Artificial Neural Networks + Back Propagation

Artificial Neural Networks + Back Propagation

1. Neuron

$σ$ = sigmoid / ReLU / tanh. ReLU $(max(0,s))$ is default (fast, no saturation for $s>0$). Network stacks layers: $X → hidden → output$.

2. Forward Pass (example 2-2-1)

h = relu(X W₁ + b₁)   # n × 2
p = sigmoid(h W₂ + b₂) # n × 1
loss = cross-entropy(p, y)

Universal approximation: one wide hidden layer can fit any continuous function (given enough units).

3. Back Propagation (chain rule)

Gradients flow output → input, reusing intermediate terms:

Written with $dL/d$ in words to stay rendering-safe. In code: cache forward activations, multiply by local derivatives backward.

4. Training Tips

Standardize inputs, init small random, mini-batch + Adam, dropout / weight decay to regularize, early stop on validation.

5. Exam Points

• Neuron + forward equations — 5 marks, very frequent.
• Backprop chain-rule idea + one-layer derivation — 7 marks, frequent.
• ReLU vs sigmoid, overfitting fixes — 3 marks, frequent.