Backpropagation Algorithm — Weight Update via Gradient Descent:

For MLP with $L$ layers, loss $E=\frac12\sum(y-o^{(L)})^2$ or cross-entropy.

Steps:

1. Forward: $net_j^{(l)}=\sum_i w_{ji}^{(l)}o_i^{(l-1)}+b_j^{(l)}$, $o_j^{(l)}=f(net_j^{(l)})$ where $f$ is sigmoid/ReLU. Propagate to output.

2. Backward (output layer $L$): Error term $\delta_j^{(L)} = (y_j - o_j^{(L)})f'(net_j^{(L)})$ (for MSE) or $(y_j - o_j)f'$ generally $ \delta^{(L)} =
   abla E \odot f'(net)$.

3. Hidden layers: $\delta_j^{(l)} = f'(net_j^{(l)})\sum_k \delta_k^{(l+1)} w_{kj}^{(l+1)}$ — error weighted by downstream weights.

4. Gradients: $\frac{\partial E}{\partial w_{ji}^{(l)}} = -\delta_j^{(l)} o_i^{(l-1)}$, $\frac{\partial E}{\partial b_j^{(l)}} = -\delta_j^{(l)}$

5. Update (SGD): $w_{ji}^{(l)} := w_{ji}^{(l)} + \eta \delta_j^{(l)} o_i^{(l-1)}$, $b_j^{(l)}:=b_j^{(l)}+ \eta \delta_j^{(l)}$

For batch: average over examples, momentum $\Delta w(t)= -\eta
abla E + \alpha\Delta w(t-1)$, Adam uses adaptive$\eta$.

Role of Gradient Descent: Iteratively moves weights opposite gradient $w:=w-\eta
abla E$to minimize loss, error surface descent. Learning rate$\eta$controls step size; too large diverges, too small slow. Backprop makes gradient computation$O(W)$ via chain rule, enabling deep learning.

Variants: Vanilla SGD (per sample), mini-batch SGD, batch GD. Improvements: momentum, RMSProp, Adam, weight decay (L2).