Backpropagation Rule — How it Helps Learning in Neural Networks:

Neural network learns by minimizing loss $L = \frac12\sum (y-\hat y)^2$ or cross-entropy via gradient descent. Backpropagation efficiently computes $\partial L/\partial w$ for all weights using chain rule, propagating error backward.

Steps:

1. Forward pass: For each layer $l$: $z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)}$, $a^{(l)}=\sigma(z^{(l)})$. Compute loss $L$.
2. Backward pass:
   • Output error: $\delta^{(L)} =
     abla_a L \odot \sigma'(z^{(L)})$
   • Propagate: $\delta^{(l)} = (W^{(l+1)T}\delta^{(l+1)})\odot \sigma'(z^{(l)})$
   • Gradients: $\frac{\partial L}{\partial W^{(l)}} = \delta^{(l)} a^{(l-1)T}$, $\frac{\partial L}{\partial b^{(l)}} = \delta^{(l)}$
3. Update: $W := W - \eta \frac{\partial L}{\partial W}$, $b:=b-\eta \frac{\partial L}{\partial b}$

Why crucial:

• Makes training deep networks computationally feasible $O(|W|)$ instead of naive $O(|W|^2)$.
• Enables learning of hierarchical features (early layers edges, deeper semantic).
• Works with any differentiable activation (sigmoid, ReLU, tanh).
• Basis for SGD, Adam optimization.

Example: 2-layer MLP with MSE: error at output $\delta = (\hat y - y)\cdot \sigma'(z)$, hidden error $\delta_h = W_2^T \delta \cdot \sigma'(z_h)$ — weight updates proportional to contribution to error (credit assignment). Without backprop, no efficient credit assignment across layers would be possible.

Extensions: Batch norm, dropout, gradient clipping to stabilize backprop, avoid vanishing/exploding gradients.