Classification by Back Propagation

1. Neural Networks for Classification

A neural network is a set of connected input/output units where each connection has a weight. Learning adjusts the weights so the network predicts the correct class label. The most popular algorithm for training multilayer feed-forward networks is back propagation (a gradient-descent method, generalised delta rule).

2. Network Structure

  INPUT LAYER        HIDDEN LAYER(S)          OUTPUT LAYER
   (attributes)      (non-linear features)    (one unit per class)
     x1 o --------\
                  \   w_ij     o  h1
     x2 o ---------+-----------> o  h2 ----> o  y1 (class A score)
                  /   +bias     o  h3  \       
     x3 o --------/                     \----> o  y2 (class B score)
   bias o --(always 1)---> feeds every unit of the next layer

Input units = one per attribute (numeric inputs usually normalised to [0,1]).
Hidden layers give the network its non-linear modelling power; one hidden layer is usually sufficient.
Output units = one per class; the highest-scoring output unit gives the predicted class (with two outputs, a single unit with threshold 0/1 may be used).

3. Forward Pass

For each unit $j$ , with inputs $O_i$ and weights $w_{ij}$ (bias treated as a weight on a constant input 1):

I_j=\sum_i w_{ij}O_i,\qquad O_j=\frac{1}{1+e^{-I_j}}\quad(\text{sigmoid / logistic activation})

Logistic properties: output range $(0,1)$ , monotonic, and conveniently

\frac{dO_j}{dI_j}=O_j\,(1-O_j)

Other activations: tanh (zero-centred), ReLU (deep nets), softmax at the output layer for multi-class probabilities.

4. Backward Pass — Error Back Propagation

For an output unit $j$ with true target $T_j$ and computed $O_j$ :

\text{Err}_j=O_j\,(1-O_j)\,(T_j-O_j)

For a hidden unit $j$ :

\text{Err}_j=O_j\,(1-O_j)\sum_k \text{Err}_k\,w_{jk}

Weights are updated (with learning rate $l$ , momentum $\alpha$ ) using the error propagated from the layer above:

w_{ij}\leftarrow w_{ij}+l\cdot \text{Err}_j\cdot O_i,\qquad \Delta w_{ij}(n)=\underbrace{l\,\text{Err}_jO_i}_{\text{current}}+\underbrace{\alpha\,\Delta w_{ij}(n-1)}_{\text{momentum}}

Full training algorithm

Initialise weights to small random numbers (-1..1 or ~0)
repeat for each training tuple:
    // forward
    for each input unit:   O_i = x_i          (normalised)
    for each hidden/output unit j: I_j = Σ w_ij O_i ; O_j = 1/(1+e^{-I_j})
    // backward
    for each output unit j: Err_j = O_j(1-O_j)(T_j - O_j)
    for each hidden unit j: Err_j = O_j(1-O_j) Σ_k Err_k w_jk
    for each weight w_ij:   w_ij += l*Err_j*O_i
until (termination condition)   // e.g. epoch error < threshold, or epochs > max

Normalise inputs to [0,1] first; the output encoding for $k$ classes is a 1-of- $k$ vector, e.g. [1,0] for class A.

5. Worked Numerical (single-unit demo)

Attribute $x=0.6$ , weight $w=0.5$ , bias weight $w_0=0.1$ , target $T=1$ , learning rate $l=0.3$ .

I=0.1+0.5\times0.6=0.40,\qquad O=\frac{1}{1+e^{-0.4}}=0.599

\text{Err}=O(1-O)(T-O)=0.599\times0.401\times(1-0.599)=0.0963

w_0 \leftarrow 0.1+0.3(0.0963)(1)=0.1289,\qquad w \leftarrow 0.5+0.3(0.0963)(0.6)=0.5173

The next pass gives $O$ closer to the target (0.62 vs 0.599 for the old weights) — the error decreases.

6. Design Issues

Issue	Guidance
Number of hidden layers/units	Usually 1 hidden layer; units $\approx$ (attributes + classes)/2 or tuned by experiment; too many → overfitting and slow training
Learning rate $l$	Too high → oscillation/divergence; too low → very slow; typical 0.05-0.5, often reduced over epochs
Momentum $\alpha$	Speeds convergence and damps oscillation (typical 0.9)
Initial weights	Small random values; identical weights break symmetry learning
Input normalisation	Scale to [0,1] (or z-score) — essential for the sigmoid
Stopping criteria	Epoch error below threshold, validation error stops improving, max epochs
Local minima	Error surface is non-convex → may reach a local minimum; use momentum, multiple random restarts, simulated annealing/genetic methods, or modern optimisers (Adam, RMSProp)

7. Advantages and Limitations

Advantages

High tolerance to noisy data; performs well when attributes are correlated, redundant or non-linearly related.
Good accuracy — often highest after tuning; naturally handles many attributes; incremental/online learning possible; parallelisable.
Well suited for continuous-valued inputs/outputs, so it also does numeric prediction (regression).

Limitations

Long training time and many parameters to tune (network topology, learning rate, momentum).
Poor interpretability — "black box"; weights and hidden units have no clear meaning, but hidden-unit clustering can reveal some structure.
Requires many design choices and normalised numeric input; categorical attributes need encoding (§ one-hot).
Can be sensitive to overfitting and to the choice of random initial weights; theoretical guarantees are weak (empirical risk minimisation).

8. Support-Vector-Machine Contrast

Back propagation uses many local minima and heuristic tuning; SVM has a global optimum for the margin and, with kernels, implicitly maps to a high-dimensional space — see the next topic.

9. Exam Points

Draw a multilayer feed-forward network and explain the back propagation algorithm with the forward/backward phase formulas — 10 marks (very frequent).
Write the pseudo-code of back propagation; explain the sigmoid and its derivative — 7 marks.
Explain the role of momentum term and the local minimum problem — 5 marks.
State advantages and limitations of neural networks for classification — 5 marks.
Small numerical: compute $\text{Err}_j$ and update a weight for one output unit — 5 marks.

Classification by Back Propagation