Basic Architecture (Concept) of Artificial Neural Networks — Beginner-Friendly Notes

1. From Perceptron to Network

One perceptron = one linear separator. Stack many → artificial neural network (ANN) that can learn non-linear boundaries (solves XOR).

2. Three Layers (conceptual — what exam expects)

Input layer (features) → Hidden layer(s) (learned features) → Output layer (prediction)

• Input layer: raw features $x$ (e.g., pixels, sensor values) — no computation, just distribution.
• Hidden layer: each neuron computes $z = w \cdot input + b$, then non-linear activation (ReLU $max(0,z)$, sigmoid, tanh). Learns intermediate representations.
• Output layer: produces final prediction — e.g., softmax for classification, linear for regression.

Key idea: Depth (number of hidden layers) and width (neurons per layer) give capacity.

3. How Data Flows (forward pass — concept)

1. Input $x$ flows to hidden neurons, each computes activation.
2. Activations flow to next layer, etc.
3. Output layer gives $\hat{y}$.
   No loops here — this is a feed-forward network (recurrent nets add loops, not in syllabus).

4. How It Learns (concept — not heavy maths)

• Define loss $L(y, \hat{y})$ (e.g., MSE, cross-entropy).
• Compute gradient of loss w.r.t. weights via backpropagation (chain rule backwards through layers).
• Update weights by gradient descent: $w \leftarrow w - \eta \cdot \partial L/\partial w$.
• Repeat over many epochs until loss plateaus.

You only need to state "learns by backprop + gradient descent" — no derivation needed for this introductory topic.

5. Why Non-Linear Activation Is Essential

Without it, stacking linear layers = single linear layer → still cannot learn curves/XOR. Activation breaks linearity.

6. One-Line Summary

A feed-forward ANN stacks input, hidden and output layers of $w \cdot x + b$ neurons with non-linear activations, learning via backpropagation and gradient descent to represent non-linear functions.

7. Quick Self-Check

1. What would happen if hidden neurons used no activation (just linear)?
2. Name one activation and its role.