Overview of Regression and Classification Concepts — Beginner-Friendly Notes

1. Both Are Supervised — The Split

• Regression: predict continuous number $y \in \mathbb{R}$ — e.g., price, temperature, GPA.
• Classification: predict discrete label $y \in \{classes\}$ — e.g., spam/ham, disease yes/no, cat/dog.

Same pipeline, different $y$ and loss function.

2. Linear Regression — The Simplest Regression

Model: $\hat{y} = w \cdot x + b$ (for vector $x$, $\hat{y} = w^T x + b$).

Loss: Mean Squared Error $MSE = \frac{1}{n} \Sigma (y_i - \hat{y}_i)^2$ — minimise it → find best $w,b$ (closed-form or gradient descent).

Intuition: fits best line/plane through points. Assumption: linear relation; fails on curves (need polynomial/features or other models).

Output: continuous; evaluation via $RMSE$, $MAE$, $R^2$.

3. k-NN (k-Nearest Neighbours) — Simple Classification & Regression

Idea: "Tell me who your neighbours are, I'll tell you who you are."

• Store all training points.
• For new $x$, find k closest training points (distance = Euclidean usually).
• Classification: vote among neighbours' labels (majority).
• Regression: average neighbours' $y$.

Pros: no training, intuitive, works for both tasks.
Cons: slow at prediction (scans all data), sensitive to scaling and $k$, curses with high dimensions.

Pick $k$: odd to avoid ties; small $k$ → noisy, flexible; large $k$ → smooth, biased. Use validation.

4. Decision Trees — Tree of Rules

Idea: recursively split data on feature thresholds to make leaves as pure as possible.

Is income > 50k? 
  No → Is student? → leaf: No / Yes
  Yes → leaf: Yes

Splitting criterion: information gain (entropy drop) or Gini impurity — pick split that most separates classes.

For regression: leaf predicts mean $y$ of its points; split minimises variance (MSE).

Pros: interpretable, handles mixed types, non-linear.
Cons: overfits easily → needs pruning or ensemble.

5. Small Comparison Table (exam favourite)

	k-NN	Decision Tree	Linear Regression
Type	non-parametric, instance-based	non-parametric, rule-based	parametric
Training	none (lazy)	builds tree	fits weights
Handles non-linear	yes (with enough data)	yes	no (linear only)
Interpretability	low	high	medium (weights)
Use when	small data, baseline	explainable rules needed	linear trend suspected

6. Choosing (beginner heuristic)

• Continuous $y$ + linear trend → linear regression.
• Need simple explainable classifier → decision tree.
• Need simple strong baseline + small data → k-NN.

7. One-Line Summary

Regression predicts continuous $y$ (e.g., linear regression with MSE) while classification predicts discrete labels via instance voting (k-NN) or recursive splitting (decision trees), each trading simplicity, interpretability and capacity for non-linear patterns.

8. Quick Self-Check

1. Predict price from area — regression or classification?
2. Why does k-NN struggle if features are on very different scales?