Linear Regression

Linear Regression

1. Model

Predict continuous $y$ as linear function:

Stacked form with bias column: $y_{pred} = X w$.

2. Least Squares + Normal Equation

Minimize sum of squared residuals. Gradient zero gives:

Geometrically: projection of $y$ onto column space of $X$.

points (1,3),(2,5),(3,7) → y = 2x+1, zero residual
X has ones column; solve 2x2 system → w = [1,2]

3. Probabilistic View

Gaussian noise assumption makes least squares = maximum likelihood. Gauss-Markov: under 5 classical assumptions OLS is BLUE (best linear unbiased).

4. Fit Quality

$R²=1$ perfect, $0$ = mean baseline. Adjusted $R²$ penalizes junk features. Plot residuals vs predicted: funnel = heteroscedasticity, curve = nonlinearity.

5. Exam Points

• Derive normal equation — 5 marks, asked nearly every year.
• 3-point numerical ($y=2x+1$ style) — 5 marks.
• $R²$ + BLUE statement — 3-5 marks, frequent.