Linear Algebra

Linear Algebra

1. Why It Matters

Every dataset is a matrix $X$ with $n$ rows (examples) and $d$ columns (features). Every model is matrix math: predictions, gradients, eigen-directions.

2. Objects

Object	Shape	Example
Scalar	single number $a$	learning rate $η = 0.01$
Vector	$x$ in $R^{d}$	$x = [x₁, x₂, x₃]$
Matrix	$A$ in $R^{m × n}$	dataset $X$
Tensor	multi-way array	image $224 × 224 × 3$

Transpose flips rows/cols. Dot product sums pairwise products.

3. Norms (size of vector)

• $L₂$ :

• $L₁$ : sum of $|x_{i}|$
• Example: $x = [3, 4]$ → $L₂ = 5$, $L₁ = 7$.

4. Matrix Rules

If $A$ is $m × n$ and $B$ is $n × p$, then $C = A B$ is $m × p$. Rules: $A B ≠ B A$ in general,

Never compute an explicit inverse in code — use QR / SVD solve.

A = [1 2 | 3 4], x = [5, 6] → A x = [17, 39]
computed as: 1*5+2*6=17, 3*5+4*6=39

5. Rank, Determinant, Eigen, SVD

• Rank: independent columns. Low rank → redundant features.
• det = 0 → singular, no unique solve (needs Ridge).
• Eigen:

— $v$ only stretched by $λ$. Example $λ = 5, 2$ for $[4 1 | 2 3]$.

• SVD:

— best rank-$k$ approximation, backbone of PCA and recommenders.

• Gradient of squared loss uses transpose:

6. Exam Points

• Compute norm, inverse, eigenvalues of 2×2 — 5 marks, numerical favourite.
• State $(A B)^{T}$ rule, $A B ≠ B A$ — 2 marks.
• SVD form + one ML use — 5 marks.
• Link det = 0 → regularization — 3 marks.