GMMs

GMMs

1. Model

Data = mix of $K$ Gaussians with weights $π_{k}$ (sum $=1$):

Soft clustering: point belongs to all components with probabilities (responsibilities), unlike $k$-means hard assignment.

2. Training by EM

E:

M: $μ_{k}$ = weighted mean with $γ$, $Σ_{k}$ = weighted covariance, $π_{k}$ = mean $γ$.

3. GMM vs K-means

	K-means	GMM
Assignment	hard (nearest)	soft (probabilities)
Shape	spheres	ellipses (full $Σ$)
Needs	$k$	$K$ + init (often $k$-means)
Picks $K$ by	elbow	BIC / validation likelihood

4. Practice

Init by $k$-means, regularize $Σ$ (add small $εI$) to avoid collapse on one point, restart 5-10×.

5. Exam Points

• Write mixture form + responsibility — 7 marks, very frequent.
• GMM vs $k$-means — 3-5 marks, frequent.
• Singularity fix ($εI$) — 2 marks.