Bayesian Learning — ML, MAP, Bayes Estimates, Conjugate Priors

Bayesian Learning — ML, MAP, Bayes Estimates, Conjugate Priors

1. Bayes Rule

• $p(θ)$ prior, $p(\text{data}|θ)$ likelihood, $p(θ|\text{data})$ posterior.

2. Three Estimates

Method	Objective	Notes
ML	max $p(\text{data}|θ)$	no prior, overfits when data small
MAP	max $p(θ|\text{data})$ = max likelihood × prior	regularized (prior = penalty)
Bayes (full)	mean over posterior	best average, needs integration

3. Example — Coin Bias

Heads $h=7$, tails $t=3$. ML: $7/10 = 0.7$. MAP with uniform prior (add-1 smoothing): $(7+1)/(10+2) ≈ 0.67$. MAP shrinks to prior when data few.

4. Conjugate Priors (why easy)

Prior + likelihood stay in same family → closed-form posterior. Beta + Binomial → Beta; Gauss + Gauss → Gauss. No sampling needed.

5. Exam Points

• State Bayes rule + ML vs MAP vs Bayes — 7 marks, very frequent.
• Coin numerical (ML vs MAP) — 5 marks, frequent.
• Define conjugate prior + one pair — 3 marks, frequent.