Hypothesis Testing

Hypothesis Testing

1. Purpose

Decide if an observed win (A 82% vs B 80%) is real or luck. Null $H₀$: no difference. Compute p-value = probability of ≥ observed gap under $H₀$. Reject $H₀$ if $p < 0.05$.

2. Test Types

Situation	Test
Proportion (accuracy)	binomial / paired McNemar on same folds
Mean over CV folds	paired t-test on fold differences
Many models	ANOVA + post-hoc (Bonferroni)

Large $|t|$ → small $p$ → significant.

3. Errors

	$H₀$ true	$H₀$ false
Keep $H₀$	correct	Type II (miss)
Reject $H₀$	Type I (false alarm, rate $α$)	power

Multiple testing inflates Type I — divide $α$ by number of tests.

4. Practice

Compare on same CV splits (paired), report mean ± std + $p$, not just best fold.

5. Exam Points

• Define $H₀$, p-value, Type I/II — 5 marks, very frequent.
• Paired t over folds steps — 5 marks, frequent.
• Multiple-comparison correction — 2 marks.