Hypothesis Space and Inductive Bias

Hypothesis Space and Inductive Bias

1. Definitions

• Hypothesis space $H$: all models learner can express (e.g. all lines, all depth-5 trees).
• Inductive bias: assumptions that rank hypotheses (prefer simpler, smoother, sparser).

Without bias, infinitely many $f$ fit train data — no reason to pick one for test.

2. Examples of Bias

Model	Bias
Linear regression	$y$ is linear in $x$
$k$-NN	nearby $x$ share $y$
Decision tree (limited depth)	axis-aligned, simple splits win
Ridge ($L₂$)	small weights win
CNN	locality + translation invariance

3. Size Matters

Small $H$ → high bias (may underfit). Huge $H$ → high variance (may overfit). Validation picks middle.

4. No Free Lunch (one line)

Averaged over all possible problems, no learner beats random — gains come from bias matching the real problem.

5. Exam Points

• Define $H$ + bias with example — 5 marks, very frequent.
• How bias controls overfitting — 3 marks, frequent.
• No Free Lunch statement — 2 marks.