Linear Classification and Advanced Methods · Machine Learning

Bayes Optimal Classifier

1. Definition

Predict the class with highest true posterior $p(y|x)$ . No other classifier using same features can beat its expected accuracy — it is the theoretical ceiling.

y* = \text{argmax}_{c} p(y=c|x)

Error of $y*$ = Bayes error (irreducible noise overlap).

2. Why Unreachable

True $p(y|x)$ unknown — we estimate it (Naive Bayes, logistic, nets). Gap to Bayes error = modelling + estimation error.

3. Example

p(\text{spam}|x) = 0.7, \quad p(\text{ham}|x) = 0.3 → \text{predict spam}

Even optimal errs 30% here — overlapping classes mean no perfect classifier exists.

4. Use

Benchmark: 99% Bayes-optimal is excellent if Bayes error is 1%; poor if Bayes error is 0.01%. Justifies more features (lowers Bayes error) vs better model (closes gap).

5. Exam Points

Define Bayes optimal + Bayes error — 5 marks, very frequent.
Why no classifier beats it — 3 marks, frequent.
Irreducible vs reducible error — 2 marks.

Budget AppAndroid App

Pocketly on Android

Log daily food, travel, and hostel expenses instantly. Set monthly spending limits and see where your money goes without complex spreadsheets.

Install Pocketly

Previous topicDecision Trees Next topicNaive Bayes

← All Machine Learning modules