Naïve Bayesian Classification — Why "Naïve" & Main Ideas

Why is it called "naïve"?

The naïve Bayes classifier applies Bayes' theorem but makes one strong ("naïve") simplifying assumption: it assumes that all attributes are conditionally independent of one another given the class label. In other words, the value of one attribute is assumed to have no influence on any other attribute once the class is known:

$$P(X \mid C) = P(x_1, x_2, \dots, x_n \mid C) = \prod_{k=1}^{n} P(x_k \mid C)$$

This independence assumption is rarely true in real-world data — for example, "age" and "income", or "smoking" and "lung disease", are usually correlated. Because this simplifying assumption is unrealistic, the classifier is called naïve.

Major Ideas of Naïve Bayesian Classification

1. Estimate probabilities from training data. For each class $C_i$ compute:

   • the class prior $P(C_i)$ — the proportion of training tuples belonging to $C_i$;
   • the conditional probabilities $P(x_k \mid C_i)$ for each attribute value $x_k$ (from frequency counts; Gaussian density for continuous attributes).

2. Classify by the maximum-a-posteriori (MAP) rule. For a new tuple $X = (x_1, \dots, x_n)$, Bayes' theorem gives:

$$P(C_i \mid X) = \frac{P(X \mid C_i)\, P(C_i)}{P(X)}$$

Since $P(X)$ is the same for every class, assign $X$ to the class with the maximum value of $P(X \mid C_i)\, P(C_i)$:

$$\text{assign } X \text{ to } C_i \text{ that maximises } P(C_i)\prod_{k=1}^{n} P(x_k \mid C_i)$$

3. Independence assumption makes it computable — the joint probability becomes a simple product of per-attribute probabilities, so the classifier is very fast even for high-dimensional data.

4. Laplacian (add-one) smoothing — avoids a single zero conditional probability $P(x_k \mid C_i)=0$ from forcing the whole product to zero.

Advantages: extremely fast, works well on high-dimensional data such as text, robust to irrelevant attributes, probabilistically principled. Major limitation: the independence assumption is usually violated in real data, which can reduce accuracy — yet in practice (e.g., spam filtering) it still performs surprisingly well.