Classification & Prediction + Naïve Bayesian Classification

Classification and Prediction — Definitions

• Classification is a supervised learning process that builds a model from labelled training data and then uses it to assign new/unseen objects to discrete, predefined class labels (e.g., predict whether a loan is low-risk or high-risk). Its output is categorical.
• Prediction is the general task of estimating unknown values. When the value being predicted is numeric/continuous (e.g., predicting the price of a house), it is called prediction or regression; when the unknown value is a categorical class label, it is classification. Prediction is governed by a mean squared error style loss, whereas classification deals with accuracy/error rate.

Bayesian Classification

Bayesian classification is a statistical classification technique based on Bayes' Theorem. It assigns a tuple $X$ to the class $C_i$ for which the posterior probability $P(C_i \mid X)$ is maximum:

$$P(C_i \mid X) = \frac{P(X \mid C_i)\, P(C_i)}{P(X)}$$

Because $P(X)$ is the same for all classes, the maximum a posteriori (MAP) decision rule is:

$$\text{choose } C_i \text{ such that } P(X \mid C_i)\, P(C_i) \text{ is maximum}$$

• $P(C_i)$ — prior probability of class $C_i$ (estimated from training-data frequencies).
• $P(X \mid C_i)$ — likelihood of observing $X$ given the class.

Naïve Bayes — a Simple Example

The naïve assumption is that all attributes are conditionally independent given the class, so:

$$P(X \mid C_i) = \prod_{k=1}^{n} P(x_k \mid C_i)$$

Example: Classify a fruit by its attributes.

Fruit	Colour	Shape	Taste
Apple	Red	Round	Sweet
Apple	Green	Round	Sweet
Mango	Yellow	Oval	Sweet
Mango	Yellow	Oval	Sour

Let a new fruit have attributes $X = (\text{Colour} = \text{Yellow},\ \text{Shape} = \text{Oval})$. We compute:
$P(\text{Mango}) = 2/4 = 0.5$, $P(\text{Apple}) = 0.5$.

• $P(Yellow \mid Mango) = 2/2 = 1$, $P(Oval \mid Mango) = 2/2 = 1$
• $P(Yellow \mid Apple) = 0/2 = 0$, so $P(X \mid Apple) = 0$.

Score mango = $0.5 \times (1 \times 1) = 0.5$; apple score $= 0$. Since $0.5 > 0$, the new fruit is classified as Mango.

Advantages: simple, fast, easily handles high-dimensional data, works well in spam filtering and text classification. Limitation: the independence assumption is often violated in real data.