Under-fitting and Over-fitting of a Classification Model

Over-fitting (Overfitting)

Overfitting occurs when a classification model fits the training data too well — including its random noise, outliers, and accidental patterns — at the cost of generalisation to new/unseen data. Such a model typically has very low (even zero) training error but high test error.

When does it happen?

• The model is too complex for the underlying problem (e.g., a very deep decision tree, a high-degree polynomial, too many neural-network hidden units).
• Training continues too long, so the learner starts memorising noise.
• The training set is small and noisy relative to the model's capacity.

Under-fitting (Underfitting)

Underfitting occurs when the model is too simple to capture the genuine structure of the data, so it fails to learn even the training set: both training and test errors are high (a high-bias situation).

When does it happen?

• The model is too simple (e.g., a linear model for a curved relationship, a one-leaf tree).
• Too few relevant attributes are provided, or the data are over-aggregated/discretized.
• Training is stopped too early, or the wrong model class is used.

          Training error   Test error
Overfit:     very low           high
Underfit:     high              high
(good model): low-mid          low-mid (small gap)

Diagnosis & remedies:

• Overfit: reduce model complexity, use cross-validation, prune trees, add more data, apply regularisation, early stopping, or ensemble (bagging/random forest).
• Underfit: increase model complexity, engineer/add more relevant features, choose a richer model class, and ensure training runs long enough.

Both problems stem from mismatch between model complexity and the data's true complexity — too complex ⇒ overfit; too simple ⇒ underfit.

Analogy

Fitting a curve to measurements: overfit = wiggly curve passing exactly through every noisy point (terrible on new points); underfit = a straight line through strongly curved data (misses the trend).