What are Over-fitting and Under-fitting? When do they happen?

Over-fitting

Over-fitting — the model learns the training data too precisely, capturing random noise, outliers, and chance coincidences in addition to the true signal. Result: near-zero training error but poor accuracy on new/unseen data (high variance).

When does it occur?

• Model too complex — too many parameters/leaves compared with the amount of data (e.g., a fully-grown, unpruned decision tree; a very deep neural net; a high-order polynomial).
• Training for too many epochs/iterations, especially on small, noisy data sets.
• Using too many irrelevant/multivalued attributes so the learner finds spurious patterns.

Under-fitting

Under-fitting — the model is too simple to represent the underlying relationship, so it fails to capture even the main trend: both training and test errors are high (high bias).

When does it occur?

• Model too simple for the data (e.g., a linear model fitted to clearly non-linear data; a single-leaf tree).
• Not enough useful features/attributes; inappropriate discretization.
• Training stopped too early or the wrong type of model is chosen.

          Training   Test
Overfit:     low       high      (low bias, high variance)
Underfit:   high       high      (high bias)
Good:       low-mid    low-mid   (balanced)

Why (the bias–variance trade-off)

Overfitting is driven by increased variance: excessively flexible models respond to every random fluctuation. Underfitting is driven by high bias: overly rigid models cannot express the data's true relation. The ideal model complexity balances both.

When they matter / detection

The clearest sign is a large gap between training and validation (test) accuracy — low train / high test ⇒ over-fit; both high ⇒ under-fit. We detect and fix them with cross-validation, pruning /regularisation, adjusting model complexity, and gathering more or better data.