Overfitting and Underfitting:

Overfitting: Model learns training data too well, including noise/outliers; high variance, low bias. Memorization. $Error_{train} \ll Error_{test}$. Example: Degree-10 polynomial fitting 10 points with wiggling curve passing through all points.

Underfitting: Model too simple to capture underlying pattern; high bias, low variance. $Error_{train} \approx Error_{test}$ but both high. Example: Linear line fitting parabolic data.

Model complexity view: Train vs Validation error vs capacity (U-shaped test error). Optimal is intermediate capacity (sweet spot) where bias-variance balanced.

2D Example:

Data: Parabolic trend $y = x^2 + noise$, points near U-shape.

• Underfit (linear, degree 1): Straight line cutting through U, misses curvature → large error on both train and test.

  y
  *   *
   \ /     data *
    .      linear ___  (misses curve)
   / \
  *   *
  

• Good fit (degree 2): Parabola $y= w_0+w_1x+w_2x^2$ captures trend smoothly, generalizes.

• Overfit (degree 9+): Wiggly curve passing exactly through every noisy point, oscillates wildly between points — zero train error but huge test error.

Remedies:

• Underfit: increase complexity, add features, reduce regularization, longer training.
• Overfit: more data, regularization (L1/L2), pruning, dropout, early stopping, cross-validation, feature selection, reduce degree.

Visual: Plot $x$ vs $y$, noisy quadratic data; underfit = straight line under, overfit = wiggly interpolating, good = smooth quadratic.