Prediction

Prediction

1. Classification vs Prediction

Aspect	Classification	Prediction (numeric / regression)
Output	Discrete class label (yes/no, A/B/C)	Continuous numeric value (price, temperature, demand)
Model	Classifier	Predictor / regressor
Evaluation	Accuracy, error rate, confusion matrix	MAE, MSE, RMSE, $R^2$, correlation
Example	'Will the customer default?'	'What will be the sales next quarter?'

Both are supervised learning of a mapping from attributes to a target; prediction is used where exact labels are replaced by numeric estimates.

2. Main Prediction (Regression) Techniques

(a) Linear regression — the simplest, assumes a linear relationship:

with the least-squares solution:

(b) Multiple linear regression — several independent attributes:

Weights are found by solving the normal equations $W = (X^{T}X)^{-1}X^{T}Y$; it can also be solved by gradient descent (the basis of neural networks). Categorical attributes are converted with dummy variables.

(c) Non-linear / other models

Method	Idea
Polynomial regression	$y = w_0+w_1x+w_2x^2+\dots$ — curves are fitted by treating powers as new attributes
Logistic regression	$P(y=1)=\frac{1}{1+e^{-(w_0+\sum w_ix_i)}}$ — classification and probability estimation
Generalised linear models (GLM)	Link function + distribution (Poisson, gamma) — insurance claims, counts
Regression trees (model trees)	Leaves carry a linear equation, not a class; M5 / M5' algorithms
Neural networks (MLP, RBF)	Universal approximators for non-linear regression
Support vector regression (SVR)	$\epsilon$-insensitive tube, kernels for non-linearity
k-NN regression / LWR	Locally weighted average of neighbours
Time-series models (AR, MA, ARIMA, exponential smoothing, Holt-Winters)	Trend, seasonality and autocorrelation in sequences
Ensembles (bagging, boosting)	Combine many weak regressors (random forest regression, gradient boosting)

3. Worked Least-Squares Example

Data: $(x,y) = (1,2), (2,4), (3,5), (4,8)$ → $\bar x = 2.5$, $\bar y = 4.75$.

Model: $y = 1.9x$. Prediction at $x=5$ → $y = 9.5$. Residuals: $-0.1,\ +0.2,\ -0.7,\ +0.4$ (small, so a good linear fit).

4. Accuracy of a Predictor

Example — actual 20, 30, 40, 50; predicted 21, 28, 42, 48:
MAE $= (1+2+2+2)/4 = 1.75$; MSE $= (1+4+4+4)/4 = 3.25$; RMSE $= 1.80$; mean $\bar y = 35$, so relative RMSE $\approx 5.1\%$.

5. Handling Overfitting and Underfitting

• Overfitting — the model fits noise (too many attributes/parameters, too deep a tree): low training error, high test error. Fix with pruning, regularisation ($L_1$ Lasso, $L_2$ Ridge), more data, cross-validation.
• Underfitting — the model is too simple: high error on both. Fix with more/better features, a more flexible model.

Regularisation shrinks coefficients:

6. Choosing a Model / Avoiding Overfitting

Split data into training / validation / test; use cross-validation for tuning; prefer the simplest model that fits (Occam's razor); check the bias-variance trade-off; use domain constraints.

7. Applications of Prediction

Sales and demand forecasting, financial time-series forecasting, price and risk estimation, medical dosage/prognosis, weather and energy-load prediction, click-through/response prediction, predictive maintenance, customer lifetime value.

8. Exam Points

• Explain linear and multiple regression for prediction with the least-squares formulas — 10 marks (frequent).
• Numerical: fit a regression line from given data and predict — 5-7 marks (very frequent).
• Compute MAE, MSE, RMSE, relative error and $R^2$ for predicted vs actual values — 5 marks.
• Difference between classification and prediction — 3-5 marks.
• Discuss overfitting and regularisation — 5 marks.