Evaluation and Cross Validation

Evaluation and Cross Validation

1. Why Not Train Error

Train error is optimistic (model saw the answers). Need held-out data for honest score.

2. Split Protocol

Train (fit) → validation (tune) → test (report once). Never tune on test.

3. Metrics

Task	Metric	Formula
Regression	MAE, RMSE

|
| Classification | accuracy, precision, recall, F1 |

|
| Ranking | AUC | area under ROC |

Confusion matrix: TP, FP, TN, FN → $P = TP / (TP + FP)$, $R = TP / (TP + FN)$.

4. $k$-Fold Cross Validation

split data into k=5 equal folds
for i=1..5: train on 4 folds, score on fold i
report mean ± std of 5 scores

Use stratified (preserve class ratio) for classification, time-split (no future → past leak) for series.

5. Exam Points

• $k$-fold steps + why — 5-7 marks, very frequent.
• Precision, recall, F1 from matrix — 5-mark numerical, frequent.
• Validation vs test role — 3 marks, frequent.