Evaluation Metrics (Classification):

Let TP, TN, FP, FN from confusion matrix.

• Accuracy $= (TP+TN)/(TP+TN+FP+FN)$ — fraction correct. Fails on imbalanced data.
• Precision $= TP/(TP+FP)$ — of predicted positives, how many truly positive. Important when FP costly (spam filter).
• Recall / Sensitivity / TPR $= TP/(TP+FN)$ — of actual positives, how many found. Important when FN costly (cancer detection).
• F1-score $= 2PR/(P+R) = 2TP/(2TP+FP+FN)$ — harmonic mean, balances P & R.
• Others: Specificity $=TN/(TN+FP)$, ROC-AUC.

Numerical Example:
Suppose 100 patients: TP=40, FP=10, FN=20, TN=30

• Accuracy $=(40+30)/100=0.70$
• Precision $=40/(40+10)=0.80$
• Recall $=40/(40+20)=0.667$
• F1 $=2*0.8*0.667/(0.8+0.667)=0.727$

Regression metrics: MSE $=\frac1n\sum(y-\hat y)^2$, RMSE, MAE, $R^2$.

Importance of Cross-Validation:

Hold-out single split is noisy. k-Fold CV (usually $k=5,10$) partitions data into $k$ folds, trains on $k-1$, validates on remaining, rotates, averages error: $CV_k = \frac1k\sum_{i=1}^k Err_i$.

Benefits:

1. Unbiased estimate of generalization: Uses all data for both training and validation, reduces variance vs single split.
2. Model selection / hyperparameter tuning: Choose $\lambda, k, C$ without touching test set.
3. Detects overfitting: Large gap train vs CV error signals overfit.
4. Stratified CV: preserves class proportions for imbalance.

After CV, retrain on full training data, report final performance on held-out test set never used in CV. LOOCV ($k=n$) unbiased but expensive; $k=10$ is good bias-variance trade-off.