Experimental Framework for Evaluating a Classification Model

The aim is to measure how well the model will work on unseen data, reliably and fairly.

Step 1: Prepare the data

Clean the data, handle missing values, encode and scale features. Fix a random seed so results can be repeated.

Step 2: Training and testing datasets

• Hold-out: split the data, for example 70% training / 30% testing (or 80/20). Use a stratified split so the class proportions are the same in both sets.
• Training set: used to build (fit) the model.
• Validation set (optional): used to tune hyperparameters (e.g. tree depth).
• Test set: used only once at the end; never used for training or tuning, to avoid leakage.

Step 3: Cross-validation

k-fold cross-validation (k = 10 is common):

1. Split the data into k equal folds.
2. Train on k-1 folds, test on the remaining fold.
3. Repeat k times so every fold is used once for testing.
4. Average the k scores.

This gives a more reliable estimate than a single split and reduces overfitting. Variants: stratified k-fold, leave-one-out (small data) and bootstrap.

Step 4: Confusion matrix

	Predicted Positive	Predicted Negative
Actual Positive	TP	FN
Actual Negative	FP	TN

Step 5: Evaluation measures

Measure	Formula	Meaning
Accuracy	(TP + TN) / (TP + TN + FP + FN)	Overall fraction correct; misleading for imbalanced classes
Precision	TP / (TP + FP)	Of the predicted positives, how many are truly positive
Recall (sensitivity, TPR)	TP / (TP + FN)	Of the actual positives, how many were found
F1-score	2 x Precision x Recall / (Precision + Recall)	Harmonic mean of precision and recall
Specificity	TN / (TN + FP)	Of the actual negatives, how many were correct
False positive rate (FPR)	FP / (FP + TN)	Used on the ROC curve

Example: TP = 40, FP = 10, FN = 5, TN = 45 (100 instances)

• Accuracy = 85/100 = 0.85
• Precision = 40/50 = 0.80
• Recall = 40/45 = 0.889
• F1 = 2(0.80)(0.889)/(0.80 + 0.889) = 0.842

Step 6: ROC-based evaluation

The ROC curve plots TPR (y-axis) against FPR (x-axis) at different classification thresholds.

• A curve close to the top-left corner means a good model.
• The diagonal line is random guessing.
• AUC (area under the curve) summarizes performance: 1.0 is perfect, 0.5 is random.
• ROC is useful for comparing models and for imbalanced data, together with precision-recall curves.

Step 7: Compare and report

Compare against a baseline (such as majority class), compare several models on the same folds, report the mean and standard deviation of the metrics, and check with a significance test if needed.

How ensemble methods improve predictive performance

An ensemble combines several models so that the combined prediction is better than any single model.

Method	Idea	Effect
Bagging (e.g. Random Forest)	Train models on bootstrap samples; vote or average	Reduces variance and overfitting
Boosting (e.g. AdaBoost, Gradient Boosting)	Train models in sequence; each focuses on the previous errors	Reduces bias, gives high accuracy
Stacking	A meta-model learns to combine the outputs of different models	Uses the strengths of diverse models

Ensembles work because different models make different errors, and combining them cancels many of those errors. Ensembles should themselves be evaluated with the same cross-validation framework.

Conclusion

A sound framework uses a clean train/test split, cross-validation, several metrics (accuracy, precision, recall, F1, ROC-AUC) and, where needed, ensembles, to get a trustworthy estimate of how the model will perform in practice.