Accuracy and Error Measures

1. Classifier Accuracy, Error Rate and the Confusion Matrix

For a test set of $n$ tuples (binary classes: positive P, negative N):

\text{Accuracy} = \frac{TP+TN}{TP+TN+FP+FN},\qquad \text{Error rate} = 1-\text{Accuracy} = \frac{FP+FN}{n}

Confusion / contingency matrix

	Predicted Positive	Predicted Negative	Total
Actual Positive	TP (true positive)	FN (false negative)	P
Actual Negative	FP (false positive)	TN (true negative)	N

Term	Definition
Sensitivity / Recall / True Positive Rate	$\dfrac{TP}{P}=\dfrac{TP}{TP+FN}$
Specificity / True Negative Rate	$\dfrac{TN}{N}=\dfrac{TN}{TN+FP}$
Precision	$\dfrac{TP}{TP+FP}$
F1 / F-score	$\dfrac{2 \times precision \times recall}{precision + recall}$
False Positive Rate	$\dfrac{FP}{N}=1-$ specificity
False Negative Rate	$\dfrac{FN}{P}=1-$ sensitivity

Worked example — 200 test tuples, 120 positive, 80 negative. Classifier predicts 105 positive of which 95 are correct, and 95 negative of which 70 are correct:

TP=95,\ FP=10,\ FN=25,\ TN=70

\text{Accuracy}=\frac{95+70}{200}=82.5\%,\quad \text{Error}=17.5\%,\quad \text{Sensitivity}=\frac{95}{120}=79.2\%,\quad \text{Specificity}=\frac{70}{80}=87.5\%

\text{Precision}=\frac{95}{105}=90.5\%,\qquad F_1=\frac{2(0.905)(0.792)}{0.905+0.792}=0.845

2. Which Measure to Use?

Accuracy alone is misleading with imbalanced classes. For 1000 tuples with 990 negative and 10 positive, a "always negative" classifier has 99% accuracy but 0% recall and 0% F1.

Rare disease / fraud detection → maximise recall, accept lower precision.
Spam filtering / credit approval → emphasise precision.
Balanced view → F1 or AUC.
Cost-sensitive → weighted error:

E_{\text{cost}}=\sum_{ij} C(i,j)\cdot \text{count}(\text{predicted } i,\ \text{actual } j),\qquad \text{Cost per tuple}=\frac{E_{\text{cost}}}{n}

3. ROC Curves and AUC

Plot TPR (sensitivity) against FPR (1 − specificity) while varying the classification threshold; a perfect classifier is the point (0,1) and a random guess is the diagonal.

\text{AUC} = \int_{0}^{1}\text{TPR}\,d(\text{FPR}),\qquad 0.5 = \text{random},\ 1.0 = \text{perfect}

 TPR
 1.0 |---------___------  model B (better, larger AUC)
     |      __/
     |    _/  ... model A (moderate)
     |  _/   /
 0.5 | /   /  ---- random classifier (diagonal)
     |/   /
 0.0 +-------------------- FPR
     0.0        0.5      1.0

Other tools: precision-recall curve, lift chart (cumulative gain), cost curve.

4. Measures for Numeric Predictors

\text{MAE}=\frac{1}{n}\sum_{i}|y_i-\hat y_i|,\qquad \text{MSE}=\frac{1}{n}\sum_i (y_i-\hat y_i)^2,\qquad \text{RMSE}=\sqrt{\text{MSE}}

\text{Relative error}=\frac{|y_i-\hat y_i|}{|y_i|},\qquad \text{RMSE}_{rel}=\frac{\text{RMSE}}{\bar y},\qquad R^{2}=1-\frac{\sum(y_i-\hat y_i)^2}{\sum (y_i-\bar y)^2}

MSE is sensitive to outliers (squared); MAE is more robust. RMSE has the same unit as the target, so it is more interpretable.

5. Estimators of Accuracy — Sampling and Confidence Intervals

Accuracy measured on a finite test set is only an estimate. If the model makes $x$ correct predictions out of $n$ independent trials, the error follows a binomial distribution whose standard deviation is

\text{stddev (error)} = \sqrt{\frac{acc\,(1-acc)}{n}}

$100(1-\alpha)$ % confidence interval for accuracy:

acc \pm z_{(1-\alpha/2)}\sqrt{\frac{acc(1-acc)}{n}}\qquad (z \approx 1.645\ \text{for }90\%,\ 1.96\ \text{for }95\%,\ 2.58\ \text{for }99\%)

Numerical example — accuracy 0.85 on 200 tuples: $\sigma=\sqrt{0.85\times0.15/200}=0.0252$ ; the 95% CI is $0.85 \pm 1.96(0.0252) = 0.85\pm0.049$ , i.e. [80.1%, 89.9%]. Larger test sets give tighter intervals ( $\sigma \propto 1/\sqrt{n}$ ).

Comparing two classifiers honestly is harder than it looks — the tests are not independent (same training/test data), so use paired methods:

Method	Idea
McNemar's test	Compares the two classifiers on a $2\times2$ table of disagreement counts $f_{01}, f_{10}$ ; $\chi^2=\frac{(\|f_{01}-f_{10}\|-1)^2}{f_{01}+f_{10}}$ with 1 d.o.f., or an exact binomial test
$k$ -fold cross-validated paired $t$ -test	Compare the per-fold differences of error rates
Sign test / Wilcoxon signed-rank	Non-parametric comparisons

6. Occam's Razor, Bias–Variance and Model Cost

Between two models with equal test error, prefer the simpler one (fewer parameters/rules/leaves) — it is more likely to generalise.

\mathbb{E}[\text{error}]=\text{Bias}^2+\text{Variance}+\text{noise},\qquad \text{Underfit} \Rightarrow \text{high bias},\quad \text{Overfit} \Rightarrow \text{high variance}

Expected error, or a pessimistic estimate used for pruning a tree (with $f=\text{error at node}$ , $N$ = tuples at the node, $k$ = number of leaves):

e=\frac{f+\frac{k}{2}}{N},\qquad \text{or Laplace-corrected } \frac{f+1}{N+k}

7. Other Useful Measures

Measure	Formula	Note
Lift	$\frac{P(y\|X)}{P(y)}$	>1 means the rule improves on the base rate
Kappa (Cohen)	$\frac{acc-P_{rand}}{1-P_{rand}}$	Agreement corrected for chance
Geometric mean	$\sqrt{sensitivity \times specificity}$	Robust for imbalance
Matthews correlation (MCC)	$\frac{TP\cdot TN - FP\cdot FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}$	Balanced single number (−1..+1)
Brier score	$\frac{1}{n}\sum (p_i-y_i)^2$	Calibration of predicted probabilities
Log loss	$-\frac{1}{n}\sum[y_i\ln p_i + (1-y_i)\ln(1-p_i)]$	Penalises confident wrong probabilities

8. Exam Points

Build a confusion matrix from given data and compute accuracy, error, sensitivity, specificity, precision, F1 — 10 marks (very frequent numerical).
Explain ROC and AUC with a sketch — 5 marks.
Compute a confidence interval for accuracy and interpret it — 5 marks.
Explain MSE, RMSE, MAE and $R^2$ for predictors — 5 marks.
Why is accuracy misleading for imbalanced data? — 3 marks.

Accuracy and Error Measures