Issues Regarding Classification and Prediction

Issues Regarding Classification and Prediction

1. Definitions

• Classification — builds a model (classifier) from labelled training data to predict a categorical/discrete class label for new tuples.
• Prediction — models a continuous-valued function (numeric prediction / regression) to forecast unknown or missing values.

 Training data (x, y)  --[learning algorithm]-->  MODEL  --[test data]-->  accuracy
                                                        |
                                            new unlabeled tuple --> predicted class/value

2. Two-Phase Process

Phase	Activity
Learning (training)	Analyse training tuples; build the model — classification rule, decision tree, mathematical formula, neural network
Classification / prediction (testing)	Estimate the accuracy of the model on a test set using a confusion matrix; if acceptable, deploy it to classify new tuples

Accuracy = percentage of test tuples correctly classified; for numeric prediction the error measure is typically MAE or RMSE.

3. Major Issues (with remedies)

#	Issue	Why it matters	Remedy
1	Data preparation	Dirty/missing/noisy data → poor models	Cleaning, imputation, normalisation, discretisation, feature construction
2	Relevant attribute selection (feature selection)	Irrelevant/redundant attributes confuse the learner and waste time	Information gain, Gini, $\chi^2$, wrapper/filter/embedded methods, PCA
3	Overfitting vs underfitting	Too complex model memorises noise; too simple misses structure	Pruning, cross-validation, regularisation, more data, simpler models
4	Model selection (bias-variance trade-off)	Choosing the wrong family/kernel/complexity	Validation curves, cross-validation, Occam's razor + MDL
5	Curse of dimensionality	Distances lose meaning in high dimensions; data becomes sparse	Dimensionality reduction, feature selection, kernels
6	Class imbalance / rare classes	Model biased towards the majority class	Resampling (over/under), SMOTE, cost-sensitive learning, better metrics (recall, F1, AUC)
7	Cost sensitivity	False negative (missed fraud) can cost far more than false positive	Cost matrix, weighted accuracy, threshold tuning
8	Missing / inconsistent attribute values	Many algorithms break; can bias results	Imputation, special 'unknown' branch, algorithms that tolerate missing values (C4.5, k-NN variants)
9	Comparing classifiers	Which model is genuinely better?	Statistical tests (t-test, paired t, McNemar), ROC/AUC comparison
10	Computational efficiency & scalability	Large data and many attributes; training time	Sampling, parallel/distributed algorithms, incremental learning
11	Interpretability	Users reject black boxes in regulated domains	Prefer rules/decision trees; hybrid models, explanation methods
12	Concept drift & data shift	Patterns change over time (fraud, demand)	Periodic retraining, drift detection, incremental learning, monitoring
13	Noisy & outlier data	Distorts splits/weights	Binning, smoothing, outlier detection, robust methods
14	Privacy / security	Personal data restrictions	Anonymisation, differential privacy, federated learning

4. Overfitting in Detail

• Overfitting: the model fits the training data too well (including noise) but generalises poorly → high training accuracy, low test accuracy. Cause: complex model + small/noisy sample.

• Underfitting: the model is too simple to capture the underlying pattern → poor on both training and test data.
• Bias–variance decomposition of expected squared error:

Remedies: tree pruning, cross-validation to pick complexity, regularisation (L1/L2), early stopping, ensembling, more training data.

5. Comparing Classification and Prediction

Aspect	Classification	Prediction (numeric)
Output	Discrete class label	Continuous value
Typical methods	Decision tree, Bayes, k-NN, SVM, neural nets, rules	Regression (linear/multiple), regression trees/model trees, neural nets, SVR
Accuracy measure	Accuracy, error rate, precision/recall, F1, ROC	MAE, MSE, RMSE, $R^2$, relative error
Example	Loan: approve/deny; churn: yes/no	Sales forecast, temperature, stock price, credit score

6. Ways of Speeding Up Classification (Zhang's / decision-tree options)

• Incremental / online learning — update the model as new data arrives.
• Attribute-subset selection, discretisation, sampling.
• Sampling + parallelism, density-based / compressive / class-relevance data reduction.
• Multidimensional index trees (k-d tree, ball tree) for k-NN speed-up.

7. Exam Points

• Explain the two-phase classification process (learning + testing) with a diagram — 5-7 marks.
• Discuss the issues in classification and prediction, with remedies — 10 marks (very frequent).
• Explain overfitting/underfitting with bias-variance trade-off — 5-7 marks.
• Why is attribute selection important? Compare filters, wrappers and embedded methods — 5 marks.