Classification and Prediction — Definitions & Major Issues

Definitions

Classification is a supervised learning process. A model (classifier) is first built from a labelled training set — tuples whose class labels are known — using a learning algorithm (e.g., decision tree, naïve Bayes, neural network, SVM). The learned classifier is then used to assign new, unseen tuples to one of the discrete, predefined classes. It is a two-step process: model construction (training) followed by model usage (classification). Its output is categorical, and its quality is measured by classification accuracy (the percentage of correctly classified test tuples).

Prediction is the broader task of estimating unknown values from historical data.

• When the unknown value is numeric/continuous, it is called prediction or regression (e.g., predicting house prices) and is judged by error measures such as mean squared error.
• When the unknown value is categorical, the task reduces to classification.

Thus classification is a special case of prediction for discrete labels, while regression is prediction for continuous values.

Major Issues in Classification and Prediction

Three major groups of issues must be addressed:

1. Preparing the data:

• Missing/incomplete data — tuples with missing attribute values may need filling, or the tuple discarded.
• Noise and outliers — remove/smooth noisy points and outliers so they do not distort the model.
• Attribute relevance & redundancy — irrelevant (e.g., a student ID) or redundant attributes must be removed to avoid overfitting and slow training.
• Normalisation / scaling — techniques like min-max or z-score normalisation prevent attributes with larger numeric ranges from dominating distance-based approaches.

2. Data availability and system storage:

• With very large datasets, some algorithms cannot hold all data in memory; techniques such as focused sampling and incremental/streaming induction are needed.
• Training and test data must be carefully partitioned (e.g., holdout or $k$-fold cross-validation) so the estimated accuracy is reliable.

3. Model quality — accuracy, speed and interpretability:

• Misclassification costs / false positives vs false negatives — in medicine or fraud detection a wrong prediction has different costs, so accuracy alone may not suffice.
• Speed — time to build the model (training) and to apply it on new objects (testing).
• Linearity vs non-linearity — whether a linear model suffices or a non-linear model is required.
• Robustness to noise and outliers — the model should not change drastically because of noisy data.
• Interpretability — the model should be understandable to the user where necessary (e.g., decision-tree rules vs a black-box neural net).
• Overfitting / generalisation — comparing model performance on unseen test data (holdout) so the learner does not memorise the training data.

In short, the main issues come down to clean, well-scaled data, a sound training/test framework, and a model that balances accuracy, speed, and interpretability.