Data Preprocessing Techniques in Data Mining

Definition. Data preprocessing is the set of operations applied to raw data before mining, in order to improve data quality. Real-world data is typically incomplete, noisy, inconsistent and heterogeneous; mining on dirty data produces misleading results.

1. Data cleaning

1. Missing values — ignore the record, fill in manually, or infer automatically: a global constant, the attribute mean/mode, the most probable value (estimated via regression or a decision tree), or KNN imputation.
2. Noise reduction — binning (equal-width / equal-depth), regression (fit a smooth curve to the data), clustering (points far from every cluster are treated as outliers and removed), or sorting followed by smoothing.
3. Inconsistency and duplicates — resolve naming and code conflicts (CS vs Computer Sci), enforce domain and integrity checks, and remove duplicate records.

2. Data integration

• Combines data from several databases, flat files or cubes into a single store.
• Schema integration — match entity and attribute names (cust-id with customer-id) and resolve structural conflicts.
• Redundancy detection — correlation analysis, $\chi^2$ tests and covariance identify duplicate or closely related attributes; highly correlated features can otherwise create spurious patterns.
• Conflict resolution — reconcile different units, scales and encoding conventions (e.g. kilograms vs pounds) using metadata.

3. Data reduction

Obtain a smaller representation of the data while keeping the mined results essentially the same.

• Dimensionality reduction — feature selection (relief, $\chi^2$) and feature extraction (PCA, SVD, wavelets) to combat the curse of dimensionality.
• Numerosity reduction — parametric methods (regression, log-linear models) or non-parametric methods (histograms, sampling, clustering).
• Compression — lossless vs lossy compression (wavelets, principal components).
• Generalisation — replace low-level values by higher-level concepts through a concept hierarchy (street → city → country).

4. Data transformation

• Normalisation — scale attributes to a common range, essential for distance-based methods such as KNN, K-means and SVM:
  • min–max: $v' = \dfrac{v - \min}{\max - \min}$
  • z-score: $v' = \dfrac{v - \bar{v}}{\sigma}$
  • decimal scaling by a power of 10.
• Encoding — binarisation / one-hot encoding of nominal attributes.
• Aggregation — summarise daily sales into monthly or yearly totals.
• Discretisation — equal-width or equal-depth (quantile) binning, or generalisation into intervals via concept hierarchies.
• Feature construction — derive new meaningful attributes (e.g. $\text{BMI} = \text{weight} / \text{height}^2$) to expose patterns that are hidden in the raw fields.

Conclusion

Preprocessing is iterative and typically occupies the majority of a project's effort (commonly quoted as 60–80%), but it is decisive: no classifier can rescue poorly prepared data, so cleaning, integration, reduction and transformation must be completed before any mining algorithm is run.