Data Preprocessing Techniques

Data preprocessing prepares raw data for mining by making it complete, consistent, accurate and compact. Real data is usually dirty (missing, noisy, inconsistent), so preprocessing is needed to get reliable results.

Main techniques

Technique	Purpose	Methods
Data cleaning	Fix errors in the data	Fill missing values (ignore tuple, mean/median, most probable value); smooth noise (binning, regression, clustering); detect outliers; resolve inconsistencies
Data integration	Combine data from several sources	Schema integration, entity resolution, remove duplicates, detect redundancy with correlation and chi-square
Data transformation	Convert data into suitable forms	Normalization, aggregation, generalization, attribute construction, encoding
Data reduction	Get a smaller representation with the same analytical results	Data cube aggregation, attribute subset selection, dimensionality reduction (PCA), numerosity reduction (sampling, histograms), compression
Data discretization	Replace numeric values with intervals or labels	Equal-width and equal-frequency binning, histogram and clustering analysis, concept hierarchies

Normalization formulas

• Min-max: v' = (v - min) / (max - min) x (new_max - new_min) + new_min
• Z-score: v' = (v - mean) / standard deviation
• Decimal scaling: v' = v / 10^j, where j is the smallest integer such that max(|v'|) < 1

Example (min-max to [0, 1]): values 200, 300, 400, 600, 1000: min = 200, max = 1000. The value 400 becomes (400 - 200)/(1000 - 200) = 0.25.

Binning example

Sorted data: 4, 8, 15, 21, 21, 24, 25, 28, 34. Equal-depth bins of size 3: {4, 8, 15}, {21, 21, 24}, {25, 28, 34}. Smoothing by bin means: {9, 9, 9}, {22, 22, 22}, {29, 29, 29}.

Why it matters

Preprocessing improves accuracy, efficiency and reliability of the mining results (garbage in, garbage out) and often takes most of the project effort.