Preprocessing Pipeline for a Dataset with Numerical, Categorical and Missing Attributes

Raw data is usually incomplete, noisy and inconsistent. The steps below are applied in this order: clean, integrate, transform, reduce, discretize.

Pipeline

Step	What to do	Justification
1. Data cleaning	Missing values: numerical attributes get mean/median (median if skewed); categorical attributes get the mode or an "Unknown" label; if over about 50% of a column is missing, drop it. Noise: smooth by binning or regression. Outliers: detect with IQR or z-score; cap or remove if they are errors. Inconsistency: standardize codes and formats (e.g. "M"/"Male")	Algorithms cannot handle missing values and noise lowers accuracy
2. Data integration	Merge multiple sources, resolve schema and entity conflicts (same customer, different IDs), remove duplicates. Check redundant attributes with correlation (numerical) or chi-square (categorical)	Avoids duplicates and conflicting values and keeps one consistent view
3. Data transformation	Normalize numerical attributes (min-max to [0, 1] or z-score); encode categorical attributes (one-hot, label encoding); apply aggregation or log transform for skewed data	Distance-based methods (KNN, K-means) would otherwise be dominated by large-scale attributes; most algorithms need numbers
4. Data reduction	Attribute subset selection (drop irrelevant/redundant), PCA for dimensionality reduction, sampling (numerosity reduction) if the data is huge	Lowers storage and computation time and reduces the curse of dimensionality
5. Discretization	Convert continuous attributes into intervals using equal-width or equal-frequency binning, and build concept hierarchies (age to young / middle / old)	Needed for association rules and decision trees, and gives more understandable patterns

Why this order

Cleaning must come first, because wrong or missing values would distort later steps. Integration comes before transformation so that scaling is done on the final merged data. Reduction and discretization come last, once the data is clean and consistent.

Example

Attribute Income (numerical, 8% missing): fill with median, cap outliers at the 99th percentile, normalize to [0, 1], then bin into Low / Medium / High. Attribute City (categorical): fix spelling variants and one-hot encode.

Conclusion

The pipeline produces complete, consistent, scaled and compact data, which gives more accurate and reliable mining results.