Data Quality Problems and Data-Cleaning Strategies

Real data is often incomplete, noisy and inconsistent. Cleaning fixes these problems before mining.

Problems and strategies

Problem	Cause	Cleaning strategies
Missing values	Faulty entry, equipment failure, non-response	(1) Ignore the tuple (when many values are missing); (2) fill manually; (3) use a global constant ("Unknown"); (4) fill with the mean/median of the attribute; (5) use the class-wise mean; (6) use the most probable value by regression, decision tree or Bayes
Noisy data (random error or variance)	Faulty instruments, entry errors	Binning (smooth by bin means, medians or boundaries); regression (fit a function); clustering (values outside clusters are noise); human inspection
Inconsistent data	Different formats or codes across sources, violated rules	Apply integrity constraints and rules, standardize formats and codes, use reference data, correct manually or with tools, and track lineage
Redundant attributes	Same information in several attributes (derived or duplicated)	Correlation analysis (Pearson r for numeric, chi-square for categorical); drop one of the pair; attribute selection
Outliers	Errors or genuine rare events	Detect with IQR, z-score, clustering, DBSCAN, LOF; remove, cap (winsorize), transform, or keep if they are genuine and important

Comparison of the strategies

Strategy	Advantage	Disadvantage
Ignore tuple	Simple	Loses data; biased if missing is not random
Mean / median fill	Easy, keeps all records	Reduces variance; may distort relations
Most probable value (model-based)	Most accurate, uses other attributes	More complex and costly
Binning	Simple smoothing	Loses detail; choice of bins matters
Regression smoothing	Preserves the trend	Needs a suitable model
Clustering for noise/outliers	Detects groups and unusual points	Depends on the algorithm and its parameters
Correlation-based removal	Reduces redundancy and speeds up mining	Correlation does not always mean redundancy
Removing outliers	Cleaner model	May discard valuable rare events (e.g. fraud)

Choosing a strategy

• Use the median for skewed data and model-based filling when accuracy matters.
• Remove outliers only if they are errors; keep genuine ones, because they may be the target of the analysis (fraud detection).
• Always apply cleaning rules consistently and document them.

Conclusion

Each data problem needs a suitable strategy; simple methods are fast but approximate, while model-based methods are more accurate but more expensive. Good cleaning improves the reliability of the mined results.