Data Cleaning & Approaches for Missing Values

What is Data Cleaning?

Data cleaning (data cleansing/scrubbing) is the preprocessing step that detects and corrects (or removes) inaccurate, incomplete, inconsistent, or noisy data from a dataset — filling missing values, smoothing noise, identifying/handling outliers, and resolving inconsistencies — so that mining algorithms operate on trustworthy, high-quality data.

Approaches for Handling Missing Values

1. Ignore the tuple
   Discard the entire record. Only reasonable when the missing attribute is essential (e.g., the class label) and the dataset is large enough to absorb the loss.

2. Fill in manually
   A domain expert inspects and enters the correct value. Accurate but time-consuming and infeasible for large datasets.

3. Use a global constant
   Replace all missing entries with a fixed placeholder (e.g., "Unknown", -999). Simple but risky — the mining algorithm may misinterpret the placeholder as a meaningful pattern.

4. Use the attribute mean/median
   Fill numeric gaps with the mean (or median, for skewed distributions) of that attribute computed over all available tuples.

5. Use the mean/median for the same class
   More accurate than a global mean — uses the mean of tuples belonging to the same class as the one with the missing value.

6. Use the most probable value
   Predict the missing value using inference-based methods: regression, Bayesian formalism, or decision-tree induction based on other attributes. This is the most sophisticated and generally most accurate method, since it uses the relationships already present in the data.

                Missing Value Handling
                        │
   ┌─────────┬──────────┼──────────┬────────────┬───────────┐
Ignore    Manual     Global     Attribute    Class-wise   Predicted
tuple      fill      constant   mean/median  mean/median  (regression/
                                                            Bayes/tree)

In practice, methods 4–6 (statistical/model-based imputation) are most commonly used in real mining pipelines because they preserve dataset size while introducing minimal bias, compared to simply dropping records.