Handling Missing Values and Noisy Data

Both are core data cleaning operations, because models and rules mined from poor-quality input are unreliable. The two problems are distinct: a missing value is an absent attribute value, while noise is a random error or deviation in a measured variable.

A. Methods for missing values

1. Ignore the tuple — discard the record entirely. Acceptable only when the missing fraction is very small and records are processed one at a time; otherwise it introduces bias.
2. Fill in manually — a domain expert supplies the value. Most accurate, but expensive, slow and subjective.
3. Use a special value — record an explicit marker such as 'unknown' or '?', which is natural for categorical attributes.
4. Use a central tendency value — substitute the attribute mean, median or mode. A refinement is to fill by class mean (e.g. average salary of the employee's own designation rather than the company-wide average).
5. Predict the most probable value — infer the missing value from the remaining attributes using decision trees, linear regression or Bayesian reasoning. This is generally the most reliable automatic method.

B. Methods for noisy data

1. Binning — sort the values, partition them into equal-frequency bins, then smooth each value by its bin mean, its bin median, or by the bin boundaries (minimum or maximum of the bin). This is local, content-independent smoothing.
2. Regression — fit a function (linear or multiple regression) to the data and smooth the values by fitting them to the regression line, which also absorbs some of the random variation.
3. Clustering — detect and remove outliers; points that do not belong to any cluster or lie far from all cluster centroids are treated as noise and either corrected or discarded.
4. Human inspection — plot the data and let an expert identify and correct obviously wrong values.

Choosing a method

Situation	Preferred method
Very few incomplete records	Ignore the tuple
Categorical attribute	Special 'unknown' value or the mode
Numerical attribute, class label known	Class mean
Other attributes informative about the missing one	Predict most probable value (regression / decision tree)
Skewed numerical variable to be smoothed	Binning
Isolate and remove outliers	Clustering
Small dataset, high cost of error	Manual entry

Applying these steps before mining substantially improves the accuracy, validity and credibility of the discovered patterns.