Handling Duplicate and Redundant Data

Duplicate tuples are exact (or near-exact) repeated records; redundant attributes are columns that can be derived from other columns (adding no new information). Both inflate storage and can bias mining results (e.g., an over-represented duplicate customer skews clustering/statistics).

Methods

1. Duplicate detection & removal

   • Exact match: simple SELECT DISTINCT / sort-and-compare to find and drop identical rows.
   • Near-duplicate/fuzzy match: use similarity measures (edit distance, Jaccard) for records that are "almost" the same due to typos/format differences (e.g., "Jon Smith" vs "John Smith"), then merge or drop.

2. Correlation analysis for redundant attributes

   • Numeric attributes — compute the Pearson correlation coefficient; a high $|r|$ (close to ±1) suggests redundancy, and one of the two attributes can be dropped.
   • Categorical attributes — use the chi-square (χ²) test of independence; a high χ² value with low p-value indicates the attributes are strongly related and possibly redundant.

3. Covariance analysis — for numeric attributes, examine how they vary together; strong covariance patterns can also flag redundancy.

4. Careful integration design — during data integration, applying entity identification (recognizing the same real-world entity across different sources with different keys/names) prevents introducing duplicates in the first place.

Duplicate Tuples          Redundant Attributes
      │                          │
  Exact match  Fuzzy match   Correlation (numeric)  χ² test (categorical)
      │             │             │                       │
   Remove       Merge/Remove   Drop one if |r| high   Drop one if dependent

In short: duplicates are handled by detection (exact or fuzzy) and removal/merging, while redundant attributes are identified via statistical correlation/independence tests and pruned before mining.