PyQdeck LogoPyQdeckSubjects
Back to the 2022 paper

Module 2: Data Mining and Association Rule Mining

20227m

In real-world data, tuples with duplicate and redundant values for some attributes are a common occurrence. Describe various methods for handling this problem.

Search GoogleAsk AI
Worked SolutionAI Assisted

Handling Duplicate and Redundant Data

Duplicate tuples are exact (or near-exact) repeated records; redundant attributes are columns that can be derived from other columns (adding no new information). Both inflate storage and can bias mining results (e.g., an over-represented duplicate customer skews clustering/statistics).

Methods

  1. Duplicate detection & removal

    • Exact match: simple SELECT DISTINCT / sort-and-compare to find and drop identical rows.
    • Near-duplicate/fuzzy match: use similarity measures (edit distance, Jaccard) for records that are "almost" the same due to typos/format differences (e.g., "Jon Smith" vs "John Smith"), then merge or drop.

  2. Correlation analysis for redundant attributes

    • Numeric attributes — compute the Pearson correlation coefficient; a high ∣r∣|r| (close to ±1) suggests redundancy, and one of the two attributes can be dropped.
    • Categorical attributes — use the chi-square (χ²) test of independence; a high χ² value with low p-value indicates the attributes are strongly related and possibly redundant.

  3. Covariance analysis — for numeric attributes, examine how they vary together; strong covariance patterns can also flag redundancy.

  4. Careful integration design — during data integration, applying entity identification (recognizing the same real-world entity across different sources with different keys/names) prevents introducing duplicates in the first place.

Duplicate Tuples          Redundant Attributes
      │                          │
  Exact match  Fuzzy match   Correlation (numeric)  χ² test (categorical)
      │             │             │                       │
   Remove       Merge/Remove   Drop one if |r| high   Drop one if dependent

In short: duplicates are handled by detection (exact or fuzzy) and removal/merging, while redundant attributes are identified via statistical correlation/independence tests and pruned before mining.

Similar questions

Data MiningIn real-world data, tuples with missing values for some attributes are a common occurrence. List at least two methods for handling this problem.20212mData MiningWhat is data cleaning? Describe various approaches for cleaning data having missing values.20205mDATABASE MANAGEMENT SYSTEMWhich of the following is known as a set of entities of the same type that share same properties or attributes? - (i) Relation set - (ii) Tuples - (iii) Entity set - (iv) Entity relation model20242mDatabase Management SystemThe term _____ is used to refer to a row. (i) attribute (ii) tuple (iii) field (iv) instance20182m
PreviousWhat is data mining? Differentiate data mining with traditional database system.NextA database has five transactions. Let min sup = 60% and min con f = 80% : | TID | items_bought | |---|---| | T100 | {M, O, N, K, E, Y} | | T200 | {D, O, N, K, E, Y} | | T300 | {M, A, K, E} | | T400 | {M, U, C, K, Y} | | T500 | {C, O, O, K, I, E} | Find all frequent item sets using Apriori and FP-growth, respectively. Compare the efficiency of the two mining processes.