Data Cleaning & Data Integration

What is Data Cleaning? Why is it needed?

Data Cleaning (data scrubbing) is the preprocessing step that detects and fixes (or removes) inaccurate, incomplete, noisy, or inconsistent data before mining.

Typical cleaning tasks:

• Missing values — filled via ignoring the tuple, manual entry, global constant, attribute/class mean, or predicted values (regression/Bayesian/decision-tree inference).
• Noisy data — smoothed using binning, regression, or clustering (to detect and remove outliers).
• Inconsistent data — resolved using domain knowledge, functional dependencies, or manual correction (e.g., fixing "NY" vs "New York" mismatches).

Why it's needed: real-world data is "dirty" — incomplete (missing attribute values), noisy (containing errors/outliers), and inconsistent (discrepancies across records). Mining on dirty data produces unreliable, misleading, or outright wrong patterns — "garbage in, garbage out." Clean data is a prerequisite for trustworthy analysis.

Dirty Data ──[Data Cleaning]──▶ Clean Data ──▶ reliable mining results
   │
   ├─ missing values
   ├─ noise/outliers
   └─ inconsistencies

What is Data Integration? Its Role in Data Mining

Data Integration merges data from multiple heterogeneous sources (different databases, flat files, ERP systems, external feeds) into a single, coherent data store for mining.

Key challenges handled during integration:

1. Entity identification problem — recognizing that the same real-world entity may be represented differently across sources (e.g., "customer_id" in one DB vs "cust_no" in another) — requires matching schema and metadata.
2. Redundancy and correlation analysis — an attribute may be redundant if it can be derived from another attribute (present in a different source under a different name); correlation tests (Pearson r for numeric, chi-square for categorical) help detect and eliminate such redundancy.
3. Detection and resolution of data value conflicts — the same real-world entity may have different attribute values in different sources due to different scales, encodings, or units (e.g., weight in kg vs lbs).

Source A (RDBMS) ──┐
Source B (Files)  ──┼──▶ [Data Integration: entity matching, redundancy
Source C (ERP)    ──┘     removal, conflict resolution] ──▶ Unified Dataset

Role in Data Mining

Data integration ensures that the mining algorithms operate on a single, consistent view of the data rather than fragmented, conflicting sources. Without proper integration:

• Duplicate records (from unresolved entity matches) can bias frequency-based mining (e.g., inflating support counts in association rule mining).
• Conflicting values for the same entity can corrupt classification/clustering models.
• Redundant attributes can slow down mining and distort distance/similarity-based algorithms.

In short: data cleaning ensures the data is accurate, while data integration ensures the data is complete and consistent across sources — together they form the foundation that makes all downstream mining results trustworthy.