Data Preprocessing

1. Why Preprocess?

Real-world data is dirty: it is incomplete, noisy, inconsistent and huge. Mining raw data gives wrong models — garbage in, garbage out. Preprocessing typically consumes 60-70% of mining effort and improves accuracy, speed and interpretability.

 RAW DATA
    |
    v
 [ DATA CLEANING ] -- fill missing, smooth noise, remove outliers, fix inconsistency
    |
    v
 [ DATA INTEGRATION ] -- combine multiple sources, resolve schema/entity conflicts
    |
    v
 [ DATA TRANSFORMATION ] -- normalise, aggregate, derive attributes
    |
    v
 [ DATA REDUCTION ] -- dimensionality / numerosity reduction, compression
    |
    v
 [ DISCRETIZATION & CONCEPT HIERARCHY ] -- continuous -> intervals, generalise values
    |
    v
 PREPROCESSED DATA  -->  DATA MINING

2. Data Quality Dimensions

Dimension	Question	If violated
Accuracy	Correct values?	Wrong patterns
Completeness	All required values present?	Biased models
Consistency	Same value everywhere?	Contradictions
Timeliness	Up to date?	Stale decisions
Believability / Interpretability	Trustworthy and understandable?	Low user acceptance

3. Task 1 — Data Cleaning

Fill missing values, smooth noisy data, identify/remove outliers, resolve inconsistencies. (Discussed in detail under Data Cleaning.)

4. Task 2 — Data Integration

Combine data from multiple databases, files or services into a coherent store:

Schema integration — entity identification (e.g. cust_id in CRM = customer_no in ERP).
Redundancy handling — correlation analysis; derive redundant attributes from others.
Tuple duplication — detect and merge duplicates.
Data value conflicts — different units/scales (kg vs lb, INR vs USD).

Correlation between attributes $A$ and $B$ :

r_{A,B}=\frac{\sum_{i=1}^{n}(a_i-\bar A)(b_i-\bar B)}{(n-1)\,\sigma_A \sigma_B},\qquad -1 \le r \le +1

Redundant if $r_{A,B}$ is significantly large (positive or negative); chi-square test used for nominal attributes.

5. Task 3 — Data Transformation

Smoothing — remove noise (binning, regression, clustering).
Attribute construction — new derived attributes (area = length × width).
Aggregation — summarise (daily → monthly sales).
Normalisation — scale to a common range:

Min–max: $v' = \dfrac{v - \min_A}{\max_A - \min_A}\,(new\_max_A - new\_min_A) + new\_min_A$

Z-score: $v' = \dfrac{v - \bar A}{\sigma_A}$

Decimal scaling: $v' = \dfrac{v}{10^{j}}$ where $j$ = smallest integer with $\max|v'| < 1$

Generalisation / discretisation — replace raw values with higher-level concepts.
Encoding — one-hot, label encoding for categorical attributes.

6. Task 4 — Data Reduction

Obtain a reduced representation that is much smaller but produces the same/similar mining result:

Dimensionality reduction — attribute subset selection, principal component analysis (PCA), wavelet transform, forward/backward/stepwise selection, decision-tree induction for attribute ranking.
Numerosity reduction — regression/log-linear models, histograms, clustering, sampling (SRSWOR, SRSWR, stratified, cluster), data cube aggregation.
Data compression — lossless (Huffman, LZW) or lossy (wavelet, PCA, JPEG).

PCA:\ X_{n\times m}\ \to\ \text{top-}k\ \text{principal components},\qquad \text{variance retained} = \frac{\sum_{i=1}^{k}\lambda_i}{\sum_{i=1}^{m}\lambda_i}

7. Descriptive Data Summarisation (Basic Statistics)

Central tendency for $x_1,\dots,x_n$ :

\bar x = \frac{1}{n}\sum_{i=1}^{n} x_i,\qquad \text{midrange}=\frac{\min+\max}{2},\qquad \text{mode} = \text{most frequent value}

Median = middle value after sorting (average of two middle values for even $n$ ).

Dispersion:

\sigma^2=\frac{1}{n}\sum_{i=1}^{n}(x_i-\bar x)^2\ \ \text{(population)},\qquad s^2=\frac{1}{n-1}\sum_{i=1}^{n}(x_i-\bar x)^2\ \ \text{(sample)}

Quartiles, IQR $= Q_3 - Q_1$ , five-number summary = $(\min, Q_1, \text{median}, Q_3, \max)$ , and outlier fences

\text{outlier if } x < Q_1 - 1.5\,IQR \ \ \text{or}\ \ x > Q_3 + 1.5\,IQR

Graphical tools: histogram, boxplot, quantile plot, Q-Q plot, scatter plot, loess (local regression) curve, pie/bar charts.

Numerical example — data: 4, 8, 15, 21, 24, 25, 28, 34 (n = 8)

mean = 159 / 8 = 19.875, midrange = (4 + 34)/2 = 19
median = (21 + 24)/2 = 22.5, mode = none (all single)
$s^2 \approx 121.27$ , $s \approx 11.01$
$Q_1 = 15$ (median of 4, 8, 15, 21 → 11.5 if averaged), $Q_3 = 25$ , IQR $\approx 25 - 11.5 = 13.5$

8. Why Preprocessing Matters — Summary Table

Problem	Symptom in mining	Preprocessing fix
Missing values	Algorithm fails/biased	Imputation or tuple removal
Noise & outliers	Overfitting, wrong centroids	Binning, clustering, regression
Redundant attributes	Slower, correlated	Correlation/PCA, attribute selection
Different scales	Distance-based methods dominated by large-range attribute	Normalisation, z-score
Continuous attributes	Some algorithms need categorical	Discretisation, concept hierarchy

9. Exam Points

Explain preprocessing tasks (cleaning, integration, reduction, transformation) with diagram — 10 marks (very frequent).
Normalisation numerical: min-max, z-score, decimal scaling — 5 marks.
Mean, median, mode, variance, quartiles, IQR numerical — 5 marks.
Correlation coefficient interpretation — 3-5 marks.

Data Preprocessing