Data Mining and the KDD Process

Definition. Data mining is the task of discovering valid, novel, potentially useful and ultimately understandable patterns — models, rules, clusters, associations — from large volumes of data. It sits at the intersection of database systems, statistics and machine learning, turning stored data into knowledge that supports decision making. A pattern counts as knowledge only if it is previously unknown, holds on new data, and is useful to the user; mining is not a random hunt for anything unusual.

Examples

• A supermarket mines point-of-sale baskets and finds the rule {bread, butter} -> milk with, say, support 8% and confidence 70%, which guides shelf placement and bundle offers.
• A bank mines card transaction histories to flag fraud by matching each transaction against the customer's normal spending profile.
• A telecom operator clusters customers by usage patterns to design targeted tariff plans.

Steps in the KDD Process

Knowledge Discovery in Databases (KDD) is the overall, iterative procedure for converting raw data into usable knowledge; data mining is the single step inside it that extracts the patterns. The standard steps are:

1. Data cleaning — remove noise and inconsistent records (missing values, duplicates, wrong entries).
2. Data integration — combine data from multiple databases, files or schemas into a coherent store.
3. Data selection — retrieve only the data relevant to the analysis task.
4. Data transformation — normalise, aggregate, summarise and discretise into forms suitable for mining.
5. Data mining — apply intelligent methods (classification, clustering, association rule mining, regression, etc.) to extract patterns.
6. Pattern evaluation — identify genuinely interesting, valid and useful patterns using objective measures (support, confidence, accuracy) together with domain knowledge.
7. Knowledge presentation — present the mined knowledge through visualisation, graphs, reports, queries or rules.

Raw data -> Cleaning -> Integration -> Selection -> Transformation
                                                        |
Knowledge <- Evaluation <- Data mining <----------------+

Because any step may expose problems that send the process back to an earlier step, KDD is a loop rather than a one-pass pipeline.