Short Note on Issues and Challenges in Data Mining

Data mining offers great benefits but raises a number of technical, methodological and social issues. They are commonly grouped as follows:

1. Mining methodology issues

1. Handling noisy, incomplete and inconsistent data - real data contain missing values, errors and outliers; mining a dirty database produces wrong patterns, so data cleaning, integration and preprocessing are mandatory.
2. Scalability and efficiency - algorithms must handle terabytes of data, high-dimensional data and evolving data streams; frequent pattern mining on dense data can be combinatorially explosive.
3. Mining diverse data types - relational data, transactions, graphs, spatial, temporal, multimedia, text, web and stream data each need different models, so one technique does not fit all.
4. Pattern evaluation and interestingness - a huge number of patterns can be produced, most of which are redundant or trivial; the challenge is to measure interestingness (objective measures such as support, confidence, lift; subjective measures such as novelty, usefulness) and to show only significant patterns.
5. Imbalance and cost-sensitive learning - in domains like fraud or disease detection the positive class is rare and a wrong prediction has a very different cost from another.

2. Performance-related issues

• Interactive mining: the user needs multiple rounds with varying minimum-support thresholds, constraints and ad-hoc queries, so results should be produced quickly rather than after long batches.
• Parallel, distributed and incremental mining to reduce response time on large or streaming data.
• Incorporation of background knowledge - user beliefs, taxonomy and constraints should guide the search without over-constraining it.

3. Issues relating to data quality and diversity

• Data may be heterogeneous, distributed, legacy or schema-less; integration creates redundancy and conflicts.
• High dimensionality, sparse feature spaces and irrelevant attributes degrade accuracy (the curse of dimensionality).
• Evolving concepts: models and patterns can become obsolete as data distributions change over time (concept drift).

4. Social and ethical issues

1. Data privacy and security - mining can reveal sensitive personal information (e.g. health or financial status) even from seemingly anonymous data; techniques such as k-anonymity, differential privacy and access control are needed.
2. Data ownership, copyright and access - who owns mined results? Is the use of web content legitimate?
3. Misuse of results - profiling, discrimination, spam and manipulation; results may be over-trusted because they look precise.
4. Interpretability - results must be understandable to non-expert decision makers, otherwise they are not used.
5. Legal and professional responsibility - who is answerable when an automatic decision is wrong (loan rejected, patient misdiagnosed)?

In short, the main challenges are the quality and scale of data, the correct evaluation of discovered patterns, and the privacy, security and ethical use of the knowledge obtained.