Text Mining & Mining the World Wide Web

Text Mining — Definition and Challenges

Text mining (text data mining / knowledge discovery in text) is the process of discovering interesting, novel, and useful knowledge from large collections of unstructured text documents (e.g., web pages, news articles, e-mails, research papers). It applies techniques from data mining, machine learning, NLP (natural language processing), and information retrieval to go beyond simple keyword searches.

Main challenges of text mining:

1. Unstructured nature — text has no fixed schema; information must first be extracted/structured.
2. High dimensionality — documents contain thousands of distinct words/phrases; representing them as features creates very sparse, huge vectors.
3. Ambiguity and semantics — words often have multiple meanings (polysemy) and synonyms; understanding context is difficult.
4. Noisy content — typographical errors, abbreviations, stop words, and slang degrade quality.
5. Multilingual / varied sources — different languages, formats, and lengths make consistent analysis hard.
6. Evolving language — new words, terms, and jargon appear continuously.

Mining the World Wide Web — Challenges and Techniques

Challenges in mining the WWW (beyond traditional data mining):

• Huge, dynamic, and heterogeneous data — billions of pages, changing constantly.
• Unstructured and semi-structured content — most pages are free text or loosely structured HTML rather than rigid database tables.
• Link/authority biases — some pages attract more links regardless of true worth; link spam and hidden text mislead naive metrics.
• Redundancy & duplication — the same information on many mirrors and machine-generated pages.
• Heterogeneous formats — HTML, PDF, images, video, XML, etc.

Techniques.

• Information retrieval / indexing — term weighting (TF-IDF), Boolean/phrase queries, inverted indexes and search engines.
• Web structure mining — using the hyperlink graph (PageRank, HITS) to rank and find authoritative sources.
• Web usage mining — analysing server logs, sessions, and cookies to understand user access patterns.
• Web content mining — page-based (text/HTML tags, wrapper induction) and search-based (meta-search / aggregation of multiple engines) approaches.

Important: because the web is so noisy, dynamic, and huge, knowledge discovery there relies on a combination of IR (to narrow candidates), NLP (to interpret text), and graph analysis (to respect the link structure of pages). It is harder than mining a well-structured relational database because the source data are largely unstructured and continuously changing.