Short Note on Text Mining

1. Definition

Text mining (text data mining) is the process of discovering previously unknown, interesting and useful patterns, associations, concepts and knowledge from large amounts of unstructured or semi-structured text - documents, web pages, e-mails, reports, social-media posts, patents, medical records. It sits at the intersection of information retrieval, natural language processing (NLP), machine learning and statistics. Unlike a keyword search, which only retrieves documents, text mining analyses them to extract knowledge; unlike classical NLP, whose goal is linguistic understanding, its goal is decision-relevant patterns.

2. General process

1. Collect the document corpus from the source (database, web, intranet).
2. Pre-processing / text preparation: tokenisation, sentence segmentation, removal of stop words (the, is, and), noise removal (punctuation, HTML tags), stemming / lemmatisation (running -> run), and sometimes part-of-speech tagging.
3. Text representation: convert text into a machine-usable form - bag-of-words, term frequency - inverse document frequency (TF-IDF) weighting, n-grams, and the vector space model (each document = a numeric vector).
4. Analysis / mining:
   • text categorisation and classification (e-mail spam vs. not spam, news by topic),
   • text clustering (grouping similar documents with no labels),
   • information extraction (entities, relations, events from text),
   • association analysis between terms,
   • sentiment / opinion analysis (positive vs. negative reviews),
   • topic modelling (e.g. Latent Dirichlet Allocation),
   • text summarisation and relationship mining.
5. Evaluation and visualisation of the discovered patterns.

3. Applications

Spam filtering, automatic news categorisation, search-engine ranking, sentiment analysis of product reviews and movie ratings, resume screening, legal/medical document discovery (e-discovery), plagiarism detection, chatbots and question-answering systems, patent and scientific literature analysis, and detection of fraudulent insurance claims.

4. Challenges

• Data is unstructured, heterogeneous and noisy, so meaning must be reconstructed.
• Very high dimensionality and sparsity of the term vector (tens of thousands of terms, most zero).
• Ambiguity: polysemy (one word, many meanings - e.g. bank), synonyms (car = automobile), idioms, sarcasm and domain-dependent language.
• Mixed languages, abbreviations, spelling variants and evolving slang.
• Large volume requires scalable algorithms; evaluation is difficult because there is no single correct answer and labelled training corpora are expensive to build.
• Privacy of the texts being mined.