Data Mining

Solution

Correct Answer: (iii) 0.04

Explanation

Term frequency (TF) = (number of times the term occurs in the document) / (total number of words in the document)

TF = 4 / 100 = 0.04

TF is the first part of the TF-IDF weight used in text mining: TF-IDF = TF x log(N / df).

Why other options are incorrect:

• (i) 0.004 = 4/1000, (ii) 0.4 = 4/10, (iv) 4 is the raw count, not the frequency ratio.

Answer: (iii) 0.04

Answer: (i) Geographic-based data

Spatial data mining is the process of discovering interesting, useful patterns from large spatial (geographic) databases — i.e., data that references objects by their location in space, such as maps, satellite imagery, GPS traces, and location-tagged records.

Examples of spatial data:

• Satellite remote-sensing images (land cover, weather patterns),
• Mapping data from Geographic Information Systems (GIS),
• Location data from mobile phones/sensors,
• Earthquake or disease-spread records with co-ordinates.

Techniques include spatial clustering (e.g., finding densely populated regions), spatial outlier detection (e.g., detecting anomalies in pollution monitors), and spatial association rules (e.g., "regions near a river usually have fertile soil").

Why the others are wrong:

• (ii) Text documents — that is text mining.
• (iii) Financial data only — that is financial data mining; spatial mining is not limited to financial data.
• (iv) Audio signals — that is multimedia mining.

Hence spatial data mining deals with geographic-based data.

Solution

Correct Answer: (i) Web structure mining

Explanation

Web structure mining analyses the link structure of the Web: hyperlinks between pages and the relationships among them. It is used to find authoritative pages, hubs and communities. Well-known algorithms: PageRank and HITS.

The three types of web mining:

• Web content mining - the content of pages (text, images)
• Web structure mining - hyperlinks between pages
• Web usage mining - user access logs and click patterns

Why other options are incorrect:

• (ii) Numerical integration - a mathematical technique, unrelated.
• (iii) Web usage mining - studies user behavior from logs, not links.
• (iv) Data normalization - a preprocessing step.

Answer: (i) Web structure mining

Mining the World Wide Web

Definition. Web mining is the automatic discovery of useful patterns and knowledge from web documents, from the link structure of the web, and from the records of how users interact with web sites. Because the web is a huge, distributed, semi-structured and constantly changing hypertext repository, ordinary data mining techniques are adapted to work on HTML/XML text, hyperlink graphs and server or clickstream data.

Types / Process of Web Mining

1. Web content mining — mines the actual content of pages: text, images, audio, metadata and embedded tables. It uses text categorisation, information extraction, wrapper induction and document clustering, and can work either on the raw pages or on the result sets returned by search engines.
2. Web structure mining — mines the hyperlink structure to evaluate importance and relatedness of pages. PageRank, the hub-and-authority (HITS) model and discovery of web communities are the standard examples.
3. Web usage mining — mines user-interaction data such as server/access logs, clickstreams, cookies and registration data. The process is:
   • Data collection — gather server logs, proxy logs, client-side records.
   • Data preprocessing — clean the logs, identify users and sessions, reconstruct visit sequences.
   • Pattern discovery — apply clustering, association rules and sequential pattern mining.
   • Pattern analysis — extract actionable knowledge for personalisation, recommendation and site redesign.

Web Mining vs Data Mining

Basis	Data mining	Web mining
Source	Centralised databases / data warehouses	Distributed web pages, links and logs
Data type	Mostly structured tuples and attributes	Semi-structured / unstructured HTML, XML, text, images, clickstreams
Data quality	Curated, consistent, pre-processed	Noisy, redundant, inconsistent formats, missing values
Schema	Fixed and known in advance	No global schema; structure must be extracted
Scale and change	Relatively static	Extremely large and continuously updated
Relationships	Foreign keys and explicit relations	Hyperlinks, page hierarchy, user sessions
Goal	Hidden patterns, rules, clusters, models	Page relevance, site structure, user behaviour, personalisation

In short, web mining is data mining applied to web-derived data, but it must additionally cope with hyperlink structure, unstructured text and highly dynamic, noisy content.

Challenges and Techniques of Multimedia Data Mining

Definition. Multimedia data mining is the process of discovering interesting patterns, structures and knowledge from multimodal, largely unstructured data such as images, audio, video, graphics and text.

1. Challenges

1. Heterogeneity. Different media have completely different formats, structures and feature spaces, so neither a single schema nor a single distance measure fits all of them.
2. Unstructured nature. Multimedia objects carry no predefined schema or attributes; standard relational mining techniques do not apply directly.
3. The semantic gap. Low-level feature values (pixel intensities, colour histograms, MFCC coefficients) do not map directly onto the high-level meaning a user has in mind, such as a dog, a goal or a mood.
4. High dimensionality. Descriptors such as SIFT, HOG or deep CNN embeddings produce hundreds or thousands of dimensions per object, triggering the curse of dimensionality.
5. Volume. Video and audio streams generate data at very high rates, so storage, indexing and bandwidth become bottlenecks.
6. Temporal and spatial correlation. Successive video frames are highly redundant and audio samples are correlated, so the effective sample size is far smaller than the raw record count suggests.
7. Variability and noise. Changes in viewpoint, lighting, scale and background, plus compression artefacts, degrade feature stability.
8. Scarce labelled data. Manual annotation is expensive, so supervised learners often lack enough ground truth.

2. Techniques used

Technique	Idea / representative methods
Content-based image retrieval (CBIR)	Match images by colour histograms, texture (Gabor, GLCM), shape descriptors and keypoints such as SIFT/SURF/ORB
Feature extraction and descriptors	Standardised descriptors such as MPEG-7 visual and aural features
Audio mining	MFCCs, spectrograms, pitch and energy features; speech-to-text followed by text mining
Video mining	Shot and scene detection, keyframe extraction, temporal segmentation, activity recognition
Text and metadata mining	Mine captions, tags and OCR-extracted text; NLP for semantic annotation
Deep learning	CNNs for images, RNNs/transformers for sequences, autoencoders and transfer learning to obtain embeddings
Similarity search and indexing	KD-trees, locality-sensitive hashing for fast approximate nearest-neighbour retrieval
Multimodal fusion	Combine visual, audio and textual features by early or late fusion for joint analysis
Clustering and classification	Group visually or semantically similar objects; supervised models on annotated examples

3. Typical workflow

1. Ingest the media and extract features or learned embeddings.
2. Reduce dimensionality (e.g. PCA) and index the feature vectors.
3. Run the mining tasks: clustering, classification, similarity retrieval, summarisation.
4. Narrow the semantic gap using user feedback and domain knowledge.

Conclusion: the central difficulty is the semantic gap compounded by scale and dimensionality; modern practice answers it with robust descriptors, deep embeddings and multimodal fusion.

Challenges in Mining Web Data and the Techniques that Address Them

Definition. Web data mining is the discovery of useful patterns and knowledge from web content, web structure (hyperlinks) and web usage (clickstreams), covering the three sub-areas content mining, structure mining and usage mining.

1. Challenges

1. Heterogeneity. Web sources differ in format, language, schema and encoding, so data arriving from many sites has to be reconciled before it can be mined.
2. Unstructured and semi-structured content. HTML pages have no fixed schema; useful facts are mixed with navigation, ads and layout markup.
3. Sheer volume and rapid growth. The Web is huge and its content changes constantly, so crawlers and indexes must scale and stay fresh.
4. Noise, irrelevance and spam. Keyword stuffing, link farms and irrelevant blocks pollute both the text and the link graph.
5. Redundancy. The same fact or near-identical documents appear on many sites, inflating counts and distorting associations.
6. Graph nature of data. Pages are connected by hyperlinks, so the independence assumptions of classical mining do not hold.
7. Diversity of users and context. The same page serves many audiences, so results must be personalised and context aware.
8. Privacy, security and access limits. Login walls, robots rules and sensitive personal data restrict what may be collected and stored.

2. Techniques used to address them

Challenge	Technique
Scale and growth	Focused/declarative crawling, distributed crawlers, incremental and update-aware indexing
Unstructured text	Information extraction, NLP, DOM-aware parsing, wrappers that lift structured records out of HTML
Redundancy and quality	Near-duplicate detection (shingling, MinHash), deduplication, spam and boilerplate filtering
Finding important pages	Link analysis: PageRank and HITS, using hyperlink structure as a popularity vote
Organisation of pages	Web page classification and clustering with TF-IDF features plus link-based refinements
User behaviour	Web usage mining: sessionisation of clickstreams, Markov models, sequential pattern mining
Integration of sources	Record linkage, schema matching and data fusion across heterogeneous sites
Retrieval at scale	Inverted indexes, TF-IDF/BM25 ranking and search-engine query processing

3. Example workflow

1. Crawl a focused set of pages under the robots rules.
2. Clean each page: strip boilerplate, extract the main text and any structured records.
3. Filter duplicates and spam; build a link graph and compute PageRank.
4. Classify pages by topic, cluster similar ones, then mine usage sessions for behaviour patterns.

Conclusion: web mining is hard mainly because the data is heterogeneous, huge, noisy and interconnected; crawling control, text and link analysis, deduplication and usage mining are the standard answers to those four difficulties.

Spatial Data Mining & Types of Web Mining

What is Spatial Data Mining?

Spatial data mining is the process of discovering interesting, useful, and previously unknown patterns from large spatial (geographic) databases — data in which objects are described by their location in space (e.g., maps, satellite imagery, GPS traces).

The distinguishing feature is that ordinary mining techniques must be extended to account for spatial relationships such as distance, adjacency, overlap, and containment between neighbouring objects. Common tasks include:

• Spatial clustering — grouping geographically close objects (e.g., identifying high-crime zones).
• Spatial outlier detection — finding locations whose attribute values differ sharply from their neighbourhood (e.g., an unusually high-pollution monitor).
• Spatial association rules — e.g., "regions that are close to a city centre and near a transport hub tend to have high land value".
• Spatial trend / classification — predicting categorical values from spatial features.

Examples of spatial applications:

• Satellite remote-sensing (land-use/land-cover classification, weather & vegetation monitoring).
• GIS (Geographic Information Systems) for urban planning and route optimisation.
• Location-based services (traffic prediction, targeted advertising) using mobile GPS data.
• Environmental monitoring (earthquake prediction, pollution mapping, disease-spread tracking).

Web Mining and its Major Types

Web mining is the application of data-mining techniques to discover useful patterns from web data — the content of web pages, the hyperlink structure of the web, and usage records of how users interact with websites. Its three major types are:

1. Web Content Mining — discovers useful information from the content of web pages (text, images, structured data). It includes retrieval (search), classification and clustering of pages, and extraction of structured information (e.g., product metadata). Techniques involve NLP, text mining, and wrapper induction.

2. Web Structure Mining — analyses the hyperlink topology — that is the graph of pages and links — to infer authority, ranking, and communities. Examples: Google's PageRank, HITS (hyperlink-induced topic search). Two aspects: intra-document structure and inter-document hyperlinks.

3. Web Usage Mining — mines the log records generated by servers, browsers, and proxies when users browse a website — that is, how users access the site. Its goal is usage patterns (frequent navigation paths, sessions) for applications such as personalisation, pre-fetching, improving site design, and e-commerce recommendation.

Summary table:

Type	Data source	Goal
Content mining	Page content (text, media)	Extract/classify useful information
Structure mining	Hyperlink graph	Discover authority/hub pages, communities
Usage mining	Server/browser logs, cookies	Understand & predict user behaviour

Challenges in Mining Spatial, Multimedia, Text and Web Data

These are complex data objects: they have rich structure, relationships between objects, or no fixed format, unlike simple records in a table.

Challenges by data type

Data type	Main challenges
Spatial (maps, GIS, satellite data)	Objects are related by location and distance, so records are not independent (spatial autocorrelation); complex data types (points, lines, polygons); topological relations (near, inside, overlaps); different scales and resolutions; very large volume; uncertain or noisy measurements
Multimedia (images, audio, video)	Huge size; high-dimensional features; the semantic gap (low-level features such as colour or pitch vs human meaning); similarity search is costly; temporal order in video and audio; lack of labels
Text	Unstructured; very high dimensionality and sparse vectors; ambiguity (synonyms, polysemy); language dependence, slang, spelling errors; context and sarcasm are hard to capture
Web	Enormous and growing size; dynamic, constantly changing pages; heterogeneous formats; noisy, low-quality and spam content; need to use hyperlink structure and user logs; privacy concerns; distributed data

Why conventional data mining techniques need modification

Classic algorithms (Apriori, decision trees, K-means) assume flat, structured records with independent objects and simple numeric or categorical attributes. Complex objects break these assumptions:

1. Dependence between objects: nearby spatial objects or linked web pages influence each other, so the independence assumption fails; methods must use spatial or link relationships.
2. No ready attributes: images, audio and text must first go through feature extraction (colour histograms, TF-IDF, embeddings) before they can be mined.
3. Special similarity measures: Euclidean distance on raw pixels or words is not meaningful; measures such as cosine similarity, spatial distance or edit distance are needed.
4. High dimensionality and sparsity: requires dimensionality reduction (PCA, LSA) and feature selection.
5. Scalability and indexing: need special index structures (R-trees, inverted indexes) and efficient algorithms.
6. Semantics and context: need ontologies, language models and domain knowledge to interpret meaning.
7. Mixed data: objects can combine several media (text + image + links), so integrated methods are needed.

Examples of adapted techniques

• Spatial: spatial association rules, spatial clustering (DBSCAN), spatial data cubes.
• Multimedia: content-based image retrieval, feature-based classification.
• Text: vector space model, TF-IDF, topic models.
• Web: PageRank, HITS, web usage mining on logs.

Conclusion

Because complex objects are structured, interdependent, high-dimensional and semantic, standard mining methods must be extended with feature extraction, special similarity measures, relationship-aware algorithms and scalable indexing.

Text Mining & Mining the World Wide Web

Text Mining — Definition and Challenges

Text mining (text data mining / knowledge discovery in text) is the process of discovering interesting, novel, and useful knowledge from large collections of unstructured text documents (e.g., web pages, news articles, e-mails, research papers). It applies techniques from data mining, machine learning, NLP (natural language processing), and information retrieval to go beyond simple keyword searches.

Main challenges of text mining:

1. Unstructured nature — text has no fixed schema; information must first be extracted/structured.
2. High dimensionality — documents contain thousands of distinct words/phrases; representing them as features creates very sparse, huge vectors.
3. Ambiguity and semantics — words often have multiple meanings (polysemy) and synonyms; understanding context is difficult.
4. Noisy content — typographical errors, abbreviations, stop words, and slang degrade quality.
5. Multilingual / varied sources — different languages, formats, and lengths make consistent analysis hard.
6. Evolving language — new words, terms, and jargon appear continuously.

Mining the World Wide Web — Challenges and Techniques

Challenges in mining the WWW (beyond traditional data mining):

• Huge, dynamic, and heterogeneous data — billions of pages, changing constantly.
• Unstructured and semi-structured content — most pages are free text or loosely structured HTML rather than rigid database tables.
• Link/authority biases — some pages attract more links regardless of true worth; link spam and hidden text mislead naive metrics.
• Redundancy & duplication — the same information on many mirrors and machine-generated pages.
• Heterogeneous formats — HTML, PDF, images, video, XML, etc.

Techniques.

• Information retrieval / indexing — term weighting (TF-IDF), Boolean/phrase queries, inverted indexes and search engines.
• Web structure mining — using the hyperlink graph (PageRank, HITS) to rank and find authoritative sources.
• Web usage mining — analysing server logs, sessions, and cookies to understand user access patterns.
• Web content mining — page-based (text/HTML tags, wrapper induction) and search-based (meta-search / aggregation of multiple engines) approaches.

Important: because the web is so noisy, dynamic, and huge, knowledge discovery there relies on a combination of IR (to narrow candidates), NLP (to interpret text), and graph analysis (to respect the link structure of pages). It is harder than mining a well-structured relational database because the source data are largely unstructured and continuously changing.

Major Techniques in Text Mining

Text mining extracts useful patterns and knowledge from unstructured text such as emails, reviews, news and documents.

Text preprocessing (first step)

Tokenization, lower-casing, stop-word removal, stemming/lemmatization (running to run), and spelling correction.

1. Document representation

• Bag of words: each document is a vector of word counts, ignoring order.
• Vector space model (VSM): documents and queries are vectors in term space; similarity is measured by cosine similarity.
• N-grams: sequences of n words capture some word order.
• Embeddings (word2vec, BERT): dense vectors that capture meaning.

2. Term weighting

Gives more importance to informative words.

• TF = (count of term in document) / (total words in document). Example: 4 occurrences in 100 words gives TF = 0.04.
• IDF = log(N / df), where N is the number of documents and df the number containing the term.
• TF-IDF = TF x IDF: high for a word frequent in one document but rare overall.

3. Text classification

Assign documents to predefined categories (supervised): spam filtering, sentiment analysis, topic labelling.

• Methods: Naive Bayes, SVM, KNN, decision trees, neural networks.

4. Text clustering

Group similar documents without labels (unsupervised): grouping news stories or support tickets.

• Methods: K-means on TF-IDF vectors, hierarchical clustering, topic models (LDA).

5. Information extraction

Pull structured facts out of text.

• Named entity recognition (people, places, organizations), relation extraction (who works for whom), event extraction; filling templates and databases.

Challenges

Challenge	Problem	Remedies
High dimensionality	Vocabulary can be tens of thousands of terms; vectors are very sparse; slow and prone to overfitting (curse of dimensionality)	Stop-word removal, stemming, feature selection (chi-square, information gain), dimensionality reduction (LSA/SVD, PCA), embeddings
Semantic ambiguity	Polysemy (one word, many meanings, e.g. "bank"), synonymy (many words, one meaning), context, idioms, sarcasm, negation	Word sense disambiguation, ontologies and WordNet, latent semantic analysis, contextual embeddings, part-of-speech tagging

Other issues: multiple languages, noisy and informal text, very large corpora.

Conclusion

Text mining converts raw text into numeric representations (VSM with TF-IDF), then applies classification, clustering and extraction. Its main difficulties are the huge sparse feature space and the ambiguity of natural language, which are handled with dimensionality reduction and semantic methods.

Short Note on Text Mining

1. Definition

Text mining (text data mining) is the process of discovering previously unknown, interesting and useful patterns, associations, concepts and knowledge from large amounts of unstructured or semi-structured text - documents, web pages, e-mails, reports, social-media posts, patents, medical records. It sits at the intersection of information retrieval, natural language processing (NLP), machine learning and statistics. Unlike a keyword search, which only retrieves documents, text mining analyses them to extract knowledge; unlike classical NLP, whose goal is linguistic understanding, its goal is decision-relevant patterns.

2. General process

1. Collect the document corpus from the source (database, web, intranet).
2. Pre-processing / text preparation: tokenisation, sentence segmentation, removal of stop words (the, is, and), noise removal (punctuation, HTML tags), stemming / lemmatisation (running -> run), and sometimes part-of-speech tagging.
3. Text representation: convert text into a machine-usable form - bag-of-words, term frequency - inverse document frequency (TF-IDF) weighting, n-grams, and the vector space model (each document = a numeric vector).
4. Analysis / mining:
   • text categorisation and classification (e-mail spam vs. not spam, news by topic),
   • text clustering (grouping similar documents with no labels),
   • information extraction (entities, relations, events from text),
   • association analysis between terms,
   • sentiment / opinion analysis (positive vs. negative reviews),
   • topic modelling (e.g. Latent Dirichlet Allocation),
   • text summarisation and relationship mining.
5. Evaluation and visualisation of the discovered patterns.

3. Applications

Spam filtering, automatic news categorisation, search-engine ranking, sentiment analysis of product reviews and movie ratings, resume screening, legal/medical document discovery (e-discovery), plagiarism detection, chatbots and question-answering systems, patent and scientific literature analysis, and detection of fraudulent insurance claims.

4. Challenges

• Data is unstructured, heterogeneous and noisy, so meaning must be reconstructed.
• Very high dimensionality and sparsity of the term vector (tens of thousands of terms, most zero).
• Ambiguity: polysemy (one word, many meanings - e.g. bank), synonyms (car = automobile), idioms, sarcasm and domain-dependent language.
• Mixed languages, abbreviations, spelling variants and evolving slang.
• Large volume requires scalable algorithms; evaluation is difficult because there is no single correct answer and labelled training corpora are expensive to build.
• Privacy of the texts being mined.