Mining the World Wide Web

Q: Name and explain the categories of web mining.

Content mining analyses page text and media. Structure mining analyses the hyperlink graph. Usage mining analyses logs and click behaviour of users.

Q: Differentiate PageRank and HITS.

PageRank gives one global, query-independent score per page. HITS gives two scores (hub and authority) computed on a query-specific subgraph.

Q: What are hubs and authorities?

An authority is a page with many good in-links on a topic. A hub is a page that points to many good authorities.

Q: What is a wrapper?

A program or learned set of rules that extracts structured records (for example product and price) from semi-structured HTML pages.

Web mining applies data mining techniques to discover patterns from the content, link structure and usage of the Web. The Web is a huge, dynamic, hyperlinked and semi-structured collection, so it needs special methods.

Important

The three categories of web mining (content, structure, usage), PageRank and the web usage mining pipeline are the core of this topic. Each is a 7-10 mark question.

1. Why the Web is Special

Characteristic	Consequence for mining
Huge and growing	Needs scalable, distributed algorithms
Hyperlinked (a graph)	The link structure itself is knowledge (PageRank, HITS)
Semi-structured and heterogeneous	HTML has no fixed schema, so wrappers are needed
Dynamic and noisy	Broken links, duplicates, spam, constant change
Distributed and autonomous	No central control, many formats
Mostly text and media	Needs text and multimedia mining
User behaviour is logged	Query logs and clickstreams enable usage mining

2. Three Categories of Web Mining

MermaidDiagram

Rendering diagram…

	Content	Structure	Usage
Data	Page text and media	Hyperlink graph	Logs, clickstream
Techniques	IR, NLP, classification, clustering	PageRank, HITS, link analysis	Association, sequences, clustering
Typical use	Search, comparison shopping	Ranking, communities	Personalization, caching, site redesign

3. Web Content Mining

Goal: extract knowledge from the content of pages.

Unstructured text: topic discovery, classification, clustering, sentiment, summarization (TF-IDF with Naive Bayes or SVM).
Semi-structured pages: wrapper induction learns extraction rules from labelled pages to pull out records (such as product name and price from HTML tables).
Multimedia: colour, texture and shape features for images, speech-to-text for audio, key frames for video.
Applications: search and ranking, price comparison, news aggregation, building knowledge bases and knowledge graphs.

4. Web Structure Mining

Goal: discover knowledge from the hyperlink graph $G = (V, E)$ where $V$ is the set of pages and $E$ the hyperlinks.

NoteKey idea

A link from page A to page B is a vote of confidence for B. Pages that receive many good votes are important.

(a) PageRank

The importance of page $A$ depends on the importance of the pages that link to it:

PR(A) = (1 - d) + d \sum_{T_i \to A} \frac{PR(T_i)}{C(T_i)}

$d$ is the damping factor (about 0.85).
$C(T_i)$ is the number of outgoing links of page $T_i$ .
The $(1 - d)$ term models a random surfer who jumps to a random page, and handles dangling pages.
It is computed iteratively (power iteration) and is query-independent.

(b) HITS (Hubs and Authorities)

Each page gets two scores:

a_p = \sum_{q:\, q \to p} h_q \qquad\qquad h_p = \sum_{q:\, p \to q} a_q

Authority: a page with many good in-links from hubs.
Hub: a page that links to many good authorities.

MermaidDiagram

Rendering diagram…

	PageRank	HITS
Scores per page	One	Two (hub and authority)
Query dependence	Independent of the query	Computed on a query-specific subgraph
Typical use	Global ranking	Topic-focused ranking

(c) Other structure tasks

Link prediction, community detection (dense bipartite cores), detecting web spam and link farms, and combining text with links.

5. Web Usage Mining

Goal: discover patterns from logs: who visited what, when and how.

Log sources: web-server logs, proxy-server logs, client-side logs and application or query logs.

A typical access-log line:

203.197.118.4 - - [20/Sep/2026:10:15:22 +0530] "GET /dm/notes/unit5.pdf HTTP/1.1" 200 51234 "-" "Mozilla/5.0"

Fields: IP, user, timestamp, method and URL, status code, bytes, referrer, user agent.

Usage mining pipeline

Data collection

Gather raw server, proxy, client or query logs.

Useful measures:

\text{Session length} = \#\text{pageviews}, \qquad \text{Conversion rate} = \frac{\#\text{orders}}{\#\text{visits}}

Pattern type	Example finding	Action
Association	60% of users who view eligibility also view fees	Add a direct link
Sequential path	`home` then `semester6` then `data-mining` then `notes`	Cache these pages
Clustering	Exam-focused vs concept-focused users	Two landing pages
Classification	Predict which sessions end in a download	Tune content

6. Integrated View

MermaidDiagram

Rendering diagram…

7. Applications and Challenges

Applications	Challenges
Search-engine ranking, focused crawling	Scalability: billions of pages, huge logs
Spam and fraud detection	Spam and adversarial content: link farms, click fraud, fake reviews
Recommender systems, personalized portals	Hidden / deep web and constantly changing pages
E-commerce: market basket, price comparison	Privacy and ethics of user behaviour data
E-learning, ad targeting	Heterogeneity: schema-less HTML, ambiguous language
Brand and sentiment monitoring	Evaluation: no gold standard for "interesting" patterns

Exam TrapTypical questions

Draw and explain the three categories of web mining with techniques and applications (10 marks, very frequent).
PageRank with formula and damping factor; also HITS (7-10 marks).
The web usage mining pipeline: cleaning, user and session identification, pattern discovery, analysis (10 marks).
Compare content, structure and usage mining (5 marks).
Short note on web log formats or challenges (5 marks).

Frequently Asked Questions

QWhat is web mining?

Web mining is the use of data mining techniques to discover useful information and patterns from the content, hyperlink structure and usage of the World Wide Web.

QName and explain the categories of web mining.

Content mining analyses page text and media. Structure mining analyses the hyperlink graph. Usage mining analyses logs and click behaviour of users.

QWhat is PageRank and what is the damping factor?

PageRank scores a page by the importance of the pages linking to it: $PR(A) = (1-d) + d \sum PR(T_i)/C(T_i)$ . The damping factor $d$ (about 0.85) is the probability that the random surfer follows a link instead of jumping to a random page. It also solves the problem of dangling pages.

QDifferentiate PageRank and HITS.

PageRank gives one global, query-independent score per page. HITS gives two scores (hub and authority) computed on a query-specific subgraph.

QWhat are hubs and authorities?

An authority is a page with many good in-links on a topic. A hub is a page that points to many good authorities.

QWhat is a wrapper?

A program or learned set of rules that extracts structured records (for example product and price) from semi-structured HTML pages.

QWhat data sources are used in web usage mining?

Web-server logs, proxy-server logs, client-side logs (cookies, scripts) and query or application logs.

QWhy is pre-processing important in usage mining?

Raw logs contain robot traffic, failed requests and mixed users. Cleaning, user identification, session identification and path completion are needed so that discovered patterns are reliable.

QList some challenges of web mining.

Scalability, spam and link farms, dynamic and hidden content, privacy, heterogeneous formats and the lack of a gold standard for evaluation.

Mining the World Wide Web

Mining the World Wide Web

1. Why the Web is Special

2. Three Categories of Web Mining

3. Web Content Mining

4. Web Structure Mining

(a) PageRank

(b) HITS (Hubs and Authorities)

(c) Other structure tasks

5. Web Usage Mining

Usage mining pipeline

Data collection

6. Integrated View

7. Applications and Challenges

Frequently Asked Questions

Coursify — Personal AI Course Generator