Mining the World Wide Web

Definition. Web mining is the automatic discovery of useful patterns and knowledge from web documents, from the link structure of the web, and from the records of how users interact with web sites. Because the web is a huge, distributed, semi-structured and constantly changing hypertext repository, ordinary data mining techniques are adapted to work on HTML/XML text, hyperlink graphs and server or clickstream data.

Types / Process of Web Mining

1. Web content mining — mines the actual content of pages: text, images, audio, metadata and embedded tables. It uses text categorisation, information extraction, wrapper induction and document clustering, and can work either on the raw pages or on the result sets returned by search engines.
2. Web structure mining — mines the hyperlink structure to evaluate importance and relatedness of pages. PageRank, the hub-and-authority (HITS) model and discovery of web communities are the standard examples.
3. Web usage mining — mines user-interaction data such as server/access logs, clickstreams, cookies and registration data. The process is:
   • Data collection — gather server logs, proxy logs, client-side records.
   • Data preprocessing — clean the logs, identify users and sessions, reconstruct visit sequences.
   • Pattern discovery — apply clustering, association rules and sequential pattern mining.
   • Pattern analysis — extract actionable knowledge for personalisation, recommendation and site redesign.

Web Mining vs Data Mining

Basis	Data mining	Web mining
Source	Centralised databases / data warehouses	Distributed web pages, links and logs
Data type	Mostly structured tuples and attributes	Semi-structured / unstructured HTML, XML, text, images, clickstreams
Data quality	Curated, consistent, pre-processed	Noisy, redundant, inconsistent formats, missing values
Schema	Fixed and known in advance	No global schema; structure must be extracted
Scale and change	Relatively static	Extremely large and continuously updated
Relationships	Foreign keys and explicit relations	Hyperlinks, page hierarchy, user sessions
Goal	Hidden patterns, rules, clusters, models	Page relevance, site structure, user behaviour, personalisation

In short, web mining is data mining applied to web-derived data, but it must additionally cope with hyperlink structure, unstructured text and highly dynamic, noisy content.