Web MiningBy:-

Vineeta

8pgc18

M.Tech (II Semester)

Introduction Why we need ? What is it ? How it is different from classical data mining ? What are the problems ? Role of web mining Web mining Taxonomy Applications

Why we need Web Mining?

Explosive growth of amount of content on the internet

Web search engines return thousands of results so difficult to browse

Online repositories are growing rapidly

Using web mining web documents can easily be BROWSED,

ORGANISED and CATALOGED with minimal human

intervention

What is it? Web mining - data mining techniques to automatically

discover and extract information from web documents/services

Knowledge

www

How does it differ from “classical” Data Mining?

The web is not a relation Textual information and linkage structure

Usage data is huge and growing rapidly Google’s usage logs are bigger than their web crawl Data generated per day is comparable to largest

conventional data warehouses

Ability to react in real-time to usage patterns No human in the loop

Web Mining: Problems The “abundance” problem Limited coverage of the Web Limited query interface based on keyword-oriented

search Limited customization to individual users Dynamic and semi structured

Role of web mining Finding Relevant Information

Creating knowledge from Information available

Personalization of the information

Learning about customers / individual users

Web Mining Taxonomy

Web Mining

Web ContentMining

Web StructureMining

Web UsageMining

Identify informationwithin given webpages

Distinguish personalhome pages fromother web pages

Understand access patterns and the trendsto improve structure

Uses interconnectionsbetween web pages to give weight to the pages

Web Content Mining Web Content Mining is the process of extracting

useful information from the contents of Web documents.

Content data corresponds to the collection of facts a Web page was designed to convey to the users. It may consist of text, images, audio, video, or structured records such as lists and tables.

Research activities in this field also involve using techniques from other disciplines such as Information Retrieval (IR) and natural language processing (NLP).

Web Content Mining

Web Content Mining

Agent Based Approach

InformationFiltering &

Categorization

MultilevelDatabases

PersonalizedWeb Agent

Database Approach

IntelligentSearchAgent

Web QuerySystems

Intelligent Search Agents Concentrate on searching relevant information using

the characteristics of a particular domain to interpret and organize the collected information.

It can be further classified into two types: Interpretation Based on Pre-Specified Information:

Examples: Harvest, FAQFinder, Information Manifold, OCCAM

Interpretation Based on Unfamiliar Source: Example: ShopBot

ShopBot A ShopBot is an autonomous software agent that

comb the internet providing users with low price product or product recommendations.

A ShopBot basically looks for product information from a variety of vendor sites using the general information about the product domain.

The following example displays a shopBot at www.allbookstores.com.

Information Filtering & Categorization

This makes use of various information retrieval techniques and characteristics of hypertext web documents to interpret and categorize data.

Examples: HyPursuit, BO (Bookmark Organizer).

Bookmark Organizer (BO) Makes use of hierarchical clustering techniques and

involves user interaction to organize a collection of web documents.

It operates in two modes: Automatic Manual

Frozen Nodes: In a hierarchical structure, if we freeze a node N, then the subtree rooted at N represents a coherent group of documents.

Personalized Web Agents This category of Web agents learn user preferences

and discover Web information sources based on these preferences, and those of other individuals with similar interests.

Examples: WebWatcher PAINT Syskill&Webert GroupLens Firefly

Multilevel Databases Layer 0 :

Unstructured, massive and global information base. Layer 1:

Derived from lower layers. Relatively structured. Obtained by data analysis, transformation &

Generalization. Higher Layers (Layer n):

Further generalization to form smaller, better structured databases for more efficient retrieval.

Web Query System These systems attempt to make use of:

Standard database query language – SQL Structural information about web documents Natural language processing for queries made in www

searches. Examples:

WebLog: Restructuring extracted information from Web sources.

W3QL: Combines structure query (organization of hypertext) and content query (information retrieval techniques).

Web Structure Mining

Web Structure Mining is the process of discovering structure information from the Web. This type of mining can be performed either at the (intra-page) document level or at the (inter-page) hyperlink level.The research at the hyperlink level is also called

HYPERLINK ANALYSIS

Web Structure Mining

Different Algorithms for Web Structures: Page-Rank Method

Sergey Brin and Lawrence Page: The anatomy of a large-scale hypertextual web search engine. In Proc. Of WWW, pages 107–117, Brisbane, Australia, 1998.

CLEVER Method

http://www.almaden.ibm.com/projects/clever.shtml

Page-Rank Method Introduced by Brin and Page (1998) Used in Google Search Engine Mine hyperlink structure of web to produce ‘global’

importance ranking of every web page Web search result is returned in the rank order Treats link as like academic citation Assumption: Highly linked pages are more ‘important’

than pages with a few links A page has a high rank if the sum of the ranks of its

back-links is high

Backlink

Link Structure of the Web

CLEVER Method CLient–side EigenVector-Enhanced Retrieval Developed by a team of IBM researchers at IBM

Almaden Research Centre Ranks pages primarily by measuring links between

them Continued refinements of HITS ( Hypertext Induced

Topic Selection) Basic Principles – Authorities, Hubs

Good hubs points to good authorities Good authorities are referenced by good hubs

Web Usage Mining

Web usage mining also known as Web log mining mining techniques to discover interesting usage

patterns from the data derived from the interactions of the users while surfing the web

mining Web log records to discover user access patterns of Web pages

Web Usage Mining – Three Phases

Web Usage Mining Pre processing consists of converting the usage, content, and

structure information contained in the various available data sources into the data abstractions necessary for pattern discovery

Pattern discovery draws upon methods and algorithms developed from several fields such as statistics, data mining, machine learning and pattern recognition.

The motivation behind pattern analysis is to filter out uninteresting rules or patterns from the set found in the pattern discovery phase. The exact analysis methodology is usually governed by the application for which Web mining is done.

Applications

Personalized experience in B2C e-commerce –Amazon.com

Web search –Google Web-wide user tracking –DoubleClick Understanding user communities –AOL Understanding auction behavior –eBay Personalized web portal –MyYahoo

Conclusion Web mining - data mining techniques to

automatically discover and extract information from Web documents/services (Etzioni, 1996). Web mining research – integrate research from several research communities (Kosala and Blockeel, July 2000) such as:

Database (DB) Information retrieval (IR) The sub-areas of machine learning (ML) Natural language processing (NLP)

References mandolin.cais.ntu.edu.sg/wise2002/web-mining-

WISE-30 David Gibson, Jon Kleinberg, and Prabhakar

Raghavan. Inferring web communities from link topology. In Conference on Hypertext and Hypermedia. ACM, 1998.

www.iprcom.com/papers/pagerank/ http://maya.cs.depaul.edu/~mobasher/webminer/

survey/node23.html

References http://en.wikipedia.org/wiki/Web_mining

http://en.wikipedia.org/wiki/Shop_bot

Y. S. Mareek and I. Z. B. Shaul. Automatically organizing bookmarks per contents. Proc. Fifth International World Wide Web Conference, May 6-10 1996.

Cooley, R., B. Mobasher, et al. (1997). Web Mining: Information and Pattern Discovery on the World Wide Web, Proc. IEEE Intl. Conf. Tools with AI, Newport Beach, CA, pp. 558-567, 1997.

References R. Kosala. and H. Blockeel, Web Mining Research:

A Survey, SIGKDD Explorations, 2(1):1-15, 2000.

R. Cooley, B. Mobasher, and J. Srivastava. Data preparation for mining world wide web browsing patterns. Journal of Knowledge and Information Systems 1, 5-32, 1999

S. Chakrabarti, Data mining for hypertext: A tutorial survey. ACM SIGKDD Explorations, 1(2):1-11, 2000System, 1(1), 1999

THANK YOU!!