Web Usage MiningPart-1

Web Usage MiningIt’s main goal is to:Discover usage patterns from web data

in order to understand and better serve the needs of web based applications

Web Usage MiningWeb usage mining consists of three

phases Preprocessing Pattern discovery Pattern analysis

Generated by users’ interaction with the Web, data sources include:

web-server access logs proxy-server logs browser logs user profiles registration data user sessions and transactions cookies user queries bookmark data mouse clicks and scrolls

Web-Usage Mining

Web-Log ProcessingA server log:

set of files consisting of the details of an activity performed by a server

files are automatically created and maintained by the server

The World Wide Web Consortium (W3C) has specified a standard format for web-server log files

There are other proprietary formats for web-server logs.

Most web logs contain: IP address of the client making the request date and time of the request URL of the requested page number of bytes sent to serve the request user agent (such as a web browser or web crawler) referrer (the URL that triggered the request)

Logs can all be stored in one file A better alternative is to separate:

access log error log referrer log

Web-Log Processing

Format of Web Logs

Common log format(http://www. W3.org/Daemon/User/Config/Logging.html#common-logfile-format)

Examples of Common Log Format140.14.6.11 - pawan [06/Sep/2001:10:46:07 -0300] "GET /s.htm HTTP/1.0"

200 2267

140.14.7.18 - raj [06/Sep/2001:11:23:53 -0300] "POST /s.cgi HTTP/1.0" 200 499

GET request that retrieves a file s.htm POST request sends data to a program s.cgi Fields:

client machine’s IP address (140.14.6.11) RFC 1413 identity of the client is missing (-) Date and time Request Error code Number of bytes transferred

Examples of Common Log Format

An example of a log file in extended format

Format of Web Logs#Version: version of the extended log file format

used#Fields: fields recorded in the log#Software: software that generated the log#Start-Date: date and time at which the log was

started#End-Date: date and time at which the log was

finished#Date: date and time at which the entry was

added#Remark: Comments that are ignored by

analysis tools

The directives #Version and #Fields are mandatory and must appear before all the entries

Each field in the #Fields directive can be specified in one of the following ways: an identifier; e.g., time an identifier with a prefix separated by a

hyphen; e.g., cs-method a prefix following a header in parentheses;

e.g., sc(Content-type)

Format of Web Logs

No prefixes for date, time, time-taken, bytes, cached

Prefixes for ip, dns, status, comment, method, uri, uri-stem, uri-query, host

Prefixes can be:cs client to serversc server to clientsr server to remote server (this prefix is used by proxies)rs remote server to server (this prefix is used by proxies)x application-specific identifier

Format of Web Logs

Analyzing Web logs

Analyzing Web Logs

General Summary from Analog

Analyzing Web Logs

Monthly report from Analog

Analyzing Web Logs

Daily summary from Analog

Hourly summary from Analog

Analyzing Web Logs

Organization report from Analog

Organization report from Analog

Search-word report from Analog

Operation-system report from Analog

Status-code report from Analog

File size report from Analog

File type report from Analog

Directory report from Analog

FRequest report from Analog

Analysis of Clickstream: Studying Navigation Paths

Clickstream using Pathalizer with seven link specification

Analysis of Clickstream: Studying

Navigation Paths

Analysis of Clickstream: Studying

Navigation Paths

Clickstream using Pathalizer with twenty link specification

Visualizing Individual User Sessions

A brief on-campus session identified by StatViz that browses the bulletin board

Visualizing Individual User Sessions

A brief off-campus session identified by StatViz with three distinct activities

Visualizing Individual User Sessions

A long on-campus session identified by StatViz with multiple activities

Caution in Interpreting Web-Access Logs Requests may not always reach the server as they

may be served from a proxy server’s cache You do not really know:

Identity of readers Number of visitors Number of visits User’s navigation path through the site Entry point and referral How users left the site or where they went next How long people spent reading each page How long people spent on the site

Turner (2004)I’ve presented a somewhat negative view here, emphasizing what you can’t find out. Web statistics are still informative: it’s just important not to slip from “this page has received 30,000 requests” to “30,000 people have read this page.” In some sense these problems are not really new to the web—they are present just as much in print media too. For example, you only know how many magazines you’ve sold, not how many people have read them. In print media we have learnt to live with these issues, using the data which are available, and it would be better if we did on the Web too, rather than making up spurious numbers.