web archive analysis - rutgers...
TRANSCRIPT
2
Access Web Archive Data
Wayback Machine
Search
3
Enable Research & Analysis
Analysis Workflow
5
Data: WARC
Data written by web crawlers Web ARchive Container File - WARC (ISO standard) Revision of the ARC file format Each file contains a series of concatenated records
– Full HTTP request/response records
– Metadata records (links, crawler path, encoding etc.)
– Records to store duplicate detection events
– Records to support segmentation and conversion
6
Derived Data: CDX
Index for Wayback Machine Space delimited text file Contains only essential fields needed by Wayback
– URL, Timestamp, Content Digest
– MIME type, HTTP Status Code
– Redirect URL, meta tags, size
– WARC filename and file offset of record
7
Wayback Machine
8
Growth of content
9
Growth of content
10
Rate of duplication
11
Breakdown by Year-First-Crawled
Log Analysis (Hive/Pig/Giraph)
CDX Warehouse Crawl Log Warehouse Distribution of HTTP status codes, MIME types Find timeout errors, duplicate content, crawler traps,
robots exclusions Trace path of the crawler
13
Derived Data: Parsed Text
Input to build text indexes for Search Text is extracted from (W)ARC files HTML boilerplate is stripped out Also contains metadata for each record
– URL, Timestamp, Content Digest, Record Length
– MIME type, HTTP status code
– Title, description and meta keywords
– Links with anchor text
14
Derived Data: WAT
Extensible Metadata format Essential metadata for many types of analyses Avoids barriers to data exchange: copyright, privacy Less data than WARC, more than CDX WAT records are WARC metadata records Contains for every HTML page in the WARC,
– Title, description and meta keywords
– Embeds and outgoing links with alt / anchor text
15
Text Analysis (Pig/Mahout)
Text extracted from WARC / Parsed Text / WAT files Use curated collections to train Classifiers Cluster documents in collections Topic Modeling
– Discover topics
– Study how topics evolve over time Compare how a page describes itself (meta text) vs.
how other pages linking to it describe it (anchor text)
16
Link Analysis (Pig/Giraph)
Links extracted from crawl logs / WARC metadata records / Parsed Text / WAT files
Indegree and Outdegree information Inter-host and Intra-host link information Study how linking behavior changes over time Rank resources by PageRank
– Identify important resources
– Prioritize crawling of missing resources Find possible spam pages by running biased PageRank
algorithms
17
Completeness
18
PageRank over Time
19
PageRank over Time
20
PageRank over Time
21
Archive Analysis Workshop
Generate derivatives: CDX, WAT, Parsed Text Set up CDX Warehouse using Hive Extract links from WARCs / WAT / Parsed Text Extract text from WARCs / WAT / Parsed Text Generate Archival web graphs
– Assign integer / fingerprint ID to URLs
– Represent graph as an adjacency list using these IDs and the timestamp info
Generate host and domain graphs
22
Archive Analysis Workshop
Text Analysis using Pig / Mahout
– Extract top terms using TF-IDF
– Prepare text for analysis with Mahout Link Analysis with Pig / Giraph
– Degree Analysis
– PageRank
– Find common links between entities Data extraction
– Repackage subset of data into new (W)ARCs
23
Archive Analysis Workshop
https://webarchive.jira.com/wiki/display/Iresearch/Web+Archive+Analysis+Workshop