tpdl 2015 - profiling web archives

Profiling Web Archives

Sawood Alam and Michael L. NelsonComputer Science Department, Old Dominion University

Norfolk, Virginia - 23529

Herbert Van de Sompel, Lyudmila L. Balakireva, and Harihar ShankarLos Alamos National Laboratory, Los Alamos, NM

David S. H. RosenthalStanford University Libraries, Stanford, CA

Supported in part by the International Internet Preservation Consortium (IIPC)

Memento Aggregator

Long Tail of Archives

Long Tail of Archives

● 400B+ web pages at IA do not cover everything

● Top three archives after IA produce full TimeMap 52% of the time (AlSum et al, TPDL 2013)

● Targeted crawls● Special focus archives● Restricted resources● Private archives

Archive Profile

● High-level summary of an archive● Predicts presence of mementos of a URI-R

in an archive● Provides various statistics about the

holdings● Small in size● Publicly available● Easy to update and partially patch● Useful for Memento query routing and other

things

Available Profiling Resources

● Client request● Archive response● Archive index (CDX files)

A Client Request

An Archive Response

A CDX Snippet

Profiling Strategies

● Complete URI-R Profiling (1 URI-R = 1 Profile Key)

○ bbc.co.uk/images/logo.png?w=90○ cnn.com/2014/03/15/?id=128734

● TLD-only Profiling (1 TLD = 1 Profile Key)

○ com)/○ uk)/

● Middle Ground○ uk,co)/○ uk,co,bbc)/images○ uk,co,bbc)/0/2/1○ com,cnn)/ 201309 ar

Frequency Measurements

CDXJ Serialization

URI-Key Generation

Profile Merging

Base profile

New profile

Merged profile

Dataset

● Three archives● Four sample query sets● 23 profiles for each archive and sample set

Archives

Archive URI-Rs URI-Ms Size

Archive-It 1.9B 5.3B 1.8TB

UKWA 0.7B 1.7B 0.5TB

Stanford 12M 25M 8.3GB

Sample Query Sets

Sample In Archive-It In UKWA In Stanford

DMOZ 4.097% 1.912% 0.034%

MementoProxy 4.182% 0.179% 0.046%

IAWayback 3.716% 0.231% 0.039%

UKWayback 0.108% 0.034% 0.002%

Sample Size: 1M URIs Each

Evaluation

● Relate CDX Size, URI-M, URI-R, and URI-Key

● Analyze profile growth● Estimate Relative Cost● Evaluate Routing Precision vs. Relative Cost

CDX Size vs URI-M (UKWA 10 Years)

Alpha: 175 bytes per CDX line

URI-M vs URI-R (UKWA 10 Years)

Gamma: 2.46 K : 2.686Beta: 0.911

Space Cost (UKWA 7 Years)

Phi: 8.5e-07 -- 0.70583

Time Cost (UKWA 7 Years)

Tau: 5.7e-05 -- 6.2e-05CDX: 45GBURI-Ms: 181MURI-Rs: 96MTime: 3 hours

Resource Requirement

Archive-It

Stanford

Cost vs Precision

Group Cost Precision

G1 (H1P0/TLD) Bound by # of TLDs < 0.05

G2 (H3P0, DDom, DSub, DPth, DQry) < 0.01 ≈ 2 * G1

G3 (DIni) ≈ 2 * G2 ≈ (3--4) * G1

G4 (HxP1) ≈ 5 * G3 ≈ (5--7) * G1

G5 (Higher HmPn) 0.4 -- 0.7 Not Explored

G6 (URIR) 1.0 1.0

Future Work

● Generating sample URI sets● Profiling via sampling● Language profiles● Evaluation of combination profiles such as

URI-Key along with Datetime● Profiles for usage other than Memento

routing, such as, site classification based profiles (e.g., news, wiki, social media, blog etc.)

Conclusions

● Generated profiles with different policies for two archives

● Examined cost-precision tradeoffs of various policies

● Related CDX Size, URI-M, URI-R, and URI-Key

● Gained up to 22% routing precision with <5% relative cost without any false negatives

● Code @ GitHub:/oduwsdl/archive_profiler