Internet Content as Research Data

Australian National University August 2012, Canberra

Monica Omodei

Research Examples •  Social networking •  Lexicography •  Linguistics •  Network Science

•  Political Science •  Media Studies •  Contemporary history

Data-driven science is migrating from the natural sciences to humanities and social science

Talk  Structure  

•  Exis0ng  web  archives  •  Web  archive  use  cases  •  Bringing  archives  together  •  Crea0ng  your  own  archive  •  It’s  ge>ng  harder  –  challenges  •  Web  data  mining  &  analysis        

Exis0ng  web  archives    

•  Internet  Archive  •  Common  Crawl    •  Pandora  Archive  •  Internet  Memory  Founda0on  Archive  •  Other  na0onal  archives  •  Research,  University  Library  archives    

Common  Collec0on  Strategies  

•  Crawl  Scope  &  Focus  1)  Thema0c/Topical  (elec0ons,  events,  global  warming…)  2)  Resource-­‐specific  (video,  pdf,  etc.)  3)  Broad  survey  (domain  wide  for  .com/.net/.org/.edu/.gov)  4)  Exhaus0ve  (end  of  life, closure crawls, natl domains)  5)  Frequency-­‐Based    

•  Key  Inputs:  nomina0ons  from  subject  ma^er  experts,  prior  crawl  data,  registry  data,  trusted  directories,  wikipedia,  twi^er  

Internet Archive’s Web Archive

Positives – Very broad – 175+ billion web instances – Historic – started 1996 – Publicly accessible – Time-based URL search – API access – Not constrained by legislation – covered by

fair use and fast take-down response

Internet  Archive’s  Web  Archive  Negatives

– Because of size can’t search by keyword – Because of size crawling is fully automated –

ergo QA is not possible  

Common  Crawl  

•  Non-­‐profit  founda0on  building  an  open  crawl  of  the  web  to  seed  research  and  innova0on  

•  Currently  5  billion  pages  •  Stored  on  Amazon’s  S3    •  Accessible  via  MapReduce  processing  in  Amazon’s  EC2  compute  cloud  

•  Wholesale  extrac0on,  transforma0on,  and  analysis  of  web  data  cheap  and  easy  

Common  Crawl  

Nega0ves  •  Not  designed  for  human  browsing  but  for  machine  access  

•  Objec0ve  is  to  support  large-­‐scale  analysis  and  text  mining/indexing  –  not  long-­‐term  preserva0on  

•  Some  costs  are  involved  for  direct  extrac0on  of  data  from  S3  storage  using  Requester-­‐Pays  API    

Pandora  Archive  •  Posi0ves  

– Quality  checked  – Targeted  Australian  content  with  selec0on  policy  – Historical  –  started  1996  – Bibliocentric  approach  –web  sites/publica0ons  selected  for  archiving  are  catalogued  (see  Trove)  

– Keyword  search  – Publicly  accessible  – You  can  nominate  Australian  web  sites  for  inclusion  -­‐  pandora.nla.gov.au/registra0on_form.html  

Pandora  Archive  

•  Nega0ves  –  labour  intensive  thus  quite  small  – significant  content  missed  because  permission  to  copy  refused  

•  Situa0on  will  improve  markedly  if  Legal  Deposit  provisions  extended  to  digital  publica0ons  

•  Broader  coverage  will  be  achieved  when  infrastructure  is  upgraded  hence  reducing  labour  costs  for  checking/fixing  crawls  

Pandora  Archive  Stats  

•  Size  –  6.32  TB  •  Number  of  Files    >  140  million  •  Number  of  ‘0tles’  >  30.5K  •  Number  of  0tle  instances  >  73.5K  

Which archived sites are popular ?  •  Measure: filtered, aggregated web access

log data which counts access to “titles”"•  Examined top 30 archived titles (# of

accesses) for each year 2009 to 2012"•  Selected some to examine and speculate

as to why they might be popular"•  Selected those with consistently high

ranking, and ones that were very variable between years  

Reasons for popularity of archived version  

•  Were once popular and are now decommissioned, particularly if domain name continues to exist and redirects to the archive"

•  May not be that popular as live sites but their live site links prominently to Pandora as an archive for their content"

•  Popular referencing sources cite the archive as well as the live site (if it still exists)  

Improving visibility and usage of Pandora archive  

•  Articles about interesting content on the Australia Web Archives blog –http://blogs.nla.gov.au/australias-web-archives/"

•  More effort to identify archived sites that are no longer ʻliveʼ"

•  Market automatic redirect services to web site owners/managers"

•  Allow Google to index archive content for ʻnon-liveʼ sites (problematic)"

•  Install Twittervane - draws  site  nomina0ons  for  archiving  based  on  trending  Twi^er  topics.      "

.au  Domain  Annual  Snapshots  •  Annual  crawls  since  2005  commissioned  from  Internet  Archive  

•  Includes  sites  on  servers  located  in  Australia  as  well  as  .au  domain  

•  Robots.txt  respected  except  for  inline  images  and  stylesheets  

•  No  public  access  –  researcher  access  protocols  are  being  developed  

•  Full  text  search  –  suited  to  searching  archives  •  Separate  .gov  crawl  publicly  accessible  soon  

Australian  web  domain  crawls  

Year   2005   2006   2007   2008   2009   2011  

Files   185  million  

596  million  

516  million  

1  billion   765  million  

660  million  

Hosts  crawled  

811,523   1,046,038   1,247,614   3,038,658   1,074,645   1,346,549  

Size  (TBs)   6.69   19.04   18.47   34.55   24.29   30.71  

Internet  Memory  Founda0on  •  Number  of  European  partners    •  LiWA  –  Living  Web  Archives:  next  genera0on  Web  archiving  methods  and  tools    

•  LAWA  –  Longitudinal  Analy0cs  of  Web  Archive  Data:  experimental  testbed  for  large-­‐scale  data  analy0cs  

•  ARCOMEM  (Collect-­‐All  ARchives  to  COmmunity  MEMories)  leveraging  social  media  for  Intelligent  Preserva0on    

•  SCAPE  –  Scalable  Preserva0on  Environments  

Other  Na0onal  Archives  •  List  of  Interna0onal  Internet  Preserva0on  Consor0um  member  archives  –  netpreserve.org/about/archiveList.php  

•  Some  are  whole  domain  archives,  some    are  selec0ve  archives,  many  are  both  

•  Some  have  public  access,  others  you  will  need  to  nego0ate  access  for  research  

•  Most  archives  have  been  collected  using  the  heritrix  open-­‐source  crawler  and  thus  use  the  standard  format  (warc  ISO  format)  

Research  Archives  •  California  Digital  Library  •  Harvard  University  Libraries  •  Columbia    University  Libraries  •  University  of  North  Texas  ….  and  many  more    •  WebCITE  -­‐  webcita0on.org  (cita0on  service  archive)  

Example:  Columbia  University  •  Member  of  the  IIPC  •  They  use  the  ArchiveIt  service  •  A  Research  library  that  sees  web  archiving  as  fundamental  to  their  collec0ng    

•  They  complement  and  coordinate  with  other  web  archives  

•  Their  collec0ng  focus  is  thema0c  –  eg  human  rights,  historic  preserva0on,  NY  religious  ins0tu0ons  

•  They  also  archive  web  content  as  part  of  personal  and  organisa0onal  archives  (c.f.  manuscripts  coll)  

•  Archive  their  own  web  site  regularly  

Bringing  Archives  Together  

•  Common  standards  and  APIs  •  Memento  project  –  adding  0me  to  the  web  

– Aggregates  CDX  files  (URL  index)  from  mul0ple  archives  

– Has  a  Firefox  plug-­‐in  which  allows  0me-­‐based  browsing  

–  Ini0a0ve  of  Los  Alamos  Laboratories  – See  h^p://www.mementoweb.org/demo/  

 

Common  Use  Cases  for  a  web  archive  

•  Content  discovery  •  Nostalgia  queries  •  Web  site  restora0on  and  file  recovery  •  Domain  name  valua0on  •  Fall-­‐back  for  link-­‐rot  •  Prior  art  analysis  and  patent/copyright  infringement  research  

•  Legal  cases  •  Topic  analysis,  web  trends  analysis,  popularity  analysis,  network  analysis,  linguis0c  analysis  

Create  your  own  Archive  

•  Use  a  subscrip0on  service  •  Build  your  own  web  archiving  infrastructure  with  open  source  sonware  (  ie  Heritrix  and  Wayback)  

•  Use  web  cita0on  services  that  create  archive  copies  as  you  bookmark  pages  

Subscrip0on  Services  

•  archive-­‐it.org  (service  operated  by  non-­‐profit  Internet  Archive  since  2006)  

•  archivethe.net  (service  operated  by  non-­‐profit    Internet  Memory  Founda0on)  

•  California  Digital  Library  Web  Archiving  Service  -­‐  cdlib.org/services/uc3/was.html  

•  OCLC  Harvester  Service  -­‐  oclc.org/webharvester/overview/default.htm  

Install  web  archiving  system  locally  

•  Easy-­‐to-­‐deploy  web  archiving  toolkit  not  yet  available    

•  Ins0tu0onal  web  archiving  infrastructure  is  feasible  and  has  been  established  at  a  number  of  universi0es  for  use  by  researchers  –  needs  IT  systems  engineers  to  set  up  though  

•  Archives  can  be  deposited  with  the  NLA  for  long-­‐term  preserva0on  

Personal  Web  Archiving  

•  WARCreate  –  recently  released  free  tool  which  creates  wayback-­‐consumable  warc  files  from  any  web  page  

•  Google  Chrome  extension  •  Enables  preserva0on  by  users  from  their  desktop  •  Can  target  content  unreachable  by  crawlers  •  Brings  WARC  to  personal  digital  archiving  •  What  you  do  with  the  WARC  files  is  up  to  you  •  Install  suite  provided  to  set  up  local  Wayback  instance  and  Memento  0megate  

Current  challenges  

•  Database-­‐driven  features  and  func0ons  •  Complex  and  varying  URI  formats  and  non-­‐standard  link  implementa0ons  eg  Twi^er  

•  Dynamically  generated  ever-­‐changing  URIs  – For  serving  the  same  resources  

•  Rich  Media  –  eg  streamed  media  with  custom  apps  and  ant-­‐collec0on  measures  

•  Scripted  incremental  display  and  page-­‐loading  

…  more…  

•  Scripted  HTML  forms  •  Mul0-­‐sourced  embedded  material  •  Dynamic  authen0ca0on  e.g.  captchas,  cross-­‐site  authen0ca0on,  user-­‐sensi0ve  embeds  

•  Alternate  display  based  on  browser  or  device,  or  other  parameter  

•  Site  architecture  designed  to  inhibit  crawling  and  indexing  –  but  if  poorly  done  even  ‘polite’  harvesters  like  Heritrix  may  crash  their  server  

..  but  wait,  there’s  more  …  

•  Server-­‐side  scripts  and  remote  procedure  calls  –  the  full  variety  of  paths  through  a  site  are  now  onen  hidden  in  remote/opaque  server-­‐side  code  –  not  a  new  problem  but  now  effects  80+%  of  online  resources  

•  HTML  5  web  sockets  –  effec0vely  codifies  incremental  updates  without  page  reloads  

•  Mobile  publishing  

Transac0onal  Web  Archiving  •  Useful  for  ins0tu0onal  archiving    

– Best  for  record-­‐keeping  purposes  -­‐  when  challenged  in  court  about  content  on  web  site  

– Can  be  used  to  ensure  URL  persistence  eg  when  site  has  a  make-­‐over  –  can  intercept  404s      

– No  ‘gaps’  c.f.  crawl  approach  –  every  change  in  accessed  content  is  archived  

– However  requires  code  snippet  to  be  installed  on  web  server  

– Open  source  sonware  being  developed  by  Los  Alamos  Labs  

Innovation is increasingly driven from Large scale Data Analysis

Need fast iteration to understand the right questions to ask More minds able to contribute = more value (perceived and real) placed on the importance of the data Increased demand for/value of the data = more funding to support it Need to surface the Information amongst all that data…

Web Data Mining & Analysis – What is it? Why Do It?

Platform & Toolkit: Overview

•  Software – Apache Hadoop – Apache Pig

•  Data/File format – WARC – CDX – WAT (new!)

Apache Hadoop

•  HDFS – Distributed storage – Durable, default 3x replication – Scalable: Yahoo! 60+PB HDFS

•  MapReduce – Distributed computation – You write Java functions – Hadoop distributes work across cluster – Tolerates failures

File formats and data: WARC

File formats and data: CDX

•  Index used to browse WARC-based archive •  Space-delimited text file •  Only essential the essential metadata needed

by Wayback – URL – Content Digest – Capture Timestamp – Content-Type – HTTP response code – etc.

File formats and data: WAT

•  Yet Another Metadata Format! ☺ ☹ •  Not preservation format •  Data exchange and analysis •  Less than full WARC, more than CDX •  Essential metadata for many types of analysis •  Avoids barriers to data exchange: copyright,

privacy •  Work-in-progress: we want your feedback

File formats and data: WAT •  WAT is WARC ☺

– WAT records are WARC metadata records

– WARC-Refers-To header identifies original WARC record

•  WAT payload is JSON – Compact – Hierarchical – Supported by every

programming environ

File formats & data: •  CDX: 53 MB •  WAT: 443 MB •  WARC: 8,651 MB

Some  References  

•  h^p://en.wikipedia.org/wiki/Web_archiving  •  h^p://netpreserve.org/about/archiveList.php  •  Web  Archives:  The  Future(s)  -­‐  h^p://www.netpreserve.org/publica0ons/2011_06_IIPC_WebArchives-­‐TheFutures.pdf  

•  h^p://matkelly.com/warcreate/  •  Common  Crawl:  h^p://commoncrawl.org/data/accessing-­‐the-­‐data/  

Contacts  •  Webarchive  @  nla.gov.au  •  Secretariat  @  internetmemory.org  •  Queries  about  the  internet  archive  web  archive  h^p://iawebarchiving.wordpress.com/  

•  Queries  about  Archive-­‐It  service  h^p://www.archive-­‐it.org/contact-­‐us  

momodei  @  nla.gov.au  (un0l  31  Aug  2012  )  or  monica.omodei  @  gmail.com