ruest and milligan - the great warc adventure
DESCRIPTION
The presentation at Nick Ruest (York University) and I gave at the Association of Canadian Archivists' annual meeting in Victoria, BC on 26 June 2014.TRANSCRIPT
![Page 1: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/1.jpg)
The Great WARC Adventure
Nick Ruest (York University)Ian Milligan (University of Waterloo)
![Page 2: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/2.jpg)
Today’s Talk
* A brief historical overview of web archiving* How to capture and preserve* Discoverability and usability (with open source software)* Interplay of the archivist and the historian* Piecing the story together* Internet Archive isn’t the only way
![Page 3: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/3.jpg)
Historical overview of web archiving
![Page 4: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/4.jpg)
Going back to 1995/1996
We do not know today what Mozart sounded like on the keyboard ... What will future generations know of our history? ... But digital technology seemed to come to the rescue, allowing indefinite storage without loss. Now we find that digital information too, has its dark side. (Michael Lesk, 1995)
![Page 5: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/5.jpg)
![Page 6: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/6.jpg)
![Page 7: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/7.jpg)
The Internet Archive
![Page 8: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/8.jpg)
But that’s not all...
![Page 9: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/9.jpg)
and… [http://freedaleaskey.plggta.org/]
![Page 10: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/10.jpg)
How to capture and preserve
![Page 11: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/11.jpg)
![Page 12: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/12.jpg)
![Page 13: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/13.jpg)
https://digital.library.yorku.ca/yul-92640/biafra
WARC?
https://digital.library.yorku.ca/yul-92640/biafra
![Page 14: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/14.jpg)
PROVENANCE!
![Page 15: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/15.jpg)
![Page 16: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/16.jpg)
![Page 17: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/17.jpg)
![Page 18: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/18.jpg)
wget --mirror --page-requisites --warc-file=THIS_AWESOME_SITE http:
//thisawesomesite.ca
Light weight
![Page 19: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/19.jpg)
Heritrix
Industrial strength
![Page 20: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/20.jpg)
using open source software
Discoverable and usable
![Page 21: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/21.jpg)
![Page 22: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/22.jpg)
$$$
![Page 23: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/23.jpg)
Can we do this with Open Source software?
![Page 24: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/24.jpg)
DrupalFedora CommonsIslandoraHeritrixwkhtmltopdfwkhtmltoimage
Use the tools you know
![Page 25: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/25.jpg)
Can we talk OAIS?
![Page 26: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/26.jpg)
HeritrixWgetwkhtmltopdfwkhtmltoimage
...and Bash!
SIP - Submission Information Package
![Page 27: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/27.jpg)
DrupalIslandoraFedora CommonsWeb ARChive SPChecksumChecksum checkerPremisFITS
AIP - Archival Information Package
![Page 28: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/28.jpg)
DrupalIslandoraFedora CommonsWeb ARChive SP
DIP - Dissemination Information Package
![Page 29: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/29.jpg)
Interplay of the archivist and historian
![Page 30: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/30.jpg)
An Ideal Case Study?
![Page 31: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/31.jpg)
A Historian in the Archive
* WARC-Tools (https://code.google.com/p/warc-tools/)* By Date (awesome!)
![Page 32: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/32.jpg)
A Historian in the Archive
* Don’t read the comments* Disqus
![Page 33: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/33.jpg)
A Historian in the Archive
![Page 34: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/34.jpg)
Learning from Word Frequency
![Page 35: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/35.jpg)
![Page 36: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/36.jpg)
A Historian in the Archive (distant reading)
![Page 37: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/37.jpg)
A Historian in the Archive (distant reading)
![Page 38: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/38.jpg)
A Historian in the Archive (distant reading)
![Page 39: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/39.jpg)
Keywords = Gotta Know What You’re Looking For
![Page 40: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/40.jpg)
![Page 41: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/41.jpg)
A Historian in the Archive (distant reading) - search ‘edwin mellen’
![Page 42: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/42.jpg)
A Historian in the Archive (distant reading) - search ‘librarians’
![Page 43: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/43.jpg)
A Historian in the Archive (distant reading) - search ‘dale askey’
![Page 44: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/44.jpg)
Problem: still need to know what you’re looking
for!
![Page 45: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/45.jpg)
A Historian in the Archive (distant reading)
![Page 46: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/46.jpg)
A Historian in the Archive (distant reading)
![Page 47: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/47.jpg)
Helps to piece the story together from massive
web archives
![Page 48: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/48.jpg)
Internet Archive isn’t the only way!
![Page 49: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/49.jpg)
…but they created the Web Archiving Lifecycle
![Page 50: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/50.jpg)
![Page 51: Ruest and Milligan - The Great WARC Adventure](https://reader034.vdocuments.mx/reader034/viewer/2022042518/559bcabc1a28ab64558b4637/html5/thumbnails/51.jpg)