dcc endeavour-2006

a centre of expertise in data curation and preservation

Funded by:This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 2.5 UK: Scotland License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/2.5/scotland/ ; or, (b) send a letter to Creative Commons, 543 Howard Street, 5th Floor, San Francisco, California, 94105, USA.

Experience is a hard teacher…

Curation and the Digital Record

Chris Rusbridge

Endeavor EndUser 2006



"Experience is a hard teacher because she gives the test first,

the lesson afterwards”• Vernon Sanders Law, ex baseball player

• (Or perhaps, in the case of digital preservation, the test occurs long after you are dead?)



Contents

• Curation• Sustainability• Data resources• Preservation & curation issues• OAIS Review



Curation

• Data increasingly important as evidence• Experimental verifiability (the basis of science)• Unrepeatable observations & experiments

(particularly environmental in broadest sense)• Legal, compliance & transactions• Cultural resources

• For evidential value, data must be curated



Curation

• “Maintaining and adding value to a trusted body of digital information for current and future use”



Lynch remarks

• Closing the 2005 Curation Conference• 3 views of digital curation

• Collection as a living thing• Whole life process, evolving object(s)• Finite process, handover to preservation



•This is what you do!This is what you do!



Sustainability and exit strategy

• Most critical resource for curation: present and future money supply!

• Plan for the long term, but have a succession plan

• Sustained approach not project mentality



Sustainability and exit strategy

• Most critical resource for curation: present and future money supply!

• Plan for the long term, but have a succession plan

• Sustained approach not project mentality

•This is what you do!This is what you do!



Some illustrations: UK census• 1881 census (UKDA)

• Hand-written individual return forms: data conversion issue (reference form available): digitisation and access issues

• 1961 census (TNA/NDAD)• First using computers to analyse (first major UK-wide

computer project?); individual returns closed until 2062: data preservation issue!!!

• 2001 census (ONS/CDU) • Data corrections and adjustments: curation issue



Curation of emailsLots of metadata and context (RFC 822)Often highly distributedSplit conversationsUnknown numbers of copiesPersonal choice of clients

• Legal requirements!• Controlled filing and controlled deletion

needed…



Online Public Access Catalogues

• Long term, curated databases• Often high quality (not always)• Well known interchange standards (MARC),

classification standards (several), name authorities…

• Still significant problems combining sources



… The database provides indexing of 527 key international English-language business periodicals including Business Week, Forbes, The Wall Street Journal, The New York Times and more. Also included are product reviews, interviews, biographical sketches, corporate profiles, reports of associations, societies and conferences. Broad areas of coverage include accounting, acquisitions and mergers, advertising, banking, chemicals, …


Pre 2000

Post 2000

© Chris Rusbridge

•Peter Buneman





© Chris Rusbridge

•Peter Buneman

•Storage–Single-source–Volatile–Centralised–Internal DBMS format

•No standards for citation•No historical record•Mind-boggling legal issues

• Storage–Redundant, Distributed–Persistent–Readable

• Clear standards for citation• Historical record (old data is useful)• Well understood ownership/IP


Endeavor EndUser 2006SDSS (Visual)

TWOMASS (Infrared)

Slide from Rajendra Bose


Endeavor EndUser 2006 Slide from Rajendra Bose



Example…• National Virtual Observatory

• Johns Hopkins press release: “Scientists working to create the NVO, an online portal for astronomical research unifying dozens of large astronomical databases, confirmed discovery of [a] new brown dwarf recently. The star emerged from a computerized search of information on millions of astronomical objects in two separate astronomical databases. Thanks to an NVO prototype, that search, formerly an endeavor requiring weeks or months of human attention, took approximately two minutes.”



Context

• Data meaningless without context• Linkage• Metadata of many kinds• Workflow!

• Provenance • Computational lineage • Authenticity



Access and re-use

• Ethics and rights control access• Weak in expressing this long-term

• Collaboration tools• Annotation, discussion, review• Re-use leading to change and development

• “Publication”• Not just in “print”• Underlying data should be “published”, too

• Citation…



Preservation & curation• Use preserves• Money preserves• Redundancy good, monoculture bad?

• LOCKSS-type & other approaches…

• Bits are fragile and robust• Don’t rely on portable media• Look after them well

• Technology changes…• How fast? What impact?

• Metadata matters! (Know what you’ve got)



Formats, migration, significant properties…

• “We MUST preserve the look and feel!”

• Well…• Think about a book like “Kenilworth” by Walter

Scott• Think about the BBC Domesday emulation

• You may be better with a preserved “desiccated” version… than nothing at all!



The Project Gutenberg EBook of Kenilworth, by Sir Walter Scott

This eBook is for the use of anyone anywhere at no cost and withalmost no restrictions whatsoever. You may copy it, give it away orre-use it under the terms of the Project Gutenberg License includedwith this eBook or online at www.gutenberg.org

Title: KenilworthAuthor: Sir Walter ScottRelease Date: February 21, 2006 [EBook #1606]Language: EnglishCharacter set encoding: ASCII

*** START OF THIS PROJECT GUTENBERG EBOOK KENILWORTH ***Produced by An Anonymous Volunteer and David Widger

KENILWORTH.

by Sir Walter Scott, Bart.

INTRODUCTION

A certain degree of success, real or supposed, in the delineation ofQueen Mary, naturally induced the author to attempt something similarrespecting "her sister and her foe," the celebrated Elizabeth. Hewill not, however, pretend to have approached the task with the samefeelings; for the candid Robertson himself confesses having felt theprejudices with which a Scottishman is tempted to regard the subject;…



QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

•But there •ARE limits!



Preservation is not cheap

• But it’s not expensive…




• But it’s not expensive…• Compared with the alternative!




• But it’s not expensive…• Compared with the alternative!

•Postcard •Sent to me•anonymously



Curation: whose job is it?

• Yours!• With your archivists• And your Records Managers• And your scientists and scholars…



Preservation & curation

• We can’t do it alone• Collective responsibility

• We can’t rely on anyone else• Institutional responsibility



It’s about time…• From the very short

• Good management (don’t under-estimate but don’t over-estimate)

• Through the medium term• Curation: use it or lose it• Gather ye metadata while ye may!• Preservation relay

• To the very long term• High commitment, high cost, high risk• Harder to do en masse



Supplier role?

• Work together with libraries…• Multi-supplier, Multi-platform• Open source mix• The library is not simple any more

• Library 2.0?• Power of crowds, economy of attention,

generation X…• Wikicat?



QuickTime™ and aTIFF (LZW) decompressor




Supplier role?• Work together with libraries…

• Multi-supplier, Multi-platform• Open source mix• The library is not simple any more

• Library 2.0?• Power of crowds, economy of attention, generation X…• Wikicat?

• Web 2.0?• Mix, mashup• What you see is… not there?



BEWARE WEB 2.0!!!

QuickTime™ and aTIFF (Uncompressed) decompressor




OAIS• “Announcement of a Comment Period for the Five

Year Review of the Reference Model for an Open Archival Information System (OAIS) Standard”• “… must be reviewed every five years and a determination

made to reaffirm, modify, or withdraw the existing standard.”• “…any revision must remain backward compatible with

regard to major terminology and concepts.”• “… we do not plan to expand the general level of detail”• “… reduce ambiguities and fill in any missing or weak

concepts”

• Make suggestions and express interest until 30/10/06• [email protected]



To close…

• Your library is currently taking the curation test…

• Your children will learn the answer!• But

dcc endeavour-2006

Technology

future money

wall street

sir walter

preservation

product reviews

york times

long term

preservation