sound directions digital preservation and access for global audio heritage indiana university...

Sound Directions

Digital Preservation and Access

for Global Audio Heritage

Indiana UniversityHarvard University

Archives of Traditional MusicArchives of Traditional MusicIndiana UniversityIndiana University

Established 1948 (or 1954?)Established 1948 (or 1954?)110,000 recordings110,000 recordings1890s to present1890s to presentField—30%Field—30%World music traditionsWorld music traditionsEndangered/extinct world languagesEndangered/extinct world languages

Collaboration between Harvard University and Collaboration between Harvard University and Indiana UniversityIndiana UniversityPhase 1 an R&D project funded by NEHPhase 1 an R&D project funded by NEHFocus on preservationFocus on preservation

Sound DirectionsSound DirectionsDigital Preservation and Access Digital Preservation and Access

for Global Audio Heritagefor Global Audio Heritage



Project PartnersProject Partners

Archives of Traditional Music, Indiana UniversityArchives of Traditional Music, Indiana UniversityArchive of World Music, Harvard UniversityArchive of World Music, Harvard UniversityHarvard College Library Audio Preservation Harvard College Library Audio Preservation ServicesServicesDigital Library Program, Indiana UniversityDigital Library Program, Indiana UniversityOffice for Information Systems, Harvard UniversityOffice for Information Systems, Harvard University



ObjectivesObjectives

Research best practices in areas without Research best practices in areas without standards or best practicesstandards or best practicesDevelop best practices to meet existing Develop best practices to meet existing and emerging standardsand emerging standardsTest existing and emerging standards/best Test existing and emerging standards/best practices with a real world projectpractices with a real world project



ResultsResults

Publication—Publication—Sound Directions: Best Practices for Audio Sound Directions: Best Practices for Audio PreservationPreservationSoftware toolsSoftware toolsDevelopment of audio preservation Development of audio preservation systemssystemsPreservation of field collectionsPreservation of field collections

www.dlib.indiana.edu/projects/sounddirections

Audio Preservation SystemsAudio Preservation Systems

Standards and Best PracticesStandards and Best Practices

Ensure QualityEnsure QualityProvide Philosophical/Ethical FoundationProvide Philosophical/Ethical FoundationEncourage SustainabilityEncourage SustainabilityFoster InteroperabilityFoster InteroperabilityProvide a Migration PathProvide a Migration Path


Standards and Best PracticesStandards and Best Practices

IASA-TC 03 and TC 04IASA-TC 03 and TC 04Broadcast Wave FormatBroadcast Wave FormatAES31-3AES31-3AES-X98-B and CAES-X98-B and CMETSMETS


What tasks or functions are required What tasks or functions are required for successful audio preservation?for successful audio preservation?

Preservation operations/activities as systemsPreservation operations/activities as systemsGeneral Systems TheoryGeneral Systems Theory


Common sense definition of system:Common sense definition of system:

Set of interacting units or elementsSet of interacting units or elementsForms an integrated wholeForms an integrated wholePerforms a functionPerforms a function


A few basic principles:A few basic principles:

Each element/part affects the wholeEach element/part affects the wholeWhole is greater than sum of partsWhole is greater than sum of partsInputs and outputsInputs and outputs

IU Audio Preservation SystemIU Audio Preservation System

What should we preserve?What should we preserve?

Selection for PreservationSelection for Preservation

Analysis of research valueAnalysis of research valueEvaluation of preservation condition and riskEvaluation of preservation condition and risk

Data Collection/Analysis

Research ValueScore

Condition/Risk(FACET)

Score

Combined SelectionScore

Collection Ranking

Curatorial Review

Selection forPreservation


Field Audio Collection Evaluation ToolField Audio Collection Evaluation Tool(FACET)(FACET)

Point-based, collection levelPoint-based, collection levelAnalyzes data on condition of field formatsAnalyzes data on condition of field formatsReturns a risk assessment scoreReturns a risk assessment scoreFormats documentFormats documentProcedures documentProcedures documentAppendicesAppendices


Where should preservation work be Where should preservation work be done?done?

In-house or outsource?In-house or outsource?Issues: studio space, technical expertise, Issues: studio space, technical expertise, amount of work, future location of expertiseamount of work, future location of expertiseCritical listening spacesCritical listening spacesDevelopment of preservation studioDevelopment of preservation studio


Who should do preservation work?Who should do preservation work?

Audio engineerAudio engineerImportance of analog playback stageImportance of analog playback stageProject assistantProject assistantProgrammerProgrammerDigital librariansDigital librarians

Audio Engineer

Preservation TransferPreservation Master FilesTechnical MD Collection

ChecksumsBWAV MD

ADL’sSignal Processing

Project Assistant

Content DivisionProduction Masters

QCADL’s

Workflow ManagementCollection SetupIngestion Process

ProgrammerSoftware/ScriptDevelopment

Digital Library Program

Preservation Repository ServicesDeliverables

Access System


What is the target preservation format?What is the target preservation format?

Digital fileDigital fileBroadcast Wave Format (BWF or BWAV)Broadcast Wave Format (BWF or BWAV)

Preservation involves a long-term responsibility to the Preservation involves a long-term responsibility to the digital filedigital file


Broadcast Wave FormatBroadcast Wave Format

Audio file format based on .wav filesAudio file format based on .wav filesEBU 1996 for the exchange of filesEBU 1996 for the exchange of files““Open source” Open source” Recommended by IASA, AES, NARAS, Sound Recommended by IASA, AES, NARAS, Sound Directions for preservationDirections for preservation““Chunk” for metadata residing with the fileChunk” for metadata residing with the fileTime stampTime stamp


Broadcast Wave FormatBroadcast Wave Format

Metadata elements include:Metadata elements include:

Description of the sound sequenceDescription of the sound sequenceName of the originatorName of the originatorDate/timeDate/timeCoding history (signal chain components)Coding history (signal chain components)Format independent, sample accurate time Format independent, sample accurate time stampstamp““Catastrophic” metadataCatastrophic” metadata


How should we define the filesHow should we define the files we create?we create?

What is in them?What is in them?How are they created?How are they created?What do they represent?What do they represent?


Best Practice DocumentsBest Practice Documents

Preservation (Archival) Master Files:Preservation (Archival) Master Files:

UnmodifiedUnmodifiedNo subjective alterations or improvementsNo subjective alterations or improvementsPreserve history, not re-write itPreserve history, not re-write itAs true to the original source as possibleAs true to the original source as possible


Preservation (Archival) Master Files Preservation (Archival) Master Files at the ATM:at the ATM:

Complete, unaltered stream from playback Complete, unaltered stream from playback machinemachineCarrier of raw material from transferCarrier of raw material from transferNo editing, signal processing, data reduction, No editing, signal processing, data reduction, gain manipulation, announcements (slates)gain manipulation, announcements (slates)24 bit, 96 kHz24 bit, 96 kHz


Production Master Files:Production Master Files:

Used to generate further derivativesUsed to generate further derivativesOptimized—signal processing, editingOptimized—signal processing, editingProduced for all content and retainedProduced for all content and retainedContent of one FaceContent of one FaceUsually 24/96Usually 24/96

IU Audio Preservation SystemsIU Audio Preservation Systems

Best PracticesBest Practices

Define purpose of every digital fileDefine purpose of every digital fileWritten guidelines on characteristics of filesWritten guidelines on characteristics of filesWritten guidelines on “technical” and content Written guidelines on “technical” and content editseditsMaintain common reference timelineMaintain common reference timeline


How do we ensure authenticity over How do we ensure authenticity over time?time?

Data integrity checkingData integrity checking““Checksums”Checksums”MD5 hash or algorithmMD5 hash or algorithm


MD5 algorithmMD5 algorithm

All files with enduring valueAll files with enduring valueAs soon as possibleAs soon as possibleCritical metadata stored in database and in Critical metadata stored in database and in preservation packagepreservation packageVerify before trustingVerify before trusting


How do we render the preserved How do we render the preserved content understandable and content understandable and

manageable?manageable?

Descriptive MetadataDescriptive MetadataAdministrative—Technical MetadataAdministrative—Technical MetadataAdministrative—Digital ProvenanceAdministrative—Digital ProvenanceAdministrative—Rights ManagementAdministrative—Rights ManagementStructural MetadataStructural Metadata


Audio Technical Metadata CollectorAudio Technical Metadata Collector

(ATMC(ATMC))

Enter/edit technical and structural metadataEnter/edit technical and structural metadataAudio object and process history metadataAudio object and process history metadataEnter/edit audio object evaluationsEnter/edit audio object evaluationsParse files to collect metadataParse files to collect metadata


Audio Technical Metadata CollectorAudio Technical Metadata Collector

(ATMC(ATMC))

Generate MD5Generate MD5Relationships diagramsRelationships diagramsAES-compliant XML exportAES-compliant XML exportJava Swing applicationJava Swing application


How do we assure and control the How do we assure and control the quality of the work?quality of the work?

Quality control and quality assuranceQuality control and quality assuranceQC at ATM: aural, visual, software toolsQC at ATM: aural, visual, software toolsCollection setup—preliminary transfersCollection setup—preliminary transfersRole of permanent staffRole of permanent staffQA at ATMQA at ATM


How do we store data after capture?How do we store data after capture?

Local, interim storageLocal, interim storageBackup copies at each stageBackup copies at each stageATM NASATM NASAdditional redundant copyAdditional redundant copy

Migration decision

Workflow management

Workflow management / scheduling

Cleaning or physical restoration as needed

System / ProjectPlanning & DevelopmentFundingPersonnel / VendorEquipment

Software ToolsCreation / maintenance of software and scripts

Selection for PreservationAssess research valueEvaluate conditionConsider political, technical, and other issuesEstablish priorities

Digitization Analog playbackA/D conversionCreation of Preservation Master FilesLocal filenames

Digitization Technical metadata Structural metadata ChecksumsQuality controlLocal storage solution

Post-Transfer ProcessingQuality controlGeneration of derivativesMarking areas of interest in filesSignal processing (if appropriate)

Preliminary Work / Pilot Project

Exploratory transfers and metadata collectionQuality controlReassessment of digitization plan

Collection SetupGather and assess documentationEvaluate collection needs / condition Assess cataloging / descriptive metadata issuesDevelop digitization planAssess and calibrate equipment

Ingestion into / Copy to Long-Term Storage

SolutionPreservation packages

Periodic EvaluationData integrity checkingFormat obsolescence analysis

MigrationNew carrierNew format

Director

Project DevelopmentSelection for Preservation

Archivist

SelectionPreview CollectionsQC Documentation

Librarian

Cataloging Issues

Associate Director

Project ManagementSelection—Format IssuesScheduling Coordination

QCR&D

Audio Engineer

Preservation TransferPreservation Master FilesTechnical MD Collection

ChecksumsBWAV MD

ADL’sSignal Processing

Project Assistant

Content DivisionProduction Masters

QCADL’s

Workflow ManagementCollection SetupIngestion Process

ProgrammerSoftware/ScriptDevelopment

Digital Library Program

Preservation Repository ServicesDeliverables

Access System



Project FutureProject Future

““Preservation” Phase funded by NEHPreservation” Phase funded by NEHIncrease throughputIncrease throughputSimultaneous transferSimultaneous transferIndiana automationIndiana automationRelease ATMC, Audio Object Manager, APXERelease ATMC, Audio Object Manager, APXEDevelop access system for field collectionsDevelop access system for field collections

Sound Directions

Digital Preservation and Access

for Global Audio Heritage

Indiana UniversityHarvard University

www.dlib.indiana.edu/projects/sounddirections

[email protected]