Archives Enabled: Case Studies on the Digitisation and Subsequent Usage of Audiovisual Collections

Download Archives Enabled: Case Studies on the Digitisation and Subsequent Usage of Audiovisual Collections

Post on 12-Apr-2017

444 views

Category:

Technology

0 download

TRANSCRIPT

PowerPoint Presentation

ARCHIVES ENABLEDCase Studies on the Digitisation and Subsequent Usage of Audiovisual Collections

1

DAMsmart Established in 2007.

Production Facility in Canberra.

Digitisation of Film, Video and Audio.

Tailored Solutions Based on Requirements.

2

Following are some of the more recent clients that we have worked for digitising legacy audiovisual collections, and conforming digital audiovisual collections for preservation. Nelson Mandela, can share sites.3

RICH END-USER EXPERIENCEMetadata to enable deep search of an oral history archive.

CUSTOMER EFFICIENCYDelivering a transmission ready broadcast archive.

CONSERVATION = RECOVERY The importance of media preparation in the digitisation process.

Todays presentation will portray the ability for DAMsmart to be adaptable to very different requirements, with quality outcomes the consistent across all 3 examples.

The first example will detail the process to enable full searchability across a significant Australian audiovisual oral history collection.

The second case study will discuss the design and fixed workflows to create very broadcast specific outputs from a videotape collection

And the third example will look at some very specialised conservation treatment carried out to allow for great access methods by a national archive.4

Case Study 1:

RICH END-USER EXPERIENCE

Australians At War Film Archive: searchable collection.

5

Australians at War Film Archive

2000 PEOPLE

www.DAMsmart.com.au

In 2001 The Department of Veterans Affairs undertook a large scale oral history project to record over 2000 people as a detailed personal record of all major conflicts and peace keeping missions Australia has been involved in. A broad cross section of society was considered, from war veterans, women on the home front, through to pilots and nurses. .

6

Australians at War Film Archive

0199-1-20030497-1-2003

1145-1-002

1182-1-20031352-1-2004

1414-1-2004

The production was huge, The interviewing process for the archive was conducted by two person teams in every state and territory across Australia. Each team interviewed in 'tours' of eight weeks, filming 300 hours of material per team, per tour.

DVCAM Tapes were sent back from the field to be duplicated and transcribed interviews into text with 30 second timecode points relating to the original DVCAM tapes. 7

Australians at War Film Archive

www.DAMsmart.com.au

Fast Forward to 2013 an a looming challenge was becoming apparent, DVCam tapes were considered vulnerable and obsolete. Therefore digitisation of the collection was decided upon, and through this process it also provided an excellent access opportunity; to make use of the great transcriptions and relate it directly with the moving image.8

The following diagram outlines the digitisation signal path for this particular project. Post digitisation, DAMsmart was required to import all video files and metadata into a Media Asset Management system which included transcripts for viewing. As we had already archived high resolution files to LTO, the next task was to create the collection access layer that was fully searchable, and highly secure, with users and groups with unique privileges, as some interviews are embargoed for various reasons.

This is a high level overview of the process, and today I would like to focus on one requirement of interest which is to link the existing transcript to the now digitised collection to allow for a much more rich user experience.9

The original transcripts were in XML form and there was a reasonable amount of work that was required to make them compatible with the new Media Asset Management system.10

Australians at War Film ArchiveAn app was needed to test for legal timecode using many variables.

Eg: 01:00:01;30 000:01:30

11,000 hrs of content= 1,320,000 30 second timecode points!

So through some initial testing it was discovered that some of the source transcriptions timecode had non compatible or illegal characters (which is fairly understandable considering the transcripts were hand typed, and the size of the project.) So an application was created to parse all timecode points in the transcription and correct where needed.11

One Interview Containing Multiple TapesTape 1

01:00:32:00

Q: Can you give us a summary of your life?

A: I was born on the 19th of August, 1917 in Windsor. My father was away at the war and I didn't see him until I was two. I grew up, on his return - in 1919, we

01:01:00:00 moved to Caulfield where I lived until the Second World War. I went to Caulfield North State School. And the local church, and had a pleasant childhood although with memories of war. In 1937,

01:01:30:00 I'll go back a bit, I attended Caulfield North Central School, became dux there and then went to Wesley like my father on a scholarship. I changed careers - choices - while at Wesley, and decided to become an actuary. I joined the militia in 1937 and I was a sergeant by the time the war broke out.

01:02:00:00 I was in the 2nd Medium Brigade and was seconded to other regiments for particular reasons. The 15th Field Artillery Regiment. Later I returned to service with the Mediums

01:02:30:00 but in 19 - I was commissioned in 1940, too, I was transferred to the 2nd Field Artillery Training Regiment in Puckapunyal, later moving to Greta. And I tried to get into the air force at one stage, but the war took an unusual turn. I

01:03:00:00 was - people were competing, well I was competing for where to serve. Although I tried to get into the air force I finally was transferred to Melbourne to become a ballistician in the last two years of the war, helping to make range stables for the short 25 pounder gun. During the war, in September '41, I married and had

01:03:30:00 a wonderful marriage lasting nearly 54 years. After the war I resumed my actuarial studies and qualified; I became manager of the Collective Insurance. At the age of 60 I was

Each tape started with an incremental timecode hour depending what tape it was in the order of recording. As an example, Tape 1 starts at 01:00:00:00, Tape 2 as 02:00:00:00, and so on. Another application was required that used the change in hour as a delimiter and split the XML to match the video file.

12

Australians at War Film Archive01:03:00:00 it leads back to the, the fogey, old Masters I had at Melbourne Grammar School.

Q: Can I ask how your mother came to be living in Australia and where she came from in England?

A: I dont know where she came from but her mother was an English widow and she had four daughters and she emigrated to Australia with these four daughters. And my mother was the eldest of the daughters. And she was a powerful old lady my grandmother. 01:03:00:00

Another problem was with characters that were throwing back issues when importing. Even though XML is human and machine readable, that doesnt necessarily mean the data within the XML is fully compatible. As an example, we had perfectly legitimate carriage returns that looked correct when reading the transcript, however would upset the import process and only import a portion of the transcript.

13

Australians at War Film Archive Transcript

01:03:00:00 Marker 6Eventit leads back to the, the fogey, old Masters I had at Melbourne Grammar School. Q: Can I ask how your mother came to be living in Australia and where she came from in England? A: I dont know where she came from but her mother was an English widow and she had four daughters and she emigrated to Australia with these four daughters. And my mother was the eldest of the daughters. And she was a powerful old lady my grandmother

To get around this, DAMsmart created another app to extract all words and recreate all words with spaces only.

14

Australians at War Film ArchiveImport into catdv

Once the XML were cleansed and updated we went through the import procedure to make the proxy files viewable in CatDV, and match the transcript to the video making it searchable to the word.

15

Along with the DAMsmart Digitisation and asset management creation, A new website that was released earlier this year allow for the same detailed searchability as the MAM system, with video or audio only representations available.

There are technologies such as phonetic search that have high accuracy that would be considered for collections that dont have existing transcripts.As with the Australians at War Film Archive, the true richness of any oral history collection is having a searchable transcription. Searchability applies to any audiovisual recording that is valued in a collection that warrants access .

Following is a clip that was produced from an interview in the archive created for an awareness campaign that was screened on Qantas flights.16

17

Case Study 2: CUSTOMER EFFICIENCYRadio Television Hong Kong: The Broadcast Ready Archive

18

OBJECTIVES

Radio Television Hong Kong needed to meet programming requirements.Television Archive locked in obsolete formats.Ensure the content is secured for future generations.DAMsmart delivered over 600,000 assets.

2014 marked a new era for RTHK, with the introduction of its own digital terrestrial television channels.

This shift in operations saw RTHK look to leverage its extensive production archive to meet its programing needs.

To achieve this, RTHK went to public tender with the following objectives in mind:

significant increases in production efficiencies and an increase in content needed for digital broadcast;

reduced risk by removing storage and access overheads inherent to physical videotape,

and, importantly, ensure the content is secured for future generations. DAMsmart was awarded the digitisation contract in May 2014.

By the end of the project, we delivered over 600,000 files including video and quality control reports. With an output of such high numbers, process and workflows had to be locked down to ensure a high quality and accurate outcome.19

21,000 hours of archive materialAcross 40,000 Digital Betacams

RTHK identified a priority collection, which consisted of 21,000 hours of content across 40,000 individual Digital Betacam cassettes.

20

It all starts with the media: physical tape.

Digitisation of the mouldy tapes could only be achieved because of the intensive conservation, repair and preparation work carried out on the cassettes.DAMsmart commenced digitisation of the clean tapes and running in parallel we commenced cleaning the 10,000 mouldy Digital Betacam cassettes.I wont go into too much detail here regarding the cleaning process as I will discuss this in the next case study.

21

SEGMENTATIONDAMsmart REGISTRATIONCLEANINGDIGITISATIONQCARCHIVINGRTHK DIGITISATION Workflow

Barcodes were placed on the tape at point of registration to enable robotic scanning and control.

Cassettes were loaded into our robotic library in batches of 60. A predetermined project templated was automatically loaded into 5 encoders. This contains media compression settings, SD to HD up conversion settings, signal analysis, and checksum creation.

High quality PAL to 1080 25i SD to HD conversion was hardware based.

The tape was digitised as a whole tape, with 1 master file. There is further reprocessing , Qcing and file integrity checks through out the whole process, however today we are focusing on the segmentation process of the master file.

22

EPISODE 1EPISODE 2MASTER FILE

SEGMENTED FILESEGMENTED FILE

Master files could have 2 to 20 individual programs on the one tape.

To be truly transmission ready, extraction of segments lifted from the master file was required.

Through initial testing, DAMsmart selected a command line application originally designed for lossless insertion of advertisements in MPEG2 transport Streams.

This segmentation process was automated, fast and maximised the quality of the video as it was lossless extraction, compared with a compute intensive and lossy transcoding process.

This enabled the archive to be active and online ready for on-air requirements without the need for further processing. 23

180792010131130211310131130211316484601SD00001807920101030000064846ACQ_20040131_NO18079202140325041101140325041116484701SD00001807920201030000064847ACQ_20040131_NO180791010131130311320131130311326484801SD00001807910101030000064848ACQ_20040131_NO18079102134624471101134624471116484901SD00001807910201030000064849ACQ_20040131_NO180817010134131411400134131411406485001SD00001808170101030000064850ACQ_20040131_NO180817021401245110501401245110506485101SD00001808170201030000064851ACQ_20040131_NO180818010130130111310130130111316485201SD00001808180101030000064852ACQ_20040131_NO18081802134724481101134724481116485301SD00001808180201030000064853ACQ_20040131_NO180725010131125511240131125511246485401SD00001807250101030000064854ACQ_20040131_NO18072502134124471106134124471166485501SD00001807250201030000064855ACQ_20040131_NO158851010131124711160131124711166485601SD00001588510101030000064856ACQ_20040131_NO158851021333244711141333244711146485701SD00001588510201030000064857ACQ_20040131_NO158839010132124911170132124911176485801SD00001588390101030000064858ACQ_20040131_NO

Clean, accurate data.

This process could only have been automated due to the very high percentage of frame accurate timecode metadata provided by RTHK. We were unsure of this at the begging of the project, so we retained the master file on LTO if there were inaccuracies.

All segmented files were viewed to ensure in and out timecode accuracy.24

RTHK INGEST AND STORAGE

On return of the content RTHK had 3 Sata Docks ingest stations used to copy the data from hard drives and operators QCd files before moving to a shared online archive for broadcast ready playout.25

Delivered and Returned

50 tonnes of videotape.155 x 4 TB hard drives.652 TB x 2 data output.Over 600,000 assets delivered.

Across 8 batches, DAMsmart and our logistics partner delivered and returned 50 tonnes of archive videotape without damage or loss of tape,

And By the end of 2015 delivered a total of 155 4xTB SATA Hard Drives. The total data output for the project was 1.3 PetaBytes.In total over 600,000 digital assets were delivered, including high definition XDCAM mxf files for broadcast, mp4 proxies, Quality Control reports and metadata documentation.So as with the Australians at War oral history project, metadata is critical to fully leverage the all the advantages of having a digital collection.26

Case Study 3:

CONSERVATION = RECOVERY

National Archives of Fiji: recovering a degraded archive.

The transformation started with organisation.

Archives of the Pacific have several very real challenges around audiovisual preservation: including funding, climate and awareness.

Through continued lobbying, involvement in professional associations and promoting interconnected support the National Archives of Fiji with DAMsmart successfully completed restoration and digitisation of 2000 hours of irreplaceable audiovisual heritage which has enabled NAF new methods of enabling access for the people of Fiji.

28

200,000 photographs and negs2000 hours of historical footageDumpedUnorganisedAdvanced State of Decay

Despite numerous attempts to convince the original record owners of the need to implement management plans for the continued accessibility of their AV holdings, nothing was done. Then these materials were brought to the archives for temporary storage while the owners relocated premises.Temporary eventually became permanent and the Ministry took action to save these materials.

29

Consultation and Partnership

Through consultation with SEAPAVAA, and advice from PARBICA, ICA and UNESCOs Memory of the World programme, The National Archives of Fiji sort out the right partner for audiovisual preservation and digitisation work.DAMsmart was commissioned by the National Archives of Fiji to inspect the ageing audiovisual collection.DAMsmart spent 2 days in Suva examining and counting collection holdings across two locations National Archives of Fiji and the Fiji Film and TV Unit.

In summary, the general condition of the audiovisual collection was very poor, with significant signs of degradation and decomposition across all videotape and film formats. Environmental storage issues in the past, combined with no availability of replay equipment had conspired to make the entire collection inaccessible and un-useable for The National Archives of Fiji.

30

From the assessment it was found that

A very high percentage of sampled U-matic and betacam tapes were suffering from severe mould infestation and hydrolysis.

31

Film inspected by DAMsmart were found to be suffering from advanced decomposition as a result of Vinegar Syndrome. These films became blocked, or the layers of film wound onto the reel stick together and could not be separated without specialist conservation treatments.

32

At risk Slide

Many hours were spent on the conservation.

www.DAMsmart.com.au

a tender was awarded to DAMsmart to treat the items in Canberra, digitise the collection, and create a digital archive based on LTO and LTFS.

Material was packed and shipped to the DAMsmart facility for processing.

Registration and conservation was the first process for the collection.

For cleaning videotape, Specialist anti-static synthetic fibre was used with small amounts of isopropyl alcohol to remove mould and clean the tape. Where tape was previously spliced, had become detached from the leader it was then needed to be respliced with videotape splicing tape. The upmost care was needed during this process, as this technique was obviously quite intensive, yet extremely effective and well worth the procedure. As we had a 98% recovery success rate across all formats.

33

Project Deliverables

Digitisation.Quality Control on video files.Creation and Delivery of a Media Asset Management System.

Once physical restoration of the collection was complete, digitisation , QC possess , import and archive into a turn key LTO LTFS system.

2nd Copy off site.34

Following the successful conclusion of the project in December 2013, NAF formed a partnership with the agency which created the footage to enable the curation and repackaging of the footage into a digestible educational programme called Back in Time which is broadcast on free to air television. This ensures that the public have ready access to their heritage at no cost to themselves nor The National Archives of Fiji. The programme has been very positively received.

35

The footage is now being used by the Ministry of Taukei (Indigenous) Affairs to revitalise dormant cultural practices in the community. Back in Time is about to enter a third season, to run over 40 episodes. The first two seasons ran over 70 episodes.

NAF have also designed innovative access programs for the people of Fiji such as government roadshows and open day initiatives, taking the information to local communities.

36

The National Archives of Fiji now also has a new website that provides access to some of the digitised films and videos.

To quote Opeta Alefaio, Director of the National Archives of Fiji:

This footage has (re)introduced the public of Fiji to cultural artefacts and practices that had passed from memory. As Pacific societies become more "modern" parts of their culture have become dormant. Salvaging the content from the inaccessible and failing physical archive has highlighted this in a very tangible way, and provided a means to (re)discover that knowing a way to reconnect with lost elements of the Fijian identity, which current and future generations must continually have access to.37

RICH END-USER EXPERIENCEMetadata to enable deep search of an oral history archive.

CUSTOMER EFFICIENCYDelivering a transmission ready broadcast archive.

CONSERVATION = RECOVERY The importance of media preparation in the digitisation process.

In conclusion, Id like to reiterate that not 2 projects are the same for DAMsmart, and we tailor our services to get the best out of the collections. Thankyou!

38

andrew@damsmart.com.auwww.DAMsmart.tvTwitter - @DAMsmart_LinkedIn - linkedin.com/company/damsmart-

Recommended

View more >