1

Moving type from past to Moving type from past to present: present: chronicling Australia chronicling Australia

through the digitisation of through the digitisation of newspapers.newspapers.

Cathy Pilgrim – Director, Australian Newspaper Digitisation Program (ANDP)Cathy Pilgrim – Director, Australian Newspaper Digitisation Program (ANDP)

12th Australasian Congress on Genealogy and Heraldry, Kings College, 12th Australasian Congress on Genealogy and Heraldry, Kings College, AucklandAuckland

19 January 2009 19 January 2009

2

Presentation outlinePresentation outline Comments on newspaper publishing in Australia Comments on newspaper publishing in Australia

Australian Newspaper Plan (ANPlan)Australian Newspaper Plan (ANPlan)

Australian Newspaper Digitisation Program (ANDP) Australian Newspaper Digitisation Program (ANDP) objectivesobjectives

ANDP workflow overviewANDP workflow overview

Australian Newspapers beta serviceAustralian Newspapers beta service

Future directions Future directions

3

Front page of the first Front page of the first Australian newspaperAustralian newspaper

The Sydney Gazette and New The Sydney Gazette and New South Wales Advertiser, March 5, South Wales Advertiser, March 5, 18031803

4

Website:Website: www.nla.gov.au/anplanwww.nla.gov.au/anplan

5

History of ANPlanHistory of ANPlan

Established in 1992 as the National Plan for Established in 1992 as the National Plan for Australian Newspapers (NPLAN)Australian Newspapers (NPLAN)

Name changed to Australian Newspaper Plan Name changed to Australian Newspaper Plan (ANPlan) in 2006(ANPlan) in 2006

An initiative of the Consortium of Australian An initiative of the Consortium of Australian State Libraries (CASL) - now know as NSLA State Libraries (CASL) - now know as NSLA (National and State Libraries Australasia) (National and State Libraries Australasia)

Managed by the National Library since 2001Managed by the National Library since 2001

6

Collecting Collecting

Partners are required to:Partners are required to:

Collect hardcopies of all newspapers as Collect hardcopies of all newspapers as publishedpublished

Identify, locate and collect missing titles and Identify, locate and collect missing titles and issues issues

7

PreservationPreservation

Partners are required to:Partners are required to:

Retain one hardcopy of every newspaper titleRetain one hardcopy of every newspaper title

Create an archival standard master Create an archival standard master reproduction and a working copy of every reproduction and a working copy of every titletitle

Provide appropriate housing and Provide appropriate housing and management of all copiesmanagement of all copies

8

AccessAccess

Partners are required to:Partners are required to:

Catalogue all print and microfilm holdings of Catalogue all print and microfilm holdings of newspapers into newspapers into Libraries AustraliaLibraries Australia cataloguecatalogue

Provide easy access pathways to the content Provide easy access pathways to the content of each titleof each title

9

Increase access to Australian newspapers Increase access to Australian newspapers

Build a national service that will provide free Build a national service that will provide free online access from the first Australian online access from the first Australian newspaper published in 1803 through to the newspaper published in 1803 through to the end of 1954 end of 1954

Key Features of the serviceKey Features of the service Online accessOnline access Freely availableFreely available Full text searchableFull text searchable

Program Objectives Program Objectives

10

National National ContentContent

Initial focus on Initial focus on major titles from major titles from each state and each state and territoryterritory

Anticipate that Anticipate that ‘regional’ titles ‘regional’ titles will be contributed will be contributed laterlater

Coverage: Coverage: published between published between 1803 – 19541803 – 1954(out of copyright)(out of copyright)

West Australian

Northern Territory Times

Courier Mail

Advertiser

Sydney Morning Herald

Sydney Gazette

Argus

Mercury

Canberra Times

11

$1 Million Grant from the Vincent Fairfax Family $1 Million Grant from the Vincent Fairfax Family Foundation to digitise Foundation to digitise The Sydney Morning Herald The Sydney Morning Herald to 1954to 1954

12

1. Identify and locate selected newspaper 1. Identify and locate selected newspaper microfilmmicrofilm

13

2. Digitisation2. Digitisation

14

3. Quality Assure digital images3. Quality Assure digital imagesUse 2 widescreen Use 2 widescreen monitors placed monitors placed vertically. Can view vertically. Can view complete page within complete page within context of issue.context of issue.

Add metadata, Add metadata, identify missing and identify missing and duplicate pages duplicate pages within an issue. within an issue.

Prepare batches to Prepare batches to send for OCR send for OCR processing.processing.

15

4. Optical Character Recognition (OCR) of pages and article zoning4. Optical Character Recognition (OCR) of pages and article zoning

16

Article categoriesArticle categories NewsNews

Includes a wide range of subject matter such as Includes a wide range of subject matter such as current affairs, law courts and crime, official current affairs, law courts and crime, official appointments and notices, commerce and appointments and notices, commerce and business news, sporting news and social news, business news, sporting news and social news, etc.etc.

AdvertisingAdvertising

Display advertising and classified advertising Display advertising and classified advertising including items and services for sale, including items and services for sale, employment notices, public and personal employment notices, public and personal notices, etc.notices, etc.

17

Article categoriesArticle categories Family NoticesFamily Notices

Birth, death and marriage notices and Birth, death and marriage notices and related announcements including weddings, related announcements including weddings, anniversaries, in memoriam, bereavement, anniversaries, in memoriam, bereavement, birthdays, congratulations, etc.birthdays, congratulations, etc.

Detailed Lists, Results and GuidesDetailed Lists, Results and GuidesIncludes sporting results, radio and Includes sporting results, radio and television guides, weather forecasts, television guides, weather forecasts, election results, education results and election results, education results and courses, stock market lists, crossword courses, stock market lists, crossword puzzles, word games, etc.puzzles, word games, etc.

18

Processing in Processing in IndiaIndia

19Beta service: www.ndp.beta.nla.gov.au

20Search results

21Page view

22Article View

23Correct electronically translated text

24Add tags to articles

25Tag cloud

26Add note/comment to article

27

Examples of comments:Examples of comments:

The name Rudolph Membergle should The name Rudolph Membergle should read Rudolph Mornbergerread Rudolph Mornberger

Typhus and Colonial Fever outbreak in Typhus and Colonial Fever outbreak in Toowoomba HospitalToowoomba Hospital

The writer would appear to be Corporal The writer would appear to be Corporal John Heaney No 91, 6th Bn, who was John Heaney No 91, 6th Bn, who was wounded in the left leg at Gallipoli on 14 wounded in the left leg at Gallipoli on 14 July 1915July 1915

28Title list

29Title information and browse

30Feedback

31

32

33

Summary needsSummary needs

1.1. Small enhancements to advanced searchSmall enhancements to advanced search2.2. Major enhancements to text correction Major enhancements to text correction

feature and related items (want it easier feature and related items (want it easier to do more)to do more)

3.3. Creation of networking features for user Creation of networking features for user community (help, share, communicate, community (help, share, communicate, work with each other)work with each other)

4.4. Transparency of users and system (open Transparency of users and system (open user profiles, statistics, ranking, about)user profiles, statistics, ranking, about)

34

Functionality and designFunctionality and design

Derivative sizes and zoom technology testingDerivative sizes and zoom technology testing

Search and Browse featuresSearch and Browse features

Display of results and refinement of resultsDisplay of results and refinement of results

User interaction with source (web 2.0)User interaction with source (web 2.0)

Interface designInterface design

35

Prototype wireframePrototype wireframe

36

Prototype service developed and tested with Australian state librariesPrototype service developed and tested with Australian state libraries

37Beta service: www.ndp.beta.nla.gov.au

38www.news.google.com.au/archivesearch

39http://www.nla.gov.au/ndp

40

QuestionsQuestions

Australian Newspapers beta serviceAustralian Newspapers beta service

http://http://ndpbeta.nla.gov.au/ndp/del/homendpbeta.nla.gov.au/ndp/del/home

41

UsersUsers

Registered login is optionalRegistered login is optional

Registered users 1,488Registered users 1,488 Unique visitors 94,000Unique visitors 94,000 78% of users in Australia78% of users in Australia Average time on beta = 18 minsAverage time on beta = 18 mins

42

Search ActivitySearch Activity

2 million keyword searches2 million keyword searches Between 18,500- 28,000 searches Between 18,500- 28,000 searches

per dayper day + Unknown amount of browsing by + Unknown amount of browsing by

date, title, state, category.date, title, state, category.

43

User InteractionUser Interaction 1 million lines of OCR corrected 1 million lines of OCR corrected 60,000 articles corrected60,000 articles corrected 18,000 tags added18,000 tags added 800 comments added800 comments added

Top corrector 60,000 lines in 2000 articlesTop corrector 60,000 lines in 2000 articles Correction 24 hrs a day/7 days a week.Correction 24 hrs a day/7 days a week. Peaks at 3pm and 9pmPeaks at 3pm and 9pm Top Text-Correctors Hall of FameTop Text-Correctors Hall of Fame

44

FeedbackFeedback

Feedback surveyFeedback survey ‘‘Contact Us’ web formContact Us’ web form Observation on user activity and statsObservation on user activity and stats Blogs and ForumsBlogs and Forums Direct e-mail/phoneDirect e-mail/phone Suggestions from ANDP team and Suggestions from ANDP team and

ANPlanANPlan

45

Positive General Positive General CommentsComments

Keyword searching (quick and Keyword searching (quick and effective)effective)

Speed of finding thingsSpeed of finding things Available online from home 24/7 Available online from home 24/7

(saves travel, convenient)(saves travel, convenient) Free (too good to be true)Free (too good to be true) Easier than microfilmEasier than microfilm Good proposed body of content Good proposed body of content Text correction is fantasticText correction is fantastic

46

Summary needsSummary needs

1.1. Small enhancements to advanced searchSmall enhancements to advanced search2.2. Major enhancements to text correction Major enhancements to text correction

feature and related items (want it easier feature and related items (want it easier to do more)to do more)

3.3. Creation of networking features for user Creation of networking features for user community (help, share, communicate, community (help, share, communicate, work with each other)work with each other)

4.4. Transparency of users and system (open Transparency of users and system (open user profiles, statistics, ranking, about)user profiles, statistics, ranking, about)

47

Infrastructure and Infrastructure and StorageStorage

Online Storage – 70 TB:Online Storage – 70 TB: Working space for images in processing 40TB for 1 million pagesWorking space for images in processing 40TB for 1 million pages Search and delivery derivatives 30TB for 3 million pagesSearch and delivery derivatives 30TB for 3 million pages XML files, database systems and indexes 1 TBXML files, database systems and indexes 1 TB

Offline Storage – unlimited for master images on tape.Offline Storage – unlimited for master images on tape.