crowdsourcing historical research

Crowdsourcing Historical Research

Claudine ChionhDrupal Downunder 2012

Founders and Survivors

• Study of the 73,000 convicts transported to Van Diemen's Land (Tasmania) between 1803 and 1853

• Records from the convict system and elsewhere• Health, environment, lifestyle, wellbeing• Effects on health and resilience of descendants

http://foundersandsurvivors.org/

Goals of the project

• Compile (health and demographic) data about this population from a range of sources

• Enable other researchers to use this data• Explore quantitative and geographic tools and analyses that are

not commonly used in historical research• Combine professional expertise with the enthusiasm of

volunteers

Some research projects

• Morbidity and mortality on the voyage to Australia• Crime and convicts in Tasmania, 1853-1900• Fertility decline in late C19 Tasmania• Prostitution and female convicts• Tracing convicts' descendants who served in WWI

http://foundersandsurvivors.org/research

Project staff

• Historians• Demographers• Epidemiologists• Two part-time developers

Who are our users?

• Research team• Other interested researchers• Genealogists/family historians• Local historians

Data sources

• Conduct records• Surgeons' journals• Newspaper reports• Births, deaths, marriages• Parish records• Family histories, memories, legends

Official/formal sources

Records from the convict system

• Trial, conviction documents• Conduct records• Ship surgeons' journals• Permissions to marry• Ticket of leave

Outside the convict system

• Births, deaths, marriages• Later convictions

Paper databases

Broader historical context:

• Mass transportation• Modern record-keeping and statistics

Informal sources

• Newspaper reports• Family history: primary sources, compiled genealogies, anecdote

and legend

Our volunteers

• Amateur historians, genealogists• Librarians• IT specialists

How volunteers can contribute

• Individual convict biographies• Tracing batches of convicts in ships

Solutions

• XML database• Drupal• Google Docs

The Founders and Survivors database

• XML (based on Text Encoding Initiative http://www.tei-c.org/)• BaseX XML database engine http://basex.org/

Experimenting with Drupal

• Used an older version of Migrate to import some tabular data as nodes

• Problem of scale: 73,000 convicts• XML approach proved to be more efficient

Getting data into our system

Formal sources

• Collected by archives and individual researchers• CSV, Excel, Filemaker, Access ...• Incorporated into BaseX database with Perl scripts

Informal sources

• Individual convicts' life histories are captured in a Drupal content type ('Community contributed content')

• Some sub-projects also capture summary data in Google spreadsheets

Viewing data

• Master database in BaseX: presented in XSLT, different views for logged in researchers and others

• Community contributed content (CCC): Drupal• Two-way link between master database and CCC• Google spreadsheets prepopulated with links to corresponding

records in master database

Data capture

• Convict biographies captured in Drupal – Community Contributed Content (CCC)

• Linked to entry in XML database• Perl scripts to incorporate CCC records into master database

XML entry for an individual convict

Prepopulated Drupal form

Community contributed content

Ships (batches of data)

• Tracing all convicts on a ship• Summary data in Google Spreadsheets• Spreadsheets are prepopulated from the master database

Ship summary data in Google Spreadsheets

Drupal can't do everything

• Scale• Complexity• Expertise

Where Drupal is appropriate for our project

• Web frontend• Data capture• Collaboration, forums

Summary

• Massive XML database with complex relations• Drupal for capturing slightly complex data and facilitating

collaboration• Google Spreadsheets for capturing tabular data

Questions?

Founders and Survivors

http://foundersandsurvivors.org/

inquiries@foundersandsurvivors.org

Claudine Chionh

http://www.onefewercar.net/

claudine@chionh.org