bhl tech overview for bhl-europe

Freeland. BHL Tech Overview. 12 May 2009 www.biodiversitylibrary.org

BHL Technology Overview

Chris Freeland

Technical Director, BHL

Director of Bioinformatics, Missouri Botanical Garden


About BHL: Usage, History


Goals of BHL

• Scan public domain biodiversity literature.

• Negotiate rights to digitize copyrighted materials.

• Ingest content digitized by others.

• Provide interfaces & APIs for repository.– GUIs– Services for data mining & citation resolution

http://www.biodiversitylibrary.org

http://www.biodiversitylibrary.org/


• More than:33,000 volumes

13.3 million pages

• Avg. monthly growth rate1,500 volumes

600,000 pages

Now Online


Monthly Usage Stats

• 45,000 unique users

• 250,000 pageviews


History

• Preliminary work: MOBOT’s Botanicus– http://www.botanicus.org

• Funded by Keck Foundation & IMLS

• Working demonstration of how nomenclators/databases can link into digitized scientific literature

http://www.botanicus.org/


Architecture


Distributed

• Digitized content on Internet Archive servers in California

• Metadata index on MOBOT servers in Missouri

• Image server on MBL servers in Massachusetts

• Nice, but not global

Freeland. BHL Tech Overview. 12 May 2009 www.biodiversitylibrary.orgMOBOT

Petabox cluster

Internet Archive

Image Server

MBL


Scanning Workflow


Scanning OperationsBHL uses scanning centers established by

Internet Archive for mass scanning.

Some partner libraries also scan in-house.

Want to expand international footprint:

•mirrored content•ingest from global data providers

Locations of BHL/IA Scanning Centers

http://www.archive.org/

Workflow

Selection Preparation

Post Production(Re)publication

Digitization

Conservation


Open Access DataFlora medica, oder, Abbildung der wichtigsten officinellen Pflanzen…[Heft 1-18]

Publisher: Jena,August Schmid,1831 [i.e. 1829-1831].

PDF

OCR

XML

JP2

http://www.biodiversitylibrary.org/bibliography/6155



Complexities of distributed, mass scanningfrom NYBG

from Smithsonian


Post Processing & Derivatives


Derivatives

• JPEG2000 (JP2) images

• OCR: ABBY FineReader

• PDF: LuraTech PDF Compressor

• XML metadata


Name Finding via TaxonFinder

http://www.ubio.org/index.php?pagename=soap_methods/taxonFinder

Raw Image Converted to text via OCRName finding via TaxonFinder Extract namesSubmit to NameBankSOAP response

Name Finding in action

with Taxonomic Intelligence…


Name Finding Stats to date*

• Have mined more than 42 million name string occurrences

• More than 30 million name strings verified by NameBank– 1.5 million unique

*12 May 2009


Content Delivery


OCR error rate for names only

1 Insert Space 8 n->v

2 Omit Space 9 l->i

3 e->c 10 r->i

4 u->I 11 u->ii

5 u->n 12 h->l

6 i->l 13 h->ii

7 c->e 14 e->o

Top OCR errors

35.16%

Study in 2008 found that for sample population of 3,003 names, 1,056 were incorrectly transcribed by OCR.

http://biodiversitylibrary.blogspot.com/2008/10/evaluation-of-taxonomic-name-finding.html

http://biodiversitylibrary.blogspot.com/2008/10/evaluation-of-taxonomic-name-finding.html


Current image delivery: djatoka

• Images stored as JPEG2000 (.jp2)

• Decoded & delivered to browser via djatoka– Open source JP2 image server– Developed by digital librarians– Scalable– Rapid development cycle (v1.1)– Growing community of users

djatoka

Browser IIPViewer

www.biodiversitylibrary.org

.jp2

.jpg

IA

/page/1274907

pageid: 1274907

BHLdb

http://www.archive.org/download/mushroomsofameri00palm/.../mushroomsofameri00palm_0010.jp2

images.biodivlibrary.org

A user requests Mushrooms of America, edible and poisonous, Plate X:http://www.biodiversitylibrary.org/page/1274907

locate:

BHL/IA architecture

St. Louis

San Francisco

Woods Hole



New delivery option: IA Bookreader

• Open source

• Example: Flora medicahttp://www.us.archive.org/GnuBook/?id=floramedicaodera118diet#229

IA Book Viewer

http://www.us.archive.org/GnuBook/?id=floramedicaodera118diet#229


APIs & Data Sharing

• Name Service (Documentation)

– REST: XML or JSON

• Data Export (Documentation)

– Monthly export of BHL titles, volumes, pages, names, other metadata in delimited files


*Soon: Citation resolver via OpenURL

Beetle, A. A. 1977. Noteworthy grasses from Mexico V. Phytologia 37(4): 317–407.

http://example.edu/cgi?url_ver=Z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:article&rft.jtitle=Phytologia&rft.atitle=Noteworthy+grasses+from+Mexico&rft.aulast=Beetle&rft.aufirst=A&rft.date=1977&rft.volume=37&rft.issue=4&rft.spage=317&rft.epage=407


Articles


Article repository

• Needed a way to display these PDFs

• Wanted to extend contribution functionality to users

• “Safe harbor” model– BHL provides platform– Community provides content

• Scientists, students, libraries


http://cite.biodiversitylibrary.org

• Drupal with Biblio module

• Multi-lingual interface

• Customizable display, layout

• Solr search/faceting

• OAI & other services for discovery/sharing


Outreach


BHL Blog

• Updates

• Announcements

• 1,500 users / month


Twitter

• twitter.com/BioDivLibrary

• Communication tool– Connecting with LinkedData community, other

users– Receiving assistance, guidance– FAST turnaround


If BHL-E is not a Research Project…


Technologies in hand:

• TaxonFinder

• djatoka

• IA Bookreader

• Drupal/Biblio

• OAI-PMH

• OpenURL

• Fedora Commons


Needed:

• Deduplication Tools

• Storage

• OCR

• Markup/rekeying

• UI/UX

• Interface translation

• Data synchronization


Thank youChris Freeland

4344 Shaw Blvd.

St. Louis, MO 63110

[email protected]

http://www.biodiversitylibrary.org

bhl tech overview for bhl-europe

Technology

phytologia rft

mexico rft

beetle rft

bhldb http

address http

pdf ocr xml jp2

operations bhl

flora medica http