Proposal for Synergistic Name Extraction from Historical Text Documents

Synergistic Name Extraction

• System operation– Shift burden to the system to the extent possible– Smoothly ranges from fully manual to fully automatic– Full control over level of automation

• Expectations– Immediate improvement (click to fill-in form ready for tech transfer)

• Accuracy can be as good as manual extraction (can guarantee)• Time to extract reduced (likely significantly; potential reduction to 0)

– Can use while still being studied and initially developed• Green Interaction

– Improves with use– Learns from observation and being corrected– Can be bootstrapped from scratch (needed for new language)

Synergy: The interaction of two or more agents or forces so that their combined effect is greater than the sum of their individual effects.

Levels of Automation• Manual

– Type in all info of interest, both stated and inferred– Is the church’s current extraction system (except using DEG’s form-based interface)

• Manual minimal– Click-only form fill-in of stated information– Reasoner provides inferred information– Manual check

• all info of interest displayed• correction option through manual editing

• Synergistic– Initial automatic form fill-in of info, both stated and inferred– Manual check and edit

• Automatic with auditing– Automatic form fill-in for a batch of names– Auditor samples and checks accuracy

• If accuracy deemed sufficient, accept• If insufficient, redo synergistically

• Automatic without auditing– Fully automatic extraction and linking to the FamilySearch tree– Patrons notified when viewing information extracted automatically

• Patrons can view source and extraction & inference results• Patrons can, indeed have the responsibility to, make corrections

Obituaries Project• Demo with the synergistic system in mind• Demo

– 1998 system demo– Korean & French demo– FROntIER demo

• Includes extraction and inference• caution: knowledge engineering not complete

– Synergistic (mock-up of editing)– Manual click-only demo– NewsBank ??

• Can we strike a deal now based on click-only?• Should we first run a pilot experiment to provide numbers for decision making?

– Fully manual to establish baseline– Manual-minimal to see if we’re enough better than the baseline to commit now– Do knowledge engineering for FROntIER obit extraction to see if we can either:

» Develop and use automated synergistic system» Accept the accuracy levels to go fully automatic

Scanned Book Project• Alter envisioned experiment (slightly)

– New objective: investigate accuracy & cost wrt levels of automation– Only a slight change – should be acceptable to all

• Missionary task changes– Single form with fields for all info of interest– Two modes of operation (with different groups of missionaries)

• Type all• Click-only (& let FROntIER get inferred info)

• Thesis experiment changes– No change to the experiment we’re planning– Additional follow-on experimental work

• Using the OntologyEditor, display and check results (as directed by low precision & recall) • Edit by hand and rerun to get improved precision & recall results

• What we can learn by the slightly altered experiment– Type-in missionaries establish baseline– Click-only missionaries yield estimated results for manual-minimal level– Knowledge-engineering experiment:

• Estimates the cost and expertise of knowledge engineering (as originally planned)• Establishes the potential for the fully automatic level (as originally planned)• Roughly estimates accuracy & cost for the synergistic level (i.e., with added fix)