unsd census workshop day 2 - session 7 data capture: intelligent character recognition andy tye –...

UNSD Census WorkshopDay 2 - Session 7

Data Capture: Intelligent Character Recognition

Andy Tye – International Manager

DRS are Worldwide specialists in Census data capture www.drs.co.uk

Data Capture

Intelligent Character Recognition (ICR)

Elements

• Form design• Hardware/Software requirements

– Scanners– Computer infrastructure

• Workflow• Accuracy• Advantages• Disadvantages


Data Capture - ICR

Forms design

• Typical stock grade paper (90GSM)

• Corner Stones advised

• Dropout colour is recommended but not essential


Hardware requirements

• Image Scanners– TWAIN or ISIS

• Database Server (Full redundancy)• Storage Server – Terabytes

– (Raid 5, Mirrored, etc.)

• Network (Gb preferred)

• Administrator PC• CS-Pro PCs• Key correction PCs (Verification)• Character Inspection PCs

– (Mass verification - optional)• Scanner PCs• Automatic data capture PCs

Software requirements

• MS-SQL or other database• Data Storage, Archive and Retrieval• Backup Software

• Software for Administrator PC• CS-Pro for analysis and reporting PCs• Software for Key correction PCs• Software for Character inspection PCs• Software for Scanner PCs• Software for automatic data capture

Data Capture - ICR


Data Capture - ICR

Typical Workflow

ICR


Data Capture - ICR

Typical Workflow

Paper Movement – Processing Centre/s


Data Capture - ICR

Typical Workflow

Receiving


Data Capture - ICR


Typical Workflow

Logging/Checking

• Open Batch

• Verify Contents

• Register Batch

Data Capture - ICR


Typical Workflow

Sifting

• Orientation

• Other Forms

Data Capture - ICR


Typical Workflow

Spine removal

• Cut Booklets

• 30,000/day

Data Capture - ICR


Typical Workflow

Scanning

• Double Sided

• High Speed

• Double Detection

• Ease of Use

Data Capture - ICR


Typical Workflow

Scanning/sorting

• Automatic Identification

• Data Capture

Data Capture - ICR


Typical Workflow

Storage

• Conditions

• Retrieval

• Space

Data Capture - ICR

Typical Workflow

Image Movement/Data Extraction – Processing Centre/s


Data Capture - ICR


Typical Workflow

Image interpretation

• Automated Process

• Background Task

• Page Identification

• De-skew

• Image Clean up

• Pre-defined Areas

Data Capture - ICR


Typical Workflow

Character inspection

• Tiling

• High Confidence

• Operator Decision

• Field Context

• Tall to Short

Data Capture - ICR


Typical Workflow

Key correction

• Low Confidence

• Operator Decision

• From Context

• External Verification

Data Capture - ICR


Typical Workflow

Key Correction

• ASCII File

• CSV Format

• 1 Line/Form

• CSPro Import

Data Capture - ICR

Typical Workflow

ICR


Data Capture - ICR

Accuracy

This is always the first question

Handprint• Numeric only in isolated fields 98%• Numeric only in semi constrained fields 95-96%• Alpha upper case only 90%• Alpha lower case only 85-87%• Alpha mixed case 75-80%• Alpha/Numeric mixed case 50% or less

– reduce by 5% if there are special characters not a-z and 0-9

The accuracy level post data correction (e.g. the final output accuracy) should be 100% (subject to good operators)


Data Capture - ICR

Accuracy continued…

The accuracy of all modern ICR engines are pretty much comparable

The major differences with suppliers solutions are the methods and workflow utilised with each offering

False positive detection takes 10 times longer than entry of characters recognised with low confidence – false positives (substitutions) are the most expensive errors


Data Capture - ICR

Accuracy continued…

Accuracy can be improved by:

• Restricting the responses to any given question

• Using external verification

• Using multiple ICR engines to ‘vote’ which is expensive

• Training your ICR engines on local hand writing styles (If possible)


Data Capture - ICR

Advantages

• No specialist hardware required

• An image archive can be automatically produced of every form

• Very high speed scanning can be achieved

• Both OMR and ICR can be interpreted using ICR software

• Forms designed for ICR relatively easy to fill in. Locally printed forms can be used.

• Allows capturing much more complex data than with OMR alone


Data Capture - ICR

Disadvantages

• Significant hardware/software and trained IT staff will be required

• Accuracy dependant on manual intervention

• High calibre IT staff are required to support the ICR system

• More complex cost/benefit analysis than with OMR alone


Data Capture - ICR

Indicative Costs

For 65 Million Population Census (20M Single Sided A4 household form)

Processing period of 12 Weeks (8 hours/day 5 days/week)

• Hardware $800k-$1M in total

• Software $700k-$1.3M in total

Total Indicative Costs are $1.5M to $2.3M

• No. of Staff 100-190 in total– 6-10 Managers – 94-180 PC Operators


Data Capture - OMR

Summary

ICR offers considerable flexibility at the cost of higher skilled IT personnel

The single most important factor for timely and accurate data capture is to make sure

‘the forms are filled in correctly and are returned in good condition’


UNSD Census WorkshopDay 2 - Session 7

Thank you for listening

Andy Tye – International Manager


unsd census workshop day 2 - session 7 data capture: intelligent character recognition andy tye –...

Documents