Projects

CS 661

DAS 02, Princeton, NJ• OCR Features and Systems

– Degradation models, script ID, Bilingual OCR, Kannada OCR, Tamil OCR, mp versus hw checks, traffic ticket reading

• Handwriting Recognition– Stochastic models, holistic methods, Japanese OCR

• Classifiers and Learning– Multi-classifier systems

• Layout Analysis– Skew correction, geometric methods, test/graphics separation, logical

labeling

• Tables and Forms– Detecting tables in HTML documents, use of graph grammars, semantics

• Text Extraction• Indexing and Retrieval• Document Engineering• New Applications

– CAPTCHA, Tachograph chart system, accessing driving directions

ICDAR 03, Edinburgh, UK

• Multiple Classifiers• Postal Automation and Check Processing• Document Understanding• HMM Classifiers• Segmentation• Character Recognition• Graphics Recognition• Non-Latin Alphabets- Kanji/Chinese, Korean/Hangul,

Arabic/Indian• Web Documents, Video• Word Recognition• Image Processing• Writer Identification• Forms and Tables

Project Assignments

Faisal Farooq Multilingual Digital Library- Indexing, Retrieval, Script discrimination

Swapnil Khedekar Multilingual document layout analysis, OCR

Kompalli Surya Multilingual OCR using HMMs

Lei Hansheng Off-line and on-line handwriting integration and matching

Sumit Manocha Fingerprint image enhancement and minutiae extraction

Lin Yu-Hsuan ** Multiple Classifier Combination- multiple modlaities

Praveer Mansukhani Interactive Handwriting Recognition Model

Amalia Rusu Handwritten Captchas

Sutanto Adi ** Indirect biometric data extraction from medical forms

Multilingual Digital Library

Query Result

Control Panel

Query Input

Telugu and Arabic modules under development

Multilingual DIA and OCR

Text/Image Separation

Intervals between peaks

Line Separation• Ascenders & descenders interfering with lines

• Region-growing approach• In Devanagari, single word is a single

connected component• Grow regions using horizontally adjacent

components

Word Separation

• In Devanagari, all characters in a word are glued together by Shirorekha

• Vertical Projection profile easily separates words

Multilingual OCR using HMMs

Continuous Attributes

grapheme pos orientation angle

Down cusp

3.0 -90o

Up loop

Down arc

Stochastic Model

Observations

Integrating Online and Offline Handwriting Recognition

Structural FeaturesBAG

JunctionLoops

LoopTurns

End

End

Feature Extraction and Ordering

Critical node: removal disconnects a connected component.

2-degree critical nodes keep feature ordering from left to right.

LeftComponent

RightComponent

Loop

EndTurns

Junction

LoopsEnd

Turns

Fingerprint Enhancement and Feature Extraction

Fingerprint Recognition

Orientation maps and minutiae detection

Preprocessing Operations

Filtering

•Image Enhancement

•Image Segmentation

•Correlation among fingers

Multiple Classifier Systems

Combination and Dynamic Selection[Govindaraju and Ianakiev, MCS 2000]

WR 1

WR 2

WR 3+Lexicon

1

Top 5

<55Top 50

image

•Optimization problem

•Combinatorial explosion in

•arrangement of recognizers

•lexicon reduction levels

Lexicon Density[Govindaraju, Slavik, and Xue, IEEE PAMI 2002]

Lexicon 1 Lexicon 2

Me MeHe MemoSo MemoryTo MemoirsIn Mellon

Interactive Handwriting Recognition

Handwriting Recognition

Context Ranked Lexicon

Multiple Choice Question

ContextRanked Lexicon

Interactive Models[McClelland and Rumelhart, Psychological Review, 1981]

ABLE TRIPTRAP

A TN

Words

Letters

Features

Handwritten CAPTCHAs

“CAPTCHAs”: Completely Automated Public Turing

Tests to Tell Computers & Humans Apart

• challenges can be generated & graded automatically (i.e. the judge is a machine)• accepts virtually all humans, quickly & easily• rejects virtually all machines• resists automatic attack for many years (even assuming that its algorithms are known?)

NOTE: the machine administers, but cannot pass the test!L. von Ahn, M. Blum, N.J. Hopper, J. Langford, “CAPTCHA: Using Hard AI Problems For Security,” Proc., EuroCrypt 2003, Warsaw, Poland, May 4-8, 2003 [to appear].

Yahoo!’s present CAPTCHA: “EZ-Gimpy”

• Randomly pick: one English word, deformations, degradations,

occlusions, colored backgrounds, etc

• Better tolerated by users• Now used on a large scale to protect various

services• Weaknesses: a single typeface, English lexicon

Indirect Biometrics from Medical Forms Images

Hard biometrics

Face

Eye :Retina & Iris

Fingerprint

Hand Geometry

Handwriting

Speech

DNA

Soft biometrics

Age

Ethnicity

Nationality

Build

Gait

Mannerisms

Writing style

(Semantic)

Derived biometrics

Text/News

WWW

Indirect biometrics

Driver’s License

Medical Records

INS Forms

The Biometrics Spectrum

•Biometric Consortium (www.biometrics.org) lists several products:

–Faces (30); Fingerprints (50); Hand geometry (30); Handwriting (5); Iris (5); Multimodal (6); Retinal (2); Vein (3); Voice (22); Other (20)

–NONE on soft biometrics

–NONE on the fusion of indirect and derived biometrics

NYS EMS PCR FormNYS PCR Example

Thousands are filed a day.Passed from EMS to Hospital.

PCR Purpose:– Medical care/diagnosis– Legal Documentation– Quality Assurance

EMS AbbreviationsCOPD Chronic Obstructive Pulmonary DiseaseCHF Congestive Heart FailureD/S Dextrose in SalinePID Pelvic Inflammatory DiseaseGSW Gunshot WoundNKA No known allergiesKVO Keep vein openNaCL Sodium Chloride

Medical Text Recognition and Data Mining