7956890_11-231353_570

FILE HISTORY

US 7,956,890

PATENT:

INVENTORS:

TITLE:

APPLICATIONNO:

FILED:

ISSUED:

COMPILED:

7,956,890

Cheng, Ken P.

Chang, Edward Y.

Wang, Yuan-Fang

Adaptive multi-modal integratedbiometric identification detection andsurveillance systems

US2005231353A

19 SEP 2005

07 JUN 2011

05 AUG 2011

7,956,890

ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION

AND SURVEILLANCE SYSTEMS 5330.07 (SMC)

Transaction History

Date Transaction Description

9/19/2005 Initial Exam Team nn

9/28/2005 IFW Scan & PACR Auto Security Review

10/4/2005 Cleared by OIPE CSR

10/11/2005 Notice Mailed--Application Incomplete--Filing Date Assigned

1/13/2006 Payment of additional filing fee/Preexam

A statement by one or more inventors satisfying the requirement under 35 USC1/13/2006 115, Oath of the.Applic

1/13/2006 Applicant has submitted new drawings to correct Corrected Papers problems

1/20/2006 Application Dispatched from OIPE

1/23/2006 Application Is Now Complete

6/16/2006 IFW TSS Processing by Tech Center Complete

6/16/2006 Case Docketed to Examiner in GAU

10/16/2006 Information Disclosure Statement considered

10/16/2006 Reference capture on IDS

10/16/2006 Information Disclosure Statement (IDS) Filed

10/16/2006 Information Disclosure Statement (IDS) Filed

2/22/2007 Petition Entered

5/18/2007 Correspondence Address Change

5/18/2007 'Change in Power of Attorney (May Include Associate POA)

5/23/2007 Mail-Petition Decision - Dismissed



3/27/2008 Transfer Inquiry to GAU





10/19/2009 Petition Entered

12/3/2009 Mail-Petition Decision - Granted

12/3/2009 Petition Decision - Granted

12/3/2009 Correspondence Address Change


2/23/2010

4/7/2010

4/8/2010

5/12/2010

5/13/2010

12/21/2010

12/27/2010

2/1/2011

2/1/2011

2/7/2011

2/10/2011

3/24/2011

3/24/2011

3/28/2011

3/28/2011

3/28/2011

3/28/2011

3/28/2011

3/28/2011

4/6/2011

4/6/2011

4/6/2011

4/15/2011

4/15/2011

4/18/2011

4/21/2011

4/21/2011

4/21/2011

4/25/2011

5/10/2011

5/10/2011

5/18/2011

6/7/2011

6/7/2011

Case Docketed to Examiner in GAU



Non-Final Rejection

Mail Non-Final Rejection

Aband. for Failure to Respond to O. A.

Mail Abandonment for Failure to Respond to Office Action

Response after Non-Final Action

Petition Entered

Correspondence Address Change

Change in Power of Attorney (May Include Associate POA)

Mail-Petition to Revive Application - Granted

Petition to Revive Application - Granted

Information Disclosure Statement considered

Information Disclosure Statement considered

Reference capture on IDS

Electronic Information Disclosure Statement

Information Disclosure Statement (IDS) Filed

Information Disclosure Statement (IDS) Filed

Date Forwarded to Examiner

Mail Notice of Rescinded Abandonment

Notice of Rescinded Abandonment in TCs

Document Verification

Notice of Allowance Data Verification Completed

Mail Notice of Allowance

Statement Filed Indicating a Loss of Entitlement to Small Entity Status

Issue Fee Payment Verified

Issue Fee Payment Received

Application Is Considered Ready for Issue

Dispatch-to FDC

Dispatch to FDC

Issue Notification Mailed

Recordation of Patent Grant Mailed

Patent Issue Date Used in PTA Calculation

Application/Control No. Applicant(s)/Patent Under Reexamination

Issue Classification 11231353 CHENG ET AL.SIl I I I Examiner Art UnitJerome Grant II 2625

ORIGINAL. INTERNATIONAL CLASSIFICATION

CLASS SUBCLASS CLAIMED NON-CLAIMED

348 143 H 0 4 N 9 / 47(2006.0)

G 0 6 K 9/00 (200601.01)CROSS REFERENCE(S)

CLASS SUBCLASS (ONE SUBCLASS PER BLOCK)382 115 119 118

O Claims renumbered in the same order as presented by applicant O CPA O T.D. O R.1.47

Final Original Final Original Final Original Final Original Final Original Final Original Final Original Final Original

1 - 17

2 18

3 - 19

4 20

5

6

7

8

9

10

- 11

12

13

1 14

2 15

16

NONETotal Claims Allowed:

2(Assistant Examiner) (Date)/Jerome Grant II/Primary Examiner.Art Unit 2625 4-11-11 O.G. Print Claim(s) O.G. Print Figure

(Primary Examiner) (Date) 1 6 and 7

U.S. Paent and Trademark Office Part of Paper No. 20110411

~ Rejected - Cancelled N Non-Elected A Appeal

= Allowed + Restricted I Interference O Objected


CLAIM DATEFinal Original 05/10/2010

1 REJ

2

3

4

5

6

8

9

10

11

12

13

14 'OB

15 OBJ

16 REJ

17

18

19

20

U.S. Patent and Trademark Office

ApplicationlControl No. Applicant(s)/Patent UnderReexamination

Index of Claims 11231353 CHENG ET AL.

||I| I IIIIIIExaminer Art Unit

Jerome Grant II 2625

"

Part of PaperNo.: 20100510


CLAIM DATEFinal Original 05/10/2010 04/11/2011

1 REJ

2

3

4

5

6

7

8

9

10

11

12

13

14 'OB -=

15 OBJ =

16 REJ -

17

18

19

20


Application/Control No. Applicant(s)/Patent UnderReexamination

Index of Claims 11231353 CHENG ET AL.

||| || ||| Examiner Art Unit


Part of Paper No. : 20110411

Application/Control No.. Applicant(s)/Patent UnderReexamination

Search Notes 11231353 CHENG ET AL.I Examiner Art Unit


SEARCHED

Class Subclass Date Examiner713 186 5-10 JG

202 '___ _' _320 '

201 '200 '

340 506 '358 143 '_ _' _147 '' 161 .'

169 '707 103 '

4382 103 '

209276277 '289 '291 "293294295 '305 5-10 JG'

_ _' _282 '115 !

107190

_ _' _116 '345 629 '348 143 '

151155 '161 '169 '

709 227 '' 215 ' "

203 '315 '

U.S. Patent and Trademark Office Part of Paper No.: 20100510

SEARCH NOTES

Search Notes Date ExaminerEast Notes 5-10 JG


INTERFERENCE SEARCH

Class Subclass Date Examiner

I

Part of Paper No.: 20100510

SEARCHED

Class Subclass Date Examiner119S128

SEARCH NOTES

Search Notes -Date ExaminerEast Notes 5-10 JGEast Notes 4-11 JG

INTERFERENCE SEARCH


' _ _ _ 115 '

__ __ __ 119

348 143

U.S. Patent and Trademark Office Part of.Paper No.:' 201104111


Search Notes 11231353 CHENG ET AL.

I IIII I|IIExaminer Art Unit


SEARCHED


202 '320 '201 '200 '

340 506 '358 143 '

147 '' 161

169707 103 '

4 9

382 103 '209 '276277 '289 '291 "

293' 294' 295' 305 5-10 JG'

282_ _' _115

107' 190

116345 629348 143

151155161169

709 227215 ' "

' 203' 315382 118 4-11- 'JG

Part of Paper No.: 20110411U.S. Patent and Trademark Office

EAST Search History

EAST Search History

EAST Search History (Prior Art)

Ref :SHits 'Search Query :DBs Default :Plurals Time Stamp

Operator .'t'i ........ i 'B'i .. ~ ~......... ........................ ........... 2 i~; 'ii9.... ....L i 24 :(security or secure or surveillance or survaillance or confidenital or secret or reconnesance or USiMUB; OR (OFF ?2010/04119

recon$14 or dandesti$6 or watch$4) near7 (camera or sensor or photo adj (conduct$4 or USPAT ;21:43sensor or read$4 or camera or diode) or cod or cmd or scan$4 or snap adj shot) with (person or USOCR; FPRS;child or adult or threat or dclient or object or subject or people or terrorist or man or woman or EPO; JFPO;individual or inhabitant)near5 (biometri$6 or hair or eye or contour or contor or shape or DERWENT;:signature or physique or physical or profile or hair adj line or skin adj (tone or color or hue) or IlBM TDBskin or dothes or shirt or pants or dress or skirt or gown or garment$4) same (profile or map orhistory or memory or register or buffer or library or table or dossier or list or chart or registrar)'and @ad<"20050921".... ....; i ...... .............. ...............................:: :.......::. .. .. .. .. .. .. .. ........................................................................ ........... .............. .... ............... .................. ... ......

1L2 35 (security or secure or surveillance or survaillance or confidenital or secret or reconnesance or US-FGPUJB; OR OFF 12010/04/19recon$14 or dandesti$6 or watch$4) near7 (camera or sensor or photo adj (conduct$4 or USPAT; i :21:43sensor or read$4 or camera or diode) or ccd or cmd or scan$4 or snap adj shot) with (person or USOCR; FPRS;

:child or adult or threat or dclient or object or subject or people or terrorist or man or woman or EPO; JPO;individual or inhabitant)near5 (biometri$6 or hair or eye or contour or face or facial or contor or DERWENT;

:shape or signature or physique or physical or profile or hair adj line or skin adj (tone or color or IBMTDB: hue) or skin or dothes or shirt or pants or dress or skirt or gown or garment$4) same (profile or: map or history or memory or register or buffer or library or table or dossier or list or chart orregistrar) and @ad<"20050921"

L.3 42 (security or secure or surveillance or survaillance or confidenital or secret or reconnesance or USFPGPUB; OR OFF 2010/04/19recon$14 or dandesti$6 or watch$4) with (camera or sensor or photo adj (conduct$4 or sensor USPAT; 21:43or read$4 or camera or diode) or cod or cmd or scan$4 or snap adj shot) with (person or child USOCR; FPRS;cor adult or threat or dient or object or subject or people or terrorist or man or woman or EPO; JPO;individual or inhabitant)near5 (biometri$6 or hair or eye or contour or face or facidal or contor or DERWENT;shape or signature or physique or physical or profile or hair adj line or skin adj (tone or color or IBMTDB

;hue) or skin or dothes or shirt or pants or dress or skirt or gown or garment$4) same (profile or -

:map or history or memory or register or buffer or library or table or dossier or list or chart or i:registrar) and (358/143,144,161,169,400,500,484,483.cds. or 707/103,4.cds. or382/103,209,276,277,289,291,293,294,295,305,282,115,107,190,116.cds. or 345/629.ccls. or?

t348/143,151,155,161,169.ccls. or 709/227,215,203,315.cds. or 713/186,202,320,201,200.ccls.:or 340506.cds.) not L1 not L2

4119110 9:45:24 PM

EAST Search History

EAST Search History


....... ...... I , .... e::......... e r .... ...... ............ i i .......... ........ .....a i... ....................ura is...me .. a... ..... :Ref Hits Search Query DBs Default Plurals Time StampOperator

L1 3 (security or secure or . USPGPUB OR OFF 2010/04/19surveillance or USPAT; :11:53survaillance or confienital USOCR; FPRS;or secret or reconnesance EPO; JPO;or reconessan$5 or DERWENT;clandesti$6) near5 I BMTDB(camera or sensor orphoto adj (conduct$4 orsensor or read$4 or

;camera or diode) or ccdor cmd or scan$4 or snapadj shot) with (person orchild or adult or threat orclient or object or subjector people or terrorist orman or woman orindividual or inhabitant)near5 (biometri$6 or hairPor eye or contour orcontor or shape orsignature or profile or hairadj line or skin adj (tone

nr mv'in~rr N hipr cin nr

clothes or shirt or pantsor dress or skirt or gownor garment$4) same(progile or map or historyor memory or register orbuffer or library) and@ad<"20050921"

I? 2 15 (security or secure or UJS-PGPLJB:

-surveillance or USPATsurvaillance or USOCR; FPRS;confidenital or secret or EPO; JPO;=reconnesance or recon DERWENT;$14 or clandesti$6) near5 IBMTDB(camera or sensor orphoto adj (conduct$4 orsensor or read$4 orcamera or diode) or ccdor cmd or scan$4 or snapadj shot) with (person orihil A h,1 th - tn

c III UorI IUUaUI ior itreorLUI

client or object or subjector people or terrorist orman or woman or

OR OFF ... .. ..................... .......2010/04/1911:57

file:///CI/Documents%20and%20Settings/jgrant/My%20Docu...31353/EASTSearchHistory. 11231353_AccessibleVersion.htm (1 of 3)4/19/10 1:16:29 PM

i

i

i

i

i

a

---------------------- ------------------------

i

:;:;:;

i

i

i

i

i

iz

EAST Search History

individual or inhabitant)Snear5 (biometri$6 or hair

or eye or contour orcontor or shape or

:signature or profile or hairadj line or skin adj (toneor color or hue) or skin orclothes or shirt or pantsor dress or skirt or gownor garment$4) same

, (profile or map or historyor memory or register orbuffer or library or tableor dossier or chart orregistrar) and@ad<"20050921.. .......... ... ... S i r c ; e r............-

L3 24 (security or secure orsurveillance orsurvaillance orconfidenital or secret orreconnesance or recon$14 or clandesti$6 orwatch$4) near7 (cameraor sensor or photo adj(conduct$4 or sensor orread$4 or camera ordiode) or ccd or cmd orscan$4 or snap adj shot)with (person or child oradult or threat or client orobject or subject orpeople or terrorist or manor woman or individual orinhabitant)near5 (biometri$6 or hair or eye orcontour or contor orshape or signature orphysique or physical orprofile or hair adj line orskin adj (tone or color orhue) or skin or clothes orshirt or pants or dress orskirt or gown or garment$4) same (profile or mapor history or memory or:register or buffer orlibrary or table or dossieror list or chart orregistrar) and@ad<"20050921"

iUS- PGPt JRUUSPAT;USOCR; FPRS;EPO; JPO;DERWENT;IBM TDB

file:///CI/Docunents%20and%20Settings/jgrant/My%20Docu...31353/EASTSearchHistory. 11231353_AccessibleVersion.htm (2 of 3)4/19/10 1:16:29 PM

ICOFF 2010/04/1912:05

T.......................................... ..................... ...............................-...... ~.

:;:;:;:;:;:;

i

iE

EAST Search History

(security or secure orsurveillance orsurvaillance orconfidenital or secret orreconnesance or recon$14 or clandesti$6 orwatch$4) near7 (cameraor sensor or photo adj(conduct$4 or sensor or!read$4 or camera ordiode) or ccd or cmd orscan$4 or snap adj shot)with (person or child oradult or threat or client orobject or subject orpeople or terrorist or manor woman or individual orinhabitant)near5 (biometri$6 or hair or eye orcontour or face or facialor contor or shape orsignature or physique orphysical or profile or hairadj line or skin adj (toneor color or hue) or skin orclothes or shirt or pantsor dress or skirt or gownor garment$4) same(profile or map or historyor memory or register orbuffer or library or tableor dossier or list or chartor registrar) and@ad<"20050921"

iUS-Ffl:;.PIiR- 1 OR ...................... ...... FF......... ,2f01/04/19............:USPAT;USOCR; FPRS;EPO; JPO;DERWENT;I BMTDB

4/19/10 1:16:09 PM

file:///CI/Docwnents%20and%20Setings/jgrant/My%20Docu...31353/EASTSearchHistory. 11231353_AccessibleVersion.htm (3 of 3)4/19/10 1:16:29 PM

L4 35........... . . .. . . . . .. . . . . .. . . . . . . .. . . . . .. . . . . .. . . . . . .. .. . . ... .. ..'""

:;

:;

:;

::

:;

::

:,

:;

:

:;

_,

i

i

't

i

i

i

i

i

i

i

i

i

. ........

;v Zvi w v vii v

;i r) "nai

i

i

't

i

11Z:U5

...... , ........... ................................ ....................... :

:;:;

................................... .

ii

EAST Search History

EAST Search History


Ref # Hits Search Query DBs ~ Default PluraIs Time StampOperator.......... ........ ........... ...........:: ................................................ .......... ..... .... ............... . ............. ............... ... ............. ......... ..

L1 22 (security or secure or US-PGPUB,; OR OFF 2011/04/11:i :: :2'2 12 3surveillance or USPAT; USOCR; 12:38

:survaillance or confiential :FPRS; EPO;or secret or reconnesaince JPO;or recon$14 or dlandest$6 DERWENT;or watch$4) near7 I;IBMTDB(camera or sensor orphoto adj (conduct$4 orsensor or read$4 orcamera or diode or ccd orcmd or scan$4 or snap adjshot)) with (person orchild or adult or threat orclient or object or subjector people or terrorrist orman or woman or ,individiual or inhabitant)near5 (biometri$6 or hairor eye or contour or face'or facial or contor orcontour or shape orsignature or phsique orphusical or profile or hair:adj line or skin adj (toneor color or hue or tint) or

:clothes or skin or shirt orpants or dress or skirt orgown or garment$4) same(profile or map or historyor memory or register orbuffer or library or tableor dossier or list or chartor registrar) and@ad< 20050921'"'.... . .......... . ... . ......e.. ......."u ..r.or................ . ..... . ....... i6 ... ............. ..... .....

S1 32808 (recipient or user or USPGPUB; OR OFF 2010/12/11:operator or client or USPAT; USOCR; 08:53personell or personal or FPRS; EPO;person) near4 (setting JPO;address or parameers or DERWENT;meta or data) same I BM_TDB(server or exchange orsaff or forward or modemor packet$4) same(display or monitor orscreen or Icd or window orview$4) and (protocol orprotocal or http or smtp orxml or remote adj

iprocedure) and

file:///CI/Documents%20and%20Settings/jgrant/My%20Doc...1353/EASTSearchHistory.11231353_AccessibleVersion.htm (1 of 26)4/11/1112:42:38 PM

@ad< "20071213"

S2 16598 (recipient or user or US-PGPUB; OR OFF 2010/12/11operator or client or USPAT; USOCR; 08:55personell or personal or FPRS; EPO;person) near4 (setting JPO;address or parameers or DERWENT;meta or data) same IBM _TDB(server or exchange or'.saff or forward or modemor packet$4) same(display or monitor orscreen or Icd or window orview$4) and (protocol orprotocal or http or smtp orxml or remote adjprocedure) and@ad<"20071213" and(scan$4 or read$4 orcopier) near4 (data oremail or electronic adjmail or message or copyor data)' ............i ................................... ............. . ... .. ... . .............. ..... ........... ............... . ..... :.. . .. ... ..... ... .. .. . ....... ............ .... ................ .... . ..... ..... .. . .... .... ....:

S3 ;5193 (recipient or user or . US-PGPUB; OR OFF 2010/12/11operator or client or =USPAT; USOCR; :08:56personell or personal or FPRS; EPO;person) near4 (setting or ,JPO;address or parameers or DERWENT;meta or data) with (server ;IBMTDBor exchange or saff orforward or modem orpacket$4) with (display ormonitor or screen or Icd or

:window or view$4) and(protocol or protocal orhttp or smtp or xml orremote adj procedure)and @ad<"20071213" and(scan$4 or read$4 orcopier) near4 (data oremail or electronic adjmail or message or copyor data)........ ............ ... ..............! ... ........... . .... .............. . .. .. ..... .... ..... ..... ..... .... .................................... .:

S4 ......... 2562 (recipient or user or ............... US-PGPUB; ........... OR... .......................... OFF ...... 2010/12/11............operator or client or 'USPAT; USOCR; 08:58personell or personal or FPRS; EPO;Iperson) near4 (setting or JPO;address or parameers or DERWENT;meta or data) near9 I BMTDB(server or exchange orsaff or forward or modemor packet$4) with (displayor monitor or screen or Icdor window or view$4) and

;(server or http or smtp orxml or transmitter) near8(protocol or protocal orhttp or smtp or xml or

file:///Cl/Documents%20and%20Settings/jgrant/My%20Doc...1353/EASTSearchHistory. 11231353AccessibleVersion.htm (2 of 26)4/11/11 12:42:38PM

EAST Search History

remote adj procedure)and @ad<"20071213" and(scan$4 or read$4 orcopier) near4 (data oremail or electronic adj:mail or message or copyor data). . . .. . . . . . .. . . . . .. . . . .. : . . .. .. . . . . . .. . . . . . .. . . . . .. . .. . .. . . . . . . . . . . .. . . . . .. . . . . .. . . . . . . .. .. .. . .. . .. . . . . . .. . . . . . . . . . .. . . . . .. . . .. .

2928 :(recipient or user or :US-PGPUB;operator or client or USPAT; USOCR;personell or personal or FPRS; EPO;:person) near4 (setting or JPO;address$4 or indicat$4 or :DERWENT;select$4 or parameters or ;IBMTDBmeta or data) near7(server or exchange orsaff or forward or modemor packet$4) with (displayor monitor or screen or Icdor window or view$4) and(server or http or smtp orxml or transmitter) near4(protocol or protocal orhttp or smtp or xml orremote adj procedure)and @ad<"20071213" and(scan$4 or read$4 orcopier) near4 (data oremail or electronic adjmail or message or copyor data)....................................... .Y. ........ Y..... ............................................................

1437 (recipient or user or US-PGPUB;operator or client or USPAT; USOCR;personell or personal or FPRS; EPO;person) near4 (setting or JPO;address$4 or indicat$4 or DERWENT;select$4 or parameters or IBMTDB:meta or data) near4(server or exchange orsaff or forward or modemor packet$4) with (displayor monitor or screen or Icdor window or view$4) and(server or http or smtp orxml or transmitter) near4(protocol or protocal orhttp or smtp or xml orremote adj procedure)and @ad<"20071213" and(scan$4 or read$4 orcopier) near4 ( server orsaff or packet or modemor email or electronic adj(mail or message or copyor data))

.... . .. . . . . .. . . . . .. . . ;.. .. . . . . . .. . . . . .. . . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . .. . . . .. . . . . . . . . . .. . . . .. . . . . .. . . . . .. . . . . .. . . . . . .. . .

file:///Cl/Documents%20and%2OSettings/jgrant/My%2ODoc:..1353/EASTSearchHistory.11231353_AccessibleVersion.htm (3 of 26)4/11/11 12:42:38PM

EAST Search History

S6 OFF 1

.1

OR

OR

i

i

i

i

i

i

i

i

i'

2010/12/1109:12

............2010/12/1109:16

:,

ii

i

i..................... :

......................

;OFF

... .... ......... ... ...... ...........OR..................................

: :

i

........

::Q96 (rrinient or ir r or ii -P(Pi IRU S-GPiS8

S9

file:///CI/Documents%20and%20Settings/jgrant/My%20Doc...1353/EASTSearchHistory. 11231353_AccessibleVersion.htm (4 of 26)4/11/11 12:42:38 PM

EAST Search History

OR :;OFFii' v

i\ 'm

'.

' , .... .. .... ., .. ... , !S i Ioperator or client or USPAT; USOCR;

personell or personal or FPRS; EPO;:person) near4 (setting or JPO;address$4 or indicat$4 or DERWENT;select$4 or key$4 or input IBM_TDB$5 or choosing or chosen)near4 (server or exchangeor saff or forward ormodem or packet$4) with(display or monitor orscreen or Icd or window orview$4) and (server orhttp or smtp or xml ortransmitter) near4(protocol or protocal orhttp or smtp or xml orremote adj procedure)and @ad<"20071213" and(scan$4 or read$4 orcopier) near4 ( server orsaff or packet or modemor http or email or

:electronic adj (mail or:message or copy or data))

4359 (recipient or user or US-PGPUB;operator or client or USPAT; USOCR;personell or personal or FPRS; EPO;person) near4 (setting or :JPO;address$4 or indicat$4 or DERWENT;select$4 or key$4 or input IBMTDB$5 or choosing or chosen)near4 (server or exchangeor saff or forward ormodem or packet$4) and(display or monitor or

:screen or Icd or window orview$4) near4 (image oroutput or picture or viewor image or data) and(server or http or smtp orxml or transmitter) near4(protocol or protocal orhttp or smtp or xml orremote adj procedure)and @ad<"20071213" and(scan$4 or read$4 or

:copier) near4 ( server orsaff or packet or modem

: or http or email orelectronic adj (mail orimessage or copy or data))

: 2010/12/11!09:19

vi

? . .. . .. . .. . . . .. . . . . . . . . . . . ..

OR (iOFF..............

ia

';

:, ................. ............. ......................... ......... ......................... ................:................ ...................... ; ;.............................

:;:;:;:;:;:;

:;

i

i

i

i

i

i

2010/12/1109:22

i

i

i

EAST Search History

Si 0 599 (recipient or user or US-PGPUB;operator or client or USPAT; USOCR;personell or personal or FPRS; EPO;person) near4 (setting or :JPO;address$4 or indicat$4 or DERWENT;select$4 or key$4 or input I BMTDB$5 or choosing or chosen):near4 (server or exchangeor saff or forward ormodem or packet$4) and(display or monitor or

:screen or Icd or window orview$4) near3 (image oroutput or picture or viewor image or data) same(server or http or smtp orxml or transmitter) near4 "

;(protocol or protocal orhttp or smtp or xml orremote adj procedure)and @ad<"20071213" and

:(scan$4 or read$4 or:copier) near4 ( server orsaff or packet or modemor http or email or

:electronic adj (mail ormessage or copy or data))

2................. 20080068642". pn..... .......... US-P PUB;...........

USPAT; USOCR;FPRS; EPO;

•JPo;DERWENT;IBMTDB

1925 (print$4 or copier eor mfp US-'PGPUB;or record$4) same USPAT; USOCR;(material or content or FPRS; EPO;page or sheet or original JPO;or file or image or picture DERWENT;or top or bottom) near5 IIBMTDB(part or portion or sectionor region or line orparagraph) same (control$4 or correct$4 or adjust$4 or modification or

:modifying or chang$4 or:altering) near3 (resolutionor gradation or pitch)and@ad<,"20080317"

.;09:23

i '

i

i

i

i

i

i

i '

i.

i

i

i' .

i

i

ia

,OFF

i

..................... .... ...................................

OFF ::2011 / 03/0707:40

file:///CI/Documents%20and%20Settings/jgrant/My%20Doc...1353/EASTSearchHistory. 11231353 AccessibleVersion.htm (5of 26)4/11/11 12:42:38 PM

OR

................... .. .. ...................................... : ........................................,OR ................... ;.......:;OFF......... ;2 10/12/1.1............ .

V I l

2'

t

C

i t

2011/02/2806:37

.;

:,

:;:;

:;

:;

:;

i

........................................

OR

> . . . . . . . . .. . . . . . . . . . .. . . .

EAST Search History

1011 (print$4 or copier or mfpor record$4) same(material or content orpage or sheet or originalor file or image or pictureor top or bottom) near5(part or portion or sectionor region or line orparagraph) near9 (control$4 or correct$4 or adjust$4 or modification ormodifying or chang$4 oraltering) near3 (resolutionor gradation or pitch)and@ad< "20080317"

744 (........................rint$4 or copier or m f............744 :!(print$4 or copier or mfpS14 -

(print$4 or copier or mfpor record$4) with(materialor content or page orsheet or original or file orimage or picture or top orbottom) near5 (part orportion or section orregion or line orparagraph) near6 (control$4 or correct$4 or adjust$4 or modification or:modifying or chang$4 oraltering) near3 (resolutionor gradation or pitch)and@ad<"20080317"

US-PGPUB;USPAT; USOCR;FPRS; EPO;JPO;DERWENT;I BM TDB

:i

US-PGPUB;USPAT; USOCR;FPRS; EPO;JPO;DERWENT;IBM TDB

i -

.i] OFF :;2011 /03/07:;07:43

................ ........

OFF

OFF ......

:;:,:,:;:,:,:,:;:;:;:,:,:,:;

:;:;:;:;:;:;

:;:;

i

file:///CI/Documents%20and%20Settingsjgrant/My%20Doc...1353/EASTSearchistory.11231353_AccessibleVersion.htm (6 of 26)4/11/11 12:42:38 PM

or record$4) with(materialor content or page orsheet or original or file orimage or picture or top orbottom) near5 (part orportion or section orregion or line orparagraph) near9 (control$4 or correct$4 or adjust$4 or modification ormodifying or chang$4 or

:altering) near3 (resolutionor gradation or pitch)and@ad< "20080317"

S15 597

... .... .. ..................... .. :- ----- ----. ............

.... ........................................ .. .~~........~..........................

........................ ---- - - -- - .---- -- - -- - -- ........................................

.................

i

..... .,.......... ..

iS13

US-PGPUB;USPAT; USOCR;FPRS; EPO;JPO;DERWENT;

i BMTDB

::;:,:;:,:,:;

,.

; vvi v

'IOR;vi i

i

2011/03/0707:44

2011/03/0707:47

:S16 "419 " (print$4 or copier or mfp) US-PGPUB; OR OFF 2011/03/07with(material or content USPAT; USOCR; 07:49or page or sheet or FPRS; EPO;original or file or image or JPO;picture or top or bottom) DERWENT;near5 (part or portion or ;BM TDBsection or region or line orparagraph) near6 (control$4 or correct$4 or adjust$4 or modification ormodifying or chang$4 or

S altering) near3 (resolutionor gradation or pitch)and@ad< "20080317"

S17 141 (user or client or operator US-PGPUB; OR OFF 2011/03/08or person) near3 (icon or USPAT; USOCR; 19:10module or machine or FPRS; EPO;comparaor or processor or JPO;interface or keybaord or DERWENT;select$4 adj means or . I BM _TDBseelct$4) near4 (select$4or input$4 or designat$4or choos$4 or indicat$4 orkey$4 or indicat$4) near5(process or operation orresolution or size or coloror black adj white or printadj (function orapplication)) and (reenteror resumbit$4 or:redesignat$4 or key adj inor re adj enter or re adjsubmit$4 or re adj transmit$4 or retransmission orretransmitting ) near6

;(icon or module or;machine or comparator or:processor or interface orkeyboard or select$4 adjmeans or select$4 orprocess or opepration orresoloution or size or coloror black adj white or printadj (function or applica$4))

S18 2579755 (print$4 or attribute or job US-PGPUB; OR " OFF 2011/03/10or control or process$4 or USPAT; USOCR; 07:34customiz$5) and (print$4 FPRS; EPO;or attribute or job or JPO;control or process$4 or DERWENT;customiz$5) near4 (print I BMTDB$4 or output$4 or record$ 4)...... .... ..... .. ... ..... ..... ..... . .. . .... ..... ... .... ... ... .... .. ... .. . ... .. ..... .. . .. ... .. .. .. ... .. .. ..... ..... ... ... .. .. .... .. ... .... .. .. ...... . ... . .. ... .... .. ..... ... ..... ... .. .. ... .

file:///ClfDocuments%20and%20Settings/jgrant/My%20Doc...1353/EASTSearchHistory.11231353_AccessibleVersion.htm (7 of 26)4/11/1112:42:38 PM

EAST Search History

EAST Search History

S19 393331 (print$4 or attribute or job US-PGPUB; OR OFF 2011/03/10or control or process$4 or USPAT; USOCR; 07:45customiz$5) and (print$4 FPRS; EPO;or attribute or job or JPO;control or process$4 or DERWENT;customiz$5) near4 (print IBM_TDB$4 or output$4 or record$4) and (digital) near4(computer or process$4 orimage or data or file) and@ad<"20080321". ....... . .............. .......... .......... .0 8 3 !......................... ... .... ...... -- .. .. .......... ...................... .. .. ........................

S20 53550 (print$4 or attribute or job US-PGPUB; . OR OFF 2011/03/10or control or process$4 or USPAT; USOCR; :07:47customiz$5)near3 (sheet FPRS; EPO;or page or paper) and JPO;(print$4 or attribute or job DERWENT;or control or process$4 or :IBMTDBcustomiz$5) near4 (print$4 or output$4 or record$4) and (digital) near4(computer or process$4 orimage or data or file) and@ad< 20080321". ....... .... ............ .......... ........ . .......... ............ : ... ..... .................................

521 33678 (print$4 or attribute or job US-PGPUB; OR OFF 2011/03/10or control or process$4 or USPAT; USOCR; :07:48customiz$5)near3 (sheet FPRS; EPO;or page or paper) same JPO;(print$4 or attribute or job DERWENT;or control or process$4 or IlBMTDBcustomiz$5) near4 (print$4 or output$4 or record$4) and (digital) near4(computer or process$4 orimage or data or file) and@ad<"20080321"

d < .......... .................... ............. ........00 8 .3 .. ".. .............................................................. .: ....................................!.. ..........................................................;52 11 (print$4 or attribute or job US-PGPUB; OR OFF 2011/03/10

or control or process$4 or USPAT; USOCR; '07 :51customiz$5)near3 (sheet FPRS; EPO;or page or paper) near9 JPO;(print$4 or output$4 or DERWENT;record$4) near3 (print$4 IBMTDBor attribute or job or

.control or process$4 orcustomiz$5)near3 (sheetor page or paper) anddigitial near3(computer orprocess$4 or image or

:data or file) and@ad<"20080321"

ie:///C .................I.........../Documents%2 and%2 Setings/jgrant/My%2 Doc...353EASTSearchsory. 23353AccessibleVersion. htm (8 f26 /1/11124238PM .............................

ile:///CI/Documents%20and%20Settings/jgrant/My%20Doc...1353/EASTSearchHistory.11231353_AccessibleVersion.htm (8 of 26)4/11/11 12:42:38 PM

EAST Search History

,S23 7 (print$4 or attribute or job US-PGPUB; OR OFF 2011/03/10or control or process$4 or USPAT; USOCR; _07:52customiz$5)near3 (sheet IFPRS; EPO;or page or paper) and JPO;(print$4 or output$4 or DERWENT;record$4) near3 (print$4 ;BM TDBor attribute or job orcontrol or process$4 orcustomiz$5)near3 (sheetor page or paper) anddigitial near3(computer orprocess$4 or image or

:data or file) and@ad< "20080321". . . . ... .. ...... ...................................................... ................ ................ .. ............ .... ................... .. ...... ...... ..........................

8242122..... 6 (print$4 or attribute or job US-PGPUB; OR OFF 2011/03/10or control or process$4 or USPAT; USOCR; 07:52customiz$5)near3 (sheet FPRS; EPO;or page or paper) and JPO;(print$4 or output$4 or :ERWENT;

: record$4) near3 (print$4 IBMTDBor attribute or job orcontrol or process$4 orcustomiz$5)near3 (sheetor page or paper) anddigital near3(computer orprocess$4 or image or

* data or file) and:: "2: :

@ad< 20080321"...........25 4726 (print$4 or attribute or job US-PPUB; OR OFF 2011/03/10

or control or process$4 or USPAT; USOCR; 07:53customiz$5)near3 (sheet FPRS; EPO;or page or paper) near9 JPO;

* '(print$4 or output$4 or DERWENT;record$4) near3 (print$4 IBM_TDBor attribute or job orcontrol or process$4 orcustomiz$5)near3 (sheetor page or paper) same

:digital near3(computer orprocess$4 or image ordata or file) and@ad< "20080321".

S26 5237 (print$4 or attribute or job US-PGPUB; OR OFF 2011/03/10or control or process$4 or :USPAT; USOCR; :;07:56customiz$5)near3 (sheet FPRS; EPO;or page or paper) near9 JPO;(print$4 or output$4near2 DERWENT;(device or machine or 'IBM_TDBapparatus) or record$4)near3 (print$4 or attributeor job or control or process:$4 or customiz$5)near3(sheet or page or paper)same digital near3(computer or process$4 or

;image or data or file) and

file:///CI/Documents%20and%20Settings/jgrant/My%20Doc...1353/EASTSearchHistory.11231353_AccessibleVersion.htm (9 of 26)4/11/1l 112:42:38 PM

EAST Search History

~~~@ad<"20080321"S 114FA (attrihIt0 A r inh nr rcontrol :H R-P(GPi iP* ?OR ;OfFF :2011/03M/10l

or process$4 or customiz USPAT; USOCR;$5)near3 (sheet or page FPRS; EPO;or paper) near9(print$4 or JPO;output$4near2 (device or DERWENT;machine or apparatus) or IBMTDBrecord$4) near3 (print$4or attribute or job orcontrol or process$4 orcustomiz$5)near3 (sheetor page or paper) samedigital near3(computer orprocess$4 or image ordata or file) and@ad< "20080321"

846 (attribute or job or controlor process$4 or customiz$5)near3 (sheet or pageor paper) near9(print$4 oroutput$4 near2 (device ormachine or apparatus) orrecord$4) near3 (attributeor job or control or process:$4 or customiz$5)near3(sheet or page or paper)same digital near3(computer or process$4 orimage or data or file) and@ad<"20080321"

450 (attribute or job or control:Ior process$4 or customiz I$5)near3 (sheet or pageor paper) near9(print$4 oroutput$4 near2 (device or I

:machine or apparatus) orrecord$4) near3 (attributeor job or control or process:$4 or customiz$5)near3(sheet or page or paper)with digital near3(computer or process$4 orimage or data or file) and@ad< "20080321"

US-PGPUB;USPAT; USOCR;FPRS; EPO;JPO;DERWENT;IBMTDB


OR

VI l VI I :;C-v /VV/ IV

2011/03/1007:59

OR ............................ OFF ....... 2011/03/10............08:00

file:///Cf/Documents%20and%20Settings/jgrant/My%20Doc...1353/EASTSearchHistory.11231353_AccessibleVersion.htm (10 of 26)4/11/11 12:42:38 PM

S28

S29

i

g

i

i

. .. . . . . . . . . .. .

NOFF,

:;:;

;VI 1

i';

;.

........

EAST Search History

S30 846 digital near3( computer or US-PGPUB; OR OFF. 2011/03/10process$4 or image or USPAT; USOCR; 08:02data or file)same(attribute FPRS; EPO;or job or control or process JPO;$4 or customiz$5)near3 DERWENT;(sheet or page or paper) IBMTDBnear9(print$4 or output$4near2 (device or machineor apparatus) or record$4)near3 (attribute or job orcontrol or process$4 orcustomiz$5)near3 (sheetor page or paper) and

d<"20080321"... .. .. .. .. .. .. .. .. .. .. .. ... .. .. .. .. .. .. .. .. .. .. .. .. .... ...... ... ................ ........U .. ......... O. ................................. ........ ... ... 1 /.. . 0 3.. .. .. ........... ;. .. . .. .

S31 .. 846 S30 or S29 USPGPUB; OR OFF 2011/03/10USPAT; USOCR ;08:03FPRS; EPO;JPO;DERWENT;IBMTDB

S32 677 digital near3(computer or US-PGPUB; OR OF 2011/03/10process$4 or image or :USPAT; USOCR 08:05data or file)same(attribute FPRS; EPO;or job or control or process :JPO;$4 or customiz$5)near3 DERWENT;(sheet or page or paper) :IBMTDBnear5(print$4 or output$4near2 (device or machineor apparatus) or record$4)near3 (attribute or job orcontrol or process$4 orcustomiz$5)near3 (sheetor page or paper) and@ad< "20080321 "ii ....i ' Y :. . ....... ........ ........ ....... .:...... ....... ..... .::.. ... ... .. ... ... .. .............. .. ... ... .... ... ....... ................................. .". ... ...i[ 6 ' '5 " ............... ...

S33 337 digital near3(computer or US-PPUB, OR OFF 2011/03/10process$4 or image or USPAT; USOCR; 08:06data or file)with(attribute.. FPRS; EPO;or job or control or process JPO;$4 or customiz$5)near3 DERWENT;(sheet or page or paper) IBM_TDBnear5(print$4 or output$4near2 (device or machineor apparatus) or record$4)near3 (attribute or job or

:control or process$4 orcustomiz$5)near3 (sheetor page or paper) and .@ad<"20080321"

.,,.

file:///CI/Documents%20and%20Settings/jgrant/My%20Doc...1353/EASTSearchHistory. 11231353_AccessibleVersion.htm(11 of 26)4/11/11 12:42:38 PM

EAST Search History

S34 ;151 digital near3(computer or US-PGPUB; OR ;OFF " 2011/03/10process$4 or image or USPAT; USOCR; 09:09data or file)with(attribute FPRS; EPO;or job or control or process JPO;

'_ $4 or customiz$5)near3 DERWENT;(sheet or page or paper) IBMTDBnear5(print$4 or output$4near2 (device or machineor apparatus) or record$4)near3 (attribute or job orcontrol or process$4 orcustomiz$5)near3 (sheetor page or paper) and@ad<"20080321" and("358"/$.ccls. or "347"/$.ccIs. or "399"/$.ccls. or"355"/$.ccls.)

S35 1 .digital near3(computer or US-PGPUB OR OFF . 2011/03/10process$4 or image or USPAT; USOCR; 09 : 11data or file)with(attribute FPRS; EPO;or job or control or process JPO;$4 or customiz$5)near3 DERWENT;(sheet or page or paper) IBM TDBnear5(print$4 or output$4near2 (device or machine'_or apparatus) or record$4)near3 (attribute or job orcontrol or process$4 orcustomiz$5) near3 (sheetor page or paper) and(yacoub.inv. or liang.inv.or zhou.inv.)

S36 400 S30 and ("358"/$.ccls. or US-PGPUB; OR OFF 2011/03/10"347"/$.ccls. or "399"/$. 'USPAT; USOCR, :09:15ccls. or "355"/$.ccls.) FPRS; EPO;

JPo;DERWENT;I BM TDB

S37 249 ............ S30 and ('358"/$.ccls. or 'US-PGPUB; OR ..........OFF 2011/03/10"347"/$.ccls. or "399"/$. USPAT; USOCR; 09:15ccls. or "355"/$.ccls.) not FPRS; EPO;S33 JPO;

DERWENT;,S. ...................................... .......................................................... ......-T .................i.. ...................... ........... ...... ......................................... ........

file:CllBMTDocuments%20and%20SetngsjgrantMy%20Doc...1353EASTSearchHistory.11231353_AccessibleVersion.htm (12 of 26/1/11 12:42:38 PM

file:///Cl/Documents%20and%20Sertings/jgrant/My%20Doc...1353/EASTSearchHistory.11231l353_AccessibleVersion.htm (12 of 26)4/11/1112:42:38 PM

EAST Search History

2(

:;

:;

080 :(recipient or user oroperator or client orpersonell or personal or.person) near4 (setting oraddress or parameters or:meta or data) near7(server or exchange orsaff or forward or modemor packet$4) with (displayor monitor or screen or Icdor window or view$4) and(server or http or smtp orxml or transmitter) near4(protocol or protocal orhttp or smtp or xml or

:remote adj procedure)and @ad< "20071213" and(scan$4 or read$4 or

:copier) near4 (data oremail or electronic adj:mail or message or copyor data)

US-PGPUB;USPAT; USOCR;FPRS; EPO;JPo;DERWENT;I BM_TDB

S38

S39.

S40

OR OFF 2011/03/10...........22:06

......................................... ........ . ..":- -- - -- -- - -- - - -

OR OFF 2011/03/1110:35

......................... .... ............,

2011/03/1110:40

file:///CI/Documents%20and%20Settings/jgrant/My%20Doc...1353/EASTSearchHistory.11231353_AccessibleVersion.htm (13 of 26)4/11/11 12:42:38 PM

17359 (print$4 or (image or data :US-PGPUB;or picture) near4 (formulat USPAT; USOCR;$4 or form$4 or print$4 or FPRS; EPO;output$4 or unit or device JPO;or apparatus)) near9 DERWENT;(thumb adj drive or disc or IBMTDBdisk or dvd or flash adjdrive or cd or memory adjcard or external adj (fufferor stor$4 or memory orregister)) same (display or.

:monitor or Icd or screen orwindow) with (indicat$4or select$4 or keyboard orkeypad or designat$4 orchoose or choice or enter)and @ad< "20080221" '

26 (print$4 or (image or data US-PGPUB;or picture) near4 (formulat :USPAT; USOCR;$4 or form$4 or print$4 or :FPRS; EPO;output$4 or unit or device JPO;or apparatus)) near4 :DERWENT;(thumb adj drive. or disc or IBM TDBdisk or dvd or flash adj.-drive or cd or memory adjcard or external adj (fufferor stor$4 or memory orregister)) near9 (displayor monitor or Icd or screenor window) with (indicat$4 or select$4 orSkeyboard or-keypad ordesignat$4 or choose orchoice or enter)with (fileor address or identification

: - ",- .....................r~\~rr\ . ............. ......... ..... ... .. ..... ..... ... . .... 111-11.1-11.

1

nR 1OFF;vn .,, i

EAS' Search History

or identify or name)near3-(name or paper or file orjob or document or sheet):and @ad<."20080221"

:CA i....... :i 00...........

(print$4 or (image or data:or picture) near4 (formulat$4 or form$4 or print$4 oroutput$4 or unit or deviceor apparatus)) with(thumb adj drive or disc ordisk or dvd or flash adjdrive or cd or memory adjcard or external adj (fufferor stor$4 or memory orregister)) with (display or:monitor or Icd or screen orwindow) with (indicat$4or select$4 or keyboard orkeypad or designat$4 orchoose or choice or enter)with (file or document orsheet or address oridentification or identify orname)near3 (name orpaper or file or job ordocument or sheet) and.@ad< "20080221" and("358"/$.ccls. or "355"/$.ccls. or "347"/$.ccls. or"399"/$.ccls.)

... . .. . . . . . .. . . . . .. . . . . .. . . . . .. . . . . . .. . . . . .. . . . . . .. . . . . .. . . . .. . . . . .. .

;188


iSi4];;..,-r

i

;;

S42

(print$4 or (image or dataor picture) near4 (formulat$4 or form$4 or print$4 oroutput$4 or unit or deviceor apparatus)) with(thumb adj drive or disc ordisk or dvd or flash adjdrive or cd or memory adjcard or external adj (fufferor stor$4 or memory orregister)) with (display or:monitor or Icd or screen or:window) with (indicat$4or select$4 or keyboard orkeypad or designat$4 orchoose or choice or enter)

:with (file or document orsheet or address oridentification or identify orname)near3 (name orpaper or file or job orIdocument or sheet) andIa@ad< "20080221"

.... .... .. .. .. .... .. .... .. .. ..... .... ... .. .. ...... ... .. ..~~~~_

i

:;

US-PGPUB; OR OFF 20103/11USPAT; USOCR; 10:44FPRS; EPO;JPO;DERWENT;IBM TDB.

. . . . . . . ... . . .. . . .. . . . . . . . . . . . . . . . . . .. . . .. . . . . .. ".. . . . ...... .. .. . ............................ . ...... . " . . . ........ .. . . . . :

US-PGPUB; ... OR ............. OFF 2011/03/11USPAT; USOCR; 10:48FPRS; EPO;JPO;DERWENT;IBM TDB

:: -- 1i! :,' "

'i

............- " . ............. ....."""""" ................

EAST Search History

:(print$4 or (image or dataor picture) near4 (formulat$4 or form$4 or print$4 oroutput$4 or unit or deviceor apparatus)) with(thumb adj drive or disc ordisk or dvd or flash adjdrive or cd or memory adjcard or external adj (fufferor stor$4 or memory orregister)) with (display or:monitor or Icd or screen or:window) with (indicat$4or select$4 or keyboard orkeypad or designat$4 orchoose or choice or enter).with (file or document or:sheet or address oridentification or identify orname)near3 (name or:paper or file or job ordocument or sheet) and:hatori.inv.

.. . .. . . . . .. . . . . .:: . ... ...........: .... ............. ........................... . ...

12 "1085740"

_US-PGPUB;USPAT; USOCR;FPRS;EPO;JPoDERWENT;IBM_TDB

S43... 0..

....................... .. ................................ ............................... ................................. ....... ,,................................. .....

6612 (software or program or US-PGPUB; " ORinstruction) near4 USPAT; USOCR;(simplex or duplex or color :FPRS; EPO;or hue or monochrom$4 JPO;or (header near3 set) or DERWENT;sortS4 or ((print or IBM_TDB

:record) near2 set$4) or((print$4 or recorder orrecording or recorded orrecord) near3 (id oridentification or name)) orstaple or sortS5 or ((paperor page or sheet) near2(type or kind ior class)))and (switch$5 ior selected

'. 'or selector or selecting or:choose or choosing orchosen or designating ordesignated ior designatoror designation or key orkeying) near3 (simplex orduplex or color or hue ortincture or tint orchromatic or (headernear2 set) or sortS4 oradf) and@ad<"20071127"



'iS44

S45

OR OFF

OFF

..... ..... ..... ..... .............. .2011/03/1316:36

2011/03/1401:15

LIIII\~~~~ ~~. EII\\~\ . ... .l ... ......... ~

.................. ................ ..... . .... ..... .. . .. ........ ........ ........ .......

i

i

.r .................. .................i ;

O R ............................ :;O F......... ;2011 /03/ 1 ...........

i

VI 1 ;iVl 1 "LV 1 1/ VV/ 1 1

: n.nn

:;

i

i

i

i

i

i

i

i

X12:08

.t

.....,

i

... .. .. .7 ... .. ... .... . .. ... ..: : --- -- --- - -- -:

i

i

EAST Search History

S46 . 2,

847

21904 (eeprom or ram or rom orstore or flash adj drive or:mrmory or register orbuffer or storage or cahseor disk or disc or dvd orcd) near4 (address ornumber or packetS2 orprotocoal or protocol ornetwork adj (address orformat or version orlanguage or number) orbuad adj (rate or speed))same (eeprom or ram orrom or store or flash adjdrive or mrmory orregister or buffer orstorage or cahse or disk ordisc or dvd or cd) near4( text or image or data orfile or picture or pictorialor video or audio)

17328 (eeprom or ram or rom orstore or flash adj drive or.:mrmory or register orbuffer or storage or cahseor disk or disc or dvd orcd) near4 (address ornumber or packet$2 orprotocoal or protocol ornetwork adj (address orformat or version orlanguage or number) orbaud adj (rate or speed))same (eeprom or ram orrom or store or flash adjdrive or mrmory orregister or buffer orstorage or cahse or disk ordisc or dvd or cd) near4( text or image or data orfile or picture or pictorialor video or audio) and(delet$4 or eras$5 orcancel$7 or zero$4 or blot$7 or modification ormodifying or abat$5 ordecreas$$ or transferring)near4 (address or numberor packet$2 or protocal orprotocol or network adj(address or format orversion or language ornumber)) near7 (eepromor ram or rom or store orflash adj drive or memory:or register or buffer or::storage or cahse or disk or

USPAT; USOCR;FPRS; EPO;,JPO;DERWENT;,I BM_TDB

~US-PGPUB,USPAT; USOCR;FPRS; EPO;JPO;DERW ENT;

iBMTDB

ile:///CI/Documents%20and%20Settings/jgrantIMy%20Doc...1353/EASTSearchHistory.11231353_AccessibleVersion.htm (16 of 26)4/11/11 12:42:38 PM

OFF 1!2011/04/0121:31

. ,,,,,,,,,,,,,,..,,,..,,..,,.,,,,,,,,,,,.,,,,,,,,..,,,,...,............ ,,.., ...............

vn

i

i

OR

1OFF"""""""""'

........... ................ .2011/04/0121:36

. ....................

a

....................."""~~""~"~""'OFF

:;i

i

i

:,i

EAST Search History

disc or dvd or cd) and@ad< "20080402".. ..... ...................................... ............ ....... ...... ............... .............. i

(eeprom or ram or rom or US-PGPUB;store or flash adj drive or USPAT; USOCR;memory or register or FPRS; EPO;buffer or storage or cashe JPO;or disk or disc or dvd or DERWENT;cd) near4 (address or IBM_TDBnumber or packet$2 orprotocal or, protocol ornetwork adj (addressorformat or version orlanguage or number) orbaud adj (rate or speed)):with (eeprom or ram orrom or store or flash adjdrive or memory or .register or buffer orstorage or cashe or disk ordisc or dvd or cd) near4( text or image or data orfile or picture or pictorialor video or audio) and(delet$4 or eras$5 orcancel$7 or zero$4 or blot$7 or abat$5 or decreas$$or transferring) near4(address or number orpacket$2 or protocal orprotocol or network adj(address or format orversion or language ornumber)) near7 (eepromor ram or rom or store orflash adj drive or memoryor register or buffer or

:storage or cashe or disk orIdisc or dvd or cd) and@ad< "20080402". . . . .. . . . .. . . . . .. . . . . . .: :. . . .. . . . . .. . . . . . .. . . . . .. . . . . . .. . . . . .. . . . .. . . . . .. . . . . . .. . . . . .. ...:: .. . . . . . .. . . . . .. . . . . . . . . . . .. . . . .. . . . . .. ;.

17185 (eeprom or ram or rom or US-PGPUB;store or flash adj drive,or USPAT; USOCR;memory or register or FPRS; EPO;buffer or storage or cashe JPO;or disk or disc or dvd or DERWENT;cd) near4 (address or IBM_TDBnumber or packet$2 orprotocal or protocol ornetwork adj (address orformat or version orlanguage or number) orbaud adj (rate or speed))near8 (eeprom or ram orrom or store or flash adj

. drive or memory orregister or buffer orstorage or cashe or disk or

: disc or dvd or cd) near4

:OFF:nR

OR

2011 /04/01:;21:40

OFF 2011/04/0121:46

file:///C/Documents%2and%2Settings/jgrant/My%20Doc...1353/EASTSearchHisory11231353_AccessibleVersion.htm (17 of 26)4/11/11 12:42:38 PM

S49

.........................................................................................................

:;:

;vri

i

;,,.

:;

,:;

:;

:_';

:;:;

i

r

i

i

i

i

. v

i

i

i

i

i

i

i

i

i

i

i

i

;

i"

i

i

i

i

i

i

i

i

? ..................... ........................

: :

EAST Search History

( text or image or data orfile or picture or pictorialor video or audio) and(delet$4 or eras$5 orcancel$7 or zero$4 or blot$7 or abat$5 or decreas$4or transferring) near4(address or number orpacket$2 or protocal or:protocol or network adj(address or format or

:version or language ornumber)) near7 (eeprom

:or ram or rom or store orflash adj drive or memory:or register or buffer or:storage or cashe or disk or:disc or dvd or cd) andS@ad<"20080402"

S50 ;8166 :(eeprom or ram or rom or:store or flash adj drive or:memory or register or:buffer or storage or cashe:or disk or disc or dvd orcd) near3 (address ornumber or packet$2 orprotocal or protocol or

:network adj (address orformat or version or:language or number) or

:baud adj (rate or speed))near8 (eeprom or ram orrom or store or flash adj

:drive or memory orregister or buffer or

:storage or cashe or disk or'disc or dvd or cd) near2(text or image or data orfile or picture or pictorial:or video or audio) and(delet$4 or eras$5 or

:cancel$7 or zero$4 or blot$7 or abat$5 or decreas$4:or transferring) near3(address or number orpacket$2 or protocal or:protocol or network adj(address or format or

:version or language ornumber)) near3 (eepromor ram or rom or store orflash adj drive or memory

:or register or buffer or:storage or cashe or disk or:disc or dvd or cd) and@ad< 20080402"

IUS-PGPUB;USPAT; USOCR;FPRS; EPO;

RJPO;DERWENT;IBM_TDB

fie:///C/Documents%20and%20Setings/jgrant/My%20Doc...1353/EASTSearchHistory.11231353_AccessibleVersion.htm (18 of 26)4/11/11 12:42:38 PM

OFF i

;

:;,;c

.. .. . .. . .. .. ._ .... .1........................ ...

3

2011/04/0121:53

........... ..

.;

. .

EAST Search History

!51.

(eeprom or ram or rom orstore or flash adj drive ormemory.or register orbuffer or storage or casheor disk or disc or dvd orcd) near3 (address ornumber or packet$2 orprotocal or protocol or

:network adj (address orformat or version orlanguage or number) orbaud adj (rate or speed))near5 (eeprom or ram orrom or store or flash adj'drive or memory orregister or buffer orstorage or cashe or disk or

:disc or dvd or cd) near2(text or image or data orfile or picture or pictorial

':JSPAT tUSOCR: _'15i2589


:

;w i

S52 2153

:;

:;

n~~~\_~~__~~~__~_~__~_\\\~ __~__~~_

................. : . . . .. . . .. .. . . . .... . . .. . . . . .. .. .. . . . . . .. . . . .. . . . ... . . . . .. . . . .. . . . . .. . .. . . . .. .. .

i

(eeprom or ram or rom orstore or flash adj drive ormemory or register orbuffer or storage or casheor disk or disc or dvd orcd) near3 (addressornumber or packet$2 or

:protocal or protocol ornetwork adj (address orformat or version orlanguage or number) orbaud adj (rate or speed))near5 (eeprom or ram orrom or store or flash adj

:drive or memory orregister or buffer orstorage or cashe or disk ordisc or dvd or cd) near2(text or image or data orfile or picture or pictorialor video or audio)same(delet$4 or eras$5 orcancel$7 or zero$4 or blot$7 or abat$5 or decreas$4or transferring) near3(address or number orpacket$2 or protocal orprotocol or network adj(address or format orversion or language ornumber)) near3 (eepromor ram or romor store orflash adj drive or memoryor register or buffer orstorage or cashe or disk ordisc or dvd or cd) and:@ad< "20080402"

... : ;.. ............................ .......... . .. . .. . .. . .

: .. .. .. ..

FPRS; EPO;=.JPO;DERWENT;IBM TDB j

US-PGPUB; OR OFFUSPAT; USOCR;FPRS; EPO;JPO;.DERWENT;IBMTDB

.l

....... 5 )'" .........2011/04/0122:05

EAST Search History

or video) with (delet$4 oreras$5 or cancel$5 or zero$4 or blot$5 or abat$5 ordecreas$4 or transferring).near3 (address or numberor packet$2 or protocal orprotocol or network adj(address or format orversion or language ornumber)) near3 (eepromor ram or rom or store orflash adj drive or memoryor register or buffer orstorage or cashe or disk or

.disc or dvd or cd) and@ad< "20080402"

~S53... iUS-PGPUBI;

.USPAT; USOCR;FPRS; EPO;JPO;DERWENT;IBMTDB


(eeprom or ram or rom orstore or flash adj drive or:memory or register orbuffer or storage or casheor disk or disc or dvd orcd) near3 (address ornumber or packet$2 orprotocal or protocol ornetwork adj (address orformat or version orlanguage or number) orbaud adj (rate or speed))near5 (eeprom or ram orrom or store or flash adj

:drive or memory orregister or buffer orstorage or cashe or disk ordisc or dvd or cd) near2(text or image or data orfile or picture or pictorialor video) near6 (delet$4or eras$5 or cancel$5 orzero$4 or blot$5 or abat$5or decreas$4 ortransferring) near3(address or number orpacket$2 or protocal orprotocol or network adj(addressor format orversion or language ornumber)) near3 (eepromor ram or rom or store orflash adj drive or memoryor register or buffer orstorage or cashe or disk ordisc or dvd or cd) and

S@ad<o"20080402"

3853 O R .............................

i

ii

i

i

i,

i

i

i

i

i

i

i

i

i

,i

i

;OFF 2011/04/0122:08

~i

: :

EAST Search History

;

:_

:;

:;

:;

:_

:;

S54 3853 (eeprom or ram or rom orUS-PGPUB;:store or flash adj drive or USPAT; USOCR;:memory or register or FPRS; EPO;buffer or storage or cashe JPO;:or disk or disc or dvd or :DERWENT;cd) near3 (address or IBMTDBnumber or packet$2 orprotocal or protocol ornetwork adj (address orformat or version orlanguage or number) orbaud adj (rate or speed))near5 (eeprom or ram orrom or store or flash adj

:drive or memory orregister or buffer orstorage or cashe or disk ordisc or dvd or cd) near2(text or image or data orfile or picture or pictorialor video) near6 (delet$4:or eras$5 or cancel$5 or:zero$4 or blot$5 or abat$5:or decreas$4 ortransferring) near3(address or number orpacket$2 or protocal orprotocol or network adj(address or format orversion or language ornumber)) near3 (eepromor ram or rom or store orflash adj drive or memoryor register or buffer or:storage or cashe or disk ordisc or dvd or cd) and@ad< "20080402"

S55 1936 (eeprom or ram'or rom or US-PGPUB;store or flash adj drive or USPAT; USOCR;memory or register or FPRS; EPO;buffer or storage or cashe JPO;or disk or disc or dvd or DERWENT;cd) near3 (address or IBMTDBnumber or packet$2 orprotocal or protocol or

:network adj (address orformat or version orlanguage or number) or

:baud adj (rate or speed))near5 (eeprom or ram orrom or store or flash adjdrive or memory orregister or buffer or

:storage or cashe or disk or:disc or dvd or cd) near2(text or image or data or

Ifile or picture or pictorial

file:///Cl/Documents%20and%20Settingsjgrant/My%20Doc...1353/EASTSearchHistory. 11231353_AccessibleVersion.htm (21 of 26)4/11/11 12:42:39 PM

O R ............................ O FF ......... 2011/04/01............22:13

OR OFF '2011/04/0122:22

i ::!

EAST Search H istory

or video) with (delet$4 oreras$5 or cancel$5 or zero$4 or blot$5 ortransferring) near3(address or number orpacket$2 or protocal orprotocol or network adj(address or format orversion or language ornumber)) near3 (eepromor ram or rom or store orflash adj drive or memoryor register or buffer or:storage or cashe or disk or,disc or dvd or cd) and@ad< "20080402"

S56

S56

US- PGPUB;

USPAT; USOCR;FPRS; EPO;JPo;

DERWENT;IBMTDB

OFF 2011/04/0122:28

file:///CI/Documents%20and%20Setings/jgrant/My%20Doc...1353/EASTSearchHistory.11231353_AccessibleVersion.htin (22 of 26)4/11/11 12:42:39 PM

1975 (eeprom or ram or rom orstore or flash adj drive ormemory or register orbuffer or storage or casheor disk or disc or dvd orcd) near3 (address ornumber or packet$2 orprotocal or protocol ornetwork adj (address orformat or version orlanguage or number) orbaud adj (rate or speed))near5 (eeprom or ram orrom or store or flash adjdrive or memory orregister or buffer orstorage or cashe or disk ordisc or dvd or cd) near2(text or image or data orfile or picture or pictorialor video) and (control$4or process$4 or select$4or switch$4 or key$4 ordesignat$5) near4 (delet$4 or eras$5 or cancel$5or zero$4 or blot$5 ortransferring) near4(address or number orpacket$2 or protocal orprotocol or network adj(address or format orversion or language ornumber)) near4 (eepromor ram or rom or store orflash adj drive or memory:or register or buffer orstorage or cashe or disk or

:disc or dvd or cd) and@ ad.......... '2 0 0 8 0 4 0 2"............. . . ........... ............

........................... .

I I

i

i

i

i

i

i

i

i

i;'

i

i

ii

i

i

i

i

;'

;

i

i

i

ia

i

i

i

i

.. .................... :...............................................................

EAST Search History

S57

....................... ,. .................... ..........................................

4 (choic$4 or keying orselect$4 or chos$4 ordiscriminat$4 or interpret$4 or predict$4 orapproximat$4 or adjust$4)near4 (language or fomrator page adj descript$4 orpdl or descript$4 near4language or page adj job

S or printer adj job or jobadj language or printeradj job) with (stitch orfailure or error oraberration or noise orincompatible) near4

;(language or fomrat or:page adj descript$4 or pd!or descript$4 near4

:,,

:;

:;

i

2

t

................ .................... .... . . . ........................ .


ii!

S58

i590 (eeprom or ram or rom. or:store or flash adj drive ormemory orregister or .buffer or storage or casheor disk or disc or dvd orod) near3 (address or:number or packet$2 orprotocal or protocol or:network adj (address orformat or version orlanguage or number) orbaud adj (rate or speed))near5 (eeprom or ram orrom or store or flash.adjdrive or memory orregister or buffer orstorage or cashe or disk or:disc or dvd or cd) near2(text or image or data orfile or picture or pictorialor video) same (control$4or process$4 or select$4or switch$4 or key$4 or

:designat$5) near4 (delet$4 or eras$5 or cancel$5or zero$4 or blot$5 ortransferring) near4(address or number orpacket$2 or protocal orprotocol or network adj(address or format orversion or language or.number)) near4 (eepromor ram or rom or store orflash adj drive or memoryor register or buffer or

:storage or cashe or disk ordisc or dvd or cd) and@ad< '"20080402"

L.


:.....................

OR OFF 2011/04/0122:31

:.......... ....... ,; ....

OR OFF 2011/04/06

21:48

:I3

US-PGPUB;USPAT; USOCR;FPRS; EPO;JPO;DERWENT;IBM TDB

..

33

:

3:i

' i

i

i

EAST Search History

language or page adj jobor printer adj job or jobadj language or printeradj job) with (choic$4 orkeying or select$4 or chos$4 or discriminat$4 orinterpret$4 or predict$4 orapproximat$4 or adjust$4)near4 (rank or order orpriority or preference or:position or list or type orkind) near7 (language orfomrat or page adj descript$4 or pdl or descript$4near4 language or pageadj job or printer adj jobor job adj language orprinter adj job) and.@ad< "20080723"

0 :(choic$4 or keying orselect$4 or chos$4 ordiscriminat$4 or interpret$4 or predict$4 or,approximat$4 or adjust$4)near6 (language or formator page adj descript$4 orpdl or descript$4) near6,(language or page adj jobor printer adj job or jobadj language or printeradj job) same (stitch orfailure or error oraberration or noise orincompatible) near6(language or fomrat orpage adj descript$4 or pdlor descript$4) near6(language or page adj jobor printer adj job or jobadj language or printeradj job) with (choic$4,orkeying or select$4 or chos$4 or discriminat$4 orinterpret$4 or predict$4 orapproximat$4 or adjust$4)near5 (rank or order orpriority or preference or:position or list or type orkind) near7 (language orformat or page adj descript$4 or pdl or descript$4)near4 (language or pageadj job or printer adj jobor job.adj language orprinter adj job) and:@ad< "20080723"

....... . . . . . .. . . . . . .. . . . . . .. . . . . .. . . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . . . .. .

US-PGPUB;USPAT; USOCR;FPRS; EPO;JPO;DERWENT;IBM_TDB

file:///CI/Documents%20and%20Settings/jgrant/My%20Doc...1353/EASTSearchistory.11231353_AccessibleVersion.htm (24 of 26)4/11/11 12:42:39 PM

S59 OFF:............ 8"..... .....2011/04/0621:53

................. , . ... . . . . . . . . .. . . . . .. . . .. . . . . .. . . . . . . . . .. . . . . .. . . . .. . .. . . . . . . . .. . . . .. . . . .. . . .. .. . . . . ... . . . .. . .. . . . .. . . . . . . . . ... .. . . . . . .. . . . .. .

t

. .. . . . . . . .. . . . . . . . .. . . . . . . . .. . . .

EAST Search History

........ii[6 ......... .......i[ i o..or .,e .... r...............:S60 0 (choic$4 or keying orselect$4 or chos$4 ordiscriminat$4 or interpret$4 or predict$4 orapproximat$4 or adjust$4)

:near6 (language or formator page adj descript$4 orpdl or descript$4) near6(language or page adj jobor printer adj job or jobadj language or printer

:adj job) and (stitch orfailure or error oraberration or noise orincompatible) near6(language or format or

:page adj descript$4 or pdlor descript$4) near6(language or page adj jobor printer adj job or job

:adj language or printer:adj job) with (choic$4 or,keying or select$4 or chos$4 or discriminat$4 orinterpret$4 or predict$4 orapproximat$4 or adjust$4)

:near5 (rank or order orpriority or preference or

:position or list or type or:kind) near7 (language orformat or page adj descript$4 or pdl or descript$4)near4 (language or page

:adj job or printer adj job:or job adj language or"printer adj job) and:@aid< "20080723"---.. .. ..... .. .. .. ---,,. .: -.. ..... ..... ... ... .... .... ... ...... ..... ... ....... .. ..... ..............

(choic$4 or keying orselect$4 or chos$4 ordiscriminat$4 or interpret$4 or predict$4 orapproximat$4 or adjust$4):with (language or fomrator page adj descript$4 orpdl or descript$4 near4language or page adj jobor printer.adj job or jobadj language or printeradj job) with (stitch orfailure or.error oraberration or noise orincompatible) with(language or fomrat orpage adj descript$4 or pdlor descript$4 near4language or page adj jobor printer adj job or job

I

US- PGPUB;USPAT; USFPRS; EPOJPO;DERWENT;IBMTDB

OCR;

C. . '3 n9nh1i f Ai flg

21: 55US-PGPUB;USPAT; USOCR;FPRS; EPO;JPO;DERWENT;IBM_TDB

.

'Z

'Z

't

'Z

file:///CI/Documents%20and%20SettingsljgrntlMy%20Doc...1353/EASTSearchHistory. 11231353_AccessibleVersion.htm (25 of 26)4/11/11 12:42:39 PM

S61 OFF""" ~"""""" ~""""""""" .. .. . . . . . . .. . . . . . . . .. . . . . . . .~"""""""""""""""' """"""""""""""""""""

i

:;

,.i

i

:,

i

Z

,.

.

t

:,

:;

2011/04/0919:33

174

adj language or printeradj job) with (choic$4 orkeying or select$4 or chos$4 or discriminat$4 orinterpret$4 or predict$4 orapproximat$4 or adjust$4)with(rank or order or;priority or preference orposition or list or type orkind) with (language orfomrat or page adj descript$4 or pdl or descript$4near4 language or pageadj job or printer adj jobor job adj language orprinter adj job) and@ad< "20080723"...........: .................... ........ i d < , 0 8 ... ...... .......... ........

4/11/11 12:41:57 PMH:\ East\ Workspaces\ 11955924.wsp

file:///CI/Documents%20and%20Setings/jgrant/My%20Doc...1 353/EASTSearchHistory. 11231353_AccessibleVersion.htm (26 of 26)4/11/11 12:42:39 PM

EAST Search History

.. ........ .. . . . . . . . . . .. .. . . . . . . . .. . . . . ~mrr~rmrmrrrr . . . . . . . . . . . . .. . . . . . . .. . . ._ . . . . . . . .. . . .. . . . .. . . . . . . . . .. . . .

11 01 IIIll 111 IllllIl ll i |I I I I 1111|1|III |II IIIIUS007956890B2

(12) United States PatentCheng et al.

(10) Patent No.:(45) Date of Patent:

US 7,956,890 B2Jun. 7, 2011

(54) ADAPTIVE MULTI-MODAL INTEGRATEDBIOMETRIC IDENTIFICATION DETECTIONAND SURVEILLANCE SYSTEMS

(75) Inventors: Ken P. Cheng, Saratoga, CA (US);Edward Y. Chang, Santa Barbara, CA(US); Yuan-Fang Wang, Goleta, CA(US)

(73) Assignee: Proximex Corporation, Sunnyvale, CA(US)

( * ) Notice: Subject to any disclaimer, the term of thispatent is extended or adjusted under 35U.S.C. 154(b) by 1517 days.

(21) Appl. No.: 11/231,353

(22) Filed: Sep. 19, 2005

(65) Prior Publication Data

US 2006/0093190 Al May 4, 2006

(56) References Cited

U.S. PATENT I)OCUMENTS

5,258,837 A5,473,369 A5,479,574 A5,701,398 A5,769,074. A5,835,901 A

11/199312/199512/199512/1997

6/199811/1998

GormleyAbeGlier et al.Glier et al.Barnhill et al.Duvoisin et al.

(Continued)

FOREIGN PATENT DOCUMENTS

JP 02004295798 * 10/2004

(Continued)

OTHER PUBLICATIONS

Goh et al., "Robust Perceptual Color Identification" U.S. Appl. No.11/229,091, ;filed Sep. 16, 2005.Belhumeur, A. et al. (1997). "Eigenfaces vs. Fisherfaces: recognitionusing class specific linear projection", IEEE Transactions on PatternAnalysis and Machine Intelligence 19(7): 711-720.

(Continued)

Primary Examiner - Jerome Grant, II(74) Attorney, Agent, or Firm - Peters Verny, LLP

Related U.S. Application Data

Provisional application No. 60/610,998, filed on Sep.17, 2004.

(51) Int. CI.HO4N 9/47 (2006.01)GO6K 9/00 (2006.01)

(52) U.S. Cl. ......... 348/143; 382/115; 382/119; 382/118

Field of Classification Search .................. 713/186,713/202, 320, 201, 200; 340/506; 358/143,

358/147, 161, 169; 707/103, 4; 382/103,382/209, 276, 277, 289, 291, 293, 294, 295,382/305, 282, 115, 107, 190, 116, 118, 119,382/128; 345/629; 348/143, 151, 155, 161,

348/169; 709/227, 215, 203, 315See application file for complete search history.

(57) ABSTRACT

A surveillance system is provided that includes at least onesensor disposed in a security area of a surveillance region tosense an occurrence of a potential security breach event; aplurality of cameras is disposed in the surveillance region; atleast one camera of the plurality has a view of the security areaand can be configured to automatically gather biometricinformation concerning at least one subject person in thevicinity of the security area in response to the sensing of apotential security breach event; one or more other of theplurality of cameras can be configured to search for the atleast one subject person; a processing system is programmedto produce a subject dossier corresponding to the at least onesubject person to match biometric information of one or morepersons captured by one or more of the other cameras withcorresponding biometric information in the subject dossier.

2 Claims, 8 Drawing Sheets

Wb _

A. Fro ntmg B. S~ drau C. Pfl OOve 0. Stk b torTmea E. a o

FacialAgent

HNormoffd Face MMgeNonna~zod poe CuveNoimasned Cd= & Tae

A. Raw FIhgewit B. Eieadad Feipip"lme

Pingrpinrt , hparpz SignatureAgent In PEG-? Desciptor

IK0101 Flngeqla5744W eEncoiwo

ProtintexKnowledge Sae.s

FcI snasue inMPEG-7Descrl

ProximoxKnowledge Server

(60)

(58)

r h

US 7,956,890 B2Page 2

U.S. PATENT DOCUMENTS

5,838,4655,912,9805,991,4296,008,9126,072,4966,157,4696,248,0636,306,0876,335,9856,404,9006,408,404

6,591,2246,604,0936,609,1986,625,3156,628,8296,687,8316,697,1036,707,4876,711,5876,757,6686,816,1846,906,7096,909,745

6,944,3196,963,8996,968,0066,970,5136,970,5826,976,2077,023,9137,028,0347,058,2047,116,3537,127,4647,136,5247,187,7837,196,7227,213,1747,242,4237,242,8107,269,5917,277,4857,286,6877,295,2287,319,7967,366,1747,376,2767,382,2497,384,5907,385,5907,456,7277,522,1867,525,5767,602,9477,746,3807,777,783

2001/00487652002/00973222002/01387682002/01901192003/00745902003/01076532003/01154742003/0202 1022004/00011492004/00224422004/0059953

2004/00813382004/01005632004/01176382004/01365742005/00525322005/01002092005/01057642005/01324142005/02220812005/0265607

2006/00178072006/01120392006/02030902006/02211842006/02222092006/02796282006/02849782007/01!464842007/01540882007/01861162008/00431022008/00684642008/00795542008/00887062008/01065972008/01309492008/02185902008/02945882009/03228732010/0002082

AlAlAlAlAlAlAlAlAlAl *Al *AlAlAlAlAlAlAlAl*Al

1/20065/20069/2006

10/200610/2006

12/200612/2006

6/20077/20078/20072/20083/20084/20084/20085/20086/20089/2008

11/200812/20091/2010

Lee et al.Wang et al.Wang et al.Vallone et al.Zhang et al.................. 382/107FlemingGirgensohn et al.Horton et al.Goh et al.Clemmensen et al. ....... 713/186Rowe ct al ................... 348/143Kitagawa et al.BoiceGirgensohn et al.Amini et al.Ivanov et al.Park et al.Morris et al.Reilly et al .................. 348/143Buehler et al.

AA

AAAABIBIBIBIB!BIB!BIB2BIBI *BI1BIB2BIBI*BIBIB!BIBIB1B2B1BIB2B2B2B2BIB2B2B2B2B2B2BIB2B2BlB2B2 *B2B2B2B2B2B2BI *B2BIAlAlAlAlAl*AlAl *AlAlAlAl *AlAlAl*AlAlAlAl*AlAlAl

11/19986/1999

11/199912/19996/2000

12/20006/2001

10/20011/20026/20026/2002

7/20038/20038/20039/20039/20032/20042/20043/20043/20046/2004

11/20046/20056/20059/2005

11/200511/200511/200511/200512/20054/20064/20066/2006

10/200610/200611/20063/20073/20075/20077/20077/20079/2007

10/200710/200711/2007

1/20084/20085/20086/20086/20086/2008

11/20084/20094/2009

10/20096/20108/2010

12/20017/20029/2002

12/20024/20036/20036/2003

10/20031/20042/20043/20044/20045/20046/20047/20043/20055/20055/20056/2005

10/200512/2005

Satou et al.HunkeCoffin et al ................... 382/118Sato et al.Guenter et al.MesthaBarnhill et al.Barnhill et al.Sambonsugi et al.Qian et al.Ladwig

Sullivan et al.Etzion et al.Wood et al.Laumeyer et al.ChasenAlbaugh et al ................... 726/7Fernandez et al.Aman et al.DufauxGoebel et al.Brill et al..................... 348/143Larkin etal.Puri et al.Huang et al.Fernandez et al.Puri et al.Puri et al.LangleyRujAn et al.Monroe et al.Wesinger et al.Hildreth et al.Hobson et al.Wesinger et al.Goh et al.Moon et al.White et al.Dahlquist et al.LinChangWesinger et al.PuriLindwurm et al.Roberts et al.SharpMacFaden et al.Shniberg et al. ............. 382/224FancellaKelly et al.Millar et al.Pinter et al.Arpa et al.Kannermark et al.Lemelson et al ............. 382/116Maruya etal.Chin et al.Yi et al.Monroe et al.Murakami et al.HuffmanFogle et al. .................. 713/320Utsumi et al..Khan et al.......... 713/186Shiota et al.Smith et al.Kim et al.Purnell .................. .... 713/202T'akenakaSablak et al.M onroe ....................... 713/186Kozakaya et al.Elooz et al.Lewis et al.Han et al................... 382/100Bentley et al.Jakobson et al.Chang


WO 2007/044037 Al 4/2007

OTHER PUBLICATIONSBrunelli, R. and D. Falavigna. (1995). "Person identification 'usingmultiple cues," IEEE Transactions on Pattern Analysis and MachineIntelligence 17(10): 955-966.Brunelli, R. et al. (1995). "Automatic Person Recognition by UsingAcoustic and Geometric Features", Machine Vision andApplications8: 317-325.Hong, Lin and Anil K. Jain. (1998). "Integrating faces and finger-prints for personal identification," IEEE Transactions on PatternAnalysis and Machine Intelligence 20(12): 1295-1307.International Search Report mailed on Apr. 2006 for PCT PatentApplication No. PCT/US05/33378 filed on Sep.,19, 2005, one page.Jain, A. K. et al. (1997). "On-Line Fingerprint Verification," IEEETransactions on Pattern Analysis and Machine Intelligence archive19(4): 302-314.Kittler, J. et al. (1998). "On combining classifiers", IEEE Transac-tions on Pattern Analysis and Machine Intelligence 20(3): 226-239.Lu X et al. (2003). "Combing classifiers for face recognition", IEEEInternational Conference on Multimedia Systems and Expo, Balti-more, MD, July.Maio, D. et al. (2002). "FVC2000: fingerprint verification competi-tion", IEEE Transactions on Pattern Analysis and Machine Intelli-gence 24(3): 402-412.Phillips, P.J. et al. (2000). "The FERET evaluation methodology forface-recognition algorithms", IEEE Transactions on Pattern Analysisand Machine Intelligence 22(10): 1090-1104.Senior, A. (2001). "A combination fingerprint classifier", IEEETransactions on Pattern Analysis and Machine Intelligence 23(10):1165 1174.Turk, A. and A. Pentland. (1991). "Eigenfaces for Recognition".Journal of Cognitive Neuroscience 3 (1): 71-86.U.S. Appl. No. 12/838,973, Hu Chin, Multi-Video Navigation, filedJul. 19, 2010.U.S. Appl. No. I11/728,404, Hu Chin, Multi-Video Navigation Sys-tem, filed Mar. 23, 2007.PCT/US05/44656 International Search Report and Written Opinion,Jun. 26, 2006.PCT/US05/43808 International Search Report and Written Opinion,Oct. 10, 2007.PCT/US05/33378 International Search Report and Written Opinion,Apr. 26, 2006.PCT/US05/33750 International Search Report and Written Opinion,May 2, 2007.PCT/US05/16961 International Search Report and Written Opinion,Oct. 17, 2006.VidSys, Inc. Complaint filed in the US District Court for the Easterndistrict of Virginia on Oct. 19, 2010.Redstone Integrated Solutions Documents, 2003.U.S. Appl. No. 95/001,525 Reexamination Request for 7,777,783,filed Jan. 21, 2011.

* cited by examiner

US 7,956,890 B2

.1.1

a

LQ.4w

W)

ci

V

0

V

CU(/)

LL-

U. S. Patent .Jun. 7, 2011 Sheet 1 of 8

US 7,956,890 B2

'. r

E"a0

E0O

w

mlaI

C

00

LL

Jun. 7, 2011 Sheet 2 of 8U. S. Patent

US 7,956,890 B2

L

1...

C')

00

0)

0

C

0.Q

M~

U.S. Patent Jun. 7, 2011 Sheet 3 of 8

US 7,956,890 B2

cL

IKawch 4T

- '-1;f

I1I

*Iiif-T

C

1

IIII q,

LIL

Jun. 7, 2011 Sheet 4 of 8U. S. Patent

ac

c

iii 4 t

I

tim1f"Ma

m

a

U.S. Patent Jun. 7, 2011 Sheet 5 of 8 US 7,956,890 B2

L.

a)

0

1)

iL.

1 .

i

I

II

... .... ... .. .. ... .. .. .. ... .. .. ... .. .. ... .. .. .

U.S. Patent Jun. 7, 2011 Sheet 6 of 8 US 7,956,890 B2

liii LI,

MisLsIl

,S O

'ci

(a)

C10--~i

ti

AV~

m,

cs

M-

,w '

00FF

zz. """.v y

.

i

US 7,956,890 B2

I2

I-

CD00Q

O

gv

ti

W.~ P4G

tic

C

tmfz

Sheet 7 of 8Jun. 7, 2011.U. S. Patent

-- r

U.S. Patent Jun. 7, 2011 Sheet 8 'of 8 US 7,956,890 B2

V.x M

mr

X1.

1 r RI

W fiau ." rOL . oA O el t0

ML

ro

-uw Cool

k

400

7: ^ towo '

d s n/ r A n

nr

IA

r Ki fil

C

a N

CM r

silo%K

n

&woo P

h i

H

n

N1

d

OA

u..

sm

C

y

IIG

f1rffiIfli1"14

I I1@liliEIfll I

ti8tt1t1811111

11 illtilutltII1lIlulsuluI

~1

_C

a

LL

p

C3

L U.dz

LDi

US 7,956,890 B2

ADAPTIVE MULTI-MODAL INTEGRATEDBIOMETRIC IDENTIFICATION DETECTION

AND SURVEILLANCE SYSTEMS

CROSS-REFERENCE TO RELATEDAPPLICATION

The present application claims the benefit of earlier filedprovisional patent application, U.S. Application No. 60/610,998, filed on Sep. 17, 2004, and entitled "Adaptive Multi-Modal Integrated Biometric Identification Detection Sys-tems," which is hereby incorporated by reference as if fullyset forth herein.

B3ACKGROUND OF T"I'HE INVENTION

1. Field of the InventionThe invention relates in general to biometric identification,

and more particularly, to a surveillance system using biomet-ric identification.

2. Brief Description of the Related ArtThe state of the art of applying biometric technologies to

authenticate and positively determine the identity of a personis still faced with several technical challenges. Specifically,the challenges can be categories into two aspects: data acqui-sition and data matching. Data acquisition deals with acquir-ing biometric data from individuals. Data matching dealswith matching biometric data both quickly and accurately.These challenges can beexplained by a port-entry scenario.In such a setting, it is difficult to obtain certain biometric datasuch as DNA and voice samples of individuals. For biometricdata that can be more easily acquired, such as face images andfingerprints, the acquired data quality can vary greatlydepending on acquisition devices, environmental factors(e.g., lighting condition), and individual corporation.Tradeoffs exist between intrusiveness of data collection, datacollection speed, and data quality.

Once after the needed data have been acquired, conductingmatching in a very large database can be very time-consum-ing. It goes without saying that unless a system can acquireand match data both timely and accurately, the system ispractically useless in improving public security, where theinconvenience due to the intrusive data-acquisition processand the time-consuming matching process ought to be mini-mized.

A biometric system typically aims to address either one ofthe following issues: 1) Authentication: is the person the onehe/she claims to be? 2) Recognition: who a person is? In thefirst case, data acquisition is voluntary and matching is donein a one-to-one fashion-matching the acquired data with thedata stored on an ID card or in a database. In the second case,individuals may not be cooperating, and the system mustconduct searches in a very large repository.

The prior art in biometric can be discussed in two parts:single-modal solutions and multi-modal solutions. Severalsystems have been built to use one of the following singlemodal: facial data, voice, fingerprint, iris or DNA. The effec-tiveness of these single-modal approaches can be evaluated inthree metrics: the degree of intrusiveness, speed and accuracy.From the perspective of a user, acquiring face modal can bethe most noninvasive method, when video cameras aremounted in the distance. However, the same conveniencenature often compromises data quality. An intrusive faceacquisition method is to acquire frontal face features, whichrequires corporation from individuals. Voice is another popu-lar modal. However, traditional voice-recognition fails mis-erable when voice samples of multiple individuals are simul[

taneously captured or when background noise exists. Evenwhen the acquired voice data can be "pure," existing signalprocessing and matching techniques can hardly achieve rec-ognition accuracy of more than 50%. The next popular modal

5 is fingerprint, which can achieve much higher recognitionaccuracy at the expense of intrusive data acquisition andtime-consuming data matching. Finally, DNA is by far themost accurate recognition technique, and the accompanyinginconvenience in data acquisition and the computational

10 complexity are both exceedingly high. Summarizing thesingle model approach, non-intrusive data-acquisition tech-niques tend to suffer from low recognition accuracy, andintrusive data-acquisition techniques tend to suffer from longcomputational time

15 As to multimodal techniques, there have been several priorart United States Patents and Patent Applications disclosetechniques. However, as will be further discussed below,these disclosures do not provide scalable means to deal withtradeoffs between non-intrusiveness, speed and accuracy

20 requirements. These disclosures may fix their system con-figuration for a particular application, and cannot adapt toqueries of different requirements and of different applica-tions.

Wood et al. disclose in U.S. Pat. No. 6,609,198 a security25 architecture using the information provided in a single sign-

on in multiple information resources. Instead of using a singleauthentication scheme for all information resources, the secu-rity architecture associates trust-level requirements withinformation resources. Authentication schemes (e.g., those

30 based on passwords, certificates, biometric techniques, smartcards, etc.) are employed depending on the trust-levelrequirement(s) of an information resource (or informationresources) to be accessed. Once credentials have beenobtained for an entity and the entity has been authenticated to

35 a given trust level, access is granted, without the need forSfurther credentials and authentication, to informationresources for which the authenticated trust level is sufficient.The security architecture also allows upgrade of credentialsfor a given session. The credential levels and upgrade scheme

40 may be useful for a log-on session; however, such architectureand method of operations do not provide a resolution for highspeed and high accuracy applications such as passenger secu-rity check in an airport.

Sullivan et al. discloseinU.S. Pat. No. 6,591,224 a method45 and apparatus for providing a standardized measure of accu-

racy of each biometric device in a biometric identity authen-tication system having multiple users. A statistical databaseincludes continually updated values of false acceptance rateand false rejection rate for each combination of user, biomet-

50 ric device and biometric device comparison score. False' acceptance rate data are accumulated each time a user suc-

cessfully accesses the system, by comparing the user's cur-rently obtained biometric data with stored templates of allother users of the same device. Each user is treated as an

55 "impostor" with respect to the other users, and the probabilityof an impostor's obtaining each possible comparison score iscomputed with accumulated data each time a successfulaccess is made to the system. The statistical database alsocontains a false rejection rate, accumulated during .a test

60 phase, for each combination of user, biometric device andbiometric device comparison score. By utilizing a biometricscore normalizer, Sullivan's method and apparatus may beuseful for improving the accuracy of a biometric devicethrough acquiring more training data.

65 Murakami et al. disclose is a Patent Publication 20,020,138,768 entitled "Method for biometric authenticationthrough layering biometric traits," a portable biometric

US 7,953

authentication system having a single technology for measur-ing multiple, varied biological traits to provide individualauthentication based on a combination of biological traits. Atleast one of these biometric traits is a live physiological trait,such as a heartbeat waveform, that is substantially-but notnecessarily completely unique to the population of individu-als. Preferably, at least one of the identifying aspects of thebiological traits is derived from a measurement taken byreflecting light off the subdermal layers of skin tissue. TheMurakami et al. approach is limited by the more intrusivemeasurement techniques to obtain data such as heartbeatwaveform and reflecting light off the subdermal layers of skintissue. These data are not immediately available in a typicalsecurity check situation to compare with the biometric data,e.g., heart beat waveforms and reflection light from subder-mal layers from the skin of a targeted searching object. Fur-thermore, the determination or the filtering of persons' iden-tity may be too time consuming and neither appropriate fornor adaptive to real time applications.

Langley discloses in US Patent Application 20,020,126,881, entitled "Method and system for identity verificationusing multiple simultaneously scanned biometric images," amethod to improve accuracy and speed of biometric identityverification process by use of multiple simultaneous scans ofbiometric features of a user, such as multiple fingerprints,using multiple scanners of smaller size than would be neededto accommodate all of the fingerprints in a single scanner, andusing multiple parallel processors, or a single higher speedprocessor, to process the fingerprint data more efficiently.Obtaining biometric data from multiple user features by useof multiple scanners increases verification accuracy, but with-out the higher cost and slower processing speed that would beincurred if a single large scanner were to be used for improvedaccuracy. The methods according to Langley may provide theadvantages of speed and accuracy improvements. However,the nature of requiring multiple scans makes data acquisitiontime-consuming and intrusive.

On the academia side, much research effort has beengeared toward analyzing data from individual biometricchannels (e.g., voice, face, fingerprint, please see the refer-ence list for a partial list), less emphasis has been placed oncomparing the performance of different approaches or comb-ing information from multiple biometric channels to improveidentification. Some notable exceptions are discussed below.In Hong Lin, JainA. K., Integrating faces and fingerprints forpersonal identification, IEEE Transactions on Pattern Analy-sis and Machine Intelligence, Vol. 20, No. 12, December1998, pp. 1295-1307, the authors report an automated personidentification system that combines face and fingerprintinformation. The face recognition method employed is thetraditional eigen face approach, M. Turk and A. Pentland,Eigenfaces for Recognition, J. Cognitive Neuroscience Vol.3, No. 1, 1991, pp. 71-96, which computes a set oforthonor-mal bases (eigen faces) of the database images using theprincipal component analysis. Face images are then approxi-mated by their projection onto the orthonormal Eigen facebases, and compared using Euclidean distances. For finger-print, the authors extend their previous work, Jain, A. K.; LinHong; Bolle, R.; On-line fingerprint verification, PatternAnalysis and Machine Intelligence, Vol. 19, No. 4, April1997, pp. 302-314, to extract minutiaes from fingerprintimages. They then align two fingerprint images by computingthe transformation (translation and rotation) between them.Minutiaes are strung together into a string representation anda dynamic programming-based algorithm is used to computethe minimum edit distance between the two input fingerprintstrings. Decision fusion is achieved by cross validation of the

top matches identified by the two modules, with matchingresults weighed by their confidence or accuracy levels. Theperformance of the system is validated on a database of about640 face and 640 fingerprint images.

5 In Phillips, Henson Moon; Rive, S E A.; Russ, The FER-RET evaluation methodology for face-recognition algo-rithms, IEEE Transactions on Pattern Analysis and Machine

SIntelligence, Vol. 22, No. 10, October 2000, pp. 1090-1104,the Michigan State University research group extends their

10 information fusion framework to include more modalities. Inparticular, images of a subject's right hand were captured, andfourteen features comprising the lengths of the fingers, widthsof the fingers, and widths ofthe palm at various locations ofthe hand. Euclidean distance metric was used to compare

15 feature vectors. Simple sum rules, decision tree and lineardiscriminant function are used for classification. It isobserved that a personal ID system using three modules out-performs that uses only two of the three modules. While thisis an interesting experiment, the data set used is small and

20 there is no accepted universal standard in using hand imagesin biometrics.

In R. Brunelli, D. Falavigna, T. Poggio and L. Stringa,Automatic Person Recognition by Using Acoustic and Geo-metric Features, Machine Vision and Applications 1995, Vol.

25 8 pp. 317-325, an automated person recognition system usingvoice and face signatures is presented. The speaker recogni-tion subsystem utilizes acoustic parameters (log-energy out-puts and their first-order time derivatives from 24 triangularband-pass filters) computed from the spectrum of short-time

30 Windows of the speech signal. The face recognition sub-system is based on geometric data represented by a vectordescribing discriminant facial features such as positions andwidths of the nose and mouth, chin shape, thickness and shapeof the eyebrows, etc. The system captures static images of the

35 test subjects and the test subjects are also asked to utter tendigits from zero to nine for use in the speaker ID subsystem.Each subsystem then computes the distances of the test sub-ject's speech and face signatures with those stored in thedatabases. Decisions from the two ID modules are combined

40 by computing a joint matching score that is the sum of the twoindividual matching scores, weighted by the correspondingvariance. Experimental results show that integration of visualand acoustic information enhances both performance andreliability of the separate systems. The above system was later

45 improved upon in Brunelli, R.; Falavigna, D., Person identi-fication using multiple cues,. IEEE Transactions on PatternAnalysis and Machine Intelligence, Vol. 17, No. 10, October1995, pp.955-966, where multiple classifiers are used in theface recognition subsystems, and the matching score normal-

5o ization process is made more robust using robust statisticalmethods.

In Kittler, J.; Hatef, M.; Duin, R. P. W.; Matas, J., Oncombining classifiers, IEEE Transactions on PatternAnalysisand Machine Intelligence, Vol. 20, No. 3, March 1998, pp.

55 226-239, a performance study of various ensemble classifi-cation scheme is presented. It is shown that many existingdecision aggregation rules are actually simplifications basedon the more general Bayesian rule. The authors compare theperformance of different decision aggregation rules (max,

60 min, median, and majority voting rule) by performing an. experiment in biometrics. Three modules are used: frontal

faces, face profiles, and voiceprints. Simple correlation-basedand distance-based matching is performed on frontal facesand face profiles, respectively, by finding a geometric trans-

65 formation that minimizes the differences in intensity. It isshown that a simple aggregation scheme by summing theresults from individual classifiers actually perform the best.

US 7,956,890 B25

In Lu X; WangY; and Jain A, Combing classifiers for facerecognition, IEEE International Conference on MultimediaSystems and Expo, Baltimore, Md., July 2003, three well-known appearance-based face recognition methods, namelyPCA, M. Turk and A. Pentland, Eigenfaces for Recognition, 5J. Cognitive Neuroscience Vol. 3, No. 1, 1991, pp. 71-96,ICA, and LDA, Belhumeur, P. N.; Hespanha, J. P.; Kriegman,D. J., Eigenfaces vs. Fisherfaces: recognition using classspecific linear projection, IEEE Transactions on PatternAnalysis and Machine Intelligence, Vol. 19, No. 7, July 1997, topp. 711-720, are used for face image classification. Two com-bination strategies, the sum rule and RBF network, are used tointegrate the outputs from these methods. Experimentalresults show that while individual methods achieve recogni-tion rates between 80% and 88%, the ensemble classifier 15boosts the performance to 90%, using either the sum rule orRBF network. In Senior, A., A combination fingerprint clas-sifier, IEEE Transactions on Pattern Analysis and MachineIntelligence, Vol. 23, No. 10, October 2001, pp. 1165-1174, asimilar multi-classifier scheme, this time for fingerprint clas- 20

sification, is proposed. Hidden Markov Models and decisiontrees are used to recognize ridge structures of the fingerprint.The accuracy of the combination classifier is shown to behigher than that of two state-of-the-art systems tested underthe same condition. These studies represent encouraging 25results that validate our multi-modal approach, though only asingle biometric channel, either face or fingerprint, not acombination of biometric channels, is used in these studies.

Maio, D.; Maltoni, D.; Cappelli, R.; Wayman, J. L.; Jain, A.K., FVC2000: fingerprint verification competition, IEEE 30Transactions on Pattern Analysis and Machine Intelligence,Vol. 24, No.3 , March 2002, pp. 402-412, documents a fin-gerprint verification competition that was carried out in con-junction with the International Conference on Pattern Recog-nition (ICPR) in 2000 (a similar contest was held again in 352002). The aim is to take the first step towards the establish-ment of a common basis to better understand the state-of-the-art and what can be expected from the fingerprint technologyin the future. Over ten participants, including entries fromboth academia and industry, took part. Four different data- 40bases, two created with optical sensors, one with a capacitivesensor, and one synthesized, were used in the validation. Boththe enrollment error (if a training image can be ingested intothe database or not) and the matching error (ifa test image canbe assigned the correct label or not) and the average time of 45enrollment and matching are documented.

A study, that is similar in spirit but compares the perfor-mance of face recognition algorithms, is reported in Phillips,P. J.; Hyeonjoon Moon; Rizvi, S. A.; Rauss, P. J., The FERETevaluation methodology for face-recognition algorithms, 50IEEE Transactions on Pattern Analysis and Machine Intelli-gence, Vol.22, No. 10, October 2000, pp.1 0 9 0 -1 104. A subsetof the Feret database (a gallery of over 3000 images) was usedin the study. Ten different algorithms, using a wide variety oftechniques, such as PCA and Fischer discriminant, were 55tested. Cumulative matching scores as a function of matchingranks in the database are tabulated and used to compare theperformance of different algorithms. This study was repeatedthree times, in August 1994, March 1995, and July 1996.What is significant about this study is that the performance of 60the face recognition algorithms improved over the three tests,while the test condition became more challenging (withincreasingly more images in the test datasets).

As can be seen from the above brief survey, multi-modalbiometrics holds a lot of promise. It is likely that much more 65accurate classification results can be obtained by intelligentlyfusing the results from multiple biometric channels given

performance requirements. While it is important to keep onimproving the accuracy and applicability of individual bio-metric sensors and recognizers, the performance of a biomet-ric system can be boosted significantly by judiciously andintelligently employing and combining mtiltiple biometricchannels.

While there have seen significant research activities insingle- and multi-channel biometry over the past decade, thestate-of-the-art is still wanting in terms of speed and accuracy.Therefore, a need still exists in the art to provide new andimproved methods and system configurations to increase thespeed and accuracy of biometric identity verification anddeterminations such that the above-mentioned difficulties andlimitations may be resolved. The present invention meets thisneed.

SUMMARY OF THE INVENTION

One embodiment of the invention provides'a novel surveil-lance method. An event sensor such as, a camera, chemicalsensor, motion detector, unauthorized door access sensor, forexample, is disposed to sense an occurrence of a potentialsecurity breach event. A camera with a view of the area inwhich an event is sensed gathers biometric information con-cerning a subject person in the vicinity of the event at aboutthe time the event is sensed. A subject dossier is producedcontaining biometric information relating to the subject per-son sensed by the camera with the view of the area. Biometricinformation of persons captured on one or more other surveil-lance cameras in the general vicinity of the event is matchedagainst corresponding biometric information in the subjectdossier.

Another embodiment of the invention provides a new sur-veillance system.A sensor is disposed in a surveillance regionto sense an occurrence of a security breach event. The systemincludes a plurality of cameras. At least one camera of theplurality has a view of the security area and can be configuredto automatically gather biometric information concerning asubject person in the vicinity of an area where the eventoccurred in response to the sensing of the event. One or moreof the plurality of cameras can be configured to search for thesubject person. The surveillance system also includes a pro-cessing system which can be programmed to produce a sub-ject dossier corresponding to the subject person. The process-ing system also can be programmed to match biometricinformation of one or more persons captured by one or moreof the cameras with corresponding biometric information inthe subject dossier.

These and other features and advantages of the inventionsill be apparent from the following description of embodi-ments thereof in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustrative showing a map of an airport pas-senger terminal and its immediate vicinity protected by asurveillance system of one embodiment of the invention andalso showing several pop-up views relating to event alerts inaccordance With the embodiment.

FIG. 2 is another view of the map of FIG. 1 showing zoomto detail maps of different portions of the overall passengerterminal map.

FIG. 3 is an illustrative drawing of example security areaswithin the surveillance region of FIGS. 1-2 outfitted withevent sensors.

US 7,956,890 B27

FIG. 4 is an illustrative block level hardware diagram of a estimatedsurveillance system in accordance with an embodiment of the include teinvention, tion of tra

FIG. 5 is an illustrative block diagram level drawing of a more pensystem architecture of an embodiment of the invention that 5 print, irisincorporates the system hardware of FIG. 4. selected t

FIG. 6 is an illustrative flow diagram showing gathering within thand conversion of facial feature data to a facial feature signa- example.ture. individua

FIG. 7 is an illustrative flow diagram showing gathering 10 weight. Hand conversion of fingerprint feature data to a fingerprint being as rsignature. his jacket

FIG. 8 is an illustrative flow diagram showing gathering person sitand conversion of DNA data to a DNA signature. One and weigembodiment of the invention may employ a DNA fingerprint 15 Systemfor identification purposes. region for

one embcDETAILED DESCRIPTION OF THE PREFERRED one or mo

EMBODIMENT event in t20 center. Th

The following description is presented to enable any per- before theson skilled in the art to make and use the invention, and is tor at theprovided in the context of particular applications and their viduals atrequirements. Various modifications to the preferred embodi- prior to thfments will be readily apparent to those skilled in the art, and 25 at the scethe generic principles defined herein maybe applied to other and trackembodiments and applications without departing from the within thspirit and scope of the invention. Moreover, in the following The sydescription, numerous details are set forth for the purpose of incremenexplanation. However, one of ordinary skill in the art will 30 mation inrealize that the invention can be practiced without the use of ject dossthose specific details. In other instances, well-known struc- weight, v,tures and devices are shown in block diagram from in order such as fanot, to obscure the description of the invention with unnec- be used tessary detail. Thus, the present invention is not intended to be 35 camera'slimited to the embodiments shown, but is to be accorded the cious perwidest scope consistent with the principles and features dis- from a suclosed herein, make a m

dates actuSystem Overview 40 mation c

additionaOne embodiment of the invention involves an intelligent subject d

surveillance system. A plurality of cameras, some with and then maysome without overlapping fields of view, are distributed individuathroughout a surveillance region. Intelligent computer soft- 45ware based agents process information captured by one or.more of the cameras to produce a subject dossier indicative ofthe identity of a person whose images have been captured by One erone or more of the cameras. Information for a subject dossier airport sealso may be gathered through other modalities such as voice 50 compriserecognition, iris scan, or fingerprint, for example. The system passengerincludes multiple event sensors, which may include the cam- outside theras, chemical sensors, infrared sensors, or other security terminal.alarm sensors that trigger an alert, upon sensing an occur- airport parence of a predetermined category of event requiring height- 55 tected byened vigilance. For example, an alarm may be triggered when inventiona locked door is opened without proper access permission or an associwhen an unauthorized person enters a restricted area or when The surva vehicle is parked in a restricted area. More specifically, a ger arrivsubject dossier is produced for individuals in the vicinity of 60 and a tenthe location of an alarm-triggering event. For instance, a view are,subject dossier may be produced for persons captured in a of the pasvideo camera image at or near a door in the surveillance FIG. 2region at about the time when an unauthorized opening of the to detaildoor is detected by an event sensor. 65 terminal

A subject dossier may include soft biometric information, displayecalso referred to as "soft" features such as clothing color, correlate

Sheight and weight. A subject dossier also maymporal information, such as walking speed or direc-vel. In addition, a subject dossier also may includemanent information such as facial features, finger-scan, voiceprint and DNA. Soft features may be

o be especially useful for relocating an individuale surveillance region, especially in a crowd, forFor instance, it maybe relatively easy to identifyls based upon clothing color or estimated height and[owever, soft features have the disadvantage of noteliable or permanent over time. If a person takes off, then an identifying color feature may be lost. If as down, then it may become impossible to use heightht information to pick that person out of a crowd.Ssensors continually monitor the surveillancer the occurrence of one or more suspicious events. Inodiment, the system directs a live video feed fromre cameras having the location ofan alert-triggeringheir field of view to a console in a manned controlhe system also may direct video images captured juste event to the control center console. Thus, an opera-console can observe behavior of suspicious indi-the scene of the event in real time and immediately

he event. A subject dossier produced for individualsne of the event can be used to automatically identifya suspect individual present at the scene of the evente surveillance area after the occurrence of the event.stem may employ information in a subject dossiertally. For instance, the system may prioritize infor-the subject dossier. Certain information in the sub-ier such as clothing color, estimated height andalking pattern or gait and certain key facial featuresacial shape, facial hair, skin color, or hair color mayto make an initial estimate of which persons in afield of view are candidates for a match to a suspi-son identified in response to an alert. Other featuresabject dossier then may be added incrementally toore careful assessment of whether identified candi-ually match the suspect. Alternatively, as more infor-oncemrning a suspicious person becomes available,l features may be added incrementally to a suspect'sossier for that person. This additional informationSbe used to more effectively locate and track thel within the surveillance region.

Surveillance Region

mbodiment of the invention is configured for use incurity. In this embodiment, the surveillance regions an airport passenger terminal and the surroundingr ground transport loading/unloading zone directlyhe terminal and the aircraft parking area adjacent theFIG. 1 is an illustrative drawing of a map of anassenger terminal and its immediate vicinity pro-Sa surveillance system of one embodiment of the.The system includes multiple cameras, each withated field of view, some of which are overlapping.eillance region has multiple areas including passen-al and departure areas, a passenger departure shopsrace. Groups of cameras with overlapping fields ofdeployed to capture images within different regionsssenger arrival and passenger departure areas.is another view of the map of FIG. I showing zoommaps of different portions of the overall passengermap. The illustrative maps of FIGS. 1-2 can be.d on a control terminal so that an operator can easilyan alert to a specific area an airport surveillance

US 7,956,890 B2

region. For instance, if an alert is triggered in the arrivalsregion shown in FIG. 1, then an operator may request theleft-most zoom shown in FIG. 2 in order to quickly picture theairport layout in the vicinity of the alert. Additional zoommaps (not shown) may be provided for numerous locations 5such as security gates, check-in counters, airport fairway,parking area, access entrance, check-in counters, etc. Eachdifferent area may be associated with a group of cameras andevent sensors.

Event sensors are disposed at selected locations within the 10

surveillance region. FIG. 3 is an illustrative drawing ofexample security areas within the surveillance region ofFIGS. 1-2 outfitted with event sensors. A first security areacomprises a door. The door may be equipped with a sensor, 15such as a mechanical sensor, that detects unauthorized open-ing of the door. A second security area comprises a window.The window may be associated with a mechanical sensor thatdetects when the window has been broken. A third securityrepresents a threshold to a restricted area. The restricted area 20may be equipped with motion detectors that detect the pres-ence of persons in a restricted area. Cameras situated through-out the surveillance region also may serve as event sensors.For example, the system may employ a monitoring rulewhereby a camera monitors a particulararea of the passenger 25terminal. Ifa person is loitering in that area, defined by failingto move beyond a 15 foot radius for more than 60 seconds,then a low level alert is declared, the camera zooms in, and theface of the loitering person is matched against the faces ofpersons on a watch list, for example. 30

Landmarks are defined in the security areas for purpose ofestimating height and weight and direction and speed of travelof a suspect individual. For instance, a landmark such as acountertop may be identified, and processing of a cameraimage may be calibrated to estimate a person's height relative 35to the land marked countertop. A group of multiple structures,such as telephone booths, lounge areas, signs or countertops,within a field of view of one or more of a group of camerascovering a security area may be identified. Processing ofcamera images from the group of cameras may be used to 40estimate the direction and speed at which a suspect is movingbased upon the sequence and timing of his passing the landmarked structures.

Although the surveillance region in this one example isdescribed in terms of an airport passenger terminal, it will be 45appreciated that the invention is not restricted to an airportterminal. Moreover, the surveillance region need not be acontinuous local area. Event sensors and surveillance cam-eras may be disposed over disparate areas and be in commu-nication with a control center via a network such as the 50internet, for example.

System Architecture

FIG. 4 is an illustrative block level hardware diagram of a 55surveillance system in accordance with an embodiment of theinvention. The system includes multiple data collectionagents, a knowledge server, a local knowledge server data-base, an application server, a middle-tier database, web serv-ers, a browser based control console and one or more client 60applications such as ComputerAided Dispatch system, build-ing management system, access control system, etc. It shouldbe understood that the various components shown are merelyillustrative. Each agent may gather information from numer-ous sources, such as the cameras shown in FIG. 1, distributed 65throughout a surveillance region. Moreover, for example, theknowledge server and the application server can be imple-

10"mented across'multiple hardware systems or as different pro-cesses within a single hardware system.

A security agent is a process that spans many tasks tocollect information about subject(s). For example, a securityagent may spawn multiple data collection agents include afacial features, fingerprint, DNA, clothing color, subject gait,subject height and weight, skin color/tone, hair color/tone,subject direction and voiceprint, for example. Each data col-lection. task produces different information about an indi-vidual. More specifically, each produces a signature indica-tive of some identifying aspect of a person under surveillance.For instance, a facial features agent uses facial informationcaptured by one or more cameras to produce a signatureindicative of an individual's facial features. 'Similarly, forexample, a clothing color agent uses clothing color informa-tion captured by one or more cameras to produce a signatureindicative of the color of an individual's clothing color. Thus,the multiple agents can produce multiple different signatures,each indicative of one or more different identifying feature ofan individual.

The agents provide the signatures to the knowledge server,which aggregates signatures for each given person undersurveillance into a subject dossier for that person. The knowl-edge server indexes the signatures within a given subjectdossier to permit incremental searches for individuals withinthe search region. The knowledge server also may performclassification and matching. The local knowledge server data-base stores the digital signatures and corresponding indexinginformation.

The web services is the component that provides the inter-faces via Web Server which is usually part of an operatingsystem. For example, web services provides the interfaces forour internal components or external systems via Web Server(such as Microsoft IIS on Windows, orApache on Linux). Allthe interfaces to the system are via HTTP or HTTPS usingport 80. Doing so, our system can run across firewall. Basi-cally, the Web Services componentjust exposes our systeminterface to the outside world via Web Server.

The application server is the component that provides thatdatabase access to the user interface component, and per-forms session management which includes authenticationand authorization. The middle-tier database serves as thelocal database for the application server.

FIG. 5 is an illustrative block diagram level drawing of asystem architecture of an embodiment of the invention thatincorporates the system hardware of FIG. 4. A user.interface(UI) provides an operator of the system with real-time.infor-mation concerning alert events within the surveillance region.The UI may provide maps of the entire surveillance region,including zoom maps. It can display alerts from differentsensors including cameras, digital video recorders, accesscontrol, bio-chemical detectors, etc. It may display videos ofa security area in which an alert has been triggered, detailedimages of suspect individuals and details of one ormore alertsthat have been triggered.

Referring again to FIG. 1, there is show an example of a UIdisplay screen in with pop-up display showing variousimages relating to one or more alerts. In the center of thescreen is map of a surveillance region. The operator can beselectively enlarge, minimize or close each pop-up. A VideoReview display provides a video image of the security regionat about the time of an alert. An incident Detection displayprovides detailed information concerning an alert event. Inthis example, the alert event involved an individual tailgatingat a commuter door. A Suspect Description display provides

US 7,956,890 B2

identifying information concerning an individual under sur-veillance based upon information gathered into a subjectdossier produced for the person. A Detailed Images displayprovides pictures of a suspect individual captured by one ormore surveillance cameras. A Potential Identification displayprovides images of the suspect together with images of one ormore people whose facial features closely match those of thesuspect. The potential matches are based upon a facial featuresignature provided by the facial feature agent. Across thebottom of the map, there is a chart listing briefly summarizingmultiple alert situations. The operator may selectively accesspop-up screens for these alert situation.

Thus, the UI advantageously displays a variety of informa-tion aggregated in response to one or more alerts. In a typicalairport security region, for example, there may be severalhundred cameras dispersed throughout a large physical area.Moreover, there may be only a few operators monitoring oneor more UI consoles. Depending upon the rules for monitor-ing and declaring alerts, alerts may occur frequently or infre-quently. The UI of one embodiment of the invention directs anoperator to areas of a surveillance region that are subject toalert and provides pertinent information concerning the alertso that the operator can efficiently manage security from acontrol center. The UI also allows an operator to quicklyinvestigate and simultaneously keep abreast of multiple alertevents.

Furthermore, as explained more fully below, informationfrom different sensing devices is correlated to facilitate track-ing of a suspect within a security region. For instance, softbiometric information and temporal information is used tolocate a suspect as he or she travels within the security region.In one embodiment, a dashed line can be produced on a maponthe display showing a path followed by a suspect within thesurveillance region. Information from different data collec-tion agents may be fused in order to more accurately identifyand track an individual. Therefore, the operator can use the UIto evaluate an alert event, to identify and track a suspect. Theoperator may use this information as a basis to send informa-tion to a responder to intercede or deal with an alert incident.

Knowledge Services are implemented as an applicationrunning on the knowledge server. Knowledge Services cor-relate and analyze signature information provided by differ-ent sensory devices (i.e., data gathering agents). The Knowl-edge Services assemble and index subject dossiers, and whenappropriate, fuse signature information for improved classi-fication results. The Knowledge Services also generate, acti-vate or deactivate rules and send/control rules and instructionto the Rules and Agent Manager.

The Rules and Agent Manager also is implemented on theknowledge server. The Rules andAgent Manager manages allother agents and manages rules that can be sent to each agent.It correlates information from agents. It can also escalate analert if the alert is not acknowledged by an operator within agiven timeframe and/or similar alerts happen repeatedlywithin a given time span (e.g. within 2 hours). Both theKnowledge Service and the Rules and Agent Manager aretheprimary components for aggregating, categorizing biometricsignatures which are parts of object dossiers. It also performsother tasks such as task assignment/tracking, load balancingtasks among agents, and interacting with data access compo-nents.

The following are examples of rules that may be imple-mented by the system.

Rules:

Actor Action5 Person Walk though lane against direction of traffic

Person TailgatingPerson LoiteringPerson PiggybackPerson Traveler screeningPerson Walk in restricted area

10 Vehicle Park overtimeVehicle Park in restricted area

15The Person-Loitering rule involves the following criteria:

Radius 15 footDuration 20 secondsAlert Severity Low

20 Response Zoom in face to match "watch list"

25 The Person-Tailgating Rule involves the following criteria:

Alert Severity LowResponse Zoom in face to match watch list"

30

The correlation Monitoring Rule for the occurrence of aPerson-Loitering event AND a Person-Tailgating event

involving the same person is as follows:

35

Alert Severity CriticalAction Acknowledge Loitering and Tailgating alerts and deliver

alarm to operator console

40 As described above the UI, may display several categoriesof information concerning an alert. The Knowledge Serviceand the Rules and Agent Manager provide the correlationbetween events and data sources and subject dossiers thatpermit an operator to view a map of the location of an alert,

45 soft-biometric data of a suspect and video playback, forexample. More particularly, these components provide a linkto a map and zoom stored in the middle tier database, link tovideo feeds for video view real-time monitoring or playback

5 ofrecordedvideo clips and stored in a Digital Video Recordersystem and provide the subject dossier information.

The Middle Tier Data Access runs on the applicationserver. It controls the database including functions such asquery, add, delete, index. Indexing biometric signatures and

55 updating subject dossiers are done by this component.A (Security) Agent is implemented as an application run-

ning on the knowledge server that controls and manages thedata gathering sensors. In the case of cameras or DVRs, it canalso perform video analytic using Computer Vision technol-

60 ogy. Those tasks include background subtraction, image sta-bilization, object detection, object classification, object track-ing, and object identification. It can also control themovement of Pan/Tilt/Zoom (PTZ) cameras, manage areas ofinterest within the field of view of the camera (called Mouse/

65 Man Trap), and collect video streams from DVR or cameras.It also has a scheduler that controls when rules or videoanalytic are performed.

US 7,956,890 B2

A Sensory Device Directory Access and Video Server isimplemented as an application that has access to the knowl-edge server manages and provides information regardirig sen-sor devices or other subsystems. Basically, it is a softwarelayer that enables the overall system to handle differentmakes/models of sensor devices.

The Web Services is the component provided by operatingsystems or web servers. It manages other components,spawns or deletes services as necessary. It can also listen tomessages from other systems. The Web Services providesinterfaces to the system via Web Services running as part of aWeb Server. The system provides a library resided on a spe-cific directory, and the Web Server (which is usually part ofthe operating system) will use it to interpret interface requeststo our system.

Tracking, Facial Recognition, Fingerprint recognition, andother biometric identification are done at the (Security)agents. Biometric signatures are collected and generated atthe agents, and sent to the Rules-and-Agent Manger. TheKnowledge Services and the Rule-and-Agent Manager col-lectively collect biometric signatures and object trackinglocations, and then generate and manage subject dossiers. Adescribed above, a subject dossier includes information aboutobject (e.g., person) such as, biometric information/signa-tures, soft biometric information (hair color, skin tone/color,weight or build, height, etc.) and other temporal information(e.g., speed, direction, location, past activities, informationthat the operator is looking for, etc.). Data fusion is performedby the Knowledge Services and the Rules and Agent Man-ager. Data required or generated by each of the componentsare saved and retrievable via the Middle-tier/Data Accesscomponent, which in turn utilizes a relational database suchas Microsoft SQL Server.

Subject Dossier

Data gathering agents collect data concerning a subjectperson from different sources. The Knowledge Servicesaggregate the data into subject dossier. The data aggregatedinto a given dossier may include different digital signaturesproduced by different data gathering agents. A subject dossieralso may include fused data signatures produced by the fusionof data gathered from multiple data sources having differentdata modalities.

The following is an example of information in a subjectdossier.

Subject Dossier:

Facial Features Signature (e.g., nose shape and size, face width,distance between eye corners, skin color (light, medium, dark),nose angle (profile view)Soft Biometrics Signature (e.g., clothing color, height, weight)Temporal Information Signature (e.g., direction of travel, speed,past places visited/path)Fingerprint SignatureVoice Print SignatureIris Scan SignatureDNA Analysis Signature

The information in a subject dossier is indexed so that it canbe used to more efficiently identify and track suspects and toavoid false alarms. More particularly, a dossier is indexed sothat certain information such as soft biometrics can be used toscreen candidates within a surveillance for closer study andalso to predict likely places within a surveillance region tolook for a suspect. For instance, soft biometric information

such as clothing color, height and weight may be employed toselect candidates for further investigation. For example, theKnowledge Services may be programmed to cause the Secu-rity Agents to search for a match between clothing color in a

5 subject dossier of a suspect and clothing color of unidentifiedpersons in a surveillance region. If a match is found, then theKnowledge Service may cause the Security Agents to per-form an analysis of whether facial features in the subjectdossier match facial features of the person with matching

10to color clothing. Moreover, temporal information provided in asubject dossier such as direction and speed of travel of asuspect may trigger the Knowledge Services to alert onlycertain sensory devices, such as a group of cameras in an areaof the surveillance region where the suspect is headed, to be

15 on the lookout for the suspect.A subject dossier may be incremented as more information

concerning a suspect is gathered. For example, initially, onlysoft biometric information such as clothing color and esti-mated height and weight might be available. Subsequently,

20 more information such as a facial feature signature or a voiceprint may become available and will be added to the subjectdossier. Newly received data from these multiple sources maybe fused with previous data by the Knowledge Services as itis received.

25 A subject dossier is a record stored in a computer readablemedium that can be easily accessed by security agents and aconsole operator. The dossier is structured to separate softbiometric information and temporal data from other biomet-ric information. Soft biometric and temporal information

30 generally can be characterized as being easier to obtain anduseful for tracking purpose, but not very reliable for definitiveidentification purposes. Other biometric information, such asfingerprints, voiceprints and an iris scan are more reliable, butmore difficult to obtain. Thus, soft biometric and temporal

35 data can be used advantageously to track an individual untilmore reliable information, such as detailed facial features orfingerprints can be obtained to provide a more reliable iden-tification.

40 Data Gathering Agents

The surveillance system of one embodiment employs mul-tiple streams of data including one or more of, facial features,vocal, fingerprint, iris scan, DNA data, soft biometric data,

45 temporal data and fused data.FIG. 6 is an illustrative flow diagram showing gathering

and conversion of facial feature data to a facial feature signa-ture. Facial feature A comprises a front image of a face that issegmented into a plurality of local areas as an irreducible set

50 of image building elements to extract a set of local featuresthat can be mapped into a mathematical formula. Facial fea-ture B comprises is a side image that is also separated into aset of irreducible image building elements for extracting localfeatures. Facial feature C comprises a side profile curve that is

55 also collected for use in the identity check and authenticationprocesses. Facial features D and E comprise skin color andtone and hair color. These facial feature data are collectedfrom several video key frames taken from a parallax camera.

These facial feature data are used to produce a facial fea-60 tures signature. In one embodiment, the Knowledge Services

which applies an MPEG-7 descriptor, e.g., a facial recogni-tion descriptor, representing a projection of a face vector ontoa set of basis vectors that span the space of possible facevectors and the projection of the face from a side view defined

65 by a profile curve. The face recognition feature sets areextracted from a normalized face image and a normalizedprofile curve. The normalized face image includes 56 lines

US 7,956,890 B215

with 46 intensity values in each line. The centers of the twoeyes in each face image are located on the 24 h row and the16"' and 31" column for the right and left eye respectively.This normalized image is then used to extract the one dimen-sional face vector that includes the luminance pixel valuesfrom the normalized face image arranged into a one dimen-sional vector using a raster scan starting at the top-left cornerof the image and finishing at the bottom right corner of theimage. The face recognition feature set is then calculated byprojecting the one-dimensional face vector onto the spacedefined by a set of basis vectors. By using the front image, theside image, the profile curve, the skin color and tone and thehair color, the accuracy of identity authentication is signifi-cantly improved.

A voiceprint signature also can be produced for identitycheck and authentication over a telephone, for example. Avoiceprint is particularly useful because it is totally noninva-sive. In one embodiment, a multi-dimensional voice identifi-cation process may be employed to generate a speaker's voicesignature by processing pitch contour vectors, time signature,beat number vector and voice shape defined by audio wave-forms of the speaker. For example, one embodiment appliespitch models for different pitch intervals, which are defined tobe the difference between the semitones of two adjacentnodes:

Pitch Interval=[(log(current pitch)-log(previous

pitch)]/log 21112

FIG. 7 is an illustrative flow diagram showing gatheringand conversion of fingerprint feature data to a fingerprintsignature. A raw image of a fingerprint is converted into a setof fingerprint codes. The set of codes has a more compactformat, e.g., IKENDI Fingerprint Pattern Format, which isbased on encoding the friction ridges into a set of directioncodes. The coded fingerprint is converted to fingerprint sig-nature in an MPEG-7 descriptor.

FIG. 8 is an illustrative flow diagram showing gatheringand conversion of DNA data to a DNA signature. Oneembodiment of the invention may employ a DNA fingerprintfor identification purposes. A complete DNA profile includes13 short tandem repeats (STRs) with repeats of four or fivenucleotides in addition to a sex marker. Each STR has variousexpected length and is located on different chromosomes ordifferent ends of the same chromosome and each is indepen-dently inherited. FIG. 8 show respectively the human chro-mosomes with STR names and locations and three or fourdifferent polymorphisms labeled with each of four fluores-cent dyes. The DNAs of different lengths are separated by gelelectrophoresis. Since it is desirable to detect all differentDNAs in one signal identification process, different colors'ofdyes are used to mark different DNAs that have same length.Appropriate dyes are employed in a PCR operation with STRprimers to separate the DNAs based on length and color to getaccurate DNA fingerprint in a single DNA identification pro-cess. The DNA profile signature is generated in the presentinvention by using STRs and STR types, e.g., {STR Name,Type}, {STR Name, Type} where STR Names are {TPOX,DSS1358, FGA, D5S818, CSF1PO, D7S820, D8S1179,TH01, VWA, D13S317, D16S539, D18S51, D21S11, SEX,etc.} Types are required to make sure other DNA sequencesmay use the repeat number of alleles instead of hetero/ho-mozygous, e.g., {Heterozygous, Homozygous}. DNAsamples for identity check and authentication may includehair, saliva, and blood. Samples are collected and their signa-tures are stored in a database. New sample can be collectedand analyzed (but not in real time) using DNA arrays/chips,

16GeneChip, Verigene ID, traditional PCR, or Forensic STRAnalysis methods. The result signature will be matched withthe signatures in the database.

FIG. 8 illustrates genomic barcodes based on a standard5 Universal Product Codes for identifying retailed products by

employing ten alternate numerals at eleven positions to gen-erate one hundred billion unique identifiers. One embodimentof the invention applies the barcode techniques for DNAfingerprint identification process. Special considerations are

10 focused on the facts that the repeat polymorphisms are foundmainly in intergenic (nongene) regions of chromosomes,especially near the centromeres and that the polymorphismsalways exist in a pair in this case, one from each cop of

15 chromosome 1. At a polymorphic locus (location), differentnumbers of a repeated unit create different alleles. Further-more, repeated sequences of 9-80 nucleotides are referred toas Variable Number Tandem Repeats (VNTRs). This VNTR

Shas a 16 nucleotide repeat. Repeated sequences of 2 to 820 nucleotides are referred to as Short Tandem. Repeats (STRs).

This STR has four nucleotide repeat. In a general genomicbarcode system, huge number of string of sites are generatedwith four alternate nucleotides, i.e., adenine, guanine,cytosine, thymine, at each position. A survey of just fifteen of

25 these nucleotide positions would create a possibility of 415,i.e., one billion codes. In the present invention, only fourteenSTRs and types are employed to generate barcodes that areeasier to analyze with much smaller amount of data to processand that can be more conveniently searched with existing

30 search engine, e.g., Google search engine.Soft biometric information, such as clothing' color may be

captured using cameras calibrated in accordance with a pro-cess disclosed in commonly assigned co-pending U.S. patent

35 application Ser. No. Not Yet Known, filed Sep. 16, 2005,entitled "Robust Perceptual Color Identification," invented byK. Goh, E. Y. Chang and Y. F Wang, which is expresslyincorporated by reference in its entirety into this applicationthrough this reference. This patent application addresses a

40 problem of camera-based sensors perceiving an article ofclothing as having a slightly different color when viewedfrom different angles or under different lighting conditions.The patent application proposes the representing color of anarticle of clothing using a "robust perceptual color".

45 Data from different modalities may be fused by the Knowl-edge Services for classification and identification purposeswithout suffering the "curse of dimensionality using tech-niques taught in commonly assigned co-pending U.S. patentapplication Ser. No. 11/129,090, filed May 13, 2005, entitled,

50 Multimodal High-Dimensional Data Fusion for Classifica-tion and Identification, invented by E. Y. Chang, which isexpressly incorporated herein in its entirety by this reference.

SData may be incrementally added to a classification and iden-tification process by the Knowledge Services using tech-

55 niques taught by commonly assigned co-pending U.S. patentapplication Ser. No. 11/230,932, filed Sep. 19, 2005, entitled,Incremental Data Fusion and Decision Making, invented byYuan-Fang Wang, which is expressly incorporated herein inits entirety by this reference.

60 While the invention has been described with reference tovarious illustrative features, aspects and embodiments, it willbe appreciated that the invention is susceptible of variousmodifications and other embodiments, other than those spe-cifically shown and described. The invention is therefore to be

65 broadly construed as including all such alternative variations,modifications and other embodiments within the spirit andscope as hereinafter claimed.

US 7,956,890 B2

What is claimed is:1. A surveillance method comprising:using at least one event sensor disposed in a security area of

a surveillanceregion to sense an occurrence of a poten-tial security breach event; 5

using at least one camera with a view of the security area inwhich the event is sensed to gather biometric informa-tion concerning at least one person in the vicinity of thesecurity area at about the time of the sensing of the event:

producing a subject dossier corresponding to the at least 10o

one person, the subject dossier including at least twobiometric signatures;

matching biometric information of one or more personscaptured by one or more other cameras in the vicinity of 15the at least one camera with corresponding biometricinformation in the subject dossier; and

fusing the at least two signatures and including the fusedsignature in the subject dossier.

2. A surveillance method of comprising:using at least one event sensor disposed in a security area of

a surveillance region to sense an occurrence of a poten-tial security breach event;

using at least one camera with a view of the security area inwhich the event is sensed to gather biometric informa-tion concerning at least one person in the vicinity of thesecurity area at about the time of the sensing of the event:

producing a subject dossier corresponding to the at leastone person, the subject dossier including at least twobiometric signatures;

matching biometric information of one or more personscaptured by one or more other cameras in the vicinity ofthe at least one camera with corresponding biometricinformation in the subject dossier; and

incrementally fusing the at least two signatures and includ-ing the fused signature in the subject dossier.

* * * * *

of 1

STATES PATENT AND TRADEMARK

BIB DATA SHEET

www uspto. gov

CONFIRMATION NO. 4531

SERIAL NUMBER FILING or 371(c) CLASS GROUP ART UNIT ATTORNEY DOCKETDATE NO.

11/231,353 09/19/2005 348 2625 5330.07 (SMC)

RULE

APPLICANTSKen Prayoon Cheng, Saratoga, CA;Edward Y. Chang, Santa Barbara, CA;Yuan-Fang Wang, Goleta, CA;

** CONTINUING DATA *************************This appln claims benefit of 60/610,998 09/17/2004

** FOREIGN APPLICATIONS ************************

** IF REQUIRED, FOREIGN FILING LICENSE GRANTED ** ** SMALL ENTITY **

10/07/2005

Foreign Priority claimed O Yes aNo

35 USC 119(a-d) conditions met Q Yes O No OVerified and /JEROME GRANT II/Acknowledged Examiners signatureTn

ADDRESS

PETERS VERNY, L.L.P.425 SHERMAN AVENUESUITE 230PALO ALTO, CA 94306UNITED STATES

TITLE

Adaptive multi-modal integrated biometric identification detection and surveillance systems

Q All Fees

Q 1.16 Fees (Filing)FILING FEE FEES: Authority has been given in Paper

RECEIVED No. to charge/credit DEPOSIT ACCOUNT D 1.17 Fees (Processing Ext. of time)

565 No. for following: lQ 1.18 Fees (Issue)

Q Other

I Credit

BIB (Rev. 05/07).

UNITED STATrES PATENT AND TRADEMARK OFFICEUNITED STATES DEPARTMENT OF COMMERCEUnited States Patent and Trademark OtheAddms COMMISSIONER FOR PATENTS

P.O. Box 1450Alexadris, Vubie 22313.-1450wuwuipto.gma

CONFIRMATION NO. 4531BI lllIlI IllaISh 11eeItINBib Data Sheet

FILING OR 371(c)SERIAL NUMBER DATE CLASS GROUP ART UNIT ATTORNEYSERIL NUBER LASSDOCKET NO.

11/231,353 09/19/2005 348 262553.TM5330.07 (SMC)

RULE

APPLICANTSKen Prayoon Cheng, Saratoga, CA;Edward Y. Chang, Santa Barbara, CA;Yuan-Fang Wang, Goleta, CA;

* CONTINUING DATA *************************

This appln claims benefit of 60/610,998 09/17/2004'* FOREIGN APPLICATIONS ********************

IF REQUIRED, FOREIGN FILING LICENSE GRANTED* 10/07/2005

Foreign Priority claimed Q yes Q noSTATE OR SHEETS TOTAL INDEPENDENT

35 USC 119 (a-d) conditions 0 yes no Met after COUNTRY DRAWING CLAIMS CLAIMSmet Allowance CA 8 20 3Verified andAcknowledged Examiner's Signature Initials

ADDRESS23308

TITLEADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCESYSTEMS

.All Fees

Q1.16 Fees ( Filing )

FILING FEE FEES: Authority has been given in PaperQ 1.17 Fees ( Processing Ext. ofRECEIVED No. to charge/credit DEPOSIT ACCOUNT time)

565 No. for following: Q 1.18 Fees (Issue)

QOther

I QCredit

of I

UNITED STATES PATENT AND TRADEMARK OFFICEUNITED STATES DEPARTMENT OF COMMERCEUnited States Patent and Trademark OfficeAdd e : COMMISSIONER FOR PATENTS

P.O. Bax 1450Alexadria., Vginia 22313-1450

pw.uryto.gov

CONFIRMATION NO. 4531I IlIaI10Il OIlOl ilheItflI l IBib Data Sheet

FILING OR 371(c)

SERIAL NUMBER DATE CLASS GROUP ART UNIT ATTORNEY11/231,353 09/19/2005 382 2624 DOCKET NO.

RULE

PPLICANTSKen Prayoon Cheng, Saratoga, CA;Edward Y. Chang, Santa Barbara, CA;Yuan-Fang Wang, Goleta, CA;

* CONTINUING DATA ************************

This appln claims benefit of 60/610,998 09/17/2004

* FOREIGN APPLICATIONS ********************

IF REQUIRED, FOREIGN FILING LICENSE GRANTED** SMALL ENTITY *** 10/07/2005

Foreign Priority claimed Q yesQ noy1ci Es n eafe STATE OR SHEETS TOTAL INDEPENDENT

SUSyes 119no- Met after COUNTRY DRAWING CLAIMS CLAIMSVerified and Aowan CA 8 20 3

cknowledged Examiner's Signature. Initials

DDRESS65761

TITLE

Adaptive multi-modal integrated biometric identification detection and surveillance systems

- All Fees

I ' 1.16 Fees ( Filing )

FILING FEE FEES: Authority has been given in Paper O 1.17 Fees ( Processing Ext. ofRECEIVED No. to charge/credit DEPOSIT ACCOUNT time)

565 No. for following:Q 1.18 Fees (Issue)

Other

_ Credit

UNITED STATES PATENT AND ThADEMARK OFFIGEUNITED STATES DEPARTMENT OF COMMERCEUnited States Patent and Trademark OfieAdd renm: COMMISSIONER FOR PATENTS

P.O. Box 1450Alexandria, Vuginia 22313-1450www.uspto.gov

lillH lii llIU llHaII UUO lli l hlleitBib Data Sheet


FILING OR 371(c)

SERIAL NUMBER DATE CLASS GROUP ART UNIT ATTORNEY DOCKET11/231,353 09/19/2005 348 2621 NO.

RULEI

PPLICANTSKen Prayoon Cheng, Saratoga, CA;Edward Y. Chang, Santa Barbara, CA;Yuan-Fang Wang, Goleta, CA;

* CONTINUING DATA **********************

This appin claims benefit of 60/610,998 09/17/2004

* FOREIGN APPLICATIONS *******************

IF REQUIRED, FOREIGN FILING LICENSE GRANTED ** ** SMALL ENTITY **10/07/2005

Foreign Priority claimed Q yesQ no5 1oreign Priority claimed yes no QSTATE OR SHEETS TOTAL INDEPENDENT

35 USC 119 (a-d) conditions yes no Met after COUNTRY DRAWING CLAIMS CLAIMSet Allowance CA 8 20 3erified andCA823cknowledged Examiner's Signature Initials

ADDRESSKEN P. CHENG0691 REID LANE

SARATOGA, CA95070-5325

TITLE

daptive multi-modal integrated biometric identification detection and surveillance systems

Q All Fees

i.1.16 Fees ( Filing )

FILING FEE FEES: Authority has been given in Paper Q 1.17 Fees ( Processing Ext. ofRECEIVED No. to charge/credit DEPOSIT ACCOUNT time)

565 No. for following: 1 Fe ssuI X1.18 Fees( Issue)

Q Other

i-Credit

Page 1 of 1

OO

o D. ,l

o C(71 co~

-40

PTO/SB/05 (05-05)Approved for use through 07/31/2006. OMB 0651-0032

U.S. Patent and Trademark Office. U.S. DEPARTMENT OF COMMERCEUnder the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of miformation unless it displays a valid OMB control number.

UTILITY Attorney Docket No. 577832000200First Inventor Ken Prayoon CHENGPATENT APPLICATION

ADAPTIVE MULTI-MODAL INTEGRATEDTRANSMITTAL Tiue BIOMETRIC IDENTIFICATION DETECTION AND

(ONLY FOR NEW NONPROVISIONAL APPLICATIONS UNDER SURVEILLANCE SYSTEMS37 CFR 1.53(8)) Express Mail Label No. EL 968416615 US

APPLICATION ELEMENTS Commissioner for PatentsSee MPEP chapter 600 concerning utility patent application contents. ADDRESS TO: P.O. Box 1450___Alexandria, VA 22313-1450

1. [ Fee Transmittal Form (e.g., PTOISBI/17), ACCOMPANYING APPULICATION PARTS(Submit en original and a dupicete for fee processing)

2. D Applicant laims small entity status. 9. O Assignment Papers (cover sheet & document(s))2. e D 7FR1.27. Ppr cvr &dcmn~)

3. []Specification (TotalPages 32 ] Name of Assignee:Both the clairm and abstract must start on a new page(For information on the prefered arrangement, see MPEP. 608.01(a))

4. Drawing(s) (35 U.S.C. 113) [Total Sheets 6 ]

5. Oath or Declaration [Total Sheets ] 10. 37 CFR 3.73(b) Statement [7Power of(when there is an assignee) LJ Attorney

a.Q Newly executed (original or copy) 11. English Translation Document (f applicable)

b. [7Acopyfrom a prior application (37 CFR 1.63(d)) 12. Information Disclosure Statement (PTO/SB/08 or PTO-1449)L.J(for contiualt~io~i'sional with Box 1e completed)"i7DELETION OF INVENTOR(S) '[ Copies of foreign patent documents,

SSigned statement attached deleting inventor(s) named in the publications. & other Informationprior application, see 37 CFR 1.63(dX)(2) and 1.33(b). 13. ] Preliminary Amendment

6. [ Application Data Sheet. See 37 CFR 1.76 (3 pages)

7. [7CD-ROM or CD-R in duprlicate. large table or 14. X Return Receipt Postcard (MPEP 503)Computer Program (Appendix) (Should be specifically itemized)

D Landscape Table on CD8. Nucleotide andlor Amino Acid Sequence Submission 15. [ Certified Copy of Priority Document(s)

(if applicable, Items a. - c. are required) (f folbren priority is claimed)a. Computer Readable Form (CRF) 16. [ Nonpublication Request under 35 U.S.C.122 (bX2XBXI).

i . Computer Readable Form (CRF) Appilcant must attach form PTO/SB/35 or equivalent

ii. F]Transfer Request (37 CFR 1.821(e)) 17. Otherb. Specification Sequence Listing on:

i. ] CD-ROM or CD-R (2 copies); or II.D Paperc. O Statements verifying identity of above copies

18. If a CONTINUING APPLICATION, check appropriate box, and supply the requisite information below and in the first sentence of thespecification following the title, or in an Application Data Sheet under 37 CFR 1.76:

] ContinuationF Divisional Continuation-in-part (CIP) of prior application No.:

Prior application Information: Examiner Art Unit:

19. CORRESPONDENCE ADDRESS

S The address associated with Customer Number 20872 OR [] Correspondence address below

Name

Address

City State Zip Code

Country .elep Email Addres s

Signature Date September 19, 2005Name tpeRegistration No.(PrintlType) tp C. DurantAttome/Agent 31,506

sf-2002707 vl

O

I--V

C)

I hereby certify that this correspondence is being deposited with the.S. Postal ic.s Express Mail, Airbill No. EL 968416615 US,in an envelope addressed to: Commissioner for Patents, P.O. inia22313-1450, on the date shown below.

Dated: September 19, 2005 Signature: -_,_ -__-__ . .._ .Todd V. Leone

ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATIONDETECTION AND SURVEILLANCE SYSTEM

Inventor: Ken Prayoon Cheng, Edward Y. Chang, Yuan-Fang Wang

CROSS-REFERENCE TO RELATED APPLICATION

[00011] The present application claims the benefit of earlier filed provisional patent

application, U.S. Application No. 60/610,998, filed on September 17, 2004, and entitled

"Adaptive Multi-Modal Integrated Biometric Identification Detection Systems," which is

hereby incorporated by reference as if fully set forth herein.

BACKGROUND OF THE INVENTION

Field of the Invention:

[0002] The invention relates in general to biometric identification, and more

particularly, to a surveillance system using biometric identification.

Brief Description of the Related Art:

[0003] The state of the art of applying biometric technologies to authenticate and

positively determine the identity of a person is still faced with several technical

challenges. Specifically, the challenges can be categories into two aspects: data

acquisition and data matching. Data acquisition deals with acquiring biometric data from

individuals. Data matching deals with matching biometric data both quickly and

accurately. These challenges can be explained by a port-entry scenario. In such a setting,

it is difficult to obtain certain biometric data such as DNA and voice samples of

individuals. For biometric data that can be more easily acquired, such as face images and

fingerprints, the acquired data quality can vary greatly depending on acquisition devices,

environmental factors (e.g., lighting condition), and individual corporation. Tradeoffs

exist between intrusiveness of data collection, data collection speed, and data quality.

[0004] Once after the needed data have been acquired, conducting matching in a very

large database can be very time-consuming. It goes without saying that unless a system

can acquire and match data both timely and accurately, the system is practically useless

ti

in improving public security, where the inconvenience due to the intrusive data-

acquisition process and the time-consuming matching process ought to be minimized.

100051 A biometric system typically aims to address either one of the following

issues: 1) Authentication: is the person the one he/she claims to be? 2) Recognition:

who a person is? In the first case, data acquisition is voluntary and matching is done in a

one-to-one fashion---matching the acquired data with the data stored on an ID card or in a

database. In the second case, individuals may not be cooperating, and the system must

conduct searches in a very large repository.

[0006] The prior art in biometric can be discussed in two parts: single-modal

solutions and multi-modal solutions. Several systems have been built to use one of the

following single modal: facial data, voice, fingerprint, iris or DNA. The effectiveness of

these single-modal approaches can be evaluated in three metrics: the degree of

intrusiveness, speed and accuracy. From the perspective of a user, acquiring face modal

can be the most noninvasive method, when video cameras are mounted in the distance.

However, the same convenience nature often compromises data quality. An intrusive

face acquisition method is to acquire frontal face features, which requires corporation

from individuals. Voice is another popular modal. However, traditional voice-

recognition fails miserable when voice samples of multiple individuals are

simultaneously captured or when background noise exists. Even when the acquired voice

data can be "pure," existing signal processing and matching techniques can hardly

achieve recognition accuracy of more than 50%. The next popular modal is fingerprint,

which can achieve much higher recognition accuracy at the expense of intrusive data

acquisition and time-consuming data matching. Finally, DNA is by far the most accurate

recognition technique, and the accompanying inconvenience in data acquisition and the

computational complexity are both exceedingly high. Summarizing the single model

approach, non-intrusive data-acquisition techniques tend to suffer from low recognition

accuracy, and intrusive data-acquisition techniques tend to suffer from long

computational time

[0007] As to multimodal techniques, there have been several prior art United States

Patents and PatentApplications disclose techniques. However, as will be further

discussed below, these disclosures do not provide scalable means to deal with tradeoffs

between non-intrusiveness, speed and accuracy requirements. These disclosures may fix

their system configuration for a particular application, and cannot adapt to queries of

different requirements and of different applications.

[0008] Wood et al. disclose in US Patent 6,609,198 a security architecture using the

information provided in a single sign-on in multiple information resources. Instead of

using a single authentication scheme for all information resources, the security

architecture associates trust-level requirements with information resources.

Authentication schemes (e.g., those based on passwords, certificates, biometric

techniques, smart cards, etc.) are employed depending on the trust-level requirement(s) of

an information resource (or information resources) to be accessed. Once credentials have

been obtained for an entity and the entity has been authenticated to a given trust level,

access is granted, without the need for further credentials and authentication, to

information resources for which the authenticated trust level is sufficient. The security

architecture also allows upgrade of credentials for a given session. The credential levels

and upgrade scheme may be useful for a log-on session; however, such architecture and

method of operations do not provide a resolution for high speed and high accuracy

applications such as passenger security check in an airport.

[0009] Sullivan et al. disclose in US Patent 6,591,224 a method and apparatus for

providing a standardized measure of accuracy of each biometric device in a biometric

identity authentication system having multiple users. A statistical database includes

continually updated values of false acceptance rate and false rejection rate for each

combination of user, biometric device and biometric device comparison score. False

acceptance rate data are accumulated each time a user successfully accesses the system,

by comparing the user's currently obtained biometric data with stored templates of all

other users of the same device. Each user is treated as an "impostor" with respect to the

other users, and the probability of an impostor's obtaining each possible comparison score

is computed with accumulated data each time a successful access is made to the system.

The statistical database also contains a false rejection rate, accumulated during a test

phase, for each combination of user, biometric device and biometric device comparison

score. By utilizing a biometric score normalizer, Sullivan's method and apparatus may

be useful for improving the accuracy of a biometric device through acquiring more

training data.

[0010] Murakami et al. disclose is a Patent Publication 20,020,138,768 entitled

"Method for biometric authentication through layering biometric traits," a portable

biometric authentication system having a single technology for measuring multiple,

varied biological traits to provide individual authentication based on a combination of

biological traits. At least one of these biometric traits is a live physiological trait, such as

a heartbeat waveform, that is substantially-but not necessarily completely unique to the

population of individuals. Preferably, at least one of the identifying aspects of the

biological traits is derived from a measurement taken by reflecting light off the

subdermal layers of skin tissue. The Murakami et al. approach is limited by the more

intrusive measurement techniques to obtain data such as heartbeat waveform and

reflecting light off the subdermal layers of skin tissue. These data are not immediately

available in a typical security check situation to compare with the biometric data, e.g.,

heart beat waveforms and reflection light from subdermal layers from the skin of a

targeted searching object. Furthermore, the determination or the filtering of persons'

identity may be too time consuming and neither appropriate for nor adaptive to real time

applications..

[0011] Langley discloses in US Patent Application 20,020,126,881, entitled "Method

and system for identity verification using multiple simultaneously scanned biometric

images," a method to improve accuracy and speed of biometric identity verification

process by use of multiple simultaneous scans of biometric features of a user, such as

multiple fingerprints, using multiple scanners of smaller size than would be needed to

accommodate all of the fingerprints in a single scanner, and using multiple parallel

processors, or a single higher speed processor, to process the fingerprint data more

efficiently. Obtaining biometric data from multiple user features by use of multiple

scanners increases verification accuracy, but without the higher cost and slower

processing speed that would be incurred if a single large scanner were to be used for

improved accuracy. The methods according to Langley may provide the advantages of

speed and accuracy improvements. However, the nature of requiring multiple scans

makes data acquisition time-consuming and intrusive:

[00121 On the academia side, much research effort has been geared toward analyzing

data from individual biometric channels (e.g., voice, face, fingerprint, please see the

reference list for a partial list), less emphasis has been placed on comparing the

performance of different approaches or combing information from multiple biometric

channels to improve identification. Some notable exceptions are discussed below. In

Hong Lin, Jain A. K., Integrating faces and fingerprints for personal identification, IEEE

Transactions on Pattern Analysis and Machine Intelligence, Vol. 20, No. 12, Dec. 1998,

pp. 1295 - 1307, the authors report an automated person identification system that

combines face and fingerprint information. The face recognition method employed is the

traditional eigen face approach, M. Turk and A. Pentland, Eigenfaces for Recognition, J.

Cognitive Neuroscience Vol. 3, No. 1, 1991, pp. 71-96, which computes a set of

orthonormal bases (eigen faces) of the database images using the principal component

analysis. Face images are then approximated by their projection onto the orthonormal

Eigen face bases, and compared using Euclidean distances. For fingerprint, the authors

extend their previous work, Jain, A. K.; Lin Hong; Bolle, R.; On-line fingerprint

verification, Pattern Analysis and Machine Intelligence, Vol. 19, No. 4, April 1997, pp.

302 - 314, to extract minutiaes from fingerprint images. They then align two fingerprint

images by computing the transformation (translation and rotation) between them.

Minutiaes are strung together into a string representation and a dynamic programming-

based algorithm is used to compute the minimum edit distance between the two input

fingerprint strings. Decision fusion is achieved by cross validation of the top matches

identified by the two modules, with matching results weighed by their confidence or

accuracy levels. The performance of the system is validated on a database of about 640

face and 640 fingerprint images.

10013] In Phillips, Henson Moon; Rive, SEA.; Russ, The FERRET evaluation

methodology for face-recognition algorithms, IEEE Transactions on Pattern Analysis and

Machine Intelligence, Vol. 22, No. 10, Oct. 2000, pp. 1090- 1104, the Michigan State

University research group extends their information fusion framework to include more

modalities. In particular, images of a subject's right hand were captured, and fourteen

features comprising the lengths of the fingers, widths of the fingers, and widths of the

palm at various locations of the hand. Euclidean distance metric was used to compare

feature vectors. Simple sum rules, decision tree and linear discriminant function are used

for classification. It is observed that a personal ID system using three modules

outperforms that uses only two of the three modules. While this is an interesting

experiment, the data set used is small and there is no accepted universal standard in using

hand images in biometrics.

10014] In R. Brunelli, D. Falavigna, T. Poggio and L. Stringa, Automatic Person

Recognition by Using Acoustic and Geometric Features, Machine Vision and

Applications 1995, Vol. 8 pp. 317-325, an automated person recognition system using

voice and face signatures is presented. The speaker recognition subsystem utilizes

acoustic parameters (log-energy outputs and their first-order time derivatives from 24

triangular band-pass filters) computed from the spectrum of short-time windows of the

speech signal. The face recognition subsystem is based on geometric data represented by

a vector describing discriminant facial features such as positions and widths of the nose

and mouth, chin shape, thickness and shape of the eyebrows, etc. The system captures

static images of the test subjects and the test subjects are also asked to utter ten digits

from zero to nine for use in the speaker ID subsystem. Each subsystem then computes

the distances of the test subject's speech and face signatures with those stored in the

databases. Decisions from the two ID modules are combined by computing a joint

matching score that is the sum of the two individual matching scores, weighted by the

corresponding variance. Experimental results show that integration of visual and acoustic

information enhances both performance and reliability of the separate systems. The

above system was later improved upon in Brunelli, R.; Falavigna, D., Person

identification using multiple cues, IEEE Transactions on Pattern Analysis and Machine

Intelligence, Vol. 17, No. 10, Oct. 1995, pp. 955-966, where multiple classifiersare

used in the face recognition subsystems, and the matching score normalization process is

made more robust using robust statistical methods.

[0015] In Kittler, J.; Hatef, M.; Duin, R.P.W.; Matas, J., On combining classifiers,,

IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 20, No. 3,

Mar. 1998, pp. 226-239, a performance study of various ensemble classification scheme

is presented. It is shown that many existing decision aggregation rules are actually

simplifications based on the more general Bayesian rule.. The authors compare the

performance of different decision aggregation rules (max, min, median, and majority

voting rule) by performing an experiment in biometrics. Three modules are used: frontal

faces, face profiles, and voiceprints. Simple correlation-based and distance-based

matching is performed on frontal faces and face profiles, respectively, by finding a

geometric transformation that minimizes the differences in intensity. It is shown that a

simple aggregation scheme by summing the results from individual classifiers actually

perform the best.

[0016] In Lu X; Wang Y; and Jain A, Combing classifiers for face recognition, IEEE

International Conference on Multimedia Systems and Expo, Baltimore, MD, July 2003,

three well-known appearance-based face recognition methods, namely PCA, M. Turk and

A. Pentland, Eigenfaces for Recognition, J. Cognitive Neuroscience Vol. 3, No. 1, 1991,

pp. 71-96, ICA, and LDA, Belhumeur, P.N.; Hespanha, J.P.; Kriegman, D.J., Eigenfaces

vs. Fisherfaces: recognition using class specific linear projection, IEEE Transactions on

Pattern Analysis and Machine Intelligence, Vol. 19, No. 7, Jul. 1997, pp. 711-720, are

used for face image classification. Two combination strategies, the sum rule and RBF

network, are used to integrate the outputs from these methods. Experimental results

show that while individual methods achieve recognition rates between 80% and 88%, the

ensemble classifier boosts the performance to 90%, using either the sum rule or RBF

network. In Senior, A., A combination fingerprint classifier, IEEE Transactions on

Pattern Analysis and Machine Intelligence, Vol. 23, No. 10, Oct. 2001, pp. 1165-1174, a

similar multi-classifier scheme, this time for fingerprint classification, is proposed.

Hidden Markov Models and decision trees are used to recognize ridge structures of the

fingerprint. The accuracy of the combination classifier is shown to be higher than that of

two state-of-the-art systems tested under the same condition. These studies represent

encouraging results that validate our multi-modal approach, though only a single

biometric channel, either face or fingerprint, not a combination of biometric channels, is

used in these studies.

[0017] Maio, D.; Maltoni, D.; Cappelli, R.; Waymani, J.L.; Jain, A.K., FVC2000:

fingerprint verification competition, IEEE Transactions on Pattern Analysis and Machine

Intelligence, Vol. 24, No. 3 , March 2002, pp. 402 - 412, documents a fingerprint

verification competition that was carried out in conjunction with the International

Conference on Pattern Recognition (ICPR) in 2000 (a similar contest was held again in

2002). The aim is to take the first step towards the establishment of a common basis to

better understand the state-of-the-art and what can be expected from the fingerprint

technology in the future. Over ten participants, including entries from both academia and

industry, took part. Four different databases, two created with optical sensors, one with a

capacitive sensor, and one synthesized, were used in the validation. Both the enrollment

error (if a training image can be ingested into the database or not) and the matching error

(if a test image can be assigned the correct label or not) and the average time of

enrollment and matching are documented.

10018] A study, that is similar in spirit but compares the performance of face

recognition algorithms, is reported in Phillips, P.J.; Hyeonjoon Moon; Rizvi, S.A.; Rauss,

P.J., The FERET evaluation methodology for face-recognition algorithms, IEEE

Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 10, Oct. 2000,

pp. 1090- 1104. A subset of the Feret database (a gallery of over 3000 images) was used

in the study. Ten different algorithms, using a wide variety of techniques, such as PCA

and Fischer discriminant, were tested. Cumulative matching scores as a function of

matching ranks in the database are tabulated and used to compare the performance of

different algorithms. This study was repeated three times, in August 1994, March 1995,

and July 1996. What is significant about this study is that the performance of the face

recognition algorithms improved over the three tests, while the test condition became

more challenging (with increasingly more images in the test datasets).

10019] As can be seen from the above brief survey, multi-modal biometrics holds a

lot of promise. It is likely that much more accurate classification results can be obtained

by intelligently fusing the results from multiple biometric channels given performance

requirements. While it is important to keep on improving the accuracy and applicability

of individual biometric sensors and recognizers, the performance of a biometric system

can be boosted significantly by judiciously and intelligently employing and combining

multiple biometric channels.

[0020] While there have seen significant research activities in single- and multi-

channel biometry over the past decade, the state-of-the-art is still wanting in terms of

speed and accuracy. Therefore, a need still exists in the art to provide new and improved

methods and system configurations to increase the speed and accuracy ofbiometric

identity verification and determinations such that the above-mentioned difficulties and

limitations may be resolved. The present invention meets this need.

SUMMARY OF THE INVENTION

[0021] One embodiment of the invention provides a novel surveillance method. An

event sensor such as, a camera, chemical sensor, motion detector, unauthorized door

access sensor, for example, is disposed to sense an occurrence of a potential security

breach event. A camera with a view of the area in which an event is sensed gathers

biometric information concerning a subject person in the vicinity of the event at about the

time the event is sensed. A subject dossier is produced containing biometric information

relating to the subject person sensed by the camera with the view of the area. Biometric

information of persons captured on one or more other surveillance cameras in the general

vicinity of the event is matched against corresponding biometric information in the

subject dossier.

[0022] Another embodiment of the invention provides a new surveillance system. A

sensor is disposed in a surveillance region to sense an occurrence of a security breach

event. The system includes a plurality of cameras. At least one camera of the plurality

has a view of the security area and can be configured to automatically gather biometric

information concerning a subject person in the vicinity of an area where the event

occurred in response to the sensing of the event. One or more of the plurality of cameras

can be configured to search for the subject person. The surveillance system also includes

a processing system which can be programmed to produce a subject dossier

corresponding to the subject person. The processing system also can be programmed to

match biometric information of one or more persons captured by one or more of the

cameras with corresponding biometric information in the subject dossier.

100231 These and other features and advantages of the invention sill be apparent from

the following description of embodiments thereof in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is an illustrative showing a map of an airport passenger terminal and

its immediate vicinity protected by a surveillance system of one embodiment of the

invention and also showing several pop-up views relating to event alerts in accordance

with the embodiment.

[0025] Figure 2 is another view of the map of Figure 1 showing zoom to detail maps

of different portions of the overall passenger terminal map.

[0026] Figure 3 is an illustrative drawing of example securityareas within the

surveillance region of Figures 1-2 outfitted with event sensors.

[0027] Figure 4 is an illustrative block level hardware diagram of a surveillance

system in accordance with an embodiment of the invention.

[0028] Figure 5 is an illustrative block diagram level drawing of a system

architecture of an embodiment of the invention that incorporates the system hardware of

Figure 4.

[0029] Figure 6 is an illustrative flow diagram showing gathering and conversion of

facial feature data to a facial feature signature.


fingerprint feature data to a fingerprint signature.


DNA data to a DNA signature. One embodiment of the invention may employ a DNA

fingerprint for identification purposes.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

[0032] The following description is presented to enable any person skilled in the art

to make and use the invention, and is provided in the context of particular applications

and their requirements. Various modifications to the preferred embodiments will be

readily apparent to those skilled in the art, and the generic principles defined herein may

be applied to other embodiments and applications without departing from the spirit and

scope of the invention. Moreover, in the following description, numerous details are set

forth for the purpose of explanation. However, one of ordinary skill in the art will realize

that the invention can be practiced without the use of those specific details. In other

instances, well-known structures and devices are shown in block diagram from in order

not to obscure the description of the invention with unnecessary detail. Thus, the present

invention is not intended to be limited to the embodiments shown, but is to be accorded

the widest scope consistent with the principles and features disclosed herein.

System Overview

[0033] One embodiment of the invention involves an intelligent surveillance system.

A plurality of cameras, some with and some withoutoverlapping fields of view, are

distributed throughout a surveillance region. Intelligent computer software based agents

process information captured by one or more of the cameras to produce a subject dossier

indicative of the identity of a person whose images have been captured by one or more of

the cameras. Information for a subject dossier also may be gathered through other

modalities such as voice recognition, iris scan, or fingerprint, for example. The system

includes multiple event sensors, which may include the cameras, chemical sensors,

infrared sensors, or other security alarm sensors that trigger an alert, upon sensing an

occurrence of a predetermined category of event requiring heightened vigilance. For

example, an alarm may be triggered when a locked door is opened without proper access

permission or when an unauthorized person enters a restricted area or when a vehicle is

parked in a restricted area. More specifically, a subject dossier is produced for

individuals in the vicinity of the location of an alarm-triggering event. For instance, a

subject dossier may be produced for persons captured in a video camera image at or near

a door in the surveillance region at about the time when an unauthorized opening of the

door is detected by an event sensor.

[0034] A subject dossier may include soft biometric information, also referred to as

"soft" features such as clothing color, estimated height and weight. A subject dossier

also may include temporal information, such as walking speed or direction of travel. In

addition, a subject dossier'also may include more permanent information such as facial

features, fingerprint, iris scan, voiceprint and DNA. Soft features maybe selected to be

especially useful for relocating an individual within the surveillance region, especially in

a crowd, for example. For instance, it may be relatively easy to identify individuals

based upon clothing color or estimated height and weight. However, soft features have

the disadvantage of not being as reliable or permanent over time. If a person takes off his

jacket, then an identifying color feature may be lost. If a person sits down, then it may

become impossible to use height and weight information to pick that person out of a

crowd.

[0035] System sensors continually monitor the surveillance region for the occurrence

of one or more suspicious events. In one embodiment, the system directs a live video

feed from one or more cameras having the location of an alert-triggering event in their

field of view to a console in a manned control center. The system also may direct video

images captured just before the event to the control center console. Thus, an operator at

the console can observe behavior of suspicious individuals at the scene of the event in

real time and immediately prior to the event. A subject dossier produced for individuals

at the scene of the event can be used to automatically identify and track a suspect

individual present at the scene of the event within the surveillance area after the

occurrence of the event.

[00361 The system may employ information in a subject dossier incrementally. For

instance, the system may prioritize information in the subject dossier. Certain

information in the subject dossier such as clothing color, estimated height and weight,

walking pattern or gait and certain key facial features such as facial shape, facial hair,

skin color, or hair color may be used to make an initial estimate of which persons in a

camera's field of view are candidates for a match to a suspicious person identified in

response to an alert. Other features from a subject dossier then may be added

incrementally to make a more careful assessment of whether identified candidates

actually match the suspect. Alternatively, as more information concerning a suspicious

person becomes available, additional features may be added incrementally to a suspect's

subject dossier for that person. This additional information then may be used to more

effectively locate and track the individual within the surveillance region.

Surveillance Region

[0037] One embodiment of the invention is configured for use in airport security. In

this embodiment, the surveillance region comprises an airport passenger terminal and the

surrounding passenger ground transport loading/unloading zone directly outside the

terminal and the aircraft parking area adjacent the terminal. Figure 1 is an illustrative

drawing of a map of an airport passenger terminal and its immediate vicinity protected by

a surveillance system of one embodiment of the invention. The system includes multiple

cameras, each with an associated field of view, some of which are overlapping. The

surveillance region has multiple areas including passenger arrival and departure areas, a

passenger departure shops and a terrace. Groups of cameras with overlapping fields of

view are deployed to capture images within different regions of the passenger arrival and

passenger departure areas.

[00381 Figure 2 is another view of the map of Figure 1 showing zoom to detail maps

of different portions of the overall passenger terminal map. The illustrative maps of

Figures 1-2 can be displayed on a control terminal so that an operator can easily correlate

an alert to a specific area an airport surveillance region. For instance, if an alert is

triggered in the arrivals region shown in Figure 1, then an operator may request the left-

most zoom shown in Figure 2 in order to quickly picture the airport layout in the vicinity

of the alert. Additional zoom maps (not shown) may be provided for numerous

locations such as security gates, check-in counters, airport fairway, parking area, access

13

entrance, check-in counters, etc. Each different area may be associated with a group of

cameras and event sensors.

[00391 Event sensors are disposed at selected locations within the surveillance region.

Figure 3 is an illustrative drawing of example security areas within the surveillance

region of Figures 1-2 outfitted with event sensors. A first security area comprises a door.

The door may be equipped with a sensor, such as a mechanical sensor, that detects

unauthorized opening of the door. A second security area comprises a window. The

window may be associated with a mechanical sensor that detects when the window has

been broken. A third security represents a threshold to a restricted area. The restricted

area may be equipped with motion detectors that detect the presence of persons in a

restricted area. Cameras situated throughout the surveillance' region also may serve as

event sensors. For example, the system may employ a monitoring rule whereby a

camera monitors a particular area of the passenger terminal. If a person is loitering in

that area, defined by failing to move beyond a 15 foot radius for more than 60 seconds,

then a low level alert is declared, the camera zooms in, and the face of the loitering

person is matched against the faces of persons on a watch list, for example.

[00401 Landmarks are defined in the security areas for purpose of estimating height

and weight and direction and speed of travel of a suspect individual. For instance, a

landmark such as a countertop may be identified, and processing of a camera image may

be calibrated to estimate a person's height relative to the land marked countertop. A

group of multiple structures, such as telephone booths, lounge areas, signs or countertops,

within a field of view of one or more of a group of cameras covering a security area may

be identified. Processing of camera images from the group of cameras may be used to

estimate the direction and speed at which a suspect is moving based upon the sequence

and timing of his passing the land marked structures.

[00411 Although the surveillance region in this one example is described in terms of

an airport passenger terminal, it will be appreciated that the invention is not restricted to

an airport terminal. Moreover, the surveillance region need not be a continuous local

area. Event sensors and surveillance cameras may be disposed over disparate areas and

be in communication with a control center via a network such as the internet, for

example.

System Architecture

[0042] Figure 4 is an illustrative block level hardware diagram of a surveillance

system in accordance with an embodiment of the invention. The system includes

multiple data collection agents, a knowledge server, a local knowledge server database,

an application server, a middle-tier database, web servers, a browser based control

console and one or more client applications such as Computer Aided Dispatch system,

building management system, access control system, etc. It should be understood that the

various components shown are merely illustrative. Each agent may gather information

from numerous sources, such as the cameras shown in Figure 1, distributed throughout a

surveillance region. Moreover, for example, the knowledge server and the application

server can be implemented across multiple hardware systems or as different processes

within a single hardware system.

[0043] A security agent is a process that spans many tasks to collect information

about subject(s). For example, a security agent may spawn multiple data collection

agents include a facial features, fingerprint, DNA, clothing color, subject gait, subject

height and weight, skin color/tone , hair color/tone, subject direction and voiceprint, for

example. Each data collection task produces different information about an individual.

More specifically, each produces a signature indicative of some identifying aspect of a

person under surveillance. For instance, a facial features agent uses facial information

captured by one or more cameras to produce a signature indicative of an individual's

facial features. Similarly, for example, a clothing color agent uses clothing color

information captured by one or more cameras to produce a signature indicative of the -

color of an individual's clothing color. Thus, the multiple agents can produce multiple

different signatures, each indicative of one or more different identifying feature of an

individual.

[0044] The agents provide the signatures to the knowledge server, which aggregates

signatures for each given person under surveillance into a subject dossier for that person.

The knowledge server indexes the signatures within a given subject dossier to permit

incremental searches for individuals within the search region. The knowledge server also

may perform classification and matching. The local knowledge server database stores the

digital signatures and corresponding indexing information.

[0045] The web services is the component that provides the interfaces via Web

Server which is usually part of an operating system. For example, web services provides

the interfaces for our internal components or external systems via Web Server (such as

Microsoft IIS on Windows, or Apache on Linux). All the interfaces to the system are via

HTTP or HTTPS using port 80. Doing so, our system can run across firewall. Basically,

the Web Services component just exposes our system interface to the outside world via

Web Server.

[0046] The application server is the component that provides that database access to

the user interface component, and performs session management which includes

authentication and authorization. The middle-tier database serves as the local database

for the application server.

[0047] Figure 5 is an illustrative block diagram level drawing of a system

architecture of an embodiment of the invention that incorporates the system hardware of

Figure 4. A user interface (UI) provides an operator of the system with real-time

information concerning alert events within the surveillance region. The UI may provide

maps of the entire surveillance region, including zoom maps. It can display alerts from

different sensors including cameras, digital video recorders, access control, bio-chemical

detectors, etc. It may display videos of a security area in which an alert has been

triggered, detailed images of suspect individuals and details of one or more alerts that

have been triggered.

[0048] Referring again to Figure 1, there is show an example of a UI display screen

in with pop-up display showing various images relating to one or more alerts. In the

center of the screen is map of a surveillance region. The operator can be selectively

enlarge, minimize or close each pop-up. A Video Review display provides a video image

of the security region at about the time of an alert. An Incident Detection display

provides detailed information concerning an alert event. In this example, the alert event

involved an individual tailgating at a commuter door. A Suspect Description display

provides identifying information concerning an individual under surveillance based upon

information gathered into a subject dossier produced for the person. A Detailed Images

display provides pictures of a suspect individual captured by one or more surveillance

cameras. A Potential Identification display provides images of the suspect together with

images of one or more people whose facial features closely match those of the suspect.

The potential matches are based upon a facial feature signature provided by the facial

feature agent. Across the bottom of the map, there is a chart listing briefly summarizing

multiple alert situations. The operator may selectively access pop-up screens for these

alert situation.

[0049] Thus, the UI advantageously displays a variety of information aggregated in

response to one or more alerts. In a typical airport security region, for example, there

may be several hundred cameras dispersed throughout a large physical area. Moreover,

there may be only a few operators monitoring one or more UI consoles. Depending upon

the rules for monitoring and declaring alerts, alerts may occur frequently or infrequently.

The UI of one embodiment of the invention directs an operator to areas of a surveillance

region that are subject to alert and provides pertinent information concerning the alert so

that the operator can efficiently manage security from a control center. The UI also

allows an operator to quickly investigate and simultaneously keep abreast of multiple

alert events.

[0050] Furthermore, as explained more fully below, information from different

sensing devices is correlated to facilitate tracking of a suspect within a security region.

For instance, soft biometric information and temporal information is used to locate a

suspect as he or she travels within the security region. In one embodiment, a dashed line

can be produced on a map on the display showing a path followed by a suspect within the

surveillance region. Information from different data collection agents may be fused in

order to more Iccurately-identify and track an individual. Therefore, the operator can use

the UI to evaluate an alert event, to identify and track a suspect. The operator may use

this information as a basis to send information to a responder to intercede or deal with an

alert incident.

[0051] Knowledge Services are implemented as an application running on the

knowledge server. Knowledge Services correlate and analyze signature information

provided by different sensory devices (i.e., data gathering agents). The Knowledge

Services assemble and index subject dossiers, and when appropriate, fuse signature

information for improved classification results. The Knowledge Services also generate,

activate or deactivate rules and send/control rules and instruction to the Rules and Agent

Manager.

[00521 The Rules and Agent Manager also is implemented on the knowledge server.

The Rules and Agent Manager manages all other agents and manages rules that can be

sent to each agent. It correlates information from agents. It can also escalate an alert if

the alert is not acknowledged by an operator within a given timeframe and/or similar

alerts happen repeatedly within a given time span (e.g. within 2 hours). Both the

Knowledge Service and the Rules and Agent Manager are the primary components for

aggregating, categorizing biometric signatures which are parts of object dossiers. It also

performs other tasks such as task assignment/tracking, load balancing tasks among

agents, and interacting with data access components.

[00531 The following are examples of rules that may be implemented by the system.

100541 Rules:

Actor Action

Person Walk though lane against direction of traffic

Person Tailgating

Person Loitering

Person Piggyback

Person Traveler screening

Person Walk in restricted area

Vehicle Park overtime

Vehicle Park in restricted area

[0055] The Person-Loitering rule involves the following criteria:

Radius 15 foot

Duration 20 seconds

Alert Severity Low

Response Zoom in face to match "watch list"

100561 The Person-Tailgating Rule involves the following criteria:

Alert Severity Low

Response Zoom in face to match "watch list"

[0057] The correlation Monitoring Rule for the occurrence of a Person-Loiteringevent AND a Person-Tailgating event involving the same person is as follows:

Alert CriticalSeverity

Action Acknowledge Loitering and Tailgating alerts and deliver alarm tooperator console

[0058] As described above the UI, may display several categories of information

concerning an alert. The Knowledge Service and the Rules and Agent Manager provide

the correlation between events and data sources and subject dossiers that permit an

operator to view a map of the location of an alert, soft-biometric data of a suspect and

video playback, for example. More particularly, these components provide a link to a

map and zoom stored in the middle tier database, link to video feeds for video view real-

time monitoring or playback of recorded video clips and stored in a Digital Video

Recorder system and provide the subject dossier information.

[0059] The Middle Tier Data Access runs on the application server. It controls the

database including functions such as query, add, delete, index. Indexing biometric

signatures and updating subject dossiers are done by this component.

iA

[00601 A (Security) Agent is implemented as an application running on the

knowledge server that controls and manages the data gathering sensors. In the case of

cameras or DVRs, it can also perform video analytic using Computer Vision technology.

Those tasks include background subtraction, image stabilization, object detection, object

classification, object tracking, and object identification. It can also control the movement

of Pan/Tilt/Zoom (PTZ) cameras, manage areas of interest within the field of view of the

camera (called Mouse/Man Trap), and collect video streams from DVR or cameras. It

also has a scheduler that controls when rules or video analytic are performed.

[00611 A SensoryDevice Directory Access and Video Server is implemented as an

application that has access to the knowledge server manages and provides information

regarding sensor devices or other subsystems. Basically, it is a software layer that enables

the overall system to handle different makes/models of sensor devices.

[00621 The Web Services is the component provided by operating systems or web

servers. It manages other components, spawns or deletes services as necessary. It can

also listen to messages from other systems. The Web Services provides interfaces to the

system via Web Services running as part of a Web Server. The system provides a library

resided on a specific directory, and the Web Server (which is usually part of the operating

system) will use it to interpret interface requests to our system.

[0063] Tracking, Facial Recognition, Fingerprint recognition, and other biometric

identification are done at the (Security) agents. Biometric signatures are collected and

generated at the agents, and sent to the Rules-and-Agent Manger. The Knowledge

Services and the Rule-and-Agent Manager collectively collect biometric signatures and

object tracking locations, and then generate and manage subject dossiers. A described

above, a subject dossier includes information about object (e.g., person) such as,

biometric information/signatures, soft biometric information (hair color, skin tone/color,

weight or build, height, etc.) and other temporal information (e.g., speed, direction,

location, past activities, information that the operator is looking for, etc.). Data fusion is

performed by the Knowledge Services and the Rules and Agent Manager. Data required

or generated by each of the components are saved and retrievable via the

Middle-tier/Data Access component, which in turn utilizes a relational database such as

Microsoft SQL Server.

Subject Dossier

[0064] Data gathering agents collect data concerning a subject person from different

sources. The Knowledge Services aggregate the data into subject dossier. The data

aggregated into a given dossier may include different digital signatures produced by

different data gathering agents. A subject dossier also may include fused data signatures

produced by the fusion of data gathered from multiple data sources having different data

modalities.

[0065] The following is an example of information in a subject dossier.

[0066] Subject Dossier:

Facial Features Signature (e.g., nose shape and size, face width, distance between eyecorners, skin color (light, medium, dark), nose angle (profile view)

Soft Biometrics Signature (e.g., clothing color, height, weight)

Temporal Information Signature (e.g., direction of travel, speed, past places visited/path)

Fingerprint Signature

Voice Print Signature

Iris Scan Signature

DNA Analysis Signature

[0067] The information in a subject dossier is indexed so that it can be used to more

efficiently identify and track suspects and to avoid false alarms. More particularly, a

dossier is indexed so that certain information such as soft biometrics can be used to

screen candidates within a surveillance for closer study and also to predict likely places

within a surveillance region to look for a suspect. For instance, soft biometric

information such as clothing color, height and weight may be employed to select

candidates for further investigation. For example, the Knowledge Services may be

programmed to cause the Security Agents to search for a match between clothing color in

a subject dossier of a suspect and clothing color of unidentified persons in a surveillance

region. If a match is found, then the Knowledge Service may cause the Security Agents

to perform an analysis of whether facial features in the subject dossier match facial

features of the person with matching color clothing. Moreover, temporal information

provided in a subject dossier such as direction and speed of travel of a suspect may

trigger the Knowledge Services to alert only certain sensory devices, such as a group of

cameras in an area of the surveillance region where the suspect is headed, to be on the

lookout for the suspect.

[00681 A subject dossier may be incremented as more information concerning a

suspect is gathered. For example, initially, only soft biometric information such as

clothing color and estimated height and weight might be available. Subsequently, more

information such as a facial feature signature or a voice print may become available and

will be added to the subject dossier. Newly received data from these multiple sources

may be fused with previous data by the Knowledge Services as it is received.

[00691 A subject dossier is a record stored in a computer readable medium that can be

easily accessed by security agents and a console operator. The dossier is structured to

separate soft biometric information and temporal data from other biometric information.

Soft biometric and temporal information generally can be characterized as being easier to

obtain and useful for tracking purpose, but not very reliable for definitive identification

purposes. Other biometric information, such as fingerprints, voiceprints and an iris scan

are more reliable, but more difficult to obtain. Thus, soft biometric and temporal data can

be used advantageously to track an individual until more reliable information, such as

detailed facial features or fingerprints can be obtained to provide a more reliable

identification.

Data Gathering Agents

[0070] The surveillance system of one embodiment employs multiple streams of data

including one or more of, facial features, vocal, fingerprint, iris scan, DNA data, soft

biometric data, temporal data and fused data.


facial feature data to a facial feature signature. Facial feature A comprises a front image

of a face that is segmented into a plurality of local areas as an irreducible set of image'

building elements to extract a set of local features that can be mapped into a

mathematical formula. Facial feature B comprises is a side image that is also separated

into a set of irreducible image building elements for extracting local features. Facial

feature C comprises a side profile curve that is also collected for use in the identity check

and authentication processes. Facial features D and E comprise skin color and tone and

hair color. These facial feature data are collected from several video key frames taken

from a parallax camera:

[0072] These facial feature data are used to produce a facial features signature. In

one embodiment, the Knowledge Services which applies an MPEG-7 descriptor, e.g., a

facial recognition descriptor, representing a projection of a face vector onto a set of basis

vectors that span the space of possible face vectors and the projection of the face from a

side view defined by a profile curve. The face recognition feature sets are extracted from

a normalized face image and a normalized profile curve. The normalized face image

includes 56 lines with 46 intensity values in each line. The centers of the two eyes in each

face image are located on the 24 th row and the 16th and 3 1 st column for the right and left

eye respectively. This normalized image is then used to extract the one dimensional face

vector that includes the luminance pixel values from the normalized face image arranged

into a one dimensional vector using a raster scan starting at the top-left corner of the

image and finishing at the bottom right corner of the image. The face recognition feature

set is then calculated by projecting the one-dimensional face vector onto the space.

defined by a set of basis vectors. By using the front image, the side image, the profile

curve, the skin color and tone and the hair color, the accuracy of identity authentication is

significantly improved.

23.

[00731 A voiceprint signature also can be produced for identity check and

authentication over a telephone, for example. A voiceprint is particularly useful because

it is totally noninvasive. In one embodiment, a multi-dimensional voice identification

process may be employed to generate a speaker's voice signature by processing pitch

contour vectors, time signature, beat number vector and voice shape defined by audio

waveforms of the speaker. For example, one embodiment applies pitch models for

different pitch intervals, which are defined to be the difference between the semitones of

two adjacent nodes:

Pitch Interval = [(log (current pitch)-log (previous pitch)]/log 21/12

[00741 Figure 7 is an illustrative flow diagram showing gathering and conversion of

fingerprint feature data to a fingerprint signature. A raw image of a fingerprint is

converted into a set of fingerprint codes. The set of codes has a more compact format,

e.g., IKENDI Fingerprint Pattern Format, which is based on encoding the friction ridges

into a set of direction codes. The coded fingerprint is converted to fingerprint signature

in an MPEG-7 descriptor.


DNA data to a DNA signature. One embodiment of the invention may employ a DNA

fingerprint for identification purposes. A complete DNA profile includes 13 short

tandem repeats (STRs) with repeats of four or five nucleotides in addition to a sex

marker. Each STR has various expected length and is located on different chromosomes

or different ends of the same chromosome and each is independently inherited. Figure 8

show respectively the human chromosomes with STR names and locations and three or

four different polymorphisms labeled with each of four fluorescent dyes. The DNAs of

different lengths are separated by gel electrophoresis. Since it is desirable to detect all

different DNAs in one signal identification process, different colors of dyes are used to

mark different DNAs that have same length. Appropriate dyes are employed in a PCR

operation with STR primers to separate the DNAs based on length and color to get

accurate DNA fingerprint in a single DNA identification process. The DNA profile

signature is generated in the present invention by using STRs and STR types, e.g., {STR

Name, Type), {STR Name, Type) where STR Names are {TPOX, DSS1358, FGA,

D5S818, CSFIPO, D7S820, D8S1179, THO1, VWA, D13S317, D16S539, D18S51,

D21S11, SEX, etc.} Types are required to make sure other DNA sequences may use the

repeat number of alleles instead of hetero/homozygous, e.g., {Heterozygous,

Homozygous}. DNA samples for identity check and authentication may include hair,

saliva, and blood. Samples are collected and their signatures are stored in a database.

New sample can be collected and analyzed (but not in real time) using DNA arrays/chips,

GeneChip, Verigene ID, traditional PCR, or Forensic STR Analysis methods. The result

signature will be matched with the signatures in the database.

10076] Figure 8 illustrates genomic barcodes based on a standard Universal Product

Codes for identifying retailed products by employing ten alternate numerals at eleven

positions to generate one hundred billion unique identifiers. One embodiment of the

invention applies the barcode techniques for DNA fingerprint identification process.

Special considerations are focused on the facts that the repeat polymorphisms are found

mainly in intergenic (nongene) regions of chromosomes, especially near the centromeres

and that the polymorphisms always exist in a pair in this case, one from each cop of

chromosome 1. At a polymorphic locus (location), different numbers of a repeated unit

create different alleles. Furthermore, repeated sequences of 9-80 nucleotides are referred

to as Variable Number Tandem Repeats (VNTRs). This VNTR has a 16 nucleotide

repeat. Repeated sequences of 2 to 8 nucleotides are referred to as Short Tandem.

Repeats (STRs). This STR has four nucleotide repeat. In a general genomic barcode

system, huge number of string of sites are generated with four altemrnate nucleotides, i.e.,

adenine, guanine, cytosine, thymine, at each position. A survey of just fifteen of these

nucleotide positions would create a possibility of 415, i.e., one billion codes. In the

present invention, only fourteen STRs and types are employed to generate barcodes that

are easier to analyze with much smaller amount of data to process and that can be more

conveniently searched with existing search engine, e.g., Google search engine.

[0077] Soft biometric information, such as clothing color may be captured using

cameras calibrated in accordance with a process disclosed in commonly assigned co-

pending U. S. Patent Application Serial No. Not Yet Known, filed September 16, 2005,

entitled "Robust Perceptual Color Identification," invented by K. Goh, E. Y. Chang and

Y. F Wang, which is expressly incorporated by reference in its entirety into this

application through this reference. This patent application addresses a problem of

camera-based sensors perceiving an article of clothing as having a slightly different color

when viewed from different angles or under different lighting conditions. The patent

application proposes the representing color of an article of clothing using a "robust

perceptual color".

[00781 Data from different modalities may be fused by the Knowledge Services for

classification and identification purposes without suffering the "curse of dimensionality

using techniques taught in commonly assigned co-pending U. S. Patent Application

Serial No. 11/129,090, filed May 13, 2005, entitled, Multimodal High-Dimensional Data

Fusion for Classification and Identification, invented by E. Y. Chang, which is expressly

incorporated herein in its entirety by this reference. Data may be incrementally added to

a classification and identification process by the Knowledge Services using techniques

taught by commonly assigned co-pending U. S. Patent Application Serial No._, filed

September 19, 2005, entitled, Incremental Data Fusion and Decision Making, invented by

Yuan-Fang Wang, which is expressly incorporated herein in its entirety by this reference.

[0079] While the invention has been described with reference to various illustrative

features, aspects and embodiments, it will be appreciated that the invention is susceptible

of various modifications and other embodiments, other than those specifically shown and

described. The invention is therefore to be broadly construed as including all such

alternative variations, modifications and other embodiments within the spirit and scope as

hereinafter claimed.

26

CLAIMS

What is claimed is:

1. A surveillance method comprising:

using at least one event sensor disposed in a security area of a surveillance region to

sense an occurrence of a potential security breach event;

using at least one camera with a view of the security area in which the event is sensed to

gather biometric information concerning at least one person in the vicinity of the security

area at about the time of the sensing of the event;

producing a subject dossier corresponding to the at least one person; and

matching biometric information of one or more persons captured by one or more other

cameras in the vicinity of the at least one camera with corresponding biometric

information in the subject dossier.

2. The method of claim 1,

wherein the at least one camera gathers clothing color; and

wherein matching involves matching clothing color of one or more persons captured by

the one or more other cameras with clothing color in the subject dossier.


wherein the at least one camera gathers a facial feature signature; and

wherein matching involves matching facial feature signature of one or more persons

captured by the one or more other cameras with a facial feature signature in the subject

dossier.


wherein the at least one camera gathers hair color; and

wherein matching involves matching hair color of one or more persons captured by the

one or more other cameras with hair color in the subject dossier.


wherein the at least one camera gathers skin color; and

wherein matching involves matching skin color of one or more persons captured by the

one or more other cameras with skin color in the subject dossier.


wherein the at least one camera gathers one or more of clothing color, facial features,

skin color and hair color; and

wherein matching involves matching at least one of clothing color, facial features, skin

color and hair color of one or more persons captured by the one or more other cameras

with corresponding one or more of clothing color, facial features, skin color and hair

color in the subject dossier.


wherein the at least one camera gathers one or both of estimated height and weight; and

wherein matching involves matching at least one of height and weight of one or more

persons captured by the one or more other cameras with corresponding or both of

estimated height and weight in the subject dossier.


wherein the at least one camera gathers estimated direction of travel of the at least one

person; and further including

determining which cameras are in the vicinity of the at least one camera based upon the

estimated direction of travel.


wherein the at least one camera gathers estimated speed of travel of the at least one

person; and further including


estimated speed of travel.


wherein the at least one camera gathers estimated direction of travel and estimated speed

of travel of the at least one person; and further including


estimated direction of travel and the estimated speed of travel.



skin color, hair color, estimated height and estimated weight;


color, hair color, estimated height and estimated weight of one or more persons captured

by the one or more other cameras with corresponding one or more of clothing color,

facial features, skin color, hair color, estimated height and estimated weight in the subject

dossier.



skin color, hair color, estimated height and estimated weight;

wherein the at least one camera gathers one or both of estimated direction of travel and

estimated speed of travel of the at least one person; and further including

determining which cameras are in the vicinity of the at least one camera based upon at

least one of the estimated direction of travel and the estimated speed of travel; and


color, hair color, estimated height and estimated weight of one or more persons captured

by the one or more other cameras with corresponding one or more of clothing color,

facial features, skin color, hair color, estimated height and estimated weight in the subject

dossier.

13. The method of claim 1 further including:

providing to a control console display an image of a person captured by the one or more

other cameras having biometric information matching biometric information in the

subject dossier.


wherein the subject dossier includes at least two biometric signatures; and further

including:

fusing the at least two signatures and including the fused signature in the subject dossier.


wherein the subject dossier includes at least two biometric signatures; and further

including:

incrementally fusing the at least two signatures and including the fused signature in the

subject dossier.


wherein the at least one camera with a view of the security area also serves as an event

sensor.

17. A surveillance system comprising:

at least one sensor disposed in a security area of a surveillance region to sense an

occurrence of a potential security breach event;

a plurality of cameras;

wherein the at least one camera of the plurality has a view of the security area and can be

configured to automatically gather biometric information concerning at least one subject

person in the vicinity of the security area in response to the sensing of a potential security

breach event in the security area;

wherein one or more other of the plurality of cameras can be configured to search for the

at least one subject person;

a processing system;

wherein the processing system is programmable to produce a subject dossier

corresponding to the at least one subject person; and

wherein the processing system is programmable to match biometric information of one or

more persons captured by one or more of the other cameras with corresponding biometric

information in the subject dossier.

18. The system of claim 17,

wherein the at least one camera serves as an event sensor.

19. The system of claim 17,

wherein the processing system is programmable to produce a subject dossier that includes

at least one or more of clothing color, facial features, skin color, hair color, estimated

height and estimated weight;

wherein the processing system is programmable to match at least one of clothing color,

facial features, skin color, hair color, estimated height and estimated weight of one or

more persons captured by the one or more other cameras with corresponding one or more

of clothing color, facial features, skin color, hair color, estimated height and estimated

weight in the subject dossier.

20. A surveillance system comprising:

a plurality of sensors disposed in a surveillance region to sense an occurrence of a

potential security breach event;

a plurality of cameras disposed in the surveillance region,

wherein one or more of the plurality of cameras can be configured to automatically gather

one or both of biometric information and temporal identifying information concerning at

least one subject person in the vicinity of such camera, in response to sensing of a

potential security breach event in the vicinity of such camera;

wherein one or more of the plurality of cameras can be configured to search for the at

least one subject person;

a processing system;

wherein the processing system is programmable to produce a subject dossier including

one or both of biometric information or temporal identifying information corresponding

to the at least one subject person; and

wherein the processing system is programmable to match one or both of biometric

information or temporal identifying information of one or more persons captured by one

or more cameras configured to search for the at least one subject person.

31

ABSTRACT OF THE DISCLOSURE

A surveillance system is provided that includes at least one sensor disposed in a security

area of a surveillance region to sense an occurrence of a potential security breach event; a

plurality of cameras is disposed in the surveillance region; at least one camera of the

plurality has a view of the security area and can be configured to automatically gather

biometric information concerning at least one subject person in the vicinity of the

security area in response to the sensing of a potential security breach event; one or more

other of the plurality of cameras can be configured to search for the at least one subject

person; a processing system is programmed to produce a subject dossier corresponding to

the at least one subject person to match biometric information of one or more persons

captured by one or more of the other cameras with corresponding biometric information

in the subject dossier.

,App .No.. ,1o v'e. ssigneo ,.ocKet No.: bi'IdJ2UUU-UUInventor: Ken P. CHENG et al.Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC

IDENTIFICATION DETECTION AND, etc.

BEST AVAILABLE COPY

v4

1L

A pp ,0o.: ,o, t eiAssigneo .ocKe 1 .4o.. wi adUUUUuInventor: Ken P. CHENG et al.Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC


NI

l6

Bt

.App N.o.: . o r e Assigneo Joc<et No.: 577832Inventor: Ken P. CHENG et al.Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC


c-

o

U=

3200200

R

'im

O)C)

im

0a)

c0

0.0.

I

rArrT %All AM ; r(

BF-1 AAILDLG r

CFn

App No.: Not Yet Assigneo .JocKet po.: a1 1dauUUUU Inventor: Ken P. CHENG et al.Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC

IDENTIFICATION DETECTION AND, etc.Sheet 5 of 6

J

Fig. I is a system diagram of a biometric identity check and authentication system of thisinvention

mm'i

;;jim.

r*-m

mmw

rwwmaum

A Fb a na kC. PUofhMPa . SIh ITm L'H* coor

Agent

NMora i0fei nawgorat Pofs cur.

NonAted Cda 5&Ton.

0~IKnowfdpSe srvi

Fig. 2 Diagrams for illustrating the facial identification data implemented in this invention.

BEST AVAILABLE COP

rsc SipnowmIMPE-7 Dsscutw

Proxli nex Mnde~m O_ AM & R .

49

I

iveylw s aAw

App No.: Not Yet Assigned .)OCKet No.: 577832000200Inventor: Ken P. CHENG et a1.Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC


p.

--~-----------

v %-; i PVLL ~

App No.: Not Yet Assigned Docket No.: 577832000200Inventor: Ken P. CHENG et al.Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC

IDENTIFICATION DETECTION AND,etc.Sheet 6 of 6

A.PwR~gpPt B. Encoed RFvmp t

L iIKEIDIFbowseput574$M SE r r

1G

Fin M6 79 oo

I>LJProxbewx-msd srw

Figs. 3A and 3B illustrate the raw-ridge image and the image of the encoded ridge-skeletons afterthe raw-ridge processed by the fingerprint verification process of this invention.

ALUE CuREN fD altow

11N)l'W tVe+rr O.6 019 D 11" IfU VWA

S nr i so s 1 va DaY,30 DR1T9 7 P OIaAIslA. HUa O with SR Nmu& Ct/POms Hm OvomoSm al wh w

A. Hunan Onmosonaswth STR Nana &teasons .Hun= Orw moabld wth Raoreset arDs

DNA

Ae***rk

Q. Getmn* 3wFCA

00

% C DA FlngsrprktImkn3hInUPEW -i Dewu

Knowh S"e Bever

Figs. 4A and 4B show respectively the human chromosomes with STR names and

locations and three or four different polymorphisms labeled with each of four fluorescent dyes.

REST AVAILABLF C OP'

I

'i

ell

i

Application Data Sheet

Application Information

Application Type::

Subject Matter::

Suggested Group Art Unit::

CD-ROM or CD-R?::

Sequence submission?::

Computer Readable Form (CRF)?::

Title::

Attorney Docket Number::

Request for Early Publication?::

Request for Non-Publication?::

Total Drawing Sheets::

Small Entity?::

Petition included?::

Secrecy Order in Parent Appl.?::

Regular

Utility

Not Yet Assigned

None

None

No

ADAPTIVE MULTI-MODAL INTEGRATED

BIOMETRIC IDENTIFICATION

DETECTION AND SURVEILLANCE

SYSTEMS

577832000200

No

No

6

Yes

No .

No

Applicant Information

Applicant Authority Type::

Primary Citizenship Country::

Status::

Given Name::

Middle Name::

Family Name::

City of Residence::

State or Province of Residence::

Country of Residence::

Inventor

US

Full Capacity

Ken

Prayoon

CHENG

Saratoga

California

US

s-200269sv1 Page 1 Initial 09/19/05

Street of mailing address::

City of mailing address::

State or Province of mailing address::

Postal or Zip Code of mailing address::



Status::

Given Name::

Middle Name::

Family Name::,

City of Residence::

State or Province of Residence::.








Status::

Given Name::

Family Name::

City of Residence::







20691 Reid Lane

Saratoga

California

95070

Inventor

US

Full Capacity

Edward

Y.

CHANG

Santa Barbara

California

US

816 Dorado Drive

Santa Barbara

California

93111

Inventor

US

Full Capacity

Yuan-Fang

WANG

Goleta

California

US

5849 Via Fiori Lane

Goleta

California

93117

Page 2sf-2002696vi Initial 09/19/05

Correspondence Information

Correspondence Customer Number::

Representative Information

Representative Customer Number::

20872

20872

Domestic Priority Information

Application:: Continuity Type:: Parent Application:: Parent Filing Date::This application An application 60/610,998 09/17/2005

claiming the benefitunder 35 USC119(e)

Foreign Priority Information

Assignee Information

Assignee name:: Proximex Corporation

Initial 09/19/05Page 3sf-2002696v

UNITED SIATES'PATENT AND ThADEMARK OFFIcESUNITED STATES DEPARTMENT OF COMMERCEUunited States Patent and Trademark OfroeAddres COMMISSlONE.R FOR PATENTS

P.O. Box 1450AJk niu, Vimiia 22313-1450www.1apio.jr

APPLICATION NUMBER FLING OR 371 (c) DATE FIRST NAMED APPLICANT ATTORNEY DOCKET NUMBER

11/231,353 09/19/2005 Ken Prayoon Cheng 577832000200

20872MORRISON & FOERSTER LLP425 MARKET STREETSAN FRANCISCO, CA 94105-2482

CONFIRMATION NO. 4531FORMALITIESLETTER

Date Mailed: 10/11/2005

NOTICE TO FILE MISSING PARTS OF NONPROVISIONAL APPLICATION

FILED UNDER 37 CFR 1.53(b)

Filing Date Granted

Items Required To Avoid Abandonment:

An application number and filing date have been accorded to this application. The item(s) indicated below,however, are missing. Applicant is given TWO MONTHS from the date of this Notice within which to file allrequired items and pay any fees required below to avoid abandonment. Extensions of time may be obtained byfiling a petition accompanied by the extension fee under the provisions of 37 CFR 1.136(a).

* The statutory basic filing fee is missing.Applicant must submit $ 150 to complete the basic filing fee for a small entity.

* The oath or declaration is missing. A properly signed oath or declaration in compliance with 37 CFR 1.63,identifying the application by the above Application Number and Filing Date, is required.Note: If a petition under 37 CFR 1.47 is being filed, an oath or declaration in compliance with 37 CFR 1.63signed by all available joint inventors, or if no inventor is available by a party with sufficient proprietaryinterest, is required.

The application is informal since it does not comply with the regulations for the reason(s) indicated below.

The required item(s) identified below must be timely submitted to avoid abandonment:

* Replacement drawings in compliance with 37 CFR 1.84 and 37 CFR 1.121(d) are required. The drawingssubmitted are not acceptable because:

The drawings submitted to the Office are not electronically reproducible. Drawingsheets must be submitted, on paper, which is flexible, strong, white, smooth, non-shiny,and durable (see 37 CFR 1.84(e)). See Figure(s) 1 - 8.

Applicant is cautioned that correction of the above items may cause the specification and drawings page count toexceed 100 pages. If the specification and drawings exceed 100 pages, applicant will need to submit the requiredapplication size fee.

The applicant needs to satisfy supplemental fees problems indicated below.

UNITED STATESPATENT AND TRADE K OFFIGE

I


* To avoid abandonment, a surcharge (for late submission of filing fee, search fee, examination fee or oath ordeclaration) as set forth in 37 CFR 1.16(f) of $65 for a small entity in compliance with 37 CFR 1.27, must besubmitted with the missing items identified in this letter.

SUMMARY OF FEES DUE:

Total additional fee(s) required for this application is $565 for a Small Entity

* $150 Statutory basic filing fee.* $65 Surcharge.

* The application search fee has not been paid. Applicant must submit $250 to complete the search fee.* The application examination fee has not been paid. Applicant must submit $100 to complete the

examination fee for a small entity in compliance with 37 CFR 1.27

Replies should be mailed to: Mail Stop Missing Parts

Commissioner for Patents

P.O. Box 1450

Alexandria VA 22313-1450

A copy of this notice MUST be returned with the reply.

Office ial Patent Examination (571) 272-4000, or 1-800-PTO-9199, or 1-800-972-6382PART 3 - OFFICE COPY

PTO/SB/21 (09-04)Approved for use through 07131/2006. OMB 0651-0031

UII S Patent and Trademark Offic- II E S. PARTMeN;T OF COMMRCE

sf-2062538 v1

JAN 13 2006

I hereby certify that this correspondence is being deposited with the U.S. Postal Service with sufficient postage as First Class Mail, inan envelope addressed to: Mail Stop MISSING PARTS. Commissioner for Paten P.O. Box 1450, Alexandria, Virginia 22313-1450.on the date shown below.

Dated: January 11. 2006 Signature: Todd V. Leone

r the Paperwork Reduction Act of 1995, no persons are required to res and to a collection of Information unless it displays a valid OMB control number.Application Number .11/231,353

TRANSMITTAL Filing Date September 19, 2005

FORM First Named Inventor Ken Prayoon CHENGArt Unit 2621

(to be used for all correspondence after initial filing)

Examiner Name Not Yet Assigned

Total Number of Pages in This Submission 30 Attomey Docket Number 577832000200

ENCLOSURES (Check all that apply)

X- Fee Transmittal Form (orig.+copy) - Drawing(s) (8 pages) ] After Allowance Communication.(2 pages) J to TC

F Fee Attached [ Ucensing-related Papers • Appeal Communication to Board ofAnnnlc anrtl InterfRrArr. A

SAmendment/Reply Petition Appeal Communication to TC(Aooeal Notice. Brief. Reolv Brief)

SAfter Final [ Pet ition to Convert to a Proprietary InformationProvisional Application

SAffidavits/declaration(s) F Power of Attomey, Revocatio dress Status LetterU~ Change of Correspondence Addresswith Form PTO/S/96 (6 pages)

[X Extension of Time Request (1 page)D Terminal Disclaimer [ Other Enclosure(s) (pleaseIdentify below):

[ Express Abandonment Request F Request for Refund (D Submission of Drawings (1 page)

W i Copy of Notice to File Missing

L Information Disclosure Statement CD, Number of CD(s) Parts, et. (2 pages)cordationCD. umbr ofCD~) _____ f® Assignment with RecordationD Form Cover Sheet (5 pages)

[]Certified Copy of Priority Landscape Table on CD Declaration for Utility or Design

Application Using an Application

- Reply to Missing Parts/ Remarks Data Sheet (1 page)Incomplete Application ® Application Data Sheet (3 pages)

"DReply to Missing Parts under ® Return Receipt Postcard37 CFR 1.52 or 1.53

SIGNATURE OF APPLICANT, ATTORNEY, OR AGENTFirm Name MORRI & FOERSTER LLP Customer No. 20872

Signature

Printed name obert E. Scheid for Stephen C. Durant

Date January 11, 2006 Reg. No. 42,126 for 31,506

PTOISB/17 (12-04v2)Approved for use through 713112006. OMB 0651-0032

U.S. Patent and Trademark Office: U.S. DEPARTMENT OF COMMERCEUnder the Paperwork Reduction Act of 1995, noneror are reuired to resoond to a collection of information .nlres it dislnay avairl OMR 'ntrol numher

Tee pursuan ote Mowat a ppropjnuons Act, 2 (H J. ). M.

FEE TRANSMITTALFor FY 2005

SApplicant claims small entity status. See 37 CFR 1.27

TOTAL AMOUNT OF PAYMENT ($) 665.00

1. BASIC FILING, SEARCH, AND EXAMINATION FEESFILING FEES

Small Entity

.Application NumberFiling DateFirst Named InventorExaminer Name

Art Unit

SEARCH FEESSmall Entity

Application Type Fee (t Fe Fee($1) FeJUtility 300 150 500 250Design 200 100 100 50Plant 200 100 300 150Reissue 300 150 500 250Provisional 200 100 0 0

2. EXCESS CLAIM FEESFee DescriptionEach claim over 20 (including Reissues)Each independent claim over 3 (including Reissues)Multiple dependent claims

Total Claims Extra Claims Fee($ Fee Paid ($)20 -= 0 x 0 = 0

I

11/231,353

September 19, 2005Ken Prayoon CHENGNot Yet Assigned2621

EXAMINATION FEESSmall Entity

Fee ) . Fee st200 100

130 65160 80

600 300

0 0

Fees Paid ($)500.00

Small EntityFee (t Fee(S

50 25

200 100

360 180

Multiple Dependent ClaimsFee Fee Paid ($S)

0 0lndep. Claims Extra Claims Fee(S) Fee Paid ($)

3 -= 0 x 0 03. APPLICATION SIZE FEE

If the specification and drawings exceed 100 sheets of paper (excluding electronically filed sequence or computerlistings under 37 CFR 1.52(e)), the application size fee due is $250 ($125 for small entity) for each additional 50sheets or fraction thereof. See 35 U.S.C. 41(a)(1)(G) and 37 CFR I.16(s).Total Sheets Extra Sheets Number of each additional 50 or fraction thereof Fee(S) Fee P

- 100 = 150 (round up to a whole number) x =-

4. OTHER FEE(S) Fees PNon-English Specification, $130 fec (no small entity discount)Other (e.g., late filing surcharge): 8021 Recording each patent assignment, agreement or ... 40.

2251 Extension for response within first month 60.2051 Surcharge - late filing fee, search fee, etc. 65.

' Robert E. Scheid (Reg. No. 42,216)for Stephen C. Durant (Rea. No. 31,506)

aid ($)

aid ($)

000000

Date

sf-2062504 vi

0A 'R

Complete if KnownEffective on 127/08/2004.

s usat to the Cmc #ad dnn ain f r i M 005 R H R1

577832000200Attorney Docket No.

METHOD OF PAYMENT (check all that apply)

ICheck D Credit Card Money Order iNone L Other (please identify):

W Deposit Account Deposit Account Number 03-1952 Depost Account Name: Morrison & Foerster LLP

For the above-identified deposit account, the Director is hereby authorized to: (check all that apply)

I Charge fee(s) indicated below ICharge fee(s) indicated below, except for the filing fee

Charge any additional fee(s) or underpayment of Credit any overpaymentsfee(s) under 37 CFR 1.16 and 1.17

FEE CALCULATION

I hereby certify that this correspondence is being deposited with the U.S. Postal Service with sufficient postage as First Class Mail, inan envelope addressed to: Box MISSING PARTS. Commissionerr Patents, P. 1450, Alexandria, Virginia 22313-1450, on thedate shown below.

Dated: January 11, 2006 Signature: Todd V. Leone

. ... r... ..

i

VIIVQ -uIQ r 01A ll ---uuuN -W VI jv. , 'IV '-Q--lN QI Iw-ull- 1 1 O r r ld l 1 .

I

qCPTO/SB/22 (12-04)SL ] 0 6 Approved for use through 7/31/2006. OMB 0651-0031JAz . . . U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE

4.1 Under the Paperwork Reduction Act of 1995. no rersons are reauired to resoond to a collection of infrmationunle if dinlavs a valid OMR nntrnl numhprriI .. ..... -... ..... . ...... - r .. ... ... 1 -.... ..r ....u ,a y , p . .... ... ..... ... ........ m N lr lw lU R JJ I p r' d Y.. ..... ..... .... ........PETITION FOR EXTENSION OF TIME UNDER 37 CFR 1.136(a) Docket Number (Optional)

FY 2005 577832000200(Fees pursuant to the Consolidated Appropriations Act, 2005'(H.R. 4818).)

Application Number 11/231,353 Filed September 19, 2005

ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION ANDFor SURVEILLANCE SYSTEMS

Art Unit 2621 Examiner Not Yet Assigned

This is a request under the provisions of 37 CFR 1.136(a) to extend the period for filing a reply in the aboveidentified application.

The requested extension and fee are as follows

OFII-IO

(check time period desired and enter the appropriate fee below):

Fee Small Entity FeeOne month (37 CFR 1.17(a)(1)) $120 $60 $ 60.00

D Two months (37 CFR 1.17(a)(2)) $450 $225 $

D Three months (37 CFR 1.17(a)(3)) $1020 $510 $

D Four months (37 CFR 1.17(a)(4)) $1590 $795 $

D Five months (37 CFR 1.17(a)(5)) $2160 $1080 $

Applicant claims small entity status. See 37 CFR 1.27.

A check in the amount of the fee is enclosed.

Payment by credit card. Form PTO-2038 is attached.

The Director has already been authorized to charge fees in this application to a Deposit Account.

The Director is hereby authorized to charge any fees which may be required, or credit any overpayment, toDeposit Account Number 03-1952 c.. . Fee

Transmittal form (PTO/SB/17) is attached to thissubmission in duplicate.

I am the i

DD

applicant/inventor.

assignee of record of the entire interest. See 37 CFR 3.71.Statement under 37 CFR 3.73(b) is enclosed. (Form PTO/SB/96).

attorney or agent of record. Registration Number

see below

January 11, 2006

Robert E. Scheid - Reg. No. 42,126for Stephen C. Durant - Reg. No. 31,506

Date

(415) 268-6982Typed or printed name Telephone Number

NOTE: Signatures of all the inventors or assignees of record of the entire interest or their representative(s) are required. Submit multiple forms if morethan one signature is required, see below.

S Total of 1 forms are submitted.

01/17/2006 NNGUYEN1 00000032 031952

05 FC:2251 60.00 DA

11231353

sf-2062483 v1

I hereby certify that this correspondence is being deposited with the U.S. Postal Service with sufficient postage as First Class Mail, inan envelope addressed to: Mail Stop MISSING PARTS. Commis for P.O. Box 1450, Alexandria, Virginia 22313-1450,on the date shown below. • ,

Dated: January 11, .2006 Signature: ' f. Y 4-7 Todd V. Leone

li231353

\k8G7.

k I A

a

Appl. No.: 11/231,353 Docket No.: 577832000200Inventor: Ken P. CHENG et al.Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC

IDENTIFICATION DETECTION AND SURVEILLANCESYSTEMS

REPLACEMENT SHEET Sheet 1 of 8)

BEST AVAILABLE COPY

'1

c .

0-

r t0

C.

0

0

.0j

'ai

r-,mD-I

-I-U

3DAI-


IDENTIFICATION DETECTION AND SURVEILLANCESYSTEMSH

REPLACEMENT SHEET. Sheet 2 of 8.,

*.

N

n0

C

y

MO

mm

0

0.2

::,.:.

Appl. No.: 11/231,353 Docket No.: 5778320002CInventor: Ken P. CHENG et al.Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC


REPLACEMENT SHEET Sheet 3 o

-ns~

-- - - - - - - - - N00

f 8

*0

0

-I

0

Ch,

MIL

2)'

Appl. No.: 11/231,353 Docket No.: 5778320002CInventor: Ken P. CHENG et al.Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC


REPLACEMENT SHEET Sheet 4 o

'1

00

f 8

41 e

3 egib

s'

A

e

s

U

- ------------- \

A'v

1

c


IDENTIFICATION DETECTION AND SURVEILLANCE.SYSTEMSH

REPLACEMENT SHEET Sheet 5 of 8

- -.. .. . . . . . . . . . .. . . . . .. . . . . . . . . . . . . . . .. . . . . . . .

0

-MI

'1- I




-T

O

F0

CD

W

0v

-'1

O0f

C7

gIITHI

!IIlla

111111~a

>1

D~

'1

~

a,o 5

e"I

Appl. No.: 11/231,353 DocKet No.: 5778320002(inventor: Ken P. CHENG et al.Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC


REPLACEMENT SHEET Sheet 7 of

a

8a . t'

0 0

ti

0

0tK

km

N,

'1

0*

%B

if 8)I

c

.... ,_________ -_00

mp&-'

Appl. No.: 11/231,353 - Docket No.: 577832000200 "

Inventor: Ken P. CHENG et al.Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC



'D

I Iiacari;[a:ima

.1000:luflg[a a1tii.1aisai

IIijlI#E4aalr:uII

Z

I

ao = =.....= ...;

0 w .6"

.,

4

Iif

fta

C

.1

O6 q

- .J4.1Ni

M~ *D

"1

,mt

1 IJJ'-L BII

'1.mU( 0

zElI!

Di

II

ir

Pao/sa~o 0446)

U. patentOWTre wmnk ofs: U.& oEPARTMWNT ofCooMME1EUrdetim Ppeiwoil eucs o aa1 9& no en m . tei soob"ob~n t. n u f w l tyvbo " v~aleoe nr~ut.

I hereby revoke all previous powers of attorney given i the application Iderntfled in tme attached statement under37 CFR 3.73(b).

OR[]Ptsutlonei N) mined below (MVopncwUln paenet practUoners an lb be named, then a cia Mmremer must be useak

as eftomunju) .ror n es) to repen t dMuelpned ba~ wlmftmitd Stems Patent nd Tademk OIke (USPTO) In connection pmeny and ci pausal eppir , s sae enad Op tb Mtie w~aeilm oem e ie b USPTO essigrran ns.cds oraseipgment doanits

aced to toInn n m cewndance wth 37CFR 3.73(0).

Plasea dienge tiehe snadenca address for the .ppleadon ldutlld hI Vwattahedulaentunder 37 CFR 3.73(b) to:

j] The addsussinodhted wthCusbmwerNumber: IIF~inne

p ky1 Istus IName

Caunby Telephone EmmA

Aeskpm Name and Addmes:Proxkmx Corptxation6 Results WayCuperdM.oCaIfOmia 95014

A copy of fhb forut opethe wilh a .beeenulmde 37 CFR 3.73(b) (Form PTOI88IS or eqiduland) Is reikd to beflied in each epptlcadtlon hiwfldhOft sarmnia used. The iemnt aider 37 CFR LT7*() may be canylated by on. of

Ohe psactdo m ar it-kItahis fore N the appointed preditloner Insmsarked toasWon behalf of the asslgneaaawn t miat le Inwhich Viii Power of Attorney Is to be fiLed

The invidwiJ wnose seta- a nd rite hksuqplied beow.s anmht d aon behalf ofihe assignee

Nane NaneRe

Signature I AIL (I&99.-ADate 2L6 osNamfe k KW PRq OoW -"Telephone Lfp-'7 g-TWOe ge

Nam

f

POWER OF ATTORNEY TO PROSECUTE APPUCA710NS BEFORE THE USPTO

I

I hereby certify that this correspondence is being deposited with the U.S. PostalService with sufficient postage as First Class Mail, in an envelope addressed to:Mail Stop MISSING PARTS, Commissioner for Patents P . Box 1450,Alexandria. Virginia 22313-1450, on the date s n w.

Dated: January 11, 2006 Signature:Todd V. Leone

Docket No.: 577832000200(PATENT)

JAN 1 3 2006 IN THE UNITED

e Patent Application ofKen Prayoon CHENG et al.

STATES PATENT AND TRADEMARK OFFICE

Application No.: 11/231,353

Filed: September 19, 2005

For: ADAPTIVE MULTI-MODAL INTEGRATEDBIOMETRIC IDENTIFICATION DETECTIONAND SURVEILLANCE SYSTEMS

Confirmation Number: 4531

Art Unit: 2621

Examiner: Not Yet Assigned

SUBMISSION OF DRAWINGS

Mail Stop MISSING PARTSCommissioner for PatentsP.O. Box 1450Alexandria, Virginia 22313-1450

Dear Sir:

Submitted herewith is one set (eight sheets, eight figures) of drawings for filing in the

above-identified Patent application. Kindly substitute the enclosed drawings for the drawings

submitted with the originally filed application.

Dated: January 11, 2006 Respectfully submitted,

B/ Robert E. Scheid - Reg. No. 42,126

for Stephen C. Durant - Reg. No.: 31,506MORRISON & FOERSTER LLP425 Market StreetSan Francisco, California 94105-2482Tel.: (415) 268-6982Fax: (415) 268-7522

sf-2062451 vI

o'-0-PF &C

of 2

AND TRADEMARK OFFICE)P COMI51ERCEark ofic

www.uptlo'v-

APPLICATION NUMBER FILING OR 371 (c)DATE FIRST NAMED APPLICANT ATTORNEY DOCKET NUMBER


CONFIRMATION NO. 453120872 FORMALITIESMORRISON & FOERSTER LLP LETTER425 MARKET STREETSAN FRANCISCO, CA 94105-2482


NOTICE TO FILE MISSING PARTS OF NONPROVISIONAL APPLICATION01/17/2006 NNGUYEHN1 00000032 031952 11231353 FILED UNDER 37 CFR 1.53(b)FILED UNDER 37 CFR 1.53(b)

01 FC:2011 150.00 DA02 FC:2051 65.00 DA Filing Date Granted03 FC:2111 250.00 DA Filing Date Granted04 FC:2311 100.00 DA

Items Required To Avoid Abandonment:

An application number and filing date have been accorded to this application. The item(s) indicated below,however, are missing. Applicant is given TWO MONTHS from the date of this Notice within which to file allrequired items and pay any fees required below to avoid abandonment. Extensions of time may be obtained byfiling a petition accompanied by the extension fee under the provisions of 37 CFR 1.136(a).

* The statutory basic filing fee is missing.Applicant must submit $ 150 to complete the basic filing fee for a small entity.

* The oath or declaration is missing. A properly signed oath or declaration in compliance with 37 CFR 1.63,identifying the application by the above Application Number and Filing Date, is required.Note: If a petition under 37 CFR 1.47 is being filed, an oath or declaration in compliance with 37 CFR 1.63signed by all available joint inventors, or if no inventor is available by a party with sufficient proprietaryinterest, is required.

The application is informal since it does not comply with the regulations for the reason(s) indicated below.


* Replacement drawings in compliance with 37 CFR 1.84 and 37 CFR 1.121(d) are required. The drawingssubmitted are not acceptable because:

The drawings submitted to the Office are not electronically reproducible. Drawingsheets must be submitted on paper, which is flexible, strong, white, smooth, non-shiny,and durable (see 37 CFR 1.84(e)). See Figure(s) 1 - 8.

Applicant is cautioned that correction of the above items may cause the specification and drawings page count toexceed 100 pages. If the specification and drawings exceed 100 pages, applicant will need to submit the requiredapplication size fee.

The applicant needs to satisfy supplemental fees problems indicated below.

of 2


* To avoid abandonment, a surcharge (for.late submission of filing fee, search fee, examination fee or oath ordeclaration) as set forth in 37 CFR 1.16(f) of $65 for a small entity in compliance with 37 CFR 1.27, must besubmitted with the missing items identified in this letter.

SUMMARY OF FEES DUE:

Total additional fee(s) required for this application is $565 for a Small Entity

* $150 Statutory basic filing fee.* $65 Surcharge.

* The application search fee has not been paid. Applicant must submit $250 to complete the search fee.* The application examination fee has not been paid. Applicant must submit $100 to complete the

examination fee for a small entity in compliance with 37 CFR 1.27

Replies should be mailed to: Mail Stop Missing Parts

Commissioner for Patents

P.O. Box 1450

Alexandria VA 22313-1450

A copy of this notice MUST be returned with the reply.

Office n', atent Examination (571) 272-4000, or 1-800-PTO-9199, or 1-800-972-6382PART 2 - COPY TO BE RETURNED WITH RESPONSE

JN1IPTO/SBl95 (09-04)

Approved for use through 07/31/2006. OMB 0651-0031U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE

work Reduction Act of 1995. no persons are required to respond to a collection of information unless it disiays a vad OMB control number

STATEMENT UNDER 37 CFR 3.73(b)

Applicant/Patent Owner: Ken Prayoon CHENG et al.

Application No.IPatent No.: 11/231,353 Filed/Issue Date: September 19, 2005

ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION ANDEntitled: SURVEILLANCE SYSTEMS

Proximex corporation(Name of Assignee) (Type of Assignee, e.g., corporation, partnership, university, government agency etc

states that it is:

1. j the assignee of the entire right, title, and interest; or

2. an assignee of less than the entire right, title and interest.The extent (by percentage) of its ownership interest is %

in the patent application/patent identified above by virtue of either:

A. R An assignment from the inventor(s) of the patent application/patent identified above. The assignmentwas recorded In the United States Patent and Trademark Office at Reel,Frame , or for which a copy thereof is attached.

OR

B. A chain of title from the inventor(s), of the patent application/patent identified above, to the currentassignee as shown below:

1. From: To:

The document was recorded in the United States Patent and Trademark Office atReel , Frame , or for which a copy thereof is attached.

2. From: To:


3. From: To:


Additional documents in the chain of title are listed on a supplemental sheet.

W Copies of assignments or other documents in the chain of title are attached.[NOTE: A separate copy (i.e., a true copy of the original assignment document(s)) must besubmitted to Assignment Division in accordance with 37 CFR Part 3, if the assignment is to berecorded in the records of the USPTO. See MPEP 302.08]

The undersigned tit )f elow) is authorized to act on behalf of the assignee.

. January 11. 2006Signature

Robert E. Scheid - Reg. No. 42,126fnr StPnh.n C ulrant - RPn Nn '1 .snR'

Printed or Typed Name

Attomey or agent under 37 CFR 1.34Title

Date

(415) 268-6982Telephone Number

sf-2062491 v1

I hereby certify that this correspondence is being deposited with the U.S. Postal Service with sfficient postage as First Glass Mail, in

c.)

Attorney Docket No.: 577832000200

ASSIGNMENTJOINT

THIS ASSIGNMENT, by Ken Prayoon CHENG, Edward Y. CHANG, and Yuan-Fang WANG(hereinafter referred to as the assignors), residing at 20691 Reid Lane, Saratoga, CA 95070; 816 Dorado Drive,Santa Barbara, CA 93111; and 5849 Via Fiori Lane, Goleta, CA 93117, respectively, witnesseth:

WHEREAS, said assignor has invented certain new and useful improvements in ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEMS,set forth in an application for Letters Patent of the United States, bearing Serial No. 11/231,353 and filed onSeptember 19, 2005; and

WHEREAS, Proximex Coroporation, a corporation duly organized under and pursuant to the laws ofCalifornia and having its principal place of business at 6 Results Way, Cupertino, CA 95014 (hereinafter referredto as the assignee) is desirous of acquiring the entire right, title and interest in and to said inventions and saidapplication for Letters Patent of the United States, and in and to any Letters Patent or Patents, United States orforeign, to be obtained therefor and thereon:

NOW, THEREFORE, in consideration of One Dollar ($1.00) and other good and sufficient consideration,the receipt of which is hereby acknowledged, said assignor has sold, assigned, transferred and set over, and bythese presents does sell, assign, transfer and set over, unto said assignee*, its successors, legal representatives andassigns, the entire right, title and interest in and to the above-mentioned inventions, application for Letters Patent,and any and all Letters Patent or Patents in the United States of America and all foreign countries which may begranted therefor and thereon, and in and to any and all divisions, continuations and continuations-in-part of saidapplication, or reissues or extensions of said Letters Patent or Patents, and all rights under the InternationalConvention for the Protection of Industrial Property, the same to be held and enjoyed by said assignee, for its ownuse and the use of its successors, legal representatives and assigns, to the full end of the term or terms for whichLetters Patent or Patents may be granted, as fully and entirely as the same would have been held and enjoyed bythe assignor, had this sale and assignment not been made.

AND for the same consideration, said assignor hereby covenants and agrees to and with said assignee itssuccessors, legal representatives and assigns, that, at the time of execution and delivery of these presents, saidassignor is the sole and lawful owner of the entire right, title and interest in and to said inventions and theapplication for Letters Patent above-mentioned, and that the same are unencumbered and that said assignor hasgood and full right and lawful authority to sell and convey the same in the manner herein set forth.

AND for the same consideration, said assignor hereby covenants and agrees to and with said assignee, itssuccessors, legal representatives and assigns, that said assignor will, whenever counsel of said assignee, or thecounsel of its successor, legal representatives and assigns, shall advise that any proceeding in connection with saidinventions, or said application for Letters Patent, or any proceeding in connection with Letters Patent for saidinventions in any country, including interference proceedings, is lawful and desirable, or that any division,continuation or continuation-in-part of any application for Letters Patent or any reissue or extension of any LettersPatent, to be obtained thereon, is lawful and desirable, sign all papers and documents, take all lawful oaths, and doall acts necessary or required to be done for the procurement, maintenance, enforcement and defense of LettersPatent for said inventions, without charge to said assignee, its successors, legal representatives and assigns, but atthe cost and expense of said assignee, its successors, legal representatives and assigns.

1sf-2045551

Attoray Docket No.: 577832000200

AND said assignor hereby requests the Commissionar of Pata nts to issue said Letters Patent of the United

States to said assignee as the assignee of said inventions and the Letters Patent to be issued theron for the sole use

of said assignee, its suocessos, legal representiv and assigns.

DatI4/o5

Daft

Dat

KEso Payoon CHENG

Edward Y. CHANG

Yen-Fang WANG

sf-2045551

J

1W


AND said assignor hereby requests the Commissioner of Patents to issue said Letters Patent of the UnitedStates to said assignee as the assignee of said inventions and the Letters Patent to be issued thereon for the sole useof said assignee, its successors, legal representatives and assigns.

8Ken Prayoon CHENG

Edward Y. CHANG

Yuan-Fang WANG

sf-2045551

-J

CD~

Date

/DateDate

Date


AND said assignor hereby requests the Commissioner of Patents to issue said Letters Patent of the UnitedStates to said assignee as the assignee of said inventions and the Letters Patent to be issued thereon for the sole useof said assignee, its successors, legal representatives and assigns.

Date

Date

DateDate

Ken Prayoon CHENG

EdwaY. CHANG

u -Fang WANG

sf-2045551

JaPTOr58l01A (08a4)

Appiored br us. ftoughO1A1r WS OM; OSS14MU.S Poft end Taidam* Oft*; UAL DEPARTMENT OF COMMERCE

Roo cfonAc d t9gkimpmemcnii edb bl aarc9aldlnbrma0onv+le Ro cs a Oe18alw ho nuRba.

FULL NAME OF INVENTOR(S)

Inventor one Ken Pr CHENG

Sinatue: __ Ctizen of United States

Inventor two: Edward Y. CHANG

Snatur: Cuzen ot United States

Inwentor tree: Yuan-FanR WANG

Slpate:C aitzen oa United States

gaato Otur f:

AdBla nvesoroma Wga emstamve ebeb ned an adlona (s) aaed heto.

Page li

,-2045545

L

t:

DECLARATION (37 CFR 1.63) FOR UTIL OR DESIGN APPUCATION USING ANAPPLICATION DATA SHEET (37 CFR 1.76)

ADAPTIVE MULTMODAL INTEGRATED BIOMETRIC IDENTIFICATION@f IIVS DETECTION AND SURVEILLANCE SYSTEMS

As he below named kwentur(s), Uwe declaretant

Tes dedaralin is directed to

EJ me attached aonab[ AppocaOn Na 11/231,353 , filed an 0911912005

El n amended an (H1 Ofawble)

We bdereva ht w e ainme o okinal md list Irwtat") t te aut~ec ma r whimch is c~imed and for wch a patent

Ue have reviews and wderstand fte cnts of tabo ied appilcat Ictudiog clak, as amended by wayamendment specIcaly rwfened to above;

e anowlaedge te dty to dtsclose to the Unted States Patent and Trademark Oface a if.omaon knw to mea s to bematedal to patanabMliy as da ined in 37 CFR 1.5, bctud t for mtinua*n ert appam mateeal i umn whi

became available been ie fig date n Oe pior applcaton and ft nationa or PCT htomallonal Bil data at Oecontinane alapicdn

AI stdltsments made herein d myowr owam knoxmlmde 4 e u, m staments made herein on bomatlon end belief marebeleved to be trum aid faeuat lease eatement were made thGe kntowedge that wM fale statements and the eare purnsatite by tie or ilmprimstw, or both, uder 18 U.S.C. 1001, and may jeopardbe the vslyid atthe appcation oraty paten ibsubigthemon.

~ ~VwY M w. .M ,-0WwVWz - wMmu 0

JAN 13 2006cPTO SB/1A (09-04)

" Apprmved for use through 07/31/2006. OMB 0851-0032U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE

r the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it dhp a valid OMB ontrol nurner.

FULL NAME OF INVENTOR(S)

Inventor one: Ken Prayoon CHENG

Signature: Citizen of: United States

Inventor two: Edward Y. CHANG

Signature: / .Citizen of:, United States

inventor three: Yuan-Fang WANG

Signature: Citizen of: United States

Inventor four.

Signature: Citizen of:

] Additional Inventors ora legal represenatve are being named on additional form(s) attached hereto.

Page 1 of I

sf-2045545

8

.J

CD-,

DECLARATION (37 CFR 1.63) FOR UTILITY OR DESIGN APPLICATION USING ANAPPLICATION DATA SHEET (37 CFR 1.76)

.... ..to ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATIONTitle of Invention DETECTION AND SURVEILLANCE SYSTEMS

As the below named inventor(s), I/we declare that:

This declaration Is directed to:

] The atlached application, or

W Application No. 11/231,353 , filed on 09/19/2005

[ as amended on (if applicable);

Iwe believe that I/we am/are the original and first Inventor(s) of the subject matter which is claimed and for which a patent issought;

I/we have reviewed and understand the contents of the above-identified application, including the claims, as amended by anyamendment specifically referred to above;

I/we acknowledge the duty to disclose to the United States Patent and Trademark Office all Information known to me/us to bematerial to patentability as defined in 37 CFR 1.56, including for continuation-in-part applications, material information whichbecame available between the filing date of the prior application and the national or PCT intemrnational filing date of thecontinuation-in-part application.

All statements made herein of my/our own knowledge are true, all statements made herein on information and belief arebelieved to be true, and further that these statements were made with the knowledge that willful false statements and the likeare punishable by fine or Imprisonment, or both, under 18 U.S.C. 1001, and may jeopardize the validity of the application orany patent issuing thereon.

091 n 00

o'P

JAM 13 ?006

u- i:dmimum db ae. o Lis: a rfl o

DECL:ARAlON p(37CFR 1.63)FOR IIT lfY OR'DEStGN :APPuCAT~1N UIG ViiAPPUCA1!ON DATASW (3 FR1J76

7ii~~o ir r:ADAP7 VE R1UL1Ti-MDALI N EGRA'TED:8WONE?'RC'IDENT1CATION 1_DETECTtON ANdD :SRVEtZ:ACE STEMS:

As' tbe Gow'ri18gi enlmtir s). M~se ears. a

LIII aeat1don ..........

Uet e heuweQamid~zwsaid ; cp~sd. ie"liaa Uid: aimr wzen: b. yy

!Aa p :: s geu12O d s ot e t ;~ii n~i t Kga ri orhirrt isi~ri adt~p,"rtby,: 161? im e al oiy is Pte1 a T ue t Sk nt~ia t 'Go ri.+ ds Z %

l ,, 1ad;"tlei:Cn me a ~e~ti p r o~d~a'r cu ; At? i:[T n re".wlpaCV .b~o r:td."2S8

AD 6W + . /1!4:q% ± : .A .. g" i ije - 'WR7

7a, { ajr bu orhrsrte;a.d,.iavir 18.t ;S:CS if eI 1;'andm oU a2 ev afd s c)i d m

!Paga1I cus -2045545

'. .-

Ca)

mmJw

IE4

au ism KenPraycon CHENG

__________________ U:1ni' ed Stats

p c Ediiard Y_. CHANG0

iln o Uem Y FangWANG ______________

ir4O f or muCprsco met a: na anl XIAA a

o'PJAN t g10 pp0lication Data Sheet

Application Information

Application Type::

Subject Matter::

Suggested Group Art Unit::

CD-ROM or CD-R?::

Sequence submission?::

Computer Readable Form (CRF)?::

Title::

Attorney Docket Number::

Request for Early Publication?::

Request for Non-Publication?::

Total Drawing Sheets::

Small Entity?::'

Petition included?::

Secrecy Order in Parent Appl.?::

Applicant Information



Status::

Given Name::

Middle Name::

Family Name::

City of Residence::






Regular

Utility

Not Yet Assigned

None

None

No

ADAPTIVE MULTI-MODAL INTEGRATEDBIOMETRIC IDENTIFICATIONDETECTION AND SURVEILLANCESYSTEMS577832000200

No

No

8

No

No

No

Inventor

US

Full Capacity

Ken

P.

CHENG

Saratoga

California

US

20691 Reid Lane

Saratoga

California

sf-2062435 v1 Page 1 initial 01111/2006



Status::

Given Name::

Middle Name::.

Family Name::

City of Residence::








Status::


Given Name::

Family Name::

City of Residence::







Correspondence Information

Correspondence Customer Number::

Inventor

US

Full Capacity

Edward

Y.

CHANG

Santa Barbara

California

US

20691 Reid Lane

Saratoga

California

95070-5325

Inventor

Full Capacity

US

Yuan-Fang

WANG

Goleta

California

US

5849 Via Fiori Lane

Goleta

California

93117-1838

20872

initial 01/11/2006Page 2sf-2062435 vl

Representative Information

Representative Customer Number::

Domestic Priority Information

Foreign Priority Information

Assignee Information

Assignee name::

initial 01/1112006

20872

Proximex

Page 3sf-2062435 v1

This Page is Inserted by IFW Indexing and ScanningOperations and is not part of the Official Record

BEST AVAILABLE IMAGES

Defective images within this document are accurate representations of the originaldocuments submitted by the applicant.

Defects in the images include but are not limited to the items checked:

Q BLACK BORDERS

Q IMAGE CUT OFF AT TOP, BOTTOM OR SIDES

O FADED TEXT OR DRAWING

Q BLURRED OR ILLEGIBLE TEXT OR DRAWING

O SKEWED/SLANTED IMAGES

COLOR OR BLACK AND WHITE PHOTOGRAPHS

] GRAY SCALE DOCUMENTS

O LINES OR MARKS ON ORIGINAL DOCUMENT

L REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY

D OTHER:

IMAGES ARE BEST AVAILABLE COPY.As rescanning these documents will not correct the imageproblems checked, please do not report these problems tothe IFW Image Problem Mailbox.


O Fee Transmittal Form,

D Fee Attached

SAmendment/Reply

D After Final

] Affidavits/declaration(s)

D Extension of Time Request

D Express Abandonment Request

x Information Disclosure Statement(5 pages)

] Certified Copy of PriorityDocument(s)

D Reply to Missing Parts/Incomplete Application

D Reply to Missing Parts under37 CFR 1.52 or 1.53

D Drawing(s)

D Licensing-related Papers

] Petition

SPetition to Convert to aProvisional Application

[] Power of Attorney, RevocationChange of Correspondence Address

D Terminal Disclaimer

I Request for Refund

CD, Number of CD(s)

D Landscape Table on CD

SAfter Allowance Communicationto TC

D Appeal Communication to Board ofAppeals and Interferences

SAppeal Communication to TC(Appeal Notice, Brief, Reply Brief)

L] Proprietary Information

SStatus Letter

Other Enclosure(s) (pleaseIdentify below):

(CD One (1) copy of each of twelve(12) cited non-patent literaturereterences

®) Return Receipt Postcard

Remarks

Total page count of this submission as cited above does not includethe enclosed copies of cited references.

SIGNATURE Cq APPLICANT, ATTORNEY, OR AGENT

Firm Name MO IS N & FOERST LLP Customer No. 20872

Signature

Printed name Stephen C. Durant

Date October 13, 2006 Reg. No. 31,506

sf-2209783 vi

I hereby certify that this paper (along with any paper referred to as being attached or enclosed) is being deposited with the U.S. Postal Service onthe date shown below with sufficient postage as First Class Mail, in envelope dressed to: Mail Stop Amendment, Commissioner forPatents. P.O. Box 1450, Alexandria, Virginia 22313-1450.

Dated: October 13. 2006 Signature:Todd V. Leone

1'

I

SPaperwork Reduction Act of 1995, no persons are required to res

TRANSMITTALFORM


Total Number of Pages in This Submission 6

PTO/SB/21 (09-04)Approved for use through 0713112006. OMB 0651-0031

U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCEond to a conltection of information unless it disolays a valid OMB control number.

Application Number

Filing Date

First Named Inventor

Art Unit

Examiner Name

Attomey Docket Number

11/231,353

September 19, 2005

Ken P. CHENG.

2621

Not Yet Assigned

577832000200

sf-2209783 v1


O Fee Transmittal Form Drawing(s) After Allowance CommunicationL uto TC

SFee Attached O Licensing-related Papers Appeal Communication to Board ofAppeals and Interferences

] Amendment/Reply F Petition Appeal Communication to TCeLi(Appeal Notice, Brief, Reply Brief)

j After Final [ Petition to Convert to a Proprietary InformationProvisional Application

[ Affidavitsdeclaration(s) Power ofAttomey, Revocationess D Status Letter

Change of Correspondence Address FSExtension of Time Request: Terminal Disclaimer -OtherEnclosure(s) (please

Express Abandonment Request 0 Request for Refund D One (1) copy of each of twelve(12) cited non-patent literatureO Information Disclosure Statement CD, Number of CD(s) references

X(5 pages) CD, Number of CD(s)

® Return Receipt Postcard

F Certified Copy of Priority Landscape Table on CDDocument(s)

SReply to Missing Parts/ Remarks

Incomplete Application

] Reply to Missing Parts under Total page count of this submission as cited above does not include37 CFR 1.52 or 1.53 the enclosed copies-of cited references.

SIGNATURE9 APPLICANT, ATTORNEY, OR AGENT

Firm Name MO IS N & FOERST LLP Customer No. 20872

Signature

Printed name Stephen C. Durant


I hereby certify that this paper (along with any paper referred to as being attached or enclosed) is being deposited with the U.S. Postal Service onthe date shown below with sufficient postage as First Class Mail, in envelope adressed to: Mail Stop Amendment. Commissioner forPatents, P.O. Box 1450, Alexandria, Virginia 22313-1450.

Dated: October 13. 2006 Signature:Todd V. Leone

_ . _ . ,._ .I .. .. . .IIG I r--, . . . . . .... . .. .. .... . ... .

1'

r

:;;r

L .

I

I hereby certify that this correspondence and any enclosures referenced thereineing deposited with the U.S. Postal Service on the date first shown belowsufficient postage as First Class Mail, in an envelope addressed to: MailAmendment. Commissioner for Patents, P.O. Box 140, Alexandria,

lia 2313-1450.

: October 13, 2006 Signature:

PatentDocket No. 577832000200

IN THE UNITED STATES PATENT AND TRADEMARK OFFICE

In re Patent Application of:Ken P. CHENG et al.

Serial No.: 11/231,353

Filing Date: September 19, 2005

For: ADAPTIVE MULTI-MODALINTEGRATED BIOMETRICIDENTIFICATION DETECTIONAND SURVEILLANCE SYSTEMS


Group Art Unit: 2621

INFORMATION DISCLOSURESTATEMENT UNDER 37 C.F.R. § 1.97 & 1.98

Mail Stop AMENDMENTCommissioner for PatentsP.O. Box 1450Alexandria, Virginia 22313-1450

Dear Sir:

Pursuant to 37 C.F.R. § 1.97 and § 1.98, Applicants submit for consideration in the

above-identified application the documents listed on the attached Form PTO/SB/08a/b. Copies of

foreign documents and non-patent literature are submitted herewith. Document no. 10 on the

attached Form PTO/SB/O8a/b is a pending non-published U.S. patent application, and in accordance

with the Waiver of the Copy Requirement in 37 CFR 1.98 for Cited Pending U.S. Patent

Applications, a copy is not submitted herewith. The Examiner is requested to make these documents

of record.

sf-2185936

II

Application No. 11/231,353 .2Filed: 09/19/2005Information Disclosure Statement

This Information Disclosure Statement is submitted:

Docket No. 577832000200

I With the application; accordingly, no fee or separate requirements are required.

O Before the mailing of a first Office Action after the filing of a Request for Continued

Examination under § 1.114. However, if applicable, a certification under 37 C.F.R. § 1.97

(e)(1) has been provided.

® Within three months of the application filing date or before mailing of a first Office

Action on the merits; accordingly, no fee or separate requirements are required.

However, if applicable, a certification under 37 C.F.R. § 1.97 (e)(1) has been provided.

L After receipt of a first Office Action on the merits but before mailing of a final Office Action

or Notice of Allowance.

L A fee is required. A check in the amount of is enclosed.

I A fee is required. Accordingly, a Fee Transmittal form (PTO/SB/17) is attached to

this submission in duplicate.

I A Certification under 37 C.F.R. § 1.97(e) is provided above; accordingly; no fee is

believed to be due.

I After mailing of a final Office Action or Notice of Allowance, but before payment of the

issue fee.

O A Certification under 37 C.F.R. § 1.97(e) is provided above and a check in the

amount of is enclosed.

L A Certification under 37 C.F.R. § 1.97(e) is provided above and a Fee Transmittal

form (PTO/SB/17 is attached to this submission in duplicate.)

Applicants would appreciate the Examiner initialing and returning the Form

PTO/SB/08a/b, indicating that the information has been considered and made of record herein.

The information contained in this Information Disclosure Statement under

37 C.F.R. § 1.97 and § 1.98 is not to be construed as a representation that: (i) a complete search has

been made; (ii) additional information material to the examination of this application does not exist;

(iii) the information, protocols, results and the like reported by third parties are accurate or enabling;

or (iv) the above information constitutes prior art to the subject invention.

sf-2185936

Application No. 1 1/231,353 3 Docket No. 577832000200Filed: 09/19/2005Information Disclosure Statement

In the unlikely event that the transmittal form is separated from this document and the

Patent and Trademark Office determines that an extension and/or other relief (such as payment of a

fee under 37 C.F.R. § 1.17 (p)) is required, Applicants petition for any required relief including

extensions of time and authorize the Commissioner to charge the cost of such petition and/or other

fees due in connection with the filing of this document to Deposit Account No. 03-1952

referencing 577832000200.

Dated: O C ?c I I, dob Respectfully submitted,

Step en C. urantRegistration No.: 31,506

MORRISON & FOERSTER LLP425 Market StreetSan Francisco, California 94105-2482(415) 268-6982

sf-2185936

ALTERNATIVE TO PTO/SBlO8alb (07-05)

te for form 1449/PTO

ORMATION DISCLOSURESTATEMENT BY APPLICANT

(Use as many sheets as necessary)

Sheet 1 of 2

Application Number

Filing Date


Art Unit

Examiner Name


11/231,353

September 19, 2005

Ken P. CHENG

2621

Not Yet Assigned

577832000200

U.S. PATENT DOCUMENTSDocument Number Publication Date Name of Patentee or Pages. Columns. Lines, Where

Examiner Cite Publication Date Name of Patentee or Relevant Passages or RelevantInitials' No.' Number-Kind Code' (ifknown) MM-DD-YYYY Applicant of Cited Figures Appear

Document

1. US-6,591,224 07-08-2003 Sullivan et al.2. US-6,609,198 08-19-2003 Wood et al.3. US-6,697,103-B1 02-24-2004 Fernandez et al.4.- US-6,970,582-A1 11-29-2005 Langley5. US-20020138768-A1 09-26-2002 Murakami et al.6. US-20020190119-A1 12-19-2002 Huffman7. US-20040081338-A1 04-29-2004 Takenaka8. US-20050265607-A1 12-01-2005 Chang9. US-20060112039-A1 05-25-2006 Wang et al.


Examiner Cile Foreign Patent Document Publication Pages. Columns. Lines.Date , FName of Patentee or Where Relevant Passages

Initals* No.' Country Code'-Numbe'-Kind Codes(ifknown) MM-DD-YYYY Applicant of Cited Document or Relevant Figures Appear T

1 1I I I 1'EXAMINER: Initial if information considered, whether or not citation is in conformance with MPEP 609. Draw line through citation if not in conformance and notconsidered. Include copy of this form with next communication to applicant. ' Applicants unique citation designation number (optional). ' See Kinds Codes ofUSPTO Patent Documents at www.uspto.gov or MPEP 901.04. ' Enter Office that issued the document, by the two-letter code (WIPO Standard ST.3). ' ForJapanese patent documents, the indication of the year of the reign of the Emperor must precede the serial number of the patent document. 'Kind of document bythe appropriate symbots as indicated on the document under WIPO Standard ST. 16 if possible. Applicant is to place a check mark here if English languageTranslation is attached.

NON PATENT LITERATURE DOCUMENTSInclude name of the author (in CAPITAL LETTERS), title of the article (when appropriate), title of the item (book,

Examiner Cite magazine, journal, serial, symposium, catalog, etc.), date, page(s), volume-issue number(s), publisher, city TInitials No.and/or country where published.

10. Goh et al., "Robust Perceptual Color Identification" US-11/229,091,Filed: 9/16/2005

11. Belhumeur, A. et al. (1997). "Eigenfaces vs. Fisherfaces: recognitionusing class specific linear projection", IEEE Transactions on PatternAnalysis and Machine Intelligence 19(7): 711-720.

12. Brunelli, R. and D. Falavigna. (1995). "Person identification usingmultiple cues," IEEE Transactions on Pattern Analysis and MachineIntelligence 17(10): 955-966.

13. Brunelli, R. et al. (1995). "Automatic Person Recognition by UsingAcoustic and Geometric Features", Machine Vision and Applications8: 317-325.

14. HONG, Lin and Anil K. Jain. (1998). "Integrating faces and fingerprintsfor personal identification," IEEE Transactions on Pattern Analysis

sf- 2185930

Complete if Known

Examiner DateSignature I IConsidered

0

I

ALTERNATIVE TO PTO/SB/08ab (07-05)

Complete if KnownSubstitute for form 1449/PTO

Application Number 11/231,353

INFORMATION DISCLOSURE Filing Date September 19, 2005

STATEMENT BY APPLICANT FirstNamed Inventor Ken P. CHENGArt Unit 2621

(Use as many sheets as necessary) Examiner Name Not Yet Assigned

sheet 2 of 2 Attorney Docket Number 577832000200

and Machine Intelligence 20(12): 1295 - 1307.15. International Search Report mailed on April 2006 for PCT Patent

Application Number PCT/US05/33378 filed on September 19, 2005,one page.

16. JAIN, A. K. et al. (1997). "On-Line Fingerprint Verification," IEEETransactions on Pattern Analysis and Machine Intelligence archive19(4): 302- 314.

17. Kittler, J. et al. (1998). "On combining classifiers", IEEE Transactionson Pattern Analysis and Machine Intelligence 20(3): 226-239.

18. Lu X et al. (2003). "Combing classifiers for face recognition", IEEEInternational Conference on Multimedia Systems and Expo, Baltimore,MD, July.

19. Maio, D. et al. (2002). "FVC2000: fingerprint verification competition",IEEE Transactions on Pattern Analysis and Machine Intelligence 24(3):402 -412.

20. Phillips, P.J. et al. (2000). "The FERET evaluation methodology forface-recognition algorithms", IEEE Transactions on Pattern Analysisand Machine Intelligence 22(10): 1090- 1104.

21. Senior, A. (2001). "A combination fingerprint classifier", IEEETransactions on Pattern Analysis and Machine Intelligence 23(10): "1165 1174.

22. TURK, A. and A. Pentland. (1991). "Eigenfaces for Recognition".Journal of Cognitive Neuroscience 3 (1): 71-86.

'EXAMINER: Initial if reference considered, whether or not citation is in conformance with MPEP 609. Draw line through citation if not in conformance and not

considered. Include copy of this form with next communication to applicant.

'Applicanrts unique citation designation number (optional). 'Applicant Is to place a check mark here if English language Translation is attached.

ExaminerDateSignature Considered

sf- 2185930

L

Other Prior Art

According to the information contained in form PTO-1449 or PTO-892, there are one or more other

prior art/non-patent literature documents missing from the original file history record obtained from

the United States Patent and Trademark Office. Upon your request we will attempt to obtain these

documents from alternative resources. Please note that additional charges will apply for this service.

This page is not part of the official USPTO record. It has been determined that content identifiedon this document is missing from the original file history record.

V .,

1 INTRODUCTION

Within the last several years, numerous algorithms havebeen proposed for face recognition; for detailed surveys see[1), [2]. While much progress has been made toward recog-nizing faces under small variations in lighting, facial ex-pression and pose, reliable techniques for recognition undermore extreme variations have proven elusive.

In this paper, we outline a new approach for face recogni-tion--one that is insensitive to large variations in lightingand facial expressions. Note that lighting variability includesnot only intensity, but also direction and number of lightsources. As is evident from Fig. 1, the same person, with thesame facial expression, and seen from the same viewpoint;can appear dramatically different when light sources illumi-nate the face from different directions. See also Fig. 4.

Our approach to face recognition exploits two observations:

1) All of the images of a Lambertian surface, taken froma fixed viewpoint, but under varying illumination, liein a 3D linear subspace of the high-dimensional imagespace [3).

2) Because of regions of shadowing, specularities, andfacial expressions, the above observation does not ex-actly hold, In practice, certain regions of the face mayhave variability from image to image that often devi-ates significantly from the linear subspace, and, con-sequently, are less reliable for recognition.

We make use of these observations by finding a linear

projection of the faces from the high-dimensional image

* The authors are with the Center for Computational Vision and Control, Dept.of Electrical Engineering, Yale University, Newt Haven, CT 06520-8267.E-mail: lbethumneur, [email protected], [email protected].

Manuscript received 15 Feb. 1996 revised 20 Mar. 1997. Recommended for accep-tance by I. Daugman.For information on obtaining reprints of this article, please send e-mail to:[email protected], and reference IEEECS Log Number 104797.

space to a significantly lolve: dimensional feature spacewhich is insensitive both to variation in lighting directionand facial expression: We choose projection directions thatare nearly orthogonal to the within-class scatter, projectingaway variations in lighting mnd facial expression whilemaintaining discriminability. Our method Fisherfaces, aderivative of Fisher's Linear Discriminant (FLD) [4], [5),maximizes the ratio of between-class scatter to that ofwithin-class scatter.

The Eligenface method is also based on linearly project-ing the image space to a low c.imensional feature space [6),17), 181. However, the Eigenface method, which uses princi-pal components analysis (PCA) for dimensionality reduc-tion, yields projection directions that maximize the totalscatter across all classes, i.e., a ross all images of all faces. Inchoosing the projection which maximizes total scatter, PCAretains unwanted variations due to lighting and facialexpression. As illustrated in Figs. 1 and 4 and stated byMoses et al., "the variations between the images of the sameface due to illumination and viewing direction are almostalways larger than image var ations due to change in faceidentity" (9]. Thus, while the PCA projections are optimal

Fig. 1. The same person seen uncer different lighting conditions canappear dramatically different: In the left image, the dominant lightsource is nearly head-on; in the right image, the dominant light sourceis from above and to the right.

0162-828/97/1S10.00 0 1997 IEEE

Application No. 1 1/231,35371IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 7, JULY 1997 Application No. 11231,353 711

Docket No. 577832000200

SEigenfaces vs. Fisherfaces: RecognitionUsing Class Specific Linear Projection

Peter N. Belhumeur, Joio P. Hespanha, and David J. Krie ;man

Abstract-We develop a face recognition algorithm which is insensitive to large variation in lighting direction and lacial expression.Taking a pattern classilication approach, we consider each pixel in an image as a coordinate in a high-dimensional space. We lakeadvantage of the observation that the images of a particular face, under varying illumination but fixed pose, lie in a 3D linearsubspace of the highdimensional image space-if the face is a Lambertian surface without shadowing.. Iowever, since faces arenot truly Lambertian surfaces and do indeed produce self-shadowing, images will deviate from this linear subspace. Rather thanexplicitly modelingj this deviation, we linearly project the image into a subspace in a manner which discounts those regions of thelace with large deviation. Our projection method is based on Fisher's Linear Discriminant and produces vell separated classes in alow-dimensional subspace, even under severe variation In lighting and facial expressions. The Elgenface technique, another methodbased on linearly projecting the image space to a low dimensional subspace, has similar computational requirements. Yet, extensiveexperimental results demonstrate that the proposed "Fisherface" method has error rates that are lower than those of the Eigentacetechnique for tests on the Harvard and Yale Face Databases.

Index Terms-Appearance-based vision, face recognition, illuminatiorn invariance, Fisher's linear discriminant.

+,

I

I

712 IEEE TRANSACTIONS ON PATT

for reconstruction from a low, dimensional basis, they maynot be optimal from a discrimination standpoint. .

We should point out that Fisher's Linear Discriminant is

a "classical" technique in pattern recognition [4], first de-

veloped by Robert Fisher in 1936 for taxonomic classifica-tion [5]. Depending upon the features being used, it has

been applied in different ways in computer vision and evenin face recognition. Cheng et al. presented a method thatused Fisher's discriminator for face recognition, wherefeatures were obtained by a polar quantization of the shape[10]. Baker and Nayar have developed a theory of patternrejection which is based on a two class linear discriminant[11]. Contemporaneous with our work [12], Cui et al. appliedFisher's discriminator (using different terminology, they

call it the Most Discriminating Feature-MDF) in a methodfor recognizing hand gestures [13]. Though no implemen-tation is reported, they also suggest that the method can beapplied to face recognition under variable illumination.,In the sections to follow, we compare four methods for

face recognition under variation in lighting and facial ex-pression; correlation, a variant of the linear subspacemethod suggested by [3], the Eigenface method [6], [71, [8],and the Fisherface method developed here. The compari-sons are done using both a subset of the Harvard Database(330 images) [14], [15] and a database created at Yale (160images). In tests on both databases, the Fisherface methodhad lower error rates than any of the other three methods.Yet, no claim is made about the relative performance of .

these algorithms on much larger databases.We should also point out that we have made no attempt

to deal with variation in pose. An appearance-basedmethod such as ours can be extended to handle limitedpose variation using either a multiple-view representation,such as Pentland et al's. view-based Eigenspace [16] or Mu-rase and Nayar's appearance manifolds [171. Other ap-proaches to face recognition that accommodate pose varia-

tion include [18], [19], [20]. Furthermore, we assume that

the face has been located and aligned within the image, asthere are numerous methods for finding faces in scenes121], [221, [20], [231, [241, 125], [7].

2 IMETHODS

The problem can be simply stated: Given a set of face im-ages labeled with the person's identity (the learning set) andan unlabeled set of face images from the same group ofpeople (the test set), identify each person in the test images.

In this section, we examine four pattern classificationtechniques for solving the face recognition problem, com-paring methods that have become quite popular in the facerecognition literature, namely correlation [26] and Eigen-face methods 16], [7], [8], with alternative methods devel-oped by the authors. We approach this problem within thepattern classification paradigm, considering each of thepixel values in a sample image as a coordinate in a high-dimensional space (the image space).

2.1 CorrelationPerhaps, the. simplest classification scheme is a nearestneighbor classifier in the image space [26]. Under this

ERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 7, JULY 1997

scheme, an image in the test set.is recognized (classifiecV byassigning to it the label of the closest point in the learningset, where distances are measured in the image space. If all

of the images are normalized to have zero mean and unitvariance, .then this procedure is equivalent to choosing the

image in the learning set that best correlates with the test

image. Because of the normalization process, the result isindependent of light source intensity and-the effects of avideo camera's automatic gain control.

This procedure, which subsequently is referred to as cor-relation, has several well-known disadvantages. First, if theimages in the learning. set and test set are gathered undervarying lighting conditions, then the corresponding points

in the image space may not be tightly clustered. So, in order

for this method to work reliably under variations in light-ing; we would need a learning set which densely sampledthe continuum of possible lighting conditions. Second, cor-relation is computationally expensive. For.recognition, wemust correlate the image of the test face with each image inthe learning set; in an effort to reduce the computationtime, implementors [27] of the algorithm described in [26]developed special purpose VLSI hardware. Third, it re-quires large amounts of storage-the learning set mustcontain numerous images of each person. ,

2.2 Eigenfaces

As correlation methods are computationally expensive and

require great amounts of storage, it is natural to pursuedimensionality reduction schemes. A technique now com-monly used for dimensionality reduction in computer vi-sion-particularly in face recognition-is principal compo-nents analysis (PCA) [14), [17], [61, [7], [8]. PCA techniques,also known as Karhunen-Loeve methods, choose a dimen-sionality reducing linear projection that maximizes thescatter of all projected samples.

More formally, let us consider a set of N sample images

{x,x 2, ... , xN) taking values in an n-dimensional image

space, and assume that each image belongs to one of cclasses {X1,X 2 ... X,.. Let us also consider a linear trans-

formation mapping the original n-dimensional image spaceinto an m-dimensional feature space, whereim < n. The new

feature vectors Yk e IR"' are defined by the following lineartransformation:

Yk = WTk k =, 2,..., N

whete W E R""' is a matrix with orthonormal columns.If the total scatter matrix S. is defined as

N

ST = (Xk - 4Xk

k=1

where n is the number of sample images, and 4e R " is themean image of all samples, then after applying the linear

transformation WT, the scatter of the transformed feature

vectors {YtYY2 -...- YN}is WTSTW. In PCA, the projection

WeP is chosen to maximize the determinant of the total

scatter matrix of the projected samples, i.e.,

BELHUMEUR ET AL.: EIGENFACES VS. FISHERFACES: RECOGNITION USING CLASS SPECIFIC LINEAR PROJECTION

W0r,= arg maxIWTSTWI

= [ W2 ... wm]

where [w; i = 1, 2,..., m) is the set of n-dimensional eig

vectors of Sr corresponding to the m largest eigenvalu

Since these eigenvectors have the same dimension asoriginal images, they are referred to as Eigenpictures in

and Eigenfaces in (7], [8]. If classification is performeding a nearest neighbor classifier in the reduced feat

space and m is chosen to be the number of images N intraining set, then the Eigenface method is equivalent tocorrelation method in the previous section.

A drawback of this approach is that the scatter bemaximized is due not only to the between-class scatter thuseful for classification, but also to the within-class scathat, for classification purposes, is unwanted informat

Recall the comment by Moses et al. [9]: Much of the variafrom one image to the next is due to illumination chan

Thus if PCA is presented with images of faces under varyillumination, the projection matrix W, will contain pri

pal components (i.e., Eigenfaces) which retain, in the !jected feature space, the variation due lighting. Coiquently, the points in the projected space will not beclustered, and worse, the classes may be smeared together

It has been suggested that by discarding the three nsignificant principal components, the variation due

lighting is reduced. The hope is that if the first princcomponents capture the variation due to lighting,

better clustering of projected samples is achieved bynoring them. Yet, it is unlikely that the first several pripal components correspond solely to variation in lightas a consequence, information that is useful for discrimtion may be lost.

2.3 Linear SubspacesBoth correlation and the Eigenface method are expectesuffer under variation in lighting direction. Neither metexploits the observation that for a Lambertian surface A

out shadowing, the images of a particular face lie in alinear subspace.

Consider a point p on a Lambertian surface illumin

by a point light source at infinity. Let s e R3 be a colhvector signifying the product of the light source interwith the unit vector for the light source direction. Whersurface is viewed by a camera, the resulting image interof the point p is given by

E(p) = a(p)n(p)T s

where n(p) is the unit inward normal vector to the sui

at the point p, and a(p) is the albedo of the surface at pThis shows that the image intensity of the point p is li

on s E R3 . Therefore, in the absence of shadowing, gthree images of a Lambertian surface from the same vpoint taken under three.known, linearly, independentsource directions, the albedo and surface normal cairecovered, this is the well known method of photomstereo [29], [30]. Alternatively, one can reconstruct the

age of the surface under an arbitrary lighting direction by alinear combination of the three original images, see 13].

(2) For classification, this fact h.as great importance: It showsthat, for a fixed viewpoint, the images of a Lambertian sur-

en- face lie in a 3D linear subspace of the high-dimensional im-.age space. This observation suggests a simple classification

es. algorithm to recognize Lambe:tian surfaces-insensitive tothe a wide range of lighting conditions. .[61 For each face, use three or r.iore images taken under dif-us- ferent lighting directions to construct a 3D basis for the lin-ure ear subspace. Note that the three basis vectors have thethe- same dimensionality as the training images and can be

the thought of as basis images. 'o perform recognition, we

simply compute the distance cf a new image to each linear"ing subspace and choose the face corresponding to the shortestat is distance. We call this recogniion scheme the Linear Sub-tter space method. We should point out thai this method is aion. variant of the photometric alignment method proposed intion [3), and is a special case of th. more elaborate recognitionges. method described in [15]. Subsequently, Nayar and Murase'ing have exploited the apparent lir earity of lighting to augmentnci- their appearance manifold (31]

pro- If there is no noise or shacowing, the Linear Subspacenrise- algorithm would achieve error free classification under anyvell lighting conditions, provided the surfaces obey the Lam-

. bertian reflectance model. Nevertheless, there are severalnost compelling reasons to look elsewhere. First, due to self-Sto shadowing, specularities, and facial expressions, some re-

ipal gions in images of the face have variability that does not

hen agree with the linear subspace. model. Given enough im-

; ig- ages of faces, we should be able to learn which regions are

nci- good for recognition and-which regions are not. Second, toing; recognize a test image; we mu:;t measure thedistance to the

ina- linear subspace for each perscn. While this is an improve-ment over a correlation scherre that needs a large numberof images to represent the variability of each class, it iscomputationally expensive. Finally, from a storage stand-

d to point, the Linear Subspace alg;orithm must keep three im-hod ages in memory for every person.,ith-

3D 2.4 FisherfacesThe previous algorithm takes advantage of the fact that,

ated under admittedly idealized conditions, the variation within

umn class lies in a linear subspace of the image space. Hence, the

Isity classes are convex, and, therefore, linearly separable. One

the can perform dimensionality reduction using linear projec-

Isity tion and still preserve linear :eparability. This is a strongargument in favor of using linear methods for dimension-ality reduction in the face recognition problem, at least

(3) when one seeks insensitivity tc lighting conditions.

ace Since the learning set is labeled, it makes sense to useface this information to build a more reliable method for re-

[281. ducing the dimensionality of the feature space. Here wenear argue that using class specific: linear methods for dimen-

iven sionality reduction and simple classifiers in the reducediew- feature space, one may get better recognition rates thanlight with either the Linear Subspace.method or the Eigenfacen be method. Fisher's Linear Discri:ninant (FLD) [5) is an exam-etric ple of a class specific method, in the sense that it tries toSim- "shape" the scatter in order :o make it more reliable for

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 7, JULY 1997

classification. This method selects W in [1] in sucha vthat the ratio of the between-class scatter and the witclass scatter is maximized.

Let the between-class scatter matrix be defined asC

SA ° Ni X( i - " )(P i - )T.

i=1

and the within-class scatter matrix be defined as

Sw = (x - i)(xk - )T

i=1 xEX,

where p is the mean image of class Xi, and Ni is the n

ber of samples in class Xi . If Sw is nonsingular, the optil

projection Wop is chosen as the matrix with orthonor:

columns which maximizes the ratio of the determinanthe between-class scatter matrix of the projected samplethe determinant of the within-class scatter matrix ofprojected samples, i.e.,

1 1WSBWWopt = arg maxW W

= [w1 w 2 ... W ]

where {w i i = 1, 2,..., m} is the set of generalized ei

vectors of SA and SW corresponding to the m largest ge

alized eigenvalues {; I i = 1,2,..., m}, i.e.,

SBw; = ,A;Sww, i = 1,2,...,m

Note that there are at most c - 1 nonzero generalized eilvalues, and so an upper bound on m is c - 1, where c isnumber of classes. See [4].

To illustrate the benefits of class specific linear pr<tion, we constructed a low dimensional analogue toclassification problem in which the samples from.each <lie near a linear subspace. Fig. 2 is a comparison of Iand FLD for a two-class problem in which the samples feach class are randomly perturbed in a direction per]dicular to a linear subspace. For this example, N = 20, nand m = 1. So, the samples from each class lie near apassing through. the origin in the 2D feature space. 1PCA and FLD have been used to project the points frondown to ID. Comparing the two projections in the figPCA actually smears the classes together so that they arelonger linearly separable in the projected space. It isthat, although PCA achieves larger total scatter,achieves greater between-class scatter, and, consequeclassification is simplified.

In the face recognition problem, one is confronted

the difficulty that the within-class scatter matrix Sw e

is always singular. This stems from the fact that the rar

Sw is at most N - c, and, in general, the number of imin the learning set N is much smaller than the numbspixels in each image n. This means that it is possibchoose the matrix W such that the within-class scatter aprojected samples can be made exactly zero.

In order to overcome the complication of a singular

we propose an alternative to the criterion in (4).

o

i

..'

4: 4

the 0feature 1

Fig. 2. A comparison of principal component analysis (PCA) andFisher's linear discriminant (FLD) for a two class problem where datafor each class lies near a linear subspace.

(4) method, which we call Fisherfaces, avoids this problem byprojecting the image set to a lower dimensional space so

gen- that the resulting within-class scatter matrix Sw is nonsin-

ner- gular. This is achieved by using PCA to reduce the dimen-

sion of the feature space to N - c, and then applying the

standard FLD defined by (4) to reduce the dimension to c - 1.More formally, Wopt is given by

- -n

WT WT Tt Wfld pea (5) .

where

W,= arg mrnaxiWTSrW

WTWTaSWcapWfld = arg max TT

Note that the optimization for Wp is performed over

n x (N - c) matrices with orthonormal columns, while the

optimization for Wfld is performed over (N - c) x m matrices

with orthonormal columns. In computing Wp, we have

thrown away only thesmallest c - 1 principal components.There are certainly other ways of reducing the within-

class scatter while preserving between-class scatter. Forexample, a second method which we are currently investi-gating chooses W to maximize the between-class scatter ofthe projected samples after having first reduced the within-class scatter. Taken to an extreme, we can maximize thebetween-class scatter of the projected samples subject to theconstraint that the within-class scatter is zero, i.e.,

W0pt = arg maxWTSW

where "W is the set of n x m matrices with orthonormal col-umns contained in the kernel of S,.

/

* 1~1

. . . . . . . . . . . .

+ / . 0

,o

b 0 classi1\4 + :class 2

EIM BEST AVAILABLE COPYBELHUMEUR ET AL.: EIGENFACES VS. FISHERFACES: RECOGNITION USING CLASS SPECIFIC LINEAR PROJECTION

3 EXPERIMENTAL RESULTS

in this section, we present and discuss each of the afore-mentioned face recognition techniques using two differentdatabases. Because of the specific hypotheses that .wewanted to test about the relative performance of the consid-ered algorithms, many of 'the standard databases were in-appropriate. So, we have used a database from the HarvardRobotics Laboratory in which lighting has been systemati-cally varied. Secondly, we have constructed a database atYale that includes variation in both facial expression andlighting. 1

3.1 Variation in LightingThe first experiment was designed to test the hypothesisthat under variable illumination, face recognition algo-rithmins will perform better if they exploit the fact that im-ages of a Lambertian surface lie in a linear subspace. Morespecifically, the recognition error rates for all four algo-rithms described in. Section 2 are compared using an im-age database constructed by Hallinan at the Harvard Ro-botics Laboratory 114), [15). In each image in this data-base, a subject held his/her head steady while being illu-minated by a dominant light source. The space of lightsource directions, which can be parameterized by spheri-,cal angles, was then sampled in 15° increments. See Fig. 3.From this database, we used 330 images of five people (66

of each). We extracted five subsets to quantify the effectsof varying lighting. Sample images from each subset areshown in Fig. 4.

Subset 1 contains 30 images for which both the longitudi-nal and latitudinal angles of light source direction are,within 15° of the camera axis, including the lighting

1. The Yale database is available for download from http://cvc.yaleedu.

Subset 1 Subset 2

Subset 1

Subset 2

Subset 3

Subset 4

Subset 5

Fig. 3. The highlighted lines of longitude and latitude indicate the lightsource directions for Subsets 1 through 5. Each intersection of a lon-gitudinal and latitudinal line on the 'ight side of the illustration has acorresponding image in the database.

direction coincident with the camera's optical axis.Subset 2 contains 45 images for which the greater of the

longitudinal and latitudinal 'angles of light source di-rection are 30° from the camera axis.

Subset 3 contains 65 images for which the greater of thelongitudinal and latitudinal angles of light source di-rection are 45° from the camera axis.

Subset 4 contains 85 images :!or which the greater of thelongitudinal and latitudi:nal angles of light source di-rection are 60° from the camera axis.

Subset 5 contains 105 images for which the greater of thelongitudinal and latitudihal angles of light source di-rection are 75° from the camera axis.

For all experiments, classification was performed using anearest neighbor classifier. Al training images of an indi-

Subset 3 Subset 4 Subset 5

Fig. 4. Example images from each subset of the Harvard Database used to test the four algorithms.

r._,

45

40.

E ~sr

30

S25

20

15

e 10

0Sub!

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 7, JULY 1997

Eigenlace (10)

-- Eigenlace (10)

S Egentace (10) w/o first 3 '- . .. CorrelationS/- -Corretion -'. -Elgenfece (10)

--- Llnear Subspace _ . ... .. w/o first 3

--e- Fishe dace ... .-....

- -

'a_.... ... . / . , -. ._ Linear' ,

Subspace

. ,-. . -- -Fisherface

set 1 Subset 2

Lighting Direction SubsetS9bset 3

Extrapolating from Subset 1Method Reduced Error Rate (% ) ..

Space Subset 1 Subset 2 Subset 3Eigenface 4 0.0 31.1 47.7

10 0.0 4.4 41.5

Eigenface 4 0.0 13.3 41.5w/o ist,3 10 0.0 .4.4 27.7

Correlation 29 0.0 . 0.0 33.9Linear Subspace 15 0.0 4.4 9.2

Fisherface 4 0.0 0.0 4.6

Fig. 5. Extrapolation: When each of the methods is trained on images with near frontal illumination (Subset 1),.the graph and corresponding table show

the relative performance under extreme light source conditions.

vidual were projected into, the feature space. The imageswere cropped within the face so that the contour of thehead was excluded. For the Eigenface and correlation tests,the images were normalized to have zero mean and unitvariance, as this ,improved the performance of these meth-ods. For the Eigenface method,.results are shown when tenprincipal components were used. Since it has been sug-gested that the first three principal components are primar-ily due to lighting variation and that recognition rates canbe improved by eliminating them, crror rates are also pre-sented using principal components four through thirteen.

We performed two experiments on the Harvard Data-base: extrapolation and interpolation. In the extrapolationexperiment, each method was trained on samples fromSubset 1 and then tested using samples from Subsets 1, 2,and 3. Since there are 30 images in the training set, cor-relation is equivalent to the Eigenface method using 29principal components. Fig. 5 shows the result from thisexperiment.

In the interpolation experiment, each method was trainedon Subsets 1 and 5 and then tested the methods on Subsets 2,3, and 4. Fig. 6 shows the result from this experiment.

These two experiments reveal a number of interestingpoints:

1) All of the algorithms perform perfectly when lightingis nearly frontal. However, as lighting is moved off

2. We have observed that the error rates are reduced for all methods whenthe contour is included and the subject is in front of a uniform background.However, all methods performed worse when the background varies.

3. To test the methods with an image from Subset 1, that image was removedfrom the training set, i.e., we employed the "leaving-one-out" strategy 141..

axis, there is a significant performance differencebetween the two class-specific methods and the Ei-genface method.

2) It has also been noted that the Eigenface method isequivalent to correlation when the number of Eigen-faces equals the size of the training.set [17], and sinceperformance increases with the dimension 'of the ei-genspace, the Eigenface method should do no betterthan correlation [26]. This is empirically demonstratedas well.

3) In the Eigenface method, removing the first threeprincipal components results in better performanceunder variable lighting conditions.

4) While the Linear Subspace method has error rates thatare competitive with the Fisherface method, it re-quires storing more than three times as much infor-mation and takes three times as long.

5) The Fisherface method had error rates lower than theEigenface method and required less computation time.

3.2 Variation in Facial Expression, Eye Wear, andLighting

Using 'a second database constructed at the Yale Center forComputational Vision and Control, we designed tests to de-termine how the methods compared under a different rangeof conditions. For sixteen subjects, ten images were acquiredduring one session in front of a simple background. Subjectsincluded females and males (some with facial hair), andsome wore glasses. Fig. 7 shows ten images of one subject.The first image was taken under ambient lighting in a neutralfacial expression and the person wore glasses. In the second

r m .I

L.R ,GAV AILABLE COPYBELHUMEUR ET AL.: EIGENFACES VS. FISHERFACES: RECOGNITION USING CLA SSP IFIC LINEAR PROJECTIOrI

Err0

rate

(9/

Subsel 2 Subset 4.,jbsel 3Lighting Direction Subset

Eigms Wae (10)

Eige iace (10)w/o I rsl 3

Corntaion

FiSNhe C

subs 3ace

Interpolatinq between Subsets 1 and 5Method Reduced Error Rate (%

Space Subset 2. Subset 3 Subs(et 4Eigentace 4 53.3 75.4 52.

10 11.11 33.9 20.0Eigenface 4 31.11 60.0 29.w/o 1st 3 10 6.7 20.0 12.9

Correlation 129 0.0 21.54 7.1Linear Subspace 15 0.0 1.5 0.0

Fisherface 4 0.0 0.0 1.2

Fig. 6. Interpolation: When each of the methods is trained on images from both near frontal and extreme lighting (Subsetscorresponding table show the relative performance under intermediate lighting conditions.

image, the glasses were removed. If the person normallywore glasses, those were used; if not, a random pair was bor-rowed. Images 3-5 were acquired by illuminating the face in'a neutral expression with a Luxolamp in three positions. The

last five images were acquired under ambient lighting with

different expressions (happy, sad, winking, sleepy, and sur-

prised). For the Eigenface and correlation tests, the images

were normalized to have zero mean and unit variance, as thisimproved the performance of' these methods. The imageswere manually centered and cropped to two different scales:The larger images included the full face and part of the back-ground while the closely cropped ones included internalstructures such as the brow, eyes, nose, mouth,' and chin, butdid not extend to the occluding contour.

1 and 5), the graph and

In this test, error rates were determined by the "leaving-one-out" strategy [4]: To clas:;ify an image of a person, thatimage was removed from the: data set and the dimension-ality reduction matrix W wa, computed. All images in the

database, excluding the test image, were then projected

down into the reduced spact: to be used for classification.Recognition was performed using a nearest neighbor classi-

fier. Note that for this test, eac:h person in the learning set isrepresented by the projectiorn. of ten images, except for thetest person who is represented by only nine.

In general, the performance of the Eigenface methodvaries with the number of piincipal components. Thus, be-

fore comparing the Linear Subspace and Fisherface methodswith the Eigenface method, we first performed an experi-

Fig. 7. The Yale database contains 160 frontal face images .covering 16 individuals taken under 10 different conditions: A normal image underambient lighting, one with or without glasses, three images taken with different point light sources, and five dif;erent facial expressions.

;

'

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 7, JULY 1997 '

Eigenface

- IEigenface w/o firstthree components

---- ------------- ------ Fisherface(7.3%)

50 100

Number of Principal Components

Fig. 8. As demonstrated on the Yale Database, the variation in performance of the Eigenface method depends on the number of principal compo-nonts retained. Dropping the first three appears to improve performance.

ea CloseCrop® uFaceJ

Eigentace(30) Correlation Unear -igenacet.u,Subspace w/o first 3

Recognition Algorithm

Fisherface

"Leaving-One-Out" of Yale DatabaseMethod Reduced Error Rate (%)

Space lose Crop Full FaceEigenlace 30 24.4 19.4Eigenfacew/o 1st 3 30 15.3 10.8

Correlation 160 23.9 20.0Linear 48 21.6 . 15.6

SubspaceFisherface 15 7.3 0.6

Fig. 9. The graph and corresponding table show the relative performance of the algorithms when applied to the Yale Database which containsvariation in facial expression and lIghting.

ment to determine the number of principal componentsyielding the lowest error rate. Fig. 8 shows a plot of error

rate vs. the number of principal components, for the closely

cropped set, when the initial three principal componentswere retained and when they were dropped.

The relative performance of the algorithms is self evident

in Fig. 9. The Fisherface method had error rates that werebetter than half that of any other method. It seems that theFisherface method chooses the set of projections which per-forms well over a range of lighting variation, facial expres-sion variation, and presence of glasses.

Note that the Linear Subspace method faired compara-tively worse in this experiment than in the lighting experi-

ments in the previous section. Because of variation in facial

expression, the images no longer lie in a linear subspace.Since the Fisherface method. tends to discount those por-tions of the image that are not significant for recognizing an

individual, the resulting projections W tend to mask theregions of the face that are highly variable. For example, thearea around the mouth is discounted, since it varies quite a

bit for different facial expressions. On the other hand; thenose, cheeks, and brow are stable over the within-class

r 25r02r

a 1 5

at t0oe

5

(%)0

0

_ v .

1 1*

. BEST AVAILABLE COPYBELHUMEUR ET AL.: EIGENFACES VS. FISHERFACES: RECOGNITION USING CLASS SPECIFIC LINEAR PROJECTION

variation and are more significant for recognition. Thus, weconjecture that Fisherface methods, which tend to reducewithin-class scatter for all classes, should produce projec-tion directions that are also good for recognizing other facesbesides the ones in the training set.

All of the algorithms performed better on the images ofthe full face. Note that there is a dramatic improvement inthe Fisherface method where the error rate was reducedfrom 7.3 percent to 0.6 percent. When the method is trainedon the entire face, the pixels corresponding to the occludingcontour of the face are chosen as good features for dis-criminating between individuals, i.e., the overall shape ofthe face is a powerful feature in face identification. As apractical note, however, it is expected that recognition rateswould have been much lower for the full face images if thebackground or hair styles had varied and may even havebeen worse than the closely cropped images.

3.3 Glasses RecognitionWhen using class specific projection methods, the learningset can be divided into classes in different manners. Forexample, rather than selecting the classes to be individualpeople, the set of images can be divided into two classes:"wearing glasses" and "not wearing glasses." With only twoclasses, the images can be projected to a line using theFisherface methods. Using PCA, the choice of the Eigenfacesis independent of the class definition.

In this experiment, the data set contained 36 imagesfrom a superset of the Yale Database, half with glasses. Therecognition rates were obtained.by cross validation, i.e., toclassify the images of each person, all images of that personwere removed from the database before the projection ma-trix W was computed. Table 1 presents the error rates fortwo different methods.

TABLE 1COMPARATIVE RECOGNITION ERROR RATES FOR GLASSES/

NO GLASSES RECOGNITION USING THE YALE DATABASE

Glasses Reco nitionMethod Reduced Space Error Rate

(%Z) .,

PCA 10. 52.6Fisheriace 1 5.3

PCA had recognition rates near chance, since, in mostcases, it classified both images with and without.glasses tothe same class. On the other hand, the Fisherface methodscan be viewed as deriving a template which is suited forfinding glasses and ignoring other characteristics of the face.This conjecture is supported by observing the Fisherface inFig. 10 corresponding to the projection matrix W. Naturally,it is expected that the same techniques could be applied toidentifying facial expressions where the set of training im-ages is divided into classes based on the facial expression.

4 CONCLUSION

The experiments suggest a number of conclusions:

1) All methods perform well if presented with an imagein the test set which is similar to an image in the train-ing set.

Fig. 10. The left image is an image from the Yale Database of a personwearing glasses. The right image s the Fisherface used for determin-ing if a person is wearing glasses.

2) The Fisherface method appears to be the best at ex-trapolating and interpolating over variation in lighting,although the Linear Sub ;pace method is a close second.

3) Removing the largest three principal components doesimprove the performan:e of the Eigenface method inthe presence of lighting variation, but does notachieve error rates as low as some of the other meth-ods described here.

4) In the limit, as more principal components are used inthe Eigenface method, performance approaches thatof correlation. Similarly, when the first three principalcomponents have been removed, performance im-proves as the dimension ality of the feature space is in-creased. Note, however, that performance seems tolevel off at about 45 principal components. Sirovitchand Kirby found a simi ar point of diminishing returnswhen using Eigenfaces.to represent faceimages [6].

5) The Fisherface method appears to be the best at simul-taneously handling variation in lighting and expres-sion. As expected, the Linear Subspace method sufferswhen confronted with variation in facial expression.

Even with this extensive experimentation, interestingquestions remain: H-ow well does the Fisherface methodextend to large databases. Can variation in lighting condi-tions be accommodated if some of the individuals are onlyobserved under one lighting -ondition?

Additionally, current face detection methods are likely tobreak down under extreme 1: ghting conditions such as Sub-sets 4 and 5 in Fig. 4, and io new detection methods areneeded to support the algorithms presented in this paper.Finally, when shadowing dominates, performance degradesfor all of the presented recog:aition methods, and techniquesthat either model or. mask he shadowed regions may beneeded. We are currently invwestigating models for repre-senting the set of images of a:, object under all possible illumi-nation conditions, and have shown that the set of n-pixel im-ages of an. object of any shape and with an arbitrary reflec-tance function, seen under ;ill possible illumination condi-tions, forms a convex cone in R" [32]. Furthermore, and mostrelevant to this paper, it appears that this convex illuminationcone lies close to a low-dimer sional linear subspace [14].

ACKNOWLEDGMENTS

P.N. Belhumeur was suppor:ed by ARO grant DAAHO4-95-1-0494. J.P. Hespanha was supported by the U.S. National

4 p. *~ ,' ,

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO.-7. JULY 1997

Science Foundation Grant ECS-9206021, AFOSR GrantF49620-94-1-0181, and ARO Grant DAAHO4-95-1-0114. D.J.Kriegman was supported by the U.S. National ScienceFoundation under an NYI,.LRI-9257990 and by ONR N00014-

93-1-0305. The authors would like to thank Peter Hallinan for

providing the Harvard Database, and Alan Yuille and David

Mumford for many useful discussions.

REFERENCES[1) R. Chellappa, C. Wilson, and S. Sirohey, "Human and Machine

Recognition of Faces: A Survey," Proc. IEEE, vol. 83, no. 5, pp. 705-740, 1995.

12) A. Samal and P. lyengar, "Automatic Recognition and Analysis ofHuman Faces and Facial Expressions: A Survey," Pattern Recogni-tion, vol. 25, pp. 65-77, 1992.

[3) A. Shashua, "Geometry and Photometry in 3D Visual Recognition,"PhD thesis, Massachusetts Institute of Technology, 1992.

[4) R. Duda and P. Hart, Pattern Classification and Scene Analysis. NewYork: Wiley, 1973.

15] R.A. Fisher, "The Use of Multiple Measures in Taxonomic Prob-lems," Ann. Eugenics, vol. 7, pp. 179-188,1936.

(6) L. Sirovitch and M: Kirby, "Low-Dimensional Procedure for theCharacterization of Human Faces," J. Optical Soc. of Am. A, vol. 2,pp. 519-524,1987.

17] M. Turk and A. Pentland, "Eigenfaces for Recognition," J. Cogni-trive Neuroscience, vol. 3, no. 1, 1991.

(81 M. Turk and A. Pentland, "Face Recognition Using Eigenfaces,"Proc. IEEE Conf on Computer Vision and Pattern Recognition, 1991,pp. 586-591.

[91 Y. Moses, Y. Adini, and S. Ullman, "Face Recognition: The Prob-lem of Compensating for Changes in Illumination Direction,"European Conf Computer Vision, 1994, pp.. 286-296.

[10] Y. Cheng, K. Liu, J. Yang, Y. Zhuang, and N. Gu, "Human FaceRecognition Method Based on the Statistical Model of Small Sam-ple Size," SPIE Proc. Intelligent Robots and Computer Vision X: Algo-rithms and Technology, 1991, pp. 85-95.

[11] S. Baker and S.K. Nayar, "Pattern Rejection," Proc. IEEE Conf.Computer Vision and Pattern Recognition, 1996, pp. 544-549.

(12] P.N. Belhumeur, J.P. Hespanha, and D.J. Kriegman, "Eigenfacesvs. Fisherfaces: Recognition Using Class Specific Linear Projection,"European Conf. Computer Vision, 1996, pp. 45-58.

(13] Y. Cui, D. Swets, and J. Weng, "Learning-Based Hand Sign Rec-ognition Using SHOSLIF-M," Int'l Conf. on Computer Vision, 1995,pp. 631-636:

1141 P. Hallinan, "A Low-Dimensional Representation of Human Facesfor Arbitrary Lighting Conditions," Proc. IEEE Conf Computer Visionand Pattern Recognition, 1994, pp. 995-999.

115) P. Hallinan, "A Dcformable Model for Face Recognition UnderArbitrary Lighting Conditions," PhD thesis, Harvard Univ., 1995.

[16) A. Pentland, B. Moghaddam, and Starner, "View-Based andModular Eigenspaces for Face Recognition," Proc. IEEE ConJ.Computer Vision and Pattern Recognition, 1994, pp. 84-91.

(17] H. Murase and S. Nayar, "Visual Learning and Recognition of 3-D Ob-jects from Appearance," Int'l J. Computer Vision, vol. 14, pp. 5-24,1995.

118] D. Beymer, "Face Recognition Under Varying Pose," Proc. IEEEConf Computer Vision and Pattern Recognition, 1994, pp. 756-761.

119) A. Gee and R. Cipolla, "Determining the Gaze of Faces in Images,"Image and Vision Computing, vol. 12, pp. 639-648, 1994.

120) A. Lanitis, C.J. Taylor, and T.F. Cootes, "A Unified Approach toCoding and Interpreting Face Images," Int'l Conf. Computer Vision,1995, pp. 368-373.

[21) Q. Chen, H. Wu, and M. Yachida, "Face Detection by Fuzzy PatternMatching," Int'Il Conf. Computer Vision, 1995, pp. 591-596.

(22) I. Craw, D. Tock, and A: Bennet, "Finding Face Features," Proc.European Conf Computer Vision, 1992, pp. 92-96.

[231 T. Leung, M. Burl, and P. Perona, "Finding Faces in ClutteredScenes Using Labeled Random Graph Matching," Int'l Conf. Con-puter Vision, 1995, pp. 637-644.

[24) K. Matsuno, C.W. Lee, S. Kimura, and S. Tsuji, "Automatic Rec-ognition of Human Facial Expressions," Int'l Conf Computer Vision,1995, pp. 352-359.

(25) B. Moghaddam and A. Pentland, "Probabilistic Visual Learning forObject Detection," Int'l Conf. Computer Vision, 1995, pp. 786-793.

126] R..Brunelli and T. Poggio, "Face Recognition: Features vs. Ternm-plates," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15,no. 10, pp. 1,042-1,053, Oct. 1993.

[27] J.M. Gilbert and W. Yang, "A Real-Time Face Recognition SystemUsing Custom VLSI Hardware," Proc. IEEE Workshop on ComputerArchitectures for Machine Perception, 1993, pp. 58-66.

[28] B.K.P. Horn, Computer Vision. Cambridge, Mass.: MIT Press, 1986.[29) W.M. Silver, Determining Shape and Reflectance Using Multiple 'Im-

ages, PhD thesis, Massachusetts Institute of Technology, 1980.[30] R.J. Woodham, "Analysing Images of Curved Surfaces," Artificial

Intelligence, vol. 17, pp. 117-140,1981.[31) S. Nayar and H. Murase, "Dimensionality of Illumination in Ap-

pearance Matching," IEEE Conf Robotics and Automation, 1996.(32] P.N. Belhumeur and D.J. Kriegman, "What is the Set of Images of

an Object under all Possible Lighting Conditions?," IEEE Proc.Conf Computer Vision and Pattern Recognition, 1996.

Peter N. Belhiumeur receiveo nhis ScB degree(with highest honors) in computer and informationengineering from Brown University in 1985. Hereceived an SM and PhD from Harvard Universityin 1991 and 1993, respectively, where he studiedunder a Harvard Fellowship. In 1994, he spend ahalf-year as a Postdoctoral Fellow at the Univer-sity of Cambridge's Sir Isaac Newton Institute forMathematical Sciences.

Currently, he Is an assistant professor of elec-trical engineering at Yale University. He teaches

courses on signals and systems, pattern and object recognition, andcomputer vision. He is a member of the Yale Center for ComputationalVision and Control and a member of the Army research Center for Imag-ing Science. He has published over twenty papers.on Image processingand 'computational vision. He is a recipient of a U.S. National ScienceFoundation Career Award; he has been awarded a Yale University JuniorFaculty Fellowship for the Natural Sciences; and he won the IEEE BestPaper Award for his work on characterizing the set of images of an objectunder variable illumination.

Joao P. Hespanha received the Licenciatura andMS degrees in electrical and computer engineer-ing from Instituto Superior Tecnico, Lisbon, Portu-gal, In 1991 and 1993, respectively, and a secondMS degree in electrical engineering from YaleUniversity, New Haven, Connecticut, in 1994,where he is currently pursuing the PhD degree inengineering and applied science.

From 1987 to 1989, he was a research assis-tant at Institute de Engenharia de Sistemas eComputadores (INESC) in Lisbon, Portugal, and,

from 1989 to 1990, was an instructor at Fundo para o DesenvolvimentoTecnol6gico (FUNDTEC) in the areas of electronics, computer science,and robotics. From 1.992 to 1993, he was a working partner in Sociedadede Projectos em Sistemas e Computadores, Lda., also in Lisbon. Hisresearch interests include nonlinear control, both robust and adaptive,hybrid systems, switching control, and the application of vision to robotics.

David J. Kriegman received the BSE degree(summa cum laude) in electrical engineering andcomputer science in 1983 from Princeton Uni-versity, where he was awarded the.Charles IraYoung Award for electrical engineering research.He received the MS degree in 1984 and PhD in1989 in electrical engineering from StanfordUniversity, where he studied under a HertzFoundation Fellowship.

Currently, he is an associate professor at theCenter for LComputautional Vision and Control in the

Departments of Electrical Engineering and Computer Science at YaleUniversity and was awarded a U.S. National Science Foundation YoungInvestigator Award in 1992. His paper on characterizing the set of imagesof an object under variable illumination received the best paper at the1996 IEEE Conference on Computer Vision and Pattemrn Recognition. Hehas published over sixty papers on object representation and recognition,illumination modeling, geometry of curves and surfaces, structure frommotion, robot planning, and mobile robot navigation.

-",,-,,mmmmlmll,

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL 17, NO. 10, OCTOBER 1995

Person Identification Using Multiple CuesRoberto Brunelli and Daniele Falavigna

Abstract-This paper presents a person Identification systembased on acoustic and visual features. The system is organized asa set of non-homogeneous classifiers whose outputs are Integratedafter a normalization step. In particular, two classifiers based onacoustic features and three based on visual ones provide data foran Integration module whose performance Is evaluated. A noveltechnique for the integration of multiple classifiers at an hybridrank/measurement level Is introduced using HyperBF networks.Two different methods for the rejection of an unknown personare Introduced. The performance of the integrated system isshown to be superior to that of the acoustic and visual subsys.-tems. The resulting Identification system can be used to log per-sonal access and, with minor modifications, as an identity verifi-cation system.

Index Terms-Template matching, robust statistics, correla-tion, face recognition, speaker recognition, learning, classifica-tion.

I. INTRODUCTION

THE identification of a person interacting with computersrepresents an important task for automatic systems in the

area of information retrieval, automatic banking, control ofaccess to security areas, buildings, and so on. The need for areliable identification of interacting users is obvious. At thesame time it is well known that the security of such systems istoo often violated in every day life. The possibility of integrat-ing multiple identification cues, such as password, identifica-tion card, voice, face, fingerprints, and the like, will, in prin-ciple, enhance the security of a system to be used by a selectedset of people.

This paper describes in detail the theoretical foundationsand design methodologies of a person recognition system thatis part of MAIA, the integrated AI project under developmentat IRST [ I .

Previous works about speaker recognition [2], [3] have pro-posed methods for classifying and combining acoustic featuresand for normalizing [4], [5] the various classifier scores. Inparticular, score normalization is a fundamental step when asystem is required to confirm or reject the identity given by theuser (user verification): In this case, in fact, the identity is ac-cepted or rejected according to a comparison with a preesti-mated threshold. Since the integration of voice and images inan identification system is a new concept, new methods forboth classifier normalization and integration were investigated.Effective ways of rejecting an unknown person by consideringscore and rank information and for comparing images with

Manuscript received July 28, 1994; revised April 17. 1995. Recommendedfor acceptance by B. Dom./ The authors are with the Istituto per la Ricerca Scientifica e Tecnologica,1-38050 Povo, Trento, ITALY; e-mail: [email protected], [email protected].

IEEECS Log Number P95111.

improved similarity measures are proposed. A simple methodfor adapting the acoustic models of the speakers to a real op-erating environment was also developed.

The speaker and face recognition systems are decomposedinto two and three single feature classifiers, respectively. Theresulting five classifiers produce nonhomogeneous lists ofscores that are combined using two different approaches. Inthe first approach, the scores are normalirzed through a robustestimate of the location and scale parameters of the corre-sponding distributions. The normalized scores are then com-bined using a weighted geometric average and the final identi-fication is accepted or rejected according to the output of alinear classifier, based on score and rank information derivedfrom the available classifiers. Within the second approach, theproblem of combining the normalized outputs of multipleclassifiers and of accepting/rejecting the resulting identifica-tion is considered a learning task. A mapping from the scoresand ranks of the classifiers into the interval (0, 1) is approxi-mated using a HyperBF network. A final threshold is then in-troduced based on cross-validation. System performance isevaluated and discussed for both strategies. Because of thenovelty of the problem, standard databases for system trainingand test are not yet available. For this reason, the experimentsreported in this paper are based on data collected at IRST. Asystem implementation operating in real-time is available andwas tested on a variety of IRST researchers and visitors. Thejoint use of acoustic and visual features proved effective inincreasing system performance and reliability.

The system described here represents an improvement overa recently patented identification system based on voice andface recognition [6], [7]. The two systems differ in many ways:In the latter the speaker and face recognition systems are notfurther decomposed into classifiers, the score normalizationdoes not rely on robust statistical techniques, and, finally, therejection problem is not addressed.

The next sections will introduce the speaker and face rec-ognition systems. The first approach to the integration of clas-sifiers and the linear accept/reject rule for the final systemidentification are then discussed. Finally, the novelrank/measurement level integration strategy using a HyperBFnetwork is introduced with a detailed report on system per-formance.

II. SPEAKER RECOGNITION

The voice signal contains two types of information: individ-ual and phonetic. They have mutual effects and are difficult toseparate; this represents one of the main problems in the de-velopment of automatic speaker and speech recognition sys-tems. The consequence is that speaker recognition systems

0162-8828/95504.00 1995 IEEE

Application No. 11/231,353Docket No. 577832000200

956 IEEE TRANSACTIONS ON PATTERN AN

perform better on speech segments having specific phoneticcontents while speech recognition systems provide higher ac-curacy when tuned on the voice of a particular speaker. Usu-ally the acoustic parameters for a speech/speaker recognizerare derived by applying a bank of band-pass filters to adjacentshort time windows of the input signal. The energy outputs ofthe filters, for various frames, provide a good domain repre-sentation. Fig. 1 gives an example of such an analysis. Thespeech waveforms correspond to utterances of the Italian digit4 (Ikwat:rol) by two different speakers. The energy outputs ofa 24 triangular band-pass filter bank are represented below thespeech waveforms (darker regions correspond to higher energyyalues).

UFig. I. Acoustic analysis of two utterances of the digit 4 (/kwat:ro/) by twodifferent speakers.

in the past years several methods and systems for speakeridentification [8), (31 were proposed that perform more or lessefficiently depending on the text the user is required to utter(in general, systems can be distinguished into text dependentor text independent), the length of the input utterance, thenumber of people in the reference database, and, finally, thetime interval between test and training recordings.

For security applications, it is desirable that the user utter adifferent sentence during each interaction. The content of theutterance can then be verified to ensure that the system is notcheated by prerecorded messages. For this work, a text inde-pendent speaker recognition system based on vector quantiza-tion (VQ) 19] was built. While it cannot yet verify the contentof the utterance, it can be modified (using supervised cluster-ing or other techniques) to obtain this result.

A block diagram of the system is depicted in Fig. 2. In thesystem, each reference speaker is represented by means of twosets of vectors (codebooks) that describe his/her acoustic char-acteristics. During identification, two sets of acoustic features(static and dynamic), derived from the short time'spectralanalysis of the input speech signal, are classified by evaluatingtheir distances from the prototype vectors contained in thespeaker codebook couples. In this way, two lists of scores aresent to the integration module. In the following, both the spec-tral analysis and vector quantization techniques will be describedin more detail (see also [10) and a reference book such as [ 11).

ALYSIS AND MACHINE INTELLIGENCE. VOL. 17, NO. 10, OCTOBER 1995

Fig. 2. The speaker recognition system based on VectorQuantization.

Since the power spectrum of the speech signal decreases asfrequency increases a preemphasis filter that enhances thehigher frequencies is applied to the sampled input signal. Thetransfer function of the filter is H(z) = 1/(1 - 0.95 z ).

The preemphasized signal, x(n), I 5 n 5 N, is subdividedinto frames y;(n). I <5 t 5 T, having length L. Each frame isobtained by multiplying x(n) by a Hamming window h,(n):

y(n) = x(n) h(n)N

SIlb-

.Y

yI....._... -.............

IA~a

VW

C(21(n- tS)" L L

h, (n) = 0.54 - 0.46. co (, t S - L < n S tS + 2 (2)L2 2

In the equation above, L represents the length, in samples,of the Hamming window and S is the analysis step (also ex-pressed in samples). For the system, L and S were chosen tocorrespond to 20 ms and 10 ms, respectively.

The signal is multiplied by an Hamming window (raisedcosine) to minimize the sidelobe effects on the spectrum of theresulting sequence y,(n).

The acoustic analysis of each frame is performed as follows:

1) the power spectrum of the sequence yt,(n) is evaluated;2) a bank of Q = 24 triangular filters, spaced according to a.

logarithmic scale (Mel scale), is applied to the powerspectrum and the energy outputs s,, I 5 q 5 Q, fromeachfilter are evaluated;

3) the Mel Frequency Cepstrum Coefficients (MFCC) 1121,.0p, 5< p 5 P=8, are computed, from the filter bank out-puts, according to the following equation:

t2 -'

q _ I a0m = t og(sr,,, os p q 2JQq=) Q

the MFCCs are arranged into a vector, 0,, which is calledstatic, since it refers to a single speech frame;

4) to account for the transitional information contained inthe speech signal a linear fit is applied to the componentsof seven adjacent MFCCs; the resulting regression coef-

Sficients are arranged into a vector that is called dynamic;5) a binary variable is finally evaluated that allows marking

the frame as speech or background noise; this parameteris computed by means of the algorithm described in [13].

The Mel scale is motivated by auditory analysis of sounds. Theinverse Fourier transform of the log-spectrum (cepstrum) pro-vides parameters that improves performance at both speechand speaker recognition [ ll ], [ 12). Furthermore, the Euclid-ean distance between two cepstral vectors represents a goodmeasure for comparing the corresponding speech spectra. Thestatic and dynamic 8D vectors related to windows marked asbackground noise are not considered during both system

BRUNELLI AND FALAVIGNA: PERSON IDENTIFICATION USING MULTIPLE CUES

training and testing. As previously said, VQ is used to designthe static and dynamic codebooks of a given reference speaker,say the ith one. Starting from a set of training vectors (static ordynamic) O9 = ( 011, ... , Ox), derived from a certain number ofutterances, the objective is to find a new set

•= .E, .... r ), with M < K, that represents well theacoustic characteristics of the given speaker. To do this aclustering algorithm, similar to that described in [10], is ap-plied to the 9

j set. The algorithm makes use of an iterativeprocedure that allows determination of codebook centroids,W,, by minimizing their average distance, D(O, 'Y), from thetraining vectors:

K "

D(641,7)= - min[d(ik.Vim), (4)K k= M=

The distance d(Oa, V,,) is defined as follows:

d(O, Vim) = (rk - Vim)' W"(O - VJm) (5)

In the equation above t denotes transpose and W is the co-variance matrix of the training vectors. The matrix W is esti-mated from'the training data of all the speakers in the refer-ence database. This matrix was found to be approximatelydiagonal, so that only the diagonal elements are used to evalu-ate distances.

In the recognition phase the distances, Dsi, DD,, between thestatic and dynamic vector sequences, derived from the inputsignal, and the corresponding speaker codebooks are evaluatedand sent to the integration module. '

If e = { ... , Or) is the static (or dynamic) input sequenceand Y'I is the ith static (or dynamic) codebook, then the totalstatic (or dynamic) distance will be:

T M

T= s1 il1 (6)

where 1 is the total number of speakers in the reference data-base.

To train the system, 200 isolated utterances of the Italiandigits (from 0 to 9) were collected for each reference user. Therecordings were realized by means of a digital audio tape(DAT): The signal on the DAT tape, sampled at 48 kHz, wasdown sampled to 16 kHz, manually end-pointed, and stored ona computer disk. The speech training material was analyzedand clustered as previously described. As demonstrated in 19),system performance depends on both input utterance lengthand codebook size; preliminary experiments have suggestedthat the speaker, to be identified, should utter a string of atleast seven digits in a continuous way and in whatever order.In the reported experiments the number of digits was keptequal to seven and the codebook size was set to M = 64 be-cause higher values did not improve recognition accuracy.Furthermore, if input signal duration is too short, the systemrequires the user to repeat the digit string.

To evaluate integrated system performance (see Sec-tion IV.A) the reference users interacted three times with thesystem during three different sessions. The test sessions werecarried out in an office environment using an ARIEL board asacquisition channel. Furthermore the test phase was performed

about five months after the training recordings. Due to both thedifferent background noise and acquisition conditions betweentraining and test, the codebooks must be adapted. Adaptation.means designing a new codebook, starting from) a given one,that better resembles the acoustic characteristics of both theoperating environment and the acquisition channel, Adaptationshould also take into account variations in time of thespeaker's voice (intraspeaker'variations). Adaptation requiresthe use of few utterances to modify the codebook as it is notnecessary to design it from scratch (this would require at least30-40 seconds of speech). In our case, the adaptation vectorsare derived from the digit strings uttered by the users during asingle test session. The dynamic codebooks were not adaptedsince they represent temporal variations of the speech spectraand therefore they are less sensitive to both intraspeaker voicevariability and acquisition channel variations.

The adaptation process of the ith codebook, C; can be sum-marized as follows:

1) the mean vectors p, and v1 of the adaptation vectors andof the given codebook respectively are evaluated;

2) the difference vector , = p- v1 is evaluated;3) the vectors of C, are shifted by a quantity equal to A, ob-

taining a new set C' = (c, + 4, ... , ca + A,); the set C'is placed in the region of the adaptation vectors;

4) the adaptation vectors are clustered using the set C, as

initial estimate of the centroids; therefore a new set of

centroids 0, = (on, .... oi) and the corresponding cell

occupancies N, = nn .... n) are evaluated;5) the adapted codebook { V); is obtained according to the

following equation:

,1 =c,,,+ l-e-a o,,-ca) 1<m5M (7)

In the equation above the parameter n1,,, determines thefraction of deviation vector 6,,, = (o., -c,,,), that has to

be summed to the initial centroid c,! . Equation 7 is asimple method to modify the centroids of a codebook ac-cording to the number of data available for their esti-

mates. ,,, can be zerowhen the utterance used for adap-tation does not contain sounds whose spectra are relatedto the mth centroid.

For the system, a was chosen equal to 0.1. The two shiftsapplied by the adaptation procedure can be interpreted as fol-lows:

I) 4,, the major shift, accounts for environment and channelvariations with respect to training;

2) 6,,,,, the minor shift, accounts for intraspeaker voicevariations in time.

IT. FACE RECOGNITION

Person identification through face recognition is the mostfamiliar among the possible identification strategies. Severalautomatic or semiautomatic systems were realized since theearly seventies-albeit with varying degree of success. Differ-ent techniques were proposed, ranging from the geometrical

S 957

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 17, NO. 10, OCTOBER 1995

description of salient facial features to the expansion of a dized image of the face on an appropriate basis of images(141 for references). The strategy used by the described sysis essentially based on the comparison, at the pixel levelselected regions of the face (14]. A set of regions, respectiencompassing the eyes, nose, and mouth of the user toidentified are compared with the corresponding regions stcin the database for each reference user (see Fig. 3). The imshould represent a frontal view of the user face without maexpressions. As will be clear from the detailed descriptthese constraints could be relaxed at the cost of storinhigher number of images per user in the database. The fumental steps of the face recognition process are the followi

1) acquisition of a frontal view of the user face;2) geometrical normalization of the digitized image;3) intensity normalization of the image;4) comparison with the images-stored in the database.

igit-(see

tem, ofvely

beoredagesrkedion,

g anda-ng:

Fig. 3. The highlighted regions represent the templates used for identification.

The image of the user face is acquired with a CCD camera anddigitized with a frame grabber.

To compare the resulting image with those stored in the da-tabase, it is necessary to register the image: It has to be trans-lated, scaled, and rotated so that the coordinates of a set ofreference points take corresponding standard values. As fron-tal views are considered, the centers. of the pupils represent anatural set of control points that can be located with good ac-curacy. Eyes can be found through the following steps:

I1) locate the (approximate)symmetry axis of the face;2) locate the left/right eye by using an eye template for

which the location of the pupil is known; if the confi-dence of the eye location is not sufficiently high, declarefailure (the identification system will use only acousticinformation);

3) achieve translation, scale, and rotation invariance by fix-ing the origin of the coordinate system at the midpoint ofthe interocular segment and the interocular distance andthe direction of the eye-to-eye axis at predefined values.

Under the assumption that the user face is approximately verti-cal in the digitized image, a good estimate of the coordinate Sof the symmetry axis is given by

S= median(Pv(IlI* Kvl)i (8)

where * represent convolution, I the image, Kv the convolutionkernel [-1, 0, I', Pv the vertical projection whose index i runsover the columns of the image. The face can then be split ver.-

tically into two, slightly overlapping parts containing the leftand right eye, respectively. The illumination under which theimage is taken can impair the template matching process usedto locate the eye.' To minimize this effect a filter, .N(1), isapplied to image I

if .N': 1

if N~'>*i

where

= I1* KG() (10)

and KG, is a Gaussian kernel whose ar is related to the ex-pected interocular distance 4,. The arithmetic operations acton the values of corresponding pixels. The process mapping Iinto Nreduces the influence of ambient lighting while keepingthe necessary image details. This is mainly due to the removalof linear intensity gradients that are mapped to the constantvalue 1. Extensive experiments, using ray-tracing and texture-mapping techniques to generate synthetic images under a widerange of lighting directions have shown that the local contrastoperator of (9) exhibits a lower illumination sensitivity thanother operators such as the laplacian, the gradient magnitudeor direction 115] and that there is an optimal value of the pa-rameter a (approximately equal to the iris radius).

The same filter is,applied to the eye templates. The templatematching process is based on the algorithm of hierarchicalcorrelation proposed by Burt (16]. Its final result is a map ofcorrelation values: the center of gravity of the pixels withmaximum value representing the location of the eye. Once thetwo eyes have been located, the confidence of the localizationis expressed by a coefficient, CE, that measures the symmetryof the eye positions with respect to the symmetry axis, thehorizontal alignment and the scale relative to that of the eyetemplates:

e 2 m ax(C1,C ) ' (11)

where C, and C, represent the (maximum) correlation value forthe left/right eye, s the interocular distance expressed as amultiple of the interocular distance of the eyes used as tem-plates, AO represents the angle of the interocular axis with re-spect to the horizontal axis, while a, and a, represent toler-ances on the deviations from the prototype scale and orienta-tion.

The first factor in the RHS of (11) is the average correlation.value of the left and right eye: The higher it is the better thematch with the eye templates. The second factor represents thesymmetry of the correlation values and equals I when the twovalues are identical. The third and fourth factors allow weigh-ing the deviation from both the assumed scale and (horizontal)orientation of the interocular axis, respectively. The parame-ters of the Gaussians, a, and a, were determined by theanalysis of a set of interactions.

If the value of CE is too low, the face recognition systemdeclares failure and the identification proceeds using the

N'


acoustic features alone. Otherwise, the image is translated,scaled and rotated to match the location of the pupils to that ofthe database images. In the reported experiments the interocu-lar distance was set equal to 28 pixels. Alternative techniquesfor locating eyes are reported in [17), 118)]. Due to the geo-metrical standardization, the subimages containing the eyes,nose, and mouth are approximately characterized by the samecoordinates in every image. These regions are extracted fromthe image of the user face and compared in turn to the corre-sponding regions extracted from the database entries, previ-ously filtered according to (9) and (10). Let us introduce asimilarity measure C based on the computation of the Lt norm

of a vector I IIL = , ix, and on the corresponding dis-lance dQ,, (x.y)= r- ylj,:

d(x, y)C(x,y)= 1- d(x,) (12)

The Lt distance of two vectors is mapped by C(.,.) into theinterval [0, 1], higher values representing smaller distances.This definition can be easily adapted to the comparison of im-ages. For the comparison to be useful when applied to realimages, it is necessary to normalize the images so that theyhave the same average intensity p and standard deviation (orscale) a. The latter is particularly sensitive to values far fromthe average p so that the scale of the image intensity distribu-tion can be better estimated by the following quantity:

I n

7 , = - x; - l (13)nfl

where the image is considered as a one dimensional vector x.The matching of an image B to an image A can then be quanti-fled by the maximum value of C(A, B), obtained by sliding the'smaller of the two images over the larger one. A major advan-tage of the image similarity computed according to (12) overthe more common estimate given by the cross-correlation co='efficient (19), based on the L. norm, is its reduced sensitivityto small amounts of unusually high differences between corre-sponding pixels. These differences are often due to noise orimage specularities such as iris highlights. A detailed analysisof the similarity measure defined in (12) is given in [20]. Analternative technique for face identification is reported in 121).Let us denote with { UJ,,, ,..,s the set of images available forthe kth user. A comparison can now be made between a set ofregions of the unknown image N( and the corresponding re-gions of the database images. The regions currently used bythe system correspond to the eyes, nose and mouth. A list ofsimilarity scores is obtained for each region ., of image U,,,:

{Sa } = maxC(Ra ().7 a(Uk,)) (14)

where R,(JV) represents a region of Neontaining JF, with aframe whose size is related to the interocular distance. Thelists of matching scores corresponding to eyes, nose, andmouth are then available for further processing. The distribu-tion of the correlation values for corresponding features of thesame person and of different people are reported in Fig. 4.

o grtwe,,

0.0 0.2' .0.4 0.6 oacores

1.0

Fig. 4. The distribution of the correlation values for corresponding features ofthe same person and of different people.

Integration with the scores derived from the acoustic analy-sis can now be performed with a single or double step process.In the first case, the two acoustic and the three. visual scoresare combined simultaneously, while in the second the acousticand visual scores are first combined separately and the finalscore is given by the integration of the outputs of the speakerand face recognition systems (see [7) for an example of thelatter). The next section will introduce two single-step integra-tion strategies for classifiers working at the measurement level.

IV. INTEGRATION

The use of multiple cues, such as face and voice, providesin a natural way the information.necessary to build a reliable,high performance system. Specialized subsystems can identify(or verify) each of the previous cues and the resulting outputscan then be combined into a unique decision by some integra-tion process. The objective of this section is to describe andevaluate some integration strategies. The use of multiple cuesfor person recognition proved beneficial for both system per-formance and reliability.'

A simplified taxonomy of multiple classifier systems is re-ported in [22). Broadly speaking, a classifier can output infor-mation at one of the following levels:

* The abstract level: the output is a subset of the possibleidentification labels, without any qualifying information;

* The rank level: the output is a subset of the possible la-bels, sorted by decreasing confidence (which is not sup-plied); ,

* The measurement level: the output is a subset of labelsqualified by a confidence measure.

The level at which the.different classifiers of a compositesystem work clearly constrains the ways their responses can bemerged. The first of the following sections will address theintegration of the speaker/face recognition systems at themeasurement level. The possibility of rejecting a user as un-known will then be discussed. Finally, a novel, hybrid levelapproach to the integration of a set of classifiers will be pre-sented.

1. Two aspects of reliability are critical for a person identification system:The first is the ability of rejecting a uscr as unknown, the second is the pos-sibility of working with a reduccd input, such as only the speech signal or theface image.


A. Measurement Level Integration

The acoustic and visual identification systems already con-stitute a multiple classifiersystem. However, both the acousticand visual classifiers can be further split into several subsys-tems, each one based on a single type of feature. In our system,five, classifiers were considered (see Sections II and III)working on the static, dynamic acoustic features, and on theeyes, nose and mouth regions.

A critical point in the design of an integration procedure atthe measurement level is that of measurement normalization.In fact, the responses of the different classifiers usually havedifferent scales (and possibly offsets), so that a sensible com-bination of the outputs can proceed only after the scores areproperly normalized. As already detailed, the outputs of theidentification systems are not homogeneous: the acoustic fea-tures provide distances while the visual ones provide correla-tion values. A first step towards the normalization of the scoresis to reverse the sign of distances, thereby making them con-cordant with the correlation values: the higher the value, the

more similar the input patterns. Inspection of the score distri-butions shows them to be markedly unimodal and roughlysymmetrical. A simple way to normalize scores is to estimatetheir average values and standard deviations so that distribu-tions can be translated and rescaled in order to have zero aver-age and unit variance. The values can then be forced into astandard interval, such as (0, 1), by means of an hyperbolictangent mapping. The normalization of the scores can rely on afixed set of parameters, estimated from the score distributionsof a certain number of interactions, or can be adaptive, estimat-ing the parameters from the score distribution of the currentinteraction. The latter strategy was chosen mainly because ofits ability to cope with variations such as different speech ut-terance length without the need to re-estimate the normaliza-tion parameters.

The estimation of the location and scale parameters of thedistribution should make use of robust statistical techniques(23), [24]. The usual arithmetic average and standard deviationare not well suited to the task: they are highly sensitive to out-lier points and could give grossly erroneous estimates. Alter-native estimators exist that are sensitive to the main bulk of thescores (i.e. the central part of a unimodal symmetric distribu-tion) and are not easily misled by points,in the extreme tails ofthe distribution. The median and the Median Absolute Devia-tion (MAD) are examples of such location and scale estimatorsand can be used to reliably normalize the distribution of thescores. However, the median and the MAD estimators have alow efficiency relative to the usual arithmetic average andstandard deviation. A.class of robust estimators with higherefficiency was introduced by Hampel under the name of tanh-estimators and is used in the current implementation of thesystem (see [23) for a detailed description). Therefore each listof scores (SY) .I,..., from classifier j, being I the number ofpeople in the reference database, can be transformed into anormalized list by the following mapping:

So =i- tanh 0.01 S~l a +1 E(0,1)2 creans

(15)

where p,,, and oh are the average and standard deviationestimates of the scores {Sy), ,....,; as given by the Hampel es-timators. An example of distributions of the resulting normal-ized scores is reported in Fig. 5 for each of the five featuresused in the classification.

Oistrioulions of Normolie ScorescSt

1 , o Sz400 - rt. -.

o r230 --- - r - - &r3

200 - - - -,

0-0.48 0.50 0.52

Normolzeo0 SCcore

Fig. 5. The density distribution of the normalized scores for each of the clas-sifiers: S1, S2 represent the static and dynamic speech scores while Fl, F2.and F3 represent the eyes, nose, and mouth scores, respectively.

In the following formulas; a subscript index im indicates themth entry within the set of scores sorted by decreasing value.The normalized scores can be integrated using a weightedgeometric average:

=i (n S . jJ )lW1 (16)

where the weights wj represent an estimate of the score disper-sion in the right tail of the corresponding distributions:

S'~ -0.5w S' l-0.51.0 (17)

S;1 - 0.5

The main reason suggesting the use of geometric average forthe integration of scores relies on probability: If we assumethat the features are independent the probability that a featurevector corresponds to a given person can be computed by tak-ing the product of the probabilities of each single feature. Thenormalized scores could then be considered as equivalent toprobabilities. Another way of looking at the geometric averageis that of predicate conjunction using a continuous logic. [25),[26]. The weights reflect the importance of the different fea-tures (or predicates). As defined in (17), each feature is givenan importance proportional to the separation of the two bestscores. If the classification provided by a single feature is am-biguous, it is given low weight. A major advantage of (16) isthat it does not require a detailed knowledge of how each fea-ture is distributed (as would be necessary when using a Bayesapproach). This eases the task of building a system that inte-grates many features.

The main performance measure of the system is the per-centage of persons correctly recognized. Performance can befurther qualified by the average value of the following ratio Rx:

S;i -SiR - isis/. (18)max)(5;,)- S1,


The ratio R, measures the separation of the correct match SXfrom the wrong ones. This ratio is invariant against the scaleand location parameters of the" integrated score distributionand can be used to compare different integration strategies(weighted/unweighted geometric average, adaptive/fixed nor-malization). The weighted geometric average of the scoresadaptively normalized exhibits the best performance and sepa-ration among the various schemes on the available data.

Experiments have been carried out using data acquired'during three different test sessions. Of the 89 persons stored inthe database, 87 have interacted with the system in one ormore sessions. One of the three test sessions was used to adaptthe acoustic and visual databases (in the last case the images ofthe session were simply added to those available); therefore,session one was used to adapt session two and session two toadapt session three. As the number of interactions for eachadapted session is 82,.the total number of test interactions was164. The recognition performance and the average value of R,for the different separate features and for their integration arereported in Table 1.:

TABLE ITHE RECOGNITmON PERFORMANCE AND AVERAGE SEPARATION RATIO R FOREACH SINGLE FEATURE AND POR THEIR INTEGRATION. DATA.ARE BASED ON

164 REAL INTERACTIONS AND A DATABASE OF 89 USERS.

can then be quantified through several measures. The decisionabout whether the confidence is sufficient to accept the systemoutput can be based on one or several of them. In the proposedsystem, a linear classifier, based on absolute and relativescores, ranks and their dispersion, will be used to accept/rejectthe final result. The following issues will be discussed:

1) degree of dependence of the features used;2) choice of the confidence measures to be used in the ac-

cept/reject rule;3) training and test of the linear classifier used to implement

the accept/reject rule.

As a preliminary step, the independence of the features usedin the identification process will be evaluated. It is known thatthe higher the degree of independence, the higher the informa-tion provided to the classifier. Let us consider a couple offeatures X and Y. Let {(x, yi) = I.....t represent the correspondingnormalized scores. They can be considered as random samplesfrom a population with a bivariate distribution function. Let A,be the rank of x, among x,, ..., x, when they are arranged indescending order, and BI the rank of y, among Y, .... y, definedsimilarly to A,. Spearman's rank correlation [27) is defined by:

_ _ .. o .-(19)

Feature Recognition (%) R

Voice 88 1.14Static 77 1.08

Dynamic 71 1.08

Face 91 1.56

Eyes 80 1.25Nose 77 1.25Mouth -83 1.28

ALL 98 1.65

B. Rejection

An important capability of a classifier is to reject input pat-terns that cannot be classified in any of the available classeswith a sufficiently high degree of confidence. For a personverification system, the ability to reject an impostor is critical.The following paragraphs introduce a rejection strategy thattakes into account the level of agreement of all the differentclassifiers in the identification of the best candidate.

A simple measure of confidence is given by the integratedscore itself: the higher the value, the higher the confidence of.the identification. Another is given by the difference of the twobest scores: It is a measure of how sound the ranking of thebest candidate is. The use of independent features (or featuresets) also provides valuable information in the form of therankings of the identification labels across the classifier out-puts: If the pattern does not .belong to any of the knownclasses, its rank will vary significantly from classifier to clas-sifier. On the contrary, if the pattern belongs to one of theknown classes, rank agreement will be consistently high. Theaverage rank and the rank dispersion across the classifiers canthen be used to quantify the agreement of the classifiers in thefinal identification. The confidence in the final identification

IXAI - )2 X(Bi--

where A and B are the average values of (AI) and (B,),.re-spectively. An important characteristic of rank correlation is itsnon-parametric nature. To assess the independence, of the fea-tures it is not necessary to know the bivariate distribution fromwhich the (XI, YI) are drawn, since the distribution of theirranks is known, under the assumption of independence. It turnsout that

1-2(20)

TABLE IITHE RANK CORRELATION VALUE OF THE COUPLES OF FEATURES. THE

PARDfrHESIZE) VALUES REPRESEMT THE SIONIFICANCE OF THE CORRELATION.St AND S2 REPRESENT THE DYNAMIC AND STATIC ACOUSTIC FEATURES

RESPECnvELY; Fl, F2, F3 REPRESENT THE EYES, NOSE AND MOUTHn

SI SZ Fl F2 F3S 1 1.00 0.64 0.06 0.08 0.04

(1.00) (1.00) (0.72) (0.77) (0.66)S2 1.00 0.03 0.07 0.03

(1.00) (0.61) (0.75) 0.61Fl 1.00 0.11 0.10

S(1.00) (0.84 (0.83)F2 1.00" 0.50

(1.00) (1.00)F3 1.00

(1.00)

is distributed approximately as a Student's distribution withI- 2 degrees of freedom [271; It is then possible to assess thedependence of the different features used by computing therank correlation of each couple and by testing the correspond-

961


ing significance. Results for the features used in the systemdeveloped are given in Table II.SThe acoustic features are clearly correlated, as well as the

nose and mouth features. The latter correlation is due to theoverlapping of the nose and mouth regions, which was foundto be necessary in order to use facial regions characterized bythe same coordinates for the whole database. Acoustic andvisual features are independent, as could be expected..'

Score

,Rar etgma

Fig. 6. Let us represent the match with the database entries by means of theintegrated score, the standard deviation of the rankings across the differentfeatures and the normalized ratio of the first to second best integrated score.The resulting three dimensional points are plotted and marked with a - ifthey represent a correct match or with a x if the match is incorrect. Visualinspection of the resulting point distribution shows that the two classes ofpoints can be separated well by using a plane.

The feasibility of using a linear classifier was investigatedby looking at the distribution of acceptable and non-acceptable 1 best candidates in a 3D space whose coordinatesare the integrated score, a normalized ratio of the first-to sec-ond best score and the standard deviation of the rankings. Ascan be seen in Fig. 6 a linear classifier seems to be appropri-ate. The full vector de Rta used as input to the linear classi-fier is given by:

1) the integrated score, St, of the best candidate;2) the normalized ratio of the first to the second best inte-

grated score:

Sh -0.5Sq-0.5

3) the minimum and maximum ranks of the first and secondfinal best candidates (four entries);

4) the rank standard deviation of the first and second finalbest candidates (two entries);

5) the individual ranks of the first and second final bestcandidates (10 entries).

To train the linear classifier the following procedure was used.A set of positive examples (pi) is derived from the data rela-tive to the persons correctly classified by the system. A set ofnegative examples (nj) is given by the data relative to the bestcandidate when the system did not classify the user correctly.The set of negative examples can be augmented by the data ofthe best candidate when the correct entry is removed from the

2. Unacceptable best candidates derive from two sources: misclassified us-ers from real interactions and best candidates from virtual interactions, char-acterized by the removal of the user entry from the data base.

database, thereby simulating the interaction with a stranger.The linear discriminant function defined by the vector w canbe found by minimizing the following error:

/2 /2

E= a 1- + 1-# l I-t +e leS+e-( .,-,,il wN,) 1 wnA J.J

(22)

where a and represent the weight to be attributed to falsenegatives and to false positives respectively and 1= 18 is thedimensionality of the input vectors. When a = = 1, E repre-sents the output error of a linear perceptron with a symmetricsigmoidal unit.

Final acceptance or rejection of an identification, associatedto a vector d, is done according to the simple rule:

wad i + w,+, > 0 accept1=1

wd, + w+ I 5 0 reject

(23)

(24)

Note that the LHSs of (23) and (24) represent the signed dis-tance, in arbitrary units, of point d from the plane defined by wthat divides the space into two semispaces. Points lying in thecorrect semispace contribute to E inversely to their distancefrom plane w. Points lying near the plane contribute with a or0 while points lying in the wrong semispace and at great dis-tance from the discriminating plane contribute with 2aor 2/3.If the two classes of points are linearly separable it is possibleto drive E to zero (see [281, [291). A stochastic minimizationalgorithm (301, (31] was used to minimize E.

When the system is required to work in a strict mode (no er-rors allowed, that is, no strangers accepted), >> a should beconsidered in the' training phase'. Note that a similar discrimi-nant function can be computed for each of the recognition sub-systems (i.e., face recognition and voice recognition), therebyenabling the system to reject an identification when it is notsufficiently certain even when not all of the identification cuesare available. The training/test of the classifier followed aleave-one-out strategy to maximize the number of data avail-able in the training phase [32]. The classifier is trained by us-ing all but one of the available samples and tested on the ex-cluded one. Theperformance of the classifier can be evaluatedby excluding in turn each of the available samples and averag-ing the classification error.

In the reported experiments, the available examples weregrouped per interacting user. The leave-one-out method wasthen applied to the resulting 87 sets (the number of users thatinteracted with the system) to guarantee the independence ofthe training and test sets.

Each set was used in turn for testing, leaving the remaining86 for training. The results are reported in Table Ill. A completeoperating characteristic curve for the integrated performanceshown in Table l is reported in Fig. 7 where the stranger-acceptedand familiar-rejected rates at different fPa ratios are plotted.

"

l

r "

34 r1 "

M "

BRUNELLI AND FALAVIONA: PERSON IDENTIFICATION USING MULTIPLE CUBS

TABLE llERROR RATES OF THE SUBSYSTEMS AND OF THE COMPLETE SYSTEM WHEN A

REJECION TNhRESHOLD Is INTRODUCED. DATA ARE BASED ON THE SUBSET OF

INmRACfONS FOR WHICH BOTH FACE AND SPEECH DATA WERE AVAn.ILABLE(i 55'Our OF 164)

Error (%)

Face

Stranger accepted 4.0Familiar rejected 8.0

Familiar misrecog 0.5

VoiceStranger accepted 14.0Familiar rejected 27.0

Familiar misrecog. 1.0

Integrated

Stranger accepted 0.5Familiar rejected 1.5

Familiar misrecog. 0.0

o Slongr occropdo for.wlo rejected

Let ( C) be the set of classifiers. Each of them associates toeach person X some numerical data Xj that can be considered avector. By comparison with the ith database enitry, a normal-ized similarity score Su can be computed. Each score Sy can beassociated to its rank rU in the list of scores produced by clas-sifier C. The output of each classifier can then be regarded asa list of couples {((Sj, r )) =l.....t where I represents the numberof people in the reference database. A mapping is sought £o1such that:

Ssris)= if i is the correci label for X (25)Ail(Soj,rit, .... Sis~rs)= tews (25)

If, after mapping the list of scores, more than a label quali-fies, the system rejects the identification. It is possible to relaxthe definition of £4o by letting the value of the mapping spanthe whole interval [0, 1]. In this way the measurement levelcharacter of the classification can be retained. The new map-ping £ can be interpreted as a fuzzy predicate. The followingfocuses on the fuzzy variant, from which the original formula-tion can be obtained by introducing a threshold m

o = O(£(x4 ) - (0) (26)

when false' positives and false negatives are

Similar experiments were run on the acoustic and visualfeatures separately and are also reported in Table III. The re-suits show that the use of the complete set of features providesa relevant increase in reliable performance over the separatesubsystems.

C. Hybrid Level Integration' In this sub-section, a hybrid rank/measurement level at

which multiple classifiers can be combined will be introduced.The approach is to reconstruct a mapping from the sets ofscores, and corresponding ranks, into the set 10, I). Thematching to each of the database entries, as described by avector of five scores and the corresponding ranks should bemapped to 1, if it corresponds to the correct label, and to 0otherwise. The reconstruction of the mapping proceeds alongthe following steps:

1) find a set of positive and negative examples;2) choose a parametric family of mappings;3) choose the set of parameters.for which the corresponding

mapping minimizes a suitable error measure over thetraining examples.

Another way to look at the reconstruction of the mapping isto consider the problem as a learning task, where, given a setof acceptable and non acceptable inputs, the system should beable to appropriately classify unseen data.

where 0() is the Heavyside unit-step function and xj = (S 1, r 1,* , Sa, r15es) is a ten-dimensional vector containing the featurenormalized matching scores and corresponding ranks. Thegoal is to approximate the characteristic function of the correctmatching vectors as a sum of Gaussian bumps. Therefore thesearch for £ is conducted within the following family of func-tions:

L(;{cata}aT-= c aG(IX-tall))a

(27)

where

G(x) e-'I2

Q~ -.) = 1-1.

(28)

(29)

.(30)

p1 being a diagonal matrix with positive entries, x, t, e R'

and c, e R. The approximating function can be represented asa HyperBF network [33] whose topology is reported in Fig. 8.The sigmoidal mapping is required to ensure that the codomainbe restricted to the interval (0, 1). The location Ia, shape £, andheight ca of each bump are chosen by minimizing the follow-ing error measure:

E = y - a caG -to)I a

(31)

where ({ (xu, yu) ), is a set of examples (points at which the valueof the mapping to be recovered is known). The first subscript Idenotes the database entry from which xy is derived and the'second subscriptj represents the example.

0.0 0.5 1.0 1.5 2.0P/a

Fig. 7. System performanceweighted differently.

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTtLLIGENCE, VOL. 17, NO. 10, OCTOBER 1995

Fig. 8. The function used to approximate the mapping from the score/rankdomain into the interval (0, 1) can be epmsented as a HyperBP network.

The required value of the mapping at xu is 1 when i is thecorrect label (class) for the jth example and 0 otherwise. Theerror measure E is minimized over the parameter space((ca, t,)), by means of a stochastic algorithm with adaptivememory (311. The number of free parameters involved in theminimization process dictates the use of a large set of exam-ples. As a limited number of real interactions was available, aleave-one-out strategy was used for training and testing thesystem as for the linear classifier previously described. Fromeach of the available user-system interactions, a virtual inter-action was derived by removing from-the database the entry ofthe interacting user, thereby simulating an interaction with astranger. For each interaction j

1) the vector corresponding to the correct database entryprovides a positive example;

2) the vectors of the first ten, incorrect, entries of the realinteraction (as derived from sorting the integrated scoresof Section IV.A) and the vectors of the first ten entries ofthe virtual interaction provide the negative examples.

The reason for using only the first ten incorrect entries is thatthe matching scores decay quickly with rank position in thefinal score list and additional examples would not providemore information. Data from different interactions of the sameuser were then grouped. The resulting set of examples wasused to generate an equal number of different training/testingset pairs. Each set was used in turn for testing, leaving the re-maining ones for training. The problem of matching the num-ber of free parameters in the approximation function to thecomplexity of the problem was solved by testing the perform-ance of networks with increasing size. For each network size, avalue for threshold ca of (26) was computed to minimize thetotal error defined as the sum of the percentage of acceptedstrangers, misrecognized and rejected database persons. InFig. 9, the total error is reported as a function of the networksize. Note that the threshold is computed on the test set, so thatit gives an optimistic estimate. To obtain a correct estimate ofsystem performance, a cross-validation approach was used forthe net giving the best (optimistic) total error estimate. Let[o, W1] be the interval over which the total error assumes itsminimum value (see Fig. 10 ). The threshold value can be cho-sen as:

* ob favoring acceptance over rejection;- (a+,)2;* col favoring rejection over acceptance.

The resulting performance is reported in Table IV. Note that

using o the system was able to reject all of the strangers,which is the ultimatWrequirement for a reliable system, miss-ing only 3.5% of the known users.

TABLE IVTHE PERFORMANCE OPFTHE SYSTEM WHEN USING A HYPERBF NETWORK

wrm 21 UNr's To PERFORM ScoRs WrTGRATION

0 Stranger Familiar Familiar misre-accepted rejected cog. (%)

%(%)

0.5 3.0 0.0(4 + av)/2 0.5 3.0 0.0

,. 0.0 3.5 0.0

Totlo Error

0.08

0.04

0,02

A no0 10 20

Network units

Fig. 9. Tihe total error achieved by networks with different numbers of units.The total error is computed by summing the percentage of accepted strangers,misrecognized, and rejected database people. For each net size, a thresholdwas chosen to minimize the cumulative error.

Error estimateso Satrnger ocpteo fa;e rejecteda lorita mimc.

Fig. 10. Enmor percentages as a function of the rejection threshold for a Gaus-sian-based expansion.

V. CONCLUSIONS

A system that combines acoustic and visual cues in order toidentify a person has been described. The speaker recognitionsub-system is based on vector quantization of the acoustic pa-rameter space and includes an adaptation phase of the code-books to the test environment. A different method to performspeaker recognition, which makes use of the Hidden MarkovModel technique is under investigation.

A face recognition subsystem also was described. It is basedon the comparison of facial features at the pixel level using asimilarity measure based on the L, norm.

The two subsystems provide a multiple classifier system. Inthe implementation described, five classifiers (two acousticand three visual) were considered. The multiple classifier op-erates in two steps. In the first one, the input scores are nor-malized using robust estimators of location and scale. In the

I-

964

BRUNELLI AND FALAVIGNA: PERSON IDENTIFICATION USING MULTIPLE CUBS

second step, the scores are combined using a weighted geo-metric average. The weights are adaptive and depend on thescore distributions. While normalization is fundamental tocompensate for input variations (e.g., variations of illumina-tion, background noise conditions, utterance length and ofspeaker voices), weighting emphasizes the classification powerof the most reliable classifiers. The. ;use of multiple cues,acoustic and visual, proved to be effeg'e in improving per-formance. The correct identification rate of the integrated sys-tem is 98% which represents a significant improvement withrespect to the 88% and 91% rates provided by the speaker andface recognition systems respectively. Future use of the hiddenMarkov model technique is expected to improve performanceof the VQ-based speaker recognizer.

An important capability of the multiple classifier itself is therejection of the input data when they can not be matched withsufficient confidence to any of the database entries.

An accept/reject rule is introduced by means of a linearclassifier based on measurement and rank information derivedfrom the five recognition systems. A novel, alternative, ap-proach to the integration of multiple classifiers at the hybridrank/measurement level is also presented. The problem ofcombining the outputs of a set of classifiers is considered as alearning task. A mapping from the scores of the classifiers andtheir ranks into the interval (0, 1) is approximated using a Hy-perBF network. A final rejection/acceptance threshold is thenintroduced using the cross-validation technique. System per-formance is evaluated on data acquired during real interactionsof the users in the reference database. Performance of the twotechniques is similar. ,

The current implementation of the system is working on anHP 735 workstation with a Matrox Magic frame grabber. Inorder to optimize system throughput, it relies on a hierarchicalmatch with the face database.

The incoming picture, represented by a set of features iscompared at low resolution with the cqmplete database. Foreach person in the database, the most similar feature, amongthe set of available images, is chosen and the location of thebest matching position stored. The search is then continued atthe upper resolution level by limiting the search to the mostpromising candidates at the previous level.

These candidates are selected by integrating their facescores according to the procedure described in Section IV.A.All available data must be used to secure a reliable normaliza-tion of the scores. However, new scores at higher resolutionare computed only for a selected subset of persons and thisconstitutes a problem for the integration procedure. In fact,scores from image comparisons at different levels would bemixed, similarity values deriving from lower resolutions beingusually higher. To overcome this difficulty, the scores from theprevious level are reduced (scaled) by the highest reductionfactor obtained comparing the newly computed scores to thecorresponding previous ones.

The performance, measured on the data sets used for the re-ported experiments, does not decrease and the overall identifi-cation time (face and voice processing) is approximately fiveseconds.

The same approach, using codebooks of reduced size could

be applied to the speaker identification system, thereby in-creasing system throughput. Adding a subject to the databaseis a simple task for both subsystems. This is due to the modu-larity of the databases, each subject being described independ-ently of the others. The integration strategy itself does not re-quire any update. The rejection and the combined identifica-tion/rejection procedures do require updating. However, thetraining.of the linear perceptron and of the HyperBF networkcan be configured more as a refinement of a suboptimal solu-tion (available from the.previous database) than as the compu-tation of a completely unknown set of optimal parameters.While the system, as presented, is mainly an identificationsystem, a small modification transforms it into a verificationsystem. For each person in the database it is possible to selecta subset containing the most similar people (as determined bythe identification system). When the user must be verified theidentification system can be used using the appropriate subset,thereby limiting, the computational effort, and verifying theidentity of the user by means of the techniques reported in thepaper.

Future work will have the purpose of further improving theglobal efficiency of the system with the investigation of moreaccurate and reliable rejection methods.

ACKNOWLEDGMENTS

The authors would like to thank Dr. L. Stringa,Prof. T. Poggio and Prof. R. de Mori for valuable suggestionsand discussions. The authors are grateful to the referees formany valuable comments.

REFERENCES

(1) T. Poggio and L Stringa. "A project for an intelligent system: Visionand learning." Int'I J. Quantum Chemistry, vol. 42, pp. 727-739, 1992.

12) F.K. Soong and A.E. Rosenberg, "On the use of instantaneous and transi-tional spectral information in speaker recognition," IEEE Trans Acoustic,Speech, and Signal Processing, vol. 36, no. 6, pp. 871-879, 1988.

(3) S. Fumi, "Cepstrum analysis technique for automatic speaker verifica-tion," IEEE Trans Acoustic, Speech, and Signal Processing, vol. 29,no. 1, pp. 254-272, 1981.

[4] A.E. Rosenberg, J. DeLong, C.H. Lee,. B.H. Juang, and F.K. Soong,'The use of cohort normalized scores for speaker verification," Proc.ICSLP, vol. I, pp. 599-602, Banff, Canada, Oct. 1992.

(5) T. Matsui and S. Furui, "Similarity normalization method for speakerverification based on a posteriori probability," Proc. ESCA Workshopon Automatic Speaker Recognition Identification Verfication, pp. 59-62, Martigny, Switzerland, Apr. 1994.

(6) R. Brunelli, D. Falavigna, T. Poggio, and L Stringa, "A recognitionsystem, particularly for recognizing people." patent no. 93112738,1993.Priority IT/i 1.08.92/IT T0920695.

(7) R. Brunelli, T. Poggio, D. Falavigna, and L Stringa. "Automatic personrecognition by using acoustic and geometric features," Tech. Report9307-43, I.R.S.T., 1993, to appear in Machine Vision and Applications.

(8) G.R. Doddington, "Speaker recognition, identifying people by theirvoices," Proc. IEEE, vol.73, no. I1, 1985.

19] A.E. Rosenberg and F.K. Soong. "Evaluation of a vector quantization Talkerrecognition system in text independent and text dependent modes," Com-puter Speech and Language, vol. 2, no. 3-4. pp. 143-157. 1987.

(101 H. Gish J. Makhoul, S. Roucos, "Vector quantization in speech coding,"Proc. IEEE, vol. 73, no. 1), pp. 1551-1588. 1985.

965.

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL 17, NO. 10, OCTOBER 1995

[(11) D. O'Shaughnessy, Speech communication. Addison-Wesley, 1987.(121 P. Melmerstein and S.B. Davis, "Comparison of parametric repesenta

tions for monosyllabic word recognition in continuosly spoken son-tences," IEEE Trans Acoustic, Speech, and Signal Processing, vol 28,no. 4, pp. 357-366, 1980.

(13) G. Carli and R. Gretter "A start-cnd point detection algorithm for a real-time acoustic front-end based on dsp32c vine board," Proc. ICSPAT,pages 1,011-1,017, Boston, Nov. 1992.

[14] R. Brunelli and T. Poggio, '"Pace Recognition: Features versus Tem-plates," IEEE Trans. Pattern Analysis and Machine Intelligence,vol. 15, no. 10, pp. 1,042-1,052. 1993.

[15) R. Brunelli, "Estimation of pose and illuminant direction for face proc-essing," A.L Memo No. 1499, Massachusetts Inst. of Technology, 1994.P.J. Burt, "Smart sensing within a pyramid vision machine," Proc.IEEE, vol. 76, no. 8, pp. 1,006-1,015, 1988.

[(16) P. W. Hallinan, "Recognizing human eyes," SPIE Proc., vol. 1570,pp. 214-226, 1991.

(17) L Stringa. "Eyes detection for face recognition," Applied ArtificialIntelligence, vol. 7, pp. 365-382, 1993.

(18] D.H. Ballard and C.M. Brown, Computer Vision. Englewood Cliffs,N.J.: Prentice Hall, 1982.

(191 R. Brunelli and S. Messelodi, "Robust estimation of correlation: Withapplication to computer vision," Pattern Recognition, vol. 28, no. 6,pp. 833-861, 1995.

(20) L Stringa, "Automatic face recognition using directional derivatives,"Tech. Report 9205-04, i.R.S.T., 1991.

(21] L Xu, A. Krzyzak, and C.Y. Suen, "Methods of combining multipleclassifiers and their applications to handwriting recognition," IEEETrans. Systems, Man, and Cybernetics, vol. 22, no. 3, pp. 418-435, 1992.

(22) FR. Hampel, PJ. Rousseeuw, E.M. Ranchetti, and W.A. Stahel, Robust Statis-tics:. The Approach Bared a Influence Fwunctio. John Wiley & Sons, 1986.

[23) P. J. Huber, Robust Statistics. Wiley, 1981.(24) P.B. Bonissone and K.S. Decker, "Selecting uncertainty calculi and

granularity: An experiment in trading off precision and complexity,"J.F. Lemmer, LN. Karnak, eds., Uncertainty in Artificial Intelligence,pp. 217-247, North Holland, 1986.

(25) P.B. Bonissone, S.S. Gans, and K.S. Decker, "Rum: A layered architec.ture for reasoning with uncertainty," Proc: 10th Int'l Joint Conf onArtificial Intelligence, pp. 891-898, Milan, Aug. 1987.

(26) R.V. Hogg and A.T. Craig, Intro. to Statistics. Collier-Macmillan, 1978.[27) R.O. Duda and P.E. Hart, Pattern Recognition and Scene Analysis, New

York: Wiley, 1973.(28) Y.-H. Pao, Adaptive Pattern Recognition and Neural Networks. Read-

ing, Mass: Addison-Wesley, 1989.(29) R. Brunelli. On training neural nets through stochastic minimization,

Neural Networks, vol. 7, no, 9, pp. 1405-1412, 1996.[30) R. Brunelli and G. Tecchiolli. Stochastic minimization with adaptive

memory, J. of Computational and Applied Mathematics, pp. 329-343, 1995.[31) K Pukma Inoduction to salistical pareon gnitionrL Academic Press, 1990.[32) T. Poggio and P. Girosi, Regularization algorithms for learning that are

equivalent to multilayer networks, Science, voL 247, pp. 978-982,1990.

Roberto Brunelli meceivesl his de~ree in nhvdca.--.-.-.- Brune... - r--- .. ... d ... p.. , --- ,

with honors, from the University of Trento in 1986.He joined IRST in 1987 where he works in theComputer Vision Group. In the past, he was in-volved in research on computer vision tools, analysisof aerial images, development of algorithms workingon compressed description of binary images, neuralnetworks, face recognition, and optimization. Hiscurrent major involvement is in the MINDEX Proj.e,'c aim at the' de~ulo mPnt ofinnovatiu tools for

content based image retrieval in large databases. Hiscurrent interests include optimization, robust statistics, object recognition,and machine learning.

Daniele Falavigna received his degree in electronicengineering from the University of Padova in 1985.He joined IRST in 1988 wher he works in theSpeech Recognition Group. In the past, his'researchhasincluded acoustic modeling of speech, automaticsegmentation and labeling, and speaker identifica-tion. His current major involvement is in the devel-opment of speech recognition systems. His currentinterests include speech analysis, speech recogni-tion, speaker identification and verification, and

^i"^^a "

9t

4

sign processing.1Z

Machine Vision and Applications (1995) 8:317-325


Machine Vision andApplicationso Springer-Verlag 1995

Automatic person recognition by acoustic and geometric features*R. Brunelli, D. Falavigna, T. Poggio**, L. Stringa

Institute for Scientific and Technological Research, 1-38050 Povo, Trento, Italy

Abstract. This paper describes a multisensorial person-iden-tification system in which visual and acoustic cues are usedjointly for person identification. A simple approach, basedon the fusion of the lists of scores produced independentlyby a speaker-recognition system and a face-recognition sys-tem, is presented. Experiments are reported that show thatthe integration of visual and acoustic information enhancesboth the performance and the reliability of the separate sys-tems. Finally, two network architectures, based on radialbasis-function theory, are proposed to describe integrationat various levels of abstraction.

Key words: Face recognition - Speaker identificationi -

Classification

1 Introduction

This paper describes an automatic person-recognition systemthat uses both acoustic features, derived from the analysis ofa given speech signal, and visual features, related to distinc-tive parameters of the face of the person who spoke. Visualand acoustic cues are used jointly for person identification:several methods of combining the acoustic and visual in-formation at various levels are described. Experiments arepresented that show the superior performance of the wholesystem with respect to both the speaker recognition system(SRS) and the face recognition system (FRS). The systemcan be used for either identification (e.g., as an electronicconcierge to recognize people in small organizations) or ver-ification applications (e.g., as a smart door key to control theentrance of a house) even if, in our experiments, we havestressed only the identification function.

The SRS uses acoustic parameters computed from thespectrum of short-time windows of the speech signal, whilethe FRS is based on geometric data represented by a vector.describing discriminant facial features such as the position

* Italian Patent No. T092A000695. European extension in progress." Anificial Intelligence Laboratory, Massachusetts Institute of Technol-

ogy, Cambridge. MA 02139, USACorrespondence to: R. Bnrunei

and width of the nose, chin shape, and so on. We have con-sidered various ways of combining the SRS and the FRSresults:

1. At the level of the outputs of the single classifiers: the listof scores (probability estimates or distances) producedindependently by the two systems can be used in anintegrated classifier system (i.e., the two sets of scoresare optimally weighted and summed to produce a uniquefinal list).

2. At the level of the measurements carried out on both thespeech and visual signals: acoustic and geometric param-eters are considered as a unique vector, lying in the carte-sian product of the acoustic and visual spaces,. which willbe successively classified by a speaker specific classifier.Examples of such a classifier are: a bayesian classifier, amultilayer perceptron (MLP) classifier or a radial basisfunction (RBF) classifier.

The experiments reported in this paper are based on thefirst integration strategy. Some HyperBF network architec-tures supporting the second integration strategy are also de-scribed. The paper is organized as follows:

- Section 2 describes the overall system architecture.- Section 3 presents the acoustic and visual databases.- Section 4 analyzes the SRS and FRS used for this

work.- Section 5 describes the method used in this. work to

integrate the two systems. An alternative strategy, currentlyunder investigation, is also presented.

- Section 6 reports the results of the experiment(s.

2 System overview

The basic structure of the identification system is depicted inFig. 1. An attention module detects changes in the image cap-tured by a CCD. camera by using background subtraction andthresholding. Whenever a significant change in the scene isdetected, a snapping module is activated. This module waitsuntil a stable scene is detected (a still person staring intothe camera) and checks for the satisfaction -of simple con-straints (mainly on the changed area) before unlocking theseparate action of the two recognition systems. The snapped

the DAT (sampled at 48 kHz) was downsampled to 16 dkHz,manually endpointed, and stored on a computer disk.

3.2 The visual database

The database is composed of 188 images, 4 per person. Ofthe 4 pictures available, the first 2 'were taken in the samesession (at an interval of a few minutes), while the otherpictures were taken at intervals of 2-4 weeks. The pictureswere taken with a CCD camera at a resolution of 512 x 512pixels as frontal views. The subjects were asked to lookinto the camera, but no particular efforts were made to en-sure perfectly frontal images. .The illumination was partiallycontrolled. The same powerful light was used, but the en-vironment in which the pictures were taken was exposed tosunlight.,through windows.

The pictures were taken randomly during daytime/Thedistance of the subject from the camera was fixed only ap-proximately, so that scale variations of as much as 30% werepossible.

Fig. 1. The flow of activation of the various modulesmultisensorial identification system

belonging to the

image is fed directly to the face-recognition system, whilethe system asks the person to utter a sequence of digits ina continuous way. A boundary detection module, applied tothe input speech signal, separates the voice signal from thebackground noise, determining both the starting and endingtimes of each voice segment and the corresponding dura-tion. If the total duration of the voice segments detected bythe start-end point-detection module is not long enough, thesystem asks the user to read the requested digits again.

The outputs of the SRS and FRS are fed into an inte-gration module; we discuss its architecture in the followingsections. The output of the integration module is sent to thelast module that identifies the user.

3,Test database description

The acoustic and the visual databases consist of 42 and 47people, respectively; the database used in the experimentsis composed of 33 people, without shaves and moustache,belonging to both databases. The FRS and SRS were trainedseparatelyfor each person in the two databases while testing(see Sect. 5) was done on the common 33 people database.

3.1 The acoustic database

The database consists of 8400 isolated Italian digits, fromzero to nine. Each speaker uttered 200 digits in five record-ing sessions: during each session the speaker provided fourrepetitions of each digit. The entire database was collectedover a period of about two months. Each recording session isseparated from the previous one by an interval varying from3 to 10 days. The Sessions were recorded in an acousticallyisolated room of our Institute by means of digital audio tape(DAT) equipment (SONY TCD-DIO). The digital signal on

4 The recognition systems

In this section the SRS and the FRS used in the experimentsare described. Both systems operate in two steps:

1. A sequence of parameter vectors or a single parametervector is derived from the speech signal or the visualsignal, respectively.

2. The input parameter vectors are matched on the basis ofdistance measures with respect to reference models, sothat each system produces a list of scores labeled withthe identifier of the corresponding reference model.

4.1 The speaker-recognition system

Automatic speaker-recognition is a topic that has been widelyinvestigated in the past years (Atal 1976; Doddington 1985;Furui 1981). Recently, some methods based on the process-ing and modeling techniques also used for speech recogni-tion have been studied and tested (Rosenberg 1987; , Tishby1991).

Speaker-recognition systems can be divided, accordingto the application area, into speaker-verification or speaker-identification systems. A speaker-verification system sim-ply checks (confirms or not) the identity claimed by a per-son (e.g., before she/he accesses a reserved place or ser-vice). A speaker-identification system must determine who(within a known group of people) gave the (speech) sig-nal. Speaker-recognition systems can also be text dependent(i.e., the speaker must utter a predefined key sentence) ortext independent. The major problem in speaker-recognitionis represented by intersession variability (variability of thespeech signal with time) due both to different recording ortransmission conditions and to intrinsic variability in peo-ple's voices.

The SRS used for this work is based on vector quantiza-tion (VQ) and is similar to the one described in Rosenberg(1987). A block diagram of this system is depicted in Fig. 2.

The system measures the distances between two distinctsets of acoustic parameters (static and dynamic) and corre-sponding prototypes or codebooks, derived, during a trainingphase, from speaker-specific speech material.

4.1.1 Speech signal analysis

The speech signal, sampled at 16 kHz and quantized over 16bits, is first pre-emphasized by using a digital filter havinga transfer function H(z) = 1 - 0.95 x z- 1 . We analyze thepre-emphasized signal every 10 ms, using a 20 ms Hammingwindow and, for each window position, we compute thefollowing parameters:

1. Eight Mel scale cepstral coefficients (Davis 1980) com-puted from the log-energy outputs of a 24 triangularband-pass filter bank applied to the power spectrum ofthe given window. These parameters are said to be static.

2. The corresponding first-order time derivatives, evaluatedby means of a linear fit over nine frames of static param-eters, centered on the given window. These parameiersare said to be dynamic.

4.1.2 The VQ-based speaker-recognition system

As seen previously, the reference models of each speaker(Fig. 2) consist of two codebooks: one corresponding to thestatic parameters and the other to the dynamic ones. Thecodebooks were generated by applying the Linde-Buzo-Grayalgorithm (Makhoul 1985) to a set of shonrt-time spectral vec-tors (static or dynamic), derived from 100 digits belongingto the first 2.5 recording sessions of each speaker. This algo-rithm searches for prototype vectors that minimize a globaldistortion measure defined on the given training set.

A weighted euclidean distance was used for the code-book design and recognition. The weights correspond to theinverse of the pooled variances of the components of thetraining vectors averaged over all training utterances andspeakers. Therefore, if 0i, Oi are two parameter vectors, theirdistance is defined as:

k- 2 (1)kI

where ak represents the average pooled variance of the kthcomponent of the parameter vector. As previously seen, p isequal to 8.

During the recognition phase, the distances (Fig. 2) be-tween the sequences of static and dynamic vectors ob-tained from the input speech signal and the correspond-ing codebooks of each speaker are evaluated. Therefore, if9 = 01, .... ,OT is the static (or dynamic) input sequence,and T, = i,;..., iyj,, are the vectors of the jth static (ordynamic) codebook, then the total static (or dynamic) dis-tortion is defined as:

1 T MD(9, lP)) = , min d(s, ,j4) (2)

t=

The static and dynamic distances are then normalized withrespect to their average values, computed on the training

set, and summed. The performance of the system dependson both the acoustic resolution (i.e., the number M of vec-tors in each speaker codebook) and the number L of digitscontained in' the input signal. The average identification errorevaluated on a test set composed of 100 digits per speaker(belonging to the last 2.5 recording sessions) is 49% forM = 4, L = 1 and 0% for M = 64, L = 3. In Fig. 3,the results for spectral resolutions varying from 4 to 64 andutterance lengths varying from I to 10 digits are reported.

4.2 The face-recognition system

Every day, humans recognize people by visual cues withoutapparent effort. The ease with which we recognize familiarpeople by their faces has led us to underestimate the diffi-culty of the problem. Extended psychophysical experimentshave revealed the fact that, even for people, recognition re-quires a lot of processing and is by no means an innateability.

There are two main strategies for automatic face recog-nition, both of which can be said to mimic some of theprocesses used by people: .

1. The iconic strategy is based on the comparison of suit-ably preprocessed image patches. Recognition is ef-fected by comparing (e.g., through the value of cross-correlation or some other suitable distance measure) anunknown image with stored templates of distinctive fa-cial regions (Baron 1981; Brunelli and Poggio 1993;Stringa 1992b, 1992; Turk and Pentland 1991).

2. The geometric strategy computes a set of geometric fea-tures that describe the size and the layout of the vari-ous features of the faces, and recognition proceeds bycomparing the unknown descriptive vector with a set ofreference vectors (known people) stored in a database(Bichsel 1991; Brunelli and Poggio 1992, 1993; Cot-trell and Fleming 1990; Craw et al. 1987; Kanade 1973;Nakamura et al. 1991).

Several approaches can be classified within this simple tax-onomy, and a comparison of the two basic strategies can befound in Brunelli and Poggio (1993). In this paper we focuson the geometric strategy. The reason is twofold:

1. It gives a more compact representation and guarantees ahigh speed in the recognition process.

2. It gives an example of how good performance results canbe obtained by integrating simple (and fast) recognitionmodules that do not have a very high performance whenconsidered separately.

As will be apparent from the discussion of the integrationstrategies, the use of a template-matching strategy fits in anatural way. Template matching typically compares suitablypreprocessed images (or patches of images) representing thesalient features of the face pattern. Whether the comparisonis done by a suitable distance measure.or by a correlationcoefficient, the result can readily be incorporated into theproposed system, alternatively or in conjunction with theinformation explicitly used.

A set of geometrical features describing a frontal viewof a face is computed automatically in the following steps(Brunelli and Poggio 1993):

320

Codebooks for Speaker N.1

Codebooks for Speaker N. K

% identification error

p.

- 14

0.00 5.00 l0.0(

Spectra Res.. 64

Spectral Res.: 32Spcrl Res.. 16

Spectral Res.. 8Spctra Res.: 4

Fig. 2. Block diagram of the vector-quantization-basedspeaker-recognition system

Fig. 3. Error rate of the speaker-recognition system (SRS),evaluated on the whole 42-speaker test set as a function of thenumber of digits in the input signal: various curves correspondto various spectral resolution values

Fig. 4. Geometrical features (black) used in the face-

nr fdi to recognition experiments (information on the eyebrow arch is

50.00

45.00

40.00

35.00

30.00

25.00

20.00

15.00

10.00

5.00

0.00

I"

-

10.0(0.00

I.

* -'' " ' -

Bi not eported

. 5.00 ,

321

1. Eyes are located so that the image can be normalizedboth in size and rotation in the image plane (Stringa1993).

2. An average face model is used to focus the system pro-gressively, in a sequential way, on the various regions ofthe face so that relevant feature points can be computed.

3. A descriptive vector is built from the relative positionsof the feature points.

This results in the computation of 35 geometrical features(Fig. 4) that can be used for recognition:

1. Eyebrow thickness and vertical position relative to thecenter of the eye

2. A coarse description of the left eyebrow arch (8 mea-surements)

3. Vertical position and width of the nose4. Vertical position, width (upper and lower lips) and height

of the mouth5. Eleven radii describing the chin shape6. Bigonial breadth (face width at nose position)7. Zygomatic breadth (face width halfway between nose tip

and eyes)

A detailed description of the algorithms employed can befound in Brunelli and Poggio (1993). Classification can thenbe based on a Bayes classifier. We make the simplifying as-sumption that the measurements of the various features sharethe same gaussian distribution for all the people apart fromtheir average value (Bichsel 1991). The covariance matrix Ecan then be estimated and classification can be based (Dudaand Hart 1973) on the following distance, which is relatedto the probability of the given measurement:

d(x, m) = (x - mj)T E - t (x - mj) (3)

The identification of an unknown vector is taken to bethat of the nearest vector in. the database of known people.The database used for the reported experiments consists of132 vectors, 4 per person, representing the complete set of35 geometrical features.

The performance is estimated on a round-robin recog-nition experiment in which three of the available vectorsare used to compute an average representative vector, whilethe remaining one is used for testing. The results for thisdatabase were found to be 92% correct.

5 Integration of vision and voice

In this section, we briefly describe methods for integratingthe two classifiers into a more robust recognition system.There are two general classes of possible architectures. Thefirst is based on the integration of the output of two inde-pendent classifiers, one for the voice and one for the face.The second one is based on the direct combination of voiceand face features. The first approach is simpler and is usedin the reported integration experiments. The second methodis more speculative and is suitable for a larger-scale appli-cation.

5.1 Score integration

Both classifiers are essentially nearest-neighbor with a suit-ably defined metric. As already mentioned, when givenan. unknown input, they generate a list of possible labels,marked with their distance from the input vector. The list issorted in order of increasing distance so that the first elementof the list should be taken as the correct matching label. Themain difficulty in combining the results of the two classifiersis given by the inhomogeneous distances they produce. Thesimplest way to fix this is by inverse standard deviation nor-malization. Given the two lists, if d,,, and d, represent thedistances computed by the face recognizer and the speakerrecognizer, respectively for the ith reference, and .a and o9are the corresponding variances, a combined distance can bedefined as

D.=dv, d,Ov

0y

A natural way to look at the answer of a nearest-neighborclassifier is to map it into a list of scores rather than a listof distances. A possible mapping is:

d2 .-si

where z stands alternatively for Vision or Speech. This map-ping associates a value, in the.interval (0, 1] with a distance.In some sense, the higher the score the more likely thatthe correspondence is correct. Each list could be normalizedfurther by imposing the following condition:

zsz 1

The resulting lists could be given a Bayes interpretation, sug-gesting the following integration strategy in the hypothesisthat the two systems are independent:

Si = Sv, X Sa

As the two recognition systems do not have the same perfor-mance, it is natural to introduce a weighted merged score:

( (w)

The optimal weight w can be found by maximizing the per-formance of the integrated system on one of the availabletest sets.

5.2 A more integrated system

A closer integration of the two recognition systems couldbe ,made with a network that learns the relative reliabilityand discriminating power of the various features. We pro-pose two such architectures based on HyperBF Networks.Before introducing the architectures, let us briefly recall thebasic theory of these networks in the general framework ofsolving learning problems through multivariate function re-construction (Poggio and Girosi 1990).

Whenever the examples in a learning task are given innumeric form, learning can be seen as a problem of surfacereconstruction from sparse data (the examples). It is fairlyevident that, as it is, the task of reconstructing a surface

322

given its value at some points is ill posed. To make theproblem well behaved some additional constraints must beimposed. The most important constraint, as far as learningis concerned, is smoothness. If the reconstructed function isrequired to be smooth, we are assured that generalizationfrom available examples is indeed possible. The assumptionof smoothness allows us to formulate a variational problemwhose solution is the surface we are looking for:

N

H[f)= (y, - f(xi))2 + A0[fJi=1

where the first term measures the distance between the dataand the desired solution f; the second term is a functionalreflecting the cost associated with the deviation from someconstraint.reflecting some a priori knowledge. The unknownfunction can be considered as a linear combination of thefollowing form:

f(x)= cjcG(Ix - tlIw)I

(10)

where tj are called expansion centers and W is a square ma-trix used to compute the norm of a vector (x-t.)TWTW(x -

tj). Function G could be, among others, a gaussian or a mul-tiquadric function [for a more complete introduction to thetheory of HyperBF Network see Poggio and Girosi (1990)].The variational problem can be solved by finding the co-efficients, the centers and the metric. This can be done byminimizing the reconstruction error on the available exam-ples and the smoothness enforcing term.

The network architectures we propose for the integratedrecognition system are shown in Fig. 5. The underlying ideais to build a set of HyperBF modules, and 'train each moduleto recoristruct the characteristic function of a given person tobe identified. The module is supposed to output 1 on inputscorresponding to the given person and 0 otherwise. Eachmodule is trained with both positive and negative examples(competitive learning) so that both optimal example selectionand adaptive metric can be used profitably.

A typical choice for the basis function in this type ofnetwork would be the gaussian function.

The first network (Fig. 5b) is closer, to the simple ap-proach outlined. The modules in the first layer represent theoutputs of recognizers based on various types of data (suchas visual, static, or dynamic acoustic information). The mod-ules can be trained separately, taking advantage of the re-duced input dimensionality. The integration is operated bythe module in the second layer that is trained, again on aper person basis, to maximize its output on the vectors ofthe corresponding person. We call the resulting network ascore-integration network. The approach detailed in the cur-rent paper can be seen as a simplified version of this typeof network in which the first layer is removed.

The second network architecture (Fig. 5c) that we pro-pose is a feature-integration network. The input to the Hy-perBF modules is taken to be the cartesian product of thevarious inputs, and the module takes advantage of the simul-taneous knowledge of the available information. This net-work requires a more difficult training due to the extendedinput dimensionality.

C

Fig. 5. a The basic HyperBF'module; b a score-level integration network;c a feature-level integration network

6 Experiments and results

The strategy we choose to integrate the two recognition sys-tems is that of score fusion. The distances given, by eachof the systems have been transformed into score lists usingexponential mapping and inverse variance normalization, asexplained in Sect. 5.1.

As we said previously, the database used for the inte-gration experiments consists of 33 people. For each face 4lists of scores were available, while for each speaker and foreach utterance length, varying from I to 6 digits, we ran-domly selected-8 lists of scores from those corresponding tocodebook sizes equal ,to 4. In this way, a total of 32 inte-gration experiments could be performed for each person andfor each input utterance length (from I to 6 digits). One-ofthese 32 sets of score lists was used to estimate the optimalintegration weight (Fig. 6) while performance was measuredon the remaining sets. Useful data on the robustness of theclassification are given by an estimate of the intraclass vari-ability as opposed to the inter-class variability. We can dothis using the so-called min/max ratio (Poggio and Edel-man 1990; Poggio and Girosi 1989), which is defined as theminimum distance on a wrong correspondence over the max-imum distance from the 'correct correspondence. To estimatethe optimal weighting factor, the interval (0, 1], representingthe weight of the FRS, was evenly sampled and the valuemaximizing the performance was chosen (ties were resolved

.correct identification

1 M

0.00 0.50 1.00FRS weight

Min/Max ratio

3.80

3.60

3.40

310

3.00

2.80

2.60

2.40

2.20

2.00

1.80

1.60

1.40

1.20 0 .0

Fig. 6. Correct performance of the integrated system as a function of the weightassigned to the face-recognition system (FRS) using a score-level integrationmethod; various curves correspondto various numbers of digits in the inputspeech signal. The spectral resolution of the speaker-recognition system (SRS)is 4

Fig. 7. Performance of the face-recognition system (FRS) (horizontal line) andof both the speaker-recognition sytem (SRS) and the integrated system versusthe number of digits in the input speech signal; the spectral resolution of theSRS is 4; the vertical bars represent the standard deviation of the quantitymeasured

Fig. 8. Mia/Max ratio of the face-recognition, speaker-recognition, and inte-grated systems (these last two are functions of the number of digits in the6.00 input speech signal); the spectral resolution of the SRS is 4; the vertical bars

nr. of digits represent the standard deviation of the quantity measured

I I - I

ar. of digits

14 c

T-I -c

_l-

correct identification

I~

I

A A

i

-

-

1-

2.00 4.00

by maximizing the min/max ratio). The performance of theSRS, FRS, and integrated systems are quantified by:

I. The average performance (Fig.7)2. The min/max ratio (Fig. 8)

The analysis of this integrated performance (Fig. 7) showsthat, even with the simple integration scheme, nearly perfectrecognition is achieved. The FRS gave 92% correct resultswhile the SRS, using different speaker models, gave resultsvarying from 51% to 100% correct. The results of the in-tegrated system, using the less complex speaker model arealready up to 95% correct, and 100%o is achieved with aspeaker model of low complexity. Further benefits of inte-gration are evident in Fig. 8, where the min/max ratio, whichrepresents an estimate of the average separation amongclasses, is plotted. The increased class separation of the in-tegrated system, as measured by the mi/max ratio, suggeststhat rejection can be introduced with a more limited impacton performance than on either of the two separate systems.The curves in Fig. 6 represent the performance of the inte-grated system as a function of the weight used to merge thescores. As can be seen, the performance is a smooth func-tion of the weight. This means.that the the system is notvery sensitive to the weighting factor. As could easily beanticipated from the performance of the two systems, theoptimal combining weight shifts from a vision dominance ata low number of digits toward a voice dominance at an highnumber of digits. The same result can be reached by passingfrom lower to higher spectral resolutions of the SRS.

7 Conclusions'

In this paper, the superior performance that can be attainedby using multisensorial input has been demonstrated. Inte-gration of two recognizing systems, based on speech and vi-sion, respectively, greatly improves the performance, reach-ing the state of the art with a low-to-moderate complexity inthe constituent systems. Finally, some speculative networkarchitectures for integration have been proposed for use onlarger databases that currently require computationally ex-pensive recognition systems. A system based on the inte-gration strategy described and working in an uncontrolledenvironment is currently under evaluation. Preliminary re-suits suggest that the use of multiple identification cues hasa major impact, not only on absolute performance, but alsoon the ability of the system to reject an unknown user.

Acknowledgements. This research was done within MAIA, the integratedArtificial Intelligence project under development at IRST (Svinga, 01991a:Poggio and Stringa, 1992). We thank Dr. S. Messelodi for kindly sharingthe C language implementation of the attention and snapping module.

References

1. Ata B (1976) Automatic recognition of speakers from their voices.Proc IEEE 64:460-475

2. Baron RJ (1981) Mechanisms of human facial recognition, Int J ManMachine Stud 15:137-178

3. Bichsel M (1991) Strategies of robust object recognition for the iden-tification of human faces. Phd Thesis, The Swiss Federal Institute ofTechnology, Zilrich

4. Brunelli R, Poggio T (1992) Face recognition through geometrical fea-tures. In: Sandini G (ed) ECCV'92, Santa Margherita Ligure. Springer,Berlin Heidelberg New York, pp 792-800

5. Brunelli R,.Poggio T (1993) Face recognition: features versus tem-plates. IEEE trans Pantt Anal Machine Intell 15:1042-1052

6. Cottrell G, Fleming M (1990) Face recognition using unsupervised fea-ture extraction Proceedings of the International Neural Network Con-ference, Paris, Kluwer, July, pp 322-335

7. Craw 1, Ellis H, Lishman JR (1987) Automatic extractoin of face fea-tures. Part Recogn Lett 5:183-187

8. Davis SB, Melmerstein P (1980) Comparison of parametric representa-tion for monosyllabic word recognition in continuosly spoken sentencesIEEE Trans Acoustic Speech Signal Processing -28:357-366

9. Doddington GR (1985). Speaker recognition, identifying people by theirvoices. Proc IEEE 73

10. Duda RO, Hart PE (1973) Pattern recognition and scene analysis. Wi-ley, New York

11. Furul S (1981) Cepstrum analysis technique for automatic speaker ver-ification. IEEE Trans Acoustic Speech Signal Processing 29:254-272

12. Kanade T (1973) Picture processing by comoputer complex and recog-nition of human faces. Technical Report, Department of InformationScience, Kyoto University, Kyoto, Japan

13. Makhoul J, Gish H, Roucos S (1985) Vector quantization in speechcoding. Proc IEEE 73:1551-1588

14. Nakamura O, Mathur S, Minami T (1991).Identification of human facesbased on isodensity maps. Part Recogn 24:263-272

15. Poggio T, Edelman S (1990).A network that learns to recognize three-dimensional objects. Nature 343:1-3

16. Poggio T, Girosi F (1989) A theory of networks for approximation.and learning. Artificial Intelligence Lab A.I. Memo No. 1140, Mas-sachusetts Institute of Technology, Boston, Mass

17. Poggio T, Girosi F (1990) Networks for approximation and learning.Proc IEEE 78:1481-1497

18. Poggio T, Stringa L (1992) A project for an intelligent system: visionand learning. nt J Quantum Chem 42:727-739

19. Rosenberg AE, Soong FK (1987) Evaluation of a vector quantiza-tion talker recognition system in text independent and text dependentmodes. Comput Speech Language 2:143-157

20. Stringa L (1991a) An integrated approach to artificial intelligence: theMAIA Project. bTchnical Report 9110-26, Institute for Scientific andTechnological Research, Trento, Italy

21. Stringa L (1991b) Automatic Face Recognition using DirectinalDerivatives. Technical Report 9205-04, I.R.S.T. Institute for Scientificand Technological Research. Trento, Italy

22. Stringa L (1992) S-net implementation of a face recognizer based ondirectional derivatives. In: Caianiello ER (ed) Proceedings of the 5thItalian Workshop on Neural Nets, Vietri, World Scientific, Singapore,pp 329-333.

23. Stringa L (1993) Eyes detection nfor face recognition. Appl Artif Intell7:365-382

24. Tishby NZ (1991) On the application of mixture AR hidden 'markovmodels to text independent speaker recognition. IEEE Trans SignalProcessing 39:563-570

25. Turk M, Pentland A (1991) Eigenfaces for recognition. J CognitiveNeurosci 3:71-86

325

Roberto Bnmelli was born in Trento, Italy, in 1961 ..Herceived his degreein Physics cum laude from the University of Trenrnto in 1986. He joinedthe Institute for Scientific and Technological Research (IRST) in 1987 asresearcher in computer vision. His interests include object-recognition andlearning.

Danlele Falavigna was born in Verona, Italy. in 1960. He received hisdegree in Electronic Engeneering from the University of Padua in 1985. In1988 he joined IRST, where he is currently engaged as Senior Researcherof.the Acoustic Processing and Speech Recognition Group. His currnenlresearch interests are signal processing and acoustic modeling for speechand speaker recognition.

Tomaso Poggio was born in Genoa. Italy, in 1947. He received his PhD inTheoretical Physics from the University of Genoa in 1970. From 1971 to1982, he was a Scientific Assistant at the Max Planck Institute for BiologicalKybernetics in Tuebingen, Germany. Since 1982, he has been a Professorat the Massachusetts Institute of Technology, affiliated with the Departmentof Brain and Cognitive Sciences and the Artificial Intelligence Laboratory.In 1988 he was named to the Uncas and Helen Whitaker Professorship,and in 1992 he became a founding co-director of the Center for Biologicaland Computational Learning. He is a Fellow of the American Associationof Artificial Intelligence, and a Corporate Fellow of Thinking MachinesCorporation.

Lulgi Stringa was born in Genoa, Italy, in 1939. He graduated in Physicsfrom the University of Genoa. His main interests are computer architectures;pattern recognition and artificial intelligence. He taught these subjects at theUniversities of Genoa, Rome, and Trento, Italy. He has designed and de-veloped innovative industrial systems, such as multiprocessor-based MailAddress Readers. Joining SELENIA in ELSAG in 1969, he became ViceDirector General in 1979. In 1980 he became Director General of SELE-NIA, and he was appointed Managing Director of the SELENIA-ELSAGGroup in 1983. He left SELENIA in 1985 to become Director of the lnsti-tute for Scientific and Technological Research (IRST) in Trento.

Application No: 11/231,353IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 20, NO. 12, DECEMBER 1998. Docket No. 577832000200

Integrating Faces and Fingerprintsfor Personal Identification

Lin Hong and Anil Jain, Fellow, IEEE

Abstract-An automatic personal identification system based solely on fingerprints or faces is often not able to meet the systemperformance requirements. Face recognition is fast but not extremely reliable, while fingerprint verification is reliable but inefficient indatabase retrieval. We have developed a prototype biometric system which integrates faces and fingerprints. The systemovercomes the limitations of face recognition systems as well as fingerprint verification systems. The integrated prototype systemoperates in the identification mode with an admissible response time. The identity established by the.system is more reliable thanthe Identity established by a face recognition system. In addition, the proposed decision fusion scheme enables performanceimprovement by integrating multiple cues with different confidence measures. Experimental results demonstrate that our systemperforms very well. It meets the response time as well as the accuracy requirements.

Index Terms-Biometrics, fingerprint matching, minutiae, face recognition, eigenface, decision fusion.

-- -+----------

1 INTRODUCTION

WITH the evolution of information technology, our so-ciety is becoming more and more electronically con-

nected. Daily transactions between individuals or betweenindividuals and various organizations are conducted in-creasingly through highly interconnected electronic de-vices. The capability of automatically establishing theidentity of individuals is thus essential to the reliability ofthese transactions. Traditional personal identification ap-proaches which use "something that you know," such as aPersonal Identification Number (PIN). or "something thatyou have," such as an ID card are not sufficiently reliableto satisfy the security requirements of electronic transac-tions because they lack the capability to differentiate be-tween a genuine individual and an impostor whofraudulently acquires the access privilege 117]. Biometrics,which refers to identification of an individual based onher physiological or behavioral characteristics, relies on"something which you are or you do" to "ake a personalidentification and, therefore, inherently has the capabilityto differentiate between a genuine individual and afraudulent impostor [17], 1271.

Any human physiological or behavioral characteristiccan be used as a biometric characteristic (indicator) to make apersonal identification as long as it satisfies the followingrequirements [6], [17]:

1) universality, which means that each person shouldhave the characteristic;

2) uniqueness, which indicates that no two personsshould be the.same in terms of the characteristic;

SThe authors are with the Department of Computer Science and Engineer-ing, Michigan State University. East Lansing. MI 48824-1226.E-mail: (honglin, [email protected].

Manuscript received 6 Oct. 1 997: revised 8 Sept. 1998. Recommended for accep-tance by V. Nalwa.For Information on obtaining reprints of this article, please send e-mail to:[email protected], and reference IEEECS Log Number 1 07416.

3) permanence, which means that theshould not be changeable; and

4) collectability, which indicates that thecan be measured quantitatively.

characteristic,

characteristic

However, in practice, a biometric characteristic that satisfiesall the above requirements may not always be feasible for apractical biometric system. In a practical biometric system,there are a number of other issues which should be consid-ered, including [6), 1171:

1) performance, which refers to the achievable identifica-tion accuracy, speed, robustness, the resource re-quirements to achieve the desired identification accu-racy and speed, as well as operational or environ-mental factors that affect the identification accuracyand speed;

2) acceptability, which indicates the extent to which peo-ple are willing to accept a particular biometrics intheir daily life: and

3) circumvention, which reflects how easy it is to fool thesystem by fraudulent methods.

A practical biometric system should be able to:

1) achieve an' acceptable identification accuracy, andspeed with a reasonable resource requirements; .

.2) not be harmful to the subjects and be accepted by theintended population; and

3) be sufficiently robust to various fraudulent methods.

Currently, there are mainly nine different biometric tech-niques that are either widely used or under investigation,including [271:

* face,* facial thermogram," fingerprint,* hand geometry,* hand vein,* iris,

0162-8828/98/510.000 1998 IEEE

1296 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 20, NO. 12, DECEMBER 1998

Fig. 1. Examples of biometric characteristics (indicators). (a) Face.. (b) Facial thermogram. (c) Fingerprint. (d) Hand vein. (e) Retinal scan.

Enrollment

IIK'

Fig. 2. A generic biometric system architecture.

* retinal pattern,* signature, and* voice-print

(some of the examples are shown in Fig. 1) 161, 171, 181, (161,(171. All these biometric techniques have their own advan-tages and disadvantages and are admissible depending onthe application domain.

A generic biometric system architecture is depicted inFig. 2. Logically, it can be divided into two modules:

1) enrollment module and2) identification module.

During the enrollment phase, the biometric'characteristic ofan individual is first scanned by the biometric reader toproduce a digital representation of the characteristic. Inorder to facilitate the matching and identification, the digi-tal representation is usually further processed by a featureextractor to generate a compact but expressive representa-tion, called a template. Depending on the application, thetemplate may be stored in the central database of thebiometric system or be recorded in the smart card or mag-netic card issued to the individual. The identification mod-ule is responsible for identifying individuals at the point-of-access. In the operational phase, the biometric reader cap-tures the characteristic of the individual to be identified and,converts it to a raw digital format, which is further proc-essed by the feature extractor to produce a compact repre-

sentation that is of the same format as the template. Theresulting representation is fed to the feature matcher whichcompares it against the template(s) to' establish the identity.

1.1 Operational ModeA biometric system may operate in

1) the verification mode or2) the identification mode 117].

A biometric system operating in the verification modeauthenticates an individual's identity by comparing the in-dividual only with his/her own template(s) (Am I whom Iclaim I am?). It conducts one-to-one comparison'to deter-mine whether the identity claimed by the individual is trueor not. A biometric system operating in the identificationmode recognizes an individual by searching the entire tem-plate database for a match (Who am I?). It conducts one-to-many comparisons to establish the identity of the individual.

Generally, it is more difficult to design an identificationsystem than to design a verification system (171. For a veri-fication system, the major challenge is the system accuracy.It is usually not very difficult to meet the response timerequirement, because only one-to-one comparison is con-ducted: On the other -hand, for an identification system,both the accuracy and speed are critical. An identificationsystem needs to explore the entire template database toestablish an identity. Thus, more requirements are irriposed'on the feature extractor and, especially, the feature matcher.

(b) (c) (d) (e)

'

t .

1

1

I

HONG AND JAIN: INTEGRATING FACES AND FINGERPRINTS FOR PERSONAL IDENTIFICATION

Some biometric approaches are more suitable for operatingin the identification mode than the others. For example,although significant progress has been made in fingerprintidentification and a number of fingerprint classification andmatching techniques have been proposed, it is still notpractical to conduct a real-time search even on a relativelysmall-size fingerprint database (several thousand images)without dedicated hardware matchers, external alignment,and multiple-fingerprint indexing mechanism [21]. On theother hand, it is feasible to design a face-recognition systemoperating in the identification mode, because

1) face comparison is a relatively less expensive opera-tion and

2) efficient indexing techniques are available and theperformance is admissible [23].

1.2 Identification AccuracyDue to intraclass variations in the blometric characteristics,the identity can be established only with certain confidence.A decision made by a biometric system is either a "genuineindividual" type of decision or an "impostor" type of deci-sion 171. 1171. For each type of decision, there are two possi-ble outcomes, true or false. Therefore, there are a total of fourpossible outcomes:

1) a genuine individual is accepted,2) a genuine individual is rejected,3) an impostor is rejected, and4) an impostor is accepted.

Outcomes I and 3 are correct, whereas outcomes 2 and 4 areincorrect. The confidence associated with different deci-sions may be characterized by the genuine distribution andthe impostor distribution, which are used to establish twoerror rates:

1) false acceptance rate (FAR), which is defined as theprobability of an impostor being accepted as a genu-ine individual and

2) false reject rate (FRR), which is defined as the prob-ability of a genuine individual being rejected as animpostor.

FAR and FRR are dual of each other. A small FRR usuallyleads to a larger FAR, while a smaller FAR usually implies alarger FRR. Generally, the system performance requirementis specified in terms of FAR 1171. A FAR of zero means thatno impostor is accepted as a genuine.individual.

In order to build a biometric system that is able to op-erate efficiently in identification mode and achieve desir-able accuracy, an integration scheme which combines twoor more different biometric approaches may be necessary.For example, a biometric approach that is suitable for op-erating in the identification mode may be used to indexthe template database and a biometric approach that 'isreliable in deterring impostors may be used to ensure theaccuracy. Each biometric approach provides a certain con-fidence about the identity being established. A decisionfusion scheme which exploits all the Information at the'output of each approach can be used to-make a more reli-able decision.

We introduce a prototype integrated biometric systemwhich makes personal identification by integrating bothfaces and fingerprints. The prototype integrated biometricsystem shown in Fig. 3 operates in the identification mode.The proposed system integrates two different biometricapproaches (face recognition and fingerprint verification)and incorporates a decision fusion module toimprove theidentification performance.

.In the following sections, we will describe each compo-nent of the proposed integrated system. Section 2 addressesthe face-recognition technique being employed. Section 3presents the fingerprint-verification module along with mi-nutiae extraction and minutiae matching. A decision fusionframework which integrates faces and fingerprints is formu-lated in Section 4. Experimental results on the MSU finger-print database captured with an online fingerprint scannerand public-domain face databases are described in Section 5.Finally, the summary and conclusions are given in Section 6.

2 FACE RECOGNITION

Face recognition is an active area of research with applica-tions ranging from static, controlled mug-shot verificationto dynamic, uncontrolled face identification in. a cluttered.background [5]. In the context of personal identification,face recognition usually refers to static, controlled full-frontal portrait recognition [5]. By static, we mean that thefacial portraits used by the face-recognition system are stillfacial images (intensity or range). By controlled, we meanthat the type of background, illumination, resolution of theacquisition devices, and the distance between the acquisi-tion devices and faces, etc. are essentially fixed during theimage acquisition process. Obviously, in such a controlledsituation, the segmentation task is relatively simple(and theintraclass variations are small.

During the past 25 years, a substantial amount of researcheffort has been devoted to face recognition 15]. 125]. (1). In theearly 1970s, face recognition was mainly based on measuredfacial attributes, such as eyes, eyebrows, nose, lips, chinshape, etc. 15]. Due to lack of computational resources 'andbrittleness of feature extraction algorithms, only a very lim-ited number of tests were conducted and the recognitionperformance of face-recognition systems was far from desir-able. [5]. After the dormant 1980s, there was a resurgence inface-recognition research in the early 1990s. In addition to con-tinuing efforts on attribute-based techniques [5],. a number ofnew face-recognition techniques,were proposed, including:

* principle component analysis (PCA) [22], 1121, [241,* linear discriminant analysis (LDA) 123],* singular value decomposition (SVD) 1101, and* a variety of neural network-based techniques 1251.

The performance of these approaches is impressive. It wasconcluded that "face-recognition algorithms were devel-.oped and were sufficiently mature that they can be portedto real-time .experimental/demonstration system" [1191.

Generally, there are two major tasks in face recognition:

1). locating faces in input images and2) recognizing the located faces.

1297

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 20, NO. 12, DECEMBER 1998

Fig. 3. System architecture of the prototype integrated biometric identification system.

Face location itself continues to be a challenging problemfor uncontrolled and cluttered images [5]. Fortunately, inthe context of personal identification, the background iscontrolled or almost controlled, so face location is generallynot considered to be a difficult problem. Face recognitionfrom a general viewpoint also remains an open problembecause transformations such as position, orientation, andscale'and changes in illumination produce large intraclassvariations 1191. Again, in the context of personal identifica-tion, the variations in acquired face images can be restrictedto a certain range which enables the current techniques toachieve a desirable performance 151. 1191.

In our system. the eigenface approach is used for thefollowing reasons:

1) in the context of personal identification, the back-ground, transformations, and illumination can becontrolled,

2) eigenface approach has a compact representation-afacial image can be concisely represented by a featurevector with a few elements,

3) it is feasible to index an eigenface-based template da-tabase using different indexing techniques such thatthe retrieval can be conducted efficiently 123),

4) the eigenface approach is a generalized templatematching approach which was demonstrated to bemore accurate than the attribute-based approach inone study (4).

The eigenface-based face recognition consists of the fol-lowing two stages [241:

1) training stage, in which a set of N training face imagesare collected; eigenfaces that correspond to the Mhighest eigenvalues are computed from the trainingset; and each face is represented as a point in the M-dimensional eigenspace, and

2) operational stage, in which each test image is first pro-jected onto the M-dimensional eigenspace; the M-dimensional face representation is then deemed as afeature vector and fed to a classifier to establish theidentity of the individual.

A Wx H face image I(x, y) can be represented as a Wx H-dimensional feature vector by concatenating the rows of I(x, y)together. Thus, each W x H face image becomes a point inthe W x H-dimensional space. The total number of pixels ina face image is typically large, on the order of several thou-sands for even small image sizes. Face images in such ahigh-dimensional space are not randomly distributed.Therefore, it is efficient and beneficial to project them to alower-dimensional subspace using principle componentanalysis 124]. Let 4', W,;., N denote the. N W x H-dimensional training vectors with zero-mean. Let the Mbasis vectors, u,, u, ...., u be a set of orthonormal vectorsthat best describe, the distribution of face images in the M-dimensional subspace (elgenspace), M 5 N. The kth eigen-vector, u, A = 1, 2; .... M, is computed such that 124]

Ak l N T(k4 )2(

i=1

is maximum, subject to

1298


-I

r '" ' '

1. -..-a t 4

17J ' -;,,

Fig. 4. First 10 eigenfaces obtained from 542 images of size 92 x 112, which are listed from left to right and top to bottom in a decreasing order ofthe corresponding eigenvalues.

r u J if i = jusu = 0 otherwise (2)

The value A, is the kth largest eigenvalue of the covariancematrix E which can be estimated using the training samplesby

- 1 " 7= N 1 ; .(3)

1=l1

The vector u, is the kth eigenvector of the covariance matrixE corresponding to A~,.

With the M-dimensional eigenspace defined, trainingvectors, T,, .'", N, can be represented as a set of M-dimensional feature vectors, (1,, (12, N:

k = uT I, i= 1, 2, ... , N, (4)

where u = (u,, u, .... , u,). Fig. 4 shows the first 10 eigenfacescorresponding to the 10 largest eigenvalues, which werecomputed based on 542 training images (of size 92 x 112).

In the operational phase, a detected face image, F, whichis normalized to zero mean, is vectorized and projectedonto the eigenvectors according to 11 = u 'F. With bothtraining samples and test samples being projected onto anM-dimensional eigenspace, face recognition can be accom-plished by a classifier operating in the eigenspace. In thecontext of personal identification, only a very limited num-ber of.training samples are available for each individual(17]. Thus, a k-nearest neighbor classifier is typically used,in which the distance, d, called Distance From FeatureSpace (DFFS) 1241 between a template, 0, and a test pattern,fl, is defined as 1' - nil, where 11 11 denotes L norm.

3 FINGERPRINT VERIFICATION'

A fingerprint is the pattern of ridges and furrows on thesurface of a fingertip. It is formed by the accumulation ofdead, cornified cells that constantly slough as scales fromthe exposed surface (14). Its formation Is determined in the

fetal period [15]. Humans have used fingerprints for per-sonal identification for a long time. The biological proper-ties of fingerprints are well understood which are summa-rized as follows:

1) individual epidermal ridges and furrows have differ-ent characteristics for different fingerprints;

2) the configuration types are individually variable, butthey vary within limits which allow for systematicclassification;

3) the configurations and minute details of individualridges and furrows are permanent and do not changewith time except by. routine injury, scratches, andscarring, as may be seen in Fig. 5 and Fig. 9 [15].

The uniqueness of a fingerprint is exclusively deter-mined by the local ridge characteristics and their relation-ships. Fingerprint matching generally, depends on thecomparison of local ridge characteristics and their rela-tionships [14], [11], (17]. A total of 150 different local ridgecharacteristics, called minute details, have been identified[14]. These local ridge characteristics are not evenly dis-tributed. Most of them depend heavily on the impressionconditions and quality of fingerprints and are rarely ob-served in fingerprints. The two most prominent ridgecharacteristics, called minutiae, are ridge ending and ridgebifurcation. A ridge ending is defined as the point where aridge ends abruptly. A ridge bifurcation is defined as thepoint where a ridge forks or diverges into branch ridges.A fingerprint typically contains about 40 to 100 minutiae.Examples of minutiae are shown in Fig. 5c. For a givenfingerprint, a minutia can be characterized by its type, itsx and y coordinates, and its direction, 6, whose definitionsare also shown in Fig. 5c.

Fingerprint verification consists of two main stages ill ),11,4]:

1) minutiae extraction and2) minutiae matching.

~LU~ i~J

1299


Ridn Ening Ride Bi) ungi

Fig. 5. Fingerprints and minutiae. (a) and (b) Two different impressions of the same finger. (c) Ridge ending and ridge bifurcation.

(a) (b)

(c)-~--~

(d)

Fig. 6. Results of our minutiae extraction algorithm on a fingerprint image (512 x 512) captured with an optical scanner. (a) Input image. (b) Ori-entation field. (c) Ridge map. (d) Extracted minutiae:

Due to a number of factors such as aberrant formations ofepidermal ridges of fingerprints, postnatal marks, occupa-tional marks, problems with acquisition devices, etc., ac-quired fingerprint images may not always have well-defined ridge structures. Thus, a reliable minutiae extrac-tion algorithm should not assume perfect ridge structuresand should degrade gracefully with the quality of finger-print images. We have developed a minutiae extraction al-gorithm [1ll) based on the algorithm proposed in [20). Itmainly consists of three steps:

1) orientation field (ridge flow) estimation, in which the orn-entation field of input fingerprint images is estimatedand the region of interest is located,

2) ridge extraction, in which ridges are extracted andthinned, and

3) minutiae detection and postprocessing, in which minutiaeare extracted from the thinned ridge maps and refined.

For each detected minutia, the following parameters arerecorded:

* x-coordinate,* y-coordinate,* orientation, which is defined as the local ridge orien-

tation of the associated ridge, and* the associated ridge.

The recorded ridges which are used for alignment in theminutiae matching are represented as one-dimensional dis-crete signals which are normalized by the average inter-ridge distance. In an automatic fingerprint identification,ridge endings and ridge bifurcations are usually not dif-ferentiated from one another. Therefore, no minutiae type

information is recorded. A minutia is completely deter-mined by its position and orientation. Fig. 6 shows the re-sults of our minutiae extraction algorithm on a fingerprintimage captured with an optical scanner.

The minutiae matching determines whether, two minu-tiae patterns are from the same finger or not. A similaritymetric between two minutiae patterns is defined and athresholding on the similarity value is performed. By rep-resenting minutiae patterns as two-dimensional "elastic"point patterns, the minutiae matching may be accom-plished by an "elastic" point pattern matching as long as itcan automatically establish minutiae correspondences (inthe presence of translation, rotation, and deformations)and detect spurious minutiae and missing minutiae. We;have developed an alignment-based "elastic" matchingalgorithm 11l, which is capable of finding the correspon-dences between minutiae without resorting to an exhaus-tive search and has the ability to adaptively compensatefor the nonlinear deformations and inexact transforma-tions between different fingerprints. The alignment-basedmatching algorithm decomposes the minutiae matchinginto two stages:

1) Alignment stage, where transformations such as trans-lation, rotation, and scaling between an input and atemplate in the database are estimated, and the inputminutiae are aligned with the template minutiae ac-cording to the estimated parameters; and

2) Matching stage, where both the input minutiae and thetemplate minutiae are converted to "strings" in the polarcoordinate system, and an "elastic"" string matching al-gorithm is used to match the resulting strings, and

J

1300


finally, the normalized number of. corresponding mi-nutiae pairs is reported.

Let

P= (x,".y ,opT .... (x .yP9e1

and

Q = x I y e Q , .... , y ,Q((XQ 'y QQ) ( 9 y 9))

denote the p minutiae in the template and the q minutiae inthe input image, respectively. The alignment-based match-ing algorithm is depicted as follows:

I) Estimate the translation and rotation parameters be-tween the ridge associated with each input minutiaand the ridge associated with each template minutiaand align the two minutiae patterns according to theestimated parameters.

2) Convert the template pattern and input pattern intothe polar coordinate representations with respect tothe corresponding minutiae on which alignment isachieved and represent them as two symbolic stringsby concatenating each minutia in an.increasing orderof radial angles:

P = ((P PP)T ... r e ,0 (5)

Q=((i9.e TO?) ( Q) T). (6)Q = , e , ..... r e O

where r , eP , and 0 , represent the correspondingradius, radial. angle, and normalized minutiae orien-tation with respect to the reference minutiae, (x, y,. 0),respectively.

3) Match the resulting strings P and Q with a modifieddynamic-programming algorithm to find the "editdistance" between P and Q.

4) Use the minimum edit distance between P and Q toestablish the correspondence of the minutiae betweenP and Q. The matching score, S, is then defined as:

100M~qS= _1M (7)Pq

where MQ is the number of 'minutiae which fall in thebounding boxes of template minutiae. The boundingbox of a minutia specifies the allowable tolerance inthe positions of the corresponding input minutiaewith respect to the template minutiae. Fig. 7 shows anexample of minutiae matching.

4 DECISION FUSION

Decision fusion which integrates multiple cues has provedbeneficial for improving the accuracy of a recognition .sys-tem'12), 13), 1131. Generally, multiple cues may be integratedat one of the following three different levels 131:

1) Abstract level; the output from each module is only aset of possible labels without any confidence associ-ated with the labels: in this case, the simple majority

Fig. 7. Fingerprint matching.

Srule may be employed to reach a more reliable deci-sion [26];

2) Rank level; the output from each module is a set ofpossible labels ranked by decreasing confidence val-ues, but the confidence values themselves are notspecified;

3) Measurement level; the output from each module is aset of possible labels with associated confidence val-ues; in this case, more accurate decisions can.be madeby integrating different confidence measures to .amore informative confidence measure'.

In our system, the decision fusion Is designed to operate atthe measurement level. Each of the top.n possible identitiesestablished by the face recognition module is verified bythe fingerprint verification module. In order to carry outsuch a decision fusion scheme, we need to define a measurethat indicates the confidence of the decision criterion and adecision fusion criterion.

As discussed in Section 1, the confidence of a given deci-sion criterion may be characterized by its FAR (false accep-tance rate). In order to estimate FAR, the impostor distribu-tion needs to be computed. How should we compute theimpostor distribution? In practice, It can only be estimatedfrom empirical data. But, this estimation problem requiressome care. In the context of personal identification, the re-quired FAR value is often a very small number (< 1 per-cent) [171. If the parametric form of the underlying impos-tor distribution is not known, nonparametric. techniquesneed to be used. In order to guarantee that the estimatedimpostor distribution is reliable for characterizing the smallFARs. a large representative test set that satisfies the fol-lowing two requirements is needed: It should be largeenough to represent the population, and it should containenough samples from each category of the population. Theabove requirements are not easily satisfied in practice. Anextrapolation based on the knowledge of the parametricform of the underlying impostor distribution is needed.

4.1 Impostor Distribution for Fingerprint VerificationA model that can precisely characterize the impostor distri-bution of a minutia matching algorithm is not easy, since:

1) the minutiae in a fingerprintare distributed randomlyin the region of interest;

1301

'IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 20, NO. 12, DECEMBER 1998

2) the region of interest of each input fingerprint may bedifferent;

3) each input fingerprint tends to have a different num-ber of minutiae;.

4) there may be a significant number of spurious minu-tiae and missing minutiae;

5) sensing, sampling, and feature extraction may resultin errors in minutiae positions; and

6) sensed fingerprints may have different distortions.

However, it is possible to obtain a general model of theoverall impostor distribution by making some-simplifyingassumptions.

Let us assume that the input fingerprint and the tem-plate have already been registered and the region of interestof both the input fingerprint and the template is of the samesize, a W x W (for example. 500 x 500) region. The W x Wregion is tessellated into small cells of size w x w which areassumed to be sufficiently large (for example, 40 x 40) suchthat possible deformation and transformation errors arewithin the bound specified by the cell size, Therefore, thereare a total of w x W (= N,) different cells in the region of

interest of a fingerprint. Further, assume that each finger-print has the same number of minutiae, Nm (< N), which aredistributed randomly in different cells and each cell con-tains at most one minutiae. Each minutia is directed to-wards one of the D (for example, eight) possible orienta-tions with equal probability. Thus, for a given cell, theprobability, Pm,,, that the cell is empty with no minutiae

present is N and.the probability, P, that the cell has a mi-

nutia that is directed toward a specific orientation is

A pair of corresponding minutiae between a template andan input is considered to be identical if and only if they arein the cells at the same position and directed in the samedirection (see Fig. 8). With the above simplifying assump-

* tions, the number of corresponding minutiae pairs betweenany two randomly selected minutiae patterns Is a randomvariable, Y, which has a binomial distribution with pa-rameters N, and P [18]:

NIg(Y = Y) = m PY(1 - P)(N.-Y) (8)

y!(Nm - y)I

The probability that the number of corresponding minutiaepairs between any two sets of minutiae patterns is less thana given threshold value, y, is

y-I

G(y) = g(Y < y) = g(k). (9)k=0

The decision made by the proposed minutiae matchingalgorithm for an input fingerprint and a template is, basedon the comparison of the "normalized" number of corre-sponding minutiae pairs against a threshold. Therefore,under the assumption that minutiae in the region of interestof fingerprints of different individuals are randomly dis-tributed, the probability that an impostor. I. is accepted is{(1 - G(y)}, where y, is the number of corresponding minu-tiae pairs between the impostor and the individual whomthe impostor claims to be.

- - ~ -

- -

-- # I .,.- I /

Template Minutiae Set Input Minutiae Set

Fig. 8. Minutiae matching model, whereand a dashed line indicates a mismatch.

a solid line indicates a match

4.2 Impostor Distribution for Face RecognitionThe characterization of impostor distribution for face rec-ognition is more difficult. Due to the relatively low dis-crimination capability of face recognition, this moduleneeds to keep the top n matches to improve the likelihoodthat the genuine individual will be identified if he or she isin the database.

Let 0,, 2, .... (N be the N face templates stored in the

database. The top n matches, (D;, ( ...., , are obtainedby searching through the entire database, in which N com-parisons are conducted explicitly (in the linear search case)or implicitly (in organized search cases such as the treesearch). The top n matches are arranged in the increasingorder of DFFS (Distance From Feature Space, Section 2)values. The smaller the DFFS value, the more likely it is thatthe match is correct. Since the relative distances betweenconsecutive DFFSs are invariant to the mean shift of theDFFSs, it is beneficial to use relative instead of absoluteDFFS values. The probability that a retrieved top n match isincorrect is different for different ranks. The impostor dis-tribution should be a decreasing function of rank order andit is a function of both the relative DFFS values, A, and therank order, i: '

F()P (). (10)

where F(A) represents the probability that the consecutiveDFFS values between impostors and their claimed indi-viduals at rank i are larger than a value A, and Ps,(i) repre-sents the probability that the retrieved match at rank i is animpostor. In practice, F,(A) and P,() need to be estimatedfrom empirical data.

In order to simplify the analysis, we assume that eachindividual has only one face template in the database. Thus,there are a total of N individuals enrolled in the databaseand I,, I2, .... IN are used as identity indicators, Let X' denotethe DFFS between an individual and his/her own templatewhich is a random variable with density function f(X) and

let X, X, ..... X_,1 denote the DFFS values between anindividual and the templates of the other individuals in thedatabase, which are random variables with density functions.

1302


(x) I(x), .... f_ X ) .respectively Assume that

X" and X2. X2 ...... X_, are statistically independent and

X = (x) = ... _,XX= fX). For an individ-

ual, I. which has a template stored in the database, {@,, O,... ,, the rank, 1, of X" among XZ , XP , .... XNI is aran-

dom variable with probability

S(N- 1)! N--i)P(1 = - i (1-i-0 '1-p( -- ) 1)(=)=i!(N-1-i)Ip'(1-p)(l)

where

P f X )dX df . (12)

When p < 1 and N is sufficiently large, P(1) may be ap-proximated by a Poisson distribution [18),

e(- )a -P(1) T ' (13)

where a np. Obviously, P(1) is exactly the probability thatmatches at rank i are genuine individuals. Therefore,

P,,(i) = 1 - P(l= i). (14)

Although the assumption that X2,X2, .... X,_, are i.i.d.may not be true in practice, It is still reasonable to use theabove parametric form to estimate the probability that.re-trieved matches at rank I are impostors. Our experimentalresults support this claim.

Without any loss of generality, we assume that, for agiven individual, fl, X, XZ, .... , XN_ are arranged in in-creasing order of values. Define the non-negative distancebetween the (i+1)th and ith DFFS values as the ith DFFSdistance,

Ai = X - X. l 15 N- 1. (15)

The distribution, f(A). of the ith distance, A,, is obtainedfrom the joint distribution w,(X, A) of the ith value, X, andthe ith distance. , .,

f(, 1) = w(X, 6.)dX, (16)

w,(Xf, A) = CF(4 'I1 - P(X°+ )] ]'f(x)f (X'+ 6). (17)

(N- 1)!C = (18)(i-1)!(N -2-i)!' (18)

where F(Xp) = f f''(XP)dX' [91. With the distribution,

f,(A), of the ith distance defined, the probability that theDFFS of the impostor at rank i is larger than a thresholdvalue, A, is

(A) = f(A,)dA,. (19)

The above equations do not make any assumptions

about the distributions of X .X , ..... X_, as long as theyare i.i.d. The equations also hold even if the mean values of

X2, X , .... XNI shift. Therefore, it can tolerate, to a cer-tain extent, DFFS variations which is a desirable property.

In our system, we assume that XXZ .. X_,. are dis-tributed with a Gaussian distribution with unknown meanand variance.

4.3 Decision FusionThe impostor distribution for face recognition and the im-postor distribution for fingerprint verification provide con-fidence measures for each of the top n matches retrieved byface recognition module. Without a loss of generality, weassume that at most one of the n possible identities estab-lished by the facerecognition module for a given individualis the genuine identity of the individual. The final decisionby Integration either rejects all the n possibilities or acceptsonly one of them as thegenuine identity. In practice, it isusually specified that the FAR of the system should be lessthan a given value [171. Therefore, the goal of decision fu-sion, in essence, is to derive a decision criterion which satis-fies the FAR specification.

It is reasonable to assume that the DFFS between twodifferent individuals is statistically independent of the fin-gerprint matching score between them; facial similaritybetween two individuals does not imply that they havesimilar fingerprints, and vice versa. This assumptionshould not be confused with the situation where an im-postor tries to fool the system by counterfeiting the faceand/or fingerprints of the genuine individual. LetF(A)Pd,(i) and C(Y) denote the impostor distribution atrank i for face-recognition and fingerprint-verification mod-ules, respectively. T'he composite impostor distribution atrank i may be defined as

H,(A, Y) = F,(A)P.,,(i)C(Y). (20)

Let I,, I .... I,, denote the n possible identities'establishedby face recognition, {X,,.X 2 . . . , X,) denote the correspondingn DFFSs, (Y,, YZ, ...; Y) denote the corresponding n finger-print matching scores, and FAR o denote the specified valueof FAR. The final decision, ID(fl), for a given individual flis determined by the following criterion:

ID(nfl) =

JI iHk (Ak , Yk) < FAR 0, and

. Hk(Ak' k) = min{H,(A, Y),.. . H e(A., YA4imposter otherwise

(21)

where A, = X,,, - X,. Since H,(A, Y) defines the probabilitythat an impostor is accepted at rank i with consecutiverelative DFFS. A. and fingerprint matching score, Y, theabove decision criterion satisfies the FAR specification.

Note that the decision criterion in (21) depends on thenumber of individuals, N, enrolled in the database, since F,depends on N. However, it does not mean that F has to berecomputed whenever a new individual is enrolled in thedatabase. In fact, if N > 1, the corresponding Fs for differ-ent values of N are quite similar to one another. On theother hand, the decision criterion still satisfies the FARspecification when Nincreases, though it may not be able totake full advantage of the information contained in the Ncomparisons. In practice, an update scheme which re-computes the decision criterion whenever the number of

1303


.qr

.(b) (c)

;'(

(e) (f)

Fig. 9. Face and fingerprint pairs; the face images (92 x 112) are from the Olivetti Research Lab; the fingerprint images (640 x 480) are capturedwith a scanner manufactured by Digital Biometrics.

added individuals is larger than a prespecified value can beused to guarantee that the decision criterion exploits all theavailable information.

5 'EXPERIMENTAL RESULTS

The integrated biometric system was tested on the MSUfingerprint database and a public domain face database.The MSU fingerprint database contains a total of 1,500 fin-gerprint images (640 x 480) from 150 individuals with 10images per individual, which were captured with an opticalscanner manufactured by Digital Biometrics. When thesefingerprint images were captured. no restrictions on theposition, orientation, and impression pressure were im-posed. The fingerprint images vary in quality. Approxi-mately 90 percent of the fingerprint images in the MSU da-tabase are of reasonable quality similar to the imagesshown in Fig. 9b and Fig. 9d. Images of poor quality withexamples shown in Fig. 9f and Fig. 9h are mainly due tolarge creases and smudges in ridges, dryness of the im-pressed finger, and high impression pressure. The face da-tabase contains a total of 1,132 images of 86 individuals; 400images of 40 individuals with 10 images per individual arefrom the Olivetti Research Lab. 300 images of 30 individu-als with 10 images per individual are from the University ofBern, and 432 images of 16 individuals with 27 images perindividual are from the MIT Media Lab. The images wereresampled from the original sizes to a fixed size of 92 x 112and normalized to zero mean.

We randomly selected 640 fingerprints of 64 individuals,as the training set and the remaining as the test set. Themean and standard deviation of the impostor distribution(Fig. 10a) were estimated to be 0.70 and 0.64 from the403,200 (640 x 630) impostor matching scores of "all against

all" verification test by fitting the probability model de-scribed in Section 4.1, respectively. A total of 542 face im-ages were usedas training samples. Since variations in po-sition, orientation, scale, and illumination exist' in the facedatabase, the 542 training samples were selected such thatthe representative views are included. Eigenfaces were esti-mated from the 542 training samples and the first 64 eigen-faces wereused for face recognition. The top n = 5 impostordistributions were approximated. Generally, the larger, thevalue of n, the lower the false reject rate of face recognition.However, as n increases, more candidates need to be verifiedby fingerprint verification. There is obviously a trade-offbetween the accuracy and speed of a biometric system. Fig. 10bshows the impostor distribution at rank no. 1.

We randomly assigned each of the remaining 86 indi-viduals in the MSU fingerprint database to an individual inthe face database (see Fig. 9 for some examples). Since the.DFFS value between two different individuals is statisti-cally independent of the fingerprint matching scores be-tween the two individuals, such a random assignment of aface to a fingerprint is admissible. One. fingerprint for eachindividual is randomly selected as the template for the in-dividual. To simulate the practical identification scenario,each of the remaining 590 faces was paired with a finger-print to produce a test pair, In the test,' with a prespecifiedconfidence value (FAR), for each of the 590 fingerprint andface pairs, the top five matches are retrieved using face rec-ognition. Then fingerprint verification is applied to each ofthe top five matches, and a final decision is made by-deci-sion fusion.

The prespecified FAR for a biometric system is usuallyvery small (< 0.0001). In order to demonstrate that the,.biometric system does meet such a specification, a largenumber of representative samples are needed. Unfortunately,

(a)

. . ; ' . ,

HONG AND JAIN: INTEGRATING FACES AND FINGERPRINTS FOR PERSONAL IDENTIFICATION'

nmatching cor lirst DFFS distanoe

Fig. 10. Impostor distributions. (a) Impostor distribution for fingerprint verification; the mean and standard deviation of the impostor distribution areestimated to be 0.70 and 0.64, respectively. (b) The impostor distribution for face recognition at rank No. 1, where the stars (') represent empiricaldata and the solid curve represents the fitted distribution; the mean square error between the empirical distribution and the fitted distribution is 0.0014.

obtaining such a large number of test samples is both ex-pensive and time consuming.: In our test, we reuse faces bydifferent assignment practices-each time, a different fin-gerprint Is assigned to a given face to form a face and fin-gerprint probe pair. Obviously, such a reuse scheme mightresult in unjustified performance improvement. In order todiminish the possible gain in performance due to such areuse scheme, we multiplied the estimated impostor distri-bution for face recognition by a constant of 1.25, which es-sentially reduces contribution of face recognition to the fi-nal decision by a factor of 1.25. On the other hand, finger-print verification operates in the one-to-one verificationmode, so different assignments may be deemed as differentimpostor forgeries. Therefore, the test results using such arandom assignment scheme are able to reasonably estimatethe underlying performance numbers. In our test, 1,000 dif-ferent assignments were tried. A total of 590,000 (590 x 1,000)face and fingerprint test pairs were generated and tested.The FRRs of our system with respect to different prespeci-fied FARs, as well as the FRRs obtained by "all-to-all" verifi-cations using only fingerprints (2,235.000 = 1,500 x 1,490tests) or faces (342,750 = 350 x (590 - 5) + 240 x (590 - 15)tests) are listed in Table 1. Note that the FRRs in integrationcolumn include the error rate (1.8 percent) of genuine indi-viduals not present in the top five matches. The receiveroperating curves are plotted in Fig. 11, in which theauthentic acceptance rate (the percentage of genuine indi-viduals being accepted, i.e., 1 - FRR) is plotted against FAR..We can conclude from these test results that the integrationof fingerprints and faces does result in a significantly betterrecognition performance.

In order for an automatic personal identification system tobe acceptable in practice, the response time of the systemneeds to be within a few seconds. Table 2 shows that our im-plemented system does meet the response time requirement.

6 SUMMARY AND CONCLUSIONS

We have developed a prototype biometric system whichintegrates faces and fingerprints in authenticating a per-sonal identification. The proposed system overcomes thelimitations of both face-recognition systems and finger-print-verification systems. The integrated system operatesin the identification mode. The decision-fusion schemeformulated in the system enables performance improve-ment by integrating multiple cues with different confi-dence measures. Experimental result's demonstrate thatour system performs very well. It meets the response timeas well as the accuracy requirements.

TABLE 1FALSE REJECT RATES (FRR) ON THE TEST SET WITH

DIFFERENT VALUES OF FAR

FAR False Reject Rates (FRR)Face Fingerprint Integration

1% 15.8% 3.9% 1.8%

0.1% 42.2% 6.9% 4.4%

0.01% 61.2% 10.6% 6.6%0.001% 64.1% 14.9% 9.8%

The false reject rates of face recognition are obtained based on 342,750 pair-wisecomparisons; the false reject rates of fingerprint verification are obtained

based on 2,235,000 pairwise comparisons; the false reject rates of the inte-

grated system are obtained based on 590, 000 probes.

"TABLE 2AVERAGE CPU TIME FOR-ONE TEST ON A

SUN SPARC 20 WORKSTATION.

Face Fingerprint TotalRecognition Verification (seconds)

(seconds) (seconds)

0.9 3.2 4.1

1305


10t

10"" 10"" 10" . tO' 10'False Acceptanco Rate 1%)

Fig. 11. Receiver operating curves; the vertical axis is (1-FRR).

The decision-fusion scheme formulated in this paper maybe applied to similar scenario in other domains to provide abetter discrimination performance. For example, in imagedatabase retrieval, a less reliable but computationally attrac-tive algorithm may be used to retrieve the top n matches;then a more reliable, but computationally more expensivealgorithm may be used to verify the top n matches; and fi-nally an integrated decision criterion may be used to reach amore reliable decision.

We must point out that the proposed system has beendesigned for a template database containing several thou-sand templates. Since it has not yet been shown that facerecognition is sufficiently efficient in correctly retrieving asmall number of top matches from a huge template data-base with millions of templates, our approach may not scaleup very well. In addition,. our decision fusion scheme as-sumes that the similarity values between faces are statisti-cally independent of the similarity values between finger-prints. While the assumption is valid for fingerprints andfaces, it may not be true for other biometric characteristics.

The specified FAR of a deployed biometric system isusually a very small number (< 1 percent). In order to pro-vide a more convincing demonstration that the system canmeet such a specification; large representative test samplesare needed. We are in the process of conducting such a teston a larger face and fingerprint database.

ACKNOWLEDGMENTS

We gratefully acknowledge our many useful discussionswith Sharath Pankanti and Ruud Bolle of the IBM T. J. Wat-son Research Lab.

REFERENCES

Ill) J. Atick, P. Griffin. and A. Redlich. "Statistical Appiroach to ShapeFrom Shading: Reconstruction of 3D Face Surfaces From Single2D Images." Neural Computation, to appear.

121 E.S. Bigun. J. Bigun. B. Duc, and S. Fischer, "Expert Conciliationfor Multi Modal Person Authentication Systems by Bayesian Sta-tistics," Proc. First lnt ' Conf. 'Audio Video-Based Personal Authentica-tion. pp. 327-334. Crans-Montana. Switzerland. Mar. 1997.

13) R. Brunelli and D. Falavigna, "Personal Identification Using Mul-tiple Cues," IEEE Trans. Pattern Analysis and Machine Intelligence.vol. 17, no. 10, pp. 955-966, Oct. 1995.

[4) R. Brunelli and T Poggio, "Face Recognition: Features VersusTemplates," IEEE Trans. Pattern Analysis and Machine Intelligence,vol. 15, no. 10. pp. 1,042-1,052, Oct. 1993.

15) R. Chellappa, C. Wilson, and A. Sirohey, "Human and MachineRecognition of Faces: A Survey" Proc. IEEE, vol. 83. no. 5. pp. 705-740, 1995.

161 R. Clarke, "Human Identification in Information Systems: Man-agement Challenges and Public Policy Issues," Information Tech-nology & People, vol. 7, no. 4, pp. 6=37, 1994.

171 J.C. Daugman, "High Confidence Visual Recognition of Personsby a Test of Statistical Independence," IEEE Trans. Pattern Analy-sis and Machine Intelligence, vol, 15, no. 11, pp. 1,148-1,161, Nov.1993.

18) S.C. Davies,, "Touching Big Brother: How Biometric Technology .Will Fuse Flesh and Machine.".. Information Technology & People,vol. 7, no. 4, pp. 60-69. 1994.

19) EJ. Cumbel, Statistics of Extremes. New York: Columbia Univ.Press. 1958.

|10) Z. Hong, "Algebraic Feature Extraction of Image for Recogni-lion," Pattern Recognition. vol: 24. no. 2, pp: 211-219.'1991.

Ill) A. lain. L. Hong, and R. Bolle, "On-Line Fingerprint Verification,"IEEE Trans. Pattern Analysis and Machine Intelligence. vol. 19. no. 4,pp. 302-314, Apr. 1997.

112) M. Kirby and L. Sirovich, "Application of the Karhunen-Loeve'Procedure for the Characterization of Human Faces," IEEE Trans.Pattern Analysis and Machine Intelligence, vol. 12, no. I, pp. 103-108, Jan. 1990.

113) J. Kittler, Y. Li, J. Matas, and M.U. Sanchez, "Combining Evidencein Multimodal Personal Identity Recognition Systems," Proc. FirstInt'l Conf. Audio Video-Based Personal Authentication, pp., 327-334,Crans-Montana, Switzerland, Mar. 1997.

114) H.C. Lee and R.E. Caensslen, Advances in Fingerprint Technology.New York: Elsevier, 1991.

115) A. Moenssens, Fingerprint Techniques. London: Chilton BookCompany, 1971,

116) V. Nalwa, "Automatic On-Line Signature Verification," Proc. IEEE,vol. 85, no. 2, pp. 213-239, 1997.

1171 E. Newham, The Biometric Report. New York: SJB Services, 1995.1181 A. Papoulls, Probability, Random Variables, and Stochastic Processes.

New York: McGraw-Hill, 1965.1191 P.J. Phillips, PJ. Rauss, and S.Z. Der, FERET (Face Recognition Tech-

nology) Recognition Algorithm Development and Test Results. Adel-phi, Md.: U.S. government publication. ALR-TR-995, Army Re-search Laboratory. 1996..

120) N. Ratha, S. Chen, and A.K. Jain, "Adaptive Flow OrientationBased Feature Extraction in Fingerprint Images," Pattern Recogni-tion, vol. 28, no. 11, pp. 1,657-1,672. 1995.

1211 N. Ratha, K. Karu, S. Chen, and AK. Jain. "A Real-Time MatchingSystem for Large Fingerprint Database," IEEE, Trans, PatternAnalysis and Machine Intelligence, vol. 18, no. 8, pp. 799-813. Aug.1996.

1221 L. Sirovich and M. Kirby. "Low Dimensional, Procedure for Char-acterization of Human Faces." J. Optical Soc. Am., vol. 4, no. 3.pp. 519-524, 1987.

123) D.L. Swets and J. Weng. "Using Discriminant Eigenfeatures forImage Retrieval," IEEE Trans. Pattern Analysis and Machine Intelli-gence. vol. 18, no. 8. pp. 831-836. Aug. 1996.

124) M. Turk and A. Pentland, "Eigenfaces. for Recognition," ]: Cogni-tive Neumroscience,,vol. 3. no. 1. pp. 71-86. 1991.

(25) D. Valertin, H. Abdi, A.). O'Toole, and C. Cottrell. "ConnectionistModels of Face Processing: A Survey," Pattern Recognition. vol. 27.,no. 9, pp. 1,209-1,230,1994.

126) Y.A. Zuev and S.K. Ivanov, "The Voting as a Way to Increase theDecision Reliability," Proc. Foundations of Information/Decision Fu-sion With Applications to Eng. Problems, pp. 206-210, Washington,D.C., Aug. 1996.

1271 A.K. Jain, R. Bolle, and S. Pankanti. eds.. Biometrics: Personal Iden-tilication in Networked Society. Norwell. Mass.: Kluwer AcademicPublishers, in press.

1306


Lin Hong received the BS and MS degrees incomputer science from Sichuan University,China, in 1987 and 1990, respectively, and thePhD degree in computer science from MichiganState University in 1998. His currently researchinterests include multimedia, biometrics, datamining, pattern recognition, image processing,and computer vision application. He is nowworking at Visionics Corporation.

Anil Jain is a university distinguished professorand chair of the Department of Computer Sci-ence at Michigan State University: His researchinterests include statistical pattern recognition,Markov random fields, texture analysis, neuralnetworks, document image analysis, fingerprintmatching, and 3D object recognition. He re-ceived the best paper awards in 1987 and 1991and certificates for outstanding contributions in1976, 1979, 1992, and 1997 from the PatternRecognition Society. He also received the 1996

IEEE Transactions on Neural Networks Outstanding Paper Award. Hewas the editor-in-chief of the IEEE Transactions on Pattern Analysisand Machine Intelligence (1990-1994). He is the coauthor of Algo-rithms for Clustering Data (Prentice-Hall, 1988), has edited the bookReal-Time Object Measurement and Classification (Springer-Verlag,1988), and coedited the books, Analysis and Interpretation of RangeImages (Springer-Verlag, 1989), Markov Random Fields (AcademicPress, 1992), Artificial Neural Networks and Pattern Recognition (El-sevier, 1993), 3D Object Recognition (Elsevier, 1993), and BIOMET-RICS: Personal Identification in Networked Society to be published byKluwer in 1998. He is a fellow of the IEEE and IAPR and has receiveda Fulbright research award.

1307

'' ''

This Page is Inserted by IFW Indexing and Scanning

Operations and is not part of the Official Record




D BLACK BORDERS

L IMAGE CUT OFF AT TOP, BOTTOM OR SIDES

ADED TEXT OR DRAWING

I BLURRED OR ILLEGIBLE TEXT OR DRAWING

D SKEWED/SLANTED IMAGES

Q COLOR OR BLACK AND WHITE PHOTOGRAPHS

I GRAY SCALE DOCUMENTS

D LINES OR MARKS ON ORIGINAL DOCUMENT

Q REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY

Q OTHER:



INTERNATIONAL SEARCH REPORTInternational application No.

PCTIUS05/33378

k. CLASSIFICATION OF SUBJECT MATTERIPC: GO6K 9/00( 2006.01),9/64( 2006.01);HO4N 9/47( 2006.01),7/18( 2006.01);HO4B 17/00( 2006.01)

USPC: 382/11 5-118,124;348/143-161;455/67.12According to Intemrnational Patent Classification (IPC) or to both national classification and IPC

B. FIELDS SEARCHED

Minimum documentation searched (classification system followed by classification symbols)U.S.: 382/115-118,124;348/143-161;455/67.12

Documentation searched other than minimum documentation to the extent that such documents are included in the fields searched

Electronic data base consulted during the international search (name of data base and, where practicable, search terms used)

C. DOCUMENTS CONSIDERED TO BE RELEVANT

Category * Citation of document, with indication, where appropriate, of the relevant passages Relevant to claim No.

X US 2002/0190119 A (HUFFMAN) 19 December 2002 (19.12.2002), entire document 1-20

X US 200410081338 Al I (TAKENAKA) 29 April 2004 (29.4.2004), entire document - 1-20

X US 6,697,103 BI (FERNANDEZ et al) 24 February 2004 (24.02.2004), column 4 lines 23- I, 17,2067, column 6 lines 1-30 Figures I and 3-4.

SFurther documents are listed in the continuation of Box C. See patent family annex.

* Special categories of cited documents: "T" later document published after the international filing date or prioritydate and not in conflict with the application but cited to understand the

"A" docmuit defining the gencaml state of the art which is not considered to be of principlc or theory undedying the invention

particular relevance"X" document of particular relevance; the claimed invention cannot be

"E" ealier application or patent published on or after the international filing date considered novel or cannot be considered to involve an inventive stepwhen the docunent is taken alone

"L" document which many tduow doubts on priority claim(s) or which is cited toestablish the publication date of another citatidoo or other special reason (as "Y" document of particular relevancc; the claimed invention cannot be

specified) considered to involve an inventive step when the docuncut is conmbus

with one or more other isucb documents, such combination being

"O" document referring to an oral disclosure, use, exhibition or other means obvious to a person stilled in the al

P" documwent published prior to the international filing date but later than the -&" document mcembrner of the same patent family

priority date claimed

Date of the actual completion of the international search Date of mailing of the intemrn s scl ep 'j

13 March 2006 (13.03.2006) M f l

Name and mailing address of the ISA/US Authorized officer

Mail Stop PCT, Atn: ISA/US Jingge WuCommissioner for Patents

P.O. Box 1450 22313-1450 Telephone No. 571-272-7361Facsimilexandria, Virginia 2 2320 3-1450Facsimile No. (571) 273-3201

Form PCT/ISA/210 (second sheet) (Apn)l 2t))-

Application No. 11/231,35,3Docket No. 577832000200

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 4, APRIL 1997

On-Line Fingerprint VerificationAnil Jain, Fellow, IEEE, Lin Hong, and Ruud Bolle, Fellow, IEEE

Abstract-Fingerprint verification is one of the most reliable personal identification methods. However, manual fingerprintverification is so tedious, time-consuming, and expensive that it is incapable of meeting today's increasing performancerequirements. An automatic fingerprint identification system (AFIS) is widely needed. It plays a very important role in forensic andcivilian applications such as criminal identification, access control, and ATM card verification. This paper describes the design andimplementation of an on-line fingerprint verification system which operates in two stages: minutia extraction and minutia matching.An improved version of the minutia extraction algorithm proposed by Ratha et al., which is much faster and more reliable, isimplemented for extracting features from an input fingerprint image captured with an on-line inkless scanner. For minutia matching,an alignment-based elastic matching algorithm has been developed. This algorithm is capable of finding the correspondencesbetween minutiae in the input image and the stored template without resorting.to exhaustive search and has the ability of adaptivelycompensating for the nonlinear deformations and inexact pose transformations between fingerprints. The system has been testedon two sets of fingerprint images captured with inkless scanners. The verification accuracy is found to be acceptable. Typically, acomplete fingerprint verification procedure takes, on an average, about eight seconds on a SPARC 20 workstation. Theseexperimental results show that our system meets the response time requirements of on-line verification with high accuracy.

Index Terms-Biometrics, fingerprints, matching, verification, minutia, orientation field, ridge extraction.

---------- + ----------

1 INTRODUCTION

F INGERPRINTS are graphical flow-like ridges present onhuman fingers. They have been widely used in personal'

identification for several centuries 111. The validity of theiruse has been well established. Inherently, using currenttechnology fingerprint identification is much more reliablethan other kinds of popular personal identification methodsbased on signature, face, and speech 1111, (31, 1151. Al-though fingerprint verification is usually associated withcriminal identification and police work, it has now becomemore popular in civilian applications such as access control,financial security, and verification of firearm purchasersand driver license applicants 11il1, 131. Usually, fingerprintverification is performed manually by professional finger-print experts. However, manual fingerprint' verification isso tedious, time-consuming, and expensive that it does notmeet the performance requirements of the new applica-tions. As a result, automatic fingerprint, identification sys-tems (AFIS) are in great demand I1ll. Although significantprogress has been made in designing automatic fingerprintidentification systems over the past 30 years, a number ofdesign factors (lack of reliable minutia extraction algorithms,difficulty in quantitatively defining a reliable match between .fingerprint images, fingerprint classification, etc.) create bot-tlenecks in achieving the desired performance i111).

* A. lain and L. Hong are with the Pattern Recognition and Image Process-ing Laboratory, Department of Computer Science, Michigan State Univer-sity, East Lansing, MI 48824. E-mail: (fain, honglin)0cps.msu.edu.

* R. Bolle is with the Exploratory Computer Vision Croup, IBM. TJ. WatsonResearch Center. Yorktown Heights; NY 10598.Email: [email protected].

Manuscript received Feb. 2, 1996; revised Oct. 21. 1996. Recommended for accep-tance by B. Dom.For information on obtaining reprints of this article. please send e-mail to:[email protected]. and reference IEEECS Log Number P96113.

An automatic fingerprint identification system is con-cerned with some or all of the following issues:

* Fingerprint Acquisition: How to acquire fingerprint im-ages and how to represent them in a proper format.

* Fingerprint Verification: To determine whether twofingerprints are from the same finger.

* Fingerprint Identification: To search for a query finger-print in a database.

* Fingerprint Classification: To assign a given fingerprintto one of the prespecified categories according to itsgeometric appearance.

A number of methods are used to acquire fingerprints.Among them, the inked impression method remains themost popular. It has been essentially a standard techniquefor fingerprint acquisition for more than 100 years [31. Thefirst step in capturing an inked impression of a fingerprintis to place a few dabs of ink on a slab then rolling it outsmoothly with a roller until the slab is covered with a thin,even layer of ink. Then the finger is rolled from one side ofthe nail to the other side over the inked slab which inks theridge patterns on top of the finger completely. After that,the finger is rolled on a piece of paper so that the inked im-pression of the ridge pattern of the finger appears on thepaper. Obviously, this method is time-consuming and un-suitable for an on-line fingerprint verification system. In-kless fingerprint scanners are now available which are ca-pable of directly acquiring fingerprints in digital form. Thismethod eliminates the intermediate digitization process ofinked fingerprint impressions and makes it possible tobuild an on-line system. Fig. I shows the two inkless fin-gerprint scanners used in our verification system. Finger-print images captured with the inked impression methodand the inkless impression method are shown in Fig. 2.

0162-8828197510.00 01997 IEEE

JAIN ET AL.: ON-LINE FINGERPRINT VERIFICATION

(a) (b)Fig. 1. Inkless' fingerprint scanners: (a) Manufactured by Identlx.(b) Manufactured by Digital Biometrics.

(a) (b)

Fig. 2. Comparison of fingerprint images captured by different meth-ods. (a) Inked impression method (from NIST database). (b) Inklessimpression method (with a scanner manufactured by Digital Biometrics).

I-f'

(d) (e) (f)Fig. 3. A coarse-level fingerprint classification into six categories:(a) Arch. (b) Tented arch. (c) Right loop. (d) Left loop. (e) Whorl. (f) Twinloop.

The goal of fingerprint classification is to assign a givenfingerprint to a specific category according to its geometricproperties (Fig. 3 shows a coarse-level fingerprint classifi-cation). The main purpose of fingerprint classification is tofacilitate the management of large fingerprint databasesand to speedup the process of fingerprint matching. Gener-ally, manual fingerprint classification is performed within aspecific framework such as the well-known Henry system131. Different frameworks use different sets of properties.

t

However, no matter what type of framework is used, theclassification is based on ridge patterns, local ridge orienta-tions and minutiae. Therefore, if these properties can bedescribed quantitatively and extracted automatically from afingerprint image then fingerprint classification will be-come an easier task. During the past several years, a num-ber of researchers have attempted to solve the fingerprintclassification problem 11)1, 131, 19), 110), 126). Unfortunately,their efforts have not resulted in the desired accuracy. Algo-rithms reported in the literature classify fingerprints into five,or six categories with about 90 percent classification accuracyon a medium size test set (several thousand images) 191, 1101,126). However, to achieve a higher recognition accuracy witha large number of categories still remains a difficult problem.

Fingerprint verification determines whether two finger-prints are from the same finger or not. It is widely believedthat if two fingerprints are from the same source, then theirlocal ridge structures (minutia details) match each othertopologically 1111, 13]. Eighteen different types of local ridgedescriptions have . been identified . 1111. The two mostprominent structures are ridge endings and ridge bifurca-tions which are usually called minutiae. Fig. 4 shows ex-amples of ridge endings and ridge bifurcations. Based onthis observation and by representing the minutiae as apoint pattern, an automatic fingerprint verification problemmay be reduced to a point pattern matching (minutiamatching) problem. In the ideal case, if

1) the correspondences between the template and inputfingerprint are known,

2) there are no deformations such as translation, rotationand nonlinear deformations, etc. between them, and

3) each minutia present in a fingerprint image is exactlylocalized, then fingerprint verification consists of thetrivial task of counting the number of spatiallymatching pairs between the two images.

Ridge Ending Ridge Bifurcation

Fig. 4. Ridge ending and ridge bifurcation.

However, in practice

1) no correspondence is known beforehand,2) there are relative translation, rotation and nonlinear

deformations between template minutiae and inputminutiae,

3) spurious minutiae are present in both templates andinputs, and

4) some minutiae are missed,

Therefore, in order for a fingerprint verification algorithmto operate under such circumstances, it is necessary toautomatically obtain minutia correspondences, to recoverdeformations, and to detect spurious minutiae from finger-print images. Unfortunately, this goal is quite difficult toachieve. Fig. 5 illustrates the difficulty with an example oftwo fingerprint images ofthe same finger.

per: -"s

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELUGENCE, VOL. 19, NO. 4, APRIL 1997

an -- *.- .4-.-

Fig. 5. Two different fingerprint images from the same finger. In orderto know the correspondence between the minutiae of these two finger-print images, all the minutiae must be precisely localized and the de-formations must be recovered.

Fingerprint identification refers to the process ofmatching a query fingerprint against a given fingerprintdatabase to establish the identity of an individual. Its goalis to quickly determine whether a query fingerprint ispresent in the database and to retrieve those which aremost similar to the query from the database.' The criticalissues here are both retrieval speed and accuracy. In fact,this problem relates to a number of techniques studiedunder the auspices of computer visiori, pattern recogni-tion, database, and parallel processing. Olperational fin-gerprint retrieval systems are being used by various lawenforcement agencies Ill).

In this paper, we will introduce an on-line fingerprintverification system whose purpose is to capture finger-print images using an inkless scanner and to comparethem with those stored in the database in "real time."Such a system has great utility in a variety of personalidentification and access control applications: The overallblock diagram of our system is shown in Fig. 6. It operatesas follows:

I) Off-line phase: Several impressions (depending on thespecification of the system) of the fingerprint of a per-son to be verified are first captured and processed bya feature extraction module; the extracted features arestored as templates in a database for later use;

2) On-line phase: The individual to be verified giveshis/her identity and places his/her finger on the in-kless fingerprint scanner, minutia points are extractedfrom the captured fingerprint image; these minutiaeare then fed to a matching module, which matchesthem against his/her own templates in the database.

-:-

• I. n . . . ..n.n .... ...

. ...... - -.. "

, l I ... Ni, .imj,,a NI ,,,u 1)flgJ~mntVij"h

Fig. 6. Overview of our on-line fingerprint verification system.

The following two modules are the main components ofour on-line fingerprint verification system:

* Minutiae extraction. Minutiae are ridge, endings orridge blfurcatlons. Generally, if a perfect segmenta-tion can be obtained, then minutia extraction is just atrivial task of extracting singular points in a thinnedridge map. However, in practice, it is not always pos-sible to obtain a perfect ridge map. Some global heu-ristics need to be used to overcome this limitation.

* Minutia matching. Minutia matching, because of de-formations in sensed fingerprints, is an elasticmatching of point patterns without knowing their

.correspondences beforehand. Generally, finding thebest match between two point patterns is intractableeven if minutiae are exactly located and no deforma-tions exist between these two point patterns. The ex-istence of deformations makes the minutia matchingmuch more difficult.

For segmentation and minutia extraction, a modifiedversion of the minutia extraction algorithm proposed in 1181is implemented which is much faster and more reliable forminutia extraction. We propose a hierarchical approach toobtain a smooth orientation field estimate of the input fin-gerprint image, which greatly improves the performance ofminutia extraction. For minutia matching, we propose analignment-based elastic matching algorithm. This algorithmis capable of finding the correspondences between minutiaewithout resorting to an exhaustive search and has the abil-ity to adaptively compensate for the nonlinear deforma-tions and inexact pose transformations between differentfingerprints. Experimental results show that our systemachieves excellent performance in a real environment.

In the following sections we will describe in detail ouron-line fingerprint verification system. Section 2 mainlydiscusses the fingerprint feature extraction module. Sec-tion 3 presents our minutia matching algorithm. Experi-mental results on two fingerprint databases captured withtwo different inkless scanners are described in Section 4.Section 5 contains the summary and discussion.

2 MINUTIA EXTRACTION

It is widely known that a professional fingerprint examinerrelies on minute details of ridge structures to make finger-print. identifications [1111, 13). The topological structure ofthe minutiae of a fingerprint is unique, and invariant withaging and impression deformations [11), 131. This impliesthat fingerprint identification can be based on the topologi-cal structural matching of these minutiae. This reduces thecomplex fingerprint verification to minutia matching proc-ess which, in fact, is a sort of point pattern matching withthe capability of tolerating, to some restricted extent, de-formations of the input point patterns. Therefore, the firststage in an automatic fingerprint verification procedure isto extract minutiae from fingerprints. In our on-line finger-print verification system, we have implemented a minutiaextraction algorithm which is an improved version of themethod proposed by Ratha et al. 1181. Its overall flowchartis depicted in Fig. 7. We assume that the resolution of inputfingerprint images is 500 dpi.


Fingerprint Smoothin Orientation FingerprintField Region

Image Filer Estimation Localization

Minutiac Minutia RidgeExtracnon Thinning Extraction

S...............................................................

Fig. 7. Flowchart of the minutia extraction algorithm.

2.1 Estimation of Orientation FieldA number of methods have been proposed to estimate theorientation field of flow-like patterns 117). In our system, anew hierarchical implementation of Rao's. algorithm 1171 isused. Rao's algorithm consists of the following main steps:

1) Divide the input fingerprint image into blocks of sizeWx W.

2) Compute the.gradients Gx and Cy at each pixel in eachblock.

3) Estimate the local orientation of each block using thefollowing formula:

1W - 1 2Gx(i, j)G,(ij)S'(1)o = 2tan-' i ,=,I=,(Gx(i.j)

- Gy(i, j)) 1

where W is the size of the block, and G, and Cy arethe gradient magnitudes in x and y directions, re-spectively.

The orientation field of a good quality fingerprint imagecan be reasonably estimated with this algorithm. However,the presence of high-curvature ridges, noise, smudges, andbreaks in ridges leads to a poor estimate of the local orien-tation field. A postprocessinrig procedure needs to be ap-plied to overcome this limitation. In our system, the fol-lowing iterative steps are added to improve an inconsistentorientation field:

* Compute the consistency level of the orientation fieldin the local neighborhood of a block (i, j) with thefollowing formula:

C = - (i',j) - 0(i.j)2 (2)oN*N (r._)ED

10' - BI=

d if(d = (0' - + 360) mod 360) < 180d - 180 otherwise

where D represents the local neighborhood around

the block (i, J) (in our system, the size of.D is 5 x 5); Nis the number of blocks within D; 0(i',j') and 6(i,j)

are local ridge orientations at blocks (i', j') and (i, j),

respectively.

* If the consistency level (2) is above a certain threshold

ST. then the local orientations around this region are,reestimated at a lower resolution level until it is belowa certain level. With this post-smoothing scheme, afairly smooth orientation field estimate can be ob-tained. Fig. 8 shows the orientation field of a finger-print image estimated with our new algorithm:

(a) Rao's method. (b) Hierarchical method.Fig. 8. Comparison of orientation fields by Rao's method and theproposed hierarchical method; the block size (W x WI) is 16 x 16 andthe size of 0 is 5 x 5.

After.the orientation field of an input fingerprint imageis estimated, a segmentation algorithm which is based onthe local variance of gray level is used to locate the regionof interest from the fingerprint image. In our segmentationalgorithm, we assuinme. that there is only one fingerprint pre-sent in the image.

2.2 Ridge Detection

After the orientation field of the input image is estimatedand the fingerprint region is located, the next step of ourminutia exaction algorithm, is ridge detection. The mostsalient property corresponding to ridges in a fingerprintimage is the fact that gray level values on ridges attain theirlocal maximaalong the normal directions of local ridges.Therefore, pixels can be identified to be ridge pixels basedon this property. In our minutia detection algorithm, a fin-gerprint image is first convolved With the following twomasks, h1(x, y u, v) and hb(x, y u. v),, of size L x H (on anaverage 11 x 7 in our system), respectively. These twomasks are capable of adaptively accentuating the localmaximum gray level values along the normal direction ofthe local ridge direction:

h,(x. y: u: v) =

I e-d

I -

0

if vctg(0(x.y)) - 2cos(H(xy))2 cos(e(x, y))

(4)ifu = (vctg(0(x,y)), v E

otherwise


h(x, y: u. v)=

S( .)

2n e if U - vctg(0(x. y)) + H V E v n2 cos(9(x, y))

ifu = (vctg((x.y))), ve

otherwise

Lsin((x, y)) Lsin(o(x,y))2= - 2 2

Although the above heuristics do delete a large percentageof spurious minutiae, many spurious minutiae still survive.The reason is that the above processing relies on local ridgeinformation. If this information itself is unreliable, then theabove heuristics have no way of differentiating false minu-tiae from true minutiae. Therefore, a refinement which is

(5) based on structural information is necessary. Our refine-ment algorithm eliminates the spurious minutiae based onthe following rules:

* If several minutiae form a cluster in a small region,then remove all of them except for the one nearest tothe cluster center.

* If two minutiae are located close enough, facing eachother, but no ridges lie between them, then removeboth of them.

where 9(x, y) represents the local ridge direction at pixel(x, y). If both the gray level values at pixel (x, y) of the con-volved images are larger than a certain threshold Tidge, thenpixel (x, y) is labeled as a ridge. By adapting the maskwidth to the width of the local ridge, this algorithm canefficiently locate the ridges in a fingerprint image.

However, due to the presence of noise, breaks, andsmudges, etc. In the Input Image, the resulting binary ridgemap often contains holes and speckles. When ridge skele-tons are used for the detection of minutiae, the presence ofsuch holes and speckles will severely handicap the per-formance of our minutia extraction algorithm because theseholes and, speckles may drastically change the skeleton ofthe ridges. Therefore, a hole and speckle removal procedureneeds to be applied before ridge thinning.

After the above steps are performed on an input finger-print image., a relatively smooth ridge map of the finger-print Is obtained. The next step of our minutia detectionalgorithm is to thin the ridge map and locate the minutiae.

2.3 Minutia DetectionMinutia detection is a trivial task when an Ideal thinnedridge map is obtained. Without a loss of generality, we as-sume that if a pixel is on a thinned ridge (eight-connected),then it has a value 1, and 0 otherwise. Let (x, y) denote apixel on a thinned ridge, and N o, N i, .... N denote its eight

neighbors. A pixel (x, y) is a ridge ending if( , N) = 1

and a ridge bifurcation if (8 0 N)> 2. However, the

presence of undesired spikes and breaks present in athinned ridge map may lead to many spurious minutiaebeing detected. Therefore, before the minutia detection, asmoothing procedure is applied to remove spikes and tojoin broken ridges. Our ridge smoothing algorithm uses thefollowing heuristics:

* If a branch in a ridge map is roughly orthogonal tothe local ridge directions and. its length is less than aspecified threshold T. then it will.be removed.

* If a break in a ridge is short enough and no otherridges pass through it, then it will be connected.

(a) Input image.

(c) Fingerprint region

(c) Fingerprint region.

(e) Thlinned ridge map.

(b) Orientation field.

(d) Ridge map.

() extracted minutiae.(f) extracted minutiae.

i

Fig. 9. Results of our minutia extraction algorithm on a fingerprint im-age (512 x 512) captured with an inkless scanner. (a) Input image.(b) Orientation field superimposed on the input image. (c) Fingerprintregion. (d) Extracted ridges. (e) Thinned ridge map. (f) Extracted minu-tiae and their orientations superimposed on the input image.

I - t

0

JAIN ET AL: ON-LINE FINGERPRINT VERIFICATION

After the above refinement procedure is performed, thesurviving minutiae are treated as true minutiae. Althoughthe above heuristics can not ensure a perfect location ofeach minutia, they are able to delete several spurious mi-nutiae. For each surviving minutia, the following parame-ters are recorded:

I) x-coordinate,2) y-coordinate,3) orientation which is defined as the local ridge orien-

tation of the associated ridge, and4) the associated ridge.

The recorded ridges are represented as one-dimensionaldiscrete signals which are normalized by the average inter-ridge distance. These recorded ridges are used for align-ment in the minutia matching phase. Fig. 9 shows the re-sults of our minutia extraction algorithm on a fingerprintimage captured with an inkless scanner.

3 MINUTIA MATCHING.

Generally, an automatic fingerprint verification/identifica-tion is achieved with point pattern matching (minutiaematching) instead of a pixel-wise matching or a ridge pat-tern matching of fingerprint images. A number of pointpattern matching algorithms have been proposed In theliterature 123), 111, 1211, 1161. Because a general pointmatching problem is essentially intractable, features associ-ated with each point and their spatial properties such as therelative distances between points are often used in these al-gorithms to reduce the exponential number of search paths.

The relaxation approach [161 iteratively adjusts the con-fidence level of each corresponding pair based on its con-sistency with other pairs until a certain criterion is satisfied.Although a number of modified versions of this algorithmhave been proposed 'to reduce the matching complexity123), these algorithms are inherently slow because of theiriterative nature.

The Hough 'transform-based approach proposed byStockman et al. 1221 converts point pattern matching to aproblem of detecting the highest peak in the Hough spaceof transformation parameters. It discretizes the transforma-tion parameter space and accumulates evidence in the dis-cretized space by deriving transformation parameters thatrelate two point patterns using a substructure or featurematching technique. Karu and Jain 181 proposed a hierar-chical Hough transform-based registration algorithm whichgreatly reduced the size of accumulator array by a mul-tiresolution approach. However, if the number of minutiapoint is less than 30, then it is very difficult to accumulateenough evidence in the Hough transform space for a reli-able match.

Another approach to point matching is based on energyminimization. This approach defines a cost function basedon an initial set of possible correspondences and uses anappropriate optimization algorithm such as genetic algo-rithm [1) and simulated annealing 1211 to find a possiblesuboptimal match. These methods tend to be very slow andare unsuitable for an on-line fingerprint verification system.

In our system, an alignment-based matching algorithm is

implemented. Recognition by alignment has received agreat deal of attention during the past few years 1121, be-cause it is simple in theory, efficient in discrimination, andfast in speed. Our alignment-based matching algorithmdecomposes the minutia matching into two stages:

1) Alignment stage, where transformations such astranslation, rotation and scaling between an input anda template in the database are estimated and the inputminutiae are aligned with the template minutiae ac-cording to the estimated parameters; and

2) Matching stage, where both the input minutiae and thetemplate minutiae are converted to polygons in thepolar coordinate system and an elastic string match-ing algorithm is used to match the resulting polygons.

3.1 Alignment of Point PatternsIdeally, two sets of planar point patterns can be alignedcompletely by two corresponding point pairs. ,A truealignment between two point patterns can be obtained bytesting all possible corresponding point pairs and selectingthe optimal one. However, due to the presence of noise anddeformations, the input minutiae cannot always be alignedexactly with respect to those of the templates. In' order toaccurately recover pose transformations between two pointpatterns, a relatively large number of corresponding pointpairs need to be used. This leads to a prohibitively. largenumber of possible correspondences to be tested. Therefore,an alignment by corresponding point pairs is not practicaleven though it is feasible.

Y

Templat Minuti

(6 %.1y)

npu Minultu Inulu KiJl

Fig. 10. Alignment of the input ridge and the template ridge.

It is well known that corresponding curve segments arecapable of aligning two point patterns with a high accuracyin the presence of noise and deformations. Each minutia ina fingerprint is associated with a ridge. It is clear that a truealignment can be achieved by aligning .correspondingridges (see Fig. 10). During the minutiae detection stage,when a minutia is. extracted and recorded, the ridge onwhich it resides is also recorded. This ridge is representedas a planar curve with its origin coincident with the minutiaand its x-coordinate being in the same direction as the.di-rection of the minutia. Also, this planar curve is normalizedwith the average inter-ridge distance. By matching theseridges, the relative pose transformation between the inputfingerprint and the template can be accurately estimated.

To be specific, let Rd and R denote the sets of ridges asso-


dciated with the minutiae in input image and template, re-'spectively. Our alignment algdrithm can be described interms of the following steps:

1) For each ridge d e Rd. represent it as an one-dimensional discrete signal and match it against each

ridge, D E RD according to the following formula:

'dL dD 2

io-=0

where L is the minimal length of the two ridges and d,

and D; represent the distances from point i on theridges d and D to the x-axis, respectively. The sam-pling interval on a ridge, is set to the average inter-

ridge distance. If the matching score S (0 < S < 1) is

larger than a certain threshold Tr, then go to step 2.,otherwise continue to match the next pair of ridges.

2) Estimate the pose transformation between the twoSridges (Fig. 10). Generally, a least-square method canbe used to estimate the pose transformation. How-ever, in our system, we observe that the followingmethod is capable of achieving the same accuracy with

less computation. The translation vector (Ax, Ay) be-tween the two corresponding ridges is computed by

AX l Xd

y yd D (8)

where (xd, y)T and (x D, yo)T are thex and y coordi-nates of the two minutiae, which are called referenceminutiae, associated with the ridges d and D, respec-

tively. The rotation angle Ao between the two ridgesis computed by

i=0

where L is the minimal length of the two ridges d and

D; y, and F' are radial angles of the ith point on theridge with respect to the reference minutia associatedwith the two ridges d and D, respectively. The scaling

.. factor between the input and template images Is as-sumed to be one. This is reasonable, because finger-print images are captured with the same device inboth the off-line processing phase and the on-lineverification phase.

3) Denote the minutia (x, y. Od)T .based on which thepose transformation parameters are estimated, as thereference minutia. Translate and rotate all the N inputminutiae with respect to this reference minutia, ac-cording to the following formula:

I x cosA= Ay +sin AGA A 0

sin A0- cos Ae. 0

0ily, - yo (10)0)1 - e

where (x,. yi. 0 ) T (i = 1, 2, .... N), represents an input

minutia and (x A , yA, 0A) represents the corre-

sponding aligned minutia.

3.2 Aligned Point Pattern MatchingIf two identical point patterns are exactly aligned with eachother, each pair of corresponding points is completely coin-cident. In such a case, a point pattern matching can be sim-ply achieved by counting the number of overlapping pairs.However, in practice. such a situation is not encountered.On the one hand, the error in determining and localizingminutia hinders the alignment algorithm to recover therelative pose transformation exactly, while on the otherhand, our alignment scheme described above does notmodel, the nonlinear deformation of fingerprints which isan inherent propertyof fingerprint impressions. With theexistence of such a nonlinear deformation, it is impossibleto exactly recover the position of each input minutia withrespect to its.corresponding minutia in the template. There-fore, the aligned point pattern matching algorithm needs to,

be elastic which means that it should be capable of tolerat-ing,.to some extent, the deformations due to inexact extrac-tion of minutia positions and nonlinear deformations. Usu-ally, such an elastic matching can be achieved by placing abounding box around each template minutia, which speci-fies all the possible positions of the corresponding inputminutia with respect to the template minutia, and restrict-ing the corresponding minutia in the input image to bewithin this box 1181. This method does not provide a satis-factory performance in practice, because local deformationsmay be small while the accumulated global deformationscan be quite large. We have implemented an adaptive elas-tic matching algorithm with the ability to compensate theminutia localization errors and nonlinear deformations.

Let

P =( x ylO , . (XP, y.))

denote the set of M minutiae in the template and

Q = x . yX , , ... x y 0QQ=((x, .. . (.(N ,YN'eN)T)

denote the set of N minutiae in the input image which isaligned with the above template with respect to a givenreference minutia point. The steps in our elastic point pat-tern matching algorithm are given below:

1) Convert each minutia point to the polar coordinatesystem with respect to the corresponding referenceminutia on which the alignment.is performed:

(x; -

eI = tan

where (x;, y;, 0;) are.the coordinates of a .minutia,

(11)


(xr, yr t) are the coordinates of the reference minu-

tia, and (r. et, 60)T is the representation of the minutia

in polar coordinate system (r represents the radial

distance, ej represents the radial angle, and 0 repre-sents the orientation of the minutia with respect to thereference minutia).

2) Represent the template and the input minutiae in thepolar coordinate system as symbolic strings by con-catenating each minutia in the increasing order of ra-dial angles:

P = (r,, e, OP) .. , r , e , p )T)

QP = ((r Q, e, O)T ... r, , e, )T)

(12)

(13)

the cost function of a string matching algorithm can achievea certain amount of error tolerance. Given two strings PPand Qp of lengths M and N, respectively, the edit distance,C(M, N), in our algorithm is recursively defined with thefollowing equations:.

C(m, n) =

O

S C(m-l,n)+0fl

min C(m.n- l)+ C(m- ,n- 1)+ w(m,n)J

Il ar-rnQI+P 3e+yeA

if m= O,or n= 0

0< mM.and 0 <n 5 N

if Irrf -r <.<De < e and AO.< c

(15)

(16)

where (r!, ef, 0) and (,Q, e2.9 Q) represent the cor-

responding radius, radial angle, and normalized mi-nutia orientation with respect to the reference minu-tia, respectively.

3) Match the resulting strings Pp and Qp with a dynamic-programming algorithm 141 to find the edit distancebetween Pp and Qp which is described below.

4) Use the edit distance between Pp and Qp to establishthe correspondence of the minutiae between Pp andQp. The matching score, Mg, is then computed ac-cording to the following formula:

100NjojrMpq = OON r

S max{ M, N)}(14)

where Npair is the number of the minutiae which fall inthe bounding boxes of template minutiae. The maxi-mum and minimum values of the matching score are100 and 1, respectively. The former value indicates aperfect match, while the later value indicates nomatch at all.

Minutia matching in the polar coordinate has several ad-vantages. We have observed that the nonlinear deformationof fingerprints has a radial property.. In other words, thenonlinear deformation in a fingerprint impression usuallystarts from a certain point (region) and nonlinearly radiatesoutward. Therefore, it is beneficial to model it in the polarspace. At the same time, it is much easier to formulate rota-tion, which constitutes the main part of the alignment errorbetween an input image and a template, in the polar spacethan in the Cartesian space. The symbolic string generatedby concatenating points in an increasing order of radialangle in polar coordinate uniquely represents a point pat-tern. This reveals that the point pattern matching can beachieved with a string matching algorithm.

A number of string matching algorithms have been re-ported in the literature 141. Here, we are interested in incor-porating an elastic criteria into a string matching algorithm.Generally, string matching can be thought of as the maxi-mization/minimization of a certain cost function such asthe edit distance. Intuitively, including an elastic term in

otherwise

Ae =

.a if (a = e - e + 360) mod 360) < 180

- 180 otherwise

Ae =

a if a = (m - O + 360) mod 360) < 180

a- 180 otherwise

(17)

S(18)

where a. /3, and y are the weights associated with eachcomponent, respectively; 8, e, and e specify the boundingbox; and n is a pre-specified penalty for a mismatch. Suchan edit distance, to some extent, captures the elastic prop-erty of string matching. It represents a cost of changing onepolygon to the other. However, this scheme can only toler-ate, but not compensate for, the adverse effect on matchingproduced by the inexact localization of minutia and nonlin-ear deformations. Therefore, an adaptive mechanism isneeded. This adaptive mechanism should be able to trackthe local nonlinear deformation and inexact alignment andtry to alleviate them during the minimization process.However, we do not expect that this adaptive mechanismcan handle the 'order flip" of minutiae, which, to someextent, can be solved by an exhaustive reordering andmatching within a local angular window.

In our matching algorithm, the adaptation is achievedby adjusting the bounding box (Fig. 11) when an inexactmatch is found during the matching process. It can berepresented as follows:

w

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 4, APRIL 1997'

ar - rf + e + y - if

images. Fig. 12 shows the results of applying the matchingalgorithm to an input minutia set and a template.

,(m, n)

< Ae(m, n)

< eh(m, n)

Template minutia

,lnui'

(19)

A <

Fig. 11. Bounding box and its adjustment.

(20)

(a) (b)

5,(m + 1, n + 1) = S,(m, n) + rlra

6 h(m+ 1, n+ 1) = ± h(m, n) + rAr

(21)

(22)

El(m+ 1, n + I) = E(m, n) +nAe (23)

eh(m+ 1, n+ 1) = Eh(m,n) + /e a (24)

where w(m, n) represents the penalty for matching a pair of

minutiae (r ,e em, ) and (rQ, eQ, 9Q) T , (m, n), Sh(m, n),

el(m, n), and Eh(m, n) specify the adaptive bounding box in

the polar coordinate system (radius and radial angle); and rjis the learning rate. This elastic string matching algorithmhas a number of parameters which are critical to its per-formance. We have empirically determined the values ofthese parameters as follows: 61(0, 0) = -8; 8.h(0,. 0) = +8;

Ej(0, 0) = -7.5: eh(0, 0) = +7.5; e = 30; a = 1.0; /3 = 2.0;: y = 0.1;

S= 200(a + / + 7); r1 = 0.5. The values of 61(0, 0), 6h(0, 0),

F;(0, 0). and eh(0,. 0) depend on the resolution of fingerprint

-r'

(c) / (d)Fig. 12. Results of applying the matching algorithm to an input minutiaset and a template. (a) Input minutia set. (b) Template minutia set.(c) Alignment result based on the minutiae marked with green circles.(d) Matching result where template minutiae and their correspon-dences are connected by green lines.

4 EXPERIMENTAL RESULTS

We have tested our on-line fingerprint verification systemon two sets of fingerprint images captured with two differ-ent inkless fingerprint scanners. Set I contains 10 imagesper finger from 18 individuals for a total of 180 fingerprintimages, which were captured with a scanner manufacturedby Identix. The size of these images is 380 x 380. Set 2 con-tains 10 images per finger from 61 individuals for a total of610 fingerprint images, which were captured with a scannermanufactured by Digital Biometrics. The size of these im-ages is 640 x 480. When these fingerprint images were. cap-

otherwise

8,(m, n)< (ri,;-Qn

< Eh(m,n)

E,(m, n)< Ae< eh(m. n)

(r" - Qrf

A e

AO < C

otherwise

(Ae.) -

JAIN ET AL.: ON-LINE FINGERPRINT VERIFICATION 311

tured, no restrictions on the position and orientation of fin-gers were imposed. The captured fingerprint images varyin quality. Figs. 13 and 14 show some of the fingerprint im-ages in our database. Approximately 90 percent of the fin-gerprint images in our database are of reasonable qualitysimilar to those shown in Figs. 13 and 14, while about10 percent of the fingerprint images in our database are notof good quality (Fig. 15). which are mainly due to largecreases and smudges In ridges and dryness of the im-pressed finger. First, we report some initial results on fin-gerprint matching, followed by fingerprint verification. Thereasons why we did not use NIST-9 fingerprint database1251.to test the performance of our system are as follows:

we concentrate on live-scan verification, andNIST-9 fingerprint database is a very difficult finger-print database which contains a large number of fin-gerprint images of poor quality and no result hasbeen reported from other on-line verification systemsfor comparison.

Fig. 13. Fingerprint images captured with a scanner manufactured byIdentix; the size of these images is 380 x 380; all the three images arefrom the same individuals finger.

Fig. 14. Fingerprint images captured with a scanner manufactured byDigital Biometrics; the size of these images is 640 x 480; all the threeimages are from the same individual's finger.

a0-

Nr

It~ ~~ -. W1- -t" b

Fig. 15. Fingerprint images of poor quality.

4.1 MatchingEach fingerprint in the test set was matched with the otherfingerprints in the set. A matching was labeled correct if the

matched fingerprint was among the nine other fingerprintsof the-same individual, and incorrect otherwise. A total of32,220 (180 x 179) matchings have been performed on testSet I and 371,490 (610 x 609) matchings on test Set 2. Thedistributions of correct and incorrect matching scores areshown in Fig. 16. It can be seen from this figure that thereexist two peaks in the distribution of matching scores. Onepronounced peak corresponds. to the incorrect matchingscores which is located at a value around 10, and the otherpeak which resides at a value of 40 is associated with thecorrect matching scores. This indicates that our algorithm iscapable of differentiating fingerprints at a high correct rateby setting an appropriate value of the threshold. Table 1.shows the verification rates and reject rates with differentthreshold values. The reject rate is defined as the percent-age of correct fingerprints with their matching scores belowthe threshold value. As we have observed, both the incor-rect matches and the high reject rates are due to fingerprintimages with poor quality such as those shown in Fig. 15. Wecan improve these matching results by ensuring that the da-tabase does not contain such poor quality fingerprint images.

: .

Si

-- Corrt

0 10 20 30 40 50 60 70 80 90 10

Normalized Matching Score

(a) Identix

0 10 20 30 40 50 60 70 80 90

Normalized Matching Score

(b) Diaital Biometrics

100

Fig. 16. Distributions of correct and incorrect matching scores; verticalaxis represents distribution of matching scores in percentage. (a) Dis-tribution of matching scores on test set 1 (180 images). (b) Distributionof matching scores on lest set 2 (610 images).

'i~ ~ ~ ~ .. _. Correct- incorrect

k .

.'

" L' w .

i

-- I 1 f f

r.'


TABLE 1THE VERIFICATION RATES AND REJECT RATES

ON TEST SETS WITH DIFFERENT THRESHOLD VALUES

Threshold Verification RejectValue Rate. Rate .

20 99.839% 11.23%22 919.947 % 13.33 %24 99.984 % 16.48 %26 99.994 % 20.49 %28 99.996 % 25.19 %30 100 % 27.72 %

(a) Using Identix system (180 images).

Threshold Verification RejectValue Rate Rate

20 99.426 % 11.23 %22 99.863% 14.55%24 99.899% 16.78 %26 99.969 % 20.20 %28 99.989 % 23.15%30 *99.999 % 27.45 %

(b) Using Digital Biometrics system (610 images).

TABLE 2MATCHING RATES ON TEST SETS USING

THE LEAVE-ONE-OUT METHOD

(a) Using Identix system (180 images).

Number of Matching RateBest Matches

1 92.13%2 94.40 %3 97.06 %4 97.67 %5 98.44 %6 99.11%7 99.70% '8 99.79 %9 99.91%

(b) Using Digital Biometrics System (610 images).

TABLE 3AVERAGE CPU TIME FOR MINUTIA EXTRACTIONAND MATCHING ON A SPARC 20 WORKSTATION

Minutia Extraction(seconds)

Minutia Matching(seconds)

5.35 2.55 7.90

4.2 VerificationIn on-line verification, a user indicates his/her identity.Therefore, the system matches the input fingerprint imageonly to his/her stored templates. To determine the verifica-tion accuracy of our system, we used each one of our data-base images as an input fingerprint which needs to be veri-fled. An input fingerprint image was matched against allthe nine other images of the same finger. If more than onehalf of the nine matching scores exceeded the thresholdvalue of 25, then the input fingerprint image is said to befrom the same finger as the templates and a valid verifica-tion is established. With this scheme, a 100 percent verifica-tion rate can be achieved with a reject rate around16 percent on both test sets. Again, this reject rate can bereduced by preprocessing the database to remove thestored templates of poor quality. This demonstrates that; inpractice, using a k-nearest neighbor type of matching is ade-.quate for a successful verification. Table 2 shows the match-ing rate which is defined as the percentage of the correct fin-gerprints (of the same finger) present among the best n (n = 1,.... 9) matches.

For an on-line fingerprint verification system to be ac-ceptable in practice, its response time needs to be within afew seconds. Table 3 shows the CPU requirements of oursystem. The CPU time for one verification, including fin-gerprint image acquisition, minutia extraction and minutiamatching, is, on an average, approximately eight seconds

on a SPARC 20 workstation. It indicates that our on-linefingerprint verification system does meet the response timerequirement of on-line verification.

The number of tests done on an automatic fingerprintidentification system is never enough. Performance meas-ures are as much a function of the algorithm as they are afunction of the database used for testing. The bi6 metricscommunity is slow at establishing benchmarks and the ul-timate performance numbers of a fingerprint verificationsystem are those which you find in a deployed system.Therefore, one can carry out only a limited amount of test-ing in a. laboratory environment to show the anticipatedsystem performance. Even in field testing, real performancenumbers are. not important-it's often the perceived per-formance which is crucial.

5 CONCLUSIONSWe have designed and implemented an on-line fingerprintverification system which operates in two stages: minutiaextraction and minutia matching. A modified version of theminutia extraction algorithm proposed in 118) is used in oursystem which is much faster and more reliable. A new hier-archical orientation field estimation algorithm results in asmoother orientation field which greatly improves the per-formance of the minutia, extraction. An alignment-based

,elastic matching algorithm is proposed for minutia match-ing. This algorithm is quite fast, because it is capable of

Number of Matching RateBest Matches

1 91.17%2 94.72 %3 96.89 %4 98.17%5 98.89 %6 99.39 %7 99.72 %8 99.83 %9' 99.94 %

Total(seconds)


finding the correspondences between minutia points with-

out resorting to an exhaustive search. At the same time, this

matching algorithm has a good performance, because it has

the ability to adaptively compensate for the nonlinear de-formations and inexact pose transformations between dif-ferent fingerprints. Experimental results show that oursystem achieves excellent performance in a realistic oper-

ating environment. It also meets the response time re-quirement of on-line verification.

Based on the experimental results, we observe that the

matching errors in our system mainly result from incorrectminutiae extraction and inaccurate alignment. We observethat a number of factors are detrimental to the correct loca-

tion of minutia. Among them, poor image quality is themost serious one. Therefore, in the future, our efforts willbe focused on global image enhancement schemes. Anotherissue related to minutia detection is to incorporate a struc-tural-based model in minutia detection which extracts mi-nutiae based on their local ridge formations. For elasticmatching, an important aspect is to utilize additional in-formation (e.g., neighboring ridges) about a minutia to in-

crease the accuracy of alignment.

ACKNOWLEDGMENTS

We gratefully acknowledge our useful discussions withNalina Ratha and Shaoyun Chen, and we would like tothank Nalina Ratha and Chitra Dorai fortheir careful com-ments on earlier versions of this paper.

REFERENCES

IllI N. Ansari, M.H. Chen, and E.S.H. Hou, "A Genetic Algorithm forPoint Pattern Matching," Chapt. 13. B. Soucek and the IRISCroup, eds., Dynamic. Genetic, and Chaotic Programming. NewYork: John Wiley & Sons, 1992.

121 P.E. Danielsson and Q.Z. Ye, "Rotation-Invariant Operators Ap-plied to Enhancement of Fingerprints." Proc. Eighth ICPR. pp. 329-333, Rome, 1988.

131 Federal Bureau of Investigation, The Science of Fingerprints: Classi-fication and Uses. Washington. D.C.: U.S. Government Printing Of-fice, 1984.

141 T.H. Cormen, C.E. Lelserson, and R.L. Rivest. Introduction to Algo-rithms. New York: McGraw-Hill, 1990.

151 D.C.D. Hung, "Enhancement and Feature Purification of Finger-print Images," Pattern Recognition, vol. 26, no. I1. pp. 1,661-1,671,1993.

161 S. Cold and A. Rangarajan, "A Graduated Assignment Algorithmfor Graph Matching," Research Report YALEU/DCS/RR-1062,Yale Univ., Dept. of Computer Science, 1995.

171 L. O'Gorman and J.V. Nickerson. "An Approach to FingerprintFilter Design," Pattern Recognition, vol. 22, no. 1, pp. 29-38, 1989.

181 K. Karu and A.K. Jain, "Fingerprint Registration," Research Re-port, Michigan State Univ., Dept. of Computer Science. 1995.

191 K. Karu and A.K. Jain; "Fingerprint Classification:." Pattern Recog-nition, vol. 29. no. 3, pp. 389-404. 1996.

1101 M. Kawagoe and A. Tojo, "Fingerprint Pattern Classification,"Pattern Recognition, vol. 17. no. 3, pp. 295-303, 1984.

111) H.C. Lee and R.E. Caensslen, eds.. Advances in Fingerprint Technol.ogy. New York: Elsevier. 1991.

1121 D.P. Huttenlocher and S. Ullman, "Object Recognition UsingAlignment." Proc. First Intl Conf. Computer Vision, pp. 102-111,London, 1987.

1131 Z.R. Li and D.P. Zhang, "A Fingerprint Recognition System WithMicro-Computer." Proc. Sixth ICPR, pp. 939-941, Montreal, 1984.

1141 L. Coetzee and E.C. Botha, "Fingerprint Recognition in LowQuality Images." Pattern Recognition. vol. 26, no. 10, pp. 1,441-1.460. 1993.

1151 B. Miller. "Vital Signs of Identity." IEEE Spectrum. vol. 31. no. 2,PP. 22-30, 1994.

1161 A. Ranade and A Rosenfeld, "Point Pattern Matching by Relaxa-tion," Pattern Recognition, vol. 12, no. 2, pp. 269-275, 1993.

1171 A.R. Rao. A Taxonomy for Texture Description and Identilfication.New York: Springer-Verlag, 1990.

181 N. Ratha, S. Chen, and A.K. Jain, "Adaptive Flow OrientationBased Feature Extraction in Fingerprint Images." Pattern Recogni-tion, vol. 28, no. 11. pp. 1,657-1,672, 1995.

1191 A. Sherstinsky and R.W. Picard, "Restoration and Enhancementof. Fingerprint Images Using M-Latiice-A Novel Non-Linear Dy:namical Systemn," Proc. 12th ICPR-B, pp. 195-200, Jerusalem. 1994.

1201 D.B.G. Sherlock, D.M. Monro. and K. Millard, "Fingerprint En-hancement by Directional Fourier Filtering," lEE Proc. Vis. ImageSignal Processing, vol. 141, no. 2, pp. 87-94. 1994. ,

121] J.P.P. Starink and E. Backer, "Finding Point Correspondence Us-ing Simulated Annealing," Pattern Recognition, vol. 28, no. 2,pp. 231-240. 1995. -

1221 C. Stockman, S. Kopstein, and S. Benett, "Matching Images toModels for Registration and Object Detection via Clustering."IEEE Transactions on Partern Analysis and Machine Intelligence,vol. 4, no. 3, pp. 229-241, 1982.

1231 J. Ton and A.K. Jain. "Registering Landsat Images by PointMatching," IEEE Trans. Geoscience and Remote Sensing, vol. 27,no. 5, pp. 642-651, 1989.

1241 V.V. Vinod and S. Chose, "Point Matching Using AsymmetricNeural Networks," Pattern Recognition, vol. 26, no. 8, pp. 1,207-1,214, 1993.

1251 C.I. Watsorn and C.L. Wilson, NIST Special Database (4, FingerprintDatabase, National Institute of Standards and Technology. Mar.1992.

126) C.L. Wilson, G.T. Gandela, and C.I. Watson, "Neural-NetworkFingerprint Classification." ]J..'Artificial Neural Networks, vol. 1,no. 2, pp. 203-228, 1994.

1271 Q. Xiao and Z. Bian, "An Approach to Fingerprint Identificationby Using the Attributes of Feature Lines of Fingerprint," Proc.Seventh ICPR, pp. 663-665, Paris, 1986.

* ,


Anil Jain received a BTech degree in 1969from the Indian Institute of Technology, Kan-pur, India, and the MS and PhD degrees inelectrical engineering from the Ohio StateUniversity in 1970 and 1973, respectively. Hejoined the faculty of Michigan State Universityin 1974, where he currently holds the rank ofUniversity Distinguished Professor in the Dept.of Computer Science. Dr. Jain has made sig-nificant contributions and published a largenumber uof papers on the following topics: sta-

tistical pattern recognition, exploratory pattern analysis, Markov ran-dom fields, texture analysis, interpretation of range images, neuralnetworks, document image analysis, and 3D object recognition. Sev-eral of his papers have been reprinted in edited volumes on imageprocessing and pattern recognition. He received the best paper awardsin 1987 and 1991 and certificates for outstanding contributions in 1976,1979, and 1992 from the Pattern Recognition Society. He also receivedthe 1996 IEEE Transactions on Neural Networks Outstanding PaperAward. Dr. Jain was the editor-in-chief of the IEEE Transactions onPattern Analysis and Machine Intelligence (1990-1994), and he alsoserves as an associate editor for Pattern Recognition, Pattern Recog-nition Letters, IEEE Transactions on Neural Networks, Applied Intelli-gence, and the Journal of Mathematical Imaging and Vision. He is thecoauthor of Algorithms for Clustering Data (Prentice Hall, 1988), hasedited Real-Time Object Measurement and Classification (Springer-Verlag, 1988), and coedited Analysis and Interpretation of Range Im-ages (Springer-Verlag, 1989), Markov Random Fields (AcademicPress, 1992), Artificial Neural Networks and Pattern Recognition(Elsevier, 1993), and 3D Object Recognition (Elsevier, 1993). He is aFellow of the IEEE and IAPR.

Lin Hong received the BS and MS degrees incomputer science from Sichuan University,China,. in 1987 and 1990, respectively. He iscurrently a PhD student in the Dept. of Com-puter Science, Michigan State University. Hiscurrent research interests include pattern rec-ognition, image processing, biometrics, andcomputer graphics.

Ruud Bolle (S'82-M'84-F'96) received thebachelor's degree in analog electronics in 1977and the master's degree in electrical engi-neering in 1980, both from Delft University ofTechnology, Delft, The Netherlands. In 1983he received the master's degree in appliedmathematics and in 1984 the PhD in electricalengineering from Brown University, Provi-dence, Rhode Island. In 1984 he became aresearch staff member .in the Artificial Intelli-ence Grou Com uter S ience D% t t thI

g p, p c ep ., a eIBM Thomas J. Watson Research Center. In 1988 he became man-ager of the newly formed Exploratory Computer Vision Group, which ispart of IBM's digital library effort. His current research interests arevideo database indexing, video processing, and biometrics. Dr. Bolle isa Fellow of the IEEE. He is on the Advisory Council of IEEE Transac-tions on Pattern Analysis and Machine Intelligence, and he is associateeditor of Computer Vision and Image Understanding. He is the guesteditor of a special issue on computer vision applications for network-centric computers in Computer Vision and Image Understanding.

9I '

Application No. I1/231,353Docket No. 577832000200

IEEE TRANSACTIONS'ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. z.. ,v. ,,, ,,v ,,

On Combining ClassifiersJosef Kittler, Member, IEEE Computer Society, Mohamrnad Hatef, Robert P.W. Duin, and Jiri Matas

Abstract-We develop a common theoretical tramework'for combining classifiers which use distinct pattern representations andshow that many existing schemes can be considered as special cases of compound classification where all the patternrepresentations are used jointly to make, a decision. An experimental comparison of various classifier combination schemesdemonstrates that the combination rule developed under the most restrictive assumptions-the sum rule-outperforms otherclassifier combinations schemes. A sensitivity analysis o the various schemes to estimation errors is carried out to show that thisfinding can be justified theoretically.

Index Terms-Classification, classifier combination, error sensitivity.

1 INTRODUCTION

T HE ultimate goal of designing pattern recognition sys-tems is to achieve the best possible classification per-

formance for the task at hand. This objective traditionallyled to the development of different classification schemesfor any pattern recognition problem to be solved. The re-suits of an experimental assessment of the different designswould then be the basis for choosing one of the classifiersas a final solution to the problem. It had been observed insuch design studies, that although one of the designswould yield the best performance, the sets of patterns mis-classified-by the different classifiers would not necessarilyoverlap. This suggested that different classifier designs po-tentially offered complementary information about thepatterns to be classified which could be harnessed to im-prove the performance of the selected classifier....

These observations motivated the relatively recent inter-est in combining classifiers. The idea is not to rely on a sin-gle decision making scheme. Instead, all the designs, ortheir subset, are used for decision making by combiningtheir individual opinions to derive a consensus decision.'Various classifier combination schemes have been devisedand it has been experimentally demonstrated that some ofthem consistently outperform a single best classifier. How-ever, there is presently inadequate understanding why'some combination schemes are better than others and in,what circumstances.

The two main reasons for combining classifiers are effi-ciency and accuracy. To increase efficiency one can adoptmultistage combination rules whereby objects are classifiedby a simple classifier using a small set of cheap features in

* 1. K tiler and J. Matas are w th the Centre for Vs on, Speech, and S gnalProcess ng, School of Electron c Eng neer ng, Informat on Technology, andMathemat cs, Un vers ty of Surrey, Gu ldford GU2 5XH, Un led K ng-dom. E-ma 1: j.k [email protected].

* M. Hatef s w th ERA Technology. Ltd., Cleeve Road, Leatherhead KT22.7SA, Un ted K ngdom. E-ma I: [email protected].

* R.P.W. Du n s w th the Department of Appl ed Phys cs, Delft Un vers tyof Technology, Lorentzweg 1, 2628 C] Delft, The Netherlands.E-ma 1: [email protected].

Manuscr pl rece ved 17 June 1996; rev sed 16 Jan. 1998. Recommended for accep-tance by 1.1. Hull.For nfornal on on obta n ng repr nis of th s art cle, please send e-ma I to:tpam @compuler.org, and reference IEEECS Log Number 106327...

combination with a reject option. For the more difficult ob-jects more complex procedures, possibly based on differentfeatures, are used (sequential or pipelined [17], [7], or hier-archical [24], [16]). Other studies in the gradual reduction ofthe set of possible classes are [8], [6], [14], [21]. The combi-nation of ensembles of neural networks (based on differentinitialisations), has been studied in the neural network lit-erature, e.g., [11], [4], 15], [10], [15], [18].

An important issue in combining classifiers is that this isparticularly useful if they are different, see [1). This can beachieved by using different feature sets [23], [13) as well asby different training sets, randomly selected 112], [22] orbased on a cluster analysis [3]. A possible application of amultistage classifier is that it may stabilize the training ofclassifiers based on a small sample size, e.g.; by the use ofbootstrapping [27], [19]. Variance reduction is studied in[30], [31] in the context of a multiple discriminant functionclassifier and in [35] for multiple probabilistic classifiers.Classifier combination strategies may reflect the local com-petence of individual experts as exemplified in [32) or thetraining process may aim to encourage some experts toachieve local decision making superiority as in the boosting'method of Freund [28) and Shapire [29].

An interesting issue in the research concerning classifierensembles is the way they are combined. If only labels areavailable a majority vote [14], 19] is used. Sometimes the usecan be made of a label ranking [2), [13]. If continuous out-puts like posteriori probabilities are supplied, an average orsome other linear combination have been.suggested [11),[23], [25], [33); It depends on the nature of the input classi-fiers and the feature space whether this can be theoreticallyjustified. An' interesting study on these possibilities is givenin [10), [26], [34]. If the classifier outputs are interpreted asfuzzy membership values, belief values or evidence, fuzzyrules [4), [5], belief functions and Dempster-Shafer tech-niques [9], [18), [20], [23] are used. Finally it is possible totrain the output classifier separately using the outputs ofthe input classifiers as new features [15], [22], [36).

From the point of view of their analysis, there aire basicallytwo classifier combination scenarios. In the first scenario, allthe 'classifiers use the same representation of the input pat-tern. A typical example of this category is a set of k-nearest

0162-8828/98/510.000 1998 IEEE

,

KITTLER ET AL.: ON COMBINING CLASSIFIERS

neighbor classifiers, each using the same measurement vec-tor, but. different classifier parameters (number of nearestneighbors k, or distance metrics used for determining thenearest neighbors). Another example is a set of designs basedon a neural network classifier of fixed architecture but havingdistinct sets of weights which have been obtained by meansof different training strategies. In this case, each classifier, fora given input pattern, can be considered to produce an esti-mate of the same a posteriori class probability.

In the second scenario, each classifier uses its own repre-sentation of.the input pattern. In other words, the meas-urements extracted from the pattern are unique to each-clas-sifier. An important application of combining classifiers inthis scenario is the possibility to integrate physically differenttypes of measurements/features. In this case, it is no longerpossible to consider the computed a posteriori probabilitiesto be estimates of the same functional value, as the classifica-tion systems operate in different measurement spaces.

In this paper, we focus on classifier combination in thesecond scenario. We develop a common theoretical frame-work for classifier combination and show that many exist-ing schemes, can be considered as special cases of com--pound classification where all the representations are usedjointly to make a decision. We demonstrate thatunder dif-ferent assumptions and using different approximations wecan derive the commonly used classifier combinationschemes such as the product rule, sum rule, min rule, maxrule, median rule, and majority voting. The various classi-fier combilation ;schemes are then compared experimen-tally. A surprising outcome of the comparative study is thatthe combination rule developed under the most restrictiveassumptions-the sum rule-outperforms other classifiercombinations schemes. To explain this empirical finding,we investigate the sensitivity of various schemes to estima-tion errors. The sensitivity analysis shows that the sum ruleis most resilient to estimation errors.

In summary, the contribution of the paper is twofold.First of all, we provide a theoretical underpinning-of manyexisting classifier combination schemes for fusing the deci-sions of multiple experts, each employing,a distinct patternrepresentation. Furthermore, our analysis of the sensitivityof these schemes to estimation errors enhances the under-standing of their properties. As a byproduct, we also offer amethodological machinery which can be used for developingother classifier combination strategies and for predictingtheir behavior. However, it cannot be overemphasized thatthe problem of classifier combination is very complex andthat there are many issues begging explanation. These in-clude the effect of individual expert error distributions onthe choice of a combination strategy, explicit differentiationbetween decision ambiguity competence and confidence,and the relationship between dimensionality reduction andmultiple expert fusion, with its implicit dimensionality ex-pansion. Also, many practical decision making schemes arevery complex, of sequential kind, with special rules to han-dle rejects and exceptions and it is currently difficult to en-visage how the results of this paper could be made to bearon the design of such schemes. The theoretical frameworkand analysis presented is only a small step towards a con-siderably improved understanding of classifier combina-

tion which will be needed in order to harness the benefitsof multiple expert fusion to their full potential.

The paper is organized as follows. In Section 2, we for-mulate the classifier combination problem and introducethe necessary notation. In this section, we also derive thebasic classifier combination schemes: the product rule andthe sum rule. These two basic schemes are then developedinto other classifier combination strategies in Section 3. Thecombination rules derived- in Sections 2 and 3 are experi-mentally compared in Sections 4 and 5. Section 6 investi-gates the sensitivity of the basic classifier combination rulesto estimation errors. Finally, Section 7 summarizes the mainresults of the paper and offers concluding remarks.

2 THEORETICAL FRAMEWORK

Consider a pattern recognition problem where pattern Z is

to be assigned to one of the mn possible classes (01,..., em).Let us.assume that we have R classifiers each representingthe given pattern by a distinct measurement vector. Denote

the measurement vector used by the th classifier by x. .Inthe measurement space each class sk is modeled by the

probability density function. p(x Ik) and its a priori prob-

ability of occurrence is denoted P(wk). We shall consider

the models to be mutually exclusive which means that onlyone model can be associated with each pattern.

Now, according to the Bayesian theory, given measure-

ments x, = 1, ..., R, the pattern, Z, should be assigned toclass ca provided the a posteriori probability of that inter-

pretation is maximum, i.e.

ass gn .Z - a

P(Wix, ..., xR)=maxP(wkIxp...,xR)

The Bayesian decision rule (1) states that in order toutilize all the available information correctly to reach a de-cision, it is essential to compute the probabilities of thevarious hypotheses by considering all the measurementssimultaneously. This is, of course, a correct statement of theclassification problem but it may not be a practicableproposition. The computation of the a posteriori probabilityfunctions would depend on the knowledge of high-ordermeasurement statistics described in terms of joint probabil-

ity density- functions p(xl,-.. . xR.k) which would be diffi-

cult to infer. We shall therefore attempt to simplify theabove rule and express it in terms of decision support com-putations performed by the individual classifiers, each ex-

ploiting only the information conveyed by vector x. Weshall see that this will not only make rule (1) computation-ally manageable, but also it will lead to combination ruleswhich are commonly used in practice., Moreover, this ap-proach will provide a scope for the development of a rangeof efficient classifier combination strategies.

We shall commence from rule .(1) and consider how itcan be expressed under certain assumptions. Let us rewrite

the a posteriori probability P(wkx ... , xR)' using the Bayes

theorem. We have

I ' I

227'

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE'INTELLIGENCE, VOL. 20, NO. 3, MARCH 1998

P(W, x1,..., xR)- p (= 1 .... .l' )P(Co) (2)p(x,... ,xR)

where p(x1, ..... , xa) is the unconditional me'asurement jointprobability density. The latter can be expressed in terms ofthe conditional measurement distributions as

m

p(x 1,..., XR) p(x,... R. )P( )

and therefore, in the following, we can concentrate only onthe numerator terms of (2).

2.1 Product Rule

As already pointed out, p(x 1,..., XRt(ck) represents the joint

probability distribution of the measurements extracted bythe classifiers. Let us assume that the representations usedare conditionally statistically independent. The use of dif-ferent representations may be a probable cause of such in-dependence in special cases. We will investigate the conse-quences of.this assumption and write

p(X1,...,XRI ok) = l P(X Ialk)

where p(x I qy) is the measurement process model of the threpresentation. Substituting from (4) and (3) into (2) wefind

P( x,... ) P( ) 1p(x , )

and using (5) in (1), we obtain the decision rule

ass gn Z - oi

P(W,) x (o )= max P(Wk) p(x IK)=1 k=I =1

or in terms of the a posteriori probabilities yieldedrespective classifiers

ass gn Z -- o f

P-(R-)( w)f P(wjIx) = max P-(R-i)(w)kI P(oklx )(7)=1 =1

The decision rule (7) quantifies the likelihood of a hypothe-sis by combining the a posteriori pr6babilities generated bythe individual classifiers by means of a product rule. It is,effectively a severe rule of fusing the classifier outputs as itis sufficient for a single recognition engine to inhibit a par-'ticular interpretation by outputting a close to zero prob-ability for it. As we shall see below,. this has a rather unde-sirable implication on the decision rule combination as allthe classifiers, in the worst case, will have to provide theirrespective opinions for a hypothesized class identity to beaccepted or rejected.

2.2 Sum RuleLet us consider decision rule (7) in more detail. In someapplications it may be appropriate further to assume thatthe a posteriori probabilities computed by the respective

classifiers will not deviate dramatically from the priorprobabilities. This is a rather strong assumption but it maybe readily satisfied when the available observational dis-criminatory information is highly ambiguous due to highlevels of noise. In such a situation we can assume that the aposteriori probabilities can be expressed as

P(Okl )= P(Wk)(l + 6k )

where bk satisfies 6 k << 1. Substituting (8) for the a poste-riori probabilities in (7), we find

P (R-I)(1-I P(WI ) - P(-)-(I + )-1 =1 '

(9)

If we expand the product and neglect any terms of secondand higher order, we can approximate the right-hand sideof (9) as

(10)P(, ) (1 + k)= P(k ,) +P(wk,) S ,=1 =1

Substituting (10) and (8) into (7), we obtain a sum decisionrule

ass gn ZR

=1

max (1 - R)P(k) + P()lx)k=1

=1

(11)

2.3 CommentsBefore proceeding, in the next section, to develop specificclassifier combination strategies based on decision rules (7)and (11), let us pause to elaborate on the assumptions madeto derive the product and sumr rules. We concede that theconditional independence assumption may be deemed tobe unrealistic in many situations. However, three importantpoints should be borne in mind before dismissing the re-suits of the rest of the paper:r

* For some applications, the conditional independenceassumption will hold.

* For many applications, this assumption will providean adequate and workable approximation of the real-ity which may be more complex. One could draw aparallel here between the Gaussian assumption fre-quently made even in situations where the class dis-tributions patently do not obey the exponential lawbut still this simplification yields acceptable results.

* Finally, and perhaps most importantly, we shall see inthe next section that all the derived classifier combi-nation schemes based on this assumption are rou-tinely used in practice. The analysis presented in thepaper therefore provides a plausible theoretical un-derpinning of these combination rules and , therebydraws attention to the underlying assumptions behindthese schemes which the users may not be aware of.

As far as the sum rule is concerned, the assumption thatthe posterior class probabilities do not deviate greatly fromthe priors will be unrealistic in most applications. When


observations x, =, 1, ..., R on a pattern convey significadiscriminatory information the sum approximation of tlproduct in (10) will introduce gross approximation errolHowever, we shall show, in Section 6 that the injectionthese errors will be compensated by a relatively low sen:tivity of the approximation to estimation errors. '

3 CLASSIFIER COMBINATION STRATEGIES

The decision rules (7) and (11) constitute the basic schemfor classifier combination. Interestingly, many commonused classifier combination strategies can be developfrom these rules by noting that

ass gn Z'. - iR mmin P( x)= max min P(k x )= k=l =1.

(17)

S3.3 Median RuleNote that under the equal prior assumption, the sum rule in(11) can be viewed to be computing the average a posterioriprobability for each class over all the classifier outputs, i.e.,

ass gn Z - 4

SP(ox ) = ma P(wk=1 =1

(18)

fi k k) - x 1 RR

RR-< P(ox ) max P(wkx) (12)

=1

The relationship (12) suggests that the product and sumcombination rules can be approximated by the above upperor lower bounds, as appropriate. Furthermore, the harden-

ing of the a posteriori probabilities P(coklx ) to produce bi-

nary valued functions Ak as

Ak = if P( = m1x P@(x

0 otherwise

Thus, the rule assigns a pattern to that class the average aposteriori probability of which is maximum. If any of theclassifiers outputs an a posteriori probability for some classwhich is an outlier, it will affect the average and this in turncould lead to an.incorrect decision. It is well known that arobust estimate of the mean is the median. It could there-fore be more appropriate to base the combined decision onthe median of the a posteriori probabilities. This then leadsto the following rule:

ass gn Z'-w i ffR m R

med P(.x ) = max med P(COwkx ).1 k=1 =1(13)

results in combining decision outcomes rather than com-bining a posteriori probabilities. These approximations leadto the following rules:

3.1 Max RuleStarting from (11) and approximating the sum by themaximum of the posterior probabilities, we obtain

ass gn Z i

(1- R)P(wm) + R max P(w x ) =m a "

max (1- R)P(wk) + R max P(ak lx ) (

which under the assumption of equal priors simplifies to

ass gn Z :- W

max P(w Ix max maxP(W x) (15)=1 k=1 =1

3.2 Min Rule.Starting from (7) and bounding the product of posteriorprobabilities from above we obtain

ass gn Z -+ c i

P< <a )min P@ ix) =

max P-(-1)() mn P(wkIx )

k=) =1

which under the assumption of equal priors simplifies to

(19)

3.4 Majority Vote RuleStarting from (11) under the assumption of equal priors andby hardening the probabilities according to (13), we find

ass gn Z -* (1)

R' m R

ml J k=1=1

(20)'

Note that for each class 0 k the sum on the right handside of (20) simply counts the votes received for this hy-pothesis from the individual classifiers. The class whichreceives the largest number of votes is then selected as theconsensus (majority) decision.

All the above combination schemes and their relation-ships are represented in Fig. 5.

4 EXPERIMENTAL COMPARISON OF CLASSIFIERCOMBINATION RULES: IDENTITY VERIFICATION

The first experiment is concerned with the problem of per-sonal identity verification. Three different sensing modali-ties of biometric information are used.to check the claimedidentity of an individual: frontal face, face profile, andvoice. The verification methods using these biometricsensing modalities have been developed as part of theEuropean Union project in Advance Communication Tech-nologies and Services M2VTS as described in [411, [441, 143].The design of the verification modules and their perform-ance testing gas been carried out using the M2VTS database

142] made up of about eight seconds of speech and videodata for 37 clients taken five times (five shots) over a periodof one month. The image resolution is 286 x 350 pixels.

229

230 IEEE TRANSACTIONS ON PA-TE

4.1 Frontal FaceThe face verification system used in the experiments is de-scribed in detail in [41]. It is based on robust correlation of afrontal face image of the client and the stored face templatecorresponding to the claimed identity. A search for the op-timum correlation is performed in the space of all validgeometric and photometric transformations of the inputimage to obtain the best possible match with respect to thetemplate. The geometric transformation includes transla-tion, rotation and scaling, whereas the photometric trans-formation corrects for a change of the mean level of illumi-.nation. The search technique for the optimal transformationparameters is based on random exponential perturbations.Accordingly, at each stage the transformation between thetest and reference images is perturbed by: a random vectordrawn from an exponential distribution and' the change isaccepted if it leads to an improvement of a matching crite-rion. Computational efficiency is achieved by means ofrandom sampling based on Sobel sequences which, allowfaster convergence as compared to uniform sampling.

The score function adopted rewards a large overlap be-tween the transformed face image and the template, andthe similarity, of the intensity distributions of the two im-ages. The degree of similarity is measured with a robustkernel. This ensures that gross errors due to, for instance,hair style changes do not swamp the cumulative error be-tween the matched images. In other words, the matching isbenevolent, aiming to find as large areas of theface as pos-sible supporting a close agreement between the respectivegray-level profiles of the two images. The gross errors willbe reflected in a reduced overlap between the two frameswhich is taken into account in the overall matching crite-rion. The system is trained very easily by means of storingone or more templates for each client. Each reference imageis segmented to create a face mask which excludes thebackground and the torso as these are likely to change overtime. The testing is performed on an independent test datacomposed of 37 clients and 37 x 36 impostors.

4.2 Face Profile,The verification approach involves a comparison of a can-didate profile with the template profile of the claimedidentity. The candidate image profile is extracted from theface profile images by means of color-based segmentation.The similarity of the two profiles is measured using theChamfer distance computed sequentially [44]. The effi-ciency of the verification process is aided by precomputinga distance map for each reference profile. The map storesthe distance of each pixel in the face profile image to thenearest point on the profile. As the candidate profile can besubject to translation, rotation and scaling, the objective of.the matching stage is to compensate for such geometrictransformation. The parameters of the compensating trans-formation are determined by minimizing the chamfer dis-tance between the template and the transformed candidateprofile. The optimization is carried out using a simplex al-gorithm which requires only the distance function evalua-tion and no derivatives. The convergence of the simplexalgorithm to a local minimum is prevented by a careful ini-tialization of the transformation parameters. The translation

RN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 20, NO. 3, MARCH 1998

parameters are estimated by comparing the position of thenose tip in the two matched profile. The scale factor is de-rived from the comparison of the profile heights and therotation is initially set to zero. Once the optimal set of trans-formation parameters is determined, the user is accepted orrejected depending on the relationship of the minimalchamfer distance to a prespecified threshold.

The system is trained on the first three shots. One profileper client per shot is stored in the training set. From thethree profiles for each-client a single reference profile is se-lected by pairwise comparison of the profile images. Theprofile yielding the lowest matching distance to the othertwo images is considered as the best representative of thetriplet. The trained system is tested on Shot 4 profiles' Asthere are 37 users in the M2VTS database the testing in-volves 37 correct authentication matches and 37 x 36 im-poster tests. The acceptance threshold is selected from theReceiver Operating Characteristic so as to produce equalerror rate (false rejection and false acceptance).

4.3 VoiceThe personal identity verification based on voice.employs atext dependent approach described in [43). It is assumedthat the audio input signal representing the uttered se-quence of digits from zero to nine can be segmented intoindividual words. Both, the segmentation of the speechdata and the claimed identity verification is accomplishedusing speech and speaker recognition methods based onHidden Markov Models. The audio signal is first trans-formed into a multivariate time series of linear predictivecepstral coefficients: During training, digit HMMs aretrained using segmented speech data from three shots ofthe M2VTS database. The digit models have the samestructure, with the number of states being digit specific.The models allocate one state per phoneme and one stateper transition between phonemes. A single Gaussian mix-ture is used to model the distribution of the cepstral coeffi-cient vectors within one state.

STwo models are acquired for each digit: the client model,and the world model. The latter, which is common to allusers, captures the variability of the uttered sound in alarge database. The verification of a claimei identity isbased on a score computed as the sum over the individualdigits of the log likelihood ratio of the claimed model andthe world model normalized by the number of cepstral co-efficient frames. The score is mapped on the interval zero-one using a sigmoid function. The performance is assessedusing an independent test set.

4.4 Experimental ResultsThe equal error rates obtained using the individual sensingmodalities are shown in Table 1. The table shows that thelowest rate of 1.4 percent was achieved using voice basedverification. The face profile verification' produced an equalerror rate of 8.5 percent whereas the frontal face methodyielded 12.2 percent. The soft decisions output by the threeverification systems were then combined using the variousclassifier combination strategies discussed in Section 3.

The validity of the conditional independence assump-tion was tested by computing the average within class corre-


TABLE 1EQUAL ERROR RATES

soo

100

Too

soo

soo

1000

method EER (%)

frontal 12.2profile 8.5speech 1.4

sum 0.7product 1.4maximum 12.2median 1.2minimum 4.5

KEr

100 200 300 400 500 600 700 800 900 1000

Fig. 1. Correlation of face profile; frontal face, and speech data.

lation matrix for the data used in decision making. Sincethe overall dimensionality of the data exceeds tens of thou-sands, it is impossible to present a full view of the correla-tions between the measurements of the respective modali-ties. However, by adopting a visual representation of thecorrelation matrix, we will be able to look at the correla-tions at least in a representative subspace of this highly di-

mensional feature space. This subspace was created bytaking as features 500 samples of the face image gray levelstaken at prespecified spatial locations..Each profile imagewas represented by 60 sample points evenly distributedalong the profile. The sampling is registered with respect tothe tip of the nose and the sampling interval normalized bythe nose length. The profile landmarks needed for the reg-istratiori and normalization can be easily detected. For thespeech data, we took the first 100 frames from each of thefirst five cepstral coefficients. The utterances for each clientwere first time warped using a client specific template. Thiscreated a client representation subspace of 1,060 dimen-sions. In particular, the face profile variables occupy thefirst 60 dimensions, followed by 500 frontal face imagesamples, and finally 5 x 100 speech measurements. The av-erage within class correlation matrix was computed by re-moving the class conditional mean of each variable. Theresulting vectors of deviations from the means were used tocompute the elements of the average within class covari-ance matrix. These were then normalized by dividing eachjih element by the product of stanidard deviations of the th

i

i

7

BEST AVAILABLE OPY

and jth component of the vector of deviations. This nor-malisation process produced average within class correla-tions taking values in the interval [-1, 1]. For display pur-poses, we have taken the absolute value of these correlationcoefficients. The result of this representation of variablecorrelations is a matrix with all elements on the diagonalequal to unity (displayed as gray level 255) and thestrength of correlation between one and zero mappedonto the gray-level scale 255/to 0. The correlation matrix isshown in Fig. 1;

The correlation matrix exhibits a block diagonal struc-ture, which suggests that the observations generated byeach modality are class conditionally dependent. The cor-Srelation are particularly strong between the features of theface profiles and similarly between those of the speechutterances. They are weaker for the features of the faceimage. Owing to the random spatial sampling of the faceimage, the spatial ordering of the successive features isdestroyed and consequently the correlation matrix blockcorresponding to the facial data has a random structure(with the exception of the diagonal elements). Note that

the correlations between features from different modali-ties are considerably weaker than within modality corre-lations. This applies in particular to the correlations be-tween the frontal face and the other two modalities. Thereis a small subset of the face profile variables for which the

correlations are not insignificant but on the whole theconditional independence assumption may be consideredto hold.

Next, the three biometric modalities were combined us-ing the fusion strategies discussed in Section 3. The resultspresented in Table 1 show the benefits of classifier combi-

nation. It is interesting to note that the sum rule outper-

formed all the other combination strategies and also theindividually best expert.

5 EXPERIMENTAL COMPARISON OF CLASSIFIER

COMBINATION RULES: HANDWRITTEN DIGITRECOGNITION

As a second domain to assess the classifier combinationstrategies, we used the problem of handwritten characterrecognition. The task is to recognize totally unconstrainedhandwritten numerals. Samples are images of isolatednumeric chaiacters takeh from addresses on the letter en-velopes provides by the U.S Postal Service.

The database used is the CEDAR-CDROM producedby the Center of Excellence for Document Analysis andRecognition, at the State University of.New York, Buffalo.Images are scanned from dead-letter envelopes providedby the U.S. Postal Service. We used the BR and BS sets ofthe database that consist of bitonal isolated images ofnumeric characters. BR set contains 18,468 samples and isused as a training set while BS set (2,213 samples) servedas a test set.

Four types of classifiers are first applied to perform theclassification individually. We used structural [38], Gaus-sian, Neural Network, and Hidden Markov Model classifi-ers [40].

ii i

i ?.

1

i

III

I

tt

I,

~aS~t~j~bi~~e4~P~W~I~E~d~~B~FI~1~Z1

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 20, NO. 3, MARCH 1998

5.1 Character RepresentationFour different representations are used as follows:

1) Pixel-level representation: in the Gaussian classifiericase the bitonal image of each numeric character isscaled into 10 x 10 gray-level image. The character isthus represented by a 100-dimensional vector inwhich each dimension is the gray level of the corre-sponding pixel.

2) Complex object representation: this is used in the caseof the structural classifier. The bitonal image is firstskeletonized using some thinning process. Theskeleton of the character is then decomposed into anumber of primitives. Each primitive being either aline, curve or a loop is parameterized using a numberof unary measurements such as the size, direction, etc.In addition, a number of binary measurements areextracted to describe the geometrical relations be-tween each primitive and its neighbors. A more de-tailed description of this representation is presentedin [39].

3) In the HMM classifier, the 2D image isrepresented astwo ID signals by using the horizontal and verticalprofiles. The profile consists of the binary pattern ofthe image across the horizontal/vertical line. Thispattern is quantized into 16 vectors in the codebook.Each pattern is therefore given the index of the closestvector in the codebook. Further, the center of gravityof each line in the profile is calculated and also quan-tized (to 13 levels). The feature space thus consists oftwo indices one for the pixel pattern and the. other forthe center of gravity. More details on this representa-tion can be found in [40].

4) The pixel representation in Item I is used as a startingpoint to derive a distinct character description by thehidden layer of the neural network employed as oneof the classifiers.

5.2 Classification5.2.1 Structural Classifier.

Handwritten characters have natural structures as they aregenerally composed of number of smaller, elements withcertain topological relations. To recognize a character, weneed to identify its 'basic .primitives and the particularstructural relations between them.

The. binary image is first skeletonized, then decom-posed into number of primitives where junctions and re-flection points serve as breaking points. Both symbolicand numeric attributes are used to describe the structureof the character. Firstly, primitives are categorized intoone of three types using a discretizing criterion: zero-line,one-curve, or two-loop. The connectivity between theprimitives is encoded to reflect the topological structureof the character. The character code consists of the code ofeach primitive which in turn consists of the type of theprimitive, the number of the neighbors on the first endpoint and their types, and the number of the neighbors onthe second endpoint and their types. For example thecode:

(1, 200, 0), (2, 10, 10), (0, 210, 12), (0, 210, 0)

Fig. 2. The prototype tree structure.

represents a character consisting of four primitives. The.first primitive is a curve (1) connected to two primitives inthe first end point, both of them being lines (200). The otherendpoint is not connected to any primitive (0).,

Numeric information is also used to characterize unaryattributes of primitives and relations-and binary relationsbetween primitives. The length of the primitive, its direc-tion, the degree of curvature are some of the unary meas-urements used. Example of the binary measurements usedare the direction of the line connecting the centers of theprimitive and its neighbor as well as the direction of theline connecting the terminal point of the primitive and thecenter.of the neighbor. Each class is represented by one ormore prototypes. In our scheme prptotypes are generatedfrom the training samples. Samples of 'each class are di-vided into small groups by means of levels of clustering.The first is to group all samples with, the same number ofprimitives in a cluster.'Each cluster is called Np-Group andis further divided according to the types of the primitives.For example, samples that consist of a curve and two linesare grouped together. Each such group or cluster is calledtype-Group and further divided into a number of clusterseach containing samples that have the same structuralcode. Cluster in this level is called code-Group. Finally,each code-Group is further divided' using the dynamicclustering algorithm [37) where each of the clusters pro-duced is called dist-Group. The mean, variance and theactual range around the mean are calculated for each of theunary and binary measurements to characterize the par-ticular cluster. The prototypes of all classes are saved in thismultilevel tree structure (Fig. 2).

5.2.2 The Classification Scheme

An unknown sample is tested first at the root of the proto-type tree to decide the right Np-group. In the next level, itis checked to select the right type-Group and eventually itreaches the appropriate code-Group. If no code-Group isfound, the class of the sample is reported as unknown. Oth-erwise, the sample will be checked against all prototypes in


the code-Group to find the closest candidate(s) to the sam-ple. First, the probabilistic relaxation algorithm [38) is usedto find the correspondence between the primitives in thesample and those in the prototype. Then, a distance meas-ure is used to quantify the similarity between the sampleand each candidate. It is pertinent to point out that ameaningful measure can be defined because each sample iscompared only to prototypes that have the same number ofprimitives as well as connectivity (adjacency matrix). Thismeans that they have the same number of dimensions.Moreover, after finding the correspondence between theprimitives in the sample and the prototype through thematching process, the attribute vectors associated with thesample and prototypes respectively can be considered asbelonging to the same space. This facilitates the task offinding a metric for measuring the distance between them.We used the Euclidean distance first, but due to the factthat this distance does not take into account second-orderstatistics of each measurement, the results were not satis-factory. On the other hand, using distance measure thatexploits second order statistics, such as the Mahalanobisdistance, requires a large number of samples in each cluster.Due to the large variability in the structure of the hand-written characters there are prototypes that contain only afew samples in the training set which makes the estimate ofthese statistics unsatisfactory. Consequently, we chose amodified Euclidean distance whereby the difference in eachdimension between the measurement vector in the sampleand that in prototype is penalized if it exceeds a certainvalue. The value is chosen to be some weight multiplied bythe standard deviation of that. particular measurement. Themodified distance therefore is:

dine = YFx - m ) 2 (21)

where:

F(y) = cII if Iy>Oa(22)1y otherwise (22)

and m is the mean of the th feature while a is its standarddeviation. O is the threshold constant. K is a penalizingweight. The values of 0 and Kare selected experimentally.

An estimate of the a priori probability is then computedas follows:

e- da

me2 P(. "

P( Ix) = 2 () (23)

P k x-dme P(k )

where P(wo) is the a posteriori probability estimated fromthe number of samples in each cluster that generated theprototype. The sample is then assigned the class that hasthe maximum P(41 x). Note that when no prototypematches the sample structure it is assigned zero a posterioriprobability for all classes.

5.2.3 Gaussian ClassifierThe classes in the feature space are assumed to possess anormal distribution:

p(x2 )= (2)I e-(x-m )TE'(x-m )

where m is the mean vector and E is the covariance matrixof class . They are estimated in the training phase from thetraining data set. d is the number of dimensions in the fea-ture space.

The a posteriori probability is then calculated:

p(xl- )P(w)P(a jx) = P( k)P(wk) (25)

5.2.4 Hidden Markov Models ClassifierHidden Markov Models (HMMs), a popular method of sta-tistical representation in speech processing is based on therepresentation of an object as a random process that gener-ates a sequence of states. The model consists of a number ofstates with their probabilities as well as probabilities asso-ciated with the transition from one state to another.

The character in this classifier is scanned vertically andhorizontally to generate thecorresponding vertical andhorizontal profiles. The vertical profile consists of the rowsin.the binary image while the horizontal profile consists ofthe columns. Each state represents ,a pattern of binary pixelsin each line along the profile. The number of possible pat-

terns (states) can be numerous. For example, in a 32 x 32, 32

binary image there are 2 possible combinations. To reduce

the number of possible states, the training patterns areclustered and the centroid of each cluster serves as a refer-ence vector in a code book (Vector Quantization). An un-known sample is compared to each reference in the code-book and assigned the'index of the closest one. The code-book is generated using the k-means clustering algorithmwith k = 16, resulting in a 16-vector codebook. In the clus-tering process some distance measure is required to deter-mine how close a sample is to its cluster in order to decidethat it should be kept in the cluster or moved to another(closer) one. Hamming distance is a natural choice whendealing with binary vectors. The Hamming distance,'how-ever, is known to be sensitive to the shift between two bi-nary patterns. Slight shifts are inevitable in a problem likecharacter recognition. A Shift Invariant Hamming distance(the minimum Hamming Distance between two discretevectors when they are allowed to slide on each other) isused. The same advantageous property of shift invariancecan be undesirable in some cases. For example, the profileof letter "q" 'and "d" would appear to have the same code-book index: Therefore, another measure is used to distin-guish between such instances. The center of gravity of lineis calculated and then subtracted from the running averageof the last three lines. The relative center of gravity is inturn quantized to 13 levels. The state representation is thusreduced to a pair of numbers-one represents the pixelpattern index and the other is the relative center of gravity.

The discrete hidden Markov models are generated usingthe Baum;Welch reestimation procedure while a scoringmechanism based on the Viterbi algorithm is used in thetest phase. The scoring result reflects the likelihood of thesample to be generated by the class model. These score val-ues are used as the soft-level assignment of the classifier (asposteriori probabilities estimates).

233

(24)


TABLE 2THE CLASSIFICATION RATE FOR EACH CLASSIFIER

Individual classifier Classification rate %Structural: 90.85Gaussian: 93.93

Neural Net: 93.2HMM: 94.77

TABLE 3THE CLASSIFICATION RATE USINGDIFFERENT COMBINING SCHEMES

Combining rule Classification rate %

Majority Vote: 97.96Sum rule: 98.05Max rule: 93.93Min rule: 86.00

Product rule: 84.69Median rule: 98.19

5.2.5 Neural Network Classifier

Our next classifier is a feed forward neural network (Multi-layer Perceptron) trained as a pattern classifier. The mo-mentum Back-propagation algorithm is used to train thenetwork. The network consists of 100 nodes in the inputlayer (corresponding to the 100 dimensions in the featurespace), 25 nodes in the hidden layer, and 10 nodes in theoutput layer. Each node in the output layer is associatedwith one class and its output O, with [zero to one] range,reflects the response of the network to the correspondingclass <o. To facilitate a soft-level combination the responsesare normalized and used as estimates of the a posterioriprobability of the classes as

OP(w IX) = Ik ok

5.3 The Combination SchemeIn this expert fusion experiment, due to staff changes, wewere unable to compute the within class correlation matrixfor the different representations used. We can, therefore,only hope that the distinct representations used by the in-dividual experts satisfythe assumption of class conditionalindependence at least approximately. Six different combi-nation schemes are applied under the assumption of equalpriors and their results are compared. These schemes can bedivided into two groups according to the format of the in-dividual classifiers used by the combiner. Hard-level com-bination uses the output of the classifier after it is hard-thresholded (binarized). Soft-level combination on the otherhand uses the estimates of a posteriori probability of theclass by each classifier. The majority vote combiner is a rep-'resentative of the first category while the-five different op-erators are the soft-level combiners. Table 2 shows the resultsof classification of the individual classifiers while the re-sults of different combining schemes are shown in Table 3.

Note that the worst results are achieved when using theproduct rule which are similar to the performance of the m nrule. The results using these two rules are worse than anyof the individual classifiers as well, and the reason is that ifany of the classifiers reports the correct class a posteriori

d 6(1) 0->8 (2) 0->8

(6) 2->4 (7) 2->3

4 4q

/(3) 0->8

/(4) I->7 (5) I->7

(8) 2->6 (9) 3->5 (I0) 3->5

Y - 6L(11) 4->7 (12) 4->9 (13) 4->9 (14) 5->3 (15)6->4

(71(16) 7>

Fig. 3; Samples misclassifiedthe sum-rule combiner.

(19) 8->4 (20) 9->4

by the HMM classifier and corrected by

probability as zero, the output will be zero, and the correctclass cannot be identified. Therefore, the final result re-ported by the combiner in such cases is either a wrong class(worst case) or a reject (when all of the classes are assignedzero a posteriori probability). Another interesting outcomeof our experiments is that the Sum rule as well as the med anrule have the best classification results. The major ty voterule is very close in performance to the mean and medianrules. The Max rule is still better than any of the individualclassifiers, with the exception of the HMM classifier.

5.4 Analysis of the ResultsWe analysed the results in more detail to see how the per-formance of the system' improves through decision com-bination. HMM classifier that yields the best classificationrate among individual classifiers is chosen as a reference.

Twenty examples of the samples misclassified by theHMM classifier and corrected by the sum-rule combiner areshown in Fig. 3. The numbers below each character repre-sent the true class and that assigned by the HMM classifier,respectively. Although the HMM classifier scored quitewell in the overall classification, it seemed to have failed toclassify samples that otherwise look easy to recognize.

Table 4 contains the coirresponding sampleswith the aposteriori probabilities estimated by each classifier. Thetable shows a clear difference in the values assigned tosome of the samples by different classifiers. While one ofthe classifiers is 100 percent sure about the classification ofthe sample (the probability estimate is 1.0), the HMM clas-sifier is 100 percent sure that it is not the true class (its esti-mate is zero). Note that 66 of the 107 of the misclassifiedsamples are corrected by the simple sum rule combiner.

An important requirement for a combiner that uses theoutput of the individual classifiers is that the classifiers

44st, oe 5


TABLE 4SAMPLES MISCLASSIFIED BY HMM CLASSIFIER

True class HMM decision Structural Neural Net. Gaussian HMM1 0 8 0.95 0.99 1:00 0.002 10 8 0.71 0.05 1.00 0.00

3 0 8 1.00 0.56 0.04 0.004 1 7 0.17 1.00 1.00 0.005 1 7 0.58 1.00 1.00 0.126 2 4 0.1 0.99 1.00 0.007 2 6 1.00 0.99 1.00 0.258 2 3 0.96 0.00 1.00 0.00

9 3 5 0.73 '0.95 1.00 0.0010 3 5 0.1 1.00 1.00 0.0011 4 7 1.00 0.91 1.00 0.3912 4 9 0.1 1.00 1.00 .0.00

13 4 9 1.00 1.00 1.00 0.1514 5 3 0.71 0.86 0.99 0.00

15 6 4 0.97 1.00 1.00 0.0016 7 3 1.00 0.92 0.00 0.00

17 7 4 0.1 0.97 1.00 0.0018 7 9 0.70 0.74 0.97 0.0019 8 4 0.75 0.59 ' 1.00 0.1720 9 4 0.98 0.98 1.00 0.00

True class, class ass gned by the HMM class f er and the a poster or probab It es est mated by each class f er.

should not be strongly correlated in their "misclassification."That is, classifiers should not agree with each other whenthey misclassify a sample, or at least they should not assignthe same incorrect class to asample. This requirement canbe satisfied to a certain extent by

1) using different representations for the object (differentfeature sets) and

2) using a different classification principle for each ofthe individual classifiers.

Using different representations (feature sets) leads, in manycases, to a reduction in the correlation between the outputs of.individual classifiers, since there is almost always less corre-lation between the input vectors using different representa-tions than when using the same set of features. Different clas-sifiers usually use different assumptions about the structureof the data and the stochastic model that generates it. Thisleads: to a different estimate of the a posteriori probabilitiesespecially around the Bayes decision boundaries.

It is also. pertinent to look at the samples that are misclas-sified by the combiner to see whether there was full correla-tion between all the classifiers in their decision. Thirty sam-ples out of the 43 misclassified samples are correctly classi-fied by at least one classifier. Fig. 4 displays some of the mis-classified samples by the sum-rule combiner. In Fig. 4a, thesamples are not recognized by any of the individual classifi-ers. In Figs. 4b, 4c, 4d, and 4e, samples are correctly classifiedby the classifier indicated below each sample.

6 ERROR SENSITIVITY

A somewhat surprising outcome of the experimental com-parison of the classifier combination rules reported in Sec-tions 4 and 5 is that the sum rule (11), which has been de-veloped under the strongest assumptions, namely, those of

* conditional independence of the respective represen-tations used by the individual classifiers and

* classes being highly ambiguous (observations en-hance the a priori class probabilities only slightly)

appear to produce the most reliable decisions. In this sec-tion, we shall investigate the sensitivity of the product rule(7) and the sum rule (11) to estimation errors. We shallshow that the sum rule is much less affected by estimationerrors. This theoretically established behavior is consistentwith the experimental findings.

In the developments in Sections 2 and 3, we assumed

that the a posteriori class probabilities P(w; I x), in terms ofwhich'the various classifier combination rules are defined,are computed correctly. In fact, each classifier will produceonly an estimate of this probability, which we shall denote

(wil x ). The estimate deviates from the true probability by

error e, i.e.,

(27)

It is these estimated probabilities that enter the classifiercombination rules rather than the true probabilities:

Let us now consider the effect of the estimation errors onthe classifier combination rules. Substituting (27) into (7)we have

ass gn Z ---

p-(R i) IP(wix ) x+ e =

max P'('"frk)! lP((ckox ) + ek J (28)

P~il x )P- P(wilx ) +ei

236 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 20, NO. 3, MARCH 1998

0 ==> 8

/ /.I => 4 (struct+neur)

A. .AI ==> 2 (ncur+hmm)

3 ==> 5(gaus)r

3 ==> 5(gaus )

3 ==> 2 (hmm)

9 ==> 7 (struct+hmm)

V F'0 ==> unknown (ncur only)

.m~m

JUNI

9 ==> 5 (gaus)

4 => 8 (hmm)

0 ==> 8 (strucI+neur)

7 ==> 4 (neur only)

8 a8 ==-> 0 (gaus+struct)

(E)

5 =_> 8 (hmun)

Combin mis= 41 / 2011

stmct = 16

gaus = 5

correct by indiv = 30

neurhmm

=8

= II

Fig. 4. Samples misclassified by the sum-rule combiner. (a) Samples not classified correctly by any individual classifier. (b) Samples classified

correctly by the structural classifier. (c) By the Neural Network classifier. (d) By the Gaussian classifier. (e) By the HMM classifier.

Under the assumption that ek <<strong and may not represent the

further assuming that P(wklx )product term as

P(oklx) which is rather

worst case scenario, and

0 we can rearrange the

Comparing (7) and (31) it is apparent that each term (classok hypothesis) in the error free classifier combination rule (7)is affected by error factor

(32)

"R R R

[P(WkIX) +ek [ kfi P(XkIX 11 + k)

-1 =1=1k x ) l

which can then be linearized as

Substituting (30) into (28) we get +

Substituting (30) into (28) we get

)I Re]iF 'max iP-("-)(Wk)l P(oix P( I )+ e

k=1 L _ P(WX )

ass gn Z -i 0ji

(29)A similar analysis of the sum rule (11) commences with

ass gn

(30)

Z - mi f

S(1- R)P(0) + RfP(jI@ x + ei ] =

( Rmax{(1 - R)P(w) + Y[P(wk x )+ek ]

which can be rewritten as =1which can be rewritten as

(31)

(D)

(33)

0

R ek

1 + I P(Wklx


Fig. 5. Classifier combination schemes.

ass gn Z -+ i f1+ (35)

(1- R)P(wa) + P(C, x) 1+

Sek e

max (1- R)P(ck) +x P(w1x + R ' (34)k=1 _ 1:P(wkX /

=1A comparison of (11) and (34) shows that each term in

the error free classifier combination rule (11) is affected byerror factor

Comparing error factors (32) and (35), it transpires thatthe sensitivity to errors of the former is much more dra-matic than that of the latter. Note that since the a posteriori

class probabilities are less than unity, each error' ek in (32) is

amplified by .The compounded effect of all theseP(Wo jx

amplified errors is equivalent to their sum. In contrast, inthe sum rule, the errors are not amplified. On the contrary,their compounded effect,; which is also computed as a sum,is scaled by the sum of the a posteriori probabilities. For themost probable class, this sum is likely to be greater than onewhich will result in the dampening of the errors. Thus, thesum decision rule is much more resilient to estimation er-

I


rors and this may be a plausible explanation of the superior

performance of this cominbination strategy that we observed

experimentally in Sections 4 and 5, or at least a contributing

factor to it. It follows, therefore, that the sum classifier

combination rule is not only a very simple and intuitive

technique of improving the reliability of decision making

based on different classifier opinions but it is also remarka-

bly robust.

7 CONCLUSIONS

The problem of combining classifiers which use different

representations of the patterns to be classified was studied.

We have developed a common theoretical framework for

classifier combination and showed that many existing

schemes can be considered as special cases of compound

classification where all the, pattern representations are used

jointly to make a decision. We have demonstrated that un-

der different assumptions and using different approxima-

tions we can derive the commonly used classifier combina-

tion schemes such as the product rule,.sum rule, min rule,

max rule, median rule, and majority voting. The various

classifier combination schemes were compared experi-

mentally. A surprising outcome of the comparative study

was that. the combination rule developed under the most

restrictive assumptions-the sum rule---outperformed

other classifier combinations schemes. Toexplain this em-

pirical finding, we investigated the sensitivity of various

schemes to estimation errors. The sensitivity analysis has

shown that the sum rule is most resilient to estimation er-

rors and this may provide a plausible explanation for its

superior performance.

ACKNOWLEDGMENTS

This work was supported by the Science and EngineeringResearch Council, UK (GR/K68165) and by the European

Union ACTS Project M2VTS. The authors would like to

thank Andrew Elms for making available the classification

results obtained using his HMM character recognizer and

Kenneth Jonsson and Medha Pandit for providing the

frontal face image and the voice data, respectively. We are

also indebted to Stephane Pigeon for providing the face

profile data and verification results and to Gilbert Maitre

for making available the voice-based verification decisions,

which were then used in fusion experiments.

REFERENCES[1) K.M. Ali and M.J. Pazzani, "On the Link Between Error Correla-

tion and Error Reduction in Decision Tree Ensembles," TechnicalReport 95-38, ICS-UCI, 1995.

12) S.C. Bagui and N.R. Pal, "A Multistage Generalization of the RankNearest Neighbor Classification Rule," Pattern Recogn t on Letters,vol. 16, no. 6, pp. 601-614, 1995.

13] J. Cao, M. Ahmadi, and M. Shridhar, "Recognition of Handwrit-ten Numerals With Multiple Feature and Multistage Classifier,"Pattern Recogn t on, vol. 28, no. 2, pp. 153-160, 1995.

14) S.B. Cho and J.H. Kim, "Combining Multiple Neural Networks byFuzzy Integral for Robust Classification," IEEE Trans. Systems,Man, and Cybernet cs, vol. 25, no. 2, pp. 380-384,1995.

15) S.B. Cho and J.H. Kim, "Multiple Network Fusion Using FuzzyLogic," IEEE Trans. Neural Networks, vol. 6, no. 2, pp. 497-501,1995.

[6) D.A. Denisov and A.K. Dudkin, "Model-Based ChromosomeRecognition Via Hypotheses Construction/Verification," PatternRecogn t on Letters, vol. 15, no. 3, pp. 299-.307, 1994.

[7) H. E)-Shishini, M.S. Abdel-Mottaleb, M. EI-Raey, and A. Shoukry,"A Multistage Algorithm for Fast Classification of Patterns," Pat-tern Recogn t on Letters, vol. 10, no. 4, pp. 211-215, 1989.

[8) "M.C. Fairhurst and H.M.S. Abdel Wahab, "An Interactive Two-Level Architecture for a Memory Network Pattern Classifier," Pat-tern Recogn t on Letters, vol. 11, no. 8, pp. 537-540, 1990.

[9) J. Franke and E. Mandler, "A Comparison of Two Approaches forCombining the Votes of Cooperating Classifiers," Proc. 11th IAPRInt't Conf. Pattern Recognition, Conf. B: Pattern Recognition Meth-odology and Systems, vol. 2, pp. 611-614; 1992.

[101 L.K. Hansen and P. Salamon, "Neural Network Ensembles," IEEETrans. Pattern Analys s and Mach ne Intell gence, vol. 12, no. 10, pp.993-1,001, Oct. 1990.

[11) Hashem and B. Schmeiser, "Improving Model Accuracy UsingOptimal Linear Combinations of Trained Neural Networks," IEEETrans. Neural Networks, vol. 6, no. 3, pp. 792-794, 1995.

[12] T.K. Ho, "Random Decision Forests," Third Int'l Conf DocumentAnalysis and Recognition, pp. 278-282, Montreal, 14-16 Aug. 1995.

[131 T.K. Ho, J.J. Hull, and S.N. Srihari, "Decision Combination inMultiple.Classifier Systems," IEEE Trans. Pattern Analys s and Ma-ch ne Intell gence, vol. 16, no. 1, pp. 66-75, Jan. 1994.

[14) F. Kimura and M. Shridhar, "Handwritten Numerical RecognitionBased on Multiple Algorithms," Pattern Recogn t on, vol. 24, no. 10,Spp, 969-983,1991.

[15) A. Krogh. and J. Vedelsby, "Neural Network Ensembles, CrossValidation, and Active Learning," Advances n Neural Informat onProcess ng Systems 7, G. Tesauro, D.S. Touretzky, and T.K. Leen,eds. Cambridge, Mass.: MIT Press, 1995.

[16) M.W. Kurzynski, "On the Identity of Optimal Strategies for Multi-stage Classifiers," Pattern Recogn t on Letters, vol. 10, no. 1, pp. 39-46,1989.

[17) P. Pudil, J. Novovicova, S. Blaha, and J. Kittler, "Multistage Pat-tern Recognition With Reject Option," Proc. 11th (APR Int'l ConfPattern Recognition, Conf. B: Pattern Recognition Methodologyand Systems, vol. 2, pp. 92-95, 1992.

[181 G. Rogova, "Combining the Results of Severa) Neural NetworkClassifiers," Neural Networks, vol. 7, no. 5, pp. 777-781, 1994.

[19) M. Skurichina and R.EPW. Duin, "Stabilizing Classifiers for.VerySmall Sample Sizes," Proc. 11th IAPR Int'l Conf. Pattern Recogni-tion, Vienna, 1996.

[20] V. Tresp and M. Taniguchi, "Combining Estimators Using Non-Constant Weighting Functions," Advances n Neural I nformat onProcess ng Systems 7, G. Tesauro, D.S. Touretzky, and T.K. Leen,eds. Cambridge, Mass.: MIT Press, 1995.

1211 C.H. Tung, H.J. Lee, and J.Y. Tsai, "Multi-Stage Pre-CandidateSelection in Handwritten Chinese Character Recognition Sys-tems," Pattern Recogn t on, vol. 27; no. 8, pp. 1,093-1,102, 1994.

[22) D.H. Wolpert, "Stacked Generalization," Neural Networks, vol. 5,no. 2, pp. 241-260, 1992.

123) L. Xu, A. Krzyzak, and C.Y. Suen, "Methods of Combining Multi-ple Classifiers and Their Applications to Handwriting Recogni-tion," IEEE Trans. Systems, Man, and Cybernet cs, vol. 22, no. 3,pp. 418-435, 1992.

[24) J.Y. Zhou and T. Pavlidis, "Discrimination of Characters by aMulti-Stage Recognition Process," Pattern Recogn t on, vol. 27,no. 11, pp. 1,539-1,549, 1994.

[25) J. Kittler, M. Hatef, and R.P.W. Duin, "Combining Classifiers,"Proc. 13th Int'i Conf Pattern Recognition, vol. 2, Track B, pp. 897-901, Vienna, 1996.

[261 J. Kittler, J. Matas, K. Jonsson, and M.U. Ramos SAnchez, "Com-bining Evidence in Personal Identity Verification Systems," Pat-tern Recogn t on Letters, pp. 845-852, 1997.

[27) L. Breiman, "Bagging Predictors," Technical Report 421, Dept. ofStatistics, Univ. of California at Berkeley, 1994.

[28] Y. Freund and R.E. Shapire, "Experiments With a New BoostingAlgorithm," Proc. 13th Int'l Conf. Machine Learning, 1996.

[29) R.E. Shapire, Y. Freund, P. Bartlett, and W.S. Lee, "Boosting theMargin: A New Explanation for the Effectiveness of Voting Meth-ods," Proc. 14th Int'l Conf. Machine Learning, 1997.

1[30) K. Tumrner and J. Ghosh, "Analysis of Decision Boundaries in Line-arly Combined Neural Classifiers," Pattern Recogn t on, vol. 29,pp. 341-348, 1996.


(31) K. Turner and J. Chosh, "Classifier Combining: Analytical Resultsand Implications," Proc. Nat'l Conf. Artificial Intelligence, Portland,Ore., 1996.

132] K.S. Woods, K. Bowyer, and W.P Kergelmeyer, "Combination ofMultiple Classifiers Using Local Accuracy Estimates," Proc. CVPR'96, pp. 391-396, 1996.

(33] J. Kittler, A. Hojjatoleslami, and T. Windeatt, "Weighting Factorsin Multiple Expert Fusion," Proc. Br t sh Mach ne Vs on Conf, Col-chester, England, pp. 41-50, 1997.

[34] J. Kittler, A. Hojjatoleslami, and T. Windeatt, "Strategies for Com-bining Classifiers Employing Shared and Distinct Pattern Repre-sentations," Pattern Recogn I on Letters, to appear.

[35] J. Kittler, "Improving Recognition Rates by Classifier Combina-tion: A Theoretical Framework," Front ers of Handwr t ng Recog-n t on 5, A.G. Downton and S. lmpedovo, eds. World Scientific,pp. 231-247, 1997.

[36) T.S. Huang and C.Y Suen, "Combination of Multiple Experts forthe Recognition of Unconstrained Handwritten Numerals," IEEETrans. Pattern Analys s and Mach ne Intell gence, vol. 17, no. 1, pp. 90-94, Jan. 1995.

[37] P.A. Devijver and J. Kittler, Pattern Recogn t on: A Stat st cal Ap-proach. Englewood Cliffs, N.J.: Prentice Hall, 1982.

[38) M. Hatef and J. Kittler, "Constraining Probabilistic Re!axation'With Symbolic Attributes," Proc. Sixth Int'l Conf. Computer Analy-sis of Images and Patterns, V. Hlavac and R. Sara, eds., pp. 862-867,Prague, 1995.

[39) M. Hatef and J. Kittler, "Combining Symbolic With Numeric At-tributes in Multiclass Object Recognition Problems," Proc. SecondInt'l Conf Image Processing, vol. 3, pp. 364-367, Washington, D.C.,1995.

(40] A.J. Elms, "A Connected Character Recogniser Using LevelBuilding of HMMs," Proc. 12th IAPR Int'l Conf Neural Networks,Conf. B: Pattern Recognition Methodology and Systems, vol. 2,pp. 439-441, 1994.

[41] J. Matas, K. Jonsson, and J. Kittler, "Fast Face Localisation andVerification," A. Clark, ed., Bir t sh Mach ne Vs on Conf., pp. 152-161, BMVA Press, 1997.

[42] S. Pigeon and L. Vandendrope, "The M2VTS Multimodal FaceDatabase (Release 1.00)," J. Bigun, C. Chollet, and G. Borgefors,eds., Aud o- and V deo-Based B ometr c Person Authent cat on, pp.403-409. Springer, 1997.

(43] D. Genoud, G. Gravier, F. Bimbot, and G. Chollet, "CombiningMethods to Improve the Phone Based Speaker Verification Deci-sion," Proc. Int'l Conf. Speech and Language Processing, vol-3, pp.1,756-1,760, Philadelphia, 1996.

(44] S. Pigeon and L. Vandendrope, "Profile Authentication Using aChamfer Matching Algorithm," J. Bigun, G. Chollet, and G. Bor-gefors, eds., Aud o- and V deo-Based B ometr c PersonAuthent cat on,pp. 185-192. Springer, 1997.

Josef Kittler graduated from the University ofCambridge in electrical engineering in 1971, wherehe also obtained his PhD in pattern recognition in1974, and the ScD degree in 1991. He joined theDepartment of Electronic and Electrical Engineer-ing of Surrey University in 1986 where he is a pro-fessor in charge of the Centre for Vision, Speech,and Signal Processing. He has worked on varioustheoretical aspects of pattern recognition and onmany applications including automatic inspection,

Mohamad Hatef received the BSc degree inelectrical engineering from the University ofBaghdad, Iraq in 1982. After graduation, heworked automatic control. He joined the Univer-sity of Surrey to pursue postgraduate studies in1991. Since 1995, he has been with ERA Tech-nology, where he works on image and videocompression.

Robert P.W. .Duin studied applied physics atDelft University of Technology in the Nether-lands. In 1978, he received the PhD degree for athesis on the accuracy of statistical pattern rec-ognizers. In his research, he included variousaspects of the automatic interpretation of meas-urements, learning systems, and classifiers.Around 1980, he integrated the Delft ImageProcessor, a reconfigurable pipelined machine, inan interactive research environment for imageprocessing. In connection with this, he Initiated

several projects on the comparison and evaluation of parallel architec-tures for image processing and pattern recognition. At the moment, hismain research interest is the comparison oi neural networks with thetraditional pattern recognition classifiers for learning. In 1994, he stayedas a visiting professor at the University Teknologi Malaysia. In 1995, hespent his sabbatical leave at the University'of Surrey in the Vision,Speech and Signal Processing Group.

Heheld official positions in both, the Dutch Society for Pattern Rec-ognition' and Image Processing as well as the International Associationfor Pattern Recognition (IAPR). He has been a member of the organiz-ing committees of international conferences on signal processing (Eu-sipco 86) and pattemrecognition (ICPR 92), as well as of the scientificcommittees of many conferences on pattern recognition, image proc-essing, and computer vision.

At present, he is an associate professor of the Faculty of AppliedSciences of Delft University of Technology. He leads several projectson pattern recognition and neural network research, sponsored byboth, the Dutch government and the industry. He is the author of alarge number of scientific papers and has served as an associate editorfor IEEE Transactions on Pattern Analysis and Machine Intelligence,Pattemrn Recognition Letters, and for Pattern Analysis and Applications.Several PhD students are coached by him. His teaching includes un-dergraduate, graduate, and postgraduate courses on statistics, patternrecognition, neural networks, and image processing.

JIri (George) Matas received the MSc degree(with honors) in electrical engineering from theCzech Technical University in Prague, Czecho-slovakia, in 1987., and a PhD degree from theUniversity of Surrey, UK in 1995. He currently isa research fellow both with the Centre for Vision,Speech, and Signal Processing at the Universityof Surrey and the Centre for Machine Perceptionat the Czech Technical University. His interestinclude pattern recognition and computer vision.

ECG diagnosis, remote sensing. robotics, speechrecognition, and document processing. His current research interestsinclude pattern recognition, image processing, and computer vision.

He has co-authored a book with the title Pattern Recognition: AStatistical Approach," published by Prentice-Hall. He has publishedmore than 300 papers. He is a member of the editorial boards of Pat-tern Recognition Journal, Image and Vision Computing, Pattern Rec-ognition Letters, Pattern Recognition and Artificial Intelligence, andMachine Vision and Applications.

239

f7.


IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24. NO. 3, MARCH 2002

FVC2000: Fingerprint Verification Competition

Dario Maio, Member, IEEE, Davide Maltoni,Raffaele Cappelli, J.L. Wayman, and.

Anil K. Jain, Fellow, IEEE

Abstract-Reliable and accurate fingerprint recognition is a. challenging patternrecognition problem, requiring algorithms robust in many contexts. FVC2000competition attempted to establish the first common benchmark, allowingcompanies and academic institutions to unambiguously compare performanceand track improvements in their fingerprint recognition algorithms. Threedatabases were created using different state-of-the-art sensors and a fourthdatabase was artificially generated; 11 algorithms were extensively tested on thefour data sets. We believe that FVC2000 protocol,. databases, and results will beuselul to all practitioners in the field not only as a benchmark for improvingmethods, but also for enabling an unbiased evaluation ot algorithms.

Index Terms-Fingerprint verification, performance evaluation, biometricsystems.

1 INTRODUCTION

IN the last decade, interest in fingerprint-based biometric systems

has grown significantly 19]. Activity on this topic increased in both

academia and industry as several research groups and companies

developed new algorithms and techniques for fingerprint recogni-

tion and as many new fingerprint acquisition sensors were

launched into the marketplace.

Nevertheless, to date only a few benchmarks have been

available for comparing developments in fingerprint verification.

Developers usually perform internal tests over self-collected

databases. In practice, the only public domain data sets are the

US National Institute of Standards and Technology (NIST) CD-

ROMs [20), [21) containing thousands of images scanned from

paper cards where fingerprints were impressed by rolling "nail to

nail" inked fingers. Since these images significantly differ from

those acquired by optical or solid state sensors, they are not well-

suited for testing "online" fingerprint systems [9], although they,

constitute an excellent benchmark for AFIS (Automated Finger-

print Identification Systems) developments (11] and fingerprint

classification studies [41. NIST recently released a database

containing digital videos of live-scan fingerprint data [22]; since

this database was specifically collected for studying plastic

distortion affecting the online acquisition process 15], [6] and the

impact of finger rotation, it models only certain fingerprint

variations and it is not recommendable for a general evaluation

of verification algorithms.The lack of standards has unavoidably led to the dissemination

of confusing, incomparable, and irreproducible results, sometimes

* D. Maio, D. Maltoni, and R. Cappelli are with Biometric System Lab(BIOLAB)-DEIS, University of Bologna, via.Sacchi 3, 47023 Cesena,Italy. E-mail: (maio, maltoni, cappelli)@csr.unibo.it..

* J.L. Wyman is with the US National Biometric Test Center, College oJEngineering, San lose State University, San Jose, CA 95192.E-mail: [email protected].

* AK. Jain is with the Pattern Recognition and Image ProcessingLaboratory, Michigan State University, East Lansing, MI 48824.E-mail: [email protected]

Manuscript received 09 Oct. 2000; revised 27 Feb. 2001; accepted 15 May2001.Recommended for acceptance by D. Kriegman.Fr infor mation on obtaining reprints of this article, please send e-mail to:[email protected], and reference IEEECS Lag Number 112958.

0162-8828/02/517.00 0 2002 IEEE

embedded in research papers and sometimes enriching thecommercial claims of marketing brochures.

The aim of this initiative was to take the first'step towards the

establishment of a common basis, both for academia and industry, to

better understand the state-of-the-art and what can be expected

from the fingerprint technology in the future. Analogous efforts

have been recently carried out for other biometric characteristics

(e.g., face [151, [121) and, in general, for other classical pattern

recognition tasks ([171,[1 ], 18], [18)). We decided to pose this effort as

an international open competition to boost interest and give our

results larger visibility. The 15th International Conference on

Pattern Recognition (ICPR 2000) was ideal for this purpose. Starting

in late spring 1999, when the FVC2000 Web site (7) was set up, we

broadly publicized this event, inviting all companies and research

groups we were aware of to take part.

From the beginning, we stated that the competition was not

meant as an official performance certification of the participant

biometric systems, as:

* The databases used in this contest. have not been acquired

in a real environment and according to a formal protocol

[23], [16], [19], [2] (also refer to [24] for an example of

performance evaluation on real applications).. Only parts of the participants software are evaluated by

using images from sensors not native to each system. In fact,

fingerprint-based biometric systems often implement pro-prietary solutions to improve robustness and accuracy (e.g.,

quality control modules to reject poor quality fingerprints,visual feedback to help the user in optimally positioning '

his/her finger, using multiple fingerprint instances to build

more reliable templates, etc.) and these contributions are

here discounted.* According to the definition reported in [16], [19], FVC2000

should be conceived as a technology evaluation (with some

analogies with the FERET contest organized by Philips on

face recognition (15]). In fact, quoting [2]:

"The goal of a technology evaluation is to compare competingalgorithms from a single technology. Testing of all algorithms isdone on a standardized database collected by a "universal" sensor.Nonetheless, performance against this database will depend uponboth the environment and the population in which it was collected.

SConsequently, the "three bears" rule might be applied, attempting tocreate a database that is neither too difficult nor too easy for thealgorithms to be tested. Although sample or example data may bedistributed for developmental or tuning purposes prior to the test,.the actual testing must be done on data which has not beenpreviously seen by algorithm developers. Testing is done using"offline" processing of the data. Because the database is fixed,results of technology tests are, repeatable."

In FVC2000, the "universal" sensor is actually a collection of

four different sensors/technologies to better cover the recent

advances in fingerprint sensing techniques and to avoid favoring

a particular algorithm through the choice of a specific sensor. In fact,

databases 1 and 2 were collected by using two small-size and low-

cost sensors (optical and capacitive, respectively). Database 3 was

collected by using a higher quality (large area) optical sensor.

Finally, images in database 4 were synthetically generated by using

the approach described in 13]. Each of the four databases contained

880 fingerprints from 110 different fingers, collected using the

"three bears rule" (not too easy, not too hard), based on our.prior

subjective experiences with fingerprint recognition algorithms; in

particular, on the one hand, we discarded fingerprint images we

considered completely intractable even for a human expert, on the

other hand, we avoided collecting perfect fingerprints which will be

very easy for a matching algorithm;, some internally developed


TABLE 1List of Participants

ID' . . ~Oraoization. TyeCETP CEFET-PR / Antheus Technologia Ltda (Brasil) AcademicCSPN Centre for'Signal Processing, Nanyang Technological Universit (Singapore) AcademicCWAI Centre for Wavelets, Approximation and Information Processing, Department of Academic

Mathematics, National University of Singapore (Singapore)DITI Ditto Information & Technology Inc. (Korea) CommercialFPIN FingerPin AG (Switzerland) I Commercial

KRDL Kent Ridge Digital Labs (Singapore) AcademicNCMI Natural Sciences and Mathematics, Institute of Informatics (Madedonia) AcademicSAGI SAGEM SA (France) i CommercialSAG2 SAGEM SA (France) CommercialUINi inha University (Korea) AcademicUTWE University of Twente, Electical Engineering (Netherlands) ! Academic

A tour digit ID was assigned to each algorithm. (Sagem SA submitted two different algorithms).

TABLE 2The Four FVC2000 Databases

Sensor Type . Image Size Set A (wxd) Set B (wxd) Resolution

DBI Optical Sensor 300x300 100x8 110x8 500 dpi

DB2 Capacitive Sensor 256x364 100x8 1l0x8 500 dpiDB3 Optical Sensor 448x478 100x8 10x8 500 dpiDB4 Synthetic Generator 240x320 100x8 10x8 About 500 dpi

I In the artifical generation, the resolution is controlled by the average ridge-line interdistance; this input parameter was estimated from a real 500 dpi fingerprint database.

algorithms helped us in accomplishing this task. Each database was working on the submitted executables to complete their evaluation

split into a sequestered "test" set of 800 images (set A) and an open by August 2000.

"training" set of 80 images (set B), made available to participants for Once all the executables were submitted, feedback was sent to

algorithm tuning. The samples in each set B were chosen to be as the participants by providing them the results of their algorithms

much as possible representative of the variations and difficulties in over training set B (the same data set they had previously been

the corresponding set A; to this purpose fingerprints were given) to allow them to verify that neither run time problems nor

automatically sorted by quality as in [14] and samples covering hardware-dependent misbehaviors were occurring on our side.

the whole range of quality were included in set B. A final visual Section 2 describes the four databases used; in Section 3, we

inspection of the obtained data sets was carried out to assure that present the criteria and the procedures used for performance

"dry," "wet," "scratched," "distorted," and "markedly rotated" evaluation. Section 4 reports the overall performance of the

fingerprints were also adequately represented. . participating algoithms on each database and concludes with a

As initially specified in the call for participation "FVC2000 comparison of the! average results. Finally, in Section 5, we draw

compeltition focuses only on fingerprint verification (1-1 matching) and some concluding remarks and discuss how we intend to continue

not on fingerprint identification (1-N matching)" [9]. Each participant supporting this initiative in the future.was required to submit two executable computer programs: the first

enrolling a fingerprint image and producing the corresponding 2 DATABASEStemplate, the second matching a fingerprint template against a

fingerprint image. Participants were allowed to submit four distinct Four different databases (hereinafter DB1, DB2, DB3, and DB4)

configuration files, to adjust the algorithms internal parameters were collected by using the following sensors/technologies [(10]:

according to each specific database; configuration files could also * DB1: optical sensor "Secure Desktop Scanner" by KeyTroniccontain precomputed data, to save time during enrollment and *. DB2: capacitive sensor "TouchChip" by ST Microelectronics

matching. For practical testing reasonis, the maximum response time * DB3: optical sensor "DFR-90" by Identicator Technology

of the algorithms was limited to 15 seconds for each enrollment and * DB4: synthetically generated based on the method pro-

five seconds for each matching (on a Pentium III-450 MHz posed in [3]. /

machine). Each database is 110 fingers wide (w) and eight impressions per

In March 2000, after several months of active promotion, we finger deep (d) (880 fingerprints in all); fingers from 101 to 110 (set B)

had 25 volunteering participants (about 50 percent from academia were made available to the participants to allow parameter tuning

and 50 percent from industry), far more than our initial before the submission of the algorithms; the benchmark is then

expectation. By the end of April, the training sets were released constituted by fingers numbered from 1 to 100 (set A). For a system

to the participants. evaluation, the size of the above four databases is certainly not

After the submission deadline (une 2000) for the executables, sufficient to estimate the performance with high confidence.

the number of participants decreased to 11 (most of the initiallytheinumer ofparicitsdrease to 11 (msto the nitill 1. These sensors are identified in order to clearly specify the features of

registered companies withdrew). In any case, the number of the databases. None of the authors have any proprietary interests in these

participants (see Table 1) was more than anticipated, so we started companies or products.

Fig. 1. S

scale fat

Fig. 2. S

However, in a technology evaluation (like, FVC2000), the aim is to

capture the variability and the difficulties of the problem at hand

and to investigate how the different algorithms deal with them. For

this purpose, the size of our databases are adequate.Table 2 summarizes the global features of the four databases,

and Fig. 1 shows a sample image from each one of them.

It is worth emphasizing that the choice of providing more than

one database is not aimed at comparing different acquisition

technologies and devices; the results obtained by the algorithms on

the different databases should not be conceived as a qualitymeasure of the corresponding sensors, since the acquisition

conditions and the volunteer crew of each database are different.To summarize, DB1 and DB2 have the following features:

* The fingerprints are mainly from 20 to 30 year-old students(about 50 percent male).

* Up to four fingers were collected for each volunteer(forefinger and middle finger of both the-hands).

* The images were taken from untrained people intwo different sessions and no efforts were made to assurea minimum acquisition quality.

* . All the images from the'same individual were acquired byinterleaving.the acquisition of the different fingers (e.g.,

H 2002

e same

first sample of left forefinger, first sample of right forefinger, first sample,. of left middle, first sample of rightmiddle, second sample of the left forefinger, ...).

* The presence of the fingerprint cores and deltas is notguaranteed since no attention was paid on checking thecorrect finger position on the sensor.

* The sensor platens were not systematically cleaned (asusually suggested by the vendors).

* The acquired fingerprints were manually analyzed toassure that the maximum rotation is approximately.in therange [-15u, 150] and that each pair of impressions of thesame finger has a nonnull overlapping area.

Database DB3 was collected as follows:

* The fingerprints are from 19 volunteers between th. agesof five to 73 (55 percent male).

* One-third of the volunteers were over 55 years of age.* One-third of the volunteers were under 18 years of age.* One-sixthlof the volunteers were under seven years of age

(childrens fingerprints constitute an interesting case study,since the usable image area is small and the ridge-linedensity is high).

* Two images of up to six fingers (thumb, fore, and middleon left and right hands) were taken without interleaving

IEEE TF

Fig. 3. Images from DB3; all the samples are from different fingers and are roug

from each volunteer at each session and no efforts weremade to assure a minimum acquisition quality.

* Each volunteer was seen at four sessions, with no morethan two sessions on any single day.

* The time gap between the first and last sessions was atleast three days and as long as three months, dependingupon volunteer.

* The sensor plate was systematically cleaned betweenimage acquisitions.

* At one session with each volunteer, fingers were cleanedwith rubbing alcohol and dried.

* Some part of the core was apparent in each image, but carewas taken to avoid a complete overlap between consecu-tive images taken during a single session.

* The acquired fingerprints were manually analyzed toassure that the maximum rotation is approximately in therange [-15 °, 15° arid that each pair of impressions of thesame finger has a nonnull overlapping area.

Figs. 2 and 3 show some sample images taken from DB3.The collection of DB4 requires some explanation: In general, the

use of artificial images for testing biometric systems is not

considered to be the "best practice" [19]. Although, this may bethe case for performance evaluation in real applications, we believe

that in a technology evaluation event such as FVC2000, the use of

synthetic images has three main advantages:

* It supplies images which are native to none of theparticipant algorithms, thus providing a fair comparison.

* Synthetic fingerprint databases can be created at a verylow cost. Acquiring a large number of fingerprints fortesting purposes may be problematic due to the great

hly ordered by quality (top-left: high quality, bottom-right: low quality).

amount of time and resources required and to the privacylegislation (which in some countries prohibits the diffusionof such personal information. Furthermore, once a data-base has been "used," its utility is limited since, forsuccessive testing of algorithms, a new unknown databaseshould be used.

* It is possible to adjust the database difficulty by tuningdifferent kinds of perturbations (e.g., maximum amount ofrotation arid translation, and the amount of skin distortion).

If the generated artificial images were not a suitable simulation

of real fingerprint patterns, the comparisons on the synthetic

ZeroFMR

Ic threshold I

toFig. 4EER,

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACH

TABLE 3Algorithm Performance over DB1 Sorted by EER

Algorithm EER REJENROLL REJMATCH Avg Enroll Avg Match(%) (%) (%) Time (sec.) Time (sec.)

Sagl 0.67 0.00 0.00 12.48 0.96Sag2 1.17 0.00 0.00 10.88 0.88Ceip 5.06 0.00 0.00 0.81 0.89Cwai 7.06 3.71 3.90 0.22 0.32Cspn 7.60 0.00 0.00 10.17 0.17Unve 7.98 0.00' 0.00 110.40 2.10Krdl 10.66 6.43 6.59 1.00 1.06Fpin 13.46 0.00 0.00 10.83 0.87Uinh 21.02 1.71 5.08 I 0.53 0.56Dii 23.63 '0.00 0.00 0.65 0.72

Ncmi 49.11 0.00, 0.12 ;1.13 1.34


Algorithm EER REJENROLL REJUATCH Avg Enroll Avg Match(%) (%) (%) Time (sec.) Time (sec.)

Sa I 0.61 0.00 0.00 1 2.63 1.03Sag2 0.82 0.00 0.00 1 0.93 0.93Cspn 2.75 0.00 0.00 1 0:17 0.17Cwai 3.01 1.29 1.29, 10.23 0.30Cetp 4.63 0.00 0.09 i 0.85 0.98Krdl 8.83 3.29 4.41 11.16 2.88Utwe 10.65 0.00 0.00 1.10.42 2.12Fpin 11.14 0.00 0.00 1 1.16 1.24Diti 13.83 0.00 -0.00 1 1.21 1.28Uinh . 15.22 0.86 4.08 1 0.60 0.65Ncmi 46.15 0.00 0.00 1 1.28 1.57

database would be misleading. Furthermore, in order to improve

the performance, ad hoc algorithms could be designed/tuned

according to the same assumptions which model the syntheticgeneration. However, the presence of three real. databases in

FVC2000 provides a natural way to check the validity of the results

on DB4.The parameters of the synthetic generator were tuned to

emulate a low-cost sensor with a small acquisition area; the

maximum rotation and displacement and skin-distortion are

adjusted to roughly reproduce the perturbations in the three

previous databases.

3 PERFORMANCE EVALUATION

For each database, we will refer to the jth fingerprint sample of the

ith finger as F,.j, i = 1... 100, and j = 1... 8, and to the correspond-

ing template (computed from F) as T.For each database and for each algorithm:

S The templates Ti, i= 1...100, and j= 1...7 are com-puted from the corresponding Fj and stored on a disk; oneof the following three kinds of rejection can happen foreach image Fi:

I. F (Fail): the algorithm declares that it cannot enrollthe fingerprint image.

2. T (Timeout): the enrollment exceeds the maximumallowed time (15 seconds).

3. C (Crash): the algorithm crashes during fingerprintprocessming.

The three types of rejections are added and stored inREJE'NOL.

RJNRO L,*

* Each fingerprint template T is matched, against thefingerprint images Fuk (j < k < 8) and the correspondingGenuine Matching Scores gmsift are stored.2 The ntumber ofmatches (denoted as NGRA - Number of GenuineRecognition Attempts) is ((8 x 7)/2) x 100 = 2,800 in caseREJENRao LL = 0. The failed, timeout (five seconds). andcrash rejections are accumulated into REJNoRA; no gmsj isstored in this case.

* Each fingerprint template Ti, i = 1...100 is matchedagainst the first fingerprint image from different fingersFkt (i < k < 100) and the corresponding Impostor MatchingScores ims;k are stored. The number of matches (denoted asNIRA - Number of Impostor Recognition Attempts) is((100 x 99)/2)= 4,950 in case REJENROLL = 0. The failed,timeout (5 seconds) and crash rejections are accumulatedinto REJiA; no imsak is stored in this case.

* The genuine matching score distribution and the impostormatching score distribution are computed (actually, the

2. If g is matched with h, the symmetric match (i.e., h against g) is notexecuted.

INE INTELLIGENCE, VL 4 O ,MRH20

-1 --

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24. NO. 3, MARCH 2002


Algorithm EER REJENROLL REJMATCH Avg Enroll Avg Match(%) ) (%) Time (sec. Time (sec.)

Sag 1 3.64 0.00 0.00 5.70 2.13

Sag2 4.01 0.00 0.00 11.94 1.94

Cspn 5.36 0.57 1.24 0.35 0.36

Cerp 8.29 0.00 0.00 11.49 1.66

Cwaui 11.94 12.86 8.00 0.46 0.57

Krdl 12.20 6.86 5.12 11.48 1.60Uinh 16.32 10.29 7.64 1.28 1.36Utwe 17.73 0.00 0.00 10.44 2.31

Diii 22.63 00.00 .00 ' 2.59 2.67Fpin 23.18 0.00 0.00 . 2.13 2.19

Ncni 47.43 0.00 0.01 2.25 2.75


Algorithm EER REJENoRII. REJMA1CH Avg Enroll Avg Match

(%) (%) (%) , Tinme (sec.) Time (sec.)

Sag 1.99 0.00 0.00 1.90 0.77Sag2 3.11 0.00 0.00 0.69 0.69Cspn 5.04 0.00 0.00 0. 11 0.11

Cwai 6.30 0.00 0.00 ;0.16. 0.20

Cerp 7.29 0.00 0.00 0.65 0.72Krdl 12.08 10.86 10.24 10.70 0.79

Fpin 16.00 0.00 0.00 10.77 0.80

Diti 23.80 0.00 0.00 10.52 0.60

Urwe 24.59 0.00 0.00 10.42 4.17

Uinh 24.77 2.14 4.28 10.42 0.45

Ncmi. 48.67 0.00 0.25 1.08 *, 1.19

term "distribution" denotes a histogram) and graphically

reported to show how the algorithm "separates" the twoclasses.. In fingerprint verification, higher scores are"

associated with more closely matching images.* The FMR(t) (False Match Rate) and FNMR(t) (False

NonMatch Rate) curves are computed from the above

distributions for t ranging from 0 to 1.3 Given a threshold t,FMR(t) denotes the percentage of imsek 2 t, and

FNMR(t) denotes the percentage of gmsj. < t. Actually,

since FMR and FNMR are used in the contest to comparethe performance of different algorithms, FMR andFNMR are "corrected" to keep into account rejections

stored in REJNI1A and REJNGRA:

FMR(t) = card{irnsik ims, > t}

NIRA

FNMR(t) = card{gms,t I gmsi. < t) + REJNcGrtFNMR~t) = NGRA.

where card denote the cardinality of a given set. Thiscorrection assumes that a failure to match is always treated '

by the system as a "nonmatch" (matching score < 0).

3. FMR and FNMR are often referred as FAR (False Acceptance Rate)and FRR (False Rejection Rate) respectively, but the FAR/FRR notation ismisleading in some applications. For example, in a welfare benefits system,which uses fingerprint identification to prevent multiple payments underfalse identity, the system "falsely accepts" an applicant if his/herfingerprint is "falsely rejected"; otherwise, a "false acceptance" causes.a,"false rejection."'

S A ROC (1Receiving Operating Curve) is obtained, wherepairs (FMR(t), FNMR(t)) are plotted for the same value oft; in particiular, for t = 0, FMR = 1, and FNMR = 0, whilefor t > 1, FMR = 0, and FNMR= 1. The ROC curve isdrawn in log-log scales for better comprehension.

* The EquallError Rate EER is computed as the point where

FNMR(t) = FMR(t) (see Fig. 4); in practice, the matching.

score distributions (histograms) are not continuous and a

crossover point might not exist. In this case, we report the

interval [EERc,,,,,, EERhih]. In Appendix A, an operational

definition of EER is given.* ZeroFMR is defined as the lowest FNMR at which no

False Matches occur and ZeroFNMR is defined as thelowest FiIR at which no False NonMatches occur (Fig. 4):

ZeroFMR(t) = min t{FNMR(t) I FMR(ti) = 0})

ZeroFNMR(t) = mint{FMR(t) I FNMR(t) = 0)}

Both ZeroFMR and ZeroFNMR may not exist; in such acase, we assign to them the value 1.

* The average enroll time is calculated as the averageCPU time for a single enrollment operation, and averagematch time as the average CPU time for a single matchoperation between a template and a test"image.

4 RESULTS

This section reports the performance of the tested algorithms oneach of the four databases (Tables 3, 4, 5, and 6) and the average

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24,

TABLE 7Average Performance over the Four Databases Sorted by Avg EER

Algorithm Avg EER Avg Avg Avg Enroll Avg Match

(%) REJENo.L(%) REJMATCH (%) Time (sec.) Time (sec.)SagI 1.73 0.00 0.00 3.18 1.22.

Sag2 2.28 0.00 . 0.00 1.11 1.11

Cspn 5.19 0.14 0.31 0.20 .0.20

Cetrp 6.32 0.00 0.02 0.95 1.06

Cwai 7.08 4,46 3.14 0.27 0.35

Krdl 10.94 6.86 6.52 1.08 1.58

Utwe 15.24 0.00 0.00 10.42 2.67

Fpin 15.94 0.00 0.00 1.22 1.27

Uinh 19.33 3.75 5.23 0.71 0.76

Diti 20.97 0.00 0.00 1.24 1.32

Ncmi 47.84 0.00 0.09 1.44 1.71

NO. 3. MARCH 2002

results over the four databases (Table 7). Fig. 5 shows the ROC foi rDB3, which proved to be the most difficult data set. The notation'

introduced in Section 3 is used in both the graphics and tables, with

the only exception of reporting REJENROLL as a percentage valueand to collapse both REJNCRA and REJxIRA into a single valueREJNIATCI:

REJN ATCII =NIRA • REJNIA + NGRA • REJNcRA

NIRA + NGRA

For a, correct interpretation of the results, EER alone is not a

sufficient metric; REJENROLL should be also taken into account.

For each algorithm, detailed results (including genuine and

impostor distributions, FMR and FNMR curves, NGRA,

NIRA, ... ) are reported in [13). Due to lack of space, Appendix

B of this paper presents only detailed results of the SAG1

algorithm which had the best accuracy in our competition.

5 CONCLUSIONS

Most of the algorithms submitted to the competition performed

well, if we take into account the difficulty of adapting a given

FNMR1

10-1

10-2

1A-3FMR

-U- Sag1

SSag2

-- +- O3pn----- atp-- f- ONai

-- Krdl

---- L we

---- Fpin

--- Unh

--- Itrvi

Fig. 5. ROC curves on 083. Each point denotes a pair (FMR(t), FNMR(t)) for a given value oft.

I

'10-5 10-4 .10-3 . 10-2 10.1

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 3, - MARCH 2002

atcd,R

tl 2 I 12

Fig. 6. Computing the EER interval. On the lop an example is given where an EER point exists. On the bottom, two cases are shown where an EER point does not exist

and the corresponding intervals are highlighted.

Average enroll time: 2.48 secondsAverage match lime: 0.96 seconds

REJENROLL NGRA NIRA REJNcGA REJIs,0.00% (F:0 T:0 C:0) 2800 4950 0.00% (F:0 T:0 C:0 0.00% (F:0 T:0 C:0)

EER. EER* ZeroFMR ZeroFNM R0.67% (0.67%-0.68% 0.67% (0.67%-0.68%) 2.11% 53.13%

0

Score distributions FMR(t) and FNMR(t)

Fig. 7. Performance ol algorithm Sogl on database DBI 4A.

algorithm to new types of images. In particular, algorithms Sagl and and on what we learned from this experience, we can make the

Sag2 showed the best accuracy and Cspn exhibited a good trade-off following observations:

between accuracy and efficiency.Table 7 highlights a significant gap in the performance of the

different algorithms and it would be extremely interesting to

understand the reasons for such differences. To this purpose, after

the presentation of the results, we asked the participants to

provide some technical details about their methods, but only a few

of them responded (the responses can be found at the FVC2000

Web site [7]); in any case, on the basis of the participant responses

* A coarse analysis of the errors on genuine attemptsshowed that most of the errors were made by the

,algorithms on about 15-20 percent poor-quality finger-prints in each database. In other words, we could claimthat a 20-80 rule is valid: that is, 20 percent of the databaseis responsible for 80 percent of the errors.

* The most accurate algorithm (Sagl) takes a lot of time forenrollment (3.18 sec with respect to a median enrollmenttime of 1.08 sec). This suggests that an accurate image

FNMR

ROC curve


Average enroll time: 2.63 secondsAverage match time: 1.03 seconds

.REJgago : NGRA NIRA REJyGgA REJ0.00% (F:0 T:0 C:0) 2800 4950 I 0.00% (F:0 T:0 C:0) 0.00% (F:0 T:0 C:0)

EER > EER* ZeroFMR ZeroFNMR0.61% 0.61% 1.36% 50.69%

nimit FMR FNMR FNMR

I ,. ....

S10

20% to7

0%

e0 s vshd 0 1 B 104 10- 107 101 F MR

Score.distributions FMR(t) and FNMR(t) ROC curve

8. Performance of algorithm Sag] on database DB2..A.

Average enroll time: 5.70 secondsAverage match time: 2.13 seconds

REJE 1,1. NGRA INRA I REJ~ REJNRA0.00% (F:0 T:0 C:0) 2800 4950 0.00% (F:0 T:0 C:0) 0.00% (F:0 T:0 C:0)

EER EER* ZeroFMR ZeroFNMR3.64% 3.64% 6.82% 100.00%

0o dstrtoon 1

Score distributions FMR(I) and FNMR(t)

Periormance ol algorithm Sugl on database DB3.A.

enhancement and feature extraction is really important forimproving the matching accuracy. Furthermore, featureextraction seems to perform asymmetrically, since theaverage matching time (which also includes the featureextraction time for the test image) is substantially lowerthan a single enrollment time.

* The fastest algorithms (Cspn) extracts minutiae by an

adaptive tracing of the gray-level ridges, without a priori

binarization and thinning (which are time consuming

tasks) [14], (25) and exploits local minutiae arrangement tospeed-up the initial steps of minutiae matching (26].

FNMRFM FNMRDo7.

50A"

40%

2%

Fig. 9.

ROC curve

v~ i

410",


Average enroll time: I1.90 secondsAverage match time: 0.77 seconds

REJaEN-jL NGRA NIRA REJNcG. REJNmA0.00% (F:0 T:0 C:0) 2800 4950 0.00% (F:0 T:0 C:0) 0.00% (F:0 T:0 C:0)

EER ' EER* ZeroFMR ZeroFNMR .1.99% (1.98%-2.00%) 1.99% (1.98%-2.00%) 6.71% 100.00%

Score distributions FMR(t) and FNMR(t)

FNMR

1o .

30 • . .

10%,

10010 0 10' 100 102 . 10 F MA

ROC curve

Fig. 10. Performance of algorithm Sagl on database DB4LA.

Databases DBI and DB2 proved to be "easier" than DB3, eventhough the sensor used for DB3 is of higher quality. This meansthat the acquisition conditions and the volunteer population canhave a stronger impact on the performance than sensor quality.

The synthetically-generated database (DB4) was demonstrated

to be adequate for FVC2000 purposes: in particular, from Tables 3, 4,5, and 6, it is evident that the algorithm ranking on DB4 is quite

similar to the other databases, proving that no algorithm wasfavored or penalized by the synthetic images. In particular, if analgorithm performs well on real fingerprints, then it also performswell on synthetic fingerprints and vice versa. The visual analysis ofimpostor and genuine distributions (see [13]) definitely supportsthis claim, since no significant differences are seen between the DB4graphics and the others.

Once again we would like to remark that the results reported heredo not necessarily reflect the performance that the participatingalgorithms would achieve in a real environment or when embeddedinto a complete biometric system. In any event, we believe thatFVC2000 results:

* provide a useful overview of the state-of-the-art in this field,* allow researchers and companies to test their algorithms

over common databases collected using state-of-the-artsensors, and

* provide guidance to the participants for improving theiralgorithms.

In future, we intend to continue supporting this initiative as follows:

* The existing FVC2000 Web site [71 will be maintained todiffuse FVC2000 results and to promote FVC2000 testingprotocol as a standard for technological evaluations.

* Companies and academic research groups will be allowedto test new algorithms or improved versions of existing

algorithms on the FVC2000 benchmark databases and toadd their results to the FVC2000 Web site. New entries willbe kept isolated from the original entries, since hereafter,the full databases are known, in advance, which couldallow algorithm tuning to give unfair advantage to newparticipants.

* The second Fingerprint Verification Competition (FVC2002)has been scheduled and its results will be presented at the16th International Conference of Pattern Recognition.

SGenerating synthetic fingerprint databases for futureevaluations will be further investigated.

APPENDIX A

An operational procedure for computing EER (interval), given'afinite number of genuine and impostor matching scores, is reported

in the following. Lett = - max

S(gms.a U{ { t I FNMR ( t ) FMR(t)},to t{ gmsij) U {imsek)

and .

t tE {gms,) U {imsA.) {t FNMR(t) > FMR(t)).

The EER interval is defined as:

IEER6.,,EER,I=

{ IFNMR(t).FMR(t,)) if FNMR(t)+FMR(t,)<FMR(t2)+FNMR(tz)

{ IFMR(ts),FNMR(t)) otherwise

and EER is estimated as (EERone + EERhigh)/2 (see Fig. 6).

APPENDIX BPlease see Figs. 7, 8, 9, and 10.

I..

412 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24. NO. 3. MARCH 2002

REFERENCES

(1) J. Barron,. D. Fleet, and S. Beauchermmin, "Systems and Experiment:Performance of Optical Flow Techniques,," Int'l I. Computer Vision, vol. 12,no. 1, pp. 43-77, 1994.

12) BIOTEST: European Esprit Project No: 21978 (http://www.npl.co.uk/npl/sections/this/biotest/index.html), 1998.

[3) R. Cappelli, A. Erol, D. Maio, and D. Maltoni, "Synthetic Fingerprint-imageGeneration," Proc. 15th Int'l Conf. Pattern Recognition (ICPR 2000), pp. 475-478, Sept. 2000.

[4) R. Cappelli, D. Maio, and D. Maltorni, "Combining Fingerprint Classifiers,"Proc. First Int'l Workshop Multiple Classifier Systems (MCS2000), pp.351-361,June 2000.

I5) R. Cappelli, D. Maio, and D. Maltoni, "Modelling Plastic Distortion inFingerprint Images," Proc. Second Int'l Conf Advances in Pattern Recognition(ICAPR 2001), pp. 369-376, Mar. 2001.

(6) C. Dorai, N.K. Ratha, and R.M. Bolle, "Detecting Dynamic Behaviour inCompressed Fingerprint Videos: Distortion," Proc. Computer Vision andPattern Recognition, (CVPR 2000), vol. II, pp. 320-326, June 2000

(7) FVC2000 Web site: http://bias.csr.unibo.it/2000. Sept. 2000.18) A. Hoover et at., "An Experimental Comparison of Range Image

Segmentation Algorithms," IEEE Trans. Pattern Analysis and MachineIntelligence, vol. 18, no. 7. pp. 673-689, July 1996.

(91 Biometrics-Personal Identification in Networked Society. A.K. Jain, R.Bollearind, S. Pankanti, eds. Kiuwer Academic, 1999.

(10] A.K. Jain, S. Prabhakar, and A. Ross, "Fingerprint Matching: DataAcquisition and Performance Evaluation," MSU Technical Report TR99-14, 1999.

II I). H.C. Lee and R.E. Goensslen, "Advances in Fingerprint Technology," 1991.(12) J. Matas et al., "Comparison of Face Verification Results on the XM2VTS

Database," Proc. 15th Int'l Conf. Pattern Recognition, vol. 4, pp. 858-863, Sept.2000.

(13) D. Maio, D. Maltoni, R. Cappelli, J.L. Wayman, and A.K. Jain, "FVC2000:Fingerprint Verification Competition," DEIS Technical Report BL-09-2000,Univ. of Bologna; available online at: http://bias.csr.unibo.it/fvc. 2000.

[(14) D Maio and D. Maltoni, "Direct Gray-Scale Minutiae Detection inFingerprints," IEEE Trans. Pattern Analysis Machine Intelligence, vol. 19,no. 1, pp. 27-40, 1997.

(115] P.). Phillips, H. Moon, S.A. Rizvi, and P.J. Rauss, "The FERET EvaluationMethodology for Face-Recognition Algorithms," IEEE Trans. PatternAnalysis and Machine Intelligence, vol. 22, no. 10, pp. 1090-1104, Oct. 2000.

[16) P.J. Phillips, A. Martin, C.L. Wilson, and M. Przybocky, "An Introduction toEvaluating Biometric Systems," Computer, Feb. 2000,

(17) P.J. Phillips and K.W. Bowyer, "Introduction to Special Section onEmpirical Evaluation of Computer Vision Algorithms," IEEE Trans. PatternAnalysis and Machine Intelligence, vol. 21, no. 4, pp. 289-290, Apr. 1999.

[18) T. Randen and J.H. Husoy, "Filtering for Texture Classification: AComparative Study," IEEE Trans. Pattern Analysis and Machine Intelligence,vol. 21, no. 4, pp. 291-310, Apr. 1999.

(19) "Biometrics Working Group, Best Practices in Testing and ReportingPerformance of Biometric Devices,"UK Government's HYPERLINKhttp://www.alb.org.uk/bwg/index.html, Jan. 2000. (available online at:http://www.afb.org.uk/bwg/bestprac0.pdf).

120) C.I. Watson and C.L. Wilson, NIST Special Database 4, Fingerprint Database.US Nat'l Institute of Standards and Technology, 1992.

(21) C.I. Watson, NIST Special Database 14, Fingerprint Database. US Nat'l Inst.Standards and Technology, 1993.

[22) C.I. Watson, "NIST Special Standard Reference Database 24, NIST DigitalVideo of Live-Scan Fingerprint Database," US Nat'l Inst. Standards andTechnology, 1998.

[23) J.L. Wayman, "Technical Testing and Evaluation of Biometric Devices,"Biometrics Personal Identification in Networked Society, A. Jain, et al. eds.,Kluwer Academic, 1999.

[24) J.L. Wayman, "The Philippine AFIS Benchmark Test," Nat'l Biometric TestCenter Collected Works, 1997-2000, Sept. 2001), (available online at: http://www.engr.sjsu.edu/biometrics/collected).

(25) X. Jiang et al., "Minutiae Extraction by Adaptive Tracing the Gray LevelRidge of the Fingerprint Image," Proc. IEEE Int'l Conf. Image Processing (ICIP'99), 1999.

126) X. Jiang et al., "Fingerprint Minutiae Matching Based on the Local AndGlobal Structures," Proc. 15th Int'l Conf. Pattern Recognition, Sept. 2000.

u For more information on this or any other computing topic, please visit ourDigital Library at http://computer.org/publications/dilb.

Appeared in Proc. of ICME, July 2003


COMBINING CLASSIFIERS FOR FACE RECOGNITION

Xiaoguang Lu*, Yunhong Wangt, Anil K. Jain*

*Department of Computer Science & Engineering, Michigan State, UniversityEast Lansing, MI 48824

{lvxiaogu, jain} @cse.msu.edu

tNational Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of SciencesP.O. Box 2728, Beijing 100080, P. R. China

wangyh@ nlpr.ia.ac.cn

ABSTRACT

Current two-dimensional face recognition approaches canobtain a good performance only under constrained environ-ments. However, in the real applications, face appearancechanges significantly due to different illumination, pose, andexpression. Face recognizers based on different representa-tions of the input face images have different sensitivity tothese variations. Therefore, a combination of different faceclassifiers which can integrate the complementary informa-tion should lead to improved classification accuracy. We usethe sum rule and RBF-based integration strategies to com-bine three commonly used face classifiers based on PCA,ICA and LDA representations. Experiments conducted ona face database containing 206 subjects (2,060 face images)show that the proposed classifier combination approachesoutperform individual classifiers.

1. INTRODUCTION

Human face recognition has a tremendous potential in awide variety of commercial and law enforcement applica-tions. Considerable research efforts have been devoted tothe face recognition problem over the past decade [I]. Al-though there are a number of face recognition algorithmswhich work well in constrained environments, face recog-nition is still an open and very challenging problem in realapplications.

Among face recognition algorithms, appearance-basedapproaches (2][3][4][5] are the most popular. These ap-proaches utilize the pixel intensity or intensity-derived fea-tures. Several such systems have been successfully devel-oped and installed [1][6)17][8]. However, appearance-basedmethods do not perform well in many real-world situations,

This research was supported by NSF IUC on Biometrics (CITeR), atwest Virginia University.

where the query test face appearance is significantly differ-ent from the training face data, due to variations in pose,lighting and expression. Some examples of these varia-tions for one of the subjects in our database are illustrated inFig. 1. While a robust classifier could be designed to handle,any one of these variations, it is extremely difficult for anappearance-based approach to deal with all of these varia-tions. Each individual classifier has different sensitivity todifferent changes in the facial appearance. It has been re-ported that each appearance-based method shows differentlevels of performance on different subsets of images [6],suggesting that different classifiers contribute complemen-tary information to the classification task. A combination'scheme involving different face classifiers, which integratesvarious information sources, is likely to improve the overallsystem performance.

L~ .. ~'. " -

Fig. 1. Facial variations under different lighting conditionsand facial expressions for the same subject [9].

The classifier combination can be implemented at twolevels, feature level and decision level. We use the deci-sion level combination that is more appropriate when thecomponent classifiers use different types of features. Kit-tier [10] provides a theoretical framework to combine vari-ous classifiers at the decision level. Many practical applica-tions of combining multiple classifiers have been developed.Brunelli and Falavigna [11)] presented a person identifica-tion system by combining outputs from classifiers based onaudio and visual cues. Jain et al. [12) integrated multiplefingerprint matchers to develop a robust fingerprint verifica-tion system. Hong and Jain [I 3] designed a decision fusionscheme to combine faces and fingerprint for personal iden-tification. Marcialis and Roli [14] exploited the fusion of

,,~

.l..

Fig. 2. Classifier combination system framework.

PCA and LDA for face verification.We propose two combination strategies, sum rule and

RBF network, to integrate the outputs of three well-knownappearance-based face recognition methods, namely PCA[2], ICA [3] and LDA [4][5]. Our combination strategy isdesigned at the decision level, utilizing all the available in-formation, i.e. a subset of (face) labels along with a confi-dence value, called the matching score provided by each ofthe three face recognition method.

2. CLASSIFIER INTEGRATION

Our combination scheme is illustrated in Fig. 2. While thisframework does not limit the number of component classi-fiers,, we currently use only three classifiers, namely, PCA,ICA and LDA. Following two strategies are provided for in-tegrating outputs of individual classifiers, (i) the sum rule,and (ii) a RBF network as.a classifier, using matching scoresas the input feature vectors.

2.1. Appearance-based Face Classifiers

Three appearance-based classifiers, PCA [21, ICA [3] andLDA 141)[51 have been implemented. In each of these ap-proaches, the 2-dimensional face image is considered as avector, by concatenating each row (or column) of the im-age. Each classifier has 'its own representation (basis vec-tors) of a high dimensional face vector space. By projectingthe face vector to the basis vectors, the projection coeffi-cients are used as the feature representation of each faceimage. The matching score between the test face image andtraining data is calculated as the cosine value of the anglebetween their coefficients vectors.

Let X = (xt,x2,... ,x, .. ,vN) represent the n x N

data matrix, where each xi is a face vector of dimension n,

concatenated from a p x p face image, where pxp = n. Heren represents the total number of pixels in the face image andN is the number of face images in the training set. The meanvector of the training images p = EN= i Xi is subtractedfrom each image vector. All the three representations canbe considered as a linear transformation from the originalimage vector to a projection feature vector, i.e.

Y = WTX, (1)

where Y is the d x N feature vector matrix, d is the di-mension of the feature vector, and W is the transformationmatrix. Note that d < < n.

I. PCA [2]. The Principal Component Analysis basisvectors are defined as the eigenvectors of the scattermatrix ST,

(2)SST = (xI (xi )T

The transformation matrix WPCA is composed of theeigenvectors corresponding to the d largest eigenval-ues. After applying the projection, the input vector(face) in an n-dimensional space is reduced to a fea-ture vector in a d-dimensional subspace.

2. ICA [3]. Bartlett et al. [31 provided two architecturesbased on Independent Component Analysis, statisti-cally independent basis images and a factorial coderepresentation, for the face recognition task. The ICAseparates the high-order moments of the input in ad-dition to the second-order moments utilized in PCA.Both the architectures lead to a similar performance.There is no special order imposed on the ICA basisvectors.

3. LDA [4][5]. The Linear Discriminant Analysis findsa transform WLDA, such that

WTSBW

WLDA = arg maX WT S W ' (3)w WT'S W

where So is the between-class scatter matrix and Swis the within-class scatter matrix, defined as

SB = Ni(xi - p)(xi - )T, (4)i=1

C

Sw = (xk - -)(xk - i)T. (5)i=1 xkeX,

in the above expression, Ni is the number of train-ing samples in class i, c is the number of distinctclasses, Pi is the mean vector of samples belonging toclass i and Xi represents the set of samples belongingto class i.

2.2. Integration Strategy

Kittler [10] analyzed several classifier combination rulesand concluded that the sum rule (defined below) outper-forms other combination schemes based on empirical ob-servations. Unlike explicitly setting up combination rules,it is possible to design a new classifier using the outputs ofindividual classifiers as features to this new classifier. Weadopt the RBF network [15] as this new classifier. Givenm templates in the training set, m matching scores will beoutput for each test image from each classifier. We considerthe following two integration strategies

I. Strategy 1: Sum Rule. The combined matching scoreis calculated as

MScomb = MSPCA + MSICA + MSLDA. (6)

For a given test sample, Output the class with thelargest value of MScomb.

2. Strategy II: RBF network. For each test image, them matching scores obtained from each classifier areused as a feature vector. Concatenating these featurevectors derived from three classifiers results in a fea-ture vector of size 3m. An RBF network is designedto use this new feature vector as the input to generateclassification results. We adopt a 3-layer RBF net-work. The input layer has 3m nodes and the outputhas c nodes, where c is the total number of classes(number of distinct faces). In the output layer, theclass corresponding to the node with the maximumoutput is assigned to the input image. The number ofnodes in the hidden layer is constructed empirically,.depending on the sizes of the input and output layers.

3. EXPERIMENTS AND DISCUSSION

Our database is a collection of four different face databases,available in the public domain (seetable 1). There are 206subjects with 10 images per subject for a total of 2,060 im-ages. Face images selected are near frontal and contain vari-ations in pose, illumination and expression. Some imagesin the individual databases are not selected forour experi-ments; these face images have out-of-plane rotation by morethan 45 degrees in the NLPR+MSU database and face im-ages with occlusions due to sun glasses or a scarf in the ARdatabase. Sample images from the databases are shown inFig. 3. Face images are closely cropped to include only theinternal facial structures such as the eyebrows, eyes, noseand mouth, and aligned by the centers of the two eyes. Allcropped images are resized to 42 x 42. Each image vectoris normalized to be of unit length.

Table 1. Database description.Face database No. of Variations

subjects includedORL [16] 40 Slight pose

and expressionYale [9) 15 Illumination

and expressionAR [17] 120 Illumination

and expression

NLPR+MSU 31 Slight pose(collected by the authors) and expression

Fig. 3. Representative face images in the database. (a) ORL,(b) Yale, (c) AR and (d) NLPR+MSU.

The entire face database is divided into two parts..Nineimages of each subject are used to construct the training dataand the remaining one is used for testing. This partition isrepeated 10 different times so that every image of the sub-ject can be used for testing. The classification accuracy isthe average of these ten different tests.

All the individual classifiers use the cosine value of theangle between the two projection coefficient vectors (onefrom the database image and the other from the test image)as the matchinig score. Database image with the best matchis used to determine the classification of the input image.The sum rule is applied to the matching score outputs ofthe three classifiers. The database image with the maximum

sum score is output as the final result. The recognition ac-

curacies of different face recognition approaches are listed'in table 2. The cumulative match score vs. rank curve [6) is

used to show the performance of each classifier, see Fig. 4.Since our RBF network outputs the final label, no rank in-

formation is available. As a result, we cannot compute thecumulative match score vs. rank curve for RBF combina-tion.

Table 2. Recognition accuracy of different classifiers.PCA ICA LDA Sum rule RBF based79.1% 88.1% 81.0% 90.0% 90.2%

_ . . • .

0.95 0

0

S Suo.... .. m.r.u

" 0.8 -- PCAE 0 ICAU -o- LDA

0.75; -+- Sum rule

0.71 2 3 4 5 6 7 8 9 10Rank

Fig. 4. Cumulative match score vs. rank curve for the sumrule.

Table 2 and figure 4 show that the combined classifiers,based on both the sum-rule and RBF network, outperformeach individual classifier.

4. CONCLUSIONS AND FUTURE WORK

An integration scheme, which combines the output match-ing scores of three well-known face recognition approaches,is proposed to improve the performance of a face identi-fication system. Two combination strategies, sum rule andRBF-based integration, are implemented to combine the out-

put information of three individual classifiers, namely PCA,ICA and LDA. The proposed system framework is scal-able; other face recognition modules can be easily addedinto this framework. Experimental results are encourag-ing, illustrating that both the combination strategies lead tomore accurate face recognition than that made by any oneof the individual classifiers. We are currently investigatingthe weighted sum rule based on the user-specific matchingscore distribution.

5. REFERENCES

[I] W. Zhao, R. Chcllappa, A. Rosenfeld, and P.J. Phillips,"Face recognition: A literature survey," CVL Tech-

nical Report, University of Maryland, Oct. 2000,<ftp://ftp.cfar.umd.edu/TRs/CVL-Reports-2000/TR4;167-zhao.ps.gz>.

[2] M. Turk and A. Pentland, "Eigenfaces for recognition," Jour-nal of Cognitive Neuroscience, vol. 3, no. 1, pp. 71-86, Mar.1991.

[3] M.S. Bartlett, H.M. Lades, and T.J. Sejnowski, "Independentcomponent representations for face recognition," in Proceed-ings of the SPIE, 1998, vol. 3299, pp. 528-539.

[4] P. N. Belhumeur, J. P. Hespanha, and D. J. Kriegman,"Eigenfaces vs. Fisherfaces: Recognition using class spe-cific linear projection," IEEE Trans. Pattern Analysis andMachine Intelligence, vol. 19, no. 7, pp. 711-720, Jul. 1997.

[5] D. L. Swets and J. Weng, "Using discriminant eigenfeaturesfor image retrieval," IEEE Trans. Pattern Analysis and Ma-chine Intelligence, vol. 18, no. 8, pp. 831-836, 1996.

[6] P. Jonathon Phillips, Hyeonjoon Moon, Syed A. Rizvi, andPatrick J. Rauss, "The feret evaluation methodology for face-recognition algorithms," IEEE Trans. Pattern Analysis andMachine. Intelligence, vol. 22, no. 10, pp. 1090-1104, 2000.

[7] ,ldentix, <http://www.identix.comn/>, Minnetonka, MN.

Viisage, <http://www.viisage.com/>, Littleton, MA.

Yale University face database,

<http://cvc.yale.edu/projects/yalefaces/yalefaces.html>.

[101 J. Kittler, M. Hatef, R. Duin, and J. Matas, "On combin-ing classifiers," IEEE Trans. Pattern Analysis and MachineIntelligence, vol. 20, no. 3, pp. 226-239, 1998.

[(11I R. Brunelli and D. Falavigna, "Person identification using.multiple cues," IEEE Trans. Pattern Analysis and MachineIntelligence, vol. 17, no. 10, pp. 955-966, Oct. 1995.

[121 A. K. Jain, S. Prabhakar, and S.'Chen, "Combining multiplematchers for a high security fingerprint verification system,"Pattern Recognition Letters. vol. 20. no. 11-13. pp. 1371-1379, 1999.

[I 31 L. Hong and A.K Jain, "Integrating faces and fingerprint forpersonal identification," IEEE Trans. Pattern Analysis andMachine Intelligence, vol. 20, no. 12, pp. 1295-1307, 1998.

1141 G. L. Marcialis and F. Roli, "Fusion of Ida and pca for faceverification," in Biometric Authentication, Jun. 2002, vol.LNCS 2359, pp. 30-37.

[15] C. M. Bishop, Neural Networks for Pattern Recognition, Ox-ford University Press, UK, 1995.

[16] Ferdinando Samaria and Andy Harter, "Parameterisation of astochastic model for human face identification," in Proc. 2ndIEEE Workshop on Applications of Computer Vision, Sara-sota FL, Dec. 1994.

[17] A.M. Martinez and R. Benavente, "The ar face database,"CVC Tech. Report # 24, Jun. 1998.

I I I I •

2-0,Application No. 11/231,353

Docket No. 577832000200

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE. VOL. 22, NO. 10, OCTOBER 2000

The FERET Evaluation Methodologyfor Face-Recognition Algorithms

P. Jonathon Phillips, Member, IEEE, Hyeonjoon Moon, Member, IEEE,Syed A. Rizvi, Member, IEEE, and Patrick J. Rauss

Abstract-Two of the most critical requirements in support oi producing reliable face-recognition systems are a large database offacial images and a testing procedure to evaluate systems. The Face Recognition Technology (FERET) program has addressed bothissues through the FERET database of facial images and the establishment of the FERET tests: To date, 14,126 images from1,199 individuals are included in the'FERET database, which is divided into development and sequestered portions of the database. InSeptember. 1996, the FERET program administered the third in a series of FERET lace-recognition tests. The primary objectives of thethird test were to 1) assess the state of the art, 2) identify future areas of research, end 3) measure algorithm performance.

Index Terms-Face recognition, algorithm evaluation, FERET database.

1 INTRODUCTION

O VER the last decade, face recognition has become anactive area of research in computer vision, neuroscience,

and psychology. Progress has advanced to the point that;"face-recognition systems are being .demonstrated in real-world settings [6]. The rapid development of face recognitionis due to a combination of factors: active development ofalgorithms, the availability of a large database of facialimages, and a method for evaluating the performance of face-recognition algorithms. The FERET database and evaluationmethodology address the latter two points and are de factostandards. There have been three FERET evaluations, withthe most recent being the September 1996 FERET test.

The September 1996 FERET test provides a comprehensivepicture of the state-of-the-art in face recognition from stillimages. This was accomplished by evaluating the algorithms'ability on different scenarios, categories of images, andversions of algorithms. Performance was computed foridentification arid verification scenarios. In an identificationapplication, an algorithm is presented with an unknown facethat is to be identified, whereas, in a verification application,an algorithm is presented with a face and a claimed identity,and the algorithm either accepts or rejects the claim. In thispaper, we describe the FERET database and the September.1996 FERET evaluation protocol and present identificationresults. Verification results are presented in Rizvi et al. [10].

* P.]. Phillips is with the National Institute of Standards and Technology,100 Bureau Dr. STOP 8940, Gaithersburg, MD 20899-8940.E-mail: [email protected].

* H. Moon is with Lau Technologies, 30 Porter Road, Littleton, MA 01460.E-mail: [email protected].

* S.A. Rizvi is with the Department of Engineering Science and Physics,College of Staten Island/CUNY, Staten Island, NY 10314.E-mail: [email protected].

* P.J. Rauss is with the Army Research Laboratory, 2800 Powder Mill Road,Adelphi, MD 20783-1197. E-mail: [email protected].

Manuscript received 2 Nov. 1998; revised 24 Sept. 1999; accepted 19 May2000.Recommended for acceptance by D.J. Kriegman.For information on obtaining reprints of this article, please send e-mail to:[email protected], and reference IEEECS Log Number 108156:

The FERET tests model the following face recognitionapplications: identification from large law enforcementdatabases and verification from biometric signatures storedon smart cards. For both applications, there are a limitednumber of facial images per person and the face represen-tation is learned (or decided) prior to people being enrolledin the system.

In the Federal Bureau of Investigation's (FBI) IntegratedAutomated Fingerprint Identification System (IAFIS), theonly required mugshot is a full frontal image [2]. The IAFISstores digital fingerprints and mugshots and will be themain depository of criminal fingerprints and mugshots inthe United States. Other examples of large databases withone image per person are photographs from driverslicenses, passports, and visas.

When the IAFIS is fully operational, it is expected toreceive 5,000 mugshots per day (1,800,000 per year).Because of the large number of mugshots, it is not practicalto continually update the representation. Updating therepresentation would require training from millions of facesand updating millions of database records.

For verification applications where biometric signaturesare stored on smart card, a user inserts a smart card into anelectronic reader and provides a new biometric signature tothe system. The system then reads the biometric signaturestored on the smart card and compares it with the. newsignature. Based on the comparison, the claimed identity iseither accepted or rejected. Because of the limited amount ofstorage space, a facial image cannot be stored on a smartcard and a representation of the face must be stored. Thus,once the first person is enrolled in the system, it is notpossible to update the facial represeritation. Also, because oflimited storage space, the representation of only one facialimage is stored on a smart.card.

The FERET was a general evaluation designed tomeasure performance of laboratory algorithms on theFERET database. The main goals of the FERET evaluation.were to assess the state-of-the-art and the feasibility ofautomatic face recognition. Thus, the FERET test did not

0162-8828/00/510.00 0 2000 IEEE

V;

1090

PHILLIPS ET AL.: THE FERET EVALUATION METHODOLOGY FOR FACE-RECOGNITION ALGORITHMS

explicitly measure the effect on performance of individualcomponents of an algorithm nor did the test measureperformance under operational scenarios. An operationaltest evaluates algorithms in an orderly and scientificmanner under all conditions in which a system will operate.

To obtain a robust assessment of performance, algorithmswere evaluated against different categories of images. Thecategories were broken out by a lighting change; peoplewearing glasses, and the time between the acquisition dateof the database image and the image presented to thealgorithm. By listing performance in these categories, abetter understanding of the face recognition field in general,as well as the strengths and weakness of individualalgorithms is obtained. This detailed analysis helps assesswhich applications can be successfully addressed.

All face recognition algorithms known to the authorsconsist of two parts: 1) face.localization and normalizationand 2) face identification. We use the term face localizationand normalization to differentiate it from face detection. Indetection, the task is to find all faces in an image, wherethere can be multiple or no faces in the image. In theFERET evaluation, there is one face in an image. In the firstpart of an algorithm, the face is located in an image andthen the face is normalized into a standard position for therecognition portion of the algorithm. Usually, normal-ization requires that a set of facial features is actuallylocated to within a couple of pixels.

Algorithms that consist of both parts are referred to asfully automatic algorithms and those that consist of only thesecond part are partially automatic algorithms. (A glossary ofterms is in the Appendix.) The September 1996 testevaluated both fully and partially automatic algorithms.Partially automatic algorithms are given a facial image andthe coordinates of the centers of the eyes. Fully automaticalgorithms are only given facial images.

The availability of the FERET database and evaluationmethodology has made a significant difference in theprogress of development of face-recognition algorithms.Before the FERET database was created, a large number ofpapers reported outstanding recognition results (usually> 95 percent correct recognition) on limited-size databases'(usually < 50 individuals). (In fact, this is still true.) Onlya few of these algorithms reported results on imagesutilizing a common database, let alone met the desirablegoal.of being evaluated on a standard testing protocol thatincluded separate training and testing sets. As a conse-quence, there was no method to make informed compar-isons among various algorithms.

The FERET database has made it possible for researchersto develop algorithms on a common database and to reportresults in the literature using this database. Results reportedin the literature do not provide a direct comparison amongalgorithms because each researcher reports results usingdifferent assumptions, scoring methods, and images. Theindependently administered FERET test allows for a directquantitative assessment of the relative strengths andweaknesses of different approaches.

More importantly, the FERET database and tests clarifythe current state of the art in face recognition and point outgeneral directions for future research. The FERET tests

allow the computer vision community to assess overallstrengths and weaknesses in the field, not only on the basisof the performance of an individual algorithm, but also onthe aggregate performance of all algorithms tested. Throughthis type of assessment, the community learns in an openmanner of the important technical, problems to be ad-dressed and how the community is progressing towardsolving these problems.

2 BACKGROUND

The first FERET tests took place in August 1994 andMarch 1995 (for details of these tests and the FERET databaseand program, see Phillips and Rauss [61, Phillips et al. [7], andRauss et al'. [8]). The FERET database collection began inSeptember 1993 along with the FERET program.

The August 1994' test established, for the first time, aperformance baseline for face-recognition algorithms. Thistest was designed to measure performance on algorithmsthat could automatically locate, normalize, and identify'faces from a database. The test consisted of three subtests,each with a different gallery and probe set. The gallerycontains the set of known individuals. An image of anunknown face presented to the-algorithm is called a probe,and the collection of probes is called the probe set. Sincethere is only one face iran image, sometimes "probe" refersto the identity of the person in a probe image. The firstsubtest examined the'ability of algorithms to recognize facesfrom a gallery of 316 individuals. The second was the false-alarm test, which measured how well an algorithm rejectsfaces not in the gallery. The third baselined the effects ofpose changes on performance.

The second FERET test, which took place in March 1995,measured progress .since August 1994. and evaluatedalgorithms on larger galleries. The March 1995 evaluationconsisted of a single test with a gallery of 817 knownindividuals. One emphasis of the test was on probe sets thatcontained duplicate probes. A duplicate probe is usually animage of a person whose corresponding gallery image wastaken on a different day. (Technically, the probe and galleryimages were from different image sets; see description ofthe FERET database below.)

The FERET database is designed to advance the state ofthe art in face recognition, with the collected imagesdirectly supporting both algorithm development and theFERET evaluation tests. The database is divided into adevelopment set, provided to researchers, and a set ofsequestered images for testing. The images in the develop-ment set are representative of the sequestered images.

The facial images were collected in 15 sessions betweenAugust 1993 and July 1996. Collection sessions lasted one ortwo days. In an effort to maintain a degree of consistency,throughout the database, the same physical setup .andlocation was used in each photography session. However,because the equipment' had to be reassembled for eachsession, there was variation from session to session (Fig. 1).

Images of an individual were acquired in sets of 5 to 11images. Two frontal views were taken (fa and fb); a differentfacial expression was requested for the second frontal image.For 200 sets'of images, a third frontal image was taken with adifferent camera and different lighting (this is referred to as

1091


duplicate I fc duplicate II

Fig. 1. Examples of different categories of probes (image). The duplicate I

fa images were taken at least one year apart.

the fc image). The remaining images were collected at variousaspects between right and left profile. To add simplevariations to the database, photographers sometimes took asecond set of images for which the subjects were asked to puton their glasses and/or pull their hair back. Sometimes asecond set of images of a person was taken on a later date;such a set of images is referred to as a duplicate set. Suchduplicates sets result in variations in scale, pose, expression,and illumination of the face.

By July 1996, 1,564 sets of images were in the database,consisting of 14,126 total images. The database contains.1,199 individuals and 365 duplicate sets of images. For somepeople, more than two years elapsed between their first andmost recent sittings, with some subjects being photo-graphed multiple times (Fig. 1). The development portionof the database consisted of 503 sets of images and wasreleased to researchers. The remaining images weresequestered.

3 TEST DESIGN

3.1 Test Design Principles.The FERET September 1996 evaluation protocol wasdesigned to assess the state of the art, advance the state ofthe art, and point to future directions of research. Tosucceed at this, the evaluation design must solve the threebears problemh. The test cannot be too hard nor too easy. If thetest is too easy, the testing process becomes an exercise in"tuning" existing algorithms. If the test is too hard, the testis beyond the ability of existing algorithmic techniques. Theresults from the test are poor and do not allow for anaccurate assessment of algorithmic capabilities.

The solution to the three bears problem is through theselection of images used in the evaluation and theevaluation protocol. Tests are administered using anevaluation protocol that states the mechanics of the testsand the manner in which the test will be scored. In facerecognition, the protocol states the number of images ofeach person in the test, how the output from the algorithmis recorded; and how the performance results are reported.

The characteristics and quality of the images are majorfactors in determining the difficulty of the problem beingevaluated. For example, if faces are in a predeterminedposition in the images, the problem is different from that forimages in which the faces can be located anywhere in the

image was taken within one year of the fa image and the duplicate II and

image. In the FERET database, variability was introducedby the inclusion of images taken at different dates andlocations (see Section 2). "This resulted in changes inlighting, scale, and background.

The testing protocol is based on a set of. designprinciples. The design principles directly relate the evalua-tion to the face recognition problem being evaluated.. ForFERET,, the applications are searching large databases andverifying identities stored on smart cards. Stating the designprinciples allows one to assess how appropriate theFERET test is for a particular face recognition algorithm.Also, design principles assist in determining if an evalua-tion methodology for testing algorithm(s) for a particularapplication is appropriate. Before discussing the designprinciples, we state the evaluation protocol.

In the testing protocol, an algorithm is given two sets ofimages: the target set and the query set. We introduce thisterminology to distinguish these sets from the gallery andprobe sets that are used in computing performancestatistics. *For all results in this paper, the images in thegalleries and probe sets were distinct. The target setis givento the algorithm as the set of known facial images. Theimages in the query set consist of unknown facial images tobe identified. For each image qi in the query set Q, analgorithm reports a similarity s,(k) between qi and eachimage tk in the target set T. The testing protocol is designedso that each algorithm can use a different similaritymeasure and we do not compare similarity measures fromdifferent algorithms. The key property of the new protocol,which allows for greater flexibility in scoring, is that, forany two images qi and tk, we know si(k).

Multiple galleries and probe sets can be constructed fromthe target and query sets. A gallery g is a subset of the targetset. Similarly, a probe set P is a subset of the query set. For agiven gallery g and probe set P, the performance scores arecomputed by examination of similarity measures s1(k) suchthat qi E P and tk E 9.

Using target and query sets allows us to computeperformance for different categories of images. Possibleprobe categories include: 1) gallery and probe images takenon the same day, 2) duplicates taken within a week of thegallery image, and 3) duplicates where the time between theimagesis atleastone year. We cancreate a gallery of 100 peopleand estimate an algorithm's performance by. recognizingpeople in this gallery. Using this as a starting point, we can

fa

1092

1093'PHILLIPS ET AL.: THE FERET EVALUATION METHODOLOGY FOR FACE-RECOGNITION ALGORITHMS

Target set

Fig. 2. Schematic of the FERET testing procedure.

then create galleries of 200,300, ...1, 000 people and deter-Smine how performance changes as the size of the galleryincreases. Another avenue of investigation is to createn different galleries of size 100 and calculate the variation inalgorithm performance with the different galleries.

We now list the three design principles. First, all faces inthe target set are treated as unique faces. This allows us toconstruct multiple galleries with one image per person. Inpractice, this condition is enforced by giving every image inthe target and query set a unique random identification.

The second design principle is that training is completedprior to the start of the test. This forces each algorithm tohave a general representation for faces, not a representationtuned to a specific gallery. The third design rule is that allalgorithms compute a similarity measure between allcombinations of images from the targetand query sets.

3.2 Test DetailsIn the September 1996 FERET test, the target set contained3,323 images and the query set 3,816 images. The target setconsists of fa and fb frontal images. The query set consistedof all the images in the target set plus the fc, rotated images,and digitally modified images. The digitally modifiedimages in the query set were designed to test the effectsof illumination and scale. (Results from the rotated anddigitally modified images are not reported here.) All theresults reported in this article are generated from galleriesthat are subsets of this target set and probe sets that aresubsets of this query set. For each query image q,, analgorithm outputs the. similarity measure s,(k) for allimages tk in the target set. For a given query image q;, thetarget images tk are sorted by the similarity scores s;(.).,Since the target 'set is a subset of the query set, the testoutput contains the similarity score between all images inthe target set. (Note: Having the target set as subset of thequery set does not constitute training and testing on the.same images. This is because the face representation islearned prior to the start of the test.)

There were two versions of the September 1996 test. Thetarget and query sets were the same for each version. Thefirst version tested partially automatic algorithms byproviding them with a list of images in the target and

query sets and the coordinates of the centers of the eyes forimages in the target and query sets. In the second version ofthe test, the coordinates of the eyes were not provided. Bycomparing the performance between the two versions, weestimate performance of the face-locating portion of a fullyautomatic algorithm at the system level.

The test was administered at each group's site under thesupervision of one of the authors. Each group. had threedays to complete the test on less than 10 UNIX workstations(this limit was not reached). We did not record the time ornumber of workstations because execution times can varyaccording to the type of machines used, machine andnetwork configuration, and the amount of time that thedevelopers spent optimizing their code (we wanted toencourage algorithm development, not code optimization).We imposed the time limit to encourage the development ofalgorithms that could be incorporated into operational,fieldable systems.

The target and query sets consisted of images from boththe developmental and sequestered portions of the FERETdatabase. Only images from the FERET database wereincluded in the test; however, algorithm developers werenot prohibited from using images outside the FERETdatabase to develop or tune parameters in their algorithms.

The FERET test is designed to measure laboratoryperformance. The test is not concerned with speed of theimplementation, real-time implementation issues, andspeed and accuracy trade-offs. These issues and othersneed to be addressed in an operational, fielded system andwere beyond the scope of the September 1996 FERET test.

Fig. 2 presents a schematic of the testing procedure. To,ensure that matching was not done by file name, we gavethe images random names. A rough estimate of the pose ofeach face was provided to each testee. Example poseestimates provided.vere: frontal, and quarter and half right.

4 DECISION THEORY AND PERFORMANCEEVALUATION

The basic models for evaluating the performance of an, algorithm are the closed and open universes. In the closed

universe, every probe is in the gallery. In an open universe,some probes are not in the gallery. Both models reflect

II

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE. VOL. 22, NO. 10, OCTOBER 2000

TABLE 1Representation and Similarity Metric for Algorithms Evaluated

Algorithm Represenation Similarity measure

Excalibur Co. Unknown UnknownMIT Media Lab 95 PCA L2

MIT Media Lab 96 PCA-difference space MAP Bayesian StatisticMichigan St. U. . Fischer discriminant L2

Rutgers U.. Greyscale projection Weighted L 1

U. of So.: CA. Dynamic Link.Architecture Elastic graph matching

(Gabor Jets)U. of MD 96 Fischer discriminant L2

U. of MD 97 Fischer discriminant Weighted L 2

Baseline PCA LiBaseline Correlation Angle

different and important aspects of face-recognition algo-rithms and report different performance statistics. The openuniverse model is used to evaluate verification applications.The FERET scoring procedures for verification is given inRizvi et al. [10].

The closed-universe model allows one to ask how goodan algorithm is at identifying a probe image; the question isnot always "is the top match correct?" but "is the correctanswer in the top n matches?" This lets one know howmany images have to be examined to get a desired level ofperformance. The performance statistics are reported ascumulative match scores, which are plotted on a graph. Thehorizontal axis of the graph is rank and the vertical axis isthe probability of identification, (PI) (or percentage ofcorrect matches).

The computation of an identification score is quitesimple. Let P be .a probe set and 11 be the size of P. Wescore probe set P against gallery 9g, where 9 = {g,..., g})and P= {pl,... ,p,v}, by comparing the similarity scores s&()such that pi E P and gk E G. For each probe image pi E P,we sort si(.) for all gallery images gk E 9. We assume that asmaller similarity score implies a closer match. The functionid(i) gives the index of the gallery image of the person inprobe pi, i.e., pi is an image of the person in gid(i). A probe piis correctly identified if si(id(i)) is the smallest score for9k E 9. A probe pi is in the top n if si (id(i)) is one of the nthsmallest scores s(') for gallery G. Let Rr, denote the numberof probes in the top n. We reported R,z/IPI, the fraction ofprobes in the top n.

In reporting identification performance results, we statethe size of the gallery and the number of probes scored. Thesize of the gallery is the number of different faces (people)contained in the images that are in the gallery. For all resultsthat we report, there is one image per person in the gallery;thus, the size of the gallery is also the number of images inthe gallery. The number of probes scored (also, size of theprobe set) is 1I1. The probe set may contain more than oneimage of a person and the probe set may not contain animage of everyone in the gallery. Every image in the probeset has a corresponding image in the gallery.

5 LATEST TEST RESULTS

The September 1996 FERET test was designed to measurealgorithm performance for identification and verificationtasks. In this article, we' report identification results.Verification results are reported in Rizvi et al. [9), [10]. Wereport results for 12 algorithms that include 10 partiallyautomatic algorithms and two fully automatic algorithms.The test was administered in September 1996 andMarch 1997 (see Table 1 for the representation andsimilarity metric for each algorithm and Table 2 for detailsof when the test was administered to which groups and.which version of the test was taken). Two of thesealgorithms were developed at the MIT Media Laboratory.The first was' the same algorithm that was tested inMarch 1995. This algorithm was retested so that improve-ment since March 1995 could be measured. The secondalgorithm was based on more recent work [3], [4).Algorithms were also tested from Excalibur Corporation(Carlsbad, California), Michigan State University (MSU)

.[111 [161, Rutgers University [13], the University of South-ern California (USC) 114], and two from the University ofMaryland (UMD) [1, [15], [16). The first algorithm fromUMD was tested in September 1996 and a second version ofthe algorithm was tested in March 1997. For the fullyautomatic version of the test, algorithms from MIT and USCwere evaluated.

The final two algorithms were our implementation ofnormalized correlation and a principal components analysis(PCA) based algorithm [5], [121. These algorithms prcovide aperformance baseline. In our implementation of the PCA-based algorithm, all images were 1) translated, rotated, andscaled so that the.centers of the eyes were placed on specificpixels, 2) faces were masked to remove background and hair,and 3) the nonmasked facial pixels were processed by ahistogram equalization algorithm. The training set consistedof 500 faces. Faces were represented by their projection ontothe first 200 eigenvectors and were identified by a nearest-neighbor classifier using the L1 metric. For normalizedcorrelation, the images were 1) translated, rotated, and scaledso that the centers of the eyes were placed on specific pixelsand 2) faces were masked to remove background and hair.

1094


TABLE 2List of Groups That Took the September 1996 Test Broken Out by Versions Taken and

Dates Administered (the 2 by MIT Indicates that Two Algorithms were Tested)

Test DateSeptember March

Version of test Group 1996 1997 BaselineFully Automatic MIT Media Lab (3,4) *

U. of So. California (14] *Eye Coordinates Given Baseline PCA (5,12] *

Baseline Correlation *Excalibur Corp. *MIT Media Lab 2

Michigan State U. (11,16])Rutgers U. (13] *

U. Maryland [1,15,16) * *USC *

We report identification scores for four categories ofprobes. For three of the probe categories, performance wascomputed using the same gallery. For the fourth category, asubset of the first gallery was used. The first gallery consistedof images of 1,196 people with one image per person. For the1,196 people, the target and query sets contain fa andfb images from the same set. (The FERET images werecollected in sets and, in each session, there are two frontalimages, fa and fb, see Section 2.) One of these images wasplaced in the gallery and the other was placed in the FB probeset. The FB probes were the first probe category. (Thiscategory is denoted by FB to differentiate it from the fb imagesin the FERET database.) (Note: the query set contained all theimages in the target set, so the probe set is a subset of the queryset.) Also, none of the faces in the gallery images wore glasses.Thus, the FB probe set consisted of probe images taken on thesame day and under the same illumination conditions as thecorresponding gallery image.

The second probe category contained all duplicatefrontal images in the FERET database for the galleryimages. We refer to this category as the duplicate I probes.The third category was the fc probes (images taken thesame day as the corresponding gallery image, butwith adifferent camera and lighting). The fourth category con-sisted of duplicates where there was at least one yearbetween the acquisition of the probe image and correspond-ing gallery image, i.e., the gallery images were acquiredbefore January 1995 and the probe images were acquiredafter January 1996. We refer to this category as theduplicate II probes. The gallery for the FB, duplicate 1,and fc probes was the same. The gallery for duplicate IIprobes was a subset of 864 images from the gallery for theother categories.

5.1 Partially Automatic AlgorithmsIn this section, we report results for the partially automaticalgorithms. Table 3 shows the categories corresponding tothe figures presenting the results, type of results, and size ofthe gallery and probe sets (Figs. 3, 4, 5, and 6). The resultsfor each probe category are presented on two graphs. Onegraph shows performance, for algorithms tested in

September 1996 and the baseline algorithms. The othershows performance for algorithms tested in March 1997, thebaseline algorithms, and the UMD algorithm tested in

SSeptember 1996 (this shows improvement between tests).(The results are reported as cumulative match scores.)

In Figs. 7 and 8, we compare the difficulty of different

probe sets. Whereas Figs. 4, 5, and 6 report identification

Sperformance for each algorithm, Fig. 7 shows a single curvethat is an average of the identification performance of all

algorithms for each probe category: For example, the firstranked score for duplicate I probe sets is computed from an

average of the first ranked score for all algorithms in Fig. 4.

In Fig. 8, we presented current upper bound for perfor-

mance on partially automatic algorithms for each probe

category. For each category of probe, Fig. 8 plots thealgorithm with the highest top rank score (R1). Figs.7 and 8

report performance of four categories of probes, FB,duplicate I, fc, and duplicate II.

5.2 Fully Automatic PerformanceIn this section, we report performance for the fully

automatic algorithms of the MIT Media Lab and USC. Toallow for a comparison between the partially and fully

automatic algorithms, we plot the results for the partiallyand fully automatic algorithms from both institutions. Fig: 9

shows performance for FB probes and Fig. 10 shows

performance for duplicate I probes. (The gallery and probesets are the same as in Section 5.1.)

TABLE 3Figures Reporting Results for Partially Automatic Algorithms

Performance is Broken Out by Probe Category

Figure no. Probe Category Gallery size Probe set size3 FB 1196 11954 duplicate I 1196, 7225 fc . 1196 1946 duplicate II 864 234

1095

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL..22, NO. 10, OCTOBER 2000

10 15 20 25' 30 35 40 45 50Rank

(a)

5 10 15 20 25Rank

30 35. 40 45 50

"(b)

Fig. 3. Identification performance against FB probes. (a) Partially automatic algorithms tested in September 1996. (b) Partially automatic algorithms

tested in March 1997.

5.3 Variation in PerformanceFrom a statistical point of view, a face-recognition algorithmestimates the identity of a face. Consistent with this view,we can ask about the change in performance of .an

algorithm: "For a given category of images, how doesperformance change if the algorithm is given a differentgallery and probe set?" In Tables 4 and 5, we show howalgorithm performance varies if the people in the gallerieschange. For this experiment, we constructed six galleries of

approximately 200 individuals, in which an individual was

in only one gallery. (The number of people contained withineach gallery versus the number of probes scored is given inTables 4 and 5.) Results are reported for the partiallyautomatic algorithms. For the results in this section, weorder algorithms by their top rank score on each gallery; for

example, in Table 4,'the UMD March 1997 algorithm scoredhighest on gallery 1 and the baseline PCA and correlationtied for ninth place. Also. included in this table is average

1

0.95

0.9

0.85

0.8

0 75

0.95

0.9

0.85

0.8.

0.75

UMD9 *.o7 AC-'.-

UMU 90Baseline Cor --Baseline EF --.- x---

I I I I 1 I I i I

1096

E


5 10 15 20 25 30 35 40Rank

(a)

45 50

"- a -a ""- -s 0 1.. "O©O.a -I

5 10 15 20 25Rank

USC -.. -.UMD 97 ...... O

Baseline EF ---X--- -Baseline Cor -+-

UMO 96-"

35 40 45 50

(b)

Fig. 4. Identification performance against all duplicate I probes. (a) Partially automatic algorithms tested in September 1996. (b) Partially automatic

algorithms tested in March 1997.

performance for all algorithms. Table 4 reports results for corresponding duplicate probes. No scFB probes. Table 5 is organized in the same manner as Table 5 for gallery 6 because there are r

Table 4,.except that duplicate I probes are scored. Tables 4 gallery.and 5 report results for the same gallery. The galleries were

constructed by placing images within the galleries by 6 DISCUSSION AND CONCLUSIONchronological order in which the images were collected(the first gallery contains the first images collected and the In this paper, we presented. the Sept

sixth gallery contains the most recent images collected). In evaluation protocol for face recognitiTable 5, mean age refers to the average time between protocol was designed so that pe

collection of images conkained in the gallery and the measured on different galleries and

ores are reported in

no duplicates for this

tember 1996 FERET

on algorithms. Therforinance can be

probe sets and on

0.8

0.7

0.6

0.5

0.4

0.3

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

.

1097

r


15 20 25Rank

30 35 40 45 50'

Fig. 5. Identification performance against fc probes. (a) Partially automatic algorithms tested in September 1996. (b) Partially automatic algorithms

tested in March 1997.

identification and verification tasks. (Verification resultsmentioned in this section are from Rizvi et al. [9], [10].)

The September 1996 test was the latest FERET evaluation(the others were the August 1994 and March 1995 tests.[7]).One of the main goals of the FERET evaluations was toencourage and measure improvements in the performance offace recognition algorithms, which is seen in the September1996 FERET test. The first case is the improvement inperformance of the MITMedia Lab September 1996 algorithmover the March 1995 algorithm; the second is the.

improvement of the UMD algorithm between September1996 and March 1997.

By looking at progress over the series of FERETevaluations, one sees that substantial progress has beenmade in face recognition. The most direct method is tocompare the performance of,fully automatic algorithms onfb probes (the two earlier FERET evaluations onlyevaluated fully automatic algorithms). The best top rankscore for fb probes on the August 1994 evaluation was78 percent on a gallery of 317 individuals and, for

V 9 f ',.UM 96

Baseline EF --- x--Baseline Cor --

5 10

1098

I ........


.. a pp .. O -

fi e ... _I!"lkdli0 " p '3 g

- -K.. i!** -- 00,CB I 'f.. ... 477 7OC"

MIT 96 -.-. -Baseline EF --- x---'Excalibur

Rutgers -.- .-MIT 95 ...... ......

Baseline Cor ---UJMD 96... -

MSU

5 10 15 20 25 30 35 40 45Rank

(a)

0.8

0.2

0

5 10 15 20 25Rank

Fig. 6. Identification performance against duplicatealgorithms tested in March 1997.

March 1995, the top score was 93 percent on a gallery of 831individuals (7). This compares to 87 pecent in September1996 and ,95 percent in March 1997 (gallery of 1,196individuals). This method shows, that over the course ofthe FERET evaluations, the absolute scores increased as thesize of the database increased. The March 1995 score wasfrom one of the MIT Media Lab algorithms and representsan increase from 76 percent in March 1995.

On duplicate I probes, MIT Media Lab improved from39 percent '(March 1995) to 51 percent (September 1996);

30 35 40 45 50

II probes. (a) Partially automatic algorithms tested in September 1996. (b) Partially automatic

USC's performance remained approximately the same at57-58 percent between March 1995 and March 1997. Thisimprovement in performance. was achieved while thegallery size increased and the number of duplicate I probesincreased from 463 to 722. While increasing the number ofprobes does not necessarily increase the difficulty ofidentification tasks, we argue that the September 1996duplicate I probe set was more difficult to process than theMarch 1995 set. The September 1996 duplicate I probe setcontained the duplicate II probes and the March 1995

e

eUU1

a WWI

USC -"-*-Baseline EF ---x---

. UMD 97 aBaseline Cor ----

UMD 96 ..

x-X-k' "' E:i!-)-?'X- it "K" "-" -k x x

1099

0


0 S 10 15 20 25Rank

35 40 45

Fig. 7. Average identification performance of partially automatic algorithms on each probe category.

duplicate I probe set did not contain a similar class ofprobes. Overall, the duplicate II probe set was the mostdifficult probe set.

Another goal, of the FERET evaluations is to identifyareas of strengths and weaknesses in the field of facerecognition. We addressed this issue by reporting perfor-mance for multiple galleries and probe sets and differentprobe categories. From this evaluation, we concluded thatalgorithm performance is dependent on the gallery andprobe sets. We observed variation in performance due tochanging the gallery and probe set within a probe category

a0LiN,

LiV-'U

a,

'U

'U" "

0.4

0.2

0

0 5 10 15 20 25Rank

and by changing probe categories. The effect of changingthe gallery while keeping the probe category constant isshown in Tables 4 and 5. For fb probes, the range forperformance is 80 percent to 94 percent; for duplicate Iprobes, the range is 24 percent to 69 percent. Equallyimportant, Tables 4 and 5 show the variability in relativeperformance levels. For example, in Table 5, UMD

September 1996 duplicate performance varies betweennumber three and nine, while at the same time' there arealgorithms that consistently outperform other algorithms.Of the algorithms tested in September 1996, the

30 35 40 45 SO

Fig. B. Current upper bound identification performance of partially automatic algorithm for each probe category.

0.8

0V

U

Li4Jf

E

U

0.6

0.4

Duplicate II probes -o-.

; fflfl

-x-X°

'* e O e a.O. .'u --

o eo 'O

-4.0

X O'0/0J

0.2

n

" ,.?4=rt--K-X-X... X .f - X-7 j 277- )(-- -',, -% -i -) o- X-... . . .- X(- XX - - 4

F -'

. '. F ' ' '-

. FB probes -- +--. e '.o fc probes ---x---

0o ." ouplicate I probes .*.. .,Duplicate II probes - .

--

1 e 1 i '1 1

1 1 I I I

1100

FB probes -

Duplicate.I probes

p

E

V

A I A I J m

0

w


.5 10 15 20 25Rank

.35 40 45

Fig. 9. Identification performance of fully automatic algorithms against partially automatic algorithms for FB probes.

September 1996 MIT algorithm, clearly outperformed theother algorithms. In addition, the September 1996 MITalgorithms and the algorithms tested in March 1997 (UMD

March 1997 and USC) outperformed the other algorithmstested. This shows that, despite the overall variation inperformance, definite conclusions about algorithm perfor-mance can be made. These conclusions are consistent withFigs. 4, 5, and 6.

The variation in Tables 4 and 5 is because traditionalmethod of calculating error bars and confidence regions donot apply to face recognition. These traditional methods

0toL

U

N

"r4.,

N

E

require that each run of the decision problem be made withthe same classes, .i.e., character recognition with the26 letters in the English alphabet. However, in facerecognition, changing the people in the gallery changesthe underlying classification problem. (Remember, each

person is a different class.) Computing error bars withdifferent people in the gallery is equivalent to computingerror bars for a character recognition .system using

performance from different sets of characters.Similar results were found in Moon and Phillips [5] in

their study of principal component analysis-based face

10 15 20 25 30Rank .

35 40 45 50

Fig. 10. Identification performance of fully automatic algorithms against partially automatic algorithms for duplicate I probes.

1101

0.98

0.96

0.94

0.92

0.9

0.88

0.86

0.84

0.82

08

. 4 -:- X X '

-X- X- X- X- K- NI"!

- -x- "

x USC partially automatic.---*-..x MIT partially automatic ---

USc fully automatic "e-.- MIT fully automatic --- x---

x

, I I I I.

0.0

I I I I I I I I I

SIEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 22, NO. 10, OCTOBER 2000

TABLE 4Variations in Identification Performance on Six Different Galleries on FB Probes

Images in Each of the Galleries do not Overlap, Ranks Range from 1-10

Algorithm Ranking by Top MatchGallery Size /Scored Probes

200/200 200/200 200/200 200/200 200/199 196/196Algorithm gallery 1 gallery 2 gallery 3 gallery 4 gallery,.5 gallery 6

Baseline PCA 9 10 8: 8 10 8Baseline correlation 9 9 9 6 9 10

Excalibur Corp. 6 7 . 7 5 7 6MIT Sep96 , 4 2 1 1 3 3MIT Mar95 7 5 4 4 5 7

Michigan State Univ. 3 4 5 8 4 4Rutgers Univ. 7 8 9 6 7 9UMD Sep96 4 6 6 10 5 5UMDMar97 1 1 3 2 2 1

USC 2 3 2 2 1 1Average Score 0.935 0.857 0.904 - 0.918 0.843 0.804

recognition algorithms. This shows that an area of futureresearch is measuring the effect of changing galleries andprobe sets and statistical measures that characterize thesevariations.

Figs. 7 and 8 show probe categories characterized bydifficulty. These figures show that fb probes are the easiestand. duplicate I1I probes are the most difficult. On average,duplicate I probes are easier to.identify than fc probes.However, the best performance on fc probes is significantlybetter ,than the best performance on duplicate I and[I probes. This comparative analysis shows that futureareas of research include processing of duplicate I probesand developing methods to compensate for changes inillumination.

The scenario being tested contributes to algorithmperformance. For identification, the MIT Media Lab

algorithm was clearly the best algorithm tested in September1996. However, for verification, there was not an algorithmthat was a top performer forall probe categories. Also, for thealgorithms tested in March 1997, the USC algorithmperformed overall better than the UMD algorithm foridentification; however; for verification, UMD overall per-formed better. This shows. that performance on one task is notnecessarily predictive of performance on a different task.

The September 1996 FERET evaluation shows that definiteprogress is being made in face recognition and that the upperbound in performance has not been reached. The improve-ment in performance documented in this paper showsdirectly that the FERET series of evaluations. has made asignificant contribution to face recognition. This conclusion isindirectly supported by 1) the improvement in performancebetween the algorithms tested in September 1996 and

TABLE 5Variations in Identification Performance on Five Different Galleries on Duplicate Probes

Images in Each of the Galleries do not Overlap, Ranks Range from 1-10

Algorithm Ranking by Top MatchGallery Size / Scored Probes

200/143 200/64 200/194 200/277 200/44Mean Age of Probes (months) 9.87 3.56 5.40 10.70 3.45

Algorithm gallery 1 gallery 2 gallery 3 gallery 4 gallery 5S Baseline PCA 6 . 10 5 5 9

Baseline correlation 10 7 ' 6 6 8Excalibur Corp. 3 5 4 4 3

MIT Sep96 2 1 2 2 3MIT Mar95 7 4 7 8 10

Michigan State Univ.. 9 6 8 10 6Rutgers Univ. 5 7 10 7 6UMD Sep96 7 9 9 9 3UMD Mar97 4 2 3 3 1

USC 1 3 1 1 1Average Score 0.238 0.620 0.645 0.523 0.687

1102


March 1997, 2) the number of papers that use FERET imagesand report experimental results using FERET images, and3) the number of groups that participated in the September1996 test.

APPENDIX

GLOSSARY OF TECHNICAL TERMS

Duplicate. A probe image of a person whose correspondinggallery image was taken from a different image set.'Usually, a duplicate is taken on a different day than thecorresponding gallery image.

Duplicate I probes. Set of duplicate probes for a gallery.

Duplicate II probes. Set of duplicate probes where there isat least one year between the acquisition of thecorresponding probe and gallery images.

FB 'probes. Probes taken from the same image set as thecorresponding gallery images.

fc probes. Probes taken on the same day, but with differentillumination from the corresponding gallery images.

Fully automatic algorithm. Anr algorithm that can locate aface in an image and recognize the face.

Gallery. In computing performance, scores, images of the .set of known individuals. The gallery is used incomputing performance after a FERET test is adminis-tered. A gallery is a subset of a target set. A target set cangenerate multiple galleries.

Probe. Image containing the face of an unknown individualthat is presented to an algorithm to be recognized. Probecan also refer to the identity of the person in a probeimage.

Partially automatic algorithm. An algorithm that requiresthat the centers of the eyes are provided prior torecognizing a face.

Probe set. A set of probe images used in computingalgorithm performance. The probe set is used incomputing performance after the FERET test is adminis-tered. A probe set is a subset of a query set. A query setcan generate multiple probe sets.

Query set. The set of unknown images presented to thealgorithm when a test is administered. See probe set.

Target set. The set of known images presented 'to thealgorithm when a test is administered. See gallery.

ACKNOWLEDGMENTS

The work reported here is part of the Face RecognitionTechnology (FERET) program, which is sponsored by theUS Department of Defense Counterdrug TechnologyDevelopment Program. Portions of this work was donewhile P.J. Phillips was at the US Army Research Laboratory(ARL). P.J. Phillips would like to acknowledge the supportof the National Institute of Justice.

REFERENCES11] K. Etemad and R. Chellappa,' "Discriminant Analysis for

Recognition of Human Face Images," J. Optical Soc. Am. A,vol. 14, pp. 1,724-1,733, Aug. 1997.

[2] R.M. McCabe, "Best Practice Recommendation for the Capture ofMugshots Version," 1997. 2.0.http://www. nist. gov/itl/div894/894.03/face/face.html.

(3] B. Moghaddam, C. Nastar, and A. Pentland, "Bayesian FaceRecognitionUsing Deformable Intensity Surfaces," Proc. Computer.Vision and Pattern Recognition '96, pp. 638-645, 1996.

14) B. Moghaddam and A. Pentland, "Probabilistic Visual Learningfor Object Detection," IEEE Trans. Patteni Analysis and MachineIntelligence, vol. 17, no. 7, pp. 696-710, July 1997.

15] H. Moon and P.J. Phillips, "Analysis of PCA-Based FaceRecognition Algorithms," Empirical 'Evaluation Techniques inComputer Vision, K.W. Bowyer and P.J. Phillips, feds., pp. 57-71,Los Alamitos, Calif.: IEEE CS Press, 1998. ,

[6] P.J. Phillips and P. Rauss, "The Face Recognition Technology(FERET) Program," Proc. Office of Nat'l Drug Control Policy, CTACInt'l Technology Symp., pp. 8-11, Aug. 1997.

[71 P.J. Phillips, H. Wechsler, J. Huang, and P. Rauss, "The FERETDatabase and Evaluation Procedure for Face-Recognition Algo-rithms," Image and Vision Computing J., vol. 16, no. 5, pp: 295-306,1998.

[8] P. Rauss, P.J. Phillips, A.T. DePersia, and M. Hamilton, "TheFERET (Face Recognition Technology) Program," Surveillance andAssessment Technology for Law Enforcement, SPIE, vol. 2,935, pp. 2-11, 1996.

[9] S. Rizvi, P.J. Phillips, and H. Moon, "The FERET VerificationTesting Protocol for Face Recognition Algorithms," TechnicalReport NISTIR 6,281, Nat'l Inst. Standards and Technology,http://www.nist.gov/itl/div894/894.03/pubs.html#face. 1998.

[10] S. Rizvi, P.J. Phillips, and H. Moon, "The FERET VerificationTesting Protocol for Face Recognition Algorithms," Image andVisioi Computing J., to appear.

1 I] D. Swets and J. Weng, "Using Discriminant Eigenfeatures forImage Retrieval," IEEE Trans. Pattern Analysis and MachineIntelligence, vol. 18, no. 8, pp. 831-836, Aug. 1996.

[12] M. Turk and A. Pentland, "Eigenfaces for Recognition," J.Cognitive Neuroscience, vol. 3, no..I, pp. 71-86, 1991.'

[13] J. Wilder, "Face Recognition Using Transform Coding ofGrayscale Projection.Projections and the Neural Tree Network,"Artifical Neural Networks with Applications in Speech and Vision,R.J. Mammone, ed., pp. 520-536, Chapman Hall, 1994.

[14] L. Wiskott, J.-M. Fellous, N. Kruger, and C. von der Malsburg,"Face Recognition by Elastic Bunch Graph Matching," IEEE Trans.Pattern Analysis and Machine Intelligence, vol. 17, no. 7, pp. 775-779,July 1997.

[151 W. Zhao, R. Chellappa, and A. Krishnaswamy, "DiscriminantAnalysis of Principal Components for Face Recognition," Proc.Third Int'l Conf Automatic Face and Gesture Recognition, pp. 336-341,1998..

[16] W. Zhao, A. Krishnaswamy, R. Chellappa, D. Swets, and J. Weng,"Discriminant Analysis of Principal Components for Face Recog-nition," Face Recognition: From Theory to Applications, H. Wechsler,P.J. Phillips, V. Bruce, F.F. Soulie, and T.S. Huang, eds., pp. 73-85,Berlin: Springer-Verlag, 1998.

1103

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, . VOL. 22, NO. 10, OCTOBER 2000

P. Jonathon Phillips received the BS degree inmathematics In 1983 and the MS in electronicand computer engineering in 1985 from GeorgeMason University and the PhD degree inoperations research from Rutgers University in1996. He is a leading technologist in the fields ofcomputer vision, biometrics, face recognition,and human identification. He works at theNational Institute of Standards and Technology(NIST), where he is currently detailed to the

Delense Advanced Projects Agency to manage the Human Identificationat a Distance (HumanlD) program. Prior to this, he served as projectleader for the Visual Image Processing Group's Human Identificationproject. His current research interests include computer vision, identify-ing humans from visual imagery, face recognition, biometrics, digitalvideo processing, developing methods for evaluating biometric algo-rithms, and computational psycho-physics. Prior to joining NIST, hedirected the Face Recognition Technology (FERET) program at the USArmy Research Laboratory. He developed and designed the FERETdatabase collection and FERET evaluations, which are the de factostandards for the face recognition community. Also, he has conductedresearch in face recognition, biomedical imaging, computationalpsychophysics, and autonomous target recognition. Dr. Phillips wascodirector of the NATO Advanced Study Institute on Face Recognition:From Theory to Applications, coorganizer of the, First and SecondWorkshops on Empirical Evaluation Methods for Computer VisionAlgorithms, and coprogram chair of the Second Intemrnational Con-ference on Audio and Video-Based Biometric Authentication. He isguest coeditor of the special section on empirical evaluation of computeralgorithms in the IEEE Transactions on Pattern Analysis and MachineIntelligence (PAMI) and the special issue of Computer Vision and ImageUnderstanding (CVIU) on empirical evaluation. He has coedited twobooks. The first, Face Recognition: From Theory to Applications, wascoedited with Harry Wechsler, Vicki Bruce, Francoise Fogelman-Soulie, .and Thomas Huang. The second, Empirical Evaluation Techniques inComputer Vision, was coedited with Kevin Bowyer. He is a member ofthe IEEE.

Hyeonjoon Moon received the BS degree inelectronics and computer engineering fromKorea University, Seoul, in 1990, the MS, andPhD degrees in electrical and computer engi-neering from the State University of New York atBuffalo, in 1992 and 1999, .respectively. From1993 to 1994, he was a systems engineer atSamsung Data Systems in Seoul, Korea. From1996 to 1999, he was a research associate at theUS Army Research Laboratory in Adelphi, Mary-

Syed A. Rizvi (S'92-M'96) received the BScdegree (honors) from the University of Engineer-ing and Technology, Lahore, Pakistan, the MS,and PhD degrees from the State. University ofNew York (SUNY) at Buffalo, in 1990, 1993, and1996, respectively, all in electrical engineering.From May 1995 to July 1996, he was a researchassociate with the US Army Research Labora-tory, Adelphi, Maryland, where he developedcoding and automatic target recoanition algo-

rithms for FLIR imagery. Since September 1996, he has been anassistant professor with the Department of Engineering Science andPhysics at the College of Staten Island of the City University of NewYork. From 1996 to 1998, he. was a consultant with the US ArmyResearch Laboratory, collaborating in research for image compressionand automatic target recognition. His current research interests includeimage and video coding, applications of artificial neural networks toimage processing, and automatic target recognition. He has publishedmore than 60 technical articles in his area of research. He is a memberof SPIE and the IEEE.

Patrick J. Rauss receiveo mthe BS degree inengineering physics from Lehigh University.in1987. He received the MEng degree in appliedremote sensing and geo-ilnformation systemsfrom the University of Michigan in 2000. He hasworked as a civilian researcher for the US Armysince 1988, first with the Night Vision andElectro-Optics Directorate and, since 1992 withthe Army Research Laboratory's EO-IR ImageProcessing Branch. He worked closely withD"r Phillins n the FERET nrnnram from 1995

to 1997. His current research interests are automated processing.olhyperspectral imagery for material classification and using supervised,adaptive leamrning techniques for hyperspectral and computer visionapplications. Over the years, he has developed signal and imageprocessing tools and techniques for a wide range of sensors includingX-ray fluorescent spectrometers, midwave and long-wave infraredradiometers, Fourier transform infrared spectrometers, forward lookinginfrared imagers, and hyperspectral imagers.

land. Currently, he is a senior research scientist at Lau Technologies inLittleton, Massachusetts. His research interests include image proces-sing, neural networks, computer vision, and pattern recognition. He Is amember of the IEEE..



a loo

, A Combination Fingerprint ClassifierAndrew Senior

Abstract-Fingerprint classification is an important indexing method for any large scale fingerprint recognition system or database asa method for reducing the number of fingerprints that need to be searched when looking for a matching print. Fingerprints are generallyclassified into broad categories based on global characteristics. This paper describes novel methods of classification using hiddenMarkov models (HMMs) and decision trees to recognize the ridge structure of the print, without needing to detect singular points. Themethods are compared and combined with a standard fingerprint classification algorithm and results for the combination are presentedusing a standard database of fingerprint images. The paper also describes a method for achieving any level of accuracy required of thesystem by sacrificing the efficiency of the classifier. The accuracy of the combination classifier is shown to be higher than that of twostate-of-the-art systems tested under the same conditions.

Index Terms-Henry fingerprint classification, hidden Markov models, decision trees, neural networks, NIST database.

+.

1 INTRODUCTION

THE classification of fingerprints has long been animportant part of any fingerprinting system. A partition

of fingerprints into groups of broadly similar patternsallows filing and retrieval of large databases of fingerprintsfor. quick reference. Currently, interest in fingerprintclassification is stimulated by its use in automatic finger-'print identification systems (AFIS). In an AFIS, the goal is tofind a match for a probe fingerprint in the' database ofenrolled prints, possibly numbering many millions. Classi-fication is used in an AFIS to reduce the size of the searchspace to fingerprints of the same class before attemptingexact matching.

The most widely used system of fingerprint classificationis the Henry system and its variants [1]. Examples from fiveofthe main classes of the Henry system are shown in Fig. 1.

Many previous authors have developed automatedsystems to classify fingerprints, using a wide variety oftechniques. Cappelli et al. [2] provide a recent review of anumber of methods that have been used and Section 8 inthis'paper presents results from other authors.

Most automatic systems use the Henry classes and theseare important for existing AFIS databases and systemswhich require compatibility with human classifications,either because of legacy data or because some manualintervention is necessary in the process, requiring the use ofhuman-interpretable classes. A variety of approaches toclassification has been tried, the most fundamental being asyntactic analysis of the relative positions and number ofcore and delta points in the print. The core and delta points,shown in Fig..1, are the singular points in the flow of theridges. Finding these points in the image is a difficult imageprocessing task, particularly with poor quality images, butif found reliably, the classification is simple. 13], [4]. Maio

* The author is with the IBM T.J. Watson Research Center, PO Box 704,Yorktown Heights, NY 10598-0218. E-mail: [email protected].

Manuscript received 15 Dec. 1999; revised 24 Nov. 2000; accepted 17 May2001.Recommended for acceptance by R. Kumar.For information on obtaining reprints of this. article, please seund e-mail to:[email protected], and reference IEEECS Log Number 111092.

and Maltoni [5] use a structural analysis of the direction ofridges in the print, without needing to find core and deltapoints. Blue et al. [6] and Candela et al. [7] use the corelocation to center their representation scheme, which isbased on a principal components analysis (PCA) of ridgedirections, and then they use a variety of classifiers. Haliciand Ongun [8] similarly use PCA-projected, core-centeredridge directions, but classified with a self-organizing map;and Jain et al. [9] also use the core for translationSinvariance, using a Cabor filter representation and ak-Nearest Neighbor classifier.

For situations where there is no need to use existingclasses, some researchers have developed systems which relyon machine-generated classes or dispense with classes allStogether and use "continuous" classification [2], [101, 18], [11].Here, the criterion is not adherence to the Henry classes, butmerely consistency among classifications of different printsfrom the same finger. Fingerprints are represented by pointsin a feature space on which some distance measure is defined.Test fingerprints are matched against all those in the databasefalling within some radius of the test print. By increasing theradius, classification can be made arbitrarily accurate,reducing errors by increasing the size of the search spaceand, hence, search time. Continuous classification holds theprospect of circumventing the difficult and restrictive Henryclassification problem and has produced the best results ofrecent years, but has disadvantages besides the uninterpret-ability mentioned above. Using Henry classes, the portions ofthe database that must be searched are always the same,allowing for rigid segmentation of the database and a prioridesign)of the' search strategy. A continuous system presentsan entirely different subset of the database for every matchingoperation, complicating and slowing the matching..

This paper describes a combination of novel approachesto fingerprint classification using the Henry system. Thesystem described has been designed to operate on bothrolled and "'dab" fingerprints, where some of the structuralinformation tised by other systems (such as the deltaposition) may not be available in the fingerprint image. Thesystem described has been tested on the NIST Specialdatabase 4 [12] database of fingerprint images and results

0162-8828/01/$10.00 0 2001 IEEE

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 23, NO. 10. OCTOBER 2001

Left loop Right loop Whorl0 338'R 0.317 0.279

Arch Tented arch0.037 0.029

Fig. 1. Examples of five fingerprint categories, marked with core anddelta points, with their frequencies of occurrence.

are presented. Further, 'a method of measuring theefficiency of a classification algorithm is described, allowinga principled comparison of this algorithm with previouspublished works. Finally, a method for achieving arbitraryaccuracy is described, allowing the Henry classifier to be

used with the flexibility of continuous classifiers. Thismethod trades off accuracy against classifier efficiency,allowing an imperfect classifier to be used in 'a real-worldsystem, while retainingall the advantages of a traditionalHenry system.

The approach taken here is of a combination ofclassifiers, each using different features and with differenterrors on test data. Two novel classifiers are described,using two-dimensional hidden Markov models (HMMs)and decision trees. In addition to showing that these.classifiers perform well on the classification problem, andwithout the need for core/delta information, this papershows-that the combination of classifiers provides a wayforward for the improvement of fingerprint classificationin the same way as recent improvements in isolatedhandwritten character recognition performance have beenlargely brought about not by better classifiers but bycombinations of different classifiers. The classifiers aretested in isolation and in combination with the Probabil-istic Neural Network classifier and pseudoridge tracerfrom the PCASYS system described by Candela et al. [7].The experiments are all performed on discrete, Henryclassification, but the system could be extended tocontinuous classification, or classification with unsuper-vised clustering, using such techniques as unsupervisedK-means HMM clustering [13].

The following sections describe the HMM classifier(previously described in [14]), the decision tree classifier,and PCASYS classifiers. In Section.5, classification basedupon the outputs of these classifiers is then described aswell as combining the classifiers to improve accuracy.Section 6 describes a measure of efficiency of the classifierand shows how arbitrary efficiency can be achieved.Section 7 presents results for the classifiers and their

.

1166

combinations, and results are compared with previouslypublished results in Section 8.

2 CLASSIFICATION BY HIDDEN MARKOV MODEL

Hidden Mairkov models are a form of stochastic finite stateautomaton well-suited to pattern recognition.and success-fully applied to speech recognition [15], [16] and otherproblems. They are appropriate to the problem posed herebecause of their ability to classify patterns based on a largequantity of features whose number is variable and whichhave certain types of.underlying structure, especially if thatstructure results in stationarity of the feature distributionsover some spatial or temporal period. Such structure is foundin fingerprints, where ridge orientations, spacings, andcurvatures are, for the most part, only slowly varying acrossthe print:' In a fingerprint, the basic class information can beinferred from syntactic analysis of singular points, but canalso be seen in the general pattern of the ridges-the way anonexpert human would classify prints. The HMM is able tostatistically model the different structures of the ridgepatterns by accumulations of evidence across the wholeprint, without relying on singular point extraction.,

2.1 Ridge Extraction

The-system deals with fingerprint images stored as arrays ofgray levels and obtained with a scanner or camera device-either from an inked fingerprint on paper or as a "live-scan"directly from the finger. For much of. the work in this paper,the NIST-4 [12] database of rolled fingerprint images hasbeen used since this provides a large number (4,000) offingerprints with associated class labels. In addition, part ofthe NIST-9 database has been used.

The features provided to the recognizer are based on the

characteristics of the intersections of ridges with a set offiducial lines that are laid across the fingerprint image . Tofind the ridge locations, a number of image processingtechniques are used [17], summarized as follows:

1. Initial segmentation: The PCASYS algorithm forextracting a central fingerprint region from a fullrolled print is used on prints from the NIST-9database. NIST-4 is already segmented at this level;

2. Smoothing;3. Finding the predominant direction in each of an

array of blocks' covering the image;4. Segmenting the image into the area of the print

(foreground) and the unwanted background, basedon the strength of directionality found in each block;

5. Applying directional filters to highlight the ridgesand detecting pixels that are parts of ridges;

6. Thinning the ridge image so that each ridge is leftrepresented by an eight-connected, one-pixel-wideline termed the skeleton.

2.2 Feature ExtractionGiven the skeleton image of the ridges, parallel fiduciallines are laid across the image at an angle 4, as shown inFig. 2, and each one followed in turn. For each intersectionof a fiducial line with a ridge, a feature is generated. Eachfeature consists of a number of measurements, chosen to

SENIOR: A COMBINATION FINGERPRINT CLASSIFIER

Fig. 2. A sample fingerprint showing horizontal fiduclal lines (p = 0).

characterize the ridge behavior and its development at theintersection point:

the distance since the last intersection;'the angle of intersection;the change in angle since the last intersection;the curvature of the ridge at the intersection.

The angle features (2) can be seen to contain similarinformation to the coarse direction field calculated in thepreprocessing stages of this system and used by othersystems as the feature set for classification [71. However,this representation allows a higher resolution representa-tion of the fingerprints, and allows more information to berepresented (e.g., ridge spacing and curvature). Furthermeasurements could also be taken at each point.

The measurements of each feature. are termed a frameand the frames, R. for the ith fiducial line are collectivelytermed a row, A, whose ordering is preserved. For eachorientation 0 of fiducial lines, a separate representationm = { Ri, Vi) of the print is obtained. In this research, only

horizontal and vertical lines have been used, giving featuresRh and 7R', respectively, but other angles may allow furtherinformation to be captured.

2.3 Hidden Markov ModelsTypically, HMMs are one-dimensional structures suitablefor analyzing temporal data. Here, the data are two-dimensional, but the process of feature extraction can alsobe described as a one-dimensional array of one-dimensionalrow processes. Thus, we can apply a "two-dimensionalhidden Markov model," similar to that of Agazzi et al. 118],which consists of a nesting of row models within whole-print models, as shown in Fig. 3.

For classification, a model M, is constructed for eachclass, c, and the maximum likelihood class is chosen aftercalculating the probability of the data R given the model:argmaxc P(RTM c).

2.3.1 Row ModelingTo simplify the analysis of the model, first consider a rowmodel modeling a single row of fingerprint data. Each rowmodel Mi, is a conventional HM-1M and consists of a numberof states which model the small, stationary regions in a row.Any row R4 is assumed to have been generated by the rowautomaton transiting from state to state, producing theframes in the observed order at each transition, with Sijbeing the kth state in the sequence whereby Mi produces Ri

Row. 1

Row 2

Row 3

Fig. 3. A schematic of the two-dimensional structure of the HMM, .showing three row models of live states each, forming a global model fora single class.

(k corresponds to time in -a temporal HMM). The statetransitions are controlled by probabilities P(S,|ISi _ )trained with certain constraints: The state must monotoni-

cally increase Si. > Sij ,,for k > k' and it is possible to skipstates at the edge of the print. Because of the nature of the

printing process whereby, especially for dabs, it is to be

expected that edge regions of the fingerprint will be missing

but the central regions will always be present, only states atthe edge of the print may be skipped. This effectively

constrains the initial state distribution P(Sijo).The frame emission probabilities are modeled with

mixtures of diagonal covariance, multivariate Gaussian

distributions. Thus, for any frame R.i, it is possible tocalculate-the likelihood P(RIk SiSqk) of it occurring in any state

Sk. With these likelihoods, for any row model, the like-lihood of any row can be approximated by the maximumlikelihood of any state sequence aligning the features and

states calculated as a product of frame likelihoods and

transition probabilities for the state sequence:

P(RiM)j) max P(RiolSijo)P(Sijo)s~j

I P( jSjk)P( SqIjSqik,).k

The models are initialized by using an equal-lengthalignment with the frames evenly distributed across thestates of the model. After estimating the initial parametervalues, using smooth equal-length alignment [19],Viterbi alignment is used to find the maximum-likelihoodalignment of frames with states, which is used forretraining. Around two iterations of training.are necessaryto achieve good classification performance.

2.3.2 Global ModelThe global model is the same as a row model, except that itsstates are row models and its frames are whole rows. Thus,for each model c:

1167


P(RIMC) max P(RolMso)P(Ms).

" P(RkIMs')P(S' ISk,),

where S' is an alignment specifying which row model Ms.models the row of data Rk.

2.4 Multiple HMM Classifiers

For each orientation of fiducial lines, a separate classifiercan be made. Since the errors of the different classifiers willbe different, a combination of their scores may yield betteraccuracy. Denoting by Me, M', the class c models trainedwith vertical and horizontal features, respectively, andassuming, independence, the likelihood of the data iswritten as:

P(^,7'ICc) = P(R IM")P(R'|M'). (3)

Fusion of multiple classifiers is treated in more detail inSection 5.3.

3 DEcISION TREE CLASSIFIERS

To provide a supplementary classification, hopefully givinguncorrelated errors, another type of features has beenextracted and classified with a decision tree approach. Suchdecision, trees are built using techniques based upon thoseof Amit et al. 1201. These authors tackled a number ofproblems including that of digit recognition--classifyingimages of the digits "0" to "9."

The technique used by Amit et al. for constructingdecision trees involves the generation of a large number ofsimple features. Each feature in isolation provides littleinformation about the classification decision, for example,the existence of an edge at a particular location in an imagemay give little clue as to the digit's identity. However,combinations of such features can represent much impor-tant information needed to make an accurate classificationdecision. Amit et al. describe a procedure for makingdecision trees by growing questions based upon suchcombinations of simple features.

The procedure has been adopted here for fingerprintclassification and involves an initial feature extractionphase, followed by question building which createsinformative questions assisting in classification. These'complex questions are combined in a hierarchical manner,to form decision trees which are used for classification.Because the trees are constructed ,stochastically, treesconstructed for the same problem have different perfor-mances and, as is common with decision tree classifiers,multiple trees are combined to give the final classification.

3.1 Feature ExtractionThis second classifier was designed to give a secondopinion on the classification of a fingerprint image. Forthis purpose, the errors in classification should be asuncorrelated as possible with those made by the HMM,thus a different set of features was generated for thisclassification method. Again, the motivation is to considerdistributed information from across the fingerprint without.extraction of singular points. Because the class informationis implicit in the shapes of ridges, features that are easily

curve

bottomn ,

right

curve

Fig. 4. A single ridge showing the extracted leatures for the decision

tree: curvature maximum and, left, right, bottom, and top, turning points.

and reliably extracted and which encode the ridge shape ina simple, concise manner were chosen.

The initial preprocessing used is identical to that of theHMM classifier, up to the extraction of ridges (Section 2.1),but, instead of taking features at intersections with fiduciallines, features are generated at salient points on the ridges.The features consist of curvature maxima and four axis-parallel turning points (g = 0 or = 0 for a ridgerepresented as the parametric curve (x(s), y(s)) and dis-tinguished by the sign of the second derivative). Someexample features are shown in Fig. 4. For each feature, thefeature type and location (in pixels at 500 dpi) is recorded.These features are all based on local computations on theridges and, again, avoid the extraction of global featuressuch as core and delta points. Again, they are invariant totranslation and to small amounts of rotation. These featuresare also appropriate for the classification of dabs since themajority of features in a rolled print also occur in the regiontypically imaged in a dab.

3.2 Decision TreesA binary decision tree is constructed as a hierarchy ofbinary questions [21]. Questions are logical statementsabout the features that may be present in the fingerprintarid about the relations between those features; for a givenfingerprint, a question is either true or false. At the "top"level, each test sample is asked the first question. Accordingto the test sample data, the question returns either true orfalse and the branch to the second-level is determined. Onwhichever branch is chosen, a second level question isasked and a further bifurcation is induced. In this way, eachtest sample descends the tree by a route dependent on itsfeatures and arrives at a leaf node. Leaf nodes are labeledaccording to the classes of the test data that arrived there. Ina simple classification problem, leaf nodes will be pure--i.e., receive only training samples from a single class andthe unambiguous classification of any test sample arrivingthere would be the class of the training data at that node.For more complex problems, the leaf nodes contain mixeddata 'and the test data is labeled with a- probabilitydistribution across the classes.

Fig. 5 shows a small decision tree with two levels. Eachof the three nodes of the tree contains a question of the formspecified in Section 3.3. At each node and at the leaves, aclass histogram with four classes is shown, indicating thereduction in entropy as the tree is traversed.. The root node

1168


A

Fig. 5. A two-level decision tree, showing hypothetical class distributions at each node. Each node has a question formed of a list of feature typesthat must be present and a list of relationships between them that must be true for, the question to return "yes."

has all classes equally likely, with no discrimination, andthe other nodes have successively stronger discriminationbetween the classes.

3.3 QuestionsA question consists of a list of required features and a set ofrelations between them in the same manner as those ofAmit et al. Each feature is specified as one of the fivefeatures described above (four turning points and curvaturemaximum). Relations are all of the form "x is direction ofy," where direction is one of North, South, East, West.

Optionally, a question can also impose a distance limit-that the feature must be within a certain distance.Experimentation led us to use two distance bands: featureswithin 0.1" and features within 0.2". An example questionmay specify that there is a maximum of curvature East of a

lower turning point which is itself East of, and within 0.2"of, a maximum of curvature. Other questions are shown inthe nodes of Fig. 5. Questions are constructed in such a waythat every feature is related to at least one feature in thefeature list and so that every pair of features can have atmost one relation.

Given a new print, the print can be tested by a searchthat determines if the question can be fulfilled by thefeatures of the print.

3.4 Question ConstructionDuring tree construction questions are constructed ran-domly as follows:

I. Select any feature class as the first feature.2. Test the data separation. If more than 2/3 of training

data at this node reply yes, refine the question andrepeat this step. If less than 1/3 reply yes, discardthis question and construct a new question. Other-wise, evaluate the question.

Refining the question consists of adding extra restrictions,which inevitably make a "yes" answer less likely. Theproportion of samples answering "yes" can be reduced inone of two ways. First, a feature can be added. In this case, arandom feature type is chosen and added to the list. A randomrelation is chosen to relate it to a randomly chosen feature

already in the list. Second, if there are twoor more features inthe question and some pair has no relation between them, thenan additional relation can be added to the question betweenany pair of features that are as yet unrelated.

When adding a relation is not possible, a feature isadded. Otherwise, a random choice is made; biased towardadding a relation since this keeps the number of featureslower, limiting the dimensionality of the search space foranswering questions and making testing faster.

Having arrived at a question which. channels approxi-mately half the data to each of the "yes" and "no" sides, thequestion is evaluated. The measure of the effectiveness of aquestion is the change of entropy in the distributions beforeand after applying the question. Classes at the root nodehave high entropy, but the leaf nodes should have very lowentropy (be "purer"). The relative entropy of the outputdistributions for a node is computed for many randomlyconstructed, candidate questions and the question with thehighest entropy change is chosen.

A tree is recursively constructed until the leaves are pureor until a maximum depth (typically 7) is reached. Fig. 6shows the effect of varying the depth of the trees and thenumber of trees used. Multiple trees are merged by

s , , , . , o -

Depth 5 -....

20

. :2 .....

2 4 0 0S 2 e4 10

Number of trees

Fig. 6. Raw error rates (no priors, unweighted) plotted against number of

th e +

i

1169

32 64 , . 129


multiplying the leaf node distributions class-by-class, as inSection 2.4.

4 PCASYS

Candela et al. [7] have described a fingerprint classifier .called PCASYS, which is based upon a probabilistic neuralnetwork (PNN) classifying features consisting of principal-component projected orientation vectors. The orientationsof the ridges taken in a 28x30 grid of points around the coreare reduced in dimensionality with principal componentsanalysis. The resulting 64-dimensional vectors are classifiedwith the PNN. They have published results and made theirsoftware available, making possible a realistic comparisonwith this system. Lumini et al. [10] have used this softwareand extended it to provide continuous classification. Toprovide an alternative classification method and anenhanced combination classifier, PCASYS has been testedon the same testing data as classified by the HMM anddecision tree.classifiers. Results are presented for PCASYSalone and in combination with the other classifiers.

PCASYS incorporates a pseudoridge tracer which detectsupward curving ridges and is able to correctly identifysome whorl prints, but provides no information todistinguish among the other classes. This effectivelypenalizes the other classes when returning a "yes" answer.PCASYS also exploits prior information to improve itsaccuracy (see Section 5.1).

5 POSTPROCESSING AND CLASSIFIERCOMBINATION

Given the raw classifiers presented above, a number ofsteps must be taken to apply the classifiers to a test set. Thefollowing sections describe using class priors to enhanceclassifier accuracy, weighting the results to predict behavioron true test sets and methods for the combination ofmultiple classifiers.

5.1 Class PriorsBecause the classes that are used are not equal in frequency ofoccurrence, calculating the posterior probability of a class,given the data, requires the product of the data likelihood,given the class c and the prior probability of the class:

P(c|Z) oc P(RJI.Mc)P(c). (4)

The class priors have been estimated by Wilson et al. [22] on222 million fingerprints. (The proportions are 0.037, 0.338,0.317, 0.029, 0.279 for arch, left loop, right loop, tented arch,and whorl, respectively).

5.2 Class WeightingSince the NIST-4 database (and, correspondingly, the testset used here) has equal numbers of prints from each class,to obtain a good estimate of the true test-condition accuracy,the results must be weighted according to the truefrequencies of occurrence, using the same procedure ofWilson et al. [22]. Otherwise, a classifier good at recognizingarches, which are rare, would appear better on this test setthan in the real world, or on a representative test set wherethis ability is rarely called upon.

5.3 Classifier CombinationFour classifiers (counting the PCASYS pseudoridge classi-fier) are available in this work. Each by itself is capable ofclassifying fingerprints with a limited.accuracy. However,each classifier uses different features and methods, so theerrors produced by each classifier should be somewhatuncorrelated. In this situation, combining the results of the.classifiers should produce an improved classifier with'lower error rate: Many other authors have tackled theproblems of decision fusion and, here, we take two simpleapproaches,

5.3.1 Linear Likelihood Fusion

The first method of combination is a probabilistic approachsince the output of each classifier is a probability distribu-tion P(cJlV) across the classes c.

Strictly speaking, if each classifier gave a true probabilityout and, with N independent classifiers operating onfeatures RZ', the posterior probability would be:

However, in practice, the probabilities are correlated andhave varying reliabilities. The HMM probabilities are the

product of many correlated probabilities and the PNNalready incorporates prior information. To correct for theseeffects, weights w are introduced to balance the classifiercombination. Working in the log domain, with normal-ization constant k:

logP(cR , ... , RN) = k + logP(c)+ EwP(Z' Jc). (6)

For simple classification, the class

argmaxe log P(c|J',..., N)

is chosen as the correct answer. Finding the weights w;,however, is a difficult problem. Estimation of weights by linesearches on the training set fails to generalize well to the testset, so the followingtrained approach was used, which isfound to achieve accuracies close to those obtained whenoptimizing linear weights by line search on the test set.

5.3.2 Neural Network Fusion

The second fusion approach is to use a backpropagationneural network. Here, the class probabilities for all theclassifiers are combined in a neural network, trained tooutput the true class on the training set. Additionally, fourestimates of the fingerprint quality [23] are supplied to thenetwork, though their effect is not significant. Training usesa momentum-based weight update scheme and Softmaxoutputs 124], giving an output class probability distribution.Training samples are weighted to simulate a uniform priorand the output probabilities are multiplied by the classprior 5.1 when testing. Separate networks are trained tocombine the HMM and decision tree or to combine all fourclassifiers. To generate enough training data for theneural network, the first half of the NIST-4 database wassupplemented with 5,400 prints from the NIST-9 database(Volume 2, CDs 1, 2, 3).

1170

P(cl7Z',... , 7ZN). P(c) r P(Vlc).


6 CLASSIFIER EFFICIENCY

Since the purpose of a fingerprint classifier is to partition la.fingerprint databases, in addition to the classificataccuracy-the proportion of classifications that givecorrect class-the classification efficiency must alsoconsidered. Since .many authors use different classesconsistent measure of efficiency, as described here,essential for the comparison of results. An efficiency measalso permits the evaluation of rejection and backing-strategies described in Section 6.2.

The classification efficiency can be considered a.,measure of reduction of search space. In practice,proportion of the database to be searched will varyeach query, so, over a test set, the average efficiency cancalculated as:

Number of matches required with no classifier

Number of matches required when classifier is used'

where an.exhaustive 1 : many match against a databas

N prints is counted as N matches. If a perfect classifie

used to classify M prints prior to matching agains

database of N prints, any of the MPc test prints in cla.

(which occurs with probability Pc) need only be tes

against the NPc database prints of class c. Thus, the t

number of matches required is now Z, NMP insteac

NM. The efficiency of a perfect classifier using these cla.,

is thus P. Using the five NIST-4 classes and

frequencies of Section 5.1, this gives an efficiency of 3

Merging arch and tented arch classes only reduces

efficiency to 3.37 since this distinction so rarely needs tc

made. As can be seen, the imbalance of the class pr

makes the efficiency significantly lower than would

obtained with five equally frequent classes (an efficienc

5). In practice, the efficiency of a fallible classifier

deviate' from this value-for instance, a classifier wl

consistently mistakes all prints for arches will have

efficiency of 15 (1/0.066).

6.1 . Alternate Classes

NIST-4 provides alternate class labels for 17.5 percent of

prints [12, p. 8, but these are ignored in this worl

classification being deemed correct only if it matches

primary class label. Allowing matches with the altern

label too would increase the recognition rates but wclower the efficiency of the classifier since such prints, w

enrolled, would have to be stored twice (under both clas

in our database, resulting in extra searches every time

secondary class was searched.

6.2 Backing-OffPrevious classification works have quoted error rateswould be unacceptable in real-world systems. It is clearaccuracy can be a trade-off for efficiency-searching nthan just the top one class will give higher accuracy

lower efficiency [25]. If a reliable measure of confidence in

rge the classifier's answer is available, it is possible to deviseion methods to adjust the reliance on the classifier answer when

the that classification is uncertain, and thus reducing thebe number of errors made. Some classifiers [7] have used a

, a rejection mechanism, which improves the accuracy at theis cost of not pruning the search with those prints that are

ure rejected. This: section proposes a more complex scheme to-off allow graceful and efficient "backing-off" of classifier

reliance based on a likelihood ratio confidence measure.s a It is clear that if the likelihoods for the top two classes are

the very different, then the classifier can be considered to beith more "confident" in its answer than if the two likelihoodsbe

are similar (when it would only take a small perturbation to

change the ranks of the answers). Thus, the likelihood ratio

(7) of the top two answers is examined. If this is less than an

empirically determined threshold, then the top choice is

of deemed to be not confident and the top two classes are

r is jointly returned as the answer (increasing the proportion ofthe database subsequently searched by the 1:many match-

t a er). Similarly, the likelihood ratio of the second and third

ss c choices is compared to a threshold to allow backing-off to

ted three classes. Repeating the procedure, if all the likelihoods

otal are similar, the classifier will return a "don't know" answer

I of and all classes must be searched. Moretraditional rejection

sses strategies (e.g., [22]) use a criterion to back off directly fromthe "top-choice only" to the "don't know" answer without

thet allowing as rich a classification..39. The efficiency of the classifier when allowing backing-off

the is now:

Sbe MN

iors ',, ' (8)

bewhere ir,,, 1 > rn,,, > 0, is the proportion of the database

y of searched for query print' m..will. Adjusting the likelihood ratio threshold allows arbitrary

hich accuracy to be 'obtained. A large threshold would give a

an null classifier with 100 percent accuracy but an efficiency of

one. A threshold of zero would give the basic top-one

accuracy and maximum efficiency (3.37 for the four class

problem). Adjusting the threshold allows us to set thethe overall classifier accuracy to that deemed necessary for thek, a whole system. However, it should be noted that thisthe arbitrary classification accuracy 'is achieved within the

nate context of a Henry classification system where the portionsuld of the database to be searched will always conform to thehen .Henry classifications and, thus, allow the database parti-ses) tioning and search to be designed to operate on priorthe knowledge, not having to cope with dynamically changing

subsets, as in continuous classification.In fact, the efficiency loss (i.e., extra search time) of

that searching the next class is dependent on the frequency of

that that class, so a more advanced backing-off algorithm should

lore take this into account to achieve a better trade-off ofbut accuracy for efficiency.

1171


TABLE 1Error Rates, Testing Different Combinations of the Classifiers

Classifier Error (%)HMM Horizontal (7 ") features 20.8HMM Vertical (1?) features 13.8HMM Both with prior 12.8DT with prior 14.4HMM + DT + prior 9.0PCASYS (PNN only) ' 8.5PCASYS (PNN + Pseudoridge) 6.1HMM +DT +PCASYS (no prior) 6.8HMM +DT +PCASYS (prior) 5.1

The HMM classifier used here has eight rows of eight states each anduses a shared pool of 200 Gaussians with four-dimensional features.The decision tree (DT) classifier is a mixture of 30 trees, each with sevenlevels. The three combination results use a neural network. ThePCASYS PNN+Pseudoridge classifier result uses a heuristic combina-tion [7] which incorporates the priors.

7 RESULTS

Following the practice of Jain et al. [9], the system has been

trained and tested on the NIST-4 database of fingerprintimages. The training set was composed of the first

2,000 prints and the test set was the remaining 2,000 printsfrom NIST-4 (1,000 pairs, so no test prints had correspond-ing prints in the training data). The primary class labelsgiven with the databases were used, but, since the efficiencyis hardly affected, the classifier was only trained to

distinguish four classes, treating-arch and tented arch asidentical. Table I shows the error rates for various

combinations of classifiers. All results presented in thistable are weighted, as in Section 5.2, to simulate the naturalfrequencies of occurrence of the fingerprint classes.

Table 2 shows the confusion matrix for the four-classifierneural network fusion with priors, but without the classweighting, and shows the distribution of misclassificationsand the error rate for each of the four classes used.

Fig. 7 shows the trade-off between error rate andefficiency obtainable by varying the likelihood ratio usedfor backing-off.

8 PREVIOUS RESULTS

Table 3 shows the results achieved by a number offingerprint classification systems that have previously been

published. These points are plotted in Fig. 8 along with thecurve' of possible operating points for the combinationpresented here and the continuous systems of Cappelli et al.

[2], Lumini et al. [10], and Halici and Ongun [8]. For each

TABLE 2Confusion Matrix lor the NIST-4 Test Set (without Reweighting)

True Assigned Class ErrorClass A/T W L R RateA/T 637 5 66 112 22.3%W . 0 384 4 14 4.5%L 10 2 370 4 4.2%Rt 8 2 .2 380 3.1%

The table shows the assigned classes for fingerprints falling into each ofthe true classes and a per-class error rate.

5

I,4

03

2

1

1.5 2.5Efficiency

3. 3.5

Fig. 7. A graph ol error rate against efficiency for the combinationdescribed here. The curve shows the trade-off of efficiency against errorrate for a variety of likelihood ratio thresholds. Other algorithms testedunder the same conditions are also plotted, along with the FBIperformance target.

system, the efficiency of the classifier is shown with the

corresponding error rate. In some cases, authors have used

rejection strategies where a proportion 7r of prints, are

rejected, making the identification system search the whole

database (with an efficiency of 1). Assuming these.rejectedprints are uniformly distributed across the database, the

efficiency of the combined system is

E, =E(1 -7r) +7r,'

where E is the natural efficiency using a classifier returninga single class. (This is the value plotted in Fig. 8 whereappropriate.)

8.1 ComparisonBecause of the estimation of efficiency in the case ofrejection and because of the wide range of testing condi-tions previously used, the figure and table present resultswhich are not always directly comparable. In particular, theerror rate of the PCASYS system is 7.8 percent under theconditions described by Candela et al. [71. However,. whentheir software is run on this test set and scoring in a mannerconsistent with the results presented here, with classweighting, the error rate was 6.1 percent (11.4 percentwithout weighting), a figure in which the results of. thecombined classifier here should be compared. Similaily,jain et al. quote an accuracy of 5.2 percent with 1.8 percentrejection, but, if the data from the confusion matrix [9, table3] are scored using the class frequencies found in real data(Fig. 1), the accuracy is 6.2 percent.' These two are the. onlysystems for which truly comparable results were available.Table 4 shows the error rates for the uniform testingconditions and these are plotted in Fig. 7.

One limitation ,of some previous works is simply thatlittle can be inferred from the results presented when thetest sets are so small or where the test set is not trulyindependent of the training set. For example, [28] .derive the

1. A weighted average of the class error rates 7.4 percent, 7.3 percent,5.0 percent, 1.4 percent, 6.0 percent, counting Arch/Tented Arch confusionsas correct. For these results, however, where an alternate class label is given(cf. Section 6.1), either answer is considered to be correct, giving a higheraccuracy than would be obtained under the same conditions we have used.

LTop~ 4- o "...+Jain h al.

PCASYS 0.. j


TABLE 3A Comparison-of Published Fingerprint Classification Methods

Authors & Year Classes Efficiency Error % Reject Test setOhteru et al. 1974 (3) 3 1.95 14 102 good qualityRao & Balck 1980 [26 6 3.9 8.3 60Kawagoe & Tojo 1984 [27] 7 2.6 8.5 94Wilson et al. 1993 122] 5 3.39 4.6 10 weighted 2000 NIST-4Blue et al. 1994 6) 5 3.39 7.2 weighted 2000 NIST-4Candela et al. 1995 7] 5 3.39 7.8 2700 NIST-14

5. 3.39 3.6 10 2700 NIST-14PNN only 5 3.39 9.5 2700 NIST-14Pal & Mitra 1996 28] 5 3.6 18+ 45 (training set)Fitz & Green 1996 (29) 3 1.95 15 . 40Karu & Jain 1996 [4) 4 3.37 7.0 10% 4000 NIST-4 (priors).

4 3.37 ' 9.4 4000 NIST-4 (priors)4 3.37 6.1 10% 4000 NIST-44 3.37 8.6 10% 4000 NIST-9

Halici & Ongun 1996 8) continuous 3.33 4.0 250 NISTJain et al. 1999 (9] 4 3.37 5.2 1.8 % second half of NIST 4

test set from the training set using artificial noise processes.Although [22] and [6] use the NIST-4 database, they test onsecond imprints of the same fingers that were used fortraining, an unrealistic scenario for which classification byrecognition of the fingerprints would result in much lowerrecognition rates and efficiencies on the order of manythousands. Fitz and Green [29] average over five samples ofa fingerprint before attempting classification.

A final problem with previous work is that the accuraciesof the systems are simply not high enough. If one is to getthe full filtering effect of the classification, only the top classmust be chosen and it has been seen that the classificationaccuracies for the top class (no paper presents any otheraccuracy, such as top 2, etc.) is never high enough to beused in a real system. The higher accuracies that areobtained are achieved by rejecting difficult prints, so thefiltering achieved is even lower. Karu and Jain [4] quote theacceptable error rate for the FBI as being 1 percent at20 percent rejection rate. With four classes, using (9), this isequivalent to a filtering efficiency of 2.816, a performanceachieved by the combination classifier described here andby no previous system, as shown in Fig. 7.-

18

16

14

12

10

w8

W6

4

2

0.

Fig. 8. A graph of error rate against efficiency for a number of publishedalgorithms.

9 CONCLUSIONS

This paper has proposed two new, effective methods for

fingerprint classification which do not require core and delta

information and which have been designed to work on both

dabs and rolled prints. The combination of classifiers

described here produces significantly better results than

any of the component classifiers. Existing Henry fingerprint

classifier accuracies fall short of what is required to make a

significant contribution to an AFI system..This paper has proposed a method for comparing the

efficiencies of different classification schemes and describesa system for achieving an arbitrary degree of accuracy froma classification system while evaluating the effect of thetrade-off. By this means, current fingerprint classifiers canbe rendered of use in an AFI' system. The new classificationcombination can achieve a.filtering efficiency of 2.8, with anerror rate of only 1.0 percent, meeting the FBI requirementsfor a classification system, and is the first system known tothe author to acheive. this. Performance for this Henrysystem is comparable to the performance .of continuousclassifiers and extensions are envisaged to adapt themethods here for non-Henry and continuous classification.

ACKNOWLEDGMENTS

The author would like to thank the reviewers for theirsuggestions, and Ruud Bolle, Sharath Pankanti, and NaliniRatha at IBM for assistance throughout this work.

TABLE 4Comparative Error Rates and Efficiencies for Three

Systems on the Second Half of the NIST-4 DataUsing True Class Frequency Weightings

System . Error (%) Efficiency

Combination classifier 5.1 3.37PCASYS 6.1 3.37Jain et at. 6.2 3.32

1173

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELUGENCE, VOL. 23, NO. 10, OCTOBER 2001

REFERENCES[1] Fed. Bureau of Investigations, The Science of Fingerprints (Classifica-

tion and Uses), 12-84 ed., US Dept. of Justice, Superintendent ofDocuments, US Govt. Printing Office, Washington DC, 1984.

12) R. Cappelli, A. Lumini, D. Maio, and D. Maltoni, "FingerprintSClassification by Directional Image Partitioning," IEEE Trians.Pattern Analysis and Machine Intelligence, vol. 21, no. 5, pp. 402421,May 1999.

[3) S. Ohteru, H. Kobayashi, T. Kato, F. Noda, and H. Kimura,"Automated Fingerprint Classifier," Proc. Int'l Conf. PatternRecognition, pp. 185-189, 1974.

[4) K. Karu and A.K. Jain, "Fingerprint Classification," PatternRecognition, vol. 29, no. 3, pp. 389-404, 1996.

[5) D. Maio and D: Maltoni, "A Structural Approach to FingerprintClassification," Proc. 13th Int'l Conf Pattern Recognition, vol. 3,pp. 578-585, 1996.

(6) J.L. Blue, G.T. Candela, P.J. Grother, R. Chellappa, C.L. Wilson,and J.D. Blue, "Evaluation of Pattern Classifiers for Fingerprintand OCR Application," Pattern Recognition, vol. 27, pp. 485-501,1994.

(7) C.T. Candela, P.J. Grother, C.I. Watson, R.A. Wilkinson, and C.L.Wilson, "PCASYS-A Pattern-Level Classification AutomationSystem for Fingerprints," Technical Report NISTIR 5647, Nat'lInst. of Standards and Technology, Apr. 1995.

[8) U. Halici and G. Ongun, "Fingerprint Classification through Self-Organizing Feature Maps Modified to Treat Uncertainties," Proc.IEEE, vol. 84, no. 10, pp. 1497-1512, Oct. 1996.

[9) A.K. Jain, S. Prabhakar, and L. Hong, "A Multichannel Approachto Fingerprint Classification," IEEE Trans. Pattern Analysis andMachine Intelligence, vol. 21, no. 4, pp. 348-359, Apr. 1999.

[10] A. Lumini, D. Maio, and D. Maltoni, "Continuous versusExclusive Classification for Fingerprint Retrieval," Pattern Recog-nition Letters, vol. 18, pp. 1027-1034, 1997.

[Ill) T. Kamei and M. Mizoguchi, "Fingerprint Preselection UsingEigenfeatures," Proc. Computer Vision and Pattern Recognition,pp. 918-923,1998.

[12] C.I. Watson and CL. Wilson, "NIST Special Database 4:Fingerprint Database," technical report, Nat'l Inst. of Standardsand Technology, Mar. 1992.

[113] M.P. Perrone and S.D. Connell, "K-Means Clustering for HiddenMarkov Models," Proc. Int'l Workshop Frontiers in HandwritingRecognition, no. 7, pp. 229-238, 2000.

114) A. Senior, "A Hidden Markov Model Fingerprint Classifier," Proc.Asilomar Conf. Signals, Systems, and Computers, 1997.

[15) L.R. Rabiner and B.H. Juang, "An Introduction to Hidden MarkovModels," IEEE ASSP Magazine, vol. 3,; no: 1, pp. 4-16, Jan. 1986.

(16) P.C. Woodland, J.J. Odell, V.V. Valtchev, and S.J. Young, "LargeVocabulary Continuous Speech Recognition Using HTK," Proc.IEEE Int'l Conf. Acoustics, Speech, and Signal Processing, vol. 2,pp. 125-128, Apr. 1994.

[17) N.K. Ratha, K. Karu, S. Chen, and A.K. Jain, "A Real-TimeMatching System for Large Fingerprint Databases," IEEE Trans.Pattern Analysis iand Machine Intelligence, vol. 18, no. 8, pp. 799-813,Aug. 1996.

1181 O.E. Agazzi, S.-S. Kuo, E. Levin, and R. Pieraccini, "Connectedand Degraded Text Recognition Using Planar Hidden MarkovModels," Proc. IEEE Int'l Conf. Acoustics, Speech, and SignalProcessing, vol. V, pp. 113-116, 1993.

[19) K.S. Nathan, A. Senior, and J. Subrahmonia, "Initialization ofHidden Markov Models for Unconstrained On-Line HandwritingRecognition," Proc. IEEE Int'l Conf. Acoustics, Speech, and SignalProcessing, vol. 6, pp. 3503-3506, 1996.

[20] Y. Amit, D. Geman, and K. Wilder, "Joint Induction of ShapeFeatures and Tree Classifiers," IEEE Trans. Pattern Analysis andMachine Intelligence, vol. 19, no. 11, pp. 1300-1305, Nov. 1997.

[21) L. Breiman, Classification and Regression Trees. Wadsworth Int'lGroup, 1984.

122) C.L. Wilson, G.T. Candela, and C.I. Watson, "Neural NetworkFingerprint Classification," J. Artificial Neural Networks, vol. 1,no. 2, pp. 203-228, 1993.

[23) R.M. Bolle, S. Pankanti, and Y.-S. Yao, "System and Method forDetermining the Quality of Fingerprint Images," US Patent5963656, Oct. 1999.

124) A. Senior and T. Robinson, "An Off-Line Cursive HandwritingRecognition System," IEEE Trans. Pattern Analysis and MachineIntelligence, vol. 20, no. 3, pp. 309-321, Mar. 1998.

[25) D.P. Mandal, C.A. Murthy, and S.K. Pal, "Formulation of aMultivalued Recognition System," IEEE Trans. Systems, Man, and

SCybernetics, vol. 22, no. 4, pp. 607-620, July/Aug. 1992.[26) K. Rao and K. Balck; ."Type Classification of Fingerprints: A

Syntactic Approach," IEEE Trans. Pattern Analysis and MachineIntelligence, vol. 2, no. 3, pp. 223-231, May 1980.

[27) M. Kawagoe and A. Tojo, "Fingerprint Pattern Classification,"Pattern Recognition, vol. 17, no. 3, pp. 295-303, 1984.

[28) S.K. Pal and S. Mitra, "Noisy Fingerprint Classification UsingMultilayer Perceptron with Fuzzy Geometrical and Textural:Features," Fuzzy Sets and Systems, vol. 80, pp. 121-132, 1996.

[29) A.P. Fitz and R.J. Green, "Fingerprint Classification Using aHexagonal Fast Fourier Transform," Pattern Recognition, vol. 29,no. 10, pp. 1587-1597, 1996.

Andrew Senior received the PhD degree fromCambridge University in 1994 for his work on"O-Line Cursive Handwriting Recognition UsingRecurrent Neural Networks." Previously, heworked on continuous speech recognition atthe French research lab LIMSI. Joining IBM'sT.J. Watson Research Center 'in 1994, hecontinued research into handwriting recognition,before joining the Exploratory Computer VisionGropJn where he ha~ worked on finnmrint and

more recently, face recognition. He has created IBM's face detection,tracking and recognition system and cofounded IBM's audio-visualspeech effort. He is the author of numerous scientific papers, andUS patents and is workshop chair for the IEEE 2001 Workshop onRecognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems.

SFor more information on this or any other computing topic,please visit our Digital Library at http://computer.org/publications/dlib.

1174

BEST AVAILABLE COPYApplication No. 11/231,353Docket No. 577832000200

Eigenfaces for Recognition.

Matthew Turk and Alex PentlandVision and Modeling GroupThe Media LaboratoryMassachusetts Institute Of Technology

Abstract

We have developed a near-real-time computer systemi thatcan locate and track a subjects head, and then recognize theperson by comparing characteristics of the face to those ofknown individuals. The computational approach taken in thissystem is motivated by both physiology and information theory.as well as by the practical requirements of near-real-time per.formance and accurict. Our approach treats the face recog-nition problem as an intrinsically two.dimensional (2-D)

recognition problem rather than requiring recovery of three-dimensional geometry, taking advantage of the fact that facesare normally upright and thus may be described by a small setof 2-D characteristic views. The system functions by projecting

INTRODUCTION

The face is our primary focus of attention' in social in-tercourse, playing a major role in conveying identity and

emotion. Although the ability to infer intelligence or.

character from facial appearance is suspect, the humanability to recognize faces is remarkable'. We can recog-nize thousands of faces learned throughout our lifetimeand identify familiar faces at a glance even after years ofseparation. This skill is quite robust, despite largechanges in the visual stimulus due to viewing conditions,expression, aging, and distractions ' such as glasses orchanges in -hairstyle or facial hair. As a consequence thevisual processing of human faces has fascinated philos-'ophers and scientists for centuries, including figures such,as Aristotle and Darwin..

Computational models of face recognition, in partic-ular, are interesting because they can contribute not onlyto theoretical insights but also to practical applications'.Computers that recognize faces could be applied to awide variety of problems, including criminal identifica-tion, security systems, image and. film processing, andhuman-computer interaction. For example, the ability tomodel a particular face and distinguish it from a largenumber of stored face models would nuke it possibleto vastly improve criminal identification. Even the abilityto merely detect faces, as opposed to recognizing them.

© 1991 Alan 3cdjuse(m Insrituie of Tecrno(oin,

face images onto a feature space that spans th signilicantvariations among known face images. The significant featuresare known as "eigenfaces," because they are the eigenvectors(principal components) of the set of faces; they do not neces-sarily correspond to features such as eyes, ears. and noses. Theprojection operation characterizes, an individual face by aweighted sum of the eigenface features, and so to recognize aparticular face it is necessary only to compare these weights tothose of known individuals. Some particular advanrtages of ourapproach are that it provides for the ability to learn and laterrecognize new faces in an unsupervised manner, and that it iseasy to implement using a neural network architec.iare. U

can be important. Detecting faces in photographs, forinstance, is an. important problem in automating colorfilm development, since the effect of many enhancementand noise reduction techniques depends on the picturecontent (e.g., faces should not be tinted green, whileperhaps grass should).

Unfortunately, developing a computational model offace recognition is quite difficult, because faces are com-plex, multidimensional, and meaningful visual stimuli.They are a' natural class of objects, and stand in starkcontrast to sine wave gratings. the "blocks world," andother artificial stimuli used in human and computer vi-sion research (Davies. Ellis, & Shepherd, 1981). Thus

unlike most early visual fuinctions, for which we mayconstruct detailed models of retinal or striate activity,face recognition is a very high level task for which con-putational approaches can currently only suggest broadconstraints on the corresponding neural activity.

We therefore 'focused our research toward developinga sort of early, preattentive pattern recognition capabilitythat does not depend on having three-dimensional in-formation or detailed geometry. Our goal, which webelieve we have reached, was to develop a computationalmodel of face recognition that is fast, reasonably simple.and accurate in constrained environments such as anoffice or a household. In addition the approach is bio-logically implemeniable and is in concert with prelirni-

Journal of Cognitive Neuroscience Volunme 3, ,Number 1

~bg--

',1 * .'T

nary findings in the physiology and psychology of facerecognition.

The scheme is based on an information theory ap-proach that decomposes face images into a small set ofcharacteristic feature images called "eigenfaces," whichmay be thought of as the principal components of theinitial training set of face images. Recognition is per-formed by projecting a new image into the subspacespanned by the eigenfaces ("face space") and then clas-sifying the face by comparing its.position in face spacewith the positions of known individuals.

Automatically learning and later recognizing new facesis practical within this framework. Recognition underwidely varving conditions is achieved by training on alimited number of characteristic views (e.g.. ai "straighton" view, a -i5° view, and a profile view). The approachhas advantages over other face recognition schemes inits speed and simplicity, learning capacity, and insensitiv-ity to small or gradual changes in the face image.

Background and Related Work

Much of the work in computer recognition of faces hasfocused on detecting individual features such as the eyes,nose, mouith, and head outline, and defining a face modelby the position, size, and relationships among these fea-tures. Such approaches have proven difficult to extendto multiple views, and have often been quite fragile,requiring a good initial guess to guide them. Researchin human strategies of face recognition, moreover hasshown that individual features and their immediate re-lationships comprise an insufficient representation to ac-count for the performance of adult. human faceidentification (Carey & Diamond, 1977). Nonetheless,.this approach to face recognition remains the most pop-ular one in the computer vision literature.

Bledsoe (1966a.b) was the first to attempt semiauto-mated face recognition with a hybrid human--computersystem that classified faces on the basis of fiducial inmarksentered on p)hotographs by hand. Parameters for theclassification were normalized distances and ratiosamong points such as eve corners, mouth corners, nosetip, and chin point. Later work at Bell Labs (Goldstein.Harmon, & Lesk, 1971: Harmon, 1971) developed a vec-tor of up to 21 features. and recognized faces usingstandard pattern classification techniques. The chosenfeatures were largely subjective evaluations (e.g., shadeof hair, length of ears, lip thickness) made by humansubjects, each of which would be quite difficult toautomate.

An early paper by Fischler and Elschlager (1973) at-tempted to measure similar features automatically. Theydescribed a linear embedding algorithm that used localfeature template matching and a global measure of fit tolind and measure facial features. This template matchingapproach has been continued and improved by the re-cent work of Yuille, Cohen, and Hallinan (1989) (see

Yuille, this volume). Their strategy is based on "deform-able templates," which are parameterized models of theface and its features in which the parameter values aredetermined by interactions with the image.

Connectionist approaches to face identification seek tocapture the configurational, or gestalt-like nature of thetask. Kohonen (1989) and Kohonen and Lahtio (1981)describe an associative network with a simple learningalgorithm that can recognize (classify) face images andrecall a face image from an incomplete or noisy versioninput to the network. Fleming and Cpttrell (1990) extendthese ideas using nonlinear units, training the system bybackpropagation. Stonhamn's WISARD system (1986) is agencral-purpose pattern recognition device based on:neural net principles. It has been applied with somesuccess to binary face images, recognizing both identityand expression. Most connectionist systems dealing withfaces (see also Midorikawa. 1988; O'Toole, Millward, &Anderson, 1988) treat the input image as a general 2-Dpattern, and can make no explicit use of the conifigura-tional properties of a face. Moreover, some of thesesystems require an inordinate number of training ex-amples to achieve a reasonable level of performance.Only very simple systems have been explored to date,and it is unclear how they will scale to larger problems.

Others have approached automated face recognitionby characterizing a face by a set of geometric parametersand.performing pattern recognition based on the param-eters (e.g., Kaya & Kobayashi, 1972; Cannon, Jones,Campbell, & Morgan, 1986; Craw, Ellis, & Lishman, 1987;\Wong, Law, & Tsaug, 1989). Kanade's (1973) face identi-fication system was the first (and still one of the few)systems in which all steps of the recognition processwere automated, using a top-down control strategy di-rected by a generic model of expected feature charac-teristics. His system calculated a setof facial parametersfrom a single face image and used a pattern classificationtechnique to match the face from a known get, a purelystatistical approach depending primarily on local histo-gram analysis and absolute gray-scale values.

Recent work by Burt (1988a,b) uses a "smart sensing"approach based on multiresolution template matching,This coarse-to-fine strategy uses a special-purpose com-puter built to calculate multiresolution pyramid imagesquickly, and has been demonstrated identifying peoplein near-real-time. This system works well under limitedcircumstances, but should suffller from the typical prob-lems of correlation-based matching, including sensitivityto image size and noise. The face niodels are built byhand from face images.

THE EIGENFACE APPROACH

Much of the previous work on automated face recogni-tion has ignored the issue of just what aspects of the facestimulus are important for identification. This suggestedto us that an information theory approach of coding and

72 Journal of Cognitir e .Veuroscience Volume 3, Number 1

decoding face images may give insight into the infor-mation content of face images, emphasizing the signifi-cant local and global "features." Such features niay ormay not be directly related to our intuitive notion of facefeatures such as the eyes, nose, lips, and hair. This mayhave important implications for the use of identificationtools such as Identikit and Photofit (Bruce, 1988).

In the language of information theory, we want toextract the relevant information in a face image, encodeit as efficiently as possible, and compare one face encod-ing with a database of models encoded similarly. A simpleapproach to extracting the information contained in animage of a face is to somehow capture the variation in acollection of face images, independent of any judgmentof features, and use this information to encode and com-pare individual face images.

In mathematical terms, we wish to find the principalcomponents of the distribution of faces, or the eigenvec-tors of the covariance matrix of the set of face images,treating an image as a point (or vector) in a very highdimensional space. The eigenvectors are ordered, eachone accounting'for a different amount of the variationamong the face images.

These eigenvectors can be thought of as a set of fea-tures that together characterize the variation betweenface images. Each image location contributes more orless to each eigenvector, so that we can display the ei-genvector as a sort of ghostly face which we call aneigenface. Some of the faces we studied are illustratedin Figure 1, and the corresponding eigenfaces are shownin Figure 2. Each eigenface deviates from uniform graywhere some facial feature differs among the set of train-ing faces; they are a sort of map of the variations betweenfaces.

Each individual face can be represented exactly interms of a linear combination of the eigenfaces. Eachface can also be approximated using only the "best"eigenfaces--those that have the largest eigenvalues, andwhich therefore account for the most variance withinthe set of face images. The best M1 eigenfaces span anMA-dimensional subspace-"face space"--of all possibleimages.

The idea of using eigenfaces was motivated by a tech-nique developed by Sirovich and Kirby (1987) and Kirbyand Sirovich (1990) for efficiently representing picturesof faces using principal component analysis. Starting withan ensemble of original face images, they calculated abest coordinate system for image compression, whereeach coordinate is actually an image that they termed aneigenpicture. They argued that. at least in principle, an-collection of face images can be approximately recon-structed by storing a small collection of weights for eachface and a small set of standard pictures (the eigenpic-tures). The weights describing each face are found by

projecting the face image onto each eigenpicture.It occurred to us that if a multitude of face images can

be reconstructed by weighted sums of a small collection

of characteristic features or eigenpictures, perhaps anefficient way to learn and recognize faces would be tobuild up the characteristic features by experience overtime and recognize particular faces by comparing, thefeature weights needed to (approximately) reconstructthem with the weights associated with known individuals.Each individual, therefore, would be characterized bythe small set of feature or eigenpicture weights neededto describe and reconstruct them-an extremely corn-

pact representation when compared with the imagesthemselves.

This approach to face recognition involves the follow-ing initialization operations:

1. Acquire an initial setof face images (the trainingset).

2. Calculate the eigenfaces from the training set, keep-ing only the M images that correspond to the highesteigenvalues. These M images define the jace space. Asnew faces are experienced, the eigenfaces can be up-

dated or recalculated.3. Calculate the corresponding distribution in M-di-

mensional weight space for each known individual, byprojecting their face images onto the 'face space.

These operations can also be performed from timeto time whenever there is free excess computationalcapacity.

Having initialized the system,- the following steps arethen used to recognize new face images:

1. Calculate a set of weights based on the input imageand the Ma eigenfaces by projecting the input image onto

each of the eigenfaces.2. Determine if the image is a face at all (whether

known or unknown) by checking to see if the image issufficiently close to "face space."

3. If it is aface, classify the weight pattern as either aknown person or as unknown.4. (Optional) Update the eigenfaces and/or weight

patterns.5. (Optional) If the same unknown face is seen several

times, calculate its characteristic weight pattern and in-corporate into the knaown faces.

Calculating Eigenfaces

Let a face image I(x.y) be a two-dimensional N by N arrayof (8-bit) intensity values. An image may also be consid-ered as a vector of dimension N2 , so that a typical imageof size 256 by 256 becomes a vector of dimension 65,536.or, equivalently, a point in 65,536-dimensional space. Anensemble of images, then, maps to a collection of pointsin this huge space.

Images of faces, being similar in overall configuration.will not be randomly distributed in this huge image spaceand thus can be described by a relatively low dimen-sional subspace. The main idea of the principal compo-

Turk and Pentland 73

Figure i. (a)Face inmaigesused as the training .et.

nent analysis ( or Karhunen-Loeve expansion) is to findthe vectors that best account for the distribution of faceimages within the entire image space. Tl'hese vectors de-fine the subspace of face images. which we call "facespace. Each vector is of length N. describes an N by .Vimage, and is a linear combination of the original faceimages. Because these vectors are the eigenvectors ofthe covariance matrix corresponding to the original faceimages, and because they are face-like in appearance, werefer to them as "eigenfaces." Some examples of eigen-faces are shown in Figure 2.

Let the training set of face images be . _.. ...Fr.. The average face of the set is defined by T =

2',2', F,,. Each face differs from the average by thevector ), = F, - V. An example training set is shownin Figure la. with the average face T shown in Figurelb. This set .of vetry large vectors is then subject to prin.cipal component analysis, which seeks a set of Mi ortho-normal vectors. u,,. which best describes the distributionof the data. The kth vector. uk, is chosen such that

,is a maximum subject to 1iS a mlaximum.m subiect to

= i, if!. = kI " " , 0, otherwise (2)

The vectors uk and scalars XA. are the eigenvectors andeigenvalues, respectively, of the covariance matrix

C

=.._ if

where the matrix A = (4), ... .,i). 'The matrix C.however, is N" by .V, and determining the N2 eigenvec-tors and eigenvalues is an intractable task for typicalimage sizes. We need a computationally feasible methodto find these eigenvectors.if the number of data points in the image space is less

than the dimension of the space .(M1 < NZ), there will beonly M - 1, rather than N2, meaningful eigenvectors.(The remaining eigenvectors will have associated eigen-values of zero.) Fortunately we can solve for the N2-dimiensional, eigenvectors in this case by first solving forthe eigenvectors of an M by M matrx-e.g., solving a16 .x 16 matrix rather than a 16.384 x 16,384 matrix-

74 Journal of Cognitiv e .Veuroscitence Volume 3, Number I

Figure 1. (b) The average face '1.

Figure 2. Seven of the eigenfaces calculated from the input imagesof Figure .,

and then taking appropriate linear combinations of theface images ,;. C6nsider the eigetwvectors v: of A'A .-uchthat

AfAv, = ,v, t-t )

PremultiplVing both sides by A. we have

.A A v, = -\v, t5)

from which we see that Av, are the eigenvectors of C =AA.

Following this analysis, we construct the Al by MAl matrixL - AlA, where L,,,,, = 4,,,, and find the Ml.eigenvec.tors, v,, of L. These vectors determine linear combina-tions of the M training set face images to form theeigenfaces ui.

U, = vu / = 1, . . ;:,

With this analysis the calculations are greatly reduced,from, the order of the number of pixels in the images(Ni) to the order of the number of images in the trainingset (M). In practice, the training set of face images willbe relatively small (l N), and the calculations becomequite manageable. The associated eigenvalies allow usto rank the eigenvectors according to their usefulness incharacterizing the variation among the images. Figure 2shows the top seven eigenfaces derived from the inputimages of Figure 1.

Using Eigenfaces to Classify a Face Image

The eigenface images calculated from the eigenvectorsof L span a basis set with which to describe face images.Sirovich and Kirby (1987) evaluated a limited version ofthis framework on an ensemble of M = 115 images ofCaucasian males, digitized in a controlled manner, andfound that about 40 eigenfaces were sufficient for a verygood description of the set of face images. With Ml' =40 eigenfaces. RMS pixel-by-pixel errors in representingcropped versions of'f6ce images were about 2%.

Since the eigenfaces seem adequate for describing faceimages under very controlled conditions, we decided toinvestigate their usefulness as a tool for face identifica-tion In practice, a smaller M' is sufficient for identifica-tion, since accurate reconstruction of the image is not arequiirement. In this framework, identification becomesa pattern recognition task. The eigenfaces span an M'-dimensional subspace of the original NV' image space.The MAl' significant eigenvectors of the L matrix are chosenas those with the largest associated eigenvalues. In manyof our test cases. based on Al = 16 face images, l' = 7eigenfaces were used.

A new face image (r) is transformed into itseigenfacecomponents (projected into "face space") by a simpleoperation,

Wk = u[( r- q')

for k = 1 .... , A1'. This describes a set of point-b)y-p6intimage multiplications and summations, operations per-formed at approximately frame rate on current imageprocessing hardware. Figure 3 shows an image and itsprojection into the seven-dimensional face space.

The weights form a vector f1. = (w,, w2 . .. w.r] thatdescribes the contribution of each eigenface in repre-senting the input face image, treating the eigenfaces as abasis set for face images. The vector may then be used


in a standard pattern recognition algorithm to find whichof a number of predefined face classes, if aiy, best de-scribes the face. The simplest method. for determiningwhich face class provides the best description of an input

face image is to find the face class k that minimizes the

Euclidian distance

= ( A IO)11 • (8)

where lk is a vector describing the kth face class. Theface classes fl are calculated by averaging the results ofthe eigenface representation over a small number of faceimages (as few as one) of each individual. A face isclassified as belonging.to class k when the minimum Ekis below some chosen threshold 0. Otherwvise the faceis classified as ':unknown," and optionally used to createa new face class.

Because creating the vector of weights is equivalent toprojecting the original face image onto the low-dimen-sional face space: many images (most of them lookingnothing like a face) will project onto a given patternvector. This is not a problem for the system, however,

since the distance E between the image and the facespace is simply the squared distance between the mean-adjusted input image ( = F - T and Or = w1 ,u;,its projection onto face space:

E = ib- 4'dir

Thus there are four possibilities for an input image andits pattern vector: (1) near face space and near a faceclass, (2) near face space but not near a known face class,(3) distant from face space and near a face class, and (4)distant from face space and not near a known face class.

In the first case, an individual is recognized and iden-tified. In the second case, an unknown individual is pres-ent. The last two cases indicate that the image is not aface image. Case three typically shows up as a false pos-itive in most recognition systems: in our framework,however, the false recognition may be detected becauseof the significant distance between the image and thesubspace of expected tface images. Figure 4 shows someimages and their projections into face space and gives ameasure of distance from the face space for each.

Summary of Eigenface RecognitionProcedure

To summarize, the eigenf aces approach to face recogni-tion involves the following steps:

1. Collect a set of characteristic face images of theknown individuals. This set should include a number ofimages for each person, with some variation in expres-sion and in the lighting. (Say four images of ten people,so .M' = -10.)

2. Calculate the (40 x 40) matrix L, find its eigenvec-tors and eigenvalues, and choose the MAl' eigenvectors

with the highest associated eigenvalues. (Let Al' = 10 inthis example.)

3. Combine the normalized training set of images ac-cording to Eq. (6) to produce the (M' = 10) eigenfacesu .

4. For each known individual, calculate the class vec-tor Ok by averaging the eigenface pattern vectors f (fromEq. (8)) calculated from the original (four) images of theindividual. Choose a threshold 0, that defines the maxi-mum allowable distance from any face class, and athreshold 06. that defines the maximum allowable dis-tance from face space [according to Eq. (9)].

5. For each new face image to be identified, calculateits pattern vector Q, the distances E, to each known class,and the distance e to face space. If the minimum distanceek < 08 and the distance E < 0,, classify the input faceas the individual associated with class vector St.. If theminimum distance e > 0, but distance E < 0,, then theimage may be classifed as "unknown," and optionallyused to begin a new face class.

6. If the new image is classified as a known individual.this image may be added to the original set of familiarface images, and the eigenfaccs may be recalculated(steps 1-4). This gives the opportunity to modify the facespace as the system encounters more instances of known

In our current system calculation of the eigenfaces isdone offline as part of the training. The recognitioncurrently takes about 400 msec running rather ineffi-ciently in Lisp on a Sun4, using face images of size 128 x128. With some special-purpose hardware, the currentversion could un' at close to frame rate: (33 msec).

Designing a practical system for face recognitionwithin this framework requires assessing the tradeoffsbetween generality, required accuracy, and speed. If theface recognition task is restricted to a small set of people(such as the members of a family or a small company),a small set of cigenfaces is adequate to span the faces of,interest. If the system is to learn new.faces or representmany people. a larger basis set of eigenfaces will berequired. The results of Sirovich and Kirby (1987) andKirby and Sirovich (1990) for coding of face images givessome evidence that even if it werenecessary to representa large segment of.the population, the numnher of eigen-faces needed would still he relatively small.

Locating and Detecting Faces

The analysis in the preceding sections assumes we havea centered face image, the same size as the trainingimages and the eigenfaces. We need some way, then, tolocate a face in a scene to do the recognition. We havedeveloped two schemes to locate and/or track faces, us-ing motion detection and manipulation of the images in"face space".

76 Journal of Cognitive Neuroscience

feces .

Volumerr 3, Number 1

Figure 3. An original face image and its projection onto the facespace delined by the eigenfaces of Figure 2.

Motion Detecting and Head Tracking

People are constantly moving. Even while sitting, wefidget and adjust our body position, nod our heads, lookaround, and such. In the case of a single person movingin a static environment, a simp!e motion detection andtracking algorithm, depicted in Figure 5, will locate andtrack the position of the head. Simple spatiotemporalfiltering (e.g., frame 'differencing) accentuates image lo-cations that change with time, so a moving person "lightsup" In the filtered image. If the image "lights up" at all,motion is detected and the presence of a person ispostulated.

Ater thresholding the filtered image to produce abinary motion image, we analyze the "motion blobs" overtime to decide if the motion is caused by a personmoving and to determine head position. A few simplerules are applied, such a:is "the head is the small upperblob above a larger blob (the body)," and "head motionmust be reasonably slow and contiguous" (heads are notexpected to jump around the image erratically). Figure6 shows an image with the head located, along with thepath of the head in the preceding sequence of frames.

The motion image also a llows for an estimate of scale.The size of the blob that is assumed to be the movinghead determines the size of the subimage to send to therecognition stage. This subimage is rescaled to fit thedimensions of the eigenfaces.

Using "Face Space" to Locate the Face

We can also use knowledge of the face space to locatefaces in single images. either as an alternative to locating

faces from motion (e.g., if there is too little motion ormany moving objects) or as a method of achieving more

precision than is possible by use of motion trackingalone. This method allows us to recognize the presenceof faces apart from the task of identifying them.

As seen in Figure 4, images of faces do not changeradically when projected into the face space, while theprojection of nonface images appears quite different.This basic idea is used to detect the presence of faces ina scene: at every location in the image, calculate thedistance E between the local subimage and face spaceThis distance from face space is used as a measure of"faceness." so the result of calculating the distance fromface space at even point in the image is a ''face map'e(x,y). Figure 7 shows an image and its face map-lowvalues (the dark area) indicate the presence of a face.

Ulnfortunatelyv, direct application of Eq. (9) is ratherexpensive. We have therefore developed a simpler, moreefficient method of calculating the face map E(x '), whichis described as follows.

To calculate the face map at every pixel of an image(xy), we need to project the subimage centered at that

pixel onto face space, then subtract the projection fromthe original. To project a subimage r onto face space.we must first subtract the mean image. resulting in i =r = V. With At being the projection of 0 onto facespace. the.distance measure at a given image location isthen

- (t -I r) 7QF - 4)r)-(D T( --Of o

Trk anti Pentland. 77

(10.)

__

Note to the reader:

There are typos here that have haunted me for years, so here is a briefcorrection. First, the third line in equation (10) should not have aplus sign - rather, the plus should be replaced by a negative. Also,

the second term of the fourth line should be OTq f rather than $Of.

This carries into equation (11). However, since the last term in thethird line of equation (10) is equal to zero (due to the fact they'reSperpendicular), this means that these two terms are actually equivalent

- i.e.,

So even though the wrong terms are written in the derivation, it's

actually still correct.

Matthew Turk

Figure 4. Three images andtheir projections onto the facespace defined by the eigen-faces of Figure 2. The relativemeasures of distance from facespace are (a) 29.8, (b) 58.5.(c) 5217.4. Images (a) and (b)are in the original training set.

since Wr 1 (4 -(r). Because 4f is a linear comnbin;

of the eigenfaces (Or = -.= w;u,) and the eigenare orthonormal vectors.,

r .4 -,p I

and

E(.;y) = FT ; ) ,) 1)(x,.) - X &;(.v..y . (i2)

where e(x, y) and w(.x, y) are scalar functions of imagelocation, and Q(x,y) is a vector function of image loc'l-tion.

The second term of Eq. (12) is calculated in practiceby a correlation with die L eigenfaces:

78 Jor1rnal of Cognilive eur-oscience

ition -1 (x, y) = =im'(x, v)u,faces = :. [F(x, ) - r)ru

= > , ([(.4;y)u, - IWuI,= 1)., I((.1y') 0 u; - 'u;]

(11) where @ is the correlation operator. TheEq.. (12) becomes

(13)

first term of

/ (14)

so that

Ei,')=[ T( ;y)r(.r. y) - 2r(...')® +~±'14 +

( (F'(x j:O®u, - w p ul 15

Vo(,unw 3. VNmber I

Figure 5. T'h'e hcad ira king ntl Itxaing ;sriein.

works, these computationssimple neural network.

can be implemented by

Figure 6. The head has been located-theto the face recognition process, Also showntracktd. over several previous I'rames.

image in the box is ;enti. the path of the head

Since the average face W and the eigenlices u, are fixed.the terms 'Wr. and 0 u, may be computed aheadof time.

Titus the computati()on nOf the tee nmap inyolves onl"L + I correlations over the input image and the com-putation of the first iernim r'(.L, ) y). This is comn-puted by squaring the input ima,e I(.Xv, y.) and. at eachimage location. summing the squared values of the localsubimage. As discussed in the section on Neural Net-

Learning to Recognize New Faces

The concept of face'space allows the ability to learn andsubsequently recognize new faces in an unsupew'isedmanner. When an image is sufficiently close to face spacebut is not classified as one of the familiar faces, it isinitially labeled as "unknown." The computer stbres thepattern vector and the corresponding unknown image.if a collection of "unknown" pattern vectors cluster inthe pattern space, the presence of a new but unidentifiedface is postulated.

The images corresponding to the pattern vectors inthe cluster are then checked for similarity by requiringthat the distance from each .'imige to the mean of theimages is less than a predefined threshold. If the imagespass the similarity test, the average of the feature vectors,.is added to the database of known faces. Occasionally,the eigenfaces may be recalculated using these storedimages as part of the new training set,

Other Issues

A number of other issues must be addressed to obtain arobust working system. In this section we. will brieflymention these issues and indicate methods of solution.

Eliminating the Background

In the preceding analysis we have ignored the effect ofthe background. In practice, the background can signif-icantly effect the recognition performance, since the ei-


--

Figure 7. (a) Original image. (b) The corresponding face map, where low values (dark areas) indicate the presence of a face.

genface analysis as described above does notdistinguishthe face from the rest of the image. in the experimentsdescribed in the section on Experiments with Eigenfaces.,the background was a significant part of the image usedto classify the faces.

To deal with this problem without having to solveother difficult vision problems (such as robust segmen-tation of the head), we have multiplied the input faceimage by a two-dimensional gaussian window centeredon the face, thus diminishing the background and accen-tuating the middle of the face. Experiments in humanstrategies of face recognition (Hay & Young, 1982) citethe importance. of the internal facial features for recog-nition of familiar faces. Deemphasizing the outside ofthe face is also a practical consideration since changinghairstyles may otherwise negatively affect the recogni-tion.

Scale (Head Size) and Orlentation Irwariance

The experiments in the section on Database of Faceimages show that recognition performance decreaisesquickly as the head size, or scale, is misjudged. The headsize in the input image must be close to that of theeigenfaces for the system to work well. The motion :anal-ysis gives an estimate of head size, from which the faceimage is rescaled to the eigenface size.

Another approach to the scale problem, which may besel)arate from or in addition to the motion estimate, isto use multiscale eigenfaces, in which an input lFace imageis compared with eigenfaces at a number of scales. Inthis case the image will appear to be near the face spaceof only the closest scale eigenfaces. Equivalently, we can

scale the input image to multiple sizes and use the scalethat results in the smallest distance measure to face space.

Although the eigenfaces approach is not extremelysensitive to head orientation (i.e.:, sideways tilt of thehead), a non-upright view will cause some performancedegradation.. An accurate estimate of the head tilt willcertainly benefit the recognition. Again, two simple meth-ods have been considered and tested. The first is tocalculate the orientation of the motion blob of the head.This is less reliableas the shape tends toward a circle,however. Using the fact that faces are reasonably sym-metric patterns, at least for frontal views, we have usedsimple symmetr, operators to estimate head orierntation.Once the orientation is estimated; the image can berotated to align the head with the eigenfaces.

Distribution in Face Space

The nearest-neighbor classification previously describedassumes a Gaussian distribution in face space' of an in-dividual's feature vectors O. Since there is no a priorireason to assume any particular distribution, we want tocharacterize it rather than assume it is gaussian. Nonlin-ear networks such as described in Fleming and Cottrell(1990) seem to be a promising way to learn the facespace distributions by example.

Muldtiple Views

We *are currently extending the system to deal with otherthan full frontal views by defining a limited number offace classes for each known person corresponding tocharacteri-istic views. For example, an individual may berepresented by face classes corresponding to a frontal

80 Journal of Cognitive leuroscience V'olumne X Number lw

face view, side views, at ± -45°, and right and left profileviews. Under most viewing conditions these seem to besufficient to recognize a face anywhere from frontal toprofile view, because the real view can be approximatedby interpolation among the fixed views.

EXPERIMENTS WITH EIGENFACES

To assess the viability of this approach to face'recogni-lion, we have performed experiments with stored face

images and built a system. to locate and recognize facesin a dynamic environment. We first created a large da:-tabase of face images collected under a wide range ofimaging conditions. Using this database we have con-ducted several experiments to assess the performanceunder known variations of lIghting, scale, and orienta-tion. The results of these experiments and early experi-ence with the near-real-time system are reported in thissection.

Database of Face Images

The images fronm Figure la were taken from a databaseof over 2500 face images digitized under controlled con-ditions. Sixteen subjects were digitized at all combina-tions of three head orientations, three head sizes orscales, and three lighting conditions. A six level Gaussianpyramid was constructed for each image, resulting inimage resoltition from 512 x 512 pixels down to 16.x16 pixels. Figure 8 shows the images from one pyramidlevel for one individual.

In the first experiment the effects of varying lighting,size, and head orientation were investigated using thecomplete database of 2500 images of the 16 individualsshown in Figure la. Various groups of 16 images wereselected and used as the training set. Within each trainingset there was one image of each person, all taken underthe samrne conditions of lighting, iluage Siz, anrd headorientation. All images in the database were then classi-fled as being one of these sixteen individuals (i.e., thethreshold 0, was effectively infinite, so that no faces wererejected as unknown). Seven eigenfaces were used inthe classification process.

Statistics were collected measuring the mean accuracyas a function of the difference between the training con-ditions and the test conditions. The independent varia-bles were difference in illumination, imaged head size,head orientation, and combinations of illumination, size.and orientation.

Figure 9 shows results of these experiments for thecase of infinite 0,. The graphs of the figure show thenumber of correct classifications for varying conditionsof lighting, size. and head orientation, averaged over thenumber of experiments. For this case where every, faceimage is classified as known, the system achieved ap-proximately 9605 correct classification averaged over

lighting variation, 85% correct averaged over orientationvariation, and 64% correct averaged over size variation.

As can be seen from these graphs, changing lightingconditions causes relatively few errors, while perfor-mance drops dramatically with size change. This is notsurprising, since under lighting changes alone the neigh-borhood pixel correlation remains high, but under sizechanges the correlation from. one image to another islargely lost. It is clear that there is a need for a multiscaleapproach, so that faces at a particular size are comparedwith one another. One method of accomplishing this isto make sure that each "face class" includes images ofthe iidividual at several different sizes, as was discussedin the section on Other Issues.

In a second experiment the same procedures werefollowed. but the acceptance threshold 6, was also var-ied, At low values of 0E, only images that project veryclosely to the known face classes will be recognized, sothat there will be few errors but many of the images willhe rejected as unknown. At high values of O. most imageswill be classified, but there will be more errors: Adjusting0e to achieve 100% accurate recognition boosted theunknown rates to 19% while varving lighting, 39% fororientation, and 60% for size. Setting the unknown ratearbitrarily to 20% resulted in correct recognition ratesof 100%. 94%. and 7".% respectively.

These experiments show an increase of performanceaccuracy as the threshold decreases. This can be tunedto achieve effectively perfect recognition as the thresholdtends to zero, but at the cost of many images beingrejected as unknown. The tradeoff between rejection rateand recognition accuracvy will be different for each of thevarious face recognition applications. 1lowever. whatwould be most desirable is to have a way of setting thethreshold high, so that few known face images are re-jected as unknown. while at the same.time detecting theincorrect classifications. That is, we would like to in-crease the efficiency (.the d-prime) of the recognitionprocess.

Ofie way of accomplishing this is to also examine the(normalized) Euclidian distance between an image aindface space as a whole. Because the projection onto theeigenface vectors is a many-to-one mapping, there is apotentially unlimited number of images that can projectonto the eigenfaces in the same manner. i.e.. produce.the same weights. Many of these will look nothing like aface. as shown in Figure -tc. This approach was describedin the section on Using "Face Space" to Locate the Faceas a method of identifying likely face subimages.

Real-Time Recognition

We have used the techniques described above to builda system that locates and recognizes faces ,in near-real-time in a reasonably unstructured environment. Figure10 shows a diagramn of the system. A fixed camera, mon-'itoring part of a room, is connected to a.Datacube image

Tur an Petlad 8

I

FIgure 8. Vari3tion of face in:tges foi- onr individu:aI: rhree head .'izvs'. threc lighting conditins. n:d three head orienuraions.

processing system, which resides on the bus of a. Sun 3/160. The Datacube digitizes the video image and per.forms spatiotemporal filtering, thresholding, and sub-sampling at frame rate (30 frames/sec)., (The images aresubsampled to speed up the motion analysis.)

The motion detection and analy'sis programs run onthe Sun 3/160, first detecting a moving object and thentracking the motion and applying simple rules to deter.mine if it is tracking a head. When a head is found. thesubimage, centered on the head, is sent to another com-puter (a Sun Sparcstation) that is running'the fatce rec-ognition program (although it could be running on thesame computer as the motion program). Using the dis-tance-from-face.space measure. the image is either re-

jected as not a face, recognized as one of a group offamiliar faces. or determined to be an unknown face.

Recognition occurs in this system at rates of up to twoor three times per second. Until motion is detected, oras long as the image is not perceived to be a face, thereis no output.. When a face is recognized, the image ofthe identified individual is displayed on the Sun monitor.

RELATIONSHIP TO BIOLOGY ANDNEURAL NETWORKS

Biological Motivations

High-level recognition tasks are typically modeled as re-quiring mnianv stages of processing, e.g., the Marr (1982)

82 Journal of Cognitive Neurosciunce Volume 3, NVumber 1

7.36

.. «

(a)

6.36 .4

1.61"."1.0 ? \\

'.0

(b)

11.8

11.0

6.3M

-.0 0r

7 (M.

1636

12.

1.06 2.66

(f)

..11.0 T

62.4l

1.6 I

7.10eIO t "- rt- -

Figure 9. Resuts of experiments measuring recognition perfor-mance using eigenfaces. Each graph shows averaged performance asthe lighting conditions, head aize. and head orientation vary-the y-.axis depicts number of correct classifications tout of 16). The peak(1616 correct) in each graph results from recognizing the particulartraining set perfectly. The other twvo graph points reveal the declinein performance as lthe following parameters are varied: (a) lighting,(b) head size (scale),- (c) orientation, (d) orientation and lighting,(e) orientation and size ( #1i. (f) orientation and size ( #2), (g) sizeand lighting. (h) size and lighting (#2).

paradigm of progressing from images to surfaces tcthree-dimensional models to matched models. Howeverthe early development and the extreme rapidity of facerecognition makes it appear likely that there must alscbe a recognition mechanism based on some fast, low.level, two-dimensional image processing.

On strictly phenomenological grounds, such a facerecognition mechanism is plausible because faces arctypically seen in a limited range of views, and are a vernimportant stimulus for humans from birth. The existenceof such a mechanism is also supported by the results o.a number of physiological experiments in monkey cortexclaiming to isolate neurons that respond selectively t.faces (e.g., see Perrett. Rolls. & Caan, 1982. Perrett, Mistlin, & Chitty. 198': Bruce, Desimone. & Gross. 1981Desimone. Albright, Gross, & Bruce, 1984; Rolls, BaylisHasselmo, & Nalwa. 1989). In these experiments, somet

, , 1linear network that implements a significant part of thesystem. The input layer receives the input (centered andnormalized) face image, with one element per imagepixel, cir N elements..' The weights from the input layerto the hidden layer correspond to the eigenfaces, so thatthe value of each hidden unit is the dot product of, theinput image and the corresponding eigenface: w; = ru,. The hidden units, then, form the pattern vector f =1wt, W1 . . WE].

The output laver produces.the face space projectionof the input image when the output weights also corre-spond to the eigenfaces (mirroring the input weights).

, Adding two nonlinear components we construct Figure12, which produces the pattern class fl, face space pro-jection (r, distance measure d (between the image and

S its face space projection), and a classification vector. Theclassification vector is comprised of a unit for eachknown face defining the pattern space distances e;. Theunit with the smallest value, if below the specified thresh..old 0e, reveals the identity of the input face imlage.

Parts of the network of Figure 12 are similar to thef ., associative networks of Kohonen (1989) and Kohonen

and Lehtio (1981). These networks implement a learnedstimulus-response mapping, in which the learning phase

- modifies the connection weights. An autoassociative.net-

S work implements the projection onto face space. Simi-, larly, reconstruction using eigenfaces can be used to"recall a partially occluded face, as shown in Figure 13.

Turk and Pentland 8.3

cells were sensitive to identity, some to "faceness," andsome only to particular views (such as frontal or profile).

Although we do not claim that biological systems have"eigenface cells" or process faces in the same way as theeigenface approach, there are a number of qualitativesimilarities between our approach and current under-standing of human face recognition. For instance, rela-tively small changes cause the recognition to degradegracefully, so that partially occluded faces can be recog-nized, as has been demonstrated in single-cell recordingexperiments. Gradual changes due to aging are easilyhandled by the occasional recalculation of the eigenfaces,so that the system is quite tolerant to everi large changesas long as they occur over a long period of time. If,however, a large change occurs quickly--e.g., additionof a disguise or change of facial hair-then the eigenfacesapproach will be fooled, as are people in conditions ofcasual observation.

Neural Networks

Although we have presented the eigenfaces approach toface recognition as an informination-processing model, itmay be implemented using simple parallel computingelements, as in a connectionist system or artificial neuralnetwork. Figure 11 shows a three-layer fully connected

Figure 10. System diagram ofthe face recognition system.

Figure 11. Three-layer linearnetwork for eigenface calcula-tion. The symmetric weights u,are the eigenfaces, and thehidden units reveal the projec-tion of the input image ( ontothe eigenlaces. The output (tis the face space projection ofthe input image.

CONCLUSION

Early attempts at making computers recognize faces werelimited by the use of impoverished face models andfeature descriptions (e.g., locating features from an edgeimage and matching simple distances and ratios), assum-ing that a face is no more than the sum of its parts, theindividual features. Recent attempts using parameterizedfeature models and multiscale matching look morepromising, but still face severe problems before they aregenerally applicable. Current connectionist approachestend to hide much of the pertinent information in theweights that makes it difficult to modify and evalua:tepanrts of the approach.

The eigenface approach to face recognition waLs mo-tivated by information theory, leading to the idea ofbasing face recognition on a small set of image featuresthat best approximates the set of known face images.without requiring that they correspond to our intuitivenotions of facial parts and features. Although it is not anelegant solution to the general recognition problem. the

eigenface approach does provide a practical solution thatis well fitted to the problem of face recognit.ion. It is fast.relatively simple, and has been shown to work well in aconstrained environment. It can also be implementedusing modules of connectionist or neural networks.

It is important to note that many applications of facerecognition do not require perfect identification, al-though most require a low false-positive rate. In search-ing a large databa:se of faces, for example, it may bepreferable to flind a small set of likely matches to presentto the user. For applications such as security systems orhuman-computer interactio6rn. the sy'stem will normallybe able to "view" the subject for a few seconds or min-utes, and thus will have a number of chances to recognizethe person, Our experiments show that the eigenfacetechnique can be made to perform at very high accuracy,although with a substantial "unknown" rejection rate, andthus is potentially well suited to these applications.

We are currently investigating in more detail the issuesof robustness to changes in lighting, head size, and headorientation, automatically learning new face;. incorpo-

84 fournal of Cognitiie Neuroscience

Output layer Qr

Iui

Hidden layerf2

Input layer )

Volume 3, VNumber I

Ui

II

Figure 12. Collection of networks to implement computation of the pattern vector, projection into face space. distance from face spacemeasure, and identilication.

Figure 13. (a) I':.lnillyv occluded face image and (b) its reconslrucion using the 'igeinlices.

T rk anl Pentland . 85

rating a limited number of characteristic views for eachindividual, and the tradeoffs between the number ofpeople the system needs to recognize and- the numberof eigenfaces necessary for unambiguous classification.In addition to recognizing faces, we are also beginningefforts to use eigenface analysis to determine the genderof the subject and to interpret facial expressions, twoimportint face processing problems that complement thetask of face recognition.

REFERENCES

Bledsoe, W. W. (1966a). The model method in facial recogni.-tion. Panoramic Research Inc., Palo Alto, CA. Rep. PRI:15,August.

Bledsoe, W W. (1966b). Man-machine facial recognition. Pan-oramic Research Inc., Palo Alto, CA, Rep. PRI:22, August.

Bruce, V. (1988). Recognising faces. .'H-illsdale, NJ: Erlbaum.Bruce, C. J., Desimone, R, & Gross, C. G. (1981).Journal of

Neuropl.siolog; 46, 369-384.Burt, P. (1988a). Algorithms and architectures for smart sen-

sing. Proceedings of the Image Understanding Workshop,April..

Burt. P. (1988b). Smart sensing within a Pramid Vision Mvla-chine. Proceedings of IEEE, 76(8), 139-153.

Cannon, S. R., Jones, G. W., Campbell, R., & Morgan, N.W. (1986). A computer vision system for identification of indi-viduals. Proceedings of IECON, 1.

Carey. S.. & Diamond, R. (1977): From piecemeal to configu-rational representation of faces. Science, 195, 312-313.

Craw, Ellis, & Lishman (1987). Automatic extraction of facefeatures. Pattern Recognition Letters, 5, 183-187.

Davies, Ellis, & Shepherd (Eds.). (1981). Perceiving and re-mernbering fjtaces. London: Academic Press.

Desimone, R., Albright, T. D., Grass, C. G.. & Bruce, C. J.(1984). Stimulus-selective properties of inf'rir tempoil. ;.neurons in the macaque. Neuroscience. 4, 2051-2068.

Fischler. MNI. A., & Elschlager, R. A. (1973). The representationand matching of pictorial structures. IEEE Transactions onComputers, c-22(1).

Fleming, M., & Cottrell. G. (1990). Categorization of faces us-ing unsupervised feature extraction. Proceedings of IJCNVN-90,2.

Goldstein, Harmon, & Lesk (1971). Identification of humanfaces. Proceedings IEEE, 59, 748.

Harmon, L. D. (1971). Some aspects of recognition of humanfaces. In O. J. Grusser & R. Klinke (Eds.), Pattern recogni-.

tion iri biological and technical ystem. Berlin: Springer-Verlag.

Hay, D. C., & Young, .k W. (1982). The human face. in A. W.Ellis (Ed.), Normality and patibology in cognitive functions.London: Academic Press.

Kanade, T. (1973). Picture processing system by computercomplex and recognition of human faces. Dept. of Informa-tion Science. Kyoto University.

Kaya, Y., &Kobavashi, K. (1972). A basic study on human facerecognition. In S. Watanabe (Ed.), Frontiers of pattern recognition. New York: Academic Press.

Kirby, MN., & Sirovich. L (1990). Application of the Karhuneri-Loeve procedure for the characterization of human faces.IEEE Transactions on Pattern Analysiv anti Machine Intelli.gence, 12(1).

Kohonen, T. (1989). Selforganization and associative memor Berlin: Springer-Verlag.

Kohonen, T., & .Lehtio, P. (1981). Storage and processing ofinformation in distributed associative memotry systems. InG. E. Hinton & J. A. Anderson (Eds.), Parallel models ofassociative mnemor: Hillsdale, NJ: Erlbaum, pp. 105-143.

Marr, D. (1982). Vision. San Francisco: W. H-. Freeman.Midorikawa, H. (1988). The face pattern identification by

back-propagation learning procedure. Abstracts of the FirstAnnrmual INNS Aleeting, Boston, p. 515.

O'Toole, Millward. & Anderson (1988). A physical systemn ap-proach to recognition memory for spatially transformedfaces. Neural Networks, ,1. 179-199.

Perrett, D. L., Mistlin, A. J., & Chitrv, A. J. (1987). Visual newurones responsive to faces. TINS, 10(9), 358-364.

Perrett, Rolls, & Caan (1982). Visual neurones responsive tofaces in the monkey temporal conrtex. hLxperintental BrainResearch, 47, 329-342.

Rolls. E. T., B3aylis, G. C.. Hasselmo, M. E., & Nalwa, V. (1989).The effectof learning on the face selective responses ofneurons in the cortex in the. superior temporal sulcus ofthe monkey. Experinmental Brain Research, 76, 153-164;

Sirovich, L., & Kirby, M. (1987). Low-dimensional procedureS7,fOr,.the characterization of human faces. Journal of the Op-

tical Socit of jAmerica A, 4(3). 519-524.Stonham. T. J. (1986). Practical face recognitioni and verifica-

tion with WISARD. In H. Ellis. M. Jeeves, F..Newcombe, &A. Young (Eds.), Aspects of face proccsirit. Dordrecht: Mar-tinus Nijhoff.

Wong, K., Law. H., & Tsang, P. (1989). A system for recognis-ing human faces. Proceedinrgs of ICASSP, May, 1638-1641

Yuille, A. L, Cohen, D. S., & Hallinan, P. W. (1989). Featureextraction from faces using deformable templates. Proceed-ings of CVPR. San Diego, CA, June.

86 Journal of Cognitive Neuroscience Volume 3, tNumber 1

THIS PAGE BLANK (usPTO)

This Page is Inserted by IFW Indexing and Scanning

Operations and is not part of the Official Record




Q BLACK BORDERS

Q IMAGE CUT OFF AT TOP, BOTTOM OR SIDES

ADED TEXT OR DRAWING

U BLURRED OR ILLEGIBLE TEXT OR DRAWING

I SKEWED/SLANTED IMAGES//

Q COLOR OR BLACKAND WHITE PHOTOGRAPHS

U .GRAY SCALE DOCUMENTS.

O LINES OR MARKS ON ORIGINAL DOCUMENT

O REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY

U OTHER:


THIS PAGE BLANK (usPo)

-FEB22. 2007 2:34PM MOFO 28TH FL

MORRISON IOERSTER

NO. 247 P. 1425 MARKET STREETSAN FRACICOCALIFORNIA 94]05-2488

TBLEPHONE. 415.266.7000FACSM=- 415.2A875Z

W W .MOFO.COM

MOA~t3OI 4 a OBaRSTR LLP

NBW YORK, $4 14FANCISCO,LOS APIGULBS. HALO ALTO,SAN 1;I200. WAAWINOTON, D.C.

'DSNVBR, 1NQRTI1UN VIRGIIA,ORANGE COUNTY, IACR.AUBNTO,WALNUT C1PUIC, CENTURV CITY

TOKCYO, LONDlON, BRIJTNO." II iI q,

1VJP'.FItowe 3CEVE

^ ^" """"' iHAMOHAT, nOd KONG,SINGAPOAS, IkUSSILS DCIE

i'.ro m Ax C (ENTER

NAME: FACSIMILE: TELEPHONE:

Centralized Facsimile Center 571-273-8300ooUS Patent and Trademark Office

FROM: Peter J. Ylm DATE: February 22, 2007

Number of pages 4 Originals Will NOT Followwith cover page:

Preparer of this slip has confirmed that facsimile number given is correct: 656vlhl

Comments:

Attorney Docket: 577832000200Art Unit: 2621Examiner: Not assigned yetApplication No. 11/231,353Filed: September 19,2005Title: INCREMENTAL DATA FUSION AND DECISION MAKING SYSTEM ANDASSOCIATED METHODInventor: Ken P. CHENG, et al.

Documents Enclosed:I. Request To Withdraw As Attorney Or Agent And Change Of CorrespondenceAddress, in triplicate, 3 pages

To ensure compliance with requirements imposed by the United States Internal RevenueService, Morrison & Foerster LLP informs you that, if any advice concerning one or moreU.S. Federal tax issues is contained in this facsimile (incliuding any attachments), suchadvice is not intended or written to be used, and cannot be used, for the purpose of (i)avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing orrecommending to another party any transaction or matter addressed herein.

+++++******************************************

CAUTION -CONFIDENTIAL

This facsimile contains confidential information that may also be privileged. Unless youare the addressee (or authorized to receive for the addressee); you may not copy, use, ordistribute it. If you have received it in error, please advise Morrison & Foerster LLP

immediately by telephone or facsimile and return it pronmptly by mail.

SF-2273872

IF YOU DO NOT RECEIVE ALL OF THE PAGES, PLEASE CALLVICKI HENRY AT 415.268.6023 AS SOON AS POSSIBLE,

PAGE 114' RCVD AT 222200715:32:49 PM [Eastem Standard Time]' SVR:USPT0EFXRF2/12'DNIS:2738300' CSD:415 2687522' DURATION (mm.ss):0130

3 22 2007

TIB22. 2007 2:34PM MOFO 28TH FLRECEIVED

CENTRAL FAX CENTER

FEB 2 2 2007

PT01Sl83 (01.06)Approved for use through 12/31/2008. OMB 0551-4003

U.S3. Patent and mdemark 0ff00; U.S. DEPARTM~N OP COMMERCE

Under the Papework Redu lan Act of 199 no e ia r uimreid to eo co10 le.tonIn lnn rmatioa unees it cdsplays a vald OMB contro number.


Filing Date September 19, 2005REQUEST FOR WITHDRAWAL First Named inventor Ken P. CHENG

AS ATTORNEY OR AGENT......AND CHANGE OF Art Unit 2621

CORRESPONDENCE ADDRESS Examiner Name ' .Not Yet.Assigned

Attomey Docket Number 877832000200

To: Commissioner for PatentsP.O. Box 1450Alexandria. VA 22313-1450

Please withdraw me as attorney or agent for the above Identified ptenrt application, and

Q all the attomeyalagentra of record.

the attorneyslagents (with registration numbers) listed on the attached paper(s), or

the attomeys/agents associated with Customer Number 20872

NOTE: This box can only be checked when the power of attomrney of record In the application is to all thepractitioners associated with a customer number.

The reasons for this request are:The practitioners have been discharged by the client. The client has reqJested transfer to newcounsel.

CORRESPONDENCE ADDFESS

1. Q The correspondenoe address is NOT affected by thin withdrawal.

2. D Change the correspondence address and direct all future correspondence to:

] The address associated with Customer Number. "___

OR

Q irm or Stephen C. DurantInMnual Name Novak Druce + Quigg LLP

Address Spear Street Tower - 35th FloorOne Market Plaza

city San Francisco state CA .... 94105

CountryTelephone 41 8814 Email [email protected]

signatureName fetrJ. im Rbistratiion No. 44,417

Date February 1", 2007 Teephone No. (415) 268-6373NOTE Wit4hanwal is ercvl when appoved raher than wen 9c0oed.4.loss ere at least 30 days between approve a windraat and1he expiralian dare of a tho period for epnse orpassbe extenalon period, the reqesto withdrawI s notnnallydsappved

I hereby certify that this paper is being transmtted by la simie to the p nt and Tmder% Offi#arimlins no. (071) 2734300. on te dateshownbelow. .. __ __,,_( M ey

Dated:Pebrua.2007. .natum (Vici..nty)

sf-2273801

PAGE 214' RCVD AT 21222007 5:32:49PM [Eastern Standard Time)' SVR:USPTOEFXRF.212' DNIS:2738g300' CSID:415 2687522' DURATION (m-.ss):01.30

NO. 247 P. 2

--- --

RECEIVED-fE B:'22. 2007 2:35PM MOFO 28TH FL CENTRALFAXCENTER NO. 247 P. 3

FEB 2 2 2007

P'rolsatas tocalpproved for use through 12/31 200. OMB 0651-0D35

US. Patent and Tfdectmal Office; U.S. DEPARTMENT OF COMMERC

Urder Oe PaoeRp R0Uon Acts of 1995. no glasonse aqufred i IBesrad to a oledian of ationMileSS I disay a vald OMB ontrlnumber

App5cation Number 11/231,353

Filing Date September 19, 2005

AS ATTORNEY OR AGENT .Unit.2521

AND CHANGE OF Art Unit 221

S CORRESPONDENCE ADDRESS Examiner Name Not Yet Assigned..... "..... .... - - - - Attomey Docket Number 577832000200

To: Commissioner for PatentsP.O. Box 1450Alexandria, VA 22313-1450

Please withdraw me as attorney or agent for the above identified patentl applicatton, and

Sall the attorneys/agents of record.

the attornmays/agenmts (with registration numbers) lated on the attahed paper(s), or

nx the attomeystagents assadatad with Customer Number 20672

NOTE: This box can only be checked when the power of attorney o record in the application is to all thepraclitioners associated with a customer number.

The reasons forthis request areThe practitioners have been discharged by the client. The client has requested transfer to new

counsel.

CORRESPONDENCE ADD ESS

1. The correspondenoe address is NOT affected by this withdrawal.

2. ©' Change the correspondence address and d'uect al future correspondence to:

The address associated with Customer Number 1OR

Firme Stephen C. DurantlnMdiU NaTmre Novak Druce + Quigg LLP


city San Francisco stale CA Zip 94105

Country

Telephone 41 38T Emal [email protected]

Name r J. Vim Registration No. 44,417

Date February ,2007 Telephone No. (415) 268-6373NOTE Wmhdwal i s ffocve o when appmroved raerhan when received. Unle sMere a at lcst a 30 0y between appoveo wrdr.thawer andm caphUon datef a erOme io fad forepo o possbicexteatorn perdnd, the equest &b wihuwis salbe dryovwd

I herwbycerarythat hs paper isbeing trenspitted by fcsiileto te ent and Tad k Oftce. facite no. (571)273.800, on the dateshowlntblouw. •".Dated: Feb Signature: (Vidd Moini

sf-2273801

PAGE 314 RCVD AT 22212007 5:32:49 PM [Eastemrn Standard Time]J SVR:USPTO.EFXRF2l12' DNIS:2?3300' CSID:415 2687522 DURATION (mm-.ss):01.30

---- ~~~-

- TE 22. 2007 2:35PM MOFO 28TH FL NO. 247 P. 4

PTOISB~83 (01406)Approved ?r use through 12/30LZOO5. OMB S05-0035

U.. Paltent and Traflma k ica; U.S. DEPARTMENT OF COMMERCEUnderthe Pe aonak edcR'il Atd of 1995. no pemai som M qredto I~sIondt3s lcin iomatla nunlsst des l a vid OMB control number.

Application Number 111231,353

Filing Date September 19, 2005REQUEST FOR WITHDRAWAL First Named Inventor KeniP. CHENG

A8 ATTORNEY OR AGENTAND CHANGE OF Art Unit 2621

CORRESPONDENCE ADDRESS Examiner Name Not Yet Assigned. . ..

Attomey Docket Number 577832000200

MOME WIthdre a eis effe mwhen eprved rathrV Yan r AotoUd IMOfs Macre ae : t baW0 da ; O bc Moen approval 1 wlfd uwel andfits os~mfon chic ofafine peioforbr empose orpoar'a b e io d ft enrueab wVlfjWris no'nuaR~daapwvVWl.

I hereby oaftty that this paperis bc ng VuwmfsVd by f to ft Prn and TO Office. facoanle To.C(71) 8734300.aono e

Dated: 8ipnaturs; f (VKM cidHomsy)

sf-2273801

PAGE 414' RCVD AT 2212001 5:32:49 PM [Easternm Standard Time]' SVR:USPTO0EFXRF.212' DNIS:2138300' CSID:415 2587522'DURATION (mm.ss):01.30

To: Commissioner for PatentsP.O. 8ox 1450Alexandria, VA 22313-1450

Please withdraw me as attorney or agent forthe above identified patent application, and

[ all the attomeys/agents of record.

[the attorneys/agents (with registration numbers) listed on the attached paper(s), or

the attomrneyalegents associated with Cuslomer Number 20872

NOTE: This box can only be checked when the power of attorney of record in the application Is to all thepractitioners associated with a customrer number.

The reasons for this request are:The practitioners have been discharged by the dent. The client as requested transfer to newcounsel.

CORRESPONDENCE ADDSS

1. .The correspondence address is NOT affected by this withdraweL

2. Change the correspondence address end direct all future cor fespondence tO:

n The addres associated with Customer Number'. ,1OR

Fae or Stephen C. DurantlndivuaI Name Novak Druoe + Quig LLP


city San Francisco State CA Zip 94105

Country

Telennone 4_1 9814 Emailn [email protected] eterJ im Registration No. 44,417

Date February , 2007 Telephone No. (415) 268-6373

i i

IL•

, 1 .......

1

Fme~

Electronic Acknowledgement Receipt

EFS ID: 1754177

Application Number: 11231353

International Application Number:


Title of Invention: Adaptive multi-modal integrated biometric identification detection andsurveillance systems

First Named Inventor/Applicant Name: Ken Prayoon Cheng

Customer Number: 20872

Filer: Stephen Christopher Durant/Marcia Shea

Filer Authorized By: Stephen Christopher Durant

Attorney Docket Number: 577832000200

Receipt Date: 08-MAY-2007

Filing Date: 19-SEP-2005

Time Stamp: 12:59:25

Application Type: Utility

Payment information:

Submitted with Payment no

File Listing:DoumntMulti .Pages

Document Document Description File Name File Size(Bytes) Multi PagesNumber Part /.zip (if appl.)

1 Power of Attorney POA.pdf 45264 no1

Warnings:

Information:

2 Assignee showing of ownership per Statement.pdf 41914 no 137 CFR 3.73(b).

Warnings:

Information:

Total Files Size (in bytes) 87178

This Acknowledgement Receipt evidences receipt on the noted date by the USPTO of the indicated documents,characterized by the applicant, and including page counts, where applicable. It serves as evidence of receiptsimilar to a Post Card, as described in MPEP 503.

New Applications Under 35 U.S.C. 111If a new application is being filed and the application includes the necessary components for a filing date (see37 CFR 1.53(b)-(d) and MPEP 506), a Filing Receipt (37 CFR 1.54) will be issued in due course and the dateshown on this Acknowledgement Receipt will establish the filing date of the application.

National Stage of an International Application under 35 U.S.C. 371If a timely submission to enter the national stage of an international application is compliant with the conditionsof 35 U.S.C. 371 and other applicable requirements a Form PCT/DO/EO/903 indicating acceptance of theapplication as a national stage submission under 35 U.S.C. 371 will be issued in addition to the Filing Receipt,in due course.

New International Application Filed with the USPTO as a Receiving OfficeIf a new international application is being filed and the international application includes the necessarycomponents for an international filing date (see PCT Article 11 and MPEP 1810), a Notification of theInternational Application Number and of the International Filing Date (Form PCT/RO/105) will be issued in duecourse, subject to prescriptions concerning national security, and the date shown on this AcknowledgementReceipt will establish the international filing date of the application.

POWER OF ATTORNEY TO PROSECUTE APPLICATIONSBEFORE THE USPTO

I hereby revoke all previous powers of attorney given in the application identified in theattached statement under 37 CFR 3.73(b).

I hereby appoint:

Practitioners associated with the Customer Number: 65761

as attorney(s) or agent(s) to represent the undersigned before the United States Patentand Trademark Office (USPTO) in connection with any and all patent applicationsassigned only to the undersigned according to the USPTO assignment records orassignment documents attached to this form in accordance with 37 CFR 3.73(b).

Please change the correspondence address for the application identified in the attachedstatement under 37 CFR 3.73(b) to:

The address associated with Customer Number: 65761

Assignee Name and Address:

Proximex Corporation440 N. Wolfe RoadSunnyvale, California 94085

A copy of this form, together with a statement under 37 CFR 3.73(b) (Form PTO/SB/96 orequivalent) is required to be filed in each application in which this form is used. Thestatement under 37 CFR 3.73(b) may be completed by one of the practitioners appointedin this form If the appointed practitioner is authorized to act on behalf of the assignee,and must identify the application in which this Power of Attorney is to be filed.

SIGNATURE of Assignee of Record

The individual whose signature and title is supplied below is authorized to act on behalfof the assignee.

Ken P. Cheng DateTitle: President

Telephone: o .2. - 5O


U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCEUnder the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number.


Applicant/Patent Owner: Ken Prayoon CHENG et al.

Application No./Patent No.: 11/231,353 Filed/Issue Date: September 19, 2005

Entitled: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICIATION DETECTION AND SURVEILLANCE SYSTEMS

PrnximPx Corporatinn , a .rpnratinn(Name of Assignee) (Type of Assignee, e.g., corporation, partnership, university, government agency, etc.)

states that it is:1. the assignee of the entire right, title, and interest; or

2. O an assignee of less than the entire right, title and interest

(The extent (by percentage) of its ownership interest is %)


A.Q An assignment from the inventor(s) of the patent application/patent identified above. The assignment was recordedin the United States Patent and Trademark Office at Reel 017473 , Frame 0506 , or for which a copythereof is attached.

ORB.0- A chain of title from the inventor(s), of the patent application/patent identified above, to the current assignee as follows:

1. From: To:


2. From: . To:The document was recorded in the United States Patent and Trademark Office atReel , Frame _, or for which a copy thereof is attached.

3. From: To:The document was recorded in the United States Patent and Trademark Office atReel , Frame , or for which a copy thereof is attached.

Ol Additional documents in the chain of title are listed on a supplemental sheet.

Q As required by 37 CFR 3.73(b)(1)(i), the documentary evidence of the chain of title from the original owner to theassignee was, or concurrently is being, submitted for recordation pursuant to 37 CFR 3.11.

[NOTE: A separate copy (i.e., a true copy of the original assignment document(s)) must be submitted to AssignmentDivision in accordance with 37 CFR Part 3, to record the assignment in the records of the USPTO. See MPEP302.08]

The undersigned (whose title is supplied below) is authorized to act on behalf of the assignee.

/Stephen C. Durant/ Mavy 8. 2007

Signature Date

Steohen C. Durant 415.814 6163

Printed or Typed Name Telephone Number

Attorney for AssigneeTitle

This collection of information is required by 37 CFR 3.73(b). The information is required to obtain or retain a benefit by the public which is to file (and by theUSPTO to process) an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.11 and 1.14. This collection is estimated to take 12 minutes tocomplete, including gathering, preparing, and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Anycomments on the amount of time you require to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer,U.S. Patent and Trademark Office, U.S. Department of Commerce, P.O. Box 1450, Alexandria, VA 22313-1450. DO NOT SEND FEES OR COMPLETEDFORMS TO THIS ADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450, Alexandria, VA 22313-1450.

If you need assistance in completing the form, call 1-800-PTO-9199 and select option 2.

UNITED STATES PATENT AND 'IThADEMARK OFFICEUNITED STATES DEPARTMENT OP COMMERCEUnited States Patent and Trademark OmoeAddrem:COMMISSIONER FOR PATENTS

P.O. Box 1450Alexandria, Vuirgia 2233-1450wwws.upto.go

APPLICATION NUMBER FILING OR 371 (c) DATE KFIRST NAMED APPLICANT ATTY. DOCKET NO/TLE



*C00000000023958100"

65761NOVAK DRUCE & QUIGG, LLP(SAN FRANCISCO)1000 LOUISIANA STREETFIFTY-THIRD FLOORHOUSTON, TX 77002


NOTICE OF ACCEPTANCE OF POWER OF ATTORNEY

This is in response to the Power of Attorney filed 05/08/2007.

The Power of Attorney in this application is accepted. Correspondence in this application will be mailed to theabove address as provided by 37 CFR 1.33.

Office of Initial Patent Examination (571) 272-4000, or 1-800-PTO-9199OFFICE COPY

--

UNITED STATES PATENT AND TADEMARK OFFIGEUNITED STATES DEPARTMENT OF COMMERCEUnited States Patent and Trademark Office'Address:COMMISSIONER FOR PATENTS

P.O. Box 1450Alexandria. VugBiia 22313.1450ww.uwptogov

APPLICATION NUMBER FILING OR 371 (c) DATE FIRST NAMED APPLICANT ATTY. DOCKET NO./TITLE




II00EII3I80IHIG UOH*HIIIIllH1111 llIlIl*0C000000023958091"


NOTICE REGARDING CHANGE OF POWER OF ATTORNEY


* The Power of Attorney to you in this application has been revoked by the assignee who has intervened asprovided by 37 CFR 3.71. Future correspondence will be mailed to the new address of record(37 CFR 1.33).

Officeofnitia Patent Examination (571) 272-4000, or 1-800-PTO-9199OFFICE COPY.

N 0

-- ~---- ------ I

--

UNITED STATES PATENT AND TRADEMARK OFFICEUNITED STATES DEPARTMENT OF COMMERCEUnited States Patent and Trademark OfficeAddreou: COMMISSIONER FOR PATENTS

P.O. Box 1450Alxandria, Vu ia 22313-1450

ww~rto~

APPLICATION NUMBER FILING OR 371 (c) DATE FIRST NAMED APPLICANT ATTY. DOCKET NO/TITLE



*00C000000023958100*

65761NOVAK DRUCE & QUIGG, LLP(SAN FRANCISCO)1000 LOUISIANA STREETFIFTY-THIRD FLOORHOUSTON, TX 77002



This is in response to the Power of Attorney filed 05108/12007.

The Power of Attorney in this application is accepted. Correspondence in this application will be mailed to theabove address as provided by 37 CFR 1.33.

Office of Initial Patent Examination (571) 272-4000, or 1-800-PTO-9199ATTORNEY/APPLICANT COPY

Page 1 of 1

I7-I

_ __ .....

UNITED STATES PATENT AND M ADEMARK OFFIGEUNITED STATES DEPARTMENT OF COMMERCEUnited States Patent and Trademark OMiceAddma: COMMISSIONER FOR PATENTS

P.O. B= 1450AIxaiam Vuni 22313-1450

APPLICATION NUMBER FILING OR 371 (c) DATE FIRST NAMED APPLICANT ATTY. DOCKET NO.TLE




0OC00000023958091*




* The Power of Attorney to you in this application has been revoked by the assignee who has intervened asprovided by 37 CFR 3.71. Future correspondence will be mailed to the new address of record(37 CFR 1.33).

Office of Initial Patent Examination (571) 27-4000, or 1-800-PTO-9199FORMER ATTORNEY/AGENT COPY

Page 1 of I

--

UNITED STATES PATENT AND TRADEMARK OFFICE

Commissioner for PatentsUnited States Patent and Trademark Office

P.O. Box 1450Alexandria, VA 22313-1450

www.uspto.gov

MORRISON & FOERSTER LLP.425 MARKET STREETSAN FRANCISCO CA 94105-2482

In re Application of

CHENG, Ken P. et al.Application No. 11/231,353Filed: September 19, 2005Attorney Docket No. 577832000200

This is a decision on the Request to§ 1.36(b), filed February 22, 2007.

COPY MAILED

MAY 2 32007

OFFICE OF PETITIONS

S DECISION ON PETITIONS TO WITHDRAWS FROM RECORD

Withdraw as attorney or agent of record under 37 C.F.R.

The request is NOT APPROVED as moot.

A review of the file record indicates that the power of attorney to Morrison &been revoked by the assignee of the patent application on May 08, 2007.request to withdraw under 37 C.F.R. § 1.36(b) is moot.

Foerster LLP hasAccordingly, the

All future communications from the Office will continue to be directed to the below-listedaddress until otherwise notified by applicant.

Telephone inquires concerning this decision should be directed to Michelle R. Eason at 571-272-4231.

Terri WilliamsPetitions ExaminerOffice of Petitions

cc: NOVAK DRUCE & QUIGG, LLP(SAN FRANCISCO)1000 LOUISIANA STREETFIFTY-THIRD FLOORHOUSTON, TX 77002


EFS ID: 6286123




Title of Invention: Adaptive multi-modal integrated biometric identification detection andTitle of Invention: suvilnc. ytm

surveillance systems



Filer: Tracy Wesley Druce/Amy Drury

Filer Authorized By: Tracy Wesley Druce

Attorney Docket Number: 8193.003.NPUS00

Receipt Date: . 19-OCT-2009



Application Type: Utility under 35 USC 111(a)


Submitted with Payment no

File Listing:

Document File Size(Bytes)/ Multi Pages

Number Document Description File Name Message Digest Part /.zip (if appl.)

170989Petition to withdraw attorney or agent 8193_003_NPUS00_Withdrawa no794

1 (5883) I.pdf no4b8f43036c2c2c615b79cc36e48994630a6

99fd

Warnings:

Information:


This Acknowledgement Receipt evidences receipt on the noted date by the USPTO of the indicated documents,characterized by the applicant, and including page counts, where applicable. It serves as evidence of receipt similar to aPost Card, as described in MPEP 503.

New Applications Under 35 U.S.C. 111If a new application is being filed and the application includes the necessary components for a filing date (see 37 CFR1.53(b)-(d) and MPEP 506), a Filing Receipt (37 CFR 1.54) will be issued in due course and the date shown on thisAcknowledgement Receipt will establish the filing date of the application.

National Stageof an International Application under 35 U.S.C. 371If a timely submission to enter the national stage of an international application is compliant with the conditions of 35U.S.C. 371 and other applicable requirements a Form PCT/DO/EO/903 indicating acceptance of the application as anational stage submission under 35 U.S.C. 371 will be issued in addition to the Filing Receipt, in due course.

New International Application Filed with the USPTO as a Receiving OfficeIf a new international application is being filed and the international application includes the necessary components foran international filing date (see PCT Article 11 and MPEP 1810), a Notification of the International Application Numberand of the International Filing Date (Form PCT/RO/105) will be issued in due course, subject to prescriptions concerningnational security, and the date shown on this Acknowledgement Receipt will establish the international filing date ofthe application.

Doc Code: PET.POA.WDRWDocument Description: Petition to withdraw attorney or agent (SB83) PTO/SB/83 (11-08)

Approved for use through 11/30/2011. OMB 0651-0035U.S. Patentand Trademark Office, U.S. DEPARTMENT OF COMMERCE

Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number.

Application Number 11/231,353REQUEST FOR WITHDRAWAL Filing Date 09-19-2005

AS ATTORNEY OR AGENT First Named Inventor Ken Pravoon ChenqclAND CHANGE OF Art Unit 2621

CORRESPONDENCE ADDRESS Examiner Name DASTOURI, MEHRDADAttorney Docket Number 8193.003.NPUS00 ,

To: Commissioner for PatentsP.O. Box 1450Alexandria, VA 22313-1450

Please withdraw me as attorney or agent for the above identified patent application, and

Sall the practitioners of record;

Q the practitioners (with registration numbers) of record listed on the attached paper(s); or

the practitioners of record associated with Customer Number: 65761

NOTE: The immediately preceding box should only be marked when the practitioners were appointed using the listedCustomer Number.

The reason(s) for this request are those described in 37 CFR :

D 10.40(b)(1) Q 10.40(b)(2) 10.40(b)(3) 10.40(b)(4)

D 10.40(c)(1)(i) 1 10.40(c)(1)(ii)Q 10.40(c)(1)(iii) jj 10.40(c)(1)(iv)

- 10.40(c)(1)(v) I 10.40(c)(1)(vi) E 10.40(c)(2) 10.40(c)(3

-1 10.40(c)(4) 11 10.40(c)(5) D 10.40(c)(6)Please explain below:

CertificationsCheck each box below that is factually correct. WARNING: If a box is left unchecked, the request will likely notbe approved.

1. 1I/We have given reasonable notice to the client, prior to the expiration of the response period, that thepractitioner(s) intend to withdraw from employment.

2. T I/We have delivered to the client or a duly authorized representative of the client all papers and property(including funds) to which the client is entitled.

3. I/We have notified the client of any responses that may be due and the time frame within which theclient must respond.Please provide an explanation, if necessary:

As of August 14, 2008, the client has directed us to cease representation and transfer all files to Stephen C.Durant, who has not yet filed a new Power of Attorney or Change of Address.

[Page 1 of 2]This collection of information is required by 37 CFR 1.36. The information is required to obtain or retain a benefit by the public which is to file (and by the USPTOto process) an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.11 and 1.14. This collection is estimated to take 12 minutes to complete,including gathering, preparing, and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Any commentson the amount of time you require to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S. Patentand Trademark Office, U.S. Department of Commerce, P.O. Box 1450, Alexandria, VA 22313-1450. DO NOT SEND FEES OR COMPLETED FORMS TO THISADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450, Alexandria, VA 22313-1450.



U.S. Patent and Trademark Office, U.S. DEPARTMENT OF COMMERCEUnder the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number.

REQUEST FOR WITHDRAWALAS ATTORNEY OR AGENT

AND CHANGE OF CORRESPONDENCE ADDRESSComplete the following section only when the correspondence address will change. Changes of address will only be accepted to aninventor or an assignee that has properly made itself of record pursuant to 37 CFR 3.71.

Change the correspondence address and direct all future correspondence to:

A. D The address of the inventor or assignee associated with Customer Number:

ORS Inventor or PROXIMEXCORPORATION

B. Assignee name

Address 440 N. Wolfe Road

City Sunnyvale State CA Zip 94085 Country US

Telephone (408) 216-5190 Email

I am authorized to sign on behalf of myself and all withdrawing practitioners.

Signature /Tracy W. Druce/

Name Tracy W. Druce Registration No. 35,493

Address NOVAK, DRUCE & QUIGG LLP, 1000 LOUISIANA STREET, FIFTY-THIRD FLOOR

City Houston State TX Zip 77002 Country US

Date October 19, 2009 Telephone No. 713-571-3400

NOTE: Withdrawal is effective when approved rather than when received.

[Page 2 of 2]This collection of information is required by 37 CFR 1.36. The information is required to obtain or retain a benefit by the public which is to file (and by the USPTOto process) an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.11 and 1.14. This collection is estimated to take 12 minutes to complete,including gathering, preparing, and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Any commentson the amount of time you require to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S. Patentand Trademark Office, U.S. Department of Commerce, P.O. Box 1450, Alexandria, VA 22313-1450. DO NOT SEND FEES OR COMPLETED FORMS TO THISADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450, Alexandria, VA 22313-1450.


of 2

Todd Leone

From: Tracy Druce

Sent: Thursday, August 14, 2008 2:25 PM

To: Ken P. Cheng

Cc: Matt Todd; Durant, Stephen C.; Jim Barth; Raj Tatini; Todd Leone; Vincent M. DeLuca; GregoryNovak

Subject: RE: Proximex file transfer request to Durant at Duane Morris

Dear Mr. Cheng:

These files will go today or tomorrow.

Thank you,

Tracy

Tracy Wesley DruceNovak Druce & Quigg LLPWells Fargo Plaza, 53rd Floor1000 Louisiana StreetHouston, TX 77002

713.571.3400 (Telephone)713.456.2836 (Fax)877.529.7894 (US Toll Free)[email protected] our website at www.novakdruce.com

Contidentiality Notice: This email ard any attachments contain information from the law firm of Novak DrucE'c, I..P,which may be confidential and/or privileged. The information is intended to be for the use of the individual or entitynamed on this email. If you are not the intended recipient, be aware that rany disclosure, copying, distribution oruse of the contents of this email is prohibited. If you receive this email in error, please notify us by reply emailimmediately so that we can arrange for the retrieval of the original documents at no cost to you.

From: Ken P. Cheng [mailto:[email protected]]Sent: Thursday, August 14, 2008 2:14 PMTo: Tracy DruceCc: Matt Todd; Durant, Stephen C.; Jim Barth; Raj TatiniSubject: Proximex file transfer request to Durant at Duane Morris

Dear Mr. Druce:

This is to inform you that Proximex Corporation ("Proximex") wishes to transfer responsibilityfor patent matters that you are handling on behalf of Proximex to Stephen C. Durant. We direct NovakDruce & Quigg LLP to discontinue its representation of Proximex and immediately transfer the patentfiles and any other materials including, but not limited to, those described below, to:

Stephen C. Durant

8/14/2008

of 2

Duane Morris LLP,2000 Spear Street TowerOne Market PlazaSan Francisco, California 94105-1104Specifically, we would like the transfer to include the following Proximex files and materials identifiedbelow:

1. All files and other property of Proximex including, without limitation, all patent applicationfiles, attorney work files, collections of prior art, correspondence files and other files and materials;

2. A printout, and an electronic file, containing all items identified in the Report docketed inNovak Druce & Quigg LLP's docket system for Proximex including all actions due or that arescheduled due for a future date;

3. A CD-Rom or Flash drive containing an MSWord version of the Proximex patent applications,drawings and any other items as filed for the patent application files identified in the Report; and

4. Any other Proximex files or documentation subsequently identified and requested by Mr.Durant or any other member of Duane Morris LLP pertaining to items identified in numbered paragraph1 above.

'Please immediately transfer the requested material as there may be many urgent issues pendingand under consideration. When the materials are ready for delivery, please call Duane Morris LLP at415-947-3023 to arrange for a courier. I appreciate your assistance. If you have any questions, pleasecontact me at your convenience.

Very truly yours,

PROXIMEX CORPORATION

Ken P. ChengFounder

cc: Stephen C. Durant, Esq.Matt Todd, Esq.

Ken Prayoon ChengCTO

Surveillint'"Discover. Connect. Resolve.

Proximex Corporation Wok: 408.524.1510

440 N. Wolfe Road Moi- 408.828.1838

Sunnyvale, CA 94085 : 408.524.1528

www.proximex.com , mah: Ken.Cheng)Proximex.com

8/14/2008




www.uspto.gov

i

SAN FRANCISCO OFFICE OFNOVAK, DRUCE & QUIGG LLP1000 LOUISIANA STREETFIFTY-THIRD FLOOR COPY MAILEDHOUSTON TX 77002

DEC 0 3 2009

OFFICE OF PETITIONS

In re Application of

CHENG, Ken Prayoon et al.Application No. 11/231,353 : DECISION ON PETITIONFiled: September 19, 2005 : TO WITHDRAWAttorney Docket No. 8193.003.NPUS00 FROM RECORD

This is a decision on the Request to Withdraw as attorney or agent of record under 37 C.F.R. §1.36(b), filed October 19, 2009.

The request is APPROVED. i

A grantable request to withdraw as. attorney/agent of record must be signed by everyattorney/agent seeking to withdraw or; contain a clear indication that one attorney is signing onbehalf of another/others. The Office requires the practitioner(s) requesting withdrawal to certifythat he, she, or they have: (1) given reasonable notice to the client, prior to the expiration of theresponse period, that the practitioner(s)1intends to withdraw from employment; (2) delivered to theclient or a duly authorized representative of the client all papers and property (including funds) towhich the client is entitled; and (3) notified the client of any responses that may be due and thetime frame within which the client must respond, pursuant 37 CFR 10.40(c).

The request was signed by Tracy W. Diruce on behalf of all attorneys of record who are associatedwith customer No. 65761. All attorneys/agents associated with have been withdrawn. Applicant isreminded that there is no attorney of record at this time.

I

The request to change the correspondence address of record is not acceptable as the requestedcorrespondence address is not that of: (1) the first named signing inventor; or (2) an interveningassignee of the entire interest under 37 C.F.R 3.71. All future communications from the Officewill be directed to the first named signing inventor.

i

,.

Application No. 11/231,353

In order to request or take action in a patent matter, the assignee must establish its ownership of thepatent to the satisfaction of the Director. In this regard, a Statement under 37 CFR 3.73(b) musthave either: (i) documentary evidence of a chain of title from the original owner to the assignee(e.g., copy of an executed assignment), and a statement affirming that the documentary evidence ofthe chain of title from the original owner to the assignee was or concurrently is being submitted forrecordation pursuant to § 3.11; or (ii) a statement specifying where documentary evidence of achain of title from the original owner to the assignee is recorded in the assignment records of theOffice (e.g., reel and frame number).

There are no outstanding office actions at this time.

Telephone inquiries concerning this decision should be directed to the undersigned at 571-272-4231.

Michelle R. EasonParalegal SpecialistOffice of Petitions

cc: KEN P. CHENG20691 REID LANESARATOGA, CA 95070-5325

cc: PROXIMEX CORPORATION440 N. WOLFE ROADSUNNYVALE, CA 94085

Page 2

UNITED STATES PATENT AND TRADEMARK OFFICEUNITED STATF. DEPARTMENT OF COMMERCEUnited States Patent and Trademark OfficeAddress: COMMISSIONER FOR PATENTS

P.O. Box 1450Alexanhdria, Vria 22313-1450www.uspto.gov

APPLICATION NUMBER FILING OR 371(C) DATE

11/231,353 09/19/2005

65761SAN FRANCISCO OFFICE OFNOVAK, DRUCE & QUIGG LLP1000 LOUISIANA STREETFIFTY-THIRD FLOORHOUSTON, TX 77002

FIRST NAMED APPLICANT ATTY. DOCKET NO.TITLE

Ken Prayoon Cheng 8193.003.NPUS00CONFIRMATION NO. 4531

POWER OF ATTORNEY NOTICE




* The withdrawal as attorney in this application has been accepted. Future correspondence will be mailed to thenew address of record. 37 CFR 1.33.

/mreason/

Office of Data Management, Application Assistance Unit (571) 272-4000, or (571) 272-4200, or 1-888-786-0101

page 1 of 1

UNITED STATES PATENT AND TRADEMARK OFFICEUNITED STATES DEPARTMENT OF COMMERCEUnited States Patent and Trademark OfficeAddress: COMMISSIONER FOR PATENTS

P.O. Box 1450Alexandria, Virginia 22313-1450www.uspto.gov

APPLICATION NO. FILING DAT]

11/231,353 09/19/2005

'7590

E FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO.

Ken Prayoon Cheng 4531

/:)V .r u .u EXAMINERKEN P. CHENG20691 REID LANE GRANT II, JEROME

SARATOGA, CA 95070-5325ART UNIT PAPER NUMBER

2625

MAIL DATE DELIVERY MODE

05/13/2010 PAPER

Please find below and/or attached an Office communication concerning this application or proceeding.

The time period for reply, if any, is set in the attached communication.

PTOL-90A (Rev. 04/07)

]]

5 ci~ l

Application No. Applicant(s)

11/231,353 CHENG ET AL.

Office Action Summary Examiner Art Unit


U.S. Patent and Trademark OfficePTOL-326 (Rev. 08-06) Office Action Summary Part of Paper No/Mail Date 20100510

Part of Paper No./Mail Date 20100510

-- The MAILING DATE of this communication appears on the cover sheet with the correspondence address --Period for Reply

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS,WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION.- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed

after SIX (6) MONTHS from the mailing date of this communication.- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication.- Failure to reply within the set or extended period for reply will,; by statute, cause the application to become ABANDONED (35 U.S.C. § 133).

Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce anyeamrned patent term adjustment. See 37 CFR 1.704(b).

Status

1)[ Responsive to communication(s) filed on

2a)O This action is FINAL. 2b)® This action is non-final.

3)O- Since this application is in condition for allowance except for formal matters, prosecution as to the merits is

closed in accordance with the practice under Ex parte Quayle, 1935 C.D. 11, 453 O.G. 213.

Disposition of Claims

4)® Claim(s) 1-20 is/are pending in the application.

4a) Of the above claim(s) is/are withdrawn from consideration.

5)- Claim(s) is/are allowed.

6)® Claim(s) 1-13 and 16-20 is/are rejected.

7)® Claim(s) 14 and 15 is/are objected to.

8) Claim(s) are subject to restriction and/or election requirement.

Application Papers

9)L The specification is objected to by the Examiner.

10)r The drawing(s) filed on _ is/are: a)LI accepted or b)L objected to by the Examiner.

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a).

Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d).

11)O The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152.

Priority under 35 U.S.C. § 119

12)I Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f).

a)Q All b)L Some * c)O None of:

1.0 Certified copies of the priority documents have been received.

2.0 Certified copies of the priority documents have been received in Application No.

3.- Copies of the certified copies of the priority documents have been received in this National Stage

application from the International Bureau (PCT Rule 17.2(a)).

* See the attached detailed Office action for a list of the certified copies not received.

Attachment(s)

1) ® Notice of References Cited (PTO-892) 4) O Interview Summary (PTO-413)

2) I Notice of Draftsperson's Patent Drawing Review (PTO-948) Paper No(s)/Mail Date.

3) ® Information Disclosure Statement(s) (PTO/SB/08) 5) O- Notice of Informal Patent Application

Paper No(s)/Mail Date 10/06. 6) E Other:

Office Action Summary

Application/Control Number: 11/231,353 Page 2Art Unit: 2625

Detailed Action

1. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for

all obviousness rejections set forth in this Office action:

(a) A patent may not be obtained though the invention is not identically disclosed or described as setforth in section 102 of this title, if the differences between the subject matter sought to be patented andthe prior art are such that the subject matter as a whole would have been obvious at the time theinvention was made to a person having ordinary skill in the art to which said subject matter pertains.Patentability shall not be negatived by the manner in which the invention was made.

Claims 1, 2, 5-13 and 16-20 are rejected under 35 U.S.C. 103(a) as being

unpatentable over Reilly.

With respect to claim 1, Reilly teaches a surveillance method, performed by the system

of figure 1, comprising at least one event sensor 26 in a security area to detect a

potential security breach (see para. 69, line 10); using at least one camera 28 (see

para. 52, lines 1-12) with a view of the security areas in which the event is sensed to

gather biometric data (see para. 52, lines 19-29; converting at least one person in the

vicinity of the sensing event (see para. 54); producing a subject dossier (para. 53, lines

27-29 and para. 46, lines 1-8) corresponding to the at least one person and detecting

biometric information of one or more persons with respect to the subject dossier. See

para. 48, lines 11-19 and 23-34.

Application/Control Number: 11/231,353 Page 3

Art Unit: 2625

Reilly teaches all of the subject matter claimed except for the specific language of

matching biometric information. But rather, Reilly uses the term "detecting" biometric

data with respect to a stored dossier of data on several persons stored in a memory

device 42.

It would have been obvious to one of ordinary skill in the art that "matching" and

"detecting" are synonymous terms for comparing live biometric data against pre-stored

biometric data to determined the identification of persons under surveillance.

With respect to claim 2, while Reilly teaches system 20 for identification of the clothing

of an individual under observation, Reilly does not specifically teach identifying the color

or clothes. However, Reilly teaches identifying image information about the clothing.

See para. 46, lines 5-8. it would have been obvious to one of ordinary skill in the art

that other information about clothes would include the color, hue or tint of clothing worn

by the person(s) under surveillance.

With respect to claim 5, see para. 48, lines 14 and 15. The motivation to claim 5 is the

same as that to claim 1.


With respect to claims 6 and 11, Reilly teaches wherein the at least one camera 28

gathers one or more clothing information (para. 46, lines 5-8), facial features (para. 48,

lines 12-14), skin color (para. 48, lines 14-15).

Reilly teaches gathering biometric data (see para. 52, lines 19-29); converting at least

one person in the vicinity of the sensing event (see para. 54); producing a subject

dossier (para. 53, lines 27-29 and para. 46, lines 1-8) corresponding to the at least one

person and detecting biometric information of one or more persons with respect to the

subject dossier. See para. 48, lines 11-19, 23-34.

Reilly teaches system 20 for identification of the clothing of an individual under

observation. Reilly does not specifically teach identifying the color or clothes. However,

Reilly teaches identifying image information about the clothing. See para. 46, lines 5-8.

it would have been obvious to one of ordinary skill in the art that other information about

clothes would include the color, hue or tint of clothing worn by the person(s) under

surveillance.




device 42.





With respect to claim 7, the limitation of the claim can be ascertained from individual

clothing portions i.e., size of the clothing and its contour upon the persons in

surveillance. Hence, the height and weight ca be estimated. See also para. 46. See

para. 52, line 15-19.

With respect to claim 8-10, Reilly teaches image cameras for detecting the position of

objects with respect to other objects, using acoustic wave generation and other

waveforms. Assuming Reillly uses NTSC cameras, it would have been obvious to

study the speed of the subject by comparing the pixel movement of the target subject in

relationship with the frame rate of the camera. Bt knowing the pixel change rate and

the frame rate, the speed of the person under surveillance can be detected. Assuming,

Reilly does not use the NTSC camera, it would have been obvious to use the acoustic

or other wave forms taught by Reilly for measuring the presence of the moving object

with respect to other surrounding objects. The rate of change of the objects with

respect to other objects can be used to detect a velocity of a target by Doppler

frequency processing. The velocity of the moving object can be determined using the

Doppler technique.


Art Unit: 2625

Moreover, it would have been obvious to one of ordinary skill in the art that with the

determination of the movement of the target under surveillance, the direction of the

camera can be determined. With the determination of the direction, other control

switches that control the operation of the network of cameras, can be used to activate a

camera in a network that covers the peripheral of the zone in which the camera had last

detected the subject, for the purpose of tracking the speed and direction of a person

under surveillance.

With respect to claim 12, Reilly teaches system 20 for identification of the clothing of an

individual under observation, Reilly does not specifically teach identifying the color or

clothes. However, Reilly teaches identifying image information about the clothing. See

para. 46, lines 5-8. It would have been obvious to one of ordinary skill in the art that

other information about clothes would include the color, hue or tint of clothing worn by

the person(s) under surveillance.

Regarding the weight and height of the person under observation, the limitation of the

claim can be ascertained from individual clothing portions i.e., size of the clothing and

its contour upon the persons in surveillance. Hence, the height and weight can be

estimated. See also para. 46. See para. 52, lines 15-19.


Reilly teaches image cameras for detecting the position of objects with respect to other

objects, using acoustic wave generation and. other waveforms. Assuming Reillly uses

NTSC cameras, it would have been obvious to study the speed of the subject by

comparing the pixel movement of the target subject in relationship with the frame rate of

the camera. By knowing the pixel change rate and the frame rate, the speed of the

person under surveillance can be detected. Assuming, Reilly does not use the NTSC

camera, it would have been obvious to use the acoustic or other wave forms taught by

Reilly for measuring the presence of the moving object with respect to other surrounding

objects. The rate of change of the objects with respect to other objects can be used to

detect a velocity of a target by Doppler frequency processing. The velocity of the

moving object can be determined using the Doppler technique.

Moreover, it would have been obvious to one of ordinary skill in the art that with the

determination of the movement of the target under surveillance, the direction of the

camera can be determined. With the determination of the direction, other control

switches that control the operation of the network of cameras, can be used to activate a

camera in a network that covers the peripheral of the zone in which the camera had last

detected the subject, for the purpose of tracking the speed and direction of a person

under surveillance.

With respect to claim 13, Reilly teaches a display 132, see also para. 51.


With respect to claim 16, see para. 52, lines 1-12.

With respect to claim 17 and 20, Reilly teaches a surveillance system as shown by

figure 1, at least one sensor disposed in the security area of a surveillance region as

claimed, see para. 69, line 10; a plurality of cameras 9see para. 52, lines 1-12; wherein

the cameras automatically generate biometric data (see para. 52, lines 19-29)

concerning a subject in the surveillance region, see also para. 54; where in one or more

other plurality of cameras search for other subject persons (see para. 53, lines 7-11; a

processing system (40); wherein the processing system is programmable (see para.

48, lines 11-19 and para. 49 and 50); and wherein the processing system is

programmable (see para. 49) to detect biometric information in relation to other stored

dossier of a subject. See also para. 48, lines 11-19




device 42.





Art Unit: 2625

With respect to claim 18, see para. 52, lines 1-12.

With respect to claim 19, Reilly teaches wherein the processing system 40 is

programmable (para. 49) to produce a subject dossier that includes one or more

clothing, facial features, skin color, hair color and estimated height and weight. Reilly

teaches wherein the at least one camera 28 gathers one or more clothing information,

facial features (para. 48, lines 12-14) , skin color (para. 48, lines 14-15).


observation, Reilly does not specifically teach identifying the color or clothes. However,


It would have been obvious to one of ordinary skill in the art that other information about

clothes would include the color, hue or tint of clothing worn by the person9s) under

surveillance.

Note that the weight and height of an individual claim can be ascertained from individual

clothing portions i.e., size of the clothing and its contour upon the persons in

surveillance. Hence, the height and weight ca be estimated. See also para. 46.. See

para. 52, line 15-19.



observation, Reilly does not specifically teach identifying the color or clothes. However,


it would have been obvious to one of ordinary skill in the art that other information about

clothes would include the color, hue or tint of clothing worn by the person(s) under

surveillance.

Reilly teaches gathering biometric data (see para. 52, lines 19-29) converting at least

one person in the vicinity of the sensing event (see para. 54); producing a subject

dossier (para. 53, lines 27-29 and para. 46, lines 1-8) corresponding to the at least one

person and detecting biometric information of one or more persons with respect to the

subject dossier. See para. 48, lines 11-19, 23-34.




device 42.


Art Unit: 2625




2.

Claims 3 and 4 are rejected under 35 U.S.C. 103(a) as being unpatentable over

Reilly in view of Monroe.

With respect to claim 3, Reilly teaches identifying a person as described at para.

52, lines 19-29. Reilly teaches identifying facial features (see para. 48, lines 12-14)

However, specific reference to facial signatures is not taught.

Monroe teaches at paragraphs. 12, 137, 139 andl40 for performing facial

signatures and comparing the facial attributes against facial dossiers stored in data

bases to be used by Intelligence organizations.


It would have been obvious to one of ordinary skill in the art to modify system 20

of Reilly for the purpose of including facial recognition as one of the identification

methods for identifying a person under survelliance.

With respect to claim 4, note that hair color would be identified as a facial

attribute such as mustaches, eyebrows, beards and eyelashes. The motivation of the

rejection of claim 4 is the same as that to claim 3.

3.

Claims Objected

Claims 14 and 15 are objected to as being dependent upon a rejected base

claim, but would be allowable if rewritten in independent form including all of the

limitations of the base claim and any intervening claims.


4. Any inquiry concerning this communication or earlier communications from

the examiner should be directed to Jerome Grant II whose telephone number is 571-

272-7463. The examiner can normally be reached on Mon.-Fri. from 9:00 to 5:00.

If attempts to reach the examiner by telephone are unsuccessful, the examiner's

supervisor, Edward Coles, can be reached on 571-272-7402. The fax phone number

for the organization where this application or proceeding is assigned is 571-273-8300.

Information regarding the status of an application may be obtained from the

Patent Application Information Retrieval (PAIR) system. Status information for

published applications may be obtained from either Private PAIR or Public PAIR.

Status information for unpublished applications is available through Private PAIR only.

For more information about the PAIR system, see http://pair-direct.uspto.gov. Should

you have questions on access to the Private PAIR system, contact the Electronic

Business Center (EBC) at 866-217-9197 (toll-free).

/Jerome Grant II/Primary Examiner, Art Unit 2625


11/231,353 CHENG ET AL.Notice of References Cited CHENG ET AL.

Examiner Art Unit

Jerome Grant II 2625 Page 1 of 2


Document Number DateCountry Code-Number-Kind Code MM-YYYY Name Classification

* A US-7,376,276 05-2008 Shniberg et al. 382/224

* B US-2004/0117638 06-2004 Monroe, David A. 713/186

* C US-2004/0059953 03-2004 Purnell, John 713/202

* D US-2003/0115474 06-2003 Khan et al. 713/186

* E US-2003/0074590 04-2003 Fogle et al. 713/320

* F US-5,991,429 11-1999 Coffin et al. 382/118

* G US-6,816,184 11-2004 Brill et al. 348/143

* H US-6,687,831 02-2004 Albaugh et al. 726/7

* I US-2009/0322873 12-2009 Reilly et al. 348/143

* j US-2008/0043102 02-2008 ROWE et al. 348/143

* K US-2007/0186116 08-2007 Clemmensen et al.. 713/186

* L US-2006/0222209 10-2006 Zhang et al. 382/107

* M US-2005/0105764 05-2005 Han et al. 382/100


Document Number DateCountry Code-Number-Kind Code MM-YYYY Country Name Classification

N JP02004295798 10-2004 Japan Sakaibara H04N 7/18

O

P

Q

R

S

T

NON-PATENT DOCUMENTS

* Include as applicable: Author, Title Date, Publisher, Edition or Volume, Pertinent Pages)

U

V

W

X

"A copy of this reference is not being furnished with this Office action. (See MPEP § 707.05(a).)Dates in MM-YYYY format are publication dates. Classifications may be US or foreign.

U.S. Patent and Trademark OfficePTO-892 (Rev. 01-2001) Part of Paper' No. 20100510Notice of References Cited


11/231,353 CHENG ET AL.Notice of References Cited

Examiner Art Unit

Jerome Grant II 2625 Page 2 of 2


Document Number DateCountry Code-Number-Kind Code MM-YYYY

* A US-7,602,947 10-2009 Lemelson et al. 382/116

B US-

C US-

D US-

E US-

F US-

G US-

H US-

I US-

J US-

K US-

L US-

M US-


Document Number DateCountry Code-Number-Kind Code MM-YYYY Country Name Classification

N

O

P

Q

R

S

T

NON-PATENT DOCUMENTS

* Include as applicable: Author, Title Date, Publisher, Edition or Volume, Pertinent Pages)

U

V

W

X

*Ann.~ k;-- on. -- fro~~ .,t k:rf-nC Q^RDOf A ~nA copy o f this reference is not being furnisneo with this uOffice action. (ee MPEP §u.u5a).)

Dates in MM-YYYY format are publication dates. Classifications may be US or foreign.

U.S. Patent and Trademark OfficePTO-892 (Rev. 01-2001) Notice of References Cited Part of Paper No. 20100510

PAT-NO: JP02004295798A

DOCUMENT-IDENTIFIER:JP 2004295798 A

TITLE: SECURITY SYSTEM

PUBN-DATE: October 21, 2004

INVENTOR-INFORMATION:

NAME COUNTRY

SAKAKIBARA, NOBUHIRON/A

KASANO, NORIHIRO N/A

ASSIGNEE-INFORMATION:

NAME COUNTRY

JAPAN BEST RESCUE SYSTEM KKN/A

OCEAN NETWORK CO LTD N/A

APPL-NO: JP2003090397

APPL-DATE:March 28, 2003

INT-CL G08B013/194 , G06T003/00 G06T007/60 , G08B025/00 ,(IPC): G08B025/10 , H04N007/18

ABSTRACT:

PROBLEM TO BE SOLVED: To provide a low cost security system whichcan decide whether a photographed moving object is an intruderaccurately and actuate a notifying means, only if the object is anintruder.

SOLUTION: The security system is composed of a monitoring camera51 takes photographs of a moving object and outputs it as animation

data; a frame stripping off means 54 inputs the animation data anddivide it into frame images with a certain time interval; aconversion means 58 decides whether two frame.images separated inturn have any change on the image by pixel and transforms the frameimage into numerical string data, on the basis of determined results;a code-generating means 59 generates geometrical codes, correspondingto a shape profile of the moving object, on the basis of the

numerical string data, a deciding means 63 decides whether the moving

4/19/10, EAST Version: 2.4.1.1

object is a human being, on the basis of the geometrical codes; andan notification control means. The notification control meansactuates a notifying means, when the moving object is decided asbeing a human being.

COPYRIGHT: (C)2005,JPO&NCIPI

4/19/10, EAST Version:.2.4..1

er the Paperwork Reduction Act of 1995, no personsare required to res

TRANSMITTALFORM


Total Number of Pagesin This Submission 6

PTO/SB/21 (09-04)Approved for use through 0713112006. OMB 0651-0031

U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE___ I t._ mIU -1 nor a n.n:..res:.,.....:..... - 1 lc...aa Ia vu MD .- nrJ numoer.

Application Number

Filing Date

First Named Inventor.

Art Unit

Examiner Name


11/231,353

September 19, 2005

Ken P. CHENG

2621

Not Yet Assigned

577832000200


S Fee Transmittal Form Drawing(s) D After Allowance CommunicationeeTa r L._J rto TC

Fee Attached Licensing-related Papers D Appeal Communication to Board ofL FAppeals and Interferences

F] Amendment/Reply ] Petition i Appeal Communication to TCL_ 1 (Appeal Notice, Brief, Reply Brief)

After Final D Petition to Convert to a F Proprietary Information

Provisional Application

] Affidavits/declaration(s) Power of Attorney, Revocation ddress Status LetterLi Change of Correspondence Address L

Extension of Time Request ITerm inal D isclaim er Other Enclosure(s) (please

0 Express Abandonment Request F] Request for Refund 0 One (1) copy of each of twelve

(12) cited non-patent literature

Information Disclosure Statement C, Number of CD(s) references

(5 pages) Q CD, Number of CD(s)0 Return Receipt Postcard

] Certified Copy of Priority Landscape Table on CDDocument(s)

D Reply to Missing Parts/ RemarksIncomplete Application

] Reply to Missing Parts under Total page count of this submission as cited above does not includel37 CFR 1.52 or 1.53 the enclosed copies-of cited references.

SIGNATURE APPLICANT, ATTORNEY, OR AGENT

Firm Name MO IS N & FOERS LLP Customer No. 20872

Signature

Printed name 'Stephen C. Durant


sf-2209783 vl

I hereby certify that this paper (along with any paper referred to as being attached or enclosed) is being deposited with the U.S. Postal Service onthe date shown below with sufficient postage as First Class Mail, in envelope adressed to: Mail Stop Amendment. Commissioner forPatents. P.O. Box 1450. Alexandria. Virginia 22313-1450.

Dated, Octobeflr 13 2006 S nature:

Todd V. Leone

i ii Ai • IU 1 t I1'all irurlrrr r r YI IIGJ.I \

I

I

ond t a metnn o mormalin unless n lspavs a vaco uec conuol numoer.

al : c r . g,@,,wl.

, certify that this correspondence and any enclosures referenced thereinng deposited with the U.S. Postal Service on the date first shown belowficient postage as First Class Mail, in an envelope addressed to: Mailmendment, Commissioner for Patents, P.O. Box 14F, Alexandria,2313-1450.

October 13. 2006 Signature:

PatentDocket No. 577832000200


In re Patent Application of:Ken P. CHENG et al.

Serial No.: 11/231,353

Filing Date: September 19, 2005

For: ADAPTIVE MULTI-MODALINTEGRATED BIOMETRICIDENTIFICATION DETECTIONAND SURVEILLANCE SYSTEMS


Group Art Unit: 2621

INFORMATION DISCLOSURESTATEMENT UNDER 37 C.F.R. § 1.97 & 1.98

Mail Stop AMENDMENTCommissioner for PatentsP.O. Box 1450Alexandria, Virginia 22313-1450

Dear Sir:

Pursuant to 37 C.F.R. § 1.97 and § 1.98, Applicants submit for consideration in the

above-identified application the documents listed on the attached Form PTO/SB/08a/b. Copies of

foreign documents and non-patent literature are submitted herewith. Document no. 10 on the

attached Form PTO/SB/O8a/b is a pending non-published U.S. patent application, and in accordance

with the Waiver of the Copy Requirement in 37 CFR 1.98 for Cited Pending U.S. Patent

Applications, a copy is not submitted herewith. The Examiner is requested to make these documents

of record.

sf=-2185936

Application No. 11/231,353 2 Docket No. 577832000200Filed: 09/19/2005Information Disclosure Statement

This Information Disclosure Statement is submitted:

Q With the application; accordingly, no fee or separate requirements are required.

I Before the mailing of a first Office Action after the filing of a Request for Continued

Examination under § 1.114. However, if applicable, a certification under 37 C.F.R. § 1.97

(e)(1) has been provided.

® Within three months of the application filing date or before mailing of a first Office

Action on the merits; accordingly, no fee or separate requirements are required.

However, if applicable, a certification under 37 C.F.R. § 1.97 (e)(1) has been provided.

L After receipt of a first Office Action on the merits but before mailing of a final Office Action

or Notice of Allowance.

] A fee is required. A check in the amount of_ is enclosed.

[- A fee is required. Accordingly, a Fee Transmittal form (PTO/SB/17) is attached to

this submission in duplicate.

[O A Certification under 37 C.F.R. § 1.97(e) is provided above; accordingly; no fee is

believed to be due.

O After mailing of a final Office Action or Notice of Allowance, but before payment of the

issue fee.

O A Certification under 37 C.F.R. § 1.97(e) is provided above and a check in the

amount of_ is enclosed.

L A Certification under 37 C.F.R. § 1.97(e) is provided above and a Fee Transmittal

form (PTO/SB/17 is attached to this submission in duplicate.)

Applicants would appreciate the Examiner initialing and returning the Form

PTO/SB/08a/b, indicating that the information has been considered and made of record herein.

The information contained in this Information Disclosure Statement under

37 C.F.R. § 1.97 and § 1.98 is not to be construed as a representation that: (i) a complete search has

been made; (ii) additional information material to the examination of this application does not exist;

(iii) the information, protocols, results and the like reported by third parties are accurate or enabling;

or (iv) the above information constitutes prior art to the subject invention.

sf-2185936

Application No. 11/231,353 3 Docket No. 577832000200Filed: 09/19/2005Information Disclosure Statement

In the unlikely event that the transmittal form is separated from this document and the

Patent and Trademark Office determines that an extension and/or other relief (such as payment of a

fee under 37 C.F.R. § 1.17 (p)) is required, Applicants petition for any required relief including

extensions of time and authorize the Commissioner to charge the cost of such petition and/or other

fees due in connection with the filing of this document to Deposit Account No. 03-1952

referencing 577832000200.

Dated: 0ct- I ) ado Respectfully submitted,

ByStep en C. urant

Registration No.: 31,506MORRISON & FOERSTER LLP425 Market StreetSan Francisco, California 94105-2482(415) 268-6982

sf-2185936

ALTERNATIVE TO PTO/SB/08alb (07-05)

te for form 1449/PTO

ORMATION DISCLOSURESTATEMENT BY APPLICANT

(Use as many sheets as necessary)

Sheet 1 of 2

Application Number

Filing Date


Art Unit

Examiner Name

Attorney Docket Number

11/231,353

September 19, 2005

Ken P. CHENG

2621

Not Yet Assigned

577832000200

U.S. PATENT DOCUMENTSDocument Number Pages. Columns. Lines, Where

Examiner Cite Publcation Date Name of Patentee or Relevant Passages or RelevantInitials' No.' Number-Kind Codea' (if known) MM-OD-YYYY Applicant of Cited Figures Appear

Document

1. US-6,591,224 07-08-2003 Sullivan et al.2. US-6,609,198 08-19-2003 Wood et al.3. US-6,697,103-B1 02-24-2004 Fernandez et al.4.- US-6,970,582-A1 11-29-2005 Langley5. US-20020138768-A1 09-26-2002 Murakami et al.6. US-20020190119-A1 12-19-2002 Huffman7. US-20040081338-A1 04-29-2004 Takenaka8. US-20050265607-A1 12-01-2005 Chang9. US-20060112039-A1 05-25-2006 Wang et al.


Examiner Cite Foreign Patent Document Publication Pages, Columns. Lines,EamiN CiDate Name of Patentee or Where Relevant Passages

Intials' No.' CountryCoda-Numbe-Gnd Code(ifknown) MM-DD-YYYY Applicant of Cited Document or Relevant Figures Appear To

I I I I

*EXAMINER: Initial if information considered, whether or not citation is in conformance with MPEP 609. Draw line through citation if not in conformance and notconsidered. Include copy of this form with next communication to applicant. ' Applicants unique citation designation number (optional). ' See Kinds Codes ofUSPTO Patent Documents at www.usplo.gov or MPEP 901.04. 3 Enter Office that issued the document, by the two-letter code (WIPO Standard ST.3). 'ForJapanese patent documents, the indication of the year of the reign of the Emperor must precede the serial number of the patent document. sKind of document bythe appropriate symbols as indicated on the document under WIPO Standard ST. 16 if possible. Applicant is to place a check mark here if English languageTranslation is attached.

NON PATENT LITERATURE DOCUMENTS

- Include name of the author (in CAPITAL LETTERS), title of the article (when appropriate), title of the item (book,Initials' No.' magazine, journal, serial, symposium, catalog, etc.), date, page(s), volume-issue number(s), publisher, city T

and/or countrywhere published.

10. h et al., "Robust Perceptual Color Identification" US-11/229,091,d: 9/16/2005

11. Belhumeur, A. et al. (1997). "Eigenfaces vs. Fisherfaces: recognitionusing class specific linear projection", IEEE Transactions on PatternAnalysis and Machine Intelligence 19(7): 711-720.

12. Brunelli, R. and D. Falavigna. (1995). "Person identification usingmultiple cues," IEEE Transactions on Pattern Analysis and MachineIntelligence 17(10): 955-966.

13. Brunelli, R. et al. (1995). "Automatic Person Recognition by UsingAcoustic and Geometric Features", Machine Vision and Applications8: 317-325.

14. HONG, Lin and Anil K. Jain. (1998). "Integrating faces and fingerprintsfor personal identification," IEEE Transactions on Pattern Analysis

E x a n e r JeromeGrant Dat eISigInature I !eoeGrn I Cons idered I05/10/2010

sf- 2185930

ALL REFERENCES CONSIDERED EXCEPT WHERE LINED THROUGH. /JG/

Complete if Known

/JG/'4

I 1

s.

ALTERNATIVE TO PTOISB/08a/b (07-05)

Complete if KnownSubstitute for form 14491PTO


INFORMATION DISCLOSURE Filing Date September 19, 2005

STATEMENT BY APPLICANT First Named Inventor Ken P. CHENGArt Unit 2621

(Use as many sheets as necessary) Examiner Name Not Yet Assigned

Sheet 2 of 2 Attorney Docket Number 577832000200

and Machine Intelligence 20(12): 1295 - 1307.15. International Search Report mailed on April 2006 for PCT Patent

Application Number PCT/US05/33378 filed on September 19, 2005,one page.

16. JAIN, A. K. et al. (1997). "On-Line Fingerprint Verification," IEEETransactions on Pattern Analysis and Machine Intelligence archive19(4): 302- 314.

17. Kittler, J. et al. (1998). "On combining classifiers", IEEE Transactionson Pattern Analysis and Machine Intelligence 20(3): 226-239.

18. Lu X et al. (2003). "Combing classifiers for face recognition", IEEEInternational Conference on Multimedia Systems and Expo, Baltimore,MD, July.

19. Maio, D. et al. (2002). "FVC2000: fingerprint verification competition",IEEE Transactions on Pattern Analysis and Machine Intelligence 24(3):402 -412.

20. Phillips, P.J. et al. (2000). "The FERET evaluation methodology forface-recognition algorithms", IEEE Transactions on Pattern Analysisand Machine Intelligence 22(10): 1090- 1104.

21. Senior, A. (2001). "A combination fingerprint classifier", IEEETransactions on Pattern Analysis and Machine Intelligence 23(10):1165 1174.

22. TURK, A. and A. Pentland. (1991). "Eigenfaces for Recognition".Journal of Cognitive Neuroscience 3 (1): 71-86.

'EXAMINER: Initial it reference considered, whether or not citation is in conformance with MPEP 609. Draw line through citation if not in conformance and not

considered. Include copy of this form with next communication to applicant.

'Applicant's unique citation designation number (optional). 2Appicant is to place a check mark here if English language Translation is attached.

Examiner /Jeroe Grant / at e 05/10/2010

sf-2185930ALL REFERENCES CONSIDERED EXCEPT WHERE LINED THROUGH. /JG!



APPLICATION NO. FLING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO.


7590

KEN P. CHENG20691 REID LANESARATOGA, CA 95070-5325

SEXAMINERGRANT II, JEROME

ART UNIT PAPER NUMBER

2625

MAIL DATE I DELIVERY MODE

12/27/2010 PAPER



PTOL-90A (Rev. 04/07)

I

I

I

12/27/2010

Application No. Applicant(s)

11/231,353 CHENG ET AL.Notice of Abandonment Examiner Art Unit


-- The MAILING DATE of this communication appears on the cover sheet with the correspondence address--

This application is abandoned in view of:

1. ® Applicant's failure to timely file a proper reply to the Office letter mailed on 13 May 2010.(a) O A reply was received on (with a Certificate of Mailing or Transmission dated _ ), which is after the expiration of the

period for reply (including a total extension of time of _ month(s)) which expired on(b) O A proposed reply was received on , but it does not constitute a proper reply under 37 CFR 1.113 (a) to the final rejection.

(A proper reply under 37 CFR 1.113 to a final rejection consists only of: (1) a timely filed amendment which places theapplication in condition for allowance; (2) a timely filed Notice of Appeal (with appeal fee); or (3) a timely filed Request forContinued Examination (RCE) in compliance with 37 CFR 1.114).

(c) O A reply was received on but it does not constitute a proper reply, or a bona fide attempt at a proper reply, to the non-final rejection. See 37 CFR 1.85(a) and 1.111. (See explanation in box 7 below).

(d) ® No reply has been received.

2. O Applicant's failure to timely pay the required issue fee and publication fee, if applicable, within the statutory period of three monthsfrom the mailing date of the Notice of Allowance (PTOL-85).

(a) O The issue fee and publication fee, if applicable, was received on _ (with a Certificate of Mailing or Transmission dated.), which is after the expiration of the statutory period for payment of the issue fee (and publication fee) set in the Notice of

Allowance (PTOL-85).

(b) O The submitted fee of $. is insufficient. A balance of $ is due.

The issue fee required by 37 CFR 1.18 is $ . The publication fee, if required by 37 CFR 1.18(d), is $ .

(c) O The issue fee and publication fee, if applicable, has not been received.

3.0 Applicant's failure to timely file corrected drawings as required by, and within the three-month period set in, the Notice ofAllowability (PTO-37).

(a) O Proposed corrected drawings were received on _ (with a Certificate of Mailing or Transmission dated , which isafter the expiration of the period for reply.

(b) O No corrected drawings have been received.

4. O The letter of express abandonment which is signed by the attorney or agent of record, the assignee of the entire interest, or all ofthe applicants.

5. O The letter of express abandonment which is signed by an attorney or agent (acting in a representative capacity under 37 CFR1.34(a)) upon the filing of a continuing application.

6. O The decision by the Board of Patent Appeals and Interference rendered on _ and because the period for seeking court reviewof the decision has expired and there are no allowed claims.

7. O The reason(s) below:


Petitions to revive under 37 CFR 1.137(a) or (b), or requests to withdraw the holding of abandonment under 37 CFR 1.181, should be promptly filed tominimize any negative effects on patent term.

IU.S Patent anrademark Office

PTO-1432 (Rev: 04-01) Notice of Abandonment Part of Paper No. 20101220


EFS ID: 9357324




Title of Invention: Adaptive multi-modal integrated biometric identification detection andTitle of Invention: suvi.nc ytm

surveillance systems


KEN P. CHENG

20691 REID LANE

Correspondence Address:

SARATOGA CA 95070-5325

US-

Filer: Robert Hayden

Filer Authorized By:

Attorney Docket Number:

Receipt Date: 01-FEB-2011





Submitted with Payment yes

Payment Type Deposit Account

Payment was successfully received in RAM $810

RAM confirmation Number 7121.

Deposit Account 161331

Authorized User

The Director of the USPTO is hereby authorized to charge indicated fees and credit any overpayment as follows:

Charge any Additional Fees required under 37 C.F.R. Section 1.21 (Miscellaneous fees and charges)

File Listing:

Document File Size(Bytes)/ Multi PagesNumber Document Description File Name Message Digest Part /.zip (ifappl.)

205807Petition for review by the Office of sb0064.pdf no 3

Petitions.99076d22654093bf1773808ccc01b73cdbd

6539f

Warnings:

Information:

5330_07-Amendment-A.pdf

Multipart Description/PDF files in .zip description

Document Description Start End

Amendment/Req. Reconsideration-After Non-Final Reject 1 1

Claims 2 3

Applicant Arguments/Remarks Made in an Amendment 4 4

Warnings:

Information:

349199

3 5330_07-POA-and-373b.PDF1b9f4f7007956da6b27B4451b250a049f9

4170a

Multipart Description/PDF files in .zip description

Document Description Start End

Power of Attorney 1 1

Assignee showing of ownership per 37 CFR 3.73(b). 2 2

Warnings:

Information:

4 Fee Worksheet (PTO-875) fee-info.pdf

Warnings:

Information:

1




National Stage of an International Application under 35 U.S.C. 371If a timely submission to enter the national stage of an international application is compliant with the conditions of 35U.S.C. 371 and other applicable requirements a Form PCT/DO/EO/903 indicating acceptance of the application as anational stage submission under 35 U.S.C. 371 will be issued in addition to the Filing Receipt, in due course.


Doc Code: PET.OPDocument Description: Petition for Review by the Office of Petitions PTO/SB/64 (07-09)

Approved for use through 07/31/2012. OMB 0651-0031U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE

Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number.

PETITION FOR REVIVAL OF AN APPLICATION FOR PATENT Docket Number (Optional)ABANDONED UNINTENTIONALLY UNDER 37 CFR 1.137(b) 5330.07 (SMC)

First named inventor: Ken Prayoon Cheng

Application No.: 11/231,353 Art Unit: 2625

Filed: 09-19-2005 Examiner: JEROME GRANT II

Title: Adaptive Multi-Modal Integrated Biometric Identification Detection and Surveillance Systems

Attention: Office of PetitionsMail Stop PetitionCommissioner for PatentsP.O. Box 1450Alexandria, VA 22313-1450FAX (571) 273-8300

NOTE: If information or assistance is needed in completing this form, please contact PetitionsInformation at (571) 272-3282.

The above-identified application became abandoned for failure to file a timely and proper reply to a notice or action by theUnited States Patent and Trademark Office. The date of abandonment is the day after the expiration date of the period setfor reply in the office notice or action plus any extensions of time actually obtained.

APPLICANT HEREBY PETITIONS FOR REVIVAL OF THIS APPLICATION

NOTE: A grantable petition requires the following items:(1) Petition fee;(2) Reply and/or issue fee;(3) Terminal disclaimer with disclaimer fee - required for all utility and plant applications filed

before June 8, 1995; and for all design applications; and(4) Statement that the entire delay was unintentional

1. Petition Fee

Q~ Small entity-fee $ 810 (37 CFR 1.17(m)). Application claims small entity status. See 37 CFR 1.27.

Q Other than small entity-fee $ (37 CFR 1.17(m))

2. Reply and/or feeA. The reply and/or fee to the above-noted Office action in

the form of a written reply (identify type of reply):

D- has been filed previously on

is enclosed herewith.

B. The issue fee and publication fee (if applicable) of $

D has been paid previously on

Sis enclosed herewith.[Page 1 of 2]'

This collection of information is required by 37 CFR 1.137(b). The information is required to obtain or retain a benefit by the public which is to file (and by the USPTO toprocess) an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.11 and 1.14. This collection is estimated to take 1.0 hour to complete, includinggathering, preparing, and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Any comments on the amount oftime you require to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S. Patent and Trademark Office,U.S. Department of Commerce, P.O. Box 1450, Alexandria, VA 22313-1450. DO NOT SEND FEES OR COMPLETED FORMS TO THIS ADDRESS. SEND TO: MailStop Petition, Commissioner for Patents, P.O. Box 1450, Alexandria, VA 22313-1450.



U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCEUnder the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number

3. Terminal disclaimer with disclaimer fee

Q~ Since this utility/plant application was filed on or after June 8, 1995, no terminal disclaimer is required.

O A terminal disclaimer (and disclaimer fee (37 CFR 1.20(d)) of $ for a small entity or $ forother than a small entity) disclaiming the required period of time is enclosed herewith (see PTO/SB/63).

4. STATEMENT: The entire delay in filing the required reply from the due date for the required reply until the filing of agrantable petition under 37 CFR 1.137(b) was unintentional. [NOTE: The United States Patent and Trademark Office mayrequire additional information if there is a question as to whether either the abandonment or the delay in filing a petitionunder 37 CFR 1.137(b) was unintentional (MPEP 711.03(c), subsections (lll)(C) and (D)).]

WARNING:Petitioner/applicant is cautioned to avoid submitting personal information in documents filed in a patent application that may contributeto identity theft. Personal information such as social security numbers, bank account numbers, or credit card numbers (other than acheck or credit card authorization form PTO-2038 submitted for payment purposes) is never required by the USPTO to support apetition or an application. If this type of personal information is included in documents submitted to the USPTO, petitioners/applicantsshould consider redacting such personal information from the documents before submitting them to the USPTO. Petitioner/applicant isadvised that the record of a patent application is available to the public after publication of the application (unless a non-publicationrequest in compliance with 37 CFR 1.213(a) is made in the application) or issuance of a patent. Furthermore, the record from anabandoned application may also be available to the public if the application is referenced in a published application or an issued patent(see 37 CFR 1.14). Checks and credit card authorization forms PTO-2038 submitted for payment purposes are not retained in theapplication file and therefore are not publicly available.

/Robert Hayden, #42,645/ February 1, 2011

SignatureRobert Hayden

Date42,645

Type or Printed name Registration Number, If applicable

Peters Verny, LLP 650.324.1677Address Telephone Number

425 Sherman Avenue, Suite 230, Palo Alto, CA 94306Address

Enclosures: i Fee Payment

/IIReply

i Terminal Disclaimer Form

i Additional sheets containing statements establishing unintentional delay

Other: Power of Attorney and Statement under 3.73(b)

[Page 2 of 21

CERTIFICATE OF MAILING OR TRANSMISSION [37 CFR 1.8(a)]I hereby certify that this correspondence is being:

SDeposited with the United States Postal Service on the date shown below with sufficient postage asQ first class mail in an envelope addressed to: Mail Stop Petition, Commissioner for Patents, P. O. Box

1450, Alexandria, VA 22313-1450.

QTransmitted by facsimile on the date shown below to the United States Patent and Trademark Officeat (571) 273-8300.

Date Signature

Typed or printed name of person signing certificate

Privacy Act Statement

The Privacy Act of 1974 (P.L. 93-579) requires that you be given certain information inconnection with your submission of the attached form related to a patent application or patent.Accordingly, pursuant to the requirements of the Act, please be advised that: (1) the generalauthority for the collection of this information is 35 U.S.C. 2(b)(2); (2) furnishing of theinformation solicited is voluntary; and (3) the principal purpose for which the information is usedby the U.S. Patent and Trademark Office is to process and/or examine your submission relatedto a patent application or patent. If you do not furnish the requested information, the U.S. Patentand Trademark Office may not be able to process and/or examine your submission, which mayresult in termination of proceedings or abandonment of the application or expiration of thepatent.

The information provided by you in this form will be subject to the following routine uses:

1. The information on this form will be treated confidentially to the extent allowed under theFreedom of Information Act (5 U.S.C. 552) and the Privacy Act (5 U.S.C. 552a). Recordsfrom this system of records may be disclosed to the Department of Justice to determinewhether the Freedom of Information Act requires disclosure of these records.

2. A record from this system of records may be disclosed, as a routine use, in the course ofpresenting evidence to a court, magistrate, or administrative tribunal, includingdisclosures to opposing counsel in the course of settlement negotiations.

3. A record in this system of records may be disclosed, as a routine use, to a Member ofCongress submitting a request involving an individual, to whom the record pertains,when the individual has requested assistance from the Member with respect to thesubject matter of the record.

4. A record in this system of records may be disclosed, as a routine use, to a contractor ofthe Agency having need for the information in order to perform a contract. Recipients ofinformation shall be required to comply with the requirements of the Privacy Act of 1974,as amended, pursuant to 5 U.S.C. 552a(m).

5. A record related to an International Application filed under the Patent Cooperation Treatyin this system of records may be disclosed, as a routine use, to the International Bureauof the World Intellectual Property Organization, pursuant to the Patent CooperationTreaty.

6. A record in this system of records may be disclosed, as a routine use, to another federalagency for purposes of National Security review (35 U.S.C. 181) and for review pursuantto the Atomic Energy Act (42 U.S.C. 218(c)).

7. A record from this system of records may be disclosed, as a routine use, to theAdministrator, General Services, or his/her designee, during an inspection of recordsconducted by GSA as part of that agency's responsibility to recommend improvements inrecords management practices and programs, under authority of 44 U.S.C. 2904 and2906. Such disclosure shall be made in accordance with the GSA regulations governinginspection of records for this purpose, and any other relevant (i.e., GSA or Commerce)directive. Such disclosure shall not be used to make determinations about individuals.

8. A record from this system of records may be disclosed, as a routine use, to the publicafter either publication of the application pursuant to 35 U.S.C. 122(b) or issuance of apatent pursuant to 35 U.S.C. 151. Further, a record may be disclosed, subject to thelimitations of 37 CFR 1.14, as a routine use, to the public if the record was filed in anapplication which became abandoned or in which the proceedings were terminated andwhich application is referenced by either a published application, an application open topublic inspections or an issued patent.

9. A record from this system of records may be disclosed, as a routine use, to a Federal,State, or local law enforcement agency, if the USPTO becomes aware of a violation orpotential violation of law or regulation.

IN THEUNITED STATES PATENT AND TRADEMARK OFFICE

APPLICANTS:

SERIAL NO.:

CONF. NO.:

FILING DATE:

TITLE:

EXAMINER:

ART UNIT:

ATTY. DKT. NO.:

Ken Prayoon Cheng et al.

11/231,353

4531

September 19, 2005

Adaptive Multi-Modal Integrated Biometric Identification

Detection and Surveillance Systems

Jerome Grant II

2625

5330.07 (SMC)

Amendment A

Sir:

In response to the Office action mailed on May 13, 2010, please amend the

application as follows and consider the following remarks. A Petition for Revival of an

Application for Patent Abandoned Unintentionally under 37 CFR 1.137(b) and the fee

required under 37 CFR 1.17(m) are submitted herewith.

5330.07 (SMC)1

IN THE CLAIMS:

The following listing of claims will replace all prior versions, and listings, of claims in

the application.

1-13. (cancelled)

14. (currently amended) A The surveillance method of claim fu , ther including

comprising:

using at least one evet one event sensor disposed in a security area of a security area of a surveillance region

to sense an occurrence of a potential security breach event;

using at least one camera with a view of the security area in which the event is

sensed to gather biometric information concerning at least one person in

the vicinity of the security area at about the time of the sensing of the

event;

producing a subject dossier corresponding to the at least one person, wherein the

subject dossier including includes at least two biometric signatures;

matching biometric information of one or more persons captured by one or more

other cameras in the vicinity of the at least one camera with corresponding

biometric information in the subject dossier; and fu her-includm , ,

fusing the at least two signatures arind including the fused signature in the subject

dossier.

5330.07 (SMC) 2

15. (currently amended) A The surveillance method of claim 1 further including

comprising:

using at least one event sensor disposed in a security area of a surveillance region

to sense an occurrence of a potential security breach event;

using at least one camera with a view of the security area in which the event is

sensed to gather biometric information concerning at least one person in

the vicinity of the security area at about the time of the sensing of the

event,

producing a subject dossier corresponding to the at least one person, wherein the

subject dossier including ineludes at least two biometric signatures;

matching biometric information of one or more persons captured by one or more

other cameras in the vicinity of the at least one camera with corresponding

biometric information in the subject dossier; and fuher-i+ udg

incrementally fusing the at least two signatures and including the fused signature

in the subject dossier.

16-20. (cancelled)

5330.07 (SMC)

REMARKS

Claims 14 and 15 are presently pending following the cancellation of claims 1-13

and 16-20 herein. The Applicants thank the Examiner for the indication of allowable

subject matter in both pending claims. Each claim has been amended to place the claim

in independent form. In view of the cancellation of claims 1-13 and 16-20, their

rejections under 35 U.S.C. § 103(a) as unpatentable over Reilly et al. (USPGP

2009/0322873) either alone or in combination with Monroe (USPGP 2004/0117638) are

deemed to be moot. The Applicants, however, do not agree that the cancelled claims are

obvious over the cited references, reserve the right to pursue these claims in a

continuation application, and cancel them herein merely to expedite the allowance of the

present application.

Both pending claims are now allowable and the Applicants therefore request a

Notice of Allowance from the Examiner. Should the Examiner have questions, the

Applicants' undersigned attorney may be reached at the number provided.

Ken Prayoon Cheng et al.

Dated: February 1, 2011 By: /Robert Hayden, #42,645/Robert Hayden, Reg. No. 42,645Peters Verny, LLP425 Sherman Avenue, Suite 230Palo Alto, CA 94306TEL: (650) 324-1677FAX: (650) 324-1678

5330.07 (SMC)



POWER OF ATTORNEY TO PROSECUTE APPLICATIONS BEFORE THE USPTO

I hereby revoke all previous powers of attorney given in the application identified in the attached statement under37 CFR 3.73(b).I hereby appoint:

r Practitioners associated with the Customer Number:

OR

23308S Practitioner(s) named below (if more than ten patent practitioners are to be named, then a customer number must be used):

Name Registration Name RegistrationNumber . Number

as attorney(s) or agent(s) to represent the undersigned before the United States Patent and Trademark Office (USPTO) in connection withany and all patent applications assigned ony to the undersigned according to the USPTO assignment records or assignment documentsattached to this form in accordance with 37 CFR 3.73(b).

Please change the correspondence address for the application identified in the attached statement under 37 CFR 3.73(b) to:

SThe address associated with Customer Number: 23308OR

Firm orIndividual Name

Address

City State Zip

Country

Telephone Email

Assignee Name and Address:

Proximex Corporation440 N. Wolfe Rd.Sunnyvale, CA94085

A copy of this form, together with a statement under 37 CFR 3.73(b) (Form PTOISBI96 or equivalent) is required to befiled in each application in which this form is used. The statement under 37 CFR 3.73(b) may be completed by one ofthe practitioners appointed in this form if the appointed practitioner is authorized to act on behalf of the assignee,and must identify the application in which this Power of Attorney is to be filed.

SIGNATURE of Assignee of RecordThe individual whose signature and title is supplied below is authorized to act on behalf of the assignee

.Signature ~.Date Jr'/20Name Ken Prayoon Cheng Telephone 408-524-1510

Title President

If you need ass/stance in completing the form, call 1-800-PTO-9199 and select option 2

This o lliuun u Informaun i required by 3/ l.R 1.3 1, 1.. and 1.3.. me inrormation is require to obtain or retain a beneit by the public which is to file (andby the USPTO to process) an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.11 and 1.14. This collection is estimated to take 3 minutesto complete, including gathering, preparing, and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Anycomments on the amount of time you require to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer,U.S. Patent and Trademark Office, U.S. Department of Commerce, P.O. Box 1450, Alexandria, VA 22313-1450. DO NOT SEND FEES OR COMPLETEDFORMS TO THIS ADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1460, Alexandria, VA 22313-1450.

I

Thi. / AII NIAn A7 IM-Minn ;n r .. l urar/ ti~'17 rco 4 '11 1 ']n -- A 4 ',o 'r"- : f i. w1: w :_ . .. - r _..-_._ - l L.. ..+




Applicant/Patent Owner: Ken Pravoon Chena et al

Application No./Patent No.: 11/231,353 Filed/Issue Date: September 19, 2005

Entitled: Adaptive multi-modal integrated biometric identification detection and surveillance systems

Prnximpx Conrpnratinn , a nCrpnm rAtinn(Name of Assignee) (Type of Assignee, e.g., corporation, partnership, university, government agency, etc.)

states that it is:1, the assignee of the entire right, title, and interest; or

2. O an assignee of less than the entire right, title and interest(The extent (by percentage) of its ownership interest is %)


A.QAn assignment from the inventor(s) of the patent application/patent identified above. The assignment was recordedin the United States Patent and Trademark Office at Reel 017473 , Frame 0506 , or for which a copythereof is attached.

ORB.4 A chain of title from the inventor(s), of the patent application/patent identified above, to the current assignee as follows:

1. From: To:The document was recorded in the United States Patent and Trademark Office atReel , Frame . , or for which a copy thereof is attached.

2. From: To:The document was recorded in the United States Patent and Trademark Office atReel , Frame , or for which a. copy. thereof is attached.

3. From: 'To:The document was recorded in the United States Patent and Trademark Office atReel _, Frame , or for which a copy thereof is attached.

- Additional documents in the chain of title are listed on a supplemental sheet.

W As required by 37 CFR 3.73(b)(1)(i), the documentary evidence of the chain of title from the original owner to theassignee was, or concurrently is being, submitted for recordation pursuant to 37 CFR 3.11.

[NOTE: A separate copy (i.e., a true copy of the original assignment document(s)) must be submitted to AssignmentDivision in accordance with 37 CFR Part 3, to record the assignment in the records of the USPTO. See MPEP302.08]

The undersigned (whose title is supplied ow) is authorized to act on behalf of the assignee.

V a

Signature.

Ken Pravoon Chen

Printed or Typed Name

PresidentTitle

Date

408-524-1510

Telephone Number

This collection of information is required by 37 CFR 3.73(b). The information is required to obtain or retain a benefit by the public which is to file (and by theUSPTO to process) an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.11 and 1.14. This collection is estimated to take 12 minutes tocomplete, including gathering, preparing, and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Anycomments on the amount of time you require to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer,U.S. Patent and Trademark Office. U.S. Department of Commerce, P.O. Box 1450, Alexandria, VA 22313-1450. DO NOT SEND FEES OR COMPLETEDFORMS TO THIS ADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450, Alexandria, VA 22313-1450.


Electronic Patent Application Fee Transmittal


Filing Date: 19-Sep-2005




Attorney Docket Number:

Filed as Small Entity

Utility under 35 USC 111(a) Filing Fees

Sub-Total inDescription Fee Code Quantity AmountS Dt

USD($)

Basic Filing:

Pages:

Claims:

Miscellaneous-Filing:

Petition:

Petition-revive unintent.abandoned appl 2453 1 810 810

Patent-Appeals-and-Interference:

Post-Allowance-and-Post-Issuance:

Extension-of-Time:

Sub-Total in

Description Fee Code Quantity. Amount USD($)USD($)

Miscellaneous:

Total in USD ($) 810o


APPLICATION NUMBER FILING OR 371(C) DATE

11/231,353 09/19/2005

23308PETERS VERNY, L.L.P.425 SHERMAN AVENUESUITE 230PALO ALTO, CA 94306

UNITED STATES DEPARTMENT OF COMMERCEUnited States Patent and Trademark OfficeAddress: COMMISSIONER FOR PATENTS

P.O Box 1450Alexandria, Virginia 22313-1450www.uspto.gov

FIRST NAMED APPLICANT I ATTY. DOCKET NO.ITLE

Ken Prayoon Cheng

CONFIRMATION NO. 4531POA ACCEPTANCE LETTER




The Power of Attorney in this application is accepted. Correspondence in this application will be mailed to theabove-address as provided by 37 CFR 1.33.

/ddinh/

Office of Data Management, Application Assistance Unit (571) 272-4000, or (571) 272-4200, or 1-888-786-0101

page 1 of 1




www.uspto.gov

PETERS VERNY, L.L.P.425 SHERMAN AVENUESUITE 230PALO ALTO, CA 94306

In re Application ofKen Prayoon Cheng, et. al.Application No. 11/231,353Filed: September 19, 2005Attorney Docket No. 5330.07 (SMC)

This is a decision on the petition underthe above-identified application.

MAILEDMAR 2 4 2011

OFFICE OFPETITIONS

ON PETITION

37 CFR 1.137(b), filed February 1, 2011, to revive

The application became abandoned for failure to file a reply to the non-final Office actionmailed May 13, 2010. A Notice of Abandonment was mailed on December 27, 2010.

Since the petition satisfies the requirements of 37 CFR 1.137(b) in that petitioner hassupplied (1) the reply in the form of an amendment; (2) the petition fee of $810; and (3) aproper statement of unintentional delay, the petition is GRANTED.

This application file is being referred to Technology Center Art Unit 2625 for review of theamendment submitted with the present petition.

concerning this decision should be directed to the undersigned at


EFS ID: 9753945









Attorney Docket Number: 5330.07 (SMC)

Receipt Date: 28-MAR-2011








RAM confirmation Number 3529


Authorized User

File Listing:

Document File Size(Bytes)/ Multi PagesNumber Document Description File NameMessageDigest Part/.zip (ifappl.)

Number Message Digest Part /.zip (if appl.)

615001Information Disclosure Statement (IDS)615001

5330_07-DS-03-2011 .pdf no 12Filed (SB/08)

409757129e88fe4a4c43ab71b96f040bel26977

Warnings:

Information:

A U.S. Patent Number Citation is required in the Information Disclosure Statement (IDS) form. You may remove the form to add the requireddata in order to correct the Informational Message or if you chose not to, the image of the form will be processed and be made availablewithin the Image File Wrapper (IFW) system. However, no data will be extracted from this form. Any additional data such as Foreign PatentDocuments or Non Patent Literature will be manually reviewed and keyed into USPTO systems.

50122Information Disclosure Statement (IDS) 53307-copending-appspdf

2 Filed (5B/08)5330_07-copending-apps.pdf7901e270f528803ce5854d34dc3d3375b

ac90

Warnings:

Information:

This is not an USPTO supplied IDS fillable form

416756PCTU50516961_ISR-and-WO.

3 NPL Documents pdf45db33c9d57414b678a942b7dbceccbe39

e569c3

Warnings:

Information:

NPL Documents

Warnings:

Information:

239134PCTUS0533750_ISR-and-WO.

L Documents pdfb400384634effbdbb567252c895f2075103

6c9e6

Warnings:

Information:

NPL Documents

Warnings:

Information:

7 NPL Documents

Warnings:

Information:

692624NPL Documents VidSys-Complaint.PDF no 11

le9549d560462af13f763309f0d6abb2f37107c7

Warnings:

Information:

8461864

NPL Documents Redstone_2003.PDF45e491b6al4175e5fbaa5c28blf 119268d4

9056

Warnings:

Information:

34951397777783-Reexamination-

10 NPL Documents Request.pdfRequest.pdf

780f67314b28b29c4326a30d2dddf07ac784fa5e

Warnings:

Information:

1705533

11 Foreign Reference WOO7044037.pdf

62bede2b099ab34e6be840acc707670f2f7((391

Warnings:

Information:

30157

Fee Worksheet (PTO-875) fee-info.pdf

4fc573053163947fecb4b7066b481 lbaaOE04db

Warnings:

Information:

Total Files Size (in bytes): 16642679



National Staae of an International AppDDlication under 35 U.S.C. 371If a timely submission to enter the national stage of an international application is compliant with the conditions of 35U.S.C. 371 and other applicable requirements a Form PCT/DO/EO/903 indicating acceptance of the application as anational stage submission under 35 U.S.C. 371 will be issued in addition to the Filing Receipt, in due course.

New International Annlication Filed with the USPTO as a Receivina OfficeIf a new international application is being filed and the international application includes the necessary components foran international filing date (see PCT Article 11 and MPEP 1810), a Notification of the International Application Numberand of the International Filing Date (Form PCT/RO/105) will be issued in due course, subject to prescriptions concerningnational security, and the date shown on this Acknowledgement Receipt will establish the international filing date ofthe application.

,.r ~~-~~ ~- ~ ~ r.. .. y ...~-~~~~~-~-- ~ -- ~-- ~ ~ ~ - ~ -

I

IN THE


APPLICANTS:

APPLICATION NO.:

CONFIRMATION NO.:

FILED:

TITLE:

EXAMINER:

GROUP ART UNIT:

ATTY.DKT.NO.:

Ken Prayoon Cheng

11/231,353

4531

September 19, 2005

Adaptive Multi-Modal Integrated Biometric IdentificationDetection and Surveillance Systems

Jerome Grant, II

2625

5330.07 (SMC)

MAIL STOP AMENDMENTCOMMISSIONER FOR PATENTSP.O. BOX 1450ALEXANDRIA, VA 22313-1450

List of Co-Pending Patent Applications That May Be DirectedTowards Similar Subject Matter

Examiner's Serial, First-Named Title

Initials Number Inventor Filing Date

12/838,973 Hu Chin Multi-Video Navigation 7-19-2010

11/728,404 Hu Chin Multi-Video Navigation System 3-23-2007

Examiner's Date

Signature:

Doc code: IDS PTO/SBi/08oa (01-10)Doc odes : noApproved for use through 07131/2012. OMB 0651-0031

Doc description: Information Disclosure Statement (IDS) Filed U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCEUnder the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it contains a valid OMB control number.

Application Number 11231353

Filing Date 2005-09-19

INFORMATION DISCLOSURE First Named Inventor Ken Prayoon Cheng

STATEMENT BY APPLICANT Art Unit12625( Not for submission under 37 CFR 1.99)

Examiner Name Jerome Grant, II

Attorney Docket Number 5330.07 (SMC)

U.S.PATENTS Remove

' Pages,Columns,Lines where

Examiner Cite Kind Name of Patentee or Applicant Pages,Columns,Lines whereExamiNoi Patent Number Cd Issue Date Naeocent Relevant Passages or Relevant

Initial* No Codel of cited DocumentFigures Appear

1 6604093 B1 2003-08-05 Etzion et al.

2 6906709 B1 2005-06-14 Larkin et al.

3 6408404 B1 2002-06-18 Michael Ladwig

4 6711587 B1 2004-03-23 Dufaux

5 6628829 B1 2003-09-30 Chasen

6 5912980 1999-06-15 Hunke

7 7136524 B1 2006-11-14 Goh et al.

8 6072496 2000-06-06 Guenter et al.

EFS Web 2.1.17




STATEMENT BY APPLICANT Art Unit 2625( Not for submission under 37 CFR 1.99)



9 6625315 B2 2003-09-23 Laumeyer et al.

10 6404900 B1 2002-06-11 Qian et al.

11 6707487 B1 2004-03-16 Aman et al:

12 5473369 1995-12-05 Abe

13 7777783 B1 2010-08-17 Chin et al.

14 7242810 B2 2007-07-10 Chang

15 6008912 1999-12-28 Sato et al.

16 5838465 1998-11-17 Satou et al.

17 6335985 B1 2002-01-01 Sambonsugi et al.

18 7187783 B2 2007-03-06 Moon et al.

19 6157469 2000-12-05 Mestha

EFS Web 2.1.17


Filing Date 2005-09-19INFORMATION DISCLOSUREINFORMATION DISCLOSURE First Named Inventor Ken Prayoon Cheng

STATEMENT BY APPLICANT Art Unit 25Art Unit 12625

( Not for submission under 37 CFR 1.99)Examiner Name Jerome Grant, II


20 7525576 B2 2009-04-28 Kannermark et al.

21 7382249 B2 2008-06-03 Fancella

22 7384590 B2 2008-06-10 Kelly et al.

23 7023913 B1 2006-04-04 Monroe et al.

24 7116353 B2 2006-10-03 Hobson et al.

25 7456727 B2 2008-11-25 Pinter et al.

26 7058204 B2 2006-06-06 Hildreth et al.

27 7385590 B2 2008-06-10 Millar et al.

28 7522186 B2 2009-04-21 Arpa et al.

29

30

EFS Web 2.1.17




STATEMENT BY APPLICANT Art Unit' 2625(Not for submission under 37 CFR 1.99)



31

32

33 7196722 B2 2007-03-27 White et al.

34 7295228 B2 2007-11-13 Roberts et al.

35 7242423 B2 2007-07-10 Lin

36 5258837 1993-11-02 Gormley

37 5479574 1995-12-26 Glier et al.

38 5701398 1997-12-23 Glier et al.

39 6757668 B1 2004-06-29 Goebel et al.

40 5769074 1998-06-23 Barnhill et al.

41 6306087 B1 2001-10-23 Barnhill et al.

EFS Web 2.1.17




STATEMENT BY APPLICANT Art Unit 2625Art Unit 12625(Not for submission under 37 CFR 1.99)



42 6248063 B1 2001-06-19 Barnhill et al.

43 5835901 1998-11-10 Duvoisin et al.

44 7366174 B2 2008-04-29 MacFaden et al.

45 7319796 B1 2008-01-15 Sharp

46 7286687 B2 2007-10-23 Lindwurm et al.

47 7277485 B1 2007-10-02 Puri

48 7269591 B2 2007-09-11 Wesinger et al.

49 7213174 B2 2007-05-01 Dahlquist et al.

50 7127464 B2 2006-10-24 Wesinger et al.

51 7028034 B2 2006-04-11 Wesinger et al

52 6976207 B1 2005-12-13 Rujn et al.

EFS Web 2.1.17


Filing Date 2005-09-19INFORMATION DISCLOSURE First Named Inventor Ken Prayoon Cheng

STATEMENT BY APPLICANT Art Unit 2625( Not for submission under 37 CFR 1.99)


Attorney Docket Number- 5330.07 (SMC)

53 6970513 B1 2005-11-29 Puri et al.

54 6968006 B1 2005-11-22 Puri et al.

55 6963899 B1 2005-11-08 Fernandez et al.

56 6909745 B1 2005-06-21 Puri et al.

57 6944319 B1 2005-09-13 Huang et al.

58 7746380 B2 2010-06-29 Maruya et al.

If you wish to add additional U.S. Patent citation information please click the Add button. Add

U.S.PATENT APPLICATION PUBLICATIONS Remove

. Pages,Columns,Lines whereExaminer Publication Kind Publication Name of Patentee or Applicant PagesColumnsLines where

Cite No Relevant Passages or RelevantInitial* Number Codel Date of cited DocumentFge saer

Figures Appear

1 20050222081 Al 2005-10-06 Jakobson et al.

2 20080294588 Al 2008-11-27 Morris et al.

3 20100002082 Al 2010-01-07 Buehler et al.

EFS Web 2.1.17




STATEMENT BY APPLICANT Art Unit' 2625( Not for submission under 37 CFR 1.99)



4 20030202102 Al 2003-10-30 Shiota et al.

5 20040100563 Al 2004-05-27 Sablak et al.

6 20030107653 Al 2003-06-12 Utsumi et al.

7 20010048765 Al 2001-12-06 Yi et al.

8 20070146484 Al 2007-06-28 Horton et al.

9 20020097322 Al 2002-07-25 Monroe et al.

10 20040001149 Al 2004-01-01 Smith et al.

11 20080218590 Al 2008-09-11 Park et al.

12 20080068464 Al 2008-03-20 Kitagawa et al.

13 20080106597 Al 2008-05-08 Amini et al.

14 20060279628 Al 2006-12-14 Fleming

EFS Web 2.1.17




STATEMENT BY APPLICANT Art Unit 2625Art Unit 2625( Not for submission under 37 CFR 1.99)



15 20060221184 Al 2006-10-05 Vallone et al.

16 20070146484 Al 2007-06-28 Horton et al.

17 20050132414 Al 2005-06-16 Bentley et al.

18 20080190949 Al 2008-08-14 Ivanov et al.

19 20050052532 Al 2005-03-10 Elooz et al.

20 20080079554 Al 2008-04-03 Boice

21 20060017807 Al 2006-01-26 Lee etal.

22 20040136574 Al 2004-07-15 Kozakaya et al.

23 20030169908 Al 2003-09-11 Kim et al.

24 20050100209 Al 2005-05-12 Lewis et al.

25 20040022442 Al 2004-02-05 Kim et al.

EFS Web 2.1.17



STATEMENT BY APPLICANT Art Unit 2625(Not for submission under 37 CFR 1.99)



26 20060203090 Al 2006-09-14 Wang et al.

27 20070154088 Al 2007-07-05 Goh et al.

28 20060284978 Al 2006-12-21 Girgensohn et al.

29 20080088706 Al 2008-04-17 Girgensohn et al.

If you wish to add additional U.S. Published Application citation information please click the Add button Add

FOREIGN PATENT DOCUMENTS Remove

Pages,Columns,LinesName of Patentee or whrRevat TExaminer Cite Foreign Document Country Kind Publication Applicant of citedwhere Relevant

Applicant of cited TInitial* No Number3 Code2 i Code4 Date Passages or Relevant

DocumentFigures Appear

PROXIMEX1 2007/044037 WO Al 2007-04-19 RO ICORPORATION

If you wish to add additional Foreign Patent Document citation information please click the Add button Add

NON-PATENT LITERATURE DOCUMENTS Remove

Include name of the author (in CAPITAL LETTERS), title of the article (when appropriate), title of the itemExaminer CiteInitials* No (book, magazine, journal, serial, symposium, catalog, etc), date, pages(s), volume-issue number(s), T5

Initials No publisher, city and/or country where published.

1 PCT/USO5/44656 International Search Report and Written Opinion, June 26, 2006 []

2 PCT/USO5/43808 International Search Report and Written Opinion, October 10, 2007 []

EFS Web 2.1.17




STATEMENT BY APPLICANT Art Unit 2625



EFS Web 2.1.17

3 PCT/US05/33378 International Search Report and Written Opinion, April 26, 2006

4 PCT/US05/33750 International Search Report and Written Opinion, May 2, 2007 O

5 PCT/US05/16961 International Search Report and Written Opinion, October 17, 2006 O-

6 VidSys, Inc. Complaint filed in the US District Court for the Eastern district of Virginia on October 19, 2010

7 Redstone Integrated Solutions Documents, 2003 I

8 95/001,525 Reexamination Request for 7,777,783, filed January 21, 2011 0

If you wish to add additional non-patent literature document citation information please click the Add button Add

EXAMINER SIGNATURE

Examiner Signature Date Considered

*EXAMINER: Initial if reference considered, whether or not citation is in conformance with MPEP 609. Draw line through a

citation if not in conformance and not considered. Include copy of this form with next communication to applicant.

1 See Kind Codes of USPTO Patent Documents at www.USPTO.GOV or MPEP 901.04. 2 Enter office that issued the document, by the two-letter code (WIPOStandard ST.3). 3 For Japanese patent documents, the indication of the year of the reign of the Emperor must precede the serial number of the patent document.4 Kind of document by the appropriate symbols as indicated on the document under WIPO Standard ST.16 if possible. 5 Applicant is to place a check mark here iiEnglish language translation is attached.



STATEMENT BY APPLICANT Art Unit 2625(Not for submission under 37 CFR 1.99)



CERTIFICATION STATEMENT

Please see 37 CFR 1.97 and 1.98 to make the appropriate selection(s):

That each item of information contained in the information disclosure statement was first cited in any communicationQ from a foreign patent office in a counterpart foreign application not more than three months prior to the filing of the

information disclosure statement. See 37 CFR 1.97(e)(1).

OR

That no item of information contained in the information disclosure statement was cited in a communication from aforeign patent office in a counterpart foreign application, and, to the knowledge of the person signing the certificationafter making reasonable inquiry, no item of information contained in the information disclosure statement was known to

I any individual designated in 37 CFR 1.56(c) more than three months prior to the filing of the information disclosurestatement. See 37 CFR 1.97(e)(2).

O See attached certification statement.

Q The fee set forth in 37 CFR 1.17 (p) has been submitted herewith.

Ii A certification statement is not submitted herewith.SIGNATURE

A signature of the applicant or representative is required in accordance with CFR 1.33, 10.18. Please see CFR 1.4(d) for theform of the signature.

Signature /Robert Hayden, #42,645/ Date (YYYY-MM-DD) 2011-03-28

Name/Print Robert Hayden Registration Number 42,645

This collection of information is required by 37 CFR 1.97 and 1.98. The information is required to obtain or retain a benefit by thepublic which is to file (and by the USPTO to process) an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR1.14. This collection is estimated to take 1 hour to complete, including gathering, preparing and submitting the completedapplication form to the USPTO. Time will vary depending upon the individual case. Any comments on the amount of time yourequire to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S.Patent and Trademark Office, U.S. Department of Commerce, P.O. Box 1450, Alexandria, VA 22313-1450. DO NOT SENDFEES OR COMPLETED FORMS TO THIS ADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450, Alexandria,VA 22313-1450.

EFS Web 2.1.17


The Privacy Act of 1974 (P.L. 93-579) requires that you be given certain information in connection with your submission of theattached form related to a patent application or patent. Accordingly, pursuant to the requirements of the Act, please be advisedthat: (1) the general authority for the collection of this information is 35 U.S.C. 2(b)(2); (2) furnishing of the information solicitedis voluntary; and (3) the principal purpose for which the information is used by the U.S. Patent and Trademark Office is toprocess and/or examine your submission related to a patent application or patent. If you do not furnish the requestedinformation, the U.S. Patent and Trademark Office may not be able to process and/or examine your submission, which mayresult in termination of proceedings or abandonment of the application or expiration of the patent.


1. The information on this form will be treated confidentially to the extent allowed under the Freedom of Information Act(5 U.S.C. 552) and the Privacy Act (5 U.S.C. 552a). Records from this system of records may be disclosed to theDepartment of Justice to determine whether the Freedom of Information Act requires disclosure of these record s.

2. A record from this system of records may be disclosed, as a routine use, in the course of presenting evidence to acourt, magistrate, or administrative tribunal, including disclosures to opposing counsel in the course of settlementnegotiations.

3. A record in this system of records may be disclosed, as a routine use, to a Member of Congress submitting arequest involving an individual, to whom the record pertains, when the individual has requested assistance from theMember with respect to the subject matter of the record.

4. A record in this system of records may be disclosed, as a routine use, to a contractor of the Agency having need forthe information in order to perform a contract. Recipients of information shall be required to comply with therequirements of the Privacy Act of 1974, as amended, pursuant to 5 U.S.C. 552a(m).

5. A record related to an International Application filed under the Patent Cooperation Treaty in this system of recordsmay be disclosed, as a routine use, to the International Bureau of the World Intellectual Property Organization, pursuantto the Patent Cooperation Treaty.

6. A record in this system of records may be disclosed, as a routine use, to another federal agency for purposes ofNational Security review (35 U.S.C. 181) and for review pursuant to the Atomic Energy Act (42 U.S.C. 218(c)).

7. A record from this system of records may be disclosed, as a routine use, to the Administrator, General Services, orhis/her designee, during an inspection of records conducted by GSA as part of that agency's responsibility torecommend improvements in records management practices and programs, under authority of 44 U.S.C. 2904 and2906. Such disclosure shall be made in accordance with the GSA regulations governing inspection of records forthispurpose, and any other relevant (i.e., GSA or Commerce) directive. Such disclosure shall not be used to makedeterminations about individuals.

8. A record from this system of records may be disclosed, as a routine use, to the public after either publication ofthe application pursuant to 35 U.S.C. 122(b) or issuance of a patent pursuant to 35 U.S.C. 151. Further, a recordmay be disclosed, subject to the limitations of 37 CFR 1.14, as a routine use, to the public if the record was filed inan application which became abandoned or in which the proceedings were terminated and which application isreferenced by either a published application, an application open to public inspections or an issued patent.

9. A record from this system of records may be disclosed, as a routine use, to a Federal, State, or local lawenforcement agency, if the USPTO becomes aware of a violation or potential violation of law or regulation.

EFS Web 2.1.17


PCT/US05/44656

A. CLASSIFICATION OF SUBJECT MATTERIPC: GO6K 9/00( 2006.01),9/62( 2006.01)

USPC: 382/167,156According to International Patent Classification (IPC) or to both national classification and IPC

B. FIELDS SEARCHED

Minimum documentation searched (classification system followed by classification symbols)U.S. :382/167,156





X US 6,711,587 (Dufaux) 23 March 2004 (21.03.2004), column 1, line 50-column 2, line 20-21-- 15), also column 5, lines 35-45.---y 22-23

Y US 6,628,829 BI( CHASEN) 30 September 2003 (30.09.2003) column 8. 22

Y US 5,912,980 (HUNKE) 15 June 1999 (15.06.1999) column 13-column 14. 23

S Further documents are listed in the continuation of Box C. E See patent family annex.

* Special categories of cited documents: "T" later document published after the international filing date or prioritydate and not in conflict with the application but cited to understand the

"A" document defining the general state of the art which is not considered to be of principle or theory underlying the inventionparticular relevance

"X" document of particular relevance; the claimed invention cannot be"E" earlier application or patent published on or after the International filing date considered novel or cannot be considered to involve an inventive step

when the document is taken alone."L" document which may throw doubts on priority claim(s) or which is cited to

establish the publication date of another citation or other special reason (as "Y" document of particular relevance; the claimed invention cannot bespecified) considered to involve an inventive step when the document is combined

with one or more other such documents, such combination being"O" document referring to an oral disclosure, use, exhibition or other means obvious to a person skilled in the art

"P" document published prior to the international filing date but later than the "&" document member of the same patent familypriority date claimed

Date of the actual completion of the international search Date of mailing o e in 4 na search report

03 May 2006 (03.05.2006)J e

Name and mailing address of the ISA/US thorized officerMail Stop PCT, Attn: ISA/USCommissioner for PatentsnggeWuP.O. Box 1450Alexandria, Virginia 22313-1450 Telephone No. 70 -5631

Facsimile No. (571) 273-3201

Form PCT/ISA/210 (second sheet) (April 2005)L

PATENT COOPERATION TREATYFrom theINTERNATIONAL SEARCHING AUTHORITY

To:STEVEN COLBYCARR & FERRELL LLP2200 GENG ROADPALO ALTO, CA 94303

Dr '

WRITTEN OPINION OF THEINTERNATIONAL SEARCHING AUTHORITY

(PCT Rule 43bis.1)

1. This opinion contains indications relating to the following items:

S Box No. I

E Box No. II

E Box No. III

E Box No. IV

0 Box No. V

Basis of the opinion

Priority

Non-establishment of opinion with regard to novelty, inventive step and industrial applicability

Lack of unity of invention

Reasoned statement under Rule 43bis.1(a)(i) with regard to novelty, inventive step or industrialapplicability; citations and explanations supporting such statement

Q Box No. VI Certain documents cited

Q Box No. VII Certain defects in the international application

S Box No. VIII Certain observations on the international application

2. FURTHER ACTION

If a demand for international preliminary examination is made, this opinion will be considered to be a written opinion of theInternational Preliminary Examining Authority ("IPEA") except that this does not apply where the applicant chooses anAuthority other than this one to be the IPEA and the chosen IPEA has notified the International Bureau under Rule 66. 1bis(b)that written opinions of this International Searching Authority will not be so considered.

If this opinion is, as provided above, considered to be a written opinion of the IPEA, the applicant is invited to submit to theIPEA a written reply together, where appropriate, with amendments, before the expiration of 3 months from the date ofmailing of Form PCT/ISA/220 or before the expiration of 22 months from the priority date, whichever expires later.

For further options, see Form PCT/ISA/220.

3. For further details, see notes to Form PCT/ISA/220. .

Name and mailing address of the ISA/ US Date of completion of this rized officeMail Stop PCT, Anttn: ISA/US opinionBAYATCommissioner for PatentsP.O. Box 1450 03 May 2006 (03.05.2006)Alexandria, Virginia 22313-1450 Telephone No. 1-2 7444

Facsimile No. (571) 273-3201Form PCT/ISA/237 (cover sheet) (April 2005)

Date of mailing(day/month/year) .7 .J N 2006

Applicant's or agent's file reference FOR FURTHER ACTION

PA3358PCT See paragraph 2 belowPA3358PCT

International application No. International filing date (day/month/year) Priority date (day/month/year)

PCT/US05/44656 09 December 2005 (09.12.2005) 16 September 2006 (16.09.2006)International Patent Classification (IPC) or both national classification and IPC

IPC: GO6K 9/00( 2006.01),9/62( 2006.01)USPC: 382/162,167,156Applicant

PROXIMEX CORPORATION'

i

TO"-


International application No.

PCT/US05/44656

Box No. I Basis of this opinion

1. With regard to the language, this opinion has been established on the basis of:

the international application in the language in which it was filed

D a translation of the international application into , which is the language of a translation furnished for the

purposes of international search (Rules 12.3(a) and 23.1l(b)).

2. With regard to any nucleotide and/or amino acid sequence disclosed in the international application and necessary to theclaimed invention, this opinion has been established on the basis of:

a. type of material

Q a sequence listing

D table(s) related to the sequence listing

b. format of material

D on paper

D in electronic form

c.time of filing/furnishing

D contained in the international application as filed.

D filed together with the international application in electronic form.

D furnished subsequently to this Authority for the purposes of search.

3. D In addition, in the case that more than one version or copy of a sequence listing and/or table(s) relating thereto has beenfiled or furnished, the required statements that the information in the subsequent or additional copies is identical to that inthe application as filed or does not go beyond the application as filed, as appropriate, were furnished.

4. Additional comments:

Form PCT/ISA/237(Box No. I) (April 2005)--- '-- ~ '---~-

International application No.WRITTEN OPINION OF THE , PCT/US05/44656

INTERNATIONAL SEARCHING AUTHORITY

Box No. V Reasoned statement under Rule 43 bis.1(a)(i) with regard to novelty, inventive step or industrialapplicability; citations and explanations supporting such statement

1. Statement

Novelty (N) Claims 1-19,22-23 YES

Claims 20 and 21 NO.

Inventive step (IS) Claims 1-19 YES

Claims 20-23 NO

Industrial applicability (IA) Claims 1-23 YES

Claims NONE NO

2. Citations and explanations:

Please See Continuation Sheet

Form PCT/ISA/237 (Box No. V) (April 2005)


International application No.PCT/US05/44656

Supplemental BoxIn case the space in any of the preceding boxes Is not sufficient.

V. 2. Citations and Explanations:Claims 1-23 meet the criteria set out in PCT Article 33(4), and thus have industrial applicability because the subject matterclaimed can be made or used in industry.

Claims 20-21 lack novelty under PCT Article 33(2) as being anticipated by Dufaux (US 6,711,857). In regard toclaim 20, Dufaux provides for means for acquiring a sequence of images including and object (col.1 line 62-col.2 line 1, notefro selecting a representative image from a video for applying face detection to a video); means for determining a robustperceptual color of the object (col.1 lines 54-60, note skin-color detection) using at least two of a pixel level analysis:(col.11lines 24-34, note skin colored pixels is used to determine the most interesting shot), a frame level analysis ( col.1 line 67-col.2 line 1, also col. 11 lines 31-34, note motion activity) and a sequence level analysis( col.1 line 67-col.2 line 1).

With regard to claim 21, Dufaux provides for a means for training a statistical classification function to identify arobust perceptual color from an observer color (col.1 lines 54-61, note learning based system).

Claim 22 lacks an inventive step under PCT Article 33(3) as being obvious over Dufaux (US 6,711,857) In view ofChasen (6,628,829). As to claim 22, Dufaux provides for compensatefor lighting conditions, then histogram equalization isperformed to compensate for differences in camera input gains and to improve contrast (col.5 lines 35-40). Dufaux does notprovide for color drift. Chasen provides for color drift (col. 8 lines 10-20) the prior art of Dufaux and Chasen are combinablebecause they are from the same field of endeavor (color correction). It would have been obvious to a person of ordinary skillin the art to incorporate the teaching of Chasen (col.8 lines 10-20), with the system and method of Dufaux, because theinvention of Chasen relates to automated analysis of a digital image of the target color to produce a closest match from aknown color database, see the background of the Invention).

Claim 23 lacks an inventive step under PCT Article 33(3) as being obvious over Dufaux (US 6,711,587) in view ofHunk ( US 5,912,980), in regard to claim 23, Dufaux provides for means for acquiring a sequence of images including andobject (col.1 line 62-col.2 line 1, note fro selecting a representative image from a video for applying face detection to avideo); means for determining a robust perceptual color of the object (col.1 lines 54-60, note skin-color detection) using atleast two of a pixel level analysis (col. 11 lines 24-34, note skin colored pixels is used to determine the most interesting shot),a frame level analysis ( col.1 line 67-col.2 line 1, also col.11 lines 31-34, note motion activity) and a sequence levelanalysis( col.1 line 67-col.2 line 1). Dufaux does not provide for means for determining the distance between a set ofobserved color data generated using an object of unknown robust perceptual color and a set of color data generated usingan object of know robust perceptual color. Hunke provides for means for determining the distance (col.13 lines 32-36, note

Form PCT/ISA/237 (Supplemental Box) (April 2005)


WRITTEN OPINION OF THE PCT/US05/44656


Supplemental BoxIn case the space in any of the preceding boxes is not sufficient.

difference measurement and the formula) between a set of observed color data generated using an object of unknown.robust perceptual color (col.13 lines 22-25, note to track a green) and a set of color data generated using an object of know

robust perceptual color (col.13 line 1, note ITCC which refer to Individual target color classifier). It would have been obviousto a person of ordinary skill in the art at time the invention was made to incorporate the teaching of Hunk with the systemand method of Dufux, because the invention of Hunk relates to the image processing in general and particularly to the

processing the Images by methods designed to detect, locate and track distinctive target objects in the image, see field ofthe invention.

Claims 1-19 meet the criteria set out in PCT Article 33(2)-(3), because the prior art does not teach or fairly suggest,for the sequence level evaluation Including analysis of a second plurality of pixels representative of the object, the second

plurality of pixels being derive from a plurality of images including the object under a variety of observation conditions. As

cited in claim 1. further the prior art of Dufaux failed to teach or suggest for using the at least a frame level analysis and a

sequence level analysis, the image processor including frame level logic, sequence level logic, and a drift matrix storageconfigured to store a color drift matrix for use in the frame level analysis. As cited in claim 14.


(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT)

(19) World Intellectual Property OrganizationInternational Bureau I 111 M11 II M11111 1111 1111 MU111111111 Ili li lIII

(43) International Publication Date PCT (10) International Publication Number

1 December 2005 (01.12.2005) WO 2005/114557 A3

(51) International Patent Classification:GO6K 9/68 (2006.01) GO6K 9/00 (2006.01)GO6K 9/70 (2006.01)

(21) International Application Number:PCT/US2005/016961

(22) International Filing Date: 13 May 2005 (13.05.2005)

(25) Filing Language: English

(26) Publication Language:,

(30) Priority Data:60/571,036 13 May 2004 (13.05.2004) US

(71) Applicant (for all designated States except US): PROX-IMEX [US/US]; 20691 Reid Lane, Saratoga, CA 95070(US).

(72) Inventor; and(75) Inventor/Applicant,(for US only): CHANG, Edward, Y.

[US/US]; 816 Dorado Drive, Santa Barbara, CA 93111(US).

(74) Agents: DURANT, Stephen, C. et al.; Morrison & Fo-erster LLP, 425 Market Street, San Francisco, CA 94105-2482 (US).

(81) Designated States (unless otherwise indicated, for everykind of national protection available): AE, AG, AL,, AM,AT, AU, AZ, BA, BB, BG, BR, BW, BY, BZ, CA, CH, CN,CO, CR, CU, CZ, DE, DK, DM, DZ, EC, EE, EG, ES, Fl,GB, GD, GE, GH, GM, HR, HU, ID, IL, IN, IS, JP, KE,KG, KM, KP, KR,.KZ, LC, LK, LR, LS, Ul, LU, LV, MA,MD, MG, MK, MN, MW, MX, MZ, NA, NG, NI, NO, NZ,OM, PG, PH, PL, PT, RO, RU, SC, SD, SE, SG, SK, SL,SM, SY, TJ, TM, TN, TR, TT, TZ, UA, UG, US, UZ, VC,VN, YU, ZA, ZM, ZW.

(84) Designated States (unless otherwise indicated, for everykind of regional protection available): ARTPO (BW, GH,GM, KE, LS, MW, MZ, NA, SD, SL, SZ, TZ, UG, ZM,ZW), Eurasian (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM),European (AT, BE, BG, CH, CY, CZ, DE, DK, EE, ES, FI,FR, GB, GR, HU, IE, IS, IT, LT, LU, MC, NL, PL, PT, RO,SE, SI, SK, TR), OAPI (BF, BJ, CF, CG, CI, CM, GA, GN,GQ, GW, ML, MR, NE, SN, TD,"'TG).

Published:Swith international search report

(88) Date of publication of the international search report:18 January 2007

For two-letter codes and other abbreviations, refer to the "Guid-ance Notes on Codes and Abbreviations" appearing at the begin-ning of each regular issue of the PCT Gazette.

In

(54) Title: MIlLTIMODAL HIGH-DIMENSIONAL DATA FUSION FOR CLASSIFICATION AND IDENTIFICATION

(57) Abstract: A method is provided for evaluating identity of an object, the method including: converting feature information

representing the object to a plurality of mathematically defined conmponents; grouping the components into multiple modalities;producing respective first prediction information for each respective modality wherein the respective prediction information for each

respective modality is based upon respective components grouped into that respective modality; and producing second prediction

information based upon the respective first prediction information produced for the multiple respective modalities.

I

0

N

O

INTERNATIONAL SEARCH REPORT International application No.

PCT/US05/16961

A. CLASSIFICATION OF SUBJECT MATTERIPC: GO6K 9168( 2006.01),9/70( 2006.01)

G06K 9/00(2006.01)

USPC: 382/115,227According to International Patent Classification (IPC) or to both national classification and IPC

B. FIELDS SEARCHED

Minimum documentation searched (classification system followed by classification symbols)U.S.: 382/115,227


Electronic data base consulted during the international search (name of data base and, where practicable, search terms used)IEEE


Category * Citation of document, with indication, where appropriate, of the relevant passages Relevant to claim No..

X TAX et al. Combining multiple classifiers by averaging or by multiplying. Pattern 1,2,12,13,20,25,26,34--- Recognition Society. 2000. Pages 1475-1485. ----.------Y 6, 11, 17,:21, 30-32

A US 2003/0169908 Al (Kim et al) 11 Sept 2003 (11,09,2003), pages 1-10 1-37

P US 2004/0136574 Al (Kozakaya et al) 15 Jul 2004 (15,07,2004), pages 1-10 1-37

A US 6,944,319 (Huang et al) 13 Sept 2005 (13,09,2005), col. 9-14 ... 1-37

SFurther documents are listed in the continuation of Box C. [] See patent family annex.

* Special categories of cited documents: "T' later document piublished after the international filing date or prioritydate and not in conflict with the application but cited to understand the

"A" document defining the general state of the art which is not considered to be of principle or theory underlying the invenitionparticular relevance

"X" document of particular relevance; the claimed invention cannot be"E" earlier application or patent published on or after the international filing date considered novel or cannot be considered to involve an inventive step

when the document is taken alone"L" document which may throw doubts on priority claim(s) or which is cited to

establish the publication date of another citation or other special reason (as "Y" document of particular relevance; the claimed invention cannot bespecified) considered to involve an inventive step when the document is

combined with one or more other such documents, such combination"O" document referring to an oral disclosure, use, exhibition or other means being obvious to a person skilled in the art

"P" document published prior to the international filing date but later than the "&" document member of the same patent familypriority date claimed

Date of the actual completion of the international search Date of mail 7 ch report

09 August 2006 (09.08.2006)Name and mailing address of the ISA/US Authorized officer

Mail Stop PCT, Attn: ISAfUSCommissioner for Patents Jingge Wu

P.O. Box 1450Alexandria, Virginia 22313-1450 Telephone No. 703-306-0377

Facsimile No. (571) 273-3201

Form PCT/ISA/210 (second sheet) (April 2005)

From theINTERNATIONAL SEARCHING AUTHORITY

To:STEPHEN C. DURANTMORRISON & FOERSTER LLP425 MARKET STREETSAN FRANCISCO, CA 94105-2482

PATENT COOPERATION TREATY

PCT,WRITTEN OPINION OF THE


(PCT Rule 43bis.1)

Date of mailing > 1 0 DT 20Id/month/year) 17OC LApplicant's or agent's file reference FOR FURTHER ACTION

577832000140 See paragraph 2 below


PCT/USO5/16961 13 May 2005 (13.05.2005) 13 May 2004 (13.05.2004)

International Patent Classification (IPC) or both national 'classification and IPC

IPC: G06K 9/68( 2006.01),9/70(2006.01) G06K 9/00( 2006.01)USPC: 382/115,227Applicant

PROXIMEX ,


® Box No. I Basis of the opinion

Box No. II Priority

SBox No. III Non-establishment of opinion with regard to n

SBox No. IV Lack of unity of invention

® Box No. V Reasoned statement under Rule 43bis.1(a)(i) wapplicability; citations and explanations suppor

D Box No. VI Certain documents cited

D Box No. VII Certain defects in the international application

Z Box No. VIII Certain observations on the international appli

ovelty, inventive step and industrial applicability

'ith regard to novelty, inventive step or industrialting siuch statement

cation

2. FURTHER ACTION

If a demand for international preliminary examination is made, this opinion will be considered to be a written opinion of the

International Preliminary Examining Authority ("IPEA") except that this does not apply where the applicant chooses an

Authority other than this one to be the IPEA and the chosen IPEA has notified the International Bureau under Rule 66. 1bis(b)that written opinions of this International Searching Authority will not be so considered.

If this opinion is, as provided above, considered to be a written opinion of the IPEA, the applicant is invited' to submit to the

IPEA a written reply together, where appropriate, with amendments, before the expiration of 3 months from the date of mailing

of Form PCT/ISA/220 or before the expiration of 22 months from the priority date, whichever expires later.


3. For further details, see notes to Form PCT/ISA/220.

Name and mailing address of the ISA] US Date of completion of this opinion Authorized office.

Mail Stop PCT, Attn: ISA/US i/ .Commissioner for Patents 14 August 2006 (14.08.2006) Jingge Wu

P.O. Box 1450Alexandria, Virginia 22313-1450 Telephone No. 703-306-0377 .

Facsimile No. (571) 273-3201

Form PCT/ISA/237 (cover sheet) (April 2005)

INTERN SEARC.I AUTHORNTY

WRITTEN OPINION OF THE International application No.

INTERNATIONAL SEARCHING AUTHORITY PCTI/USO5/16961


1. With regard to the language, this opinion has been established on the basis of:

Sthe international application in the language in which it was filed

OI a translation of the international application into , which is the language of a translation furnished for the purposes ofinternational search (Rules 12.3(a) and 23.1(b)).

2. With regard to any nucleotide and/or amino acid sequence disclosed in the international application and necessary to the claimedinvention, this opinion has been established on the basis of:

a. type of material

O a sequence listing

j table(s) related to the sequence listing


D on paper



L contained in the international application as filed.

D filed together with the international application in electronic form.

L furnished subsequently to this Authority for the purposes of search.

3. Q In addition, in the case that more than one version or copy of a sequence listing and/or table(s) relating thereto has been filedor furnished, the required statements that the information in the subsequent or additional copies is identical to that in theapplication as filed or does not go beyond the application as filed, as appropriate, were furnished.


Form PCT/ISA/237(Box No. I) (April 2005)_ _ __ _ ______

j

Caims PeaseSee CoInternational application No. 'WRITTEN OPINION OF THE PCT/US05/16961


Box No. V Reasoned statement under Rule 43 bis.1(a)(i) with regard to novelty, inventive step or industrial .applicability; citations and explanations supporting such statement

1. Statement

Novelty (N) Claims Please See Continuation Sheet YESClaims Please See Continuation Sheet NO

Inventive step (IS) Claims Please See Continuation Sheet, YESClaims Please See Continuation Sheet NO

Industrial applicability (IA) Claims Please See Continuation Sheet YESClaims Please See Continuation Sheet NO


Please See Continuation Sheet

Form PCT/ISA/237 (Box No. V) (April 2005)


INTERNATIONAL SEARCHING AUTHORITY PCT/US05/16961

Form ?.tfISA/2iI3 (BIOX NO. yInI) (ApriI LU0)

Box No. VIII Certain observations on the international application

The following observations on the clarity of the claims, description, and drawings or on the questions whether the claims are fullysupported by the description, are made:

The drawings are objected to under PCT Rule 66.2(a)(iii) as containing the following defect(s) in the form or content thereof: Thedrawings are objected to because they are of insufficient quality for publication.

Claims 14, 22 are objected to under PCT Rule 66.2(a)(v) as lacking clarity under PCT Article 6 because claims 14 and 22 are indefinitefor the following reason(s):

Referring to claim 14, the phrase "the returned previously stored modality vectors" in line 8 lacks antecedent basis. It appearsthat the applicant intended the phrase to read "the located previously stored modality vectors." Appropriate correction is required.

Referring to claim 22, the phrase "means for producing second prediction information" is indefinite because there is norecitation of producing "first prediction information".

"^


WRITTEN OPINION OF THE PCT/US05/16961



V.1. Reasoned Statements:The opinion as to Novelty was positive (Yes)with respect to claims 3-11, 14-19, 21-24, 27-33, 35-37The opinion as to Novelty was negative (No) with respect to claims 1, 2, 12, 13, 20, 25, 26, 34The opinion as to Inventive Step was positive (Yes)with respect to claims 3-5, 7-10, 14-16, 18-19, 22-24, 27-29, 33, 36-37The opinion as to Inventive Step was negative(NO) with respect to claims 1, 2, 6, 11, 12, 13, 17, 20, 21, 25, 26, 30-32, 34, 35The opinion as to Industrial Applicability was positive (YES) with respect to claims 1-37The opinion as to Industrial Applicability was negative(NO) with respect to claims NONEI

V. 2. Citations and Explanations:Claims 1, 2, 12, 13, 20, 25, 26, 34 lack novelty under PCT Article 33(2) as being anticipated by the article entitled "Combining multipleclassifiers by averaging or by multiplying?" by Tax et al. ("Tax").

Referring to claim 1, Tax discloses a method of evaluating identity of an object comprising:converting feature information representing the object to a plurality of mathematically defined components (feature sets)

[pages 1479-1480, section 5.1]; !grouping the components into multiple modalities [page 1480];/producing respective first prediction information for each respective modality wherein the respective prediction information for

each respective modality is based upon respective components grouped into that respective modality (page 1481, section 5.2); andproducing second prediction information based upon the respective first prediction information produced for the multiple

respective modalities (pages 1476-1479, sections 2-4 and pages 1481-1482, section 5.2. Note that the product combination of theclassifiers produces second prediction information).

Referring to claim 2, Tax further discloses that corrverting involves mapping the feature information to components in amathematically different space from the original feature space (pages 1480-1481).

Referring to claim 12, Tax further discloses that producing respective first prediction information involves producingrespective first classification prediction information for each respective modality (page 1481, section 5.2).

.Referring to claim 13, Tax further discloses that that producing respective first prediction information involves producingrespective first classification prediction information for each respective modality (page 1481, section 5.2), and producing secondprediction information involves producing second classification prediction information based upon the respective first classificationprediction information (pages 1476-1479, sections 2-4 and pages 1481-1482, section 5.2).

Referring to claim 20, Tax discloses a system for evaluating identity of an object comprising:means for converting feature information representing the object to a plurality of mathematically defined components (feature

sets) [pages 1479-1480, section 5.1]means for grouping the components into multiple modalities [page 1480];multiple first classifiers, each associated with modality, each producing respective first classification prediction information for

each respective modality, based upon respective components grouped into that respective modality (page 1481, section 5:2); anda second classifier producing second classification prediction information based upon a non-linear combination of the

respective first classification prediction information produced for the multiple respective modalities (pages 1476-1479, sections 2-4 andpages 1481-1482, section 5.2. Note that the product-combination of the classifiers produces second prediction information based on anon-linear combination ofrescleetive first classification prediction information)


International application No.WRITTEN OPINION OF THE PCT/USO5/16961



Referring to claim 25, see the discussion in view of Tax of at least claim 1 above.Referring to claim 26, see the discussion in view of Tax of at least claim 2 above.Referring to claim 34, see the discussion of at least claim 20 above.

Claims 6, I11, 17, 21, 30-32, 35 lack an inventive step under PCT Article 33(3) as being obvious over the article entitled "Combiningmultiple classifiers by averaging or by multiplying?" by Tax et al. ("Tax") in view of applicant's admission of known prior art("Admission").

Referring to claim 6, Tax does not explicitly disclose that grouping the components involves ensuring that the total dimensionswithin individual modalities is below a prescribed threshold based upon the curse of dimensionality. However, this feature wasexceedingly well known in the art. For example, Admission discloses grouping of components that involves ensuring that the totaldimensions within individual modalities is below a prescribed threshold (20 to 30) based upon the curse of dimensionality (page 13,paragraph 50).

Tax and Admission are combinable because they are both concerned with multi-level classifiers. At the time of the invention,it would have been obvious to a person of ordinary skill in the art to modify Tax in view of Admission. The suggestion/motivation fordoing so would have been to reduce the effects due to the curse of dimensionality (Admission, page 13, paragraph 50). Therefore, itwould have been obvious to combine Tax with Admission to obtain the invention as specified in claim 6.

Referring to claim 11, Tax further discloses that the grouping involves grouping the components into multiple modalities so asto minimize correlation among modalities (page 1481, last paragraph on the left column. Note that modalities comprise independent datasets).

Tax does not explicitly disclose that grouping the components involves ensuring that the total dimensions within individualmodalities is below a prescribed threshold based upon the curse of dimensionality. However, this feature was exceedingly well known inthe art. For example, Admission discloses grouping of components that involves ensuring that the total dimensions within individualmodalities is below a prescribed threshold (20 to 30) based upon the curse of dimensionality (page 13, paragraph 50).

Tax and Admission are combinable because they are both concerned with multi-level classifiers. At the time of the invention,it would have been obvious to a person of ordinary skill in the art to modify Tax in view of Admission. The suggestion/motivation fordoing so would have been to reduce the effects due to the curse of dimensionality (page 13, paragraph 50). Therefore, it would havebeen obvious to combine Tax with Admission to obtain the invention as specified in claim 11.

Referring to claim 17, Tax discloses a method of evaluating identity of an object comprising:converting feature information representing the object to a plurality of mathematically defined components (feature sets)

[pages 1479-1480, section 5.1];grouping the components into multiple modalities so as to minimize correlation among modalities [page 1480-1481];producing respective first prediction information for each respective modality wherein the respective prediction information for

each respective modality is based upon respective components grouped into that respective modality (page 1481, section 5:2); andproducing second classification prediction information based upon a non-linear combination of the respective first

classification prediction information produced for the multiple respective modalities (pages 1476-1479, sections 2-4 and pages 1481-

1482, section 5.2. Note that the product combination of the classifiers produces second prediction information that is a non-linearcombination of the respective first classification prediction information).

Tax does not explicitly disclose that grouping the components involves limiting the dimensions within individual modalities is

below a prescribed threshold based upon the curse of dimensionality. However, this feature was exceedingly well lkniown in the art. For

example, Admission discloses grouping of components that involves limiting dimensions within individual modalities is below a

prescribed threshold (20 to 30) based upon the curse of dimensionality (page 13, paragraph 50). ,

Tax and Admission are combinable because they are both concerned with multi-level classifiers. At the time of the invention,it would have been obvious to a person of ordinary skill in the art to modify Tax in view of Admission. The suggestion/motivation fordoing so would have been to reduce the effects due to the curse of dimensionality (page 13, paragraph 50). Therefore, it would have

been obvious to combine Tax with Admission to obtain the invention as specified in claim 17.Referring to claim 21, see the discussion of at least claim 6 above.Referring to claim 31, see the discussion of at least claim 11 above.Referring to claim 32, see the discussion of at least claim 17 above.Referring to claim 35, see the discussion of at least claim 17 above.


(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION.TREATY (PCT)

(19) World Intellectual Property OrganizationInternational Bureau "I ll II 1l1 ll Illl l Ill1ll II ll l M l Il

(43) International Publication Date (10) International Publication Number

30 March 2006 (30.03.2006) PCT ;WO 2006/034135 A3

(51) International Patent Classification:GO6K 9/00 (2006.01) HO04N 7/18 (2006.01)..GO6K 9/64 (2006.01) 110H4B 17/00 (2006.01)HO4N 9/47 (2006.01)

(21) International Application NumberCPCT/US2005/033378

(22) International Filing Date:19 September 2005 (19.09.2005)

(25) Filing Language:

(26) Publication Language:

(30) Priority Data:60/610,998

English

English

17 September 2004 (17.09.2004) US

(71) Applicant (for all designated States except US): PROX-IMEX [US/US]; 20691 Reid Lane, Saratoga, CA 95070

(US).

(72)(75)

Inventors; andInventors/Applicants (for US only): CHENG, Ken,Prayoon [US/US]; 20691 Reid Lane, Saratoga, CA 95070

(US). CHANG, Edward, Y. [IS/US]; 816 Dorado Drive,Santa Barbara, CA 93111 (US). WANG, Yuan, Fang[US/US]; 5849 Via Fiori Lane, Goleta, CA 93117 (US).

(54) Title: ADAPTIVELANCE SYSTEM

(74) Agents: DURANT, Stephen, C. ct a].; Morrison & Focr-ster LLP, 425 Market Street, San Francisco, Ca 94105-2482(US).,

(81) Designated States (unless otherwise indicated, for everykind of national protection available): AE, AG, AL, AM,AT, AU, AZ, BA, BB, BG, BR, BW, BY, BZ, CA, CH, CN,CO, CR, CU, CZ, DE, DK, DM, DZ, EC, EE, EG, ES, FI,

GB, GD, GE, GH, GM, HR, HU, ID, IL, IN, IS, JP, KE,KG, KM, KP,;KR, KZ, LC, LK, LR, LS, LT, LU, LV, LY,MA, MD, MG, MK, MN, MW, MX, MZ, NA, NG, NI, NO,NZ, OM, PG, PH, PL, PT, RO, RU, SC, SD, SE, SG, SK,

SL, SM, SY, TJ, TM, TN, TR, TT, TZ, UA, UG, US, UZ,VC, VN, YIJ, ZA, ZM, ZW..

(84) Designated States (unless otherwise indicated, for every

kind of regional protection available): AR1PO (BW, GH,GM, KE, 1LS, MW, MZ, NA, SD;. SI,, SZ, TZ, 110, ZM,

ZW), Eurasian (AM, AZ,-BY, KG, KZ, MD, RU, TJ, TM),European (AT, BE, BG, CH, CY, CZ, DE, DK, EE, ES, Fl;

FR, GB, GR, HU, IE, IS, IT, LT, LU, LV, MC, NL, PL, PT,

RO, SE, SI, SK, TR), QAPI (BF, BJ, CF, CG, CI, CM, GA,.GN, GQ, GW, ML, MR,. NE, SN, TD, TG).

Published:- with international search report

(88) Date of publication of the International search report:13 July 2006

[Continued on next page]

MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEIL-. ,.

Configure Maps (Environment Admin)

"M wl, r-. .. .. ..fr~mt d o n pa '" '

()

0

to

M

M

m

mmm

O

mmm

m

(57) Abstract: Asurveillance system is provided that includes at least one sensor disposed in a security area of a surveillance

region to sense an occurrence of a potential security breach event; a plurality of cameras is disposed in the surveillance region; at

least one camera of the plurality has a view of the security area and can be configured to automatically gather biometric information

concerning at least one subject person in the vicinity of the security area in response to the sensing of a potential security breach

event; one or more other of the plurality of cameras can be configured to search for the at least one subject person; a processing

system is programmed to produce a subject dossier corresponding to the at least one subject person to match biometric information of

one or more persons captured by one or more of the other cameras with corresponding biometric information in the subject dossier.

W O 2006/034135 A3 IIIll11111IIIIIIIIIIIIIIIIIIIIIIIII III IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII

For two-letter codes and other abbreviations, refer to the "Guid-ance Notes on Codes and Abbreviations" appearing at the begin-ning of each regular issue of the PCT Gazette.


PCT/US05/33378

A. CLASSIFICATION OF SUBJECT MATFTERIPC: GO6K 9/00( 2006.01),9/64('2006.01I);HO4N 9/47( 2006.01),7/18( 2006.0 1);HO4B 17/00( 2006.01)

USPC: 382/115-118,124;348/143-161;455/67.12 'According to International Patent Classification (IPC) or to both national classification and IPC

B. FIELDS SEARCHED .

Minimum documentation searched (classification system followed by classification symbols)U.S.: 3821115-118,124;348/143-161;455/67.12





X US 2002/0190119 Al (HUFFMAN) 19 December 2002 (19.12.2002), entire document 1-20

X US 2004/0081338 Al (TAKENAKA) 29 April 2004 (29.4.2004), entire document 1-20

X US 6,697,103 BI (FERNANDEZ et al) 24 February 2004 (24.02.2004), column 4 lines 23- I, 17, 20

67, column 6 lines 1-30 Figures 1 and 3-4.

I, .mE

- Further documents are listed in the continuation of Box C. [E See patent family annex.

* Special categories of cited documents: "T" later document published after the intemrnational filing date or prioritydate and not in conflict with the application but cited to understand the

"A" document defining the general state of the art which is not considered to be of principle or theory underlying the invention

particular relevance"X" document ofparticular relevance; the claimed invention cannot be

"E" earlier application or patent published on or after the international filing date considered novel or cannot be considered to involve an inventive stepwhen the document is taken alone

"L" document which may throw doubts on priority claim(s) or which is'cited toestablish the publication date of another citation or other special reason (as "Y' document of particular relevance; the claimed irivention cannot be

specified) considered to involve an inventive step when the document is combinedwith one or more other such documents, such combination being

"0" document referring to an oral disclosure, use, exhibition or other means obvious to a person skilled in the art

"P" document published prior to the intemrnational filing date btit later than the "&" document member of the same patent family

priority date claimed

Date of the actual completion of the international search Date of mailing of the nt tlp s port

13 March 2006 (13.03.2006)

Name and mailing address of the ISA/US Authorized officer

Mail Stop PCT, Attn: ISA/US Jingge WuCommissioner for PatentsP.O. Box 1450 TlpoeN.512276Alexandria, Virginia 22313-1450 Telephone No. 571-272-7361

Facsimile No. (571) 273-3201

Form PCT/ISA/210 (second sheet) (April 2005)I \ ' , ' -



To:STEPHEN C. DURANTMORRISON& FOERSTER LLP425 MARKET STREETSAN FRANCISCO, CA 94105-2482


577832000240 ' See paragraph 2 below


PCT/US05/33378 19 September 2005 (19.09.2005) 17 September 2004 (17.09.2004 )

International Patent Classification (IPC) or both national classification and IPC

IPC: Please See Continuation SheetUSPC: 382/115-118,124;348/143-161;455/67.12 ,Applicant

PROXIMEX


"C,"WRITTEN OPINION OF THE


(PCT Rule 43bis.1)

Date of mailingday month/ ear 2 6.APR 2006.


Z Box No. I Basis of the opinion

D Box No. II Priority

Box No. III Non-establishment of-opinion with regard to novelty, inventive step and industrial applicability

. j Box No. IV Lack of unity of invention

Z Box No. V Reasoned statement under Rule 43bis.I (a)(i) with regard to novelty, inventive step or industrialapplicability; citations and explanations supporting such statement

SBox No. VI Certain documents cited

Box No. VII Certain defects in the international application

Box No. VIII Certain observations on the international application

2. FURTHER ACTION

If a demand, for international preliminary examination is made, this opinion will be considered to be a written opinion of theInternational Preliminary Examining Authority ("IPEA") except that this does not apply where the applicant chooses anAuthority other than this one to be the IPEA and the chosen IPEA has notified the International Bureau under Rule 66.1bis(b)that written opinions of this International Searching Authority will not be so considered.

If this opinion is, as provided above, considered to be a written opinion of the IPEA, the applicant is invited to submit to theIPEA a written reply together, where appropriate, with amendments,, before the expiration of 3 months from the date of mailingof Form PCT/ISA/220 or before the expiration of 22 months from the priority date, whichever expires later.



Name and mailing address of the ISA/ US Date of completion of this opinion Authorized officerMail Stop PCT, Attn: ISA/USCommissioner for Patents 13 March 2006 (13.03.2006) Jingge WuP.O. Box 1450Alexandria, Virginia 22313-1450 Telephone No. 571-272-7361

Facsimile No. (571) 273-3201

__



PCT/US05/33378

I

Li furnished subsequently to this Authority for the purposes of search.

3. L In addition, in the case that more than one version or copy of a sequence listing and/or table(s) relating thereto has been filedor furnished, the required statements that the information in the subsequent or additional copies is identical to that in theapplication as filed or does not go beyond the application as filed, as appropriate, were furnished.


Form PCT/ISA/237(Box No. I) (April 2005)


1. With regard to the language, this opinion has beeh established on the basis of:

Sthe international application in. the language in which it was filed

Li- a translation of the international application into ____, which is the language of a translation furnished for the purposes ofinternational search (Rules 12.3(a) and 23.1(b)).


a. type of material

L a sequence listing

O- table(s) related to the sequence listing

b. format of material '

L on paper

O in electronic form


O- contained in the international application as filed.

O filed together with the international application in electronic form.

International application No.WRITTEN OPINION OF THE PCT/US05/33378


Box No. V Reasoned statement under Rule 43 bis.1(a)(i) with regard to novelty, Inventive step or industrial

applicability; citations and explanations supporting such statement

1. Statement

Novelty (N) Claims NONE YES

Claims 1-20 NO

Inventive step (IS) Claims NONE YES

Claims 1-20 NO

Industrial applicability (IA) Claims 1-20 YES

Claims NONE NO


Claims 1-20 lack the novelty under PCT Article 33(2) as being anticipated by Huffman (US 2002/0190119 Al).

Huffman discloses claims 1-20. For example, Huffman discloses a surveillance method comprising, using at least one eventsensor disposed in a security area of a surveillance region to sense an occurrence of potential security breach event (page 2 paragraphs17, 18 and 21); using at least one camera with a view of the security area in which the event is sensed to gather biometric informationconcerning at least one person in the vicinity of the security area at about the time of the sensing of the event (Figures 2 and 3);producing a subject dossier corresponding to the at least one person (page 2, paragraph 22); and matching biometric information of oneor more persons captured by one or more other cameras in the vicinity of the at leas tone camera with corresponding biometricinformation in the subject dossier (page 3 paragraphs 32 and 33).

Form PCT/1SA/237 (Box No. V) (April 2005). . . . . . . ... . . .. ... . . .. A ^ ^ .


International application No.PCT/US05/33378


Continuatioi of IPC:G06K 9/00( 2006.01),9/64( 2006.0 1);HO4N 9/47( 2006.01),7/18( 2006.01);HO4B 17/00( 2006.01)


I ' '

(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT)

(19) World Intellectual Property Organization V III 1 11 V V III III I I V I III IIIInternational Bura l l l Ml llcato licIainub

(43) International Publication Date PCT (10) International Publication Number

2 1 eptemer 200 2O 1.097.22006)

(51) International Patent Classification:GO6N 3/00 (2006.01) GO6N 3/08 (2006.01)GO6N 3/063 (2006.01) GO6N 3/04 (2006.01)

(21) International Application Number:PCT/US2005/033750

(22) International Filing Da




19 September 2005 (19.09.2005)

English

"English

17 September 2004 (17.09.2004)

S (71) Applicant (for all designated States except US): PROX-IMEX [US/US]; 20691 Reid Lane, Saratoga, CA 95070

- (US).

(72). Inventor; and(75) Inventor/Applicant (for US only): WANG, Yuan Fuang

[US/US]; 5849 Via Fiori Lane, Golta, CA 93117 (US).

(74) Agents: DURANT, Stephen, C. et al.; Morrison & Fo-

erster LLP, 425 Market Street, San Francisco, CA 94105-2482 (US).

C ,

(81) Designated States (unless otherwise indicated, for everykind of national protection available): AE, AG, AL, AM,AT, AU, AZ, BA, BB, BG, BR, BW, BY, BZ, CA, CH, CN,CO, CR, CU, CZ, DE, DK, DM, DZ, EC, EE, EG, ES, FI,GB, GD, GE, GH, GM, HR, HU, ID, IL, IN, IS, JP, KE,KG, KM, KP, KR, KZ, LC, LK, LR, LS, LT, LU, LV, LY,MA, MD, MG, MK, MN, MW, MX, MZ, NA, NG, NI, NO,NZ, OM, PG, PH, PL, PT, RO, RU, SC, SD, SE, SG, SK,SL, SM, SY, TJ, TM, TN, TR, TT, TZ, UA, UG, US, UZ,VC, VN, YU, ZA, ZM, ZW.

(84) Designated States (unless otherwise indicated, for everykind of regional jprotection available): ARTPO (BW, GH,GM, KE, LS, MW, MZ, NA, SD, SL, SZ, TZ, UG, ZM,ZW), Eurasian (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM),European (AT, BE, BG, CH, CY, CZ, DE, DK, EE, ES, Fl,FR, GB, GR, HU, IE, IS, IT, LT, LU, LV, MC, NL, PL, PT,RO, SE,,SI, SK,TR), OAPI (BF, BJ, CF, CG, CI, CM, GA,GN, GQ, GW, ML, MR, NE, SN, TD, TG).

Published:- with international search report

(88) Date of publication of the international search report:26 July 2007

For two-letter codes and other abbreviations, refer to.the "Guid-

ance Notes on Codes and Abbreviations" appearing at the begin-

ning of each regular issue of the PCT Gazette.

(54) Title: INCREMENTAL DATA FUSION AND DECISION MAKING SYSTEM AND ASSOCIATED METHOD

(57) Abstract: A computer implemented (Figure 7) adaptive ensemble classifier is provided which includes: a plurality classifiers

(Figure 1, Ddf); a decision structure that maps respective classifier combinations to respective classification decision results (Figure

5C, 512); and a plurality of respective sets of weights associated with respective classifier combinations.

0

WO 2006/098766 A31)

3THNATMONAL BEARCI[ IMP'ORfT ttatitanm appl1oation No.

PCD'V805/.33750

AL. CiLASSIOA ON OF SUBJsCT WiA'fI'DIP: G6N /0( 2006.01 ),3/0 3( 2006.01 ),3/08( 2006.01),3%04( 200b. 61)

USPC: 706/2;706141;706/20;706/26Accrding to IntDemational Patent Chmi on~t (W2C) or to bot natloaai clasLcaiion and IPC

Minimum acgwrontatiois asaehd (alassotion eyatem followed by classificaton symbols)U.S.: 706125;706/41;706120;7D6I26

Dcaauniorrttidon searched other than Mmtt wum docurnutaon to 'Cho extent that =u6 dcumeants are Icluded in the f1olde searchedUJS latent datubaeo; IRI

Elchtso data bas econaulted duding the inmtional uessac anv of data us amud, why eprwiouble, sarclx t c ed)IBS

C, DOCUM ITS CONSWFAUED TO BE REtLSVA)Xf''1

Category " Mar=io of duct mrsnt4 with jcndAcat luu, whore approprilote, of tbarelevant pssages Relevant to claim N.

A US 5.4o-,574 A(GLMR~ eC al.) 26 D~ecember 1995 (2?. I 25995) 2column 2, line 29 to ooltwint 3, fine;

A, US 5,70t398 A(Gl cc aL) 23 oembeT 1997 (23.12.1997)12column 2, lino 29 to colur a 3, lime 5

A US 6,757,668 B1 A(COSBEL ticof.) 29 June 2004 (29?.06.2004), 2columnt 3, litres 28-43

A US 6,248,063 B i &(DARNHILL ctel.) 19 June 20091 (19.06,2001)>-0colutmn 7. line 39 to column, 9, tins 57

A, US 5,769,074 A(BARNMEL of a.) 23 Ju~ne 1998'(23.06.1~98) 1-20columnt 7, 39 to colurem 9, line 59

A U'S 6,306,087131 A(BAM L Let al.) 23 October 2001 (23,10.2011) 1-20column 3,9 to oolurnn 9, line 35

El Fu rtlicr doouztn na tare listed In th coud auuion df ox a. Q see platen lniy-azmex." 8Aa aatoeruia oafcca dSofooti~T seen Wr do lditt ntn a 'be MMtunadouul fiffnj dat e orptcty

A' ioctnen cta~~lbsganiultut of~ u~ wlohtn aom onadaid t hadata anid not to wtflhetvk Me~ Ia pplooettou btm shadi to dund i = the"A" dooamnt d tiuWSthe garotel cat* otbo ar wpiah iy ea or idmd to o of udnolpl or (Ity Mdybe thlfamda

°x" dount 'taliotlom~tlavanoot tboulaitrwt {itrtsNront catutl t°V" erlier apliton erpiat4t 5natahed en oratorlis iznt lonal Olhng daft e eonsmdet tanbn be cos werd to Involve an fzwunttw stop hIS mc

"L" dgounise( wilkb uuy lIsow daubo an pdadWt eldttt(o) or WWhr6 Is aoW ed icatublivh th lmhoua dace caeolhr dchodan or er spuot moao (a "'Y duuant1~ of utttoulnfoteevuste de; 4bm clafoind nid otmmt be

xpeoiiescadod to InvolVe an Iavasdvo atop When the damuitan iIs aominadwlth oite nr morm Qtheruplh 4*auibnum nwh c mbntlou bt io

"O)" de nma malng to an otal daleauro. use, o d~blhn or odiemflleamz abvloosa too ia omt s~lkd in the nit1111 damunnt pbialled pior to Ows Intosnclloaal fiingdsic but IOloIU n m &". doatizat MlO4Umr ofrthe sap aatIuil

Data of the uiuual aonitpIn of the intemauuzicnl search. Dato of mnelding of t~le iztern~ianal semni report

29 August 2005 (29.08.200t5)1)2f YNatru-w and tnailing address ofithe MWS u itho caro ~ I' '

Cmsirp ACT, PAnntSACmiSione fCo Patens Anhn KughP.O. Box L4$0 C7)Alamudia, Vicrtnl2313-140O 'Talapnolii No. (0)3830

Facsimnile No. (571) 273-3201__________________________F~onn PCTL/ISA/210 (second sheat) (Apri 2005)

PATENT COOPERATION TREATYFrom theINTERNATIONAL SEARCHING AUTHORITY

To:STEPHEN C. DURANTMORRISON & FOERSTER LLP425 MARKET STREETSAN FRANCISCO, CA 94105-2482

PCTWRITTEN OPINION OF THE


(PCT Rule 43bis.1)

Date of mailing,da /month/ ear



SBox No. I Basis of the opinion

SBox No. 11. Priority

D Box No. III Non-establishment of opinion with regard to novelty, inventive step and industrial applicability

Box No. IV Lack of unity of invention

Z Box No. V Reasoned statement under Rule 43bis. 1 (a)(i) with regard to novelty, inventive step or industrialapplicability; citations and explanations supporting such statement

D Box No. VI Certain documents cited

Box No. VII Certain defects in the international application

D Box No. VIII Certain observations on the international application

2. FURTHER ACTIONIf a demand for international preliminary examination is made, this opinion will be considered to be a written opinion of theInternational Preliminary Examining Authority ("IPEA") except that this does not apply where the applicant chooses anAuthority other than this one to be the IPEA and the chosen IPEA has notified the International Bureau under Rule 66.1 bis(b)that written opinions of this International Searching Authority will not be so considered.

If this opinion is, as provided above, considered to be a written opinion of the IPEA, the applicant is invited to submit to theIPEA a written reply together, where appropriate, with amendments, before the expiration of 3 months from the date of mailingof Form PCT/ISA/220 or before the expiration of 22 months from the priority date, whichever expires later.For further options, see Form PCT/ISA/220.


I

TO: "

STEPHEN C. DURANT

MORRISON & FOERSTER

LLP

425 MARKET

STREET

SAN FRANCISCO,

CA 94105-2482

Applicant's or agent's file reference FOR FURTHER ACTION ' U577832000340 See paragraph 2 below

577832000340International application No. International filing date (day/month/year) Priority date (day/month/year)

PCT/US05/33750 19 September 2005 (19.09.2005) 17 September 2004 (17.09.2004)International Patent Classification (IPC) or both national classification and IPC

IPC: GO6N 3/00(2006.01);GO6N 3/063( 2006.01);GO6N 3/08( 2006.01);GO6N 3/04(2006.01)USPC: 706/25;706/41';706/20;706/26Applicant

PROXIMEX- -- :

Name and mailing address of the ISA/ USMail Stop PCT, Attn: ISA/USCommissioner for PatentsP.O. Box 1450Alexandria, Virginia 22313-1450

Facsile No. (571) 273-3201



PCT/US05/33750

1. With regard to the i'nguage, this opinion has been established on the basis of:

, the international application in the language in which it was filed

[I a translation of the international application into _ which is the laninternational search (Rules 12.3(a) and 23.1(b)).

guage of a translation furnished for the purposes of


a. type of material

- a sequence listing

Stable(s) related to the sequence listing


O ' on paper


c.time. of filing/furnishing

[I contained in the international.application as filed.

Q filed tbgether with the international application in electronic form.

Q" furnished subsequently to this Authority for the purposes of search.

3. [- In addition, in the case that more than one version or copy of a sequence listing and/or table(s) relating thereto has been filedor furnished, the required statements that the information in the subsequent or additional copies is identical to that in theapplication as filed or does not go beyond the application as filed, as appropriate, were furnished.


Form PCT/ISA/237(Box No. I) (April 2005)



International application No.PCTIUS05/33750

Box No. V Reasoned statement under Rule 43 bis.1(a)(i) with regard to novelty, inventive step or industrialapplicability; citations and explanations supporting such statement

1. Statement

Claims 1-20 YESClaims NONE

Inventive step (IS) Claims 1-20 YESClaims NONE

7'

Industrial applicability (IA) Claims 1-20

Claims NONE


Claims 1-20 meets the criteria set forth in PCT Article 33(2-4) whereby the prior art does not explicitly teach or render obviousapplicant's claimed invention. Specifically, a computer implemented adaptive ensemble classifier is provided which includes: a pluralityof classifiers; a decision structure that maps respective classifier combinations to respective classification decision results; and a pluralityof respective sets of weights associated with respective classifier combinations.

Form rLu11IA/Z3 / (tOX NO. V) (April 2005)

Novelty (N)

YES

NO

NO

NO

r ~1 1-~-_ -___-_ -

I

-~~~--- NOrINO

(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION T'REATY (PCT)

(19) World Intellectual Property Organization

International Bureau IIl 111 I111111 11 11111 11111 liIll 1111I 11111 11111 11111 11111 11111li1111111 11111111) Iii

(43) International Publication Date (10) International Publication Number

3 May 2007 (03.05.2007) PCT WO 2007/050104 A3

(51) International Patent Classification:11HO4N 7/18 (2006.01)

(21) International Application Number:PCT/ JS2005/043808

(22) International Filing Dat




4 December 2005 (04.12.2005)

English

English

4 December 2004 (04.12.2004) US

(71) Applicant (for all designated States except US): PROX-IMEX CORPORATION [IJS/1JS]; 6 Reuslts Way, Cuper-tino, CA 95014 (US).

(72) Inventors: WANG, Yuan-Fang; 5849 Via Fiori Lane,Goleta, CA 93117 (US). CHANG, Edward; 816 DoradoDrive, Santa Barbara, CA 93111 (US). CHENG, Ken,Prayoon; 20691 Reid Lane, Saratoga, CA 95070 (US).

(74) Agent: LIN, Bo-in; 13445 Mandoli Drive, Los Altos Hills,CA 94022 (US).

(81) Designated States (unless otherwise indicated, for everykind of national protection available): AE, AG, AL, AM,

AT, AU, AZ, BA, BB, BG, BR, BW, BY, BZ, CA, CH, CN,CO, CR, CU, CZ, DE, DK, DM, DZ, EC, EE, EG, ES, FI,GB, GD, GE, GH, GM, HR, HU, ID, IL, IN, IS, JP, KE,KG, KM, KN, KP, KR, KZ,"LC, LK, LR, LS, LT, LU, LV,LY, MA, MD, MG, MK, MN, MW, MX, MZ, NA, NG, NI,NO, NZ, OM, PG, PH, PL, PT, RO, RU, SC, SD, SE, SG,SK, SL, SM, SY, TJ, TM, TN, TR, TT, TZ, UA, UG, US,UZ, VC, VN, YU, ZA, ZM, ZW.

(84) Designated States (unless otherwise indicated, for everykind of regional protection available): ARIPO (BW, GH,GM, KE, LS, MW, MZ, NA, SD, SL, SZ, TZ, UG, ZM,ZW), Eurasian (AM,AZ, BY, KG, KZ, MD, RIJ, T.1, TM),European (Al', BE, BG, CH, CY, CZ, DE, DK, EE, ES, Fl,FR, GB, GR, HU, IE, IS, IT, LT, LU, LV, MC, NL, PL, PT,RO, SE, SI, SK, TR), OAPI (BF, BJ, CF, CG, CI, CM, GA,GN, GQ, GW, Ml,, MR, NE, SN, TD, TG).

Published:- with international search report- before the expiration of the time limit for amending the

claims and to be republished in the event of receipt of

amendments

(88) Date of publication of the international search report:13 December 2007

For two-letter codes and other abbreviations, refer to the "Guid-

ance Notes on Codes and Abbreviations" appearing at the begin-

ning of each regular issue of the PCT Gazette.

(54) Title: VIDEO SURVEILLANCE USING STATIONARY-DYNAMIC CAMERA ASSEMBLIES

(57) Abstract: A video surveillance system includes multiple video cameras. The surveillance system is configured with an ar-

rangement to separate the surveillance functions and assign different surveillance functions to different cameras. A master camera is

assigned the surveillance of large area surveillance and tracking of object movement while one or more slave cameras are provided

to dynamically rotate and adjust focus to obtain clear image of the moving objects as detected by the master camera. Algorithms to

adjust the focus-of-attention are disclosed to effectively carry out the tasks by a slave camera under the command of a master camera

to obtain images of a moving object with clear feature detections.

C

M

V

O

O

V /

O

O

N

I

r'

a

INTERNATIONAL SEARCH REPORT International application No.

PCT/US 05143808

A. CLASSIFICATION OF SUBJECT MATTERIPC(8) - H04N 7/18 (2007.01)USPC - 348/143

According to International Patent Classification (IPC) or to both national classification and IPC

B. FIELDS SEARCHED

Minimum documentation searched (classification system 'followed by classification symbols)

USPC: 348/143

Documentation searched other than minimum documentation to the extent that such documents are included in the fields searchedUS: 375/240.01; 348/143, 152, E7.086

Electronic data base consulted during the international search (name of data base and, where practicable, search terms used)PubWEST(PGPB,, USPT, USOC, EPAB, JPAB)Search Terms: video and camera and (surveillance or security), master and slave, processor, master camera, slave camera, multipledegrees of freedom, degrees of freedom, freedom near3 degree; GOOGLE.com/patents: master and slave video cameras with zoomcontrol ,


Category* Citation of document, with indication, where appropriate, of the relevant passages Relevant to claim No.

X US 6,724,421 B1 (GLATT) 20 April 2004 (20.04.2004), [abstract], col. 1, In 10-12, col. 3, in 2-3, 1 - 33- 8-20, Fig. 1, claim 3. ---y 34

Y US 6,795,106 B1 (COOPER) 21 September 2004 (21.09.2004), [abstract], col. 7, 11-16 34

I Further documents are listed in the continuation of Box C. []* Special categories of cited documents: "T" later document published after the international filing date or priority"A" document defining the general state of the art which is not considered date and not in conflict with the application but cited to understand

to be of particular relevance the principle or theory underlying the invention

"E" earlier application or patent but published on or after the international "X" document of particular relevance; the claimed invention cannot befiling date considered novel' or cannot be considered to involve an inventive

"L" document which may throw doubts on priority claim(s) or which is step when the document is taken alonecited to establish the publication date of another citation or other "Y", document of particular relevance; the-claimed invention cannot bespecial reason (as specified) considered to involve an inventive step when the document is

"O" document referring to an oral disclosure, use, exhibition or other combined with one or more other such documents, such combinationmeans being obvious to a person skilled in the art

"P" document published prior to the intemrnational filing date but later than "&" document member of the same patent familythe priority date claimed

Date of the actual completion of the international search Date of mailing of the international search report

08 August 2007 (08.08.2007) 1 00 CT 2007Name and mailing address of the ISA/US Authorized officer:

Mail Stop PCT, Attn: ISA/US, Commissioner for Patents Lee W. YoungP.O. Box 1450, Alexandria, Virginia 22313-1450 PCTHepd 571-272-4300Facsimile No. 571-273-3201 PCT OSP:671-272-7774

Form PCT/ISA/210 (second sheet) (April 2007)



PCTWRITTEN OPINION OF THE


(PCT Rule 43bis.1)

Date of mailing 1 0 CT 2007(day/mont ear)


PROXIMEX0402 See paragraph 2 below '

International application No. International filing date (day/monthlyear) - Priority date (day/month/year)

PCT/US 05/43808 04 December 2005 (04.12.2005) 04 December 2004 (04.12.2004)

International Patent Classification (PC) or both national classification and IPC .IPC(8) - H04N 7/18 (2007.01)USPC - 348/143Applicant

PROXIMEX CORPORATION


SBox No. I Basis of the opinion

] Box No. II Priority

[ Box No. III Non-establislihment of opinion with regard to novelty, inventive step and industrial applicability

SBox No. IV Lack of unity of invention

SBox No. V Reasoned statement under Rule'43bis. i (a)(i) with regard to novelty, inventive step or industrial applicability;citations and explanations supporting such statement

[ Box No. VI Certain documents cited

SBox No. VII Certain defects in the intern

[ Box No. VIII Certain observations on the

ational application

international application

2. FURTHER ACTIONIf a demand for international preliminary examination is made, this opinion will be considered to be a written opinion of theInternational Preliminary Examining Authority ("IPEA") except that this does not apply where the applicant chooses an Authorityother than this one to be the IPEA and the chosen IPEA has notified the International Bureau tinder Rule 66. lbis(b) that writtenopinions of this International Searching Authority will not be so considered..

If this opinion is, as provided above, considered to be a written opinion of the IPEA, the applicant is invited to submit to the IPEAa written reply together, where appropriate, with amendments, before the expiration of 3 months from the date of mailing of FormPCT/ISA/220 orbefore the expiration of 22 months from the priority date, Whichever expires later.

For further options; see Form PCT/ISA/220.


Name and mailing address of the ISA/US Date of completion of this opinion Authorized officer:Mail Stop PCT, Attn: ISA/US Lee W. YoungCommissioner for Patents 10 August 2007 (10.08.2007)P.O. Box 1450, Alexandria, Virginia 22313-1450 PCT Holpdesic 571-272-4300

Facsimile No. 571-273-3201 PCT OSP: 571-272-7774


To: BO-IN LIN13445 MANDOLI DRIVELOS ALTOS HILLS, CA 94022

1

WRITTEN OPINION, OF THE International application No.INTERNATIONAL SEARCHING AUTHORITY

PCT/US 05/43808


1. With regard to the language, this opinion has been established on the basis of

[ the international application in the language in which it was filed.

] a translation of the international application into which is, the language of atranslation furnished for the purposes of international search (Rules 12.3(a) and 23.1(b)).

2. This opinion has been established taking into account the rectification of an obvious mistake authorized by or notifiedto this Authority under Rule 91 (Rule 43bis. I(a))

3. With regard to any nucleotide and/or amino acid sequence disclosed in the international application, this opinion has been

established on the basis of:

a. type of material

L a sequence listing

LI table(s) related to the sequence listing

.b. format of material

Li on paper

- in electronic form

c. time of filing/furnishing

[-] contained in the international applicatioin as filed

i filed together with the international application in electronic form

[ furnished subsequently to this Authority for the purposes of search

4. L In addition, in the case that more than one version or copy of a sequence listing and/or table(s) relating thereto has beenfiled or furnished, the required statements thit the information in the subsequent or additional copies is identical to thatin the application as filed or does not go beyond the application as filed, as appropriate, were furnished.;


+.

Form PCT/ISA/237 (Box No. I) (April 2007)


INTERNATIONAL SEARCHING AUTHORITY PCT/US 05/43808PCT/US 05/43808

Box No. V Reasoned statement under Rule 43bis.1(a)(i) with regard to novelty, inventive step or industrial applicability;citations and explanations supporting such statement

1. Statement '

Novelty (N) Claims 34 ( YES

Claims 1 -33 NO

Inventive step (IS) Claims NONE. YES

Claims 1 -34. NO

Industrial applicability (IA) Claims 1 -34 YES

Claims NONE NO

2. Citations and explanations:Claims 1-33 lack novelty under PCT Article 33(2) as being anticipated by US 6,724,421 B1 (Glatt).

Regarding claims 1, Glatt teaches a video surveillance system comprising: at least two video cameras performing a surveillance by usinga cooperative and hierarchical control process ([abstract], col, 3, In 8-14).

Regarding claim 2, Glatt teaches two video cameras comprising a first video camera functioning as a master camera for commanding asecond video camera functioning as a slave camera (stationary pilot camera ("master") for monitoring the area and one or more moveableslave cameras for monitoring at least part of the area, [abstract]).

Regarding claim 3, Glatt teaches that the two cameras further controlled by a control processor (Such a camera incorporates a controllerin the form of Z180 processor, col. 3, In 2-3).

Regarding claim 4, Glatt teaches that the two cameras further controlled by a control processor embodied in a computer (col. 3, In 8-14).

Regarding claim 5, Glatt teaches that at least one of the cameras are mounted on a movable platform (the slave camera is movable, claim-3).

Regarding claim 6, Glatt teaches that at least one of the cameras having a flexibility of multiple degrees of freedormn (DOFs) (The ability to.pan and tilt allows each slave camera to track the movement of an object (an intruder. for example) within the cell to which the camera isassigned and into an adjacent cell, col. 3, In 17-20).

Regarding claim 7, Glatt teaches that at least one of the cameras having a rotational flexibility for pointing to different angular directions(col. 3, In 17-20).

Regarding claim 8, Glatt teaches that at least one of the cameras having an automatically adjustable focus length (Computer 41 isprogrammed to instruct, via bus 45, any of cameras 16, 18, 20 and 22 to pan, tilt, zoom, focus, col. 3, In 15-17).

Regarding claim 9, Glatt teaches that at least one of the cameras is provided to receive a command from another camera to automaticallyadjust a focus length (The pilot camera produces a signal representative of the area. The location of a moving object in the area monitoredby the pilot camera is determined. A signal is produced representing the location of the object. The slave cameras track the object basedon the signal representing the location of the object, [abstract]).

Regarding claim 10, Glatt teaches that the surveillance comprises at least three cameras configured in a co-linear configuration (Fig. 1).

Regarding claim 11, Glatt teaches that the surveillance comprises at least three cameras configured in a planar configuration (Fig. 1).

Regarding claim 12, Glatt teaches that the surveillance comprises at least three cameras with one master camera and two slave camerasdisposed on either sides the master camera (Fig. 1).

Regarding claim 13, Glatt teaches a video surveillance system comprising: a controller controlling a camera for performing a stationaryglobal large-filed-of-view surveillance and a dynamic selective focus-of-attention surveillance by implementing a cooperative andhierarchical control process ([abstract], col. 3, In 8-14).

Regarding claim 14, Glatt teaches that the cooperative and hierarchical control process controlling the camera to function as a stationarycamera to carry out the global large-field-of-view surveillance and dynamic camera to carry out the selective-focus-of-attentionsurveillance ([abstract]).

Regarding claim 15, Glatt teaches that the controller is embodied in a computer (cot. 3, In 8-14).

Regarding claim 16, Glatt teaches that the camera is mounted on a movable platform (the slave camera is movable, claim-3).

-(Continued on Supplemental Pages)-

Form PCT/ISA/237 (Box No. V) (April 2007)__


INTERNATIONAL SEARCHING AUTHORITY' PCT/US 05/43808


Supplemental Box

In case the space in any of the preceding boxes is not sufficient.Continuation of:

V.2. Citations and explanations:

Regarding claim 17, Glatt teaches that the camera having a flexibility of multiple degrees of freedom (DOFs) (an intruder for example)within the cell to which the camera is assigned and into an adjacent cell, col. 3, In 17-20).

Regarding claim 18, Glatt teaches that the controller Is embodied in the camera as an embedded processor (Such a camera incorporatesa controller in the form of Z180 processor., col. 3, In 2-3). "

Regarding claim 19, Glatt teaches a video surveillance system for a large area comprising several compartmentalized zones comprising: a

stationary video camera for monitoring the large area and at least two dynamic video cameras for monitoring the severalcompartmentalized zones wherein the global video camera and the dynamic video cameras are operated according to a cooperative andhierarchical control process (A video surveillance system for monitoring an area is made up of a stationary pilot camera for monitoring thearea and one or more moveable slave cameras for monitoring at least part of the area, [abstract]).

Regarding claim 20, Glatt teaches that the stationary video camera functioning as a master camera for commanding the dynamic video

cameras functioning as slave cameras (stationary pilot camera ("master") for monitoring the area and one or more moveable slavecameras for monitoring at least part of the area, [abstract]).

Regarding claim 21, Glatt teaches that the stationary camera and the dynamic cameras are further controlled by a control processor (Sucha camera incorporates a controller in the form of Z180 processor, col. 3, In 2-3).

Regarding claim 22, Glatt teaches that the stationary camera and the dynamic cameras are further controlled by a control processorembodied in a computer (col. 3, In 8-14).

Regarding claim 23, Glatt teaches that at least one of the video cameras are mounted on a movable platform (the slave camera is

movable, claim-3).

Regarding claim 24, Glatt teaches that at least one of the video cameras having a flexibility of multiple 25 degrees of freedom (DOFs) (an

intruder for example) within the cell to which the camera is assigned and into an adjacent cell, col. 3, In 17-20).

Regarding claim 25, Glatt teaches that at least one of the video cameras having a rotational flexibility for pointing to different angulardirections (The ability to pan and tilt allows each slave camera to track the movement of an object (an intruder for example) within the.cell

to which the camera is assigned and into an adjacent cell, col. 3, In 17-20).

Regarding claim 26, Glatt teaches that at least one of the video cameras having an automatically adjustable focus length (Computer 41 is

programmed to Instruct, via bus 45, any of cameras 16, 18, 20 and 22 to pan, tilt, zoom, focus, col. 3, in 15-17).

Regarding claim 27, Glatt teaches that at least one of the dynamic cameras is provided to receive a command transmitted as wireless

signals from the stationary camera (The signal representative of the area is compressed and transmitted over a communications channel

for remote monitoring, [abstract)).

Regarding claim 28, Glatt teaches that the stationary video camera and the dynamic video cameras are configured in a co-linear

configuration (Fig. 1).

Regarding claim 29, Glatt teaches that the stationary video camera and the dynamic video cameras are configured in a planar

configuration (Fig. 1).

Regarding claim 30, Glatt teaches that the stationary camera and the dynamic cameras are further controlled by a control processor

embodied in the stationary video camera (a pilot camera controls the operation of one or more slave cameras, col. 1, in 10-12).

Regarding claim 31, Glatt teaches a video surveillance camera comprising: a global large-filed-of-view surveillance lens and a dynamic

selective-focus-of-attention surveillance lens ([abstract]); and an embedded controller for controlling the-video surveillance camera to

implement a cooperative and hierarchical control process for operating with the global large-filed-of-view surveillance lens and the dynami

selective-focus-of-attention surveillance lens ([abstract]).

Regarding claim 32, Glatt teaches that the camera is mounted on a movable platform (the slave camera is movable, claim-3).

Regarding claim 33, Glatt teaches that the camera having a flexibility of multiple degrees of freedom (DOFs) (an intruder for example)

within the cell to which the camera is assigned and into an adjacent cell., col. 3, In 17-20).

-(Continued on Supplemental Pages)-


INTERNATIONAL SEARCHING AUTHORITY PCT/US 05/43808

Supplemental Box

In case the space in any of the preceding boxes is not sufficient.Continuation of:

/.2. Citations and explanations:

lialm 34 lacks an Inventive step under PCT Article 33(3) as being obvious over Glatt in view of US 6,795,106 B1 (Cooper).

Regarding claim 34, Glatt does not teach that that controller is embodied in the camera as an application specific integrated circuit (ASIC)processor. However, Cooper teaches a camera controller (abstract], and that the controller 202 is intended to represent any of a numberof altemative controllers available in the art including, but not limited to, a microcontroller, a central processing unit (a.k.a., processor), anApplication Specific Integrated Circuit (ASIC), and the like., col. 7, 11-16). It would have been obvious to one skilled in the art to combine

he teachings of Glatt with those of Cooper because both teaching camera systems that use controllers. Furthermore, an ASIC would

allow the desired functionality to be designed into the controller.

Claims 1 - 34 have industrial applicability as defined by PCT Article 33(4), because the subject matter can be made or used in industry.


Case 1:10-cv-01 1....- JCC-IDD Document 1 Filed 10/1... 0 Page 1 of 11

IN THE UNITED STA'T'ES D)ISTrRICT COURTFOR THE EASTERN DISTRICT OF VIRGINIA

AI.EXANDRIA DIVISION

VIDSYS, INC., a Delaware Corporation8219 Leesburg PikeSuite 250Vienna, VA 22182

Plaintiff,

PROXIMEX CORPORATION, a DelawareCorporation440 N. Wolfe RoadSunnyvale, CA 94085

Defendant.; eenat

)

)

Civil Action No. (0 Co / /iJ7b

JURY TRIAl.. DEMANDED

CON'PILAI NT Folt_[),Ci LA.RA'l1OIZ .J UIGEM1":NT

Plaintif; VidSys, Inc., ("VidSys" or "Plaintifl") by and through its ubdersi'gned counsel,.

brings this action against defendimnt, Proximcx Corporation; ('"Proximex" or "Defndant") and

alleges as follows:

INTRODI)UCTION

1. Plaintiff VidSys respectively requests that this Court enter an Order declaring (1) that

VidSys does not infringe U.S. Patent No. 7,777,783 (the '783 Patent), (2) that the '783 Patent is

invalid, and (3) that VidSys did not breach a Mutual Non-Disclosure Agreement hetween the

parties, dated November 14, 2006, or misappropriate any trade secret from Defenldant.

I L I~-7-----̂-------- ----~j;_,,,,.

I ,. ,.,,,~..,,.....,.,..,...... ........,........ ~...... ........ ..........1.....,, .. ,, I I

I

Case 1:10-cv-011oS-JCC -IDD Document 1.. Filed ,10/T 10 Page 2 of 11

7. VidSys is a provider of Physical Security information Management ("PSIM") software

marketed as "VidShield" and "RiskShield." PSIM software systems are designed and optimized

to integrate and analyze information from traditional security devices and systems, and present

the necessary data to automatically or manually analyze the situation in real time or thereafter.

VidSys' PSIM software (VidShield) includes features known as "Virtual Tracker" and "Back

Tracker" which provide a user with control over security camera video display operation.I-

8. Proximex also is a provider of PSIM software and, upon information and belief, has

offices throughout the United States, including in California, Colorado, Connecticut, North

Carolina, Tennessee, Texas, and Washington, D.C. Proximex's PSIM software includes a

feature known as "EZ Track," which, upon information and belief, also provides a user with

control over security camera video display operation.

9.( VidSys and Proximex are direct competitors in the PSIM software market. Upon

information and belief, VidSys' and Proximex's competing products include the Virtual Tracker,

Back Tracker, and EZ Track features described above.

10. VidSys and Proximex have been in engaged in competition for the business of a major

prospective customer ("Customer"). In connection with this competition, VidSys has installed

its PSIM software, which includes the Virtual Tracker and Back Tracker features, in a Customer

demonstration computer laboratory located in Herndon, Virginia, which is located in this judicial

district. Upon information and belief, Proximex has installed its PSIM software, which includes

the EZ Track feature, in the, same Customer demonstration computer laboratory located in

Hemdon, Virginia in competition for Customer's business. Upon information and belief, VidSys

Case 1:10-cv-01185-JCC -IDD Document I Filed 10/19/10 Page 3 of 11

and Proximex have had their respective PSIM software. products installed in Customer's

Herndon facility for approximately two years.

1. VidSys permits Customer to use its PSIM software, which includes the Virtual Tracker

and Back Tracker features, to demonstrate for the Customer the capabilities of the software. As

a result, Customer enjoys at least an implied license to use VidSys' software. VidSys does not

have information as to whether or not Proximex licenses Customer to use its PSIM software in

Herndon, Virginia. However, if Customer does not at least enjoy an implied license of the

Proximex software for demonstration purposes, then upon information and belief, there would be

no point in Proximex's installing the software in Customer's Herndon, Virginia facility.

12. Upon information and belief, Proximex has installed its PSIM software in the Reston,

Virginia facility of a well-known information technology company ("Technology Company") for

evaluation of such software. Upon information and belief, the installation with thisTechnology

Company was in place approximately in the late 2008 to early 2009 time frame.

13. Recently, there have been one or more announcements in the press that Proximex was

awarded a key patent for its EZ Track technology. One such announcement is attached hereto as

Exhibit 1, and incorporated herein by reference. Based on the date of the announcement, and

upon information and belief, the key patent referenced is the '783 Patent.

VIDSYS' RELATIONSHIP WITH PROXIMEX

14. VidSys, from time to time, explores and, if appropriate, partners with third party vendors

to explore business opportunities with prospective customers seeking PSIM solutions.

N.

Case 1:10-cv-011 f85-JCC -IDD Document 1 Filed 10/1 '0 Page 4 of 11 .

15. In 2006, VidSys and Proximex communicated with each other via multiple telephone

calls and emails to and fromn a location in this judicial district in order to explore 'a business

opportunity with a major national retailer. The contemplated business opportunity involved an

in-store security surveillance solution, or what is now referred to as a PSIM solution.

16. In anticipation ofjointly submitting a proposal to this same retailer, the parties executed a

Mutual Non-Disclosure Agreement ("NDA") on November 14, 2006. A copy of the NDA. is

attached as Exhibit 2, and incorporated herein by reference. '

17. The NDA explicitly provides under Section 2 that Confidential Information does not

"include any information which Recipient can establish (i) was publicly known and made

generally available in the public domain prior to the time of disclosure to Recipient by Discloser;

(ii) becomes publicly known and made generally available after disclosure to Recipient by

Discloser through no action or inaction of Recipient; or (iii) is in the possession of Recipient,

without confidentiality restrictions, at the time of disclosure by Discloser as shown by

Recipient's files and records immediately prior to the time of disclosure."

18. Under Section 9, the NDA further provides that its Term, shall "survive until such time as

all Confidential Information disclosed hereunder becomes publicly known and made generally

available through no action or inaction of Recipient."

19. After executing the NDA, Proximex telephoned and emailed VidSys' office in Virginia

on several occasions in' order to exchange information regarding VidSys' and Proximex's PSIM

technology and to finalize the proposal for the aforementioned prospective retailer customer.

The parties jointly cooperated in providing information for the proposal.

Case 1:10-cv-01 i 5-JCC -IDD Document 1 Filed 10/1Y,10 Page 5 of 11

20.- VidSys sent the proposal from its Virginia office, on behalf of both VidSys and

Proximex, to the retailer, but for reasons unrelated to the proposal's merits, the business

opportunity was no longer of interest to the retailer.

21. The parties had no further communications about their respective products and services

over the ensuing three years until Proximex sent the threatening letter that gave rise to this

controversy.

PROXIMEX'S ACCUSATIONS AND THREATS

22. Upon information and belief, the '783 Patent was recently issued by the United States

Patent Office on August 17, 2010.

23, On September 29, 2010, outside counsel for Proximex sent a cease and desist letter to

VidSys in which he accused VidSys of making improper use of Proximex's proprietary

technology and product portfolio, including infringing the '783 patent, misappropriating

Proximex's trade secrets, and violating the NDA. A true and correct copy of Proximex's

counsel's letter ("the September 29 letter") is attached as Exhibit 3, and incorporated herein by

reference.

24. Among other things, the September 29, letter asserts that:

(a) Proximex is the "owner of all right, title and interest in and to U.S. Patent-No.7,777,783 'Multi-Video Navigation System' (the 'Proximex Patent');"

(b) "The Proximex Patent covers systems and methods of synchronizing video datagenerated using multiple video cameras and automatically generating a stitchedvideo sequence based on the user selection of video camera, embodiments ofwhich include tracking of a target between video cameras;"

Case 1:10-cv-011d5-JCC -IDD Document1 Filed 10/1@,10 Page 6 of 11

(c) "Proximex currently markets and sells its Proximex EZ TrackTM a multi-videonavigation solution that easily tracks individuals across multiple camera viewsand different types of cameras that is based on the Proximex Patent;"

(d) "[S]ubject to the terms of a Non-Disclosure Agreement, dated November 14,2006, (the "NDA"), a copy of which is also enclosed herewith, VidSys has beenin possession of Proximex proprietary confidential information since at leastNovember 2006 when Proximex disclosed such confidential information toVidSys;"

(e) VidSys' products called the VidSys VidShield and RiskShield "contains a VirtualTracker feature that enables easy tracking of an. object in 'real time and. BackTracker, which allows operators to go back in time, using recorded video forforensic analysis of events;"

(f) "This product [feature] therefore appears to infringe the Proximex Patentdescribed above;"

25. The September 29 letter demanded that VidSys "agree to stop selling and offering to sell

any component of the VidShield, RiskShield or any other VidSys product that infringes upon the

Proximex Patent or Proximex's confidential information" no later than October 8, 2010.

26. VidSys did not receive the September 29 letter until approximately a day before the

response deadline of October 8, 2010 because the letter had been sent. to VidSys' Virginia

address listed in the NDA which is no longer the current address for VidSys.

27. VidSys develops, 'manufactures, offers for sale and sells one or more products with

features of the type accused of infringement by Proximex, and continues to develop,

manufacture, offer for sale and sell such products and services in Virginia. .Proximex'sconduct,

directed towards VidSys in Virginia, has created a reasonable apprehension on. the part of

VidSys that it will be faced with an infringement suit if it continues' to make, use, sell and offer

to sell its PSIM products, including VidShield and/or RiskShield products with the Virtual

Tracker and/or Back Tracker features.

Case 1:10-cv-011b5-JCC -IDD Document 1 Filed 10/ 10 Page 7 of 11

28. VidSys has not infringed, and is not now infringing the '783 Patent, directly or indirectly,

nor has VidSys induced or contributed to others' infringement of the '783 Patent.

29. Additionally, the '783 Patent is invalid because, inter alia, the alleged claimed inventions

fail to satisfy the conditions for patentability specified in 35 U.S.C. §§ 101, 102, 103 and/or 112.

30. VidSys independently developed its Virtual Tracker and Back Tracker features, as well

as its other software products and services, without the use of any Proximex trade secret

information and without violation of the terms of the NDA.

31. Moreover, upon information and belief, the information provided to VidSys by Proximex

does not constitute information that is protected as trade secret information.

32. Upon information and belief, Proximex's asserted patent rights, purported trade secrets,

and purported proprietary confidential information disclosed pursuant to the NDA, all, according

to Proximex, concern the same technology, namely Proximex's EZ Track. feature and VidSys'

Virtual Tracker and Back Tracker features.

33. If VidSys were to acquiesce to Proximex's demands, VidSys would suffer irreparable

harm to its business which would have an impact on VidSys in this judicial district.

COUNT I

(Non-Infringement of the '783 Patent)

34. VidSys repeats and re-alleges paragraphs 1-33 as if fully set forth herein.

35. ' Based on Proximex's conduct purposefully directed towards VidSys in Virginia, Plaintiff

has a reasonable apprehension that it will be faced with an infringement suit if it continues

Case 1:10-cv-01185-JCC -IDD Document 1 Filed 10/19110 Page 8 of 11

making, using, selling or offering to sell its PSIM software including its VidShield and

RiskShield Virtual Tracker and Back Tracker features.

36. As a result of Pioximex's conduct as outlined above, an actual controversy now exists

between VidSys and Proximex concerning whether the VidShield and RiskShield Virtual

Tracker and Back Tracker features infringe any claim of the '783 Patent, which is attached

hereto as Exhibit 4.

37. VidSys has not and does not infringe the '783 Patent.

38. Therefore, VidSys seeks entry of a declaratory judgment that it does not infringe the '783

Patent.

COUNT II

(Declaration of Invalidity of the '783 Patent)

39. VidSys repeats and re-alleges paragraphs 1-38 as if fully set forth herein.

40. Based on Proximex's conduct as outlined above, VidSys has a reasonable apprehension

that it will be faced with an infringement suit if it continues making, using, selling or offering to

sell its PSIM software including its VidShield and RiskShield Virtual Tracker and Back Tracker

features. C

41. VidSys has not and does not infringe the '783 Patent. Moreover, none of the claims of

the '783 Patent are valid. Patent invalidity is a defense to patent infringement, and thus an actual

controversy now exists between VidSys and Proximex concerning. the validity of the claims of

the '783 Patent.

Case 1:10-cv-0175-JCC -IDD Document 1 Filed 10/19/10 Page 9 of 11

42. Therefore, VidSys seeks entry of a declaratory judgment that all claims of the '783 Patent

are invalid because they fail to satisfy the conditions for patentability specified in 35 U.S.C. §§

101, 102, 103 and/or 112.

COUNT III

(Declaration That VidSys Has Not Breached the NDA or Misappropriated Trade Secrets)

43. .VidSys repeats and re-alleges paragraphs 1-42 as if fully set forth herein.

44. Proximex has accused VidSys of breaching the NDA and misappropriating trade secret

information based on Proximex's contention that VidSys has used Proximex's purportedly

confidential, proprietary, or trade secret information relating to the technology described and/or

claimed in the '783 Patent.

45. Contrary to Proximex's' accusations, VidSys has not breached the NDA or

misappropriated any Proximex trade secret information.

46. An actual controversy exists between VidSys and Proximex with respect to Defendant's

accusation that VidSys breached the NDA and misappropriated trade secret information because

the allegedly misappropriated information:

(a) under Section 2 of the NDA, was generally known to the public or to persons in'the PSIM industry at the time ofdisclosure to VidSys or since the time ofdisclosure to VidSys;

(b) did not survive the "Term" of the NDA as provided, Section 9 because suchinformation has become publicly known and made generally available through noaction or inaction of VidSys;

(c) did not derive any independent economic value from not being generally knownto the public or to persons in the PSIM industry;

Case 1:10-cv-O11 5-JCC -IDD Document 1 Filed 10/0 'iO Page.lO of11

(d) was not used or disclosed by VidSys in violation of the terms of the NDA; and/or

(e) did not qualify as trade secret information under any applicable state or federallaw.

47. Therefore, VidSys seeks entry of a declaratory judgment that it has not breached the

NDA and that it has not misappropriated any Proximex trade secret information.

PRAYER FOR RELIEF

Wherefore, VidSys prays that the Court enter judgment in its favor and against Defendant

Proximex, providing the following relief:

A. Entry of Declaratory Judgment that Plaintiff VidSys does not infringe the '783 Patent;

B. Entry of Declaratory Judgment that each and every claim of the '783 Patent is invalid,

void, and without force or effect;

C. Entry of Declaratory Judgment that Plaintiff did not breach the NDA and did not,

misappropriate any Proximex trade secret information;

D. A finding that this case is an exceptional case and award Plaintiff VidSys reasonable

attorneys' fees, costs, and expenses for this action pursuant to 35 U.S.C. § 285;

E. An award of costs incurred in connection with this suit; and

F. An award of such further and other relief as the Court deems appropriate.

Case 1:10-cv-01 65-JCC -IDD Document 1 Filed 10/1 .1 0 Page 11 of 11

JURY DEMAND

VidSys hereby demands a trial by jury of all counts of this Complaint which are.

permitted to be tried by a jury.

Date: October 19, 2010

Respectfully submitted,

David R. Yohannan ( inia Bar No. 37464)Stephen R. Freeland (Virginia BarNo. 72947)Kelley Drye & Warren, LLP3050 K Street, N.W., Suite 400Washington, D.C. 20007-5108Phone: 202-342-8400Fax: 202-342-8451E-mail: dyohannan(ibkelleydrye.comE-Mail: sfrceland(@kelleydrye.com

Attorneys for Plaintiff VidSys, Inc.

N

.. o, :0;Tr o:

on-~'>. ~Oj":.~

L r

c5 1

441C,)04~

0

I

.c ..

I~ ~

:: '::~1: ;'~ : ::%I' ' :::I r

"

:

I,.:i

iij~~ .,

': '

i

'

0-

r '~&.

r C

5I

r~~~~ ~ W "w'44jl i+ ,: ~Tx,.y, :. ", .;-t.. ':t 4" '': f e;N; ......ci 6}i yri' jf .", ..

/t . t e J~sR + ' ,It'~s A ! 4' , ~

t tii~ '. sr'i; 3 .§tRK ;"J'iMx 4VrK" Aid .',' ' S " lS J f ?1 ' Mr,;+C " ; t r I 1 N , dr. ( C

" Y f J f u Ur ,m l

w

LF-

0w

4

LU

N.

0J

' 0

C,)C0

C',et'

H0Lu

ZLu

-~4-a

~0o:

.V :c..").T

VI'-a)

0o

aaQ

Ci

UW

C,)

m0

a 0.

o a_

L.

o

aE

- C

L.muWO

a)~OCJ~ ~ C,) ~eE~au~

0~ W a ~

* U U00

0

I

S ...

aI ...

a

(: '3~:LL:

w

01wr

0

0

' , . ' . ,.

U.:.

r i .". ii

-a..

0

CV:C.

0

rn'0;

o

O

Q.t0 '.

C,

(I

C

-6-4

w'. wry ' :::.: I

..k .... k, .Cl)

F-U :I'.:~ ::~

M

(N

(N

09

''

I,~ ~ ~:::. :

w3

HG.2*'

4 .

S.4

*4

5

U

0

.

D.

* 0

C4N-0

M (~Q-I. ~

~~1)00L

U 0

sI 0

N00'

V7.--

0 0D

H

LU

LiLPJ

rJ

Lu

I-

00~

M

Go

II

i

woo..

40D~ O ::

0)

.0..,*., -~ 0C -'-'a)o

~ 0)co C(VL~jCWL~QQL.

a a)inc~'o C/*J~a) C*+-~a ~ Cl)> >c~QQQL.4in~ O~L L. 0 ) C/) ~a a ~0

* U U U

C,)

0

Cl)c

Ul:

0

0

000

C?.

w

V"

W

Im

. ,. ..

O ., ~ !• ..

Jm

00

O0a

1 1. 7

'I'I

' ''' '

: .

":

I :: ::~

: :

I

:

:~)

:: I

~:'

" ~': '' ~: ::::::: ~':,:: \ i

:

~: ..:: : . . . .:

.i:'

o:.

cd

::::.: .,: :: : :!:i . (:I : : ..~.:. ii ;j~ii~9~

i.~ .:::: .i.Ij: 1.: :.: . . :: :.:

U

Ik

0ilia-U0 '

"0

"0

0

00

" I

U Co .

4-,Co

Co Co

_ X U)

U)I

3; 0Co. _

L...ao -

Coa)Co

a)a) aCo

a) $a)~a)> c~ L~

~.. (~)

T c~)(~ ~*

Eu U

"S

c:

Ci)0Q

Wa

wHIF3

S%3

0

0 0:

aa-. '. .:

a) .

0rY.Cl)' ...

0U-

0'. :rn~: ;. .:.

E t~..'...::. ':'.::';)o:~ . ......-' ' o ' '; ~ ~ ( . i:: " I . ~ . ' .. ; '' .:..'

_~r *. Q4. £ C D :

~ U U :-' : ,' .:

: ::~

:::.::.. ~ ... : .

i

F-

hA

VILi

(I)

c

c0i~

9<0LL~a)

m

u-u-rn-i

C..4-

.4

1

pp

4

J.

(N

U U U

w.

b

Ww

0

0 *

O .

C/)mmmllN~~

0

. ,

:. .

0

i

• O

- - - : - ; ~ - :-- -: :.:: ic :'i: :I '::i:: :i I .:: iii 'i:: ' :

: : '~ :~ :

w"'V

vI"

0 0

C,,

C,)3 .(/):

" Q .0 :1' x"

i

M

O'

N

00

Li

C?.NOZ

/

:l"

rio,.

0uCl: U-L '

O ',CN1

it

LiJ

H

,,Ine

oo-

Cl) C)

C. _ _

H

Zw

a

0

0

0

U

U

-cJQ)

:.:

0

0Oe

1".

N '

C?

w-

I"

E

,a

U

C

*U

.

Co:

ca) 0.E~ 0

w

0w

0

N

00

..... C.L -.

-U "

U0O

.0

0

0-

0d

Li

fm'

w

.ME

Cl

Cl

L

0)"

. v

0 W L, _ ..U)o.~ oA

w

z

Z'

4

aN

4.w

Lu.V

0

" o

C,))

_UC,)

.. A,

0lw'.

_ C,

' .:4 -

'.i* (i c5. - c .-- Idg-)

(I) cq)l0_ 0A

. .: .

O

N0

Q:

U .0 -n

0

0v r 7 l.U

OA

J

aiW

0

w

H

C,, .

00.Q

.rrV

0) Cl

- - -- .1(J) L. C,, CIY*

06c1)'<~,.. U a> 0

c:~)-zo O)XC~3L~

00 ~LUQQU U U U

. . . :

.0

0.

ely

.4.

0

.o

Cl

LU

* C')

0

H

wa).U

oT

44:

. -f,: :C3K

S

: Q'

E ":i C '

a®

HZ

0

, .

U

Q

U

.,.

. . ,:

. o,.

N.

r :

: :

...~.~ . .. : ~ :~.:~~.~ :~

.. .~ ~;-

~

: ~ :":

r : ~ . ' ,:

:

.

. ..: .: . .; : .. ,~

:

: :

: : :: :

'''~:: ,

:ii

:

::

::::

g

:: :: ::'' ' "' :'

Volume 1 Number 1 August 2003

Inside this IssueRedstone LaunchesN~w e ttar

Steve Browt Head.Int&Wte nt TansportationSyste.m (ITS) rogramns

Intelligent Transportaton:Smnart C.Np Solutions WithSmart Plate%

Now Pmduct Ratlut -fsualPucsu.lt dule

NeA. siLe Prevtw

Reatvn Irttegxate. Soatutlois5085 List DriveSuite 110olorado Springs. CO 80919

?1 9 - -12.5

www,Redstunie-S.coni

Redstone LaunchesNewsletterNow, Redstone Friends and Familycan get The Redstone Newsdelivered fresh to your e-mail boxlike clockwork every -- well, oftenenough, don't you worry.

You will receive notice of all thehappenings and news around theRedstone enterprise including newfaces, business developmentactivities, new products andprojects, training and professionaldevelopment programs, whitepapers and technical writings.Stories and infonnation arewelcomed at the Redstone NewsEditor's Desk atRharrison@redstone -is.com.

Steve Brown ToHead IntelligentTransportationSystems ProgramsRedstone is pleased to announce thatSteve Brown will direct theIntelligent Transportation SystemsProgram. A nationally recognizedexpert, Steve Brown has beenassociated with the transportationindustry since 19.75. Steve hasserved as President of IntegratedSoftware Solutions, Inc. since 1993.ISS developed the TransLink ClosedLoop System for centralized controland monitoring of the WapitiMicrosystems' 170

controller firmware. TransLink hasbeen adopted by 9 state agencies andover 50 local agencies in the US,Canada and Mexico. Steve beganhis career as a field supervisor forSignal Maintenance, Inc., a trafficsignal maintenance company inSouthern California, responsible forthe maintenance and operation ofsignalized intersections throughoutthe Southern California area. Steve'sexperience includes 16 years withSafetran Traffic' Systems, Inc. inColorado Springs, as Manager ofTechnical Sales and Services. Forthe last three years, Steve was LeadSoftware Engineer and AssociateVice President for AECOM SystemsIntegration Group.

As Director of ITS Programs, Stevehas already made presentations onRedstone solutions to keytransportation groups in sevenstates. He will coordinate productsales efforts Smart Plate Solutions intolling, parking, and traffic securityaccess.

Continued on Page 2

' ,....~ .~-- - -~I-----~-R^ I~ Wa~

Redstone New's

August 2003

The new smart chip technology inthe Redstone SmnartPlateTM canlocate, store, and manage data assetsfor a wide variety of client needs.SmartPlateTM technology is used forautomatic identification anddetection of vehicles. It is aninput/output device for Redstone'sIntelligent Transportation Systems(ITS) solutions.

IntelligentTransportation: SmartChip Solutions withSmart Plates

Redstone SmartchipTr technologiesemit unique ID's over approved RFfrequencies, much like the lightfrom the beacon on a radio tower,but from within a tiny emitter thatcan be the size of a dime. Thesechips can be deployed in a numberof sizes and shapes, including theSmartplate license plate that willread at up to 200 feet and speeds ofup to 100 mph. The Smartchlp TMtechnologies provide Redstone-ISsoftware with a method to identifyand track mobile assets. Mobile'assets can vary from vehicles toidentification cards.

The smart plate is used in fleetmanagement, automatic vehicleidentification, security accesscontrol, and asset management.The package includes RFIDtransponders, antenna, receivers, andconinunication modules, whichinteract as a system with the smartplate. Redstone provides the front-end software package for thissystem.

These Redstone technologies, whencombined, can provide atransportation client with completetracking, event management, timestamp, physical location and anyother specific tracking capabilitiesrequired by the application. Throughsimple repackaging, SmartPlateTMcan also be used to track containers,airport ground vehicles (includingairplanes on the tarmac), nursinghome patients, employee badges, orvirtually anything that can passthrough a standard reader.

New Product Rollout -VisualPursuitIn August, Redstone will announcethe availability of VisualP ursuit, asoftware package that can be usedby security personnel, facilitycommand-and-control operators, or .any other operator monitoring videocameras inside or outside a facilityto easily track a moving subject..

Surveillance, early intervention in aproblem situation and rapidemergency management are allbenefits of this enhanced operatorcontrol product. The onlyrequirement that an area needs tohave in order for VisualPursuit to beused is adequate coverage bysecurity cameras.

VlsualPursuit uses an array of camera inputs totrack a subject through a facility

VisualPursuit expands Redstone'sintelligent facilities package forsurveillance. It includes remotemonitoring of buildings, complexes,and parking structures. The trackingfeature makes the operations andcontrol center more effective in theuse of video monitoring andinteractions with security personnel.Operator efficiency, flexibility, anddynamic support will improvemonitoring and response, increasingthe value of any camera surveillancesystem large or small.VisualPursuit can be installed inconunercial offce buildings,airports, hospitals, prisons, jails,laboratories, secured buildings orcomplexes such as courthouses,government offices, conventioncenters, event and sports facilities -any area where better surveillance,situation and emergencymanagement are important.

Next Issue: VisualPusuit PriductI vcrvicw, MapTrack Preiew, and

Security White Paper

Gate Access Cointrol

2'

Redstone Intergrated Solutions - Products And Services

Home I Portfolio i <<Products & Services>> I Demos & Downloads I News I Contact Us

e. Redstone Products * Professional Serviceso Redstone o System Integration and

Console Software Engineeringo MapTrack o Testing.and Software Qualityo Visual Pursuit . Assuranceo ResourceTrack o Support Serviceso SmartPlate o Project Managemento Traffic View o Configuration Management

Automation. Standardization. Simplification. Innovation. These four principles underlie everything we do atRedstone.

Through the use of technology, command-and-control of facilities has become easier and more cost-efficient. The newest breakthrough has come in the form of the Redstone Console. It is a software suite thatallows an operator to visually monitor the facilities, computer resources, systems, even individual cars, andinteract with them all from a single interface.

The Redstone technology is superior to other alternatives because it provides you with unparalleledmonitoring and control capabilities. It gives you the power to efficiently control your facility, and does it with-fewer people on staff. The Redstone console is designed in a way that reduces the learning curve, whilekeeping intuitiveness and multifaceted functionality. In the following paragraphs we will be disecting thefunctionality of the Redstone Console in order to give you a better Idea of how the software operates. Thefive main functions of the Redstone Console are:

* Camera Monitoring* Device Tracking* Building Automation* Resource Monitoring* Command & Control

Camera Monitoring/Facility SecuritySystems - VisualPursuit

return to top

Redstone's VisualPursuit enhances an operator's effectiveness by allowing the operator to easily track amoving subject throughout a facility from a single monitor. It accomplishes this by consolidating the videofrom physically adjacent surveillance cameras into one screen, increasing the field of view of a physicallocation dramatically.

VisualPursuit is a part of the Redstone family of software and when used with the full. system, adds subject-tracking and wide-area monitoring capabilities to an operator's toolkit.

VisualPursuit's display combines the output from several surveillance cameras into a single, MS Windows-standard screen.

* The VisualPursuit screen contains a Subject View and Locator Views.* The Subject View displays the current location of the moving subject.* Up to 8 Locator Views display the output of the cameras closest to the one trained on the subject.* Track a subject by clicking the Locator View that the person has moved into. The Locator View

becomes the new Subject View, and all Locator Views are automatically reassigned and updated.

VisualPursuit works with Redstone's MapTrack or ResourceTrack products:

http://web.archive.org/web/2004060601 4343/redstone-is.com/products_services.html '

Page .l of 13

10/20/2010

Redstone Intergrated Solutions - Products And Services Page 2 of 13

* Click,a camera symbol on MapTrack or ResourceTrack to.start a VisualPursuit session.* MapTrack's camera symbols are color-coded so that you always know what cameras are being used

in VisualPursuit.* From MapTrack, monitor and control devices at the same time you're engaged in a VisualPursuit

(lock doors, zoom cameras, etc).

Track the subject's location and direction of movement on MapTrack's interface.

VisualPursuit is an important element of Redstone's Intelligent Facilities package. VisualPursuit makes anoperator more.effective and efficient by allowing her to easily track a moving subject using one monitor. It'also allows her to. quickly scan an area covered by security cameras without having to view severalmonitors.

Applications of VisualPursuit include commercial office buildings, cities, airports, hospitals, sports facilities,prisons, pipeline companies, refineries, and convention centers to name a few. Basically, any area.thatneeds better surveillance and/or emergency management will benefit from the installation of Redstone'sVisualPursuit.

VisualPursuit can be used to monitor and/or control:

* Facilities: casinos, housing complexes, office buildings, convention centers, governmental buildings,prisons, corporate campuses, high-rises

* Airports: access and security* Educational Campuses:* Complex Infrastructures such as chemical plants, pipelines, manufacturing complexes, city

intersections* Entertainment complexes: amusement parks, zoos, sports facilities, museums

VisualPursuit in Action:

1. The picture below is a screenshot of the Visual Pursuit interface. This view shows nine cameras. Thelarge, central window contains the output from the current main tracking camera;, other cameras inproximity to the current tracking camera are displayed in the other eight windows. By clicking on oneof these adjacent views, you can change it to be VisualPursuit's main' camera view. The adjacentviews would then automatically update, based on the new current tracking camera. This is an exampleof how you would use it to follow a person through a facility.

2. A series of LEDs give you an idea of where the current tracking camera is, in relation to anothercamera.

3. There is also a map layout that can assist you with visualizing exactly where each camera is in yourfacility.

<< Click Here for close up shot >>

(Tracking Devices) Inteligent Transportation,_Smart Plates

return to top

The new smart chip technology in the Redstone SmartPlate TM can locate, store, and manage data assets fora wide variety of client needs. SmartPlateTM technology is used for automatic identification and detection ofvehicles.

t...

http://web. archive.org/web/20040606014343/redstone-is.com/products_services.html 10/20/2010


The SmartChip emit unique IDs over approved radio frequencies, much like the light from a beacon on aradio tower, but from within a tiny emitter the size of a •dime. These chips can be manufactured in a variety ofsizes and shapes, including the SmartPlate TM license plate whose signal can be read at up to 200 ft awayand at speeds of up to 100 mph. The SmartChip technology provides Redstone software with a method toidentify and track mobile "assets" such as vehicles and identification cards or security badges.

The SmartPlate" m is used in fleet management, automatic vehicle identification, security access control, andasset management. The package includes RFID transponders, antennae, receivers, and communicationmodules that interact as a system with the SmartPlate TM . Redstone provides the front-end software for thissystem. These Redstone technologies, when combined, can provide a transportation client with completetracking, event management, time-stamp, physical location and any other specific tracking capabilitiesrequired by the application. Through simple repackaging, SmartPlateTM can also be used to track containers,airport ground vehicles (including airplanes on the tarmac), nursing home patients, employee badges, orvirtually anything that can pass through a standard reader.

The applications for this software are:

* Redstone SmartPlate AVI systems: Traffic Light Priority, Passenger Information, TerminalManagement, Access Control, Fleet Management, Speed Measurement, Automatic Vehicle Detection

* Traffic Flow Information.with untagged metal detection feature: Vehicle Counting, Ramp Metering,Incident Detection, Average Speed of Vehicles, Vehicle Density

* Transportation: Traffic Flow Measurement, Speed Measurement, Identification of Smart PlateVehicles, Traffic Light Priority for buses, emergency and Smart Plate: vehicles, Bus PassengerInformation at Bus Stops, Bus Terminal Management, Access Control, Road Pricing/Tolling Systems,Tunnel and Hazardous Materials Management.

Building Automation - MapTrack

return to top

Redstone's MapTrack is an easy-to-use, graphical interface. Users can quickly locate, view, and/or controlcomputer systems, video, security cameras, audio, card access readers, doors, and, a host of other devices,all from a real-time, interactive map interface. MapTrack can display maps or diagrams at the level of detail.required by the user. /

MapTrack is a part of the Redstone family of software products and when used with the full system, addsdigital, interactive map and device control capabilities to an operator's toolkit.

MapTrack's display combines digital maps or diagrams With graphical symbols that can represent thelocations of items such as:

* security cameras, doors, and windows* computers/components in an equipment rack* valves on a pipeline* traffic and railroad signals* alarms (motion detectors, fire alarms, etc.)* steps in a process diagram* and many other devices

These graphical symbols are more than simple static representations, •they can:

* display a device's status (alarming, locked/unlocked, on/off) by 'a change in color or by flashing* allow device interaction and control* allow viewing of device output" give the operator a quick entry into another Redstone module such as VisualPursuit.

MapTrack is an important element of Redstone's Intelligent Facilities package. MapTrack makes an operatormore effective and efficient by using a single, real-time, map-based interface through which an operator can

http://web.archive.org/web/20040606014343/redstone-is.conm/products_services.html. 10/20/2010


monitor and respond to events over an entire facility - all from one or many workstations. The look and feelof this interface can be customized to suit the needs of the client.

Applications of MapTrack include commercial office buildings, cities, airports,. network operations centers,hospitals, sports facilities, prisons, pipeline companies, refineries, and convention centers to name a few.Basically, any area that needs better facilities management, surveillance, and/or emergency managementwill benefit from the installation of Redstone's MapTrack.

MapTrack can be used to monitor and/or control:

* Facilities: access and security, climate control, emergency systems, lighting* Refineries and chemical plants: security, emergency response, product delivery, product status* Pipelines: emergency response, security, product delivery, product status, cathodic protection* Networks and hardware: system status, hardware status, connectivity

<< Click Here for close up shot >>

<< Click here fora close up shot >>

Redstone's software allows for remote monitoring of numerous buildings, complexes, and parking structuresby the property owner for the purpose of reducing'liability, reducing operating .costs, increasing safety andsecurity, reducing loss of assets, and increasing the owner's capability to proactively or quickly respond to avariety of incidents, ranging from simple facility maintenance issues to criminal activity. It also offers theoption to hand-off control of subsystems to emergency response teams during an event (if so equipped.)

Monitoring Resources - ResourceTrack

return to top

A command and control console would not be complete without a way to monitor computer and electricalresources. This is where ResourceTrack comes into play. ResourceTrack is the tool used to monitor anymachine in the facility. It is very important to know the status of computer resources that control power,alarms, communication, etc.

Command & Control -The Redstone Console

return to top

< Click Here for close up shot >>

The Redstone Console is the powerhouse of the entire suite of Redstone software. The Console is theengine that allows Redstone to easily integrate disparate systems into one user interface. It provides thecommon operating framework, device monitoring and control, interface features, device communications,and database used by the rest of the Redstone software system.

The Console's display is called Resource Navigator. Navigator. is the primary "window" into the RedstoneConsole. It allows the user to view, monitor, and control every device, file, and system integrated into theRedstone system.

* Locating devices to control or files to view is as easy as navigating in MS Windows Explorer.* Click acamera-icon in the Resource Tree and view its video output to the right.* Click a filename or a website and view its content without leaving the application. VisualPursuit,

MapTrack and ResourceTrack provide different interfaces into the Redstone Console. These optionalinterfaces allow you to:

* Easily track a moving subject or monitor a wide area of a facility from one interface.* View and/or control elements of a facility from a graphical, map-like interface.

http://web.archive.org/web/20040606014343/redstone-is .com/products_services.html

Page 4 of 13

10/20/2010


* Track a moving subject on video, at the same time you monitor their movement across the facilitymap.

Applications of the Redstone Console include commercial office buildings, cities, airports, networkoperations centers, hospitals, sports facilities, museums, prisons, pipeline companies, refineries, andconvention centers to name a few. Basically, any area that needs better surveillance, facilities management,or emergency management will benefit from the installation of the Redstone Console.

The Redstone Console can be used in the following industries:

* Facilities: casinos, housing complexes, office buildings, convention centers, governmental buildings,prisons, corporate campuses, high-rise buildings

* Airports, Seaports, Train and Subway Stations* Educational or Corporate Campuses* Network Operations Centers and other Command-and-Control installations* Complex Infrastructures such as chemical plants, pipelines, manufacturing complexes, city

intersections, emergency management organizations* Entertainment complexes: amusement parks, zoos, sports facilities, museums* Many more industries can benefit from the integration capabilities of the Redstone Console

Professional Services

return to top

The Professional Services group is responsible for providing fulllintegration, support services, and softwareengineering. Included is turnkey implementation and integration, testing, 24x7 maintenance monitoringprograms, documentation, product development, and many other types of services as described in thefollowing sections.

The three key areas of Redstone's Professional Services are System Integration, Software Engineering, andSupport. System Integration involves working with a customer to bring their hardware, software, andcomputer systems into one, easy-to-manage unit. Software Engineering includes.developing software tosupport a customer's integration effort, upgrading and expanding Redstone's current software, anddeveloping new products to address industry problems. Support includes customer support at all levels, fromTier 1 through system integration support.

Project Management and Configuration Management are also important parts of the Professional Servicesgroup. All projects undertaken by Redstone, whether they are internal development or client projects, will beproperly managed in accordance with industry standards. Configuration Management ensures a traceablehistory and change tracking for code and documentation, is key component of software releases, and allowsRedstone to ensure that the product the customer receives is the most current and accurate available.

System Integration and Software Engineering

return to top

Redstone's Systems Integration and Software Engineering process work in tandem and follow and industry-standard software development process: Scoping, Process Mapping, Design, Documentation, Testing/QA,and Installation. Project Scoping encompasses evaluating the current status of a system or facility, gatheringand writing Requirements, and compiling and writing Specifications (hardware and software). After Project.Scoping is complete, the Design phase of a project starts. After approval of the Design, we write allappropriate Documentation for the solution, then compile a Test/QA Plan and perform Testing. Only after thesolution has passed Testing will we proceed to the Installation phase.

System Integration can be standalone hardware/systems integration, or work together with SoftwareEngineering to deliver a fully customized software solution that fulfills a client's hardware and softwarerequirements in one project. Additionally, Software Engineering can be a standalone process, when workingon custom or unique software development for a customer, or the creation of new Redstone software .products in-house.

http://web.archive.org/web/20040606014343/redstone-is.com/products_services.html

Page 5 of 13

10/20/2010


Software Engineering

return to top

The Software Engineering process employed at Redstone closely follows the System Integration process,with the addition of developing and testing code. Software Engineering can be a standalone projector alsoemployed as an adjunct to a Systems Integration, and vice versa.

Services Offered

return to top.

Whether the project is Systems Integration or Software Engineering, Redstone follows the same process inorder to bring the project to a successful completion.. This process is based on SEI-CMM recommendations(Software Engineering Institute, Capability Maturity Model) and includes the following phases: Scoping,Design, Build, Documentation, Testing/QA; and Installation.

Scoping

return to top


Page 6 of 13

10/20/2010


The Scoping Phase of a project involves working with a customer to gather and write requirements, usingthose requirements to write hardware and/or software specifications, and process mapping.

Standard Deliverables:

* Scoping Study Document* Requirements* Functional Specifications* Process Map* Project Plan and Timeline

Scoping Study

return to top

As the first step in a Systems Integration or Software Engineering project, the Scoping Study is designed todetermine all variables influencing a project. Because systems have often been developed and managedindependently of each other, many organizations can be unaware of all of the complexities involved inintegrating systems or developing new software. A Scoping Study will help identify these complexities andplan. processes and mechanisms for dealing with these issues.

Some of the activities involved in this phase may include:

* Facility tours* Interviews with client Subject Matter Experts, Project Managers, potential users, Information

Technology representatives, and Administration* Observation of the day-to-day operations surrounding the client's current systems

Redstone then analyzes the results of the tours, interviews; and observation sessions to focus ondetermining what needs to be done to fully integrate the client's different systems or develop software. Theresult of these analyses is a document, produced from a Scoping Study Template 'to ensure consistency,that defines the full scope of the project, implementation estimates, recommendations for integration, andmilestones for all potential activities. It also includes all information gathered during the Scoping Study forfuture use by the customer.

Requirements

return to top

The Requirements phase is a natural next step in a Systems Integration or Software Engineering project,after the Scoping Study has been approved. Requirements are mandatory unless the project manager hasgood reasoning for skipping them. Working with a customer, and working from the information obtainedduring the previous phase, Redstone will produce a document that contains both functional and non-functional requirements for completing the project. These requirements will be as detailed as possible asthey will be used throughout the remainder of the project: to create functional specifications,hardware/software design, and test plans.

Requirements for both Hardware and Software will be handled in separate sections of the same document ifboth need to be addressed in the project. Requirements will also be developed from a standard templateand brought under configuration management using CVS. Requirements are baselined, documented, andtracked through the fulfillment process with a dynamic Requirements Traceability Matrix (RTM) document..Requirements are fulfilled through the testing process outlined later.

Functional Specifications

return to top

Once the project's Requirements have been completed and approved by the.customer, Redstone will beginthe Functional Specifications phase. From the techweb.com encyclopedia:


Page 7 of 13'

10/20/2010

Redstone Intergrated Solutions - Products And ServicesFunctional Specifications are the blueprint for the design of.an information system. It provides the.documentation for the database, human and machine procedures, and all of the input, processing, andoutput detail for each data entry, query, update, and report program in the system.

Along with the Requirements and Design Document, discussed in Section 1.2.2.2 Design, the FunctionalSpecification helps to form the backbone of a Redstone project.

A Systems Analyst will develop Functional Specifications as needed for each project. The FunctionalSpecification explains the "what" of the project. It is the blueprint for how the project or application will lookand work. It details what the finished product will do, how a user will interact with it, and what it will look like.

Process Mapping

return to top

Process Mapping is the mapping of serial customer operations to the functions and capabilities of thedesigned system. This process is performed by Redstone system analysts in conjunction with customeroperators and technicians. Process mapping is applied, as an example, to hard copy checklists, or securityprocedures. The result is respectively an automated macro driven checklist, or an automated security.function driven by the software application.

Design

return to top

The Design phase of the project takes the Functional Specification a step further. While. the FunctionalSpecification defines what the project will build,. the Design phase will define how the solution will be built.

The result of this phase will be a Design Document, outlining in detail how the proposed solution will be put-together. For Software Engineering projects, the structure.of the programming, individual code modules-and .their relationships, and the parameters they pass to each other will be described. For solely SystemsIntegration projects, the Design Document will detail the configuration and relationships of the hardware,systems, and any additional software involved.

Documentation

return to top

Besides the documents described previously (Requirements, Functional Specification, Design Document),each: project can also include additional documentation as necessary, written for both end-users andsystem/administrator personnel..

User Documentation

All of the preceding documentation, outlined above, can be used as source material for the final userdocumentation. User Documentation will consist of the following (as applicable):

* User Manual - typically consists of a task-based, quick reference section and a more detailedreference manual. This document is mandatory for all Systems Integration and Software Engineeringprojects and should be written to the "lowest common denominator" or novice end-user.

* Online Help System - produced in tandem with the User Manual, a system designed as an adjunctto the program to deliver information from the user manual while the user is "in" the application. Thisis mandatory for all Software Engineering projects; as-needed for Systems Integration projects. Thisshould be written to the level of the novice end-user.

. Training Materials - training materials include a training manual, training presentations, and online.trainlng modules. The first is mandatory for.all Systems Integration and Software Engineeringprojects, the others.are optional. When training users, these documents will be written to the level ofthe novice user. When training, administrators, they will be written for a more technically experiencedaudience if necessary.


Page 8 of 13

10/20/2010


* Administrator Manual - the administrator manual is a technically detailed but easy-to-understandguide designed for long-term administration and troubleshooting of the system. This is typicallywritten for a more experienced end-user.

* Installation Guide - The installation guide is a document describing, in detail, how the system wasinstalled. It also acts as a guide for future installations or re-installations. Even though information ispresented in detail, it will be written in clear, easy-to-understand language. Written to the level of amore experienced user.

System Documentation

System documentation includes any materials relating to docurfienting the solution from the programmers' orarchitects' points-of-view.

* System Documentation can include:o Hardware Specificationso Wiring Diagramso Inline Code Documentationo Module Documentationo Device Driver Documentation

Testing and Software Quality Assurance

return to top

The Testing phase of a project comes after a software build or systems integration. Testing can encompasscomponent ("white box"), integration, system ("black box"), hardware, and documentation testing..

http://web.archive.org/web/20040606014343/redstone-is.com/products_services.html 10/20/2010

ResoeIneg e Soions II ProuctIAnISeviesIageIo 1

User ManuaI,Online Help,Training Witetrials

fTst Plans, TestCases

IItallatin andAdmihimmtraorManualsel

Figure 1: The Redstone Testing Process

An overview of the testing lifecycle is presented in Figure 2 and is explained in the following sections.


Page 10 of 13Redstone Intergrated Solutions - Products And Services

10/20/20 1 0


Figure 2: Detail of Testing Cycle

Software Quality Assurance applies to the source code and is integrated into the project throughout itslifecycle. It consists of:

Code CritiquesCode ReviewsCode WalkthroughsRefactoringExtreme Programming (XP Style)

1.4 Su..port Services

return to top

The Redstone Support group offers technical assistance to customers of both our Software Engineering andSystems Integration services.

http://web.archive.org/web/20040606014343/redstone-is.com/productsservices.html

Page 11 of 13

10/20/2010


The structure of the Redstone Support group follows a "tiered" support paradigm. The Tier 1 and 2 supportstaff are responsible for taking the initial support request from our customers and providing a basic level ofsupport. Typical problems that are resolved at this support tier are basic software set-up, basic usage ofproduct, configuration and general software upgrade issues, and known software problems. Tier 1 takes thecall from the customer and attempts to solve the problem over the phone. If more research is required, theissue is escalated to Tier 2 for research and resolution. If the problem.is beyond known fixes, or is notsolved within prescribed guidelines, the service request will be escalated to the Tier 3 support staff.

Problems that are escalated to our Tier 3 support staff are beyond basic setup and usage issues. Theseissues may need programming involvement and/or code changes to resolve. These problems may also becomplex software upgrade issues, customer application issues, or core product issues that need to beresearched, duplicated, and tested in a support lab. If code changes are required, the service request isescalated to our development staff for resolution.

Support Offerings

Redstone Technical Support Services offer several support plans to reflect different customer.needs:

* Standard Support is designed for customers with limited support needs outside of regular businesshours, with support offered from 8 am to 5 pm MST.

* Extended Support expands Redstone's basic support option to 5 am to 5 pm MST.* Premium Support provides access to Redstone support personnel 24 hours, a day, 7 days a week,

for mission critical issues.* Should a client sign up for a, lower level of support and find they need technical assistance outside

their support "window," support will be provided and billed to the customer in a separate billing. At thattime, Redstone will give the customer the option to upgrade to a higher level of support, or just paythe current support invoice.

Project Management

return to top

Project Management at Redstone follows the industry-standard, process-oriented approach.as outlined bythe SEI-CMM and Project Management Institute (PMI).

Based on the findings from the Scoping Study, and the information in the Requirements and FunctionalSpecification, the Redstone Project Manager will develop a Project Timeline using these documents as abaseline. The Timeline will be an outline of tasks, necessary meetings, and give time estimates for allphases of the project and all deliverables.

This Timeline should give the customer a +/- 10% estimate of how much work will be needed to completethe project.

Throughout the course'of the project, the Project Manager ensures:

* Adherence to the timeline and processes* Tracking of any problems or issues encountered* Efficient use of resources* Adherence to the principles of SQA* Clear communications with the client

1.6 Configuration Management

return to top

All project documentation, user documentation, system documentation, and code is version-controlled withconfiguration management software. Currently, Redstone uses CVS, Concurrent Versioning Software.


Page 12 of 13

.110/2 0/20101

RedstonIIntIIIrted Solution I Products A S I PageIIII13ofIII13-

EIE BuLLProcess

Stored Files/Builds/Releases

Configuration Management Software

InstaiLationProvess

© Redstone Integrated Solutions, 2003. All Rights Reserved.

http://web.archive.org/web/20040 606014343/redstone-is.com/products_services.html

iWorkingFoders/Files

Install Kit Buld

Page 13 of 13Redstone Intergrated Solutions - Products And Services

I-

10/20/201.0°

Redstone Integrated Solutions - Camera Monitoring with VisualPursuit

Home

Products

VisualPursuitServices

Portfolio

News-

Contact Us VisualPursuit:, Camera Monitoring / Facility Security Systems

Redstone's VisualPursuit enhances an operator's effectiveness by allowing the operator.to easily track amoving subject throughout a facility from a single monitor. It accomplishes this .by consolidating thevideo from physically adjacent surveillance cameras into one screen, increasing the field of view of aphysical location dramatically.

VisualPursuit is a part of the Redstone family 'of software and when used with the full system, addssubject-tracking and wide-area monitoring capabilities'to an operator's toolkit.

VisualPursuit's display combines the output from several surveillance cameras into a single, MSWindows-standard screen.

* The VisualPursuit screen contains a Subject View and Locator Views.* The Subject View displays the current location of the moving subject.* Up to 8 Locator Views display the output of the cameras closest to the one trained on the subject.* Track a subject by clicking the Locator View that the person has moved into. The Locator View

becomes the new Subject View, and all Locator Views are automatically reassigned and updated.

VisualPursuit works with Redstone's MapTrack or ResourceTrack products:

* Click a camera symbol on MapTrack or ResourceTrack to start a VisualPursuit session.* MapTrack's camera symbols are color-coded so that you always know what cameras are being

used in VisualPursuit.* From MapTrack, monitor and control devices at the same time you're engaged in a VisualPursuit

(lock doors, zoom cameras, etc).

Track the subject's location and direction of movement on MapTrack's interface.

VisualPursuit is an important element of Redstone's Intelligent Facilities package. VisualPursuit makesan operator more effective and efficient by allowing her to easily track a moving subject using onemonitor. It also allows her to quickly scan an area covered by security cameras without having to viewseveral monitors.

Applications of VisualPursuit include commercial office buildings, cities, airports, hospitals, sportsfacilities, prisons, pipeline companies, refineries, and convention centers to name a few. Basically, anyarea that needs better surveillance and/or emergency management will benefit from the installation ofRedstone's VisualPursuit.

http://web.archive.org/web/20041011132522/redstone-is.com/camera _monitoring.html

Page 1 of 2

10/20/2010

Redstone Integrated Solutions - Camera Monitoring with VisualPursuit

VisualPursuit can be used to monitor and/or control:

* Facilities: casinos, housing complexes, office buildings, convention centers, governmentalbuildings, prisons, corporate campuses, high-rises

* Airports: access and security* Educational Campuses:* Complex. Infrastructures such as chemical plants, pipelines, manufacturing complexes, city

intersections* Entertainment complexes: amusement parks, zoos, sports facilities, museums

VisualPursuit in Action:

* The picture to the right is a screenshot of the Visual Pursuit interface. This view shows nine f-cameras. The large, central window contains the output from the current main tracking I--icamera; other cameras in proximity to the current tracking camera are displayed in the other eightwindows. By clicking on one of these adjacent views, you can change it to be VisualPursuit's maincamera view. The adjacent views would then automatically update based on the new currenttracking camera. This is an example of how you would use it to follow a person through a facility.

* A series of LEDs give you an idea of where the current tracking camera is, in relation to -- .another camera. . J

* There is also a map layout that can assist you withvisualizing exactly where each camera is in your facility.<< Click lHere for close up shot >>

Back to Products

I I

C Redstone Integrated Solutions, 2004. All Rights Reserved.

http://web.archive.org/web/2004 1011132522/redstone-is.com/camera_monitoring.html

Page 2 of 2

I

10/20/2010 ;

Redstone Integrated Solutions - Building Automation with MapTrack

Home

Products

MapTrack .Services

Portfolio

News

Contact Us MapTrack: Building Automation

Redstone's MapTrack is an easy-to-use, graphical interface. Users can quickly locate, view, and/orcontrol computer systemns, video, security cameras, audio, card access readers, doors, and a host of otherdevices, all from a real-time, interactive map interface. MapTrack can display maps or diagrams at thelevel of detail required by the user.

MapTrack is a part of the Redstone family of software products and when used with the full system, addsdigital, interactive map and device control capabilities to an.operator's toolkit.

MapTrack's display combines digital maps or diagrams with graphical symbols that can represent thelocations of items such as:

* security, cameras, doors, and windows* computers/components in an equipment rack* valves on a pipeline* traffic and railroad signals* alarms (motion detectors, fire alarms, etc.)* steps in a process diagram* and many other devices

These graphical symbols are more than simple static representations, they can:

* display a device's status (alarming, locked/unlocked, on/off) by a change in color or by flashing. allow device interaction and control* allow viewing of device output* give the operator a quick entry into another Redstone module such as VisualPursuit.

MapTrack is an important element of Redstone's Intelligent Facilities package. MapTrack makes .an operator more effective and efficient by using a single, real-time, map-based interface through . --which an operator can monitor and respond to events' over an entire facility - all from-one or manyworkstations. The look and feel of this interface can be customized to suit the needs of the client.«.Cl ck Here for. closeup.shot.."

Applications of MapTrack include commercial office buildings, cities, airports, network operationscenters, hospitals, sports facilities, prisons, pipeline companies, refineries, and convention centers toname a few. Basically, any area that needs better facilities management, surveillance, and/or emergencymanagement will benefit from the installation of Redstone's MapTrack.

http://web. archive.org/web/2004 1011131 044/redstone-is.com/building_automation.html

Page 1 of 2

10/20/2010

Redstone Integrated Solutions - Building Automation.with MapTrack

MapTrack can be used to monitor and/or control:

* Facilities: access and security, climate control, emergency systems, lighting* Refineries and chemical plants: security, emergency response, product delivery, product status* Pipelines: emergency response, security, product delivery, product status, cathodic protection* Networks and hardware: system status, hardware status, connectivity

Redstone's software allows for remote monitoring of numerous buildings, complexes, and parkingstructures by the property owner for the purpose of reducing liability, reducing operating costs,increasing safety and security, reducing loss of assets, and increasing the owner's capability toproactively or quickly respond to a variety of incidents, ranging from simple facility maintenance issuesto criminal activity. It also offers the option to hand-off control of subsystems to emergency responseteams during an event (if so equipped.)

«< Click here for a.close up shot >

Back to Products

© Redstone Integrated Solutions, 2004. All Rights Reserved.

http://web.archive.org/web/20041 01 I1131044/redstone-is.com/building_automation.html

Page 2'of 2

I D

10/20/2010


In re U.S. Patent No.: 7,777,783

Inventors: Hu Chin et al.

Issue Date: August 17, 2010

Serial No.: 11/726,879

Filing Date: March 23, 2007

Attorney Dkt. No.: 017874-0003

For: MULTI-VIDEO NAVIGATION

Mail Stop Inter Partes ReexamDirector of the U.S. Patent and Trademark OfficeP.O. Box 1450Alexandria, VA 22313-1450

REQUEST FOR INTER PARTES REEXAMINATION OF U.S. PATENT NO. 7,777,783PURSUANT TO 35 U.S.C. § 311 AND 37 C.F.R. § 1.913

Dear Sir:

Inter Partes Reexamination under 35 U.S.C. § 311 et seq. and 37 C.F.R § 1.913

et seq. is requested for United States Patent Number 7,777,783 (the '783 patent) which

issued on. August 17, 2010 to Chin et al. from an application filed March 23,'2007. This

Request is accompanied by the fee for requesting inter partes reexamination set forth in

37 C.F.R. § 1.20(c)(2). The United States Patent and Trademark Office is hereby

authorized to charge any fee deficiency,, or credit any overpayment, to our Deposit

Account No. 03-2469.

DC' 0/YHA)IA D1/436261.1

Attorney Dkt. No. 017874-0003Reexamination Request

U.S Patent No. 7,777,783

TABLE OF CONTENTS

i. STATEMENT REGARDING PENDING LITIGATION............................. ..... 1

II. INTRO D U CT IO N ......................................................................... ... 1,

III. PROSECUTION OF THE APPLICATION FOR THE '783 PATENT...............6

IV. CITATION OF PRIOR ART PURSUANT TO 37 C.F.R. § 1.915(b)(2). ............. 17

V. STATEMENT OF SUBSTANTIAL NEW QUESTIONS OFPATENTABILITY PURSUANT TO 37 C.F.R. § 1.915(b)(3).AND DETAILED EXPLANATION...................... ...... ............ .... 19

A. Proposed 35 U.S.C. § 102 Rejections...........................................19

1. Girgensohn '706 Reference . .............. 19

2. W hite Reference............ ...... .................. .................... .... 24

3. Roberts Reference ............... ........... .......... .28

4. Redstone Reference............. ..... .......... .. .............. 33

B. Proposed 35 U.S.C. § 103 Rejections............................................36

1. Girgensohn '706 Reference ......................................... 37

a. Girgensohn '706 Reference in combinationwith Lin and Girgensohn '978 References......................37

b. Girgensohn '706 Reference incombination with Gormley Reference ........ ............. 42

2. Roberts Reference ................. .. ........................ 44

a. Roberts Reference in combinationwith Girgensohn '706 and GormleyReferences.............. ..................... . ................. 44

DC(.'1/YOllAD/436261.1



b. Roberts Reference in combination withGirgensohn '978 and Lin References........................47

3. Redstone Reference .......................... .................... 51

a. Redstone Reference in combination withBalancing Tomorrow's Technology, RedstoneIntegrated Solutions reference ................... ............. 51

b. Redstone References in combination withGirgensohn '706 and Gormley References.....................53

c. Redstone References in combination withGirgensohn '978 and Lin References ............... ......... 55

4. White Reference......................... ................... 58

a. White Reference in combinationwith Girgensohn '706 and GormleyReferences............................................. .... .58

b. White Reference in combination withGirgensohn '978 and Lin References ................ .......... 60

5. Maruya Reference ........ ..... ..................... :...................63

a. ,Maruya Reference in.combination withGirgensohn '706 and Gormley References.... .............. 63

b. Maruya Reference in combinationwith White, Girgensohn '978and Lin References .......... ... . . ........... .... ...... 67

c. Maruya Reference-in combinationwith Roberts, Girgensohn '978and Lin References .......... ............. ... 69

d. Maruya Reference in combinationwith Redstone, Girgensohn '978and Lin References................ .............. 71

S.'01/YO ID/4 3.6261 .1



6. Buehler Reference ......................................... .. 73

a. Buehler Reference in combination withGirgensohn '706 Reference............................ ..... 73

b. Buehler Reference in combinationwith White, Girgensohn '978and Lin R eferences ................................ ................. ... 77

c. Buehler Reference in combinationwith Roberts, Girgensohn '978and Lin References ......... ..................80

V I. C O N C LU S IO N ...................................................... . ................... 83

VII. LIST OF EXHIBITS AND APPENDICES..................................................83

DC01/YOIIA D/436261.1


U.S Patent No. 7;777,783

1. STATEMENT REGARDING PENDING LITIGATION

The '783 patent is currently the subject of litigation in the United States.District

Court for the Eastern District of Virginia. The case is styled: VidSys, Inc., Plaintiff v.

Proximex Corporation, Defendant, United States District Court for the Eastern District of

Virginia, Civil Action No. 1:10cv1185. The reexamination Requester, VidSys, Inc., is the

plaintiff in this litigation, and the patent owner, Proximex Corporation, is the defendant.

The subject litigation is in its very first stage in that the Complaint has been filed with the

Court, however the Complaint has not yet been served and the defendant has not

responded to the Complaint. It is believed that the '783 patent has not been adjudged

invalid or unenforceable at this time and thus remains in force.

II. INTRODUCTION

The '783 patent relates to a system which includes a plurality of video cameras

and a video tracking interface adapted to display video streams provided by the

cameras. The system may be used for surveillance of an area using multiple cameras

which have distinct fields of view relative to each other. The primary focus of .the patent

is on the video tracking interface and its arrangement and functionality.

The video tracking interface includes user controls associated with each of the

cameras. These user controls may be activated by "clicking" on them with a mouse or

touching them, for example, to select an associated camera to provide a video stream in

I)C01/YOI1A D/436261.1



a main video display. See '783 patent, 7:40-43.) In the Figure 3 embodiment of the

'783 patent the user controls 310A-310H are provided around, i.e., at a position dutside,

and along a perimeter, of the main video display 305. See '783 patent, Fig. 3. In some

embodiments, the user controls may include an optional thumbnail of video data

generated using the associated camera. See id., 8:28-33. Figures 4A, 4B and 4C

illustrate alternative arrangements of the user.controls on the video tracking interface

screen wherein the main video display itself may have user control regions within it

which may be clicked upon to select. a new camera to provide the video stream for the

main video display. See id., 7:43-63 and Figs. 4A-4C. In the Figure 4 embodiments,

the user controls are not provided around the main video display, but are instead

contained within the main, video display.

The arrangement of the user controls on the video tracking interface may be

representative of the topological relationship of the cameras associated with the main

video display and the user controls. See id., 3:51-4:24. The representative topological

relationships may be derived from GPS locations, for example, taken from the viewpoint

of the camera which is currently providing the video stream for the main video display.

If a user control is selected so that a new camera begins providing the video stream for

the main video display, the system may re-associate each of the user controls with one

or more of the other cameras so that the arrangement of the user controls is

1 The nomenclature of "X:Y" is used to refer to column number "X" and line number "Y" in U.S. patentprior art references.I.)CO1/YOHIAD/436261 . 2

Attorney Dkt, No. 017874-0003Reexamination Request


representative of the topological relationship of the cameras taken from the viewpoint of

the new camera. See id., 7:3-20. The re-association' process may be repeated each

time a new camera is selected to provide the video stream for the main video display.

The re-association process may result in some of the user controls being unused

if no camera,has a topological relationship with the "new" camera that corresponds to

the location of those particular User controls on the screen. See id., 6:43-60. Because

not all user controls will necessarily be required after "re-association" of the user

controls with the cameras, it follows that active user controls may not always be

provided along each of the four sides of the main video display after re-association. In

fact, one of ordinary skill in the art would readily understand that there may be a "re-

association" that results in as few as a single active user control being provided on only

one of the four sides of the main video display.

With reference to Figure 3, the '783 patent explains the usefulness of the,

foregoing functionality as follows:

In various embodiments, the association of User Controls 310 withmembers of Cameras 105 based on camera topology results in an intuitiveinterface for tracking of targets between video camera. For example, if thetarget moves to the right out of the field of view presented 'in Main VideoDisplay 305, then User Control 310B can be used to select the member ofCameras 105 whose field of viewthe target-is. most likely to be found next.As the target moves from camera to camera, a user can select thosemembers of User Controls 310 adjacent to the edge of Main Video Display305 at which the target disappeared.

'783 patent, 6:6-16.

DC1 /YOI IAD)/436261. I


U.S Patent No. 7,777,783'

Recorded segments of video streams may also be automatically stitched or

spliced together by the system to form a continuous sequence of video data taken from

different cameras. See id., 9:27-42. The stitched together sequence of video may be

used to track a target as it moves through the fields of view of the different cameras that

capture video of the target. The automatic stitching together of video streams may

further involve use of a preset "temporal offset." The '783 patent describes the use of

this temporal offset as follows:

The synchronization between video. data optionally includes anoffset selected to approximate the time a target would be expected totravel between members of Cameras 105. For example, if a typical traveltime between the fields of view of Cameras 105G and 105D is 15seconds, then a 15 second offset is optionally automatically used whensynchronizing video data from Camera 105G and Camera 105D. If a useris viewing video data from Camera 105G in Main Video Display 305, anduses User Control 310B to select video data generated by Camera 105D[starts by viewing video data generatedusing Camera 105D], then thevideo from Camera 105D will be offset by 15 seconds. This offset mayenable the user to see the target immediately in the field of view ofCamera 105D without having to wait for the target to travel between fieldsof view.

'783 patent, 9:5-19.

The system may also convert video data from one format to another as needed,

store video data in association with information that identifies. the camera used to

generate the video data, and utilize on-screen "thumbnail" views to provide an index of

stitched video.

The points of novelty expressed in independent Claims 1 and 5 of the '783 patent

- namely, placement of the above-described user controls around a main video displayDC1/YIOHAD/436261.1 . 4



in locations representative of the topological relationship of the cameras providing the

video, and use of an automatic "temporal offset" when stitching together video from

different cameras - were known in the art prior to the filing of the application for the '783

patent. In fact, theformer point of novelty, concerning the placement of user controls,

was known more than seven years before the filing date of the application for the '783

patent, as will be explained below.

This Request for inter partes reexamination involves ten (10) different prior art

references. Only one (1) of the prior art references relied upon for this Request was

cited during the prosecution of the application for the '783 patent. Four of these

references anticipate Claim 1 of the '708 patent under 35 U.S.C. § 102 and one of these

references anticipates all of the claims. None of these anticipatory references was

before the Office during examination of the '783 patent application. Furthermore,

combinations of the ten different.prior art references render Claims 1-8 obvious under

35 U.S.C. § 103 in a number of different combinations. Accordingly, the submitted

references raise substantial new questions of patentability.

In view of the foregoing, and pursuant to 37 C.F.R. § 1.915(b)(1), inter partes

reexamination is hereby requested for Claims 1-8 of the '783 patent. Pursuant to 37

C.F.R. § 1.915(b)(8), the real party in interest is VidSys, Inc., a Delaware corporation',

having a place of business at 8219 Leesburg Pike, Suite 250, Vienna, VA 22182

("Requester"). Pursuant to 37 C.F.R. § i.915(b)(7), Requester hereby certifies.that the

estoppel provisions of 37 C.F.R. § 1.907 do not prohibit this inter partes reexamination.1C01 /Y01.A D/436261.1 5.



Pursuant to 37 C.F.R. § 1.915(b)(5), a copy of the entire '783 patent is attached hereto

as Exhibit A, including the front face, drawings, and specification/claims (in double

column format).

III. PROSECUTION OF THE APPLICATION FOR THE '783 PATENT

A. Original Claims 1-27

The '783 patent issued on August 17, 2010, based on U.S. Application Serial

Number 11/726,879 (the '879 application) filed on March 23, 2007, by purported

inventors Hu Chin and Ken Prayoon Cheng. Thus, March 23, 2007 constitutes the

priority date against which prior art is measured (the "critical date"). The '783 patent is

entitled "Multi-Video Navigation."

As originally filed, the application for the '783 patent included twenty-seven (27)

claims, of which Claims 1, 7 and 24 were independent. Independent Claims 1, 7 and

24, as originally drafted, claimed:

1. A method comprising:

identifying a first member of a plurality of cameras;

identifying a first topological relationship between the firstmember of the plurality of cameras and a second member ofthe plurality of cameras;

identifying a second topological relationship between the firstmember of the plurality of cameras and a third member ofthe plurality of cameras, the first. topological relationshipbeing relative to a viewpoint of the first camera;,

presenting video data. generated using the first member ofthe plurality of cameras in a user interface, the user interface

DCOI/Yo AI)/4362 I .1 6



including a first user control and a second user control;.

associating the second member of the plurality of cameraswith the first user control based on the topologicalrelationship between the first member of the plurality ofcameras and the second member of the plurality of cameras;

associating the third member of the plurality of cameras withthe second user control based on the topological relationshipbetween .the first member of the plurality of cameras andthird member of the plurality of cameras;

receiving a user selection of the first user control;

presenting video data generated using the second memberof the plurality of cameras in the user interface based on thereceived user selection; and

associating the first user control and the second user controlwith members of the plurality of cameras based ontopological relationships relative to a viewpoint of the secondmember of the plurality of cameras.


viewing first video data within a user interface, the first videodata generated using a first member of a plurality of,cameras;

locating a target within the first video-data;

observing the target leave a field of view of the first memberof the plurality of cameras in a first direction relative to aviewpoint of the 'first member of a plurality of cameras, orobserving the target enter the field of view of the firstmember of the plurality of cameras from the first directionrelative to a viewpoint of the first member of the plurality'ofcameras;

selecting a second member of the plurality of cameras usinga first user control associated with the first direction. relative,to a viewpoint of the first member of the plurality of cameras;and

viewing second video data within the user interface, the

DC O/YOH A v/4 6261.1 7



second video data generated using the second member ofthe plurality of cameras.


receiving a user selection of first video data generated usinga first camera, the first video data including a first timereference;

receiving a user selection of second video data generatedusing a second camera, the second video data including asecond time reference;

automatically stitching the first video data to the secondvideo data responsive to the user selection of the, secondvideo data and using the first time reference and the secondtime reference to form a chronological video sequence; and

creating an index to the video sequence based on identitiesof the first camera and the second camera.

See March 23, 2007 Patent Appl. No. 11/726,879, at 34-35.

The following discussion of the prosecution history of the claims focuses

particularly on prosecution of Claims 1-3, 7, 16 and 18. Claims. 1-3 are of interest

because the subject matter of these claims was eventually consolidated in order to gain

allowance of issued Claim 1. Claim 18 is also of particular interest, as the subject

matter of Claim 18 was consolidated with that of base and intervening Claims 7 and 16

in order to gain allowance of the other independent issued claim (Claim 5). Many of the

other claims in the application were ultimately abandoned.

DCOI/Y ()o IA /4362 6 1.

Attorney Dkt. No..017874-0003Reexamination Request


Original claims 2 and 3 were dependent on claim 1, and claimed:

2. The method of claim 1; further including presenting thefirst user control and the second user control in the userinterface in positions representative of the first topologicalrelationship and the second topological relationship.

3. The method of claim 2, where in the first user control andthe second user control are presented in positions around apresentation of the video data generated using they firstmember of the plurality of cameras.

Original claims 16 and 18 were dependent on claim 7, and claimed:

16. The method of claim 7, further including automaticallystitching the first video data and the second video data toform a video sequence.

18. The method of claim 16, further including using atemporal offset to automatically stitch the first video data andthe second video data, the temporal offset being based onan estimated time of travel of the target.

B. The First Rejection and Response

In the first Office Action, the Examiner rejected all of the claims as being obvious

either over U.S. Pub. No. 2007/0146484A1 to Horton et at. ("Horton") alone, Horton in

view of U.S. Pub. No. 2002/0097322A1 to Monroe et al. ("Monroe"), or Horton in view of

US Pub. No. 2008/0068464A1 to Kitagawa et al. ("Kitigawa").. See May 20, 2009 Office

Action ("First Office Action"),.passim.

The applicants did not amend any of the rejected claims in response to the First

Office Action. Instead, the applicants presented only remarks in an attempt to traverse

the rejections. Notably, with respect to Claim 1, the applicants contended that HortonDCOI/YO1IAD/436261.1 9



did not teach the "step of identify[ing] a topological relationship." .June 8, 2009

Amendment A ("First Remarks"), at 12-13. The applicants attempted to distinguish

Horton by arguing that "a relationship between a camera and a target, as in Horton et

al., cannot read on a topological relationship between two cameras." Id. at 13. The

applicants also contended that "Horton et al. does not teach 'selecting a member of the

plurality of cameras using a first user control associated with the first direction relative to

a viewpoint of the first member of the plurality of cameras.'" Id. at 13.

C. The Second Rejection and Response

In a second Office Action, the Examiner again rejected all of the pending claims.

U.S. Pub. No. 2003/0202102A1 to Shiota et al. ("Shiota") replaced Horton as the

primary reference upon which the rejections were based. In the Second Office Action,

both anticipation and obviousness were asserted using Shiota alone, or in various

combinations with Monroe, U.S. Pub. No. 2004/0001149A1 to Smith ("Smith"), U.S.

Patent No. 7,456,727 to Pinter et al. ("Pinter"), U.S. Patent No. 7,525,576 to

Kannermark. et al. ("Kannermark"), and U.S. Pub. No. 2006/0221184A1 to Vallone et al.

("Vallone"). Sept. 21, 2009 Office Action ("Second Office Action"), at 16.

With respect to Claim 1, the Examiner concluded that Shiota disclosed each

element of the claim except the requirement "that the user interface includes the first

and second user controls for controlling each individual camera." Id. at 14-16. For the

same reasons as set forth in the First Office Action, the Examiner again concluded that

Monroe filled this gap, rendering Claim 1 obvious. Id. at 16.D)C0I/YOIIAI)/436261.1 1 0



The applicants responded.to the Second Office Action by again only submitting

remarks and not amending any of the claims other than to correct for grammatical

errors. The applicants argued that Shiota did not teach or suggest identifying

topological relationships between cameras as required by Claim 1. Dec. 21, 2009

Amendment B ("Second Remarks"), at 17. The applicants agreed that Shiota disclosed

"cameras pointed in different directions and that spatial relationships exist between

them." Id. According to the applicants, however, "nowhere in Shiota et al. are such

relationships noted or utilized" and they were therefore "irrelevant to the operation of the

system of Shiota et al." Id. As an example, the applicants argued that Shiota "does not

identify the angle between the viewpoints of any two cameras, and does not use such

information to direct the cameras or process images." Id. at 17-18. The applicants

argued that Monroe was irrelevant for the same reason. Id. at 18.

The applicants also argued (as they did with respect to Horton in response to the

First Office Action) that Shiota "does not teach or suggest associating cameras with

user controls based on topological relationships." Id. Specifically, the applicants

argued that "[i]n Shiota et al. there is a plurality of 'display blocks' in the display, one for

each of a plurality of cameras, and then one of two sequences is repeated as in FIGs.

4A and 4B." Id. "Either each display block is dedicated to a particular monitored area

(FIG. 4B), else each display block is dedicated to a particular camera (FIG. 4A)" Id.

According to the applicants, "[t]here is no correlation between the arrangement of the

display blocks and the relationships between cameras" and "[f]urther, the display blocksI')COI/YCII D/43626 .1 11

,, .



aren't even user controls." Id.

The applicants did not address the Second Office Action's rejection of claim 2

based on the combination of Shiota and Monroe. They simply, asserted that since

Shiota and Monroe did not render claim 1 obvious, claim 2 was not obvious either. lId.

at 18-19. The applicants did, however, respond to the rejection of claim 3, as follows:

The Applicants would also like to point out the furtherpatentability of claim[] 3 .... Claim 3 requires that, in themethod of claim 2, the first user control and the second usercontrol are presented in positions around a presentation ofthe video data generated using the first member of theplurality of cameras. The Examiner merely cites to FIGs. 3and 5 of Monroe et al. ([Second] Office action, page 17).Both drawings show a map display area and a video displayarea, and the two areas are side by side. ,The Applicantsassert that neither drawing of Monroe et al. shows usercontrols presented in positions around a presentation ofvideo data.

Id. at 19 (emphasis added).

D. The Final Rejection and Response

In a third and final Office Action, the Examiner rejected the applicants' arguments

regarding patentability. With respect to Claim 1, the Examiner first noted that the

applicants' arguments were based on a construction of claim 1 that was not supported

by the claim language:

There is no mentioning in the claimed language of identifyingthe angle between the viewpoints of any two cameras. Theclaims simply require identifying a topological relationshipbetween the cameras and this is clearly fulfilled by the

I)('OI/YOHAD)/436261.1 12



teaching of Shiota et al.. The Examiner would like to pointout that the Applicant agrees that in Shiota et al. there arecameras pointed, in different directions and that spatialrelationships exist between them. The Examiner also likes.to point out that although the Applicant states that nowherein Shiota et al: are such relationships noted or utilized, theApplicant admits that such relationships are inherentlypresent. It is for this reason that the Examiner maintains thatthe topological relationship between the cameras. exists.There are commonly identified areas of interest presentbetween the cameras.

March 9, 2010 Final Office Action ("Final Office Action"), at 7-8.

For the same reasons as those set forth in the Second Office Action, the

Examiner again rejected applicants' arguments that the combination of Shiota and

Monroe "does not teach 'or suggest associating cameras with user controls based on

topological relationships." Id. at 8.

With respect to Claim 3, the Examiner also disagreed with the applicants'

assertion that Monroe did not disclose "user controls presented in positions around a

presentation of video data." Id. at 9. The Examiner cited the Princeton online

dictionary's definition of the term "around" in concluding that it means "about or in-the

area or vicinity." Id. Under this definition of "around," "it is clear that Monroe et al. teach

this limitation." Id. The Examiner then repeated the rejections of Claims 1-3 based on

Shiota in view of Monroe that were presented in the Second Office Action. See Final

Office Action at 22-24.

I)('01/YOlAD/43621.1 13



E. Interview and Subsequent Remarks By Applicants

After the Final Office Action, an interview was held between applicants' counsel

and the Examiner. See April 6, 2010 Interview. Summary. During the interview,

agreement was reached with the Examiner that "[r]egarding independent claim 1, Shiota

et al. did not teach identifying a first topological relationship between the first and

second cameras of the plurality of cameras." Id. Claims 2-3 were not discussed during

the interview. April 8, 2010 Amendment C ("Third Remarks"), at 8.

Applicants' Third Remarks requested allowance of claims 1, 2 and 3 since claim

1 was allowable:

In view of the April 6 interview, claim 1 is allowable.Accordingly,. the Applicants request that the Examinerwithdraw the rejection of claim 1, and the rejections of claims2-6 depending therefrom, under 35 U.S.C. § 103(a).

Id. at 10.

F. Subsequent Office Action, Amendment and Notice of Allowance

.In response to Applicants' Third Remarks, the Examiner issued a non-final Office

Action. In this Office Action, the Examiner indicated that he never agreed during the

interview that Claim 1 contained allowable subject matter. See May 14, 2010 Office

Action ("May 2010 Office Action"), at 3. In this Office Action, the Examiner rejected

original Claims 1, 2, 7 and 16, among others, on new grounds, i.e., as being anticipated

by U.S. Pub. No. 2010002082A1 to Buehler et al. ("Buehler"), or obvious over Buehler

in view of U.S. Pub. No. 2005/0052532A1 to Elooz et al. ("Elooz"). Id. at 4-6, 14.CI)oYOoHAr/43261.I . 14

Attorney Dkt: No. 017874-0003Reexamination Request


Claims 3 and 18, however, were objected to as being dependent on a rejected base or

intervening claim(s), but not rejected.

Claim 2 was rejected as obvious over Buehler in view of Monroe Id. at 11.

Here, the Examiner found that Buehler disclosed all elements of Claim 1, however it did

not disclose "that the user controls are physically in positions representative of the first

topological relationship and the second topological relationship." As expressed in the

First Office Action, however, Monroe filled this gap. Id. at 11-12.

With respect to claim 3, the Examiner noted that it was "objected to.as' being

dependent upon a rejected base claim', but would be allowable if rewritten in'

independent form including all of the limitations of the base claim and any intervening

claims." Id. at 22., Claim 3 added the limitation that "the first user control and the

second user control are presented in positions around a presentation of the video data

generated using the first member of the plurality of cameras." See Third Remarks, at 3

(emphasis added).

With regard to Claim 16, the Examiner explained that Buehler taught all, elements

of the claim except for "automatically stitching the first and second video data." Id. at

14-15. Elooz was asserted to teach automatically combining portions of videos, and it

would have been obvious to combine the automatic stitching of Elooz with the teachings

of Buehler. Id. at 16.

Like with Claim 3,."with respect to claim 18, the Examiner noted that it was

"objected to as being dependent upon a rejected base claim, but would be allowable ifD(:01 /YOl IA I)/436261.1 15

Attorney Dkt, No. 017874-.0003Reexamination Request


rewritten in independent form including all of the limitations of the base claim and any

intervening claims." Id. at 22. Claim 18 added the limitation of "using a temporal offset

to automatically stitch the first video data and the second video data, the temporal offset

being based on an estimated time of travel of the tarqet." See Third Remarks, at 6

(emphasis added).

In response to the May 2010 Office Action, the applicants cancelled Claims 2, 3,

7 and 18, among others. See May 24, 2010 Amendment D ("Fourth Remarks"), at 3-4.

Simultaneously, the applicants amended Claim 1 to incorporate the limitations of

original Claims 2 and 3, and amendedClaim,16 to incorporate the limitations of original

Claims 7 and 18., Id. at 2-3, 6.. The applicants did not traverse the rejection of any of

the claims other than to state that the "[n]on-allowed claims that are cancelled herein

are cancelled merely to expedite prosecution and their cancellation should not be

construed as.an admission of unpatentability." Id. at 6.

The Examiner subsequently issued a Notice of Allowance dated June 18, 2010

("Notice of Allowance"). In the Notice of Allowance, the Examiner indicated that the only

reason for issuance was the combination of all elements of original Claim 1 With the

elements of original Claims 2 and 3,.and the combination of all elements of original

Claim 16 with the elements of original Claims 7 and 18. Notice of Allowance, at 2-3.

The addition of the language from dependent Claims 3 and 18 to the allowed claims

was highlighted in bold text, indicating that this language was considered necessary for

allowance. Buehler was cited, as the closest prior art to all issued claims. Id. at 5.I)C01/YOlIA)/436261.I 16



According to the Examiner, Buehler disclosed "methods and systems for creating video

from multiple sources to utilize intelligence to designate the most relevant sources,

facilitating their adjacent display and/or catenation of their video streams." Id.

IV. CITATION OF PRIOR ART PURSUANT TO 37 C.F.R. § 1.915(b)(2)

Based on the critical date of the '783 patent,, each of the prior art references

listed in the chart below qualifies as prior art under 35 U.S.C. §§ 102(a), 102(b) and/or

102(e). Pursuant to 37 C.F.R. § 1.915(b)(4), a copy of each patent or printed

publication relied upon or referred to is provided with this submission as Exhibits B-K.

The chart below also identifies the short cite for each reference, the statutory basis

under' which the reference is applied, and the claims to which the reference is applied

individually or in combination with other references.

SStatutory Claim(s)Exhibit Reference Short Cite Basis Applied To

U.S. Patent Appl. Pub. No.2008/0088706 Al to Girgensohn Girgensohn

B et al. filed Oct. 13, 2006 and '706 § 102(e), 1-8published Apr. 171 2008 Reference

U.S. Patent No. 7,196,722 to

C White et al.issued Mar. 27, 2007 White § 102(e), 1-8and filed May 18, 2001 Reference § 103(a)

U.S. Patent No. 7,295,228 to

Roberts et al. issued Nov. 13, Roberts § 102(e), 1-82007 and filed Oct. 29, 2002 Reference § 103(a)

I_ . _

DCO1/YOA I)/436261 .1


U.S Patent No: 7,777,783

Exhibit , Reference Short Cite Statutory Claim(s)Basis Applied To

http://web.archive.org/web/20040 Redstone606014343/redstone- Products_ § 102(b), 1-8is.com/products_services.html Services § 103(a)

ReferenceU.S. Patent No. 7,242,423 to Lin

F issued Jul. 10, 2007 and filed Lin § 102 (e) 58Aug. 8, 2003 Reference § 103(a)

U.S. Patent Appl. Pub. No.2006/0284978 Al to Girgensohn Girgensohn § 102 (a)

G et al. filed Jan. 3, 2006 and '978 § 102 (e) 3-8published Dec. 21, 2006 Reference § 103(a)

U.S. Patent No. 5,258,837 toGormley issued Nov. 2, 1993 and Reference § 102 (b) 2filed Oct. 19, 1992 Reference § 103(a)Balancing Tomorrow's SecondaryTechnology, Redstone Integrated Redstone § 102 (b) 1-8Solutions, @ 2003 Reference § 103(a)

U.S. Patent No. 7,746,380 to Maruya § 102 (e)Maruya § 102 (e) 1-8J Maruya et al. issued Jun. 29, Reference § 103(a)

2010 and filed Jun. 15, 2004U.S. Patent Appl. Pub. No.2010/0002082 Al to Buehler et

K al. filed Mar. 24, 2006 and Refrence §1-8Refrence J 103(a)

published Jan. 7, 2010"

1'.

I)C01/YOI IAD/43626 I. I



V. STATEMENT OF SUBSTANTIAL NEW QUESTIONS OF PATENTABILITYPURSUANT TO 37 C.F.R. § 1.915(b)(3) AND DETAILED EXPLANATION

A. Proposed 35 U.S.C. § .102 Rejections

1. Girgensohn '706 Reference

Claims 1-8 of the '783 patent are anticipated by U.S. Patent Appi. Pub. No.

2008/0088706 Al to Girgensohn et al. (the "Girgensohn '706" reference) (Exhibit B).

The Girgensohn '706 patent application was filed on October 13, 2006. As a result, the

Girgensohn '706 reference qualifies as prior art against the '783 patent under 35 U.S.C.

§ 102(e). A detailed comparison of the elements and limitations of Claims 1-8 of the

'783 patent with the disclosure of the Girgensohn '706 reference is provided in Appendix

A, which is incorporated herein by reference.

The Girgensohn '706 reference relates to a method of viewing video from

multiple cameras using a computer interface. See Girgensohn '706, Abstract. In

Figures 3, 6 and 7 of Girgensohn '706, the interface may display a main video stream in

a large central window 310. The main video stream is captured by one of a plurality of

cameras'selected to provide the main video stream. See Girgensohn '706, Abstract

and Para. 0035. With reference to the Girgensohn '706 figure below, "a group of

cameras are chosen that are in close spatial proximity to the main camera," to provide

the smaller thumbnail video streams 330, 350 and 360 that surround the main video

stream 310. See Girgensohn '706, Para. 0036.

.COI/YOlIAI)/436261. I 19



310 z- 330

350

360

Girgensohn '706, Fig. 3.

The placement of the thumbnail video streams 330, 350 and 360 around the

main video stream 310 corresponds to, and is representative of, the topological

relationship of the cameras providing each of the video streams. Namely; "[t]he

[thumbnail] video streams are placed around the main video stream .such that a person

walking out of the field of view of a first camera will likely appear in the field of view of

an adjacent second camera, where the video stream of the second camera in the video

display is in the same direction from the first camera as the direction in which the

person walked out of the field of view of the first camera." Girgensohn '706, Para. 0039.D)C01/YOIIAD/436261. 20


U.S Patent No. 7,777,78:3

The topological relationship of the source cameras is "calculated from the video

streams' cameras physical locations relative to the main camera." Girgensohn '706,

Para. 0041.

Further, a user can select a thumbnail camera view to be the main camera video

by clicking on a video stream in the spatial video display. See Girgensohn '706, Para.

0063. Thus, the thumbnail video streams that are placed around the main video stream

in Girgensohn '706 act as "user controls" because they may be clicked upon to select

the thumbnail video stream to become the main video stream.

It is also inherent that once a new video stream is selected to be the main video

stream, "[t]he [surrounding thumbnail] video streams are-[again] placed around the main

video stream such that a person walking.out of the field of view of a first 'camera will

likely appear in the field of view of an adjacent second camera, where the video stream

of the second camera in the video display is in the same direction from the first camera

as the direction in which the person walked out of the field of,view of the first camera."

See' Girgensohn '706, Para. 0039. If this were not the case,. the Girgensohn '706

system would not function properly to permit .a user to track a target from camera to

camera repeatedly.

The Girgensohn '706 system also stores, and provides format conversion of,

video data. See Girgensohn '706, Para. 0070. Format conversion occurs given that the

Girgensohn '706 cameras utilize JPEG which is a video compression technique that

was well known in.the art prior to the critical date. Accordingly, format conversion occursDCOI/Yo AI/43,261.) . 21



for incoming video to be converted to JPEG. Identification information for each of the

cameras is also maintained by the Girgensohn '706 system, which is evident from the

fact that "the main video stream and the other video streams have colored borders to

indicate the actual source cameras." Girgensohn:'706, Para. 0031.

The Girgensohn '706 system stitches video data from different source cameras

together. The system works with recorded video which may be captured as a result of a

user tracking a target from camera view to camera view, as explained above. During

playback, the system automatically updates the camera views that were chosen by

security personnel when the video was recorded. See Girgensohn '706, Para. 0060 and

0070. The foregoing requires that the video streams-from different cameras be stitched

together.

The Girgensohn '706 system also uses thumbnail views as an index of stitched

together video in that thumbnails are used to provide video that was taken in the past

and future relative to the video displayed in a main, camera video stream. These past

and future thumbnail views may be from any one of a number of cameras, which

requires that the video have been stitched together from different source cameras.

Video streams of these five cameras, including the .main cameravideo stream, will be displayed in four horizontal bands shown in FIG. 1,two above and two below the main ,camera video, stream display 110. Thesecond band 125 and third band 130 show video streams from cameraviews that are between 0 and 30 seconds in the past and future,respectively, relative to the main camera video stream. The top band 120and bottom band 135 show video streams from camera views that arebetween 30 seconds to 3 minutes in the past and future,. respectively,

DC01 /YOIIAD/:)43626 1.



relative to the main camera video stream. In embodiments, these rangescan be varied as desired.

Girgensohn '706, Para. 0022.

With respect to Claim 5, the Girgensohn '706 system also uses a temporal offset

which is based on the travel time of a target between camera views.

A video stream display 210 from the view of a main camera isselected to be the main video played on the video player. Controls of thevideo player are part of the video display (not shown). Two vertical bands,of video streams 220 and 230 are displayed on either side of the mainvideo stream display 210. A first band 220 to the left of the,.video streamdisplay' 210 shows three camera views that display video streams fromfive seconds prior to the time of the main video stream playing in display210. A second, band 230 to the right of the video stream display 210shows three camera views that display video. streams from five secondsafter the time of the video stream playing in display 210. A three to tensecond offset from the time of the main video stream display playinqperforms best for tracking people walking from camera view to cameraview. In embodiments, this offset can be varied as desired. The refreshrate of images in these video streams can be varied to produce effectsranging from almost fluid video playback to images that remain steady forseconds or even minutes.

Girgensohn '706, Para. 0030 (emphasis added).

In summary, the.Girgensohn '706 system provides the exact same functionality

as described and claimed in Claims 1-8 of the '783 patent. Just as with the '783

claimed invention, "[u]sers [of the Girgensohn '706 system] such as security personnel

can browse between video cameras by clicking on any of the video streams of the

temporal and geographic displays described above, such that the main playing video

stream is switched to the camera and playback position corresponding to the video

stream selected. . .Using this technique, users can follow activity from camera view toI)C:OIv/YOIHAD/136261 23



camera view." See Girgensohn '706, Para. 0060. Accordingly, Girgensohn '706 raises

a substantial new question of patentability and anticipates Claims 1-8 of the '783 patent.

2., White Reference

Claim 1 of the '783 patent is anticipated by U.S. Patent No. 7,196,722 to White et

al. (the "White" reference) (Exhibit C). The application for the White patent was filed on

May 18, 2001 and the patent issued on March 27, 2007. The White reference qualifies

as prior art against the '783 patent under 35 U.S.C. § 102(e). A detailed comparison of

the elements and limitations of Claim 1 of the '783 patent with the disclosure of the

White reference is provided in Appendix B, which is incorporated herein by reference.

The White reference discloses a system for capturing multiple images from

multiple cameras and selectively presenting desired views to a user on a display or user

interface. Multiple streams of video data are streamed to the user interface. One type of

data stream (called a thumbnail stream) is used to tell the user what video image

streams are available. For this type of video stream, each image is transmitted as a low

resolution thumbnail. One thumbnail is transmitted for each camera and the thumbnails

are presented as small images on the users screen. Another data stream (called the

focus stream) contains a series of high resolution images from a selected camera. The

images transmitted for the focus stream is displayed in a relatively large area, in the

center of the user's display screen. See White, 2:3-16 and Fig. 9.

DC('1/YO(HAD/43626 .1



In Figure 9 of White, six thumbnail images are presented on the display screen

which correspond to six different camera views. Each of these thumbnails is presented

in a location on the display which is representative of the location of the cameras

providing thumbnail streams relative to the viewpoint of the camera providing the focus

stream.

White, Fig. 9. The White reference explains this functionality as follows.

The six images captured by the camera are: a top, a bottom, a leftside, a right side, a front and a;back images (i.e. there is a lens on eachside of a cube):. These, images can be seamed. into a panorama inaccordance with the prior art and stored in a format such as anequirectangular or cubic format. With this alternative embodiment, theuser sees a display such as that illustrated in FIG. 9. At the top center ofthe display is a thumbnail 901 of a panorama. The panoramic image isformed by seaming fs].together into one panoramic image, the individualimages from the six cameras. Six thumbnails of images from the cameras(the top, bottom, left side, right side, front and back of the cube) are shownalonq the right and left edges of the display. If a user clicks on any one of

the focus stream [Ls]

DC01 YOIHAD/436260 1.1

Top Panoramic Image Thumbnail Bottom

Camera 901 Camera

Thumbnail Thumbnail

Left side Right sideCamera' Camera

Thumbnail Thumbnail

Front . Back

Camera Focus CameraThumbnail Image Thumbnail

the six thumbnails, on the right and left of the screen, --~----~~~~~~~~~~-



switched to that image stream as in the first embodiment. It is noted thatwith a panoramic image, it is usual for a viewer to select a view windowand then see the particular part of the panorama which is in the selectedview window. If the user clicks anywhere in the panorama 901, the focusstream is changed to a view window into the panorama which is centeredat the point where the user clicked. With this embodiment, stream controlhas as one input a panoramic image and the stream control selects a viewwindow from the. panorama which, is dependent upon where the userclicks on the thumbnail of the panorama. The image from this view windowis then streamed to the user as the focus image.

White, 7:38-64 (emphasis added).

It is clear from the foregoing that the White thumbnail streams are placed

"around" the central focus stream and are. placed in locations that represent the

topological relationship of the cameras providing the thumbnail streams with the camera

providing the focus stream. For example, Figure 9 shows that the thumbnail stream on

the left of the central focus stream is associated with a "left side camera" and the

thumbnail stream on the right of the central focus stream is associated with a "right side

camera." Moreover, each of the thumbnail streams can be "clicked" upon, i.e., acts as

a "user control," to select that thumbnail stream to become the focus stream.

Each time a new camera is selected to provide the central focus stream, by

clicking on a thumbnail stream for example, the White system reconfigures the overall

display so that surrounding thumbnail streams are presented in positions relative to the

viewpoint of the camera providing the focus stream, which positions are representative

of the topological relationships of the cameras. The White reference explains this

functionality as follows:

)(uO/Y1 eI243661.I 26



[A]t this point the client requests that the focus stream change.This is sent to the server as indicated by arrow 994 [Figure 12]. When theserver receives the command, it stops streaming the old focus stream andstarts streaming the new focus stream as indicated by arrow 995. A newlayout for the user's display is also sent as indicated by -arrow 996: It isnoted that a wide variety of circumstances could cause the server to sendto the client a new layout for the users display screen. When the clientreceives the new display layout, the display is reconfigured.

White, 10:5-15 (emphasis added).

Based on the foregoing, when the left side camera thumbnail is selected by

clicking on it (i.e., by clicking on the first user control), the left side camera thumbnail

view becomes the new central focus stream. This also causes a new layout to be

provided for the user's display. 'This new'layout is "reconfigured," indicating to one of

ordinary skill in the art that the surrounding thumbnails are provided in positions relative

to the viewpoint of the left side camera which is now providing the focus stream. The

conclusion that any of the seven cameras used for the Figure 9 embodiment of White

may provide the focus stream and thus the arrangement of the surrounding thumbnails

is relative to the viewpoint of any of the seven cameras when selected to provide the

focus stream is further supported by the disclosure that the "streaming would begin with

a default camera view as the focus stream." See White, 4:66-67.

The White reference discloses a camera system and display interface that

provides all of the functionality recited in Claim 1 of the '783 patent. The White system

arranges thumbnail streams, which act as user controls, around a central focus stream

in positions which represent the topological arrangement of the cameras providing the

D(: 1 Y O I IAHD/436261.1 27


U,S Patent No. 7,777,783

video streams. Each time a new camera is selected to provide the focus stream, the

display is "reconfigured" so that the user control positions on the display (i.e., the

thumbnail stream positions) are re-assigned to the cameras that have a corresponding

topological relationship with the newly selected focus stream camera. Accordingly,

White raises a substantial new question of patentability and anticipates Claim 1 of the

'783 patent.

3. Roberts Reference

Claim 1 of the '783 patent is anticipated by U.S. Patent No. 7,295,228 to Roberts

(the "Roberts" reference) (Exhibit D). The application for the patent which is the

Roberts reference was filed on October 29, 2002 and the patent issued on November

13, 2007. As a result, the Roberts reference qualifies as prior art against the '783

patent under 35 U.S.C. § 102(e). A detailed comparison of the elements and limitations

of Claim 1 of the '783 patent with the disclosure of the Roberts reference is provided in

Appendix C, which is incorporated herein by reference.

The Roberts reference discloses a method of operating a security camera

system. See Roberts, 2:17-21. The security camera system comprises a plurality of

video cameras and camera selection means which permit a user to observe a target of

interest as it moves from the field of view of camera to camera. See Roberts, 1:29-35.

Each of the cameras in the system feed their respective video signals to a camera

matrix. See Roberts, 2:54-59. With reference to Figure 1, the camera matrix 5 isI.(:'OI/YOAl D/436261 .] 28



controlled by a CPU in which "data" is retained "relating to a fixed field of view afforded

by each of the cameras" 1, 2, 3, 4, and the position of the cursor 8. A target person

may be visually followed by an operator across a current field of view of the camera

feeding image display 9, and when the target approaches an. edge of the field of view, a

device 7 (such as a mouse) may be used to mark the spot with the cursor .8, for

example by clicking on the boundary. Data identifying the marked spot together with

data appertaining to each available camera.image is then processed in the CPU .in

order to select the most suitable camera to feed the monitor 6 with the image display

from the selected camera. See Roberts, 3:2-14. "[U]pon movement of the target to a

transition region, the user control is operable to select the 'button' region in which the

target appears in a displayed image, so as to select a second camera and to employ

video signals from the second video camera to maintain target image continuity."

Roberts, 4:26-31.

The use of the Roberts system is explained as follows:

[A] target of interest can be observed by a first video camera andan image of said target be displayed on the monitor, wherein overlapimage portions at transition regions of a displayed image are provided bytwo adjacent cameras, each transition region being displayed as a 'button'region by the image display; wherein there are a plurality of such 'button'regions being arranged about or-surround[ing] a central area in an imagefield of view; wherein, upon movement of the target to a transition region,the user control is operable to select the 'button' region of an imagedisplayed, whereby to select a second camera and to employ videosignals from the second video camera.

Roberts, 2:24-35.

CO 1/YOl A1:/436261. 1 29



FIOURE I 11'

Roberts, Fig. 1.

Figure 1 of Roberts, above, shows the interface or monitor 6 screen. The "data"

referenced above, relating to the "fixed field of view afforded by each of the cameras,"

constitutes data which indicates the topological relationship of the cameras. See

Roberts, 3:2-14. In order to "select the most suitable camera to feed the monitor" after

a spot is marked by a user by clicking on the boundary 11 of the displayed image where

the target approached the edge of the field of view requires that the CPU have data that

indicates the topological relationship of the cameras in the system. See id.

With continued reference to Figure. 1, clicking on the boundary of the image

display 9 to select a new camera to provide the video for the image display may involve

I)CO 1 /YOIIAD/43626 1.



clicking on boundary "buttons" 11 as explained below.

[A] target of interest can be observed by a first video camera andan image of said target be displayed on the monitor, wherein overlap.image portions at transition regions of a displayed image are provided bytwo adjacent cameras, each transition region being displayed as a 'button'region by the image display; wherein there are a plurality of such 'button'regions being arranged about or surround[ing] a central area in an imagefield of view; wherein, upon movement of the target to a transition region,the user control is operable to select the 'button' region of an imagedisplayed; whereby to. select a second camera and to employ videosignals from the second video camera.

Roberts, 2:24-35.

In an alternative embodiment the image displayed may besurrounded by 'button regions' 11, the 'button region' (or regions) adjacentto a spot whereat a target approaches the'limit of an image being selectedusing the cursor 8, and the device 7, in order to provide information for theCPU to facilitate selection of an appropriate one of the cameras 1, 2, 3, or4 for the image display 9.

Roberts, 3:15-21.

The Roberts system utilizes "knowledge of the transition regions in each camera

where the subject leaves the field of view, and of the camera on which that region can

next be seen" to select the best camera to next provide video.for the image display 9.

See Roberts, 3:42-46. The referenced use of "knowledge of the transition regions .

.where the subject leaves the field of view," and "of the camera on which that region can

next be seen," further confirms that the Roberts CPU utilizes data that indicates the

topological relationship of the cameras in the system.

The composition of the Roberts interface or monitor 6; including the placement

'of the "button regions" 11 (i.e., user controls) or shown in Figure 1 above is identical toDC01/YOI IAD/436261.1 31



that of the Video Tracking Interface 300 shown in Figure 3 of the '783 patent, below.

300

31] 310B

310 C

FIG. 3

'783 patent, Fig.3.

Both the Roberts interface and the '783 patent interface include a central maid

video display and surrounding user controls that may be clicked upon to select the next

camera to provide a video stream for the main video display. Still further, in both

systems, the surrounding user controls are placed in locations which are' representative

of the topological relationship of the cameras associated with the user controls with the

camera providing video for the'main video display. The identification and use of these

topological relationships is apparent from perusal of the Roberts reference because

such relationships must be known in order to select the most suitable camera to nextDCOI/YO A) /436261. . ' 32

Main Video Display305

Main Video Display3O5



feed'the monitor 6 with the. image display from the selected camera and to maintain

target image continuity. See Roberts, 3:2-14 and 4:26-31.

The Roberts system also inherently re-associates each of the boundary buttons

11 with cameras in the system to be representative of the topological relationship of

such cameras with the viewpoint of the camera selected to provide the main video

display. If this were not the case, the Roberts system would not operate as described

"to maintain target image continuity." Roberts, 4:30-31. In light of the foregoing

disclosure of the Roberts reference, a substantial new question of patentability is raised

and Claim 1 is anticipated.

"4. Redstone Products_Services Reference

Claim 1 of the '783 patent is anticipated by the website of Redstone Integrated

Solutions, available at http://web.archive.org/web/20040606014343/redstone-

is.com/products_services.html (the "Redstone Products_Services reference") (Exhibit

E). The Redstone Products Services website page is indicated to have a copyright

notice of publication date of 2003, and an Internet publication date of June 6, 2004. As

a result, the Redstone Products_Services reference qualifies as prior art against the

'783 patent under 35 U.S.C. § 102(b). A detailed comparison of the elements and

limitations of Claim 1 of the '783 patent with the disclosure of the Redstone

Products_Services reference is provided in Appendix D, which is incorporated herein by

reference.DCO/YOJIAD/43(261 .1 33



The Redstone. ProductsServices reference discloses a product called

VisualPursuit which allows a surveillance operator to easily track a moving subject

through a facility from. a single monitor.. See Redstone Products_Services, 1.

"VisualPursuit's display combines the output from several surveillance cameras into a

single, MS Windows-standard screen." Id., 1. The VisualPursuit screen contains a

Subject View and Locator Views. The Subject View displays the current location of the

moving subject. Up to 8 Locator Views display the output of the cameras. closest to the

one trained on the subject. An operator may track a subject by clicking the Locator

View that the person has moved into. The Locator View becomes the new Subject

View, and all Locator Views are automatically reassigned and updated. See id.

A screenshot of the VisualPursuit interface provides views taken from nine

cameras. A large, central window contains the output from the current main tracking

camera; other cameras in proximity to the current tracking camera are displayed in the

other eight windows. By clicking on one of these adjacent views, the operator can

change it to be VisualPursuit's main camera view. The adjacent views then

automatically update based on the, new current tracking camera. See id.,.2.

VisualPursuit is described as using location information for each of the cameras

that are part of the system. Namely, a series of LED's give the operator an idea of

where the current tracking camera is, in relation to another camera. There is also a

map layout that can assist with visualizing exactly where each camera is in the facility

being monitored. See id., 2. VisuaiPursuit is also described as working withCOI/Y01 IA D/4v3o62( 1 .34



Redstone's MapTrack product, which identifies the locations of the cameras in the

system relative to each other.. See id., 1. "MapTrack's display combines digital maps or

diagrams with graphical symbols that can represent the locations of items such as..

.security cameras...". Id., 3.

The foregoing establishes that the topological relationship between the nine

cameras used for VisualPursuit were known.in order for VisualPursuit to perform the

specified functions of "display[ing] the output of the cameras closest to the one trained

on the subject" in Locator Views, and enabling a user to click on "the Locator View that

the person has moved into [so that] [t]he Locator View becomes the new Subject View,

and all Locator Views are automatically reassigned and updated."

There are multiple indications that VisualPursuit utilized the topological

relationships of the cameras when assigning Locator Views to particular cameras. First,

"VisualPursuit works with MapTrack, and MapTrack identifies the locations of the

security cameras in the system. Second, it is necessary for VisualPursuit to identify the

topological relationships of the cameras in the system in order to identify the "cameras

closest to the one trained on the subject" after a new camera is selected to provide the

Subject View. Further, the Locator Views are each a "user control" which may be

selected by clicking on it. Thus, each of the cameras that provide a Locator View are

associated with a Locator View (i.e., user control) as a result of being "closest" to (i.e.,

based on the topological relationship with) the camera providing the Subject View.

Accordingly, the Redstone Products_Services reference raises a substantial newDC01/YO1 IA^ IA)3,261.1 35



question of patentabilityas it discloses all elements of, and anticipates C.laim 1 of the

'783 patent.

B. Proposed 35 U.S.C. § 103 Rejections

The Manual of Patent Examining Procedure ("MPEP") was revised in 2007 to

incorporate the Examination Guidelines for Determining Obviousness Under 35 U.S.C.

103 in View of the Supreme Court Decision in KSR International Co. v. Teleflex -Inc., 72

.FR 57526 (Oct. 10, 2007), Off. Gaz. Pat. Office 23 (Nov. 6, 2007) ("2007 KSR

Guidelines"). See MPEP § 2141 (8 th ed. 2001) (Rev. 6, Sept. 2007). -The 2007 KSR

Guidelines identify the following six (6) nonexclusive rationales for finding a claimed

invention obvious in view of prior art. These rationales are:

(1) combining prior art elements according to known methods to yield

predictable results;

(2) simple substitution of one known element for another to obtain predictable

results;

(3) use of a known .technique to improve similar devices, methods, or

products in the same way;

(4) applying a known technique to a known device, method, or product ready

for improvement to yield predictable results;

(5) obvious to try - choosing from a finite number of identified, predictable

solutions, with a reasonable expectation of success; andDCOI/YO HAI /4 362, .1 36



(6) known work in one field of endeavor may promit variations of it for use in

either the same field or a different one based on design incentives or other

market forces if the variations are predictable to one of ordinary skill in the

art.

The inventions claimed in the '783 patent are obvious based on one or more of

the above-referenced rationales, in addition to being obvious in view of the teaching-

suggestion-motivation test that was employed prior to the KSR decision. Each of the

combinations of prior art described below satisfy at least the first five rationales because

the references relied upon all relate to very similar systems which are used to visually .

track a target from camera view to camera view on a screen which displays at least one

selected camera view. As a result, one of ordinary skill in the art would have found it

obvious to combine the teachings of these references to obtain the advantages of each

in the resulting system. Moreover, the references themselves, in explaining the

advantages of each, provide motivation to one of ordinary skill in the art to combine the

teachings of each:

1. Girgensohn '706 Reference

a. Girgensohn '706 Reference in combination with Lin andGirgensohn '978 References

The teachings of the Girgensohn '706 reference (Exhibit B) would have been

readily combined by one of ordinary skill in the art with those of U.S. Patent. No.

7,242,423 to Lin (the "Lin" reference) (Exhibit F) and U.S. Patent Appl. Pub. No.I)C(OI/YOI HAD4.3626t1.1 37



2006/0284978 Al to Girgensohn et al. (the'"Girgensohn '978" reference) (Exhibit G) to

render Claims 3-8 of the '783 patent obvious over the Girgensohn '706 reference in

view of the Lin reference and in further view of the Girgensohn '978 reference.

The application that resulted in the Lin reference was filed August 8, 2003 and

the patent issued on July 10, 2007. As a result, the Lin reference qualifies as prior art

against the '783 patent under 35 U.S.C. § 102(e). The application that resulted in the

Girgensohn '978 reference was filed January 3, 2006 and published on December 21,

2006. As a result, the 'Girgensohn '978 reference qualifies as prior art against the '783

patent under 35 U.S.C. §§ 102(e) and 102(a). A detailed comparison of the elements

and limitations of Claims 3-8 of the '783 patent with the relevant disclosure of the

Girgensohn '706, Lin and Girgensohn '978 references is provided in Appendix E, which

is incorporated herein by reference. Requester also hereby. incorporates by reference

the discussion of the Girgensohn '706 reference provided above.

The Girgensohn '706, Girgensohn '978 and Lin references all disclose systems

and methods for switching between cameras to monitor a surveillance area. See. Lin,

Abstract; see also Girgensohn '978, Para. 0008. With regard to Claims 5-8, both

Girgensohn '706 and Lin also disclose. using an automatic temporal offset to account for

the travel time of a target between the fields of view of two cameras, which limitation

was required in order to obtain allowance. of Claim 5.

In Girgensohn '706, "[a] three to ten second offset from' the time of the main

video stream display playing performs best for tracking people walking from cameraDCOI OIY()IAI)/43 261. 1 38



view to camera view. In embodiments, this offset can be varied as desired. The refresh

rate of images in these video streams can be varied to produce effects ranging from

almost fluid video playback to images that remain steady for seconds or even minutes."

Girgensohn '706, Para. 0030 (emphasis added). One of ordinary skill in the art would

readily understand that the afore-noted "offset" disclosed in Girgensohn '706 is based

on the travel time of a target between the fields of view of two cameras. However, even

if this were not the case, Lin expressly discloses use of a travel time based temporal

offset.

In Lin, "[i]f a physical path exists for an object to traverse from one zone to

another, these zones are defined to be "linked"." Lin, 4:1-3. The use of a temporal

offset for linked zones is explained further in Lin as follows:

-In a preferred embodiment this invention, each linked zone maycontain a time parameter that determines whether an event in one zone is"recent" relative to the corresponding event in another zone. For example,a disappearance from zone Z(3,1) of FIG. 2A will correspond/to animmediate appearance at the corresponding zone of camera C2, and thedefinition of "recent" for this linkage can be set to a few seconds.Conversely, an appearance at zone Z(I,I) may correspond to adisappearance from a zone-of camera C3 that occurred minutes before,and the definition of "recent" for this linkage would be set to a few minutes.

Lin, 6:5364.

The surveillance system disclosed in Lin expressly uses "a time duration

parameter corresponding to traversing between the first zone and each of the zones in

the set of linked zones." See Lin, 8:60-64. The Lin "time duration parameter" may be

based on a "minimal traversal time, a maximum traversal time, a traversal time1C)01/YOlIAD/436261.1 39



likelihood, and a probability distribution function of traversal time" from one camera's

view to another's. See Lin, 9:3-8. Thus, Lin discloses use of a temporal offset that is

"based on an estimated time of travel of the target" as required by Claim 5.

With regard to Claims 3-4 and 5-8, the Girgensohn '978 reference discloses a

system and method for automatically stitching together video recorded by different

cameras. Girgensohn '978 refers to such stitched together video as "a summary of

video from multiple cameras" which "provides a storyboard of all activity with .emphasis

on events of importance" and which "summary can be used to access the recorded

video." Girgensohn '978, Para. 0012. Using the Girgensohn '978 system, "[a] single

Manga summary for all selected cameras can be created by selecting keyframes from

those cameras." Id., Para. 0038. The identity of the cameras that are the source of the

keyframes is known because "[c]olor-coding :indicates the source of each keyframe." Id.

Girgensohn '978 further discloses that the storyboard of keyframes provides an

index for a user to locate video of interest.

Users such as security personnel need to be able to select videostreams for inclusion in the storyboard and the time frame of thestoryboard. Map and timeline interface components have been designedand developed for these purposes. The map, timeline, and storyboardinteract to provide the user with the information necessary to locate videosegments of interest.

Id., Para. 0041. The Girgensohn '978 video summaries are thumbnail sized images as

shown in Figures 2-4 and 9-11. Id., Figs. 2-4 and 9-11. In view of the foregoing, the.

Girgensohn '706, Lin and Girgensoh '978 references collectively disclose all of the

DC0I/YOil iAD/436261.1 40


U.S Patent No.7,777,783

limitations of Claims 3-8.

It would have been obvious to combine the temporal offset teachings of Lin and

the automatic stitching and thumbnail indexing features of Girgensohn '978 with the

teachings of Girgensohn '706 as a result of employing simple substitution to obtain

predictable results, or as a result of using known techniques to improvesimilar methods

and systems. As noted above, all references concern. systems and methods for

switching between cameras to monitor a surveillance area. The Girgensohn '706 and

Girgensohn '978 references, which have overlapping inventorship, both disclose video

display interfaces which present surveillance video from multiple cameras in thumbnail

format. The two systems are compatible and complementary. The advantages of such

a summary are noted in the Girgensohn '978 and Lin references. See Girgensohn '978,

Para. 0008-0009, 0012-0013 and 0041; see also Lin, 3:59-4:3. Thus, one of ordinary

skill in the art would have been motivated to combine the functionality of the two

systems together to obtain a single system with improved utility. One of ordinary skill in

the art would have been further motivated to utilize the travel time temporal offset

teachings of Lin in the Girgensohn '706 system, as'the latter specifically references use

of an offset in order to account for a target walking from camera view to camera view. A

substantial new question of patentability is raised because Claims 5-8 would have been

obvious over the Girgensohn '706 reference in view of the Lin reference and in further

view of the Girgensohn '978 reference, and Claims 3-4 would have been obvious over

the Girgensohn '706 reference in view of the Girgensohn '978 reference.DCO01/YOI IAD/436261,1 41



b. Girgensohn '706 Reference in combination withGormley Reference

The teachings of the Girgensohn '706 reference (Exhibit B) would have been

readily combined by one of ordinary skill in the art with those of U.S. Patent No.

5,258,837 to Gormley (the "Gormley" reference) (Exhibit H) to render Claim 2 of the

'783 patent obvious. The Gormley reference issued as a patent on November 2, 1993

and qualifies as prior art against the '783 patent under 35 U.S.C. § 102(b). A detailed

comparison of the elements and limitations of Claim 2 of the '783 patent with the

relevant disclosure of the Girgensohn '706 and Gormley references is provided. in

Appendix, F, which is incorporated herein by, reference. Requester also hereby

incorporates by reference the discussion of the Girgensohn '706 reference provided

above.

The Girgensohn '706 reference discloses all elements of Claim 1. Claim 2 adds

the limitations of converting video data from a first format to a second format, and

storing video data generated by a video camera in association with identification

information for the camera. The Girgensohn '706 system also stores, and provides

format conversion of, video data. See Girgensohn '706, Para. 0070. Format

conversion occurs given that the Girgensohn '706 cameras 'utilize JPEG which is a

video compression technique that was well known in the art prior to the critical date.

Accordingly, format conversion occurs for incoming video to be converted to JPEG.

The Girgensohn '706 system also utilizes information to identify the cameras which are

)CO1/YOIID/43626.1 42



associated with each of the video streams that it displays. Specifically, "the main video

stream and the other video streams have colored borders to indicate the actual source

cameras." Girgensohn '706, Para. 0031. This color coding information must

necessarily be stored for video playback. See also, e.g., Girgensohn '706, Para. 0052

and 0067.

Even if the color coding information of Girgensohn '706. were not stored,

however, Gormley teaches storage of information which identifies the cameras used to

generate video streams. In this regard, Gormley states:

It may be desirable for the, images to be identified.

One way of achieving this is for each camera to have associatedwith it an identification unit which inserts an identification signal (e.g. analphanumeric message) into its. picture. This results in the messageforming an indivisible part of the image, so that the message will beenlarged together with the rest of the image if that image is selected for .the large zone.

Another way of achieving this is for the identification signals to begenerated in association with the generation of the composite picture. Thispermits the message to remain the same size regardless of the size of theassociated images. If the composite picture has neutral borders betweenthe zones, then the messages can be displayed in those borders.

Gormley, 4:10-17. In order for the Gormley system to insert identification signals as an

indivisible (i.e., permanent) part of the video images, identification information for each

of the cameras must be stored in association with the video images.

It would have been obvious for one of ordinary skill in the art to provide the video

streams of Girgensohn '706 with the identification signal of Gormley as a result of

EX() I /YOIA1/436261.1



employing simple substitution to obtain predictable results, or as a result of using known

techniques.to improve similar methods and systems. More specifically, it would have

been obvious to combine these teachings in order to improve upon and/or supplement

the method of identifying camera video streams disclosed in Girgensohn '706. The

motivation for such modification is provided in Gormley, which states "[i]t may be

desirable for the images to be identified." Gormley, 4:10. Accordingly, Claim 2 would

have been obvious over the Girgensohn '706 reference in view of the Gormley

reference, and a substantial new question of patentability is raised.

2. Roberts Reference

a. Roberts Reference in combination with Girgensohn '706and Gormley References

The teachings of the Roberts reference (Exhibit D) would have been readily

combined by one of ordinary skill in the art with those of the Girgensohn '706 (Exhibit B)

and Gormley (Exhibit H) references to render ,Claim 2 of the, '783 patent obvious. A

detailed comparison of the elements and limitations of Claim 2 of the '783 patent with.

the relevant disclosure of the Roberts, Girgensohn '706 and Gormley references is

provided in Appendix G, which is incorporated -herein by reference. Requester also

hereby incorporates by reference the discussion of the Girgensohn '706. and Gormley

references provided above.

The Roberts reference discloses all elements of Claim 1. Claim 2 adds the

limitations of converting video data from a first format to a second format, and storingIC) I /YOHIA D/436261.1 44



video data generated by a video camera in association With identification information for

the camera. The Roberts reference discloses that camera identification information is

stored in connection with the Roberts system. "Conveniently the user control includes a

CPU, which is operable to store data including, in respect to each camera, the pixel

positions of the lines at which transitions occur, and the identity of the camera to be

selected at that transition." Roberts, 1:60-63. Roberts does not expressly disclose

format conversion, however.



Girgensohn '706 cameras utilize JPEG which is a video compression technique that

was-well known in the art prior to the critical date. Accordingly, format conversion occurs

for incoming video to be converted to JPEG. It would have been obvious for one of

ordinary skill in the art to utilize JPEG with the Roberts system. As taught by

Girgensohn '706, JPEG is a format used for streaming video, such as the video

disclosed in Roberts. It would have been obvious to one of ordinary skill in the art to

provide the video streams of Roberts with the format conversion of Girgensohn '706 to

improve upon the flexibility of the Roberts system to work with different formats.

The Girgensohn '706 system also utilizes information to identify the cameras

which are associated with each of the video streams that it displays. Specifically, "the

main video stream and the other video streams have colored borders to indicate the

actual source cameras." Girgensohn '706, Para. 0031. This color coding informationDCO1/YOHAD/43626 1.1 45


U.S Patent No. 7,77.7,783

must necessarily be stored for video playback. See also, e.g., Girgensohn '706, Para.

0052 and 0067.

Even if the color coding information of Girgensohn '706 were not stored,



It may be desirable for the images to be identified.

One way of achieving this is for each camera to have associatedwith it an identification unit which inserts an identification signal (e.g, analphanumeric message) into its picture. This results in the messageforming an indivisible part of the image, so that the message will beenlarged together with the rest of the image if that image is selected forthe large zone.

Another way of achieving this is for the identification signals to begenerated in association with the generation of the composite picture. Thispermits the message to remain the same size regardless of the size of theassociated images. If the composite picture has neutral borders between'the zones, then the messages can be displayed in those borders.

Gormley, 4:10-17. In order for the Gormley system to insert identification signals

as an indivisible (i.e., permanent) part of the video images, identification

information for each of the cameras must be stored in. association with the video

images.

To the extent Roberts does not disclose storing camera identification information

in association with video data from the identified camera, it would have been obvious for

one of ordinary skill in the art to utilize the video camera identification of Girgensohn

'706 and/or the identification signal of Gormley with the. video streams of the Roberts

I)C I /YO HA D/436261. I 46.

Attorney Dkt. No. 017874-0003Reexamination. Request


system as a result of employing simple substitution to obtain predictable results, or as a

result of using known techniques to improve similar methods and systems. More

specifically, it would have been obvious to combine these teachings in order to

incorporate the method of identifying camera video streams disclosed in Girgensohn

'706 and Gormley into the Roberts system in view of the Gormley reference's

suggestion that "[i]t may be desirable for the images to be identified." Gormley, 4:10. A

substantial new question of patentability is raised because Claim 2 would have been

obvious over the Roberts reference in view of the Girgensohn '706 and Gormley

references.

b. Roberts Reference in combination with Girgensohn '978and Lin References

The teachings of the Roberts reference (Exhibit D) would have been readily-

combined by one.of ordinary skill in the art with those of the Girgensohn '978 (Exhibit B)

and Lin (Exhibit F) references to render Claims 3-8 of the '783 patent obvious. A

detailed comparison of the elements and limitations of Claims 3-8 of the '783 patent with

the relevant disclosure of the Roberts, Girgensohn '978 and Lin references is provided

in Appendix H, which is incorporated herein by reference. Requester also hereby

incorporates by. reference the discussions of the Roberts, Girgensohn '978 and Lin


The Roberts, Girgensohn '978 and Lin references all disclose systems and

methods for switching between cameras to monitor a surveillance area. See Lin,DC01/YO IAD/436261.1 47



Abstract; see also Girgensohn '978, Para. 0008. With regard to Claims 3 and 4, the

Roberts reference discloses all elements of Claim 1, from which Claims 3 and 4

depend. The Girgensohn '978 reference discloses the elements of Claims 3 and 4, as

discussed below.

With regard to Claims 5-8, Lin discloses using an automatic temporal offset to

account for the travel time of a target betweenri the fields of view of two cameras, which

limitation was required in order to obtain allowance of Claim 5. Lin expressly discloses

use of a travel time based temporal offset. In Lin, "[i]f a physical path exists for an

object to traverse from one zone to another, these zones are defined to be "linked"."

Lin, 4:1-3. The use of a temporal offset for linked zones is explained further in Lin as

follows:

In a preferred embodiment this invention, each linked zone may,contain a time parameter that determines whether an event in one zone is."recent" relative to the corresponding event in another zone. For example,a disappearance from zone Z(3,1) of FIG. 2A will correspond to animmediate appearance at the corresponding zone of camera C2, and thedefinition of "recent" for this )linkage can be 'set. to a few seconds.Conversely, an appearance at zone Z(l,I) may correspond to adisappearance from a zone of camera C3 that occurred minutes before,and the definition of "recent" for this linkage would be set-to a few minutes.

'Lin, 6:53-64.

The surveillance system disclosed 'in Lin uses "a time duration 'parameter

corresponding to traversing.between the first zone and each of the zones in the set of

linked zones." See Lin, 8:60-64. The Lin "time duration parameter" may be based on a

"minimal traversal time, a maximum traversal time, a traversal time likelihood, and aD'01/YO AI.A/436261.1 48



probability distribution function of traversal time" from one camera's view to another's.

See Lin, 9:3-8. Thus, Lin discloses use of a temporal offset that is "based on an

estimated time of travel of the target" as required by Claim 5.


system and method for automatically stitching together video .recorded by different

cameras. Girgensohn '978 refers to such stitched together video as "a summary of

video from multiple cameras" which "provides a storyboard of all activity with emphasis


video." Girgensohn '978, Para. 0012. Using the Girgensohn '978 system, "[a] single

Manga summary for all selected cameras can be created by selecting keyframes from

those cameras." Id., Para. 0038. The identity of the cameras that are the source of the

keyframes is known because "[c]olor-coding indicates the source of each keyframe." Id.

Girgensohn '978 further discloses that the storyboard of keyframes provides an

index for a user to locate video of interest.

Users such as security personnel need, to be able to select videostreams for inclusion in the storyboard, and the time frame of thestoryboard. Map and timeline interface components have been designedand developed for these purposes. The map, timeline, and ,storyboardinteract to provide the user with the information necessary to locate videosegments of interest.

Id., Para. 0041. The Girgensohn '978 video summaries. are thumbnail sized images as

shown in Figures 2-4 and 9-11. Id., Figs. 2-4 and 9-11. In view of the foregoing, the

Roberts, Lin and Girgensohn '978 references collectively disclose all of the limitations of

DC YO o AD/4362,1 49



Claims 3-8.

It would have been obvious to combine the temporal offset teachings of Lin and

the automatic stitching and thumbnail indexing features of Girgensohn '978 with the

teachings of Roberts as a result of employing simple substitution to obtain predictable

results, or as a result of using known techniques to improve similar methods and

systems. As noted above, all references concern systems and methods for switching

between cameras to monitor a surveillance area. Modifying the Roberts system to

stitch together video streams generated by the cameras using the methods of

Girgensohn '978 and the temporal offset of Lin would have provided an efficient

summary of video taken from multiple cameras. The advantages of such a summary,

and the motivation for creating such summary, are noted in the Girgensohn '978 and Lin

references. See Girgensohn '978, Para. 0008-0009, 0012-0013 and 0041; see also Lin,

3:59-4:3.

One of ordinary skill in the art would have been further motivated to incorporate

the automatic stitching capability of the Girgensohn '978 system into the Roberts

system because Girgensohn '978 discloses that security personnel "need to be able to

select video streams for inclusion in" a summary of video taken from different cameras.

Thumbnails are taught in Girgensohn '978 to be used to index such stitched video.

Thus, one of ordinary skill in the art would have been motivated to use the thumbnail

indexing and stitching.capabilities of Girgensohn '978 in the Roberts system.

A substantial new question of patentability is raised because Claims 5-8 wouldIDC01/YOHAD/)1 326 1. 50



have been obvious over the Roberts reference in view of the Lin reference and in further

view of the Girgensohn '978 reference, and Claims,3- 4 would have been obvious over

the Roberts reference in view of the Girgensohn '978 reference.

3. Redstone Reference

a. Redstone Reference in combination with BalancingTomorrow's Technology, Redstone Integrated SolutionsReference

Claim 1 of the '783 patent is believed to be anticipated by the Redstone

Products Services reference (Exhibit E). However, to the extent the Redstone

Products Services reference does not anticipate Claim 1, the teachings of the

Redstone Products Services reference would have been readily combined by one of

ordinary skill in the art with those of the Balancing Tomorrow's Technology, Redstone

Integrated Solutions reference (Exhibit I) to render Claim 1 obvious in view of the two

references. The Balancing Tomorrow's Technology, Redstone Integrated Solutions/

reference bears a 2003 copyright. notice and is internally dated August 26, 2003. As a

result, the Balancing Tomorrow's Technology, Redstone Integrated Solutions reference

qualifies as prior art against the '783 patent under 35 U.S.C. § 102(b). The Redstone

Products_Services reference and the Balancing Tomorrow's Technology, Redstone

Integrated Solutions reference are collectively referred to as the "Redstone references."

A detailed comparison of the elements and limitations of Claim 1 of the '783 patent with

the relevant disclosure of the Redstone references is provided in Appendix I, which isJ

DC'01 /YOH AI)/43626, 1. I 51 '



incorporated herein by reference. Requester also hereby incorporates by reference the

'discussion of the Redstone Products_Services reference provided above.

The Redstone Products_Services reference discloses a product called

VisualPursuit which allows a. surveillance operator to easily track a moving subject

through a facility from a single monitor. See Redstone Products_Services, 1.

"VisualPursuit's display combines the output from several surveillance cameras into a

single, MS Windows-standard screen." Id:, 1. The VisualPursuit screen contains a

Subject View and Locator Views. The Subject View displays the current location .of the

moving subject. Up to 8 Locator Views display the output of the cameras closest.to the

one trained on the subject. An operatr .may track a subject by clicking the Locator

View that the person has moved into. The Locator View becomes the new Subject

View, and all Locator Views are automatically reassigned and updated. See id.

The Balancing Tomorrow's Technology, Redstone Integrated Solutions reference

supplements the teaching of the Redstone ProductsServices reference by showing an

example of the actual VisualPursuit screen. The VisualPursuit screen includes a large

central Subject View surrounded by Locator Views. It would have been obvious to one

of ordinary skill in the art to combine the teachings of the two Redstone references, as

both describe features of the same product called VisualPursuit' Accordingly, Claim 1

would have been obvious over the Redstone Products_Services reference in view of the

Balancing Tomorrow's Technology, Redstone Integrated Solutions reference, and. a

substantial new question of patentability is raised.I)COI/YOHlA1/436261.1 52



b. Redstone References in combination with Girgensohn'706 and Gormley References

The teachings of the Redstone references (Exhibits E and. ) would have been

readily combined by one of ordinary skill in the art with those of the, Girgensohn '706

(Exhibit B) and Gormley (Exhibit H) references to render Claim 2 of the '783 patent

obvious. A detailed comparison of the elements and limitations of Claim 2 of the '783

patent with the relevant disclosure of the -Redstone, Girgensohn '706 and Gormley

references is provided in Appendix J, which is incorporated herein. by reference.

Requester also hereby incorporates -by reference the discussion of the Redstone,

Girgensohn '706 and Gormley references provided above.

The Redstone references disclose all elements of Claim 1. Claim 2 adds the

limitations of converting video data from a first format to a second format, and storing

video data generated by a video camera in association with identification information for

the camera.



Girgensohn '706 cameras utilize JPEG which is .a video compression technique that

was well known in the art prior to the critical date. Accordingly, format conversion occurs

for incoming video to be converted to JPEG. It would have been obvious for one of

ordinary skill in the art to utilize JPEG with the Redstone VisualPursuit system. As'

taught by Girgensohn '706, JPEG is a format used for streaming video, such as theICOI/YOIIAI)/436261.I 53


U.S Patent No. 7,777,783,

video disclosed in the Redstone references. It would have been obvious to one of

ordinary skill in the art to provide the video streams of VisualPursuit with the format

conversion of Girgensohn '706 to improve upon the flexibility of VisualPursuit to work

with different formats.




actual source cameras." Girgensohn '706, Para. 0031. This color coding information

must necessarily be stored for video playback. See also, e.g., Girgensohn '706,- Para.

0052 and 0067.

Even if the color coding information of Girgensohn '706 were not stored,



It may be desirable for the images to be identified.

One way of achieving this is for each camera to have associatedwith it an identification unit which inserts an. identification signal (e.g. analphanumeric message) into its picture. This results in the messageforming an indivisible part of the image, so that the message will beenlarged together with the rest of.the image if that image is selected forthe large zone.

Another way of achieving this is for the identification signals to begenerated in association with the generation of the composite picture. Thispermits the message to remain the same size regardless of the size of theassociated images. If the composite picture has neutral borders betweenthe zones, then the messages can be displayed in those borders.

I)(l/YOI IAD/43626 . 1 54



Gormley, 4:10-17. In order for the Gormley system to insert identification signals

as an indivisible (i.e., permanent) part of the video images, identification

information for each of the cameras must be stored in association with the video

images.

It would have been obvious for one of ordinary skill in the art to utilize the video

camera identification of Girgensohn '706 and/or the identification signal of Gormley, with

the video streams of the VisualPursuit system as a result of employing simple

substitution to obtain predictable results, or as a result of using known techniques to

improve similar methods and systems. More specifically, it would have been obvious to

combine these teachings in order to incorporate the method of identifying camera video

streams disclosed in Girgensohn '706 and Gormley into the VisualPursuit system based

on the suggestion in Gormley that "[i]t may be desirable for the images to be. identified."

Gormley, 4:10. Accordingly, Claim 2 would have been obvious over the Redstone

references in view of the Girgensohn '706 and Gormley references, and a substantial

new question of patentability is raised.

c. Redstone References in combination with Girgensohn'978 and Lin References

The teachings of the Redstone references (Exhibits E and I) would have been

readily combined by one of ordinary skill in the art with those of the Girgensohn '978

(Exhibit B) and Lin (Exhibit F) references to render Claims 3-8 of the .'783 patent

obvious. A detailed comparison of the elements and limitations of Claims 3-8 of theCO.:0/YOi A D436261 .1 55



'783 patent with the relevant disclosure of the Redstone, Girgensohn '978 and Lin

references is provided in Appendix K, which is incorporated herein by reference.

Requester also hereby incorporates by reference the discussion of the Redstone,

Girgensohn '978 and Lin references provided above.

The Redstone, Girgensohn '978 and Lin references all disclose. systems and

methods for switching between cameras to monitor a surveillance area. See Lin,

Abstract; see also Girgensohn '978, Para. 0008. The Redstone references disclose a

system which is used to track a target as it moves between the fields of view of multiple

video cameras. Girgensohn '978 discloses a video surveillance system that stitches

together streams of surveillance video captured by different cameras in the system, and

uses thumbnails to index the stitched together. video. It would' have been obvious to

modify the Redstone system to stitch together video streams generated by the cameras

in the Redstone system and create an index using thumbnails, using the methods of

Girgensohn '978 in order to provide a readily accessible summary of video taken from

multiple cameras. The advantages of.(i.e. mptivation for using) such a summary and

indexing are noted in the Girgensohn '978 reference. See Girgensohn '978, Para.

0008-0010, 0012-0013 and 0041.

Lin discloses use of a temporal offset that is based on the travel time between

camera fields of view when switching from viewing the video from one camera to that of

an adjacent camera. Specifically, Lin discloses a system that uses "a time duration

parameter corresponding to traversing between the first zone and each of the zones in)C:()/YOHA D/436261.1 56




based on a "minimal traversal time, a maximum traversal time, a traversal time



"based on an estimated time of travel of the target" as required by Claim 5. It would

have been obvious to modify the Redstone VisualPursuit system to stitch together video

streams generated by the cameras in the Redstone system, using the methods of

Girgensohn '978 and the temporal offset of Lin in order to provide an efficient summary

of video taken from multiple cameras. The advantages of such a summary are noted in

the Girgensohn '978 and Lin references. See Girgensohn '978, Para. 0008-0009, 0012-

0013 and 0041; see also Lin, 3:59-4:3.

' With regard to Claims 3-4 and 5-6, the Redstone references disclose all

elements of Claim 1, from which Claims 3 and 4 depend, as discussed above and in

Section V.B.2.b., and the Girgensohn '978 reference discloses a system and method for

automatically stitching together video recorded by different cameras. Girgensohn '978

refers to such stitched together video as "a summary of video from multiple cameras"

which "provides a storyboard of all activity with emphasis on events of importance" and

which "summary can be used to access the recorded video." Girgensohn '978, Para.

0012. The Girgensohn '978 video summaries are thumbnail sized images as shown in

Figures 2-4 and 9-11. Id., Figs. 2-4 and 9-11.

With regard to Claims 5-8, the Redstone references disclose all the elements ofDC('1/YOlAD/436261.1 57



Claims 5, 7 and 8, except for stitching video and using a temporal offset in doing so. Lin

discloses using an automatic temporal offset to account for the travel time of a target

between the fields of view of two cameras, as discussed above and in Section V,B.2.b.

In view of the foregoing, a substantial new question of patentability is raised as

Claims 5-8 would have been obvious over the Redstone references in view of the Lin,

reference and in further view of the Girgensohn '978 reference, and Claims 3-4 would

have been obvious over the Redstone references in view of the Girgensohn '978

reference.

4. White Reference

a. White Reference in combination with Girgensohn '706and' Gormley References

The teachings of the White reference (Exhibit C) would have been readily


and Gormley (Exhibit. H) references to render Claim 2 of the '783 patent obvious. A

detailed comparison of the elements and limitations of Claim 2 of the '783 patent with

the relevant disclosure of the White, Girgensohn '706 and Gormley references, is

provided in Appendix L, which is incorporated herein by reference. Requester also

hereby incorporates by reference the discussion of the White, Girgensohn '706 and

Gormley references provided above.

The White reference discloses a.,camera system and display interface that

DCO) /YOIIAD/43(261.1


U.S Patent No. 7.,777,783

provides all of the functionality recited in Claim 1 of the '783 patent. The White system

arranges thumbnail streams, which act as user controls, around a central focus stream

in positions which represent the topological arrangement of the cameras providing the

video streams. Each time a new camera is selected to provide the focus stream, the

display is "reconfigured" so that the user control positions on the display (i.e., the

thumbnail stream positions) are; re-assigned to the cameras that have a corresponding

topological relationship with the newly selected focus stream camera.,

Claim 2 adds the limitations of converting video data from a first format to a

second format, and storing video data generated by a video camera in association with

identification information for the camera. As explained above, the Girgensohn '706

system provides format conversion of video data.. See Girgensohn '706, Para. 0070.

Format conversion occurs given that the Girgensohn '706 cameras utilize JPEG which

is a video compression technique that was well known in the art prior to the critical date.

Accordingly, format conversion occurs for incoming video to be converted to JPEG. It

would have been obvious for one of ordinary ,skill in the art to utilize JPEG with the

White system to improve upon the flexibility of the White system to work with different

formats.




actual source cameras." Girgensohn '706, Para. 0031. This color coding informationDCOY011/YOAD/436261. 59



must necessarily be stored for video playback. See also, e.g., Girgensohn '706, Para.

0052 and 0067. Even if the color coding information of Girgensohn '706 were not

stored, however, Gormley teaches storage of information which identifies the cameras

used to generate video streams. See.Gormley, 4:10-17.

It would have been obvious for one of ordinary skill in the art to utilize the video

camera identification of Girgensohn '706 and/or the identification signal of Gormley, with

the White system as a result of employing simple substitution to obtain predictable

results, or as a result of using known techniques to improve similar methods and

systems. More specifically, it would have been obvious to combine these teachings, in

order to incorporate the method of identifying camera video streams disclosed in

Girgensohn '706 and Gormley into the White system based on the suggestion in

Gormley that "[i]t may be desirable for the images to be identified." Gormley, 4:10.

Accordingly, Claim 2 would have been obvious over the White reference in view of the

Girgensohn '706 and Gormley references, and a substantial new question of

patentability is raised.

b. White Reference in combination with Girgensohn '978and Lin References

The teachings of the White reference (Exhibit C) would have been readily


and Lin (Exhibit F) references, to render Claims 3-8 of the '783 patent obvious. A

DCO /YOIAI)/436261. I 60

Attorney Dkt. No.. 017874-0003Reexamination Request


detailed comparison of the elements and limitations of Claims 3-8 of the '783 patent with

the relevant disclosure of the White, Girgensohn '978 and Lin references is provided in

Appendix M, which is incorporated herein by reference. Requester also hereby

incorporates by reference the discussion of the White, Girgensohn '978 and Lin


The White, Girgensohn '978 and Lin references all disclose systems 'and

methods for switching between, cameras to monitor a surveillance area. See Lin,

Abstract; see also Girgensohn '978, Para. 0008. With regard to Claims 3-4, the White

reference discloses all elements of Claim 1, from which Claims 3 and 4 depend. The

Girgensohn '978 reference discloses the elements of Claims 3-4, as discussed above in

Section 2.b.

With regard to Claims 5-8, the White reference discloses all the elements of

Claims 7 and 8, and Lin discloses using an automatic temporal offset to account for the

travel time of a target between the fields of view of two cameras, as discussed in

Section V.B.2.b. Specifically, Lin discloses a system that uses "a time duration

parameter corresponding to traversing between the first zone and each of the zones, in

the set of linked zones." See Lin, 8:60-64.. The Lin "time duration parameter" may be


likelihood, and a probability 'distribution function of traversal time" from one camera's


"based on an estimated time of travel of the target" as required by Claim 5.I)C0 /YOHAD/43626i.1 61




system and method for automatically stitching together video recorded by different

.cameras. Girgensohn '978 refers to such stitched together video 'as "a summary of

video from multiple cameras" which "provides a storyboard of all activity with emphasis


video." Girgensohn '978, Para. 0012. The Girgensohn '978 video)summaries are

thumbnail sized images as shown in Figures 2-4 and 9-11. Id., Figs. 2-4 and 9-11. In

view of the foregoing, the White, Lin and, Girgensohn '978 references collectively

disclose all of the limitations of Claims 3-8.

The White reference discloses a system which is used to track a target as it

moves between the fields of view of. multiple video cameras. Girgensohn '978 discloses

a video surveillance system that stitches together streams of surveillance video

captured by different cameras in the system, and uses thumbnails to index the stitched

together video. It would have been obvious to modify the White system to stitch

together video streams generated by the cameras in the White system and create an

index using thumbnails, using the methods of Girgensohn '978 in order to provide a

readily accessible summary of video taken from multiple cameras. The advantages of

(i.e. motivation for using) such a summary and indexing are noted in the Girgensohn

'978 reference. See Girgensohn '978, Para. 0008-0010, 0012-0013 and 0041.

Lin discloses use of a temporal offset that is based on the travel time between

camera fields of view when switching from viewing the video from one camera to that ofDC 01/YO IA /436261.1 62



an adjacent camera. It would have been obvious to modify the White system to stitch

together video streams generated by the cameras in it using the methods of Girgensohn

'978 and the temporal offset of Lin in order to provide an efficient summary of video

taken from multiple cameras. The advantages of such a summary are noted in the

Girgensohn '978 and Lin references. See Girgensohn '978, Para. 0008-0009, 0012-

0013 and 0041; see also Lin, 3:59-4:3. In view of the foregoing, a substantial new

question of patentability is raised because Claims 3-8 would have been obvious over

the White reference in view of the Lin reference and in further view of the Girgensohn

'978 reference.

5. Maruya Reference

a. Maruya Reference in combination with Girgensohn '706and Gormley References

Claims 1-8 of the '783 patent are rendered obvious .over 'U.S. Patent No.

7,746,380 to Maruya et al. (the "Maruya reference") (Exhibit J) in view of the

Girgensohn '706 reference (Exhibit B) with respect to Claims 1 and 3-8, and in further

view of the Gormley reference (Exhibit H) with respect to Claim 2. The application for

the Maruya patent was filed on June 15, 2004 and the patent issued on June 29, 2010.

The Maruya reference qualifies as prior art against the '783 patent under 35 U.S.C. §

102(e). A detailed comparison of the elements.and limitations of Claim 2 of the '783

patent with the relevant disclosure of the Maruya, Girgensohn '706 and GormleyCO1/Y.OHADI/4362 1.I 63



references is provided in Appendix N, which is incorporated herein by reference.

Requester also hereby incorporates by reference the discussion of the Girgensohn '706

and Gormley references provided above.

The Maruya reference is directed to a video surveillance system which tracks a

tracking target using a plurality of surveillance cameras. See Maruya, Abstract. The

Maruya system stores map information on a surveillance zone, information on camera

locations, and imaging range information for the cameras so that the topological

relationship of the cameras is known. See Maruya, 1:59-2:2. The system "detects the

moving direction of a tracking target based on a picture including the tracking target

from one camera, selects at least one other camera which should shoot the tracking

target next, and generates a display picture including a picture from the one camera and

a picture from the at least one other camera, so as to monitor a moving object as the

tracking target." Maruya, 1:59-2:2.

With respect to the user interface, the Maruya system "arranges the pictures from

the one camera and at least one. other camera at locations which can be approximated

to the camera locations on the map to generate a display picture.. In other words, the

invention arranges a next camera picture of the tracking target in the direction the

tracking target moves within the sight of the surveyor on the picture screen." .Maruya,

2:3-10. Thus, the pictures from the cameras are arranged, in a manner that is

representative of the topological relationship of the cameras providing the pictures. As

shown below, in the Maruya system, video pictures from neighboring cameras A and BC'0OI/YOIiAD/4, 6 .I 64



are provided "around" the.video picture from the tracking camera, and are accordingly,

placed in positions relative to the viewpoint of the tracking camera and representative of

the actual camera locations.

FIG. 24B

Maruya, Fig. 24B.

The Maruya reference

cameras are re-arranged -on

also discloses that the video pictures from each of the

screen each time a new camera is selected to be the

tracking camera.

In case the tracking target has moved in accordance with themoving example shown in FIG. 17A, the screens shown in FIG. 17B areoutput on the surveillance terminal screen in the locations (a), (b) and (c)of FIG. 17A. In the location (a), a picture from the camera cl is displayedon the left as a picture from the tracking camera and a picture, from thecamera c2 in the moving direction of the tracking target is displayed on theright. When the person moves to the location (b), the tracking camera isswitched to c2 and a picture from the camela c3 in the moving direction isdisplayed as a neighborinq camera on the riqht. When the person moves

r

65DC0:OI /YO1I I D/4362-01.



to the location (c), a, picture from the camera c3 as a tracking camera isdisplayed as [the tracking camera] and pictures from the cameras c4 andc5 on the branch passages in the moving direction are displayed on theright as pictures from the neighboring cameras."

Maruya, 9:60-10:7 (emphasis added).

In view of the foregoing, when a person moves from the field of view of a first

camera to the field of view of a second camera using the Maruya system, the picture

from the second camera becomes the central tracking camera view, and the picture

from a third camera is moved so that it is located on the a screen in a position

representative of the topological relationship of the second and. third cameras from the

viewpoint of the second camera.

With respect to Claim 1, the Maruya reference does not disclose that a user may

select a new. tracking camera by clicking on a "user control," such as one of the camera

pictures displayed on the screen. In Maruya, a new tracking camera is selected

automatically based on the movement of the tracking target. Girgensohn '706 teaches '

that the smaller surrounding video streams in Figures 3 and 7 of Girgensohn '706 can

be clicked upon so that they act as, user controls. It would have been obvious to modify

the Maruya system with the teaching of Girgensohn '706 to permit a user to click on the

camera pictures, such as those shown in Figures 24B, 26B and 27B of Maruya, to

enable a user to manually select such camera pictures. to be the main central camera

picture in order to give the user flexibility in selecting the next camera view to be the

main camera view. Thus, Claim 1 is obvious over Maruya in view of Girgensohn '706,

)COI/YOHAI)/436261 .1 66



and a substantial new question of patentability is raised.

Maruya discloses the additional limitations found in dependent Claim 2, including

format conversion and storing video data with camera identification information. See

Maruya, 4:49-57 and 6:31-42. However, to the extent Maruya does not disclose the

additional limitations found in Claim 2, these limitations are disclosed in Girgensohn

'706 and Gormley. For the reasons previously stated, it would have been obvious to

combine the teachings of Maruya, Girgensohn '706 and Gormley to render Claim 2

obvious.

With regard to Claims 3-8, to the extent Maruya does not disclose all elements of

the claims, such as stitching together video streams from different cameras; and use of

a temporal offset when performing such stitching, these limitations are disclosed in

Girgensohn '706 as noted in Appendix R. For the reasons stated above, and in the

attached Appendix R, it would have been obvious to combine the teachings of Maruya

and Girgensohn '706 to render Claims 1-8 obvious. Accordingly, a substantial new

question of patentability is raised by the combination of Maruya with Girgensohn '706

for all claims.

b. Maruya Reference in combination withWhite,Girgensohn '978 and Lin References

The teachings of the Maruya reference (Exhibit J) would have been readily

combined by one of ordinary skill in the art with those of the White (Exhibit C)

Girgensohn '978 (Exhibit B) and Lin (Exhibit F) references to render Claims 1 and 3-8 of1)CO1/YOIIAD/43 (261. 67



the '783 patent obvious. A detailed comparison of the elements and limitations of

Claims 1 and 3-8 of the '783 patent with the relevant disclosure of the Maruya, White,

Girgensohn '978 and Lin references is provided in Appendix O, which is incorporated

herein by reference. Requester also hereby incorporates by reference the discussion of

the Maruya, White, Girgensohn '978 and Lin references provided above.

The Maruya, White, Girgensohn '978 and Lin references all disclose systems and

methods for switching between cameras to monitor a surveillance area. See Lin,

Abstract; see also Girgensohn '978, Para. 0008. As noted above, the Maruya

reference discloses all elements of Claims 1 and 3-8, with the exception of using video

pictures as user controls, and stitching videos together, with or without a temporal

offset.

White teaches that thumbnail views can be clicked upon so that they act as user

controls. It would have been obvious to modify the Maruya system with the teaching of

White to permit a user to click on the camera pictures, such as those shown in Figures

24B, 26B and 27B of Maruya, to enable a user to manually select such camera pictures

to be the main central camera picture in order to give the user flexibility in selecting the

next camera view to be the main camera view.

Girgensohn '978 discloses a video surveillance system that stitches together

streams of surveillance video captured by different cameras in the system. It would

have been obvious to modify the Maruya system to stitch together video streams

generated by the cameras in the Maruya system, using the methods of Girgensohn '978DCo /YOHA D/)436261.1 68



in order to provide a summary of video taken from multiple cameras. The advantages

of such a summary are noted in the Girgensohn '978 reference. See Girgensohn '978,

Para. 0008-0009, 0012-0013 and 0041.

The Lin reference discloses using an automatic temporal offset to account for the

travel time of a target between the fields of view of two cameras, as discussed in

Section V.B.2.b. Specifically, Lin discloses a system that uses "a time duration

parameter corresponding to traversing between the first zone and each of the zones in




view to another's. See Lin, 9:3-8.' Thus, Lin discloses use of a temporal offset that is

"based on an estimated time of travel of the target" as required by Claim 5'.

In view of the foregoing, the Maruya, White, Lin and Girgensohn '978 references

collectively disclose all of the limitations of Claims 1 and 3-8. As explained above and.

in Appendix O, it would have been obvious to combine the teachings of these

references to render Claims 1 and 3-8 obvious.

c. Maruya Reference in combination with Roberts,Girgensohn '978 and Lin References


combined by one of ordinary skill in the art with those of the Roberts (Exhibit D)

Girgensohn '978 (Exhibit B) and Lin (Exhibit F) references to render Claims 1 and 3-8 ofDC01/YOHIA D/436261.1 69




Claims 1 and 3-8 of the '783 patent with the relevant disclosure of the Maruya, Roberts,

Girgensohn '978 and Lin references is provided in Appendix P, which is incorporated

herein by reference., Requester also hereby incorporates by reference the discussion of

the Maruya, Roberts, Girgensohn '978 and Lin references provided above.

The Maruya, Roberts, Girgensohn '978 and Lin references all disclose systems

and methods for switching between cameras to monitor a surveillance area. See Lin,

Abstract; see also Girgensohn '978, Para. 0008. As noted above, the Maruya reference

discloses all elements of Claims 1 and 3-8, with the exception of using video pictures as

user controls, and stitching videos together, with or without a temporal offset. The.

rationale for combining the teachings of the Maruya, Girgensohn '978 and Lin

references is also explained above and in the referenced appendices.

Maruya discloses automatic selection of the next camera view based on target

movement. Roberts teaches that user controls in the form of "button regions," can be

clicked upon and surround a display image provided by a selected camera. The "button

regions" are located along the edge of the central display image and are presented in

display locations representative of topological relationship of the cameras in the system,

relative to the viewpoint of the camera providing the: central display image. It would

have been obvious to modify the Maruya system with the teaching of Roberts to permit

a user to click on the camera pictures, such as those shown in Figures 24B, 26B and

27B of Maruya, to enable a user to manually select such camera pictures to be the mainI)c' I/YOI II 1/4 3 62G1. 70

Attorney .Dkt, No. 017874-0003Reexamination, Request


central camera picture in order to give the user flexibility in selecting the next camera

view to be the main camera view. Maruya teaches that the camera pictures which

provide the next main camera view are located closest to the edge of the main camera

view at which a target leaves the field of view of the camera that previously provided the

main camera view.

Thus, the Maruya, Roberts, Lin and Girgensohn '978 references collectively

disclose all of the limitations of Claims 1 and 3-8. As explained above and in.Appendix

P, it would have been obvious to combine the teachings ofrthese references to render

Claims 1 and 3-8 obvious. Accordingly, these references collectively raise a substantial

new question of patentability.

d. Maruya Reference in combination with Redstone,Girgensohn '978.and Lin References


combined by one of ordinary skill in the art with those of the Redstone (Exhibits E and I)

Girgensohn '978 (Exhibit B) and Lin (Exhibit F) references to render Claims 1 and 3-8 of


Claims 1 and 3-8. of the '783 patent with the relevant disclosure of the Maruya,

Redstone, Girgensohn '978 and Lin references is provided in Appendix Q, which is

incorporated herein by reference. Requester also hereby incorporates by reference the

discussion of the Maruya, Redstone, Girgensohn '978 and Lin references provided

above.I)COI/YOHAD/436261.1 71

Attorney Dkt, No. 017874-0003Reexamination Request


The Maruya, Redstone, Girgensohn '978 and Lin references all disclose systems


Abstract;' see also Girgensohn '978, Para. 00081 As noted above, the Maruya

reference discloses all elements of Claims 1 and 3-8, with the exception ,of using video

pictures as user controls, and stitching videos together, with or without a temporal

offset. The rationale for combining the teachings of the Maruya, Girgensohn '978 and

Lin references is also explained above and in the referenced appendices.

Maruya discloses automatic selection of the next camera view based on target

movement. The Redstone references teach that the thumbnails can be clicked upon so

that they 'act as.user controls. It would have been obvious to modify the Maruya system

with the teachings of the Redstone references to' permit a user to click on the camera

pictures, such as those shown in Figures 24B, 26B and 27B of Maruya, to enable a user

to manually select such camera pictures to be the main central camera picture in order

to give the user flexibility in selecting the next camera view to be the main camera view.

Thus, the. Maruya, Redstone, Lin and Girgensohn '978 references collectively

disclose all of the limitations of Claims 1 and 3-8. As explained above and in Appendix

P, it would have been obvious to combine the teachings of these references to render

Claims 1 and 3-8 obvious. Accordingly, these references collectively raise a substantial

new question of patentability.

DC01/YOHAD/436261. I



6. Buehler Reference

a. Buehler Reference in combination with Girgensohn,'706Reference

Claims 1-8 of the '783 patent are rendered obvious over U.S. Patent Appl. Pub.

No. 2010/0002082 Al to Buehler et al. (the "Buehler" reference) (Exhibit K) in view of

the Girgensohn '706 (Exhibit B) reference. The application which resulted in publication

of the Buehler reference was filed on March 24, 2006. Therefore, the Buehler reference

qualifies as prior art against the '783 patent under 35 U.S.C. § 102(e). A detailed

comparison of the elements and limitations of Claim 2 of the '783 patent with the

relevant disclosure of the Buehler and Girgensohn '706 references is provided in

Appendix R, which is incorporated herein by reference. Requester also hereby

incorporates by reference the discussion of the Girgensohn '706 reference provided

above.

The Buehlerreference was relied upon alone and in combination with other prior

art references for the final rejection of the '783 patent claims. These rejections resulted

in the applicants resorting to combining the limitations of original Claims 1, 2 and 3 to

obtain the allowance of issued Claim 1, and to combining -the limitations of original

Claims 7, 16 and 18 to obtain allowance of issued Claim 5.

The Notice' of Allowability of the application for the '783 patent included the

examiner's statement of reasons for allowance of the issued claims. The reason for

allowance of issued Claim 1 was that the closest prior art (i.e., the combination of the

DCO1/YO HI A 1'436261 .1

Attorney Dkt. No. 017874-0003Reexaminationi Request

U.S Patent No. 7,777,783.

Buehler reference with US Pub: No. 2002/0097322 Al to Monroe et al.) did. not disclose

the last element of issued Claim 1, which recites:

presenting the first user control and the second user control in the userinterface in positions representative of the first topological relationship andthe second topological relationship, where in the first user control and thesecond user control are presented in positions around a presentation ofthe video data generated using the first member of the plurality of

Scameras.

June 18, 2010 Notice of Allowance, at 3. Issued Claims 2-4 were allowed "due to

their dependence on allowed independent claim 1." Id.

Similarly, issued Claim 5 was allowed because the closest prior art (i.e., the

combination of the Buehler reference with US Pub. No. 2005/0052532 Al to Elooz et

al.) did not disclose a portion of the last element of issued Claim 5, which recites: "using

a temporal offset to automatically stitch the first video data and the second video data,

the temporal offset being based on an estimated time of travel of the target." Id., at 4.

Issued Claims 6-8 were allowed "due to their dependence on allowed independent claim

16 [issued Claim 5]." Id.. The applicants did not contest or attempt to comment on the

stated reasons for allowance.

In view of the foregoing, the Buehler reference is admitted by .the applicants to

disclose all of the limitations of Claims" 1 and 5 save for (i) presenting user controls in

positions representative of the topological relationship of the cameras, which positions

are "around" a presentation of the video data generated using a selected camera, and

(ii) using a temporal offset based on an estimated time of travel between camera fields

1)C01/YO1IAD/43,2( t. i 74



of view when stitching together video from different cameras.

As explained above and in Appendix R, the Girgensohn '706 reference discloses

(i) positioning user controls around a presentation of video data such that the user

controls are in locations representative of the topological relationship of the cameras

with which the user controls are associated, and (ii) using a temporal offset when

stitching together video from different cameras, the only claim limitations ostensibly not

disclosed in Buehler.

Buehler teaches an icon layout in which secondary video data feeds 125 may be

placed in thumbnail windows which surround a primary video data. feed 115. See.

Buehler, Fig. 1.

I)C:I/YOI1AD/4326 I. 1



120

100

10

105

25L2

Buehler, Fig. 1.

It would have been obvious for one of ordinary skill in the art to modify the

Buehler system with the teachings of Girgensohn '706 so that the Buehler secondary

video data feeds were placed in positions representative of the topological relationships

between the cameras providing the secondary camera feeds and the camera providing

the primary video data feed, taken: from the viewpoint of the camera providing the

primary video data feed. It would also have been obvious in view of Girgensohn '706 to

modify the Buehler system so that the secondary video data feeds could be clickedDcO1/YOIIA I.)430261,I 76


U:S Patent No. 7,777,783

upon (i.e., act as user controls) to select a secondary video data feed to be the primary

video data feed. One of ordinary skill in the art would have been motivated to combine

these teachings in order to satisfy the need identified in Girgensohn '706 to have a "way

for users to browse video by quickly. ..switching to another video stream to keep the

activity in view." Girgensohn '706, Para. 0005., Accordingly, the combination of the

Buehler and Girgensohn,'706 references renders Claims 1-8 obvious and raises a

substantial new question of patentability.

b. Buehler Reference in combination with White,Girgensohn '978 and Lin References

The teachings of the Buehler reference .(Exhibit K) would have been readily

combined by one of ordinary skill in the art with those of the White (Exhibit C)

Girgensohn '978 (Exhibit B) and Lin (Exhibit F) references to render Claims. 1-8 of the

'783 patent obvious. A detailed comparison of the elements and limitationsof Claims 1-

8 of the '783 patent with the relevant disclosure of the Buehler, White, Girgensohn '978

and Lin references is. provided in Appendix S, which is incorporated herein by reference.

Requester also hereby incorporates by reference the discussion of the Buehler, White,

Girgensohn '978 and Lin references provided above.

The Buehler, White, Girgensohn '978 and Lin references all disclose systems


Abstract; see also Girgensohn '978, Para. 0008. As noted above, the Buehler reference

discloses all of thelimitations of Claims 1 and 5 except (i) presenting user controls inI)(C1/YOHA1/43626 .1 77




are "around" a presentation of the video data generated using a selected camera, and


of view when stitching together video from different cameras. Buehler also does not

disclose providing a thumbnail index of stitched together video.

With regard to Claim 1, Buehler teaches an icon layout in which secondary video

data feeds may be placed in thumbnail windows which surround a primary video data

feed. See Buehler, Fig. 1. White teaches that thumbnail views may be clicked upon to

select such views to be a main video view and that the thumbnail views may be

presented in positions on the interface screen which are representative of the

topological relationships of the cameras providing the main video view and the

thumbnail views. See Section V.A.2.,above. It would have been obvious for one of

ordinary skill in the art to combine the teachings of Buehler with those of White so that

the Buehler secondary video data feeds were placed in positions representative of the

topological relationships between the cameras providing the secondary camera feeds

and the camera providing the primary video data feed, taken from the viewpoint of the.

camera providing the primary video data feed, It would also have been obvious in view

of White to modify the Buehler system so that the secondary video data feeds could be

clicked upon (i.e., act as user controls) to select a secondary video data feed to be the

primary video data feed. One of ordinary skill in the art would have been motivated to

combine the teachings of Buehler and White to take advantage of the White system'sI)CO/YOIIA)1/436261 .1 78



ability to make "multiple streams available to a user without using an undue amount of

bandwith." See White, 1:65-67.

The elements of Claims 2 and 3 - format conversion, storing camera

identification information with video data, and stitching video together from different

cameras - are disclosed in Buehler.

With regard to Claim 4, Girgensohn '978 discloses a video surveillance system

that stitches together streams of surveillance video captured by different cameras in the

system and uses thumbnails to index the stitched video. See Girgensohn '978, Para,

0041 and Figs. 2-4, 9'11. Buehler discloses stitching video streams together. See

Buehler, Para. 0053-0055. It would have been obvious to modify the Buehler system to

use thumbnails to index stitched together video streams generated by the cameras in

the Buehler system, using the methods of Girgensohn '978 in order to provide a

summary of video taken from multiple cameras. The advantages of such a summary

are noted in the Girgensohn '978 reference. See Girgensohn '978, Para. 0008-0009,

0012-0013 and 0041.

With regard to Claim 5, the Lin reference discloses using an automatic temporal

offset to account for the travel time of a target between the fields of view of two

cameras, as discussed in Section V.B.2.b, above. Specifically, Lin discloses a system

that uses "a time duration parameter corresponding to traversing between the first zone

and each of the zones in the set of linked zones." See Lin, 8:60-64. The Lin "time

duration parameter" may be based on a "minimal traversal time, a maximum traversalI)COI/YIHAD/436261. I 7Qrv



time, a traversal time likelihood, and a probability distribution function of traversal time"

from one camera's view to another's. See Lin, 9:3-8. Thus, Lin discloses use of a

temporal offset that is "based on an estimated time of travel of the target" as required by

Claim 5. The Buehler reference, in combination with the White reference .discloses the

elements of Claims 6-8!

In view of the foregoing, the Buehler, White, Lin and Girgensohn '97.8 references

collectively disclose all of the limitations of Claims 1-8. As explained above and: in.

Appendix S, -it would have been obvious to combine the teachings of these references

to render Claims 1-8 obvious. Thus, the Buehler, White, Lin and Girgensohn '978

references raise a substantial new question of patentability.

c. Buehler Reference in combination with Roberts,Girgensohn '978 and Lin References

The teachings of the Buehler reference (Exhibit K) would have been readily

combined by one of ordinary skill in the art with those of the Roberts (Exhibit D)

Girgensohn '978 (Exhibit B) and Lin (Exhibit F) references to render Claims 1-8 of the

'783 patent obvious. A detailed oomparison of the elements and limitations of Claims 1-

8 of the '783 patent with the relevant disclosure of the Buehler, Roberts, Girgensohn

'978 and Lin references is provided in Appendix T, which is incorporated herein by

reference. Requester also hereby incorporates by reference the discussion of the-

Buehler, Roberts, Girgensohn '978 and Lin references provided above.

The Buehler, Roberts, Girgensohn '978 and Lin references all disclose systemsl ('0 I /Y 1 )I1AD43r 26 I. 1 80




Abstract; see also Girgensohn '978, Para. 0008. As noted above, the Buehler reference

discloses all of the limitations of Claims 1 and 5 except (i) presenting user. controls in


are "around" a presentation of the video data generated using a selected camera, and.


of view when stitching together video from different cameras.

With regard to Claim 1, Roberts teaches that user controls in the form of "button

regions," can be clicked upon and surround a display image provided by. a selected

camera. The "button regions" are located along the edge of the central display image

and are presented in display locations representative of topological relationship of the

cameras in the system, relative to the viewpoint of the camera providing the central.

display image. It would have been obvious to modify the Buehler system with the

teaching of Roberts so that the Buehler secondary video data feeds were placed in

positions representative of the topological relationships between the cameras providing

the secondary camera feeds and the camera providing the primary video data feed,

taken from the viewpoint of the camera providing the primary video data feed. It would

also have been obvious in view of Roberts to modify the Buehler system so that the

secondary video data feeds could-be clicked upon (i.e., act as user controls) to select a

secondary video data feed to be the primary video data feed. This modification of

Buehler would enable a user.to easily select the secondary video data feeds to be theI.x:tO/YOII AD/43 62 6 1 , 81

I



primary,video data feed.

With regard to Claim 4, Girgensohn '978 discloses a video surveillance system

that stitches together streams of surveillance video captured by different cameras''in the

system and uses thumbnails to index the stitched video. See Girgensohn '978, Para.

0041 and Figs. 2-4, 9-11. Buehler discloses stitching video streams together. See

Buehler, Para, 0053-0055. It would have been obvious to modify the Buehler system to,

use thumbnails to index stitched together video streams generated by the cameras in

the. Buehler system, using the methods of Girgensohn '978 in order to provide a

summary of video taken from multiple cameras. The advantages of such a summary

are noted in the Girgensohn '978 reference. See Girgensohn '978, Para. 0008-0009,

0012-0013 and 0041.

With regard to Claim 5, the Lin reference discloses using an automatic temporal

offset to account for the travel time of a target between the fields of view of two

cameras, as discussed in Section V.B.2.b, above. Specifically, Lin discloses a system

that uses "a time duration parameter corresponding to traversing between the first zone

and each of the zones in the set of linked zones." See Lin, 8:60-64. The Lin "time

duration parameter" may be based on a "minimal traversal time, a maximum traversal

time, a traversal time likelihood, and a probability distribution function of traversal time"

from one camera's view to another's. See Lin, 9:3-8. Thus, Lin discloses use of a

temporal offset that. is "based on an estimated time of travel of the target" as required by

Claim 5.DCO(/YOIDi.)/O43G261 .1 82

Attorney Dkt. No. 017874-0003'Reexamination Request


In view of the foregoing, the Buehler, Roberts, Lin .and Girgensohn '.978

references collectively disclose all of the limitations of Claims 1-8. As explained above

and in Appendix T, it would have been obvious to combine the teachings of these

references to render Claims 1-8 obvious. Thus, the Buehler, Roberts, Lin and

Girgensohn '978 references raise a substantial new question of patentability.

VI. CONCLUSION

For the foregoing reasons, substantial new questions of patentability with respect

to all claims of the '783 patent have been raised. Reexamination of all claims (1-8) of

the '783 patent is respectfully requested. The undersigned is acting as a representative

party of the real party in interest, VidSys, Inc. pursuant to 37 C.F.R. § 1.34.

VII. LIST OF EXHIBITS AND APPENDICES

Exhibit A U.S. Patent No. 7,777,783 to Chin et al.

Exhibit B U.S. Patent Appl. Pub. No. 2008/0088706 Al to Girgensohn et al.

Exhibit C U.S. Patent No. 7,196,722 to White et al.

Exhibit D U.S. Patent No. 7,295,228 to Roberts et al.

Exhibit E http://web.archive.orq/web/20040606014343/redstone-is.com/products services.html, pp. 1-11..

Exhibit F U.S. Patent No. 7,242,423 to Lin.

Exhibit G U.S. Patent Appl. Pub. No. 2006/0284978 Al to Girgensohn et al.

)COI/YOVIA14 3(i2(1. 83

Attorney Dkt. :No. 017874-0003Reexamination Request


Exhibit H U.S. Patent No. 5,258,837 to Gormley.

Exhibit I Balancing Tomorrow's Technology, Redstone Integrated Solutions, © 2003.

Exhibit J U.S. Patent No. 7,746,380 to Maruya et al.

Exhibit K U.S. Patent Appl. Pub. No. 2010/0002082 Al to Buehler et al.

DC0I /YOI AD)/43626 1.1

Attorney. Dkt. No. 017874-0003Reexamination Request


Appendix

Appendix

Appendix

Appendix

Appendix

Appendix

Appendix

Appendix

Appendix

Appendix J

Appendix K

Appendix L

Appendix

Appendix

Appendix O

Anticipation Based On The Girgensohn '706 Reference

Anticipation Based On The White.Reference

Anticipation Based On The Roberts Reference

Anticipation Based On The Redstone Products_Services Reference

Obviousness Based On The Girgensohn '706, Girgensohn '978; and LinReferences

Obviousness Based On The Girgensohn '706 and Gormley Reference

Obviousness Based On The Roberts, Girgensohn '706 and GormleyReferences

Obviousness Based On The Roberts, Girgensohn '978 and Lin References

Obviousness Based On The Redstone Products Services Reference andSecondary Redstone Reference

Obviousness Based On The Redstone, Girgensohn '706 and GormleyReferences

Obviousness Based On The Redstone, Girgensohn '978 and LinReferences

Obviousness Based On The White, Girgensohn '706 and GormleyReferences

Obviousness Based On The White, Girgensohn '978 and Lin References

Obviousness Based On The Maruva, Girgensohn '706 and GormleyReferences.

Obviousness Based On The Maruya, White, Girgensohn '978 and LinReferences

DC01/YO1AD/4302( I. I 85

Attorney Dkt. No. 01.7874-0003Reexamination Request


Appendix P

Appendix Q

Obviousness BasedReferences


On The Maruya, Roberts, Girgensohn '978 and Lin

On The Maruya, Redstone, Girgensohn '978 and Lin

Appendix R Obviousness Based On The Buehler and Girgensohn '706 References

Appendix S

Appendix T


Obviousness BasedLin References

On The Buehler, White, Girgensohn '978 and Lin

On The Buehler, Roberts and Girgensohn '978 and

Respectfully submitted,

Dated: January 21, 2011

KELLEY DRYE & W/REN, LLC3050 K Street, N.W., Suite 400Washington, D.C. 20007(202) 342-8400

DC:OI/YoIIAn)/46261 .1 86








Filed as Large Entity


Sub-Total inDescription Fee Code Quantity Amount

USD($)

Basic Filing:

Pages:

Claims:


Petition:



Extension-of-Time:

,y ,47,

Sub-Total inDescription Fee Code Quantity Amount USD($)

USD($)

Miscellaneous:

Submission- Information Disclosure Stt 1806 1 180 180

Total in USD ($) 180

W_% *

:'p ,:



APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO.

11/231,353 , 09/19/2005 Ken Prayoon Cheng 5330.07 (SMC) ' 4531

23308 7590

PETERS VERNY, L.L.P.425 SHERMAN AVENUESUITE 230PALO ALTO, CA 94306

EXAMINER

GRANT II, JEROME


2625

MAIL DATE DELIVERY MODE

04/06/2011 PAPER



PTOL-90A (Rev. 04/07)

04/06/2011

UNITED STATES DEPARTMENT.OF COMMERCEPatent and Trademark OfficeASSISTANT SECRETARY AND COMMISSIONEROF PATENTS AND TRADEMARKSWashington. D.C. 20231

This application has been withdraw from abandoned.

Thank you,

SUPERVISORY L.. STRUMBENTS EXMR.TECHN1OlGY:.CEER 2600

"t

,f..

_ __ _

f'

Transaction History Date9 0 14Date information retrieved from USPTO Patent

Application Information Retrieval (PAIR)system records at www.uspto.gov

. Application No. Applicant(s)

11/231,353 CHENG ET AL.Notice of Allowability Examiner Art Unit


-- The MAILING DATE of this communication appears on the cover sheet with the correspondence address--All claims being allowable, PROSECUTION ON THE MERITS IS (OR REMAINS) CLOSED in this application. If not includedherewith (or previously mailed), a Notice of Allowance (PTOL-85) or other appropriate communication will be mailed in due-course. THISNOTICE OF ALLOWABILITY IS NOT A GRANT OF PATENT RIGHTS. This application is subject to withdrawal from issue at the initiativeof the Office or upon petition by the applicant. See 37 CFR 1.313 and MPEP 1308.

1. I This communication is responsive to an amendment received 4-6-11.

2. I The allowed claim(s) is/are 14 and 15.

3. QI Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f).a) O All b) O some* c) O None of the:

1. O Certified copies of the priority documents have been received.

2. O Certified copies of the priority documents have been received in Application No.

3. O Copies of the certified copies of the priority documents have been received in this national stage application from the

International Bureau (PCT Rule 17.2(a)).

* Certified copies not received:

Applicant has THREE MONTHS FROM THE "MAILING DATE" of this communication to file a reply complying with the requirementsnoted below. Failure to timely comply will result in ABANDONMENT of this application.THIS THREE-MONTH PERIOD IS NOT EXTENDABLE.

4. O A SUBSTITUTE OATH OR DECLARATION must be submitted. Note the attached EXAMINER'S AMENDMENT or NOTICE OFINFORMAL PATENT APPLICATION (PTO-1 52) which gives reason(s) why the oath or declaration is deficient.

5. O CORRECTED DRAWINGS (as "replacement sheets") must be submitted.

(a) O including changes required by the Notice of Draftsperson's Patent Drawing Review ( PTO-948) attached

1) El hereto or 2) O to Paper No./Mail Date

(b) O including changes required by the attached Examiner's Amendment / Comment or in the Office action ofPaper No./Mail Date

Identifying indicia such as the application number (see 37 CFR 1.84(c)) should be written on the drawings in the front (not the back) ofeach sheet. Replacement sheet(s) should be labeled as such in the header according to 37 CFR 1.121(d).

6. r DEPOSIT OF and/or INFORMATION about the deposit of BIOLOGICAL MATERIAL must be submitted. Note theattached Examiner's comment regarding REQUIREMENT FOR THE DEPOSIT OF BIOLOGICAL MATERIAL.

Attachment(s)1. ElO Notice of References Cited (PTO-892)

2. O Notice of Draftperson's Patent Drawing Review (PTO-948)

3. 0 Information Disclosure Statements (PTO/SB/08),Paper No./Mail Date 3/11

4. O Examiner's Comment Regarding Requirement for Deposit-of Biological Material

5. O Notice of Informal Patent Application

6. O Interview Summary (PTO-413),Paper No./Mail Date _ .

7. O Examiner's Amendment/Comment

8. 0 Examiner's Statement of Reasons for Allowance

9. Ol Other


U.S. Patent and Trademark OfficePTOL-37 (Rev. 08-06)

Noic ofAlwblt ato ae oMi ae2101

Part of Paper No./Mail Date 20110411Noice of Allowability

Application/Control Number: 11/231,353Art Unit: 2625

Reasons for Allowance

Claims 14 and 15 were previously indicated as being objected as containing

allowable matter and have been amended to include the subject matter of the

base claim.

Any inquiry concerning this communication or earlier communications from theexaminer should be directed to Jerome Grant II whose telephone number is 571-272-7463. The examiner can normally be reached on Mon.-Fri. from 9:00 to 5:00.

If attempts to reach the examiner by telephone are unsuccessful, the examiner'ssupervisor, mark Zimmerman, can be reached on 571-272-7653. The fax phonenumber for the organization where this application or proceeding is assigned is 571-273-8300.

Information regarding the status of an application may be obtained from thePatent Application Information Retrieval (PAIR) system. Status information forpublished applications may be obtained from either Private PAIR or Public PAIR.Status information for unpublished applications is available through Private PAIR only.For more information about the PAIR system, see http://portal.uspto.gov/external/portal.Should you have questions on access to the Private PAIR system, contact theElectronic Business Center (EBC) at 866-217-9197 (toll-free).

/Jerome Grant II/

Primary Examiner, Art Unit 2625

Page 2

IN THE


APPLICANTS:

APPLICATION NO.:

CONFIRMATION NO.:

FILED:

TITLE:

EXAMINER:

GROUP ART UNIT:

ATTY.DKT.NO.:

Ken Prayoon Cheng

11/231,353

4531

September 19, 2005

Adaptive Multi-Modal Integrated Biometric IdentificationDetection and Surveillance Systems

Jerome Grant, II

2625

5330.07 (SMC)

MAIL STOP AMENDMENTCOMMISSIONER FOR PATENTSP.O. BOX 1450ALEXANDRIA, VA 22313-1450

List of Co-Pending Patent Applications That May Be DirectedTowards Similar Subject Matter

Examiner's Serial First-Named TitleInitials Number Inventor Filing Date

/JG! 12/838,973 Hu Chin Multi-Video Navigation 7-19-2010

IJGI 11/728,404 Hu Chin Multi-Video Navigation System 03-23-2007

Examiner's DateSignature: /Jerome Grant II/ 04/11/2011

Doc code: IDS PTO/SBI08a (01-10)Approved for use through 07131/2012. OMB 0651-0031

Doc description: Information Disclosure Statement (IDS) Filed U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCEUnder the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it contains a valid OMB control number.




STATEMENT BY APPLICANT Art Unit' 2625( Not for submission under 37 CFR 1.99)



U.S.PATENTS Remove

Pages,Columns, Lines whereExamine Cite Kind Name of Patentee or Applicant

Patent Number Issue Date Relevant Passages or RelevantInitial* No Codel of cited Document

Figures Appear

/J /1 6604093 B1 2003-08-05 Etzion et al.

2 6906709 B1 2005-06-14 Larkin et al.

/JG/ 3 6408404 B1 1 2002-06-18 Michael Ladwig

/JG 4 6711587 B1 2004-03-23 Dufaux

/JG,5 6628829 B1 2003-09-30 Chasen

6 5912980 1999-06-15 Hunke

/JG/

/JG/ 7 7136524 B1 2006-11-14. Goh et al.

/JG/ 8 6072496 2000-06-06 Guenter et al.

EFS Web 2.1.17


Filing Date 2005-09-19IN FORMATION DISCLOSURE First Named Inventor Ken Prayoon Cheng

STATEMENT BY APPLICANT Art Unit12625( Not for submission under 37 CFR 1.99)



9 6625315 B2 2003-09-23 Laumeyer et al./JG/

/JG/ 10 6404900 B1 2002-06-11 Qian et al.

/JG/ 11 6707487 B1 2004-03-16 Aman et al.

/JG! 12' 5473369 1995-12-05 Abe

/JG/ 13 7777783 B1 2010-08-17 Chin et al.

/JG/ 14 7242810 B2 2007-07-10 Chang

/JG/ 15 6008912 1999-12-28 Salo et al.

/JGi 16 5838465 1998-11-17 Satou et al.

17 6335985 B1 2002-01-01 Sambonsugi et al./JG/

/JG/ 18 7187783 B2 2007-03-06 Moon et al.

/JG/ 19 6157469 2000-12-05 Mestha

/Jlerome Grant II/ 04/11/2011EFS Web 2.1.17


Filing Date 2005-09-19INFORMATION DISCLOSURE First Named Inventor Ken Prayoon ChengSTATEMENT BY APPLICANT ArtUnit 2625( Not for submission under 37 CFR 1.99)



/JG/ 20 7525576 B2 2009-04-28 Kannermark et al.

21 7382249 B2 2008-06-03 Fancella

/JG/

/JG/ 22 7384590 B2 2008-06-10 Kelly et al.

/JG/ 23 7023913 B1 2006-04-04 Monroe et al.

/JG/ 24 7116353 B2 2006-10-03 Hobson et al.

/JG/ 25 7456727 B2 2008-11-25 Pinter et al.

/JG/ 26 7058204 B2 2006-06-06 Hildreth et al.

/JG/ 27 7385590 B2 2008-06-10 Millar et al.

/JG/ 28 7522186 B2 2009-04-21 Arpa et al.

29

30

EFS Web 2.1.17/Jerome Grant II/ 04/11/2011


Filing Date 2005-09-19INFORMATION DISCLOSURE First Named Inventor Ken Prayoon ChengSTATEMENT BY APPLICANT Art Unit 2625( Not for submission under 37 CFR 1.99)



31

32

/JGI 33 7196722 B2 2007-03-27 White et al.

34 7295228 B2 2007-11-13 Roberts et al./JG/

!JG/ 35 7242423 B2 2007-07-10 Lin

/JG/ 36 5258837 1993-11-02 Gormley

/JG 37 5479574 1995-12-26 Glier et al.

/JG/ 38 5701398 1997-12-23 Glier et al.

/JG/ 39 6757668 B1 2004-06-29 Goebel et al.

/JG/ 40 5769074 1998-06-23 Bamhill et al.

/JG/ 41 6306087 B1 2001-10-23 Bamhill et al.

/Jerome Granlt II/EFS Web 2.1.17 04/11/201,

Application Number 1.1231353

Filing Date 2005-09-19INFORMATION DISCLOSURE First Named Inventor Ken Prayoon ChengSTATEMENT BY APPLICANT Artnit 2625

Art Unit 2625( Not for submission under 37 CFR 1.99)



/JG/ 42 6248063 B1 2001-06-19. Bamhill et al.

/JG! 43 5835901 1998-11-10 Duvoisin et al.

/JG/ 44 7366174 B2 2008-04-29 MacFaden et al.

/JG/45 7319796 B1 2008-01-15 Sharp

/JG/ 46 7286687 B2 2007-10-23 Lindwurm et al.

/JG/47 7277485 B1 2007-10-02 Puri.

/JG, 48 7269591 B2 2007-09-11 Wesinger et al.

/JG/ 49 7213174 B2 . 2007-05-01 Dahlquist et al.

/JG! 50 7127464 B2 2006-10-24 Wesinger etal.

/JG! 51 7028034 B2 2006-04-11 Wesinger et al

/JG 52 6976207 B 2005-12-13 u et.52 6976207 B1 2005-12-13 RujBn et al.

EFS Web 2.1.17 /Jerome Grant II/ 04/11/2011


Filing Date 2005-09-19IN FORMATION DISCLOSURE First Named Inventor Ken Prayoon ChengSTATEMENT BY APPLICANT Art Unit 2625



/JG/ 53 6970513 B1 2005-11-29 Puri et al.

/JG! 54 6968006 B1 2005-11-22 Puri et al.

/JG/ 55 6963899 B1 2005-11-08 Femandez et al.

/JG 56 6909745 B1 2005-06-21 Purl et al.

/JG/ 57 6944319 B1 2005-09-13 Huang et al.

/JG/ 58 7746380 B2 2010-06-29 Maruya et al.

If you wish to add additional U.S. Patent citation information please click the Add button. Add

U.S.PATENT APPLICATION PUBLICATIONS Remove

Pages,Columns,Lines whereExamine Publication Kind Publication Name of Patentee or ApplicantPagesColumnsLines whereCite No Relevant Passages or Relevant

Initial* Number Codel Date of cited DocumentFigures Appear

/JG/ 1 20050222081 Al 2005-10-06 Jakobson et al.

2 20080294588 Al 2008-11-27 Morris et al./JG

/JG/ 3 20100002082 Al 2010-01-07 Buehler et al.

04/11/2011/Jerome Grant II!EFS Web 2.1.17


Filing Date 2005-09-19IN FORMATION DISCLOSURE First Named Inventor Ken Prayoon ChengSTATEMENT BY APPLICANT Art Unit12625( Not for submission under 37 CFR 1.99)



/JG!/ 4 20030202102 Al 2003-10-30 Shiota et al.

/JG/ 5 20040100563 Al 2004-05-27 Sablak et al.

6 20030107653 Al 2003-06-12 Utsumi et al.

/JG/

/JG/ 7 20010048765 Al 2001-12-06 Yi et al.

/JG/ 8 20070146484 Al 2007-06-28 Horton et al.

/JG/ 9 20020097322 Al 2002-07-25 Monroe et al.

/JG/ 10 20040001149 Al 2004-01-01 Smith et al.

/JG/ 11 20080218590 Al 2008-09-11 Park et al.

/JG/ 12 20080068464 Al 2008-03-20 Kitagawa et al.

IJGI 13 20080106597 Al 2008-05-08 Amini et al.

/JG/ 14 20060279628 Al 2006-12-14 Fleming

EFS Web 2.1.17 /Jerome GrantIll/ 04/11/2011





/JG/ 15 20060221184 Al 2006-10-05 Vallone et al.

/JG/ 16 20070146484 Al 2007-06-28 Horton et al.

/JGi 17 20050132414 Al 2005-06-16 Bentley et al.

/JG/ 18 20080190949 Al 2008-08-14 Ivanov et al.

/JG/ 19 20050052532 Al 2005-03-10 Elooz et al.

/JG/ 20 20080079554 Al 2008-04-03 Boice

/JG/ 21 20060017807 Al 2006-01-26 Lee et al.

/JG, 22 20040136574 Al 2004-07-15 Kozakaya et al.

23 20030169908 Al 2003-09-11 Kim et al.

/JG/ 24 20050100209 Al 2005-05-12 Lewis et al.

/JG/ 25 20040022442 Al 2004-02-05 Kim et al.



Filing Date 2005-09-19IN FORMATION DISCLOSURE ,First Named Inventor Ken Prayoon Cheng

STATEMENT BY APPLICANT ArtUnit 2625( Not for submission under 37 CFR 1.99)



/JG/ 26 20060203090 Al 2006-09-14 Wang et al.

/JG/ 27 20070154088 Al 2007-07-05 Goh et al.

iJGI 28 20060284978 Al 2006-12-21 Girgensohn et al.

/JG/ 29 20080088706 Al 2008-04-17 Girgensohn et al.

If you wish to add additional U.S. Published Application citation information please click the Add button Add

FOREIGN PATENT DOCUMENTS Remove

Pages,Columns,LinesName of Patentee' orExaminer Cite Foreign Document Country Kind Publication where RelevantInitial* No Number3 Code 2 i Code4 Date Applicant of cited Passages or Relevant

DocumentFigures Appear

PROXIMEX/JG/ 1 2007/044037 WO Al 2007-04-19 CORPORATIONMX

If you wish to add additional Foreign Patent Document citation information please click the Add button Add

NON-PATENT LITERATURE DOCUMENTS Remove

Include name of the author (in CAPITAL LETTERS), title of the article (when appropriate), title of the itemExaminer CiteInitials* No (book, magazine, journal, serial, symposium, catalog, etc), date, pages(s), volume-issue number(s), Ts

publisher, city and/or country where published.

.JG/ 1 PCT/US05/44656 Intemational Search Report and Written Opinion, June 26, 2006 -

iJG/ 2 PCT/US05/43808 International Search Report and Written Opinion, October 10, 2007 E

/Jerome Grant I/l04/11/2011

EFS Web 2.1.17





EFS Web 2.1.17

/JG/ 3 PCT/US05/33378 Intemational Search Report and Written Opinion, April 26, 2006

4 PCT/US05/33750 Intemational Search Report and Written Opinion, May 2, 2007

IJG!

/JG/ 5 PCT/US05/16961 Intemational Search Report and Written Opinion, October 17, 2006

/JG/6 VidSys, Inc. Complaint filed in the US District Court for the Eastem district of Virginia on October 19,

/JGI 7 Redstone Integrated Solutions Documents, 2003

8 951001,525 Reexamination Request for 7,777,783, filed January 21, 2011

If you wish to add additional non-patent literature document citation information please click the Add button

EXAMINER SIGNATURE

Examiner Signature /Jerome Grant II/ Date Considered 04/11/2011

*EXAMINER: Initial if reference considered, whether or not citation is in conformance with MPEP 609. Draw line through acitation if not in conformance and not considered. Include copy of this form with next communication to applicant.

1 See Kind Codes of USPTO Patent Documents at www.USPTO.GOV or MPEP 901.04. 2 Enter office that issued the document, by the two-letter code (WIPOStandard ST.3). 3 For Japanese patent documents, the indication of the year of the reign of the Emperor must precede the serial number of the patent document4 Kind of document by the appropriate symbols as indicated on the document under WIPO Standard ST.16 if possible. 5 Applicant is to place a check mark here ifEnglish language translation is attached.



NOTICE OF ALLOWANCE AND FEE(S) DUE

23308 7590

PETERS VERNY , L.L.P.425 SHERMAN AVENUESUITE 230PALO ALTO, CA 94306

04/18/2011EXAMINER

GRANT II, JEROME


2625

DATE MAILED: 04/18/2011


11/231,353 09/19/2005 Ken Prayoon Chcng 5330.07 (SMC) 4531

TITLE OF INVENTION: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCESYSTEMS

APPLN. TYPE SMALL ENTITY ISSUE FEE DUE PUBLICATION FEE DUE PREV. PAID ISSUE FEE TOTAL FEE(S) DUE DATE DUE

nonprovisional YES $755 $300 $0 $1055 07/18/2011

THE APPLICATION IDENTIFIED ABOVE HAS BEEN EXAMINED AND IS ALLOWED FOR ISSUANCE AS A PATENT.PROSECUTION ON THE MERITS IS CLOSED. THIS NOTICE OF ALLOWANCE IS NOT A GRANT OF PATENT RIGHTS.TIHIS APPLICATION IS SUBJECT TO WITHDRAWAL FROM ISSUE AT THE INITIATIVE OF THE OFFICE OR UPONPETITION BY THE APPLICANT. SEE 37 CFR 1.313 AND MPEP 1308.

THE ISSUE FEE AND PUBLICATION FEE (IF REQUIRED) MUST BE PAID WITHIN THREE MONTHS FROM THEMAILING DATE OF THIS NOTICE OR THIS APPLICATION SHALL BE REGARDED AS ABANDONED. THISSTATUTORY PERIOD CANNOT BE EXTENDED. SEE 35 U.S.C. 151. THE ISSUE. FEE DUE INDICATED ABOVE DOESNOT REFLECT A CREDIT FOR ANY PREVIOUSLY PAID ISSUE FEE IN THIS APPLICATION. IF AN ISSUE FEE HASPREVIOUSLY BEEN PAID IN THIS APPLICATION (AS SHOWN ABOVE), THE RETURN OF PART B OF THIS FORMWILL. BE CONSIDERED A REQUEST TO REAPPLY THE PREVIOUSLY PAID ISSUE FEE TOWARD THE ISSUE FEE NOWDUE.

HOW TO REPLY TO THIS NOTICE:

I. Review the SMALL ENTITY status shown above.

If the SMALL ENTITY is shown as YES, verify your currentSMALL ENTITY status:

A. If the status is the same, pay the TOTAL FEE(S) DUE shownabove.

B. If the status above is to be removed, check box 5b on Part B -Fee(s) Transmittal and pay the PUBLICATION FEE (if required)and twice the amount of the ISSUE FEE shown above, or

If the SMALL ENTITY is shown as NO:

A. Pay TOTAL FEE(S) DUE shown above, or

B. If applicant claimed SMALL ENTITY status before, or is nowclaiming SMALL ENTITY status, check box 5a on Part B - Fee(s)Transmittal and pay the PUBLICATION FEE (if required) and 1/2the ISSUE FEE shown above.

II. PART B - FEE(S) TRANSMITTAL, or its equivalent, must be completed and returned to the United States Patent and Trademark Office(USPTO) with your ISSUE FEE and PUBLICATION FEE (if required). If you are charging the fee(s) to your deposit account, section "4b"of Part B - Fee(s) Transmittal should be completed and an extra copy of the form should be submitted. If an equivalent of Part B is filed, arequest to reapply a previously paid issue fee must be clearly made, and delays in processing may occur due to the difficulty in recognizingthe paper as an equivalent of Part B.

III. All communications regarding this application must give the application number. Please direct all communications prior to issuance toMail Stop ISSUE FEE unless advised to the contrary.

IMPORTANT REMINDER: Utility patents issuing on applications filed on or after Dec. 12, 1980 may require payment ofmaintenance fees. It is patentee's responsibility to ensure timely payment of maintenance fees when due.

Page 1 of 3

PTOL-85 (Rev. 02/11)

PART B - FEE(S) TRANSMITTAL

Complete and send this form, together with applicable fee(s), to: Mail Mail Stop ISSUE FEECommissioner for PatentsP.O. Box 1450Alexandria, Virginia 22313-1450

or Fax (571)-273-2885

INSTRUCTIONS: This form should be used for transmitting the ISSUE FEE and PUBLICATION FEE (if required). Blocks 1 through 5 should be completed whereappropriate. All further correspondence including the Patent, advance orders and notification of maintenance fees will be mailed to the current correspondence address asindicated unless corrected below or directed otherwise in Block 1, by (a) specifying a new correspondence address; and/or (b) indicating a separate "FEE ADDRESS" formaintenance fee notifications.

CURRENT CORRESPONDENCE ADDRESS (Note: Use Block I for any change of address) Note: A certificate of mailing can only be used for domestic mailings of theFee(s) Transmittal. This certificate cannot be used for any other accompanyingpapers. Each additional paper, such as an assignment or formal drawing, musthave its own certificate of mailing or transmission.

23308 7590 04/18/2011

PETERS VERNY , L.L.P. Certificate of Mailing or TransmissionI hereby certify that this Fee(s) Transmittal is being deposited with the United

425 SHERMAN AVENUE States Postal Service with sufficient postage for first class mail in an envelope

SUITE 230 addressed to the Mail Stop ISSUE FEE address above, or being facsimile

(Depositors name)

(Signature)

(Date)

APPLICATION NO. FILING DATE FIRST NAMED INVENTOR . ATT'IORNEY DOCKET NO. CONFIRMATION NO.

11/231,353 09/19/2005 Ken Prayoon Cheng 5330.07 (SMC) 4531

TITLE OF INVENTION: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCESYSTEMS

APPLN. TYPE SMALL ENTITY ISSUE FEE DUE PUBLICATION FEE DUE PREV. PAID ISSUE FEE TOTAL.FEE(S) DUE DATE DUE

nonprovisional YES $755 $300 $0 $1055 07/18/2011

EXAMINER ART UNIT CLASS-SUBCLASS

GRANT II, JEROME 2625 348-143000

1. Change of correspondence address or indication of "Fee Address" (37 2. For printing on the patent front page, listCFR 1.363). (1) the names of up to 3 registered patent attorneys 1

Q0 Change of correspondence address (or Change of Correspondence or agents OR, alternatively,Address form PTO/SB/122) attached. (2) the name of a single firm (having as a member a 2[ "Fee Address" indication (or "Fee Address" Indication form registered attorney or agent) and the names of up toPTO/SB/47; Rev 03-02 or more recent) attached. Use of a Customer 2 registered patent attorneys or agents. If no name is 3Number is required. listed, no name will be printed.

3. ASSIGNEE NAME AND RESIDENCE DATA TO BE PRINTED ON THE PATENT (print or type)

PLEASE NOTE: Unless an assignee is identified below, no assignee data will appear on the patent. If an assignee is identified below, the document has been filed forrecordation as set forth in 37 CFR 3.11. Completion of this form is NOT a substitute for filing an assignment.

(A) NAME OF ASSIGNEE (B) RESIDENCE: (CITY and STATE OR COUNTRY)

Please check the appropriate assignee category or categories (will not be pr;inted on the patent) : 0 Individual Q Corporation or other private group entity Government

4a. The following fee(s) are submitted: 4b. Payment of Fee(s): (Please first reapply any previously paid issue fee shown above)

Q Issue Fee D A check is enclosed.

Q Publication Fee (No small entity discount permitted) O Payment bycredit card. Form PTO-2038 is attached.

Q Advance Order - # of Copies Q The Director is hereby authorized to charge the required fee(s), any deficiency, or credit anyoverpayment, to Deposit Account Number • (enclose an extra copy of this form).

5. Change in Entity Status (from status indicated above)

Q a. Applicant claims SMALL ENTITY status. See 37 CFR 1.27. O b. Applicant is no longer claiming SMALL ENTITY status. See 37 CFR 1.27(g)(2).

NOTE: The Issue Fee and Publication Fee (if required) will not be accepted from anyone other than the applicant; a registered attorney or agent; or the assignee or other party ininterest as shown by the records of the United States Patent and Trademark Office.

Authorized Signature . Date

Typed or printed name Registration No.

This collection of information is required by 37 CFR 1.311. The information is required to obtain or retain a benefit by the public which is to file (and by the USPTO to process)an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.14. This collection is estimated to take 12 minutes to complete, including gathering, prepanng, andsubmitting the completed application form to the USPTO. Time will vary depending upon the individual case. Any comments on the amount of time you require to completethis form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S. Patent and Trademark Office, U.S. Department of Commerce, P.O.Box 1450, Alexandria, Virginia 22313-1450. DO NOT SEND FEES OR COMPLETED FORMS TO THIS ADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450,Alexandria, Virginia 22313-1450.Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number.

OMB 0651-0033 U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCEPTOL-85 (Rev. 02/11) Approved for use through 08/31/2013.





23308 7590


04/18/2011EXAMINER

GRANT II, JEROME

ART UNIT. PAPER NUMBER

2625

DATE MAILED: 04/18/2011

Determination of Patent Term Adjustment under 35 U.S.C. 154 (b)(application filed on or after May 29, 2000)

The Patent Term Adjustment to date is 1127 day(s). If the issue fee is paid on the date that is three months after themailing date of this notice and the patent issues on the Tuesday before the date that is 28 weeks (six and a halfmonths) after the mailing date of this notice, the Patent Term Adjustment will be 1127 day(s).

If a Continued Prosecution Application (CPA) was filed in the above-identified application, the filing date thatdetermines Patent Term Adjustment is the filing date of the most recent CPA.

Applicant will be able to obtain more detailed information by accessing the Patent Application Information Retrieval(PAIR) WEB site (http://pair.uspto.gov).

Any questions regarding the Patent Term'Extension or Adjustment determination should be directed to the Office ofPatent Legal Administration at (571)-272-7702. Questions relating to issue and publication fee payments should bedirected to the Customer Service Center of the Office of Patent Publication at 1-(888)-786-0101 or (571)-272-4200.

Page 3 of 3

PTOL-85 (Rev. 02/11)


The Privacy Act of 1974 (P.L. 93-579) requires that you be given certain information in connection withyour submission of the attached form related to a patent application or patent. Accordingly, pursuant tothe requirements of the Act, please be advised that: (1) the general authority for the collection of thisinformation is 35 U.S.C. 2(b)(2); (2) furnishing of the information solicited is voluntary; and (3) theprincipal purpose for which the information is used by the U.S. Patent and Trademark Office is to processand/or examine your submission related to a patent application or patent. If you do not furnish therequested information, the U.S. Patent and Trademark Office may not be able to process and/or examineyour submission, which may result in termination of proceedings or abandonment of the application orexpiration of the patent.


1. The information on this form will be treated confidentially to the extent allowed under the Freedomof Information Act (5 U.S.C. 552) and the Privacy Act (5 U.S.C 552a). Records from this system ofrecords may be disclosed to the Department of Justice to determine whether disclosure of theserecords is required by the Freedom of Information Act.

2. A record from this system of records may be disclosed, as a routine use, in the course of presentingevidence to a court, magistrate, or administrative tribunal, including disclosures to opposing counselin the course of settlement negotiations.

3. A record in this system of records may be disclosed, as a routine use, to a Member of Congresssubmitting a request involving an individual, to whom the record pertains, when the individual hasrequested assistance from the Member with respect to the subject matter of the record.

4. A record in this system of records may be disclosed, as a routine use, to a contractor of the Agencyhaving need for the information in order to perform a contract. Recipients of information shall berequired to comply with the requirements of the Privacy Act of 1974, as amended, pursuant to 5U.S.C. 552a(m).

5. A record related to an International Application filed under the Patent Cooperation Treaty in thissystem of records may be disclosed, as a routine use, to the International Bureau of the WorldIntellectual Property Organization, pursuant to the Patent Cooperation Treaty.

6. A record in this system of records may be disclosed, as a routine use, to another federal agency forpurposes of National Security review (35 U.S.C. 181) and for review pursuant to the Atomic EnergyAct (42 U.S.C. 218(c)).

7. A record from this system of records may be disclosed, as a routine use, to the Administrator,General Services, or his/her designee, during an inspection of records conducted by GSA as part ofthat agency's responsibility to recommend improvements in records management practices andprograms, under authority of 44 U.S.C. 2904 and 2906. Such disclosure shall be made in accordancewith the GSA regulations governing inspection of records for this purpose, and any other relevant(i.e., GSA or Commerce) directive. Such disclosure shall not be used to make determinations aboutindividuals.

8. A record from this system of records may be disclosed, as a routine use, to the public after eitherpublication of the application pursuant to 35 U.S.C. 122(b) or issuance of a patent pursuant to 35U.S.C. 151. Further, a record may be disclosed, subject to the limitations of 37 CFR 1.14, as aroutine use, to the public if the record was filed in an application which became abandoned or inwhich the proceedings were terminated and which application is. referenced by either a publishedapplication, an application open to public inspection or an issued patent.

9. A record from this system of records may be disclosed, as a routine use, to a Federal, State, or locallaw enforcement agency, if the USPTO becomes aware of a violation or potential violation of law orregulation.


EFSID: 9933650




ADAPTIVE MU LTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATIONTitle of Invention:

DETECTION AND SURVEILLANCE SYSTEMS






Receipt Date: 21-APR-2011










Authorized User

File Listing:

Documentument Description File ame File Size(Bytes)/ Multi PagesNumber Message Digest Part /.zip (if appl.)

1 Issue Fee Payment (PTO-85 B) . 5330_07-issue-fee.pdf

Warnings:

Information:

2 Fee Worksheet (PTO-875) fee-info.

Warnings:

Information:

Total Files Size (in bytes) 12?071

This Acknowledgement Receipt evidences receipt on the noted date by the USPTO of the indicated documents,characterized by the applicant, and includingpage counts, where applicable. It serves as evidence of receipt similar to aPost Card, as described in MPEP 503.

New Applications Under 35 U.S.C. 111If a new application is being filed and the application includes the necessary components for a filing date (see 37CFR1.53(b)-(d) and MPEP 506), a Filing Receipt (37 CFR 1.54) will be issued in due course and the date shown on thisAcknowledgement Receipt will establish the filing date of the application.



PART B - FEE(S) TRANSMITTAL

Complete and send this form, together with applicable fee(s), to: Mail Mail Stop ISSUE FEECommissioner for PatentsP.O. Box 1450Alexandria, Virginia 22313-1450

or Fax (571)-273-2885

INSTRUCTIONS: This form should be used for transmitting the ISSUE FEE and PUBLICATION FEE (if required). Blocks 1 through 5 should be completed whereappropriate. All further correspondence including the Patent, advance orders and notification of maintenance fees will be mailed to the current correspondence address asindicated unless corrected below or directed otherwise in Block 1, by (a) specifying a new correspondence address; and/or (b) indicating a separate "FEE ADDRESS" formaintenance fee notifications.

CURRENT CORRESPONDENCE ADDRESS (Note: Use Block I for any change of address) .Note: A certificate of mailing can only be used for domestic mailings of theFee(s) Transmittal. This certificate cannot be used for any other accompanyingpapers. Each additional paper, such as an assignment or formal drawing, musthave its own certificate of mailing or transmission.

23308 7590 04/18/2011

PETERS VERNY , L.L.P. Certificate of Mailing or Transmission425 SHERMAN AVENUE I hereby certify that this Fee(s) Transmittal is being deposited with the United

States Postal Service with sufficient postage for first class mail in an envelopeSUITE 230 addressed to the Mail Stop ISSUE FEE address above, or being facsimile

PALO ALTO, CA 94306 transmitted to the USPTO (571) 273-2885, on the date indicated below.

(Depositor's name)

(Signature)

(Date)



TITLE OF INVENTION: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND .SURVEILLANCESYSTEMS

APPLN. TYPE SMALL ENTITY ISSUE FEE DUE PUBLICATION FEE DUE I PREV. PAID ISSUE FEE TOTAL FEE(S) DUE DATE DUE

nonprovisional YES $755 .. $300 $0 $1055 07/18/2011

EXAMINER ART UNIT CLASS-SUBCLASS

GRANT II, JEROME 2625 348-143000

1. Change of correspondence address or indication of "Fee Address" (37 2. For printing on the patent front page, listCFR 1.363). (1) the names of up to 3 registered patent attorneys I Peters Verny, LLP

O Change of correspondence address (or Change of Correspondence or agents OR, alternatively,Address form PTO/SB/122) attached. (2) the name of a single firm (having as a member a 2O "Fee Address" indication (or "Fee Address" Indication form registered attorney or agent) and the names of up toPT1O/SB/47; Rev 03-02 or more recent) attached. Use of a Customer 2 registered patent attorneys or agents. If no name is 3Number is required. listed, no name will be printed.

3. ASSIGNEE NAME AND RESIDENCE DATA TO BE PRINTED ON THE PATENT (print or type)

PLEASE NOTE: Unless an assignee is identified below, no assignee data will appear on the patent. If an assignee is identified below, the document has been filed forrecordation as set forth in 37 CFR 3.11. Completion of this form is NOT a substitute for filing an assignment.

(A) NAME OF ASSIGNEE

Proximex Corporation(B) RESIDENCE: (CITY and STATE OR COUNTRY)

Sunnyvale, CA

Please check the appropriate assignee category or categories (will not be printed on the patent) : 0 Individual Q§ Corporation or other private group entity O Government

4a. The following fee(s) are submitted: 4b. Payment of Fee(s): (Please first reapply any previously paid issue fee shown above)

SIssue Fee O A check is enclosed.

[Publication Fee (No small entity discount permitted) O Payment by credit card. Form PTO-2038 is attached.

O Advance Order - # of Copies [_[The Director is hereby authorized to charge the required fee(s), any deficiency, or credit anyoverpayment, to Deposit Account Number -1-64- 33-1-- (enclose an extra copy of this form).

5. Change in Entity Status (from status indicated above)

Q a. Applicant claims SMALL ENTITY status. See 37 CFR 1.27. b. Applicant is no longer claiming SMALL ENTITY status. See 37 CFR 1.27(g)(2).

NO1TE: The Issue Fee and Publication Fee (if required) will not be accepted from anyone other than the applicant; a registered attorney or agent; or the assignee or other party ininterest as shown by the records of the United States Patent and Trademark Office.

Authorized Signature /Robert Hayden, #42,645/ Date April 21, 2011

Typed or printed name Robert Hayden Registration No. 42,645

'This collection of information is required by 37 CFR 1.311. The information is required to obtain or retain a benefit by the public which is to file (and by the USPTO to process)an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.14. This collection is estimated to take 12 minutes to complete,.including gathering, preparing, andsubmitting the completed application form to the USPTO. Time will vary depending upon the individual case. Any comments on the amount of time you require to completethis form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S. Patent and Trademark Office, U.S. Department of Commerce, P.O.Box 1450, Alexandnria, Virginia 22313-1450. DO NOT SEND FEES OR COMPLETED FORMS TO THIS ADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450,Alexandria, Virginia 22313-1450.Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number.

OMB 0651-0033 U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCEPTOL-85 (Rev. 02/11) Approved for use through 0831/2013.




Title of Invention: ADAPTIVE MU LTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATIONDETECTION AND SURVEILLANCE SYSTEMS






Sub-Total inDescription Fee Code Quantity Amount Sub-Total in

USD($)

Basic Filing:

Pages:

Claims:


Petition:



Utility Appl issue fee 1501 1 1510 1510

Certificate of correction 1811 1 100 100


USD($)

Extension-of-Time:

Miscellaneous:



EFS ID: 9985391




ADAPTIVE MU LTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATIONTitle of Invention: DETECTION AND SURVEILLANCE SYSTEMS



Filer: Steven M. Colby/Melinda Tompkins

Filer Authorized By: Steven M. Colby


Receipt Date: 29-APR-2011







Payment was successfully received in RAM $ 300



Authorized User

The Director of the USPTO is hereby authorized to charge indicated fees and credit any overpayment as follows:

Charge any Additional Fees required under 37 C.F.R. Section 1.21 (Miscellaneous fees and charges)

File Listing:

cument D o cument Description File Name File Size(Bytes)/ Multi PagesNumber Message Digest Part /.zip (if appl.)

30395

1 Fee Worksheet (PTO-875) fee-info.pdf no 24dcaSOf2177504634c6901126a1129c4603

188ff

Warnings:

Information:

Total Files Size (in bytes): 30395






Application Number: 1123,1353


Title of Invention: 'ADAPTIVE MU LTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATIONDETECTION AND SURVEILLANCE SYSTEMS


Filer: Steven M. Colby/Melinda Tompkins




Sub-Total inDescription Fee Code Quantity Amount

USD($)

Basic Filing:

Pages:

Claims:


Petition:

Patent-Appea Is-and-Interference:


Publ. Fee- early, voluntary, or normal 1504 1 300 300

Extension-of-Time:


USD($)

Miscellaneous:


entitled "Robust Perceptual Color Identification," invented by K. Goh, E. Y. Chang and

Y. F Wang, which is expressly incorporated by reference in its entirety into this

application through this reference. This patent application addresses a problem of .

camera-based sensors perceiving an article of clothing as having a slightly different color

when viewed from. different angles or under different lighting conditions. The patent

application proposes the representing color:of an article of clothing using a "robust

perceptual color".

[0078] Data from different modalities may be fused by the Knowledge Services for

classification and identification purposes without suffering the "curse of dimensionality

using techniques taught in commonly assigned co-pending U. S. Patent Application

Serial No. 11/129,090, filed May 13, 2005, entitled, Multimodal High-Dimensional Data

Fusion for Classification and Identification, invented by E. Y. Chang, which is expressly

incorporated herein in its entirety by this reference. Data may be incrementally added to

a classification and identification process by the Knowledge Services using techniquesChange(s) applied i 1/zo230932zto document, taught by commonly assigned co-pending U. S. Patent Application Serial No., filed

/A. .M., September 19, 2005, entitled, Incremental Data Fusion and Decision Making, invented by+/z/zo 1i Yuan-Fang Wang, which is expresslyincorporated herein in its entirety by this reference.

10079] While the invention has been described with reference to various illustrative

features, aspects and embodiments, it will be appreciated that the invention is susceptible

of various modifications and other embodiments, other than those specifically shown and

described. The invention is therefore to be broadly construed as including all such

alternative variations, modifications and other embodiments within the spirit and scope as

hereinafter claimed.

26


Filing Date 2005-09-19IN FORMATION DISCLOSURE First Named Inventor Ken Prayoon Cheng

STATEMENT BY APPLICANT ArtUnit 2625( Not for submission under 37 CFR 1.99)



IJG/ 15 20060221184 Al 2006-10-05 Vallone et al.

/JG/ 16 20070146484 Al 2007-06-28 Horton et al.

/JG/ 17 20050132414 Al 2005-06-16 Bentley et al.

a 1 ied 19989 94. Al .009 4084 Ivanov, et al.ang azooso 3099 o6/zoo8

.J.F//20/J/ 19 20050052532 Al 2005-03-10 Elooz et al.

/JG/ 20 20080079554 Al 2008-04-03 Boice

/JG/ 21 20060017807 Al 2006-01-26 Lee et al.

/JG: 22 20040136574 Al 2004-07-15 Kozakaya et al.

23 20030169908 Al 2003-09-11 Kim et al.

/JG/ 24 20050100209 Al 2005-05-12 Lewis et al.

/JG/ 25 20040022442 Al 2004-02-05 Kim et al.




APPLICATION NO. ISSUE DAE PATENT NO. A'IORNEY DOCKET NO. CONFIRMATION NO.

11/231,353 06/07/2011 7956890 5330.07 (SMC) 4531

23308 7590 05/18/2011


ISSUE NOTIFICATION

The projected patent number and issue date are specified above.

Determination of Patent Term Adjustment under 35 U.S.C. 154 (b)(application filed on or after May 29, 2000)

The Patent Term Adjustment is 1517 day(s). Any patent to issue from the above-identified application willinclude an indication of the adjustment on the front page.

If a Continued Prosecution Application (CPA) was filed in the above-identified application; the filing date thatdetermines Patent Term Adjustment is the filing date of the most recent CPA.

Applicant will be able to obtain more detailed information by accessing the Patent Application InformationRetrieval (PAIR) WEB site (http://pair.uspto.gov).

Any questions regarding the Patent Term Extension or Adjustment determination should be directed to theOffice of Patent Legal Administration at (571)-272-7702. Questions relating to issue and publication feepayments should be directed. to the Application Assistance Unit (AAU) of the Office of Data Management(ODM) at (571)-272-4200.

APPLICANT(s) (Please see PAIR WEB site http://pair.uspto.gov for additional applicants):

Ken Prayoon Cheng, Saratoga, CA;Edward Y. Chang, Santa Barbara, CA;Yuan-Fang Wang, Goleta, CA;

IR103 (Rev. 10/09)


U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCEUnder the Paperwork Reduction Act of 1995. no persons are required to respond to a collection of information unless it displays a valid OMB control number.

PATENT APPLICATION FEE DETERMINATION RECORD Application or Docket NumberSubstitute for Form PTO-875 11231353

APPLICATION AS FILED - PART I(Column 1) (Column 2)

FOR NUMBER FILED NUMBER EXTRABASIC FEE(37 CFR 1.16(a), (b), or (c))SEARCH FEE(37 CFR 1.16(k). (i). or (m))EXAMINATION FEE(37 CFR 1.16(o), (p), or (q))TOTAL CLAIMS 20(37 CFR 1.16(i)) minus 20 =INDEPENDENT CLAIMS 3(37 CFR 1.16(h)) minus 3 =

If the specification and drawings exceed 100APPLICATION SIZE sheets of paper, the application size fee due isFEE 5250 ($125 for small entity) for each additional(37 CFR 1.16(s)) 50 sheets or fraction thereof. See

35 U.S.C. 41(a)(1)(G) and 37 CFR

MULTIPLE DEPENDENT CLAIM PRESENT (37 CFR 1.160))

If the difference in column 1 is less than zero, enter "0" in column 2.

APPLICATION AS AMENDED - PART II

(Column 1) (Column 2) (Column 3)CLAIMS HIGHEST

REMAINING NUMBER PRESENTSI AFTER PREVIOUSLY EXTRAZ AMENDMENT PAID FORW Total

(37 CFR 1.16(I)) Minus

Z Independentw (37 CFR1.16(h)) Minus

< Application Size Fee (37 CFR 1.16(s))

FIRST PRESENTATION OF MULTIPLE DEPENDENT CLAIM (37 CFR 1.16())

(Column 1) (Column 2) (Column 3)

m

zwgo

zwQ

SMALL ENTITY

RATE ($)

X$ 25=

X$100=

NIA

TOTAL

SMALL E

RATE ($)

X =

x =

TOTALADD'T FEE

RATE (5)

X =

X =

N/A

TOTALADD'T FEE

FEE (S)

150

250

100

500

NTITY

ADDI-TIONALFEE (S)

ADDI-TIONALFEE (5)

OR

OR

OR

OR

OR

OR

OR

SOR

OR

OR

OR

OR

OTHER THANSMALL ENTITY

RATE (5)

X$50=

X$200=

TOTAL

FEE (S)

OTHER THANSMALL ENTITY

ADDI-RATE (5) TIONAL

FEE (5)

X =

X =

N/A

TOTALADD'T FEE

RATE (5)

X =

X =

N/A

TOTALADD'T FEE

ADDI-TIONALFEE (S)

If the entry in column 1 is less than the entry in column 2, write *0' in column 3.If the "Highest Number Previously Paid For' IN THIS SPACE is less than 20, enter "20'.If the "Highest Number Previously Paid For IN THIS SPACE is less than 3, enter'3'.The "Highest Number Previously Paid For" (Total or Independent) is the highest number found in the appropriate box in column 1.

This collection of information is required by 37 CFR 1.16. The information is required to obtain or retain a benefit by the public which is to file (and by theUSPTO to process) an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.14. This collection is estimated to take 12 minutes to complete,including gathering, preparing, and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Any commentson the amount of time you require to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S. Patenand Trademark Office, U.S. Department of Commerce, P.O. Box 1450, Alexandria, VA 22313-1450. DO NOT SEND FEES OR COMPLETED FORMS TO THISADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450, Alexandria, VA 22313-1450.


CLAIMS HIGHESTREMAINING NUMBER PRESENT

AFTER PREVIOUSLY EXTRAAMENDMENT PAID FOR

Total Minus(37 CFR 1.16(i)) Minus =Independent M(37 CFR 1.16(h)) inus =

Application Size Fee (37 CFR 1.16(s))FIRST PRESENTATION OF MULTIPLE DEPENDENT CLAIM (37 CFR 1.16(j))

I

I

-- I



PATENT APPLICATION FEE DETERMINATION RECORD Application or Docket Number Filing Date

Substitute for Form PTO-875 11/231,353 09/19/2005 O To be Mailed

APPLICATION AS FILED - PART I. OTHER THAN(Column 1) (Column 2) SMALL ENTITY I OR SMALL ENTITY

NUMBER FILED

N/A

N/A

N/A

minus 20 =

minus 3 =

NUMBER EXTRA

N/A

N/A

N/A

If the specification and drawings exceed 100sheets of paper, the application size fee dueis $250 ($125 for small entity) for eachadditional 50 sheets or fraction thereof. See35 U.S.C. 41(a)(1)(G) and 37 CFR 1.16(s).

O MULTIPLE DEPENDENT CLAIM PRESENT (37 CFR 1.16(j))

* If the difference in column 1 is less than zero, enter "0" in column 2.

APPLICATION AS AMENDED - PART II


CLAIMS HIGHEST

02/01/2011 REMAINING NUMBER PRESENT02/01/201 AFTER PREVIOUSLY EXTRA

Z AMENDMENT PAID FORS Total 37 CFR 2 Minus =* 20 = 0

SIndependent Minus 3 =Z 37nCFR 1. * 2 Minus -3 = 0

U E Application Size Fee (37 CFR 1.16(s))

E FIRST PRESENTATION OF MULTIPLE DEPENDENT CLAIM (37 CFR 1.16(j))


CLAIMS HIGHESTREMAINING NUMBER PRESENT

AFTER PREVIOUSLY EXTRAAMENDMENT PAID FOR

Z Total (37 CFR Minus I2 independent Minus "**-( 37.CFR 1.16(h)} -

Z 1 l Application Size Fee (37 CFR 1.16(s))

Q E FIRST PRESENTATION OF MULTIPLE DEPENDENT CLAIM (37 CFR 1.16(j))

* If the entry in column 1 is less than the entry in column 2, write "0" in column 3.

** If the "Highest Number Previously Paid For" IN THIS SPACE is less than 20, enter "20".

*"* If the "Highest Number Previously Paid For" IN THIS SPACE is less than 3, enter "3".

RATE ($)

TOTAL

FEE ($)

SMALL ENTITY

RATE ($)

X $26 =

X $110=

TOTALADD'LFEE

ADD'LFEE

ADDITIONALFEE ($)

0

0

0

N/A

TOTAL

FEE ($)

OTHER THAN

OR SMALL ENTITY

RATE ($)

OR $x =

OR X $ =

OR

TOTALOR ADD'L

FEE

RATE ($)

X$ =

X$ =

TOTALOR ADD'L

FEE

ADDITIONALFEE ($)

ADDITIONALFEE ($)

Legal Instrument Examiner:/SHARON WEST/

The "Highest Number Previously Paid For" (Total or Independent) is the highest number found in the appropriate box in column 1.

This collection of information is required by 37 CFR 1.16. The information is required to obtain or retain a benefit by the public which is to file (and by the USPTO toprocess) an application. Confidentiality is governed by 35 U.S.C. 122 and 37CFR 1.14. This collection is estimated to take'12 minutes to complete, including gathering,preparing, and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Any comments on the amount of time yourequire to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S. Patent and Trademark Office, U.S.Department of Commerce, P.O. Box 1450, Alexandria, VA 22313-1450. DO NOT SEND FEES OR COMPLETED FORMS TO THISADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450, Alexandria, VA 22313-1450.


O BASIC FEE(37 CFR 1.16a), (b , or (c))

O SEARCH FEE(37 CFR 1.16(k)l (i, or (m))

O EXAMINATION FEE(37 CFR 1.16(o), (p), or (q))

TOTAL CLAIMS(37 CFR 1.16i))

INDEPENDENT CLAIMS(37 CFR 1.16h))

OAPPLICATION SIZE FEE(37 CFR 1.16(s))

MMMM9 0

MOMI r-,RATE ($)

X $

X $

X $

X $

1

Table of Contents i

MicroPatent Patent Index - an enhanced INPADOC database

MPI Family Report (Family Bibliographic and Legal Status)

In the MPI Family report, all publication stages are collapsed into a single record, based on identical application data. The bibliographic information displayed in the collapsed record is taken from the latest publication.

Report Created Date: 2011-08-05

Name of Report:

Number of Families: 1

Comments:

Table of Contents

1. US7956890B2 20110607 PROXIMEX CORP US Adaptive multi-modal integrated biometric identification detection and surveillance systems ......................... 2

http://www.micropat.com

Family Bibliographic and Legal Status 1


Family1

4 records in the family, collapsed to 2 records.

WO2006034135A3 20060713WO2006034135A2 20060330

(ENG) ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEM

Assignee: PROXIMEX US

Inventor(s): CHENG KEN PRAYOON US ; CHANG EDWARD Y US ; WANG YUAN FANG US

Application No: US 2005033378 W

Filing Date: 20050919

Issue/Publication Date: 20060713

Abstract: (ENG) <emi file="US2005033378_13072006_pf_fp.g4" he="199MM" wi="271MM"/><p>A surveillance system is provided that includes at least one sensor disposed in a security area of a surveillance region to sense an occurrence of a potential security breach event; a plurality of cameras is disposed in the surveillance region; at least one camera of the plurality has a view of the security area and can be configured to automatically gather biometric information concerning at least one subject person in the vicinity of the security area in response to the sensing of a potential security breach event; one or more other of the plurality of cameras can be configured to search for the at least one subject person; a processing system is programmed to produce a subject dossier corresponding to the at least one subject person to match biometric information of one or more persons captured by one or more of the other cameras with corresponding biometric information in the subject dossier. </p>

Priority Data: US 61099804 20040917 P;

Related Application(s): 20060713 200628 3 R4

IPC (International Class): G06K00900; H04N00947; G06K00964; H04N00718; H04B01700

ECLA (European Class): G06K00962F3M; G08B013196; H04N00718C

Designated Countries:----Designated States: AE AG AL AM AT AU AZ BA BB BG BR BW BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE EG ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KM KP KR KZ LC LK LR LS LT LU LV LY MA MD MG MK MN MW MX MZ NA NG NI NO NZ OM PG PH PL PT RO RU SC SD SE SG SK SL SM SY TJ TM TN TR TT TZ UA UG US UZ VC VN YU ZA ZM ZW ----Regional Treaties: AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LT LU LV MC NL PL PT RO SE SI SK TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG BW GH GM KE LS MW MZ NA SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM

Publication Language: ENG

Agent(s): DURANT, Stephen, C. et al. Morrison & Foerster LLP, 425 Market Street, San Francisco, Ca 94105-2482 US

Legal Status: There is no Legal Status information available for this patent

Family Bibliographic and Legal Status 2


US7956890B2 20110607US2006093190A1 20060504

(ENG) Adaptive multi-modal integrated biometric identification detection and surveillance systems

Assignee: PROXIMEX CORP US

Inventor(s): CHENG KEN P US ; CHANG EDWARD Y US ; WANG YUAN-FANG US

Application No: US 23135305 A

Filing Date: 20050919

Issue/Publication Date: 20110607

Abstract: (ENG) A surveillance system is provided that includes at least one sensor disposed in a security area of a surveillance region to sense an occurrence of a potential security breach event; a plurality of cameras is disposed in the surveillance region; at least one camera of the plurality has a view of the security area and can be configured to automatically gather biometric information concerning at least one subject person in the vicinity of the security area in response to the sensing of a potential security breach event; one or more other of the plurality of cameras can be configured to search for the at least one subject person; a processing system is programmed to produce a subject dossier corresponding to the at least one subject person to match biometric information of one or more persons captured by one or more of the other cameras with corresponding biometric information in the subject dossier.

Priority Data: US 23135305 20050919 A Y; US 61099804 20040917 P Y;

Related Application(s): 11/231353 20050919 20060093190 US; 60/610998 20040917 US

IPC (International Class): H04N00947; G06K00900

ECLA (European Class): G06K00900V4

US Class: 348143; 382115; 382119; 382118

Publication Language: ENG

Filing Language: ENG

Agent(s): Peters Verny, LLP

Examiner Primary: Grant, II, Jerome

Assignments Reported to USPTO:Reel/Frame: 17473/0506 Date Signed: 20051206 Date Recorded: 20060113 Assignee: PROXIMEX 6 RESULTS WAY CUPERTINO CALIFORNIA 95014

Assignor: CHENG, KEN P.; CHANG, EDWARD Y.; WANG, YUAN-FANGCHENG, KEN P.; CHANG, EDWARD Y.; WANG, YUAN-FANG

Corres. Addr: STEPHEN C. DURANT MORRISON & FOERSTERLLP 425 MARKET STREET SAN FRANCISCO, CALIFORNIA 94105-2482

Brief: ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS).

Legal Status:Date +/- Code Description20060113 () AS New owner name: PROXIMEX, CALIFORNIA; : ASSIGNMENT

OF ASSIGNORS INTEREST;ASSIGNORS:CHENG, KEN P.;CHANG, EDWARD Y.;WANG, YUAN-FANG;SIGNING DATES FROM 20051206 TO 20051215;REEL/FRAME:017473/0506;

USPTO Maintenance Report

Patent Bibliographic Data 08/05/2011 12:33 PM

Patent Number: 7956890 Application

Number: 11231353

Issue Date: 06/07/2011 Filing Date: 09/19/2005

Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND

Status: 4th year fee window opens: 06/07/2014 Entity: Large

Window Opens: 06/07/2014 Surcharge

Date: 12/09/2014 Expiration: N/A

Fee Amt Due: Window not open

Surchg Amt Due:

Window not open

Total Amt Due:

Window not open

Fee Code: 1551 MAINTENANCE FEE DUE AT 3.5 YEARS

Surcharge Fee Code:

Most recent events (up to 7):

No Maintenance History Found --- End of Maintenance History ---

Address for fee purposes:

PETERS VERNY , L.L.P. 425 SHERMAN AVENUE SUITE 230 PALO ALTO CA 94306

7956890_11-231353_570

Documents

gau petition

gau case

final original final

o claims

gau transfer inquiry

patent issue date

nonfinal action petition

rej date