Application NO: Filed: Issued

FILE HISTORY US 7,956,890
PATENT: INVENTORS:
7,956,890 Cheng, Ken P. Chang, Edward Y. Wang, Yuan-Fang
TITLE:
Adaptive multi-modal integrated biometric identification detection and surveillance systems
APPLICATION US2005231353A NO: FILED: ISSUED: COMPILED:
19 SEP 2005 07 JUN 2011 05 AUG 2011
7,956,890
ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEMS 5330.07 (SMC) Transaction History
Date 9/19/2005 9/28/2005 10/4/2005 10/11/2005 1/13/2006 Transaction Description Initial Exam Team nn IFW Scan & PACR Auto Security Review Cleared by OIPE CSR Notice Mailed--Application Incomplete--Filing Date Assigned Payment of additional filing fee/Preexam
A statement by one or more inventors satisfying the requirement under 35 USC 1/13/2006 115, Oath of the.Applic 1/13/2006 Applicant has submitted new drawings to correct Corrected Papers problems 1/20/2006 Application Dispatched from OIPE 1/23/2006 Application Is Now Complete 6/16/2006 IFW TSS Processing by Tech Center Complete 6/16/2006 Case Docketed to Examiner in GAU 10/16/2006 Information Disclosure Statement considered 10/16/2006 Reference capture on IDS 10/16/2006 Information Disclosure Statement (IDS) Filed 10/16/2006 Information Disclosure Statement (IDS) Filed 2/22/2007 Petition Entered 5/18/2007 Correspondence Address Change 5/18/2007 'Change in Power of Attorney (May Include Associate POA) 5/23/2007 1/15/2008 1/15/2008 3/27/2008 5/4/2008 5/4/2008 6/12/2008 7/23/2008 10/19/2009 12/3/2009 12/3/2009 12/3/2009 2/22/2010 Mail-Petition Decision - Dismissed Case Docketed to Examiner in GAU Case Docketed to Examiner in GAU Transfer Inquiry to GAU Transfer Inquiry to GAU Transfer Inquiry to GAU Transfer Inquiry to GAU Case Docketed to Examiner in GAU Petition Entered Mail-Petition Decision - Granted Petition Decision - Granted Correspondence Address Change Case Docketed to Examiner in GAU
2/23/2010 4/7/2010 4/8/2010 5/12/2010 5/13/2010 12/21/2010 12/27/2010 2/1/2011 2/1/2011 2/7/2011 2/10/2011 3/24/2011 3/24/2011 3/28/2011
Case Docketed to Examiner in GAU Case Docketed to Examiner in GAU Case Docketed to Examiner in GAU Non-Final Rejection Mail Non-Final Rejection Aband. for Failure to Respond to O. A. Mail Abandonment for Failure to Respond to Office Action Response after Non-Final Action Petition Entered Correspondence Address Change Change in Power of Attorney (May Include Associate POA) Mail-Petition to Revive Application - Granted Petition to Revive Application - Granted Information Disclosure Statement considered Information Disclosure Statement considered Reference capture on IDS Electronic Information Disclosure Statement Information Disclosure Statement (IDS) Filed Information Disclosure Statement (IDS) Filed Date Forwarded to Examiner Mail Notice of Rescinded Abandonment Notice of Rescinded Abandonment in TCs Document Verification Notice of Allowance Data Verification Completed Mail Notice of Allowance Statement Filed Indicating a Loss of Entitlement to Small Entity Status Issue Fee Payment Verified Issue Fee Payment Received Application Is Considered Ready for Issue Dispatch-to FDC Dispatch to FDC Issue Notification Mailed Recordation of Patent Grant Mailed Patent Issue Date Used in PTA Calculation
3/28/2011
3/28/2011
3/28/2011
3/28/2011 3/28/2011
4/6/2011
4/6/2011
4/6/2011
4/15/2011
4/15/2011
4/18/2011 4/21/2011 4/21/2011
4/21/2011
4/25/2011 5/10/2011 5/10/2011 5/18/2011 6/7/2011 6/7/2011
Application/Control No.
Applicant(s)/Patent Under Reexamination CHENG ET AL.
Issue Classification
SIlII I
11231353
Examiner
Jerome Grant II
SUBCLASS CLAIMED
H G 0 0 4 6 N K
Art Unit
2625
INTERNATIONAL CLASSIFICATION
NON-CLAIMED
9 / 47(2006.0) 9/00 (200601.01)
ORIGINAL.
CLASS
348 143
CROSS REFERENCE(S)
CLASS
382 115
SUBCLASS (ONE SUBCLASS PER BLOCK)

119 118
O
Final
Claims renumbered in the same order as presented by applicant

Original 1 2 3 4 5 6 7 8 9 10
11 12 13
O
Final
CPA
Original Final
T.D.
Original Final
R.1.47
Original Final Original
Final
-
Original 17 18
Final
Original
Final
Original
19 20
1 2
14 15
16
NONE Total Claims Allowed: (Assistant Examiner) /Jerome Grant II/ Primary Examiner.Art Unit 2625 (Primary Examiner)
U.S. Paent and Trademark Office
2 (Date) 4-11-11 (Date) O.G. Print Claim(s) 1

Part of Paper
O.G. Print Figure 6 and 7

No. 20110411
ApplicationlControl No.
Applicant(s)/Patent Under Reexamination
Index of Claims
11231353
CHENG ET AL.
||I|
"
I IIIIIIExaminer
Jerome Grant II
Art Unit
2625
Rejected
Cancelled
Non-Elected
Appeal
Allowed
Restricted
Interference
O
CPA
O
O
T.D.
Objected
O
R.1.47
O Claims renumbered in the same order as presented by applicant
CLAIM
Final Original 1 2 3 4 5 6 8 9 10 11 12 13 14 15 16 17 18 19 20 'OB OBJ REJ 05/10/2010 REJ
DATE
U.S. Patent and Trademark Office
Partof PaperNo.: 20100510
Applicant(s)/Patent Under Reexamination
Index of Claims
11231353
CHENG ET AL.
||| || |||
Examiner
Art Unit
2625
Jerome Grant II
Claims renumbered in the same order as presented by applicant
O DATE
CPA
T.D.
R.1.47
CLAIM
Final Original 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 'OB OBJ REJ -= =
-
05/10/2010 04/11/2011 REJ
Part of Paper No. : 20110411
Application/Control No..
Applicant(s)/Patent Under
Reexamination
Search Notes
11231353
Examiner
CHENG ET AL.
Art Unit
Jerome Grant II
2625
SEARCHED Class 713

_ _320_'
Subclass 186 202 201 200 506 143 161 169 103 4 103 209 276 277 289 291 293 294 295 305
115
Date 5-10 '__ '
Examiner JG ' ' ' ' '

.'
340 358
_ _147 _'
' 707 382
' ' '
' ' "
' 5-10
_ _282_'
JG' '
!
107 190
_ _116 _'
345 348
629 143
151
' ' ' ' ' ' ' " ' '
709 '
155 161 169 227 215 203 315
'
Part
of Paper
No.:
20100510
SEARCH NOTES
Search Notes East Notes Date 5-10 Examiner JG
INTERFERENCE SEARCH Class Subclass Date Examiner
Part of Paper No.: 20100510
SEARCHED
Class
119 S128
Subclass
Date
Examiner
SEARCH NOTES
Search Notes East Notes East Notes -Date 5-10 4-11 JG JG Examiner
INTERFERENCE SEARCH
Class 382
'
__ _ __ __ __
Subclass 118
115 119
Date 4-11
Examiner JG
'
348
143
Part of.Paper
No.:'
201104111
Applicant(s)/Patent Under
Reexamination
Search Notes
11231353
CHENG ET AL.
I I|IIExaminer IIII
Jerome Grant II
Art Unit
2625
SEARCHED
Class 713 186 202 320 201 200 506 143 147 161 169 103
4
Subclass
Date 5-10
Examiner JG ' ' ' ' ' ' '
340 358 ' 707 382
'
9
' ' '

_ _115_'
103 209 276 277 289 291 293 294 295 305 282 107 190 116 629 143 151 155 161 169 227 215 203 315 118
' ' ' '

"
5-10
JG'
' 345 348
709 ' ' 382
'
"
4-11-
'JG
Part of Paper No.: 20110411
EAST History Search
EAST Search History EAST Search History (Prior Art) Ref 'Search :SHits Query :DBs :Plurals Time Stamp Default . 'ii9.... 'B'i. . . . . . . . . . . . . . . . 2i~;.... ..~......... ~ Operator .. (OFF ?2010/04119
;21:43
........ i 't'i
.. i.. .. .. .... ..
35 1L2
OR L i 24 :(security or secure or surveillance or survaillance or confidenital or secret or reconnesance or USiMUB; USPAT recon$14 or dandesti$6 or watch$4) near7 (camera or sensor or photo adj (conduct$4 or or sensor or read$4 or camera or diode) or codcmd or scan$4 or snap adj shot) with (person or USOCR; FPRS; or object child or adult or threat or dclient or subject or people or terrorist or man or woman or EPO; JFPO; DERWENT; individual or inhabitant)near5 (biometri$6 or hair or eye or contour or contor or shape or TDB :signature or physique or physical or profile or hair adj line or skin adj (tone or color or hue) or IlBM skin or dothes or shirt or pants or dress or skirt or gown or garment$4) same (profile or map or history or memory or register or buffer or library or table or dossier or list or chart or registrar) ; :::. . . .: . 'and @ad<"20050921" . OR (security or secure or surveillance or survaillance or confidenital or secret or reconnesance or US-FGPUJB; USPAT; sensor or photo adj (conduct$4 or recon$14 or dandesti$6 or watch$4) near7 (camera or sensor or read$4 or camera or diode) or ccd or cmd or scan$4 or snap adj shot) with (person or USOCR; FPRS; or object :child or adult or threat or dclient or subject or people or terrorist or man or woman or EPO; JPO; individual or inhabitant)near5 (biometri$6 or hair or eye or contour or face or facial or contor or DERWENT; :shape or signature or physique or physical or profile or hair adj line or skin adj (tone or color or IBMTDB : or skin or dothes or shirt or pants or dress or skirt or gown or garment$4) same (profile or hue) : or history or memory or register or buffer or library or table or dossier or list or chart or map registrar) and @ad<"20050921" (security or secure or surveillance or survaillance or confidenital or secret or reconnesance or recon$14 or dandesti$6 or watch$4) with (camera or sensor or photo adj (conduct$4 or sensor or read$4 or camera or diode) or cod or cmd or scan$4 or snap adj shot) with (person or child or adult or threat or dient or object or subject or people or terrorist or man or woman or or contor individual or inhabitant)near5 (biometri$6 or hair or eye or contour or face or facidal or shape or signature or physique or physical or profile or hair adj line or skin adj (tone or color or ;hue) or skin or dothes or shirt or pants or dress or skirt or gown or garment$4) same (profile or :map or history or memory or register or buffer or library or table or dossier or list or chart or :registrar) and (358/143,144,161,169,400,500,484,483.cds.or 707/103,4.cds. or or 382/103,209,276,277,289,291,293,294,295,305,282,115,107,190,116.cds. 345/629.ccls. or? t348/143,151,155,161,169.ccls. or 709/227,215,203,315.cds. or 713/186,202,320,201,200.ccls. not :or 340506.cds.) not L1 L2
............ .... .............. ...................... ..... ..... .... .... .. ............. .. ............................................................. ........... ... . ............ .. .. . .
OFF
i
12010/04/19
:21:43
L.3 42
OR USFPGPUB; USPAT; USOCR; FPRS;c EPO; JPO; DERWENT; IBMTDB

-
OFF
2010/04/19 21:43
4119110 9:45:24 PM
EAST Search History
EAST Search History EAST Search History (Prior Art)
.......I ,.... .... r ............ ii .......... .......................... . . . Hits Search Query e::............... DBs e a i... Plurals Time .. a... : ..... urais...me Stamp ..... Default Ref
Operator
L1
(security or secure or . surveillance or survaillance or confienital or secret or reconnesance or reconessan$5 or clandesti$6) near5 (camera or sensor or photo adj (conduct$4 or sensor or read$4 or ;camera or diode) or ccd or cmd or scan$4 or snap adj shot) with (person or child or adult or threat or client or object or subject or people or terrorist or man or woman or individual or inhabitant) near5 (biometri$6 or hairP or eye or contour or contor or shape or signature or profile or hair adj line or skin adj (tone
nr mv'in~rr
USPGPUB USPAT; USOCR; FPRS; EPO; JPO; DERWENT; I BMTDB
OR
OFF
2010/04/19 :11:53
N hipr
cin nr
clothes or shirt or pants or dress or skirt or gown or garment$4) same (progile or map or history or memory or register or buffer or library) and @ad<"20050921"
I? 2
15
i i
(security or secure or
UJS-PGPLJB:
OR
---------------------- ------------------------
OFF ...
.. ..................... ....... 2010/04/19

11:57
-surveillance
or
USPAT
survaillance or confidenital or secret or =reconnesance or recon $14 or clandesti$6) near5 (camera or sensor or photo adj (conduct$4 or sensor or read$4 or camera or diode) or ccd or cmd or scan$4 or snap adj shot) with (person or
c III
ihil
USOCR; FPRS; EPO; JPO; DERWENT; IBMTDB

i :; :; :;
UorI
IUUaUIior itreorLUI
thh,1 -
tn
client or object or subject or people or terrorist or man or woman or
i i
z
file:///CI/Documents%20and%20Settings/jgrant/My%20Docu...31353/EASTSearchHistory.
11231353_AccessibleVersion.htm (1 of 3)4/19/10 1:16:29 PM
EAST Search History
.......... .. ...
L3
individual or inhabitant) Snear5 (biometri$6 or hair or eye or contour or contor or shape or :signature or profile or hair adj line or skin adj (tone or color or hue) or skin or clothes or shirt or pants or dress or skirt or gown or garment$4) same , (profile or map or history or memory or register or buffer or library or table or dossier or chart or registrar) and @ad<"20050921 ...Si r c;e r............TiUS- PGPt JRU 24 (security or secure or surveillance or USPAT; USOCR; FPRS; survaillance or EPO; JPO; confidenital or secret or DERWENT; reconnesance or recon IBM TDB $14 or clandesti$6 or watch$4) near7 (camera or sensor or photo adj (conduct$4 or sensor or read$4 or camera or diode) or ccd or cmd or scan$4 or snap adj shot) with (person or child or adult or threat or client or object or subject or people or terrorist or man or woman or individual or inhabitant)near5 (biometri $6 or hair or eye or contour or contor or shape or signature or physique or physical or profile or hair adj line or skin adj (tone or color or hue) or skin or clothes or shirt or pants or dress or skirt or gown or garment $4) same (profile or map or history or memory or :register or buffer or library or table or dossier or list or chart or registrar) and @ad<"20050921"
.......................................... .....................
OFF IC
...............................-...... ~.
2010/04/19 12:05
:; :; i :; i :; :; E :;
file:///CI/Docunents%20and%20Settings/jgrant/My%20Docu...31353/EASTSearchHistory. 11231353_AccessibleVersion.htm (2 of 3)4/19/10 1:16:29 PM
EAST Search History
'""
............ .. . . . . .. . . . . . . .. . . . . .. . . . . .. . . . . . .. .. . . ... .. .. . . .. . . .
L4
35
:; i i :;
't :; :: :; :: i i i i
:, :; i : :;
i i i i
_,
. ........
...... ,
1 iUS-Ffl:;.PIiR(security or secure or surveillance or USPAT; survaillance or USOCR; FPRS; EPO; JPO; confidenital or secret or reconnesance or recon DERWENT; $14 or clandesti$6 or I BMTDB watch$4) near7 (camera or sensor or photo adj (conduct$4 or sensor or !read$4 or camera or diode) or ccd or cmd or scan$4 or snap adj shot) with (person or child or adult or threat or client or object or subject or people or terrorist or man or woman or individual or inhabitant)near5 (biometri $6 or hair or eye or contour or face or facial or contor or shape or signature or physique or physical or profile or hair adj line or skin adj (tone or color or hue) or skin or clothes or shirt or pants or dress or skirt or gown or garment$4) same (profile or map or history or memory or register or buffer or library or table or dossier or list or chart or registrar) and @ad<"20050921" ....................... ...........
................................ :
OR ......................
;v
......Zvi
i
FF......... ,2f01/04/19............:
w ;i
11Z:U5
r) "na
vii
't
:; :;
i i
................................... .
4/19/10 1:16:09 PM
file:///CI/Docwnents%20and%20Setings/jgrant/My%20Docu...31353/EASTSearchHistory. 11231353_AccessibleVersion.htm (3 of 3)4/19/10 1:16:29 PM
EAST Search History
EAST Search History EAST Search History (Prior Art) Ref #

..........
Hits
........ ...........
Search Query
........... ................................................ ::
DBs
.......... ..... ....
~
............... .
Default
Operator
............. ............... ...
PluraIs Time Stamp

............. .. .........
L1 :i
22 ::
.... . .......... .r.or................ . ..... . ....... . . ...e.. . . . "u . .. . . i6... .............

S1 32808 (recipient or user or :operator or client or personell or personal or person) near4 (setting address or parameers or meta or data) same (server or exchange or saff or forward or modem or packet$4) same (display or monitor or screen or Icd or window or view$4) and (protocol or protocal or http or smtp or xml or remote adj iprocedure) and USPGPUB; USPAT; USOCR; FPRS; EPO; JPO; DERWENT; I BM_TDB OR OFF
(security or secure or :2'2 surveillance or :survaillance or confiential or secret or reconnesaince or recon$14 or dlandest$6 or watch$4) near7 (camera or sensor or photo adj (conduct$4 or sensor or read$4 or camera or diode or ccd or cmd or scan$4 or snap adj shot)) with (person or child or adult or threat or client or object or subject or people or terrorrist or man or woman or , individiual or inhabitant) near5 (biometri$6 or hair or eye or contour or face' or facial or contor or contour or shape or signature or phsique or phusical or profile or hair :adj line or skin adj (tone or color or hue or tint) or :clothes or skin or shirt or pants or dress or skirt or gown or garment$4) same (profile or map or history or memory or register or buffer or library or table or dossier or list or chart or registrar) and @ad< 20050921'"'
US-PGPUB,; USPAT; USOCR; :FPRS; EPO; JPO; DERWENT; I;IBMTDB
OR
OFF
2011/04/11 12 3 12:38
.....
2010/12/11 08:53
.....
file:///CI/Documents%20and%20Settings/jgrant/My%20Doc...1353/EASTSearchHistory.11231353_AccessibleVersion.htm
(1 of 26)4/11/1112:42:38 PM
EAST Search History
@ad< "20071213" S2 OR US-PGPUB; (recipient or user or USPAT; USOCR; or operator or client FPRS; EPO; personell or personal or JPO; person) near4 (setting DERWENT; address or parameers or IBM _TDB meta or data) same (server or exchange or'. saff or forward or modem or packet$4) same (display or monitor or screen or Icd or window or view$4) and (protocol or protocal or http or smtp or xml or remote adj procedure) and @ad<"20071213" and (scan$4 or read$4 or copier) near4 (data or email or electronic adj mail or message or copy :.. '.. .. .......or data) .....i... ..... .. .................... ............. . ... .. ... . .............. ..... ........... ............... . ..... . .. ... ..... ... . 16598 ;5193 (recipient or user or. operator or client or personell or personal or person) near4 (setting or address or parameers or meta or data) with (server or exchange or saff or forward or modem or packet$4) with (display or monitor or screen or Icd or :window or view$4) and (protocol or protocal or http or smtp or xml or remote adj procedure) OR US-PGPUB; =USPAT; USOCR; FPRS; EPO; ,JPO; DERWENT; ;IBMTDB OFF 2010/12/11 08:55
.. .... .... ..... .. . .... .... ....: ... ..... ..... .. .. . ........ .. . .. .............
S3
OFF
2010/12/11 :08:56
and @ad<"20071213" and

(scan$4 or read$4 or copier) near4 (data or email or electronic adj mail or message or copy ....... ... ... . .. or . ............ ... ...... !.. data)
..... . .... . ..... ..............
.... ....... . .. .....
.....
... .: ...... .................................... 08:58
S4 ......... 2562
2010/12/11............ OFF ...... .......................... US-PGPUB; ........... (recipient or user or ............... OR...
'USPAT; USOCR; operator or client or FPRS; EPO;I personell or personal or JPO; person) near4 (setting or DERWENT; address or parameers or meta or data) near9 I BMTDB (server or exchange or saff or forward or modem or packet$4) with (display or monitor or screen or Icd or window or view$4) and ;(server or http or smtp or xml or transmitter) near8
(protocol or protocal or
http or smtp or xml or
file:///Cl/Documents%20and%20Settings/jgrant/My%20Doc...1353/EASTSearchHistory.
11231353AccessibleVersion.htm (2 of 26)4/11/11 12:42:38PM
EAST Search History
. . . . .. . . . . . .. . . . . .. . . . .. :. . .. .. . . . . . .. . . . . . .. . . . . .. . .. . .. . . . . . . . . . . .. . . . . .. . . . . .. . . . . . . .. .. .. . .. . .. . . . . . .. . . . . . . . . . .. . . . . .. . . ..
remote adj procedure) and @ad<"20071213" and (scan$4 or read$4 or copier) near4 (data or email or electronic adj :mail or message or copy or data)
i i i i i i i i
.. ... ... . ............ ...... .........
OR..................................
i'
S6
2928
.......................................
........
:US-PGPUB; :(recipient or user or OR USPAT; USOCR; operator or client or FPRS; EPO; personell or personal or JPO; :person) near4 (setting or address$4 or indicat$4 or :DERWENT; select$4 or parameters or ;IBMTDB meta or data) near7 (server or exchange or saff or forward or modem or packet$4) with (display or monitor or screen or Icd or window or view$4) and (server or http or smtp or xml or transmitter) near4 (protocol or protocal or http or smtp or xml or remote adj procedure) and @ad<"20071213" and (scan$4 or read$4 or copier) near4 (data or email or electronic adj mail or message or copy or data)
.Y. ........ ............................................................ Y.....
OFF
2010/12/11 1 09:12
.1
::
:,
i i
i ..................... : ......................
............
2010/12/11 09:16
1437
. . .. . . . . .. . . . . .. . . ...
(recipient or user or operator or client or personell or personal or person) near4 (setting or address$4 or indicat$4 or select$4 or parameters or :meta or data) near4 (server or exchange or saff or forward or modem or packet$4) with (display or monitor or screen or Icd or window or view$4) and (server or http or smtp or xml or transmitter) near4 (protocol or protocal or http or smtp or xml or remote adj procedure) and @ad<"20071213" and (scan$4 or read$4 or copier) near4 ( server or saff or packet or modem or email or electronic adj (mail or message or copy or data))
US-PGPUB; OR USPAT; USOCR; FPRS; EPO; JPO; DERWENT; IBMTDB
;OFF
.. . . . ;... . .. . . . . .. . . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . .. . . . .. . . . . . . . . . .. . . . .. . . . . .. . . . . .. . . . . .. . . . . . .. . .
file:///Cl/Documents%20and%2OSettings/jgrant/My%2ODoc:..1353/EASTSearchHistory.11231353_AccessibleVersion.htm
(3 of 26)4/11/11 12:42:38PM
EAST Search History
:,.................
...................... ......................... :................ ................ ............. ......................... .........
; ;.............................
iS8
::Q96 v
S i
ii
'
or Ioperator or client personell or personal or :person) near4 (setting or address$4 or indicat$4 or select$4 or key$4 or input $5 or choosing or chosen) near4 (server or exchange or saff or forward or modem or packet$4) with (display or monitor or screen or Icd or window or view$4) and (server or http or smtp or xml or transmitter) near4 (protocol or protocal or http or smtp or xml or remote adj procedure) and @ad<"20071213" and (scan$4 or read$4 or copier) near4 ( server or saff or packet or modem or http or email or :electronic adj (mail or
i\ m . ,
or r (rrinient or ir ' ' ' .... .. .... .,
iiS-GP IRU -P(Pi
USPAT; USOCR; FPRS; EPO; JPO; DERWENT; IBM_TDB
..
...
OR
vi
:; :; :; :; :; :; :;
:;OFF
i i i i i
: 2010/12/11
!09:19
i i
:message or copy or data))

S9
.........
. . . ?. .... . . . .. . . . . . . . . .
. . ..
4359
(recipient or user or operator or client or personell or personal or person) near4 (setting or address$4 or indicat$4 or select$4 or key$4 or input $5 or choosing or chosen) near4 (server or exchange or saff or forward or modem or packet$4) and (display or monitor or :screen or Icd or window or view$4) near4 (image or output or picture or view or image or data) and (server or http or smtp or xml or transmitter) near4 (protocol or protocal or http or smtp or xml or remote adj procedure) and @ad<"20071213" and (scan$4 or read$4 or :copier) near4 ( server or saff or packet or modem :or http or email or electronic adj (mail or imessage or copy or data))
US-PGPUB; USPAT; USOCR; FPRS; EPO; :JPO; DERWENT; IBMTDB
OR
(i OFF.....
2010/12/11 09:22
i a
';
file:///CI/Documents%20and%20Settings/jgrant/My%20Doc...1353/EASTSearchHistory. 11231353_AccessibleVersion.htm (4 of 26)4/11/11 12:42:38 PM
EAST Search History
Si 0
. ...................
..
..
......................................
599
(recipient or user or operator or client or personell or personal or person) near4 (setting or address$4 or indicat$4 or select$4 or key$4 or input $5 or choosing or chosen) :near4 (server or exchange or saff or forward or modem or packet$4) and (display or monitor or :screen or Icd or window or view$4) near3 (image or output or picture or view or image or data) same (server or http or smtp or xml or transmitter) near4 ;(protocol or protocal or http or smtp or xml or remote adj procedure) and @ad<"20071213" and :(scan$4 or read$4 or :copier) near4 ( server or saff or packet or modem or http or email or :electronic adj (mail or
,OR ................... ;.......:;OFF......... ;2 10/12/1.1............ US-PGPUB; .;09:23 USPAT; USOCR; FPRS; EPO; i :JPO; DERWENT; IBMTDB
V I l 2'
.......................................
'
t C
i i i
.;
"
:, :; i :; :; i
'
i.
i :; i i' :; i .
message or copy or data)) .......... US-P PUB;........... 2................. 20080068642". pn.....

USPAT; USOCR; FPRS; EPO;
i i a
........................................
OR
,OFF
2011/02/28 06:37
JPo;
DERWENT; IBMTDB
..................... .... ...................................
1925
(print$4 or copier eor mfp

or record$4) same (material or content or page or sheet or original or file or image or picture or top or bottom) near5 (part or portion or section or region or line or paragraph) same (control $4 or correct$4 or adjust $4 or modification or :modifying or chang$4 or :altering) near3 (resolution or gradation or pitch)and
US-'PGPUB;
USPAT; USOCR; FPRS; EPO; JPO; DERWENT; IIBMTDB
OR
OFF
03/07 ::2011 /
07:40
@ad<,"20080317"
>. . . . . . . .. . . . . . . . . . .. . . . .
file:///CI/Documents%20and%20Settings/jgrant/My%20Doc...1353/EASTSearchHistory. 11231353 AccessibleVersion.htm (5of 26)4/11/11 12:42:38 PM
EAST Search History
................. ..... .,.......... ..
... .... ..................... :- ---------. ............ .. ..
iS13
1011
:i
S14 -
744
(print$4 or copier or mfp US-PGPUB; or record$4) same USPAT; USOCR; (material or content or FPRS; EPO; page or sheet or original JPO; or file or image or picture DERWENT; I BM TDB or top or bottom) near5 (part or portion or section or region or line or paragraph) near9 (control $4 or correct$4 or adjust $4 or modification or modifying or chang$4 or altering) near3 (resolution or gradation or pitch)and @ad< "20080317" . (....................... rint$4 or copier or mf............ :!(print$4 or copier or mfp US-PGPUB;
; vvi v .... ........................................
'IOR
;vi
.i]OFF
:; :, :, :; :, :, :, :; :; :; :, :, :, :;
:;2011 /03/07 :;07:43
:; :; :; :; :; :; :; :; i : :; :, :; :, :, :; ................ ........
..
.~~........~..........................
,.
OFF
2011/03/07
........................ ---- ----
or record$4) with(material USPAT; USOCR; FPRS; EPO; or content or page or sheet or original or file or JPO; image or picture or top or DERWENT; bottom) near5 (part or i BMTDB portion or section or region or line or paragraph) near9 (control $4 or correct$4 or adjust $4 or modification or modifying or chang$4 or :altering) near3 (resolution or gradation or pitch)and @ad< "20080317"
.---- -- - -- - -........................................
07:44
S15
597
(print$4 or copier or mfp or record$4) with(material or content or page or sheet or original or file or image or picture or top or bottom) near5 (part or portion or section or region or line or paragraph) near6 (control $4 or correct$4 or adjust $4 or modification or :modifying or chang$4 or altering) near3 (resolution or gradation or pitch)and @ad<"20080317"
US-PGPUB; USPAT; USOCR;
OFF ...... 2011/03/07

07:47
FPRS; EPO;
JPO; DERWENT; IBM TDB
file:///CI/Documents%20and%20Settingsjgrant/My%20Doc...1353/EASTSearchistory.11231353_AccessibleVersion.htm
(6 of 26)4/11/11 12:42:38 PM
EAST Search History
:S16 "419
"
(print$4 or copier or mfp)

with(material or content or page or sheet or original or file or image or picture or top or bottom) near5 (part or portion or section or region or line or paragraph) near6 (control $4 or correct$4 or adjust $4 or modification or modifying or chang$4 or Saltering) near3 (resolution or gradation or pitch)and @ad< "20080317"
US-PGPUB;
USPAT; USOCR; FPRS; EPO; JPO; DERWENT; ;BM TDB
OR
OFF
2011/03/07 07:49
S17
141
(user or client or operator or person) near3 (icon or module or machine or comparaor or processor or interface or keybaord or select$4 adj means or . seelct$4) near4 (select$4 or input$4 or designat$4 or choos$4 or indicat$4 or key$4 or indicat$4) near5 (process or operation or resolution or size or color or black adj white or print adj (function or application)) and (reenter or resumbit$4 or :redesignat$4 or key adj in or re adj enter or re adj submit$4 or re adj transmit $4 or retransmission or retransmitting ) near6 ;(icon or module or ;machine or comparator or :processor or interface or
US-PGPUB; USPAT; USOCR; FPRS; EPO; JPO; DERWENT; IBM _TDB
OR
OFF
2011/03/08 19:10
keyboard or select$4 adj

means or select$4 or process or opepration or
resoloution or size or color

or black adj white or print adj (function or applica $4)) (print$4 or attribute or job or control or process$4 or customiz$5) and (print$4 or attribute or job or control or process$4 or customiz$5) near4 (print $4 or output$4 or record
S18
2579755
US-PGPUB; USPAT; USOCR; FPRS; EPO; JPO; DERWENT; I BMTDB
OR
"
OFF
2011/03/10 07:34
........ ..... . .. ..... ..... ....
$4)...... ... .... ... ..... .... ... .... ... . ... ..... . .. ...... ... ....... ... .... .... .... ...... .... .. ... .. .. .. .. ... .. ..... ... .. .. . ... ..
... .. ... ... .. . .... ..... ... ..... ..
file:///ClfDocuments%20and%20Settings/jgrant/My%20Doc...1353/EASTSearchHistory.11231353_AccessibleVersion.htm (7 of 26)4/11/1112:42:38 PM
EAST Search History
S19
393331
. ....... . . . . . . . .
S20 53550
-- .. ...................... .. .......... ............................ .......... ...... 083!.. . . . . . . . . . . .

OR US-PGPUB; . USPAT; USOCR; FPRS; EPO; JPO; DERWENT; :IBMTDB OFF 2011/03/10 :07:47
(print$4 or attribute or job or control or process$4 or customiz$5) and (print$4 or attribute or job or control or process$4 or customiz$5) near4 (print $4 or output$4 or record $4) and (digital) near4 (computer or process$4 or image or data or file) and .......... . .... @ad<"20080321" ... (print$4 or attribute or job or control or process$4 or customiz$5)near3 (sheet or page or paper) and (print$4 or attribute or job or control or process$4 or customiz$5) near4 (print $4 or output$4 or record $4) and (digital) near4 (computer or process$4 or image or data or file) and . ....... .... .......... ............ @ad< 20080321"
US-PGPUB; USPAT; USOCR; FPRS; EPO; JPO; DERWENT; IBM_TDB
OR
OFF
2011/03/10 07:45
........
. ..........
............ : ... ........................ ..... OFF 2011/03/10 :07:48
521
33678
..... .............< 00......"........................................:..................................!.... . ............................. ..... ...... d ..8.3................. . . .......... . .. .. .... .. .................. . ......
;52 11 US-PGPUB; USPAT; USOCR; FPRS; EPO; JPO; DERWENT; IBMTDB OR OFF
(print$4 or attribute or job or control or process$4 or customiz$5)near3 (sheet or page or paper) same (print$4 or attribute or job or control or process$4 or customiz$5) near4 (print $4 or output$4 or record $4) and (digital) near4 (computer or process$4 or image or data or file) and @ad<"20080321" (print$4 or attribute or job or control or process$4 or customiz$5)near3 (sheet or page or paper) near9 (print$4 or output$4 or record$4) near3 (print$4 or attribute or job or .control or process$4 or customiz$5)near3 (sheet or page or paper) and digitial near3(computer or process$4 or image or :data or file) and @ad<"20080321"
US-PGPUB; OR USPAT; USOCR; FPRS; EPO; JPO; DERWENT; IlBMTDB
'07 :51
2011/03/10
ie:///C I.........../Documents%2 ................. and%2 Setings/jgrant/My%2 Doc...353EASTSearchsory.
23353AccessibleVersion. htm (8 f26 /1/11124238PM
.............................
ile:///CI/Documents%20and%20Settings/jgrant/My%20Doc...1353/EASTSearchHistory.11231353_AccessibleVersion.htm
(8 of 26)4/11/11 12:42:38 PM
EAST Search History
,S23
.. . .
OFF 2011/03/10 OR (print$4 or attribute or job US-PGPUB; _07:52 or control or process$4 or USPAT; USOCR; customiz$5)near3 (sheet IFPRS; EPO; JPO; or page or paper) and DERWENT; (print$4 or output$4 or record$4) near3 (print$4 ;BM TDB or attribute or job or control or process$4 or customiz$5)near3 (sheet or page or paper) and digitial near3(computer or process$4 or image or :data or file) and ... .. @ad< "20080321" . ..... ...................................................... ..... ... .. ... ... ... ... .. ... ..... .. .... ............ ...... ................... .. ....................... ......
8242122.....
(print$4 or attribute or job US-PGPUB;

or control or process$4 or customiz$5)near3 (sheet or page or paper) and (print$4 or output$4 or : record$4) near3 (print$4 or attribute or job or control or process$4 or customiz$5)near3 (sheet or page or paper) and digital near3(computer or process$4 or image or data or file) and "2: :: @ad< 20080321" USPAT; USOCR; FPRS; EPO; JPO; :ERWENT; IBMTDB
OR
OFF
2011/03/10
07:52
25
4726
........... (print$4 or attribute or job US-PPUB;

or control or process$4 or customiz$5)near3 (sheet or page or paper) near9 '(print$4 or output$4 or record$4) near3 (print$4 or attribute or job or control or process$4 or customiz$5)near3 (sheet or page or paper) same :digital near3(computer or process$4 or image or data or file) and @ad< "20080321". USPAT; USOCR; FPRS; EPO; JPO; DERWENT; IBM_TDB
OR
OFF
2011/03/10
07:53
S26
5237
(print$4 or attribute or job US-PGPUB; OR or control or process$4 or :USPAT; USOCR; customiz$5)near3 (sheet FPRS; EPO; or page or paper) near9 JPO; (print$4 or output$4near2 DERWENT; (device or machine or 'IBM_TDB apparatus) or record$4) near3 (print$4 or attribute or job or control or process: $4 or customiz$5)near3 (sheet or page or paper) same digital near3 (computer or process$4 or ;image or data or file) and
OFF
2011/03/10 :;07:56
(9 of 26)4/11/1l112:42:38 PM
EAST Search History
~~~@ad<"20080321"
S 114FA (attrihIt0 Ar inh nr rcontrol :H R-P(GPi iP* ?OR ;OfFF :2011/03M/10l
or process$4 or customiz $5)near3 (sheet or page or paper) near9(print$4 or output$4near2 (device or machine or apparatus) or record$4) near3 (print$4 or attribute or job or control or process$4 or customiz$5)near3 (sheet or page or paper) same digital near3(computer or process$4 or image or data or file) and @ad< "20080321"
. .. . . . . . . . . .. .
USPAT; USOCR;
VI l
i
VI I
:;C-v /VV/
IV
FPRS; EPO; JPO; DERWENT; IBMTDB
S28
846
(attribute or job or control or process$4 or customiz $5)near3 (sheet or page or paper) near9(print$4 or output$4 near2 (device or machine or apparatus) or record$4) near3 (attribute or job or control or process: $4 or customiz$5)near3 (sheet or page or paper) same digital near3 (computer or process$4 or image or data or file) and @ad<"20080321"
, ;VI :; :;
NOFF
1
2011/03/10 07:59
i';
;.
........
S29
450
(attribute or job or control:I US-PGPUB;

or process$4 or customiz IUSPAT; USOCR; FPRS; EPO; $5)near3 (sheet or page or paper) near9(print$4 or JPO; output$4 near2 (device or IDERWENT; :machine or apparatus) or IBMTDB record$4) near3 (attribute or job or control or process: $4 or customiz$5)near3 (sheet or page or paper) with digital near3 (computer or process$4 or image or data or file) and @ad< "20080321"
OR ............................ OFF ....... 2011/03/10............
08:00
file:///Cf/Documents%20and%20Settings/jgrant/My%20Doc...1353/EASTSearchHistory.11231353_AccessibleVersion.htm
(10 of 26)4/11/11 12:42:38 PM
EAST Search History
S30
846
digital near3( computer or process$4 or image or data or file)same(attribute or job or control or process $4 or customiz$5)near3 (sheet or page or paper) near9(print$4 or output$4 near2 (device or machine or apparatus) or record$4) near3 (attribute or job or control or process$4 or customiz$5)near3 (sheet or page or paper) and
OFF.
2011/03/10 08:02
..... .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .... .. .. .. .. .. ...
S31..
846
d<"20080321" S30 or S29
...... ...
................
........ U ..
.........
O. .................................
........ ... . ..
1 /..3.. ............ .. .. 0 ;.. . . . .
USPGPUB; OR USPAT; USOCR;08:03 FPRS; EPO; JPO; DERWENT; IBMTDB OR
OFF
2011/03/10
S32
677
digital near3(computer or US-PGPUB; process$4 or image or :USPAT; USOCR data or file)same(attribute FPRS; EPO; or job or control or process :JPO; $4 or customiz$5)near3 DERWENT; (sheet or page or paper) :IBMTDB near5(print$4 or output$4 near2 (device or machine or apparatus) or record$4) near3 (attribute or job or control or process$4 or customiz$5)near3 (sheet or page or paper) and digital near3(computer or process$4 or image or data or file)with(attribute.. or job or control or process $4 or customiz$5)near3 (sheet or page or paper) near5(print$4 or output$4 near2 (device or machine or apparatus) or record$4) near3 (attribute or job or :control or process$4 or customiz$5)near3 (sheet or page or paper) and @ad<"20080321"
OF
2011/03/10
08:05
ii' Y:. ....... .:...... ....... . . . .::.. ... .. ... ................i[ ''5 " ... .. .i . @ad< "20080321 " ........ ........ ....... ... ... .. .. ... ... ... .... ............................. ".. 6 ............ . ...
S33 337 US-PPUB, OR USPAT; USOCR; FPRS; EPO; JPO; DERWENT; IBM_TDB OFF 2011/03/10 08:06
.,,.
file:///CI/Documents%20and%20Settings/jgrant/My%20Doc...1353/EASTSearchHistory.
11231353_AccessibleVersion.htm(11 of 26)4/11/11 12:42:38 PM
EAST Search History
S34
;151
'_
digital near3(computer or process$4 or image or data or file)with(attribute or job or control or process $4 or customiz$5)near3 (sheet or page or paper) near5(print$4 or output$4 near2 (device or machine or apparatus) or record$4) near3 (attribute or job or control or process$4 or customiz$5)near3 (sheet or page or paper) and @ad<"20080321" and ("358"/$.ccls. or "347"/$. ccIs. or "399"/$.ccls. or "355"/$.ccls.) near3(computer or process$4 or image or data or file)with(attribute or job or control or process $4 or customiz$5)near3 (sheet or page or paper) near5(print$4 or output$4 near2 (device or machine'_ or apparatus) or record$4) near3 (attribute or job or control or process$4 or customiz$5) near3 (sheet or page or paper) and (yacoub.inv. or liang.inv. or zhou.inv.)
;OFF "
2011/03/10 09:09
S35
1 .digital
OR US-PGPUB USPAT; USOCR; FPRS; EPO; JPO; DERWENT; IBM TDB
OFF.
2011/03/10 09 : 11
S36
400
S30 and ("358"/$.ccls. or

"347"/$.ccls. or "399"/$. ccls. or "355"/$.ccls.)
US-PGPUB;
'USPAT; USOCR,
OR
OFF
2011/03/10 :09:15
FPRS; EPO;
JPo;
DERWENT; IBM TDB
S37
S........ .................................... ... - ......... i......... .... ..................... .... ... .. .. T .......... ... .
249 ............('358"/$.ccls. or S30 and "347"/$.ccls. or "399"/$. ccls. or "355"/$.ccls.) not S33
'US-PGPUB;
..........OFF OR
USPAT; USOCR; FPRS; EPO; JPO; DERWENT;,
2011/03/10 09:15
file:CllBMTDocuments%20and%20SetngsjgrantMy%20Doc...1353EASTSearchHistory.11231353_AccessibleVersion.htm (12 26/1/11 12:42:38 of PM
file:///Cl/Documents%20and%20Sertings/jgrant/My%20Doc...1353/EASTSearchHistory.11231l353_AccessibleVersion.htm (12 of 26)4/11/1112:42:38 PM
EAST Search History
r~\~rr\ . ............. ",- .....................
......... ..... ...
.. ..... ........
. ....
111-11.1-11.
S38
080 2(
:; :;
:(recipient or user or operator or client or personell or personal or. person) near4 (setting or address or parameters or :meta or data) near7 (server or exchange or saff or forward or modem or packet$4) with (display or monitor or screen or Icd or window or view$4) and (server or http or smtp or xml or transmitter) near4 (protocol or protocal or http or smtp or xml or :remote adj procedure) and @ad< "20071213" and (scan$4 or read$4 or :copier) near4 (data or email or electronic adj :mail or message or copy or data)
US-PGPUB; OR USPAT; USOCR; FPRS; EPO;
OFF
2011/03/10...........
22:06
JPo;
DERWENT; IBM_TDB
. ......... . .... ... . . .".. .. ..... .. .... ... .... . ... . . .

:-- - ---- -- - -
S39.
17359
(print$4 or (image or data :US-PGPUB; OR or picture) near4 (formulat USPAT; USOCR; $4 or form$4 or print$4 or FPRS; EPO; output$4 or unit or device JPO; or apparatus)) near9 DERWENT; (thumb adj drive or disc or IBMTDB disk or dvd or flash adj drive or cd or memory adj card or external adj (fuffer or stor$4 or memory or register)) same (display or. :monitor or Icd or screen or window) with (indicat$4 or select$4 or keyboard or keypad or designat$4 or choose or choice or enter) and @ad< "20080221" ' (print$4 or (image or data or picture) near4 (formulat $4 or form$4 or print$4 or output$4 or unit or device or apparatus)) near4 (thumb adj drive. or disc or disk or dvd or flash adj.drive or cd or memory adj card or external adj (fuffer or stor$4 or memory or register)) near9 (display or monitor or Icd or screen or window) with (indicat $4 or select$4 or Skeyboard or-keypad or designat$4 or choose or choice or enter)with (file or address or identification US-PGPUB; :USPAT; USOCR; :FPRS; EPO; JPO;
nR ;vn
OFF
2011/03/11 10:35
.........................
.... ............,
S40
26
1OFF .,,
2011/03/11 10:40
:DERWENT;
IBM TDB
(13 of 26)4/11/11 12:42:38 PM
EAS' Search History
or identify or name)near3 -(name or paper or file or job or document or sheet) :and @ad<."20080221"
:CA
'i
iSi4] ;;..,-r
i.......
:i
00...........
............-
".
.............
"""""" .....
................
;188
i i
:; ;;
(print$4 or (image or data or picture) near4 (formulat $4 or form$4 or print$4 or output$4 or unit or device or apparatus)) with (thumb adj drive or disc or disk or dvd or flash adj drive or cd or memory adj card or external adj (fuffer or stor$4 or memory or register)) with (display or :monitor or Icd or screen or :window) with (indicat$4 or select$4 or keyboard or keypad or designat$4 or choose or choice or enter) :with (file or document or sheet or address or identification or identify or name)near3 (name or paper or file or job or Idocument or sheet) and Ia@ad< "20080221"
US-PGPUB;
OR
OFF
20103/11
10:44
JPO;
DERWENT; IBM TDB.
. . .. . . . . . . ..
..
..
..
. ..
.. ..
.. .
. ..
..
..
..
..
. . . ..
. . . ..... .. .. . ........ ".. . ...... . . ............ .... .. . . . .." . . . .. ....... .
.... .... .. .. .. .... .. .... .. .. ..... .... ... .. .. ...... ... .. ..
~~~~_
S42
(print$4 or (image or data :or picture) near4 (formulat $4 or form$4 or print$4 or output$4 or unit or device or apparatus)) with (thumb adj drive or disc or disk or dvd or flash adj drive or cd or memory adj card or external adj (fuffer or stor$4 or memory or register)) with (display or :monitor or Icd or screen or window) with (indicat$4 or select$4 or keyboard or keypad or designat$4 or choose or choice or enter) with (file or document or sheet or address or identification or identify or name)near3 (name or paper or file or job or document or sheet) and "20080221" and
US-PGPUB; ...
OR .............
OFF
2011/03/11
USPAT; USOCR; FPRS; EPO; JPO; DERWENT; IBM TDB
10:48
:: -i!
1 :,'
"
.@ad<
... . ..
("358"/$.ccls. or "355"/$.
. . . . . .. . . . . .. . . . . .. . . . . .. . . . . . ..
ccls. or "347"/$.ccls. or "399"/$.ccls.)

. . . .
.. . . . . . .. . . . . .. . . . .. . . . . .. .
(14 of 26)4/11/11 12:42:38 PM
EAST Search History
S43...
0..
LIIII\~~~~ . ...
'iS44
.................. ................ ..... . .... ..... .. . i; ..... ,
.. . .. . . . . .. . . . . .::.
:(print$4 or (image or data or picture) near4 (formulat $4 or form$4 or print$4 or output$4 or unit or device or apparatus)) with (thumb adj drive or disc or disk or dvd or flash adj drive or cd or memory adj card or external adj (fuffer or stor$4 or memory or register)) with (display or :monitor or Icd or screen or :window) with (indicat$4 or select$4 or keyboard or keypad or designat$4 or choose or choice or enter) (file or document or :sheet or address or identification or identify or name)near3 (name or :paper or file or job or document or sheet) and :hatori.inv. ........ ...................... .... ......... . .... .... .....
~~.
.l ... EII\\~\ .........
_US-PGPUB; USPAT; USOCR; FPRS;EPO;

.t
O R ............................ :;O VI ;iVl

1
F.........
1
;2011 /03/ 1 ...........

"LV :
1 1/
VV/
i :; i i i
n.nn
X12:08
JPo
DERWENT; IBM_TDB
i i i
i i
.with
........ ........ ........ ....... ..
. . ...
12
.r
....... ....... . . .
........ ....... . .
"1085740"
US-PGPUB;
USPAT; USOCR;
OR
OFF
.......... .............. . ..... ..... 2011/03/13

16:36
FPRS; EPO; JPO;

DERWENT; IBMTDB
i i
.......................
.. ................................
...............................
-- - -: : --- --- -: . .. ...7 ... .. ... .. ... .. ... .. .. ................................. ....... ,,................................. .....
S45
6612
'.
(software or program or instruction) near4 (simplex or duplex or color or hue or monochrom$4 or (header near3 set) or sortS4 or ((print or :record) near2 set$4) or ((print$4 or recorder or recording or recorded or record) near3 (id or identification or name)) or staple or sortS5 or ((paper or page or sheet) near2 (type or kind ior class))) and (switch$5 ior selected 'or selector or selecting or :choose or choosing or chosen or designating or designated ior designator or designation or key or keying) near3 (simplex or duplex or color or hue or tincture or tint or chromatic or (header near2 set) or sortS4 or adf) and @ad<"20071127"
US-PGPUB; " OR USPAT; USOCR; :FPRS; EPO; JPO; DERWENT; IBM_TDB
OFF
2011/03/14 01:15
(15 of 26)4/11/11 12:42:38 PM
EAST Search History
"""""""""'
. ,,,,,,,,,,,,,,..,,,..,,..,,.,,,,,,,,,,,.,,,,,,,,..,,,,...,............
,,.., vn
...............
S46 .
2,21904
(eeprom or ram or rom or store or flash adj drive or :mrmory or register or buffer or storage or cahse or disk or disc or dvd or cd) near4 (address or number or packetS2 or protocoal or protocol or network adj (address or format or version or language or number) or buad adj (rate or speed)) same (eeprom or ram or rom or store or flash adj drive or mrmory or register or buffer or storage or cahse or disk or disc or dvd or cd) near4 ( text or image or data or file or picture or pictorial
USPAT; USOCR; FPRS; EPO;,
1!2011/04/01 OFF 1OFF 21:31

:,i
JPO;
DERWENT;, IBM_TDB
:;
i i i
i a i . .................... .....................
847
17328
or video or audio) (eeprom or ram or rom or ~US-PGPUB,

store or flash adj drive or. :mrmory or register or buffer or storage or cahse or disk or disc or dvd or cd) near4 (address or number or packet$2 or protocoal or protocol or network adj (address or format or version or language or number) or baud adj (rate or speed)) same (eeprom or ram or rom or store or flash adj drive or mrmory or register or buffer or storage or cahse or disk or disc or dvd or cd) near4 ( text or image or data or file or picture or pictorial or video or audio) and (delet$4 or eras$5 or cancel$7 or zero$4 or blot $7 or modification or modifying or abat$5 or decreas$$ or transferring) near4 (address or number or packet$2 or protocal or protocol or network adj (address or format or version or language or number)) near7 (eeprom or ram or rom or store or flash adj drive or memory :or register or buffer or ::storage or cahse or disk or USPAT; USOCR; FPRS; EPO;
"""~~""~"~""'
OR
OFF
........... ................ . 2011/04/01

21:36
JPO;
DERW ENT;
iBMTDB
ile:///CI/Documents%20and%20Settings/jgrantIMy%20Doc...1353/EASTSearchHistory.11231353_AccessibleVersion.htm
(16 of 26)4/11/11 12:42:38 PM
EAST Search History
:;:
disc or dvd or cd) and .. . . ......................................... @ad< "20080402" ............ ....... ...... ............... .............. i :nR (eeprom or ram or rom or US-PGPUB; store or flash adj drive or USPAT; USOCR; memory or register or FPRS; EPO; buffer or storage or cashe JPO; or disk or disc or dvd or DERWENT; cd) near4 (address or IBM_TDB number or packet$2 or protocal or,protocol or network adj (addressor format or version or language or number) or baud adj (rate or speed)) :with (eeprom or ram or rom or store or flash adj drive or memory or . register or buffer or storage or cashe or disk or disc or dvd or cd) near4 (text or image or data or file or picture or pictorial or video or audio) and (delet$4 or eras$5 or cancel$7 or zero$4 or blot $7 or abat$5 or decreas$$ or transferring) near4 (address or number or packet$2 or protocal or protocol or network adj (address or format or version or language or number)) near7 (eeprom or ram or rom or store or flash adj drive or memory or register or buffer or :storage or cashe or disk or Idisc or dvd or cd) and @ad< . .. . . . . .. . . . . . .. . . . . .. . . . . . . . .. . . . .. . . . . .. . . . . . . ::.. .. . . . . .. . . . ."20080402".. . . . . .. . . . . . .. . . . . .. . .... . . . . . .. . . . . .. . . . . . . . . . . .. . . . .. . . . . .. ;. . ::
;vri i
.........................................................................................................
:OFF
;, ,. r :; i ,:; :; :_ '; :; :; i i i
2011 /04/01
:;21:40
. v
i i
i i
i i i i
i i i i ; i" i i
i i i i i i ..................... ?
........................
S49
17185
(eeprom or ram or rom or store or flash adj drive,or memory or register or buffer or storage or cashe or disk or disc or dvd or cd) near4 (address or number or packet$2 or protocal or protocol or network adj (address or
format or version or
US-PGPUB; OR USPAT; USOCR; FPRS; EPO; JPO; DERWENT; IBM_TDB
OFF
2011/04/01 21:46
language or number) or baud adj (rate or speed)) near8 (eeprom or ram or rom or store or flash adj . drive or memory or register or buffer or storage or cashe or disk or :disc or dvd or cd) near4
::
file:///C/Documents%2and%2Settings/jgrant/My%20Doc...1353/EASTSearchHisory11231353_AccessibleVersion.htm of 26)4/11/11 12:42:38 PM (17
EAST Search History
S50
;8166
( text or image or data or file or picture or pictorial or video or audio) and (delet$4 or eras$5 or cancel$7 or zero$4 or blot $7 or abat$5 or decreas$4 or transferring) near4 (address or number or packet$2 or protocal or :protocol or network adj (address or format or :version or language or number)) near7 (eeprom :or ram or rom or store or flash adj drive or memory :or register or buffer or :storage or cashe or disk or :disc or dvd or cd) and S@ad<"20080402" :(eeprom or ram or rom or :store or flash adj drive or :memory or register or :buffer or storage or cashe :or disk or disc or dvd or cd) near3 (address or number or packet$2 or protocal or protocol or :network adj (address or format or version or :language or number) or :baud adj (rate or speed)) near8 (eeprom or ram or rom or store or flash adj :drive or memory or register or buffer or :storage or cashe or disk or 'disc or dvd or cd) near2 (text or image or data or file or picture or pictorial :or video or audio) and (delet$4 or eras$5 or :cancel$7 or zero$4 or blot $7 or abat$5 or decreas$4 :or transferring) near3 (address or number or packet$2 or protocal or :protocol or network adj (address or format or :version or language or number)) near3 (eeprom or ram or rom or store or flash adj drive or memory :or register or buffer or :storage or cashe or disk or :disc or dvd or cd) and
3 IUS-PGPUB; USPAT; USOCR; FPRS; EPO;

.. .. . .. . .. .. ._ ... .... .1
OFF ........................
: ;, ;c
2011/04/01
21:53
RJPO;
DERWENT; IBM_TDB
.; . .
@ad< 20080402"
.. ...........
fie:///C/Documents%20and%20Setings/jgrant/My%20Doc...1353/EASTSearchHistory.11231353_AccessibleVersion.htm
(18 of 26)4/11/11 12:42:38 PM
EAST Search History
!51.
;w
:
n~~~\_~~__~~~__~_~__~_\\\~ __~__~~_
i
:;
i2589
:;
. . : ....... . ;.. .. ...... ...... ...... . . ... . . . . . . . . .. ....... ................. : . . . .. . . .. .. . . . .... . . .. . . . . .. .. .. . . . . . .. . . . .. . . . .... . . . .. . . . .. . . . . .. . .. . . . .. .. .
(eeprom or ram or rom or store or flash adj drive or memory or register or buffer or storage or cashe or disk or disc or dvd or cd) near3 (addressor number or packet$2 or :protocal or protocol or network adj (address or format or version or language or number) or baud adj (rate or speed)) near5 (eeprom or ram or rom or store or flash adj :drive or memory or register or buffer or storage or cashe or disk or disc or dvd or cd) near2 (text or image or data or file or picture or pictorial or video or audio)same (delet$4 or eras$5 or cancel$7 or zero$4 or blot $7 or abat$5 or decreas$4 or transferring) near3 (address or number or packet$2 or protocal or protocol or network adj (address or format or version or language or number)) near3 (eeprom or ram or romor store or flash adj drive or memory or register or buffer or storage or cashe or disk or disc or dvd or cd) and :@ad< "20080402"
':JSPAT tUSOCR: _'15 FPRS; EPO; =.JPO; DERWENT; IBM TDB j

: . .. . .. ..
.l
S52
2153
(eeprom or ram or rom or store or flash adj drive or memory.or register or buffer or storage or cashe or disk or disc or dvd or cd) near3 (address or number or packet$2 or protocal or protocol or :network adj (address or format or version or language or number) or baud adj (rate or speed)) near5 (eeprom or ram or rom or store or flash adj 'drive or memory or register or buffer or storage or cashe or disk or :disc or dvd or cd) near2 (text or image or data or file or picture or pictorial
US-PGPUB;
JPO;.
OR
OFF
....... 5)'" .........

2011/04/01 22:05
DERWENT; IBMTDB
file:///CI/Documents%20and%20Settings/jgrant/My%2Doc...1353/EASTSearchHistory. 11231353_AccessibleVersion.htm (19 of 26)4/11/11 12:42:38 PM
EAST Search History
or video) with (delet$4 or eras$5 or cancel$5 or zero $4 or blot$5 or abat$5 or decreas$4 or transferring) .near3 (address or number or packet$2 or protocal or protocol or network adj (address or format or version or language or number)) near3 (eeprom or ram or rom or store or flash adj drive or memory or register or buffer or storage or cashe or disk or .disc or dvd or cd) and @ad< "20080402"
i i i i i i,
i i i i i i i
i i
, i
~S53...
3853
(eeprom or ram or rom or store or flash adj drive or :memory or register or buffer or storage or cashe or disk or disc or dvd or cd) near3 (address or number or packet$2 or protocal or protocol or network adj (address or format or version or language or number) or baud adj (rate or speed)) near5 (eeprom or ram or rom or store or flash adj :drive or memory or register or buffer or storage or cashe or disk or disc or dvd or cd) near2 (text or image or data or file or picture or pictorial or video) near6 (delet$4 or eras$5 or cancel$5 or zero$4 or blot$5 or abat$5 or decreas$4 or transferring) near3 (address or number or packet$2 or protocal or protocol or network adj (addressor format or version or language or number)) near3 (eeprom or ram or rom or store or flash adj drive or memory or register or buffer or storage or cashe or disk or disc or dvd or cd) and
i iUS-PGPUBI;
............................. OR
;OFF
.USPAT; USOCR; FPRS; EPO;
2011/04/01 22:08
JPO;
::DERWENT;
IBMTDB
S@ad<o"20080402"
file:///CI/Documents%20and%2Settings/jgrant/My%20Doc...1353/EASTSearchHistory.
1231353_AccessibleVersion.htm (20 of 26)4/11/11 12:42:39 PM
EAST Search History
S54
3853
(eeprom or ram or rom orUS-PGPUB;

:store or flash adj drive or :memory or register or buffer or storage or cashe :or disk or disc or dvd or cd) near3 (address or number or packet$2 or protocal or protocol or network adj (address or format or version or language or number) or baud adj (rate or speed)) near5 (eeprom or ram or rom or store or flash adj :drive or memory or register or buffer or storage or cashe or disk or disc or dvd or cd) near2 (text or image or data or file or picture or pictorial or video) near6 (delet$4 :or eras$5 or cancel$5 or :zero$4 or blot$5 or abat$5 :or decreas$4 or transferring) near3 (address or number or packet$2 or protocal or protocol or network adj (address or format or version or language or number)) near3 (eeprom or ram or rom or store or flash adj drive or memory or register or buffer or :storage or cashe or disk or disc or dvd or cd) and @ad< "20080402" USPAT; USOCR; FPRS; EPO; JPO; :DERWENT; IBMTDB
OR............................ OFF ......... 2011/04/01............

22:13
:_
:;
::!
S55
1936
:;
:;
:_
(eeprom or ram'or rom or store or flash adj drive or memory or register or buffer or storage or cashe or disk or disc or dvd or cd) near3 (address or number or packet$2 or protocal or protocol or :network adj (address or format or version or language or number) or :baud adj (rate or speed)) near5 (eeprom or ram or rom or store or flash adj drive or memory or
US-PGPUB; USPAT; USOCR; FPRS; EPO; JPO; DERWENT; IBMTDB
OR
OFF
'2011/04/01 22:22
register or buffer or
:storage or cashe or disk or :disc or dvd or cd) near2 (text or image or data or Ifile or picture or pictorial
:;
file:///Cl/Documents%20and%20Settingsjgrant/My%20Doc...1353/EASTSearchHistory. 11231353_AccessibleVersion.htm (21 of 26)4/11/11 12:42:39 PM
EAST Search H istory
S56
S56 1975
.. .................... :...............................................................
or video) with (delet$4 or eras$5 or cancel$5 or zero $4 or blot$5 or transferring) near3 (address or number or packet$2 or protocal or protocol or network adj (address or format or version or language or number)) near3 (eeprom or ram or rom or store or flash adj drive or memory or register or buffer or :storage or cashe or disk or ,disc or dvd or cd) and @ad< "20080402" (eeprom or ram or rom or US- PGPUB; store or flash adj drive or USPAT; USOCR; memory or register or FPRS; EPO; buffer or storage or cashe JPo; DERWENT; or disk or disc or dvd or IIIBMTDB cd) near3 (address or number or packet$2 or protocal or protocol or network adj (address or format or version or language or number) or baud adj (rate or speed)) near5 (eeprom or ram or rom or store or flash adj drive or memory or register or buffer or storage or cashe or disk or disc or dvd or cd) near2 (text or image or data or file or picture or pictorial or video) and (control$4 or process$4 or select$4 or switch$4 or key$4 or designat$5) near4 (delet $4 or eras$5 or cancel$5 or zero$4 or blot$5 or transferring) near4 (address or number or packet$2 or protocal or protocol or network adj (address or format or version or language or number)) near4 (eeprom or ram or rom or store or flash adj drive or memory :or register or buffer or storage or cashe or disk or :disc or dvd or cd) and '2008 ........... ............ . @ad..........04.02".............
i i
...........................
OFF
i i
2011/04/01 22:28
i i ;'
i i i i i
i i i ;' ; i i i i a i i i
file:///CI/Documents%20and%20Setings/jgrant/My%20Doc...1353/EASTSearchHistory.11231353_AccessibleVersion.htin
(22 of 26)4/11/11 12:42:39 PM
EAST Search History
.................... ................
:,,
S57
ii! 590
....
. .
........................
:;
:;
(eeprom or ram or rom. or :store or flash adj drive or memory orregister or . buffer or storage or cashe or disk or disc or dvd or od) near3 (address or :number or packet$2 or protocal or protocol or :network adj (address or format or version or language or number) or baud adj (rate or speed)) near5 (eeprom or ram or rom or store or flash.adj drive or memory or register or buffer or storage or cashe or disk or :disc or dvd or cd) near2 (text or image or data or file or picture or pictorial or video) same (control$4 or process$4 or select$4 or switch$4 or key$4 or :designat$5) near4 (delet $4 or eras$5 or cancel$5 or zero$4 or blot$5 or transferring) near4 (address or number or packet$2 or protocal or protocol or network adj (address or format or version or language or. number)) near4 (eeprom or ram or rom or store or flash adj drive or memory or register or buffer or :storage or cashe or disk or disc or dvd or cd) and
OR US-PGPUB; USPAT; USOCR; FPRS; EPO; JPO; DERWENT; IBMTDB
OFF
2011/04/01 22:31
,;.... ....... :..........
:.....................
@ad< '"20080402"
L. ....................... ,..................... ..........................................
S58
'
(choic$4 or keying or select$4 or chos$4 or discriminat$4 or interpret $4 or predict$4 or approximat$4 or adjust$4) near4 (language or fomrat or page adj descript$4 or pdl or descript$4 near4 language or page adj job S or printer adj job or job adj language or printer adj job) with (stitch or failure or error or aberration or noise or incompatible) near4 ;(language or fomrat or :page adj descript$4 or pd! or descript$4 near4
OR US-PGPUB; USPAT; USOCR;
OFF
2011/04/06
21:48
:
FPRS; EPO; JPO;

DERWENT; IBM TDB
3 3
:I3
3 :i
..
(23 of 26)4/11/11 12:42:39 PM
EAST Search History
.................
. ... . . . . . . . . .. . . . . .. .
language or page adj job or printer adj job or job adj language or printer adj job) with (choic$4 or keying or select$4 or chos $4 or discriminat$4 or interpret$4 or predict$4 or approximat$4 or adjust$4) near4 (rank or order or priority or preference or :position or list or type or kind) near7 (language or fomrat or page adj descript $4 or pdl or descript$4 near4 language or page adj job or printer adj job or job adj language or printer adj job) and .@ad< "20080723"
. .. . . . . .. . . . . . . . . .. . . . . .. . . . .. . .. . . . . . . . .. . . . .. . . . .. . . ..
. . . . . ... . . . .. . .. . . . .. . . . . . . . . .. .. . . . . . .. . . . .. .
S59
:............
..... 8".....
. .... ... .
. . . .. . . . . . .. . . . . . .. . . . . .. . . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . . . .. .. . . . . . . .. . . . . . . . .. . . . . . . . .. . . . ..
:(choic$4 or keying or US-PGPUB; select$4 or chos$4 or USPAT; USOCR; discriminat$4 or interpret FPRS; EPO; $4 or predict$4 or JPO; ,approximat$4 or adjust$4) DERWENT; near6 (language or format IBM_TDB or page adj descript$4 or pdl or descript$4) near6 ,(language or page adj job or printer adj job or job adj language or printer adj job) same (stitch or failure or error or aberration or noise or incompatible) near6 (language or fomrat or page adj descript$4 or pdl or descript$4) near6 (language or page adj job or printer adj job or job adj language or printer adj job) with (choic$4,or keying or select$4 or chos $4 or discriminat$4 or interpret$4 or predict$4 or approximat$4 or adjust$4) near5 (rank or order or priority or preference or :position or list or type or kind) near7 (language or format or page adj descript t $4 or pdl or descript$4) near4 (language or page adj job or printer adj job or job.adj language or printer adj job) and :@ad< "20080723"
OFF
2011/04/06 21:53
file:///CI/Documents%20and%20Settings/jgrant/My%20Doc...1353/EASTSearchistory.11231353_AccessibleVersion.htm
(24 of 26)4/11/11 12:42:39 PM
EAST Search History
:S60
........ii[6 i(choic$4 or keying orr.. . . . . . . .,e .... 0 ......... ....... [ i o..or

select$4 or chos$4 or discriminat$4 or interpret $4 or predict$4 or approximat$4 or adjust$4) :near6 (language or format or page adj descript$4 or pdl or descript$4) near6 (language or page adj job or printer adj job or job adj language or printer :adj job) and (stitch or failure or error or aberration or noise or incompatible) near6 (language or format or :page adj descript$4 or pdl or descript$4) near6 (language or page adj job or printer adj job or job :adj language or printer :adj job) with (choic$4 or ,keying or select$4 or chos $4 or discriminat$4 or interpret$4 or predict$4 or approximat$4 or adjust$4) :near5 (rank or order or priority or preference or :position or list or type or :kind) near7 (language or format or page adj descript $4 or pdl or descript$4) near4 (language or page :adj job or printer adj job :or job adj language or "printer adj job) and ..... .. - ,,. -............... ......."20080723"....... .............. ......--.. ---.. .: ..... :@aid< .............. .......
US-PGPUB; USPAT; USOCR; FPRS; EPO; JPO; DERWENT; IBM_TDB
C.
n9nh1i
f Ai flg
'3
21: 55
.
i :; ,. i
'Z
'Z
:,
't
,.
'Z
:, :;
S61
~""" """"""""""""' """"""""""""""""""""
174
""" ~""""""
~"""""""""
. .. . . . . . .
.. . . . . . . . .. . . . . . . .
US- PGPUB; (choic$4 or keying or select$4 or chos$4 or USPAT; US OCR; FPRS; EPO discriminat$4 or interpret $4 or predict$4 or JPO; approximat$4 or adjust$4) DERWENT; :with (language or fomrat IBMTDB or page adj descript$4 or pdl or descript$4 near4 language or page adj job or printer.adj job or job adj language or printer adj job) with (stitch or failure or.error or aberration or noise or incompatible) with (language or fomrat or page adj descript$4 or pdl or descript$4 near4 language or page adj job or printer adj job or job
OFF
2011/04/09 19:33
file:///CI/Documents%20and%20SettingsljgrntlMy%20Doc...1353/EASTSearchHistory. 11231353_AccessibleVersion.htm
(25 of 26)4/11/11 12:42:39 PM
EAST Search History
..........
adj language or printer adj job) with (choic$4 or keying or select$4 or chos $4 or discriminat$4 or interpret$4 or predict$4 or approximat$4 or adjust$4) with(rank or order or ;priority or preference or position or list or type or kind) with (language or fomrat or page adj descript $4 or pdl or descript$4 near4 language or page adj job or printer adj job or job adj language or printer adj job) and i@ad< "20080723". . d< , 0 8 .... ........ ...........:.............................. ........
. . . . . . . .. . . .. . . . .. . . . . . . . . .. . . ... . . . . . . . . . .. .. .
. . . . . . .. . . . .
~mrr~rmrmrrrr
. . . . . . . . . . . . .. . . . . . . .. . . .
4/11/11 12:41:57 PM H:\ East\ Workspaces\ 11955924.wsp
file:///CI/Documents%20and%20Setings/jgrant/My%20Doc...1 353/EASTSearchHistory. 11231353_AccessibleVersion.htm (26 of 26)4/11/11 12:42:39 PM
1101 llII I1111|1|III IIIll IllllIl i |I |II 111 IIII

US007956890B2
(12)
United States Patent

Cheng et al.
ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEMS Inventors: Ken P. Cheng, Saratoga, CA (US); Edward Y. Chang, Santa Barbara, CA (US); Yuan-Fang Wang, Goleta, CA (US) Assignee: Proximex Corporation, Sunnyvale, CA (US) JP Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 1517 days.
(10) Patent No.: (45) Date of Patent:

(56)
US 7,956,890 B2
Jun. 7, 2011
(54)
References Cited U.S. PATENT I)OCUMENTS A 11/1993 Gormley

A A A A A 12/1995 12/1995 12/1997 6/1998 11/1998 Abe
(75)
5,258,837 5,473,369 5,479,574 5,701,398 5,769,074. 5,835,901
Glier et al. Glier et al. Barnhill et al. Duvoisin et al. (Continued)
(73)
(* )
Notice:
FOREIGN PATENT DOCUMENTS 02004295798 * 10/2004 (Continued) OTHER PUBLICATIONS
(21) (22) (65)
Appl. No.: 11/231,353 Filed: Sep. 19, 2005 Prior Publication Data US 2006/0093190 Al May 4, 2006
Goh et al., "Robust Perceptual Color Identification" U.S. Appl. No. 11/229,091,;filed Sep. 16, 2005. Belhumeur, A. et al. (1997). "Eigenfaces vs. Fisherfaces: recognition using class specific linear projection", IEEE Transactions on Pattern Analysis and Machine Intelligence 19(7): 711-720. (Continued) Primary Examiner - Jerome Grant, II (74) Attorney, Agent, or Firm - Peters Verny, LLP ABSTRACT (57) A surveillance system is provided that includes at least one sensor disposed in a security area of a surveillance region to sense an occurrence of a potential security breach event; a plurality of cameras is disposed in the surveillance region; at least one camera of the plurality has a view of the security area and can be configured to automatically gather biometric information concerning at least one subject person in the vicinity of the security area in response to the sensing of a potential security breach event; one or more other of the plurality of cameras can be configured to search for the at least one subject person; a processing system is programmed to produce a subject dossier corresponding to the at least one subject person to match biometric information of one or more persons captured by one or more of the other cameras with corresponding biometric information in the subject dossier. 2 Claims, 8 Drawing Sheets
Related U.S. Application Data (60) Provisional application No. 60/610,998, filed on Sep. 17, 2004. Int. CI. HO4N 9/47
(51)
(2006.01)
(52) (58)
GO6K 9/00 U.S. Cl. .........
(2006.01) 348/143; 382/115; 382/119; 382/118
Field of Classification Search .................. 713/186, 713/202, 320, 201, 200; 340/506; 358/143, 358/147, 161, 169; 707/103, 4; 382/103, 382/209, 276, 277, 289, 291, 293, 294, 295, 382/305, 282, 115, 107, 190, 116, 118, 119, 382/128; 345/629; 348/143, 151, 155, 161, 348/169; 709/227, 215, 203, 315 See application file for complete search history.
Wb
A. Fro ntmg B.S~drau C. Pfl OOve Stk 0. b torTmea
E. a
A. FIhgewit B.Eieadad Raw Feipip"lme
Pingrpinrt
Agent Facial Agent

HNormoffd MMge Face Nonna~zod poe Cuve Noimasned Cd= &Tae FcI snasue in MPEG-7Descrl IK0101 Flngeqla Proximox Knowledge Server
In PEG-? Desciptor
hparpz Signature
5744W eEncoiwo Protintex Knowledge Sae.s
US 7,956,890 B2
Page 2
U.S. PATENT DOCUMENTS

5,838,465 5,912,980 5,991,429 6,008,912 6,072,496 6,157,469 6,248,063 6,306,087 6,335,985 6,404,900 6,408,404 6,591,224 6,604,093 6,609,198 6,625,315 6,628,829 6,687,831 6,697,103 6,707,487 6,711,587 6,757,668 6,816,184 6,906,709 6,909,745 A
A
A A A A BI BI BI
BI B!
BI B! BI B2 BI BI * BI1 BI B2 BI
11/1998 Satou et al. 6/1999 Hunke 11/1999 Coffin et al ................... 12/1999 Sato et al. 6/2000 Guenter et al. 12/2000 Mestha 6/2001 Barnhill et al. 10/2001 Barnhill et al. 1/2002 Sambonsugi et al. 6/2002 Qian et al. 6/2002 Ladwig 7/2003 8/2003 8/2003 9/2003 9/2003 2/2004 2/2004 3/2004 3/2004 6/2004 11/2004 6/2005 6/2005 9/2005 11/2005 11/2005 11/2005 11/2005 12/2005 4/2006 4/2006 6/2006 10/2006 10/2006 11/2006 3/2007 3/2007 5/2007 7/2007 7/2007 9/2007 10/2007 10/2007
382/118
BI*
BI BI 6,944,319 B! 6,963,899 BI 6,968,006 BI 6,970,513 B1 6,970,582 B2 6,976,207 B1 7,023,913 BI 7,028,034 B2 7,058,204 B2 7,116,353 B2 7,127,464 B2 7,136,524 BI 7,187,783 B2 7,196,722 B2 7,213,174 B2 7,242,423 B2 7,242,810 B2 7,269,591 B2 7,277,485 BI 7,286,687 B2 7,295,228 B2 7,319,796 Bl 7,366,174 B2 7,376,276 B2 * 7,382,249 B2 7,384,590 B2 7,385,590 B2 7,456,727 B2 7,522,186 B2 7,525,576 B2 7,602,947 BI * 7,746,380 B2 7,777,783 BI 2001/0048765 Al 2002/0097322 Al 2002/0138768 Al 2002/0190119 Al 2003/0074590 Al* 2003/0107653 Al 2003/0115474 Al * 2003/0202 102 Al 2004/0001149 Al 2004/0022442 Al 2004/0059953 Al * 2004/0081338 Al 2004/0100563 Al 2004/0117638 Al* 2004/0136574 Al 2005/0052532 Al 2005/0100209 Al 2005/0105764 Al* 2005/0132414 Al 2005/0222081 Al 2005/0265607 Al
11/2007
1/2008 4/2008 5/2008 6/2008 6/2008 6/2008 11/2008 4/2009 4/2009 10/2009 6/2010 8/2010 12/2001 7/2002 9/2002 12/2002 4/2003 6/2003 6/2003 10/2003 1/2004 2/2004 3/2004 4/2004 5/2004 6/2004 7/2004 3/2005 5/2005 5/2005 6/2005 10/2005 12/2005
Sullivan et al. Etzion et al. Wood et al. Laumeyer et al. Chasen Albaugh et al ................... 726/7 Fernandez et al. Aman et al. Dufaux Goebel et al. Brill et al..................... 348/143 Larkin etal. Puri et al. Huang et al. Fernandez et al. Puri et al. Puri et al. Langley RujAn et al. Monroe et al. Wesinger et al. Hildreth et al. Hobson et al. Wesinger et al. Goh et al. Moon et al. White et al. Dahlquist et al. Lin Chang Wesinger et al. Puri Lindwurm et al. Roberts et al. Sharp MacFaden et al. Shniberg et al. ............. 382/224 Fancella Kelly et al. Millar et al. Pinter et al. Arpa et al. Kannermark et al. Lemelson et al ............. 382/116 Maruya etal. Chin etal. Yi et al. Monroe et al. Murakami et al. Huffman Fogle et al. .................. 713/320 Utsumi et al.. Khan et al.......... 713/186 Shiota et al. Smith et al. Kim et al. Purnell .................. .... 713/202 T'akenaka Sablak et al. M onroe ....................... 713/186 Kozakaya et al. Elooz et al. Lewis et al. Han et al................... 382/100 Bentley et al. Jakobson et al. Chang
2006/0017807 2006/0112039 2006/0203090 2006/0221184 2006/0222209 2006/0279628 2006/0284978 2007/01!46484 2007/0154088 2007/0186116 2008/0043102 2008/0068464 2008/0079554 2008/0088706 2008/0106597 2008/0130949 2008/0218590 2008/0294588 2009/0322873 2010/0002082
Al Al Al Al Al Al Al Al Al
1/2006 5/2006 9/2006 10/2006 10/2006
12/2006 12/2006 6/2007 7/2007 Al * 8/2007 Al * 2/2008 Al 3/2008 Al 4/2008 Al 4/2008 Al 5/2008 Al 6/2008 Al 9/2008 Al 11/2008
Al*
Al
12/2009
1/2010
Lee et al. Wang et al. Wang et al. Vallone et al. Zhang et al.................. 382/107 Fleming Girgensohn et al. Horton et al. Goh et al. Clemmensen et al. ....... 713/186 Rowe ct al ................... 348/143 Kitagawa et al. Boice Girgensohn et al. Amini et al. Ivanov et al. Park et al. Morris et al. Reilly et al .................. 348/143 Buehler et al.
FOREIGN PATENT DOCUMENTS WO 2007/044037 Al 4/2007
OTHER PUBLICATIONS Brunelli, R. and D. Falavigna. (1995). "Person identification 'using multiple cues," IEEE Transactions on Pattern Analysis and Machine Intelligence 17(10): 955-966. Brunelli, R. et al. (1995). "Automatic Person Recognition by Using Acoustic and Geometric Features", Machine Vision andApplications 8: 317-325. Hong, Lin and Anil K. Jain. (1998). "Integrating faces and fingerprints for personal identification," IEEE Transactions on Pattern Analysis and Machine Intelligence 20(12): 1295-1307. International Search Report mailed on Apr. 2006 for PCT Patent Application No. PCT/US05/33378 filed on Sep.,19, 2005, one page. Jain, A. K. et al. (1997). "On-Line Fingerprint Verification," IEEE Transactions on Pattern Analysis and Machine Intelligence archive 19(4): 302-314. Kittler, J. et al. (1998). "On combining classifiers", IEEE Transactions on Pattern Analysis and Machine Intelligence 20(3): 226-239. Lu X et al. (2003). "Combing classifiers for face recognition", IEEE International Conference on Multimedia Systems and Expo, Baltimore, MD, July. Maio, D. et al. (2002). "FVC2000: fingerprint verification competition", IEEE Transactions on Pattern Analysis and Machine Intelligence 24(3): 402-412. Phillips, P.J. et al. (2000). "The FERET evaluation methodology for face-recognition algorithms", IEEE Transactions on Pattern Analysis and Machine Intelligence 22(10): 1090-1104. Senior, A. (2001). "A combination fingerprint classifier", IEEE Transactions on Pattern Analysis and Machine Intelligence 23(10): 1165 1174. Turk, A. and A. Pentland. (1991). "Eigenfaces for Recognition". Journal of Cognitive Neuroscience 3 (1): 71-86. U.S. Appl. No. 12/838,973, Hu Chin, Multi-Video Navigation, filed Jul. 19, 2010. U.S. Appl. No. I11/728,404, Hu Chin, Multi-Video Navigation System, filed Mar. 23, 2007. PCT/US05/44656 International Search Report and Written Opinion, Jun. 26, 2006. PCT/US05/43808 International Search Report and Written Opinion, Oct. 10, 2007. PCT/US05/33378 International Search Report and Written Opinion, Apr. 26, 2006. PCT/US05/33750 International Search Report and Written Opinion, May 2, 2007. PCT/US05/16961 International Search Report and Written Opinion, Oct. 17, 2006. VidSys, Inc. Complaint filed in the US District Court for the Eastern district of Virginia on Oct. 19, 2010. Redstone Integrated Solutions Documents, 2003. U.S. Appl. No. 95/001,525 Reexamination Request for 7,777,783, filed Jan. 21, 2011. * cited by examiner
U. S.
Patent
.Jun.
7, 2011
Sheet 1 of 8
US 7,956,890 B2
ci
.1.1
0
LL-
a
L 4w
Q.
W) CU (/)
U. S.
Patent
Jun. 7, 2011
Sheet 2 of 8
US 7,956,890 B2
'. r
E
"a
E
0 O
LL
w
ml
aI
U.S. Patent
Jun. 7, 2011
Sheet 3 of 8
US 7,956,890 B2
L 1...
C')
M~
0 0
0)
C
0.
Q
U. S.
Patent
Jun. 7, 2011
Sheet 4 of 8
US 7,956,890 B2
cL
IKawch
ac
4T
I
c
4 t
*Iii
f-T
1
C
iii
'-1;f
III
I
I1
m1
ti
m a
q,
LIL
f"Ma
U.S. Patent
Jun. 7, 2011
Sheet 5 of 8
US 7,956,890 B2
i I II
1)
iL.
L.
a)
0
... ... .. .. ... .. .. .. ... .. .. ... .. .. ... .. .. . .. ..
U.S. Patent
Jun. 7, 2011
Sheet 6 of 8
US 7,956,890 B2
liii LI,
ti
MisLsIl
,SO A
V~
'ci
i (a)
,w
'
m,
00FF cs
. """.v y
M-
-~i
10-
zz
U. S.
Patent
Jun.
7, 2011.
Sheet 7 of 8
US 7,956,890 B2
I2
I-
CD
0 0Q O
tm fz
gv
ti
W.~
4G
tic C
-- r
U.S. Patent
Jun. 7, 2011
Sheet
8 'of
US 7,956,890 B2
V.
x M
N1
d
m
r
OA
LD
X1.
u..
s
1 W r fi RI
II
G
au
OL .
." r
oA O
el
t0
ML
ro
uw
Cool
400 7: ^ towo
d
/ r
LL
'
s A n n
n
r IA r Ki
fil
f rffiIfli1"14
C
I
p
C3
I1@liliEIfll
C
M r
ti8tt1t1811111 illtilut 11 ltII1lIlulsuluI
silo%
n K
&woo
h
y
P i H
~1
L U.d z
_C
n
US 7,956,890 B2
ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEMS CROSS-REFERENCE TO RELATED APPLICATION The present application claims the benefit of earlier filed provisional patent application, U.S. Application No. 60/610, 998, filed on Sep. 17, 2004, and entitled "Adaptive MultiModal Integrated Biometric Identification Detection Systems," which is hereby incorporated by reference as if fully set forth herein. B3ACKGROUND OF T"I'HE INVENTION 1. Field of the Invention The invention relates in general to biometric identification, and more particularly, to a surveillance system using biometric identification. 2. Brief Description of the Related Art The state of the art of applying biometric technologies to authenticate and positively determine the identity of a person is still faced with several technical challenges. Specifically, the challenges can be categories into two aspects: data acquisition and data matching. Data acquisition deals with acquiring biometric data from individuals. Data matching deals with matching biometric data both quickly and accurately. These challenges can beexplained by a port-entry scenario. In such a setting, it is difficult to obtain certain biometric data such as DNA and voice samples of individuals. For biometric data that can be more easily acquired, such as face images and fingerprints, the acquired data quality can vary greatly depending on acquisition devices, environmental factors (e.g., lighting condition), and individual corporation. Tradeoffs exist between intrusiveness of data collection, data collection speed, and data quality. Once after the needed data have been acquired, conducting matching in a very large database can be very time-consuming. It goes without saying that unless a system can acquire and match data both timely and accurately, the system is practically useless in improving public security, where the inconvenience due to the intrusive data-acquisition process and the time-consuming matching process ought to be minimized. A biometric system typically aims to address either one of the following issues: 1) Authentication: is the person the one he/she claims to be? 2) Recognition: who a person is? In the first case, data acquisition is voluntary and matching is done in a one-to-one fashion-matching the acquired data with the data stored on an ID card or in a database. In the second case, individuals may not be cooperating, and the system must conduct searches in a very large repository. The prior art in biometric can be discussed in two parts: single-modal solutions and multi-modal solutions. Several systems have been built to use one of the following single modal: facial data, voice, fingerprint, iris or DNA. The effectiveness of these single-modal approaches can be evaluated in three metrics: the degree of intrusiveness, speed and accuracy. From the perspective of a user, acquiring face modal can be the most noninvasive method, when video cameras are mounted in the distance. However, the same convenience nature often compromises data quality. An intrusive face acquisition method is to acquire frontal face features, which requires corporation from individuals. Voice is another popular modal. However, traditional voice-recognition fails miserable when voice samples of multiple individuals are simul[ taneously captured or when background noise exists. Even when the acquired voice data can be "pure," existing signal processing and matching techniques can hardly achieve recognition accuracy of more than 50%. The next popular modal 5 is fingerprint, which can achieve much higher recognition accuracy at the expense of intrusive data acquisition and time-consuming data matching. Finally, DNA is by far the most accurate recognition technique, and the accompanying inconvenience in data acquisition and the computational 10 complexity are both exceedingly high. Summarizing the single model approach, non-intrusive data-acquisition techniques tend to suffer from low recognition accuracy, and intrusive data-acquisition techniques tend to suffer from long computational time 15 As to multimodal techniques, there have been several prior art United States Patents and Patent Applications disclose techniques. However, as will be further discussed below, these disclosures do not provide scalable means to deal with tradeoffs between non-intrusiveness, speed and accuracy 20 requirements. These disclosures may fix their system configuration for a particular application, and cannot adapt to queries of different requirements and of different applications. Wood et al. disclose in U.S. Pat. No. 6,609,198 a security 25 architecture using the information provided in a single signon in multiple information resources. Instead of using a single authentication scheme for all information resources, the security architecture associates trust-level requirements with information resources. Authentication schemes (e.g., those 30 based on passwords, certificates, biometric techniques, smart cards, etc.) are employed depending on the trust-level requirement(s) of an information resource (or information resources) to be accessed. Once credentials have been obtained for an entity and the entity has been authenticated to 35 a given trust level, access is granted, without the need for Sfurther credentials and authentication, to information resources for which the authenticated trust level is sufficient. The security architecture also allows upgrade of credentials for a given session. The credential levels and upgrade scheme 40 may be useful for a log-on session; however, such architecture and method of operations do not provide a resolution for high speed and high accuracy applications such as passenger security check in an airport. Sullivan et al. discloseinU.S. Pat. No. 6,591,224 a method 45 and apparatus for providing a standardized measure of accuracy of each biometric device in a biometric identity authentication system having multiple users. A statistical database includes continually updated values of false acceptance rate and false rejection rate for each combination of user, biomet50 ric device and biometric device comparison score. False ' acceptance rate data are accumulated each time a user successfully accesses the system, by comparing the user's currently obtained biometric data with stored templates of all other users of the same device. Each user is treated as an 55 "impostor" with respect to the other users, and the probability of an impostor's obtaining each possible comparison score is computed with accumulated data each time a successful access is made to the system. The statistical database also contains a false rejection rate, accumulated during .a test 60 phase, for each combination of user, biometric device and biometric device comparison score. By utilizing a biometric score normalizer, Sullivan's method and apparatus may be useful for improving the accuracy of a biometric device through acquiring more training data. 65 Murakami et al. disclose is a Patent Publication 20,020, 138,768 entitled "Method for biometric authentication through layering biometric traits," a portable biometric
US 7,95
3
authentication system having a single technology for measuring multiple, varied biological traits to provide individual authentication based on a combination of biological traits. At least one of these biometric traits is a live physiological trait, such as a heartbeat waveform, that is substantially-but not necessarily completely unique to the population of individuals. Preferably, at least one of the identifying aspects of the biological traits is derived from a measurement taken by reflecting light off the subdermal layers of skin tissue. The Murakami et al. approach is limited by the more intrusive measurement techniques to obtain data such as heartbeat waveform and reflecting light off the subdermal layers of skin tissue. These data are not immediately available in a typical security check situation to compare with the biometric data, e.g., heart beat waveforms and reflection light from subdermal layers from the skin of a targeted searching object. Furthermore, the determination or the filtering of persons' identity may be too time consuming and neither appropriate for nor adaptive to real time applications. Langley discloses in US Patent Application 20,020,126, 881, entitled "Method and system for identity verification using multiple simultaneously scanned biometric images," a method to improve accuracy and speed of biometric identity verification process by use of multiple simultaneous scans of biometric features of a user, such as multiple fingerprints, using multiple scanners of smaller size than would be needed to accommodate all of the fingerprints in a single scanner, and using multiple parallel processors, or a single higher speed processor, to process the fingerprint data more efficiently. Obtaining biometric data from multiple user features by use of multiple scanners increases verification accuracy, but without the higher cost and slower processing speed that would be incurred if a single large scanner were to be used for improved accuracy. The methods according to Langley may provide the advantages of speed and accuracy improvements. However, the nature of requiring multiple scans makes data acquisition time-consuming and intrusive. On the academia side, much research effort has been geared toward analyzing data from individual biometric channels (e.g., voice, face, fingerprint, please see the reference list for a partial list), less emphasis has been placed on comparing the performance of different approaches or combing information from multiple biometric channels to improve identification. Some notable exceptions are discussed below. In Hong Lin, JainA. K., Integrating faces and fingerprints for personal identification, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 20, No. 12, December 1998, pp. 1295-1307, the authors report an automated person identification system that combines face and fingerprint information. The face recognition method employed is the traditional eigen face approach, M. Turk and A. Pentland, Eigenfaces for Recognition, J. Cognitive Neuroscience Vol. 3, No. 1, 1991, pp. 71-96, which computes a set oforthonormal bases (eigen faces) of the database images using the principal component analysis. Face images are then approximated by their projection onto the orthonormal Eigen face bases, and compared using Euclidean distances. For fingerprint, the authors extend their previous work, Jain, A. K.; Lin Hong; Bolle, R.; On-line fingerprint verification, Pattern Analysis and Machine Intelligence, Vol. 19, No. 4, April 1997, pp. 302-314, to extract minutiaes from fingerprint images. They then align two fingerprint images by computing the transformation (translation and rotation) between them. Minutiaes are strung together into a string representation and a dynamic programming-based algorithm is used to compute the minimum edit distance between the two input fingerprint strings. Decision fusion is achieved by cross validation of the top matches identified by the two modules, with matching results weighed by their confidence or accuracy levels. The performance of the system is validated on a database of about 640 face and 640 fingerprint images. 5 In Phillips, Henson Moon; Rive, S E A.; Russ, The FERRET evaluation methodology for face-recognition algorithms, IEEE Transactions on Pattern Analysis and Machine SIntelligence, Vol. 22, No. 10, October 2000, pp. 1090-1104, the Michigan State University research group extends their 10 information fusion framework to include more modalities. In particular, images of a subject's right hand were captured, and fourteen features comprising the lengths of the fingers, widths of the fingers, and widths ofthe palm at various locations of the hand. Euclidean distance metric was used to compare 15 feature vectors. Simple sum rules, decision tree and linear discriminant function are used for classification. It is observed that a personal ID system using three modules outperforms that uses only two of the three modules. While this is an interesting experiment, the data set used is small and 20 there is no accepted universal standard in using hand images in biometrics. In R. Brunelli, D. Falavigna, T. Poggio and L. Stringa, Automatic Person Recognition by Using Acoustic and Geometric Features, Machine Vision and Applications 1995, Vol. 25 8 pp. 317-325, an automated person recognition system using voice and face signatures is presented. The speaker recognition subsystem utilizes acoustic parameters (log-energy outputs and their first-order time derivatives from 24 triangular band-pass filters) computed from the spectrum of short-time 30 Windows of the speech signal. The face recognition subsystem is based on geometric data represented by a vector describing discriminant facial features such as positions and widths of the nose and mouth, chin shape, thickness and shape of the eyebrows, etc. The system captures static images of the 35 test subjects and the test subjects are also asked to utter ten digits from zero to nine for use in the speaker ID subsystem. Each subsystem then computes the distances of the test subject's speech and face signatures with those stored in the databases. Decisions from the two ID modules are combined 40 by computing a joint matching score that is the sum of the two individual matching scores, weighted by the corresponding variance. Experimental results show that integration of visual and acoustic information enhances both performance and reliability of the separate systems. The above system was later 45 improved upon in Brunelli, R.; Falavigna, D., Person identification using multiple cues,. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 17, No. 10, October 1995, pp.955-966, where multiple classifiers are used in the face recognition subsystems, and the matching score normal5o ization process is made more robust using robust statistical methods. In Kittler, J.; Hatef, M.; Duin, R. P. W.; Matas, J., On combining classifiers, IEEE Transactions on PatternAnalysis and Machine Intelligence, Vol. 20, No. 3, March 1998, pp. 55 226-239, a performance study of various ensemble classification scheme is presented. It is shown that many existing decision aggregation rules are actually simplifications based on the more general Bayesian rule. The authors compare the performance of different decision aggregation rules (max, 60 min, median, and majority voting rule) by performing an . experiment in biometrics. Three modules are used: frontal faces, face profiles, and voiceprints. Simple correlation-based and distance-based matching is performed on frontal faces and face profiles, respectively, by finding a geometric trans65 formation that minimizes the differences in intensity. It is shown that a simple aggregation scheme by summing the results from individual classifiers actually perform the best.
US 7,956,890 B2 5
In Lu X; WangY; and Jain A, Combing classifiers for face recognition, IEEE International Conference on Multimedia Systems and Expo, Baltimore, Md., July 2003, three wellknown appearance-based face recognition methods, namely PCA, M. Turk and A. Pentland, Eigenfaces for Recognition, J. Cognitive Neuroscience Vol. 3, No. 1, 1991, pp. 71-96, ICA, and LDA, Belhumeur, P. N.; Hespanha, J. P.; Kriegman, D. J., Eigenfaces vs. Fisherfaces: recognition using class specific linear projection, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 19, No. 7, July 1997, pp. 711-720, are used for face image classification. Two combination strategies, the sum rule and RBF network, are used to integrate the outputs from these methods. Experimental results show that while individual methods achieve recognition rates between 80% and 88%, the ensemble classifier boosts the performance to 90%, using either the sum rule or RBF network. In Senior, A., A combination fingerprint classifier, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 23, No. 10, October 2001, pp. 1165-1174, a similar multi-classifier scheme, this time for fingerprint classification, is proposed. Hidden Markov Models and decision trees are used to recognize ridge structures of the fingerprint. The accuracy of the combination classifier is shown to be higher than that of two state-of-the-art systems tested under the same condition. These studies represent encouraging results that validate our multi-modal approach, though only a single biometric channel, either face or fingerprint, not a combination of biometric channels, is used in these studies. Maio, D.; Maltoni, D.; Cappelli, R.; Wayman, J. L.; Jain, A. K., FVC2000: fingerprint verification competition, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24, No.3 , March 2002, pp. 402-412, documents a fingerprint verification competition that was carried out in conjunction with the International Conference on Pattern Recognition (ICPR) in 2000 (a similar contest was held again in 2002). The aim is to take the first step towards the establishment of a common basis to better understand the state-of-theart and what can be expected from the fingerprint technology in the future. Over ten participants, including entries from both academia and industry, took part. Four different databases, two created with optical sensors, one with a capacitive sensor, and one synthesized, were used in the validation. Both the enrollment error (if a training image can be ingested into the database or not) and the matching error (ifa test image can be assigned the correct label or not) and the average time of enrollment and matching are documented. A study, that is similar in spirit but compares the performance of face recognition algorithms, is reported in Phillips, P. J.; Hyeonjoon Moon; Rizvi, S. A.; Rauss, P. J., The FERET evaluation methodology for face-recognition algorithms, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.22, No. 10, October 2000, pp. 1 0 9 0 - 1 104. A subset of the Feret database (a gallery of over 3000 images) was used in the study. Ten different algorithms, using a wide variety of techniques, such as PCA and Fischer discriminant, were tested. Cumulative matching scores as a function of matching ranks in the database are tabulated and used to compare the performance of different algorithms. This study was repeated three times, in August 1994, March 1995, and July 1996. What is significant about this study is that the performance of the face recognition algorithms improved over the three tests, while the test condition became more challenging (with increasingly more images in the test datasets). As can be seen from the above brief survey, multi-modal biometrics holds a lot of promise. It is likely that much more accurate classification results can be obtained by intelligently fusing the results from multiple biometric channels given performance requirements. While it is important to keep on improving the accuracy and applicability of individual biometric sensors and recognizers, the performance of a biometric system can be boosted significantly by judiciously and 5 intelligently employing and combining mtiltiple biometric channels. While there have seen significant research activities in single- and multi-channel biometry over the past decade, the state-of-the-art is still wanting in terms of speed and accuracy. to Therefore, a need still exists in the art to provide new and improved methods and system configurations to increase the speed and accuracy of biometric identity verification and determinations such that the above-mentioned difficulties and 15 limitations may be resolved. The present invention meets this need. SUMMARY OF THE INVENTION
20
25
30
35
40
45
50
One embodiment of the invention provides'a novel surveillance method. An event sensor such as, a camera, chemical sensor, motion detector, unauthorized door access sensor, for example, is disposed to sense an occurrence of a potential security breach event. A camera with a view of the area in which an event is sensed gathers biometric information concerning a subject person in the vicinity of the event at about the time the event is sensed. A subject dossier is produced containing biometric information relating to the subject person sensed by the camera with the view of the area. Biometric information ofpersons captured on one or more other surveillance cameras in the general vicinity of the event is matched against corresponding biometric information in the subject dossier. Another embodiment of the invention provides a new surveillance system.A sensor is disposed in a surveillance region to sense an occurrence of a security breach event. The system includes a plurality of cameras. At least one camera of the plurality has a view of the security area and can be configured to automatically gather biometric information concerning a subject person in the vicinity of an area where the event occurred in response to the sensing of the event. One or more of the plurality of cameras can be configured to search for the subject person. The surveillance system also includes a processing system which can be programmed to produce a subject dossier corresponding to the subject person. The processing system also can be programmed to match biometric information of one or more persons captured by one or more of the cameras with corresponding biometric information in the subject dossier. These and other features and advantages of the invention sill be apparent from the following description of embodiments thereof in conjunction with the drawings. BRIEF DESCRIPTION OF THE DRAWINGS
55
FIG. 1 is an illustrative showing a map of an airport passenger terminal and its immediate vicinity protected by a surveillance system of one embodiment of the invention and 60 also showing several pop-up views relating to event alerts in accordance With the embodiment. FIG. 2 is another view of the map of FIG. 1 showing zoom to detail maps of different portions of the overall passenger terminal map. 65 FIG. 3 is an illustrative drawing of example security areas within the surveillance region of FIGS. 1-2 outfitted with event sensors.
US 7,956,890 B2 7
estimated Sheight and weight. A subject dossier also may FIG. 4 is an illustrative block level hardware diagram of a surveillance system in accordance with an embodiment of the include te mporal information, such as walking speed or direction of tra vel. In addition, a subject dossier also may include invention, more pen manent information such as facial features, fingerFIG. 5 is an illustrative block diagram level drawing of a system architecture of an embodiment of the invention that 5 print, iris scan, voiceprint and DNA. Soft features may be o selected t be especially useful for relocating an individual incorporates the system hardware of FIG. 4. within th e surveillance region, especially in a crowd, for FIG. 6 is an illustrative flow diagram showing gathering example. For instance, it maybe relatively easy to identify and conversion of facial feature data to a facial feature signabased upon clothing color or estimated height and individua ls ture. FIG. 7 is an illustrative flow diagram showing gathering 10 weight. H[owever, soft features have the disadvantage of not being as r eliable or permanent over time. If a person takes off and conversion of fingerprint feature data to a fingerprint his jacket ,then an identifying color feature may be lost. If a signature. person sit s down, then it may become impossible to use height FIG. 8 is an illustrative flow diagram showing gathering information to pick that person out of a crowd. and weig ht and conversion of DNA data to a DNA signature. One System Ssensors continually monitor the surveillance embodiment of the invention may employ a DNA fingerprint 15 the region forr occurrence of one or more suspicious events. In for identification purposes. one embc odiment, the system directs a live video feed from DETAILED DESCRIPTION OF THE PREFERRED one ormo re cameras having the location ofan alert-triggering EMBODIMENT event in t heir field of view to a console in a manned control 20 center. Thhesystem also may direct video images captured just before theeevent to the control center console. Thus, an operaThe following description is presented to enable any person skilled in the art to make and use the invention, and is tor at the console can observe behavior of suspicious indiprovided in the context of particular applications and their viduals at the scene of the event in real time and immediately requirements. Various modifications to the preferred embodiprior to thf event. A subject dossier produced for individuals he ments will be readily apparent to those skilled in the art, and 25 at the sce ne of the event can be used to automatically identify the generic principles defined herein maybe applied to other and track a suspect individual present at the scene of the event embodiments and applications without departing from the within th esurveillance area after the occurrence of the event. spirit and scope of the invention. Moreover, in the following The sy stem may employ information in a subject dossier description, numerous details are set forth for the purpose of incremen tally. For instance, the system may prioritize inforexplanation. However, one of ordinary skill in the art will 30 mation in the subject dossier. Certain information in the subrealize that the invention can be practiced without the use of ject doss ier such as clothing color, estimated height and those specific details. In other instances, well-known strucweight, v,alking pattern or gait and certain key facial features such as faacial shape, facial hair, skin color, or hair color may tures and devices are shown in block diagram from in order to not, to obscure the description of the invention with unnecbe used t make an initial estimate of which persons in a essary detail. Thus, the present invention is not intended to be 35 camera's field of view are candidates for a match to a suspilimited to the embodiments shown, but is to be accorded the cious per son identified in response to an alert. Other features from a suabject dossier then may be added incrementally to widest scope consistent with the principles and features disclosed herein, make a m ore careful assessment of whether identified candidates actu ually match the suspect. Alternatively, as more inforSystem Overview 40 mation c oncemrning a suspicious person becomes available, additiona l features may be added incrementally to a suspect's One embodiment of the invention involves an intelligent subject d ossier for that person. This additional information surveillance system. A plurality of cameras, some with and then may Sbe used to more effectively locate and track the some without overlapping fields of view, are distributed individua l within the surveillance region. throughout a surveillance region. Intelligent computer soft- 45 ware based agents process information captured by one or. Surveillance Region more of the cameras to produce a subject dossier indicative of the identity of a person whose images have been captured by One er mbodiment of the invention is configured for use in one or more of the cameras. Information for a subject dossier airport se curity. In this embodiment, the surveillance region also may be gathered through other modalities such as voice 50 comprise s an airport passenger terminal and the surrounding recognition, iris scan, or fingerprint, for example. The system passengerr ground transport loading/unloading zone directly includes multiple event sensors, which may include the camoutside thheterminal and the aircraft parking area adjacent the eras, chemical sensors, infrared sensors, or other security terminal. FIG. 1 is an illustrative drawing of a map of an alarm sensors that trigger an alert, upon sensing an occurairport paassenger terminal and its immediate vicinity prorence of a predetermined category of event requiring height- 55 tected by Sa surveillance system of one embodiment of the ened vigilance. For example, an alarm may be triggered when invention .The system includes multiple cameras, each with a locked door is opened without proper access permission or an associ ated field of view, some of which are overlapping. when an unauthorized person enters a restricted area or when The surv eillance region has multiple areas including passena vehicle is parked in a restricted area. More specifically, a al ger arrivand departure areas, a passenger departure shops subject dossier is produced for individuals in the vicinity of 60 and a ten race. Groups of cameras with overlapping fields of the location of an alarm-triggering event. For instance, a view are, deployed to capture images within different regions subject dossier may be produced for persons captured in a of the passsenger arrival and passenger departure areas. video camera image at or near a door in the surveillance FIG. 2 is another view of the map of FIG. I showing zoom region at about the time when an unauthorized opening of the to detail maps of different portions of the overall passenger 65 terminal map. The illustrative maps of FIGS. 1-2 can be. door is detected by an event sensor. A subject dossier may include soft biometric information, displayec don a control terminal so that an operator can easily also referred to as "soft" features such as clothing color, correlate an alert to a specific area an airport surveillance
US 7,956,890 B2
10"
mented across'multiple hardware systems or as different proregion. For instance, if an alert is triggered in the arrivals cesses within a single hardware system. region shown in FIG. 1, then an operator may request the A security agent is a process that spans many tasks to left-most zoom shown in FIG. 2 in order to quickly picture the collect information about subject(s). For example, a security airport layout in the vicinity of the alert. Additional zoom maps (not shown) may be provided for numerous locations 5 agent may spawn multiple data collection agents include a facial features, fingerprint, DNA, clothing color, subject gait, such as security gates, check-in counters, airport fairway, subject height and weight, skin color/tone, hair color/tone, parking area, access entrance, check-in counters, etc. Each subject direction and voiceprint, for example. Each data coldifferent area may be associated with a group of cameras and lection. task produces different information about an indievent sensors. Event sensors are disposed at selected locations within the 10 vidual. More specifically, each produces a signature indicative of some identifying aspect of a person under surveillance. surveillance region. FIG. 3 is an illustrative drawing of example security areas within the surveillance region of For instance, a facial features agent uses facial information captured by one or more cameras to produce a signature FIGS. 1-2 outfitted with event sensors. A first security area comprises a door. The door may be equipped with a sensor, 15 indicative of an individual's facial features. 'Similarly, for example, a clothing color agent uses clothing color informasuch as a mechanical sensor, that detects unauthorized opention captured by one or more cameras to produce a signature ing of the door. A second security area comprises a window. indicative of the color of an individual's clothing color. Thus, The window may be associated with a mechanical sensor that detects when the window has been broken. A third security the multiple agents can produce multiple different signatures, represents a threshold to a restricted area. The restricted area 20 each indicative of one or more different identifying feature of may be equipped with motion detectors that detect the presan individual. ence of persons in a restricted area. Cameras situated throughThe agents provide the signatures to the knowledge server, out the surveillance region also may serve as event sensors. which aggregates signatures for each given person under For example, the system may employ a monitoring rule surveillance into a subject dossier for that person. The knowlwhereby a camera monitors a particulararea of the passenger 25 edge server indexes the signatures within a given subject terminal. Ifa person is loitering in that area, defined by failing dossier to permit incremental searches for individuals within to move beyond a 15 foot radius for more than 60 seconds, the search region. The knowledge server also may perform then a low level alert is declared, the camera zooms in, and the classification and matching. The local knowledge server dataface of the loitering person is matched against the faces of base stores the digital signatures and corresponding indexing 30 information. persons on a watch list, for example. Landmarks are defined in the security areas for purpose of The web services is the component that provides the interestimating height and weight and direction and speed of travel faces via Web Server which is usually part of an operating of a suspect individual. For instance, a landmark such as a system. For example, web services provides the interfaces for countertop may be identified, and processing of a camera our internal components or external systems via Web Server image may be calibrated to estimate a person's height relative 35 (such as Microsoft IIS on Windows, orApache on Linux). All to the land marked countertop. A group of multiple structures, the interfaces to the system are via HTTP or HTTPS using such as telephone booths, lounge areas, signs or countertops, port 80. Doing so, our system can run across firewall. Basiwithin a field of view of one or more of a group of cameras cally, the Web Services componentjust exposes our system covering a security area may be identified. Processing of group of cameras may be used to 40 interface to the outside world via Web Server. camera images from the The application server is the component that provides that estimate the direction and speed at which a suspect is moving database access to the user interface component, and perbased upon the sequence and timing of his passing the land forms session management which includes authentication marked structures. and authorization. The middle-tier database serves as the Although the surveillance region in this one example is described in terms of an airport passenger terminal, it will be 45 local database for the application server. FIG. 5 is an illustrative block diagram level drawing of a appreciated that the invention is not restricted to an airport system architecture of an embodiment of the invention that terminal. Moreover, the surveillance region need not be a incorporates the system hardware of FIG. 4. A user.interface continuous local area. Event sensors and surveillance cam(UI) provides an operator of the system with real-time.inforeras may be disposed over disparate areas and be in communication with a control center via a network such as the 50 mation concerning alert events within the surveillance region. The UI may provide maps of the entire surveillance region, internet, for example. including zoom maps. It can display alerts from different System Architecture sensors including cameras, digital video recorders, access control, bio-chemical detectors, etc. It may display videos of FIG. 4 is an illustrative block level hardware diagram of a 55 a security area in which an alert has been triggered, detailed surveillance system in accordance with an embodiment of the images of suspect individuals and details ofone ormore alerts invention. The system includes multiple data collection that have been triggered. agents, a knowledge server, a local knowledge server dataReferring again to FIG. 1, there is show an example of a UI base, an application server, a middle-tier database, web servdisplay screen in with pop-up display showing various ers, a browser based control console and one or more client 60 images relating to one or more alerts. In the center of the applications such as ComputerAided Dispatch system, buildscreen is map of a surveillance region. The operator can be ing management system, access control system, etc. It should selectively enlarge, minimize or close each pop-up. A Video be understood that the various components shown are merely Review display provides a video image of the security region illustrative. Each agent may gather information from numerat about the time of an alert. An incident Detection display ous sources, such as the cameras shown in FIG. 1, distributed 65 provides detailed information concerning an alert event. In throughout a surveillance region. Moreover, for example, the this example, the alert event involved an individual tailgating knowledge server and the application server can be impleat a commuter door. A Suspect Description display provides
US 7,956,890 B2
identifying information concerning an individual under surveillance based upon information gathered into a subject dossier produced for the person. A Detailed Images display provides pictures of a suspect individual captured by one or more surveillance cameras. A Potential Identification display 5 provides images of the suspect together with images of one or more people whose facial features closely match those of the suspect. The potential matches are based upon a facial feature signature provided by the facial feature agent. Across the 10 bottom of the map, there is a chart listing briefly summarizing multiple alert situations. The operator may selectively access pop-up screens for these alert situation. Thus, the UI advantageously displays a variety of information aggregated in response to one or more alerts. In a typical 15 airport security region, for example, there may be several hundred cameras dispersed throughout a large physical area. Moreover, there may be only a few operators monitoring one or more UI consoles. Depending upon the rules for monitoring and declaring alerts, alerts may occur frequently or infre- 20 quently. The UI of one embodiment of the invention directs an operator to areas of a surveillance region that are subject to alert and provides pertinent information concerning the alert so that the operator can efficiently manage security from a 25 control center. The UI also allows an operator to quickly investigate and simultaneously keep abreast of multiple alert events.
Rules:
Actor Person Person
Action Walk though lane against direction of traffic Tailgating
Person
Person Person Person Vehicle Vehicle
Loitering
Piggyback Traveler screening Walk in restricted area Park overtime Park in restricted area
The Person-Loitering rule involves the following criteria: Radius Duration Alert Severity Response 15 foot 20 seconds Low Zoom in face to match "watch list"
The Person-Tailgating Rule involves the following criteria: Alert Severity Response Low Zoom in face to match watch list"
Furthermore, as explained more fully below, information from different sensing devices is correlated to facilitate track- 30 ing of a suspect within a security region. For instance, soft biometric information and temporal information is used to The correlation Monitoring Rule for the occurrence of a Person-Loitering event AND a Person-Tailgating event locate a suspect as he or she travels within the security region. involving the same person is as follows: In one embodiment, a dashed line can be produced on a map onthe display showing a path followed by a suspect within the 35 Alert Severity Critical surveillance region. Information from different data collecAction Acknowledge Loitering and Tailgating alerts and deliver tion agents may be fused in order to more accurately identify alarm to operator console and track an individual. Therefore, the operator can use the UI to evaluate an alert event, to identify and track a suspect. The 40 As described above the UI, may display several categories operator may use this information as a basis to send informaof information concerning an alert. The Knowledge Service tion to a responder to intercede or deal with an alert incident. and the Rules and Agent Manager provide the correlation Knowledge Services are implemented as an application between events and data sources and subject dossiers that running on the knowledge server. Knowledge Services corpermit an operator to view a map of the location of an alert, relate and analyze signature information provided by differ- 45 soft-biometric data of a suspect and video playback, for ent sensory devices (i.e., data gathering agents). The Knowlexample. More particularly, these components provide a link edge Services assemble and index subject dossiers, and when to a map and zoom stored in the middle tier database, link to appropriate, fuse signature information for improved classivideo feeds for video view real-time monitoring or playback fication results. The Knowledge Services also generate, acti5 ofrecordedvideo clips and stored in a Digital Video Recorder vate or deactivate rules and send/control rules and instruction system and provide the subject dossier information. to the Rules and Agent Manager. The Middle Tier Data Access runs on the application The Rules and Agent Manager also is implemented on the server. It controls the database including functions such as knowledge server. The Rules andAgent Manager manages all query, add, delete, index. Indexing biometric signatures and other agents and manages rules that can be sent to each agent. 55 updating subject dossiers are done by this component. It correlates information from agents. It can also escalate an A (Security) Agent is implemented as an application runalert if the alert is not acknowledged by an operator within a ning on the knowledge server that controls and manages the given timeframe and/or similar alerts happen repeatedly data gathering sensors. In the case of cameras or DVRs, it can within a given time span (e.g. within 2 hours). Both the also perform video analytic using Computer Vision technolKnowledge Service and the Rules and Agent Manager arethe 60 ogy. Those tasks include background subtraction, image staprimary components for aggregating, categorizing biometric bilization, object detection, object classification, object tracksignatures which are parts of object dossiers. It also performs ing, and object identification. It can also control the other tasks such as task assignment/tracking, load balancing movement of Pan/Tilt/Zoom (PTZ) cameras, manage areas of tasks among agents, and interacting with data access compointerest within the field of view of the camera (called Mouse/ nents. 65 Man Trap), and collect video streams from DVR or cameras. The following are examples of rules that may be impleIt also has a scheduler that controls when rules or video mented by the system. analytic are performed.
US 7,956,890 B2
A Sensory Device Directory Access and Video Server is implemented as an application that has access to the knowledge server manages and provides information regardirig sensor devices or other subsystems. Basically, it is a software layer that enables the overall system to handle different makes/models of sensor devices. The Web Services is the component provided by operating systems or web servers. It manages other components, spawns or deletes services as necessary. It can also listen to messages from other systems. The Web Services provides interfaces to the system via Web Services running as part of a Web Server. The system provides a library resided on a specific directory, and the Web Server (which is usually part of the operating system) will use it to interpret interface requests to our system. Tracking, Facial Recognition, Fingerprint recognition, and other biometric identification are done at the (Security) agents. Biometric signatures are collected and generated at the agents, and sent to the Rules-and-Agent Manger. The Knowledge Services and the Rule-and-Agent Manager collectively collect biometric signatures and object tracking locations, and then generate and manage subject dossiers. A described above, a subject dossier includes information about object (e.g., person) such as, biometric information/signatures, soft biometric information (hair color, skin tone/color, weight or build, height, etc.) and other temporal information (e.g., speed, direction, location, past activities, information that the operator is looking for, etc.). Data fusion is performed by the Knowledge Services and the Rules and Agent Manager. Data required or generated by each of the components are saved and retrievable via the Middle-tier/Data Access component, which in turn utilizes a relational database such as Microsoft SQL Server. such as clothing color, height and weight may be employed to select candidates for further investigation. For example, the Knowledge Services may be programmed to cause the Security Agents to search for a match between clothing color in a 5 subject dossier of a suspect and clothing color of unidentified persons in a surveillance region. If a match is found, then the Knowledge Service may cause the Security Agents to perform an analysis of whether facial features in the subject dossier match facial features of the person with matching 10to clothing. Moreover, temporal information provided in a color subject dossier such as direction and speed of travel of a suspect may trigger the Knowledge Services to alert only certain sensory devices, such as a group of cameras in an area of the surveillance region where the suspect is headed, to be 15 on the lookout for the suspect. A subject dossier may be incremented as more information concerning a suspect is gathered. For example, initially, only soft biometric information such as clothing color and estimated height and weight might be available. Subsequently, 20 more information such as a facial feature signature or a voice print may become available and will be added to the subject dossier. Newly received data from these multiple sources may be fused with previous data by the Knowledge Services as it is received. 25 A subject dossier is a record stored in a computer readable medium that can be easily accessed by security agents and a console operator. The dossier is structured to separate soft biometric information and temporal data from other biometric information. Soft biometric and temporal information 30 generally can be characterized as being easier to obtain and useful for tracking purpose, but not very reliable for definitive identification purposes. Other biometric information, such as fingerprints, voiceprints and an iris scan are more reliable, but more difficult to obtain. Thus, soft biometric and temporal 35 data can be used advantageously to track an individual until Subject Dossier more reliable information, such as detailed facial features or fingerprints can be obtained to provide a more reliable idenData gathering agents collect data concerning a subject tification. person from different sources. The Knowledge Services aggregate the data into subject dossier. The data aggregated Data Gathering Agents into a given dossier may include different digital signatures 40 produced by different data gathering agents. A subject dossier The surveillance system of one embodiment employs mulalso may include fused data signatures produced by the fusion tiple streams of data including one or more of, facial features, of data gathered from multiple data sources having different vocal, fingerprint, iris scan, DNA data, soft biometric data, data modalities. The following is an example of information in a subject 45 temporal data and fused data. FIG. 6 is an illustrative flow diagram showing gathering dossier. and conversion of facial feature data to a facial feature signature. Facial feature A comprises a front image of a face that is segmented into a plurality of local areas as an irreducible set Subject Dossier: 50 of image building elements to extract a set of local features that can be mapped into a mathematical formula. Facial feaFacial Features Signature (e.g., nose shape and size, face width, ture B comprises is a side image that is also separated into a distance between eye corners, skin color (light, medium, dark), nose angle (profile view) set of irreducible image building elements for extracting local Soft Biometrics Signature (e.g., clothing color, height, weight) features. Facial feature C comprises a side profile curve that is Temporal Information Signature (e.g., direction of travel, speed, 55 also collected for use in the identity check and authentication past places visited/path) processes. Facial features D and E comprise skin color and Fingerprint Signature Voice Print Signature tone and hair color. These facial feature data are collected Iris Scan Signature from several video key frames taken from a parallax camera. DNA Analysis Signature These facial feature data are used to produce a facial fea60 tures signature. In one embodiment, the Knowledge Services The information in a subject dossier is indexed so that it can which applies an MPEG-7 descriptor, e.g., a facial recognibe used to more efficiently identify and track suspects and to tion descriptor, representing a projection of a face vector onto avoid false alarms. More particularly, a dossier is indexed so a set of basis vectors that span the space of possible face that certain information such as soft biometrics can be used to vectors and the projection of the face from a side view defined screen candidates within a surveillance for closer study and 65 by a profile curve. The face recognition feature sets are also to predict likely places within a surveillance region to extracted from a normalized face image and a normalized look for a suspect. For instance, soft biometric information profile curve. The normalized face image includes 56 lines
US 7,956,890 B2
15
16
with 46 intensity values in each line. The centers of the two GeneChip, Verigene ID, traditional PCR, or Forensic STR eyes in each face image are located on the 2 4 h row and the Analysis methods. The result signature will be matched with 16"' and 31" column for the right and left eye respectively. the signatures in the database. This normalized image is then used to extract the one dimenFIG. 8 illustrates genomic barcodes based on a standard sional face vector that includes the luminance pixel values 5 Universal Product Codes for identifying retailed products by from the normalized face image arranged into a one dimenemploying ten alternate numerals at eleven positions to gensional vector using a raster scan starting at the top-left corner erate one hundred billion unique identifiers. One embodiment of the image and finishing at the bottom right corner of the of the invention applies the barcode techniques for DNA image. The face recognition feature set is then calculated by fingerprint identification process. Special considerations are projecting the one-dimensional face vector onto the space 10 focused on the facts that the repeat polymorphisms are found defined by a set of basis vectors. By using the front image, the mainly in intergenic (nongene) regions of chromosomes, side image, the profile curve, the skin color and tone and the especially near the centromeres and that the polymorphisms hair color, the accuracy of identity authentication is signifialways exist in a pair in this case, one from each cop of cantly improved. 15chromosome 1. At a polymorphic locus (location), different A voiceprint signature also can be produced for identity numbers of a repeated unit create different alleles. Furthercheck and authentication over a telephone, for example. A more, repeated sequences of 9-80 nucleotides are referred to voiceprint is particularly useful because it is totally noninvaas Variable Number Tandem Repeats (VNTRs). This VNTR sive. In one embodiment, a multi-dimensional voice identifiShas a 16 nucleotide repeat. Repeated sequences of 2 to 8 cation process may be employed to generate a speaker's voice 20 nucleotides are referred to as Short Tandem. Repeats (STRs). signature by processing pitch contour vectors, time signature, This STR has four nucleotide repeat. In a general genomic beat number vector and voice shape defined by audio wavebarcode system, huge number of string of sites are generated forms of the speaker. For example, one embodiment applies with four alternate nucleotides, i.e., adenine, guanine, pitch models for different pitch intervals, which are defined to cytosine, thymine, at each position. A survey of just fifteen of be the difference between the semitones of two adjacent 25 these nucleotide positions would create a possibility of 415, nodes: i.e., one billion codes. In the present invention, only fourteen STRs and types are employed to generate barcodes that are Pitch Interval=[(log(current pitch)-log(previous easier to analyze with much smaller amount of data to process pitch)]/log 21112 and that can be more conveniently searched with existing FIG. 7 is an illustrative flow diagram showing gathering 30 search engine, e.g., Google search engine. and conversion of fingerprint feature data to a fingerprint Soft biometric information, such as clothing' color may be signature. A raw image of a fingerprint is converted into a set captured using cameras calibrated in accordance with a proof fingerprint codes. The set of codes has a more compact cess disclosed in commonly assigned co-pending U.S. patent format, e.g., IKENDI Fingerprint Pattern Format, which is based on encoding the friction ridges into a set of direction 35 application Ser. No. Not Yet Known, filed Sep. 16, 2005, entitled "Robust Perceptual Color Identification," invented by codes. The coded fingerprint is converted to fingerprint sigK. Goh, E. Y. Chang and Y. F Wang, which is expressly nature in an MPEG-7 descriptor. incorporated by reference in its entirety into this application FIG. 8 is an illustrative flow diagram showing gathering through this reference. This patent application addresses a and conversion of DNA data to a DNA signature. One embodiment of the invention may employ a DNA fingerprint 40 problem of camera-based sensors perceiving an article of clothing as having a slightly different color when viewed for identification purposes. A complete DNA profile includes 13 short tandem repeats (STRs) with repeats of four or five from different angles or under different lighting conditions. nucleotides in addition to a sex marker. Each STR has various The patent application proposes the representing color of an expected length and is located on different chromosomes or article of clothing using a "robust perceptual color". different ends of the same chromosome and each is indepen- 45 Data from different modalities may be fused by the Knowldently inherited. FIG. 8 show respectively the human chroedge Services for classification and identification purposes mosomes with STR names and locations and three or four without suffering the "curse of dimensionality using techdifferent polymorphisms labeled with each of four fluoresniques taught in commonly assigned co-pending U.S. patent cent dyes. The DNAs of different lengths are separated by gel application Ser. No. 11/129,090, filed May 13, 2005, entitled, electrophoresis. Since it is desirable to detect all different 50 Multimodal High-Dimensional Data Fusion for ClassificaDNAs in one signal identification process, different colors'of tion and Identification, invented by E. Y. Chang, which is dyes are used to mark different DNAs that have same length. expressly incorporated herein in its entirety by this reference. Appropriate dyes are employed in a PCR operation with STR SData may be incrementally added to a classification and idenprimers to separate the DNAs based on length and color to get tification process by the Knowledge Services using techaccurate DNA fingerprint in a single DNA identification pro- 55 niques taught by commonly assigned co-pending U.S. patent cess. The DNA profile signature is generated in the present application Ser. No. 11/230,932, filed Sep. 19, 2005, entitled, invention by using STRs and STR types, e.g., {STR Name, Incremental Data Fusion and Decision Making, invented by Type}, {STR Name, Type} where STR Names are {TPOX, Yuan-Fang Wang, which is expressly incorporated herein in DSS1358, FGA, D5S818, CSF1PO, D7S820, D8S1179, its entirety by this reference. TH01,VWA, D13S317, D16S539, D18S51, D21S11, SEX, 60 While the invention has been described with reference to etc.} Types are required to make sure other DNA sequences various illustrative features, aspects and embodiments, it will may use the repeat number of alleles instead of hetero/hobe appreciated that the invention is susceptible of various mozygous, e.g., {Heterozygous, Homozygous}. DNA modifications and other embodiments, other than those spesamples for identity check and authentication may include cifically shown and described. The invention is therefore to be hair, saliva, and blood. Samples are collected and their signa- 65 broadly construed as including all such alternative variations, tures are stored in a database. New sample can be collected modifications and other embodiments within the spirit and and analyzed (but not in real time) using DNA arrays/chips, scope as hereinafter claimed.
US 7,956,890 B2
What is claimed is: 1. A surveillance method comprising: using at least one event sensor disposed in a security area of a surveillanceregion to sense an occurrence of a potential security breach event; 5 using at least one camera with a view of the security area in which the event is sensed to gather biometric information concerning at least one person in the vicinity of the security area at about the time of the sensing of the event: producing a subject dossier corresponding to the at least 10o one person, the subject dossier including at least two biometric signatures; matching biometric information of one or more persons captured by one or more other cameras in the vicinity of 15 the at least one camera with corresponding biometric information in the subject dossier; and fusing the at least two signatures and including the fused signature in the subject dossier. 2. A surveillance method of comprising: using at least one event sensor disposed in a security area of a surveillance region to sense an occurrence of a potential security breach event; using at least one camera with a view of the security area in which the event is sensed to gather biometric information concerning at least one person in the vicinity of the security area at about the time of the sensing of the event: producing a subject dossier corresponding to the at least one person, the subject dossier including at least two biometric signatures; matching biometric information of one or more persons captured by one or more other cameras in the vicinity of the at least one camera with corresponding biometric information in the subject dossier; and incrementally fusing the at least two signatures and including the fused signature in the subject dossier.
* * * * *
Page 1 of 1
STATES PATENT AND TRADEMARK
www uspto. gov
BIB DATA SHEET

CONFIRMATION NO. 4531 SERIAL NUMBER 11/231,353 FILING or 371(c) DATE 09/19/2005 RULE APPLICANTS Ken Prayoon Cheng, Saratoga, CA; Edward Y. Chang, Santa Barbara, CA; Yuan-Fang Wang, Goleta, CA;
** CONTINUING DATA *************************
CLASS 348
GROUP ART UNIT 2625
ATTORNEY DOCKET NO. 5330.07 (SMC)
This appln claims benefit of 60/610,998 09/17/2004

** FOREIGN APPLICATIONS ************************
** IF REQUIRED, FOREIGN FILING LICENSE GRANTED **** SMALL ENTITY 10/07/2005

Foreign Priority claimed No 35 USC 119(a-d) conditions met Q Yes /JEROME GRANT II/ Verified and Examiners signatureTn Acknowledged
**
O Yes aNo O
ADDRESS PETERS VERNY, L.L.P. 425 SHERMAN AVENUE SUITE 230 PALO ALTO, CA 94306 UNITED STATES TITLE Adaptive multi-modal integrated biometric identification detection and surveillance systems Q All Fees
Q
1.16 Fees (Filing)
FILING FEE
FEES: Authority has been given in Paper
RECEIVED 565
No. No.
to charge/credit DEPOSIT ACCOUNT for following:
lQ
D 1.17 Fees (Processing Ext. of time) 1.18 Fees (Issue)

Q Other
I Credit
BIB (Rev. 05/07).
UNITED STATrES PATENT AND TRADEMARK OFFICE

UNITED STATES DEPARTMENT OF COMMERCE United States Patent and Trademark Othe
Addms COMMISSIONER FOR PATENTS P.O. Box 1450 Alexadris, Vubie 22313.-1450 wuwuipto.gma
BI
11eeItIN lllIlI IllaISh

FILING OR 371(c) DATE SERIL NUBER 09/19/2005 RULE
CONFIRMATION NO. 4531
Bib Data Sheet
SERIAL NUMBER 11/231,353
GROUP ART UNIT CLASS LASSDOCKET 262553.TM 348
ATTORNEY NO. 5330.07 (SMC)
APPLICANTS Ken Prayoon Cheng, Saratoga, CA; Edward Y. Chang, Santa Barbara, CA; Yuan-Fang Wang, Goleta, CA;
* CONTINUING DATA *************************

'* FOREIGN APPLICATIONS ********************
IF REQUIRED, FOREIGN FILING LICENSE GRANTED * 10/07/2005

Foreign Priority claimed Q yes Q no Met after 35 USC 119 (a-d) conditions 0 yes no met Allowance Verified and Examiner's Signature Acknowledged
STATE OR
COUNTRY CA
DRAWING 8
SHEETS
CLAIMS 20
TOTAL
INDEPENDENT
CLAIMS 3
Initials
ADDRESS 23308 TITLE ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE
SYSTEMS
.All Fees Q1.16 Fees (Filing ) FILING FEE RECEIVED 565 FEES: Authority has been given in PaperQ to charge/credit DEPOSIT ACCOUNT No. No. for following: 1.17 Fees ( Processing Ext. of time) Q 1.18 Fees (Issue) QOther
I QCredit
Page 1 of I
UNITED STATES PATENT AND TRADEMARK OFFICE

UNITED STATES DEPARTMENT OF COMMERCE United States Patent and Trademark Office Add e : COMMISSIONER FOR PATENTS P.O. 1450 Bax Alexadria., Vginia 22313-1450 pw.uryto.gov
lI IIlIaI10Il OIlOl ilheItflI

Bib Data Sheet
FILING OR 371(c) SERIAL NUMBER 11/231,353 DATE 09/19/2005 RULE CLASS 382 GROUP ART UNIT 2624 ATTORNEY DOCKET NO.
PPLICANTS Ken Prayoon Cheng, Saratoga, CA; Edward Y. Chang, Santa Barbara, CA; Yuan-Fang Wang, Goleta, CA;
* CONTINUING DATA ************************

* FOREIGN APPLICATIONS ********************
IF REQUIRED, FOREIGN FILING LICENSE GRANTED** SMALL ENTITY

* 10/07/2005 Foreign Priority claimed Q yesQ no
**
SUSyes 119noVerified and

cknowledged
y1ci
Met after
Initials
Es n eafe
Aowan
Examiner's Signature.
STATE OR COUNTRY CA
SHEETS DRAWING 8
TOTAL CLAIMS 20
INDEPENDENT CLAIMS 3
DDRESS 65761 TITLE Adaptive multi-modal integrated biometric identification detection and surveillance systems
I '
All Fees
1.16 Fees ( Filing )
FILING FEE RECEIVED 565
FEES: Authority has been given in Paper to charge/credit DEPOSIT ACCOUNT No. No. for following:Q
1.17 Fees ( Processing Ext. of time) 1.18 Fees (Issue) Other

_
Credit
Page 1 of 1
UNITED STATES PATENT AND ThADEMARK OFFIGE
UNITED STATE S DEPARTMENT OF COMMERCE United States Patent and Trademark Ofie Addrenm: COMMISSIONER FOR PATENTS P.O.Box1450 Alexandria, Vuginia22313-1450
www.uspto.gov
UUO llIU llHaII lillH li hlleit lli l

Bib Data Sheet
CONFIRMATION NO. 4531 FILING OR 371(c)
SERIAL NUMBER 11/231,353

I
DATE
09/19/2005
CLASS
348
GROUP ART UNIT

2621
ATTORNEY DOCKET
NO.
RULE PPLICANTS Ken Prayoon Cheng, Saratoga, CA; Edward Y. Chang, Santa Barbara, CA; Yuan-Fang Wang, Goleta, CA;
* CONTINUING DATA **********************
This appin claims benefit of 60/610,998 09/17/2004

* FOREIGN APPLICATIONS *******************
IF REQUIRED, FOREIGN FILING LICENSE GRANTED ** **SMALL ENTITY **

10/07/2005
Foreign Priority claimed 5 Priority claimed 1oreign

35 USC 119 (a-d) conditions et
erified andCA823 cknowledged
yesQ no yes no
QSTATE
Met after
Initials
OR COUNTRY
yes Allowance
no
CA
SHEETS DRAWING
TOTAL CLAIMS
INDEPENDENT
CLAIMS
20
Examiner's
Signature
ADDRESS KEN P. CHENG 0691 REID LANE SARATOGA, CA95070-5325 TITLE daptive multi-modal integrated biometric identification detection and surveillance systems
Q
All Fees
i.1.16 Fees ( Filing FILING FEE RECEIVED 565 FEES: Authority has been given in Paper to charge/credit DEPOSIT ACCOUNT No. for following: No.
Q 1.17 Fees ( Processing Ext. of time) ssu 1 Fe IFees( Issue) X1.18 Q Other i-Credit
O
OO
.
,l
(71
co~
-4
D C 0
PTO/SB/05 (05-05) Approved for use through 07/31/2006. OMB 0651-0032 U.S. Patent and Trademark Office. U.S. DEPARTMENT OF COMMERCE Under the Paperwork Reduction Act of 1995, no persons arerequired to respond to a collection of miformation itdisplays a valid OMB control number. unless
I--V
UTILITY PATENT APPLICATION TRANSMITTAL

(ONLY FOR NEW NONPROVISIONAL APPLICATIONS UNDER 37CFR 1.53(8))
Attorney Docket No.

First Inventor
577832000200
Ken Prayoon CHENG
C)
Tiue
ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND

SURVEILLANCE SYSTEMS
EL 968416615 US
Commissioner for Patents
Express Mail Label No.
See MPEP chapter 600 concerning utility patent application contents. ___Alexandria, 1.[ 2. D Fee Transmittal Form (e.g., PTOISBI/17), (Submit original adupicete for fee en and processing) Applicant laims small entity status.
APPLICATION ELEMENTS
ADDRESS
TO:
P.O. Box 1450

VA 22313-1450
ACCOMPANYING APPULICATION PARTS 9.
2. e 7FR1.27. 3. []Specification (TotalPages Both clairm and the abstract start a new must on page
4. Drawing(s) (35 U.S.C. 113) [Total Sheets [Total Sheets
Assignment Papers (coversheet & document(s))
32
6
Ppr Name of Assignee:
cvr
&dcmn~)
(For information on the prefered arrangement, see MPEP.608.01(a)) ] ] 10. 11. 12. 37 CFR 3.73(b) Statement (when there is an assignee) [7Power of LJ Attorney
5. Oath orDeclaration a.Q
Newly executed (original or copy)
English Translation Document (f applicable) Information Disclosure Statement (PTO/SB/08 or PTO-1449) Copies of foreign patent documents, publications. &other Information ] Preliminary Amendment
b. [7Acopyfrom a prior application (37 CFR 1.63(d)) L.J(for contiualt~io~i'sionalwithBox completed)" 1e i7DELETION OF INVENTOR(S) '[ SSigned statement attached deleting inventor(s) named the in prior application, 37CFR 1.63(dX)(2)1.33(b). see and 6. [ Application Data Sheet. See 37 CFR 1.76 (3 pages)
13.
7.[7CD-ROM or CD-R induprlicate. large table or Computer Program (Appendix) Landscape Table on CD 8. Nucleotide andlor Amino Acid Sequence Submission (if applicable, Items a.- c. are required) a. Computer Readable Form (CRF) Computer Readable Form (CRF) i . ii. F]Transfer Request (37 CFR 1.821(e)) b. Specification Sequence Listing on: i. ] CD-ROM or CD-R (2 copies); or
c.
14. X Return Receipt Postcard (MPEP 503) (Should be specificallyitemized) 15.[ 16. [ Certified Copy of Priority Document(s) (f folbren priority isclaimed) Nonpublication Request under 35 U.S.C.122 (bX2XBXI). Appilcant must attach form PTO/SB/35 or equivalent Other
17. II.D Paper
Statements verifying identity of above copies
18. If a CONTINUING APPLICATION, check appropriate box, and supply the requisite information below and in the first sentence of the specification followingthe title, or in an Application Data Sheet under 37 CFR 1.76:
] ContinuationF
Divisional
Continuation-in-part (CIP)
of prior application No.:

Art Unit:
Prior applicationInformation: Examiner
19. CORRESPONDENCE ADDRESS S The address associated with Customer Number Name Address City
Country .elep
20872
OR
[]
Correspondence address below
State
Email Addres
s
Zip Code
Signature
Name
Date tpeRegistration
September 19, 2005

No.
(PrintlType) tp
C. DurantAttome/Agent
31,506
I hereby certify that this correspondence is being deposited with the.S. Postal in an envelope addressed to: Commissioner for Patents, P.O.
Dated: September 19, 2005
ic.s
Express Mail, Airbill No. EL 968416615 US, inia22313-1450, on the date shown below.
Signature:
-_,_
-__-__
Todd V. Leone
.._
sf-2002707 vl
ti
ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEM Inventor: Ken Prayoon Cheng, Edward Y. Chang, Yuan-Fang Wang CROSS-REFERENCE TO RELATED APPLICATION [00011] The present application claims the benefit of earlier filed provisional patent application, U.S. Application No. 60/610,998, filed on September 17, 2004, and entitled "Adaptive Multi-Modal Integrated Biometric Identification Detection Systems," which is hereby incorporated by reference as if fully set forth herein. BACKGROUND OF THE INVENTION Field of the Invention: [0002] The invention relates in general to biometric identification, and more
particularly, to a surveillance system using biometric identification. Brief Description of the Related Art: [0003] The state of the art of applying biometric technologies to authenticate and
positively determine the identity of a person is still faced with several technical challenges. Specifically, the challenges can be categories into two aspects: data acquisition and data matching. Data acquisition deals with acquiring biometric data from individuals. Data matching deals with matching biometric data both quickly and accurately. These challenges can be explained by a port-entry scenario. In such a setting, it is difficult to obtain certain biometric data such as DNA and voice samples of individuals. For biometric data that can be more easily acquired, such as face images and fingerprints, the acquired data quality can vary greatly depending on acquisition devices, environmental factors (e.g., lighting condition), and individual corporation. Tradeoffs exist between intrusiveness of data collection, data collection speed, and data quality. [0004] Once after the needed data have been acquired, conducting matching in a very
large database can be very time-consuming. It goes without saying that unless a system can acquire and match data both timely and accurately, the system is practically useless
in improving public security, where the inconvenience due to the intrusive dataacquisition process and the time-consuming matching process ought to be minimized.
100051
A biometric system typically aims to address either one of the following
issues: 1) Authentication: is the person the one he/she claims to be? 2) Recognition: who a person is? In the first case, data acquisition is voluntary and matching is done in a one-to-one fashion---matching the acquired data with the data stored on an ID card or in a database. In the second case, individuals may not be cooperating, and the system must conduct searches in a very large repository. [0006] The prior art in biometric can be discussed in two parts: single-modal
solutions and multi-modal solutions. Several systems have been built to use one of the following single modal: facial data, voice, fingerprint, iris or DNA. The effectiveness of these single-modal approaches can be evaluated in three metrics: the degree of intrusiveness, speed and accuracy. From the perspective of a user, acquiring face modal can be the most noninvasive method, when video cameras are mounted in the distance. However, the same convenience nature often compromises data quality. An intrusive face acquisition method is to acquire frontal face features, which requires corporation from individuals. Voice is another popular modal. However, traditional voicerecognition fails miserable when voice samples of multiple individuals are simultaneously captured or when background noise exists. Even when the acquired voice data can be "pure," existing signal processing and matching techniques can hardly achieve recognition accuracy of more than 50%. The next popular modal is fingerprint, which can achieve much higher recognition accuracy at the expense of intrusive data acquisition and time-consuming data matching. Finally, DNA is by far the most accurate recognition technique, and the accompanying inconvenience in data acquisition and the computational complexity are both exceedingly high. Summarizing the single model approach, non-intrusive data-acquisition techniques tend to suffer from low recognition accuracy, and intrusive data-acquisition techniques tend to suffer from long computational time
[0007]
As to multimodal techniques, there have been several prior art United States
Patents and PatentApplications disclose techniques. However, as will be further discussed below, these disclosures do not provide scalable means to deal with tradeoffs between non-intrusiveness, speed and accuracy requirements. These disclosures may fix their system configuration for a particular application, and cannot adapt to queries of different requirements and of different applications. [0008] Wood et al. disclose in US Patent 6,609,198 a security architecture using the
information provided in a single sign-on in multiple information resources. Instead of using a single authentication scheme for all information resources, the security architecture associates trust-level requirements with information resources. Authentication schemes (e.g., those based on passwords, certificates, biometric techniques, smart cards, etc.) are employed depending on the trust-level requirement(s) of an information resource (or information resources) to be accessed. Once credentials have been obtained for an entity and the entity has been authenticated to a given trust level, access is granted, without the need for further credentials and authentication, to information resources for which the authenticated trust level is sufficient. The security architecture also allows upgrade of credentials for a given session. The credential levels and upgrade scheme may be useful for a log-on session; however, such architecture and method of operations do not provide a resolution for high speed and high accuracy applications such as passenger security check in an airport. [0009] Sullivan et al. disclose in US Patent 6,591,224 a method and apparatus for
providing a standardized measure of accuracy of each biometric device in a biometric identity authentication system having multiple users. A statistical database includes continually updated values of false acceptance rate and false rejection rate for each combination of user, biometric device and biometric device comparison score. False acceptance rate data are accumulated each time a user successfully accesses the system, by comparing the user's currently obtained biometric data with stored templates of all other users of the same device. Each user is treated as an "impostor" with respect to the other users, and the probability of an impostor's obtaining each possible comparison score is computed with accumulated data each time a successful access is made to the system.
The statistical database also contains a false rejection rate, accumulated during a test phase, for each combination of user, biometric device and biometric device comparison score. By utilizing a biometric score normalizer, Sullivan's method and apparatus may be useful for improving the accuracy of a biometric device through acquiring more training data. [0010] Murakami et al. disclose is a Patent Publication 20,020,138,768 entitled
"Method for biometric authentication through layering biometric traits," a portable biometric authentication system having a single technology for measuring multiple, varied biological traits to provide individual authentication based on a combination of biological traits. At least one of these biometric traits is a live physiological trait, such as a heartbeat waveform, that is substantially-but not necessarily completely unique to the population of individuals. Preferably, at least one of the identifying aspects of the biological traits is derived from a measurement taken by reflecting light off the subdermal layers of skin tissue. The Murakami et al. approach is limited by the more intrusive measurement techniques to obtain data such as heartbeat waveform and reflecting light off the subdermal layers of skin tissue. These data are not immediately available in a typical security check situation to compare with the biometric data, e.g., heart beat waveforms and reflection light from subdermal layers from the skin of a targeted searching object. Furthermore, the determination or the filtering of persons' identity may be too time consuming and neither appropriate for nor adaptive to real time applications.. [0011] Langley discloses in US Patent Application 20,020,126,881, entitled "Method
and system for identity verification using multiple simultaneously scanned biometric images," a method to improve accuracy and speed of biometric identity verification process by use of multiple simultaneous scans of biometric features of a user, such as multiple fingerprints, using multiple scanners of smaller size than would be needed to accommodate all of the fingerprints in a single scanner, and using multiple parallel processors, or a single higher speed processor, to process the fingerprint data more efficiently. Obtaining biometric data from multiple user features by use of multiple scanners increases verification accuracy, but without the higher cost and slower
processing speed that would be incurred if a single large scanner were to be used for improved accuracy. The methods according to Langley may provide the advantages of speed and accuracy improvements. However, the nature of requiring multiple scans makes data acquisition time-consuming and intrusive: [00121 On the academia side, much research effort has been geared toward analyzing
data from individual biometric channels (e.g., voice, face, fingerprint, please see the reference list for a partial list), less emphasis has been placed on comparing the performance of different approaches or combing information from multiple biometric channels to improve identification. Some notable exceptions are discussed below. In Hong Lin, Jain A. K., Integrating faces and fingerprints for personal identification, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 20, No. 12, Dec. 1998, pp. 1295 - 1307, the authors report an automated person identification system that combines face and fingerprint information. The face recognition method employed is the traditional eigen face approach, M. Turk and A. Pentland, Eigenfaces for Recognition, J. Cognitive Neuroscience Vol. 3, No. 1, 1991, pp. 71-96, which computes a set of orthonormal bases (eigen faces) of the database images using the principal component analysis. Face images are then approximated by their projection onto the orthonormal Eigen face bases, and compared using Euclidean distances. For fingerprint, the authors extend their previous work, Jain, A. K.; Lin Hong; Bolle, R.; On-line fingerprint verification, Pattern Analysis and Machine Intelligence, Vol. 19, No. 4, April 1997, pp. 302 - 314, to extract minutiaes from fingerprint images. They then align two fingerprint images by computing the transformation (translation and rotation) between them. Minutiaes are strung together into a string representation and a dynamic programmingbased algorithm is used to compute the minimum edit distance between the two input fingerprint strings. Decision fusion is achieved by cross validation of the top matches identified by the two modules, with matching results weighed by their confidence or accuracy levels. The performance of the system is validated on a database of about 640 face and 640 fingerprint images. 10013] In Phillips, Henson Moon; Rive, SEA.; Russ, The FERRET evaluation
methodology for face-recognition algorithms, IEEE Transactions on Pattern Analysis and
Machine Intelligence, Vol. 22, No. 10, Oct. 2000, pp. 1090- 1104, the Michigan State University research group extends their information fusion framework to include more modalities. In particular, images of a subject's right hand were captured, and fourteen features comprising the lengths of the fingers, widths of the fingers, and widths of the palm at various locations of the hand. Euclidean distance metric was used to compare feature vectors. Simple sum rules, decision tree and linear discriminant function are used for classification. It is observed that a personal ID system using three modules outperforms that uses only two of the three modules. While this is an interesting experiment, the data set used is small and there is no accepted universal standard in using hand images in biometrics. 10014] In R. Brunelli, D. Falavigna, T. Poggio and L. Stringa, Automatic Person
Recognition by Using Acoustic and Geometric Features, Machine Vision and Applications 1995, Vol. 8 pp. 317-325, an automated person recognition system using voice and face signatures is presented. The speaker recognition subsystem utilizes acoustic parameters (log-energy outputs and their first-order time derivatives from 24 triangular band-pass filters) computed from the spectrum of short-time windows of the speech signal. The face recognition subsystem is based on geometric data represented by a vector describing discriminant facial features such as positions and widths of the nose and mouth, chin shape, thickness and shape of the eyebrows, etc. The system captures static images of the test subjects and the test subjects are also asked to utter ten digits from zero to nine for use in the speaker ID subsystem. Each subsystem then computes the distances of the test subject's speech and face signatures with those stored in the databases. Decisions from the two ID modules are combined by computing a joint matching score that is the sum of the two individual matching scores, weighted by the corresponding variance. Experimental results show that integration of visual and acoustic information enhances both performance and reliability of the separate systems. The above system was later improved upon in Brunelli, R.; Falavigna, D., Person identification using multiple cues, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 17, No. 10, Oct. 1995, pp. 955-966, where multiple classifiersare used in the face recognition subsystems, and the matching score normalization process is made more robust using robust statistical methods.
[0015]
In Kittler, J.; Hatef, M.; Duin, R.P.W.; Matas, J., On combining classifiers,,
IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 20, No. 3, Mar. 1998, pp. 226-239, a performance study of various ensemble classification scheme is presented. It is shown that many existing decision aggregation rules are actually simplifications based on the more general Bayesian rule.. The authors compare the performance of different decision aggregation rules (max, min, median, and majority voting rule) by performing an experiment in biometrics. Three modules are used: frontal faces, face profiles, and voiceprints. Simple correlation-based and distance-based matching is performed on frontal faces and face profiles, respectively, by finding a geometric transformation that minimizes the differences in intensity. It is shown that a simple aggregation scheme by summing the results from individual classifiers actually perform the best. [0016] In Lu X; Wang Y; and Jain A, Combing classifiers for face recognition, IEEE
International Conference on Multimedia Systems and Expo, Baltimore, MD, July 2003, three well-known appearance-based face recognition methods, namely PCA, M. Turk and A. Pentland, Eigenfaces for Recognition, J. Cognitive Neuroscience Vol. 3, No. 1, 1991, pp. 71-96, ICA, and LDA, Belhumeur, P.N.; Hespanha, J.P.; Kriegman, D.J., Eigenfaces vs. Fisherfaces: recognition using class specific linear projection, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 19, No. 7, Jul. 1997, pp. 711-720, are used for face image classification. Two combination strategies, the sum rule and RBF network, are used to integrate the outputs from these methods. Experimental results show that while individual methods achieve recognition rates between 80% and 88%, the ensemble classifier boosts the performance to 90%, using either the sum rule or RBF network. In Senior, A., A combination fingerprint classifier, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 23, No. 10, Oct. 2001, pp. 1165-1174, a similar multi-classifier scheme, this time for fingerprint classification, is proposed. Hidden Markov Models and decision trees are used to recognize ridge structures of the fingerprint. The accuracy of the combination classifier is shown to be higher than that of two state-of-the-art systems tested under the same condition. These studies represent encouraging results that validate our multi-modal approach, though only a single
biometric channel, either face or fingerprint, not a combination of biometric channels, is used in these studies. [0017] Maio, D.; Maltoni, D.; Cappelli, R.; Waymani, J.L.; Jain, A.K., FVC2000:
fingerprint verification competition, IEEE Transactions on Pattern Analysis and Machine

Intelligence, Vol. 24, No. 3 , March 2002, pp. 402 - 412, documents a fingerprint
verification competition that was carried out in conjunction with the International Conference on Pattern Recognition (ICPR) in 2000 (a similar contest was held again in 2002). The aim is to take the first step towards the establishment of a common basis to better understand the state-of-the-art and what can be expected from the fingerprint technology in the future. Over ten participants, including entries from both academia and industry, took part. Four different databases, two created with optical sensors, one with a capacitive sensor, and one synthesized, were used in the validation. Both the enrollment error (if a training image can be ingested into the database or not) and the matching error (if a test image can be assigned the correct label or not) and the average time of enrollment and matching are documented. 10018] A study, that is similar in spirit but compares the performance of face
recognition algorithms, is reported in Phillips, P.J.; Hyeonjoon Moon; Rizvi, S.A.; Rauss, P.J., The FERET evaluation methodology for face-recognition algorithms, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 10, Oct. 2000, pp. 1090- 1104. A subset of the Feret database (a gallery of over 3000 images) was used in the study. Ten different algorithms, using a wide variety of techniques, such as PCA and Fischer discriminant, were tested. Cumulative matching scores as a function of matching ranks in the database are tabulated and used to compare the performance of different algorithms. This study was repeated three times, in August 1994, March 1995, and July 1996. What is significant about this study is that the performance of the face recognition algorithms improved over the three tests, while the test condition became more challenging (with increasingly more images in the test datasets). 10019] As can be seen from the above brief survey, multi-modal biometrics holds a
lot of promise. It is likely that much more accurate classification results can be obtained
by intelligently fusing the results from multiple biometric channels given performance requirements. While it is important to keep on improving the accuracy and applicability of individual biometric sensors and recognizers, the performance of a biometric system can be boosted significantly by judiciously and intelligently employing and combining multiple biometric channels. [0020] While there have seen significant research activities in single- and multi-
channel biometry over the past decade, the state-of-the-art is still wanting in terms of speed and accuracy. Therefore, a need still exists in the art to provide new and improved methods and system configurations to increase the speed and accuracy ofbiometric identity verification and determinations such that the above-mentioned difficulties and limitations may be resolved. The present invention meets this need. SUMMARY OF THE INVENTION [0021] One embodiment of the invention provides a novel surveillance method. An event sensor such as, a camera, chemical sensor, motion detector, unauthorized door access sensor, for example, is disposed to sense an occurrence of a potential security breach event. A camera with a view of the area in which an event is sensed gathers biometric information concerning a subject person in the vicinity of the event at about the time the event is sensed. A subject dossier is produced containing biometric information relating to the subject person sensed by the camera with the view of the area. Biometric information of persons captured on one or more other surveillance cameras in the general vicinity of the event is matched against corresponding biometric information in the subject dossier. [0022] Another embodiment of the invention provides a new surveillance system. A
sensor is disposed in a surveillance region to sense an occurrence of a security breach event. The system includes a plurality of cameras. At least one camera of the plurality has a view of the security area and can be configured to automatically gather biometric information concerning a subject person in the vicinity of an area where the event occurred in response to the sensing of the event. One or more of the plurality of cameras can be configured to search for the subject person. The surveillance system also includes
a processing system which can be programmed to produce a subject dossier corresponding to the subject person. The processing system also can be programmed to match biometric information of one or more persons captured by one or more of the cameras with corresponding biometric information in the subject dossier. 100231 These and other features and advantages of the invention sill be apparent from
the following description of embodiments thereof in conjunction with the drawings. BRIEF DESCRIPTION OF THE DRAWINGS [0024] Figure 1 is an illustrative showing a map of an airport passenger terminal and its immediate vicinity protected by a surveillance system of one embodiment of the invention and also showing several pop-up views relating to event alerts in accordance with the embodiment. [0025] Figure 2 is another view of the map of Figure 1 showing zoom to detail maps
of different portions of the overall passenger terminal map. [0026] Figure 3 is an illustrative drawing of example securityareas within the
surveillance region of Figures 1-2 outfitted with event sensors. [0027] Figure 4 is an illustrative block level hardware diagram of a surveillance
system in accordance with an embodiment of the invention. [0028] Figure 4. [0029] Figure 6 is an illustrative flow diagram showing gathering and conversion of Figure 5 is an illustrative block diagram level drawing of a system
architecture of an embodiment of the invention that incorporates the system hardware of
facial feature data to a facial feature signature.
[0030]
Figure 7 is an illustrative flow diagram showing gathering and conversion of
fingerprint feature data to a fingerprint signature.
[0031]
Figure 8 is an illustrative flow diagram showing gathering and conversion of
DNA data to a DNA signature. One embodiment of the invention may employ a DNA fingerprint for identification purposes. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT [0032] The following description is presented to enable any person skilled in the art
to make and use the invention, and is provided in the context of particular applications and their requirements. Various modifications to the preferred embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the invention. Moreover, in the following description, numerous details are set forth for the purpose of explanation. However, one of ordinary skill in the art will realize that the invention can be practiced without the use of those specific details. In other instances, well-known structures and devices are shown in block diagram from in order not to obscure the description of the invention with unnecessary detail. Thus, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein. System Overview [0033] One embodiment of the invention involves an intelligent surveillance system.
A plurality of cameras, some with and some withoutoverlapping fields of view, are distributed throughout a surveillance region. Intelligent computer software based agents process information captured by one or more of the cameras to produce a subject dossier indicative of the identity of a person whose images have been captured by one or more of the cameras. Information for a subject dossier also may be gathered through other modalities such as voice recognition, iris scan, or fingerprint, for example. The system includes multiple event sensors, which may include the cameras, chemical sensors, infrared sensors, or other security alarm sensors that trigger an alert, upon sensing an occurrence of a predetermined category of event requiring heightened vigilance. For example, an alarm may be triggered when a locked door is opened without proper access permission or when an unauthorized person enters a restricted area or when a vehicle is parked in a restricted area. More specifically, a subject dossier is produced for
individuals in the vicinity of the location of an alarm-triggering event. For instance, a subject dossier may be produced for persons captured in a video camera image at or near a door in the surveillance region at about the time when an unauthorized opening of the door is detected by an event sensor. [0034] A subject dossier may include soft biometric information, also referred to as
"soft" features such as clothing color, estimated height and weight. A subject dossier also may include temporal information, such as walking speed or direction of travel. In addition, a subject dossier'also may include more permanent information such as facial features, fingerprint, iris scan, voiceprint and DNA. Soft features maybe selected to be especially useful for relocating an individual within the surveillance region, especially in a crowd, for example. For instance, it may be relatively easy to identify individuals based upon clothing color or estimated height and weight. However, soft features have the disadvantage of not being as reliable or permanent over time. If a person takes off his jacket, then an identifying color feature may be lost. If a person sits down, then it may become impossible to use height and weight information to pick that person out of a crowd. [0035] System sensors continually monitor the surveillance region for the occurrence
of one or more suspicious events. In one embodiment, the system directs a live video feed from one or more cameras having the location of an alert-triggering event in their field of view to a console in a manned control center. The system also may direct video images captured just before the event to the control center console. Thus, an operator at the console can observe behavior of suspicious individuals at the scene of the event in real time and immediately prior to the event. A subject dossier produced for individuals at the scene of the event can be used to automatically identify and track a suspect individual present at the scene of the event within the surveillance area after the occurrence of the event. [00361 The system may employ information in a subject dossier incrementally. For
instance, the system may prioritize information in the subject dossier. Certain information in the subject dossier such as clothing color, estimated height and weight,
walking pattern or gait and certain key facial features such as facial shape, facial hair, skin color, or hair color may be used to make an initial estimate of which persons in a camera's field of view are candidates for a match to a suspicious person identified in response to an alert. Other features from a subject dossier then may be added incrementally to make a more careful assessment of whether identified candidates actually match the suspect. Alternatively, as more information concerning a suspicious person becomes available, additional features may be added incrementally to a suspect's subject dossier for that person. This additional information then may be used to more effectively locate and track the individual within the surveillance region. Surveillance Region [0037] One embodiment of the invention is configured for use in airport security. In this embodiment, the surveillance region comprises an airport passenger terminal and the surrounding passenger ground transport loading/unloading zone directly outside the terminal and the aircraft parking area adjacent the terminal. Figure 1 is an illustrative drawing of a map of an airport passenger terminal and its immediate vicinity protected by a surveillance system of one embodiment of the invention. The system includes multiple cameras, each with an associated field of view, some of which are overlapping. The surveillance region has multiple areas including passenger arrival and departure areas, a passenger departure shops and a terrace. Groups of cameras with overlapping fields of view are deployed to capture images within different regions of the passenger arrival and passenger departure areas. [00381 Figure 2 is another view of the map of Figure 1 showing zoom to detail maps
of different portions of the overall passenger terminal map. The illustrative maps of Figures 1-2 can be displayed on a control terminal so that an operator can easily correlate an alert to a specific area an airport surveillance region. For instance, if an alert is triggered in the arrivals region shown in Figure 1, then an operator may request the leftmost zoom shown in Figure 2 in order to quickly picture the airport layout in the vicinity of the alert. Additional zoom maps (not shown) may be provided for numerous locations such as security gates, check-in counters, airport fairway, parking area, access
13
entrance, check-in counters, etc. Each different area may be associated with a group of cameras and event sensors. [00391 Event sensors are disposed at selected locations within the surveillance region.
Figure 3 is an illustrative drawing of example security areas within the surveillance region of Figures 1-2 outfitted with event sensors. A first security area comprises a door. The door may be equipped with a sensor, such as a mechanical sensor, that detects unauthorized opening of the door. A second security area comprises a window. The window may be associated with a mechanical sensor that detects when the window has been broken. A third security represents a threshold to a restricted area. The restricted area may be equipped with motion detectors that detect the presence of persons in a restricted area. Cameras situated throughout the surveillance' region also may serve as event sensors. For example, the system may employ a monitoring rule whereby a camera monitors a particular area of the passenger terminal. If a person is loitering in that area, defined by failing to move beyond a 15 foot radius for more than 60 seconds, then a low level alert is declared, the camera zooms in, and the face of the loitering person is matched against the faces of persons on a watch list, for example. [00401 Landmarks are defined in the security areas for purpose of estimating height
and weight and direction and speed of travel of a suspect individual. For instance, a landmark such as a countertop may be identified, and processing of a camera image may be calibrated to estimate a person's height relative to the land marked countertop. A group of multiple structures, such as telephone booths, lounge areas, signs or countertops, within a field of view of one or more of a group of cameras covering a security area may be identified. Processing of camera images from the group of cameras may be used to estimate the direction and speed at which a suspect is moving based upon the sequence and timing of his passing the land marked structures. [00411 Although the surveillance region in this one example is described in terms of
an airport passenger terminal, it will be appreciated that the invention is not restricted to an airport terminal. Moreover, the surveillance region need not be a continuous local area. Event sensors and surveillance cameras may be disposed over disparate areas and
be in communication with a control center via a network such as the internet, for example. System Architecture [0042] Figure 4 is an illustrative block level hardware diagram of a surveillance
system in accordance with an embodiment of the invention. The system includes multiple data collection agents, a knowledge server, a local knowledge server database, an application server, a middle-tier database, web servers, a browser based control console and one or more client applications such as Computer Aided Dispatch system, building management system, access control system, etc. It should be understood that the various components shown are merely illustrative. Each agent may gather information from numerous sources, such as the cameras shown in Figure 1, distributed throughout a surveillance region. Moreover, for example, the knowledge server and the application server can be implemented across multiple hardware systems or as different processes within a single hardware system. [0043] A security agent is a process that spans many tasks to collect information
about subject(s). For example, a security agent may spawn multiple data collection agents include a facial features, fingerprint, DNA, clothing color, subject gait, subject height and weight, skin color/tone , hair color/tone, subject direction and voiceprint, for example. Each data collection task produces different information about an individual. More specifically, each produces a signature indicative of some identifying aspect of a person under surveillance. For instance, a facial features agent uses facial information captured by one or more cameras to produce a signature indicative of an individual's facial features. Similarly, for example, a clothing color agent uses clothing color information captured by one or more cameras to produce a signature indicative of the color of an individual's clothing color. Thus, the multiple agents can produce multiple different signatures, each indicative of one or more different identifying feature of an individual. [0044] The agents provide the signatures to the knowledge server, which aggregates
-
signatures for each given person under surveillance into a subject dossier for that person.
The knowledge server indexes the signatures within a given subject dossier to permit incremental searches for individuals within the search region. The knowledge server also may perform classification and matching. The local knowledge server database stores the digital signatures and corresponding indexing information. [0045] The web services is the component that provides the interfaces via Web
Server which is usually part of an operating system. For example, web services provides the interfaces for our internal components or external systems via Web Server (such as Microsoft IIS on Windows, or Apache on Linux). All the interfaces to the system are via HTTP or HTTPS using port 80. Doing so, our system can run across firewall. Basically, the Web Services component just exposes our system interface to the outside world via Web Server. [0046] The application server is the component that provides that database access to
the user interface component, and performs session management which includes authentication and authorization. The middle-tier database serves as the local database for the application server. [0047] Figure 5 is an illustrative block diagram level drawing of a system
architecture of an embodiment of the invention that incorporates the system hardware of Figure 4. A user interface (UI) provides an operator of the system with real-time information concerning alert events within the surveillance region. The UI may provide maps of the entire surveillance region, including zoom maps. It can display alerts from different sensors including cameras, digital video recorders, access control, bio-chemical detectors, etc. It may display videos of a security area in which an alert has been triggered, detailed images of suspect individuals and details of one or more alerts that have been triggered. [0048] Referring again to Figure 1, there is show an example of a UI display screen
in with pop-up display showing various images relating to one or more alerts. In the center of the screen is map of a surveillance region. The operator can be selectively enlarge, minimize or close each pop-up. A Video Review display provides a video image of the security region at about the time of an alert. An Incident Detection display
provides detailed information concerning an alert event. In this example, the alert event involved an individual tailgating at a commuter door. A Suspect Description display provides identifying information concerning an individual under surveillance based upon information gathered into a subject dossier produced for the person. A Detailed Images display provides pictures of a suspect individual captured by one or more surveillance cameras. A Potential Identification display provides images of the suspect together with images of one or more people whose facial features closely match those of the suspect. The potential matches are based upon a facial feature signature provided by the facial feature agent. Across the bottom of the map, there is a chart listing briefly summarizing multiple alert situations. The operator may selectively access pop-up screens for these alert situation. [0049] Thus, the UI advantageously displays a variety of information aggregated in
response to one or more alerts. In a typical airport security region, for example, there may be several hundred cameras dispersed throughout a large physical area. Moreover, there may be only a few operators monitoring one or more UI consoles. Depending upon the rules for monitoring and declaring alerts, alerts may occur frequently or infrequently. The UI of one embodiment of the invention directs an operator to areas of a surveillance region that are subject to alert and provides pertinent information concerning the alert so that the operator can efficiently manage security from a control center. The UI also allows an operator to quickly investigate and simultaneously keep abreast of multiple alert events. [0050] Furthermore, as explained more fully below, information from different
sensing devices is correlated to facilitate tracking of a suspect within a security region. For instance, soft biometric information and temporal information is used to locate a suspect as he or she travels within the security region. In one embodiment, a dashed line can be produced on a map on the display showing a path followed by a suspect within the surveillance region. Information from different data collection agents may be fused in order to more Iccurately-identify and track an individual. Therefore, the operator can use the UI to evaluate an alert event, to identify and track a suspect. The operator may use
this information as a basis to send information to a responder to intercede or deal with an alert incident. [0051] Knowledge Services are implemented as an application running on the
knowledge server. Knowledge Services correlate and analyze signature information provided by different sensory devices (i.e., data gathering agents). The Knowledge Services assemble and index subject dossiers, and when appropriate, fuse signature information for improved classification results. The Knowledge Services also generate, activate or deactivate rules and send/control rules and instruction to the Rules and Agent Manager. [00521 The Rules and Agent Manager also is implemented on the knowledge server.
The Rules and Agent Manager manages all other agents and manages rules that can be sent to each agent. It correlates information from agents. It can also escalate an alert if the alert is not acknowledged by an operator within a given timeframe and/or similar alerts happen repeatedly within a given time span (e.g. within 2 hours). Both the Knowledge Service and the Rules and Agent Manager are the primary components for aggregating, categorizing biometric signatures which are parts of object dossiers. It also performs other tasks such as task assignment/tracking, load balancing tasks among agents, and interacting with data access components. [00531 The following are examples of rules that may be implemented by the system.
100541
Actor
Rules:
Action
Person Person Person Person Person Person
Walk though lane against direction of traffic Tailgating Loitering Piggyback Traveler screening Walk in restricted area
Vehicle Vehicle [0055] Radius Duration Alert Severity Response 100561 Response
Park overtime Park iA in restricted area
The Person-Loitering rule involves the following criteria: 15 foot 20 seconds Low Zoom in face to match "watch list" The Person-Tailgating Rule involves the following criteria: Low Zoom in face to match "watch list"
Alert Severity
[0057] The correlation Monitoring Rule for the occurrence of a Person-Loitering event AND a Person-Tailgating event involving the same person is as follows: Alert Severity Action Critical Acknowledge Loitering and Tailgating alerts and deliver alarm to
operator console
[0058]
As described above the UI, may display several categories of information
concerning an alert. The Knowledge Service and the Rules and Agent Manager provide the correlation between events and data sources and subject dossiers that permit an operator to view a map of the location of an alert, soft-biometric data of a suspect and video playback, for example. More particularly, these components provide a link to a map and zoom stored in the middle tier database, link to video feeds for video view realtime monitoring or playback of recorded video clips and stored in a Digital Video Recorder system and provide the subject dossier information. [0059] The Middle Tier Data Access runs on the application server. It controls the
database including functions such as query, add, delete, index. Indexing biometric signatures and updating subject dossiers are done by this component.
[00601
A (Security) Agent is implemented as an application running on the
knowledge server that controls and manages the data gathering sensors. In the case of cameras or DVRs, it can also perform video analytic using Computer Vision technology. Those tasks include background subtraction, image stabilization, object detection, object classification, object tracking, and object identification. It can also control the movement of Pan/Tilt/Zoom (PTZ) cameras, manage areas of interest within the field of view of the camera (called Mouse/Man Trap), and collect video streams from DVR or cameras. It also has a scheduler that controls when rules or video analytic are performed. [00611 A SensoryDevice Directory Access and Video Server is implemented as an
application that has access to the knowledge server manages and provides information regarding sensor devices or other subsystems. Basically, it is a software layer that enables the overall system to handle different makes/models of sensor devices. [00621 The Web Services is the component provided by operating systems or web
servers. It manages other components, spawns or deletes services as necessary. It can also listen to messages from other systems. The Web Services provides interfaces to the system via Web Services running as part of a Web Server. The system provides a library resided on a specific directory, and the Web Server (which is usually part of the operating system) will use it to interpret interface requests to our system. [0063] Tracking, Facial Recognition, Fingerprint recognition, and other biometric
identification are done at the (Security) agents. Biometric signatures are collected and generated at the agents, and sent to the Rules-and-Agent Manger. The Knowledge Services and the Rule-and-Agent Manager collectively collect biometric signatures and object tracking locations, and then generate and manage subject dossiers. A described above, a subject dossier includes information about object (e.g., person) such as, biometric information/signatures, soft biometric information (hair color, skin tone/color, weight or build, height, etc.) and other temporal information (e.g., speed, direction, location, past activities, information that the operator is looking for, etc.). Data fusion is performed by the Knowledge Services and the Rules and Agent Manager. Data required or generated by each of the components are saved and retrievable via the
Middle-tier/Data Access component, which in turn utilizes a relational database such as Microsoft SQL Server. Subject Dossier [0064] Data gathering agents collect data concerning a subject person from different sources. The Knowledge Services aggregate the data into subject dossier. The data aggregated into a given dossier may include different digital signatures produced by different data gathering agents. A subject dossier also may include fused data signatures produced by the fusion of data gathered from multiple data sources having different data modalities. [0065] [0066] The following is an example of information in a subject dossier. Subject Dossier:
Facial Features Signature (e.g., nose shape and size, face width, distance between eye corners, skin color (light, medium, dark), nose angle (profile view) Soft Biometrics Signature (e.g., clothing color, height, weight) Temporal Information Signature (e.g., direction of travel, speed, past places visited/path) Fingerprint Signature Voice Print Signature Iris Scan Signature DNA Analysis Signature [0067] The information in a subject dossier is indexed so that it can be used to more
efficiently identify and track suspects and to avoid false alarms. More particularly, a dossier is indexed so that certain information such as soft biometrics can be used to screen candidates within a surveillance for closer study and also to predict likely places within a surveillance region to look for a suspect. For instance, soft biometric information such as clothing color, height and weight may be employed to select
candidates for further investigation. For example, the Knowledge Services may be programmed to cause the Security Agents to search for a match between clothing color in a subject dossier of a suspect and clothing color of unidentified persons in a surveillance region. If a match is found, then the Knowledge Service may cause the Security Agents to perform an analysis of whether facial features in the subject dossier match facial features of the person with matching color clothing. Moreover, temporal information provided in a subject dossier such as direction and speed of travel of a suspect may trigger the Knowledge Services to alert only certain sensory devices, such as a group of cameras in an area of the surveillance region where the suspect is headed, to be on the lookout for the suspect. [00681 A subject dossier may be incremented as more information concerning a
suspect is gathered. For example, initially, only soft biometric information such as clothing color and estimated height and weight might be available. Subsequently, more information such as a facial feature signature or a voice print may become available and will be added to the subject dossier. Newly received data from these multiple sources may be fused with previous data by the Knowledge Services as it is received. [00691 A subject dossier is a record stored in a computer readable medium that can be
easily accessed by security agents and a console operator. The dossier is structured to separate soft biometric information and temporal data from other biometric information. Soft biometric and temporal information generally can be characterized as being easier to obtain and useful for tracking purpose, but not very reliable for definitive identification purposes. Other biometric information, such as fingerprints, voiceprints and an iris scan are more reliable, but more difficult to obtain. Thus, soft biometric and temporal data can be used advantageously to track an individual until more reliable information, such as detailed facial features or fingerprints can be obtained to provide a more reliable identification. Data Gathering Agents
[0070]
The surveillance system of one embodiment employs multiple streams of data
including one or more of, facial features, vocal, fingerprint, iris scan, DNA data, soft biometric data, temporal data and fused data. [0071] Figure 6 is an illustrative flow diagram showing gathering and conversion of
facial feature data to a facial feature signature. Facial feature A comprises a front image of a face that is segmented into a plurality of local areas as an irreducible set of image' building elements to extract a set of local features that can be mapped into a mathematical formula. Facial feature B comprises is a side image that is also separated into a set of irreducible image building elements for extracting local features. Facial feature C comprises a side profile curve that is also collected for use in the identity check and authentication processes. Facial features D and E comprise skin color and tone and hair color. These facial feature data are collected from several video key frames taken from a parallax camera: [0072] These facial feature data are used to produce a facial features signature. In
one embodiment, the Knowledge Services which applies an MPEG-7 descriptor, e.g., a facial recognition descriptor, representing a projection of a face vector onto a set of basis vectors that span the space of possible face vectors and the projection of the face from a side view defined by a profile curve. The face recognition feature sets are extracted from a normalized face image and a normalized profile curve. The normalized face image includes 56 lines with 46 intensity values in each line. The centers of the two eyes in each
face image are located on the
2 4 th
row and the
16 th
and
3 1 st
column for the right and left
eye respectively. This normalized image is then used to extract the one dimensional face vector that includes the luminance pixel values from the normalized face image arranged into a one dimensional vector using a raster scan starting at the top-left corner of the image and finishing at the bottom right corner of the image. The face recognition feature set is then calculated by projecting the one-dimensional face vector onto the space. defined by a set of basis vectors. By using the front image, the side image, the profile curve, the skin color and tone and the hair color, the accuracy of identity authentication is significantly improved.
23.
[00731
A voiceprint signature also can be produced for identity check and
authentication over a telephone, for example. A voiceprint is particularly useful because it is totally noninvasive. In one embodiment, a multi-dimensional voice identification process may be employed to generate a speaker's voice signature by processing pitch contour vectors, time signature, beat number vector and voice shape defined by audio waveforms of the speaker. For example, one embodiment applies pitch models for different pitch intervals, which are defined to be the difference between the semitones of two adjacent nodes: Pitch Interval = [(log (current pitch)-log (previous pitch)]/log 21/12 [00741 Figure 7 is an illustrative flow diagram showing gathering and conversion of
fingerprint feature data to a fingerprint signature. A raw image of a fingerprint is converted into a set of fingerprint codes. The set of codes has a more compact format, e.g., IKENDI Fingerprint Pattern Format, which is based on encoding the friction ridges into a set of direction codes. The coded fingerprint is converted to fingerprint signature in an MPEG-7 descriptor. [0075] Figure 8 is an illustrative flow diagram showing gathering and conversion of
DNA data to a DNA signature. One embodiment of the invention may employ a DNA fingerprint for identification purposes. A complete DNA profile includes 13 short tandem repeats (STRs) with repeats of four or five nucleotides in addition to a sex marker. Each STR has various expected length and is located on different chromosomes or different ends of the same chromosome and each is independently inherited. Figure 8 show respectively the human chromosomes with STR names and locations and three or four different polymorphisms labeled with each of four fluorescent dyes. The DNAs of different lengths are separated by gel electrophoresis. Since it is desirable to detect all different DNAs in one signal identification process, different colors of dyes are used to mark different DNAs that have same length. Appropriate dyes are employed in a PCR operation with STR primers to separate the DNAs based on length and color to get accurate DNA fingerprint in a single DNA identification process. The DNA profile signature is generated in the present invention by using STRs and STR types, e.g., {STR
Name, Type), {STR Name, Type) where STR Names are {TPOX, DSS1358, FGA, D5S818, CSFIPO, D7S820, D8S1179, THO1, VWA, D13S317, D16S539, D18S51, D21S11, SEX, etc.} Types are required to make sure other DNA sequences may use the repeat number of alleles instead of hetero/homozygous, e.g., {Heterozygous, Homozygous}. DNA samples for identity check and authentication may include hair, saliva, and blood. Samples are collected and their signatures are stored in a database. New sample can be collected and analyzed (but not in real time) using DNA arrays/chips, GeneChip, Verigene ID, traditional PCR, or Forensic STR Analysis methods. The result signature will be matched with the signatures in the database.
10076]
Figure 8 illustrates genomic barcodes based on a standard Universal Product
Codes for identifying retailed products by employing ten alternate numerals at eleven positions to generate one hundred billion unique identifiers. One embodiment of the invention applies the barcode techniques for DNA fingerprint identification process. Special considerations are focused on the facts that the repeat polymorphisms are found mainly in intergenic (nongene) regions of chromosomes, especially near the centromeres and that the polymorphisms always exist in a pair in this case, one from each cop of chromosome 1. At a polymorphic locus (location), different numbers of a repeated unit create different alleles. Furthermore, repeated sequences of 9-80 nucleotides are referred to as Variable Number Tandem Repeats (VNTRs). This VNTR has a 16 nucleotide repeat. Repeated sequences of 2 to 8 nucleotides are referred to as Short Tandem. Repeats (STRs). This STR has four nucleotide repeat. In a general genomic barcode system, huge number of string of sites are generated with four altemrnate nucleotides, i.e., adenine, guanine, cytosine, thymine, at each position. A survey of just fifteen of these nucleotide positions would create a possibility of 415, i.e., one billion codes. In the present invention, only fourteen STRs and types are employed to generate barcodes that are easier to analyze with much smaller amount of data to process and that can be more conveniently searched with existing search engine, e.g., Google search engine. Soft biometric information, such as clothing color may be captured using cameras calibrated in accordance with a process disclosed in commonly assigned copending U. S. Patent Application Serial No. Not Yet Known, filed September 16, 2005, [0077]
entitled "Robust Perceptual Color Identification," invented by K. Goh, E. Y. Chang and Y. F Wang, which is expressly incorporated by reference in its entirety into this application through this reference. This patent application addresses a problem of camera-based sensors perceiving an article of clothing as having a slightly different color when viewed from different angles or under different lighting conditions. The patent application proposes the representing color of an article of clothing using a "robust perceptual color". [00781 Data from different modalities may be fused by the Knowledge Services for
classification and identification purposes without suffering the "curse of dimensionality using techniques taught in commonly assigned co-pending U. S. Patent Application Serial No. 11/129,090, filed May 13, 2005, entitled, Multimodal High-Dimensional Data Fusion for Classification and Identification, invented by E. Y. Chang, which is expressly incorporated herein in its entirety by this reference. Data may be incrementally added to a classification and identification process by the Knowledge Services using techniques taught by commonly assigned co-pending U. S. Patent Application Serial No._, filed September 19, 2005, entitled, Incremental Data Fusion and Decision Making, invented by Yuan-Fang Wang, which is expressly incorporated herein in its entirety by this reference. [0079] While the invention has been described with reference to various illustrative
features, aspects and embodiments, it will be appreciated that the invention is susceptible of various modifications and other embodiments, other than those specifically shown and described. The invention is therefore to be broadly construed as including all such alternative variations, modifications and other embodiments within the spirit and scope as hereinafter claimed.
26
CLAIMS What is claimed is: 1. A surveillance method comprising:
using at least one event sensor disposed in a security area of a surveillance region to sense an occurrence of a potential security breach event; using at least one camera with a view of the security area in which the event is sensed to gather biometric information concerning at least one person in the vicinity of the security area at about the time of the sensing of the event; producing a subject dossier corresponding to the at least one person; and matching biometric information of one or more persons captured by one or more other cameras in the vicinity of the at least one camera with corresponding biometric information in the subject dossier. 2. The method of claim 1,
wherein the at least one camera gathers clothing color; and wherein matching involves matching clothing color of one or more persons captured by the one or more other cameras with clothing color in the subject dossier. 3. The method of claim 1,
wherein the at least one camera gathers a facial feature signature; and wherein matching involves matching facial feature signature of one or more persons captured by the one or more other cameras with a facial feature signature in the subject dossier. 4. The method of claim 1,
wherein the at least one camera gathers hair color; and wherein matching involves matching hair color of one or more persons captured by the one or more other cameras with hair color in the subject dossier. 5. The method of claim 1,
wherein the at least one camera gathers skin color; and
wherein matching involves matching skin color of one or more persons captured by the one or more other cameras with skin color in the subject dossier. 6. The method of claim 1,
wherein the at least one camera gathers one or more of clothing color, facial features, skin color and hair color; and wherein matching involves matching at least one of clothing color, facial features, skin color and hair color of one or more persons captured by the one or more other cameras with corresponding one or more of clothing color, facial features, skin color and hair color in the subject dossier. 7. The method of claim 1,
wherein the at least one camera gathers one or both of estimated height and weight; and wherein matching involves matching at least one of height and weight of one or more persons captured by the one or more other cameras with corresponding or both of estimated height and weight in the subject dossier. 8. The method of claim 1,
wherein the at least one camera gathers estimated direction of travel of the at least one person; and further including determining which cameras are in the vicinity of the at least one camera based upon the estimated direction of travel. 9. The method of claim 1,
wherein the at least one camera gathers estimated speed of travel of the at least one person; and further including determining which cameras are in the vicinity of the at least one camera based upon the estimated speed of travel. 10. The method of claim 1,
wherein the at least one camera gathers estimated direction of travel and estimated speed of travel of the at least one person; and further including
determining which cameras are in the vicinity of the at least one camera based upon the estimated direction of travel and the estimated speed of travel. 11. The method of claim 1,
wherein the at least one camera gathers one or more of clothing color, facial features, skin color, hair color, estimated height and estimated weight; wherein matching involves matching at least one of clothing color, facial features, skin color, hair color, estimated height and estimated weight of one or more persons captured by the one or more other cameras with corresponding one or more of clothing color, facial features, skin color, hair color, estimated height and estimated weight in the subject dossier. 12. The method of claim 1,
wherein the at least one camera gathers one or more of clothing color, facial features, skin color, hair color, estimated height and estimated weight; wherein the at least one camera gathers one or both of estimated direction of travel and estimated speed of travel of the at least one person; and further including determining which cameras are in the vicinity of the at least one camera based upon at least one of the estimated direction of travel and the estimated speed of travel; and wherein matching involves matching at least one of clothing color, facial features, skin color, hair color, estimated height and estimated weight of one or more persons captured by the one or more other cameras with corresponding one or more of clothing color, facial features, skin color, hair color, estimated height and estimated weight in the subject dossier. 13. The method of claim 1 further including:
providing to a control console display an image of a person captured by the one or more other cameras having biometric information matching biometric information in the subject dossier. 14. The method of claim 1 further including:
wherein the subject dossier includes at least two biometric signatures; and further including: fusing the at least two signatures and including the fused signature in the subject dossier. 15. The method of claim 1 further including:
wherein the subject dossier includes at least two biometric signatures; and further including: incrementally fusing the at least two signatures and including the fused signature in the subject dossier. 16. The method of claim 1 further including:
wherein the at least one camera with a view of the security area also serves as an event sensor. 17. A surveillance system comprising:
at least one sensor disposed in a security area of a surveillance region to sense an occurrence of a potential security breach event; a plurality of cameras; wherein the at least one camera of the plurality has a view of the security area and can be configured to automatically gather biometric information concerning at least one subject person in the vicinity of the security area in response to the sensing of a potential security breach event in the security area; wherein one or more other of the plurality of cameras can be configured to search for the at least one subject person; a processing system; wherein the processing system is programmable to produce a subject dossier corresponding to the at least one subject person; and wherein the processing system is programmable to match biometric information of one or more persons captured by one or more of the other cameras with corresponding biometric information in the subject dossier.
18.
The system of claim 17,
wherein the at least one camera serves as an event sensor. 19. The system of claim 17,
wherein the processing system is programmable to produce a subject dossier that includes at least one or more of clothing color, facial features, skin color, hair color, estimated height and estimated weight; wherein the processing system is programmable to match at least one of clothing color, facial features, skin color, hair color, estimated height and estimated weight of one or more persons captured by the one or more other cameras with corresponding one or more of clothing color, facial features, skin color, hair color, estimated height and estimated weight in the subject dossier. 20. A surveillance system comprising:
a plurality of sensors disposed in a surveillance region to sense an occurrence of a potential security breach event; a plurality of cameras disposed in the surveillance region, wherein one or more of the plurality of cameras can be configured to automatically gather one or both of biometric information and temporal identifying information concerning at least one subject person in the vicinity of such camera, in response to sensing of a potential security breach event in the vicinity of such camera; wherein one or more of the plurality of cameras can be configured to search for the at least one subject person; a processing system; wherein the processing system is programmable to produce a subject dossier including one or both of biometric information or temporal identifying information corresponding to the at least one subject person; and wherein the processing system is programmable to match one or both of biometric information or temporal identifying information of one or more persons captured by one or more cameras configured to search for the at least one subject person.
31
ABSTRACT OF THE DISCLOSURE A surveillance system is provided that includes at least one sensor disposed in a security area of a surveillance region to sense an occurrence of a potential security breach event; a plurality of cameras is disposed in the surveillance region; at least one camera of the plurality has a view of the security area and can be configured to automatically gather biometric information concerning at least one subject person in the vicinity of the security area in response to the sensing of a potential security breach event; one or more other of the plurality of cameras can be configured to search for the at least one subject person; a processing system is programmed to produce a subject dossier corresponding to the at least one subject person to match biometric information of one or more persons captured by one or more of the other cameras with corresponding biometric information in the subject dossier.
,.ocKet No.: bi'IdJ2UUU-UU ,App .No.. ,1o v'e. ssigneo Inventor: Ken P. CHENG et al. Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND, etc.
v4
1L
BEST AVAILABLE COPY
A pp ,0o.: ,o, t eiAssigneo
.ocKe 1 .4o.. wi adUUUUu Inventor: Ken P. CHENG et al. Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND, etc.
NI
l6
Bt
.App N.o.: . o re Assigneo Joc<et No.: 577832 3200200 Inventor: Ken P. CHENG et al. Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND, etc.
CFn
'im
U=
c-
O) C)
0
a)
im
c
0
I
0. 0.
rArrT
BF-1
%All
AM
r(
r AAILDLG
App No.: Not Yet Assigneo
.JocKet po.: a1 1dauUUUU Inventor: Ken P. CHENG et al. Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND, etc. Sheet 5 of 6
J
AM _ R &
Proxlinex
Mnde~m O
49
iveylw s
aAw
Fig. I is a system diagram of a biometric identity check and authentication system of this invention
mm'i ;;jim.
A
r*-m
mmw
rwwmaum
ITm L'H* coor
Fb na
akC.
PUofhMPa
. SIh
Agent
NMora i0fei
orat
NonAted Cda 5&Ton.

KnowfdpSe srvi
nawg Pofs cur.
0~I
rsc
MPE-7 Dsscutw
SipnowmI
Fig. 2 Diagrams for illustrating the facial identification data implemented in this invention.
BEST AVAILABLE COP
App No.: Not Yet Assigned Inventor: Ken P. CHENG et a1. IDENTIFICATION DETECTION AND, etc.
-~----------.)OCKet No.: 577832000200
Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC
p.
%-; i
PVLL
Docket No.: 577832000200 App No.: Not Yet Assigned Inventor: Ken P. CHENG et al. Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND,etc. Sheet 6 of 6
'i
ell
A.PwR~gpPt
L i
B. Encoed RFvmp t
Fin M6 79 oo
IKEIDIFbowseput
574$MSE r r
I>LJ
Proxbewx
1G msd
srw
Figs. 3A and 3B illustrate the raw-ridge image and the image of the encoded ridge-skeletons after the raw-ridge processed by the fingerprint verification process of this invention.
ALUE CuREN
fD
altow
'W tVe+rr
S
O.6 019
so s va
DaY,30
11N)l
D 11"
DR1T9 7
IfU
VWA
OIaAIsl
nr
A. HUa with SR Nmu& O
Ct/POms Hm OvomoSm al
.Hun= Orw moabld
wh
wth
A. Hunan Onmosonaswth
STR Nana &teasons
RaoresetarDs
00
DNA
Ae***rk
C FlngsrprktImkn3 DA
hInUPEW-i Dewu
Q. Getmn* 3wFCA
Knowh S"e Bever
Figs. 4A and 4B show respectively the human chromosomes with STR names and locations and three or four different polymorphisms labeled with each of four fluorescent dyes.
REST AVAILABLF COP'
Application Data Sheet Application Information Application Type:: Subject Matter:: Suggested Group Art Unit:: CD-ROM or CD-R?:: Sequence submission?:: Computer Readable Form (CRF)?:: Title:: Regular Utility Not Yet Assigned None None No ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEMS Attorney Docket Number:: Request for Early Publication?:: Request for Non-Publication?:: Total Drawing Sheets:: Small Entity?:: Petition included?:: Secrecy Order in Parent Appl.?:: 577832000200 No No 6 Yes No No .
Applicant Information Applicant Authority Type:: Primary Citizenship Country:: Status:: Given Name:: Middle Name:: Family Name:: City of Residence:: State or Province of Residence:: Country of Residence:: Inventor US Full Capacity Ken Prayoon CHENG Saratoga California US
s-200269sv1
Page 1
Initial 09/19/05
Street of mailing address:: City of mailing address:: State or Province of mailing address:: Postal or Zip Code of mailing address:: Applicant Authority Type:: Primary Citizenship Country:: Status:: Given Name:: Middle Name:: Family Name::, City of Residence:: State or Province of Residence::. Country of Residence:: Street of mailing address:: City of mailing address:: State or Province of mailing address:: Postal or Zip Code of mailing address:: Applicant Authority Type:: Primary Citizenship Country:: Status:: Given Name:: Family Name:: City of Residence:: State or Province of Residence:: Country of Residence:: Street of mailing address:: City of mailing address:: State or Province of mailing address:: Postal or Zip Code of mailing address::
20691 Reid Lane Saratoga California 95070
Inventor US Full Capacity Edward Y. CHANG Santa Barbara California US 816 Dorado Drive Santa Barbara California 93111
Inventor US Full Capacity Yuan-Fang WANG Goleta California US 5849 Via Fiori Lane Goleta California 93117
sf-2002696vi
Page 2
Initial 09/19/05
Correspondence Information Correspondence Customer Number:: Representative Information Representative Customer Number:: 20872
20872
Domestic Priority Information Application:: This application Continuity Type:: An application claiming the benefit under 35 USC 119(e) Parent Application:: 60/610,998 Parent Filing Date:: 09/17/2005
Foreign Priority Information
Assignee Information Assignee name:: Proximex Corporation
sf-2002696v
Page 3
Initial 09/19/05
UNITED SIATES'PATENT AND ThADEMARK OFFIcE K OFFIGE AND TRADE

UNITED STATESPATENT
SUNITED STATES DEPARTMENT OF COMMERCE Uunited States Patent and Trademark Ofroe Addres COMMISSlONE.R FOR PATENTS
P.O. Box 1450 AJk niu, Vimiia 22313-1450 www.1apio.jr
APPLICATION NUMBER
FLING OR 371 (c) DATE
FIRST NAMED APPLICANT
ATTORNEY DOCKET NUMBER
11/231,353
09/19/2005
Ken Prayoon Cheng
577832000200 CONFIRMATION NO. 4531 FORMALITIES LETTER
20872 MORRISON & FOERSTER LLP 425 MARKET STREET SAN FRANCISCO, CA 94105-2482
Date Mailed: 10/11/2005
NOTICE TO FILE MISSING PARTS OF NONPROVISIONAL APPLICATION

FILED UNDER 37 CFR 1.53(b) Filing Date Granted Items Required To Avoid Abandonment: An application number and filing date have been accorded to this application. The item(s) indicated below, however, are missing. Applicant is given TWO MONTHS from the date of this Notice within which to file all required items and pay any fees required below to avoid abandonment. Extensions of time may be obtained by filing a petition accompanied by the extension fee under the provisions of 37 CFR 1.136(a). * The statutory basic filing fee is missing. Applicant must submit $ 150 to complete the basic filing fee for a small entity. * The oath or declaration is missing. A properly signed oath or declaration in compliance with 37 CFR 1.63, identifying the application by the above Application Number and Filing Date, is required. Note: If a petition under 37 CFR 1.47 is being filed, an oath or declaration in compliance with 37 CFR 1.63 signed by all available joint inventors, or if no inventor is available by a party with sufficient proprietary interest, is required. The application is informal since it does not comply with the regulations for the reason(s) indicated below. The required item(s) identified below must be timely submitted to avoid abandonment: * Replacement drawings in compliance with 37 CFR 1.84 and 37 CFR 1.121(d) are required. The drawings submitted are not acceptable because: The drawings submitted to the Office are not electronically reproducible. Drawing sheets must be submitted, on paper, which is flexible, strong, white, smooth, non-shiny, and durable (see 37 CFR 1.84(e)). See Figure(s) 1 - 8. Applicant is cautioned that correction of the above items may cause the specification and drawings page count to exceed 100 pages. If the specification and drawings exceed 100 pages, applicant will need to submit the required application size fee. The applicant needs to satisfy supplemental fees problems indicated below.
The required item(s) identified below must be timely submitted to avoid abandonment: * To avoid abandonment, a surcharge (for late submission of filing fee, search fee, examination fee or oath or declaration) as set forth in 37 CFR 1.16(f) of $65 for a small entity in compliance with 37 CFR 1.27, must be submitted with the missing items identified in this letter. SUMMARY OF FEES DUE: Total additional fee(s) required for this application is $565 for a Small Entity * $150 Statutory basic filing fee. * $65 Surcharge. * The application search fee has not been paid. Applicant must submit $250 to complete the search fee. * The application examination fee has not been paid. Applicant must submit $100 to complete the examination fee for a small entity in compliance with 37 CFR 1.27
Replies should be mailed to:
Mail Stop Missing Parts Commissioner for Patents P.O. Box 1450 Alexandria VA 22313-1450
A copy of this notice MUST be returned with the reply.
Office
ial Patent Examination (571) 272-4000, or 1-800-PTO-9199, or 1-800-972-6382 PART 3 - OFFICE COPY
JAN 13 2006
UII Patent S
Approved for use through 07131/2006. OMB 0651-0031 r the Paperwork Reduction Act of 1995, no persons are required to res and to a collection of Information unless itdisplays a valid OMB control number.
and Trademark OfficII S. E PARTMeN;T OF COMMRCE
PTO/SB/21 (09-04)
Application Number
.11/231,353
TRANSMITTAL FORM
(to be used for all correspondence after initial filing)
Filing Date First Named Inventor

Art Unit Examiner Name
September 19, 2005
Ken Prayoon CHENG

2621 Not Yet Assigned 577832000200
Total Number of Pages in This Submission
30
Attomey Docket Number
ENCLOSURES (Check all that apply)

X- Fee Transmittal Form (orig.+copy) (2 pages) J Drawing(s) (8 pages) Ucensing-related Papers Petition
[
] After Allowance
to TC
Annnlc
Communication.
Fee Attached
Appeal Communication to Board of

anrtl InterfRrArr. A
SAmendment/Reply
Appeal Communication to TC (Aooeal Notice. Brief. Reolv Brief) Proprietary Information Status Letter
SAfter Final
SAffidavits/declaration(s)
Pet iti to Convert to a on Provisional Application
with Form PTO/S/96 (6 pages) page)D Terminal Disclaimer [X Extension of Time Request (1
U~
Change of Correspondence Address
Power of Attomey, Revocatio dress
Other Enclosure(s) (please

Identify below): of Drawings (1 page)
[ LInformation Disclosure Statement

Express Abandonment Request
Request for Refund

Wi
(D Submission
[]Certified Copy of Priority Reply to Missing Parts/ to "DReply Missing Parts under 37 CFR 1.52 or 1.53
CD, Number of CD(s) CD. ofCD~) _____ D umbr Landscape Table on CD Remarks
Copy of Notice to File Missing (2 et. pages)cordation Parts, f Assignment with Recordation Form Cover Sheet (5 pages) Declaration for Utility or Design Application Using an Application Data Sheet (1 page)
Application Data Sheet (3 pages)
Incomplete Application
Return Receipt Postcard
Firm Name Signature

Printed name
MORRI
SIGNATURE OF APPLICANT, ATTORNEY, OR AGENT & FOERSTER LLP
Customer No. 20872
obert E. Scheid for Stephen C. Durant January 11, 2006

Reg. No.
Date
42,126 for 31,506
I hereby certify that this correspondence is being deposited with the U.S. Postal Service with sufficient postage as First Class Mail, in an envelope addressed to: Mail Stop MISSING PARTS. Commissioner for Paten P.O. Box 1450, Alexandria, Virginia 22313-1450.
on the date shown below.
Dated: January 11. 2006
Signature:
Todd V. Leone
sf-2062538 v1
0A
'R
PTOISB/17 (12-04v2) Approved for use through 713112006. OMB 0651-0032 U.S. Patent and Trademark Office: U.S. DEPARTMENT OF COMMERCE Under uIQr - 01A ll Reduction ActVI jv. of 1995, noneror QI Iw-ull- to resoond to O r are a collection of information ld .nlres it dislnay .. avairl OMR 'ntrol numher . r l 1 '-Q--lN reuired uuuN --VIIVQ the Paperwork . ... r...
-W ,'IV
i fain ppropjnuons Act,
11
Effective on 127/08/2004.
Tee
s pursuan ote
usat to the Cmc Mowat
#ad
dnn
005 H R M (H J.
R1
M. ).
FEE TRANSMITTAL
For FY 2005
SApplicant claims small entity status. See 37 CFR 1.27 TOTAL AMOUNT OF PAYMENT ($) 665.00
.Application Number Filing Date First Named Inventor Examiner Name Art Unit Attorney Docket No.
Complete if Known 11/231,353 September 19, 2005 Ken Prayoon CHENG i
Not Yet Assigned

2621
577832000200
METHOD OF PAYMENT (check all that apply)
ICheck
Credit Card
Money Order
iNone
Other (please identify):
Deposit Account
Name: Deposit Number 03-1952 Depost Account Account
Morrison & Foerster LLP
For the above-identified deposit account, the Director is hereby authorized to: (check all that apply) fee(s) indicated below, except for the filing fee Charge any additional fee(s) or underpayment of Credit any overpayments fee(s) under 37 CFR 1.16 and 1.17 FEE CALCULATION 1. BASIC FILING, SEARCH, AND EXAMINATION FEES FILING FEES SEARCH FEES EXAMINATION FEES Small Entity Small Entity Small Entity Application Type Fee (t Fe Fee($1) FeJ Fees Paid ($) Fee ) . Fee st Utility 300 150 500 250 200 100 500.00 I Design 200 100 100 50 130 65 Plant 200 100 300 150 160 80 Reissue 300 150 500 250 600 300 Provisional 200 100 0 0 0 0 Small Entity 2. EXCESS CLAIM FEES Fee(S Fee (t Fee Description Each claim over 20 (including Reissues) 50 25 Each independent claim over 3 (including Reissues) 200 100 Multiple dependent claims 360 180 Total Claims Extra Claims Fee($ Fee Paid ($) Multiple Dependent Claims 20 -= 0 x 0 = 0 Fee Fee Paid ($S) 0 0 lndep. Claims Extra Claims Fee(S) Fee Paid ($) 3 -= 0 x 0 0 3. APPLICATION SIZE FEE If the specification and drawings exceed 100 sheets of paper (excluding electronically filed sequence or computer listings under 37 CFR 1.52(e)), the application size fee due is $250 ($125 for small entity) for each additional 50 sheets or fraction thereof. See 35 U.S.C. 41(a)(1)(G) and 37 CFR I.16(s).
Total Sheets - 100 = Extra Sheets 150 Number of each additional 50 or fraction thereof (round up to a whole number) x Fee(S) =Fee P
I Charge fee(s) indicated below
ICharge
aid ($)
4. OTHER FEE(S) Non-English Specification, $130 fec (no small entity discount) Other (e.g., late filing surcharge): 8021 Recording each patent assignment, agreement or ... 2251 Extension for response within first month 2051 Surcharge - late filing fee, search fee, etc.
Fees Paid ($) 40. 00 60. 00 65. 00
' Robert E. Scheid (Reg. No. 42,216) for Stephen C. Durant (Rea. No. 31,506)
Date
I hereby certify that this correspondence is being deposited with the U.S. Postal Service with sufficient postage as First Class Mail, in an envelope addressed to: Box MISSING PARTS. Commissionerr Patents, P. 1450, Alexandria, Virginia 22313-1450, on the date shown below. Dated: January 11, 2006 Signature: Todd V. Leone
sf-2062504 vi
SL ] \k8
k I A
06 4.1 riI
JAz .
.. .... -... .. . ......- r..... 1 ,a.... , .. . .. .. .. .... lw ....... d Y.. ..... ........ . ... ... -.... y pr . . mN. .. lU R JJ I p r' ..... .... lr . Docket Number (Optional) PETITION FOR EXTENSION OF TIME UNDER 37 CFR 1.136(a) FY 2005 577832000200
G7.
Under the Paperwork Reduction Act of 1995. no
(12-04) qCPTO/SB/22 Approved for use through 7/31/2006. OMB 0651-0031 . U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE dinlavs a valid OMR nntrnl numhprrersons are reauired to resoond to a collection of infrmationunle if
(Fees pursuant to the Consolidated Appropriations Act, 2005'(H.R. 4818).)
Application Number
11/231,353
Filed
September 19, 2005
ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND ForSURVEILLANCE SYSTEMS Art Unit 2621 Examiner Not Yet Assigned
This is a request under the provisions of 37 CFR 1.136(a) to extend the period for filing a reply in the above identified application.
The requested extension and fee are as follows (check time period desired and enter the appropriate fee below):
One month (37 CFR 1.17(a)(1)) Fee $120 $450 $1020 $1590 $2160 Small Entity Fee $60 $225 $510 $795 $1080 $ $ $ $ $ 60.00
Two months (37 CFR 1.17(a)(2)) Three months (37 CFR 1.17(a)(3)) Four months (37 CFR 1.17(a)(4)) Five months (37 CFR 1.17(a)(5))
D
D D
FI O I-I O
Applicant claims small entity status. See 37 CFR 1.27. A check in the amount of the fee is enclosed. Payment by credit card. Form PTO-2038 is attached. The Director has already been authorized to charge fees in this application to a Deposit Account. The Director is hereby authorized to charge any fees which may be required, or credit any overpayment, to Deposit Account Number 03-1952 c.. . Fee Transmittal form (PTO/SB/17) is attached to this submission in duplicate.
I am the
applicant/inventor. assignee of record of the entire interest. See 37 CFR 3.71. Statement under 37 CFR 3.73(b) is enclosed. (Form PTO/SB/96). attorney or agent of record. Registration Number
D D
see below January 11, 2006 Date
Robert E. Scheid - Reg. No. 42,126 for Stephen C. Durant - Reg. No. 31,506 Typed or printed name
than one signature isrequired, see below.
(415) 268-6982 Telephone Number
NOTE: Signatures of all the inventors or assignees of record of theentire interest or theirrepresentative(s) are required. Submit multiple forms if more
S
05 FC:2251
Total of
forms aresubmitted.
01/17/2006 NNGUYEN1 00000032 031952 60.00 DA
li231353 11231353
I hereby certify that this correspondence is being deposited with the U.S. Postal Service with sufficient postage as First Class Mail, in an envelope addressed to: Mail Stop MISSING PARTS. Commis for P.O. Box 1450, Alexandria, Virginia 22313-1450,
on the date shown below. Dated: January 11, .2006
, Signature:
'
f.
4-7
Todd V. Leone
sf-2062483 v1
Docket No.: 577832000200 Appl. No.: 11/231,353 Inventor: Ken P. CHENG et al. Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEMS REPLACEMENT SHEET Sheet 1 of 8)
'ai
0r-,
mD
rt
0 -I -I
-U
'1 c.
C. 0
3D
A
I0
.0j
BEST AVAILABLE COPY
Appl. No.: 11/231,353 Docket No.: 577832000200 Inventor: Ken P. CHENG et al. Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEMSH REPLACEMENT SHEET. Sheet 2 of 8.,
n
0
C
y
*.
mm
0 MO
: :,.:.
0.
Appl. No.: 11/231,353 Docket No.: 5778320002C 00 Inventor: Ken P. CHENG et al. Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEMS REPLACEMENT SHEET Sheet 3 o f 8
- --
-- ---
- N
*0
0
-I
-n s~
0 Ch,
MIL
2)'
00 Docket No.: 5778320002C Appl. No.: 11/231,353 Inventor: Ken P. CHENG et al. Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEMS Sheet 4 o f8 REPLACEMENT SHEET
-------------\
A 'v
'1
e
41
A e
s
e 3
gib
s'
Docket No.: 577832000200 Appl. No.: 11/231,353 Inventor: Ken P. CHENG et al. Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE. SYSTEMSH REPLACEMENT SHEET Sheet 5 of 8
..
. .. . . .- -..
..
. ..
..
..
..
. . .
. .
. .
. . .
. .
. .
.. .
. .
. .
. .
-MI
'1
I
Appl. No.: 11/231,353 Docket No.: 577832000200 Inventor: Ken P. CHENG et al. Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEMS REPLACEMENT SHEET Sheet 6 of 8
>1
D~
-T
F0
CD
W
0v
'1
~
-'1
a,
5
e"I
O
0f
gIITHI
111111~a
C7
!IIlla
Appl. No.: 11/231,353 DocKet No.: 5778320002(00 inventor: Ken P. CHENG et al. Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEMS Sheet 7 ofif8)I REPLACEMENT SHEET
-_ ....,_________
8a
t'
mp&-'
0 0
%B
'1
ti
0t K
0*
km
N,
Docket No.: 577832000200 Appl. No.: 11/231,353 Inventor: Ken P. CHENG et al. Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEMS Sheet 8 of 8 REPLACEMENT SHEET
"
'D
Z
Di
I I O6 q
I
'1 .
mU
Iiacari;[a:ima
w 0
.6"
ao = =.....= ... ;
Ni
.J4.1
.1000:luflg[a a
1tii.1aisai
if
.,
IIijlI#E4aalr:uII
M~
*D
ft
a
(0
I!
ir
El
z
.1
"1 ,mt
IJJ'-L
BII
Pao/sa~o 0446)
Urdetim Ppeiwoil
eucs
aa1 9&
no
en
m .
tei soob"ob~n
U. patentOWTre wmnk ofs: U.& oEPARTMWNT ofCooMME1E nr~ut. t. n u f w l tyvbo " v~aleoe
POWER OF ATTORNEY TO PROSECUTE APPUCA710NS BEFORE THE USPTO
I hereby revoke all previous powers 37 CFR 3.73(b).

I
of attorney given i
the application Iderntfled in tme attached statement under
[]Ptsutlonei
OR
N) mined below (MVopncwUln paenet practUoners

Nam
an
lbbe named,
then a cia Mmremer must be useak
Nane
NaneRe
t dMuelpned ba~ wlmftmitd Stems Patent nd Tademk OIke (USPTO) In connection pm as eftomunju) .ror n es) repen e ie b USPTO essigrran ns.cds oraseipgment doanits eny and ci pausal eppir , s sae enad Op Mtie w~aeilm oem tb aced to toInn n mcewndance wth 37CFR 3.73(0).
to
Plasea
j] The addsussinodhted wthCusbmwerNumber:

F~inne
1 ky
dienge tiehe snadenca address for the .ppleadon ldutlld hI Vwattahedulaentunder 37 CFR 3.73(b) to:
Istus
Telephone
IName
EmmA
Caunby
Aeskpm Name and Addmes:

Proxkmx Corptxation 6 Results Way
CuperdM.oCaIfOmia 95014
A copy awn
Ohe psactdo m
t
flied ineach epptlcadtlon hiwfldhOft sarmniaused.

miat
ar it-kItahis fore Nthe
of forut opethe fhb

le
wilh a .beeenulmde 37 CFR 3.73(b) (Form PTOI88IS or eqiduland)
Inwhich Viii Power
appointed preditloner Insmsarked toasWon

of Attorney Is to be fiLed
The
iemnt aider 37 CFR LT7*() may be canylated
Is reikd to be by on. of behalf of the asslgneaa
The invidwiJ
wnose seta-
and ritehksuqplied beow.s anmht
d aon behalf ofihe assignee
Signature
Namfe
I AIL
ge
(I&99.-ADate
TWOe
k KW PRq OoW
-"Telephone Lfp-'7
2L6 os
g-
I hereby certify that this correspondence is being deposited with the U.S. Postal Service with sufficient postage as First Class Mail, in an envelope addressed to: Mail Stop MISSING PARTS, Commissioner for Patents P . Box 1450, Alexandria. Virginia 22313-1450, on the date s n w. Dated: January 11, 2006 Signature:
o'-0-PF
&C
Todd V.Leone
Docket No.: 577832000200 (PATENT)
JAN 13 2006
IN THE UNITED STATES PATENT AND TRADEMARK OFFICE e Patent Application of Ken Prayoon CHENG et al.
Application No.: 11/231,353 Filed: September 19, 2005 For: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEMS
Confirmation Number: 4531 Art Unit: 2621 Examiner: Not Yet Assigned
SUBMISSION OF DRAWINGS
Mail Stop MISSING PARTS Commissioner for Patents P.O. Box 1450 Alexandria, Virginia 22313-1450 Dear Sir: Submitted herewith is one set (eight sheets, eight figures) of drawings for filing in the above-identified Patent application. Kindly substitute the enclosed drawings for the drawings submitted with the originally filed application.
Dated: January 11, 2006
Respectfully submitted,
/ Robert E. Scheid - Reg. No. 42,126

for Stephen C. Durant - Reg. No.: 31,506 MORRISON & FOERSTER LLP 425 Market Street San Francisco, California 94105-2482 Tel.: (415) 268-6982 Fax: (415) 268-7522
sf-2062451 vI
Page 1 of 2
AND TRADEMARK OFFICE

)PCOMI51ERCE
ark
ofic
www.uptlo'v-
APPLICATION NUMBER
FILING OR 371 (c)DATE
ATTORNEY DOCKET NUMBER
11/231,353
09/19/2005
Ken Prayoon Cheng
577832000200 CONFIRMATION NO. 4531 FORMALITIES LETTER
NOTICE TO FILE MISSING PARTS OF NONPROVISIONAL APPLICATION

01/17/2006 NNGUYEHN1 00000032 031952 01 02 03 04 FC:2011 FC:2051 FC:2111 FC:2311 150.00 65.00 250.00 100.00 DA DA DA DA 11231353 FILED UNDER 37 CFR 1.53(b) UNDER 37 CFR 1.53(b)
Filing Date Granted
Filing Date Granted
Items Required To Avoid Abandonment: An application number and filing date have been accorded to this application. The item(s) indicated below, however, are missing. Applicant is given TWO MONTHS from the date of this Notice within which to file all required items and pay any fees required below to avoid abandonment. Extensions of time may be obtained by filing a petition accompanied by the extension fee under the provisions of 37 CFR 1.136(a). * The statutory basic filing fee is missing. Applicant must submit $ 150 to complete the basic filing fee for a small entity. * The oath or declaration is missing. A properlysigned oath or declaration in compliance with 37 CFR 1.63, identifying the application by the above Application Number and Filing Date, is required. Note: If a petition under 37 CFR 1.47 is being filed, an oath or declaration in compliance with 37 CFR 1.63 signed by all available joint inventors, or if no inventor is available by a party with sufficient proprietary interest, is required. The application is informal since it does not comply with the regulations for the reason(s) indicated below. The required item(s) identified below must be timely submitted to avoid abandonment: * Replacement drawings in compliance with 37 CFR 1.84 and 37 CFR 1.121(d) are required. The drawings submitted are not acceptable because: The drawings submitted to the Office are not electronically reproducible. Drawing sheets must be submitted on paper, which is flexible, strong, white, smooth, non-shiny, and durable (see 37 CFR 1.84(e)). See Figure(s) 1 - 8. Applicant is cautioned that correction of the above items may cause the specification and drawings page count to exceed 100 pages. If the specification and drawings exceed 100 pages, applicant will need to submit the required application size fee.
The applicant needs to satisfy supplemental fees problems indicated below.
Page 2 of 2
The required item(s) identified below must be timely submitted to avoid abandonment: * To avoid abandonment, a surcharge (for.late submission of filing fee, search fee, examination fee or oath or declaration) as set forth in 37 CFR 1.16(f) of $65 for a small entity in compliance with 37 CFR 1.27, must be submitted with the missing items identified in this letter. SUMMARY OF FEES DUE: Total additional fee(s) required for this application is $565 for a Small Entity * $150 Statutory basic filing fee. * $65 Surcharge. * The application search fee has not been paid. Applicant must submit $250 to complete the search fee. * The application examination fee has not been paid. Applicant must submit $100 to complete the examination fee for a small entity in compliance with 37 CFR 1.27
Replies should be mailed to:
Mail Stop Missing Parts Commissioner for Patents P.O. Box 1450 Alexandria VA 22313-1450
A copy of this notice MUST be returned with the reply.
Office
n',atent Examination (571) 272-4000, or 1-800-PTO-9199, or 1-800-972-6382

PART 2 - COPY TO BE RETURNED WITH RESPONSE
JN1IPTO/SBl95
(09-04)
Approved for use through 07/31/2006. OMB 0651-0031 U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE work Reduction Act of 1995. no persons are required to respond to a collection of information unless it disiays a vad OMB control number
STATEMENT UNDER 37 CFR 3.73(b)

Applicant/Patent Owner: Application No.IPatent No.: Entitled: Ken Prayoon CHENG et al. 11/231,353 Filed/Issue Date: September 19, 2005
ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEMS Proximex
corporation
(Type of Assignee, e.g.,corporation, partnership, university, government agency etcc.)
(Name of Assignee)
states that it is: 1. 2.
the assignee of the entire right, title, and interest; or
an assignee of less than the entire right, title and interest. The extent (by percentage) of its ownership interest is in the patent application/patent identified above by virtue of either: A. R
An assignment from the inventor(s) of the patent application/patent identified above. The assignment was recorded In the United States Patent and Trademark Office at Reel, Frame , or for which a copy thereof is attached. A chain of title from the inventor(s), of the patent application/patent identified above, to the current assignee as shown below:
1. From: To:
OR B.
The document was recorded in the United States Patent and Trademark Office at Reel , Frame , or for which a copy thereof is attached.
2. From: To:
The document was recorded in the United States Patent and Trademark Office at Reel , Frame , or for which a copy thereof is attached.
3. From: To:
The document was recorded in the United States Patent and Trademark Office at Reel , Frame , or for which a copy thereof is attached. Additional documents in the chain of title are listed on a supplemental sheet.
Copies of assignments or other documents in the chain of title are attached. [NOTE: A separate copy (i.e., a true copy of the original assignment document(s)) must be submitted to Assignment Division in accordance with 37 CFR Part 3, if the assignment is to be recorded in the records of the USPTO. See MPEP 302.08] tit
The undersigned
)f
Signature
elow) is authorized to act on behalf of the assignee. .
January 11. 2006

Date (415) 268-6982 Telephone Number
Robert E. Scheid - Reg. No. 42,126 fnr StPnh.n C ulrant - RPn Nn '1 .snR' Printed or Typed Name Attomey or agent under 37 CFR 1.34 Title
I hereby
certify that
this
correspondence
is being
deposited
with the
U.S. Postal
Service with sfficient
postage
as
First Glass
Mail, in
sf-2062491 v1
Attorney Docket No.: 577832000200
ASSIGNMENT JOINT THIS ASSIGNMENT, by Ken Prayoon CHENG, Edward Y. CHANG, and Yuan-Fang WANG (hereinafter referred to as the assignors), residing at 20691 Reid Lane, Saratoga, CA 95070; 816 Dorado Drive, Santa Barbara, CA 93111; and 5849 Via Fiori Lane, Goleta, CA 93117, respectively, witnesseth: WHEREAS, said assignor has invented certain new and useful improvements in ADAPTIVE MULTIMODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEMS, set forth in an application for Letters Patent of the United States, bearing Serial No. 11/231,353 and filed on September 19, 2005; and WHEREAS, Proximex Coroporation, a corporation duly organized under and pursuant to the laws of California and having its principal place of business at 6 Results Way, Cupertino, CA 95014 (hereinafter referred to as the assignee) is desirous of acquiring the entire right, title and interest in and to said inventions and said application for Letters Patent of the United States, and in and to any Letters Patent or Patents, United States or foreign, to be obtained therefor and thereon: NOW, THEREFORE, in consideration of One Dollar ($1.00) and other good and sufficient consideration, the receipt of which is hereby acknowledged, said assignor has sold, assigned, transferred and set over, and by these presents does sell, assign, transfer and set over, unto said assignee*, its successors, legal representatives and assigns, the entire right, title and interest in and to the above-mentioned inventions, application for Letters Patent, and any and all Letters Patent or Patents in the United States of America and all foreign countries which may be granted therefor and thereon, and in and to any and all divisions, continuations and continuations-in-part of said application, or reissues or extensions of said Letters Patent or Patents, and all rights under the International Convention for the Protection of Industrial Property, the same to be held and enjoyed by said assignee, for its own use and the use of its successors, legal representatives and assigns, to the full end of the term or terms for which Letters Patent or Patents may be granted, as fully and entirely as the same would have been held and enjoyed by the assignor, had this sale and assignment not been made. AND for the same consideration, said assignor hereby covenants and agrees to and with said assignee its successors, legal representatives and assigns, that, at the time of execution and delivery of these presents, said assignor is the sole and lawful owner of the entire right, title and interest in and to said inventions and the application for Letters Patent above-mentioned, and that the same are unencumbered and that said assignor has good and full right and lawful authority to sell and convey the same in the manner herein set forth. AND for the same consideration, said assignor hereby covenants and agrees to and with said assignee, its successors, legal representatives and assigns, that said assignor will, whenever counsel of said assignee, or the counsel of its successor, legal representatives and assigns, shall advise that any proceeding in connection with said inventions, or said application for Letters Patent, or any proceeding in connection with Letters Patent for said inventions in any country, including interference proceedings, is lawful and desirable, or that any division, continuation or continuation-in-part of any application for Letters Patent or any reissue or extension of any Letters Patent, to be obtained thereon, is lawful and desirable, sign all papers and documents, take all lawful oaths, and do all acts necessary or required to be done for the procurement, maintenance, enforcement and defense of Letters Patent for said inventions, without charge to said assignee, its successors, legal representatives and assigns, but at the cost and expense of said assignee, its successors, legal representatives and assigns.
sf-2045551
Attoray Docket No.: 577832000200
AND said assignor hereby requests the Commissionar of Patants to issue said Letters Patent of the United States to said assignee as the assignee of said inventions and the Letters Patent to be issued theron for the sole use
of said assignee, its suocessos, legal representiv
and assigns.
DatI4/o5
J
KEso Payoon CHENG
1W
Daft
Edward Y. CHANG
Dat
Yen-Fang WANG
sf-2045551
AND said assignor hereby requests the Commissioner of Patents to issue said Letters Patent of the United States to said assignee as the assignee of said inventions and the Letters Patent to be issued thereon for the sole use of said assignee, its successors, legal representatives and assigns.
8
-J
Date
Ken Prayoon CHENG
CD~
/Date Date
Edward Y. CHANG
Date
Yuan-Fang WANG
sf-2045551
AND said assignor hereby requests the Commissioner of Patents to issue said Letters Patent of the United States to said assignee as the assignee of said inventions and the Letters Patent to be issued thereon for the sole use of said assignee, its successors, legal representatives and assigns.
Date
Ken Prayoon CHENG
Date
EdwaY. CHANG
Date Date
u -Fang WANG
sf-2045551
Ja
PTOr58l01A (08a4) Appiored br U.S Roo cfonAc d Poft end Taidam* us. Oft*;
t9gkimpmemcnii
ftoughO1A1r WS OM; OSS14M UAL DEPARTMENT OF COMMERCE

Ro cs a Oe18alw wMmu ho nuRba. 0 -
edb ~
~VwY
bl
aarc9aldlnbrma0onv+le M w. .M
,-0WwVWz
OR DESIGN APPUCATION USING AN DECLARATION (37 CFR 1.63) FOR UTIL APPLICATION DATA SHEET (37 CFR 1.76) @f IIVS
ADAPTIVE MULTMODAL INTEGRATED BIOMETRIC IDENTIFICATION
DETECTION AND SURVEILLANCE SYSTEMS
declaretant As he below named kwentur(s), Uwe
Tes dedaralin is directed to
EJ me attached aonab
L We bdereva
ht
[ AppocaOn Na
11/231,353
an
, filed an
0911912005
(H1
El
n amended
Ofawble)
a patent
we ainme o okinal md list Irwtat") t te aut~ec ma r whimch is c~imed

of tabo ied appilcat Ictudiog
and wch for
t:
amendment specIcaly rwfened to above;
Ue have reviews and wderstand fte cnts
clak, as amended by way
became available been ie fig date n Oe pior applcaton and

continane alapicdn
e anowlaedge te dty to dtsclose to the Unted States Patent and Trademark Oface a if.omaon knw to mea s to be matedal to patanabMliy as da ined in37 CFR 1.5, bctud t for mtinua*n ert appam mateeal i umn whi
ft
nationa or PCT
htomallonal Bil
data at
Oe
AI stdltsments made herein d myowr owam knoxmlmde 4 e u, m staments made herein on bomatlon end belief mare beleved to be trum aid faeuat lease eatement were made thGe kntowedge that wM fale statements and the e are purnsatite by tie or ilmprimstw, or both, uder 18 U.S.C. 1001, and may jeopardbe the vslyid atthe appcation or
aty paten ibsubigthemon.
FULL NAME OF INVENTOR(S)

Inventor
one
__
Ken Pr
CHENG
Ctizen of
United States
Sinatue:
Inventor two: Snatur:
Edward Y.CHANG Cuzen ot Yuan-FanR WANG aitzen oa United States United States
Inwentor tree:
Slpate:C
gaato
AdBla
Otur nvesoroma Wga emstamve ebeb ned an
f: adlona
(s) aaed heto.
Page li
,-2045545
JAN 13 2006
SB/1A (09-04) Apprmved for use through 07/31/2006. OMB 0851-0032 U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE r the Paperwork Reduction Act of1995, no persons requiredrespond to collection of information it dhp are to a unless a valid OMB ontrol nurner. " cPTO
DECLARATION (37 CFR 1.63) FOR UTILITY OR DESIGN APPLICATION USING AN APPLICATION DATA SHEET (37 CFR 1.76)
Title of Invention
.... ..
to
ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION
DETECTION AND SURVEILLANCE SYSTEMS
8
.J
, CD
As the below named inventor(s), I/we declare that: This declaration Is directed to:
]
W
The atlached application, or
Application No.
11/231,353
,filed on
09/19/2005
(if applicable);
as amended on
Iwe believe that I/we am/are the original and first Inventor(s) of the subject matter which is claimed and for which a patent is sought; I/we have reviewed and understand the contents of the above-identified application, including the claims, as amended by any amendment specifically referred to above; I/we acknowledge the duty to disclose to the United States Patent and Trademark Office all Information known to me/us to be material to patentability as defined in 37 CFR 1.56, including for continuation-in-part applications, material information which became available between the filing date of the prior application and the national or PCT intemrnational filing date of the continuation-in-part application. All statements made herein of my/our own knowledge are true, all statements made herein on information and belief are believed to be true, and further that these statements were made with the knowledge that willful false statements and the like are punishable by fine or Imprisonment, or both, under 18 U.S.C. 1001, and may jeopardize the validity of the application or any patent issuing thereon.
FULL NAME OF INVENTOR(S)
Inventor one:
Ken Prayoon CHENG
Signature: Inventor two:

Signature:
Citizen of: Edward Y. CHANG / .Citizen

of:,
United States
United States
inventor three:
Signature:
Inventor four.
Yuan-Fang WANG
Citizen of: United States
Signature:
Citizen
of:
additional form(s) attached hereto.
] Additional Inventors ora legal represenatve are being named on
Page 1 of I
sf-2045545
'.
.-
o'P
JAM 13 ?006
u- i:dmimum db ae. o
Lis:
rfl o
DECL:ARAlON p(37CFR 1.63)FOR IITlfY OR'DEStGN :APPuCAT~1N UIG Vii APPUCA1!ON DATASW (3 FR1J76
7ii~~o ir As'
EGRA'TED:8WONE?'RC'IDENT1CATION r:ADAP7 VE R1UL1Ti-MDALI N _DETECTtON ANdD :SRVEtZ:ACE STEMS: tbe Gow'ri18gi enlmtir s). M~se ears. a
Ca)
LIII
J m m w IE4
Uet e heuweQamid~zwsaid ;
aeat1don
091 .......... n
00
cp~sd. ie"liaa
Uid:
aimr
wzen: b. yy
!Aa ::
ri adt~p,"rtby,:
geu12Od eal
161?
ot
~e~ti
e
im
;~ii n~i
Kga ri
t Sk
orhirrt
isi~
ri.+Zds %
Pte1 oiy isT ue a p r o~d~a'r cu ; At?

g" : .A
nt~ia t 'Go
,,
1ad;"tlei:Cn mea
+ . /1!4:q%
i:[T n re".wlpaCV .b~o

..
-
r:td."2S8 d
AD
6W
ije
7a, {
ajr bu orhrsrte;a.d,.iavir
18.t ;S:CS 1;'andm if eI
oU
a2 ev afd c)i s
'WR7
au
ism
p c
KenPraycon CHENG
__________________
U:1ni' ed Stats
Ediiard Y_. CHANG0
iln
Uem Y
FangWANG
______________
ir4O f
or
muCprsco
met
a:
na
anl
XIAA
!Paga1I
s -2045545
cu
o'P
JAN t g10 pp0lication Data Sheet Application Information Application Type:: Subject Matter:: Suggested Group Art Unit:: CD-ROM or CD-R?:: Sequence submission?:: Computer Readable Form (CRF)?:: Title:: Regular Utility Not Yet Assigned None None No ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEMS 577832000200 No No 8 No No No
Attorney Docket Number:: Request for Early Publication?:: Request for Non-Publication?:: Total Drawing Sheets:: Small Entity?::' Petition included?:: Secrecy Order in Parent Appl.?:: Applicant Information Applicant Authority Type:: Primary Citizenship Country:: Status:: Given Name:: Middle Name:: Family Name:: City of Residence:: State or Province of Residence:: Country of Residence:: Street of mailing address:: City of mailing address:: State or Province of mailing address::
sf-2062435 v1
Inventor US Full Capacity Ken P. CHENG Saratoga California US 20691 Reid Lane Saratoga California Page 1
initial 01111/2006
Applicant Authority Type:: Primary Citizenship Country:: Status:: Given Name:: Middle Name::. Family Name:: City of Residence:: State or Province of Residence:: Country of Residence:: Street of mailing address:: City of mailing address:: State or Province of mailing address:: Postal or Zip Code of mailing address::
Inventor US Full Capacity Edward

Y.
CHANG Santa Barbara California
US
20691 Reid Lane Saratoga California 95070-5325 Inventor Full Capacity
Applicant Authority Type:: Status:: Primary Citizenship Country:: Given Name:: Family Name:: City of Residence:: State or Province of Residence:: Country of Residence:: Street of mailing address:: City of mailing address:: State or Province of mailing address:: Postal or Zip Code of mailing address:: Correspondence Information Correspondence Customer Number::
US
Yuan-Fang WANG Goleta California US 5849 Via Fiori Lane Goleta California 93117-1838
20872
sf-2062435 vl
Page 2
initial 01/11/2006
Representative Information Representative Customer Number:: Domestic Priority Information 20872
Foreign Priority Information
Assignee Information Assignee name:: Proximex
sf-2062435 v1
Page 3
initial 01/1112006
This Page is Inserted by IFW Indexing and Scanning Operations and is not part of the Official Record BEST AVAILABLE IMAGES
Defective images within this document are accurate representations of the original documents submitted by the applicant.
Defects in the images include but are not limited to the items checked:
Q BLACK BORDERS Q IMAGE CUT OFF AT TOP, BOTTOM OR SIDES
O FADED
TEXT OR DRAWING
Q BLURRED OR ILLEGIBLE TEXT OR DRAWING
SKEWED/SLANTED IMAGES COLOR OR BLACK AND WHITE PHOTOGRAPHS
] GRAY SCALE DOCUMENTS
O
L
LINES OR MARKS ON ORIGINAL DOCUMENT REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY OTHER:
IMAGES ARE BEST AVAILABLE COPY. As rescanning these documents will not correct the image problems checked, please do not report these problems to
the IFW Image Problem Mailbox.
Fee Transmittal Form,
D Drawing(s)
SAfter Allowance Communication

to TC D Appeal Communication to Board of Appeals and Interferences SAppeal Communication to TC (Appeal Notice, Brief, Reply Brief)
D Fee Attached
SAmendment/Reply
D Licensing-related Papers
]
Petition
D After Final
SPetition to Convert to a
]
D
Affidavits/declaration(s)
[]
D
I I
Provisional Application
L]
Proprietary Information
Power of Attorney, Revocation

Terminal Disclaimer
SStatus Letter
Other Enclosure(s) (please Identify below): (CD (1) copy of each of twelve One (12) cited non-patent literature reterences
Extension of Time Request Express Abandonment Request Information Disclosure Statement (5 pages)
D
x
Request for Refund CD, Number of CD(s)
] Certified Copy of Priority Document(s) D

Reply to Missing Parts/ Incomplete Application
) Return
D Landscape Table on CD
Receipt Postcard
Remarks Total page count of this submission as cited above does not include the enclosed copies of cited references.
Reply to Missing Parts under 37 CFR 1.52 or 1.53
1'
SIGNATURE Cq APPLICANT, ATTORNEY, OR AGENT

Firm Name
MO
IS
N & FOERST
LLP
Customer No. 20872
Signature Printed name Date Stephen C. Durant
October 13, 2006
Reg. No.
31,506
I hereby certify that this paper (along with any paper referred to as being attached or enclosed) is being deposited with the U.S. Postal Service on dressed to: Mail Stop Amendment, Commissioner for envelope the date shown below with sufficient postage as First Class Mail, in Patents. P.O. Box 1450, Alexandria, Virginia 22313-1450. Dated: October 13. 2006 Signature: Todd V. Leone
sf-2209783 vi
:;;r
SPaperwork Reduction Act of 1995, no persons
PTO/SB/21 (09-04) Approved for use through 0713112006. OMB 0651-0031 U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE are required to resond to a conltection of information unless it.disolays.a valid.OMB control number. L . I .. I.. IIG . r--, . . . .... .. ....... .... . _. _ . ,._ .
Application Number
I 11/231,353
TRANSMITTAL

Art Unit
September 19, 2005 Ken P. CHENG. 2621

Not Yet Assigned
FORM
Examiner Name Total Number of Pages in This Submission 6 Attomey Docket Number
577832000200
'
Fee Transmittal Form L
uto
Drawing(s) Licensing-related Papers Petition Li(Appeal Petition to Convert to a Provisional Application Power ofAttomey, Revocation Change of Correspondence Address Terminal Disclaimer
ess
After Allowance Communication

TC
SFee Attached
] Amendment/Reply
O
F
Appeal Communication to Board of

Appeals and Interferences Appeal Communication to Brief) Notice, Brief, Reply TCe Proprietary Information
j
[
After Final Affidavitsdeclaration(s)
D F
Status Letter (please
SExtension of Time Request:

Express Abandonment Request
-OtherEnclosure(s)
Request for Refund CD, Number of CD(s) CD,Number ofCD(s) Landscape Table on CD
O X(5
F
D One (1) copy of each of twelve (12) cited non-patent literature

references Return Receipt Postcard
Information Disclosure Statement pages)
Certified Copy of Priority Document(s)
SReply to Missing Parts/

Incomplete Application
Remarks
] Reply to
Missing Parts under
37 CFR 1.52 or 1.53
Total page count of this submission as cited above does not include the enclosed copies-of cited references.
SIGNATURE9 Firm Name Signature Printed name Date
APPLICANT, ATTORNEY, OR AGENT
MO
IS N & FOERST
LLP
Customer No. 20872
Stephen C. Durant October 13, 2006

Reg. No.
131,506
I hereby certify that this paper (along with any paper referred to as being attached or enclosed) is being deposited with the U.S. Postal Service on envelope adressed to: Mail Stop Amendment. Commissioner for the date shown below with sufficient postage as First Class Mail, in Patents, P.O. Box 1450, Alexandria, Virginia 22313-1450. Dated: October 13. 2006 Signature: Todd V. Leone
sf-2209783 v1
I hereby certify that this correspondence and any eing deposited with the U.S. Postal Service sufficient postage as First Class Mail, in an Amendment. Commissioner for Patents, lia 2313-1450.
:
enclosures referenced therein

on the date first shown below envelope addressed to: Mail P.O. Box 140, Alexandria,
October 13,
2006
Signature:
Patent Docket No. 577832000200
IN THE UNITED STATES PATENT AND TRADEMARK OFFICE

In re Patent Application of: Ken P. CHENG et al. Serial No.: 11/231,353 Filing Date: September 19, 2005 For: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEMS
I I
Examiner: Not Yet Assigned Group Art Unit: 2621
INFORMATION DISCLOSURE STATEMENT UNDER 37 C.F.R. 1.97 & 1.98 Mail Stop AMENDMENT Commissioner for Patents P.O. Box 1450 Alexandria, Virginia 22313-1450 Dear Sir: Pursuant to 37 C.F.R. 1.97 and 1.98, Applicants submit for consideration in the above-identified application the documents listed on the attached Form PTO/SB/08a/b. Copies of foreign documents and non-patent literature are submitted herewith. Document no. 10 on the attached Form PTO/SB/O8a/b is a pending non-published U.S. patent application, and in accordance with the Waiver of the Copy Requirement in 37 CFR 1.98 for Cited Pending U.S. Patent Applications, a copy is not submitted herewith. The Examiner is requested to make these documents of record.
sf-2185936
Application No. 11/231,353 Filed: 09/19/2005 Information Disclosure Statement
.2
Docket No. 577832000200
This Information Disclosure Statement is submitted:
With the application; accordingly, no fee or separate requirements are required. Before the mailing of a first Office Action after the filing of a Request for Continued Examination under 1.114. However, if applicable, a certification under 37 C.F.R. 1.97 (e)(1) has been provided.
Within three months of the application filing date or before mailing of a first Office Action on the merits; accordingly, no fee or separate requirements are required. However, if applicable, a certification under 37 C.F.R. 1.97 (e)(1) has been provided.
After receipt of a first Office Action on the merits but before mailing of a final Office Action or Notice of Allowance.
L
I I I
A fee is required. A check in the amount of this submission in duplicate.
is enclosed.
A fee is required. Accordingly, a Fee Transmittal form (PTO/SB/17) is attached to A Certification under 37 C.F.R. 1.97(e) is provided above; accordingly; no fee is believed to be due.
After mailing of a final Office Action or Notice of Allowance, but before payment of the issue fee.
O L
A Certification under 37 C.F.R. 1.97(e) is provided above and a check in the
amount of
is enclosed.
A Certification under 37 C.F.R. 1.97(e) is provided above and a Fee Transmittal form (PTO/SB/17 is attached to this submission in duplicate.) Applicants would appreciate the Examiner initialing and returning the Form
PTO/SB/08a/b, indicating that the information has been considered and made of record herein. The information contained in this Information Disclosure Statement under 37 C.F.R. 1.97 and 1.98 is not to be construed as a representation that: (i) a complete search has been made; (ii) additional information material to the examination of this application does not exist; (iii) the information, protocols, results and the like reported by third parties are accurate or enabling; or (iv) the above information constitutes prior art to the subject invention.
sf-2185936
Application No. 1 1/231,353 Filed: 09/19/2005 Information Disclosure Statement
Docket No. 577832000200
In the unlikely event that the transmittal form is separated from this document and the Patent and Trademark Office determines that an extension and/or other relief (such as payment of a fee under 37 C.F.R. 1.17 (p)) is required, Applicants petition for any required relief including extensions of time and authorize the Commissioner to charge the cost of such petition and/or other fees due in connection with the filing of this document to Deposit Account No. 03-1952 referencing 577832000200.
Dated: O C
?c I I,
dob
Step en C. urant Registration No.: 31,506 MORRISON & FOERSTER LLP 425 Market Street San Francisco, California 94105-2482 (415) 268-6982
sf-2185936
ALTERNATIVE TO PTO/SBlO8alb (07-05)

0
te for form 1449/PTO Application Number
Complete if Known
11/231,353 September 19, 2005 Ken P. CHENG
ORMATION DISCLOSURE STATEMENT BY APPLICANT

(Use as many sheets as necessary)
Filing Date First Named Inventor Art Unit Examiner Name
2621 Not Yet Assigned 577832000200

I
Sheet
of

Examiner Initials' Cite No.' Document Number Number-Kind Code' (ifknown) Publication Date Publication Date MM-DD-YYYY Name of Patentee or Name of Patentee or Applicant of Cited Document Pages. Columns. Lines, Where Relevant Passages or Relevant Figures Appear
1. 2. 3. 4.5. 6. 7. 8. 9.
Examiner Cile Initals* 1 No.'
US-6,591,224 US-6,609,198 US-6,697,103-B1 US-6,970,582-A1 US-20020138768-A1 US-20020190119-A1 US-20040081338-A1 US-20050265607-A1 US-20060112039-A1

Foreign Patent Document
07-08-2003 08-19-2003 02-24-2004 11-29-2005 09-26-2002 12-19-2002 04-29-2004 12-01-2005 05-25-2006

Publication Date MM-DD-YYYY
Sullivan et al. Wood et al. Fernandez et al. Langley Murakami et al. Huffman Takenaka Chang Wang et al.
Pages. Columns. Lines. Where Relevant Passages orRelevant Figures Appear T
FOREIGN PATENT DOCUMENTS

Country Code'-Numbe'-Kind Code (ifknown) 1I
s
FName of Patentee or Applicant of Cited Document
'EXAMINER: Initial if information considered, whether or not citation is in conformance with MPEP 609. Draw line through citation if not in conformance and not considered. Include copy of this form with next communication to applicant. ' Applicants unique citation designation number (optional). ' See Kinds Codes of USPTO Patent Documents at www.uspto.gov or MPEP 901.04. ' Enter Office that issued the document, by the two-letter code (WIPO Standard ST.3). ' For Japanese patent documents, the indication of the year of the reign of the Emperor must precede the serial number of the patent document. 'Kind of document by the appropriate symbots as indicated on the document under WIPO Standard ST. 16 if possible. Applicant is to place a check mark here if English language Translation is attached.
NON PATENT LITERATURE DOCUMENTS

Examiner Include name of the author (in CAPITAL LETTERS), title of the article (when appropriate), title of the item (book, Cite magazine, journal, serial, symposium, catalog, etc.), date, page(s), volume-issue number(s), publisher, city Initials No.and/or country where published. T
10. 11.
Goh et al., "Robust Perceptual Color Identification" US-11/229,091,

Filed: 9/16/2005
12.
13.
Belhumeur, A. et al. (1997). "Eigenfaces vs. Fisherfaces: recognition using class specific linear projection", IEEE Transactions on Pattern Analysis and Machine Intelligence 19(7): 711-720. Brunelli, R. and D. Falavigna. (1995). "Person identification using multiple cues," IEEE Transactions on Pattern Analysis and Machine Intelligence 17(10): 955-966. Brunelli, R. et al. (1995). "Automatic Person Recognition by Using Acoustic and Geometric Features", Machine Vision and Applications
8: 317-325.
14.
HONG, Lin and Anil K. Jain. (1998). "Integrating faces and fingerprints for personal identification," IEEE Transactions on Pattern Analysis
IConsidered
Examiner Signature I
Date
sf- 2185930
ALTERNATIVE TO PTO/SB/08ab (07-05)
Complete if Known Substitute for form 1449/PTO Application Number
11/231,353
INFORMATION DISCLOSURE
Filing Date
STATEMENT BY APPLICANT
FirstNamed Inventor
September 19, 2005 CHENG Ken P. 2621 Not Yet Assigned 577832000200
sheet
of
Attorney Docket Number
15.
16.
17. 18.
and Machine Intelligence 20(12): 1295 - 1307. International Search Report mailed on April 2006 for PCT Patent Application Number PCT/US05/33378 filed on September 19, 2005, one page. JAIN, A. K. et al. (1997). "On-Line Fingerprint Verification," IEEE Transactions on Pattern Analysis and Machine Intelligence archive 19(4): 302- 314. Kittler, J. et al. (1998). "On combining classifiers", IEEE Transactions on Pattern Analysis and Machine Intelligence 20(3): 226-239. Lu X et al. (2003). "Combing classifiers for face recognition", IEEE International Conference on Multimedia Systems and Expo, Baltimore,
MD, July.
19.
Maio, D. et al. (2002). "FVC2000: fingerprint verification competition", IEEE Transactions on Pattern Analysis and Machine Intelligence 24(3):
402 -412.
20.
21.
Phillips, P.J. et al. (2000). "The FERET evaluation methodology for face-recognition algorithms", IEEE Transactions on Pattern Analysis and Machine Intelligence 22(10): 1090- 1104. Senior, A. (2001). "A combination fingerprint classifier", IEEE " Transactions on Pattern Analysis and Machine Intelligence 23(10):
1165 1174.
22.
TURK, A. and A. Pentland. (1991). "Eigenfaces for Recognition". Journal of Cognitive Neuroscience 3 (1): 71-86.
'EXAMINER: Initial if reference considered, whether ornot citation is in conformance with MPEP 609. Draw line through citation if not in conformance and not considered. Include copy of this form with next communication to applicant. designation number (optional). 'Applicant Is to place a check mark here if English language Translation is attached. 'Applicanrts unique citation
ExaminerDate
Signature sf- 2185930
Considered
Other Prior Art

According to the information contained in form PTO-1449 or PTO-892, there are one or more other prior art/non-patent literature documents missing from the original file history record obtained from the United States Patent and Trademark Office. Upon your request we will attempt to obtain these documents from alternative resources. Please note that additional charges will apply for this service.
This page is not part of the official USPTO record. It has been determined that content identified on this document is missing from the original file history record.
., IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 7, JULY 1997
Application No. 1 1/231,35371

Application No. 11231,353 Docket No. 577832000200
711
SEigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection

Peter N. Belhumeur, Joio P. Hespanha, and David J. Krie ;man
Abstract-We develop a face recognition algorithm which is insensitive to large variation in lighting direction and lacial expression. Taking a pattern classilication approach, we consider each pixel inan image as a coordinate in a high-dimensional space. We lake advantage of the observation that the images of a particular face, under varying illumination but fixed pose, lie in a 3D linear subspace of the highdimensional image space-if the face is a Lambertian surface without shadowing.. Iowever, since faces are not truly Lambertian surfaces and do indeed produce self-shadowing, images will deviate from this linear subspace. Rather than explicitly modelingj this deviation, we linearly project the image into a subspace in a manner which discounts those regions of the lace with large deviation. Our projection method is based on Fisher's Linear Discriminant and produces vell separated classes ina low-dimensional subspace, even under severe variation In lighting and facial expressions. The Elgenface technique, another method based on linearly projecting the image space to a low dimensional subspace, has similar computational requirements. Yet, extensive experimental results demonstrate that the proposed "Fisherface" method has error rates that are lower than those of the Eigentace technique for tests on the Harvard and Yale Face Databases. Index Terms-Appearance-based vision, face recognition, illuminatiorn invariance, Fisher's linear discriminant.
+,
1 INTRODUCTION Within the last several years, numerous algorithms have been proposed for face recognition; for detailed surveys see [1), [2]. While much progress has been made toward recognizing faces under small variations in lighting, facial expression and pose, reliable techniques for recognition under more extreme variations have proven elusive. In this paper, we outline a new approach for face recognition--one that is insensitive to large variations in lighting and facial expressions. Note that lighting variability includes not only intensity, but also direction and number of light sources. As is evident from Fig. 1, the same person, with the same facial expression, and seen from the same viewpoint; can appear dramatically different when light sources illuminate the face from different directions. See also Fig. 4. Our approach to face recognition exploits two observations: 1) All of the images of a Lambertian surface, taken from a fixed viewpoint, but under varying illumination, lie in a 3D linear subspace of the high-dimensional image space [3). 2) Because of regions of shadowing, specularities, and facial expressions, the above observation does not exactly hold, In practice, certain regions of the face may have variability from image to image that often deviates significantly from the linear subspace, and, consequently, are less reliable for recognition. We make use of these observations by finding a linear projection of the faces from the high-dimensional image
* The authors are with the Center for Computational Vision and Control, Dept. of Electrical Engineering, Yale University, NewtHaven, CT 06520-8267. E-mail: lbethumneur, kriegngmanl@yale.edu, hespanha@yale.edu. Manuscript received 15 Feb. 1996 revised 20 Mar. 1997. Recommended for acceptance by I. Daugman.
For informationon obtaining reprints of this article, please send e-mail to: transpami@computer.org, and reference IEEECS Log Number 104797.
I
space to a significantly lolve: dimensional feature space which is insensitive both to variation in lighting direction and facial expression: We choose projection directions that are nearly orthogonal to the within-class scatter, projecting away variations in lighting mnd facial expression while maintaining discriminability. Our method Fisherfaces, a derivative of Fisher's Linear Discriminant (FLD) [4], [5), maximizes the ratio of between-class scatter to that of within-class scatter. The Eligenface method is also based on linearly projecting the image space to a low c.imensional feature space [6), 17), 181. However, the Eigenface method, which uses principal components analysis (PCA) for dimensionality reduction, yields projection directions that maximize the total scatter across all classes, i.e., a ross all images of all faces. In choosing the projection which maximizes total scatter, PCA retains unwanted variations due to lighting and facial expression. As illustrated in Figs. 1 and 4 and stated by Moses et al., "the variations between the images of the same face due to illumination and viewing direction are almost always larger than image var ations due to change in face identity" (9]. Thus, while the PCA projections are optimal
Fig. 1. The same person seen uncer different lighting conditions can appear dramatically different: In the left image, the dominant light source is nearly head-on; in the right image, the dominant light source is from above and to the right.
0162-828/97/1S10.00 0 1997 IEEE
712
IEEE TRANSACTIONS ON PATT ERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 7, JULY 1997
for reconstruction from a low, dimensional basis, they may not be optimal from a discrimination standpoint. . We should point out that Fisher's Linear Discriminant is a "classical" technique in pattern recognition [4], first developed by Robert Fisher in 1936 for taxonomic classification [5]. Depending upon the features being used, it has been applied in different ways in computer vision and even in face recognition. Cheng et al. presented a method that used Fisher's discriminator for face recognition, where features were obtained by a polar quantization of the shape [10]. Baker and Nayar have developed a theory of pattern rejection which is based on a two class linear discriminant [11]. Contemporaneous with our work [12], Cui et al. applied Fisher's discriminator (using different terminology, they call it the Most Discriminating Feature-MDF) in a method for recognizing hand gestures [13]. Though no implementation is reported, they also suggest that the method can be applied to face recognition under variable illumination. the ,In sections to follow, we compare four methods for face recognition under variation in lighting and facial expression; correlation, a variant of the linear subspace method suggested by [3], the Eigenface method [6], [71, [8], and the Fisherface method developed here. The comparisons are done using both a subset of the Harvard Database (330 images) [14], [15] and a database created at Yale (160 images). In tests on both databases, the Fisherface method had lower error rates than any of the other three methods. Yet, no claim is made about the relative performance of these algorithms on much larger databases. We should also point out that we have made no attempt to deal with variation in pose. An appearance-based method such as ours can be extended to handle limited pose variation using either a multiple-view representation, such as Pentland et al's. view-based Eigenspace [16] or Murase and Nayar's appearance manifolds [171. Other approaches to face recognition that accommodate pose variation include [18], [19], [20]. Furthermore, we assume that the face has been located and aligned within the image, as there are numerous methods for finding faces in scenes
scheme, an image in the test set.is recognized (classifiecV by assigning to it the label of the closest point in the learning set, where distances are measured in the image space. If all of the images are normalized to have zero mean and unit variance, .then this procedure is equivalent to choosing the image in the learning set that best correlates with the test image. Because of the normalization process, the result is independent of light source intensity and-the effects of a video camera's automatic gain control. This procedure, which subsequently is referred to as correlation, has several well-known disadvantages. First, if the images in the learning. set and test set are gathered under varying lighting conditions, then the corresponding points in the image space may not be tightly clustered. So, in order for this method to work reliably under variations in lighting; we would need a learning set which densely sampled the continuum of possible lighting conditions. Second, correlation is computationally expensive. For.recognition, we must correlate the image of the test face with each image in the learning set; in an effort to reduce the computation time, implementors [27] of the algorithm described in [26] developed special purpose VLSI hardware. Third, it requires large amounts of storage-the learning set must , contain numerous images of each person.
2.2 Eigenfaces
As correlation methods are computationally expensive and . require great amounts of storage, it is natural to pursue dimensionality reduction schemes. A technique now commonly used for dimensionality reduction in computer vision-particularly in face recognition-is principal components analysis (PCA) [14), [17], [61, [7], [8]. PCA techniques, also known as Karhunen-Loeve methods, choose a dimensionality reducing linear projection that maximizes the scatter of all projected samples. More formally, let us consider a set of N sample images {x,x 2 , ... , xN) taking values in an n-dimensional image space, and assume that each image belongs to one of c classes {X1,X 2 ...X,..Let us also consider a linear trans121], [221, [20], [231, [241, 125], [7]. formation mapping the original n-dimensional image space into an m-dimensional feature space, whereim < n. The new 2 IMETHODS feature vectors Yk e IR"' are defined by the following linear The problem can be simply stated: Given a set of face im- transformation: ages labeled with the person's identity (the learning set) and k =, 2,..., N Yk = WTk an unlabeled set of face images from the same group of people (the test set), identify each person in the test images. whete W E R""' is a matrix with orthonormal columns. In this section, we examine four pattern classification If the total scatter matrix S. is defined as N techniques for solving the face recognition problem, comparing methods that have become quite popular in the face (Xk - 4Xk ST = recognition literature, namely correlation [26] and Eigenk=1 face methods 16], [7], [8], with alternative methods develwhere n is the number of sample images, and 4e R " is the oped by the authors. We approach this problem within the pattern classification paradigm, considering each of the mean image of all samples, then after applying the linear pixel values in a sample image as a coordinate in a high- transformation WT, the scatter of the transformed feature dimensional space (the image space). vectors {YtYY2 -...-YN}is WTSTW. In PCA, the projection WeP is chosen to maximize the determinant of the total 2.1 Correlation scatter matrix of the projected samples, i.e., Perhaps, the. simplest classification scheme is a nearest neighbor classifier in the image space [26]. Under this
BELHUMEUR ET AL.: EIGENFACES VS. FISHERFACES: RECOGNITION USING CLASS SPECIFIC LINEAR PROJECTION
age of the surface under an arbitrary lighting direction by a linear combination of the three original images, see 13]. (2) For classification, this fact h.as great importance: It shows =[ W2 ... wm] that, for a fixed viewpoint, the images of a Lambertian surset of n-dimensional eig en- face lie in a 3D linear subspace of the high-dimensional im-. where [w; i = 1, 2,..., m) is the age space. This observation suggests a simple classification vectors of Sr corresponding to the m largest eigenvalu es. algorithm to recognize Lambe:tian surfaces-insensitive to Since these eigenvectors have the same dimension as the a wide range of lighting conditions. . For each face, use three or r.iore images taken under diforiginal images, they are referred to as Eigenpictures in [61 and Eigenfaces in (7], [8]. If classification is performed us- ferent lighting directions to construct a 3D basis for the lining a nearest neighbor classifier in the reduced feat ure ear subspace. Note that the three basis vectors have the space and m is chosen to be the number of images N in the- same dimensionality as the training images and can be training set, then the Eigenface method is equivalent to the thought of as basis images. 'o perform recognition, we correlation method in the previous section. simply compute the distance cf a new image to each linear A drawback of this approach is that the scatter be "ing subspace and choose the face corresponding to the shortest is distance. We call this recogniion scheme the Linear Submaximized is due not only to the between-class scatter th at classification, but also to the within-class sca tter space method. We should point out thai this method is a useful for that, for classification purposes, is unwanted informat ion. variant of the photometric alignment method proposed in Recall the comment by Moses et al. [9]: Much of the varia tion [3), and is a special case of th. more elaborate recognition from one image to the next is due to illumination chan ges. method described in [15]. Subsequently, Nayar and Murase earity of lighting to augment have exploited the apparent lir Thus if PCA is presented with images of faces under vary projection matrix W, will contain pri nci- their appearance manifold (31] illumination, the If there is no noise or shacowing, the Linear Subspace propal components (i.e., Eigenfaces) which retain, in the ! nrise- algorithm would achieve error free classification under any jected feature space, the variation due lighting. Coi quently, the points in the projected space will not be vell lighting conditions, provided the surfaces obey the Lam. bertian reflectance model. Nevertheless, there are several clustered, and worse, the classes may be smeared together nost compelling reasons to look elsewhere. First, due to selfIt has been suggested that by discarding the three n significant principal components, the variation due Sto shadowing, specularities, and facial expressions, some relighting is reduced. The hope is that if the first princ ipal gions in images of the face have variability that does not components capture the variation due to lighting, hen agree with the linear subspace. model. Given enough imbetter clustering of projected samples is achieved by ; ig- ages of faces, we should be able to learn which regions are noring them. Yet, it is unlikely that the first several princi- good for recognition and-which regions are not. Second, to pal components correspond solely to variation in light ing; recognize a test image; we mu:;t measure thedistance to the as a consequence, information that is useful for discrim ina- linear subspace for each perscn. While this is an improvement over a correlation scherre that needs a large number tion may be lost. of images to represent the variability of each class, it is 2.3 Linear Subspaces computationally expensive. Finally, from a storage standBoth correlation and the Eigenface method are expecte d to point, the Linear Subspace alg;orithm must keep three imsuffer under variation in lighting direction. Neither met hod ages in memory for every person. ,ithexploits the observation that for a Lambertian surface A out shadowing, the images of a particular face lie in a 3D 2.4 Fisherfaces The previous algorithm takes advantage of the fact that, linear subspace. on a Lambertian surface illumin ated under admittedly idealized conditions, the variation within Consider a point p 3 by a point light source at infinity. Let s e R be a colhumn class lies in a linear subspace of the image space. Hence, the signifying the product of the light source inter Isity classes are convex, and, therefore, linearly separable. One vector with the unit vector for the light source direction. Wher the can perform dimensionality reduction using linear projecsurface is viewed by a camera, the resulting image interIsity tion and still preserve linear :eparability. This is a strong argument in favor of using linear methods for dimensionof the point p is given by ality reduction in the face recognition problem, at least T (3) when one seeks insensitivity tc lighting conditions. E(p) = a(p)n(p) s the learning set is labeled, it makes sense to use where n(p) is the unit inward normal vector to the sui aceface Since this information to build a more reliable method for reat the point p, and a(p) is the albedo of the surface at p [281. ducing the dimensionality of the feature space. Here we This shows that the image intensity of the point p is li near argue that using class specific: linear methods for dimenon s E R3 . Therefore, in the absence of shadowing, g iven sionality reduction and simple classifiers in the reduced three images of a Lambertian surface from the same v iew- feature space, one may get better recognition rates than point taken under three.known, linearly, independent light with either the Linear Subspace.method or the Eigenface source directions, the albedo and surface normal cai n be method. Fisher's Linear Discri:ninant (FLD) [5) is an examrecovered, this is the well known method of photom etric ple of a class specific method, in the sense that it tries to stereo [29], [30]. Alternatively, one can reconstruct the Sim- "shape" the scatter in order :omake it more reliable for W 0r,= arg maxIWTSTWI
'ing
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 7, JULY 1997
classification. This method selects W in [1] in sucha v that the ratio of the between-class scatter and the wit class scatter is maximized. Let the between-class scatter matrix be defined as
SA
C
X( Ni
i - " )(P i -
)T.
* 1~1
i=1
and the within-class scatter matrix be defined as

+
/
.. .
.. . . .. . .
. .
Sw =
(x i=1 xEX,
i)(xk -
)T
i
..'
where p is the mean image of class Xi, and N i is the n ber of samples in class Xi . If Sw is nonsingular, the optil projection Wop is chosen as the matrix with orthonor: columns which maximizes the ratio of the determinan the between-class scatter matrix of the projected sample the determinant of the within-class scatter matrix of the projected samples, i.e., Wopt = arg maxW
= [w
1
,o
4 : 4
b \4
classi1
+ :class 2
feature 1
1WSBW
W
W ]
Fig. 2. A comparison of principal component analysis (PCA) and Fisher's linear discriminant (FLD) for a two class problem where data for each class lies near a linear subspace.
w2
...
(4)
where {w i i = 1, 2,..., m} is the set of generalized ei genvectors of SA and SW corresponding to the m largest ge neralized eigenvalues {; I i = 1,2,..., m}, i.e.,
SBw; =,A;Sww,
method, which we call Fisherfaces, avoids this problem by projecting the image set to a lower dimensional space so that the resulting within-class scatter matrix Sw is nonsingular. This is achieved by using PCA to reduce the dimension of the feature space to N - c, and then applying the standard FLD defined by (4) to reduce the dimension to c - 1. More formally, Wop t is given by
WT t WT Wfld T pea
i = 1,2,...,m
-n -
Note that there are at most c - 1 nonzero generalized eil values, and so an upper bound on m is c - 1, where c is number of classes. See [4]. To illustrate the benefits of class specific linear pr< tion, we constructed a low dimensional analogue to classification problem in which the samples from.each < lie near a linear subspace. Fig. 2 is a comparison of I and FLD for a two-class problem in which the samples f each class are randomly perturbed in a direction per] dicular to a linear subspace. For this example, N = 20, n and m = 1. So, the samples from each class lie near a passing through. the origin in the 2D feature space. 1 PCA and FLD have been used to project the points fron down to ID. Comparing the two projections in the fig PCA actually smears the classes together so that they are longer linearly separable in the projected space. It is that, although PCA achieves larger total scatter, achieves greater between-class scatter, and, conseque classification is simplified. In the face recognition problem, one is confronted the difficulty that the within-class scatter matrix Sw e is always singular. This stems from the fact that the rar Sw is at most N - c, and, in general, the number of im in the learning set N is much smaller than the numbs pixels in each image n. This means that it is possib choose the matrix W such that the within-class scatter a projected samples can be made exactly zero. In order to overcome the complication of a singular we propose an alternative to the criterion in (4).
(5) .
where W,= arg mrnaxiWTSrW WTWTaSWcap Wfld = arg max
TT
is performed over
Note that the optimization for Wp
n x (N - c) matrices with orthonormal columns, while the optimization for Wfld is performed over (N - c) x m matrices with orthonormal columns. In computing Wp, we have thrown away only thesmallest c - 1 principal components. There are certainly other ways of reducing the withinclass scatter while preserving between-class scatter. For example, a second method which we are currently investigating chooses W to maximize the between-class scatter of the projected samples after having first reduced the withinclass scatter. Taken to an extreme, we can maximize the between-class scatter of the projected samples subject to the constraint that the within-class scatter is zero, i.e., W0pt = arg
maxWTSW
where "W is the set of n x m matrices with orthonormal columns contained in the kernel of S,.
/
EIM
BEST AVAILABLE COPY
3 EXPERIMENTAL RESULTS
in this section, we present and discuss each of the aforementioned face recognition techniques using two different databases. Because of the specific hypotheses that .we wanted to test about the relative performance of the considered algorithms, many of 'the standard databases were inappropriate. So, we have used a database from the Harvard Robotics Laboratory in which lighting has been systematically varied. Secondly, we have constructed a database at Yale that includes variation in both facial expression and lighting. 1
Subset 1 Subset 2 Subset 3 Subset 4 Subset 5
3.1 Variation in Lighting Fig. 3. The highlighted lines oflongitude and latitude indicate the light The first experiment was designed to test the hypothesis source directions for Subsets 1 through 5. Each intersection ofa lonthat under variable illumination, face recognition algo- gitudinal and latitudinal line on the 'ight side of the illustration has a rithmins will perform better if they exploit the fact that im- corresponding image inthe database. ages of a Lambertian surface lie in a linear subspace. More specifically, the recognition error rates for all four algodirection coincident with the camera's optical axis. rithms described in. Section 2 are compared using an im- Subset 2 contains 45 images for which the greater of the age database constructed by Hallinan at the Harvard Rolongitudinal and latitudinal 'angles of light source dibotics Laboratory 114), [15). In each image in this datarection are 30 from the camera axis. base, a subject held his/her head steady while being illu- Subset 3 contains 65 images for which the greater of the minated by a dominant light source. The space of light longitudinal and latitudinal angles of light source disource directions, which can be parameterized by spheri-, rection are 45 from the camera axis. cal angles, was then sampled in 15 increments. See Fig. 3. Subset 4 contains 85 images :!or which the greater of the From this database, we used 330 images of five people (66 longitudinal and latitudi:nal angles of light source diof each). We extracted five subsets to quantify the effects rection are 60 from the camera axis. of varying lighting. Sample images from each subset are Subset 5 contains 105 images for which the greater of the shown in Fig. 4. longitudinal and latitudihal angles of light source diSubset 1 contains 30 images for which both the longitudirection are 75 from the camera axis. nal and latitudinal angles of light source direction are, For all experiments, classification was performed using a within 15 of the camera axis, including the lighting nearest neighbor classifier. Al training images of an indi1. The Yale database is available for download from http://cvc.yaleedu.
Subset 1
Subset 2
Subset 3
Subset 4
Subset 5
r._,
Fig. 4. Example images from each subset of the Harvard Database used to test the four algorithms.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 7, JULY 1997
45
Eigenlace (10) 40. -S
Eigenlace (10) Egentace (10) w/o first 3

'-
E~s r
30 S25 20 15
..
S/
..
Correlation
(10) 3 w/ofirst
- -Corretion
-'. _ ...
.
-Elgenfece
...
--- Llnear Subspace

dace --e- Fishe
. -....
_....
'a
... . / . , -. ._ Linear' ,
e 10
.
,-. .
Subspace
--Fisherface
Sub! 1 set
Subset 2
S9bset 3
Lighting Direction Subset Extrapolating from Subset 1 Error Rate (%) Reduced Subset 2 Subset 1 Space 31.1 0.0 4
10 0.0 4.4
Method Eigenface Eigenface

w/o ist,3
.. Subset 3 47.7
41.5
4
10
0.0
0.0
13.3
.4.4
41.5
27.7
Correlation Linear Subspace Fisherface
29 15 4
0.0 . 0.0 0.0
0.0 4.4 0.0
33.9 9.2 4.6
graph and corresponding table show Fig. 5. Extrapolation: When each of the methods is trained on images with near frontal illumination (Subset 1),.the
the relative performance under extreme light source conditions. vidual were projected into, the feature space. The images were cropped within the face so that the contour of the head was excluded. For the Eigenface and correlation tests, the images were normalized to have zero mean and unit variance, as this ,improved the performance of these methods. For the Eigenface method,.results are shown when ten principal components were used. Since it has been suggested that the first three principal components are primarily due to lighting variation and that recognition rates can be improved by eliminating them, crror rates are also presented using principal components four through thirteen. We performed two experiments on the Harvard Database: extrapolation and interpolation. In the extrapolation experiment, each method was trained on samples from Subset 1 and then tested using samples from Subsets 1, 2, and 3. Since there are 30 images in the training set, correlation is equivalent to the Eigenface method using 29 principal components. Fig. 5 shows the result from this experiment. In the interpolation experiment, each method was trained on Subsets 1 and 5 and then tested the methods on Subsets 2, 3, and 4. Fig. 6 shows the result from this experiment. These two experiments reveal a number of interesting points: 1) All of the algorithms perform perfectly when lighting is nearly frontal. However, as lighting is moved off
2. We have observed that the error rates are reduced for all methods when the contour is included and the subject is in front of a uniform background. However, all methods performed worse when the background varies. 3. To test the methods with an image from Subset 1, that image was removed from the training set, i.e., we employed the "leaving-one-out" strategy 141..
axis, there is a significant performance difference between the two class-specific methods and the Eigenface method. 2) It has also been noted that the Eigenface method is equivalent to correlation when the number of Eigenfaces equals the size of the training.set [17], and since performance increases with the dimension 'of the eigenspace, the Eigenface method should do no better than correlation [26]. This is empirically demonstrated as well. 3) In the Eigenface method, removing the first three principal components results in better performance under variable lighting conditions. 4) While the Linear Subspace method has error rates that are competitive with the Fisherface method, it requires storing more than three times as much information and takes three times as long. 5) The Fisherface method had error rates lower than the Eigenface method and required less computation time. 3.2 Variation in Facial Expression, Eye Wear, and Lighting Using 'a second database constructed at the Yale Center for Computational Vision and Control, we designed tests to determine how the methods compared under a different range of conditions. For sixteen subjects, ten images were acquired during one session in front of a simple background. Subjects included females and males (some with facial hair), and some wore glasses. Fig. 7 shows ten images of one subject. The first image was taken under ambient lighting in a neutral facial expression and the person wore glasses. In the second
,GAV L.R
BELHUMEUR ET AL.: EIGENFACES VS. FISHERFACES: RECOGNITION USING CLA SSP
AILABLE COPY
IFIC LINEAR PROJECTIOrI
E
r
r
0
r
Eigms (10) Wae
a t
e
Eige iace (10)
w/o rsl I 3
Corntaion
FiSNhe C
(9/
Subsel 2
subs 3ace
,jbsel 3
4. Subset
Lighting Direction Subset Interpolatinq between Subsets 1 and 5 Error Rate (% Reduced Subset 3 Subset 2. Space
4 53.3 75.4
Method
Eigentace
Subs(et 4
52.
10
11.11
33.9
20.0
Eigenface w/o 1st 3 Correlation Linear Subspace Fisherface
4 10 129 15 4
31.11 6.7 0.0 0.0 0.0
60.0 20.0 21.54 1.5 0.0
29. 12.9 7.1 0.0 1.2
Fig. 6. Interpolation: When each of the methods is trained on images from both near frontal and extreme lighting (Subsets 1and 5), the graph and corresponding table show the relative performance under intermediate lighting conditions. image, the glasses were removed. If the person normally wore glasses, those were used; if not, a random pair was borrowed. Images 3-5 were acquired by illuminating the face in' a neutral expression with a Luxolamp in three positions. The last five images were acquired under ambient lighting with different expressions (happy, sad, winking, sleepy, and surprised). For the Eigenface and correlation tests, the images were normalized to have zero mean and unit variance, as this improved the performance of' these methods. The images were manually centered and cropped to two different scales: The larger images included the full face and part of the background while the closely cropped ones included internal structures such as the brow, eyes, nose, mouth,' and chin, but did not extend to the occluding contour. In this test, error rates were determined by the "leavingone-out" strategy [4]: To clas:;ify an image of a person, that image was removed from the: data set and the dimensionality reduction matrix W wa, computed. All images in the database, excluding the test image, were then projected down into the reduced spact: to be used for classification. Recognition was performed using a nearest neighbor classifier. Note that for this test, eac:h person in the learning set is represented by the projectiorn. of ten images, except for the test person who is represented by only nine. In general, the performance of the Eigenface method varies with the number of piincipal components. Thus, before comparing the Linear Subspace and Fisherface methods with the Eigenface method, we first performed an experi-
Fig. 7. The Yale database contains 160 frontal face images .covering 16 individuals taken under 10 different conditions: A normal image under ambient lighting, one with or without glasses, three images taken with different point light sources, and five dif;erent facial expressions.
'
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 7, JULY 1997 '
r 25
r
02 r
a
15
Eigenface
a
t t0o
Eigenface w/o first three components

Fisherface(7.3%)
e
5
----
-------------
------
(%)
0
*
0
1
50
1
100
Number of Principal Components

Fig. 8. As demonstrated on the Yale Database, the variation in performance of the Eigenface method depends on the number of principal compononts retained. Dropping the first three appears to improve performance.
ea CloseCrop
uFaceJ
Eigentace(30)
Correlation
Subspace
Unear
-igenacet.u,
w/o first 3
Fisherface
Recognition Algorithm "Leaving-One-Out" of Yale Database Error Rate (%) Reduced Method Full Face lose Crop Space 19.4 24.4 30 Eigenlace Eigenface
w/o 1st 3 30 15.3 10.8
Correlation
Linear Subspace
160
48
23.9
21.6
.
20.0
15.6
Fisherface
variation in facial expression and lIghting.
15
7.3
0.6
Fig. 9. The graph and corresponding table show the relative performance of the algorithms when applied to the Yale Database which contains Note that the Linear Subspace method faired comparatively worse in this experiment than in the lighting experiments in the previous section. Because of variation in facial expression, the images no longer lie in a linear subspace. Since the Fisherface method. tends to discount those portions of the image that are not significant for recognizing an individual, the resulting projections W tend to mask the regions of the face that are highly variable. For example, the area around the mouth is discounted, since it varies quite a bit for different facial expressions. On the other hand; the nose, cheeks, and brow are stable over the within-class
ment to determine the number of principal components yielding the lowest error rate. Fig. 8 shows a plot of error rate vs. the number of principal components, for the closely cropped set, when the initial three principal components were retained and when they were dropped. The relative performance of the algorithms is self evident in Fig. 9. The Fisherface method had error rates that were better than half that of any other method. It seems that the Fisherface method chooses the set of projections which performs well over a range of lighting variation, facial expression variation, and presence of glasses.
p.
*~
,'
BEST AVAILABLE COPY
variation and are more significant for recognition. Thus, we conjecture that Fisherface methods, which tend to reduce within-class scatter for all classes, should produce projection directions that are also good for recognizing other faces besides the ones in the training set. All of the algorithms performed better on the images of the full face. Note that there is a dramatic improvement in the Fisherface method where the error rate was reduced from 7.3 percent to 0.6 percent. When the method is trained on the entire face, the pixels corresponding to the occluding contour of the face are chosen as good features for discriminating between individuals, i.e., the overall shape of the face is a powerful feature in face identification. As a practical note, however, it is expected that recognition rates would have been much lower for the full face images if the background or hair styles had varied and may even have been worse than the closely cropped images. 3.3 Glasses Recognition When using class specific projection methods, the learning set can be divided into classes in different manners. For example, rather than selecting the classes to be individual people, the set of images can be divided into two classes: "wearing glasses" and "not wearing glasses." With only two classes, the images can be projected to a line using the Fisherface methods. Using PCA, the choice of the Eigenfaces is independent of the class definition. In this experiment, the data set contained 36 images from a superset of the Yale Database, half with glasses. The recognition rates were obtained.by cross validation, i.e., to classify the images of each person, all images of that person were removed from the database before the projection matrix W was computed. Table 1 presents the error rates for two different methods. TABLE 1
COMPARATIVE RECOGNITION ERROR RATES FOR GLASSES/
NO GLASSES RECOGNITION USING THE YALE DATABASE
Fig. 10. The left image is an image from the Yale Database of a person wearing glasses. The right image s the Fisherface used for determining if a person is wearing glasses. 2) The Fisherface method appears to be the best at extrapolating and interpolating over variation in lighting, although the Linear Sub ;pace method is a close second. 3) Removing the largest three principal components does improve the performan:e of the Eigenface method in the presence of lighting variation, but does not achieve error rates as low as some of the other methods described here. 4) In the limit, as more principal components are used in the Eigenface method, performance approaches that of correlation. Similarly, when the first three principal components have been removed, performance improves as the dimension ality of the feature space is increased. Note, however, that performance seems to level off at about 45 principal components. Sirovitch and Kirby found a simi ar point of diminishing returns when using Eigenfaces.to represent faceimages [6]. 5) The Fisherface method appears to be the best at simultaneously handling variation in lighting and expression. As expected, the Linear Subspace method suffers when confronted with variation in facial expression. Even with this extensive experimentation, interesting questions remain: H-ow well does the Fisherface method extend to large databases. Can variation in lighting conditions be accommodated if some of the individuals are only observed under one lighting -ondition? Additionally, current face detection methods are likely to ghting conditions such as Subbreak down under extreme 1: sets 4 and 5 in Fig. 4, and io new detection methods are needed to support the algorithms presented in this paper. Finally, when shadowing dominates, performance degrades for all of the presented recog:aition methods, and techniques that either model or. mask he shadowed regions may be needed. We are currently invwestigating models for reprepossible illumisenting the set of images of a:, object under all nation conditions, and have shown that the set of n-pixel images of an. object of any shape and with an arbitrary reflectance function, seen under ;ill possible illumination conditions, forms a convex cone in R" [32]. Furthermore, and most relevant to this paper, it appears that this convex illumination cone lies close to a low-dimer sional linear subspace [14]. ACKNOWLEDGMENTS P.N. Belhumeur was suppor:ed by ARO grant DAAHO4-951-0494. J.P. Hespanha was supported by the U.S. National
Method PCA Fisheriace
Glasses Reco nition Reduced Space 10. 1
Error Rate (%Z) ., 52.6
5.3
PCA had recognition rates near chance, since, in most cases, it classified both images with and without.glasses to the same class. On the other hand, the Fisherface methods can be viewed as deriving a template which is suited for finding glasses and ignoring other characteristics of the face. This conjecture is supported by observing the Fisherface in Fig. 10 corresponding to the projection matrix W. Naturally, it is expected that the same techniques could be applied to identifying facial expressions where the set of training images is divided into classes based on the facial expression.
4
CONCLUSION 1) All methods perform well if presented with an image in the test set which is similar to an image in the training set.
The experiments suggest a number of conclusions:
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO.-7. JULY 1997
Science Foundation Grant ECS-9206021, AFOSR Grant F49620-94-1-0181, and ARO Grant DAAHO4-95-1-0114. D.J. Kriegman was supported by the U.S. National Science Foundation under an NYI,.LRI-9257990 and by ONR N0001493-1-0305. The authors would like to thank Peter Hallinan for providing the Harvard Database, and Alan Yuille and David Mumford for many useful discussions.
REFERENCES
[1) R. Chellappa, C. Wilson, and S. Sirohey, "Human and Machine Recognition of Faces: A Survey," Proc. IEEE, vol. 83, no.5, pp. 705740, 1995. 12) A. Samal and P. lyengar, "Automatic Recognition and Analysis of Human Faces and Facial Expressions: A Survey," Pattern Recognition, vol. 25, pp. 65-77, 1992. [3) A. Shashua, "Geometry and Photometry in 3D Visual Recognition," PhD thesis, Massachusetts Institute of Technology, 1992. [4) R. Duda and P. Hart, Pattern Classification and Scene Analysis. New York: Wiley, 1973. 15] R.A. Fisher, "The Use of Multiple Measures in Taxonomic Problems," Ann. Eugenics,vol. 7, pp. 179-188,1936. (6) L. Sirovitch and M: Kirby, "Low-Dimensional Procedure for the Optical Soc. of Am. A, vol. 2, Characterization of Human Faces," J. pp. 519-524,1987. Cogni17]M. Turk and A. Pentland, "Eigenfaces for Recognition," J. trive Neuroscience, vol. 3, no. 1, 1991. (81 M. Turk and A. Pentland, "Face Recognition Using Eigenfaces," Proc. IEEE Conf on Computer Vision and Pattern Recognition, 1991, pp. 586-591. [91 Y. Moses, Y. Adini, and S. Ullman, "Face Recognition: The Problem of Compensating for Changes in Illumination Direction," European Conf Computer Vision, 1994, pp.. 286-296. [10] Y. Cheng, K. Liu, J. Yang, Y. Zhuang, and N. Gu, "Human Face Recognition Method Based on the Statistical Model of Small Sample Size," SPIE Proc. Intelligent Robots and Computer Vision X: Algorithms and Technology, 1991, pp. 85-95. [11] Baker and S.K. Nayar, "Pattern Rejection," Proc. IEEE Conf. S. Computer Vision and Pattern Recognition, 1996, pp. 544-549. (12] P.N. Belhumeur, J.P. Hespanha, and D.J. Kriegman, "Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection," EuropeanConf.Computer Vision, 1996,pp. 45-58. (13] Y. Cui, D. Swets, and J. Weng, "Learning-Based Hand Sign Recognition Using SHOSLIF-M," Int'l Conf. on Computer Vision, 1995, pp. 631-636: 1141 P. Hallinan, "A Low-Dimensional Representation of Human Faces for Arbitrary Lighting Conditions," Proc. IEEE Conf Computer Vision and Pattern Recognition, 1994, pp. 995-999. 115) P. Hallinan, "A Dcformable Model for Face Recognition Under Arbitrary Lighting Conditions," PhD thesis, Harvard Univ., 1995. [16) A. Pentland, B. Moghaddam, and Starner, "View-Based and Modular Eigenspaces for Face Recognition," Proc. IEEE ConJ. Computer Vision and PatternRecognition, 1994, pp. 84-91. (17] H. Murase and S.Nayar, "Visual Learning and Recognition of 3-D Objects from Appearance," Int'l J. Computer Vision, vol. 14, pp. 5-24,1995. 118] D. Beymer, "Face Recognition Under Varying Pose," Proc. IEEE Conf Computer Vision and Pattern Recognition, 1994, pp. 756-761. 119) A. Gee and R. Cipolla, "Determining the Gaze of Faces in Images," Image and Vision Computing, vol. 12, pp. 639-648, 1994. 120) A. Lanitis, C.J. Taylor, and T.F. Cootes, "A Unified Approach to Coding and Interpreting Face Images," Int'l Conf.Computer Vision, 1995, pp. 368-373. [21) Q. Chen, H. Wu, and M. Yachida, "Face Detection by Fuzzy Pattern Matching," Int'Il Conf. Computer Vision, 1995, pp. 591-596. (22) I. Craw, D. Tock, and A: Bennet, "Finding Face Features," Proc. European Conf Computer Vision, 1992, pp. 92-96. [231 T. Leung, M. Burl, and P. Perona, "Finding Faces in Cluttered Scenes Using Labeled Random Graph Matching," Int'l Conf. Conputer Vision, 1995, pp. 637-644. [24) K. Matsuno, C.W. Lee, S. Kimura, and S. Tsuji, "Automatic Recognition of Human Facial Expressions," Int'l Conf Computer Vision, 1995, pp. 352-359. (25) B. Moghaddam and A. Pentland, "Probabilistic Visual Learning for Object Detection," Int'l Conf. Computer Vision, 1995, pp. 786-793.
126] R..Brunelli and T. Poggio, "Face Recognition: Features vs. Ternmplates," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 10, pp. 1,042-1,053, Oct. 1993. [27] J.M. Gilbert and W. Yang, "A Real-Time Face Recognition System Using Custom VLSI Hardware," Proc. IEEE Workshop on Computer Architectures for Machine Perception, 1993, pp. 58-66. [28] B.K.P. Horn, Computer Vision. Cambridge, Mass.: MIT Press, 1986. [29) W.M. Silver, Determining Shape and Reflectance Using Multiple'Images, PhD thesis, Massachusetts Institute of Technology, 1980. [30] R.J. Woodham, "Analysing Images of Curved Surfaces," Artificial Intelligence, vol. 17, pp. 117-140,1981. [31) S.Nayar and H. Murase, "Dimensionality of Illumination in Appearance Matching," IEEE Conf Robotics and Automation, 1996. (32] P.N. Belhumeur and D.J. Kriegman, "What is the Set of Images of an Object under all Possible Lighting Conditions?," IEEE Proc. Conf Computer Vision and Pattern Recognition, 1996. Peter N. Belhiumeur receiveo nhis ScB degree (with highest honors) in computer and information engineering from Brown University in 1985. He received an SM and PhD from Harvard University in 1991 and 1993, respectively, where he studied under a Harvard Fellowship. In 1994, he spend a half-year as a Postdoctoral Fellow at the University of Cambridge's Sir Isaac Newton Institute for Mathematical Sciences. Currently, he Is an assistant professor of electrical engineering at Yale University. He teaches courses on signals and systems, pattern and object recognition, and computer vision. He is a member of the Yale Center for Computational Vision and Control and a member of the Army research Center for Imaging Science. He has published over twenty papers.on Image processing and 'computational vision. He is a recipient of a U.S. National Science Foundation Career Award; he has been awarded a Yale University Junior Faculty Fellowship for the Natural Sciences; and he won the IEEE Best Paper Award for his work on characterizing the set of images of an object under variable illumination. Joao P. Hespanha received the Licenciatura and MS degrees in electrical and computer engineering from Instituto Superior Tecnico, Lisbon, Portugal, In 1991 and 1993, respectively, and a second MS degree in electrical engineering from Yale University, New Haven, Connecticut, in 1994, where he is currently pursuing the PhD degree in engineering and applied science. From 1987 to 1989, he was a research assistant at Institute de Engenharia de Sistemas e Computadores (INESC) in Lisbon, Portugal, and,
from 1989 to 1990, was an instructor at Fundo para o Desenvolvimento Tecnol6gico (FUNDTEC) in the areas of electronics, computer science, and robotics. From 1.992 to 1993, he was a working partner in Sociedade de Projectos em Sistemas e Computadores, Lda., also in Lisbon. His research interests include nonlinear control, both robust and adaptive, hybrid systems, switching control, and the application of vision to robotics.
David J. Kriegman received the BSE degree (summa cum laude) in electrical engineering and computer science in 1983 from Princeton University, where he was awarded the.Charles Ira Young Award for electrical engineering research. He received the MS degree in 1984 and PhD in 1989 in electrical engineering from Stanford University, where he studied under a Hertz Foundation Fellowship. Currently, he is an associate professor at the Center for LComputautional Vision and Control in the Departments of Electrical Engineering and Computer Science at Yale University and was awarded a U.S. National Science Foundation Young Investigator Award in 1992. His paper on characterizing the set of images of an object under variable illumination received the best paper at the 1996 IEEE Conference on Computer Vision and Pattemrn Recognition. He has published over sixty papers on object representation and recognition, illumination modeling, geometry of curves and surfaces, structure from motion, robot planning, and mobile robot navigation.
-",,-,,mmmmlmll,
Application No. 11/231,353 Docket No. 577832000200

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL 17, NO. 10, OCTOBER 1995
Person Identification Using Multiple Cues

Roberto Brunelli and Daniele Falavigna
Abstract-This paper presents a person Identification system
based on acoustic and visual features. The system is organized as a set of non-homogeneous classifiers whose outputs are Integrated after a normalization step. In particular, two classifiers based on acoustic features and three based on visual ones provide data for an Integration module whose performance Is evaluated. A novel technique for the integration of multiple classifiers at an hybrid rank/measurement level Is introduced using HyperBF networks. Two different methods for the rejection of an unknown person are Introduced. The performance of the integrated system is shown to be superior to that of the acoustic and visual subsys.tems. The resulting Identification system can be used to log personal access and, with minor modifications, as an identity verification system. Index Terms-Template matching, robust statistics, correlation, face recognition, speaker recognition, learning, classifica-
tion.
I. INTRODUCTION
THE
identification of a person interacting with computers represents an important task for automatic systems in the area of information retrieval, automatic banking, control of access to security areas, buildings, and so on. The need for a reliable identification of interacting users is obvious. At the same time it is well known that the security of such systems is too often violated in every day life. The possibility of integrating multiple identification cues, such as password, identification card, voice, face, fingerprints, and the like, will, in principle, enhance the security of a system to be used by a selected set of people. This paper describes in detail the theoretical foundations and design methodologies of a person recognition system that is part of MAIA, the integrated AI project under development at IRST [ I . Previous works about speaker recognition [2], [3] have proposed methods for classifying and combining acoustic features and for normalizing [4], [5] the various classifier scores. In particular, score normalization is a fundamental step when a system is required to confirm or reject the identity given by the user (user verification): In this case, in fact, the identity is accepted or rejected according to a comparison with a preestimated threshold. Since the integration of voice and images in an identification system is a new concept, new methods for both classifier normalization and integration were investigated. Effective ways of rejecting an unknown person by considering score and rank information and for comparing images with
Manuscript received July 28, 1994; revised April 17. 1995. Recommended for acceptance by B. Dom. / The authors are with the Istituto per la Ricerca Scientifica e Tecnologica, 1-38050 Povo, Trento, ITALY; e-mail: brunelli@irst.itc.it, falavi@irst.ite.it. IEEECS Log Number P95111.
improved similarity measures are proposed. A simple method for adapting the acoustic models of the speakers to a real operating environment was also developed. The speaker and face recognition systems are decomposed into two and three single feature classifiers, respectively. The resulting five classifiers produce nonhomogeneous lists of scores that are combined using two different approaches. In the first approach, the scores are normalirzed through a robust estimate of the location and scale parameters of the corresponding distributions. The normalized scores are then combined using a weighted geometric average and the final identification is accepted or rejected according to the output of a linear classifier, based on score and rank information derived from the available classifiers. Within the second approach, the problem of combining the normalized outputs of multiple classifiers and of accepting/rejecting the resulting identification is considered a learning task. A mapping from the scores and ranks of the classifiers into the interval (0, 1) is approximated using a HyperBF network. A final threshold is then introduced based on cross-validation. System performance is evaluated and discussed for both strategies. Because of the novelty of the problem, standard databases for system training and test are not yet available. For this reason, the experiments reported in this paper are based on data collected at IRST. A system implementation operating in real-time is available and was tested on a variety of IRST researchers and visitors. The joint use of acoustic and visual features proved effective in increasing system performance and reliability. The system described here represents an improvement over a recently patented identification system based on voice and face recognition [6], [7]. The two systems differ in many ways: In the latter the speaker and face recognition systems are not further decomposed into classifiers, the score normalization does not rely on robust statistical techniques, and, finally, the rejection problem is not addressed. The next sections will introduce the speaker and face recognition systems. The first approach to the integration of classifiers and the linear accept/reject rule for the final system identification are then discussed. Finally, the novel rank/measurement level integration strategy using a HyperBF network is introduced with a detailed report on system performance. II. SPEAKER RECOGNITION The voice signal contains two types of information: individual and phonetic. They have mutual effects and are difficult to separate; this represents one of the main problems in the development of automatic speaker and speech recognition systems. The consequence is that speaker recognition systems
1995 IEEE
0162-8828/95504.00
956
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE. VOL. 17, NO. 10, OCTOBER 1995
perform better on speech segments having specific phonetic contents while speech recognition systems provide higher accuracy when tuned on the voice of a particular speaker. Usually the acoustic parameters for a speech/speaker recognizer are derived by applying a bank of band-pass filters to adjacent short time windows of the input signal. The energy outputs of the filters, for various frames, provide a good domain representation. Fig. 1 gives an example of such an analysis. The speech waveforms correspond to utterances of the Italian digit 4 (Ikwat:rol) by two different speakers. The energy outputs of a 24 triangular band-pass filter bank are represented below the speech waveforms (darker regions correspond to higher energy yalues).
VW
Fig. 2. The speaker recognition system based on VectorQuantization.
Since the power spectrum of the speech signal decreases as frequency increases a preemphasis filter that enhances the higher frequencies is applied to the sampled input signal. The transfer function of the filter is H(z) = 1/(1 - 0.95 z ). The preemphasized signal, x(n), I 5 n 5 N, is subdivided into frames y;(n). I <5t 5 T, having length L. Each frame is obtained by multiplying x(n) by a Hamming window h,(n): N y(n) = x(n) h(n)
S h, (n) = 0.54 - 0.46. co
Ilb-
C
yI....._...
.Y
IA~a
-.............
(21(n- tS)" (, t L2
L L S< n S tS + 2 2
(2)
U
Fig. I. Acoustic analysis of two utterances of the digit 4 (/kwat:ro/) by two different speakers.
In the equation above, L represents the length, in samples, of the Hamming window and S is the analysis step (also expressed in samples). For the system, L and S were chosen to correspond to 20 ms and 10 ms, respectively. The signal is multiplied by an Hamming window (raised cosine) to minimize the sidelobe effects on the spectrum of the resulting sequence y,(n). The acoustic analysis of each frame is performed as follows:
1) the power spectrum of the sequence yt,(n) is evaluated;
in the past years several methods and systems for speaker identification [8), (31 were proposed that perform more or less efficiently depending on the text the user is required to utter (in general, systems can be distinguished into text dependent or text independent), the length of the input utterance, the number of people in the reference database, and, finally, the time interval between test and training recordings. For security applications, it is desirable that the user utter a different sentence during each interaction. The content of the utterance can then be verified to ensure that the system is not cheated by prerecorded messages. For this work, a text independent speaker recognition system based on vector quantization (VQ) 19] was built. While it cannot yet verify the content of the utterance, it can be modified (using supervised clustering or other techniques) to obtain this result. A block diagram of the system is depicted in Fig. 2. In the system, each reference speaker is represented by means of two sets of vectors (codebooks) that describe his/her acoustic characteristics. During identification, two sets of acoustic features (static and dynamic), derived from the short time'spectral analysis of the input speech signal, are classified by evaluating their distances from the prototype vectors contained in the speaker codebook couples. In this way, two lists of scores are sent to the integration module. In the following, both the spectral analysis and vector quantization techniques will be described in more detail (see also [10) and a reference book such as [ 11).
2) a bank of Q = 24 triangular filters, spaced according to a. logarithmic scale (Mel scale), is applied to the power spectrum and the energy outputs s,, I 5 q 5 Q, fromeach filter are evaluated; 3) the Mel Frequency Cepstrum Coefficients (MFCC) 1121,. 0p, 5< 5 P=8, are computed, from the filter bank outp puts, according to the following equation:
t2 -'
q_ I
os p q
a
Q
0m = q=) t og(sr,,,
2JQ
the MFCCs are arranged into a vector, 0,, which is called static, since it refers to a single speech frame; 4) to account for the transitional information contained in the speech signal a linear fit is applied to the components of seven adjacent MFCCs; the resulting regression coefSficients are arranged into a vector that is called dynamic; 5) a binary variable is finally evaluated that allows marking the frame as speech or background noise; this parameter is computed by means of the algorithm described in [13]. The Mel scale is motivated by auditory analysis of sounds. The inverse Fourier transform of the log-spectrum (cepstrum) provides parameters that improves performance at both speech and speaker recognition [ ll], [ 12). Furthermore, the Euclidean distance between two cepstral vectors represents a good measure for comparing the corresponding speech spectra. The static and dynamic 8D vectors related to windows marked as background noise are not considered during both system
BRUNELLI AND FALAVIGNA: PERSON IDENTIFICATION USING MULTIPLE CUES
S 957
training and testing. As previously said, VQ is used to design the static and dynamic codebooks of a given reference speaker, say the ith one. Starting from a set of training vectors (static or dynamic) O9= ( 011, ... , Ox), derived from a certain number of utterances, the objective is to find a new set r ), with M < K, that represents well the .... acoustic characteristics of the given speaker. To do this a
= .E,
about five months after the training recordings. Due to both the different background noise and acquisition conditions between training and test, the codebooks must be adapted. Adaptation. means designing a new codebook, starting from) a given one, that better resembles the acoustic characteristics of both the operating environment and the acquisition channel, Adaptation should also take into account variations in time of the algorithm, similar to that described in [10], is ap- speaker's voice (intraspeaker'variations). Adaptation requires clustering plied to the 9 j set. The algorithm makes use of an iterative the use of few utterances to modify the codebook as it is not procedure that allows determination of codebook centroids, necessary to design it from scratch (this would require at least W,, by minimizing their average distance, D(O, 'Y), from the 30-40 seconds of speech). In our case, the adaptation vectors are derived from the digit strings uttered by the users during a training vectors: single test session. The dynamic codebooks were not adapted " D(641,7)= min[d(ik.Vim), (4) since they represent temporal variations of the speech spectra K k= M= and therefore they are less sensitive to both intraspeaker voice variability and acquisition channel variations. The distance d(Oa, V,,) is defined as follows: The adaptation process of the ith codebook, C; can be sumd(O, Vim) = (rk - Vim)' W"(O - VJm) (5) marized as follows: In the equation above t denotes transpose and W is the co1) the mean vectors p, and v1 of the adaptation vectors and variance matrix of the training vectors. The matrix W is estiof the given codebook respectively are evaluated; mated from'the training data of all the speakers in the refer2) the difference vector , = p- v1 is evaluated; ence database. This matrix was found to be approximately 3) the vectors of C, are shifted by a quantity equal to A, obdiagonal, so that only the diagonal elements are used to evalutaining a new set C' = (c, + 4, ... , ca + A,); the set C' ate distances. is placed in the region of the adaptation vectors; In the recognition phase the distances, Dsi, DD,, between the 4) the adaptation vectors are clustered using the set C, as static and dynamic vector sequences, derived from the input initial estimate of the centroids; therefore a new set of signal, and the corresponding speaker codebooks are evaluated centroids 0, = (on, .... oi) and the corresponding cell and sent to the integration module. '
K
If = { ... , Or) is the static (or dynamic) input sequence and Y'Iis the ith static (or dynamic) codebook, then the total
static (or dynamic) distance will be:

T
T=
occupancies N, = nn .... n) are evaluated; 5) the adapted codebook { V); is obtained according to the following equation: ,1 =c,,,+ l-e-a
1 il1 (6)
o,,-ca)
1<m5M
(7)
In the equation above the parameter n,,, determines the 1 where 1 is the total number of speakers in the reference datafraction of deviation vector 6,,, = (o., -c,,,), that has to base. be summed to the initial centroid c,! . Equation 7 is a To train the system, 200 isolated utterances of the Italian simple method to modify the centroids of a codebook acdigits (from 0 to 9) were collected for each reference user. The cording to the number of data available for their estirecordings were realized by means of a digital audio tape mates. ,,,can be zerowhen the utterance used for adap(DAT): The signal on the DAT tape, sampled at 48 kHz, was tation does not contain sounds whose spectra are related down sampled to 16 kHz, manually end-pointed, and stored on to the mth centroid. a computer disk. The speech training material was analyzed in 19), For the system, a was chosen equal to 0.1. The two shifts and clustered as previously described. As demonstrated performance depends on both input utterance length applied by the adaptation procedure can be interpreted as folsystem and codebook size; preliminary experiments have suggested lows: that the speaker, to be identified, should utter a string of at I) 4,, the major shift, accounts for environment and channel least seven digits in a continuous way and in whatever order. variations with respect to training; In the reported experiments the number of digits was kept 2) 6,,,,, the minor shift, accounts for intraspeaker voice equal to seven and the codebook size was set to M = 64 bevariations in time. cause higher values did not improve recognition accuracy. Furthermore, if input signal duration is too short, the system IT. FACE RECOGNITION requires the user to repeat the digit string. To evaluate integrated system performance (see SecPerson identification through face recognition is the most tion IV.A) the reference users interacted three times with the familiar among the possible identification strategies. Several system during three different sessions. The test sessions were automatic or semiautomatic systems were realized since the carried out in an office environment using an ARIEL board as early seventies-albeit with varying degree of success. Differacquisition channel. Furthermore the test phase was performed ent techniques were proposed, ranging from the geometrical
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 17, NO. 10, OCTOBER 1995
description of salient facial features to the expansion of a digitized image of the face on an appropriate basis of images (see (141 for references). The strategy used by the described sys tem is essentially based on the comparison, at the pixel level , of selected regions of the face (14]. A set of regions, respecti vely encompassing the eyes, nose, and mouth of the user to be identified are compared with the corresponding regions stcored in the database for each reference user (see Fig. 3). The images should represent a frontal view of the user face without marked expressions. As will be clear from the detailed descript ion, these constraints could be relaxed at the cost of storin g a higher number of images per user in the database. The fundamental steps of the face recognition process are the followi ng: 1) acquisition of a frontal view of the user face; 2) geometrical normalization of the digitized image; 3) intensity normalization of the image; 4) comparison with the images-stored in the database.
tically into two, slightly overlapping parts containing the left and right eye, respectively. The illumination under which the image is taken can impair the template matching process used to locate the eye.' To minimize this effect a filter, .N(1), is applied to image I
if
.N': 1
N'
if
I 1* KG()
N~'>*i
where
=
(10)
The Fig. 3. highlighted regions represent the templates used for identification.
The image of the user face is acquired with a CCD camera and digitized with a frame grabber. To compare the resulting image with those stored in the database, it is necessary to register the image: It has to be translated, scaled, and rotated so that the coordinates of a set of reference points take corresponding standard values. As frontal views are considered, the centers. of the pupils represent a naturalset of control points that can be located with good accuracy. Eyes can be found through the following steps: I1) locate the (approximate)symmetry axis of the face; 2) locate the left/right eye by using an eye template for which the location of the pupil is known; if the confidence of the eye location is not sufficiently high, declare failure (the identification system will use only acoustic
and KG, is a Gaussian kernel whose ar is related to the expected interocular distance 4,. The arithmetic operations act on the values of corresponding pixels. The process mapping I into Nreduces the influence of ambient lighting while keeping the necessary image details. This is mainly due to the removal of linear intensity gradients that are mapped to the constant value 1. Extensive experiments, using ray-tracing and texturemapping techniques to generate synthetic images under a wide range of lighting directions have shown that the local contrast operator of (9) exhibits a lower illumination sensitivity than other operators such as the laplacian, the gradient magnitude or direction 115] and that there is an optimal value of the parameter a (approximately equal to the iris radius). The same filter is,applied to the eye templates. The template matching process is based on the algorithm of hierarchical correlation proposed by Burt (16]. Its final result is a map of correlation values: the center of gravity of the pixels with maximum value representing the location of the eye. Once the two eyes have been located, the confidence of the localization is expressed by a coefficient, CE, that measures the symmetry of the eye positions with respect to the symmetry axis, the horizontal alignment and the scale relative to that of the eye templates: e
2
max(C1,C )
'
(11)
where C, and C, represent the (maximum) correlation value for the left/right eye, s the interocular distance expressed as a multiple of the interocular distance of the eyes used as templates, AO represents the angle of the interocular axis with respect to the horizontal axis, while a, and a, represent tolerinformation); ances on the deviations from the prototype scale and orienta3) achieve translation, scale, and rotation invariance by fix- tion. ing the origin of the coordinate system at the midpoint of The first factor in the RHS of (11) is the average correlation. the interocular segment and the interocular distance and value of the left and right eye: The higher it is the better the the direction of the eye-to-eye axis at predefined values. match with the eye templates. The second factor represents the Under the assumption that the user face is approximately verti- symmetry of the correlation values and equals I when the two cal in the digitized image, a good estimate of the coordinate S values are identical. The third and fourth factors allow weighing the deviation from both the assumed scale and (horizontal) of the symmetry axis is given by orientation of the interocular axis, respectively. The parameS= median(Pv(IlI* Kvl)i (8) ters of the Gaussians, a, and a, were determined by the where * represent convolution, I the image, Kv the convolution analysis of a set of interactions. If the value of CE is too low, the face recognition system Pv kernel [-1, 0, I', the vertical projection whose index i runs
over the columns of the image. The face can then be split ver.-
declares failure and the identification proceeds using the
acoustic features alone. Otherwise, the image is translated, scaled and rotated to match the location of the pupils to that of the database images. In the reported experiments the interocular distance was set equal to 28 pixels. Alternative techniques for locating eyes are reported in [17), 118)]. Due to the geometrical standardization, the subimages containing the eyes, nose, and mouth are approximately characterized by the same coordinates in every image. These regions are extracted from the image of the user face and compared in turn to the corresponding regions extracted from the database entries, previously filtered according to (9) and (10). Let us introduce a similarity measure C based on the computation of the Lt norm
o grtwe,,
0.0
0.2' .0.4
0.6
oa
1.0
cores
Fig. 4. The distribution of the correlation values for corresponding features of
the same person and of different people.
of a vector I IIL =
lance dQ,, (x.y)=
, ix, and on the corresponding dis-
rylj,:
C(x,y)= 1- d(x,)
d(x,y)
(12)
The Lt distance of two vectors is mapped by C(.,.) into the interval [0, 1], higher values representing smaller distances. This definition can be easily adapted to the comparison of images. For the comparison to be useful when applied to real images, it is necessary to normalize the images so that they have the same average intensity p and standard deviation (or scale) a. The latter is particularly sensitive to values far from the average p so that the scale of the image intensity distribution can be better estimated by the following quantity:
Integration with the scores derived from the acoustic analysis can now be performed with a single or double step process. In the first case, the two acoustic and the three. visual scores are combined simultaneously, while in the second the acoustic and visual scores are first combined separately and the final score is given by the integration of the outputs of the speaker and face recognition systems (see [7) for an example of the latter). The next section will introduce two single-step integration strategies for classifiers working at the measurement level. IV. INTEGRATION
The use of multiple cues, such as face and voice, provides in a natural way the information.necessary to build a reliable, high performance system. Specialized subsystems can identify I n (or verify) each of the previous cues and the resulting outputs (13) x; - l 7 , = can then be combined into a unique decision by some integranfl tion process. The objective of this section is to describe and where the image is considered as a one dimensional vector x. evaluate some integration strategies. The use of multiple cues The matching of an image B to an image A can then be quantifor person recognition proved beneficial for both system perfled by the maximum value of C(A, B), obtained by sliding the' formance and reliability.' smaller of the two images over the larger one. A major advanA simplified taxonomy of multiple classifier systems is retage of the image similarity computed according to (12) over ported in [22). Broadly speaking, a classifier can output inforthe more common estimate given by the cross-correlation co=' mation at one of the following levels: efficient (19), based on the L.norm, is its reduced sensitivity * The abstract level: the output is a subset of the possible to small amounts of unusually high differences between correidentification labels, without any qualifying information; sponding pixels. These differences are often due to noise or * The rank level: the output is a subset of the possible laimage specularities such as iris highlights. A detailed analysis bels, sorted by decreasing confidence (which is not supof the similarity measure defined in (12) is given in [20]. An plied); , alternative technique for face identification is reported in 121). * The measurement level: the output is a subset of labels the Let us denote with {UJ,,, ,..,s set of images available for qualified by a confidence measure. the kth user. A comparison can now be made between a set of regions of the unknown image N( and the corresponding reThe level at which the.different classifiers of a composite gions of the database images. The regions currently used by system work clearly constrains the ways their responses can be the system correspond to the eyes, nose and mouth. A list of merged. The first of the following sections will address the similarity scores is obtained for each region ., of image U,,,: integration of the speaker/face recognition systems at the
{Sa
} = maxC(Ra ().7 a(Uk,))
(14)
where R,(JV) represents a region of Neontaining JF, with a frame whose size is related to the interocular distance. The lists of matching scores corresponding to eyes, nose, and mouth are then available for further processing. The distribution of the correlation values for corresponding features of the same person and of different people are reported in Fig. 4.
measurement level. The possibility of rejecting a user as unknown will then be discussed. Finally, a novel, hybrid level approach to the integration of a set of classifiers will be presented.
1. Two aspects of reliability are critical for a person identification system: The first is the ability of rejecting a uscr as unknown, the second is the possibility of working with a reduccd input, such as only the speech signal or the
face image.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 17, NO. 10, OCTOBER
1995
A. Measurement Level Integration
The acoustic and visual identification systems already constitute a multiple classifiersystem. However, both the acoustic and visual classifiers can be further split into several subsystems, each one based on a single type of feature. In our system, five, classifiers were considered (see Sections II and III) working on the static, dynamic acoustic features, and on the eyes, nose and mouth regions. A critical point in the design of an integration procedure at the measurement level is that of measurement normalization. In fact, the responses of the different classifiers usually have different scales (and possibly offsets), so that a sensible combination of the outputs can proceed only after the scores are properly normalized. As already detailed, the outputs of the identification systems are not homogeneous: the acoustic features provide distances while the visual ones provide correlation values. A first step towards the normalization of the scores is to reverse the sign of distances, thereby making them concordant with the correlation values: the higher the value, the more similar the input patterns. Inspection of the score distributions shows them to be markedly unimodal and roughly symmetrical. A simple way to normalize scores is to estimate their average values and standard deviations so that distributions can be translated and rescaled in order to have zero average and unit variance. The values can then be forced into a standard interval, such as (0, 1), by means of an hyperbolic tangent mapping. The normalization of the scores can rely on a fixed set of parameters, estimated from the score distributions of a certain number of interactions, or can be adaptive, estimating the parameters from the score distribution of the current interaction. The latter strategy was chosen mainly because of its ability to cope with variations such as different speech utterance length without the need to re-estimate the normalization parameters. The estimation of the location and scale parameters of the distribution should make use of robust statistical techniques (23), [24]. The usual arithmetic average and standard deviation are not well suited to the task: they are highly sensitive to outlier points and could give grossly erroneous estimates. Alternative estimators exist that are sensitive to the main bulk of the scores (i.e. the central part of a unimodal symmetric distribution) and are not easily misled by points,in the extreme tails of the distribution. The median and the Median Absolute Deviation (MAD) are examples of such location and scale estimators and can be used to reliably normalize the distribution of the scores. However, the median and the MAD estimators have a low efficiency relative to the usual arithmetic average and standard deviation. A.class of robust estimators with higher efficiency was introduced by Hampel under the name of tanhestimators and is used in the current implementation of the system (see [23) for a detailed description). Therefore each list of scores (SY) .I,..., from classifier j, being I the number of people in the reference database, can be transformed into a normalized list by the following mapping:
So =i- tanh 0.01 S~l

2
+1 E(0,1)
(15)
creans
where p,,, and oh are the average and standard deviation estimates of the scores {Sy), ,....,; by the Hampel esas given timators. An example of distributions of the resulting normalized scores is reported in Fig. 5 for each of the five features used in the classification.
Oistrioulions Normolie of Scores 1 -. cSt , o Sz rt. o r2 &r3
400
30
---
200
-,
00.48 0.50 0.52 Normolzeo0 SCcore
Fig. 5. The density distribution of the normalized scores for each of the classifiers: S1, S2 represent the static and dynamic speech scores while Fl, F2.
and F3 represent the eyes, nose, and mouth scores, respectively.
In the following formulas; a subscript index imindicates the mth entry within the set of scores sorted by decreasing value. The normalized scores can be integrated using a weighted
geometric average:
=i
(nS . jJ 1 )lW
(16)
where the weights wj represent an estimate of the score dispersion in the right tail of the corresponding distributions:
S'~ 0.5 S' l-0.51.0

S; 1 - 0.5
(17)
The main reason suggesting the use of geometric average for the integration of scores relies on probability: If we assume that the features are independent the probability that a feature vector corresponds to a given person can be computed by taking the product of the probabilities of each single feature. The normalized scores could then be considered as equivalent to probabilities. Another way of looking at the geometric average
is that of predicate conjunction using a continuous logic. [25),
[26]. The weights reflect the importance of the different features (or predicates). As defined in (17), each feature is given an importance proportional to the separation of the two best scores. If the classification provided by a single feature is ambiguous, it is given low weight. A major advantage of (16) is that it does not require a detailed knowledge of how each feature is distributed (as would be necessary when using a Bayes approach). This eases the task of building a system that integrates many features. The main performance measure of the system is the percentage of persons correctly recognized. Performance can be further qualified by the average value of the following ratio Rx:
R S;i - S i
isis/.
(18)
max)(5;,)- S1,
961
The ratio R,measures the separation of the correct match SX from the wrong ones. This ratio is invariant against the scale and location parameters of the" integrated score distribution and can be used to compare different integration strategies (weighted/unweighted geometric average, adaptive/fixed normalization). The weighted geometric average of the scores adaptively normalized exhibits the best performance and separation among the various schemes on the available data. Experiments have been carried out using data acquired' during three different test sessions. Of the 89 persons stored in the database, 87 have interacted with the system in one or more sessions. One of the three test sessions was used to adapt the acoustic and visual databases (in the last case the images of the session were simply added to those available); therefore, session one was used to adapt session two and session two to adapt session three. As the number of interactions for each adapted session is 82,.the total number of test interactions was 164. The recognition performance and the average value of R, for the different separate features and for their integration are reported in Table 1.:
TABLE I
THE RECOGNITmON PERFORMANCE AND AVERAGE SEPARATION RATIO R FOR
EACH SINGLE FEATURE AND POR THEIR INTEGRATION. DATA.ARE BASED ON 164 REAL INTERACTIONS AND A DATABASE OF 89 USERS.
can then be quantified through several measures. The decision about whether the confidence is sufficient to accept the system output can be based on one or several of them. In the proposed system, a linear classifier, based on absolute and relative scores, ranks and their dispersion, will be used to accept/reject the final result. The following issues will be discussed: 1) degree of dependence of the features used; 2) choice of the confidence measures to be used in the accept/reject rule; 3) training and test of the linear classifier used to implement the accept/reject rule. As a preliminary step, the independence of the features used in the identification process will be evaluated. It is known that the higher the degree of independence, the higher the information provided to the classifier. Let us consider a couple of features X and Y.Let {(x, yi) =I.....t the corresponding represent normalized scores. They can be considered as random samples from a population with a bivariate distribution function. Let A,
be the rank of x, among
x,,
..., x,when they are arranged in
y, descending order, and BI the rank of y, among Y,....defined similarly to A,. Spearman's rank correlation [27) is defined by:
__
o . ..
(19)
Feature Voice Static Dynamic Face Eyes Nose

Mouth
Recognition (%) 88 77 71 91 80 77
-83
R 1.14 1.08 1.08 1.56 1.25 1.25

1.28
IXAI
-
)2
X(Bi-
ALL
98
1.65
where A and B are the average values of (AI) and (B,),.respectively. An important characteristic of rank correlation is its non-parametric nature. To assess the independence, of the features it is not necessary to know the bivariate distribution from which the (XI, YI) are drawn, since the distribution of their ranks is known, under the assumption of independence. It turns out that
1-2
B. Rejection An important capability of a classifier is to reject input patterns that cannot be classified in any of the available classes with a sufficiently high degree of confidence. For a person verification system, the ability to reject an impostor is critical. The following paragraphs introduce a rejection strategy that takes into account the level of agreement of all the different classifiers in the identification of the best candidate. A simple measure of confidence is given by the integrated score itself: the higher the value, the higher the confidence of. the identification. Another is given by the difference of the two best scores: It is a measure of how sound the ranking of the best candidate is. The use of independent features (or feature sets) also provides valuable information in the form of the rankings of the identification labels across the classifier outputs: If the pattern does not .belong to any of the known classes, its rank will vary significantly from classifier to classifier. On the contrary, if the pattern belongs to one of the known classes, rank agreement will be consistently high. The average rank and the rank dispersion across the classifiers can then be used to quantify the agreement of the classifiers in the final identification. The confidence in the final identification
(20)
TABLE II THE RANK CORRELATION VALUE OF THE COUPLES OF FEATURES. THE PARDfrHESIZE) VALUES REPRESEMT THE SIONIFICANCE OF THE CORRELATION.
St AND S2 REPRESENT THE DYNAMIC AND STATIC ACOUSTIC FEATURES
RESPECnvELY; Fl, F2, F3 REPRESENT THE EYES, NOSE AND MOUTHn
SI S S2
Fl 1.00 1 (1.00)
SZ 0.64
(1.00)
Fl 0.06
(0.72)
F2 0.08
(0.77)
F3 0.04
(0.66)
1.00 (1.00)
0.03 (0.61) 1.00 S(1.00)
F2
F3
0.07 (0.75) 0.11 (0.84 1.00" (1.00)
0.03 0.61 0.10 (0.83) 0.50 (1.00)

1.00 (1.00)
is distributed approximately as a Student's distribution with

I- 2 degrees of freedom [271; It is then possible to assess the dependence of the different features used by computing the rank correlation of each couple and by testing the correspond-
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 17, NO. 10, OCTOBER 1995
ing significance. Results for the features used in the system developed are given in Table II. SThe acoustic features are clearly correlated, as well as the nose and mouth features. The latter correlation is due to the overlapping of the nose and mouth regions, which was found to be necessary in order to use facial regions characterized by the same coordinates for the whole database. Acoustic and visual features are independent, as could be expected..'
Score
database, thereby simulating the interaction with a stranger. The linear discriminant function defined by the vector w can be found by minimizing the following error:
E= a 1/2 /2
tS+e-( .,-,,il 1 le wN,) +e
1-# Il
wnA
J.J
(22)
where a and represent the weight to be attributed to false negatives and to false positives respectively and 1= 18 is the
dimensionality of the input vectors. When a = = 1, E repre-
"
l r "
,Rar etgma
sents the output error of a linear perceptron with a symmetric sigmoidal unit. Final acceptance or rejection of an identification, associated to a vector d, is done according to the simple rule: wad i + w,+, > 0
1=1
34
r1 "
accept
(23)
"
Fig. 6. Let us represent the match with the database entries by means of the integrated score, the standard deviation of the rankings across the different of features and the normalized ratio the first to second best integrated score. The resulting three dimensional points are plotted and marked with a - if they represent a correct match or with a x if the match is incorrect. Visual inspection of the resulting point distribution shows that the two classes of points can be separated well by using a plane.
wd, + w+
reject
(24)
The feasibility of using a linear classifier was investigated by looking at the distribution of acceptable and nonacceptable 1 best candidates in a 3D space whose coordinates are the integrated score, a normalized ratio of the first-to second best score and the standard deviation of the rankings. As can be seen in Fig. 6 a linear classifier seems to be appropria ate. The full vector de R t used as input to the linear classifier is given by: 1) the integrated score, St, of the best candidate; 2) the normalized ratio of the first to the second best integrated score: Sh -0.5 Sq-0.5 3) the minimum and maximum ranks of the first and second final best candidates (four entries); 4) the rank standard deviation of the first and second final best candidates (two entries); 5) the individual ranks of the first and second final best candidates (10 entries). To train the linear classifier the following procedure was used.
A set of positive examples (pi) is derived from the data rela-
tive to the persons correctly classified by the system. A set of negative examples (nj) is given by the data relative to the best candidate when the system did not classify the user correctly. The set of negative examples can be augmented by the data of the best candidate when the correct entry is removed from the
2. Unacceptable best candidates derive from two sources: misclassified users from real interactions and best candidates from virtual interactions, characterized by the removal of the user entry from the data base.
Note that the LHSs of (23) and (24) represent the signed distance, in arbitrary units, of point d from the plane defined by w that divides the space into two semispaces. Points lying in the correct semispace contribute to E inversely to their distance from plane w. Points lying near the plane contribute with a or while points lying in the wrong semispace and at great distance from the discriminating plane contribute with 2aor 2/3. If the two classes of points are linearly separable it is possible to drive E to zero (see [281, [291). A stochastic minimization algorithm (301, (31] was used to minimize E. When the system is required to work in a strict mode (no errors allowed, that is, no strangers accepted), >> a should be considered in the' training phase'. Note that a similar discriminant function can be computed for each of the recognition subsystems (i.e., face recognition and voice recognition), thereby enabling the system to reject an identification when it is not sufficiently certain even when not all of the identification cues are available. The training/test of the classifier followed a leave-one-out strategy to maximize the number of data available in the training phase [32]. The classifier is trained by using all but one of the available samples and tested on the excluded one. Theperformance of the classifier can be evaluated by excluding in turn each of the available samples and averaging the classification error. In the reported experiments, the available examples were grouped per interacting user. The leave-one-out method was then applied to the resulting 87 sets (the number of users that interacted with the system) to guarantee the independence of the training and test sets. Each set was used in turn for testing, leaving the remaining 86 for training. The results are reported inTable Ill. A complete operating characteristic curve for the integrated performance shown in Table l is reported in Fig. 7 where the stranger-accepted and familiar-rejected rates at different fPa ratios are plotted.
BRUNELLI AND FALAVIONA: PERSON IDENTIFICATION USING MULTIPLE CUBS
TABLE ll ERROR RATES OF THE SUBSYSTEMS AND OF THE COMPLETE SYSTEM WHEN A
REJECION TNhRESHOLD Is INTRODUCED. DATA ARE BASED ON THE SUBSET OF
INmRACfONS FOR WHICH BOTH FACE AND SPEECH DATA WERE AVAn.ILABLE
(i 55'Our OF 164) Error (%) Face
Stranger accepted Familiar rejected Familiar misrecog

Voice Stranger accepted Familiar rejected Familiar misrecog.
Integrated Stranger accepted Familiar rejected Familiar misrecog.
4.0 8.0 0.5
Let ( C) be the set of classifiers. Each of them associates to each person X some numerical data Xj that can be considered a vector. By comparison with the ith database enitry, a normalized similarity score Su can be computed. Each score Sy can be associated to its rank rU in the list of scores produced by classifier C. The output of each classifier can then be regarded as a list of couples {((Sj, r )) =l.....t where I represents the number of people in the reference database. A mapping is sought o1 such that: Ssris)= if i is the correci label for X (25)
Ail(Soj,rit, .... Sis~rs)=
tews
(25)
14.0 27.0 1.0

0.5 1.5 0.0
oSlongr occropd
o for.wlo rejected
If, after mapping the list of scores, more than a label qualifies, the system rejects the identification. It is possible to relax the definition of 4o by letting the value of the mapping span the whole interval [0, 1]. In this way the measurement level character of the classification can be retained. The new mapping can be interpreted as a fuzzy predicate. The following focuses on the fuzzy variant, from which the original formulation can be obtained by introducing a threshold m (26) where 0() is the Heavyside unit-step function and xj = (S 1, r 1, * , Sa, res) is a ten-dimensional vector containing the feature 1 5 normalized matching scores and corresponding ranks. The goal is to approximate the characteristic function of the correct matching vectors as a sum of Gaussian bumps. Therefore the search for is conducted within the following family of functions:
o = O((x4 ) -
(0)
0.0
0.5
1.0 P/a
1.5
2.0
Fig. 7. System performance when false' positives and false negatives are weighted differently.
L(;{cata}aT-=
where
G(x) e-'I2
c aG(IX-tall))
a
Similar experiments were run on the acoustic and visual features separately and are also reported in Table III. The resuits show that the use of the complete set of features provides a relevant increase in reliable performance over the separate subsystems. C. Hybrid Level Integration ' In this sub-section, a hybrid rank/measurement level at which multiple classifiers can be combined will be introduced. The approach is to reconstruct a mapping from the sets of scores, and corresponding ranks, into the set 10, I).The matching to each of the database entries, as described by a vector of five scores and the corresponding ranks should be mapped to 1,if it corresponds to the correct label, and to 0 otherwise. The reconstruction of the mapping proceeds along the following steps: 1) find a set of positive and negative examples; 2) choose a parametric family of mappings; 3) choose the set of parameters.for which the corresponding mapping minimizes a suitable error measure over the training examples. Another way to look at the reconstruction of the mapping is to consider the problem as a learning task, where, given a set of acceptable and non acceptable inputs, the system should be able to appropriately classify unseen data.
(27)
(28) (29)
Q~ -. )=
1-1.
.(30)
p1 being a diagonal matrix with positive entries, x, t, e R' and c, e R. The approximating function can be represented as a HyperBF network [33] whose topology is reported in Fig. 8. The sigmoidal mapping is required to ensure that the codomain be restricted to the interval (0, 1). The location Ia, shape , and height ca of each bump are chosen by minimizing the following error measure:
E=
I
y - a
a
caG
-to)
(31)
is (xu, where ({ yu) ), a set of examples (points at which the value of the mapping to be recovered is known). The first subscript I denotes the database entry from which xy is derived and the
'second subscriptj represents the example.
964
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTtLLIGENCE, VOL. 17, NO. 10, OCTOBER 1995
using o the system was able to reject all of the strangers,

which is the ultimatWrequirement for a reliable system, missing only 3.5% of the known users.
TABLE IV
THE PERFORMANCE OPFTHE SYSTEM WHEN USING A HYPERBF NETWORK
wrm 21 UNr's To PERFORM ScoRs
WrTGRATION
0
Fig. 8. The function used to approximate the mapping from the score/rank domain into the interval (0, 1) can be epmsented as a HyperBP network.
Stranger accepted
%(%)
Familiar rejected
Familiar misrecog. (%)
(4 +av)/2 ,.
0.5 0.5 0.0
3.0 3.0 3.5
0.0 0.0 0.0
The required value of the mapping at xu is 1 when i is the correct label (class) for the jth example and 0 otherwise. The error measure E is minimized over the parameter space ((ca, t,)), by means of a stochastic algorithm with adaptive memory (311. The number of free parameters involved in the minimization process dictates the use of a large set of examples. As a limited number of real interactions was available, a leave-one-out strategy was used for training and testing the system as for the linear classifier previously described. From each of the available user-system interactions, a virtual interaction was derived by removing from-the database the entry of the interacting user, thereby simulating an interaction with a stranger. For each interaction j 1) the vector corresponding to the correct database entry provides a positive example; 2) the vectors of the first ten, incorrect, entries of the real interaction (as derived from sorting the integrated scores of Section IV.A) and the vectors of the first ten entries of the virtual interaction provide the negative examples. The reason for using only the first ten incorrect entries is that the matching scores decay quickly with rank position in the final score list and additional examples would not provide more information. Data from different interactions of the same user were then grouped. The resulting set of examples was used to generate an equal number of different training/testing set pairs. Each set was used in turn for testing, leaving the remaining ones for training. The problem of matching the number of free parameters in the approximation function to the complexity of the problem was solved by testing the performance of networks with increasing size. For each network size, a value for threshold ca of (26) was computed to minimize the total error defined as the sum of the percentage of accepted strangers, misrecognized and rejected database persons. In Fig. 9, the total error is reported as a function of the network size. Note that the threshold is computed on the test set, so that it gives an optimistic estimate. To obtain a correct estimate of system performance, a cross-validation approach was used for the net giving the best (optimistic) total error estimate. Let [o, W] be the interval over which the total error assumes its 1 minimum value (see Fig. 10 ). The threshold value can be chosen as:
TotloError
0.08
I-
0.04
0,02
A no
10
20 Network units
Fig. 9. Tihe total error achieved by networks with different numbers of units. The total error is computed by summing the percentage of accepted strangers, misrecognized, and rejected database people. For each net size, a threshold was chosen to minimize the cumulative error.
Error estimates oSatrnger ocpte

o fa;e
rejected
a lorita mimc.
Fig. 10. Enmor percentages as a function of the rejection threshold for a Gaussian-based expansion. V. CONCLUSIONS
A system that combines acoustic and visual cues in order to
* ob favoring acceptance over rejection;
- (a+,)2;
* col favoring rejection over acceptance. The resulting performance is reported in Table IV. Note that
identify a person has been described. The speaker recognition sub-system is based on vector quantization of the acoustic parameter space and includes an adaptation phase of the codebooks to the test environment. A different method to perform speaker recognition, which makes use of the Hidden Markov Model technique is under investigation. A face recognition subsystem also was described. It is based on the comparison of facial features at the pixel level using a similarity measure based on the L,norm. The two subsystems provide a multiple classifier system. In the implementation described, five classifiers (two acoustic and three visual) were considered. The multiple classifier operates in two steps. In the first one, the input scores are normalized using robust estimators of location and scale. In the
BRUNELLI AND FALAVIGNA: PERSON IDENTIFICATION USING MULTIPLE CUBS
965.
second step, the scores are combined using a weighted geometric average. The weights are adaptive and depend on the score distributions. While normalization is fundamental to compensate for input variations (e.g., variations of illumination, background noise conditions, utterance length and of speaker voices), weighting emphasizes the classification power of the most reliable classifiers. The. ;use of multiple cues, acoustic and visual, proved to be effeg'e in improving performance. The correct identification rate of the integrated system is 98% which represents a significant improvement with respect to the 88% and 91% rates provided by the speaker and face recognition systems respectively. Future use of the hidden Markov model technique is expected to improve performance of the VQ-based speaker recognizer. An important capability of the multiple classifier itself is the rejection of the input data when they can not be matched with sufficient confidence to any of the database entries. An accept/reject rule is introduced by means of a linear classifier based on measurement and rank information derived from the five recognition systems. A novel, alternative, approach to the integration of multiple classifiers at the hybrid rank/measurement level is also presented. The problem of combining the outputs of a set of classifiers is considered as a learning task. A mapping from the scores of the classifiers and their ranks into the interval (0, 1) is approximated using a HyperBF network. A final rejection/acceptance threshold is then introduced using the cross-validation technique. System performance is evaluated on data acquired during real interactions of the users in the reference database. Performance of the two techniques is similar. , The current implementation of the system is working on an HP 735 workstation with a Matrox Magic frame grabber. In order to optimize system throughput, it relies on a hierarchical match with the face database. The incoming picture, represented by a set of features is compared at low resolution with the cqmplete database. For each person in the database, the most similar feature, among the set of available images, is chosen and the location of the best matching position stored. The search is then continued at the upper resolution level by limiting the search to the most promising candidates at the previous level. These candidates are selected by integrating their face scores according to the procedure described in Section IV.A. All available data must be used to secure a reliable normalization of the scores. However, new scores at higher resolution are computed only for a selected subset of persons and this constitutes a problem for the integration procedure. In fact, scores from image comparisons at different levels would be mixed, similarity values deriving from lower resolutions being usually higher. To overcome this difficulty, the scores from the previous level are reduced (scaled) by the highest reduction factor obtained comparing the newly computed scores to the corresponding previous ones. The performance, measured on the data sets used for the reported experiments, does not decrease and the overall identification time (face and voice processing) is approximately five seconds. The same approach, using codebooks of reduced size could
be applied to the speaker identification system, thereby increasing system throughput. Adding a subject to the database is a simple task for both subsystems. This is due to the modularity of the databases, each subject being described independently of the others. The integration strategy itself does not require any update. The rejection and the combined identification/rejection procedures do require updating. However, the training.of the linear perceptron and of the HyperBF network can be configured more as a refinement of a suboptimal solution (available from the.previous database) than as the computation of a completely unknown set of optimal parameters. While the system, as presented, is mainly an identification system, a small modification transforms it into a verification system. For each person in the database it is possible to select a subset containing the most similar people (as determined by the identification system). When the user must be verified the identification system can be used using the appropriate subset, thereby limiting, the computational effort, and verifying the identity of the user by means of the techniques reported in the paper. Future work will have the purpose of further improving the global efficiency of the system with the investigation of more accurate and reliable rejection methods.
ACKNOWLEDGMENTS
The authors would like to thank Dr. L. Stringa, Prof. T. Poggio and Prof. R. de Mori for valuable suggestions
and discussions. The authors are grateful to the referees for

many valuable comments. REFERENCES
(1) T. Poggio and L Stringa. "A project for an intelligent system: Vision and learning." Int'I J. Quantum Chemistry, vol. 42, pp. 727-739, 1992. 12) F.K. Soong and A.E. Rosenberg, "On the use of instantaneous and transitional spectral information in speaker recognition," IEEE Trans Acoustic, Speech, and Signal Processing, vol. 36, no. 6, pp. 871-879, 1988. (3) S. Fumi, "Cepstrum analysis technique for automatic speaker verification," IEEE Trans Acoustic, Speech, and Signal Processing, vol. 29, no. 1, pp. 254-272, 1981. [4] A.E. Rosenberg, J. DeLong, C.H. Lee,. B.H. Juang, and F.K. Soong,
'The use of cohort normalized scores for speaker verification," Proc.
ICSLP, vol. I, pp. 599-602, Banff, Canada, Oct. 1992. T. Matsui and S. Furui, "Similarity normalization method for speaker verification based on a posteriori probability," Proc. ESCA Workshop on Automatic Speaker Recognition Identification Verfication, pp. 5962, Martigny, Switzerland, Apr. 1994. (6) R. Brunelli, D. Falavigna, T. Poggio, and L Stringa, "A recognition system, particularly for recognizing people." patent no. 93112738,
(5)
1993.Priority IT/i 1.08.92/IT T0920695.
(7) R. Brunelli, T. Poggio, D. Falavigna, and L Stringa. "Automatic person recognition by using acoustic and geometric features," Tech. Report 9307-43, I.R.S.T., 1993, to appear in Machine Vision and Applications.
(8) G.R. Doddington, "Speaker recognition, identifying people by their voices," Proc. IEEE, vol.73, no. I1, 1985.
19]
A.E. Rosenberg and F.K. Soong. "Evaluation of a vector quantization Talker recognition system in text independent and text dependent modes," Computer Speech and Language, vol. 2, no. 3-4. pp. 143-157. 1987.
(101 H. Gish J. Makhoul, S. Roucos, "Vector quantization in speech coding,"

Proc. IEEE, vol. 73, no. 1), pp. 1551-1588. 1985.
9t 4
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL 17, NO. 10, OCTOBER 1995
Roberto Brunelli meceivesl his de~ree in nhvdca., [(11)D. O'Shaughnessy, Speech communication. Addison-Wesley, 1987. --.-.-.Brune... .. r--... d ... p.. --, with honors, from the University of Trento in 1986. (121 P. Melmerstein and S.B. Davis, "Comparison of parametric repesenta He joined IRST in 1987 where he works in the tions for monosyllabic word recognition in continuosly spoken sonComputer Vision Group. In the past, he was intences," IEEE Trans Acoustic, Speech, and Signal Processing, vol 28, volved in research on computer vision tools, analysis no. 4, pp. 357-366, 1980. of aerial images, development of algorithms working (13) G. Carli and R. Gretter "Astart-cnd point detection algorithm for a realon compressed description of binary images, neural time acoustic front-end based on dsp32c vine board," Proc. ICSPAT, networks, face recognition, and optimization. His pages 1,011-1,017, Boston, Nov. 1992. current major involvement is in the MINDEX Proj. at the' de~ulo mPnt ofinnovatiu tools for e,'c aim [14] R. Brunelli and T. Poggio, '"Pace Recognition: Features versus Temcontent based image retrieval in large databases. His plates," IEEE Trans. Pattern Analysis and Machine Intelligence, current interests include optimization, robust statistics, object recognition, vol. 15, no. 10, pp. 1,042-1,052. 1993. [15) R. Brunelli, "Estimation of pose and illuminant direction for face proc- and machine learning. essing," A.L Memo No. 1499, Massachusetts Inst. of Technology, 1994. Daniele Falavigna received his degree in electronic P.J. Burt, "Smart sensing within a pyramid vision machine," Proc. engineering from the University of Padova in 1985. IEEE, vol. 76, no. 8, pp. 1,006-1,015, 1988. He joined IRST in 1988 wher he works in the [(16) P. W. Hallinan, "Recognizing human eyes," SPIE Proc., vol. 1570, Speech Recognition Group. In the past, his'research pp. 214-226, 1991. hasincluded acoustic modeling of speech, automatic (17) L Stringa. "Eyes detection for face recognition," Applied Artificial segmentation and labeling, and speaker identificaIntelligence, vol. 7, pp. 365-382, 1993. tion. His current major involvement is in the devel(18] D.H. Ballard and C.M. Brown, Computer Vision. Englewood Cliffs, opment of speech recognition systems. His current interests include speech analysis, speech recogniN.J.: Prentice Hall, 1982. tion, speaker identification and verification, and (191 R. Brunelli and S. Messelodi, "Robust estimation of correlation: With " ^i"^^a sign processing.1Z application to computer vision," Pattern Recognition, vol. 28, no. 6, pp. 833-861, 1995. (20) L Stringa, "Automatic face recognition using directional derivatives," Tech. Report 9205-04, i.R.S.T., 1991. (21] L Xu, A. Krzyzak, and C.Y. Suen, "Methods of combining multiple classifiers and their applications to handwriting recognition," IEEE Trans. Systems, Man, and Cybernetics, vol. 22, no. 3, pp. 418-435, 1992. (22) FR. Hampel, PJ. Rousseeuw, E.M. Ranchetti, and W.A. Stahel, Robust Statistics:.The Approach Bared a Influence Fwunctio. John Wiley & Sons, 1986. [23) P.J. Huber, Robust Statistics. Wiley, 1981. (24) P.B. Bonissone and K.S. Decker, "Selecting uncertainty calculi and granularity: An experiment in trading off precision and complexity," J.F. Lemmer, LN. Karnak, eds., Uncertainty in Artificial Intelligence, pp. 217-247, North Holland, 1986. (25) P.B. Bonissone, S.S. Gans, and K.S. Decker, "Rum: A layered architec. ture for reasoning with uncertainty," Proc: 10th Int'l Joint Conf on Artificial Intelligence, pp. 891-898, Milan, Aug. 1987. (26) R.V. Hogg and A.T. Craig, Intro. to Statistics. Collier-Macmillan, 1978. [27) R.O. Duda and P.E. Hart, Pattern Recognition and Scene Analysis, New York: Wiley, 1973. (28) Y.-H. Pao, Adaptive Pattern Recognition and Neural Networks. Reading, Mass: Addison-Wesley, 1989. (29) R. Brunelli. On training neural nets through stochastic minimization, Neural Networks, vol. 7, no, 9, pp. 1405-1412, 1996. [30) R. Brunelli and G. Tecchiolli. Stochastic minimization with adaptive memory, J. of Computational andApplied Mathematics, pp. 329-343, 1995. Academic Press, 1990. gnitionrL [31) K Pukma Inoduction to salistical pareon [32) T. Poggio and P. Girosi, Regularization algorithms for learning that are equivalent to multilayer networks, Science, voL 247, pp. 978-982,1990.
Machine Vision and Applications (1995) 8:317-325
Machine Vision and Applications

o Springer-Verlag 1995
Automatic person recognition by acoustic and geometric features*

R. Brunelli, D. Falavigna, T. Poggio**, L. Stringa
Institute for Scientific and Technological Research, 1-38050 Povo, Trento, Italy
Abstract. This paper describes a multisensorial person-identification system in which visual and acoustic cues are used jointly for person identification. A simple approach, based on the fusion of the lists of scores produced independently by a speaker-recognition system and a face-recognition system, is presented. Experiments are reported that show that the integration of visual and acoustic information enhances both the performance and the reliability of the separate systems. Finally, two network architectures, based on radial basis-function theory, are proposed to describe integration at various levels of abstraction.
Key words: Face recognition - Speaker identificationi
-
and width of the nose, chin shape, and so on. We have considered various ways of combining the SRS and the FRS
results:
1. At the level of the outputs of the single classifiers: the list of scores (probability estimates or distances) produced independently by the two systems can be used in an
integrated classifier system (i.e., the two sets of scores
Classification
are optimally weighted and summed to produce a unique final list). 2. At the level of the measurements carried out on both the speech and visual signals: acoustic and geometric parameters are considered as a unique vector, lying in the cartesian product of the acoustic and visual spaces,. which will be successively classified by a speaker specific classifier. Examples of such a classifier are: a bayesian classifier, a multilayer perceptron (MLP) classifier or a radial basis function (RBF) classifier. The experiments reported in this paper are based on the first integration strategy. Some HyperBF network architectures supporting the second integration strategy are also described. The paper is organized as follows: - Section 2 describes the overall system architecture. - Section 3 presents the acoustic and visual databases. - Section 4 analyzes the SRS and FRS used for this work. - Section 5 describes the method used in this. work to integrate the two systems. An alternative strategy, currently under investigation, is also presented. - Section 6 reports the results of the experiment(s. 2 System overview The basic structure of the identification system is depicted in Fig. 1. An attention module detects changes in the image captured by a CCD. camera by using background subtraction and thresholding. Whenever a significant change in the scene is detected, a snapping module is activated. This module waits
until a stable scene is detected (a still person staring into
1 Introduction This paper describes an automatic person-recognition system that uses both acoustic features, derived from the analysis of a given speech signal, and visual features, related to distinctive parameters of the face of the person who spoke. Visual and acoustic cues are used jointly for person identification: several methods of combining the acoustic and visual information at various levels are described. Experiments are presented that show the superior performance of the whole system with respect to both the speaker recognition system (SRS) and the face recognition system (FRS). The system can be used for either identification (e.g., as an electronic concierge to recognize people in small organizations) or verification applications (e.g., as a smart door key to control the entrance of a house) even if, in our experiments, we have stressed only the identification function. The SRS uses acoustic parameters computed from the spectrum of short-time windows of the speech signal, while the FRS is based on geometric data represented by a vector. describing discriminant facial features such as the position
* Italian Patent No. T092A000695. European extension in progress. "Anificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge. MA 02139, USA R. Correspondence to: Bnrunei
the camera) and checks for the satisfaction -of simple constraints (mainly on the changed area) before unlocking the separate action of the two recognition systems. The snapped
the DAT (sampled at 48 kHz) was downsampled to 16 dkHz, manually endpointed, and stored on a computer disk. 3.2 The visual database The database is composed of 188 images, 4 per person. Of taken in the same the 4 pictures available, the first 2 session (at an interval of a few minutes), while the other pictures were taken at intervals of 2-4 weeks. The pictures were taken with a CCD camera at a resolution of 512 x 512 pixels as frontal views. The subjects were asked to look into the camera, but no particular efforts were made to ensure perfectly frontal images. .The illumination was partially controlled. The same powerful light was used, but the environment in which the pictures were taken was exposed to sunlight.,through windows. The pictures were taken randomly during daytime/The distance of the subject from the camera was fixed only approximately, so that scale variations of as much as 30% were possible.
'were
Fig. 1. The flow of activation of the various modules belonging to the multisensorial identification system
4 The recognition systems
image is fed directly to the face-recognition system, while the system asks the person to utter a sequence of digits in a continuous way. A boundary detection module, applied to the input speech signal, separates the voice signal from the background noise, determining both the starting and ending times of each voice segment and the corresponding duration. If the total duration of the voice segments detected by the start-end point-detection module is not long enough, the system asks the user to read the requested digits again. The outputs of the SRS and FRS are fed into an integration module; we discuss its architecture in the following sections. The output of the integration module is sent to the last module that identifies the user. 3,Test database description The acoustic and the visual databases consist of 42 and 47 people, respectively; the database used in the experiments is composed of 33 people, without shaves and moustache, belonging to both databases. The FRS and SRS were trained separatelyfor each person in the two databases while testing (see Sect. 5) was done on the common 33 people database. 3.1 The acoustic database The database consists of 8400 isolated Italian digits, from zero to nine. Each speaker uttered 200 digits in five recording sessions: during each session the speaker provided four repetitions of each digit. The entire database was collected over a period of about two months. Each recording session is separated from the previous one by an interval varying from 3 to 10 days. The Sessions were recorded in an acoustically isolated room of our Institute by means of digital audio tape (DAT) equipment (SONY TCD-DIO). The digital signal on
In this section the SRS and the FRS used in the experiments are described. Both systems operate in two steps: 1. A sequence of parameter vectors or a single parameter vector is derived from the speech signal or the visual signal, respectively. 2. The input parameter vectors are matched on the basis of distance measures with respect to reference models, so that each system produces a list of scores labeled with the identifier of the corresponding reference model. 4.1 The speaker-recognition system Automatic speaker-recognition is a topic that has been widely investigated in the past years (Atal 1976; Doddington 1985; Furui 1981). Recently, some methods based on the processing and modeling techniques also used for speech recognition have been studied and tested (Rosenberg 1987; , Tishby 1991). Speaker-recognition systems can be divided, according to the application area, into speaker-verification or speakeridentification systems. A speaker-verification system simply checks (confirms or not) the identity claimed by a person (e.g., before she/he accesses a reserved place or service). A speaker-identification system must determine who (within a known group of people) gave the (speech) signal. Speaker-recognition systems can also be text dependent (i.e., the speaker must utter a predefined key sentence) or text independent. The major problem in speaker-recognition is represented by intersession variability (variability of the speech signal with time) due both to different recording or transmission conditions and to intrinsic variability in people's voices. The SRS used for this work is based on vector quantization (VQ) and is similar to the one described in Rosenberg (1987). A block diagram of this system is depicted in Fig. 2.
The system measures the distances between two distinct sets of acoustic parameters (static and dynamic) and corresponding prototypes or codebooks, derived, during a training phase, from speaker-specific speech material. 4.1.1 Speech signal analysis The speech signal, sampled at 16 kHz and quantized over 16 bits, is first pre-emphasized by using a digital filter having a transfer function H(z) = 1 - 0.95 x z- 1 . We analyze the pre-emphasized signal every 10 ms, using a 20 ms Hamming window and, for each window position, we compute the following parameters: 1. Eight Mel scale cepstral coefficients (Davis 1980) computed from the log-energy outputs of a 24 triangular band-pass filter bank applied to the power spectrum of the given window. These parameters are said to be static. 2. The corresponding first-order time derivatives, evaluated by means of a linear fit over nine frames of static parameters, centered on the given window. These parameiers are said to be dynamic. 4.1.2 The VQ-based speaker-recognition system As seen previously, the reference models of each speaker (Fig. 2) consist of two codebooks: one corresponding to the static parameters and the other to the dynamic ones. The codebooks were generated by applying the Linde-Buzo-Gray algorithm (Makhoul 1985) to a set of shonrt-time spectral vectors (static or dynamic), derived from 100 digits belonging to the first 2.5 recording sessions of each speaker. This algorithm searches for prototype vectors that minimize a global distortion measure defined on the given training set. A weighted euclidean distance was used for the codebook design and recognition. The weights correspond to the inverse of the pooled variances of the components of the training vectors averaged over all training utterances and speakers. Therefore, if 0i, Oi are two parameter vectors, their distance is defined as:
kkI
2
set, and summed. The performance of the system depends on both the acoustic resolution (i.e., the number M of vectors in each speaker codebook) and the number L of digits contained in' the input signal. The average identification error evaluated on a test set composed of 100 digits per speaker (belonging to the last 2.5 recording sessions) is 49% for M = 4, L = 1 and 0% for M = 64, L = 3. In Fig. 3, the results for spectral resolutions varying from 4 to 64 and utterance lengths varying from I to 10 digits are reported. 4.2 The face-recognition system Every day, humans recognize people by visual cues without apparent effort. The ease with which we recognize familiar people by their faces has led us to underestimate the difficulty of the problem. Extended psychophysical experiments have revealed the fact that, even for people, recognition requires a lot of processing and is by no means an innate ability. There are two main strategies for automatic face recognition, both of which can be said to mimic some of the processes used by people: . 1. The iconic strategy is based on the comparison of suitably preprocessed image patches. Recognition is effected by comparing (e.g., through the value of crosscorrelation or some other suitable distance measure) an unknown image with stored templates of distinctive facial regions (Baron 1981; Brunelli and Poggio 1993; Stringa 1992b, 1992; Turk and Pentland 1991). 2. The geometric strategy computes a set of geometric features that describe the size and the layout of the various features of the faces, and recognition proceeds by comparing the unknown descriptive vector with a set of reference vectors (known people) stored in a database (Bichsel 1991; Brunelli and Poggio 1992, 1993; Cottrell and Fleming 1990; Craw et al. 1987; Kanade 1973; Nakamura et al. 1991). Several approaches can be classified within this simple taxonomy, and a comparison of the two basic strategies can be found in Brunelli and Poggio (1993). In this paper we focus on the geometric strategy. The reason is twofold: 1. It gives a more compact representation and guarantees a high speed in the recognition process. 2. It gives an example of how good performance results can be obtained by integrating simple (and fast) recognition modules that do not have a very high performance when considered separately. As will be apparent from the discussion of the integration strategies, the use of a template-matching strategy fits in a natural way. Template matching typically compares suitably preprocessed images (or patches of images) representing the salient features of the face pattern. Whether the comparison is done by a suitable distance measure.or by a correlation coefficient, the result can readily be incorporated into the proposed system, alternatively or in conjunction with the information explicitly used. A set of geometrical features describing a frontal view of a face is computed automatically in the following steps (Brunelli and Poggio 1993):
(1)
where ak represents the average pooled variance of the kth component of the parameter vector. As previously seen, p is equal to 8. During the recognition phase, the distances (Fig. 2) between the sequences of static and dynamic vectors obtained from the input speech signal and the corresponding codebooks of each speaker are evaluated. Therefore, if
9 = 01, .... ,OT is the static (or dynamic) input sequence, and T, = i,;..., iyj,, are the vectors of the jth static (or
dynamic) codebook, then the total static (or dynamic) distortion is defined as: D(9, lP)) = ,
t=
TM
min d(s, ,j4)
(2)
The static and dynamic distances are then normalized with respect to their average values, computed on the training
320
Codebooks for Speaker N.1
Codebooks for Speaker N. K
% identification error 50.00
Spectra Res.. 64
45.00
40.00 35.00 30.00 25.00
Spectral Res.: 32 SpcrlRes.. 16 * -'' " Spectral Res.. 8

Spctra Res.: 4
' -
20.00
p.
15.00 10.00 5.00

I"
14
I.
0.00
0.00 0.00
5.00 . 5.00
l0.0( 10.0(,
nr fdi to
Bi
Fig. 2. Block diagram of the vector-quantization-based speaker-recognition system Fig. 3. Error rate of the speaker-recognition system (SRS), evaluated on the whole 42-speaker test set as a function of the number of digits in the input signal: various curves correspond to various spectral resolution values Fig. 4. Geometrical features (black) used in the facerecognition experiments (information on the eyebrow arch is
not eported
321
1. Eyes are located so that the image can be normalized both in size and rotation in the image plane (Stringa 1993). 2. An average face model is used to focus the system progressively, in a sequential way, on the various regions of the face so that relevant feature points can be computed. 3. A descriptive vector is built from the relative positions of the feature points. This results in the computation of 35 geometrical features (Fig. 4) that can be used for recognition: 1. Eyebrow thickness and vertical position relative to the center of the eye 2. A coarse description of the left eyebrow arch (8 measurements) 3. Vertical position and width of the nose 4. Vertical position, width (upper and lower lips) and height of the mouth 5. Eleven radii describing the chin shape 6. Bigonial breadth (face width at nose position) 7. Zygomatic breadth (face width halfway between nose tip and eyes) A detailed description of the algorithms employed can be found in Brunelli and Poggio (1993). Classification can then be based on a Bayes classifier. We make the simplifying assumption that the measurements of the various features share the same gaussian distribution for all the people apart from their average value (Bichsel 1991). The covariance matrix E can then be estimated and classification can be based (Duda and Hart 1973) on the following distance, which is related to the probability of the given measurement:
d(x, m) = (x - mj)T E
- t
5.1 Score integration Both classifiers are essentially nearest-neighbor with a suitably defined metric. As already mentioned, when given an. unknown input, they generate a list of possible labels, marked with their distance from the input vector. The list is sorted in order of increasing distance so that the first element of the list should be taken as the correct matching label. The main difficulty in combining the results of the two classifiers is given by the inhomogeneous distances they produce. The simplest way to fix this is by inverse standard deviation normalization. Given the two lists, if d,,, and d, represent the distances computed by the face recognizer and the speaker recognizer, respectively for the ith reference, and .a and o9 are the corresponding variances, a combined distance can be defined as
D.=dv,
Ov
d,
0
A natural way to look at the answer of a nearest-neighbor classifier is to map it into a list of scores rather than a list of distances. A possible mapping is:
-si
d2 .
where z stands alternatively for Vision or Speech. This mapping associates a value, in the.interval (0, 1]with a distance. In some sense, the higher the score the more likely that the correspondence is correct. Each list could be normalized further by imposing the following condition: zsz
1
(x - mj)
(3)
The resulting lists could be given a Bayes interpretation, suggesting the following integration strategy in the hypothesis that the two systems are independent:
Si = Sv, X Sa
The identification of an unknown vector is taken to be that of the nearest vector in. the database of known people. The database used for the reported experiments consists of 132 vectors, 4 per person, representing the complete set of 35 geometrical features. The performance is estimated on a round-robin recognition experiment in which three of the available vectors are used to compute an average representative vector, while the remaining one is used for testing. The results for this database were found to be 92% correct.
As the two recognition systems do not have the same performance, it is natural to introduce a weighted merged score:
(
(w)
The optimal weight w can be found by maximizing the performance of the integrated system on one of the available test sets. 5.2 A more integrated system
5 Integration of vision and voice In this section, we briefly describe methods for integrating the two classifiers into a more robust recognition system. There are two general classes of possible architectures. The first is based on the integration of the output of two independent classifiers, one for the voice and one for the face. The second one is based on the direct combination of voice and face features. The first approach is simpler and is used in the reported integration experiments. The second method is more speculative and is suitable for a larger-scale application.
A closer integration of the two recognition systems could be ,made with a network that learns the relative reliability and discriminating power of the various features. We propose two such architectures based on HyperBF Networks. Before introducing the architectures, let us briefly recall the basic theory of these networks in the general framework of solving learning problems through multivariate function reconstruction (Poggio and Girosi 1990). Whenever the examples in a learning task are given in numeric form, learning can be seen as a problem of surface reconstruction from sparse data (the examples). It is fairly evident that, as it is, the task of reconstructing a surface
322
given its value at some points is ill posed. To make the problem well behaved some additional constraints must be imposed. The most important constraint, as far as learning is concerned, is smoothness. If the reconstructed function is required to be smooth, we are assured that generalization from available examples is indeed possible. The assumption of smoothness allows us to formulate a variational problem whose solution is the surface we are looking for:
N
H[f)=
i=1
(y,
f(xi))2 + A0[fJ
where the first term measures the distance between the data and the desired solution f; the second term is a functional reflecting the cost associated with the deviation from some constraint.reflecting some a priori knowledge. The unknown function can be considered as a linear combination of the following form: f(x)=
I
cjcG(Ix - tlIw)
(10)
where tj are called expansion centers and W is a square matrix used to compute the norm of a vector (x-t.)TWTW(x tj). Function G could be, among others, a gaussian or a multiquadric function [for a more complete introduction to the theory of HyperBF Network see Poggio and Girosi (1990)]. The variational problem can be solved by finding the coefficients, the centers and the metric. This can be done by minimizing the reconstruction error on the available examples and the smoothness enforcing term. The network architectures we propose for the integrated recognition system are shown in Fig. 5. The underlying idea is to build a set of HyperBF modules, and 'train each module to recoristruct the characteristic function of a given person to be identified. The module is supposed to output 1 on inputs corresponding to the given person and 0 otherwise. Each module is trained with both positive and negative examples (competitive learning)so that both optimal example selection and adaptive metric can be used profitably. A typical choice for the basis function in this type of network would be the gaussian function. The first network (Fig. 5b) is closer, to the simple approach outlined. The modules in the first layer represent the outputs of recognizers based on various types of data (such as visual, static, or dynamic acoustic information). The modules can be trained separately, taking advantage of the reduced input dimensionality. The integration is operated by the module in the second layer that is trained, again on a per person basis, to maximize its output on the vectors of the corresponding person. We call the resulting network a score-integration network. The approach detailed in the current paper can be seen as a simplified version of this type of network in which the first layer is removed. The second network architecture (Fig. 5c) that we propose is a feature-integration network. The input to the HyperBF modules is taken to be the cartesian product of the various inputs, and the module takes advantage of the simultaneous knowledge of the available information. This network requires a more difficult training due to the extended input dimensionality.
C Fig. 5. a The basic HyperBF'module; b a score-level integration network; c a feature-level integration network
6 Experiments and results
The strategy we choose to integrate the two recognition systems is that of score fusion. The distances given, by each of the systems have been transformed into score lists using exponential mapping and inverse variance normalization, as explained in Sect. 5.1. As we said previously, the database used for the integration experiments consists of 33 people. For each face 4 lists of scores were available, while for each speaker and for each utterance length, varying from I to 6 digits, we randomly selected-8 lists of scores from those corresponding to codebook sizes equal ,to 4. In this way, a total of 32 integration experiments could be performed for each person and for each input utterance length (from I to 6 digits). One-of these 32 sets of score lists was used to estimate the optimal integration weight (Fig. 6) while performance was measured on the remaining sets. Useful data on the robustness of the classification are given by an estimate of the intraclass variability as opposed to the inter-class variability. We can do this using the so-called min/max ratio (Poggio and Edelman 1990; Poggio and Girosi 1989), which is defined as the minimum distance on a wrong correspondence over the maximum distance from the 'correct correspondence. To estimate the optimal weighting factor, the interval (0, 1], representing the weight of the FRS, was evenly sampled and the value maximizing the performance was chosen (ties were resolved
.correct identification
1 M
correct identification
I - I
~
0.00 0.50
I
1.00 FRS weight
ar. of digits
Min/Max ratio
A
3.80
3.60 3.40 310

14 c
3.00 2.80 2.60 i 2.40 2.20 2.00 1.80 1.60 1.40 1.20 Fig. 6. Correct performance of the integrated system as a function of the weight assigned to the face-recognition system (FRS) using a score-level integration method; various curves correspondto various numbers of digits in the input speech signal. The spectral resolution of the speaker-recognition system (SRS) is 4 Fig. 7. Performance of the face-recognition system (FRS) (horizontal line) and of both the speaker-recognition sytem (SRS) and the integrated system versus the number of digits in the input speech signal; the spectral resolution of the SRS is 4; the vertical bars represent the standard deviation of the quantity measured Fig. 8. Mia/Max ratio of the face-recognition, speaker-recognition, and integrated systems (these last two are functions of the number of digits in the input speech signal); the spectral resolution of the SRS is 4; the vertical bars represent the standard deviation of the quantity measured
T
_l-I
-c
0 2.00
.0 4.00
16.00 nr. of digits
by maximizing the min/max ratio). The performance of the SRS, FRS, and integrated systems are quantified by: I. The average performance (Fig.7) 2. The min/max ratio (Fig. 8) The analysis of this integrated performance (Fig. 7) shows that, even with the simple integration scheme, nearly perfect recognition is achieved. The FRS gave 92% correct results while the SRS, using different speaker models, gave results varying from 51% to 100% correct. The results of the integrated system, using the less complex speaker model are already up to 95% correct, and 100%o is achieved with a speaker model of low complexity. Further benefits of integration are evident in Fig. 8, where the min/max ratio, which represents an estimate of the average separation among classes, is plotted. The increased class separation of the integrated system, as measured by the mi/max ratio, suggests that rejection can be introduced with a more limited impact on performance than on either of the two separate systems. The curves in Fig. 6 represent the performance of the integrated system as a function of the weight used to merge the scores. As can be seen, the performance is a smooth function of the weight. This means.that the the system is not very sensitive to the weighting factor. As could easily be anticipated from the performance of the two systems, the optimal combining weight shifts from a vision dominance at a low number of digits toward a voice dominance at an high number of digits. The same result can be reached by passing from lower to higher spectral resolutions of the SRS. 7 Conclusions' In this paper, the superior performance that can be attained by using multisensorial input has been demonstrated. Integration of two recognizing systems, based on speech and vision, respectively, greatly improves the performance, reaching the state of the art with a low-to-moderate complexity in the constituent systems. Finally, some speculative network architectures for integration have been proposed for use on larger databases that currently require computationally expensive recognition systems. A system based on the integration strategy described and working in an uncontrolled environment is currently under evaluation. Preliminary resuits suggest that the use of multiple identification cues has a major impact, not only on absolute performance, but also on the ability of the system to reject an unknown user.
Acknowledgements. This research was done within MAIA, the integrated Artificial Intelligence project under development at IRST (Svinga, 01991a: Poggio and Stringa, 1992). We thank Dr. S. Messelodi for kindly sharing the C language implementation of the attention and snapping module.
References 1. Ata B (1976) Automatic recognition of speakers from their voices. Proc IEEE 64:460-475 2. Baron RJ (1981) Mechanisms of human facial recognition, Int JMan Machine Stud 15:137-178 3. Bichsel M (1991) Strategies of robust object recognition for the identification of human faces. Phd Thesis, The Swiss Federal Institute of Technology, Zilrich 4. Brunelli R, Poggio T (1992) Face recognition through geometrical features. In: Sandini G (ed) ECCV'92, Santa Margherita Ligure. Springer, Berlin Heidelberg New York, pp 792-800 5. Brunelli R,.Poggio T (1993) Face recognition: features versus templates. IEEE trans Pantt Anal Machine Intell 15:1042-1052 6. Cottrell G,Fleming M (1990) Face recognition using unsupervised feature extraction Proceedings of the International Neural Network Conference, Paris, Kluwer, July, pp 322-335 7. Craw 1, Ellis H, Lishman JR (1987) Automatic extractoin of face features. Part Recogn Lett 5:183-187 8. Davis SB, Melmerstein P (1980) Comparison of parametric representation for monosyllabic word recognition in continuosly spoken sentences IEEE Trans Acoustic Speech Signal Processing -28:357-366 9. Doddington GR (1985). Speaker recognition, identifying people by their voices. Proc IEEE 73 10. Duda RO, Hart PE (1973) Pattern recognition and scene analysis. Wiley, New York 11. Furul S (1981) Cepstrum analysis technique for automatic speaker verification. IEEE Trans Acoustic Speech Signal Processing 29:254-272 12. Kanade T (1973) Picture processing by comoputer complex and recognition of human faces. Technical Report, Department of Information Science, Kyoto University, Kyoto, Japan 13. Makhoul J, Gish H, Roucos S (1985) Vector quantization in speech coding. Proc IEEE 73:1551-1588 14. Nakamura O, Mathur S, Minami T (1991).Identification of human faces based on isodensity maps. Part Recogn 24:263-272 15. Poggio T, Edelman S (1990).A network that learns to recognize threedimensional objects. Nature 343:1-3 16. Poggio T, Girosi F (1989) A theory of networks for approximation . and learning. Artificial Intelligence Lab A.I. Memo No. 1140, Massachusetts Institute of Technology, Boston, Mass 17. Poggio T, Girosi F (1990) Networks for approximation and learning. Proc IEEE 78:1481-1497 18. Poggio T, Stringa L (1992) A project for an intelligent system: vision and learning. nt J Quantum Chem 42:727-739 19. Rosenberg AE, Soong FK (1987) Evaluation of a vector quantization talker recognition system in text independent and text dependent modes. Comput Speech Language 2:143-157 20. Stringa L (1991a) An integrated approach to artificial intelligence: the MAIA Project. bTchnical Report 9110-26, Institute for Scientific and Technological Research, Trento, Italy 21. Stringa L (1991b) Automatic Face Recognition using Directinal Derivatives. Technical Report 9205-04, I.R.S.T. Institute for Scientific and Technological Research. Trento, Italy 22. Stringa L (1992) S-net implementation of a face recognizer based on directional derivatives. In: Caianiello ER (ed) Proceedings of the 5th Italian Workshop on Neural Nets, Vietri, World Scientific, Singapore, pp 329-333. 23. Stringa L (1993) Eyes detection nfor face recognition. Appl Artif Intell 7:365-382 24. Tishby NZ (1991) On the application of mixture AR hidden 'markov models to text independent speaker recognition. IEEE Trans Signal Processing 39:563-570 25. Turk M, Pentland A (1991) Eigenfaces for recognition. J Cognitive Neurosci 3:71-86
325 Lulgi Stringa was born in Genoa, Italy, in 1939. He graduated in Physics from the University of Genoa. His main interests are computer architectures; pattern recognition and artificial intelligence. He taught these subjects at the Universities of Genoa, Rome, and Trento, Italy. He has designed and developed innovative industrial systems, such as multiprocessor-based Mail Address Readers. Joining SELENIA in ELSAG in 1969, he became Vice Director General in 1979. In 1980 he became Director General of SELENIA, and he was appointed Managing Director of the SELENIA-ELSAG Group in 1983. He left SELENIA in 1985 to become Director of the lnstitute for Scientific and Technological Research (IRST) in Trento.
Roberto Bnmelli was born in Trento, Italy, in 1961 ..Herceived his degree in Physics cum laude from the University of Trenrnto in 1986. He joined the Institute for Scientific and Technological Research (IRST) in 1987 as researcher in computer vision. His interests include object-recognition and learning. Danlele Falavigna was born in Verona, Italy. in 1960. He received his degree in Electronic Engeneering from the University of Padua in 1985. In 1988 he joined IRST, where he is currently engaged as Senior Researcher of.the Acoustic Processing and Speech Recognition Group. His currnenl research interests are signal processing and acoustic modeling for speech and speaker recognition. Tomaso Poggio was born in Genoa. Italy, in 1947. He received his PhD in Theoretical Physics from the University of Genoa in 1970. From 1971 to 1982, he was a Scientific Assistant at the Max Planck Institute for Biological Kybernetics in Tuebingen, Germany. Since 1982, he has been a Professor at the Massachusetts Institute of Technology, affiliated with the Department of Brain and Cognitive Sciences and the Artificial Intelligence Laboratory. In 1988 he was named to the Uncas and Helen Whitaker Professorship, and in 1992 he became a founding co-director of the Center for Biological and Computational Learning. He is a Fellow of the American Association of Artificial Intelligence, and a Corporate Fellow of Thinking Machines Corporation.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 20, NO. 12, DECEMBER 1998.
Application No: 11/231,353 Docket No. 577832000200
Integrating Faces and Fingerprints for Personal Identification

Lin Hong and Anil Jain, Fellow, IEEE
Abstract-An automatic personal identification system based solely on fingerprints or faces is often not able to meet the system performance requirements. Face recognition is fast but not extremely reliable, while fingerprint verification is reliable but inefficient in database retrieval. We have developed a prototype biometric system which integrates faces and fingerprints. The system overcomes the limitations of face recognition systems as well as fingerprint verification systems. The integrated prototype system operates in the identification mode with an admissible response time. The identity established by the.system is more reliable than the Identity established by a face recognition system. In addition, the proposed decision fusion scheme enables performance improvement by integrating multiple cues with different confidence measures. Experimental results demonstrate that our system performs very well. It meets the response time as well as the accuracy requirements. Index Terms-Biometrics, fingerprint matching, minutiae, face recognition, eigenface, decision fusion. --+----------
INTRODUCTION
3) permanence, which means that the characteristic, should not be changeable; and 4) collectability, which indicates that the characteristic can be measured quantitatively. However, in practice, a biometric characteristic that satisfies all the above requirements may not always be feasible for a practical biometric system. In a practical biometric system, there are a number of other issues which should be consid1171: ered, including [6), 1) performance, which refers to the achievable identification accuracy, speed, robustness, the resource requirements to achieve the desired identification accuracy and speed, as well as operational or environmental factors that affect the identification accuracy and speed; 2) acceptability,which indicates the extent to which people are willing to accept a particular biometrics in their daily life: and 3) circumvention, which reflects how easy it is to fool the system by fraudulent methods. A practical biometric system should be able to: 1) achieve an' acceptable identification accuracy, and speed with a reasonable resource requirements; . .2) not be harmful to the subjects and be accepted by the intended population; and 3) be sufficiently robust to various fraudulent methods. Currently, there are mainly nine different biometric techniques that are either widely used or under investigation, including [271: * * " * * * face, facial thermogram, fingerprint, hand geometry, hand vein, iris,
WITH
sothe evolution of information technology, our ciety is becoming more and more electronically connected. Daily transactions between individuals or between individuals and various organizations are conducted increasingly through highly interconnected electronic devices. The capability of automatically establishing the identity of individuals is thus essential to the reliability of these transactions. Traditional personal identification approaches which use "something that you know," such as a Personal Identification Number (PIN). or "something that you have," such as an ID card are not sufficiently reliable to satisfy the security requirements of electronic transactions because they lack the capability to differentiate between a genuine individual and an impostor who fraudulently acquires the access privilege 117]. Biometrics, which refers to identification of an individual based on her physiological or behavioral characteristics, relies on "something which you are or you do" to "ake a personal identification and, therefore, inherently has the capability to differentiate between a genuine individual and a fraudulent impostor [17], 1271. Any human physiological or behavioral characteristic can be used as a biometric characteristic(indicator) to make a personal identification as long as it satisfies the following requirements [6], [17]: 1)universality, which means that each person should have the characteristic; 2) uniqueness, which indicates that no two persons should be the.same in terms of the characteristic;
SThe authors are with the Department ofComputer Science and Engineer-
ing, Michigan State University. East Lansing.MI 48824-1226. E-mail: (honglin,jain@cse.msu.edu. Manuscript received 6 Oct. 1997: revised 8 Sept. 1998. Recommended for acceptance by V. Nalwa. For Information on obtaining reprints ofthis article, please send e-mail to:
tpami@computer.org, and reference IEEECS Log Number 107416.
0162-8828/98/510.000 1998 IEEE
1296
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 20, NO. 12, DECEMBER 1998
(b)
(c)
(d)
(e)
Fig. 1. Examples of biometric characteristics (indicators). (a) Face.. (b) Facial thermogram. (c) Fingerprint. (d) Hand vein. (e) Retinal scan.
Enrollment
'
K'
t 1 1
II
. I
Fig. 2. A generic biometric system architecture.
* retinal pattern, * signature, and * voice-print (some of the examples are shown in Fig. 1) 161, 171, 181, (161, (171. All these biometric techniques have their own advantages and disadvantages and are admissible depending on the application domain. A generic biometric system architecture is depicted in Fig. 2. Logically, it can be divided into two modules: 1) enrollment module and 2) identification module. During the enrollment phase, the biometric'characteristic of an individual is first scanned by the biometric reader to produce a digital representation of the characteristic. In order to facilitate the matching and identification, the digital representation is usually further processed by a feature extractor to generate a compact but expressive representation, called a template. Depending on the application, the template may be stored in the central database of the biometric system or be recorded in the smart card or magnetic card issued to the individual. The identification module is responsible for identifying individuals at the point-ofaccess. In the operational phase, the biometric reader captures the characteristic of the individual to be identified and, converts it to a raw digital format, which is further processed by the feature extractor to produce a compact repre-
sentation that is of the same format as the template. The resulting representation is fed to the feature matcher which compares it against the template(s) to' establish the identity. 1.1 Operational Mode A biometric system may operate in 1) the verification mode or 2) the identification mode 117]. A biometric system operating in the verification mode authenticates an individual's identity by comparing the individual only with his/her own template(s) (Am I whom I claim I am?). It conducts one-to-one comparison'to determine whether the identity claimed by the individual is true or not. A biometric system operating in the identification mode recognizes an individual by searching the entire template database for a match (Who am I?). It conducts one-tomany comparisons to establish the identity of the individual. Generally, it is more difficult to design an identification system than to design a verification system (171. For a verification system, the major challenge is the system accuracy. It is usually not very difficult to meet the response time requirement, because only one-to-one comparison is conducted: On the other -hand, for an identification system, both the accuracy and speed are critical. An identification system needs to explore the entire template database to establish an identity. Thus, more requirements are irriposed' on the feature extractor and, especially, the feature matcher.
HONG AND JAIN: INTEGRATING FACES AND FINGERPRINTS FOR PERSONAL IDENTIFICATION
1297
Some biometric approaches are more suitable for operating in the identification mode than the others. For example, although significant progress has been made in fingerprint identification and a number of fingerprint classification and matching techniques have been proposed, it is still not practical to conduct a real-time search even on a relatively small-size fingerprint database (several thousand images) without dedicated hardware matchers, external alignment, and multiple-fingerprint indexing mechanism [21]. On the other hand, it is feasible to design a face-recognition system operating in the identification mode, because 1) face comparison is a relatively less expensive operation and 2) efficient indexing techniques are available and the performance is admissible [23]. 1.2 Identification Accuracy Due to intraclass variations in the blometric characteristics, the identity can be established only with certain confidence. A decision made by a biometric system is either a "genuine individual" type of decision or an "impostor" type of decision 171. 1171. For each type of decision, there are two possible outcomes, true or false. Therefore, there are a total of four possible outcomes: 1) a genuine individual is accepted, 2) a genuine individual is rejected, 3) an impostor is rejected, and 4) an impostor is accepted. Outcomes I and 3 are correct, whereas outcomes 2 and 4 are incorrect. The confidence associated with different decisions may be characterized by the genuine distribution and the impostor distribution, which are used to establish two error rates: 1) false acceptance rate (FAR), which is defined as the probability of an impostor being accepted as a genuine individual and 2) false reject rate (FRR), which is defined as the probability of a genuine individual being rejected as an impostor. FAR and FRR are dual of each other. A small FRR usually leads to a larger FAR, while a smaller FAR usually implies a larger FRR. Generally, the system performance requirement is specified in terms of FAR 1171. A FAR of zero means that no impostor is accepted as a genuine.individual. In order to build a biometric system that is able to operate efficiently in identification mode and achieve desirable accuracy, an integration scheme which combines two or more different biometric approaches may be necessary. For example, a biometric approach that is suitable for operating in the identification mode may be used to index the template database and a biometric approach that 'is reliable in deterring impostors may be used to ensure the accuracy. Each biometric approach provides a certain confidence about the identity being established. A decision fusion scheme which exploits all the Information at the' output of each approach can be used to-make a more reliable decision.
We introduce a prototype integrated biometric system which makes personal identification by integrating both faces and fingerprints. The prototype integrated biometric system shown in Fig. 3 operates in the identification mode. The proposed system integrates two different biometric approaches (face recognition and fingerprint verification) and incorporates a decision fusion module toimprove the identification performance. .In the following sections, we will describe each component of the proposed integrated system. Section 2 addresses the face-recognition technique being employed. Section 3 presents the fingerprint-verification module along with minutiae extraction and minutiae matching. A decision fusion framework which integrates faces and fingerprints is formulated in Section 4. Experimental results on the MSU fingerprint database captured with an online fingerprint scanner and public-domain face databases are described in Section 5. Finally, the summary and conclusions are given in Section 6. 2 FACE RECOGNITION
Face recognition is an active area of research with applications ranging from static, controlled mug-shot verification to dynamic, uncontrolled face identification in. a cluttered. background [5]. In the context of personal identification, face recognition usually refers to static, controlled fullfrontal portrait recognition [5]. By static, we mean that the facial portraits used by the face-recognition system are still facial images (intensity or range). By controlled, we mean that the type of background, illumination, resolution of the acquisition devices, and the distance between the acquisition devices and faces, etc. are essentially fixed during the image acquisition process. Obviously, in such a controlled situation, the segmentation task is relatively simple(and the intraclass variations are small. During the past 25 years, a substantial amount of research effort has been devoted to face recognition 15]. 125]. (1). In the early 1970s, face recognition was mainly based on measured facial attributes, such as eyes, eyebrows, nose, lips, chin shape, etc. 15]. Due to lack of computational resources 'and brittleness of feature extraction algorithms, only a very limited number of tests were conducted and the recognition performance of face-recognition systems was far from desirable. [5]. After the dormant 1980s, there was a resurgence in face-recognition research in the early 1990s. In addition to continuing efforts on attribute-based techniques [5],. a number of new face-recognition techniques,were proposed, including: * * * * principle component analysis (PCA) [22], 1121, [241, linear discriminant analysis (LDA) 123], singular value decomposition (SVD) 1101, and a variety of neural network-based techniques 1251.
The performance of these approaches is impressive. It was concluded that "face-recognition algorithms were devel-. oped and were sufficiently mature that they can be ported to real-time .experimental/demonstration system" [1191. Generally, there are two major tasks in face recognition: 1). locating faces in input images and 2) recognizing the located faces.
1298
Fig. 3. System architecture of the prototype integrated biometric identification system.
Face location itself continues to be a challenging problem 1) training stage, in which a set of N training face images for uncontrolled and cluttered images [5]. Fortunately, in are collected; eigenfaces that correspond to the M the context of personal identification, the background is highest eigenvalues are computed from the training controlled or almost controlled, so face location is generally set; and each face is represented as a point in the Mnot considered to be a difficult problem. Face recognition dimensional eigenspace, and from a general viewpoint also remains an open problem 2) operationalstage, in which each test image is first probecause transformations such as position, orientation, and jected onto the M-dimensional eigenspace; the Mscale'and changes in illumination produce large intraclass dimensional face representation is then deemed as a variations 1191. Again, in the context of personal identificafeature vector and fed to a classifier to establish the tion, the variations in acquired face images can be restricted identity of the individual. to a certain range which enables the current techniques to A Wx H face image I(x, y) can be represented as a Wx Hachieve a desirable performance 151. 1191. dimensional feature vector by concatenating the rows of I(x, y) In our system. the eigenface approach is used for the together. Thus, each W x H face image becomes a point in following reasons: the W x H-dimensional space. The total number of pixels in 1) in the context of personal identification, the back- a face image is typically large, on the order of several thouground, transformations, and illumination can be sands for even small image sizes. Face images in such a controlled, high-dimensional space are not randomly distributed. 2) eigenface approach has a compact representation-a Therefore, it is efficient and beneficial to project them to a facial image can be concisely represented by a feature lower-dimensional subspace using principle component vector with a few elements, analysis 124]. Let 4', W,;., N denote the. N W x H3) it is feasible to index an eigenface-based template dadimensional training vectors with zero-mean. Let the M tabase using different indexing techniques such that basis vectors, u,, u, .... u be a set of orthonormal vectors , the retrieval can be conducted efficiently 123), that best describe, the distribution of face images in the M4) the eigenface approach is a generalized template dimensional subspace (elgenspace), M 5 N. The kth eigenmatching approach which was demonstrated to be vector, u, A = 1, 2; .... M, is computed such that 124] more accurate than the attribute-based approach in Ak l N T(k4 )2( one study (4). The eigenface-based face recognition consists of the following two stages [241:
i=1
is maximum, subject to
1299
-I
r
1. -..-
'"
'
'
t
17J '
4
-;,,
~LU~
i~J
Fig. 4. First 10 eigenfaces obtained from 542 images of size 92 x 112, which are listed from left to right and top to bottom in a decreasing order of the corresponding eigenvalues.
r u J usu = 0
if i = j otherwise
fetal period [15]. Humans have used fingerprints for per-
(2) sonal identification for a long time. The biological properties of fingerprints are well understood which are summarized as follows: 1) individual epidermal ridges and furrows have different characteristics for different fingerprints; 2) the configuration types are individually variable, but they vary within limits which allow for systematic classification; 3) the configurations and minute details of individual ridges and furrows are permanent and do not change with time except by. routine injury, scratches, and scarring, as may be seen in Fig. 5 and Fig. 9 [15]. The uniqueness of a fingerprint is exclusively determined by the local ridge characteristics and their relationships. Fingerprint matching generally, depends on the comparison of local ridge characteristics and their relationships [14], [11], (17]. A total of 150 different local ridge characteristics, called minute details, have been identified [14]. These local ridge characteristics are not evenly distributed. Most of them depend heavily on the impression conditions and quality of fingerprints and are rarely observed in fingerprints. The two most prominent ridge characteristics, called minutiae, are ridge ending and ridge bifurcation. A ridge ending is defined as the point where a ridge ends abruptly. A ridge bifurcation is defined as the point where a ridge forks or diverges into branch ridges. A fingerprint typically contains about 40 to 100 minutiae. Examples of minutiae are shown in Fig. 5c. For a given fingerprint, a minutia can be characterized by its type, its x and y coordinates, and its direction, 6, whose definitions are also shown in Fig. 5c. Fingerprint verification consists of two main stages ill ),
The value A, is the kth largest eigenvalue of the covariance matrix E which can be estimated using the training samples by =
-
N1
"
.(3)
1=l1
The vector u, is the kth eigenvector of the covariance matrix E corresponding to A~,. With the M-dimensional eigenspace defined, training vectors, '", . N, can be represented as a set of Mdimensional feature vectors, (1,, (12, N:
T,,
= uT
I,
i= 1, 2, ... , N,
(4)
where u = (u,, u, .... ,u,). Fig. 4 shows the first 10 eigenfaces corresponding to the 10 largest eigenvalues, which were computed based on 542 training images (of size 92 x 112). In the operational phase, a detected face image, F, which is normalized to zero mean, is vectorized and projected onto the eigenvectors according to 11 = u ' With both F. training samples and test samples being projected onto an M-dimensional eigenspace, face recognition can be accomplished by a classifier operating in the eigenspace. In the context of personal identification, only a very limited number of.training samples are available for each individual (17]. Thus, a k-nearest neighbor classifier is typically used, in which the distance, d, called Distance From Feature Space (DFFS) 1241 between a template, 0, and a test pattern, fl, is defined as - nil, where 11 denotes L norm. 11
1'
3 FINGERPRINT VERIFICATION'
A fingerprint is the pattern of ridges and furrows on the surface of a fingertip. It is formed by the accumulation of dead, cornified cells that constantly slough as scales from the exposed surface (14). Its formation Is determined in the
11,4]:
1) minutiae extraction and 2) minutiae matching.
1300
Ening Ridn
Rideungi Bi)
Fig. 5. Fingerprints and minutiae. (a) and (b) Two different impressions of the same finger. (c) Ridge ending and ridge bifurcation.
(c)-~--~
(a)
(b)
(d)
Fig. 6. Results of our minutiae extraction algorithm on a fingerprint image (512 x 512) captured with an optical scanner. (a) Input image. (b) Orientation field. (c) Ridge map. (d) Extracted minutiae:
Due to a number of factors such as aberrant formations of epidermal ridges of fingerprints, postnatal marks, occupational marks, problems with acquisition devices, etc., acquired fingerprint images may not always have welldefined ridge structures. Thus, a reliable minutiae extraction algorithm should not assume perfect ridge structures and should degrade gracefully with the quality of fingerprint images. We have developed a minutiae extraction algorithm [1ll) based on the algorithm proposed in [20). It mainly consists of three steps: 1) orientation field (ridge flow) estimation, in which the ornentation field of input fingerprint images is estimated and the region of interest is located, 2) ridge extraction, in which ridges are extracted and thinned, and 3) minutiae detection and postprocessing, in which minutiae are extracted from the thinned ridge maps and refined. For each detected minutia, the following parameters are recorded: * x-coordinate, * y-coordinate, * orientation, which is defined as the local ridge orientation of the associated ridge, and * the associated ridge. The recorded ridges which are used for alignment in the minutiae matching are represented as one-dimensional discrete signals which are normalized by the average interridge distance. In an automatic fingerprint identification, ridge endings and ridge bifurcations are usually not differentiated from one another. Therefore, no minutiae type
information is recorded. A minutia is completely determined by its position and orientation. Fig. 6 shows the results of our minutiae extraction algorithm on a fingerprint image captured with an optical scanner. The minutiae matching determines whether, two minutiae patterns are from the same finger or not. A similarity metric between two minutiae patterns is defined and a thresholding on the similarity value is performed. By representing minutiae patterns as two-dimensional "elastic" point patterns, the minutiae matching may be accomplished by an "elastic" point pattern matching as long as it can automatically establish minutiae correspondences (in the presence of translation, rotation, and deformations) and detect spurious minutiae and missing minutiae. We; have developed an alignment-based "elastic" matching algorithm 11l, is capable of finding the corresponwhich dences between minutiae without resorting to an exhaustive search and has the ability to adaptively compensate for the nonlinear deformations and inexact transformations between different fingerprints. The alignment-based matching algorithm decomposes the minutiae matching into two stages: 1) Alignment stage, where transformations such as translation, rotation, and scaling between an input and a template in the database are estimated, and the input minutiae are aligned with the template minutiae according to the estimated parameters; and 2) Matching stage, where both the input minutiae and the template minutiae are converted to "strings" in the polar coordinate system, and an "elastic"" string matching algorithm is used to match the resulting strings, and
1301
finally, the normalized number of. corresponding minutiae pairs is reported. Let
P=
(x,".y ,opT
....
(x
.yP9e1
and = x y QQ) Q Q((XQ 'y I e Q , .... (

9 y
,y ,
9))
denote the p minutiae in the template and the q minutiae in the input image, respectively. The alignment-based matching algorithm is depicted as follows: I) Estimate the translation and rotation parameters between the ridge associated with each input minutia and the ridge associated with each template minutia and align the two minutiae patterns according to the estimated parameters. 2) Convert the template pattern and input pattern into the polar coordinate representations with respect to the corresponding minutiae on which alignment is achieved and represent them as two symbolic strings by concatenating each minutia in an.increasing order of radial angles:
Fig. 7. Fingerprint matching.
P=
((P PP)T
...
e ,0
(5)
T
Srule may be employed to reach a more reliable decision [26]; 2) Rank level; the output from each module is a set of possible labels ranked by decreasing confidence values, but the confidence values themselves are not specified; 3) Measurement level; the output from each module is a set of possible labels with associated confidence values; in this case, more accurate decisions can.be made by integrating different confidence measures to .a more informative confidence measure'.
In our system, the decision fusion Is designed to operate at the measurement level. Each of the top.n possible identities where r , eP , and 0 , represent the corresponding established by the face recognition module is verified by radius, radial. angle, and normalized minutiae orien- the fingerprint verification module. In order to carry out such a decision fusion scheme, we need to define a measure tation with respect to the reference minutiae, (x, y,. 0), that indicates the confidence of the decision criterion and a respectively. decision fusion criterion. 3) Match the resulting strings P and Q with a modified As discussed in Section 1, the confidence of a given decidynamic-programming algorithm to find the "edit sion criterion may be characterized by its FAR (false accepdistance" between P and Q. tance rate). In order to estimate FAR, the impostor distribu4) Use the minimum edit distance between P and Q to tion needs to be computed. How should we compute the establish the correspondence of the minutiae between impostor distribution? In practice, It can only be estimated P and Q. The matching score, S, is then defined as: from empirical data. But, this estimation problem requires 100M~q some care. In the context of personal identification, the reS= _1M Pq (7) quired FAR value is often a very small number (< 1 percent) [171. If the parametric form of the underlying imposwhere MQ is the number of 'minutiae which fall in the tor distribution is not known, nonparametric. techniques bounding boxes of template minutiae. The bounding need to be used. In order to guarantee that the estimated box of a minutia specifies the allowable tolerance in impostor distribution is reliable for characterizing the small the positions of the corresponding input minutiae FARs. a large representative test set that satisfies the folwith respect to the template minutiae. Fig. 7 shows an lowing two requirements is needed: It should be large example of minutiae matching. enough to represent the population, and it should contain enough samples from each category of the population. The above requirements are not easily satisfied in practice. An 4 DECISION FUSION extrapolation based on the knowledge of the parametric Decision fusion which integrates multiple cues has proved form of the underlying impostor distribution is needed. beneficial for improving the accuracy of a recognition .system'12), 13), 1131. Generally, multiple cues may be integrated 4.1 Impostor Distribution for Fingerprint Verification A model that can precisely characterize the impostor distriat one of the following three different levels 131: 1) Abstract level; the output from each module is only a bution of a minutia matching algorithm is not easy, since:
Q=((i9.e ,e , =
TO?)
.....
(r
O e Q)
).
(6)
set of possible labels without any confidence associated with the labels: in this case, the simple majority
1) the minutiae in a fingerprintare distributed randomly inthe region of interest;
1302
'IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 20, NO. 12, DECEMBER 1998
2) the region of interest of each input fingerprint may be different; 3) each input fingerprint tends to have a different number of minutiae;. 4) there may be a significant number of spurious minutiae and missing minutiae; 5) sensing, sampling, and feature extraction may result in errors in minutiae positions; and 6) sensed fingerprints may have different distortions. However, it is possible to obtain a general model of the overall impostor distribution by making some-simplifying assumptions. Let us assume that the input fingerprint and the template have already been registered and the region of interest of both the input fingerprint and the template is of the same size, a W x W (for example. 500 x 500) region. The W x W region is tessellated into small cells of size w x w which are assumed to be sufficiently large (for example, 40 x 40) such that possible deformation and transformation errors are within the bound specified by the cell size, Therefore, there are a total of w x W (= N,) different cells in the region of interest of a fingerprint. Further, assume that each fingerprint has the same number of minutiae, Nm (< N), which are distributed randomly in different cells and each cell contains at most one minutiae. Each minutia is directed towards one of the D (for example, eight) possible orientations with equal probability. Thus, for a given cell, the probability, Pm,,, that the cell is empty with no minutiae present is
N
-# I /
.,.
Template Minutiae Set
Input Minutiae Set
Fig. 8. Minutiae matching model, where a solid line indicates a match and a dashed line indicates a mismatch.
4.2 Impostor Distribution for Face Recognition

The characterization of impostor distribution for face recognition is more difficult. Due to the relatively low discrimination capability of face recognition, this module needs to keep the top n matches to improve the likelihood that the genuine individual will be identified if he or she is in the database. Let 0,, 2,.... (N be the N face templates stored in the database. The top n matches, .... , are obtained , by searching through the entire database, in which N comparisons are conducted explicitly (in the linear search case) or implicitly (in organized search cases such as the tree search). The top n matches are arranged in the increasing order of DFFS (Distance From Feature Space, Section 2) values. The smaller the DFFS value, the more likely it is that the match is correct. Since the relative distances between consecutive DFFSs are invariant to the mean shift of the DFFSs, it is beneficial to use relative instead of absolute DFFS values. The probability that a retrieved top n match is incorrect is different for different ranks. The impostor distribution should be a decreasing function of rank order and it is a function of both the relative DFFS values, A, and the rank order, i: '
(D;, (
and.the probability, P, that the cell has a mi-
nutia that is directed toward a specific orientation is A pair of corresponding minutiae between a template and an input is considered to be identical if and only if they are in the cells at the same position and directed in the same direction (see Fig. 8). With the above simplifying assump* tions, the number of corresponding minutiae pairs between any two randomly selected minutiae patterns Is a random variable, Y, which has a binomial distribution with parameters N, and P [18]:
g(Y = Y) =
NI m
PY(1 - P)(N.-Y)
(8)
y!(Nm - y)I The probability that the number of corresponding minutiae pairs between any two sets of minutiae patterns is less than a given threshold value, y, is
y-I
F()P ().
(10)
G(y) = g(Y < y) =

k=0
g(k).
(9)
The decision made by the proposed minutiae matching algorithm for an input fingerprint and a template is, based on the comparison of the "normalized" number of corresponding minutiae pairs against a threshold. Therefore, under the assumption that minutiae in the region of interest of fingerprints of different individuals are randomly distributed, the probability that an impostor. I. is accepted is {(1- G(y)}, where y, is the number of corresponding minutiae pairs between the impostor and the individual whom the impostor claims to be.
where F(A) represents the probability that the consecutive DFFS values between impostors and their claimed individuals at rank i are larger than a value A, and Ps,(i)represents the probability that the retrieved match at rank i is an impostor. In practice, F,(A) and P,() need to be estimated from empirical data. In order to simplify the analysis, we assume that each individual has only one face template in the database. Thus, there are a total of N individuals enrolled in the database and I,I2, INare used as identity indicators, Let X' denote , .... the DFFS between an individual and his/her own template which is a random variable with density function f(X) and let X, X, ..... X_,1 denote the DFFS values between an individual and the templates of the other individuals in the database, which are random variables with density functions.
1303
(x) I(x), .... f_ X ) .respectively Assume that

X" and X2. X2 ...... X_, are statistically independent and
4.3 Decision Fusion The impostor distribution for face recognition and the im... ,, the rank, 1, of X" among XZ , XP , .... XNI is aran- postor distribution for fingerprint verification provide condom variable with probability fidence measures for each of the top n matches retrieved by P(1 = i S(N- 1)! '1-p( N--i) -- ) (1-i-0 recognition module. Without a loss of generality, we 1) face assume that at most one of the n possible identities estab(=)=i!(N-1-i)Ip'(1-p)(l) lished by the facerecognition module for a given individual where is the genuine identity of the individual. The final decision by Integration either rejects all the n possibilities or accepts P f X )dX df . (12) only one of them as thegenuine identity. In practice, it is usually specified that the FAR of the system should be less When p < 1 and N is sufficiently large, P(1) may be apthan a given value [171. Therefore, the goal of decision fuproximated by a Poisson distribution [18), sion, in essence, is to derive a decision criterion which satisfies the FAR specification. e(- )a P(1) T ' (13) It is reasonable to assume that the DFFS between two different individuals is statistically independent of the finwhere a np. Obviously, P(1) is exactly the probability that gerprint matching score between them; facial similarity matches at rank i are genuine individuals. Therefore, between two individuals does not imply that they have P,,(i) = 1 - P(l= i). (14) similar fingerprints, and vice versa. This assumption should not be confused with the situation where an imAlthough the assumption that X2,X2, .... X,_, are i.i.d. postor tries to fool the system by counterfeiting the face may not be true in practice, It is still reasonable to use the and/or fingerprints of the genuine individual. Let above parametric form to estimate the probability that.reF(A)Pd,(i) and C(Y) denote the impostor distribution at trieved matches at rank I are impostors. Our experimental rank i for face-recognition and fingerprint-verification modresults support this claim. ules, respectively. T'he composite impostor distribution at Without any loss of generality, we assume that, for a rank i may be defined as given individual, fl, X, XZ, .... , XN_ are arranged in in(20) H,(A, Y) = F,(A)P.,,(i)C(Y). creasing order of values. Define the non-negative distance Let I,, I .... I,, denote the n possible identities'established between the (i+1)th and ith DFFS values as the ith DFFS by face recognition, {X,,.X 2 . . . , X,) denote the corresponding distance, Ai = X - X. l 15 N- 1. (15) n DFFSs, (Y,, YZ, ...; Y) denote the corresponding n fingerprint matching scores, and FAR o denote the specified value The distribution, f(A). of the ith distance, A,, is obtained of FAR. The final decision, ID(fl), for a given individual fl from the joint distribution w,(X, A) of the ith value, X, and is determined by the following criterion: the ith distance. ,, . ID(nfl) = f(, ) = 1
fX). For an individual, I. which has a template stored in the database, {@,, O,
X = = ... _,XX=
(x)
In our system, we assume that XXZ .. X_,. are distributed with a Gaussian distribution with unknown mean and variance.
w(X, 6.)dX,
(16)
JI
iHk
(A
k , Yk) < FAR 0, and
w,(Xf, A) = CF(4 'I1 - P(X+ )] ]'f(x)f (X'+6). (17)

(N- 1)! C = (i-1)!(N -2-i)!'
.
imposter
Hk(Ak' k) = min{H,(A, Y),.. . H otherwise
e(A., YA4
(18)
(18)
(21)
where A, = X,,, - X,. Since H,(A, Y) defines the probability that an impostor is accepted at rank i with consecutive relative DFFS. A. and fingerprint matching score, Y, the above decision criterion satisfies the FAR specification. Note that the decision criterion in (21) depends on the number of individuals, N, enrolled in the database, since F, depends on N. However, it does not mean that F has to be recomputed whenever a new individual is enrolled in the database. In fact, if N > 1, the corresponding Fs for different values of N are quite similar to one another. On the other hand, the decision criterion still satisfies the FAR specification when Nincreases, though it may not be able to take full advantage of the information contained in the N comparisons. In practice, an update scheme which recomputes the decision criterion whenever the number of
where F(Xp) = f
f''(XP)dX'
[91. With the distribution,
f,(A), of the ith distance defined, the probability that the DFFS of the impostor at rank i is larger than a threshold value, A, is
(A) = f(A,)dA,. (19)
The above equations do not make any assumptions about the distributions of X .X , ..... X_, as long as they are i.i.d. The equations also hold even if the mean values of X2, X , .... XNI shift. Therefore, it can tolerate, to a certain extent, DFFS variations which is a desirable property.
;'(
.qr
(a)
.(b)
(c)
. .; '
(e)
(f)
Fig. 9. Face and fingerprint pairs; the face images (92 x 112) are from the Olivetti Research Lab; the fingerprint images (640 x 480) are captured with a scanner manufactured by Digital Biometrics.
all" verification test by fitting the probability model described in Section 4.1, respectively. A total of 542 face images were usedas training samples. Since variations in position, orientation, scale, and illumination exist' in the face database, the 542 training samples were selected such that 5 'EXPERIMENTAL RESULTS the representative views are included. Eigenfaces were estiThe integrated biometric system was tested on the MSU mated from the 542 training samples and the first 64 eigenfingerprint database and a public domain face database. faces wereused for face recognition. The top n = 5 impostor The MSU fingerprint database contains a total of 1,500 fin- distributions were approximated. Generally, the larger, the gerprint images (640 x 480) from 150 individuals with 10 value of n, the lower the false reject rate of face recognition. images per individual, which were captured with an optical However, as n increases, more candidates need to be verified scanner manufactured by Digital Biometrics. When these by fingerprint verification. There is obviously a trade-off fingerprint images were captured. no restrictions on the between the accuracy and speed of a biometric system. Fig. 10b position, orientation, and impression pressure were im- shows the impostor distribution at rank no. 1. We randomly assigned each of the remaining 86 indiposed. The fingerprint images vary in quality. Approxiof the fingerprint images in the MSU da- viduals in the MSU fingerprint database to an individual in mately 90 percent tabase are of reasonable quality similar to the images the face database (see Fig. 9 for some examples). Since the. shown in Fig. 9b and Fig. 9d. Images of poor quality with DFFS value between two different individuals is statistiexamples shown in Fig. 9f and Fig. 9h are mainly due to cally independent of the fingerprint matching scores belarge creases and smudges in ridges, dryness of the im- tween the two individuals, such a random assignment of a pressed finger, and high impression pressure. The face da- face to a fingerprint is admissible. One. fingerprint for each tabase contains a total of 1,132 images of 86 individuals; 400 individual is randomly selected as the template for the inimages of 40 individuals with 10 images per individual are dividual. To simulate the practical identification scenario, from the Olivetti Research Lab. 300 images of 30 individu- each of the remaining 590 faces was paired with a fingerals with 10 images per individual are from the University of print to produce a test pair, In the test,' with a prespecified Bern, and 432 images of 16 individuals with 27 images per confidence value (FAR), for each of the 590 fingerprint and individual are from the MIT Media Lab. The images were face pairs, the top five matches are retrieved using face recresampled from the original sizes to a fixed size of 92 x 112 ognition. Then fingerprint verification is applied to each of and normalized to zero mean. the top five matches, and a final decision is made by-deciWe randomly selected 640 fingerprints of 64 individuals, sion fusion. as the training set and the remaining as the test set. The The prespecified FAR for a biometric system is usually mean and standard deviation of the impostor distribution very small (< 0.0001). In order to demonstrate that the,. (Fig. 10a) were estimated to be 0.70 and 0.64 from the biometric system does meet such a specification, a large 403,200 (640 x 630) impostor matching scores of "all against number of representative samples are needed. Unfortunately, added individuals is larger than a prespecified value can be used to guarantee that the decision criterion exploits all the available information.
HONG AND JAIN: INTEGRATING FACES AND FINGERPRINTS FOR PERSONAL IDENTIFICATION'
1305
nmatching cor
lirst DFFS distanoe
Fig. 10. Impostor distributions. (a) Impostor distribution for fingerprint verification; the mean and standard deviation of the impostor distribution are estimated to be 0.70 and 0.64, respectively. (b) The impostor distribution for face recognition at rank No. 1, where the stars (') represent empirical data and the solid curve represents the fitted distribution; the mean square error between the empirical distribution and the fitted distribution is 0.0014.
obtaining such a large number of test samples is both ex- 6 SUMMARY AND CONCLUSIONS pensive and time consuming.: In our test, we reuse faces by We have developed a prototype biometric system which different assignment practices-each time, a different finintegrates faces and fingerprints in authenticating a pergerprint Is assigned to a given face to form a face and finsonal identification. The proposed system overcomes the gerprint probe pair. Obviously, such a reuse scheme might limitations of both face-recognition systems and fingerresult in unjustified performance improvement. In order to print-verification systems. The integrated system operates diminish the possible gain in performance due to such a in the identification mode. The decision-fusion scheme reuse scheme, we multiplied the estimated impostor distriformulated in the system enables performance improvebution for face recognition by a constant of 1.25, which esment by integrating multiple cues with different confisentially reduces contribution of face recognition to the fidence measures. Experimental result's demonstrate that nal decision by a factor of 1.25. On the other hand, fingerprint verification operates in the one-to-one verification our system performs very well. It meets the response time as well as the accuracy requirements. mode, so different assignments may be deemed as different impostor forgeries. Therefore, the test results using such a random assignment scheme are able to reasonably estimate TABLE 1 FALSE REJECT RATES (FRR) ON THE TEST SET WITH the underlying performance numbers. In our test, 1,000 difDIFFERENT VALUES OF FAR ferent assignments were tried. A total of 590,000 (590 x 1,000) face and fingerprint test pairs were generated and tested. FAR False Reject Rates (FRR) The FRRs of our system with respect to different prespeciFace Fingerprint Integration fied FARs, as well as the FRRs obtained by "all-to-all" verifi1% 15.8% 3.9% 1.8% cations using only fingerprints (2,235.000 = 1,500 x 1,490 0.1% 42.2% 6.9% 4.4% tests) or faces (342,750 = 350 x (590 - 5) + 240 x (590 - 15) 0.01% 61.2% 10.6% 6.6% tests) are listed in Table 1. Note that the FRRs in integration 0.001% 64.1% 14.9% 9.8% column include the error rate (1.8 percent) of genuine indiThe false reject rates of face recognition are obtained based on 342,750 pairviduals not present in the top five matches. The receiver wisecomparisons; the false reject rates of fingerprint verification are obtained operating curves are plotted in Fig. 11, in which the based on 2,235,000 pairwise comparisons; the false reject rates of the inteauthentic acceptance rate (the percentage of genuine indigrated system are obtained based on 590,000 probes. viduals being accepted, i.e., 1 - FRR) is plotted against FAR.. We can conclude from these test results that the integration "TABLE 2 of fingerprints and faces does result in a significantly better AVERAGE CPU TIME FOR-ONE TEST ON A recognition performance. SUN SPARC 20 WORKSTATION. In order for an automatic personal identification system to Face Fingerprint Total be acceptable in practice, the response time of the system Recognition Verification (seconds) needs to be within a few seconds. Table 2 shows that our im(seconds) (seconds) plemented system does meet the response time requirement.
0.9 3.2 4.1
1306
13) R. Brunelli and D. Falavigna, "Personal Identification Using Multiple Cues," IEEE Trans. Pattern Analysis and Machine Intelligence. vol. 17, no. 10, pp. 955-966, Oct. 1995. [4) R. Brunelli and T Poggio, "Face Recognition: Features Versus Templates," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 10. pp. 1,042-1,052, Oct. 1993. 15) R. Chellappa, C. Wilson, and A. Sirohey, "Human and Machine Recognition of Faces: A Survey" Proc. IEEE, vol. 83. no. 5. pp. 705740, 1995. 161 R. Clarke, "Human Identification in Information Systems: Management Challenges and Public Policy Issues," Information Technology & People, vol. 7, no. 4, pp. 6=37, 1994. 171 J.C. Daugman, "High Confidence Visual Recognition of Persons by a Test of Statistical Independence," IEEE Trans. Pattern Analysis and Machine Intelligence, vol, 15, no. 11,pp. 1,148-1,161, Nov. 1993. 18) S.C. Davies,, "Touching Big Brother: How Biometric Technology . Will Fuse Flesh and Machine.".. Information Technology & People, vol. 7, no. 4, pp. 60-69. 1994. 19) EJ. Cumbel, Statistics of Extremes. New York: Columbia Univ. Press. 1958. |10) Z. Hong, "Algebraic Feature Extraction of Image for Recognilion," Pattern Recognition. vol: 24. no. 2, 211-219.'1991. pp: Ill) A. lain. L. Hong, and R. Bolle, "On-Line Fingerprint Verification," IEEE Trans. Pattern Analysis and Machine Intelligence. vol. 19. no. 4, pp. 302-314, Apr. 1997. 112) M. Kirby and L. Sirovich, "Application of the Karhunen-Loeve 'Procedure for the Characterization of Human Faces," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 12, no. I, pp. 103108, Jan. 1990. 113) J. Kittler, Y. Li, J. Matas, and M.U. Sanchez, "Combining Evidence in Multimodal Personal Identity Recognition Systems," Proc. First Int'l Conf. Audio Video-Based Personal Authentication, pp., 327-334, Crans-Montana, Switzerland, Mar. 1997. 114) H.C. Lee and R.E. Caensslen, Advances in Fingerprint Technology. New York: Elsevier, 1991. 115) A. Moenssens, Fingerprint Techniques. London: Chilton Book Company, 1971, 116) V. Nalwa, "Automatic On-Line Signature Verification," Proc. IEEE, vol. 85, no. 2, pp. 213-239, 1997. 1171 E. Newham, The Biometric Report. New York: SJB Services, 1995. 1181 A. Papoulls, Probability, Random Variables, and Stochastic Processes. New York: McGraw-Hill, 1965. 1191 P.J. Phillips, PJ. Rauss, and S.Z. Der, FERET (Face Recognition Technology) Recognition Algorithm Development and Test Results. Adelphi, Md.: U.S. government publication. ALR-TR-995, Army Research Laboratory. 1996.. 120) N. Ratha, S. Chen, and A.K. Jain, "Adaptive Flow Orientation Based Feature Extraction in Fingerprint Images," Pattern Recognition, vol. 28, no. 11, pp. 1,657-1,672. 1995. 1211 N. Ratha, K. Karu, S. Chen, and AK. Jain. "A Real-Time Matching System for Large Fingerprint Database," IEEE, Trans, Pattern Analysis and Machine Intelligence, vol. 18, no. 8, pp. 799-813. Aug. 1996. 1221 L. Sirovich and M. Kirby. "Low Dimensional, Procedure for Characterization of Human Faces." J. Optical Soc. Am., vol. 4, no. 3. pp. 519-524, 1987. 123) D.L. Swets and J. Weng. "Using Discriminant Eigenfeatures for Image Retrieval," IEEE Trans. Pattern Analysis and Machine Intelligence. vol. 18, no. 8. pp. 831-836. Aug. 1996. 124) M. Turk and A. Pentland, "Eigenfaces. for Recognition," ]:Cognitive Neumroscience,,vol. 3. no. 1. pp. 71-86. 1991. (25) D. Valertin, H. Abdi, A.). O'Toole, and C. Cottrell. "Connectionist Models of Face Processing: A Survey," Pattern Recognition. vol. 27., no. 9, pp. 1,209-1,230,1994. 126) Y.A. Zuev and S.K. Ivanov, "The Voting as a Way to Increase the Decision Reliability," Proc. Foundations of Information/Decision Fusion With Applications to Eng. Problems, pp. 206-210, Washington, D.C., Aug. 1996. 1271A.K. Jain, R. Bolle, and S. Pankanti. eds.. Biometrics: Personal Identilication in Networked Society. Norwell. Mass.: Kluwer Academic Publishers, in press.
10
10""
False Acceptanco Rate
10""
10"
. tO'
1%)
10'
Fig. 11. Receiver operating curves; the vertical axis is (1-FRR). The decision-fusion scheme formulated in this paper may be applied to similar scenario in other domains to provide a better discrimination performance. For example, in image database retrieval, a less reliable but computationally attractive algorithm may be used to retrieve the top n matches; then a more reliable, but computationally more expensive algorithm may be used to verify the top n matches; and finally an integrated decision criterion may be used to reach a more reliable decision. We must point out that the proposed system has been designed for a template database containing several thousand templates. Since it has not yet been shown that face recognition is sufficiently efficient in correctly retrieving a small number of top matches from a huge template database with millions of templates, our approach may not scale up very well. In addition,. our decision fusion scheme assumes that the similarity values between faces are statistically independent of the similarity values between fingerprints. While the assumption is valid for fingerprints and faces, it may not be true for other biometric characteristics. The specified FAR of a deployed biometric system is usually a very small number (< 1 percent). In order to provide a more convincing demonstration that the system can meet such a specification; large representative test samples are needed. We are in the process of conducting such a test on a larger face and fingerprint database. ACKNOWLEDGMENTS We gratefully acknowledge our many useful discussions with Sharath Pankanti and Ruud Bolle of the IBM T. J. Watson Research Lab. REFERENCES Ill) J. Atick, P. Griffin. and A. Redlich. "Statistical Appiroach to Shape From Shading: Reconstruction of 3D Face Surfaces From Single 2D Images." Neural Computation, to appear. 121 E.S. Bigun. J. Bigun. B. Duc, and S. Fischer, "Expert Conciliation for Multi Modal Person Authentication Systems by Bayesian Statistics," Proc. First lnt ' Conf. 'Audio Video-Based Personal Authentication. pp. 327-334. Crans-Montana. Switzerland. Mar. 1997.
1307
Lin Hong received the BS and MS degrees in computer science from Sichuan University, China, in 1987 and 1990, respectively, and the PhD degree in computer science from Michigan State University in 1998. His currently research interests include multimedia, biometrics, data mining, pattern recognition, image processing, and computer vision application. He is now working at Visionics Corporation.
Anil Jain is a university distinguished professor and chair of the Department of Computer Science at Michigan State University: His research interests include statistical pattern recognition, Markov random fields, texture analysis, neural networks, document image analysis, fingerprint matching, and 3D object recognition. He received the best paper awards in 1987 and 1991 and certificates for outstanding contributions in 1976, 1979, 1992, and 1997 from the Pattern Recognition Society. He also received the 1996 IEEE Transactions on Neural Networks Outstanding Paper Award. He was the editor-in-chief of the IEEE Transactions on Pattern Analysis and Machine Intelligence (1990-1994). He is the coauthor of Algorithms for Clustering Data (Prentice-Hall, 1988), has edited the book Real-Time Object Measurement and Classification (Springer-Verlag, 1988), and coedited the books, Analysis and Interpretation of Range Images (Springer-Verlag, 1989), Markov Random Fields (Academic Press, 1992), Artificial Neural Networks and Pattern Recognition (Elsevier, 1993), 3D Object Recognition (Elsevier, 1993), and BIOMETRICS: Personal Identification in Networked Society to be published by Kluwer in 1998. He is a fellow of the IEEE and IAPR and has received a Fulbright research award.
''
''
Defects in the images include but are not limited to the items checked: D
BLACK BORDERS IMAGE CUT OFF AT TOP, BOTTOM OR SIDES ADED TEXT OR DRAWING
I BLURRED
OR ILLEGIBLE TEXT OR DRAWING
D SKEWED/SLANTED IMAGES
Q COLOR OR BLACK AND WHITE PHOTOGRAPHS
I GRAY SCALE DOCUMENTS

D
LINES OR MARKS ON ORIGINAL DOCUMENT
Q REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY
Q OTHER:
IMAGES ARE BEST AVAILABLE COPY.
As rescanning these documents will not correct the image

problems checked, please do not report these problems to the IFW Image Problem Mailbox.
INTERNATIONAL SEARCH REPORT
International application No. PCTIUS05/33378
k. CLASSIFICATION OF SUBJECT MATTER GO6K 9/00( 2006.01),9/64( 2006.01);HO4N 9/47( 2006.01),7/18( 2006.01);HO4B 17/00( 2006.01) IPC:
382/11 5-118,124;348/143-161;455/67.12 USPC: According to Intemrnational Patent Classification (IPC) or to both national classification and IPC
B.
FIELDS SEARCHED
Minimum documentation searched (classification system followed by classification symbols) U.S.: 382/115-118,124;348/143-161;455/67.12 Documentation searched other than minimum documentation to the extent that such documents are included in the fields searched
Electronic data base consulted during the international search (name of data base and, where practicable, search terms used)
C.
DOCUMENTS CONSIDERED TO BE RELEVANT
Category * X X X
Citation of document, with indication, where appropriate, of the relevant passages US 2002/0190119 A (HUFFMAN) 19 December 2002 (19.12.2002), entire document US 200410081338 AlI (TAKENAKA) 29 April 2004 (29.4.2004), entire document US 6,697,103 BI (FERNANDEZ et al) 24 February 2004 (24.02.2004), column 4 lines 2367, column 6 lines 1-30 Figures I and 3-4.
Relevant to claim No. 1-20 1-20 I, 17,20
SFurther documents are listed in the continuation of Box C.

* "A"
of Special categories cited documents:
See patent family annex.

"T" later document published after theinternational filing dateor priority the date andnot in conflict with theapplication but cited to understand principlc or theory undedying the invention the of document particular relevance; claimed invention cannot be to novel or cannot be considered involve aninventive step considered whenthedocunent is takenalone be invention cannot of document particular relevancc; theclaimed to considered involve an inventive step whenthedocuncut is conmbus suchcombination being documents, with one or more other isucb obvious to a person stilled in theal of mcembrnerthesamepatent family document
stateof the art which isnot considered to be of docmuit defining the gencaml particular relevance "X" ealier application or patentpublished onor after the international filing date tduow document which many doubts on priority claim(s) or which is cited to or citatidoo otherspecial reason(as the establish publication dateof another specified) referring to anoral disclosure, use,exhibition or other means document published prior to the international filing datebut later than the documwent priority date claimed -&" "Y"
"E" "L"
"O" P"
Date of the actual completion of the international search 13 March 2006 (13.03.2006) Name and mailing address of the ISA/US Mail Stop PCT, Atn: ISA/US Commissioner for Patents P.O. Box 1450 22313-1450 3-1450 2 Facsimilexandria, Virginia Facsimile No. (571) 273-3201 2320
Form PCT/ISA/210 (second sheet) (Apn)l 2t))
Date of mailing of the intemrn Authorized officer Jingge Wu Telephone No. 571-272-7361
scl
M f
ep
l
'j
Application No. 11/231,35,3 Docket No. 577832000200

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 4, APRIL 1997
On-Line Fingerprint Verification

Anil Jain, Fellow, IEEE, Lin Hong, and Ruud Bolle, Fellow, IEEE
Abstract-Fingerprint verification is one of the most reliable personal identification methods. However, manual fingerprint verification is so tedious, time-consuming, and expensive that itis incapable of meeting today's increasing performance requirements. An automatic fingerprint identification system (AFIS) is widely needed. It plays a very important role inforensic and civilian applications such as criminal identification, access control, and ATM card verification. This paper describes the design and implementation of an on-line fingerprint verification system which operates intwo stages: minutia extraction and minutia matching. An improved version of the minutia extraction algorithm proposed by Ratha et al., which is much faster and more reliable, is implemented for extracting features from an input fingerprint image captured with an on-line inkless scanner. For minutia matching, an alignment-based elastic matching algorithm has been developed. This algorithm iscapable of finding the correspondences between minutiae inthe input image and the stored template without resorting.to exhaustive search and has the ability of adaptively compensating for the nonlinear deformations and inexact pose transformations between fingerprints. The system has been tested on two sets of fingerprint images captured with inkless scanners. The verification accuracy is found to be acceptable. Typically, a complete fingerprint verification procedure takes, on an average, about eight seconds on a SPARC 20 workstation. These experimental results show that our system meets the response time requirements of on-line verification with high accuracy. Index Terms-Biometrics, fingerprints, matching, verification, minutia, orientation field, ridge extraction.
---------+ ----------
INTRODUCTION
INGERPRINTS
are graphical flow-like F human fingers. They have been widelyridges present on used in personal'
An automatic fingerprint identification system is concerned with some or all of the following issues:
identification for several centuries 111. The validity of their * Fingerprint Acquisition: How to acquire fingerprint imuse has been well established. Inherently, using current ages and how to represent them in a proper format. technology fingerprint identification is much more reliable * Fingerprint Verification: To determine whether two than other kinds of popular personal identification methods fingerprints are from the same finger. based on signature, face, and speech 1111, (31, 1151. Al* FingerprintIdentification: To search for a query fingerthough fingerprint verification is usually associated with print in a database. criminal identification and police work, it has now become * FingerprintClassification: To assign a given fingerprint more popular in civilian applications such as access control, to one of the prespecified categories according to its financial security, and verification of firearm purchasers geometric appearance. and driver license applicants 11il1, 131. Usually, fingerprint verification is performed manually by professional finger- A number of methods are used to acquire fingerprints. print experts. However, manual fingerprint' verification is Among them, the inked impression method remains the so tedious, time-consuming, and expensive that it does not most popular. It has been essentially a standard technique meet the performance requirements of the new applica- for fingerprint acquisition for more than 100 years [31. The tions. As a result, automatic fingerprint, identification sys- first step in capturing an inked impression of a fingerprint tems (AFIS) are in great demand I1ll. Although significant is to place a few dabs of ink on a slab then rolling it out progress has been made in designing automatic fingerprint smoothly with a roller until the slab is covered with a thin, identification systems over the past 30 years, a number of even layer of ink. Then the finger is rolled from one side of design factors (lack of reliable minutia extraction algorithms, the nail to the other side over the inked slab which inks the difficulty in quantitatively defining a reliable match between . ridge patterns on top of the finger completely. After that, fingerprint images, fingerprint classification, etc.) create bot- the finger is rolled on a piece of paper so that the inked impression of the ridge pattern of the finger appears on the tlenecks in achieving the desired performance i111). paper. Obviously, this method is time-consuming and unsuitable for an on-line fingerprint verification system. Inkless fingerprint scanners are now available which are capable of directly acquiring fingerprints in digital form. This * A.lain and L. Hong are with the Pattern Recognition and Image Processmethod eliminates the intermediate digitization process of ing Laboratory, Departmentof Computer Science, Michigan State Univerinked fingerprint impressions and makes it possible to sity, East Lansing, MI 48824. E-mail: (fain, honglin)0cps.msu.edu. build an on-line system. Fig. I shows the two inkless fin* R. Bolle is with the Exploratory Computer Vision Croup, IBM. TJ.Watson Research Center. Yorktown Heights; NY 10598. gerprint scanners used in our verification system. FingerEmail: bolle@watson.ibm.com. print images captured with the inked impression method Manuscript received Feb. 2, 1996; revised Oct. 21. 1996. Recommended for accepand the inkless impression method are shown in Fig. 2.
tance by B. Dom. For information on obtaining reprints of this article. please send e-mail to: transpami@computer.org. and reference IEEECS Log Number P96113.
0162-8828197510.00 01997 IEEE
JAIN ET AL.: ON-LINE FINGERPRINT VERIFICATION

t
(a)
(b)
Fig. 1. Inkless' fingerprint scanners: (a) Manufactured by Identlx. (b) Manufactured by Digital Biometrics.
(a)
(b)
Fig. 2. Comparison of fingerprint images captured by different methods. (a) Inked impression method (from NIST database). (b) Inkless impression method (with a scanner manufactured by Digital Biometrics).
However, no matter what type of framework is used, the classification is based on ridge patterns, local ridge orientations and minutiae. Therefore, if these properties can be described quantitatively and extracted automatically from a fingerprint image then fingerprint classification will become an easier task. During the past several years, a number of researchers have attempted to solve the fingerprint classification problem 11)1, 131, 19), 110), 126). Unfortunately, their efforts have not resulted in the desired accuracy. Algorithms reported in the literature classify fingerprints into five, or six categories with about 90 percent classification accuracy on a medium size test set (several thousand images) 191, 1101, 126). However, to achieve a higher recognition accuracy with a large number of categories still remains a difficult problem. Fingerprint verification determines whether two fingerprints are from the same finger or not. It is widely believed that if two fingerprints are from the same source, then their local ridge structures (minutia details) match each other topologically 1111, 13]. Eighteen different types of local ridge descriptions have . been identified . 1111. The two most prominent structures are ridge endings and ridge bifurcations which are usually called minutiae. Fig. 4 shows examples of ridge endings and ridge bifurcations. Based on this observation and by representing the minutiae as a point pattern, an automatic fingerprint verification problem may be reduced to a point pattern matching (minutia matching) problem. In the ideal case, if 1) the correspondences between the template and input fingerprint are known, 2) there are no deformations such as translation, rotation and nonlinear deformations, etc. between them, and 3) each minutia present in a fingerprint image is exactly localized, then fingerprint verification consists of the trivial task of counting the number of spatially matching pairs between the two images.
I-f'
Ridge Ending Fig. 4. Ridge ending and ridge bifurcation.
Ridge Bifurcation
per:
-"s
However, in practice 1) no correspondence is known beforehand, 2) there are relative translation, rotation and nonlinear deformations between template minutiae and input minutiae, 3) spurious minutiae are present in both templates and inputs, and 4) some minutiae are missed, Therefore, in order for a fingerprint verification algorithm to operate under such circumstances, it is necessary to automatically obtain minutia correspondences, to recover deformations, and to detect spurious minutiae from fingerprint images. Unfortunately, this goal is quite difficult to achieve. Fig. 5 illustrates the difficulty with an example of two fingerprint images ofthe same finger.
(d) (e) (f) Fig. 3. A coarse-level fingerprint classification into six categories: (a)Arch. (b)Tented arch. (c) Right loop. (d) Left loop. (e)Whorl. (f)Twin loop. The goal of fingerprint classification is to assign a given fingerprint to a specific category according to its geometric properties (Fig. 3 shows a coarse-level fingerprint classification). The main purpose of fingerprint classification is to facilitate the management of large fingerprint databases and to speedup the process of fingerprint matching. Generally, manual fingerprint classification is performed within a specific framework such as the well-known Henry system 131. Different frameworks use different sets of properties.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELUGENCE, VOL. 19, NO. 4, APRIL 1997
The following two modules are the main components of our on-line fingerprint verification system: * Minutiae extraction. Minutiae are ridge, endings or ridge blfurcatlons. Generally, if a perfect segmentation can be obtained, then minutia extraction is just a trivial task of extracting singular points in a thinned ridge map. However, in practice, it is not always possible to obtain a perfect ridge map. Some global heuristics need to be used to overcome this limitation. * Minutia matching. Minutia matching, because of deformations in sensed fingerprints, is an elastic matching of point patterns without knowing their .correspondences beforehand. Generally, finding the best match between two point patterns is intractable even if minutiae are exactly located and no deformations exist between these two point patterns. The existence of deformations makes the minutia matching much more difficult. For segmentation and minutia extraction, a modified version of the minutia extraction algorithm proposed in 1181 is implemented which is much faster and more reliable for minutia extraction. We propose a hierarchical approach to obtain a smooth orientation field estimate of the input fingerprint image, which greatly improves the performance of minutia extraction. For minutia matching, we propose an alignment-based elastic matching algorithm. This algorithm is capable of finding the correspondences between minutiae without resorting to an exhaustive search and has the ability to adaptively compensate for the nonlinear deformations and inexact pose transformations between different fingerprints. Experimental results show that our system achieves excellent performance in a real environment. In the following sections we will describe in detail our on-line fingerprint verification system. Section 2 mainly discusses the fingerprint feature extraction module. Section 3 presents our minutia matching algorithm. Experimental results on two fingerprint databases captured with two different inkless scanners are described in Section 4. Section 5 contains the summary and discussion.
an-
*.-
.4-.-
Fig. 5. Two different fingerprint images from the same finger. In order to know the correspondence between the minutiae of these two fingerprint images, all the minutiae must be precisely localized and the deformations must be recovered. Fingerprint identification refers to the process of matching a query fingerprint against a given fingerprint database to establish the identity of an individual. Its goal is to quickly determine whether a query fingerprint is present in the database and to retrieve those which are most similar to the query from the database.' The critical issues here are both retrieval speed and accuracy. In fact, this problem relates to a number of techniques studied under the auspices of computer visiori, pattern recognition, database, and parallel processing. Olperational fingerprint retrieval systems are being used by various law enforcement agencies Ill). In this paper, we will introduce an on-line fingerprint verification system whose purpose is to capture fingerprint images using an inkless scanner and to compare them with those stored in the database in "real time." Such a system has great utility in a variety of personal identification and access control applications: The overall block diagram of our system is shown in Fig. 6. It operates as follows: I) Off-line phase: Several impressions (depending on the specification of the system) of the fingerprint of a person to be verified are first captured and processed by a feature extraction module; the extracted features are stored as templates in a database for later use; 2) On-line phase: The individual to be verified gives his/her identity and places his/her finger on the inkless fingerprint scanner, minutia points are extracted from the captured fingerprint image; these minutiae are then fed to a matching module, which matches them against his/her own templates in the database.
-:-
2 MINUTIA EXTRACTION
It is widely known that a professional fingerprint examiner relies on minute details of ridge structures to make fingerprint. identifications [1111, 13). The topological structure of the minutiae of a fingerprint is unique, and invariant with aging and impression deformations [11), 131. This implies that fingerprint identification can be based on the topological structural matching of these minutiae. This reduces the complex fingerprint verification to minutia matching process which, in fact, is a sort of point pattern matching with the capability of tolerating, to some restricted extent, deformations of the input point patterns. Therefore, the first stage in an automatic fingerprint verification procedure is to extract minutiae from fingerprints. In our on-line fingerprint verification system, we have implemented a minutia extraction algorithm which is an improved version of the method proposed by Ratha et al. 1181. Its overall flowchart is depicted in Fig. 7. We assume that the resolution of input fingerprint images is 500 dpi.
. I.. .. . n n.n
,... I l
.... .
)flgJ~mntVij"h Ni, .imj,,a
...... NI ,,,u
..
1
.."
Fig. 6. Overview of our on-line fingerprint verification system.
Fingerprint
Smoothin
Image Filer
Orientation Field
Estimation
Fingerprint Region
Localization
Minutiac
Minutia
Extracnon Thinning
Ridge
Extraction
* If the consistency level (2) is above a certain threshold ST. then the local orientations around this region are, reestimated at a lower resolution level until it is below a certain level. With this post-smoothing scheme, a fairly smooth orientation field estimate can be obtained. Fig. 8 shows the orientation field of a fingerprint image estimated with our new algorithm:
S...............................................................
Fig. 7. Flowchart of the minutia extraction algorithm. 2.1 Estimation of Orientation Field A number of methods have been proposed to estimate the orientation field of flow-like patterns 117). In our system, a new hierarchical implementation of Rao's. algorithm 1171 is used. Rao's algorithm consists of the following main steps: 1) Divide the input fingerprint image into blocks of size Wx W. 2) Compute the.gradients Gx and Cy at each pixel in each block. 3) Estimate the local orientation of each block using the following formula:
12Gx(i, j)G,(ij)
-
(a) Rao's method. (b)Hierarchical method. Fig. 8. Comparison of orientation fields by Rao's method and the proposed hierarchical method; the block size (W x WI) 16 x 16 and is the size of 0 is 5 x 5. After.the orientation field of an input fingerprint image is estimated, a segmentation algorithm which is based on the local variance of gray level is used to locate the region of interest from the fingerprint image. In our segmentation algorithm, we assuinme. that there is only one fingerprint present in the image. 2.2 Ridge Detection After the orientation field of the input image is estimated and the fingerprint region is located, the next step of our minutia exaction algorithm, is ridge detection. The most salient property corresponding to ridges in a fingerprint image is the fact that gray level values on ridges attain their local maximaalong the normal directions of local ridges. Therefore, pixels can be identified to be ridge pixels based on this property. In our minutia detection algorithm, a fingerprint image is first convolved With the following two masks, h (x, y u, v) and hb(x, y u. v),, of size L x H (on an 1 average 11 x 7 in our system), respectively. These two masks are capable of adaptively accentuating the local maximum gray level values along the normal direction of the local ridge direction:
h,(x. y: u: = v)
I e-d
= 2tan-'
1W -
S'(1) ,=,I=,(Gx(i.j) i
Gy(i, j))
where W is the size of the block, and G,and Cy are the gradient magnitudes in x and y directions, respectively. The orientation field of a good quality fingerprint image can be reasonably estimated with this algorithm. However, the presence of high-curvature ridges, noise, smudges, and breaks in ridges leads to a poor estimate of the local orientation field. A postprocessinrig procedure needs to be applied to overcome this limitation. In our system, the following iterative steps are added to improve an inconsistent orientation field: * Compute the consistency level of the orientation field in the local neighborhood of a block (i, j) with the following formula:
= oN*
N
(i',j) - 0(i.j) 2
(r._)ED
(2)
10' - BI=
d - 180 d if(d = (0' otherwise
+ 360) mod 360) < 180 I

-
if
vctg(0(x.y)) - 2cos(H(xy))
2 cos(e(x, y))
where D represents the local neighborhood around the block (i, J) (in our system, the size of.D is 5 x 5); N is the number of blocks within D; 0(i',j') and 6(i,j) are local ridge orientations at blocks (i', j') and (i, j), respectively.
ifu = (vctg(0(x,y)), v E
otherwise
(4)
h(x, y: u. v)= S( 2ne if U - vctg(0(x. y)) +

H 2 cos(9(x, y))
.)
v VE n
(5)
- t
ifu =
(vctg((x.y))),
ve
Although the above heuristics do delete a large percentage of spurious minutiae, many spurious minutiae still survive. The reason is that the above processing relies on local ridge information. If this information itself is unreliable, then the above heuristics have no way of differentiating false minutiae from true minutiae. Therefore, a refinement which is based on structural information is necessary. Our refinement algorithm eliminates the spurious minutiae based on the following rules: * If several minutiae form a cluster in a small region, then remove all of them except for the one nearest to the cluster center. * If two minutiae are located close enough, facing each other, but no ridges lie between them, then remove both of them.
otherwise
2= - Lsin((x, y)) 2
Lsin(o(x,y)) 2
where 9(x, y) represents the local ridge direction at pixel (x, y). If both the gray level values at pixel (x, y) of the convolved images are larger than a certain threshold Tidge, then pixel (x, y) is labeled as a ridge. By adapting the mask width to the width of the local ridge, this algorithm can efficiently locate the ridges in a fingerprint image. However, due to the presence of noise, breaks, and smudges, etc. In the Input Image, the resulting binary ridge map often contains holes and speckles. When ridge skeletons are used for the detection of minutiae, the presence of such holes and speckles will severely handicap the performance of our minutia extraction algorithm because these holes and, speckles may drastically change the skeleton of the ridges. Therefore, a hole and speckle removal procedure needs to be applied before ridge thinning. After the above steps are performed on an input fingerprint image., a relatively smooth ridge map of the fingerprint Is obtained. The next step of our minutia detection algorithm is to thin the ridge map and locate the minutiae. 2.3 Minutia Detection Minutia detection is a trivial task when an Ideal thinned ridge map is obtained. Without a loss of generality, we assume that if a pixel is on a thinned ridge (eight-connected), then it has a value 1, and 0 otherwise. Let (x, y) denote a pixel on a thinned ridge, and N o , N i, .... N denote its eight neighbors. A pixel (x, y) is a ridge ending if(
and a ridge bifurcation if
(a) Input image.
(b) Orientation field.
(c) Fingerprint region (c)
Fingerprint region.
(d) Ridge map.
, N) = 1
(8
0 N)> 2. However, the
presence of undesired spikes and breaks present in a thinned ridge map may lead to many spurious minutiae being detected. Therefore, before the minutia detection, a smoothing procedure is applied to remove spikes and to join broken ridges. Our ridge smoothing algorithm uses the following heuristics: * If a branch in a ridge map is roughly orthogonal to the local ridge directions and. its length is less than a specified threshold T. then it will.be removed. * If a break in a ridge is short enough and no other ridges pass through it, then it will be connected.
(e) Thlinned ridge map.
() extracted minutiae. (f)extracted minutiae.

i
Fig. 9. Results of our minutia extraction algorithm on a fingerprint image (512 x 512) captured with an inkless scanner. (a) Input image. (b) Orientation field superimposed on the input image. (c) Fingerprint region. (d) Extracted ridges. (e) Thinned ridge map. (f) Extracted minu-
tiae and their orientations superimposed on the input image.
JAIN ET AL: ON-LINE FINGERPRINT VERIFICATION
After the above refinement procedure is performed, the surviving minutiae are treated as true minutiae. Although the above heuristics can not ensure a perfect location of each minutia, they are able to delete several spurious minutiae. For each surviving minutia, the following parameters are recorded: I) x-coordinate, 2) y-coordinate, 3) orientation which is defined as the local ridge orientation of the associated ridge, and 4) the associated ridge. The recorded ridges are represented as one-dimensional discrete signals which are normalized by the average interridge distance. These recorded ridges are used for alignment in the minutia matching phase. Fig. 9 shows the results of our minutia extraction algorithm on a fingerprint image captured with an inkless scanner. 3 MINUTIA MATCHING.
implemented. Recognition by alignment has received a great deal of attention during the past few years 1121, because it is simple in theory, efficient in discrimination, and fast in speed. Our alignment-based matching algorithm decomposes the minutia matching into two stages: 1) Alignment stage, where transformations such as translation, rotation and scaling between an input and a template in the database are estimated and the input minutiae are aligned with the template minutiae according to the estimated parameters; and 2) Matching stage, where both the input minutiae and the template minutiae are converted to polygons in the polar coordinate system and an elastic string matching algorithm is used to match the resulting polygons. 3.1 Alignment of Point Patterns Ideally, two sets of planar point patterns can be aligned completely by two corresponding point pairs. ,A true alignment between two point patterns can be obtained by testing all possible corresponding point pairs and selecting the optimal one. However, due to the presence of noise and deformations, the input minutiae cannot always be aligned exactly with respect to those of the templates. In' order to accurately recover pose transformations between two point patterns, a relatively large number of corresponding point pairs need to be used. This leads to a prohibitively. large number of possible correspondences to be tested. Therefore, an alignment by corresponding point pairs is not practical even though it is feasible.
Y
Templat Minuti
Generally, an automatic fingerprint verification/identification is achieved with point pattern matching (minutiae matching) instead of a pixel-wise matching or a ridge pattern matching of fingerprint images. A number of point pattern matching algorithms have been proposed In the literature 123), 111, 1211, 1161. Because a general point matching problem is essentially intractable, features associated with each point and their spatial properties such as the relative distances between points are often used in these algorithms to reduce the exponential number of search paths. The relaxation approach [161 iteratively adjusts the confidence level of each corresponding pair based on its consistency with other pairs until a certain criterion is satisfied. Although a number of modified versions of this algorithm have been proposed 'to reduce the matching complexity 123), these algorithms are inherently slow because of their iterative nature. The Hough 'transform-based approach proposed by Stockman et al. 1221 converts point pattern matching to a problem of detecting the highest peak in the Hough space of transformation parameters. It discretizes the transformation parameter space and accumulates evidence in the discretized space by deriving transformation parameters that relate two point patterns using a substructure or feature matching technique. Karu and Jain 181 proposed a hierarchical Hough transform-based registration algorithm which greatly reduced the size of accumulator array by a multiresolution approach. However, if the number of minutia point is less than 30, then it is very difficult to accumulate enough evidence in the Hough transform space for a reliable match. Another approach to point matching is based on energy minimization. This approach defines a cost function based on an initial set of possible correspondences and uses an appropriate optimization algorithm such as genetic algorithm [1) and simulated annealing 1211 to find a possible suboptimal match. These methods tend to be very slow and are unsuitable for an on-line fingerprint verification system. In our system, an alignment-based matching algorithm is
(6 %.1y) npu Minultu Inulu KiJl
Fig. 10. Alignment of the input ridge and the template ridge.
It is well known that corresponding curve segments are capable of aligning two point patterns with a high accuracy in the presence of noise and deformations. Each minutia in a fingerprint is associated with a ridge. It is clear that a true alignment can be achieved by aligning . corresponding ridges (see Fig. 10). During the minutiae detection stage, when a minutia is. extracted and recorded, the ridge on which it resides is also recorded. This ridge is represented as a planar curve with its origin coincident with the minutia and its x-coordinate being in the same direction as the.direction of the minutia. Also, this planar curve is normalized with the average inter-ridge distance. By matching these ridges, the relative pose transformation between the input fingerprint and the template can be accurately estimated. To be specific, let Rd and R denote the sets of ridges asso-
dciated with the minutiae in input image and template, re-' spectively. Our alignment algdrithm can be described in terms of the following steps: 1) For each ridge d e Rd. represent it as an onedimensional discrete signal and match it against each ridge, D E RD according to the following formula:
where (x,. yi. minutia
0)
(i = 1, 2,.... N), represents an input

A ,
and (x
yA, A) 0
represents
the corre-
sponding aligned minutia.
3.2 Aligned Point Pattern Matching If two identical point patterns are exactly aligned with each other, each pair of corresponding points is completely coin'dL dD 2 cident. In such a case, a point pattern matching can be simply achieved by counting the number of overlapping pairs. io-=0 However, in practice. such a situation is not encountered. where L is the minimal length of the two ridges and d, On the one hand, the error in determining and localizing minutia hinders the alignment algorithm to recover the and D; represent the distances from point i on the relative pose transformation exactly, while on the other ridges d and D to the x-axis, respectively. The sam- hand, our alignment scheme described above does not pling interval on a ridge, is set to the average inter- model, the nonlinear deformation of fingerprints which is ridge distance. If the matching score S (0 < S < 1) is an inherent propertyof fingerprint impressions. With the larger than a certain threshold Tr, then go to step 2., existence of such a nonlinear deformation, it is impossible to exactly recover the position of each input minutia with otherwise continue to match the next pair of ridges. 2) Estimate the pose transformation between the two respect to its.corresponding minutia in the template. Thereneeds to, Sridges (Fig. 10). Generally, a least-square method can fore, the aligned point pattern matching algorithm be elastic which means that it should be capable of toleratbe used to estimate the pose transformation. However, in our system, we observe that the following ing,.to some extent, the deformations due to inexact extracmethod is capable of achieving the same accuracy with tion of minutia positions and nonlinear deformations. Usually, such an elastic matching can be achieved by placing a less computation. The translation vector (Ax, Ay) bebounding box around each template minutia, which specitween the two corresponding ridges is computed by fies all the possible positions of the corresponding input minutia with respect to the template minutia, and restrictXd AX l ing the corresponding minutia in the input image to be (8) D yd y within this box 1181. This method does not provide a satisfactory performance in practice, because local deformations D where (xd, y)T and (x , yo)T are thex and y coordimay be small while the accumulated global deformations nates of the two minutiae, which are called reference can be quite large. We have implemented an adaptive elasminutiae, associated with the ridges d and D, respectic matching algorithm with the ability to compensate the tively. The rotation angle Ao between the two ridges minutia localization errors and nonlinear deformations. is computed by Let P =( x
i=0
ylO ,
(XP, y.)) 0Q
denote the set of M minutiae in the template and where L is the minimal length of the two ridges d and D; y, and F'are radial angles of the ith point on the ridge with respect to the reference minutia associated with the two ridges d and D, respectively. The scaling .. factor between the input and template images Is assumed to be one. This is reasonable, because fingerprint images are captured with the same device in both the off-line processing phase and the on-line verification phase. 3) Denote the minutia (x, y. Od)T . based on which the pose transformation parameters are estimated, as the reference minutia. Translate and rotate all the N input minutiae with respect to this reference minutia, according to the following formula: sin A0 cos Ae . 0 Q
Q=((x, = x .,yX , .. . (.(N ,YN'eN)T)
...
denote the set of N minutiae in the input image which is aligned with the above template with respect to a given reference minutia point. The steps in our elastic point pattern matching algorithm are given below: 1) Convert each minutia point to the polar coordinate system with respect to the corresponding reference minutia on which the alignment.is performed: (x; eI = tan
(11)
= Ay +sin AG 0 A A
x cosA
0ily, - yo
0)1- e
(10)
where (x;, y;, 0;) are.the coordinates of a .minutia,
r (x yr ,
t)are the coordinates of the reference minu-
T tia, and (r. e,60)is the representation of the minutia t
in polar coordinate system (r represents the radial distance, ej represents the radial angle, and 0 represents the orientation of the minutia with respect to the reference minutia). 2) Represent the template and the input minutiae in the polar coordinate system as symbolic strings by concatenating each minutia in the increasing order of radial angles:
the cost function of a string matching algorithm can achieve a certain amount of error tolerance. Given two strings PP and Qp of lengths M and N, respectively, the edit distance, C(M, N), in our algorithm is recursively defined with the following equations:. C(m,n)=
O
if m= O,or n= 0
S C(m-l,n)+0fl
min 0< mM. C(m.n- l)+ C(m- ,n- 1)+ w(m,n)J and 0 <n 5 N
(15)
P=
(r,, e, OP)
((r Q, O)T e,
.. , r ,e ,
p )T)
(12)
QP =
...
r, , e,
)T)
(13)
where (r!, ef, 0)
and (,Q, e2.9 Q) represent the corAe
Il
=
ar
-rnQI+P
3e+yeA
<.< if Irrf -r De < e and AO.< otherwise
(16)
responding radius, radial angle, and normalized minutia orientation with respect to the reference minutia, respectively. 3) Match the resulting strings Pp and Qp with a dynamicprogramming algorithm 141 to find the edit distance between Pp and Qp which is described below. 4) Use the edit distance between Pp and Qp to establish the correspondence of the minutiae between Pp and Qp. The matching score, Mg, is then computed according to the following formula: 100Njojr OON r Mpq = S max{ M, N)}
.a
if (a = e - e + 360) mod 360) < 180 (17)

- 180 otherwise
Ae
if a = (m - O + 360) mod 360) < 180
S(18)
(14)
a- 180 otherwise where a. /3, and y are the weights associated with each component, respectively; 8, e, and e specify the bounding is a pre-specified penalty for a mismatch. Such box; and an edit distance, to some extent, captures the elastic property of string matching. It represents a cost of changing one polygon to the other. However, this scheme can only tolerate, but not compensate for, the adverse effect on matching produced by the inexact localization of minutia and nonlinear deformations. Therefore, an adaptive mechanism is needed. This adaptive mechanism should be able to track the local nonlinear deformation and inexact alignment and try to alleviate them during the minimization process. However, we do not expect that this adaptive mechanism can handle the 'order flip" of minutiae, which, to some extent, can be solved by an exhaustive reordering and matching within a local angular window. In our matching algorithm, the adaptation is achieved by adjusting the bounding box (Fig. 11) when an inexact match is found during the matching process. It can be represented as follows:
where Npair is the number of the minutiae which fall in the bounding boxes of template minutiae. The maximum and minimum values of the matching score are 100 and 1, respectively. The former value indicates a perfect match, while the later value indicates no match at all. Minutia matching in the polar coordinate has several advantages. We have observed that the nonlinear deformation of fingerprints has a radial property.. In other words, the nonlinear deformation in a fingerprint impression usually starts from a certain point (region) and nonlinearly radiates outward. Therefore, it is beneficial to model it in the polar space. At the same time, it is much easier to formulate rotation, which constitutes the main part of the alignment error between an input image and a template, in the polar space than in the Cartesian space. The symbolic string generated by concatenating points in an increasing order of radial angle in polar coordinate uniquely represents a point pattern. This reveals that the point pattern matching can be achieved with a string matching algorithm. A number of string matching algorithms have been reported in the literature 141. Here, we are interested in incorporating an elastic criteria into a string matching algorithm. Generally, string matching can be thought of as the maximization/minimization of a certain cost function such as the edit distance. Intuitively, including an elastic term in
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 19, NO. 4, APRIL 1997'
images. Fig. 12 shows the results of applying the matching algorithm to an input minutia set and a template.
,(m, n)
Templateminutia
ar
- rf +
e+
- if
< Ae(m, n) < eh(m, n) (19)
,lnui'
A <
otherwise
Fig. 11. Bounding box and its adjustment.
8,(m, n) < (ri,;-Qn

< Eh(m,n)
(r" Ae
Qrf
E,(m, n) < Ae < eh(m. n)
(20)
(Ae.)
AO < C
(a)
(b)
otherwise
5,(m + 1, + 1) = S,(m, n) + rlra n

6
(21)
(22) (23)
-r'
h(m+ 1,n+ 1) =
h(m, n) + rAr
El(m+ 1,n + I) = E(m, n) +nAe

eh(m+ 1, n+ 1) =Eh(m,n) +
(c)
(d)
/e
(24)
where w(m, n) represents the penalty for matching a pair of minutiae (r ,eem, ) and (rQ, eQ, 9Q) T , (m, n), Sh(m, n),
Fig. 12. Results of applying the matching algorithm to an input minutia set and a template. (a) Input minutia set. (b) Template minutia set. (c)Alignment result based on the minutiae marked with green circles. (d) Matching result where template minutiae and their correspondences are connected by green lines. 4 EXPERIMENTAL RESULTS
el(m, n), and Eh(m, n) specify the adaptive bounding box in the polar coordinate system (radius and radial angle); and rj is the learning rate. This elastic string matching algorithm has a number of parameters which are critical to its performance. We have empirically determined the values of 8 these parameters as follows: 61(0, 0) = -8; .h(0,. 0) = +8; Ej(0, 0)= -7.5: eh(0, 0) = +7.5; e = 30; a = 1.0; /3 = 2.0;: y = 0.1; S= 200(a + / + 7); r1= 0.5. The values of 61(0, 0), 6h(0, 0),
F;(0, 0). and eh(0,. 0) depend on the resolution of fingerprint
We have tested our on-line fingerprint verification system on two sets of fingerprint images captured with two different inkless fingerprint scanners. Set I contains 10 images per finger from 18 individuals for a total of 180 fingerprint images, which were captured with a scanner manufactured by Identix. The size of these images is 380 x 380. Set 2 contains 10 images per finger from 61 individuals for a total of 610 fingerprint images, which were captured with a scanner manufactured by Digital Biometrics. The size of these images is 640 x 480. When these fingerprint images were. cap-
311
tured, no restrictions on the position and orientation of fingers were imposed. The captured fingerprint images vary in quality. Figs. 13 and 14 show some of the fingerprint images in our database. Approximately 90 percent of the fingerprint images in our database are of reasonable quality similar to those shown in Figs. 13 and 14, while about 10 percent of the fingerprint images in our database are not of good quality (Fig. 15). which are mainly due to large creases and smudges In ridges and dryness of the impressed finger. First, we report some initial results on fingerprint matching, followed by fingerprint verification. The reasons why we did not use NIST-9 fingerprint database 1251.to test the performance of our system are as follows: we concentrate on live-scan verification, and NIST-9 fingerprint database is a very difficult fingerprint database which contains a large number of fingerprint images of poor quality and no result has been reported from other on-line verification systems for comparison.
matched fingerprint was among the nine other fingerprints of the-same individual, and incorrect otherwise. A total of 32,220 (180 x 179) matchings have been performed on test Set I and 371,490 (610 x 609) matchings on test Set 2. The distributions of correct and incorrect matching scores are shown in Fig. 16. It can be seen from this figure that there exist two peaks in the distribution of matching scores. One pronounced peak corresponds. to the incorrect matching scores which is located at a value around 10, and the other peak which resides at a value of 40 is associated with the correct matching scores. This indicates that our algorithm is capable of differentiating fingerprints at a high correct rate by setting an appropriate value of the threshold. Table 1. shows the verification rates and reject rates with different threshold values. The reject rate is defined as the percentage of correct fingerprints with their matching scores below the threshold value. As we have observed, both the incorrect matches and the high reject rates are due to fingerprint images with poor quality such as those shown in Fig. 15. We can improve these matching results by ensuring that the database does not contain such poor quality fingerprint images.
-: .
Corrt
Si
Fig. 13. Fingerprint images captured with a scanner manufactured by Identix; the size of these images is 380 x 380; all the three images are from the same individuals finger.
r.'
0
i
10
20
30
40 50 60 70 Normalized Matching Score
80
90
10
(a)Identix
Fig. 14. Fingerprint images captured with a scanner manufactured by Digital Biometrics; the size of these images is 640 x 480; all the three images are from the same individual's finger.
a0-
'i~
~~ ~
..
_. -
Correct
incorrect
' .
" L' .w
Nr
It~
~~
b W1-
-.
-t"
--
10
20
30
40
50
60
70
80
90
100
Normalized Matching Score
Fig. 15. Fingerprint images of poor quality. 4.1 Matching Each fingerprint in the test set was matched with the other fingerprints in the set. A matching was labeled correct if the
axis represents distribution of matching scores in percentage. (a) Distribution of matching scores on test set 1 (180 images). (b)Distribution of matching scores on lest set 2 (610 images).
(b) Diaital Biometrics Fig. 16. Distributions of correct and incorrect matching scores; vertical
TABLE 1 THE VERIFICATION RATES AND REJECT RATES ON TEST SETS WITH DIFFERENT THRESHOLD VALUES Threshold Verification Reject Value Rate. Rate . 20 99.839% 11.23% 22 919.947 % 13.33 % 24 99.984 % 16.48 % 26 99.994 % 20.49 % 28 99.996 % 25.19 % 30 100 % 27.72 % (a) Using Identix system (180 images). Threshold Value 20
22
Verification Rate 99.426 %

99.863%
Reject Rate 11.23 %

14.55%
24
26
99.899%
99.969 %
16.78 %
20.20 %
28 30
99.989 % *99.999 %
23.15% 27.45 %
(b) Using Digital Biometrics system (610 images).
TABLE 2
MATCHING RATES ON TEST SETS USING THE LEAVE-ONE-OUT METHOD
Number of Matching Rate Best Matches 1 91.17% 2 94.72 % 3 96.89 % 4 98.17% 5 98.89 % 6 99.39 % 7 99.72 % 8 99.83 % 9' 99.94 % (a)Using Identix system (180 images).
Number of Matching Rate Best Matches 1 92.13% 2 94.40 % 3 97.06 % 4 97.67 % 5 98.44 % 6 99.11% 7 99.70% ' 8 99.79 % 9 99.91% (b)Using Digital Biometrics System (610 images).
TABLE 3 AVERAGE CPU TIME FOR MINUTIA EXTRACTION AND MATCHING ON A SPARC 20 WORKSTATION Minutia Extraction (seconds) 5.35 Minutia Matching (seconds) 2.55 Total (seconds) 7.90
4.2 Verification on a SPARC 20 workstation. It indicates that our on-line In on-line verification, a user indicates his/her identity. fingerprint verification system does meet the response time Therefore, the system matches the input fingerprint image requirement of on-line verification. only to his/her stored templates. To determine the verificaThe number of tests done on an automatic fingerprint tion accuracy of our system, we used each one of our data- identification system is never enough. Performance measbase images as an input fingerprint which needs to be veri- ures are as much a function of the algorithm as they are a fled. An input fingerprint image was matched against all function of the database used for testing. The bi 6 metrics the nine other images of the same finger. If more than one community is slow at establishing benchmarks and the ulhalf of the nine matching scores exceeded the threshold timate performance numbers of a fingerprint verification value of 25, then the input fingerprint image is said to be system are those which you find in a deployed system. from the same finger as the templates and a valid verifica- Therefore, one can carry out only a limited amount of testtion is established. With this scheme, a 100 percent verifica- ing in a. laboratory environment to show the anticipated tion rate can be achieved with a reject rate around system performance. Even in field testing, real performance 16 percent on both test sets. Again, this reject rate can be numbers are. not important-it's often the perceived perreduced by preprocessing the database to remove the formance which is crucial. stored templates of poor quality. This demonstrates that; in practice, using a k-nearest neighbor type of matching is ade-. quate for a successful verification. Table 2 shows the match- 5 CONCLUSIONS ing rate which is defined as the percentage of the correct fin- We have designed and implemented an on-line fingerprint gerprints (of the same finger) present among the best n (n = 1, verification system which operates in two stages: minutia extraction and minutia matching. A modified version of the .... 9) matches. For an on-line fingerprint verification system to be ac- minutia extraction algorithm proposed in 118) is used in our ceptable in practice, its response time needs to be within a system which is much faster and more reliable. A new hierfew seconds. Table 3 shows the CPU requirements of our archical orientation field estimation algorithm results in a system. The CPU time for one verification, including fin- smoother orientation field which greatly improves the pergerprint image acquisition, minutia extraction and minutia formance of the minutia, extraction. An alignment-based matching, is, on an average, approximately eight seconds ,elastic matching algorithm is proposed for minutia matching. This algorithm is quite fast, because it is capable of
* ,
finding the correspondences between minutia points without resorting to an exhaustive search. At the same time, this matching algorithm has a good performance, because it has the ability to adaptively compensate for the nonlinear deformations and inexact pose transformations between different fingerprints. Experimental results show that our system achieves excellent performance in a realistic operating environment. It also meets the response time requirement of on-line verification. Based on the experimental results, we observe that the matching errors in our system mainly result from incorrect minutiae extraction and inaccurate alignment. We observe that a number of factors are detrimental to the correct location of minutia. Among them, poor image quality is the most serious one. Therefore, in the future, our efforts will be focused on global image enhancement schemes. Another issue related to minutia detection is to incorporate a structural-based model in minutia detection which extracts minutiae based on their local ridge formations. For elastic matching, an important aspect is to utilize additional information (e.g., neighboring ridges) about a minutia to increase the accuracy of alignment. ACKNOWLEDGMENTS We gratefully acknowledge our useful discussions with Nalina Ratha and Shaoyun Chen, and we would like to thank Nalina Ratha and Chitra Dorai fortheir careful comments on earlier versions of this paper. REFERENCES
1151 B. Miller. "Vital Signs of Identity." IEEE Spectrum. vol. 31. no. 2,
PP. 22-30, 1994. 1161 Ranade and A Rosenfeld, "Point Pattern Matching by RelaxaA. tion," Pattern Recognition, vol. 12, no. 2, pp. 269-275, 1993. 1171 Rao. A Taxonomy for Texture Description and Identilfication. A.R. New York: Springer-Verlag, 1990. 181 N. Ratha, S. Chen, and A.K. Jain, "Adaptive Flow Orientation Based Feature Extraction in Fingerprint Images." Pattern Recognition, vol. 28, no. 11.pp. 1,657-1,672, 1995. 1191 A. Sherstinsky and R.W. Picard, "Restoration and Enhancement of. Fingerprint Images Using M-Latiice-A Novel Non-Linear Dy: namical Systemn," Proc. 12th ICPR-B, pp. 195-200, Jerusalem. 1994. 1201 D.B.G. Sherlock, D.M. Monro. and K. Millard, "Fingerprint Enhancement by Directional Fourier Filtering," lEE Proc. Vis. Image Signal Processing, vol. 141, no. 2, pp. 87-94. 1994. , 121] J.P.P. Starink and E. Backer, "Finding Point Correspondence Using Simulated Annealing," Pattern Recognition, vol. 28, no. 2, pp. 231-240. 1995. 1221 C. Stockman, S. Kopstein, and S. Benett, "Matching Images to Models for Registration and Object Detection via Clustering." IEEE Transactions on Partern Analysis and Machine Intelligence, vol. 4, no. 3, pp. 229-241, 1982. 1231 J. Ton and A.K. Jain. "Registering Landsat Images by Point Matching," IEEE Trans. Geoscience and Remote Sensing, vol. 27, no. 5, pp. 642-651, 1989. 1241 V.V. Vinod and S. Chose, "Point Matching Using Asymmetric Neural Networks," Pattern Recognition, vol. 26, no. 8, pp. 1,2071,214, 1993. 1251 C.I. Watsorn and C.L. Wilson, NIST Special Database (4, Fingerprint Database, National Institute of Standards and Technology. Mar. 1992. 126) C.L. Wilson, G.T. Gandela, and C.I. Watson, "Neural-Network Fingerprint Classification." ]J..'Artificial Neural Networks, vol. 1, no. 2, pp. 203-228, 1994. 1271 Q. Xiao and Z. Bian, "An Approach to Fingerprint Identification by Using the Attributes of Feature Lines of Fingerprint," Proc. Seventh ICPR, pp. 663-665, Paris, 1986.
IllI Ansari, M.H.Chen, and E.S.H. Hou, N.
"A Genetic Algorithm for Point Pattern Matching," Chapt. 13. B. Soucek and the IRIS Croup, eds., Dynamic. Genetic, and Chaotic Programming. New York: John Wiley & Sons, 1992. 121 P.E. Danielsson and Q.Z. Ye, "Rotation-Invariant Operators Applied to Enhancement of Fingerprints." Proc. Eighth ICPR. pp. 329333, Rome, 1988. 131 Federal Bureau of Investigation, The Science of Fingerprints: Classification and Uses. Washington. D.C.: U.S. Government Printing Office, 1984. 141 T.H. Cormen, C.E. Lelserson, and R.L. Rivest. Introduction to Algorithms. New York: McGraw-Hill, 1990. 151 D.C.D. Hung, "Enhancement and Feature Purification of Fingerprint Images," Pattern Recognition, vol. 26, no. I1. pp. 1,661-1,671, 1993. 161S. Cold and A. Rangarajan, "A Graduated Assignment Algorithm for Graph Matching," Research Report YALEU/DCS/RR-1062, Yale Univ., Dept. of Computer Science, 1995. 171 L. O'Gorman and J.V. Nickerson. "An Approach to Fingerprint Filter Design," Pattern Recognition, vol. 22, no. 1, pp. 29-38, 1989. 181 K. Karu and A.K. Jain, "Fingerprint Registration," Research Report, Michigan State Univ., Dept. of Computer Science. 1995. "Fingerprint Classification:." Pattern Recog191 K. Karu and A.K. Jain; nition, vol. 29. no. 3, pp. 389-404. 1996. 1101 Kawagoe and A. Tojo, "Fingerprint Pattern Classification," M. PatternRecognition, vol. 17. no. 3, pp. 295-303, 1984. 111) H.C. Lee and R.E. Caensslen, eds.. Advances in Fingerprint Technol. ogy. New York: Elsevier. 1991. 1121 D.P. Huttenlocher and S. Ullman, "Object Recognition Using Alignment." Proc. First Intl Conf. Computer Vision, pp. 102-111, London, 1987. 1131 Z.R. Li and D.P. Zhang, "A Fingerprint Recognition System With Micro-Computer." Proc. Sixth ICPR, pp. 939-941, Montreal, 1984. 1141 L. Coetzee and E.C. Botha, "Fingerprint Recognition in Low Quality Images." Pattern Recognition. vol. 26, no. 10, pp. 1,4411.460. 1993.
9I
'
Anil Jain received a BTech degree in 1969 from the Indian Institute of Technology, Kanpur, India, and the MS and PhD degrees in electrical engineering from the Ohio State University in 1970 and 1973, respectively. He joined the faculty of Michigan State University in 1974, where he currently holds the rank of University Distinguished Professor in the Dept. of Computer Science. Dr. Jain has made significant contributions and published a large number uof papers on the following topics: statistical pattern recognition, exploratory pattern analysis, Markov random fields, texture analysis, interpretation of range images, neural networks, document image analysis, and 3D object recognition. Several of his papers have been reprinted in edited volumes on image processing and pattern recognition. He received the best paper awards in 1987 and 1991 and certificates for outstanding contributions in 1976, 1979, and 1992 from the Pattern Recognition Society. He also received the 1996 IEEE Transactions on Neural Networks Outstanding Paper Award. Dr. Jain was the editor-in-chief of the IEEE Transactions on Pattern Analysis and Machine Intelligence (1990-1994), and he also serves as an associate editor for Pattern Recognition, Pattern Recognition Letters, IEEE Transactions on Neural Networks, Applied Intelligence, and the Journal of Mathematical Imaging and Vision. He is the coauthor of Algorithms for Clustering Data (Prentice Hall, 1988), has edited Real-Time Object Measurement and Classification (SpringerVerlag, 1988), and coedited Analysis and Interpretation of Range Images (Springer-Verlag, 1989), Markov Random Fields (Academic Press, 1992), Artificial Neural Networks and Pattern Recognition (Elsevier, 1993), and 3D Object Recognition (Elsevier, 1993). He is a Fellow of the IEEE and IAPR.
Lin Hong received the BS and MS degrees in computer science from Sichuan University, China,. in 1987 and 1990, respectively. He is currently a PhD student in the Dept. of Computer Science, Michigan State University. His current research interests include pattern recognition, image processing, biometrics, and computer graphics.
Ruud Bolle (S'82-M'84-F'96) received the bachelor's degree in analog electronics in 1977 and the master's degree in electrical engineering in 1980, both from Delft University of Technology, Delft, The Netherlands. In 1983 he received the master's degree in applied mathematics and in 1984 the PhD in electrical engineering from Brown University, Providence, Rhode Island. In 1984 he became a research staff member .in the Artificial Intellig p c ep ., a e IBM Thomas J. Watson Research p, Center. In 1988 he became manager of the newly formed Exploratory Computer Vision Group, which is part of IBM's digital library effort. His current research interests are video database indexing, video processing, and biometrics. Dr. Bolle is a Fellow of the IEEE. He is on the Advisory Council of IEEE Transactions on Pattern Analysis and Machine Intelligence, and he is associate editor of Computer Vision and Image Understanding. He is the guest editor of a special issue on computer vision applications for networkcentric computers in Computer Vision and Image Understanding.
ence Grou
Com uter S ience D% t
t thI
, IEEE TRANSACTIONS'ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.
Application No. I1/231,353

Docket No. 577832000200
z..
,v.
,,,
,,v
,,
On Combining Classifiers
Josef Kittler, Member, IEEE Computer Society, Mohamrnad Hatef, Robert P.W. Duin, and Jiri Matas
Abstract-We develop a common theoretical tramework'for combining classifiers which use distinct pattern representations and show that many existing schemes can be considered as special cases of compound classification where all the pattern representations are used jointly to make, a decision. An experimental comparison of various classifier combination schemes demonstrates that the combination rule developed under the most restrictive assumptions-the sum rule-outperforms other classifier combinations schemes. Asensitivity analysis o the various schemes to estimation errors is carried out to show that this finding can be justified theoretically. Index Terms-Classification, classifier combination, error sensitivity.
INTRODUCTION
HE ultimate goal of designing pattern recognition sys-
tems is to achieve the best possible classification performance for the task at hand. This objective traditionally led to the development of different classification schemes for any pattern recognition problem to be solved. The resuits of an experimental assessment of the different designs would then be the basis for choosing one of the classifiers as a final solution to the problem. It had been observed in such design studies, that although one of the designs would yield the best performance, the sets of patterns misclassified-by the different classifiers would not necessarily overlap. This suggested that different classifier designs potentially offered complementary information about the patterns to be classified which could be harnessed to improve the performance of the selected classifier.... These observations motivated the relatively recent interest in combining classifiers. The idea is not to rely on a single decision making scheme. Instead, all the designs, or their subset, are used for decision making by combining their individual opinions to derive a consensus decision.' Various classifier combination schemes have been devised and it has been experimentally demonstrated that some of them consistently outperform a single best classifier. However, there is presently inadequate understanding why' some combination schemes are better than others and in, what circumstances. The two main reasons for combining classifiers are efficiency and accuracy. To increase efficiency one can adopt multistage combination rules whereby objects are classified by a simple classifier using a small set of cheap features in * 1.K tiler and J.Matas are w
th the Centre for Vs on, Speech, and S gnal Process ng, School of Electron c Eng neer ng, Informat on Technology, and Mathemat cs, Un vers ty of Surrey, Gu ldford GU2 5XH, Un led K ngdom. E-ma 1: tller@ee.surrey.ac.uk. j.k * M. Hatef s w th ERA Technology.Ltd., Cleeve Road, Leatherhead KT22. 7SA, Un ted K ngdom. E-ma I:mhate@ee.surrey.ac.uk. * R.P.W. Du n s w ththe Department of Appl ed Phys cs, Delft Un vers ty of Technology, Lorentzweg 1, 2628 C] Delft, The Netherlands. E-ma 1:bob@ph.tn.tudelfl.nl.
Manuscr plrece ved 17 June 1996; rev sed 16 Jan. 1998. Recommendedfor acceptance by 1.1. Hull.
For nfornal on on obta n ng repr nis of th sart cle, please send e-ma I to: tpam @compuler.org, and reference IEEECS Log Number 106327...
combination with a reject option. For the more difficult objects more complex procedures, possibly based on different features, are used (sequential or pipelined [17], [7], or hierarchical [24], [16]). Other studies in the gradual reduction of the set of possible classes are [8], [6], [14], [21]. The combination of ensembles of neural networks (based on different initialisations), has been studied in the neural network literature, e.g., [11], [4], 15], [10], [15], [18]. An important issue in combining classifiers is that this is particularly useful if they are different, see [1). This can be achieved by using different feature sets [23], [13) as well as by different training sets, randomly selected 112], [22] or based on a cluster analysis [3]. A possible application of a multistage classifier is that it may stabilize the training of classifiers based on a small sample size, e.g.; by the use of bootstrapping [27], [19]. Variance reduction is studied in [30], [31] in the context of a multiple discriminant function classifier and in [35] for multiple probabilistic classifiers. Classifier combination strategies may reflect the local competence of individual experts as exemplified in [32) or the training process may aim to encourage some experts to achieve local decision making superiority as in the boosting' method of Freund [28) and Shapire [29]. An interesting issue in the research concerning classifier ensembles is the way they are combined. If only labels are available a majority vote [14], 19] is used. Sometimes the use can be made of a label ranking [2), [13]. If continuous outputs like posteriori probabilities are supplied, an average or some other linear combination have been.suggested [11), [23], [25], [33); It depends on the nature of the input classifiers and the feature space whether this can be theoretically justified. An' interesting study on these possibilities is given in [10), [26], [34]. If the classifier outputs are interpreted as fuzzy membership values, belief values or evidence, fuzzy rules [4), [5], belief functions and Dempster-Shafer techniques [9], [18), [20], [23] are used. Finally it is possible to train the output classifier separately using the outputs of the input classifiers as new features [15], [22], [36). From the point of view of their analysis, there aire basically two classifier combination scenarios. In the first scenario, all the 'classifiers use the same representation of the input pattern. A typical example of this category is a set of k-nearest
0162-8828/98/510.000 1998 IEEE
'
KITTLER ET AL.: ON COMBINING CLASSIFIERS
227'
neighbor classifiers, each using the same measurement vector, but. different classifier parameters (number of nearest neighbors k, or distance metrics used for determining the nearest neighbors). Another example is a set of designs based on a neural network classifier of fixed architecture but having distinct sets of weights which have been obtained by means of different training strategies. In this case, each classifier, for a given input pattern, can be considered to produce an estimate of the same a posteriori class probability. In the second scenario, each classifier uses its own representation of.the input pattern. In other words, the measurements extracted from the pattern are unique to each-classifier. An important application of combining classifiers in this scenario is the possibility to integrate physically different types of measurements/features. In this case, it is no longer possible to consider the computed a posteriori probabilities to be estimates of the same functional value, as the classification systems operate in different measurement spaces. In this paper, we focus on classifier combination in the second scenario. We develop a common theoretical framework for classifier combination and show that many existing schemes, can be considered as special cases of com-pound classification where all the representations are used jointly to make a decision. We demonstrate thatunder different assumptions and using different approximations we can derive the commonly used classifier combination schemes such as the product rule, sum rule, min rule, max rule, median rule, and majority voting. The various classifier combilation ;schemes are then compared experimentally. A surprising outcome of the comparative study is that the combination rule developed under the most restrictive assumptions-the sum rule-outperforms other classifier combinations schemes. To explain this empirical finding, we investigate the sensitivity of various schemes to estimation errors. The sensitivity analysis shows that the sum rule is most resilient to estimation errors. In summary, the contribution of the paper is twofold. First of all, we provide a theoretical underpinning-of many existing classifier combination schemes for fusing the decisions of multiple experts, each employing,a distinct pattern representation. Furthermore, our analysis of the sensitivity of these schemes to estimation errors enhances the understanding of their properties. As a byproduct, we also offer a methodological machinery which can be used for developing other classifier combination strategies and for predicting their behavior. However, it cannot be overemphasized that the problem of classifier combination is very complex and that there are many issues begging explanation. These include the effect of individual expert error distributions on the choice of a combination strategy, explicit differentiation between decision ambiguity competence and confidence, and the relationship between dimensionality reduction and multiple expert fusion, with its implicit dimensionality expansion. Also, many practical decision making schemes are very complex, of sequential kind, with special rules to handle rejects and exceptions and it is currently difficult to envisage how the results of this paper could be made to bear on the design of such schemes. The theoretical framework and analysis presented is only a small step towards a considerably improved understanding of classifier combina-
tion which will be needed in order to harness the benefits
of multiple expert fusion to their full potential.

The paper is organized as follows. In Section 2, we formulate the classifier combination problem and introduce the necessary notation. In this section, we also derive the basic classifier combination schemes: the product rule and
the sum rule. These two basic schemes are then developed
into other classifier combination strategies in Section 3. The combination rules derived- in Sections 2 and 3 are experi-
mentally compared in Sections 4 and 5. Section 6 investigates the sensitivity of the basic classifier combination rules to estimation errors. Finally, Section 7 summarizes the main results of the paper and offers concluding remarks.
2 THEORETICAL FRAMEWORK
Consider a pattern recognition problem where pattern Z is possible classes (01,..., em). to be assigned to one of the mn Let us.assume that we have R classifiers each representing the given pattern by a distinct measurement vector. Denote the measurement vector used by the th classifier by x. .In the measurement space each class sk is modeled by the probability density function. p(x Ik) and its a priori probability of occurrence is denoted P(wk). We shall consider the models to be mutually exclusive which means that only one model can be associated with each pattern. Now, according to the Bayesian theory, given measurements x, = 1, ..., R, the pattern, Z, should be assigned to class ca provided the a posteriori probability of that interpretation is maximum, i.e.
ass gn
.Z - a
P(Wix, ..., xR)=maxP(wkIxp...,xR)

The Bayesian decision rule (1) states that in order to utilize all the available information correctly to reach a decision, it is essential to compute the probabilities of the various hypotheses by considering all the measurements simultaneously. This is, of course, a correct statement of the classification problem but it may not be a practicable proposition. The computation of the a posteriori probability functions would depend on the knowledge of high-order measurement statistics described in terms of joint probability density- functions p(xl,-.. . xR.k) which would be difficult to infer. We shall therefore attempt to simplify the above rule and express it in terms of decision support computations performed by the individual classifiers, each exploiting only the information conveyed by vector x. We shall see that this will not only make rule (1) computationally manageable, but also it will lead to combination rules which are commonly used in practice., Moreover, this approach will provide a scope for the development of a range of efficient classifier combination strategies. We shall commence from rule .(1) and consider how it can be expressed under certain assumptions. Let us rewrite the a posteriori probability P(wkx ... , xR)' using the Bayes theorem. We have
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE'INTELLIGENCE, VOL. 20, NO. 3, MARCH 1998
P(W,
x1,...,
xR)- =
p(
1 ....
)P(Co) .l' p(x,... ,xR)
(2)
where p(x1, ..... , xa) is the unconditional me'asurement joint probability density. The latter can be expressed in terms of the conditional measurement distributions as
m
classifiers will not deviate dramatically from the prior probabilities. This is a rather strong assumption but it may be readily satisfied when the available observational discriminatory information is highly ambiguous due to high levels of noise. In such a situation we can assume that the a posteriori probabilities can be expressed as
p(x 1,..., XR)
p(x,...
R. )P( )
P(Okl )= P(Wk)(l
6
6k )
and therefore, in the following, we can concentrate only on the numerator terms of (2). 2.1 Product Rule As already pointed out, p(x 1,..., XRt(ck) represents the joint probability distribution of the measurements extracted by the classifiers. Let us assume that the representations used are conditionally statistically independent. The use of different representations may be a probable cause of such independence in special cases. We will investigate the consequences of.this assumption and write p(X1,...,XRI
ok) =
where bk satisfies k << 1. Substituting (8) for the a posteriori probabilities in (7), we find
P(R-I)(1-I
-1
P(WI )-
P(-)-(I
=1
+
'
(9)
If we expand the product and neglect any terms of second and higher order, we can approximate the right-hand side of (9) as
P(,
)
=1
(1
k)=
P(k +P(wk,) ,)
=1
S,
(10)
Substituting (10) and (8) into (7), we obtain a sum decision rule
l P(X Ialk)
ass gn
Z
R
where p(x I qy) is the measurement process model of the th representation. Substituting from (4) and (3) into (2) we find
=1
P(x,... )) P(
max (1- R)P(k) +
P()lx)
=1
(11)
p(x 1
, )
k=1
and using (5) in (1), we obtain the decision rule

ass gn
Z - oi
P(W,)
(o x
=1
)= max P(Wk)
k=I
=1
p(x IK)
or in terms of the a posteriori probabilities yielded respective classifiers ass gn
2.3 Comments Before proceeding, in the next section, to develop specific classifier combination strategies based on decision rules (7) and (11), let us pause to elaborate on the assumptions made to derive the product and sumr rules. We concede that the conditional independence assumption may be deemed to be unrealistic in many situations. However, three important points should be borne in mind before dismissing the resuits of the rest of the paper: r * For some applications, the conditional independence assumption will hold. * For many applications, this assumption will provide an adequate and workable approximation of the reality which may be more complex. One could draw a parallel here between the Gaussian assumption frequently made even in situations where the class distributions patently do not obey the exponential law but still this simplification yields acceptable results. * Finally, and perhaps most importantly, we shall see in the next section that all the derived classifier combination schemes based on this assumption are routinely used in practice. The analysis presented in the paper therefore provides a plausible theoretical underpinning of these combination rules and ,thereby draws attention to the underlying assumptions behind these schemes which the users may not be aware of. As far as the sum rule is concerned, the assumption that the posterior class probabilities do not deviate greatly from the priors will be unrealistic in most applications. When
Z -- o
f )(7)
=1
P-(R-)( w)f P(wjIx) = max P-(R-i)(w)kI P(oklx

=1
The decision rule (7) quantifies the likelihood of a hypothesis by combining the a posteriori pr6babilities generated by the individual classifiers by means of a product rule. It is, effectively a severe rule of fusing the classifier outputs as it is sufficient for a single recognition engine to inhibit a par'ticular interpretation by outputting a close to zero probability for it. As we shall see below,. this has a rather undesirable implication on the decision rule combination as all the classifiers, in the worst case, will have to provide their respective opinions for a hypothesized class identity to be accepted or rejected. 2.2 Sum Rule Let us consider decision rule (7) in more detail. In some applications it may be appropriate further to assume that the a posteriori probabilities computed by the respective
229
observations x, =, 1, ..., R on a pattern convey significa discriminatory information the sum approximation of tl product in (10) will introduce gross approximation errol However, we shall show, in Section 6 that the injection these errors will be compensated by a relatively low sen: tivity of the approximation to estimation errors. '
ass gn
Z'. m k=l
i
=1.
min P(
=
x)= max min P(k x )
(17)
3 CLASSIFIER COMBINATION STRATEGIES

The decision rules (7) and (11) constitute the basic schem for classifier combination. Interestingly, many common used classifier combination strategies can be develop from these rules by noting that
S3.3 Median Rule Note that under the equal prior assumption, the sum rule in (11) can be viewed to be computing the average a posteriori probability for each class over all the classifier outputs, i.e.,
ass gn
SP(ox =1
Z-
4
P(wk
=1
) = ma
(18)
fi
k) x
RR
RR
-<
=1
P(ox
) max P(wkx)
(12)
The relationship (12) suggests that the product and sum combination rules can be approximated by the above upper or lower bounds, as appropriate. Furthermore, the hardening of the a posteriori probabilities P(coklx ) to produce binary valued functions Ak as
Thus, the rule assigns a pattern to that class the average a posteriori probability of which is maximum. If any of the classifiers outputs an a posteriori probability for some class which is an outlier, it will affect the average and this in turn could lead to an.incorrect decision. It is well known that a robust estimate of the mean is the median. It could therefore be more appropriate to base the combined decision on the median of the a posteriori probabilities. This then leads to the following rule: ass gn
R
Z'-w i
m R
k=1 =1
ff
(19)
Ak
if P(
0 otherwise
= m1x P@(x
med P(.x ) = max med P(COwkx )
(13)
.1
3.4 Majority Vote Rule

Starting from (11) under the assumption of equal priors and by hardening the probabilities according to (13), we find ass gn
R'
ml J
results in combining decision outcomes rather than combining a posteriori probabilities. These approximations lead to the following rules: 3.1 Max Rule Starting from (11) and approximating the sum by the maximum of the posterior probabilities, we obtain
-*
(1)
(20)'
k=1=1
ass gn
(1- R)P(wm) + R max P(w x

m a "
)=
(
max (1- R)P(wk) + R max P(aklx)
Note that for each class 0 k the sum on the right hand side of (20) simply counts the votes received for this hypothesis from the individual classifiers. The class which receives the largest number of votes is then selected as the consensus (majority) decision. All the above combination schemes and their relationships are represented in Fig. 5. 4 EXPERIMENTAL COMPARISON OF CLASSIFIER COMBINATION RULES: IDENTITY VERIFICATION
which under the assumption of equal priors simplifies to
ass gn
max P(w Ix =1
Z :W
max maxP(W x) k=1 =1 (15)
3.2 Min Rule. Starting from (7) and bounding the product of posterior probabilities from above we obtain ass gn
P< <a
Z -+ c )min
P@ix)
max P-(-1)()
k=)
mn P(wkIx )
=1
which under the assumption of equal priors simplifies to
The first experiment is concerned with the problem of personal identity verification. Three different sensing modalities of biometric information are used.to check the claimed identity of an individual: frontal face, face profile, and voice. The verification methods using these biometric sensing modalities have been developed as part of the European Union project in Advance Communication Technologies and Services M2VTS as described in [411, [441, 143]. The design of the verification modules and their performance testing gas been carried out using the M2VTS database 142] made up of about eight seconds of speech and video data for 37 clients taken five times (five shots) over a period of one month. The image resolution is 286 x 350 pixels.
230
IEEE TRANSACTIONS ON PA-TE RNANALYSIS AND MACHINE INTELLIGENCE, VOL. 20, NO. 3, MARCH 1998
4.1 Frontal Face The face verification system used in the experiments is described in detail in [41]. It is based on robust correlation of a frontal face image of the client and the stored face template corresponding to the claimed identity. A search for the optimum correlation is performed in the space of all valid geometric and photometric transformations of the input image to obtain the best possible match with respect to the template. The geometric transformation includes translation, rotation and scaling, whereas the photometric transformation corrects for a change of the mean level of illumi-. nation. The search technique for the optimal transformation parameters is based on random exponential perturbations. Accordingly, at each stage the transformation between the test and reference images is perturbed by: a random vector drawn from an exponential distribution and' the change is accepted if it leads to an improvement of a matching criterion. Computational efficiency is achieved by means of random sampling based on Sobel sequences which, allow faster convergence as compared to uniform sampling. The score function adopted rewards a large overlap between the transformed face image and the template, and the similarity, of the intensity distributions of the two images. The degree of similarity is measured with a robust kernel. This ensures that gross errors due to, for instance, hair style changes do not swamp the cumulative error between the matched images. In other words, the matching is benevolent, aiming to find as large areas of theface as possible supporting a close agreement between the respective gray-level profiles of the two images. The gross errors will be reflected in a reduced overlap between the two frames which is taken into account in the overall matching criterion. The system is trained very easily by means of storing one or more templates for each client. Each reference image is segmented to create a face mask which excludes the background and the torso as these are likely to change over time. The testing is performed on an independent test data composed of 37 clients and 37 x 36 impostors. 4.2 Face Profile, The verification approach involves a comparison of a candidate profile with the template profile of the claimed identity. The candidate image profile is extracted from the face profile images by means of color-based segmentation. The similarity of the two profiles is measured using the Chamfer distance computed sequentially [44]. The efficiency of the verification process is aided by precomputing a distance map for each reference profile. The map stores the distance of each pixel in the face profile image to the nearest point on the profile. As the candidate profile can be subject to translation, rotation and scaling, the objective of. the matching stage is to compensate for such geometric transformation. The parameters of the compensating transformation are determined by minimizing the chamfer distance between the template and the transformed candidate profile. The optimization is carried out using a simplex algorithm which requires only the distance function evaluation and no derivatives. The convergence of the simplex algorithm to a local minimum is prevented by a careful initialization of the transformation parameters. The translation
parameters are estimated by comparing the position of the nose tip in the two matched profile. The scale factor is derived from the comparison of the profile heights and the rotation is initially set to zero. Once the optimal set of transformation parameters is determined, the user is accepted or rejected depending on the relationship of the minimal chamfer distance to a prespecified threshold. The system is trained on the first three shots. One profile per client per shot is stored in the training set. From the three profiles for each-client a single reference profile is selected by pairwise comparison of the profile images. The profile yielding the lowest matching distance to the other two images is considered as the best representative of the triplet. The trained system is tested on Shot 4 profiles' As there are 37 users in the M2VTS database the testing involves 37 correct authentication matches and 37 x 36 imposter tests. The acceptance threshold is selected from the Receiver Operating Characteristic so as to produce equal error rate (false rejection and false acceptance). 4.3 Voice The personal identity verification based on voice.employs a text dependent approach described in [43). It is assumed that the audio input signal representing the uttered sequence of digits from zero to nine can be segmented into individual words. Both, the segmentation of the speech data and the claimed identity verification is accomplished using speech and speaker recognition methods based on Hidden Markov Models. The audio signal is first transformed into a multivariate time series of linear predictive cepstral coefficients: During training, digit HMMs are trained using segmented speech data from three shots of the M2VTS database. The digit models have the same structure, with the number of states being digit specific. The models allocate one state per phoneme and one state per transition between phonemes. A single Gaussian mixture is used to model the distribution of the cepstral coefficient vectors within one state. STwo models are acquired for each digit: the client model, and the world model. The latter, which is common to all users, captures the variability of the uttered sound in a large database. The verification of a claimei identity is based on a score computed as the sum over the individual digits of the log likelihood ratio of the claimed model and the world model normalized by the number of cepstral coefficient frames. The score is mapped on the interval zeroone using a sigmoid function. The performance is assessed using an independent test set. 4.4 Experimental Results The equal error rates obtained using the individual sensing modalities are shown in Table 1.The table shows that the lowest rate of 1.4 percent was achieved using voice based verification. The face profile verification' produced an equal error rate of 8.5 percent whereas the frontal face method yielded 12.2 percent. The soft decisions output by the three verification systems were then combined using the various classifier combination strategies discussed in Section 3. The validity of the conditional independence assumption was tested by computing the average within class corre-
BEST AVAILABLE OPY

and jth component of the vector of deviations. This normalisation process produced average within class correlations taking values in the interval [-1, 1]. For display purposes, we have taken the absolute value of these correlation coefficients. The result of this representation of variable correlations is a matrix with all elements on the diagonal equal to unity (displayed as gray level 255) and the
TABLE 1
EQUAL ERROR RATES method frontal profile speech sum product maximum median minimum EER (%) 12.2 8.5 1.4 0.7 1.4 12.2 1.2 4.5
strength of correlation between one and zero mapped onto the gray-level scale 255/to 0. The correlation matrix is
shown in Fig. 1; The correlation matrix exhibits a block diagonal structure, which suggests that the observations generated by each modality are class conditionally dependent. The cor-
Too
soo
soo
ii iK E r
tt
I
I
?.
Srelation are particularly strong between the features of the face profiles and similarly between those of the speech utterances. They are weaker for the features of the face image. Owing to the random spatial sampling of the face image, the spatial ordering of the successive features is
destroyed and consequently the correlation matrix block corresponding to the facial data has a random structure (with the exception of the diagonal elements). Note that
soo
100
I,
I i
1
the correlations between features from different modalities are considerably weaker than within modality correlations. This applies in particular to the correlations between the frontal face and the other two modalities. There is a small subset of the face profile variables for which the correlations are not insignificant but on the whole the conditional independence assumption may be considered to hold.
1000
100 200 300 400 500 600
~aS~t~j~bi~~ e4~P~W~I~E~d~ ~B~FI~1~Z1

700 800 900 1000
Fig. 1. Correlation of face profile; frontal face, and speech data.
Next, the three biometric modalities were combined using the fusion strategies discussed in Section 3. The results
presented in Table 1 show the benefits of classifier combination. It is interesting to note that the sum rule outper-
lation matrix for the data used in decision making. Since the overall dimensionality of the data exceeds tens of thousands, it is impossible to present a full view of the correlations between the measurements of the respective modalities. However, by adopting a visual representation of the correlation matrix, we will be able to look at the correlations at least in a representative subspace of this highly dimensional feature space. This subspace was created by taking as features 500 samples of the face image gray levels taken at prespecified spatial locations..Each profile image was represented by 60 sample points evenly distributed along the profile. The sampling is registered with respect to the tip of the nose and the sampling interval normalized by the nose length. The profile landmarks needed for the registratiori and normalization can be easily detected. For the speech data, we took the first 100 frames from each of the first five cepstral coefficients. The utterances for each client were first time warped using a client specific template. This created a client representation subspace of 1,060 dimensions. In particular, the face profile variables occupy the first 60 dimensions, followed by 500 frontal face image samples, and finally 5 x 100 speech measurements. The average within class correlation matrix was computed by removing the class conditional mean of each variable. The resulting vectors of deviations from the means were used to compute the elements of the average within class covariance matrix. These were then normalized by dividing each jih element by the product of stanidard deviations of the th
formed all the other combination strategies and also the individually best expert. 5
EXPERIMENTAL COMPARISON OF CLASSIFIER COMBINATION RULES: HANDWRITTEN DIGIT RECOGNITION
As a second domain to assess the classifier combination strategies, we used the problem of handwritten character recognition. The task is to recognize totally unconstrained handwritten numerals. Samples are images of isolated numeric chaiacters takeh from addresses on the letter envelopes provides by the U.S Postal Service. The database used is the CEDAR-CDROM produced by the Center of Excellence for Document Analysis and Recognition, at the State University of.New York, Buffalo. Images are scanned from dead-letter envelopes provided by the U.S. Postal Service. We used the BR and BS sets of the database that consist of bitonal isolated images of numeric characters. BR set contains 18,468 samples and is used as a training set while BS set (2,213 samples) served as a test set. Four types of classifiers are first applied to perform the classification individually. We used structural [38], Gaussian, Neural Network, and Hidden Markov Model classifiers [40].
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 20, NO. 3, MARCH 1998
5.1 Character Representation Four different representations are used as follows: 1) Pixel-level representation: in the Gaussian classifieri case the bitonal image of each numeric character is scaled into 10 x 10 gray-level image. The character is thus represented by a 100-dimensional vector in which each dimension is the gray level of the corresponding pixel. 2) Complex object representation: this is used in the case of the structural classifier. The bitonal image is first skeletonized using some thinning process. The skeleton of the character is then decomposed into a number of primitives. Each primitive being either a line, curve or a loop is parameterized using a number of unary measurements such as the size, direction, etc. In addition, a number of binary measurements are extracted to describe the geometrical relations between each primitive and its neighbors. A more detailed description of this representation is presented in [39]. 3) In the HMM classifier, the 2D image isrepresented as two ID signals by using the horizontal and vertical profiles. The profile consists of the binary pattern of the image across the horizontal/vertical line. This pattern is quantized into 16 vectors in the codebook. Each pattern is therefore given the index of the closest vector in the codebook. Further, the center of gravity of each line in the profile is calculated and also quantized (to 13 levels). The feature space thus consists of two indices one for the pixel pattern and the. other for the center of gravity. More details on this representation can be found in [40]. 4) The pixel representation in Item I is used as a starting point to derive a distinct character description by the hidden layer of the neural network employed as one of the classifiers. 5.2 Classification 5.2.1 Structural Classifier. Handwritten characters have natural structures as they are generally composed of number of smaller, elements with certain topological relations. To recognize a character, we need to identify its 'basic .primitives and the particular structural relations between them. The. binary image is first skeletonized, then decomposed into number of primitives where junctions and reflection points serve as breaking points. Both symbolic and numeric attributes are used to describe the structure of the character. Firstly, primitives are categorized into one of three types using a discretizing criterion: zero-line, one-curve, or two-loop. The connectivity between the primitives is encoded to reflect the topological structure of the character. The character code consists of the code of each primitive which in turn consists of the type of the primitive, the number of the neighbors on the first end point and their types, and the number of the neighbors on the second endpoint and their types. For example the code: (1, 200, 0), (2, 10, 10), (0, 210, 12), (0, 210, 0)
Fig. 2. The prototype tree structure. represents a character consisting of four primitives. The. first primitive is a curve (1) connected to two primitives in the first end point, both of them being lines (200). The other endpoint is not connected to any primitive (0)., Numeric information is also used to characterize unary attributes of primitives and relations-and binary relations between primitives. The length of the primitive, its direction, the degree of curvature are some of the unary measurements used. Example of the binary measurements used are the direction of the line connecting the centers of the primitive and its neighbor as well as the direction of the line connecting the terminal point of the primitive and the center.of the neighbor. Each class is represented by one or more prototypes. In our scheme prptotypes are generated from the training samples. Samples of 'each class are divided into small groups by means of levels of clustering. The first is to group all samples with, the same number of primitives in a cluster.'Each cluster is called Np-Group and is further divided according to the types of the primitives. For example, samples that consist of a curve and two lines are grouped together. Each such group or cluster is called type-Group and further divided into a number of clusters each containing samples that have the same structural code. Cluster in this level is called code-Group. Finally, each code-Group is further divided' using the dynamic clustering algorithm [37) where each of the clusters produced is called dist-Group. The mean, variance and the actual range around the mean are calculated for each of the unary and binary measurements to characterize the particular cluster. The prototypes of all classes are saved in this multilevel tree structure (Fig. 2). 5.2.2 The Classification Scheme An unknown sample is tested first at the root of the prototype tree to decide the right Np-group. In the next level, it is checked to select the right type-Group and eventually it reaches the appropriate code-Group. If no code-Group is found, the class of the sample is reported as unknown. Otherwise, the sample will be checked against all prototypes in
233
the code-Group to find the closest candidate(s) to the sample. First, the probabilistic relaxation algorithm [38) is used to find the correspondence between the primitives in the sample and those in the prototype. Then, a distance measure is used to quantify the similarity between the sample and each candidate. It is pertinent to point out that a meaningful measure can be defined because each sample is compared only to prototypes that have the same number of primitives as well as connectivity (adjacency matrix). This means that they have the same number of dimensions. Moreover, after finding the correspondence between the primitives in the sample and the prototype through the matching process, the attribute vectors associated with the sample and prototypes respectively can be considered as belonging to the same space. This facilitates the task of finding a metric for measuring the distance between them. We used the Euclidean distance first, but due to the fact that this distance does not take into account second-order statistics of each measurement, the results were not satisfactory. On the other hand, using distance measure that exploits second order statistics, such as the Mahalanobis distance, requires a large number of samples in each cluster. Due to the large variability in the structure of the handwritten characters there are prototypes that contain only a few samples in the training set which makes the estimate of these statistics unsatisfactory. Consequently, we chose a modified Euclidean distance whereby the difference in each dimension between the measurement vector in the sample and that in prototype is penalized if it exceeds a certain value. The value is chosen to be some weight multiplied by the standard deviation of that. particular measurement. The modified distance therefore is:
dine =
where m is the mean vector and E is the covariance matrix of class . They are estimated in the training phase from the training data set. d is the number of dimensions in the feature space. The a posteriori probability is then calculated:
P(a jx)
p(xl- )P(w)
P(
k)P(wk)
(25)
5.2.4 Hidden Markov Models Classifier Hidden Markov Models (HMMs), a popular method of statistical representation in speech processing is based on the representation of an object as a random process that generates a sequence of states. The model consists of a number of states with their probabilities as well as probabilities associated with the transition from one state to another. The character in this classifier is scanned vertically and horizontally to generate the- corresponding vertical and horizontal profiles. The vertical profile consists of the rows in.the binary image while the horizontal profile consists of the columns. Each state represents ,a pattern of binary pixels in each line along the profile. The number of possible patterns (states) can be numerous. For example, in a 32 x 32 32 , binary image there are 2 possible combinations. To reduce the number of possible states, the training patterns are clustered and the centroid of each cluster serves as a reference vector in a code book (Vector Quantization). An unknown sample is compared to each reference in the codebook and assigned the'index of the closest one. The codebook is generated using the k-means clustering algorithm with k = 16, resulting in a 16-vector codebook. In the clustering process some distance measure is required to determine how close a sample is to its cluster in order to decide that it should be kept in the cluster or moved to another (closer) one. Hamming distance is a natural choice when dealing with binary vectors. The Hamming distance,'however, is known to be sensitive to the shift between two binary patterns. Slight shifts are inevitable in a problem like character recognition. A Shift Invariant Hamming distance (the minimum Hamming Distance between two discrete vectors when they are allowed to slide on each other) is used. The same advantageous property of shift invariance can be undesirable in some cases. For example, the profile of letter "q" 'and "d" would appear to have the same codebook index: Therefore, another measure is used to distinguish between such instances. The center of gravity of line is calculated and then subtracted from the running average of the last three lines. The relative center of gravity is in turn quantized to 13 levels. The state representation is thus reduced to a pair of numbers-one represents the pixel pattern index and the other is the relative center of gravity. The discrete hidden Markov models are generated using the Baum;Welch reestimation procedure while a scoring mechanism based on the Viterbi algorithm is used in the test phase. The scoring result reflects the likelihood of the sample to be generated by the class model. These score values are used as the soft-level assignment of the classifier (as posteriori probabilities estimates).
YFx
-m
)2
(21)
where: F(y) =
cII if Iy>Oa(22) otherwise 1y
(22)
and m is the mean of the th feature while a is its standard deviation. O is the threshold constant. K is a penalizing weight. The values of 0 and Kare selected experimentally. An estimate of the a priori probability is then computed as follows:
e
- da e2
= P( Ix)
Pk
P(. 2
()
"
(23)
x-dme
P(k )
where P(wo) is the a posteriori probability estimated from the number of samples in each cluster that generated the prototype. The sample is then assigned the class that has the maximum P(41 x). Note that when no prototype matches the sample structure it is assigned zero a posteriori probability for all classes. 5.2.3 Gaussian Classifier The classes in the feature space are assumed to possess a normal distribution:
p(x2
)= (2)I
e-(x-m )T E'(x-m )
(24)
TABLE 2
THE CLASSIFICATION RATE FOR EACH CLASSIFIER
Individual classifier Structural: Gaussian: Neural Net:

HMM:
Classification rate % 90.85 93.93 93.2

94.77
d 6
(1) 0->8 (2) 0->8 (3) 0->8
/ /
(4) I->7 (5) I->7
TABLE 3
THE CLASSIFICATION RATE USING
DIFFERENT COMBINING SCHEMES
44st, oe
(6) 2->4 (7) 2->3 (8) 2->6
5
(9) 3->5 (I0) 3->5
Combining rule Majority Vote: Sum rule: Max rule: Min rule: Product rule: Median rule:
Classification rate % 97.96 98.05 93.93 86.00 84.69 98.19
4 4qY - 6L
(11) 4->7 (12) 4->9 (13) 4->9 (14) 5->3 (15)6->4
5.2.5 Neural Network Classifier Our next classifier is a feed forward neural network (Multilayer Perceptron) trained as a pattern classifier. The momentum Back-propagation algorithm is used to train the network. The network consists of 100 nodes in the input layer (corresponding to the 100 dimensions in the feature space), 25 nodes in the hidden layer, and 10 nodes in the output layer. Each node in the output layer is associated with one class and its output O, with [zero to one] range, reflects the response of the network to the corresponding class <o. To facilitate a soft-level combination the responses are normalized and used as estimates of the a posteriori probability of the classes as
O
(71
(16) 7> (19) 8->4 (20) 9->4 Fig. 3; Samples misclassified by the HMM classifier and corrected by the sum-rule combiner.
P(w IX)=
Ik ok
5.3 The Combination Scheme In this expert fusion experiment, due to staff changes, we were unable to compute the within class correlation matrix for the different representations used. We can, therefore, only hope that the distinct representations used by the individual experts satisfythe assumption of class conditional independence at least approximately. Six different combination schemes are applied under the assumption of equal priors and their results are compared. These schemes can be divided into two groups according to the format of the individual classifiers used by the combiner. Hard-level combination uses the output of the classifier after it is hardthresholded (binarized). Soft-level combination on the other hand uses the estimates of a posteriori probability of the class by each classifier. The majority vote combiner is a rep-' resentative of the first category while the-five different operators are the soft-level combiners. Table 2 shows the results of classification of the individual classifiers while the results of different combining schemes are shown in Table 3. Note that the worst results are achieved when using the product rule which are similar to the performance of the m n rule. The results using these two rules are worse than any of the individual classifiers as well, and the reason is that if any of the classifiers reports the correct class a posteriori
probability as zero, the output will be zero, and the correct class cannot be identified. Therefore, the final result reported by the combiner in such cases is either a wrong class (worst case) or a reject (when all of the classes are assigned zero a posteriori probability). Another interesting outcome of our experiments is that the Sum rule as well as the med an rule have the best classification results. The major ty vote rule is very close in performance to the mean and median rules. The Max rule is still better than any of the individual classifiers, with the exception of the HMM classifier. 5.4 Analysis of the Results We analysed the results in more detail to see how the performance of the system' improves through decision combination. HMM classifier that yields the best classification rate among individual classifiers is chosen as a reference. Twenty examples of the samples misclassified by the HMM classifier and corrected by the sum-rule combiner are shown in Fig. 3. The numbers below each character represent the true class and that assigned by the HMM classifier, respectively. Although the HMM classifier scored quite well in the overall classification, it seemed to have failed to classify samples that otherwise look easy to recognize. Table 4 contains the coirresponding sampleswith the a posteriori probabilities estimated by each classifier. The table shows a clear difference in the values assigned to some of the samples by different classifiers. While one of the classifiers is 100 percent sure about the classification of the sample (the probability estimate is 1.0), the HMM classifier is 100 percent sure that it is not the true class (its estimate is zero). Note that 66 of the 107 of the misclassified samples are corrected by the simple sum rule combiner. An important requirement for a combiner that uses the output of the individual classifiers is that the classifiers
TABLE 4
SAMPLES MISCLASSIFIED BY HMM CLASSIFIER
True class
1 2 0 10
HMM decision
8 8
Structural
0.95 0.71
Neural Net.
0.99 0.05
Gaussian
1:00 1.00
HMM
0.00 0.00
1.00
0.56
0.04
0.00
0.00 0.12 0.00 0.25 0.00 0.00 0.00 0.39 .0.00 0.15 0.00 0.00 0.00 0.00 0.00 0.17 0.00
4 1 5 1 6 2 7 2 8 2 3 9 10 3 11 4 12 4 13 4 14 5 15 6 16 7 17 7 18 7 19 8 20 9 True class, class ass gned by the HMM
7 0.17 7 0.58 4 0.1 6 1.00 3 0.96 5 0.73 5 0.1 7 1.00 9 0.1 9 1.00 3 0.71 4 0.97 3 1.00 4 0.1 9 0.70 4 0.75 4 0.98 class f er and the a poster or probab It
1.00 1.00 1.00 1.00 0.99 1.00 0.99 1.00 0.00 1.00 '0.95 1.00 1.00 1.00 0.91 1.00 1.00 1.00 1.00 1.00 0.86 0.99 1.00 1.00 0.92 0.00 0.97 1.00 0.74 0.97 0.59 ' 1.00 0.98 1.00 es mated by each class f er. est
should not be strongly correlated in their "misclassification." That is, classifiers should not agree with each other when they misclassify a sample, or at least they should not assign the same incorrect class to asample. This requirement can be satisfied to a certain extent by 1) using different representations for the object (different feature sets) and 2) using a different classification principle for each of the individual classifiers. Using different representations (feature sets) leads, in many cases, to a reduction in the correlation between the outputs of. individual classifiers, since there is almost always less correlation between the input vectors using different representations than when using the same set of features. Different classifiers usually use different assumptions about the structure of the data and the stochastic model that generates it. This leads: to a different estimate of the a posteriori probabilities especially around the Bayes decision boundaries. It is also. pertinent to look at the samples that are misclassified by the combiner to see whether there was full correlation between all the classifiers in their decision. Thirty samples out of the 43 misclassified samples are correctly classified by at least one classifier. Fig. 4 displays some of the misclassified samples by the sum-rule combiner. In Fig. 4a, the samples are not recognized by any of the individual classifiers. In Figs. 4b, 4c, 4d, and 4e, samples are correctly classified by the classifier indicated below each sample. 6 ERROR SENSITIVITY
* conditional independence of the respective representations used by the individual classifiers and * classes being highly ambiguous (observations enhance the a priori class probabilities only slightly) appear to produce the most reliable decisions. In this section, we shall investigate the sensitivity of the product rule (7) and the sum rule (11) to estimation errors. We shall show that the sum rule is much less affected by estimation errors. This theoretically established behavior is consistent with the experimental findings. In the developments in Sections 2 and 3, we assumed that the a posteriori class probabilities P(w; I x), in terms of which'the various classifier combination rules are defined, are computed correctly. In fact, each classifier will produce only an estimate of this probability, which we shall denote
(wilx
). The estimate deviates from the true probability by
error e, i.e.,
P~il x )PP(wilx ) +ei
(27)
It is these estimated probabilities that enter the classifier combination rules rather than the true probabilities: Let us now consider the effect of the estimation errors on the classifier combination rules. Substituting (27) into (7) we have
ass gn
Z ---
p-(R
i)
IP(wix )x+ e
A somewhat surprising outcome of the experimental comparison of the classifier combination rules reported in Sections 4 and 5 is that the sum rule (11), which has been developed under the strongest assumptions, namely, those of
max P'('"frk)! lP((ckox
) + ek J
(28)
236
0 ==> 8
/.
9 ==> 7 (struct+hmm) 0 ==> 8 (strucI+neur)
I => 4 (struct+neur)
I ==> 2 (ncur+hmm)
.A A. V
F'
7 ==> 4 (neur only)
.m~m
0 ==> unknown (ncur only)
3 ==> 5(gaus)r
JUNI
3 ==> 5(gaus )
9 ==> 5 (gaus)
8 ==-> 0 (gaus+struct)
(D)
(E)
3 ==> 2 (hmm)
4 => 8 (hmm)
5 =_> 8 (hmun)
Combin mis= 41 / 2011

stmct gaus = 16 = 5
correct by indiv = 30
neur
hmm
=8 = II
Fig. 4. Samples misclassified by the sum-rule combiner. (a) Samples not classified correctly by any individual classifier. (b) Samples classified correctly by the structural classifier. (c) By the Neural Network classifier. (d) By the Gaussian classifier. (e) By the HMM classifier.
Under the assumption that ek
<< P(oklx) which is rather
strong and may not represent the worst case scenario, and further assuming that P(wklx ) 0 we can rearrange the product term as
"R R R
Comparing (7) and (31) it is apparent that each term (class ok hypothesis) in the errorfree classifier combination rule (7) is affected by error factor
R
1 +
P(Wklx
(32)
[P(WkIX) +ek
-1
kfi P(XkIX
=1=1k
11
k)
lx
(29)
A similar analysis of the sum rule (11) commences with

ass gn Zmi
x
which can then be linearized as
f
+ ei
(30)
Substituting (30) into (28) we get
Substituting (30) into (28) we get +
S(1- R)P(0) + RfP(jI@
max{(1 -( R)P(w) + which can be rewritten as

which be rewritten as can
)+ek ] Y[P(wk x R
(33)
ass gn
-i
0ji
=1
max
k=1 L
iP-("-)(Wk)l
P( I)+ P(oix
)I Re] iF '
_
e
(31)
P(WX )
Fig. 5. Classifier combination schemes.
ass gn
Z -+
I
e
1+
(35)
(1- R)P(wa) +
P(C, x)
1+ Comparing error factors (32) and (35), it transpires that the sensitivity to errors of the former is much more dramatic than that of the latter. Note that since the a posteriori
max (1- R)P(ck)

k=1
+x
_
Sek
P(w1x + R '
(34)
/
class probabilities are less than unity, each error' ek in (32) is .The compounded effect of all these amplified by
P(Wo jx
1:P(wkX
=1
A comparison of (11) and (34) shows that each term in the error free classifier combination rule (11) is affected by error factor
amplified errors is equivalent to their sum. In contrast, in the sum rule, the errors are not amplified. On the contrary, their compounded effect,; which is also computed as a sum, is scaled by the sum of the a posteriori probabilities. For the most probable class, this sum is likely to be greater than one which will result in the dampening of the errors. Thus, the sum decision rule is much more resilient to estimation er-
rors and this may be a plausible explanation of the superior performance of this cominbination strategy that we observed experimentally in Sections 4 and 5, or at least a contributing factor to it. It follows, therefore, that the sum classifier combination rule is not only a very simple and intuitive technique of improving the reliability of decision making based on different classifier opinions but it is also remarkably robust.
D.A. Denisov and A.K. Dudkin, "Model-Based Chromosome Recognition Via Hypotheses Construction/Verification," Pattern Recogn t on Letters, vol. 15, no. 3, pp. 299-.307, 1994. [7) H. E)-Shishini, M.S. Abdel-Mottaleb, M. EI-Raey, and A. Shoukry, "A Multistage Algorithm for Fast Classification of Patterns," Pattern Recogn t on Letters, vol. 10, no. 4, pp. 211-215, 1989. "M.C. Fairhurst and H.M.S. Abdel Wahab, "An Interactive Two[8) Level Architecture for a Memory Network Pattern Classifier," Pattern Recogn t on Letters, vol. 11, no. 8, pp. 537-540, 1990. [9) J. Franke and E. Mandler, "A Comparison of Two Approaches for Combining the Votes of Cooperating Classifiers," Proc. 11th IAPR Int't Conf. Pattern Recognition, Conf. B: Pattern Recognition Methodology and Systems, vol. 2, pp. 611-614; 1992. 7 CONCLUSIONS [101 L.K. Hansen and P. Salamon, "Neural Network Ensembles," IEEE Trans. PatternAnalys s and Mach ne Intell gence, vol. 12, no. 10, pp. The problem of combining classifiers which use different 993-1,001, Oct. 1990. representations of the patterns to be classified was studied. [11) Hashem and B. Schmeiser, "Improving Model Accuracy Using We have developed a common theoretical framework for Optimal Linear Combinations of Trained Neural Networks," IEEE Trans. Neural Networks, vol. 6, no. 3, pp. 792-794, 1995. classifier combination and showed that many existing [12] T.K. Ho, "Random Decision Forests," Third Int'l Conf Document schemes can be considered as special cases of compound Analysis and Recognition, pp. 278-282, Montreal, 14-16 Aug. 1995. classification where all the, pattern representations are used [131 T.K. Ho, J.J. Hull, and S.N. Srihari, "Decision Combination in jointly to make a decision. We have demonstrated that unMultiple.Classifier Systems," IEEE Trans. Pattern Analys s and Mach ne Intell gence, vol. 16, no. 1, pp. 66-75, Jan. 1994. der different assumptions and using different approxima[14) F. Kimura and M. Shridhar, "Handwritten Numerical Recognition tions we can derive the commonly used classifier combinaBased on Multiple Algorithms," Pattern Recogn t on, vol. 24, no. 10, tion schemes such as the product rule,.sum rule, min rule, Spp, 969-983,1991. max rule, median rule, and majority voting. The various [15) A. Krogh. and J. Vedelsby, "Neural Network Ensembles, Cross Validation, and Active Learning," Advances n Neural Informat on classifier combination schemes were compared experiProcess ng Systems 7, G. Tesauro, D.S. Touretzky, and T.K. Leen, mentally. A surprising outcome of the comparative study eds. Cambridge, Mass.: MIT Press, 1995. was that. the combination rule developed under the most [16) M.W. Kurzynski, "On the Identity of Optimal Strategies for Multirestrictive assumptions-the sum rule---outperformed stage Classifiers," PatternRecogn ton Letters, vol. 10, no. 1, pp. 39-46, 1989. other classifier combinations schemes. Toexplain this em[17) P. Pudil, J. Novovicova, S. Blaha, and J. Kittler, "Multistage Patpirical finding, we investigated the sensitivity of various tern Recognition With Reject Option," Proc. 11th (APR Int'l Conf schemes to estimation errors. The sensitivity analysis has Pattern Recognition, Conf. B: Pattern Recognition Methodology and Systems, vol. 2, pp. 92-95, 1992. shown that the sum rule is most resilient to estimation er[181 G. Rogova, "Combining the Results of Severa) Neural Network rors and this may provide a plausible explanation for its Classifiers," Neural Networks, vol. 7, no. 5, pp. 777-781, 1994. superior performance. [19) M. Skurichina and R.EPW. Duin, "Stabilizing Classifiers for.Very Small Sample Sizes," Proc. 11th IAPR Int'l Conf. Pattern Recognition, Vienna, 1996. ACKNOWLEDGMENTS [20] V. Tresp and M. Taniguchi, "Combining Estimators Using NonConstant Weighting Functions," Advances n Neural I nformat on This work was supported by the Science and Engineering Process ng Systems 7, G. Tesauro, D.S. Touretzky, and T.K. Leen, Research Council, UK (GR/K68165) and by the European eds. Cambridge, Mass.: MIT Press, 1995. 1211 C.H. Tung, H.J. Lee, and J.Y. Tsai, "Multi-Stage Pre-Candidate Union ACTS Project M2VTS. The authors would like to Selection in Handwritten Chinese Character Recognition Systhank Andrew Elms for making available the classification tems," Pattern Recogn t on, vol. 27; no. 8, pp. 1,093-1,102, 1994. results obtained using his HMM character recognizer and [22) D.H. Wolpert, "Stacked Generalization," Neural Networks, vol. 5, no. 2, pp. 241-260, 1992. Kenneth Jonsson and Medha Pandit for providing the 123) L. Xu, A. Krzyzak, and C.Y.Suen, "Methods of Combining Multifrontal face image and the voice data, respectively. We are ple Classifiers and Their Applications to Handwriting Recognialso indebted to Stephane Pigeon for providing the face tion," IEEE Trans. Systems, Man, and Cybernet cs, vol. 22, no. 3, profile data and verification results and to Gilbert Maitre pp. 418-435, 1992. [24) J.Y. Zhou and T. Pavlidis, "Discrimination of Characters by a for making available the voice-based verification decisions, Multi-Stage Recognition Process," Pattern Recogn t on, vol. 27, which were then used in fusion experiments. no. 11, pp. 1,539-1,549, 1994. [25) J. Kittler, M. Hatef, and R.P.W. Duin, "Combining Classifiers," Conf Proc. 13th Int'i Pattern Recognition, vol. 2, Track B, pp. 897REFERENCES 901, Vienna, 1996. [1) K.M. Ali and M.J. Pazzani, "On the Link Between Error CorrelaKittler, J. Matas, K. Jonsson, and M.U. Ramos SAnchez, "Com[261 J. bining Evidence in Personal Identity Verification Systems," Pattion and Error Reduction in Decision Tree Ensembles," Technical tern Recogn ton Letters, pp. 845-852, 1997. Report 95-38, ICS-UCI, 1995. [27) L. Breiman, "Bagging Predictors," Technical Report 421, Dept. of 12) S.C. Bagui and N.R. Pal, "A Multistage Generalization of the Rank Statistics, Univ. of California at Berkeley, 1994. Nearest Neighbor Classification Rule," Pattern Recogn t on Letters, Y. [28] Freund and R.E. Shapire, "Experiments With a New Boosting vol. 16, no. 6, pp. 601-614, 1995. Algorithm," Proc. 13th Int'l Conf.Machine Learning, 1996. 13]J. Cao, M. Ahmadi, and M. Shridhar, "Recognition of Handwrit[29) R.E. Shapire, Y. Freund, P. Bartlett, and W.S. Lee, "Boosting the ten Numerals With Multiple Feature and Multistage Classifier," Margin: A New Explanation for the Effectiveness of Voting MethPattern Recogn t on, vol. 28, no. 2, pp. 153-160, 1995. ods," Proc. 14th Int'l Conf.Machine Learning, 1997. 14) S.B.Cho and J.H. Kim, "Combining Multiple Neural Networks by 1[30) K. Tumrner and J. Ghosh, "Analysis of Decision Boundaries in LineFuzzy Integral for Robust Classification," IEEE Trans. Systems, arly Combined Neural Classifiers," Pattern Recogn t on, vol. 29, vol. Man, and Cybernet cs, 25, no. 2, pp. 380-384,1995. pp. 341-348, 1996. 15)S.B. Cho and J.H. Kim, "Multiple Network Fusion Using Fuzzy Logic," IEEE Trans. Neural Networks, vol. 6, no. 2, pp. 497-501, 1995. [6)
239
(31) K. Turner and J. Chosh, "Classifier Combining: Analytical Results and Implications," Proc. Nat'l Conf. Artificial Intelligence, Portland, Ore., 1996. 132] K.S. Woods, K. Bowyer, and W.P Kergelmeyer, "Combination of Multiple Classifiers Using Local Accuracy Estimates," Proc. CVPR '96, pp. 391-396, 1996. (33] J. Kittler, A. Hojjatoleslami, and T. Windeatt, "Weighting Factors in Multiple Expert Fusion," Proc. Br t sh Mach ne Vs on Conf, Colchester, England, pp. 41-50, 1997. [34] J. Kittler, A. Hojjatoleslami, and T. Windeatt, "Strategies for Combining Classifiers Employing Shared and Distinct Pattern Representations," Pattern Recogn I on Letters, to appear. [35] J. Kittler, "Improving Recognition Rates by Classifier Combination: A Theoretical Framework," Front ers of Handwr t ng Recogn t on 5, A.G. Downton and S. lmpedovo, eds. World Scientific, pp. 231-247, 1997. [36) T.S. Huang and C.Y Suen, "Combination of Multiple Experts for the Recognition of Unconstrained Handwritten Numerals," IEEE Trans. Pattern Analys s and Mach ne Intell gence, vol. 17, no. 1, pp. 9094, Jan. 1995. [37] P.A. Devijver and J. Kittler, Pattern Recogn t on: A Stat st cal Approach. Englewood Cliffs, N.J.: Prentice Hall, 1982. [38) M. Hatef and J. Kittler, "Constraining Probabilistic Re!axation' With Symbolic Attributes," Proc. Sixth Int'l Conf. Computer Analysis of Images and Patterns,V. Hlavac and R. Sara, eds., pp. 862-867, Prague, 1995. [39) M. Hatef and J. Kittler, "Combining Symbolic With Numeric Attributes in Multiclass Object Recognition Problems," Proc. Second Int'l Conf Image Processing, vol. 3, pp. 364-367, Washington, D.C., 1995. (40] A.J. Elms, "A Connected Character Recogniser Using Level Building of HMMs," Proc. 12th IAPR Int'l Conf Neural Networks, Conf. B: Pattern Recognition Methodology and Systems, vol. 2, pp. 439-441, 1994. [41] J. Matas, K. Jonsson, and J. Kittler, "Fast Face Localisation and Verification," A. Clark, ed., Bir t sh Mach ne Vs on Conf., pp. 152161, BMVA Press, 1997. [42] S. Pigeon and L. Vandendrope, "The M2VTS Multimodal Face Database (Release 1.00)," J. Bigun, C. Chollet, and G. Borgefors, eds., Aud o- and V deo-Based B ometr c Person Authent cat on, pp. 403-409. Springer, 1997. (43] D. Genoud, G. Gravier, F. Bimbot, and G. Chollet, "Combining Methods to Improve the Phone Based Speaker Verification Decision," Proc. Int'l Conf. Speech and Language Processing, vol-3, pp. 1,756-1,760, Philadelphia, 1996. (44] S. Pigeon and L. Vandendrope, "Profile Authentication Using a Chamfer Matching Algorithm," J. Bigun, G. Chollet, and G. Borgefors, eds., Aud o- and V deo-Based B ometr c PersonAuthent cat on, pp. 185-192. Springer, 1997. Josef Kittler graduated from the University of Cambridge in electrical engineering in 1971, where he also obtained his PhD in pattern recognition in 1974, and the ScD degree in 1991. He joined the Department of Electronic and Electrical Engineering of Surrey University in 1986 where he is a professor in charge of the Centre for Vision, Speech, and Signal Processing. He has worked on various theoretical aspects of pattern recognition and on many applications including automatic inspection, ECG diagnosis, remote sensing. robotics, speech recognition, and document processing. His current research interests include pattern recognition, image processing, and computer vision. He has co-authored a book with the title Pattern Recognition: A Statistical Approach," published by Prentice-Hall. He has published more than 300 papers. He is a member of the editorial boards of Pattern Recognition Journal, Image and Vision Computing, Pattern Recognition Letters, Pattern Recognition and Artificial Intelligence, and Machine Vision and Applications.
f7.
Mohamad Hatef received the BSc degree in electrical engineering from the University of Baghdad, Iraq in 1982. After graduation, he worked automatic control. He joined the University of Surrey to pursue postgraduate studies in 1991. Since 1995, he has been with ERA Technology, where he works on image and video compression.
Robert P.W. .Duin studied applied physics at Delft University of Technology in the Netherlands. In 1978, he received the PhD degree for a thesis on the accuracy of statistical pattern recognizers. In his research, he included various aspects of the automatic interpretation of measurements, learning systems, and classifiers. Around 1980, he integrated the Delft Image Processor, a reconfigurable pipelined machine, in an interactive research environment for image several projects on the comparison and evaluation of parallel architectures for image processing and pattern recognition. At the moment, his main research interest is the comparison oi neural networks with the traditional pattern recognition classifiers for learning. In 1994, he stayed as a visiting professor at the University Teknologi Malaysia. In 1995, he spent his sabbatical leave at the University'of Surrey in the Vision, Speech and Signal Processing Group. Heheld official positions in both, the Dutch Society for Pattern Recognition' and Image Processing as well as the International Association for Pattern Recognition (IAPR). He has been a member of the organizing committees of international conferences on signal processing (Eusipco 86) and pattemrecognition (ICPR 92), as well as of the scientific committees of many conferences on pattern recognition, image processing, and computer vision. At present, he is an associate professor of the Faculty of Applied Sciences of Delft University of Technology. He leads several projects on pattern recognition and neural network research, sponsored by both, the Dutch government and the industry. He is the author of a large number of scientific papers and has served as an associate editor for IEEE Transactions on Pattern Analysis and Machine Intelligence, Pattemrn Recognition Letters, and for Pattern Analysis and Applications. Several PhD students are coached by him. His teaching includes undergraduate, graduate, and postgraduate courses on statistics, pattern recognition, neural networks, and image processing. JIri (George) Matas received the MSc degree (with honors) in electrical engineering from the Czech Technical University in Prague, Czechoslovakia, in 1987., and a PhD degree from the University of Surrey, UK in 1995. He currently is a research fellow both with the Centre for Vision, Speech, and Signal Processing at the University of Surrey and the Centre for Machine Perception at the Czech Technical University. His interest include pattern recognition and computer vision.
processing. In connection with this, he
Initiated
Application No. 11/231,353 Docket No. 577832000200 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24. NO. 3, MARCH 2002
FVC2000: Fingerprint Verification Competition

Dario Maio, Member, IEEE, Davide Maltoni, Raffaele Cappelli, J.L. Wayman, and. Anil K. Jain, Fellow, IEEE
Abstract-Reliable and accurate fingerprint recognition is a. challenging pattern recognition problem, requiring algorithms robust in many contexts. FVC2000 competition attempted to establish the first common benchmark, allowing companies and academic institutions to unambiguously compare performance and track improvements intheir fingerprint recognition algorithms. Three databases were created using different state-of-the-art sensors and a fourth database was artificially generated; 11 algorithms were extensively tested on the four data sets. We believe that FVC2000 protocol,. databases, and results will be uselul to all practitioners in the field not only as a benchmark for improving methods, but also for enabling an unbiased evaluation ot algorithms. Index Terms-Fingerprint verification, performance evaluation, biometric systems.
embedded in research papers and sometimes enriching the commercial claims of marketing brochures. The aim of this initiative was to take the first'step towards the establishment of a common basis, both for academia and industry, to better understand the state-of-the-art and what can be expected from the fingerprint technology in the future. Analogous efforts have been recently carried out for other biometric characteristics (e.g., face [151, [121) and, in general, for other classical pattern [18)). recognition tasks ([171,[1 ], 18], We decided to pose this effort as an international open competition to boost interest and give our results larger visibility. The 15th International Conference on Pattern Recognition (ICPR 2000) was ideal for this purpose. Starting in late spring 1999, when the FVC2000 Web site (7) was set up, we broadly publicized this event, inviting all companies and research groups we were aware of to take part. From the beginning, we stated that the competition was not meant as an official performance certification of the participant biometric systems, as: * The databases used in this contest. have not been acquired in a real environment and according to a formal protocol [23], [16], [19], [2] (also refer to [24] for an example of performance evaluation on real applications). Only parts of the participants software are evaluated by using images from sensors not native to each system. In fact, fingerprint-based biometric systems often implement proprietary solutions to improve robustness and accuracy (e.g., quality control modules to reject poor quality fingerprints, visual feedback to help the user in optimally positioning ' his/her finger, using multiple fingerprint instances to build more reliable templates, etc.) and these contributions are here discounted. According to the definition reported in [16], [19], FVC2000 should be conceived as a technology evaluation (with some analogies with the FERET contest organized by Philips on face recognition (15]). In fact, quoting [2]:
INTRODUCTION
IN the last decade, interest in fingerprint-based biometric systems has grown significantly 19]. Activity on this topic increased in both academia and industry as several research groups and companies developed new algorithms and techniques for fingerprint recognition and as many new fingerprint acquisition sensors were launched into the marketplace. Nevertheless, to date only a few benchmarks have been available for comparing developments in fingerprint verification. Developers usually perform internal tests over self-collected databases. In practice, the only public domain data sets are the US National Institute of Standards and Technology (NIST) CDROMs [20), [21) containing thousands of images scanned from paper cards where fingerprints were impressed by rolling "nail to nail" inked fingers. Since these images significantly differ from those acquired by optical or solid state sensors, they are not wellsuited for testing "online" fingerprint systems [9], although they, constitute an excellent benchmark for AFIS (Automated Fingerprint Identification Systems) developments (11] and fingerprint classification studies [41. NIST recently released a database containing digital videos of live-scan fingerprint data [22]; since this database was specifically collected for studying plastic distortion affecting the online acquisition process 15], [6] and the impact of finger rotation, it models only certain fingerprint variations and it is not recommendable for a general evaluation of verification algorithms. The lack of standards has unavoidably led to the dissemination of confusing, incomparable, and irreproducible results, sometimes * D. Maio, D. Maltoni, and R. Cappelli are with Biometric System Lab (BIOLAB)-DEIS, University of Bologna, via.Sacchi 3, 47023 Cesena, Italy. E-mail: (maio, maltoni, cappelli)@csr.unibo.it.. * J.L. Wyman is with the US National Biometric Test Center, College oJ Engineering, San lose State University, San Jose, CA 95192. E-mail: jlwayman@aol.com. * AK. Jain is with the Pattern Recognition and Image Processing Laboratory, Michigan State University, East Lansing, MI 48824. E-mail: jain@cps.msu.edu Manuscript received 09 Oct. 2000; revised 27 Feb. 2001; accepted 15 May 2001. Recommended for acceptance by D. Kriegman. Fr information on obtaining reprints of this article, please send e-mail to: tpami@computer.org,and reference IEEECS Lag Number 112958.
0162-8828/02/517.00
"The goal of a technology evaluation is to compare competing algorithms from a single technology. Testing of all algorithms is done on a standardized database collected by a "universal" sensor. Nonetheless, performance against this database will depend upon both the environment and the population in which it was collected. SConsequently, the "three bears" rule might be applied, attempting to create a database that is neither too difficult nor too easy for the algorithms to be tested. Although sample or example data may be distributed for developmental or tuning purposes prior to the test,. the actual testing must be done on data which has not been previously seen by algorithm developers. Testing is done using "offline" processing of the data. Because the database is fixed, results of technology tests are, repeatable." In FVC2000, the "universal" sensor is actually a collection of four different sensors/technologies to better cover the recent advances in fingerprint sensing techniques and to avoid favoring a particular algorithm through the choice of a specific sensor. In fact, databases 1 and 2 were collected by using two small-size and lowcost sensors (optical and capacitive, respectively). Database 3 was collected by using a higher quality (large area) optical sensor. Finally, images in database 4 were synthetically generated by using Each the approach described in 13]. of the four databases contained 880 fingerprints from 110 different fingers, collected using the "three bears rule" (not too easy, not too hard), based on our.prior subjective experiences with fingerprint recognition algorithms; in particular, on the one hand, we discarded fingerprint images we considered completely intractable even for a human expert, on the other hand, we avoided collecting perfect fingerprints which will be very easy for a matching algorithm;, some internally developed
0 2002 IEEE
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, TABLE 1 List of Participants
NO. 3, MARCH 2002
. . ~Oraoization. CEFET-PR / Antheus Technologia Ltda (Brasil) Centre for'Signal Processing, Nanyang Technological Universit (Singapore) Centre for Wavelets, Approximation and Information Processing, Department of Mathematics, National University of Singapore (Singapore) DITI Ditto Information & Technology Inc. (Korea) I FPIN FingerPin AG (Switzerland) KRDL Kent Ridge Digital Labs (Singapore) Natural Sciences and Mathematics, Institute of Informatics (Madedonia) NCMI i SAGEM SA (France) SAGI SAGEM SA (France) SAG2 UINi inha University (Korea) ! UTWE University of Twente, Electical Engineering (Netherlands) A tour digit ID was assigned to each algorithm. (Sagem SA submitted two different algorithms). CETP CSPN CWAI TABLE 2 The Four FVC2000 Databases Sensor Type Optical Sensor . Image Size Set A (wxd) Set B (wxd) 110x8 100x8 300x300
ID'
Tye Academic Academic Academic Commercial Commercial Academic Academic Commercial Commercial Academic Academic
DBI
Resolution 500 dpi
DB2 DB3 DB4
Capacitive Sensor Optical Sensor Synthetic Generator
256x364 448x478 240x320
100x8 100x8 100x8
1l0x8 10x8 10x8
500 dpi 500 dpi About 500 dpi
average ridge-line interdistance; this input parameter was estimatedfrom a real 500 dpi fingerprintdatabase. I the artificalgeneration, the resolution iscontrolled by the In algorithms helped us in accomplishing this task. Each database was split into a sequestered "test" set of 800 images (set A) and an open "training" set of 80 images (set B), made available to participants for algorithm tuning. The samples in each set B were chosen to be as much as possible representative of the variations and difficulties in the corresponding set A; to this purpose fingerprints were automatically sorted by quality as in [14] and samples covering the whole range of quality were included in set B. A final visual inspection of the obtained data sets was carried out to assure that "dry," "wet," "scratched," "distorted," and "markedly rotated"
. fingerprints were also adequately represented. As initially specified in the call for participation "FVC2000 compeltition focuses only on fingerprint verification (1-1 matching) and not on fingerprint identification (1-N matching)" [9]. Each participant was required to submit two executable computer programs: the first enrolling a fingerprint image and producing the corresponding template, the second matching a fingerprint template against a fingerprint image. Participants were allowed to submit four distinct configuration files, to adjust the algorithms internal parameters
working on the submitted executables to complete their evaluation by August 2000. the Once all executables were submitted, feedback was sent to the participants by providing them the results of their algorithms over training set B (the same data set they had previously been given) to allow them to verify that neither run time problems nor hardware-dependent misbehaviors were occurring on our side. Section 2 describes the four databases used; in Section 3, we present the criteria and the procedures used for performance evaluation. Section 4 reports the overall performance of the participating algoithms on each database and concludes with a comparison of the! average results. Finally, in Section 5, we draw some concluding remarks and discuss how we intend to continue supporting this initiative in the future. 2 DATABASES
Four different databases (hereinafter DB1, DB2, DB3, and DB4) were collected by using the following sensors/technologies [(10]: * *. * * DB1: optical sensor "Secure Desktop Scanner" by KeyTronic DB2: capacitive sensor "TouchChip" by ST Microelectronics DB3: optical sensor "DFR-90" by Identicator Technology DB4: synthetically generated based on the method pro/ posed in [3].
according to each specific database; configuration files could also contain precomputed data, to save time during enrollment and matching. For practical testing reasonis, the maximum response time of the algorithms was limited to 15 seconds for each enrollment and five seconds for each matching (on a Pentium III-450 MHz machine). In March 2000, after several months of active promotion, we had 25 volunteering participants (about 50 percent from academia and 50 percent from industry), far more than our initial expectation. By the end of April, the training sets were released to the participants. After the submission deadline (une 2000) for the executables, theinumer of participants decreased to the number ofparicitsdrease to 11 (mstothe nitill (most of the initially
Each database is 110 fingers wide (w) and eight impressions per finger deep (d) (880 fingerprints in all); fingers from 101 to 110 (set B) were made available to the participants to allow parameter tuning before the submission of the algorithms; the benchmark is then constituted by fingers numbered from 1 to 100 (set A). For a system evaluation, the size of the above four databases is certainly not sufficient to estimate the performance with high confidence.
1. These sensors are identified in order to clearly specify the features of the databases. None of the authors have any proprietary interests in these
registered companies withdrew). In any case, the number of

participants (see Table 1) was more than anticipated, so we started
companies or products.
H 2002
Fig. 1. S scale fat
e same
Fig. 2. S
However, in a technology evaluation (like, FVC2000), the aim is to capture the variability and the difficulties of the problem at hand and to investigate how the different algorithms deal with them. For this purpose, the size of our databases are adequate. Table 2 summarizes the global features of the four databases, and Fig. 1 shows a sample image from each one of them. It is worth emphasizing that the choice of providing more than one database is not aimed at comparing different acquisition technologies and devices; the results obtained by the algorithms on the different databases should not be conceived as a quality measure of the corresponding sensors, since the acquisition conditions and the volunteer crew of each database are different. To summarize, DB1 and DB2 have the following features: * The fingerprints are mainly from 20 to 30 year-old students (about 50 percent male). * Up to four fingers were collected for each volunteer (forefinger and middle finger of both the-hands). * The images were taken from untrained people in two different sessions and no efforts were made to assure a minimum acquisition quality. * . All the images from the'same individual were acquired by interleaving.the acquisition of the different fingers (e.g.,
* *
first sample of left forefinger, first sample of right fore finger, first sample,. of left middle, first sample of right middle, second sample of the left forefinger, ...). The presence of the fingerprint cores and deltas is not guaranteed since no attention was paid on checking the correct finger position on the sensor. The sensor platens were not systematically cleaned (as usually suggested by the vendors). The acquired fingerprints were manually analyzed to assure that the maximum rotation is approximately.in the 0 range [-15u, 15 ] and that each pair of impressions of the same finger has a nonnull overlapping area.
Database DB3 was collected as follows: * * * * The fingerprints are from 19 volunteers between th. ages of five to 73 (55 percent male). One-third of the volunteers were over 55 years of age. One-third of the volunteers were under 18 years of age. One-sixthlof the volunteers were under seven years of age (childrens fingerprints constitute an interesting case study, since the usable image area is small and the ridge-line density is high). Two images of up to six fingers (thumb, fore, and middle on left and right hands) were taken without interleaving
IEEE TF
Fig. 3. Images from DB3; all the samples are from different fingers and are roug hly ordered by quality (top-left: high quality, bottom-right: low quality). from each volunteer at each session and no efforts were made to assure a minimum acquisition quality. * Each volunteer was seen at four sessions, with no more than two sessions on any single day. * The time gap between the first and last sessions was at least three days and as long as three months, depending upon volunteer. * The sensor plate was systematically cleaned between image acquisitions. * At one session with each volunteer, fingers were cleaned with rubbing alcohol and dried. * Some part of the core was apparent in each image, but care was taken to avoid a complete overlap between consecutive images taken during a single session. * The acquired fingerprints were manually analyzed to assure that the maximum rotation is approximately in the range [-15 , 15 arid that each pair of impressions of the same finger has a nonnull overlapping area. Figs. 2 and 3 show some sample images taken from DB3. The collection of DB4 requires some explanation: In general, the use of artificial images for testing biometric systems is not considered to be the "best practice" [19]. Although, this may be the case for performance evaluation in real applications, we believe that in a technology evaluation event such as FVC2000, the use of synthetic images has three main advantages: * * It supplies images which are native to none of the participant algorithms, thus providing a fair comparison. Synthetic fingerprint databases can be created at a very low cost. Acquiring a large number of fingerprints for testing purposes may be problematic due to the great amount of time and resources required and to the privacy legislation (which in some countries prohibits the diffusion of such personal information. Furthermore, once a database has been "used," its utility is limited since, for successive testing of algorithms, a new unknown database should be used. It is possible to adjust the database difficulty by tuning different kinds of perturbations (e.g., maximum amount of rotation arid translation, and the amount of skin distortion).
If the generated artificial images were not a suitable simulation of real fingerprint patterns, the comparisons on the synthetic
ZeroFMR
Ic
Fig. 4 EER,
threshold
I
to
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACH INE
INTELLIGENCE,
VL
,MRH20
TABLE 3 -1 -Algorithm Performance over DB1 Sorted by EER
Algorithm
EER (%)
REJENROLL (%)
REJMATCH (%)
Avg Enroll Time (sec.)
Avg Match Time (sec.)
Sagl Sag2 Ceip

Cwai Cspn
Unve
0.67 1.17 5.06

7.06
0.00 0.00 0.00

3.71
0.00 0.00 0.00

3.90
12.48 10.88 0.81
0.96 0.88 0.89

0.32
7.60
7.98 10.66
0.00
0.00' 6.43
0.00
0.00 6.59
10.17
0.22
0.17
2.10 1.06
Krdl
110.40 1.00
Fpin Uinh Dii

Ncmi
13.46 21.02 23.63 49.11
0.00 1.71 '0.00 0.00,
0.00 5.08 0.00 0.12
10.83 I0.53 0.65 ;1.13
0.87 0.56 0.72 1.34
TABLE 4 Algorithm Performance over DB2 Sorted by EER
Algorithm Sa I
EER
REJENROLL
REJUATCH
Avg Enroll
Avg Match
(%)
0.61
(%)
0.00
(%)
0.00
Time (sec.)
1 2.63
Time (sec.)
1.03
Sag2 Cspn Cwai Cetp

Krdl Utwe Fpin Diti Uinh Ncmi
0.82 2.75 3.01 4.63

8.83
0.00 0.00 1.29 0.00

3.29
0.00 0.00 1.29, 0.09

4.41
10.93 10:17 10.23 i 0.85

11.16
0.93 0.17 0.30 0.98

2.88
10.65
11.14 13.83 15.22 46.15
0.00
0.00 0.00 0.86 0.00
0.00
0.00
-0.00
1.10.42
1 1.16 11.21 10.60 1 1.28
2.
2.12
1.24 1.28 0.65 1.57
4.08 0.00
database would be misleading. Furthermore, in order to improve the performance, ad hoc algorithms could be designed/tuned according to the same assumptions which model the synthetic generation. However, the presence of three real. databases in FVC2000 provides a natural way to check the validity of the results on DB4. The parameters of the synthetic generator were tuned to emulate a low-cost sensor with a small acquisition area; the maximum rotation and displacement and skin-distortion are adjusted to roughly reproduce the perturbations in the three previous databases.
T (Timeout): the enrollment exceeds the maximum allowed time (15 seconds). 3. C (Crash): the algorithm crashes during fingerprint processming. The three types of rejections are added and stored in REJE'NOL.
RJNRO L,*
PERFORMANCE EVALUATION
For each database, we will refer to the jth fingerprint sample of the i= ith finger as F,.j, 1... 100, and j = 1... 8, and to the corresponding template (computed from F) as T. For each database and for each algorithm: S The templates Ti, i= 1...100, and j= 1...7 are computed from the corresponding Fj and stored on a disk; one of the following three kinds of rejection can happen for each image Fi: I. F (Fail): the algorithm declares that it cannot enroll the fingerprint image.
Each fingerprint template T is matched, against the fingerprint images Fuk (j < k < 8) and the corresponding 2 Genuine Matching Scores gmsift are stored. The ntumber of matches (denoted as NGRA - Number of Genuine Recognition Attempts) is ((8 x 7)/2) x 100 = 2,800 in case REJENRao LL = 0. The failed, timeout (five seconds). and crash rejections are accumulated into REJNoRA; no gmsj is stored in this case. Each fingerprint template Ti, i = 1...100 is matched against the first fingerprint image from different fingers Fkt (i < k < 100) and the corresponding Impostor Matching Scores ims;k are stored. The number of matches (denoted as NIRA - Number of Impostor Recognition Attempts) is ((100 x 99)/2)= 4,950 in case REJENROLL = 0. The failed, timeout (5 seconds) and crash rejections are accumulated into REJiA; no imsak is stored in this case. The genuine matching score distribution and the impostor matching score distribution are computed (actually, the
2. If g is matched with h, the symmetric match (i.e., h against g) is not executed.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,
VOL. 24. NO. 3, MARCH 2002
TABLE 5 Algorithm Performance over DB3 Sorted by EER
Algorithm
Sag 1 Sag2 Cspn Cerp Cwaui
EER (%)
3.64 4.01 5.36 8.29 11.94
REJENROLL
) 0.00 0.00 0.57 0.00 12.86
REJMATCH
(%) 0.00 0.00 1.24 0.00 8.00
Avg Enroll Time (sec.

5.70 11.94 0.35 11.49 0.46
Avg Match Time (sec.)

2.13 1.94 0.36 1.66 0.57
Krdl Uinh
Utwe
12.20 16.32
17.73
6.86 10.29
0.00
5.12 7.64
0.00
11.48 1.28
10.44
1.60 1.36
2.31
Diii
Fpin Ncni
22.63
23.18 47.43
00.00
0.00 0.00
.00 '
0.00 0.01
2.59
. 2.13 2.25
2.67
2.19 2.75
TABLE 6 Algorithm Performance over DB4 Sorted by EER Algorithm EER (%) REJENoRII. (%) Avg Enroll REJMA1CH (%) , Tinme (sec.) Avg Match Time (sec.)
Sag Sag2
Cspn Cwai
1.99 3.11
5.04 6.30
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
1.90 0.69 0. 11
;0.16.
0.77 0.69
0.11 0.20
Cerp
Krdl Fpin Diti Urwe Uinh Ncmi.
7.29
12.08 16.00 23.80 24.59 24.77 48.67
0.00
10.86 0.00 0.00 0.00 2.14 0.00
0.00
10.24 0.00 0.00 0.00 4.28 0.25
0.65
10.70
10.77
0.72
0.79 0.80 0.60 4.17 0.45 1.19
10.52 10.42
10.42
1.08
*,
term "distribution" denotes a histogram) and graphically reported to show how the algorithm "separates" the two classes.. In fingerprint verification, higher scores are" associated with more closely matching images. The FMR(t) (False Match Rate) and FNMR(t) (False NonMatch Rate) curves are computed from the above Given a threshold t, distributions for t ranging from 0 to 1.3 FMR(t) denotes the percentage of imsek 2 t, and FNMR(t) denotes the percentage of gmsj. < t. Actually, since FMR and FNMR are used in the contest to compare the performance of different algorithms, FMR and FNMR are "corrected" to keep into account rejections stored in REJNI1A and REJNGRA: FMR(t) = card{irnsik
ims, NIRA
>
S A ROC (1Receiving Operating Curve) is obtained, where pairs (FMR(t), FNMR(t)) are plotted for the same value of t; particiular, for t = 0, FMR = 1, and FNMR = 0, while in for t > 1, FMR = 0, and FNMR= 1. The ROC curve is drawn in log-log scales for better comprehension. * The EquallError Rate EER is computed as the point where FNMR(t) = FMR(t) (see Fig. 4); in practice, the matching. score distributions (histograms) are not continuous and a crossover point might not exist. In this case, we report the interval [EERc,,,,,, EERhih].In Appendix A, an operational definition of EER is given. ZeroFMR is defined as the lowest FNMR at which no * False Matches occur and ZeroFNMR is defined as the lowest FiIR at which no False NonMatches occur (Fig. 4): ZeroFMR(t) = min t{FNMR(t)
t}
I FMR(ti)
= 0})
FNMR(t) = card{gms,t I gmsi. < t) + REJNcGrt FNMR~t) = NGRA. where card denote the cardinality of a given set. This ' correction assumes that a failure to match is always treated score < 0). by the system as a "nonmatch" (matching 3. FMR and FNMR are often referred as FAR (False Acceptance Rate) and FRR (False Rejection Rate) respectively, but the FAR/FRR notation is misleading in some applications. For example, in a welfare benefits system, which uses fingerprint identification to prevent multiple payments under false identity, the system "falsely accepts" an applicant if his/her fingerprint is "falsely rejected"; otherwise, a "false acceptance" causes.a, "false rejection."' *
ZeroFNMR(t) = mint{FMR(t) I FNMR(t) = 0)} Both ZeroFMR and ZeroFNMR may not exist; in such a case, we assign to them the value 1. The average enroll time is calculated as the average CPU time for a single enrollment operation, and average match time as the average CPU time for a single match operation between a template and a test"image. RESULTS
This section reports the performance of the tested algorithms on each of the four databases (Tables 3, 4, 5, and 6) and the average
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, TABLE 7 Average Performance over the Four Databases Sorted by Avg EER Algorithm SagI Sag2 Cspn Cetrp Cwai Krdl Utwe Fpin Uinh Diti Ncmi Avg EER Avg 0.00 0.00 . 0.14 0.00 4,46 6.86 0.00 0.00 3.75 0.00 0.00 Avg 0.00 0.00 0.31 0.02 3.14 6.52 0.00 0.00 5.23 0.00 0.09 Avg Enroll
VOL. 24, NO. 3. MARCH 2002
Avg Match
(%)
1.73 2.28 5.19 6.32 7.08 10.94 15.24 15.94 19.33 20.97 47.84
REJENo.L(%) REJMATCH (%)
Time (sec.)
3.18 1.11 0.20 0.95 0.27 1.08 10.42 1.22 0.71 1.24 1.44
Time (sec.)
1.22. 1.11 .0.20 1.06 0.35 1.58 2.67 1.27 0.76 1.32 1.71
r results over the four databases (Table 7). Fig. 5 shows the ROC foi DB3, which proved to be the most difficult data set. The notation' introduced in Section 3 is used in both the graphics and tables, with the only exception of reporting REJENROLL as a percentage value and to collapse both REJNCRA and REJxIRA into a single value
REJNIATCI:
For each algorithm, detailed results (including genuine and impostor distributions, FMR and FNMR curves, NGRA, NIRA, ... ) are reported in [13). Due to lack of space, Appendix B of this paper presents only detailed results of the SAG1 algorithm which had the best accuracy in our competition.
REJN ATCII =
NIRA REJNIA + NGRA REJNcRA NIRA + NGRA
CONCLUSIONS
For a, correct interpretation of the results, EER alone is not a sufficient metric; REJENROLL should be also taken into account.
Most of the algorithms submitted to the competition performed well, if we take into account the difficulty of adapting a given
FNMR
1
I
-USag1 SSag2
-- +- O3pn
------ f--
atp
ONai
Krdl L we
Fpin
10-1
-------
-----
Unh
Itrvi
10-2
1A-3
'10-5 10-4 .10-3 . 10-2 10.1
FMR
Fig. 5. ROC curves on 083. Each point denotes a pair (FMR(t), FNMR(t)) for a given value oft.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 3, - MARCH 2002
atcd ,R
tl
12
Fig. 6. Computing the EER interval. On the lop an example is given where an EER point exists. On the bottom, two cases are shown where an EER point does not exist
and the corresponding intervals are highlighted. Average enroll time: 2.48 seconds
Average match lime: 0.96 seconds

REJENROLL 0.00% (F:0 T:0 C:0) NGRA 2800 NIRA 4950 REJNcGA 0.00% (F:0 T:0 C:0
REJIs,
0.00% (F:0 T:0 C:0)
EER. 0.67% (0.67%-0.68%
EER* 0.67% (0.67%-0.68%)
ZeroFMR 2.11%
FNMR
ZeroFNM R 53.13%
Score distributions
Fig. 7. Performance ol algorithm Sogl on database DBI 4A.
FMR(t) and FNMR(t)
ROC curve
algorithm to new
types of images.
In particular, algorithms Sagl and
and on what we learned from this experience, we can make the following observations: * A coarse analysis of the errors on genuine attempts showed that most of the errors were made by the ,algorithms on about 15-20 percent poor-quality fingerprints in each database. In other words, we could claim that a 20-80 rule is valid: that is, 20 percent of the database is responsible for 80 percent of the errors. The most accurate algorithm (Sagl) takes a lot of time for enrollment (3.18 sec with respect to a median enrollment time of 1.08 sec). This suggests that an accurate image
Sag2 showed the best accuracy and Cspn exhibited a good trade-off between accuracy and efficiency. Table 7 highlights a significant gap in the performance of the different algorithms and it would be extremely interesting to understand the reasons for such differences. To this purpose, after the presentation of the results, we asked the participants to provide some technical details about their methods, but only a few of them responded (the responses can be found at the FVC2000 Web site [7]); in any case, on the basis of the participant responses
410",
VOL. 24, NO. 3, MARCH 2002
Average enroll time: 2.63 seconds Average match time: 1.03 seconds R : . EJgago 0.00% (F:0 T:0 C:0)
EER 0.61% >
NGRA 2800
EER* 0.61%
NIRA 4950
REJyGgA 0.00% (F:0 T:0 C:0)

ZeroFMR 1.36%
REJ 0.00% (F:0 T:0 C:0)

ZeroFNMR 50.69%
nimit
FMR
FNMR
FNMR
,.
....
S10
20%
to7
0%
e0 s
vshd
104
10-
107
101
FMR
Score.distributions
8. Performance of algorithm Sag] on database DB2..A.
FMR(t) and FNMR(t)
ROC curve
Average enroll time: 5.70 seconds Average match time: 2.13 seconds
REJE 1,1. 0.00% (F:0 T:0 C:0) EER

3.64%
NGRA 2800 EER*

3.64%
INRA
4950
REJ~ 0.00% (F:0 T:0 C:0) ZeroFMR

6.82%
REJNRA 0.00% (F:0 T:0 C:0) ZeroFNMR

100.00%
Do7.
FM
FNMR
FNMR
50A"
40%
2%
dstrtoon
v~ i
Score distributions
Fig. 9. Periormance ol algorithm Sugl on database DB3.A.
FMR(I) and FNMR(t)
ROC curve
enhancement and feature extraction is really important for

improving the matching accuracy. Furthermore, feature extraction seems to perform asymmetrically, since the average matching time (which also includes the feature extraction time for the test image) is substantially lower than a single enrollment time.
The fastest algorithms (Cspn) extracts minutiae by an adaptive tracing of the gray-level ridges, without a priori binarization and thinning (which are time consuming tasks) [14], (25) and exploits local minutiae arrangement to speed-up the initial steps of minutiae matching (26].
VOL. 24, NO. 3, MARCH 2002
Average enroll time: I1.90 seconds

Average match time: 0.77 seconds
REJaEN-jL
0.00% (F:0 T:0 C:0) EER
1.99% (1.98%-2.00%)
NGRA
2800 ' EER*
NIRA
4950
REJNcG.
0.00% (F:0 T:0 C:0) ZeroFMR
6.71%
REJNmA
0.00% (F:0 T:0 C:0) ZeroFNMR
100.00%
1.99% (1.98%-2.00%)
FNMR
1o
10%,
30
100 10 0
10'
100
102
10
FMA
Score distributions
Fig. 10. Performance of algorithm Sagl on database DB4LA.
FMR(t) and FNMR(t)
ROC curve
Databases DBI and DB2 proved to be "easier" than DB3, even though the sensor used for DB3 is of higher quality. This means that the acquisition conditions and the volunteer population can have a stronger impact on the performance than sensor quality. The synthetically-generated database (DB4) was demonstrated to be adequate for FVC2000 purposes: in particular, from Tables 3, 4, 5, and 6, it is evident that the algorithm ranking on DB4 is quite similar to the other databases, proving that no algorithm was favored or penalized by the synthetic images. In particular, if an algorithm performs well on real fingerprints, then it also performs well on synthetic fingerprints and vice versa. The visual analysis of impostor and genuine distributions (see [13]) definitely supports this claim, since no significant differences are seen between the DB4 graphics and the others. Once again we would like to remark that the results reported here do not necessarily reflect the performance that the participating algorithms would achieve in a real environment or when embedded into a complete biometric system. In any event, we believe that FVC2000 results: provide a useful overview of the state-of-the-art in this field, allow researchers and companies to test their algorithms over common databases collected using state-of-the-art sensors, and * provide guidance to the participants for improving their algorithms. In future, we intend to continue supporting this initiative as follows: * The existing FVC2000 Web site [71 will be maintained to diffuse FVC2000 results and to promote FVC2000 testing protocol as a standard for technological evaluations. Companies and academic research groups will be allowed to test new algorithms or improved versions of existing * *
algorithms on the FVC2000 benchmark databases and to add their results to the FVC2000 Web site. New entries will be kept isolated from the original entries, since hereafter, the full databases are known, in advance, which could allow algorithm tuning to give unfair advantage to new participants. * The second Fingerprint Verification Competition (FVC2002) has been scheduled and its results will be presented at the 16th International Conference of Pattern Recognition. SGenerating synthetic fingerprint databases for future evaluations will be further investigated.
APPENDIX A
An operational procedure for computing EER (interval), given'a finite number of genuine and impostor matching scores, is reported in the following. Let
tto t{
max
{t
S(gms.a U{
gmsij)
U {imsek)
I FNMR ( t )
FMR(t)},
and
tE {gms,)
U {imsA.) {t FNMR(t) > FMR(t)).
The EER interval is defined as:

IEER6.,,EER,I=
{ IFNMR(t).FMR(t,))
IFMR(ts),FNMR(t))
if FNMR(t)+FMR(t,)<FMR(t2)+FNMR(tz)
otherwise
and EER is estimated as (EERone + EERhigh)/2 (see Fig. 6). APPENDIX B Please see Figs. 7, 8, 9, and 10.
I..
412
VOL. 24.
NO. 3. MARCH 2002
REFERENCES
J. Barron,. D. Fleet, and S. Beauchermmin, "Systems and Experiment: Performance of Optical Flow Techniques,," Int'l I. Computer Vision, vol. 12, no. 1, pp. 43-77, 1994. 12) BIOTEST: European Esprit Project No: 21978 (http://www.npl.co.uk/npl/ sections/this/biotest/index.html), 1998. [3) R.Cappelli, A. Erol, D. Maio, and D. Maltoni, "Synthetic Fingerprint-image Generation," Proc. 15th Int'l Conf. Pattern Recognition (ICPR 2000), pp. 475478, Sept. 2000. [4) R.Cappelli, D. Maio, and D. Maltorni, "Combining Fingerprint Classifiers," Proc. First Int'l Workshop Multiple Classifier Systems (MCS2000), pp.351-361, June 2000. I5) R. Cappelli, D. Maio, and D. Maltoni, "Modelling Plastic Distortion in Fingerprint Images," Proc. Second Int'l Conf Advances in Pattern Recognition (ICAPR 2001), pp. 369-376, Mar. 2001. (6) C. Dorai, N.K. Ratha, and R.M. Bolle, "Detecting Dynamic Behaviour in Compressed Fingerprint Videos: Distortion," Proc. Computer Vision and Pattern Recognition, (CVPR 2000), vol. II, pp. 320-326, June 2000 (7) FVC2000 Web site: http://bias.csr.unibo.it/2000. Sept. 2000. 18) A. Hoover et at., "An Experimental Comparison of Range Image Segmentation Algorithms," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 7. pp. 673-689, July 1996. (91 Biometrics-Personal Identification in Networked Society. A.K. Jain, R. Bollearind, S. Pankanti, eds. Kiuwer Academic, 1999. (10] A.K. Jain, S. Prabhakar, and A. Ross, "Fingerprint Matching: Data Acquisition and Performance Evaluation," MSU Technical Report TR9914, 1999. III). H.C. Lee and R.E. Goensslen, "Advances in Fingerprint Technology," 1991. (12) J. Matas et al., "Comparison of Face Verification Results on the XM2VTS Database," Proc. 15th Int'l Conf. Pattern Recognition, vol. 4, pp. 858-863, Sept. 2000. (13) D. Maio, D. Maltoni, R. Cappelli, J.L. Wayman, and A.K. Jain, "FVC2000: Fingerprint Verification Competition," DEIS Technical Report BL-09-2000, Univ. of Bologna; available online at: http://bias.csr.unibo.it/fvc. 2000. [(14) D Maio and D. Maltoni, "Direct Gray-Scale Minutiae Detection in Fingerprints," IEEE Trans. Pattern Analysis Machine Intelligence, vol. 19, no. 1, pp. 27-40, 1997. (115] Phillips, H. Moon, S.A. Rizvi, and P.J.Rauss, "The FERET Evaluation P.). Methodology for Face-Recognition Algorithms," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 10, pp. 1090-1104, Oct. 2000. [16) P.J.Phillips, A. Martin, C.L. Wilson, and M. Przybocky, "An Introduction to Evaluating Biometric Systems," Computer, Feb. 2000, (17) P.J. Phillips and K.W. Bowyer, "Introduction to Special Section on Empirical Evaluation of Computer Vision Algorithms," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 4, pp. 289-290, Apr. 1999. [18) T. Randen and J.H. Husoy, "Filtering for Texture Classification: A Comparative Study," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 4, pp. 291-310, Apr. 1999. (19) "Biometrics Working Group, Best Practices in Testing and Reporting Performance of Biometric Devices,"UK Government's HYPERLINK http://www.alb.org.uk/bwg/index.html, Jan. 2000. (available online at: http://www.afb.org.uk/bwg/bestprac0.pdf). 120) C.I. Watson and C.L. Wilson, NIST Special Database 4, Fingerprint Database. US Nat'l Institute of Standards and Technology, 1992. (21) C.I. Watson, NIST Special Database 14, Fingerprint Database. US Nat'l Inst. Standards and Technology, 1993. [22) C.I. Watson, "NIST Special Standard Reference Database 24, NIST Digital Video of Live-Scan Fingerprint Database," US Nat'l Inst. Standards and Technology, 1998. [23) J.L. Wayman, "Technical Testing and Evaluation of Biometric Devices," Biometrics Personal Identification in Networked Society, A. Jain, et al. eds., Kluwer Academic, 1999. [24) J.L. Wayman, "The Philippine AFIS Benchmark Test," Nat'l Biometric Test Center Collected Works, 1997-2000, Sept. 2001),(available online at: http:// www.engr.sjsu.edu/biometrics/collected). (25) X. Jiang et al., "Minutiae Extraction by Adaptive Tracing the Gray Level Ridge of the Fingerprint Image," Proc. IEEE Int'l Conf. Image Processing (ICIP '99), 1999. 126) Jiang et al., "Fingerprint Minutiae Matching Based on the Local And X. Global Structures," Proc. 15th Int'l Conf. Pattern Recognition, Sept. 2000. (1)
u For more information on this or any other computing topic, please visit our Digital Library at http://computer.org/publications/dilb.
Appeared in Proc. of ICME, July 2003
COMBINING CLASSIFIERS FOR FACE RECOGNITION

Xiaoguang Lu*, Yunhong Wangt, Anil K. Jain*
*Department of Computer Science & Engineering, Michigan State, University

East Lansing, MI 48824
{lvxiaogu, jain} @cse.msu.edu tNational Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences P.O. Box 2728, Beijing 100080, P. R. China wangyh@ nlpr.ia.ac.cn
ABSTRACT Current two-dimensional face recognition approaches can obtain a good performance only under constrained environments. However, in the real applications, face appearance changes significantly due to different illumination, pose, and expression. Face recognizers based on different representations of the input face images have different sensitivity to these variations. Therefore, a combination of different face classifiers which can integrate the complementary information should lead to improved classification accuracy. We use the sum rule and RBF-based integration strategies to combine three commonly used face classifiers based on PCA, ICA and LDA representations. Experiments conducted on a face database containing 206 subjects (2,060 face images) show that the proposed classifier combination approaches outperform individual classifiers. 1. INTRODUCTION Human face recognition has a tremendous potential in a wide variety of commercial and law enforcement applications. Considerable research efforts have been devoted to the face recognition problem over the past decade [I]. Although there are a number of face recognition algorithms which work well in constrained environments, face recognition is still an open and very challenging problem in real applications. Among face recognition algorithms, appearance-based approaches (2][3][4][5] are the most popular. These approaches utilize the pixel intensity or intensity-derived features. Several such systems have been successfully developed and installed [1][6)17][8]. However, appearance-based methods do not perform well in many real-world situations,
This research was supported by NSF IUC on Biometrics (CITeR), at
where the query test face appearance is significantly different from the training face data, due to variations in pose, lighting and expression. Some examples of these variations for one of the subjects in our database are illustrated in Fig. 1. While a robust classifier could be designed to handle ,any one of these variations, it is extremely difficult for an appearance-based approach to deal with all of these variations. Each individual classifier has different sensitivity to different changes in the facial appearance. It has been reported that each appearance-based method shows different levels of performance on different subsets of images [6], suggesting that different classifiers contribute complementary information to the classification task. A combination 'scheme involving different face classifiers, which integrates various information sources, is likely to improve the overall system performance.
,,~
L~
.l..
.. ~'.
"
west Virginia University.
Fig. 1. Facial variations under different lighting conditions and facial expressions for the same subject [9]. The classifier combination can be implemented at two levels, feature level and decision level. We use the decision level combination that is more appropriate when the component classifiers use different types of features. Kittier [10] provides a theoretical framework to combine various classifiers at the decision level. Many practical applications of combining multiple classifiers have been developed. Brunelli and Falavigna [11)] presented a person identification system by combining outputs from classifiers based on audio and visual cues. Jain et al. [12) integrated multiple fingerprint matchers to develop a robust fingerprint verification system. Hong and Jain [I 3] designed a decision fusion scheme to combine faces and fingerprint for personal identification. Marcialis and Roli [14] exploited the fusion of
Fig. 2. Classifier combination system framework. PCA and LDA for face verification. We propose two combination strategies, sum rule and RBF network, to integrate the outputs of three well-known appearance-based face recognition methods, namely PCA [2], ICA [3] and LDA [4][5]. Our combination strategy is designed at the decision level, utilizing all the available information, i.e. a subset of (face) labels along with a confidence value, called the matching score provided by each of the three face recognition method. 2. CLASSIFIER INTEGRATION Our combination scheme is illustrated in Fig. 2. While this framework does not limit the number of component classifiers,, we currently use only three classifiers, namely, PCA, ICA and LDA. Following two strategies are provided for integrating outputs of individual classifiers, (i) the sum rule, and (ii) a RBF network as.a classifier, using matching scores as the input feature vectors. 2.1. Appearance-based Face Classifiers Three appearance-based classifiers, PCA [21, ICA [3] and LDA 141)[51 have been implemented. In each of these approaches, the 2-dimensional face image is considered as a vector, by concatenating each row (or column) of the image. Each classifier has 'its own representation (basis vectors) of a high dimensional face vector space. By projecting the face vector to the basis vectors, the projection coefficients are used as the feature representation of each face image. The matching score between the test face image and training data is calculated as the cosine value of the angle between their coefficients vectors.
Let X = (xt,x2,... ,x, .. ,vN) represent the n x N
concatenated from a p x p face image, where pxp = n. Here n represents the total number of pixels in the face image and N is the number of face images in the training set. The mean vector of the training images p = EN= i Xi is subtracted from each image vector. All the three representations can be considered as a linear transformation from the original image vector to a projection feature vector, i.e.
Y = WTX,
(1)
where Y is the d x N feature vector matrix, d is the dimension of the feature vector, and W is the transformation matrix. Note that d < < n. I. PCA [2]. The Principal Component Analysis basis vectors are defined as the eigenvectors of the scatter matrix ST, (2)
SST = (xI
(xi
)T
The transformation matrix WPCA is composed of the eigenvectors corresponding to the d largest eigenvalues. After applying the projection, the input vector (face) in an n-dimensional space is reduced to a feature vector in a d-dimensional subspace. 2. ICA [3]. Bartlett et al. [31 provided two architectures based on Independent Component Analysis, statistically independent basis images and a factorial code representation, for the face recognition task. The ICA separates the high-order moments of the input in addition to the second-order moments utilized in PCA. Both the architectures lead to a similar performance. There is no special order imposed on the ICA basis vectors.
data matrix, where each xi is a face vector of dimension n,
3. LDA [4][5]. The Linear Discriminant Analysis finds a transform WLDA, such that WTSBW WLDA = arg maX WT S W ' (3) w WT'S W where So is the between-class scatter matrix and Sw is the within-class scatter matrix, defined as SB =
i=1
C
3. EXPERIMENTS AND DISCUSSION

Our database is a collection of four different face databases,
Ni(xi - p)(xi -
)T,
(4)
Sw =
i=1 xkeX,
(xk -
--)(xk i)T.
(5)
available in the public domain (seetable 1). There are 206 subjects with 10 images per subject for a total of 2,060 images. Face images selected are near frontal and contain variations in pose, illumination and expression. Some images in the individual databases are not selected forour experiments; these face images have out-of-plane rotation by more than 45 degrees in the NLPR+MSU database and face images with occlusions due to sun glasses or a scarf in the AR database. Sample images from the databases are shown in Fig. 3. Face images are closely cropped to include only the internal facial structures such as the eyebrows, eyes, nose
and mouth, and aligned by the centers of the two eyes. All cropped images are resized to 42 x 42. Each image vector is normalized to be of unit length. Table 1.Database description. Face database No. of Variations subjects included ORL [16] 40 Slight pose and expression Yale [9) 15 Illumination and expression Illumination AR [17] 120
and expression
in the above expression, Ni is the number of training samples in class i, c is the number of distinct classes, Pi is the mean vector of samples belonging to class i and Xi represents the set of samples belonging to class i. 2.2. Integration Strategy Kittler [10] analyzed several classifier combination rules and concluded that the sum rule (defined below) outperforms other combination schemes based on empirical observations. Unlike explicitly setting up combination rules, it is possible to design a new classifier using the outputs of individual classifiers as features to this new classifier. We adopt the RBF network [15] as this new classifier. Given m templates in the training set, m matching scores will be output for each test image from each classifier. We consider the following two integration strategies I. Strategy 1: Sum Rule. The combined matching score is calculated as MScomb = MSPCA + MSICA + MSLDA. (6)
NLPR+MSU
31
Slight pose
(collected by the authors)
and expression
For a given test sample, Output the class with the largest value of MScomb. 2. Strategy II: RBF network. For each test image, the m matching scores obtained from each classifier are used as a feature vector. Concatenating these feature vectors derived from three classifiers results in a feature vector of size 3m. An RBF network is designed to use this new feature vector as the input to generate classification results. We adopt a 3-layer RBF network. The input layer has 3m nodes and the output has c nodes, where c is the total number of classes (number of distinct faces). In the output layer, the class corresponding to the node with the maximum output is assigned to the input image. The number of nodes in the hidden layer is constructed empirically,. depending on the sizes of the input and output layers.
Fig. 3. Representative face images in the database. (a) ORL, (b) Yale, (c) AR and (d) NLPR+MSU. The entire face database is divided into two parts..Nine images of each subject are used to construct the training data and the remaining one is used for testing. This partition is repeated 10 different times so that every image of the subject can be used for testing. The classification accuracy is the average of these ten different tests. All the individual classifiers use the cosine value of the angle between the two projection coefficient vectors (one from the database image and the other from the test image) as the matchinig score. Database image with the best match is used to determine the classification of the input image. The sum rule is applied to the matching score outputs of the three classifiers. The database image with the maximum
sum score is output as the final result. The recognition accuracies of different face recognition approaches are listed' in table 2. The cumulative match score vs. rank curve [6) is used to show the performance of each classifier, see Fig. 4. Since our RBF network outputs the final label, no rank information is available. As a result, we cannot compute the cumulative match score vs. rank curve for RBF combination. Table 2. Recognition accuracy of different classifiers.
5. REFERENCES [I] W. Zhao, R. Chcllappa, A. Rosenfeld, and P.J. Phillips, CVL Tech"Face recognition: A literature survey," nical Report, University of Maryland, Oct. 2000, <ftp://ftp.cfar.umd.edu/TRs/CVL-Reports-2000/TR4;167zhao.ps.gz>. [2] M. Turk and A. Pentland, "Eigenfaces for recognition," Journo. nal of Cognitive Neuroscience, vol. 3, 1, pp. 71-86, Mar. 1991. M.S. Bartlett, H.M. Lades, and T.J. Sejnowski, "Independent [3] component representations for face recognition," in Proceedings of the SPIE, 1998, vol. 3299, pp. 528-539. P. [4] N. Belhumeur, J. P. Hespanha, and D. J. Kriegman, "Eigenfaces vs. Fisherfaces: Recognition using class specific linear projection," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 711-720, Jul. 1997. [5]D. L. Swets and J. Weng, "Using discriminant eigenfeatures for image retrieval," IEEE Trans. PatternAnalysis and Machine Intelligence, vol. 18, no. 8, pp. 831-836, 1996. [6] P. Jonathon Phillips, Hyeonjoon Moon, Syed A. Rizvi, and Patrick J. Rauss, "The feret evaluation methodology for facerecognition algorithms," IEEE Trans. Pattern Analysis and Machine.Intelligence, vol. 22, no. 10, pp. 1090-1104, 2000. [7] ,ldentix, <http://www.identix.comn/>, Minnetonka, MN.
Viisage, <http://www.viisage.com/>, Littleton, MA. database, face University Yale <http://cvc.yale.edu/projects/yalefaces/yalefaces.html>.
PCA 79.1%
ICA 88.1%
_
LDA
I
Sum rule
I
RBF based
81.0%
. .
90.0%
90.2%
.
0.95
0
0
S
" 0.8 E
..
Suo.... m.r.u
-- PCA 0 ICA -o- LDA
-+- Sum rule
0.75;
0.71
10
Rank
Fig. 4. Cumulative match score vs. rank curve for the sum rule. Table 2 and figure 4 show that the combined classifiers, based on both the sum-rule and RBF network, outperform each individual classifier.
[101 J. Kittler, M. Hatef, R. Duin, and J. Matas, "On combining classifiers," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 3, pp. 226-239, 1998. [(11I R. Brunelli and D. Falavigna, "Person identification using .multiple cues," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 10, pp. 955-966, Oct. 1995. [121 A. K. Jain, S. Prabhakar, and S.'Chen, "Combining multiple matchers for a high security fingerprint verification system," Pattern Recognition Letters. vol. 20. no. 11-13. pp. 13711379, 1999. [I L. Hong and A.K Jain, "Integrating faces and fingerprint for 31 personal identification," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 12, pp. 1295-1307, 1998. 1141 G. L. Marcialis and F. Roli, "Fusion of Ida and pca for face verification," in Biometric Authentication, Jun. 2002, vol. LNCS 2359, pp. 30-37. [15] C. M. Bishop, Neural Networksfor Pattern Recognition, Oxford University Press, UK, 1995. [16] Ferdinando Samaria and Andy Harter, "Parameterisation of a stochastic model for human face identification," in Proc. 2nd IEEE Workshop on Applications of Computer Vision, Sarasota FL, Dec. 1994. [17] A.M. Martinez and R. Benavente, "The ar face database," CVC Tech. Report # 24, Jun. 1998.
4. CONCLUSIONS AND FUTURE WORK An integration scheme, which combines the output matching scores of three well-known face recognition approaches, is proposed to improve the performance of a face identification system. Two combination strategies, sum rule and RBF-based integration, are implemented to combine the output information of three individual classifiers, namely PCA, ICA and LDA. The proposed system framework is scalable; other face recognition modules can be easily added into this framework. Experimental results are encouraging, illustrating that both the combination strategies lead to more accurate face recognition than that made by any one of the individual classifiers. We are currently investigating the weighted sum rule based on the user-specific matching score distribution.
2-0
V;
,Application No.11/231,353 Docket No. 577832000200

1090
The FERET Evaluation Methodology for Face-Recognition Algorithms

P. Jonathon Phillips, Member, IEEE, Hyeonjoon Moon, Member, IEEE, Syed A. Rizvi, Member, IEEE, and Patrick J. Rauss
Abstract-Two of the most critical requirements in support oi producing reliable face-recognition systems are a large database of facial images and a testing procedure to evaluate systems. The Face Recognition Technology (FERET) program has addressed both issues through the FERET database of facial images and the establishment of the FERET tests: To date, 14,126 images from 1,199 individuals are included inthe'FERET database, which is divided into development and sequestered portions of the database. In September. 1996, the FERET program administered the third ina series of FERET lace-recognition tests. The primary objectives of the third test were to 1)assess the state of the art, 2) identify future areas of research, end 3) measure algorithm performance. Index Terms-Face recognition, algorithm evaluation, FERET database.
1 INTRODUCTION
an VER the last decade, face recognition has become active area of research in computer vision, neuroscience, and psychology. Progress has advanced to the point that;" face-recognition systems are being .demonstrated in realworld settings [6]. The rapid development of face recognition is due to a combination of factors: active development of algorithms, the availability of a large database of facial images, and a method for evaluating the performance of facerecognition algorithms. The FERET database and evaluation methodology address the latter two points and are de facto standards. There have been three FERET evaluations, with the most recent being the September 1996 FERET test. The September 1996 FERET test provides a comprehensive picture of the state-of-the-art in face recognition from still images. This was accomplished by evaluating the algorithms' ability on different scenarios, categories of images, and versions of algorithms. Performance was computed for identification arid verification scenarios. In an identification application, an algorithm is presented with an unknown face that is to be identified, whereas, in a verification application, an algorithm is presented with a face and a claimed identity, and the algorithm either accepts or rejects the claim. In this paper, we describe the FERET database and the September. 1996 FERET evaluation protocol and present identification results. Verification results are presented in Rizvi et al. [10].
* P.]. Phillips is with the National Institute of Standards and Technology,
100 Bureau Dr. STOP 8940, Gaithersburg,MD 20899-8940. E-mail: jonathon@nist.gov. * H. Moon is with Lau Technologies, 30 Porter Road, Littleton, MA 01460. E-mail: hm@lautechnologies.com. * S.A. Rizvi is with the Department of Engineering Science and Physics, College of Staten Island/CUNY, Staten Island, NY 10314. E-mail: rizvi@wagner.csi.cuny.edu. * P.J. Rauss is with the Army Research Laboratory, 2800 Powder Mill Road, Adelphi, MD 20783-1197. E-mail: rauss@erim-it.com. Manuscript received 2 Nov. 1998; revised 24 Sept. 1999; accepted 19 May 2000.
Recommended for acceptance by D.J. Kriegman.
For information on obtaining reprints of this article, please send e-mail to: tpami@computer.org, and reference IEEECS Log Number 108156:
The FERET tests model the following face recognition applications: identification from large law enforcement databases and verification from biometric signatures stored on smart cards. For both applications, there are a limited number of facial images per person and the face representation is learned (or decided) prior to people being enrolled in the system. In the Federal Bureau of Investigation's (FBI) Integrated Automated Fingerprint Identification System (IAFIS), the only required mugshot is a full frontal image [2]. The IAFIS stores digital fingerprints and mugshots and will be the main depository of criminal fingerprints and mugshots in the United States. Other examples of large databases with one image per person are photographs from drivers licenses, passports, and visas. When the IAFIS is fully operational, it is expected to receive 5,000 mugshots per day (1,800,000 per year). Because of the large number of mugshots, it is not practical to continually update the representation. Updating the representation would require training from millions of faces and updating millions of database records. For verification applications where biometric signatures are stored on smart card, a user inserts a smart card into an electronic reader and provides a new biometric signature to the system. The system then reads the biometric signature stored on the smart card and compares it with the. new signature. Based on the comparison, the claimed identity is either accepted or rejected. Because of the limited amount of storage space, a facial image cannot be stored on a smart card and a representation of the face must be stored. Thus, once the first person is enrolled in the system, it is not possible to update the facial represeritation. Also, because of limited storage space, the representation of only one facial image is stored on a smart.card. The FERET was a general evaluation designed to measure performance of laboratory algorithms on the FERET database. The main goals of the FERET evaluation .were to assess the state-of-the-art and the feasibility of automatic face recognition. Thus, the FERET test did not
0162-8828/00/510.00 0 2000 IEEE
PHILLIPS ET AL.: THE FERET EVALUATION METHODOLOGY FOR FACE-RECOGNITION ALGORITHMS
1091
explicitly measure the effect on performance of individual components of an algorithm nor did the test measure performance under operational scenarios. An operational test evaluates algorithms in an orderly and scientific manner under all conditions in which a system will operate. To obtain a robust assessment of performance, algorithms were evaluated against different categories of images. The categories were broken out by a lighting change; people wearing glasses, and the time between the acquisition date of the database image and the image presented to the algorithm. By listing performance in these categories, a better understanding of the face recognition field in general, as well as the strengths and weakness of individual algorithms is obtained. This detailed analysis helps assess which applications can be successfully addressed. All face recognition algorithms known to the authors consist of two parts: 1) face.localization and normalization and 2) face identification. We use the term face localization and normalization to differentiate it from face detection. In detection, the task is to find all faces in an image, where there can be multiple or no faces in the image. In the FERET evaluation, there is one face in an image. In the first part of an algorithm, the face is located in an image and then the face is normalized into a standard position for the recognition portion of the algorithm. Usually, normalization requires that a set of facial features is actually located to within a couple of pixels. Algorithms that consist of both parts are referred to as fully automatic algorithms and those that consist of only the second part are partially automatic algorithms. (A glossary of terms is in the Appendix.) The September 1996 test evaluated both fully and partially automatic algorithms. Partially automatic algorithms are given a facial image and the coordinates of the centers of the eyes. Fully automatic algorithms are only given facial images. The availability of the FERET database and evaluation methodology has made a significant difference in the progress of development of face-recognition algorithms. Before the FERET database was created, a large number of papers reported outstanding recognition results (usually > 95 percent correct recognition) on limited-size databases' (usually < 50 individuals). (In fact, this is still true.) Only a few of these algorithms reported results on images utilizing a common database, let alone met the desirable goal.of being evaluated on a standard testing protocol that included separate training and testing sets. As a consequence, there was no method to make informed comparisons among various algorithms. The FERET database has made it possible for researchers to develop algorithms on a common database and to report results in the literature using this database. Results reported in the literature do not provide a direct comparison among algorithms because each researcher reports results using different assumptions, scoring methods, and images. The independently administered FERET test allows for a direct quantitative assessment of the relative strengths and weaknesses of different approaches. More importantly, the FERET database and tests clarify the current state of the art in face recognition and point out general directions for future research. The FERET tests
allow the computer vision community to assess overall strengths and weaknesses in the field, not only on the basis of the performance of an individual algorithm, but also on the aggregate performance of all algorithms tested. Through this type of assessment, the community learns in an open manner of the important technical, problems to be addressed and how the community is progressing toward solving these problems.
BACKGROUND
The first FERET tests took place in August 1994 and March 1995 (for details of these tests and the FERET database and program, see Phillips and Rauss [61, Phillips et al. [7], and Rauss et al'. [8]). The FERET database collection began in September 1993 along with the FERET program. The August 1994' test established, for the first time, a performance baseline for face-recognition algorithms. This test was designed to measure performance on algorithms that could automatically locate, normalize, and identify' faces from a database. The test consisted of three subtests, each with a different gallery and probe set. The gallery contains the set of known individuals. An image of an unknown face presented to the-algorithm is called a probe, and the collection of probes is called the probe set. Since there is only one face iran image, sometimes "probe" refers to the identity of the person in a probe image. The first subtest examined the'ability of algorithms to recognize faces from a gallery of 316 individuals. The second was the falsealarm test, which measured how well an algorithm rejects faces not in the gallery. The third baselined the effects of pose changes on performance. The second FERET test, which took place in March 1995, measured progress .since August 1994. and evaluated algorithms on larger galleries. The March 1995 evaluation consisted of a single test with a gallery of 817 known individuals. One emphasis of the test was on probe sets that contained duplicate probes. A duplicate probe is usually an image of a person whose corresponding gallery image was taken on a different day. (Technically, the probe and gallery images were from different image sets; see description of the FERET database below.) The FERET database is designed to advance the state of the art in face recognition, with the collected images directly supporting both algorithm development and the FERET evaluation tests. The database is divided into a development set, provided to researchers, and a set of sequestered images for testing. The images in the development set are representative of the sequestered images. The facial images were collected in 15 sessions between August 1993 and July 1996. Collection sessions lasted one or two days. In an effort to maintain a degree of consistency ,throughout the database, the same physical setup .and location was used in each photography session. However, because the equipment' had to be reassembled for each session, there was variation from session to session (Fig. 1). Images of an individual were acquired in sets of 5 to 11 images. Two frontal views were taken (fa and fb); a different facial expression was requested for the second frontal image. For 200 sets'of images, a third frontal image was taken with a different camera and different lighting (this is referred to as
1092
VOL. 22, NO. 10, OCTOBER 2000
fa
duplicate I
fc
duplicate II
Fig. 1. Examples of different categories of probes (image). The duplicate I image was taken within one year of the fa image and the duplicate II and fa images were taken at least one year apart.
the fc image). The remaining images were collected at various aspects between right and left profile. To add simple variations to the database, photographers sometimes took a second set of images for which the subjects were asked to put on their glasses and/or pull their hair back. Sometimes a second set of images of a person was taken on a later date; such a set of images is referred to as a duplicate set. Such duplicates sets result in variations in scale, pose, expression, and illumination of the face. By July 1996, 1,564 sets of images were in the database, consisting of 14,126 total images. The database contains .1,199 individuals and 365 duplicate sets of images. For some people, more than two years elapsed between their first and most recent sittings, with some subjects being photographed multiple times (Fig. 1). The development portion of the database consisted of 503 sets of images and was released to researchers. The remaining images were sequestered. 3 TEST DESIGN
3.1 Test Design Principles. The FERET September 1996 evaluation protocol was designed to assess the state of the art, advance the state of the art, and point to future directions of research. To succeed at this, the evaluation design must solve the three bears problemh. The test cannot be too hard nor too easy. If the test is too easy, the testing process becomes an exercise in "tuning" existing algorithms. If the test is too hard, the test is beyond the ability of existing algorithmic techniques. The results from the test are poor and do not allow for an accurate assessment of algorithmic capabilities. The solution to the three bears problem is through the selection of images used in the evaluation and the evaluation protocol. Tests are administered using an evaluation protocol that states the mechanics of the tests and the manner in which the test will be scored. In face recognition, the protocol states the number of images of each person in the test, how the output from the algorithm is recorded; and how the performance results are reported. The characteristics and quality of the images are major factors in determining the difficulty of the problem being evaluated. For example, if faces are in a predetermined position in the images, the problem is different from that for images in which the faces can be located anywhere in the
image. In the FERET database, variability was introduced by the inclusion of images taken at different dates and locations (see Section 2). "This resulted in changes in lighting, scale, and background. The testing protocol is based on a set of. design principles. The design principles directly relate the evaluation to the face recognition problem being evaluated.. For FERET,, the applications are searching large databases and verifying identities stored on smart cards. Stating the design principles allows one to assess how appropriate the FERET test is for a particular face recognition algorithm. Also, design principles assist in determining if an evaluation methodology for testing algorithm(s) for a particular application is appropriate. Before discussing the design principles, we state the evaluation protocol. In the testing protocol, an algorithm is given two sets of images: the target set and the query set. We introduce this terminology to distinguish these sets from the gallery and probe sets that are used in computing performance statistics. *For all results in this paper, the images in the galleries and probe sets were distinct. The target setis given to the algorithm as the set of known facial images. The images in the query set consist of unknown facial images to be identified. For each image qi in the query set Q, an algorithm reports a similarity s,(k) between qi and each image tk in the target set T. The testing protocol is designed so that each algorithm can use a different similarity measure and we do not compare similarity measures from different algorithms. The key property of the new protocol, which allows for greater flexibility in scoring, is that, for any two images qi and tk, we know si(k). Multiple galleries and probe sets can be constructed from the target and query sets. A gallery g is a subset of the target set. Similarly, a probe set P is a subset of the query set. For a given gallery g and probe set P, the performance scores are computed by examination of similarity measures s1(k) such that qi E P and tk E 9. Using target and query sets allows us to compute performance for different categories of images. Possible probe categories include: 1) gallery and probe images taken on the same day, 2) duplicates taken within a week of the gallery image, and 3) duplicates where the time between the imagesis atleastone year. We cancreate a gallery of 100 people and estimate an algorithm's performance by. recognizing people in this gallery. Using this as a starting point, we can
'PHILLIPS ET AL.: THE FERET EVALUATION METHODOLOGY FOR FACE-RECOGNITION ALGORITHMS
1093
Target set
I I
Fig. 2. Schematic of the FERET testing procedure.
000 then create galleries of 200,300, ...1, people and deterSmine how performance changes as the size of the gallery increases. Another avenue of investigation is to create n different galleries of size 100 and calculate the variation in algorithm performance with the different galleries. We now list the three design principles. First, all faces in the target set are treated as unique faces. This allows us to construct multiple galleries with one image per person. In practice, this condition is enforced by giving every image in the target and query set a unique random identification. The second design principle is that training is completed prior to the start of the test. This forces each algorithm to have a general representation for faces, not a representation tuned to a specific gallery. The third design rule is that all algorithms compute a similarity measure between all combinations of images from the targetand query sets.
query sets and the coordinates of the centers of the eyes for images in the target and query sets. In the second version of the test, the coordinates of the eyes were not provided. By comparing the performance between the two versions, we estimate performance of the face-locating portion of a fully automatic algorithm at the system level. The test was administered at each group's site under the supervision of one of the authors. Each group. had three days to complete the test on less than 10 UNIX workstations (this limit was not reached). We did not record the time or number of workstations because execution times can vary according to the type of machines used, machine and network configuration, and the amount of time that the developers spent optimizing their code (we wanted to encourage algorithm development, not code optimization). We imposed the time limit to encourage the development of algorithms that could be incorporated into operational, fieldable systems. 3.2 Test Details The target and query sets consisted of images from both In the September 1996 FERET test, the target set contained the developmental and sequestered portions of the FERET 3,323 images and the query set 3,816 images. The target set database. Only images from the FERET database were consists of fa and fb frontal images. The query set consisted included in the test; however, algorithm developers were of all the images in the target set plus the fc, rotated images, not prohibited from using images outside the FERET and digitally modified images. The digitally modified database to develop or tune parameters in their algorithms. images in the query set were designed to test the effects The FERET test is designed to measure laboratory of illumination and scale. (Results from the rotated and performance. The test is not concerned with speed of the digitally modified images are not reported here.) All the implementation, real-time implementation issues, and results reported in this article are generated from galleries speed and accuracy trade-offs. These issues and others that are subsets of this target set and probe sets that are need to be addressed in an operational, fielded system and were beyond the scope of the September 1996 FERET test. subsets of this query set. For each query image q,, an Fig. 2 presents a schematic of the testing procedure. To, algorithm outputs the. similarity measure s,(k) for all images tk in the target set. For a given query image q;, the ensure that matching was not done by file name, we gave target images tk are sorted by the similarity scores s;(.)., the images random names. A rough estimate of the pose of Since the target 'set is a subset of the query set, the test each face was provided to each testee. Example pose output contains the similarity score between all images in estimates provided.vere: frontal, and quarter and half right. the target set. (Note: Having the target set as subset of the query set does not constitute training and testing on the. 4 DECISION THEORY AND PERFORMANCE same images. This is because the face representation is EVALUATION learned prior to the start of the test.) There were two versions of the September 1996 test. The The basic models for evaluating the performance of an target and query sets were the same for each version. The , algorithm are the closed and open universes. In the closed universe, every probe is in the gallery. In an open universe, first version tested partially automatic algorithms by with a list of images in the target and some probes are not in the gallery. Both models reflect providing them
1094
TABLE 1 Representation and Similarity Metric for Algorithms Evaluated Algorithm Excalibur Co. MIT Media Lab 95 MIT Media Lab 96 Michigan St. U. Rutgers U.. U. of So.: CA. U. of MD 96 U. of MD 97 Baseline Baseline Represenation Unknown PCA PCA-difference space Fischer discriminant Greyscale projection Dynamic Link.Architecture (Gabor Jets) Fischer discriminant Fischer discriminant PCA Correlation Similarity measure Unknown L2 MAP Bayesian Statistic L2 Weighted L 1 Elastic graph matching L2 Weighted L 2 Li Angle
different and important aspects of face-recognition algo- 5 LATEST TEST RESULTS rithms and report different performance statistics. The open The September 1996 FERET test was designed to measure universe model is used to evaluate verification applications. algorithm performance for identification and verification The FERET scoring procedures for verification is given in tasks. In this article, we' report identification results. Rizvi et al. [10]. Verification results are reported in Rizvi et al. [9), [10]. We The closed-universe model allows one to ask how good report results for 12 algorithms that include 10 partially an algorithm is at identifying a probe image; the question is automatic algorithms and two fully automatic algorithms. not always "is the top match correct?" but "is the correct The test was administered in September 1996 and answer in the top n matches?" This lets one know how March 1997 (see Table 1 for the representation and many images have to be examined to get a desired level of similarity metric for each algorithm and Table 2 for details performance. The performance statistics are reported as of when the test was administered to which groups and. cumulative match scores, which are plotted on a graph. The which version of the test was taken). Two of these horizontal axis of the graph is rank and the vertical axis is algorithms were developed at the MIT Media Laboratory. the probability of identification, (PI) (or percentage of The first was' the same algorithm that was tested in March 1995. This algorithm was retested so that improvecorrect matches). The computation of an identification score is quite ment since March 1995 could be measured. The second probe set and 11 be the size of P. We algorithm was based on more recent work [3], [4). simple. Let P be .a Algorithms were also tested from Excalibur Corporation where 9 = {g,..., g}) score probe set P against gallery 9g, and P= {pl,... ,p,v}, by comparing the similarity scores s&() (Carlsbad, California), Michigan State University (MSU) such that pi E P and gk E G. For each probe image pi E P, .[111 [161, Rutgers University [13], the University of Southwe sort si(.) for all gallery images gk E 9. We assume that a ern California (USC) 114], and two from the University of smaller similarity score implies a closer match. The function Maryland (UMD) [1, [15], [16). The first algorithm from id(i) gives the index of the gallery image of the person in UMD was tested in September 1996 and a second version of 1997. For the fully probe pi, i.e., pi is an image of the person in gid(i). A probe pi the algorithm was tested in March automatic version of the test, algorithms from MIT and USC is correctly identified if si(id(i)) is the smallest score for were evaluated. (id(i)) is one of the nth 9k E 9. A probe pi is in the top n if si The final two algorithms were our implementation of Let smallest scores s(') for gallery G. Rr, denote the number normalized correlation and a principal components analysis of probes in the top n.We reported R,z/IPI, the fraction of (PCA) based algorithm [5], [121. These algorithms prcovide a probes in the top n. performance baseline. In our implementation of the PCAIn reporting identification performance results, we state based algorithm, all images were 1) translated, rotated, and the size of the gallery and the number of probes scored. The scaled so that the.centers of the eyes were placed on specific size of the gallery is the number of different faces (people) pixels, 2) faces were masked to remove background and hair, contained in the images that are in the gallery. For all results and 3) the nonmasked facial pixels were processed by a that we report, there is one image per person in the gallery; histogram equalization algorithm. The training set consisted thus, the size of the gallery is also the number of images in of 500 faces. Faces were represented by their projection onto the gallery. The number of probes scored (also, size of the the first 200 eigenvectors and were identified by a nearestprobe set) is 1I1. probe set may contain more than one neighbor classifier using the L1 metric. For normalized The image of a person and the probe set may not contain an correlation, the images were 1) translated, rotated, and scaled image of everyone in the gallery. Every image in the probe so that the centers of the eyes were placed on specific pixels set has a corresponding image in the gallery. and 2) faces were masked to remove background and hair.
1095
TABLE 2 List of Groups That Took the September 1996 Test Broken Out by Versions Taken and Dates Administered (the 2 by MIT Indicates that Two Algorithms were Tested) Test Date March 1997 * * * * 2 * *
Version of test Fully Automatic Eye Coordinates Given
Group MIT Media Lab (3,4) U. of So. California (14] Baseline PCA (5,12] Baseline Correlation Excalibur Corp. MIT Media Lab Michigan State U. (11,16]) Rutgers U. (13] U. Maryland [1,15,16) USC
September 1996 *
Baseline
* *
We report identification scores for four categories of probes. For three of the probe categories, performance was computed using the same gallery. For the fourth category, a subset of the first gallery was used. The first gallery consisted of images of 1,196 people with one image per person. For the 1,196 people, the target and query sets contain fa and fb images from the same set. (The FERET images were collected in sets and, in each session, there are two frontal images, fa and fb, see Section 2.) One of these images was placed in the gallery and the other was placed in the FB probe set. The FB probes were the first probe category. (This category is denoted by FB to differentiate it from the fb images inthe FERET database.) (Note: the query set contained all the images in the target set, so the probe set is a subset of the query set.) Also, none of the faces in the gallery images wore glasses. Thus, the FB probe set consisted of probe images taken on the same day and under the same illumination conditions as the corresponding gallery image. The second probe category contained all duplicate frontal images in the FERET database for the gallery images. We refer to this category as the duplicate I probes. The third category was the fc probes (images taken the same day as the corresponding gallery image, butwith a different camera and lighting). The fourth category consisted of duplicates where there was at least one year between the acquisition of the probe image and corresponding gallery image, i.e., the gallery images were acquired before January 1995 and the probe images were acquired after January 1996. We refer to this category as the duplicate II probes. The gallery for the FB, duplicate 1, and fc probes was the same. The gallery for duplicate II probes was a subset of 864 images from the gallery for the other categories. 5.1 Partially Automatic Algorithms In this section, we report results for the partially automatic algorithms. Table 3 shows the categories corresponding to the figures presenting the results, type of results, and size of the gallery and probe sets (Figs. 3, 4, 5, and 6). The results for each probe category are presented on two graphs. One graph shows performance, for algorithms tested in
September 1996 and the baseline algorithms. The other shows performance for algorithms tested in March 1997, the baseline algorithms, and the UMD algorithm tested in
SSeptember 1996 (this shows improvement between tests). (The results are reported as cumulative match scores.)
In Figs. 7 and 8, we compare the difficulty of different
probe sets. Whereas Figs. 4, 5, and 6 report identification Sperformance for each algorithm, Fig. 7 shows a single curve
that is an average of the identification performance of all algorithms for each probe category: For example, the first ranked score for duplicate I probe sets is computed from an average of the first ranked score for all algorithms in Fig. 4. In Fig. 8, we presented current upper bound for performance on partially automatic algorithms for each probe category. For each category of probe, Fig. 8 plots the Figs.7 and 8 algorithm with the highest top rank score (R1). report performance of four categories of probes, FB, duplicate I, fc, and duplicate II. 5.2 Fully Automatic Performance In this section, we report performance for the fully automatic algorithms of the MIT Media Lab and USC. To allow for a comparison between the partially and fully automatic algorithms, we plot the results for the partially and fully automatic algorithms from both institutions. Fig: 9 shows performance for FB probes and Fig. 10 shows performance for duplicate I probes. (The gallery and probe sets are the same as in Section 5.1.) TABLE 3 Figures Reporting Results for Partially Automatic Algorithms Performance is Broken Out by Probe Category Figure no. 3 4 5 6 Probe Category FB duplicate I fc . duplicate II Gallery size 1196 1196, 1196 864 Probe set size 1195 722 194 234
1096
VOL..22,
NO. 10, OCTOBER 2000
0.95
0.9
0.85
0.8
0 75
10
15
20
25'
Rank
30
35
40
45
50
(a)
0.95
0.9 7 0.85 AC-'.UMD9 *.o
0.8.
E
I 5 I 10 I 15 I 20 1 25
Rank
Baseline Cor -Baseline EF --.- x---
UMU
90
0.75
I 30
I 35.
i 40
I 45
50
"(b) Fig. 3. Identification performance against FB probes. (a) Partially automatic algorithms tested in September 1996. (b) Partially automatic algorithms tested in March 1997.
5.3 Variation in Performance From a statistical point of view, a face-recognition algorithm estimates the identity of a face. Consistent with this view, we can ask about the change in performance of .an algorithm: "For a given category of images, how does performance change if the algorithm is given a different gallery and probe set?" In Tables 4 and 5, we show how algorithm performance varies if the people in the galleries change. For this experiment, we constructed six galleries of
approximately 200 individuals, in which an individual was in only one gallery. (The number of people contained within each gallery versus the number of probes scored is given in Tables 4 and 5.) Results are reported for the partially automatic algorithms. For the results in this section, we order algorithms by their top rank score on each gallery; for example, in Table 4,'the UMD March 1997 algorithm scored highest on gallery 1 and the baseline PCA and correlation tied for ninth place. Also. included in this table is average
1097
0.8
0.7
0.6
0.5
0.4
0.3 5 10 15 20 25 Rank (a)

1
30
35
40
45
50
0.9
0.8
"-0a -a""1.. -s
"OO.a
-I
0.7
0.6
0.5
USC -.. -.
0.4
O UMD 97 ...... X--- Baseline EF --Baseline Cor -+UMO 96-"

r
0.3
10
15
20
25 Rank
35
40
45
50
(b) Fig. 4. Identification performance against all duplicate I probes. (a) Partially automatic algorithms tested in September 1996. (b) Partially automatic algorithms tested in March 1997. performance for all algorithms. Table 4 reports results for FB probes. Table 5 is organized in the same manner as Table 4,.except that duplicate I probes are scored. Tables 4 and 5 report results for the same gallery. The galleries were constructed by placing images within the galleries by chronological order in which the images were collected (the first gallery contains the first images collected and the sixth gallery contains the most recent images collected). In Table 5, mean age refers to the average time between collection of images conkained in the gallery and the corresponding duplicate probes. No sc ores are reported in Table 5 for gallery 6 because there are r no duplicates for this gallery.
DISCUSSION AND CONCLUSION
In this paper, we presented. the Septtember 1996 FERET evaluation protocol for face recogniti on algorithms. The protocol was designed so that pe rforinance can be measured on different galleries and probe sets and on
1098
VOL. 22, NO. 10,
OCTOBER 2000
V9 f
',.UM
96
Baseline EF --- x-Baseline I ........ Cor --
10
15
20
25 Rank
30
35
40
45
50'
Fig. 5. Identification performance against fc probes. (a) Partially automatic algorithms tested in September 1996. (b) Partially automatic algorithms tested in March 1997.
identification and verification tasks. (Verification results mentioned in this section are from Rizvi et al. [9], [10].) The September 1996 test was the latest FERET evaluation (the others were the August 1994 and March 1995 tests.[7]). One of the main goals of the FERET evaluations was to encourage and measure improvements in the performance of face recognition algorithms, which is seen in the September 1996 FERET test. The first case is the improvement in performance of the MITMedia Lab September 1996 algorithm over the March 1995 algorithm; the second is the.
improvement of the UMD algorithm between September 1996 and March 1997. By looking at progress over the series of FERET evaluations, one sees that substantial progress has been made in face recognition. The most direct method is to compare the performance of,fully automatic algorithms on fb probes (the two earlier FERET evaluations only evaluated fully automatic algorithms). The best top rank score for fb probes on the August 1994 evaluation was 78 percent on a gallery of 317 individuals and, for
1099
..
fi
-
pp a ..
477
O 00,CB 7OC"
e ...
i!** -K.. --
" _I!"lkdli0
I 'f..
p'3 g
...
e
U1 eU
MIT 96 -.-.
Baseline EF --- x--'Excalibur Rutgers -.- ....... MIT 95 ...... Baseline Cor --- UJMD 96...
MSU
10
15
20
25 Rank
30
35
40
45
(a)
WWI a 0.8
"-" -k x "K" x
"' x-X-k' E:i!-)-?'X-
it
USC -"-*-
0.2
x Baseline EF -----a . UMD 97 Baseline Cor ---UMD 96 ..
0
0
10
15
20
25 Rank
30
35
40
45
50
Fig. 6. Identification performance against duplicate II probes. (a) Partially automatic algorithms tested in September 1996. (b) Partially automatic algorithms tested in March 1997.
March 1995, the top score was 93 percent on a gallery of 831 individuals (7). This compares to 87 pecent in September 1996 and ,95 percent in March 1997 (gallery of 1,196 individuals). This method shows, that over the course of the FERET evaluations, the absolute scores increased as the size of the database increased. The March 1995 score was from one of the MIT Media Lab algorithms and represents an increase from 76 percent in March 1995. On duplicate I probes, MIT Media Lab improved from 39 percent '(March 1995) to 51 percent (September 1996);
USC's performance remained approximately the same at 57-58 percent between March 1995 and March 1997. This improvement in performance. was achieved while the gallery size increased and the number of duplicate I probes increased from 463 to 722. While increasing the number of probes does not necessarily increase the difficulty of identification tasks, we argue that the September 1996 duplicate I probe set was more difficult to process than the March 1995 set. The September 1996 duplicate I probe set contained the duplicate II probes and the March 1995
1100
FB
probes
Duplicate.I probes
0.8
p
-x-X '*
o O '
Duplicate II probes -o-.

fflfl ;
0 V U Li
4J
0.6
a.O.
.'u
--
0.4
E U
-4
0 .
X O'0
0J
0.2
n
V
A
1
A
1
I
I
J
I
m
I
10
15
20
25
Rank
35
40
45
Fig. 7. Average identification performance of partially automatic algorithms on each probe category.
duplicate I probe set did not contain a similar class of probes. Overall, the duplicate II probe set was the most difficult probe set. Another goal, of the FERET evaluations is to identify areas of strengths and weaknesses in the field of face recognition. We addressed this issue by reporting performance for multiple galleries and probe sets and different probe categories. From this evaluation, we concluded that algorithm performance is dependent on the gallery and probe sets. We observed variation in performance due to changing the gallery and probe set within a probe category
and by changing probe categories. The effect of changing the gallery while keeping the probe category constant is shown in Tables 4 and 5. For fb probes, the range for performance is 80 percent to 94 percent; for duplicate I probes, the range is 24 percent to 69 percent. Equally important, Tables 4 and 5 show the variability in relative performance levels. For example, in Table 5, UMD September 1996 duplicate performance varies between number three and nine, while at the same time' there are algorithms that consistently outperform other algorithms. Of the algorithms tested in September 1996, the
X(-
... . .
,.?4=rt--K-X-X... " XX - - 4 .-
-'X f
- .X-7
277--',,-%-i )(-'-
-)o- X-
'.
''
a
0 Li Li
. '.o e ." 0o . .,Duplicate
N,
x fc probes -----ouplicate I probes .*. II probes - .
FB probes -- +--
V'U
a, 'U 'U ""
0.4
0.2
0
0
'1
10
15
20
25 Rank
30
35
40
45
SO
Fig. B.Current upper bound identification performance of partially automatic algorithm for each probe category.
1101
0.98 0.96 0.94 0.92 0.9 0.88 0.86 0.84 0.82

08
,
-x- "
-:
-X X '
X-X- X- X- K-NI"!
x x
USC partially automatic.---*-.. MIT partially automatic --USc fully automatic "e-. MIT fully automatic --- x---
0.
I
0
I.
.5
10
15
20
25 Rank
.35
40
45
Fig. 9. Identification performance of fully automatic algorithms against partially automatic algorithms for FB probes.
September 1996 MIT algorithm, clearly outperformed the other algorithms. In addition, the September 1996 MIT algorithms and the algorithms tested in March 1997 (UMD March 1997 and USC) outperformed the other algorithms tested. This shows that, despite the overall variation in performance, definite conclusions about algorithm performance can be made. These conclusions are consistent with Figs. 4, 5, and 6. The variation in Tables 4 and 5 is because traditional method of calculating error bars and confidence regions do not apply to face recognition. These traditional methods
require that each run of the decision problem be made with the same classes, .i.e., character recognition with the 26 letters in the English alphabet. However, in face recognition, changing the people in the gallery changes the underlying classification problem. (Remember, each person is a different class.) Computing error bars with different people in the gallery is equivalent to computing error bars for a character recognition . system using performance from different sets of characters. Similar results were found in Moon and Phillips [5] in their study of principal component analysis-based face
to
0 L U N
4., "r
N E
10
15
20
25 Rank .
30
35
40
45
50
Fig. 10. Identification performance of fully automatic algorithms against partially automatic algorithms for duplicate I probes.
1102
SIEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,
VOL. 22,
TABLE 4 Variations in Identification Performance on Six Different Galleries on FB Probes Images in Each of the Galleries do not Overlap, Ranks Range from 1-10 Algorithm Ranking by Top Match Gallery Size /Scored Probes 200/200 200/200 200/200 200/199 gallery 2 gallery 3 gallery 4 gallery,.5 10 8: 8 10 9 9 6 9 7 . 7 5 7 2 1 1 3 5 4 4 5 4 5 8 4 8 9 6 7 6 6 10 5 1 3 2 2
Algorithm Baseline PCA Baseline correlation Excalibur Corp. MIT Sep96 , MIT Mar95 Michigan State Univ. Rutgers Univ. UMD Sep96 UMDMar97
200/200 gallery 1 9 9 6 4 7 3 7 4 1
196/196 gallery 6 8 10 6 3 7 4 9 5 1
USC
Average Score
2
0.935
3
0.857
2
0.904 -
2
0.918
1
0.843
1
0.804
recognition algorithms. This shows that an area of future research is measuring the effect of changing galleries and probe sets and statistical measures that characterize these variations. Figs. 7 and 8 show probe categories characterized by difficulty. These figures show that fb probes are the easiest and. duplicate I1I probes are the most difficult. On average, duplicate I probes are easier to.identify than fc probes. However, the best performance on fc probes is significantly better ,than the best performance on duplicate I and [I probes. This comparative analysis shows that future areas of research include processing of duplicate I probes and developing methods to compensate for changes in illumination. The scenario being tested contributes to algorithm performance. For identification, the MIT Media Lab
algorithm was clearly the best algorithm tested in September 1996. However, for verification, there was not an algorithm that was a top performer forall probe categories. Also, for the algorithms tested in March 1997, the USC algorithm performed overall better than the UMD algorithm for identification; however; for verification, UMD overall performed better. This shows. that performance on one task is not necessarily predictive of performance on a different task. The September 1996 FERET evaluation shows that definite progress is being made in face recognition and that the upper bound in performance has not been reached. The improvement in performance documented in this paper shows directly that the FERET series of evaluations. has made a significant contribution to face recognition. This conclusion is indirectly supported by 1) the improvement in performance between the algorithms tested in September 1996 and
TABLE 5 Variations in Identification Performance on Five Different Galleries on Duplicate Probes Images in Each of the Galleries do not Overlap, Ranks Range from 1-10 Algorithm Ranking by Top Match Gallery Size / Scored Probes
200/143
Mean Age of Probes (months) Algorithm S Baseline PCA Baseline correlation Excalibur Corp. MIT Sep96 MIT Mar95 Michigan State Univ.. Rutgers Univ. UMD Sep96 UMD Mar97 USC Average Score 9.87 gallery 1 6 10 3 2 7 9 5 7 4 1 0.238
200/64
3.56 gallery 2 10 7 ' 5 1 4 6 7 9 2 3 0.620
200/194
5.40 gallery 3 5 6 4 2 7 8 10 9 3 1 0.645
200/277
10.70 gallery 4 5 6 4 2 8 10 7 9 3 1 0.523
200/44
3.45 gallery 5 9 8 3 3 10 6 6 3
1
1 0.687
1103
March 1997, 2) the number of papers that use FERET images and report experimental results using FERET images, and 3) the number of groups that participated in the September 1996 test.
REFERENCES
Target set. The set of known images presented 'to the algorithm when a test is administered. See gallery. ACKNOWLEDGMENTS
11] K. Etemad and R. Chellappa,' "Discriminant Analysis for Recognition of Human Face Images," J. Optical Soc. Am. A, vol. 14, pp. 1,724-1,733, Aug. 1997. [2] R.M. McCabe, "Best Practice Recommendation for the Capture of Mugshots Version," 1997. 2.0.http://www. nist. gov/itl/div894/ 894.03/face/face.html. APPENDIX (3] B. Moghaddam, C. Nastar, and A. Pentland, "Bayesian Face GLOSSARY OF TECHNICAL TERMS RecognitionUsing Deformable Intensity Surfaces," Proc. Computer. Vision and Pattern Recognition '96, pp. 638-645, 1996. Duplicate. A probe image of a person whose corresponding 14) B. Moghaddam and A. Pentland, "Probabilistic Visual Learning gallery image was taken from a different image set.' for Object Detection," IEEE Trans. Patteni Analysis and Machine Intelligence, vol. 17, no. 7, pp. 696-710, July 1997. Usually, a duplicate is taken on a different day than the 15]H. Moon and P.J. Phillips, "Analysis of PCA-Based Face corresponding gallery image. Recognition Algorithms," Empirical 'Evaluation Techniques in Computer Vision, K.W. Bowyer and P.J. Phillips, feds., pp. 57-71, Duplicate I probes. Set of duplicate probes for a gallery. Los Alamitos, Calif.: IEEE CS Press, 1998. , Duplicate II probes. Set of duplicate probes where there is [6] P.J. Phillips and P. Rauss, "The Face Recognition Technology (FERET) Program," Proc. Office of Nat'l Drug Control Policy, CTAC at least one year between the acquisition of the Int'l Technology Symp., pp. 8-11, Aug. 1997. corresponding probe and gallery images. [71 P.J. Phillips, H. Wechsler, J. Huang, and P. Rauss, "The FERET Database and Evaluation Procedure for Face-Recognition AlgoFB Probes taken from the same image set as the rithms," Image and Vision Computing J., vol. 16, no. 5, pp: 295-306, corresponding gallery images. 1998. fc probes. Probes taken on the same day, but with different [8] P. Rauss, P.J. Phillips, A.T. DePersia, and M. Hamilton, "The FERET (Face Recognition Technology) Program," Surveillance and illumination from the corresponding gallery images. Assessment Technology for Law Enforcement, SPIE, vol. 2,935, pp. 211, 1996. Fully automatic algorithm. Anralgorithm that can locate a [9] S. Rizvi, P.J. Phillips, and H. Moon, "The FERET Verification face in an image and recognize the face. Testing Protocol for Face Recognition Algorithms," Technical Report NISTIR 6,281, Nat'l Inst. Standards and Technology, Gallery. In computing performance, scores, images of the . http://www.nist.gov/itl/div894/894.03/pubs.html#face. 1998. set of known individuals. The gallery is used in [10] Rizvi, P.J. Phillips, and H. Moon, "The FERET Verification S. computing performance after a FERET test is adminisTesting Protocol for Face Recognition Algorithms," Image and Visioi Computing J., appear. tered. A gallery is a subset of a target set. A target set can to 1 D. Swets and J. Weng, "Using Discriminant Eigenfeatures for I] generate multiple galleries. Image Retrieval," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 8, pp. 831-836, Aug. 1996. Probe. Image containing the face of an unknown individual that is presented to an algorithm to be recognized. Probe [12] M. Turk and A. Pentland, "Eigenfaces for Recognition," J. Cognitive Neuroscience, vol. 3, no..I, pp. 71-86, 1991.' can also refer to the identity of the person in a probe [13] J. Wilder, "Face Recognition Using Transform Coding of image. Grayscale Projection.Projections and the Neural Tree Network," Artifical Neural Networks with Applications in Speech and Vision, Partially automatic algorithm. An algorithm that requires R.J. Mammone, ed., pp. 520-536, Chapman Hall, 1994. that the centers of the eyes are provided prior to [14] L. Wiskott, J.-M. Fellous, N. Kruger, and C. von der Malsburg, "Face Recognition by Elastic Bunch Graph Matching," IEEE Trans. recognizing a face. PatternAnalysis and Machine Intelligence, vol. 17, no. 7, pp. 775-779, Probe set. A set of probe images used in computing July 1997. algorithm performance. The probe set is used in [151 W. Zhao, R. Chellappa, and A. Krishnaswamy, "Discriminant Analysis of Principal Components for Face Recognition," Proc. computing performance after the FERET test is adminisThird Int'l Conf Automatic Face and Gesture Recognition, pp. 336-341, tered. A probe set is a subset of a query set. A query set 1998.. can generate multiple probe sets. [16] Zhao, A. Krishnaswamy, R. Chellappa, D. Swets, and J. Weng, W. "Discriminant Analysis of Principal Components for Face RecogQuery set. The set of unknown images presented to the nition," Face Recognition: From Theory to Applications, H. Wechsler, algorithm when a test is administered. See probe set. P.J. Phillips, V. Bruce, F.F. Soulie, and T.S. Huang, eds., pp. 73-85, Berlin: Springer-Verlag, 1998.
'probes.
The work reported here is part of the Face Recognition Technology (FERET) program, which is sponsored by the US Department of Defense Counterdrug Technology Development Program. Portions of this work was done while P.J. Phillips was at the US Army Research Laboratory (ARL). P.J. Phillips would like to acknowledge the support of the National Institute of Justice.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, . VOL. 22,
Syed A. Rizvi (S'92-M'96) received the BSc P. Jonathon Phillips received the BS degree in degree (honors) from the University of Engineermathematics In 1983 and the MS in electronic ing and Technology, Lahore, Pakistan, the MS, and computer engineering in 1985 from George and PhD degrees from the State. University of Mason University and the PhD degree in New York (SUNY) at Buffalo, in 1990, 1993, and operations research from Rutgers University in 1996, respectively, all in electrical engineering. 1996. He is a leading technologist in the fields of From May 1995 to July 1996, he was a research computer vision, biometrics, face recognition, associate with the US Army Research Laboraand human identification. He works at the tory, Adelphi, Maryland, where he developed National Institute of Standards and Technology coding and automatic target recoanition algo(NIST), where he is currently detailed to the rithms for FLIR imagery. Since September 1996, he has been an Delense Advanced Projects Agency to manage the Human Identification at a Distance (HumanlD) program. Prior to this, he served as project assistant professor with the Department of Engineering Science and Physics at the College of Staten Island of the City University of New leader for the Visual Image Processing Group's Human Identification project. His current research interests include computer vision, identify- York. From 1996 to 1998, he. was a consultant with the US Army ing humans from visual imagery, face recognition, biometrics, digital Research Laboratory, collaborating in research for image compression and automatic target recognition. His current research interests include video processing, developing methods for evaluating biometric algoimage and video coding, applications of artificial neural networks to rithms, and computational psycho-physics. Prior to joining NIST, he directed the Face Recognition Technology (FERET) program at the US image processing, and automatic target recognition. He has published more than 60 technical articles in his area of research. He is a member Army Research Laboratory. He developed and designed the FERET database collection and FERET evaluations, which are the de facto of SPIE and the IEEE. standards for the face recognition community. Also, he has conducted research in face recognition, biomedical imaging, computational Patrick J. Rauss receiveo mthe degree in BS engineering physics from Lehigh University.in psychophysics, and autonomous target recognition. Dr. Phillips was 1987. He received the MEng degree in applied codirector of the NATO Advanced Study Institute on Face Recognition: remote sensing and geo-ilnformation systems From Theory to Applications, coorganizer of the, First and Second from the University of Michigan in 2000. He has Workshops on Empirical Evaluation Methods for Computer Vision worked as a civilian researcher for the US Army Algorithms, and coprogram chair of the Second Intemrnational Consince 1988, first with the Night Vision and ference on Audio and Video-Based Biometric Authentication. He is Electro-Optics Directorate and, since 1992 with guest coeditor of the special section on empirical evaluation of computer the Army Research Laboratory's EO-IR Image algorithms in the IEEE Transactions on Pattern Analysis and Machine Processing Branch. He worked closely with Intelligence (PAMI) and the special issue of Computer Vision and Image D"r Phillins n the FERET nrnnram from 1995 Understanding (CVIU) on empirical evaluation. He has coedited two to 1997. His current research interests are automated processing.ol books. The first, Face Recognition: From Theory to Applications, was coedited with Harry Wechsler, Vicki Bruce, Francoise Fogelman-Soulie, . hyperspectral imagery for material classification and using supervised, and Thomas Huang. The second, Empirical Evaluation Techniques in adaptive leamrning techniques for hyperspectral and computer vision Computer Vision, was coedited with Kevin Bowyer. He is a member of applications. Over the years, he has developed signal and image processing tools and techniques for a wide range of sensors including the IEEE. X-ray fluorescent spectrometers, midwave and long-wave infrared radiometers, Fourier transform infrared spectrometers, forward looking Hyeonjoon Moon received the BS degree in infrared imagers, and hyperspectral imagers. electronics and computer engineering from Korea University, Seoul, in 1990, the MS, and PhD degrees in electrical and computer engineering from the State University of New York at Buffalo, in 1992 and 1999, .respectively. From 1993 to 1994, he was a systems engineer at Samsung Data Systems in Seoul, Korea. From 1996 to 1999, he was a research associate at the US Army Research Laboratory in Adelphi, Maryland. Currently, he is a senior research scientist at Lau Technologies in Littleton, Massachusetts. His research interests include image processing, neural networks, computer vision, and pattern recognition. He Is a member of the IEEE..
VOL. 23,
NO. 10,
OCTOBER 2001
Application No. 11/231,353 Docket No. 577832000200 a loo
,A Combination Fingerprint Classifier

Andrew Senior
Abstract-Fingerprint classification is an important indexing method for any large scale fingerprint recognition system or database as a method for reducing the number of fingerprints that need to be searched when looking for a matching print. Fingerprints are generally classified into broad categories based on global characteristics. This paper describes novel methods of classification using hidden Markov models (HMMs) and decision trees to recognize the ridge structure of the print, without needing to detect singular points. The methods are compared and combined with a standard fingerprint classification algorithm and results for the combination are presented using a standard database of fingerprint images. The paper also describes a method for achieving any level of accuracy required of the system by sacrificing the efficiency of the classifier. The accuracy of the combination classifier is shown to be higher than that of two state-of-the-art systems tested under the same conditions. Index Terms-Henry fingerprint classification, hidden Markov models, decision trees, neural networks, NIST database.
+.
1 INTRODUCTION
THE
and Maltoni [5] use a structural analysis of the direction of an classification of fingerprints has long been important part of any fingerprinting system. A partition ridges in the print, without needing to find core and delta of fingerprints into groups of broadly similar patterns points. Blue et al. [6] and Candela et al. [7] use the core allows filing and retrieval of large databases of fingerprints location to center their representation scheme, which is based on a principal components analysis (PCA) of ridge for. quick reference. Currently, interest in fingerprint classification is stimulated by its use in automatic finger-' directions, and then they use a variety of classifiers. Halici print identification systems (AFIS). In an AFIS, the goal is to and Ongun [8] similarly use PCA-projected, core-centered find a match for a probe fingerprint in the' database of ridge directions, but classified with a self-organizing map; enrolled prints, possibly numbering many millions. Classi- and Jain et al. [9] also use the core for translation fication is used in an AFIS to reduce the size of the search Sinvariance, using a Cabor filter representation and a k-Nearest Neighbor classifier. space to fingerprints of the same class before attempting For situations where there is no need to use existing exact matching. The most widely used system of fingerprint classification classes, some researchers have developed systems which rely is the Henry system and its variants [1]. Examples from five on machine-generated classes or dispense with classes all ofthe main classes of the Henry system are shown in Fig. 1. Stogether and use "continuous" classification [2], [101, 18], [11]. Here, the criterion is not adherence to the Henry classes, but Many previous authors have developed automated of merely consistency among classifications of different prints systems to classify fingerprints, using a wide variety techniques. Cappelli et al. [2] provide a recent review of a from the same finger. Fingerprints are represented by points number of methods that have been used and Section 8 in in a feature space on which some distance measure is defined. Test fingerprints are matched against all those in the database this'paper presents results from other authors. Most automatic systems use the Henry classes and these falling within some radius of the test print. By increasing the are important for existing AFIS databases and systems radius, classification can be made arbitrarily accurate, which require compatibility with human classifications, reducing errors by increasing the size of the search space either because of legacy data or because some manual and, hence, search time. Continuous classification holds the intervention is necessary in the process, requiring the use of prospect of circumventing the difficult and restrictive Henry human-interpretable classes. A variety of approaches to classification problem and has produced the best results of classification has been tried, the most fundamental being a recent years, but has disadvantages besides the uninterpretsyntactic analysis of the relative positions and number of ability mentioned above. Using Henry classes, the portions of core and delta points in the print. The core and delta points, the database that must be searched are always the same, shown in Fig..1, are the singular points in the flow of the allowing for rigid segmentation of the database and a priori strategy. A continuous system presents ridges. Finding these points in the image is a difficult image design)of the' search an entirely different subset of the database for every matching processing task, particularly with poor quality images, but if found reliably, the classification is simple. 13], [4]. Maio operation, complicating and slowing the matching.. This paper describes a combination of novel approaches to fingerprint classification using the Henry system. The system described has been designed to operate on both * The author is with the IBM T.J. Watson Research Center, PO Box 704, Yorktown Heights, NY 10598-0218. E-mail: aws.@watson.ibm.com. rolled and "'dab"fingerprints, where some of the structural Manuscript received 15 Dec. 1999; revised 24 Nov. 2000; accepted 17 May information tised by other systems (such as the delta 2001. position) may not be available in the fingerprint image. The Recommended for acceptance by R. Kumar. e-mail to: system described has been tested on the NIST Special For information on obtaining reprints of this. article, please seund tpami@computer.org, and reference IEEECS Log Number 111092. database 4 [12] database of fingerprint images and results
0162-8828/01/$10.00
0 2001
IEEE
1166
VOL. 23, NO. 10. OCTOBER 2001
combinations, and results are compared with previously published results in Section 8.
CLASSIFICATION BY HIDDEN MARKOV MODEL
Left loop
0 .338'R
Right loop 0.317
Whorl
0.279
Arch 0.037
Tented arch 0.029
Fig. 1. Examples of five fingerprint categories, marked with core and delta points, with their frequencies of occurrence. are presented. Further, 'a method of measuring the efficiency of a classification algorithm is described, allowing a principled comparison of this algorithm with previous published works. Finally, a method for achieving arbitrary accuracy is described, allowing the Henry classifier to be used with the flexibility of continuous classifiers. This method trades off accuracy against classifier efficiency, allowing an imperfect classifier to be used in 'a real-world system, while retainingall the advantages of a traditional Henry system. The approach taken here is of a combination of classifiers, each using different features and with different errors on test data. Two novel classifiers are described, using two-dimensional hidden Markov models (HMMs) and decision trees. In addition to showing that these. classifiers perform well on the classification problem, and without the need for core/delta information, this paper shows-that the combination of classifiers provides a way forward for the improvement of fingerprint classification in the same way as recent improvements in isolated handwritten character recognition performance have been largely brought about not by better classifiers but by combinations of different classifiers. The classifiers are tested in isolation and in combination with the Probabilistic Neural Network classifier and pseudoridge tracer from the PCASYS system described by Candela et al. [7]. The experiments are all performed on discrete, Henry classification, but the system could be extended to continuous classification, or classification with unsupervised clustering, using such techniques as unsupervised K-means HMM clustering [13]. The following sections describe the HMM classifier (previously described in [14]), the decision tree classifier, and PCASYS classifiers. In Section.5, classification based upon the outputs of these classifiers is then described as well as combining the classifiers to improve accuracy. Section 6 describes a measure of efficiency of the classifier and shows how arbitrary efficiency can be achieved. Section 7 presents results for the classifiers and their
Hidden Mairkov models are a form of stochastic finite state automaton well-suited to pattern recognition.and successfully applied to speech recognition [15], [16] and other problems. They are appropriate to the problem posed here because of their ability to classify patterns based on a large quantity of features whose number is variable and which have certain types of.underlying structure, especially if that structure results in stationarity of the feature distributions over some spatial or temporal period. Such structure is found in fingerprints, where ridge orientations, spacings, and curvatures are, for the most part, only slowly varying across the print:' In a fingerprint, the basic class information can be inferred from syntactic analysis of singular points, but can also be seen in the general pattern of the ridges-the way a nonexpert human would classify prints. The HMM is able to statistically model the different structures of the ridge patterns by accumulations of evidence across the whole print, without relying on singular point extraction., Ridge Extraction The-system deals with fingerprint images stored as arrays of gray levels and obtained with a scanner or camera deviceeither from an inked fingerprint on paper or as a "live-scan" directly from the finger. For much of. the work in this paper, the NIST-4 [12] database of rolled fingerprint images has been used since this provides a large number (4,000) of fingerprints with associated class labels. In addition, part of the NIST-9 database has been used. The features provided to the recognizer are based on the characteristics of the intersections of ridges with a set of . fiducial lines that are laid across the fingerprint image To find the ridge locations, a number of image processing techniques are used [17], summarized as follows: 2.1 Initial segmentation: The PCASYS algorithm for extracting a central fingerprint region from a full rolled print is used on prints from the NIST-9 database. NIST-4 is already segmented at this level; 2. Smoothing; 3. Finding the predominant direction in each of an array of blocks' covering the image; 4. Segmenting the image into the area of the print (foreground) and the unwanted background, based on the strength of directionality found in each block; 5. Applying directional filters to highlight the ridges and detecting pixels that are parts of ridges; 6. Thinning the ridge image so that each ridge is left represented by an eight-connected, one-pixel-wide line termed the skeleton. 1. 2.2 Feature Extraction Given the skeleton image of the ridges, parallel fiducial lines are laid across the image at an angle 4,as shown in Fig. 2, and each one followed in turn. For each intersection of a fiducial line with a ridge, a feature is generated. Each feature consists of a number of measurements, chosen to
SENIOR: A COMBINATION FINGERPRINT CLASSIFIER
1167
Row. 1
Row 2
Fig. 2. A sample fingerprint showing horizontal fiduclal lines (p = 0). characterize the ridge behavior and its development at the intersection point: the distance since the last intersection;' the angle of intersection; the change in angle since the last intersection; the curvature of the ridge at the intersection. The angle features (2) can be seen to contain similar information to the coarse direction field calculated in the preprocessing stages of this system and used by other systems as the feature set for classification [71. However, this representation allows a higher resolution representation of the fingerprints, and allows more information to be represented (e.g., ridge spacing and curvature). Further measurements could also be taken at each point. The measurements of each feature. are termed a frame and the frames, R. for the ith fiducial line are collectively termed a row, A, whose ordering is preserved. For each orientation 0 of fiducial lines, a separate representation m = {Ri, Vi) of the print is obtained. In this research, only horizontal and vertical lines have been used, giving features R h and 7R', respectively, but other angles may allow further information to be captured. 2.3 Hidden Markov Models Typically, HMMs are one-dimensional structures suitable for analyzing temporal data. Here, the data are twodimensional, but the process of feature extraction can also be described as a one-dimensional array of one-dimensional row processes. Thus, we can apply a "two-dimensional hidden Markov model," similar to that of Agazzi et al. 118], which consists of a nesting of row models within wholeprint models, as shown in Fig. 3. For classification, a model M, is constructed for each class, c, and the maximum likelihood class is chosen after calculating the probability of the data R given the model: argmaxc P(RTM c). 2.3.1 Row Modeling To simplify the analysis of the model, first consider a row model modeling a single row of fingerprint data. Each row model Mi, a conventional HM-1M and consists of a number is of states which model the small, stationary regions in a row. Any row R4 is assumed to have been generated by the row automaton transiting from state to state, producing the frames in the observed order at each transition, with Sij being the kth state in the sequence whereby Mi produces Ri
Row 3
Fig. 3. A schematic of the two-dimensional structure of the HMM, . showing three row models of live states each, forming a global model for a single class. (k corresponds to time in -a temporal HMM). The state transitions are controlled by probabilities P(S,|ISi _ ) trained with certain constraints: The state must monotonically increase Si. > Sij,,for k > k' and it is possible to skip states at the edge of the print. Because of the nature of the printing process whereby, especially for dabs, it is to be expected that edge regions of the fingerprint will be missing but the central regions will always be present, only states at the edge of the print may be skipped. This effectively constrains the initial state distribution P(Sijo). The frame emission probabilities are modeled with mixtures of diagonal covariance, multivariate Gaussian distributions. Thus, for any frame R.i, it is possible to calculate-the likelihood P(RIkSiSqk) of it occurring in any state Sk. With these likelihoods, for any row model, the likelihood of any row can be approximated by the maximum likelihood of any state sequence aligning the features and states calculated as a product of frame likelihoods and transition probabilities for the state sequence: P(RiM)j) max P(RiolSijo)P(Sijo) s~j
I
k
P( jSjk)P(SqIjSqik,).
The models are initialized by using an equal-length alignment with the frames evenly distributed across the states of the model. After estimating the initial parameter values, using smooth equal-length alignment [19], Viterbi alignment is used to find the maximum-likelihood alignment of frames with states, which is used for retraining. Around two iterations of training.are necessary to achieve good classification performance. 2.3.2 Global Model The global model is the same as a row model, except that its states are row models and its frames are whole rows. Thus, for each model c:
1168
P(RIMC)
max P(RolMso)P(Ms).
curve
" P(RkIMs')P(S' ISk,),

where S' is an alignment specifying which row model Ms. models the row of data Rk. 2.4 Multiple HMM Classifiers For each orientation of fiducial lines, a separate classifier can be made. Since the errors of the different classifiers will be different, a combination of their scores may yield better accuracy. Denoting by Me, M', the class c models trained with vertical and horizontal features, respectively, and assuming, independence, the likelihood of the data is written as:
right
bottomn ,
curve
Fig. 4. A single ridge showing the extracted leatures for the decision tree: curvature maximum and, left, right, bottom, and top, turning points.
P(^,7'ICc) = P(R IM")P(R'|M').
(3)
Fusion of multiple classifiers is treated in more detail in Section 5.3.
DEcISION TREE CLASSIFIERS
To provide a supplementary classification, hopefully giving uncorrelated errors, another type of features has been extracted and classified with a decision tree approach. Such decision, trees are built using techniques based upon those of Amit et al. 1201. These authors tackled a number of problems including that of digit recognition--classifying images of the digits "0" to "9." The technique used by Amit et al. for constructing decision trees involves the generation of a large number of simple features. Each feature in isolation provides little information about the classification decision, for example, the existence of an edge at a particular location in an image may give little clue as to the digit's identity. However, combinations of such features can represent much important information needed to make an accurate classification decision. Amit et al. describe a procedure for making decision trees by growing questions based upon such combinations of simple features. The procedure has been adopted here for fingerprint classification and involves an initial feature extraction phase, followed by question building which creates informative questions assisting in classification. These' complex questions are combined in a hierarchical manner, to form decision trees which are used for classification. Because the trees are constructed ,stochastically, trees constructed for the same problem have different performances and, as is common with decision tree classifiers, multiple trees are combined to give the final classification.
and reliably extracted and which encode the ridge shape in a simple, concise manner were chosen. The initial preprocessing used is identical to that of the HMM classifier, up to the extraction of ridges (Section 2.1), but, instead of taking features at intersections with fiducial lines, features are generated at salient points on the ridges. The features consist of curvature maxima and four axis= 0 for a ridge parallel turning points (g = 0 or represented as the parametric curve (x(s), y(s)) and distinguished by the sign of the second derivative). Some example features are shown in Fig. 4. For each feature, the feature type and location (in pixels at 500 dpi) is recorded. These features are all based on local computations on the ridges and, again, avoid the extraction of global features such as core and delta points. Again, they are invariant to translation and to small amounts of rotation. These features are also appropriate for the classification of dabs since the majority of features in a rolled print also occur in the region typically imaged in a dab.
3.2 Decision Trees A binary decision tree is constructed as a hierarchy of binary questions [21]. Questions are logical statements about the features that may be present in the fingerprint arid about the relations between those features; for a given fingerprint, a question is either true or false. At the "top" level, each test sample is asked the first question. According to the test sample data, the question returns either true or false and the branch to the second-level is determined. On whichever branch is chosen, a second level question is asked and a further bifurcation is induced. In this way, each test sample descends the tree by a route dependent on its features and arrives at a leaf node. Leaf nodes are labeled according to the classes of the test data that arrived there. In a simple classification problem, leaf nodes will be pure-i.e., receive only training samples from a single class and 3.1 Feature Extraction the unambiguous classification of any test sample arriving there would be the class of the training data at that node. This second classifier was designed to give a second opinion on the classification of a fingerprint image. For For more complex problems, the leaf nodes contain mixed data 'and the test data is labeled with a- probability this purpose, the errors in classification should be as uncorrelated as possible with those made by the HMM, distribution across the classes. Fig. 5 shows a small decision tree with two levels. Each thus a different set of features was generated for this classification method. Again, the motivation is to consider of the three nodes of the tree contains a question of the form distributed information from across the fingerprint without. specified in Section 3.3. At each node and at the leaves, a class histogram with four classes is shown, indicating the extraction of singular points. Because the class information is implicit in the shapes of ridges, features that are easily reduction in entropy as the tree is traversed.. The root node
1169
Fig. 5. A two-level decision tree, showing hypothetical class distributions at each node. Each node has a question formed of a list of feature types that must be present and a list of relationships between them that must be true for, the question to return "yes." has all classes equally likely, with no discrimination, and the other nodes have successively stronger discrimination between the classes. 3.3 Questions A question consists of a list of required features and a set of relations between them in the same manner as those of Amit et al. Each feature is specified as one of the five features described above (four turning points and curvature maximum). Relations are all of the form "x is direction of y," where direction is one of North, South, East, West. Optionally, a question can also impose a distance limitthat the feature must be within a certain distance. Experimentation led us to use two distance bands: features within 0.1" and features within 0.2". An example question may specify that there is a maximum of curvature East of a lower turning point which is itself East of, and within 0.2" of, a maximum of curvature. Other questions are shown in the nodes of Fig. 5. Questions are constructed in such a way that every feature is related to at least one feature in the feature list and so that every pair of features can have at most one relation. Given a new print, the print can be tested by a search that determines if the question can be fulfilled by the features of the print. 3.4 Question Construction During tree construction questions are constructed randomly as follows: Select any feature class as the first feature. Test the data separation. If more than 2/3 of training data at this node reply yes, refine the question and repeat this step. If less than 1/3 reply yes, discard this question and construct a new question. Otherwise, evaluate the question. Refining the question consists of adding extra restrictions, which inevitably make a "yes" answer less likely. The proportion of samples answering "yes" can be reduced in one of two ways. First, a feature can be added. In this case, a random feature type is chosen and added to the list. A random relation is chosen to relate it to a randomly chosen feature I. 2. already in the list. Second, if there are twoor more features in the question and some pair has no relation between them, then an additional relation can be added to the question between any pair of features that are as yet unrelated. When adding a relation is not possible, a feature is added. Otherwise, a random choice is made; biased toward adding a relation since this keeps the number of features lower, limiting the dimensionality of the search space for answering questions and making testing faster. Having arrived at a question which. channels approximately half the data to each of the "yes" and "no" sides, the question is evaluated. The measure of the effectiveness of a question is the change of entropy in the distributions before and after applying the question. Classes at the root node have high entropy, but the leaf nodes should have very low entropy (be "purer"). The relative entropy of the output distributions for a node is computed for many randomly constructed, candidate questions and the question with the highest entropy change is chosen. A tree is recursively constructed until the leaves are pure or until a maximum depth (typically 7) is reached. Fig. 6 shows the effect of varying the depth of the trees and the number of trees used. Multiple trees are merged by
s
20
o
the
+
5 Depth -....
i
. :2 ..... 2 S 4 2
0
e4
0 10
32
64 , .
129
Number of trees
Fig. 6. Raw error rates (no priors, unweighted) plotted against number of
1170
VOL. 23,
NO. 10,
OCTOBER 2001
multiplying the leaf node distributions class-by-class, as in Section 2.4.
4 PCASYS
Candela et al. [7] have described a fingerprint classifier called PCASYS, which is based upon a probabilistic neural network (PNN) classifying features consisting of principalcomponent projected orientation vectors. The orientations of the ridges taken in a 28x30 grid of points around the core are reduced in dimensionality with principal components analysis. The resulting 64-dimensional vectors are classified with the PNN. They have published results and made their software available, making possible a realistic comparison with this system. Lumini et al. [10] have used this software and extended it to provide continuous classification. To provide an alternative classification method and an enhanced combination classifier, PCASYS has been tested on the same testing data as classified by the HMM and decision tree.classifiers. Results are presented for PCASYS alone and in combination with the other classifiers. PCASYS incorporates a pseudoridge tracer which detects upward curving ridges and is able to correctly identify some whorl prints, but provides no information to distinguish among the other classes. This effectively penalizes the other classes when returning a "yes" answer. PCASYS also exploits prior information to improve its accuracy (see Section 5.1).
5.3 Classifier Combination Four classifiers (counting the PCASYS pseudoridge classifier) are available in this work. Each by itself is capable of classifying fingerprints with a limited.accuracy. However, each classifier uses different features and methods, so the . errors produced by each classifier should be somewhat uncorrelated. In this situation, combining the results of the. classifiers should produce an improved classifier with 'lower error rate: Many other authors have tackled the problems of decision fusion and, here, we take two simple approaches, 5.3.1 Linear Likelihood Fusion The first method of combination is a probabilistic approach since the output of each classifier is a probability distribution P(cJlV) across the classes c. Strictly speaking, if each classifier gave a true probability out and, with N independent classifiers operating on features RZ', the posterior probability would be:
P(cl7Z',... , 7ZN). P(c)
P(Vlc).
POSTPROCESSING AND CLASSIFIER COMBINATION
However, in practice, the probabilities are correlated and have varying reliabilities. The HMM probabilities are the product of many correlated probabilities and the PNN already incorporates prior information. To correct for these effects, weights w are introduced to balance the classifier combination. Working in the log domain, with normalization constant k: logP(cR , ... , RN) = k + logP(c)+ EwP(Z'Jc). For simple classification, the class (6)
Given the raw classifiers presented above, a number of steps must be taken to apply the classifiers to a test set. The following sections describe using class priors to enhance classifier accuracy, weighting the results to predict behavior on true test sets and methods for the combination of multiple classifiers. 5.1 Class Priors Because the classes that are used are not equal in frequency of occurrence, calculating the posterior probability of a class, given the data, requires the product of the data likelihood, given the class c and the prior probability of the class: P(c|Z) oc P(RJI.Mc)P(c). (4)
argmaxe log P(c|J',...,
N)
is chosen as the correct answer. Finding the weights w;, however, is a difficult problem. Estimation of weights by line searches on the training set fails to generalize well to the test set, so the followingtrained approach was used, which is found to achieve accuracies close to those obtained when optimizing linear weights by line search on the test set. 5.3.2 Neural Network Fusion The second fusion approach is to use a backpropagation neural network. Here, the class probabilities for all the classifiers are combined in a neural network, trained to output the true class on the training set. Additionally, four estimates of the fingerprint quality [23] are supplied to the network, though their effect is not significant. Training uses a momentum-based weight update scheme and Softmax outputs 124], giving an output class probability distribution. Training samples are weighted to simulate a uniform prior and the output probabilities are multiplied by the class prior 5.1 when testing. Separate networks are trained to combine the HMM and decision tree or to combine all four classifiers. To generate enough training data for the neural network, the first half of the NIST-4 database was supplemented with 5,400 prints from the NIST-9 database (Volume 2, CDs 1, 2, 3).
The class priors have been estimated by Wilson et al. [22] on 222 million fingerprints. (The proportions are 0.037, 0.338, 0.317, 0.029, 0.279 for arch, left loop, right loop, tented arch, and whorl, respectively). 5.2 Class Weighting Since the NIST-4 database (and, correspondingly, the test set used here) has equal numbers of prints from each class, to obtain a good estimate of the true test-condition accuracy, the results must be weighted according to the true frequencies of occurrence, using the same procedure of Wilson et al. [22]. Otherwise, a classifier good at recognizing arches, which are rare, would appear better on this test set than in the real world, or on a representative test set where this ability is rarely called upon.
1171
lower efficiency [25]. If a reliable measure of confidence in rge the classifier's answer is available, it is possible to devise Since the purpose of a fingerprint classifier is to partition la. ion methods to adjust the reliance on the classifier answer when fingerprint databases, in addition to the classificat accuracy-the proportion of classifications that give the that classification is uncertain, and thus reducing the correct class-the classification efficiency must also be number of errors made. Some classifiers [7] have used a considered. Since .many authors use different classes ,a rejection mechanism, which improves the accuracy at the consistent measure of efficiency, as described here, is cost of not pruning the search with those prints that are essential for the comparison of results. An efficiency meas ure rejected. This: section proposes a more complex scheme to also permits the evaluation of rejection and backing--off allow graceful and efficient "backing-off" of classifier strategies described in Section 6.2. reliance based on a likelihood ratio confidence measure. The classification efficiency can be considered a.,s a It is clear that if the likelihoods for the top two classes are measure of reduction of search space. In practice, the very different, then the classifier can be considered to be proportion of the database to be searched will vary ith more "confident" in its answer than if the two likelihoods each query, so, over a test set, the average efficiency can be are similar (when it would only take a small perturbation to calculated as: change the ranks of the answers). Thus, the likelihood ratio Number of matches required with no classifier (7) of the top two answers is examined. If this is less than an Number of matches required when classifier is used' empirically determined threshold, then the top choice is the top two classes are where an.exhaustive 1 : many match against a databas of deemed to be not confident and jointly returned as the answer (increasing the proportion of r is N prints is counted as N matches. If a perfect classifie the database subsequently searched by the 1:many matchused to classify M prints prior to matching agains ta er). Similarly, the likelihood ratio of the second and third database of N prints, any of the MPc test prints in cla.ssc choices is compared to a threshold to allow backing-off to (which occurs with probability Pc) need only be tes ted three classes. Repeating the procedure, if all the likelihoods against the NPc database prints of class c. Thus, the t otal are similar, the classifier will return a "don't know" answer number of matches required is now Z, NMP insteac I of and all classes must be searched. Moretraditional rejection from NM. The efficiency of a perfect classifier using these cla.,sses strategies (e.g., [22]) use a criterion to back off directly the "top-choice only" to the "don't know" answer without P. Using the five NIST-4 classes and the t allowing as rich a classification. is thus The efficiency of the classifier when allowing backing-off frequencies of Section 5.1, this gives an efficiency of 3 .39.
CLASSIFIER EFFICIENCY
Merging arch and tented arch classes only reduces the efficiency to 3.37 since this distinction so rarely needs tc Sbe made. As can be seen, the imbalance of the class pr iors makes the efficiency significantly lower than would be obtained with five equally frequent classes (an efficienc y of 5). In practice, the efficiency of a fallible classifier will. deviate' from this value-for instance, a classifier wl hich consistently mistakes all prints for arches will have an efficiency of 15 (1/0.066). 6.1 . Alternate Classes NIST-4 provides alternate class labels for 17.5 percent of the prints [12, p. 8, but these are ignored in this worlk,a classification being deemed correct only if it matches the nate primary class label. Allowing matches with the altern label too would increase the recognition rates but wc uld lower the efficiency of the classifier since such prints, w hen enrolled, would have to be stored twice (under both clas ses) in our database, resulting in extra searches every time the secondary class was searched. 6.2 Backing-Off Previous classification works have quoted error rates that would be unacceptable in real-world systems. It is clear that accuracy can be a trade-off for efficiency-searching n lore than just the top one class will give higher accuracy but
is now: MN
'
(8) ',,
1> where ir,,, rn,,,> 0, is the proportion of the database searched for query print' m..
Adjusting the likelihood ratio threshold allows arbitrary accuracy to be 'obtained. A large threshold would give a null classifier with 100 percent accuracy but an efficiency of one. A threshold of zero would give the basic top-one accuracy and maximum efficiency (3.37 for the four class problem). Adjusting the threshold allows us to set the overall classifier accuracy to that deemed necessary for the whole system. However, it should be noted that this arbitrary classification accuracy 'is achieved within the context of a Henry classification system where the portions of the database to be searched will always conform to the .Henry classifications and, thus, allow the database partitioning and search to be designed to operate on prior knowledge, not having to cope with dynamically changing subsets, as in continuous classification. In fact, the efficiency loss (i.e., extra search time) of searching the next class is dependent on the frequency of that class, so a more advanced backing-off algorithm should take this into account to achieve a better trade-off of accuracy for efficiency.
VOL. 23,
TABLE 1 Error Rates, Testing Different Combinations of the Classifiers Error (%) Classifier 20.8 HMM Horizontal (7 ") features 13.8 HMM Vertical (1?) features 12.8 HMM Both with prior 14.4 DT with prior 9.0 HMM + DT + prior 8.5 PCASYS (PNN only) ' 6.1 PCASYS (PNN + Pseudoridge) 6.8 HMM +DT +PCASYS (no prior) 5.1 HMM +DT +PCASYS (prior) HMM classifier used here has eight rows of eight states each and The uses a shared pool of 200 Gaussians with four-dimensional features. The decision tree (DT) classifierisa mixture of 30 trees, each with seven levels. The three combination results use a neural network. The PCASYS PNN+Pseudoridge classifier result uses a heuristiccombination [7] which incorporates the priors. 5
I,4
LTop~"...+ o 4Jain h al.
PCASYS 0..
03
2 1
1.5
Efficiency
2.5
3.
3.5
RESULTS
Fig. 7. A graph ol error rate against efficiency for the combination described here. The curve shows the trade-off of efficiency against error rate for a variety of likelihood ratio thresholds. Other algorithms tested under the same conditions are also plotted, along with the FBI performance target. system, the efficiency of the classifier is shown with the corresponding error rate. In some cases, authors have used rejection strategies where a proportion 7r of prints, are rejected, making the identification system search the whole database (with an efficiency of 1). Assuming these.rejected prints are uniformly distributed across the database, the efficiency of the combined system is E, =E(1 -7r) +7r,' where E is the natural efficiency using a classifier returning a single class. (This is the value plotted in Fig. 8 where appropriate.) 8.1 Comparison in the case of rejection and because of the wide range of testing conditions previously used, the figure and table present results which are not always directly comparable. In particular, the error rate of the PCASYS system is 7.8 percent under the conditions described by Candela et al. [71. However,. when their software is run on this test set and scoring in a manner consistent with the results presented here, with class weighting, the error rate was 6.1 percent (11.4 percent without weighting), a figure in which the results of. the combined classifier here should be compared. Similaily, jain et al. quote an accuracy of 5.2 percent with 1.8 percent rejection, but, if the data from the confusion matrix [9, table 3] are scored using the class frequencies found in real data (Fig. 1), the accuracy is 6.2 percent.' These two are the. only
Because of the estimation of efficiency
Following the practice of Jain et al. [9], the system has been trained and tested on the NIST-4 database of fingerprint images. The training set was composed of the first 2,000 prints and the test set was the remaining 2,000 prints from NIST-4 (1,000 pairs, so no test prints had corresponding prints in the training data). The primary class labels given with the databases were used, but, since the efficiency is hardly affected, the classifier was only trained to distinguish four classes, treating-arch and tented arch as identical. Table I shows the error rates for various combinations of classifiers. All results presented in this table are weighted, as in Section 5.2, to simulate the natural frequencies of occurrence of the fingerprint classes. Table 2 shows the confusion matrix for the four-classifier neural network fusion with priors, but without the class weighting, and shows the distribution of misclassifications and the error rate for each of the four classes used. Fig. 7 shows the trade-off between error rate and efficiency obtainable by varying the likelihood ratio used for backing-off. 8 PREVIOUS RESULTS
Table 3 shows the results achieved by a number of fingerprint classification systems that have previously been published. These points are plotted in Fig. 8 along with the curve' of possible operating points for the combination presented here and the continuous systems of Cappelli et al. [2], Lumini et al. [10], and Halici and Ongun [8]. For each TABLE 2 Confusion Matrix lor the NIST-4 Test Set (without Reweighting) True Class
A/T W . L
systems for which truly comparable results were available.

Table 4 shows the error rates for the uniform testing conditions and these are plotted in Fig. 7. One limitation ,of some previous works is simply that little can be inferred from the results presented when the test sets are so small or where the test set is not truly independent of the training set. For example, [28] .derive the
1. A weighted average of the class error rates 7.4 percent, 7.3 percent,
Assigned Class L W A/T

637 0 10 5 384 2 66 4 370 .2
R
112 14 4
Error Rate
22.3% 4.5% 4.2%
Rt 8 2 380 3.1% 5.0 percent, 1.4 percent, 6.0 percent, counting Arch/Tented Arch confusions The table shows the assignedclasses for fingerprints falling into each of as correct. For these results, however, where an alternate class label is given (cf. Section 6.1), either answer is considered to be correct, giving a higher the true classes and a per-class error rate. we have used.
accuracy than would be obtained under the same conditions
1173
TABLE 3 A Comparison-of Published Fingerprint Classification Methods

Authors & Year Ohteru et al. 1974 (3) Rao & Balck 1980 [26 Kawagoe & Tojo 1984 [27] Wilson et al. 1993 122] Blue et al. 1994 6) Candela et al. 1995 7] PNN only Pal & Mitra 1996 28] Fitz & Green 1996 (29) Karu & Jain 1996 [4) Classes 3 6 7 5 5 5 5. 5 5 3 4 4 4 4 continuous 4 Efficiency 1.95 3.9 2.6 3.39 3.39 3.39 3.39 3.39 3.6 1.95 3.37 3.37 ' 3.37 3.37 3.33 3.37 Error % 14 8.3 8.5 4.6 7.2 7.8 3.6 9.5 18+ 15 . 7.0 9.4 6.1 8.6 4.0 5.2 Reject Test set 102 good quality 60 94 weighted 2000 NIST-4 weighted 2000 NIST-4 2700 NIST-14 2700 NIST-14 2700 NIST-14 45 (training set) 40 4000 NIST-4 (priors). 4000 NIST-4 (priors) 4000 NIST-4 4000 NIST-9 250 NIST second half of NIST 4
10
10
10% 10% 10% 1.8 %
Halici & Ongun 1996 8) Jain et al. 1999 (9]
test set from the training set using artificial noise processes. Although [22] and [6] use the NIST-4 database, they test on second imprints of the same fingers that were used for training, an unrealistic scenario for which classification by recognition of the fingerprints would result in much lower recognition rates and efficiencies on the order of many thousands. Fitz and Green [29] average over five samples of a fingerprint before attempting classification. A final problem with previous work is that the accuracies of the systems are simply not high enough. If one is to get the full filtering effect of the classification, only the top class must be chosen and it has been seen that the classification accuracies for the top class (no paper presents any other accuracy, such as top 2, etc.) is never high enough to be used in a real system. The higher accuracies that are obtained are achieved by rejecting difficult prints, so the filtering achieved is even lower. Karu and Jain [4] quote the acceptable error rate for the FBI as being 1 percent at 20 percent rejection rate. With four classes, using (9), this is equivalent to a filtering efficiency of 2.816, a performance achieved by the combination classifier described here and by no previous system, as shown in Fig. 7.18 16 14 12
10
CONCLUSIONS
This paper has proposed two new, effective methods for fingerprint classification which do not require core and delta information and which have been designed to work on both dabs and rolled prints. The combination of classifiers described here produces significantly better results than any of the component classifiers. Existing Henry fingerprint classifier accuracies fall short of what is required to make a significant contribution to an AFI system. .This paper has proposed a method for comparing the efficiencies of different classification schemes and describes a system for achieving an arbitrary degree of accuracy from a classification system while evaluating the effect of the trade-off. By this means, current fingerprint classifiers can be rendered of use in an AFI' system. The new classification combination can achieve a.filtering efficiency of 2.8, with an error rate of only 1.0 percent, meeting the FBI requirements for a classification system, and is the first system known to the author to acheive. this. Performance for this Henry system is comparable to the performance .of continuous classifiers and extensions are envisaged to adapt the methods here for non-Henry and continuous classification. ACKNOWLEDGMENTS The author would like to thank the reviewers for their suggestions, and Ruud Bolle, Sharath Pankanti, and Nalini Ratha at IBM for assistance throughout this work. TABLE 4 Comparative Error Rates and Efficiencies for Three Systems on the Second Half of the NIST-4 Data Using True Class Frequency Weightings
System . Error (%) Efficiency
w8
W6
4 2
0.
Fig. 8. A graph of error rate against efficiency for a number of published algorithms.
Combination classifier PCASYS

Jain et at.
5.1 6.1
6.2
3.37 3.37
3.32
1174
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELUGENCE,
REFERENCES
Fed. Bureau of Investigations, The Science of Fingerprints(Classification and Uses), 12-84 ed., US Dept. of Justice, Superintendent of Documents, US Govt. Printing Office, Washington DC, 1984. 12) R. Cappelli, A. Lumini, D. Maio, and D. Maltoni, "Fingerprint SClassification by Directional Image Partitioning," IEEE Trians. Pattern Analysis and Machine Intelligence, vol. 21, no. 5, pp. 402421, May 1999. [3) S. Ohteru, H. Kobayashi, T. Kato, F. Noda, and H. Kimura, "Automated Fingerprint Classifier," Proc. Int'l Conf. Pattern Recognition, pp. 185-189, 1974. [4) K. Karu and A.K. Jain, "Fingerprint Classification," Pattern Recognition, vol. 29, no. 3, pp. 389-404, 1996. [5) D. Maio and D: Maltoni, "A Structural Approach to Fingerprint Classification," Proc. 13th Int'l Conf Pattern Recognition, vol. 3, pp. 578-585, 1996. (6) J.L. Blue, G.T. Candela, P.J. Grother, R. Chellappa, C.L. Wilson, and J.D. Blue, "Evaluation of Pattern Classifiers for Fingerprint and OCR Application," Pattern Recognition, vol. 27, pp. 485-501, 1994. (7) C.T. Candela, P.J. Grother, C.I. Watson, R.A. Wilkinson, and C.L. Wilson, "PCASYS-A Pattern-Level Classification Automation System for Fingerprints," Technical Report NISTIR 5647, Nat'l Inst. of Standards and Technology, Apr. 1995. [8) U. Halici and G. Ongun, "Fingerprint Classification through SelfOrganizing Feature Maps Modified to Treat Uncertainties," Proc. IEEE, vol. 84, no. 10, pp. 1497-1512, Oct. 1996. [9) A.K. Jain, S. Prabhakar, and L. Hong, "A Multichannel Approach to Fingerprint Classification," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 4, pp. 348-359, Apr. 1999. [10] A. Lumini, D. Maio, and D. Maltoni, "Continuous versus Exclusive Classification for Fingerprint Retrieval," Pattern Recognition Letters, vol. 18, pp. 1027-1034, 1997. [Ill) T. Kamei and M. Mizoguchi, "Fingerprint Preselection Using Eigenfeatures," Proc. Computer Vision and Pattern Recognition, pp. 918-923,1998. [12] C.I. Watson and CL. Wilson, "NIST Special Database 4: Fingerprint Database," technical report, Nat'l Inst. of Standards and Technology, Mar. 1992. [113] M.P. Perrone and S.D. Connell, "K-Means Clustering for Hidden Markov Models," Proc. Int'l Workshop Frontiers in Handwriting Recognition, no. 7, pp. 229-238, 2000. 114) A. Senior, "A Hidden Markov Model Fingerprint Classifier," Proc. Asilomar Conf. Signals, Systems, and Computers, 1997. [15) L.R. Rabiner and B.H. Juang, "An Introduction to Hidden Markov Models," IEEE ASSP Magazine, vol. 3,; no: 1, pp. 4-16, Jan. 1986. (16) P.C. Woodland, J.J. Odell, V.V. Valtchev, and S.J. Young, "Large Vocabulary Continuous Speech Recognition Using HTK," Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing, vol. 2, pp. 125-128, Apr. 1994. [17) N.K. Ratha, K. Karu, S. Chen, and A.K. Jain, "A Real-Time Matching System for Large Fingerprint Databases," IEEE Trans. PatternAnalysis iand Machine Intelligence, vol. 18, no. 8, pp. 799-813, Aug. 1996. 1181 O.E. Agazzi, S.-S. Kuo, E. Levin, and R. Pieraccini, "Connected and Degraded Text Recognition Using Planar Hidden Markov Models," Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing, vol. V, pp. 113-116, 1993. [19) K.S. Nathan, A. Senior, and J. Subrahmonia, "Initialization of Hidden Markov Models for Unconstrained On-Line Handwriting Recognition," Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing, vol. 6, pp. 3503-3506, 1996. [20] Y. Amit, D. Geman, and K. Wilder, "Joint Induction of Shape Features and Tree Classifiers," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 11, pp. 1300-1305, Nov. 1997. [21) L. Breiman, Classification and Regression Trees. Wadsworth Int'l Group, 1984. 122) C.L. Wilson, G.T. Candela, and C.I. Watson, "Neural Network Fingerprint Classification," J. Artificial Neural Networks, vol. 1, no. 2, pp. 203-228, 1993. [23) R.M. Bolle, S. Pankanti, and Y.-S. Yao, "System and Method for Determining the Quality of Fingerprint Images," US Patent 5963656, Oct. 1999. 124) A. Senior and T. Robinson, "An Off-Line Cursive Handwriting Recognition System," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 3, pp. 309-321, Mar. 1998. [1]
[25) D.P. Mandal, C.A. Murthy, and S.K. Pal, "Formulation of a Multivalued Recognition System," IEEE Trans. Systems, Man, and SCybernetics, vol. 22, no. 4, pp. 607-620, July/Aug. 1992. [26) K. Rao and K. Balck; ."Type Classification of Fingerprints: A Syntactic Approach," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 2, no. 3, pp. 223-231, May 1980. [27) M. Kawagoe and A. Tojo, "Fingerprint Pattern Classification," Pattern Recognition, vol. 17, no. 3, pp. 295-303, 1984. [28) S.K. Pal and S. Mitra, "Noisy Fingerprint Classification Using Multilayer Perceptron with Fuzzy Geometrical and Textural: Features," Fuzzy Sets and Systems, vol. 80, pp. 121-132, 1996. Green, "Fingerprint Classification Using a [29) A.P. Fitz and R.J. Hexagonal Fast Fourier Transform," Pattern Recognition, vol. 29, no. 10, pp. 1587-1597, 1996. Andrew Senior received the PhD degree from Cambridge University in 1994 for his work on "O-Line Cursive Handwriting Recognition Using Recurrent Neural Networks." Previously, he worked on continuous speech recognition at the French research lab LIMSI. Joining IBM's T.J. Watson Research Center 'in 1994, he continued research into handwriting recognition, before joining the Exploratory Computer Vision GropJn where he ha~ worked on finnmrint and more recently, face recognition. He has created IBM's face detection, tracking and recognition system and cofounded IBM's audio-visual speech effort. He is the author of numerous scientific papers, and US patents and is workshop chair for the IEEE 2001 Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in RealTime Systems.
SFor more information on this or any other computing topic, please visit our Digital Library at http://computer.org/publications/dlib.
~bg--
BEST AVAILABLE COPY Eigenfaces for Recognition.

Matthew Turk and Alex Pentland
Vision and Modeling Group The Media Laboratory Massachusetts Institute Of Technology
Abstract
We have developed a near-real-time computer systemi that can locate and track a subjects head, and then recognize the person by comparing characteristics of the face to those of known individuals. The computational approach taken in this system is motivated by both physiology and information theory. as well as by the practical requirements of near-real-time per. formance and accurict. Our approach treats the face recognition problem as an intrinsically two.dimensional (2-D) recognition problem rather than requiring recovery of threedimensional geometry, taking advantage of the fact that faces are normally upright and thus may be described by a small set of 2-D characteristic views. The system functions by projecting face images onto a feature space that spans th signilicant variations among known face images. The significant features are known as "eigenfaces," because they are the eigenvectors (principal components) of the set of faces; they do not necessarily correspond to features such as eyes, ears. and noses. The projection operation characterizes, an individual face by a weighted sum of the eigenface features, and so to recognize a particular face it is necessary only to compare these weights to those of known individuals. Some particular advanrtages of our approach are that it provides for the ability to learn and later recognize new faces in an unsupervised manner, and that it is easy to implement using a neural network architec.iare. U
INTRODUCTION The face is our primary focus of attention' in social intercourse, playing a major role in conveying identity and emotion. Although the ability to infer intelligence or. character from facial appearance is suspect, the human ability to recognize faces is remarkable'. We can recognize thousands of faces learned throughout our lifetime and identify familiar faces at a glance even after years of separation. This skill is quite robust, despite large changes in the visual stimulus due to viewing conditions, expression, aging, and distractions 'such as glasses or changes in -hairstyle or facial hair. As a consequence the visual processing of human faces has fascinated philos-' ophers and scientists for centuries, including figures such, as Aristotle and Darwin.. Computational models of face recognition, in particular, are interesting because they can contribute not only to theoretical insights but also to practical applications'. Computers that recognize faces could be applied to a wide variety of problems, including criminal identification, security systems, image and. film processing, and human-computer interaction. For example, the ability to model a particular face and distinguish it from a large number of stored face models would nuke it possible to vastly improve criminal identification. Even the ability to merely detect faces, as opposed to recognizing them.
3cdjuse(m Insrituie of Tecrno(oin, 1991 Alan
can be important. Detecting faces in photographs, for instance, is an. important problem in automating color film development, since the effect of many enhancement and noise reduction techniques depends on the picture content (e.g., faces should not be tinted green, while perhaps grass should). Unfortunately, developing a computational model of face recognition is quite difficult, because faces are complex, multidimensional, and meaningful visual stimuli. They are a' natural class of objects, and stand in stark contrast to sine wave gratings. the "blocks world," and other artificial stimuli used in human and computer vision research (Davies. Ellis, & Shepherd, 1981). Thus unlike most early visual fuinctions, for which we may construct detailed models of retinal or striate activity, face recognition is a very high level task for which conputational approaches can currently only suggest broad constraints on the corresponding neural activity. We therefore 'focused our research toward developing a sort of early, preattentive pattern recognition capability that does not depend on having three-dimensional information or detailed geometry. Our goal, which we believe we have reached, was to develop a computational model of face recognition that is fast, reasonably simple. and accurate in constrained environments such as an office or a household. In addition the approach is biologically implemeniable and is in concert with prelirniJournalof Cognitive Neuroscience Volunme 3, ,Number 1
',1 *
.'T
nary findings in the physiology and psychology of face

recognition.
The scheme is based on an information theory approach that decomposes face images into a small set of characteristic feature images called "eigenfaces," which may be thought of as the principal components of the initial training set of face images. Recognition is performed by projecting a new image into the subspace spanned by the eigenfaces ("face space") and then classifying the face by comparing its.position in face space with the positions of known individuals. Automatically learning and later recognizing new faces is practical within this framework. Recognition under widely varving conditions is achieved by training on a limited number of characteristic views (e.g.. ai "straight on" view, a -i5view, and a profile view). The approach has advantages over other face recognition schemes in its speed and simplicity, learning capacity, and insensitivity to small or gradual changes in the face image. Background and Related Work Much of the work in computer recognition of faces has focused on detecting individual features such as the eyes, nose, mouith, and head outline, and defining a face model by the position, size, and relationships among these features. Such approaches have proven difficult to extend to multiple views, and have often been quite fragile, requiring a good initial guess to guide them. Research in human strategies of face recognition, moreover has shown that individual features and their immediate relationships comprise an insufficient representation to account for the performance of adult. human face identification (Carey & Diamond, 1977). Nonetheless,. this approach to face recognition remains the most popular one in the computer vision literature. Bledsoe (1966a.b) was the first to attempt semiautomated face recognition with a hybrid human--computer system that classified faces on the basis of fiducial inmarks entered on p)hotographs by hand. Parameters for the classification were normalized distances and ratios among points such as eve corners, mouth corners, nose tip, and chin point. Later work at Bell Labs (Goldstein. Harmon, & Lesk, 1971: Harmon, 1971) developed a vector of up to 21 features. and recognized faces using standard pattern classification techniques. The chosen features were largely subjective evaluations (e.g., shade of hair, length of ears, lip thickness) made by human subjects, each of which would be quite difficult to automate. An early paper by Fischler and Elschlager (1973) attempted to measure similar features automatically. They described a linear embedding algorithm that used local feature template matching and a global measure of fit to lind and measure facial features. This template matching approach has been continued and improved by the recent work of Yuille, Cohen, and Hallinan (1989) (see
72
Yuille, this volume). Their strategy is based on "deformable templates," which are parameterized models of the face and its features in which the parameter values are determined by interactions with the image. Connectionist approaches to face identification seek to capture the configurational, or gestalt-like nature of the task. Kohonen (1989) and Kohonen and Lahtio (1981) describe an associative network with a simple learning algorithm that can recognize (classify) face images and recall a face image from an incomplete or noisy version input to the network. Fleming and Cpttrell (1990) extend these ideas using nonlinear units, training the system by backpropagation. Stonhamn's WISARD system (1986) is a gencral-purpose pattern recognition device based on: neural net principles. It has been applied with some success to binary face images, recognizing both identity and expression. Most connectionist systems dealing with faces (see also Midorikawa. 1988; O'Toole, Millward, & Anderson, 1988) treat the input image as a general 2-D pattern, and can make no explicit use of the conifigurational properties of a face. Moreover, some of these systems require an inordinate number of training examples to achieve a reasonable level of performance. Only very simple systems have been explored to date, and it is unclear how they will scale to larger problems. Others have approached automated face recognition by characterizing a face by a set of geometric parameters and.performing pattern recognition based on the parameters (e.g., Kaya & Kobayashi, 1972; Cannon, Jones, Campbell, & Morgan, 1986; Craw, Ellis, & Lishman, 1987; \Wong, Law, & Tsaug, 1989). Kanade's (1973) face identification system was the first (and still one of the few) systems in which all steps of the recognition process were automated, using a top-down control strategy directed by a generic model of expected feature characteristics. His system calculated a setof facial parameters from a single face image and used a pattern classification technique to match the face from a known get, a purely statistical approach depending primarily on local histogram analysis and absolute gray-scale values. Recent work by Burt (1988a,b) uses a "smart sensing" approach based on multiresolution template matching, This coarse-to-fine strategy uses a special-purpose computer built to calculate multiresolution pyramid images quickly, and has been demonstrated identifying people in near-real-time. This system works well under limited circumstances, but should suffller from the typical problems of correlation-based matching, including sensitivity to image size and noise. The face niodels are built by hand from face images. THE EIGENFACE APPROACH Much of the previous work on automated face recognition has ignored the issue of just what aspects of the face stimulus are important for identification. This suggested to us that an information theory approach of coding and
Volume 3, Number 1
Journalof Cognitir e .Veuroscience
decoding face images may give insight into the information content of face images, emphasizing the significant local and global "features." Such features niay or may not be directly related to our intuitive notion of face features such as the eyes, nose, lips, and hair. This may have important implications for the use of identification tools such as Identikit and Photofit (Bruce, 1988). In the language of information theory, we want to extract the relevant information in a face image, encode it as efficiently as possible, and compare one face encoding with a database of models encoded similarly. A simple approach to extracting the information contained in an image of a face is to somehow capture the variation in a collection of face images, independent of any judgment of features, and use this information to encode and compare individual face images. In mathematical terms, we wish to find the principal components of the distribution of faces, or the eigenvectors of the covariance matrix of the set of face images, treating an image as a point (or vector) in a very high dimensional space. The eigenvectors are ordered, each one accounting'for a different amount of the variation among the face images. These eigenvectors can be thought of as a set of features that together characterize the variation between face images. Each image location contributes more or less to each eigenvector, so that we can display the eigenvector as a sort of ghostly face which we call an eigenface. Some of the faces we studied are illustrated in Figure 1, and the corresponding eigenfaces are shown
of characteristic features or eigenpictures, perhaps an efficient way to learn and recognize faces would be to build up the characteristic features by experience over time and recognize particular faces by comparing, the feature weights needed to (approximately) reconstruct them with the weights associated with known individuals. Each individual, therefore, would be characterized by the small set of feature or eigenpicture weights needed to describe and reconstruct them-an extremely cornpact representation when compared with the images themselves. This approach to face recognition involves the following initialization operations: 1. Acquire an initial setof face images (the training set). 2. Calculate the eigenfaces from the training set, keeping only the M images that correspond to the highest eigenvalues. These M images define the jace space. As new faces are experienced, the eigenfaces can be updated or recalculated. 3. Calculate the corresponding distribution in M-dimensional weight space for each known individual, by projecting their face images onto the 'face space. These operations can also be performed from time to time whenever there is free excess computational capacity. Having initialized the system,- the following steps are then used to recognize new face images: 1. Calculate a set of weights based on the input image and the Maeigenfaces by projecting the input image onto each of the eigenfaces. 2. Determine if the image is a face at all (whether known or unknown) by checking to see if the image is sufficiently close to "face space." 3. If it is aface, classify the weight pattern as either a known person or as unknown. 4. (Optional) Update the eigenfaces and/or weight patterns.
in Figure 2. Each eigenface deviates from uniform gray

where some facial feature differs among the set of training faces; they are a sort of map of the variations between
faces. Each individual face can be represented exactly in

terms of a linear combination of the eigenfaces. Each face can also be approximated using only the "best" eigenfaces--those that have the largest eigenvalues, and which therefore account for the most variance within the set of face images. The best M1eigenfaces span an MA-dimensional subspace-"face space"--of all possible images. The idea of using eigenfaces was motivated by a technique developed by Sirovich and Kirby (1987) and Kirby and Sirovich (1990) for efficiently representing pictures of faces using principal component analysis. Starting with an ensemble of original face images, they calculated a best coordinate system for image compression, where each coordinate is actually an image that they termed an eigenpicture. They argued that. at least in principle, ancollection of face images can be approximately reconstructed by storing a small collection of weights for each face and a small set of standard pictures (the eigenpictures). The weights describing each face are found by projecting the face image onto each eigenpicture. It occurred to us that if a multitude of face images can be reconstructed by weighted sums of a small collection
5.(Optional) If the same unknown face is seen several

times, calculate its characteristic weight pattern and incorporate into the knaown faces.
Calculating Eigenfaces Let a face image I(x.y) be a two-dimensional N by N array of (8-bit) intensity values. An image may also be considered as a vector of dimension N2 , so that a typical image of size 256 by 256 becomes a vector of dimension 65,536. or, equivalently, a point in 65,536-dimensional space. An ensemble of images, then, maps to a collection of points in this huge space. Images of faces, being similar in overall configuration. will not be randomly distributed in this huge image space and thus can be described by a relatively low dimensional subspace. The main idea of the principal compoTurk and Pentland 73
Figure i. (a)Face inmaiges used as the training .et.
nent analysis ( or Karhunen-Loeve expansion) is to find the vectors that best account for the distribution of face images within the entire image space. Tl'hese vectors define the subspace of face images. which we call "face space. Each vector is of length N. describes an N by .V image, and is a linear combination of the original face images. Because these vectors are the eigenvectors of the covariance matrix corresponding to the original face images, and because they are face-like in appearance, we refer to them as "eigenfaces." Some examples of eigenfaces are shown in Figure 2. Let the training set of face images be . _.. ... = Fr..The average face of the set is defined by 2',2', F,,. Each face differs from the average by the vector ), = F, - V. An example training set is shown in Figure la. with the average face T shown in Figure lb. This set .of vetry large vectors is then subject to prin. cipal component analysis, which seeks a set of Mi orthonormal vectors. u,,. which best describes the distribution of the data. The kth vector. uk, is chosen such that
"
"
, 0,
i,
if!. = k otherwise
(2)
The vectors uk and scalars XA. are the eigenvectors and eigenvalues, respectively, of the covariance matrix
C
=.._if
where the matrix A = (4),
...
.,i). matrix C. 'The
iS a mlaximum.m subiect to
a maximum,is to subject
however, is N" by .V, and determining the N2 eigenvectors and eigenvalues is an intractable task for typical image sizes. We need a computationally feasible method to find these eigenvectors. if number of data points in the image space is less the than the dimension of the space .(M1NZ), there will be < only M - 1,rather than N 2, meaningful eigenvectors. (The remaining eigenvectors will have associated eigenvalues of zero.) Fortunately we can solve for the N2dimiensional, eigenvectors in this case by first solving for the eigenvectors of an M by M matrx-e.g., solving a 16 .x 16 matrix rather than a 16.384 x 16,384 matrixVolume 3, Number I
74
Journal of Cognitiv e .Veuroscitence
Following this analysis, we construct the Al by MAl matrix L - AlA, where L,,,,, = 4,,,, and find the Ml.eigenvec. tors, v,, of L. These vectors determine linear combinations of the M training set face images to form the
eigenfaces ui.
U, =
vu
/ = 1, . . ;:,
With this analysis the calculations are greatly reduced, from, the order of the number of pixels in the images (Ni) to the order of the number of images in the training set (M). In practice, the training set of face images will N), and the calculations become be relatively small (l quite manageable. The associated eigenvalies allow us to rank the eigenvectors according to their usefulness in characterizing the variation among the images. Figure 2 shows the top seven eigenfaces derived from the input
images of Figure 1.
Figure 1. (b) The
average
face '1.
Using Eigenfaces to Classify a Face Image The eigenface images calculated from the eigenvectors of L span a basis set with which to describe face images. Sirovich and Kirby (1987) evaluated a limited version of this framework on an ensemble of M = 115 images of Caucasian males, digitized in a controlled manner, and found that about 40 eigenfaces were sufficient for a very good description of the set of face images. With Ml' = 40 eigenfaces. RMS pixel-by-pixel errors in representing cropped versions of'f6ce images were about 2%. Since the eigenfaces seem adequate for describing face images under very controlled conditions, we decided to investigate their usefulness as a tool for face identification In practice, a smaller M' is sufficient for identification, since accurate reconstruction of the image is not a requiirement. In this framework, identification becomes a pattern recognition task. The eigenfaces span an M'dimensional subspace of the original NV'image space. The MAl' significant eigenvectors of the L matrix are chosen as those with the largest associated eigenvalues. In many of our test cases. based on Al = 16 face images, l' = 7 eigenfaces were used. A new face image (r) is transformed into itseigenface components (projected into "face space") by a simple operation,
Wk = u[( rq')
Figure 2. Seven of the eigenfaces calculated from the input images
of Figure .,
and then taking appropriate linear combinations of the face images ,;. C6nsider the eigetwvectors v: of A'A .-uch that
AfAv, = ,v, t-t )
PremultiplVing both sides by A. we have

.AA v, = -\v, t5)
from which we see that Av, are the eigenvectors of C =

AA.
for k = 1....A1'. This describes a set of point-b)y-p6int , image multiplications and summations, operations performed at approximately frame rate on current image processing hardware. Figure 3 shows an image and its projection into the seven-dimensional face space. The weights form a vector f1. = (w,, w2 . .. w.r] that describes the contribution of each eigenface in representing the input face image, treating the eigenfaces as a basis set for face images. The vector may then be used
Turk and Pentland 75
in a standard pattern recognition algorithm to find which of a number of predefined face classes, if aiy, best describes the face. The simplest method. for determining
which face class provides the best description of an input face image is to find the face class k that minimizes the
with the highest associated eigenvalues. (Let
Al'
= 10 in
this example.) 3. Combine the normalized training set of images according to Eq. (6) to produce the (M' = 10) eigenfaces
u
Euclidian distance = (
IO)11 A
(8)
where lk is a vector describing the kth face class. The face classes fl are calculated by averaging the results of the eigenface representation over a small number of face
images (as few as one) of each individual. A face is classified as belonging.to class k when the minimum Ek
is below some chosen threshold 0. Otherwvise the face is classified as ':unknown," and optionally used to create a new face class. Because creating the vector of weights is equivalent to projecting the original face image onto the low-dimensional face space: many images (most of them looking nothing like a face) will project onto a given pattern
vector. This is not a problem for the system, however,
4. For each known individual, calculate the class vector Ok by averaging the eigenface pattern vectors (from Eq. (8)) calculated from the original (four) images of the individual. Choose a threshold 0, that defines the maximum allowable distance from any face class, and a threshold 06.that defines the maximum allowable distance from face space [according to Eq. (9)]. 5. For each new face image to be identified, calculate its pattern vector Q, the distances E, to each known class, and the distance e to face space. If the minimum distance ek < 08 and the distance E < 0,, classify the input face as the individual associated with class vector St.. If the minimum distance e > 0, but distance E< 0,, then the
image may be classifed as "unknown," and optionally
since the distance E between the image and the face space is simply the squared distance between the meanadjusted input image ( = F its projection onto face space:
E=
T and
Or
1 w ,u;,
used to begin a new face class. 6. If the new image is classified as a known individual. this image may be added to the original set of familiar face images, and the eigenfaccs may be recalculated (steps 1-4). This gives the opportunity to modify the face space as the system encounters more instances of known
feces .
ib-
4'dir
Thus there are four possibilities for an input image and its pattern vector: (1) near face space and near a face class, (2) near face space but not near a known face class, (3) distant from face space and near a face class, and (4) distant from face space and not near a known face class. In the first case, an individual is recognized and identified. In the second case, an unknown individual is present. The last two cases indicate that the image is not a face image. Case three typically shows up as a false positive in most recognition systems: in our framework, however, the false recognition may be detected because of the significant distance between the image and the subspace of expected tface images. Figure 4 shows some images and their projections into face space and gives a measure of distance from the face space for each. Summary of Eigenface Recognition Procedure To summarize, the eigenf aces approach to face recognition involves the following steps: 1. Collect a set of characteristic face images of the known individuals. This set should include a number of images for each person, with some variation in expression and in the lighting. (Say four images of ten people,
so .M' =
In our current system calculation of the eigenfaces is done offline as part of the training. The recognition currently takes about 400 msec running rather inefficiently in Lisp on a Sun4, using face images of size 128 x 128. With some special-purpose hardware, the current version could un' at close to frame rate: (33 msec). Designing a practical system for face recognition within this framework requires assessing the tradeoffs between generality, required accuracy, and speed. If the face recognition task is restricted to a small set of people (such as the members of a family or a small company), a small set of cigenfaces is adequate to span the faces of, interest. If the system is to learn new.faces or represent many people. a larger basis set of eigenfaces will be required. The results of Sirovich and Kirby (1987) and Kirby and Sirovich (1990) for coding of face images gives some evidence that even if it werenecessary to represent a large segment of.the population, the numnher of eigenfaces needed would still he relatively small.
Locating and Detecting Faces
-10.)
2. Calculate the (40 x 40) matrix L, find its eigenvectors and eigenvalues, and choose the MAl' eigenvectors
The analysis in the preceding sections assumes we have a centered face image, the same size as the training images and the eigenfaces. We need some way, then, to locate a face in a scene to do the recognition. We have developed two schemes to locate and/or track faces, using motion detection and manipulation of the images in "face space".
76
Journalof Cognitive Neuroscience
Volumerr
3,
Number
__
Figure 3. An original face image and its projection onto the facespace delined by the eigenfaces of Figure 2.
Motion Detecting and Head Tracking People are constantly moving. Even while sitting, we fidget and adjust our body position, nod our heads, look around, and such. In the case of a single person moving in a static environment, a simp!e motion detection and tracking algorithm, depicted in Figure 5, will locate and track the position of the head. Simple spatiotemporal filtering (e.g., frame 'differencing) accentuates image locations that change with time, so a moving person "lights up" In the filtered image. If the image "lights up" at all, motion is detected and the presence of a person is postulated. Ater thresholding the filtered image to produce a binary motion image, we analyze the "motion blobs" over time to decide if the motion is caused by a person moving and to determine head position. A few simple rules are applied, such a:is "the head is the small upper blob above a larger blob (the body)," and "head motion must be reasonably slow and contiguous" (heads are not expected to jump around the image erratically). Figure 6 shows an image with the head located, along with the path of the head in the preceding sequence of frames. The motion image also a llows for an estimate of scale. The size of the blob that is assumed to be the moving head determines the size of the subimage to send to the recognition stage. This subimage is rescaled to fit the dimensions of the eigenfaces.
faces from motion (e.g., if there is too little motion or many moving objects) or as a method of achieving more precision than is possible by use of motion tracking alone. This method allows us to recognize the presence of faces apart from the task of identifying them. As seen in Figure 4, images of faces do not change radically when projected into the face space, while the projection of nonface images appears quite different. This basic idea is used to detect the presence of faces in a scene: at every location in the image, calculate the distance E between the local subimage and face space
This distance from face space is used as a measure of

"faceness." so the result of calculating the distance from face space at even point in the image is a ''face map' e(x,y). Figure 7 shows an image and its face map-low values (the dark area) indicate the presence of a face. Ulnfortunatelyv, direct application of Eq. (9) is rather expensive. We have therefore developed a simpler, more efficient method of calculating the face map E(x '), which is described as follows. To calculate the face map at every pixel of an image (xy), we need to project the subimage centered at that pixel onto face space, then subtract the projection from the original. To project a subimage r onto face space. we must first subtract the mean image. resulting in i = r = V. With At being the projection of 0 onto face space. the.distance measure at a given image location is then (t
Using "FaceSpace" to Locate the Face We can also use knowledge of the face space to locate faces in single images. either as an alternative to locating
-I
r) 7QF
4)r)
T( ---Ofo (D
(10.)
Trk anti Pentland.
77
Note to the reader: There are typos here that have haunted me for years, so here is a brief correction. First, the third line in equation (10) should not have a plus sign - rather, the plus should be replaced by a negative. Also,
the second term of the fourth line should be
OTq f rather
than
$Of.
This carries into equation (11). However, since the last term in the third line of equation (10) is equal to zero (due to the fact they're Sperpendicular), this means that these two terms are actually equivalent i.e.,
So even though the wrong terms are written in the derivation, it's actually still correct. Matthew Turk
Figure 4. Three images and their projections onto the face space defined by the eigenfaces of Figure 2. The relative measures of distance from face space are (a) 29.8, (b) 58.5. (c) 5217.4. Images (a) and (b) are in the original training set.
(r). Because 4f is a linear comnbin; since Wr 1 (4 = of the eigenfaces (Or = -. w;u,) and the eigen faces are orthonormal vectors.,
4
r.
I -,p
ition
-1
(x,y) =
=im'(x, v)u,
= :.[F(x, ) - r)ru = > ,([(.4;y)u, - IWuI, u; 'u;] = 1)., I((.1y')
(13)
(11)
where @ is the correlation operator. The first term of Eq.. (12) becomes
and E(.;y) = FT ; ,) 1)(x,.) ) (14)
X &;(.v..y. (i2)
so
that
Ei,')=[
where e(x, y) and w(.x,y) are scalar functions of image location, and Q(x,y) is a vector function of image loc'ltion. The second term of Eq. (12) is calculated in practice by a correlation with die L eigenfaces:
78
T(
;y)r(.r. y)
2r(...')
+~'14 +
15
(F'(x j:Ou, - w p ul (
Jor1rnalof Cognilive eur-oscience
Vo(,unw 3. VNmber I
--
Figure 5.
T'h'e
hcad
ira king
ntl Itxaing ;sriein.
works, these computations can be implemented by simple neural network. Learning to Recognize New Faces The concept of face'space allows the ability to learn and subsequently recognize new faces in an unsupew'ised manner. When an image is sufficiently close to face space but is not classified as one of the familiar faces, it is initially labeled as "unknown." The computer stbres the pattern vector and the corresponding unknown image. if a collection of "unknown" pattern vectors cluster in the pattern space, the presence of a new but unidentified face is postulated. The images corresponding to the pattern vectors in the cluster are then checked for similarity by requiring that the distance from each .'imige to the mean of the images is less than a predefined threshold. If the images pass the similarity test, the average of the feature vectors ,.is added to the database of known faces. Occasionally, the eigenfaces may be recalculated using these stored images as part of the new training set, Other Issues A number of other issues must be addressed to obtain a robust working system. In this section we. will briefly mention these issues and indicate methods of solution.
Eliminating the Background
Figure 6. The head has been located-the image in the box is ;ent to the face recognition process, Also shown i. the path of the head tracktd. over several previous I'rames.
Since the average face and the eigenlices u, are fixed. the terms 'Wr. and 0 u, may be computed ahead of time. Titus the computati()on nOf the tee nmap inyolves onl" L + I correlations over the input image and the computation of the first iernim r'(.L, ) y). This is comnputed by squaring the input ima,e I(.Xv, y.) and. at each image location. summing the squared values of the local subimage. As discussed in the section on Neural Net-
In the preceding analysis we have ignored the effect of the background. In practice, the background can significantly effect the recognition performance, since the eiTurk and Pentland 79
Figure 7. (a) Original image. (b) The corresponding face map, where low values (dark areas) indicate the presence of a face.
genface analysis as described above does notdistinguish the face from the rest of the image. in the experiments described in the section on Experiments with Eigenfaces., the background was a significant part of the image used to classify the faces. To deal with this problem without having to solve other difficult vision problems (such as robust segmentation of the head), we have multiplied the input face image by a two-dimensional gaussian window centered on the face, thus diminishing the background and accentuating the middle of the face. Experiments in human strategies of face recognition (Hay & Young, 1982) cite the importance. of the internal facial features for recognition of familiar faces. Deemphasizing the outside of the face is also a practical consideration since changing hairstyles may otherwise negatively affect the recognition.
scale the input image to multiple sizes and use the scale that results in the smallest distance measure to face space. Although the eigenfaces approach is not extremely of sensitive to head orientation (i.e.:, sideways tilt the head), a non-upright view will cause some performance degradation.. An accurate estimate of the head tilt will certainly benefit the recognition. Again, two simple methods have been considered and tested. The first is to calculate the orientation of the motion blob of the head. This is less reliableas the shape tends toward a circle, however. Using the fact that faces are reasonably symmetric patterns, at least for frontal views, we have used simple symmetr, operators to estimate head orierntation. Once the orientation is estimated; the image can be rotated to align the head with the eigenfaces. Distribution in Face Space The nearest-neighbor classification previously described assumes a Gaussian distribution in face space' of an individual's feature vectors O. Since there is no a priori reason to assume any particular distribution, we want to characterize it rather than assume it is gaussian. Nonlinear networks such as described in Fleming and Cottrell (1990) seem to be a promising way to learn the face space distributions by example. Muldtiple Views We *arecurrently extending the system to deal with other than full frontal views by defining a limited number of face classes for each known person corresponding to characteri-istic views. For example, an individual may be represented by face classes corresponding to a frontal
V'olumne X Number lw
Scale (Head Size) and Orlentation Irwariance The experiments in the section on Database of Face images show that recognition performance decreaises quickly as the head size, or scale, is misjudged. The head size in the input image must be close to that of the eigenfaces for the system to work well. The motion :analysis gives an estimate of head size, from which the face image is rescaled to the eigenface size. Another approach to the scale problem, which may be sel)arate from or in addition to the motion estimate, is to use multiscale eigenfaces, in which an input lFace image is compared with eigenfaces at a number of scales. In this case the image will appear to be near the face space of only the closest scale eigenfaces. Equivalently, we can
80 Journal of Cognitive leuroscience
face view, side views, at -45, and right and left profile views. Under most viewing conditions these seem to be sufficient to recognize a face anywhere from frontal to profile view, because the real view can be approximated by interpolation among the fixed views.
EXPERIMENTS WITH EIGENFACES To assess the viability of this approach to face'recognilion, we have performed experiments with stored face
images and built a system. to locate and recognize faces in a dynamic environment. We first created a large da:tabase of face images collected under a wide range of imaging conditions. Using this database we have conducted several experiments to assess the performance under known variations of lIghting, scale, and orientation. The results of these experiments and early experience with the near-real-time system are reported in this section.
Database of Face Images
The images fronm Figure la were taken from a database of over 2500 face images digitized under controlled conditions. Sixteen subjects were digitized at all combinations of three head orientations, three head sizes or scales, and three lighting conditions. A six level Gaussian pyramid was constructed for each image, resulting in image resoltition from 512 x 512 pixels down to 16.x 16 pixels. Figure 8 shows the images from one pyramid level for one individual. In the first experiment the effects of varying lighting, size, and head orientation were investigated using the complete database of 2500 images of the 16 individuals shown in Figure la. Various groups of 16 images were selected and used as the training set. Within each training set there was one image of each person, all taken under the samrne conditions of lighting, iluage Siz, anrd head orientation. All images in the database were then classifled as being one of these sixteen individuals (i.e., the threshold 0, was effectively infinite, so that no faces were rejected as unknown). Seven eigenfaces were used in the classification process. Statistics were collected measuring the mean accuracy as a function of the difference between the training conditions and the test conditions. The independent variables were difference in illumination, imaged head size, head orientation, and combinations of illumination, size. and orientation. Figure 9 shows results of these experiments for the case of infinite 0,. The graphs of the figure show the number of correct classifications for varying conditions of lighting, size. and head orientation, averaged over the number of experiments. For this case where every, face image is classified as known, the system achieved approximately 9605 correct classification averaged over
lighting variation, 85% correct averaged over orientation variation, and 64% correct averaged over size variation. As can be seen from these graphs, changing lighting conditions causes relatively few errors, while performance drops dramatically with size change. This is not surprising, since under lighting changes alone the neighborhood pixel correlation remains high, but under size changes the correlation from. one image to another is largely lost. It is clear that there is a need for a multiscale approach, so that faces at a particular size are compared with one another. One method of accomplishing this is to make sure that each "face class" includes images of the iidividual at several different sizes, as was discussed in the section on Other Issues. In a second experiment the same procedures were followed. but the acceptance threshold 6, was also varied, At low values of 0E, only images that project very closely to the known face classes will be recognized, so that there will be few errors but many of the images will he rejected as unknown. At high values of O.most images will be classified, but there will be more errors: Adjusting 0eto achieve 100% accurate recognition boosted the unknown rates to 19% while varving lighting, 39% for orientation, and 60% for size. Setting the unknown rate arbitrarily to 20% resulted in correct recognition rates of 100%. 94%. and 7".% respectively. These experiments show an increase of performance accuracy as the threshold decreases. This can be tuned to achieve effectively perfect recognition as the threshold tends to zero, but at the cost of many images being rejected as unknown. The tradeoff between rejection rate and recognition accuracvy will be different for each of the various face recognition applications. 1lowever. what would be most desirable is to have a way of setting the threshold high, so that few known face images are rejected as unknown. while at the same.time detecting the incorrect classifications. That is, we would like to increase the efficiency (.the d-prime) of the recognition
process.
Ofie way of accomplishing this is to also examine the (normalized) Euclidian distance between an image aind face space as a whole. Because the projection onto the eigenface vectors is a many-to-one mapping, there is a potentially unlimited number of images that can project onto the eigenfaces in the same manner. i.e.. produce .the same weights. Many of these will look nothing like a face. as shown in Figure -tc. This approach was described in the section on Using "Face Space" to Locate the Face as a method of identifying likely face subimages. Real-Time Recognition We have used the techniques described above to build near-reala system that locates and recognizes faces ,in in a reasonably unstructured environment. Figure time 10 shows a diagramn of the system. A fixed camera, mon-' itoring part of a room, is connected to a.Datacube image
Tur
an Petlad
FIgure 8. Vari3tion of face in:tges foi- onr individu:aI: rhree head .'izvs'. threc lighting conditins.
n:d three head orienuraions.
processing system, which resides on the bus of a. Sun 3/ 160. The Datacube digitizes the video image and per. forms spatiotemporal filtering, thresholding, and subsampling at frame rate (30 frames/sec)., (The images are subsampled to speed up the motion analysis.) The motion detection and analy'sis programs run on the Sun 3/160, first detecting a moving object and then tracking the motion and applying simple rules to deter. mine if it is tracking a head. When a head is found. the subimage, centered on the head, is sent to another computer (a Sun Sparcstation) that is running'the fatce recognition program (although it could be running on the same computer as the motion program). Using the distance-from-face.space measure. the image is either re-
jected as not a face, recognized as one of a group of familiar faces. or determined to be an unknown face. Recognition occurs in this system at rates of up to two or three times per second. Until motion is detected, or as long as the image is not perceived to be a face, there is no output.. When a face is recognized, the image of the identified individual is displayed on the Sun monitor. RELATIONSHIP TO BIOLOGY AND
NEURAL NETWORKS
Biological Motivations High-level recognition tasks are typically modeled as requiring mnianv stages of processing, e.g., the Marr (1982)
82
Journal of Cognitive Neurosciunce
Volume
3,
NVumber
1"."1.0 ? \\
1.6 7.36 .. '.0
1636
1
(b) (a)
11.8 1.06 11.0 2.66 2.
6.3M -. 0r 0 6.36 .4
(f)
(M.
cells were sensitive to identity, some to "faceness," and some only to particular views (such as frontal or profile). Although we do not claim that biological systems have "eigenface cells" or process faces in the same way as the eigenface approach, there are a number of qualitative similarities between our approach and current understanding of human face recognition. For instance, relatively small changes cause the recognition to degrade gracefully, so that partially occluded faces can be recognized, as has been demonstrated in single-cell recording experiments. Gradual changes due to aging are easily handled by the occasional recalculation of the eigenfaces, so that the system is quite tolerant to everi large changes as long as they occur over a long period of time. If, however, a large change occurs quickly--e.g., addition of a disguise or change of facial hair-then the eigenfaces approach will be fooled, as are people in conditions of casual observation.
11.0 T
62.4l
..
Neural Networks
I "- rt-
1.6 7.10 eIO t
Figure 9. Resuts of experiments measuring recognition performance using eigenfaces. Each graph shows averaged performance as the lighting conditions, head aize. and head orientation vary-the y-. axis depicts number of correct classifications tout of 16). The peak (1616 correct) in each graph results from recognizing the particular training set perfectly. The other twvo graph points reveal the decline in performance as lthe following parameters are varied: (a) lighting, (b) head size (scale),(c) orientation, (d) orientation and lighting, (e) orientation and size ( #1i. orientation and size ( #2), (g) (f) size and lighting. (h) size and lighting (#2). 1wt, W1
Although we have presented the eigenfaces approach to face recognition as an informination-processing model, it may be implemented using simple parallel computing elements, as in a connectionist system or artificial neural network. Figure 11 shows a three-layer 1 fully connected , , linear network that implements a significant part of the system. The input layer receives the input (centered and normalized) face image, with one element per image pixel, cir N elements..'The weights from the input layer to the hidden layer correspond to the eigenfaces, so that the value of each hidden unit is the dot product of, the input image and the corresponding eigenface: w; = r u,. The hidden units, then, form the pattern vector f =
. .
WE].
The output laver produces.the face space projection of the input image when the output weights also correparadigm of progressing from images to surfaces tc spond to the eigenfaces (mirroring the input weights). three-dimensional models to matched models. However , Adding two nonlinear components we construct Figure the early development and the extreme rapidity of face 12, which produces the pattern class face space projection (r, distance measure d (between the image and recognition makes it appear likely that there must alsc be a recognition mechanism based on some fast, low. S its face space projection), and a classification vector. The level, two-dimensional image processing. classification vector is comprised of a unit for each On strictly phenomenological grounds, such a face known face defining the pattern space distances e;. The recognition mechanism is plausible because faces arc unit with the smallest value, if below the specified thresh. typically seen in a limited range of views, and are a vern .old 0e, reveals the identity of the input face imlage. important stimulus for humans from birth. The existence Parts of the network of Figure 12 are similar to the of such a mechanism is also supported by the results o.f ., associative networks of Kohonen (1989) and Kohonen a number of physiological experiments in monkey cortex and Lehtio (1981). These networks implement a learned claiming to isolate neurons that respond selectively t. stimulus-response mapping, in which the learning phase faces (e.g., see Perrett. Rolls. & Caan, 1982. Perrett, Mist - modifies the connection weights. An autoassociative.netlin, & Chitty. 198': Bruce, Desimone. & Gross. 1981 S work implements the projection onto face space. SimiDesimone. Albright, Gross, & Bruce, 1984; Rolls, Baylis , larly, reconstruction using eigenfaces can be used to Hasselmo, & Nalwa. 1989). In these experiments, somet "recall a partially occluded face, as shown in Figure 13.
fl,
Turk and Pentland
8.3
II
Figure 10. System diagram of the face recognition system.
Figure 11. Three-layer linear network for eigenface calculation. The symmetric weights u, are the eigenfaces, and the hidden units reveal the projection of the input image ( onto the eigenlaces. The output (t is the face space projection of the input image.
Output layer Qr Ui
I
ui Hidden layer
f2
Input layer
)
eigenface approach does provide a practical solution that is well fitted to the problem of face recognit.ion. It is fast. relatively simple, and has been shown to work well in a constrained environment. It can also be implemented using modules of connectionist or neural networks. It is important to note that many applications of face recognition do not require perfect identification, although most require a low false-positive rate. In searching a large databa:se of faces, for example, it may be preferable to flind a small set of likely matches to present to the user. For applications such as security systems or human-computer interactio6rn. the sy'stem will normally be able to "view" the subject for a few seconds or minutes, and thus will have a number of chances to recognize the person, Our experiments show that the eigenface technique can be made to perform at very high accuracy, although with a substantial "unknown" rejection rate, and thus is potentially well suited to these applications. We are currently investigating in more detail the issues of robustness to changes in lighting, head size, and head orientation, automatically learning new face;. incorpoVolume 3, VNumber I
CONCLUSION Early attempts at making computers recognize faces were limited by the use of impoverished face models and feature descriptions (e.g., locating features from an edge image and matching simple distances and ratios), assuming that a face is no more than the sum of its parts, the individual features. Recent attempts using parameterized feature models and multiscale matching look more promising, but still face severe problems before they are generally applicable. Current connectionist approaches tend to hide much of the pertinent information in the weights that makes it difficult to modify and evalua:te panrts of the approach. The eigenface approach to face recognition waLs motivated by information theory, leading to the idea of basing face recognition on a small set of image features that best approximates the set of known face images. without requiring that they correspond to our intuitive notions of facial parts and features. Although it is not an elegant solution to the general recognition problem. the
84
fournal of Cognitiie Neuroscience
Figure 12. Collection of networks to implement computation of the pattern vector, projection into face space. distance from face space
measure, and identilication.
Figure 13. (a) I':.lnillyv occluded face image and (b)
its reconslrucion
using the 'igeinlices.
rk anl Pentland .
85
rating a limited number of characteristic views for each individual, and the tradeoffs between the number of people the system needs to recognize and- the number of eigenfaces necessary for unambiguous classification. In addition to recognizing faces, we are also beginning
efforts to use eigenface analysis to determine the gender
tion iri biological and technical ystem. Berlin: Springer-
Verlag.
Hay, D. C., & Young, .k W. (1982). The human face. in A. W. Ellis (Ed.), Normality and patibology in cognitive functions. London: Academic Press. Kanade, T. (1973). Picture processing system by computer complex and recognition of human faces. Dept. of Information Science. Kyoto University. Kaya, Y., &Kobavashi, K. (1972). A basic study on human face recognition. In S. Watanabe (Ed.), Frontiersof pattern rec ognition. New York: Academic Press. Kirby, MN., Sirovich. L (1990). Application of the Karhuneri& Loeve procedure for the characterization of human faces. IEEE Transactions on Pattern Analysiv anti Machine Intelli. gence, 12(1). Kohonen, T. (1989). Selforganization and associative mem or Berlin: Springer-Verlag. Kohonen, T., & .Lehtio, P. (1981). Storage and processing of information in distributed associative memotry systems. In G. E. Hinton & J. A. Anderson (Eds.), Parallel models of associative mnemor: Hillsdale, NJ: Erlbaum, pp. 105-143. Marr, D. (1982). Vision. San Francisco: W. H-. Freeman. Midorikawa, H. (1988). The face pattern identification by back-propagation learning procedure. Abstracts of the First Annrmual INNS Aleeting, Boston, p. 515. O'Toole, Millward. & Anderson (1988). A physical systemn approach to recognition memory for spatially transformed faces. Neural Networks, ,1. 179-199. Perrett, D. L.,Mistlin, A. J., & Chitrv, A. J. (1987). Visual newu rones responsive to faces. TINS, 10(9), 358-364. Perrett, Rolls, & Caan (1982). Visual neurones responsive to faces in the monkey temporal conrtex. hLxperintental Brain Research, 47, 329-342. Rolls. E. T., B3aylis, G. C.. Hasselmo, M. E., & Nalwa, V. (1989). The effectof learning on the face selective responses of neurons in the cortex in the. superior temporal sulcus of the monkey. Experinmental Brain Research, 76, 153-164; Sirovich, L., & Kirby, M. (1987). Low-dimensional procedure S7,fOr,.the characterization of human faces. Journalof the Optical Socit of jAmerica A, 4(3). 519-524. Stonham. T. J. (1986). Practical face recognitioni and verification with WISARD. In H. Ellis. M. Jeeves, F..Newcombe, & A. Young (Eds.), Aspects of face proccsirit. Dordrecht: Martinus Nijhoff. Wong, K., Law. H., & Tsang, P. (1989). A system for recognising human faces. Proceedinrgs of ICASSP, May, 1638-1641 Yuille, A. L, Cohen, D. S., & Hallinan, P. W. (1989). Feature extraction from faces using deformable templates. Proceedings of CVPR.San Diego, CA, June.
of the subject and to interpret facial expressions, two importint face processing problems that complement the task of face recognition. REFERENCES
Bledsoe, W. W. (1966a). The model method in facial recogni.tion. Panoramic Research Inc., Palo Alto, CA. Rep. PRI:15, August. Bledsoe, W W. (1966b). Man-machine facial recognition. Panoramic Research Inc., Palo Alto, CA, Rep. PRI:22, August. Bruce, V. (1988). Recognising faces. .'H-illsdale, NJ: Erlbaum. Bruce, C.J., Desimone, R, & Gross, C. G. (1981).Journal of Neuropl.siolog; 46, 369-384. Burt, P. (1988a). Algorithms and architectures for smart sensing. Proceedings of the Image UnderstandingWorkshop,
April..
Burt. P. (1988b). Smart sensing within a Pramid Vision Mvlachine. Proceedings of IEEE, 76(8), 139-153. Cannon, S. R., Jones, G. W., Campbell, R., & Morgan, N.W. (1986). A computer vision system for identification of individuals. Proceedings of IECON, 1. Carey. S.. & Diamond, R. (1977): From piecemeal to configurational representation of faces. Science, 195, 312-313. Craw, Ellis, & Lishman (1987). Automatic extraction of face features. Pattern Recognition Letters, 5, 183-187. Davies, Ellis, & Shepherd (Eds.). (1981). Perceiving and remernbering fjtaces. London: Academic Press. Desimone, R., Albright, T. D., Grass, C. G.. & Bruce, C. J. (1984). Stimulus-selective properties of inf'rir tempoil. ;. neurons in the macaque. Neuroscience. 4, 2051-2068. Fischler. MNI. & Elschlager, R. A. (1973). The representation A., and matching of pictorial structures. IEEE Transactions on Computers, c-22(1). Fleming, M., & Cottrell. G. (1990). Categorization of faces using unsupervised feature extraction. Proceedings of IJCNVN-
90,2.
Goldstein, Harmon, & Lesk (1971). Identification of human faces. Proceedings IEEE, 59, 748. Harmon, L. D. (1971). Some aspects of recognition of human faces. In O. J. Grusser & R. Klinke (Eds.), Pattern recogni-.
86
Journal of Cognitive Neuroscience
Volume 3,
tNumber
THIS PAGE BLANK (usPTO)
Defects in the images include but are not limited to the items checked:
Q BLACK BORDERS
Q IMAGE CUT OFF AT TOP, BOTTOM OR SIDES
ADED TEXT OR DRAWING
BLURRED OR ILLEGIBLE TEXT OR DRAWING
I SKEWED/SLANTED IMAGES
//
Q COLOR OR BLACKAND WHITE PHOTOGRAPHS
U .GRAY
O
SCALE DOCUMENTS.
LINES OR MARKS ON ORIGINAL DOCUMENT OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY
O REFERENCE(S) U
OTHER:
IMAGES ARE BEST AVAILABLE COPY. As rescanning these documents will not correct the image problems checked, please do not report these problems to the IFW Image Problem Mailbox.
THIS PAGE BLANK (usPo)
FEB22. 2007
2:34PM
MOFO 28TH FL
425 MARKET STREET
SAN FRACICO
MOA~t3OI 4a
NO. 247
OBaRSTR LLP
P. 1
MORRISON IOERSTER
CALIFORNIA 94]05-2488 TBLEPHONE. 415.266.7000 FACSM=- 415.2A875Z W W .MOFO.COM

^ ^" """"'
NBW YORK, $4 14FANCISCO, LOS APIGULBS.HALO ALTO,

SAN
1;I200. WAAWINOTON, D.C. 'DSNVBR, 1NQRTI1UN VIRGIIA,

ORANGE COUNTY, IACR.AUBNTO, WALNUT C1PUIC, CENTURV CITY TOKCYO, LONDlON, BRIJTNO. iI q, 1 VJP'.FItowe " II
SINGAPOAS, iHAMOHAT, nOd
3CEVE
DCIE m Ax
IkUSSILS KONG,
i'.ro
NAME: Centralized Facsimile Center US Patent and Trademark Office FROM: Peter J. Ylm 4
FACSIMILE: 571-273-8300oo
TELEPHONE:
322 2007
C (ENTER
DATE:
February 22, 2007
Number of pages with cover page:
Originals Will NOT Follow
Preparer of this slip has confirmed that facsimile number given is correct: 656vlhl Comments: Attorney Docket: 577832000200 Art Unit: 2621
Examiner: Not assigned yet

Application No. 11/231,353 Filed: September 19,2005 Title: INCREMENTAL DATA FUSION AND DECISION MAKING SYSTEM AND ASSOCIATED METHOD Inventor: Ken P. CHENG, et al. Documents Enclosed: I. Request To Withdraw As Attorney Or Agent And Change Of Correspondence Address, in triplicate, 3 pages To ensure compliance with requirements imposed by the United States Internal Revenue Service, Morrison & Foerster LLP informs you that, if any advice concerning one or more U.S. Federal tax issues is contained in this facsimile (incliuding any attachments), such advice is not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. +++++****************************************** CAUTION -CONFIDENTIAL This facsimile contains confidential information that may also be privileged. Unless you are the addressee (or authorized to receive for the addressee); you may not copy, use, or distribute it. If you have received it in error, please advise Morrison & Foerster LLP immediately by telephone or facsimile and return it pronmptly by mail.
IF YOU DO NOT RECEIVE ALL OF THE PAGES, PLEASE CALL VICKI HENRY AT 415.268.6023 AS SOON AS POSSIBLE,
SF-2273872
PAGE 114' RCVD AT 222200715:32:49 PM [Eastem Standard Time]' SVR:USPT0EFXRF2/12'DNIS:2738300' CSD:415 2687522' DURATION (mm.ss):0130
RECEIVED TIB22. 2007 2:34PM MOFO 28TH FL

CENTRAL FAX CENTER
NO. 247
P. 2
FEB 22 2007
of Under the Papework Redu lan Act 199 no e r
ia
to uimreid
U.S. Patent and mdemark 0ff00; DEPARTM~N OP COMMERCE U.S3. avald OMB contro number. unees rmatioa itcdsplays le.tonIn lnn eo co10
Approved for use through 12/31/2008. OMB 0551-4003
PT01Sl83 (01.06)
REQUEST FOR WITHDRAWAL AS ATTORNEY OR AGENT...... Art Unit AND CHANGE OF Examiner Name CORRESPONDENCE ADDRESS
Application Number Filing Date First Named inventor
11/231,353 September 19, 2005 Ken P. CHENG 2621 ' .Not Yet.Assigned
877832000200
Attomey Docket Number ----
To:
Commissioner for Patents P.O. Box 1450 Alexandria. VA 22313-1450
Please withdraw me as attorney or agent for the above Identified ptenrt application, and Q all the attomeyalagentra of record. the attorneyslagents (with registration numbers) listed on the attached paper(s), or 20872 the attomeys/agents associated with Customer Number NOTE: This box can only be checked when the power of attomrney of record Inthe application is to all the practitioners associated with a customer number.
The reasons for this request are: The practitioners have been discharged by the client. The client has reqJested transfer to new counsel. CORRESPONDENCE ADDFESS 1. Q 2. D The correspondenoe address is NOT affected by thin withdrawal. Change the correspondence address
and direct all future correspondence to:

"___
]
OR
Q
The address associated with Customer Number.

irm or
InMnual Name
Address
Stephen C. Durant Novak Druce + Quigg LLP Spear Street Tower - 35th Floor One Market Plaza
San Francisco city Country 8814 Telephone 41
state
CA
....
94105
Email steve.durant@rnovakdruce.com
signature
Name
Date
fetrJ. im
February
Rbistratiion No.
2007 Teephone No.
44,417
(415) 268-6373
1",
at least 30 days between approve a windraat and NOTE Wit4hanwal is ercvl when appoved raher than wen 9c0oed.4.loss ere 1he expiralian dare of a tho period for epnse orpassbeextenalon period, the reqesto withdrawI s notnnallydsappved
the transmtted byla simie to p nt and Tmder% Offi#arimlins no. (071) 2734300. on te date I hereby certify that this paper is being M ey __ __,,_( .. shownbelow.
Dated:Pebrua.2007. .natum (Vici..nty)
sf-2273801
PAGE 214' RCVD AT 21222007 5:32:49PM [Eastern Standard Time)' SVR:USPTOEFXRF.212' DNIS:2738g300' CSID:415 2687522' DURATION (m-.ss):01.30
RECEIVED
-fEB:'22. 2007 2:35PM MOFO 28TH FL
CENTRALFAXCENTER NO. 247
P. 3
FEB 22 2007
P'rolsatas toc alpproved
Urder Oe PaoeRp R0Uon
of Acts1995. noglasonse aqufred i
US. Patentand Tfdectmal ationMileSS I disay a vald OMB ontrlnumber IBesrad to a oledian of
OMB 0651-0D35 for use through 12/31 Office; U.S. DEPARTMENT OF COMMERC
200.
App5cation Number Filing Date
11/231,353 September 19,2005
AS ATTORNEY OR AGENT
AND CHANGE OF
S CORRESPONDENCE
.Unit.2521
Art Unit 221
---- ~~~-
..... "..... ....
ADDRESS
- -
Examiner Name
Not Yet Assigned

577832000200
To:
Commissioner for Patents P.O. Box 1450 Alexandria, VA 22313-1450
Please withdraw me as attorney or agent for the above identified patentl applicatton, and
Sall the attorneys/agents of record.

the attornmays/agenmts (with registration numbers) lated on the attahed paper(s), or
20672 nx the attomeystagents assadatad with Customer Number NOTE: This box can only be checked when the power of attorney o record inthe application is to all the
praclitioners associated with a customer number. The reasons forthis request are The practitioners have been discharged by the client. The client has requested transfer to new
counsel. CORRESPONDENCE ADD ESS

1. The correspondenoe address is NOT affected by this withdrawal. 2. ' Change the correspondence address and d'uect al future correspondence to:
The address associated with Customer Number

OR
Firme
Address
city Country
Stephen C. Durant lnMdiU NaTmre Novak Druce + Quigg LLP
Spear Street Tower - 36th Floor

One Market Plaza San Francisco stale CA
Zip
94105
Telephone 41
Signature
38T
r J. Vim
Emal steve.duLant@novakdruce.com
Registration No. 44,417
Name
Date
NOTE erOme io fad forepo m caphUon datefa
,2007 February o Wmhdwal is ffocve when appmroved raerhan when received.
Telephone No.
(415) 268-6373
Unle sMere a at lcsta30 0y between appoveo wrdr.thawer and o possbicexteatorn perdnd, the equest &bwihuwis salbe dryovwd
I herwbycerarythat hs paper isbeing trenspitted by fcsiileto te

showlntblouw.
".
ent and Tad
k Oftce. facite no.(571)273.800, on the date (Vidd Moini
Dated: Feb
Signature:
sf-2273801
RCVD PAGE 314 AT 22212007 5:32:49 PM [Eastemrn Standard Time]J SVR:USPTO.EFXRF2l12' DNIS:2?3300' CSID:415 2687522 DURATION (mm-.ss):01.30
TE
22. 2007 2:35PM
MOFO 28TH FL
Fme~
qredto
NO. 247
P. 4
Underthe Pe aonak
som edcR'il Atdof 1995. no pemai M
PTOISB~83 (01406) Approved ?r use through 12/30LZOO5. OMB S05-0035 U.. Paltent and Traflma k ica; U.S. DEPARTMENT OF COMMERCE l a vid OMB control number. iomatla nunlsst des I~sIondt3s lcin
Application Number Filing Date First Named Inventor Art Unit Examiner Name Attomey Docket Number
111231,353 September 19, 2005 KeniP. CHENG

2621
REQUEST FOR WITHDRAWAL A8 ATTORNEY OR AGENT AND CHANGE OF CORRESPONDENCE ADDRESS
Not Yet Assigned. 577832000200

i
..
i
To:

P.O. 8ox 1450
Alexandria, VA 22313-1450 Please withdraw me as attorney or agent forthe above identified patent application, and
[
all the attomeys/agents of record. [the attorneys/agents (with registration numbers) listed on the attached paper(s), or 20872 the attomrneyalegents associated with Cuslomer Number NOTE: This box can only be checked when the power of attorney of record inthe application Is to all the practitioners associated with a customrer number.
The reasons for this request are: The practitioners have been discharged by the dent. The client as requested transfer to new counsel. CORRESPONDENCE ADDSS 1. 2.
n
.The correspondence address is NOT affected by this withdraweL Change the correspondence address end direct all future cor
The addres associated with Customer Number'.
fespondence tO:
,1
OR Stephen C. Durant lndivuaI Name Novak Druoe + Quig LLP Spear Street Tower - 35th Floor Address One Market Plaza State San Francisco city
Faeor
CA
Zip
94105
Country
Telennone 4_1
Signature
Name
9814
eterJ im
Emailn steve.durant@novakdruce.com
Date
(415) 268-6373 Telephone No. , 2007 :t ; bc Yan r AotoUd IMOfs Macre ae baW0 da O Moen approval wlfd uwel and eis MOME WIthdre a effe mwhen eprved rathrV io d ft enrueab wVlfjWris no'nuaR~daapwvVWl. fits os~mfon chic ofafine peioforbrempose orpoar'a b e February
IL
Ihereby oaftty that this paperis bc ng VuwmfsVd

Dated:
by f
to ft Prn
and TO
f
Office. facoanle
To.C(71) 8734300.aono e
(VKM cidHomsy)
8ipnaturs;
sf-2273801
PAGE 414' RCVD AT 2212001 5:32:49 PM [Easternm Standard Time]' SVR:USPTO0EFXRF.212' DNIS:2138300' CSID:415 2587522'DURATION (mm.ss):01.30
Electronic Acknowledgement Receipt

EFS ID: Application Number: International Application Number: Confirmation Number:
4531
1754177
11231353
Title of Invention:
First Named Inventor/Applicant Name: Customer Number: Filer: Filer Authorized By: Attorney Docket Number: Receipt Date: Filing Date: Time Stamp: Application Type:
Ken Prayoon Cheng
20872 Stephen Christopher Durant/Marcia Shea Stephen Christopher Durant

577832000200
08-MAY-2007 19-SEP-2005 12:59:25 Utility
Payment information:
Submitted with Payment no
File Listing:
Document Number 1 Warnings: DoumntMulti Document Description File Name File Size(Bytes) .Pages Multi Pages Part /.zip (if appl.) no1
Power of Attorney
POA.pdf
45264
Information: 2 Assignee showing of ownership per 37 CFR 3.73(b). Statement.pdf 41914 no 1
Warnings: Information: Total Files Size (in bytes) 87178
This Acknowledgement Receipt evidences receipt on the noted date by the USPTO of the indicated documents, characterized by the applicant, and including page counts, where applicable. It serves as evidence of receipt similar to a Post Card, as described in MPEP 503. New Applications Under 35 U.S.C. 111 If a new application is being filed and the application includes the necessary components for a filing date (see 37 CFR 1.53(b)-(d) and MPEP 506), a Filing Receipt (37 CFR 1.54) will be issued in due course and the date shown on this Acknowledgement Receipt will establish the filing date of the application. National Stage of an International Application under 35 U.S.C. 371 If a timely submission to enter the national stage of an international application is compliant with the conditions of 35 U.S.C. 371 and other applicable requirements a Form PCT/DO/EO/903 indicating acceptance of the application as a national stage submission under 35 U.S.C. 371 will be issued in addition to the Filing Receipt, in due course. New International Application Filed with the USPTO as a Receiving Office If a new international application is being filed and the international application includes the necessary components for an international filing date (see PCT Article 11 and MPEP 1810), a Notification of the International Application Number and of the International Filing Date (Form PCT/RO/105) will be issued in due course, subject to prescriptions concerning national security, and the date shown on this Acknowledgement Receipt will establish the international filing date of the application.
POWER OF ATTORNEY TO PROSECUTE APPLICATIONS BEFORE THE USPTO

I hereby revoke all previous powers of attorney given in the application identified in the attached statement under 37 CFR 3.73(b). I hereby appoint: Practitioners associated with the Customer Number: 65761
as attorney(s) or agent(s) to represent the undersigned before the United States Patent and Trademark Office (USPTO) in connection with any and all patent applications assigned only to the undersigned according to the USPTO assignment records or assignment documents attached to this form in accordance with 37 CFR 3.73(b). Please change the correspondence address for the application identified in the attached statement under 37 CFR 3.73(b) to: The address associated with Customer Number: Assignee Name and Address: Proximex Corporation 440 N. Wolfe Road Sunnyvale, California 94085 A copy of this form, together with a statement under 37 CFR 3.73(b) (Form PTO/SB/96 or equivalent) is required to be filed ineach application in which this form is used. The statement under 37 CFR 3.73(b) may be completed by one of the practitioners appointed in this form Ifthe appointed practitioner is authorized to act on behalf of the assignee, and must identify the application inwhich this Power of Attorney is to be filed. SIGNATURE of Assignee of Record The individual whose signature and title is supplied below is authorized to act on behalf of the assignee. Ken P. Cheng Title: President Telephone: o Date 65761
.2. -
5O
PTO/SB/96 (09-06) Approved for use through 03/31/2007. OMB 0651-0031 U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number.
STATEMENT UNDER 37 CFR 3.73(b) Applicant/Patent Owner: Ken Prayoon CHENG et al. Application No./Patent No.: 11/231,353 Filed/Issue Date: September 19, 2005
Entitled: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICIATION DETECTION AND SURVEILLANCE SYSTEMS
PrnximPx Corporatinn (Name of Assignee)
,a
.rpnratinn (Type of Assignee, e.g., corporation, partnership, university, government agency, etc.)
states that it is: the assignee of the entire right, title, and interest; or 1. 2.
O an assignee of less than the entire right, title and interest

(The extent (by percentage) of its ownership interest is %)
in the patent application/patent identified above by virtue of either: A.Q An assignment from the inventor(s) of the patent application/patent identified above. The assignment was recorded , or for which a copy , Frame 0506 in the United States Patent and Trademark Office at Reel 017473 thereof is attached. OR A B.0- chain of title from the inventor(s), of the patent application/patent identified above, to the current assignee as follows:
1. From:
To:
The document was recorded in the United States Patent and Trademark Office at
Reel , Frame
,
or for which a copy thereof is attached.
To: . 2. From: The document was recorded in the United States Patent and Trademark Office at
Reel 3. From: Reel , Frame , Frame
_,
or for which a copy thereof is attached. To: , or for which a copy thereof is attached.
Ol
Additional documents in the chain of title are listed on a supplemental sheet.
Q As required by 37 CFR 3.73(b)(1)(i), the documentary evidence of the chain of title from the original owner to the assignee was, or concurrently is being, submitted for recordation pursuant to 37 CFR 3.11. [NOTE: A separate copy (i.e., a true copy of the original assignment document(s)) must be submitted to Assignment Division in accordance with 37 CFR Part 3, to record the assignment in the records of the USPTO. See MPEP 302.08] The undersigned (whose title is supplied below) is authorized to act on behalf of the assignee. /Stephen C. Durant/ Signature Steohen C. Durant Printed or Typed Name Attorney for Assignee Title
This collection of information is required by 37 CFR 3.73(b). The information is required to obtain or retain a benefit by the public which is to file (and by the USPTO to process) an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.11 and 1.14. This collection is estimated to take 12 minutes to complete, including gathering, preparing, and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Any comments on the amount of time you require to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S. Patent and Trademark Office, U.S. Department of Commerce, P.O. Box 1450, Alexandria, VA 22313-1450. DO NOT SEND FEES OR COMPLETED FORMS TO THIS ADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450, Alexandria, VA 22313-1450.
Mavy 8. 2007 Date 415.814 6163 Telephone Number
If you need assistance in completing the form, call 1-800-PTO-9199 and select option 2.
UNITED STATES PATENT AND 'IThADEMARK OFFICE

UNITED STATES DEPARTMENT OP COMMERCE United States Patent and Trademark Omoe
Addrem:COMMISSIONER FOR PATENTS P.O. Box 1450 Alexandria, Vuirgia 2233-1450
wwws.upto.go
APPLICATION NUMBER
FILING OR 371 (c) DATE
KFIRST NAMED APPLICANT
ATTY. DOCKET NO/TLE
11/231,353
09/19/2005
Ken Prayoon Cheng
577832000200 CONFIRMATION NO. 4531
65761 NOVAK DRUCE & QUIGG, LLP (SAN FRANCISCO) 1000 LOUISIANA STREET FIFTY-THIRD FLOOR HOUSTON, TX 77002
*C00000000023958100"
NOTICE OF ACCEPTANCE OF POWER OF ATTORNEY

This is in response to the Power of Attorney filed 05/08/2007. The Power of Attorney in this application is accepted. Correspondence in this application will be mailed to the above address as provided by 37 CFR 1.33.
--
Office of Initial Patent Examination (571) 272-4000, or 1-800-PTO-9199 OFFICE COPY
UNITED STATES PATENT AND TADEMARK OFFIGE I -- ~---------
UNITED STATES DEPARTMENT OF COMMERCE United States Patent and Trademark Office' Address:COMMISSIONER FOR PATENTS P.O.Box1450 VugBiia Alexandria. 22313.1450 ww.uwptogov
APPLICATION NUMBER
ATTY. DOCKET NO./TITLE
11/231,353
09/19/2005
Ken Prayoon Cheng
llIlIl II00EII3I80IHIG UOH*HIIIIllH1111 *0C000000023958091"

NOTICE REGARDING CHANGE OF POWER OF ATTORNEY

This is in response to the Power of Attorney filed 05/08/2007. * The Power of Attorney to you in this application has been revoked by the assignee who has intervened as provided by 37 CFR 3.71. Future correspondence will be mailed to the new address of record(37 CFR 1.33).
--
Officeofnitia Patent Examination (571) 272-4000, or 1-800-PTO-9199 OFFICE COPY.
Page 1 of 1
UNITED
STATES PATENT AND TRADEMARK OFFICE I
STATES DEPARTMENT OF COMMERCE United States Patent and Trademark Office Addreou: COMMISSIONER FOR PATENTS
P.O. Box 1450 Alxandria, Vu ia
ww~rto~
I UNITED
7-
22313-1450
APPLICATION NUMBER
ATTY. DOCKET NO/TITLE
11/231,353
09/19/2005
Ken Prayoon Cheng
65761 NOVAK DRUCE & QUIGG, LLP (SAN FRANCISCO) 1000 LOUISIANA STREET FIFTY-THIRD FLOOR HOUSTON, TX 77002
*00C000000023958100*

This is in response to the Power of Attorney filed 05108/12007. The Power of Attorney in this application is accepted. Correspondence in this application will be mailed to the above address as provided by 37 CFR 1.33.
__
.....
Office of Initial Patent Examination (571) 272-4000, or 1-800-PTO-9199 ATTORNEY/APPLICANT COPY
Page 1 of I
ADEMARK OFFIGE UNITED STATES PATENT AND M
UNITED STATES DEPARTMENT OF COMMERCE United States Patent and Trademark OMice Addma: COMMISSIONER FOR PATENTS
P.O. B= 1450
AIxaiam
Vuni
22313-1450
APPLICATION NUMBER
ATTY. DOCKET NO.TLE
11/231,353
09/19/2005
Ken Prayoon Cheng
0OC00000023958091*

This is in response to the Power of Attorney filed 05/08/2007. * The Power of Attorney to you in this application has been revoked by the assignee who has intervened as provided by 37 CFR 3.71. Future correspondence will be mailed to the new address of record(37 CFR 1.33).
--
Office of Initial Patent Examination (571) 27-4000, or 1-800-PTO-9199 FORMER ATTORNEY/AGENT COPY

Commissioner for Patents United States Patent and Trademark Office P.O. Box 1450 Alexandria, VA 22313-1450
www.uspto.gov
MORRISON & FOERSTER LLP. 425 MARKET STREET SAN FRANCISCO CA 94105-2482
COPY MAILED
MAY 2 32007
OFFICE OF PETITIONS
In re Application of CHENG, Ken P. et al. Application No. 11/231,353 Filed: September 19, 2005 Attorney Docket No. 577832000200 S DECISION ON PETITION S TO WITHDRAW S FROM RECORD
This is a decision on the Request to Withdraw as attorney or agent of record under 37 C.F.R. 1.36(b), filed February 22, 2007. The request is NOT APPROVED as moot. A review of the file record indicates that the power of attorney to Morrison & Foerster LLP has been revoked by the assignee of the patent application on May 08, 2007. Accordingly, the request to withdraw under 37 C.F.R. 1.36(b) is moot. All future communications from the Office will continue to be directed to the below-listed address until otherwise notified by applicant. Telephone inquires concerning this decision should be directed to Michelle R. Eason at 571-2724231.
Terri Williams Petitions Examiner Office of Petitions
cc:
NOVAK DRUCE & QUIGG, LLP (SAN FRANCISCO) 1000 LOUISIANA STREET FIFTY-THIRD FLOOR HOUSTON, TX 77002

EFS ID: Application Number: International Application Number: Confirmation Number: 4531 6286123 11231353
Title of Title of Invention:Invention:
Adaptive multi-modal integrated biometric identification detection and suvilnc. ytm surveillance systems
First Named Inventor/Applicant Name: Customer Number: Filer: Filer Authorized By: Attorney Docket Number: Receipt Date: Filing Date: Time Stamp: Application Type: .
Ken Prayoon Cheng 65761 Tracy Wesley Druce/Amy Drury Tracy Wesley Druce 8193.003.NPUS00 19-OCT-2009 19-SEP-2005 15:48:38 Utility under 35 USC 111(a)
Submitted with Payment no
File Listing:
Document Number Document Description
Petition to withdraw attorney or agent
File Name
8193_003_NPUS00_Withdrawa
File Size(Bytes)/ Message Digest 170989

b8f43036c2c2c615b79cc36e48994630a6 99fd
Multi Part /.zip

no794
Pages (if appl.)
(5883)
I.pdf
no4
Warnings: Information:
Total Files Size (in bytes)
170989
This Acknowledgement Receipt evidences receipt on the noted date by the USPTO of the indicated documents, characterized by the applicant, and including page counts, where applicable. It serves as evidence of receipt similar to a Post Card, as described in MPEP 503. New Applications Under 35 U.S.C. 111 If a new application is being filed and the application includes the necessary components for a filing date (see 37 CFR 1.53(b)-(d) and MPEP 506), a Filing Receipt (37 CFR 1.54) will be issued in due course and the date shown on this Acknowledgement Receipt will establish the filing date of the application. National Stageof an International Application under 35 U.S.C. 371 If a timely submission to enter the national stage of an international application is compliant with the conditions of 35 U.S.C. 371 and other applicable requirements a Form PCT/DO/EO/903 indicating acceptance of the application as a national stage submission under 35 U.S.C. 371 will be issued in addition to the Filing Receipt, in due course. New International Application Filed with the USPTO as a Receiving Office If a new international application is being filed and the international application includes the necessary components for an international filing date (see PCT Article 11 and MPEP 1810), a Notification of the International Application Number and of the International Filing Date (Form PCT/RO/105) will be issued in due course, subject to prescriptions concerning national security, and the date shown on this Acknowledgement Receipt will establish the international filing date of the application.
Doc Code: PET.POA.WDRW Document Description: Petition
PTO/SB/83 (11-08) Approved for use through 11/30/2011. OMB 0651-0035 U.S. Patentand Trademark Office, U.S. DEPARTMENT OF COMMERCE Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. to withdraw attorney or agent (SB83)
REQUEST FOR WITHDRAWAL

AS ATTORNEY OR AGENT
Application Number Filing Date First Named Inventor
11/231,353
09-19-2005
Ken Pravoon Chenqcl
AND CHANGE OF CORRESPONDENCE ADDRESS

To: Commissioner for Patents P.O. Box 1450 Alexandria, VA 22313-1450
Art Unit
Examiner Name Attorney Docket Number
2621
DASTOURI, MEHRDAD 8193.003.NPUS00 ,
Please withdraw me as attorney or agent for the above identified patent application, and
Sall the practitioners of record;

Q the practitioners (with registration numbers) of record listed on the attached paper(s); or the practitioners of record associated with Customer Number: 65761
NOTE: The immediately preceding box should only be marked when the practitioners were appointed using the listed Customer Number. The reason(s) for this request are those described in 37 CFR :
D
D
-
10.40(b)(1) 10.40(c)(1)(i)
10.40(c)(1)(v)
10.40(b)(2)
10.40(b)(3) 10.40(c)(1)(iii)
10.40(b)(4)
1
I
10.40(c)(1)(ii)Q 10.40(c)(1)(vi)
jj
10.40(c)(1)(iv)
-1
10.40(c)(4)
11
10.40(c)(5)
E D
10.40(c)(2)
10.40(c)(3
10.40(c)(6)Please explain below:
Certifications
Check each box below that is factually correct. WARNING: If a box is left unchecked, the request will likely not
be approved.
1. 1I/We have given reasonable notice to the client, prior to the expiration of the response period, that the practitioner(s) intend to withdraw from employment.
I/We have delivered to the client or a duly authorized representative of the client all papers and property 2. T (including funds) to which the client is entitled.
3.
I/We have notified the client of any responses that may be due and the time frame within which the
client must respond. Please provide an explanation, if necessary: As of August 14, 2008, the client has directed us to cease representation and transfer all files to Stephen C. Durant, who has not yet filed a new Power of Attorney or Change of Address.
[Page 1 of 2] This collection of information is required by 37 CFR 1.36. The information is required to obtain or retain a benefit by the public which is to file (and by the USPTO to process) an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.11 and 1.14. This collection is estimated to take 12 minutes to complete, including gathering, preparing, and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Any comments on the amount of time you require to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S. Patent and Trademark Office, U.S. Department of Commerce, P.O. Box 1450, Alexandria, VA 22313-1450. DO NOT SEND FEES OR COMPLETED FORMS TO THIS ADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450, Alexandria, VA 22313-1450.
PTO/SB/83 (11-08) Approved for use through 11/30/2011. OMB 0651-0035 U.S. Patent and Trademark Office, U.S. DEPARTMENT OF COMMERCE Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number.
REQUEST FOR WITHDRAWAL AS ATTORNEY OR AGENT AND CHANGE OF CORRESPONDENCE ADDRESS

Complete the following section only when the correspondence address will change. Changes of address will only be accepted to an inventor or an assignee that has properly made itself of record pursuant to 37 CFR 3.71.
Change the correspondence address and direct all future correspondence to: A. D OR S B. The address of the inventor or assignee associated with Customer Number:
Inventor or Assignee name
PROXIMEXCORPORATION
Address
440 N. Wolfe Road State CA Zip 94085

Email
City Sunnyvale
Telephone (408) 216-5190
Country US
I am authorized to sign on behalf of myself and all withdrawing practitioners.

Signature /Tracy W. Druce/
Name
Tracy W. Druce
Address NOVAK, DRUCE & QUIGG LLP, 1000 LOUISIANA STREET, FIFTY-THIRD FLOOR City Houston Date October 19, 2009 State TX Zip 77002 Country US
Telephone No. 713-571-3400
NOTE: Withdrawal is effective when approved rather than when received.
[Page 2 of 2] This collection of information is required by 37 CFR 1.36. The information is required to obtain or retain a benefit by the public which is to file (and by the USPTO to process) an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.11 and 1.14. This collection is estimated to take 12 minutes to complete, including gathering, preparing, and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Any comments on the amount of time you require to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S. Patent and Trademark Office, U.S. Department of Commerce, P.O. Box 1450, Alexandria, VA 22313-1450. DO NOT SEND FEES OR COMPLETED FORMS TO THIS ADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450, Alexandria, VA 22313-1450. If you need assistance in completing the form, call 1-800-PTO-9199 and select option 2.
Page 1 of 2
Todd Leone
From: Sent: To: Cc: Tracy Druce Thursday, August 14, 2008 2:25 PM Ken P. Cheng Matt Todd; Durant, Stephen C.; Jim Barth; Raj Tatini; Todd Leone; Vincent M. DeLuca; Gregory Novak
Subject: RE: Proximex file transfer request to Durant at Duane Morris Dear Mr. Cheng: These files will go today or tomorrow. Thank you, Tracy Tracy Wesley Druce Novak Druce & Quigg LLP Wells Fargo Plaza, 53rd Floor 1000 Louisiana Street Houston, TX 77002 713.571.3400 (Telephone) 713.456.2836 (Fax) 877.529.7894 (US Toll Free) tracy.druce@novakdruce.com Visit our website at www.novakdruce.com Contidentiality Notice: This email ard any attachments contain information from the law firm of Novak DrucE'c, I..P, which may be confidential and/or privileged. The information is intended to be for the use of the individual or entity named on this email. If you are not the intended recipient, be aware that rany disclosure, copying, distribution or use of the contents of this email is prohibited. Ifyou receive this email in error, please notify us by reply email immediately so that we can arrange for the retrieval of the original documents at no cost to you.
From: Ken P. Cheng [mailto:ken.cheng@proximex.com] Sent: Thursday, August 14, 2008 2:14 PM To: Tracy Druce Cc: Matt Todd; Durant, Stephen C.; Jim Barth; Raj Tatini Subject: Proximex file transfer request to Durant at Duane Morris
Dear Mr. Druce: This is to inform you that Proximex Corporation ("Proximex") wishes to transfer responsibility for patent matters that you are handling on behalf of Proximex to Stephen C. Durant. We direct Novak Druce & Quigg LLP to discontinue its representation of Proximex and immediately transfer the patent files and any other materials including, but not limited to, those described below, to: Stephen C. Durant
8/14/2008
Page 2 of 2
Duane Morris LLP, 2000 Spear Street Tower One Market Plaza San Francisco, California 94105-1104 Specifically, we would like the transfer to include the following Proximex files and materials identified below: All files and other property of Proximex including, without limitation, all patent application 1. files, attorney work files, collections of prior art, correspondence files and other files and materials; A printout, and an electronic file, containing all items identified in the Report docketed in 2. Novak Druce & Quigg LLP's docket system for Proximex including all actions due or that are scheduled due for a future date; A CD-Rom or Flash drive containing an MSWord version of the Proximex patent applications, 3. drawings and any other items as filed for the patent application files identified in the Report; and Any other Proximex files or documentation subsequently identified and requested by Mr. 4. Durant or any other member of Duane Morris LLP pertaining to items identified in numbered paragraph 1 above. 'Please immediately transfer the requested material as there may be many urgent issues pending and under consideration. When the materials are ready for delivery, please call Duane Morris LLP at 415-947-3023 to arrange for a courier. I appreciate your assistance. If you have any questions, please contact me at your convenience. Very truly yours, PROXIMEX CORPORATION
Ken P. Cheng Founder cc: Stephen C. Durant, Esq. Matt Todd, Esq.
Ken Prayoon Cheng CTO Surveillint'" Discover. Connect. Resolve. Proximex Corporation 440 N. Wolfe Road Sunnyvale, CA 94085 www.proximex.com Wok: 408.524.1510 Moi408.828.1838 : 408.524.1528 , mah: Ken.Cheng)Proximex.com
8/14/2008
,.

www.uspto.gov i
SAN FRANCISCO OFFICE OF NOVAK, DRUCE & QUIGG LLP 1000 LOUISIANA STREET FIFTY-THIRD FLOOR HOUSTON TX 77002
COPY MAILED
DEC 0 3 2009 OFFICE OF PETITIONS
In re Application of CHENG, Ken Prayoon et al. Application No. 11/231,353 Filed: September 19, 2005 Attorney Docket No. 8193.003.NPUS00
: :
DECISION ON PETITION TO WITHDRAW FROM RECORD
This is a decision on the Request to Withdraw as attorney or agent of record under 37 C.F.R. 1.36(b), filed October 19, 2009. The request is APPROVED. i
A grantable request to withdraw as. attorney/agent of record must be signed by every attorney/agent seeking to withdraw or; contain a clear indication that one attorney is signing on behalf of another/others. The Office requires the practitioner(s) requesting withdrawal to certify that he, she, or they have: (1) given reasonable notice to the client, prior to the expiration of the response period, that the practitioner(s) 1 intends to withdraw from employment; (2) delivered to the client or a duly authorized representative of the client all papers and property (including funds) to which the client is entitled; and (3) notified the client of any responses that may be due and the time frame within which the client must respond, pursuant 37 CFR 10.40(c). The request was signed by Tracy W. Diruce on behalf of all attorneys of record who are associated with customer No. 65761. All attorneys/agents associated with have been withdrawn. Applicant is reminded that there is no attorney of record at this time.
I
The request to change the correspondence address of record is not acceptable as the requested correspondence address is not that of: (1) the first named signing inventor; or (2) an intervening assignee of the entire interest under 37 C.F.R 3.71. All future communications from the Office will be directed to the first named signing inventor.
Application No. 11/231,353
Page
In order to request or take action in a patent matter, the assignee must establish its ownership of the patent to the satisfaction of the Director. In this regard, a Statement under 37 CFR 3.73(b) must have either: (i) documentary evidence of a chain of title from the original owner to the assignee (e.g., copy of an executed assignment), and a statement affirming that the documentary evidence of the chain of title from the original owner to the assignee was or concurrently is being submitted for recordation pursuant to 3.11; or (ii) a statement specifying where documentary evidence of a chain of title from the original owner to the assignee is recorded in the assignment records of the Office (e.g., reel and frame number). There are no outstanding office actions at this time. Telephone inquiries concerning this decision should be directed to the undersigned at 571-2724231.
Michelle R. Eason
Paralegal Specialist Office of Petitions
cc:
KEN P. CHENG 20691 REID LANE SARATOGA, CA 95070-5325 PROXIMEX CORPORATION 440 N. WOLFE ROAD SUNNYVALE, CA 94085
cc:
UNITED STATES PATENT AND
TRADEMARK OFFICE
UNITED STATF. DEPARTMENT OF COMMERCE United States Patent and Trademark Office Address: COMMISSIONER FOR PATENTS
P.O. Box 1450 Vria 22313-1450 Alexanhdria, www.uspto.gov
APPLICATION NUMBER
FILING OR 371(C) DATE
ATTY. DOCKET NO.TITLE
11/231,353 65761 SAN FRANCISCO OFFICE OF NOVAK, DRUCE & QUIGG LLP 1000 LOUISIANA STREET FIFTY-THIRD FLOOR HOUSTON, TX 77002
09/19/2005
Ken Prayoon Cheng
8193.003.NPUS00 CONFIRMATION NO. 4531 POWER OF ATTORNEY NOTICE

This is in response to the Power of Attorney filed 10/19/2009. * The withdrawal as attorney in this application has been accepted. Future correspondence will be mailed to the new address of record. 37 CFR 1.33.
/mreason/
Office of Data Management, Application Assistance Unit (571) 272-4000, or (571) 272-4200, or 1-888-786-0101
page 1 of 1

UNITED STATES DEPARTMENT OF COMMERCE United States Patent and Trademark Office Address: COMMISSIONER FOR PATENTS P.O. Box 1450 Alexandria, Virginia 22313-1450 www.uspto.gov
APPLICATION NO.
11/231,353
FILING DAT]E
09/19/2005
5 ci~ l
FIRST NAMED INVENTOR Ken Prayoon Cheng

.r .u u
ATTORNEY DOCKET NO.
'7590 /:)V KEN P. CHENG 20691 REID LANE
EXAMINER
GRANT II, JEROME

ART UNIT 2625 PAPER NUMBER
SARATOGA, CA 95070-5325
] ]
MAIL DATE
DELIVERY MODE
05/13/2010
PAPER
Please find below and/or attached an Office communication concerning this application or proceeding.
The time period for reply, if any, is set in the attached communication.
PTOL-90A (Rev. 04/07)
Application No. 11/231,353
Applicant(s) CHENG ET AL.

Art Unit 2625
Office Action Summary
Examiner Jerome Grant II
-- The MAILING DATE of this communication appears on the cover sheet with the correspondence address -Period for Reply A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION.
- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed after SIX (6) MONTHS from the mailing date of this communication. - If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. - Failure to reply within the set or extended period for reply will,; by statute, cause the application to become ABANDONED (35 U.S.C. 133). Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any eamrned patent term adjustment. See 37 CFR 1.704(b).
Status Responsive to communication(s) filed on 1)[ 2b) This action is non-final. 2a)O This action is FINAL. 3)O- Since this application is in condition for allowance except for formal matters, prosecution as to the merits is closed in accordance with the practice under Ex parte Quayle, 1935 C.D. 11, 453 O.G. 213. Disposition of Claims 4) Claim(s) 1-20 is/are pending in the application. is/are withdrawn from consideration. 4a) Of the above claim(s) is/are allowed. 5)- Claim(s) 6) Claim(s) 1-13 and 16-20 is/are rejected. 7) Claim(s) 14 and 15 is/are objected to. are subject to restriction and/or election requirement. Claim(s) 8) Application Papers 9)L The specification is objected to by the Examiner. is/are: a)LI accepted or b)L objected to by the Examiner. 10)r The drawing(s) filed on _ Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 11)O The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. Priority under 35 U.S.C. 119 12)I Acknowledgment is made of a claim for foreign priority under 35 U.S.C. 119(a)-(d) or (f). a)Q All b)L Some * c)O None of: 1.0 Certified copies of the priority documents have been received. 2.0 Certified copies of the priority documents have been received in Application No. Copies of the certified copies of the priority documents have been received in this National Stage 3.application from the International Bureau (PCT Rule 17.2(a)). the attached detailed Office action for a list of the certified copies not received. * See
Attachment(s) 1) Notice of References Cited (PTO-892) 2) I Notice of Draftsperson's Patent Drawing Review (PTO-948) 3) Information Disclosure Statement(s) (PTO/SB/08) Paper No(s)/Mail Date 10/06.
4)
O Interview Summary (PTO-413) Paper No(s)/Mail Date. Notice of Informal Patent Application 5) O6) E Other:
PTOL-326 (Rev. 08-06)
Part of
Paper No/Mail Date 20100510

Date 20100510
Part of Paper No./Mail
Application/Control Number: 11/231,353 Art Unit: 2625 Detailed Action
Page 2
1.
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for
all obviousness rejections set forth in this Office action:

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claims 1, 2, 5-13 and 16-20 are rejected under 35 U.S.C. 103(a) as being unpatentable over Reilly.
With respect to claim 1, Reilly teaches a surveillance method, performed by the system of figure 1, comprising at least one event sensor 26 in a security area to detect a potential security breach (see para. 69, line 10); using at least one camera 28 (see para. 52, lines 1-12) with a view of the security areas in which the event is sensed to gather biometric data (see para. 52, lines 19-29; converting at least one person in the vicinity of the sensing event (see para. 54); producing a subject dossier (para. 53, lines 27-29 and para. 46, lines 1-8) corresponding to the at least one person and detecting biometric information of one or more persons with respect to the subject dossier. See para. 48, lines 11-19 and 23-34.
Application/Control Number: 11/231,353 Art Unit: 2625 Reilly teaches all of the subject matter claimed except for the specific language of
Page 3
matching biometric information. But rather, Reilly uses the term "detecting" biometric data with respect to a stored dossier of data on several persons stored in a memory device 42.
It would have been obvious to one of ordinary skill in the art that "matching" and "detecting" are synonymous terms for comparing live biometric data against pre-stored biometric data to determined the identification of persons under surveillance.
With respect to claim 2, while Reilly teaches system 20 for identification of the clothing of an individual under observation, Reilly does not specifically teach identifying the color or clothes. However, Reilly teaches identifying image information about the clothing. See para. 46, lines 5-8. it would have been obvious to one of ordinary skill in the art that other information about clothes would include the color, hue or tint of clothing worn by the person(s) under surveillance.
With respect to claim 5, see para. 48, lines 14 and 15. The motivation to claim 5 is the same as that to claim 1.
Application/Control Number: 11/231,353 Art Unit: 2625
Page 4
With respect to claims 6 and 11, Reilly teaches wherein the at least one camera 28 gathers one or more clothing information (para. 46, lines 5-8), facial features (para. 48, lines 12-14), skin color (para. 48, lines 14-15). Reilly teaches gathering biometric data (see para. 52, lines 19-29); converting at least one person in the vicinity of the sensing event (see para. 54); producing a subject dossier (para. 53, lines 27-29 and para. 46, lines 1-8) corresponding to the at least one person and detecting biometric information of one or more persons with respect to the subject dossier. See para. 48, lines 11-19, 23-34.
Reilly teaches system 20 for identification of the clothing of an individual under observation. Reilly does not specifically teach identifying the color or clothes. However, Reilly teaches identifying image information about the clothing. See para. 46, lines 5-8. it would have been obvious to one of ordinary skill in the art that other information about clothes would include the color, hue or tint of clothing worn by the person(s) under surveillance.
Reilly teaches all of the subject matter claimed except for the specific language of matching biometric information. But rather, Reilly uses the term "detecting" biometric data with respect to a stored dossier of data on several persons stored in a memory device 42.
Application/Control Number: 11/231,353 Art Unit: 2625 It would have been obvious to one of ordinary skill in the art that "matching" and
Page 5
"detecting" are synonymous terms for comparing live biometric data against pre-stored biometric data to determined the identification of persons under surveillance.
With respect to claim 7, the limitation of the claim can be ascertained from individual clothing portions i.e., size of the clothing and its contour upon the persons in surveillance. Hence, the height and weight ca be estimated. See also para. 46. See para. 52, line 15-19.
With respect to claim 8-10, Reilly teaches image cameras for detecting the position of objects with respect to other objects, using acoustic wave generation and other waveforms. Assuming Reillly uses NTSC cameras, it would have been obvious to study the speed of the subject by comparing the pixel movement of the target subject in relationship with the frame rate of the camera. Bt knowing the pixel change rate and the frame rate, the speed of the person under surveillance can be detected. Assuming, Reilly does not use the NTSC camera, it would have been obvious to use the acoustic or other wave forms taught by Reilly for measuring the presence of the moving object with respect to other surrounding objects. The rate of change of the objects with respect to other objects can be used to detect a velocity of a target by Doppler frequency processing. The velocity of the moving object can be determined using the Doppler technique.
Page 6
Moreover, it would have been obvious to one of ordinary skill in the art that with the determination of the movement of the target under surveillance, the direction of the camera can be determined. With the determination of the direction, other control switches that control the operation of the network of cameras, can be used to activate a camera in a network that covers the peripheral of the zone in which the camera had last detected the subject, for the purpose of tracking the speed and direction of a person under surveillance.
With respect to claim 12, Reilly teaches system 20 for identification of the clothing of an individual under observation, Reilly does not specifically teach identifying the color or clothes. However, Reilly teaches identifying image information about the clothing. See para. 46, lines 5-8. It would have been obvious to one of ordinary skill in the art that other information about clothes would include the color, hue or tint of clothing worn by the person(s) under surveillance. Regarding the weight and height of the person under observation, the limitation of the claim can be ascertained from individual clothing portions i.e., size of the clothing and its contour upon the persons in surveillance. Hence, the height and weight can be estimated. See also para. 46. See para. 52, lines 15-19.
Page 7
Reilly teaches image cameras for detecting the position of objects with respect to other objects, using acoustic wave generation and. other waveforms. Assuming Reillly uses NTSC cameras, it would have been obvious to study the speed of the subject by comparing the pixel movement of the target subject in relationship with the frame rate of the camera. By knowing the pixel change rate and the frame rate, the speed of the person under surveillance can be detected. Assuming, Reilly does not use the NTSC camera, it would have been obvious to use the acoustic or other wave forms taught by Reilly for measuring the presence of the moving object with respect to other surrounding objects. The rate of change of the objects with respect to other objects can be used to detect a velocity of a target by Doppler frequency processing. The velocity of the moving object can be determined using the Doppler technique.
Moreover, it would have been obvious to one of ordinary skill in the art that with the determination of the movement of the target under surveillance, the direction of the camera can be determined. With the determination of the direction, other control switches that control the operation of the network of cameras, can be used to activate a camera in a network that covers the peripheral of the zone in which the camera had last detected the subject, for the purpose of tracking the speed and direction of a person under surveillance.
With respect to claim 13, Reilly teaches a display 132, see also para. 51.
Page 8
With respect to claim 16, see para. 52, lines 1-12.
With respect to claim 17 and 20, Reilly teaches a surveillance system as shown by figure 1, at least one sensor disposed in the security area of a surveillance region as claimed, see para. 69, line 10; a plurality of cameras 9see para. 52, lines 1-12; wherein the cameras automatically generate biometric data (see para. 52, lines 19-29) concerning a subject in the surveillance region, see also para. 54; where in one or more other plurality of cameras search for other subject persons (see para. 53, lines 7-11; a processing system (40); wherein the processing system is programmable (see para. 48, lines 11-19 and para. 49 and 50); and wherein the processing system is programmable (see para. 49) to detect biometric information in relation to other stored dossier of a subject. See also para. 48, lines 11-19
It would have been obvious to one of ordinary skill in the art that "matching" and "detecting" are synonymous terms for comparing live biometric data against pre-stored biometric data to determined the identification of persons under surveillance.
Page 9
With respect to claim 18, see para. 52, lines 1-12.
With respect to claim 19, Reilly teaches wherein the processing system 40 is programmable (para. 49) to produce a subject dossier that includes one or more clothing, facial features, skin color, hair color and estimated height and weight. Reilly
teaches wherein the at least one camera 28 gathers one or more clothing information, facial features (para. 48, lines 12-14) , skin color (para. 48, lines 14-15). Reilly teaches system 20 for identification of the clothing of an individual under observation, Reilly does not specifically teach identifying the color or clothes. However, Reilly teaches identifying image information about the clothing. See para. 46, lines 5-8. It would have been obvious to one of ordinary skill in the art that other information about clothes would include the color, hue or tint of clothing worn by the person9s) under surveillance.
Note that the weight and height of an individual claim can be ascertained from individual clothing portions i.e., size of the clothing and its contour upon the persons in surveillance. Hence, the height and weight ca be estimated. See also para. 46.. See para. 52, line 15-19.
Page 10
Reilly teaches system 20 for identification of the clothing of an individual under observation, Reilly does not specifically teach identifying the color or clothes. However, Reilly teaches identifying image information about the clothing. See para. 46, lines 5-8. it would have been obvious to one of ordinary skill in the art that other information about clothes would include the color, hue or tint of clothing worn by the person(s) under surveillance.
Reilly teaches gathering biometric data (see para. 52, lines 19-29) converting at least one person in the vicinity of the sensing event (see para. 54); producing a subject dossier (para. 53, lines 27-29 and para. 46, lines 1-8) corresponding to the at least one person and detecting biometric information of one or more persons with respect to the subject dossier. See para. 48, lines 11-19, 23-34.
Application/Control Number: 11/231,353 Art Unit: 2625 It would have been obvious to one of ordinary skill in the art that "matching" and
Page 11
"detecting" are synonymous terms for comparing live biometric data against pre-stored biometric data to determined the identification of persons under surveillance.
2. Claims 3 and 4 are rejected under 35 U.S.C. 103(a) as being unpatentable over Reilly in view of Monroe.
With respect to claim 3, Reilly teaches identifying a person as described at para. 52, lines 19-29. Reilly teaches identifying facial features (see para. 48, lines 12-14) However, specific reference to facial signatures is not taught.
Monroe teaches at paragraphs. 12, 137, 139 andl40 for performing facial signatures and comparing the facial attributes against facial dossiers stored in data bases to be used by Intelligence organizations.
Page 12
It would have been obvious to one of ordinary skill in the art to modify system 20 of Reilly for the purpose of including facial recognition as one of the identification methods for identifying a person under survelliance.
With respect to claim 4, note that hair color would be identified as a facial attribute such as mustaches, eyebrows, beards and eyelashes. The motivation of the rejection of claim 4 is the same as that to claim 3.
3.
Claims Objected
Claims 14 and 15 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Page 13
4.
Any inquiry concerning this communication or earlier communications from
the examiner should be directed to Jerome Grant II whose telephone number is 571272-7463. The examiner can normally be reached on Mon.-Fri. from 9:00 to 5:00. If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Edward Coles, can be reached on 571-272-7402. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/Jerome Grant II/ Primary Examiner, Art Unit 2625
Notice of References Cited
11/231,353 Examiner Jerome Grant II

CHENG ET AL.
Art Unit 2625
Applicant(s)/Patent Under Reexamination CHENG ET AL. Page 1 of 2
Document Number
Country Code-Number-Kind Code
Date
MM-YYYY Name Classification
* * * * * * * * * * * * *
A B C D E F G H I j K L M
US-7,376,276 US-2004/0117638 US-2004/0059953 US-2003/0115474 US-2003/0074590 US-5,991,429 US-6,816,184 US-6,687,831 US-2009/0322873 US-2008/0043102 US-2007/0186116 US-2006/0222209 US-2005/0105764
Document Number Country Code-Number-Kind Code
05-2008 06-2004 03-2004 06-2003 04-2003 11-1999 11-2004 02-2004 12-2009 02-2008 08-2007 10-2006 05-2005
Date MM-YYYY
Shniberg et al. Monroe, David A. Purnell, John Khan et al. Fogle et al. Coffin et al. Brill et al. Albaugh et al. Reilly et al. ROWE et al. Clemmensen et al.. Zhang et al. Han et al.
FOREIGN PATENT DOCUMENTS Country Name
382/224 713/186 713/202 713/186 713/320 382/118 348/143 726/7 348/143 348/143 713/186 382/107 382/100
Classification
N
O P
JP02004295798
10-2004
Japan
Sakaibara
H04N 7/18
Q
R S T NON-PATENT DOCUMENTS
*
U
Include as applicable: Author, Title Date, Publisher, Edition or Volume, Pertinent Pages)
X
"A copy of this reference is not being furnished with this Office action. (See MPEP 707.05(a).) Dates inMM-YYYY format are publication dates. Classifications may be US or foreign.
PTO-892 (Rev. 01-2001)
Part of Paper'No. 20100510
Application/Control No. 11/231,353
Applicant(s)/Patent Under Reexamination CHENG ET AL. Art Unit
Examiner
Jerome Grant II
U.S. PATENT DOCUMENTS Document Number Country Code-Number-Kind Code * A B C D E F G H I J K L M US-7,602,947 USUSUSUSUSUSUSUSUSUSUSUSFOREIGN PATENT DOCUMENTS Document Number Country Code-Number-Kind Code N O P Q R S T NON-PATENT DOCUMENTS * Date MM-YYYY Country Name Date MM-YYYY 10-2009 Lemelson et al.
2625
Page 2 of 2
382/116
Classification
Include as applicable: Author, Title Date, Publisher, Edition or Volume, Pertinent Pages)
X
Q^RDOf A .,t k:rf-nC -- fro~~ on. *Ann.~ k;-A copy of this reference is not being furnisneo with this uOffice action. (ee MPEP u.u5a).) Dates in MM-YYYY format are publication dates. Classifications may be US or foreign. U.S. Patent and Trademark Office PTO-892 (Rev. 01-2001) Notice of References Cited
~n
Part of Paper No. 20100510
PAT-NO: TITLE:
PUBN-DATE:
JP02004295798A SECURITY SYSTEM

October 21, 2004
DOCUMENT-IDENTIFIER:JP 2004295798 A
INVENTOR-INFORMATION: NAME COUNTRY
SAKAKIBARA, NOBUHIRON/A N/A KASANO, NORIHIRO
ASSIGNEE-INFORMATION: NAME COUNTRY
JAPAN BEST RESCUE SYSTEM KKN/A N/A OCEAN NETWORK CO LTD
APPL-NO:
JP2003090397 2003
APPL-DATE:March 28, INT-CL (IPC):
G08B013/194 , G06T003/00 G08B025/10 , H04N007/18
G06T007/60 , G08B025/00 ,
ABSTRACT:
PROBLEM TO BE SOLVED: To provide a low cost security system which can decide whether a photographed moving object is an intruder accurately and actuate a notifying means, only if the object is an intruder. SOLUTION: The security system is composed of a monitoring camera 51 takes photographs of a moving object and outputs it as animation data; a frame stripping off means 54 inputs the animation data and divide it into frame images with a certain time interval; a conversion means 58 decides whether two frame.images separated in turn have any change on the image by pixel and transforms the frame image into numerical string data, on the basis of determined results; a code-generating means 59 generates geometrical codes, corresponding
to a shape profile of the moving object, on the basis of the
numerical string data, a deciding means 63 decides whether the moving
4/19/10, EAST Version: 2.4.1.1
object is a human being, on the basis of the geometrical codes; and an notification control means. The notification control means actuates a notifying means, when the moving object is decided as being a human being. COPYRIGHT: (C)2005,JPO&NCIPI
4/19/10, EAST Version:.2.4..1
er Paperwork Reduction Act of 1995, no personsare the

i ii Ai I
vu MD .- nrJ numoer. Ia - 1 lc...aa nor a n.n:..res:.,.....:..... ond a metnn required to res ___ I tt._-1 mIU o mormalin unless n lspavs a vaco uec conuol numoer.
U 1 t I1'all irurlrrr r r YI IIGJ.I \
PTO/SB/21 (09-04) Approved for use through 0713112006. OMB 0651-0031 U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE
Application Number Filing Date First Named Inventor. Art Unit
11/231,353 September 19, 2005 Ken P. CHENG 2621 Not Yet Assigned 577832000200
I
TRANSMITTAL
FORM
Examiner Name Total Number of Pagesin This Submission 6 Attomey Docket Number
Fee Transmittal Form r eeTa

L Fee Attached FAppeals
L._J
Drawing(s)
rto Licensing-related Papers
D D
1 F
L
After Allowance Communication TC Appeal Communication to Board of and Interferences
F]
Amendment/Reply After Final
L_
D
Petition Petition to Convert to a Provisional Application
i Appeal
Communication to TC (Appeal Notice, Brief, Reply Brief) Proprietary Information
Affidavits/declaration(s)
Li

s a m m n Ter i alD i cl i er
Power of Attorney, Revocationddress
Status Letter
Extension of Time Request
I
0
Other Enclosure(s)
( l a se pe
0
D
Express Abandonment Request Information Disclosure Statement
F]
Q
Request for Refund
C, Number ofCD(s)
CD, Number of CD(s)
Landscape Table on CD Remarks
One (1) copy of each of twelve (12) cited non-patent literature references Return Receipt Postcard
(5 pages)
] Certified Copy of Priority

Document(s) Reply to Missing Parts/ Incomplete Application
] Reply to Missing Parts under l37 CFR 1.52 or 1.53
Total page count of this submission as cited above does not include the enclosed copies-of cited references.
SIGNATURE
APPLICANT, ATTORNEY, OR AGENT
Firm Name Signature

Printed Date
MO
IS
N & FOERS
LLP
Customer No. 20872
name
'Stephen C. Durant October 13, 2006 Reg. No. 31,506
I hereby certify that this paper (along with any paper referred to as being attached or enclosed) is being deposited with the U.S. Postal Service on envelope adressed to: Mail Stop Amendment. Commissioner for the date shown below with sufficient postage as First Class Mail, in Patents. P.O. Box 1450. Alexandria. Virginia 22313-1450.
Dated, Octobeflr 13
al : c
r .
2006
nature: g,@,,wl.
Todd V. Leone
sf-2209783 vl
, certify that this correspondence and any enclosures referenced therein ngdeposited with the U.S. Postal Service on the date first shown below
ficient postage as First Class Mail, in an envelope addressed to: Mail

mendment, Commissioner for Patents, P.O. Box 14F, Alexandria, 2313-1450.
October 13. 2006 Signature:
Patent Docket No. 577832000200
IN THE UNITED STATES PATENT AND TRADEMARK OFFICE In re Patent Application of: Ken P. CHENG et al. Serial No.: 11/231,353 Filing Date: September 19, 2005 For: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEMS
Examiner: Not Yet Assigned Group Art Unit: 2621
INFORMATION DISCLOSURE STATEMENT UNDER 37 C.F.R. 1.97 & 1.98 Mail Stop AMENDMENT Commissioner for Patents P.O. Box 1450 Alexandria, Virginia 22313-1450 Dear Sir: Pursuant to 37 C.F.R. 1.97 and 1.98, Applicants submit for consideration in the above-identified application the documents listed on the attached Form PTO/SB/08a/b. Copies of foreign documents and non-patent literature are submitted herewith. Document no. 10 on the attached Form PTO/SB/O8a/b is a pending non-published U.S. patent application, and in accordance with the Waiver of the Copy Requirement in 37 CFR 1.98 for Cited Pending U.S. Patent Applications, a copy is not submitted herewith. The Examiner is requested to make these documents of record.
sf=-2185936
Application No. 11/231,353 Filed: 09/19/2005 Information Disclosure Statement
Docket No. 577832000200
This Information Disclosure Statement is submitted: Q With the application; accordingly, no fee or separate requirements are required. Before the mailing of a first Office Action after the filing of a Request for Continued Examination under 1.114. However, if applicable, a certification under 37 C.F.R. 1.97
(e)(1) has been provided.
Within three months of the application filing date or before mailing of a first Office Action on the merits; accordingly, no fee or separate requirements are required. However, if applicable, a certification under 37 C.F.R. 1.97 (e)(1) has been provided.
After receipt of a first Office Action on the merits but before mailing of a final Office Action or Notice of Allowance. ] [[O A fee is required. A check in the amount of_ is enclosed. A fee is required. Accordingly, a Fee Transmittal form (PTO/SB/17) is attached to this submission in duplicate. A Certification under 37 C.F.R. 1.97(e) is provided above; accordingly; no fee is believed to be due.
After mailing of a final Office Action or Notice of Allowance, but before payment of the issue fee.
O
L
A Certification under 37 C.F.R. 1.97(e) is provided above and a check in the amount of_ is enclosed. A Certification under 37 C.F.R. 1.97(e) is provided above and a Fee Transmittal form (PTO/SB/17 is attached to this submission in duplicate.) Applicants would appreciate the Examiner initialing and returning the Form
PTO/SB/08a/b, indicating that the information has been considered and made of record herein. The information contained in this Information Disclosure Statement under 37 C.F.R. 1.97 and 1.98 is not to be construed as a representation that: (i) a complete search has been made; (ii) additional information material to the examination of this application does not exist; (iii) the information, protocols, results and the like reported by third parties are accurate or enabling; or (iv) the above information constitutes prior art to the subject invention.
sf-2185936
Application No. 11/231,353 Filed: 09/19/2005
Docket No. 577832000200
Information Disclosure Statement
In the unlikely event that the transmittal form is separated from this document and the Patent and Trademark Office determines that an extension and/or other relief (such as payment of a fee under 37 C.F.R. 1.17 (p)) is required, Applicants petition for any required relief including extensions of time and authorize the Commissioner to charge the cost of such petition and/or other fees due in connection with the filing of this document to Deposit Account No. 03-1952 referencing 577832000200.
Dated: 0ct-
I ) ado
By
Step en C. urant Registration No.: 31,506 MORRISON & FOERSTER LLP 425 Market Street San Francisco, California 94105-2482 (415) 268-6982
sf-2185936
ALTERNATIVE TO PTO/SB/08alb (07-05)
Complete if Known
te for form 1449/PTO
Application Number
11/231,353 September 19, 2005 Ken P. CHENG 2621 Not Yet Assigned
ORMATION DISCLOSURE STATEMENT BY APPLICANT

(Use as many sheets as necessary) Sheet I 1 of 2
Filing Date First Named Inventor Art Unit
Examiner Name
577832000200

Examiner Initials' Cite No.' Document Number Number-Kind (ifknown) Codea' Publcation Date MM-OD-YYYY Name of Patentee or Applicant of Cited
Document
Pages. Columns. Lines, Where Relevant Passages or Relevant Figures Appear
1. 2. 3. 4.5. 6. 7. 8. 9.
Examiner Cite
US-6,591,224 US-6,609,198 US-6,697,103-B1 US-6,970,582-A1 US-20020138768-A1 US-20020190119-A1 US-20040081338-A1 US-20050265607-A1 US-20060112039-A1

Foreign Patent Document
07-08-2003 08-19-2003 02-24-2004 11-29-2005 09-26-2002 12-19-2002 04-29-2004 12-01-2005 05-25-2006

Publication
MM-DD-YYYY
Sullivan et al. Wood et al. Fernandez et al. Langley Murakami et al. Huffman Takenaka Chang Wang et al.
Pages, Columns. Lines,
Name of Patentee or Applicant of Cited Document Where Relevant Passages or Relevant Figures Appear
FOREIGN PATENT DOCUMENTS

EamiN CiDate Intials' No.' CountryCoda-Numbe-Gnd Code(ifknown)
To I I
*EXAMINER: Initial if information considered, whether or notcitation is in conformance with MPEP 609. Draw line through citation if not in conformance and not considered. Include copy of this form with next communication to applicant. ' Applicants unique citation designation number (optional). ' See Kinds Codes of USPTO Patent Documents at www.usplo.gov or MPEP 901.04. 3Enter Office that issued the document, by the two-letter code (WIPO Standard ST.3). 'For Japanese patent documents, theindication of the year of thereign of the Emperor must precede the serial number of the patent document. sKind of document by the appropriate symbols as indicated on the document under WIPO Standard ST. 16 if possible. Applicant is to place a check mark here if English language Translation is attached.
NON PATENT LITERATURE DOCUMENTS

Initials'
-
No.'
Include name of the author (in CAPITAL LETTERS), title of the article (when appropriate), title of the item (book,
magazine, journal, serial, symposium, catalog, etc.), date, page(s), volume-issue number(s), publisher, city and/or countrywhere published.
/JG/
'4
s.
10. 11.
12.
13.
h et al., "Robust Perceptual Color Identification" US-11/229,091, d: 9/16/2005 Belhumeur, A. et al. (1997). "Eigenfaces vs. Fisherfaces: recognition using class specific linear projection", IEEE Transactions on Pattern Analysis and Machine Intelligence 19(7): 711-720. Brunelli, R. and D. Falavigna. (1995). "Person identification using multiple cues," IEEE Transactions on Pattern Analysis and Machine Intelligence 17(10): 955-966. Brunelli, R. et al. (1995). "Automatic Person Recognition by Using Acoustic and Geometric Features", Machine Vision and Applications
8: 317-325.
14.
ExI n e ISig anaturer I
HONG, Lin and Anil K. Jain. (1998). "Integrating faces and fingerprints for personal identification," IEEE Transactions on Pattern Analysis
!eoeGrn I JeromeGrant
C at e idered D ns o
I05/10/2010
sf- 2185930
ALL REFERENCES CONSIDERED EXCEPT WHERE LINED THROUGH. /JG/
ALTERNATIVE TO PTOISB/08a/b (07-05)
Substitute for form 14491PTO Application Number
Complete if Known
11/231,353
September 19, 2005 Ken P.CHENG

2621 Not Yet Assigned
Art Unit
Examiner Name
Sheet
of
577832000200
15.
16.
17. 18.
and Machine Intelligence 20(12): 1295 - 1307. International Search Report mailed on April 2006 for PCT Patent Application Number PCT/US05/33378 filed on September 19, 2005, one page. JAIN, A. K. et al. (1997). "On-Line Fingerprint Verification," IEEE Transactions on Pattern Analysis and Machine Intelligence archive 19(4): 302- 314. Kittler, J. et al. (1998). "On combining classifiers", IEEE Transactions on Pattern Analysis and Machine Intelligence 20(3): 226-239. Lu X et al. (2003). "Combing classifiers for face recognition", IEEE International Conference on Multimedia Systems and Expo, Baltimore,
MD, July.
19.
Maio, D. et al. (2002). "FVC2000: fingerprint verification competition", IEEE Transactions on Pattern Analysis and Machine Intelligence 24(3):
402 -412.
20.
21.
Phillips, P.J. et al. (2000). "The FERET evaluation methodology for face-recognition algorithms", IEEE Transactions on Pattern Analysis and Machine Intelligence 22(10): 1090- 1104. Senior, A. (2001). "A combination fingerprint classifier", IEEE Transactions on Pattern Analysis and Machine Intelligence 23(10):
1165 1174.
22.
TURK, A. and A. Pentland. (1991). "Eigenfaces for Recognition". Journal of Cognitive Neuroscience 3 (1): 71-86.
'EXAMINER: Initial it reference considered, whether or not citation is in conformance with MPEP 609. Draw linethrough citation if not in conformance and not considered. Include copy of thisform with next communication to applicant. 'Applicant's unique citation designation number (optional). Appicant is to place a check mark here if English language Translation isattached.
2
Examiner
sf-2185930
/Jeroe Grant /
at e
05/10/2010
ALL REFERENCES CONSIDERED EXCEPT WHERE LINED THROUGH. /JG!

UNITED STATES DEPARTMENT OF COMMERCE United States Patent and Trademark Office
Address: COMMISSIONER FOR PATENTS P.O. Box 1450 Alexandria, Virginia 22313-1450 www.uspto.gov
APPLICATION NO.
FLING DATE 09/19/2005

7590 12/27/2010
ATTORNEY DOCKET NO.
I
11/231,353
KEN P. CHENG 20691 REID LANE SARATOGA, CA 95070-5325
SEXAMINER
GRANT II, JEROME
ART UNIT
2625
PAPER NUMBER
MAIL DATE 12/27/2010
DELIVERY MODE PAPER
Please find below and/or attached an Office communication concerning this application or proceeding.
The time period for reply, if any, is set in the attached communication.
PTOL-90A (Rev. 04/07)
Application No.
Applicant(s) CHENG ET AL. Art Unit 2625
Notice of Abandonment
-- The MAILING DATE of this communication appears on the cover sheet with the correspondence address--
This application is abandoned in view of: 1. Applicant's failure to timely file a proper reply to the Office letter mailed on 13 May 2010. (a) O A reply was received on (with a Certificate of Mailing or Transmission dated _ ), which is after the expiration of the month(s)) which expired on period for reply (including a total extension of time of _ (b) O A proposed reply was received on , but it does not constitute a proper reply under 37 CFR 1.113 (a) to the final rejection. (A proper reply under 37 CFR 1.113 to a final rejection consists only of: (1) a timely filed amendment which places the application in condition for allowance; (2) a timely filed Notice of Appeal (with appeal fee); or (3) a timely filed Request for Continued Examination (RCE) in compliance with 37 CFR 1.114). (c)
A reply was received on but it does not constitute a proper reply, or a bona fide attempt at a proper reply, to the nonfinal rejection. See 37 CFR 1.85(a) and 1.111. (See explanation in box 7 below).
(d) No reply has been received. 2. O Applicant's failure to timely pay the required issue fee and publication fee, if applicable, within the statutory period of three months from the mailing date of the Notice of Allowance (PTOL-85). (a) O The issue fee and publication fee, if applicable, was received on _ (with a Certificate of Mailing or Transmission dated .), which is after the expiration of the statutory period for payment of the issue fee (and publication fee) set in the Notice of Allowance (PTOL-85). (b) O The submitted fee of $. (c) is insufficient. A balance of $
.
is due. .
The issue fee required by 37 CFR 1.18 is $
The publication fee, if required by 37 CFR 1.18(d), is $
O The issue fee
and publication fee, if applicable, has not been received.
3.0 Applicant's failure to timely file corrected drawings as required by, and within the three-month period set in, the Notice of Allowability (PTO-37). (a) O Proposed corrected drawings were received on _ (with a Certificate of Mailing or Transmission dated , which is after the expiration of the period for reply. (b)
No corrected drawings have been received.
4. O The letter of express abandonment which is signed by the attorney or agent of record, the assignee of the entire interest, or all of the applicants. 5. O The letter of express abandonment which is signed by an attorney or agent (acting in a representative capacity under 37 CFR 1.34(a)) upon the filing of a continuing application. 6. O The decision by the Board of Patent Appeals and Interference rendered on of the decision has expired and there are no allowed claims. 7. O The reason(s) below:
_
and because the period for seeking court review
Petitions to revive under 37 CFR 1.137(a) or (b), or requests to withdraw the holding of abandonment under 37 CFR 1.181, should be promptly filed to minimize any negative effects on patent term.
IU.S Patent anrademark PTO-1432 (Rev: Office 04-01)
Notice of Abandonment
Part
of
Paper
No.
20101220

Title of Title of Invention:
Invention:
Adaptive multi-modal integrated biometric identification detection and suvi.nc ytm

surveillance systems
First Named Inventor/Applicant Name:
Ken Prayoon Cheng KEN P.CHENG
20691 REID LANE Correspondence Address: SARATOGA

US-
CA
95070-5325
Filer: Filer Authorized By: Attorney Docket Number: Receipt Date: Filing Date: Time Stamp: Application Type:
Robert Hayden
01-FEB-2011 19-SEP-2005 19:03:33 Utility under 35 USC 111(a)
Submitted with Payment Payment Type Payment was successfully received in RAM
yes
Deposit Account $810
RAM confirmation Number Deposit Account Authorized User
7121. 161331
The Director of the USPTO is hereby authorized to charge indicated fees and credit any overpayment as follows: Charge any Additional Fees required under 37 C.F.R. Section 1.21 (Miscellaneous fees and charges)
File Listing:
Document
Number Document Description File Name
File Size(Bytes)/
Message Digest 205807
Multi
Part /.zip
Pages
(ifappl.)
Petition for review by the Office of Petitions. Warnings: Information:
sb0064.pdf
no
99076d22654093bf1773808ccc01b73cdbd 6539f
5330_07-Amendment-A.pdf
Multipart Description/PDF files in .zip description Document Description Amendment/Req. Reconsideration-After Non-Final Reject Start 1 End 1
1
3
Claims
Applicant Arguments/Remarks Made in an Amendment Warnings: Information:
349199
5330_07-POA-and-373b.PDF
1b9f4f7007956da6b27B4451b250a049f9 4170a
Multipart Description/PDF files in .zip description Document Description Power of Attorney Start 1 End 1
Assignee showing of ownership per 37 CFR 3.73(b). Warnings: Information:
Fee Worksheet (PTO-875)
fee-info.pdf
Total Files Size (in bytes)
608278
Doc Code: PET.OP Document Description: Petition for Review by the Office of Petitions
PETITION FOR REVIVAL OF AN APPLICATION FOR PATENT ABANDONED UNINTENTIONALLY UNDER 37 CFR 1.137(b)
First named inventor: Ken Prayoon Cheng
Docket Number (Optional)
5330.07 (SMC)
Application No.: 11/231,353 Filed: 09-19-2005
Art Unit: 2625
Examiner: JEROME GRANT II
Title: Adaptive Multi-Modal Integrated Biometric Identification Detection and Surveillance Systems Attention: Office of Petitions Mail Stop Petition Commissioner for Patents P.O. Box 1450 Alexandria, VA 22313-1450 FAX (571) 273-8300 NOTE: If information or assistance is needed in completing this form, please contact Petitions Information at (571) 272-3282. The above-identified application became abandoned for failure to file a timely and proper reply to a notice or action by the United States Patent and Trademark Office. The date of abandonment is the day after the expiration date of the period set for reply in the office notice or action plus any extensions of time actually obtained. APPLICANT HEREBY PETITIONS FOR REVIVAL OF THIS APPLICATION NOTE: A grantable petition requires the following items: (1) Petition fee; (2) Reply and/or issue fee; (3) Terminal disclaimer with disclaimer fee - required for all utility and plant applications filed before June 8, 1995; and for all design applications; and (4) Statement that the entire delay was unintentional 1. Petition Fee Q~ Small entity-fee $ 810 Q Other than small entity-fee $ (37 CFR 1.17(m)). Application claims small entity status. See 37 CFR 1.27. (37 CFR 1.17(m))
2. Reply and/or fee A. The reply and/or fee to the above-noted Office action in the form of a written reply (identify type of reply):
DB.
has been filed previously on
is enclosed herewith. The issue fee and publication fee (if applicable) of $
has been paid previously on
Sis
enclosed herewith.
[Page 1 of 2]' This collection of information is required by 37 CFR 1.137(b). The information is required to obtain or retain a benefit by the public which is to file (and by the USPTO to process) an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.11 and 1.14. This collection is estimated to take 1.0 hour to complete, including gathering, preparing, and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Any comments on the amount of time you require to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S. Patent and Trademark Office, U.S. Department of Commerce, P.O. Box 1450, Alexandria, VA 22313-1450. DO NOT SEND FEES OR COMPLETED FORMS TO THIS ADDRESS. SEND TO: Mail
Stop Petition, Commissioner for Patents, P.O. Box 1450, Alexandria, VA 22313-1450. If you need assistance in completing the form, call 1-800-PTO-9199 and select option 2.
PTO/SB/64 (07-09) Approved for use through 07/31/2012. OMB 0651-0031 U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number
3. Terminal disclaimer with disclaimer fee Q~ Since this utility/plant application was filed on or after June 8, 1995, no terminal disclaimer is required.
O A terminal disclaimer (and disclaimer fee (37 CFR 1.20(d)) of $
for a small entity or $ other than a small entity) disclaiming the required period of time is enclosed herewith (see PTO/SB/63).
for
4. STATEMENT: The entire delay in filing the required reply from the due date for the required reply until the filing of a grantable petition under 37 CFR 1.137(b) was unintentional. [NOTE: The United States Patent and Trademark Office may require additional information if there is a question as to whether either the abandonment or the delay in filing a petition under 37 CFR 1.137(b) was unintentional (MPEP 711.03(c), subsections (lll)(C) and (D)).]
WARNING:
Petitioner/applicant is cautioned to avoid submitting personal information in documents filed in a patent application that may contribute to identity theft. Personal information such as social security numbers, bank account numbers, or credit card numbers (other than a check or credit card authorization form PTO-2038 submitted for payment purposes) is never required by the USPTO to support a petition or an application. If this type of personal information is included in documents submitted to the USPTO, petitioners/applicants should consider redacting such personal information from the documents before submitting them to the USPTO. Petitioner/applicant is advised that the record of a patent application is available to the public after publication of the application (unless a non-publication request in compliance with 37 CFR 1.213(a) is made in the application) or issuance of a patent. Furthermore, the record from an abandoned application may also be available to the public if the application is referenced in a published application or an issued patent (see 37 CFR 1.14). Checks and credit card authorization forms PTO-2038 submitted for payment purposes are not retained in the application file and therefore are not publicly available. /Robert Hayden, #42,645/ Signature Robert Hayden Type or Printed name Peters Verny, LLP Address 425 Sherman Avenue, Suite 230, Palo Alto, CA 94306 Address February 1, 2011 Date 42,645 Registration Number, If applicable 650.324.1677 Telephone Number
Enclosures:
Fee Payment
/IIReply i Terminal
Disclaimer Form
Additional sheets containing statements establishing unintentional delay Other: Power of Attorney and Statement under 3.73(b)
CERTIFICATE OF MAILING OR TRANSMISSION [37 CFR 1.8(a)] I hereby certify that this correspondence is being: SDeposited with the United States Postal Service on the date shown below with sufficient postage as Q first class mail in an envelope addressed to: Mail Stop Petition, Commissioner for Patents, P. O. Box 1450, Alexandria, VA 22313-1450. QTransmitted by facsimile on the date shown below to the United States Patent and Trademark Office at (571) 273-8300. Date Signature
Typed or printed name of person signing certificate

[Page 2 of 21
Privacy Act Statement

The Privacy Act of 1974 (P.L. 93-579) requires that you be given certain information in connection with your submission of the attached form related to a patent application or patent. Accordingly, pursuant to the requirements of the Act, please be advised that: (1) the general authority for the collection of this information is 35 U.S.C. 2(b)(2); (2) furnishing of the information solicited is voluntary; and (3) the principal purpose for which the information is used by the U.S. Patent and Trademark Office is to process and/or examine your submission related to a patent application or patent. If you do not furnish the requested information, the U.S. Patent and Trademark Office may not be able to process and/or examine your submission, which may result in termination of proceedings or abandonment of the application or expiration of the patent. The information provided by you in this form will be subject to the following routine uses: 1. The information on this form will be treated confidentially to the extent allowed under the Freedom of Information Act (5 U.S.C. 552) and the Privacy Act (5 U.S.C. 552a). Records from this system of records may be disclosed to the Department of Justice to determine whether the Freedom of Information Act requires disclosure of these records. 2. A record from this system of records may be disclosed, as a routine use, in the course of presenting evidence to a court, magistrate, or administrative tribunal, including disclosures to opposing counsel in the course of settlement negotiations. 3. A record in this system of records may be disclosed, as a routine use, to a Member of Congress submitting a request involving an individual, to whom the record pertains, when the individual has requested assistance from the Member with respect to the subject matter of the record. 4. A record in this system of records may be disclosed, as a routine use, to a contractor of the Agency having need for the information in order to perform a contract. Recipients of information shall be required to comply with the requirements of the Privacy Act of 1974, as amended, pursuant to 5 U.S.C. 552a(m). 5. A record related to an International Application filed under the Patent Cooperation Treaty in this system of records may be disclosed, as a routine use, to the International Bureau of the World Intellectual Property Organization, pursuant to the Patent Cooperation Treaty. 6. A record in this system of records may be disclosed, as a routine use, to another federal agency for purposes of National Security review (35 U.S.C. 181) and for review pursuant to the Atomic Energy Act (42 U.S.C. 218(c)). 7. A record from this system of records may be disclosed, as a routine use, to the Administrator, General Services, or his/her designee, during an inspection of records conducted by GSA as part of that agency's responsibility to recommend improvements in records management practices and programs, under authority of 44 U.S.C. 2904 and 2906. Such disclosure shall be made in accordance with the GSA regulations governing inspection of records for this purpose, and any other relevant (i.e., GSA or Commerce) directive. Such disclosure shall not be used to make determinations about individuals. 8. A record from this system of records may be disclosed, as a routine use, to the public after either publication of the application pursuant to 35 U.S.C. 122(b) or issuance of a patent pursuant to 35 U.S.C. 151. Further, a record may be disclosed, subject to the limitations of 37 CFR 1.14, as a routine use, to the public if the record was filed in an application which became abandoned or in which the proceedings were terminated and which application is referenced by either a published application, an application open to public inspections or an issued patent. 9. A record from this system of records may be disclosed, as a routine use, to a Federal, State, or local law enforcement agency, if the USPTO becomes aware of a violation or potential violation of law or regulation.
APPLICANTS: SERIAL NO.: CONF. NO.: FILING DATE: TITLE:
Ken Prayoon Cheng et al. 11/231,353 4531 September 19, 2005 Adaptive Multi-Modal Integrated Biometric Identification Detection and Surveillance Systems
EXAMINER: ART UNIT: ATTY. DKT. NO.:
Jerome Grant II 2625 5330.07 (SMC)
Amendment A
Sir: In response to the Office action mailed on May 13, 2010, please amend the application as follows and consider the following remarks. A Petition for Revival of an Application for Patent Abandoned Unintentionally under 37 CFR 1.137(b) and the fee required under 37 CFR 1.17(m) are submitted herewith.
5330.07 (SMC)1
IN THE CLAIMS: The following listing of claims will replace all prior versions, and listings, of claims in the application.
1-13. (cancelled)
14. (currently amended) A The surveillance method of claim, fu therincluding comprising: area a event using at least one evet sensor disposed in a security area of a surveillance region to sense an occurrence of a potential security breach event; using at least one camera with a view of the security area in which the event is sensed to gather biometric information concerning at least one person in the vicinity of the security area at about the time of the sensing of the
event;
producing a subject dossier corresponding to the at least one person, wherein the subject dossier including includes at least two biometric signatures; matching biometric information of one or more persons captured by one or more other cameras in the vicinity of the at least one camera with corresponding , , biometric information in the subject dossier; and fu her-includm fusing the at least two signatures arind including the fused signature in the subject dossier.
5330.07 (SMC)
15. (currently amended) A The surveillance method of claim 1 further including comprising: using at least one event sensor disposed in a security area of a surveillance region to sense an occurrence of a potential security breach event; using at least one camera with a view of the security area in which the event is sensed to gather biometric information concerning at least one person in the vicinity of the security area at about the time of the sensing of the
event,
producing a subject dossier corresponding to the at least one person, wherein the subject dossier including ineludes at least two biometric signatures; matching biometric information of one or more persons captured by one or more other cameras in the vicinity of the at least one camera with corresponding biometric information in the subject dossier; and fuher-i+ udg incrementally fusing the at least two signatures and including the fused signature in the subject dossier.
16-20. (cancelled)
5330.07 (SMC)
REMARKS
Claims 14 and 15 are presently pending following the cancellation of claims 1-13 and 16-20 herein. The Applicants thank the Examiner for the indication of allowable subject matter in both pending claims. Each claim has been amended to place the claim in independent form. In view of the cancellation of claims 1-13 and 16-20, their rejections under 35 U.S.C. 103(a) as unpatentable over Reilly et al. (USPGP 2009/0322873) either alone or in combination with Monroe (USPGP 2004/0117638) are deemed to be moot. The Applicants, however, do not agree that the cancelled claims are obvious over the cited references, reserve the right to pursue these claims in a continuation application, and cancel them herein merely to expedite the allowance of the present application.
Both pending claims are now allowable and the Applicants therefore request a Notice of Allowance from the Examiner. Should the Examiner have questions, the Applicants' undersigned attorney may be reached at the number provided.
Ken Prayoon Cheng et al.
Dated: February 1, 2011
By:
/Robert Hayden, #42,645/ Robert Hayden, Reg. No. 42,645 Peters Verny, LLP 425 Sherman Avenue, Suite 230 Palo Alto, CA 94306 TEL: (650) 324-1677 FAX: (650) 324-1678
5330.07 (SMC)
POWER OF ATTORNEY TO PROSECUTE APPLICATIONS BEFORE THE USPTO

I hereby revoke all previous powers of attorney given in the application identified in the attached statement under 37 CFR 3.73(b). I hereby appoint:
r Practitioners associated with the Customer Number:
OR
23308
Registration Number . Name Registration Number
S Practitioner(s) named below (if more

Name
than ten patent practitioners are to be named, then a customer number must be used):
as attorney(s) or agent(s) to represent the undersigned before the United States Patent and Trademark Office (USPTO) in connection with any and all patent applications assigned ony to the undersigned according to the USPTO assignment records or assignment documents attached to this form in accordance with 37 CFR 3.73(b). Please change the correspondence address for the application identified in the attached statement under 37 CFR 3.73(b) to:
SThe
OR
Firm or
address associated with Customer Number:
23308
State Zip
Individual Name Address City
Country
Telephone Email
Assignee Name and Address:
Proximex Corporation 440 N. Wolfe Rd. Sunnyvale, CA 94085 A copy of this form, together with a statement under 37 CFR 3.73(b) (Form PTOISBI96 or equivalent) is required to be filed in each application in which this form is used. The statement under 37 CFR 3.73(b) may be completed by one of the practitioners appointed in this form if the appointed practitioner is authorized to act on behalf of the assignee, and must identify the application in which this Power of Attorney is to be filed.
SIGNATURE of Assignee of Record
The individual whose signature and title is supplied below is authorized to act on behalf of the assignee
.Signature
Name Title
Thi.
/ AII NIAn
~.Date
Ken Prayoon Cheng
President
A7
Jr'/20
Telephone 408-524-1510
rco
.. l urar/
This olliuun u Informaun i application.by 3/ l.R 1.3 1, 1.. and 1.3.. me inrormation is require to obtain or retain a beneit by the public which is to file (and required by the USPTO to process) an Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.11 and 1.14. This collection is estimated to take 3 minutes
IM-Minn
;n
ti~'17
'11
1 ']n
--
4 ',o
'r"-
i.
w1: w
:_
..
_..-_._
L..
..+
to complete, including gathering, preparing, and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Any comments on the amount of time you require to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S. Patent and Trademark Office, U.S. Department of Commerce, P.O. Box 1450, Alexandria, VA 22313-1450. DO NOT SEND FEES OR COMPLETED FORMS TO THIS ADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1460, Alexandria, VA 22313-1450.
If you need ass/stance in completing the form, call 1-800-PTO-9199 and select option 2
STATEMENT UNDER 37 CFR 3.73(b) Applicant/Patent Owner: Ken Pravoon Chena et al Application No./Patent No.: 11/231,353 Filed/Issue Date: September 19, 2005
Entitled: Adaptive multi-modal integrated biometric identification detection and surveillance systems
Prnximpx Conrpnratinn (Name of Assignee)
nCrpnm rAtinn (Type of Assignee, e.g., corporation, partnership, university, government agency, etc.)
states that it is: 1, the assignee of the entire right, title, and interest; or 2.
O an assignee
of less than the entire right, title and interest (The extent (by percentage) of its ownership interest is
%)
inthe patent application/patent identified above by virtue of either: A.QAn assignment from the inventor(s) of the patent application/patent identified above. The assignment was recorded in the United States Patent and Trademark Office at Reel 017473 , Frame 0506 , or for which a copy thereof is attached.
OR
B.4 A chain of title from the inventor(s), of the patent application/patent identified above, to the current assignee as follows:
1. From:
To:
Reel
2. From:
, Frame
. , or for which a copy thereof is attached.

To:
Reel
3. From:
, Frame
'To:
_,
, or for which a. copy. thereof is attached.
Reel Frame , or for which a copy thereof is attached.
Additional documents in the chain of title are listed on a supplemental sheet.
As required by 37 CFR 3.73(b)(1)(i), the documentary evidence of the chain of title from the original owner to the assignee was, or concurrently is being, submitted for recordation pursuant to 37 CFR 3.11. [NOTE: A separate copy (i.e., a true copy of the original assignment document(s)) must be submitted to Assignment Division in accordance with 37 CFR Part 3, to record the assignment in the records of the USPTO. See MPEP 302.08] The undersigned (whose title is supplied ow) is authorized to act on behalf of the assignee.
V
a
Signature.
Ken Pravoon Chen Printed or Typed Name President Title
Date 408-524-1510 Telephone Number
This collection of information is required by 37 CFR 3.73(b). The information is required to obtain or retain a benefit by the public which is to file (and by the USPTO to process) an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.11 and 1.14. This collection is estimated to take 12 minutes to complete, including gathering, preparing, and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Any comments on the amount of time you require to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S. Patent and Trademark Office. U.S. Department of Commerce, P.O. Box 1450, Alexandria, VA 22313-1450. DO NOT SEND FEES OR COMPLETED FORMS TO THIS ADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450, Alexandria, VA 22313-1450.
Electronic Patent Application Fee Transmittal

Application Number: Filing Date: 11231353 19-Sep-2005
Title of Invention:
First Named Inventor/Applicant Name: Filer: Attorney Docket Number: Filed as Small Entity
Ken Prayoon Cheng Robert Hayden
Utility under 35 USC 111(a) Filing Fees

Description Fee Code Quantity AmountS
Sub-Total in Dt
USD($)
Basic Filing:
Pages:
Claims:
Miscellaneous-Filing:
Petition:
Petition-revive unintent.abandoned appl
2453
810
810
Patent-Appeals-and-Interference:
Post-Allowance-and-Post-Issuance:
Extension-of-Time:
Description
Fee Code
Quantity.
Amount
Sub-Total in USD($)
USD($)
Miscellaneous:
Total in USD ($)
810o
UNITED STATES PATENT AND
TRADEMARK
OFFICE
UNITED STATES DEPARTMENT OF COMMERCE United States Patent and Trademark Office Address: COMMISSIONER FOR PATENTS
P.O Box 1450 Alexandria, Virginia 22313-1450 www.uspto.gov
APPLICATION NUMBER
FILING OR 371(C) DATE
ATTY. DOCKET NO.ITLE
11/231,353 23308 PETERS VERNY, L.L.P. 425 SHERMAN AVENUE SUITE 230 PALO ALTO, CA 94306
09/19/2005
Ken Prayoon Cheng
CONFIRMATION NO. 4531 POA ACCEPTANCE LETTER

This is in response to the Power of Attorney filed 02/01/2011. The Power of Attorney in this application is accepted. Correspondence in this application will be mailed to the above-address as provided by 37 CFR 1.33.
/ddinh/
Office of Data Management, Application Assistance Unit (571) 272-4000, or (571) 272-4200, or 1-888-786-0101
page 1 of 1

www.uspto.gov
PETERS VERNY, L.L.P. 425 SHERMAN AVENUE SUITE 230 PALO ALTO, CA 94306
MAILED
MAR 24 2011
In re Application of Ken Prayoon Cheng, et. al. Application No. 11/231,353 Filed: September 19, 2005 Attorney Docket No. 5330.07 (SMC)
OFFICE OFPETITIONS
ON PETITION
This is a decision on the petition under 37 CFR 1.137(b), filed February 1, 2011, to revive the above-identified application. The application became abandoned for failure to file a reply to the non-final Office action mailed May 13, 2010. A Notice of Abandonment was mailed on December 27, 2010. Since the petition satisfies the requirements of 37 CFR 1.137(b) in that petitioner has supplied (1) the reply in the form of an amendment; (2) the petition fee of $810; and (3) a proper statement of unintentional delay, the petition is GRANTED. This application file is being referred to Technology Center Art Unit 2625 for review of the amendment submitted with the present petition. concerning this decision should be directed to the undersigned at

Title of Invention:
Ken Prayoon Cheng 23308 Robert Hayden
5330.07 (SMC) 28-MAR-2011 19-SEP-2005 16:22:22 Utility under 35 USC 111(a)
Submitted with Payment Payment Type Payment was successfully received in RAM RAM confirmation Number Deposit Account Authorized User yes Deposit Account $180 3529 161331
File Listing:
Document Number
Number
Document Description
File NameMessageDigest
File Size(Bytes)/
Message Digest
Multi Part/.zip
Part /.zip
Pages (ifappl.)
(if appl.)
615001 Information Disclosure Statement (IDS)615001 5330_07-DS-03-2011 .pdf Filed

(SB/08)
no
409757129e88fe4a4c43ab71b96f040bel 26977
12
A U.S. Patent Number Citation is required in the Information Disclosure Statement (IDS) form. You may remove the form to add the required data in order to correct the Informational Message or if you chose not to, the image of the form will be processed and be made available within the Image File Wrapper (IFW) system. However, no data will be extracted from this form. Any additional data such as Foreign Patent Documents or Non Patent Literature will be manually reviewed and keyed into USPTO systems. 53307-copending-appspdf Information Disclosure Statement (IDS) Filed (5B/08)5330_07-copending-apps.pdf 50122
7901e270f528803ce5854d34dc3d3375b ac90
This is not an USPTO supplied IDS fillable form PCTU50516961_ISR-and-WO. pdf
45db33c9d57414b678a942b7dbceccbe39 e569c3
416756 3 NPL Documents
NPL Documents
PCTUS0533750_ISR-and-WO. pdf 239134
b400384634effbdbb567252c895f2075103 6c9e6
LDocuments
NPL Documents
NPL Documents
692624 NPL Documents VidSys-Complaint.PDF
le9549d560462af13f763309f0d6abb2f371 07c7
no
11
8461864
NPL Documents
Redstone_2003.PDF
45e491b6al4175e5fbaa5c28blf 119268d4 9056
Warnings: Information: 7777783-ReexaminationRequest.pdf

Request.pdf
3495139
780f67314b28b29c4326a30d2dddf07ac78 4fa5e
10
NPL Documents
1705533
11
Foreign Reference
WOO7044037.pdf
62bede2b099ab34e6be840acc707670f2f7( (391
30157
fee-info.pdf
4fc573053163947fecb4b7066b481 lbaaOE 04db
Warnings: Information: Total Files Size (in bytes): 16642679
This Acknowledgement Receipt evidences receipt on the noted date by the USPTO of the indicated documents, characterized by the applicant, and including page counts, where applicable. It serves as evidence of receipt similar to a Post Card, as described in MPEP 503. New Applications Under 35 U.S.C. 111 If a new application is being filed and the application includes the necessary components for a filing date (see 37 CFR 1.53(b)-(d) and MPEP 506), a Filing Receipt (37 CFR 1.54) will be issued in due course and the date shown on this Acknowledgement Receipt will establish the filing date of the application.
I
National Staae of an International AppDDlication under 35 U.S.C. 371 If a timely submission to enter the national stage of an international application is compliant with the conditions of 35 U.S.C. 371 and other applicable requirements a Form PCT/DO/EO/903 indicating acceptance of the application as a national stage submission under 35 U.S.C. 371 will be issued in addition to the Filing Receipt, in due course. New International ,.r r.. ~- ~~~-~~ ~ Annlication Filed with the USPTO as a Receivina ...Office ~ ~-~~~~~-~-- y - ~ -~-- ~ ~ ~ .. If a new international application is being filed and the international application includes the necessary components for an international filing date (see PCT Article 11 and MPEP 1810), a Notification of the International Application Number and of the International Filing Date (Form PCT/RO/105) will be issued in due course, subject to prescriptions concerning national security, and the date shown on this Acknowledgement Receipt will establish the international filing date of the application.
IN THE
UNITED STATES PATENT AND TRADEMARK OFFICE APPLICANTS: APPLICATION NO.: CONFIRMATION NO.: FILED: TITLE: Ken Prayoon Cheng 11/231,353 4531 September 19, 2005 Adaptive Multi-Modal Integrated Biometric Identification Detection and Surveillance Systems Jerome Grant, II 2625 5330.07 (SMC)
EXAMINER: GROUP ART UNIT: ATTY.DKT.NO.:
MAIL STOP AMENDMENT COMMISSIONER FOR PATENTS P.O. BOX 1450 ALEXANDRIA, VA 22313-1450 List of Co-Pending Patent Applications That May Be Directed Towards Similar Subject Matter Examiner's Initials Serial, Number
12/838,973 11/728,404
First-Named Inventor
Hu Chin Hu Chin
Title Filing Date

Multi-Video Navigation Multi-Video Navigation System 7-19-2010 3-23-2007
Examiner's
Signature:
Date
Doc code: IDS
Doc odes : noApproved Doc description: Information Disclosure Statement (IDS) Filed
U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE
PTO/SBi/08oa (01-10) for use through 07131/2012. OMB 0651-0031
Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it contains a valid OMB control number.
Application Number Filing Date
11231353 2005-09-19
First Named Inventor
Ken Prayoon Cheng
( Not for submission under 37 CFR 1.99)
Art Unit12625
Examiner Name Jerome Grant, II 5330.07 (SMC)
U.S.PATENTS '
Remove Pages,Columns,Lines where
Examiner Cite ExamiNoi No Initial*
Patent Number
Kind Cd Codel
Issue Date
Name of Patentee or Applicant Pages,Columns,Lines where Relevant Passages or Relevant Naeocent of cited Document
Figures Appear
6604093
B1
2003-08-05
Etzion et al.
6906709
B1
2005-06-14
Larkin et al.
6408404
B1
2002-06-18
Michael Ladwig
6711587
B1
2004-03-23
Dufaux
6628829
B1
2003-09-30
Chasen
5912980
1999-06-15
Hunke
7136524
B1
2006-11-14
Goh et al.
6072496
2000-06-06
Guenter et al.
EFS Web 2.1.17
11231353 2005-09-19 Ken Prayoon Cheng
Art Unit
Examiner Name
2625
Jerome Grant, II
5330.07 (SMC)
6625315
B2
2003-09-23
Laumeyer et al.
10
6404900
B1
2002-06-11
Qian et al.
11
6707487
B1
2004-03-16
Aman et al:
12
5473369
1995-12-05
Abe
13
7777783
B1
2010-08-17
Chin et al.
14
7242810
B2
2007-07-10
Chang
15
6008912
1999-12-28
Sato et al.
16
5838465
1998-11-17
Satou et al.
17
6335985
B1
2002-01-01
Sambonsugi et al.
18
7187783
B2
2007-03-06
Moon et al.
19
6157469
2000-12-05
Mestha
EFS Web 2.1.17
Application Number
11231353 2005-09-19
Ken Prayoon Cheng
Filing Date
INFORMATION
DISCLOSURE
(Not for submission under 37 CFR 1.99)
Art Unit Art Examiner Name
Unit
12625 25 Jerome Grant, II 5330.07 (SMC)
20
7525576
B2
2009-04-28
Kannermark et al.
21
7382249
B2
2008-06-03
Fancella
22
7384590
B2
2008-06-10
Kelly et al.
23
7023913
B1
2006-04-04
Monroe et al.
24
7116353
B2
2006-10-03
Hobson et al.
25
7456727
B2
2008-11-25
Pinter et al.
26
7058204
B2
2006-06-06
Hildreth et al.
27
7385590
B2
2008-06-10
Millar et al.
28
7522186
B2
2009-04-21
Arpa et al.
29
30
EFS Web 2.1.17
First Named Inventor Art Unit'

Examiner Name
2625
Jerome Grant, II 5330.07 (SMC)
31
32
33
7196722
B2
2007-03-27
White et al.
34
7295228
B2
2007-11-13
Roberts et al.
35
7242423
B2
2007-07-10
Lin
36
5258837
1993-11-02
Gormley
37
5479574
1995-12-26
Glier et al.
38
5701398
1997-12-23
Glier et al.
39
6757668
B1
2004-06-29
Goebel et al.
40
5769074
1998-06-23
Barnhill et al.
41
6306087
B1
2001-10-23
Barnhill et al.
EFS Web 2.1.17
Art Unit Art Unit

Examiner Name Attorney Docket Number
2625 12625
42
6248063
B1
2001-06-19
Barnhill et al.
43
5835901
1998-11-10
Duvoisin et al.
44
7366174
B2
2008-04-29
MacFaden et al.
45
7319796
B1
2008-01-15
Sharp
46
7286687
B2
2007-10-23
Lindwurm et al.
47
7277485
B1
2007-10-02
Puri
48
7269591
B2
2007-09-11
Wesinger et al.
49
7213174
B2
2007-05-01
Dahlquist et al.
50
7127464
B2
2006-10-24
Wesinger et al.
51
7028034
B2
2006-04-11
Wesinger et al
52
6976207
B1
2005-12-13
Rujn et al.
EFS Web 2.1.17
11231353 2005-09-19 Ken Prayoon Cheng 2625 Jerome Grant, II 5330.07 (SMC)

Attorney Docket Number-
53
6970513
B1
2005-11-29
Puri et al.
54
6968006
B1
2005-11-22
Puri et al.
55
6963899
B1
2005-11-08
Fernandez et al.
56
6909745
B1
2005-06-21
Puri et al.
57
6944319
B1
2005-09-13
Huang et al.
58
7746380
B2
2010-06-29
Maruya et al. Add Remove
If you wish to add additional U.S. Patent citation information please click the Add button. U.S.PATENT APPLICATION PUBLICATIONS Examiner Cite No Initial* Publication Number Kind Publication Codel Date
Pages,Columns,Lines where . Name of Patentee or Applicant PagesColumnsLines where Relevant Passages or Relevant saer of cited DocumentFge Figures Appear
20050222081
Al
2005-10-06
Jakobson et al.
20080294588
Al
2008-11-27
Morris et al.
20100002082
Al
2010-01-07
Buehler et al.
EFS Web 2.1.17
Art Unit'
Examiner Name
2625
20030202102
Al
2003-10-30
Shiota et al.
20040100563
Al
2004-05-27
Sablak et al.
20030107653
Al
2003-06-12
Utsumi et al.
20010048765
Al
2001-12-06
Yi et al.
20070146484
Al
2007-06-28
Horton et al.
20020097322
Al
2002-07-25
Monroe et al.
10
20040001149
Al
2004-01-01
Smith et al.
11
20080218590
Al
2008-09-11
Park et al.
12
20080068464
Al
2008-03-20
Kitagawa et al.
13
20080106597
Al
2008-05-08
Amini et al.
14
20060279628
Al
2006-12-14
Fleming
EFS Web 2.1.17
Art Unit Art Unit

Examiner Name
2625 2625 Jerome Grant, II

5330.07 (SMC)
15
20060221184
Al
2006-10-05
Vallone et al.
16
20070146484
Al
2007-06-28
Horton et al.
17
20050132414
Al
2005-06-16
Bentley et al.
18
20080190949
Al
2008-08-14
Ivanov et al.
19
20050052532
Al
2005-03-10
Elooz et al.
20
20080079554
Al
2008-04-03
Boice
21
20060017807
Al
2006-01-26
Lee etal.
22
20040136574
Al
2004-07-15
Kozakaya et al.
23
20030169908
Al
2003-09-11
Kim et al.
24
20050100209
Al
2005-05-12
Lewis et al.
25
20040022442
Al
2004-02-05
Kim et al.
EFS Web 2.1.17
Application Number
Art Unit
Examiner Name
26
20060203090
Al
2006-09-14
Wang et al.
27
20070154088
Al
2007-07-05
Goh et al.
28
20060284978
Al
2006-12-21
Girgensohn et al.
29
20080088706
Al
2008-04-17
Girgensohn et al. Add Remove
If you wish to add additional U.S. Published Application citation information please click the Add button FOREIGN PATENT DOCUMENTS Examiner Cite Initial* No Foreign Document Number3 Country Code 2 i Kind Publication Code4 Date
Name of Patentee or Pages,Columns,Lines T whrRevat Relevant Applicant of citedwhere T Applicant of cited Passages or Relevant DocumentFigures Appear PROXIMEX RO I
2007/044037
WO
Al
2007-04-19
CORPORATION
If you wish to add additional Foreign Patent Document citation information please click the Add button NON-PATENT LITERATURE DOCUMENTS Examiner Cite Initials* No InitialsNo
Add Remove T5
Include name of the author (in CAPITAL LETTERS), title of the article (when appropriate), title of the item (book, magazine, journal, serial, symposium, catalog, etc), date, pages(s), volume-issue number(s), publisher, city and/or country where published.
PCT/USO5/44656 International Search Report and Written Opinion, June 26, 2006
[]
PCT/USO5/43808 International Search Report and Written Opinion, October 10, 2007
[]
EFS Web 2.1.17
INFORMATION DISCLOSURE STATEMENT BY APPLICANT


PCT/US05/33378 International Search Report and Written Opinion, April 26, 2006
PCT/US05/33750 International Search Report and Written Opinion, May 2, 2007
PCT/US05/16961 International Search Report and Written Opinion, October 17, 2006
O-
VidSys, Inc. Complaint filed in the US District Court for the Eastern district of Virginia on October 19, 2010
Redstone Integrated Solutions Documents, 2003
95/001,525 Reexamination Request for 7,777,783, filed January 21, 2011
0
Add
If you wish to add additional non-patent literature document citation information please click the Add button EXAMINER SIGNATURE Examiner Signature Date Considered
*EXAMINER: Initial if reference considered, whether or not citation is in conformance with MPEP 609. Draw line through a citation if not in conformance and not considered. Include copy of this form with next communication to applicant.
1See Kind Codes of USPTO Patent Documents at www.USPTO.GOV or MPEP 901.04. 2 Enter office that issued the document, by the two-letter code (WIPO Standard ST.3). 3For Japanese patent documents, the indication of the year of the reign of the Emperor must precede the serial number of the patent document. 4 Kind of document by the appropriate symbols as indicated on the document under WIPO Standard ST.16 if possible. 5Applicant is to place a check mark here ii English language translation is attached.
EFS Web 2.1.17
Application Number

Art Unit
Examiner Name
CERTIFICATION STATEMENT Please see 37 CFR 1.97 and 1.98 to make the appropriate selection(s):
That each item of information contained in the information disclosure statement was first cited in any communication from a foreign patent office in a counterpart foreign application not more than three months prior to the filing of the information disclosure statement. See 37 CFR 1.97(e)(1).
OR That no item of information contained in the information disclosure statement was cited in a communication from a foreign patent office in a counterpart foreign application, and, to the knowledge of the person signing the certification after making reasonable inquiry, no item of information contained in the information disclosure statement was known to any individual designated in 37 CFR 1.56(c) more than three months prior to the filing of the information disclosure statement. See 37 CFR 1.97(e)(2).
O
Q
See attached certification statement. The fee set forth in 37 CFR 1.17 (p) has been submitted herewith.
Ii A certification statement is not submitted herewith.

SIGNATURE A signature of the applicant or representative is required in accordance with CFR 1.33, 10.18. Please see CFR 1.4(d) for the form of the signature. Signature Name/Print /Robert Hayden, #42,645/ Robert Hayden Date (YYYY-MM-DD) Registration Number 2011-03-28 42,645
This collection of information is required by 37 CFR 1.97 and 1.98. The information is required to obtain or retain a benefit by the public which is to file (and by the USPTO to process) an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.14. This collection is estimated to take 1 hour to complete, including gathering, preparing and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Any comments on the amount of time you require to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S. Patent and Trademark Office, U.S. Department of Commerce, P.O. Box 1450, Alexandria, VA 22313-1450. DO NOT SEND FEES OR COMPLETED FORMS TO THIS ADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450, Alexandria, VA 22313-1450.
EFS Web 2.1.17
The Privacy Act of 1974 (P.L. 93-579) requires that you be given certain information in connection with your submission of the attached form related to a patent application or patent. Accordingly, pursuant to the requirements of the Act, please be advised that: (1) the general authority for the collection of this information is 35 U.S.C. 2(b)(2); (2) furnishing of the information solicited is voluntary; and (3) the principal purpose for which the information is used by the U.S. Patent and Trademark Office is to process and/or examine your submission related to a patent application or patent. If you do not furnish the requested information, the U.S. Patent and Trademark Office may not be able to process and/or examine your submission, which may result in termination of proceedings or abandonment of the application or expiration of the patent. The information provided by you in this form will be subject to the following routine uses: 1. The information on this form will be treated confidentially to the extent allowed under the Freedom of Information Act (5 U.S.C. 552) and the Privacy Act (5 U.S.C. 552a). Records from this system of records may be disclosed to the Department of Justice to determine whether the Freedom of Information Act requires disclosure of these record s. A record from this system of records may be disclosed, as a routine use, in the course of presenting evidence to a court, magistrate, or administrative tribunal, including disclosures to opposing counsel in the course of settlement negotiations. A record in this system of records may be disclosed, as a routine use, to a Member of Congress submitting a request involving an individual, to whom the record pertains, when the individual has requested assistance from the Member with respect to the subject matter of the record. A record in this system of records may be disclosed, as a routine use, to a contractor of the Agency having need for the information in order to perform a contract. Recipients of information shall be required to comply with the requirements of the Privacy Act of 1974, as amended, pursuant to 5 U.S.C. 552a(m). A record related to an International Application filed under the Patent Cooperation Treaty in this system of records may be disclosed, as a routine use, to the International Bureau of the World Intellectual Property Organization, pursuant to the Patent Cooperation Treaty. A record in this system of records may be disclosed, as a routine use, to another federal agency for purposes of National Security review (35 U.S.C. 181) and for review pursuant to the Atomic Energy Act (42 U.S.C. 218(c)). A record from this system of records may be disclosed, as a routine use, to the Administrator, General Services, or his/her designee, during an inspection of records conducted by GSA as part of that agency's responsibility to recommend improvements in records management practices and programs, under authority of 44 U.S.C. 2904 and 2906. Such disclosure shall be made in accordance with the GSA regulations governing inspection of records forthis purpose, and any other relevant (i.e., GSA or Commerce) directive. Such disclosure shall not be used to make determinations about individuals.
2.
3.
4.
5.
6.
7.
8.
A record from this system of records may be disclosed, as a routine use, to the public after either publication of the application pursuant to 35 U.S.C. 122(b) or issuance of a patent pursuant to 35 U.S.C. 151. Further, a record may be disclosed, subject to the limitations of 37 CFR 1.14, as a routine use, to the public if the record was filed in an application which became abandoned or in which the proceedings were terminated and which application is referenced by either a published application, an application open to public inspections or an issued patent.
9.
A record from this system of records may be disclosed, as a routine use, to a Federal, State, or local law enforcement agency, if the USPTO becomes aware of a violation or potential violation of law or regulation.
EFS Web 2.1.17

CLASSIFICATION OF SUBJECT MATTER A. GO6K 9/00( 2006.01),9/62( 2006.01) IPC:
International application No. PCT/US05/44656
382/167,156 USPC: According to International Patent Classification (IPC) or to both national classification and IPC
B.
FIELDS SEARCHED
Minimum documentation searched (classification system followed by classification symbols) U.S. :382/167,156 Documentation searched other than minimum documentation to the extent that such documents are included in the fields searched
C.
Category *
DOCUMENTS CONSIDERED TO BE RELEVANT Citation of document, with indication, where appropriate, of the relevant passages US 6,711,587 (Dufaux) 23 March 2004 (21.03.2004), column 1, line 50-column 2, line X 15), also column 5, lines 35-45.---y Y Y US 6,628,829 BI( CHASEN) 30 September 2003 (30.09.2003) column 8. US 5,912,980 (HUNKE) 15 June 1999 (15.06.1999) column 13-column 14.
Relevant to claim No. 20-21 22-23 22 23
S
*
Further documents are listed in the continuation of Box C.

Special categories of cited documents:
E
"T"

later document published after the international filing date or priority date and notin conflict with the application butcited to understand the
"A" "E" "L"
document defining the general state of the art which is notconsidered tobe of particular relevance the earlier application or patent published on or after International filing date document which may throw doubts on priority claim(s) or which is cited to
establish the publication date of another citation or other special reason (as
principle or theory underlying the invention "X" document of particular relevance; the claimed invention cannot be considered novel or cannot be considered to involve an inventive step when the document is taken alone.
document of particular relevance; the claimed invention cannot be
"Y"
specified) "O"
"P"
document referring to an oral disclosure, use, exhibition or other means

document published prior tothe international filing date but later than the priority date claimed "&"
document is combined considered to involve an inventive step when the withone or more other such documents, such combination being obvious to a person skilled in the art
document member of the same patent family
Date of the actual completion of the international search 03 May 2006 (03.05.2006)J Name and mailing address of the ISA/US Mail Stop PCT, Attn: ISA/US Commissioner for PatentsnggeWu P.O. Box 1450 Alexandria, Virginia 22313-1450 Facsimile No. (571) 273-3201 Form PCT/ISA/210 (second sheet) (April 2005)
Date of mailing o e e in 4 thorized officer
na search report
Telephone No. 70
-5631
L
PATENT COOPERATION TREATY

From the INTERNATIONAL SEARCHING AUTHORITY TO"To: STEVEN COLBY CARR & FERRELL LLP 2200 GENG ROAD PALO ALTO, CA 94303
Dr '
WRITTEN OPINION OF THE
INTERNATIONAL SEARCHING AUTHORITY

(PCT Rule 43bis.1)
Date of mailing
Applicant's or agent's file reference PA3358PCT PA3358PCT
(day/month/year)
.7 .J N
2006
FOR FURTHER ACTION See paragraph 2 below International filing date (day/month/year) Priority date (day/month/year) 16 September 2006 (16.09.2006)
International application No.
09 December 2005 (09.12.2005) PCT/US05/44656 International Patent Classification (IPC) or both national classification and IPC IPC: GO6K 9/00( 2006.01),9/62( 2006.01) USPC: 382/162,167,156 Applicant PROXIMEX CORPORATION' 1. This opinion contains indications relating to the following items:
S
E
Box No. I
Basis of the opinion Priority Non-establishment of opinion with regard to novelty, inventive step and industrial applicability Lack of unity of invention Reasoned statement under Rule 43bis.1(a)(i) with regard to novelty, inventive step or industrial applicability; citations and explanations supporting such statement Certain documents cited Certain defects in the international application Certain observations on the international application
Box No. II Box No. III Box No. IV

Box No. V
E
E 0
Q Q
Box No. VI Box No. VII
Box No. VIII
2. FURTHER ACTION If a demand for international preliminary examination is made, this opinion will be considered to be a written opinion of the International Preliminary Examining Authority ("IPEA") except that this does not apply where the applicant chooses an Authority other than this one to be the IPEA and the chosen IPEA has notified the International Bureau under Rule 66. 1bis(b) that written opinions of this International Searching Authority will not be so considered. If this opinion is, as provided above, considered to be a written opinion of the IPEA, the applicant is invited to submit to the IPEA a written reply together, where appropriate, with amendments, before the expiration of 3 months from the date of mailing of Form PCT/ISA/220 or before the expiration of 22 months from the priority date, whichever expires later. For further options, see Form PCT/ISA/220. 3. For further details, see notes to Form PCT/ISA/220. . Name and mailing address of the ISA/ US Commissioner for Patents P.O. Box 1450 Alexandria, Virginia 22313-1450 Facsimile No. (571) 273-3201 Form PCT/ISA/237 (cover sheet) (April 2005)
Mail Stop PCT, Anttn: ISA/US
Date of completion of this

opinionBAYAT
rized office
03 May 2006 (03.05.2006) Telephone No. 1-2 7444
WRITTEN OPINION OF THE INTERNATIONAL SEARCHING AUTHORITY

Box No. I Basis of this opinion
1. With regard to the language, this opinion has been established on the basis of: the international application in the language in which it was filed
, which is the language of a translation furnished for the a translation of the international application into purposes of international search (Rules 12.3(a) and 23.1l(b)).
2. With regard to any nucleotide and/or amino acid sequence disclosed in the international application and necessary to the claimed invention, this opinion has been established on the basis of: a. type of material Q a sequence listing table(s) related to the sequence listing
D
b.
format of material
D D
on paper in electronic form
c.time of filing/furnishing
D D
D
contained in the international application as filed. together with the international application in electronic form. filed furnished subsequently to this Authority for the purposes of search.
3.
In addition, in the case that more than one version or copy of a sequence listing and/or table(s) relating thereto has been filed or furnished, the required statements that the information in the subsequent or additional copies is identical to that in the application as filed or does not go beyond the application as filed, as appropriate, were furnished.
4. Additional comments:
--- '-- ~'---~Form PCT/ISA/237(Box No. I) (April 2005)
WRITTEN OPINION OF THE , INTERNATIONAL SEARCHING AUTHORITY
Box No. V Reasoned statement under Rule 43 bis.1(a)(i) with regard to novelty, inventive step or industrial applicability; citations and explanations supporting such statement 1. Statement Novelty (N) Claims 1-19,22-23 Claims 20 and 21 Claims 1-19 Claims 20-23 Claims 1-23 Claims NONE YES NO. YES NO YES NO
Inventive step (IS)
Industrial applicability (IA)
2. Citations and explanations: Please See Continuation Sheet
Form PCT/ISA/237 (Box No. V) (April 2005)

Supplemental Box In case the space in any of the preceding boxes Is not sufficient.
V. 2. Citations and Explanations: Claims 1-23 meet the criteria set out in PCT Article 33(4), and thus have industrial applicability because the subject matter claimed can be made or used in industry. Claims 20-21 lack novelty under PCT Article 33(2) as being anticipated by Dufaux (US 6,711,857). In regard to claim 20, Dufaux provides for means for acquiring a sequence of images including and object (col.1 line 62-col.2 line 1, note fro selecting a representative image from a video for applying face detection to a video); means for determining a robust perceptual color of the object (col.1 lines 54-60, note skin-color detection) using at least two of a pixel level analysis:(col.11 lines 24-34, note skin colored pixels is used to determine the most interesting shot), a frame level analysis ( col.1 line 67col.2 line 1, also col. 11 lines 31-34, note motion activity) and a sequence level analysis( col.1 line 67-col.2 line 1). With regard to claim 21, Dufaux provides for a means for training a statistical classification function to identify a robust perceptual color from an observer color (col.1 lines 54-61, note learning based system). Claim 22 lacks an inventive step under PCT Article 33(3) as being obvious over Dufaux (US 6,711,857) In view of Chasen (6,628,829). As to claim 22, Dufaux provides for compensatefor lighting conditions, then histogram equalization is performed to compensate for differences in camera input gains and to improve contrast (col.5 lines 35-40). Dufaux does not provide for color drift. Chasen provides for color drift (col. 8 lines 10-20) the prior art of Dufaux and Chasen are combinable because they are from the same field of endeavor (color correction). It would have been obvious to a person of ordinary skill in the art to incorporate the teaching of Chasen (col.8 lines 10-20), with the system and method of Dufaux, because the invention of Chasen relates to automated analysis of a digital image of the target color to produce a closest match from a known color database, see the background of the Invention). Claim 23 lacks an inventive step under PCT Article 33(3) as being obvious over Dufaux (US 6,711,587) in view of Hunk (US 5,912,980), in regard to claim 23, Dufaux provides for means for acquiring a sequence of images including and object (col.1 line 62-col.2 line 1, note fro selecting a representative image from a video for applying face detection to a video); means for determining a robust perceptual color of the object (col.1 lines 54-60, note skin-color detection) using at least two of a pixel level analysis (col. 11 lines 24-34, note skin colored pixels is used to determine the most interesting shot), a frame level analysis ( col.1 line 67-col.2 line 1,also col.11 lines 31-34, note motion activity) and a sequence level analysis( col.1 line 67-col.2 line 1). Dufaux does not provide for means for determining the distance between a set of observed color data generated using an object of unknown robust perceptual color and a set of color data generated using an object of know robust perceptual color. Hunke provides for means for determining the distance (col.13 lines 32-36, note Form PCT/ISA/237 (Supplemental Box) (April 2005)
Supplemental Box In case the space in any of the preceding boxes is not sufficient. difference measurement and the formula) between a set of observed color data generated using an object of unknown. robust perceptual color (col.13 lines 22-25, note to track a green) and a set of color data generated using an object of know robust perceptual color (col.13 line 1, note ITCC which refer to Individual target color classifier). It would have been obvious to a person of ordinary skill in the art at time the invention was made to incorporate the teaching of Hunk with the system and method of Dufux, because the invention of Hunk relates to the image processing in general and particularly to the processing the Images by methods designed to detect, locate and track distinctive target objects in the image, see field of the invention. Claims 1-19 meet the criteria set out in PCT Article 33(2)-(3), because the prior art does not teach or fairly suggest, for the sequence level evaluation Including analysis of a second plurality of pixels representative of the object, the second plurality of pixels being derive from a plurality of images including the object under a variety of observation conditions. As cited in claim 1. further the prior art of Dufaux failed to teach or suggest for using the at least a frame level analysis and a sequence level analysis, the image processor including frame level logic, sequence level logic, and a drift matrix storage configured to store a color drift matrix for use in the frame level analysis. As cited in claim 14.
Form PCT/ISA/237 (Supplemental Box) (April 2005)
(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT)
(19) World Intellectual Property Organization
International Bureau
(43) International Publication Date
MU111111111 M11111 111II 1111 I M11 1111

PCT
Ili li lIII
(10) International Publication Number
1 December 2005 (01.12.2005)

(51) International Patent Classification: GO6K 9/00 (2006.01) GO6K 9/68 (2006.01) GO6K 9/70 (2006.01) (21) International Application Number: PCT/US2005/016961 (22) International Filing Date: (25) Filing Language: (26) Publication Language:, (30) Priority Data: 60/571,036 13 May 2005 (13.05.2005) English
WO 2005/114557 A3
(81) Designated States (unless otherwise indicated,for every kind of national protection available): AE, AG, AL,, AM, AT, AU, AZ, BA, BB, BG, BR, BW, BY, BZ, CA, CH, CN, CO, CR, CU, CZ, DE, DK, DM, DZ, EC, EE, EG, ES, Fl, GB, GD, GE, GH, GM, HR, HU, ID, IL, IN, IS, JP, KE, KG, KM, KP, KR,.KZ, LC, LK, LR, LS, Ul, LU, LV, MA, MD, MG, MK, MN, MW, MX, MZ, NA, NG, NI, NO, NZ, OM, PG, PH, PL, PT, RO, RU, SC, SD, SE, SG, SK, SL, SM, SY, TJ, TM, TN, TR, TT, TZ, UA, UG, US, UZ, VC, VN, YU, ZA, ZM, ZW. (84) Designated States (unless otherwise indicated,for every kind of regionalprotection available): ARTPO (BW, GH, GM, KE, LS, MW, MZ, NA, SD, SL, SZ, TZ, UG, ZM, ZW), Eurasian (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), European (AT, BE, BG, CH, CY, CZ, DE, DK, EE, ES, FI, FR, GB, GR, HU, IE, IS, IT, LT, LU, MC, NL, PL, PT, RO, SE, SI, SK, TR), OAPI (BF, BJ, CF, CG, CI, CM, GA, GN, GQ, GW, ML, MR, NE, SN, TD,"'TG). Published: Swith internationalsearch report (88) Date of publication of the international search report: 18 January 2007 For two-letter codes and other abbreviations,refer to the "Guidance Notes on Codes and Abbreviations"appearing at the beginning of each regular issue of the PCT Gazette.
13 May 2004 (13.05.2004)
US
(71) Applicant (for all designated States except US): PROXIMEX [US/US]; 20691 Reid Lane, Saratoga, CA 95070 (US). (72) Inventor; and (75) Inventor/Applicant,(for US only): CHANG, Edward, Y. [US/US]; 816 Dorado Drive, Santa Barbara, CA 93111 (US). (74) Agents: DURANT, Stephen, C. et al.; Morrison & Foerster LLP, 425 Market Street, San Francisco, CA 941052482 (US).
I
In
(54) Title: MIlLTIMODAL HIGH-DIMENSIONAL DATA FUSION FOR CLASSIFICATION AND IDENTIFICATION
0
(57) Abstract: A method is provided for evaluating identity of an object, the method including: converting feature information representing the object to a plurality of mathematically defined conmponents; grouping the components into multiple modalities; producing respective first prediction information for each respective modality wherein the respective prediction information for each respective modality is based upon respective components grouped into that respective modality; and producing second prediction information based upon the respective first prediction information produced for the multiple respective modalities.

A. CLASSIFICATION OF SUBJECT MATTER IPC: GO6K 9168( 2006.01),9/70( 2006.01) G06K 9/00(2006.01)
382/115,227 USPC: According to International Patent Classification (IPC) or to both national classification and IPC
B.
FIELDS SEARCHED
Minimum documentation searched (classification system followed by classification symbols) U.S.: 382/115,227 Documentation searched other than minimum documentation to the extent that such documents are included in the fields searched
Electronic data base consulted during the international search (name of data base and, where practicable, search terms used) IEEE C. X --Y A P A DOCUMENTS CONSIDERED TO BE RELEVANT Citation of document, with indication, where appropriate, of the relevant passages TAX et al. Combining multiple classifiers by averaging or by multiplying. Pattern Recognition Society. 2000. Pages 1475-1485. Relevant to claim No.. 1,2,12,13,20,25,26,34 ----.-----6, 11, 17,:21, 30-32 1-37 1-37 ... 1-37
Category *
US 2003/0169908 Al (Kim et al) 11 Sept 2003 (11,09,2003), pages 1-10 US 2004/0136574 Al (Kozakaya et al) 15 Jul 2004 (15,07,2004), pages 1-10 US 6,944,319 (Huang et al) 13 Sept 2005 (13,09,2005), col. 9-14
SFurther documents are listed in the continuation

* "A" Special categories of cited documents:
of Box C.
[]
"T' "X"

later document piublished after the international filing date or priority date and not in conflict with theapplication but cited to understand the principle or theory underlying the invenition document of particular relevance; the claimed invention cannot be considered novel or cannot be considered to involve an inventive step when the document is taken alone document of particular relevance; the claimed invention cannot be
document defining the general state of the art which is not considered to be of particular relevance earlier application or patent published on or after the international filing date document which may throw doubts on priority claim(s) or which is cited to establish the publication date of another citation or other special reason (as
"E" "L"
"Y"
specified)
"O" document referring to anoral disclosure, use, exhibition or other means
considered to involve an inventive step when the document is

combined with one or more other such documents, such combination being obvious to a personskilled in the art
"P"
document published prior to the international filing date but later than the
priority date claimed
"&"
document member of the same patent family
Date of the actual completion of the international search 09 August 2006 (09.08.2006) Name and mailing address of the ISA/US
Mail Stop PCT, Attn: ISAfUS
Date of mail
ch report
Authorized officer
Jingge Wu
P.O. Box 1450
Alexandria, Virginia 22313-1450

Facsimile No. (571) 273-3201
Telephone No. 703-306-0377
Form PCT/ISA/210 (second sheet) (April 2005)

From the INTERNATIONAL SEARCHING AUTHORITY AUTHORNTY SEARC.I INTERN To: STEPHEN C. DURANT MORRISON & FOERSTER LLP 425 MARKET STREET SAN FRANCISCO, CA 94105-2482
PCT,
(PCT Rule 43bis.1)
Date of mailing
d/month/year)
>
1 0 DT 20I 17OC L
Priority date (day/month/year) 13 May 2004 (13.05.2004)
Applicant's or agent's file reference 577832000140 International application No.
FOR FURTHER ACTION See paragraph 2 below International filing date (day/month/year)
13 May 2005 (13.05.2005) PCT/USO5/16961 International Patent Classification (IPC) or both national 'classification and IPC G06K 9/68( 2006.01),9/70(2006.01) G06K 9/00( 2006.01) IPC: 382/115,227 USPC: Applicant PROXIMEX ,
1. This opinion contains indications relating to the following items: Box No. I Box No. II Basis of the opinion Priority Non-establishment of opinion with regard to n ovelty, inventive step and industrial applicability Lack of unity of invention Reasoned statement under Rule 43bis.1(a)(i) w'ith regard to novelty, inventive step or industrial applicability; citations and explanations suppor ting siuch statement Certain documents cited Certain defects in the international application Certain observations on the international appli cation
SBox No. III SBox No. IV

Box No. V Box No. VI
D Z
D Box No. VII

Box No. VIII
2. FURTHER ACTION If a demand for international preliminary examination is made, this opinion will be considered to be a written opinion of the International Preliminary Examining Authority ("IPEA") except that this does not apply where the applicant chooses an Authority other than this one to be the IPEA and the chosen IPEA has notified the International Bureau under Rule 66. 1bis(b) that written opinions of this International Searching Authority will not be so considered. If this opinion is, as provided above, considered to be a written opinion of the IPEA, the applicant is invited' to submit to the IPEA a written reply together, where appropriate, with amendments, before the expiration of 3 months from the date of mailing of Form PCT/ISA/220 or before the expiration of 22 months from the priority date, whichever expires later. For further options, see Form PCT/ISA/220. 3. For further details, see notes to Form PCT/ISA/220. Name and mailing address of the ISA] US Mail Stop PCT, Attn: ISA/US Commissioner for Patents P.O. Box 1450 Alexandria, Virginia 22313-1450 Facsimile No. (571) 273-3201 Form PCT/ISA/237 (cover sheet) (April 2005) Date of completion of this opinion 14 August 2006 (14.08.2006) Authorized office. i/ Jingge Wu
Telephone No. 703-306-0377 .

INTERNATIONAL SEARCHING AUTHORITY Box No. I Basis of this opinion
International application No. PCTI/USO5/16961
1. With regard to the language, this opinion has been established on the basis of: Sthe international application in the language in which it was filed
OI
a translation of the international application into international search (Rules 12.3(a) and 23.1(b)).
which is the language of a translation furnished for the purposes of
2. With regard to any nucleotide and/or amino acid sequence disclosed in the international application and necessary to the claimed invention, this opinion has been established on the basis of: a. type of material
a sequence listing table(s) related to the sequence listing
j
b.
format of material
on paper in electronic form
D
L
c.time of filing/furnishing contained in the international application as filed. filed together with the international application in electronic form. furnished subsequently to this Authority for the purposes of search.
D
L
3. Q
_ _ __ _ ______ Form PCT/ISA/237(Box No. I) (April 2005)
application ' No. WRITTEN OPINION OF THE Caims PeaseSee CoInternational PCT/US05/16961 INTERNATIONAL SEARCHING AUTHORITY Box No. V Reasoned statement under Rule 43 bis.1(a)(i) with regard to novelty, inventive step or industrial . applicability; citations and explanations supporting such statement 1. Statement Novelty (N) Claims Please See Continuation Sheet Claims Please See Continuation Sheet Claims Please See Continuation Sheet, Claims Please See Continuation Sheet Claims Please See Continuation Sheet Claims Please See Continuation Sheet YES NO YES NO YES NO
Inventive step (IS)
2. Citations and explanations: Please See Continuation Sheet
Form PCT/ISA/237 (Box No. V) (April 2005)
International application No. INTERNATIONAL SEARCHING AUTHORITY Box No. VIII Certain observations on the international application PCT/US05/16961
The following observations on the clarity of the claims, description, and drawings or on the questions whether the claims are fully supported by the description, are made: The drawings are objected to under PCT Rule 66.2(a)(iii) as containing the following defect(s) in the form or content thereof: The drawings are objected to because they are of insufficient quality for publication. Claims 14, 22 are objected to under PCT Rule 66.2(a)(v) as lacking clarity under PCT Article 6 because claims 14 and 22 are indefinite for the following reason(s): Referring to claim 14, the phrase "the returned previously stored modality vectors" in line 8 lacks antecedent basis. It appears that the applicant intended the phrase to read "the located previously stored modality vectors." Appropriate correction is required. Referring to claim 22, the phrase "means for producing second prediction information" is indefinite because there is no recitation of producing "first prediction information".
Form ?.tfISA/2iI3
"^
(BIOX NO. yInI) (ApriI
LU0)
WRITTEN OPINION OF THE INTERNATIONAL SEARCHING AUTHORITY Supplemental Box In case the space in any of the preceding boxes is not sufficient.
V.1. The The The The The The
Reasoned opinion as opinion as opinion as opinion as opinion as opinion as
Statements: to Novelty was positive (Yes)with respect to claims 3-11, 14-19, 21-24, 27-33, 35-37 to Novelty was negative (No) with respect to claims 1, 2, 12, 13, 20, 25, 26, 34 to Inventive Step was positive (Yes)with respect to claims 3-5, 7-10, 14-16, 18-19, 22-24, 27-29, 33, 36-37 to Inventive Step was negative(NO) with respect to claims 1, 2, 6, 11, 12, 13, 17, 20, 21, 25, 26, 30-32, 34, 35 to Industrial Applicability was positive (YES) with respect to claims 1-37 to Industrial Applicability was negative(NO) with respect to claims NONE I
V. 2. Citations and Explanations: Claims 1, 2, 12, 13, 20, 25, 26, 34 lack novelty under PCT Article 33(2) as being anticipated by the article entitled "Combining multiple classifiers by averaging or by multiplying?" by Tax et al. ("Tax"). Referring to claim 1, Tax discloses a method of evaluating identity of an object comprising: converting feature information representing the object to a plurality of mathematically defined components (feature sets) ! [pages 1479-1480, section 5.1]; grouping the components into multiple modalities [page 1480];/ producing respective first prediction information for each respective modality wherein the respective prediction information for each respective modality is based upon respective components grouped into that respective modality (page 1481, section 5.2); and producing second prediction information based upon the respective first prediction information produced for the multiple respective modalities (pages 1476-1479, sections 2-4 and pages 1481-1482, section 5.2. Note that the product combination of the classifiers produces second prediction information). Referring to claim 2, Tax further discloses that corrverting involves mapping the feature information to components in a mathematically different space from the original feature space (pages 1480-1481). Referring to claim 12, Tax further discloses that producing respective first prediction information involves producing respective first classification prediction information for each respective modality (page 1481, section 5.2). .Referring to claim 13, Tax further discloses that that producing respective first prediction information involves producing respective first classification prediction information for each respective modality (page 1481, section 5.2), and producing second prediction information involves producing second classification prediction information based upon the respective first classification prediction information (pages 1476-1479, sections 2-4 and pages 1481-1482, section 5.2). Referring to claim 20, Tax discloses a system for evaluating identity of an object comprising: means for converting feature information representing the object to a plurality of mathematically defined components (feature sets) [pages 1479-1480, section 5.1] means for grouping the components into multiple modalities [page 1480]; multiple first classifiers, each associated with modality, each producing respective first classification prediction information for each respective modality, based upon respective components grouped into that respective modality (page 1481, section 5:2); and a second classifier producing second classification prediction information based upon a non-linear combination of the respective first classification prediction information produced for the multiple respective modalities (pages 1476-1479, sections 2-4 and pages 1481-1482, section 5.2. Note that the product-combination of the classifiers produces second prediction information based on a non-linear combination ofrescleetive first classification prediction information) Form PCT/ISA/237 (Supplemental Box) (April 2005)
WRITTEN OPINION OF THE INTERNATIONAL SEARCHING AUTHORITY Supplemental Box In case the space in any of the preceding boxes is not sufficient.
International application No. PCT/USO5/16961
Referring to claim 25, see the discussion in view of Tax of at least claim 1 above. Referring to claim 26, see the discussion in view of Tax of at least claim 2 above. Referring to claim 34, see the discussion of at least claim 20 above. Claims 6, I11,17, 21, 30-32, 35 lack an inventive step under PCT Article 33(3) as being obvious over the article entitled "Combining multiple classifiers by averaging or by multiplying?" by Tax et al. ("Tax") in view of applicant's admission of known prior art ("Admission"). Referring to claim 6, Tax does not explicitly disclose that grouping the components involves ensuring that the total dimensions within individual modalities is below a prescribed threshold based upon the curse of dimensionality. However, this feature was exceedingly well known in the art. For example, Admission discloses grouping of components that involves ensuring that the total dimensions within individual modalities is below a prescribed threshold (20 to 30) based upon the curse of dimensionality (page 13, paragraph 50). Tax and Admission are combinable because they are both concerned with multi-level classifiers. At the time of the invention, it would have been obvious to a person of ordinary skill in the art to modify Tax in view of Admission. The suggestion/motivation for doing so would have been to reduce the effects due to the curse of dimensionality (Admission, page 13, paragraph 50). Therefore, it would have been obvious to combine Tax with Admission to obtain the invention as specified in claim 6. Referring to claim 11, Tax further discloses that the grouping involves grouping the components into multiple modalities so as to minimize correlation among modalities (page 1481, last paragraph on the left column. Note that modalities comprise independent data sets). Tax does not explicitly disclose that grouping the components involves ensuring that the total dimensions within individual modalities is below a prescribed threshold based upon the curse of dimensionality. However, this feature was exceedingly well known in the art. For example, Admission discloses grouping of components that involves ensuring that the total dimensions within individual modalities is below a prescribed threshold (20 to 30) based upon the curse of dimensionality (page 13, paragraph 50). Tax and Admission are combinable because they are both concerned with multi-level classifiers. At the time of the invention, it would have been obvious to a person of ordinary skill in the art to modify Tax in view of Admission. The suggestion/motivation for doing so would have been to reduce the effects due to the curse of dimensionality (page 13, paragraph 50). Therefore, it would have been obvious to combine Tax with Admission to obtain the invention as specified in claim 11. Referring to claim 17, Tax discloses a method of evaluating identity of an object comprising: converting feature information representing the object to a plurality of mathematically defined components (feature sets) [pages 1479-1480, section 5.1]; grouping the components into multiple modalities so as to minimize correlation among modalities [page 1480-1481]; producing respective first prediction information for each respective modality wherein the respective prediction information for each respective modality is based upon respective components grouped into that respective modality (page 1481, section 5:2); and producing second classification prediction information based upon a non-linear combination of the respective first classification prediction information produced for the multiple respective modalities (pages 1476-1479, sections 2-4 and pages 14811482, section 5.2. Note that the product combination of the classifiers produces second prediction information that is a non-linear combination of the respective first classification prediction information). Tax does not explicitly disclose that grouping the components involves limiting the dimensions within individual modalities is below a prescribed threshold based upon the curse of dimensionality. However, this feature was exceedingly well lkniown in the art. For example, Admission discloses grouping of components that involves limiting dimensions within individual modalities is below a , prescribed threshold (20 to 30) based upon the curse of dimensionality (page 13, paragraph 50). Tax and Admission are combinable because they are both concerned with multi-level classifiers. At the time of the invention, it would have been obvious to a person of ordinary skill in the art to modify Tax in view of Admission. The suggestion/motivation for doing so would have been to reduce the effects due to the curse of dimensionality (page 13, paragraph 50). Therefore, it would have been obvious to combine Tax with Admission to obtain the invention as specified in claim 17. Referring to claim 21, see the discussion of at least claim 6 above. Referring to claim 31, see the discussion of at least claim 11 above. Referring to claim 32, see the discussion of at least claim 17 above. Referring to claim 35, see the discussion of at least claim 17 above.
(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION.TREATY (PCT) (19) World Intellectual Property Organization
International
Bureau
"I
ll
II
ll
Illl l
II Ill1ll 1l1 ll l M l
Il
(43) International Publication Date
30 March 2006 (30.03.2006)

(51) International Patent Classification: HO04N 7/18 (2006.01).. GO6K 9/00 (2006.01) 110H4B 17/00 (2006.01) GO6K 9/64 (2006.01) HO4N 9/47 (2006.01)
PCT
;WO 2006/034135 A3
(74) Agents: DURANT, Stephen, C. ct a].; Morrison & Focrster LLP, 425 Market Street, San Francisco, Ca 94105-2482 (US)., (81) Designated States (unless otherwise indicated,for every kind of nationalprotection available): AE, AG, AL, AM, AT, AU, AZ, BA, BB, BG, BR, BW, BY, BZ, CA, CH, CN, CO, CR, CU, CZ, DE, DK, DM, DZ, EC, EE, EG, ES, FI, GB, GD, GE, GH, GM, HR, HU, ID, IL, IN, IS, JP, KE, KG, KM, KP,;KR, KZ, LC, LK, LR, LS, LT, LU, LV, LY, MA, MD, MG, MK, MN, MW, MX, MZ, NA, NG, NI, NO, NZ, OM, PG, PH, PL, PT, RO, RU, SC, SD, SE, SG, SK, SL, SM, SY, TJ, TM, TN, TR, TT, TZ, UA, UG, US, UZ, VC, VN, YIJ, ZA, ZM, ZW.. (84) Designated States (unless otherwise indicated,for every kind of regional protection available): AR1PO (BW, GH, GM, KE, 1LS, MW, MZ, NA, SD;. SI,, SZ, TZ, 110, ZM, ZW), Eurasian (AM, AZ,-BY, KG, KZ, MD, RU, TJ, TM), European (AT, BE, BG, CH, CY, CZ, DE, DK, EE, ES, Fl; FR, GB, GR, HU, IE, IS, IT, LT, LU, LV, MC, NL, PL, PT, RO, SE, SI, SK, TR), QAPI (BF, BJ, CF, CG, CI, CM, GA,. GN, GQ, GW, ML, MR,. NE, SN, TD, TG). Published: with internationalsearch report (88) Date of publication of the International search report: 13 July 2006 [Continuedon next page]
(21) International Application NumberC PCT/US2005/033378 (22) International Filing Date: 19 September 2005 (19.09.2005) (25) Filing Language: (26) Publication Language: (30) Priority Data: 60/610,998 English English
17 September 2004 (17.09.2004)
US
(71) Applicant (forall designated States except US): PROXIMEX [US/US]; 20691 Reid Lane, Saratoga, CA 95070 (US). (72) Inventors; and (75) Inventors/Applicants (for US only): CHENG, Ken, Prayoon [US/US]; 20691 Reid Lane, Saratoga, CA 95070 (US). CHANG, Edward, Y. [IS/US]; 816 Dorado Drive, Santa Barbara, CA 93111 (US). WANG, Yuan, Fang [US/US]; 5849 Via Fiori Lane, Goleta, CA 93117 (US).
SURVEIL(54) Title: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND ,. . LANCE SYSTEM
to M
()
M m mmm O
Configure Maps (Environment Admin)
m
mm m
wl,r-.
dn '" opa '

"M
.fr~mt
.. .. .
0
(57) Abstract: Asurveillance system is provided that includes at least one sensor disposed in a security area of a surveillance region to sense an occurrence of a potential security breach event; a plurality of cameras is disposed in the surveillance region; at least one camera of the plurality has a view of the security area and can be configured to automatically gather biometric information concerning at least one subject person in the vicinity of the security area in response to the sensing of a potential security breach event; one or more other of the plurality of cameras can be configured to search for the at least one subject person; a processing of system is programmed to produce a subject dossier corresponding to the at least one subject person to match biometric information or more of the other cameras with corresponding biometric information in the subject dossier. one or more persons captured by one
O 2006/034135 A3
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII IIIll11111IIIIIIIIIIIIIIIIIIIIIIIII III
For two-letter codes and other abbreviations,refer to the "Guidance Notes on Codes andAbbreviations" appearingat the beginning of each regular issue of the PCT Gazette.

A. IPC:
CLASSIFICATION OF SUBJECT MATFTER GO6K 9/00( 2006.01),9/64('2006.01I);HO4N 9/47( 2006.01),7/18( 2006.0 1);HO4B 17/00( 2006.01)
382/115-118,124;348/143-161;455/67.12 ' USPC: According to International Patent Classification (IPC) or to both national classification and IPC
B.
FIELDS SEARCHED .
Minimum documentation searched (classification system followed by classification symbols) U.S.: 3821115-118,124;348/143-161;455/67.12 Documentation searched other than minimum documentation to the extent that such documents are included in the fields searched
C.
DOCUMENTS CONSIDERED TO BE RELEVANT Citation of document, with indication, where appropriate, of the relevant passages Category * US 2002/0190119 Al (HUFFMAN) 19 December 2002 (19.12.2002), entire document X X X US 2004/0081338 Al (TAKENAKA) 29 April 2004 (29.4.2004), entire document US 6,697,103 BI (FERNANDEZ et al) 24 February 2004 (24.02.2004), column 4 lines 2367, column 6 lines 1-30 Figures 1 and 3-4.
Relevant to claim No. 1-20 1-20 I,17, 20
I,
.mE
- Further documents are listed in the continuation of Box C.

*
[E
"T"

filing dateor priority later document published after the intemrnational the date and notin conflict with the application but cited to understand principle or theory underlying the invention document ofparticular relevance; the claimed invention cannot be considered novel or cannot be considered to involve an inventive step when the document is taken alone document of particular relevance; the claimed irivention cannot be combined considered to involve an inventive step when the document is
with one or more other such documents, such combination being
Special categories of cited documents: document defining thegeneral stateof the art which is not considered to be of particular relevance earlier application or patent published on or after the international filing date document which may throw doubts on priority claim(s) or which is'cited to establish the publication dateof another citation or other special reason (as specified)
"A"
"X"
"E" "L"
"Y'
"0"
"P"
document referring to anoral disclosure, use,exhibition or other means

later thanthe date document published prior to the intemrnational filingbtit priority date claimed
"&"
obvious to aperson skilled in the art

document member of the samepatent family
Date of the actual completion of the international search

13 March 2006 (13.03.2006)
Date of mailing of the nt Authorized officer Jingge Wu
tlp
port
Name and mailing address of the ISA/US Mail Stop PCT, Attn: ISA/US Commissioner for Patents Alexandria, Virginia 22313-1450 Facsimile No. (571) 273-3201 \ ' I Form PCT/ISA/210 (second sheet) (April 2005)
P.O. Box 1450
TlpoeN.512276 Telephone No. 571-272-7361

, ' -
__

From the
INTERNATIONAL SEARCHING AUTHORITY

To: STEPHEN C. DURANT MORRISON& FOERSTER LLP 425 MARKET STREET SAN FRANCISCO, CA 94105-2482
"C,"
(PCT Rule 43bis.1) Date of mailing day month/ ear FOR FURTHER ACTION ' See paragraph 2 below International filing date (day/month/year)
2 6.APR 2006.
Applicant's or agent's file reference 577832000240 International application No.
Priority date (day/month/year) 17 September 2004 (17.09.2004

)
PCT/US05/33378 19 September 2005 (19.09.2005) International Patent Classification (IPC) or both national classification and IPC IPC: Please See Continuation Sheet USPC: 382/115-118,124;348/143-161;455/67.12 Applicant PROXIMEX 1. This opinion contains indications relating to the following items: ,
Z
j
Box No. I
Basis of the opinion Priority Non-establishment of-opinion with regard to novelty, inventive step and industrial applicability Lack of unity of invention Reasoned statement under Rule 43bis.I (a)(i) with regard to novelty, inventive step or industrial applicability; citations and explanations supporting such statement Certain documents cited Certain defects in the international application Certain observations on the international application
D Box No. II
Box No. III . Box No. IV Box No. V
SBox No. VI
Box No. VII Box No. VIII
2. FURTHER ACTION If a demand, for international preliminary examination is made, this opinion will be considered to be a written opinion of the International Preliminary Examining Authority ("IPEA") except that this does not apply where the applicant chooses an Authority other than this one to be the IPEA and the chosen IPEA has notified the International Bureau under Rule 66.1bis(b) that written opinions of this International Searching Authority will not be so considered. If this opinion is, as provided above, considered to be a written opinion of the IPEA, the applicant is invited to submit to the IPEA a written reply together, where appropriate, with amendments,, before the expiration of 3 months from the date of mailing of Form PCT/ISA/220 or before the expiration of 22 months from the priority date, whichever expires later. For further options, see Form PCT/ISA/220. 3. For further details, see notes to Form PCT/ISA/220. Name and mailing address of the ISA/ US Mail Stop PCT, Attn: ISA/US Commissioner for Patents P.O. Box 1450 Alexandria, Virginia 22313-1450 Facsimile No. (571) 273-3201 Form PCT/ISA/237 (cover sheet) (April 2005) Date of completion of this opinion 13 March 2006 (13.03.2006) Authorized officer Jingge Wu Telephone No. 571-272-7361
Box No. I Basis of this opinion
1. With regard to the language, this opinion has beeh established on the basis of: Sthe international application in. the language in which it was filed
Li-
a translation of the international application into ____, which is the language of a translation furnished for the purposes of international search (Rules 12.3(a) and 23.1(b)).
2. With regard to any nucleotide and/or amino acid sequence disclosed in the international application and necessary to the claimed invention, this opinion has been established on the basis of: a. type of material
a sequence listing
O- table(s) related to the sequence listing

b. format of material '
on paper
in electronic form
c.time of filing/furnishing
O- contained in the international application as filed. O filed together with the international application in electronic form.
Li
3.
furnished subsequently to this Authority for the purposes of search.
Form PCT/ISA/237(Box No. I) (April 2005)
Box No. V Reasoned statement under Rule 43 bis.1(a)(i) with regard to novelty, Inventive step or industrial applicability; citations and explanations supporting such statement 1. Statement Novelty (N) Claims NONE Claims 1-20 Inventive step (IS) Claims NONE Claims 1-20 Industrial applicability (IA) Claims 1-20 Claims NONE YES NO YES NO YES NO
2. Citations and explanations: Claims 1-20 lack the novelty under PCT Article 33(2) as being anticipated by Huffman (US 2002/0190119 Al). Huffman discloses claims 1-20. For example, Huffman discloses a surveillance method comprising, using at least one event sensor disposed in a security area of a surveillance region to sense an occurrence of potential security breach event (page 2 paragraphs 17, 18 and 21); using at least one camera with a view of the security area in which the event is sensed to gather biometric information concerning at least one person in the vicinity of the security area at about the time of the sensing of the event (Figures 2 and 3); producing a subject dossier corresponding to the at least one person (page 2, paragraph 22); and matching biometric information of one or more persons captured by one or more other cameras in the vicinity of the at leas tone camera with corresponding biometric information in the subject dossier (page 3 paragraphs 32 and 33).
Form PCT/1SA/237 (Box No. V) (April 2005)
...
..
...
. .
..
A^ ^

I Supplemental Box In case the space in any of the preceding boxes is not sufficient. '

'
Continuatioi of IPC: G06K 9/00( 2006.01),9/64( 2006.0 1);HO4N 9/47( 2006.01),7/18( 2006.01);HO4B 17/00( 2006.01)
(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT)
(19) World Intellectual Property Organization
11 III1V V
l l
III IIII I
l
III
licIainub llcato
III
International Bura (43) International Publication Date

2 1 eptemer 200
Ml
PCT
(10) International Publication Number WO 2006/098766 A31)
2O 1.097.22006)
(51) International Patent Classification: GO6N 3/08 (2006.01) GO6N 3/00 (2006.01) GO6N 3/04 (2006.01) GO6N 3/063 (2006.01) (21) International Application Number: PCT/US2005/033750 (22) International Filing Da 19 September 2005 (19.09.2005) (25) Filing Language: (26) Publication Language: (30) Priority Data: 60/610,916
S
(81) Designated States (unless otherwise indicated,for every kind of nationalprotection available): AE, AG, AL, AM, AT, AU, AZ, BA, BB, BG, BR, BW, BY, BZ, CA, CH, CN, CO, CR, CU, CZ, DE, DK, DM, DZ, EC, EE, EG, ES, FI, GB, GD, GE, GH, GM, HR, HU, ID, IL, IN, IS, JP, KE, KG, KM, KP, KR, KZ, LC, LK, LR, LS, LT, LU, LV, LY, MA, MD, MG, MK, MN, MW, MX, MZ, NA, NG, NI, NO, NZ, OM, PG, PH, PL, PT, RO, RU, SC, SD, SE, SG, SK, SL, SM, SY, TJ, TM, TN, TR, TT, TZ, UA, UG, US, UZ, VC, VN, YU, ZA, ZM, ZW. (84) Designated States (unless otherwise indicated,for every kind of regional jprotection available): ARTPO (BW, GH, GM, KE, LS, MW, MZ, NA, SD, SL, SZ, TZ, UG, ZM, ZW), Eurasian (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), European (AT, BE, BG, CH, CY, CZ, DE, DK, EE, ES, Fl, FR, GB, GR, HU, IE, IS, IT, LT, LU, LV, MC, NL, PL, PT, RO, SE,,SI, SK,TR), OAPI (BF, BJ, CF, CG, CI, CM, GA, GN, GQ, GW, ML, MR, NE, SN, TD, TG). Published: with internationalsearch report (88) Date of publication of the international search report: 26 July 2007 For two-letter codes and other abbreviations,refer to.the "Guidance Notes on Codes and Abbreviations"appearing at the beginning of each regular issue of the PCT Gazette.
English "English
17 September 2004 (17.09.2004)
(71) Applicant (for all designated States except US): PROXIMEX [US/US]; 20691 Reid Lane, Saratoga, CA 95070 (US). (72). Inventor; and (75) Inventor/Applicant (for US only): WANG, Yuan Fuang [US/US]; 5849 Via Fiori Lane, Golta, CA 93117 (US). (74) Agents: DURANT, Stephen, C. et al.; Morrison & Foerster LLP, 425 Market Street, San Francisco, CA 941052482 (US).
(54) Title: INCREMENTAL DATA FUSION AND DECISION MAKING SYSTEM AND ASSOCIATED METHOD
(57) Abstract: A computer implemented (Figure 7) adaptive ensemble classifier is provided which includes: a plurality classifiers (Figure 1, Ddf); a decision structure that maps respective classifier combinations to respective classification decision results (Figure 5C, 512); and a plurality of respective sets of weights associated with respective classifier combinations.
3THNATMONAL BEARCI[
AL.
IMP'ORfT
ttatitanm appl1oation No. PCD'V805/.33750
CiLASSIOA
IP: USPC:
ON OF SUBJsCT WiA'fI'D 6 G6N /0( 2006.01 ),3/0 3( 2006.01 ),3/08( 2006.01),3%04( 200b. 1) 706/2;706141;706/20;706/26
Accrding to IntDemational Patent Chmi on~t (W2C)
or to bot
natloaai clasLcaiion and IPC
Minimum acgwrontatiois asaehd (alassotion eyatem U.S.: 706125;706/41;706120;7D6I26

Dcaauniorrttidon searched other than UJS latent datubaeo; IRI
followed by classificaton
symbols)
Mmtt wum
docurnutaon to 'Cho extent that =u6 dcumeants are Icluded in the
f1olde searched
c ed)
Elchtso data bas econaulted
duding the inmtional uessac anv of data us amud,
why eprwiouble,
sarclx t
IBS
C,
A
A, A
DOCUM
Category "
A A, A
REtLSVA)Xf''1 ITS CONSWFAUED TO Mar=io of duct mrsnt4 with jcndAcat luu, whore approprilote, US 5.4o-,574 A(GLMR~ eC al.) 26 D~ecember 1995 (2?. 25995) column 2, line 29 to ooltwint 3, fine; aL) 23 oembeT 1997 (23.12.1997)12 US 5,70t398 A(Gl column 2, lino 29to colur a 3, lime 5 US 6,757,668 B1 A(COSBEL ticof.) 29 June 2004 (29?.06.2004), columnt 3, litres 28-43 US 6,248,063 B i &(DARNHILL ctel.) 19 June 20091 (19.06,2001)>-0 colutmn 7. line 39 to column, 9, tins 57 US 5,769,074 A(BARNMEL of a.) 23 Ju~ne 1998'(23.06.1~98) columnt 7, 39 to colurem 9, line 59 LLet al.) 23 October 2001 (23,10.2011) U'S 6,306,087131 A(BAM
BE
of tbarelevant
pssages
Relevant to claim N.
2
2
cc
1-20
1-20
column
3,9 to oolurnn 9,
line
35
El
see platen lniy-azmex. Q lditt ntn a 'be MMtunadouul fiffnj dat e orptcty seen Wr do 8Aaaatoeruia oafcca dSofooti~T Ia onadaid t hadata anid not to wtflhetvk Me~ pplooettou btm shadi to dund i = the A' ioctnen cta~~lbsganiultut of~ u~ wlohtn aom or idmd udnolpl thlfamda Mdybe (Ity to o of iy dooamnt otbo ar wpiah ea "A" garotel d tiuWSthe x" dount 'taliotlom~tlavanoot tboulaitrwt {itrtsNront catutl t V" erlier apliton erpiat4t 5natahed en oratorlis iznt lonal Olhng daft e eonsmdet tanbn be cos werd to Involve an fzwunttw stop dgounise( wilkb uuy lIsow daubo an pdadWt eldttt(o) or WWhr6 Is aoW ed i "L" 4bm clafoind nid otmmt be duuant1~ of utttoulnfoteevuste de; "'Y th lmhoua dace caeolhr dchodan or er spuot moao (a catublivh xpeoiiescadod to InvolVe an Iavasdvo atop When the damuitaniIs aominad wlth oite nr mormQtheruplh 4*auibnum nwh c mbntlou bt io abvloosa too ia omt s~lkd in the nma malng to an otaldaleauro. use, o d~blhn or odiemflleamz "O)" de Fu rtlicr doouztn na tare listed In th coud auuion df ox a.
"
cat*
hIS mc
nit
1111
Intosnclloaal fiingdsic but IOloIU n damunnt pbialled pior to Ows
&".
ofrthe doatizat MlO4Umr
sap
aatIuil
Data of the uiuual aonitpIn of the intemauuzicnl 29 August 2005 (29.08.200t5)1)2f

Natru-w and tnailing address ofithe ACT, PAnntSA Cmsirp Patens CmiSione fCo Alamudia,
search.
Dato of mnelding of iztern~ianal semni report t~le
Y
u
MWS
itho
caro
I'
'
P.O. Box L4$0
Anhn
Kugh
Facsimnile No. (571) 273-3201__________________________ F~onn PCTL/ISA/210 (second sheat) (Apri 2005)
Vicrtnl2313-140O
'Talapnolii No. (0)3830
C7)
I
From the INTERNATIONAL SEARCHING AUTHORITY "
LLP C. DURANT To: FOERSTER STEPHEN &STREET 94105-2482 CA STEPHEN C. MORRISON DURANT MARKET MORRISON & FOERSTER LLP 425 FRANCISCO, 425 MARKET STREET SAN
TO:
PCT
WRITTEN OPINION OF THE INTERNATIONAL SEARCHING AUTHORITY (PCT Rule 43bis.1) Date of mailing,
da /month/ ear
SAN FRANCISCO, CA 94105-2482
Applicant's or agent's file reference 577832000340 577832000340 International application No.
FOR FURTHER ACTION ' U See paragraph 2 below International filing date (day/month/year) Priority date (day/month/year) 17 September 2004 (17.09.2004)
PCT/US05/33750 19 September 2005 (19.09.2005) International Patent Classification (IPC) or both national classification and IPC
IPC: GO6N 3/00(2006.01);GO6N 3/063( 2006.01);GO6N 3/08( 2006.01);GO6N 3/04(2006.01) USPC: 706/25;706/41';706/20;706/26 Applicant PROXIMEX
-:
1. This opinion contains indications relating to the following items: SBox No. I Basis of the opinion Priority Non-establishment of opinion with regard to novelty, inventive step and industrial applicability Lack of unity of invention Reasoned statement under Rule 43bis.1(a)(i) with regard to novelty, inventive step or industrial applicability; citations and explanations supporting such statement
Certain documents cited
SBox No. 11.
D Z
D
Box No. III Box No. IV Box No. V

Box No. VI
Box No. VII
Certain defects in the international application Certain observations on the international application
D Box No. VIII
2. FURTHER ACTION If a demand for international preliminary examination is made, this opinion will be considered to be a written opinion of the International Preliminary Examining Authority ("IPEA") except that this does not apply where the applicant chooses an Authority other than this one to be the IPEA and the chosen IPEA has notified the International Bureau under Rule 66.1 bis(b) that written opinions of this International Searching Authority will not be so considered. If this opinion is, as provided above, considered to be a written opinion of the IPEA, the applicant is invited to submit to the IPEA a written reply together, where appropriate, with amendments, before the expiration of 3 months from the date of mailing of Form PCT/ISA/220 or before the expiration of 22 months from the priority date, whichever expires later. For further options, see Form PCT/ISA/220. 3. For further details, see notes to Form PCT/ISA/220. Name and mailing address of the ISA/ US
Mail Stop PCT, Attn: ISA/US

P.O. Box 1450 Alexandria, Virginia 22313-1450
Facsile No. (571) 273-3201
Form PCT/ISA/237 (cover sheet) (April 2005)
WRITTEN OPINION OF THE INTERNATIONAL SEARCHING AUTHORITY Box No. I Basis of this opinion
1. With regard to the i'nguage, this opinion has been established on the basis of: , the international application in the language in which it was filed [I a translation of the international application into international search (Rules 12.3(a) and 23.1(b)).
_
which is the lan guage of a translation furnished for the purposes of
2. With regard to any nucleotide and/or amino acid sequence disclosed in the international application and necessary to the claimed invention, this opinion has been established on the basis of: a. type of material a sequence listing
Stable(s)
b.
related to the sequence listing
format of material
O ' on paper
D
[I
Q Q" in electronic form c.time.of filing/furnishing contained in the international.application as filed. filed tbgether with the international application in electronic form. furnished subsequently to this Authority for the purposes of search.
3. [-
Form PCT/ISA/237(Box No. I) (April 2005)
International application No. PCTIUS05/33750
Box No. V Reasoned statement under Rule 43 bis.1(a)(i) with regard to novelty, inventive step or industrial applicability; citations and explanations supporting such statement 1. Statement Novelty (N) Claims 1-20
Claims NONE -~~~---
YES NOr
INO
Inventive step (IS)
Claims 1-20 Claims NONE Claims 1-20 Claims NONE
YES NO 7'
YES
NO
NO
2. Citations and explanations: Claims 1-20 meets the criteria set forth in PCT Article 33(2-4) whereby the prior art does not explicitly teach or render obvious applicant's claimed invention. Specifically, a computer implemented adaptive ensemble classifier is provided which includes: a plurality of classifiers; a decision structure that maps respective classifier combinations to respective classification decision results; and a plurality of respective sets of weights associated with respective classifier combinations.
~1 1-~-_ -___-_ r Form rLu11IA/Z3 / (tOX NO. V) (April 2005)
(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION T'REATY (PCT) (19) World Intellectual Property Organization International Bureau (43) International Publication Date
IIl
Iii 11111111) 11111 liIll 111 11 11111 11111 11111 I111111 11111 1111I11111 11111li1111111
3 May 2007 (03.05.2007)

(51) International Patent Classification: 11HO4N 7/18 (2006.01)
PCT
WO 2007/050104 A3
AT, AU, AZ, BA, BB, BG, BR, BW, BY, BZ, CA, CH, CN, CO, CR, CU, CZ, DE, DK, DM, DZ, EC, EE, EG, ES, FI, GB, GD, GE, GH, GM, HR, HU, ID, IL, IN, IS, JP, KE, KG, KM, KN, KP, KR, KZ,"LC, LK, LR, LS, LT, LU, LV, LY, MA, MD, MG, MK, MN, MW, MX, MZ, NA, NG, NI, NO, NZ, OM, PG, PH, PL, PT, RO, RU, SC, SD, SE, SG, SK, SL, SM, SY, TJ, TM, TN, TR, TT, TZ, UA, UG, US, UZ, VC, VN, YU, ZA, ZM, ZW. (84) Designated States (unless otherwise indicated,for every kind of regional protection available): ARIPO (BW, GH, GM, KE, LS, MW, MZ, NA, SD, SL, SZ, TZ, UG, ZM, ZW), Eurasian (AM,AZ, BY, KG, KZ, MD, RIJ, T.1, TM), European (Al', BE, BG, CH, CY, CZ, DE, DK, EE, ES, Fl, FR, GB, GR, HU, IE, IS, IT, LT, LU, LV, MC, NL, PL, PT, RO, SE, SI, SK, TR), OAPI (BF, BJ, CF, CG, CI, CM, GA, GN, GQ, GW, Ml,, MR, NE, SN, TD, TG). Published: with internationalsearch report before the expiration of the time limit for amending the claims and to be republished in the event of receipt of amendments (88) Date of publication of the international search report: 13 December 2007 For two-letter codes and other abbreviations,refer to the "Guidance Notes on Codes and Abbreviations" appearingat the beginning of each regular issue of the PCT Gazette.
(21) International Application Number: PCT/ JS2005/043808 (22) International Filing Dat 4 December 2005 (04.12.2005) (25) Filing Language: (26) Publication Language: (30) Priority Data: 60/633,166 English English US
4 December 2004 (04.12.2004)
(71) Applicant (for all designated States except US): PROXIMEX CORPORATION [IJS/1JS]; 6 Reuslts Way, Cupertino, CA 95014 (US). (72) Inventors: WANG, Yuan-Fang; 5849 Via Fiori Lane, Goleta, CA 93117 (US). CHANG, Edward; 816 Dorado Drive, Santa Barbara, CA 93111 (US). CHENG, Ken, Prayoon; 20691 Reid Lane, Saratoga, CA 95070 (US). (74) Agent: LIN, Bo-in; 13445 Mandoli Drive, Los Altos Hills, CA 94022 (US).
I
C
(81) Designated States (unless otherwise indicated,for every kind of nationalprotection available): AE, AG, AL, AM,
(54) Title: VIDEO SURVEILLANCE USING STATIONARY-DYNAMIC CAMERA ASSEMBLIES
r'
O
O
V /
O N
an ar(57) Abstract: A video surveillance system includes multiple video cameras. The surveillance system is configured with rangement to separate the surveillance functions and assign different surveillance functions to different cameras. A master camera is assigned the surveillance of large area surveillance and tracking of object movement while one or more slave cameras are provided todynamically rotate and adjust focus to obtain clear image of the moving objects as detected by the master camera. Algorithms to adjust the focus-of-attention are disclosed to effectively carry out the tasks by a slave camera under the command of a master camera to obtain images of a moving object with clear feature detections.
International application No. PCT/US 05143808
A.
CLASSIFICATION OF SUBJECT MATTER
IPC(8) - H04N 7/18 (2007.01) USPC - 348/143

According to International Patent Classification (IPC)or to both national classification and IPC B. FIELDS SEARCHED Minimum documentation searched (classification system 'followed by classification symbols)
USPC: 348/143
Documentation searched other than minimum documentation to the extent that such documents are included in the fields searched US: 375/240.01; 348/143, 152, E7.086
PubWEST(PGPB,, USPT, USOC, EPAB, JPAB) Search Terms: video and camera and (surveillance or security), master and slave, processor, master camera, slave camera, multiple degrees of freedom, degrees of freedom, freedom near3 degree; GOOGLE.com/patents: master and slave video cameras with zoom control , C. DOCUMENTS CONSIDERED TO BE RELEVANT Citation of document, with indication, where appropriate, of the relevant passages
US 6,724,421 B1 (GLATT) 20 April 2004 (20.04.2004), [abstract], col. 1, In 10-12, col. 3, in 2-3,
Category*
X
Relevant to claim No.

1 - 33
8-20, Fig. 1, claim 3.
--34
US 6,795,106 B1 (COOPER) 21 September 2004 (21.09.2004), [abstract], col. 7, 11-16
34
I
*
Further documents are listed in the continuation of Box C.
[]
"A" "E" "L" "O" "P"
"T" later document published after the international filing date or priority Special categories of cited documents: date and not in conflict with the application but cited to understand document defining the general state of the art which is not considered the principle or theory underlying the invention particular relevance to be of earlier application or patent but published on or after the international "X" document of particular relevance; the claimed invention cannot be considered novel' or cannot be considered to involve an inventive filing date document is taken alone step when the document which may throw doubts on priority claim(s) or which is cited to establish the publication date of another citation or other "Y", document of particular relevance; the-claimed invention cannot be considered to involve an inventive step when the document is special reason (as specified) combined with one or more other such documents, such combination document referring to an oral disclosure, use, exhibition or other being obvious to a person skilled in the art means document published prior to the intemrnational filing date but later than "&" document member of the same patent family
the priority date claimed
Date of the actual completion of the international search
Date of mailing of the international search report
08 August 2007 (08.08.2007)

Name and mailing address of the ISA/US Mail Stop PCT, Attn: ISA/US, Commissioner for Patents P.O. Box 1450, Alexandria, Virginia 22313-1450 Facsimile No. 571-273-3201
00 CT 2007
Lee W. Young
Authorized officer: 571-272-4300 PCTHepd PCTOSP:671-272-7774
Form PCT/ISA/210 (second sheet) (April 2007)

From the INTERNATIONAL SEARCHING AUTHORITY To: BO-IN LIN 13445 MANDOLI DRIVE LOS ALTOS HILLS, CA 94022
PCT
WRITTEN OPINION OF THE INTERNATIONAL SEARCHING AUTHORITY (PCT Rule 43bis.1)
Date of mailing
(day/mont reference Applicant's or agent's file PROXIMEX0402 International application No. PCT/US 05/43808
ear)
1 0
CT 2007
FOR FURTHER ACTION See paragraph 2 below ' International filing date (day/monthlyear) 04 December 2005 (04.12.2005) Priority date (day/month/year) 04 December 2004 (04.12.2004)
International Patent Classification (PC) or both national classification and IPC . IPC(8) - H04N 7/18 (2007.01) USPC - 348/143 Applicant PROXIMEX CORPORATION
1. This opinion contains indications relating to the following items: SBox No. I Basis of the opinion
Priority
]
[
Box No. II
Box No. III
Non-establislihment of opinion with regard to novelty, inventive step and industrial applicability Lack of unity of invention Reasoned statement under Rule'43bis. i (a)(i) with regard to novelty, inventive step or industrial applicability; citations and explanations supporting such statement
Certain documents cited
SBox No. IV
SBox No. V
Box No. VI
SBox No. VII [ Box No. VIII
Certain defects in the intern ational application

Certain observations on the international application
2. FURTHER ACTION If a demand for international preliminary examination is made, this opinion will be considered to be a written opinion of the International Preliminary Examining Authority ("IPEA") except that this does not apply where the applicant chooses an Authority other than this one to be the IPEA and the chosen IPEA has notified the International Bureau tinder Rule 66. lbis(b) that written opinions of this International Searching Authority will not be so considered.. Ifthis opinion is, as provided above, considered to be a written opinion of the IPEA, the applicant is invited to submit to the IPEA a written reply together, where appropriate, with amendments, before the expiration of 3 months from the date of mailing of Form PCT/ISA/220 orbefore the expiration of 22 months from the priority date, Whichever expires later. For further options; see Form PCT/ISA/220. 3. For further details, see notes to Form PCT/ISA/220.
Name and mailing address of the ISA/US

Mail Stop PCT, Attn: ISA/US Commissioner for Patents
P.O. Box 1450, Alexandria, Virginia 22313-1450
Date of completion of this opinion 10 August 2007 (10.08.2007)
Authorized officer: Lee W. Young

PCT Holpdesic 571-272-4300 PCT OSP: 571-272-7774
Facsimile No. 571-273-3201
Form PCT/ISA/237 (cover sheet) (April 2007)
WRITTEN OPINION, OF THE INTERNATIONAL SEARCHING AUTHORITY Box No. I Basis of this opinion
International application No. PCT/US 05/43808
1. With regard to the language, this opinion has been established on the basis of
the international application in the language in which it was filed. a translation of the international application into which is, the language of a translation furnished for the purposes of international search (Rules 12.3(a) and 23.1(b)). This opinion has been established taking into account the rectification of an obvious mistake authorized by or notified to this Authority under Rule 91 (Rule 43bis.I(a))
2.
3. With regard to any nucleotide and/or amino acid sequence disclosed in the international application, this opinion has been established on the basis of: a. type of material
a sequence listing table(s) related to the sequence listing
LI
Li
-
.b. format of material on paper in electronic form
c. time of filing/furnishing
[-]
i [
4.
contained in the international applicatioin as filed filed together with the international application in electronic form furnished subsequently to this Authority for the purposes of search
In addition, in the case that more than one version or copy of a sequence listing and/or table(s) relating thereto has been filed or furnished, the required statements thit the information in the subsequent or additional copies is identical to that in the application as filed or does not go beyond the application as filed, as appropriate, were furnished.;
+.
Form PCT/ISA/237 (Box No. I) (April 2007)
WRITTEN OPINION OF THE INTERNATIONAL SEARCHING AUTHORITY Box No. V
PCT/US 05/43808 PCT/US 05/43808
Reasoned statement under Rule 43bis.1(a)(i) with regard to novelty, inventive step or industrial applicability; citations and explanations supporting such statement
1.
Statement ' Novelty (N) Claims Claims Inventive step (IS) Claims Claims Industrial applicability (IA) Claims Claims 34 1 - 33 NONE. 1 - 34. 1 -34 NONE ( YES NO YES NO YES NO
2. Citations and explanations: Claims 1-33 lack novelty under PCT Article 33(2) as being anticipated by US 6,724,421 B1 (Glatt). Regarding claims 1, Glatt teaches a video surveillance system comprising: at least two video cameras performing a surveillance by using a cooperative and hierarchical control process ([abstract], col, 3, In 8-14). Regarding claim 2, Glatt teaches two video cameras comprising a first video camera functioning as a master camera for commanding a second video camera functioning as a slave camera (stationary pilot camera ("master") for monitoring the area and one or more moveable slave cameras for monitoring at least part of the area, [abstract]). Regarding claim 3, Glatt teaches that the two cameras further controlled by a control processor (Such a camera incorporates a controller in the form of Z180 processor, col. 3, In 2-3). Regarding claim 4,Glatt teaches that the two cameras further controlled by a control processor embodied in a computer (col. 3, In 8-14). Regarding claim 5,Glatt teaches that at least one of the cameras are mounted on a movable platform (the slave camera is movable, claim -3). Regarding claim 6,Glatt teaches that at least one of the cameras having a flexibility of multiple degrees of freedormn (DOFs) (The ability to. to pan and tilt allows each slave camera to track the movement of an object (an intruder. for example) within the cell which the camera is assigned and into an adjacent cell, col. 3, In 17-20). Regarding claim 7, Glatt teaches that at least one of the cameras having a rotational flexibility for pointing to different angular directions (col. 3, In 17-20). Regarding claim 8, Glatt teaches that at least one of the cameras having an automatically adjustable focus length (Computer 41 is programmed to instruct, via bus 45, any of cameras 16, 18, 20 and 22 to pan, tilt, zoom, focus, col. 3, In 15-17). Regarding claim 9,Glatt teaches that at least one of the cameras is provided to receive a command from another camera to automatically adjust a focus length (The pilot camera produces a signal representative of the area. The location of a moving object in the area monitored by the pilot camera is determined. A signal is produced representing the location of the object. The slave cameras track the object based on the signal representing the location of the object, [abstract]).
Regarding claim 10, Glatt teaches that the surveillance comprises at least three cameras configured in a co-linear configuration (Fig. 1).
Regarding claim 11, Glatt teaches that the surveillance comprises at least three cameras configured in a planar configuration (Fig. 1). Regarding claim 12, Glatt teaches that the surveillance comprises at least three cameras with one master camera and two slave cameras disposed on either sides the master camera (Fig. 1). Regarding claim 13, Glatt teaches a video surveillance system comprising: a controller controlling a camera for performing a stationary global large-filed-of-view surveillance and a dynamic selective focus-of-attention surveillance by implementing a cooperative and hierarchical control process ([abstract], col. 3, In 8-14). Regarding claim 14, Glatt teaches that the cooperative and hierarchical control process controlling the camera to function as a stationary camera to carry out the global large-field-of-view surveillance and dynamic camera to carry out the selective-focus-of-attention surveillance ([abstract]). Regarding claim 15, Glatt teaches that the controller is embodied in a computer (cot. 3, In 8-14). Regarding claim 16, Glatt teaches that the camera is mounted on a movable platform (the slave camera is movable, claim-3). -(Continued on Supplemental Pages)-
__ Form PCT/ISA/237 (Box No. V) (April 2007)
WRITTEN OPINION OF THE INTERNATIONAL SEARCHING AUTHORITY Supplemental Box In case the space in any of the preceding boxes is not sufficient. Continuation of: V.2. Citations and explanations:
International application No. ' PCT/US 05/43808
Regarding claim 17, Glatt teaches that the camera having a flexibility of multiple degrees of freedom (DOFs) (an intruder for example) within the cell to which the camera is assigned and into an adjacent cell, col. 3, In 17-20). Regarding claim 18, Glatt teaches that the controller Is embodied in the camera as an embedded processor (Such a camera incorporates " a controller in the form of Z180 processor., col. 3, In 2-3). Regarding claim 19, Glatt teaches a video surveillance system for a large area comprising several compartmentalized zones comprising: a stationary video camera for monitoring the large area and at least two dynamic video cameras for monitoring the several compartmentalized zones wherein the global video camera and the dynamic video cameras are operated according to a cooperative and hierarchical control process (A video surveillance system for monitoring an area is made up of a stationary pilot camera for monitoring the area and one or more moveable slave cameras for monitoring at least part of the area, [abstract]). Regarding claim 20, Glatt teaches that the stationary video camera functioning as a master camera for commanding the dynamic video cameras functioning as slave cameras (stationary pilot camera ("master") for monitoring the area and one or more moveable slave cameras for monitoring at least part of the area, [abstract]). Regarding claim 21, Glatt teaches that the stationary camera and the dynamic cameras are further controlled by a control processor (Such a camera incorporates a controller in the form of Z180 processor, col. 3, In 2-3). Regarding claim 22, Glatt teaches that the stationary camera and the dynamic cameras are further controlled by a control processor embodied in a computer (col. 3, In 8-14). Regarding claim 23, Glatt teaches that at least one of the video cameras are mounted on a movable platform (the slave camera is movable, claim-3). Regarding claim 24, Glatt teaches that at least one of the video cameras having a flexibility of multiple 25 degrees of freedom (DOFs) (an intruder for example) within the cell to which the camera is assigned and into an adjacent cell, col. 3, In 17-20). Regarding claim 25, Glatt teaches that at least one of the video cameras having a rotational flexibility for pointing to different angular directions (The ability to pan and tilt allows each slave camera to track the movement of an object (an intruder for example) within the.cell to which the camera is assigned and into an adjacent cell, col. 3, In 17-20). Regarding claim 26, Glatt teaches that at least one of the video cameras having an automatically adjustable focus length (Computer 41 is programmed to Instruct, via bus 45, any of cameras 16, 18, 20 and 22 to pan, tilt, zoom, focus, col. 3, in 15-17). Regarding claim 27, Glatt teaches that at least one of the dynamic cameras is provided to receive a command transmitted as wireless signals from the stationary camera (The signal representative of the area is compressed and transmitted over a communications channel for remote monitoring, [abstract)). Regarding claim 28, Glatt teaches that the stationary video camera and the dynamic video cameras are configured in a co-linear configuration (Fig. 1). Regarding claim 29, Glatt teaches that the stationary video camera and the dynamic video cameras are configured in a planar configuration (Fig. 1). Regarding claim 30, Glatt teaches that the stationary camera and the dynamic cameras are further controlled by a control processor embodied in the stationary video camera (a pilot camera controls the operation of one or more slave cameras, col. 1, in 10-12). Regarding claim 31, Glatt teaches a video surveillance camera comprising: a global large-filed-of-view surveillance lens and a dynamic selective-focus-of-attention surveillance lens ([abstract]); and an embedded controller for controlling the-video surveillance camera to implement a cooperative and hierarchical control process for operating with the global large-filed-of-view surveillance lens and the dynami selective-focus-of-attention surveillance lens ([abstract]). Regarding claim 32, Glatt teaches that the camera is mounted on a movable platform (the slave camera is movable, claim-3). Regarding claim 33, Glatt teaches that the camera having a flexibility of multiple degrees of freedom (DOFs) (an intruder for example) within the cell to which the camera is assigned and into an adjacent cell., col. 3, In 17-20). -(Continued on Supplemental Pages)-

INTERNATIONAL SEARCHING AUTHORITY Supplemental Box In case the space in any of the preceding boxes is not sufficient.

PCT/US 05/43808
Continuation of:
/.2.
Citations and explanations:
34 lialm lacks an Inventive step under PCT Article 33(3) as being obvious over Glatt in view of US 6,795,106 B1 (Cooper). Regarding claim 34, Glatt does not teach that that controller is embodied in the camera as an application specific integrated circuit (ASIC) processor. However, Cooper teaches a camera controller (abstract], and that the controller 202 is intended to represent any of a number a ofaltemative controllers available in the art including, but not limited to, microcontroller, a central processing unit (a.k.a., processor), an Application Specific Integrated Circuit (ASIC), and the like., col. 7, 11-16). It would have been obvious to one skilled in the art to combine camera systems that use controllers. Furthermore, an ASIC would he teachings of Glatt with those of Cooper because both teaching allow the desired functionality to be designed into the controller. Claims 1 - 34 have industrial applicability as defined by PCT Article 33(4), because the subject matter can be made or used in industry.
Case 1:10-cv-01 1....- JCC-IDD
Document 1
Filed 10/1... 0 Page 1 of 11
IN THE UNITED STA'T'ES D)ISTrRICT COURT FOR THE EASTERN DISTRICT OF VIRGINIA AI.EXANDRIA DIVISION
I L I~-7----- ~j;_,,,,. ---^-------VIDSYS, INC., a Delaware Corporation 8219 Leesburg Pike Suite 250 Vienna, VA 22182
Civil Action No.
(0
J7b
Co / /i
) JURY TRIAl.. DEMANDED
Plaintiff,
PROXIMEX CORPORATION, a Delaware

Corporation
440 N. Wolfe Road Sunnyvale, CA 94085
Defendant. eenat
,.
,.................. ~...... 1.....,, ,.,,,~..,,.....,.,..,...... ........ ........ .. ,, I I CON'PILAI NT Folt_[),CiLA.RA'l1OIZ .JUIGEM1":NT
Plaintif; VidSys, Inc., ("VidSys" or "Plaintifl") by and through its ubdersi'gned counsel,. brings this action against defendimnt, Proximcx Corporation; ('"Proximex" or "Defndant") and alleges as follows: INTRODI)UCTION 1. Plaintiff VidSys respectively requests that this Court enter an Order declaring (1) that
VidSys does not infringe U.S. Patent No. 7,777,783 (the '783 Patent), (2) that the '783 Patent is invalid, and (3) that VidSys did not breach a Mutual Non-Disclosure Agreement hetween the parties, dated November 14, 2006, or misappropriate any trade secret from Defenldant.
Case 1:10-cv-011oS-JCC -IDD
Document 1.. Filed ,10/T 10 Page 2 of 11
7.
VidSys is a provider of Physical Security information Management ("PSIM") software
marketed as "VidShield" and "RiskShield." PSIM software systems are designed and optimized
to integrate and analyze information from traditional security devices and systems, and present the necessary data to automatically or manually analyze the situation in real time or thereafter. VidSys' PSIM software (VidShield) includes features known as "Virtual Tracker" and "Back Tracker" which provide a user with control over security camera video display operation.
I-
8.
Proximex also is a provider of PSIM software and, upon information and belief, has
offices throughout the United States, including in California, Colorado, Connecticut, North Carolina, Tennessee, Texas, and Washington, D.C. Proximex's PSIM software includes a
feature known as "EZ Track," which, upon information and belief, also provides a user with control over security camera video display operation. 9.( VidSys and Proximex are direct competitors in the PSIM software market. Upon
information and belief, VidSys' and Proximex's competing products include the Virtual Tracker, Back Tracker, and EZ Track features described above. 10. VidSys and Proximex have been in engaged in competition for the business of a major
prospective customer ("Customer"). In connection with this competition, VidSys has installed its PSIM software, which includes the Virtual Tracker and Back Tracker features, in a Customer demonstration computer laboratory located in Herndon, Virginia, which is located in this judicial
district. Upon information and belief, Proximex has installed its PSIM software, which includes
the EZ Track feature, in the, same Customer demonstration computer laboratory located in
Hemdon, Virginia in competition for Customer's business. Upon information and belief, VidSys
Case 1:10-cv-01185-JCC -IDD Document I
Filed 10/19/10 Page 3 of 11
and Proximex have had their respective PSIM software. products installed in Customer's Herndon facility for approximately two years. 1. VidSys permits Customer to use its PSIM software, which includes the Virtual Tracker
and Back Tracker features, to demonstrate for the Customer the capabilities of the software. As a result, Customer enjoys at least an implied license to use VidSys' software. VidSys does not have information as to whether or not Proximex licenses Customer to use its PSIM software in Herndon, Virginia. However, if Customer does not at least enjoy an implied license of the Proximex software for demonstration purposes, then upon information and belief, there would be no point in Proximex's installing the software in Customer's Herndon, Virginia facility. 12. Upon information and belief, Proximex has installed its PSIM software in the Reston,
Virginia facility of a well-known information technology company ("Technology Company") for evaluation of such software. Upon information and belief, the installation with thisTechnology Company was in place approximately in the late 2008 to early 2009 time frame. 13. Recently, there have been one or more announcements in the press that Proximex was
awarded a key patent for its EZ Track technology. One such announcement is attached hereto as Exhibit 1, and incorporated herein by reference. Based on the date of the announcement, and upon information and belief, the key patent referenced is the '783 Patent. VIDSYS' RELATIONSHIP WITH PROXIMEX 14. VidSys, from time to time, explores and, if appropriate, partners with third party vendors
to explore business opportunities with prospective customers seeking PSIM solutions.
N.
Case 1:10-cv-011 f85-JCC -IDD Document 1 Filed 10/1 '0
Page 4 of 11 .
15.
In 2006, VidSys and Proximex communicated with each other via multiple telephone
calls and emails to and fromn a location in this judicial district in order to explore 'a business opportunity with a major national retailer. The contemplated business opportunity involved an in-store security surveillance solution, or what is now referred to as a PSIM solution. 16. Inanticipation ofjointly submitting a proposal to this same retailer, the parties executed a
Mutual Non-Disclosure Agreement ("NDA") on November 14, 2006. A copy of the NDA. is attached as Exhibit 2, and incorporated herein by reference. ' 17. The NDA explicitly provides under Section 2 that Confidential Information does not
"include any information which Recipient can establish (i) was publicly known and made generally available in the public domain prior to the time of disclosure to Recipient by Discloser; (ii) becomes publicly known and made generally available after disclosure to Recipient by Discloser through no action or inaction of Recipient; or (iii) is in the possession of Recipient, without confidentiality restrictions, at the time of disclosure by Discloser as shown by Recipient's files and records immediately prior to the time of disclosure." 18. Under Section 9, the NDA further provides that its Term, shall "survive until such time as
all Confidential Information disclosed hereunder becomes publicly known and made generally available through no action or inaction of Recipient." 19. After executing the NDA, Proximex telephoned and emailed VidSys' office in Virginia
on several occasions in'order to exchange information regarding VidSys' and Proximex's PSIM technology and to finalize the proposal for the aforementioned prospective retailer customer. The parties jointly cooperated in providing information for the proposal.
Case 1:10-cv-01 i 5-JCC -IDD Document 1
Filed 10/1Y,10 Page 5 of 11
20.-
VidSys sent the proposal from its Virginia office, on behalf of both VidSys and
Proximex, to the retailer, but for reasons unrelated to the proposal's merits, the business opportunity was no longer of interest to the retailer. 21. The parties had no further communications about their respective products and services
over the ensuing three years until Proximex sent the threatening letter that gave rise to this controversy. PROXIMEX'S ACCUSATIONS AND THREATS 22. Upon information and belief, the '783 Patent was recently issued by the United States
Patent Office on August 17, 2010. 23, On September 29, 2010, outside counsel for Proximex sent a cease and desist letter to
VidSys in which he accused VidSys of making improper use of Proximex's proprietary technology and product portfolio, including infringing the '783 patent, misappropriating Proximex's trade secrets, and violating the NDA. A true and correct copy of Proximex's
counsel's letter ("the September 29 letter") is attached as Exhibit 3, and incorporated herein by reference. 24. Among other things, the September 29, letter asserts that: (a) (b) Proximex is the "owner of all right, title and interest in and to U.S. Patent-No. 7,777,783 'Multi-Video Navigation System' (the 'Proximex Patent');" "The Proximex Patent covers systems and methods of synchronizing video data generated using multiple video cameras and automatically generating a stitched video sequence based on the user selection of video camera, embodiments of which include tracking of a target between video cameras;"
Case 1:10-cv-011d5-JCC -IDD Document1
Filed 10/1@,10 Page 6 of 11
(c)
M "Proximex currently markets and sells its Proximex EZ Track T a multi-video navigation solution that easily tracks individuals across multiple camera views and different types of cameras that is based on the Proximex Patent;"
(d)
"[S]ubject to the terms of a Non-Disclosure Agreement, dated November 14,
2006, (the "NDA"), a copy of which is also enclosed herewith, VidSys has been in possession of Proximex proprietary confidential information since at least November 2006 when Proximex disclosed such confidential information to VidSys;" (e) VidSys' products called the VidSys VidShield and RiskShield "contains a Virtual Tracker feature that enables easy tracking of an. object in 'real time and. Back Tracker, which allows operators to go back in time, using recorded video for forensic analysis of events;"
(f)
25.
"This product [feature] therefore appears to infringe the Proximex Patent

described above;"
The September 29 letter demanded that VidSys "agree to stop selling and offering to sell
any component of the VidShield, RiskShield or any other VidSys product that infringes upon the
Proximex Patent or Proximex's confidential information" no later than October 8, 2010. 26. VidSys did not receive the September 29 letter until approximately a day before the
response deadline of October 8, 2010 because the letter had been sent. to VidSys' Virginia
address listed in the NDA which is no longer the current address for VidSys. 27. VidSys develops, 'manufactures, offers for sale and sells one or more products with
features of the type accused of infringement by Proximex, and continues to develop, manufacture, offer for sale and sell such products and services in Virginia. .Proximex'sconduct, directed towards VidSys in Virginia, has created a reasonable apprehension on. the part of VidSys that it will be faced with an infringement suit if it continues' to make, use, sell and offer to sell its PSIM products, including VidShield and/or RiskShield products with the Virtual Tracker and/or Back Tracker features.
Case 1:10-cv-011b5-JCC -IDD
Document 1
Filed 10/
10 Page 7 of 11
28.
VidSys has not infringed, and is not now infringing the '783 Patent, directly or indirectly,
nor has VidSys induced or contributed to others' infringement of the '783 Patent. 29. Additionally, the '783 Patent is invalid because, inter alia, the alleged claimed inventions
fail to satisfy the conditions for patentability specified in 35 U.S.C. 101, 102, 103 and/or 112. 30. VidSys independently developed its Virtual Tracker and Back Tracker features, as well
as its other software products and services, without the use of any Proximex trade secret information and without violation of the terms of the NDA. 31. Moreover, upon information and belief, the information provided to VidSys by Proximex
does not constitute information that is protected as trade secret information. 32. Upon information and belief, Proximex's asserted patent rights, purported trade secrets,
and purported proprietary confidential information disclosed pursuant to the NDA, all, according to Proximex, concern the same technology, namely Proximex's EZ Track. feature and VidSys' Virtual Tracker and Back Tracker features. 33. If VidSys were to acquiesce to Proximex's demands, VidSys would suffer irreparable
harm to its business which would have an impact on VidSys in this judicial district. COUNT I (Non-Infringement of the '783 Patent) 34. 35. VidSys repeats and re-alleges paragraphs 1-33 as if fully set forth herein. Based on Proximex's conduct purposefully directed towards VidSys in Virginia, Plaintiff
'
has a reasonable apprehension that it will be faced with an infringement suit if it continues
Case 1:10-cv-01185-JCC -IDD
Document 1
Filed 10/19110
Page 8 of 11
making, using, selling or offering to sell its PSIM software including its VidShield and RiskShield Virtual Tracker and Back Tracker features. 36. As a result of Pioximex's conduct as outlined above, an actual controversy now exists
between VidSys and Proximex concerning whether the VidShield and RiskShield Virtual Tracker and Back Tracker features infringe any claim of the '783 Patent, which is attached hereto as Exhibit 4. 37. 38. Patent. COUNT II (Declaration of Invalidity of the '783 Patent) 39. 40. VidSys repeats and re-alleges paragraphs 1-38 as if fully set forth herein. Based on Proximex's conduct as outlined above, VidSys has a reasonable apprehension VidSys has not and does not infringe the '783 Patent. Therefore, VidSys seeks entry of a declaratory judgment that it does not infringe the '783
that it will be faced with an infringement suit if it continues making, using, selling or offering to sell its PSIM software including its VidShield and RiskShield Virtual Tracker and Back Tracker features. 41.
C
VidSys has not and does not infringe the '783 Patent. Moreover, none of the claims of
the '783 Patent are valid. Patent invalidity is a defense to patent infringement, and thus an actual controversy now exists between VidSys and Proximex concerning. the validity of the claims of the '783 Patent.
Case 1:10-cv-0175-JCC -IDD
Document 1
Filed 10/19/10 Page 9 of 11
42.
Therefore, VidSys seeks entry of a declaratory judgment that all claims of the '783 Patent
are invalid because they fail to satisfy the conditions for patentability specified in 35 U.S.C. 101, 102, 103 and/or 112. COUNT III (Declaration That VidSys Has Not Breached the NDA or Misappropriated Trade Secrets) 43. 44. . VidSys repeats and re-alleges paragraphs 1-42 as if fully set forth herein. Proximex has accused VidSys of breaching the NDA and misappropriating trade secret
information based on Proximex's contention that VidSys has used Proximex's purportedly confidential, proprietary, or trade secret information relating to the technology described and/or claimed in the '783 Patent. 45. Contrary to Proximex's' accusations, VidSys has not breached the NDA or
misappropriated any Proximex trade secret information. 46. An actual controversy exists between VidSys and Proximex with respect to Defendant's
accusation that VidSys breached the NDA and misappropriated trade secret information because the allegedly misappropriated information: (a) under Section 2 of the NDA, was generally known to the public or to persons in 'the PSIM industry at the time ofdisclosure to VidSys or since the time of disclosure to VidSys; did not survive the "Term" of the NDA as provided, Section 9 because such information has become publicly known and made generally available through no action or inaction of VidSys; did not derive any independent economic value from not being generally known to the public or to persons in the PSIM industry;
(b)
(c)
Case 1:10-cv-O11 5-JCC -IDD
Document 1
Filed 10/0 'iO
Page.lO of11
(d)
was not used or disclosed by VidSys in violation of the terms of the NDA; and/or
(e) 47.
did not qualify as trade secret information under any applicable state or federal law.
Therefore, VidSys seeks entry of a declaratory judgment that it has not breached the
NDA and that it has not misappropriated any Proximex trade secret information.
PRAYER FOR RELIEF Wherefore, VidSys prays that the Court enter judgment in its favor and against Defendant Proximex, providing the following relief: A. B. Entry of Declaratory Judgment that Plaintiff VidSys does not infringe the '783 Patent; Entry of Declaratory Judgment that each and every claim of the '783 Patent is invalid, void, and without force or effect; C. Entry of Declaratory Judgment that Plaintiff did not breach the NDA and did not, misappropriate any Proximex trade secret information; D. A finding that this case is an exceptional case and award Plaintiff VidSys reasonable attorneys' fees, costs, and expenses for this action pursuant to 35 U.S.C. 285; E. F. An award of costs incurred in connection with this suit; and An award of such further and other relief as the Court deems appropriate.
Case 1:10-cv-01 65-JCC -IDD Document 1
Filed 10/1 .1 0 Page 11 of 11
JURY DEMAND VidSys hereby demands a trial by jury of all counts of this Complaint which are. permitted to be tried by a jury.
Date: October 19, 2010 Respectfully submitted,
inia Bar No. 37464) David R. Yohannan ( Stephen R. Freeland (Virginia BarNo. 72947) Kelley Drye & Warren, LLP 3050 K Street, N.W., Suite 400 Washington, D.C. 20007-5108 Phone: 202-342-8400 Fax: 202-342-8451 E-mail: dyohannan(ibkelleydrye.com E-Mail: sfrceland(@kelleydrye.com
Attorneysfor PlaintiffVidSys, Inc.
441
C,)
0
4~
N
I~ ~
.. o, :
I
'
0;Tr o:
on-~
'>.
.c ..
':'
~Oj":.~
L r
;'~ ' ' ::~1: :::%I
::
:: :I
'
r
i
c5
"
I ,. :i iij~ .,~
w
L
F4
0
-
0
r
'~&.
LU
N.
0J
rC
5I
jf .", '': ci 6}i yri'~ .y,", .;-t.. ':t 4" f e;N; ...... :.
t
e
r~~~~
W "w'44jl i+ ,:
~Tx,
,
..
.
/t
J~sR
'
,It'~s
A !
4'
. sr'i;
3
',' S
ii~ .tRK ;"J'iMx 4VrK"

' " lS J f ?1
t
'"
'
r I 1 N ,
"fJ f u,m Y l Ur
Mr,;+C ; t
dr.
Aid
( C
ZLu
H 0
C,) C
Lu
'
C', et'
a)
VI'-
-~4-a
0o
~0o:
.V
a aQ
:c..").T
Ci
UW
C,)
m
a o
_
0.
a L.
o a E
-
C
L.muWO
a)~O
CJ~ ~
C,)
~eE~au~
0 ~
a ~
00
01
wr
' ,
'
,.
'
'
0
0
w'. ' : wry
U.:.
ri .". ii
::.: I
(I '0; rn -a..
S ...
aI ...
o
O
~ ~ : ::. :
I,
V: CC.
-6-4
'3 (:
~:LL:
kk, .. .... .
Cl)
M
C,
0 Q.t '.
F-
(N (N 09
U ::~ :I'.:~
w3
HG
.2*'
.4
.
*4
5
D.
* 0
C4
N-0
(~QI. ~
~~1)
00 L
V7.
--
0D
sI
00'
LU Li
rJ
I I i
LPJ
Lu
I-
0~
woo..
40D~ O ::
Go
C?.
V"
C,)
0
0)
Ul:
Cl)c
.0..,*., C o
~
-~
-'-'a)
0)co
0 C
0 0 0
(VL~jCW L~QQL.
a a)inc~' o C/*J~a) a ~ > >c~Q QQL.4in~

L
C*+-~ Cl) O~ ~0
00
a
*
L.
a
U
0 )
C/)
W
Im
: :~) ::
'I'
I
~
. ,. ..
:' ': ''" ~ ~ :::::::: ~': : ! ..
' '' '' :. ":
,:: \ i
., O ~
Jm
0
0
I:: ::~ ~: : . . . .: ..::
.i
:'
::
O
0 a
.: :::: ., : :I :::
: ..~.:.
:!:i (;j~ii~9~ . ii I 1. 7 1 i.~.:::: .i. Ij .: : . 1.::: . :: .:
o:.
cd
Ik
"0 "0
0ilia
-U0 '
"
0 00
Co
4-,
Co Co Co X U) U)
I
"S
c:
3;
Co.
_
0
L...
Ci)
a o
Co
a)
Co
Q a)a) a)
>
a
Co
L~
~..
$a)~a)
c~ c~)(~
(~)
~*
Eu
w
HIF3
Wa
S%3
.a) 0rY. Cl)' ...

0
U-
:::
:rn~: 0'.
;. .:.
-'' ' 'o

0
: .':'.::';)o:~ ...... t~..'... . E

; " I ~. . i::~ ~' . ; '' .:..' (
..
_~r
*.
Q4
C D
:,'~ .:
:-'
U
:: :.:: ..~ :. ... i
aa-..: '. 0 0:
FhA
VI
Li
(I)
c0i~
.4
1 C..4pp
9<0
LL~a)
m
J.
u-u-rn-i
UU U
(N
w.
0
O .
.,
:.
0 *
0
C/)mmmllN~~ O 0
: :.:: : :: ' :
:I
-ic :
: -; '::i: iii :: '~ :~ :
~ :i I
- :-:'i .: 'i
w
"'V
i
vI"
C,, C,)3 .(/):
"
Q :1' .0 x"
0
M
O'
00
Li
C?.N
OZ
/
it
:l"
rio,.
0u
Cl:
U-L
'
O ',
CN1
LiJ
,,
Ine
oo-
Cl)
C)
C.
Zw
H
a
-cJQ)
:.:
0
0
U
0
Oe
1".
N '
w
C?
wI"
E
,a
Co:
ca) 0 .E~ 0 U
*U
0 N
00
Li
fm'
w
.....
C.L
-.
U 0
O
.0
0 0-
-U
"
0d
.ME
Cl Cl
0)"
W L,
U)
o.~ oA
z
Z' 4
C,)) _U
C,)
a
N
4. w
Lu.
..
A,
0
_
0lw'. C,
" o
OA
J
aiW
O N 0
'
.:4
'.i* (i
c5
.-Idg-)
(I)
: . . .
cq)l0
_ 0
0A
Q:
U .0 -n
l.U r 7
w
H
C,,
00
.Q
.rrV
..
0)
Cl .0
-
ely
(J)
<~,..
.1
L.
06c1)' U 0
a>
C,,
CIY*
.4.
zo
00
c:~)
O)XC~3L~ U U U U
0.
0 .o Cl
~LUQQ
LU
U
* 0
a).
C')
oT
44:
f,:
:C3K
Z H
::
:.:. .;:.. . .. ,~
...~.~ .. : ~ :~ . :~.:~~.~
.,.
~ :~ :" :
: ~. ' ,
... ~ ~;
: :: :
,:
: : :
. ,
''' ~:: , : :
:
E
i
Q'
::
":
C '
::: : :
:ii
.o
,. N.
:: :''' "' ::: :'
TAB
Volume 1 Number 1
August 2003
Redstone
Redstone Launches Newsletter
Inside this Issue
Redstone Launches
New's
e N~w ttar
Now, Redstone Friends and Family can get The Redstone News delivered fresh to your e-mail box
like clockwork every -- well, often
Steve Browt
Head.
enough, don't you worry. You will receive notice of all the happenings and news around the Redstone enterprise including new faces, business development activities, new products and projects, training and professional development programs, white papers and technical writings. Stories and infonnation are welcomed at the Redstone News Editor's Desk at Rharrison@redstone -is.com.
nt Int&Wte Tansportation Intelligent Transportaton:
Syste.m (ITS) rogramns Smnart C.Np Solutions With Smart Plate% Now Pmduct RatlutfsualPucsu.lt dule
NeA. siLe
Prevtw
Steve Brown To Head Intelligent Transportation Systems Programs

Reatvn Irttegxate. Soatutlois
5085 List Drive Suite 110 olorado Springs. CO 80919 ?1 9- -12.5
www,Redstunie-S.coni
Redstone is pleased to announce that Steve Brown will direct the Intelligent Transportation Systems Program. A nationally recognized expert, Steve Brown has been associated with the transportation industry since 19.75. Steve has served as President of Integrated Software Solutions, Inc. since 1993. ISS developed the TransLink Closed Loop System for centralized control and monitoring of the Wapiti Microsystems' 170
controller firmware. TransLink has been adopted by 9 state agencies and over 50 local agencies in the US, Canada and Mexico. Steve began his career as a field supervisor for Signal Maintenance, Inc., a traffic signal maintenance company in Southern California, responsible for the maintenance and operation of signalized intersections throughout the Southern California area. Steve's experience includes 16 years with Safetran Traffic' Systems, Inc. in Colorado Springs, as Manager of Technical Sales and Services. For the last three years, Steve was Lead Software Engineer and Associate Vice President for AECOM Systems Integration Group. As Director of ITS Programs, Steve has already made presentations on Redstone solutions to key transportation groups in seven states. He will coordinate product sales efforts Smart Plate Solutions in tolling, parking, and traffic security access.
Continued on Page 2
'
,. . ~
Wa~ .~---~I-----~-R^ - I~
August 2003
The new smart chip technology in M the Redstone SmnartPlateT can locate, store, and manage data assets for a wide variety of client needs. SmartPlateTM technology is used for
automatic identification and
detection of vehicles. It is an input/output device for Redstone's Intelligent Transportation Systems (ITS) solutions.
Transportation: Smart Chip Solutions with Smart Plates

Redstone SmartchipTr technologies emit unique ID's over approved RF frequencies, much like the light from the beacon on a radio tower, but from within a tiny emitter that can be the size of a dime. These chips can be deployed in a number of sizes and shapes, including the Smartplate license plate that will read at up to 200 feet and speeds of up to 100 mph. The SmartchlpTM technologies provide Redstone-IS software with a method to identify and track mobile assets. Mobile' assets can vary from vehicles to identification cards.
Intelligent
The smart plate is used in fleet management, automatic vehicle identification, security access control, and asset management. The package includes RFID transponders, antenna, receivers, and coninunication modules, which interact as a system with the smart plate. Redstone provides the frontend software package for this system. These Redstone technologies, when combined, can provide a transportation client with complete tracking, event management, time stamp, physical location and any other specific tracking capabilities required by the application. Through simple repackaging, SmartPlateTM can also be used to track containers, airport ground vehicles (including airplanes on the tarmac), nursing home patients, employee badges, or virtually anything that can pass through a standard reader.
VlsualPursuit uses an array of camera inputs to track a subject through a facility
New Product Rollout VisualPursuit

In August, Redstone will announce the availability of VisualP ursuit, a software package that can be used by security personnel, facility command-and-control operators, or . any other operator monitoring video cameras inside or outside a facility to easily track a moving subject.. Surveillance, early intervention in a problem situation and rapid emergency management are all benefits of this enhanced operator control product. The only requirement that an area needs to have in order for VisualPursuit to be used is adequate coverage by security cameras.
VisualPursuit expands Redstone's intelligent facilities package for surveillance. It includes remote monitoring of buildings, complexes, and parking structures. The tracking feature makes the operations and control center more effective in the use of video monitoring and interactions with security personnel. Operator efficiency, flexibility, and dynamic support will improve monitoring and response, increasing the value of any camera surveillance system large or small. VisualPursuit can be installed in conunercial offce buildings, airports, hospitals, prisons, jails, laboratories, secured buildings or complexes such as courthouses, government offices, convention centers, event and sports facilities any area where better surveillance, situation and emergency management are important.
Next Issue: VisualPusuit Priduct vcrvicw, MapTrack Preiew, and Security White Paper
Gate Access Cointrol
2'
TAB E
Redstone Intergrated Solutions - Products And Services
Page .l of
13
Home I Portfolio i <<Products & Services>>
I Demos & Downloads I News I Contact Us
e. Redstone Products * Professional Services o System Integration and o Redstone Software Engineering Console o Testing.and Software Quality o MapTrack Assurance o Visual Pursuit . o Support Services o ResourceTrack o Project Management o SmartPlate o Configuration Management o Traffic View Automation. Standardization. Simplification. Innovation. These four principles underlie everything we do at Redstone. Through the use of technology, command-and-control of facilities has become easier and more costefficient. The newest breakthrough has come in the form of the Redstone Console. It is a software suite that allows an operator to visually monitor the facilities, computer resources, systems, even individual cars, and interact with them all from a single interface. The Redstone technology is superior to other alternatives because it provides you with unparalleled monitoring and control capabilities. It gives you the power to efficiently control your facility, and does it withfewer people on staff. The Redstone console is designed in a way that reduces the learning curve, while keeping intuitiveness and multifaceted functionality. In the following paragraphs we will be disecting the functionality of the Redstone Console in order to give you a better Idea of how the software operates. The five main functions of the Redstone Console are: * * * * * Camera Monitoring Device Tracking Building Automation Resource Monitoring Command & Control
Camera Monitoring/Facility SecuritySystems - VisualPursuit

return to top Redstone's VisualPursuit enhances an operator's effectiveness by allowing the operator to easily track a moving subject throughout a facility from a single monitor. It accomplishes this by consolidating the video from physically adjacent surveillance cameras into one screen, increasing the field of view of a physical location dramatically. VisualPursuit is a part of the Redstone family of software and when used with the full. system, adds subjecttracking and wide-area monitoring capabilities to an operator's toolkit. VisualPursuit's display combines the output from several surveillance cameras into a single, MS Windowsstandard screen. * * * * The VisualPursuit screen contains a Subject View and Locator Views. The Subject View displays the current location of the moving subject. Up to 8 Locator Views display the output of the cameras closest to the one trained on the subject. Track a subject by clicking the Locator View that the person has moved into. The Locator View becomes the new Subject View, and all Locator Views are automatically reassigned and updated.
VisualPursuit works with Redstone's MapTrack or ResourceTrack products:
http://web.archive.org/web/2004060601 4343/redstone-is.com/products_services.html
'
10/20/2010
Page 2 of 13
* Click,a camera symbol on MapTrack or ResourceTrack to.start a VisualPursuit session. * MapTrack's camera symbols are color-coded so that you always know what cameras are being used in VisualPursuit. * From MapTrack, monitor and control devices at the same time you're engaged ina VisualPursuit (lock doors, zoom cameras, etc). Track the subject's location and direction of movement on MapTrack's interface. VisualPursuit is an important element of Redstone's Intelligent Facilities package. VisualPursuit makes an operator more.effective and efficient by allowing her to easily track a moving subject using one monitor. It' also allows her to. quickly scan an area covered by security cameras without having to view several monitors. Applications of VisualPursuit include commercial office buildings, cities, airports, hospitals, sports facilities, prisons, pipeline companies, refineries, and convention centers to name a few. Basically, any area.that needs better surveillance and/or emergency management will benefit from the installation of Redstone's VisualPursuit. VisualPursuit can be used to monitor and/or control: * Facilities: casinos, housing complexes, office buildings, convention centers, governmental buildings, prisons, corporate campuses, high-rises * Airports: access and security * Educational Campuses: * Complex Infrastructures such as chemical plants, pipelines, manufacturing complexes, city intersections * Entertainment complexes: amusement parks, zoos, sports facilities, museums VisualPursuit in Action: 1. The picture below is a screenshot of the Visual Pursuit interface. This view shows nine cameras. The large, central window contains the output from the current main tracking camera;, other cameras in proximity to the current tracking camera are displayed in the other eight windows. By clicking on one of these adjacent views, you can change it to be VisualPursuit's main' camera view. The adjacent views would then automatically update, based on the new current tracking camera. This is an example of how you would use it to follow a person through a facility.
2. A series of LEDs give you an idea of where the current tracking camera is, in relation to another camera.
3. There is also a map layout that can assist you with visualizing exactly where each camera is in your facility. << Click Here for close up shot >>
(Tracking Devices) Inteligent Transportation,_Smart Plates

return to top
T The new smart chip technology in the Redstone SmartPlate M can locate, store, and manage data assets for a wide variety of client needs. SmartPlateTM technology is used for automatic identification and detection of vehicles.
t...
http://web. archive.org/web/20040606014343/redstone-is.com/products_services.html
10/20/2010
Page 3 of 13
The SmartChip emit unique IDs over approved radio frequencies, much like the light from a beacon on a radio tower, but from within a tiny emitter the size of a dime. These chips can be manufactured in a variety of T sizes and shapes, including the SmartPlate M license plate whose signal can be read at up to 200 ft away and at speeds of up to 100 mph. The SmartChip technology provides Redstone software with a method to identify and track mobile "assets" such as vehicles and identification cards or security badges. The SmartPlate" m is used in fleet management, automatic vehicle identification, security access control, and asset management. The package includes RFID transponders, antennae, receivers, and communication M modules that interact as a system with the SmartPlate T . Redstone provides the front-end software for this system. These Redstone technologies, when combined, can provide a transportation client with complete tracking, event management, time-stamp, physical location and any other specific tracking capabilities required by the application. Through simple repackaging, SmartPlateTM can also be used to track containers, airport ground vehicles (including airplanes on the tarmac), nursing home patients, employee badges, or virtually anything that can pass through a standard reader. The applications for this software are: * Redstone SmartPlate AVI systems: Traffic Light Priority, Passenger Information, Terminal Management, Access Control, Fleet Management, Speed Measurement, Automatic Vehicle Detection * Traffic Flow Information.with untagged metal detection feature: Vehicle Counting, Ramp Metering, Incident Detection, Average Speed of Vehicles, Vehicle Density * Transportation: Traffic Flow Measurement, Speed Measurement, Identification of Smart Plate Vehicles, Traffic Light Priority for buses, emergency and Smart Plate: vehicles, Bus Passenger Information at Bus Stops, Bus Terminal Management, Access Control, Road Pricing/Tolling Systems, Tunnel and Hazardous Materials Management.
Building Automation - MapTrack

return to top
Redstone's MapTrack is an easy-to-use, graphical interface. Users can quickly locate, view, and/or control computer systems, video, security cameras, audio, card access readers, doors, and, a host of other devices, all from a real-time, interactive map interface. MapTrack can display maps or diagrams at the level of detail. / required by the user. MapTrack is a part of the Redstone family of software products and when used with the full system, adds digital, interactive map and device control capabilities to an operator's toolkit. MapTrack's display combines digital maps or diagrams With graphical symbols that can represent the locations of items such as: * * * * * * * security cameras, doors, and windows computers/components in an equipment rack valves on a pipeline traffic and railroad signals alarms (motion detectors, fire alarms, etc.) steps in a process diagram and many other devices
These graphical symbols are more than simple static representations, they can: * display a device's status (alarming, locked/unlocked, on/off) by 'a change in color or by flashing * allow device interaction and control * allow viewing of device output " give the operator a quick entry into another Redstone module such as VisualPursuit. MapTrack is an important element of Redstone's Intelligent Facilities package. MapTrack makes an operator more effective and efficient by using a single, real-time, map-based interface through which an operator can
http://web.archive.org/web/20040606014343/redstone-is.conm/products_services.html.
10/20/2010
Page 4 of 13
monitor and respond to events over an entire facility - all from one or many workstations. The look and feel of this interface can be customized to suit the needs of the client. Applications of MapTrack include commercial office buildings, cities, airports,. network operations centers, hospitals, sports facilities, prisons, pipeline companies, refineries, and convention centers to name a few. Basically, any area that needs better facilities management, surveillance, and/or emergency management will benefit from the installation of Redstone's MapTrack. MapTrack can be used to monitor and/or control: * * * * Facilities: access and security, climate control, emergency systems, lighting Refineries and chemical plants: security, emergency response, product delivery, product status Pipelines: emergency response, security, product delivery, product status, cathodic protection Networks and hardware: system status, hardware status, connectivity
<< Click Here for close up shot >>
<< Click here fora close up shot >>
Redstone's software allows for remote monitoring of numerous buildings, complexes, and parking structures by the property owner for the purpose of reducing'liability, reducing operating .costs, increasing safety and security, reducing loss of assets, and increasing the owner's capability to proactively or quickly respond to a variety of incidents, ranging from simple facility maintenance issues to criminal activity. It also offers the option to hand-off control of subsystems to emergency response teams during an event (if so equipped.)
Monitoring Resources - ResourceTrack

return to top A command and control console would not be complete without a way to monitor computer and electrical resources. This is where ResourceTrack comes into play. ResourceTrack is the tool used to monitor any machine in the facility. It is very important to know the status of computer resources that control power, alarms, communication, etc.
Command & Control -The Redstone Console

return to top
< Click Here for close up shot >> The Redstone Console is the powerhouse of the entire suite of Redstone software. The Console is the engine that allows Redstone to easily integrate disparate systems into one user interface. It provides the common operating framework, device monitoring and control, interface features, device communications, and database used by the rest of the Redstone software system. The Console's display is called Resource Navigator. Navigator. is the primary "window" into the Redstone Console. It allows the user to view, monitor, and control every device, file, and system integrated into the Redstone system. * Locating devices to control or files to view is as easy as navigating in MS Windows Explorer. the * Click acamera-icon in Resource Tree and view its video output to the right. * Click a filename or a website and view its content without leaving the application. VisualPursuit, MapTrack and ResourceTrack provide different interfaces into the Redstone Console. These optional interfaces allow you to: * Easily track a moving subject or monitor a wide area of a facility from one interface. * View and/or control elements of a facility from a graphical, map-like interface.
http://web.archive.org/web/20040606014343/redstone-is .com/products_services.html
10/20/2010

* Track a moving subject on video, at the same time you monitor their movement across the facility map.
Page 5 of 13
Applications of the Redstone Console include commercial office buildings, cities, airports, network operations centers, hospitals, sports facilities, museums, prisons, pipeline companies, refineries, and convention centers to name a few. Basically, any area that needs better surveillance, facilities management, or emergency management will benefit from the installation of the Redstone Console. The Redstone Console can be used in the following industries: * Facilities: casinos, housing complexes, office buildings, convention centers, governmental buildings, prisons, corporate campuses, high-rise buildings * Airports, Seaports, Train and Subway Stations * Educational or Corporate Campuses * Network Operations Centers and other Command-and-Control installations * Complex Infrastructures such as chemical plants, pipelines, manufacturing complexes, city intersections, emergency management organizations * Entertainment complexes: amusement parks, zoos, sports facilities, museums * Many more industries can benefit from the integration capabilities of the Redstone Console
Professional Services
return to top
The Professional Services group is responsible for providing fulllintegration, support services, and software engineering. Included is turnkey implementation and integration, testing, 24x7 maintenance monitoring programs, documentation, product development, and many other types of services as described in the following sections. The three key areas of Redstone's Professional Services are System Integration, Software Engineering, and Support. System Integration involves working with a customer to bring their hardware, software, and computer systems into one, easy-to-manage unit. Software Engineering includes.developing software to support a customer's integration effort, upgrading and expanding Redstone's current software, and developing new products to address industry problems. Support includes customer support at all levels, from Tier 1 through system integration support. Project Management and Configuration Management are also important parts of the Professional Services group. All projects undertaken by Redstone, whether they are internal development or client projects, will be properly managed in accordance with industry standards. Configuration Management ensures a traceable history and change tracking for code and documentation, is key component of software releases, and allows Redstone to ensure that the product the customer receives is the most current and accurate available.
System Integration and Software Engineering

return to top Redstone's Systems Integration and Software Engineering process work in tandem and follow and industrystandard software development process: Scoping, Process Mapping, Design, Documentation, Testing/QA, and Installation. Project Scoping encompasses evaluating the current status of a system or facility, gathering and writing Requirements, and compiling and writing Specifications (hardware and software). After Project. Scoping is complete, the Design phase of a project starts. After approval of the Design, we write all appropriate Documentation for the solution, then compile a Test/QA Plan and perform Testing. Only after the solution has passed Testing will we proceed to the Installation phase. System Integration can be standalone hardware/systems integration, or work together with Software Engineering to deliver a fully customized software solution that fulfills a client's hardware and software requirements in one project. Additionally, Software Engineering can be a standalone process, when working on custom or unique software development for a customer, or the creation of new Redstone software . products in-house.
http://web.archive.org/web/20040606014343/redstone-is.com/products_services.html
10/20/2010
Page 6 of
13
Software Engineering
return to top The Software Engineering process employed at Redstone closely follows the System Integration process, with the addition of developing and testing code. Software Engineering can be a standalone projector also employed as an adjunct to a Systems Integration, and vice versa.
Services Offered
return to top. Whether the project is Systems Integration or Software Engineering, Redstone follows the same process in order to bring the project to a successful completion.. This process is based on SEI-CMM recommendations (Software Engineering Institute, Capability Maturity Model) and includes the following phases: Scoping, Design, Build, Documentation, Testing/QA; and Installation.
Scoping
return to top
10/20/2010

The Scoping Phase of a project involves working with a customer to gather and write requirements, using those requirements to write hardware and/or software specifications, and process mapping. Standard Deliverables: * * * * * Scoping Study Document Requirements Functional Specifications Process Map Project Plan and Timeline
Page 7 of 13'
Scoping Study
return to top As the first step in a Systems Integration or Software Engineering project, the Scoping Study is designed to determine all variables influencing a project. Because systems have often been developed and managed independently of each other, many organizations can be unaware of all of the complexities involved in integrating systems or developing new software. A Scoping Study will help identify these complexities and plan. processes and mechanisms for dealing with these issues. Some of the activities involved in this phase may include: * Facility tours * Interviews with client Subject Matter Experts, Project Managers, potential users, Information Technology representatives, and Administration * Observation of the day-to-day operations surrounding the client's current systems Redstone then analyzes the results of the tours, interviews; and observation sessions to focus on determining what needs to be done to fully integrate the client's different systems or develop software. The result of these analyses is a document, produced from a Scoping Study Template 'to ensure consistency, that defines the full scope of the project, implementation estimates, recommendations for integration, and milestones for all potential activities. It also includes all information gathered during the Scoping Study for future use by the customer.
Requirements
return to top The Requirements phase is a natural next step in a Systems Integration or Software Engineering project, after the Scoping Study has been approved. Requirements are mandatory unless the project manager has good reasoning for skipping them. Working with a customer, and working from the information obtained during the previous phase, Redstone will produce a document that contains both functional and nonfunctional requirements for completing the project. These requirements will be as detailed as possible as they will be used throughout the remainder of the project: to create functional specifications, hardware/software design, and test plans. Requirements for both Hardware and Software will be handled in separate sections of the same document if both need to be addressed in the project. Requirements will also be developed from a standard template and brought under configuration management using CVS. Requirements are baselined, documented, and tracked through the fulfillment process with a dynamic Requirements Traceability Matrix (RTM) document.. Requirements are fulfilled through the testing process outlined later.
Functional Specifications
return to top Once the project's Requirements have been completed and approved by the.customer, Redstone will begin the Functional Specifications phase. From the techweb.com encyclopedia:
10/20/2010

Functional Specifications are the blueprint for the design of.an information system. It provides the. documentation for the database, human and machine procedures, and all of the input, processing, and output detail for each data entry, query, update, and report program in the system. Along with the Requirements and Design Document, discussed in Section 1.2.2.2 Design, the Functional Specification helps to form the backbone of a Redstone project.
Page 8 of 13
A Systems Analyst will develop Functional Specifications as needed for each project. The Functional Specification explains the "what" of the project. It is the blueprint for how the project or application will look and work. It details what the finished product will do, how a user will interact with it, and what it will look like.
Process Mapping
return to top Process Mapping is the mapping of serial customer operations to the functions and capabilities of the designed system. This process is performed by Redstone system analysts in conjunction with customer operators and technicians. Process mapping is applied, as an example, to hard copy checklists, or security procedures. The result is respectively an automated macro driven checklist, or an automated security. function driven by the software application.
Design
return to top The Design phase of the project takes the Functional Specification a step further. While. the Functional Specification defines what the project will build,. the Design phase will define how the solution will be built. The result of this phase will be a Design Document, outlining in detail how the proposed solution will be puttogether. For Software Engineering projects, the structure.of the programming, individual code modules-and . their relationships, and the parameters they pass to each other will be described. For solely Systems Integration projects, the Design Document will detail the configuration and relationships of the hardware, systems, and any additional software involved.
Documentation
return to top Besides the documents described previously (Requirements, Functional Specification, Design Document), each: project can also include additional documentation as necessary, written for both end-users and system/administrator personnel.. User Documentation All of the preceding documentation, outlined above, can be used as source material for the final user documentation. User Documentation will consist of the following (as applicable): * User Manual - typically consists of a task-based, quick reference section and a more detailed reference manual. This document is mandatory for all Systems Integration and Software Engineering projects and should be written to the "lowest common denominator" or novice end-user. * Online Help System - produced in tandem with the User Manual, a system designed as an adjunct to the program to deliver information from the user manual while the user is "in" the application. This is mandatory for all Software Engineering projects; as-needed for Systems Integration projects. This should be written to the level of the novice end-user. . Training Materials - training materials include a training manual, training presentations, and online .trainlng modules. The first is mandatory for.all Systems Integration and Software Engineering projects, the others.are optional. When training users, these documents will be written to the level of the novice user. When training, administrators, they will be written for a more technically experienced audience if necessary.
10/20/2010
Page 9 of 13
* Administrator Manual - the administrator manual is a technically detailed but easy-to-understand guide designed for long-term administration and troubleshooting of the system. This is typically written for a more experienced end-user. * Installation Guide - The installation guide is a document describing, in detail, how the system was installed. It also acts as a guide for future installations or re-installations. Even though information is presented in detail, it will be written in clear, easy-to-understand language. Written to the level of a more experienced user. System Documentation System documentation includes any materials relating to docurfienting the solution from the programmers' or architects' points-of-view. * System Documentation can include: o Hardware Specifications o Wiring Diagrams o Inline Code Documentation o Module Documentation o Device Driver Documentation
Testing and Software Quality Assurance

return to top The Testing phase of a project comes after a software build or systems integration. Testing can encompass component ("white box"), integration, system ("black box"), hardware, and documentation testing..
10/20/2010
Redstone Intergrated
Solutions - Products And Services
Page 10 of
13
ResoeIneg e
Soions II
ProuctIAnISeviesIageIo
User ManuaI, Online Help,

Training Witetrials
fTst Plans, Test

Cases
IItallatin and Admihimmtraor

Manualsel
Figure 1: The Redstone Testing Process An overview of the testing lifecycle is presented in Figure 2 and is explained in the following sections.
10/20/20 10
Page 11 of
13
Figure 2: Detail of Testing Cycle Software Quality Assurance applies to the source code and is integrated into the project throughout its lifecycle. It consists of: Code Critiques
Code Reviews
Code Walkthroughs Refactoring Extreme Programming (XP Style)
1.4 Su..port Services

return to top The Redstone Support group offers technical assistance to customers of both our Software Engineering and Systems Integration services.
http://web.archive.org/web/20040606014343/redstone-is.com/productsservices.html
10/20/2010
Page 12 of 13
The structure of the Redstone Support group follows a "tiered" support paradigm. The Tier 1 and 2 support staff are responsible for taking the initial support request from our customers and providing a basic level of support. Typical problems that are resolved at this support tier are basic software set-up, basic usage of product, configuration and general software upgrade issues, and known software problems. Tier 1 takes the call from the customer and attempts to solve the problem over the phone. If more research is required, the issue is escalated to Tier 2 for research and resolution. If the problem.is beyond known fixes, or is not solved within prescribed guidelines, the service request will be escalated to the Tier 3 support staff. Problems that are escalated to our Tier 3 support staff are beyond basic setup and usage issues. These issues may need programming involvement and/or code changes to resolve. These problems may also be complex software upgrade issues, customer application issues, or core product issues that need to be researched, duplicated, and tested in a support lab. If code changes are required, the service request is escalated to our development staff for resolution. Support Offerings Redstone Technical Support Services offer several support plans to reflect different customer.needs: * Standard Support is designed for customers with limited support needs outside of regular business hours, with support offered from 8 am to 5 pm MST. * Extended Support expands Redstone's basic support option to 5 am to 5 pm MST. * Premium Support provides access to Redstone support personnel 24 hours, a day, 7 days a week, for mission critical issues. * Should a client sign up for a,lower level of support and find they need technical assistance outside their support "window," support will be provided and billed to the customer in a separate billing. At that time, Redstone will give the customer the option to upgrade to a higher level of support, or just pay the current support invoice.
Project Management
return to top Project Management at Redstone follows the industry-standard, process-oriented approach.as outlined by the SEI-CMM and Project Management Institute (PMI). Based on the findings from the Scoping Study, and the information in the Requirements and Functional Specification, the Redstone Project Manager will develop a Project Timeline using these documents as a baseline. The Timeline will be an outline of tasks, necessary meetings, and give time estimates for all phases of the project and all deliverables. 10% estimate of how much work will be needed to complete This Timeline should give the customer a +/the project. Throughout the course'of the project, the Project Manager ensures: * * * * * Adherence to the timeline and processes Tracking of any problems or issues encountered Efficient use of resources Adherence to the principles of SQA Clear communications with the client
1.6 Configuration Management

return to top All project documentation, user documentation, system documentation, and code is version-controlled with configuration management software. Currently, Redstone uses CVS, Concurrent Versioning Software.
.110/2 0/20101
Page 13 of
13
RedstonIIntIIIrted Products A Solution I I-
PageIIII13ofIII13-
Working Foders/Files
EIE i
Stored Files/Builds/Releases Configuration Management Software
BuLL
Process
InstaiLation
Provess
Install Kit
Buld
Redstone Integrated Solutions, 2003. All Rights Reserved.
http://web.archive.org/web/20040 6 06014343/redstone-is.com/products_services.html
10/20/201.0
TAB F
Redstone Integrated Solutions - Camera Monitoring with VisualPursuit
Page 1 of 2
Home Products
Services Portfolio
News-
VisualPursuit
Contact Us
VisualPursuit:,Camera Monitoring / Facility Security Systems Redstone's VisualPursuit enhances an operator's effectiveness by allowing the operator.to easily track a moving subject throughout a facility from a single monitor. It accomplishes this .by consolidating the video from physically adjacent surveillance cameras into one screen, increasing the field of view of a physical location dramatically. VisualPursuit is a part of the Redstone family 'of software and when used with the full system, adds subject-tracking and wide-area monitoring capabilities'to an operator's toolkit. VisualPursuit's display combines the output from several surveillance cameras into a single, MS Windows-standard screen. * * * * The VisualPursuit screen contains a Subject View and Locator Views. The Subject View displays the current location of the moving subject. Up to 8 Locator Views display the output of the cameras closest to the one trained on the subject. Track a subject by clicking the Locator View that the person has moved into. The Locator View becomes the new Subject View, and all Locator Views are automatically reassigned and updated.
VisualPursuit works with Redstone's MapTrack or ResourceTrack products: * Click a camera symbol on MapTrack or ResourceTrack to start a VisualPursuit session. * MapTrack's camera symbols are color-coded so that you always know what cameras are being used in VisualPursuit. * From MapTrack, monitor and control devices at the same time you're engaged in a VisualPursuit (lock doors, zoom cameras, etc). Track the subject's location and direction of movement on MapTrack's interface. VisualPursuit is an important element of Redstone's Intelligent Facilities package. VisualPursuit makes an operator more effective and efficient by allowing her to easily track a moving subject using one monitor. It also allows her to quickly scan an area covered by security cameras without having to view several monitors. Applications of VisualPursuit include commercial office buildings, cities, airports, hospitals, sports facilities, prisons, pipeline companies, refineries, and convention centers to name a few. Basically, any area that needs better surveillance and/or emergency management will benefit from the installation of Redstone's VisualPursuit.
http://web.archive.org/web/20041011132522/redstone-is.com/camera _monitoring.html
10/20/2010
Redstone Integrated Solutions - Camera Monitoring with VisualPursuit

VisualPursuit can be used to monitor and/or control:
Page 2 of 2
* Facilities: casinos, housing complexes, office buildings, convention centers, governmental buildings, prisons, corporate campuses, high-rises * Airports: access and security * Educational Campuses: * Complex. Infrastructures such as chemical plants, pipelines, manufacturing complexes, city intersections * Entertainment complexes: amusement parks, zoos, sports facilities, museums VisualPursuit in Action: * The picture to the right is a screenshot of the Visual Pursuit interface. This view shows nine fI--i cameras. The large, central window contains the output from the current main tracking tracking camera are displayed in the other eight camera; other cameras in proximity to the current windows. By clicking on one of these adjacent views, you can change it to be VisualPursuit's main camera view. The adjacent views would then automatically update based on the new current tracking camera. This is an example of how you would use it to follow a person through a facility. * A series of LEDs give you an idea of where the current tracking camera is, in relation to . another camera. * There is also a map layout that can assist you with visualizing exactly where each camera is in your facility. << Click lHere for close up shot >>
-- .
J
Back to Products
I I
C Redstone Integrated Solutions, 2004. All Rights Reserved.
http://web.archive.org/web/2004 1011132522/redstone-is.com/camera_monitoring.html
10/20/2010
TAB G
Redstone Integrated Solutions - Building Automation with MapTrack
Page 1 of 2
Home Products
Services
MapTrack .
Portfolio
News
Contact Us
MapTrack: Building Automation Redstone's MapTrack is an easy-to-use, graphical interface. Users can quickly locate, view, and/or control computer systemns, video, security cameras, audio, card access readers, doors, and a host of other devices, all from a real-time, interactive map interface. MapTrack can display maps or diagrams at the level of detail required by the user. MapTrack is a part of the Redstone family of software products and when used with the full system, adds digital, interactive map and device control capabilities to an.operator's toolkit. MapTrack's display combines digital maps or diagrams with graphical symbols that can represent the locations of items such as: * * * * * * * security, cameras, doors, and windows computers/components in an equipment rack valves on a pipeline traffic and railroad signals alarms (motion detectors, fire alarms, etc.) steps in a process diagram and many other devices
These graphical symbols are more than simple static representations, they can: * . * * display a device's status (alarming, locked/unlocked, on/off) by a change in color or by flashing allow device interaction and control allow viewing of device output give the operator a quick entry into another Redstone module such as VisualPursuit.
MapTrack is an important element of Redstone's Intelligent Facilities package. MapTrack makes . an operator more effective and efficient by using a single, real-time, map-based interface through . -which an operator can monitor and respond to events' over an entire facility - all from-one or many workstations. The look and feel of this interface can be customized to suit the needs of the client. .Cl ck Here for. closeup.shot.." Applications of MapTrack include commercial office buildings, cities, airports, network operations centers, hospitals, sports facilities, prisons, pipeline companies, refineries, and convention centers to name a few. Basically, any area that needs better facilities management, surveillance, and/or emergency management will benefit from the installation of Redstone's MapTrack.
http://web. archive.org/web/2004 1011131 044/redstone-is.com/building_automation.html
10/20/2010
Redstone Integrated Solutions - Building Automation.with MapTrack

MapTrack can be used to monitor and/or control: * * * *
Page 2'of 2
Facilities: access and security, climate control, emergency systems, lighting Refineries and chemical plants: security, emergency response, product delivery, product status Pipelines: emergency response, security, product delivery, product status, cathodic protection Networks and hardware: system status, hardware status, connectivity
Redstone's software allows for remote monitoring of numerous buildings, complexes, and parking structures by the property owner for the purpose of reducing liability, reducing operating costs, increasing safety and security, reducing loss of assets, and increasing the owner's capability to proactively or quickly respond to a variety of incidents, ranging from simple facility maintenance issues to criminal activity. It also offers the option to hand-off control of subsystems to emergency response teams during an event (if so equipped.) < Click here for a.close up shot > Back to Products
Redstone Integrated Solutions, 2004. All Rights Reserved.
http://web.archive.org/web/20041 01 I1131044/redstone-is.com/building_automation.html
10/20/2010

In re U.S. Patent No.: 7,777,783 Inventors: Hu Chin et al. Issue Date: August 17, 2010 Serial No.: 11/726,879 Filing Date: March 23, 2007 Attorney Dkt. No.: 017874-0003
For: MULTI-VIDEO NAVIGATION
Mail Stop Inter Partes Reexam Director of the U.S. Patent and Trademark Office P.O. Box 1450 Alexandria, VA 22313-1450 REQUEST FOR INTER PARTES REEXAMINATION OF U.S. PATENT NO. 7,777,783 PURSUANT TO 35 U.S.C. 311 AND 37 C.F.R. 1.913 Dear Sir: Inter Partes Reexamination under 35 U.S.C. 311 et seq. and 37 C.F.R 1.913 et seq. is requested for United States Patent Number 7,777,783 (the '783 patent) which issued on.August 17, 2010 to Chin et al. from an application filed March 23,'2007. This Request is accompanied by the fee for requesting inter partes reexamination set forth in 37 C.F.R. 1.20(c)(2). The United States Patent and Trademark Office is hereby authorized to charge any fee deficiency,, or credit any overpayment, to our Deposit Account No. 03-2469.
DC' 0/YHA)IA
D1/436261.1
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 TABLE OF CONTENTS
i.
II. III. IV. V.
STATEMENT REGARDING PENDING LITIGATION............................. ..... 1 INTRO DUCT IO N......................................................................... ... 1,
PROSECUTION OF THE APPLICATION FOR THE '783 PATENT...............6 CITATION OF PRIOR ART PURSUANT TO 37 C.F.R. 1.915(b)(2). ............. 17 STATEMENT OF SUBSTANTIAL NEW QUESTIONS OF PATENTABILITY PURSUANT TO 37 C.F.R. 1.915(b)(3). AND DETAILED EXPLANATION...................... ...... ............ A.
....
19
Proposed 35 U.S.C. 102 Rejections...........................................19 1.

2.
Girgensohn '706 Reference .

W hite Reference............ ...... ..................
..............
.................... ....
19
24
3. 4. B.
Roberts Reference ............... Redstone Reference.............
........... ..... ..........
.......... ..
.28 .............. 33
Proposed 35 U.S.C. 103 Rejections............................................36 1. Girgensohn '706 Reference ......................................... a. b. 2. 37
Girgensohn '706 Reference in combination with Lin and Girgensohn '978 References......................37 Girgensohn '706 Reference in combination with Gormley Reference ........ .. ............. 42 44
Roberts Reference ................. a.
........................
Roberts Reference in combination with Girgensohn '706 and Gormley ..................... . References..............
................. 44
DC(.'1/YOllAD/436261.1
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 b. 3. Roberts Reference in combination with Girgensohn '978 and Lin References........................47 .................... 51
Redstone Reference .......................... a.
Redstone Reference in combination with Balancing Tomorrow's Technology, Redstone Integrated Solutions reference ...................
............. 51
b. c. 4.
Redstone References in combination with Girgensohn '706 and Gormley References.....................53 Redstone References in combination with Girgensohn '978 and Lin References ............... ......... 55
White Reference......................... ................... 58 a. White Reference in combination with Girgensohn '706 and Gormley References............................................. White Reference in combination with Girgensohn '978 and Lin References ................ .....
....
.58
b. 5.
.......... 60
Maruya Reference ........ a. b.
..................... :...................63
,Maruya Reference in.combination with Girgensohn '706 and Gormley References.... .............. 63 Maruya Reference in combination with White, Girgensohn '978 and Lin References .......... ... . . ........... Maruya Reference-in combination with Roberts, Girgensohn '978 and Lin References .......... ............. Maruya Reference in combination with Redstone, Girgensohn '978 and Lin References................
.... ...... 67
c.
... 69
d.
.............. 71
S.'01/YO ID/4 3.6261 .1
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 Buehler Reference ......................................... a. b. Buehler Reference in combination with Girgensohn '706 Reference............................ .. 73
6.
..... 73
Buehler Reference in combination with White, Girgensohn '978 ... and Lin R eferences ................................ ................. 77 Buehler Reference in combination with Roberts, Girgensohn '978 and Lin References .........
c.
.................. 80 ................... 83
V I. VII.
CO N CLU S IO N...................................................... .
LIST OF EXHIBITS AND APPENDICES..................................................83
DC01/YOIIA D/436261.1
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7;777,783
1.
STATEMENT REGARDING PENDING LITIGATION The '783 patent is currently the subject of litigation in the United States.District
Court for the Eastern District of Virginia. The case is styled: VidSys, Inc., Plaintiff v. Proximex Corporation, Defendant, United States District Court for the Eastern District of Virginia, Civil Action No. 1:10cv1185. The reexamination Requester, VidSys, Inc., is the plaintiff in this litigation, and the patent owner, Proximex Corporation, is the defendant. The subject litigation is in its very first stage in that the Complaint has been filed with the Court, however the Complaint has not yet been served and the defendant has not responded to the Complaint. It is believed that the '783 patent has not been adjudged invalid or unenforceable at this time and thus remains in force.
II.
INTRODUCTION The '783 patent relates to a system which includes a plurality of video cameras
and a video tracking interface adapted to display video streams provided by the cameras. The system may be used for surveillance of an area using multiple cameras which have distinct fields of view relative to each other. The primary focus of .the patent is on the video tracking interface and its arrangement and functionality. The video tracking interface includes user controls associated with each of the cameras. These user controls may be activated by "clicking" on them with a mouse or touching them, for example, to select an associated camera to provide a video stream in
I)C01/YOI1A D/436261.1
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 a main video display. See '783 patent, 7:40-43.) In the Figure 3 embodiment of the '783 patent the user controls 310A-310H are provided around, i.e., at a position dutside, and along a perimeter, of the main video display 305. See '783 patent, Fig. 3. In some embodiments, the user controls may include an optional thumbnail of video data generated using the associated camera. See id., 8:28-33. Figures 4A, 4B and 4C
illustrate alternative arrangements of the user.controls on the video tracking interface screen wherein the main video display itself may have user control regions within it which may be clicked upon to select. a new camera to provide the video stream for the main video display. See id., 7:43-63 and Figs. 4A-4C. In the Figure 4 embodiments, the user controls are not provided around the main video display, but are instead contained within the main, video display. The arrangement of the user controls on the video tracking interface may be representative of the topological relationship of the cameras associated with the main video display and the user controls. See id., 3:51-4:24. The representative topological relationships may be derived from GPS locations, for example, taken from the viewpoint of the camera which is currently providing the video stream for the main video display. If a user control is selected so that a new camera begins providing the video stream for the main video display, the system may re-associate each of the user controls with one or more of the other cameras so that the arrangement of the user controls is
1 The nomenclature of "X:Y" is used to refer to column number "X" and line number "Y" in U.S. patent prior art references.
I.)CO1/YOHIAD/436261 .
Attorney Dkt, No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 representative of the topological relationship of the cameras taken from the viewpoint of the new camera. See id., 7:3-20. The re-association' process may be repeated each time a new camera is selected to provide the video stream for the main video display. The re-association process may result in some of the user controls being unused if no camera,has a topological relationship with the "new" camera that corresponds to the location of those particular User controls on the screen. See id., 6:43-60. Because not all user controls will necessarily be required after "re-association" of the user controls with the cameras, it follows that active user controls may not always be provided along each of the four sides of the main video display after re-association. In fact, one of ordinary skill in the art would readily understand that there may be a "reassociation" that results in as few as a single active user control being provided on only one of the four sides of the main video display. With reference to Figure 3, the '783 patent explains the usefulness of the, foregoing functionality as follows: In various embodiments, the association of User Controls 310 with members of Cameras 105 based on camera topology results in an intuitive interface for tracking of targets between video camera. For example, if the target moves to the right out of the field of view presented 'in Main Video Display 305, then User Control 310B can be used to select the member of Cameras 105 whose field of viewthe target-is. most likely to be found next. As the target moves from camera to camera, a user can select those members of User Controls 310 adjacent to the edge of Main Video Display 305 at which the target disappeared. '783 patent, 6:6-16.
DC1 /YOI IAD)/436261. I
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783' Recorded segments of video streams may also be automatically stitched or spliced together by the system to form a continuous sequence of video data taken from different cameras. See id., 9:27-42. The stitched together sequence of video may be used to track a target as it moves through the fields of view of the different cameras that capture video of the target. The automatic stitching together of video streams may further involve use of a preset "temporal offset." The '783 patent describes the use of this temporal offset as follows: The synchronization between video. data optionally includes an offset selected to approximate the time a target would be expected to travel between members of Cameras 105. For example, if a typical travel time between the fields of view of Cameras 105G and 105D is 15 seconds, then a 15 second offset is optionally automatically used when synchronizing video data from Camera 105G and Camera 105D. If a user is viewing video data from Camera 105G in Main Video Display 305, and uses User Control 310B to select video data generated by Camera 105D [starts by viewing video data generated- using Camera 105D], then the video from Camera 105D will be offset by 15 seconds. This offset may enable the user to see the target immediately in the field of view of Camera 105D without having to wait for the target to travel between fields of view. '783 patent, 9:5-19. The system may also convert video data from one format to another as needed, store video data in association with information that identifies. the camera used to generate the video data, and utilize on-screen "thumbnail" views to provide an index of stitched video. The points of novelty expressed in independent Claims 1 and 5 of the '783 patent - namely, placement of the above-described user controls around a main video display . 4 DC1/YIOHAD/436261.1
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 in locations representative of the topological relationship of the cameras providing the video, and use of an automatic "temporal offset" when stitching together video from different cameras - were known in the art prior to the filing of the application for the '783 patent. In fact, theformer point of novelty, concerning the placement of user controls, was known more than seven years before the filing date of the application for the '783 patent, as will be explained below. This Request for inter partes reexamination involves ten (10) different prior art references. Only one (1) of the prior art references relied upon for this Request was cited during the prosecution of the application for the '783 patent. Four of these
references anticipate Claim 1 of the '708 patent under 35 U.S.C. 102 and one of these references anticipates all of the claims. None of these anticipatory references was Furthermore,
before the Office during examination of the '783 patent application.
combinations of the ten different.prior art references render Claims 1-8 obvious under 35 U.S.C. 103 in a number of different combinations. Accordingly, the submitted references raise substantial new questions of patentability. In view of the foregoing, and pursuant to 37 C.F.R. 1.915(b)(1), inter partes reexamination is hereby requested for Claims 1-8 of the '783 patent. Pursuant to 37 C.F.R. 1.915(b)(8), the real party in interest is VidSys, Inc., a Delaware corporation', having a place of business at 8219 Leesburg Pike, Suite 250, Vienna, VA 22182 ("Requester"). Pursuant to 37 C.F.R. i.915(b)(7), Requester hereby certifies.that the estoppel provisions of 37 C.F.R. 1.907 do not prohibit this inter partes reexamination.
1C01 /Y01.A D/436261.1
5.
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 Pursuant to 37 C.F.R. 1.915(b)(5), a copy of the entire '783 patent is attached hereto as Exhibit A, including the front face, drawings, and specification/claims (in double column format).
III.
PROSECUTION OF THE APPLICATION FOR THE '783 PATENT A. Original Claims 1-27
The '783 patent issued on August 17, 2010, based on U.S. Application Serial Number 11/726,879 (the '879 application) filed on March 23, 2007, by purported inventors Hu Chin and Ken Prayoon Cheng. Thus, March 23, 2007 constitutes the priority date against which prior art is measured (the "critical date"). The '783 patent is entitled "Multi-Video Navigation." As originally filed, the application for the '783 patent included twenty-seven (27) claims, of which Claims 1, 7 and 24 were independent. Independent Claims 1, 7 and 24, as originally drafted, claimed: 1. A method comprising: identifying a first member of a plurality of cameras; identifying a first topological relationship between the first member of the plurality of cameras and a second member of the plurality of cameras; identifying a second topological relationship between the first member of the plurality of cameras and a third member of the plurality of cameras, the first. topological relationship being relative to a viewpoint of the first camera;, presenting video data. generated using the first member of the plurality of cameras in a user interface, the user interface
DCOI/Yo AI)/4362 I .1
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783
including a first user control and a second user control;. associating the second member of the plurality of cameras with the first user control based on the topological relationship between the first member of the plurality of cameras and the second member of the plurality of cameras; associating the third member of the plurality of cameras with the second user control based on the topological relationship between .the first member of the plurality of cameras and third member of the plurality of cameras; receiving a user selection of the first user control; presenting video data generated using the second member of the plurality of cameras in the user interface based on the received user selection; and associating the first user control and the second user control with members of the plurality of cameras based on topological relationships relative to a viewpoint of the second member of the plurality of cameras. 7. A method comprising: viewing first video data within a user interface, the first video data generated using a first member of a plurality of, cameras; locating a target within the first video-data; observing the target leave a field of view of the first member of the plurality of cameras in a first direction relative to a viewpoint of the 'first member of a plurality of cameras, or observing the target enter the field of view of the first member of the plurality of cameras from the first direction relative to a viewpoint of the first member of the plurality'of cameras; selecting a second member of the plurality of cameras using a first user control associated with the first direction. relative, to a viewpoint of the first member of the plurality of cameras; and viewing second video data within the user interface, the
DC O/YOH A v/4 6261.1
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 second video data generated using the second member of the plurality of cameras. 24. A method comprising: receiving a user selection of first video data generated using a first camera, the first video data including a first time reference; receiving a user selection of second video data generated using a second camera, the second video data including a second time reference; automatically stitching the first video data to the second video data responsive to the user selection of the, second video data and using the first time reference and the second time reference to form a chronological video sequence; and creating an index to the video sequence based on identities of the first camera and the second camera. See March 23, 2007 Patent Appl. No. 11/726,879, at 34-35. The following discussion of the prosecution history of the claims focuses particularly on prosecution of Claims 1-3, 7, 16 and 18. Claims. 1-3 are of interest
because the subject matter of these claims was eventually consolidated in order to gain allowance of issued Claim 1. Claim 18 is also of particular interest, as the subject matter of Claim 18 was consolidated with that of base and intervening Claims 7 and 16 in order to gain allowance of the other independent issued claim (Claim 5). Many of the other claims in the application were ultimately abandoned.
DCOI/Y ()o IA /4362 6 1.
Attorney Dkt. No..017874-0003 Reexamination Request U.S Patent No. 7,777,783 Original claims 2 and 3 were dependent on claim 1, and claimed: 2. The method of claim 1; further including presenting the first user control and the second user control in the user interface in positions representative of the first topological relationship and the second topological relationship. 3. The method of claim 2, where in the first user control and the second user control are presented in positions around a presentation of the video data generated using they first member of the plurality of cameras. Original claims 16 and 18 were dependent on claim 7, and claimed: 16. The method of claim 7, further including automatically stitching the first video data and the second video data to form a video sequence. 18. The method of claim 16, further including using a temporal offset to automatically stitch the first video data and the second video data, the temporal offset being based on an estimated time of travel of the target. The First Rejection and Response
B.
In the first Office Action, the Examiner rejected all of the claims as being obvious either over U.S. Pub. No. 2007/0146484A1 to Horton et at. ("Horton") alone, Horton in view of U.S. Pub. No. 2002/0097322A1 to Monroe et al. ("Monroe"), or Horton in view of US Pub. No. 2008/0068464A1 to Kitagawa et al. ("Kitigawa").. See May 20, 2009 Office Action ("First Office Action"),.passim. The applicants did not amend any of the rejected claims in response to the First Office Action. Instead, the applicants presented only remarks in an attempt to traverse the rejections. Notably, with respect to Claim 1, the applicants contended that Horton 9 DCOI/YO1IAD/436261.1
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 did not teach the "step of identify[ing] a topological relationship." Amendment A ("First Remarks"), at 12-13. .June 8, 2009
The applicants attempted to distinguish
Horton by arguing that "a relationship between a camera and a target, as in Horton et al., cannot read on a topological relationship between two cameras." Id. at 13. The applicants also contended that "Horton et al. does not teach 'selecting a member of the plurality of cameras using a first user control associated with the first direction relative to a viewpoint of the first member of the plurality of cameras.'" Id. at 13. C. The Second Rejection and Response
In a second Office Action, the Examiner again rejected all of the pending claims. U.S. Pub. No. 2003/0202102A1 to Shiota et al. ("Shiota") replaced Horton as the primary reference upon which the rejections were based. In the Second Office Action, both anticipation and obviousness were asserted using Shiota alone, or in various combinations with Monroe, U.S. Pub. No. 2004/0001149A1 to Smith ("Smith"), U.S. Patent No. 7,456,727 to Pinter et al. ("Pinter"), U.S. Patent No. 7,525,576 to Kannermark. et al. ("Kannermark"), and U.S. Pub. No. 2006/0221184A1 to Vallone et al. ("Vallone"). Sept. 21, 2009 Office Action ("Second Office Action"), at 16. With respect to Claim 1, the Examiner concluded that Shiota disclosed each element of the claim except the requirement "that the user interface includes the first and second user controls for controlling each individual camera." Id. at 14-16. For the same reasons as set forth in the First Office Action, the Examiner again concluded that Monroe filled this gap, rendering Claim 1 obvious. Id. at 16.
D)C0I/YOIIAI)/436261.1
10
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 The applicants responded.to the Second Office Action by again only submitting remarks and not amending any of the claims other than to correct for grammatical errors. The applicants argued that Shiota did not teach or suggest identifying
topological relationships between cameras as required by Claim 1. Dec. 21, 2009 Amendment B ("Second Remarks"), at 17. The applicants agreed that Shiota disclosed "cameras pointed in different directions and that spatial relationships exist between them." Id. According to the applicants, however, "nowhere in Shiota et al. are such
relationships noted or utilized" and they were therefore "irrelevant to the operation of the system of Shiota et al." Id. As an example, the applicants argued that Shiota "does not identify the angle between the viewpoints of any two cameras, and does not use such information to direct the cameras or process images." Id. at 17-18. The applicants
argued that Monroe was irrelevant for the same reason. Id. at 18. The applicants also argued (as they did with respect to Horton in response to the First Office Action) that Shiota "does not teach or suggest associating cameras with user controls based on topological relationships." Id. Specifically, the applicants
argued that "[i]n Shiota et al. there is a plurality of 'display blocks' in the display, one for each of a plurality of cameras, and then one of two sequences is repeated as in FIGs. 4A and 4B." Id. "Either each display block is dedicated to a particular monitored area (FIG. 4B), else each display block is dedicated to a particular camera (FIG. 4A)" Id. According to the applicants, "[t]here is no correlation between the arrangement of the display blocks and the relationships between cameras" and "[f]urther, the display blocks
I')COI/YCII D/43626 .1
,, .
11
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 aren't even user controls." Id. The applicants did not address the Second Office Action's rejection of claim 2 based on the combination of Shiota and Monroe. They simply, asserted that since
Shiota and Monroe did not render claim 1 obvious, claim 2 was not obvious either. lId. at 18-19. The applicants did, however, respond to the rejection of claim 3, as follows: The Applicants would also like to point out the further patentability of claim[] 3 .... Claim 3 requires that, in the method of claim 2, the first user control and the second user control are presented in positions around a presentation of the video data generated using the first member of the plurality of cameras. The Examiner merely cites to FIGs. 3 and 5 of Monroe et al. ([Second] Office action, page 17). Both drawings show a map display area and a video display area, and the two areas are side by side. ,The Applicants assert that neither drawing of Monroe et al. shows user controls presented in positions around a presentation of video data. Id. at 19 (emphasis added).
D.
The Final Rejection and Response In a third and final Office Action, the Examiner rejected the applicants' arguments
regarding patentability.
With respect to Claim 1, the Examiner first noted that the
applicants' arguments were based on a construction of claim 1 that was not supported by the claim language: There is no mentioning in the claimed language of identifying the angle between the viewpoints of any two cameras. The claims simply require identifying a topological relationship between the cameras and this is clearly fulfilled by the 12 I)('OI/YOHAD)/436261.1
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 teaching of Shiota et al.. The Examiner would like to point out that the Applicant agrees that in Shiota et al. there are cameras pointed, in different directions and that spatial relationships exist between them. The Examiner also likes. to point out that although the Applicant states that nowhere in Shiota et al: are such relationships noted or utilized, the Applicant admits that such relationships are inherently present. It is for this reason that the Examiner maintains that the topological relationship between the cameras. exists. There are commonly identified areas of interest present between the cameras. March 9, 2010 Final Office Action ("Final Office Action"), at 7-8. For the same reasons as those set forth in the Second Office Action, the Examiner again rejected applicants' arguments that the combination of Shiota and Monroe "does not teach 'or suggest associating cameras with user controls based on topological relationships." Id. at 8. With respect to Claim 3, the Examiner also disagreed with the applicants' assertion that Monroe did not disclose "user controls presented in positions around a presentation of video data." Id. at 9. The Examiner cited the Princeton online
dictionary's definition of the term "around" in concluding that it means "about or in-the area or vicinity." Id. Under this definition of "around," "it is clear that Monroe et al. teach this limitation." Id. The Examiner then repeated the rejections of Claims 1-3 based on Shiota in view of Monroe that were presented in the Second Office Action. See Final Office Action at 22-24.
I)('01/YOlAD/43621.1
13
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 Interview and Subsequent Remarks By Applicants
E.
After the Final Office Action, an interview was held between applicants' counsel and the Examiner. See April 6, 2010 Interview. Summary. During the interview,
agreement was reached with the Examiner that "[r]egarding independent claim 1, Shiota et al. did not teach identifying a first topological relationship between the first and second cameras of the plurality of cameras." Id. Claims 2-3 were not discussed during the interview. April 8, 2010 Amendment C ("Third Remarks"), at 8. Applicants' Third Remarks requested allowance of claims 1, 2 and 3 since claim 1 was allowable: In view of the April 6 interview, claim 1 is allowable. Accordingly,. the Applicants request that the Examiner withdraw the rejection of claim 1, and the rejections of claims 2-6 depending therefrom, under 35 U.S.C. 103(a). Id. at 10. F. Subsequent Office Action, Amendment and Notice of Allowance
.In response to Applicants' Third Remarks, the Examiner issued a non-final Office Action. In this Office Action, the Examiner indicated that he never agreed during the interview that Claim 1 contained allowable subject matter. See May 14, 2010 Office
Action ("May 2010 Office Action"), at 3. In this Office Action, the Examiner rejected original Claims 1, 2, 7 and 16, among others, on new grounds, i.e., as being anticipated by U.S. Pub. No. 2010002082A1 to Buehler et al. ("Buehler"), or obvious over Buehler in view of U.S. Pub. No. 2005/0052532A1 to Elooz et al. ("Elooz").
CI)oYOoHAr/43261.I . 14
Id. at 4-6, 14.
Attorney Dkt: No. 017874-0003
Reexamination Request U.S Patent No. 7,777,783 Claims 3 and 18, however, were objected to as being dependent on a rejected base or intervening claim(s), but not rejected. Claim 2 was rejected as obvious over Buehler in view of Monroe Id. at 11.
Here, the Examiner found that Buehler disclosed all elements of Claim 1, however it did not disclose "that the user controls are physically in positions representative of the first topological relationship and the second topological relationship." As expressed in the First Office Action, however, Monroe filled this gap. Id. at 11-12. With respect to claim 3, the Examiner noted that it was "objected to.as' being dependent upon a rejected base claim', but would be allowable if rewritten in' independent form including all of the limitations of the base claim and any intervening claims." Id. at 22., Claim 3 added the limitation that "the first user control and the second user control are presented in positions around a presentation of the video data generated using the first member of the plurality of cameras." See Third Remarks, at 3 (emphasis added). With regard to Claim 16, the Examiner explained that Buehler taught all, elements of the claim except for "automatically stitching the first and second video data." Id. at 14-15. Elooz was asserted to teach automatically combining portions of videos, and it would have been obvious to combine the automatic stitching of Elooz with the teachings of Buehler. Id. at 16. Like with Claim 3,."with respect to claim 18, the Examiner noted that it was "objected to as being dependent upon a rejected base claim, but would be allowable if
D(:01 /YOl IA I)/436261.1
15
Attorney Dkt, No. 017874-.0003 Reexamination Request U.S Patent No. 7,777,783 rewritten in independent form including all of the limitations of the base claim and any intervening claims." Id. at 22. Claim 18 added the limitation of "using a temporal offset to automatically stitch the first video data and the second video data, the temporal offset being based on an estimated time of travel of the tarqet." (emphasis added). In response to the May 2010 Office Action, the applicants cancelled Claims 2, 3, 7 and 18, among others. See May 24, 2010 Amendment D ("Fourth Remarks"), at 3-4. Simultaneously, the applicants amended Claim 1 to incorporate the limitations of original Claims 2 and 3, and amendedClaim,16 to incorporate the limitations of original Claims 7 and 18., Id. at 2-3, 6.. The applicants did not traverse the rejection of any of the claims other than to state that the "[n]on-allowed claims that are cancelled herein are cancelled merely to expedite prosecution and their cancellation should not be construed as.an admission of unpatentability." Id. at 6. The Examiner subsequently issued a Notice of Allowance dated June 18, 2010 ("Notice of Allowance"). In the Notice of Allowance, the Examiner indicated that the only reason for issuance was the combination of all elements of original Claim 1 With the elements of original Claims 2 and 3,.and the combination of all elements of original Claim 16 with the elements of original Claims 7 and 18. Notice of Allowance, at 2-3. The addition of the language from dependent Claims 3 and 18 to the allowed claims was highlighted in bold text, indicating that this language was considered necessary for Buehler was cited, as the closest prior art to all issued claims. I)C01/YOlIA)/436261.I 16 allowance. Id. at 5. See Third Remarks, at 6
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 According to the Examiner, Buehler disclosed "methods and systems for creating video from multiple sources to utilize intelligence to designate the most relevant sources, facilitating their adjacent display and/or catenation of their video streams." Id. IV. CITATION OF PRIOR ART PURSUANT TO 37 C.F.R. 1.915(b)(2) Based on the critical date of the '783 patent,, each of the prior art references listed in the chart below qualifies as prior art under 35 U.S.C. 102(a), 102(b) and/or 102(e). Pursuant to 37 C.F.R. 1.915(b)(4), a copy of each patent or printed
publication relied upon or referred to is provided with this submission as Exhibits B-K. The chart below also identifies the short cite for each reference, the statutory basis under' which the reference is applied, and the claims to which the reference is applied individually or in combination with other references. Exhibit Reference U.S. Patent Appl. Pub. No. 2008/0088706 Al to Girgensohn et al. filed Oct. 13, 2006 and published Apr. 171 2008 U.S. Patent No. 7,196,722 to C White et al.issued Mar. 27, 2007 and filed May 18, 2001 U.S. Patent No. 7,295,228 to Roberts et al. issued Nov. 13, 2007 and filed Oct. 29, 2002 White Reference 102(e), 103(a) 1-8 SStatutory Basis Short Cite Girgensohn '706 Reference Claim(s) Applied To
102(e),
1-8
Roberts Reference
I_
102(e), 103(a)
1-8
. _
DCO1/YOA I)/436261 .1
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No: 7,777,783
Exhibit
Reference
Short Cite Redstone Products_ Services Reference Lin Reference Girgensohn '978 Reference
Statutory Basis 102(b), 103(a)
Claim(s) Applied To 1-8
http://web.archive.org/web/20040 606014343/redstoneis.com/products_services.html U.S. Patent No. 7,242,423 to Lin F issued Jul. 10, 2007 and filed Aug. 8, 2003 U.S. Patent Appl. Pub. No. 2006/0284978 Al to Girgensohn et al. filed Jan. 3, 2006 and published Dec. 21, 2006 U.S. Patent No. 5,258,837 to Gormley issued Nov. 2, 1993 and filed Oct. 19, 1992 Balancing Tomorrow's Technology, Redstone Integrated Solutions, @ 2003 J U.S. Patent No. 7,746,380 to Maruya et al. issued Jun. 29, 2010 and filed Jun. 15, 2004 U.S. Patent Appl. Pub. No. 2010/0002082 Al to Buehler et al. filed Mar. 24, 2006 and published Jan. 7, 2010"
102 (e) 103(a)
58
102 (a)
102 (e) 103(a) 3-8
Reference Reference Secondary Redstone Reference Maruya Maruya Reference
102 (b) 103(a)
102 (b)
103(a)
1-8 1-8
102 (e)
102 (e) 103(a)
Refrence Refrence J
1-8 103(a)
1'.
I)C01/YOI IAD/43626 I. I
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 V. STATEMENT OF SUBSTANTIAL NEW QUESTIONS OF PATENTABILITY PURSUANT TO 37 C.F.R. 1.915(b)(3) AND DETAILED EXPLANATION A. Proposed 35 U.S.C. .102 Rejections 1. Girgensohn '706 Reference
Claims 1-8 of the '783 patent are anticipated by U.S. Patent Appi. Pub. No. 2008/0088706 Al to Girgensohn et al. (the "Girgensohn '706" reference) (Exhibit B). The Girgensohn '706 patent application was filed on October 13, 2006. As a result, the Girgensohn '706 reference qualifies as prior art against the '783 patent under 35 U.S.C. 102(e). A detailed comparison of the elements and limitations of Claims 1-8 of the '783 patent with the disclosure of the Girgensohn '706 reference is provided in Appendix A, which is incorporated herein by reference. The Girgensohn '706 reference relates to a method of viewing video from multiple cameras using a computer interface. See Girgensohn '706, Abstract. In
Figures 3, 6 and 7 of Girgensohn '706, the interface may display a main video stream in a large central window 310. The main video stream is captured by one of a plurality of cameras'selected to provide the main video stream. See Girgensohn '706, Abstract
and Para. 0035. With reference to the Girgensohn '706 figure below, "a group of cameras are chosen that are in close spatial proximity to the main camera," to provide the smaller thumbnail video streams 330, 350 and 360 that surround the main video stream 310. See Girgensohn '706, Para. 0036.
.COI/YOlIAI)/436261. I
19
310
z-
330
350
360
Girgensohn '706, Fig. 3. The placement of the thumbnail video streams 330, 350 and 360 around the main video stream 310 corresponds to, and is representative of, the topological relationship of the cameras providing each of the video streams. Namely; "[t]he
[thumbnail] video streams are placed around the main video stream .such that a person walking out of the field of view of a first camera will likely appear in the field of view of an adjacent second camera, where the video stream of the second camera in the video display is in the same direction from the first camera as the direction in which the person walked out of the field of view of the first camera." Girgensohn '706, Para. 0039.
D)C01/YOIIAD/436261.
20
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,78:3 The topological relationship of the source cameras is "calculated from the video streams' cameras physical locations relative to the main camera." Para. 0041. Further, a user can select a thumbnail camera view to be the main camera video by clicking on a video stream in the spatial video display. See Girgensohn '706, Para. 0063. Thus, the thumbnail video streams that are placed around the main video stream in Girgensohn '706 act as "user controls" because they may be clicked upon to select the thumbnail video stream to become the main video stream. It is also inherent that once a new video stream is selected to be the main video stream, "[t]he [surrounding thumbnail] video streams are-[again] placed around the main video stream such that a person walking.out of the field of view of a first 'camera will likely appear in the field of view of an adjacent second camera, where the video stream of the second camera in the video display is in the same direction from the first camera as the direction in which the person walked out of the field of,view of the first camera." See' Girgensohn '706, Para. 0039. If this were not the case,. the Girgensohn '706 Girgensohn '706,
system would not function properly to permit .a user to track a target from camera to camera repeatedly. The Girgensohn '706 system also stores, and provides format conversion of, video data. See Girgensohn '706, Para. 0070. Format conversion occurs given that the Girgensohn '706 cameras utilize JPEG which is a video compression technique that was well known in.the art prior to the critical date. Accordingly, format conversion occurs
DCOI/Yo AI/43,261.) .
21
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 for incoming video to be converted to JPEG. Identification information for each of the cameras is also maintained by the Girgensohn '706 system, which is evident from the fact that "the main video stream and the other video streams have colored borders to indicate the actual source cameras." Girgensohn:'706, Para. 0031. The Girgensohn '706 system stitches video data from different source cameras together. The system works with recorded video which may be captured as a result of a user tracking a target from camera view to camera view, as explained above. During playback, the system automatically updates the camera views that were chosen by security personnel when the video was recorded. See Girgensohn '706, Para. 0060 and 0070. The foregoing requires that the video streams-from different cameras be stitched together. The Girgensohn '706 system also uses thumbnail views as an index of stitched together video in that thumbnails are used to provide video that was taken in the past and future relative to the video displayed in a main, camera video stream. These past and future thumbnail views may be from any one of a number of cameras, which requires that the video have been stitched together from different source cameras. Video streams of these five cameras, including the .main camera video stream, will be displayed in four horizontal bands shown in FIG. 1, two above and two below the main ,camera video, stream display 110. The second band 125 and third band 130 show video streams from camera views that are between 0 and 30 seconds in the past and future, respectively, relative to the main camera video stream. The top band 120 and bottom band 135 show video streams from camera views that are between 30 seconds to 3 minutes in the past and future,. respectively,
DC01 /YOIIAD/:)43626 1.
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 relative to the main camera video stream. In embodiments, these ranges can be varied as desired. Girgensohn '706, Para. 0022. With respect to Claim 5, the Girgensohn '706 system also uses a temporal offset which is based on the travel time of a target between camera views. A video stream display 210 from the view of a main camera is selected to be the main video played on the video player. Controls of the video player are part of the video display (not shown). Two vertical bands, of video streams 220 and 230 are displayed on either side of the main video stream display 210. A first band 220 to the left of the,.video stream display' 210 shows three camera views that display video streams from five seconds prior to the time of the main video stream playing in display 210. A second, band 230 to the right of the video stream display 210 shows three camera views that display video. streams from five seconds after the time of the video stream playing in display 210. A three to ten second offset from the time of the main video stream display playinq performs best for tracking people walking from camera view to camera view. In embodiments, this offset can be varied as desired. The refresh rate of images in these video streams can be varied to produce effects ranging from almost fluid video playback to images that remain steady for seconds or even minutes. Girgensohn '706, Para. 0030 (emphasis added). In summary, the.Girgensohn '706 system provides the exact same functionality as described and claimed in Claims 1-8 of the '783 patent. Just as with the '783 claimed invention, "[u]sers [of the Girgensohn '706 system] such as security personnel can browse between video cameras by clicking on any of the video streams of the temporal and geographic displays described above, such that the main playing video stream is switched to the camera and playback position corresponding to the video stream selected.
I)C:OIv/YOIHAD/136261 .
.Using this technique, users can follow activity from camera view to
23
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 camera view." See Girgensohn '706, Para. 0060. Accordingly, Girgensohn '706 raises a substantial new question of patentability and anticipates Claims 1-8 of the '783 patent.
2.,
White Reference
Claim 1 of the '783 patent is anticipated by U.S. Patent No. 7,196,722 to White et al. (the "White" reference) (Exhibit C). The application for the White patent was filed on May 18, 2001 and the patent issued on March 27, 2007. The White reference qualifies as prior art against the '783 patent under 35 U.S.C. 102(e). A detailed comparison of the elements and limitations of Claim 1 of the '783 patent with the disclosure of the White reference is provided in Appendix B, which is incorporated herein by reference. The White reference discloses a system for capturing multiple images from multiple cameras and selectively presenting desired views to a user on a display or user interface. Multiple streams of video data are streamed to the user interface. One type of data stream (called a thumbnail stream) is used to tell the user what video image streams are available. For this type of video stream, each image is transmitted as a low resolution thumbnail. One thumbnail is transmitted for each camera and the thumbnails are presented as small images on the users screen. Another data stream (called the focus stream) contains a series of high resolution images from a selected camera. The images transmitted for the focus stream is displayed in a relatively large area, in the center of the user's display screen. See White, 2:3-16 and Fig. 9.
DC('1/YO(HAD/43626 .1
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 In Figure 9 of White, six thumbnail images are presented on the display screen which correspond to six different camera views. Each of these thumbnails is presented in a location on the display which is representative of the location of the cameras providing thumbnail streams relative to the viewpoint of the camera providing the focus stream.
Top Camera Panoramic Image Thumbnail 901 Bottom Camera
Thumbnail Left side Camera'

Thumbnail
Thumbnail Right side Camera

Thumbnail
Front .
Back
Camera
Thumbnail
Focus
Image
Camera
Thumbnail
White, Fig. 9. The White reference explains this functionality as follows. The six images captured by the camera are: a top, a bottom, a left side, a right side, a front and a;back images (i.e. there is a lens on each side of a cube):. These, images can be seamed. into a panorama in accordance with the prior art and stored in a format such as an equirectangular or cubic format. With this alternative embodiment, the user sees a display such as that illustrated in FIG. 9. At the top center of the display is a thumbnail 901 of a panorama. The panoramic image is formed by seaming fs].together into one panoramic image, the individual images from the six cameras. Six thumbnails of images from the cameras (the top, bottom, left side, right side, front and back of the cube) are shown alonq the right and left edges of the display. If a user clicks on any one of --- ~~~~~~~~~~ --~the six thumbnails, on the right and left of the screen, the focus stream [Ls]
DC01 YOIHAD/436260 1.1
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 switched to that image stream as in the first embodiment. It is noted that with a panoramic image, it is usual for a viewer to select a view window and then see the particular part of the panorama which is in the selected view window. If the user clicks anywhere in the panorama 901, the focus stream is changed to a view window into the panorama which is centered at the point where the user clicked. With this embodiment, stream control has as one input a panoramic image and the stream control selects a view window from the. panorama which, is dependent upon where the user clicks on the thumbnail of the panorama. The image from this view window is then streamed to the user as the focus image. White, 7:38-64 (emphasis added). It is clear from the foregoing that the White thumbnail streams are placed "around" the central focus stream and are. placed in locations that represent the topological relationship of the cameras providing the thumbnail streams with the camera providing the focus stream. For example, Figure 9 shows that the thumbnail stream on the left of the central focus stream is associated with a "left side camera" and the thumbnail stream on the right of the central focus stream is associated with a "right side camera." Moreover, each of the thumbnail streams can be "clicked" upon, i.e., acts as
a "user control," to select that thumbnail stream to become the focus stream. Each time a new camera is selected to provide the central focus stream, by clicking on a thumbnail stream for example, the White system reconfigures the overall display so that surrounding thumbnail streams are presented in positions relative to the viewpoint of the camera providing the focus stream, which positions are representative of the topological relationships of the cameras. functionality as follows:
)(uO/Y1 eI243661.I
The White reference explains this
26
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 [A]t this point the client requests that the focus stream change. This is sent to the server as indicated by arrow 994 [Figure 12]. When the server receives the command, it stops streaming the old focus stream and starts streaming the new focus stream as indicated by arrow 995. A new layout for the user's display is also sent as indicated by -arrow 996: It is noted that a wide variety of circumstances could cause the server to send to the client a new layout for the users display screen. When the client receives the new display layout, the display is reconfigured. White, 10:5-15 (emphasis added). Based on the foregoing, when the left side camera thumbnail is selected by clicking on it (i.e., by clicking on the first user control), the left side camera thumbnail view becomes the new central focus stream. This also causes a new layout to be provided for the user's display. 'This new'layout is "reconfigured," indicating to one of ordinary skill in the art that the surrounding thumbnails are provided in positions relative to the viewpoint of the left side camera which is now providing the focus stream. The conclusion that any of the seven cameras used for the Figure 9 embodiment of White may provide the focus stream and thus the arrangement of the surrounding thumbnails is relative to the viewpoint of any of the seven cameras when selected to provide the focus stream is further supported by the disclosure that the "streaming would begin with a default camera view as the focus stream." See White, 4:66-67. The White reference discloses a camera system and display interface that provides all of the functionality recited in Claim 1 of the '783 patent. The White system arranges thumbnail streams, which act as user controls, around a central focus stream in positions which represent the topological arrangement of the cameras providing the
D(: 1YO I IAHD/436261.1
27
Attorney Dkt. No. 017874-0003 Reexamination Request U,S Patent No. 7,777,783 video streams. Each time a new camera is selected to provide the focus stream, the display is "reconfigured" so that the user control positions on the display (i.e., the thumbnail stream positions) are re-assigned to the cameras that have a corresponding topological relationship with the newly selected focus stream camera. Accordingly,
White raises a substantial new question of patentability and anticipates Claim 1 of the '783 patent.
3.
Roberts Reference
Claim 1 of the '783 patent is anticipated by U.S. Patent No. 7,295,228 to Roberts (the "Roberts" reference) (Exhibit D). The application for the patent which is the
Roberts reference was filed on October 29, 2002 and the patent issued on November 13, 2007. As a result, the Roberts reference qualifies as prior art against the '783
patent under 35 U.S.C. 102(e). A detailed comparison of the elements and limitations of Claim 1 of the '783 patent with the disclosure of the Roberts reference is provided in Appendix C, which is incorporated herein by reference. The Roberts reference discloses a method of operating a security camera system. See Roberts, 2:17-21. The security camera system comprises a plurality of video cameras and camera selection means which permit a user to observe a target of interest as it moves from the field of view of camera to camera. See Roberts, 1:29-35. Each of the cameras in the system feed their respective video signals to a camera matrix.
I.(:'OI/YOAl
See Roberts, 2:54-59.

D/436261 .]
With reference to Figure 1, the camera matrix 5 is

28
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 controlled by a CPU in which "data" is retained "relating to a fixed field of view afforded by each of the cameras" 1, 2, 3, 4, and the position of the cursor 8. A target person may be visually followed by an operator across a current field of view of the camera feeding image display 9, and when the target approaches an.edge of the field of view, a device 7 (such as a mouse) may be used to mark the spot with the cursor .8, for example by clicking on the boundary. Data identifying the marked spot together with data appertaining to each available camera.image is then processed in the CPU .in order to select the most suitable camera to feed the monitor 6 with the image display from the selected camera. See Roberts, 3:2-14. "[U]pon movement of the target to a transition region, the user control is operable to select the 'button' region in which the target appears in a displayed image, so as to select a second camera and to employ video signals from the second video camera to maintain target image continuity." Roberts, 4:26-31. The use of the Roberts system is explained as follows: [A] target of interest can be observed by a first video camera and an image of said target be displayed on the monitor, wherein overlap image portions at transition regions of a displayed image are provided by two adjacent cameras, each transition region being displayed as a 'button' region by the image display; wherein there are a plurality of such 'button' regions being arranged about or-surround[ing] a central area in an image field of view; wherein, upon movement of the target to a transition region, the user control is operable to select the 'button' region of an image displayed, whereby to select a second camera and to employ video signals from the second video camera. Roberts, 2:24-35.
CO 1/YOl A1:/436261. 1 29
FIOURE I
11'
Roberts, Fig. 1. Figure 1 of Roberts, above, shows the interface or monitor 6 screen. The "data" referenced above, relating to the "fixed field of view afforded by each of the cameras," constitutes data which indicates the topological relationship of the cameras. See
Roberts, 3:2-14. In order to "select the most suitable camera to feed the monitor" after a spot is marked by a user by clicking on the boundary 11 of the displayed image where the target approached the edge of the field of view requires that the CPU have data that indicates the topological relationship of the cameras in the system. See id. With continued reference to Figure. 1, clicking on the boundary of the image display 9 to select a new camera to provide the video for the image display may involve
I)CO 1/YOIIAD/43626 1.
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 clicking on boundary "buttons" 11 as explained below. [A] target of interest can be observed by a first video camera and an image of said target be displayed on the monitor, wherein overlap. image portions at transition regions of a displayed image are provided by two adjacent cameras, each transition region being displayed as a 'button' region by the image display; wherein there are a plurality of such 'button' regions being arranged about or surround[ing] a central area in an image field of view; wherein, upon movement of the target to a transition region, the user control is operable to select the 'button' region of an image displayed; whereby to. select a second camera and to employ video signals from the second video camera. Roberts, 2:24-35. In an alternative embodiment the image displayed may be surrounded by 'button regions' 11, the 'button region' (or regions) adjacent to a spot whereat a target approaches the'limit of an image being selected using the cursor 8, and the device 7, in order to provide information for the CPU to facilitate selection of an appropriate one of the cameras 1, 2, 3, or 4 for the image display 9. Roberts, 3:15-21. The Roberts system utilizes "knowledge of the transition regions in each camera where the subject leaves the field of view, and of the camera on which that region can next be seen" to select the best camera to next provide video.for the image display 9. See Roberts, 3:42-46. The referenced use of "knowledge of the transition regions . .where the subject leaves the field of view," and "of the camera on which that region can next be seen," further confirms that the Roberts CPU utilizes data that indicates the topological relationship of the cameras in the system. The composition of the Roberts interface or monitor 6; including the placement 'of the "button regions" 11 (i.e., user controls) or shown in Figure 1 above is identical to DC01/YOI IAD/436261.1 31
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 that of the Video Tracking Interface 300 shown in Figure 3 of the '783 patent, below.
300
31]
Main Video Display

305
310
3O5 Main Video Display

310 C
FIG. 3
'783 patent, Fig.3. Both the Roberts interface and the '783 patent interface include a central maid video display and surrounding user controls that may be clicked upon to select the next camera to provide a video stream for the main video display. Still further, in both
systems, the surrounding user controls are placed in locations which are' representative of the topological relationship of the cameras associated with the user controls with the camera providing video for the'main video display. The identification and use of these topological relationships is apparent from perusal of the Roberts reference because such relationships must be known in order to select the most suitable camera to next DCOI/YO A) /436261. . ' 32
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 feed'the monitor 6 with the. image display from the selected camera and to maintain target image continuity. See Roberts, 3:2-14 and 4:26-31. The Roberts system also inherently re-associates each of the boundary buttons 11 with cameras in the system to be representative of the topological relationship of such cameras with the viewpoint of the camera selected to provide the main video display. If this were not the case, the Roberts system would not operate as described "to maintain target image continuity." Roberts, 4:30-31. In light of the foregoing
disclosure of the Roberts reference, a substantial new question of patentability is raised and Claim 1 is anticipated.
"4.
Redstone Products_Services Reference
Claim 1 of the '783 patent is anticipated by the website of Redstone Integrated Solutions, available at http://web.archive.org/web/20040606014343/redstone-
is.com/products_services.html (the "Redstone Products_Services reference") (Exhibit E). The Redstone Products Services website page is indicated to have a copyright notice of publication date of 2003, and an Internet publication date of June 6, 2004. As a result, the Redstone Products_Services reference qualifies as prior art against the '783 patent under 35 U.S.C. 102(b). A detailed comparison of the elements and
limitations of Claim 1 of the '783 patent with the disclosure of the Redstone Products_Services reference is provided in Appendix D, which is incorporated herein by reference.
DCO/YOJIAD/43(261 .1
33
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 reference discloses a product called
The
Redstone. ProductsServices
VisualPursuit which allows a surveillance operator to easily track a moving subject through a facility from. a single monitor.. See Redstone Products_Services, 1.
"VisualPursuit's display combines the output from several surveillance cameras into a single, MS Windows-standard screen." Id., 1. The VisualPursuit screen contains a
Subject View and Locator Views. The Subject View displays the current location of the moving subject. Up to 8 Locator Views display the output of the cameras. closest to the one trained on the subject. An operator may track a subject by clicking the Locator View that the person has moved into. The Locator View becomes the new Subject View, and all Locator Views are automatically reassigned and updated. See id. A screenshot of the VisualPursuit interface provides views taken from nine cameras. A large, central window contains the output from the current main tracking camera; other cameras in proximity to the current tracking camera are displayed in the other eight windows. By clicking on one of these adjacent views, the operator can The adjacent views then
change it to be VisualPursuit's main camera view.
automatically update based on the, new current tracking camera. See id.,.2. VisualPursuit is described as using location information for each of the cameras that are part of the system. Namely, a series of LED's give the operator an idea of where the current tracking camera is, in relation to another camera. There is also a map layout that can assist with visualizing exactly where each camera is in the facility being monitored.
COI/Y01 IAD/4v3o62( 1
See id., 2.
VisuaiPursuit is also described as working with

.34
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 Redstone's MapTrack product, which identifies the locations of the cameras in the system relative to each other.. See id., 1. "MapTrack's display combines digital maps or diagrams with graphical symbols that can represent the locations of items such as.. .security cameras...". Id., 3. The foregoing establishes that the topological relationship between the nine cameras used for VisualPursuit were known.in order for VisualPursuit to perform the specified functions of "display[ing] the output of the cameras closest to the one trained on the subject" in Locator Views, and enabling a user to click on "the Locator View that the person has moved into [so that] [t]he Locator View becomes the new Subject View, and all Locator Views are automatically reassigned and updated." There are multiple indications that VisualPursuit utilized the topological relationships of the cameras when assigning Locator Views to particular cameras. First, "VisualPursuit works with MapTrack, and MapTrack identifies the locations of the security cameras in the system. Second, it is necessary for VisualPursuit to identify the topological relationships of the cameras in the system in order to identify the "cameras closest to the one trained on the subject" after a new camera is selected to provide the Subject View. Further, the Locator Views are each a "user control" which may be
selected by clicking on it. Thus, each of the cameras that provide a Locator View are associated with a Locator View (i.e., user control) as a result of being "closest" to (i.e., based on the topological relationship with) the camera providing the Subject View. Accordingly, the Redstone Products_Services reference raises a substantial new DC01/YO1 IA^ IA)3,261.1 35
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No: 7,777,783 question of patentabilityas it discloses all elements of, and anticipates C.laim 1 of the '783 patent.
B.
Proposed 35 U.S.C. 103 Rejections
The Manual of Patent Examining Procedure ("MPEP") was revised in 2007 to incorporate the Examination Guidelines for Determining Obviousness Under 35 U.S.C. 103 in View of the Supreme Court Decision in KSR International Co. v. Teleflex -Inc., 72
.FR
57526 (Oct. 10, 2007), Off. Gaz. Pat. Office 23 (Nov. 6, 2007) ("2007 KSR
Guidelines"). See MPEP 2141 (8 th ed. 2001) (Rev. 6, Sept. 2007). -The 2007 KSR Guidelines identify the following six (6) nonexclusive rationales for finding a claimed invention obvious inview of prior art. These rationales are: (1) combining prior art elements according to known methods to yield predictable results; (2) simple substitution of one known element for another to obtain predictable results; (3) use of a known .technique to improve similar devices, methods, or products in the same way; (4) applying a known technique to a known device, method, or product ready for improvement to yield predictable results; (5) obvious to try - choosing from a finite number of identified, predictable solutions, with a reasonable expectation of success; and
DCOI/YO HAI /4 362, .1
36
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 known work in one field of endeavor may promit variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art. The inventions claimed in the '783 patent are obvious based on one or more of the above-referenced rationales, in addition to being obvious in view of the teachingsuggestion-motivation test that was employed prior to the KSR decision. Each of the combinations of prior art described below satisfy at least the first five rationales because the references relied upon all relate to very similar systems which are used to visually . track a target from camera view to camera view on a screen which displays at least one selected camera view. As a result, one of ordinary skill in the art would have found it obvious to combine the teachings of these references to obtain the advantages of each in the resulting system. Moreover, the references themselves, in explaining the
(6)
advantages of each, provide motivation to one of ordinary skill in the art to combine the teachings of each: 1. Girgensohn '706 Reference a. Girgensohn '706 Reference in combination with Lin and Girgensohn '978 References
The teachings of the Girgensohn '706 reference (Exhibit B) would have been readily combined by one of ordinary skill in the art with those of U.S. Patent. No. 7,242,423 to Lin (the "Lin" reference) (Exhibit F) and U.S. Patent Appl. Pub. No.
I)C(OI/YOIHAD4.3626t1.1
37
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 2006/0284978 Al to Girgensohn et al. (the'"Girgensohn '978" reference) (Exhibit G) to render Claims 3-8 of the '783 patent obvious over the Girgensohn '706 reference in view of the Lin reference and in further view of the Girgensohn '978 reference. The application that resulted in the Lin reference was filed August 8, 2003 and the patent issued on July 10, 2007. As a result, the Lin reference qualifies as prior art against the '783 patent under 35 U.S.C. 102(e). The application that resulted in the Girgensohn '978 reference was filed January 3, 2006 and published on December 21, 2006. As a result, the 'Girgensohn '978 reference qualifies as prior art against the '783 patent under 35 U.S.C. 102(e) and 102(a). A detailed comparison of the elements and limitations of Claims 3-8 of the '783 patent with the relevant disclosure of the Girgensohn '706, Lin and Girgensohn '978 references is provided in Appendix E, which is incorporated herein by reference. Requester also hereby. incorporates by reference the discussion of the Girgensohn '706 reference provided above. The Girgensohn '706, Girgensohn '978 and Lin references all disclose systems and methods for switching between cameras to monitor a surveillance area. See. Lin, Abstract; see also Girgensohn '978, Para. 0008. With regard to Claims 5-8, both
Girgensohn '706 and Lin also disclose. using an automatic temporal offset to account for the travel time of a target between the fields of view of two cameras, which limitation was required in order to obtain allowance. of Claim 5. In Girgensohn '706, "[a] three to ten second offset from' the time of the main video stream display playing performs best for tracking people walking from camera
DCOI
OIY()IAI)/43 261. 1
38
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 view to camera view. In embodiments, this offset can be varied as desired. The refresh rate of images in these video streams can be varied to produce effects ranging from almost fluid video playback to images that remain steady for seconds or even minutes." Girgensohn '706, Para. 0030 (emphasis added). One of ordinary skill in the art would readily understand that the afore-noted "offset" disclosed in Girgensohn '706 is based on the travel time of a target between the fields of view of two cameras. However, even if this were not the case, Lin expressly discloses use of a travel time based temporal offset. In Lin, "[i]f a physical path exists for an object to traverse from one zone to another, these zones are defined to be "linked"." Lin, 4:1-3. The use of a temporal
offset for linked zones is explained further in Lin as follows: -In a preferred embodiment this invention, each linked zone may contain a time parameter that determines whether an event in one zone is "recent" relative to the corresponding event in another zone. For example, a disappearance from zone Z(3,1) of FIG. 2A will correspond/to an immediate appearance at the corresponding zone of camera C2, and the definition of "recent" for this linkage can be set to a few seconds. Conversely, an appearance at zone Z(I,I) may correspond to a disappearance from a zone-of camera C3 that occurred minutes before, and the definition of "recent" for this linkage would be set to a few minutes. Lin, 6:5364. The surveillance system disclosed in Lin expressly uses "a time duration parameter corresponding to traversing between the first zone and each of the zones in the set of linked zones." See Lin, 8:60-64. The Lin "time duration parameter" may be based on a "minimal traversal time, a maximum traversal time, a traversal time
1C)01/YOlIAD/436261.1
39
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 likelihood, and a probability distribution function of traversal time" from one camera's view to another's. See Lin, 9:3-8. Thus, Lin discloses use of a temporal offset that is "based on an estimated time of travel of the target" as required by Claim 5. With regard to Claims 3-4 and 5-8, the Girgensohn '978 reference discloses a system and method for automatically stitching together video recorded by different cameras. Girgensohn '978 refers to such stitched together video as "a summary of
video from multiple cameras" which "provides a storyboard of all activity with .emphasis on events of importance" and which "summary can be used to access the recorded video." Girgensohn '978, Para. 0012. Using the Girgensohn '978 system, "[a] single Manga summary for all selected cameras can be created by selecting keyframes from those cameras." Id., Para. 0038. The identity of the cameras that are the source of the keyframes is known because "[c]olor-coding :indicates the source of each keyframe." Id. Girgensohn '978 further discloses that the storyboard of keyframes provides an index for a user to locate video of interest. Users such as security personnel need to be able to select video streams for inclusion in the storyboard and the time frame of the storyboard. Map and timeline interface components have been designed and developed for these purposes. The map, timeline, and storyboard interact to provide the user with the information necessary to locate video segments of interest. Id., Para. 0041. The Girgensohn '978 video summaries are thumbnail sized images as shown in Figures 2-4 and 9-11. Id., Figs. 2-4 and 9-11. In view of the foregoing, the. Girgensohn '706, Lin and Girgensoh DC0I/YOil iAD/436261.1 '978 references collectively disclose all of the 40
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No.7,777,783
limitations of Claims 3-8. It would have been obvious to combine the temporal offset teachings of Lin and the automatic stitching and thumbnail indexing features of Girgensohn '978 with the teachings of Girgensohn '706 as a result of employing simple substitution to obtain predictable results, or as a result of using known techniques to improvesimilar methods and systems. As noted above, all references concern. systems and methods for switching between cameras to monitor a surveillance area. The Girgensohn '706 and Girgensohn '978 references, which have overlapping inventorship, both disclose video display interfaces which present surveillance video from multiple cameras in thumbnail format. The two systems are compatible and complementary. The advantages of such a summary are noted in the Girgensohn '978 and Lin references. See Girgensohn '978, Para. 0008-0009, 0012-0013 and 0041; see also Lin, 3:59-4:3. Thus, one of ordinary skill in the art would have been motivated to combine the functionality of the two systems together to obtain a single system with improved utility. One of ordinary skill in the art would have been further motivated to utilize the travel time temporal offset teachings of Lin in the Girgensohn '706 system, as'the latter specifically references use of an offset in order to account for a target walking from camera view to camera view. A substantial new question of patentability is raised because Claims 5-8 would have been obvious over the Girgensohn '706 reference in view of the Lin reference and in further view of the Girgensohn '978 reference, and Claims 3-4 would have been obvious over the Girgensohn '706 reference in view of the Girgensohn '978 reference. 41 DCO01/YOI IAD/436261,1
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 b. Girgensohn '706 Reference in combination with Gormley Reference
The teachings of the Girgensohn '706 reference (Exhibit B) would have been readily combined by one of ordinary skill in the art with those of U.S. Patent No. 5,258,837 to Gormley (the "Gormley" reference) (Exhibit H) to render Claim 2 of the '783 patent obvious. The Gormley reference issued as a patent on November 2, 1993 and qualifies as prior art against the '783 patent under 35 U.S.C. 102(b). A detailed comparison of the elements and limitations of Claim 2 of the '783 patent with the relevant disclosure of the Girgensohn '706 and Gormley references is provided. in Appendix, F, which is incorporated herein by, reference. Requester also hereby
incorporates by reference the discussion of the Girgensohn '706 reference provided above. The Girgensohn '706 reference discloses all elements of Claim 1. Claim 2 adds the limitations of converting video data from a first format to a second format, and storing video data generated by a video camera in association with identification information for the camera. The Girgensohn '706 system also stores, and provides format conversion of, video data. See Girgensohn '706, Para. 0070. Format
conversion occurs given that the Girgensohn '706 cameras 'utilize JPEG which is a video compression technique that was well known in the art prior to the critical date. Accordingly, format conversion occurs for incoming video to be converted to JPEG. The Girgensohn '706 system also utilizes information to identify the cameras which are
)CO1/YOIID/43626.1
42
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No: 7,777,783 associated with each of the video streams that it displays. Specifically, "the main video stream and the other video streams have colored borders to indicate the actual source cameras." Girgensohn '706, Para. 0031. This color coding information must
necessarily be stored for video playback. See also, e.g., Girgensohn '706, Para. 0052 and 0067. Even if the color coding information of Girgensohn '706. were not stored, however, Gormley teaches storage of information which identifies the cameras used to generate video streams. In this regard, Gormley states: It may be desirable for the, images to be identified. One way of achieving this is for each camera to have associated with it an identification unit which inserts an identification signal (e.g. an alphanumeric message) into its. picture. This results in the message forming an indivisible part of the image, so that the message will be enlarged together with the rest of the image if that image is selected for . the large zone. Another way of achieving this is for the identification signals to be generated in association with the generation of the composite picture. This permits the message to remain the same size regardless of the size of the associated images. If the composite picture has neutral borders between the zones, then the messages can be displayed in those borders. Gormley, 4:10-17. In order for the Gormley system to insert identification signals as an indivisible (i.e., permanent) part of the video images, identification information for each of the cameras must be stored in association with the video images. It would have been obvious for one of ordinary skill in the art to provide the video streams of Girgensohn '706 with the identification signal of Gormley as a result of
I EX()/YOIA1/436261.1
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 employing simple substitution to obtain predictable results, or as a result of using known techniques.to improve similar methods and systems. More specifically, it would have been obvious to combine these teachings in order to improve upon and/or supplement the method of identifying camera video streams disclosed in Girgensohn '706. The
motivation for such modification is provided in Gormley, which states "[i]t may be desirable for the images to be identified." Gormley, 4:10. Accordingly, Claim 2 would have been obvious over the Girgensohn '706 reference in view of the Gormley reference, and a substantial new question of patentability is raised. 2. Roberts Reference a. Roberts Reference in combination with Girgensohn '706 and Gormley References
The teachings of the Roberts reference (Exhibit D) would have been readily combined by one of ordinary skill in the art with those of the Girgensohn '706 (Exhibit B) and Gormley (Exhibit H) references to render ,Claim 2 of the, '783 patent obvious. A detailed comparison of the elements and limitations of Claim 2 of the '783 patent with. the relevant disclosure of the Roberts, Girgensohn '706 and Gormley references is provided in Appendix G, which is incorporated -herein by reference. Requester also
hereby incorporates by reference the discussion of the Girgensohn '706. and Gormley references provided above. The Roberts reference discloses all elements of Claim 1. Claim 2 adds the limitations of converting video data from a first format to a second format, and storing
IC) I /YOHIA D/436261.1
44
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 video data generated by a video camera in association With identification information for the camera. The Roberts reference discloses that camera identification information is stored in connection with the Roberts system. "Conveniently the user control includes a CPU, which is operable to store data including, in respect to each camera, the pixel positions of the lines at which transitions occur, and the identity of the camera to be selected at that transition." format conversion, however. The Girgensohn '706 system also stores, and provides format conversion of, video data. See Girgensohn '706, Para. 0070. Format conversion occurs given that the Girgensohn '706 cameras utilize JPEG which is a video compression technique that was-well known in the art prior to the critical date. Accordingly, format conversion occurs for incoming video to be converted to JPEG. It would have been obvious for one of ordinary skill in the art to utilize JPEG with the Roberts system. As taught by Roberts, 1:60-63. Roberts does not expressly disclose
Girgensohn '706, JPEG is a format used for streaming video, such as the video disclosed in Roberts. It would have been obvious to one of ordinary skill in the art to provide the video streams of Roberts with the format conversion of Girgensohn '706 to improve upon the flexibility of the Roberts system to work with different formats. The Girgensohn '706 system also utilizes information to identify the cameras which are associated with each of the video streams that it displays. Specifically, "the main video stream and the other video streams have colored borders to indicate the actual source cameras." Girgensohn '706, Para. 0031. This color coding information DCO1/YOHAD/43626 1.1 45
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,77.7,783 must necessarily be stored for video playback. See also, e.g., Girgensohn '706, Para. 0052 and 0067. Even if the color coding information of Girgensohn '706 were not stored, however, Gormley teaches storage of information which identifies the cameras used to generate video streams. In this regard, Gormley states: It may be desirable for the images to be identified. One way of achieving this is for each camera to have associated with it an identification unit which inserts an identification signal (e.g, an alphanumeric message) into its picture. This results in the message forming an indivisible part of the image, so that the message will be enlarged together with the rest of the image if that image is selected for the large zone. Another way of achieving this is for the identification signals to be generated in association with the generation of the composite picture. This permits the message to remain the same size regardless of the size of the associated images. If the composite picture has neutral borders between 'the zones, then the messages can be displayed in those borders. Gormley, 4:10-17. In order for the Gormley system to insert identification signals as an indivisible (i.e., permanent) part of the video images, identification information for each of the cameras must be stored in. association with the video images. To the extent Roberts does not disclose storing camera identification information in association with video data from the identified camera, it would have been obvious for one of ordinary skill in the art to utilize the video camera identification of Girgensohn '706 and/or the identification signal of Gormley with the. video streams of the Roberts
I)C I /YO HA D/436261. I
46.
Attorney Dkt. No. 017874-0003 Reexamination. Request U.S Patent No. 7,777,783 system as a result of employing simple substitution to obtain predictable results, or as a result of using known techniques to improve similar methods and systems. More
specifically, it would have been obvious to combine these teachings in order to incorporate the method of identifying camera video streams disclosed in Girgensohn '706 and Gormley into the Roberts system in view of the Gormley reference's suggestion that "[i]t may be desirable for the images to be identified." Gormley, 4:10. A substantial new question of patentability is raised because Claim 2 would have been obvious over the Roberts reference in view of the Girgensohn '706 and Gormley references. b. Roberts Reference in combination with Girgensohn '978 and Lin References
The teachings of the Roberts reference (Exhibit D) would have been readilycombined by one.of ordinary skill in the art with those of the Girgensohn '978 (Exhibit B) and Lin (Exhibit F) references to render Claims 3-8 of the '783 patent obvious. A
detailed comparison of the elements and limitations of Claims 3-8 of the '783 patent with the relevant disclosure of the Roberts, Girgensohn '978 and Lin references is provided in Appendix H, which is incorporated herein by reference. Requester also hereby
incorporates by. reference the discussions of the Roberts, Girgensohn '978 and Lin references provided above. The Roberts, Girgensohn '978 and Lin references all disclose systems and methods for switching between cameras to monitor a surveillance area. DC01/YO IAD/436261.1 47 See Lin,
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 Abstract; see also Girgensohn '978, Para. 0008. With regard to Claims 3 and 4, the
Roberts reference discloses all elements of Claim 1, from which Claims 3 and 4 depend. The Girgensohn '978 reference discloses the elements of Claims 3 and 4, as discussed below. With regard to Claims 5-8, Lin discloses using an automatic temporal offset to account for the travel time of a target betweenri the fields of view of two cameras, which limitation was required in order to obtain allowance of Claim 5. Lin expressly discloses use of a travel time based temporal offset. In Lin, "[i]f a physical path exists for an object to traverse from one zone to another, these zones are defined to be "linked"." Lin, 4:1-3. The use of a temporal offset for linked zones is explained further in Lin as follows: In a preferred embodiment this invention, each linked zone may, contain a time parameter that determines whether an event in one zone is. "recent" relative to the corresponding event in another zone. For example, a disappearance from zone Z(3,1) of FIG. 2A will correspond to an immediate appearance at the corresponding zone of camera C2, and the definition of "recent" for this )linkage can be 'set. to a few seconds. Conversely, an appearance at zone Z(l,I) may correspond to a disappearance from a zone of camera C3 that occurred minutes before, and the definition of "recent" for this linkage would be set-to a few minutes. 'Lin, 6:53-64. The surveillance system disclosed 'in Lin uses "a time duration
'parameter
corresponding to traversing.between the first zone and each of the zones in the set of linked zones." See Lin, 8:60-64. The Lin "time duration parameter" may be based on a "minimal traversal time, a maximum traversal time, a traversal time likelihood, and a D'01/YO AI.A/436261.1 48
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 probability distribution function of traversal time" from one camera's view to another's. See Lin, 9:3-8. Thus, Lin discloses use of a temporal offset that is "based on an
estimated time of travel of the target" as required by Claim 5. With regard to Claims 3-4 and 5-8, the Girgensohn '978 reference discloses a system and method for automatically stitching together video .recorded by different cameras. Girgensohn '978 refers to such stitched together video as "a summary of video from multiple cameras" which "provides a storyboard of all activity with emphasis on events of importance" and which "summary can be used to access the recorded video." Girgensohn '978, Para. 0012. Using the Girgensohn '978 system, "[a] single Manga summary for all selected cameras can be created by selecting keyframes from those cameras." Id., Para. 0038. The identity of the cameras that are the source of the keyframes is known because "[c]olor-coding indicates the source of each keyframe." Id. Girgensohn '978 further discloses that the storyboard of keyframes provides an index for a user to locate video of interest. Users such as security personnel need, to be able to select video streams for inclusion in the storyboard, and the time frame of the storyboard. Map and timeline interface components have been designed and developed for these purposes. The map, timeline, and ,storyboard interact to provide the user with the information necessary to locate video segments of interest. Id., Para. 0041. The Girgensohn '978 video summaries. are thumbnail sized images as shown in Figures 2-4 and 9-11. Id., Figs. 2-4 and 9-11. In view of the foregoing, the Roberts, Lin and Girgensohn '978 references collectively disclose all of the limitations of
DC
o YO AD/4362,1
49
Claims 3-8. It would have been obvious to combine the temporal offset teachings of Lin and the automatic stitching and thumbnail indexing features of Girgensohn '978 with the teachings of Roberts as a result of employing simple substitution to obtain predictable results, or as a result of using known techniques to improve similar methods and systems. As noted above, all references concern systems and methods for switching between cameras to monitor a surveillance area. Modifying the Roberts system to
stitch together video streams generated by the cameras using the methods of Girgensohn '978 and the temporal offset of Lin would have provided an efficient summary of video taken from multiple cameras. The advantages of such a summary, and the motivation for creating such summary, are noted in the Girgensohn '978 and Lin references. See Girgensohn '978, Para. 0008-0009, 0012-0013 and 0041; see also Lin, 3:59-4:3. One of ordinary skill in the art would have been further motivated to incorporate the automatic stitching capability of the Girgensohn '978 system into the Roberts system because Girgensohn '978 discloses that security personnel "need to be able to select video streams for inclusion in" a summary of video taken from different cameras. Thumbnails are taught in Girgensohn '978 to be used to index such stitched video. Thus, one of ordinary skill in the art would have been motivated to use the thumbnail indexing and stitching.capabilities of Girgensohn '978 in the Roberts system. A substantial new question of patentability is raised because Claims 5-8 would IDC01/YOHAD/)1 326 1. 50
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 have been obvious over the Roberts reference in view of the Lin reference and in further 4 view of the Girgensohn '978 reference, and Claims,3- would have been obvious over the Roberts reference in view of the Girgensohn '978 reference. 3. Redstone Reference a. Redstone Reference in combination with Balancing Tomorrow's Technology, Redstone Integrated Solutions Reference
Claim 1 of the '783 patent is believed to be anticipated by the Redstone Products Services reference (Exhibit E). However, to the extent the Redstone
Products Services reference does not anticipate Claim 1, the teachings of the Redstone Products Services reference would have been readily combined by one of ordinary skill in the art with those of the Balancing Tomorrow's Technology, Redstone Integrated Solutions reference (Exhibit I) to render Claim 1 obvious in view of the two references.
/
The Balancing Tomorrow's Technology, Redstone Integrated Solutions
reference bears a 2003 copyright. notice and is internally dated August 26, 2003. As a result, the Balancing Tomorrow's Technology, Redstone Integrated Solutions reference qualifies as prior art against the '783 patent under 35 U.S.C. 102(b). The Redstone Products_Services reference and the Balancing Tomorrow's Technology, Redstone Integrated Solutions reference are collectively referred to as the "Redstone references." A detailed comparison of the elements and limitations of Claim 1 of the '783 patent with the relevant disclosure of the Redstone references is provided in Appendix I, which is J
DC'01 /YOH AI)/43626, 1. I
51 '
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 incorporated herein by reference. Requester also hereby incorporates by reference the
'discussion of
the Redstone Products_Services reference provided above. Redstone Products_Services reference discloses a product called
The
VisualPursuit which allows a.surveillance operator to easily track a moving subject through a facility from a single monitor. See Redstone Products_Services, 1.
"VisualPursuit's display combines the output from several surveillance cameras into a single, MS Windows-standard screen." Id:, 1. The VisualPursuit screen contains a
Subject View and Locator Views. The Subject View displays the current location .ofthe moving subject. Up to 8 Locator Views display the output of the cameras closest.to the one trained on the subject. An operatr .may track a subject by clicking the Locator View that the person has moved into. The Locator View becomes the new Subject View, and all Locator Views are automatically reassigned and updated. See id. The Balancing Tomorrow's Technology, Redstone Integrated Solutions reference supplements the teaching of the Redstone ProductsServices reference by showing an example of the actual VisualPursuit screen. The VisualPursuit screen includes a large central Subject View surrounded by Locator Views. It would have been obvious to one of ordinary skill in the art to combine the teachings of the two Redstone references, as both describe features of the same product called VisualPursuit' Accordingly, Claim 1 would have been obvious over the Redstone Products_Services reference inview of the Balancing Tomorrow's Technology, Redstone Integrated Solutions reference, and. a substantial new question of patentability is raised.
I)COI/YOHlA1/436261.1
52
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 Redstone References in combination with Girgensohn '706 and Gormley References
b.
The teachings of the Redstone references (Exhibits E and. ) would have been readily combined by one of ordinary skill in the art with those of the, Girgensohn '706 (Exhibit B) and Gormley (Exhibit H) references to render Claim 2 of the '783 patent obvious. A detailed comparison of the elements and limitations of Claim 2 of the '783 patent with the relevant disclosure of the -Redstone, Girgensohn '706 and Gormley references is provided in Appendix J, which is incorporated herein. by reference. Requester also hereby incorporates
-by
reference the discussion of the Redstone,
Girgensohn '706 and Gormley references provided above. The Redstone references disclose all elements of Claim 1. Claim 2 adds the limitations of converting video data from a first format to a second format, and storing video data generated by a video camera in association with identification information for the camera. The Girgensohn '706 system also stores, and provides format conversion of, video data. See Girgensohn '706, Para. 0070. Format conversion occurs given that the Girgensohn '706 cameras utilize JPEG which is .a video compression technique that was well known in the art prior to the critical date. Accordingly, format conversion occurs for incoming video to be converted to JPEG. It would have been obvious for one of ordinary skill in the art to utilize JPEG with the Redstone VisualPursuit system. As' taught by Girgensohn '706, JPEG is a format used for streaming video, such as the
ICOI/YOIIAI)/436261.I 53
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783, video disclosed in the Redstone references. It would have been obvious to one of
ordinary skill in the art to provide the video streams of VisualPursuit with the format conversion of Girgensohn '706 to improve upon the flexibility of VisualPursuit to work with different formats. The Girgensohn '706 system also utilizes information to identify the cameras which are associated with each of the video streams that it displays. Specifically, "the main video stream and the other video streams have colored borders to indicate the actual source cameras." Girgensohn '706, Para. 0031. This color coding information must necessarily be stored for video playback. See also, e.g., Girgensohn '706,- Para. 0052 and 0067. Even if the color coding information of Girgensohn '706 were not stored, however, Gormley teaches storage of information which identifies the cameras used to generate video streams. In this regard, Gormley states: It may be desirable for the images to be identified. One way of achieving this is for each camera to have associated with it an identification unit which inserts an. identification signal (e.g. an alphanumeric message) into its picture. This results in the message forming an indivisible part of the image, so that the message will be enlarged together with the rest of.the image if that image is selected for the large zone. Another way of achieving this is for the identification signals to be generated in association with the generation of the composite picture. This permits the message to remain the same size regardless of the size of the associated images. If the composite picture has neutral borders between the zones, then the messages can be displayed in those borders.
I)(l/YOI IAD/43626 . 1
54
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 Gormley, 4:10-17. In order for the Gormley system to insert identification signals as an indivisible (i.e., permanent) part of the video images, identification information for each of the cameras must be stored in association with the video images. It would have been obvious for one of ordinary skill in the art to utilize the video camera identification of Girgensohn '706 and/or the identification signal of Gormley, with the video streams of the VisualPursuit system as a result of employing simple substitution to obtain predictable results, or as a result of using known techniques to improve similar methods and systems. More specifically, it would have been obvious to combine these teachings in order to incorporate the method of identifying camera video streams disclosed in Girgensohn '706 and Gormley into the VisualPursuit system based may on the suggestion in Gormley that "[i]t be desirable for the images to be.identified." Gormley, 4:10. Accordingly, Claim 2 would have been obvious over the Redstone
references in view of the Girgensohn '706 and Gormley references, and a substantial new question of patentability is raised. c. Redstone References in combination with Girgensohn '978 and Lin References
The teachings of the Redstone references (Exhibits E and I) would have been readily combined by one of ordinary skill in the art with those of the Girgensohn '978 (Exhibit B) and Lin (Exhibit F) references to render Claims 3-8 of the .'783 patent obvious. A detailed comparison of the elements and limitations of Claims 3-8 of the
CO.:0/YOi A D436261 .1
55
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 '783 patent with the relevant disclosure of the Redstone, Girgensohn '978 and Lin references is provided in Appendix K, which is incorporated herein by reference. Requester also hereby incorporates by reference the discussion of the Redstone, Girgensohn '978 and Lin references provided above. The Redstone, Girgensohn '978 and Lin references all disclose. systems and methods for switching between cameras to monitor a surveillance area. See Lin,
Abstract; see also Girgensohn '978, Para. 0008. The Redstone references disclose a system which is used to track a target as it moves between the fields of view of multiple video cameras. Girgensohn '978 discloses a video surveillance system that stitches together streams of surveillance video captured by different cameras in the system, and uses thumbnails to index the stitched together. video. It would' have been obvious to modify the Redstone system to stitch together video streams generated by the cameras in the Redstone system and create an index using thumbnails, using the methods of Girgensohn '978 in order to provide a readily accessible summary of video taken from multiple cameras. The advantages of.(i.e. mptivation for using) such a summary and indexing are noted in the Girgensohn '978 reference. 0008-0010, 0012-0013 and 0041. Lin discloses use of a temporal offset that is based on the travel time between camera fields of view when switching from viewing the video from one camera to that of an adjacent camera. Specifically, Lin discloses a system that uses "a time duration See Girgensohn '978, Para.
parameter corresponding to traversing between the first zone and each of the zones in
)C:()/YOHA D/436261.1
56
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 the set of linked zones." See Lin, 8:60-64. The Lin "time duration parameter" may be based on a "minimal traversal time, a maximum traversal time, a traversal time likelihood, and a probability distribution function of traversal time" from one camera's view to another's. See Lin, 9:3-8. Thus, Lin discloses use of a temporal offset that is "based on an estimated time of travel of the target" as required by Claim 5. It would have been obvious to modify the Redstone VisualPursuit system to stitch together video streams generated by the cameras in the Redstone system, using the methods of Girgensohn '978 and the temporal offset of Lin in order to provide an efficient summary of video taken from multiple cameras. The advantages of such a summary are noted in the Girgensohn '978 and Lin references. See Girgensohn '978, Para. 0008-0009, 00120013 and 0041; see also Lin, 3:59-4:3. ' With regard to Claims 3-4 and 5-6, the Redstone references disclose all elements of Claim 1, from which Claims 3 and 4 depend, as discussed above and in Section V.B.2.b., and the Girgensohn '978 reference discloses a system and method for automatically stitching together video recorded by different cameras. Girgensohn '978 refers to such stitched together video as "a summary of video from multiple cameras" which "provides a storyboard of all activity with emphasis on events of importance" and which "summary can be used to access the recorded video." Girgensohn '978, Para. 0012. The Girgensohn '978 video summaries are thumbnail sized images as shown in Figures 2-4 and 9-11. Id., Figs. 2-4 and 9-11. With regard to Claims 5-8, the Redstone references disclose all the elements of
DC('1/YOlAD/436261.1
57
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 Claims 5, 7 and 8, except for stitching video and using a temporal offset in doing so. Lin discloses using an automatic temporal offset to account for the travel time of a target between the fields of view of two cameras, as discussed above and in Section V,B.2.b. In view of the foregoing, a substantial new question of patentability is raised as Claims 5-8 would have been obvious over the Redstone references in view of the Lin, reference and in further view of the Girgensohn '978 reference, and Claims 3-4 would have been obvious over the Redstone references in view of the Girgensohn '978 reference.
4.
White Reference a. White Reference in combination with Girgensohn '706 and' Gormley References
The teachings of the White reference (Exhibit C) would have been readily combined by one of ordinary skill in the art with those of the Girgensohn '706 (Exhibit B) and Gormley (Exhibit. H) references to render Claim 2 of the '783 patent obvious. A detailed comparison of the elements and limitations of Claim 2 of the '783 patent with the relevant disclosure of the White, Girgensohn '706 and Gormley references, is provided in Appendix L, which is incorporated herein by reference. Requester also
hereby incorporates by reference the discussion of the White, Girgensohn '706 and Gormley references provided above. The White reference discloses a.,camera system and display interface that
DCO) /YOIIAD/43(261.1
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7.,777,783 provides all of the functionality recited in Claim 1 of the '783 patent. The White system arranges thumbnail streams, which act as user controls, around a central focus stream in positions which represent the topological arrangement of the cameras providing the video streams. Each time a new camera is selected to provide the focus stream, the
display is "reconfigured" so that the user control positions on the display (i.e., the thumbnail stream positions) are; re-assigned to the cameras that have a corresponding topological relationship with the newly selected focus stream camera., Claim 2 adds the limitations of converting video data from a first format to a second format, and storing video data generated by a video camera in association with identification information for the camera. As explained above, the Girgensohn '706
system provides format conversion of video data.. See Girgensohn '706, Para. 0070. Format conversion occurs given that the Girgensohn '706 cameras utilize JPEG which is a video compression technique that was well known in the art prior to the critical date. Accordingly, format conversion occurs for incoming video to be converted to JPEG. It would have been obvious for one of ordinary ,skill in the art to utilize JPEG with the White system to improve upon the flexibility of the White system to work with different formats. The Girgensohn '706 system also utilizes information to identify the cameras which are associated with each of the video streams that it displays. Specifically, "the main video stream and the other video streams have colored borders to indicate the actual source cameras."
DCOY011/YOAD/436261.
Girgensohn '706, Para. 0031. This color coding information

59
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 must necessarily be stored for video playback. See also, e.g., Girgensohn '706, Para. 0052 and 0067. Even if the color coding information of Girgensohn '706 were not
stored, however, Gormley teaches storage of information which identifies the cameras used to generate video streams. See.Gormley, 4:10-17. It would have been obvious for one of ordinary skill in the art to utilize the video camera identification of Girgensohn '706 and/or the identification signal of Gormley, with the White system as a result of employing simple substitution to obtain predictable results, or as a result of using known techniques to improve similar methods and systems. More specifically, it would have been obvious to combine these teachings, in order to incorporate the method of identifying camera video streams disclosed in Girgensohn '706 and Gormley into the White system based on the suggestion in Gormley that "[i]t may be desirable for the images to be identified." Gormley, 4:10.
Accordingly, Claim 2 would have been obvious over the White reference in view of the Girgensohn '706 and Gormley references, and a substantial new question of patentability is raised.
b.
White Reference in combination with Girgensohn '978 and Lin References
The teachings of the White reference (Exhibit C) would have been readily combined by one of ordinary skill in the art with those of the Girgensohn '978 (Exhibit B) and Lin (Exhibit F) references, to render Claims 3-8 of the '783 patent obvious. A
DCO /YOIAI)/436261. I
60
Attorney Dkt. No.. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 detailed comparison of the elements and limitations of Claims 3-8 of the '783 patent with the relevant disclosure of the White, Girgensohn '978 and Lin references is provided in Appendix M, which is incorporated herein by reference. Requester also hereby
incorporates by reference the discussion of the White, Girgensohn '978 and Lin references provided above. The White, Girgensohn '978 and Lin references all disclose systems 'and methods for switching between, cameras to monitor a surveillance area. Abstract; see also Girgensohn '978, Para. 0008. See Lin,
With regard to Claims 3-4, the White
reference discloses all elements of Claim 1, from which Claims 3 and 4 depend. The Girgensohn '978 reference discloses the elements of Claims 3-4, as discussed above in Section 2.b. With regard to Claims 5-8, the White reference discloses all the elements of Claims 7 and 8, and Lin discloses using an automatic temporal offset to account for the travel time of a target between the fields of view of two cameras, as discussed in Section V.B.2.b. Specifically, Lin discloses a system that uses "a time duration
parameter corresponding to traversing between the first zone and each of the zones, in the set of linked zones." See Lin, 8:60-64.. The Lin "time duration parameter" may be based on a "minimal traversal time, a maximum traversal time, a traversal time likelihood, and a probability 'distribution function of traversal time" from one camera's view to another's. See Lin, 9:3-8. Thus, Lin discloses use of a temporal offset that is "based on an estimated time of travel of the target" as required by Claim 5. I)C0 /YOHAD/43626i.1 61
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 With regard to Claims 3-4 and 5-6, the Girgensohn '978 reference discloses a system and method for automatically stitching together video recorded by different .cameras. Girgensohn '978 refers to such stitched together video 'as "a summary of video from multiple cameras" which "provides a storyboard of all activity with emphasis on events of importance" and which "summary can be used to access the recorded video." Girgensohn '978, Para. 0012. The Girgensohn '978 video)summaries are
thumbnail sized images as shown in Figures 2-4 and 9-11. Id., Figs. 2-4 and 9-11. In view of the foregoing, the White, Lin and, Girgensohn '978 references collectively disclose all of the limitations of Claims 3-8. The White reference discloses a system which is used to track a target as it moves between the fields of view of. multiple video cameras. Girgensohn '978 discloses a video surveillance system that stitches together streams of surveillance video captured by different cameras in the system, and uses thumbnails to index the stitched together video. It would have been obvious to modify the White system to stitch
together video streams generated by the cameras in the White system and create an index using thumbnails, using the methods of Girgensohn '978 in order to provide a readily accessible summary of video taken from multiple cameras. The advantages of (i.e. motivation for using) such a summary and indexing are noted in the Girgensohn '978 reference. See Girgensohn '978, Para. 0008-0010, 0012-0013 and 0041. Lin discloses use of a temporal offset that is based on the travel time between camera fields of view when switching from viewing the video from one camera to that of 62 DC 01/YO IA /436261.1
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 an adjacent camera. It would have been obvious to modify the White system to stitch together video streams generated by the cameras in it using the methods of Girgensohn '978 and the temporal offset of Lin in order to provide an efficient summary of video taken from multiple cameras. The advantages of such a summary are noted in the Girgensohn '978 and Lin references. See Girgensohn '978, Para. 0008-0009, 00120013 and 0041; see also Lin, 3:59-4:3. In view of the foregoing, a substantial new question of patentability is raised because Claims 3-8 would have been obvious over the White reference in view of the Lin reference and in further view of the Girgensohn '978 reference.
5.
Maruya Reference a. Maruya Reference in combination with Girgensohn '706 and Gormley References
Claims 1-8 of the '783 patent are rendered obvious .over 'U.S. Patent No. 7,746,380 to Maruya et al. (the "Maruya reference") (Exhibit J) in view of the Girgensohn '706 reference (Exhibit B) with respect to Claims 1 and 3-8, and in further view of the Gormley reference (Exhibit H) with respect to Claim 2. The application for the Maruya patent was filed on June 15, 2004 and the patent issued on June 29, 2010. The Maruya reference qualifies as prior art against the '783 patent under 35 U.S.C. 102(e). A detailed comparison of the elements.and limitations of Claim 2 of the '783 patent with the relevant disclosure of the Maruya, Girgensohn '706 and Gormley 63 CO1/Y.OHADI/4362 1.I
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 references is provided in Appendix N, which is incorporated herein by reference. Requester also hereby incorporates by reference the discussion of the Girgensohn '706 and Gormley references provided above. The Maruya reference is directed to a video surveillance system which tracks a tracking target using a plurality of surveillance cameras. See Maruya, Abstract. The
Maruya system stores map information on a surveillance zone, information on camera locations, and imaging range information for the cameras so that the topological relationship of the cameras is known. See Maruya, 1:59-2:2. The system "detects the moving direction of a tracking target based on a picture including the tracking target from one camera, selects at least one other camera which should shoot the tracking target next, and generates a display picture including a picture from the one camera and a picture from the at least one other camera, so as to monitor a moving object as the tracking target." Maruya, 1:59-2:2. With respect to the user interface, the Maruya system "arranges the pictures from the one camera and at least one. other camera at locations which can be approximated to the camera locations on the map to generate a display picture.. In other words, the invention arranges a next camera picture of the tracking target in the direction the tracking target moves within the sight of the surveyor on the picture screen." .Maruya, 2:3-10. Thus, the pictures from the cameras are arranged, in a manner that is
representative of the topological relationship of the cameras providing the pictures. As

shown below, in the Maruya system, video pictures from neighboring cameras A and B
C'0OI/YOIiAD/4, 6
.I
64
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 are provided "around" the.video picture from the tracking camera, and are accordingly, placed in positions relative to the viewpoint of the tracking camera and representative of the actual camera locations.
FIG. 24B
Maruya, Fig. 24B. The Maruya reference also discloses that the video pictures from each of the cameras are re-arranged -on screen each time a new camera is selected to be the tracking camera. In case the tracking target has moved in accordance with the moving example shown in FIG. 17A, the screens shown in FIG. 17B are output on the surveillance terminal screen in the locations (a), (b) and (c) of FIG. 17A. In the location (a), a picture from the camera cl is displayed on the left as a picture from the tracking camera and a picture, from the camera c2 in the moving direction of the tracking target is displayed on the right. When the person moves to the location (b), the tracking camera is switched to c2 and a picture from the camela c3 in the moving direction is displayed as a neighborinq camera on the riqht. When the person moves
DC0:OI /YO1I
D/4362-01.
65
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 to the location (c), a, picture from the camera c3 as a tracking camera is displayed as [the tracking camera] and pictures from the cameras c4 and c5 on the branch passages in the moving direction are displayed on the right as pictures from the neighboring cameras." Maruya, 9:60-10:7 (emphasis added). In view of the foregoing, when a person moves from the field of view of a first camera to the field of view of a second camera using the Maruya system, the picture from the second camera becomes the central tracking camera view, and the picture from a third camera is moved so that it is located on the a screen in a position representative of the topological relationship of the second and. third cameras from the viewpoint of the second camera. With respect to Claim 1, the Maruya reference does not disclose that a user may select a new. tracking camera by clicking on a "user control," such as one of the camera pictures displayed on the screen. In Maruya, a new tracking camera is selected
' automatically based on the movement of the tracking target. Girgensohn '706 teaches
that the smaller surrounding video streams in Figures 3 and 7 of Girgensohn '706 can be clicked upon so that they act as, user controls. It would have been obvious to modify the Maruya system with the teaching of Girgensohn '706 to permit a user to click on the camera pictures, such as those shown in Figures 24B, 26B and 27B of Maruya, to enable a user to manually select such camera pictures. to be the main central camera picture in order to give the user flexibility in selecting the next camera view to be the main camera view. Thus, Claim 1 is obvious over Maruya in view of Girgensohn '706,
)COI/YOHAI)/436261 .1 66
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 and a substantial new question of patentability is raised. Maruya discloses the additional limitations found in dependent Claim 2, including format conversion and storing video data with camera identification information. See Maruya, 4:49-57 and 6:31-42. However, to the extent Maruya does not disclose the additional limitations found in Claim 2, these limitations are disclosed in Girgensohn '706 and Gormley. For the reasons previously stated, it would have been obvious to combine the teachings of Maruya, Girgensohn '706 and Gormley to render Claim 2 obvious. With regard to Claims 3-8, to the extent Maruya does not disclose all elements of the claims, such as stitching together video streams from different cameras; and use of a temporal offset when performing such stitching, these limitations are disclosed in Girgensohn '706 as noted in Appendix R. For the reasons stated above, and in the
attached Appendix R, it would have been obvious to combine the teachings of Maruya and Girgensohn '706 to render Claims 1-8 obvious. Accordingly, a substantial new question of patentability is raised by the combination of Maruya with Girgensohn '706 for all claims. b. Maruya Reference in combination withWhite, Girgensohn '978 and Lin References
The teachings of the Maruya reference (Exhibit J) would have been readily combined by one of ordinary skill in the art with those of the White (Exhibit C) Girgensohn '978 (Exhibit B) and Lin (Exhibit F) references to render Claims 1 and 3-8 of 1)CO1/YOIIAD/43(261. 67
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 A detailed comparison of the elements and limitations of
the '783 patent obvious.
Claims 1 and 3-8 of the '783 patent with the relevant disclosure of the Maruya, White, Girgensohn '978 and Lin references is provided in Appendix O, which is incorporated herein by reference. Requester also hereby incorporates by reference the discussion of the Maruya, White, Girgensohn '978 and Lin references provided above. The Maruya, White, Girgensohn '978 and Lin references all disclose systems and methods for switching between cameras to monitor a surveillance area. Abstract; see also Girgensohn '978, Para. 0008. See Lin,
As noted above, the Maruya
reference discloses all elements of Claims 1 and 3-8, with the exception of using video pictures as user controls, and stitching videos together, with or without a temporal offset. White teaches that thumbnail views can be clicked upon so that they act as user controls. It would have been obvious to modify the Maruya system with the teaching of White to permit a user to click on the camera pictures, such as those shown in Figures 24B, 26B and 27B of Maruya, to enable a user to manually select such camera pictures to be the main central camera picture in order to give the user flexibility in selecting the next camera view to be the main camera view. Girgensohn '978 discloses a video surveillance system that stitches together streams of surveillance video captured by different cameras in the system. It would have been obvious to modify the Maruya system to stitch together video streams generated by the cameras in the Maruya system, using the methods of Girgensohn '978
DCo /YOHA D/)436261.1
68
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 in order to provide a summary of video taken from multiple cameras. The advantages of such a summary are noted in the Girgensohn '978 reference. See Girgensohn '978, Para. 0008-0009, 0012-0013 and 0041. The Lin reference discloses using an automatic temporal offset to account for the travel time of a target between the fields of view of two cameras, as discussed in Section V.B.2.b. Specifically, Lin discloses a system that uses "a time duration
parameter corresponding to traversing between the first zone and each of the zones in the set of linked zones." See Lin, 8:60-64. The Lin "time duration parameter" may be based on a "minimal traversal time, a maximum traversal time, a traversal time likelihood, and a probability distribution function of traversal time" from one camera's view to another's. See Lin, 9:3-8.' Thus, Lin discloses use of a temporal offset that is "based on an estimated time of travel of the target" as required by Claim 5'. In view of the foregoing, the Maruya, White, Lin and Girgensohn '978 references collectively disclose all of the limitations of Claims 1 and 3-8. As explained above and. in Appendix O, it would have been obvious to combine the teachings of these references to render Claims 1 and 3-8 obvious. c. Maruya Reference in combination with Roberts, Girgensohn '978 and Lin References
The teachings of the Maruya reference (Exhibit J) would have been readily combined by one of ordinary skill in the art with those of the Roberts (Exhibit D) Girgensohn '978 (Exhibit B) and Lin (Exhibit F) references to render Claims 1 and 3-8 of
DC01/YOHIA D/436261.1
69
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 the '783 patent obvious. A detailed comparison of the elements and limitations of Claims 1 and 3-8 of the '783 patent with the relevant disclosure of the Maruya, Roberts, Girgensohn '978 and Lin references is provided in Appendix P, which is incorporated herein by reference., Requester also hereby incorporates by reference the discussion of the Maruya, Roberts, Girgensohn '978 and Lin references provided above. The Maruya, Roberts, Girgensohn '978 and Lin references all disclose systems and methods for switching between cameras to monitor a surveillance area. See Lin,
Abstract; see also Girgensohn '978, Para. 0008. As noted above, the Maruya reference discloses all elements of Claims 1 and 3-8, with the exception of using video pictures as user controls, and stitching videos together, with or without a temporal offset. The. rationale for combining the teachings of the Maruya, Girgensohn '978 and Lin references is also explained above and in the referenced appendices. Maruya discloses automatic selection of the next camera view based on target movement. Roberts teaches that user controls in the form of "button regions," can be
clicked upon and surround a display image provided by a selected camera. The "button regions" are located along the edge of the central display image and are presented in display locations representative of topological relationship of the cameras in the system, relative to the viewpoint of the camera providing the: central display image. It would
have been obvious to modify the Maruya system with the teaching of Roberts to permit a user to click on the camera pictures, such as those shown in Figures 24B, 26B and 27B of Maruya, to enable a user to manually select such camera pictures to be the main
I)c' I/YOIII 1/4 362G1.
70
Attorney .Dkt, No. 017874-0003 Reexamination, Request U.S Patent No. 7,777,783 central camera picture in order to give the user flexibility in selecting the next camera view to be the main camera view. Maruya teaches that the camera pictures which provide the next main camera view are located closest to the edge of the main camera view at which a target leaves the field of view of the camera that previously provided the main camera view. Thus, the Maruya, Roberts, Lin and Girgensohn '978 references collectively disclose all of the limitations of Claims 1 and 3-8. As explained above and in.Appendix P, it would have been obvious to combine the teachings ofrthese references to render Claims 1 and 3-8 obvious. Accordingly, these references collectively raise a substantial new question of patentability. d. Maruya Reference in combination with Redstone, Girgensohn '978.and Lin References
The teachings of the Maruya reference (Exhibit J) would have been readily combined by one of ordinary skill in the art with those of the Redstone (Exhibits E and I) Girgensohn '978 (Exhibit B) and Lin (Exhibit F) references to render Claims 1 and 3-8 of the '783 patent obvious. A detailed comparison of the elements and limitations of Claims 1 and 3-8. of the '783 patent with the relevant disclosure of the Maruya, Redstone, Girgensohn '978 and Lin references is provided in Appendix Q, which is incorporated herein by reference. Requester also hereby incorporates by reference the discussion of the Maruya, Redstone, Girgensohn '978 and Lin references provided above.
I)COI/YOHAD/436261.1
71
Attorney Dkt, No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 The Maruya, Redstone, Girgensohn '978 and Lin references all disclose systems and methods for switching between cameras to monitor a surveillance area. Abstract;' see also Girgensohn '978, Para. 00081 See Lin,
As noted above, the Maruya
reference discloses all elements of Claims 1 and 3-8, with the exception ,of using video pictures as user controls, and stitching videos together, with or without a temporal offset. The rationale for combining the teachings of the Maruya, Girgensohn '978 and Lin references is also explained above and in the referenced appendices. Maruya discloses automatic selection of the next camera view based on target movement. The Redstone references teach that the thumbnails can be clicked upon so that they 'act as.user controls. It would have been obvious to modify the Maruya system with the teachings of the Redstone references to' permit a user to click on the camera pictures, such as those shown in Figures 24B, 26B and 27B of Maruya, to enable a user to manually select such camera pictures to be the main central camera picture in order to give the user flexibility in selecting the next camera view to be the main camera view. Thus, the. Maruya, Redstone, Lin and Girgensohn '978 references collectively disclose all of the limitations of Claims 1 and 3-8. As explained above and in Appendix P, it would have been obvious to combine the teachings of these references to render Claims 1 and 3-8 obvious. Accordingly, these references collectively raise a substantial new question of patentability.
DC01/YOHAD/436261. I
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 6. Buehler Reference a. Buehler Reference in combination with Girgensohn,'706 Reference
Claims 1-8 of the '783 patent are rendered obvious over U.S. Patent Appl. Pub. No. 2010/0002082 Al to Buehler et al. (the "Buehler" reference) (Exhibit K) in view of the Girgensohn '706 (Exhibit B) reference. The application which resulted in publication of the Buehler reference was filed on March 24, 2006. Therefore, the Buehler reference qualifies as prior art against the '783 patent under 35 U.S.C. 102(e). A detailed
comparison of the elements and limitations of Claim 2 of the '783 patent with the relevant disclosure of the Buehler and Girgensohn '706 references is provided in Appendix R, which is incorporated herein by reference. Requester also hereby
incorporates by reference the discussion of the Girgensohn '706 reference provided above. The Buehlerreference was relied upon alone and in combination with other prior art references for the final rejection of the '783 patent claims. These rejections resulted in the applicants resorting to combining the limitations of original Claims 1, 2 and 3 to obtain the allowance of issued Claim 1, and to combining -the limitations of original Claims 7, 16 and 18 to obtain allowance of issued Claim 5. The Notice' of Allowability of the application for the '783 patent included the examiner's statement of reasons for allowance of the issued claims. The reason for allowance of issued Claim 1 was that the closest prior art (i.e., the combination of the
DCO1/YO HI 1'436261 .1 A
Attorney Dkt. No. 017874-0003 Reexaminationi Request U.S Patent No. 7,777,783. Buehler reference with US Pub: No. 2002/0097322 Al to Monroe et al.) did. not disclose
the last element of issued Claim 1, which recites:
presenting the first user control and the second user control in the user interface in positions representative of the first topological relationship and the second topological relationship, where in the first user control and the second user control are presented in positions around a presentation of the video data generated using the first member of the plurality of
Scameras.
June 18, 2010 Notice of Allowance, at 3. Issued Claims 2-4 were allowed "due to
their dependence on allowed independent claim 1." Id. Similarly, issued Claim 5 was allowed because the closest prior art (i.e., the
combination of the Buehler reference with US Pub. No. 2005/0052532 Al to Elooz et al.) did not disclose a portion of the last element of issued Claim 5, which recites: "using a temporal offset to automatically stitch the first video data and the second video data, the temporal offset being based on an estimated time of travel of the target." Id., at 4. Issued Claims 6-8 were allowed "due to their dependence on allowed independent claim 16 [issued Claim 5]." Id.. The applicants did not contest or attempt to comment on the stated reasons for allowance. In view of the foregoing, the Buehler reference is admitted by .the applicants to disclose all of the limitations of Claims" 1 and 5 save for (i) presenting user controls in positions representative of the topological relationship of the cameras, which positions are "around" a presentation of the video data generated using a selected camera, and
(ii) using a temporal offset based on an estimated time of travel between camera fields
1)C01/YO1IAD/43,2( t. i
74
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 of view when stitching together video from different cameras. As explained above and in Appendix R, the Girgensohn '706 reference discloses (i) positioning user controls around a presentation of video data such that the user controls are in locations representative of the topological relationship of the cameras with which the user controls are associated, and (ii) using a temporal offset when stitching together video from different cameras, the only claim limitations ostensibly not disclosed in Buehler. Buehler teaches an icon layout in which secondary video data feeds 125 may be placed in thumbnail windows which surround a primary video data. feed 115. Buehler, Fig. 1. See.
I)C:I/YOI1AD/4326 I.1
120 100
10 105
25
L2
Buehler, Fig. 1. It would have been obvious for one of ordinary skill in the art to modify the Buehler system with the teachings of Girgensohn '706 so that the Buehler secondary video data feeds were placed in positions representative of the topological relationships between the cameras providing the secondary camera feeds and the camera providing the primary video data feed, taken: from the viewpoint of the camera providing the primary video data feed. It would also have been obvious in view of Girgensohn '706 to modify the Buehler system so that the secondary video data feeds could be clicked
DcO1/YOIIA I.)430261,I
76
Attorney Dkt. No. 017874-0003 Reexamination Request U:S Patent No. 7,777,783 upon (i.e., act as user controls) to select a secondary video data feed to be the primary video data feed. One of ordinary skill in the art would have been motivated to combine these teachings in order to satisfy the need identified in Girgensohn '706 to have a "way for users to browse video by quickly. ..switching to another video stream to keep the activity in view." Girgensohn '706, Para. 0005., Accordingly, the combination of the
Buehler and Girgensohn,'706 references renders Claims 1-8 obvious and raises a substantial new question of patentability. b. Buehler Reference in combination with White, Girgensohn '978 and Lin References
The teachings of the Buehler reference .(Exhibit K) would have been readily combined by one of ordinary skill in the art with those of the White (Exhibit C) Girgensohn '978 (Exhibit B) and Lin (Exhibit F) references to render Claims. 1-8 of the '783 patent obvious. A detailed comparison of the elements and limitationsof Claims 18 of the '783 patent with the relevant disclosure of the Buehler, White, Girgensohn '978 and Lin references is. provided in Appendix S, which is incorporated herein by reference. Requester also hereby incorporates by reference the discussion of the Buehler, White, Girgensohn '978 and Lin references provided above. The Buehler, White, Girgensohn '978 and Lin references all disclose systems and methods for switching between cameras to monitor a surveillance area. See Lin, Abstract; see also Girgensohn '978, Para. 0008. As noted above, the Buehler reference discloses all of thelimitations of Claims 1 and 5 except (i) presenting user controls in
I)(C1/YOHA1/43626 .1 77
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 positions representative of the topological relationship of the cameras, which positions are "around" a presentation of the video data generated using a selected camera, and (ii) using a temporal offset based on an estimated time of travel between camera fields of view when stitching together video from different cameras. Buehler also does not disclose providing a thumbnail index of stitched together video. With regard to Claim 1, Buehler teaches an icon layout in which secondary video data feeds may be placed in thumbnail windows which surround a primary video data feed. See Buehler, Fig. 1. White teaches that thumbnail views may be clicked upon to select such views to be a main video view and that the thumbnail views may be presented in positions on the interface screen which are representative of the topological relationships of the cameras providing the main video view and the thumbnail views. See Section V.A.2.,above. It would have been obvious for one of ordinary skill in the art to combine the teachings of Buehler with those of White so that the Buehler secondary video data feeds were placed in positions representative of the topological relationships between the cameras providing the secondary camera feeds and the camera providing the primary video data feed, taken from the viewpoint of the. camera providing the primary video data feed, It would also have been obvious in view of White to modify the Buehler system so that the secondary video data feeds could be clicked upon (i.e., act as user controls) to select a secondary video data feed to be the primary video data feed. One of ordinary skill in the art would have been motivated to combine the teachings of Buehler and White to take advantage of the White system's I)CO/YOIIA)1/436261 1 . 78
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 ability to make "multiple streams available to a user without using an undue amount of bandwith." See White, 1:65-67. The elements of Claims 2 and 3 format conversion, storing camera
identification information with video data, and stitching video together from different cameras - are disclosed in Buehler. With regard to Claim 4, Girgensohn '978 discloses a video surveillance system that stitches together streams of surveillance video captured by different cameras in the system and uses thumbnails to index the stitched video. See Girgensohn '978, Para, 0041 and Figs. 2-4, 9'11. Buehler discloses stitching video streams together. See
Buehler, Para. 0053-0055. It would have been obvious to modify the Buehler system to use thumbnails to index stitched together video streams generated by the cameras in the Buehler system, using the methods of Girgensohn '978 in order to provide a summary of video taken from multiple cameras. The advantages of such a summary are noted in the Girgensohn '978 reference. See Girgensohn '978, Para. 0008-0009, 0012-0013 and 0041. With regard to Claim 5, the Lin reference discloses using an automatic temporal offset to account for the travel time of a target between the fields of view of two cameras, as discussed in Section V.B.2.b, above. Specifically, Lin discloses a system that uses "a time duration parameter corresponding to traversing between the first zone and each of the zones in the set of linked zones." See Lin, 8:60-64. The Lin "time
duration parameter" may be based on a "minimal traversal time, a maximum traversal

I)COI/YIHAD/436261. I 7Q rv
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 time, a traversal time likelihood, and a probability distribution function of traversal time" from one camera's view to another's. See Lin, 9:3-8. Thus, Lin discloses use of a
temporal offset that is "based on an estimated time of travel of the target" as required by Claim 5. The Buehler reference, in combination with the White reference .discloses the elements of Claims 6-8! In view of the foregoing, the Buehler, White, Lin and Girgensohn '97.8 references collectively disclose all of the limitations of Claims 1-8. As explained above and: in.
Appendix S, -it would have been obvious to combine the teachings of these references to render Claims 1-8 obvious. Thus, the Buehler, White, Lin and Girgensohn '978
references raise a substantial new question of patentability. c. Buehler Reference in combination with Roberts, Girgensohn '978 and Lin References
The teachings of the Buehler reference (Exhibit K) would have been readily combined by one of ordinary skill in the art with those of the Roberts (Exhibit D) Girgensohn '978 (Exhibit B) and Lin (Exhibit F) references to render Claims 1-8 of the '783 patent obvious. A detailed oomparison of the elements and limitations of Claims 18 of the '783 patent with the relevant disclosure of the Buehler, Roberts, Girgensohn '978 and Lin references is provided in Appendix T, which is incorporated herein by reference. Requester also hereby incorporates by reference the discussion of the-
Buehler, Roberts, Girgensohn '978 and Lin references provided above. The Buehler, Roberts, Girgensohn '978 and Lin references all disclose systems
26 I 1 l ('0/Y )I1AD43r I.1 80
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 and methods for switching between cameras to monitor a surveillance area. See Lin, Abstract; see also Girgensohn '978, Para. 0008. As noted above, the Buehler reference discloses all of the limitations of Claims 1 and 5 except (i) presenting user. controls in positions representative of the topological relationship of the cameras, which positions are "around" a presentation of the video data generated using a selected camera, and. (ii) using a temporal offset based on an estimated time of travel between camera fields of view when stitching together video from different cameras. With regard to Claim 1, Roberts teaches that user controls in the form of "button regions," can be clicked upon and surround a display image provided by.a selected camera. The "button regions" are located along the edge of the central display image and are presented in display locations representative of topological relationship of the cameras in the system, relative to the viewpoint of the camera providing the central. display image. It would have been obvious to modify the Buehler system with the
teaching of Roberts so that the Buehler secondary video data feeds were placed in positions representative of the topological relationships between the cameras providing the secondary camera feeds and the camera providing the primary video data feed, taken from the viewpoint of the camera providing the primary video data feed. It would also have been obvious in view of Roberts to modify the Buehler system so that the secondary video data feeds could-be clicked upon (i.e., act as user controls) to select a secondary video data feed to be the primary video data feed. This modification of
Buehler would enable a user.to easily select the secondary video data feeds to be the
I.x:tO/YOII AD/43 62 6 1
,
81
Attorney Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 primary,video data feed. With regard to Claim 4, Girgensohn '978 discloses a video surveillance system that stitches together streams of surveillance video captured by different cameras''in the system and uses thumbnails to index the stitched video. See Girgensohn '978, Para. 0041 and Figs. 2-4, 9-11. Buehler discloses stitching video streams together. See
Buehler, Para, 0053-0055. It would have been obvious to modify the Buehler system to, use thumbnails to index stitched together video streams generated by the cameras in the. Buehler system, using the methods of Girgensohn '978 in order to provide a summary of video taken from multiple cameras. The advantages of such a summary are noted in the Girgensohn '978 reference. See Girgensohn '978, Para. 0008-0009, 0012-0013 and 0041. With regard to Claim 5, the Lin reference discloses using an automatic temporal offset to account for the travel time of a target between the fields of view of two cameras, as discussed inSection V.B.2.b, above. Specifically, Lin discloses a system that uses "a time duration parameter corresponding to traversing between the first zone and each of the zones in the set of linked zones." See Lin, 8:60-64. The Lin "time duration parameter" may be based on a "minimal traversal time, a maximum traversal time, a traversal time likelihood, and a probability distribution function of traversal time" from one camera's view to another's. See Lin, 9:3-8. Thus, Lin discloses use of a
temporal offset that. is "based on an estimated time of travel of the target" as required by Claim 5.
DCO(/YOIDi.)/O43G261 .1
82
Attorney Dkt. No. 017874-0003' Reexamination Request U.S Patent No. 7,777,783 In view of the foregoing, the Buehler, Roberts, Lin .and Girgensohn '.978 references collectively disclose all of the limitations of Claims 1-8. As explained above and in Appendix T, it would have been obvious to combine the teachings of these references to render Claims 1-8 obvious. Thus, the Buehler, Roberts, Lin and
Girgensohn '978 references raise a substantial new question of patentability.
VI.
CONCLUSION For the foregoing reasons, substantial new questions of patentability with respect
to all claims of the '783 patent have been raised. Reexamination of all claims (1-8) of the '783 patent is respectfully requested. The undersigned is acting as a representative party of the real party in interest, VidSys, Inc. pursuant to 37 C.F.R.
1.34.
VII.
LIST OF EXHIBITS AND APPENDICES U.S. Patent No. 7,777,783 to Chin et al. U.S. Patent Appl. Pub. No. 2008/0088706 Al to Girgensohn et al. U.S. Patent No. 7,196,722 to White et al. U.S. Patent No. 7,295,228 to Roberts et al. http://web.archive.orq/web/20040606014343/redstoneis.com/products services.html, pp. 1-11.. U.S. Patent No. 7,242,423 to Lin. U.S. Patent Appl. Pub. No. 2006/0284978 Al to Girgensohn et al.
83
Exhibit A Exhibit B Exhibit C Exhibit D Exhibit E Exhibit F Exhibit G
)COI/YOVIA14 3(i2(1.
Attorney Dkt. :No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783 Exhibit H Exhibit I Exhibit J Exhibit K U.S. Patent No. 5,258,837 to Gormley. Balancing Tomorrow's Technology, Redstone Integrated Solutions, 2003. U.S. Patent No. 7,746,380 to Maruya et al. U.S. Patent Appl. Pub. No. 2010/0002082 Al to Buehler et al.
DC0I /YOI AD)/43626 1.1
Attorney. Dkt. No. 017874-0003 Reexamination Request U.S Patent No. 7,777,783
Appendix Appendix Appendix Appendix Appendix Appendix Appendix Appendix Appendix
Anticipation Based On The Girgensohn '706 Reference Anticipation Based On The White.Reference Anticipation Based On The Roberts Reference Anticipation Based On The Redstone Products_Services Reference Obviousness Based On The Girgensohn '706, Girgensohn '978; and Lin References Obviousness Based On The Girgensohn '706 and Gormley Reference Obviousness Based On The Roberts, Girgensohn '706 and Gormley References Obviousness Based On The Roberts, Girgensohn '978 and Lin References Obviousness Based On The Redstone Products Services Reference and Secondary Redstone Reference
Appendix J Obviousness Based On The Redstone, Girgensohn '706 and Gormley References Appendix K Obviousness Based On The Redstone, Girgensohn '978 and Lin References Appendix L Obviousness Based On The White, Girgensohn '706 and Gormley References Appendix Appendix Obviousness Based On The White, Girgensohn '978 and Lin References Obviousness Based On The Maruva, Girgensohn '706 and Gormley References.
Appendix O Obviousness Based On The Maruya, White, Girgensohn '978 and Lin References
DC01/YO1AD/4302( I. I
85
Attorney Dkt. No. 01.7874-0003 Reexamination Request U.S Patent No. 7,777,783 Appendix P Obviousness Based On The Maruya, Roberts, Girgensohn '978 and Lin References Appendix Q Obviousness Based On The Maruya, Redstone, Girgensohn '978 and Lin References Appendix R Obviousness Based On The Buehler and Girgensohn '706 References Appendix S Obviousness Based On The Buehler, White, Girgensohn '978 and Lin References Appendix T Obviousness Based On The Buehler, Roberts and Girgensohn '978 and Lin References
Dated: January 21, 2011 KELLEY DRYE & W/REN, LLC 3050 K Street, N.W., Suite 400 Washington, D.C. 20007 (202) 342-8400
DC:OI/YoIIAn)/46261 .1
86

Title of Invention:

Filer:
Ken Prayoon Cheng

Robert Hayden
Attorney Docket Number: Filed as Large Entity
5330.07 (SMC)

Description Fee Code Quantity Amount Sub-Total in
USD($)
Basic Filing:
Pages:
Claims:
Petition:
Extension-of-Time:
,y
,47,
W_% *
Description
Fee Code
Quantity
Amount
Sub-Total in USD($) USD($)
Miscellaneous: Submission- Information Disclosure Stt 1806 1 180 180
Total in USD ($)
180
:'p
,:

P.O.Box1450 Alexandria, Virginia 22313-1450 www.uspto.gov
APPLICATION NO.
FILING DATE
FIRST NAMED INVENTOR
ATTORNEY DOCKET NO.
CONFIRMATION NO.
11/231,353
23308
,
7590
09/19/2005
04/06/2011
Ken Prayoon Cheng
5330.07 (SMC)
'
EXAMINER
4531
PETERS VERNY, L.L.P. 425 SHERMAN AVENUE SUITE 230 PALO ALTO, CA 94306
GRANT II, JEROME ART UNIT 2625 PAPER NUMBER
MAIL DATE 04/06/2011
DELIVERY MODE PAPER
Please find below and/or attached an Office communication concerning this application or proceeding. The time period for reply, if any, is set in the attached communication.
PTOL-90A (Rev. 04/07)
,f..
UNITED STATES DEPARTMENT.OF COMMERCE Patent and Trademark Office

f'
ASSISTANT SECRETARY AND COMMISSIONER OF PATENTS AND TRADEMARKS Washington. D.C. 20231
This application has been withdraw from abandoned.
Thank you,
L.. STRUMBENTS EXMR. SUPERVISORY TECHN1OlGY:.CEER 2600
"t
__ _
Transaction History Date9 0
14
Date information retrieved from USPTO Patent Application Information Retrieval (PAIR) system records at www.uspto.gov
. Application No.
Applicant(s) CHENG ET AL. Art Unit 2625
Notice of Allowability
-The MAILING DATE of this communicationappears on the cover sheet with the correspondence address-All claims being allowable, PROSECUTION ON THE MERITS IS (OR REMAINS) CLOSED in this application. If not included herewith (or previously mailed), a Notice of Allowance (PTOL-85) or other appropriate communication will be mailed in due-course. THIS NOTICE OF ALLOWABILITY IS NOT A GRANT OF PATENT RIGHTS. This application is subject to withdrawal from issue at the initiative of the Office or upon petition by the applicant. See 37 CFR 1.313 and MPEP 1308. 1.I This communication is responsive to an amendment received 4-6-11.
2. I The allowed claim(s) is/are 14 and 15. 3. QI Acknowledgment is made of a claim for foreign priority under 35 U.S.C. 119(a)-(d) or (f). b) O some* c) O None of the: a) O All 1. O Certified copies of the priority documents have been received.
O Certified copies of the priority documents have been received in Application No. 3. O Copies of the certified copies of the priority documents have been received in this national stage application from the
2. International Bureau (PCT Rule 17.2(a)). * Certified copies not received: Applicant has THREE MONTHS FROM THE "MAILING DATE" of this communication to file a reply complying with the requirements noted below. Failure to timely comply will result in ABANDONMENT of this application. THIS THREE-MONTH PERIOD IS NOT EXTENDABLE. 4. 5.
O O
A SUBSTITUTE OATH OR DECLARATION must be submitted. Note the attached EXAMINER'S AMENDMENT or NOTICE OF INFORMAL PATENT APPLICATION (PTO-1 52) which gives reason(s) why the oath or declaration is deficient.
CORRECTED DRAWINGS (as "replacement sheets") must be submitted. (a) O including changes required by the Notice of Draftsperson's Patent Drawing Review ( PTO-948) attached 1) El hereto or 2) O to Paper No./Mail Date (b)
O including changes required by the attached Examiner's Amendment /Comment or in the Office action of
Paper No./Mail Date Identifying indicia such as the application number (see 37 CFR 1.84(c)) should be written on the drawings in the front (not the back) of each sheet. Replacement sheet(s) should be labeled as such in the header according to 37 CFR 1.121(d).
6. r DEPOSIT OF and/or INFORMATION about the deposit of BIOLOGICAL MATERIAL must be submitted. Note the
attached Examiner's comment regarding REQUIREMENT FOR THE DEPOSIT OF BIOLOGICAL MATERIAL.
Attachment(s) 1. Notice of References Cited (PTO-892) 2. O Notice of Draftperson's Patent Drawing Review (PTO-948)
ElO
0
O Notice of Informal Patent Application 6. O Interview Summary (PTO-413),

5. Paper No./Mail Date
_
3.
4. O
Information Disclosure Statements (PTO/SB/08), Paper No./Mail Date 3/11 Examiner's Comment Regarding Requirement for Depositof Biological Material
7.
O Examiner's Amendment/Comment
8. 0 Examiner's Statement of Reasons for Allowance 9.
Ol
Other
Noic ofAlwblt
Noice of Allowability
ato
Part
ae
oMi
No./Mail
ae2101
Date 20110411
PTOL-37 (Rev. 08-06)
of Paper
Page 2
Reasons for Allowance
Claims 14 and 15 were previously indicated as being objected as containing allowable matter and have been amended to include the subject matter of the base claim. Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jerome Grant IIwhose telephone number is 571-2727463. The examiner can normally be reached on Mon.-Fri. from 9:00 to 5:00. If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, mark Zimmerman, can be reached on 571-272-7653. The fax phone number for the organization where this application or proceeding is assigned is 571273-8300. Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://portal.uspto.gov/external/portal. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). /Jerome Grant II/ Primary Examiner, Art Unit 2625

APPLICANTS: APPLICATION NO.: CONFIRMATION NO.: FILED: TITLE: Ken Prayoon Cheng 11/231,353 4531 September 19, 2005 Adaptive Multi-Modal Integrated Biometric Identification Detection and Surveillance Systems Jerome Grant, II 2625 5330.07 (SMC)
EXAMINER: GROUP ART UNIT: ATTY.DKT.NO.:
MAIL STOP AMENDMENT COMMISSIONER FOR PATENTS P.O. BOX 1450 ALEXANDRIA, VA 22313-1450 List of Co-Pending Patent Applications That May Be Directed Towards Similar Subject Matter Examiner's Initials Serial Number
12/838,973 11/728,404
First-Named Inventor
Hu Chin Hu Chin
Title Filing Date

Multi-Video Navigation Multi-Video Navigation System 7-19-2010 03-23-2007
/JG!
IJGI Examiner's
Signature:
/Jerome Grant II/
04/11/2011
Date
Doc code: IDS Doc description: Information Disclosure Statement (IDS) Filed
PTO/SBI08a (01-10) Approved for use through 07131/2012. OMB 0651-0031
Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it contains a valid OMB control number.
11231353 2005-09-19

Art Unit' Examiner Name
Ken Prayoon Cheng

2625 Jerome Grant, II 5330.07 (SMC)
U.S.PATENTS Examine Cite No Initial* Kind Issue Date Codel
Remove
Patent Number
Pages,Columns, Lines where Name of Patentee or Applicant Relevant Passages or Relevant of cited Document
Figures Appear
/J
/1
6604093
B1
2003-08-05
Etzion et al.
6906709
B1
2005-06-14
Larkin et al.
/JG/
6408404
B1
1 2002-06-18
Michael Ladwig
/JG 4
6711587
B1
2004-03-23
Dufaux
/JG,5
6628829
B1
2003-09-30
Chasen
5912980
1999-06-15
Hunke
/JG/
Goh et al.
/JG/ 7
7136524
B1
2006-11-14.
/JG/
6072496
2000-06-06
Guenter et al.
EFS Web 2.1.17
Application Number
11231353 2005-09-19
Ken Prayoon Cheng
IN FORMATION DISCLOSURE
Filing Date First Named Inventor Art Unit12625

Examiner Name
Jerome Grant, II
5330.07 (SMC)
6625315
B2
2003-09-23
Laumeyer et al.
/JG/
/JG/
10 6404900 B1 2002-06-11 Qian et al.
/JG/
11
6707487
B1
2004-03-16
Aman et al.
/JG!
12'
5473369
1995-12-05
Abe
/JG/ 13
7777783
B1
2010-08-17
Chin et al.
/JG/
14
7242810
B2
2007-07-10
Chang
/JG/15
6008912
1999-12-28
Salo et al.
/JGi
16
5838465
1998-11-17
Satou et al.
17
6335985
B1
2002-01-01
Sambonsugi et al.
/JG/
/JG/ 18 B2 2007-03-06 Moon et al.
7187783
/JG/ 19
6157469
2000-12-05
Mestha
EFS Web 2.1.17
/Jlerome Grant II/
04/11/2011
11231353 2005-09-19
Ken Prayoon Cheng
ArtUnit
Examiner Name
2625
Jerome Grant, II
5330.07 (SMC)
/JG/
20
7525576
B2
2009-04-28
Kannermark et al.
21
7382249
B2
2008-06-03
Fancella
/JG/
/JG/ 22
7384590 B2 2008-06-10 Kelly et al.
/JG/
23
7023913
B1
2006-04-04
Monroe et al.
/JG/ 24
7116353
B2
2006-10-03
Hobson et al.
/JG/
25
7456727
B2
2008-11-25
Pinter et al.
/JG/
26
7058204
B2
2006-06-06
Hildreth et al.
/JG/ 27
7385590
B2
2008-06-10
Millar et al.
/JG/ 28
7522186
B2
2009-04-21
Arpa et al.
29
30
EFS Web 2.1.17
/Jerome Grant II/
04/11/2011
11231353 2005-09-19
Ken Prayoon Cheng
Art Unit
Examiner Name
2625
Jerome Grant, II
5330.07 (SMC)
31
32
/JGI 33
7196722
B2
2007-03-27
White et al.
/JG/
34
7295228
B2
2007-11-13
Roberts et al.
!JG/
35
7242423
B2
2007-07-10
Lin
/JG/ 36
5258837
1993-11-02
Gormley
/JG
37
5479574
1995-12-26
Glier et al.
/JG/ 38
5701398
1997-12-23
Glier et al.
/JG/ 39
6757668
B1
2004-06-29
Goebel et al.
/JG/ 40
5769074
1998-06-23
Bamhill et al.
/JG/ 41
6306087
B1
2001-10-23
Bamhill et al.
EFS Web 2.1.17
/Jerome
Granlt II/
04/11/201,
1.1231353 2005-09-19
Ken Prayoon Cheng
Artnit Art Unit

Examiner Name
2625 2625
Jerome Grant, II
5330.07 (SMC)
/JG/
42
6248063
B1
2001-06-19.
Bamhill et al.
/JG!
43
5835901
1998-11-10
Duvoisin et al.
/JG/ 44
7366174
B2
2008-04-29
MacFaden et al.
/JG/
45 7319796 B1 2008-01-15 Sharp
/JG/ 46
7286687
B2
2007-10-23
Lindwurm et al.
/JG/
47
7277485
B1
2007-10-02
Puri.
/JG, 48
7269591
B2
2007-09-11
Wesinger et al.
/JG/ 49
7213174
B2
. 2007-05-01
Dahlquist et al.
/JG!
50
7127464
B2
2006-10-24
Wesinger etal.
/JG! 51
7028034
B2
2006-04-11
Wesinger et al
/JG 52 52
6976207 6976207
B B1
2005-12-13 2005-12-13
RujBn et al.
et.
EFS Web 2.1.17
/Jerome Grant II/
04/11/2011
11231353 2005-09-19
Ken Prayoon Cheng 2625 Jerome Grant, II


5330.07 (SMC)
/JG/
53
6970513
B1
2005-11-29
Puri et al.
/JG!
54
6968006
B1
2005-11-22
Puri et al.
/JG/ 55
6963899
B1
2005-11-08
Femandez et al.
/JG
56
6909745
B1
2005-06-21
Purl et al.
/JG/
57
6944319
B1
2005-09-13
Huang et al.
/JG/
58
7746380
B2
2010-06-29
Maruya et al.
If you wish to add additional U.S. Patent citation information please click the Add button. U.S.PATENT APPLICATION PUBLICATIONS Examine Cite No Initial* Publication Number Kind Publication Codel Date
Add Remove
Pages,Columns,Lines where Name of Patentee or ApplicantPagesColumnsLines where Relevant Passages or Relevant of cited Document Figures Appear
/JG/
20050222081
Al
2005-10-06
Jakobson et al.
/JG
20080294588
Al
2008-11-27
Morris et al.
/JG/ 3
20100002082
Al
2010-01-07
Buehler et al.
EFS Web 2.1.17
/Jerome Grant II!
04/11/2011
11231353 2005-09-19
Ken Prayoon Cheng

First Named Inventor Art Unit12625

Examiner Name
Jerome Grant, II
5330.07 (SMC)
/JG!/ 4
20030202102
Al
2003-10-30
Shiota et al.
/JG/
20040100563
Al
2004-05-27
Sablak et al.
20030107653
Al
2003-06-12
Utsumi et al.
/JG/
/JG/ 7 20010048765 Al 2001-12-06 Yi et al.
/JG/
20070146484
Al
2007-06-28
Horton et al.
/JG/
20020097322
Al
2002-07-25
Monroe et al.
/JG/ 10
20040001149
Al
2004-01-01
Smith et al.
/JG/ 11
20080218590
Al
2008-09-11
Park et al.
/JG/ 12
20080068464
Al
2008-03-20
Kitagawa et al.
IJGI
13
20080106597
Al
2008-05-08
Amini et al.
/JG/
14
20060279628
Al
2006-12-14
Fleming
EFS Web 2.1.17
/Jerome GrantIll/
04/11/2011
11231353 2005-09-19
Ken Prayoon Cheng
First Named Inventor Art Unit12625 Examiner Name
/JG/ 15
20060221184
Al
2006-10-05
Vallone et al.
/JG/ 16
20070146484
Al
2007-06-28
Horton et al.
/JGi
17
20050132414
Al
2005-06-16
Bentley et al.
/JG/ 18
20080190949
Al
2008-08-14
Ivanov et al.
/JG/ 19
20050052532
Al
2005-03-10
Elooz et al.
/JG/ 20
20080079554
Al
2008-04-03
Boice
/JG/
21
20060017807
Al
2006-01-26
Lee et al.
/JG,
22
20040136574
Al
2004-07-15
Kozakaya et al.
23
20030169908
Al
2003-09-11
Kim et al.
/JG/
24
20050100209
Al
2005-05-12
Lewis et al.
/JG/ 25
20040022442
Al
2004-02-05
Kim et al.
EFS Web 2.1.17
/Jerome Grant II/
04/11/2011
Application Number
Filing Date ,First Named Inventor ArtUnit Examiner Name
2625
/JG/ 26
20060203090
Al
2006-09-14
Wang et al.
/JG/
27
20070154088
Al
2007-07-05
Goh et al.
iJGI
28
20060284978
Al
2006-12-21
Girgensohn et al.
/JG/
29
20080088706
Al
2008-04-17
Girgensohn et al. Add Remove Pages,Columns,Lines where Relevant Passages or Relevant Appear
If you wish to add additional U.S. Published Application citation information please click the Add button FOREIGN PATENT DOCUMENTS Examiner Cite Foreign Document No Number 3 Initial* Country Code 2 i Kind Publication Code4 Date Name of Patentee'or Applicant of cited
DocumentFigures
/JG/ 1
2007/044037
WO
Al
2007-04-19
PROXIMEX CORPORATIONMX Add Remove
If you wish to add additional Foreign Patent Document citation information please click the Add button NON-PATENT LITERATURE DOCUMENTS Examiner Cite Initials* No
Include name of the author (in CAPITAL LETTERS), title of the article (when appropriate), title of the item (book, magazine, journal, serial, symposium, catalog, etc), date, pages(s), volume-issue number(s), publisher, city and/or country where published.
Ts
.JG/ 1
PCT/US05/44656 Intemational Search Report and Written Opinion, June 26, 2006
iJG/
PCT/US05/43808 International Search Report and Written Opinion, October 10, 2007
/Jerome Grant I/l

EFS Web 2.1.17
04/11/2011
11231353 2005-09-19
Ken Prayoon Cheng
First Named Inventor Art Unit2625

Examiner Name
Jerome Grant, II
5330.07 (SMC)
/JG/
26, 2006 PCT/US05/33378 Intemational Search Report and Written Opinion, April
PCT/US05/33750 Intemational Search Report and Written Opinion, May 2, 2007
IJG!
/JG/
PCT/US05/16961 Intemational Search Report and Written Opinion, October 17, 2006
/JG/
VidSys, Inc. Complaint filed in the US District Court for the Eastem district of Virginia on October 19,
/JGI
2003 Redstone Integrated Solutions Documents,
951001,525 Reexamination Request for 7,777,783, filed January 21, 2011
If you wish to add additional non-patent literature document citation information please click the Add button EXAMINER SIGNATURE Examiner Signature
/Jerome Grant II/
Date Considered
04/11/2011
*EXAMINER: Initial if reference considered, whether or not citation is in conformance with MPEP 609. Draw line through a citation if not in conformance and not considered. Include copy of this form with next communication to applicant.
1See Kind Codes of USPTO Patent Documents at www.USPTO.GOV or MPEP 901.04. 2 Enter office that issued the document, by the two-letter code (WIPO Standard ST.3). 3 For Japanese patent documents, the indication of the year of the reign of the Emperor must precede the serial number of the patent document possible. 5Applicant is to place a check mark here if 4Kind of document by the appropriate symbols as indicated on the document under WIPO Standard ST.16 if English language translation is attached.
EFS Web 2.1.17

P.O. Box 1450 Alexandria, Virginia 22313-1450 www.uspto.gov
NOTICE OF ALLOWANCE AND FEE(S) DUE

23308 7590 04/18/2011 EXAMINER GRANT II, JEROME ART UNIT 2625 DATE MAILED: 04/18/2011 PAPER NUMBER
PETERS VERNY , L.L.P. 425 SHERMAN AVENUE SUITE 230 PALO ALTO, CA 94306
APPLICATION NO. 11/231,353 SYSTEMS
FILING DATE 09/19/2005
FIRST NAMED INVENTOR Ken Prayoon Chcng
TITLE OF INVENTION: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE
APPLN. TYPE
SMALL ENTITY
ISSUE FEE DUE
PUBLICATION FEE DUE
PREV. PAID ISSUE FEE
TOTAL FEE(S) DUE
DATE DUE
nonprovisional
YES
$755
$300
$0
$1055
07/18/2011
THE APPLICATION IDENTIFIED ABOVE HAS BEEN EXAMINED AND IS ALLOWED FOR ISSUANCE AS A PATENT. PROSECUTION ON THE MERITS IS CLOSED. THIS NOTICE OF ALLOWANCE IS NOT A GRANT OF PATENT RIGHTS. TIHIS APPLICATION IS SUBJECT TO WITHDRAWAL FROM ISSUE AT THE INITIATIVE OF THE OFFICE OR UPON
PETITION BY THE APPLICANT. SEE 37 CFR 1.313 AND MPEP 1308.
THE ISSUE FEE AND PUBLICATION FEE (IF REQUIRED) MUST BE PAID WITHIN THREE MONTHS FROM THE
MAILING DATE OF THIS NOTICE OR THIS APPLICATION SHALL BE REGARDED AS ABANDONED. THIS
STATUTORY PERIOD CANNOT BE EXTENDED. SEE 35 U.S.C. 151. THE ISSUE. FEE DUE INDICATED ABOVE DOES NOT REFLECT A CREDIT FOR ANY PREVIOUSLY PAID ISSUE FEE IN THIS APPLICATION. IF AN ISSUE FEE HAS
PREVIOUSLY BEEN PAID IN THIS APPLICATION (AS SHOWN ABOVE), THE RETURN OF PART B OF THIS FORM
WILL. BE CONSIDERED A REQUEST TO REAPPLY THE PREVIOUSLY PAID ISSUE FEE TOWARD THE ISSUE FEE NOW
DUE. HOW TO REPLY TO THIS NOTICE: I. Review the SMALL ENTITY status shown above. If the SMALL ENTITY is shown as YES, verify your current SMALL ENTITY status: A. If the status is the same, pay the TOTAL FEE(S) DUE shown above. B. If the status above is to be removed, check box 5b on Part B Fee(s) Transmittal and pay the PUBLICATION FEE (if required) and twice the amount of the ISSUE FEE shown above, or If the SMALL ENTITY is shown as NO: A. Pay TOTAL FEE(S) DUE shown above, or B. If applicant claimed SMALL ENTITY status before, or is now claiming SMALL ENTITY status, check box 5a on Part B - Fee(s) Transmittal and pay the PUBLICATION FEE (if required) and 1/2 the ISSUE FEE shown above.
II. PART B - FEE(S) TRANSMITTAL, or its equivalent, must be completed and returned to the United States Patent and Trademark Office (USPTO) with your ISSUE FEE and PUBLICATION FEE (if required). If you are charging the fee(s) to your deposit account, section "4b" of Part B - Fee(s) Transmittal should be completed and an extra copy of the form should be submitted. If an equivalent of Part B is filed, a request to reapply a previously paid issue fee must be clearly made, and delays in processing may occur due to the difficulty in recognizing the paper as an equivalent of Part B. III. All communications regarding this application must give the application number. Please direct all communications prior to issuance to Mail Stop ISSUE FEE unless advised to the contrary.
IMPORTANT REMINDER: Utility patents issuing on applications filed on or after Dec. 12, 1980 may require payment of
maintenance fees. It is patentee's responsibility to ensure timely payment of maintenance fees when due. Page 1 of 3
PTOL-85 (Rev. 02/11)
PART B - FEE(S) TRANSMITTAL Complete and send this form, together with applicable fee(s), to: Mail Mail Stop ISSUE FEE P.O. Box 1450
Alexandria, Virginia 22313-1450 or Fax (571)-273-2885

INSTRUCTIONS: This form should be used for transmitting the ISSUE FEE and PUBLICATION FEE (if required). Blocks 1 through 5 should be completed where appropriate. All further correspondence including the Patent, advance orders and notification of maintenance fees will be mailed to the current correspondence address as indicated unless corrected below or directed otherwise in Block 1, by (a) specifying a new correspondence address; and/or (b) indicating a separate "FEE ADDRESS" for maintenance fee notifications. Note: A certificate of mailing can only be used for domestic mailings of the CURRENT CORRESPONDENCE ADDRESS (Note: Use Block I for any change of address) Fee(s) Transmittal. This certificate cannot be used for any other accompanying papers. Each additional paper, such as an assignment or formal drawing, must have its own certificate of mailing or transmission. 04/18/2011 7590 23308
PETERS VERNY , L.L.P.
Certificate of Mailing or Transmission

I hereby certify that this Fee(s) Transmittal is being deposited with the United States Postal Service with sufficient postage for first class mail in an envelope addressed to the Mail Stop ISSUE FEE address above, or being facsimile
(Depositors name)
SHERMAN SUITE 230

425
AVENUE
(Signature) (Date) APPLICATION NO. 11/231,353 FILING DATE 09/19/2005 FIRST NAMED INVENTOR Ken Prayoon Cheng . ATT'IORNEY DOCKET NO. 5330.07 (SMC) CONFIRMATION NO. 4531
TITLE OF INVENTION: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEMS
APPLN. TYPE
SMALL ENTITY
ISSUE FEE DUE
PUBLICATION FEE DUE
PREV. PAID ISSUE FEE
TOTAL.FEE(S) DUE
DATE DUE
nonprovisional
EXAMINER
YES
$755
ART UNIT
$300
CLASS-SUBCLASS
$0
$1055
07/18/2011
GRANT II, JEROME
2625
348-143000
2. For printing on the patent front page, list (1) the names of up to 3 registered patent attorneys or agents OR, alternatively, (2) the name of a single firm (having as a member a registered attorney or agent) and the names of up to 2 registered patent attorneys or agents. If no name is listed, no name will be printed. 1 2 3
1. Change of correspondence address or indication of "Fee Address" (37 CFR 1.363). Q0 Change of correspondence address (or Change of Correspondence Address form PTO/SB/122) attached. [ "Fee Address" indication (or "Fee Address" Indication form PTO/SB/47; Rev 03-02 or more recent) attached. Use of a Customer Number is required.
3. ASSIGNEE NAME AND RESIDENCE DATA TO BE PRINTED ON THE PATENT (print or type)
PLEASE NOTE: Unless an assignee is identified below, no assignee data will appear on the patent. If an assignee is identified below, the document has been filed for recordation as set forth in 37 CFR 3.11. Completion of this form is NOT a substitute for filing an assignment. (B) RESIDENCE: (CITY and STATE OR COUNTRY) (A) NAME OF ASSIGNEE
Please check the appropriate assignee category or categories (will not be pr;inted on the patent) : 4a. The following fee(s) are submitted: Q Issue Fee Q Publication Fee (No small entity discount permitted)
Individual Q Corporation or other private group entity
Government
4b. Payment of Fee(s): (Please first reapply any previously paid issue fee shown above)
D A check is enclosed. O Payment bycredit card. Form PTO-2038 is attached.

Q The Director is hereby authorized to charge the required fee(s), any deficiency, or credit any (enclose an extra copy of this form). overpayment, to Deposit Account Number
Advance Order - # of Copies
5. Change in Entity Status (from status indicated above)
Q a. Applicant claims SMALL ENTITY status. See 37 CFR 1.27.
O b. Applicant is no longer claiming
SMALL ENTITY status. See 37 CFR 1.27(g)(2).
NOTE: The Issue Fee and Publication Fee (if required) will not be accepted from anyone other than the applicant; a registered attorney or agent; or the assignee or other party in interest as shown by the records of the United States Patent and Trademark Office. Authorized Signature Typed or printed name . Date Registration No.
This collection of information is required by 37 CFR 1.311. The information is required to obtain or retain a benefit by the public which is to file (and by the USPTO to process) an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.14. This collection is estimated to take 12 minutes to complete, including gathering, prepanng, and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Any comments on the amount of time you require to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S. Patent and Trademark Office, U.S. Department of Commerce, P.O. Box 1450, Alexandria, Virginia 22313-1450. DO NOT SEND FEES OR COMPLETED FORMS TO THIS ADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450, Alexandria, Virginia 22313-1450. Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number.
PTOL-85
(Rev.
02/11)
Approved
for use
through
08/31/2013.
OMB 0651-0033

APPLICATION NO. 11/231,353 23308 7590
FILING DATE 09/19/2005

04/18/2011
EXAMINER

425 SHERMAN AVENUE
GRANT II, JEROME
SUITE 230 PALO ALTO, CA 94306
ART UNIT.
2625
PAPER NUMBER
DATE MAILED: 04/18/2011
Determination of Patent Term Adjustment under 35 U.S.C. 154 (b) (application filed on or after May 29, 2000) The Patent Term Adjustment to date is 1127 day(s). If the issue fee is paid on the date that is three months after the mailing date of this notice and the patent issues on the Tuesday before the date that is 28 weeks (six and a half months) after the mailing date of this notice, the Patent Term Adjustment will be 1127 day(s). If a Continued Prosecution Application (CPA) was filed in the above-identified application, the filing date that determines Patent Term Adjustment is the filing date of the most recent CPA. Applicant will be able to obtain more detailed information by accessing the Patent Application Information Retrieval (PAIR) WEB site (http://pair.uspto.gov). Any questions regarding the Patent Term'Extension or Adjustment determination should be directed to the Office of Patent Legal Administration at (571)-272-7702. Questions relating to issue and publication fee payments should be directed to the Customer Service Center of the Office of Patent Publication at 1-(888)-786-0101 or (571)-272-4200.
Page 3 of 3
PTOL-85 (Rev. 02/11)

The Privacy Act of 1974 (P.L. 93-579) requires that you be given certain information in connection with your submission of the attached form related to a patent application or patent. Accordingly, pursuant to the requirements of the Act, please be advised that: (1) the general authority for the collection of this information is 35 U.S.C. 2(b)(2); (2) furnishing of the information solicited is voluntary; and (3) the principal purpose for which the information is used by the U.S. Patent and Trademark Office is to process and/or examine your submission related to a patent application or patent. If you do not furnish the requested information, the U.S. Patent and Trademark Office may not be able to process and/or examine your submission, which may result in termination of proceedings or abandonment of the application or expiration of the patent. The information provided by you in this form will be subject to the following routine uses: 1. The information on this form will be treated confidentially to the extent allowed under the Freedom of Information Act (5 U.S.C. 552) and the Privacy Act (5 U.S.C 552a). Records from this system of records may be disclosed to the Department of Justice to determine whether disclosure of these records is required by the Freedom of Information Act. 2. A record from this system of records may be disclosed, as a routine use, in the course of presenting evidence to a court, magistrate, or administrative tribunal, including disclosures to opposing counsel in the course of settlement negotiations. 3. A record in this system of records may be disclosed, as a routine use, to a Member of Congress submitting a request involving an individual, to whom the record pertains, when the individual has requested assistance from the Member with respect to the subject matter of the record. 4. A record in this system of records may be disclosed, as a routine use, to a contractor of the Agency having need for the information in order to perform a contract. Recipients of information shall be required to comply with the requirements of the Privacy Act of 1974, as amended, pursuant to 5 U.S.C. 552a(m). 5. A record related to an International Application filed under the Patent Cooperation Treaty in this system of records may be disclosed, as a routine use, to the International Bureau of the World Intellectual Property Organization, pursuant to the Patent Cooperation Treaty. 6. A record in this system of records may be disclosed, as a routine use, to another federal agency for purposes of National Security review (35 U.S.C. 181) and for review pursuant to the Atomic Energy Act (42 U.S.C. 218(c)). 7. A record from this system of records may be disclosed, as a routine use, to the Administrator, General Services, or his/her designee, during an inspection of records conducted by GSA as part of that agency's responsibility to recommend improvements in records management practices and programs, under authority of 44 U.S.C. 2904 and 2906. Such disclosure shall be made in accordance with the GSA regulations governing inspection of records for this purpose, and any other relevant (i.e., GSA or Commerce) directive. Such disclosure shall not be used to make determinations about individuals. 8. A record from this system of records may be disclosed, as a routine use, to the public after either publication of the application pursuant to 35 U.S.C. 122(b) or issuance of a patent pursuant to 35 U.S.C. 151. Further, a record may be disclosed, subject to the limitations of 37 CFR 1.14, as a routine use, to the public if the record was filed in an application which became abandoned or in which the proceedings were terminated and which application is. referenced by either a published application, an application open to public inspection or an issued patent. 9. A record from this system of records may be disclosed, as a routine use, to a Federal, State, or local law enforcement agency, if the USPTO becomes aware of a violation or potential violation of law or regulation.

EFSID: Application Number: International Application Number: Confirmation Number: 4531 9933650 11231353
Title of Invention:
ADAPTIVE MU LTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEMS
Ken Prayoon Cheng 23308 Robert Hayden
5330.07 (SMC) 21-APR-2011 19-SEP-2005 17:53:02 Utility under 35 USC 111(a)
Submitted with Payment Payment Type Payment was successfully received in RAM RAM confirmation Number Deposit Account Authorized User yes Deposit Account $1610 4856 161331
File Listing:
Documentument Number Description File ame File Size(Bytes)/ Message Digest Multi Part /.zip Pages (if appl.)
Issue Fee Payment (PTO-85 B).
5330_07-issue-fee.pdf
fee-info.
Warnings: Information: Total Files Size (in bytes) 12?071
This Acknowledgement Receipt evidences receipt on the noted date by the USPTO of the indicated documents, characterized by the applicant, and includingpage counts, where applicable. It serves as evidence of receipt similar to a Post Card, as described in MPEP 503. New Applications Under 35 U.S.C. 111 If a new application is being filed and the application includes the necessary components for a filing date (see 37CFR 1.53(b)-(d) and MPEP 506), a Filing Receipt (37 CFR 1.54) will be issued in due course and the date shown on this Acknowledgement Receipt will establish the filing date of the application. National Stage of an International Application under 35 U.S.C. 371 If a timely submission to enter the national stage of an international application is compliant with the conditions of 35 U.S.C. 371 and other applicable requirements a Form PCT/DO/EO/903 indicating acceptance of the application as a national stage submission under 35 U.S.C. 371 will be issued in addition to the Filing Receipt, in due course. New International Application Filed with the USPTO as a Receiving Office If a new international application is being filed and the international application includes the necessary components for an international filing date (see PCT Article 11 and MPEP 1810), a Notification of the International Application Number and of the International Filing Date (Form PCT/RO/105) will be issued in due course, subject to prescriptions concerning national security, and the date shown on this Acknowledgement Receipt will establish the international filing date of the application.
PART B - FEE(S) TRANSMITTAL
Complete and send this form, together with applicable fee(s), to: Mail Mail Stop ISSUE FEE Commissioner for Patents P.O. Box 1450 Alexandria, Virginia 22313-1450 or Fax (571)-273-2885
INSTRUCTIONS: This form should be used for transmitting the ISSUE FEE and PUBLICATION FEE (if required). Blocks 1 through 5 should be completed where appropriate. All further correspondence including the Patent, advance orders and notification of maintenance fees will be mailed to the current correspondence address as indicated unless corrected below or directed otherwise in Block 1, by (a) specifying a new correspondence address; and/or (b) indicating a separate "FEE ADDRESS" for maintenance fee notifications. CURRENT CORRESPONDENCE ADDRESS (Note: Use Block I for any change of address) .Note: A certificate of mailing can only be used for domestic mailings of the Fee(s) Transmittal. This certificate cannot be used for any other accompanying papers. Each additional paper, such as an assignment or formal drawing, must have its own certificate of mailing or transmission. 7590 04/18/2011 23308 Certificate of Mailing or Transmission PETERS VERNY , L.L.P. I hereby certify that this Fee(s) Transmittal is being deposited with the United 425 SHERMAN AVENUE States Postal Service with sufficient postage for first class mail in an envelope addressed to the Mail Stop ISSUE FEE address above, or being facsimile SUITE 230 transmitted to the USPTO (571) 273-2885, on the date indicated below.
PALO ALTO, CA 94306
(Depositor's name) (Signature) (Date) APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO.
11/231,353
09/19/2005
Ken Prayoon Cheng
5330.07 (SMC)
4531
TITLE OF INVENTION: ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND .SURVEILLANCE SYSTEMS
APPLN. TYPE nonprovisional
SMALL ENTITY YES
ISSUE FEE DUE $755 ..
PUBLICATION FEE DUE I PREV. PAID ISSUE FEE $300 $0
TOTAL FEE(S) DUE $1055
DATE DUE 07/18/2011
EXAMINER GRANT II, JEROME
ART UNIT 2625
CLASS-SUBCLASS 348-143000
2. For printing on the patent front page, list (1) the names of up to 3 registered patent attorneys or agents OR, alternatively, (2) the name of a single firm (having as a member a registered attorney or agent) and the names of up to 2 registered patent attorneys or agents. If no name is listed, no name will be printed. I 2 3
1. Change of correspondence address or indication of "Fee Address" (37 CFR 1.363). address (or Change of Correspondence Address form PTO/SB/122) attached. "Fee Address" indication (or "Fee Address" Indication form PT1O/SB/47; Rev 03-02 or more recent) attached. Use of a Customer Number is required.
Peters
O Change of correspondence O
Verny,
LLP
3. ASSIGNEE NAME AND RESIDENCE DATA TO BE PRINTED ON THE PATENT (print or type)
PLEASE NOTE: Unless an assignee is identified below, no assignee data will appear on the patent. If an assignee is identified below, the document has been filed for recordation as set forth in 37 CFR 3.11. Completion of this form is NOT a substitute for filing an assignment.
(A) NAME OF ASSIGNEE
(B) RESIDENCE: (CITY and STATE OR COUNTRY)
Proximex Corporation
Sunnyvale, CA 0
Individual
Please check the appropriate assignee category or categories (will not be printed on the patent) : 4a. The following fee(s) are submitted: SIssue Fee [Publication Fee (No small entity discount permitted)
Q Corporation or other private group entity O Government
4b. Payment of Fee(s): (Please first reapply any previously paid issue fee shown above)
O Advance
O A check is enclosed. O Payment by credit card. Form PTO-2038
Order - # of Copies
is attached. [_[The Director is hereby authorized to charge the required fee(s), any deficiency, or credit any overpayment, to Deposit Account Number -1-64- 33-1-- (enclose an extra copy of this form). b. Applicant is no longer claiming SMALL ENTITY status. See 37 CFR 1.27(g)(2).
5. Change in Entity Status (from status indicated above) Q a. Applicant claims SMALL ENTITY status. See 37 CFR 1.27. NO1TE: The Issue Fee and Publication Fee (if required) will not be accepted from anyone other than the applicant; a registered attorney or agent; or the assignee or other party in interest as shown by the records of the United States Patent and Trademark Office. Authorized Signature
/Robert Hayden, #42,645/
Date
Registration No.
April 21, 2011 42,645
Typed or printed name
Robert Hayden
'This
collection of information is required by 37 CFR 1.311. The information is required to obtain or retain a benefit by the public which is to file (and by the USPTO to process) an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.14. This collection is estimated to take 12 minutes to complete,.including gathering, preparing, and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Any comments on the amount of time you require to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S. Patent and Trademark Office, U.S. Department of Commerce, P.O.
Box 1450, Alexandnria, Virginia 22313-1450. DO NOT SEND FEES OR COMPLETED FORMS TO THIS ADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450, Alexandria, Virginia 22313-1450.
Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number.
PTOL-85
(Rev.
02/11)
Approved
for use
through
0831/2013.
OMB 0651-0033

Title of Invention:

Filer:
Ken Prayoon Cheng

Robert Hayden
Attorney Docket Number: Filed as Large Entity
5330.07 (SMC)

Description Fee Code Quantity Amount
Sub-Total in
USD($)
Sub-Total in
Basic Filing:
Pages:
Claims:
Petition:
Utility Appl issue fee Certificate of correction 1501 1811 1 1 1510 100 1510 100
Description
Fee Code
Quantity
Amount
Sub-Total in USD($)
USD($)
Extension-of-Time:
Miscellaneous:
Total in USD ($)
1610

Title of Invention:
Ken Prayoon Cheng 23308 Steven M.Colby/Melinda Tompkins Steven M.Colby 5330.07 (SMC) 29-APR-2011 19-SEP-2005 12:30:32 Utility under 35 USC 111(a)
Submitted with Payment Payment Type Payment was successfully received in RAM RAM confirmation Number Deposit Account Authorized User The Director of the USPTO is hereby authorized to charge indicated fees and credit any overpayment as follows: Charge any Additional Fees required under 37 C.F.R. Section 1.21 (Miscellaneous fees and charges) yes Deposit Account $300 13122 161331
File Listing:
cument D
Number
cument Description
File Name
File Size(Bytes)/
Message Digest
30395
Multi
Part /.zip
Pages
(if appl.)
fee-info.pdf
4dcaSOf2177504634c6901126a1129c4603 188ff
no
Warnings: Information: Total Files Size (in bytes): 30395

Application Number: Filing Date: 1123,1353 19-Sep-2005
Title of Invention:
'ADAPTIVE MU LTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEMS
First Named Inventor/Applicant Name: Filer: Attorney Docket Number: Filed as Large Entity Utility under 35 USC 111(a) Filing Fees
Ken Prayoon Cheng Steven M. Colby/Melinda Tompkins 5330.07 (SMC)
Description
Fee Code
Quantity
Amount
Sub-Total in USD($)
Basic Filing:
Pages:
Claims:
Petition:
Patent-Appea Is-and-Interference:
Post-Allowance-and-Post-Issuance: Publ. Fee- early, voluntary, or normal Extension-of-Time: 1504 1 300 300
Description
Fee Code
Quantity
Amount
Sub-Total in USD($)
USD($)
Miscellaneous:
Total in USD ($)
300
entitled "Robust Perceptual Color Identification," invented by K. Goh, E. Y. Chang and Y. F Wang, which is expressly incorporated by reference in its entirety into this application through this reference. This patent application addresses a problem of . camera-based sensors perceiving an article of clothing as having a slightly different color when viewed from. different angles or under different lighting conditions. The patent application proposes the representing color:of an article of clothing using a "robust perceptual color". [0078] Data from different modalities may be fused by the Knowledge Services for
classification and identification purposes without suffering the "curse of dimensionality using techniques taught in commonly assigned co-pending U. S. Patent Application Serial No. 11/129,090, filed May 13, 2005, entitled, Multimodal High-Dimensional Data Fusion for Classification and Identification, invented by E. Y. Chang, which is expressly incorporated herein in its entirety by this reference. Data may be incrementally added to a classification and identification process by the Knowledge Services using techniques
applied i 1/zo230932z
Change(s) /A. .M.,
to document, +/z/zo 1i
taught by commonly assigned co-pending U. S. Patent Application Serial No.,
filed
September 19, 2005, entitled, Incremental Data Fusion and Decision Making, invented by Yuan-Fang Wang, which is expresslyincorporated herein in its entirety by this reference. 10079] While the invention has been described with reference to various illustrative
features, aspects and embodiments, it will be appreciated that the invention is susceptible of various modifications and other embodiments, other than those specifically shown and described. The invention is therefore to be broadly construed as including all such alternative variations, modifications and other embodiments within the spirit and scope as hereinafter claimed.
26
11231353 2005-09-19
Ken Prayoon Cheng
First Named Inventor ArtUnit

Examiner Name
2625
Jerome Grant, II
5330.07 (SMC)
IJG/
15
20060221184
Al
2006-10-05
Vallone et al.
/JG/ 16
20070146484
Al
2007-06-28
Horton et al.
/JG/
17
20050132414
Al
2005-06-16
Bentley et al.
a 1 ied angazooso
1998994. 3099
Al
.009
4084 Ivanov,et al.
o6/zoo8
.J.F/
/20/J/ 19 20050052532 Al 2005-03-10 Elooz et al.
/JG/ 20
20080079554
Al
2008-04-03
Boice
/JG/ 21
20060017807
Al
2006-01-26
Lee et al.
/JG: 22
20040136574
Al
2004-07-15
Kozakaya et al.
23
20030169908
Al
2003-09-11
Kim et al.
/JG/ 24
20050100209
Al
2005-05-12
Lewis et al.
/JG/
25
20040022442
Al
2004-02-05
Kim et al.
EFS Web 2.1.17
/Jerome Grant II/
04/11/2011

APPLICATION NO.
ISSUE DAE
PATENT NO.
A'IORNEY DOCKET NO.
CONFIRMATION NO.
11/231,353
23308 7590
06/07/2011
05/18/2011
7956890
5330.07 (SMC)
4531
425 SHERMAN AVENUE SUITE 230 PALO ALTO, CA 94306
ISSUE NOTIFICATION The projected patent number and issue date are specified above. Determination of Patent Term Adjustment under 35 U.S.C. 154 (b) (application filed on or after May 29, 2000) The Patent Term Adjustment is 1517 day(s). Any patent to issue from the above-identified application will include an indication of the adjustment on the front page. If a Continued Prosecution Application (CPA) was filed in the above-identified application; the filing date that determines Patent Term Adjustment is the filing date of the most recent CPA. Applicant will be able to obtain more detailed information by accessing the Patent Application Information Retrieval (PAIR) WEB site (http://pair.uspto.gov). Any questions regarding the Patent Term Extension or Adjustment determination should be directed to the Office of Patent Legal Administration at (571)-272-7702. Questions relating to issue and publication fee payments should be directed. to the Application Assistance Unit (AAU) of the Office of Data Management
(ODM) at (571)-272-4200.
APPLICANT(s) (Please see PAIR WEB site http://pair.uspto.gov for additional applicants): Ken Prayoon Cheng, Saratoga, CA; Edward Y. Chang, Santa Barbara, CA;
Yuan-Fang Wang, Goleta, CA;
IR103 (Rev. 10/09)
PTO/SB/06 (12-04) Approved for use through 7/31/2006. OMB 0651-0032 U.S. Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE Under the Paperwork Reduction Act of 1995. no persons are required to respond to a collection of information unless itdisplays a valid OMB control number.
PATENT APPLICATION FEE DETERMINATION RECORD

Substitute for Form PTO-875
Application or Docket Number
11231353
SMALL ENTITY
RATE ($) FEE (S)
APPLICATION AS FILED - PART I

(Column 1)
FOR BASIC FEE (37 CFR 1.16(a), (b), or (c)) SEARCH FEE (37 CFR 1.16(k). (i).or (m)) EXAMINATION FEE (37 CFR 1.16(o), (p),or (q)) TOTAL CLAIMS (37 CFR 1.16(i)) INDEPENDENT CLAIMS (37 CFR 1.16(h)) APPLICATION SIZE FEE (37 CFR 1.16(s)) NUMBER FILED
(Column 2)
NUMBER EXTRA
OR
OTHER THAN SMALL ENTITY

RATE (5) FEE (S)
150 250 100 20 3

minus 20 = minus 3 = If the specification and drawings exceed 100 sheets of paper, the application size fee due is 5250 ($125 for small entity) for each additional 50 sheets or fraction thereof. See 35 U.S.C. 41(a)(1)(G) and 37 CFR
X$ 25= X$100=
OR
X$50= X$200=
MULTIPLE DEPENDENT CLAIM PRESENT (37 CFR 1.160)) If the difference in column 1 is less than zero, enter "0" in column 2.
NIA TOTAL
500
TOTAL
APPLICATION AS AMENDED - PART II

(Column 1) CLAIMS REMAINING AFTER AMENDMENT Total Minus (37 CFR 1.16(I)) Independent Minus (37 CFR1.16(h)) Application Size Fee (37 CFR 1.16(s)) FIRST PRESENTATION OF MULTIPLE DEPENDENT CLAIM (37 CFR 1.16()) (Column 2) HIGHEST NUMBER PREVIOUSLY PAID FOR (Column 3) PRESENT EXTRA X x
SMALL ENTITY
RATE ($) ADDITIONAL FEE (S)
OR
OTHER THAN SMALL ENTITY

RATE (5) ADDITIONAL FEE (5)
SI Z W Z w
= =
OR OR OR
X
X
=
=
<
TOTAL ADD'T FEE
I OR
N/A TOTAL ADD'T FEE
(Column 1) m
CLAIMS REMAINING AFTER AMENDMENT
(Column 2)
HIGHEST NUMBER PREVIOUSLY PAID FOR
(Column 3)
PRESENT EXTRA RATE (5) ADDITIONAL FEE (5)
SOR
ADDIRATE (5)
TIONAL
FEE (S)
z go w z w
Q
Total
(37 CFR 1.16(i))
Minus
Minus
=
=
X X
= =
OR OR OR OR
X X
= =
Independent
(37 CFR 1.16(h))
inus
Application Size Fee (37 CFR 1.16(s))

FIRST PRESENTATION OF MULTIPLE DEPENDENT CLAIM (37 CFR 1.16(j))
--
N/A TOTAL
ADD'T FEE
N/A
TOTAL ADD'T FEE
If the entry in column 1 is less than the entry in column 2, write *0' in column 3. If the "Highest Number Previously Paid For' IN THIS SPACE is less than 20, enter "20'. If the "Highest Number Previously Paid For INTHIS SPACE is less than 3, enter'3'. The "Highest Number Previously Paid For" (Total or Independent) is the highest number found inthe appropriate box in column 1. This collection of information is required by 37 CFR 1.16. The information is required to obtain or retain a benefit by the public which is to file (and by the USPTO to process) an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR 1.14. This collection is estimated to take 12 minutes to complete, including gathering, preparing, and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Any comments on the amount of time you require to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S. Paten and Trademark Office, U.S. Department of Commerce, P.O. Box 1450, Alexandria, VA 22313-1450. DO NOT SEND FEES OR COMPLETED FORMS TO THIS ADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450, Alexandria, VA 22313-1450. Ifyou need assistance in completingthe form, call 1-800-PTO-9199 and select option 2.
PATENT APPLICATION FEE DETERMINATION RECORD

Substitute for Form PTO-875
Application or Docket Number
Filing Date
11/231,353
09/19/2005
O To be Mailed
APPLICATION AS FILED - PART I.

(Column 1) MOMI
NUMBER FILED
OTHER THAN
(Column 2) SMALL ENTITY I
RATE ($) FEE ($)
OR
r-,
SMALL ENTITY
RATE ($) FEE ($)
NUMBER EXTRA
O O O
BASIC FEE
(37 CFR 1.16a), (b , or (c))
N/A N/A N/A

minus 20 = minus 3 =
N/A N/A N/A

X $ X
N/A
SEARCH FEE
(37 CFR 1.16(k)l (i, or (m))
EXAMINATION FEE
(37 CFR 1.16(o), (p),or (q))
TOTAL CLAIMS (37 CFR 1.16i)) INDEPENDENT CLAIMS (37 CFR 1.16h))
X $
OAPPLICATION
SIZE FEE (37 CFR 1.16(s))
If the specification and drawings exceed 100 sheets of paper, the application size fee due is $250 ($125 for small entity) for each additional 50 sheets or fraction thereof. See 35 U.S.C. 41(a)(1)(G) and 37 CFR 1.16(s).
O MULTIPLE DEPENDENT CLAIM PRESENT (37 CFR 1.16(j))

in * If the difference in column 1 is less than zero, enter "0" column 2. TOTAL
1
OR
TOTAL
APPLICATION AS AMENDED - PART II

OTHER THAN
(Column 1) CLAIMS (Column 2) HIGHEST NUMBER (Column 3) PRESENT
SMALL ENTITY
RATE ($) ADDITIONAL FEE ($)
SMALL ENTITY RATE ($) ADDITIONAL FEE ($)
02/01/2011 02/01/201
Z S Total 37 CFR
REMAINING
AFTER
PREVIOUSLY
PAID FOR Minus
Minus
EXTRA
AMENDMENT
2
*2
=*20
= 0
= =
X $26 =
SIndependent 37nCFR1. Z
Minus
-3
X $110=
0 0
OR
OR
$x
X $
=
=
E U
Application Size Fee (37 CFR 1.16(s)) FIRST PRESENTATION OF MULTIPLE DEPENDENT CLAIM (37 CFR 1.16(j)) TOTAL ADD'L FEE (Column 1) CLAIMS REMAINING AFTER AMENDMENT (Column 2) HIGHEST NUMBER PREVIOUSLY PAID FOR Minus
Minus "**-
OR
OR
TOTAL ADD'L FEE
(Column 3) MMMM9
0
ADDITIONAL FEE ($)
Total (37 CFR

independent
37.CFR 1.16(h)}
2
(
PRESENT EXTRA
RATE ($)
X$ X$
= =
Z1 Q
l Application Size Fee (37 CFR 1.16(s))
FIRST PRESENTATION OF MULTIPLE DEPENDENT CLAIM (37 CFR 1.16(j)) ADD'L FEE OR TOTAL ADD'L FEE
* If the entry in column 1 is less than the entry in column 2,write "0" column 3. in **If "Highest Number Previously Paid For" IN THIS SPACE is less than 20, enter "20". the
Legal Instrument Examiner: /SHARON WEST/
*"* the "Highest Number Previously Paid For" IN THIS SPACE is less than 3, enter "3". If The "Highest Number Previously Paid For" (Total or Independent) is the highest number found in the appropriate box in column 1. This collection of information is required by 37 CFR 1.16. The information is required to obtain or retain a benefit by the public which is to file (and by the USPTO to process) an application. Confidentiality is governed by 35 U.S.C. 122 and 37CFR 1.14. This collection is estimated to take'12 minutes to complete, including gathering, preparing, and submitting the completed application form to the USPTO. Time will vary depending upon the individual case. Any comments on the amount of time you require to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S. Patent and Trademark Office, U.S. Department of Commerce, P.O. Box 1450, Alexandria, VA 22313-1450. DO NOT SEND FEES OR COMPLETED FORMS TO THIS
ADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450, Alexandria, VA 22313-1450.
Table of Contents
MPI Family Report

Report Created Date: 2011-08-05 Name of Report: Number of Families: 1 Comments:
(Family Bibliographic and Legal Status)
In the MPI Family report, all publication stages are collapsed into a single record, based on identical application data. The bibliographic information displayed in the collapsed record is taken from the latest publication.
Table of Contents
1. US7956890B2 20110607 PROXIMEX CORP US Adaptive multi-modal integrated biometric identification detection and surveillance systems ......................... 2
MicroPatent Patent Index - an enhanced INPADOC database
Family Bibliographic and Legal Status
Family1 4 records in the family, collapsed to 2 records. WO2006034135A3 20060713 WO2006034135A2 20060330
(ENG) ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND SURVEILLANCE SYSTEM Assignee: PROXIMEX US
Inventor(s): CHENG KEN PRAYOON US ; CHANG EDWARD Y US ; WANG YUAN FANG US Application No: US 2005033378 W Filing Date: 20050919 Issue/Publication Date: 20060713 Abstract: (ENG) <emi file="US2005033378_13072006_pf_fp.g4" he="199MM" wi="271MM"/><p>A surveillance system is provided that includes at least one sensor disposed in a security area of a surveillance region to sense an occurrence of a potential security breach event; a plurality of cameras is disposed in the surveillance region; at least one camera of the plurality has a view of the security area and can be configured to automatically gather biometric information concerning at least one subject person in the vicinity of the security area in response to the sensing of a potential security breach event; one or more other of the plurality of cameras can be configured to search for the at least one subject person; a processing system is programmed to produce a subject dossier corresponding to the at least one subject person to match biometric information of one or more persons captured by one or more of the other cameras with corresponding biometric information in the subject dossier. </p> Priority Data: US 61099804 20040917 P; Related Application(s): 20060713 200628 3 R4 G06K00900; H04N00947; G06K00964; H04N00718; H04B01700 G06K00962F3M; G08B013196; H04N00718C
IPC (International Class): ECLA (European Class):
Designated Countries: ----Designated States: AE AG AL AM AT AU AZ BA BB BG BR BW BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE EG ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KM KP KR KZ LC LK LR LS LT LU LV LY MA MD MG MK MN MW MX MZ NA NG NI NO NZ OM PG PH PL PT RO RU SC SD SE SG SK SL SM SY TJ TM TN TR TT TZ UA UG US UZ VC VN YU ZA ZM ZW ----Regional Treaties: AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LT LU LV MC NL PL PT RO SE SI SK TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG BW GH GM KE LS MW MZ NA SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM Publication Language: ENG Agent(s): DURANT, Stephen, C. et al. Morrison & Foerster LLP, 425 Market Street, San Francisco, Ca 94105-2482 US
Legal Status: There is no Legal Status information available for this patent
Family Bibliographic and Legal Status
US7956890B2 20110607 US2006093190A1 20060504

(ENG) Adaptive multi-modal integrated biometric identification detection and surveillance systems Assignee: PROXIMEX CORP US
Inventor(s): CHENG KEN P US ; CHANG EDWARD Y US ; WANG YUAN-FANG US Application No: US 23135305 A Filing Date: 20050919 Issue/Publication Date: 20110607 Abstract: (ENG) A surveillance system is provided that includes at least one sensor disposed in a security area of a surveillance region to sense an occurrence of a potential security breach event; a plurality of cameras is disposed in the surveillance region; at least one camera of the plurality has a view of the security area and can be configured to automatically gather biometric information concerning at least one subject person in the vicinity of the security area in response to the sensing of a potential security breach event; one or more other of the plurality of cameras can be configured to search for the at least one subject person; a processing system is programmed to produce a subject dossier corresponding to the at least one subject person to match biometric information of one or more persons captured by one or more of the other cameras with corresponding biometric information in the subject dossier. Priority Data: US 23135305 20050919 A Y; US 61099804 20040917 P Y; Related Application(s): 11/231353 20050919 20060093190 H04N00947; G06K00900 G06K00900V4 US; 60/610998 20040917 US
IPC (International Class): ECLA (European Class):
US Class: 348143; 382115; 382119; 382118 Publication Language: ENG Filing Language: ENG Agent(s): Peters Verny, LLP
Examiner Primary: Grant, II, Jerome Assignments Reported to USPTO: Reel/Frame: 17473/0506 Date Signed: 20051206 Date Recorded: 20060113 Assignee: PROXIMEX 6 RESULTS WAY CUPERTINO CALIFORNIA 95014 Assignor: CHENG, KEN P.; CHANG, EDWARD Y.; WANG, YUAN-FANG Corres. Addr: STEPHEN C. DURANT MORRISON & FOERSTERLLP 425 MARKET STREET SAN FRANCISCO, CALIFORNIA 94105-2482 Brief: ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Legal Status: Date 20060113
+/()
Code AS
Description New owner name: PROXIMEX, CALIFORNIA; : ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CHENG, KEN P.;CHANG, EDWARD Y.;WANG, YUAN-FANG;SIGNING DATES FROM 20051206 TO 20051215;REEL/FRAME:017473/0506;
USPTOMaintenanceReport
Patent Bibliographic Data Patent Number: Issue Date: Title: Status: Window Opens: Fee Amt Due: Fee Code: Surcharge Fee Code: Most recent events (up to 7): Address for fee purposes: No Maintenance History Found --- End of Maintenance History --PETERS VERNY , L.L.P. 425 SHERMAN AVENUE SUITE 230 PALO ALTO CA 94306 7956890 06/07/2011 Application Number: Filing Date:
08/05/2011 12:33 PM 11231353 09/19/2005
ADAPTIVE MULTI-MODAL INTEGRATED BIOMETRIC IDENTIFICATION DETECTION AND 4th year fee window opens: 06/07/2014 06/07/2014 Window not open 1551 Surcharge Date: Surchg Amt Due: 12/09/2014 Window not open Entity: Expiration: Total Amt Due: Large N/A Window not open
MAINTENANCE FEE DUE AT 3.5 YEARS

Application NO: Filed: Issued

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Application NO: Filed: Issued

Caricato da

Copyright:

Formati disponibili

FILE HISTORY US 7,956,890

7,956,890 Cheng, Ken P. Chang, Edward Y. Wang, Yuan-Fang

Adaptive multi-modal integrated biometric identification detection and surveillance systems

APPLICATION US2005231353A NO: FILED: ISSUED: COMPILED:

19 SEP 2005 07 JUN 2011 05 AUG 2011

Applicant(s)/Patent Under Reexamination CHENG ET AL.

SUBCLASS (ONE SUBCLASS PER BLOCK)

Claims renumbered in the same order as presented by applicant

2 (Date) 4-11-11 (Date) O.G. Print Claim(s) 1

O.G. Print Figure 6 and 7

Applicant(s)/Patent Under Reexamination

O Claims renumbered in the same order as presented by applicant

U.S. Patent and Trademark Office

Partof PaperNo.: 20100510

Applicant(s)/Patent Under Reexamination

Claims renumbered in the same order as presented by applicant

05/10/2010 04/11/2011 REJ

U.S. Patent and Trademark Office

Part of Paper No. : 20110411

SEARCHED Class 713

Date 5-10 '__ '

Examiner JG ' ' ' ' '

' 707 382

' ' '

' ' "

155 161 169 227 215 203 315

U.S. Patent and Trademark Office

INTERFERENCE SEARCH Class Subclass Date Examiner

U.S. Patent and Trademark Office

Part of Paper No.: 20100510

U.S. Patent and Trademark Office

Examiner JG ' ' ' ' ' ' '

340 358 ' 707 382

' ' '

' ' ' '

' 345 348

709 ' ' 382

U.S. Patent and Trademark Office

Part of Paper No.: 20110411

EAST History Search

OR USFPGPUB; USPAT; USOCR; FPRS;c EPO; JPO; DERWENT; IBMTDB

EAST Search History

EAST Search History EAST Search History (Prior Art)

USPGPUB USPAT; USOCR; FPRS; EPO; JPO; DERWENT; I BMTDB

.. ..................... ....... 2010/04/19

USOCR; FPRS; EPO; JPO; DERWENT; IBMTDB

client or object or subject or people or terrorist or man or woman or

11231353_AccessibleVersion.htm (1 of 3)4/19/10 1:16:29 PM

EAST Search History

file:///CI/Docunents%20and%20Settings/jgrant/My%20Docu...31353/EASTSearchHistory. 11231353_AccessibleVersion.htm (2 of 3)4/19/10 1:16:29 PM

EAST Search History

file:///CI/Docwnents%20and%20Setings/jgrant/My%20Docu...31353/EASTSearchHistory. 11231353_AccessibleVersion.htm (3 of 3)4/19/10 1:16:29 PM

EAST Search History

EAST Search History EAST Search History (Prior Art) Ref #

PluraIs Time Stamp

.... . .......... .r.or................ . ..... . ....... . . ...e.. . . . "u . .. . . i6... .............

US-PGPUB,; USPAT; USOCR; :FPRS; EPO; JPO; DERWENT; I;IBMTDB

EAST Search History

and @ad<"20071213" and

..... . .... . ..... ..............

.... ....... . .. .....

... .: ...... .................................... 08:58

11231353AccessibleVersion.htm (2 of 26)4/11/11 12:42:38PM