cerias talk on testing and evaluation
DESCRIPTION
This slidedeck highlights the recent tech talk for CERIAS.TRANSCRIPT
![Page 1: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/1.jpg)
BIOMETRICS – TESTING
AND EVALUATION
![Page 2: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/2.jpg)
• Traditional biometric testing
• Algorithm testing
• Well established metrics
• Well understood testing methodologies
• Operational testing
• Harder to do
• Access to environments
• Test methodologies dependent in some cases on the test
BIOMETRIC TESTING
![Page 3: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/3.jpg)
•Essentially trying to understand how a system
performs
• Maybe more fundamentally – who or what is
causing the errors?
TESTING AND EVALUATION
![Page 4: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/4.jpg)
PROBABLY NOT MISSING,
BUT HARD TO DO
![Page 5: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/5.jpg)
• There have been several papers on the contribution of individual error on performance
• What causes these errors?
• There are some papers that examine meta-data and the contribution of variables (age) and examines the training of algorithms
GAPS IN BIOMETRIC TESTING
![Page 6: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/6.jpg)
•Training
• How do users get accustomed to devices?
• Can they remember how to use them?
• How do we provide good training to the users that has a consistent message?
GAPS IN BIOMETRIC TESTING
![Page 7: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/7.jpg)
• Accessibility
• How many people know people who have problems with interacting with a biometric systems?
• How do we deal with accessibility and usability issues?
• Hearing and sight issues
GAPS IN BIOMETRIC TESTING
![Page 8: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/8.jpg)
• Human Factors
• Testing and evaluating biometric systems by looking at how the users interact with the system
• Are performance results different in an operational environment than collect in a lab?
• Are these performance results due to the environment?
GAPS IN BIOMETRIC TESTING
![Page 9: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/9.jpg)
• Is the error always subject centric?
• The role of the device?
• The role of the operator?
GAPS IN BIOMETRIC TESTING
![Page 10: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/10.jpg)
FILLING IN THOSE GAPSA WORK IN PROGRESS
![Page 11: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/11.jpg)
• Devices are:
• Now more complex
• Variety of form factors, deployments, applications
• In the hands of more customers
• They become more demanding
• Technology development cycle is short
WORK IN PROGRESS
![Page 12: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/12.jpg)
• What are biometrics?
• Why should we test?
• Complexities of testing today
• Examples of testing
• Operational / Scenario / Hybrid
• Our approach to testing
• Upcoming tests
TODAY – AN UPDATE TO TESTING AND
EVALUATION
![Page 13: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/13.jpg)
•How do we identify people?
•As we move around, the need to identify
individuals increases
INITIAL CONCEPTS
![Page 14: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/14.jpg)
•Biometric – of or having to do with biometrics
•Biometrics – automated recognition of
individuals based on their behavioral and
biological characteristics
TWO MAIN DEFINITIONS
![Page 15: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/15.jpg)
•Biometrics appear in a number of different
environments:
• Large scale government programs
• Individual identification on a laptop
INITIAL CONCEPTS
![Page 16: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/16.jpg)
•Border control – US VISIT and Global Entry
•Apple fingerprint sensors
TWO EXAMPLES
![Page 17: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/17.jpg)
•How do we test and evaluate these products to
make sure that we have good performance?
TWO VERY DIFFERENT SCENARIOS
![Page 18: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/18.jpg)
• International Center for Biometric Research, and its predecessors have been testing and evaluating biometric products for over 14 years.
• The devices have changed over time, but the needs of testing and evaluation haven’t
• Companies want to test and evaluate products to prove that they will work in the marketplace
ICBR HISTORY
![Page 19: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/19.jpg)
•Anyone know what this
is?
BIOMETRIC DEVICES AND
MODALITIES
![Page 20: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/20.jpg)
• Iris camera
•Now what year was this
launched?
BIOMETRIC DEVICES AND
MODALITIES
![Page 21: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/21.jpg)
• Iris camera
•2001
BIOMETRIC DEVICES AND
MODALITIES
![Page 22: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/22.jpg)
• We have tested mobile iris, eye vein, voice, signature, face, palm, and two-factor authentications, all since 2001
• However, today, testing and evaluation is a lot more complex
• Biometrics has a huge market opportunity, but to deploy successfully, we need to test
SO MOBILE ISNT NEW!
![Page 23: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/23.jpg)
TESTING AND EVALUATION
– THINGS TO CONSIDER
![Page 24: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/24.jpg)
• What is the purpose of the test?
• To get an idea of the performance of the biometric
• To get an idea of the usability
• To calculate the throughput of an ABC gate
• To understand the differences of whether there is a difference in one group or another
• Take the example of the phone versus the border gate – what are we interested in?
WHAT ARE WE EVALUATING
![Page 25: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/25.jpg)
•Understanding the problem statement is key
for a good test protocol
•Tests are expensive and time consuming, and
we want to be able to collect the data
efficiently, accurately, and within budget
WHAT ARE WE EVALUATING
![Page 26: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/26.jpg)
• Discuss with the clients the different aspects of evaluation that you can do
• Such as straight performance through to understanding the issues relating to performance
• Sample size – the complexity of the test will drive the sample size
WHAT ARE WE EVALUATING
![Page 27: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/27.jpg)
•Scenario
•Technology
•Operational
DIFFERENT TYPES OF TESTS
![Page 28: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/28.jpg)
•The matching algorithms are then tested on a
sequestered dataset
TECHNOLOGY EVALUATIONS
![Page 29: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/29.jpg)
•Takes place in a testing facility
•Matchers are installed in an “office
environment” and biometric devices are tested
SCENARIO EVALUATIONS
![Page 30: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/30.jpg)
•Volunteers are recruited
•Group uses the system to collect over time
•This creates a set of databases that can be
used for technology evaluations later on
SCENARIO EVALUATIONS
![Page 31: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/31.jpg)
• From Mansfield and Wayman:
• Provide a framework for developing and fully describing test protocols
• Help avoid systematic bias due to incorrect data collection or analytic procedures in evaluation
• To help testers achieve the best possible estimate of field performance while expanding the minimum effort in conducting their evaluation
• To improve the understanding of the limits of applicability of test results and test methods
SCENARIO EVALUATIONS
![Page 32: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/32.jpg)
•This type of evaluation is used to determine
the performance of a biometric system in a
real-world environment
OPERATIONAL EVALUATION
![Page 33: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/33.jpg)
• A combination of technology and scenario testing
• We want to solve these additional problems of what are these biometric system errors, and how can we fix them
• So we do:
• test protocol development, human subject testing, many different modalities
• Usability testing
• Surveys
• Focus groups
OUR TESTING
![Page 34: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/34.jpg)
USABILITY TESTING
![Page 35: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/35.jpg)
HUMAN BIOMETRIC SENSOR
INTERACTION
![Page 36: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/36.jpg)
HBSI MODEL
![Page 37: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/37.jpg)
•ABC gates are important, but these are
complicated from the perspective of biometric
testing
HBSI MODEL ADVANCES
![Page 38: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/38.jpg)
•Token- the passport
•People traveling- potential incorrect passport
•Throughput issues- how to measure
CHALLENGES
![Page 39: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/39.jpg)
TOKEN HBSI MODEL
Token HBSI Model
![Page 40: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/40.jpg)
TOKEN HBSI MODEL
![Page 41: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/41.jpg)
TOKEN HBSI MODEL
![Page 42: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/42.jpg)
TOKEN HBSI MODEL
![Page 43: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/43.jpg)
• HBSI and Border Gates
• Building a border gate in the center to test various technologies, including iris, fingerprint, documents
• Throughput research
• Usability using the Kinect
• Novel performance metrics such as the Stability Score Index
TESTING AND EVALUATION RESEARCH
![Page 44: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/44.jpg)
• Mobile devices are important as well, and we continue to work in this area:
• Wild testing
• Illumination / Noise / Controlled environments
• Voice, face, signature, palm, and multi-factor authentication
TESTING AND EVALUATION RESEARCH
![Page 45: Cerias talk on testing and evaluation](https://reader034.vdocuments.mx/reader034/viewer/2022051817/54821f235906b5d3048b460b/html5/thumbnails/45.jpg)
QUESTIONS