caveon webinar series - integrating data forensics into the entire test security process - april...
TRANSCRIPT
Caveon Webinar Series
www.caveon.com 1
Integrating Data Forensics
into the Entire
Test Security Process
April 29, 2015
Dennis Maynes, Chief Scientist
Jennifer Miller, Data Forensics Coordinator
Caveon Test Security
Agenda for Today
• Test Security Process
• Data Forensics Integration
• Application to Selected
Situations
• Summary
www.caveon.com 2
Test Security Process
www.caveon.com 3
Security is a Process, Not a State
www.caveon.com 4
Measure and Manage
Respond
Protect
DetectImprove
Four-Fold Protection
www.caveon.com 5
Prevention
Deterrence
Insurance
Quick Response
Detection Elements
www.caveon.com 6
Proctoring
Tip Lines
Data Forensics
Web Patrol
Site Monitoring
Response to Breaches
www.caveon.com 7
Incident
Response
Investigations
Communications
Sanctions
Threats
Attacks
Breaches
Vulnerabilities
Improvement of Test Security
8
Plan
Do
Check
Act
Data Forensics Integration
www.caveon.com 9
What is Data Forensics?
www.caveon.com 10
• Data Forensics is the science of gathering evidence of potential test security breaches from the test response data
• There are clues in the data relating to:– Collusion– Use of recalled questions– Rogue review courses– Testing sites with poor security– Exams and items that have been disclosed
“We balance probabilities and choose the most likely. It is the scientific use of the imagination.” – Sherlock Holmes, The Hound of the Baskervilles
Data Forensics Measurement
www.caveon.com 11
Statistical Anomalies
Testing
Irregularities
Security
Violations
Security
Breaches
Test Fraud
Definitions
www.caveon.com 12
• Statistical anomalies are observed data that do
not conform to statistical models of normal test
taking.
• Testing irregularities are abnormal occurrences
which may have impacted the test administration.
• Test security violations occur when the security
protocols of the test have not been followed.
• A breach in test security is an event which has
jeopardized the fairness and the validity of the
current or future test administrations.
• Test fraud involves intent by a perpetrator to
breach the security of the test.
Detection Statistics
www.caveon.com 13
Pre-Knowledge
–Use of Braindump content
–Rogue Review courses
–Disclosure of content
–Answer key theft
Aberrance / Person-fit
Gain scores
High pass rates
Score differencing
Trojan Horse & EVT items
Response times
Collusion
–Impersonators
–Proxy test takers
–Sharing content
Similarity
Identical tests
Large clusters – similarity counts
Source-copier analysis
Shared personal information
Tampering
–Answer sheet falsification
–Score report falsification
Counts of changed answers
Associated gains
Inconsistent score data
Data Forensics Focus
• Who has gained an unfair advantage?– Test takers
• Where is the advantage concentrated?– Test sites, Schools, Review courses, Teachers
• What is the impact of advantage on test?– Items, pass rates, scores
• Are we winning the test security battle?– Trends of key indicators
www.caveon.com 14
How to Use Data Forensics
www.caveon.com 15
• Protection – Just in time analysis– Prevention - Quick detection followed by timely
response– Deterrence – “Radar patrolled”
• Detection – Monitoring– “Fire spotting”
• Response – Investigation & Sanctions– Draw inferences from data– Create presentations of inferred events
• Improvement – Metrics & Key Indicators– Use for triage– Track trends
Application to Selected
Examples
www.caveon.com 16
Proxy Test Taking
www.caveon.com 17
Proxy Test Taking
2007: Contracted with a proxy test taker for $1,000
• In a few weeks, the certificate was “awarded”
• Data analysis discovered
– The test site:
• registered with a false mailing address
• affiliated with a mobile site
• operated by the proxy test taking organization
– Tests at five more test sites were “very similar” / “in
collusion”
– Estimated number of proxy-taken exams was 500 in 6
months
We infer that:
• This organization was paid $1 million for proxy test taking
services for a single exam title in one year.
www.caveon.com 18
We Have Learned
www.caveon.com 20
• Proxy test takers
– Legitimate test sites, but…
• Front room and back room
– Operate multi-nationally
– Super-human performance
– Branching out to other
certifications
– Sophisticated
• “Whack-a-mole” – they move
on
DF Applied to Proxy Test Taking
• Identify/Monitor Individuals,
Test Sites
– Similarity
– Response Time
• Use sanctions to protect, invalidate,
publicize
• Use KPI’s to learn “Are we winning?”
www.caveon.com 21
Exam Piracy
www.caveon.com 22
Exam Piracy Case 2012
• Intercepted copy of stolen exam with 97% of items
with near-exact textual matches
• Forensics identified the author (a test taker)
• Fifteen more test takers in a one month period
were extremely similar with the author
• The similarity had a vanishingly small probability
(<10-38)– The imputed answer key had 10 wrong answers for
60 questions
– It’s more likely for the Powerball winner to win the
next 4 jackpots!
– Often, data forensics analysis is compelling!
www.caveon.com 23
We Have Learned
• Use of stolen exam content can be
prevalent (may exceed 1 in 6 test takers)
• Not just for “profiteers” anymore—small
groups
• Some test thieves have gotten smarter– Are reacting to new test design tactics
• Some users of stolen content are naïve– Education is key
– Invalidating scores will deter use of stolen
content
www.caveon.com 24
Counter with TH & EVT Items
• Build security into exams by design
– Detect users of stolen content
– Provide data required to invalidate/revoke
results
– Reduce scores for users of stolen content
– Provide intelligence about theft and use of
disclosed content
• When the compromise occurred
• How many are accessing compromised
content
– Create problems for sellers of stolen content –
unhappy customers
www.caveon.com 25
Trojan Horse Items
• Easy item which is valid but intentionally
miskeyed
• Detect use of disclosed
answer key
• Indicate when and, potentially,
where content with answer key
was stolen
• Easily explained
• Issues and difficulties– Have become known in the “cheat site” industry
– Detect access to stolen answer keys,
not stolen content
www.caveon.com 26
EVT Items
www.caveon.com 27
• Never-exposed item added to a live exam – Should have same difficulty as most items
– Several items required (10 or more)
– Probability of guessing correct answer should be low
• Compare scores between old & new items– Detect who used
compromised items
– Estimate usage rate of compromised content
– Easily explained
– Not easily detected by cheat sites
• Requires great care for defensible
measurement
DF Applied to Exam Piracy
• Use Trojan Horse and EVT questions
– Detect/measure use of stolen
content
– Impose sanctions
• Protect by determining when to republish
• Measure KPIs – Are we winning?
www.caveon.com 28
Disclosure of Content by
Teachers
www.caveon.com 29
Examples of Disclosing Answers
• 2007: Servisair instructors
disclosed answers to
candidates being trained
to de-ice aircraft
• 2011: Principal in
Mississippi instructed
teachers to “chunk-and-
redirect”
www.caveon.com 30
We Have Learned
• Insiders (such as instructors and proctors)
may compromise exams
• Test booklets are not always kept secure
• Harvested content may be shared as a
“drill-it-and-kill-it” book
• Often imputed answer keys contain errors
• Test takers may not always know something
was wrong
• Investigators may need to penetrate the
“code of silence”
www.caveon.com 31
DF Applied to Content Disclosure
• Identify/Monitor Groups, Teachers
and/or Test Sites
– Similarity (large clusters)
– Time stamps on responses
– Answer changes (electronic or on paper)
• Use sanctions to protect, invalidate,
publicize
• Use KPI’s to learn “Are we winning?”
www.caveon.com 32
Review & Wrap Up
www.caveon.com 33
Summary
• Data analytics can be applied in all security
processes
• It’s important to measure so that you can
manage
– We need to learn from our experiences and
mistakes if we do not wish to repeat them
www.caveon.com 34
Thank You!
www.caveon.com 35
Follow Caveon on twitter @caveon
Check out our blog
www.caveon.com/blog
LinkedIn Group “Caveon Test Security”
Jennifer MillerDF Coordinator
Dennis MaynesChief Scientist
@DennisMaynes
Follow Caveon on twitter @caveon
Check out our blog
www.caveon.com/blog
LinkedIn Group “Caveon Test Security”