![Page 1: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/1.jpg)
Caveon Webinar Series:
Ten Test Security Lessons from ATP
March 19, 2014
John Fremer, Ph.D.PresidentCaveon Consulting Services
Steve AddicottVice PresidentCaveon Test Security
![Page 2: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/2.jpg)
Agenda for Today
• Steve– Emphasis on Test Security– Highlights from some TERRIFIC Security Sessions– Online proctoring – Next Step in Exam Security
John– Exam Security Processes and Procedures– “Telling it to the Judge”– User Agreements– ATP Security Committee Briefing– Closing Keynote
Q&A
![Page 3: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/3.jpg)
www.caveon.com 3
Test Security is…..
![Page 4: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/4.jpg)
www.caveon.com 4
Test Security is Front Burner
![Page 5: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/5.jpg)
www.caveon.com 5
Test Security is Front Burner
![Page 6: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/6.jpg)
www.caveon.com 6
Ten Years of Test Security
Star-studded cast– Schoenig, Wollack, Cizek (and Addicott )
Key trends explored
Important “Lessons”
![Page 7: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/7.jpg)
www.caveon.com 7
Schoenig—Test Security is a Process
ACT Process:
Deter, Detect, & Decide
Supports the 3-legged stool of
![Page 8: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/8.jpg)
www.caveon.com 8
Wollack—Data Forensics Goes Mainstream
An amazing paradigm shiftPast: DF detects cheating = public distrust in
testing
Future: DF enhances validity = increased
public trust
Industry Standards for DF• Proprietary Dissemination
• CSDPTF COTS
DF moving closer to test instance
![Page 9: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/9.jpg)
www.caveon.com 9
Cizek—Test Security is about Validity (not cheating)
What we have said:
“a statistical procedure... can never
prove that, for example, teachers in a
school are cheating”
“such statistical criteria are never ‘proof’
of cheating”
![Page 10: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/10.jpg)
www.caveon.com 10
Cizek—Test Security is about Validity (not cheating)
What we can say:
“There is strong reason to be concerned that
these scores may not be valid. We recommend
that additional information be gathered or that
testing be replicated before interpreting these
scores to mean...”
![Page 11: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/11.jpg)
www.caveon.com 11
“Exam Inoculation and Other Crazy Ideas to Stop Cheaters from Passing Exams”
Burns, Van de Velde, and Maynes
• Pirates persist
• Test Design is powerful!
• Most measures are reactive, so…
• Inoculate exams against pirates
– Trojan Horse Items
– Embedded Verification Tests
– Item cloning
– DOMC
![Page 12: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/12.jpg)
www.caveon.com 12
Online Proctoring Is Here to Stay
• ATP SC orchestrated demo-rama
• Security concerns abound
• Are they justified?
• More research required
• Test Security Standards for Online Proctoring
now available
– http://bit.ly/1ifSv0F
– Caveon.com Resources White Papers
![Page 13: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/13.jpg)
www.caveon.com 13
Next Step in Exam Security’s Evolution
Institute for Exam Security (I4ES)
• An independent, non-profit association
by/for Test Security professionals
• www.cespcert.org
![Page 14: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/14.jpg)
www.caveon.com 14
Fremer Areas to be Covered
Multidisciplinary Approaches Workshop
“Telling it to the Judge”Critical Importance of User
AgreementsATP Security Committee
BreakfastClosing Keynote
![Page 15: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/15.jpg)
www.caveon.com 15
Multidisciplinary Approaches Workshop
• Program Manager, Investigator, Attorney, Test Security Guy
• Four Group Exercises• Each Dealing with Real Challenges
in Testing Programs• Common Elements across
Discussions
![Page 16: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/16.jpg)
www.caveon.com 16
One Exercise – The Context
Limited staffLimited staff expertiseLimited resources: software,
money, timeProblemEfficiently allocate resources
to threats based on – The risk they pose– The likelihood of finding a solution
![Page 17: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/17.jpg)
www.caveon.com 17
Activity #2
Assess the Risks!
![Page 18: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/18.jpg)
www.caveon.com 18
The ScenarioCBT administration, 3,000 examinees.
Following the administration, you analyze your data for statistical indicators of cheating.
180 examinees were statistically flagged for similar response patterns. They did not pass at a higher rate than expected.
Of those 180, a group of 80 examinees had been flagged in previous administrations. They did not pass at a higher rate than expected.
What are your next steps in addressing this issue?
![Page 19: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/19.jpg)
www.caveon.com 19
Common Elements Across Reports
Notify Your Boss PromptlyCollect Relevant
Information Based on Your Response Plan
Seek Appropriate Expertise– Legal – Psychometric–Messaging/Communications
![Page 20: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/20.jpg)
www.caveon.com 20
I Told It To The Judge, But S/He Didn’t Listen
Jennifer Semko – Baker & MckenzieCamille Thompson – ACTDennis Maynes - Caveon
![Page 21: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/21.jpg)
www.caveon.com 21
I Told It To The Judge (cont.)
• Advice re testifying– Answer only what you are asked– Don’t be a smart alec– KISS Principle
• Courts have been Supportive• Follow Your Policies and Procedures
or Stay Home
![Page 22: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/22.jpg)
www.caveon.com 22
I Told It to The Judge - User Agreements
User Agreement is a Binding Contract
Defines RelationshipMemorializes ObligationsYou will be Judged by How
Well you Fulfilled Your Commitments
Must Behave Responsibly and Fairly
![Page 23: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/23.jpg)
www.caveon.com 23
Poll #4
![Page 24: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/24.jpg)
www.caveon.com 24
ATP Security Committee
High Attendance at Breakfast
Great Deal of Productive Energy
Many Projects (Examples)– Removing Copyrighted Material from
the Internet– Test Taker Rights and Responsibilities
![Page 25: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/25.jpg)
www.caveon.com 25
Closing Keynote
Hilary MasonBig Data – Tremendously
UsefulGives Power to Many to
Influence EventsIt is the Effective Holders of
Information who Can Wield Power
Some Problems Cut Across Fields – Unwillingness to Act on Data Alone
![Page 26: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/26.jpg)
www.caveon.com 26
Overall Impression
Sharp Increase in Awareness of Importance of Test Security to Test Program Management
Other Domains Catching up to IT in Recognition of Problems
Focus on User Agreements and Online Proctoring
Willingness to Try New Approaches
![Page 27: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/27.jpg)
www.caveon.com 27
Questions?
Please type questions for our presenters in the GoToWebinar control panel on your screen.
![Page 28: Caveon Webinar Series - Ten Test Security Lessons Learned at ATP 2014 march 2014](https://reader038.vdocuments.mx/reader038/viewer/2022103114/55619df0d8b42ae1538b479c/html5/thumbnails/28.jpg)
www.caveon.com 28
Thank you!
John Fremer PresidentCaveon Consulting Services
Steve Addicott Vice PresidentCaveon Test Security
Follow Caveon on twitter @caveonCheck out our blog www.caveon.com/blogLinkedIn Group “Caveon Test Security”