Exam Security IssuesLegal, Psychometric, and Administrative Perspectives

Amin Saiar, PhDBob Albert, MBA

Amanda Wolkowitz, PhDJennifer Ancona Semko, JD

Overview of Presentation

1. Introduction to Security Risks and Analytics

2. Case Study: Invalidation of Examinee Scores

3. Legal Considerations in Cheating Investigation and Intervention

Introduction to Security Risks and Analytics

Amin Saiar, PhDJohn Weiner, MSGreg Hurtz, PhD

Security Analytics

• Can be used to investigate issues or events, or to identify issues to investigate

• It is most effective when combined with other methods of intervention

• Play a critical role in testing-program management

Exam Security RisksRisk Behavior Analytic

Cheating Collusion / Copying Response similarity

Piracy HarvestingTesting time vs. items answered

Proxy Stand-inBiometrics, photo,

proctor

Volatile Retake Quasi-key accessTest 2 - Test 1 comparison

Compromised content

Change in performance Pass rate analyses

Response Similarity

• Used to identify examinees with an unexpectedly similar number of matching responses with another examinee

• An examinee is flagged if number of matched responses are significantly higher from distribution of matched scores

• In such case, collusion or copying may have occurred

ABACABDCADBCCBBAAB

DBACCBCDAABCABDCAB

Collusion vs. Copying

An important distinction because the type of damage done is different and analytics differ in ability to detect

them

Copying

Copying of answers from a source

Collusion

Advance knowledge of some or all of the

items

Monte Carlo Simulations

• Simulated data created to test different indices’ ability to correctly identify collusion and copying

• Different percentages of “examinees” throughout the ability range were randomly selected to “cheat” on different percentages of items

• Response operator characteristic (ROC) curve used to measure detection ability

Indices Compared

• HT (Sijtmsa & Meijer)– Nonparametric person-fit statistic

• Kappa (Cohen)– Cohen’s kappa inter-rater agreement

coefficient for categorical items• J Indices (Weiner, Hurtz, & Saiar)

– Regression-based approaches to identify large residuals

J2 Index

1. Compute Max_nMatch by form Each candidate’s maximum # of matching responses with any other candidate

2. Conduct regression analysisTest score onto Max_nMatch

3. Compute residualsObserved minus predicted

4. Standardize residuals within sampleConversion to z-score

0 20 40 60 80 100 120

0

20

40

60

80

100

120

140

ROC Curve

• Plots true positive rate against false positive rate

• Useful in determining if actual cheaters are being detected

Area under ROC Curve

Takeaways

• Certain analytics and investigative methods are most appropriate for specific types of cheating

• The specific risks that a program is susceptible to should be identified or determined

• The use of multiple analytic tools and investigative methods is advised

Case Study:Invalidation of Examinee

Scores

Bob Albert, MBAAmanda Wolkowitz, PhD

National Dental Examining Board of Canada (NDEB)

• 1953 Act of Parliament – establish qualifying conditions for a single national standard of qualification for general dentists in Canada

• Issues certificates to those who meet the standard

10 Provincial Licensing Authorities

NDEB Certification Process

Written Examination

and OSCE

Certified and portable

Graduates of Accredited Programs

Graduates of Non-

accredited Programs

NDEB Equivalency Process

Accredited Qualifying Program

Bylaws

• 20.01 explains the consequences of compromising an examination or assessment to examinees:– Results may be voided– Privilege of repeating the examination or

assessment may be denied– A person may be dismissed immediately

prior to a full report

Assessment of Clinical Judgment (ACJ)

• 1 day, 3 booklet paper-pencil based (2 case-based and 1 radiology) with extended match questions

• Administered annually in June and December

• Must earn an equated test score of 75 to pass the exam

Sample questionQuestion: (select one or more correct answers.)There is evidence of calculus on the

Distal of tooth 1.8

Mesial of tooth 1.8

Distal of tooth 1.7

Mesial of tooth 1.7

Distal of tooth 1.6

Mesial of tooth 1.6

+.25

-1

(0)

-1

+.25

-1

-1

+.25

+.25

-1

-1

Distal of tooth 1.5

Mesial of tooth 1.5

Distal of tooth 1.4

Mesial of tooth 1.4

Distal of tooth 1.3

The 2014 ACJ Incident

• Approximately 634 participants nationally

• 203 participants at the Toronto site• Participants seated according to a

pre-set seating chart• Exam is proctored by invigilators and

supervised by a test administrator.

Suspicious Behavior Reported

• Invigilator alerted by a participant of two individuals having conversations during the exam.

• After noting the incident, participants asked to stop talking and not have communications.

• Test Administrator notified of the incident and noted the seat numbers.

Suspicious Behavior Suspected

• Exam wide answer sheets collected and scanned

• Commercial based detection analysis software not possible due to multiple choice/response item structure

• In-house group comparison statistical analysis revealed significance similarities in responses

• Alpine Testing was engaged to analyze and report on the suspicious behavior.

Suspicious Behavior Reported

What did we find?

In your testing program, have you successfully invalidated candidates’ scores based on questions about the validity of that score?

A. No.  We have not identified specific candidates whose scores were in question.

B. No. We identified candidates whose scores were in question, but did not invalidate their scores.

C. Yes. We have invalidated candidates’ scores based on questionable validity.

Case Study

• June 2014 administration– 634 total participants– 203 completed paper-pencil exam at

site of interest

Case Study

• 120 items divided into 3 “books”– Extended match multiple-choice– Each option has a point value between -

3 and 1 (including fractional amounts) associated with it

– Sum of points equals item score– Item score range: 0 to 1

1. Test StatisticsAll ASCT

SiteNon-ASCT Site

Participant Count

654 203 451

Mean 83.0 83.4 82.9SD 8.6 8.1 8.8Alpha Reliability

0.779 0.758 0.787

No suspicious findings when comparing overall test statistics.

2. Correlation Analysis

• Correlation of all 203 participants at site

Pair r p ID #1 Seat

ID #2 Seat

1 0.87 <0.001 46 452 0.61 <0.001 202 53 0.60 <0.001 93 114 0.63 <0.001 161 1495 0.62 <0.001 129 120

Participants with Both Statistically Significant Correlations and Correlations Above r = 0.60

2. Correlation Analysis

• What does this mean?– Pair 1 had < 1 in

10,000,000,000,000,000,000,000, 000,000,000,000,000 (< 1 in ten undecillion) chance of having this much similarity in their responses

– The chances of winning the Powerball are much greater (1 in 175,000,000)

3. Identical Response Option

203 participants = 20,503 paired Comparisons

Pair 1 shared identical responses to 84.2% of the items. This is over 20% moreidentical item responses than any other pair at this site.

Pair 1

4. Seating Chart

• Seats of those with significantly high correlations

4. Seating Chart

• Seats of those with high number of identical responses (at least 69)

5. Probability Analysis on Pair

• Responses of Pair 2407– 21 items in which both scored 0– Exact same response to 17 of these 21

• What is the probability that two randomly selected examinees would have 17 or more identical wrong answers on these 21 items?

5. Probability Analysis on Pair

Less than 1 in 33 million

Summary of Pair 24071. Proctor noticed test misconduct2. Highest correlation at test site with a correlation unlikely

to occur by chance3. Highest number of identical item responses that were

highly unlikely to occur by chance (100% identical in book 1 of 3 books)

4. Highest number of identical wrong responses receiving 0 points that were highly unlikely to occur by chance

5. Seated right next to each other

What happened to these candidates?

Suspicious Behavior Reprimanded

• After legal counsel and sufficient proof of compromise…

• Examinations committee voted to void the participants’ results and be “ineligible to participate in any future NDEB assessments or examinations”.

• Communicated the results to the public on the NDEB web site.

Additional Procedures

• Alpine Testing contracted to develop a design to conduct high level screening analyses of test scores.

• NDEB is integrating the design into its own custom question bank system.

• NDEB will incorporate the reports into its routine post exam item performance analyses.

Legal Considerations in Cheating Investigation and

Intervention

Jennifer Ancona Semko, JD

Key Question: Can you defend your actions?

Three fundamentals:

(1) Matter of contract(2) Deference (3) Good faith

What’s a candidate agreement?

Do I need one?• Contract: An agreement between two or more persons which creates an obligation to do or not to do a particular thing. A legal relationship consisting of the rights and duties of the contracting parties. Black’s Law Dictionary, Sixth Edition

• Your agreement with test takers defines the relationship

• Memorializes your (and their) rights and obligations

• If done properly, makes expectations (and remedies) clear

What does your candidate agreement say?• Are candidates on notice that sharing items is a breach?• Are candidates on notice that studying from recalled items

is improper?• “It must be OK . . . they told us about everything else,

but not about this.”• Did you reserve the right to invalidate scores? Suspend or

ban access to examination? To take other action?• What are the grounds for action? Is there a “catch all”?• Are candidates required to cooperate in investigations?• Do you regularly review your agreement language?• Do you have uniform security procedures and policies in

place?

When was the last time you reviewed and updated your Candidate Agreement?

A. Our program regularly reviews and updates our Candidate Agreement.

B. No one has touched our Candidate Agreement in at least a few years.

C. We have a Candidate Agreement?

Rights of Exam Administrators

• Murray v. Educational Testing Service, 170 F.3d 514 (5th Cir. 1999)– “Several courts, including this one, have recognized

the importance of allowing ETS to assure itself of the validity of students’ scores through internal review procedures.”

– “. . . ETS has an obligation to provide, or use its best efforts to provide, only valid scores to the colleges and universities that rely on ETS’s services.”

– “Moreover, ETS has the right to protect its own reputation by assuring the reliability of the information it provides.”

Rights of Exam Administrators

• Scott v. ETS, 600 A.2d 500 (N.J. Super. 1991)– “We are satisfied that the relevant public and private

interests are fairly accommodated by a procedure which permits ETS to cancel scores upon an adequate showing of substantial question as to their validity, without any necessity for a showing of actual cheating or other misconduct.”

• K.D. v. ETS, 386 N.Y.S.2d 747 (1976)– “. . . If [ETS] reasonably believed that the test scores . . .

did not accurately reflect [plaintiff’s] aptitude for law school, it acted within its right to protect its own image as well as its obligation to the schools who are its clients in canceling plaintiff’s scores . . .”

Deference to Exam Administrators

Murray v. ETS, 170 F.3d 514 (5th Cir. 1999) (SAT Exam)

• Louisiana basketball player; needed 820 on SAT

• Scored 700, then 1300– Similarity to nearby student (3 in 100 million odds)– Scored 800 on retake

• “ETS’s contract with Murray clearly and explicitly reserved to ETS the right to withhold any scores ETS had reason to believe were not valid. The only contractual duty ETS owed to Murray was to investigate the validity of Murray’s scores in good faith.”

Good faith applies to football, too

Langston v. ACT, 890 F.2d 380 (11th Cir. 1989) (ACT Exam)

• Alabama football player; scored 10 on ACT; then 20

• Inconsistent with GPA; unusual similarity to nearby student

• “Under the governing law, the outcome of plaintiff’s case does not turn on whether or not plaintiff cheated on his exam, but only on whether or not ACT carried out its contractual obligations in good faith.”

Did you act in good faith?

• Court will look for evidence of good faith actions

• Consistency, uniformity, fairness

• Consideration of all available information (including test taker’s response)

• State actors also owe Constitutional Due Process

Which creates stronger foundation for action?

• Your Candidate Agreement Says:

(A)“Candidates proven to have cheated during the examination may have their scores invalidated.”

OR

(B)“We reserve the right to cancel any test score if, in our sole opinion, there is adequate reason to question its validity.”

What is the most significant obstacle preventing you from invalidating candidate scores where appropriate?

A. Don’t have the data needed to support invalidation.

B. Lack of management/legal department support for invalidation decisions.

C. Unsure of our program’s rights to invalidate.

D. Something else.

Legal Considerations: Take-aways

• DON’T have to prove cheating• DO have to act in good faith• It’s about validity, not cheating• Matter of contract, deference, good

faith• Candidate Agreement language is

key

THANK YOU• Bob Albert, MBA

– Chief Technology Officer, National Dental Examining Board Of Canada– balbert@ndeb-bned.ca

• Amin Saiar, PhD– Manager of Psychometric Services, PSI Services LLC– amin@psionline.com

• Jennifer Ancona Semko, JD– Partner, Baker & McKenzie LLP– jennifer.semko@bakermckenzie.com

• Amanda Wolkowitz, PhD– Psychometrician, Alpine Testing Solutions– Amanda.Wolkowitz@alpinetesting.com