copyright © 1995-2012 questionmark corporation and/or questionmark computing limited, known...

Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Security Strategies for Every Stage of the Testing Process

DeDe Hedlund, Creighton UniversityJeff Place, Questionmark Evangelist

NCTA – Minneapolis, MNFriday Aug. 3, 2012, 11:30-12:30 pm


There are no shortage of test security challenges At each stage of the assessment process (Authoring,

Scheduling, Delivery, and Reporting) solutions exist to meet security challenges

Let’s Identify the threats to watch out for Show you today how certain product functionality

can meet your test security needs by walking through a typical examination process and some real life examples

Goals of this Session

Slide 2


Impersonation

Content Theft

Cheating

Threats

Slide 3


Issues

Fraud Triangle Threats Rationalization Opportunity Motivation

Impersonation Content Theft Cheating

http://blog.eric.info/wp-content/uploads/2009/11/image3.png




Higher Stakes

High

Medium

Low

Medium Stakes

Low Stakes

Low/High to High/High Stakes Tests

Slide 5

What’s at Stake?

Life and Limb

Promotion & Jobs & Legal Concern

Educational Exams

Tests

Elearning & Surveys


• Personable, consistent• Monitor vulnerable to unfair influenceTest Centers

• Convenient, personable, consistent• Monitor vulnerable to unfair influence

Events (Classrooms or Conventions)

• Monitor is less vulnerable • New and not yet widespread

Remote Real-time 360 cam

• Monitor is less vulnerable• Easy to deploy for use at home

Remote Real-time webcam

• Seems secure• Nothing for content theft

Record & Review360 or webcam

• Works for employees• Nothing for content theftUn-Monitored

Monitoring Tests Securely

Slide 6

ID F

raud

Prot

ect

Cont

ent

Min

imize

s Ch

eatin

g

√ √ √

√ √ √

√ X

√ X X

√ √ √

√

√ √ √


Test Centers

Events (Classrooms or Conventions)

Remote Real-time 360 cam

Remote Real-time webcam

Record & Review360 or webcam

Un-Monitored

Providers

Slide 7

ID F

raud

Prot

ect

Cont

ent

Min

imize

s Ch

eatin

g

√ √ √

√ √ √

√ X

√ X X

√ √ √

√

√ √ √


Create• Collaborative, secure

authoring of surveys, quizzes, tests and exams

Deliver• Blended delivery to

support a broad range of devices and security needs

Analyze• Evaluate results and

inform stakeholders with timely, meaningful reports

Assessment Management

Process


Slide 9

Authoring: Item Banking and Security

Security impacts Validity of assessment results Fairness to participants

One approach: security rights that are role-based Set access permissions by role or profile Limit an author’s or reviewer’s access to the

various functions Limit access to topic (item) folders Limit what the author may do in those folders


Slide 10

Scheduling

Create participants and schedule assessment delivery

Manage participants individually or in hierarchical groups

Schedule access to assessments:◦ Limit dates◦ Limit attempts◦ Require proctor/invigilator◦ Deliver to certified test center◦ Online or offline delivery

Email Broadcast:Notifications to Participants


Formative QuizzesCourse Evaluations

Diagnostic Tests

Low Stakes Tests

Medium Stakes Exams

High Stakes Exams

Leve

ls o

f M

onito

ring

Blended Delivery

Stakes of Assessment


Slide 12

Questionmark Secure Stops people from printing questions Stops people from typing in their own URL Always display in full screen and it’s not possible to maximize or

minimize Stops people starting a new task Does not provide menu options or icons Disables control keys and task switching Disables right-click menu options Disables the ability to start new applications Prevents going backwards to a previous page Stops people exiting in a high stakes, proctored, environment Hides the HTML source Provides an API to control certain functions of a browser from the

server Server can command Questionmark Secure to display a toolbar

Secure Browser

Regular Browser


Slide 13

Browser Check

Ensures Test Takers

use a compatible browser


All participant scoring is conducted on the Perception server (rather than on client/participant PCs)

Scoring algorithms are not sent in any way to participant PCs

Perception can be configured to use SSL to ensure communication between participants and the server are secure

Data in item repositories are encrypted (QML is encrypted) providing added item bank security

Delivery Security

Slide 14


Delivery Security Features


Question by Question (QxQ) delivery options limit the amount of content shown at one time

Randomization of questions and choices to prevent cheating

Limiting Item Exposure


Score List Report◦ IP addresses (as expected?)◦ Time taken (too quick, too long?)◦ Easy flagging of participants who get perfect scores for further

investigation Coaching Report

◦ Drill down on an individual participant, comparison to benchmark, use in appeal process

Item Analysis Report◦ Determine psychometric performance and examine item difficulty

drift over time Test Analysis Report

◦ Assessment defensibility review and test difficulty drift

Reporting

Slide 17


Case studies from Creighton University

a. What their test security needs were

b. How our software addressed those needs

c. Demo of solutions

Case studies

Slide 18


Creighton University School of Pharmacy & Health Professions

Private Jesuit University located in Omaha, NE 2001 started the first and only accredited truly distance

based full time Doctor of Pharmacy program Issues to resolve and security requirements

1. How to securely deliver and administer exams to students located throughout the country

2. Paper and pencil were too expensive and it’s difficult to distribute and collect approximately 5600 exams per semester

3. Control who sees what test and when4. Lock down the computer to eliminate printing, instant

messaging, browsing the internet during the tests5. Ability for students to “mark” an identified area of a graphic


Solutions offered to meet requirements:1. Provided online testing solution and eliminated almost all paper and pencil tests to

reduce costs2. 22 Question types including Hotspot allowing online exams to replicate past paper

exam formats with the additional ability to provide instant feedback.3. Ability to randomly deliver different variations of an exam4. Manage large item banks of questions including security for collaborative authoring

among faculty 5. Lock down browser6. Scheduling (right test to right student)

Demos7. Hotspot question type & feedback text/images8. Password system & variations of an exam9. Item Banking and security for collaborative authoring10. Questionmark Secure11. Scheduling


Slide 20


Demonstration


Variations of an Exam

Slide 22


Item Banking

Slide 23


Additional security measures1. Established guidelines for choosing new proctor sites2. Formalized the process of communicating what is expected

from a proctor site3. Proctor Certification forms are required from every student

who completes an exam4. Exam specific instructions are emailed to the proctors


Slide 24


Laptops and PCs

CustomHardware

Remote Video Monitoring

Slide 25


Candidate (Participant) Logs in

As Software Secure and Questionmark integration is for higher stakes exams a Participant will be referred to as a Candidate

Slide 26


Candidate selects exam and “Schedules Appointment”

Schedule Appointment Exam

Exams (4) Surveys (1)

Slide 27


Schedule Day and Time

Slide 28


If the exam is to be schedule in the next 3 days or if the participant might be expected to pay some of the cost

of the exam then there are more steps

Review Schedule

Slide 29


Payment Process

Slide 30


Candidate Takes Exam When It Was Scheduled


Candidate (Participant) Logs in

Slide 32


Selects Exam to Take

Exams (4) Surveys (1)

Slide 33


Camera and Microphone Access Identity Checks Captures Photo, Proctor Checks against files Proctor checks has Participant scan the room

ProctorU / Software Secure


After Authentication Candidate takes Exam Video and screens are monitored during exam

Security Checks Complete Candidate Takes Exam

Slide 35


Potential Applications for Mobile Delivery

Deliver to the device in your hand

Observational assessments Mobile test centers


Mobile Test Centers

Slide 10

• Use notebooks/3G laptops/iPads to create mobile test centers

• Enables greater flexibility in where you deliver assessments

• Assessments can be conducted in ‘on location’ environments


“Securing” an iPad…

Slide 11

• An app should prevent candidates from easily getting to other URLs

• But – access to “home button” allows task-switching

• Hardware solutions are available to enable use of iPad in a “kiosk” mode


Considerations:

Slide 12

• The higher the stakes, the higher the propensity to cheat –still require human monitoring!

• Bandwidth – Be sure to test the wifi and/or 3G signal strength prior to event


Cheat-resistant questions and assessments


Randomization Refresh items regularly Build large item banks Track item parameters Tests that are unique for each test taker make it hard

to share answers Adaptive tests provide different questions based on

responses to previous questions

Considerations for cheat-resistant questions and assessments

Slide 41


Answer choices should be roughly the same length and kept as short as possible.

Provide a minimum of three answer choices and a maximum of five. Four is considered optimal.

Keep your writing clear and concise – you’re testing knowledge, not reading comprehension.

Make sure that you’re putting the correct answer in the first two positions as often as the last two positions.

Improve Your Multiple Choice Questions

Slide 42


Avoid Previously Delivered Questions

Slide 43


Extended Matching Question Type

The number of answer options depends on the logical number of realistic options for the test taker.)

The same answer choice could be correct for more than one question in the set Some answer choices may not be the correct answer for any of the questions So it is difficult the answer this type of question correctly by chance. A well-written lead-in question is so specific that students understand what kind

of response is expected, without needing to look at the answer options. Slide 44


Scenario Based Questions

Slide 45


Slide 46


Scan a QR Code with your mobile to start an assessment!

Slide 47


Closing and questions Thank you for your time and attention! Questions/Comments? Visit our booth to talk to us more We have a whitepaper on this topic (Delivering

Computerized Assessments Safely and Securely) Visit our website to download:

http://www.questionmark.com/us/whitepapers/index.aspx

http://www.questionmark.com/us/whitepapers/index.aspx


Thank you for attending!DeDe Hedlund

[email protected]

Jeff [email protected]

https://by2prd0410.outlook.com/owa/redir.aspx?C=puKQV8d_GEGyjPuHaDCp5hZE_9mPOs8IVm1P_G89miiESnxkE67jzJbVUH5BbyoR92ODsPsh2WA.&URL=mailto:[email protected]

mailto:[email protected]

copyright © 1995-2012 questionmark corporation and/or questionmark computing limited, known...

Documents