copyright © 1995-2009 questionmark corporation and/or questionmark computing limited, known...

Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Jeff Place, US Sales ManagerGreg Pope, Analytics and Psychometric

Manager

NCTA – San AntonioThursday Aug. 6, 2009, 3-4pm

Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are

acknowledged.

1. Brief introduction to Questionmark products and services

2. Case studies from four of our customersa. What their test security needs wereb. How our software addressed those needsc. Demo of solutions

3. Questions and closing


acknowledged.

There are no shortage of test security challenges

At each stage of the assessment process (Authoring, Scheduling, Delivery, and Reporting) solutions exist to meet security challenges

We hope to show you today how our product can meet your test security needs by walking through a typical examination process and some real life examples

Slide 3


acknowledged.

Slide 4

Founded in 1988 Technologies and services that enable organizations to

measure knowledge, skills, and attitudes securely for certification, regulatory compliance and improved learning outcomes – Assessment Management System

More than 2,500 customers have purchased Perception◦ For Corporate, Government, and Education markets

More than 14,000 authoring systems installed Offices in US, UK, Belgium, Germany, Mexico Available in English, Spanish, German, French, Dutch

and Japanese


acknowledged.

Slide 5

Create questions and group into quizzes, tests,

exams and surveys using a web browser or

Windows PC

Add, import, change, and

delete participants, groups and

their schedules

Deliver assessment using a standard or secure browser, PDA’s, or using a Windows PC

Report on the results. Reports include coaching reports, item analysis, gap analysis, and others.


acknowledged.

Slide 6

Financial Services

Health / Pharma

Retail High Tech / Manufacturing

Telecommunications Energy / Utilities

Public Sector Education


acknowledged.

Slide 7

Security impacts◦ Validity of assessment results ◦ Fairness to participants

One approach: security rights that are role-based◦ Set access permissions by role or profile ◦ Limit an author’s or reviewer’s access to

the various functions◦ Limit access to topic (item) folders◦ Limit what the author may do in those

folders


acknowledged.

Slide 8

Drag and Drop Essay Explanation Fill in Blanks Hotspot *Likert Adobe Flash Adobe Captivate Matching Knowledge Matrix Survey Matrix

*Multiple Choice *Multiple Response Numeric File Upload Pull-down List Ranking Select a Blank Spoken Response *Text Match *True/False *Yes/No

Enable authors to create more interesting and probing questions

Reveals more information during an assessment in a shorter time

Not offered by simpler assessment tools in LMS’s

Enable authors to create more interesting and probing questions

Reveals more information during an assessment in a shorter time

Not offered by simpler assessment tools in LMS’s


acknowledged.

Slide 9

Create participants and schedule assessment delivery

Manage participants individually or in hierarchical groups

Schedule access to assessments:◦ Limit dates◦ Limit attempts◦ Require proctor/invigilator◦ Deliver to certified test center◦ Online or offline delivery

Email Broadcast:Notifications to Participants


acknowledged.

Slide 10

Secure BrowserSecure Browser

Web BrowserWeb Browser

Questionmark to GoPlayer

Questionmark to GoPlayer

LMS

IntegrationIntegration Windows / CDWindows / CD

Test CentersTest Centers

Printing andScanning

Printing andScanning

MobileDeviceMobileDevice


acknowledged.

Slide 11

Stops people from printing questions Stops people from typing in their own URL Always display in full screen and it’s not possible to

maximize or minimize Stops people starting a new task Does not provide menu options or icons Disables control keys and task switching Disables right-click menu options Disables the ability to start new applications Prevents going backwards to a previous page Stops people exiting in a high stakes, proctored,

environment Hides the HTML source Provides an API to control certain functions of a

browser from the server Server can command Questionmark Secure to

display a toolbar

Secure Browser

Regular Browser


acknowledged.

Slide 12

Ensures Test Takersuse a

compatible browser


acknowledged.

All participant scoring is conducted on the Perception server (rather than on client/participant PCs)

Scoring algorithms are not sent in any way to participant PCs

Perception can be configured to use SSL to ensure communication between participants and the server are secure

Data in item repositories are encrypted (QML is encrypted) providing added item bank security

Slide 13


acknowledged.


acknowledged.

Question by Question (QxQ) delivery options limit the amount of content shown at one time

Randomization of questions and choices to prevent cheating


acknowledged.

Score List Report◦ IP addresses (as expected?)◦ Time taken (too quick, too long?)◦ Easy flagging of participants who get perfect scores

for further investigation Coaching Report

◦ Drill down on an individual participant, comparison to benchmark, use in appeal process

Item Analysis Report◦ Determine psychometric performance and examine

item difficulty drift over time Test Analysis Report

◦ Assessment defensibility review and test difficulty drift

Slide 16


acknowledged.

Case studies from four of our customers

a. What their test security needs were

b. How our software addressed those needs

c. Demo of solutions

Slide 17


acknowledged.

Multi-campus community college based in Iowa with about 15,000 students

Kirkwood offered surgical technician programs to a large base of students through distance learning

Security requirements arose related to the standardized, secure, administration of assessments across the multiple remote campuses1. Authoring to be controlled by faculty at the main campus2. Delivery of assessments to be standardized across the

remote locations to ensure that student experiences were comparable

3. Assessments must be locked-down and protected so that content cannot be shared/exposed

http://www.questionmark.com/us/casestudies/kirkwood_surgical.aspx


acknowledged.

Solutions offered to meet requirements:1. Authoring: Centralize secure authoring platform

(Authoring Manager), Easy remote authoring tools (Questionmark Live)

2. Delivery: Templates ensure equal participant experience regardless of location

3. Locked down/protected: Monitor/proctor and participant logins

Demos◦ Questionmark Live◦ Monitor/proctor/test center login and participant

logins◦ Version 5 templates -

https://beta.questionmark.com/perception5/perception.php


acknowledged.

Technical college located in Rome, GA The home to many of the health education programs

offered by the Georgia Dept. of Tech and Adult Education system including Ultrasound / Sonography

Learn how to read Ultrasounds and recognize problems After the 18-month curriculum, students must be

prepared to take their National Certification Exam

Issues to resolve and security requirements1. Online testing capability to closely resemble the National

Exam2. Paper and pencil were too expensive and difficult to add

graphics 3. Ability for students to “mark” an identified area of a

graphic 4. Control who sees what test and when5. Lock down the computer to eliminate printing, instant

messaging, browsing the internet during the tests


acknowledged.

Solutions offered to meet requirements:1. Provided online testing solution and eliminated all paper and

pencil tests to reduce costs2. 22 Question types including Hotspot to resemble the National

exam with the ability provide feedback.3. Manage large item banks of questions including security for

collaborative authoring among faculty (5,000 items for mock exams)

4. Lock down browser5. Scheduling (right test to right student),

Demos1. Hotspot question type & feedback text/images2. Item Banking and security for collaborative authoring3. Questionmark Secure4. Scheduling

http://www.questionmark.com/us/casestudies/coosa_valley.aspx

Slide 21


acknowledged.

14,000 students located in Mobile College of Medicine offers courses in various

specialties to students in their first two years of medical school

Students must be prepared to take medical certification exams

Security requirements:1. P&P to online test security2. Secure testing in computer labs3. Reporting to facilitate security

http://www.questionmark.com/us/casestudies/univ_south_alabama.aspx


acknowledged.

Solutions offered to meet requirements:1. P&P to online test security: Central secure storage of

item banks, controlled delivery via secure channels2. Secure testing in computer labs: Questionmark Secure,

shuffling, randomization, proctor/student logins3. Reporting: Many reports to ensure secure testing

Demos1. Score list report: Time taken, IP address2. Coaching Report: Appeals, comparison to benchmark3. Item Analysis Report: Item difficulty drift over time and

psychometric performance4. Test Analysis Report: Defensibility and test difficulty

drift


acknowledged.

Department of Nursing in Syracuse, NY 2 year program Mission to prepare certification exam (NCLEX)

Issues to resolve and security requirements1. Remote testing2. Lock down the computer to eliminate printing,

word processing programs, browsing the internet during the tests

3. Randomization4. Remediation capabilities


acknowledged.

Solutions offered to meet requirements:1. Had to switch from P&P to online (QM used for over

100 exams)2. Secure browser provides more security in non-

proctored locations (uses video surveillance)3. Shuffling choices 4. Random administration of questions 5. Randomize within block6. Limiting the amount of attempts to take tests (4 tries)7. Host with us or in your secure environment

(accessible)Demo

1. Shuffling choices2. Random administration of questions 3. Randomize within block

http://www.questionmark.com/us/casestudies/onondaga_cc.aspx Slide 25


acknowledged.

Thank you for your time and attention! Questions/Comments? Visit our booth to talk to us more We have a whitepaper on this topic

(Delivering Computerized Assessments Safely and Securely)◦ Visit our website to download:

http://www.questionmark.com/us/whitepapers/index.aspx

Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Jeff [email protected]

Greg [email protected]

copyright © 1995-2009 questionmark corporation and/or questionmark computing limited, known...

Documents