testing is changing - questionmark & proctoru briefing

Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Testing is Changing: Practical and Secure Assessment

in the 21st Century

June, 2014

Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Agenda

8.45– 9.30 Registration9.30 – 9.45 Welcomes and introductions

9.45 – 10.55 Testing is Changing: Practical and Secure Assessment in the 21st CenturyDemonstration

10:55 – 11:15 Networking Break11.15 – 12.20 Flexibility and Security – From Test

Authoring to Delivery and ReportingDemonstration

12.20 – 12.30 Q&A

• Founded in 1988

• Offices in US, UK, Canada; Distribution partners in 27 countries

• Questionmark provides technologies and services that enable organizations to measure knowledge, skills, and attitudes securely for certification, regulatory compliance and improved learning outcomes.

Background

• 1,000 current enterprise customers worldwide

• Corporate and Government (Learning/HR function); Higher Education; Awarding Bodies and Test Publishers

Customers

About Questionmark

• Assessment Management System

• Questionmark OnDemand (SaaS/cloud)

• Questionmark Perception (installable software)

• Multilingual

• Localized for dozens of different languages and cultures

• Global 24/7 Support

Products and Services

About Questionmark

Che OsborneVP of Sales

Kerry WilliamsRegional VP Sales, Europe

Ivan ForwardAssessment Solutions

Consultant

Chloe MendoncaMarketing

Introductions: Questionmark

Don Kassner Matt Jaeh

Introductions:

How Testing is Changing

in the 21st Century

Presented by Don Kassner – President, ProctorU

33% of all job openings and nearly 50% of all new jobs created between 2008 and 2018 will require a postsecondary degree or credential.

How can organizations keep up with the demand?

An Educated Work Force

http://www.mcgraw-hillresearchfoundation.org/wp-content/uploads/the-return-on-investment-from-adult-education-and-training.pdf#page=6

70% of companies are planning to implement mobile learning in the next 2 years.

Learning in the 21st Century e-Learning - FOUR out of FIVE

companies are reducing costs, increasing the speed of engagement with learning and reducing time spent away from the business.

http://towardsmaturity.org/article/2013/01/15/towards-maturity-focus-report-mobile-learning-work/http://corporateelearningconsortium.com/wp-content/uploads/2012/12/Leadership-research.pdf

Certification testing is evolving from linear-exams to performance-based assessments

Employers want something that is outcome-based

Evolution of Certification

Long distance and associated travel costs

Not 24/7

Cannot meet scalable demand

Requires time away from business

Requires time-consuming post-exam review

Cannot catch integrity breaches in real-time

No live technical support

Secured Assessments

Onsite ProctoringAutomated Remote

Proctoring

Secured Assessments

Ability to address integrity breaches as they occur

Live technical support

24/7 operation

Multiple location redundancy

Convenient testing from home / workplace

Provides a human element of customer service

Live Online Proctoring

Identity Verification

Identity Verification Attendance Verification

Government-issued identification

Keystroke analysis

Public Records challenge-based quiz

Facial-recognition technology

Fingerprint scanner (session entry verification, not identity)

Password-protected exams

Secure Learning Management Systems

Multiple test banks

Limiting or eliminating “Slippery Slope” exams

Exam Structures

Tools to combat major sources of online plagiarism

Create a culture of assessment integrity among administrators and test-takers

Tighten proctoring requirements

Additional Preventative Measures

Demonstration

What questions do you have?

Ensuring Assessment SecuritySecure Technology and Good Practice

Software as a Service

Installable Software

Assessment Management System

European Data Centre

ISO 2700 and 9001 accredited

State-of-the-art datacentre

SaaS / Rapid Deployment

24/7 Monitoring

Global 24/7 support available

99.9+% Uptime

Protection of candidate/student PII (Personally Identifiable Information)

Protection of valuable assessment content

• Average corporate test: 15,000 GBP

• Average certification test: 100,000 to 200,000 GBP

Protect integrity of test/exam results

• A lot could be on the line…

• Reputation

• Life and Limb

Key drivers for secure assessment delivery

Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.Slide 23

Stakes of assessmentLicensure exams

Certification exams

Summative Assessments

Placement tests

Pre-certification exams

Job Task analysis surveys

Needs analysis surveys

Post-course test

Pre-course test

Formative quizzes

Course Evaluations

Satisfaction surveys

Pre-employment

Public

Certifications

& Licensing

Regulatory Compliance

Sales and Technical Channel

Verification

Threat Level in Higher StakesShort Term with Low Trust Relationships

Long Term with High Trust Relationships

Higher threats require more

“Oversight” and so cost more

Lower threats require less “Oversight”

and so cost less to administer

Impersonation Content Theft Cheating

Key Threats to be Addressed in High-stakes Exam Delivery

Tight controls over the access to content

Shuffling items and choices –limit exposure of item pool

Not exposing the scoring algorithm beyond the content repositories/databases

Securing, and only providing limited access to, the content repositories/databases

Enforceable candidate agreements

Vigorous follow up on infractions

Mitigate Leakage of Content

Randomization

…of choices …of questions

Helpful for…• Re-use of items in practice tests/quizzes• Mitigate cheating/copying from “wandering eyes”

DemonstrationSecure, Collaborative Authoring

Enforceable candidates agreement; formal honesty contracts

Monitoring (Proctoring/Invigilation)

Secure browsers/players on candidate devices

Mitigate Cheating

Questionmark SecureOne Setting – Multiple Platforms

Windows

Printing

Screen capture

Task switching

Control/system keys

Right-click / context menu options

View source

Accidentally exiting exam

Prevents….

Test Centers

Events (Classrooms or Conventions)

Dedicated Kiosk with multiple cameras

Remote real-time webcam

Record & Review / 360 or webcam

Un-Monitored

Options for Monitoring Tests

•Personable, consistent

•Monitor vulnerable to unfair influenceTest Centers

•Convenient, personable, consistent

•Monitor vulnerable to unfair influence

Events (Classrooms or Conventions)

•Monitor is less vulnerable

•New and not yet widespread

Dedicated Kiosk with multiple cameras

•Monitor is less vulnerable

•Easy to deploy for use at home

Remote Real-time webcam

•Authentication and forensic benefits

•Nothing for content theft

Record & Review360 or webcam

•Works for employees

•Nothing for content theftUn-Monitored

Monitoring Tests Securely

√ √ √

√ X X

√ √ √

Reporting and Analytics

Analysis

How are our Assessments doing? How are our Participants doing?

Did the learning investment impact

performance? Where are the skills and knowledge gaps in my

organization?

Are my questions and tests valid and reliable?

What did participants think of the learning

experience?

What did participants learn from the training

or course?

How do I coach participants to improve

performance?

How did one group of test takers compare to

another?

Analyze Time to Complete Exam

Valuable completion time analysis useful to:

• Identify potential cheating• Flag potential content theft• Determine if allotted time is

sufficient for completing the assessment

How do aggregate results compare at one center vs. another?

Analyze Results by Test Centre/Location

Valuable for spotting problems with invigilation

DemonstrationReporting and Analytics

Providing a Great Candidate Experience

Turnkey web portal solutions for awarding

bodies

Configured and branded to your organisation’s

Powered by comprehensive, scalable

and secure platform

Questionmark Portals

An Example Portal

OnDemand Portals On Your Design

Your Logo and Your Color Scheme

Simple For Candidates To RegisterSimple registration

Simple To Sign InSimple Sign In

Menu OptionsDifferent Menu Options for Guests and Candidates

Highlight Key Messages

Quick links to important messages

News Feeds

testing is changing - questionmark & proctoru briefing

Services

the easihe (e-assessment in higher education)...

questionmark vs blackboard for online tests

kristin bernor questionmark

examiner instructions for proctoru...

s t 2022 proctoru review + s t e

using questionmark through blackboard · separate page, to...

application checklist please complete the following … ·...

questionmark conference 2017 santa fe · 2017. 3. 16. ·...

evgeniy kotelva, sap che osborne, questionmark€¦ ·...

how to: setup proctoru for online exam taking · 2020. 5....

giac guide to with proctoru

2009 – current:president, proctoru inc. 2007 –...

questionmark white paper - university of maryland, baltimore

introduction to questionmark’s assessment management...

copyright © 1995-2012 questionmark corporation and/or...

apmg-international taking your online apmg-international...

hrn en 13286 - 2 zbijanje po proctoru

how pg&e’s move to the cloud has - questionmark

formative online assessments using questionmark tracey...

step 1: create your online profile with proctoru