questionmark breakfast briefing: london, september 2012

Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Breakfast Briefing

London

5-September-2012


9:00 – 9:30 Breakfast and Networking

9:30 – 10:45 • Welcomes and introductions • Brief Questionmark Overview • Assessment Technologies and Regulatory Compliance – John

Kleeman

10:45 – 11:00 Networking Break

11:00 – 12:30 Assessments Solutions enabled by Questionmark • Authoring, Delivery and Analytics • Mobile Delivery • New European Data Centre Case Study: HSBC – Matt Bushby Q&A

12:30 Close of Briefing

Agenda

Slide 2


Slide 3

About Questionmark

Founded in 1988

Technologies and services that enable organizations to measure knowledge, skills, and attitudes securely for certification, regulatory compliance and improved learning outcomes – Assessment Management System: On Demand

On Premise

More than 2,500 customers worldwide

Offices in UK, US, Belgium and Germany

Questionmark Perception available in English, Spanish, German, French, Dutch, Norwegian and Japanese Participant facing software in 20 languages

Questionmark Live SME Authoring available in 15 languages


John Kleeman, Questionmark

Chairman

Ivan Forward, Assessment Solutions

Consultant

Che Osborne, VP of Sales

Kerry Williams, UK Sales Manager

Chloe Mendonça, Marketing Assistant

Leila Homps, Account Manager

Carla Tavares, International Account

Executive

Questionmark Staff at Briefing

Slide 4


Education

• Improve learning outcomes

• Increase student satisfaction

• Data for institutional effectiveness, accreditation

Learning, Training & HR in Business and Government

• Regulatory Compliance

• Heath and Safety

• Sales and Technical Channel Verification

• Employee Selection

Awarding Bodies / Certification / Test Publishers

• Efficiently create and maintain valid, reliable tests

• Generate revenue from exams and practice tests

Questionmark enables customers to address key objectives…

Slide 5


Assessment Solutions Enabled

Diagnostic/ Placement

tests

Pre- and Post-course tests

Quizzes & Knowledge

Checks

Summative assessment

Course evaluation

Observational Assessments

Certification & licensing

exams

Job Task Analysis


Global Manufacturing and Technology Company

Deploying OnDemand for Sales Channel Verification Assessment Program to Sales Reps Worldwide


Power Generation and Distribution Utility in Midwest

Deploying Assessments for 7,000 employees to test for safety, regulatory compliance and certification of knowledge


Large Automotive Parts Retailer with 50,000+ Employees

New OnDemand Customer Deploying Product Knowledge Assessment Program at 3,000+ US Retail locations

Slide 9


Indian government agency responsible for Technical College entrance exams

All entrance exams will be delivered via Questionmark OnDemand

Slide 10


Global Services Company Deploying Questionmark for Drivers Examinations

Helped firm address security concerns

Slide 11


Large Medical Certification body deploying Questionmark for Observational Assessment

Objective Structured Clinical Examination (OSCE)

Slide 12


Large Insurance Company deploys Questionmark for

Compliance and On-boarding 200,000+ assessments annual for new hires and compliance

testing

Slide 13


Global IT company for partner certification

300,000 at home assessments

50,000 in test centers

Slide 14


Large Transportation company deploys Questionmark with

SAP Connector 15,000 Pre-employment tests


Nationwide College Entry Exams Deployed on Questionmark

in India 3 million exams per year

Slide 16


Assessment Technologies and Regulatory Compliance

John Kleeman, Questionmark Chairman


3 huge fines in summer 2012

Slide 18

Can assessments make compliance easier and reduce risk?


Setting the scene

5 reasons why assessments mitigate compliance risk 1. Demonstrate commitment

2. Find problems early

3. Document understanding after training

4. Increase knowledge / reduce forgetting

5. Reduce cost & time for compliance training

5 key pieces of practical advice on using assessments

Agenda

Slide 19


Primarily focused on financial structure

Banks need capital and must not take risks …

But employee performance is part of this

Basel guidelines say: compliance function should assist senior management in

educating staff on compliance issues

Financial Services : Basel II and III

Slide 20


Combat Terrorism Organized crime

FATF guidelines include a risk-based process

“testing to assess knowledge commensurate with the detail of information provided”

UK FSA example of poor practice: “employees were merely required to sign a form to confirm that

they were aware of the firm’s AML procedures, but their understanding was never tested.”

Financial Services : Anti-money laundering

Slide 21

http://www.fatf-gafi.org/


Two main impetuses US Foreign Corrupt Practices Act

UK Bribery Act

Both extra-territorial

Penalties can be large US$800m settlement with Siemens

US government being much more proactive

UK getting into act

Training and procedures against bribery can be a defence – and stop fines

All industries : anti-bribery and corruption

Slide 22


Consumer protection Testing sales people

Industry technical regulations

General business regulation Accuracy of accounts

Anti-competition laws

HR (e.g. equality)

Health and Safety

Privacy and data protection

Other important regulatory drivers

Slide 23


Reason #1 : Demonstrate commitment

Slide 24


Because employees follow their leaders Charles Jennings (August 2012)

“… common thread that runs through almost all high-profile compliance catastrophes. It is that the top-tier executives and middle managers in the organisations simply don’t model the behaviours that will lead to a culture of compliance”

Why is it important to demonstrate top level commitment?

http://charles-jennings.blogspot.co.uk/2012/08/compliance-training-does-it-really-work.html


Require all employees and managers/executives to Take assessments every 6 or 12 months

Remedial training and retake assessment if fail

Assessments seen as fair

Major stride in demonstrating commitment

How do assessments demonstrate commitment?

Slide 26


UK FSA 2009: “Aon Ltd should have ensured that appropriate members of

staff – particularly those in the Aviation and Energy divisions – received focused training in relation to this area and were tested on their understanding of the relevant risks involved. Effective training and testing in this regard would have emphasised to staff the importance of carrying out effective due diligence prior to authorising an Overseas Third Party for payment.”

Fine of UK£5.5m for not testing, partly due to lack of commitment

Slide 27

http://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&frm=1&source=web&cd=1&cad=rja&ved=0CCMQFjAA&url=http://www.fsa.gov.uk/pubs/final/aon.pdf&ei=4Vs2UIymL6nX0QXS8YHIDQ&usg=AFQjCNGiTQcXBKnPpAX-4-wJjSibMpatFw


Reason #2 : Find problems early

Slide 28


Surveys identify issues in culture and ethics

Most employees want to help. From employee of bank involved in recent LIBOR scandal: “Always happy to help, leave it with me, Sir”.

UK Ministry of Justice Bribery Act Guidelines “Staff surveys, questionnaires and feedback from training can also provide an important source of information on effectiveness and a means by which employees and other associated persons can inform continuing improvement of anti-bribery policies”.

How surveys can find problems early

Slide 29

http://www.cftc.gov/ucm/groups/public/@lrenforcementactions/documents/legalpleading/enfbarclaysorder062712.pdf

http://www.justice.gov.uk/downloads/legislation/bribery-act-2010-guidance.pdf


Look at topic and assessment scores By department

By geography

Over time

Look at wrong answers

Look at test failures

Assessments touch every employee: a unique tool

How tests/quizzes/exams can find problems early


Reason #3 : Document understanding after training

Slide 31


Because regulators require it common regulatory requirement, e.g. OECD recommends on

ethics/compliance programme:

measures designed to ensure periodic communication, and documented training for all levels of the company

Because makes business sense Trained employees more effective

“If you can not measure it, you can not improve it.“

Why document training for compliance?


Take attendance

Get employee signatures

Test to check understanding

Common Methods of Documenting Training

Slide 33


How to do Trainer records attendance or takes photo of attendees

Record online attendees

Pros Easy to do

Measures who turns up

Cons Did they stay the whole time?

Did they pay attention?

Did they get it?

Take attendance

Slide 34


How to do Get employees to sign a form confirming

that they attended training AND that they understand the rules

Pros Gets employee confirmation

Employee committed to follow rules

Cons People may sign but not take seriously

People may sign without attending or understanding

Get employee signatures

Slide 35


Not where safety is involved according to 2005 United States Appeal Court: “merely having an individual sign a form acknowledging his responsibility to read the safety manual is insufficient to insure that the detailed instructions contained therein have actually been communicated.”

Not according to this UK FSA example of poor practice: “… employees were merely required to sign a form to confirm that they were aware of the firm’s AML procedures, but their understanding was never tested.”

Does getting an employee to sign give protection?

Slide 36

http://www.fsa.gov.uk/pubs/other/jmlsg_guidance.pdf


How to do: After training, test employees People who fail test undergo

remedial training

Pros Genuinely check understanding Allows improvement of weaknesses Feedback to improve training

Cons More effort You have to follow through & take

action on failures

Test to check understanding

Slide 37


Reason #4 : Increase knowledge / reduce forgetting

Slide 38


Compliance Training

Slide 39

Receive information

Working memory

Long term memory


Cue

• Question

• Other stimulus

Search • Retrieve from long

term memory

Act • Act or respond

Psychology research shows retrieval practice helps future retrieval

Successful retrieval modifies memory making it more retrievable in future

Training useful if can retrieve

Slide 40


In a Perfect World…

Slide 41


In a Perfect World… we would remember everything we learn!

Slide 42


Learning in the Actual World They don’t get it All!

Slide 43


Learning Curve

Slide 44

Ouch! The Forgetting Curve

Time

Kn

ow

led

ge/S

kills

Lear

nin

g

Exp

erie

nce

Lea

rnin

g

Exp

erie

nce

Lea

rnin

g

Exp

erie

nce

Qu

esti

on

s Q

ues

tio

ns

Purpose: • Content repetition • Memory retrieval practice • Strengthens memory recall

Purpose: • Content repetition • Memory retrieval practice • Strengthens memory recall


In the Actual World… we forget some of it!

Slide 45


Slow the Forgetting Curve

Slide 46

Time

Kn

ow

led

ge/S

kills

Lear

nin

g

Exp

erie

nce

Lea

rnin

g

Exp

erie

nce

Lea

rnin

g

Exp

erie

nce

Qu

esti

on

s Q

ues

tio

ns

Qu

esti

on

s Q

ues

tio

ns

Purpose: • Memory retrieval practice • Strengthens memory recall • Reduce Forgetting Curve

Purpose: • Memory retrieval practice • Strengthens memory recall • Reduce Forgetting Curve


Reason #5 : Reduce cost & time for compliance training

Slide 47


Take test If pass, skip

training Otherwise do training

Testing out of compliance training

Slide 48


Management Report Example

A) Assessment Completions 1000

B) Pass the assessment and test out 375

C) Do the compliance training 625

D) Time for compliance training 1.5 hours

E) Hours of training avoided (B * D) 562.5

F) Average cost / hour $70

G) Money saved (E * F) $39,375

Example of how can save money

Slide 49


1. Demonstrate commitment

2. Find problems early

3. Document understanding after training

4. Increase knowledge / reduce forgetting

5. Reduce cost & time for compliance training

Summary on why to use assessments


Good practice in using assessments

5 key practical steps for compliance assessments

Slide 51


Medium and higher stakes Internal exams Post-course tests Diagnostic tests Observational/workplace assessments

Lower stakes assessments

Course evaluation surveys Employee attitude surveys Job task analysis surveys Practice tests Quizzes during learning 360 degree performance assessments

Common assessments in compliance

Slide 52


Create scenario questions Assess application not just knowledge


Evaluation

Synthesis

Analysis

Application

Comprehension

Knowledge

Questions better if use application / comprehension

NOT just knowledge

Blooms Taxonomy


What does a yellow traffic light mean? Stop

Go

Caution

If you are driving toward an intersection and the light turns from yellow to red, what should you do? Speed up and cross the

intersection

Stop suddenly

Stop gradually

Example

Slide 55


Use feedback at the topic level It is actionable

Slide 56


What to do Divide assessment by topic areas

Give individual feedback

Topic scores

Resources or advice to improve weak topics

Analyze at department or organization level

Topic feedback is actionable Direct participant to specific resources to improve learning

Direct compliance to areas knowledge is weak

How topic feedback can help


Feedback based topics of questions

Slide 58

Corrective feedback points participant to appropriate learning

resources

“Topics” of questions

answered in assessment

Score on topic


Make assessments accessible Any large organization will have visually impaired or other participants with accessibility needs

Slide 59


Accommodations to consider

Slide 60

Disability

Loss of vision

Visual impairment

D/deafness, hard of hearing

Motor impairment

Cognitive impairment

Accommodation

Screen reader support

Contrast and text size

Text alternatives

Keyboard accessible

Text size, extra time


Screenshots in questions

Use of color

Long text that is hard to read

Simulations & graphics using Flash

Time limits (may need to extend for some)

Make sure your assessment software supports accessible assessments

Be careful about

Slide 61


Have participants agree a code of conduct Help them be honest

Slide 62


Fraud needs: Motivation

Opportunity

Rationalization

Fraud triangle

Slide 63


Motivation depends on stakes

Slide 64

Higher Stakes

High

Medium

Low

Medium Stakes

Low Stakes

What’s at Stake?

Life and Limb

Promotion & Jobs & Legal Concern

Educational Exams

Tests

Elearning & Surveys


Impersonation Content Theft Cheating

Opportunity

Slide 65

Monitoring/proctoring helpful, e.g. having manager supervise employees


Reduce rationalization to cheat Make the programme fair

Communicate widely that it is fair

A code of conduct (honor code) can help

Idea borrowed from universities

Employees commit at start of assessment to be honest

Rationalization

Slide 66


Consider observational assessments Measure skills and behavior

Slide 67


Traditional assessment

• Participant is presented with questions for direct response

Observational assessment

• Observer is presented with questions to ‘rate’ participant to measure knowledge, skills and abilities

Observational assessment vs. ‘traditional’ quiz, test or exam

Slide 68


Preparation Performing the

task Wrap Up

Observer typically rates participant in 3 areas

Smartphones/tablets make more practical


1. Create scenario questions

2. Use feedback at the topic level

3. Make assessments accessible

4. Have participants agree a code of conduct

5. Consider observational assessments

Actionable good practice

Slide 70


Excellent book on assessment Criterion-Referenced Test

Development

by Shrock and Coscarelli

More good practice

Slide 71


Questionmark white paper

www.questionmark.com/uk/whitepapers

More good practice

Slide 72

http://www.questionmark.com/uk/whitepapers


Your questions

Slide 73


Break

Slide 74


Create • Collaborative, secure

authoring of surveys, quizzes, tests and exams

Deliver • Blended delivery to

support a broad range of devices and security needs

Analyze • Evaluate results and

inform stakeholders with timely, meaningful reports

Assessment Management

System


Create • Collaborative, secure

authoring of surveys, quizzes, tests and exams

Deliver • Blended delivery to

support a broad range of devices and security needs

Analyze • Evaluate results and

inform stakeholders with timely, meaningful reports

Software-as-a-Service

For OnPremise Deployment


New European Data Centre

Slide 77

ISO 2700 and 9001 accredited

State-of-the-art datacenter

SaaS / Rapid Deployment

24/7 Monitoring

Global 24/7 support available

99.9+% Uptime


Authoring

Delivery

Analytics

Integration

Slide 78


Author, Deliver and Analyse via the Cloud

Internet


Distributed authoring and item management

Internet

Assessments

Questions

Windows based

Authoring Manager


Slide 81

Wizards to create 21 item types

Drag and Drop *Essay *Explanation Fill in Blanks Hotspot *Likert Adobe Flash Adobe Captivate Matching Knowledge Matrix Survey Matrix

*Multiple Choice *Multiple Response Numeric File Upload *Pull-down List Ranking Select a Blank *Text Match *True/False *Yes/No

Enable authors to create more interesting and probing questions

Reveals more information during an assessment in a shorter time

Not offered by simpler assessment tools in LMS’s


Subject Matter Experts Author

Questions in Questionmark Live

Questionmark Live: Simplify Item Writing, Content Harvesting from SMES

Perception Repository

SME content

SME questions

SME questions


Interactive Translations Using Questionmark TMS

Translations using 3rd Party Tools

Translation Management System

Translation methods

Slide 83

Export to XLIFF (XML)

Translate using 3rd Party Tools

Import from XLIFF (XML)

Translation

Management

System

Manages translation process ….does not do translation

Set Up Project using TMS

On-The-Fly


Demonstration

Slide 84


Authoring

Delivery

Analytics

Integration

Slide 85


Formative Quizzes Course Evaluations

Diagnostic Tests

Low Stakes Tests

Medium Stakes Exams

High Stakes Exams

Levels

of M

onitoring

Blended Delivery

Stakes of Assessment


Browser Support Critical for Reliable Participant Experience

Slide 87


Browser Support Critical for Reliable Participant Experience

Slide 88


Support for Leading Browsers

Slide 89

Internet Explorer

Firefox Safari

for Mac Chrome

Safari for iOS 4 & 5

Android 2+


Auto Sense Device/Browser

Slide 90

Senses Device/Browser

Questionmark Packages the Assessment for Device/Browser/iframe

On-the-fly auto-sizing of skin and controls

Sends HTML


Demonstration

Slide 91


Mobile Delivery of Assessments

Observational assessments

Mobile exam rooms

Deliver to the device in your

hand


Mobile Apps

Android

iPhone, iPad and iPod touch

http://itunes.apple.com/us/app/questionmark-for-ipad/id377556777?mt=8


Observer rates participants’

performance, skills and abilities

Preparation Performing the

task Wrap Up

Observational Assessment


Uses of Observational Assessment

Certification Process

• Observational assessments can be used as critical components of the certification process – in some cases are required by regulatory authorities.

Medical and Dental School exams:

• Medical and dental schools make extensive use of observational assessments such as the Objective Structured Clinical Examination (OSCE).

Equipment operation:

• Observational assessment can be used to record and rate operators' skills, abilities, safety practices and adherence to required procedures


~70% from Experience

~20% from Others

~10% from Study

Effective application of 70+20+10 requires flexible learning and assessment technologies

Slide 96

Wikipedia: http://en.wikipedia.org/wiki/70/20/10_Model

Dr. Charles Jennings

http://en.wikipedia.org/wiki/70/20/10_Model




Wikis

Blogs

Forums

Social Networking

Portals

Technologies used to support social/informal learning initiatives

Slide 97


Embedding Assessments in Wikis, blogs, portals


Benefits of Embedding in Blogs, Portals, Wiki’s

For Learners For Organizations

Focus the learners’ attention

Practice memory recall

Identify what was learned

Provide feedback to assist the learner to: Correct misconceptions

Instill confidence

Point to relevant material

Embrace 70+20+10 with confidence

Easily use assessments throughout the learning process

Provide seamless user experience

Centralized management of assessment content and results

Slide 99


Authoring

Delivery

Analytics

Integration

Slide 100


Did the learning investment impact

performance?

Where are the skills and

knowledge gaps in my organization?

Are my questions and tests valid and

reliable?

What did participants think

of the learning experience?

What did participants learn from the training

or course?

How do I coach participants to

improve performance?

How did one group of test takers compare to

another?

Analyze and report on results


Demonstration

Slide 102


Authoring

Delivery

Analytics

Integration

Slide 103

Can I integrate Questionmark with various other Enterprise Systems?

https://www.questionmark.com/us/aicc_information.aspx

http://www.viewcentral.com/

http://www.sumtotalsystems.com/

http://www.plateau.com/

http://www.trainingpartner.com/

http://www.meridianksi.com/


Integrating with SharePoint

Slide 105

Embedding assessment lists using

SharePoint Connector for OnDemand

Embedding assessment lists and assessments using

Windows Authentication


SAP Connector

Slide 106


Blackboard Connector (v8 and 9)

Slide 107


• Content

• Delivery

• Results

Secure

• Millions of tests delivered

• 99.99% Uptime

• Tests available globally

Scalable

• Author in Browser

• Multilingual

• Mobile, Blended Delivery

Agile

• Accessible

• Reliable

• 24/7 Support

Trustable

Why Questionmark?


Ways to Evaluate

Slide 109

Try Questionmark OnDemand

• 30 Day Trial – Software as a Service

• https://www.questionmark.com/ondemand

Try Questionmark Perception

• Download – 30 day trial

• https://www.questionmark.com/download

https://www.questionmark.com/ondemand

https://www.questionmark.com/download


Slide 110

Ways to Learn More

Questionmark Blog!

• http://blog.questionmark.com

Web Seminars

• http://www.questionmark.com/uk/seminars

White Papers

• http://www.questionmark.com/whitepapers

Manuals, Knowledge Base and Best Practices Guides

• http://www.questionmark.com/perception/help

http://blog.questionmark.com/

http://www.questionmark.com/uk/seminars

http://www.questionmark.com/whitepapers

http://www.questionmark.com/perception/help


The Role of Assessments in Mitigating Risk for Financial Services Organizations

Assessments through the Learning Process

Security of Questionmark’s OnDemand Service

Alignment, Impact and Measurement with the A-model

Embedded Assessments: Building Knowledge Checks, Surveys and Other Assessments into Learning Materials

How Questionmark Aids ISO 10667 Compliance

Recommended White Papers

Slide 111

https://www.questionmark.com/whitepapers

https://www.questionmark.com/whitepapers


Thank you for attending – Keep in Touch!

Slide 112

blog.questionmark.com

www.questionmark.com/newsletter

twitter.com/questionmark

linkedin.com Look for Questionmark under “Groups” and join

youtube.com/questionmarkvideo

flickr.com/questionmarkpictures

facebook.com Like us on Facebook

http://www.questionmark.com/newsletter

questionmark breakfast briefing: london, september 2012

Technology

questionmark authoring

questionmark staff

safety sales

assessments solutions

networking break11

student satisfaction

customers worldwide

close of briefing copyright