assess to comply: how else can you be sure that employees understand? ecoa 2012

Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Assess to comply: How else can you be sure that

employees understand?John Kleeman, Questionmark (www.questionmark.com)

ECOA 20th Annual Ethics & Compliance Conference St Louis, October 2012

http://www.questionmark.com/


John Kleeman MA MBCS C.Eng Worked in assessment software 25 years Chairman and Founder of Questionmark Questionmark used by 1000+ organizations for

compliance and other assessments Contact:

Email: [email protected] Twitter : @johnkleeman

John Kleeman

Slide 2

@theECOA | #AECC2012

mailto:[email protected]

http://twitter.com/johnkleeman

http://twitter.com/johnkleeman


Increasing regulation Companies expected to perform Business and regulators international

(e.g. FCPA, UK Bribery Act) Regulators levying heavier penalties

Several companies fined US$300m+ in 2012

Can assessments make compliance easier and reduce risk?

Setting the scene

Slide 3



5 reasons why assessments mitigate compliance risk1. Demonstrate commitment2. Find problems early3. Document understanding after training4. Increase knowledge / reduce forgetting5. Reduce cost & time for compliance training

5 key pieces of practical advice on using assessments

Agenda

Slide 4



Medium and higher stakes Internal exams Post-course tests Diagnostic tests Observational/workplace assessments

Lower stakes assessments Course evaluation surveys Employee attitude surveys Job task analysis surveys Practice tests Quizzes during learning 360 degree performance assessments

Common assessments in compliance

Slide 5



Reason #1 : Demonstrate commitment

Slide 6



Because employees follow their leaders Charles Jennings (August 2012)

“… common thread that runs through almost all high-profile compliance catastrophes. It is that

the top-tier executives and middle managers in the organisations simply don’t model the behaviours that will lead to a culture of compliance”

Why is it important to demonstrate top level commitment?


http://charles-jennings.blogspot.co.uk/2012/08/compliance-training-does-it-really-work.html


If you require all employees and managers to Take assessments every 6 or 12 months Remedial training and retake assessment if fail

Assessments seen as fair Major stride in demonstrating commitment

How do assessments demonstrate commitment?

Slide 8



UK FSA 2009: “Aon Ltd should have ensured that appropriate members of

staff – particularly those in the Aviation and Energy divisions – received focused training in relation to this area and were tested on their understanding of the relevant risks involved. Effective training and testing in this regard would have emphasised to staff the importance of carrying out effective due diligence prior to authorising an Overseas Third Party for payment.”

Fine of UK£5.5m for not testing, partly due to lack of commitment

Slide 9


http://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&frm=1&source=web&cd=1&cad=rja&ved=0CCMQFjAA&url=http://www.fsa.gov.uk/pubs/final/aon.pdf&ei=4Vs2UIymL6nX0QXS8YHIDQ&usg=AFQjCNGiTQcXBKnPpAX-4-wJjSibMpatFw


Reason #2 : Find problems early

Slide 10



Surveys identify issues in culture and ethics Most employees want to help. From employee of bank

involved in recent LIBOR scandal:“Always happy to help, leave it with me, Sir”.

UK Ministry of Justice Bribery Act Guidelines“Staff surveys, questionnaires and feedback from training can also provide an important source of information on effectiveness and a means by which employees and other associated persons can inform continuing improvement of anti-bribery policies”.

How surveys can find problems early

Slide 11


http://www.cftc.gov/ucm/groups/public/@lrenforcementactions/documents/legalpleading/enfbarclaysorder062712.pdf

http://www.justice.gov.uk/downloads/legislation/bribery-act-2010-guidance.pdf


Look at topic and assessment scores By department By geography Over time

Look at wrong answers Look at test failures Assessments touch every employee: a unique tool

How tests/quizzes/exams can find problems early



Reason #3 : Document understanding after training

Slide 13



Because regulators require it common regulatory requirement, e.g. OECD recommends on

ethics/compliance programme:measures designed to ensure periodic communication, and

documented training for all levels of the company

Because makes business sense Trained employees more effective “If you can not measure it, you can not improve it.“

Why document training for compliance?



Take attendance

Get employee signatures

Test to check understanding

Common Methods of Documenting Training

Slide 15



How to do Trainer records attendance or takes photo of attendees Record online attendees

Pros Easy to do Measures who turns up

Cons Did they stay the whole time? Did they pay attention? Did they get it?

Take attendance

Slide 16



How to do Get employees to sign a form

confirming that they attended training AND that they understand the rules

Pros Gets employee confirmation Employee committed to follow rules

Cons People may sign but not take seriously People may sign without attending or

understanding

Get employee signatures

Slide 17



Not where safety is involved according to 2005 United States Appeal Court:“merely having an individual sign a form acknowledging his responsibility to read the safety manual is insufficient to insure that the detailed instructions contained therein have actually been communicated.”

Complete General Const. Co. v. OSHRC, 2005 U.S. App. LEXIS 5197 (6th Cir.), March 29, 2005

Does getting an employee to sign give legal protection?

Slide 18



SEC on FCPA compliance:Titan’s only related ‘policy’ is a statement in Titan’s Code of Ethics, which all Titan employees were required to sign annually, stating employees must be familiar with and strictly adhere to such provisions … Titan did not enforce that policy nor did it provide its employees with any information concerning the FCPA or its purposes”.

UK FSA example of poor practice:“… employees were merely required to sign a form to confirm that they were aware of the firm’s AML procedures, but their understanding was never tested.”

What do regulators think?

Slide 19


http://www.sec.gov/litigation/complaints/comp19107.pdf

http://www.fsa.gov.uk/pubs/other/jmlsg_guidance.pdf


How to do: After training, test employees People who fail test undergo remedial

training Pros

Genuinely check understanding Allows improvement of weaknesses Feedback to improve training

Cons More effort You have to follow through & take

action on failures

Test to check understanding

Slide 20



Reason #4 : Increase knowledge / reduce forgetting

Slide 21



Because of the famous arch?

Should it be for Washington University in St. Louis psychology research?

Why is St Louis famous?



Compliance Training

Slide 23

Receive informati

on

Working memory

Long term

memory



Cue •Question•Other stimulus

Search •Retrieve from long term memory

Act •Act or respond

Research at Washington University in St. Louis (and elsewhere) shows retrieval practice helps future retrieval

Successful retrieval modifies memory making it more retrievable in future

Training useful if can retrieve

Slide 24



120 students divided into groups randomly Given reading comprehension passages used in TOEFL Some students

Study for 14 mins Some students

Study for 7 mins Recall material for 7 mins

Compared results after 5 mins and 1 week

Roediger & Karpicke experiment 2006 (in St. Louis)

Slide 25



After 5 mins After 1 week0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

RestudyQuiz

How much learners retained

Slide 26



SAT II Bloom Definition Bloom Application0%

10%

20%

30%

40%

50%

60%

No quizQuiz

Marsh, Roediger, Bjork & Bjork 2007. Multiple choice quizzes aid long term retention

Slide 27



Slide 28

Learning in the Actual WorldThey don’t get it All!



Learning Curve

Slide 29

Ouch! The Forgetting Curve

Time

Know

ledg

e/Sk

ills

Lear

ning

Ex

perie

nce

Lea

rnin

g E

xper

ienc

e

Lea

rnin

g E

xper

ienc

e

Que

stion

s

Que

stion

s

Purpose:• Content repetition• Memory retrieval practice• Strengthens memory recall

Purpose:• Content repetition• Memory retrieval practice• Strengthens memory recall



In the Actual WorldThey Forget It!

Slide 30



Slow the Forgetting Curve

Slide 31

Time

Know

ledg

e/Sk

ills

Lear

ning

Ex

perie

nce

Lea

rnin

g E

xper

ienc

e

Lea

rnin

g E

xper

ienc

e

Que

stion

s

Que

stion

s

Que

stion

s

Que

stion

s

Purpose:• Memory retrieval practice• Strengthens memory recall• Reduce Forgetting Curve

Purpose:• Memory retrieval practice• Strengthens memory recall• Reduce Forgetting Curve



Reason #5 : Reduce cost & time for compliance training

Slide 32



Take test If pass, skip training Otherwise do training

Testing out of compliance training

Slide 33



Management Report Example

A) Assessment Completions 1000

B) Pass the assessment and test out 375

C) Do the compliance training 625

D) Time for compliance training 1.5 hours

E) Hours of training avoided (B * D) 562.5

F) Average cost / hour $70

G) Money saved (E * F) $39,375

Example of how can save money

Slide 34



Many regulators permit “testing out” to reduce compliance training ABA Bank Compliance magazine article 2009 interviewed

some US regulators OTS: Testing out acceptable OCC: Agree, providing testing program well structured. May be

some exceptions FDIC : Agree

Check with your regulator

Will the regulator allow?


http://www.aba.com/Products/bankcompliance/Documents/8bdcd6492072495e84d8322ce10cd0e9BCJF2009Training.pdf


1. Demonstrate commitment2. Find problems early3. Document understanding after training4. Increase knowledge / reduce forgetting5. Reduce cost & time for compliance training

Summary on why to use assessments



Good practice in using assessments5 key practical steps for compliance assessments

Slide 37



Create scenario questionsAssess application not just knowledge



Evaluation

Synthesis

Analysis

Application

Comprehension

Knowledge

Questions better if use application / comprehension

NOT just knowledge

Blooms Taxonomy


What does a yellow traffic light mean? Stop Go Caution

If you are driving toward an intersection and the light turns from yellow to red, what should you do? Speed up and cross the

intersection Stop suddenly Stop gradually

Example

Slide 40



Use feedback at the topic levelIt is actionable

Slide 41



What to do Divide assessment by topic areas Give individual feedback

Topic scores Resources or advice to improve weak topics

Analyze at department or organization level

Topic feedback is actionable Direct participant to specific resources to improve learning Direct compliance to areas knowledge is weak

How topic feedback can help



Feedback based topics of questions

Slide 43

Corrective feedback points participant to appropriate

learning resources

“Topics” of questions

answered in assessment

Score on

topic


Make assessments accessibleAny large organization will have visually impaired or other participants with accessibility needs

Slide 44



Accommodations to consider

Slide 45

Disability

Loss of vision

Visual impairment

D/deafness, hard of hearing

Motor impairment

Cognitive impairment

Accommodation

Screen reader support

Contrast and text size

Text alternatives

Keyboard accessible

Text size, extra time


Screenshots in questions Use of color Long text that is hard to read Simulations & graphics using Flash Time limits (may need to extend for some)

Make sure your assessment software supports accessible assessments

Be careful about

Slide 46



Have participants agree a code of conductHelp them be honest

Slide 47



Fraud needs: Motivation Opportunity Rationalization

Fraud triangle

Slide 48



Motivation depends on stakes

Slide 49

Higher Stakes

High

Medium

Low

Medium Stakes

Low Stakes

What’s at Stake?

Life and Limb

Promotion & Jobs & Legal Concern

Educational Exams

Tests

Elearning & Surveys


Impersonation Content Theft

Cheating

Opportunity

Slide 50

Monitoring/proctoring helpful, e.g. having manager supervise employees



Reduce rationalization to cheat Make the programme fair Communicate widely that it is fair A code of conduct (honor code) can help

Idea borrowed from universities Employees commit at start of assessment to be honest

Rationalization

Slide 51



Consider observational assessmentsMeasure skills and behavior

Slide 52



Traditional assessment• Participant is presented with questions for direct

response

Observational assessment• Observer is presented with questions to ‘rate’

participant to measure knowledge, skills and abilities

Observational assessment vs. ‘traditional’ quiz, test or exam

Slide 53


Preparation

Performing the task Wrap Up

Observer typically rates participant in 3 areas

Smartphones/tablets make more practical


1. Create scenario questions2. Use feedback at the topic level3. Make assessments accessible4. Have participants agree a code of conduct5. Consider observational assessments

Actionable good practice

Slide 55



Excellent book on assessment Criterion-Referenced Test Development Shrock and Coscarelli

Questionmark white papers on good practice in compliance and learning assessments www.questionmark.com/whitepapers

Papers by Professor Roediger at Washington University in St Louis http://psych.wustl.edu/memory/publications/

What’s the best way to document that training has taken place Article by John Kleeman in Compliance & Ethics

Professional, July/August 2012

More good practice and resources

Slide 56

http://www.questionmark.com/whitepapers

http://psych.wustl.edu/memory/publications/

https://www.questionmark.com/us/news/Documents/Compliance-and-Ethics_0712_Kleeman.pdf


Your questions

Slide 57
