the blueprint for radically improved training...
TRANSCRIPT
![Page 1: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/1.jpg)
Phone: 888‐579‐9814Email: info@work‐learning.comWebsite: Work‐Learning.comAudits: LearningAudit.netSmiles: SmileSheets.comBlog: WillAtWorkLearning.comTwitter: @WillWorkLearn
Will Thalheimer, PhDPresidentWork‐Learning Research, Inc.Somerville, Massachusetts, US
July 2016
The Blueprint for Radically Improved Training Evaluations
“The goal of training evaluation is not to prove the value of training; the goal of evaluation is to
improve the value of training.” (p. 94‐95)
Tim Mooney and Rob BrinkerhoffCourageous Training:
Bold Actions for Business Results.
![Page 2: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/2.jpg)
Demonstrate the Value
Improve theLearning
Three Reasons to Use Measurement in Learning
# 1
# 2
# 3
The Kirkpatrick 4-Level ModelLevel 1
LearnerReaction
Level 2
Tests of Learning
Level 3
On‐the‐Job Behavior
Level 4
Business Results
![Page 3: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/3.jpg)
“Training Evaluations”
reaction sheetshappy sheets
level 1’s
Bridging Gap between Research and Practice
![Page 4: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/4.jpg)
Research
Simplicity is the Ultimate Sophistication
Leonardo da Vinci
![Page 5: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/5.jpg)
The Decisive Dozenfor Learning Design and Learning Measurement
http://is.gd/ddResearch
1. Content
2. ExposureBaseline
3. Guiding Attention
4. Creating Correct Conceptions
5. Repetition
6. Feedback
7. Variation
Engagement & Understanding
8. Retrieval Practice
9. Context Alignment
10. SpacingRemembering
11. Persuasion
12. PerseveranceApplication
![Page 6: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/6.jpg)
http://Work-Learning.com/catalog.html
![Page 7: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/7.jpg)
Quite simply, the BEST book on smile sheet creation and utilization, Period!
Karl M. KappProfessor of Instructional Technology
Bloomsburg University
Thoughtful and sensible advice for feedback tools that will provide valid and actionable data.
Robert O. BrinkerhoffProfessor Emeritus, Western Michigan
University & Director, Brinkerhoff Evaluation Institute
Evidence‐based practice at the master level.
Julie DirksenAuthor of Design For How People Learn
Background
![Page 8: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/8.jpg)
This year, improving your smile sheetswill likely be the MOST IMPORTANTthing you can do to improve learning!
Professional Development
Pro
1. Red‐flagging training programs that are not sufficiently effective.
2. Gathering ideas for ongoing updates and revision of a learning program.
3. Judging strengths and weaknesses of a pilot program to enable revision.
4. Providing instructors with feedback to aid their development.
5. Helping learners reflect on and reinforce what they learned.
6. Helping learners determine what (if anything) they plan to do with their learning.
7. Capturing learner satisfaction data to understand—and make decisions that relate to—the reputation of the training and/or the instructors.
8. Upholding the spirit of common courtesy by giving learners a chance for feedback.
9. Enabling learner frustrations to be vented—to limit damage from negative back‐channel communications.
Modified based on work by Robert Brinkerhoff
Reasons for Smile Sheets
![Page 9: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/9.jpg)
# 1
We Work in aLearning-to-Performance
Ecosystem
LearningIntervention
PerformanceSituation
LearningOutcomes
IndividualResults
OrganizationalResults
On-the-Job Learning
Prompting
The Learning Landscape
Learning
Remembering
On-the-Job Performance
Later on the job, learners remember
what they’ve learned.
The learners get a return on their efforts.
Performance can be prompted
through job aids, signage,
intuitive cues, performance
support, management, etc.
Learners can learn on-the-job through
trial & practice, insight learning, help from others,
social media, studying on their
own, etc.
© Copyright 2009-2016 Work-Learning Research, Inc.
YouTube: http://is.gd/LearningLandscape
![Page 10: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/10.jpg)
# 2
We Should NOTExpect Perfection
20
Measurement doesn’t
illuminate everything.
Measurement
Measurement illuminates
some things.
![Page 11: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/11.jpg)
There is no perfect measurement tool!
How would you rate your organization’s smile sheets?
Terrible
1
Bad
2
Okay
3
Good
4
Given, that there is NO perfect
measurement…
![Page 12: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/12.jpg)
# 3
The Kirkpatrick ModelControls our Thinking!
The Kirkpatrick 4-Level ModelLevel 1
LearnerReaction
Level 2
Tests of Learning
Level 3
On‐the‐Job Behavior
Level 4
Business Results
![Page 13: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/13.jpg)
LearningIntervention
PerformanceSituation
LearningOutcomes
IndividualResults
OrganizationalResults
On-the-Job Learning
Prompting
The Learning Landscape
Learning
Remembering
On-the-Job Performance
© Copyright 2009-2016 Work-Learning Research, Inc.Regular
smile sheets
Immediate tests of
remembering & decision-
making
Does Job Performance
Improve?
Does Business
Performance Improve?
IR
L1-Delayed
L2-Delayed
L3-Supports
PD
RS JS PS
L1
L2Are Prompt Supports
Used? Effective?
Are Just-in-Time Learning
Supports Used?
Effective?
Are Retrieval Supports
Used? Effective?
Smile sheets
Retrieval &
Decision Making
Evaluation of Coaching & Learner Learning
Delayed smile
sheets
Delayed tests of remembering & decision-making
Do the learners actually benefit
from the learning in their work or
career?
Does the work environment
support performance?
Are Prompting Devices Used?
Effective?
On smile sheets, ask about
motivation to apply
L1
L2
L3
L4
Problems with the Kirkpatrick/Phillips 4- or 5-Level Models of Learning Evaluation
• Pushes us to focus on weighing outcomes. Is largely silent on learning support and learning-design improvement.
• Training centric. Ignores prompting mechanisms & on-the-job learning.
• Ignores the role that management and the business side must play.
• Implies that higher levels are more important than lower levels.
• Ignores the causal chain from learning to remembering to performance to results.
• Ignores the fact that learners forget and that learning interventions can be good at creating understanding but poor at minimizing forgetting.
• Pushes us to value learner ratings as predictive of learning and on-the-job performance.
![Page 14: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/14.jpg)
“Historically, organizations and training researchers have relied on Kirkpatrick’s [4-Level] hierarchy as a framework for evaluating training programs…
[Unfortunately,] The Kirkpatrick framework has a number of theoretical and practical shortcomings.
[It] is antithetical to nearly 40 years of research on human learning, leads to a checklist approach to evaluation (e.g., ‘we are measuring Levels 1 and 2, so we need to measure Level 3’), and, by ignoring the actual purpose for evaluation, risks providing no information of value to stakeholders… (p. 91)
http://is.gd/TrainingResearch2012
# 4
Most Smile Sheets Stink!
![Page 15: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/15.jpg)
Level 1to
Level 3
r=.16
Alliger, Tannenbaum, Bennett, Traver, & Shotland (1997). A meta-analysis of the relations among training criteria.
Personnel Psychology, 50, 341-357.
Very Weak Relationship between Levels
Correlation between levels?
Level 1to
Level 2
r=.09
Sitzmann, T., Brown, K. G., Casper, W. J., Ely, K., & Zimmerman, R. D. (2008). A review and meta-analysis of the nomological network of trainee reactions. Journal of Applied
Psychology, 93, 280-295.
Correlation between levels?
No
Practical Significance
Weak Relationship is below .30 and .09 is VERY WEAK
So…SMILE SHEETS tell us VERY LITTLE about Learning
Level 1to
Level 2
r=.09
![Page 16: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/16.jpg)
# 5
Likert-Like Scales &Numeric Responses
create Poor Data
Likert-like Scales provide Poor DataA. Strongly AgreeB. AgreeC. Neither Agree Nor DisagreeD. DisagreeE. Strongly Disagree
54321
4.1
Compared to:
• Previous• Standard• Others
Sharon Shrock and Bill Coscarelli, authors of the classic text, now in its third edition, Criterion‐Referenced Test Development, offer the following wisdom:
On using Likert‐type Descriptive Scales (of the kind that uses response words such as “Agree,” “Strongly Agree,” etc.):
“…the resulting scale is deficient in that the [response words] are open to many interpretations.” (p. 188)
![Page 17: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/17.jpg)
We’d like to trust our learners…
Research shows that learnersdon’t always know their own learning…
Learners are Overly Optimistic Zechmeister & Shaughnessy (1980).
Learners can’t always OvercomeFaulty Prior KnowledgeKendeou & van den Broek (2005).
Learners Fail to Properly Use ExamplesRenkl (1997).
Learners Fail to Give ThemselvesRetrieval PracticeKarpicke, Butler, & Roediger (2009).
Two Recent Reviews Emphasize Learners’ Lack of Knowledge of LearningBrown, Roediger & McDaniel (2014); Kirschner & van Merriënboer (2013).
![Page 18: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/18.jpg)
Transmogrify
Strongly AgreeAgree
Neither Agree Nor DisagreeDisagree
Strongly Disagree
We Start with Fuzzy
Adjectives
54321
Magically We Turn Adjectives
Into Numbers
We Average
Responses, Losing
More Info
3.9
4.2
3.7
4.0
4.1
4.0
4.2
3.4
We Choose One
Question and Report Results
4.1
Garbage In -- Garbage Out!
When Our Smile Sheet Questions areBiased, Irrelevant, Unanswerable
![Page 19: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/19.jpg)
Garbage In -- Garbage Out!
# 6
The Critical Importanceof Remembering!
![Page 20: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/20.jpg)
LearningIntervention
PerformanceSituation
LearningOutcomes
IndividualResults
OrganizationalResults
The Learning Landscape
On-the-Job Performance
Remembering
Learning
© Copyright 2009-2016 Work-Learning Research, Inc., www.work-learning.com
0
20
40
60
80
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
After LearningDuring Learning
Learning and Forgetting Curves
Learners
Learners
If our learners start here.
But end up here.
Have we maximized the
learning benefits?
Less Remembering
More Remembering
© Copyright 2012-2016 by Work-Learning Research, Inc. (www.work-learning.com)
![Page 21: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/21.jpg)
0
20
40
60
80
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
After LearningDuring Learning
On-the-JobLearning Curves
On-the-JobForgetting
Curves
LearningCurve
Learning and Forgetting Curves
There are many possible after-training results,
depending on:
(1) Design of the Learning(2) After-Learning Follow-up
© Copyright 2012-2016 by Work-Learning Research, Inc. (www.work-learning.com)
YouTube: http://is.gd/LearningForgettingCurves
0
20
40
60
80
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
After LearningDuring Learning
On-the-JobLearning Curves
On-the-JobForgetting
Curves
LearningCurve
Learning and Forgetting Curves
© Copyright 2012-2016 by Work-Learning Research, Inc. (www.work-learning.com)
What does an end-of-course
assessment tell us?
![Page 22: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/22.jpg)
0
20
40
60
80
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
After LearningDuring Learning
Learning and Forgetting Curves
© Copyright 2012-2016 by Work-Learning Research, Inc. (www.work-learning.com)
If we measure
here?BiasedMetric!!
Understanding
Remembering
Application
DelayedTests
# 7
We must go beyondSmile Sheets!
![Page 23: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/23.jpg)
Smile Sheets should be ONLY ONE PARTof our learning evaluation efforts
Smile Sheets
Understanding
Remembering
Motivation to Apply
After Supports
Meeting Target Goals?
Job Performance
Organizational Results
Learner Expectations
Other Expectations
Supports:
Management Support?
Workplace Obstacles?
Reinforcement?
Reminders?
Learners able to:
Understand?
Remember?
Make Decisions?
Apply the Learning?
# 8
Smile Sheetscan be Improved!
![Page 24: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/24.jpg)
My Journey in tryingTo create a better
Smile Sheet
My “New” Smile Sheet
![Page 25: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/25.jpg)
Specific Concepts
How much Value?
ConceptNew?
HowWell
Taught?
Simple Overall Ratings
Helping Learners Calibrate
![Page 26: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/26.jpg)
Best Feedback Comes from Comments
# 9
Performance-FocusedSmile Sheets
![Page 27: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/27.jpg)
http://is.gd/TrainingResearch2012
http://is.gd/ddResearch
Ultimate Goal
MAXIMALLY EFFECTIVE
SMILE SHEET
Primary Goals Secondary Goals Tertiary Goals
LEARNINGEFFECTIVE?
Will the learning be effective in supporting
on-the-job performance?
RESULTS ACTIONABLE?
Will the Smile-Sheet results communicate
with clarity and urgency to guide
action?
LearnersUnderstand?
LearnersRemember?
LearnersMotivated to Apply?
After-Training Supports in Place?
Learners Smile Sheet Decisions
Accurate? Avoiding GIGO?
Do Smile Sheet Results Distinguish between Different
Levels of Success?
Are We Measuring the Things that
Matter?
Are We Using the Smile Sheet
Opportunity to Educate Our
Stakeholders?
LearnersEngaged
Cognitive Supports Effective
Realistic Retrieval
SpacingSit-Action Triggers
Belief in Value of Concepts
Self-Efficacy in Skill Area
InoculatedJob Aids
Supervisors Follow-up
Do learners remember enough to answer the questions, are the questions focused on most
important factors, are the answers calibrated to provide granularity, are leading questions avoided, do questions avoid areas of bias?
From the information, can we determine whether a course needs to be maintained, improved, or removed? Are we avoiding
numeric averages that discourage a standards-based decision on success and failure?
While smile sheets are not capable on their own to determine effectiveness, we should at
least try to examine the four goals, (1) understanding, (2) remembering, (3)
motivation to apply, (4) after-training support.
We should use smile sheets to send stealth messages to our stakeholders, including senior decision makers, instructors,
instructional designers.
Three Key Goals:
1. Are the Questions Well-Designed?
2. Are Learners Making Good Smile Sheet Decisions?
3. Are The Data we’re Getting Clear and Actionable?
Ensuring that we are focused on the science-
of-learning factors that matter!
![Page 28: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/28.jpg)
Ultimate Goal
MAXIMALLY EFFECTIVE
SMILE SHEET
Primary Goals Secondary Goals Tertiary Goals
LEARNING EFFECTIVE?
Will the learning be effective in supporting
on-the-job performance?
RESULTS ACTIONABLE?
Will the Smile-Sheet results communicate
with clarity and urgency to guide
action?
LearnersUnderstand?
LearnersRemember?
LearnersMotivated to Apply?
After-Training Supports in Place?
Learners Smile Sheet Decisions
Accurate? Avoiding GIGO?
Do Smile Sheet Results Distinguish between Different
Levels of Success?
Are We Measuring the Things that
Matter?
Are We Using the Smile Sheet
Opportunity to Educate Our
Stakeholders?
LearnersEngaged
Cognitive Supports Effective
Realistic Retrieval
SpacingSit-Action Triggers
Belief in Value of Concepts
Self-Efficacy in Skill Area
InoculatedJob Aids
Supervisors Follow-up
Do learners remember enough to answer the questions, are the questions focused on most
important factors, are the answers calibrated to provide granularity, are leading questions avoided, do questions avoid areas of bias?
From the information, can we determine whether a course needs to be maintained, improved, or removed? Are we avoiding
numeric averages that discourage a standards-based decision on success and failure?
While smile sheets are not capable on their own to determine effectiveness, we should at
least try to examine the four goals, (1) understanding, (2) remembering, (3)
motivation to apply, (4) after-training support.
We should use smile sheets to send stealth messages to our stakeholders, including senior decision makers, instructors,
instructional designers.
What Most Smile Sheets
Provide
QUATERNARYGOALS
Traditional Smile Sheets
Learners rate instructors as credible and engaging?
Learners say that classroom environment was conducive to
learning?
Learners satisfied with experience?
Learners think course was well-organized?
Ultimate Goal Primary Goals Secondary Goals Tertiary Goals
In regard to the course topics taught, HOW ABLE ARE YOU to put what you’ve learned into practice on the job?
A. I’m NOT AT ALL ABLE to put the concepts into practice.
B. I have GENERAL AWARENESS of the concepts taught, but I will need more training/practice/guidance/experience TO DO ACTUAL JOB TASKS using the concepts taught.
C. I am ABLE TO WORK ON ACTUAL JOB TASKS, but I’LL NEED MORE HANDS-ON EXPERIENCE to be fully competent in using the concepts taught.
D. I am ABLE TO PERFORM ACTUAL JOB TASKS at a FULLY-COMPETENT LEVEL in using the concepts taught.
E. I am ABLE TO PERFORM ACTUAL JOB TASKS at an EXPERT LEVEL in using the concepts taught.
A Better Smile Sheet Question
![Page 29: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/29.jpg)
Ultimate Goal Primary Goals Secondary Goals Tertiary Goals
HOW ABLE ARE YOU to put what you’ve learned into practice on the job? Choose One.
A. I am NOT AT ALL ready to use the skills taught.
B. I need MORE GUIDANCE to be GOOD at using these skills.
C. I need MORE EXPERIENCE to be GOOD at using these skills.
D. I am FULLY COMPETENT in using these skills.
E. I am CAPABLE at an EXPERT LEVEL in using these skills.
A Better Smile Sheet Question -- Shorter
![Page 30: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/30.jpg)
© Copyright 2009 Work-Learning Research, Inc.
Now that you’ve taken the course, how well do you feel you understand the concepts taught in the course?
A. I have some significant CONFUSIONS AND/OR BLIND SPOTS.
B. I have a BASIC FAMILIARITY with the concepts.
C. I have a SOLID UNDERSTANDING of the concepts.
D. I have a COMPREHENSIVE UNDERSTANDING of the concepts.
E. I have an EXPERT‐LEVEL UNDERSTANDING of the concepts.
Standards
Unacceptable
Unlikely
?
Acceptable
?
AcceptableUnacceptable
AcceptableSuperior
Traditional Smile Sheet
Design Learning
Design Smile Sheet
Deploy‐‐‐‐‐
Collect Data
Examine Results
Determine Standards
Performance‐Focused Smile
Sheet
Design Learning
Design Smile Sheet
Deploy‐‐‐‐‐
Collect Data
Examine Results
Determine Standards
OPEN TO
BIAS
![Page 31: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/31.jpg)
Ultimate Goal Primary Goals Secondary Goals Tertiary Goals
In regard to the concepts taught in the course, how motivated WILL YOU BE to UTILIZE these skills in your work?
A. I will NOT MAKE THIS A PRIORITY when I get back to my day-to-day job.
B. I will make this a PRIORITY—BUT A LOW PRIORITY when I get back to my day-to-day job.
C. I will make this a MODERATE PRIORITY when I get back to my day-to-day job.
D. I will make this a HIGH PRIORITY when I get back to my day-to-day job.
E. I will make this one of my HIGHEST PRIORITIES when I get back to my day-to-day job.
A Question About Motivation…
Ultimate Goal Primary Goals Secondary Goals Tertiary Goals
After the course, when you begin to apply your new knowledge at your worksite, which of the following supports are likely to be in place for you? Select as many items as are likely to be true.
A. I will have ENOUGH TIME to work on applying the learning.
B. I will have my PROGRESS MONITORED BY MY SUPERVISOR in applying the learning.
C. I will have someone available TO COACH OR MENTOR ME in applying the learning.
D. I will have easy access to a COURSE INSTRUCTOR to contact for guidance and support.
E. I will have JOB AIDS to guide me in applying the learning to real job tasks.
F. I will be PERIODICALLY REMINDED of key learning concepts/skills over the next few months.
G. I will NOT get much direct support, but will rely on my own initiative.
A Question About Follow-Through…
![Page 32: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/32.jpg)
# 10
Delayed Smile Sheets
Delayed Smile Sheet
Have you used what you learned in the workshop to make a significant improvement in your work?
• No, and I doubt that I will use what I learned.
• No, but I probably will use what I learned.
• No, but I have a plan to use what I learned.
• Yes, I have already used what I learned.
![Page 33: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/33.jpg)
Are You Using What You Learned?
Yes!!
What Enabled You to Apply
What You Learned?
No
What is Holding You Back?
Feedback to:• Management• Learners• Us
Delayed Smile Sheet – Feedback for Improvement
What is holding you back from teaching others what you have learned?
• The learning content is not relevant to my colleagues’ work.
• I don’t think the learning content is valid.
• I don’t see how teaching others will benefit me.
• I can’t remember the learning content well enough.
• I’m not very good at teaching others.
• I haven’t had the time.
• I have had higher priorities.
• I have not had the resources.
• The risk of teaching this to others is too high.
• I’m still working to persuade management of the value.
• Other, please specify ____________________________
![Page 34: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/34.jpg)
Delayed Smile Sheet
―――Feedback
toManagement,
Learners,and/or Us.
What are the top 3 to 4 reasons that have enabled you to put the workshop information into practice?
• I remember (or can look up) the workshop’s learning points.
• I believe strongly in the workshop’s learning points.
• I have the authority to make the necessary changes.
• I have decided to lead a change effort to make this happen.
• I have the time to put this into practice.
• I have the resources to put this into practice.
• I will be acknowledged or rewarded if I do this.
• I will be sanctioned or punished if I do NOT do this.
• My management has made this a high priority.
• My management is fully supportive of my efforts.
• My coworkers are fully supportive of my efforts.
• My Team (or our whole unit) is working together on this.
50%
50%75%
50%
Message toManagement
“More management-
driven than self-driven.”
This learning event will make a significant contribution to my work, in terms of the following: SELECT AS MANY AS YOU LIKE!
A. Improving my personal productivity.
B. Increasing my ability to innovate.
C. Enabling me to collaborate more effectively.
D. Improving my management performance.
E. Enabling me to generate more revenue.
F. Enabling me to lower costs.
![Page 35: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/35.jpg)
# 11
Stealth Messages
Workplace Learning‐and‐Performance
Professionals
![Page 36: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/36.jpg)
1. Training is not enough.
2. Training for awareness is a weak and insufficient approach.
3. Training should specifically minimize forgetting and support remembering.
4. It is better to train a few things really well, than many things perfunctorily.
5. Training should motivate subsequent on‐the‐job application.
6. Training content must be correct, validated, and relevant.
7. Traditional smile sheets are inadequate.
8. We must measure to provide ourselves with feedback to spur continuous improvement.
9. Prompting mechanisms (like job aids and performance support) should be used—as appropriate—as an integral part of training and as a replacement for training.
10. On‐the‐job learning should be leveraged in addition to formal training.
11. Learners’ managers and others within the learners’ chain of command should support after‐training application.
12. Supervisors play a critical role in enabling the success of training and in helping people be creative on the job.
Some Stealth MessagesWe Might Want to Send
conscious conscious
filters
ProactiveReactive
![Page 37: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/37.jpg)
Etcetera
There are
Leverage Pointswithin our SOP’s
from where we can send
Stealth Messages
Recruiting of Work‐Learning Professionals
Formal Course Evaluations
TrainingRequests
(our response)
Instructor Evaluations
On‐the‐Job Performance
Data
Workplace Performance Assistance
Learning Needs Analysis
Annual Reports to Management
Course Reviews
Does the course use
methods that support
remembering?
Does the instructor use methods that support after‐
training follow‐through
?
Do we determine whether this is a training issue or a workplace
issue?
Do we ask trainers to show their consulting skills, not just their
platform skills?
Do we utilize Performance‐Focused smile
sheets & delayed smile
sheets?
What messages does a
traditional smile sheet
send?
What messages does a delayed performance‐focused smile sheet send?
What messages does a
performance‐focused smile sheet send?
![Page 38: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/38.jpg)
Summary
1. Most smile sheets provide misleading feedback.
2. Smile sheets can be significantly improved.
3. Smile Sheets should NOT be our sole source of feedback.
4. Learners don’t always know their own learning.
5. Likert‐like scales and numeric scales create a garbage‐in garbage‐out problem.
6. We should set smile‐sheet standards before we get smile‐sheet results.
7. Delayed smile sheets offer potent information.
8. We are getting very poor feedback on the success of our learning interventions.
9. We should utilize the science of learning in our learning metrics.
10. To educate our stakeholders, we should consider using stealth messaging.
11. Performance‐Focused Smile Sheets align with science‐of‐learning findings and enable better decision‐making and reporting that enables urgency and clarity.
12. The Kirkpatrick Model is incomplete and sends some wrong messages about learning measurement.
Today’s Main Points:
![Page 39: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/39.jpg)
Quite simply, the BEST book on smile sheet creation and utilization, Period!
Karl M. KappProfessor of Instructional Technology
Bloomsburg University
Thoughtful and sensible advice for feedback tools that will provide valid and actionable data.
Robert O. BrinkerhoffProfessor Emeritus, Western Michigan
University & Director, Brinkerhoff Evaluation Institute
Evidence‐based practice at the master level.
Julie DirksenAuthor of Design For How People Learn
www.SmileSheets.comuse Code: NGO16
• New Smile‐Sheet Questions – https://is.gd/NewSmiles1
• Avoid the Net Promoter Score — http://is.gd/NetPromoter
• Avoid Benchmarking — http://is.gd/DoNotBenchmark
• eLearning Manifesto 22 Principles — http://is.gd/manifesto22
• ATD Article on Smile Sheets — http://is.gd/atdsmile
• Subscription Learning — http://SubscriptionLearning.com
• Neuroscience & Learning: Not Yet! — http://is.gd/brainlearning
Subscribe to my Newsletter: — http://Work‐Learning.com/sign‐up.html
![Page 40: The Blueprint for Radically Improved Training Evaluationswillthalheimer.typepad.com/files/thalheimer----insidengo----smile... · limit damage from negative back‐channel ... good](https://reader031.vdocuments.mx/reader031/viewer/2022030421/5aa8f5067f8b9a9a188c3c49/html5/thumbnails/40.jpg)
Slides available at: www.is.gd/willstuff
Phone: 888‐579‐9814Email: info@work‐learning.comWebsite: Work‐Learning.comAudits: LearningAudit.netSmiles: SmileSheets.comBlog: WillAtWorkLearning.comTwitter: @WillWorkLearn
Will Thalheimer, PhDPresidentWork‐Learning Research, Inc.Somerville, Massachusetts, US
July 2016
The Blueprint for Radically Improved Training Evaluations
www.SmileSheets.comuse Code: NGO16
Slides available at: www.is.gd/willstuff
Thank you!!