improved learner-feedback questions - work-learning research€¦ · web viewtraditionally, smile...

Performance-Focused Learner-Survey QuestionsUpdates and Improvements

August 2019

Will Thalheimer, PhD

© Copyright 2019 by Will Thalheimer and Work-Learning Research, Inc.

All rights reserved except as follows. You may use and modify the questions recommended herein for use with learners who are employees or contractors within your organization—or for your own personal learners if you are a trainer, teacher, professor. You may NOT use these for any other purpose. You may NOT use these questions or derivatives for other organizations. You may NOT receive revenues or compensation for these questions.

This version revised September 23, 2019 to fix minor typos.

We are happy to negotiate their use, as we want learners everywhere to have a chance to provide more meaningful feedback.

We are even happier to help you—for a reasonable fee—in writing learner surveys, reviewing your draft questions, writing questions for you, speaking at your events, and leading workshops. Please reach out to Will Thalheimer to schedule an appointment or ask questions.

https://www.worklearning.com/contact/

Will Thalheimer, PhD

Work-Learning Research, Inc.

2


Somerville, Massachusetts, United States

617-718-0767

Improved Learner-Feedback Questions

Since the 2016 publication of my book, Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form, I’ve been working with organizations of many types to develop specific learner-feedback questions that enable the collection of data on learning effectiveness.

Smile sheets can be used for three main purposes: (1) to get data about learner satisfaction and the reputation of a learning intervention, (2) to get data about learning effectiveness to enable subsequent improvement in the learning, or (3) to nudge learners to take action toward learning goals.

Most of my work in this area has focused on performance-focused learner feedback—questions designed for the second purpose above, helping organizations get better feedback about the effectiveness of their learning interventions.

In working with clients over the last several years, I’ve learned a ton about what works and what doesn’t in question design. For example, I’ve learned that tailoring questions to a specific learning program is very valuable because it adds clarity and motivates learner attention.

Improving our questions is critical for two reasons. When our wording is better, we get better data. When our wording is more succinct, we get more motivated respondents. We need both.

The two most important things that I’ve learned are (1) that the process of writing questions is improved when we use thoughtful iterations utilizing multiple stakeholders and (2) that we, as question writers, must maintain humility and be aggressive in working to create continuous improvement.

In this document, I offer improvements on some of the most important smile-sheet questions that can be asked. These questions are written broadly, so they can be useful for a wide range of learning programs. They can, of course, be rewritten to be tailored to a specific learning program or a specific organization.

I recommend that you tailor your questions! I am available to help you do that or to give you feedback on your draft versions! You can see my price list here: https://www.worklearning.com/services-and-price-list/ and please do reach out to contact me to have a brief 30-minute conversation about how I can help: https://www.worklearning.com/contact/.

3


https://www.worklearning.com/services-and-price-list/

If you don’t have a budget, please find someone in your organization who has expertise on survey design and question wording, preferably not someone who rigidly adheres to problematic Likert-like or numeric scales!

4

The Rationale for Performance-Focused Question Designs

There are dozens of constructs we might target in questions we ask our learners. The constructs we choose will determine the information we capture and the conclusions we might draw from the data. Traditionally, smile sheet questions have focused on the reputation of the course and the satisfaction of the learners. Unfortunately, as research has shown, this focus pushes us to create smile sheets that are completely inadequate in determining—or even in hinting at—the effectiveness of our learning interventions.

To help guide smile-sheet design, we can use the Four Pillars of Training Effectiveness. Based on science-of-learning research, the following factors are critical to learning.

A. Do the learners UNDERSTAND the concepts targeted for comprehension?B. Will they REMEMBER what they’ve learned?C. Will they be MOTIVATED TO APPLY what they’ve learned?D. Are there AFTER-LEARNING SUPPORTS in place?

Measuring these constructs is critical to ensure that we are focusing on learning effectiveness. Of course, other constructs may be of importance as well, but at a minimum it is desirable to measure these four constructs in some manner.

What are the most critical questions we might ask then? Answering this question is difficult because each learning intervention differs along many parameters; including learners, content, technology used, time allocated, instructional goals, and stakeholder perspectives. Also, smile sheets are better seen as tools for improvement rather than as a pro-forma regimen to be delivered regardless of our strategic focus. For example, as performance-improvement tools, we might want to rotate questions to enable a coordinated organizational-wide focus on improving targeted learning factors. For example, this year we might focus on improving the realistic practice we provide to our learners. Next year, we might focus on after-learning follow-through, or spacing, or motivation to apply.

I get asked all the time, “Which questions must we include on our smile sheets?” While I strive to communicate the complexities in answering this question, I feel a responsibility to provide at least some guidance. This document contains a full list of questions that are critical. These questions fulfill the parameters of the Four Pillars of Training Effectiveness and/or have been expressed as requirements for many organizations with whom I work. So, for example, many organizations want a question focused specifically on instructor performance, so one is included here.

My advice is to utilize these questions as a starting point, perhaps hiring an outside assessment expert to ensure you’re asking questions in a manner that target what you want them to target. Getting outside help is extremely useful. Indeed, I’m one of the

5

best in the business, yet still, my questions improve tremendously by getting feedback from others and by using an iterative process.

The questions in this document represent the culmination of a long line of improvements that rely on hundreds of helpful comments from learning-and-development professionals and numerous data points from real learners engaged in real workplace learning.

Each question is presented in a tabular format to support clarity and to make the questions and answer choices easy to copy into your survey tool.

I have included a short commentary after each question—to provide context, a rationale for the question’s purpose, and specific recommendations for how to use the question.

As always, if you have suggestions for improvement, I would love to hear them!

Also, I would love to help you write questions or review your question drafts. You will be amazed at the improvements that can be made!

Go to this webpage to contact me or set up a meeting directly from my calendar to get started right now!

Contact me (Will Thalheimer, PhD)to ask a question or set up a meeting:


Have I hinted enough that I’d love to work with you and your organization?

6


BIG SMILE

7

Question1

Question on Learner Comprehension(not shown to learners)

Proposed Standards(not shown to learners)

Now that you’ve completed the learning experience, how well do you feel you understand the concepts taught? CHOOSE ONE.

A I am still at least SOMEWHAT CONFUSED about the concepts.

Alarming

B I am now SOMEWHAT FAMILIAR WITH the concepts. Unacceptable

C I have a SOLID UNDERSTANDING of the concepts. Acceptable

D I AM FULLY READY TO USE the concepts in my work. Superior

E I have an EXPERT-LEVEL ABILITY to use the concepts. Superior/Overconfident?

GENERAL NOTE: Proposed standards are ONLY a starting point. Ideally, you will negotiate standards with your key stakeholders (including and especially your learning team)—and pilot test your questions and adjust standards if evidence suggests that your standards are inappropriate. Beware, of course, of lowering your standards just because your current learning interventions are not meeting those standards. It’s perfectly fine, at first, to fail to meet the standards set. The goal is for improvement, so it’s not a disaster to fall below your standards in your initial efforts

NOTES ON THIS QUESTION: This question focuses on learners’ ability to understand what was taught. If they have confusions or learn just enough to be familiar with the concepts taught, that is not enough to enable work performance. Note that Answer C is deemed “Acceptable,” even though we might hope for more than a solid understanding. Answer D is clearly better as it implies that people can use what they’ve learned in their work. We might expect Answer D is the minimum acceptable answer, but since this question is focused on comprehension, developing a solid understanding can be deemed acceptable. Other questions will be aimed specifically at a person’s ability to use what they’ve learned.

8

Question2

Question on Learner Motivation to Apply

(not shown to learners)Proposed Standards

(not shown to learners)

In regard to the topics taught, how motivated WILL YOU BE TO USE these concepts/skills in your work? CHOOSE ONE.

A My CURRENT ROLE DOES NOT ENABLE me to use what I learned.

Alarming

B I will NOT MAKE THIS A PRIORITY when I get back to my day-to-day job.

Unacceptable

C I will make this a PRIORITY—BUT A LOW PRIORITY when I get back to my day-to-day job.

Unacceptable

D I will make this a MODERATE PRIORITY when I get back to my day-to-day job.

Unacceptable

E I will make this a HIGH PRIORITY when I get back to my day-to-day job.

Acceptable

F I will make this ONE OF MY HIGHEST PRIORITIES when I get back to my day-to-day job.

Superior

NOTES: This question focuses on learners’ motivation to apply what they’ve learned. The research on training transfer is very clear that such a motivation is highly predictive of actual success in transfer. Here, my recommendation for acceptability only gives credit for the top two answers on the assumption that if people are not highly motivated, they won’t actually do the work needed to apply the learning.

TAILORING: Again, all these questions are written in a generic format, but I highly encourage you to tailor them to the specifics of the situation. So, in this question, you could go beyond the generic version by asking something like, “How motivated will you be to use the PowerPoint skills you just learned when you create your next presentation?” Your answer choices would have to change as well, and they could get much more specific too.

9

Question3

Question on On-the-Job Performance



HOW ABLE ARE YOU to put what you’ve learned into practice in your work? CHOOSE THE ONE OPTION that best describes your current readiness.

A My CURRENT ROLE DOES NOT ENABLE me to use what I learned.

Alarming

B I AM STILL UNCLEAR about what to do, and/or why to do it.

Alarming

C I NEED MORE GUIDANCE before I know how to use what I learned.

Unacceptable

D I NEED MORE EXPERIENCE to be good at using what I learned.

Acceptable

E I CAN BE SUCCESSFUL NOW in using what I learned (even without more guidance or experience).

Superior

F I CAN PERFORM NOW AT AN EXPERT LEVEL in using what I learned.

Superior/Overconfident?

NOTE: This question is a descendant from “The World’s Best Smile Sheet Question,” a question that I once thought was great, but now I think is poor. From that original draft, I’ve improved it a half-dozen times. I point this out to emphasize that from the feedback I’ve received and the experience I’ve gained, I’ve learned that questions can always be improved—and so I continue working to make improvements. The questions here in this document are better than the previous ones, but perhaps these too will bear improvements in the future. I offer this as a lesson to you. Be open to improvements! And get feedback on your questions!

This question focuses on learners’ perception of their ability to put what they’ve learned into practice in their work. I deem it “Unacceptable” if learners need more guidance before they know how to use what they learned under the belief that most training or learning interventions should provide such guidance.

10

Question4

Question on Realistic Practice(as a proxy for “Support for Remembering”)



In which of the following activities did you spend the most time during the learning? SELECT UP TO THREE CHOICES ONLY (1, 2, 3 choices please)!

AVIEWING INFORMATION presented on a screen (example: from PowerPoint)

Acceptable

BREFLECTING ON HOW I MIGHT USE the ideas presented.

Acceptable

CEngaging in DISCUSSIONS ON HOW TO USE the ideas presented.

Acceptable

DAnswering QUIZ-LIKE QUESTIONS on the ideas presented.

Acceptable

E MAKING DECISIONS like those I will face on the job. Superior

F DOING TASKS OR ACTIVITIES like those I will face on the job.

Superior

G Engaging in ACTIVITIES NOT RELEVANT to my job. Alarming

HEngaging in RELEVANT ACTIVITIES not listed here, PLEASE SPECIFY BELOW:

Acceptable

NOTES: Learning should aim to support remembering. This is one of the most critical distinctions in the learning profession. Too often, our learning designs push only for learner comprehension. The best way to measure remembering is to assess learners after a delay of several days or more—preferably using realistic decision-making or task completion as our metrics (See LTEM Tiers 5 and 6, available at: https://www.worklearning.com/ltem). When gathering learner feedback, we can look for proxies for remembering, focused specifically on learning factors that have research-based backing in supporting remembering. Providing learners with realistic practice is one of the best ways to ensure remembering, along with spacing repetitions over time. This question is focused on gauging realistic retrieval practice. The two items that provide the best indication of this are focused on realistic decision making and realistic task practice, choices E and F above.

11

https://www.worklearning.com/ltem/

Question5

Question on After-Learning Support



After the course, when you begin to apply your new knowledge at your worksite, which of the following supports are likely to be in place for you? SELECT AS MANY ITEMS as are likely to be true.

AMY MANAGER WILL ACTIVELY SUPPORT ME with key supports like time, resources, advice, and/or encouragement.

Superior

BI will use A COACH OR MENTOR to guide me in applying the learning to my work.

Acceptable

CI will regularly receive support from A COURSE INSTRUCTOR to help me in applying the learning to my work.

Acceptable

DI will use JOB AIDS like checklists, search tools, or reference materials to guide me in applying the learning to my work.

Acceptable

EI will be PERIODICALLY REMINDED (for at least several weeks) of key concepts and skills that were taught.

Acceptable

F I will NOT get much direct support, but will rely on my own initiative.

Alarming

NOTES: The research on transfer is very clear that after-learning supports are critical for ensuring that learners can successfully apply what they’ve learned. This question gets at the most common types of after-learning support.

12

Question6

Question on Instructor(s) Performance



Which of the following are true about your course instructor(s)?

SELECT UP TO THREE ITEMS—THOSE THAT ARE THE MOST IMPORTANT in describing the performance of your instructor(s).

A Generally DID A GOOD JOB in facilitating the learning. Superior

BToo often HURRIED through the content in a SUPERFICIAL manner.

Unacceptable

C Demonstrated DEEP SUBJECT-MATTER KNOWLEDGE. Acceptable

D Was too OFTEN UNCLEAR or DISORGANIZED. Unacceptable

EShowed HIGH LEVELS OF REAL-WORLD EXPERIENCE relevant to the topic.

Acceptable

FWas SOCIALLY AWKWARD or INAPPROPRIATE to an extent that it harmed learning.

Unacceptable

GMotivated me to ENGAGE MORE DEEPLY IN THE LEARNING than I’d expected.

Acceptable

HGave us LITTLE OR NO TIME TO PRACTICE SKILLS we could use in our work.

Unacceptable

I Is a PERSON I CAME TO REALLY TRUST. Acceptable

J I wish he/she/they had PERFORMED BETTER in facilitating the learning.

Alarming

NOTES: Instructors should be judged NOT ONLY using this question, but also using the other questions in this document.

Note that learners tend to rate instructors at very high levels, which too often has made it difficult to distinguish between different levels of instructor performance. Even with the greater specificity in the answer choices here, differences may be subtle.

13

Question7

Question on the Survey Questions



We’re using a new type of question to get your feedback. Instead of using a scale from 1 to 5 or choices from strongly disagree to strongly agree, we’re presenting questions with more specific answer choices. This question is an example.

How do you feel about these new questions?

AThey HAVE ABOUT THE SAME LEVEL OF EFFECTIVENESS as the 1-to-5 questions or the strongly-disagree to strongly-agree questions.

Acceptable

BThey are BETTER because they GIVE ME MORE CLARITY about the choices I’m making.

Superior

CThey are WORSE because they TAKE MORE TIME to think through the answer choices.

Unacceptable

DTheir BENEFITS (BRINGING MORE CLARITY) OUTWEIGH their DOWNSIDES (TAKING MORE TIME).

Acceptable

ETheir BENEFITS (MORE CLARITY) ARE NOT WORTH the extra time required.

Unacceptable

In your own words, what do you like and dislike about the new questions?

NOTES: In any field, when we use new methods, we may encounter resistance. When you use this new form of learner-feedback question—even though they are demonstrably superior to traditional smile-sheet questions—you may encounter pockets of resistance in your organization. The biggest worry is that the learners won’t like the new questions because they use a lot of words. This question is designed specifically to enable you to collect data about what learners really think of the new format.

In our pilot tests, learners have overwhelmingly preferred the new questions over the old ones (about 80% prefer the new questions and 90% say the new questions are equal or better to traditional questions). Still, you may find a need for your own data—so you can show your stakeholders evidence from your own organization. You may only need to use this question a few times to satisfy your organization’s curiosity. This question can be withdrawn after you validate learner acceptance.

14

Question8

Question about Reputation of Learning

(an NPS replacement)(not shown to learners)


If someone asked you about the effectiveness of the learning experience, would you recommend the learning to them? CHOOSE ONE.

A The learning was TOO INEFFECTIVE to recommend. Alarming

B The learning was INEFFECTIVE ENOUGH THAT I WOULD BE HESITANT to recommend it.

Alarming

CThe learning was NOT FULLY EFFECTIVE, BUT I would recommend it IF IMPROVEMENTS WERE MADE to the learning.

Unacceptable

DThe learning was NOT FULLY EFFECTIVE, BUT I would still recommend it EVEN IF NO CHANGES WERE MADE to the learning.

Acceptable

E The learning was EFFECTIVE, SO I WOULD RECOMMEND IT.

Acceptable

F The learning was VERY EFFECTIVE, SO I WOULD HIGHLY RECOMMEND IT.

Superior

NOTES: This question provides data about learner satisfaction and the reputation of the course. It does NOT provide data about the effectiveness of the course. Still, this type of data can be quite important in our success—as we need stakeholder support to keep our jobs, gain resources, obtain support, and attract and motivate learners to engage in our learning interventions.

Note that this question was originally designed as a replacement for NPS-like questions used for training. The Net Promoter Score construct is not acceptable for training. See the problems with NPS here: https://www.worklearning.com/2013/12/12/net-promoter-score-maybe-fine-for-marketing-stupid-for-training/.

15

https://www.worklearning.com/2013/12/12/net-promoter-score-maybe-fine-for-marketing-stupid-for-training/

https://www.worklearning.com/2013/12/12/net-promoter-score-maybe-fine-for-marketing-stupid-for-training/

Question9

Question on What Learners Liked



Which aspects of the learning helped you the most in learning what was taught?

NOTES: Comment questions can provide the most valuable information, especially when they are preceded by well-designed prior questions—ones that focus learners’ attention on learning-effectiveness factors.

16

Question10

Question on What Learners Did NOT Like



What could have been done better to make this a more effective learning experience? Remember, your feedback is critical, especially in providing us with constructive ideas for improvement.

NOTES: It’s good to ask for both positive and negative comments and focus the comments toward learning effectiveness.

17

Question11

Question on Anything Else?(not shown to learners)


Is there anything else we should have asked? Anything else you want to tell us about the quality of the learning experience?

NOTES: This question may seem extraneous, but experience shows that some of the most insightful comments can be generated here. Usually, not many people answer it, but those who do often provide very insightful comments.

18

Final Summary

One of the most critical goals of learning evaluation is to get data that helps us make our most important decisions—usually focusing on learning design effectiveness, content relevance, and instructor performance.

The eleven questions proposed here are fundamental questions, but they are not the only questions that can be developed. Clients have asked me for help in developing questions targeting all manner of constructs, including the following:

Questions about whether a training is supporting organizational values. Questions about sensitivities involved in sexual-harassment training. Questions about the learning technologies in use. Questions about engagement and attention. Questions about management involvement. Questions for leadership and management training to get feedback from direct

reports (not just from the learners). Questions for self-study programs, asking about both the materials and the

experience. Questions for elearning, where many fewer questions can be used. Questions for conference sessions, where only three questions can be used. Questions for webinars or webinar-like training. Questions for full conference experiences. Questions for employee exit surveys. Questions tailored for different courses.

One of the most common misconceptions is that we have to use the same exact questions with the same exact wording for all of the courses in our organization. This is a mistake—one often made under the pretense that we should compare our courses or instructors to each other. Sometimes we can use a question on multiple courses, but comparisons are difficult. The same questions asked in a course on how to use PowerPoint and a course on sexual harassment will not produce comparable results!

Our goal is to help learners make clear decisions for every question they are answering. The more tailored and specific the wording, the more the questions will make sense to them, the more motivated they will be to attend to the questions, and the better our data will be. I highly recommend that you tailor your questions.

If you and your organization need help in crafting new learner surveys, or need a review of your question drafts, or want help writing questions, I would be delighted. You can schedule an appointment with me, Will Thalheimer, or just ask me a question at:


19


improved learner-feedback questions - work-learning research€¦ · web viewtraditionally, smile...

Documents