chapter six training evaluation - u.camdemyu.camdemy.com/sysdata/doc/3/3e87ab1c532751ca/pdf.pdf ·...

Chapter Six

Training Evaluation

Objectives O Explain why evaluation is important

O Identify and choose outcomes to evaluate a

training program

O Discuss the process used to plan and

implement a good training evaluation

O Discuss the strengths and weaknesses of

different evaluation designs

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.

Objectives O Choose the appropriate evaluation design

based on the characteristics of the company

and the importance and purpose of the

training

O Conduct a cost-benefit analysis for a

training program

O Explain the role of workforce analytics and

dashboards in determining the value of

training practices


Key Terms O Training effectiveness: benefits derived

training

O Training evaluation: process of determining

training effectiveness

O Training outcomes: measures to evaluate

training effectiveness

O Evaluation design: data will be collected

and which data will be collected for

training evaluation


Formative Evaluation

O Refers to evaluation of training that takes

place during program design

O Helps to ensure training is well organized,

runs smoothly, and that trainees learn and

are satisfied

O Provides information on how to make a

program better


Summative Evaluation

O Refers to evaluation conducted to

determine if training has lead to desirable

outcomes

O Examines whether trainees have improved

or acquired knowledge, skills, attitudes,

behaviors, or other outcomes

O Includes examining the business impact of

training


Importance of Evaluation

O There are multiple reasons to evaluate:

o identify a program’s strengths and

weaknesses

o assess what features of training content

and context matter

o identify which trainees benefited

o gather information for marketing

training

o determine financial benefits and costs


The Evaluation Process


Training Outcomes


Reactions

Learning &

Cognitive

Outcomes

Behavior &

Skill-Based

Outcomes

Affective

Outcomes Results

Return on

Investment

Reactions

O Trainees’ perceptions of the training

experience relating to the content,

facilities, trainer, and methods

O Key questions to consider include:

o Did the trainees like the program?

o Did the environment help learning?

o Was the material meaningful?

O Typically measured at the end of training


Learning & Cognitive

Outcomes O Relate to familiarity with information,

including principles, facts, techniques,

procedures, and processes

O Typically measured via paper-and-pencil

tests and self-assessments

O Tests often preferred over self-assessments


Behavior & Skill-Based

Outcomes O Relate to proficiency with technical skills,

motor skills, and behavior

O Include learning and transfer

O Learning often assessed via work samples

O Transfer often assessed via observation or

managerial/peer ratings


Affective Outcomes

O Include attitudes and motivation

o e.g., self-efficacy, employee engagement,

motivation to learn, tolerance for

diversity, attitudes toward safety

O The attitude of interest depends on the

training objectives

O Affective outcomes often measured via

surveys


Results

O Did training have an impact on meaningful

business outcomes?

O Outcomes used to determine the benefits

of training to the company

o e.g., reduced costs, increased employee

retention, increased sales, improved

quality or customer service


Return on Investment

O Involves comparing the training program's

benefits to its costs in monetary terms

o Benefits: value the company gains

o Direct costs: salaries for employees

involved in training, program materials,

facilities, and travel

o Indirect costs: costs not related directly

to design and delivery


Which Outcomes?


Relevance Reliability

Discrimination Practicality

Relevance O The extent to which training outcomes are

related to the learned capabilities

emphasized in the program

O Contamination refers to the inclusion of

inappropriate or irrelevant outcomes

O Deficiency refers to the omission of

important information


Deficiency, Relevance, &

Contamination


Reliability

O The extent to which outcomes can be

measured consistently over time

O Evaluators are concerned with consistency

over time, such that items do not change

in meaning or interpretation over time


Discrimination

O The extent to which measured

performance reflects a true difference

O We want tests that discriminate between

high and low performance

o A test that is too easy may not

discriminate

o In this instance, both high and low

performers would do well and appear

“good” even though they are not


Practicality

O The extent to which outcomes can be

easily measured and collected

O Companies often claim that measurement

is too burdensome


Training Evaluation Practices


Which Outcomes?

O While collecting data on all outcomes is

ideal, it may not always be necessary

O Consider the scope of the training and

practical considerations

O Do not assume that positive reactions lead

to transfer


Threats to Validity

O Threats to validity refer to factors that will

lead an evaluator to question the results

O Internal validity is the believability of the

study

O External validity is the generalizability of

the evaluation results to other groups and

situations


Controlling for Threats

O There are three ways to control for threats:

o Use pre-tests and post-tests

o Use a control group

o Random assignment of employees to

control and training groups


Post-test Only

O Involves collecting only post-training

outcomes

O Appropriate when trainees can be

expected to have similar levels of

proficiency prior to training

O Strengthened with a control group


Pre-test / Post-test

O Involves collecting both pre-training and

post-training outcomes to determine if a

change has occurred

O Is there a difference in outcomes before

and after training?


Pre-test / Post-test with

Comparison Group

O Includes pre-training and post-training

outcomes and use of a control group

O If the post-training improvement is greater

for the training group, there is evidence

that training was responsible


Time Series

O Involves collecting measures at periodic

intervals pre- and post-training

O A comparison group may be used

O The strength of this design can be

improved by using reversal, which refers to

a time period when participants no longer

receive training


Solomon Four-Group

O Combines the pre-test/post-test

comparison group design and the

posttest-only control group design

o Pre-test, treatment, post-test

o Pre-test, no treatment, post-test

o No pre-test, treatment, post-test

o No pre-test, no treatment, post-test


When Evaluation May Not

Be Necessary O Time constraints

O Managers and trainees may lack expertise

O The company may view training as an

investment from which it expects little or

no return


When Evaluation is a Must

O The training is ongoing and has the

potential to affect many employees

O The training program involves multiple

classes and a large number of trainees

O The expertise exists to evaluate

O The cost of training is significant

O There is sufficient time and interest


Cost-Benefit Analysis

O Companies may desire to quantify whether

the benefits of training outweigh the costs

O Cost-benefit analysis

o Process of determining the economic

impact of training using accounting

methods that look at training costs and

benefits


Cost Categories 1. program development or purchase

2. instructional materials

3. equipment and hardware

4. facilities

5. travel and lodging

6. salary of the trainer and support staff

7. cost of lost productivity or replacement

workers while trainees are away


Determining Benefits

O A number of methods can be used to

identify benefits

o literature that summarizes benefits

o pilot training programs

o observing successful job performers

asking trainees and managers for

estimates


Calculating Return on

Investment (ROI)

O Identify annual change in outcomes

O Place a monetary value on the outcomes

O Determine the annual change in value

O Determine training costs

O Calculate net benefit

O Divide net benefits by costs


Utility Analysis

O Assessing the dollar value of training

based on: o estimates of the difference in

performance between trained and untrained employees

o number of individuals trained o length of time training is expected to

influence performance o variability in performance in the

untrained group of employees


Success Cases & Return on

Expectations

O Success cases refer to concrete examples showing how learning has led to results the company finds worthwhile and credible

O Return on expectations (ROE) demonstrates to key stakeholders that their expectations about training have been satisfied


Big Data O Big data refer to complex datasets

compiled across different systems,

including marketing, sales, HR, finance,

accounting, customer service, and

operations

O Three dimensions characterize big data

o volume

o variety

o velocity


Big Data & Training O Big data help make decisions about human

capital based on data, rather than intuition and conventional wisdom

O Big data can be used to: o evaluate the effectiveness of programs o determine their impact on business

results o develop predictive models for

forecasting training needs, course enrollments, and outcomes


chapter six training evaluation - u.camdemyu.camdemy.com/sysdata/doc/3/3e87ab1c532751ca/pdf.pdf ·...

Documents