Evaluating Training Evaluating Training ProgramsPrograms

How can training programs be How can training programs be evaluated?evaluated?

Measures used in evaluating training Measures used in evaluating training programsprograms

Various ways of designing the Various ways of designing the evaluation proceduresevaluation procedures

Describe the measurement process Describe the measurement process itselfitself

Donald KirkpatrickDonald Kirkpatrick

Kirkpatrick developed a model of Kirkpatrick developed a model of training evaluation in 1959 training evaluation in 1959

Arguably the most widely used Arguably the most widely used approach approach

Simple, Flexible and Complete Simple, Flexible and Complete 4-level model4-level model

Measures of Training EffectivenessMeasures of Training Effectiveness

REACTION - how well trainees like a REACTION - how well trainees like a particular training program. Evaluating particular training program. Evaluating in terms of reaction is the same as in terms of reaction is the same as measuring trainees' feelings. It doesn't measuring trainees' feelings. It doesn't measure any learning that takes place. measure any learning that takes place. And because reaction is easy to And because reaction is easy to measure, nearly all training directors measure, nearly all training directors do it. do it.

Reaction (cont)Reaction (cont)

It's important to measure participants' It's important to measure participants' reactions in an organized fashion using reactions in an organized fashion using written comment sheets that have been written comment sheets that have been designed to obtain the desired reactions. designed to obtain the desired reactions.

The comments should also be designed so The comments should also be designed so that they can be tabulated and quantified. that they can be tabulated and quantified.

The training coordinator/trained observer The training coordinator/trained observer should make his own appraisal of the should make his own appraisal of the training to supplement participants' training to supplement participants' reactions. reactions.

The combination of two evaluations is The combination of two evaluations is more meaningful than either one by itself. more meaningful than either one by itself.

Reaction (cont)Reaction (cont) When training directors effectively measure When training directors effectively measure

participants' reactions and find them participants' reactions and find them favorable, they can feel proud. But they should favorable, they can feel proud. But they should also feel humble; the evaluation has only just also feel humble; the evaluation has only just begun. begun.

May have done a masterful job measuring May have done a masterful job measuring reactions, but no assurance that any learning reactions, but no assurance that any learning has taken place. Nor is that an indication that has taken place. Nor is that an indication that participants' behavior will change because of participants' behavior will change because of training. And still further away is any training. And still further away is any indication of results that can be attributed to indication of results that can be attributed to the training.the training.

Collecting reaction measures after Collecting reaction measures after training important:training important:

Memory distortion can affect Memory distortion can affect measures taken at a later point.measures taken at a later point.

There is often a low return rate for There is often a low return rate for questionnaires mailed to people long questionnaires mailed to people long after they have completed the after they have completed the training.training.

LearningLearning

Defined in a limited way: What principles, Defined in a limited way: What principles, facts, and techniques were understood facts, and techniques were understood and absorbed by trainees? (We're not and absorbed by trainees? (We're not concerned with on-the-job use of the concerned with on-the-job use of the principles, facts, and techniques.)principles, facts, and techniques.)

Here are some guideposts for Here are some guideposts for measuring learning: measuring learning:

Measure the learning of each trainee so that Measure the learning of each trainee so that quantitative results can be determined. quantitative results can be determined.

Use a before-and-after approach so that Use a before-and-after approach so that learning can be related to the program. learning can be related to the program.

As much as possible, the learning should be As much as possible, the learning should be measured on an objective basis. measured on an objective basis.

Where possible, use a control group (not Where possible, use a control group (not receiving the training) to compare with the receiving the training) to compare with the experimental group that receives the training. experimental group that receives the training.

Where possible, analyze the evaluation Where possible, analyze the evaluation results statistically so that learning can be results statistically so that learning can be proven in terms of correlation or level of proven in terms of correlation or level of confidence. confidence.

BehaviorBehavior- - Evaluation of training in terms of on-the-Evaluation of training in terms of on-the-job behavior is more difficult than reaction and learning job behavior is more difficult than reaction and learning evals, because one must consider many factors. evals, because one must consider many factors.

Here are several guideposts for evaluating training in Here are several guideposts for evaluating training in termsterms

of behavioral changes: of behavioral changes: Conduct a systematic appraisal of on-the-job Conduct a systematic appraisal of on-the-job

performance on a before-and-after basis. performance on a before-and-after basis. The appraisal of performance should be made by one or The appraisal of performance should be made by one or

more of the following groups (the more the better): more of the following groups (the more the better): trainees, trainees' supervisors, subordinates, peers, and trainees, trainees' supervisors, subordinates, peers, and others familiar with trainees' on-the-job performance. others familiar with trainees' on-the-job performance.

Conduct a statistical analysis to compare before-and-Conduct a statistical analysis to compare before-and-after performance and to relate changes to the training. after performance and to relate changes to the training.

Conduct a post-training appraisal three months or more Conduct a post-training appraisal three months or more after training so that trainees have an opportunity to put after training so that trainees have an opportunity to put into practice what they learned. Subsequent appraisals into practice what they learned. Subsequent appraisals may add to validity of the study. may add to validity of the study.

ResultsResults

The objectives of most training programs can The objectives of most training programs can be stated in terms of the desired results, such be stated in terms of the desired results, such as reduced costs, higher quality, increased as reduced costs, higher quality, increased production, and lower rates of employee production, and lower rates of employee turnover and absenteeism. turnover and absenteeism.

It's best to evaluate training programs directly It's best to evaluate training programs directly in terms of desired results. But complicated in terms of desired results. But complicated factors can make it difficult to evaluate certain factors can make it difficult to evaluate certain kinds of programs in terms of results. kinds of programs in terms of results.

It's recommended that training directors begin It's recommended that training directors begin to evaluate using the criteria in the first three to evaluate using the criteria in the first three steps: reaction, learning, and behavior. steps: reaction, learning, and behavior.

Utility AnalysisUtility Analysis

Cost-benefit analysis: compare costs of Cost-benefit analysis: compare costs of training program with the benefits received training program with the benefits received (both monetary and non-monetary)(both monetary and non-monetary)

Costs: direct costs, indirect costs, Costs: direct costs, indirect costs, overhead, development costs, and overhead, development costs, and participant compensationparticipant compensation

Benefits: improvement in trainee attitudes, Benefits: improvement in trainee attitudes, job performance, quality of work, creativityjob performance, quality of work, creativity

How Should a Training Evaluation How Should a Training Evaluation Study be Designed?Study be Designed?

Case StudyCase Study Training >>>>Measures Taken After TrainingTraining >>>>Measures Taken After Training

Problem- no measures taken prior to training so no Problem- no measures taken prior to training so no way to know whether assertiveness training brought way to know whether assertiveness training brought any changeany change

Pretest-Posttest DesignPretest-Posttest Design Measures Taken Before Training >> Training Measures Taken Before Training >> Training

>>>Measures Taken After Training>>>Measures Taken After Training Little to no value as multitude of unknown factors Little to no value as multitude of unknown factors

could be the real cause of change in performance.could be the real cause of change in performance.

A. A. PRETEST - POSTTEST METHODPRETEST - POSTTEST METHOD1. Most commonly used method in training.1. Most commonly used method in training.2. Does not clearly identify training as the reason for 2. Does not clearly identify training as the reason for

improved knowledge or performance.improved knowledge or performance.

B. B. AFTER-ONLY DESIGN WITH A CONTROL GROUPAFTER-ONLY DESIGN WITH A CONTROL GROUP1. Control group is used to determine whether training 1. Control group is used to determine whether training

made a difference.made a difference.2. No Pretests are given.2. No Pretests are given.3. Both groups take posttest after training.3. Both groups take posttest after training.4. The after-only design with a control group allows 4. The after-only design with a control group allows

trainers to tell whether changes are due to their trainers to tell whether changes are due to their programs.programs.

C. C. PRETEST-POSTTEST DESIGN WITH A PRETEST-POSTTEST DESIGN WITH A CONTROL GROUPCONTROL GROUP

1. Employees are randomly assigned to a treatment group 1. Employees are randomly assigned to a treatment group or a control group.or a control group.

2. Only treatment group receives training.2. Only treatment group receives training.3. Both groups take a posttest.3. Both groups take a posttest.4. Advantages4. Advantages

a. Pretest results ensure equality between the groups.a. Pretest results ensure equality between the groups.b. Statistical analysis determines whether differences in b. Statistical analysis determines whether differences in posttest results are significant.posttest results are significant.

D. D. TIME-SERIES DESIGNTIME-SERIES DESIGN1. Uses a number of measures both before and after 1. Uses a number of measures both before and after

training.training.2. Purpose is to establish individuals' patterns of behavior 2. Purpose is to establish individuals' patterns of behavior

and then see whether a sudden leap inand then see whether a sudden leap in performance followed a training program.performance followed a training program.3. Weakness: because of relatively long time period 3. Weakness: because of relatively long time period

covered, changes in behavior can be attributed tocovered, changes in behavior can be attributed to circumstances other than the program.circumstances other than the program.

More Sophisticated Evaluation More Sophisticated Evaluation DesignsDesigns

Solomon Four Group Design – ideal Solomon Four Group Design – ideal for ascertaining whether a training for ascertaining whether a training intervention had a desired effect on intervention had a desired effect on training behavior. Unlike the designs training behavior. Unlike the designs discussed – this design involves the discussed – this design involves the use of more than one control group. use of more than one control group.

Evaluation statisticallyEvaluation statistically

Preferred choice for analyzing Preferred choice for analyzing training intervention when training intervention when considering statistical power & lower considering statistical power & lower costs is Analysis of Variance with an costs is Analysis of Variance with an after-only control group design.after-only control group design.

The next best approach is the The next best approach is the Analysis of Covariance using the Analysis of Covariance using the pretest score as a covariant.pretest score as a covariant.

Self report -Self report - trainees are asked to evaluate trainees are asked to evaluate

themselves on variables related to the themselves on variables related to the purpose of trainingpurpose of training

measures complicate the measurement of measures complicate the measurement of change because of the problems involved change because of the problems involved in the definition of change itself. in the definition of change itself.

3 Types of Change w/ Self-Report Data 3 Types of Change w/ Self-Report Data are:are:

Alpha changeAlpha change Beta changeBeta change Gamma changeGamma change

Barriers that Discourage Training Barriers that Discourage Training Evaluation (p. 161- 163)Evaluation (p. 161- 163)

Top mgmt doesn’t usually require Top mgmt doesn’t usually require evaluationevaluation

Most senior-level training mgrs don’t Most senior-level training mgrs don’t know how to go about evaluating know how to go about evaluating training programstraining programs

Senior-level training managers don’t Senior-level training managers don’t know what to evaluateknow what to evaluate

Evaluation is perceived as costly & Evaluation is perceived as costly & risky.risky.