roi of training - isolating the results

ROI OF TRAININGROI OF TRAINING

Session 6 – Isolating the Session 6 – Isolating the Results/Effects Of the Training Results/Effects Of the Training InterventionIntervention

Pre-Amble

Isolating the results/effect of the training effort is the most difficult and also most important element of building the case for value (ROI);

Because of this, value at the higher levels is seldom determined;

However, the credibility not only of your study to show value, but also the training function is based on this element.

Pre-Amble

The complexity of isolating the effect, is a function of the variables that are in play at any moment in time in an organization

Pre-Amble

The Challenge

Pre-Amble

An acceptable degree of accuracy

External Factors

ManagementAttention

Incentives

Systems/ProceduresChanges

LearningProgram

TOTALIMPROVEMENT

AFTERPROGRAM

EFFECT OFLEARNING ON IMPROVEMENT

How then do you isolate the

effects?

Control groups Trend line analysis of performance data Participant’s estimation of impact Supervisor’s estimation of impact Management’s estimation of impact Use of previous studies Use of customer input Subordinate’s report of other factors Estimating the impact of other factors

Isolating the Effects of the Intervention

Control groups Trend line analysis of performance data Participant’s estimation of impact Supervisor’s estimation of impact Management’s estimation of impact Use of previous studies Use of customer input Subordinate’s report of other factors Estimating the impact of other factors

Isolating the Effects of the Intervention

Isolating the Effects of the Intervention

External Factors

ManagementAttention

Incentives

Systems/ProceduresChanges

LearningProgram

ELIMINATE

Control groups Trend line analysis of performance data Participant’s estimation of impact Supervisor’s estimation of impact Management’s estimation of impact Use of previous studies Use of customer input Subordinate’s report of other factors Estimating the impact of other factors

Isolating the Effects of the Intervention

Isolating the Effects of the Intervention

External Factors

ManagementAttention

Incentives

Systems/ProceduresChanges

LearningProgram

CALCULATEAND ATTRIBUTE THE

REST

Control Groups

Most accurate isolation approach; Involves the use of an experimental group

that is impacted by the intervention and a control group that is not;

Composition of both groups should be as identical as possible and, if feasible, the selection of participants for each group should be on a random basis.

Control Group Designs

Control GroupControl Group

ExperimentalGroup

M1M1

M1 Program

M2M2

M2

Question is What Do You

Control?

Case Study

Federal Express ROI analysis using control groups as a good example of the state of the art in measuring ROI. The study focused on 20 employees who went through an intense, redesigned two-week training program soon after being hired to drive company vans. Their performance was compared with a control group of 20 other new hires whose managers were told to do no more or less on-the-job training than they normally would. Performance was tracked for the two groups for 90 days in categories such as accidents, injuries, time-card errors, and domestic air-bill errors. The ten performance categories were assigned monetary values by experts from engineering, finance, and other groups. The intervention demonstrated that the performance of the highly trained employees was superior to that of the group that did not receive the upgraded training and also resulted in a 24% return on investment.

Also look at CLC Best PracticeDocument

Federal Express demonstrates: Practicality Non disruptive; Relatively accurate; Line involvement (and commitment); ROI is possible if you know what you are

aiming for (measures)

Also look at CLC Best PracticeDocument

Practical Exercise

Accra Wholesale Brickworks, find that there production targets are not being met. The Company is midsize but established in the City. It employs a production team consisting of three foreman, that supervise approximately 8 workers each and these teams take responsibility for procurement of the raw materials, mixing, filling the moulds, drying and baking. The production function is managed by a section manager.

Practical Exercise

Practical Exercise

Using the control group method: How would you selects your experimental

group vis your control group What will you control How will you apply the Trend line

Analysis Technique?

Trend-line Analysis

Basic format less accurate than control group Draw a trend-line using previous performance data

as a base When intervention is conducted, actual

performance is plotted and compared to the trend line

Improvement in actual performance over what the trend line predicted can then be reasonably attributed to the intervention

The primary advantage of this approach is that it is simple, inexpensive, and takes very little effort;

However not accurate as it does not isolate into other factors;

However a good first order analysis.

AveragePost Program 35

AveragePre Program 55

Time

Com

pla

ints

SexualHarassment

Prevention Program

Formal Internal Complaints of Sexual Harassment

O N D J F M A M J J A S O N D J F M A M J J A S O

Trend-line Analysis

Trend line Use Case Study(Micro Electronics Incorporated)

Practical Exercise

Estimate of Impact by Key Role Players

Easy to implement; Based on the assumption that participants

are capable of determining or estimating how much of a performance improvement/impact is related to the intervention;

Because their actions have produced the improvement, participants may have very accurate input on the issue;

Although an estimate, this value will usually have considerable credibility with management because participants are at the center of the change or improvement;

High face validity.

Estimate of Impact by Key Role Players

Easy to implement; Based on the assumption

that participants are capable of determining or estimating how much of a performance improvement/impact is related to the intervention;

Because their actions have produced the improvement, participants may have very accurate input on the issue;

Although an estimate, this value will usually have considerable credibility with management because participants are at the center of the change or improvement;

High face validity.

TypicalResearch Questions

The individuals who do not respond to the questionnaire or provide non- usable data on the questionnaire are assumed to report no improvement/impact;

Extreme data and unrealistic claims are omitted from the analysis;

Only annualized values are used, it is assumed that there are no benefits from the program after the first year of implementation;

The confidence level, expressed as a percent, is multiplied by the improvement value to reduce the amount of the improvement by the potential error;

If value is expressed it is also factored by the confidence value.

Adjustments to

improve validity

Estimate of Impact by Key Role Players

Estimate of Impact by Key Role Players

% Program Contributed to Improvement

Confidence of Statement as %

Factor Total Improvement (annualized)– Value

Contribution to Improvement

60 % (.6) 40% (.4) .24 1000 240

60 % (.6) 40% (.4) .24 1000 240

50% (.5) 50% (.5) .25 1000 250

Estimate of Impact by Key Role Players

The process can be

expandedto

include:

Supervisors Management Other stakeholders

Expert Input

Select carefully, based on their knowledge of the process and situation;

If capacity does not exist in the organization; Will probably use techniques that most

training practitioners can implement; Organization culture will determine whether

the outcome will be accepted; If linked to skills transfer it could be

worthwhile; Often top management place more confidence

in outside experts than in their own staff.

Obtaining Input From a Target Group

An example is the subordinates of the supervisors trained in people handling skills;

It could be possible for them to estimate how much of an improvement can be attributed to the intervention;

They can also provide input in terms of what other factors might have contributed to the improvement;

The input is usually obtained through surveys or interviews. When the survey results show significant changes in participant behavior after training and no significant change in the general work climate, the improvement in work performance, therefore, must be attributed to the changes in supervisor behavior since other factors remained constant;

This approach has some disadvantages. Data from subordinates is subjective and may be questionable because of the possibility for biased input;

Often subordinates may have difficulty in determining changes in the work climate. However, in some cases, subordinates are aware of the factors that caused changes in their work unit and can provide input about the magnitude or quantity of changes. When combined with other methods to isolate impact, this process has increased credibility.

Obtaining Input From a Target Group

In some situations it could be feasible to calculate the impact of factors (other than the intervention) that influenced the improvement and then conclude that the intervention is credited with the remaining portion;

In terms of this approach, the intervention claims credit for improvement that cannot be attributed to other factors;

The method is appropriate when the other factors are easily identified and the appropriate mechanisms are in place to calculate their impact on the improvement;

The process can be very credible if the method used to isolate the impact of other factors is credible.

Which Strategy?

Control group feasibility; Availability of historical

data; Expertise to conduct study; Capability and willingness

to estimate impact; Can non-training effects be

isolated? What are the constraints in

terms of using the data collection technique and isolation strategy.

Credibility of Data is Influenced By:

Reputation of the source of the data;

Reputation of the source of the study;

Motives of the researchers; Personal bias of audience; Methodology of the study; Assumptions made in the

analysis; Realism of the outcome data; Type of data; Scope of analysis.