bottom-line evaluation: measuring results from
TRANSCRIPT
Bottom-line Evaluation:Measuring Results from
Training and Performance Improvement
Tuesday May 19, 2003
8:00-9:30 a.m.
ASTD International Conference
Donald J. Ford, Ph.D., CPT
Training Education Management
Redondo Beach, CA 90277
BiographyDonald J. Ford, Ph.D., C.P.T., is president and owner of Training Education Management, a training and performance improvement consulting firm in Redondo Beach, California. Dr. Ford also teaches graduate courses in Human Resource Development for Cal State Northridge and Antioch University, L.A. His consulting clients include: ASTD, Toyota, Nissan Motors U.S.A., Rockwell International, U.S. Marines Corps, Orange County Transportation Authority, Glendale Memorial Hospital, and National Education Corp., among others. He has worked in the field of human resource development for eighteen years, including training management positions at Southern California Gas Company, Magnavox, Allied-Signal and Texas Instruments. He possesses another eight years of experience in public education at the secondary, adult and college levels.
He holds a B.A. and M.A. in history and a Ph.D. in education, all from UCLA. He has published 35 articles and three books on topics in training, education and business. He edited a book entitled, In Action: Designing Training Programs published by ASTD in 1996. His third book entitled, Bottom-Line Training: How to Design and Implement Programs That Boost Profits, was released by Gulf Publishing in July, 1999 and forms the basis for this presentation. Dr. Ford has presented at the ASTD International Conference four times, and at ASTD L.A. conferences and workshops.
He may be reached at:
1874 Pacific Coast Highway, Suite 205
Redondo Beach, CA 90277
Telephone/Fax: 310-316-2240
E-mail: [email protected]
Web: www.trainingeducationmanagement.com
Bottom-Line Training Design Model
• Needs Analysis• Performance Analysis• Needs assessment
• Job/Task Analysis• Learner Analysis• Context Analysis• Skill Gap Analysis
AnalyzeAnalyze
•Objective• Deliverables/Strategies• Budgets/Schedules• Project management• Blueprints/Prototypes
DesignDesign
• Materials• Tests/Assessments• Quality Control• Production
DevelopDevelop• Train the Trainer• Classroom Delivery• Non-Classroom Delivery
ImplementImplement
• Formative Evaluation• Reactions/Learning• Transfer of Training• Business Results
EvaluateEvaluate• Learning• Performance• Financial• Strategic
ResultsResults
Work Environment
Level 4:Results
• Performance• Financial
Level 4:Results
• Performance• Financial
Level 3:Job Behavior
• Learner• Organization
Level 3:Job Behavior
• Learner• Organization
Training Environment
Level 1: Reactions
• Learner• Client
Level 1: Reactions
• Learner• Client
Level 2:Learning
• Learner• Organization
Level 2:Learning
• Learner• Organization
Kirkpatrick Evaluation Model
Learning Event
Typical Evaluation Points
AnalyzeDesign/Develop
Implement
Evaluate
Evaluation Design• Tests • Surveys• Formative Evaluation
•Reactions• Pre-Test•Post-Test
•Skills Transfer •Performance•Business Results •Data Analysis •Evaluation Reports
Baseline Data:• Learners Performance
• Skill Gaps• Business Measures
Two Types of Evaluation
Formative Improve the process
& the program Focus on process
Measures: Acceptance Satisfaction Quality
Summative Sum up the impact,
Go-No Go Focus on results
Measures: Performance change Business results ROI
HPI Evaluation Model
BusinessAnalysis
PerformanceAnalysis
CauseAnalysis
InterventionSelection
InterventionImplementation Evaluation
Evaluation Baseline
Formative Evaluation
Summative Evaluation
BusinessGoals
PerformanceGaps
RootCauses
Solutions
ProjectManagement
ChangeManagement
BusinessMeasures
PerformanceMeasures
AnalysisEvaluation
EvaluationDesign
CustomerSatisfaction
Performanceand ROI
HPIOutputs
Evaluation Process
HPI Process
EvaluationOutputs
Figure 21-1: One -Shot Case Study Design
T M
T = training M = measurement
Figure 21-2: One-Group Pre-Post Evaluation Design
M = measurement (1 = Pre, 2 = Post) T = training
M1 M2T
Figure 21-5: Pre-Post Control Group (Classic Experimental Design)
M1 M2
Group 1: R M1 M2T
Group 2: R
R = randomize T = trainingM = measurement (1 = Pre, 2 = Post)
Figure 21-7:Interrupted Time Series Design
Examples: Accident rates pre and post safety training Sales revenues pre and post sales training Error rates pre and post quality training Turnover rate pre and post management training
M2 M5TM1 M6M3 M4
M1- 6 = measures at regular intervals T = training
A Training Parable
One day in Training Land, the Senior Vice-President of the company’s largest division came to request training for all of his 2,000 employees. His reason is that the company is about to launch a new line of products and services and he fears employees will not be able to deliver these successfully without being retrained. Training needs to start in six months.
The content must include: product features and benefits, sales techniques, customer service and a customer information system.
Your assignment is to design an evaluation plan for this request.
Figure 24-1:Bottom-Line Impact of Training
Strategic GrowthStrategic Growth
RevenuesRevenues
SavingsSavings
TrainingTraining
Cost AvoidanceCost Avoidance
Monthly Sales
Sales Training Impact
Product 1/02 2/02 3/02 4/02 5/02 6/02 7/02 8/02 9/02 10/02 11/02 12/02
Us 4% 3% 5% 4% 3% 5% 7% 6% 7% 8% 6% 6%
#1 Cp 46 47 43 44 46 44 43 42 40 39 41 42
#2 Cp 20 21 19 22 20 21 19 20 19 21 20 21
#3 Cp 15 14 13 14 15 13 12 12 14 14 15 14
#4 Cp 8 10 11 10 8 7 9 10 11 10 9 10
#5 Cp 7 7 9 6 7 10 10 10 9 7 8 7
New Sales Training
What was the impact of the sales training on market share? If the total annual market for these drugs is $10 million, what was
the financial benefit of the sales training?
Cancer Drug Sales as % of Market Share
Figure 23-4: Productivity Gains Caused by Workplace Literacy Training Program
Productivity Level (% of Ideal)
Training Group Monthly Productivity Average
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Training Ends= Control Group Productivity Level
What conclusions can you draw from from this chart about
the impact of the training?
If each percentage increase in productivity was worth an average of $5,000 what would be the total value (benefit) of this training?
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
Training Begins
Training Return on Investment (ROI)
Total benefits ÷ total costsExample:
$ 750,000 total benefits
$ 187,500 in total costs
750,000/187,500 = 4:1
Company receives $4 in return for each $1 spent or a return of 400%
ROI Example $$
Category Costs
Development $250,000
Direct (delivery) $250,000
Indirect (participant salaries) $500,000
Benefits
Direct (sales increase) $750,000
Indirect (lower turnover, absenteeism, higher productivity) $750,000
What is the ROI of this training?
Evaluating Behavior, Performance & Skill Transfer
Success factors: Learned new skills Desire to change Supportive work climate Supportive supervision Lack of barriers Rewards and incentives
Performance Measurement Methods
Action planning Pre-post surveys Pre-post interviews Job observation Performance documentation
Output Quality Cost Time Behavior
Figure 23-1: Sample Action Plan for Customer Service Training
Summary of Skills LearnedIn this course, I learned the following new customer service skills:
- How to greet customers. - How to identify customer needs.- How to locate customer account information in the customer database.- How to make simple changes to an account, like updating the customer’s name,
address, applying payments, waiving late charges and fees.- How to handle customer complaints and document problems.
Ways the New Skills Can be AppliedOnce I return to work, I can apply these new skills in the following ways:
- Use the standard customer greeting I learned in class with every customer.- Ask questions of the customer, following the Inquiry Matrix I learned about in class.- Listen actively to identify the customer’s real need.- Use the Customer Database (CD) to access account number in the system.- Use the system features to update the customer’s account.- Handle any complaints by using the Customer Complaint Resolution Job Aid.
Obstacles or Barriers to Skills ApplicationBarriers include:
- Lack of time to practice on the job.- Waiting time for system to access account information.- Some customers do not speak English well and are hard to understand.- Some customers are already so upset by the time they call that it is difficult
to solve their problem.
Enablers for Skill ApplicationEnablers include:
- My supervisor wants met to succeed and is willing to give me time to learn the new skills.- My co-workers are willing to help me and answer questions when they have time.- I like helping people and will try to satisfy upset customers. - The computer system is being improved to speed up access time.
Sample Action Plan for Customer Service Training, continued
Specific Steps To Be Taken and Deadlines Step: Deadline:1. Use standard greeting on all calls. Tomorrow2. Begin using the Inquiry Matrix on all calls. Tomorrow3. Practice looking up customer information. Tomorrow4. Become proficient on the Customer Database System. One month from now5. Use the Customer Complaint Resolution Job Aid to One Month from now
resolve all customer complaints.
Follow up Actions to be Taken to Review Progress- Meet with my boss to discuss this action plan within one week.- Begin using the job aids immediately to help me.- Ask co-workers for help when I need it.- Register to attend another customer service class next month.
Sample Action Plan for Customer Service Training, continued
District Manager Training Pre-Post Survey
This manager Pre Post Pre Post Pre Post Pre Post Pre Post
Communicates effectively with employees 3.5 4.5 2.4 3.0 2.1 2.9 3.0 3.3 2.8 3.4 21%
Establishes clear performance expectations and goals 2.7 4.1 1.9 2.7 3.1 3.9 3.5 3.6 2.8 3.6 29%
Provides coaching and on the job training 3.2 3.8 3.0 3.3 2.5 2.9 3.7 3.7 3.1 3.4 10%
Question Self Manager Employee Colleague Average%
Gain
1. What conclusions would you draw about the training’s impact on supervisors’ job behavior?
2. What recommendations would you make about changes to the program based on this data?
3. How could you translate this data into financial results?
Balanced Scorecard
Training Activity % Payroll
US Avg.=1.9% ‡
$/Employee/yearUS Avg.=$569 ‡
Hrs./Empl./yearUS Avg.= 36 ‡
Training Efficiency Cost/Hour
US Avg.=$28 ‡
% Billable Hrs.US Avg.=70% †
Training Results % Positive Course Ratings
Avg.=91%†
% Learning GainAvg.=45%†
% Behavior ChangeAvg.=25%†
Revenues/EmployeeAvg.=$100K†
Market/Book† Avg.=10:1
† Source=Author’s Study of High Tech firms
‡ Source: ASTD Training Data Book, 1996.
Your Scorecard
Training Activity % Payroll:_____
Training budget/Payroll
Hrs./Empl.____ Training hours/Employees
$/Empl. _____ Training budget/Employees
Training Efficiency Cost/Hr. ______
Training budget/ training hours
% Billable ____ Billable hours/staff hours
Training Results % Positive Ratings
Positive ratings/total ratings _________________ % Learning Gain
Post-pre / Pre-test scores
_________________ % Performance Gain
Post-pre / Pre-performance
_________________ Revenues/Employee
Total revenues/employees
_________________ Market/Book Ratio
Total market worth/ total physical assets
Training Design Job Aid
Analysis Phase
Needs Analysis Performance Analysis Needs Assessment Job Analysis
Task Analysis Learner Analysis Context Analysis Skill Gap Analysis
Design Phase
Training Objectives Deliverables/Strategies Budgets/Schedules
Project management Blueprints/Prototypes
Development Phase
Draft Materials Tests/Assessment
Quality Control Full-scale Production
Training Design Job Aid, continued
Implementation Phase
Train the Trainer Classroom Delivery
Non-Classroom Delivery
Evaluation Phase
Formative Reactions Learning
Transfer to Job Results/ROI
Closing commentsand Q&A
Donald J. Ford, Ph.D., CPT
Training Education Management
Redondo Beach, CA 90277
Telephone/Fax: 310-316-2240
E-mail: [email protected]
Web: www.trainingeducationmanagement.com