the bottom line on roi: measuring the impact and roi of ...€¦ · return on investment compares...
TRANSCRIPT
The Bottom Line on ROI:Measuring the Impact and ROI of
Organization Development Programs, Projects, and Initiatives
Timothy R. Brock, PhD, CPT, CRP
Director of Consulting Services
ROI Institute
BOTTOMLINE on ROIThe
2
Reaction Objectives
The intent is for you to perceive this 90-minute session content as:
Relevant to their work
Important to their success
Immediately applicable
BOTTOMLINE on ROIThe
3
After completing this session, you should be able to:
Identify the drivers for ROI
Make the business case for ROI
Explain how business alignment is achieved
Identify 7 of the 12 guiding principles
Describe the 10 steps in the ROI Methodology
Learning Objectives
BOTTOMLINE on ROIThe
4
Within two weeks of this session, you should:
Conduct a briefing on ROI.
Develop Level 3 and Level 4 objectives for a specific program.
Application Objectives
BOTTOMLINE on ROIThe
5
There is greater demand to show the value of investments in OD programs, projects, & initiatives at my organization
than in the past.
A. Yes
B. No
C. I don’t know
BOTTOMLINE on ROIThe
6
What is Value?
Exercise:
Think about a project or program that you recently implemented.
How would you define the value delivered by the program?
BOTTOMLINE on ROIThe
7
The New Definition of ValueValue Must:
Be balanced, with qualitative and quantitative data
Contain financial and nonfinancial perspectives
Reflect strategic and tactical issues
Satisfy all key stakeholders
Be consistent in collection and analysis
Be grounded in conservative standards
Come from credible sources
BOTTOMLINE on ROIThe
9
Shifting ParadigmsActivity Based Results Based
No business need for the program Program linked to specific business needs
No assessment of performance issues Assessment of performance effectiveness
No specific measurable objectivesSpecific objectives for behavior and business impact
No effort to prepare program participants to achieve results
Results expectations communicated to participants
No effort to prepare the work environment to support transfer
Environment prepared to support transfer
No efforts to build partnerships with key managers
Partnerships established with key managers & clients
No measurement of results or benefit-cost analysis
Measurement of results and benefit-cost analysis
Planning and reporting is input focused Planning and reporting is outcome focused
Source: Measuring ROI in Healthcare: Tools and Techniques to Measure the Impact and ROI in Healthcare Improvement Projects and Programs, 2013, V. V. Buzachero, J. Phillips, P. P. Phillips, and Z. L. Phillips.
BOTTOMLINE on ROIThe
10
The Executive View *Measure
CurrentMeasure
ShouldMeasure
Measure Importance
1. InputsLast year, 78,000 employees received formal learning.
94%
2. EfficiencyFormal learning costs $2.15 per hour of learning consumed.
78%
3. ReactionEmployees rated our training very high, averaging 4.2 out of 5.
53%
4. Learning92% of participants increased knowledge and skills
32%
5. ApplicationAt least 78% of employees are using the skills on the job.
11%
6. ImpactOur programs are driving our top 5 business measures in the organization.
8%
7. ROIFive ROI studies were conducted on major programs yielding an average of 68% ROI.
4%
8. AwardsOur learning and development program won an award from the Association for Talent Development.
40%
* CEO Survey—Fortune 500 and Large Private Company, ROI Institute
85%
82%
22%
28%
61%
96%
74%
44%
6
5
8
4
1
2
3
7
BOTTOMLINE on ROIThe
MeasurementLevels
Measurement Category
2006Status
2011Status
0 Inputs/IndicatorsNumber of programs, participants, costs, efficiencies
100% 100%
1 ReactionReaction and satisfaction with experience,content, value of program
92% 89%
2 LearningAcquisition of information, knowledge, skills, contacts
48% 59%
3 ApplicationUse of information, knowledge, skill, insights
11% 34%
4 ImpactChanges in business impact measures such as output, quality, time, costs
8% 21%
5 ROICompares monetary benefits to cost of program, project, or initiative
2% 11%
2011 Global Leadership Development Practices Survey Results
N=232; large organizations with leadership development budgets averaging $3 million annually.
BOTTOMLINE on ROIThe
12
The ROI Methodology
Generates Six Types of Measures:
• Reaction and Planned Action
• Learning
• Application
• Business Impact
• Return on Investment
• Intangible Measures
The ROI Methodology also includes a technique to isolate the effects of the program or solution.
BOTTOMLINE on ROIThe
14
Level Measurement Focus
1. Reaction and PlannedAction
Measures participant satisfaction with the program/project and captures planned action
2. LearningMeasures changes in knowledge andskills
3. ApplicationMeasures changes in on-the-job behavior or actions
4. Business ImpactCaptures changes in business impact measures
5. Return on InvestmentCompares monetary benefits to the costs
ROI Framework
BOTTOMLINE on ROIThe
16
Develop/ Review
Objective of Program
Develop EvaluationPlans and
Baseline Data
Collect Data During
Program
Implementation
Collect DataAfter
Program
Implementation
Evaluation Planning Data Collection
LEVEL 1: REACTION AND PLANNED ACTION
LEVEL 2:LEARNING
LEVEL 3: APPLICATION
LEVEL 4: BUSINESS IMPACT
The ROI Methodology, Part 1
BOTTOMLINE on ROIThe
17
Isolate the Effects of
Program
Convert Data To
MonetaryValue
Calculatethe Return
OnInvestment
GenerateImpact
Study
Data Analysis Reporting
LEVEL 5: ROI
Intangible Benefits
Capture Costs of Solution
Identify Intangible Measures
The ROI Methodology, Part 2
BOTTOMLINE on ROIThe
19
Guiding Principles, Part 1
1.When conducting a higher-level evaluation, collect data at lower levels.
2.When planning a higher level evaluation, the previous level of evaluation is not required to be comprehensive.
3.When collecting and analyzing data, use only the most credible sources.
4.When analyzing data, select the most conservative alternatives for calculations.
5.Use at least one method to isolate the effects of the program or project.
6.If no improvement data are available for a population or from a specific source, assume that no improvement has occurred.
BOTTOMLINE on ROIThe
20
Guiding Principles, Part 2
7.Adjust estimates of improvements for the potential error of the estimates.
8.Avoid use of extreme data items and unsupported claims when calculating ROI calculations.
9.Use only the first year of annual benefits in the ROI analysis of short-term solutions.
10.Fully load all costs of the solution, project, or program when analyzing ROI.
11.Intangible measures are defined as measures that are purposely not converted to monetary values.
12.Communicate the results of the ROI Methodology to all key stakeholders.
BOTTOMLINE on ROIThe
ROI Myths• ROI is too complex for most users.
• ROI is too expensive, consuming too many critical resources.
• If senior management does not require ROI, there is no need to pursue it.
• ROI is a passing fad.
• ROI is too subjective
• ROI is for post analysis only
24
BOTTOMLINE on ROIThe
26
The ROI Calculation
Benefit Cost Ratio (BCR) =
ROI = X 100
$300,000Program Costs
$300,000 - $200,000(Program Benefits – Program Costs)
$200,000
$200,000Net Program Benefits
Program Costs$100,000
= 1.50:1
= 0.50 X 100% = 50%
Program Benefits
BOTTOMLINE on ROIThe
28
Develop/ Review
Objective of Program
Develop EvaluationPlans and
Baseline Data
Collect Data During
Program
Implementation
Collect DataAfter
Program
Implementation
Evaluation Planning Data Collection
LEVEL 1: REACTION AND PLANNED ACTION
LEVEL 2:LEARNING
LEVEL 3: APPLICATION
LEVEL 4: BUSINESS IMPACT
The ROI Methodology, Part 1
BOTTOMLINE on ROIThe
29
Reaction & Planned Action
Learning
Application & Implementation
Isolate the Effects of the Program
Intangible Benefits
Impact
Ch
ain
of
Imp
act
BOTTOMLINE on ROIThe
30
Level 0 Input and Indicators
• Number of programs
• Participants
• Hours
• Requests
• Efficiencies
• Costs
• Time to Deliver
Level 1 Reaction and Planned Action
• Relevance*
• Importance*
• Usefulness
• Recommend to Others
• Necessary
• Practical
• Uniqueness
• Concreteness
• New Information*
• Motivation
• Appropriateness
• Intent to Use**Correlates with Application
Five Levels of Measurement: Examples
BOTTOMLINE on ROIThe
31
Level 2 Learning
• Information
• Knowledge
• Understanding
• Capability
• Contacts
• Confidence
• Perceptions
• Skills
Level 3 Application
• Use of Information
• Use of Knowledge
• Use of Skill*
• Frequency of Use*
• Completion of Actions
• Completion of Tasks
• Implementation of Ideas
• Following the Policy
• Use of Procedure
• Use of Regulation
• Success with Application
• Barriers*
• Enablers* * Essential Measures at this Level
Five Levels of Measurement: Examples
BOTTOMLINE on ROIThe
32
Level 4 Business Impact Level 5 Return on Investment• Productivity
• Quality
• Errors
• Incidents
• Re-Work
• Efficiency
• Compliance Discrepancies
• Citizen Complaints
• Costs
• Employee Engagement
• Employee Retention
• Service Delivery
• Cycle Time
• Customer Satisfaction
• ROI (%)
• Benefit/Cost Ratio
• Payback Period
Intangible Measures
…. includes a technique to isolate the effects of the communication project.
• Image
• Reputation
• Teamwork
• Stress
• Brand
Five Levels of Measurement: Examples
BOTTOMLINE on ROIThe
34
Criteria for Selecting Programs for Level 4 and Level 5 Evaluation
Benchmarking*
• Cost of the program 52%
• Importance of program to strategic objectives 50%
• Executive interest in the evaluation 48%
• Visibility of the program 45%
• Linkage of program to operational goals and issues 29%
• Life cycle of the program 14%
• Investment of time required 7%
• Size of target audience 6% *Survey of Users, N = 235
BOTTOMLINE on ROIThe
37
Program
Payoff Need
Business Need
Performance Need
Learning Need
Preference NeedInitial
Analysis
Start Here End Here
5 ROI5
4 Impact4
3 Application3
2 Learning2
1 Reaction1
Absenteeism is costing $10K/month
• Unexpected absenteeism is 9% & growing
• Benchmark data is 5%
Discussion between team member & supervisor is not occurring when there is an unplanned absence
Deficiency is counseling/ discussion skills
• One day counseling skills workshop must provide relevant, practical skills
• Facilitator-led• Participants are supervisors
• Reaction questionnaire at end of workshop• Planned Actions
• Skills practice during workshop• Participant assessments• Facilitator assessments
• Questionnaire – at 3 months• Action Plan – 6 months
• Monitor absenteeism records for 6 months
• Action Plan – 6 months
Calculate ROI/BCR
Program rated 4 out of 5 on: Need, Relevance, Practicality
+ Commitment
Applies counseling skills to standard in work-like setting
Reduce absenteeism to 5% six months after course
ROI of 25%
• Apply Action Plans• Conduct counseling sessions in 95% of
situations when an unexpected absence occurs
Business Alignment & Forecasting
Measurement & Evaluation
ROI Process Model
Needs Objectives Evaluation
ROI Objectives
Impact Objectives
Application Objectives
Learning Objectives
Reaction Objectives
Levels of Objectives
Measurement Focus
0. Inputs and IndicatorsThe input into the project in terms of scope, volume, efficiencies, costs
Participants, hours, costs, timing
1. Reaction & PlannedAction
Reaction to the project or program, including the perceived value
Relevance, importance, usefulness, appropriateness, intent to use, motivation to take action
2. LearningLearning to use the content and materials, including the confidence to use what was learned
Skills, knowledge, capacity, competencies, confidence, contacts
3. Application &Implementation
Use of content and materials in the work environment, including progress with actual items and implementation
Extent of use, task completion, frequency of use, actions completed, success with use, barriers to use, enablers to use
4. Business ImpactThe consequences of the use of the content and materials expressed as business impact measures
Productivity, revenue, quality, time, efficiency, customer satisfaction, employee engagement
Typical Measures
5. ROIComparison of monetary benefits from program to program costs
Benefit-cost ratio (BCR), ROI%, payback period
BOTTOMLINE on ROIThe
46
Matching Evaluation Levels with Objectives
1. Reaction and Planned Action
2. Learning
3. Application
4. Business Impact
5. Return on Investment
BOTTOMLINE on ROIThe
47
1. Error rates on reports will decrease by 20 percent.
1. Error rates on reports will decrease by 20 percent.
44
2. Participants will use disciplinary discussion skills in 90 percent of discussions.
2. Participants will use disciplinary discussion skills in 90 percent of discussions.
3. Customer complaints will decrease.3. Customer complaints will decrease.
4. Participants will indicate an intention to implement the new procedure within two
months.
4. Participants will indicate an intention to implement the new procedure within two
months.
5. Participants will score 75 or higher on the new product quiz.
5. Participants will score 75 or higher on the new product quiz.
33
44
11
22
1 Reaction 2 Learning 3 Application 4 Impact 5 ROI
BOTTOMLINE on ROIThe
48
6. Participants will achieve a post-test score increase of 30 percent over pretest.
6. Participants will achieve a post-test score increase of 30 percent over pretest.
7. Participants will initiate at least three cost reduction projects within 15 days.
7. Participants will initiate at least three cost reduction projects within 15 days.
8. Per project costs will be reduced by $2,000. 8. Per project costs will be reduced by $2,000.
22
33
44
9. Time to complete a project will decrease.9. Time to complete a project will decrease. 44
10. Program implementation will realize a 2:1 benefit-cost ratio within one year.
10. Program implementation will realize a 2:1 benefit-cost ratio within one year.
55
1 Reaction 2 Learning 3 Application 4 Impact 5 ROI
BOTTOMLINE on ROIThe
49
11. Program relevance to the job will receive a score of at least 4.5 out of 5.0.
11. Program relevance to the job will receive a score of at least 4.5 out of 5.0.
12. External customer satisfaction index increase by 25 percent in three months.12. External customer satisfaction index increase by 25 percent in three months.
13. CSRs representatives will handle customer complaints with the 5-step process in 95% of
complaint situations
13. CSRs representatives will handle customer complaints with the 5-step process in 95% of
complaint situations
14. Canal operators will achieve a simulation test score average of 75.
14. Canal operators will achieve a simulation test score average of 75.
15. Participants will use new customer services interaction skills with every customer at least 50%
of the time.
15. Participants will use new customer services interaction skills with every customer at least 50%
of the time.
11
44
33
22
33
1 Reaction 2 Learning 3 Application 4 Impact 5 ROI
BOTTOMLINE on ROIThe
50
17. Participants will use new software daily as reflected by an 90% score on an unscheduled
audit of use.
17. Participants will use new software daily as reflected by an 90% score on an unscheduled
audit of use. 33
18. Customer response time will be reduced. 18. Customer response time will be reduced.
19. Supervisors will conduct a meeting with direct reports to implement new policy.
19. Supervisors will conduct a meeting with direct reports to implement new policy.
20. Employees will submit ideas or suggestions for improvement in the first year.
20. Employees will submit ideas or suggestions for improvement in the first year.
44
33
33
16. Participants will rate the training as a good investment for the company.
16. Participants will rate the training as a good investment for the company.
11
1 Reaction 2 Learning 3 Application 4 Impact 5 ROI
BOTTOMLINE on ROIThe
51
This ROI Methodology is a Process Improvement toolFund, Fix, or Fold
• Just because a program has a negative ROI does not mean the program is discontinued
• Must understand what happened so we can make changes to make it positive in the future to drive business value
• Our analysis statistics indicate that 20%-30% of programs have a negative ROI with 10% of those being discontinued.
• It is rare when ROI is the only factor• Intangibles can offset a negative ROI
• Ownership, engagement, commitment, communication, teamwork, cooperation, etc.
• If it can’t produce any business value, then we have a data-driven case to let it die in peace
BOTTOMLINE on ROIThe
52
The Power of Objectives
Level 3 and Level 4 Objectives provide:
• Direction to designers and developers
• Guidance to facilitators and coordinators
• Goals for participants
• Satisfaction for program sponsors
• A framework for evaluators
BOTTOMLINE on ROIThe
53
Data Collection During and After ProgramMethod Level 1 Level 2 Level 3 Level 4
Surveys
Questionnaires
Observation
Interviews
Focus Groups
Tests/Quizzes
Demonstrations
Simulations
Action planning/improvement plans
Performance contraction
Performance monitoring
BOTTOMLINE on ROIThe
54
Isolate the Effects of
Program
Convert Data To
MonetaryValue
Calculatethe Return
OnInvestment
GenerateImpact
Study
Data Analysis Reporting
LEVEL 5: ROI
Intangible Benefits
Capture Costs of Solution
Identify Intangible Measures
The ROI Methodology, Part 2
BOTTOMLINE on ROIThe
57
Systems/ProceduresChanges
Several Factors Contribute to an Improvement After a Program in Conducted
External Factors
ManagementAttention
Incentives
ODPrograms
TOTALIMPROVEMENT
AFTERPROGRAM
EFFECT OF HR ON IMPROVEMENT
BOTTOMLINE on ROIThe
58
Isolating the Effects of the ProgramBenchmarking Data*
* Survey of Users, N = 235
BOTTOMLINE on ROIThe
59
M1 M2
M1 Program M2Experimental
Control
M2
Program M2Experimental
Control
M2
Program M2Experimental
Control
M2Control
M1
M1
Control Groups
BOTTOMLINE on ROIThe
60
2%
1%
J F M A M J J A S O N D J
1.85% Pre Program Six-Month Average
Projected Average —Using Pre Data as a
Base 1.45%
0.7% Post Program Six-MonthAverage
CPI Program Conducted
MONTHS
Reject Rate
Trend Line Analysis
5.8
BOTTOMLINE on ROIThe
61
Monthly increase in credit card accounts: 175 (fact)
Contributing Factors Consensus
Impact (%)
Average Confidence (%)
Sales Training 32% 83%
Incentives 41% 87%
Management Reinforcement
14% 62%
Market Fluctuations 11% 75%
Other _________ 2% 91%
100%
5.12
175 x = 56 x = 46.48(at least)
Participant Estimates
BOTTOMLINE on ROIThe
Influence Fact % Contrib. Est. Impact Confidence Adjusted Impact
Sales Training 175 32% 56 83% 46.48
Fact: 175 New Credit Card Accounts
% Contribution: 32%
Est. Impact: 56
Uncertainty 17%
Margin of Error: +/- 9.52
65.52
56
46.48
62
Participant Estimates
BOTTOMLINE on ROIThe
63
Isolate the Effects of
Program
Convert Data To
MonetaryValue
Calculatethe Return
OnInvestment
GenerateImpact
Study
Data Analysis Reporting
LEVEL 5: ROI
Intangible Benefits
Capture Costs of Solution
Identify Intangible Measures
The ROI Methodology, Part 2
64
Data Conversion
• Converting output to contribution – standard value
• Converting the cost of quality – standard value
• Converting employee’s time – standard value
• Using historical costs
• Using internal and external experts
• Using data from external databases
• Linking with other measures
• Using participants’ estimates
• Using supervisors’ and managers’ estimates
• Using staff estimates
Slide 64
The BOTTOMLINE on ROI
BOTTOMLINE on ROIThe
Five Steps to Data Conversion
1. Focus on the unit of measure.
2. Determine the value (V) of the unit of measure.
3. Calculate the change in performance(∆P).
4. Determine the annual change in performance (A∆P).
5. Calculate the annual monetary benefits (A∆P x V).
BOTTOMLINE on ROIThe
1. One grievance
2. V = $6,500
3. ∆P = Prevented 7 out of 10 grievances/month
4. A∆P =
5. A∆P x V =
7 x 12 = 84
84 x $6,500 = $546,000
Five Steps to Data Conversion
BOTTOMLINE on ROIThe
To Convert or Not to Convert
67
No
Is there a standard value?
Yes
Is there a method to get there?
Can we get there with minimum resources?
No
Yes
No No
Yes
Add to numerator
Leave as Intangible
Can we convince our executives of credibility in 2 minutes?
Convert data
Yes
68
Typical Intangible Benefits
Adaptability
Awards
Brand awareness
Career minded
Caring
Collaboration
Communication
Conflicts
Cooperation
Corporate social responsibility
Creativity
Culture
Customer complaints
Customer response time
Customer satisfaction
Decisiveness
Slide 68
The BOTTOMLINE on ROI
69
Typical Intangible Benefits
Employee complaints
Engagement
Execution
Image
Innovation
Job satisfaction
Leadership
Networking
Organizational climate
Organizational commitment
Partnering
Reputation
Resilience
Stress
Talent
Teamwork
Slide 69
The BOTTOMLINE on ROI
BOTTOMLINE on ROIThe
70
Isolate the Effects of
Program
Convert Data To
MonetaryValue
Calculatethe Return
OnInvestment
GenerateImpact
Study
Data Analysis Reporting
LEVEL 5: ROI
Intangible Benefits
Capture Costs of Solution
Identify Intangible Measures
The ROI Methodology, Part 2
BOTTOMLINE on ROIThe
72
Micro Level Scorecard Macro Level Scorecard
0 Indicators
1 Reaction
2 Learning
3 Application
4 Impact
5 ROI
Intangibles
0
1
2
3
4
0
1
2
3
The Bottom Line on ROI:Measuring the Impact and ROI of
Organization Development Programs, Projects, and Initiatives
Timothy R. Brock, PhD, CPT, CRP
Director of Consulting Services
ROI Institute
Thank you for attending.
BOTTOMLINE on ROIThe
Please Complete the Survey Before You Leave
Your feedback is greatly appreciated
Thank you!
BOTTOMLINE on ROIThe
77
ROI Calculation
Net Project BenefitsProject Costs
ROI =
Cost of project $230,000Benefits of project (1st year) $430,000
ROI =$430,000 - $230,000
$230,000= 0.87 x 100 = 87%
BOTTOMLINE on ROIThe
Impact Measures
78Slide 78
Output
• Units Produced
• Items Assembled
• Revenue
• Items Sold
• Forms Processed
• Inventory Turnover
• Output Per Hour
• Productivity
• Work Backlog
• Incentive Bonus
• Shipments
• New Accounts Generated
BOTTOMLINE on ROIThe
79Slide 79
Cost
• Budget Variances
• Unit Costs
• Cost by Account
• Variable Costs
• Fixed Costs
• Overhead Cost
• Operating Costs
• Number of Cost Reductions
• Project Cost Savings
• Accident Costs
• Sales Expense
Impact Measures
BOTTOMLINE on ROIThe
80Slide 80
Time
• Equipment Downtime
• Overtime
• On Time Shipments
• Time to Project Completion
• Processing Time
• Cycle Time
• Supervisory Time
• Training Time
• Repair Time
• Efficiency
• Work Stoppages
• Order Response
• Late Reporting
• Lost Time Days
Impact Measures
BOTTOMLINE on ROIThe
81Slide 81
Quality
• Scrap
• Waste
• Rejects
• Error Rates
• Rework
• Shortages
• Product Defects
• Deviation From Standard
• Product Failures
• Inventory Adjustments
• Number of Accidents
Impact Measures
BOTTOMLINE on ROIThe
82Slide 82
Customer Service
• Customer Satisfaction
Survey
• Customer Satisfaction
Index
• Customer Complaints
• Customer Comments
• Customer Defection
• Customer Retention
Work Habits
• Absenteeism
• Tardiness
• First Aid Treatments
• Violations of Safety Rules
• Excessive Breaks
Impact Measures
BOTTOMLINE on ROIThe
83Slide 83
Work Climate
• Number of Grievances
• Number of Discrimination
Charges
• Employee Complaints
• Job Satisfaction
• Organizational
Commitment
• Employee Turnover
• Reduced Litigation
Impact Measures
BOTTOMLINE on ROIThe
Case Study: The Result
84
Four months after implementing an expensive, strategic program, you conduct an impact study and discover from participants involved:
• Reacted very positively to the program and found it to be very relevant to their work;
• Learned new skills and gained new insights about themselves;
• Applied the skills and insights routinely with their teams, although they had some difficulty in a few areas;
• Improved several important work unit measures, with some measures improving as much as 28%;
• Achieved an impressive 105% return on investment; and
• Reported an increase in job satisfaction in the work unit.
Slide 84
BOTTOMLINE on ROIThe
85Slide 85
What were the objectives?
How many people were involved?
What was the scope of analysis?
What assumptions were made in the analysis?
What specific measures were improved?
When were the data collected?
What was the data collection plan
beyond reaction?
How were the needs and measures determined?
What assumptions were made in the ROI
calculations?
How were results isolated from other
factors?
What is the upfront business connection?
How are you tracking costs?
Case Study: The Result