-
Performance ManagementMaking Outcome Information Useful
Measuring Results to Improve Technical AssistanceIFC Monitoring and Evaluation Meeting
May 10, 2005
Harry P. HatryThe Urban institute
Washington, DC
-
The Urban Institute2
PROGRAM PERFORMANCEEVALUATIONS MEASUREMENT
Frequency: Irregular Regular, Continuing
Coverage: Done on only Covers mosta few programs programs
Depth of Attempts to get Only tells theInformation: at reasons for score, not WHY
poor performance
Cost High for each study Cost spread outUtility Major long-term Continuous program
Decisions Improvement
-
The Urban Institute3
1. Inputs: Costs, Staff Time
2. Outputs (Amount of Work Completed)
3. OutcomesIntermediate (including quality)End
4. EfficiencyAmount of input per unit of outputAmount of input per unit of
outcome
Types of Performance Information
-
The Urban Institute4
OUTPUTS
1. Number of clients served.
2. Number of recommendations made.
3. Number of training programs held.
OUTCOMES
1. Number of clients whose situation improved.
2a. Number and Percentage of recommendations implemented by businesses.
and2b. Number and percentage of assisted business with improved earnings.
3. Number of trainees who were helped by the program.
Output vs. Outcome Indicators
-
The Urban Institute5
1. Is the performance indicator directly related to the programs goals?
2. Can the values be measured reliably?
3. Can the measurements be obtained in a sufficiently timely way?
4. Can the measurements be made without incurring excessive cost or effort?
5. Is the performance indicator clear, understandable, and fully defined?
6. Does the data set include indicators of possible negative, unintended, effects (e.g., pollution)
7. Does the set of indicators cover inputs, outputs, outcomes, and efficiency?
8. Does the set avoid excessive duplication?
9. Does the set provide a comprehensive picture of performance?
Criteria for Selecting Performance Indicators
-
The Urban Institute6
1. Examination of Project Description
2. IFC Staff (e.g., Focus Groups)
3. Discussions With Customers
4. Other Projects, Donors, Countries
5. Logic Models
6. Common Sense
Sources for IdentifyingWhat Outcomes Should be Tracked
-
The Urban Institute7
Small and Medium Business SupportService Outcome Sequence Chart (Logic Model)
Trainingand
TechnicalAssistance
areProvided
1
Existing BusinessSeek Help
2a
PersonsSeek Help to
Start a Business2b
They Requestand Receive SubstantialAssistance
3a
They Requestand Receive SubstantialAssistance
3b
They TakeAction Based
on Assistance 4a
They TakeAction to Starta Business
4b
BusinessImproves
5a
The Businessis Started
4c
The Businessis Reasonably
Successful 5b
Jobs Created,Especially forLow IncomeHouseholds
6a
Jobs Created,Especially forLow IncomeHouseholds
6b
ReducedPoverty
7
ImprovedQualityof Life
8
-
The Urban Institute8
Small and Medium Business SupportBasic Outcome Indicators
Outreach SuccessBlock 2a: Number of existing businesses who sought assistance. Block 2b: Number of individuals who sought assistance in starting a business.
Clients Take Action Based, At Least in Part, On the AssistanceBlock 4a: Number, and percent, of clients that took action based on the assistance.Block 4b: Number, and percent, of clients that took steps to start a businessBlock 4c: Number, and percent, of clients that succeeded in starting a business.
Actions Lead to Successes Block 5a: Number, and percent, of clients whose business improved as of X months after these
actions occurred..Block 5b: Number, and percent, of clients whose new business appeared to be succeeding as of X
months after start up.
Clients Take Action Based, At Least in Part, On the AssistanceBlock 3a: Number, and percent, of those existing businesses who sought a significant
amount of assistance.Block 3b:Number, and percent, of those individuals who sought a significant amount of
assistance.
-
The Urban Institute9
Small and Medium Business SupportBasic Outcome Indicators
(continued)
Number of Persons in Poverty Reduced Block 7a/7b: Number of employees whose outcomes increased enough to get them out of poverty.
New Jobs Created, Especially for Low Income Households Block 6a: Number of new jobs created related to the action (a) in total; and (b) number of
jobs for low income families.Block 6b: Number of new jobs created related to the action (a) in total; and (b) number of
jobs for low income families.
Number of Persons in Poverty Reduced Block 8a/8b: Number of affected families who report significant improvement in their quality
of life due at least in part to the business.
-
The Urban Institute10
Outcome Sequence Chart (with Indicators) CDP for Financial and Corporate Competitiveness
Outputs Intermediate Outcomes End Outcomes
.
Mediation rules, code of conduct and standard operating manual developed
Communications strategy developed to
build public understanding of out-
of-court mediation 4. Accountants
and auditors begin using
standards and guidelines
7. Standards and information on out-of-court mediation
disseminated to the public via workshops and public awareness campaigns
11. Corporate governance enhanced
10. Speed and quality of
corporate and debt restructuring
improved
TAMC staff trained in business and debt
restructuring
Workshops on accounting
standards and practical guidelines for accountants and
auditors
1. TAMC staff improve their knowledge of
content
5. Regulations implemented and
enforced
8. Transparency in disclosure of financial
and non-financial information
12. Enhanced financial and
corporate competitiveness
2. Standards endorsed by National Judicial
Committee and implemented
Regulations issued to improve corporate
governance of financial institutions
3. Public awareness campaign implemented
9. Reduced number of discrepancies found
6. Institutional capacity of ADRO
enhanced to administer out-of-court mediation of financial disputes
Outcome Sequence Chart CDP for Financial and Corporate Competitiveness
-
The Urban Institute11
Block No.
Outcomes Outcome Indicators
1. TAMC staff improve their knowledge of content
Percent of staff obtaining higher scores on post-learning tests
2. Standards endorsed by National Judicial Committee and implemented
No. of standards endorsed by the National Judicial Committee
3. Public awareness campaign implemented
3.1 No. of TV/radio/newspaper advertisements or educational campaigns in the last 3 months 3.2 Percent of respondents who have seen any advertisements in the last 3 months
4. Accountants and auditors begin using standards and guidelines
Percent of accountants and auditors who state they are using the new guidelines
5. Regulations implemented and enforced
No. and percent of institutions adhering to the new regulations
6. Institutional capacity of ADRO enhanced to administer out-of-court mediation of financial disputes
No. of out-of-court mediations in the last 3 months
7. Standards and information on out-of-court mediation disseminated to the public via workshops and public awareness campaigns
Percent of respondents who are aware of the new standards and information on out-of-court mediation
Table of Indicators
-
The Urban Institute12
Table of Indicators (continued)Block No.
Outcomes Outcome Indicators
8. Transparency in disclosure of financial and non-financial information
8.1 Percent of respondents who say they understand the risks involved in trading with financial institutions 8.2. No. and % of respondents who state that financial information reported by institutions is more user-friendly and understandable 8.3. No. and % of respondents who state that non-financial information reported by institutions is more user-friendly and understandable
9. Reduced number of discrepancies found
No. of discrepancies found in audits
10. Speed and quality of corporate and debt restructuring improved
Percent of debt restructuring that exceeds X days
11. Corporate governance enhanced
11.1 No. and percent of institutions complying with information disclosure 11.2. No. and % of institutions not complying with information disclosure/regulations.
12. Enhanced financial and corporate competitiveness
No. and percent of listed companies showing improvements in SET principles of good corporate governance
-
The Urban Institute13
TYPICAL SERVICE QUALITY CHARACTERISTICS1. TIMELINESS/WAIT TIMES
2. STAFF HELPFULNESS/KNOWLEDGE
3. PLEASANTNESS/FRIENDLINESS
4. CONVENIENCE/ACCESSIBILITY OF LOCATION OF HOURS OF OPERATION CUSTOMER CAN REACH SOMEONE TO TALK TO
5. AWARENESS OF PROGRAM SERVICES
6. CONDITION OF FACILITIES CUSTOMERS USE
7. OVERALL CUSTOMER SATISFACTION
-
The Urban Institute14
1. The Organizations Own Records
2. Another Organizations Records (e.g. unemployment compensation information)
3. Surveys of Assisted Businesses and/or Citizens
4. Trained Observer Ratings (e.g., on-site observations of business progress)
5. Special Equipment (e.g., to measure amount of air and water pollution)
Data Sources
-
The Urban Institute15
1. Provide frequent, timely information to program staff.
2. Set targets and call for explanations for unexpected findings.
3. Disaggregate outcome data by customer and service characteristics.
4. Do regular, basic, analysis of the data, such as comparisons.
5. Hold How Are We Doing? sessions after each performance report.
6. Use Red-Yellow-Green Lights to identify programs/projects needing attention.
7. Require Service Improvement Action Plans for programs/projects with low performance.
8. Have active participation by assisted country/businesses.
Making Performance Information Really Useful
-
The Urban Institute16
Client Characteristics1. New versus Existing Business
2. Type of Business: Agriculture, Manufacturing, Banking, Retail, Services, etc.
3. Size Category Gross Sales (at start of service)
4. Size Category Number of Employees (at start of service)
5. Location: e.g., urban versus rural, district, city, etc.
6. Difficulty: e.g., very, somewhat, or not difficult.
Breakout Categories
-
The Urban Institute17
Service Characteristics
1. Amount of Assistance Provided
2. Office that Provided the Assistance
3. Mode of Service Delivery (especially useful for testing different approaches)
4. Individual Business Consultant
Breakout Categories (continued)
-
The Urban Institute18
Which Hospital Would You Choose?
-
The Urban Institute19
Compare the Latest Data:
1. To previous performance
2. To targets set by the agency
3. Among categories of customers
4. Among geographical areas
5. Among organizational units
6. By type and amount of service
7. To other governments
8. To private sector organizations
Types of Comparison
-
The Urban Institute20
Percent of Clients That Reported The Programs Assistance Had Helped Them Improve Their Business
N
Very or Somewhat
Helpful
Target
Difference (Percentage
Points) All Clients 56 50% 60% -10 Total Annual Sales Below X 23 30% 60% -30 X or Larger 33 64% 60% 4Number of Employees
1-25 10 60% 60% 0 26-50 22 55% 60% -5 51-100 18 44% 60% -16 Over 100 6 33% 60% -27Business Category Agriculture 19 53% 60% -7 Retail 3 67% 60% 7 Services 15 33% 60% -27 Banking 19 58% 60% -2
-
The Urban Institute21
Percent of Clients That Reported The Programs Assistance Had Helped Them Improve Their Business
(continued)
N
Very or Somewhat
Helpful
Target
Difference (Percentage
Points) Number of Days of Assistance
1 2 15 53% 60% -7 3 4 31 55% 60% -5 5 + 10 30% 60% -30Office Office #1 37 49% 60% -11 Office #2 19 53% 60% -7BusinessConsultant A 19 63% 60% 3 B 18 50% 60% -10 C 19 37% 60% -23
-
The Urban Institute22
Action potentially needed?
Encourage agricultural clients to attend the long program, but not services clients. Perhaps use these figures to help convince clients.
Percent of Clients Increasing RevenuesTwelve Months After Service Completion
N
Attended Short Program
Attended Long
Program Total Agriculture 80 25%
75% 63%
Services 180 50% 50% 50% Total 260 46% (120) 61% (140) 54% (260)
-
The Urban Institute23
1. Provide frequent, timely information to program staff.
2. Set targets and call for explanations for unexpected findings.
3. Disaggregate outcome data by customer and service characteristics.
4. Do regular, basic, analysis of the data, such as comparisons.
5. Hold How Are We Doing? sessions after each performance report.
6. Use Red-Yellow-Green Lights to identify programs/projects needing attention.
7. Require Service Improvement Action Plans for programs/projects with low performance.
8. Have active participation by assisted country/businesses.
Making Performance Information Really Useful
-
The Urban Institute24
How often has your water service been interrupted in the last three months?
0
10
20
30
40
50
60
70
Elbasan Korca Lushnja
perc
enta
ge o
f res
pond
ents
1-2 t imes
3-4 t imes
More than4 t imes
Don't know
-
The Urban Institute25
Program________________________
1. Performance Indicator.
2. Category of Indicator.
3. Rationale (What does it measure?)
4. Data Collection Procedure/Source.
5. Frequency of (a) Data Collection; (b) Reporting.
6. Definitions of Any Unclear Terms.
7. What Disaggregations of the P. I. Will be Tabulated?
8. What Comparisons/Benchmarks Will be Provided?
9. Who Is Responsible for the Data and Their Quality
Performance Indicator Specification Sheets