performance management: making outcome information...

Performance ManagementMaking Outcome Information Useful

Measuring Results to Improve Technical AssistanceIFC Monitoring and Evaluation Meeting

May 10, 2005

Harry P. HatryThe Urban institute

Washington, DC

The Urban Institute2

PROGRAM PERFORMANCEEVALUATIONS MEASUREMENT

Frequency: Irregular Regular, Continuing

Coverage: Done on only Covers mosta few programs programs

Depth of Attempts to get Only tells theInformation: at reasons for score, not WHY

poor performance

Cost High for each study Cost spread outUtility Major long-term Continuous program

Decisions Improvement


1. Inputs: Costs, Staff Time

2. Outputs (Amount of Work Completed)

3. OutcomesIntermediate (including quality)End

4. EfficiencyAmount of input per unit of outputAmount of input per unit of

outcome

Types of Performance Information


OUTPUTS

1. Number of clients served.

2. Number of recommendations made.

3. Number of training programs held.

OUTCOMES

1. Number of clients whose situation improved.

2a. Number and Percentage of recommendations implemented by businesses.

and2b. Number and percentage of assisted business with improved earnings.

3. Number of trainees who were helped by the program.

Output vs. Outcome Indicators


1. Is the performance indicator directly related to the programs goals?

2. Can the values be measured reliably?

3. Can the measurements be obtained in a sufficiently timely way?

4. Can the measurements be made without incurring excessive cost or effort?

5. Is the performance indicator clear, understandable, and fully defined?

6. Does the data set include indicators of possible negative, unintended, effects (e.g., pollution)

7. Does the set of indicators cover inputs, outputs, outcomes, and efficiency?

8. Does the set avoid excessive duplication?

9. Does the set provide a comprehensive picture of performance?

Criteria for Selecting Performance Indicators


1. Examination of Project Description

2. IFC Staff (e.g., Focus Groups)

3. Discussions With Customers

4. Other Projects, Donors, Countries

5. Logic Models

6. Common Sense

Sources for IdentifyingWhat Outcomes Should be Tracked


Small and Medium Business SupportService Outcome Sequence Chart (Logic Model)

Trainingand

TechnicalAssistance

areProvided

1

Existing BusinessSeek Help

2a

PersonsSeek Help to

Start a Business2b

They Requestand Receive SubstantialAssistance

3a

They Requestand Receive SubstantialAssistance

3b

They TakeAction Based

on Assistance 4a

They TakeAction to Starta Business

4b

BusinessImproves

5a

The Businessis Started

4c

The Businessis Reasonably

Successful 5b

Jobs Created,Especially forLow IncomeHouseholds

6a

Jobs Created,Especially forLow IncomeHouseholds

6b

ReducedPoverty

7

ImprovedQualityof Life

8


Small and Medium Business SupportBasic Outcome Indicators

Outreach SuccessBlock 2a: Number of existing businesses who sought assistance. Block 2b: Number of individuals who sought assistance in starting a business.

Clients Take Action Based, At Least in Part, On the AssistanceBlock 4a: Number, and percent, of clients that took action based on the assistance.Block 4b: Number, and percent, of clients that took steps to start a businessBlock 4c: Number, and percent, of clients that succeeded in starting a business.

Actions Lead to Successes Block 5a: Number, and percent, of clients whose business improved as of X months after these

actions occurred..Block 5b: Number, and percent, of clients whose new business appeared to be succeeding as of X

months after start up.

Clients Take Action Based, At Least in Part, On the AssistanceBlock 3a: Number, and percent, of those existing businesses who sought a significant

amount of assistance.Block 3b:Number, and percent, of those individuals who sought a significant amount of

assistance.


Small and Medium Business SupportBasic Outcome Indicators

(continued)

Number of Persons in Poverty Reduced Block 7a/7b: Number of employees whose outcomes increased enough to get them out of poverty.

New Jobs Created, Especially for Low Income Households Block 6a: Number of new jobs created related to the action (a) in total; and (b) number of

jobs for low income families.Block 6b: Number of new jobs created related to the action (a) in total; and (b) number of

jobs for low income families.

Number of Persons in Poverty Reduced Block 8a/8b: Number of affected families who report significant improvement in their quality

of life due at least in part to the business.


Outcome Sequence Chart (with Indicators) CDP for Financial and Corporate Competitiveness

Outputs Intermediate Outcomes End Outcomes

.

Mediation rules, code of conduct and standard operating manual developed

Communications strategy developed to

build public understanding of out-

of-court mediation 4. Accountants

and auditors begin using

standards and guidelines

7. Standards and information on out-of-court mediation

disseminated to the public via workshops and public awareness campaigns

11. Corporate governance enhanced

10. Speed and quality of

corporate and debt restructuring

improved

TAMC staff trained in business and debt

restructuring

Workshops on accounting

standards and practical guidelines for accountants and

auditors

1. TAMC staff improve their knowledge of

content

5. Regulations implemented and

enforced

8. Transparency in disclosure of financial

and non-financial information

12. Enhanced financial and

corporate competitiveness

2. Standards endorsed by National Judicial

Committee and implemented

Regulations issued to improve corporate

governance of financial institutions

3. Public awareness campaign implemented

9. Reduced number of discrepancies found

6. Institutional capacity of ADRO

enhanced to administer out-of-court mediation of financial disputes

Outcome Sequence Chart CDP for Financial and Corporate Competitiveness


Block No.

Outcomes Outcome Indicators

1. TAMC staff improve their knowledge of content

Percent of staff obtaining higher scores on post-learning tests

2. Standards endorsed by National Judicial Committee and implemented

No. of standards endorsed by the National Judicial Committee

3. Public awareness campaign implemented

3.1 No. of TV/radio/newspaper advertisements or educational campaigns in the last 3 months 3.2 Percent of respondents who have seen any advertisements in the last 3 months

4. Accountants and auditors begin using standards and guidelines

Percent of accountants and auditors who state they are using the new guidelines

5. Regulations implemented and enforced

No. and percent of institutions adhering to the new regulations

6. Institutional capacity of ADRO enhanced to administer out-of-court mediation of financial disputes

No. of out-of-court mediations in the last 3 months

7. Standards and information on out-of-court mediation disseminated to the public via workshops and public awareness campaigns

Percent of respondents who are aware of the new standards and information on out-of-court mediation

Table of Indicators


Table of Indicators (continued)Block No.

Outcomes Outcome Indicators

8. Transparency in disclosure of financial and non-financial information

8.1 Percent of respondents who say they understand the risks involved in trading with financial institutions 8.2. No. and % of respondents who state that financial information reported by institutions is more user-friendly and understandable 8.3. No. and % of respondents who state that non-financial information reported by institutions is more user-friendly and understandable

9. Reduced number of discrepancies found

No. of discrepancies found in audits

10. Speed and quality of corporate and debt restructuring improved

Percent of debt restructuring that exceeds X days

11. Corporate governance enhanced

11.1 No. and percent of institutions complying with information disclosure 11.2. No. and % of institutions not complying with information disclosure/regulations.

12. Enhanced financial and corporate competitiveness

No. and percent of listed companies showing improvements in SET principles of good corporate governance


TYPICAL SERVICE QUALITY CHARACTERISTICS1. TIMELINESS/WAIT TIMES

2. STAFF HELPFULNESS/KNOWLEDGE

3. PLEASANTNESS/FRIENDLINESS

4. CONVENIENCE/ACCESSIBILITY OF LOCATION OF HOURS OF OPERATION CUSTOMER CAN REACH SOMEONE TO TALK TO

5. AWARENESS OF PROGRAM SERVICES

6. CONDITION OF FACILITIES CUSTOMERS USE

7. OVERALL CUSTOMER SATISFACTION


1. The Organizations Own Records

2. Another Organizations Records (e.g. unemployment compensation information)

3. Surveys of Assisted Businesses and/or Citizens

4. Trained Observer Ratings (e.g., on-site observations of business progress)

5. Special Equipment (e.g., to measure amount of air and water pollution)

Data Sources


1. Provide frequent, timely information to program staff.

2. Set targets and call for explanations for unexpected findings.

3. Disaggregate outcome data by customer and service characteristics.

4. Do regular, basic, analysis of the data, such as comparisons.

5. Hold How Are We Doing? sessions after each performance report.

6. Use Red-Yellow-Green Lights to identify programs/projects needing attention.

7. Require Service Improvement Action Plans for programs/projects with low performance.

8. Have active participation by assisted country/businesses.

Making Performance Information Really Useful


Client Characteristics1. New versus Existing Business

2. Type of Business: Agriculture, Manufacturing, Banking, Retail, Services, etc.

3. Size Category Gross Sales (at start of service)

4. Size Category Number of Employees (at start of service)

5. Location: e.g., urban versus rural, district, city, etc.

6. Difficulty: e.g., very, somewhat, or not difficult.

Breakout Categories


Service Characteristics

1. Amount of Assistance Provided

2. Office that Provided the Assistance

3. Mode of Service Delivery (especially useful for testing different approaches)

4. Individual Business Consultant

Breakout Categories (continued)


Which Hospital Would You Choose?


Compare the Latest Data:

1. To previous performance

2. To targets set by the agency

3. Among categories of customers

4. Among geographical areas

5. Among organizational units

6. By type and amount of service

7. To other governments

8. To private sector organizations

Types of Comparison


Percent of Clients That Reported The Programs Assistance Had Helped Them Improve Their Business

N

Very or Somewhat

Helpful

Target

Difference (Percentage

Points) All Clients 56 50% 60% -10 Total Annual Sales Below X 23 30% 60% -30 X or Larger 33 64% 60% 4Number of Employees

1-25 10 60% 60% 0 26-50 22 55% 60% -5 51-100 18 44% 60% -16 Over 100 6 33% 60% -27Business Category Agriculture 19 53% 60% -7 Retail 3 67% 60% 7 Services 15 33% 60% -27 Banking 19 58% 60% -2


Percent of Clients That Reported The Programs Assistance Had Helped Them Improve Their Business

(continued)

N

Very or Somewhat

Helpful

Target

Difference (Percentage

Points) Number of Days of Assistance

1 2 15 53% 60% -7 3 4 31 55% 60% -5 5 + 10 30% 60% -30Office Office #1 37 49% 60% -11 Office #2 19 53% 60% -7BusinessConsultant A 19 63% 60% 3 B 18 50% 60% -10 C 19 37% 60% -23


Action potentially needed?

Encourage agricultural clients to attend the long program, but not services clients. Perhaps use these figures to help convince clients.

Percent of Clients Increasing RevenuesTwelve Months After Service Completion

N

Attended Short Program

Attended Long

Program Total Agriculture 80 25%

75% 63%

Services 180 50% 50% 50% Total 260 46% (120) 61% (140) 54% (260)


1. Provide frequent, timely information to program staff.

2. Set targets and call for explanations for unexpected findings.

3. Disaggregate outcome data by customer and service characteristics.

4. Do regular, basic, analysis of the data, such as comparisons.

5. Hold How Are We Doing? sessions after each performance report.

6. Use Red-Yellow-Green Lights to identify programs/projects needing attention.

7. Require Service Improvement Action Plans for programs/projects with low performance.

8. Have active participation by assisted country/businesses.

Making Performance Information Really Useful


How often has your water service been interrupted in the last three months?

0

10

20

30

40

50

60

70

Elbasan Korca Lushnja

perc

enta

ge o

f res

pond

ents

1-2 t imes

3-4 t imes

More than4 t imes

Don't know


Program________________________

1. Performance Indicator.

2. Category of Indicator.

3. Rationale (What does it measure?)

4. Data Collection Procedure/Source.

5. Frequency of (a) Data Collection; (b) Reporting.

6. Definitions of Any Unclear Terms.

7. What Disaggregations of the P. I. Will be Tabulated?

8. What Comparisons/Benchmarks Will be Provided?

9. Who Is Responsible for the Data and Their Quality

Performance Indicator Specification Sheets

performance management: making outcome information...

Documents