seminar on performance budgeting and fiscal transparency, tangier, morocco, april 21-23, 2009
Post on 15-Jan-2016
32 Views
Preview:
DESCRIPTION
TRANSCRIPT
Seminar on Performance Budgeting and Fiscal Transparency, Tangier, Morocco, April 21-23, 2009
Session 8. Monitoring and Evaluation:
Challenges and Issues
Nowook Park (npark@kipf.re.kr)Center for Performance Evaluation and Management Korea
Institute of Public Finance
2
Contents
Impact of performance budgeting in Korea1
Roles of the MOF, line ministries, parliament, and audit office
2
3
4 How to motivate performance? How to engage politicians?
5
Performance Information : Indicators and Evaluations
1. Impact of Performance Budgeting in Korea
4
Evaluation results Quality of performance information has not improved much Programs are showing better results
Link between evaluation results and budget Evaluation results are utilized at every stage of budget
process
Moving away from incremental budgeting Evaluated programs are subject to bigger budget change
compared to other programs
Observations from Program Review Process
5
Evaluation Results by Total Score
0
50
100
150
200
250
300
0 10 20 30 40 50 60 70 80 90 100
Total
Frequency
2005 2006 2007
6
Total EffectiveModerately
EffectiveAdequate Ineffective
2005Number 555 28 100 340 87
(%) (100.0) (5.0) (18.0) (61.3) (15.7)
2006Number 577 30 94 388 65
(%) (100.0) (5.2) (16.3) (67.2) (11.3)
2007Number 584 66 139 348 31
(%) (100.0) (11.3) (23.8) (59.6) (5.3)
Evaluation Results by Ratings
7
TotalScore(100)
Planning(30)
Management(20)
Results(50)
Sub total(30)
Design(15)
Performance Planning
(15)
2005 60.1 23.1 13.8 9.3 15.1 21.9
2006 59.9 22.9 14.3 8.6 14.7 22.3
2007 66.0 23.4 14.2 9.2 15.5 27.1
Evaluation Results by Section
8
MOSF encouraged ministries/agencies to use the results in reshuffling budget allocation
MOSF announced at least 10% budget-cut would be done to “ineffective” programs, in principle
MOSF submitted evaluation results to the National Assembly upon their request
Evaluation results are open to public since 2006
Utilization of Evaluation Results
9
-20
24
695%
CI/F
itte
d v
alu
es/a
gencyuse
0 20 40 60 80 100perf
95% CI Fitted values
agencyuse
Use of Performance Information by Agencies (2005)
10
-20
24
695%
CI/F
itte
d v
alu
es/m
pbuse
0 20 40 60 80 100perf
95% CI Fitted values
mpbuse
Use of Performance Information by MOSF (2005)
11
Use of Performance Information by Legislature (2005)
12
Rating
‘05Budget
(A)
‘06 Budget Difference Ratio
Agency(B)
MPB(C)
Final(D)
B-A C-A D-A Ratio_1 Ratio_2 Ratio_3
Effective 15,600 24,948 18,955 22,489 9,348 3,355 6,889 0.52 0.38 0.50
Mod.Effective
92,994 104,335 107,055 105,762 11,342 14,062 12,768 0.32 0.33 0.28
Adequate 208,066 204,473 195,625 201,214 -3,593 -12,441 -6,852 0.10 0.05 0.05
Ineffective 33,081 28,644 29,007 28,505 -4,437 -4,074 -4,576 -0.15 -0.25 -0.19
Total 349,740 362,400 350,642 357,970
( Unit : 100,000won, % )
Link between Evaluation and Budgeting (2005)
13
Rating ‘06
Budget(A)
‘07 Budget Difference Ratio
Agency(B)
MPB(C)
Final(D)
B-A C-A D-A Ratio_1 Ratio_2 Ratio_3
Effective 8,891 9,467 9,337 8,872 575 446 -19 0.11 0.10 0.06
Mod.Effective
33,156 35,701 35,364 35,654 2,545 2,208 2,498 0.12 0.08 0.09
Adequate 297,180 296,769 290,481 289,969 -411 -6,699 -7,211 0.10 0.04 0.03
Ineffective 11,431 6,039 5,400 5,380 -5,392 -6,031 -6,051 -0.15 -0.24 -0.25
Total 350,658 347,975 340,582 339,875
( Unit : 100,000won, % )
Link between Evaluation and Budgeting (2006)
14
Rating ‘07
Budget(A)
‘08 Budget Difference Ratio
Agency(B)
MPB(C)
B-A C-A Ratio_1 Ratio_2
Effective 17,112 18,211 17,503 1,099 391 0.18 0.12
Mod.Effective
266,051 295,121 291,319 29,070 25,268 0.20 0.13
Adequate 146,034 153,026 142,044 6,992 -3,990 0.28 0.15
Ineffective 3,870 3,457 3,066 -414 -805 -0.001 -0.15
Total 433,067 469,815 453,932
( Unit : 100,000won, % )
Link between Evaluation and Budgeting (2007)
15
Programs have been subject to larger budget change after evaluation
Coefficient of Variation in Funding Change (Excluding Programs of which funding change is greater than 200%)
CVYear
(B04-B03)/B03 (B05-B04)/B04 (B06-B05)/B05 (B07-B06)/B06 (B08-B07)/B07
2005 3.1 2.7 9.2
2006 2.7 2.7 -14.3
2007 2.5 3.1 3.9
Moving Away from Incremental Budgeting
16
Budget allocation based on performance is spreading among line ministries Utilization of SABP results in budget requests Changing practices of program management
Evaluation activities become active among line ministries Need for program evaluations are recognized and
accepted among line ministries
Cultural Changes among Line Ministries
2. Roles of the MOF, line ministries, parliament, and audit
office
18
Ministry of Strategy and Finance (MOSF)
Evaluate line ministries’ program Issuing guidelines on performance-based budgeting to line ministries/agencies Evaluating their program’s performance
Use performance information in budget formulation Encouraging line ministries/agencies to use performance evaluation results in preparing their budget requests Incorporating the performance information into its decisions during budget formulation
19
Line ministries/agencies
Producing performance information in compliance with central budget authority’s initiative Submitting strategic plans, annual performance plans
and performance reports Evaluate their programs based on SABP checklist
Conduct program evaluation Use performance information in budget request
20
The National Assembly
Before the enactment of National Finance Act, there had not been official role of the National Assembly They had requested evaluation results for budget deliberation on an ad hoc basis
The National Assembly receives annual performance plan (from 2008) and report (from 2009) with budget documents The National Assembly Budget Office intends to analyze budget in connection with performance information
21
The National Audit Office
Before the enactment of National Finance Act, there had not been official role of the National Audit Office As an independent audit office within the
Administration, it has not played any official role However, they examined the operation of
performance budgeting system and produced report
The National Audit Office is assigned with the role of verifying annual performance report from 2009
3. Performance Information : Indicators and Evaluations
23
Status
Slight increase in outcome measures but there are big room for further improvement Outcome performance measures increase by 3.2 percentage
points, while output performance measures decrease by 4.1 percentage points
More than 40% of programs do not have relevant performance indicators in SABP
Performance IndicatorsPerformance Indicators
TotalType of indicators
Input Process Output Outcome
FY 2007 2,01673
(3.6)140(7.0)
837(41.5)
966(47.9)
FY 2008 2,037111(5.4)
125(6.1)
761(37.4)
1,040(51.1)
24
ExamplesPerformance IndicatorsPerformance Indicators
Good Bad Ugly
Support the IT business
to go abroad
- Customer satisfaction
- Increase the amount of
export($)
Student Loans Program
- The amount of loans- The number of loans
Percentage of attainment- rate of planned goal achievement about employment (x)-rate of employment (o)
Recommendation :Using ‘actual value of attainment’ instead of ‘percentage of achievement’. (FY 2008 manual of KPART)
Recommendation : Using outcome measure ex) Enrollment rates: The postsecondary enrollment gap between low- and high-income high school graduates will decrease each year.
25
Problems
Difficulties to develop outcome measures in some programs Program producing results after long period
• Using milestone indicators may help
• However, using different evaluation cycle needs to be considered
Successful management of program may not change outcome measures
• Reduce scope of outcome measures
• Particular program may not worth evaluating
Data do not exist
• Investment in data is necessary
• Compare cost and benefit of producing new data
Performance IndicatorsPerformance Indicators
26
Lessons
It is desirable to develop outcome measures, but it is not possible for every program
Be aware of some exceptions and apply it
flexibly according to program type
Performance IndicatorsPerformance Indicators
27
Status
Self-Assessment of Budgetary Program Review based on the checklist Among the checklist, there is a question whether program
evaluation is conducted or not It encourage line ministries to conduct program evaluation at
least once in three years
In-depth Program Evaluation In-depth evaluation for selected programs by the central budget
authority
Program EvaluationsProgram Evaluations
28
Status - KPARTProgram EvaluationsProgram Evaluations
• Has your program been evaluated in depth using verifiable data ?• Has your program evaluation been conducted by independent organization ? ex) external institution, audit office, internal evaluation expert, etc.• Does evaluation cover important issues of program ?
Note) ‘yes’ in case that evaluation is in progress or conducted within 3 years latest
YES !YES !
Q. 3-1. Did you conduct comprehensive program evaluation objectively?
29
Problems
Little experience with program evaluation among program managers
Lack of data to prohibits meaningful program evaluation
Lack of fund to conduct program evaluation
Difficulties in maintaining independence of evaluators Trade off between independence and expertise in the
society where social ties are strong and dense
Program EvaluationsProgram Evaluations
30
Example : BadProgram EvaluationsProgram Evaluations
Promotion of the intellectual property (IP)
Obstacle
Purpose
• Measuring the impact of program performance
• Realign the way of investment
• Monitoring the performance management system
IssueDoes the number of IP center affect the number of industrial property ?
Methodology Estimation based on the fixed effect model
• Insufficient Data - absence of data which is needed to conduct the evaluation - using customer satisfaction (or number of counseling) instead of the number of intellectual property in each localities• Lack of understanding on program evaluation among program managers
Result : Couldn’t find the evidence for the regional impact of the IP center !
31
Example : GoodProgram EvaluationsProgram Evaluations
Rural development program
Purpose• Evaluate the impact of the program
• Monitoring the program in progress
IssueDoes the amount of fund improve the condition of the undeveloped-area ?
Methodology Data analysis and the survey
Result of EffectivenessFind the evidence for the regional impact! - There was positive relation between the program’s spending and the infrastructure - In addition, the infrastructure created value-added business
Result of ManagementFind the problems of the management system! - The management system is not developed and the each manager’s autonomy is limited
32
Lessons
Plan ahead for data collection Data should go hand in hand with program management Data management is a part of program management
Be creative in securing fund for evaluation Evaluation should be part of program management In budget negotiation, evaluation plan should be presented
Program EvaluationsProgram Evaluations
4. How to motivate performance? How to engage politicians?
Institutionalized Incentives for line ministries
How performance information will be used in budget preparation is stipulated in guidelines for budget preparation In principle, 10% budget cut for ineffective program is
encouraged to line ministries It draws attention from program managers but not from high
ranking decision makers
Results from SABP is used as a part of evaluation information for the whole department Now high ranking decision makers pay attention But their attention is focused more on getting good scores than
on improving programs’ performance
Forms of Information does Matter
Performance information (performance indicator) at monitoring level has not been able to draw attention from decision makers They are internally useful information, but not utilized by central
budget authority• Remains to be seen how they will be utilized by involved parties
Performance indicators are good starting point for communication rather than decision making
Performance information (program ratings) from program reviews are actively used by central budget authority It summarizes various information into simple ratings Power of Simplicity!
Politician’s Engagement
Although there is no formal mechanism for politician’s engagement, politicians show interest in performance information They requests performance information on an ad hoc basis There has not been much suspicion on the evaluation results
from the Administration so far It remains to be seen whether politicians will use
performance information more systematically with the submission of annual performance plan and report to the National Assembly,
Publicizing performance information may help to force politicians to pay attention to PI
top related