programme evaluation training slides osvaldo nestor feinstein

Bu proje Avrupa Birliği ve TürkiyeCumhuriyeti tarafından finanse edilmektedir

Programme Evaluation for Strategic Management

Osvaldo Néstor FeinsteinProgram Evaluation WorkshopAnkara, 28-29 November 2013

Bu proje Avrupa Birliği ve Türkiye Cumhuriyeti tarafından finanse

edilmektedir

Themes

1. Programme Evaluation and Strategic Management2. Performance Management & Rating systems 3. Institutionalization of Evaluation in the Public

Sector4. Assumptions Based Programme Evaluation

Framework

2


edilmektedir Programme Evaluation and the Strategic Management Cycle

• Policy Review • Strategic Planning and Selection of

Interventions/strategies • Implementation (delivery of strategies)• Performance reporting /Monitoring• Performance auditing/EVALUATIONWider

PROGRAMME EVALUATION• Policy Review


edilmektedir Programme Evaluation and Effective Governance

• Public programmes are interventions that attempt to achieve results, outputs, outcomes and impact, using public resources.

• Are government interventions achieving their intentions? Are they producing the intended results? Can they be improved?

• Programme Evaluation provides answers to these questions, contributing to effective governance.


edilmektedir Relations between Monitoring and Evaluation

• MONITORING: observing and collecting data on the implementation of a programme, including inputs, outputs and outcomes, & contextual factors

• EVALUATION: on the basis of monitoring and other data, assessing in a systematic way programme effectiveness ( achievement of outputs & outcomes),efficiency, relevance and sustainability

• Evaluation uses monitoring data and evaluation requirements determines which data should be monitored ME & EM


edilmektedir Performance monitoring and reporting and their relation to evaluation

• Performance monitoring is a continuous process of collecting and analyzing data to compare how well a programme or policy is being implemented against expected results

• Performance reporting is an instrument to convey the results of performance monitoring

• Performance monitoring and reporting are inputs for Programme Evaluation, which includes an assessment of programmes.


edilmektedir

Differences with Auditing and Investigation

• AUDITING verifies if resources were allocated for their intended uses. It does not assess objectives, which takes as a given.

• INVESTIGATION assesses the concerns and complaints with respect to the program implementation, for example, in violating rights of minorities or allegations of fraud.


edilmektedir

Key purposes of evaluation

• Evaluations for accountability: to show for which purposes resources were used. Were the programme’s objectives (expected results) achieved? Ex post

• Evaluations for learning: to draw lessons from experience and apply them to improve interventions. What knowledge or lessons can be drawn from the program’s experience? (Janus)


edilmektedir Programme Evaluation as a tool to learn from experience

Looking to the future, with a view to the past

THE PAST THE FUTURE


edilmektedir Program Evaluation and Strategic Decision Making

• Decisions concerning – which programs to close or cut, on the basis of

performance, results achieved or not achieved (accountability function of evaluation)

– which programs should be supported, and/or improved, in light of results achieved and strategic orientations (learning function of evaluation)

• Decisions influenced by political factors and by evidence on results provided by evaluations


edilmektedir Programme evaluation and strategic planning

• Decisions among alternative ways (technologies) to achieve programme’s objectives . Ex-ante use of evaluation (for example, cost benefit analysis)

• Identifying contextual factors that may require programme adaptation

• Improving stakeholder coordination during preparation of strategic plans

11


edilmektedir Program evaluation and public management reform

• Program Evaluation (PE) as an instrument of public management reform, to improve its implementation by drawing lessons from the implementation experience.

• PE provides evidence on effective and ineffective programs so as to guide resource allocation (but beware with mechanical links)

12


edilmektedir

Programs, projects and policies

• Projects: achieving objectives (e.g. reducing rural poverty) through a set of

instruments or components (e.g. microfinance)

• Programmes: set of projects or projects with several components. They include

objectives, resources and activities Programmes to be evaluated could be

interventions included in Institutional Strategic Plans.

• Policies: objectives that are implemented through a set of programmes or projects

• POLICIES PROGRAMMES PROJECTS


edilmektedir SOME EXAMPLES AMONG THE 25 PROGRAMS INCLUDED IN THE 10th NATIONAL DEVELOPMENT PLAN

• PROGRAM ON REDUCING THE INFORMAL ECONOMY• THE COMMERCIALIZATION PROGRAMME IN PRIORITY

TECHNOLOGY AREAS• IMPROVING THE HEALTH TOURISM PROGRAM• TRANSFORMATION PROGRAM FROM

TRANSPORTATION TO LOGISTICS• URBAN TRANSFORMATION PROGRAM ENHANCING

COMPETITIVENESS AND SOCIAL COHESION• INCREASING THE EFFECTIVENESS OF THE LABOUR

MARKET PROGRAM

14


edilmektedir Programme evaluation and the evaluation of the NDP

• Turkey’s 10th National Development Plan indicates that the coordination of monitoring and evaluation of the progress of the Development Plan will be performed by the Monitoring and Direction Committee of the Development Plan, which will be headed by the Undersecretary of the Ministry of Development and comprised of high level administrators from related ministries.

• In the process of planning and implementing new investments, the NDP states that there is a need for enhancing the capacity of public institutions in preparation, implementation, monitoring and evaluation issues

• Discussion question: could it be convenient for strategic management to separate evaluation from monitoring?


edilmektedir Strategic Plans and Programme Evaluation

• Strategic Plans include programmes although sometimes without calling them “programmes” (like the French gentleman who spoke prose without knowing it). “Implicit programmes”. Some examples– SP of the Ministry of Energy, its “Aims” can be evaluated as programmes (ex.

“providing diversity in resources by giving priority to domestic resources)– SP of the Ministry of Justice, its “Objectives” can be evaluated as programmes

• Performance indicators of strategic objectives are usually output- or input level indicators. Outcome indicators are rarely applied. In many cases, performance indicators are measured by the level of required inputs

• For discussion: how could better performance indicators support strategic management?

16


edilmektedir Strategic Plans and Monitoring & Evaluation

• Monitoring and evaluation is one of the least developed components of SPs among the pilot institutions

• Performance Plans do not include any section for the follow up and evaluation of the SP implementation.

• For discussíon: why is it important for strategic management to include an appropriate section dealing with evaluation?

17


edilmektedir

Different types of evaluation

• Ex-ante evaluation (appraisal)• Process evaluation• Mid-term evaluation• Completion evaluation• Ex-post evaluation – Impact evaluations• Formative (to improve) Summative (to judge)

– “when the cook tastes the soup, that's formative; – when the guests taste the soup, that's summative“

M.Scriven

18


edilmektedir Costs and Benefits of different types of evaluation

• Impact evaluations more expensive and less timely for decision making

• Higher benefits of evaluations when their timing is linked to decisions that will have to be made at certain moments or periods

• Choose approaches that will allow to deliver the evaluation in time for decision makingSee “Monitoring and Evaluation: Some Tools, Methods and Approaches”

http://www.worldbank.org/ieg/ecd/tools/


edilmektedir

Public expenditure tracking surveys (PETS)

• Public expenditure tracking surveys (PETS) track the flow of public funds and determine the extent to which resources actually reach the target groups. The surveys examine the manner, quantity, and timing of releases of resources to different levels of government, particularly to the units responsible for the delivery of social services such as health and education (“follow the money”).

• PETS are often implemented as part of larger service delivery and facility surveys which focus on the quality of service.

• Can be used to diagnose problems in service delivery quantitatively, to identify best practices and to provide Providing evidence on delays, “leakage,” and corruption.

• Supports the pursuit of accountability when little financial information is available and improves management by pinpointing bureaucratic bottlenecks in the flow of funds for service delivery.

• http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTSOCIALDEVELOPMENT/EXTPCENG/0,,contentMDK:20507700~pagePK:148956~piPK:216618~theSitePK:410306,00.html for more details, examples and other methodologies

20

http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTSOCIALDEVELOPMENT/EXTPCENG/0,,contentMDK:20507700~pagePK:148956~piPK:216618~theSitePK:410306,00.html




edilmektedir Cost-Benefit and Cost-EffectivenessAnalysis

• Tools for assessing whether or not the costs of a programme can be justified by its results. Cost-benefit analysis measures both inputs and outputs in monetary terms. Cost-effectiveness analysis estimates inputs in monetary terms and outcomes in non-monetary quantitative terms (such as improvements in student reading scores).

• Can be used to inform decisions about the most efficient allocation of resources and/or to identify programmes or projects that offer the highest rate of return on investment.

• Useful for convincing policy-makers and funders that the benefits justify the programme but data for cost-benefit calculations may not be available, and results may be highly dependent on assumptions made.

• http://ec.europa.eu/regional_policy/information/evaluations/guidance_en.cfm

21

http://ec.europa.eu/regional_policy/information/evaluations/guidance_en.cfm



edilmektedir Different evaluation approaches/techniques

• Objectives based evaluation• Goal-free evaluation

• Qualitative methods• Quantitative methods

• Mix of quantitative and qualitative methods• Triangulation

22


edilmektedir Evaluation Criteria and Evaluation Questions

• Relevance: were the objectives consistent with beneficiaries’ requirements, organizational and country priorities & policies?

• Effectiveness: to which extent the program’s objectives were achieved, or are expected to be achieved?

• Efficiency: how economically resources/inputs (funds, expertise, time, etc.) were converted into results? how reasonable were the costs?

23


edilmektedir

Four additional evaluation criteria

• Sustainability: will the results achieved be durable? Is it likely that those results will be sustainable?

• Innovativeness: were innovative approaches introduced by the program?

• Replicability and Scaling-up: to what extent the program’s approaches have been (or are likely to be) replicated and scaled up?

• Equity: how equitable by gender, social group (or other attributes) is the distribution of benefits?

24


edilmektedir

Case study exercise (I)Evaluation of the Northern Someland Rural

Development Program• With support from the XFund, the 10,000,000 euros Northern

Someland Rural Development Program aimed at providing low prized fertilizers to 5,000 farmers in the northern region of Someland, a land-locked country, where small farmers are fruit producers.

• The objective of the program was to improve the welfare of the farmers and their families. A program implementation unit was set up to organize the delivery of fertilizers and to establish links between the farmers and local institutions involved in the provision of complementary services.

• How would you evaluate this program? What would you do to evaluate the program? Please discuss with your colleagues and prepare a 5 minutes presentation

25


edilmektedir Alternative approaches to program evaluation

• Goal-free evaluation – Its strength in avoiding a limited focus on objectives can

be compensated by objectives-based evaluation with an explicit focus on “unintended consequences”

– Weak on accountability.• Impact evaluation with control groups

– Explicit counterfactual to model the without program situation

26


edilmektedir

Counterfactual in evaluation

• What would have happened WITHOUT the program? (had there been no program)

• Note that the program is “factual”, it is a fact, as a program was implemented, whereas the counterfactual is a situation that is not observed, it has to be reconstructed.

• Explicit and implicit counterfactuals, not theoretical but practical importance, rationale for the program.

27


edilmektedir Comparisons Before/ After With/Without

Without With

Before 100 100

After 150 120

Look first only at the “with” column

28


edilmektedir Comparisons Before/ After With/Without

Without With

Before 100 100

After 80 100

Look first only at the “with” column


edilmektedir

Establishing the “Before” & “Without”

• “Before” , or at the beginning of the program. Need for a BASELINE, which could be part of the diagnosis. Sometimes available census and/or household surveys can be used.

• “Without”, requires a comparison group; “control group” is a particular case. Statistical methods. Comparative analysis.

30


edilmektedir The “General Elimination Method” GEM

• Simple and powerful, introduced by Michael Scriven in 2008

• Two key steps:1. consider the set of “possible plausible causes” (including components of the contexts and/or other programs or policies)2. eliminate the less plausible possible causes

• See application and references in p.3 of • http://www.undp.org/evaluation/documents/ADR/ADR_

Reports/Chile/ADR-Chile.pdf

31

http://www.undp.org/evaluation/documents/ADR/ADR_Reports/Chile/ADR-Chile.pdf

http://www.undp.org/evaluation/documents/ADR/ADR_Reports/Chile/ADR-Chile.pdf


edilmektedir How to manage and conduct program evaluations

• The first strategic decision is choosing WHAT to evaluate. Decide (if you have degrees of freedom) on the basis of expected value added for accountability and/or learning

• The terms of reference should clarify expectations concerning the evaluation that will be carried out. Answers to WHY, HOW, and WHEN TO EVALUATE?– See “Writing Terms of Reference for an Evaluation: A How-To

Guide”– https://ieg.worldbankgroup.org/evaluations/evaluation-capacity-

development-ecd

32

https://ieg.worldbankgroup.org/evaluations/evaluation-capacity-development-ecd

https://ieg.worldbankgroup.org/evaluations/evaluation-capacity-development-ecd


edilmektedir Which type of evaluation should be used?

• Selecting the type(s) of evaluation to be used• Methods adequate to the circumstances (purpose,

availability of time and resources). Not only “fit-to-purpose”. Pragmatic

• See “Monitoring and Evaluation: Some Tools, Methods and Approaches”

• http://www.worldbank.org/ieg/ecd/tools

33


edilmektedir

Consultation with stakeholders

• Consulting with stakeholders– Why? To get their insights, and– to nurture their interest in the evaluation (which will

facilitate the use of the evaluation).• Identifying key stakeholders:

– Partners, intended beneficiaries, government officers, civil society representatives

34


edilmektedir

Reviewing relevant documentation

• In addition to PROGRAMME DOCUMENTS, reports on the country, internet searches for thesis. other evaluations and papers on the project/program area and on the instruments used (e.g., rural microfinance or agricultural extension in the area)

35


edilmektedir

APPLYING Evaluation Criteria

• Applying evaluation criteria– Relevance, Effectiveness, Efficiency, Sustainability,

Innovativeness, Replicability and Scaling-up, Performance of partners

– Ratings for the criteria– Rating overall project achievement– Provide a justification for the ratings

36


edilmektedir

Use of evaluation reports

• Using evaluation reports– Evaluation reports are important INPUTS for

evaluations, as indicated before under “reviewing documentation”.

– Evaluation reports are also key OUTPUTS of the evaluation process. Show in a clear way the links between findings and recommendations to facilitate the use of the evaluation report.

37


edilmektedir

Case studies

• For in-depth understanding of how a program has operated in a specific context

• Problem of eventual lack of representativeness• Typology of contexts/situations, and case studies

corresponding to the different contexts/situations.• Combining case studies with surveys

38


edilmektedir

Use of surveys

• Search for available surveys • If surveys are needed and are not available, ensure

that professionals with the required expertise are involved; possibility of subcontracting

• The methodology ACTUALLY used should be described in detail (for replication & interpretation)

• Pilot trials are important

39


edilmektedir

Semi-structured interviews

• Identify key informants corresponding to different perspectives

• Elaborate a brief interview guide• Team should be opened to comments, opinions and

judgements not considered in the guide• Systematize the results of the interviews

40


edilmektedir

Case study exercise (II)Evaluation of the Youth Employment Program for the Eastern

Region

• Given the importance of youth unemployment a program to increase the employability of the young was designed and implemented in the Eastern region of the country. The program included training in agricultural activities, as well as training courses for the production of furniture, which is an important economic activity in that region. In addition there were also training courses on informatics. An evaluation team was tasked to evaluate the program.

• Discuss with your colleagues how you would conduct the evaluation of the program. Prepare a brief presentation with the conclusions of your discussion.

41


edilmektedir

e-reference materials

• Glossary of Key Terms in Evaluation and Results Based Management – http://www.oecd.org/dac/evaluation/evaluationandaideffectivenessseri

es.htm

– TURKISH VERSION:– http://www.oecd.org/dac/evaluation/deerlendirmevesonuodaklyonetim

deanahtarterimlersozluu.htm

– Monitoring and Evaluation: Some Tools, Methods and Approaches• http://www.worldbank.org/ieg/ecd/tools • Impact Evaluation in Practice (2011) Eng/Fr/Sp.

– http://siteresources.worldbank.org/EXTHDOFFICE/Resources/5485726-1295455628620/ Impact_Evaluation_in_Practice.pdf

42

http://www.oecd.org/dac/evaluation/evaluationandaideffectivenessseries.htm

http://www.oecd.org/dac/evaluation/evaluationandaideffectivenessseries.htm

http://www.oecd.org/dac/evaluation/deerlendirmevesonuodaklyonetimdeanahtarterimlersozluu.htm

http://www.oecd.org/dac/evaluation/deerlendirmevesonuodaklyonetimdeanahtarterimlersozluu.htm

http://siteresources.worldbank.org/EXTEVACAPDEV/Resources/4585672-1251481378590/MandE_tools_methods_approaches.pdf

http://siteresources.worldbank.org/EXTHDOFFICE/Resources/5485726-1295455628620/Impact_Evaluation_in_Practice.pdf

http://siteresources.worldbank.org/EXTHDOFFICE/Resources/5485726-1295455628620/Impact_Evaluation_in_Practice.pdf


edilmektedir

E-evaluation guides from the EC

• On the evaluation of innovations• http://ec.europa.eu/regional_policy/sources/docgen

er/evaluation/pdf/eval2007/innovation_activities/inno_activities_guidance_en.pdf.

• On impact assessment http://ec.europa.eu/governance/impact/commission_guidelines/commission_guidelines_en.htm

• Several guides can be accessed at• http://ec.europa.eu/regional_policy/information/eval

uations/guidance_en.cfm43

http://ec.europa.eu/regional_policy/sources/docgener/evaluation/pdf/eval2007/innovation_activities/inno_activities_guidance_en.pdf



http://ec.europa.eu/governance/impact/commission_guidelines/commission_guidelines_en.htm

http://ec.europa.eu/governance/impact/commission_guidelines/commission_guidelines_en.htm




edilmektedir

Performance Management

• Performance Management, • Performance Measurement, • Performance Monitoring • Measure to Monitor & Monitor to Manage• Two frequent (and partially deceiving) quotes:

– “What gets measured gets done” Tom Peters– “What gets measured gets managed” Drucker

• Measurement facilitates Management

44


edilmektedir Performance measurement and management

• Performance measurement– What is happening with the programs?– How are the programs performing? – Are programs achieving their expected results? (the

“why” question answered by evaluations)• Performance management

– making decisions about programs and project portfolio: allocation of additional resources, stopping programs, lessons for new programs, etc.

45


edilmektedir

PM in practice & Exercise III

• What should be measured and managed?• Comparing what is actually measured with what should be

measured, which are the expected results? • The results chain: inputs--(processes)outputsoutcomes impacts

• Exercise III: QUESTIONS FOR DEBATE:• Are inputs “results” in the results chain? And outputs?• What are the implications for accountability of different

definitions of results?

46


edilmektedir

Reviewing a portfolio of programs

• Identification of “PM gaps” (expected vs actual)

• In which (type of ) programs is performance higher (lower) than expected?

• What worked? What did not work? • Why? (an important evaluation question)• What influence had the context ?• Any patterns?

47


edilmektedir

Purposes of PM

• Accountability and learning (improvement)

• Whose accountability?

• Whose learning?

• Complementarity or trade-offs?

48


edilmektedir

Lessons on PM Systems (PMS)

• Importance of ownership of the PMS• WHO are the intended users of PMS?

• Convenience of involving intended users in the design and/or redesign of the PMS

• Opportunities created by PMS to facilitate coordination and enhance effectiveness

49


edilmektedir

Lessons on PM Risks

• Nonuse and/or misuse (GAO’s findings of nonuse of performance information to manage; commitment by leaders and communication of that commitment is crucial)

• Mismeasuring and/or not measuring, not counting what counts, leaving out what is important

• Inducing short-termism• Bureaucratization, with PM being perceived as a

mechanical exercise, as a burden

50


edilmektedir

Issues of PM

• Participation of intended PM users in the review of the system

• Programs, projects and portfolio, the trees and the forest

• Partial achievement: PM when reporting period does not correspond to “gestation period”. Actual costs and likely benefits.

51


edilmektedir

Issue of Partial Achievement

Year Output (reported in year…)

Cost $

Unitary cost $/A

Revised

2011 2 A 10,000 5,000

2012 2 A 6,000 3,000 6,000A:2-2x.5

Total 4 A 16,000 4,000

52


edilmektedir

Issues of PM

• PM and its use for resource allocation, performance informed budgeting

• Quality control of data• Performance Measurement & Management• Links of PM with the M&E system• PM, comparisons and benchmarking (with

baselines, targets, other organizations)

53


edilmektedir

Rating Systems

• Trending theme: S&Poor• Why ratings?• Experience of WB & IFAD - ECG• Validation of self-evaluation ratings, • Evaluation of ratings, rating the raters• Contestability (dealing with disagreements)

54


edilmektedir

Program or Project Ratings

• Relevance of anticipated outcomes• Effectiveness (achievement of outcomes)and• Efficiency on a six-point rating scale• Highly Satisfactory, Satisfactory, Moderately Satisfactory.• Moderately Unsatisfactory, Unsatisfactory., Highly Unsat.• (risks to) sustainability of outcomes based on an

assessment of four dimensions: financial, sociopolitical, institutional framework & governance, and environmental

55


edilmektedir A six- point rating system for the evaluation criteria

• A six-point rating system for the evaluation criteria as a way to quantify qualitative judgements of evaluators– 6: highly satisfactory 5: Satisfactory– 4: moderately satisfactory– ------------------------------------------------– 3: moderately unsatisfactory– 2: unsatisfatory 1: highly unsatisfactory

56


edilmektedir Ratings for innovation, scaling-up and partners’ performance

• Good practice of IFAD’s evaluation office providing explicit ratings for the creativity, innovation and up-scaling potential of innovative projects and distinguishing between agency and partners’ performance ratings and outcome ratings

• Promotion of pro-poor innovation,replication and scaling up: The extent to which IFAD development interventions have: (i) introduced innovative approaches to rural poverty reduction; and (ii) the extent to which these interventions have been (or are likely to be) replicated and scaled up by government authorities, donor organizations,the private sector and others agencies.

• Performance of partners: assess the contribution of partners to project design, execution, monitoring and reporting, supervision and implementation support, and evaluation. The performance of each partner will be assessed on an individual basis with a view to the partner’s expected role and responsibility in the project life cycle.

57


edilmektedir Remembering the performance evaluation criteria

• Relevance : The extent to which the objectives of a program are consistent with beneficiaries’ requirements and country needs, institutional priorities and policies.

• Effectiveness: The extent to which the program’s objectives were achieved, or are expected to be achieved, taking into account their relative importance.

• Efficiency: A measure of how economically resources/inputs (funds, expertise, time, etc.) are converted into results.

58


edilmektedir

Clarifications

• A satisfactory outcome means that relevant project objectives are expected to be achieved with no, minor or moderate shortcomings at the time of evaluation.

• A likely sustainability rating means that the project is considered likely to generate continued benefits after project implementation - with no or only moderate risks.

• (definition of sustainability: The likely continuation of net benefits from a program beyond the phase of external funding support. It also includes an assessment of the likelihood that actual and anticipated results will be resilient to risks beyond the program’s life).

59


edilmektedir

SMART criteria for indicators

• Specific relating results to the achievement of one and only one objective.

• Measurable. all parties agree on what indicators cover and on practical ways to measure them

• Achievable feasible targets• Realistic levels of performance that are ambitious

but feasible..• Time-Bound, not open ended but with dates

60


edilmektedir

Use (and Misuse) of ratings

• As an input, but not the only input, for decision-making, for performance management

• For comparisons, benchmarking• For assessing and enhancing learning & quality • For aggregation of information

61


edilmektedir

Issues concerning Ratings

• Identifying critical dimensionsapplying “Occam’s razor”

d1, d2, d3, …., dn• Weighting the dimensions w1d1, w2d2, w3d3• Partial orders limits to aggregation: partial comparisons (Amartya Sen); • Transparency (avoiding “secret algorithms”)

62


edilmektedir

Synthesis• Performance management

1. Measurement, availability of performance information is not enough (possible nder-use, non-use and misuse);

2. Focus on expected results, identify PM gaps and patterns, present data with a brief narrative

3. Involve intended users in (re)design of PMS

• Rating systems1. Different approaches to ratings2. Validation of ratings3. Transparency of ratings and benchmarking

• Links between performance management and rating systems1. Ratings provide management with comparative data on performance that can be used

for aggregation, learning and accountability

63


edilmektedir Exercise IV: Case study example Performance Measurement System for a Health Program

• A country wide health program was launched with the objective of contributing to improve primary health care of the population in urban and rural areas. Different types of health interventions were considered. You have been asked to develop a performance measurement system for such a program.

• Discuss with your colleagues and prepare with your group a 5 minutes presentation.

64


edilmektedir

Learning from others

• World Bank Independent Evaluation Group (IEG)• http://ieg.worldbankgroup.org/content/ieg/en/home.

html• Project perf.assessment reports(PPAR); M&E• http://www.ifad.org/evaluation/process_methodolog

y/doc/manual.pdf• http://www.thegef.org/gef/eo_office• https://wpqr1.adb.org/LotusQuickr/ecg/Main.nsf/$de

faultview/0906CAABDA17E70A48257731002A0656/$File/PublicSectorGPS-revised.pdf?OpenElement

• (GPS Evaluation Cooperation Group)

65

http://ieg.worldbankgroup.org/content/ieg/en/home.html

http://ieg.worldbankgroup.org/content/ieg/en/home.html

http://www.ifad.org/evaluation/process_methodology/doc/manual.pdf

http://www.ifad.org/evaluation/process_methodology/doc/manual.pdf

http://www.thegef.org/gef/eo_office

https://wpqr1.adb.org/LotusQuickr/ecg/Main.nsf/$defaultview/0906CAABDA17E70A48257731002A0656/$File/PublicSectorGPS-revised.pdf?OpenElement




edilmektedir

Additional reference materials

• DAC/IEG Sourcebook for evaluating global and regional programs http://web.worldbank.org/WBSITE/EXTERNAL/EXTOED/EXTGLOREGPARPROG/0,,contentMDK:21178261~menuPK:4426473~pagePK:64829573~piPK:64829550~theSitePK:4426313,00.html

• Program Assessment Rating Tool (PART) US Gov.http://web.worldbank.org/WBSITE/EXTERNAL/EXTOED/EXTEVACAPDEV/0,,contentMDK:22321269~menuPK:4585748~pagePK:64829573~piPK:64829550~theSitePK:4585673,00.html Note: PART has been discontinued but it’s worthwhile

• A critique of the PART in http://www.businessofgovernment.org/report/performance-management-recommendations-new-administration

• Up-to-date information on the US gov. case: http://www.businessofgovernment.org/blog/business-government/omb-unveils-its-performancegov-website

66

http://web.worldbank.org/WBSITE/EXTERNAL/EXTOED/EXTGLOREGPARPROG/0,,contentMDK:21178261~menuPK:4426473~pagePK:64829573~piPK:64829550~theSitePK:4426313,00.html



http://web.worldbank.org/WBSITE/EXTERNAL/EXTOED/EXTEVACAPDEV/0,,contentMDK:22321269~menuPK:4585748~pagePK:64829573~piPK:64829550~theSitePK:4585673,00.html









http://www.businessofgovernment.org/blog/business-government/omb-unveils-its-performancegov-website

http://www.businessofgovernment.org/blog/business-government/omb-unveils-its-performancegov-website


edilmektedir

And some other useful references

• US (speech by OMB A.Director) http://www.state.gov/s/d/rm/rls/rm/2010/147424.htm

• http://web.hbr.org/email/archive/managementtip.php?date=060611 (brief note)

• UNEG Good Practice Guidelines for Follow up to Evaluations• http://www.uneval.org/papersandpubs/documentdetail.jsp?

doc_id=610• http://www.eipa.eu/files/repository/eipascope/20101022101218_Eipas

cope_2010_2_Article2.pdf

• Patria de Lancer Julnes et.al. (2008) “International Handbook of Practice-Based Performance Management” Sage

• http://www.sustainability.com/library/rate-the-raters-phase-three67

http://www.state.gov/s/d/rm/rls/rm/2010/147424.htm

http://web.hbr.org/email/archive/managementtip.php?date=060611

http://www.eipa.eu/files/repository/eipascope/20101022101218_Eipascope_2010_2_Article2.pdf

http://www.eipa.eu/files/repository/eipascope/20101022101218_Eipascope_2010_2_Article2.pdf


edilmektedir Institutionalization of Evaluation in the Public Sector - Demand

• Demand for evaluation – from the Executive (Prime Minister Office, Ministry of

Development, MOF, Other M.)– from the Legislative , Parliament, from Civil Society

• Types of demand:– Actual demand – Potential demand (there is an interest on evaluation but

funds are lacking)– Latent demand (there is no awareness of the role of

evaluation but an interest on what evaluation can provide)

68


edilmektedir Institutionalization of Evaluation in the Public Sector - Supply

• Actual Supply of Evaluation– Public sector managing the evaluations– Public sector conducting evaluations– Public sector outsourcing evaluations

• Potential Supply of Evaluations– (organizations or professionals so far not involved in

evaluations but with capacities which can be used for evaluation)

• Private consultants, universities, think tanks conducting evaluations

69


edilmektedir Institutionalization of Program Evaluation in the Public Sector

• Who would lead a program evaluation initiative in the public sector?

• A central Ministry, like the Ministry of Development in Turkey, the Ministry of Planning or of Finance in other countries

• Who will manage, conduct and use program evaluations? Sector Ministries (Health, Education, Labor…)

70


edilmektedir International Experiences in Program Evaluation

• Frequently the experiences of New Zealand, USA, Canada, the Nordic countries and the UK are highlighted. We will make reference to another OECD case more relevant for Turkey: the case of MEXICO and some other relevant experiences

• Mexico has developed a well functioning system of evaluation of social programs, introducing some innovations that are worthwhile to take into account

71


edilmektedir The Mexican Experience on social program evaluation

• The Mexican system for social program evaluation is described in http://www.coneval.gob.mx/Evaluacion/Paginas/Evaluation-and-monitoring-en.aspx

• The National Council for the Evaluation of Social Development Policy (CONEVAL), as set forth in the General Law of Social Development, has the aim to regulate and coordinate the evaluation of the Social Development Policy and Programs implemented by public dependencies.

72

http://www.coneval.gob.mx/Evaluacion/Paginas/Evaluation-and-monitoring-en.aspx

http://www.coneval.gob.mx/Evaluacion/Paginas/Evaluation-and-monitoring-en.aspx


edilmektedir More on the Mexican Evaluation Experience

• Mexico has also been developing evaluation systems at the subnational level, with support from CONEVAL, and the capital city of Mexico developed an evaluation system on its own.

• For experiences of institutionalization of impact evaluation see www.worldbank.org/ieg/ecd/docs/inst_ie_framework_me.pdf

73


edilmektedir

The experience of Austria

• From a recent diagnosis of Austrian Public Administration: input rather than output (or outcome). Concern with who gets how much, instead of what has to be the outcome

• Shift in 2009 towards an outcome orientation. Managing public adminstration based on contributions to achieve outcomes (performance management)

74


edilmektedir

The experience of Malaysia

• Malaysia public sector management reform shaped by best practices

• Framing a reform as ‘best practice’ was a way of neutralizing potential opposition from the very different and sometimes antagonistic stakeholders that the reform had to satisfy

• Successive reforms have built on previous ones: past failures have been the paradoxical foundation for present success

75


edilmektedir

More on the Malaysian experience

• The current prime minister, or his deputy, has individual meetings with the ministers responsible for the seven national key result areas every month, and with the rest of the Cabinet every six months,to review their performance against key performance indicators (KPIs) set by the prime minister himself

• Need to emphasize the importance of the implementation stage of reform, and of an appropriate diagnosis

76


edilmektedir The evaluation of Turkey’s Vocational Training Programs

• The evaluation of Turkey’s Vocational Training Programs sought to assess the effectiveness of improving vocational training programs in the context of the rapid expansion of these programs in Turkey.

• Concept Note, Baseline Report and • Evaluation Report available at• http://web.worldbank.org/WBSITE/EXTERNAL/

EXTABOUTUS/ORGANIZATION/EXTHDNETWORK/EXTHDOFFICE/0,,contentMDK:23177755~pagePK:64168445~piPK:64168309~theSitePK:5485727,00.html

77

http://siteresources.worldbank.org/EXTHDOFFICE/Resources/5485726-1334870720628/8597105-1334870749247/Turkey_Concept_Note.docx

http://siteresources.worldbank.org/EXTHDOFFICE/Resources/5485726-1334870720628/8597105-1334870749247/Turkey_Baseline_Report.docx

http://siteresources.worldbank.org/EXTHDOFFICE/Resources/5485726-1334870720628/8597105-1334870749247/Turkey_Evaluation_Report.docx


edilmektedir Exercise V: Debate on the Institutionalization of Evaluation

• Discuss the institutionalization of evaluation in the public sector of Turkey and rate demand for and supply of evaluation, as well as the links between demand and supply of evaluation, and the use of evaluation on:– a four point rating scale, with 1– as the lowest rating and 4 the highest.

• Demand: Supply: Links: Use:

78


edilmektedir Summing-up: 12 steps to conduct program evaluation

1. Identify the program’s objective (s)2. Become familiar with the modalities of implementation3. Identify the basic assumptions (the logic of the

intervention)4. Decide which will be the key indicators (could be

those included in the program, eventually adding some more)

5. Search and read bibliography and review the first four steps in light of your reading

79


edilmektedir Steps for conducting program evaluation (continued)

6. Search for and review available information sources.

7. Determine needs of additional information, if any8. Decide which methods/techniques to use9. Apply the evaluative criteria

80


edilmektedir Steps for conducting program evaluation (final)

10. Be open to the possibility of unintended consequences, positive and/or negative

11. Analyse and synthesize results by implementation modality and by area of implementation, identifying best and worst

12. Triangulate results (cross-checking different information sources)

81


edilmektedir

The end?

“This is not the end, nor the beginning of the end, but the end of the beginning” - Winston Churchill

82


edilmektedir

Thank you!

teşekkür ederim

[email protected]

83