monitoring and evaluation of international development assistance to the private sector
DESCRIPTION
Effective monitoring and evaluation (M&E) systems are essential to learning and accountability. M&E system reviews provides perspective on what is working well, where there are gaps in coverage or weaknesses that need to be addressed, how the M&E information is actually used in decision making, and whether the system is efficient. This session will demonstrate the methods, tools and results in assessing the functioning of the M&E systems of the World Bank Group’s private sector operations in two specialized agencies: the International Finance Corporation and the Multilateral Investment Guarantee Agency.TRANSCRIPT
Monitoring and Evaluation of Private Sector Development
Hiroyuki Hatashima
Private Sector Evaluation, Independent Evaluation Group (IEG)
World Bank Group
Canadian Evaluation Society Conference
June 11, 2013
2
IEG is part of a larger array of evaluation and control mechanisms
World Bank Group President / Management
Management’s Internal Accountability Framework
And Systems• Self Evaluation
• Monitoring (QAG, DOTS, AS)• Institutional Integrity
• Internal Audit• Compliance Advisor/ Ombudsman (IFC,
MIGA)
Independent Evaluation
Group
Inspection Panel
Board ofExecutive Directors
IEG seeks to address both accountability and learning
• Accountability for results:– Is the Bank Group doing what it says? – Is it achieving its objectives? – To what extent is it contributing to development outcomes?– With what results and development impact?
• Learning:– What projects and programs achieve results – relevance,
efficacy, and efficiency – and why?– How can M&E be strengthened in the Bank Group and in
client countries?
3
Three main messages
Evaluation Across Boundaries• Private Sector operations have specific data
gathering advantage/disadvantage based on their business model
• M&E systems should adopt to the business practices to be effective and efficient
• M&E can influence learning, quality of work and outcome
Outline of Presentation
1. Context of Private Sector Development Assistance
2. Types of World Bank Group Private Sector Development interventions
3. Technical Assistance projects M&E
4. Private Sector Investment projects M&E
5. Use and Influence of M&E
1.1 Context: Private Sector As Important Player
1. The private sector has taken on an essential role in contributing a wide range of development issues.
2. Potential of private sector to contribute to development solution is still underutilized.
3. M&E essential for track progress, assess outcome & performance and generating knowledge of what works, what does not, and why.
Are the monitoring and evaluation systems of IFC and MIGA equipped to inform the organizations on their performance and results?
Systems functions
• To what extent does the mechanism in place ensure that M&E systems generate credible, timely, and relevant information?
Use and Influence
• To what extent does M&E information support evidence-based decision making and learning?
Effectiveness and Efficiency
• What has been the impact of the M&E outputs and use on project quality and development outcomes?
1.2 Assessing M&E systems – Evaluation Questions
Outline of Presentation
1. Context of Private Sector Development Assistance
2. Types of World Bank Group Private Sector Development interventions
3. Technical Assistance projects M&E
4. Private sector Investment projects M&E
5. Use and Influence of M&E
The World Bank Group consists of five closely related institutions:• IBRD: The International Bank for
Reconstruction and Development• IDA: The International
Development Association• IFC: International Finance
Corporation• MIGA: The Multilateral Investment
Guarantee Agency• ICSID: The International Center for
the Settlement of Investment Disputes
1. Types of WBG Private Sector Development interventions
1. Interventions through the Public Sector to influence the Private Sector World Band Financial and Private Sector
Development projects, IFC’s Advisory Services projects to government
(Investment Climate, PPP)
2. Direct interventions in partnership with the private sector and public entities operating commercially IFC’s investments and MIGA’s guarantees
3. Blend interventions IFC’s Advisory Services to private companies, but
with a strong public good component
Outline of Presentation
1. Context of Private Sector Development Assistance
2. Types of World Bank Group Private Sector Development interventions
3. Technical Assistance projects M&E
4. Private Sector Investment projects M&E
5. Business Models and M&E
12
3.1. Technical Assistances follow logical model results measurement
ImpactOutcomesOutputsActivities Inputs
Resources used for advisory
services projects
Resources used for advisory
services projects
Programs associated with an
advisory project
Example:
– Provide advisory
services to local companies
Programs associated with an
advisory project
Example:
– Provide advisory
services to local companies
Immediate deliverables of
the project
Example:
– Diagnostic report and
recommendations made
– 10 managers were trained
– Legislation drafted
Immediate deliverables of
the project
Example:
– Diagnostic report and
recommendations made
– 10 managers were trained
– Legislation drafted
Changes in knowledge,
behaviors, and attitudes as a
result of intervention with
attribution of changes to the
project Example:
– Legislative or inst reforms
– Changes in agricultural
techniques, or CG practices
Changes in knowledge,
behaviors, and attitudes as a
result of intervention with
attribution of changes to the
project Example:
– Legislative or inst reforms
– Changes in agricultural
techniques, or CG practices
Consequences resulting from an
intervention including
improvements in clients’
performance and effects beyond
direct beneficiaries
Example:
– increased investment, sales, jobs
Consequences resulting from an
intervention including
improvements in clients’
performance and effects beyond
direct beneficiaries
Example:
– increased investment, sales, jobs
Track indicatorsProject Supervision Report
(PSR)
Supervise
Approve Disburse Completion
Commit
M&E system
Logframe,Identify indicators Project
Completion Report
(PCR)
3.2. M&E follows project cycle
Development Effectiveness FrameworkDevelopment Effectiveness Framework
Strategic Relevance
Was it the right project at the
right time?
Strategic Relevance
Was it the right project at the
right time?
Output Achievement
Were “key” deliverables achieved? To
what extent were clients satisfied
with the advisory service?
Output Achievement
Were “key” deliverables achieved? To
what extent were clients satisfied
with the advisory service?
Outcome Achievement
Were intended short & medium-term changes in
behavior achieved?
Outcome Achievement
Were intended short & medium-term changes in
behavior achieved?
Efficiency
Were costs reasonable vis-à-
vis results?
Efficiency
Were costs reasonable vis-à-
vis results?
Impact Achievement
Were intended longer-term effects achieved?
What would have happened without the intervention? (What is the counterfactual?)
Impact Achievement
Were intended longer-term effects achieved?
What would have happened without the intervention? (What is the counterfactual?)
• These combine into a “synthesis” rating
In addition: Assessment of IFC Role & Contribution: – Was IFC particularly catalytic or innovative in its advisory services?
– Was IFC able to play a unique role not easily filled by others?
• These combine into a “synthesis” rating
In addition: Assessment of IFC Role & Contribution: – Was IFC particularly catalytic or innovative in its advisory services?
– Was IFC able to play a unique role not easily filled by others?
3.2. Advisory Services Project Completion Report as Self-Evaluation
3.4 Technical Assistance M&E –ways to adopt to business practices
Progress Integration with project
cycle M&E officer playing important
and formal roles in quality assurance
Quality control cycle established
M&E officers sign-off, Development Impact Dept reviews
M&E quality improved Quality of logical framework,
baseline data, quality of results discussion – all improved
Challenges Standard indicators not
sufficient to track results Inadequate information in
PCR to justify outcome/impact ratings
Post completion follow-up system not systematic
Work quality assessment not covered in PCR
Outline of Presentation
1. Context of Private Sector Development Assistance
2. Types of World Bank Group Private Sector Development interventions
3. IFC Advisory Services (Technical Assistance) projects M&E
4. Private Sector investment projects M&E
5. Use and Influence of M&E
4.1. Characteristics of Private Sector investments supported by IFC and MIGA
Operations subject to market testDirect Beneficiaries pay for serviceSome degree of competitionSponsors motivated by profitability
Third party effects (TPE) with no market feedbackEconomic distortionsPositive and negative externalitiesProvide rationale for public intervention
Development Agency’s involvementConsistent with market principlesFocused on third party effects (TPE)
Neighbors / Environment
Neighbors / Environment
CustomersCustomersGovernment / Taxpayers
Government / Taxpayers
EmployeesEmployees
4.2 Assessing Development Results - Stakeholder Framework
Competitors
Producers ofComplementary
Products
SuppliersSuppliers
New EntrantsProject
Company
(financiers)
Project
Company
(financiers)
4.3. Performance dimensions of IFC investments and MIGA guarantees
Objective Stakeholders Criteria Standards Assessment
Development Outcome
Financiers Business success(Market test)
Market benchmarks for ROIC
Actual financial statements and objectives at approval
Customers, suppliers, employees, government
Economic sustainability(Third party effect)
Social benchmarks for ERR
Actual financial statements, adjusted by economic prices and values
Neighbors, environment
Environment and social effects (Third party effects)
WBG performance standards
Actual impact and compliance with performance standards
Competitors, new entrants, producers of complementary products
Private sector development (Third party effect)
Objectives on competition, ownership structure, demonstration effects
Qualitative assessment of achievements of objectives on competition, ownership structure, demonstration effects
Track indicators and development outcomes
Supervise
DOTS*
*IFC’s Development Outcome Tracking System
Approve Disburse Repaid / Exit
Commit
Identify indicators
ExpandedSupervision
Report (XPSR)
4.4. M&E to adopt investment project cycle
21
Development Outcome Framework
FinancialPerformance
Financiers
Economic Performance
Gov’t, taxpayersemployees, customers,
suppliers
Envi. & SocialPerformance
Neighbors & Environment
Private SectorDevelopment
Demo. effects, linkages, corp. gov, investment climate
& new entrants
+Standard Development Reach Indicators
IFC-wide: Taxes, employment (incl. female)Plus sector-specific: e.g. MSME loans, utility connections
• Development Outcome: Overall effect on country’s development
• Framework: Derived from Good Practice Standards for Private Sector Evaluation (investments)
• Each with specific indicators, benchmarks
• Development Outcome: Overall effect on country’s development
• Framework: Derived from Good Practice Standards for Private Sector Evaluation (investments)
• Each with specific indicators, benchmarks
3.2. Performance Assessment
2.2 Investment project M&E
Challenges
• Difficult to gather info beyond company”
• Gaps in tracking private sector development outcome
– Missing indicators, discrepancies between stated outcome and indicators
• Reliance on client-submitted data
– Some data based largely on assumptions
• Financial Institutions – challenges in getting sub-project level information
Advantages• Financial data- good quality
and frequently updated;• Enforcement via contracts /
covenants• Integration with project
cycle– Use of standard indicators, with
baseline and target• Quality control cycle
established –Dept. review, Development Impact Dept review, External Assurance for Annual Report
Political Risk Guarantee Business model and M&E challenges (MIGA)
Business Model•Core competency as guarantee underwriting, not financing•Arms-length relationship to project enterprises;•No statutory reporting requirement from projects;•Constraints to access –
–No membership on project company boards of directors
ChallengesIn data gatheringIn communicating with project entitiesChallenges in contextual analysis
Industry / sector intelligence
23
Function of M&E in MIGA: significant progress in intro of self-evaluation and indicators.
Achievements
•New monitoring system for Environment & Social•Mainstreaming self-evaluation•Development indicator tracking system introduced
Areas of improvementChallenges in data gatheringInconsistent E&S follow-up on missing documentsDEIS –ex-ante data not covering all indicators.Pool of evaluation not adequate to report portfolio results
24
Outline of Presentation
1. Context of Private Sector Development Assistance
2. Types of WBG Private Sector Development interventions
3. Technical Assistance projects M&E
4. Private Sector Investment projects M&E
5. Use and influence of M&E
3.1 Lessons can have a positive influence on results
IFC Investment• Lack of lessons /
application of lessons at appraisal associated with poor outcome
– 80% of unsuccessful projects – low quality at entry was main factor
– Low DO with low quality at entry = 40% of them without lessons60% of them lessons were not integrated in design.
• Improved performance by learning from past failures (Agribusiness case study)
– Newer successful projects applied lessons from earlier failed projects
IFC Advisory Lessons influence design
and implementation New template asks “how
lessons have been applied to project design”
Wealth of lessons from PCR Segmented categories of
lessons High use of past project
PCR in new projects Low use of “SmartLessons”
M&E officers review at entry
3.2 Issues have been detected during supervision and actions have been taken
IFC Investment• High quality of supervision
work: 83% high (XPSR)• Half of XPSR identified
issues during supervision, 90% of them were detected, 63% of them were fully addressed during supervision.
• Credit and Environment risk systems as the most frequently used monitoring systems
IFC Advisory PSR widely used
95% of staff used PSR to make adjustment.
88% of supervision issues identified in PCR captured in PSR.
Poor supervision critical factor of 25% of projects rated low in DE
3.3 M&E information is increasingly used in IFC Strategies• Increasing use of M&E data• Annual Strategy process
– Development impact Dept. provide data and participate discussion.
– Guidance: asks development issues to cover
• Ratings and reach data used in strategy formulation– Historical DOTS rating comparisons
between sectors – prioritization of subsector– DOTS ratings vs. profitability– Gap analysis using “reach indicators” –
strategy to sharpen focus, areas to reach (i.e. BOP)
– Development return of different options using reach data (reach per $ cost of capital)
Three main messages
• Private Sector operations have specific data gathering advantage/disadvantage based on their business model
• M&E systems should adopt to the business practices to be effective and efficient
• M&E can influence learning, quality of work and outcome
Outstanding issues
• Difficult to estimate benefit of M&E–Need innovative approach, some
assumptions about counterfactuals;
Monitoring and Evaluation of Private Sector Development
Hiroyuki Hatashima
Private Sector Evaluation, Independent Evaluation Group (IEG)
World Bank Group
Canadian Evaluation Society Conference
June 11, 2013