Approach to GEF IW SCS Approach to GEF IW SCS Impact EvaluationImpact Evaluation

Aaron ZazuetaAaron Zazueta

Reference Group MeetingReference Group Meeting

Bangkok, ThailandBangkok, Thailand

September 27, 2010September 27, 2010

OverviewOverview

2

Contents

1. Overall evaluation approach

GEF’s Approach to International Waters

Objectives of the Evaluation

Impact Evaluation Approach

Introduction to “theory of change” Approach

South China Sea BackgroundSouth China Sea Background

• Known for its rich biodiversity and natural Known for its rich biodiversity and natural resources, the South China Sea (SCS) resources, the South China Sea (SCS) provides food to 250 million people and provides food to 250 million people and directly provides a livelihood for over 100 directly provides a livelihood for over 100 million people.million people.

• 40 years of rapid economic growth have 40 years of rapid economic growth have had results that threaten the sustainability had results that threaten the sustainability of the social, economic, and ecological of the social, economic, and ecological services that the SCS providesservices that the SCS provides– Growing coastal habitat destructionGrowing coastal habitat destruction– Increased pollutionIncreased pollution– Increased overfishingIncreased overfishing

• Management of the SCS’s rich marine Management of the SCS’s rich marine resources is complicated by outstanding resources is complicated by outstanding territorial disputes.territorial disputes.3

GEF support in the SCS+ GEF support in the SCS+ • Since 1993, the GEF has allocated over Since 1993, the GEF has allocated over

$180mn USD to projects related to the $180mn USD to projects related to the SCS and surrounding areasSCS and surrounding areas

4

IW Global Projects

8 Projects$ -- M

IW Regional & National Projects

22 Projects$ 172M

Other Focal Projects

11 Projects$ 42M

SCS4 Proj

Others8 Proj

IF6 Proj

PEMSEA4 Proj

GEF Approach to International GEF Approach to International WatersWaters

• The GEF helps countries work together to secure The GEF helps countries work together to secure environmental benefits from shared surface water, environmental benefits from shared surface water, ground water, and marine ecosystems by fostering ground water, and marine ecosystems by fostering international cooperation and catalyzing action international cooperation and catalyzing action on priority transboundary water issueson priority transboundary water issues..

• The GEF has developed an International Waters The GEF has developed an International Waters approach that involves:approach that involves:

• Foundational Activities: Foundational Activities:

Build trust and confidence among countries, strengthen knowledge base on root causes, improve national capacity, and strengthen regional coordination mechanisms

• Demonstration ActivitiesDemonstration Activities

Develop, test, and adapt approaches and technologies; catalyze further action to address problems

• InvestmentsInvestmentsReplicate, upscale, and/or mainstream technologies and approaches that work5

GEF Model to International WatersGEF Model to International Waters

6

Foundational

Demonstration Investment/Mainstreaming/Upscaling

Countries establish national inter-ministerial committees to address IW transboundary concerns

Mechanisms for knowledge sharing and facilitating public participation

Build individual, organizational and institutional capacities of decision makers

Identify transboundary concerns and root causes (TDA) Countries establish interim mechanisms for cooperation

Regional and national institutions and mechanisms coordinate IW policy and monitoring efforts

Demonstration and piloting of approaches with involvement of appropriate stakeholders to address priority transboundary IW concerns

Private sector engagement Multi-county agreement

on commitments to regional mechanism and national actions through a Strategic Action Program

Mechanisms in place that provide timely and reliable information to monitor stress and status of the water body and provide a better understanding of the water body’s evolving challenges

Change in behavior leads to continued and sustained action on transboundary concerns, as well as self-sustaining policy actions, public and private sector investment, monitoring, and public involvement

Impact: reduced stress on international waters

Impact: Water/ environment/ socioeconomic status eventually improved

Replication, knowledge sharing, and upscaling of demonstrations to the appropriate scales and by the relevant stakeholders to change behavior

Actions taken at the appropriate scale and w/ involvement of relevant stakeholders (various government levels and sectors, private sector and public) to address priority concerns

In practiceIn practice

• The application of the model is The application of the model is shaped by contextshaped by context

• The sequence of interventions The sequence of interventions may varymay vary

• A limitation: The does not fully A limitation: The does not fully capture opportunistic capture opportunistic investments that might be investments that might be undertaken by following the undertaken by following the model. model.

7

Assumptions behind GEF IW ApproachAssumptions behind GEF IW Approach

• A strategic , programmatic approach of investments A strategic , programmatic approach of investments and countries’ coordinated action and countries’ coordinated action will lead to will lead to improvements improvements in in water/environment/socioeconomic statuswater/environment/socioeconomic status

• The promoted measures and solutions are well The promoted measures and solutions are well aligned aligned with both the with both the regional priorities regional priorities and the and the national priorities national priorities of the participating countries of the participating countries

• The “architecture” promoted will enableThe “architecture” promoted will enable adaptive adaptive management processesmanagement processes that will lead to the policy that will lead to the policy changes and investments needed to reduce changes and investments needed to reduce environmental stress and improve environmental environmental stress and improve environmental status. status.

• Improvement of environmental and socioeconomic Improvement of environmental and socioeconomic status will status will take place over time take place over time as long as as long as countries and other actors take countries and other actors take follow up actions follow up actions and investments and investments to address the transboundary IW to address the transboundary IW concerns concerns

• Water body impacts depend on Water body impacts depend on level of effort level of effort in in stress reduction and may be appropriate only for stress reduction and may be appropriate only for larger, programmatic approacheslarger, programmatic approaches

8

Objectives of the EvaluationObjectives of the Evaluation• Main objective: Main objective: Assess the extent Assess the extent to to

which processes, knowledge, which processes, knowledge, technologies, and capacities to which technologies, and capacities to which GEF contributes promote changes in GEF contributes promote changes in policies, technology, management policies, technology, management practices, and other behaviors that will practices, and other behaviors that will address the address the priority transboundary priority transboundary environmental concerns environmental concerns that affect that affect the social, economic, and the social, economic, and environmental services of the SCS.environmental services of the SCS.

• Evaluation will answer four main Evaluation will answer four main questions:questions:– Has support been Has support been relevantrelevant to SCS environmental threats and to SCS environmental threats and

priorities?priorities?– What are the What are the effectseffects of GEF support (positive or negative, of GEF support (positive or negative,

intended or unintended) on country efforts and environmental intended or unintended) on country efforts and environmental problems?problems?

– What are the What are the critical factors critical factors that affect likelihood that that affect likelihood that support will catalyze actions that will reduce environmental support will catalyze actions that will reduce environmental stress and improve environmental and socioeconomic status?stress and improve environmental and socioeconomic status?

– WhatWhat lessons lessons can be learned from the interventions’ can be learned from the interventions’ successes and failures that apply to the SCS and elsewhere?successes and failures that apply to the SCS and elsewhere?

9

Impact Evaluation ApproachImpact Evaluation Approach• Impact: Impact:

““positive and negative, primary and positive and negative, primary and secondary long-term effects produced by secondary long-term effects produced by a development intervention, directly or a development intervention, directly or indirectly, intended or unintended.”indirectly, intended or unintended.”(OECD Development Assistance (OECD Development Assistance Committee, 2002)Committee, 2002)

• The ROtI (The ROtI (RReview of eview of OOutcomes to utcomes to IImpacts) methodology assesses mpacts) methodology assesses progressprogress of projects’ outcomes of projects’ outcomes toward impacttoward impact, and the likelihood of , and the likelihood of achieving the ultimate goals of achieving the ultimate goals of reducing environmental stress and reducing environmental stress and improving environmental and socio-improving environmental and socio-economic status.economic status.

10

Approach ConsiderationsApproach Considerations

• Select illustrative interventionsSelect illustrative interventions

• Assess how key actors responded to Assess how key actors responded to interventions, interventions, – Assuming that response to the same Assuming that response to the same

intervention will vary from actor to actor, intervention will vary from actor to actor, time to time; thus the same intervention time to time; thus the same intervention will have different impact in different will have different impact in different places.places.

• Assess extent to which there is Assess extent to which there is movement to intermediate stagesmovement to intermediate stages

• Assess contextual factors affecting Assess contextual factors affecting decisionsdecisions

11

Introduction to Theory of Change Introduction to Theory of Change Approach Approach

• Evaluation will use the Theory of Change Evaluation will use the Theory of Change approach: a tool to assess GEF support’s approach: a tool to assess GEF support’s contribution to impactcontribution to impact– Assumes there are multiple paths toward Assumes there are multiple paths toward

impactimpact

• Three characteristics of complex socio-Three characteristics of complex socio-ecological systems merit special attention ecological systems merit special attention in impact evaluations (E. Ostrom):in impact evaluations (E. Ostrom):– PolycentricPolycentric: independent units interact with : independent units interact with

one anotherone another– Multi-scalarMulti-scalar: institutions and processes are : institutions and processes are

linked at multiple levels and system linked at multiple levels and system boundaries varyboundaries vary

– Multi-temporalMulti-temporal: diverse temporal lags and : diverse temporal lags and inherent delays exist between actions and inherent delays exist between actions and (natural and other) systems’ response. (natural and other) systems’ response. 12

Approach: Theory-based ApproachApproach: Theory-based Approach

• Programmes and projects are built Programmes and projects are built on assumptions on how and why on assumptions on how and why they are supposed to achieve their they are supposed to achieve their objectivesobjectives

• In project proposal documents, In project proposal documents, some assumptions are explicit, some assumptions are explicit, others are notothers are not

• Assumptions about how and why Assumptions about how and why impacts will be achieved must be impacts will be achieved must be made explicit before they can be made explicit before they can be evaluatedevaluated

• When causal mechanisms behind When causal mechanisms behind assumptions and project assumptions and project contextual factors are known, contextual factors are known, indicators can be identified for indicators can be identified for progress towards impactprogress towards impact

13

Impact Evaluation FrameworkImpact Evaluation Framework

Impact

ReducedThreats to

GEB

Enhancedstatus of

GEBOutcomeOutputs

State/condition

State/condition

Assumption Assumption Assumption

Threats Based AnalysisOutcomes-Impacts TOC AnalysisProject Logframe Analysis

Assess direct effects of the project

Assess how these effects are leading to impacts

Assess whether impacts have actually occurred

+ ++ +++

15

Actors

Complex Socio-Ecological SystemComplex Socio-Ecological System

Regional Factors Country Factors Country Decisions

Project Intervention

Thank you.Thank you.

• For more information on this For more information on this evaluation, please contact Aaron evaluation, please contact Aaron Zazueta, azazueta@thegef.orgZazueta, azazueta@thegef.org

16