midterm evaluation of the nama facility · nama facility, the evaluation object; the purpose of...

Final Report Appendix 1

Midterm Evaluation

of the NAMA Facility

Inception Report

Submitted to GIZ by LTS International Ltd.

30th June 2016

LTS International Ltd

Pentlands Science Park, Bush Loan

Penicuik, EH26 0PL

United Kingdom

Tel. +44 (0)131 440 5500 Fax. +44 (0)131 440

5501

Email. [email protected]

Web. www.ltsi.co.uk Twitter. @LTS_Int

Registered in Scotland Number 100833

mailto:[email protected]

http://www.ltsi.co.uk/

Acronyms

BMUB - German Federal Ministry for the Environment, Nature Conservation, Building and

Nuclear Safety

CAUES - China Association of Urban Environmental Sanitation

CCAP - Centre for Clean Air Policy

CDKN - Climate and Development Knowledge Network

CF Ready - GIZ Climate Finance Readiness

CIFs – Climate Investment Funds

COP – Conference of the Parties

CRGE - Ethiopia’s Climate Resilient Green Economy Initiative

CTCN – Climate Technology Centre and Network

DA – Development Assistance

DECC - UK Department of Energy and Climate Change

DOs – Delivery Organisations

EC – European Commission

EFKM – Danish Ministry of Energy, Utilities and Climate

ESG - Evaluation Steering Group

ET - Evaluation Team

EU – European Union

FINDETER - Financiera de Desarrollo Territorial S.A.

FIRM - Facilitating Implementation and Readiness for Mitigation

GCF – Green Climate Fund

GEF – Global Environment Facility

GGGI - Global Green Growth Institute

GHG -Greenhouse Gas

GID - General Information Documents

GIZ – Deutsche Gesellschaft für Internationale Zusammenarbeit

GNESD – Global Network on Energy for Sustainable Development

IADB, IDB – Inter American Development Bank

ICF - International Climate Fund

IGOs – Intergovernmental Organisations

ISR – Institutional Systems Review

ISRM – Institutional Systems Review Matrix

KfW - Kreditanstalt für Wiederaufbau (KfW Development Bank)

KO – Kick-off

LDCs - Least Developed Countries

LEDS - Low Emission Development Strategies

MDG – Millennium Development Goals

MEUC – Danish Ministry of Energy, Utilities and Climate

M&E - Monitoring and Evaluation

NA – not applicable

NAMA - Nationally Appropriate Mitigation Action

NDCs - Nationally Determined Contributions

NEFCO – Nordic Environment Finance Corporation

NGO – Non-governmental Organisation

NSP – NAMA Support Projects

ODA – Official Development Assistance

OECD DAC - Organisation for Economic Co-operation and Development Development

Assistance Committee

PMR - Partnership for Market Readiness

RAC NAMA - Refrigeration and Air Conditioning NAMA

SSRE - Self-Supply Renewable Energy

SNV – Netherlands Development Organisation

TOC – Theory of Change

TOD - Transit-orientated Development

TOR - Terms of Reference

TSU – Technical Support Unit

UK – United Kingdom

UDP – UNEP DTU Partnership

UNDP – United Nations Development Programme

UNEG - United Nations Evaluators’ Group

UNEP – United Nations Environment Programme

UNFCCC - United Nations Framework Convention on Climate Change

UNIDO – United Nations Industrial Development Organisation

Contents ACRONYMS .................................................................................................................................................. 1

CONTENTS ................................................................................................................................................... 4

LIST OF TABLES .............................................................................................................................................................................. 6

LIST OF FIGURES ............................................................................................................................................................................ 6

EXECUTIVE SUMMARY ............................................................................................................................... 1

1 INTRODUCTION .................................................................................................................................. 2

1.1 BACKGROUND TO THE ASSIGNMENT .......................................................................................................................... 2

1.2 PURPOSE OF THIS REPORT ............................................................................................................................................ 3

1.3 STRUCTURE OF THIS REPORT ........................................................................................................................................ 3

2 DESCRIPTION OF THE EVALUATION OBJECT ................................................................................ 5

2.1 INTRODUCTION TO NAMAS ....................................................................................................................................... 5

2.2 SOURCES OF NAMA FINANCING ............................................................................................................................... 7

2.3 THE NAMA FACILITY ................................................................................................................................................... 7

NAMA Facility Governance and Project Cycles ..................................................................................................... 10

The NAMA Facility Portfolio ......................................................................................................................................... 12

First Call for Proposals, September 2013 ................................................................................................................. 13

Second Call for Proposals, July 2014 ......................................................................................................................... 14

Third Call for Proposals, July 2015 ............................................................................................................................. 15

Concerns about the overall NSP Portfolio ............................................................................................................... 15

3 PURPOSE AND OBJECTIVES OF THE NAMA FACILITY MIDTERM EVALUATION .................... 18

3.1 THE MIDTERM EVALUATION OBJECTIVE .................................................................................................................. 18

3.2 EVALUATION STAKEHOLDERS .................................................................................................................................... 19

Insiders ................................................................................................................................................................................. 19

External stakeholders targeted for interviews ........................................................................................................ 20

Outsiders potentially relevant for outreach ............................................................................................................ 21

3.3 EVALUATION QUESTIONS .......................................................................................................................................... 21

Overarching questions leading to recommendations ......................................................................................... 21

Evaluation tasks ................................................................................................................................................................ 22

From evaluation tasks to evaluation work streams ............................................................................................. 24

Coverage Check ................................................................................................................................................................ 25

Summary Evaluation Matrix ......................................................................................................................................... 25

4 METHODOLOGY AND PRINCIPLES ................................................................................................ 28

4.1 ASSESSMENT METHODOLOGIES ................................................................................................................................ 28

Theory of Change Analysis ........................................................................................................................................... 28

Mapping and Assessment of the Project Cycle ...................................................................................................... 29

Institutional Systems Review ........................................................................................................................................ 29

Portfolio Analysis .............................................................................................................................................................. 31

4.2 OTHER WORK STREAMS ............................................................................................................................................. 31

Climate Finance Gap assessment ............................................................................................................................... 31

Perception Assessment ................................................................................................................................................... 32

4.3 DATA COLLECTION METHODOLOGIES ..................................................................................................................... 32

Desk Review of Documentation .................................................................................................................................. 32

Informant Interviews ....................................................................................................................................................... 33

Rubrics .................................................................................................................................................................................. 34

4.4 TRIANGULATION AND EVIDENCE ASSESSMENT ....................................................................................................... 36

Triangulation Protocol .................................................................................................................................................... 36

Strength of Evidence Protocol ...................................................................................................................................... 37

4.5 LIMITATIONS ................................................................................................................................................................ 38

5 APPROACH ........................................................................................................................................ 40

5.1 OVERVIEW OF EVALUATION PHASES ........................................................................................................................ 40

5.2 DESK REVIEW .............................................................................................................................................................. 40

5.3 THE INTERNAL PERSPECTIVE PHASE .......................................................................................................................... 41

5.4 CONTINUED DOCUMENT REVIEW AS NEEDED ....................................................................................................... 41

5.5 THE EXTERNAL PERPECTIVE PHASE ........................................................................................................................... 42

5.6 DATA ANALYSIS AND REPORTING ............................................................................................................................ 42

6 WORK PLAN ...................................................................................................................................... 43

6.1 NAMA FACILITY MIDTERM EVALUATION KEY DATES ........................................................................................... 43

6.2 UPDATED WORK PLAN .............................................................................................................................................. 44

6.3 ADDITIONAL PERSONNEL........................................................................................................................................... 44

Dr Martina Greib............................................................................................................................................................... 45

William Battye ................................................................................................................................................................... 45

ANNEXES .................................................................................................................................................... 46

Annex List ............................................................................................................................................................................ 46

Annex 1 - Evaluation Matrix ......................................................................................................................................... 47

Annex 2 – Strength of Evidence Protocol and Triangulation Matrix .............................................................. 52

Annex 3 – Documentation Informing Inception Phase ....................................................................................... 53

Annex 4 – Full Workplan ............................................................................................................................................... 57

Annex 5 Original Terms of Reference ....................................................................................................................... 58

List of Tables Table 1. Successful NSP applications and their status from Call 1 ..................................................... 13

Table 2. Successful NSP applications and their status from Call 2 ..................................................... 14

Table 3: Successful NSP applications and their status from Call 3 ..................................................... 15

Table 4. NAMA Facility components understanding for the Midterm Evaluation ....................... 23

Table 5. Reallocation of tasks between work streams ............................................................................ 24

Table 6. Evaluation Work Streams .................................................................................................................. 25

Table 7. Summary Evaluation Matrix (full matrix is provided in Annex 1) ....................................... 26

Table 8. Components and potential subcomponents (lines of the Institutional Systems Review

Matrix) ....................................................................................................................................................................... 30

Table 9. Stakeholder groups for key informant interviews ................................................................... 33

Table 10. Strength of Evidence protocol ...................................................................................................... 38

Table 11: Indicative Key Milestones Timeline ............................................................................................. 43

List of Figures Figure 1. NAMA Facility Governance Structure ............................................................................................ 9

Figure 2. NAMA Facility three step NAMA Support Project selection process ............................. 11

Figure 3. Global map of current NAMA Facility Portfolio ...................................................................... 12

Figure 4. NSP Outlines by Sector .................................................................................................................... 13

Figure 5. NAMA Facility Programme Rationale ......................................................................................... 23

Figure 6. Indicative Work Plan .......................................................................................................................... 44

NAMA Facility MTE Inception Report P a g e | 1

Executive Summary

Nationally Appropriate Mitigation Actions (NAMAs) are an important modality in the post-

Copenhagen and post-Paris climate finance arena. They allow the formulation and

implementation of mitigation programmes and projects in non-Annex-I countries. The

NAMA Facility exists to provide the funding for financial and technical assistance required for

the implementation of NAMAs. It is jointly funded by DE: BMUB and UK: DECC, DK: DEKM

and the European Commission. Since 2013 it has launched three calls for proposals, resulting

in 14 NAMA Support Projects (NSP). This Midterm Evaluation is tasked with understanding

strengths and weaknesses of The NAMA Facility at all levels (governance, Technical Support

Unit (TSU), NSP pipeline and portfolio); the relevance of The NAMA Facility and how

management of implementation of The NAMA Facility can be further improved.

The inception report discusses the evaluation team’s understanding of the evaluation and

the evaluation questions. In order to operationalise the numerous evaluation tasks and

questions from the Terms of Reference, the evaluation team defines six work streams: The

Theory of Change analysis, a Portfolio Assessment, an Institutional Systems Review, a Project

Cycle mapping, a Climate Finance Mapping and a Perception Assessment. The approaches

used in the work streams differ, but all work streams use varied data sources, including

documents and key stakeholder interviews.

The evaluation is structured to accommodate the specific stakeholder constellation of The

NAMA Facility. The immediate stakeholders to the evaluation include the TSU and the

Evaluation Steering Group which includes the Donors. Stakeholders that might be interested

in the results can also be found in the wider climate community, including intergovernmental

organisations and other country governments. As The NAMA Facility is strongly aware that

there might be a significant difference in perception between these groups, the overall

evaluation uses a two-step approach. In the first phase, it will focus on the inside perspective,

leading through document review and interviews with TSU, Donors and the selected NSPs

and Delivery Organisations to preliminary findings, lessons and recommendations on what

the Facility is doing right and can and should be doing better. In the second phase, this

internal view will then be complemented by the external perspective of Intergovernmental

Organisations, non-supported NSPs and observers. This will allow to clearly understand

where internal and external views diverge. In order to separate the better justified from the

not so well justified claims, the evaluation will use a triangulation protocol to collect the

relevant evidence and understand what findings, recommendations and lessons can be

helpful to enhance the strategic relevance, effectiveness, efficiency and impact of The NAMA

Facility.


1 Introduction

This section outlines the background to this assignment by providing a brief overview of The

NAMA Facility, the evaluation object; the purpose of this Inception Report and its structure.

1.1 Background to the Assignment The evaluation object, The NAMA Facility, was jointly established by the German Federal

Ministry for the Environment, Nature Conservation, Building and Nuclear Safety (BMUB) and

the UK Department of Energy and Climate Change (DECC) in 2012 to support developing

countries and emerging economies that show leadership on tackling climate change and

want to implement transformational country-led Nationally Appropriate Mitigation Actions

(NAMAs) within the existing global mitigation architecture in the short term1. In 2015, the

Danish Ministries of Climate, Energy, and Building MEUC) and the European Commission

joined The NAMA Facility as additional donors. Since its launch, The NAMA Facility has

supported three Calls for NAMA Support Project Outlines, making available accumulated

funding of more than 200 million euros.2

The expectation is that by financing and supporting the implementation of ambitious 'NAMA

Support Projects' (NSP) The NAMA Facility contributes to direct and indirect reduction of

GHG emissions and provides examples on how NAMAs may be financed and implemented.

The activities of The NAMA Facility shall encourage more countries to prepare NAMAs and

to mobilize additional public and private finance for the implementation of NAMAs. In line

with its objective to support the most ambitious and promising NAMAs with a high potential

for transformational change towards low carbon development, The NAMA Facility sets no

specific regional or sector focus.3

The NAMA Facility is governed by a Board consisting of donor representatives, which take

ultimate decisions on the selection and support of NSP Outlines. The KfW Development Bank

(KfW) and Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) were

1 NAMA Facility. (2015). General Information Document. [Online]. Available at: http://www.nama-

facility.org/fileadmin/user_upload/pdf/NAMA_Facility_General_Information_Document_2015.pdf. [Accessed 25

th May 2016].

2 NAMA Facility. (2016). Terms of Reference VN 81197008: Midterm Evaluation of the NAMA Facility.

3 Ibid.

http://www.nama-facility.org/fileadmin/user_upload/pdf/NAMA_Facility_General_Information_Document_2015.pdf

http://www.nama-facility.org/fileadmin/user_upload/pdf/NAMA_Facility_General_Information_Document_2015.pdf


commissioned to implement The NAMA Facility as 'trustees4' on behalf of the donors where

the trustee-role is not limited to the financial managements: A Technical Support Unit (TSU)

has been established to provide secretariat and management services to The NAMA Facility 5.

Like other qualified third-party organisations, KfW Development Bank and GIZ can equally

apply and act as Delivery Organisations for the implementation of individual NSPs.

Following the general implementation plan for The NAMA Facility, a midterm evaluation of

the facility’s governance, management and the NAMA Support Project portfolio was

mandated by the donors6. This midterm evaluation was commissioned as a result of this

mandate, with the contract being awarded to LTS

1.2 Purpose of this report The inception phase of this assignment focused on developing a deeper understanding of

The NAMA Facility, further developing the evaluation methodology and starting to gather

key documents for data collection. During the inception phase, the evaluation team have

refined the evaluation strategies and adjusted the evaluation methodologies and approaches

proposed in the tender, so that the questions expressed in the terms of reference and in the

discussions with The NAMA Facility TSU and the Evaluation Steering Group (ESG) in the Kick-

off meeting will be addressed in an efficient and effective manner in the evaluation.

This report details outcomes of the inception phase, and provides a rationale for subsequent

decisions surrounding LTS’s approach to the assignment. It also sets out an updated work

plan based upon findings from the inception phase. At the time of writing this final report, a

contract extension until end of November 2016 has been prepared that is reflected in this

work plan.

1.3 Structure of this report Section 1 summarises the background to the assignment. The evaluation teams

understanding of the evaluation object, The NAMA Facility, is put forward in section 2. The

midterm evaluation purpose and objectives are detailed in section 3, as well as a breakdown

of the evaluation questions and tasks. Section 4 contains the detailed evaluation

methodology. It also includes the teams approach to triangulation, strength of evidence

4 The term ‘trustee’ in the context of the NAMA Facility is used to differentiate the role of KfW and GIZ vis-à-vis

third party Delivery Organisations. It is not used in the context of its actual definition.

5 Moving forward, KfW Development Bank will no longer be involved in this (Source – Kick off meeting

discussion 10th

May 2016).

6 Kick off meeting discussion 10

th May 2016.


protocols and known limitations for this evaluation, which leads into discussion of the

approach phases in section 5. Finally, section 6 outlines the revised work plan showing the

timeline for the delivery of activities subject to client approval.


2 Description of the Evaluation

Object This section provides a summary description of The NAMA Facility including its governance,

project cycles, sources of finance, history of calls for grants and current portfolio of projects.

2.1 Introduction to NAMAs The term “NAMA” originates from the Bali Action Plan from the 13th Conference of Parties

(COP) of the United Nations Framework Convention on Climate Change (UNFCCC) in 2007,

where Parties were discussing a way to enhance national and international action on

mitigation of climate change. The Bali Action Plan has four main components: (i) Mitigation,

(ii) Adaptation, (iii) Technology, and (iv) Financing. NAMAs formed an integral part of the

mitigation component. The Bali Action Plan called for future discussions to address7.

Measurable, reportable and verifiable nationally appropriate mitigation

commitments or actions by all developed countries, and;

Nationally appropriate mitigation actions by developing country Parties,

supported and enabled by technology, financing and capacity-building, in a

measurable, reportable and verifiable manner.

The Copenhagen Accord (COP15) narrowed the concept of NAMAs to only apply to Non-

Annex 1 countries.

While the UNFCCC has not issued a precise definition for NAMAs, emerging consensus is

that a NAMA refers to any action that reduces emissions in developing countries and is

prepared under the umbrella of a national governmental initiative. They can be policies

directed at transformational change within an economic sector, or programmes or actions

across sectors for a broader national focus. NAMAs are supported and enabled by

technology, financing, and capacity-building and are aimed at achieving a reduction in

emissions relative to 'business as usual' emissions8. By moving countries towards a low-

7 UNDP. (2008). Bali Road Map: Key Issues under Negotiation.

http://www.undp.org/content/dam/undp/library/Environment%20and%20Energy/Climate%20Change/Bali_Road_Map_Key_Issues_Under_Negotiation.pdf.

8 UNFCCC., (2014). FOCUS: Mitigation - NAMAs, Nationally Appropriate Mitigation Actions. [Online]. Available

at: http://unfccc.int/focus/mitigation/items/7172.php [Accessed 31st May 2016].

http://unfccc.int/focus/mitigation/items/7172.php


carbon development trajectory, NAMAs have the potential to significantly contribute to

global efforts to reduce greenhouse gas (GHG) emissions. Simultaneously, they offer

developing countries and emerging economies a framework for combining broad-based

climate action with the achievement of sustainable development goals. This concept is

gaining momentum and many developing countries are already developing NAMAs in the

context of their national development strategies and plans9.

In practice, NAMAs are usually conceived as sector-wide transformational programmes that

are national in scope. Yet, NAMAs will have different scopes and include different activities.

Common themes for NAMAs include: 10

They contribute to the reduction or limitation of greenhouse gas emissions;

They contribute to the transformation of an economy towards low-carbon growth;

combining development and climate change mitigation;

NAMAs are carried out on a voluntary basis by developing countries;

NAMAs can be funded entirely domestically (“unilateral NAMAs”), or to rely on;

international financial or technical support (“supported NAMAs”).

The NAMA Facility has embedded the following principles into its selection criteria, which

represent the emerging consensus on best practices to promote transformational change11.

NAMAs should be country-driven and anchored in national development strategies

and plans;

NAMAs should strive to be sector-wide programmes that are national in scope, even

if regional or municipal elements could form part of the overall design;

NAMAs should consist of a combination of policies and financial mechanisms;

Policies should serve to create an enabling environment and channel financial flows

into low-carbon investments. Financial mechanisms should serve to address potential

barriers for investment and leverage potential public support for mitigation activities;

9 NAMA Facility. (2015). General Information Document. [Online]. Available at: http://www.nama-

facility.org/fileadmin/user_upload/pdf/NAMA_Facility_General_Information_Document_2015.pdf. [Accessed 25th May 2016].

10 International Partnership on Mitigation and MRV. [Online].http://mitigationpartnership.net/nationally-

appropriate-mitigation-actions-namas-0. [Accessed 26th

May 2016].

11 NAMA Facility., (2015). General Information Document. [Online]. Available at: http://www.nama-



International support for NAMAs needs to be flexible in order to provide tailor-made

solutions that are appropriate for the circumstances and capabilities of different

countries. International funds should be used to enable the implementation of

NAMAs and leverage additional public and/or private capital investment. A strategy

for self-sustained implementation at national level should be envisaged.

2.2 Sources of NAMA Financing NAMA financing can be a combination of public and private financing coming from various

national or international sources. Public financing could be either from national government

budgets or international public sources; private financing also could be from national or

international development and private financial institutions12.

The Supported NAMAs currently listed in the UNFCCC NAMA Registry are supported by13:

1) NAMA Facility

2) UNDP MDG CARBON

3) GEF Trust Fund

4) Spanish NAMA Platform

5) Inter-American Development Bank

6) Japan’s ODA for Climate change measures

However, there are a wide variety of other potential funders, most notably the Green Climate

Fund (GCF) (Refer to section 3.2 for more examples of potential funders).

2.3 The NAMA Facility Established in 2012, The NAMA Facility was officially announced by Germany and the United

Kingdom (UK) at the eighteenth session of the Conference of the Parties (COP18) in Doha

with a joint funding of €120m for the first and second call. Denmark and the European

Commission (EC) have since contributed a combined total of €25m (€15 million contribution

from the European Commission and €10 million from Denmark) for the third call. An

12 Sharma, S. & Desgain, D., (2013). Understanding the Concept of Nationally Appropriate Mitigation Action.

[Online]. Available at: http://mitigationpartnership.net/sites/default/files/2013_unep_risoe_sharma_understanding_namas.pdf [Accessed 1st June 2016].

13 UNFCCC NAMA Registry., (2016). [Online]. Available at: http://www4.unfccc.int/sites/nama. [Accessed 31

May, 2016].

http://mitigationpartnership.net/sites/default/files/2013_unep_risoe_sharma_understanding_namas.pdf

http://www4.unfccc.int/sites/nama


additional £75m of UK International Climate Fund (ICF) funding was approved in 2015,

awarded on the basis of DECC’s new business case for the 3rd and potential further calls.1415

The NAMA Facility is managed by a TSU that is jointly run by GIZ and the KfW Development

Bank.16

The German Federal Ministry for the Environment, Nature Conservation, Building and

Nuclear Safety (BMUB) and UK Department of Energy and Climate Change (DECC) sit on the

Facility’s Board, which takes all decisions on strategy, policies and guidelines and selects the

NAMA Support Projects (NSPs) for funding. The EU Commission and the Danish Ministry of

Energy, Utilities and Climate are also members of the Board as of 2015, starting with the

third call. Figure 1 illustrates The NAMA Facility’s governance structure.

14 European Commission, (2014). EU joins UK, Germany, Denmark in financing ground-breaking climate Facility.

[Online]. Available at: http://ec.europa.eu/clima/news/articles/news_2014121201_en.htm. [Accessed 2nd

June 2016]

15 Garside, B., (2015). EU nations could double current NAMA funding with new call for projects. [Online].

Available at: http://carbon-pulse.com/3305/. [Accessed 1st

June 2016].

16 Moving forward, KfW will no longer be involved in this. (Source – Kick off meeting discussion 10

th May 2016)

http://ec.europa.eu/clima/news/articles/news_2014121201_en.htm

http://carbon-pulse.com/3305/


Figure 1. NAMA Facility Governance Structure17

Until now, many NAMA support initiatives have focused on the preparation of NAMAs and

the creation of enabling environments for NAMA implementation (“NAMA readiness”).

International finance and support for the implementation of NAMAs, however, has been rare.

Particularly in the case of highly innovative NAMAs, it has proven difficult to access

commercial financing for implementation and thereby deliver concrete GHG emission

reductions18.

The NAMA Facility aims to provide financial support and technical assistance for the NAMAs

of developing countries and emerging economies that show leadership on tackling climate

change and that want to implement transformational country-led NAMAs within the existing

global mitigation architecture in the short term.

While financial support provided through The NAMA Facility is grant-based, Delivery

Organisations (DOs) and implementing partners are encouraged to use these grants to set

up mechanisms that will provide or unlock other types of financial support (such as

concessional loans or guarantee funds) to mobilise additional funding, including but not

limited to private investment. The support granted for a specific financial instrument,

including the level of concessionality of subsidised loans, and the amount of funding

17 Soren, D. (2016). NAMA Facility: Supporting the Implementation of Nationally Appropriate Mitigation

Actions (NAMAs). Presentation on 23 May 2016 at the NAMA Facility Workshop.

18 Ibid.


provided through the Facility is decided on the basis of the project appraisal for each NSP

individually19.

The Facility has a Monitoring and Evaluation (M&E) Framework and an overarching Theory of

Change (TOC). Project level reporting (through the DOs) will generate results-oriented

monitoring data, but few projects are expected to be reporting on results yet. There will be a

project level evaluation for The NAMA Facility in the future. Furthermore, there will be mid-

term and final evaluations for individual NSPs in the future, and a final programme

evaluation/evaluation of the overall NAMA Facility. The focus of this evaluation is on the

programme level of the Facility.

NAMA Facility Governance and Project Cycles

As discussed above, The NAMA Facility supports the implementation of NAMA Support

Projects (NSPs). NSPs are supported by qualified DOs that enhance the implementation

capacity of implementation partners (often national government and private sector)20.

They are selected on the basis of ‘NSP Outlines’ that are submitted in competitive calls. There

have been three calls for submission of outlines so far (September 2013, July 2014 and July

2015). Fewer projects have passed the Facility’s eligibility criteria than initially planned, with

many failing more than one criterion, e.g. formal requirements, readiness, the eligibility of

delivery organisations and embeddedness21. Of the 138 submissions (including

resubmissions), The NAMA Facility has thus far approved 11 project outlines in addition to

the initial pilot in Mexico, although the pilot and five more projects have moved from pre-

project into the implementation phase. Funding for implementation has been approved for

four projects (Indonesia, Peru, Thailand and Chile). The selection of NAMA Support Projects

follows a three-step process as set out in Figure 2.

19 Ibid.

20 Ibid.

21 NAMA Facility., (2015). General Information Document. [Online]. Available at: http://www.nama-



Figure 2. NAMA Facility three step NAMA Support Project selection process22

The call for submissions and the submission of Outlines for NAMA Support Projects forms

Step 1. The TSU then verifies the eligibility of the delivery organisations and of the NSP

outlines by applying defined criteria. Eligible projects are evaluated for ambition and

feasibility by the TSU, again following pre-defined criteria. An external assessment provides

feedback on each individual eligible project and on a sample of projects not eligible for

funding. In addition, the entire selection and rating process is audited by an external auditor.

The Board then pre-approves the NSP Outlines and mandates an appraisal process for each

selected project. Changes have been made to the upcoming fourth call in recognition of

feedback from the previous calls.

Step 2 begins with the in-depth appraisal of each pre-approved project by the respective

delivery organisations. With the successful conclusion of the appraisal process, delivery

organisations submit the Proposal for NAMA Support Projects. The TSU evaluates feasibility

22 Soren, D. (2016) NAMA Facility: Supporting the Implementation of Nationally Appropriate Mitigation Actions

(NAMAs). Presentation on 23 May 2016 at the NAMA Facility Workshop.


by applying defined criteria. The Board then grants final approval of the NAMA Support

Project and officially mandates the delivery organisations.

Step 3 is the implementation of the NAMA Support Project through the delivery

organisations.

The NAMA Facility Portfolio

Location and theme of supported projects

As discussed above The NAMA Facility is supporting 14 projects, 13 of which were awarded

through calls for proposals (Mexico is the pilot project).

Figure 3 illustrates the global project distribution pattern. The countries represented are:

Africa: Burkina Faso, Kenya, South Africa

Americas: Chile, Colombia (2), Costa Rica, Guatemala, Mexico, Peru

Asia: China, Thailand, Indonesia

Central Asia: Tajikistan

Figure 3. Global map of current NAMA Facility Portfolio23

As can be seen in Figure 4 the portfolio comprises of: Transport – 3 projects, Fridges and air-

conditioning – 2 projects, Energy (biomass & renewables) – 2 projects. There are also

projects in forestry and agriculture, waste management, urban development, new housing.

23 Soren, D. (2016). NAMA Facility: Supporting the Implementation of Nationally Appropriate Mitigation



Figure 4. NSP Outlines by Sector24

First Call for Proposals, September 2013

First Call Statistics

The first call saw the submission of 48 outlines, of which 19 were eligible. The four total

successful NSP proposals can be seen in Table 1, as well as the pilot project ’Sustainable

housing NAMA in Mexico’, totalling five projects.

Two of the four projects pre-approved in the first project call (Indonesia and Costa Rica) have

now submitted project proposals and moved towards implementation, however, their

financial components have not yet been designed in detail, and proposals also did not

contain any updated date on baselines and targets on project specific indicators.

Table 1. Successful NSP applications and their status from Call 1

Country NAMA Support

Project

Status Delivery Organisation

Mexico (pilot) Sustainable Housing NAMA

Implementation Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH, KfW Development Bank

Costa Rica Low Carbon Coffee Implementation 1st Deutsche Gesellschaft für Internationale Zusammenarbeit

24 Ibid


NAMA call (GIZ) GmbH

Indonesia NAMA SUTRI Funding for implementation approved 1st call

Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH

Chile Self-Supply Renewable Energy (SSRE) NAMA

Funding for implementation approved 1st call

Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH, KfW Development Bank

Colombia I Transit-orientated Development (TOD) NAMA

Appraisal 1st call Financiera de Desarrollo Territorial S.A. (FINDETER) through KfW Development Bank, Centre for Clean Air Policy (CCAP)

Second Call for Proposals, July 2014

Second Call Statistics

The second call resulted in 49 NAMA support outlines received, of which 23 met the

eligibility criteria. There were a total of four successful outlines, as can be seen in Table 2.

Table 2. Successful NSP applications and their status from Call 2

Country NAMA Support

Project

Status Delivery Organisation

Peru Sustainable Urban Transport NAMA Support Project

Funding for implementation approved 2nd call

Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH, KfW Development Bank

Tajikistan Forestry NAMA Support Project

Appraisal 2nd call Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH, KfW Development Bank

Thailand Refrigeration and Air Conditioning NAMA (RAC NAMA)

Funding for Implementation approved 2nd call


Burkina Faso Biomass Energy NAMA Support Project

Appraisal 2nd call SNV Netherlands Development Organisation


Third Call for Proposals, July 2015

Third Call Statistics

The Third Call resulted in 42 NAMA support outlines received, of which 22 were eligible.

There were a total of three successful outlines initially approved for appraisal and two more

after a reshaping process of NSP Outlines as can be seen in Table 3,

Table 3: Successful NSP applications and their status from Call 3

Country NAMA Support Project Status Delivery Organisation

China Integrated waste management NAMA

Appraisal 3rd call

China Association of Urban Environmental Sanitation (CAUES)

Colombia II NAMA for the domestic refrigeration sector

Appraisal 3rd call


Kenya Mass rapid transport system for Nairobi

Appraisal 3rd call

KfW Development Bank for the Financial Component, GIZ for the Technical Component

Guatemala Efficient Use of Fuel and Alternative Fuels in Indigenous and Rural Communities

Appraisal 3rd call

Inter-American Development Bank

South Africa Energy Efficiency in Public Buildings Programme in South Africa (EEPBP)

Appraisal 3rd call


Concerns about the overall NSP Portfolio

The Evaluation Team has been provided with presentations from each of three Lessons

Learned workshops that have been held by The NAMA Facility. Concerns raised in these

presentations are numerous, including the low number of A-rated projects, particularly from

Least Developed Countries (LDCs), and a proportionally high number of projects from GIZ

(leading to the impression of “German Aid for German Institutions”). A stakeholder survey

that has been made available to the team (e.g. UDP 2015) partially confirms these views, but

also highlights that perceptions range widely. As can be seen from Tables 1, 2 and 3 above,

the majority of the projects are located in South America, with four projects in Asia and only

three projects in Africa, neither of which have reached the implementation phase. Of the

successful projects, 75% use GIZ as the or one of the Delivery Organisations. Whilst there are


projects from LDCs in the portfolio, stakeholders have expressed concerns as to how to

increase this number moving forward25.

In addition, most projects are taking longer than the anticipated 6-12 months to go from

pre-approval to implementation. Notably, there are projects (Chile and Colombia I) pre-

approved in the first call that have not yet, or only partially, reached the implementation

stage. According to the call assessment reports that the team has been provided with, almost

half of all proposals for each call have been ineligible. Of those that were eligible, there have

also been a high number of non-elected proposals.

The NAMA Facility TSU has taken stock of some key areas of strength and weakness in

regular lesson learning exercises. The following are some of the key points that were

highlighted in the presentations from the lesson learning workshops, which are relevant to

this evaluation:

Many NSP Outlines scored low on feasibility and require further development on the

project structure and the financing concepts;

Project proposals often do not address how to ensure the early involvement of

financial actors in project development, which is crucial for the structuring and in-

depth preparation of financial support mechanisms;

Adequately designed financial support mechanisms are often not included but

considered key for strengthening the ambition and the transformational potential of

a NAMA as they enable the national government to mobilise additional financial

resources from different sources (private, national and international).

The documentation provided to the evaluation team by The NAMA Facility TSU suggests that

the Facility is in a constant process of self-monitoring and reflection on what could be

improved and The NAMA Facility has been evolving over time. The Facility has announced

that it is planning to implement some changes in the 4th Call design, aimed at receiving a

higher number of outlines of good quality from an enlarged circle of potential applicants.

Due to the timing and agreed scope, the 4th call is beyond the scope of this evaluation.

Anticipated changes include26:

Simplified application documents;

Extended range of eligible applicants;

25 Discussion on 23 May 2016 at the NAMA Facility Workshop.

26 Soren, D., (2016). NAMA Facility: Supporting the Implementation of Nationally Appropriate Mitigation



Increased support to selected NSPs for an extended preparation phase;

Enhanced information on the 4th Call (workshops and webinars).


3 Purpose and Objectives of The

NAMA Facility Midterm Evaluation

This section of the report details the objective of this Midterm Evaluation, the evaluation

stakeholders and audience and the evaluation questions and summary matrix.

3.1 The Midterm Evaluation Objective The subject of this Midterm Evaluation is the implementation of The NAMA Facility. The

Midterm Evaluation is strongly formative and is designed to inform further development of

the Facility and improvements in its systems. It focuses on effectiveness, efficiency and

strategic relevance; including the NSP portfolio, the Facility’s governance and its

management.

The specific objectives, as listed in the Terms of Reference (TOR), are to analyse and improve

the understanding of:

The strengths and weaknesses of The NAMA Facility at all levels (governance, TSU,

NSP pipeline and portfolio);

The relevance of The NAMA Facility to different stakeholders: beneficiaries and

recipient countries, to Donors, to Delivery Organisations;

How the management of implementation of The NAMA Facility can be further

improved, including answering questions related to the funding modalities (e.g.

looking at global competitive bids and how they have worked and not worked).

In assessing the strategic relevance of the Facility, the Midterm Evaluation will explore the

role the Facility plays in the context of the wider landscape of climate financing; including its

influence in relation to the Nationally Determined Contributions (NDCs) and Low Emission

Development Strategies (LEDS) that provide the framework within which specific NAMA

actions may be coordinated and supported. This includes understanding the implications of

the Paris Agreement and the capitalisation of the Green Climate Fund (GCF) on the strategic

relevance of the Facility, a point which has been reiterated by the donors.

The TOR further specify 17 evaluation tasks and a large number of example evaluation

questions which will be addressed in the evaluation. The NAMA Facility has already taken

steps towards addressing some of the currently better understood challenges to pipeline

development, in the preparations for the next call for Outlines. The evaluation will attempt to

understand how these ongoing changes are likely to influence the Facility and its operations,


and include this in the assessment, whilst recognising that the associated evidence base for

this part is limited.

3.2 Evaluation Stakeholders The target audience of the evaluation, the “evaluation stakeholders”, is first and foremost the

TSU and the NAMA Facility’s Donors. However, there are other groups that might be affected

by this evaluation, for at least three reasons:

1) The (potential) actions of each of these groups of stakeholders could affect the

Facility’s performance, so they need to be reflected in the Theory of Change;

2) The data collection for the evaluation will target several of these stakeholder groups

(See Table 9, Section 4);

3) We also defined as stakeholder groups others who potentially have an interest in the

evaluation findings and should be included in the dissemination of the results.

Building on these considerations and on the discussions in the context of the Kick-off

meeting, we identified three types of stakeholders to be considered in this Midterm

Evaluation: ‘Insiders’, ‘Outsiders targeted for interviews’ and ‘Outsiders potentially relevant

for outreach’. These are discussed in the following subsections. The ‘Insiders’ and ‘Outsiders

targeted for interviews’ groups also form the population of informants from which the

sampling strategy of this Midterm Evaluation has been drawn. Section 4 of this report

summarises the Evaluation Sampling Strategy building on this analysis.

Insiders

The stakeholder group that the evaluation team is terming “insiders” includes the ESG, the

TSU, NAMA Facility implementing agencies, and successful applicants, all of which will be

targeted in the data collection phase. The numbers in parentheses are rough estimates of the

population size.

ESG/funders (4): DE:BMUB, UK:DECC, DK:MEUC, European Commission

NAMA Facility Delivery Organisations (2): GIZ, KfW Development Bank

o TSU staff and consultants (~13)

o Embedded experts within GIZ and KfW Development Bank who interact with

or advise the TSU (<10)

o UDP staff who supported the last selection process (~ 2)

Successful applicants for the 12 NSPs who are in the pipeline at different stages from

calls 1-3, including:


o Partner ministries in the government (~25)

o Delivery Organisations (~6 organisations, but probably 12 responsible

individuals)

o Implementing partners (~90)

o National/International consultants supporting NSPs in pipeline who are not

formal delivery or implementation partners (~10, including, inter alia, Ecofys,

ECN, WRI, CCAP)

External stakeholders targeted for interviews

The stakeholder group the evaluation terms “outsiders” includes agencies and organisations

associated with unsuccessful NSPs and their consultants, eligible countries that have not

submitted NSPs, and current and potential NAMA funders, as well as the International

governmental organisations and groups with particular expertise in NAMAs, including those

supporting readiness. A subset of these will be targeted in the data collection phase.

“Outsiders” include:

Unsuccessful applicants for the 124 NSPs from calls 1-3, including (>500):

o Partner ministries in the government (up to 250)

o [potential] Delivery Organisations (unknown, >15 international organisations

plus national organisations)

o [potential] Implementing partners (>250)

o National/International consultants that supported unsuccessful NSPs who

were not formal delivery or implementation partners (NA)

International organisations with specialized knowledge (25+ organisations, with

overlaps):

o International governmental organisations: UNFCCC, UNDP, UNEP (incl. UDP)

o Current and potential funders of NAMA readiness and/or implementation:

Global Environment Facility (GEF), Green Climate Fund (GCF), CIFs, GIZ Climate

Finance Readiness (CF Ready), Denmark’s Facilitating Implementation and

Readiness for Mitigation (FIRM) implemented through UNDP, IADB, Japan’s

ODA for Climate Change Measures, Spanish NAMA Platform, Austria’s

Support for Sustainable Forest Management, as well as other international

development funds and other International and Local Development Banks.

o Collaborations/Partnerships: Partnership for Market Readiness (PMR), NAMA

Partnership or the International Partnership on Mitigation and MRV


o Specialised think tanks and consultants, such as the Centre for Clean Air

Policy, GNESD and CTCN Network members, World Resources Institute,

CDKN, Climate Policy Initiative, Ecofys, ECN, etc.

Outsiders potentially relevant for outreach

This secondary “outsiders” group includes eligible countries that have not sought funding, as

well as the broader development and climate change communities. It could also include

private sector, local community and civil society actors who could be interested in and/or

affected or may eventually become involved in proposing or benefiting from projects. These

groups will not be directly targeted in the data collection phase, but may be relevant for the

dissemination of results:

Countries eligible to submit NSPs but who have not yet done so. [this number needs

to be clarified with the TSU, 10-25 estimated];

Development and climate change mitigation and adaptation communities, including

international development agencies, civil society, consultants and evaluators;

Private sector investors, e.g. pension funds, growth equity funds.

3.3 Evaluation Questions

Overarching questions leading to recommendations

The most important questions for the evaluation are those for which recommendations of

the evaluation are needed. According to the TOR, these are:

Are the strategic objectives of The NAMA Facility as originally described in the KfW

Development Bank and GIZ programme offer being met? (the amended programme

offers shall be taken into account e.g. regarding the log frame and theory of change)

Is any potential (which could be used for improving effectiveness) being missed

and/or are any limitations regarding the intended objectives being encountered?

Why do so many project proposals submitted to The NAMA Facility score low on

eligibility and feasibility? If projects are deemed to be insufficiently ready to receive

funding, what are the main reasons for this, and what are the options for addressing

this?

How is The NAMA Facility being perceived by relevant stakeholders?

What are the strengths and weaknesses of the current project cycle/funding practice?

What are the main challenges that projects face when going from pre-approval to

final approval?


What are the early challenges faced by NSPs in the first six months of

implementation? How can The NAMA Facility change to further support NSPs in the

future?

The questions are very different from each other in terms of their scope and their procedural

vs. thematic vs. strategic orientation. This also implies that they are not easily fitted into the

OECD DAC criteria and suggests that an alternative approach to evaluation against the OECD

DAC criteria is required. The key work of the evaluation will be in striving to collect sufficient

evidence to formulate findings, lessons and recommendations against each of them in the

final evaluation report. The intended output of this evaluation is an evaluation report that

includes recommendations as to how The NAMA Facility can be improved and/or further

developed with regard to its governance, management, objectives/ambition as well as

communication.

Evaluation tasks

The TOR includes a detailed list of tasks across four thematic areas (1) NAMA Facility

governance, 2) NAMA Facility management, 3) NSPs and 4) cross-cutting issues)

complemented by a long list of example evaluation questions. In the tender, the team had

proposed a slightly shortened list of questions ordered by the OECD DAC criteria that had

been emphasised in the TOR (relevance, effectiveness, efficiency, impact).

Given that the thematic areas as specified in the TOR do not easily map onto the DAC

criteria, for the final evaluation matrix, the evaluation team revisited the two ordering

principles from the TOR (tasks and example evaluation questions).

Figure 5 illustrates the sequence from formulating the Facility objectives to donor decisions

(through the Board) and Facility governance to the TSU, as the focus of Facility management,

into the NSPs themselves. ‘Facility Management’ as implied by the evaluation tasks listed in

the TOR, is divided into three components: 1) Project Selection Procedures; 2) Project

Selection Criteria; and 3, all other parts of TSU management.


Figure 5. NAMA Facility Programme Rationale

This slight adjustment of the understanding of The NAMA Facility structure as compared to

the TOR is made more explicit in Table 4. It enabled the team to remap some of the tasks

from the TOR to slightly different thematic areas without changing their substance. This

facilitated the development of discrete evaluative work streams for the thematic areas. The

shifts are documented in Table 5.

Table 4. NAMA Facility components understanding for the Midterm Evaluation

TOR Evaluation Matrix Work stream (approx.)

NAMA

Facility

Governance

Governance / Institutional System / decision

making structure ISR

NAMA

Facility

Management

NAMA Facility management

Selection and contracting process for concepts

and pipeline development Project cycle analysis

NAMA

Support

Projects

NAMA Support Projects

Portfolio analysis

Criteria for concepts and DOs

Cross-

cutting

Cross-cutting issues / Climate finance and climate

negotiations THEORY OF CHANGE


NAMAs not supported; reputation and replication

Climate Finance Mapping

Perception analysis

Table 5. Reallocation of tasks between work streams

Evaluation Task stated in TOR Theme in TOR

Reallocated to theme in

Evaluation Matrix

• Examine the procedures for NSP approval and contracting

Governance Management / project selection and contracting

• Examine the procedures for project assessment, selection, M&E and quality assurance

Facility management Management / project selection and contracting

• Examine the procedures for building the quality of the pipeline including the extent and usefulness of communication between the TSU and project submitters as they develop their bids, and in providing feedback following submission.

Facility management Management / project selection and contracting

• Examine the eligibility criteria for qualified Delivery Organisations.

Facility management Criteria for concepts and DOs

• Examine the eligibility criteria and instruments for concepts and projects.

Facility management Criteria for concepts and DOs

• Examine the strategic objectives of The NAMA Facility (with respect to incentivizing replication / scale-up)

Cross-cutting issues NAMAs not supported; reputation and replication

• Examine the reputational effect of NSP Outline rejections and selections among proponents of rejected NSPs

NAMA-support projects

NAMAs not supported; reputation and replication

From evaluation tasks to evaluation work streams

Using these evaluation tasks listed in the TOR, and refined into thematic area, the evaluation

team then identified the data sources and most appropriate methodologies for assessment

against each task to complete the revised evaluation matrix.

The complete matrix of methods and steps against each evaluation task from the TOR is

provided in Annex 1. Six closely linked, but discrete evaluation work streams are defined in

the matrix, which are summarised in Table 6. Within each work stream, desk reviews and

interviews with various target groups are the main source of data.


Table 6. Evaluation Work Streams

Work

Stream

Theme of Work Stream No. of

task

Methodology

A Reconstructed Theory of Change, relevance and

effectiveness of NAMA Facility with respect to

strategic objectives, testing of Theory of Change

assumptions

1, 2, 4,

19

Theory of Change

B The NAMA Facility portfolio and results (closely

linked to Work Stream A)

18 Portfolio Analysis

C Relevance of NAMA Facility in the climate finance

landscape

3 Climate Finance

mapping

D NAMA Facility governance, management and

organisation (decision making, KfW/GIZ, donors)

5-10 Institutional

Systems Review

E NAMA Facility processes (selection criteria,

workflow, communication, support products) and

their influence on the projects

11 –

17

Project Cycle

mapping

F Perception of The NAMA Facility (Closely linked to

Work Stream A)

20 Informant

Interviews /

perception analysis

Coverage Check

As the last step in deriving the work stream packages, the evaluation team undertook a final

check to confirm that each of the example evaluation questions from the TOR are covered

with this scope of work.

Summary Evaluation Matrix

Table 7 summarises the Evaluation Matrix with respect to the key questions from the TOR.

The full Evaluation Matrix is provided in in Annex 1).


Table 7. Summary Evaluation Matrix (full matrix is provided in Annex 1)

Key Questions, as stated in the TOR Work stream and Method Data source Data level

• Are the strategic objectives of The NAMA Facility as originally described in the KfW and GIZ programme offer being met? (the amended programme offers shall be taken into account e.g. regarding the log frame and theory of change)

Multiple methods and work streams: desk review and interviews

Work stream B - Portfolio analysis project level

Work stream A – Theory of Change analysis

programme level

Work stream C - Climate Finance Gap analysis

global level

• Is any potential (which could be used for improving effectiveness) being missed and/or are any limitations regarding the intended objectives being encountered?

Work stream B - Portfolio analysis desk review and interviews project level


programme level

Work stream E - Project Cycle mapping

programme level

Work stream C- Climate Finance Gap analysis

global level

• Why do so many project proposals submitted to The NAMA Facility score low on eligibility and feasibility? If projects are deemed to be insufficiently ready to receive funding, what are the main reasons for this, and what are the options for addressing this?

Work stream E - Project cycle mapping desk review and interviews programme level


• How is The NAMA Facility being perceived by relevant stakeholders? Work stream F - Perception analysis desk review and interviews global level

• What are the strengths and weaknesses of the current project cycle/funding practice?

Work stream E - Project cycle mapping desk review and interviews programme level


Work stream D – Institutional Systems Review

programme level


programme level

• What are the main challenges that projects face when going from pre-approval to final approval?

Work stream E - Project Cycle mapping

desk review and interviews programme level

Work stream B - Portfolio Analysis project level

• What are the early challenges faced by NSPs in the first six months of Work stream B - Portfolio Analysis desk review and interviews project level


implementation? How can The NAMA Facility change to further support NSPs in the future?

Work stream C - Climate Finance Gap analysis

global level


4 Methodology and Principles

This section describes our assessment and data collection methodologies. It also includes

our approach to triangulation, strength of evidence protocol and known limitations for this

evaluation. Documentation informing this inception report can be found in Annex 3. Excel

versions of key tables have also been made available as additional documents.

4.1 Assessment Methodologies The following methodologies are anticipated for the major work streams of this evaluation.

Theory of Change Analysis

The Evaluation Team has been provided with a Theory of Change (TOC) diagram for the

NAMA Facility that outlines inputs, activities, anticipated outputs, outcomes and impacts.

However, a reconstructed Theory-of-Change-Analysis can be an important aid to clarify the

underlying understanding of the project between stakeholders and assess where

assumptions fundamental to the TOC need to be vetted with or adjusted to reality. The team

will use the founding documents of the NAMA Facility to reconstruct a TOC. The different

donor’s “accession” documents (ICF business cases, Danida Assessments, GIZ and KfW

proposals) will serve to reconstruct the understanding of the programme logic, strategic

objectives and intended outcomes, and relative weights between the various goals of the

Facility. It will also systematically extract the assumptions that have been formulated in the

documents, or that are implicit in the documents. This TOC reconstructed on the basis of the

written information will then serve as the initial hypothesis of the evaluation.

In the second step, these aspects – strategic objectives, assumptions, programme logic – will

be vetted with the internal stakeholders, some of whom have changed since the early days of

the Facility. In particular, the authors of the business case and the Danida Assessment are no

longer part of the Facility today. The new staff members will provide their views, whilst the

authors and original stakeholders will also be interviewed, and can validate their original

thinking with their own experiences to the current time.

This might lead to an update of the TOC reconstructed by the Evaluation Team, but it might

also lead to “points of contention”, i.e. diverging views between the potential and actual

impact pathways and underlying assumptions between the internal stakeholders and the

Evaluation Team. This will form the basis for the next steps (including more desk review and

more stakeholder interviews) in which the Evaluation Team will specify what evidence would

be needed to support the view of the stakeholders and look for that evidence in a strategic


manner (hopefully with the support of the stakeholder). These steps will also include external

stakeholders.

Both Insider and Outsider interviews will then be used to test the validity of the underlying

assumptions of the ToC, and to provide insights on the achievement of the Facility outputs,

outcomes and impacts, as far as possible.

Mapping and Assessment of the Project Cycle

The project cycle of The NAMA Facility has been documented in several places. Concerns

have been raised in terms of the length of the project cycle. The project cycle includes the

selection of outlines submitted in the calls, and an unexpectedly large number of outlines fail

on important criteria. Therefore, questions are raised on the criteria used in the project cycle,

- for DOs as well as NSP outlines – as well as on the steps in the project cycle.

The work stream will first map the project cycle, i.e. lay out the steps and the time they are

expected to take on the basis of the General Information Documents (GID). This will then be

compared with actuals from the successful projects (from work stream “Portfolio

Assessment”). The typical reasons for delays in the successful projects will be identified. This

will be discussed with the successful projects. It will also be compared with other climate

finance mechanisms where such information is available.

In addition, there will be an assessment of both the number of A rated projects that have

been approved, and the reasons why so few projects are selected. The reasons for rejection

will be extracted from the outline assessment overviews. This will lead to the identification of

criteria that provide no differentiating value to the selection process, and of the most critical

criteria, that cause the highest numbers of rejections. These critical criteria will then be

analysed further for their match with the strategic objectives

First, preliminary hypotheses about the relevance and impact of the criteria on the project

portfolio, the steps (and potential gaps) in the project cycle and their respective relevance

towards the objectives of the Facility will be formulated on the basis of the desk review, and

then vetted with internal and external stakeholders in the following phases. This will result in

recommendations on the project cycle and criteria.

Institutional Systems Review

The Institutional Systems Review (ISR) will focus on understanding whether governance,

management or process issues influence the likelihood of The NAMA Facility in reaching its

strategic objectives. This formative assessment may guide further development of the Facility

and its institutional arrangements. For the ISR, a matrix will be developed to review the

possible influence that various components of the institutional system – those that were

raised in the TOR, i.e. governance and management – can have on the strategic objectives.

As a starting point, the ISR matrix (ISRM) will include the sub-components mentioned in the


TOR. During the evaluation, the matrix will be populated by additional components and

criteria that develop from the findings from other methodologies, in particular the Theory of

Change analysis and the Project Cycle Mapping. Table 8 demonstrates components and

potential sub-components of the ISRM.

Table 8. Components and potential subcomponents (lines of the Institutional Systems Review

Matrix)

Dimension Subcomponents

Governance Governance structure of The NAMA Facility

Distribution of roles between donors and TSU

Roles of KfW/GIZ as TSU and DOs

Procedures for donor involvement, consultation and communication

processes, and funds management

Management Organisational set-up of the TSU and the management of The NAMA

Facility

Internal organisation of The NAMA Facility TSU, work flow, communication,

decision making, potential for learning

Financial and operational management,

Communication and outreach products27

The ISRM’s subcomponents will be combined with rubrics on a Scoring Sheet. The scoring

will be done with respect to how strongly the subcomponent influences the performance of

The NAMA Facility in attaining its strategic objectives. The questions will be phrased such

that they build on the hypotheses generated in the Project Cycle and Portfolio work

streams28.

The rubrics will provide the definition for scoring each of the criteria so that the ISRM allows

to record the assessments (rubrics discussed in Section 4.3). This will support the internal

collection of information from the interviews with key informants from the TSU and ESG, as

follows:

Each interviewee’s self-assessed score (based on the assessment criteria in the

matrix);

Each interviewee’s reasons for selecting the score (up to a maximum of three bullet

points);

Recording any documentary evidence that is provided to the assessors;

27 Communication with the projects will be assessed under the Project Cycle work stream

28 This is why the exact questions cannot be provided at this point.


Any additional information or observations made by the assessors during the

interviews.

This internal collection will initially not be shared with the evaluation stakeholders, in order

to not introduce a self-confirmation bias. However, it will support the assessment of the

evaluators, to record their external assessments in the form of:

The final score that the assessors have determined;

Justification for the score (up to a maximum of three bullet points).

For each subcomponent, one score will be selected. Evaluators will take particular care to

produce a concise and evidence-based assessment. The evidence will be much more

important than the scores for the derivation of findings, lessons and recommendations.

Portfolio Analysis

The portfolio of selected projects will be examined through desk review and interviews with

respect to the questions specified in the evaluation matrix. These include questions

regarding the NAMAs themselves, including implementation challenges, their expected

results nationally, their suitability as role models, and their contribution to The NAMA

Facility’s objectives. Specific evaluation questions have also been formulated with respect to

gender, human rights, and transformative changes. The evaluation questions will serve as the

basis for a data collection instrument that will serve as a standard information collection tool

across the nine NSPs from the first two calls. In addition, the Portfolio Analysis will have

strong links with the other work streams as these will lead to hypotheses (e.g. about how the

NAMA Facilities’ processes or selection criteria influence the portfolio) that can be confirmed

or rejected on the basis of the evidence from the portfolio review. The instrument will

therefore include questions that anticipate these information needs and include them in the

data collection efforts, to the degree possible.

The instrument contains standard project information fields, and more evaluative rubrics. It

will be used for the desk review and the interviews, and include the triangulation and

strength of evidence protocol assessments. Aggregating the results from all nine cases will

allow for the identification of patterns and for portfolio-wide findings on the evaluation

questions and hypotheses which will form the basis for recommendations on the portfolio

itself but also on many other evaluation questions.

4.2 Other work streams

Climate Finance Gap assessment

A basic premise of the NAMA Facility was that funding for the implementation of NAMAs is

not available in the current climate finance landscape. The Climate Finance Gap assessment


will map the current climate finance landscape in order to understand to what degree this is

the case – and acknowledge that the climate finance landscape has changed since the

creation of the Facility. Validated with internal and external interviews, as well as the results

from the project cycle mapping and portfolio assessment, findings, lessons and

recommendations to the current situation will be formulated. This step will be undertaken

comparatively late in the evaluation, as it is important to focus on the specific niche and

strategic objective of the NAMA Facility, which will be identified and validated with the

Steering Group in the context of the TOC analysis. These include the impact that the NAMA

Facility might have as a role model for other climate finance mechanisms and aid flows.

Perception Assessment

In the course of the ongoing learning and reflection mode of the NAMA Facility, a number of

internal hypotheses have been formed and documented that related to the question how the

NAMA Facility is perceived. These will be summarised in the context of the first document

analysis and validated in the internal interview phase, for discussion at the first debriefing

meeting. The stakeholder interviews of the “external perspective” phase will contain a small

set of questions on these hypotheses. In the Perception Assessment Work stream, these will

then be summarised for findings, conclusions, recommendations and lessons.

4.3 Data Collection Methodologies

Desk Review of Documentation

The desk review will provide most data for the assessment.

Desk reviews will be conducted in each work stream (except (F) Perception Analysis) and will

vary in scope and depth. In some work streams it will be based on a specific instrument that

operationalises the rubrics and evaluation questions and serves as one basis for the

collection of data in the interviews. In others, these instruments will be developed during the

desk review. The Evaluation Matrix contains more specific details on how and when to

conduct desk reviews to what end.

The desk reviews will include the following documents, among others:

Products of The NAMA Facility, including outreach and guidance documents;

Programme Offers (GIZ and KfW Development Bank) underlying contractual

arrangements between KfW and GIZ, as well as BMUB;

Board Decision Documents (BDDs);

The Detailed Arrangement(s), Joint Declaration of Intent (modified);

Reports to and by The NAMA Facility donors;


Board meeting notes;

Project documentation;

Assessment compilations;

Third party information, including from UNFCCC, other international and national

bodies, think tanks.

Gaps in the documentation that are identified will be noted and if possible filled in the

interviews.

Informant Interviews

As per the TOR, the evaluation team is targeting approximately 50 interviews for completion

under this evaluation. Table 9 provides a preliminary breakdown by stakeholder type. Refer

to Section 3.2 for a comprehensive list of evaluation and NAMA Facility stakeholders.

Table 9. Stakeholder groups for key informant interviews

Stakeholder Group Est. Population

Targeted for Interviews

Sampling Strategy

Insiders

ESG/Funders 4 4 Full coverage

TSU Staff ~15 8 Full coverage of organisations involved

Pipeline (calls 1-2): 9 NSPs (ministries and delivery partners)

~21 13 Full coverage of calls 1 and 2, 9 NSPs; engage ministry(s) and/or delivery partners, as appropriate

UDP 1 1

Insider subtotal 26

Outsiders

Unsuccessful NSPs (calls 1-3): (ministries and Delivery Organisations)

500+ 15 “critical case” sampling: those outlines that were closest to being funded; engage ministry(s) and/or delivery partners, as appropriate

International Organisations (specialized knowledge)

25 10 Full coverage IGOs, includes UNFCCC, GIZ, KfW, other climate finance mechanisms; purposive sampling of bilateral donors and observers; self-sampling

Outsider subtotal 24

Grand total 50

Semi-structured interviews will be conducted mainly by phone. However, the team will look

for opportunities, such as through previously scheduled workshops or international

meetings, to conduct personal interviews, as feasible. The donors will be visited at their

respective headquarters.

For the External Perspective, sampling is particularly critical. We propose to undertake critical

case sampling of the unsuccessful NSPs.


As detailed in Section 3.2, the following subgroups can be distinguished among the

international organisations with specialized knowledge:

International governmental organisations;

Current and potential funders of NAMA readiness and/or implementation;

Collaborations and partnerships;

Think tanks and consultants specialized in NAMAs.

The evaluation team propose to break out the 10 interviews anticipated for this group as

follows:

1. International governmental organisations: UNFCCC

2. International governmental organisations: UNDP

3. Climate Finance Mechanisms: GEF

4. Climate Finance Mechanisms: GCF

5. Climate Finance Mechanisms: IDB, NEFCO or World Bank

6. Think tanks/consultants/NGO (observers)

7. Think tanks/consultants/NGO (observers)

8. A UNDP or UNIDO Country Office supporting NAMA development but not applying

for NSP support

9. A bilateral programme supporting NAMA development but not applying for NSP

support

In addition, the evaluation team proposes that The NAMA Facility publishes a note about

how to access the evaluation team on its website, with an invitation to contact the evaluation

team as an interviewee (“self-sampling”) under the confidentiality codes applicable to an

evaluation. We appreciate that the same interviewees will be asked questions related to

several different work-streams of the evaluation. Interviews will be tailored to capture

evidence for each of the work-streams as relevant with the transcripts made available to the

evaluation team as a whole.

Rubrics

Rubrics are a generative, theory-based assessment. They use multiple criteria to score the

quality of a service. Rubrics are made up of29:

Evaluative criteria: the aspects of performance the evaluation focusses on;

29 Oakden, J., (2013). Evaluation Rubrics. Better Evaluation. [Online] Available at:

http://betterevaluation.org/sites/default/files/Evaluation%20rubrics.pdf [Accessed 2nd

June 2016].

http://betterevaluation.org/sites/default/files/Evaluation%20rubrics.pdf


Merit determination: the definitions of what performance looks like at each level.

Usually the evaluative criteria are developed once an understanding what the service should

be is established; drawing from the Theory of Change and a review of available literature and

the initial evaluative analysis.

In this case, we are proposing to use the rubric to support our institutional analysis, project

cycle mapping and analysis process mapping; and would first review the workflows of The

NAMA Facility and TSU, review the organisational documentation, previous lesson learning

documentation. The evaluation questions will help to determine the type of information

required for the assessment. Following the initial analysis, we will agree with the client on

the important aspects to include and the rubrics at the debriefing meeting in July. To

facilitate the determination of the rubrics, we suggest a discussion around how specific

aspects influence (or not) the ability of The NAMA Facility to attain its strategic objectives:

“Given the strategic objectives of The NAMA Facility, how will this aspect

influence its ability to reach them and what evidence is there to support this

statement?”

With the evaluation questions in mind, we expect the potential criteria will centre on the

evaluation tasks formulated in the Evaluation Matrix. Drawing on the July debriefing

meeting, the evaluation team will prepare a set of criteria describing what evidence and

observations would be expected at each of several levels to allow a rating to be

determined. A rating of the evidence for each criterion can be done on scales such as this

(following Davidson, 200530:

Excellent: (Always) Clear example of best practice in this domain; no weaknesses.

Likely that 90% or more agree with statement to a considerable or high degree.

Very good: (Almost Always) Very good to excellent practice on virtually all aspects;

storing overall but not exemplary; no weaknesses of any real consequence. Possibly

80%- 90% agree with statement to a considerable or high degree.

Good: (Mostly, with some exceptions) Reasonably good practice overall; might have a

few slight weaknesses but nothing serious. In the range of 60%-80% agree with

statement to a considerable or high degree and no more than 15% agree to a limited

or very limited degree.

Adequate: (Sometimes, with quite a few exceptions) Fair practice, some serious, but

non-fatal weaknesses on a few aspects. Around 40%–60% agree with statement to a

30 Davidson, E. J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand

Oaks: Sage


considerable or high degree and no more than 15% agree to a limited or very limited

degree.

Poor: Clear evidence of unsatisfactory functioning; serious weaknesses across the

board on crucial aspects. Probably fewer than 40% or more agree with statement to a

considerable or high degree.

As described there, the evidence for testing the evaluative criteria will come from:

Key informant interviews which combines stakeholder and evaluator expertise but

requires careful justification;

Secondary documentation and data, including prior reviews and assessments if

available.

4.4 Triangulation and Evidence Assessment

Triangulation Protocol

A systematic process of triangulation will be undertaken in order to cross-verify and validate

the findings of this Midterm Evaluation. Two forms of triangulation are possible:

Methodological Triangulation – assessment of the consistency of findings obtained

through the different data collection methods used by this Midterm Evaluation

(document review and stakeholder interviews).

Data / Sources Triangulation – assessment of the consistency of findings between

different stakeholder groups during stakeholder interviews (between donors and the

TSU, for example, and between stakeholders that are internal and external to The

NAMA Facility). Comparisons between groups will depend on there being sufficient

within-group consistency of perception. What constitutes 'sufficient consistency of

perception' within a group will be assessed on a case by case basis by the review

team.

We will use these two types of triangulation to the extent possible. In order to track where

and to what effect triangulation has been used, we are proposing to use a Triangulation

Matrix. It has been developed following the approach of the Global Environment Facility

Evaluation Office (2010)31. The Triangulation Matrix includes each of the evaluation questions

31 Global Environment Facility Evaluation Office., (2010).Methodological Note on Triangulation Analysis in

Country Portfolio Evaluations [Online]. Available at:


from the Evaluation Framework. Preliminary findings from each of the data collection

methods (and key stakeholder groups, where appropriate) will be added to the Triangulation

Matrix against the evaluation questions during brainstorming sessions at the end of each

data collection phase (after document review, after the Insider interview phase and following

the "outside" interview phase). During these sessions the evaluation team will discuss each

evaluation question in turn to identify relevant preliminary findings that have emerged

during the phase. The strength of evidence for these preliminary findings will be iteratively

monitored against the Strength of Evidence Protocol. After the final phase of data collection,

the evaluation team will ascertain which of the preliminary findings are confirmed or

challenge, in order to draw out the final results. This part of the process will be undertaken

with a final assessment of each finding against the strength of evidence protocol, to ensure

that each final result is triangulated and supported by strong evidence. The Triangulation

Matrix can be found in Annex 2.

It should be noted, however, that it will not be possible to triangulate all evidence that we

will need to use. For example, evidence on UNFCCC negotiators’ perceptions, for example

regarding the relevance of The NAMA Facility, is highly subjective and may be gained only

through interviews with this very specific group of stakeholders, and it is not possible to

triangulate via other sources. This is mitigated through questioning technique,32 the

strength-of-evidence protocol, through the explicit split into internal and external sources

and the evaluation phases, and through devoting a separated work stream to “perceptions”

which is targeting exactly these soft and subjective aspects. Here we will juxtapose all

external views to provide a picture of the voiced perceptions, which will help us understand

whether these pose an actual challenge to the effectiveness of the NAMA Facility, or not, and

what can be done about them.

Strength of Evidence Protocol

The Strength of Evidence Protocol that will be applied during this Midterm Evaluation is a

score card within the Triangulation Matrix (see Annex 2) that the evaluation team will use to

assess the strength of evidence of findings as they emerge. The protocol will include an

assessment of the confirmatory strength of evidence in relation to confirming or refuting the

change hypothesised through the theory of change.

https://www.thegef.org/gef/sites/thegef.org/files/documents/CPE_Triangulation_Analysis.pdf [Accessed 26th May 2016].

32 We will always ask open questions first where ever possible, but have our hypothesis in the background for

follow-up questions. The hypotheses also help us to ask about things that might not be in the forefront of the discussion with the stakeholders but might have come up in our internal discussions.

https://www.thegef.org/gef/sites/thegef.org/files/documents/CPE_Triangulation_Analysis.pdf


Emerging findings will be monitored iteratively at the end of each data collection phase to

enable the team to identify when there is sufficient evidence for an emerging finding to be

considered a final result. This will ensure that subsequent data collection can be focused on

data gaps and building strength of evidence against evaluation questions where emerging

findings are unclear. At the end of data collection, a final Strength of Evidence Assessment

will be made against each final result to ensure that only results that are sufficiently

triangulated and supported by evidence are included in the final report. Table 10 summarises

strength of evidence protocol.

Table 10. Strength of Evidence protocol

Strength of Evidence Criteria

High Triangulation secured through multiple

sources, no conflicting alternative findings

emerging. High levels of confirmatory

evidence.

Moderate Triangulation of at least two sources;

alternative findings present but poorly

supported by confirmatory evidence.

Low / Non-Viable Insufficient triangulation; conflicting

alternative findings; insufficient evidence for

a clear overarching finding. No confirmatory

evidence.

4.5 Limitations While the evaluation plan has been designed with critical consideration of evidence quality

and the determination to avoid bias, there are several biases that the evaluation team will

have to manage actively during the evaluation itself. These are specifically confirmation bias

and critical case sampling bias.

Confirmation bias is inherent in the selected modality, where hypotheses are formulated and

then triangulated with a successively wider range of stakeholders. In these cases, putting

forward the hypothesis sometimes induces the effect of the socially desired response, which

often implies a confirmation. The mitigation strategies that the team will intentionally utilise

are to ask open questions instead of leading questions that rephrase the hypothesis, and the

strength of evidence protocol which lend a comparatively stronger weight to written

information than to verbally conveyed information. The triangulation protocol will make very

clear the sequence of sources taken on board so that the results can be critically assessed.


The sampling for stakeholders, in particular for rejected projects and for other finance

institutions, will apply a “critical case” scheme. This implies that Project Outlines that almost

were funded, will be interviewed. Several biases can be introduced by this sampling scheme,

one being that the evaluators wrongly assess who actually is a critical case. The second major

bias being that these critical cases have particularly tainted perspectives on the Facility. The

evaluators will hedge against these latter biases by being aware of them and putting the

statements into context. Also, triangulation and the strength of evidence protocol will help

assess the value of the information. Wherever possible, third party perspectives will be

sought, for example by asking the “critical cases” to confirm each other’s assessments.

Conversely, interviewing the recipients of the funds tends to give overly positive answers

which also has to be taken into account.

Access to data is not considered a major issue here as we are mainly focusing on procedural

and perception issues. Commercial confidentiality of the private sector and privacy

protection of individuals as well as all other provisions of the United Nations Evaluators’

Group (UNEG) Code of Conduct will be maintained.

We do anticipate addressing all evaluation questions, such as those relating to development

and submission of the NSP. However, due to the scope we do not expect to get a detailed

understanding of all of the intricacies of each of the successful or unsuccessful NSP, nor the

associated NAMAs. This also means for example that we are not necessarily reassessing the

NSP itself in relation to its ability to generate transformational change or the soundness of

the financial mechanism.

A major limitation to this Midterm Evaluation is the dynamic development around NAMAs

on the country level and around financing NAMA implementation in the global Climate

Finance Arena. The projects from calls 1 – 3 were all developed before the Paris Agreement.

Internal reviews of The NAMA Facility already led to significant changes in the design of the

next call of the Facility. In that sense, the findings, lessons and recommendations that are

based on evidence from calls 1 – 3 might already be taken on board in the fourth call. In

order to keep the evaluation from falling into irrelevance the evaluation team will take the

following precautions:

Integrate the internal learning documents and the changed design of the fourth call

in the vetting of findings, lessons and recommendations;

Include in these also an assessment of how likely the changes in the fourth call are to

address the potential issues that have been identified.


5 Approach

5.1 Overview of Evaluation Phases In expanding on the approach described in the tender, a more staggered evaluation

approach was presented by the Evaluation Team at the kick-off meeting with the client. This

approach proposes three phases:

A Desk Review phase;

An Internal Perspective phase; and,

An External Perspective phase.

These phases also correspond to the steps proposed in the Evaluation Matrix (Annex 1).

5.2 Desk Review After the inception period, the Evaluation Team will commence with an in-depth document

review and portfolio analysis. Based on the evaluation questions, this will lead to hypotheses

about the reasons for the performance characteristics of The NAMA Facility. Regular

interaction with the TSU will be necessary in order to assure the completeness of the

documentation. Ad hoc, these interactions might take on the nature of interviews, but the

Evaluation Team will pay heed to record whether information is documented “on paper” or

transmitted verbally. Major outputs of the desk review phase by work stream are:

A draft reconstructed Theory of Change (Work Stream A);

Partially filled instruments for the portfolio analysis (Work Stream B);

A draft map of funding gaps (Work Stream C);

A draft instrument for the Institutional Systems Review with partial information (Work

Stream D); and,

A map of the project cycle with actual times (Work Stream E).

Triangulation discussion 1 will take place at the end of this phase within each work stream to

identify emerging hypotheses and findings to be tested in later stages.

This listing illustrates that the desk review phase might not take the same amount of time for

each work stream as the volume of documentation that needs to be reviewed in each work

stream is quite different, and some of the work streams build upon each other. However, the

team will strive to coordinate this as much as possible, and is prepared to continue

document review throughout the evaluation (see section 5.4 Continued Document Review).


5.3 The Internal Perspective Phase In the Internal Perspective phase, the hypotheses identified in the desk review phase will be

discussed with internal, ‘insider’, stakeholders, including:

The TSU;

The ESG and donors;

The proponents and implementers of approved NAMA projects.

The main purpose of this phase is to complement, confirm and/or clarify the findings from

the written documentation. This phase will lead to the formulation of preliminary findings.

The triangulation matrix will help understand where evidence for preliminary findings has

been confirmed by several sources during Triangulation Discussion 2 at the end of this

phase. The findings will be discussed with the ESG in a debriefing meeting in late July, with

the intention to expand on the hypotheses, on the one hand in terms of their evidence base

but on the other hand already with respect to potential preliminary lessons and

recommendations.

In addition, the interviews with the project proponents and DOs will serve to fill gaps in the

instruments for the portfolio assessment.

The outputs from this phase will be:

A finalised reconstructed Theory of Change (Work Stream A);

More complete instruments for the portfolio analysis (Work Stream B);

An internally validated map of funding gaps (Work Stream C);

A filled draft instrument for the ISR (Work Stream D), to be validated and amended in

the further process;

Comments and preliminary findings on the project cycle with actual times (Work

Stream E).

These outputs will serve as final lists of hypotheses for testing in the external phases. In

addition, at this point expectations for what kind of evidence could solidify the case in the

course of further triangulation will be defined. This will help strengthen the basis for

conclusions and findings.

5.4 Continued Document Review As Needed The Evaluation Matrix indicates for some evaluation tasks that a second desk review might

be necessary to develop findings and hypotheses further. In most cases the reason for this is

that only after the first interviews, the appropriate documentation (e.g. for comparison with

other financing mechanisms) can be identified. In other cases, hypotheses and findings


might stem from the interviews and might have to be triangulated with documents. This

second document review will also be documented in the Triangulation Matrix. It will partially

provide information related to the internal perspective, and partially to the external

perspective, so it cannot and does not have to be logically contained in one of these phases

but will be conducted in line with the timing of the work streams.

5.5 The External Perpective Phase As specified in the Evaluation Matrix, many of the Evaluation Questions and Tasks will benefit

from discussion with outside stakeholders. These will differ according to work stream, but in

many cases more than one work stream will have to approach the same stakeholder. The

team will conduct joint interviews to minimise the logistical and time burden on interviewees.

The main purpose of separating the external interviews from the internal stakeholder

interviews is to be able to clearly distinguish between these two perspectives which might

diverge in important points. Specifically, work stream F, focused on perceptions of the NAMA

Facility, will draw significantly on external views. Still, external experts will also provide views

that are relevant for further triangulations at least for some questions, and in particular it

should be possible to reject or confirm some of the hypotheses, preliminary findings, lessons

and recommendations.

The external view phase will result in such findings for all work streams (these can be seen in

the evaluation matrix Annex 1). It will end with a debriefing meeting in which these are

discussed with the TSU and ESG.33

5.6 Data Analysis and Reporting While data analysis is part of all earlier phases, in the final phase the team will summarise the

findings from all work streams on the basis of the Triangulation and Strength of Evidence

Matrix into a concise, 30-page report (as specified in the TOR), that documents the findings,

lessons and recommendations and briefly summarises the evidence base. Where extensive

discussions of evidence etc. are necessary, these can be provided in annexes. The draft will

be circulated to the evaluation manager (TSU) and the ESG for fact checking and comments,

leading to a revised final report.

The report will be published by the donors in line with their respective publication policies.

33 Ideally this phase would take place during August 2016, but the review team appreciates access to

stakeholders may be difficult during this time period due to holiday periods.


6 Work Plan

This updated timeframe responds to a request from the project kick-off meeting (10/5/16) to

document the new timeframe proposed for the project. The indicative timeline below (Table

11, Figure 7) will be reviewed and confirmed by the client during the inception report review.

At this point, the inception timeframe is fixed. Timings beyond the inception phase will be

re-affirmed once there is confirmation of a contract extension from the client. The timings

listed below are in line with key criteria set out in the contract.

6.1 NAMA Facility Midterm Evaluation Key

Dates The indicative key milestones34 timeline that will be reviewed and confirmed upon approval

from the client is presented in Table 11. Important dates for logistical arrangements are the

planned interviews:

TSU week of July 4th and 11th

Donors between July 15th and 22nd

Implementers the week of July 23rd.

The support of TSU and donors in accessing persons would be appreciated and will be

coordinated by the Team Leader through the TSU.

Table 11: Indicative Key Milestones Timeline

Deliverable/Milestone Date

Kick Off Meeting 10th May

Official start of inception 23rd May

Inception Report 8th June

Client De-Briefing

Meetings

Week of 25th – 29th July

Week of 19th – 23rd

September35

34 Subject to client approval of a contract extension

35 Assuming client comments are back 9 weeks prior to this in line with the TOR


Draft Final Report 14th October

Final Report 4th November36

6.2 Updated Work Plan The indicative work plan that will be confirmed upon approval from the client is

demonstrated in Figure 7. Please see Annex 4 for the full work plan.

Figure 6. Indicative Work Plan

6.3 Additional Personnel Based on some of the limitations listed in section 5.8, and on the external review of the LTS

proposal for this work (which called for greater inputs on program management), it is the

belief of the evaluation team that there would be significant value in including additional

evaluation consultants to the midterm evaluation team. To supplement the existing team

strengths, the team believes that the review would be further enhanced by additional

program management evaluation capabilities, as well as implementation experience and

familiarity with the current climate finance situation. Therefore, we have identified the

36 Assuming client comments are received on the draft report within 2 weeks, in line with the TOR


following candidates to formulate a consultant pool, in order to increase the quality and

reach of this midterm evaluation.37

Dr Martina Greib

Dr. Martina Greib is a Senior Evaluator / Project Director at Arepo Consult and focuses on

climate change mitigation and adaptation policy and programme analysis, communication,

monitoring and evaluation. On behalf of UNEP’s Evaluation Office, Martina is currently

evaluating UNEP´s “Facilitating Implementation and Readiness for Mitigation” (FIRM) project

as well as the UNEP/UNIDO CTCN programme. CTCN is the implementation arm of the

UNFCCC Technology Mechanism. FIRM was designed for national “quick start” mitigation

actions to assist developing countries in systematizing and scaling up their efforts towards

sustainable development with a focus on NAMAs in. It is one of the projects that would

develop finance-ready NAMAs for The NAMA Facility.

Martina has 20 years’ worth of experience in development assistance (DA), with short-term

and long-term assignments of development projects. For three years, she has been

implementing a DA programme in the Dominican Republic. Other programs for the

contracting agencies GIZ, EU and World Bank have brought her to Asia, Africa and Latin

America. With a Doctorate in Agricultural Economics, she is particularly able to understand

sectoral approaches and their integrated effects, on economics and carbon. She is fluent in

German, English and Spanish and has working knowledge of French.

William Battye

William is an economist with experience in the development, implementation and evaluation

of climate finance programmes and climate policy across Asia and Africa. William has a

background in the private sector and for the international public sector and has developed

projects for the carbon market, climate investment funds and other results based systems.

He has developed NAMAs and results based funds for country governments and

methodologies for the UNFCCC. William has led the development of NAMAs in energy and

water sectors and has developed climate finance investment vehicles in Singapore, Kenya

and Ethiopia. William has led the evaluation of climate investment programmes in Ethiopia,

Indonesia, Vietnam, Philippines and Cambodia. William holds and MSc in Development

Finance and a BA in Economics from the University of Manchester.

37 It should be noted that this work will still be delivered within the financial budget detailed in the original

contract, despite adding additional personnel to the team.


Annexes

Annex List

Annex 1- Evaluation Matrix

Annex 2 – Strength of Evidence Protocol and Triangulation

Matrix

Annex 3 – Documentation Informing Inception Phase

Annex 4 - Full Workplan

Annex 5 Original Terms of Reference


Annex 1 - Evaluation Matrix

Work Stream FINDINGS REGARDING

method DESK REVIEW DESK REVIEW STAKEHOLDER INTERVIEWS

"External"

1

• Examine the strategic

objectives of the NF (from the

climate finance perspective)

A.1. reconstructed

TOC, understand

relevance and

effectiveness of NF

wrt strategic

objectives

A.1.1 Desk review of

documents to formulate

a consolidated strategic

objective and a

reconstructed TOC

A.1.2 a discuss validity

of these formulations

with stakeholders /

ESG

1.4.a discuss

validity of these

formulations with

external

stakeholders

1.5 Reconstructed TOC of the facility

1.6 Findings on level of achievement

wrt outputs 1 and 2

1.7 Recommendations on

improvement of effectiveness

A.1.2.b discuss extent

of achievements on

these objectives with

ESG

1.3.b confirm

achievements

through desk

review

1.4.b. discuss

extent of

achievements on

these objectives

with external

stakeholders

2

• Consider and incorporate

lessons learnt from other

climate finance schemes and

from similar funding

mechanisms (e.g. from GEF,

ICF, CIF and from an EC

program run under CfP mode)

A.2. Testing of

assumptions

underlying the TOC

2.1 extraction of

assumptions from

foundational documents

(e.g. GIZ/KfW proposal,

DECC business case,

DANIDA appraisal

document)

2.2 review of

implementation

documents if

assumptions held true

2.3 discussion of

findings from desk

review with TSU

2.4 discussion of

findings from desk

review and from

discussion with

TSU with ESG

2.5 Comparison of

preliminary findings

from 1.4 and 2.4

with evaluation

reports and

learning documents

of GEF, ICF and CIF

2.6 validate with

GEF, GCF, PMR,

UNFCCC

2.7 Findings regarding the

assumptions in the TOC and whether

they conform with international

experiences and integrate the

lessons from other Climate Finance

Mechanisms

2.8 Recommendations on risk

mitigation and enhanced efficiency

3

• Examine the role of the NF in

the climate finance

architecture – its niche vs. the

GCF

• How relevant is the NAMA Facility for potential

submitters of project outlines?

• What is the conceptual understanding of NAMAs of

different stakeholders? Which expectations do they

have towards the NAMA Facility?

C.3. mapping of

funding gaps

3.1 mapping of funding

gaps in 2011 and today

3.2 validate with GIZ,

KfW, BMUB, DECC,

MEUC, EU

3.3 validate with

GEF, GCF, PMR,

UNFCCC

3.4 findings regarding the role of the

NF as compared to other financing

mechanisms

3.5 Recommendations on enhanced

strategic relevance wrt climate

finance landscape

4


objectives of the NF (from the

donor perspective)

• How relevant is the NAMA Facility and its current set-

up for NAMA Facility Donors? Which priority and

importance has the NAMA Facility for its Donors? combined with A.1.

4.1 discuss result of

1.1 with GIZ, KfW,

BMUB, DECC, MEUC,

EU

4.2 Findings on match of Donor

priorities with NAMA objectives and

results

4.3 Recommendations on enhanced

strategic relevance wrt donor

priorities

5

• Examine the governance

structure of the NAMA Facility

6

• Examine the division of work

between the donors and the

TSU

7

• Examine specific procedures

for donor involvement,

consultation and

communication processes, and

funds management

• To what extent does the current governance

arrangement promote learning within the NAMA

Facility, amongst donors and with wider audiences?

Are any improvements necessary? If so, what?

• How is the communication within the TSU, between

the Delivery Organizations (GIZ, KfW and Third Party

Delivery Organizations) and TSU and between TSU and

the Donors?

• Do the NAMA Facility’s funding conditions permit

Delivery Organizations and implementing partner

organizations to operate efficiently?

D.7. ISR (combined

with ISR in 5.)

7.1 add rubrics to 5.1

7.2 analyze documents

wrt procedures; map

procedures

7.2 interview individual

TSU members

regarding procedures

7.3 interview

individual donor

representatives

and DOs regarding

procedures

F.20.4 interview

DOs and DO

candidates on

perceptions of

procedures

7.5 Findings on procedures between

Donors and TSU; recommendations

5.5 Findings on governance structure

and division of work between TSU

and donors; recommendations

D.5. ISR (Thread 1

Management)

• What is the degree of delegation of decision

competence to the TSU and is this level appropriate?

Workflow / sources

STAKEHOLDER INTERVIEWS

"Internal"

5.1 development of

rubrics / criteria

5.2 application of rubrics

to desk review

5.3 discussion of

findings from desk

review with TSU

5.4 discussion of

findings from desk

review and from

discussion with

TSU with ESG

Tasks from TORs Example questions from TORs (where applicable - do

not necessarily reflect the full scope of the task, not

cros

s-cu

ttin

g is

sues

/ C

limat

e fi

nanc

e an

d cl

imat

e ne

goti

atio

ns

• What is the theoretical basis (theory of change) of

the NAMA Facility? Are the objectives clearly defined?

Is there a joint understanding of the theory of change

among major stakeholders?

• To what extent, in what circumstances and why has

the NAMA Facility achieved outputs 1 and 2, namely:

The NAMA Facility is established as a mechanism

which efficiently allocates support to the

implementation of ambitious, transformative NAMAs;

and the quality of the project pipeline has improved.

Gov

erna

nce

/ In

stit

utio

nal S

yste

m /

dec

isio

n m

akin

g st

ruct

ure




"External"

8

• Examine the organizational set-

up of the TSU and the

management of the NAMA

Facility

• How do NSP monitoring and evaluation systems feed

into the NAMA Facility framework?

D.8. ISR (combined

with ISR in 5.)


8.2 analyze documents wrt

procedures; map

procedures


TSU members regarding

organizational setup

8.3 interview

individual donor

representatives

and DOs regarding

organizational

setup

F.20.4 interview

DOs and DO

candidates on

perceptions of

organizational

setup

8.5 Findings on organizational setup

between Donors and TSU;

recommendations

9

• Examine the role of KfW/GIZ

as trustees and delivery

organizations

• What is the degree of delegation of decision

competence to the TSU and is this level appropriate?

D. 9. ISR (combined

with ISR in 5.)


9.2 analyze documents wrt

role of KfW/GIZ; map role

of KfW/GIZ


TSU members regarding

role of KfW/GIZ

9.3 interview

individual donor

representatives

and DOs regarding

role of KfW/GIZ

F.20.4 interview

DOs and DO

candidates on

perceptions of role

of KfW/GIZ



10

• Examine the internal

organization of the TSU (work

flow, communication, decision

making, potential for learning)

• How efficient is the financial management of the NF

(controlling, disbursement of funds, procurement and

contracting)?

• Is the staffing of the TSU sufficient and adequate?

D. 10. ISR (combined

with ISR in 5.)


10.2 analyze documents

wrt procedures; map

procedures

10.2 interview

individual TSU members

regarding procedures

10.3 interview

individual donor

representatives

and DOs regarding

procedures

F.20.4 interview

DOs and DO

candidates on

perceptions of

procedures



11

• Examine the communication /

support that the TSU provides

with regard to projects and

applicants

• How efficient and effective is the communication

between TSU and NSP proponents?

• Are project cycle, technical support, engagement

activities and funding instruments adequate to achieve

the NAMA Facility outcomes? Are any improvements

necessary? If so, which ones?

• How is the communication within the TSU, between the

Delivery Organizations (GIZ, KfW and Third Party Delivery

Organizations) and TSU and between TSU and the Donors?

• Are the eligibility criteria for qualified Delivery

Organisations equally understood by all stakeholders?

What are reasons to rule out third-party Delivery

Organisations?

• Are the advisory services provided by the TSU sufficient

and adequate? How are they judged by partners?

• How do NSP monitoring and evaluation systems feed

into the NAMA Facility framework?

11.1 analyse documents;

map project cycle, NSP

approval and contracting

processes, technical

support activities

undertaken by TSU with

NSP proponents, TSU and

DOs, TSU and donors,

engagement /

communication processes

within TSU, between TSU

and proponents, describe

funding instrument and its

processes, map eligibility

criteria for qualified

delivery organisations

11.2 interview TSU

regarding processes and

eligibility criteria,

special focus

communication and

support to NSPs

11.3 interview

donors regarding

processes and


special focus

communication,

and support to

NSPs

11.4 interview DOs

and DO candidates

on procedures and


11.5 Findings on adequacy of project

cycle and related processes,

procedures, support provided;

recommendations on improving the

project cycle and related processes,

special focus communication and

support to NSPs

Tasks from TORs Example questions from TORs (where applicable - do not

necessarily reflect the full scope of the task, not

necessarily clearly attributable to just one task)

Faci

lity

man

agem

ent

Workflow / sources


"Internal"




"External"

12

• Examine the procedures for

NSP approval and contracting

• Is the project cycle/funding system effectively

supporting the objectives of the NAMA Facility?

• How effective and efficient are the assessment

processes for NSP outlines and for NSP project proposals?




• Which effects were intended, which were not? Are

influences visible, if so, which ones?

As 11.1, special focus NSP

approval, contracting,

funding conditions

As 11.2special focus on

NSP approval,

contracting, funding

conditions

As 11.3 special

focus on NSP

approval,

contracting,

funding conditions

As11.4 special focus

on NSP approval,

contracting, funding

conditions

As 11.5 special focus on NSP approval,

contracting, funding conditions

13


project assessment, selection,

M&E and quality assurance



the NAMA Facility outcomes? Are any improvements

necessary? If so, which ones?

• Are the outline and proposal templates concise enough

to make the applicants understand what the NAMA

Facility is looking for?

• How effective and efficient are the assessment

processes for NSP outlines and for NSP project proposals?

• Is the project selection process suited to select those

projects that are the most relevant ones for achieving the

objectives of the NAMA Facility?

• Are the work processes between Delivery

Organisations, and between Delivery Organisations and

the TSU aligned? Are there any losses or friction and/or

unclear regulations?

• Are the monitoring and evaluation mechanisms

As 11.1, special focus on

project assessment,

monitoring, quality

assurance and whether

most relevant projects are

being selected

As 11.2, special focus on

project assessment,

monitoring, quality

assurance and whether

most relevant projects

are being selected

As 11.3, special

focus on project

assessment,

monitoring, quality

assurance and

whether most

relevant projects

are being selected comparison with GEF

As 11.4, special

focus on project

assessment,

monitoring, quality

assurance and

whether most

relevant projects

are being selected

As 11.5, special focus on project

assessment, monitoring, quality

assurance and whether most relevant

projects are being selected

14


building the quality of the

pipeline including the extent

and usefulness of

communication between the

TSU and project submitters as

they develop their bids, and in

providing feedback following

submission.



• How has communication between TSU and potential

NSPs been effective at achieving NAMA objectives? What

could be improved? How has the TSU supported new NSPs

and delivery organisations in successfully applying for

NAMAs?

• Are the advisory services provided by the TSU sufficient

and adequate? How are they judged by partners?



E.14/17 mapping of

project cycle and

response times;

validation and

benchmarking,

analysis of the root

causes for delays;

combine with 20

As 11.1, with special focus

on the bid development

process

As 11.2 with special

focus on the bid

development process


focus on the bid

development

process


focus on the bid

development

process

• Does the current General Information Document

provide sufficient guidance for potential submitters and

for Delivery Organizations?

• Are the outline and proposal templates concise enough

to make the applicants understand what the NAMA

facility is looking for?

E. link to 11 and 14 but

with special focus on

the GID and templates


on GID and templates


focus on GID and

templates


focus on GID and

templates


focus on GID and

templates As 11.5 special focus on GID

"Internal"




Sele

ctio

n a

nd

co

ntr

acti

ng

pro

cess

fo

r co

nce

pts

an

d p

ipel

ine

dev

elo

pm

ent

Workflow / sources





"External"

15

• Examine the eligibility criteria

for qualified Delivery

Organizations.

• Are the eligibility criteria for qualified Delivery

Organisations equally understood by all stakeholders?

What are reasons to rule out third-party Delivery

Organisations?



• How does the NAMA Facility consider gender and



formulating

hypotheses regarding

how eligibility criteria

for DO and other

aspects can influence


on eligibility criteria for

DOs


focus on eligibility

criteria for DOs



criteria for DOs



criteria for DOs

Findings regarding effectiveness of

each eligibility criterion (are the

contributing relevant information? Are

the influencing the portfolio?

Recommendations on changing the list

of criteria

16

• Examine the eligibility criteria

and instruments for concepts

and projects.



• How effective are the NAMA Facility criteria in ensuring

that the highest scoring projects meet the Facility’s

objectives?

• How is ‘implementation readiness’ being interpreted?

• How is the concept of ’transformational change’

implemented? (in the criteria)

• How is ‘implementation readiness’ being interpreted?


Human Rights issues?



formulating

hypotheses regarding

how effective these

eligibility criteria are

(are the contributing

relevant information?)

and if eligibility

criteria can influence

the portfolio

E16.1 add to E11.1 analysis

of key policies and

(assessment) guidance

document, review criteria

for representation of

general principles such as

Transformational change,

implementation

readiness, gender and

human rights, which are

expected to be covered in

the criteria

E16.2 interviews with

TSU and external

assessors for utilization

of criteria for

transformational

change, implementation

readiness, gender and

human rights

E16.3 interviews

with donors

regarding criteria

for

transformational

change,

implementation

readiness, gender

and human rights

cf. B: assessment of

evidence in the

portfolio regarding

transformational

change,

implementation

readiness, gender

and Human Rights

E16.4 discussion

with rejected

proposals regarding

missed

opportunities?

Findings regarding effectiveness of

each eligibility criterion (are the

contributing relevant information? Are

the influencing the portfolio?

Recommendations on changing the list

of criteria

17

• Examine the project cycle of

NSPs from NSP outline to

proposal and implementation +

evaluation (timing,

costs/resources)





E.17 mapping of

project cycle and

response times;

combine with 14

E.11 / E.14/17.1 mapping of

project cycle and response

times; draw on portfolio

review (B.18) for actuals

E.14/17.2 discuss project

cycle and average

response times with TSU

Findings regarding

- length of project cycle

- causes for delays, factors that support

acceleration

- readiness of countries, including

typical readiness gaps

recommendations on accelerating

NAMA approval and implementation

18

• Examine the portfolio (desk

analysis) consisting of 9 NAMA

Support Projects from two calls

2013-2014 with regard to the

extent that such projects are a

good fit with Nama Facility’s

objectives, in particular with

regards to delivering

transformational change and

ensuring projects are

embedded with government’s

national strategies?




• How effective are the NAMA Facility criteria in ensuring

that the highest scoring projects meet the Facility’s

objectives?



• How do NSPs intend to measure their impacts?

• How is the concept of ’transformational change’

implemented? (in the projects)

• How has the NAMA Facility contributed to induce

reflections on how to achieve transformational change in

partner countries and/or in sectors?


Human Rights issues?

• What are the greatest challenges facing successful

applicants in implementing NSPs and meeting NAMA

requirements? How can the NAMA Facility support NSPs

in the future?

B. portfolio analysis

using rubrics;

"modified process

tracing" using the

portfolio to test

hypotheses from 15

and 16.

B.18.1 develop rubrics that

apply evaluation

questions

B.18.2 apply rubrics to

each of the 9 projects in

the portfolio

B.18.3 formulate

preliminary findings on

the assessment

B.18.4 complement desk

review with interviews

with project proponents

and local stakeholders

to complete information

base

B.18.5 discuss

findings with the

TSU and ESG

B.18.6 compare

findings with criteria

and hypotheses

formulated within A,

D, E and potentially

with findings from

reviews of other CF

mechanisms; apply

"modified process

tracing"

B. Findings regarding

- expected project achievements, their

measurements, and their contribution

to the NAMA Facility's strategic

objectives

- typical challenges facing successful

applicants in meeting NAMA and

NAMA Facility requirements and

implementing NSPs

- the degree to which criteria and

procedures have influenced project

design

- transformational change, gender,

Human Rights

Workflow / sources





"Internal"

crit

eria

fo

r co

nce

pts

an

d D

Os

NA

MA

Su

pp

ort

Pro

ject

s




"External"

19


objectives of the NF (with

respect to incentivizing

replication / scale-up)

• To what extent does the current governance

arrangement promote learning within the NAMA Facility,

amongst donors and with wider audiences? Are any

improvements necessary? If so, what?

• How has the NAMA Facility contributed to the

international climate finance debate, and to efforts to

influence climate relevant national and international

policies?

• Is the TSU successful in sharing its experiences and

lessons learnt with other stakeholders in the

international climate finance discussion?

A.19 TOC analysis;

combine with A.1;

focus on replication

aspect of TOC,

effectiveness of

learning and outreach

products

A.19.1 complement Desk

Review under A.1.1 with

review of learning and

outreach products; analyse

reach figures

A.19.2 interview TSU,

ESG and direct advisors

for specific incidents

where learning /

influencing of climate

finance debate has

taken place and scale-

up/replication A.19.3 validate A 19.2

A.19.4 validate

A.19.3/2 with

outside

stakeholders

A.19 Findings regarding learning and

influence climate finance debate• How do Delivery Organizations and partners value the

activities of the NAMA Facility and of the TSU?

• How relevant is the NAMA Facility for potential

submitters of project outlines?

• How has the NAMA Facility contributed to the

international climate finance debate, and to efforts to

influence climate relevant national and international

policies?

• Examine the reputational

effect of NSP Outline rejections

and selections among

proponents of rejected NSPs

• Is the TSU successful in sharing its experiences and

lessons learnt with other stakeholders in the

international climate finance discussion?

• What is the conceptual understanding of NAMAs of

different stakeholders? (incl. rejected proposals) Which

expectations do they have towards the NAMA Facility?

Workflow / sources





"Internal"

NA

MA

s n

ot

sup

po

rted

; rep

uta

tio

n a

nd

rep

licat

ion

20

F.20.2 interview

purposive sample

of outside

stakeholders wrt

their perception of

relevance,

effectiveness,

efficiency and

impact of the

NAMA Facility and

TSU (35 interviews).

F.20 Findings regarding

- the views of the various stakeholder

groups;

- how they compare with the evidence

provided in the other workstreams

Recommendations on how to

overcome negative perceptions if any

and enhance impact of outreach.

F.20.1 interview DOs

and partners wrt their

perception of relevance,

effectiveness, efficiency

and impact of the NAMA

Facility and TSU (15

interviews).

F. Perception

assessment


Annex 2 – Strength of Evidence Protocol and Triangulation Matrix

NAMA Facility Mid-Term Review

FINDINGS EMERGING FINDINGS PRELIMINARY FINDINGS

FINAL RESULTS

STRENGTH OF EVIDENCE

METHOD DOCUMENT REVIEW

STAKEHOLDER INTERVIEWS HIGH Multiple triangulation; no conflicting findings. High level of confirmatory evidence.

SOURCES "Insiders" "Outsiders" MEDIUM Triangulation with at least two ways; alternative findings present but poorly supported with confirmatory evidence.

KEY EVALUATION QUESTIONS

Group 1

...

... Group 1

... LOW/NON-VIABLE

Insufficient triangulation; conflicting alternative findings; insufficient evidence for clear overarching finding. No confirmatory evidence.


Annex 3 – Documentation Informing Inception Phase

Document

Number

Author Year Document Title Type of

Literature

1 NAMA Facility 2015 General Information

Document

Online

Report

2 NAMA Facility 2016 Terms of Reference VN

81197008: Mid-term

evaluation of the

NAMA Facility.

Project

Document

3 UNDP 2008 Bali Road Map: Key

Issues under

Negotiation.

Online

Publication

4 UNFCCC 2014 FOCUS: Mitigation -

NAMAs, Nationally

Appropriate Mitigation

Actions.

Online

Publication

5 International Partnership

on Mitigation and MRV

Unknown Nationally Appropriate

Mitigation Actions

(NAMAs)

Website

6 Sharma, S. & Desgain, D 2013 Understanding the

Concept of Nationally


Action.

Online

Report

7 UNFCCC NAMA Registry 2016 Supported NAMAs Online

Database

8 European Commission 2014 EU joins UK, Germany,

Denmark in financing

ground-breaking

climate Facility

Online

News

Bulletin

9 Garside, B. 2014 EU nations could

double current NAMA

funding with new call

for projects.

Online

News

Bulletin

10 Soren, D 2016 NAMA Facility:

Supporting the

Implementation of

Nationally Appropriate

Mitigation Actions

Presentation

on 23 May

2016 at the

NAMA

Facility


(NAMAs). Workshop

11 Oakden, J. 2013 Evaluation Rubrics Online

Publication

12 Davidson, E. J. 2005 Evaluation

methodology basics:

The nuts and bolts of

sound evaluation.

Book

13 GEF 2010 Methodological Note

on Triangulation

Analysis in Country

Portfolio Evaluations

Online

Publication

14 Sharma, S. & Desgain, D 2014 Nationally Appropriate

Mitigation Action:

Understanding NAMA

cycle

Online

Publication

15 NAMA Facility 2015 NAMA Facility annual

report 2015

Online

Report

16 NAMA Facility, Technical

Support Unit,

2016 Financial Mechanisms

in the NAMA Facility

Online

Factsheet

17 NAMA Facility, Technical

Support Unit

2016 Financing the

Implementation of

NAMAs – lessons

learned from the

assessment of 3 Calls

of the NAMA Facility

Online

Factsheet

18 Mitigation Momentum,

Editors: Charlotte Cuntz

(Ecofys), Natalie Harms

(ECN), Noémie Klein

(Ecofys).

2016 Status Report on


Mitigation Actions

(NAMAs) Mid-year

update 2016

Online

Report


Authors: Xander van

Tilburg, Sophy Bristow,

Frauke Röser, Donovan

Escalante , Hanna Fekete

2013 Status Report on


Mitigation Actions

(NAMAs) Mid-year

update June 2013

Online

Report

20 PricewaterhouseCoopers

(PwC)

2015 Review of the

assessment process of

the project outlines

submitted to the

Online

Report


NAMA Facility during

the third call

21 Department of Energy

and Climate Change

(DECC)

2015 Nationally Appropriate

Mitigation Actions -

Second Annual Review

Online

Report

22 KPMG 2014 Review of evaluation

process for assessment

of Project Outlines of

the NAMA Facility –

Second Call

Online

Report


Ann Gardiner, Michelle

Bosquet (Ecofys), Xander

van Tilburg and Natalie

Harms (ECN Policy

Studies).

2015 Annual Status Report

on Nationally


Actions (NAMAs) 2015

Online

Report

24 KPMG 2013 Review of evaluation

process for assessment

of Project Outlines of

the NAMA Facility

Online

Report

25 NAMA and Registry Unit

of the Non-Annex I

Support Sub-

Programme of the

Mitigation, Data and

Analysis Programme

(MDA) of the United

Nations Framework

Convention on Climate

Change (UNFCCC)

Secretariat

2015 Key criteria for

selecting nationally

appropriate mitigation

actions for

international support

Online

Factsheet

26 NAMA Facility 2015 NAMA Facility Theory

of Change

Online

Publication

27 NAMA Facility TSU 2016 Information on NAMA

Support Projects and

Technical Support Unit

Document

made

available

from TSU

28 NAMA Facility TSU 2016 Role of the Steering

Committee for NAMA

Facility mid-term

Document

made

available


evaluation from TSU


and Climate Change

(DECC)

2012 Business Case and

Intervention Summary

Document

made

available

from TSU


and Climate Change

(DECC)

2014 Business Case and

Intervention Summary

Document

made

available

from TSU

31 Ministry of Foreign

Affairs of Denmark

Danida

2014 Appraisal on Support

to the Nationally


Actions Facility Under

the Climate Facility

Document

made

available

from TSU

32 NAMA Facility 2016 Procedures Overview Document

made

available

from TSU

33 NAMA Facility TSU 2016 Feedback relating to

calls

Document

made

available

from TSU


Date / week starting 30.05.16 06.06.16 13.06.16 20.06.16 27.06.16 04.07.16 11.07.16 18.07.16 25.07.16 01.08.16 08.08.16 15.08.16 22.08.16 29.08.16 05.09.16 12.09.16 19.09.16 26.09.16 03.10.16 10.10.16 17.10.16 24.10.16 31.10.16

1. Inception Report

- development of report

- finalisation after client comments

(approval of Inception Report)

2. Data Collection Phase 1 - Document

Review

- Develop document review instruments, if

needed

-write up about instruments, if applicable

- Review related to ToC

-Review for Portfolio Analysis

-Review for Climate Finance Mapping

-Review for Process Mapping

-Review for ISR

3. Data Analysis Phase 1

- Team Meeting - triangulation discussion 1;

emerging hypotheses and findings -

development of phase 2 rubrics and

instruments

-documentation of emerging hypotheses and

findings; write up of any finalised

components (process descriptions,

methodlogy so far etc.

-finalisation of rubrics and instruments for

Phase 2

4. Data Collection Phase 2 - "Insider"

informant interviews





-Review for ISR


- Triangulation discussion 2; emerging

hypotheses and findings

-documentation of emerging hypotheses and

findings; write up of any finalised

components (process descriptions,

methodlogy so far etc.

-adjustment of rubrics and instruments for

Phase 3, if needed

6. Data Collection Phase 3 - "Outsider"

informant interviews





-Review for ISR


- Team Meeting - triangulation discussion 3;

final findings

- finalise write up of analysis

8. Debriefing Meetings

9. Report Writing / Revision

-ongoing during data collection and analysis

phases

-development of final report

-draft report QA

-draft report submission

(client comments)

10. Final report submission

Annex 4 –

Full

Workplan

Annex 5 Original Terms of Reference