Evaluating land use and livelihood impacts of early forestcarbon projects: Lessons for learning about REDD+

Susan Caplow a,b,*, Pamela Jagger a,b,c,e, Kathleen Lawlor a,c, Erin Sills d,e

aCarolina Population Center, University of North Carolina at Chapel Hill, CB #8120, Chapel Hill, NC 27599-8120, USAbCurriculum for the Environment and Ecology, 223 E. Franklin St., 207 Coates Building, CB# 3275 University of North Carolina at Chapel Hill,

Chapel Hill, NC 27599-3275, USAcDepartment of Public Policy, University of North Carolina at Chapel Hill, CB#3435 Abernethy Hall, Chapel Hill, NC 27599-3435, USAdDepartment of Forestry and Environmental Resources, North Carolina State University, 3120 Jordan Hall Raleigh, NC 27695-8080, USAeCenter for International Forestry Research, P.O. Box 0113 BOCBD, Bogor 16000, Indonesia

e n v i r o n m e n t a l s c i e n c e & p o l i c y 1 4 ( 2 0 1 1 ) 1 5 2 – 1 6 7

a r t i c l e i n f o

Published on line 12 November 2010

Keywords:

REDD+

Deforestation

Impact evaluation

Forest carbon

Conservation

Socio-economic impacts

a b s t r a c t

The ‘Bali Road Map’ of UNFCCC COP-13 calls for sharing lessons learned from demonstration

activities that aim to reduce emissions from deforestation and degradation and enhance

forest carbon stocks (now known as ‘REDD+’). To develop a feasible yet rigorous strategy for

learning from these REDD+ pilots, it is critical to assess previous efforts to evaluate the

impacts of ‘pre-REDD+’ avoided deforestation projects. Further, because REDD+ remains a

politically volatile issue, with both critics and supporters pointing to the impacts (or lack

thereof) of these pre-REDD+ projects, it is important to critically examine the methods

employed to assess those impacts. We review the body of literature that makes claims about

the socioeconomic and biophysical impacts of pre-REDD+ projects. We find assessments of

outcomes or impacts for only five pre-REDD projects. The design, data collection, and

analysis methods for understanding the impacts of pre-REDD+ projects frequently lack

rigor. In particular, the counterfactual scenarios for establishing socioeconomic impacts are

vague, unscientific, or omitted completely. We conclude that drawing specific lessons from

pre-REDD+ projects for the design or evaluation of current REDD+ projects is tenuous.

Rigorous project evaluations are challenging, expensive, and time-consuming, but because

they are so critical for learning about what works for people and forests, evaluations of

current REDD+ projects must use improved methods. In particular, much better care should

be taken to construct credible – and where possible, consistent – counterfactuals for both

biophysical and socioeconomic outcomes.

# 2010 Elsevier Ltd. All rights reserved.

avai lable at www.sc iencedi rec t .com

journal homepage: www.elsevier.com/locate/envsci

1. Introduction

The ‘Bali Road Map’ of UNFCCC COP-13 calls for sharing

lessons learned from demonstration activities that aim to

reduce emissions from deforestation and degradation and

* Corresponding author at: Carolina Population Center, University of NUSA. Tel.: +1 612 578 8571.

E-mail address: scaplow@email.unc.edu (S. Caplow).

1462-9011/$ – see front matter # 2010 Elsevier Ltd. All rights reservedoi:10.1016/j.envsci.2010.10.003

enhance forest carbon stocks (now known as ‘REDD+’) (United

Nations, 2008). These sentiments were echoed at COP-15,

where REDD+ played a significant role in the Copenhagen

Accord (UNFCCC, 2009). Many sub-national REDD+ projects

have been launched since COP-13, along with concurrent

orth Carolina at Chapel Hill, CB #8120, Chapel Hill, NC 27599-8120,

d.

e n v i r o n m e n t a l s c i e n c e & p o l i c y 1 4 ( 2 0 1 1 ) 1 5 2 – 1 6 7 153

efforts to assess those projects (Sills et al., 2009; Jagger et al.,

2009; Richards and Panfil, 2010).

However, there was already a significant body of experi-

ence with avoided deforestation projects prior to COP-13.

These projects provide fertile ground for understanding the

implications of study design and methods in evaluations of

REDD+ project impacts. Further, REDD+ remains a politically

volatile issue, with debates raging about whether it will meet

its primary goal of reducing global carbon emissions and

whether it will support or undermine local livelihoods and

well-being (Sunderlin and Atmadja, 2009; Olander et al., 2009;

Skutsch and McCall, 2010). With both criticism and support of

REDD+ often based on claims about the experiences of ‘pre-

REDD+’ projects, it is important to critically examine the

methods employed in these studies.

A number of publications aggregate lessons from early

forest carbon projects, whether part of the clean development

mechanism (CDM), activities implemented jointly (AIJ), a

conservation organization’s suite of projects, or the voluntary

market (e.g. Dixon et al., 1993; Karsenty et al., 2003; DiNicola

et al., 1997; Watson et al., 2000; FAO, 2004; Bailis, 2006; Jindal

et al., 2008; Bond et al., 2009; Harvey et al., 2010). These

publications provide brief overviews of each project, and a

series of process and experience oriented ‘lessons learned’.

We are interested in disaggregating this information to

understand the underlying methods that are used to develop

claims about the potential for future REDD+ projects. What

kinds of methods were used to determine the project

outcomes reported in these studies? How do these studies

establish causation (attribution) between observed outcomes

and the interventions? And, finally, what were the outcomes

observed and impacts reported?

In this paper, we review the published and grey literature

on early avoided deforestation projects to assess whether and

how project outcomes/impacts were reported and measured.

For each study, we examine data collection methods,

biophysical and socioeconomic outcomes/impacts reported,

and how the studies establish causation between observed

outcomes and the project (i.e., impact). Finally, we discuss the

implications of our findings for future REDD+ project evalua-

tion design.

2. Background

When assessing the impacts of an intervention, it is important

to consider what would have happened in the absence of the

intervention. This is known as the counterfactual scenario.

Measuring differences between control and intervention sites

is a common way to help estimate the counterfactual.

However, using control site data alone to construct counter-

factual scenarios is insufficient because interventions are

taken up differentially across the landscape depending on

biophysical and socioeconomic characteristics and political

climate. In other words, pre-existing differences between

control and intervention areas influence the probability of

their participation, and these same differences also influence

project outcomes. In the medical field, risks of selection bias are

often addressed by randomly assigning clinical trial partici-

pants to receive either treatment or a placebo (the control).

Randomization (plus further correction with statistical tech-

niques) creates control and intervention groups that are

similar to one another in terms of potential confounders

(including characteristics that cannot be observer or mea-

sured) that affect how they respond to the treatment.

Randomization of forest conservation projects has proven

difficult for a host of reasons (Ferraro and Pattanayak, 2006).

For example, participants are often chosen precisely because

they are unique in ways that might maximize probability of

project success. Despite potential selection bias, these inter-

ventions can still be compared to counterfactuals. Perhaps the

most common statistical approach is to collect information on

potential confounders (observable socio-economic, biophysi-

cal, economic or institutional factors that influence the

outcome measure) and control for these in multivariate

regressions. This approach has some weaknesses, which

have led some researchers to turn towards ‘quasi-experimen-

tal’ impact evaluation techniques instead. Quasi-experimen-

tal techniques employ careful research design and statistical

techniques in order to overcome problems of selection bias

and construct credible counterfactuals. Experimental and

quasi-experimental impact evaluation techniques have

emerged as the standard for evidence-based decision-making

in public policy, particularly in the education, health, social,

and development policy fields (Angrist and Pischke, 2010).

The conservation field has historically suffered from a

dearth of rigorous, ex post, empirical impact evaluations of

both biophysical and human welfare outcomes (Ferraro and

Pattanayak, 2006; Tallis et al., 2008; Sutherland et al., 2004;

Lawlor et al., 2010a; Pattanayak et al., 2010). That said, in

recent years, the body of rigorous empirical studies employing

quasi-experimental techniques to evaluate the impacts of

conservation policies and programs has expanded. Notable

examples quantify the impacts of protected areas on poverty

in Costa Rica and Thailand (Andam et al., 2010) and on

deforestation in Costa Rica (Andam et al., 2008) and Sumatra

(Gaveau et al., 2007); impacts of decentralized management

on forest cover in the Himalayas (Somanthan et al., 2009) and

on welfare in Malawi (Jumbe and Angelsen, 2006) and Uganda

(Jagger, 2010); and the impacts of payments for ecosystem

services (PES) on deforestation in Costa Rica (Arriagada, 2008;

Robalino et al., 2008).

New REDD+ pilot projects could provide an excellent

opportunity to build on this recent trend towards rigorous

evaluation of the causal impacts of forest conservation

interventions. The voluntary carbon market standards require

ex ante estimates of counterfactual land use emissions (often

referred to as the ‘reference level’ or ‘baseline’) and a carefully

constructed argument that the intervention will result in

lower emissions than the counterfactual (often referred to as

‘additionality’). Thus, the voluntary carbon market clearly

embraces the type of ‘counterfactual thinking’ (Ferraro, 2009)

necessary for understanding an intervention’s true impacts.

Ex post testing of those impacts, not only on carbon but also on

local populations, biodiversity, and local ecosystem services is

critical. Further, the potential trade-offs and complementa-

rities between welfare, carbon sequestration, and other

ecosystem services at a given site are best understood if all

impacts are evaluated using the same approach, or at least the

same basic assumptions, for estimating the counterfactual.

e n v i r o n m e n t a l s c i e n c e & p o l i c y 1 4 ( 2 0 1 1 ) 1 5 2 – 1 6 7154

3. Methods

3.1. What is a ‘pre-REDD+’ project?

Our literature review focuses on ‘pre-REDD+’ projects. We

define a pre-REDD+ project as one that: (a) was launched after

UNFCCC COP-1 but before COP-13; (b) is located in a

developing (non-Annex I) country; (c) aims primarily to

reduce deforestation and forest degradation; (d) has estimat-

ed its net impact on greenhouse gas (GHG) emissions. We

include projects on avoided deforestation, avoided degrada-

tion and sustainable forest management, but not those

delivering carbon credits solely through afforestation/refor-

estation (A/R). While we acknowledge that REDD+ design

should be informed by experiences with both A/R and other

types of forest conservation projects (such as CDM, payments

for ecosystem services, and integrated conservation and

development; see Angelsen et al., 2009 and Pfaff et al., 2010 for

reviews), we specifically focus on pre-REDD+ projects because

they are the most similar prototype to new REDD+ projects in

terms of their structure, goals, and the political and social

context.

Based on these criteria, we identified 20 pre-REDD+ projects

(Table 1). Our list includes AIJ projects that are labeled ‘forest

preservation’ as well as one reforestation project and one

agricultural project focusing on forest conservation and

restoration. We also include projects supported by the World

Bank’s BioCarbon Fund, started in 2004 with the objectives of

strengthening the role of forests in climate change mitigation

and creating opportunities for the participation of sub-

Saharan Africa (World Bank Carbon Finance Unit, 2010). One

relevant Plan Vivo project, as well as three of the first projects

Table 1 – List of ‘pre-REDD+’ projects by region.

Project name Coun

Latin America

Bananal Island Carbon Sequestration Project Brazil

Bilsa Biological Reserve Ecuador

Bosques Pico Bonito Hondur

Coopeagri Project Costa R

ECOLAND: Piedras Blancas National Park Costa R

Guaraquecaba Climate Action Project Brazil

Juma Sustainable Development Reserve RED Project Brazil

Madre de Dios Amazon REDD Project Peru

Noel Kempff Mercado Climate Action Project Bolivia

Rio Bermejo Carbon Sequestration Pilot Project Argenti

Rio Bravo Carbon Sequestration Pilot Project Belize

Rio Condor Carbon Sequestration Project Chile

San Nicolas Agroforestry Colomb

Scolel Te/Fondo Bioclimatico Mexico

Africa

Ankeniheny-Mantadia-Zahamena Biodiversity

Conservation and Restoration Project

Madaga

Kasigau Corridor REDD Project Kenya

N’hambita Pilot Project Mozam

Asia

Mawas Peatland Conservation Area Project Indones

Ulu Masen Indones

Total: 20 projects, 5 with outcome/impact documents

validated by the Climate, Community, Biodiversity Alliance

(CCBA) are also included in our sample.

3.2. Literature review

We performed a systematic literature review to identify all

documents that conducted ex post evaluations of our 20 pre-

REDD+ projects or make claims regarding a project’s realized

outcomes or impacts. All forest carbon projects make ex ante

predictions about expected carbon impacts (and occasionally

expected socioeconomic impacts) based on the difference

between expected project outcomes and a projected counter-

factual scenario. We were interested specifically in ex post

(including mid-stream) evaluations of the socioeconomic and

carbon/biophysical impacts actually realized. We searched for

project names, relevant key words (such as ‘carbon sequestra-

tion’ or ‘avoided deforestation’) in Web of Science, Google,

Google Scholar, Digital Library of Commons, ProQuest, CIAO,

Tropical Forest Conservation and Development, and the

reference lists of documents found through these searches.

For each project, at least two authors conducted independent

searches for documents. We also asked project proponents and

other researchers familiar with these pre-REDD+ projects for

other relevant documents. We limited our search to published

articles, books, and reputable gray literature. We included

technical reports and NGO publications, but omitted press

releases and other short summaries. We searched for informa-

tion on all 20 pre-REDD+ projects, but found reports on the

outcomes/impacts of only five projects (Bananal in Brazil, Noel

Kempff in Bolivia, Rio Bravo in Belize, N’hambita in Mozambi-

que, and Scolel Te in Mexico), despite our broad definition of

‘impacts’ and ‘outcomes.’

try Year started Outcome/impact documents

1998 Yes

1996 No

as 2006 No

ica 2006 No

ica 1994 No

1998 No

2006 No

2005 No

1996 Yes

na 2000 No

1995 Yes

2000 No

ia 2007 No

1996 Yes

scar 2008 No

2006 No

bique 2003 Yes

ia 2005 No

ia 2007 No

e n v i r o n m e n t a l s c i e n c e & p o l i c y 1 4 ( 2 0 1 1 ) 1 5 2 – 1 6 7 155

3.3. Analysis

We tabulated data collection methods, outcomes/impacts

reported, and the approaches used to establish counterfac-

tual scenarios. To categorize the studies’ approaches to

counterfactuals, we borrow the Terrestrial Carbon Group

(2009) typology of approaches to establishing deforestation

reference levels/baselines in REDD: (1) extrapolated historical,

which assume past trends are perfect predictors of future

outcomes; (2) adjusted historical, which consider both histori-

cal trends and some kind of ‘development adjustment factor’

to predict future deforestation emissions; and (3) forward-

looking, which use models and simulation to predict future

trends by assuming changes in the underlying factors

driving forest loss. Because this typology emphasizes ex

ante approaches to counterfactuals, we add three methods

for assessing counterfactuals ex post that rely on empirical

evidence collected during or after project implementation:

(4) before–after, where data are collected at just the project

site, and it is assumed that conditions at the site would have

been static in the absence of the project; (5) control-

intervention, where data are collected after project start at

both the project site and a ‘control’ site not involved in the

project that is assumed to represent what would have

happened in the project area without the intervention; and

(6) before–after-control-intervention (BACI), where data are

collected at both the control and intervention sites both

before and after the project, and then changes in outcomes

are compared. We categorized counterfactual approaches to

both socioeconomic and biophysical counterfactuals

employed in the studies.

Finally, to understand each study’s influence on the

public’s, policymakers’, and academics’ opinions of REDD+,

we examined both popular media references and Google

Scholar citation counts. Lexus Nexus and Academic Search

Primier were used to search for references to these pre-REDD+

projects in major English language newspapers and period-

icals from around the world. Search terms included: forest

carbon; carbon offsets; REDD; and the names of the specific

projects that are included in this study. We did not set a time

bound on our search. While Google Scholar sometimes either

overestimates or underestimates the true number of citations

for each document, we believe this is the best way to obtain a

rough quantitative estimate of visibility.

4. Results

The majority of the documents reviewed lacked detail on

research designs and methods. While all six of the

counterfactual typologies were used across the pool of

studies reviewed, many studies gave only cursory attention

(if any) to construction of the counterfactual. Tables 2 and 3

summarize and compare the studies. Out of 13 biophysical

evaluations, 10 described some kind of counterfactual

design, but only 4 compared observations from the project

against a counterfactual (see Table 2 for more detail). Out of

19 socioeconomic evaluations, 7 described a counterfactual

design, and 6 of those actually compared observations on

the project to a counterfactual (Table 3).

4.1. Biophysical outcomes and impacts

4.1.1. Forest carbonThere are two components to estimating a project’s forest

carbon outcomes: the change in forest area and the change in

forest biomass (converted to carbon stocks) in this area. The

forest carbon impact of the project is obtained by comparing

this increase or decrease in total forest carbon at the project

site to the counterfactual. The primary methods used to

determine forest carbon outcomes were document review,

remote sensing, and biomass plots.

Plots are typically measured at project start to determine

carbon density estimates that are applied to both the

counterfactual and project scenarios. We found evaluations

tended to rely on carbon stock estimates from project

proponents (i.e., rarely independently measured) and that

pre-project carbon stock estimates were often used to fill in for

the post-project carbon stock estimate (i.e., transect walks

rarely redone). This has important implications, as it means

that degradation within the project site may go unreported in

either project documents or independent evaluations (Lawlor

et al., 2010b).

To establish the counterfactual area change, a variety of ex

ante methods were used. For example, May et al. (2004) used

the historical regional deforestation rate to establish the

counterfactual; Jindal (2010) added a buffer calculation to

historical deforestation rates to account for unforeseen

changes; and McNally et al. (2009) used modeling software

(GEOMOD) to simulate land use-land cover change over time

and compared deforestation rates in the project area to a

reference area.

All of the documents reviewed concluded that the projects

had a positive impact on forest conservation and reduced net

carbon emissions/removals. The reliability of these conclu-

sions is hard to assess, given that study design and data

collection were generally not carefully described.

4.1.2. Biodiversity and ecosystem services

We found that evaluations rarely measured non-carbon

biophysical impacts or general indicators of forest health.

None of the documents we reviewed collected data (or

established counterfactual scenarios) to assess impacts on

biodiversity, water, or other ecosystem services. In some

cases, biodiversity was suggested as an additional benefit of

land conservation, based on the assumption that greater

forest cover preserves more biodiversity (e.g., Robertson and

Wunder, 2005; TNC, 2009).

4.2. Socioeconomic outcomes and impacts

Key informant interviews and document review were the

primary methods used to quantify socioeconomic impacts,

most often represented by employment and income. In rare

cases household surveys were implemented to assess house-

hold level welfare outcomes (e.g., Jindal, 2010; Hegde, 2010).

Other methods included applications of the sustainable

livelihoods framework and asset pentagons (e.g., Calderon

Angeleri, 2005).

We found a few studies that explicitly established

socioeconomic counterfactuals. Some used a predicted trend

Table 2 – Biophysical impacts of pre-REDD+ projects.

Reference # Years b/nproject

start andevaluation

Datacollectionmethods

Biophysicalcounterfactual

methods

Observableoutcomescomparedagainst a

counterfactual

Reportedbiophysicaloutcomes

Biodiversityinformation

Number ofcitations(5/18/10)(Source:Google

Scholar)

Project: Bananal Island Carbon Sequestration, Brazil

May et al. (2004)/

May et al. (2005)

6/7 Biomass

measurements,

interviewed project

developers

Extrapolated historical:

determined average deforestation

rate between 1986 and

1998 to be 8 per cent in the

region. Acknowledged need for

better counterfactuals

No (1) Lowered C sequestration from

65 milllion tonnes in 30 years to

25 million in 25 years. NO

additionality claimed. Carbon goals

were not realized, had to focus on

‘‘research and social components’’

Project increased

monitoring

knowledge for

biodiversity

(and carbon)

44 (2004)/3

(2005)

Project: Noel Kempff Mercado Climate Action Project, Bolivia

Robertson and

Wunder (2005)

9 Comparative case

study analysis,

semi-structured

interviews

Adjusted historical: assuming

logging would have continued

at a lower rate due to logging

restrictions

No Though additionality is unclear, the

project appears to have had positive

environmental effects in storing

carbon and reducing threats in the

park

Now the park is

larger, can protect

more species

52

SGS (2008)/TNC

(2009)

12/13 625 permanent plots

established in NK-CAP,

remote sensing

Forward looking: area adjacent

to the park served as a

‘‘control’’ for estimate baseline,

then modeled avoided deforestation

and harvest from logging (to

address avoided degradation)

Yes Deforestation: 763 ha saved,

371,650 tCO2e. Degradation:

468,474 sqm of timber was

protected from harvest, avoided

791,443 tCO2e. All from 1997–2005.

Site selected as UNESCO world

heritage site

Additional species

protected in park

expansion area

N/A

McNally et al.

(2009)

13 Document review Forward looking: modeled land

use change using historical

data and reference area

deforestation rate

Yes (1) From 1997–2005 (phase I), park

expanded to 1.58 million hectares;

(2) 1,034,107 mtCO2 sequestered

thus far

None N/A

May et al. (2004)/

May et al. (2005)

8/9 Sustainable

development

methodology,

document review,

stakeholder interviews

None No (1) Additional 634,000 ha lowland

forest protected; (2) 3.59 m tones

of C, down from 14 and 7 mtCO2

estimates. Carbon benefits expected

to last ‘‘in perpetuity?’’ because it’s

a protected area

None 44 (2004)/3

(2005)

Rio Bravo Carbon Sequestration Pilot Project, Belize

USIJI (2000) 5 Document review,

sample plots

Extrapolated historical:

assumed a linear rate of

conversion to agriculture or

traditional logging (only

qualitative assessment)

No 5602 ha reserve and 49,985 ha in

sustainable forestry

None N/A

en

vir

on

me

nt

al

sc

ie

nc

e&

po

lic

y1

4(2

01

1)

15

2–

16

71

56

N’hambita Pilot Project, Mozambique

Jindal

(2010/2004)

7 Document review Adjusted historical: historical

deforestation rates; biomass

assessment, adjusted for buffer

No (1) 11,071 ha of avoided

deforestation; (2) 154,457 tCO2

avoided emissions

None N/A (2010)/7

(2004)

Kooistra and

Wolf (2006)

3 Sample plots,

document review, key

stakeholder interviews

None No Community nursery established

(12,000 N-fixing trees; 23,000

fruit trees; 8.2 of Jatropha plantation

established)

None N/A

Niles (2008) 5 Document review,

project visitation,

interviews; not a

complete evaluation

None No Group on track to measure

carbon accurately

Established lots of

native trees, which

reduces risk of fire

N/A

Final Report

(2008)

5 Sample plots, Landsat

imagery, measured

growth rate on

abandoned plots

Extrapolated historical:

extrapolated historical

deforestation rates

No While they have not measured

the carbon impacts yet, assuming

protection of the entire 8000 ha area

will reduce emissions by

22,900 tCO2/year

Extensive biodiversity

baseline data

collected, impact

only measured by

lack of encroachment

on park

N/A

Scolel Te, Mexico

De Jong et al.

(2007)

9 Interviews, field visits,

GIS mapping

Forward looking: ‘‘Climafor

approach’’ developed for

project-regional baseline,

combination of drivers and

accessibility, GIS maps, risk

matrix

Yes Expected deforestation and

emissions were higher than actual

figures, showing that there was

positive leakage for deforestation

None N/A

Soto-Pinto

et al. (2010)

13 Measured biomass in

plots

Control-intervention:

compared plots both inside and

outside reserve of similar types

Yes All project systems had significantly

more carbon than maize and

pastures without trees

None N/A

De Jong (2004) 7 Participatory

workshops, document

review

Extrapolated historical: created

low, medium, high future

baseline depletion. Compared

regional LULC baselines and

project baselines

No Too early to tell if the land use

practices are sustainable, but the

project is a cost-effective way to

reduce carbon emissions

None N/A

en

vir

on

me

nt

al

sc

ie

nc

e&

po

lic

y1

4(2

01

1)

15

2–

16

71

57

Table 3 – Socioeconomic impacts of pre-REDD+ projects.

Reference # Years b/nproject start

and evaluationpublication

Data collectionmethods

Socialcounterfactual

methods

Observableoutcomescomparedagainst a

counterfactual

Reported socio-economic impacts

Number ofcitations(Source:GoogleScholar)

Project: Bananal Island Carbon Sequestration, Brazil

Boyd et al. (2007) 9 Document

review + interviews

(identified using

stakeholder analysis)

None No (1) US$60 per month gross revenue to group; (2)

majority of micro-projects had not succeeded;

(3) only 1 farmer adopted new agroforestry; (4) 6

of 12 park guards employed in community; (5)

fewer jobs in logging acquisition, within one

community 50% of families lost jobs from this

3

May et al. (2004)/

May et al. (2005)

6/7 Sustainable development

methodology, document

review, stakeholder

interviews

None No (1) 4 nurseries with 5 staff, jobs during building

of research center, assistance on beekeeping

and tourism (no numbers); (2) ‘‘attracted

investment to the region’’; (3) sweet factory

financed – 60 USD/mo for group; (4) people were

not included in planning stages

44 (2004)/

3(2005)

Project: Noel Kempff Mercado Climate Action Project, Bolivia

Boyd et al. (2007) 11 Document

review + interviews

(identified using

stakeholder analysis)

None No (1) 93 micro-credit projects (48% of affected

families), though most of loans not repaid,

causing lending to stop; (2) job creation: 12 park

guards (6 for local community members); (3) job

loss: 13 of 26 total families in one community

affected by job loss; (4) property rights: rights

secured for some, while also causing conflict

over access and rights

3

Calderon Angeleri

(2005)

9 Document review;

qualitative analysis using

the sustainable livelihood

framework; partial

financial analysis of

impacts

Adjusted historical: re-

constructs conditions before

the project, performs financial

analysis: 3 alternative baselines

(steady, worst case, and best

case)

Yes (1) One community had a net negative impact

because of disproportionate job loss; (2)

qualitative assessment of impact on different

capitals (human, financial, physical, natural,

social); (3) infrastructure built, services

provided, land tenure acquired; (4) in general,

project compensated losses of the

communities; (5) compares net present value of

project to the three financial baselines, and in

almost all potential scenarios there is a net gain

from the project

N/A

en

vir

on

me

nt

al

sc

ie

nc

e&

po

lic

y1

4(2

01

1)

15

2–

16

71

58

Robertson and

Wunder (2005)

9 Comparative case study

analysis, interviews

Adjusted historical: local

people would have gradually

left when timber jobs ran out

(speculated)

No (1) 12 permanent jobs as park guards, 80 temps

as surveyors, 6 tour guides; (2) Limited

economic success, issues with distribution; (3)

limits on use rights unclear; (4) loss of

healthcare, and (5) land tenure acquired

52

SGS (2008)/TNC (2009) 12/13 Proponent experience,

document review

Before–after: comparison

of job numbers

Yes They list project activities resulting in ‘‘overall

community benefit’’ (1) project helped better

organize communities; (2) helped establish land

tenure; (3) capacity, education, and health

services provided; (4) sustainable community

forestry concession gained; (5) ecotourism

infrastructure developed

N/A

Asquith et al. (2002) 6 Reviewed documents,

conducted interviews,

rapid rural appraisal

Before–after: retrospective

recall

Yes Direct losses: 229, 800 USD plus loss of use

rights to land; direct gains: 358, 380 USD – losses

and gains vary by community

40

May et al. (2004)/

May et al. (2005)

8/9 Sustainable development

methodology, document

review, stakeholder

interviews

None No Many reported. highlights: (1) uptake of

alternative land-use practices was limited; (2)

new conservation or resource-management

rules are not always clearly understood or

accepted; (3) no dialogue/equitable

partnerships in design phase, but these

partnerships have grown. Including

marginalized community members was a

challenge; (4) most successful activities were

process related; (5) poorest community

members benefited the least; (6) limited

alternative employment generation; (7) loss of

roads formerly well kept by logging companies

44 (2004)/3

(2005)

Rio Bravo Carbon Sequestration Pilot Project, Belize

USIJI (2000) 5 None stated: likely

document review,

proponent experience

Before–after: assumes no

forestry without project

Yes Employment generated (but not how many

jobs)

N/A

Brown et al. (2004) 9 Interviews, stakeholder

analysis, multi-criteria

analysis

None No (1) Illegal use of resources in the park and

frequent conflicts over resources; (2)

insufficient financial resources and lack of

tourist; (3) forest access rule changes have been

controversial for surrounding communities; (4)

social development component limited to

economically unsustainable pilot projects; (5)

created some jobs but with low salary and

considered undesirable); (6) need for patrols to

prevent forest/resource use (evidence of

conflict)

49

Scolel Te, Mexico

en

vir

on

me

nt

al

sc

ie

nc

e&

po

lic

y1

4(2

01

1)

15

2–

16

71

59

Table 3 (Continued )Reference # Years b/n

project startand evaluation

publication

Data collectionmethods

Socialcounterfactual

methods

Observableoutcomescomparedagainst a

counterfactual

Reported socio-economic impacts

Number ofcitations(Source:GoogleScholar)

Brown et al.

(2004)

7 Key informant

interviews, multi-

stakeholder analysis, two

village comparison

None No (1) Participants felt they were not compensated

enough for maintaining plantations; (2) when

additional funding ended in the late 1990s,

community workers left and a lot of the social

support collapsed; (3) in one village, critical

voices remain active; (4) limited project

participation due to unequal land holdings

within the community; (5) lack of

comprehension in project villages of carbon

sequestration and project structure; (6) in other

village, turnover for neighborhood agents and

authorities is too quick, no institutional

knowledge was built; (7) women were excluded

49

Corbera et al.

(2007)

10 Social survey, participant

observation, comparative

study between two

communities

None No In individually led project community: weak

governance failed to bring equitable outcomes,

reinforced power balances. In communally led

project community: legitimacy of organization

was good, but perhaps not sustainable. Lacked

sensitivity to women’s issues, also unable to

challenge inequities. Overall: project has made

trade-offs between legitimacy and equity

24

Tipper (2002) 5 None stated: likely

document review,

personal experience

None No (1) 400 individual participants from 30

communities, dropout rate under 5 per cent for

past 3 years; (2) income improvements: 300–

1800 per year, per family; (3) technical skills

developed – surveying, mapping, financial

planning, silviculture

30

Corbera (2005) 8 None reported; likely

personal experience,

comparative process

analysis between two

villages

None No Institutional processes narrative: (1) project

gained legitimacy across region; (2) sometimes

representatives fail to relay information to

communities, misunderstandings ensue

30

Tschakert et al.

(2008)

11 Expert stakeholder

consultation

None No Four criteria: (1) carbon, economic, ecological,

and social; overall, the project is helping

economic and social, but not holistic enough to

be sustainable

1

N’hambita Pilot Project, Mozambique

en

vir

on

me

nt

al

sc

ie

nc

e&

po

lic

y1

4(2

01

1)

15

2–

16

71

60

Jindal (2010/2004)/

Final Report (2008)

7 Household surveys, focus

groups

BACIa: control households,

before–after data; focus groups

on project perceptions and

other changes within/outside

communities

Yes Higher incomes for microenterprise/

agroforestry participating households. No

evidence of significant changes in months of

food bought; number of income sources;

number of durables

N/A (2010)/7

(2004)/N/A

(2008)

Niles (2008) 5 Document review, project

visitation, interviews, not

a complete audit/

evaluation

None No

(1) Project people work on a daily basis with

various community members; (2) farmers who

plant 1 ha of trees receive $100 and $25 goes

into a community fund; (3) project employs

dozens of people; (4) ‘‘From a social perspective,

the project is a clear success.’’

N/A

Kooistra and

Wolf (2006)

3 Stakeholder interviews,

document reviews

None No (1) 140 people employed by the project; (2)

increase in beekeeping (project promoted) and

other livelihood diversification projects, but not

clear if they are due to project presence or not;

(3) investment in community school, improved

wells; (4) sense of empowerment in

communities

N/A

Hegde (2010) 6 Quarterly household

surveys, village meetings

Control-intervention:

compared non-participating

households

Yes (1) Participants had more cash income and

more expenditures than non-participants; (2)

statistically insignificant reduction in use of

forest for subsistence; (3) conditional incentives

diluted by splitting revenues; (4) benefits tend to

flow to male-headed households; (5) 2/3 project

revenue spent on overhead and transaction

costs; (6) local improvements in ‘‘technical

knowledge, social capital, and community

visibility’’ (118)

N/A

a ‘BACI’ stands for before–after-control-intervention.

en

vir

on

me

nt

al

sc

ie

nc

e&

po

lic

y1

4(2

01

1)

15

2–

16

71

61

Table 4 – Comparison of project and evaluation design: Noel Kempff, Scolel Te, and N’hambita.

Noel Kempff, Boliva Scolel Te, Mexico N’hambita, Mozambique

Year established 1996 1997 2002

Strategy Park consolidation (avoided logging

and smallholder agriculture)

Agroforestry and forest

conservation by smallholders

Agroforestry and fire management

by smallholders

Framework AIJ/TNC AIJ/ Plan Vivo Plan Vivo

Early research attention Winrock International Edinburgh Centre for Carbon

Management, Tyndall Center

University of Edinburgh

Biophysical indicators Avoided deforestation and

degradation in project area;

total net reduction in emissions

Reforestation and avoided

deforestation in project area;

total net reduction in emissions

Reforestation and avoided

deforestation in project area

Biophysical

counterfactual

Modeled with input from

control area

Modeled, historical, and

comparison plots

Historical, but no explicit

comparison

Welfare indicators Employment, full income,

infrastructure

Perceptions and participation Cash income, assets

employment, forest use

Welfare counterfactual Recall of pre-project conditions;

historical extrapolation

Not explicit Comparison households

e n v i r o n m e n t a l s c i e n c e & p o l i c y 1 4 ( 2 0 1 1 ) 1 5 2 – 1 6 7162

in directionality, saying for example that jobs would have

‘declined’ (rather than modeling employment trends based on

historical data) (e.g., USIJI, 2000; Robertson and Wunder, 2005).

While this method acknowledges the concept of a counterfac-

tual, assessing its validity is difficult. Others relied on before and

after data (derived from either document review or respondent

recall) (e.g., TNC, 2009; Asquith et al., 2002; USIJI, 2000). In these

cases, historical conditions are taken as the ‘without project’

scenario, and all changes are attributed to project implementa-

tion. Hegde (2010) used control-intervention design, comparing

households at control and intervention sites after project

implementation. Finally, Jindal (2010) used a before–after-

control impact (BACI) design, with measurements before and

after the project in both control and intervention villages.

Outcomes were frequently reported in terms of changes in

employment. For example, Kooistra and Wolf (2006) reported

140 jobs gained, and Robertson and Wunder (2005) reported 12

permanent park guards, 80 temporary surveyor jobs, and 6

tour guides resulting from the project. Others described

income directly generated by project activities (e.g., May et

al., 2004). These outcomes were typically determined from a

few key informant interviews or anecdotal information.

Project narratives were also a common way of reporting

socioeconomic impacts (Kooistra and Wolf, 2006; Brown et

al., 2004). Narratives construct a story of how the project has

impacted the community or households, providing a qualita-

tive picture of process, participation, and outcomes of the

project, but generally not quantifying impacts nor applying

counterfactual thinking in a critical way.

Most documents reported positive employment and

income gains. The focus on employment may be both

because the data are relatively easy to collect and because

from the perspective of local stakeholders (e.g., local

government), employment is an important and visible

metric of impact. The negative impacts reported had more

to do with process: exclusion of women, distributional

issues, tensions between the project and the community,

and/or lack of early participation in project design (e.g., May

et al., 2004; Corbera, 2005). One document we reviewed

commented on the poor quality of jobs created by the project

and noted distributional inequity (Brown et al., 2004.). Other

well-being measures such as health or literacy were absent

from the reviewed studies.

4.3. Case studies

The three projects with the most detailed published evalua-

tions were Noel Kempff, Scolel Te, and N’hambita (see Table

4). This can partly be explained by their early affiliation with

research organizations, or what we term ‘early research

patrons’, in recognition of the fact that later evaluations also

involved researchers from other institutions. The large

number of published evaluation of these projects is some-

what misleading in that they are not entirely independent of

one another (with many conducted by overlapping sets of

researchers) or of the project implementers. For example, all

projects seeking credit for reduced carbon emissions must

estimate land cover counterfactuals as their baseline or

reference level for crediting. These are reported in the

uniform reporting documents (USIJI, 2000), which are also

available for projects like Bananal and Rio Bravo. ‘Indepen-

dent’ evaluations of these projects may adopt these same

counterfactuals developed by project proponents. This is

particularly true of comparative studies that assess multiple

projects (Boyd et al., 2007; Brown et al., 2004; May et al., 2004;

Pereira, 2010).

Noel Kempff is perhaps the best known of the projects that

we reviewed, having been established and publicized by The

Nature Conservancy (TNC) as ‘proof of concept’ of forest

carbon offsets, but recently criticized by Greenpeace as a

‘carbon scam.’ Winrock International has been engaged in this

project from the beginning, developing methods, estimates,

and reports largely focused on carbon. Both TNC and Winrock

have widely publicized their work with this project, and its

high profile has attracted independent researchers, REDD+

critics (e.g. Greenpeace), and press attention (e.g. The

Guardian, Christian Science Monitor, The Economist).

In contrast, N’hambita is a much more recently established

project. The more rigorous empirical impact evaluation

methods applied to both co-benefits and carbon in this project

reflect the growing attention to ‘evidence-based’ policy-

making in conservation and to the potential ‘co-benefits’ or

impacts of REDD+ on livelihoods and biodiversity (e.g., as in

CCBA certification). The project’s early engagement with a

university as research partner probably also contributed to

greater investment in and transparency of baseline data

collection.

e n v i r o n m e n t a l s c i e n c e & p o l i c y 1 4 ( 2 0 1 1 ) 1 5 2 – 1 6 7 163

Scolel Te was built on an earlier initiative by the Mexican

government to engage in AIJ, Fondo Bioclimatico. There was

already significant research on this earlier initiative and it

continued with Scolel Te. Like N’hambita, this project was

designed under the Plan Vivo system of individual contracts

with farmers. Compared to Noel Kempff, the household scale

of participation and activities in Scolel Te and N’hambita

makes it more feasible to evaluate those projects using

empirical observations on controls because there are so many

candidate control households.

4.4. Visibility search

In Google Scholar, the documents we reviewed were generally

well cited or had no citations at all. Of the 24 documents

reviewed, 12 had citation data available, ranging from 1 to 52

citations. Five of the documents have received 30 or more

citations as of May 2010.

In the popular media we reviewed, most of the discussion

about social welfare outcomes was framed in the context of

the mechanisms projects used (or intended to use) to transfer

benefits to forest users. Several articles discussed direct

compensation (i.e. cash payments) for forest users and

communities and/or investment in community level infra-

structure including schools, hospitals and other public

infrastructure (The Economist, 2009; Girling, 2009; Vidal,

2009). There was also mention of compensation for lost

employment opportunities, e.g. in Noel Kempff, where

community members were compensated for exiting the

logging business (Spotts, 2005). We found discussion about

how lack of property rights to clarify land and carbon tenure,

and elite capture compromises the social welfare gains in pre-

REDD+ (Vidal, 2009; Hari, 2010). Several articles discussed the

‘additional advantage’ of positive social welfare impacts in a

normative sense (World Bank, 2010; Autoblog Green, 2009).

The only explicit reference to counterfactual thinking was in

the context of the N’hambita project, where it was noted that

while the project cannot eradicate poverty completely, it is

raising the standard of living above that of other comparable

villages in Mozambique (Girling, 2009).

In contrast, the popular media does discuss counter-

factuals in terms of ‘baselines’ and ‘leakages’ when consider-

ing biophysical impacts. This is the area where we observe the

most significant substantive critiques of pre-REDD+ projects in

the popular press. There are critiques of the baseline methods

used in the Juma project (The Economist, 2009), and estimates

of emissions reductions and leakage in Noel Kempff (Spotts,

2005; Lang, 2009). While several mentioned the conservation

benefits of early forest carbon projects in the context of

preserving biodiversity rich forests, we did not find explicit

discussion of the potential for either positive or negative

biodiversity outcomes.

5. Discussion

Overall, we found a very limited body of evidence from which

to draw lessons for designing evaluations of future REDD+

projects. Relative to the current state-of-the-art in empirical

impact evaluation, the design, data collection, and methods of

analysis of pre-REDD+ projects lack rigor. Evaluations

appeared in many cases to be afterthoughts, undertaken

without baseline data or experimental design built into project

design. In light of the explicit goal of ‘learning while doing’

inherent in pilot avoided deforestation and REDD+ projects,

the question of why there are so little data warrants further

consideration.

The historical context provides a partial explanation. The

AIJ projects were launched in the mid-1990s amid optimism

that avoided deforestation would become a mechanism under

the Kyoto Protocol. When this did not happen, interest faded.

This meant both a loss of funding for the projects themselves

and a loss of support for research on the projects. Out of the

eight avoided deforestation projects listed on the AIJ website,

we only found impact documents for three of them (Noel

Kempff, Rio Bravo, and Scolel Te). This probably reflects not

only a lack of evaluation, but also a lack of project

implementation. In other words, many projects appearing

on lists of pre-REDD+ projects either never got started or were

terminated prior to substantial implementation.

It can also be argued that the primary objective of these

early avoided deforestation initiatives was to establish some

kind of project on the ground. Because these were the very first

forest carbon projects, there were few standardized reporting

requirements and no quality standards. Without clear

standards, establishing monitoring and evaluation systems

was difficult. Rigorous impact evaluation was beyond their

scope, especially after avoided deforestation was excluded

from the CDM. More recently, standards for credits traded in

the voluntary carbon market (e.g., Voluntary Carbon Stan-

dards) and specific standards for forest carbon projects (e.g.,

the Climate, Community, and Biodiversity Alliance Standards)

have been developed. Experience with pre-REDD+ projects has

informed the carbon accounting methodologies for REDD+

projects being developed under these standards (Brown, 2010;

Kant, 2010) and controversies over those projects has led to

interest in standards that assess co-benefits and respect for

local stakeholders rights (Ecosecurities, 2010). However, these

new standards do not explicitly incorporate lessons from

socioeconomic evaluations of pre-REDD+ projects, perhaps in

part because those evaluations are so limited and descriptions

are so opaque.

Perhaps one of the largest barriers to effective pre-REDD+

evaluation is the long project time frames. Many projects have

time horizons ranging from 20 to 30 years or longer and there

is uncertainty about when it is reasonable to expect projects

will have quantifiable impacts. We did see a fair amount of

attention paid to establishing baselines in many cases, but

even with projects that are now 10–15 years old, there is no

definitive impact statement regarding project outcomes.

Waiting 20 to 30 years to evaluate pre-REDD+ or REDD+

projects is not a viable option. Project implementation

requires process evaluation and the use of high quality

baseline data for assessing both biophysical and social

outcomes at various stages of project (Jagger et al., 2009).

Also, validation of carbon credits, while it is an important

process, is not a perfect substitute for independent carbon

research; carbon sequestration measurement science is not

well-established, and independent research is needed to

understand whether or not the validation process is accurately

e n v i r o n m e n t a l s c i e n c e & p o l i c y 1 4 ( 2 0 1 1 ) 1 5 2 – 1 6 7164

capturing carbon sequestration values over time. The absence

of rigorous measurement of forest quality beyond carbon

suggests that we have little information about changes in the

quality of the forests and forest sustainability.

It is striking that the construction of the counterfactual is

common when considering biophysical outcomes but gener-

ally neglected when considering social outcomes. This may

simply be because the goal of these projects is emission

reductions and certification standards require construction of

a counterfactual land use/carbon scenario, so any evaluations

of social impacts are for curiosity and often conducted ex post.

However, lack of attention to counterfactuals with respect to

social outcomes also points to a deeper theoretical gap

between how social and physical scientists consider project

impacts. This divide manifests itself in terms of how one

defines project impacts and success; it is unclear whether

evaluation should seek to quantify measurable socio-econom-

ic impacts or focus on whether the project impacts less

tangible but arguably important welfare dimensions (such as

rights, participation, capacity, autonomy, or culture). Regard-

less of the outcome variables defined, the credibility of social

impact assessments that do not use quasi-experimental

methods, modeling exercises or any ‘hard’ measures of

outcomes are questionable to much of the scientific commu-

nity. We also point out that there is no discussion of leakages

as they pertain to welfare outcomes in pre-REDD+ evaluations.

Socio-economic indicators are solely focused on project sites,

neglecting general equilibrium effects that may result from

shifting economic activity.

We observe a clear separation between the research

processes measuring carbon and biophysical outcomes and

those measuring socio-economic outcomes. While the de-

sign, data collection and analysis of these areas require

different skill sets, failing to provide analysis of both

outcomes precludes any meaningful analysis of trade-offs.

Given that understanding tradeoffs (including the magnitude

of trade-offs) and the potential for win-win outcomes are

among the most urgent areas for learning needed for REDD+

(Jagger et al., 2009), this is a significant gap that needs to be

addressed.

Pre-REDD+ projects were often established in areas with

high biodiversity. Biodiversity preservation is considered a

co-benefit of pre-REDD+ and REDD+ projects. While there are

likely biodiversity gains when additional land is protected,

biodiversity outcomes are by no means optimized by

traditional placement of protected areas (Salafsky et al.,

2002). Thus, measuring biodiversity outcomes presents a

clear area for improvement in the evaluation of pre-REDD+

and REDD+ projects. There are ways to measure biodiversity

that can be incorporated into study design including many

that engage local communities in monitoring and data

collection (Elkstrom, 2008). This part of evaluation was

clearly neglected, but the global REDD+ community would

benefit from a stronger alliance with traditional conservation

organizations so that ongoing biodiversity research could be

linked to carbon outcomes (Pimm, 2010).

Our popular media and scholarly citation review indicates

that the wider community is clearly showing interest in pre-

REDD+ projects, and it is important that they have access to

rigorous empirical research. Our overall impression of the

popular press on pre-REDD projects is that it lacks specificity

in its discussion of impacts, but makes generalized comments

on the basis of the limited and largely speculative data that are

available on welfare and biophysical outcomes. There is no

attention given to using counterfactual thinking to under-

stand project outcomes. While these issues are partially

attributable to the nature and style of popular journalism,

there is a danger that the popular media will fail to fulfill its

role in providing critical analysis of important public policy

issues. Noel Kempff provides the most salient example of how

the lack of well-designed evaluation and baseline data has

allowed both supporters and detractors of REDD+ to use it as a

flagship case study of either success or failure.

6. Conclusion

We have drawn on our collective knowledge of the universe of

available information on pre-REDD+ projects to conduct our

review of methods for understanding and assessing biophysi-

cal and welfare outcomes. Our findings lead us to a conclusion

consistent with the absence of rigorous evaluations in the

environmental field, which Ferraro and Pattanayak (2006)

argue is seriously hindering advancement in our knowledge of

solving complex environment and development problems.

While the data itself was not able to provide a wealth of hard

analyses on the successes or failures of pre-REDD+ projects,

the process of assembling it and the qualitative assessments

we can make of the available data are compelling and valuable

for directing future research.

We have three main findings. First, linking co-benefits to

carbon benefits remains a significantmethodologicalchallenge,

and we acknowledge that prioritizing both in evaluation design

is difficult. Measures that use common metrics and scales for

analyzing both biophysical and welfare outcomes are not easy

to find; using mixed methods is a challenge despite gains in

interdisciplinary research. However, understanding the inter-

play between co-benefits and carbon benefits is essential to

learning how to implement sub-national REDD+ that will

deliver on the promise of incentive based green house gas

emissions reductions with favorable socioeconomic and biodi-

versity co-benefits. Indeed, REDD+ success in low income

countries may hinge on its ability to at the very least do no harm

tothe welfare of forestusers.Second,whilewe acknowledgethe

hesitanceof theconservationcommunity toundertakerigorous

evaluation work, when we consider current research and policy

analysis tools in education, public health and social policy, we

see that rigorous impact evaluation is the standard mode of

operation. Arguments about the difficulty of finding good

control groups, sample size issues, etc. can be overcome with

effective and well planned research design, sampling strategies

and adequate investment in evaluation. Finally, we have

highlighted how the scientific, donor and journalistic commu-

nities have used existing evaluations of pre-REDD+ projects. As

we move forward with investment in REDD+ projects, taking

lessons for design and implementation from the very limited

body of research is disconcerting. We should use REDD+

projects as a platform for effectively integrating more rigorous

evaluation standards into the core of the business of environ-

ment and development.

e n v i r o n m e n t a l s c i e n c e & p o l i c y 1 4 ( 2 0 1 1 ) 1 5 2 – 1 6 7 165

We argue that some, but not all, of the early REDD+ projects

need to be rigorously evaluated to inform early learning and

project design for the future. In the spirit of ‘learning while

doing’ we call for the use of research design and methods to

establish baseline data and measure both biophysical AND

social welfare outcomes in REDD+ related project assess-

ments. We also recommend using mixed methods (i.e.

ethnography, participatory rapid appraisal, socioeconomic

surveys etc.) to understand the process of implementation and

causal relationships between interventions and observed

outcomes (Jagger et al., 2010). We foresee a strong interplay

between the scientific and practitioner communities in the

design, implementation and evaluation of REDD+ projects.

Accordingly, methods should be clear and accessible to both

communities. More independent baseline analyses are needed

in order to improve the validity of our overall understandings

of additionality in REDD+ projects. There are some groups

working on this (see Brown et al., 2000) but more can be done.

Acknowledgments

We acknowledge the financial support of the Carolina

Population Center, the Center for International Forestry

Research (CIFOR) and the Research Triangle Institute (RTI).

We are also grateful to Dr. Brian Murray and the students,

guest speakers, and attendees of the Collaborative Research

Seminar on REDD+ held jointly by Duke University, North

Carolina State University, and the University of North Carolina

at Chapel Hill during the spring semester of 2010. Their

insightful comments on this research, as well as the

comments of 4 anonymous reviewers have strengthened this

study. Megha Karmalkar of the University of North Carolina

provided valuable research assistance. Any errors or omis-

sions are our own.

r e f e r e n c e s

Andam, K.S., Ferraro, P.J., Pfaff, A., Sanchez-Azofeifa, G.A.,Rabalino, J.A., 2008. Measuring the effectiveness ofprotected area networks in reducing deforestation. PNAS105 (42), 16089–16094.

Andam, K.S., Ferraro, P.J., Sims, K.R.E., Healy, A., Holland, M.B.,2010. Protected areas reduced poverty in Costa Rica andThailand. PNAS 107 (22), 9996–10001.

Angelsen, A., Brockhaus, M., Kanninen, M., Sills, E., Sunderlin,W.D., Wertz-Kanounnikoff, S., Abdel Nour, H.O. (Eds.), 2009.Realising REDD+: National Strategy and Policy Options.Center for International Forestry Research (CIFOR), Bogor,Indonesia.

Angrist, J., Pischke, J.-S., 2010. The credibility revolution inempirical economics: How better research design is takingthe con out of econometrics. In: NBER Working Paper 15794,National Bureau of Economics Research, Cambridge, MA.

Arriagada, R.A., 2008. Private provision of public goods: applyingmatching methods to evaluate payments for ecosystemservices in Costa Rica. PhD dissertation. North CarolinaState University, Raleigh, North Carolina.

Asquith, N.M., Vargas Rios, M.T., Smith, J., 2002. Can forest-protection carbon projects improve rural livelihoods?Analysis of the Noel Kempff Mercado Climate Action

Project, Bolivia. Mitigation and Adaptation Strategies forGlobal Change 7, 323–337.

Autoblog Green, 2009. Hyundai Genesis sales pay for forestpreservation and restoration projects. Autoblog Green(May).

Bailis, R., 2006. Climate change mitigation and sustainabledevelopment through carbon sequestration: experiences inLatin America. Energy for Sustainable Development 10 (4),74–87.

Bond, I., Grieg-Gran, M., Wertz-Kanounnikoff, S., Hazlewood, P.,Wunder, S., Angelsen, A., 2009. Incentives to sustain forestecosystem services: a review and lessons for REDD. NaturalResource Issues No. 16. International Institute forEnvironment and Development, London, UK, with CIFOR,Bogor, Indonesia, and World Resources Institute,Washington D.C., USA.

Boyd, E., May, P., Chang, M., Veiga, F.C., 2007. Exploringsocioeconomic impacts of forest based mitigation projects:lessons from Brazil and Bolivia. Environmental Science andPolicy 10, 419–433.

Brown, K., Adger, W.N., Boyd, E., Corbrera-Elizalde, E., Shackley,S., 2004. How do CDM projects contribute to sustainabledevelopment? Tyndall Centre for Climate Change ResearchTechnical Report 16.

Brown, S., Burnham, M., Delaney, M., Powell, M., Vaca, R.,Moreno, A., 2000. Issues and challenges for forest-basedcarbon-offset projects: a case study of the Noel KempffClimate Action Project in Bolivia. Mitigation and AdaptationStrategies for Global Change 5, 99–121.

Brown, S., 2010. Personal Communication.Calderon Angeleri, N., 2005. Livelihood impact assessment: Noel

Kempff Mercado Climate Action Project, Bolivia, November.Annex 6 of Project Design Document.

Corbera, E., 2005. Bringing development into carbon forestrymarkets: challenges and outcomes of small-scale carbonforestry activities in Mexico. In: Murdiyarso, D., Herawati,H. (Eds.), Carbon Forestry: Who Will Benefit? Proceedings ofthe Workshop on Carbon Sequestration and SustainableLivelihoods. Center for International Forestry Research,Bogor, Indonesia, pp. 42–56.

Corbera, E., Brown, K., Adger, W.N., 2007. The equity andlegitimacy of markets for ecosystem services. Developmentand Change 38 (4), 587–613.

De Jong, B.H.J., 2004. Carbon sequestration potential throughforestry activities in tropical Mexico. In: Zarin, D.J.,Alavalapati, J.R.R., Putz, F.E., Schmink, M. (Eds.), WorkingForests in the Neotropics: Conservation ThroughSustainable Management? Columbia University Press, NewYork, pp. 238–257.

De Jong, B.H.J., Esquivel Bazan, E., Quechulpa Montalvo, S., 2007.Application of the ‘‘Climafor’’ baseline to determineleakage: the case of Scolel Te. Mitigation and AdaptationStrategies for Global Change 12, 1153–1168.

DiNicola, A., Jones, D.J., Gray, G., 1997. Opportunities for forestryinvestment in Asia and the Pacific through carbon offsetinitiatives. In: Asia-pacific Forestry Sector Outlook StudyWorking Paper No. 29, Food and Agriculture Organization ofthe United Nations, Rome.

Dixon, R.K., Andrasko, K.J., Sussman, F.G., Lavinson, M.A.,Trexler, M.C., Vinson, T.S., 1993. Forest sector carbon offsetprojects: near-term opportunities to mitigate greenhousegas emissions. Water, Air, and Soil Pollution 70 (1–4), 561–577.

Ecosecurities. 2010. The Forest Carbon Offsetting Report 2010.Elkstrom, J., 2008. Low-cost Biodiversity Impact Assessment for

Multi-benefit PES Projects: Guidance and Challenges.(accessed 29 May 2010), available at:http://www.forestcarbonportal.com/sites/default/files/doc_192.pdf.

e n v i r o n m e n t a l s c i e n c e & p o l i c y 1 4 ( 2 0 1 1 ) 1 5 2 – 1 6 7166

Food and Agriculture Organization of the United Nations (FAO),2004. A review of carbon sequestration projects. FAO, Rome.

Ferraro, P.J., Pattanayak, S.K., 2006. Money for nothing? A callfor empirical evaluation of biodiversity conservationinvestments. PLoS Biology 4 (4), 482–488.

Ferraro, P.J., 2009. Counterfactual thinking and impactevaluation in environmental policy. New Directions forEvaluation 122, 75–84.

Gaveau, D.L.A., Wandono, H., Setiabudi, F., 2007. Three decadesof deforestation in southwest Sumatra: have protectedareas halted forest loss and logging, and promoted re-growth? Biological Conservation 134, 495–504.

Girling, R.C., 2009. The green shoots of hope; a community indevastated rural Mozambique is being helped to sow theseeds of its own salvation. Sunday Times (April).

Hari, J., 2010. The wrong kind of green. The Nation (March).Harvey, C.A., Zerbock, O., Papageorgiou, S., Parra, A., 2010. What

is needed to make REDD+ work on the ground? Lessonslearned from pilot forest carbon initiatives. In: ConservationInternational, Arlington, Virginia.

Hegde, R., 2010. Payments for ecosystem services and farmhousehold behaviour: The case of carbon in Mozamibque’sagroforests. PhD dissertation. University of BritishColumbia, Vancouver.

Jagger, P., 2010. Forest sector reform, livelihoods andsustainability in western Uganda. In: German, L., Karsenty,A., Tiani, A.M. (Eds.), Governing Africa’s Forests in aGlobalized World. Earthscan Publications Inc. and Centerfor International Forestry Research, London and Bogor,Indonesia.

Jagger, P., Sills, E.O., Lawlor, K., Sunderlin, W., 2010. A Guide forLearning about REDD+ Projects and Livelihood Impacts.Center for International Forestry Research, Bogor,Indonesia.

Jagger, P., Atjadja, S., Pattanayak, S.K., Sills, E., Sunderlin, W.D.,2009. Learning while doing: evaluating impacts of REDD+projects. In: Angelsen, A., with Brockhaus, M., Kanninen,M., Sills, E., Sunderlin, W.D., Wertz-Kanounnikoff, S.(Eds.), Realising REDD+: National Strategy and PolicyOptions. Center for International Forestry Research, Bogor,Indonesia.

Jindal, R., 2004. Measuring the socio-economic impact of carbonsequestration on local communities: An assessment studywith specific reference to the N’hambita Pilot Project inMozambique. MSc thesis. University of Edinburgh,Edinburgh, Scotland.

Jindal, R., Swallow, B., Kerr, J., 2008. Forestry-based carbonsequestration projects in Africa: potential benefits andchallenges. Natural Resources Forum 32, 116–130.

Jindal, R., 2010. Livelihood impacts of payments for forestrycarbon services: field evidence from Mozambique. In:Tacconi, L., Mahanty, S., Suich, H. (Eds.), Livelihoods inREDD: Payments for Environmental Services. ForestConservation and Climate Change, Edward Elgar,Cheltenham.

Jumbe, C.B.L., Angelsen, A., 2006. Do the poor benefit fromdevolution policies? Evidence from Malawi’s Forest Co-Management Program. Land Economics 82 (4), 562–581.

Kant, Z., 2010. Personal Communication.Karsenty, A., Blanco, C., Dufour, T., 2003. Forest and Climate

Change: Instruments Related to the United NationsFramework Convention on Climate Change and theirPotential for Sustainable Forest Management in Africa. FAO,Rome.

Kooistra, T., Wolf, J., 2006. External Evaluation of the MiomboCommunity Land Use and Carbon Management N’hambitaPilot Project. Institute of Atmospheric & EnvironmentalScience, School of GeoSciences, The University ofEdinburgh, Scotland.

Lang, C., 2009. Carbon Scam: The Noel Kempff Project in Bolivia.(accessed 10 September 2010), available at:http://www.redd-monitor.org/2009/10/22/carbon-scam-the-noel-kempff-project-in-bolivia/.

Lawlor, K., Weinthal, E., Olander, L.P., 2010a. Institutions andpolicies to protect rural livelihoods in REDD+ regimes.Global Environmental Politics 10 (4).

Lawlor, K., Jenkins, A., Olander, L.P., Murray, B.C., 2010b.Expanding the scope of international terrestrial carbonoptions: implications of REDD+ and beyond. In: NicholasInstitute Report, Feb. Nicholas Institute, Duke University,Durham, North Carolina.

May, P., Boyd, E., Viega, F., Chang, M., 2004. Local SustainableDevelopment Effects of Forest Carbon Projects in Brazil andBolivia: A View from the Field. International Institute forEnvironment and Development, London.

May, P., Boyd, E., Chang, M., Viega, F., 2005. Incorporatingsustainable development into carbon forest projects inBrazil and Bolivia. Estud. Soc. Agric. vol. 1, No. Se Rio deJaneiro.

McNally, R., Sage, N., Holland, T., 2009. Understanding REDD:Implications for Lao PDR, Nepal and Vietnam.Commissioned by SNV.

Niles, J.O., 2008. Independent Project Report on N’hambitaproject in Mozambique. (Accessed 13 December 2009),available at http://www.envirotrade.co.uk/documents/Niles%20Nhambita%20Nov%2026%202008.pdf.

Olander, L.P., Boyd, W., Lawlor, K., Niles, J.O., Myers Madiera, E.,2009. Responding to Concerns and Questions. NicholasInstitute Policy Brief. Nicholas Institute, Duke University,Durham, North Carolina.

Pattanayak, S.K., Wunder, S., Ferraro, P.J., 2010. Show me themoney: do payments supply ecosystem services indeveloping countries? Review of Environmental Economicsand Policy.

Pereira, S.N.C., 2010. Payment for environmental services in theAmazon forest: how can conservation and development bereconciled? The Journal of Environment & Development 19,171–190.

Pfaff, A., Sills, E.O., Amacher, G.S., Coren, M.J., Lawlor, K., Streck,C., 2010. Policy Impacts on Deforestation: Lessons Learnedfrom Past Experiences to Inform New Initiatives. TheNicholas Institute, Duke University, Durham, NC.

Pimm, S., 2010. Personal Communication.Richards, M., Panfil, S., 2010. Manual for Social Impact

Assessment of Land-Based Carbon Projects, Version 1.0.(accessed 31 August 2010), available at:In: http://www.forest-trends.org/documents/files/doc_2436.pdf.

Robalino, J., Pfaff, A., Sanchez-Azofeifa, G.A., Alpızar, F., Leon, C.,Rodrıguez, C.M., 2008. Deforestation Impacts ofEnvironmental Services Payments: Costa Rica’s PSA Program2000–2005. Environment for Development Discussion Paper08-24. Resources for the Future, Washington, DC.

Robertson, N., Wunder, S., 2005. Fresh Tracks in the Forest:Assessing Incipient Payments for Environmental ServicesInitiatives in Bolivia. Center for International ForestryResearch, Bogor, Indonesia.

Salafsky, N., Margoluis, R., Redford, K.H., Robinson, J.G., 2002.Improving the practice of conservation: a conceptualframework and research agenda for conservation science.Conservation Biology 16, 1469–1479.

SGS, 2008. The Noel Kempff climate action project: validationand verification report. Summary (November).

Skutsch, M.M., McCall, M.K., 2010. Reassessing REDD:governance, markets and the hype cycle: an editorialcomment. Climatic Change: An Interdisciplinary,International Journal Devoted to the Description, Causesand Implications of Climatic Change 100 (3–4),395–402.

e n v i r o n m e n t a l s c i e n c e & p o l i c y 1 4 ( 2 0 1 1 ) 1 5 2 – 1 6 7 167

Sills, E., Madeira, E., Sunderlin, W., Wertz-Kanounnikoff, S.S.,2009. The evolving landscape of REDD+ projects. In:Angelsen, A., with Brockhaus, M., Kanninen, M., Sills, E.,Sunderlin, W.D., Wertz-Kanounnikoff, S. (Eds.), RealisingREDD+: National Strategy and Policy Options. Center forInternational Forestry Research, Bogor, Indonesia.

Somanthan, E., Prabhakar, R., Mehta, B.S., 2009.Decentralization for cost-effective conservation. PNAS 106(11), 4143–4147.

Soto-Pinto, L., Anzueto, M., Mendoza, J., Jimenez Ferrer, G., deJong, B., 2010. Carbon sequestration through agroforestry inindigenous communities of Chiapas, Mexico. AgroforestrySystem 78, 39–51.

Spotts, P.N., 2005. Bid to team rich and poor in sustainingforests. Christian Science Monitor (December).

Sunderlin, W.D., Atmadja, S., 2009. Is REDD+ an idea whosetime has come, or gone? In: Angelsen, A., with Brockhaus,M., Kanninen, M., Sills, E., Sunderlin, W.D., Wertz-Kanounnikoff, S. (Eds.), Realising REDD+: NationalStrategy and Policy Options. Center for InternationalForestry Research, Bogor, Indonesia.

Sutherland, W.J., Pullin, A.S., Dolman, P.M., Knight, T.M., 2004.The need for evidence-based conservation. Trends inEcology and Evolution 19 (6), 305–308.

Tallis, H., Kareiva, P., Marvier, M., Chang, A., 2008. An ecosystemservices framework to support both practical conservationand economic development. PNAS 105 (28), 9457–9464.

Terrestrial Carbon Group, 2009. Tools for Setting ReferenceEmission Levels. Terrestrial Carbon Group Policy BriefNumber 2. Terrestrial Carbon Group, Washington, DC.

Tipper, R., 2002. Helping indigenous farmers to participate inthe international market for carbon services: the case ofScolel Te. In: Pagiola, S., Bishop, J., Landell-Mills, N. (Eds.),Selling Forest Environmental Services: Market-basedMechanisms for Conservation and Development. EarthscanPublishing Inc., London.

The Economist, 2009. Last gasp for the forest; Paying to savetrees. The Economist (September.).

The Nature Conservancy, 2009. Noel Kempff Mercado ClimateAction Project: A Case Study in Reducing Emissions fromDeforestation and Degradation. The Nature Conservancy,Arlington, Virginia.

The University of Edinburgh School of GeoSciences, 2008.Miombo Community Land Use & Carbon Management:N’hambita Pilot Project Final Project Report, December.University of Edinburgh, Edinburgh, Scotland.

Tschakert, P., Huber-Sannwald, E., Ojima, D.S., Raupach, M.R.,Schienke, E., 2008. Holistic, adaptive management of theterrestrial carbon cycle at local and regional scales. GlobalEnvironmental Change – Human and Policy Dimensions 18(1), 128–141.

UNFCCC, 2008.In: Report of the Conference of the Parties on itsThirteenth Session. Bali, Indonesia, 3–15 December, 2007(Decision 1/CP.13).

UNFCCC, 2009.In: Copenhagen Accord. Draft Decision of theConference of the Parties at its Fifteenth Session.Copenhagen, Denmark, 7–18 December, 2009 (FCCC/CP/2009/L.7).

United States Initiative for Joint Implementation (USIJI), 2000.Uniform reporting document for activities implemented

jointly under the pilot phase. Rio Bravo CarbonSequestration Pilot Project.

Vidal, J., 2009. Redd in Africa: how we can earn money from airby harvesting carbon. The Guardian (October).

Watson, R.T., Noble, I.R., Bolin, B., Ravindranath, N.H., Verardo,D.J., Dokken, D.J. (Eds.), 2000. Land Use, Land-Use Changeand Forestry. Cambridge University Press, Cambridge.

World Bank Carbon Finance Unit, 2010. Carbon Finance at theWorld Bank: BioCarbon Fund. (accessed 8 February, 2010),available at:In: http://wbcarbonfinance.org/Router.cfm?Page=BioCF&FID=9708&ItemID=9708&ft=About.

World Bank, 2010. Africa’s first large-scale forestry projectunder the Kyoto protocol. The World Bank (March).

Susan Caplow is a PhD student in the Curriculum for the Environ-ment and Ecology at the University of North Carolina at ChapelHill. She is also a predoctoral trainee at the Carolina PopulationCenter. She has an MSc in Environmental Sciences and Policy fromCentral European University in Budapest and a BA in Public PolicyAnalysis/Biology from Pomona College in Claremont, CA. Herresearch focuses on the impact of community participation andstakeholder engagement on conservation project outcomes.

Pamela Jagger is Assistant Professor of Environmental Policy anda Faculty Fellow with the Carolina Population Center at theUniversity of North Carolina at Chapel Hill. She has more than10 years of policy research experience examining the livelihoodimpacts of natural resource management policies in sub-SaharanAfrica. She has a PhD in Public Policy from Indiana University andan MSc in Forest Economics from the University of Alberta. Shehas worked with the International Food Policy Research Institute,Resources for the Future and the World Bank. She is a researchassociate with the Center for International Forestry Research inBogor, Indonesia.

Kathleen Lawlor is a PhD student at the Department of PublicPolicy and a Pre-doctoral Trainee at the Carolina PopulationCenter at the University of North Carolina at Chapel Hill. Shehas worked in conservation and development for 10 years. Shehas worked for Duke University’s Nicholas Institute on forestand climate policy and research; for the World Bank Group,investigating communities’ claims of social and environmentalharm; for the US Forest Service on African forestry issues;and with farmers in Cameroon as a Peace Corps Volunteer.She holds a Master of Environmental Management from DukeUniversity.

Erin Sills is an Associate Professor and coordinator of internation-al programs in the Department of Forestry and EnvironmentalResources at North Carolina State University. She received a PhDin natural resource economics from Duke University and a BA inpublic policy from Princeton University. Her research focuses onthe economics of multiple-use forest management, includingquantifying the value of non-timber benefits from forests, model-ing the behavior of forest owners and managers, and rigorousimpact evaluation of forest policies and programs. She is a Re-search Associate of the Center for International Forestry Research(CIFOR) and the Instituto do Homen e Meio Ambiente da Amazonia(Imazon).