www.3ieimpact.org

How can synthesised evidence

help reduce poverty?

Hugh Waddington,

Senior Evaluation Specialist

International Initiative for Impact Evaluation

www.3ieimpact.org

What do we mean by evidence synthesis?

• Systematic collection and presentation

of evidence on a particular topic or

programme

• Using rigorous methods

– Gap maps

– Rapid evidence assessments

– Systematic reviews

– Statistical meta-analysis

www.3ieimpact.org

Exercise 1

Do you AGREE or DISAGREE with the following statements about

evidence-informed policy making?

• Evidence needs to be drawn from all studies to be policy actionable.

• Evidence needs to be drawn from a range of academic disciplines to

be relevant.

• Existing evidence needs to be appraised and synthesised by

experts before it is used.

• Systematic reviews are relevant for policy making but not

programme design and implementation.

• I have used systematic reviews to inform my decision making.

• I know what are evidence gap maps.

• Writing more user-friendly reports will ensure evidence-informed

policy happens.

www.3ieimpact.org

Q: Why bother with rigorous evidence synthesis?

www.3ieimpact.org

Why ‘evidence synthesis-informed’

(rather than just ‘evidence-informed’)

• Where evidence is

used it is often

based on single

studies

• Where a range of

studies are used,

not clear how

representative

• Quality of evidence

used is often low

www.3ieimpact.org

What should the government do about NREGA?

“…NREGA is an absolute

disaster.”

- Rajeev Srinivasan,

Management Consultant

“The factual basis of [the

NREGA] debate has made is

possible for extremist

positions to flourish...”

- Jean Dreze and Christian

Oldiges (2012, p.21)

www.3ieimpact.org

History of Indian employment guarantee

Provides legal

right to

demand work

from the state

(1979)

Decentralised

delivery and

targeting of

women, but

poor targeting

Employment

and asset

creation, but

top-down and

poor targeting

Self-targeting,

wages paid in

cash and food

grains

Rights-based,

decentralised

delivery, self-

targeting,

employment

asset creation

www.3ieimpact.org

Evidence gap maps: rigorous evidence fast

• Thematic evidence collection on programmes,

e.g. on a range of interventions (sanitation

promotion), or a particular programme (NREGA)

• Use a framework presenting a matrix of policy

relevant interventions and process, intermediate

outcomes and impacts

• A tool to navigate the evidence base

www.3ieimpact.org

Understanding complex programmes D

em

ocra

cy

Participation in decision making

Decen

tralis

n

Build local government capacity

Co

rru

pti

on

Reduce leakage and mis-appropriation

Dir

ec

t

Beneficiaries:

- Wage income

- Consumption

- Food security

Ind

ire

ct

Beneficiaries:

- Vulnerability

- Financial inclusion

- Migration

In

dir

ec

t Economy-wide:

- Productive sector growth

- Real wage increases

Go

vern

an

ce

Liv

eli

ho

od

s

Em

po

we

rmen

t

Wo

me

n

- Intra-household power

- Burden of time

Ch

ild

ren

- Reduced child labour

- Better school outcomes

Dis

ad

van

tag

ed

- Ability to demand work

- Dignity (and stigma)

www.3ieimpact.org

0

20

40

60

80

100

120

140

160

180

200

NREGA evidence gap map summary

0 20 40 60

Empowerment

Governance

Livelihoods (indirect)

Livelihoods (direct)

0 50 100 150 200 250 300 350

Other

Governance

Performance

Asset Creation

Wages

Access

Outcomes: impact evaluations

Process: broader evidence

www.3ieimpact.org

Andhra Pradesh

Process and Implementation Outcomes and Impact

Intervention/Outcomes

Access Wages Asset Creation Performance Governance Other

Economic Direct Economic Indirect Governance Empowerment NREGA Cost-Effectiveness/ Cost benefit Analysis

Aw

are

ness

Targ

eting

Job C

ard

s

Availabili

ty

Beneficia

ry P

art

icip

ation

Dis

tance f

rom

work

sites

Work

Facilitie

s

Avera

ge W

ages (pert

ain

ing

to N

RE

GA

work

s,

season,

changes in w

ages o

ver

tim

e/r

egio

ns)

Full

Mode o

f P

aym

ent

Tim

ely

manner

Unem

plo

ym

ent A

llow

ance

Wate

r

Securi

ty/M

anagem

ent

Soil c

onserv

ation a

nd L

and

Pro

ductivity

Rura

l C

onnectivity

Work

s o

n s

pecifie

d la

nds

Oth

er

Pla

nnin

g a

nd E

xecution

Sta

ff

Fin

ancia

l

Record

Main

tenance

Innovation

GP

/GS

/Panchayat

involv

em

ent

Socia

l A

udits

Corr

uption

Com

pla

ints

and Issues

Converg

ence

B

eneficia

ry v

iew

s o

n

NR

EG

A's

influence (

perc

eiv

ed li

velih

ood

changes -

savin

gs, pri

vate

assets

, jo

b s

atisfa

ction)

Em

plo

ym

ent

Wages, A

ssets

, W

elfare

Spendin

g P

att

ern

Povert

y

Food S

ecuri

ty

Fin

ancia

l Inclu

sio

n

Mig

ration

Aff

ect on A

gri

culture

rela

ted facto

rs

Food P

rice Inflation

Pro

ductivity

Corr

uption

Em

pow

erm

ent (e

conom

ic,

political, s

ocia

l,

educational, g

ender)

Pro

gra

mm

e c

om

ponents

Government

Supply Side

Aiyar et

al., 2009 Sheahan

et al 2014 Aiyar et

al., 2009 Kumar et

al 2008

Shankar

and

Shylashri,

2010

Pankaj

and

Tankha,

2012 Reddy,

2012 Kumar et

al 2008

Pankaj

and

Tankha,

2012 Kumar et

al 2008 Reddy,

2012 Reddy,

2012 MoRD,

NS Reddy,

2012 Kumar et

al 2008

Imbert

and Papp,

2011 Sheahan

et al 2014 Dreze,

2011b Kumar et

al 2008 Afridi et

al., 2013

Shankar

and

Shylashri,

2010 Afridi et

al., 2013 CGG,

2009

Singh and

Vutukuru,

NS Sheahan

et al 2014 Aiyar et

al., 2009 Sheahan et al

2014

Singh and

Vutukuru,

NS Rao et al,

2008 MoRD,

NS Reddy,

2012

Ravi &

Aravinda,

2009

Ravi &

Aravinda,

2009

Shankar

et al.,

2011 MoRD,

NS MoRD,

NS

Singha

and

Kumar,

2014 MoRD,

NS

Singha

and

Kumar,

2014 MoRD,

NS

Pankaj

and

Tankha,

2012 Afridi et

al., 2013 MoRD,

NS

Singh and

Vutukuru,

NS Imbert and

Papp, 2011

The

National

Consortiu

m on

NREGA,

2009

Pankaj

and

Tankha,

2012

The

National

Consortiu

m on

NREGA,

2009 Maiorano,

2014

The

National

Consortiu

m on

NREGA,

2009

Pankaj

and

Tankha,

2012

Singha

and

Kumar,

2014

Singha

and

Kumar,

2014

Singha

and

Kumar,

2014

Singha

and

Kumar,

2014 Rajsekhar

, NS

Pankaj

and

Tankha,

2012

Pankaj

and

Tankha,

2012

Shankar

and

Gaiha.,

2011

MoRD,

NS

Pankaj

and

Tankha,

2012

Shankar

et al.,

2011

CGG,

2009

Singha

and

Kumar,

2014

Shankar

et al.,

2011

Shankar

and

Gaiha.,

2011

Singh and

Vutukuru,

NS

Murthya

and

Indumatib,

2011

Participant Demand

Side

Ravi et al

2013 Reddy et

al, 2010

Athukoral

a et al.,

2008 Ravi et al

2013 Jha et al.,

2012 Reddy et

al, 2010

Pellissary

and Jalan,

2011

Reddy et

al, 2010

Kareemull

a et al.,

2009

Kareemull

a et al.,

2009

Kareemull

a et al.,

2009

Kareemull

a et al.,

2009

Reddy et

al, 2010

Reddy et

al, 2010 Reddy et

al, 2010 Reddy et

al, 2010

Kareemull

a et al.,

2009 Uppal,

2009 Esteves et

al., 2013 Uppal,

2009 Ravi et al

2013 Esteves et

al., 2013

Athukoral

a et al.,

2008

Kareemull

a et al.,

2009

Esteves et

al., 2013 Uppal,

2009

Jha et al.,

2010 Jha et al.,

2012

Pellissary

and Jalan,

2011

Reddy et

al, 2010

Kareemull

a et al.,

2009

Ravi et al

2013

Uppal,

2009

All programme or

other aspects

CBGA,

2006

Galab and

Revathi,

2012 CBGA,

2006

Ravi and

Engler,

NS

Galab and

Revathi,

2012 Datta et

al., 2009 Sastry,

2007

Ravi and

Engler,

NS

Mukhopad

hyay,

2012 Kamath et

al., 2008 Datta et

al., 2009

Shah and

Jose,

2012

Shah and

Jose,

2012

Shah and

Jose,

2012

Shah and

Jose,

2012 Kamath et

al., 2008

The

National

Consortiu

m on

NREGA,

2009 Sastry,

2007

Shah and

Jose,

2012

The

National

Consortiu

m on

NREGA,

2009 Sastry,

2007 Sastry,

2007

Ravi and

Engler,

NS Johnson,

2010

Ravi and

Engler,

NS Jha et al.,

2011

Ravi and

Engler,

NS Johnson,

2009 Deininger

et al, 2013

Sastry,

2007 Datta et

al., 2009

Galab and

Revathi,

2012 Johnson,

2010

Haque,

2012

Haque,

2012 Haque,

2012 Haque,

2012 Haque,

2012 Dilip et al.,

2013 Kamath et

al., 2008 Datta et

al., 2009 Haque,

2012

The

National

Consortiu

m on

NREGA,

2009 Kamath et

al., 2008 Sastry,

2007 Datta et

al., 2009 Deininger

et al, 2013 Gehrke,

2013

Galab and

Revathi,

2012

Kamath et

al., 2008 Gaiha et

al., 2010

Mukhopad

hyay,

2012

Shah and

Jose,

2012

Pankaj,

2012

Sastry,

2007 Sastry,

2007 Sastry,

2007 Sastry,

2007 IIS, 2013

Mukhopad

hyay,

2012 Datta et

al., 2009 Datta et

al., 2009 Gaiha et

al., 2011 Jha et al.,

2011 Illuru et

al., 2013

Datta et

al., 2009

CBGA,

2006 Kamath et

al., 2008

Mukhopad

hyay,

2012

The

National

Consortiu

m on

NREGA,

2009

The

National

Consortiu

m on

NREGA,

2009

The

National

Consortiu

m on

NREGA,

2009

The

National

Consortiu

m on

NREGA,

2009 Doug,

2008 Doug,

2008 Varshney

et al 2014 Varshney

et al 2014

Sastry,

2007 Dilip et al.,

2013 Gaiha et

al, 2010 Datta et

al., 2009 Datta et

al., 2009 Datta et

al., 2009 Datta et

al., 2009

IIS, 2013

Kamath et

al., 2008 Datta et

al., 2009

CBGA,

2006

Varshney

et al 2014 Varshney

et al 2014

Gaiha et

al., 2011 Sastry,

2007

Gaiha et

al., 2010

The

National

Consortiu

m on

NREGA,

2009

Kamath et

al., 2008

Gaiha et

al., 2011

Gaiha et

al., 2010

Process and Outcomes

Evidence and

gaps

www.3ieimpact.org

Meghalaya

Process and Implementation Outcomes and Impact

Intervention/Outcomes

Access Wages Asset Creation Performance Governance

Other

Economic Direct Economic Indirect Governance Empowerment NREGA Cost-Effectiveness/ Cost

benefit Analysis

Aw

are

ne

ss

Ta

rge

tin

g

Jo

b C

ard

s

Ava

ila

bility

Be

ne

ficia

ry P

art

icip

atio

n

Dis

tan

ce

fro

m w

ork

site

s

Wo

rk F

acilitie

s

Ave

rag

e W

ag

es (p

ert

ain

ing

to N

RE

GA

wo

rks, se

aso

n, ch

an

ge

s in

wa

ge

s o

ver

tim

e/r

eg

ions)

Fu

ll

Mo

de

of

Pa

ym

en

t

Tim

ely

ma

nn

er

Une

mp

loym

ent A

llo

wance

Wa

ter S

ecu

rity

/Ma

na

gem

ent

So

il c

on

se

rva

tio

n a

nd

La

nd

Pro

du

ctivi

ty

Rura

l C

on

ne

ctivi

ty

Wo

rks o

n s

pe

cifie

d lan

ds

Oth

er

Pla

nn

ing

an

d E

xecu

tio

n

Sta

ff

Fin

an

cia

l

Reco

rd M

ain

ten

an

ce

Inn

ova

tio

n

GP

/GS

/Pa

nch

aya

t in

volv

em

en

t

So

cia

l Au

dits

Corr

up

tio

n

Com

pla

ints

an

d Issu

es

Con

verg

en

ce

Be

ne

ficia

ry v

iew

s o

n N

RE

GA

's in

flu

en

ce

( p

erc

eiv

ed

live

lih

oo

d c

han

ge

s -

sa

vin

gs, p

riva

te a

sse

ts, jo

b s

atisfa

ctio

n)

Em

plo

ym

ent

Wa

ge

s, A

sse

ts, W

elfa

re

Sp

en

din

g P

att

ern

Po

vert

y

Fo

od

Se

cu

rity

Fin

an

cia

l In

clu

sio

n

Mig

ratio

n

Ag

ricu

ltu

re re

late

d fa

cto

rs

Fo

od

Pri

ce

Infla

tio

n

Pro

du

ctivi

ty

Corr

up

tio

n

Em

po

werm

ent (e

co

no

mic

, p

olitica

l, s

ocia

l, e

du

ca

tion

al, g

en

de

r)

Pro

gra

mm

e c

om

ponents

Government Supply

Side

Singha

and

Kumar,

2014

Shah and

Jose,

2012

Singha

and

Kumar,

2014

Singha

and

Kumar,

2014

Singha

and

Kumar,

2014

Singha

and

Kumar,

2014

Singha

and

Kumar,

2014

Singha

and

Kumar,

2014

Shah and

Jose,

2012

Shah and

Jose,

2012

Shah and

Jose,

2012

Shah and

Jose,

2012

Shah and

Jose,

2012

Participant Demand

Side

All programme or

other aspects

Panda et

al., 2009 AMC, NS Mishra et

al., 2010 Mishra et

al., 2010 AMC, NS Panda et

al., 2009 Panda et

al., 2009 AMC, NS Panda et

al., 2009 Panda et

al., 2009 AMC, NS Haque,

2012 Haque,

2012 Haque,

2012 Haque,

2012 Haque,

2012 AMC, NS

Ravi and

Engler,

NS

Ravi and

Engler,

NS

Ravi and

Engler,

NS

Panda et

al., 2009

Ravi and

Engler,

NS AMC, NS AMC, NS Pankaj,

2012

AMC, NS

Ravi and

Engler,

NS

Haque,

2012

Ravi and

Engler,

NS

Big gaps

www.3ieimpact.org

0

5

10

15

20

25

2009 2010 2011 2012 2013 2014 NS

Simulation Study

Impact evaluation usingquasi-experimentalmethods

Experimental impactevaluation

Rigorous counterfactual studies on NREGA

Big

gap

s

www.3ieimpact.org

What gap maps can and can’t do

• Focus on providing a broad overview of the existing

evidence quickly (1-3 months)

• Provide recommendations for research (future evaluations

and systematic reviews)

• Can provide general policy guidance when based on

systematic reviews

• No rigorous synthesis of findings so no in-depth guidance

for policy and programmes

• ‘Global public good’, updated periodically

www.3ieimpact.org

Group exercise

• Choose a programme as a group (e.g.

NREGA, Swach Bharat, Jan Dhan, or any

other programme).

• Discuss how you would implement an

‘evidence gap map’ to be policy relevant:

– Which interventions should the map cover?

– Which outcomes should be covered?

– How would you use the information obtained?

www.3ieimpact.org

Systematic reviews

• Focus on providing an indepth analysis of the existing

evidence (12-18 months)

• Provide guidance for policy

• Provide guidance for programmes

• May provide guidance for implementers

• Provide guidance for research

• ‘Global public good’, updated periodically

www.3ieimpact.org

Systematic reviews answer relevant questions

Policy question Type of review

What difference is made by a

programme (does it work)?

Review of effects drawing on evidence

from counterfactual impact evaluations

How are programmes designed?

Portfolio review of project documentation

What are the important considerations

for implementation (barriers and

enablers)?

Review of implementation, drawing on

factual evidence e.g. process evaluations

and qualitative studies

What are stakeholders’ experiences or

preferences (eg willingness to pay)?

Review of participant views, drawing on

qualitative studies and descriptive

quantitative studies

How efficient is the programme in

turning inputs into outcomes?

Review of cost-effectiveness, cost-utility

and cost-benefit evidence

www.3ieimpact.org

Swach Bharat abhiyan

• Half of the world’s open defecators are in

India

• Implications for child health and nutrition,

women’s safety, labour productivity

• Programme objectives include:

– Eradicate open defecation

– Improve toilet facilities

– Behaviour change to healthy sanitation

practices

– Awareness raising on public health

www.3ieimpact.org

Water-related disease transmission

Source: Cairncross et al. 2010 (Wagner and Lanoix 1957)

Threshold effects?

www.3ieimpact.org

WASH systematic reviews search process

74,181 records identified through database search

Scre

en

ing

In

clu

de

d

Elig

ibili

ty

Id

en

tifi

cati

on

1,024 records identified through other sources

49,472 records after duplicates removed

1,869 records screened

47,603 records irrelevant

225 full-text articles assessed

120 full-text articles excluded

137 studies included in quantitative

synthesis of effects

44 studies included in qualitative synthesis

‘Factual’ evaluation

studies

‘Counterfactual’ impact

evaluation studies

www.3ieimpact.org

Hierarchies of evidence

Causal chain area Type of evidence Programme design

Implementation fidelity

Project and programme

documents

Beneficiary targeting

Implementation performance

Beneficiary views about programme

‘Factual’ evidence (qualitative

and/or quantitative)

Impacts on beneficiaries

Impacts on non-beneficiaries

Societal and economy-wide impacts

‘Counterfactual evidence’ (impact

evaluations and simulation

studies)

Efficiency Cost-benefit, cost-effectiveness,

cost-utility evidence

www.3ieimpact.org

Evidence from Total Sanitation Campaign

• Odisha (Clasen et al. 2014)

– Modest increase in access to latrines

– But limited use by men and children

– No effect on faecal contamination of wells

and household stored water

– No effect on faecal contamination of

hands

– No impacts on reported diarrhoea

• Madhya Pradesh (Patil et al. 2014)

• Modest increase in access to latrines

• Modest reductions in open defecation

• No effect on child health outcomes

(diarrhoea, parasite infection, anemia,

growth)

www.3ieimpact.org

Evidence from national hygiene campaigns

• Vietnam:

– Handwashing BCC (not soap) scale-up

through 1) mass media & 2) inter-personal

communication

– No health or productivity effects

• Peru:

– Hygiene promotion through 1) mass media & 2)

community level (health facilities, schools)

– No impacts health or nutrition (compliance measured)

• Bangladesh:

– SHEWA-B local community hygiene promoters

– Low compliance rates & no impacts on diarrhoea or ARIs

www.3ieimpact.org

Supply doesn’t create its own demand

• Reduction in child disease rates not observed by carers or seen as substantial enough benefit to warrant costs (money/time) – health education doesn’t work

• Adoption of innovations (social change) is a slow process (early adopters vs. laggards) – role for ‘triggering’ (CLTS)?

• Perceived benefits important (e.g. user satisfaction, time-use, safety) – facility maintenance

www.3ieimpact.org

BEHAVIOURAL CONTEXT (Pre-project period)

Individual characteristics and abilities: Age, gender, pre-existing WASH practices, knowledge, skills, self-efficacyHousehold demographics: Socioeconomic status, household structure, religionCommunity/socio-political structure: Social norms, access to WASH technologies and materials, health system structure, governmental policies and financial support

Early introduction

Communities and individuals are first introduced to WASH technologies and promotion messages. • May have previously been exposed, or may be new to campaigns.• WASH technologies or messaging may be new or exciting• Promoter may be viewed as an influential person

Maintenance during project period

Promotion continues to end of intervention period. • Health promoters assist with problems• Participants able to practice behaviours independently

Continued use (short-term) after end of project period

Intervention period ends. Participants practice WASH behaviours without study support. • Messaging and education still “fresh” in participants’ minds• Lack of reminders from promoter visits

Sustained adoption (long-term) after end of project period

Ranges from 6 months to years after project period ends. Captures true behaviour practice without influence of intervention promotion.• Habitual behaviour established• New families unfamiliar with WASH interventions may not practice

behaviour• Cost may be unsustainable

PR

OJE

CT

PER

IOD

PO

ST P

RO

JEC

T P

ERIO

DWASH Technology

Knowledge/skills of WASH behaviour

Early project period:

funding and

technology promotion

Late project period:

promotion stops,

beneficiaries assess

effectiveness

Early post-project

period: funding ends,

messages remain

fresh

Late post-project

period: equipment

break-down,

messages may be lost

www.3ieimpact.org

Jan Dhan Yojana

• Open zero-balance bank accounts for 75

million households, including providing

insurance and overdraft facilities

• Aims to eradicate poverty and empower

women

• But will the programme help to empower

women?

www.3ieimpact.org

Systematic review on micro-credit

•Effects micro-credit on women’s control over

household spending in L&MICs

•25 impact evaluations found examining

causal relationship

•Majority (15 of the 25) found a positive and

statistically significant relationship

•Do we conclude microcredit is empowering?

www.3ieimpact.org

Errors in hypothesis testing

=> This is why we need study replication

and meta-analysis

H0 ‘true’ H0 false

Do not reject H0

No error

Type II error

maybe 20% but

often 40-60%

Reject H0

Type I error

= 5%

No error

www.3ieimpact.org

Statistical meta-analysis

•Data-pooling and statistical synthesis of

independent studies

•Aggregating/cumulating samples and findings

•Seeks to measure and control bias

•Explore heterogeneity in findings

www.3ieimpact.org

Study

0

Showing the result in a forest plot

www.3ieimpact.org

Study

0 0=no effect

Showing the result in a forest plot

www.3ieimpact.org

Study Less empowerment More empowerment

0 0=no effect

Showing the result in a forest plot

www.3ieimpact.org

India

Study Less empowerment More empowerment

0 0=no effect

Showing the result in a forest plot

Confidence interval

www.3ieimpact.org

India

Study Less empowerment More empowerment

0 0=no effect

Showing the result in a forest plot

Confidence interval

-0.1

www.3ieimpact.org

India

Study Less empowerment More empowerment

0 0=no effect

Showing the result in a forest plot

Confidence interval

-0.1

Pooled effect

www.3ieimpact.org

Overall

Bangladesh

Kyrgyzstan

Morocco

South Africa

India

South Africa

India

Study Less empowerment More empowerment

0 -0.5 0 0.5

Meta-analysis of rigorous study evidence

Standardised mean difference in outcomes

www.3ieimpact.org

Implications

“There is no evidence for an effect of microcredit

on women’s control over household spending…

(and) it is therefore very unlikely that, overall,

microcredit has a meaningful and substantial

impact on empowerment processes in a broader

sense.” (Vaessen et al. 2014)

• Provide (separate) bank accounts to women?

• Complementary interventions (business training,

awareness raising)?

www.3ieimpact.org

How to make evidence synthesis more relevant?

1. Relevant approach

2. Involve users throughout the study

3. Communicate findings effectively

4. Build capacity

www.3ieimpact.org

Group exercise

How would you make evidence synthesis

relevant for the policy cycle?

• Which institutions and decision-makers

(policy/programmes) would you involve?

• When would you involve them?

• What would be the scope of the study?

• How would you communicate findings?

www.3ieimpact.org

“Our skills should be reserved for the evaluation

of policies and programs that can be applied in

more than one setting... The lack of this

knowledge makes us incompetent estimators of

programme impacts, turning out conclusions that

are not only wrong, but are often wrong in

socially destructive ways.”

Donald T. Campbell (1979, p.84)

Thanks: hwaddington@3ieimpact.org

www.3ieimpact.org

A systematic review is NOT a literature review Characteristic Literature reviews Systematic reviews and meta-

analysis

Perspective Espousal of position Neutral representation

Coverage Selective, usually published

literature only

Exhaustive (or representative),

published and unpublished

Audience Scholars Policy makers (and practitioners)

Process Unclear decision making

process

Clear protocol for inclusion or

exclusion, data extraction, analysis

and reporting

Critical appraisal At best, study design

mentioned only

In-depth assessment of internal &

external (and construct) validity

Synthesis method

(impacts) Null-hypothesis significance

testing (‘vote counting’)

Synthesis of effect sizes, studies

weighted by precision

Study team Carried out individually Conducted in teams, double-coding