doug gollin standing panel on impact assessment (spia)
TRANSCRIPT
Standing Panel on Impact AssessmentDoug Gollin, SPIA Chair, 15 Sept 2016
Contenthttp://impact.cgiar.org/
• Work since last ISPC meeting (Lima, May 2016)• First SIAC synthesis report: The “rigor revolution” in impact
assessment for agricultural researchCausal identificationMeasurementRepresentativeness of sampling
• Towards a second phase for the SIAC program
Recent calendar http://impact.cgiar.org/
June• 3ie / IFAD workshop “Designing and implementing high-quality, policy-
relevant impact evaluations” (SPIA contribution: DNA fingerprinting into IEs)• FERDI / SPIA workshop “Agricultural innovation: Learning for adopting”
(Portfolio of 4 SIAC RCTs, linking with other RCTs outside, and thinking broadly about social sciences across CGIAR + Science of innovation / impact)
• MEL CoP “Taskforce on selection of harmonized indicators” (advice / caution)
July• Impact Assessment Focal Point Meeting (our main CoP – identify CAPI need)• SIAC long-term / large-scale impact assessment studies: Mid-term meeting
(Portfolio of 7 studies, updates, course-correction)August• SIAC workshop on innovative methods for measuring adoption of
agricultural technologies (Fingerprinting, remote sensing, SMS and CAPI surveys - establishing proof of concept and thinking about scaling up)
Previous approaches to IA in the CGIARhttp://impact.cgiar.org/
• SPIA core business for a long time was to generate aggregate estimates of rates of return to investments in the CGIAR
• Methodologically simple but crude• Appropriate for the era:
• Technologies easily identified• Impacts largely unidimensional
• Impact modelled as an increase in production multiplied by a price.
• Consumers and/or producers benefit depending on assumptions• Extent of broad pool of benefits is a function of:
% adoptionAverage productivity gain per unit adoption
The ‘rigor revolution’ in IAhttp://impact.cgiar.org/
• SPIA commissioned paper to Alain de Janvry and Elisabeth Sadoulet in 2010
• Their paper “Recent Advances in Impact Analysis Methods for Ex-post Impact Assessments of Agricultural Technology: Options for the CGIAR”, published in 2011 laid foundations for big changes
• Emphasized the need to take seriously that comparisons of adopters and non-adopters do not account for all the differences between them, especially in relation to “unobservables.”
• Recommended a portfolio of RCTs
1. Causal Identificationhttp://impact.cgiar.org/
• SPIA took on board these recommendations.• Opted not to focus entirely on RCTs.• Methodologically agnostic – we want careful and appropriate
combinations of methods• More complicated question is being clear about which non-
experimental research designs are appropriate, and in what contexts
Institutionalizing new methodshttp://impact.cgiar.org/
• Quality-rating system for impact studies / claims in CGIAR launched earlier this year – no voluntary take-up
Future:• Shift into regular audits / reviews of impact claims (retrospective)
as well as continuing a forward looking advice function reviewing research designs for future impact studies (prospective)
• Periodic, predictable synthesis reports on state of knowledge of impacts
2. Measurement mattershttp://impact.cgiar.org/
Adoption• Often the critical missing data for understanding impacts from
different streams of research• Definition and measurement of adoption far from simple
Outcome variables• Productivity (Plot area measurement? Crop-cuts?)• Remote sensing for environmental benefit streams?• Data on diets? Anthropometry?
Genotype Farmer-elicited name
Maize in Uganda: SPIA / LSMS-ISA / UBoS / Diversity Arrays
• Data from 540 HHs in 45 enumeration areas
• Enumerators from UBoS trained for 1 month
• CAPI-based survey + grain-based highly-quantitative DArTSeqgenotyping
• 2% of farmers were correct about the variety they were growing
Line transects: Reference value
Enumerator with visual-aid
Self-report with visual-aid
Enumerator estimate
Self-report
Remote-sensing
Drone
On the ground or from the air? Measuring crop residue retention, Ethiopia (teff, maize, wheat, barley)
3. Statistical representativenesshttp://impact.cgiar.org
• World Bank Living Standards Measurement Study –Integrated Surveys of Agriculture (LSMS-ISA)
• 8 countries in SSA – all important to CGIAR• Average of 5,000 HHs / country, nationally
representative• Panel – visited every 2 years
SPIA role:• Surveys lack modules / questions on agricultural
technologies (varieties, NRM practices)• SPIA’s comparative advantage to work to improve
this for benefit of CGIAR as a whole
Future: • Help bring about a geographic focusing of CGIAR
Towards SIAC phase 2: (2018-2022)http://impact.cgiar.org/
1. Country baselines and monitoring in key geographies
2. Database of claims of policy influence resulting from CGIAR research
3. Maintain focused competitively-commissioned portfolio of ex-post impact assessments
4. Synthesis reports on predictable and regular production cycle
5. Audits of impact claims
6. Improving the prediction of technology success in farmers’ fields
7. Capacity-building of economics / social science function
8. More work on methods as a CGIAR-wide public good
Country baselines and monitoringhttp://impact.cgiar.org/
What?• Proposing 6 high-priority countries + possible spillover countries• Phased approach between now and 2018/19 for first waves• Partnerships with national government agencies and local implementing
bodies with expertise in survey management• Collaboration with CGIAR partners on identifying priorities and piloting
methods
Why SPIA?• CGIAR-wide public goods• Established network of potential partners (World Bank LSMS-ISA; IFAD;
Excellence in Breeding - ICRISAT/CIMMYT; Big Data – CIAT/IFPRI; Diversity Arrays)
• Independence• Long history of documenting technology adoption
SIAC PHASE 1 (piloting now) SIAC PHASE 2? 6 country CRP total (M HHs)
Overall CRP totalEthiopia Tanzania Uganda India B’desh Vietnam
% share
A4NH 0.50 0.56 1.80 2.50 3.10 8.5 20.5 41CCAFS 0.80 0.40 3.00 0.50 1.00 5.7 10.9 53DCL 8.00 10.00 18.0 25 72Fish 0.11 1.80 1.9 4.9 39FTA 4.20 1.20 1.50 6.20 2.60 15.7 41.3 38Livestock 2.05 1.44 0.31 0.48 0.16 0.11 4.6 6.5 70Maize 2.50 1.10 0.30 3.70 0.80 8.4 15.0 56PIM 1.00 5.00 2.00 1.50 9.5 14.5 66Rice 0.09 4.27 1.19 0.76 6.3 16.5 38RTB 0.30 0.80 0.30 1.4 8.0 18Wheat 2.00 0.01 8.00 0.34 10.4 17.2 60WLE 1.00 0.55 12.50 2.75 16.8 21.0 80Country total (M HHs) 22.05 10.76 4.71 52.95 12.14 4.47 107.1 201.2 53
% share of all rural HHs 127 141 72 29 52 29
N.B. - CRP data from March full proposal versions (not yet checked against revisions)