Slide 1

Slide 2

Monitoring Progress & Evaluating Impact Presented by: Cindy Banyai, Ph.D. Executive Director, efocus Institute IREX Community Solutions Program September 1, 2011 September 8, 2011

Slide 3

Learning Objective Identify and employ key methods and tools for monitoring progress and evaluating impact 2 2011 IREX CSP, Cindy Banyai, refocusinstitute.com

Slide 4

Project Management Cycle (PMC) 3 Based on Miyoshi, 2008 2011 IREX CSP, Cindy Banyai, refocusinstitute.com

Slide 5

Monitoring Progress Monitoring - mid-term and terminal evaluations Examine implementation process -- assess inputs through outputs Measure performance assess outputs to intermediate outcomes 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 4

Slide 6

Evaluating Impact Impact evaluation measures relationship between intermediate outcomes and end outcomes Sustainability evaluation measures long- term impacts, periphery effects of program 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 5

Slide 7

USAID on Impact Evaluation Measures the change in a development outcome that is attributable to a defined intervention Based on models of cause and effect Requires a credible and rigorously defined counterfactual to control for factors other than the intervention that might account for observed change Comparisons between beneficiaries that are randomly assigned to either a treatment or a control group provide the strongest evidence of a relationship between the intervention under study and the outcome measured (USAID 2011) 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 6

Slide 8

DAC 5 Evaluation Criteria Efficiency - A measure of how economically resources/inputs (funds, expertise, time, etc.) are converted to results. Effectiveness - The extent to which the development interventions objectives were achieved, or are expected to be achieved, taking into account their relative importance. 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 7

Slide 9

DAC 5 Evaluation Criteria (cont) Impact - Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. Sustainability - The continuation of benefits from a development intervention after major development assistance has been completed. The probability of continued long- term benefits. The resilience to risk of the net benefit flows over time. 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 8

Slide 10

DAC 5 Evaluation Criteria (cont) Relevance - The extent to which the objectives of a development intervention are consistent with beneficiaries requirements, country needs, global priorities and partners and donors policies. (DAC 2002) 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 9

Slide 11

10 Relationship between Evaluation Criteria & Logic Model End Outcomes Intermediate Outcomes Outputs Inputs impact effectiveness efficiency Relevance sustainability Activities Measuring performance Examining process Assessment of 5 criteria Measuring performance Examining implementing process Miyoshi 2008

Slide 12

TOOLS FOR MONITORING AND EVALUATION 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 11

Slide 13

Traditional types of data gathering Surveys Tests Interviews Observation Documents 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 12

Slide 14

Participatory types of data gathering Focus groups cooperative inquiry Training Action research project Photographs Video Plays/storytelling Metaphor drawing sculpture 13 2011 IREX CSP, Cindy Banyai, refocusinstitute.com

Slide 15

Quantitative data Need counterfactual - situation a participating subject would have experienced had he or she not been exposed to the program Baseline or pre-program data used to assess impact Predict outcomes that might result from the program (as in ex ante evaluations) Make before-and-after comparisons (also called reflexive comparisons) (WB 2010) 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 14

Slide 16

Quantitative impact evaluation Ex-ante impact evaluation - measures intended impacts of future policies/programs/projects, predicts program impacts. Looks at a potentially targeted areas current situation May involve simulations based on assumptions about how the economy works Based on structural models of the economic environment facing potential participants 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 15

Slide 17

Quantitative impact evaluation Ex-post impact evaluations - measure actual impacts accrued by beneficiaries attributable to policy/program/project Examines immediate benefits, reflects reality Sometimes miss the mechanisms underlying the programs impact on the population, which structural models aim to capture (WB 2010) 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 16

Slide 18

Quantitative methods for ex-post impact evaluation Randomized evaluations Matching methods, specifically propensity score matching (PSM) Double-difference (DD) methods Instrumental variable (IV) methods Regression discontinuity (RD) design and pipeline methods Distributional impacts Structural and other modeling approaches (WB 2010) 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 17

Slide 19

Qualitative data 3 main sources of qualitative data open- ended interviews, observations, documents (Patton 2002) Emerging sources photographs, video, artistic expressions (Hesse-Beiber and Leavy 2008) Can be coded, transformed into statistics 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 18

Slide 20

Qualitative research Looks at a phenomenon, situation in-depth and detail The researcher is the instrument validity relies on her skill, competence and rigor Internal validity when qualitative data is corroborated from multiple sources 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 19

Slide 21

Quantitative vs. Qualitative Need both to provide depth and understanding of situation Numbers balance narrative accounts Stories and visuals put a human touch on numbers 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 20

Slide 22

Determine approach and tools Use these questions to guide your selection Who is the information for and who will use the findings? What kinds of information are needed? How is the information to be used? For what purposes are the evaluation being done? When is the information needed? What resources are available to conduct the evaluation? Given the answers to the proceeding questions, which methods are appropriate? (Patton 2002) 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 21

Slide 23

Parting thoughts on M&E Collect the data you need. Use the data you collect. Doing so improves your efficiency and the clarity of your evaluations, in addition to freeing up time for other program activities. 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 22

Slide 24

Thanks for your attention! Please send questions & comments to photoevaluation@refocusinstitute. com cindy.banyai@gmail.com

Slide 25

References Banyai, Cindy. (2010). Community capacity and governance New approaches to development and evaluation. PhD diss., Ritsumeikan Asia Pacific University. Bureau for Policy, Planning, and Learning United States Agency for International Development [USAID]. (2011.) Evaluation Policy. Washington, D.C.: USAID. Development Assistance Committee [DAC] of the Organization for Economic Cooperation and Development [OECD]. (2002). Glossary of Terms in Evaluation and Results Based Management. Paris: OECD. Hesse-Beiber, Sharlene Nagy and Leavy, Patricia. (2008). Handbook of Emergent Methods. New York: Guilford Press. Japan International Cooperation Agency - Office of Evaluation and Post Project Monitoring, Planning and Evaluation Department (2004). JICA Evaluation Handbook: Practical Methods for Evaluation. Tokyo: Japan International Cooperation Agency. Khandaker, Shahidur R., Koolwal, Gayatri B., Samad, Hussain A. [WB] (2010). Handbook on Impact Evaluation: Quantitative Methods and Practices. Washington, D.C.: World Bank. Miyoshi, Koichi. (2008). What is Evaluation?. In Miyoshi, Koichi (Ed). Hyoka-ron wo Manabu Hito no tameni (For People Learning Evaluation Theory). (pp. 1-16). Tokyo. Sekaishisosha. Patton, Michael Quinn (2002). Qualitative Research & Evaluation Methods. Thousand Oaks: Sage Publications, Inc. 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 24