WHAT IS AN EVALUATION? Impact Evaluation Course Cape Town, South Africa 18 January 2016 Jeremy Magruder

Outline 1.  Evidence Based Policymaking : Why Evaluate? 2.  Levels of Evaluation

a)  Needs Assessment b)  Theory of Change c)  Process Evaluation d)  Impact Evaluation e)  Cost Effectiveness Analysis

3.  Good Impact Evaluation Questions

Outline 1.  Evidence Based Policymaking : Why

Evaluate? 2.  Levels of Evaluation

a)  Needs Assessment b)  Theory of Change c)  Process Evaluation d)  Impact Evaluation e)  Cost Effectiveness Analysis

3.  Good Impact Evaluation Questions

What is Policy? • Dic$onary:  “A  course  or  principle  of  ac$on  adopted  or  proposed  by  an  organiza$on  or  individual.”  

• Wikipedia:  “A  principle  or  protocol  to  guide  decisions  and  achieve  ra$onal  outcomes.”  

How  do  we  know  if  a  policy  achieves  ra$onal  outcomes?  

How is policy decided?

Quiz: How is policy decided?

A.  Ideology  (i.e.,  of  decision  makers,  par$es,  cons$tuencies)  

B.  Experience  (i.e.,  of  decision  makers,  par$es,  cons$tuencies)  

C.  Financial  resources/constraints  D.  Implementa$on  capacity  

E.  Scien$fic  evidence  

Imagine you are a policy maker

How do we increase school participation (enrollment and attendance)?

A  government  wants  to  improve  school  aOendance  at  primary  schools,  what  interven$ons  would  you  recommend?  

How do we increase school participation (enrollment and attendance)? A.  Text  Books    

B.  Lunch  for  free  

C.  Free  school  uniforms  

D.  Treat  intes$nal  worms  

E.  Merit  scholarships  

F.  Improve  curriculum  &  teaching  

G.  Provide  beOer  materials  

H.  Increase  awareness  of  returns  to  educa$on  

Impact evaluations can help answer these questions

How do you think increased use of evidence would change policy decisions? A.  Change  policy  priori$es  

B.  New  programs  

C.  New  implementa$on  models  

D.  Halt  of  programs  that  do  not  work  

E.  Reallocate  resources  towards  higher  impact  programs  

F.  Reallocate  resources  toward  more  cost-­‐effec$ve  programs  

What if the evidence is not there?

Why evaluate?

A.  To  know  whether  a  program  is  implemented  as  intended  

B.  To  know  whether  a  policy/  program  works  and  how  

C.  Donors  require  it  D.  To  make  our  policies/  

programs  more  performance-­‐based  

E.  To  be  able  to  improve  the  program  and  make  it  more  effec$ve,  or  abandon  the  program  if  it  is  not  

F.  To  iden$fy  if  there  are  unintended  consequences  

G.  All  of  the  above  

Evaluation

What is evaluation?

A.  A  survey  to  see  how  people  are  doing,  how  they  feel  about  things  

B.  A  structured  way  of  thinking  through  the  details  of  a  program  

C.  Roughly  the  same  thing  as  monitoring  

D.  Judgment  of  whether,  and  how  much  a  person,  organiza$on  or  program  succeeded  or  failed  

E.  All  of  the  above  

Outline 1.  Evidence Based Policymaking : Why Evaluate 2.  Levels of Evaluation

a)  Needs Assessment b)  Theory of Change c)  Process Evaluation d)  Impact Evaluation e)  Cost Effectiveness Analysis

3.  Good Impact Evaluation Questions

Levels of Programme Evaluation

Needs Assessment

Program Theory Assessment

Process Evaluation

Impact Evaluation

Cost-Effectiveness Analysis

Levels of Program Evaluation

Needs Assessment

Needs Assessment

•  Clear  sense  of  target  popula$on  •  Clear  sense  of  need  program  will  fill  •  Clear  ar$cula$on  of  program  benefits  

•  Clear  sense  of  alterna$ves  

Needs Assessment: Descriptive Data • Good  descrip$ve  data  is  cri$cal  for  understanding  the  nature  of  a  problem.  

•  Assessing  the  quality  of  descrip$ve  data:    1.  Is  the  sample  large  and  is  it  representa$ve—ie  does  it  

represent  different  types  of  people  

2.  Are  the  ques$ons  appropriate  to  the  local  context?  

Example: Needs Assessment In  2012  in  Monrovia  (pre-­‐Ebola):    •  Extremely  high  youth  unemployment  (only  20-­‐30%  of  working  age  youth  employed)  

•  Crime  and  an$-­‐social  behavior  is  very  high  

•  Post  conflict  environment:  concerns  about  social  integra$on  and  economic  opportunity  

•  2  widespread  programs  for  social  integra$on  and  economic  opportunity  of  youth  •  Life  skills  trainings  coupled  with  sports  programs    

•  Cash  for  Work  

Levels of Program Evaluation

Theory of Change

Theory of Change • How  will  the  program  address  the  needs  put  forth  in  your  needs  assessment?  • What  are  the  prerequisites  to  meet  the  needs?    

• How  and  why  are  those  requirements  currently  lacking  or  failing?    

• How  does  the  program  intend  to  target  or  circumvent  shortcomings?    

• What  services  will  be  offered?  

Example: Theory of Change

Teachers and

Coaches are hired to train young adults in life skills; curriculum developed

Classes are held and

sports events organized.

Youth Attend.

Youth Learn new life skills;

pro-social attitudes improved

Employment increases; anti-social behaviors decrease

Input Output Outcome Goal

Example: Log Frame Objectives Hierarchy

Indicators Sources of Verification

Assumptions / Threats

Impact (Goal/ Overall

objective)

Higher Employment Rates and lower crime

•  Employment Status

•  Crime/anti-social behaviors

Household survey Skill gaps and social integrations barriers to employment

Outcome (Project

Objective)

Youth learn valuable life skills and increase pro-sociality

•  Survey data •  Test Scores

•  Household surveys

•  Routine data •  Cortisol

testing?

•  Behavior change •  Value to

prosociality /life skills indicators

Outputs Youth Trained in Life Skills and engage insports events

•  Number of Youth participants

•  Attendance logs, unscheduled audits

•  Instruction is valuable

•  Youth learn from this format of course

Inputs (Activities)

Teachers and Coaches hired and trained in curriculum

•  Teachers and Coaches Hired

•  Curriculum Documents

•  Routine data * Teachers and Coaches will hold classes and follow curriculum

Levels of Program Evaluation

Process Evaluation

Process Evaluation • Are  basic  tasks  being  completed?    

• Are  the  services  being  delivered?    •  Is  the  interven$on  reaching  the  target  popula$on?    •  Is  the  interven$on  being  completed  well  or  efficiently  and  to  the  beneficiaries’  sa$sfac$on?  

Process Evaluation • Process  evalua$ons    answer  whether  the  program  or  policy  was  carried  out  as  planned.  

• Good  quality  process  assessment  goes  beyond  coun$ng  inputs  

1.  Not  just  did  the  text  books  arrive  but  are  they  being  used?      

2.  Not  just  were  people  hired,  but  how  ofen  are  they  absent?  

3.  What  percentage  of  eligible  people  take  up  the  program?  

4.  How  many  youth  par$cipated  in  the  program?  

Example: Process Evaluation

Process Evaluation

Teachers and Coaches are hired to train young adults in life skills; curriculum developed

Classes are held and

sports events organized.

Youth Attend.

Youth Learn new life skills;

pro-social attitudes improved

Employment increases; anti-social behaviors

decrease

Input Outcome Goal

Levels of Program Evaluation

Impact Evaluation

Impact Evaluation •  Process  evalua$ons  determine  if  a  program  is  running  in  the  way  it  is  supposed  to  run  •  Ex:  Did  teachers  distribute  deworming  pills?    

•  Impact  evalua$ons  determines  if  a  program  creates  a  change  in  an  outcome(s)  •  Approaches:  non-­‐experimental,  quasi-­‐experimental  and  experimental  methods  (randomized  evalua$ons)  

•  Ex.  Did  financial  rewards  for  hairdressers  result  in            customers  prac$cing  safer  sex  ?    

•  Ex.  Did  the  provision  of  deworming  pills  create        an  increase  in  school  aOendance?    

Example: Impact Evaluation

Process Evaluation Impact Evaluation

Teachers and

Coaches are hired to train young adults in life skills; curriculum developed

Classes are held and

sports events organized.

Youth Attend.

Youth Learn new life skills;

pro-social attitudes improved

Employment increases; anti-social behaviors decrease

Input Outcome Goal

Levels of Program Evaluation

Cost-Effectiveness Analysis

Outline 1.  Evidence Based Policymaking : Why Evaluate 2.  Levels of Evaluation

a)  Needs Assessment b)  Theory of Change c)  Process Evaluation d)  Impact Evaluation e)  Cost Effectiveness Analysis

3.  Good Impact Evaluation Questions

What types of questions can you answer with impact evaluation?

•  Evalua$ons  of  all  kinds  can  be  thought  to  answer  three  types  of  ques$ons  (Imas  and  Rist  2009)  :  •  Descrip(ve  ques(ons  •  Norma(ve  ques(ons  

•  Cause  and  effect  ques(ons  

•  Impact  Evalua$on  can  only  answer  cause  and  effect  ques$ons.  

Which one of these would make a good question for an impact evaluation? A.  What  share  of  kids  in  

Tanzania  drop  out  of  school  before  comple$ng  primary?  

B.  How  should  a  land  contract  be  wriOen?  

C.  Does  providing  kids  with  deworming  pills  or  school  uniforms  do  a  beOer  job  of  keeping  kids  in  school?

Which one of these would make a good question for an impact evaluation? A.  Do  s$ckers  in  a  minibus  

encouraging  passengers  to  remind  their  drivers  to  drive  safely  reduce  accidents?  

B.  What  share  of  farmers  in  Kenya  currently  live  on  less  than  $2  a  day?  

C.  Which  kind  of  fer$lizer  works  best  for  a  plot  of  maize?  

Which one of these would make a good question for a process evaluation? A.  Does  a  sexual  educa$on  

program  or  free  school  uniforms  have  a  bigger  effect  on  teenage  pregnancy  rates?    

B.  Do  teenage  girls  have  a  right  to  have  full  informa$on  regarding  sexual  educa$on?  

C.  How  does  a  sexual  educa$on  program  affect  the  pregnancy  rates  of  teenage  girls?  

D.  Are  teachers  implemen$ng  a  sexual  educa$on  program  as  designed?  

WHAT IS EVALUATION? Impact Evaluation Course Cape Town, South Africa 18 January 2016