session 4 logic models and indicators

Chris Nicoletti

Activity #267: Analysing the socio-economic

impact of the Water Hibah on beneficiary

households and communities (Stage 1)

Impact Evaluation Training Curriculum

Session 4

April 19, 2013

MEASURING RESULTS

From Promises into Evidence IMPACT EVALUATION

AND

This material constitutes supporting material for the "Impact Evaluation in Practice" book. This additional material is made freely but please acknowledge its use as follows: Gertler, P. J.; Martinez, S., Premand, P., Rawlings, L. B. and Christel M. J. Vermeersch, 2010, Impact Evaluation in Practice: Ancillary Material, The World Bank, Washington DC (www.worldbank.org/ieinpractice). The content of this presentation reflects the views of the authors and not necessarily those of the World Bank.

Some of the data collection management material was developed by Adam Ross

3

Tuesday - Session 1

INTRODUCTION AND OVERVIEW

1) Introduction

2) Why is evaluation valuable?

3) What makes a good evaluation?

4) How to implement an evaluation?

Wednesday - Session 2

EVALUATION DESIGN

5) Causal Inference

6) Choosing your IE method/design

7) Impact Evaluation Toolbox

Thursday - Session 3

SAMPLE DESIGN AND DATA COLLECTION

9) Sample Designs

10) Types of Error and Biases

11) Data Collection Plans

12) Data Collection Management

Friday - Session 4

INDICATORS & QUESTIONNAIRE DESIGN

1) Results chain/logic models

2) SMART indicators

3) Questionnaire Design

Outline: topics being covered

4

Using a Results Chain

What are the intended results of the program?

How will we achieve the intended results?

How will we know we have achieved the intended results?

A Results Chain answers 3 questions:

Impact Evaluation Training Curriculum - Activity 267

5

The Results Chain in a Typical Program

o Budget

o Staffing

o Training

o Studies

o Construction

o Training plan

completed

o Cash transfer

delivered

o Road

constructed

o School built

o New practices

adopted

o Use of the road

o School attendance

up

o Health service use

up

o Poverty reduced

o Income inequality

reduced

o Labor productivity

increased

Results-based management

Implementation Results

Financial,

human, and

other resources

mobilized to

support

activities.

Actions taken or

work performed

to convert inputs

into specific

outputs.

Project

deliverables

within the control

of implementing

agency

SUPPLY SIDE.

Use of outputs by

beneficiaries and

stakeholders outside

the control of

implementing agency

DEMAND SIDE.

Changes in

outcomes that have

multiple drivers.

INPUTS ACTIVITIES OUTPUTS OUTCOMES LONGER-TERM

OUTCOMES HIGHER ORDER

GOALS

6

Example 1: Results Chain

Activities Outputs Outcomes Longer-term

Outcomes

Education

o Teacher

training

o Textbooks

developed

o Teachers

trained in new

methods

o Textbooks

delivered

o New methods

used

o Increased

completion

rates

o Increased test

scores

o Increased labor

productivity

Health

o Doctors

hired

o Birth

attendants

trained

o New doctors

practicing

o Attendants

applying

methods

o Increased use

of health

clinics for

deliveries

o Improved

maternal

mortality

Social

Protection

and Labor

o CCTs

delivered

o Targeting

system

o MIS

o CCTs delivered

to target

households in

accordance

with conditions

o Increased food

consumption

o Increased

child health

visits

o Decreased

poverty

o Lower child

mortality

7

Identify the sequence of inputs, activities, outputs

and outcomes:


1

2

3

4

5

6

Information is available for parents about the

importance of breast feeding.

Children in community healthier.

Fewer children are having diarrheal diseases.

Mothers breast feeding rather than using formula.

New funds available to implement a health project

to reduce child malnutrition rates.

Design information campaigns on the importance

of breast feeding

8

Identify the sequence of inputs, activities, outputs

and outcomes:


1

2

3

4

5

6

Information is available for parents about the

importance of breast feeding.

Children in community healthier.

Fewer children are having diarrheal diseases.

Mothers breast feeding rather than using formula.

New funds available to implement a health project

to reduce child malnutrition rates.

Design information campaigns on the importance

of breast feeding

Input

Activity

Output

Outcome

Outcome

Outcome

Additional Land

Access to Irrigation

Anchor Farmer

Cost of Inputs

# of Crops Grown

Value of Crop Mix

Access to Markets

Access to Additional Inputs

Irrigation and Farm-management

training

Contract Farming and market

opportunities

Efficiency of Irrigation and

farming practices

Anchor Farmer Employment

Household Labor Employment

Farm Production and Output

On-Farm Income

Off Farm Income Farm Gate Prices

Primary Inputs Secondary Inputs Intermediate Impacts Outcomes

Key Solid Line indicates positive relationship

Dashed Line indicates negative relationship

Orange Line indicates that the relationship assumes significant anchor farmer Involvement

Example #3: Irrigation Activity – Logic Model

11

• Take moment to look at the Logic models that are

in the M&E plan to identify the parts that are

testable. Would you please identify the ones that

are testable, using impact evaluation?

• Remember, these are not processes and or monitoring

efforts.

• You are interested in the causal impact of an IndII

program on ________.

Activity #2 : Review the logic models

from the M&E Plan


12

SMART: Identifying good indicators

S pecific

easurable

ttributable

argeted

ealistic

M

A

R

T

13

S pecific

easurable M

Measure as closely as possible what you want to know.

Outcome: Children treated for malaria

Indicators:

1. Increased utilization of clinics

2. Increased use of malaria drugs

Be clear about how it will be measured – specific.

Indicators:

1. % of health centers without stocks of drugs x, y & z for

more than a week at a time

2. % of health centers with availability of drugs

Source: Kathouri and Kusek, 2006

14

ealistic R Data obtainable at reasonable cost, frequency and accuracy.

Indicators:

1. HIV prevalence among 15-24 year-old

pregnant women

2. HIV prevalence among the total population


ttributable A Logically and closely linked to a program’s efforts.

Indicators:

1. Life expectancy

2. % of children fully immunized at 1 year

15

argeted T Specific to the program’s target group.

Indicators:

1. % increase in employment

2. % increase in employment of graduates of

technical training center X, in the first year

after completion of training


17

• Are the components of the logic model that are

testable using SMART indicators? How might they be

refined for the purposes of impact evaluation?

• Focus on the testable part of the logic model that you

identified earlier.

• Think about the actual complexities of how you would

accurately and reliably measure your outcome of interest.

• Be creative and think about your own life… !

Activity #2 : Come up with a set of

SMART indicators for your project


18

• What do you want to know?

• Tailor your survey to capture outcome indicators of

interest Those that are suitable for impact

evaluation.

• Take the time, before you start developing your

questionnaire, to list EVERY indicator you are

interested in measuring, and every theoretical

characteristic that could affect those indicators. – This should come from your logic model!

• Know how you will do your analysis!

• Don’t figure out how to do your analysis after the

fact… start early and have a plan in place.

Before you begin working on the questionnaire…

19

• Use reliable and valid instruments.

• If you are going to have surgery on your knee do you

want the doctor to use this… or this…?

Use valid instruments…

20

• Often times, we will use existing survey

instruments that we know worked well.

• We will modify them for the cultural context that we are

working in.

• Adapt the instrument to the local language.

• A good question is one where the respondents

have the necessary knowledge to answer

• A good question is one where the respondent is

willing to provide the “true” answer.

• This is can be difficult with health- or income-related

questions.

Building of reliable instruments.

21

• Ask about first hand experience

• Ask one question at a time.

• Bad: Are you physically able to do things like walk or

carry a full water bucket without difficulty?

• Better: Are you physically able to carry a full water

bucket without difficulty?

• Obtain specific information, but do it simply

(example: total expenditures)

• Bad: How much do you spend per month on water?

• Better: How much do you spend on a single container

of water? How many containers of water do you buy

per week?

Developing a survey…

22

• Use open questions for frequencies of

undesirable behavior.

• Use familiar words important for health

questions.

• For socially-undesirable questions ask about the

past, before asking about the present.

• 1) Ask “have you ever done ______?”

• 2) “Are you currently doing ______?”

• Put sensitive questions in between less sensitive

questions, so the respondent does not get

overwhelmed.

Sensitive questions…

23

• There are 4 types of questions

• Numeric

• Pre-coded – Read the response options to the respondent

• Field-coded – Do not read the response options to the respondent

• Verbatim – Completely open-ended

TYPES OF QUESTIONS YOU CAN USE…

24

TYPES OF QUESTIONS

Example: On average How many days per week does [NAME] collect water for

drinking, cooking or bathing currently?

|__|

Example 2: What water source do you use most often for drinking?

1. WATER VENDOR (CART, WATER TRUCK, ETC.)

2. REFILL KIOSK

3. PUBLIC/COMMUNITY TAP

Example: What is your family name?

_________________________________________

What is each type of question?

Numeric

Pre-Coded

Field-Coded

Verbatim

E1.2. Does the plot belong to your HH or has it been rented / borrowed?

1 = belongs to HH

2 = rented

3 = borrowed

25

• Lowercase vs. Uppercase

QUESTION CONVENTIONS

What treatments did you receive?

SEPARATE MULTIPLE RESPONSES WITH COMMAS

SEE CODES

Example of Field-Coded question:

How many days per week would [NAME] collect water for drinking,

cooking or bathing on average in January 2010?

-88 = DO NOT KNOW

-99 = NO RESPONSE

Example of Pre-Coded question:

What symptoms did [NAME] experience?

1 Abdominal Pain

2 Frequent Evacuation

3 Watery Feces

4 Bloody Feces

5 Vomiting

-99 NO RESPONSE

READ THE

LOWERCASE

LETTERS

DO NOT

READ THE

UPPERCASE

LETTERS

26

• Parentheses

• Square brackets

• Underlined words

QUESTION CONVENTIONS

Example: How old was [NAME] on (his/her) last birthday?

Example:

Is [NAME] responsible for collecting water for drinking, cooking or

bathing from a source outside of the home currently?

Example:

How many days per week would [NAME] collect water for drinking,

cooking or bathing on average in January 2010?

27

• We use skip instructions to filter out irrelevant questions

• Skip patterns must be followed correctly

• Asking questions in a table

SKIP INSTRUCTIONS

Member ID

BASIC DEMOGRAPHICS OF HOUSEHOLD MEMBERS

B3-A B3-B B3-C B3-D B3-E B3-F B3-G B3-H

Please tell me

the name of

each of the

members of this

household,

starting with

yourself.

After yourself,

please list oldest

to youngest.

What is

[HOUSEH

OLD

MEMBER

S]’s sex?

1 Male

2 Female

What is

[NAME]’s

relations

hip to the

head of

househol

d?

SEE

CODES

How old is

[NAME]?

Years

-99 = No

Response

(IF B3-D>13

years)

Was [NAME]

employed in

January 2010?

1=Yes

2=No B3-G

-88 = Do not

know

-99 = No

Response

(IF B3-D >13

years)

What type of

employment

did [NAME]

Have in January

2010?

SEE CODES

(IF B3-D>11

years)

Is [NAME]

employed

currently?

1=Yes

2=No B3-I

-88 = Do not

know

-99 = No

Response

(IF B3-D >11

years)

What type of

employment

does [NAME]

Have

currently?

SEE CODES

01 |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_|

02 |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_|

D7-A Does your household have a piped water connection from the PDAM?

1 YES

2 NO SKIP TO D7-B

|_|

28

• It also helps to be aware of other impact

evaluations going on in your sector. You can

learn from their results when writing your survey.

• At NORC we have people whose entire career is

writing reliable questionnaires.

• There are a lot of biases that come from poor

questionnaires.

• Instrument bias – when the instrument you are using

does not capture the information you intend to capture.

This takes a lot of practice…

TEST YOUR QUESTIONNAIRE!

It saves you from having dodgy data!

30

• Evaluate your questions – including the coding

that you use.

• Focus groups

• Cognitive Testing

• Think-aloud interviews

• Evaluate your questionnaire and procedures –

especially the timing of the survey and data flows.

• Conduct full field pretest.

• Revise questionnaire and procedures

Questionnaire testing

32

• Using one of the SMART indicators that you identified

previously, I would like for us to come up with a series

of questions to capture the information necessary for

this indicator.

• Think about question format – categorical, continuous,

verbatim, etc.

• Write out the questions and make sure they capture the

information you Intend. Are they confusing? Are they

ambiguous?

• Is it a complex indicator? If so, how will you break it into

pieces so the respondent can answer more easily? – Think about your own ability to answer these questions.

Activity #3 : Questionnaire

development


33

Tuesday - Session 1

INTRODUCTION AND OVERVIEW

1) Introduction

2) Why is evaluation valuable?

3) What makes a good evaluation?

4) How to implement an evaluation?

Wednesday - Session 2

EVALUATION DESIGN

5) Causal Inference

6) Choosing your IE method/design

7) Impact Evaluation Toolbox

Thursday - Session 3

SAMPLE DESIGN AND DATA COLLECTION

9) Sample Designs

10) Types of Error and Biases

11) Data Collection Plans

12) Data Collection Management

Friday - Session 4

INDICATORS & QUESTIONNAIRE DESIGN

1) Results chain/logic models

2) SMART indicators

3) Questionnaire Design

Outline: topics being covered

Thank You!

MEASURING RESULTS

From Promises into Evidence IMPACT EVALUATION

AND

This material constitutes supporting material for the "Impact Evaluation in Practice" book. This additional material is made freely but please acknowledge its use as follows: Gertler, P. J.; Martinez, S., Premand, P., Rawlings, L. B. and Christel M. J. Vermeersch, 2010, Impact Evaluation in Practice: Ancillary Material, The World Bank, Washington DC (www.worldbank.org/ieinpractice). The content of this presentation reflects the views of the authors and not necessarily those of the World Bank.

Some of the data collection management material was developed by Adam Ross

session 4 logic models and indicators

Documents