evaluation for m&e and program managers

R&M Module 3

Evaluation for M&E and Program Managers

Introduction

To the Participants/ Expectations:

• Your name, organization, and position?• Why did you sign up for this workshop?• How do you hope this course will help you in

your job?• What areas of evaluation are you most

interested in learning about?

Course Objective

To give M&E and program managers the background needed to effectively manage

evaluations.

Learning Outcomes– To understand the principles and importance of

evaluation.– Learn basic evaluation design and methods for data

collection commonly used in field-based evaluation.– Learn key considerations in planning and managing

evaluations in the field.– Construct a learning agenda and evaluation plan for your

grant.– Learn key steps in ensuring effective communication and

utilization of evaluation findings.– Develop a terms of reference/evaluation plan.

Course OutlineDay TopicDay 1 Chapter 1: What is Evaluation?

Chapter 2: Evaluation Purpose and QuestionsDay 2 Chapter 3 : Overview of Evaluation Design and MethodsDay 3 Chapter 4: Data Sources and Collection Methods

Chapter 5: SamplingDay 4 Chapter 6: Basic Data Analysis

Chapter 7: Using and Communicating the ResultsDay 5 Chapter 8: Managing the Evaluation

Presentations of TOR/Evaluation Plans

Learning for Pact and Learning Principles

• Solicit daily feedback for ongoing feedback• Asking is learning! Open environment for

asking questions throughout the presentation.• Let us know if there are

areas/items/considerations missing• Learning by doing• Peer learning through development of TOR

(and presentations on the last day)

What is Evaluation?Session 1

Session Learning Objectives:

• Define evaluation• Explain the difference between monitoring,

evaluation, and research• Describe why evaluations are conducted • Describe different types of evaluations• Describe common barriers to evaluations

Defining EvaluationSession 1.1

What is Evaluation?

What is Evaluation?

“The systematic collection of information about the activities, characteristics, and results of programs to make judgments about the program, improve or further develop program effectiveness, inform decisions about future programming, and/or increase understanding.”

-Michael Patton (1997, 23)

What is Evaluation?

“The systematic and objective assessment of an ongoing or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact, and sustainability.”

-Organization for Economic Co-operation and Development (OECD; 2002, 21–22)

What is Evaluation?

“The systematic collection and analysis of information about the characteristics and outcomes of programs and projects as a basis for judgments, to improve effectiveness, and/or inform decisions about programming.”

-US Agency for International Development (2011, 2)

What is Evaluation?

• Systematic- grounded in a system, method, or plan

• Specific- focused on a project/program

• Versatile- ask many different types of questions

• Utility- used to inform current and future programming

Evaluation is Action-Oriented

Evaluation seeks to answer a range of questions that might lead to adjustment of project activities, including:

– Is the program addressing a real problem, and is that problem the right one to address?

– Is the intervention appropriate?

– Are additional interventions necessary to achieve the objectives?

– Is the intervention being implemented as planned?

– Is the intervention effective and resulting in the desired change at a reasonable cost?

Evaluation Versus ResearchEvaluation Versus Monitoring

Session 1.2

Evaluation vs Research

• What is an example of research? What is an example of an evaluation?

• What are main differences between evaluation and research?

Evaluation vs ResearchFACTOR RESEARCH EVALUATION

Purpose To add to knowledge in the field, develop laws and theories

To make judgments, provide information for decision making

Who sets the agenda or focus?

Researchers, academic institutions

Stakeholders and evaluator jointly

Generalizability of results

Important to add to theory/contribute to field

Less important; focus is on particulars of the program or policy and the context

Intended use of results

Usually for publication and knowledge sharing

Usually will directly affect the project’s activities or decisions of stakeholders in development of future projects

(Fitzpatrick, Sanders, and Worthen, 2011).

Monitoring vs Evaluation

• What are some key differences?

Differences between Monitoring and EvaluationEvaluation Monitoring

Subject Usually focused on strategic aspects

Addresses operational management issues

Character Flexible subject and methods Systematic

Frequency Periodic Continuous

Primary client Stakeholders and external audience

Program management

Party conducting Can be external or internal Internal

Methodology Rigorous research methods, sophisticated tools

Rapid appraisal methods

Primary focus Focus on relevance, outcomes, impact and sustainability

Focus on operational efficiency

Objectives To determine outcomes or impact, verify developmental hypothesis, and document successes and lessons

To identify and resolve implementation problems and assess progress towards objectives

(adapted from Jaszczolt, Potkański, and Alwasiak 2003)

Why Evaluate?Session 1.3

Exercise: Why Evaluate?

• Take a few minutes to reflect on what evaluation means to your organization based on your experience.

• What would you consider as the key reasons why your organization should invest in undertaking evaluation?

• Write down three points you might say to someone else to explain why evaluation is important to your project.

Why Evaluate?

Common reasons to evaluate are:

– To measure a program’s value or benefits– To get recommendations to improve a project– To improve understanding of a project– To inform decisions about future projects

Types of EvaluationsSession 1.4

Types of Evaluation

Once you determine your evaluation purpose, you will want to consider what type of evaluation will fulfill the purpose

Broad types are:– Formative evaluation– Summative evaluation– Process evaluation– Outcome evaluation– Impact evaluation

Formative Evaluations

• Undertaken during program design or early in the implementation phase to assess whether planned activities are appropriate.

• Examine whether the assumed ‘operational logic’ corresponds with the actual operations and what immediate consequences implementation may produce.

• Sub-types include: needs assessments, contextual scans, feasibility assessments, and baseline assessments

Summative Evaluations

• Final assessments done at the end of a project. • Results obtained help in making decisions about

continuation or termination or whether there would be value in scaling up or replicating the program.

• A summative evaluation determines the extent to which anticipated outcomes were produced.

• This kind of evaluation focuses on the program‘s effectiveness, assessing the results of the program.

Process Evaluations

• Examines whether a program has been implemented as intended—whether activities are taking place, whom they reach, who is conducting them, and whether inputs have been sufficient.

• Also is referred to “implementation evaluation.”

Formative and Summative Evaluations

"When the cook tastes the soup, that's formative; when the guests taste the soup, that's summative"

Bob Stake, quoted in Scriven, 1991

http://www.cerlim.ac.uk/projects/efx/toolkit/links.html

http://www.cerlim.ac.uk/projects/efx/toolkit/links.html

Outcome Evaluations

• An outcome evaluation examines a project’s short-term, intermediate, and long-term outcomes.

• While process evaluation may examine the number of people receiving services and the quality of those services, outcome evaluation measures the changes that may have resulted in people’s attitudes, beliefs, behaviors, and health outcomes.

• Outcome evaluation may also study changes in the environment, such as policy and regulatory changes.

Impact Evaluations

• Assess the long term effect of the project on its end goals, i.e. disease prevalence, resilience, or stability.

• The most rigorous types of outcome evaluations. Use statistical methods and comparison groups to attribute change to a particular project or intervention.

Linking Types of Evaluations to the Program Logic Model

Baseline/Formative

Midterm Endline Summative

Impact

Process Evaluation

Outcome

Inputs Activities Output Outcome Impact

Exercise: Choosing Evaluation Type

Identify which evaluation type is the most appropriate in each situation:

– You are interested in knowing whether the project is being implemented on budget and according the workplan.

– You are interested in knowing what the effect of the project has been.

– You want to determine whether to scale up project activities.

– You are about to begin project activities, but want to determine whether the proposed activities fit with the current context.

Internal and External EvaluationsWho conducts the evaluation?

– Internal evaluations are conducted by people within the implementing organization.

– External evaluations are conducted by people not affiliated with the implementing organization (although conducted closely with project staff).

– Hybrid evaluations are conducted with representatives from both

External Internal HybridMay be more objective May be more efficient or

nuanced due to internal understanding

May be useful in sensitive situations

May have more expertise in evaluation methods

May be more directly suited to project needs

May serve as a learning opportunity

What else?

Barriers to EvaluationsSession 1.5

Exercise: Barriers to EvaluationMany programs are not properly evaluated, and as a result it is very difficult to duplicate or scale up the interventions that are most effective.

– What are some of the barriers that prevent evaluation of programs?

– To what extent do you believe that program implementers are open to evaluating their programs and what are some of the underlying reasons for this?

– Write down five common barriers to evaluation.

A Few Barriers:• Lack of time, knowledge and skills• Lack of resources or budget• Poor design or planning• Startup activities compete with baselines• Too complex evaluation design• Fear of negative findings• Resistance to M&E as police work• Resistance to using resources for M&E• Perception that evaluations are not useful• Disinterest in mid or endline evaluations if no baseline• What else?

Evaluation Purpose and QuestionsSession 2

Session Learning Objectives:• Use logic models to explain program theory.• Write an evaluation purpose statement.• Develop evaluation questions

To begin to build terms of reference, participants will:• Describe the program using a logic model (TOR I-B and II-C). • Complete a stakeholder analysis (TOR II-A).• Write an evaluation purpose statement (TOR II-B).• Develop evaluation questions (TOR III).

Terms of Reference (TOR)• Comprehensive, written plan for the evaluation.• Articulates the evaluation’s specific purposes, the design and

data collection needs, the resources available, the roles and responsibilities of different evaluation team members, the timelines, and other fundamental aspects of the evaluation.

• Facilitates clear communication of evaluation plans to other people.

• If the evaluation will be external, the TOR helps communicate expectations to and then managing the consultant(s). A TOR template can be found in Appendix 1 (page 89).

Evaluation Plan/TOR Roadmap• Background of the evaluation• Brief description of the program• Purpose of the evaluation• Evaluation questions• Evaluation methodology• Evaluation team• Schedule and logistics• Reporting and dissemination plan• Budget• Timeline • Ethical considerations

Focusing the EvaluationSession 2.1

Focusing the Evaluation• Focusing an evaluation means determining what it’s major purpose

is.• Why do we have to do this?

– Usually a broad range of interests and expectations among stakeholders

– Evaluation resources and budgets are usually limited, so not everything can be evaluated

– Evaluations must focus on generating useful information; plenty of interesting information is not particularly useful

– In an evaluation report, the results must come across clearly so they can be actionable

• What may happen if we don’t do this?

Key Steps in Focusing the Evaluation

• Use a logic model to understand and document the program logic.

• Document key assumptions underlying the program logic.

• Engage stakeholders to determine what they need to know from the evaluation.

• Write a purpose statement for the evaluation.• Develop a realistic set of questions that will be

answered by the evaluation.

Logic Model

• Visually describe the program’s hypothesis of how project activities will create impact.

• Useful in distilling the program logic into its key components and relationships.

• Results frameworks, logframes, theories of change, and conceptual models, also facilitate visualization of program logic.

• Start with describing the project’s long term goals, and work towards the left.

Logic Model

• Program logic—also called the program theory—is the reasoning underlying the program design.

• The logic can often be expressed with if–then statements

• By making explicit the assumptions behind a program, it becomes easier to develop good evaluation questions.

Logic Model

• If we give people bednets, then they will use them over their beds.

Or • If we educate 60% of adults about mosquito

prevention, then the mosquito population will decline.

Logic Model

INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACTS

Key Assumptions

• Every program has assumptions• Use logic model can help to make those clear. • For example: a logic model could show an

expected output of administering vaccinations to 2,000 children and an outcome of fewer children getting sick as a result. What are the underlying assumptions here?

TOR Exercise: Program Background

• If you are working with a group, please break out into your group.

• Describe the program using a logic model using the TOR I-B and II-C template.

• Spend approximately 30 minutes completing this activity.

• Be prepared to share with the larger group.

Engaging Stakeholders in EvaluationSession 2.2

Stakeholder Groups

• Who are some common stakeholders?– People involved in program operations (eg. staff,

partners, funders)– People served by or affected by the program (eg.

clients, community members, officials)– People who intent to use the evaluation results

(eg. staff, funders, general public)

Participatory Evaluation

• What might be some advantages to involving program clients/beneficiaries in key evaluation processes?– Better understanding of client perspectives– Organizations more accountable to beneficiaries– Encourage an trust and transparency– Cultivates evaluative thinking and ongoing learning– Help clarify indicators and stimulate innovative ways of

measurement– May lead to more participatory decision making

The Personal Factor

Research has demonstrated the value of the personal factor, and a key component to successful use of evaluations is:

“the presence of an identifiable individual or group of people who personally care about the evaluation and the finding it generates. Where such a person or group was present, evaluations were used; where the personal factor was absent, there was a correspondingly marked absence of evaluation impact” (Patton 1997, 44).

How to involve stakeholders

• What are some ways to meaningfully engage stakeholders?

Common Pitfalls to Involving Stakeholders

• Making yourself (or evaluator) the primary stakeholder

• Identifying vague, passive audiences• Automatically assuming the funder is the

primary stakeholder• Waiting until the evaluation is finished to

identify its users

Exercise: Engaging Stakeholders• Conducting a stakeholder analysis is the optimal beginning.• To begin developing a stakeholder analysis, ask:

– Identify the different stakeholders mentioned in the case study– Who among these stakeholders should be involved in the

program evaluation?– Consider reasons why the stakeholders you listed should be

involved in the evaluation.– How might they use or be affected by the evaluation’s results?– What would be their role in the evaluation?– Complete the template provided and prepare to provide

feedback in plenary.

Exercise: Engaging Stakeholders

STAKEHOLDERS

WHO AMONG THESE STAKEHOLDERS SHOULD BE INVOLVED IN THE PROGRAM

EVALUATION?HOW MIGHT EVALUATION

RESULTS AFFECT OR BE USED BY

THE STAKEHOLDER?

WHAT WOULD BE

THE STAKEHOLDER’S ROLE

IN THE EVALUATIO

N?

Should be involved(Yes / No)

Reasons for the listed stakeholder to be

involved

Stakeholder Analysis Matrix

• The template used in is a common type of stakeholder analysis matrix. It:– Identifies key stakeholders, – Assesses their interests,– And considers how those interests affect the

project.• Considering stakeholders and their interests

systematically helps ensure that evaluation results are useful and used.

Stakeholder Analysis Matrix

Power/Interest Matrices classify stakeholders based on the power they hold and how likely they are to be interested in the evaluation results.

The Involved

High Interest/Low Power

The Crowd

Low Interest/Low Power

The Players

High Interest/High Power

The Context Setters

Low Interest/High Power

TOR Exercise: Stakeholder Analysis• If you are working with a group, please break out into

your group.• Using the template provided in the TOR (II-A), complete

the following tasks for your program:– Identify the different stakeholders– Identify what they want to know– Consider why the evaluation is important for your stakeholders– Identify how they will be involved in the evaluation

• Spend approximately 45 minutes completing this activity.• Be prepared to share with the larger group.

Writing an Evaluation Purpose Statement

Session 2.3

Evaluation Purposes

• Should be developed after program logic clear and stakeholders are engaged.

• A clear and well-written purpose statement is important in clarifying the aim that the statement

• Key questions to be addressed in the purpose statement are:– What will be evaluated?– Why are we conducting the evaluation?– How will the findings from the evaluation be used?

Evaluation Purposes

• Another way to write the purpose statement is to complete the blanks in the following sentence:

“We are conducting an evaluation of ___________________(name of program) to find out ______________________ and will use that information in order to _____________________________________.”

TOR Exercise: Evaluation Purposes


• Develop an evaluation purpose statement for your evaluation using TOR template (I-C).



Evaluation QuestionsSession 2.4

Evaluation Questions Versus Purpose

• What is the difference between the evaluation purpose and evaluation questions?

• Questions will fulfill the purpose• Further articulate what specifically the

evaluation will answer.• Evaluation questions are different from a

question in a survey/instrument. Evaluation questions are higher level, seeking to answer broader questions about the project.

Steps to Develop Evaluation Questions

1. Review the original program goals and objectives. 2. Ensure that the evaluation is relevant, be sure you know what is

important to the organization and to other stakeholders who might use the evaluation—their priorities and needs.

3. Consider the timing of the evaluation. Some questions are best asked at the beginning of the program, others at the end of the program

4. Develop a list of potential evaluation questions. Decide which questions are most important.

5. Focus on the questions for which you need answers, not those on questions whose answers would be nice to know. Keep this list short (3-6) in order to ensure that the evaluation remains focused on the most important issues.

6. Review that the questions should be answerable and realistic given the resources available.

Choosing Evaluation Questions

There may be more evaluation questions that fall under the evaluation focus than can reasonably be answered. Consider the following when prioritizing:– Can the question be answered based on available or

attainable data?– Is the question relevant to the project and evaluation

focus?– What stakeholder cares about this question?– What resources would answering this question entail?

Types of Evaluation QuestionsSession 2.5

Categories of Evaluation Questions

Descriptive

Normative

Cause/effect

Descriptive Questions

Descriptive questions focus on “what is” and provide a means to understand the present situation regarding processes, participants, stakeholder views, or environmental questions.

Descriptive Questions

Characteristics of descriptive questions:• Have answers that provide insight into what is

happening with program activities and implementation.

• Are straightforward, asking about who, what, where, when, and how.

• Can be used to describe inputs, activities, and outputs.• May include gathering opinions or perceptions of

clients or key stakeholders.

Examples: Descriptive Questions

• What did participants learn from the program?

• Who benefited most (or least) from the program?

• How did the environment change during the years the program was implemented?

Normative Questions

Normative questions compare program achievement with an established benchmark such as national standards or project targets.

Examples: Normative Questions

• Did we spend as much as we had budgeted?• Did we reach our goal of admitting 5,000

students per year?• Did we vaccinate 80% of children as planned?• Did we meet our objective of draining 100,000

hectares of land?

Cause-Effect Questions

• Cause-effect questions intend to determine if change was achieved because of project activities.

• These questions require that the program (not something else) caused any change observed, so other causes have to be eliminated.

Examples: Cause-Effect Questions

• Did the women’s empowerment program increase the income of female-headed households?

• Did malnutrition rate drop substantially (by at least 20%) among orphaned and vulnerable children targeted by the nutrition program?

• Did increased knowledge and skills in water harvesting techniques result in increased crop yield and income for the subsistence farmers?

Mixing Questions

Depending on the evaluation purpose, a mix of different types of questions may be most appropriate for the evaluation.

Exercise: Brainstorming & Prioritizing Evaluation Questions

Think about your own program and following your evaluation purpose, complete the following:• Brainstorm and identify a few key evaluation questions that

could potentially be relevant to your program• Based on your work on your organization’s evaluation purpose

statement and reflecting on your organization’s context, identify key potentially relevant evaluation questions.

• Prioritize the questions you have identified use the Prioritizing Evaluation Questions template

• This exercise will take approximately 30-45 minutes to complete.

Exercise: Prioritizing Evaluation Questions

Evaluation Question

Can this question

be answered given the program?

Which Stakeholder cares

about this?

How important

is this?

Does this involve

new data collection?

Can it be answered given your time and resources

?

Priority: High,

Medium, Low,

Eliminate

If you are working with a group, please break out into your group.• Spend approximately 30 minutes completing this activity.• Be prepared to share your statement with the group.

TOR Exercise: Developing Evaluation Questions


• Based on the previous exercise, further develop your evaluation questions (and complete the accompanying information) using TOR III.


• Be prepared to share with the group.

Evaluation Design and TypesSession 3.1

Session Learning Objectives:• Compare and contrast qualitative, quantitative, and mixed

approaches• Identify common methods used in evaluations.• Match the best method with different evaluation questions• Identify ways to avoid common pitfalls in data collection

To continue building your TOR:

• Complete the TOR Evaluation Design and Approach (TOR IV-A).

What is the Evaluation Design?

• Plan for answering the key evaluation questions.• Begin as soon as program planning begins. • The evaluation design process should involve key

stakeholders.• Encompasses a strategy to evaluate the project– Baseline– Mid-term– End-line– Types of evaluation

• Evaluation plan/strategy

Evaluation Design Specifies:

• Which people or units will be studies• How they will be selected• The kinds of comparisons that should be made• By what approach the comparisons will be

made• The timing of the evaluation• At what intervals the groups will be studied

USAID’s Evaluation Policy

“Recommitment to learn as we do, updating evaluation standards and practices to address contemporary needs”

Why evaluate?– For accountability – inform resource allocation– To learn – inform and improve project design and

implementation

USAID’s Evaluation Practices

• Integrated into the design of each project –• Relevant to future decisions• Based on best methods – counterfactual/control• Reinforcement of local capacity• Commitment to transparency• Dedicated resources• Minimized bias

Key Concepts

• Control group – counterfactual

• Bias

Control groups a few examples …

• Omega 3 fatty acids• One year project to increase learning among

school children aged 8 to 10• Measuring IQ at the beginning and at the end

of the project.

http://www.informationisbeautiful.net/visualizations/snake-oil-superfoods/

Parachute use to prevent death and trauma related to gravitational challenge: a systematic review of RCTs

• Systematic review of literature

• Main outcome measure:– Death or major trauma

Bias?

• What is bias?• Different types of bias• How can we avoid bias?

*Evaluation design can help avoid bias*

Types of Evaluation Design

• Quantitative– Experimental– Quasi-experimental– Non-experimental

• Qualitative• Mixed Methods

Experimental Design• Counterfactual – Control group

– Studies units both receiving and not receiving the intervention

– Non-treatment is a counterfactual, or picture of what probably would have happened to the treatment group if they had not received the treatment

• Random Intervention allocation– Intervention is randomly assigned, which means it is unlikely

there are systematic differences (bias) between the groups• Take measurements before and after intervention for both

groups• Enables attribution of change to the treatment

Experimental Design Challenges• Commonly used in drug trials or testing new types of treatment;

only sometimes used in social science research or evaluation• Randomization of assignment is usually not done during program

implementation; intervention communities are targeted because they are either easiest to work with or most in need

• Ethical considerations usually limit viability of this approach in field settings– Not providing program to those in need– Answering a survey or otherwise participating in data collection can be

burdensome• Can be expensive

Experimental Design• Sub-types– Pre-test and post-test control group design

– Post-test only control group design

Treatment

Control

Measurement MeasurementIntervention

MeasurementIntervention

Treatment

Control

Quasi-Experimental Design

• What is it?• What is the difference with an experimental

design?


• Compare groups that both receive and do not receive an intervention

• Groups are not randomly assigned

• Comparison groups are found that are as similar to the intervention group as possible EXCEPT that they do not receive the intervention

Quasi-Experimental Design Challenges

• How do you select your control group?• What do you have to take into account when

selecting a control group?

Quasi-Experimental Design• Commonly used in social science research• Control group:

– Adjacent communities – random selection or matching– General population

• Spill-over effects• Ethical considerations:

– What do you do about problems you identify in the comparison sites?

– How do you minimize the burden of participating in data collection?

– Sometimes projects offer comparison sites an alternate benefit.


• Select sub-types– Non-equivalent control group pre-test post-test

• Non-random comparison before/after measurement• There are a number of ways to choose non-random comparison

groups and agreed upon methods for analysing them afterwards, but it’s beyond the scope of this training to go into detail

– Generic control design• Comparison group is the general population• Requires:

– Good data on general population from same time periods– The general population to be similar on the treatment group

Threats to Quasi-Experimental Design

Consider whether the following may have influenced the comparison you are looking at:• Program design shifts after baseline and comparison group

receives treatment (or treatment group doesn’t)• Another organization begins similar work in comparison

areas• Program target beneficiaries were chosen based on specific

characteristics that can’t be replicated in a comparison (i.e. proximity to road)

• Program effects are wide-ranging, i.e. change to national policy

Non-Experimental Design

• No comparison between groups • Data are collected only from the group of

individuals that received the intervention

Non-Experimental Design

Common characteristics of this design include:• Allows documentation of status or change in

beneficiaries but not attribution because there is no counterfactual

• Can still use statistical methods to make these data more descriptive:– Measure your beneficiaries both before, during and after

intervention– Test any changes for statistical significance

• Easy to collect but not very persuasive

Interim Evaluation of Intervention

Mar Sep0

25

50

75

100

125

Number of clients linked to care

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec0

25

50

75

100

125



255075

100125



25

50

75

100

125



25

50

75

100

125


Change Made

Change Made

Change Made Change

Made

Change Made in June

Qualitative Approaches

• Qualitative evaluation approaches synthesize people’s perceptions of a situation and its meaning

• Non-numerical data include:– Open-ended interview– Focus group discussion data– Pictures– Maps– Case studies

• Qualitative approaches can still use in before/after, longitudinal, cross-sectional, or as post-test only designs

Qualitative Design

• Helpful when:– You need to know why things have happened– You don’t know exactly what results you are looking for– You want the community to be engaged in the

evaluation process– You want descriptions of change

• Not good for:– Rigorous counterfactuals– Being able to quantify change that has taken place

Quantitative vs QualitativeQualitative Approaches Quantitative Approaches

The aim is to identify common themes and patterns in how people think about and interpret something.

The aim is to classify features, count them, compare, and construct statistical models that explain what is observed as precisely as possible.

Evaluator may only know roughly in advance what he/she is looking for.

Evaluator knows clearly in advance what he/she is looking for.

Data are in the form of words, pictures, or objects.

Data are in the form of numbers.

Focuses on fewer selected cases, people, or events.

Measures a limited range of responses from more people.

Greater depth and detail is possible. Facilitates comparison and statistical aggregation for concise conclusions.

Categories for analysis must be developed and are specific to the particular evaluation.

Uses standardized measures that fit various opinions and experiences into predetermined categories, often using questions that have been verified and tested in other programs/studies.

Can ask questions about the program holistically or about specific parts of a program.

Views components of a program separately and uses data from the different pieces to describe the whole.

Mixed Approaches

• Best of both worlds• Combines qualitative and quantitative approaches• When used together, these approaches can be

stronger and give more meaningful results– Qualitative data can give rise to a quantitative strategy

once an evaluator knows what s/he is looking for– Qualitative data can give meaning to confusing

quantitative results– Quantitative data can validate qualitative articulation of

experiences

Choosing a Design

• Considerations in choosing an approach and design:– Purpose of evaluation– Time, money, and other resources– Data collected to date– Stakeholder priorities

• Often a trade-off - compromise• Can aggregate all of these into a design matrix

Design Matrix

QUESTIONS SUB-QUESTIONS

DATA COLLECTION

METHOD

DATA SOURCES

UNIT OF ANALYSIS

SAMPLING APPROACH COMMENTS

What is a good basic design?

• A good basic design maximizes the quality of the evaluation while also efficiently using resources.

• We can increase strength of the findings by minimizing threats to validity, or factors that might cause the audience to believe the evaluation is not accurate.

Threats to validity

• Threats to concept validity: measures things that aren’t relevant to the evaluation questions.

• Threats to internal validity: the data or findings are inaccurate.

• Threats to external validity: the results wouldn’t be the same elsewhere (relevant if you are interested in scaling the project up or expanding)

Good basic design

• Recognize that there are always trade offs; it will never be possible to do an experimental evaluation for every project

• Judiciously evaluate available resources to choose the approach and design that is most appropriate for your evaluation questions

TOR Exercise: Evaluation Design


• Develop a draft for TOR section IV A: Evaluation design and approach



Data Sources and Collection Methods

Session 4

Session Learning Objectives:• Compare and contrast different data collection methods.• Select practical data collection methods for a project.• Discuss ethical considerations in evaluations.

To continue to build your TOR:

• Begin filling in the design matrix, specifically columns related to data collection methods and data sources (TOR IV-B).

• Identify and describe ethical considerations connected with the evaluation (TOR X).

Data Collection

• Data for evaluation comes from a wide range of sources and can be collected through an equally wide range of techniques

• Collection methods are not necessarily quantitative or qualitative; the same method often can be adapted to collect different kinds of data

Deciding on Data Collection Methods

• Decision depends on:– What you need to know– Where the data reside– Resources and time available– Complexity of the data to be collected– Frequency of data collection

General Rules for Collecting Data• If you collect original data:

– Establish procedures, document the process and follow it– Maintain accurate records of definitions and coding of the

data– Pre-test data collection tools – Follow recommended Standard Operating Procedures for data

collection, collation, analysis and reporting (see Pact’s Data Quality Management Module for more)

• Next, we’ll look at some common methods for data collection

Existing Records• Looking at data that already

exist before analyzing it for project purposes

• Can be quantitative or qualitative

• If data were collected by a source other than the program, they are secondary data and you should ensure that the data are of good quality

• Can be very useful, but requires that data relevant to the project already exist and are accessible

Surveys• Surveys have a single questionnaire that is administered

to all respondents• Questions can be open-ended, which generates

qualitative data, or close-ended, which generates quantitative data

• Can be self-administered (cheaper) or administered by a data collector (more reliable)

• Can be a good source of aggregated and comparable data• Difficult to get the questions right (use previously

validated questions when possible)

Direct Measurements

• Possible when the outcome of interest can be directly observed. For example:– Height or weight of a person– Contents of soil or water– Presence of disease– Presence or absence of a policy

• Very reliable if done properly• Can be intrusive, time-consuming, or require

special equipment and expertise

Observation

• The evaluator directly observes activities and notes patterns, routines and other relevant data

• Useful when evaluation question can be answered this way (i.e. about service delivery)

• May be more objective than interviews with participants, but still very subjective

• Can be intrusive or influence behavior• Can be direct or participatory

Key Informant Interviews

• In-depth interviews, usually with a semi-structured open-ended questionnaire, with a handful of important stakeholders

• Helpful in getting the big picture of a project and understanding context

• Can ask some quantitative data, i.e. approximate village population, distance from road

• Which key informants are chosen can have a big influence on findings

Focus Group Discussions• Small-group (8-12 people) discussions facilitated by the

evaluator, usually using a semi-structured open-ended questionnaire

• Participants are most often somewhat homogeneous in order to encourage people to feel comfortable expressing themselves (all same gender, age range, occupation)

• If well-facilitated, can encourage discussion that yields valuable information

• Usually several FGDs representing different populations are held to provide better representation of a community

• Time consuming to analyze data properly

FGD vs KIIFactors to consider Use focus groups if Use key informants if

Group interactionInteraction may stimulate a richer and new responses

Group interaction likely to be unproductive and limited

Sensitivity of subject matter

When subject is not sensitive causing participants to withhold their opinions

When subject is sensitive and a respondent will be unwilling to discuss in a group

Depth of individual responses

Participants will be able to articulate their opinions faster to give others a chance

The topic requires greater depth of response, or subject is highly specialised

Extent of issues to be covered

Themes to be interrogated are very few

Greater volume of issues to be covered

Availability of qualified staff

Focus group facilitators need to be able to control and manage group

Interviewer need to be supportive and skilled listener.

Most Significant Change

• “Regular collection and participatory interpretation of “stories” about change” (Dart and Davies 2003)

• Beneficiaries give stories about the most significant change in their lives over a defined period of time to a panel of stakeholders, who identify the ones that best represent the project

• A larger group of stakeholders then meets to discuss the stories and their meanings

• Developed by Dart and Davieshttp://www.mande.co.uk/docs/MSCGuide.pdf

http://www.mande.co.uk/docs/MSCGuide.pdf


• What, in the last X period, was the most significant change regarding Y that happen to Z?

Most Significant ChangeMost Significant Change


• The product is not the final story, but rather the recording at each level of how/why each story was chosen; feedback is key

• Many of these stories are included in the final document

• Good method of qualitative data collection that is systematic, participatory, and open-ended

Outcome Mapping

• Illustrates a program’s theory of change in a participatory manner

• Set in motion at a workshop before program activities begin

• Best suited to learn about outcome information of complex programs

• Requires a skilled facilator

Mapping

• Mapping is the visualization of key landmarks• Can show landscape change, spark discussion,

or enable people to describe how their environments fit into their daily activities

• Can be hand-drawn or use accurate GIS data

Other Methods

• There are many, many methods.• Sites like BetterEvaluation.org can provide many

explanations and examples of how a method can be used.

• If you are unsure which method, consult a professional evaluator for advice on options.

• Generally best to suggest basic frame of acceptable methods in the terms of reference, as the evaluation will yield the required type of data.

Consider the following when choosing which method to use:

Feasibility Do you have the resources (personnel, skills, equipment, and time)?Can the method fulfill the evaluation purpose and answer the evaluation questions? What are the language/literacy requirements?

Appropriateness Does the method suit the project conditions and circumstances?Do all the stakeholders understand and agree on the method?

Validity Will it provide accurate information? Is it possible to assess the desired indicator with accuracy?

Reliability Will the method work whenever applied?Will the errors that occur be acceptable?

Consider the following when choosing which method to use:

Relevance Does the method produce information required or is it actually assessing another type of outcome?Does the method complement the basic approaches of the project, e.g., is it participatory?

Sensitive Is it sufficient to assess variations in the different population characteristics, e.g. differences between age groups or gender?Can it be adapted to changing conditions without excess loss of reliability?

Cost effective

Will the method produce useful information at relatively low cost?Is there a more cost effective alternative method?

Timeliness Does the method use staff time wisely? Will it require withdrawing staff from their usual activities, leaving project work unattended?Is there an acceptable level of delay between information collection, analysis, and use?Can the method be incorporated in other daily tasks?

Ethical Review• All study methods should undergo ethical review. Consider the

following ethical questions when thinking about your proposed design:– Is the evaluation causing undue burden on a group receiving no

program benefits if it includes a control, or raise expectations that a control group might receive services?

– Does the evaluation tool ask sensitive questions, for example regarding child abuse, insensitively or without resources to refer the respondent to appropriate resources?

– Would focus group discussions of sensitive topics result in negative consequences for the participants in their communities?

– Are children being asked questions without an adult present?– Does the evaluation protocol collect informed consent?

Exercise: Evaluation DesignRound-Robin Table Conversations• How can you have confidence that the outcomes you observe are

the result of your program and not something else?• How can you attribute the changes noticed in the intervention

population (as opposed to non-intervention population) to your program support?

• How do you ensure that the problems identified prior to implementation are the real problems a successful project needs to address?

• How would you demonstrate that the responses you obtain from your beneficiaries are a true reflection of their perceptions or characteristics?

• What are the steps you might take to establish the cause and effect of the results achieved?

TOR Exercise: Design Matrix and Ethical Considerations


• In TOR IV-B, fill in the “data collection method” and “data sources” columns.

• In TOR X, complete the section related to ethical considerations.

• Spend approximately 60 minutes to complete these two activities.


SamplingSession 4


• Identify different units of analysis in evaluations.• Compare and contrast probability and non-probability

sampling.• Identify potential biases in data collection.

To continue to build your TOR:

• Complete the design matrix, specifically columns related to unit of analysis and sampling approach (TOR IV-B).

• Describe an appropriate sampling strategy (TOR IV-C).

Sampling ApproachesSession 4.1

Sampling

Sampling is the answer to the question, “Who are we evaluating?”

145

Basic Concepts in Sampling

Unit of analysis: the person, group, place, or event of interest for the evaluation– When might you have a unit of analysis that is not

an individual?

146

Unit of Analysis ExamplesEVALUATION QUESTION UNITS OF ANALYSIS

Did the women’s empowerment program result in increased income levels among female-headed households?

Female-headed households

Did the rate of malnutrition drop substantially (by at least 20%) among orphaned and vulnerable children targeted by the nutrition program?

Orphan and vulnerable children

Did the increased knowledge and skills in water harvesting techniques increase crop yield and income for subsistence farmers?

Subsistence farmers

What impacts (positive or negative) did the intervention have on the wider community?

Community

Did the clinics that received the training implement what they learned?

Clinics

Did local governments adopt more transparent policies as a result of civil society organizations’ work?

Local governments


Study Population: the entire set of unit we are interested in

For example: – all intervention communities– all low-income people in a city – all people participating in an education program

147


Sample: a subset of a study population– Different from disaggregation– Likely, you will not collect data on every program

beneficiary for the evaluation. The ones you do collect data on are the sample.

– Good samples are representative

148

149

Sampling

There are 2 broad categories of sampling techniques:

• Probability sampling: every member of the population has an equal (or known weighted) chance of being selected for the sample

• Non-probability sampling: study participants are selected without this equal chance (some are more likely to be included in the sample than others)

Sampling Frames

• A sampling frame is necessary for true random sampling

• This is a list of the entire population of interest• If no such list exists or can be made, other

techniques must be used to approximate randomization

151

Types of Probability Sampling

• Simple random sampling• Stratified sampling• Cluster sampling• Systematic sampling

152

Simple Random Sampling (SRS)

• Every individual in the population has the same probability of being selected for the sample.

Sampling with a Non-Digital List

Example 1: Simple Random Sampling of Participants

• Take a piece of paper

• Write your Surname and initial on the piece of paper

• Then fold the piece of paper and put it in a box

• We’ll randomly select ¼ participants

Sampling with a Non-Digital List

Sampling with a Digital ListExample 2: Simple random sampling of 10 households from a list of 40 households

• We have a list of 40 heads of households. Each has a unique number, 1 through 40. We want to select 10 households randomly from this list. Using the Excel function RANDBETWEEN(x,y), we select consecutive 2-digit numbers starting from the upper left.

• If a random number matches a household's number, that household is added to the list of selected households.

• After each random number is used, it is crossed out so that it is never used again. We continue to select households until we have 10.

Challenges with Simple Random Sample

• Simple random sample is the best way – so why are we not always using it?

• Can you think of challenges with simple random sampling?

157

Possible complications of SRS

• If you don’t have a sampling frame, it can be costly to create one

• Sometimes – going to all program areas (e.g. Ethiopia) can be very expensive

• If there is a particular subset of the population of interest that is a relatively small group, (e.g. an ethnic minority) they may be left out of a random sample by chance if the sample is not very large

158

Resources to help with SRS

– http://bcs.whfreeman.com/ips4e/cat_010/applets/randomsample.html

– http://stattrek.com– http://www.socialresearchmethods.net/kb/sampp

rob.php– http://stattrek.com/Tables/Random.aspx

http://bcs.whfreeman.com/ips4e/cat_010/applets/randomsample.html

http://bcs.whfreeman.com/ips4e/cat_010/applets/randomsample.html

http://stattrek.com/

http://www.socialresearchmethods.net/kb/sampprob.php


http://stattrek.com/Tables/Random.aspx

159

Stratified Sampling

• First, divide the population of interest along a characteristic (or multiple characteristics) of interest.– Examples: gender; age group; ethnic group

• Determine how many people you want to sample from each strata (group).

• Within each strata, proceed to select individual units as you would for a simple random sample.

160

Stratified Sampling

• Assures representation of key subgroups of the population

• If sub-populations of interest are small, stratified sampling with weighted over-sampling (choosing a higher percentage of respondents from the sample than is representative of the entire population) ensures that there will be enough data from that subgroup to get precise estimates of subgroup characteristics

Example of Stratified Sampling• Let’s say we have a population of 3 ethnic groups. We are

very interested in how each of the 3 groups experienced the intervention, and making meaningful generalizations about how the intervention affected each group.

• The population breakdown is as follows:– Ethnic group 1: 85%– Ethnic group 2: 10%– Ethnic group 3: 5%

• Though groups 2 and 3 are small, they are also marginalized and may have stood to gain the most from the project.

Example of Stratified Sampling• If we sample 100 people randomly, we’re likely to have very small numbers of

people in Groups 2 and 3 (likely 10 and 5 people respectively.• We cannot make any generalizations about a population from this small of a

sample. If we want to do a simple random sample and be able to make within-group generalizations, we would have to increase the overall sample by several times.

• An alternative is to stratify the population by ethnic group and then over-sample from Groups 2 and 3.

• For example, you may choose to sample as follows:– Group 1: 50 people– Group 2: 25 people– Group 3: 25 people

• This will give more precise data for group averages. If you also need whole-population statistics, you can weight each group’s data by the group’s actual size when analysing it.

163

Pros and Cons of Stratified Sampling

• Stratified sampling is appropriate when you are very interested in data from particular subgroups and are concerned that simple random sampling will not yield those data.

• In some quasi-experimental methods, choosing samples of treatment and comparison units is a type of stratified sampling.

• Analyzing stratified data accurately requires that the percent of each subgroup in the population be known.

• As with simple random sampling, a sampling frame is necessary.

164

Examples and Illustrations of Stratified Sampling

– http://stattrek.com/Lesson6/STR.aspx– http://www.socialresearchmethods.net/kb/sampp

rob.php– http://www.marketresearchworld.net/index.php?

option=com_content&task=view&id=23&Itemid=1

http://stattrek.com/Lesson6/STR.aspx



http://www.marketresearchworld.net/index.php?option=com_content&task=view&id=23&Itemid=1

http://www.marketresearchworld.net/index.php?option=com_content&task=view&id=23&Itemid=1

If we don’t have a sampling frame?

166

Cluster Sampling

• Clusters are collections of study units that are grouped in some way.

• For example, a village is a cluster of beneficiaries; a classroom is a cluster of students.

• Cluster sampling means that the sampling happens at the Different levels. The first level is usually the cluster

• Cluster sampling can be more a more efficient way of collecting data. You can cut costs of traveling to every village by sampling villages

167

Cluster Sampling Assumptions

• Clusters are more or less the same• Study units within each cluster are

heterogeneous• Clusters do not need to be of equal size – if

that is the case : use weighting techniques to compensate for the possibility of oversampling very small clusters. (e.g. selection with probability proportional to size)

168

Cluster Sampling

There are two broad types of cluster sampling:• Single-stage cluster sampling when all units in the

selected clusters are included in the sample.– This makes sense when clustering by classroom, where it is

most efficient to test all of the students in the sampled classes.

• Multi-stage cluster sampling when a sub-sample of units in selected clusters are chosen.– This makes sense when clustering by village, where surveying

every household in a village would be time-consuming and inefficient.

169

Cluster Sampling

Cluster samples can be useful when:

• No list of the population exists• Well-defined clusters, which will often be geographic

areas, exist• It’s possible to estimate the population of each cluster• Often the total sample size and total population must

be fairly large to enable cluster sampling to be representative

Example of Cluster Sampling

List all villages/towns, with their populations; must have at least approximate idea of population

Randomly select the desired number of villages from the list

Within the village, systematically select households through a random walk method.

171

Pros and Cons of Cluster Sampling

• Complete list of clusters is more likely to exist than complete list of units

• Can be less expensive to administer surveys by cluster• Still need to have a strategy for identifying

respondents within a cluster• Need to have a relatively large population of interest

and large sample size because there is some risk that within cluster respondents will be too homogenous to adequately represent the population

172

Systematic Sampling

• Systematic sampling is different from random sampling.

• This can be done without a sampling frame.• Systematic sampling can still yield a

representative sample if done properly.

173

Systematic Sampling

• First, determine what proportion of the population you need to sample.– We will sample 10% of the population.

• Then, every nth unit that you come to is part of your sample. This is the sampling interval.– For 10%, every tenth person is part of the sample.

• The first subject should be chosen at random. Because of this, we assume that a systematic sample has the same properties as a random sample.

174

Systematic Sampling• Systematic sampling is useful when you do not have an

actual sampling frame.

• Example 1: go to every 10th house in a village along a specified path to conduct a survey.

• Example 2: poll every 10th person who leaves a voting booth

• Can be used in combination with cluster sampling or stratified sampling.

175

Pros and Cons of Systematic Sampling

• If the available data on population isn’t very good, systematic sampling can be more precise than random sampling.

• Systematic sampling will be designed differently based on context, so someone who understands the issues must be involved in design.

• Possible risk of periodicity, if for some reason the sampling interval coincides with every nth person having a special characteristic.

Non-Probability Sampling

• Non-probability sampling is non-random sampling.

• Accidental– Commonly referred to as convenience sampling.

• Purposive

Accidental Sampling• Non-probability sampling strategy that uses the most easily

accessible people to participate in a study.

• Not recommended.

• Very likely to be biased. The most easily accessible sample is likely to be similar in a confounding way.

• For example, many psychology studies use students in college psychology classes. When these studies are re-conducted with random participants, they yield very different results.

Purposive Sampling

• Non-probability sampling strategies that are deliberate

• Sampling done with some purpose in mind

• Two types:– The respondents are specifically selected to represent

“average” population– The respondents are specifically selected to represent

particular points of view, such as key informants

Purposive Sampling

• Makes sense when a population is not large and particular points of view are of interest (key informants)

• With a purposive sample, you are likely to overweight particular points of view either intentionally or due to accessibility

• Purposeful samples are NOT representative of populations

Categories of Purposive Sampling• Modal Instance Sampling: select the typical case• Expert Sampling: select people with demonstrated

expertise in the area of interest• Quota sampling: select a certain number of

respondents who fit particular characteristics• Heterogeneity: select people who are supposed to

represent different segments of the population• Snowball: get referrals of people from previous

respondents (for hard to reach populations, like illegal immigrants)

Choosing a Sampling StrategyFor probability samples, it is important to ask the following questions:• What is the unit of analysis?• Does the evaluation purpose, question, or method require a

probability sample?• What is the sampling frame?• How will we obtain the complete list of the population?• How will we ensure the list is accurate and up to date?• What sampling technique will we use and why?• What is the necessary sample size?• How was the sample size calculated?

Choosing a Sampling StrategyFor nonprobability samples, it is important to ask the following questions:• Will a nonprobability sampling technique allow us to fulfill

the evaluation purpose and question?• Are we using purposive sampling? If so, what are our

inclusion criteria?• How will we know we have reached saturation?• How will we select your participants?• How will we determine the sample size?

Sampling SizeSession 4.2

Sample Size• How many people do you need in your sample?• Representativeness: small samples are more likely to be non-

representative by chance• Comparisons: if you are comparing populations, the total

sample size will have to be larger• Differences: if you expect small changes or differences, the

sample size will have to be larger to be more precise• Use a sample size calculator to figure out how many people

you need

Sample Size

• Qualitative research– No fixed rules on sample size– Talk about “saturation”: continue interviewing until the data

seem saturated and clear patterns begin to emerge– May be informed by available resources, the topic of interest

• Quantitative Research – Sample size formulae exist– Based on the expected amount of change you expect to see,

the level of confidence you want in the results, and population size

Why Calculate Sample Size?

• Needlessly large samples may waste time, resources and money

• Small samples may lead to inaccurate results • Allows study plan and budget• Forces specification of expected results, which

helps in later analysis and program consideration of results

Sample size calculation

A number of sample size calculators exist online:• http://www.openepi.com/v37/Menu/OE_Men

u.htm• http://www.stat.ubc.ca/~rollin/stats/ssize/• http

://homepage.stat.uiowa.edu/~rlenth/Power/index.html#Download_to_run_locally

http://www.openepi.com/v37/Menu/OE_Menu.htm

http://www.openepi.com/v37/Menu/OE_Menu.htm

http://www.stat.ubc.ca/~rollin/stats/ssize/

http://www.stat.ubc.ca/~rollin/stats/ssize/

http://homepage.stat.uiowa.edu/~rlenth/Power/index.html#Download_to_run_locally



Final Remarks

• This session was mainly to help you understand matters that will be important for budgeting, workplanning, and managing consultants

• Experts will understand the most appropriate kind of sampling technique for a given project

• Be sure to appropriately document sample selection in the evaluation plan and report

Exercise: Choosing an Appropriate Sampling Approach

• Read through the case scenario “Sampling at XX Child Care Centre”

• Identify one data collection method that you would use to answer the evaluation question, “In what ways have the supervisors and managers used their learning from the workshop?”

• Recommend two sampling methods (one probability and one non-probability) and identify the advantages and disadvantages of each methods. Use the table on the handout to organize your thoughts.

TOR Exercise: Sampling


• Complete the “unit of analysis” and sampling approach columns in the Design Matrix TOR IV-B.

• Develop a draft for section IV-C: Sampling Strategy • Spend approximately 45 minutes to complete

these two activities.• Be prepared to share with the larger group.

Basic Data AnalysisSession 6


• Describe the basics of data analysis.• Prepare data for analysis.• Interpret the evaluation data and findings.

To continue building your TOR:

• Devise data analysis procedures, including a data analysis plan and dummy tables (TOR IV-D).

Data Analysis

• Data analysis is the process of turning raw data (numbers or text) into usable information.

Developing a Data Analysis Plan

• A plan should be developed before data are ever collected.

Raw data

Analysis

Useful information

Why Create a Data Analysis Plan?

• Makes sure you don’t collect unneeded data• Makes sure you do collect necessary data• Figures out the kinds of results you are interested in and

helps articulate the hypothesis• Enables other parts of data collection, such as sample

size selection• Ensures that the most appropriate methods to get the

necessary information are used• A data analysis plan should be completed at the TOR or

inception report stage.

What Does a Data Analysis Plan Include?

• The key variables for each evaluation question• The type of analysis that will be performed for

each type of variable• What kinds of comparisons will be made• How data will be presented (e.g. graphs, tables,

quotes)

Data analysis plans should be based on the evaluation questions developed at an earlier stage.

Dummy Tables

• Dummy tables are mock tables created before data analysis.

• They should be the tables we eventually want to put in the final report, but without the actual data.

• This technique enables us to better think through the key results that we will actually be showing in the report.

• This technique also helps us think through our analysis beforehand and the best way to visualize it.

Dummy Tables: Descriptive Statistics

Variable Descriptive StatisticsAge Mean

(Standard Deviation)Percent Female % FemaleEducation

No SchoolingPrimary SchoolMiddle School

College

% No schooling% Primary School% Middle School% College

Dummy Tables: Statistical Significance

Intervention Non Intervention Statistical Significance

Condom use pre-intervention

% % p=0.x

Condom use post-intervention

% % P=0.x

Dummy tables: qualitative analysis

Themes Key Informants in Town A

Key Informants in Town B

Government Corruption Hopelessness Environmental Degradation

Economic Uncertainty

Data Preparation

After the data have been collected in line with the data plan, the data must be prepared before being analyzed:

• Data Cleaning• Data Coding• Organizing data into a dataset

Data Cleaning

• Data is cleaned to ensure that “bad data” is excluded from the analysis in order to avoid drawing invalid conclusions

• Sometimes bad data is so obvious and can quickly be identified in the data set:– A six-month-old baby that weighs 50kg or – Age of a participant in a focus groups being

indicated as 2 years

Data Cleaning

• One method of cleaning data is to take a random selection of data entries and compare it to the source to check for transcription errors

• If many errors are found, a more systematic comparison is made

• Usually, data are double-entered by two different people and can be quickly checked against each other for discrepancies

Data Coding

• Coding is assigning a tag or identifier to responses • Should be done after the data is cleaned • This makes it easier to work with computer software to

undertake the analysis.• For quantitative data, numerical codes are assigned to

key variables, i.e.: 1=Male and 2=Female• For qualitative data, concepts that repeat can be

grouped together or coded with tags or numbers• All data codes should be recorded on a master codebook

to ensure accuracy later in data analysis

Datasets

• Data are organized into a dataset, which are an organized matrix of columns and rows.

• Rows: each individual respondent• Columns: Each variable• Each individual should have a unique ID

numberID Number Name Gender Province001 Vincent 1 4002 Tsakani 2 3003 Khensani 2 5

Types of Quantitative Analysis

• Descriptive measures: proportions, frequencies, rates, and ratios

• Measures of central tendency: averages• Measures of dispersion: how widely data are

spread

Descriptive MeasuresDescriptive Measures Example Proportion

Number of observations with a given characteristic divided by the total number of observations

1 out of 3 children in the study had a Vitamin A deficiency; 56% of participants completed the training

Frequency

Arrangement of values from lowest to highest with a count of the number of observations sharing each value; these counts are often converted into a percentage of the total count.

12 participants (40%) had attended school for less than 5 years, 12 participants (40%) attended school for between 5 and 8 years, and 6 participants (20%) graduated from high school.

Rate Occurrences per a certain constant over a certain time period.

The infant mortality rate is the number of deaths of infants under one year old per 1,000 live births.

Ratio Number of observations in a given group with the characteristic, divided by the number of observations in the same group without the characteristic.

81 women were married and 27 were not married. The ratio of married women to non-married women was 3:1.

Measures of Central TendencyMeasures of Central Tendency Mean The average. This is calculated by

totaling the values of all observations and dividing by the number of observations.

Participants were ages 18, 18, 20, 21, and 26. The average age of participants was 20.6.

Median The middle observation, i.e., half the observations are smaller and half are larger. This is calculated by arranging the observations from lowest to highest (or from highest to lowest), counting to the middle value, then taking the middle value for an odd number of observations and the mean of the two middle values for an even number of observations.

Participants were ages 18, 18, 20, 21, and 26. The median age of participants was 20.

Mode The value in the set that occurs most frequently.

Participants were ages 18, 18, 20, 21, and 26. The mode was 18.

Measures of DispersionMeasure of Dispersion Range The difference between the largest

observation and the smallest. This is often expressed as the largest and smallest observation rather than the difference between them.

Participants were ages 18, 18, 20, 21, and 26. The ages of participants ranged from 18 to 26.

Standard deviation

This is a measure of the spread of data around the mean, or in other words, the average of how far the numbers are from the mean. If the standard deviation is 0, then all the observations are the same.

Participants were ages 18, 18, 20, 21, and 26. The standard deviation is 2.9.

Presenting Quantitative Data

• Just embedding text in a report makes it difficult to see the big picture

• Data can be visualized in a number of ways; here, we will go through just a few of the most common

Bar Graphs

• A bar graph is used to show relationships between groups.

• The items being compared do not need to affect each other.

• Bar graphs are a good way to show big differences.

Before After0

10

20

30

40

50

60

70

80

90

100

ControlIntervention

Perc

ent

Line Graphs

• A line graph is used to show continuous data.

• It's easy to see trends by the rises and falls on a line graph.

• Line graphs are useful in depicting the course of events over time.

JanMarch May

JulySe

ptNov

0

100

200

300

400

500

600

700

800

InterventionControl

Pie Charts

• Pie charts are used to show how a part of something relates to the whole.

• This kind of graph is an effective way to show percentages.

Underweight30%

Healthy weight55%

Overweight

12%

Obese3%

Tables

• Tables can be used to present absolute numbers or percentages.

• Tables are very useful for providing detailed statistical data that may be too complex to be captured by a simple chart.

Ages 18-29

Ages 30-39

Male 24 35Female 36 32

Example of quantitative data analysis and presentation

• http://www.ted.com/talks/hans_rosling_shows_the_best_stats_you_ve_ever_seen.html

http://www.ted.com/talks/hans_rosling_shows_the_best_stats_you_ve_ever_seen.html

http://www.ted.com/talks/hans_rosling_shows_the_best_stats_you_ve_ever_seen.html

Exercise: Quantitative Data Analysis

• Form groups of four to five people• Each group should have at least one laptop• Receive spreadsheet with program data in

Microsoft Excel• Analyze data within the group• Discuss useful information and possible

conclusions

Qualitative Data

Prepare data• Converted into written form• Interviews and FGDs should be transcribed verbatim• Checked against the recording• Translation may be necessary• Documents and notes should be checked for

accuracy• Some data analysis will require software programs

such as Nvivo

Qualitative Data

Code and Analyze• Goal is to identify themes and patterns; must be

done systematically, although there are different approaches

• How is this typically done?

Qualitative Data

Common steps include:• Develop a list of expected codes (based on evaluation

questions, KII guides, etc.)• Read all of the transcripts; revise codes• Develop a codebook– ensure agreement between coders• Read the transcripts, identifying and marking sections

with the predetermined codes• If there are multiple coders, compare coding• Review the coded data and combine into set of themes

(usually 5-7)

Qualitative Data

Present data• Often in narrative description• Quotes are often included• Images can be included• Tables are used to summarize findings• Matrices can also be used to compare

themes/groups• Diagrams used to depict concepts, show a process,

etc.

TOR Exercise: Data Analysis Procedures


• Complete IV-D of the TOR to describe the data analysis plan and the steps and available tools you plan to use in both qualitative and quantitative data analysis

• Spend approximately 45 minutes to complete this activity.


Using and Communicating ResultsSession 7

Session Learning Objectives:• Maximize the use of the evaluation findings.• Write or oversee the writing of an evaluation report.• Develop a plan to share the results

TOR:• Describe the layout and content of the evaluation report

(TOR VII-A)• Develop a dissemination plan to share evlauation findings

with stakeholders (TOR VII-B)

Evaluation ReportSession 7.1

The Evaluation Report

• The evaluation report is the key product of the evaluation process.

• Its purpose is to provide :– A transparent basis for accountability for results– Recommendations for decision-making on policies

and programs – A summary of information gathered to inform

learning and improvement.

Evaluation Report Components

• A title page.• A list of acronyms and abbreviations.• A table of contents, including a list of annexes.• An executive summary (ES).• An introduction describing the program’s background

and context.• A description of the program and the logic model.• A statement of the purpose of the evaluation.

Evaluation Report Components• Key questions and a statement of the scope of the

evaluation, with information on limitations and delimitations.

• An overview of the evaluation approach and methodology.• Data sources.• Findings.• A summary and explanation of findings and interpretations.• Conclusions.• Recommendations.• Lessons, generalizations, alternatives.• Appendices, also known as annexes, including a special

methods annex.

Characteristics of a Good Evaluation Report

Review checklist on page 67-69:

Dissemination of Evaluation ResultsSession 7.2

What informs a communication strategy?

• Who needs to know or might use the results and lessons

• What they might want to know• How to reach our audience• When is the best time or opportunity to reach

the audience• How much budget, time, personnel and other

resources are needed

Dissemination plan matrix

Stakeholder Key Findings Channel of Communication

Product to Share

Donor Quality of service Sustainability

Dissemination Meeting Abstract Power Point Slides

Using a table like the one below can be helpful in figuring out what to share with who, when, and how.

While you may take all of your stakeholders into account, you may determine that just a subset of them are most important to prioritize during the evaluation dissemination.

Channels of Communication

• Besides publishing a report, keep in mind the following ways of reaching stakeholders:– Written reports and summaries– Workshops– Publications (e.g. journals, newsletters, etc.);– Participatory methods (e.g. community meetings, discussions)– Mass media (radio, TV, newspapers or press releases)– Interactive media (websites, social media, etc.)– Research or professional meetings– Political meetings

TOR Exercise: Reporting and Dissemination Plan


• Complete VII-A and VII-B of the TOR • Spend approximately 45 minutes to complete

these sections.• Be prepared to share with the larger group.

Managing the EvaluationSession 8

Session Learning Objectives:• Prepare for evaluations.• Budget for an evaluation.• Select an evaluation team.• Develop Calls for Expressions of Interest and TOR• Manage evaluation consultant.

To continue building your TOR:• Describe the evaluation team’s roles and responsibilities (TOR V).• Describe the schedule and logistics (TOR VI), and provide a timeline

(TOR IX).• Develop an evaluation budget (TOR VIII).

Pre-evaluation planningSession 8.1

Pre-Evaluation Planning

• Whether evaluations will be internal or external, it’s important to have – team – documentation

• Pre-planning ensures efficient use of resources and adequate addressing of evaluation questions.

• CRS and American Red Cross have developed a comprehensive guide to pre-evaluation planning: http://pdf.usaid.gov/pdf_docs/PNADN086.pdf

http://pdf.usaid.gov/pdf_docs/PNADN086.pdf

Steps in Pre-Evaluation Planning

1. Identify, empower, and mentor the evaluation manager: – Internal staff member who will be managing the consultant,

usually the M&E head or someone from program management. – Clarifies the reporting chain

2. Clarify relevant guidelines and requirements:– Performance Management Plan/ MER Plan– Donor guidance – The project description– Internal evaluation policy

Steps in Pre-Evaluation Planning3. Prepare the evaluation scope of work or terms of reference and workplan

4. Identify the evaluation team

5. Organize project documentation into a bibliography(See Page 75 of manual)

6. Organize project information (briefing book)

7. Plan evaluation logistics

Budgeting for EvaluationSession 8.2

1. What funds are available in the either the project or organizational budget?

2. Who will fund the costs?3. Are there restrictions on the use of funds such as

deadlines or donor-specified scopes of work?4. Will the evaluation be outsourced? If so, what costs

would be associated with outsourcing? (advertising, etc.)

5. What internal resources (staff time, facility costs) would be required?

6. What costs and resources will be associated with the activities in the evaluation plan? Brainstorm

Key Budget Considerations

Barriers to Good Budgeting

• No initial budgeting because evaluations were low priority or forgotten

• Lack of budgeting skills in staff• Limited funding for evaluation activities• M&E staff not involved in budgeting process

M &E activities are usually 5-10% of overall project budgets, sometimes including multiple evaluations.

Evaluation TeamSession 8.3

External Evaluation Teams

• External evaluation teams are solicited through expressions of interest.

• Expressions of interest typically mirror terms of reference developed for an evaluation, though they also include guidelines on the application process.

• Following the selection of a consultant, the project writes a selection memo summarizing the selection process.

Managing Consultants

• Though consultants are independent, they must still be managed in order to ensure that the evaluation products are adequate to project needs.

• Managing consultants includes:– Briefing the consultant– Reviewing and approving Inception Report ( feedback on

data collection tools and detailed plan/methodology)– Checking on data collection– Reviewing and approving drafts and final reports– Disseminating results

Final words

• Planning is one of the most important parts of evaluation

• Be sure to link activities to evaluation questions, and that evaluation questions are suited to project needs

• Communicating and acting on results is important. A high quality evaluation report is useless if the project does nothing with the findings and recommendations.

TOR Exercise: Evaluation Team Roles


• Complete TOR V to describe the evaluation team’s roles and responsibilities

• Spend approximately 30 minutes to complete this activity.


TOR Exercise: Schedule, Logistics, and Timeline


• Complete the TOR sections to describe the schedule and logistics (TOR VI); and develop a timeline (TOR IX).

• Approximately 45 minutes to complete these sections.


TOR Exercise: Budget


• Complete TOR III related to developing an evaluation budget.

• Approximately 30 minutes to complete this exercise.


TOR Exercise: Finalizing your TOR


• Review and refine your TOR• If time permits, prepare to share your full TOR

with the workshop participants for additional review/critique

• Remember, the TOR is an iterative process!

The End