m&e overview (ii)
TRANSCRIPT
-
7/27/2019 M&E Overview (II)
1/41
M&E Overview
-
7/27/2019 M&E Overview (II)
2/41
At the end of this session, participants will
be able to:
Understand concepts of monitoring andevaluation
Understand how to set goals and objectives
Prepare an M&E plan
T. Bezabih
-
7/27/2019 M&E Overview (II)
3/41
What Is Monitoring? and What IsEvaluation? How are they different?
How do they fit together?
-
7/27/2019 M&E Overview (II)
4/41
Monitoring is the routine
process of data collection
and measurement of
progress toward programobjectives.
Monitoring involves
counting what we are
doing.
Monitoring involves
routinely looking at the
quality of our services.
-
7/27/2019 M&E Overview (II)
5/41
The regular follow-up of the implementation of
planned activities. It also involves
documentation of project activities
The systematic and continuous process of
following and keeping track of indicators in
order to ensure that the project/program isproceeding according to plan and modifying the
plan as necessary.
-
7/27/2019 M&E Overview (II)
6/41
Are projected outputs being met?
Are we heading in the right direction?
Are we in good time?
Are the indicators appropriate?
Did you identify the correct problem and has this problemchanged?
Are the intervention strategies appropriate to the targetpopulation?
What can be improved in our project?
Are we utilizing resources efficiently?
Gives us the strengths and weaknesses of our project
Provides updates for stakeholders
-
7/27/2019 M&E Overview (II)
7/41
Assesses progress against setobjectives/outputs
Supervises implementation
Assesses effectiveness of implementationstrategies
Identifies new issues and/or unforeseencircumstances that may be obstacles
T. Bezabih
-
7/27/2019 M&E Overview (II)
8/41
Identifies necessary corrective measures (strategymodification)
Verifies information first-hand for immediate
feedback Strengthens relationships between collaborators(donors, implementers and beneficiaries)
Serves as a motivation to implementers andbeneficiaries
Provides an opportunity to verify whether resourcesare being used effectively (cost-effectiveness)
-
7/27/2019 M&E Overview (II)
9/41
Review existing information related to the project. Make a conceptual framework of the project for monitoring.
Identify monitoring goals and objectives.
Identify indicators. Determine which categories of workers, supervisors orothers will be responsible for the collection of each categoryof monitoring data.
Develop a timetable for frequency of monitoring.
Develop/strengthen a management information system. Develop monitoring instruments.
Conduct monitoring activities.
-
7/27/2019 M&E Overview (II)
10/41
Analyze monitoring data.
Write a report.
Make recommendations.
Implement recommendations.
Identify new indicators based on the
recommendations.
Modify the monitoring system if necessary.
Continue to monitor.
-
7/27/2019 M&E Overview (II)
11/41
Evaluation is the use of social
research methods to systematically
investigate a programs
effectiveness. Evaluation requires study design.
Evaluation sometimes requires a
control or comparison group.
Evaluation involvesmeasurements over time.
Evaluation involves special
studies.
-
7/27/2019 M&E Overview (II)
12/41
It is a means of problem verification
It maximizes utilization of resources
It identifies the strengths/weaknesses of the project
It provides information for planning and re-planning
It provides learning opportunities
It provides an opportunity for problem solving (strategymodification)
It is a basis for maintaining and/or improving the existing
strategy It measures the effectiveness of the project/program
It is a check whether the project was implementedaccording to the detailed plan/design
-
7/27/2019 M&E Overview (II)
13/41
-
7/27/2019 M&E Overview (II)
14/41
DIFFERENCES BETWEEN MONITORING
AND EVALUATION
Monitoring Evaluation
Who M. Internal management responsibility all levels
E. Usually incorporates external inputs (objectivity)
When M. Ongoing
E. Periodic mid-term, completion, ex-post
Why M. Check progress, take remedial action, update
plans
E. Learn broad lessons applicable to other
programs/projects, policy review, etc
Focus on M. Inputs, activities, outputs, process
E. Outcome, impact
-
7/27/2019 M&E Overview (II)
15/41
Lack of appreciation of the role of monitoring andevaluation.
Fear of finding mistakes/ Fear of failure
Lack of transparency and accountability by project
managers. Lack of knowledge and skills in monitoring and evaluation.
Cost of re-designing the overall project
Resistance to change by entire project staff
People are overwhelmed by more work, lack of time
Restrictive budgets (lack of funds to accommodatemonitoring and evaluation).
Fear of piracy by external evaluators.
-
7/27/2019 M&E Overview (II)
16/41
Process Evaluation
Mid-term Evaluation
Impact Evaluation Summative Evaluation
What is it?
Why is it? When?
Who conducts it?
How should the findings be used?
-
7/27/2019 M&E Overview (II)
17/41
-
7/27/2019 M&E Overview (II)
18/41
Global Fund wants to know how many PLHIV have beenreached by your HBC program this year.
A local community-based organization wants to start addressingHIV in its community with a comprehensive BCC program and
starts by collecting key bits of information to find out who ismost in need of the services.
After a year of conducting your program, you want to know ifthe budget is being spent in the most efficient way.
NAP+ is interested in finding out if the HBC services providedare being carried out according to national standards of quality.
HAPCO wants to know if the programs being carried out inAmhara Region are changing the risk behaviors of havingmultiple sexual partners in the Region.
-
7/27/2019 M&E Overview (II)
19/41
The core of any M&E system is the
goals and objectives of the program
to be monitored and evaluated.
If the program goals and objectives
are written in such a way that they
can be easily distinguished from
one another and measured, the jobof the M&E specialist will be much
easier.
-
7/27/2019 M&E Overview (II)
20/41
General statement that describes the hoped-for
result of a program (e.g., reduction of HIV
incidence).
Goals are achieved over the long term and
through the combined efforts of multiple
programs.
T. Bezabih
-
7/27/2019 M&E Overview (II)
21/41
Specific, operationalized statement detailing thedesired accomplishment of the program.
A properly stated objective is action-oriented, starts
with the word to, and is followed by an actionverb.
Objectives address questions ofwhat and when,but not why" orhow.
Objectives are stated in terms of results to beachieved, not processes or activities to be
performed.
-
7/27/2019 M&E Overview (II)
22/41
(S) Specificity Is it specific? Does it covers onlyone rather than multiple activities?
(M) Measurability Can it be measured or countedin some way?
(A) Attainability Is the objective actually doable?Can we achieve this goal?
(R) Relevance How important is this objective tothe work that we are doing? How relevant is it toachieving our goal?
(T) Time Does the objective give a timeframe bywhen the objective will be achieved, or a timeframeduring which the activity will occur?
-
7/27/2019 M&E Overview (II)
23/41
Start with the outcome of theintervention (purpose), thendefine the outputs and then
activities that would achievethe outputs. Then finally theimpact.
-
7/27/2019 M&E Overview (II)
24/41
MONITORING &EVALUATION INDICATORS
An indicator is the quantitative or qualitative
evidence that will be used to assess progress
towards an objective.
Indicators provide the basis for monitoring
progress and evaluating the achievement of
outcomes.
-
7/27/2019 M&E Overview (II)
25/41
CLASSIFICATIONS OF INDICATORS
Input indicators: measure all the investment
and recurrent cost resources (e.g. human,
financial, facility, equipment, supplies) needed
to enable the activities to be delivered
Process indicators: attempt to set standards
for the quality of activities to be carried out,
such as appropriate training methods or an
adequate supervision plan.
-
7/27/2019 M&E Overview (II)
26/41
CLASSIFICATIONS OF INDICATORS
Output indicators: measure result of inputs &process:
Level at which program is functioning (operations):
e.g. number of services delivered, medicinesprovided, IEC messages delivered
How well program is functioning in terms of itsobjectives (performance): e.g., adequacy of output
(e.g. services delivered) in terms of quality, equity,utilization/access, cost-effectiveness, efficiency,sustainability
-
7/27/2019 M&E Overview (II)
27/41
CLASSIFICATIONS OF INDICATORS
Outcome indicators: measure the main
changes in the program population that are
expected in the short-term, such as improved
knowledge, attitudes or behaviors
Impact indicators: measures a long-term
effect that the program aims to produce, such
as reduction of HIV prevalence
-
7/27/2019 M&E Overview (II)
28/41
Practical Examples
Examples of Outcomes, Indicators,
Baselines and Targets (Prevention):
Table 2 Examples of Outcomes, Indicators,
Baselines and Targets (Treatment)- Table
3
-
7/27/2019 M&E Overview (II)
29/41
FEATURES OF GOOD INDICATOR
Valid: actually measures the phenomenon (e.g.,self-reported vs. facility based data)
Reliable: produces same results when used morethan once to measure precisely the same
phenomenon
Specific: measures only the phenomenon it isintended to measure
Sensitive: reflects changes in the state of the
phenomenon Operational: measurable with developed and
tested definitions and reference standards
-
7/27/2019 M&E Overview (II)
30/41
TYPES OF INDICATORS
PROCESS - show whether the activities that wereplanned are actually being carried out, and carried outeffectively
IMPACT - to assess what progress is being made towards
reaching the objectives, and what impact the work has hadon the different groups of people affected by the work
QUANTITATIVE indicators involve the definition ofnumerical measures eg number of meetings attended
QUALITATIVE indicators refer to definingcharacteristics that cannot be quantified. For example,changes in behaviour orpeoples perceptions
PROXY indicators measure things that represent (orapproximate) changes that cannot be measured directly.
-
7/27/2019 M&E Overview (II)
31/41
CHOOSING INDICATORS
Select among possible choices based on:
What level of system youre measuring(inputs, process, outputs, outcomes)
Data availability & quality
Cost of collecting data
Comparability with other programs, projects
-
7/27/2019 M&E Overview (II)
32/41
CRITERIA FOR DEVELOPING AND SELECTING
INDICATORS
Result oriented: should focus at measuring resultexpected from the program
Direct: should always be direct as much as possible.Proxy indicators could only preferred when direct
indicators are not possible or difficult to use for differentreasons
Objective: Could be understood without differentinterpretations
One-dimensional/Independent: output indicatorscannot be used to prove achievement of purpose and
purpose indicators achievement of goal
Quantitative: (Whenever possible). However, thereare instances where qualitative indicators could be
desired or even more useful.
-
7/27/2019 M&E Overview (II)
33/41
CRITERIA FOR DEVELOPING AND
SELECTING INDICATORS
Disaggregated: (Whenever possible) usedisaggregated indicators. e.g. by sex, age,geographical location, education level, etc.
Simplicity/Unambiguous: Simple and clearly definedin the program's context
Validity/Consistency: The values of the indicatorsshould stay valid/constant as long as they are collectedin identical conditions, no matter who does the
collecting Specificity: Should measure specific conditions that
the program aims to do & change
Sensitivity: should be highly sensitive to changes in a
program situation.
-
7/27/2019 M&E Overview (II)
34/41
CRITERIA FOR DEVELOPING AND
SELECTING INDICATORS
Cost-Effectiveness/Ease of Data Collection:
Identify and select indicators for which data, time
and resource (budget) is available
Relevance/Reliable: Should be relevant to program
objectives and measure what is expected to achieve
Timeliness: Should be sensitive to the time it is
done and possible to collect the data reasonably andquickly.
-
7/27/2019 M&E Overview (II)
35/41
DATA GATHERING, ANALYSIS, AND
REPORTING STRAGEY
The HIV/AIDS Monitoring and EvaluationFramework, should clearly define:
Indicators for the specific HIV/AIDS
intervention areas Sources of data to generate the indicators
Measurement tool
Frequency of data collection
Responsible body for data collection,compilation, analysis and reporting bylevel
-
7/27/2019 M&E Overview (II)
36/41
Suggested reporting schedules for
EMSAP II and GF projects
Level of Indicators Recommended frequency of reportingExample of data collection methods used
Input/process Continuously Health and non- health servicesstatistics,
Program monitoring
Output Quarterly, semi-annually, or annually Health and non-healthservices statistics,
Program monitoring
Outcome 1 to 3 years Population-based surveys,
Health facility surveys,
Special studiesImpact 2 to 5 years Surveillance,
Population-based surveys,
Special studies
-
7/27/2019 M&E Overview (II)
37/41
Refer M&E OVER
-
7/27/2019 M&E Overview (II)
38/41
The proposal submitted to the Global Fund by
NAP+ has five major core areas: ART
adherence literacy and counseling, home based
care, job creation, food and nutrition support
and capacity building
Break into five groups and write a goal and
three SMART Objectives for each of core areasfor the M&E system.
-
7/27/2019 M&E Overview (II)
39/41
Careful selection of the questions you want answered
through monitoring and evaluation will greatly help you
develop your M&E processes and work plan.
At the outset of the planning process, program managersshould ask themselves where they want the program to take
them.
Many of these questions will be reflected in the goals and
objectives.
-
7/27/2019 M&E Overview (II)
40/41
Was the activity carried out as planned?
Did it reach its target beneficiaries?
Did any changes in exposure to HIV infectionresult?
How will the risk behaviors of the targetpopulation be affected?
What sort of coverage do you expect to have? Did STI/HIV incidence change?
How much did it cost?
T. Bezabih
-
7/27/2019 M&E Overview (II)
41/41
Refer back to your previous group activity
where you developed a goal and a set of
objectives. Now look at these goals and
objectives and come up with at least three
monitoring questions and at least two
evaluation questions.