towards an enhanced monitoring and evaluation system: how m&e can become a tool for supporting...
TRANSCRIPT
Towards an enhanced monitoring and evaluation system: how M&E can become a tool for supporting programme management
Presentation for COP RBM premium group 6 April 2011
Benedict WautersDeputy director, Flemish ESF agency
Coordinator of the COP RBM
Where are the results?
“The most evident weaknesses which indicate the need for reform of cohesion policy are: …Methodological and operational problems that have prevented
both the appropriate use of indicators and targets – for which no comparable information is available - and a satisfactory analysis of “what works” in terms of policy impact.
A remarkable lack of political and policy debate on results in terms of the well-being of people, at both local and EU level, most of the attention being focused on financial absorption and irregularities.”
Independent report “An Agenda for a Reformed Cohesion Policy” delivered at the request of Commissioner for Regional Policy, Ms Hübner.
The discussion needs to go beyond (better) formulation of indicators!
We have to look at how M&E as a wholeSupports better programme management
Contents1. Theories of change as the basis of the M&E system2. Outcome indicator formulation is based on evidence and
relates to net effects3. Programme priorities should be based on a demand logic4. Ex ante evaluation: independent review of evidence5. Generate an evidence base for actions if there is none
available6. Programmes should be dynamic7. Independent evaluation authority8. Evaluation of TA = quality management9. Programme = policy dialogue10. Unique, complex, transformational projects should not be
impeded by the proposed enhanced M&E system
1. Theories of change as the basis of the M&E system
If Member States put forward measures/types of actions in their programmes, there should also be theories of change that explain how they intends to get from these actions to outcomes theories of change not the same as what is referred to
as “intervention logics”, although they should form a better basis for elaborating these
Within these theories of change, it should be clear where intrinsic needs of target groups are located (dimensions of well-being).
Theories of change should be supported by evidence
A simple cause-effect chain
Source: Adapted from Florida Department of Children and Families
Inside intervention: activity=CONTROL=OUTPUTS
External environment: change=INFLUENCE(varying degrees)
=OUTCOMES
Improvedhealth
Improvedcareer
prospects
A simple cause-effect chain
Source: Adapted from Florida Department of Children and Families
Inside intervention: activity=CONTROL=OUTPUTS
External environment: change=INFLUENCE(varying degrees)
=OUTCOMES
Improvedhealth
Well-being
Improvedcareer
prospects
Direct influence
Contents
1. Theories of change as the basis of the M&E system2. Outcome indicator formulation is based on evidence and
relates to net effects3. Programme priorities should be based on a demand logic4. Ex ante evaluation: independent review of evidence5. Generate an evidence base for actions if there is none
available6. Programmes should be dynamic7. Independent evaluation authority8. Evaluation of TA = quality management9. Programme = policy dialogue10. Unique, complex, transformational projects should not be
impeded by the proposed enhanced M&E system
2. Outcome indicator formulation is based on evidence and relates to net effects Based on the review of the evidence:
indicators can be formulated for outcomes reasonable targets can be set
Outcome targets should always be net, never gross Gross targets are totally irrelevant for programme managers
This is equivalent to companies being interested in knowing whether needs of market segments are fulfilled but not caring whether these needs are being fulfilled by their projects.
For outcomes that are very closely connected to the (elements of) financed actions within the theory of change and that become visible almost immediately after the action, attribution is not a (huge) concern:
Assumption that outcome is not due to something else happening at the same time can be supported (eg if before and after training you take test, the test result improvement probably due to the training –of course you can never be sure about this but for managerial purposes it is reasonable)
2 -continued
Deadweight) remains a concern (how much would have happened anyway
without financed action eg people would have been trained anyway) but this
is not relevant for short term managerial purposes (as Keynes said “in the
long run, we are all dead”) where the main issue is: do people learn what they
are supposed to learn in your training?
For outcomes that are further removed from the action, rigorous
impact evaluation is required
Managerial purpose shifts to the longer run
This may range from qualitative, in depth case studies to control
group style quantitative evaluations (or a combination)
These should therefore be planned for at the start of the OP
It is not necessary to cover all outcomes (see also principle 7)
Contents
1. Theories of change as the basis of the M&E system2. Outcome indicator formulation is based on evidence and
relates to net effects3. Programme priorities should be based on a demand logic4. Ex ante evaluation: independent review of evidence5. Generate an evidence base for actions if there is none
available6. Programmes should be dynamic7. Independent evaluation authority8. Evaluation of TA = quality management9. Programme = policy dialogue10. Unique, complex, transformational projects should not be
impeded by the proposed enhanced M&E system
3. Programme priorities should be based on a demand logic Within a priority outcome objectives should be
logically connected to each other within a comprehensive theory of change
The logic of a priority therefore should be demand and NOT supply led: eg. the priorities should not be formulated on the
basis of similar actions (eg. training) without taking into account the very distinct problems, needs and target groups these actions are supposed to address eg. training for employed middle managers is a different
issue than training for unemployed Roma. Putting these two on a same priority because it is all training reflects a supply led way of thinking.
Several actions
connected inone theory of
change = basis for a
priority
Source: E valuation of Flemish ESF OP 2007-14
Contents
1. Theories of change as the basis of the M&E system2. Outcome indicator formulation is based on evidence and
relates to net effects3. Programme priorities should be based on a demand logic4. Ex ante evaluation: independent review of evidence5. Generate an evidence base for actions if there is none
available6. Programmes should be dynamic7. Independent evaluation authority8. Evaluation of TA = quality management9. Programme = policy dialogue10. Unique, complex, transformational projects should not be
impeded by the proposed enhanced M&E system
4. Ex ante evaluation: independent review of evidence Ex ante evaluation at the programme negotiation
stage should consist, inter alia, of a serious evaluation of the evidence that a theory of change will work and will be measured correctly: For actions already proposed at the negotiation stage, the
evidence base that relates to the formulated theories of change for these actions, is reviewed.
If this is not given by programme management, it can be a task for the evaluators to find the existing evidence and link it to the programme
A judgment can then be given, based on this evidence base, on how likely it is that the theor(ies)y will be effective and that there are no conflicts with the policy context that will reduce this effectiveness.
Independence of the ex ante evaluation should be ensured.
Contents
1. Theories of change as the basis of the M&E system2. Outcome indicator formulation is based on evidence and
relates to net effects3. Programme priorities should be based on a demand logic4. Ex ante evaluation: independent review of evidence5. Generate an evidence base for actions if there is none
available6. Programmes should be dynamic7. Independent evaluation authority8. Evaluation of TA = quality management9. Programme = policy dialogue10. Unique, complex, transformational projects should not be
impeded by the proposed enhanced M&E system
5. Generate an evidence base for actions if there is none available For actions where the ex ante evaluation shows there is no
sufficient evidence base, small scale experiments to understand how these actions work and what their potential is, should be conducted. Small scale experiments can be carried out quickly and at reasonable
costs. They are not meant to be identical to an actual project but to focus on those elements of an action that are considered essential and to then assess the influence of this on intended outcomes (including intermediate ones).
This also feeds target setting as net effects obtained by the experiment provide a decent basis for extrapolation.
Actions where there is no sufficient evidence base should not be financed but put on hold or at least capped in terms of the amount of finances that can flow to them until a minimal evidence base can be generated.
Contents
1. Theories of change as the basis of the M&E system2. Outcome indicator formulation is based on evidence and
relates to net effects3. Programme priorities should be based on a demand logic4. Ex ante evaluation: independent review of evidence5. Generate an evidence base for actions if there is none
available6. Programmes should be dynamic7. Independent evaluation authority8. Evaluation of TA = quality management9. Programme = policy dialogue10. Unique, complex, transformational projects should not be
impeded by the proposed enhanced M&E system
6. Programmes should be dynamic For actions that are newly launched during
the life of the programme and that were not yet in the OP when it was negotiated, there should also be a theory of change that connects these new actions to the existing theory of change within one of the priorities. This allows to situate the new actions properly
within the various OP priorities. Ex ante evaluation and indicators formulation /
target setting should happen as for the actions already known at the beginning of the OP. This includes also the obligation to conduct small scale experiments if the evidence base is not sufficient.
Contents
1. Theories of change as the basis of the M&E system2. Outcome indicator formulation is based on evidence and
relates to net effects3. Programme priorities should be based on a demand logic4. Ex ante evaluation: independent review of evidence5. Generate an evidence base for actions if there is none
available6. Programmes should be dynamic7. Independent evaluation authority8. Evaluation of TA = quality management9. Programme = policy dialogue10. Unique, complex, transformational projects should not be
impeded by the proposed enhanced M&E system
7. Independent evaluation authority An independent evaluation authority judges whether
programme management acts responsibly throughout the programme execution period: The audit authority is the guardian of compliance with the rules.
However, there is yet no guardian that watches whether programme managers are also acting responsibly (ie. learning what works and what not/less and acting on this). Combined with weak evaluation cultures and capacities in various regions and countries this is serious threat to effectiveness.
A possible solution to both issues is to set up an independent evaluation authority (IEA), at the level of the NSRF or even the programme if the latter is large enough.
The programme managers still will have the initiative to determine, supported by the IEA , the plan for specific evaluations to be carried out. This plan should be based on the overview of where the knowledge gaps in the evidence base are greatest, or where the evidence base is becoming outdated. This means it is not necessary to cover all outcomes all the time.
7. continued The independence of these evaluations will be guaranteed by
the IEA. This IEA can be located together with the audit authority, as long as it is clear that it has another finality and requires other competences than those of auditors, or it could be established as a unit reporting directly to parliament or another position that guarantees some independence e.g. at the ministry of finance.
The IEA would have the critical mass to be a centre of evaluation expertise.
They should do some of the evaluation work themselves as well as act as an intermediary to reach consultants and academics in the market, providing for a sophisticated demand side, triggering capacity building and innovation in the market as well.
The IEA should be funded by OPs technical assistance to carry out evaluations for these OPs but they can also compete commercially to provide evaluation services for non-structural funds programmes and policies, thus building a dynamic of improvement and innovation in the evaluation market.
7. continued The IEA, similar to the audit authority, should annually provide
a judgment as to how “responsible” the programme manager has acted (did they learn about and adapt their programmes). If this assurance cannot be given the EC could block payments on the basis of this. The EC will however first engage in a dialogue between the IEA and MA.
The audit authority is in charge of assessing whether the IEA is itself in compliance with rules governing the work of the IEA.
The IEA would also be in charge of setting up and conducting the ex post evaluation. In order to do this, the IEA can draw upon the evaluations commissioned by the programme managers as well a complement these with extra evaluation activities, financed directly by the DG EMPL budget and with DG EMPL as commissioner.
It should also be able to conduct the necessary ex ante evaluations referred to earlier.
Contents
1. Theories of change as the basis of the M&E system2. Outcome indicator formulation is based on evidence and
relates to net effects3. Programme priorities should be based on a demand logic4. Ex ante evaluation: independent review of evidence5. Generate an evidence base for actions if there is none
available6. Programmes should be dynamic7. Independent evaluation authority8. Evaluation of TA = quality management9. Programme = policy dialogue10. Unique, complex, transformational projects should not be
impeded by the proposed enhanced M&E system
8. Evaluation of TA = quality management A clear distinction should be made between
evaluation of the technical assistance and evaluation of the actions/projects that have been financed. The former is part of quality management and
should be executed wholly by the Managing Authority itself. A variety of quality assurance tools exist ranging from ISO to EFQM to support this.
The latter is the remit of the IEA. Both are subject matter to be discussed during
annual reviews with the EC.
Contents
1. Theories of change as the basis of the M&E system2. Outcome indicator formulation is based on evidence and
relates to net effects3. Programme priorities should be based on a demand logic4. Ex ante evaluation: independent review of evidence5. Generate an evidence base for actions if there is none
available6. Programmes should be dynamic7. Independent evaluation authority8. Evaluation of TA = quality management9. Programme = policy dialogue10. Unique, complex, transformational projects should not be
impeded by the proposed enhanced M&E system
9. Programme = policy dialogue The debate between the Commission and programme
management should include participation of the relevant representatives of the government (incl. ministers) as many of the interventions are shaped by a particular policy context. If the policy context is a problem for effective ESF
interventions (as e.g. evidenced by evaluations), then this cannot be addressed at the level of programme management. This requires the presence also of the political level.
Annual reviews should be used to discuss policy level implications and the need to change certain policies. Actions that cannot be effective under current policy conditions may be blocked by the EC.
Contents
1. Theories of change as the basis of the M&E system2. Outcome indicator formulation is based on evidence and
relates to net effects3. Programme priorities should be based on a demand logic4. Ex ante evaluation: independent review of evidence5. Generate an evidence base for actions if there is none
available6. Programmes should be dynamic7. Independent evaluation authority8. Evaluation of TA = quality management9. Programme = policy dialogue10. Unique, complex, transformational projects should not be
impeded by the proposed enhanced M&E system
10. Unique, complex, transformational projects should not be impeded by the proposed enhanced M&E system These projects that are addressing many
different issues by containing many different, customised actions, often formulated in a bottom-up fashion (e.g. transformation of an entire company to increase quality of labour of employees) with non-standard outputs should not be impeded by the desire for a priori, generic, top-down, theories of change with standard ouptuts formulated at the programme level.
Output-oriented calls
Source: Adapted from Florida Department of Children and Families
Inside intervention: activity=CONTROL=OUTPUTS
External environment: change=INFLUENCE(varying degrees)
=OUTCOMES
Improvedhealth
Well-being
Improvedcareer
prospects
Direct influence
Outcome-oriented calls
Source: Adapted from Florida Department of Children and Families
Inside intervention: activity=CONTROL=OUTPUTS
External environment: change=INFLUENCE(varying degrees)
=OUTCOMES
Improvedhealth
Well-being
Improvedcareer
prospects
Direct influence
10 - continued It is necessary to address this with M&E in two
ways: At the project level, the need to be unique has to be
addressed by a sophisticated project appraisal process (e.g. using project cycle management approaches) that includes an ex ante assessment of:
a project level theory of change (eg using problem trees) with connected unique objectives, indicators and targets
as well as a screening of the evidence which is provided by projects applicants to substantiate that the project will be effective.
This is similar to what happens at the level of the programme between the EC and programme management
Next slides show example of system as used in Flanders
Outcome-oriented planning: from problems to objectives
Global objectives More disabled persons feel appreciated
Specific objective
More disabledpersons fulfill their potential
in a work environment
Global
Specific
Operational
TO TAKE ONNOT TO TAKE ON
Strategy 1: mobilise
supply
Strategy 2: increase
demand
Project intervention logic
MiEANS
Disabled retain
security when they take a
job
Disabled not afraid to look for
work
Disabled invited more
for job inteviews
Disabled find job more easily
Disabled feel appreciated
Physical environment more adapted to disabilities
Disabled have more self-
confidence
Adapting physical
environment is
expensive
Employers make betterassessment of potentialof disabled
Disabled areable to
performbetter
Training more
adapted to
disabledAdaptatio
n of training is expensive
Objective tree derived from problem tree
Non-disabledinteract moreeasily with disabled
GOAL
Operational objectives
Employers make better
assessment of potential
of disabled
Disabled areable to
performbetter
Disabledinteract more
easily with non-disabled
indicator
indicator
indicator
2. Indicators 3. Source
Project milestones
Budget/resources
Activities
Operational
Specific
GlobalWhat is
contributed toby project
Advantage for target
derived from using
what is available
What is made available:
knowledge, practice, attitudes, services...
What is donein the project
What needsto be the case, even if operational obj.
are realised?
What needsto be the case, even
if spec. obj. is realised?
1. Project description
4. Conditions
Outcome-oriented planning: summary of project
If target group does not believe advantage is worth acquiring, they will not use what is available!
What needsto be the case, even
if activities are executed?
Non-work environmentremains supportiveof disabled persons
1. x% more disabled are invited for job
interview each year 2. x% of disabled
obtain 80% score on standard competency
test3. x% of disabled score
min satisfactoriy in team role play
1. Diversity statistics2. Assessment center3. Assessment center
x% more disabled with mild physical
handicapshave fixed contract by
2010
CVS (client tracking system)
Share of excludedgroups in total FTEs of
organisations increased to 5% by
2020
Survey question tobe included in
regular sectoralsurvey
HR officers do notchange jobs
Disabilities of participants do not deteriorate beyond
being “mild”
-Number of new job openings remains constant
or grows-Anti-discriminationlaws continue to be
enforced-Disabled apply for jobs
Activities
Operational
Specific
Global
Social inclusionincreased
More disabled persons feel appreciated
More disabledpersons fulfill their
potentialin a work
environment1. Employers make betterassessment of potentialof disabled2. Disabled are able to perform better3. Disabled interact more easily with non-disabled
1. Project description
2. Indicators 3. Source 4. Conditions
Outcome-oriented planning: example summary of project
1.1 Info sessions for HR officers1.2 Adaptation of recruitment procedures1.3 Coaching for HR officers during recruitment...
Project milestones,budget/resources:
see planning
Outcome oriented planning: application form
Application form consists of list of standard questions: Analysis:
Target group description Process of involving stakeholders Problem tree Goal tree and strategic choices Lessons learnt from the past Policy context
Formulation of the project: Logical framework Planning (incorporating some elements of outcome mapping) Management (RACI table) Evaluation
Tools/guidance are provided to enable applicants to answer each question properly
Outcome-oriented planning: project appraisal Detailed appraisal grids (criteria) for readers of
project proposals linked directly to applicant
answers:
Relevance: the project should meet demonstrated and
high priority needs: 7 tables
Feasibility: the project should be well-designed and
deliver sustainable benefits to target groups : 5 tables
Good project management: 3 tables
Outcome-oriented planning : project appraisal Detailed appraisal grids (criteria) for readers of
project proposals linked directly to applicant
answers:
Relevance: the project should meet demonstrated and
high priority needs: 7 tables
Feasibility: the project should be well-designed and
deliver sustainable benefits to target groups : 5 tables
Good project management: 3 tables
Several proposal/appraisal questions
request that evidence be providedto reasonably assert that the project will
be effective
Appraisal is linked to the proposal questions
4 All components of an evaluation strategy are discussed and they are coherent. External stakeholders incl. the final target group will supply information in order to judge the project. At the same time it is clear who will execute the evaluation and who will follow-up and a budget is fixed.
3 All components of an evaluation strategy are discussed and they are coherent. External stakeholders incl. the final target group will supply information in order to judge the project.
2 All components of an evaluation strategy are discussed but they are not all coherent.
1 Only a limited amount of the components of an evaluation strategy are discussed.
0 Evaluation is confused with monitoring.
Proposal question on self-evaluation
•Goal of evaluation•Evaluation approach •Evaluation questions •Method of the evaluation•External or internal evaluators •Steering group evaluation: •Timing of evaluation and deliverables•Budget
Appraisal
Project applicants can score themselves in
advance!
Appraisal is linked to the proposal questions
4 All components of an evaluation strategy are discussed and they are coherent. External stakeholders incl. the final target group will supply information in order to judge the project. At the same time it is clear who will execute the evaluation and who will follow-up and a budget is fixed.
3 All components of an evaluation strategy are discussed and they are coherent. External stakeholders incl. the final target group will supply information in order to judge the project.
2 All components of an evaluation strategy are discussed but they are not all coherent.
1 Only a limited amount of the components of an evaluation strategy are discussed.
0 Evaluation is confused with monitoring.
Proposal question on self-evaluation
•Goal of evaluation•Evaluation approach •Evaluation questions •Method of the evaluation•External or internal evaluators •Steering group evaluation: •Timing of evaluation and deliverables•Budget
Appraisal
Project applicants can score themselves in
advance!
Self-evaluation stimulatedbut initially more intended to
generate evaluationculture at project level rather
than be useful directly for MA
(hard to aggregate)
10- continued
At the programme level, there can still be a more
generic theory of change, based on a broadly
formulated typology of actions that are eligible to
be financed within the complex projects.
This allows a programme level view of overall
effectiveness of projects that are in themselves
formulated in unique ways but that share an underlying
common logic.
The same principles as applicable to other types of
actions (see principles 1-9) can then apply here as well.
Towards a hybrid for outcome-oriented calls?
Top down intervention theory of change for a bottom-up call:
Could be used for M&E at programme level in parallel to project level bottom-up system?
10- continued This does mean that these projects will use two
reporting mechanisms: one at the programme level to feed the programme level
monitoring system. Of course, only paths in the programme level theory of change that are connected to types of actions present in the project need to be reported on.
one at project level to report on self-defined objectives and indicators that can be unique and more specific to the project than the programme level ones.
This system can thus marry – to some extent- a top down with a bottom up logic.
It is still important to understand that to really understand what is happening in and across these projects, more developmental M&E systems may be necessary at the level of the MA (e.g. SenseMaker can code and map 95,000 stories in 24 hours –see next slides)
Thank you!