who is listening to who, how well and with what effect? by daniel ticehurst

WHO IS LISTENING TO WHO, HOW WELL AND WITH WHAT EFFECT?

DANIEL TICEHURST 1

OCTOBER 16TH, 2012

1 Project Director, Performance Management and Evaluation, HTSPE Ltd.

“Just weighing a pig doesn’t fatten it. You can weigh it all the time, but it’s not making the hog

fatter.”

President Obama. Green Bay town hall meeting, June 11th 2009

http://pifactory.wordpress.com/2009/06/16/just-weighing-a-pig-doesnt-fatten-it-obama-hint-on-testing/



ACKNOWLEDGEMENTS I’d like to thank the following people for their comments on early drafts: Andrew Temu, James Gilling, Jonathan Mitchell, Rick Davies, Harold Lockwood, Mike Daplyn, David Booth, Simon Maxwell, Ian Goldman, Owen Barder, Natasha Nel, Susie Turrall and Patricia Woods. Particular thanks go to Martine Zeuthen for her support throughout and to Larry Salmen whose comments and writings encouraged me to start and keep going. For their support to editing the first and final drafts, special thanks to Michael Flint, Clive English and Sarah Leigh-Hunt.

ContentsEXECUTIVE SUMMARY............................................................................................................2

1. INTRODUCTION...........................................................................................................7

A. WHAT ARE RESULTS?.....................................................................................7

B. WHAT ARE THE PRACTICAL DIFFERENCES BETWEEN MONITORING AND EVALUATION?.............................................................................................................8

2. THE STARTER PROBLEM...........................................................................................10

3. THE VALUE OF MONITORING IN UNDERSTANDING BENEFICIARY VALUES....................15

4. THE NEED FOR FEEDBACK LOOPS............................................................................16

5. THE IMPORTANCE OF INSTITUTIONS...........................................................................19

6. MAIN OBSERVATIONS OF CURRENT PRACTICE...........................................................21

7. CONCLUSIONS..........................................................................................................26

Who is Listening to Who, How Well and with What Effect? | Daniel Ticehurst 1

EXECUTIVE SUMMARYI am a so called Monitoring and Evaluation (M&E) specialist although, my passion is monitoring. Hence I dislike the collective term ‘M&E’. I see them as very different things. I also question the setting up of Monitoring and especially Evaluation units on development aid programmes: the skills and processes necessary for good monitoring should be an integral part of management; and evaluation should be seen as a different function. I often find that ‘M&E’ experts over-complicate the already challenging task of managing development programmes. The work of a monitoring specialist is to help instil an understanding of the scope of what a good monitoring process looks like. Based on this, it is to support those responsible for managing programmes to work together in following this process through so as to drive better, not just comment on, performance.

I have spent most of my 20 years in development aid working on long term assignments mainly in various countries in Africa and exclusively on ‘M&E’ across the agriculture and private sector development sectors. Of course, just because I have done nothing else but ‘M&E.’ does not mean I excel at both. However, it has meant that I have had opportunities to make mistakes and learn from them and the work of others.

The purpose of this paper is to stimulate debate on what makes for good monitoring. It draws on my reading of history and perceptions of current practice, in the development aid and a bit in the corporate sectors. I dwell on the history deliberately as it throws up some good practice and relevant lessons. This is particularly instructive regarding the resurgence of the aid industry’s focus on results and recent claims about scant experience in involving intended beneficiaries2 and establishing feedback loops.3 The main audience I have in mind are not those associated with managing or carrying out evaluations. Rather, this paper is aimed at managers responsible for monitoring (be they directors in Ministries, managers in consulting companies, NGOs or civil servants in donor agencies who oversee programme implementation) and will improve a neglected area.

Human behaviour is unpredictable and people’s values vary widely. In the development context, the challenges lie in how to understand the assumptions development aid programmes make about their beneficiaries. Ultimately, understanding behaviours and decisions is what economics is all about.4 One of its tasks is to show how sometimes ignorant we are in imagining what we can design to bring about change.5

As Hayak explains, often our inability to discuss seriously what really explains underlying problems in development is due to timidity about soiling our hands going from purely scientific questions into value questions.

Both Hayek and Harford argue that a subtle process of trial-and-error can produce a highly successful system. Certainly, there are no reliable models of behaviour that can predict the results of development aid programmes with certainty. Development aid

2 People or institutions who are meant to benefit from a particular development initiative.3 The Sorry State of M&E in Agriculture: Can People-centred Approaches Help? Lawrence Haddad,

Johanna Lindstrom and Yvonne Pinto. Institute of Development Studies, 2010.4 The Undercover Economist. Tim Harford. Abacus an imprint of Little, Brown Book Group 20065 The Fatal Conceit. Frederick Von Hayek. University of Chicago Press. 1991.


programmes are delivered in complex and highly unpredictable environments and thus are associated with, and subject to, all kinds of ‘jinks’ and ‘sways’. These are often overlooked and/or under-estimated in how they influence the results sought and ultimately, how they are monitored and evaluated.

Furthermore, as Rondinelli has stated, the way programmes are designed and monitored sits uncomfortably with these complexities:

“the procedures adopted for designing and implementing aid interventions often become ever more rigid and detailed at the same time as recognising that development problems are more uncertain and less amenable to systematic design, analysis and monitoring.” 6

This highlights the need to find ways of understanding values: appreciating and learning about, through feed-back, the opinions of beneficiaries in terms of their assessment of the relevance and quality of the aid received. For development to have an impact on poverty reduction, the learning process must incorporate and use the perspectives of beneficiaries.

As Barder comments, and as a recent Harvard Business Review makes explicit, approaches to gauging client feedback are under-developed for two key reasons7:

Either beneficiaries and institutions are simply not asked for their opinions due to the emphasis of monitoring on, for example, enabling subsequent impact assessment and/or limiting its enquiry to ‘tracking’ effort and spend and, if they are,

The beneficiaries’ response to the performance of those providing support or services is seldom validated by them and/or fed back in the form of remedial actions. So why bother providing feedback in the first place?

In the business world, realising that customer retention is more critical than ever, companies have ramped up their efforts to listen to customers. Many however struggle to convert their findings into practical prescriptions. Some are addressing that challenge by creating feedback loops that start at the front line such as Pfizer who uses approaches similar to what development aid refers to as participatory story telling. Unlike development aid, however, the concept of participation is applied to allowing opportunities for front line staff, in addition to their customers or beneficiaries, to tell their stories. Many companies have succeeded at retaining customers by asking them for simple feedback-and then empowering frontline employees to act swiftly on that feedback. The importance of understanding staff and client or customer satisfaction was highlighted through the balanced scorecard by Kaplan and Norton. 8

6 Development Projects as Policy Experiments. An Adaptive Approach to Development Administration. Development and Underdevelopment Series. Methuen and Co Ltd 1983

7 http://www.owen.org/blog/4018) 2010 and “Closing the Customer Feedback Loop”, By Rob Markey, Fred Reichheld and Andreas Dullweber, Harvard Business Review, December 2009

8 The Balanced Scorecard, developed by Robert Kaplan and David Norton in 1994, is a performance management tool used by managers to keep track of the execution of activities by the staff within their control and to monitor the consequences arising from these actions. Its balanced nature is how it is based around four perspectives: Financial (how do we look to shareholders?), Customer (how do we look to our customers?), Internal Business Process (What must we excel at?) and Learning


In the field of what is called Monitoring and Evaluation (M&E), few efforts try and understand behaviour. Often they tend to control expenditure and analyse other numbers and assess developmental change, but not so much values and opinions. I maintain that trying to assess 'profound' and lasting developmental impacts, in the absence of effective feedback loops, is impractical and of limited use. I further argue that this should be a core feature of any monitoring system and, for practical management reasons, should not be the sole domain of evaluation.

I do not want to come across as being too black and white or dogmatic about what constitutes Monitoring as opposed to Evaluation. Although opinions differ as to what extent Evaluation is independent of and/or relates to Monitoring, I find it useful to define the main source of differences according to: a) the responsibilities and primary users of the information generated; b) their objectives; c) their requirements for comparative analysis (across time, people and space); and d), their reference periods.

I see monitoring as having three inter-related parts:

one that is about controlling expenditures in the context of cataloguing activities, that involves a participatory approach between those responsible for delivering the support and the finance team;

another that tracks and analyses the reach of the support (ie, outputs) to intended beneficiaries these activities make available and how this varies; and

one that gauges how and to what extent beneficiaries respond to this support – their assessment of its quality, relevance and ultimately usefulness – and also how this varies among them.

The questions associated with the third component, I maintain, should not be held in abeyance pending an evaluation. Doing so begs very real questions as to the extent to which managers are accountable for the quality and relevance of the support if they are not listening to beneficiary opinion and response. Monitoring needs to be less than periodically surveying socio-economic impacts irrespective of approach but also more than just cataloguing ‘outputs and activities’ and controlling ‘spend’. 9

That what I refer to as the third component of any good monitoring system, others may see as evaluation, gives me hope: that good monitoring practice involves getting outside the office, listening to beneficiaries and taking what they say on board, re-adjusting accordingly and closing the feedback loop by letting them know what you have done with their feedback.

Of course, evaluations do this as well. But understanding the values and behaviours of beneficiaries is an approach they both share. The difference between how monitoring and evaluation try to achieve this understanding is based on approach: who does this, how often, why, with type of comparisons across people and places, and for whom?

Monitoring can and should ultimately drive better performance and involve participatory processes including, but not limited to, those between the intervention and intended

and Growth (How can we continue to improve and create value?). 9 Such surveys perhaps need doing but not by those attached to programmes.


beneficiaries (be they the poor themselves or institutions that serve them, depending on the outcome sought).10 Having the ability to listen and understand how and in what ways beneficiaries respond to development programmes, and feeding this information back to decision-makers should not be judged by academic standards alone.

I do not see the problem as an absence of tools or methods. They are there. Beneficiary Assessments is one stand out example and is not new - the approach was first developed in the late 1980s and described in 1995.11 Another is Casley and Kumar’s Beneficiary Contact Monitoring (BCM), the equivalent in addition to beneficiary assessments, to what I describe as the third component part of a monitoring system.12 Such assessments, I argue, can better enable improvements in the quality and usefulness of monitoring.

I hope this paper provides a more balanced understanding and interest in Monitoring in the face of a growing preoccupation with trying to evaluate results, including and especially impacts. I’d like to believe that it could also help take advantage of a similar movement by focussing more on the element of taking into account and learning from the views of beneficiaries in assessing the value of investments in aid and how well they are delivered. Doing this should be treated as an integral element of monitoring.

I am a ‘fan’ of logframes and value the need to develop results-chains. The major strength of the approach is that it provides an opportunity to collect evidence and think through a programme’s theory of change.

However, it is important to distinguish between the logical framework – the matrix which summarises the structure of a programme and how this broken down among the hierarchy of objectives – and the approach – the process by, and the evidence with which, this is defined. With this in mind, my qualms are about how logical frameworks are easy to be: a) mis-used through being developed without adequate participation of all stakeholders, not balancing both logical thinking and deeper critical reflection and organisations filling in the boxes to receive funding; and b) mis-managed by not being an iterative reference point for programmes that keep up to speed with the realities through providing opportunities for beneficiary assessments. There is nothing intrinsic to the process associated with developing logframes that explains the need for a separate approach built around theories of change.13

Currently, M&E processes and systems in public sector development aid at higher levels (Outcomes & Impacts) tend to be over-prescriptive and focussed on measuring pre-defined indicators within politically defined time periods – ie, elections. The really challenging questions are not how to do better monitoring, but rather (a) what are the

10 As with Michael Quinn Paton’s view on utilisation focussed evaluation, the bottom line objective for monitoring is how it really makes a difference to improving programme performance so as to enhance prospects for bringing about lasting change.

11 “…….an approach to information gathering which assesses the value of an activity as it is perceived by its principal users; . . . a systematic inquiry into people’s values and behaviour in relation to a planned or on-going intervention for social, institutional and economic change.” Lawrence F. Salmen, Beneficiary Assessment: An Approach Described, Social Development Paper Number 10 (Washington, D.C.: World Bank, July 1995), p. 1.

12 Project Monitoring and Evaluation in Agriculture by Dennis J. Casley , Krishna Kumar 1987. Johns Hopkins University Press.

13 http://web.mit.edu/urbanupgrading/upgrading/issues-tools/tools/ZOPP.html


bureaucratic pressures that lead to civil servants behaving in certain ways and (b) how to change them. 14 Typically, political time periods of five years ‘force’ over-ambition and therefore the premature measurement of developmental results. The systems civil servants are obliged to set up are limited in providing information which can help to:

A. Damp down politically-inspired over-ambition regarding outcomes and especially impacts that may inadvertently undermine the case for development aid;

B. Safeguard against the Law of Unintended Consequences (or at least illuminate where these are happening through testing the assumptions during implementation); and

C. Take account of alternative views (“theories of change”) especially those of beneficiaries and field staff regarding the quality and relevance of their support in order to help ensure the delivery of results – the true purpose of monitoring.

This can be accomplished by establishing Feedback loops based on beneficiary perceptions of the quality of project/programme services and ‘products’ (beneficiaries may be the general population and/or local institutions) and their ‘results’. These in turn require:

1) Opportunities to encourage often poor and vulnerable beneficiaries and front line staff to express their views;

2) Sufficient real-time flexibility in project/programme design to permit incorporation of feedback;

3) Commitment by managers and those responsible for the oversight of implementation to monitoring programme consequences, intended, positive or otherwise; and

4) Assurances by those with authority to allocate resources at all to validate feedback among beneficiaries and then incorporate remedial actions in projects/programmes.

The rationale of this paper is to explain some of the reasons why monitoring does not, yet could with effect and at reasonable cost, do the following:

1. Make effective contributions in delivering significant development results that matter most to beneficiaries; and

2. Better understand the ‘theory’ underlying aid programmes through monitoring processes and establishing feedback loops, in real time, with beneficiaries.15

14 Pers Comm Simon Maxwell15 This paper uses the term beneficiary in a collective sense: in relation to either the poor themselves

(for aid programmes that deliver support directly to them); or the institutions that serve them (for programmes that support for example, partner country ministries, NGOs and markets, formal and/or informal).


1. INTRODUCTIONThe importance of demonstrating Results and Value for Money has special resonance today. Aid budgets are now subject to much more scrutiny and transparency than ever before.

However, being too results-focussed within short time frames has potential drawbacks. There is now a risk that we will restrict aid to short term, measurable indicators that limit understanding how and to what extent it strengthens processes of institutional and economic change that precede and outlast donor interventions. Additonaly, measures of success that are forecast from the outset rarely involve the views of beneficiaries. Moreover, unexpected outcomes, good or bad, can matter almost as much as what programmes themselves are intended to do or achieve. We need to harness them as effective contributions to achieving change.

I am a ‘fan’ of logframes and value the need to develop results-chains. The major strength of the approach is that it provides an opportunity for stakeholders to collect evidence, think through and develop a programme’s theory of change. Part of the reflective thinking is about recognising that there is seldom one path. This is one obvious limitation in having to get a consensus on what is the best bet. A further limit is that a logframe approach can only cope with the anticipated impacts and does not take into account unanticipated consequences.

It is important to distinguish between the logical framework – the matrix which summarises the structure of a programme and he hierarchy of objectives – and the approach – the process by and the evidence with which this is defined.

My qualms are about how they are easy to be: a) mis-used through being developed without adequate participation of all stakeholders, not balancing both logical thinking and deeper critical reflection and organisations filling in the boxes to receive funding; and b) mis-managed by not being an iterative reference point for programmes that keeps up to speed with the realities through providing opportunities for beneficiary assessments. If Theories of Change can help others deepen their understanding, improve the process, explain how and why change will come about then this is all for the good, but the need for a separate process is dubious.

A. WHAT ARE RESULTS?

Development aid describes its effectiveness in terms of “results”. This covers “outputs”, “outcomes”, and “impacts”. These terms are used to tell a story often explained through what is called a results chain or Logframe. (See Diagram 1)

In simple terms: 1) outputs define the support programmes make available, in collaboration with beneficiaries, that are within their manageable control; 2) outcomes define the behavioural changes ‘stimulated’ by these outputs among the beneficiaries - and often others - that are obviously not in the control of the programme; and 3) impact refers to the contribution of the consequences of such behavioural changes typically defined in the context of national development plans and/or Millennium Development


Goals. Equally importantly, for M&E, is the differing nature of assumptions made concerning how and on what basis these changes are expected to unfold over time.

Diagram 1 A Basic breakdown of the term ‘Results’ and the importance of assumptions

There has been a resurgence of interest among donors in measuring the results of development aid as the upshot of the push for greater transparency. Over the last few years this has bought about a profusion of intellectual and institutional support on evaluation.16 However, what has been neglected in this growing focus on evaluation is monitoring – both in terms of interrogating assumptions made about the programme, and in providing feedback loops to provide beneficiaries (and frontline staff) with a meaningful opportunity to give input and shape the programme. In this context the main feature of feedback loops is to understand beneficiaries as individuals, as groups or as parts of institutions and as members of social and economic organisations that are in the development process for the long run.

B. WHAT ARE THE PRACTICAL DIFFERENCES BETWEEN MONITORING AND EVALUATION?

Often, the discipline of distinguishing one from the other is lacking. Although opinions differ as to what extent Evaluation is independent of and/or relates to Monitoring, I find it useful to define the main source of differences according to: a) the responsibilities and primary users of the information generated; b) their objectives; c) their requirements for comparative analysis (across people and space); and d), their reference periods. (See Table 1)

I see monitoring as having three inter-related parts:

one that is about controlling expenditures in the context of cataloguing activities;

16 Impact evaluation is a specific type of evaluation that tries to attribute the difference made on the lives of the poor or the performance of an institution by comparing what actually happened and what would have happened in the absence of the aid programme.


Assumptions that predict how beneficiaries are able

to and will respond

Assumptions that predict how lasting changes in these behaviours will

trigger sustained developmental processes

and benefits

another that tracks and analyses the support (ie, outputs) to beneficiaries these activities make available; and a third

that gauges how and to what extent beneficiaries respond to this support – their assessment of its quality, relevance and ultimately usefulness – and how this varies among them.

Of course evaluation is interested in this type of information as well. It is simply that it looks at these aspects over a longer reference period (say every two years), often compares this with other similar programmes and is typically done independently and usually commissioned by the donor or the country partner. Some evaluations focus more on certain criteria than others.17 Impact evaluation goes further: it tries to answer the question ‘how much development would have happened anyway in the absence of the programme/project?’ through establishing a valid counterfactual.

Table 1 What’s the Difference?

Source of Difference

Monitoring Evaluation

Responsibility and primary

users

A collaborative effort between the finance and delivery teams across the three parts of a monitoring system.

Programme Management Team

Usually done by an independent team. Donor and Partner Country Institution

Objective To improve performance based on analysing information about financial control, output tracking and beneficiary assessments.

Either proving and/or improving based on a range of different types (mid-term or process reviews and end of programme and impact evaluation), approaches and methods.

Requirements for

comparative analysis

Expenditure in relation to activity based budgets; the distribution of outputs according to targets/milestones; and differences among beneficiary groups on their assessments on the quality and relevance of the outputs and associated changes in their behaviours.

Will vary depending on the objective: for mid-term reviews it would focus on benchmarking performance with other similar programmes, with a focus on outputs and their relationship with outcomes; for end of programme and especially impact evaluation it will include comparisons across people and places to establish a valid counterfactual focussing on outcomes and impact.

Reference Period

Varies: for expenditure it is a matter of monthly comparisons; for outputs it is likely to be quarterly; for the assessments of quality and relevance of outputs by beneficiaries around every six months.

Typically two years, but for impact evaluation, if to assess lasting outputs and outcomes as well as ‘impacts’, it should be at least two years after programme closure.

17 In development aid, evaluations are built around their relative interest in the relevance, efficiency, effectiveness, impact and sustainability of a programme. Some also assess the added value of the programme and the extent to which it is coordinated with other similar programmes and harmonised with the partner country’s own systems.


2. THE STARTER PROBLEMDevelopment aid programmes are delivered in complex and highly unpredictable environments and thus are associated with, and subject to, all kinds of ‘jinks’ and ‘sways’. These are often overlooked and/or under-estimated in how they influence the results aid programmes seek and how they are monitored and evaluated.

As Rondinelli stated, the way programmes are designed and monitored sits uncomfortably with such complexities:

“the procedures adopted for designing and implementing aid interventions often become ever more rigid and detailed at the same time as recognising that development problems are more uncertain and less amenable to systematic design, analysis and monitoring..” .18

Nevertheless there are ways with which good monitoring can help managers better understand the inter-dependencies and multi-causal factors associated with programme outcomes. In general terms, these boil down to providing opportunities and discretion for beneficiaries to comment on the quality and relevance of the support on offer during implementation. However, these are not being adequately pursued for three broad reasons: 1) the belief that a robust evidence base will increase likelihood of success ; 2) belief that the perceived challenges relate more to evaluating the impacts, rather than the processes that bring these about; and 3) the term ‘implementation’ understates the complexity of the tasks involved in carrying out aid programmes that deliver the results.

Firstly, an apparent belief that improvements in programme design, which are informed by a more robust evidence base and rigorous ex-ante evaluation, will deliver some form of truth.

Recent guidance from the UK Department for International Development (DFID) rightly highlights how the appraisal should assess all costs and benefits, including all expected benefits that can be articulated now and observed post implementation and not just the benefits that can be quantified. As the guidance highlights, such changes take time to be observed and this should inform sensibly defined time periods to measure them. 19 It is of no less importance in knowing over what time period the outcomes and impacts are But, too often people feel obliged to measure the goal or the impact by the end of implementation periods rather than accepting that changes may not be discernible within this timeframe.

Cost benefit analysis has returned. However, in a review of rural development experience in the early 1980s, Johnston and Clark argued that cost-benefit analysis was only applicable to a very small number of routine and tactical problems.20 They concluded they were of little value in informing strategic and decision-making debates concerning choices over nutrition, family planning, health and agriculture programmes.

18 Development Projects as Policy Experiments. An Adaptive Approach to Development Administration. Development and Underdevelopment Series. D. Rondinelli. Methuen and Co Ltd 1983

19 Timing and Duration of Exposure in Evaluation of Social Programs, World Bank Research Observer 24(1): pp55-82. E. King and J. Berhman.

20 Redesigning Rural Development: A Strategic Perspective. B. Johnston and W. Clark. John Hopkins University Press. 1982.


Of course, there is the need to adequately design aid programmes based on a rigorous analysis of sufficient evidence. However, the consequences of these beliefs have been formulaic and overly complicated approaches to Value for Money21 coupled with pressures to monitor ‘impact’.22 Both these drivers, I argue, overshadow the need to monitor other, more important aspects in more flexible ways.

Charles Edwards Deming highlighted in the 1950s, in the context of the Japanese car industry, how “the most important things are unknown or unknowable."23 He argued that these ‘unknowns’ in trying to help Japanese car manufacturers understand the nature of United States demand for cars, can often have the greatest impact, long term, and be quite surprising. In development, like business, these are impossible to predict at the outset. Hirschman similarly concluded that if planners of World Bank projects had been able to anticipate all of the problems that later befell them, few of the projects would have been judged feasible or met with board approval.24 Robert Chambers has also made a similar point:

“Development is not like a Swiss train journey…..It is more like being in a boat at sea and trying to fish. The weather changes, the tides and currents vary, the waves come from different directions, the boat is blown about, and where the fish are and what sort they are, differs constantly.”25

Although I am not arguing against having a timetable or a logframe, there are several examples of how programmes mis-use them either by a poor process in developing them, confusing basic differences between an output and an outcome, and/or by sticking to them too rigidly. In some cases over-ambitious objectives coupled with conflating the development process in the results chain can lead to inadvertently undermining the capacity of local institutions.

As recently commented on by Lant Pritchett and others

“………….this leads to premature load bearing, in which wishful thinking about the pace of progress and unrealistic expectations about the level and rate of improvement of capability lead to stresses and demands on systems that cause capability to weaken….”26

Secondly, and following on from this belief of ‘getting it right’ at design is how the perceived challenges relate more to evaluating the impacts. As an aside, most DFID business cases are developed without having carried out a baseline survey. Many Terms of Reference (ToR) do not appear to ask certain questions as to the empirical basis on which specific design factors of programmes were informed.27

21 http://www.dfid.gov.uk/r4d/pdf/outputs/mis_spc/60797_itad-vfm-report-dec10.pdf 22 http://www.clp-bangladesh.org/outcome.php23 Out of the Crisis. Deming, W. Edwards (1986).Massachusetts Institute of Technology Press24 Development Project Observed. Brookings Institution. A. Hirschman. 196725 ZOPP Marries PRA? http://opendocs.ids.ac.uk/opendocs/bitstream/handle/123456789/285/

rc467.pdf?sequence=126 Capability Traps? The Mechanisms of Persistent Implementation Failure. Lant Pritchett, Michael

Woolcock, and MattAndrews. Centre for Global Development Working Paper 234 December 201027 After all, the purpose of baseline surveys is more than simply establishing the current values of

indicators: they are also a useful way of learning about the specific context in which the programme


http://www.clp-bangladesh.org/publication.php

http://www.dfid.gov.uk/r4d/pdf/outputs/mis_spc/60797_itad-vfm-report-dec10.pdf

These perceived challenges to evaluating impacts overshadow real opportunities to help realise these impacts through a better understanding of what beneficiaries value and need, the environments in which they live and how they respond to the support.

“There is a fundamental disconnect between the rhetoric about the need for learning in development and the reality of the monitoring procedures that funding agencies require.”28

Indeed, most ToR make scant reference to monitoring if at all beyond reporting and asking for an example of a reporting format. There is also evidence to suggest that monitoring even plays a subservient role to evaluation:

“Will the baseline data and monitoring strategy provide the data necessary to answer the evaluation questions?”29

Many argue how ‘conventional’ monitoring is just about cataloguing information from inputs-activities-outputs. Evidence supporting this claim is found in the outputs of such efforts: periodic progress reports that are driven by justifying the last six months for reasons of upward accountability. Such practice however, risks limiting opportunities for beneficiaries to shape programme performance.

© Dazed and Confused and Daniel Ticehurst ‘94

Recent opportunities to ‘improve’ monitoring are associated with ways in which to broaden its scope of enquiry at a higher level in the log frame. In many cases this has restricted monitoring to track pre-determined outcome indicators30. There is, however, the danger of

will work.28 Seeking Surprise: Rethinking Monitoring for Collective Learning in Rural Resource Management.

Published PhD Thesis, Wageningen University, Wageningen, The Netherlands Irene Guijt. 200829 Writing a Business Case. How to Note. DFID August 2011.30 In Booth and White’s analysis of DFID’s Country Strategy Papers in 2000 and Booth and Lucas

(2002) in relation to PRSP indicators both papers argued that the discussion of indicators needs to


an increasing irrelevance of information and subsequently the isolation of those responsible for collecting it. This is not an observation on approach (participatory or otherwise), rather the usefulness of the information – being clear on why it is being generated and whose decisions it needs to inform. Collecting all conceivably useful data creates confusion, often results in information graveyards and may inhibit managers from learning.31


During implementation, I contend that information attempting to measure the relative values of (socio-economic) impact indicators is of limited usefulness for those responsible for implementation. If programme managers oversee those responsible for collecting such data, it is likely they will become increasingly divorced from the programme.

“If the voyage is to be successful, the methods used to formulate and carry out development activity must encourage the process of discovery and use information and experience to chart a course of action along the way.”32

Often this type of tracking of outcome indicators can be interpreted as subservient to impact evaluation. Such ‘blind’ tracking sometimes means the process misses capturing more useful information. It is only recently that guidance, originating from GTZ in the early 1990s and now being followed by DFID, for example33, requires monitoring of the assumptions. I believe that doing this is of no less importance that monitoring the indicators themselves. Outcome mapping is not supposed to be a proxy for an impact evaluation that uses a valid counterfactual. It is a tool for managers who wish to improve

be driven by a discussion of strategy. This demands a realistic view of policy processes, which treats PRSP monitoring not as a technical exercise but as a fundamentally political one, and accepts the need for a systematic and rigorous handling of all the steps required to reach specific PRSP goals. They warn against over-reliance on indicators and argue for broader relevant information, including proxy indicators for policy design and implementation.

31 Managing Rural Development: Lessons and Methods from East Africa. Institute of Development Studies. R. Chambers and D. Belshaw. 1973.

32 Op cit, Rondinelli, 198333 How to Guidance Note on Logframes, DFID August 2011.


the relevance and quality of development operations - the purpose of monitoring. Outcome mapping is an approach, similar to Beneficiary Assessments (BA)34, to information gathering which to assess, in the case of BA the value of an output as it is perceived by its principal clients or beneficiaries. The approach is qualitative in that it attempts to derive understanding from shared experience as well as observation, and gives primacy to the centrality of the beneficiary’s point of view.

Thirdly the term ‘implementation’ understates the complexity of the tasks involved in carrying out aid programmes. Current thinking also appears to assume that achieving outcomes and impacts is relatively unproblematic. All that is needed is a robust, evidence based design and an equally credible attempt on how to evaluate outcomes and impacts.

However, we know that often the problems experienced by those responsible for managing the implementation of aid programmes is often not their fault, “but repeated implementation failures across an array of activities are not mistakes, but the visible manifestations of failure in the underlying theory of change.”35

Civil servants who oversee programme implementation are under pressure to conform to preconceived designs and bureaucratic procedures, which constrain them from dealing with unanticipated difficulties when they arise. They are often inadequately sheltered from political pressures to either: a) replicate programmes in other areas or extend their life in the same area too soon (very typical in development aid); or b) abandon them too early (very rare).

The purpose of monitoring is presented as a means to support implementation. However, intellectual and institutional support to monitoring is often lacking. For example. the Big Push Forward, a movement that offers space for discussion, debate and the exploration of appropriate approaches for assessing transformative development processes, does not explicitly mention monitoring among its clusters, although ‘Reporting’ and ‘Value for Money’ are included alongside others including Theory of Change and Impact Evaluation.36

3. THE VALUE OF MONITORING IN UNDERSTANDING BENEFICIARY VALUES

The purpose of monitoring should be driven by providing information that helps ensure aid programmes deliver lasting results. It should be a tool for managers to help them deliver programmes and projects, based on a better understanding of how beneficiaries (and so too front line staff) value and assess the support and to what extent they respond so as to help achieve the results described above. Understanding these responses, with a focus on outputs, assumptions and outcomes (not impact itself) is a means to achieving this purpose.

In the last twenty years there has been real improvement in approaches to understanding more about those who are intended to benefit from development aid. In the 1980s words 34 Toward a Listening Bank: A Review of Best Practices and the Efficacy of Beneficiary Assessment.

Lawrence Salmen. Social Development Paper Number 23 September 199835 Op cit CGD Working Paper 234 December 201036 http://bigpushforward.net/


such as ‘participation’, ‘listening’ and ‘subjective opinion’ were not important parts of the vocabulary in the development community. Although ‘participatory approaches are often limited to involving beneficiaries and not frontline staff, effective listening to beneficiaries can and does bring about changes they desire. By appreciating people’s values and perspectives Programme Leaders and managers can deliver support that brings about lasting changes of processes beneficiaries see as being in their own interest.37 I argue this listening should be an integral part of standard practice in monitoring.

There have been repeated occasions when support has been given to improve various aspects of M&E. Efforts from 1987 (Beneficiary Contact Monitoring and Beneficiary Assessments), 1997 (Participatory M&E) 2001 (Results Based Monitoring), 2003 (Citizen and Community Scorecards) 2010 (Results Oriented Monitoring and mention of beneficiary feedback systems) and, more recently, 2011 (Story Telling typically associated with Sensemaker38) are notable examples. However, compared with evaluation, there remains a relative lack of support, afforded to monitoring. Perhaps it is because, in development circles, monitoring is still considered routine and fact-oriented, limited to cataloguing and tracking progress to reduce deviations from workplan targets.

Increasing amounts of development assistance are going to fragile environments and conflict affected communities, where the potential for change is likely to be the greatest. As Martine Zeuthen commented on, however, in these places investments made in trying to understand and achieve outcomes and impact are also likely to be the most challenging. Monitoring in such environments is ever more important: the context influences the programme even more than in other relatively more stable environments. From both a learning and accountability perspective the main job of monitoring is to help inform why and how a programme can re-adjust in rapidly changing contexts. The tools to do this need to include contextual indicators and conflict mappings that assess continuously how the context is influencing the progress of the programme. In such environments aid programmes truly are policy experiments and effective monitoring could make a significant difference.

Significant investments are also being made in the area of climate change. One of the challenges facing Low Carbon Development interventions, for example, is that, alone, they will not achieve the envisaged transformation that motivates the decision behind such investments seeking to reduce CO2 emissions. The impacts of existing country policies, public/private sector behaviour and other donor activities will also have a key influence on the outcome. There is a need to enrich the understanding of these behaviours as an integral part of the monitoring process. As noted by 3ie in its systematic review of Impact Evaluation and Interventions to Address Climate Change:

“If the reduction of carbon-emitting behaviours is known one can to some extent rely on technical coefficients to calculate carbon reductions from practices like kerosene consumption and air travel. But as behavioural changes and take-up of new technologies are difficult to assess…………. theory based evaluation can improve our understanding of these processes.”

37 Bridging Diversity: Participatory Learning for Responsive Development. Lawrence Salmen and Eileen Cane. World Bank 2006.

38 David Snowden, a Welsh cognitive scientist and founder of a UK based firm called Cognitive Edge, uses this software.


4. THE NEED FOR FEEDBACK LOOPS

“The main conclusion is that as would-be change-makers, we should not try to design a better world: we should concentrate on building better feedback loops.”

Owen Barder (http://www.owen.org/blog/4018) 2010

Trying to assess 'profound' and lasting developmental impacts, in the absence of effective feedback loops, makes for a rather academic exercise rather than one that is tangible, practical and useful.

As Barder highlights, and as a recent Harvard Business Review makes explicit, approaches to gauging client feedback, through feedback loops are undeveloped for two reasons:39

Either beneficiaries and institutions are simply not asked for their opinions and, if they are,

The beneficiaries’ response to the performance of those providing support or services is seldom validated by them and/or fed back in the form of remedial actions. So why bother providing feedback in the first place?

The importance of understanding client or customer satisfaction, and indeed that of staff, was highlighted through the balanced scorecard developed by Kaplan and Norton and in 1994. 40 In the corporate world, realising that customer retention is more critical than ever, companies have ramped up their efforts to listen to customers. Some companies are addressing that challenge by creating feedback loops that start at the front line and help empower staff responsible for delivering services.

Rick Davies shared with me an exciting initiative by Pfizer to radically change the approach it used to better communicate with its sales force.41 Pfizer wanted to learn about the day to day real life experiences of its sales force, not its customers or beneficiaries, through generating different perspectives on that experience, assessing their relative strengths and weaknesses and exploring their their perceptions of the sales environment they worked in. It was based on a technique called ‘anecdote circles’ that drew out stories from lots of groups in ways that did not allow particular voices or viewpoints to dominate the group. The ‘circles’ were seen as a social, informal event with a formal information collection system within it. The main outcomes were that better results were realised through their own actions as informed by insights no other method would have provided. It was also a very cathartic process. They took actions themselves, rather than waiting for the results of a survey to cure their problems, and these were achieved at no extra effort or cost.

39 “Closing the Customer Feedback Loop”, By Rob Markey, Fred Reichheld and Andreas Dullweber, Harvard Business Review, December 2009.

40 The Balanced Scorecard, developed by Robert Kaplan and David Norton in 1994, is a performance management tool used by managers to keep track of the execution of activities by the staff within their control and to monitor the consequences arising from these actions. Its balanced nature is how it is based around four perspectives: Financial (how do we look to shareholders?), Customer (how do we look to our customers?), Internal Business Process (What must we excel at?) and Learning and Growth (How can we continue to improve and create value?).

41 Using Stories to Increase Sales at Pfizer. Nigel Edwards, Communications Director, Pfizer Business Unit. Strategic Communications, Volume 15, Issue 2. February/March 2011. Melcrum Publishing.


http://www.owen.org/blog/4018

I believe that establishing better feedback loops is an important means for changing the world for better. In other service industries understanding feedback is not a problem. I illustrate this with a hypothetical example.42 If the JAMMIN' JAVA CAFÉ in the Defence area of Lahore in Pakistan is offering a better combination of service, range of food etc, then more customers will congregate there than at the café next door. Often this next door café will try and imitate JAMMIN’ JAVA or close down.

In the public sector it is obviously different and not so straightforward. While the JAMMIN’ JAVA example explains how a market provides a short feedback loop, the feedback loop is longer and more fragile in the context of development aid. Providing opportunities for those most vulnerable (notably the extreme poor, women and ethnic minorities) with space to voice their opinions inadvertently challenges prevailing norms. While it is their right to voice their subjective opinions on the performance of others, they often lack the capacity or confidence to respond, through fear of repercussions.

In support to community action projects in northern Uganda in 1994, I provided advice to the Community Development Resource Network on approaches to what they called community based M&E. This entailed a process where the community members developed their own signs with which to assess the performance of those running community based projects and identified ways of engaging wider members of the community in discussing performance and change. Initially, though this was difficult. Children would not respond to these opportunities even in separate focus groups as initially suggested. It was only after having spent time with the parent committee and the teachers that opportunities for the children became apparent and acceptable and these were in the classroom and gave equal opportunities for boys and girls to engage.

This concern of finding ways to ‘voice’ opinions can be particularly so if beneficiaries are required to speak up in public. It is not simply the lack of opportunity that prohibits their participation, rather the types of opportunity facilitators make available, which fail to adequately take into account community dynamics. Feedback loops thus need to appreciate cultural dynamics.

My understanding of effective feedback loops is that they focus on learning about the preferences, responses and behaviours of beneficiaries and other stakeholders. Lessons I learnt in designing an M&E framework for a World Bank financed Village Infrastructure Programme in Ghana were based on synthesising participatory M&E practice on social and economic infrastructure interventions amongst NGOs such as Trend, ActionAid, Technoserve and Community Self-Help. The object was to advise on the development of a framework that provided opportunities for beneficiary communities to monitor the quality and relevance of services provided by an EU funded ported Microproject Management Unit (MPMU) to support district departments up to and after construction of social infrastructure. The findings generated recommendations concerning community generated indicators and arrangements needed to assess the district departments’ performance against baselines in: 1) supporting the Community Implementation Committees (up to construction); and 2) supporting the Parent Teacher Associations (post construction) in operating and maintaining the primary schools (in this instance) and gauging feedback from parents.

42 This example is modified from that provided in ADAPT – “Why Success Always Starts With Failure”- with kind permission from Tim Harford.


In reference to feeding back the results of development aid to UK taxpayers (ie, one important shareholder of aid), recent research by Institute of Public Policy Research (IPPR) to establish whether voters’ attitudes to aid are quite as simplistic and unchanging as some assume when they go on about proving impact and value for money, reveals some interesting findings.43 Based on a series of consultation workshops the results showed a clear appetite for a richer understanding of how aid is used and how development takes place.

“In general, this seems out of step with the current development communications’ focus on either inputs (such as ensuring aid spending reaches 0.7% of GNI) or outputs (such as simply listing the numbers of people educated or vaccinated as a result of aid). Participants at the deliberative workshops were most engaged when they heard progress stories and were intrigued to know more about how change had happened (even when this was complex). For example, hearing ‘good news stories’ about the progress of countries like Vietnam and Brazil in addressing hunger sparked a positive interest in many participants, leading to questions about how this had been achieved.”

As the UK government moves towards the next election and/or the next expenditure review, it will be interesting to see how and to what extent it responds to this feedback.

5. THE IMPORTANCE OF INSTITUTIONSIf these feedback loops are associated with effective two way communication processes, they can enhance prospects for achieving impacts that can outlast the duration of the programme/project. I believe that the lasting ‘success’ of interventions depends on the performance of the institutions in being able to offer a quality service in the long run.

“Growth and prosperity are unlikely to be maintained if the institutions which guide them are dysfunctional. And if development is to be accompanied by poverty reduction, those organisational competencies and values must also be attuned to the needs of the poor.” 44

This observation, originally made by Salmen, has more recently informed, albeit by coincidence, one of the four principles - sustainable change - underpinning the Making Markets Work for the Poor (M4P) approach. That is in addition to its other features that build on the ZOPP methodology in developing logframes set out in the late 1980’s. Rather than treating the performance of these institutions as assumptions for aid programmes that delivered services directly to the poor, approaches like M4P define these assumptions as the actual outcome. In other words, the clients or direct beneficiaries of such programmes are these institutions, not the poor.

Salmen’s work informed the advice I gave in supporting a DFID funded crop-post harvest programme from 1999-2002. The programme’s original purpose was defined in reference to poor people benefitting from the research programme. The organisations with which the programme engaged such as community based organisations (CBOs), NGOs, private

43 UK public attitudes to aid and development: Understanding public attitudes to aid and development Alex Glennie, IPPR Will Straw, IPPR Leni Wild, ODI. June 2012

44 Reducing Poverty: An Institutional Perspective Poverty and Social Policy Series Paper No. 1 Lawrence F. Salmen 1992.


companies and government research and development institutions and their capacities were treated as assumptions, if not risks to the programme’s impact! “Needs assessments” seem only to be conducted with end-users (farmers); rarely are intermediate partner organisations included in such exercises. All this does is reflect the understanding among research teams that the primary clients of research outputs are exclusively and immediately poor communities.

This could be one reason why there remains a lack of confidence in the results of impact studies that use rates of return as a measure of ‘success’ or as means to attracting funding.

"If agricultural research is such a good investment, i.e. yields very high rates of return, why the slow growth of the agricultural sector in Africa given the amount of money invested?" 45

Participatory approaches are sometimes understood to be exclusively about ways of engaging with the poor. The Pfizer example illustrates how this is not the case. This latter point reflects a blurring of the distinction between: (a) changes achieved among a small sample of farmers participating in the Research and Development (R&D) process for validating the research and its outputs; and (b) development impact, which implies significant scale and requires promotion/replication by local organisations, private and public, that are in the R&D and extension process for the long run.

The programme subsequently placed institutions, private and public, which are responsible for networking the results of research at the centre. Monitoring efforts focussed on gauging their responses to and use of the programme’s research outputs. Such responses to the programme were re-defined as the programme’s purpose or outcome. That is as opposed to treating their behaviours as assumptions. In this context, validating research on farmer fields is a poor proxy on which assess the outcome or ‘success’ of the crop-post harvest programme within the programme’s funding period.

Developing feedback loops should be an inherent feature of monitoring processes. Without them, serious questions are raised about the basis on which programmes continue to operate and how the programme is interacting with the context, and if this is still suitable even if the context has changed. Such questions should not be held in abeyance pending an evaluation. Monitoring needs to be less than periodically surveying socio-economic impacts but more than cataloguing ‘output and activity’ and controlling ‘spend’.46 Its primary purpose should be to gauge and diagnose responses and preferences, many of which relate to the assumptions and behaviour on which projects were designed, Cost Benefit Analysis (CBA) was based and/or that theories of ‘change’ make explicit.

This paper does not see the problem as an absence of tools or methods, they are there: beneficiary assessments and outcome mapping are two examples. Rather it explains how these tools can better enable improvements in the quality of monitoring. I hope this

45 Alex Macalla, Keynote Address at a workshop on Impact Assessment of Agricultural Research in Eastern and Central Africa organized by the European Consortium for Agricultural Research in the Tropics (ECART) together with the Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) and the ACP-EU Technical Centre for Agricultural and Rural Cooperation (CTA). Uganda, November 1999.

46 Such surveys may need doing but not by those attached to programmes.


provides a more balanced understanding and interest in Monitoring in the face of a growing pre-occupation with approaches mixed, randomised or otherwise, to evaluation.

6. MAIN OBSERVATIONS OF CURRENT PRACTICE The following lists and comments on current practice are based on my experience and knowledge. I am aware that there may well exist practice that differs from or contradicts what I say. My main point is that there are useful and exciting options that point to ways in which to improve how monitoring listens to and responds to beneficiary views and responses. One main challenge lies in how best to select which ones best suit which particular purpose.


The current interest among some in the development community on the theory of change is surpassed only by some of the community members’ limited capacity to develop coherent and evidenced-based logical frameworks in the past. Indeed, I often ask, what is the difference between the method and product associated with a theory of change and a logframe? Too many of the alleged advantages of using a theory of change over a logframe is an apology for neglect; that is, as opposed to an intrinsic weakness of the logframe or its methodology (if used and used well).

Impact Evaluation is currently driven by donor/public demand to demonstrate causal relationships between outputs/outcomes/impacts. Because aid is the only business of development agencies, one can understand the reasons for them in encouraging their partner governments to take on Evaluation. Examples of this include those made by the


World Bank’s Evaluation Capacity Development Initiative, established in 1998. Recently it has embraced, rather unsurprisingly, impact evaluation. Such encouragement needs to include the incentives and capacities required to evaluate development aid among partner governments. These could be limited, and could distract from their main business……of running an entire economy/country. This is especially true for partner governments where aid only accounts for a small percentage of GDP.

This more recent demand for evaluation, among the donors anyway, was prompted by a paper called When Will We Ever Learn.47 The paper’s basic conclusion was that ‘persistent shortcomings in our knowledge of the effects of social policies and programs reflect a gap in both the quantity and quality of impact evaluations.’ The paper ascribes the evaluation gap to the public good nature of impact measurement. I, like James Morton, challenge this.48 Governments and development agencies may be relatively better at monitoring and process evaluations than they are at impact evaluation. However, that is not the same as saying they good at them.

Experience, for example in the case of agricultural development projects in the 1970s/1980s, tells us it is analytically impossible to establish significant causal relationships between the provision of services and movements in the relative values of crop yields by the end of implementation periods.49 Typically, and given relationship between aid programmes and development, such changes may take years to observe.

The use of Randomised Control Trials has gained recent popularity in development aid.50

The principal strength of this is that it minimises bias (the risk of being misled by systematic errors). The use of randomised or pseudo-experimental designs to overcome this problem has its own limitations. The vanity of rigour surrounding these approaches has been questioned and is well documented.51 Suffice it to say that in development aid there are real constraints in designing programmes as classic scientific or behavioural experiments in which differences by the end of implementation periods due to ‘treatment’ are assessed through establishing a ‘valid’ counterfactual.

Although in a different context, setting overly ambitious results and trying to assess the difference aid makes in direct relation to these is a habit some find hard to let go of. The Select Committee on International Development, in its eighth report on DFID - more than 10 years ago - raised a similar point.

47 When Will We Ever Learn: Improving Lives Through Impact Evaluation. WD Savedoff, R Levine & N Birdsall, et al Centre for Global Development, 2006.

48 Why We will Never Learn. A Political Economy of Aid Effectiveness. James Morton 200949 An Overview of Monitoring and Evaluation in the World Bank, Operations and Evaluation

Department, 1994.50 Randomised Control Trials, after assessment of eligibility and recruitment of programme

beneficiaries, but before programme implementation begins, involves randomly selecting those who will and will not receive support. Random allocation is complex, but the idea is like tossing a coin. After randomization, the two groups are followed in exactly the same way, and the only differences between the support they receive for example, in terms of training, cash transfers, advice should be those intrinsic to the support being compared. The most important advantage of proper randomization is that it minimizes allocation bias, balancing both known and unknown factors among the different types of support.

51 For example: 1) Randomised Control Trials for the Impact Evaluation of Development Initiatives: A Statistician’s Point of View Carlos Barahona. ILAC Working Paper 13 2) http://findwhatworks.wordpress.com/2011/04/13/limitations-of-rcts-part-3-how-context-influences-program-execution/; and 3) Pawson, R. and Tilley, N. (1997) Realistic Evaluation, London: Sage.


http://findwhatworks.wordpress.com/2011/04/13/limitations-of-rcts-part-3-how-context-influences-program-execution/

http://findwhatworks.wordpress.com/2011/04/13/limitations-of-rcts-part-3-how-context-influences-program-execution/

“It is our view [……] that measuring DFID's performance as a Department by relating its expenditure at country levels to specific development outcomes in a disparate set of countries is so simplistic as to be uninformative and useless.”

This habit inadvertently can, does and may well again undermine the case for development aid.

At Project level, Casley and Kumar warned against such over-ambition. In Casley’s review of M&E training materials I developed for CARE Uganda in 1994, he repeated this message:

“References to baseline and terminal surveys reflect the need, but during the course emphasis will be needed on the difficulty of measuring change in socio-economic variables in a short span of years in establishing the significance of change and its attribution to aid interventions. Such surveys will not establish trends in statistically ‘noisy’ systems, but their imposition leads to expenditure of the major proportion of M&E effort on these surveys and unrealistic expectations from donor economists that a ‘rate of return’ will be calculable from these surveys.” 52

What Casley and Kumar stressed, above all, was to focus in on simple indicators of what they called beneficiary reaction. I see this type of monitoring not too dissimilar to outcome mapping.53 Outcome mapping makes clear the importance of defining the client group (or what they call boundary partners) - farmers in Casley and Kumar’s work or market players in M4P programmes –and understanding the programme’s sphere of influence.

If you read the guidance from the International Development Research Centre on Outcome Mapping you will, perhaps like me, question their claim that it is original. The definition in the footnote below is remarkably similar to beneficiary assessments developed some 13 years earlier, GTZ’s definition of a purpose in its logframe methodology developed in the early 1990s and adoption rate surveys that, if done properly, pursue a similar objective.54 This is not to say it is not useful, but serves to highlight how those interested in M&E each have their own understanding of history, just like I do.

Participatory Monitoring and Evaluation (PME) takes on many forms and is associated with a broad range of approaches, objectives and practice. Many researchers who were involved in the genesis of Rapid Rural Appraisal – Participatory Rural Appraisal and Participatory Learning and Action experiences wanted to profile M&E more clearly within the context of a project cycle. Hence the beginnings, for some anyway, of PME in the mid 1990’s. At its heart is an approach that seeks to empower beneficiaries, and initially

52 Project Monitoring and Evaluation in Agriculture by Dennis J. Casley , Krishna Kumar 1987. Johns Hopkins University Press.

53 “The originality of the methodology is its shift away from assessing the products of a program (e.g., policy relevance, poverty alleviation, reduced conflict) to focus on changes in behaviours, relationships, actions, and/or activities of the people and organizations with whom a development program works directly.” Sarah Earl, Fred Carden, Terry Smutylo /www.idrc.ca/booktique 2001.

54 “The purpose or outcome of a project needs to describe the behavioural change or action taken by the client or beneficiary group as measured by using skills or knowledge, organising something themselves or making something.” (GTZ ZOPP checklist for logframes, 1992)


through a facilitated process. My understanding is that much, but not all, practice such as Most Significant Change has an enthusiastic following in part due to it being understood to be an indicator-free way of understanding change. The process provides space for beneficiaries (and other stakeholders) to articulate and prioritise changes. Indeed, most PME is about understanding changes or impacts. More recent practice on other approaches to (participatory) story telling, share a similar focus. Such practice has the added benefit of using tools such as SMS to collect data and software to analyse and interpret the results (such as Sensemaker). As noted earlier, however, participatory approaches to M&E need not be limited to providing opportunities in facilitating opportunities and space for beneficiaries to articulate change per se.

A recent review of academic and practitioner literature explained how it sees the relationship between logframes, PME and feedback systems55. The authors describe how logframes meet the needs of senior decision-makers to summarise, organise and compare projects. PME meets the needs of field staff to work sensitively with intended beneficiaries and support their learning and empowerment. Feedback systems appear to link the two, providing performance data for managers and creating incentives for implementing staff to focus on their intended beneficiaries. Feedback systems, they state, build on the rich heritage of PME and are compatible with logframes. They may help provide a manageable and effective approach to accountability that links the means and the ends of development interventions. The authors claim that feedback systems are at an early stage of development and conclude there is a lot to learn about how and where they work best……………Issues that Beneficiary Assessments developed at the World Bank and client satisfaction surveys run by the private sector have been grappling with for at least 15 years! As the IDS Agricultural Learning and Impact Network states, current efforts to learn about and plan for impacts of agricultural projects are fractured and of variable quality.56 I would also add to this, with noise being made by some donors and the humanitarian sector generally on feedback loops, how some should not reinvent the wheel in trying to recast something old into different catchy slogans. While these are not necessarily needless, they lack efficiency.

An effort in late 2010 to map and synthesise experience and practice in M&E, prompted by the resurgence of interest in agriculture, introduced itself in the following way.

“ …the articles in this seminal (my emphasis) IDS Bulletin provide systematic evidence to lay open the widely shared secret among development practitioners that the cupboard of agricultural monitoring and evaluation (M&E) is bare. Agricultural M&E has been weak at best. Where it exists it has concentrated on tools and methods, a narrow focus on project performance ratings and ‘rates of return’ with accountability upwards to donors rather than downwards to the intended beneficiaries of programmes.”57

It concluded by arguing for what it calls a new ‘People-Centred Approach’ to M&E that demonstrates two main characteristics: a) to balances multiple accountabilities through

55 Three Approaches to Monitoring: Feedback Systems, Participatory Monitoring and Evaluation and Logical Frameworks. Alex Jacobs, Chris Barnett, Richard Ponsford. IDS Bulletin Volume 41, Issue 6, Nov 2010.

56 http://www.ids.ac.uk/go/idsproject/aline57 The Sorry State of M&E in Agriculture: Can People-centred Approaches Help? Lawrence Haddad,

Johanna Lindstrom and Yvonne Pinto. Institute of Development Studies, 2010.


greater participation in design, implementation and evaluation based on providing discretion and space for beneficiaries to comment on the performance of donors and those who provide services; b) to enhance greater organisational incentives for learning that provides opportunities for organisations to improve outcomes through a more inclusive approach to M&E. Groundbreaking? In light of the above claim I would also argue that many of the lessons and practices from some time back regarding agriculture and M&E appear to have got lost. In this context, I am referring to the 1987 Casley and Kumar publications and the Agricultural Knowledge and Information System (AKIS) series, among others. 58

There does not appear to be as much emphasis in PME on what I understand to be at the heart of beneficiary assessments – providing space and opportunity for beneficiaries to voice their subjective opinions on the performance of the programme or organisation in providing quality and relevant support. PME tends to focus, albeit very usefully, on change itself. Beneficiary Assessments were developed by the World Bank at roughly the same time as Rapid Rural Appraisal was shifting to Participatory Rural Appraisal. The difference was that beneficiary assessments were being used in the context of M&E before the advent of PME. It is also revealing how recent work through the Institute of Development Studies on feedback systems tries to identify how development agencies can implement feedback systems so as to hear systematically from intended beneficiaries: how can such systems improve aid accountability, and thus effectiveness?. 59 Attempts to answer the same questions were what drove Salmen’s work on Beneficiary Assessments nearly twenty years earlier, Casley and Kumar’s proposed Beneficiary Contact Monitoring in 1987 and what the business world has been working on since the early 1990s through adaptations of the balanced scorecard.60

In the context of Monitoring, approaches to and ways of ranking and scoring developed in the mid 1980s under the guise of Rapid Rural Appraisal offer powerful ways in understanding beneficiary preferences and views. Direct Matrix Ranking is particularly useful in assessing the satisfaction of beneficiaries concerning the performance of different service providers in primary health care or agricultural extension; and to assessing the relative strengths and weaknesses of multiple services (or outputs) from the same service provider (or programme). In my experience this can lead to instilling confidence among those who participate. More recently, these tools are more commonly used in community and citizen scorecards.61 While potentially useful, some scorecard practice may eliminate the value of candour (validity) somewhat by their emphasis on quantification.62

Monitoring is made complicated by ‘M&E experts’. They rarely challenge the strategic failings, objectives and targets that are arguably often too ambitious, very often blindly

58 Agricultural Knoweldge and Information Systems (AKIS) Good Practice Note: Monitoring and Evaluation for AKIS Projects. The World Bank Rural Development Family. December 2000 Prepared by Gary Alex and Derek Byerlee with inputs from the AKIS Thematic Team.

59 Jacobs, A., 2010, 'Creating the Missing Feedback Loop', IDS Bulletin, vol. 41, no. 6, pp. 56-6460 Project Monitoring and Evaluation in Agriculture by Dennis J. Casley , Krishna Kumar 1987. Chapter

4, pp 60-61. Johns Hopkins University Press.61 http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTSOCIALDEVELOPMENT/EXTPCENG/0,,contentMDK:20507680~pagePK:148956~piPK:216618~theSitePK:410306,00.html62 Pers com Lawrence Salmen.


measure them and offer up or sell on a preponderance of solutions and training courses. Best to keep it simple.

Beneficiary and institutional opinion on the ‘value’ they perceive from aid programmes, including their assessment of the programme’s performance to learn lessons, can get lost. Assessing Value for Money tends to obsess too much about price, procurement and fiduciary risk and attempt to integrate this with the quantum of outputs based around approaches to activity based costing63. This tends to take the focus away from managing the potential risks of development ineffectiveness, as do attempts in ‘quarantining’ indicators or specific groups of people for impact assessment purposes.

As Korten advocated, the more complex the problem, the greater the need for localised solutions and for value innovations. Such processes depend less on finding rationally ‘correct’ solutions rather than on finding ways to support beneficiaries of development aid in ways that are acceptable to them.

“The key to learning is not analytical method, but organisational process; and the central methodological concern is not with the isolation of variables or the control of bureaucratic deviations from plans, but with effectively engaging with the necessary participation of system members in contributing to the collective knowledge of the system”.64

7. CONCLUSIONS How and in what ways can monitoring be improved?

1. The ‘outputs’ of well-designed monitoring processes can contribute to the ‘results agenda’ through communicating the reach of outputs and outcomes (important pieces of the results ‘jigsaw’) and how these vary among beneficiaries. My proposed extension of the monitoring remit is essential to enhancing prospects for aid programmes to achieve lasting impacts. Attempts to assess impact are often done too soon after completion or after the extension has been approved…..

2. More important than the particular academic training of the person engaging with the intended beneficiary is empathy. Being ‘good’ at listening to and understanding how and in what ways beneficiaries respond to development programmes by rigorously analysing the facts and feeding these back to decision-makers should not be judged by academic standards.

3. Monitoring can help learn about behavioural responses to aid programmes among the poor and/or client institutions. Effective feedback loops will help promote ‘downward’ accountability.

63 Activity Based Costing (ABC) is a method for developing cost estimates in which the project/programme is broken down into discrete, quantifiable activities associated with each of its outputs.

64 Management of Social Transformation at National and Subnational Levels. Unpublished Paper, Manila. Ford Foundation. 1981


“Monitoring systems should be seen as a way in which institutional level learning can be systemised, shortening the feedback loop and being accountable to beneficiaries, not simply as a way to measure outcomes in order to justify prior investments in planning and implementation.” 65

4. There is useful reference to behavioural change and client satisfaction surveys – that point to the importance of providing periodic opportunities to ‘listen’ to beneficiary perceptions and preferences.

5. Experiences and lessons in development aid to listen to beneficiary perceptions and front line service providers should learn from similar initiatives in the business world.

6. It is important to monitor specific indicators and assumptions defined in theories of change, concerning how programmes affect and bring into effect systemic change. There are tensions between delivering direct support to generate quickly demonstrable ‘results’, that are vulnerable post programme completion, with supporting institutions who are in the development process for the long run.

7. Monitoring can help understand unintended effects by answering the question ‘whose theory of change is being monitored?’ Field staff, beneficiaries or other commentators/observers may have quite different perspectives.

Monitoring should ultimately drive better performance and involve participatory processes including, but not limited to, those between the intervention and intended beneficiaries. The principles of participation equally apply to knowledge systems of other stakeholders throughout the system, including those who deliver support to beneficiaries.

8. It is important to ensure that findings of assessments are shared with beneficiaries to close the feedback loop. Doing this will check their validity, promote accountability and help ensure what they say is acted upon.

In sum, the ultimate question for monitoring is: does the information it provides drive better performance? Processes need to:

1) Define opportunities for institutions and the poor to engage in theory of change processes, insights from which make explicit the change and the associated causal assumptions. Outcome mapping has much to offer in developing theories of change in ways that were usefully initiated by GTZ’s ZOPP methodology in the early 1990’s;

2) Provide space for institutions and the poor to periodically voice their subjective opinions on the relevance and quality of the services that aid

65 Performance and Impact Assessment: Emerging Systems Among Development Agencies. Ticehurst, D. & Cameron, C. Natural Resources Institute, Policy Series 8. 2001,


interventions make available, and to provide to feedback on the programmes’ performance and for management that is used as the basis for adjustment; and

3) Allow managers of programmes to develop inclusive ways of understanding how, and to what extent, the programme’s presence is influencing the behaviours of the institutions and their relationships with the poor. Putting to one side the predicted benefits made at design, if institutions and the poor judge the support to be of little relevance and/or poorly delivered, then this is not good value for money, irrespective of how economically sensible the support appears to be.

By amplifying the voice of the people for whom development is intended, monitoring can help empower them by giving them space to comment on the performance of aid programmes. By showing managers the value of improved communication with beneficiaries, this will encourage institutions and programmes to be more open and responsive to beneficiaries. The ultimate test of the worth of doing this is the creation of durable bridges of understanding between managers and beneficiary institutions and people such that those subject to change are empowered through being provided with a decisive voice in the management of the development aid process.


who is listening to who, how well and with what effect? by daniel ticehurst

Self Improvement