The use of evidence in achieving IRC’s mission

Background

• IRC’s vision and mission – action oriented – a think/do tank – a

change agent

• How does “evidence” fit into that? What is “evidence”?

• How does evidence fit into our vision of a learning and adaptive

WASH sector?

• What role do we have vis-à-vis collecting and using evidence?

A global movement around “evidence based approaches”

• Driven by a critique of endless cycles of implementation that don’t

seem to go anywhere

• Of ‘development by anecdote’; development by story-telling

• Move away from ideology-based interventions

• Rise of business informed ‘metrics’

• Rise of “big data” – which can be mined to identify patterns of

evidence

• Rise of ‘randomistas’

• Rise of ‘evidence’

Evidence?

• But what is evidence? And what role does it play in improved WASH services?

– Measurement of impact – WB DIME – supporting iterative learning.

Strengthening capacity to do impact evaluation

– Randomistas – Randomised Controlled Trials – the ‘gold standard’ of (medical)

research – physician envy?

• At worst – a reductionist approach that focusses on ‘the evidence’ rather than ‘the

intervention’ – looking at what works and what doesn’t and not at the underlying

causes and processes

• An approach that risks missing complex and non-linear relationships in

development ‘space’

• But …. a real opportunity for a knowledge broker … or a think tank. The question is

not ‘evidence: yes or no’ – but “what evidence – for what process”

A ‘development lab’/RCT approach

Fine in practice – but:• Where did the ‘idea’ come from?• Who owns it?• Who owns/actions the results?• What happens when it doesn’t work?

• The development lab is an international development lab

Intervention group

Control group

Any significant difference?

Our approach

• Often we do not know beforehand, which “innovation” to test – these need to emerge from the context

• Experiments are rooted into a country context

• Ownership of ideas is local (at least partially)

• Scaling happens because people ‘buy’ the idea

• But… in the testing there may be space for RCTs, or similar methods, as well as a range of other forms of

generating and using evidence

• Our development lab is a national development lab!

Accept the possibility of failure!

Evidence at multiple different scales and as part of multiple different processes

Evidence of problems: to helps kick-start a process

But.. difficult to have evidence of a concept beforehand

Evidence of successes elsewhere: to help stimulate ideas

Evidence from experiments/pilots: to feed into process

But also ….

Evidence that the whole process is functioning …….

Evidence that the whole process is leading to results: better longer lasting services.This is critical to IRC – it is proof that what we do …. works!

But what sort of evidence?

• Despite what randomistas say ….

– There is a huge range of evidence …. A broad evidence

spectrum

– From purely qualitative to hard quantitative

– From case studies to surveys/samples and …RCTs

– There is NO one gold standard. Evidence is context specific.

– Evidence is expensive

– Generating evidence must be subject to the same cost-benefit

analysis as any other intervention

Where is IRC’s strength?• Long history of qualitative/soft work – including case-studies

• Strong in mixed methods: quantifying the qualitative (QIS, MPA), but

also trials with text analysis, e.g. SenseMaker and Infolution

• Recently, rapid rise in more quantitative work linked to LCCA and service

delivery indicators

• Gaining experience at measuring outcomes of process work in a more

rigorous way (process documentation, SenseMaker – failed; QDA)

• But limited/no real experience of measuring impact (although new tools

present us with the means of doing so)

Qualitative QuantitativeMixed

Programme/process focused

Project/pilot focused

QDA

IRC reporting

Case studiesProcess

documentationQIS etc. LCCA,

SDI etc.

Country/district key metrics

Use of evidence: communicating evidence for change

• How to present evidence so that:

– It is problem/solution focused (not just descriptive)

– Provides clear suggestions for policy/practice

– Is comprehensible, useful and actionable by target audiences

• How to tell our story about evidence so that:

– We do not simply come across as defensive towards

randomistas (and fellow travelers)

– Demonstrate a real appreciation of the use of evidence in real

change processes

Discussion

• What type of (new) evidence is most important for IRC’s work

• What type of communication/medium is best suited to communicating

this?

• Where does IRC most need new capacity to develop and use evidence?

• Group into and discuss for:

– International (DP, INGO)

– National (Government: technocrats but also politicians and ministry

of finance)

– District/Local

Where does IRC need to develop new strengths?

• More trials with qualitative methods a la QDA and text analysis

• Quantitative methods for generating/interpreting evidence at

national, district, and project level

– Unlikely to do our own RCTs – but need to understand them and

their roll in wider evidence discussion

• Logic of EF is that district level is crucial

• But, communications also critical