the use of evidence in achieving ir cs mission
Post on 17-May-2015
209 Views
Preview:
DESCRIPTION
TRANSCRIPT
The use of evidence in achieving IRC’s mission
Background
• IRC’s vision and mission – action oriented – a think/do tank – a
change agent
• How does “evidence” fit into that? What is “evidence”?
• How does evidence fit into our vision of a learning and adaptive
WASH sector?
• What role do we have vis-à-vis collecting and using evidence?
A global movement around “evidence based approaches”
• Driven by a critique of endless cycles of implementation that don’t
seem to go anywhere
• Of ‘development by anecdote’; development by story-telling
• Move away from ideology-based interventions
• Rise of business informed ‘metrics’
• Rise of “big data” – which can be mined to identify patterns of
evidence
• Rise of ‘randomistas’
• Rise of ‘evidence’
Evidence?
• But what is evidence? And what role does it play in improved WASH services?
– Measurement of impact – WB DIME – supporting iterative learning.
Strengthening capacity to do impact evaluation
– Randomistas – Randomised Controlled Trials – the ‘gold standard’ of (medical)
research – physician envy?
• At worst – a reductionist approach that focusses on ‘the evidence’ rather than ‘the
intervention’ – looking at what works and what doesn’t and not at the underlying
causes and processes
• An approach that risks missing complex and non-linear relationships in
development ‘space’
• But …. a real opportunity for a knowledge broker … or a think tank. The question is
not ‘evidence: yes or no’ – but “what evidence – for what process”
A ‘development lab’/RCT approach
Fine in practice – but:• Where did the ‘idea’ come from?• Who owns it?• Who owns/actions the results?• What happens when it doesn’t work?
• The development lab is an international development lab
Intervention group
Control group
Any significant difference?
Our approach
• Often we do not know beforehand, which “innovation” to test – these need to emerge from the context
• Experiments are rooted into a country context
• Ownership of ideas is local (at least partially)
• Scaling happens because people ‘buy’ the idea
• But… in the testing there may be space for RCTs, or similar methods, as well as a range of other forms of
generating and using evidence
• Our development lab is a national development lab!
Accept the possibility of failure!
Evidence at multiple different scales and as part of multiple different processes
Evidence of problems: to helps kick-start a process
But.. difficult to have evidence of a concept beforehand
Evidence of successes elsewhere: to help stimulate ideas
Evidence from experiments/pilots: to feed into process
But also ….
Evidence that the whole process is functioning …….
Evidence that the whole process is leading to results: better longer lasting services.This is critical to IRC – it is proof that what we do …. works!
But what sort of evidence?
• Despite what randomistas say ….
– There is a huge range of evidence …. A broad evidence
spectrum
– From purely qualitative to hard quantitative
– From case studies to surveys/samples and …RCTs
– There is NO one gold standard. Evidence is context specific.
– Evidence is expensive
– Generating evidence must be subject to the same cost-benefit
analysis as any other intervention
Where is IRC’s strength?• Long history of qualitative/soft work – including case-studies
• Strong in mixed methods: quantifying the qualitative (QIS, MPA), but
also trials with text analysis, e.g. SenseMaker and Infolution
• Recently, rapid rise in more quantitative work linked to LCCA and service
delivery indicators
• Gaining experience at measuring outcomes of process work in a more
rigorous way (process documentation, SenseMaker – failed; QDA)
• But limited/no real experience of measuring impact (although new tools
present us with the means of doing so)
Qualitative QuantitativeMixed
Programme/process focused
Project/pilot focused
QDA
IRC reporting
Case studiesProcess
documentationQIS etc. LCCA,
SDI etc.
Country/district key metrics
Use of evidence: communicating evidence for change
• How to present evidence so that:
– It is problem/solution focused (not just descriptive)
– Provides clear suggestions for policy/practice
– Is comprehensible, useful and actionable by target audiences
• How to tell our story about evidence so that:
– We do not simply come across as defensive towards
randomistas (and fellow travelers)
– Demonstrate a real appreciation of the use of evidence in real
change processes
Discussion
• What type of (new) evidence is most important for IRC’s work
• What type of communication/medium is best suited to communicating
this?
• Where does IRC most need new capacity to develop and use evidence?
• Group into and discuss for:
– International (DP, INGO)
– National (Government: technocrats but also politicians and ministry
of finance)
– District/Local
Where does IRC need to develop new strengths?
• More trials with qualitative methods a la QDA and text analysis
• Quantitative methods for generating/interpreting evidence at
national, district, and project level
– Unlikely to do our own RCTs – but need to understand them and
their roll in wider evidence discussion
• Logic of EF is that district level is crucial
• But, communications also critical
top related