“Ideally, the World Bank and other development funders would require pilot programs and randomized evaluations before launching large-scale funding of new policies which are prone to evaluation, just as regulators require randomized trials before approving new drugs. In that case, randomized evaluations might play the same transformative role in social policy during the 21st Century that they played in medicine during the 20th.” (Michael Kremer, J-PAL working paper)

Claims for efficacy of RCTs and ‘randomized evaluations’

Level 5: Randomized evaluation

Level 4: Measures of the outcome before and after in multiple situations, controlling for variables

Level 3: Measures of the outcome before and after, with control group

Level 2: Measures of the outcome before and after the programme

Level 1: Correlation between a programme and a measure of outcome at one point in time

Maryland Scale

Different dimensions of impact matter in different circumstances

Tangible / Intangible outcomes

Direct / Intermediate outcomes

Exploratory / Roll-out

Different agendas of impact measurement

Participative / Technocratic

Integrating outcomes into single measure / keeping aspects disparate

Adjusting for changes in outcome versus looking for ‘best practice’

Fundamental Question: How do we see the world?

‘Positivist’ Paradigm ‘Interpretivist’ Paradigm

Absolute reality

Change in external, observable circumstances and behaviour is the most important and valid

Reality is subjective

Change in subjective experience is most important and subjective measures are equally valid

Human systems can be broken down into linear cause and effect relationships

RCT’s are rigorous and the best way of getting at objective truth

Human systems are too complex to be meaningfully broken down in that way – better to try and understand the dynamics within the system

Exploratory approaches are best

Different mindsets – two forms of measurement practice

Central task Forms of human relationships

System builders Produce a system that is as objective, robust, and quantified as possible

Expert to expert interactions designed to build up a body of knowledge

Expert to audience communications designed to disseminate knowledge

Case by case Produce an assessment that informs stakeholders of the full social value

Facilitator role played to draw out stakeholder views on key outcomes

Focus is on the ‘here and now’, not on what other assessors have done in the past

A typology of techniques

Participative

Technocratic

Integration

Disparate

Theory of change

Before and after

Statistical regression

SROI

RCT

Cost-benefit analysis

Balanced scorecard Multi-criteria dimension analysis

Rating system

Pattern recognition

Standards

No adjustment for changes in outcomes Assess changes in outcomes

Extend intermediate outcomes

Key:

Analysts tend to deploy very different techniques, reflecting different views on analytical and engagement goals

Three key issues for debate

In terms of the challenges that you are facing in your work:

• Is ‘good enough’ analysis the way forward?

• Are there practical ways to make progress on cross-sector agendas such as assessment of levels of empowerment? Are those steps worth taking?

• Are there practical ways for ‘technocrats’ to become more aware of the complex ‘human dimension’ behind their numbers?