The Importance of Negative Evidence

Rob D. van den BergMay, 2013

Setting the StageEvaluations should enable “learning from mistakes”Negative evidence (“what does not work”) could

help usEvidence movement focuses on positive evidence

If it does not work: no clue why, just stop funding If it does work: no certainty on why, just increase

fundingThe nature of positive and negative evidenceA framework for integrating negative evidence: a

Theory of No Change (Christine Woerlen)

2

Data Gathering by a TurkeyEvery day the Turkey gathers data on food, water and

security: Safe and secure environment on the farm with a fence to

keep wolves and foxes out Food and water delivered by the farmer every day

Counterfactual: Turkey’s distant cousin lives in the wild and faces many uncertainties… Often on the run from predators Organized hunts Food and water availability have wild fluctuations

High probability that life is good for a turkey on a farm

3

Turkey graphs

4

Food availability

Water availability

Predator threats

5

Food availability

Water availability

Predator threats

Farm turkey

Wild turkey Cut-off point: head of the Turkey

Black swan event (Nassim Taleb/Popper) Many data points on food and water availability and predator

threats Positive proof that farm turkeys are better off than wild

turkeys Farmer cuts off the head of the farm turkey One event proofs that the “naïve” theory is not correct Large n provides statistically significant proof for a theory One n (a black swan event) is more powerful than the

combined might of many n However large the n is, it will never deliver 100 percent proof One n may proof the theory wrong with 100 percent certainty

6

Causality in ScienceCausality refers to the relationship between two events: the

cause and the effect, where the second event is caused by the first

Scientific theories predict and explain effectsEarly 20th century: logical positivism

Logic guides deductions from general theories to set up tests Empirical data can provide positive proof of theory

Popper: logical positivism cannot escape the induction problem of Hume However many data you gather, it will never constitute positive

proof that the theory is right The proof that is scientifically and logically sound is negative proof Challenge is to falsify a theory

7

Positivism in Development Logical positivism is no longer in vogue in the natural

sciencesTesting of medicine is based on logical positivism and has

been adopted as the “gold standard” by the evidence movement

Naïve positivism has been replaced with nuanced positivism that poses a null hypothesis that should be disproved; however, this still delivers “positive” proof the treatment works

Health, Education and Economics are heavily influenced; development has followed

Large n, divided in two groups (with/without intervention) is needed for evidence

8

From zero to small nExplanatory power: zero difference in n

Theory that explains more is accepted Example: fractal geophysics (chaos theory) versus linear

geophysicsOccam’s razor: zero difference in n

Theory that is simple wins against theory that is complicated Example: Copernicus versus Ptolemy

Predictive power: one n may suffice Special theory of relativity was proven through one observation

of gravitational pull on light during a solar eclipse Falsifying a theory: one n may suffice

One black swan will disproof the theory that all swans are white (Popper)

9

Large nData on natural or human phenomena over time

Can establish historical trends – the more n the betterModeling of large n through macro-economic or other

theories Mathematical approach to “what if” questions – the more n the

betterNatural experimentation; also known as quasi-experimental

Large n is welcome but often difficult to findRandomized controlled trials

Large n is welcome but costly and difficult to control Systemic reviews

Sifting through large n to find relevant n

10

Nature of EvidenceThe term evidence increasingly refers to outcome

of research/studiesHierarchies of evidence (Campbell collaboration,

Maryland hierarchy) focus on large n onlyEvidence based on n=1 or no difference in n is no

longer recognized as such in some of the literature of the “evidence-movement”

Sciences that use n=1 or no difference in n tend to not be less in policy discussions and provide hardly any countervailing perspectives

11

Causality in EvaluationsCausality in research focuses on new subjects – to

proof or disproof causal linkages that are predicted by theory

Causality in evaluations also tackles old subjects and is not focused on proof or disproof of scientific theories, but on what works and why

Interventions take place in a mixed environment of scientific and technical certainties, unproven theories and scientifically unknown territory

Identification of possible causal linkages takes place through a theory based approach

12

13

From ToC to TonCA theory based approach may lead to a theory of change

identifying causal linkages and assumptions covering these

This may also lead to an identification of what could possibly prevent these causal linkages from “working”

It may also identify what prevents the intervention as a whole to move forward

Analogy: a car needs many working components to function as a car, but take away the wheels and it will stop moving

Identification of these factors leads to a “theory of no change”

14

Meta Analysis

Systemic Reviews go through existing evidence in research and evaluations from the perspective of a specific question Are cash transfers effective in promoting school attendance?

Many studies and evaluations do not address this question in the exact same way and are thus not accepted as evidence Health review: only 50 studies accepted from 49.000

Other forms of meta-evaluations do not pose restrictive questions but pose to explore existing evidence All quality evaluations on a subject are accepted; and quality

evidence in a bad evaluation may also be accepted Theory based approach

15

Over to Christine Woerlen!