Using programme theory for evaluation of complex interventions

Prashanth NSFaculty & PhD Scholar

Institute of Public Health, Bangalore

Outline

• Background– Interventions, HS interventions and complexity– Framing evaluation questions? “Did it…”, “What..”,

“Why (or why not?)..” or “How…”– Multiple paths

• How to use a programme theory approach in a complex local health systems intervention

Source: Pd photo/NLM, USA

“Medicine is a social science, and

politics is nothing else but

medicine on a large scale.

Medicine, as a social science, as

the science of human beings, has

the obligation to point out

problems and to attempt their

theoretical solution: the

politician, the practical

anthropologist, must find the

means for their actual solution…”

Randomise and control

Origins in drug testing for effectiveness– Outcomes measurable, verifiable; animal models

available. Environmental conditions can be controlled. • Linearity (linear and observable transitions from inputs

to outcome; environmental conditions known)• No ethical issues in setting up control• Gold-standard for effectiveness

Hazardous journeys

Smith, G. C. S., & Pell, J. P. (2003). Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. BMJ, 327(7429), 1459–61. doi:10.1136/bmj.327.7429.1459

But health systems are complex…

WHO 2000 WHO AHPSR 2008

Software neglected

Complexity & systems-thinking

• Simple, complicated and complex (Glouberman & Zimmerman)

– Examples: from recipies to rockets to children• Characteristics– Multiple interacting elements (within and outside)– Organisational structure and interactions– Multiple paths/configurations to same outcome;

same structural configuration but different outcome (path dependency)

– Unpredictability due to feedback loops

Complexity in health systems/interventions

Institute of Public Health, Bangalore

Medical research council guidance

Effectiveness or Mechanism?

Existing methods in public health research focus on effectiveness: “Did it work? To what extent”

But, middle-level managers, decision-makers are looking for: “How did it work?”, “Why did it not work for x or y?”

The evaluation question

Ideally, evaluations should be able to inform not only if a given intervention worked, but also how it worked, and why it worked for some (and not for others)

Why did Arogyashri not benefit beneficiaries from SC/ST proportional to

their population in AP?Is there a social phenomenon operating

here that could help us understand implementation of financing or other

schemes in general?

Fan, V. Y., & Mahal, A. (2011). Learning and getting better : Rigorous evaluation of health policy in India. National Medical Journal of India, 24(6), 325–327.

Answering how questions

Various methods possible (MRC guidance)

– CrCT with process evaluations, stepped-wedge designs

– Modeling: causal modelling, mathematical modelling, TdI, economic modelling

– Natural experiments & cohort studies

The programme theory approach

A programme theory is way of representing the expected relationship between the elements of the intervention and its expected outcomes.

Consider the pulse polio programme

But, for complex interventions…

Initial programme assumptions

Learning from general theory

Reconstruction

Empircise

Applications of the PT

• Theorising and understanding “how”: in parallel with RCTs (process evaluation) or preliminary to a realist evaluation (my study design)

• Informing design/development of future similar programmes

• Forms a basis for analysing the qualitative data and for using frameworks (Eg. Using MPF for understanding organisational performance)

Six-step modelVan Belle, S. B., Marchal, B., Dubourg, D., & Kegels, G.

(2010). How to develop a theory-driven evaluation design? Lessons learned from an adolescent sexual and reproductive

health programme in West Africa. BMC public health, 10, 741. doi:10.1186/1471-2458-10-741

A mixed method RE design using PT

Addressing complexity by understanding relationships rather than structural outcomes

Marchal et. al’s representation of Sicotte framework

Using PT for evaluation

Examining the data using the framework helps further refine the PT. Allows for identifying positive and negative configurations of the context-mechanism-outcome that can be emperically verified to answer the question “How does capacity-building work in this intervention?”

Key referencesPrashanth, N. S., Marchal, B., Hoeree, T., Devadasan, N., Macq, J., Kegels, G., & Criel, B. (2012). How does capacity building of health managers work? A realist evaluation study protocol. BMJ open, 2(2), e000882. doi:10.1136/bmjopen-2012-000882

Marchal, B., Van Belle, S., Van Olmen, J., Hoeree, T., & Kegels, G. (2012). Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research. Evaluation, 18(2), 192–212. doi:10.1177/1356389012442444

Van Belle, S. B., Marchal, B., Dubourg, D., & Kegels, G. (2010). How to develop a theory-driven evaluation design? Lessons learned from an adolescent sexual and reproductive health programme in West Africa. BMC public health, 10, 741. doi:10.1186/1471-2458-10-741

Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., & Petticrew, M. (2008). Developing and evaluating complex interventions: the new Medical Research Council guidance. British Medical Journal, 337(sep29 1), a1655–a1655. doi:10.1136/bmj.a1655

Marchal, B., Dedzo, M., & Kegels, G. (2010). A realist evaluation of the management of a well- performing regional hospital in Ghana. BMC Health Services Research, 10(October 2000), 24.

On Mendeley, see my reading list on critical realism and realist evaluation at:http://www.mendeley.com/groups/535071/critical-realism-and-realist-evaluation/papers/

AcknowledgementBart Criel, Guy Kegels, Jean Macq, Bruno Marchal & Tom Hoeree at ITM, AntwerpUpendra Bhojani, Tanya Seshadri, Arima Mishra & N Devadasan for discussions and re-discussions