1

Evaluating Revalidation: Lessons from the Evaluation of Complex

Interventions

IAMRA Revalidation Symposium

Sanjeev Sridharan The Evaluation Centre for Complex Health

InterventionsUniversity of Toronto &St. Michael’s Hospital

Standpoint

Evaluation and

complexity

Evaluation Framework

Application to GMC Domains

Ten Questions

3

What is evaluation? A useful but perhaps incomplete

definition • Evaluation is defined both as a means of

assessing performance and to identify alternative ways to deliver

• “evaluation is the systematic collection and analysis of evidence on the outcomes of programs to make judgments about their relevance, performance and alternative ways to deliver them or to achieve the same results.”

4

Purpose of evaluation

• Assessing merit and wortho Causal questions, RCT, observational studies

• Programme and organizational improvemento Formative evaluation

• Oversight and compliance• Knowledge development

o Neglected purpose of many evaluationso Processes and active ingredientso Testing assumptions

5

An Example: Primary Prevention Have a Heart Paisley

6

Features of complex interventions

The intervention is a theory or theories The intervention involves the actions of people. The intervention consists of a chain of steps These chains of steps or processes are often not

linear, and involve negotiation and feedback at each stage.

Interventions are embedded in social systems and how they work is shaped by this context.

Interventions are prone to modification as they are implemented.

Interventions are open systems and change through learning as stakeholders come to understand them.

7

System Dynamic Approaches (Sterman,

2006)

• Constantly changing;• Governed by feedback;• Non-linear, History-dependent;• Adaptive and evolving;• Characterized by trade-offs;• Policy resistance: “The result is policy

resistance, the tendency for interventions to be defeated by the system’s response to the intervention itself.”

8

“Solutions” Can Also Create New Problems

Meadows DH, Richardson J, Bruckmann G. Groping in the dark: the first decade of global modelling. New York, NY: Wiley, 1982.Merton RK. The unanticipated consequences of purposive social action. American Sociological Review 1936;1936:894-904.Forrester JW. Counterintuitive behavior of social systems. Technology Review 1971;73(3):53-68.

Policy resistance is the tendency for interventions to be delayed, diluted, or defeated by the response of the system to the intervention

itself. -- Meadows, Richardson, Bruckman

9

System-as-Cause

Forrester JW. Counterintuitive behavior of social systems. Technology Review 1971;73(3):53-68.Meadows DH. Leverage points: places to intervene in a system. Sustainability Institute, 1999. Available at <http://www.sustainabilityinstitute.org/pubs/Leverage_Points.pdf>.Richardson GP. Feedback thought in social science and systems theory. Philadelphia, PA: University of Pennsylvania Press, 1991.Sterman JD. Business dynamics: systems thinking and modeling for a complex world. Boston, MA: Irwin McGraw-Hill, 2000.

10

11

Intervention System

Components

of the Interventio

n

Stabilityof the

components

Impacts on evaluation

design

1. Elements of the intervention: What are the elements of a revalidation

system?

12

2. What is the theory of change by which revalidation can enhance quality of care? What are the connections between the various domains (trust, knowledge, communication, quality)?

• Context and Support Conditions• Mechanisms• Heterogeneity• Leverage

13

3. Evidence Synthesis: What can we learn from the literature on

revalidation?

14

4. Anticipated Timeline of Impact: What is your timeline of impact? Would revalidation impact quality of care within the time frame of the evaluation?

Outcome 1

Outcome 2

Outcome 3

Program

Years1 2 3 4 5 60

15

B. Structure of Evaluation Influence

5. Framework of Learning6. Pathways of evaluation influence

16

5. What do you wish to learn from the

evaluation of revalidation? Improvement vs.

Merit and Worth

17

Evaluation/Performanc

e Measureme

nt

Pathways of

Evaluation Influence: Collective, Individual

and Interperso

nal

6. How will the evaluation of Revalidation impact practice?

Impacts on Policy,

Practice and the Political Agenda

18

Design, Methods and Data

• 7. Design Issues• 8. Learning about Program Theory

19

Some ideas for design• Is a comparison group possible?• Longitudinal • Ecology of Evidence• Involving the stakeholders• Balance an improvement vs. Merit and worth lens• Multilevel• Attribution vs. Contribution

20

ProgramEvaluation Design/Met

hodProgram Impacts

Initial Program Theory

Initial Impacts

Areas of Uncertainty

Learning from

Innovative Methods

Emergent Program TheoryScenario 2

Scenario 1

(8) Learning from Methods: How do you learn along the way?

21

Spread and Sustainability

22

Program

Larger System

Spread Innovations

Components of System

Innovative Practice

Contextual Learning

(9) Framework of Spread

23

Framework of Learning

Decision to Sustain

Match between

Anticipated Findings of

Impact/Performan

ce Trajectory and Actual Performan

ce

Evaluation Results

Other Factors

(10) A Framework for Sustainability

24

How will this approach help with learning about key domains discussed by GMC?

• 1. Knowledge, skills, and performance• 2. Safety and quality• 3. Communication, partnership, and teamwork• 4. Maintaining Trust.

25

Clarifying revalidationo Theory of Change; Realist Review

• Connect the different domains• Connections between different mechanisms• Support conditions

o Timeline of Impact

o What are you uncertain of? What evidence do you need? Whom are you trying to persuade? • Ecology of Evidence

o ProcessesTough tests of processesMechanisms and differential mechanismsFormative vs. Summative

o Knowledge Translation

o Learning, accountability and accountability towards learning

26

Linking processes to outcomes

Variations in ‘revalidation’ implementation

Longitudinal

MultilevelConnect changes in practice to patient

outcomes

Patient involvement—qualitative longitudinal research