the science behind clinical systems improvementcardiac arrest, unplanned icu admissions, or...

The Science Behind ClinicalSystems Improvement

Donald M. Berwick, MD, MPP, FRCPInstitute for Healthcare Improvement

Clinical Systems Improvement SeminarUniversity of Warwick

Coventry, UK: 23 January 2008

William Stewart Halsted

2

3

Canadian Task Force:Grades of Evidence

I: One Properly Randomized Controlled TrialII: Well-Designed Controlled Trials without

RandomizationII-2: Well-Designed Cohort or Case-Control

Analytic StudiesII-3: Multiple Time Series with or without the

Intervention; or Dramatic Results in UncontrolledExperiments

III: Opinions of Respected Authorities Based onClinical Experience; Descriptive Studies andCase Reports; Reports of Expert Committees

4

Institute of Medicine:Crossing the Quality Chasm

AIMS FOR IMPROVEMENT OF HEALTH CARE

• Safety• Effectiveness• Patient-Centeredness• Timeliness• Efficiency• Equity

5

Four Concerns

• Inference and gaining knowledge

• Thresholds for action

• Bias and trust

• Hope, teamwork, civility

Dr. Hillman’s Letter: July 23, 1982

6

Dr. Hillman’s Letter

7

Dr. Hillman’s Letter

“This is a different concept to the cardiacarrest team. It is hoped that providingearlier and more expert care,cardiopulmonary arrest may be avoided.The doctors called to seriously ill medicalor surgical patients would provide rapidresuscitation and assessment.”

8

Rapid Response Teams

BMC Code Rate

0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

10.00

J an . 05F eb . 05

M a r . 0 5Ap r . 0 5

M ay . 05J un . 05

J u l. 0 5

Aug . 05

Sep t. 0 5O ct.

0 5

N ov . 05D ec . 05

J an -0 6F eb -06

M a r -06Ap r -0

6M ay -06

J un -0 6J u l- 0 6

Aug -06Sep -06

O ct- 06N ov -06

D ec -06J an -0 7

F eb -07M a r -0

7Ap r - 0

7M ay - 07

J un -0 7J u l-0 7

Aug -07Sep t-0 7

Co

de

pe

r1

00

0P

ati

en

tD

ay

sCode Rate

Mean

9


Codes Outside the ICU and RRS UtilizationDelmarva Foundation

2.4 2.2 2.3

1.9

1.4

1.9 2.01.7

2.812.1

14.3

18.6

14.4

9.7 9.710.6

11.8

9.9

0

1

2

3

4

5

Jan-06 Feb-06 Mar-06 Apr-06 May-06 Jun-06 Jul-06 Aug-06 Sep-06 Oct-06 Nov-06 Dec-06

Co

des

Out

sid

eth

eIC

Up

er1,

000

Dis

char

ges

0

2

4

6

8

10

12

14

16

18

20

RR

SC

alls

per

1,00

0D

isch

arge

s

Number of codes outside the ICU per 1,000 dischargesNumber of RRS calls per 1,000 discharges

10


Crash Call Rate

1

2

3

4

5

6

7

8

9

Jan-

05

Feb

-05

Mar

-05

Apr

-05

May

-05

Jun-

05

Jul-0

5

Aug

-05

Sep

-05

Oct

-05

Nov

-05

Dec

-05

Jan-

06

Feb

-06

Mar

-06

Apr

-06

May

-06

Jun-

06

Jul-0

6

Aug

-06

Sep

-06

Oct

-06

Nov

-06

Dec

-06

Period

Ind

ivid

ua

lV

alu

e

11


12


13


14


15


Catholic Health Initiatives System Mortality Rates

July02-Jun04 July04-Jun05 July05-June06 July06-Jun07 July07-Oct07

Baseline Mortality

IHI 100K Lives Campaign

Introduction to RRTs

100% CHI Beds

Covered By RRTsIHI 5M Lives Campaign

600+ RRT Calls

Q1 FY 08

16

17

The “100,000 Lives Campaign”

Six Changes That Save Lives

1. Deploy Rapid Response Teams

2. Deliver Reliable, Evidence-Based Care forAcute Myocardial Infarction

3. Prevent Adverse Drug Events (ADEs)

4. Prevent Central Line Infections

5. Prevent Surgical Site Infections

6. Prevent Ventilator-Associated Pneumonia

18

19

MERIT Study

“The MET system greatly increasesemergency team calling, but does notsubstantially affect the incidence ofcardiac arrest, unplanned ICU admissions,or unexpected deaths.”

MERIT Study Investigators. Introduction of the medical emergency team(MET) system: a cluster-randomized controlled trial. Lancet. 2005; 365:2091-2097

19

20

Johns Hopkins Researchers

“Given the equivocal evidence supportingthe effectiveness of RRT programs, withthe largest best-designed study showingno benefits, it is unclear why there is suchinterest in implementing this interventionand making it a care standard…”

Winters BD, Pham J, Pronovost PJ. Rapid response teams – walk, don’trun. JAMA. 2006; 296: 1645-1647.20

21

New England Journal of MedicineSounding Board

“Early trials of medical emergency teamssuggested a large potentialbenefit…However, a large, randomizedtrial subsequently showed that medicalemergency teams had no effect on patientoutcomes.”

Auerbach AD, Landefeld CS, Shojania KG. The tension between needingto improve and knowing how to do it. N Engl J Med. 2007; 357: 608-613.21

22

Consensus Conference Report

“It was emphasized that findings for oneresponse team might not…begeneralizable to other settings or overtime, and that the intervention itself shouldcontinually evolve and improve…”

Øvretveit J, Suffoletto J. Improving rapid response systems: progress,issues, and future directions. Jt Comm J Qual Patient Saf. 2007; 33: 512-519.

22

23

Consensus Conference Report

“ An RRS (rapid response system) is not adrug treatment, a hospital is not a body.Both are continually changing, complexsocial interventions and cannot beunderstood through randomized controlledtrials.” (John Øvreveit, MD)

Øvretveit J, Suffoletto J. Improving rapid response systems: progress,issues, and future directions. Jt Comm J Qual Patient Saf. 2007; 33: 512-519.

23

24

Pawson and Tilley: Realistic Evaluation

24

25

Pawson and Tilley

Pre-Test Treatment Post-Test

ExperimentalGroup

O1 X O2

Control Group O1 O2

The Classic Experimental Design: “OXO”

Pawson R, Tilley N. Realistic Evaluation. London: Sage Publications,Ltd.; 1997.25

26

Pawson and Tilley

“For us, the experimental paradigmconstitutes a heroic failure, promising somuch and yet ending up in ironicanticlimax. The underlying logic…seemsmeticulous, clear-headed, and militarilyprecise, and yet findings seem to emergein a typically non-cumulative, low-impact,prone-to-equivocation sort of way.”


27

Pawson and Tilley

Context + New Mechanism = Outcome

C + M = O


28

Pawson and Tilley

“In other words, programs work (havesuccessful ‘outcomes’) only in so far asthey introduce the appropriate ideas andopportunities (‘mechanisms’) to groups inthe appropriate social and culturalconditions (‘contexts’).”


29

Implications of Pawson and Tilley

“In other words, Rapid Response Systemswork (have successful ‘outcomes’) only inso far as they introduce the appropriateideas and opportunities (‘mechanisms’)to groups in the appropriate social andcultural conditions (‘contexts’).”


30

Pawson and Tilley

“…(E)xperimentalists have pursued toosingle-mindedly the question of whether aprogram works at the expense of knowingwhy it works.”


31

Sir Austin Bradford Hill

“All scientific work is incomplete – whether itbe observational or experimental. Allscientific work is liable to be upset ormodified by advancing knowledge. Thatdoes not confer upon us a freedom toignore the knowledge we already have, orto postpone action that it appears todemand at a given time.”

Hill AB. The environment and disease: association or causation? ProcRoyal Soc Med. 1965; 58: 295-300.31

32

Four Changes

1. Broaden methods of inference andembrace a wider range of ways to growknowledge (with no relaxation of rigor)

32

Is life this simple?

X Y

(If only it was this simple!)

Understanding Profound Knowledge

33

No, it looks more like this…

X3

X2

X1

X5

X4

Y

In this model there are numerous direct effects between theindependent and variables (the Xs) and the dependent variable (Y).

Time 1

Time 3

Time 2

Dependentor outcomevariable

Ind

ep

en

de

nt

Va

ria

ble

s

34

In this case, there are numerous direct and indirect effects betweenthe independent variables and the dependent variable. For example, X1

and X4 both have direct effects on Y plus there is an indirect effect dueto the interaction of X1 and X4 conjointly on Y.

Y

Or, probably more like this…

X3

X2

X1

X5

X4

Time 1

Time 3

Time 2

R3

R2

R1

R5

R4

RY

Key Reference on Causal Modeling

Blalock HM, ed. Causal Models in theSocial Sciences. Chicago: Aldine; 1999.

R = residuals or error termsrepresenting the effects of variables

omitted in the model.

35

You need to enjoy the messiness of life!

36

Rigor and Complexity: View #1

SimpleLinear

Cause-and-Effect

Informal Narrative StudyStories and Anecdote

“Non-Rigorous”

ComplexNon-Linear

Chaotic

Formal Evaluative DesignsRCT’s

“Rigorous”

The “Evaluation Project” –Add Rigor to the Complex

37

Rigor and Complexity – View #2

SimpleLinear

Cause-and-Effect

ComplexNon-Linear

Chaotic

“Rigorous” Learning

Poor Learning38

Evidence in Simple Systems

SimpleLinear

Cause-and-Effect

ComplexNon-Linear

Chaotic


Poor Learning

Fisher’sRCT’s

Halsted’sConclusions

39

Evidence in Complex Systems

SimpleLinear

Cause-and-Effect

ComplexNon-Linear

Chaotic


Poor Learning

Fisher’sRCT’s


RCT’s to StudyRapid Response Teams

40

Eating Soup with a Fork

SimpleLinear

Cause-and-Effect

ComplexNon-Linear

Chaotic


Poor Learning

Fisher’sRCT’s



41

Rigorous Learning inComplex Systems

SimpleLinear

Cause-and-Effect

ComplexNon-Linear

Chaotic


Poor Learning

Fisher’sRCT’s



What Methods ofLearning Go Here?

42

Rigorous Learning inComplex Systems

SimpleLinear

Cause-and-Effect

ComplexNon-Linear

Chaotic

“Rigorous”Learning

Poor Learning

Fisher’sRCT’s



•Statistical Process Control•Time Series Methods•Off-Line Design•Anthropology•Ethnography•Journalism•Etc…Etc…Etc…

43

44

Four Changes


2. Reconsider our attitudes towardthresholds for action

44

Primum non noncere

“When harm is underway, proceed urgentlyto learn how to stop it, and act urgently on

the learning.”

45

Science in Cycles of Action and Learning

PLAN – DO – STUDY - ACT

P

A

S

D

46

47

Four Changes



3. Reconsider our view of “bias” andincrease trust in knowledge of contextand mechanism

47

Auerbach AD, Landefeld CS, Shojania KG. Thetension between needing to improve andknowing how to do it. N Engl J Med. 2007; 357:608-613.

48

One View of Bias and Trust

“Even when direct financial conflicts ofinterest do not exist, any organization thathas undertaken a major campaign toimprove the quality of care has littleincentive to invest resources in a rigorousevaluation of the effects of its efforts.”

48

49

Possible Inferences…?

• “Parents have little incentive to evaluate theeffects of their parenting.”

• “Golfers have little incentive to study the effectsof their stroke.”

• “Armies have little incentive to study the effectsof their tactics.”

• “Politicians have little incentive to study theeffects of their campaign plans.”

• “Environmentalists have little incentive to studytheir effects on the environment.”

49

50

Four Changes



3. Reconsider our view of “bias” andincrease trust in knowledge of contextand mechanism

4. Reconsider mood, affect, and civility

50

the science behind clinical systems improvementcardiac arrest, unplanned icu admissions, or...

Documents