1

Choices and Outcomes: The Effects of Improvement Project

Portfolio Choices on Clinical Outcomes

Anita L. Tucker Assistant Professor, U. of Pennsylvania

Senior Fellow, Leonard Davis Institute of Health Economics

Ingrid M. NembhardDoctoral Candidate, Harvard Business SchoolHarvard Graduate School of Arts and Sciences

KLIC 4: INFORMS ConferenceNovember 6, 2006

Financial Support from HBS DOR, Wharton’s Fishman Davidson CenterIn Collaboration with Jeffrey Horbar, Richard Bohmer and Amy Edmondson

2

Background: Health Care in America

1999: Institute of Medicine (IOM) reports 100,000 Americans die annually from preventable medical errors

1998: National Roundtable reports “serious and

widespread problems” exist in American medicine” . .

Problems of underuse, overuse, misuse . . “Quality of care is

the problem.”

2003: RAND Study reports only 55% of patients receive the recommended care for their condition

2000: IOM finds quality problems are a systemic property, requiring process improvement

1996: Dartmouth Atlas Project shows inappropriately geographic variations in care

Need for improvement projects in health care organizations

The Data:

=

+

Publicity:

3

Improvement Projects

Improvement Projects “solve complex organizational problems through the work of formal teams that use a structured improvement method” (Christianson et al., 2005: p. 610)

Healthcare Project Examples Target Area• Increase handwashing (Reduce Infections)• Reduce heelsticks in Neonates (Pain management)• Increase collaboration among MDs and RNs (Staff Retention)

Portfolio of Improvement Projects: An organization’s set of improvement projects that are in progress at the same time and draw on the same limited set of human, managerial and financial resources (Cooper, Edgett & Kleinschmidt, 1999; Wheelwright & Clark, 1992)

4

Research Question

How should health care organizations structure their portfolios of improvement projects to

achieve better outcomes?

Hypotheses about Portfolio Choices1) Number of projects (+)2) Concentration within a target area (+)3) Level of evidence for portfolio (+)4) Novices should start with clinically-

oriented portfolio (+) 5) Extent of physician involvement (+)

5

Collaborative of Neonatal Intensive Care Units (NICU)

44 NICUs working together for 2 ½ years (Apr 2002 - Oct 2004)

Identified 7 target areas for improvement and 93 improvement projects across those areas

Met twice a year to learn QI methods (PDSA cycles), work on developing best practice guidelines and share experiences

In between meetings, implemented practices, conducted site visits to other NICUs and phone conferences

Research Setting: Vermont Oxford Network

Horbar, J. D. et al, 2001. Collaborative quality improvement for neonatal intensive care. Pediatrics 107 (1) 14-22.

6

Portfolio Project OptionsTarget Area Aim # of

practices

Sample practice project(s)

Pain & Sedation Decrease mean pain score by 50% during

NICU stay

10 Pain management during heelsticks

Intubation of medications

Infection Control Decrease hospital-acquired infections by

25-50% over 2 years

7 Promote hand washing education and

practices to prevent nosocomial infection

Respiratory Care Decrease chronic lung disease by 10%,

and decrease oxygen days, ventilator

days, steroid use

15 Vitamin A supplementation

Ventilation by Tidal Volume Monitoring

OB/NICU

Relations

Improve maternal & newborn caregiver

collaboration: periviability, delivery

response, comfort care

6 Design process to increase collaboration and

communication during high-risk delivery

Staff Retention Decrease staff turnover by 50% 5 Improve nurse-physician collaboration

Family-Centered

Care

Enhance ability to co-ordinate and deliver

care so the infant and family needs are met

27 Provide resource materials that depict

newborn premature infants’ maturational and

postnatal environment.

Discharge

Planning

Embed discharge planning into all aspect

of patient care & communication

23 Develop "trigger point" checklist for discharge

teaching

93 Potential Projects in the Portfolio

Cli

nic

all

y-o

rien

ted

Op

erat

ion

ally

-ori

ente

d

**Each NICU indicated which projects they included in their project portfolio to the collaborative sponsor.

7

Unique Portfolios of Practices

PBP Ho

spit

al 1

00

Hos

pita

l 101

Ho

spit

al 1

02

Hos

pita

l 103

Hos

pita

l 104

Hos

pita

l 105

1 Intubation Medications # # + #10 Pharmacological Factors affecting Opioid Tolerance # #2 Reduce frequency of tracheal suctioning o + o + o3 Reduce frequency of heelsticks + + x o + #4 Standardized Recommendations for Sucrose Analgesia o + + o # x5 Pain Management During Heelsticks + o + x6 Peripheral Vascular Procedures o + o + +7 Circumcision o + o x x8 Postoperative Pain Management x + + o x #9 Guidelines for weaning from opioids x + + #

Sum of + Have in place prior to NIC/Q 2002 1 6 3 1 6 1Sum of o Implemented during NIC/Q 2002 4 0 0 7 0 1Sum of X Working on 2 0 1 0 2 3Sum of # plan to work on 2 0 0 1 1 5Sum of not working on and don't plan to 1 4 6 1 1 0

Excerpt from the practices from Pain Management

Implemented during collaborative

Working on

Hospital 100 Implemented/Working on: Reducing frequency of tracheal suctioning, standarized sucrose analgesia, peripheral vascular procedures, circumcision, post op pain, weaning from opiods

VERSUSHospital 102: Reducing frequency of heelsticks

8

Evidence-base for the portfolio

Level of evidence for all projects within each target area assessed by target area team using Muir-Gray (MG) score to rate articles1 = strong evidence from at least one systematic review of multiple,

well-designed, randomized, controlled trials

2 = properly designed randomized control trial of appropriate size

3 = well-designed trials without randomization

4 = well-designed non-experimental trials

5 = opinions of respected authorities, based on clinical evidence,

descriptive studies or reports of expert committees

Evidence base for NICU portfolio = the average MG score of the portfolio

9

Outcome: Standardized Mortality Ratio

SMR level of analysis: Babies nested in NICU

1.LOGIT model (clustered by NICU) Outcome Death (0,1)

• Independent Variables: Established risk factors (Zupanic et al. 2006)

2.Predict probability of death for each baby

3.By NiCU, sum up probability of death, actual deaths

4.Compute ratio:

• SMR < 1 Outcomes BETTER than expected

• SMR = 1 Outcomes equal to expected

• SMR > 1 Outcomes WORSE than expected

deathsExpected

deathsActualMortalitySMR

10

Means, SD, and correlations (N=27)

^=p<.1; *=p<.05; **=p<.01

Variable Mean SD 1 2 3 4 5 6 7 8 9 101 Staff/bed 3.34 0.92

2 Cardiac Surgery 0.44 0.51 .02

3# Prior Collaboratives 0.78 0.75 .14 .07

4 QI Team Size 13.04 5.91 .12 -.20 -.28

5Teaching Hospital 0.67 0.48 .03 .32 -.21 -.21

6Total comp. projects 8.56 8.87 -.16 -.19 -.04 -.05 .05

7Total comp. LOS 3.44 5.29 -.08 -.19 .01 .03 -.08 .92**

8 Ave Evidence 3.75 0.43 -.04 .01 .01 .24 .10 .21 .39*

9% of MDs on team 0.20 0.09 -.12 .19 .47** -.34^ .07 .22 .21 -.12

10Improve. SMR LOS 04-01 -0.02 0.19 -.17 .25 .19 .13 -.07 -.48** -.42* .05 -.06

11Improve. SMR Mort. 04-01 0.15 0.82 -.07 -.12 .31 -.39* -.20 .20 .23 -.11 .03 -.39

11

OLS Regression results (H1, H3, H4)Outcome measure Improvement SMR

mortality (2004-2001)

Control Variables

Staff to beds ratio -0.336* (0.145)

Cardiac surgery provider -0.232 (0.240)

History of quality improvement 0.569* (0.208)

Team size -0.020 (0.023)

Teaching Status 0.189 (0.279)

Independent Variables

Number of projects -0.105* (0.044)

Number of projects squared 0.004** (0.001)

Evidence supporting portfolio -0.526^ (0.293)

% of MDs on QI team -3.429* (1.507)

Constant 4.139** (1.291)

Adj. R-squared 0.37

F 2.62

Sig 0.04

df 9, 16

N = 26, (std error)^ significant at 10%; * significant at 5%; ** significant at 1%

H1: Supported u-shape

H3: Less evidence->Imp

H4: Supported

12

H2: Concentrating the within a target area helps

Outcome measure Improvement SMR length of stay (2004-2001)

Control Variables

Staff to beds ratio -0.049 (0.036)

Cardiac surgery provider 0.118 (0.071)

History of quality improvement 0.082 (0.048)

Team size 0.008 (0.006)

Teaching Status -0.047 (0.077)

Independent Variables

Number of LOS projects -0.013^ (0.006)

Constant -0.007 (0.161)

Adj. R-squared 0.23

F 2.22

Sig 0.09

df 6, 19

N = 26, (std error)

^ significant at 10%; * significant at 5%; ** significant at 1%

H2: Supported

13

H5: Initial portfolio orientation mattersOutcome measure Improvement SMR mortality

(2004-2001)

Control Variables

Staff to beds ratio -0.251 (0.144)

Cardiac surgery provider

History of quality improvement

Team size

Teaching status

Independent Variables

Number of projects -0.213* (0.082)

Number of projects squared 0.005* (0.002)

Evidence supporting portfolio

% of MDs on QI team

Clinically oriented portfolio -1.751* (0.697)

Constant 2.951* (1.008)

Adj. R-squared 0.16

F (df) 7.50 (4,6)

Sig 0.02

N= 11(Robust std error)^ significant at 10%; * significant at 5%; ** significant at 1%

H5: Supported

14

Summary and Implications

An effective improvement project portfolio:

• Includes neither too few or too many projects to manage the tradeoff between synergy and distraction

• Concentrates its efforts within a target area to maximize inter-project learning

• Focuses on operationally-oriented projects which build performance-improvement capability

• For novices is clinically-oriented where clearer benchmarks are available

• Is led by a team with significant physician membership.