www.icnarc.org analysis and presentation of quality indicators dr david harrison senior...

ww

w.icn

arc.o

rg

Analysis and presentation of quality indicators

Dr David HarrisonSenior Statistician, ICNARC

Analysis and presentation of quality indicators | Dr David Harrison

ww

w.icn

arc.o

rg

Analysis and presentation of QIs• Principles of statistical process

control• Comparison among providers• Continuous monitoring over time


ww

w.icn

arc.o

rg


control– Common cause variation– Special cause variation– Control limits

• Comparison among providers• Continuous monitoring over time


ww

w.icn

arc.o

rg Principles of statistical process

control• Common cause variation

– Variation cannot be eliminated– Some variation is inherent to any

process– This is termed “common cause

variation”– To reduce common cause variation

we need to change the process


ww

w.icn

arc.o

rg

Five signatures…


ww

w.icn

arc.o

rg

They are not identical…


ww

w.icn

arc.o

rg

They are not identical…

…but they are all my signature


ww

w.icn

arc.o

rg

We could rank them…

…but this doesn’t make much sense!

1.

2.

3.

4.

5.


ww

w.icn

arc.o

rg We could reject some as low

quality…

…but they are still my signature!


ww

w.icn

arc.o

rg

This is common cause variation


ww

w.icn

arc.o

rg Principles of statistical process

control• Special cause variation

– Some variation is the result of external factors acting on a process

– This is termed “special cause variation”

– To reduce special cause variation we need to identify the source and eliminate it


ww

w.icn

arc.o

rg

Now we have a sixth signature…


ww

w.icn

arc.o

rg

Now we have a sixth signature…

…it’s a good try, but I think you can tell which one is the forgery!


ww

w.icn

arc.o

rg

This is special cause variation


ww

w.icn

arc.o

rg

Control limits• Statistical process control is all

about making allowance for common cause variation to detect special cause variation

• To do this we place control limits around a process

• Control limits represent the acceptable range of common cause variation


ww

w.icn

arc.o

rg

Control limits• Typically control limits of 2 and 3

SDs represent “alert” and “alarm”• If a system is in control:

– 95.4% of values within 2 SDs– 99.7% of values within 3 SDs


ww

w.icn

arc.o

rg


control• Comparison among providers

– League tables– Caterpillar plots– Funnel plots– Over-dispersion

• Continuous monitoring over time


ww

w.icn

arc.o

rg

Comparison among providers• I’ll assume we have a binary event

(e.g. death) and an associated risk estimate (e.g. predicted risk of death)

• Most common QI is:observed events / expected

events• (for mortality this is the standardised

mortality ratio)• How should we compare this QI

among providers (e.g. critical care units)?


ww

w.icn

arc.o

rg

League tables• Journalists love them

– High impact– Everyone wants to know who is first

and last


ww

w.icn

arc.o

rg

Seven deadliest hospitals identified

in damning Dr Foster reportDaily Telegraph, 29 November 2009

Twelve NHS trusts slammedThe Sun, 29 November 2009

Patient safety at Scarborough

Hospital ‘second worst in country’

Scarborough Evening News, 29 November 2009


ww

w.icn

arc.o

rg

League tables• Journalists love them

– High impact– Everyone wants to know who is first

and last

• Statisticians hate them– Overemphasise unimportant

differences– Even if there is no true difference,

someone will be first and someone last– No account of role of chance

(common cause variation)


ww

w.icn

arc.o

rg

Marshall & Spiegelhalter, BMJ 1998• League table of 52 IVF clinics

ranked on live birth rate• Monte Carlo simulation to put 95%

CI on ranks


ww

w.icn

arc.o

rg

Marshall & Spiegelhalter, BMJ 1998


ww

w.icn

arc.o

rg


• King’s College Hospital – sixth from bottom – is the only one that can reliably be placed in the bottom 25%


ww

w.icn

arc.o

rg


• BMI Chiltern Hospital – seventh from bottom – may not even be in the bottom 50%


ww

w.icn

arc.o

rg


• Five clinics can confidently be placed in the top quarter

*

****


ww

w.icn

arc.o

rg


• Southmead General – ranked sixth from top – may not be in the top 50%


ww

w.icn

arc.o

rg

Caterpillar plots (or forest plots)• Plot of QIs with CIs in rank order• Still a league table really• But at least acknowledges

variation by including CIs


ww

w.icn

arc.o

rg

Caterpillar plot – IV clinics


ww

w.icn

arc.o

rg

Caterpillar plot – ANZICS

• SMRs by APACHE III-J for 106 adult ICUs in Australia and New Zealand, 2004(Cook et al. Crit Care Resusc 2008)


ww

w.icn

arc.o

rg

Funnel plots• Larger sample = greater precision• If you plot QI against sample size,

you expect to see a funnel shape• We can plot funnel shaped control

limits

ww

w.icn

arc.o

rg

|


ww

w.icn

arc.o

rg

Funnel plot – ANZICS

• SMRs by APACHE III-J for 106 adult ICUs in Australia and New Zealand, 2004(Cook et al. Crit Care Resusc 2008)


ww

w.icn

arc.o

rg


• Note: use of normal distribution can result in negative confidence intervals – better methods exist


ww

w.icn

arc.o

rg


• Note: as SMR is a ratio measure, we would advocate plotting on a log scale (i.e. SMR=2 and SMR=0.5 are equidistant from SMR=1)


ww

w.icn

arc.o

rg

Funnel plot – SICSAG

• SMRs by APACHE II for 25 adult ICUs in Scotland, 2009(SICSAG Audit of critical care in Scotland 2010)


ww

w.icn

arc.o

rg

Funnel plot – SICSAG

• Note: as the model is poorly calibrated, most units are “better than average” – the funnel has been centred on the average SMR not 1


ww

w.icn

arc.o

rg

Over-dispersion• Variability more than expected by

chance• Suggests important factors that

vary among providers are not being taken into account

• Too many providers classified as “abnormal” (i.e. outside the funnel)


ww

w.icn

arc.o

rg Over-dispersion – hospital

readmissions

(Spiegelhalter. Qual Saf Health Care 2005)


ww

w.icn

arc.o

rg

Over-dispersion – what to do…?• Don’t use the indicator?• Improve risk adjustment• Adjust for it

– Estimate “over-dispersion factor” by “Winsorisation”

• Use random effects models– Assumes each provider has their

own true rate from a distribution


ww

w.icn

arc.o

rg

Example – over-dispersion factor

0.25

0.50

1.00

2.00

Mor

talit

y ra

tio -

ICN

AR

C (

200

9) m

odel

0 500 1000 1500Number of admissions

• SMRs by ICNARC model for 171 adult ICUs in England, Wales & N Ireland, 2009


ww

w.icn

arc.o

rg

Example – over-dispersion factors

0.25

0.50

1.00

2.00

Mor

talit

y ra

tio -

ICN

AR

C (

200

9) m

odel

0 500 1000 1500Number of admissions

Note: Overdispersion factor 1.4 based on 10% Winsorised

• Over-dispersion factor estimated at 1.4• Funnel widened


ww

w.icn

arc.o

rg


control• Comparison among providers• Continuous monitoring over time

– RAP chart– EWMA– VLAD– R-SPRT– CUSUM


ww

w.icn

arc.o

rg

Continuous monitoring over time• Various approaches• In general, they consist of…

– an indicator that is updated for each consecutive patient

– control limits


ww

w.icn

arc.o

rg Example for continuous

monitoring• Queen Kate Hospital• Fictitious critical care unit• Random sample of 2000 records

from the Case Mix Programme Database

• After 1000 records, outcomes changed so that an extra 6% of patients (selected at random) die

• Risk adjustment by the ICNARC (2009) model


ww

w.icn

arc.o

rg

Queen Kate Hospital – SMRs

0.7

0.9

0.8

1.0

1.2

1.4

1.6

Mor

talit

y ra

tio (

95%

CI)

Consecutive blocks of 250 patients


ww

w.icn

arc.o

rg

RAP chart• Risk-adjusted p chart• Cohort divided into discrete blocks

(e.g. 100 patients)• Indicator is observed mortality• Control limits are predicted mortality

+/- 2 or 3 SDs• Pro

– Displays observed and expected mortality

• Con– Still in blocks, not sensitive


ww

w.icn

arc.o

rg

Queen Kate Hospital – RAP chart

10%

20%

30%

40%

Mor

talit

y

0 500 1000 1500 2000Number of admissions

Observed Predicted 2 SDs 3 SDs


ww

w.icn

arc.o

rg

EWMA• Exponentially weighted moving

average• Similar to RAP but uses all data up

to the current timepoint• Data weighted by a smoothing

factor so that most recent data are given most weight


ww

w.icn

arc.o

rg

EWMA• Pro

– Displays observed and expected mortality

– Estimates updated continuously not in arbitrary blocks

• Con– Choice of smoothing factor is

important – too little smoothing and plot is unreadable, too much and plot is insensitive to changes


ww

w.icn

arc.o

rg

Queen Kate Hospital – EWMA

20%

25%

30%

35%

40%

Mor

talit

y


Observed Predicted +/- 2 SD 3 SD


ww

w.icn

arc.o

rg

VLAD• Variable life adjusted display• Cumulative observed minus

expected deaths• Pro

– Nice easy interpretation

• Con– Control limits are complex to

calculate curved functions


ww

w.icn

arc.o

rg

Queen Kate Hospital – VLAD

-20

0

20

40

60

Cum

ulat

ive

obse

rved

min

us e

xpec

ted

deat

hs



ww

w.icn

arc.o

rg

R-SPRT• Resetting sequential probability

ratio test• Tests evidence for/against a

specific hypothesis (e.g. odds of death are double that predicted by the model)

• Plot of log likelihood ratio• If bottom line is reached (strong

evidence against hypothesis) then line resets to zero


ww

w.icn

arc.o

rg

R-SPRT• Pro

– Nice statistical properties– Control limits are horizontal lines

• Con– Choice of hypothesis to test is

arbitrary – should we test for an OR of 2, 1.5,…?


ww

w.icn

arc.o

rg

Queen Kate Hospital – R-SPRT

-10

-5

0

5

10

RS

PR

T fo

r a

doub

ling

of o

dds

0 500 1000 1500 2000Case number

0.01 0.001 0.0001alpha = beta =


ww

w.icn

arc.o

rg

CUSUM• “Cumulative sum”• Log likelihood ratio – same as R-

SPRT• “Absorbing barrier” at zero (i.e.

never goes below zero)


ww

w.icn

arc.o

rg

CUSUM• Pros/Cons as for the R-SPRT plus…• Pro

– Does not allow credit to build up (as in R-SPRT) so alerts earlier

– Negative CUSUM (e.g. OR=0.5) can be plotted on the same axes

• Con– Cannot detect evidence against

hypothesis


ww

w.icn

arc.o

rg

Queen Kate Hospital – CUSUM

0

5

10

15

Log

likel

ihoo

d ra

tio fo

r O

R=

2


0.01 0.001 0.0001alpha = beta =


ww

w.icn

arc.o

rg

Which method(s) to use…?• Comparison among providers

– Funnel plot

• Continuous monitoring over time– EWMA– or R-SPRT– or CUSUM– (VLAD can be used as a display in

conjunction with, e.g., CUSUM for monitoring)

ww

w.icn

arc.o

rg

|

www.icnarc.org analysis and presentation of quality indicators dr david harrison senior...

Documents