The Selection of Early Warning Scores

Dana P. Edelson, MD, MS, FAHA

Assistant Professor, Section of Hospital Medicine

Executive Medical Director for Inpatient Quality and Safety

NTI – May 2017

Disclosures

Employment:

• The University of ChicagoResearch support:

• National Heart Lung Blood Institute of NIH

• Philips Healthcare

• Early SenseOwnership interests:

• Founder & CEO, QuantHC

• Patent pending, ARCD.P0535US.P2Other:

• Immediate Past Chair, Systems of Care Subcommittee, American Heart Association

• Co-Chair, GWTG-R Adult Research Task Force

• Member, CDC Ward Sepsis Working Group

Hospitalization Time Course

Each 1 hr delay in ICU transfer is associated with a 3% increased odds of mortality

Traditional RRT calling criteria

Call if any of these criteria are met:

Threatened airway*

Respiratory rate <5

Respiratory rate >36

Heart rate <40

Heart rate >140

Systolic Blood Pressure <90

Drop in Glasgow Coma Scale >2

Prolonged seizure activity*

4Hillman, Lancet, 2005

5

Traditional Statistics: Baseball v. Healthcare

6

Rethinking baseball statistics

7

Modified Early Warning Score (MEWS)

8

Score 3 2 1 0 1 2 3

Respiratory rate (RPM) — ≤ 8 — 9-14 15-20 21-29 ≥ 30

Heart rate (BPM) — ≤ 40 41-50 51-100 101-110 111-129 ≥ 130

Systolic BP ≤ 70 71-80 81-100 101-199 ≥ 200

Temperature (°C) — ≤35 — 35.0-38.4 — >38.5 —

AVPU — — — AlertReact to Voice

React to Pain

Unresp

Churpek, Chest, 2013

9

National Early Warning Score (NEWS)

Churpek, Chest, 2013

10

But health care has moved into the digital age

Running a paper-based tool on a computer is like …

11

Harnessing big data analytics for IHCA prevention

12

Introduction of eCARTTM– linear logistic regression

n=56,649 admission from one hospital Churpek, CCM 2014

Churpek, M, et al. Crit Care Med. 2013

eCARTTM 2

• Cubic spline logistic regression

• Utilizes 33 EHR variables: vitals, labs, demographics

• Derived and validated in >250,000 patients from five hospitals

eCART 2TM – cubic spline logistic regression

n=269,999 admission from five hospitals Churpek, AJRCCM 2014

Accuracy: eCART vs MEWS

16

Challenge: $1,000,000 to improve rental suggestions

17

Machine learning models are more accurate

Churpek, CCM, 2016

0.698

0.735

0.735

0.754

0.770

0.783

0.786

0.789

0.795

0.801

0.50 0.55 0.60 0.65 0.70 0.75 0.80

MEWS

Logistic regression (linear)

Decision tree

K-nearest neighbors

Logistic regression (splines)

Neural network

Support vector machine

Bagged trees

Gradient boosted machine

Random forest

AUC

% of observations detected that are followed by an outcome

Perc

ent

of

po

pu

lati

on

ne

eded

to

scr

een

Model accuracy comparisons

Churpek, CCM, 2016

% of observations detected that are followed by an outcome

Logistic regression (linear)

Random ForestLogistic regression (splines)

MEWS

Model accuracy comparisons

50,000 fewer false positives with random forestcompared to MEWS

Perc

ent

of

po

pu

lati

on

ne

eded

to

scr

een

Churpek, CCM, 2016

50,000 Fewer False Alarms With Random Forest over MEWS

21

Variable importance in the random forest model

22Churpek, CCM, 2016

Random forest visualization: Getting old is bad

40 is not the new 20

Age (years)

Ris

k

Churpek, CCM, 2016

Respiratory rate

Ris

k

Churpek, CCM, 2016

Random forest visualization: respiratory rate is key

Age

20

40

60

80

100

Respiratory rate 20

40

60

Ris

k o

f Event

1.4

1.6

1.8

Random forest visualization: predictors interaction

26

Systemic Inflammatory Response Syndrome (SIRS) criteria

27

Most ward patients meets SIRS criteria at some point

28

More specific than SIRS but less sensitive

29

When infection is suspected: eCART>MEWS>qSOFA>SIRS

30ATS, 2017

Real-time data-driven decision support in action

High Risk Moderate Average

The University of Chicago Medical Center

617 beds

28,726 inpatient admissions

0% 20% 40% 60% 80% 100%

Percent of Events Captured

RRT Called

ICU

Transfer

[n=383]

eCART

Cardiac

Arrest

[n=10] High risk eCART threshold met 31 hours prior to RRT

(High/Mod Risk)

Kang, CCM, 2016

Silent phase eCART implementation

Custom implementation at the University of Chicago

eCART imbedded within the EHR

Building in clinical decision support

eCART Imbedded in the Philips Guardian Platform

Summary

Data-driven early warning scores outperform traditional expert consensus

Machine learning algorithms further improve detection and reduce false alarms

Pick the best early warning score you can and skip the sepsis screening tool

Imbed it into your workflow at the point of care