engaging with data
TRANSCRIPT
Engaging with data
Martin Utley, Director of the Clinical Operational Research Unit,
University College London
Chair: Chris Sherlaw-Johnson, Senior Research Analyst,
Nuffield Trust
Building tools with clinical teams for use in monitoring outcomes.
Martin Utley
UCL Clinical Operational Research Unit
I received royalties from UCL Business in relation to co-authorship of the PRAiS software tool for monitoring 30-day survival following paediatric cardiac surgery.
Declaration
An insentient machine
? One can monitor outcomes to identify problems
One has absolute standards (some components to ¼ inch, others to a thousandth)
If there is a problem, the machine can be fixed.
To diagnose and fix problem, one has to stop production and open the box.
Stopping production comes at a cost, so one has to have a degree of
confidence that there is a genuine problem.
To what extent do these conditions apply to health care?
An insentient machine
Survival to 30 days following
paediatric cardiac surgery
Variable Life Adjusted Display (VLAD)
with Partial Risk Adjustment in Surgery (PRAiS)
Variable Life Adjusted Display (VLAD)
with Partial Risk Adjustment in Surgery (PRAiS)
Pagel C, Utley M, Crowe S, Witter T, Anderson D, Samson R, McLean A, Banks V, Tsang V, Brown K, Real time
monitoring of risk-adjusted paediatric cardiac surgery outcomes using variable life-adjusted display: implementation
in three UK centres, Heart (2013) 99:1445-50.
In use for local monitoring of outcomes at UK surgical centres since 2013.
In use for local monitoring of outcomes at UK surgical centres since 2013.
Dissemination to
wider clinical community
Clinical consultation / contribution / leadership
Development of risk
model
Design of data displays
In use for local monitoring of outcomes at UK surgical centres since 2013.
Dissemination to
wider clinical community
Clinicians led
engagement with surgical units,
National audit body & other stakeholders
All episodes in a 30% validation
sample within development set:
Mortality: 3.5%, N=7890
High risk diagnoses
Mortality: 8.8%, N=667
HLHS
Interrupted aortic arch
Pulmonary atresia (inc PA+IVS)
Comorbidity
Low risk diagnoses
Mortality: 1.4%, N=3196
Fallot/DORV-Fallot type
VSD
Pulmonary stenosis
Aortic arch obst +/- VSD/ASD
ASD
Subaortic stenosis (isolated)
Aortic regurgitation
Procedure
Medium risk diagnoses
Mortality: 4.2%, N=4027
PDA
Acquired
EMPTY/Unknown
Functionally uni-ventricular heart
TGA+VSD/DORV-TGA type
Pulmonary atresia+VSD (inc fallot type)
Tricuspid valve abnormality (inc Ebstein)
Mitral valve abnormality (inc supra, sub)
TAPVC
AVSD
Aortic valve stenosis
Misc Cong
TGA & IVS
Common arterial truncus
Normal
All episodes in a 30% validation
sample within development set:
Mortality: 3.5%, N=7890
High risk diagnoses
Mortality: 8.8%, N=667
HLHS
Interrupted aortic arch
Pulmonary atresia (inc PA+IVS)
Comorbidity
Low risk diagnoses
Mortality: 1.4%, N=3196
Fallot/DORV-Fallot type
VSD
Pulmonary stenosis
Aortic arch obst +/- VSD/ASD
ASD
Subaortic stenosis (isolated)
Aortic regurgitation
Procedure
Medium risk diagnoses
Mortality: 4.2%, N=4027
PDA
Acquired
EMPTY/Unknown
Functionally uni-ventricular heart
TGA+VSD/DORV-TGA type
Pulmonary atresia+VSD (inc fallot type)
Tricuspid valve abnormality (inc Ebstein)
Mitral valve abnormality (inc supra, sub)
TAPVC
AVSD
Aortic valve stenosis
Misc Cong
TGA & IVS
Common arterial truncus
Normal
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Year
Percentage episodes with empty co-morbidity fields
New ways of using diagnostic codes
clinically led / meaningful
then simplified for purposes of
risk model
Any old way of including comorbidity
knowingly included poor quality data
acknowledging clinical nous
commitment to long game
Development of risk
model Old ways of using operation codes
accepted existing schemes
impact on model “performance”
impact of model use
Year
% missing
comorbidity
data
PRAIS 1:
• 29 Specific procedures
• 3 diagnosis groups (low, medium, high)
• UVH
• Yes/no comorbidity
• Bypass/non-bypass
• Continuous weight
• Continuous age
• Age band (neonate/infant/child)
• Epoch effect
38 parameters
PRAIS 2:
• 16 spec proc groupings
• 11 diagnosis groupings
• UVH
• 4 “other illness” indicators
• Bypass/non-bypass
• Continuous nonlinear weight
• Continuous nonlinear age
• Epoch effect
36 parameters
PRAiS2 – developed on 21838 surgical episodes 2009-2014 from NCHDA data.
Recent update
VLADs do not give an evaluation of the
effectiveness / benefit of a surgical
programme to the patient population.
VLADs are not evidence of good or bad
performance.
DATA
patient case-mix
structure / resources
care processes
professionals
Queensland
pyramid model of
investigation
Worked with 3 sites on design of software
We improved ease of data entry / use, shared strengths and
limitations, listened to concerns.
Consultation on design of data displays
Added after consultation with units
Secondary outcomes we would have discarded if not for discussions with clinical team
Weeks following release of PRAiS 2013
Christina Pagel - How can we better support appropriate interpretation
and use of risk adjusted survival data?
PRAiS 1 launch
What?
How?
Why? PRAiS2
Research team worked with clinicians, family representatives, press officers and journalists on resources to support understanding and use of risk-adjusted mortality data.
childrensheartsurgery.info
Launch childrensheartsurgery.info
Launch childrensheartsurgery.info
What?
How?
Why?
Funding NIHR HSDR PI Victor Tsang
Health Foundation PI MU
NICOR
Funding NIHR HSDR PI Christina Pagel
Funding NIHR HSDR PI Victor Tsang & Kate Brown
Funding GOSH Charity PI Kate Brown
None of the views expressed are those of DH, NIHR, Health Foundation, GOSH Charity etc.
2014/15 - 4,207 surgical episodes with 97 deaths (mortality rate 2.3%) – 89 deaths predicted Area under ROC curve 0.86 [0.82, 0.89]
PRAiS2 performance in test data
Brown KL, Crowe S, Franklin R, McLean A, Cunningham D, Barron D, Tsang V, Pagel C, Utley M, Trends in 30-day mortality
rate and case mix for paediatric cardiac surgery in the UK between 2000 and 2010, Open Heart. 2015 Feb 14;2(1):e000157.
doi: 10.1136/openhrt-2014-000157.
Absolute survival rates completely absent from debate in 2013
Kate Brown
PRAIS 2:
• 16 spec proc groupings
• 11 diagnosis groupings
• UVH
• 4 “other illness” indicators
• Bypass/non-bypass
• Continuous nonlinear weight
• Continuous nonlinear age
• Epoch effect
36 parameters
Performance assessed using cross-validation in main data set and prospective use in 2014/15 national data set.
PRAiS risk model recently updated
Close collaboration with an expert panel including of intensive care consultants, surgeons, cardiologists, data experts, analysts