challenges of evaluating the impacts of ambulatory ... · challenges of evaluating the impacts of...
TRANSCRIPT
© Nuffield Trust28 July 2017
Challenges of Evaluating the Impacts
of Ambulatory Emergency Care
Paul Smith
Martin Bardsley
Nuffield Trust
© Nuffield Trust
• Promote independent analysis
and informed debate on
healthcare policy across the UK
• Charitable organization founded
in 1940
• Formerly a grant-giving
organization
• Since 2008 we have been
conducting in-house research
and policy analysis
• Significant interest in uses of
data linkage and predictive risk
techniques
The Nuffield Trust
William Morris
1st Viscount Nuffield
(1877 -1963)
© Nuffield Trust
Concern about trends in emergency admissions
Source: A&E Annual activity statistics, NHS and independent sector organisations in England
© Nuffield Trust
Nuffield work includes studies of….
Telehealth and Telecare – Whole System Demonstrator in 3 areas
National Integrated Care Pilots
Partnerships for Older People*
Birmingham Own Health
Virtual Wards in 4 sites*
Marie Curie Nursing Service*
NW London Integrated Care Pilot
British Red Cross ..Care in the Home
Not clear what works see Purdy et al (2012) Interventions to Reduce Unplanned Hospital
Admission: A series of systematic reviews. Bristol University Final Report)
© Nuffield Trust
Cautions in evaluation……
1. Timing….Recognise that planning and implementing large scale
service changes take time
2. Fidelity….Define the service intervention clearly – and be clear when
the model is changed
3. Sample sizes…If you want to demonstrate statistically significant
change, size and time matter
4. Range of outcomes….Hospital use and costs are not the only impact
measures
5. Evaluation methods….Carefully consider the best models for
evaluation – prospective/retrospective; formative/summative;
quant./qualitative
6. Monitoring change in process
© Nuffield Trust
Specific potential problems with evaluating AEC
• No standard AEC dataset therefore data collection inconsistent
• AEC is changing over time (number of pathways, referral routes etc.)
• Different models and interpretations of AEC and large variation as to the
range of conditions managed (DVT most common)
• Sites vary in the scale of implementation (e.g. hours service available, size
of designated ambulatory care areas)
• Majority of sites around country provided some form of ambulatory care in
2009/10 (McCallum et al, 2010) so control groups problematic
• Identifying conditions consistently over time in routine data also complicated
by coding changes (mainly HRGs)
• The year service introduced and the extent to which AEC developed is
highly variable (at certain trusts services began before 2002)
• The field of emergency and urgent care is changing rapidly so may be
difficult to isolate an AEC effect
© Nuffield Trust
Evaluation goals and analysis decisions
Two main research questions
1. Do we see an overall AEC effect?
2. Do we see network-specific effects?
• HES inpatient data from 2009/10 to 2014/15 used as majority of sites should be
using admitted patient data
• Focus mainly on 18 of the clinical scenarios found in the Directory of Ambulatory
Care that are either included in the PbR Best Practice Tariffs or were reported as
the conditions most commonly managed by sites
• Reduction in bed days for the 18 clinical scenario chosen as our main measure
as this would be expected to decrease if AEC effective irrespective of dataset
used
• Limited to patients aged 18 and over (children excluded from BPTs) and
admissions occurring Monday to Friday
• Comparator groups selected were based on membership of AEC network
© Nuffield Trust
Analysis cohorts
We used four main groups for comparison based on when the trusts joined the
AEC network :
• Cohort 1 - joined the network in September 2011, n = 10
• Cohort 2 – joined the network in September 2012, n = 12
• Cohort 3 – joined the network in April 2013, n = 10
• Remaining providers were used as a “control” (although not strictly true as
many will have AEC services), n = 110 (and 29 of these subsequently joined
the network). Specialist providers excluded from this cohort
• In total we examined 20,516,751 emergency admissions to English hospitals
between April 2009 and March 2014, encompassing 118,532,710 emergency
bed days
© Nuffield Trust
Summary
• Most sites do appear to be recording AEC activity using the admitted patient
dataset though certainly not the case for all
• Bed days appear to be the most clear-cut way of examining the impact of AEC
• There is evidence that AEC has helped reduce acute bed days for certain
conditions (with caveats around controls)
• There is also evidence of AEC network membership having an effect on bed day
reduction for specific conditions (again more thorough control groups may need
to be established though)
• There is variation between conditions in both the extent to which bed days have
been reduced, and the proportion of same-day discharges
© Nuffield Trust
Improvements and potential future work
• More rigorous definition of control groups – either drawing on more detailed
local knowledge to derive trust comparator groups or draw controls from a
different time period (say pre-AEC)
• More sophisticated modelling e.g. propensity matched controls and tracking
specific cohorts of patients who used the AEC service
• Fuller/multi-method evaluation capturing patient and staff views/experience,
locally defined metrics, and cost-benefit analysis
• More fully explore the potential for AEC – large variation between conditions
and how much of this based on casemix and how much due to the maturity of
the specific pathway?
© Nuffield Trust28 July 2017
www.nuffieldtrust.org.uk
Sign-up for our newsletterwww.nuffieldtrust.org.uk/newsletter
Follow us on Twitter:Twitter.com/NuffieldTrust
© Nuffield Trust