bcf scheme measurement and evaluation workshop · pdf file 1 the better care fund bcf scheme...
TRANSCRIPT
www.england.nhs.uk 1
The Better Care Fund
BCF Scheme Measurement and
Evaluation Workshop
(NHS England in Partnership with London
ADASS)
www.england.nhs.uk 2
Neighbourhood Rm, Coin Street Conference Centre, 108 Stamford St, South Bank, London, SE1 9NH
Tuesday 25th April 2017
@ 9 – 13:30 pm
BCF Schemes Metric and Evaluation Workshop
1. Registration and Refreshments 9:00 – 9:30 am
2. Welcome, Introductions and Regional Context - Discussion 9:30 – 9:55 am Eileen Sutton & Jane Hannon - NHS England
3. Nuffield Report and Evaluating Shifting the Balance of Care – Discussion and Q & A 9:55 – 10:35 am Natasha Curry - Nuffield Trust
4. Practical Lessons and Tips from Evaluating NHS New Care Models and Vanguards – Discussion and Q & A 10:50 - 11:30 am
Michael Lawrie - NHS England
5. Developing an evaluation framework to support areas in assessing the impact of their schemes – - Discussion and Q & A Anne Jarrett & Fiona Russell - Local Partnerships 11:30 – 12:10 pm
6. Measurement and analysis for Improvement – - Discussion and Q & A 12:40 – 13:20 pm Susanna Shouls - NHS Elect
7. Round up 13:20 – 13:30 pm
Break
Lunch
www.england.nhs.uk
BCF Schemes
Metric and
Evaluation
Workshop
www.england.nhs.uk
Context
4
• Local BCF leads have highlighted evaluating the impact of BCF schemes as a challenge
• Good practice regionally includes :
o Dashboards monitoring overall performance (Camden and Southwark have shared their work)
o Tracking outcomes for cohorts using NHS number to identify patients
o Measuring outputs and outcomes for individual schemes, including patient satisfaction
o Monitoring overall HWB area outcomes and patient satisfaction levels
o Funded external evaluations
• HWB areas have been grappling with what reasonable assumptions are when linking
scheme outcomes and outputs with systems level outcomes
• There is more expectation than ever on the BCF schemes to support system change and
there is an appetite to form a clearer picture of “what good looks like”
• Today we’ve brought together some experts to help us think about these questions.
• We’ll be asking you to share learning, views and ideas via post-it notes on the four flipcharts
• We’d like to start by asking you to take 10 minutes identify in your tables at least one
element in this area that is working well and an area of challenge
www.england.nhs.uk 5
The Better Care Fund
Natasha Curry
Senior Fellow in Health Policy
Nuffield Trust
Natasha Curry, Senior Fellow in Health Policy, Nuffield Trust
Shifting the balance of
care
Today’s presentation
7
1. Shifting care out of hospital: Why do it?
2. Community-based initiatives: What does the evidence say?
3. Shifting care out of hospital: why is it so hard?
4. Top tips for implementation and evaluation
5. Q&A
Shifting care out of hospital: why do
it?
8
The Triple Aim:
1. It’s cheaper
2. It improves quality of patient care
3. It improves health and wellbeing
Multiple policy interventions, little
shift
9
2000 2006 2013 2014
“Shift in the centre
of gravity of
spending.”
“Significant
expansion of care in
community settings.”
“Out-of-hospital care
needs to become a
much larger part of
what the NHS does.”
“Ease the pressure
on hospitals.”
But now there’s extra imperative
Trends in hospital activity 10
Financial pressures
Reference: Robertson et al (2017) The King’s Fund 11
£22bn gap by
2020
STP assumptions to bend the
demand curve & save money
12
Area No. of
STPs
Min
Reduction
Max
Reduction
Average by
2020/21
Outpatients 19 7% 30% 15.5%
Elective Inpatients 22 1.4% 16% 9.6%
A&E attendances 26 6% 30% 17%
Non-elective
inpatients
30 3% 30% 15.7%
What do systems leaders think?
13
Care in the community is cheaper and provides better
care for patients. 38%
Care in the community provides better care for
patients but is not cheaper. 38%
Care in the community is cheaper but does not provide better care for
patients. 3%
Care in the community is neither cheaper nor
provides better care for patients.
7%
I am not sure. 14%
With regard to moving care out of hospitals, which of the following statements most accurately reflects your view?
(n=58)
What does the evidence say?
What we did
15
• Reviewed evidence for 27 common initiatives intended to reduce hospital utilisation
• High quality evidence (Cochrane, RCT, systematic reviews) with ‘grey’ literature where other evidence not available
• Focused on cost savings but recognised other values
Categorised initiatives into:
1. Positive evidence re reduced costs/activity
2. Mixed or emerging evidence re reduced costs/activity
3. Evidence to suggest increased costs/activity
The state of the evidence
16
• Limited
• Small studies
• Many are poorly-constructed
• Most are single-disease focused
• Few focus on cost
• Few consider the whole system
• Context-specific
• But, many demonstrate positive impacts in terms of patient experience or outcomes
Evidence suggests some initiatives may
reduce activity and save money
Most Positive Emerging positive
Remote monitoring of people with certain
LTCs
Patients experiencing GP
continuity of care
Improved end-of-life care in the
community
Extensivist model of care for
high risk patients
Condition specific rehabilitation Social prescribing
Targeted support for self care Senior assessment in A&E
Additional clinical support to people in
nursing and care homes
Rapid access clinics for urgent
specialist assessment
Improved GP access to specialist
expertise
Ambulance/paramedic triage to the
community
Evidence suggests some initiatives may
reduce activity and save money
Most Positive Emerging positive
Remote monitoring of people with
certain LTCs
Patients experiencing GP continuity
of care
Improved end-of-life care in the
community
Extensivist model of care for high
risk patients
Condition specific rehabilitation Social prescribing
Targeted support for self care Senior assessment in A&E
Additional clinical support to people
in nursing and care homes
Rapid access clinics for urgent
specialist assessment
Improved GP access to specialist
expertise
Ambulance/paramedic triage to the
community
Evidence suggests some initiatives may
reduce activity and save money
Most Positive Emerging positive
Remote monitoring of people with
certain LTCs
Patients experiencing GP continuity
of care
Improved end-of-life care in the
community
Extensivist model of care for high
risk patients
Condition specific rehabilitation Social prescribing
Targeted support for self care Senior assessment in A&E
Additional clinical support to people
in nursing and care homes
Rapid access clinics for urgent
specialist assessment
Improved GP access to specialist
expertise
Ambulance/paramedic triage to the
community
Many initiatives may not save or may cost money
Mixed – re £ + activity May cost ££
Case management and care
coordination
Extending GP opening hours
Intermediate care: rapid response
services
Specialist support from a GP with a
special interest
Intermediate care: bed-based
services
Consultant clinics in the community
Hospital at Home NHS 111
Shared care models for the
management of chronic disease
Urgent care centres including minor
injury units (not co-located with A&E)
Virtual ward Referral management centres
Shared decision making to support
treatment choices
Direct access to diagnostics for GPs
Many initiatives may not save or may cost money
Mixed – re £ + activity May cost ££
Case management and care
coordination
Extending GP opening hours
Intermediate care: rapid response
services
Specialist support from a GP with a
special interest
Intermediate care: bed-based
services
Consultant clinics in the community
Hospital at Home NHS 111
Shared care models for the
management of chronic disease
Urgent care centers including minor
injury units (not co-located with A&E)
Virtual ward Referral management centers
Shared decision making to support
treatment choices
Direct access to diagnostics for GPs
Many initiatives may not save or may cost money
Mixed – re £ + activity May cost ££
Case management and care
coordination
Extending GP opening hours
Intermediate care: rapid response
services
Specialist support from a GP with a
special interest
Intermediate care: bed-based
services
Consultant clinics in the community
Hospital at Home NHS 111
Shared care models for the
management of chronic disease
Urgent care centers including minor
injury units (not co-located with A&E)
Virtual ward Referral management centers
Shared decision making to support
treatment choices
Direct access to diagnostics for GPs
Why is it so hard?
A gap between theory and practice
“Improvement initiatives are sometimes planned on the hard high ground, but are put into effect in the swampy lowlands.”
- Marshall and others, 2016, BMJ Quality & Safety
© Kenneth Allen
Implementation needs to take wide
range of factors into account
• Context matters
• Requires rigorous framing of the problem and contextual factors that could influence feasibility and effectiveness
• Including influencing professional behaviour such as attitudes to risk
Source: Imison and others, 2012 25
Bed use
System governance factors
• Governance models
• Commissioner behaviour/ relationships
• Provider behaviour/ relationships
• Staff beliefs and values
• Leadership
Hospital factors (supply side)
• Access (rurality)
• Internal processes – admission, treatment and discharge
Community factors
• Primary care supply and capacity
• Community care supply and capacity
• Local authority care supply and capacity
Patient factors (demand side)
• Age
• Socioeconomic status
• Sex
• Health needs
• Beliefs and values
Community
An unequal battle?
Hospital
26
Primary and community care facing
significant challenges
27
• 1/3 GP practices have a vacancy for at least one partner
• 2016 - NHS England identified 20% GP practices as vulnerable
• 1/5 district nurse posts vacant
Savings are difficult to realise in
reality
28
• Some interventions identify unmet need = increase activity
• New services can fuel supply-induced demand
• Savings depend upon capacity being taken out of hospital
What should you be mindful of in implementing & evaluating an initiative?
When is evidence ‘good enough’?
30
• Hierarchy of evidence
• Pragmatism: rigour versus
reality
• What/who is the evidence for?
Source: NIHR Evidence Works
Top tips for implementing &
measuring change
31
1. Be realistic: Planning & implementing large-scale change takes time
2. Ensure aims are clear and be explicit about desired outcomes
3. Hospital use and cost are not the only measures
4. Be flexible: Monitor and measure as you progress and adapt accordingly
5. Understand your data: what do you have? What is it telling you?
Top tips for implementing &
measuring change
32
6. Relationships matter: sets of processes can only achieve change if underpinned by good relationships and a shared vision
7. Engage staff: ultimately, they will deliver the change
8. Invest in leadership
9. Don’t get bogged down in governance: let form follow function
10. Context matters and it changes
Conclusion
33
• This agenda presents a huge challenge
• Shifting the balance of care is not easy and will take time
• Nobody can argue against the principle of better, more appropriate care closer to home
• But we cannot assume that this will save money, especially in the short term
• To succeed, we need a relentless focus on what works & constant monitoring
www.nuffieldtrust.org.uk
Follow us on Twitter – twitter.com/NuffieldTrust
Sign up for our newsletter – www.nuffieldtrust.org.uk/newsletter-signup
www.england.nhs.uk 35
The Better Care Fund
Michael Lawrie
Operational Research and Evaluation Unit
NHS England
www.england.nhs.uk 36
Lessons learned from evaluating
national transformation programmes
Michael Lawrie
Operational Research and Evaluation Unit, NHS England
www.england.nhs.uk
• NHS England’s Operational Research and Evaluation Unit was formed in 2015
• Around 35 mixed methods analysts working on a range of national programmes .
• Economists, social researchers, statisticians, operational researchers.
• We aim to embed robust evaluations, support the adoption of rapid cycle evaluation approaches (where suitable) and work alongside programmes to help them be as evidence based as possible.
• The new care models programme is currently our largest area of work.
37
Introduction
www.england.nhs.uk
• To provide some advice on how to undertake high quality evaluations, drawing on our experience of evaluating NHS England transformation programmes.
• The session will be structured around our five step rapid cycle evaluation approach.
• I propose to open up for questions after each of these five steps so that the session is as tailored to your queries as possible.
38
Aims and structure of the session
www.england.nhs.uk 39
Rapid cycle evaluation Specify the desired outcomes and
the broad policy parameters within which the programme should be
implemented and, to the extent it exists, the evidence-informed
common recipe for change.
Describe how the programme is supposed to work. Articulate, through a logic model, the proposed
interventions and the causal chain linking them to
the desired outcomes.
Measure impact through comparison with what would have
occurred (the counterfactual)
Measure what changes are being made and what is happening as a
result.
Learn, adapt and improve
www.england.nhs.uk 40
1. Specify the desired outcomes, policy parameters
and the evidence informed case for change
• Agree the evaluation questions early and prioritise them;
• Agree a range of outcome measures. Commonality will aid neatness when aggregating data (especially at a national level); but also embrace the variety with local sites defining local measures;
• Measure a few things well. But ensure that a varied mix of outputs and outcomes are established;
• Decide whether to focus evaluation resources in on a particular initiative, or on a full programme evaluation.
www.england.nhs.uk 41
A set of simple evaluation questions
that we have tried to stick to…
What is the model and how is it intended to work?
What changes are being made and what is happening as a result?
What is the impact?
What is the cost?
What is causing the impact?
What should be replicated and spread?
How should x, y, z be implemented?
www.england.nhs.uk 42
2. Describe how the programme is supposed to
work.
• We advocate the use of logic models to describe how a programme is supposed to work.
• It provides a hypothesis for the evaluator to test.
• Producing a logic model can help to tease out the full range of value of a programme.
www.england.nhs.uk 43
The activity
funded connects
to…
The outcomes of
the funding…
And onto the
longer-term
impacts…
If we deliver training to people who are
unemployed then they will improve their skills;
If they improve their skills, then they will gain in
confidence or gain a qualification;
If they gain in confidence or gain a
qualification, then they will obtain more job
interviews and job offers;
If they get a job, then their income will rise, and
there will be reduced unemployment.
www.england.nhs.uk 44
1. What is the rationale for your intervention?
[What is the problem you are trying to address?]
2. Given your rationale, what impacts are you ultimately trying to achieve?
[What would be the end measure of success for you?]
3. What short- to medium-term outcomes would generate those impacts?
[What immediate benefits would you like to see before you can achieve these longer-term success measures?]
4. What type of activities would generate those outcomes?
[What will you do differently / new practices will you introduce in your area to achieve these immediate changes?]
5. Given all of the above, what resources are required?
[What will you need to enable you to work differently / introduce new practices]
www.england.nhs.uk 45
Inputs
These are the resources available to
deliver the intervention.
This may be in terms of specific cash
funding or ‘in-kind’ contributions.
It is relatively straightforward to put a monetary value on
inputs and, in a framework of costs and benefits, inputs
are the costs.
Evaluation at this level is about
economy and the resources consumed.
Activities & Outputs
These are the things that an intervention
does, e.g. provide workshops, practical support, training etc.
Outputs are quantitative
measures of this activity, e.g. No.s of:
• Beneficiaries•Materials distributed
• Sessions held
Evaluation at this level concerns
implementation and efficiency (the
relationships between inputs and outputs).
Short-termOutcomes
It is often useful to distinguish between short- and medium-
term outcomes.
Short-term outcomes can be defined as
changes in knowledge / awareness / attitude
– e.g. ‘beneficiaries have an increased awareness of ...’
This is based on a simple model of
behavioral change, which suggests that
these changes precede changes in
behaviour or condition.
Long-termImpacts
This is the final, high-level effect of the intervention – e.g.
‘Improved life expectancy, reduced health inequalities’.
This relates closely to the original rationale
for intervention.
Impacts are subject to a very wide range of
other contextual influences (e.g.
combinations of other policies, programmes, economic conditions),
- illustrated by the very permeable line
around this box.
Context to the InterventionThese are the wider economic, social, environmental, and policy conditions. This is very important: interventions do not
take place in a vacuum and contextual factors affect the intervention and its results.
Medium-termOutcomes
Medium-term outcomes are
changes in behaviour or condition – e.g.
‘beneficiaries increase levels of physical activity’
In describing any outcomes, language suggesting change
(‘increased, reduced’) is useful.
Evaluation here is about effectiveness.
The relationship between inputs and
outcomes is the basis for cost-effectiveness / cost-benefit studies.
Rationale for Intervention
This is the justification for the selected intervention, e.g.: what is the nature and scale of the specific problem being addressed? What will happen if we ‘do nothing’? Why this intervention and not alternatives?
Source: Midlands and Lancashire CSU
www.england.nhs.uk
• One of the central evaluation questions is to try to
understand what would have happened in the
absence of the programme intervention compared to
what did happen – the impact question.
• There are numerous ways in which to do this, and the
next few slides show varying ways in which to do this
based on real life examples.
46
3. Measure impact through comparison with what
would have occurred (the counterfactual)
www.england.nhs.uk 47
Tracking the time series
www.england.nhs.uk 48
Use of simple comparators
www.england.nhs.uk 49
Use of statistical process control
www.england.nhs.uk 50
Use of statistical process control
www.england.nhs.uk 51
Controlling to counteract regression
to mean
Source: Health Foundation
www.england.nhs.uk 52
Looking at data at a more granular
level
Source: Midlands and Lancashire CSU
www.england.nhs.uk 53
4. Measure what changes are being made
and what is happening as a result
• Outputs and early outcome measures are important
leading indicators of change. E.g. patient activation
E.g. Clinician training
Initial median score 51.00
Follow up median score 60.60
Median difference 9.60
Target: Average follow up score to be at
least 50 from April 2017
Source: Vanguard local metrics return
Source: Vanguard local metrics return
www.england.nhs.uk 54
4. Measure what changes are being made
and what is happening as a result
• Input metrics are also important to consider – e.g.
what are staff doing differently in order to deliver the
changed service in question (skill mix, hours worked,
location of work).
• Qualitative insights are crucial here, both to
understand how things are changing and to explore
chains of causality. E.g. some of the vanguard
evaluations are using embedded evaluators
(Researchers in Residence).
www.england.nhs.uk
• Collate and synthesise all available evidence. Triangulate to draw firm conclusions about how the initiative (set out in the logic model) is working and test the causal chains therein.
• If resources are not available to undertake this robustly across a full programme, look in depth at particular parts of your programme – case studies.
• Set up feedback loops which will allow the evaluator / analyst to feed in regular findings to programme decision makers.
• Ensure a diversity of data are available – different stakeholders require different standards and types of evidence.
55
5. Learn, adapt and improve
www.england.nhs.uk 56
The Better Care Fund
Anne Jarrett & Fiona Russell
Operational Research and Evaluation Unit
Local Partnerships & Local
Government Authorities
The Better Care Fund
asdf
Fiona Russell
[email protected]; 07799 466328
Anne Jarrett
[email protected]; 07917 813829
25 April 2017
Supporting the evaluation of local
BCF schemes
BCF bespoke support programme for 2017-19
• Part of a suite of support commissioned by the national Better Care Support Team,
which also includes:
o Masterclasses and Regional Networking Events
o Regional Support Funding
Better Care Adviser and Multi-Disciplinary Support:
Programme objectives:
Further support:
• Delivered by LGA, as a national partner in the Better Care Support Team, with NHS
IMAS and Local Partnerships
• To secure agreement on a compliant BCF plan for 2017-19
• To support implementation of BCF schemes or performance improvements against
the BCF metrics
• To support local areas at all levels to drive their integration plans forward and achieve
greater integration of health and care, including preparing for graduation from the BCF
• To strengthen leadership to lead integration locally
The evaluation challenge… • Evaluation questions about local schemes are being framed in different
ways but there appears to be a real challenge in trying to answer them
Local BCF
schemes
Is the scheme
‘effective’?
Is it delivering
the outcomes
we expected?
How can we deliver more from
the scheme?
Is it contributing
to the achievement of BCF plan objectives?
Does it represent ‘value’ for
BCF investment?
Should we continue
the scheme?
The data challenge • Can we drawn on national data and best practice to develop a simple
framework to help support local areas with scheme evaluation?
• Can we drill down into the national data set to help frame questions about local scheme evaluation?
BCF NATIONAL DATA
• Can we identify and share effective approaches to local scheme evaluation?
LOCAL BEST PRACTICE
Develop a local framework • Evaluating the performance of local schemes
On a scale of 1 – 10 where 1 is “not at all” and 10 is “to a great
extent”, indicate the extent to which each scheme: N.B. Here are some example headings that we think should be included – however please adapt and add to these as you feel works for your local context.
Total up the
scores, and take
the lowest scorers
through the
process on the
next page.
Today’s exercise – developing a local framework
• Three tables – each taking on of the most common scheme types
(Reablement; Care at home services; Integrated care teams)
• The task is to populate a local framework, considering the following:
How would you know your chosen schemes are delivering:
• Improving outcomes
• Value for money in the long term
• Long-term capacity building for integration and new care models
• Improving patient/service user satisfaction
• Shifting provision towards prevention and community
What are the challenges of trying to answer these questions locally
What are you doing about evaluation that is proving useful to you?
What sort of support would you find valuable?
www.england.nhs.uk 63
The Better Care Fund
Susanna Shouls
Operational Research and Evaluation Unit
NHS Elect
Measurement and analysis for
Improvement
Susanna Shouls Measurement & Analysis
www.nhselect.nhs.uk Twitter @NHSElect
Why measure?
“The third habit is measurement and oversight. For many measurement of clinical operations is driven by external audiences: payers, regulators and rating agencies. Although high-value organizations share this reporting obligation, they primarily use measurement for internal process control and performance management.”
• More specific measures
• Customised to their processes
• Integral to accountability and strategic aims
Model for improvement
“Do something and see what happens” Paul Plsek, 2016
Mark Wallinger
Art on the Underground
Measurement throughout a project
Potential for improvement
Getting a baseline Did it make a
difference?
What needs to be in place to mainstream
improvement?
Evaluating worth
(benefits, costs)
A P
D S
A P
D S
A P
D S
A P
D S
A P
D S
Seven steps – measurement for improvement To reduce unplanned admissions to my acute acute hospital Outcome: ?number of unplanned admissions Process: ?number of patients seen in my alternative solution Balancing: ?a&e attendance ?satisfaction
Consistency is key here!
Baseline data …
Step 1: Decide your aim
Features of a good aims statement
A worthwhile topic
Outcome focused
Measurable
Specific population
Clear timelines
Succinct but clear
Adapted from
Tom Nolan in
The
Improvement
Guide
Beware Weasel words
“Weasel words are words that have no specific and obvious and singular meaning.
They bring no clear images to mind of what is meant.”
best practice, effective, evidence-based, excellence, high quality, high value, responsive, value, value-added, world class and many more!
Source: Stacey Barr quoting Don Watson
A word to watch out for
Does your aim include the word “by”?
For example:
“We want to reduce the number of complaints by giving our staff customer care training.”
Step 2: Select measures
?Already started?
?Use what exists?
Driver Diagrams Schematic view of a system on the left we depict outcome and as
we move right we drill down into the network of causes that drive the outcome, from ‘primary’ to ‘secondary’ drivers
Driver Diagrams
On the right we depict ideas for system changes that might ultimately impact the outcome. The diagram represents our theory about how to modify the system to change the outcome
Aim:
2 stones
lighter!
Energy Out
Energy In
Walk daily
commute
Stairs not
lift
Exercise
Reduce
alcohol
intake
Eat Less
Pedometer
Gym work
out 3 days
Squash
weekends
No pub
weekdays
Take
packed
lunch
Low fat
meals
Driver Diagrams - weight loss
Exercise 1: Discuss your aims and your key drivers
• You have 10 minutes
Understanding populations
2x2 matrix approach
Was the patient
suitable for AEC?
Was the patient seen in
AEC?
Yes
No – admitted
Yes
Right patient, right place
Missed opportunity
No
Wasted resource
- admission
- A&E
- ?avoid attendance
Right patient, right place
Assessing right place, right time of care.
Using AEC “same day emergency care” to illustrate the 2x2 matrix
Does this graph help us?
How about this?
Celebration of success???
• If 1/3 patients seen in AEC would not have previously been admitted ….
• But avoiding admission in 2/3 …
• Is this efficient or not? And how did we know?
Qualitative approach
Obvious learning points
Number of patients seen in AEC is not a sufficient to demonstrate impact – although is an easy / obvious process measure
Understanding impact on total population not just our service
Qualitative data helps to understand the “why” patient is in right place / “wrong place” as well as assessing effective processes
Steps 3-5 reflections
Improving quality in the English NHS
“Reflect, measure and learn rapidly about what is and is not working to help implementation become more successful”
“In complex adaptive systems like the NHS and its component health care organisations, learning is dynamic, plans need to be flexible and strategies emergent if they are to be effective”
Improving quality in the English NHS – A strategy for action
Kings Fund, Feb 2016
Don Berwick, Jennifer Dixon, Chris Ham
It is a waste of time collecting and analysing your data if you don't
take action on the results
Step 6 - Review Measures
How, when and where you review your measures is a key
challenge for you to take away from today
You may not get it right first time!
You may need several iterative attempts through steps 4, 5 and 6
And finally - Step 7
Resources
Guide to measurement for improvement
http://www.nhselect.nhs.uk/What-we-do/Service-Improvement--Measurement/Measurement-for-Improvement
Link to video and guide (free)
H l o io ’s v l io w o co si
http://www.health.org.uk/publication/evaluation-what-consider
Any questions?