Introduction to Measuring Quality

for Internal Service Improvement

Sebastian Yuen and Richard Wilson

Understand Variation

Measurement Questions

• How old are you?• How many qualifications do you have?• How many hours a week do you work?• How safe is your department?• How does your practice compare with peers?• What do your patients think of your service?• How do you know your project is improving care?

Types of measuresCharacteristic: Judgement Research Improvement

Aim:

Sample size:

Hypothesis:

Variation (Bias):

Determining if change is an

improvement:

Testing strategy:

Achievement of target

Obtain 100% of available, relevant data

No hypothesis

Adjust measures to reduce variation

No change focus

No tests

New knowledge

Fixed hypothesis

Design to eliminate unwanted variation

Statistical tests: (t-test, F-test,

chi square, p values)

Improvement of service

“Just enough” data, small sequential samples

Hypothesis flexible, changes with learning

Accept consistent variation

“Run Charts” or “Statistical

Process Control” (SPC) charts

“Just in case” data

One large test Test blinded

Sequential tests Tests observable

4 Hour Wait Effectiveness of Drug Implementing PEWSExample:

Reasons For Referral To Social Care

RAG Rating

What Does This Show?

“When you have two data points, it is very likely that

one will be different from the other.”

W. Edwards Deming

Is This An Improvement?

Seek Usefulness Not Perfection

Child Protection Activity

Child Protection Activity

• Is it relevant?• Is this good / bad / normal? • Is it improving over time? • Do we understand the variation?• What are we aiming for?• Does it inspire staff to improve?

Speed Of Completion Of Assessments

Speed Of Completion Of Assessments

• Is this good / bad / normal? • Is it improving over time? • Do we understand the variation?• Is this something we are trying to improve?

Plot Data Over Time

MRSA Bacteraemia (All Sources) - HPA

Annotate: Tell A Story

MRSA Bacteraemia (All Sources) - HPA

There is no data showing a problem here

There is no problem

There is no evidence for handwashing / bare below the elbows

It cannot be measured

My patients are sicker / different

The patients come from care homes

I can’t do that – I’m a surgeon!Bare below the elbows

Infection control nurses

Swab all patients on arrival

Antibiotic use controlledThe blood cultures are contaminated

Specially trained nurses do all blood cultures

FT application depends on it

Zero tolerance by DH

The data is wrong

Use Small Samples, Regularly

Measurement

“All improvement will require change,

but not all change will result in an improvement”

The Improvement Guide: A practical approach to enhancing organizational performance (2nd Edition 2009)Gerard J. Langley, Kevin M. Nolan, Thomas W. Nolan, Clifford L. Norman, Lloyd P. Provost

Developing Measures

• Name of measure• Why is it important?• Who owns it?• Operational definitions:

– Data collected, how often, by whom, calculation• Numerical goal (and target date to achieve it)• How to display data, how often, by whom?• Who will review results and take action?

LddddddddddddddddddlLlll

Driver Diagram

Use A Basket Of Measures

(Dashboard)

• Process Measures– Was the WHO Safer Surgery Checklist completed?

• Outcome Measures– Crash calls– Mortality– PROMs

• Balancing Measures– Re-admission within 30 days of discharge

Basket of Quality Measures

Frontline staff to perform, display and improve measures

What Is A Care Bundle?

The Improvement Guide: A practical approach to enhancing organizational performance (2nd Edition 2009)Gerard J. Langley, Kevin M. Nolan, Thomas W. Nolan, Clifford L. Norman, Lloyd P. Provost

What Is A Care Bundle?

• Allows systematic appraisal of clinical processes• Ensures patients receive best care 100% of time• Each component must be evidence-based• The delivery of each component can be improved• Delivery of each component must be achievable• Delivery of each component must be measurable• The bundle is more than sum of its parts

PEWS Process Bundle

Central Line Bundles

1. Wash hands2. Maximal barrier precautions on insertion3. Chlorhexidine skin antisepsis4. Optimal catheter site selection:

avoid femoral vein for central venous access5. Daily review of line necessity

with prompt removal of unnecessary lines

Central Line Bundle: Process (Luton)Central Line Compliance

Central Line Infections

Central Line Bundle: Outcome (Luton)Central Line Compliance

Central Line Infections

For Rare Outcomes:

Use Days Between Incidents

What Is A Run Chart?• A line graph that displays observed data points

plotted in chronological order

• X-axis: Time• Y-axis: Observation• Can add a mean, median or target• Annotate when changes made

• Use as a driver for change / celebration of success

0

2

4

6

8

10

12

Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar

Cum

ulat

ive

Arr

ests

Month

Salford Cardiac Arrests

Last Year (2007)

Actual (2008)

50% Reduction Target

Example Run Chart

• Only measure what you want to improve• Use a balanced set of measures• Use qualitative and quantitative data• Measure over time• Measure small samples regularly• Integrate measurement into the daily routine• Keep it simple: pen and paper can be better than Excel• Need to share the data with the team - transparency• Data enables conversation about problems & solutions

Measurement Summary

Paediatric Dashboard - WalsallDomain Measure

Safety Days since last gentamicin error / harm from medication

Safety Hospital acquired infection rate

Safety Number of incidents (level 1-3) divided by incidents (level 4-5) pm

Safety Days between never events

Pt Experience Complaints per month

Pt Experience Compliments per month

Pt Experience Patient experience survey (Picker / I Want Great Care)

Pt Experience Walkaway from PAU / Ward

Pt Experience Net Promoter Score

• Customer loyalty metric• "How likely is it that you would recommend our company

to a friend or colleague? (on a scale of 0-10)" o Promoters (9–10 rating)o Passives (7–8 rating)o Detractors (0–6 rating)

• NPS = % Promoters minus % Detractors• Interpretation:

o -100 (everybody is a detractor)o +100 (everybody is a promoter)o An NPS that is positive is goodo An NPS of +50 is excellent

Net Promoter Score

Paediatric Dashboard - Asthma

Domain Measure

Effectiveness Number of non-elective admissions with asthma, age 0-18

Effectiveness Percentage of children an Asthma Control Test (ACT) of >19 (of 25), measured at annual review by GP

Effectiveness Re-admission rate with asthma within 7 days

Effectiveness Annual UK BTS Asthma Audit

Pt Experience Patient experience survey

Pt Experience Number of days off school for children with asthma

Value Financial cost to whole healthcare economy of children with asthma

Integration How seamless was the service (from patient’s point of view)?

Paediatric Dashboard - Walsall

Domain Measure

Efficient Clinic DNA rate

Efficient New : Follow Up Rate

Efficient Financial contribution to Trust

Efficient Clinic letters sent within 48 hours

Timely Days to third next available clinic slot

Timely Days since patient left PAU without being seen

Process Measures Notes, PEWS, Drug Charts,

Staff Experience

Domain Measure

Staff Experience Staff sickness rate

Staff Experience Staff turnover

Staff Experience Staff satisfaction survey – annual

Staff Experience Number of staff being disciplined / performance managed

Staff Experience Vacant posts

Trainee Experience GMC / JEST Survey

Trainee Experience Would they recommend the department to others (NPS)?

Neonatal Outcomes

Domain Measure

Effectiveness Number of days unit closed to admissions

Effectiveness Survival to discharge

Effectiveness Grade 3 intraventricular haemorrhage (IVH)

Effectiveness Admission to Ward 21 within 7 days of discharge

Safety Healthcare-Acquired Infection rate

Pt Experience Parent experience score (internal questionnaire)

Pt Experience Complaints (days between)