PATIENT SAFETY DATA OR PATIENT SAFETY INTELLIGENCE?Selection and presentation to draw the best possible insight, and avoid common pitfalls and perils

Dr Frances Healey, RGN, RMN, PhDSenior Head of Patient Safety Intelligence, Insight & Evaluation, Patient Safety Domain, NHS England

21 May 2014

1st rule of #statisticsclubQualitative data is at least equally important, and probably much more important, than quantitative data…..

2

….. but it’s quantitative data that has the pitfalls & perils, so that is my focus today

3

4

5

Dammit Jim, I’m a doctor, not a rocket

scientist

Dammit Frances, I’m a nurse, not statistician

2nd rule of #statisticsclubYou are probably much less logical than you think you are

6

http://blogs.bmj.com/bmj/2014/05/09/tara-lamont-on-failing-well-archie-cochranes-legacy/

@TaraJLamont

Archie Cochrane

7www.england.nhs.uk

8www.england.nhs.uk

“The results at that stage showed a slight numerical advantage for those who had been treated at home. It was of course completely insignificant statistically.

“I rather wickedly compiled two reports, one reversing the numbers of deaths on the two sides of the trial. As we were going into committee, in the anteroom, I showed some cardiologists the results……..

9

“……they were vociferous in their abuse: `Archie’, they said, `we always thought you were unethical. You must stop the trial at once…’

“I let them have their say for some time and then apologised and gave them the true results, challenging them to say, as vehemently, that coronary care units should be stopped immediately.

“There was dead silence and I felt rather sick because they were, after all, my medical colleagues.”

Professor Archibald Cochrane & Max Blythe One Man's Medicine (1989) p.211

Cognitive dissonance • We have a strong need for our personal beliefs

and our personal actions to chime• The discomfort we feel when they don’t is

‘cognitive dissonance’

10

• If we believe we are part of effective, motivated, caring teams, it is very hard to also simultaneously believe:i. We haven’t achieved real

improvements in safety ii. We might be less safe

than peers

http://britishgeriatricssociety.wordpress.com/2013/05/16/all-down-to-numbers/

11

http://www.health.org.uk/multimedia/slideshow/hard-data-soft-intelligence/

Data used for reassurance

12

13

http://britishgeriatricssociety.wordpress.com/2013/12/19/fallsafe-are-culture-clashes-good-for-us/

Numbers don’t know if they are in research study or a QI project

0

5000

10000

15000

20000

25000

OUTCOME indicators PROCESS indicators

planning process indicatorsdelivery process indicator

STRUCTURE indicators CULTURE indicators

14

Types of safety measurement

Our high tech interactive facility

15

1.2.3.4.5.

1. OUTCOME indicator2. PROCESS - planning process indicator3. PROCESS - delivery process indicator 4. STRUCTURE indicator5. CULTURE indicator

16

What type of indicator?

97% of patients who need a pressure reliving mattress received it within four hours

1. OUTCOME indicator2. PROCESS - planning process indicator3. PROCESS - delivery process indicator 4. STRUCTURE indicator5. CULTURE indicator

17

What type of indicator?

2% of patients developed a pressure ulcer

1. OUTCOME indicator2. PROCESS - planning process indicator3. PROCESS - delivery process indicator 4. STRUCTURE indicator5. CULTURE indicator

18

What type of indicator?

86% of nurses agree that most pressure ulcers can be prevented

1. OUTCOME indicator2. PROCESS - planning process indicator3. PROCESS - delivery process indicator 4. STRUCTURE indicator5. CULTURE indicator

19

What type of indicator?

We have 42 pressure relieving mattresses per 100 beds

3rd rule of #statisticsclubWe don’t do structural measurement nearly often enough

20

30%

9%

26%

35%

on all wards

on most wards

on one or some wards

not on any wards

“This [MH unit for older people] has no physio input. Balance and strength assessments never get done”

“We cannot put walking frames within reach as there is no room left once you have a chair beside the bed”

Royal College of Physicians 2012 Report of the 2011 inpatient falls pilot audit www.rcplondon.ac.uk

One often used model:measurement for improvement measurement for judgement measurement for research

Alternative I’d suggest: measurement to understand prioritiesmeasurement to see how we compare to others measurement to see if we’re getting better (or worse)

21

Purposes of safety measurement

0-45-9 10-14

15-19

20-24

25-29

30-34

35-39

40-44

45-49

50-54

55-59

60-64

65-69

70-74

75-79

80-84

85-89

90-94

95-99

100+

0%2%4%6%8%

10%12%14%16%18%

Breakdown by age of falls in acute clusters

Age group

% of all repor-ted acute falls

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Acute hospitals Community hospitals Mental health units

Location of incident

Per c

ent o

f sam

ple

Apparently unwitnessed by staff

Witnessed by staff

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Acute hospitals Community hospitals Mental health units

Location of incident

Per c

ent o

f sam

ple

Apparently unwitnessed by staff

Witnessed by staff

0%

5%

10%

15%

20%

25%

30%

35%

Acute hospitals Community hospitals Mental health units

Whilst walking

From beds

Circumstances unclear

From chairs

From toilet or commode

Other

0%

5%

10%

15%

20%

25%

30%

35%

Acute hospitals Community hospitals Mental health units

Whilst walking

From beds

Circumstances unclear

From chairs

From toilet or commode

Other

00 (12 AM

- Midnight)

01 (1

AM)

02 (2

AM)

03 (3

AM)

04 (4

AM)

05 (5

AM)

06 (6

AM)

07 (7

AM)

08 (8

AM)

09 (9

AM)

10 (10 AM

)

11 (11 AM

)

12 (12 PM

- Midday

)

13 (1

PM)

14 (2

PM)

15 (3

PM)

16 (4

PM)

17 (5

PM)

18 (6

PM)

19 (7

PM)

20 (8

PM)

21 (9

PM)

22 (10 PM

)

23 (11 PM

)

0%

1%

2%

3%

4%

5%

6%

Falls incidents by hour of occurrence, for acute clusters

Hour

% of all reported acute falls

Understanding priority areas

4th rule of #statisticsclubWhen it comes to comparing outcomes, case mix really matters

23

Safety outcomes & case mix

Deandra S et al. Arch Gerontol Geriatr 56 (2013) 407–415NPSA Slips trips and falls in hospital data update NPSA 2010

0%

5%

10%

15%

20%

25%

0-4 5-9 10-14 15-19 20-24 25-29 30-34 35-39 40-44 45-49 50-54 55-59 60-64 65-69 70-74 75-79 80-84 85-89 90-94 95-99 100+

Age of patient (years)

Per

cen

t

Per cent of totalbed daysPer cent of totalfalls

0%

5%

10%

15%

20%

25%

0-4 5-9 10-14 15-19 20-24 25-29 30-34 35-39 40-44 45-49 50-54 55-59 60-64 65-69 70-74 75-79 80-84 85-89 90-94 95-99 100+

Age of patient (years)

Per

cen

t

Per cent of totalbed daysPer cent of totalfalls

Risk factors for fall in hospital Odds Ratio

History of falls 2.85 (1.14–7.15)

Cognitive impairment 1.52 (1.18–1.94)

85 years +

25

Older people are not evenly distributed

Therefore higher/lower rates of these safety outcomes compared to other trusts with very different age profiles, highly unlikely to be useful indicators of safety

All ages falls rate

0.01.02.03.04.05.06.07.08.09.0

RETIREM

ENT TO

WN B

RETIREM

ENT TO

WN D

RETIREM

ENT TO

WN A

RETIREM

ENT TO

WN C

URBAN TEACHIN

G B

URBAN TEACHIN

G C

URBAN TEACHIN

G D

URBAN TEACHIN

G A

falls

per

1,0

00 b

ed d

ays

26Royal College of Physicians 2011 The FallSae Quality Improvement project: report for the Health Foundation

50-fold differences between wards

Therefore higher/lower rates of these safety outcomes compared to other wards in the same trust, highly unlikely to be useful local indicators of safety

27 Royal College of Physicians 2011 The FallSafe Quality Improvement project: report for the Health Foundation

Comparing same speciality better but will still see volatility

OUTCOME indicators PROCESS indicators

planning process indicatorsdelivery process indicator

STRUCTURE indicators CULTURE indicators

28

Types of safety measurement

29

Recap: types of ward level indicator Hierarchy of activities not done

Therefore measuring a few processes that are easier to measure gives a good indication of what other activities will also have been delivered /not delivered

Patient Safety domain workstreams • Developing & testing new peer groups for trusts• Working with CQC, TDA & Monitor to ensure

shared understanding of what the data can and can’t tell you about safety

• Developing new public presentations of safety data:

30

Comparable safety indicators

DH – Leading the nation’s health and care

Non-comparable safety indicators

5th rule of #statisticsclubWhen it comes to measuring improvement in outcomes, size matters

33

34

0

2

4

6

8

10

12

14

16

Feb-10

Mar-10

Apr-10

May-10

Jun-10

Jul-1

0

Aug-10

Sep-10

Oct-10

Nov-10

Dec-10

Jan-11

Feb-11

Mar-11

Apr-11

May-11

Jun-11

Jul-1

1

Aug-11

Sep-11

Oct-11

Nov-11

Dec-11

Jan-12

0

2

4

6

8

10

12

14

16

0

10

20

30

40

50

60

70

80

90

100

35

Sample safety outcome indicators: scaled to ward-level*

IF these safety outcomes were distributed evenly across acute wards, an average ward would have around:

• 1 case of c difficile per year

• 1 MRSA bloodstream infection per decade

• 1 new pressure ulcer per quarter

• 1 fall with minor injury per month

• One fall with hip fracture every five years

* Approximations based on c. 5,000 acute/rehabilitation hospital wards in England, PHE trust

attributed/trust-assigned HCAI data, NRLS reported falls, assumption that acute ‘new’ p ulcer

prevalence as measured by ST represents about 4 x acute p ulcer incidence

For falls with injury 1. One ward two years2. Ten wards two years 3. One medium sized hospital two years 4. Five hospitals two years 5. Fifty hospitals two years

37

What scale & time would give you a reasonable chance of being able to distinguish a 25% improvement from natural variation ?

For hospital-associated MRSA?1. One ward two years2. Ten wards two years 3. One medium sized hospital two years 4. Five hospitals two years 5. Fifty hospitals two years

38

What scale and time would give you a reasonable chance of being able to distinguish a 25% improvement from natural variation ?

39

Collaboration is not just good for learning,

it increases sample size

6th rule of #statisticsclubYour data don’t have to be perfect – but you do need to know how imperfect they are

40

Feb-10 Mar-10 Apr-10 May-10 Jun-10 Jul-10 Aug-10 Sep-10 Oct-10 Nov-10 Dec-10 Jan-11 Feb-11 Mar-11 Apr-11 May-11 Jun-11 Jul-11 Aug-11 Sep-11 Oct-11 Nov-11 Dec-11 Jan-120

2

4

6

8

10

12

14

16

60% certain last fall was reported

77% certain last fall was reported

7th rule of #statisticsclubIf it looks too good to be true, it probably is!

42

Jan

Feb

Mar

Apr

May Jun

Aug

Oct

Nov

Jan

Oct

Dec

Aug

Sep

Oct

Nov

Dec

Jan

Feb

Mar

Apr

May Jun Jul

Aug

Sep

Oct

Nov

Dec

2008 2009 2010 2011

0

10

20

30

40

50

60

70

80 Total

0%

1%

2%

3%

4%

5%

6%

7%

8%

9%

10%

Falls

8th rule of #statisticsclubDon’t try to measure too much

43

44

45

So I’m not going to talk about 9. Beware using SPC if seasonal variation

10. Rates are usually better than numbers, but sometimes numbers are better than rates

11. There is no ‘expected rate’ of Never Events

12. You can quantitatively analyse words as well as numbers

46

47www.england.nhs.uk

“Faced with the choice between changing one’s mind and proving there is no need to do so, almost everybody gets busy on the proof ”

JK Galbraith Economics, Peace and Laughter (1971) p. 50

Thank you for listening

frances.healey@nhs.net

@FrancesHealey

Please tweetfurther #statisticsclub rules of your own