alastair macdonald- farewell to routine outcomes ...ukrcom.org/proceedings_data/farewell to routine...

Farewell to Routine ClinicalFarewell to Routine Clinical Outcomes Measurement

Alastair MacdonaldSLAMSLAM

Implementation of RCOM 1997-2007 as F/T clinical academicCopyright holder & trainer tabulated HoNOS65+

2007-2017 as 0 2 WTE chair of outcomes group SQL and SPSS2007-2017 as 0.2 WTE chair of outcomes group, SQL and SPSS analyst

Such sweet sorrowSuch sweet sorrow...

Hopes and fears for RCOM

Hope

The Third DimensionThe Third Dimension

A 3-dimension modelA 3-dimension modelof outcomes (Broadbent, 2001)

Intervention NOT YET!

HoNOS

Change

HoNOS

Condition/ C t tAge, Sex, diagnosis, severity Contextg , , g , y

$64,000 question: What interventions are associated with what clinical

outcomes?outcomes?

2001-4 in older adult services we2001 4 in older adult services we experimented with capturing care

plansplans

Differences between the wards in change in depression scoresin patients with dementia noted in 2001

0

in patients with dementia noted in 2001

re-20

ssio

n sc

o

-40

in d

epre

s

-60

% C

hang

e

-80Team associated with

200120001999

Mea

n %

-100

Ward 1

Ward 2

Better

YEAR

200120001999

Differences between the wards in planned use of sedatives and hypnoticsin patients with dementiain patients with dementia

YEAR: 2001

sed

70

60

otic

pro

po

50

ve o

r hyp

n

40

,1) S

edat

iv 30

20

Ward 2Ward 1

%in

(1,

10

Team associated with episode

Differences between the wards in outcome (depression scale change) by planned use of sedatives and hypnotics

YEAR: 2001

by planned use of sedatives and hypnoticsin patients with dementia

e

40

20si

on s

core

0

in d

epre

s -20

-40

% C

hang

e

-60Sedative or hypnotic

Ward 2Ward 1

Mea

n % -80

-100

No

YesBetter

Team associated with episode

Ward 2Ward 1

Fed back to Ward 1

Change in planned use of sedatives and hypnoticsin patients with dementia

70

in patients with dementia

60

50

40

osed

40

30

me

prop

o

20Team associated with

% in

who 10

0

Ward 1

Ward 2

YEAR

2004200320022001

Change in outcome (depression scale change) in patients with dementia

40


ore

20

0

ssio

n sc

o 0

-20

in d

epre

s

-40

Cha

nge

-60

Team associated with

Mea

n %

-80

-100

Ward 1

Ward 2

YEAR

2004200320022001200019991998

M

Change in outcome (depression scale change) in patients with dementia

40


ore

20

0

Another story ...ss

ion

sco 0

-20

in d

epre

s

-40

Cha

nge

-60

Team associated with

Mea

n %

-80

-100

Ward 1

Ward 2

YEAR

2004200320022001200019991998

M

Hope- actual interventions trapped one day

• Psychological therapies already better capturedp

• Recording of medication for community patients will become more systematicpatients will become more systematic– How can patients go on being treated for month

after month without any systematic record of what medication they are supposed to be taking?

• Electronic prescribing on inpatient units

Fear

RCOM remains a hobby interest of aRCOM remains a hobby interest of a few Trusts and individuals

Institutional attitudes to RCOM e gInstitutional attitudes to RCOM e.g.

• NHS England• Royal College of PsychiatristsRoyal College of Psychiatrists

NHS outcome frameworksNHS outcome frameworks

How progress is to be p gmeasured...

OUTCOMES ????How progress is to be p gmeasured...

The College aims to improve the outcomes of people with mental illness, and the mental health of individuals, their families and communities.communities.

H NOS f il ThHoNOS family... ThenOccasional papers recommending measuresRevalidation: “The College does recommend that psychiatrists

should be considering with colleagues the use of appropriate outcome measures as a way of working with patients tooutcome measures as a way of working with patients to determine the effectiveness of interventions. The College has produced guidance on outcome measures to be considered”p g

Attitudinal problems in the NHSAttitudinal problems in the NHSP ti t• Patients– No demand for outcomes data- would rather not know

• Clinicians• Clinicians– No culture of clinical curiosity (apart from a la Sacks);

fearful of exposure• Managers

– Do not want data muddying up decision-making• Commissioners

– No ability to process information skilfully• Politicians• Politicians

– No interest in anything beyond repeated use of the word “outcomes” without understanding itg

Hope

Closing the information loopClosing the information loop

T i l Outer spaceTypical information

fl

Outer space

flow

2. Data Input- IT SYSTEMService

1. Data Collection

Needed: 5-stage cyclical implementation process: all g y p pnecessary, none sufficient

1 C i

Comparative Effectiveness Studies,, 1. Data Collection

D t Q litStudies,, publications etc

2 Data Input- IT SYSTEM5. Feedback

• CliniciansData Quality

2. Data Input- IT SYSTEM• Clinicians

• Managers

y3. Data Extraction

4. Data Analysis

Issues• Capacity

For training in data entry (Natural language processing– For training in data entry (Natural language processing takes over from rating scales?)

– For extraction (Natural language processing shows context ( g g p gand interventions?)

– For analysis and interpretation (Grant application open)– For training in rating methods

• Active feedback unsustainable?• Dashboard feedback?• Patient-owned records/access to clinical records• Integrating CROMs and PROMs at staff-patient contact

– “Where’s my HoNOS?” in Australia

Fear

Intrusion of academicsIntrusion of academics

PontificationPontification

Bandwagon ridingBandwagon-riding

Pubmed “routine” + “outcomes”+”mental health”n=846

Pubmed “routine” + “outcomes”+”mental health”n=846

Still, not huge...

Publication without getting your bootsPublication without getting your boots dirty

• Jump through some hoops (admittedly tedious) and get access to the mental health minimum gdataset.

• Grab any data on outcomes you findGrab any data on outcomes you find– Don’t ask too many questions about how they got

there• Throw some analysis at the data

– E g Factor analysis or analysis lifted from anotherE.g. Factor analysis, or analysis lifted from another field without thought for inconsistency

• Publish!!Publish!!

Parabiaghi et al’s 2005 Reliable and Clinically Significant Change

C i f b & ’ h d• Corruption of Jacobson & Truax’s method originally developed for measures like CORE

hi h h ti d t d f hi h t twhich have normative data and for which test-retest reliability is known, neither of which is true of HoNOSof HoNOS.

• Moran, V. & Jacobs, R. (2015) Comparing the f f E li h l h l h idperformance of English mental health providers

in achieving patient outcomes, Social Science and M di i 140 127 35 d iMedicine, 140:127-35. doi: 10.1016/j.socscimed.2015.07.009

R li bl h• Reliable changeIndex

– Cronbach’s alpha measures the internal consistency of all HoNOS scales lumped together• But HoNOS was never meant to be a scale and items have low internal consistencyBut HoNOS was never meant to be a scale and items have low internal consistency • Jacobson & Truax correctly used test-retest reliability, not internal consistency. Test-

retest reliability of HoNOS is not going to be easy to estimate accurately!

• Clinically significant change

B t d ’t h d t H NOS ti i th l (“f ti l”)– But we don’t have any data on HoNOS ratings in the general (“functional”) population, so Parabiaghi decided on a formula depending on whether or not a rating had two scales scoring at least 3 rather than one or fewer scales: arbitrary!!y

80% no change!

• Very pessimistic• Arbitrary (Clinical change)Arbitrary (Clinical change)• Miscalculated (Reliable Change)• Totals never meant to be added up

Hopes

Everyone will forever follow theEveryone will forever follow the following advice...

1. Understand where RCOM fits into evidence-based practice: You cannot prove that the outcomes are caused by theYou cannot prove that the outcomes are caused by the

interventions.

Hear Hear!

But you can plausibly suggest this when you have theBut you can plausibly suggest this when you have the right context data.

The ‘hourglass’ (Salkovskis 1995)

Naturalistic observation of practice (wide = ‘noisy’)

Randomised Controlled Trial

Naturalistic observation of practiceNaturalistic observation of practice

The place of routine outcomes measurementThe place of routine outcomes measurement

Naturalistic observation of practice (wide = ‘noisy’)

Routine clinical outcomes measurement

Randomised Controlled Trial

Routine clinical outcomes measurement

Naturalistic observation of practiceNaturalistic observation of practice

2 Wh t ? U d t d th t2. Whose outcomes? Understand the outcomes grid

Patient Clinician Family Purchasers

“Stakeholders”`

Patient Clinician Family Purchasers

Reason for interest in outcomes D i d

e.g. HoNOS

Desired outcomes Choice ofChoice of measure

© Clearing House for health outcomes

3. Do not quail before cheating and unreliability; they can be measured and then accommodated in analysisand then accommodated in analysis

2304 episodes with same rater at2304 episodes with same rater at assessment & dischargeg

Median no of episodes per rater=8Median no of episodes per rater 8

44

30.00Assessment Total

25.00

TotalDischarge Total

20 00

25.00

15 00

20.00

lue

10 00

15.00

Va

5 00

10.00

0 00

5.00

45

1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101106111

Case Number

0.00


25.00


20 00

25.00

15 00

20.00

lue

10 00

15.00

Va

5 00

10.00

0 00

5.00

Cheats?

46

1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101106111

Case Number

0.00


25.00


20 00

25.00

15 00

20.00

lue Depressed?

10 00

15.00

Va

5 00

10.00

0 00

5.00

Cheats?

47

1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101106111

Case Number

0.00

Incorporating inter-rater unreliabilityinto analysis- raw data example

18

16

1414

12

10 Total1 adjusted for

% C

I 8

missing values to ma

Total2 adjusted for

1961571000873568 1961571000873568N =

HAYWORTHJOBCBN.Southw ark

95%

6

j


48Team associated with episode

16

2001

14

12Total1 adjusted for

12 missing values to ma

TOT1RND10 TOT1RND

Total2 adjusted for

% C

I

8Total2 adjusted for


110162702618441 110162702618441 110162702618441 110162702618441N =

HAYWORTHJOBCBN.Southw ark

95%

6 TOT2RND

49Team associated with episode

4. Taxpayers! Start to ask how much health gain you are getting with the

£120bn spent on the NHS!£120bn spent on the NHS!

When all they you see so far areWhen all they you see so far are outcomes for....

5. Everyone! Be a bit more thoughtful about what “quality” is in

health care serviceshealth care servicesOutcomes Hero No 1Outcomes Hero No 1

Avedis Donabedian1919-2000

Quality in:Quality in:Structure, process and outcomep

Donabedian, A (1966) Evaluating the , ( ) gquality of medical care. Milbank Memorial

Fund Quarterly 44; 166-206 Q y ;

Quality in:Quality in:Outcome, process, structurep

Is the order in which he described them

3 aspects to quality in a health service3 aspects to quality in a health serviceOutcomeOutcome• Cautioned against using outcome as only measure of

qualityquality– Cannot distinguish efficacy from effectiveness: (outcome

may be poor because right treatment badly applied or wrong treatment)

– Most important outcome may be least easy to measure, so easily measured but irrelevant outcomes are chosen (e geasily measured but irrelevant outcomes are chosen (e.g. mortality instead of disability)

– Must always take into account context; factors other than y ;intervention may be very important in determining outcome

Donabedian A (1966) Evaluating the quality of medical careDonabedian, A (1966) Evaluating the quality of medical care. Milbank Memorial Fund Quarterly 44; p169

3 aspects to quality in a health service3 aspects to quality in a health service

• Outcome• Process

– appropriateness, completeness and redundancy of • history, • physical examination • diagnostic tests

h i l i d– technical competence in procedures– evidence of preventative management, – co-ordination and continuity of care, – acceptability to patient.. Etc.

3 aspects to quality in a health service3 aspects to quality in a health service

O• Outcome• Process• Structure: the settings in which care takes place and the

“instrumentalities of which it is the product”. Includes d i i i h d di h– administrative processes that support and direct the

provision of care. – the adequacy of facilities and equipmentthe adequacy of facilities and equipment, – the qualifications of staff and their organisation, – administrative structure and operations of programs andadministrative structure and operations of programs and

institutions, – fiscal organisation …. etc.

Later development of his ideasLater development of his ideas..

StructureProcessOutcome

Quality

became

Quality ofO t

Quality of Quality of determines determines

OutcomeProcessStructure

Which led toWhich led to..

F di i iFuror redisorganisationus

5 There is nothing to fear in5. There is nothing to fear in measurement!

Outcomes Hero no 2Outcomes Hero no 2

Tom Trauer 1945 2013Tom Trauer 1945-2013

No-one objected to the introduction of a thermometer or sphygmomanometer in medicine..

6 Don’t become an outcomes6. Don t become an outcomes martyr like me... and...y

Outcomes Hero No 3Outcomes Hero No 3

Ernest Amory CodmanErnest Amory Codman1869 -1940

The End Result IdeaThe End Result Idea

Th E d R lt Id 1900 1940The End Result Idea 1900-1940"The common sense notion that every hospital should follow every

patient it treats, long enough to determine whether or not the treatment has been successful, and then to inquire 'if not, why not?' with a view of preventing similar failures in the future.“

Each patient who entered the operating room was provided with a 5-inch by 8-inch card on which the operating surgeon filled out the details of the case before and after surgery. This card was brought g y gup 1 year later, the patient was examined, and the previous years' treatment was then evaluated based on the patient's condition. This system enabled the hospital and the public to evaluate the

l f d id i i di id lresults of treatments and to provide comparisons among individual surgeons and different hospitals.

Kaska: Spine, Volume 23(5).March 1, 1998.629-633

70

Unable to get colleagues to participateUnable to get colleagues to participate

• 1914 Resigned as junior surgeon MGH in protest at seniority system that was inimical to the “end result” idea– And applied for Surgeon-in-chief position showing data on

his outcomes as superior to those of more senior surgeons– Trustees did not reply

• Organised a meeting to discuss hospital efficiency– Got Mayor of Boston to speak

President MGH wondering if clinical professors could make a

li ing itho t h mb g

Board: “If we let her know the truth about our patients do you suppose she

would still be willing to lay?”living without humbug would still be willing to lay?

G ld fGreedy Professors Golden eggs from rich residents

Greedy Professors

Outcome for CodmanOutcome for Codman

• disgrace• Poverty (temporary)• fired from his position as instructor in surgery at

HarvardHarvard • asked to resign as chairman of the local medical

societysociety• “End Result” Idea thereafter practiced mainly in his

own 10 12 bed private hospitalown 10-12 bed private hospital• Tracked and reported on 337 patients

• Hospital half-full and losing money; “outlaw” statusHospital half full and losing money; outlaw status in Boston

• Managed to do operations in other hospitals, including MGH, where his cards were still being used

• Interrupted by US involvement in the First World War• After this was heavily in debt and had to rejoin the

Establishment in Boston and become a full-time private surgeon with no time to develop the Ideaprivate surgeon with no time to develop the Idea

• Finally incorporated the Idea in a Sarcoma Registry and in a classic book on the shoulderand in a classic book on the shoulder

• Never saw implementation of Idea. Hoped that Harvard and MGH would embrace it as an honourable product of their institutions- but they did not do so in his lifetime

• Died 1940 “Although the End Result Idea may not achieve its entire fulfillment for several generations, I hope to be as content when dying as any soldier on the battlefield… Honors, except those I have thrust

lf i l b t h t b ton myself, are conspicuously absent on my chart, but I am able to enjoy the hypothesis that I may receive some from a more receptive generation”some from a more receptive generation

M l dMy last words...

說的人某事無法完成不應該中斷人做它

alastair macdonald- farewell to routine outcomes ...ukrcom.org/proceedings_data/farewell to routine...

Documents