Audit: what works and what doesn’t

Dr Jo HayesConsultant in Palliative Medicine

Marie Curie Hospice, PenarthCourse Tutor, Diploma in

Palliative Medicine / Care, Cardiff University

Overview Overview

►What audit isWhat audit is►What audit is notWhat audit is not►Why perform auditWhy perform audit►The audit cycleThe audit cycle►Standard setting Standard setting ►Types of auditTypes of audit►Good and bad Good and bad

Definition of auditDefinition of audit

► Clinical auditClinical audit is a quality improvement is a quality improvement process that seeks to improve patient care process that seeks to improve patient care and outcomes through systematic review of and outcomes through systematic review of care against explicit criteria and care against explicit criteria and implementation of changeimplementation of change

NICE: Principles for best practice in clinical audit. 2002NICE: Principles for best practice in clinical audit. 2002

► Introduced in the UK in the 1980sIntroduced in the UK in the 1980s► Part of the clinical governance framework in Part of the clinical governance framework in

the UKthe UK

History of auditHistory of audit► Crimean War 1850sCrimean War 1850s

soldiers 7 x risk of death soldiers 7 x risk of death from cholera & typhus from cholera & typhus than from battle injuriesthan from battle injuries

► Mortality decreased Mortality decreased dramatically due to:-dramatically due to:- Hygiene measures Hygiene measures Fresh waterFresh water Fresh fruit and vegFresh fruit and veg Hospital equipmentHospital equipment

What audit isWhat audit is

► A measure of A measure of compliance with compliance with defined best defined best practicepractice

► Do we know what Do we know what we should be doing?we should be doing?

► Are we doing it?Are we doing it?

What audit is notWhat audit is not

► What should What should we be doing? - we be doing? - researchresearch

► What are we What are we doing? – survey doing? – survey / review etc/ review etc

Audit vs ResearchAudit vs Research

► Tests care given Tests care given against known best against known best practicepractice

► Measures against Measures against standardsstandards

► Normal clinical Normal clinical practicepractice

► No randomisation or No randomisation or placeboplacebo

► Most relevant locallyMost relevant locally

► Creates new Creates new knowledgeknowledge

► Tests hypothesesTests hypotheses

► ExperimentalExperimental► May involve May involve

randomisation or randomisation or placeboplacebo

► GeneralisableGeneralisable

Why perform audit?Why perform audit?

To provide a high quality service and improve To provide a high quality service and improve patient carepatient care

► Compulsory for all UK doctorsCompulsory for all UK doctors► Royal colleges require ongoing departmental Royal colleges require ongoing departmental

audit to be in placed before approving audit to be in placed before approving training poststraining posts

► Meetings are educational Meetings are educational ► Encourage teamworkEncourage teamwork► Protected timeProtected time

The Audit CycleThe Audit Cycle

Set standardsObserve practice

Identify topic / issues and agree importance

Compare practice with standard

Review current practice and implement change

Compare new practice with standard

Reflect / consider new standards

What can you audit ?What can you audit ?

►Structure: premises, staffing, Structure: premises, staffing, equipment equipment

►Process: waiting times, investigationsProcess: waiting times, investigations►Outcomes:Outcomes: survival, improved survival, improved

symptoms, achieving desired place of symptoms, achieving desired place of death, improved quality of life etcdeath, improved quality of life etc

‘‘Not all that can be counted matters and Not all that can be counted matters and not all that matters can be counted.’not all that matters can be counted.’

Audit standards Audit standards

SMART StandardsSMART Standards

► SSpecific (clear, unambiguous and jargon-free)pecific (clear, unambiguous and jargon-free)► MMeasurable – and with a target attached e.g. easurable – and with a target attached e.g.

100% 100% ► AAgreed (by all concerned with delivering that greed (by all concerned with delivering that

aspect of care)aspect of care)► RRelevantelevant► TTheoretically sound (based on evidence heoretically sound (based on evidence

about best practice, reviewed and updated as about best practice, reviewed and updated as new evidence becomes available)new evidence becomes available)

UBHT Central Audit OfficeUBHT Central Audit Office

Writing standards Writing standards

Audit Criteria Audit Criteria

(Standard (Standard statements)statements)

TargeTargett

ExceptionsExceptions Source of Source of EvidenceEvidence

InstrumentInstruments for data s for data collectioncollection

‘‘Patients Patients referredreferred

to the to the communitycommunity

SPCT for urgentSPCT for urgent

review ofreview of

uncontrolleduncontrolled

symptoms symptoms shouldshould

be assessedbe assessed

within 2 days ofwithin 2 days of

referral.’referral.’

100%100% Patient choicePatient choice

Patient Patient outside of outside of community community team remit team remit e.g. hospice / e.g. hospice / acute hospitalacute hospital

All WalesAll Wales

NationalNational

Standards for Standards for

Specialist Specialist

Palliative CarePalliative Care

Cancer ServicesCancer Services

20052005

RetrospectivRetrospective review of e review of patient notes patient notes against against referral referral forms. forms.

UBHT Central Audit OfficeUBHT Central Audit Office

TargetsTargets

► Often set at 100% (or 0%)Often set at 100% (or 0%)

► Exceptions to criteria can allow 100% standard Exceptions to criteria can allow 100% standard ► May set a target below 100% to be realistic for May set a target below 100% to be realistic for

a first audit in an aspect of care where you a first audit in an aspect of care where you know that your team performs badly.know that your team performs badly.

► Figures suggested by research literature could Figures suggested by research literature could be set as a target e.g. percentage deaths at be set as a target e.g. percentage deaths at home home

Identify problem & agree need for Identify problem & agree need for auditaudit

►Team or own ideas of area/issue of Team or own ideas of area/issue of concern or with potential for concern or with potential for improvementimprovement

Staff on surgical ward 10 at Memorial Staff on surgical ward 10 at Memorial Hospital say all their palliative care Hospital say all their palliative care patients are nauseated / GPs report patients are nauseated / GPs report patients discharged with uncontrolled patients discharged with uncontrolled nauseanausea

Determine a standardDetermine a standard

► SMARTSMART► Qualified: existing (search literature) or guided Qualified: existing (search literature) or guided

by experts (ask) or agreed by team (discuss) by experts (ask) or agreed by team (discuss) ► Quantified: standards should have a percentage Quantified: standards should have a percentage

attached attached

90% patients on ward 10 should be asked about 90% patients on ward 10 should be asked about nausea as part of their initial assessment nausea as part of their initial assessment

75% patients should have acceptable relief within 75% patients should have acceptable relief within 48hrs of treatment for nausea48hrs of treatment for nausea

Measure current practiceMeasure current practice

►Retrospective/prospective (time factor)Retrospective/prospective (time factor)►Tools, scales, questionnairesTools, scales, questionnaires

Trawl patients’ notes from ward 10 for Trawl patients’ notes from ward 10 for medical/nursing entry about nausea medical/nursing entry about nausea (bias?) – define time-scale(bias?) – define time-scale

Trawl drug charts for antiemetic (bias?)Trawl drug charts for antiemetic (bias?)

Compare practice with Compare practice with standardstandard

► Simple arithmeticSimple arithmetic► If standard already met, was standard too If standard already met, was standard too

low?low?► Consider extent of likely improvementConsider extent of likely improvement

40% of patients on ward 10 had a record of 40% of patients on ward 10 had a record of enquiry about nausea in their notesenquiry about nausea in their notes

24% of nauseated patients achieved relief 24% of nauseated patients achieved relief within 48hrswithin 48hrs

Implement change & run for a Implement change & run for a timetime

►Team working needed ++Team working needed ++►Education / forms / posters / guidelines Education / forms / posters / guidelines

Barriers/cascading/resistance/ownershipBarriers/cascading/resistance/ownership►Agree time to allow change to take effectAgree time to allow change to take effect

Ward 10: teaching sessions - need to enquire about Ward 10: teaching sessions - need to enquire about nausea & management of nausea/vomiting to nausea & management of nausea/vomiting to nurses and junior drs; simple guidelines nurses and junior drs; simple guidelines flowchart devised and displayedflowchart devised and displayed

Compare modified practice Compare modified practice with standardwith standard

►Repeat survey – same time frame Repeat survey – same time frame as first viewas first view

►Arithmetic comparisonsArithmetic comparisons

63% of patients on ward 10 had a record 63% of patients on ward 10 had a record of enquiry about nausea in their notesof enquiry about nausea in their notes

18% of nauseated patients achieved 18% of nauseated patients achieved relief within 48hrsrelief within 48hrs

Reflect and consider future Reflect and consider future plansplans

►This part is very important in the This part is very important in the project…and in lifeproject…and in life

►Barriers / cascading / resistance / Barriers / cascading / resistance / ownershipownership

Ward 10 - Standards not metWard 10 - Standards not metBetter rate of enquiry about nausea Better rate of enquiry about nausea Guidelines not followed/prescribing failure as Guidelines not followed/prescribing failure as

junior doctors rotated between teaching and junior doctors rotated between teaching and re-auditre-audit

Diploma audit proposal formDiploma audit proposal form

What makes a good audit?What makes a good audit?

►Relevant topic / team ownershipRelevant topic / team ownership►Multi-professional involvementMulti-professional involvement►SMART standardsSMART standards►Good methodologyGood methodology►Change happensChange happens►Change benefits the patients / carers / Change benefits the patients / carers /

staff staff ►Complete the cycleComplete the cycle►Acknowledge the limitationsAcknowledge the limitations

What makes a bad audit?What makes a bad audit?

►Audit for audit’s sakeAudit for audit’s sake►Seen as a chore Seen as a chore ►No standards / survey / researchNo standards / survey / research►Poor data collectionPoor data collection►Unrealistic time framesUnrealistic time frames

► Inadequate time to Inadequate time to do a good jobdo a good job