Clinical Governance: An International JournalIndependent feedback on clinical audit performance: a multi-professional pilot studyPaul Bowie Pat Quinn Ailsa Power

Article information:To cite this document:Paul Bowie Pat Quinn Ailsa Power, (2009),"Independent feedback on clinical audit performance: a multi-professional pilot study", Clinical Governance: An International Journal, Vol. 14 Iss 3 pp. 198 - 214Permanent link to this document:http://dx.doi.org/10.1108/14777270910976148

Downloaded on: 27 April 2015, At: 04:28 (PT)References: this document contains references to 28 other documents.To copy this document: permissions@emeraldinsight.comThe fulltext of this document has been downloaded 763 times since 2009*

Users who downloaded this article also downloaded:Howard J. Klein, Aden E. Heuser, (2008),"The learning of socialization content: A framework forresearching orientating practices", Research in Personnel and Human Resources Management, Vol. 27 pp.279-336

Access to this document was granted through an Emerald subscription provided by 465347 []

For AuthorsIf you would like to write for this, or any other Emerald publication, then please use our Emerald forAuthors service information about how to choose which publication to write for and submission guidelinesare available for all. Please visit www.emeraldinsight.com/authors for more information.

About Emerald www.emeraldinsight.comEmerald is a global publisher linking research and practice to the benefit of society. The companymanages a portfolio of more than 290 journals and over 2,350 books and book series volumes, as well asproviding an extensive range of online products and additional customer resources and services.

Emerald is both COUNTER 4 and TRANSFER compliant. The organization is a partner of the Committeeon Publication Ethics (COPE) and also works with Portico and the LOCKSS initiative for digital archivepreservation.

*Related content and download information correct at time of download.

Dow

nloa

ded

by N

HS

ED

UC

AT

ION

FO

R S

CO

TL

AN

D A

t 04:

28 2

7 A

pril

2015

(PT

)

Independent feedback on clinicalaudit performance:

a multi-professional pilot studyPaul Bowie, Pat Quinn and Ailsa Power

NHS Education for Scotland, Glasgow, UK

AbstractPurpose – The purpose of this paper is to investigate the acceptability and educational impact ofindependent feedback on the clinical audit performance of different groups of healthcare professionalsby trained colleagues.

Design/methodology/approach – This is a pilot study involving review of the criterion audit andsignificant event analysis (SEA) attempts of west of Scotland dentists, pharmacists, physiotherapists,practice managers and nurse practitioners by trained colleagues using validated instruments. Audit,SEA and feedback reports were content-analysed. Data on pre- and post-study attitudes, experiencesand knowledge levels were collected by questionnaire. T-tests for differences in mean group scoreswere calculated, along with 95 per cent confidence intervals for mean differences. A difference in meanscores of 1.0 or greater would be indicative of educational gain.

Findings – A total of 34 participants submitted 54 audit and SEA reports, with 20 submitting both(58.9 per cent). In total, 14/20 audits (75.0 per cent) and 26/34 SEAs (76.5 per cent) contained evidence ofappropriate learning needs and action(s) implemented for healthcare improvement. Feedback focusedon knowledge and skills in applying audit methods; demonstrating insight into deficiencies;highlighting appropriate learning needs; and implementing change. Audit knowledge and skill scoresincreased by a mean difference of $1.0 for most stages of audit and SEA method (p , 0:001). Strongagreement on the value of independent feedback on clinical audit was reported.

Research limitations/implications – The study highlights some of the difficulties in applyingaudit methods across professions and highlights the added value of feedback by trained colleagues,but is limited in size.

Practical implications – Integrating clinical audit and peer feedback with continuing professionaldevelopment obligations may facilitate greater engagement and more effective quality improvement,but will require a policy change and additional resource.

Originality/value – This small study provides further evidence of the acceptability and educationalimpact of independent feedback on clinical audit performance for healthcare professionals.

Keywords Clinical audit, Peer review, Feedback, Quality improvement

Paper type Research paper

IntroductionIn the past decade a greater emphasis has been placed on the need for all healthprofessionals in the UK to participate in continuing professional development (CPD)(Scottish Office Department of Health, 1998). Reflecting on clinical practice and

The current issue and full text archive of this journal is available at

www.emeraldinsight.com/1477-7274.htm

The authors extend their thanks to all health care professionals who participated in this studyand the members of the peer group networks who provided feedback. They also thank Dr JohnMcKay, NHS Education for Scotland, for commenting on earlier drafts of this paper and Dr LilianMurray, University of Glasgow, for statistical advice. This research was funded by NHSEducation for Scotland.

CGIJ14,3

198

Received 24 October 2008Revised 18 February 2009Accepted 13 March 2009

Clinical Governance: An InternationalJournalVol. 14 No. 3, 2009pp. 198-214q Emerald Group Publishing Limited1477-7274DOI 10.1108/14777270910976148

Dow

nloa

ded

by N

HS

ED

UC

AT

ION

FO

R S

CO

TL

AN

D A

t 04:

28 2

7 A

pril

2015

(PT

)

identifying learning needs are important components of CPD (Royal College ofNursing, 2003; Scottish Executive et al., 2003; British Dental Association, 2007).Investigating significant events, undertaking clinical audit and seeking peer feedbackare all proposed as objective methods of highlighting potential learning needs(Chartered Society of Physiotherapy, 2003; McLaren et al., 2007). Participation in theseactivities may also make important organisational contributions to improvinghealthcare quality as part of the UK clinical governance (Scottish Office Department ofHealth, 1998) and patient safety agendas (National Patient Safety Agency, 2005).

However, the clinical audit evidence base suggests that there are a number ofobstacles that impede its success as an effective mechanism for change – despite itswidespread endorsement by policymakers (Department of Health, 2006) andprofessional hierarchies (Scottish Executive et al., 2003; Royal College of Nursing,2003; Chartered Society of Physiotherapy, 2003). For many health professionals audit isstill a marginal activity with low priority. Lack of engagement is a serious concern(Davies et al., 2006). Ever increasing workloads, poor leadership, inadequateunderstanding of audit methods, distrust and hostility, and limited resources andtime are often cited as factors inhibiting participation (Johnston et al., 2000; NationalInstitute for Clinical Excellence, 2002). It is assumed by policymakers that cliniciansintuitively understand audit methods and posses the necessary knowledge, skills andprofessional attributes to apply these to a satisfactory standard. The reality is thatmuch audit is poorly designed and frequently fails largely because all of thefundamental stages of the audit cycle are not always adequately completed (Bowieet al., 2007; McKay et al., 2006; Miles et al., 1996). Similar methodological failings havebeen identified for significant event analysis (SEA) – a qualitative method of audit –when undertaken by many health care teams (Bowie et al., 2008a). The consequencesare potentially serious in that crucial opportunities to improve the quality and safety ofhealthcare may be missed and finite time and resources can be wasted.

Given the reaffirmation by national decision-makers of the priority role of clinicalaudit in improving healthcare quality and safety (Healthcare Commission, 2007) andcontributing to professional appraisal and regulation (Department of Health, 2006),tackling these deficiencies and educational gaps should be of prime concern. One obviousapproach to raising audit performance is through targeted education and training atundergraduate and postgraduate level, but this is not often a priority in alreadycondensed course curricula and effective examples are limited (Campbell et al., 1993).

In recognition of the difficulties in this area, NHS Education for Scotland (NES) – aspecial health authority with responsibility for the education and training of the NHSworkforce – introduced a voluntary feedback model for reviewing the clinical auditactivities of general medical practitioners (GPs). This has enabled GPs to submitexamples of two methods of clinical audit - criterion based audit and significant eventanalysis - for independent review by trained peers as part of regional arrangements forCPD. The model was introduced in the west of Scotland deanery in 2000 andestablished nationally in 2007. It has been previously described in detail (Bowie et al.,2005b), while evidence for its acceptability, feasibility and educational impact ingeneral practice and hospital medicine has grown steadily (McKay et al., 2007, 2006,Bowie et al., 2005a; McGregor et al., 2005).

The purpose of the model is to facilitate engagement in audit methods and providedevelopmental feedback on the quality of audit attempts. The underlying philosophy is

Clinical auditperformance

199

Dow

nloa

ded

by N

HS

ED

UC

AT

ION

FO

R S

CO

TL

AN

D A

t 04:

28 2

7 A

pril

2015

(PT

)

based firmly on formative educational principles and the feedback follows best practiceby aiming to be objective, positive, fair, clear and specific. The aim of the feedback is toprovide thepractitioner with informationonhowsuccessfully aclinical audit activity wascarried out and guide individuals on improving their performance in this area(Chur-Hansen and Koopowitz, 2005). Engagement with the medical model has beenmoderate butencouraging witharound35per cent ofGPprincipals in thewestofScotlandhaving submitted at least one audit example for peer feedback (McKay et al., 2006).

This feedback model is currently limited to GPs but should in theory be transferablegiven the symmetry of CPD and clinical governance expectations for all healthcareprofessionals. Given the potential rewards for improving clinical audit practices on amulti-professional basis through a targeted educational intervention, the possibility ofextending this model of learning to other members of the healthcare workforce clearlymerits investigation. The main aim of this pilot study, therefore, was to investigate theacceptability and educational impact of independent feedback on the clinical auditperformance of different groups of healthcare professionals.

MethodsA summary of the study project design and timeline is outlined in Table I.

Participants and settingsEvery second general practice in the Greater Glasgow NHS Board area was randomlyselected from a descending alphabetical list of all 128 practices. Identified practiceswere sent a letter outlining the study purpose and inviting practices to nominate apractice manager or nurse to participate, with 30 practices expressing an initial interest(23.4 per cent).

Physiotherapists were voluntarily recruited from the NHS Greater Glasgow BackPain Team, which had agreed to pilot the use of SEA and independent feedback.Dentists and pharmacists were recruited from SEA peer review network groups newlyestablished in NHS Scotland in 2006.

Clinical audit training and allocated tasksParticipants all attended single half-day training sessions in February 2007 on clinicalaudit methods and were provided with relevant educational support materials forfuture reference. As well as a focus on understanding the technical side of auditmethods, the training emphasised the need for participants to demonstrate leadership,organisational, motivational and negotiation skills when engaging their healthcareteams. All participants agreed to attempt to complete either one or two clinical audittasks – a completed criterion-based audit cycle and/or a significant event analysis. Atime-scale of six months was set for completion of both tasks that were to be submittedin standard report formats to NES for review and feedback.

Educational review and feedbackIn this model both SEA and criterion audit have defined standardised methods (McKayet al., 2006; Bowie et al., 2007). Reviewers in all peer networks underwent appropriatetraining and calibration (a check on inter-rater reliability) to learn how to formativelyassess audit and/or SEA reports using the validated assessment instruments (Loughet al., 1995; McKay et al., 2007) and provide developmental feedback. Materials

CGIJ14,3

200

Dow

nloa

ded

by N

HS

ED

UC

AT

ION

FO

R S

CO

TL

AN

D A

t 04:

28 2

7 A

pril

2015

(PT

)

submitted by participants were screened for confidentiality issues and then forwardedto two trained reviewers in the appropriate peer group for assessment. The feedbackwas returned to a NES coordinator who undertook a third review and combined allfeedback before returning this in the form of a detailed letter to the submittingpractitioner for their consideration. The absence of established peer review groups forphysiotherapy and practice nursing led to their efforts being evaluated by reviewersfrom the GP network.

Data collectionAll participants completed a pre and post study four-part evaluation questionnaire.The first two sections focused on respondents’ experiences of audit and SEA andestimations of their knowledge and skill levels. The third section consisted of a series

Project stage Timescale Description

One October-December 2006 Training of audit/SEA reviewers (dentists andpharmacists)

Two January 2007 Recruitment of study participants and locationsPractice managers and practice nurses – NHSGreater GlasgowCommunity physiotherapists – NHS GreaterGlasgowGeneral dental practitioners – West of ScotlandCommunity pharmacists – NHS Scotland

Three February 2007 Delivery of training in audit and SEA methodsParticipants attended a single half-day introductorytraining session covering both methodsParticipants agreed to undertake an audit and/orSEA task and submit a written report to NHSEducation for Scotland for feedback from trainedcolleagues within the next six monthsParticipants complete a pre-study evaluationquestionnaire of perceived knowledge, skills andexperiences related to audit and SEA

Four April-October 2007 Independent review and feedback completedParticipants completed audit and SEA reportssubmitted for review and feedbackWritten feedback provided to participants two tofour weeks after submission

Five April-November 2007 Post-study evaluation questionnaires completedParticipants completed and returned questionnaireswithin three weeks of receiving feedback letters

Six November 2007-February 2008 Analysis and reporting of study dataAudit and SEA reports content analysed by researchteamsWritten feedback letters content analysed byresearch teamsQuestionnaire data analysed by research teamPreliminary report of findings sent to participantsand professionals heads of department

Table I.Details of project stages

and timetable inchronological date order

Clinical auditperformance

201

Dow

nloa

ded

by N

HS

ED

UC

AT

ION

FO

R S

CO

TL

AN

D A

t 04:

28 2

7 A

pril

2015

(PT

)

of attitudinal statements about the acceptability and educational impact of the receivedfeedback on their clinical audit efforts. The final section collected demographic dataand encouraged free text responses on participants’ experiences of taking part.Respondents indicated their level of agreement with all statements on a five-pointrating scale.

Data analysisAll audit and SEA reports and associated feedback letters were jointly analysed forcontent by both authors. Key themes were identified around the types of topics studiedand the main learning and change that reportedly took place. The feedback issueshighlighted by reviewers were subjected to identical analysis.

Questionnaire data were input into a Microsoft Excel spreadsheet and analysed. Preand post intervention differences in mean rating scale scores for the group werecalculated. T-tests for differences in mean group scores were calculated, along with 95per cent confidence intervals for mean differences. We determined that a difference inmean group scores of 1.0 or greater would be indicative of overall educational gain.

ResultsStudy participantsOf the 46 health care professionals who attended training sessions, 34 submitted auditmaterials for independent feedback (73.9 per cent). Follow-up telephone interviewswith the 12 non-participants identified heavy workloads and subsequent timeconstraints as mitigating factors. A breakdown of the professional characteristics ofstudy participants is outlined in Table II.

Audit and SEA reports submitted for feedbackThe 34 participants submitted 54 reports of completed audit cycles and SEA reports,with 20 submitting both (58.9 per cent). All participants in this latter group were

Factor n %

Healthcare professionCommunity pharmacist 3 8.8Physiotherapist 4 11.8Practice manager 8 23.5Practice and community nurses 12 35.3Dentist 7 20.6

Professional experience (years)Range 2-39Mean 19.8SD 9.46Median 20

Experience in current post (years)Range 1-35Mean 8.0SD 7.79Median 5

Table II.Participants’ professionaldetails and experience:(n ¼ 34)

CGIJ14,3

202

Dow

nloa

ded

by N

HS

ED

UC

AT

ION

FO

R S

CO

TL

AN

D A

t 04:

28 2

7 A

pril

2015

(PT

)

practice managers, pharmacists or practice nurses. The remaining 14 physiotherapists,dentists and a community nurse submitted a single SEA report only.

Previous experience of criterion audit and SEAIn terms of audit experience, 17 of the 20 participants (85 per cent) who submitted acriterion audit project had previously been involved in audit as part of a health careteam. Of these, ten had gained experience in their current post of being the main personleading a project (50 per cent), while eight from the whole group (40 per cent) hadpreviously written-up a completed audit cycle report.

For the 34 participants who submitted SEA reports, 28 (82 per cent) had beeninvolved previously in discussing a significant event informally with colleagues; 24(70.5 per cent) had highlighted a significant event which was discussed with thehealthcare team in a formal meeting situation; and 18 (52.9 per cent) had been the leadperson charged with analysing a significant event in their current post. A total of 19had previous experience of writing-up a completed SEA report (55.8 per cent).

Learning and changeThe review process judged 14/20 audit reports (75.0 per cent) and 26/34 SEA reports(76.5 per cent) to contain sufficient evidence of learning and change that wasappropriate to the topics being investigated. The reviewers agreed that the actionsimplemented under the leadership of participants were relevant, potentiallysustainable and likely to lead to improvements in the quality and safety ofhealthcare. Typical examples of specific learning and change resulting from clinicalaudit efforts are highlighted in Table III.

Feedback issues highlightedA range of feedback issues was identified by clinical audit reviewers (Table IV). Thesewere mainly concerned with technical and human factors related to participants’knowledge, skills and behaviour in applying both improvement methods;demonstrating insight into why aspects of health care delivery may be deficient;highlighting appropriate learning needs; and also leading the healthcare team in theimplementation of agreed change and ensuring sustainability. However, reviewers alsoraised issues and made suggestions for participants to reflect upon which were basedon their own experiences and ideas as a way of potentially further enhancing thequality and safety of patient care being described in the audit and SEA reports.

Pre- and post-study estimation of audit and SEA knowledge and skillsParticipants estimated that their levels of knowledge and skills of both audit and SEAto have increased after attempting both activities and receiving independent feedbackcompared with their estimated levels before they took part. Tables V and VI highlightstatistical post study differences in participants’ estimated knowledge and skill levelsfor most stages of the defined criterion audit and SEA methods.

Attitudes to independent feedbackLevels of agreement with a series of attitudinal statements about the value ofindependent feedback for clinical audit activities and participating in this type ofeducational model are outlined in Table VII. It is evident that the vast majority of

Clinical auditperformance

203

Dow

nloa

ded

by N

HS

ED

UC

AT

ION

FO

R S

CO

TL

AN

D A

t 04:

28 2

7 A

pril

2015

(PT

)

Pro

fess

ion

Au

dit

pro

ject

SE

Ati

tle

Mai

nle

arn

ing

nee

dor

poi

nt

Key

chan

ges

imp

lem

ente

d

Gen

eral

pra

ctic

em

anag

emen

tH

eig

ht

and

wei

gh

tm

onit

orin

gin

pat

ien

tsag

ed$

21y

ears

No

esta

bli

shed

pro

toco

lfo

rh

eig

ht

and

wei

gh

tm

easu

rem

ent

Ag

reed

stan

dar

dm

eth

odfo

rid

enti

fyin

gta

rget

pop

ula

tion

and

reco

rdin

gm

easu

rem

ents

Tra

inin

gp

rov

ided

onC

hro

nic

Dis

ease

info

rmat

ion

syst

emfo

rca

ptu

rin

gd

ata

Gen

eral

pra

ctic

en

urs

ing

Mic

roal

bu

min

uri

ate

stin

gin

dia

bet

icp

opu

lati

onM

ajor

ity

ofta

rget

pop

ula

tion

not

scre

ened

for

Mic

roal

bu

min

uri

a

Rel

evan

tcl

inic

alst

aff

atte

nd

eded

uca

tion

alse

ssio

nR

outi

ne

scre

enin

gad

apte

dto

incl

ud

esu

bje

ct-m

atte

rA

pp

rop

riat

ep

atie

nts

targ

eted

and

inv

ited

for

scre

enin

gC

omm

un

ity

ph

arm

acy

Gen

eric

ally

-pre

scri

bed

CF

C-

free

bec

loet

ason

ein

hal

ers

Fai

lin

gto

reac

hst

and

ard

sin

crea

ses

risk

sto

pat

ien

ts’

hea

lth

;m

akes

un

nec

essa

ryd

eman

ds

onp

atie

nts

’ti

me;

incr

ease

scl

inic

alw

ork

load

Ap

pro

pri

ate

pat

ien

tsta

rget

ed,

con

sult

edan

dp

resc

rip

tion

san

dre

cord

sch

ang

edac

cord

ing

ly

Gen

eral

den

tal

pra

ctic

eL

oss

ofro

otin

max

illa

ryca

nal

Roo

tre

mov

alsh

ould

not

hav

eb

een

atte

mp

ted

by

inex

per

ien

ced

pra

ctit

ion

erw

ith

out

app

rop

riat

era

dio

gra

ph

Ded

icat

edem

erg

ency

app

oin

tmen

tsl

ots

intr

odu

ced

Su

pp

ort

pro

toco

lsu

sin

glo

cal

coll

eag

ues

arra

ng

edT

rain

ing

onap

pro

pri

ate

exp

osu

reof

rad

iog

rap

hs

pro

vid

edD

enta

ln

urs

eat

ten

ded

cou

rse

onas

sist

ing

inor

alsu

rger

yG

ener

alp

ract

ice

man

agem

ent

Fri

dg

eco

nta

inin

gv

acci

nes

–te

mp

erat

ure

not

pro

per

lyse

tW

ider

pri

mar

yca

rete

amn

otaw

are

ofp

ract

ice

pro

toco

lon

mon

itor

ing

offr

idg

ete

mp

erat

ure

Tea

m-b

ased

mee

tin

gle

dto

frid

ge

bei

ng

mov

edto

new

pos

itio

nw

her

ep

ract

ice

pro

toco

lco

uld

be

mai

nta

ined

(con

tinued

)

Table III.Clinical audit projects:selected examples oflearning and change byprofession

CGIJ14,3

204

Dow

nloa

ded

by N

HS

ED

UC

AT

ION

FO

R S

CO

TL

AN

D A

t 04:

28 2

7 A

pril

2015

(PT

)

Pro

fess

ion

Au

dit

pro

ject

SE

Ati

tle

Mai

nle

arn

ing

nee

dor

poi

nt

Key

chan

ges

imp

lem

ente

d

Gen

eral

pra

ctic

en

urs

ing

Acc

iden

tal

ear

dru

mp

erfo

rati

onE

ver

yd

ayp

roce

du

res

carr

yp

oten

tial

risk

sT

ech

nic

alab

ilit

ym

ayn

eed

tob

ech

eck

edK

now

led

ge

and

equ

ipm

ent

nee

ds

up

dat

ed

Att

end

ance

atre

fres

her

cou

rse

tore

vie

wte

chn

iqu

ean

dp

roce

du

res

New

equ

ipm

ent

ord

ered

Ear

syri

ng

ep

roto

col

up

dat

ed

Com

mu

nit

yp

har

mac

yR

epea

tp

resc

rib

ing

rev

iew

Sy

stem

req

uir

edfo

rre

vie

win

gp

resc

rip

tion

sof

nu

rsin

gh

ome

resi

den

ts

For

mal

med

icat

ion

rev

iew

pol

icy

dev

elop

edfo

rg

ener

alp

ract

ice

Des

ign

ated

GP

lead

for

nu

rsin

gh

ome

liai

son

New

resi

den

tsh

ave

med

icat

ion

rev

iew

edb

yp

har

mac

ist

orle

adG

PT

rain

ing

sess

ion

onlo

cal

pre

scri

bin

gg

uid

ance

Ph

ysi

oth

erap

yM

ix-u

pov

erp

atie

nt

amb

ula

nce

tran

spor

tfo

rp

hy

siot

her

apy

Un

awar

eof

pre

-ch

eck

ing

syst

emw

ith

amb

ula

nce

tran

spor

tP

atie

nts

app

reci

ate

hon

esty

wh

ener

rors

mad

ean

dev

iden

ceof

acti

on

Mu

lti-

pro

fess

ion

alm

eeti

ng

led

ton

ewch

ang

esto

boo

kin

gsy

stem

Rot

atio

nal

staf

fin

du

ctio

np

ack

up

dat

ed

Table III.

Clinical auditperformance

205

Dow

nloa

ded

by N

HS

ED

UC

AT

ION

FO

R S

CO

TL

AN

D A

t 04:

28 2

7 A

pril

2015

(PT

)

participants were strongly positive in their response levels in agreeing about theeducational value and potential need for external and independent feedback. There wasno difference in attitudes between participants who received in-depth feedback andthose receiving limited feedback on performance i.e. confirmation of good practice. Aselection of comments from participants on their opinions of the role of this type offeedback on clinical audit performance is outlined in Table VIII.

DiscussionThis pilot study aimed to establish whether an existing medical model of external andindependent feedback on audit and SEA attempts was potentially acceptable and ofeducational value to a multi-professional group of healthcare professionals. Thefindings demonstrate that the overwhelming majority of study participants were ableto successfully complete a clinical audit task(s) within a six-month period and did notobject to the principle of independent review of their performances. The feedback theyreceived was perceived to be professionally acceptable by the vast majority ofparticipants. Similarly, the findings show that most participants reported increasedlevels of knowledge and skills in the application of both audit and SEA techniquesafter receiving their feedback. The implication is that this type of independentfeedback had a positive educational impact on participants. It may also contribute tovalidating the reported learning, change and potential healthcare improvement thatwas facilitated by most participants’ clinical audit attempts. Overall the findingssuggest that this medical model may potentially be transferable to other professions asa generic educational support mechanism for improving clinical audit practice andpotentially adding value to the CPD process. Against this background, the pilot study

Audit projects SEA reports

Unclear or confused criteriaNo clear or defined standard(s) setStandards defined not related to criteriaNo justification for standard levels setTime-scale set to achieve standards not evident orunrealisticNo evidence or insufficient evidence ofpreparation and planning or teamworkData collected were not related to criteria/standardsFailure to compare data with criteria/standardsNumerical errors in data presentedData were poorly presented or difficult to interpretFailure to implement necessary changeFailure to describe change agreed andimplementedInadequate change implementedDescription of change poor or confusedFailure to outline what was learnedFailure to explain why results improvedFailure to explain why results got worseFailure to document how change will be sustained

Not a healthcare significant eventFailure to fully describe what happenedInadequate description of why event happenedWrong interpretation of why the event happenedFailure to describe why the event happenedInadequate description of learning needs, pointsor related issuesLack of insight demonstratedFailure to describe any learning issuesInsufficient details of change describedInadequate action takenInappropriate action takenChange discussed, but failed to take necessaryactionFailure to discuss or take action

Table IV.Summary examples offeedback issues providedby reviewers toparticipants

CGIJ14,3

206

Dow

nloa

ded

by N

HS

ED

UC

AT

ION

FO

R S

CO

TL

AN

D A

t 04:

28 2

7 A

pril

2015

(PT

)

Bef

ore

Aft

er

Au

dit

kn

owle

dg

eor

skil

lM

ean

gro

up

scor

eS

DM

ean

gro

up

scor

eS

DM

ean

gro

up

dif

fere

nce

95p

erce

nt

con

fid

ence

inte

rval

(CI)

t-te

stp-

val

ue

Ch

oosi

ng

ato

pic

for

aud

it2.

761.

034.

150.

491.

451.

06to

1.84

7.85

,0.

001

Defi

nin

gau

dit

crit

eria

tob

em

easu

red

2.45

1.00

4.05

0.51

1.6

1.22

to1.

988.

72,

0.00

1S

etti

ng

stan

dar

ds

for

aud

it2.

351.

094.

000.

651.

651.

16to

2.14

7.10

,0.

001

Pre

par

ing

and

pla

nn

ing

anau

dit

pro

ject

2.45

1.15

4.10

0.55

1.65

1.19

to2.

117.

47,

0.00

1C

olle

ctin

gan

dan

aly

sin

gau

dit

dat

a2.

551.

194.

000.

561.

450.

94to

1.96

5.90

,0.

001

Ab

ilit

yto

lead

the

imp

lem

enta

tion

ofch

ang

eas

are

sult

ofau

dit

2.80

1.20

4.20

0.62

1.40

0.93

to1.

876.

29,

0.00

1P

rep

arin

ga

wri

tten

aud

itre

por

t2.

401.

054.

100.

551.

701.

24to

2.16

7.77

,0.

001

Ov

eral

lun

der

stan

din

gof

the

aud

itcy

cle

2.70

1.30

4.35

0.59

1.65

1.08

to2.

226.

02,

0.00

1

Note:

Rat

ing

Sca

le1

to5,

wh

ere

1¼

No

kn

owle

dg

eto

5¼

Hig

hly

kn

owle

dg

eab

le

Table V.Estimation of audit

knowledge and skillsbefore and after study

(n ¼ 20)

Clinical auditperformance

207

Dow

nloa

ded

by N

HS

ED

UC

AT

ION

FO

R S

CO

TL

AN

D A

t 04:

28 2

7 A

pril

2015

(PT

)

Bef

ore

Aft

er

Sta

tem

ent

Mea

ng

rou

psc

ore

SD

Mea

ng

rou

psc

ore

SD

Mea

ng

rou

pd

iffe

ren

ce95

per

cen

tco

nfi

den

cein

terv

al(C

I)p-

test

p-v

alu

e

Iu

nd

erst

and

wh

atco

nst

itu

tes

a“s

ign

ifica

nt

even

t”2.

911.

134.

640.

601.

731.

36to

2.17

8.90

,0.

001

Iu

nd

erst

and

how

toan

aly

sea

sig

nifi

can

tev

ent

2.76

1.17

4.27

0.67

1.51

1.16

to1.

958.

02,

0.00

1I

amco

nfi

den

tab

out

lead

ing

the

team

inu

nd

erta

kin

ga

sig

nifi

can

tev

ent

anal

ysi

s2.

521.

374.

180.

981.

661.

31to

2.16

8.34

,0.

001

Ik

now

how

top

rep

are

aw

ritt

enS

EA

rep

ort

2.64

1.32

4.42

0.66

1.78

1.35

to2.

248.

21,

0.00

1S

EA

can

imp

rov

eth

eq

ual

ity

and

safe

tyof

hea

lth

care

3.75

1.16

4.61

0.61

0.86

0.52

to1.

244.

93,

0.00

1S

EA

shou

ldb

eco

mp

uls

ory

asp

art

ofh

ealt

hca

rep

ract

ice

3.56

1.39

4.21

0.82

0.65

0.32

to0.

954.

08,

0.00

1

Note:

Rat

ing

scal

e1

to5,

wh

ere

1¼

Str

ong

lyd

isag

ree

to5¼

Str

ong

lyag

ree

Table VI.Levels of agreement withstatements aboutperceived knowledge andskills related tosignificant event analysisbefore and after study(n ¼ 34)

CGIJ14,3

208

Dow

nloa

ded

by N

HS

ED

UC

AT

ION

FO

R S

CO

TL

AN

D A

t 04:

28 2

7 A

pril

2015

(PT

)

Health care professional Comment

Community physiotherapist (B) “SEA is an excellent form of CPD activity. A formalreport of this with external feedback is appropriateevidence for my portfolio . . . Physiotherapy staffshould be exposed to this . . . ”

General practice nurse (A) “I totally agree that audits and SEA should be anintegral part of our professional practice. However,management have to be supportive in allocation oftime and resources . . . ”

Practice manager (B) “I found the whole process very rewarding and I amglad I took part. External peer review is very usefulas it is completely unbiased with no personal feelingtowards the person doing the work”

General practice nurse (C) “One of the main aspects I have learned from thisprogramme – TIME – staff need protected time”

Community physiotherapist (D) “A relatively new experience that was bothchallenging and rewarding. Peer review was focusedand relevant. I will continue to submit reports of anySEAs in future”

Practice manager (E) “I enjoyed completing the audit as I had noexperience. The information was very useful and Ihave used the skills gained to complete audits for theQuality Practice Award that we are workingtowards. Overall I found it very beneficial for mydevelopment”

Table VIII.Selected comments from

participants on theirstudy experiences and

attitudes towards clinicalaudit methods and the

feedback model

Statement Mean score SD

I fully understand the concept of educational reviewand feedback 4.12 0.65It is a good idea to get external, independent feedbackon improving aspects of my work 4.64 0.57I was apprehensive about having my SEA reportcritically reviewed 3.59 1.22I don’t believe external peer review of audit is feasiblein the workplace 1.89 0.93External peer review of audit should be more widelyused in health care 4.19 0.83I found the feedback I received to be educationallyusefulAudit (n ¼ 20) 4.52 0.51SEA 4.44 0.51I would submit examples for review and feedback inthe futureAudit (n ¼ 20) 3.96 0.95SEA 4.26 0.86

Note: Rating scale 1 to 5, where 1 ¼ Strongly disagree to 5 ¼ Strongly agree

Table VII.Post-study levels of

agreement withattitudinal statements

about independent reviewby trained colleagues

(n ¼ 34)

Clinical auditperformance

209

Dow

nloa

ded

by N

HS

ED

UC

AT

ION

FO

R S

CO

TL

AN

D A

t 04:

28 2

7 A

pril

2015

(PT

)

was largely successful in meeting its aim. However it is unclear if this approach isfeasible on a long-term basis without sustainable investment from relevant educationaldecision-makers.

A minority of potential study participants was unable to participate – even withinthe timeframe outlined – citing familiar well-documented barriers which confirms thatengagement in audit activity may never be a feasible proposition for some individualsin the current climate. Additionally, a lack of expertise in audit method is oftenreported as an obstacle by many healthcare professionals, with limited exposure toeducation and training a major contributory factor (Johnston et al., 2000). The findingssuggest that lack of expertise and experience was also a factor for many participantsprior to the study. Although many may also believe that they understand audit theory,we know that they can fail to grasp the range of leadership, technical and otherprofessional attributes necessary to undertake successful, completed audit projects(Johnston et al., 2000; Bowie et al., 2007). From a medical perspective, independent anddevelopmental feedback from trained colleagues has been beneficial in highlightingrelevant learning needs and providing informed guidance (McKay et al., in press a).Evidence from this study indicates that this may also be true with other health careprofessionals.

The general feedback provided to study participants not unsurprisingly replicatedthat previously reported for medical practitioners submitting both criterion audit(Bowie et al., 2007) and SEA reports (Bowie et al., 2005a) for peer review. However,while acknowledging the small study size there was a clear decrease in the volume andextent of learning and change issues raised by reviewers compared to those studiesinvolving medical practitioners. A mitigating factor may be that study participantshad a clear advantage as they were exposed to targeted education and training in bothaudit techniques prior to the study, while most doctors would have submitted withoutsuch formal and focused training.

The majority of study participants were highly experienced in terms of theiraverage length of service both as career professionals and in their current posts. Yet, interms of engagement with and exposure to the different stages of audit methods therewas wide variation in experience. Many lacked leadership experience of audit andSEA, and writing-up related reports. This perhaps reflects the lack of claritysurrounding the specific audit expectations for individual clinicians. For example, it isunclear whether “undertaking clinical audit” means taking on a project leadership roleand assuming responsibility for report authorship. Or is adopting a less active (orpassive) contributory role sufficient evidence of engagement in clinical audit? Theambiguities contained in various policy documents (Scottish Executive, 1998;Department of Health, 2006) and the lack of systems to verify engagement andperformance arguably hinder progress and thus contribute to the continued mixedfortunes of audit as an effective quality improvement method.

CPD is now a mandatory expectation of regulatory bodies for the majority ofprofessional disciplines in the UK. From this perspective, we would suggest that thevery act of submitting an audit or SEA report by a healthcare professional as part ofthe model confers either a level of confidence in their ability or potential insight intotheir educational need in these areas. But more importantly, the direct alignment of theclinical audit process with this type of CPD approach offers a multitude of potentialbenefits to the healthcare system. For the participant, it informs professional

CGIJ14,3

210

Dow

nloa

ded

by N

HS

ED

UC

AT

ION

FO

R S

CO

TL

AN

D A

t 04:

28 2

7 A

pril

2015

(PT

)

performance and provides external recognition of workplace experiential learning. Forthe patient, it enhances the quality and safety of those aspects of healthcare that areunder scrutiny. For the NHS organisation, it has a potential clinical governancefunction in facilitating engagement in audit through the CPD route and in qualityassuring performance. Finally for the policymaker and professional regulator, it addsvalue and contributes to the robustness of CPD systems through the provision ofexternal, independent and objective evidence of performance (Department of Health,2006).

If clinical audit is to be taken seriously and clinicians are to be fully engaged, thenpolicymakers will need to co-ordinate and revitalise the efforts of a range ofstakeholders including health care regulators, medical colleges, professionalassociations, patients’ representatives, health authorities and higher educationinstitutions. The chief medical officer in England has called for a “reinvigoration” ofclinical audit (Department of Health, 2006) to ensure that it “takes its rightful place” ininfluencing NHS policy in the same way research and education already contributes.Medical appraisal, particularly in general practice in NHS Scotland (Scottish Executiveet al., 2003), has focused on providing evidence of competence in clinical audit methodsas a core professional activity, which is a start in demonstrating across the boardengagement. Other clinical professions should view this approach as one way ofimproving audit engagement and assuring knowledge of a defined approach.

It is clear overall that quality improvement education and training at undergraduateand postgraduate levels, and as part of continuing professional development, shouldhave a much greater and integrated focus in all healthcare professions. A morestrategic approach to aligning audit with clinical education is necessary and we need torethink how this can be realistically achieved. A long-term educational strategy will bepivotal to equipping clinicians with the necessary skills and professional attributes toallow them to engage in the process knowledgably and with confidence. However, thiswill take political will and serious time, commitment and resources, but it has thepotential to pay dividends. For example, the summative assessment of GPs’ auditprojects as part of vocational training provided evidence of clinicians’ ability to planand execute a completed cycle of effective audit in a pressurised workplaceenvironment and in time-limited circumstances (Campbell et al., 1993).

Study limitations and strengthsThe study numbers are small so the findings must be viewed in this context.Participants were volunteers and are not representative of their professions. Thefindings are likely to be biased because of the self-selected study population, whopotentially perceived value in participation. The motivations of those who did notparticipate were not investigated. It will be necessary to explore experiences andimpact of the model with larger groups of health care professionals once peer networksare better established. The use of a questionnaire to assess pre- and post-interventionratings of knowledge, skills and attitudes is limited because responses are self-reportedand cannot be verified. Similarly, we assumed that participants took the lead role indirecting and reporting the audit projects in conjunction with their healthcare teams,but how this was conducted in practice may have varied. Improvements in the qualityand safety of care reported by participants were not independently verified.

Clinical auditperformance

211

Dow

nloa

ded

by N

HS

ED

UC

AT

ION

FO

R S

CO

TL

AN

D A

t 04:

28 2

7 A

pril

2015

(PT

)

GP reviewers evaluated the audit materials of some other professional groups,which may have been further enhanced if done by true peers. The reviewers’judgments and the feedback provided were done so using the methods outlined, butother approaches may have led to alternative conclusions. However, a key strength ofthe model is that it has been designed by practitioners for practitioners to developrather than criticise individual audit performance. Indeed, peer reviewers perceive theirrole to be a privileged and highly important professional duty.

Practical implicationsIf this model is to become more widely accepted, its practical use in the context ofsupporting professional appraisal and the clinical governance agendas of NHSorganisations must be further explored. The evidence for the feedback model in termsof its validity, acceptability and educational impact is growing (McKay et al., in press a,b) and there have been calls for it to be aligned with the GP appraisal system inScotland as one means of adding value to this process (Murie et al., in press). However,a justified limitation is whether it is “feasible” on a larger scale and on amulti-professional basis in terms of the cost implications and organisationalcommitment required.

From a health service perspective, we would suggest that setting up regionalmulti-professional peer networks and providing a minimum of training is not an overlyonerous or expensive task, given the potential rewards for CPD and the quality ofhealth care. Clinical governance leadership may be a key factor in developing this typeof initiative and providing sustainable support locally. Existing resources – such asclinical governance support staff – could arguably be targeted more effectively as theymay have a valuable role to play in managing and contributing to such an educationalmodel alongside peer networks. There is evidence that their clinical audit assessmentsas part of the feedback model are comparable with established peer reviewers (Bowieet al., 2008b), which may not be surprising because of their specialist experience inadvising healthcare professionals in these matters. A further strategy would be toutilise the experience of trained professional peers from established networks to“champion” the educational value of participation locally within both the service andtheir own professional associations, which may have a major influence from a CPDperspective.

ConclusionClinical audit attempts by healthcare professionals are often unchecked and ineffectivewhich may lead to non-engagement and missed opportunities to improve the qualityand safety of patient care. This small study adds further evidence of the potential valueof independent feedback by trained reviewers in quality assuring the audit process,validating good practice and providing guidance for improvement, where necessary.Given the overall evidence base for clinical audit, NHS organisations and policymakersshould consider exploring how engagement in this activity can be linked more directlywith CPD obligations as one way of ensuring participation and verifying the quality ofperformance.

CGIJ14,3

212

Dow

nloa

ded

by N

HS

ED

UC

AT

ION

FO

R S

CO

TL

AN

D A

t 04:

28 2

7 A

pril

2015

(PT

)

References

Bowie, P., Pope, L. and Lough, M. (2008a), “A review of the current evidence base for significantevent analysis”, Journal of Evaluation in Clinical Practice, Vol. 14 No. 4, pp. 520-36.

Bowie, P., McCoy, S., McKay, J. and Lough, M. (2005a), “Learning issues raised by the peerreview of significant event analyses in general practice”, Quality in Primary Care, Vol. 13No. 2, pp. 75-84.

Bowie, P., McKay, J., Dalgetty, E. and Lough, J.R.M. (2005b), “A qualitative study of why generalpractitioners may participate in significant event analysis and educational peerassessment”, Quality & Safety in Health Care, Vol. 14 No. 3, pp. 185-9.

Bowie, P., McKay, J., Murray, L. and Lough, M. (2008b), “Judging the quality of clinical audit bygeneral practitioners: a pilot study comparing the educational assessments of medicalpeers and NHS audit specialists”, Journal of Evaluation in Clinical Practice, Vol. 14 No. 6,pp. 1038-43.

Bowie, P., Cooke, S., Lo, P., McKay, J. and Lough, M. (2007), “The assessment of criterion auditcycles by external peer review – when is an audit not an audit?”, Journal of Evaluation inClinical Practice, Vol. 13 No. 2, pp. 352-7.

British Dental Association (2007), “Continuing professional development”, available at: www.bda.org/education/education.cfm?ContentID¼307 (accessed 12 December 2007).

Campbell, L.M., Howie, J.G.R. and Murray, T.S. (1993), “Summative assessment: a pilot project inthe west of Scotland”, British Journal of General Practice, Vol. 43 No. 375, pp. 430-4.

Chartered Society of Physiotherapy (2003), Continuing Professional Development (CPD) Briefingand Policy Statement, CSP, London.

Chur-Hansen, A. and Koopowitz, L.F. (2005), “Formative feedback in teaching undergraduatepsychiatry”, Academic Psychiatry, Vol. 29 No. 1, pp. 66-8.

Davies, H., Powell, A. and Rushmer, R. (2006), Healthcare Professionals’ Views on ClinicianEngagement in Quality Improvement: A Literature Review, The Health Foundation,London.

Department of Health (2006), Good Doctors, Safer Patients: Proposals to Strengthen the System toAssure and Improve the Performance of Doctors and to Protect the Safety of Patients.A Report by the Chief Medical Officer, Department of Health, London.

Healthcare Commission (2007), “Developing the annual health check 2008/09 – have your say”,available at: www.healthcarecommission.org.uk/publicationslibrary.cfm?fde_id¼491(accessed 5 February 2009).

Johnston, G., Crombie, I.K., Alder, E.M., Davies, H.T.O. and Millard, A. (2000), “Reviewing audit:barriers and facilitating factors for effective clinical audit”, Quality in Health Care, Vol. 9No. 1, pp. 23-36.

Lough, J.R.M., McKay, J. and Murray, T.S. (1995), “Audit and summative assessment:a criterion-referenced marking schedule”, British Journal of General Practice, Vol. 45No. 400, pp. 607-9.

McGregor, J., Bowie, P. and McKay, J. (2005), “A pilot study of the feasibility of trainee hospitaldoctors undertaking significant event analysis”, British Journal of Hospital Medicine,Vol. 66 No. 9, pp. 477-80.

McKay, J., Bowie, P. and Lough, J.R.M. (2006), “Variation in the ability of general medicalpractitioners to apply two methods of clinical audit: a 5-year study of assessment by peerreview”, Journal of Evaluation in Clinical Practice, Vol. 12 No. 6, pp. 622-9.

Clinical auditperformance

213

Dow

nloa

ded

by N

HS

ED

UC

AT

ION

FO

R S

CO

TL

AN

D A

t 04:

28 2

7 A

pril

2015

(PT

)

McKay, J., Pope, L., Bowie, P. and Lough, M. (in press a), “External feedback in general practice:a focus group study of trained peer reviewers of significant event analyses”, Journal ofEvaluation in Clinical Practice.

McKay, J., Shepherd, A., Bowie, P. and Lough, M. (in press b), “The acceptability and educationalimpact of a peer feedback model for significant event analysis in general practice”, MedicalEducation.

McKay, J., Murphy, D., Bowie, P., Schmuck, M.-L., Lough, M. and Eva, K.W. (2007),“Development and testing of an assessment instrument for the formative peer review ofsignificant event analyses”, Quality and Safety in Health Care, Vol. 16 No. 2, pp. 150-3.

McLaren, S., Woods, L., Boudioni, M., Lemma, F., Rees, S. and Broadbent, J. (2007), “Developingthe general practice manager role: managers’ experiences of engagement in continuingprofessional development”, Quality in Primary Care, Vol. 15 No. 2, pp. 85-91.

Miles, A., Bentley, P., Polychronis, A., Price, N. and Grey, J. (1996), “Clinical audit in the NationalHealth Service: fact or fiction?”, Journal of Evaluation in Clinical Practice, Vol. 2 No. 1,pp. 29-35.

Murie, J., McCrae, J. and Bowie, P. (in press), “The peer review pilot project: a potential system tosupport GP appraisal in NHS Scotland”, Education for Primary Care.

National Institute for Clinical Excellence (2002), Principles for Best Practice in Clinical Audit,Radcliffe Medical Press, Oxford.

National Patient Safety Agency (2005), Seven Steps to Patient Safety for Primary Care, NPSA,London.

Royal College of Nursing (2003), Nursing Education: A Statement of Principles, RCN, London.

Scottish Executive, NHS Education for Scotland, RCGP (Scotland) and BMA (Scotland) (2003),GP Appraisal: A Brief Guide, Scottish Executive, Edinburgh.

Scottish Office Department of Health (1998), Clinical Governance, MEL 75, Scottish Office,Edinburgh.

Further reading

Royal Pharmaceutical Society of Great Britain (2007), “Continuing professional development”,available at: www.rpsgb.org.uk/registrationandsupport/continuingprofessionaldevelopment/ (accessed 12 December 2007).

Corresponding authorPaul Bowie can be contacted at: paul.bowie@nes.scot.nhs.uk

CGIJ14,3

214

To purchase reprints of this article please e-mail: reprints@emeraldinsight.comOr visit our web site for further details: www.emeraldinsight.com/reprints

Dow

nloa

ded

by N

HS

ED

UC

AT

ION

FO

R S

CO

TL

AN

D A

t 04:

28 2

7 A

pril

2015

(PT

)

This article has been cited by:

1. Carol Fawkes, Elena Ward, Dawn Carnes. 2014. Clinical audit in osteopathic practice: A Masterclass.International Journal of Osteopathic Medicine 17, 110-118. [CrossRef]

2. Kay Donnellon, Grace Hurford, Diane L. Cox. 2013. It's good to talk: auditing clinicians' interactions withpatients in a primary care setting. Clinical Governance: An International Journal 18:3, 220-227. [Abstract][Full Text] [PDF]

Dow

nloa

ded

by N

HS

ED

UC

AT

ION

FO

R S

CO

TL

AN

D A

t 04:

28 2

7 A

pril

2015

(PT

)