automating ward evaluations a tentative first step

Journal of Clinical Nursing 1994; 3: 347-354

Automating ward evaluations: a tentative first step

L. G. MOSELEY MA, MBCSComputer Science Department, University College, Swansea SA2 8PP, UK

D. M. MEAD PhD, RGNCollege of Nursing (£ Midwifery, University College, Swansea SA2 8PP, UK

Accepted for publication 2 December 1993

Summary ' ' '

• Auditing nursing practice is a time-consuming, error-prone task.

• Feedback to individuals is highly desirable hut usually is not offered. This isparticularly difficult in large-scale studies.

-•UJr )>n:

• In this study of 654 wards a new method was attempted. Structuredquestionnaires were analysed by a computer program and feedback reportsgenerated automatically.

• Informal evaluations by the initial recipients are positive and the method seemsworthy of future development.

Keywords: audit, computer, feedback, primary nursing.

a

Introduction

Audit has become a widely used concept and activity in theUK National Health Service. It has been applied to a widevariety of activities, many of which are best categorized asaccountancy—assessing whether the patient, taxpayer, orsignificant other is obtaining value for money. However,one branch of the aetivities under this heading is clinicalaudit. In our view this must precede any other fortii ofaudit. In particular, ways of conducting effective nursingaudit need to he developed.

In this paper we address two important issues:• the need fbr a professional perspective in audit,• the need to ensure feedback to those who provided the

data which enabled the audit to take place.One of the main purposes of audit should be that theinformation gained is used to improve practice. Difficultiesarise because when audits are carried out by managementconsultants there is a danger of the professional perspec-

tive being omitted. When this occurs, a major part oftheusefulness of the exercise is lost.

In our research, of which the work reported here forms asmall part, the main objective was to determine the extentto which primary nursing is practised on wards in Wales.In order to accomplish this it was necessary first to identifythe essential, discriminating features of primary nursingand to devise a tool which would enable a ward to beanalysed in tenns of the presence or absence of thediscriminating features.

The study reported here was commissioned by theWelsh Office, and funded by the Department of Health. Itwas earried out in three phases:• preparation.• developing an instrument,• testing the instrument on a large sample.The first phase (identifying the essential, diseriminatingfeatures of primary nursing) entailed a three-round DelphiStudy. The second phase entailed devising a questionnaire

347

348 L.G. Moseley and D.M. Mead

wbicb could be used to measure tbe extent to wbieb a wardwas using primary nursing. Tbis was based on the consen-sus opinion of tbe experts used for the Delphi study. Tbisquestionnaire, and methods for analysing it, was developedas a Magnitude Ratio Scale, and will from here on bereferred to as tbe MRS. Tbis was tbe instrument wbicbwas used in tbe tbird and final pbase of tbe study, and wasapplied to a universe consisting of all bospital wards inWales.

The principal researcher was a nurse, tbe 28-memberDelpbi panel consisted of nurses and tbe 654 respondentswere also nurses, and benee tbe people to wbom we bad tofeedback any evaluative information were nurses. Giventbis structure, we believe tbat tbe danger of omitting tbeprofessional perspective has been minimized. In addition,an indieator of tbe tool's validity sbould be tbe extent towbicb it accurately described practice and was instrumen-tal in bringing about development. Several measures ofsucb validity were included in tbe original researcb. In all,tbe underlying study of wbicb tbe work bere formed partwas clearly rooted in nursing practice, and has a genuinelyprofessional perspective. We believe tbat this involvementof nurses not only avoided the dangers mentioned above,but also bad a positive effect upon tbe researcb process.

It bas to be asked of any reporting process 'Wbo makesup the audience?' A report may go to a healtb authority, ahospital, or a unit manager. Tbe report may circulate,either widely or to a select few. A seminar or study daymay be arranged to inform staff more generally of thebroad overall results. It is seldom tbe case, tbougb, tbat tbeindividuals wbose work is being audited, and wbo were tbemain providers of data, receive any structured informationabout bow tbey or tbeir wards stand, eitber relative tootber individuals or wards, or to some absolute standard.

, Tbis is comprehensible for two reasons.Firstly, given tbe scale of many audits, it may be costly

or time-consuming to do tbis. In our own case, to supple-ment our overall findings about wards, specialities, bealthauthorities, etc., we would have bad to prepare 654individual feedback reports, eacb potentially based upontbe answers to 76 questions. Tbat in itself would bave beena daunting task. Even allowing, say, only 2 b for tbepreparation of eacb report, it would have involved almost aperson-year of work for tbis task alone, ignoring everyother aspect of tbe researeb. uo lr)i

Secondly, tbere are psycbologieal factors involved. Itmigbt be embarrassing or otberwise difficult to offer anbonest (and tbus in some cases critical) evaluation ofindividuals or wards in a widely circulated report or in apublic seminar.

However, if an audit or an evaluation is to bave any

beneficial effect, it is precisely tbe individuals wbo need tobe informed, because it is tbeir bebaviour wbicb will needto cbange if improvements are to be brougbt about. Thus,tbe audit process normally omits tbe most importatitaudienee.

Tbere is also a longer-term problem. It appears to be thecase tbat, wben audit (or otber researcb) is earried out, thenurses serve as providers of data, but are rarely given thestatus of reeeivers of tbe derived information. Tbis appearsto have the effect of lessening their commitment to the useof structured, carefully gathered research data in thedevelopment of botb tbe service and tbe profession (Luker& Kenrick, 1992). If tbose wbo participate as the subjectsof research, or as providers of data, receive feedback whtchtbey find useful, they are more likely in the future both toparticipate willingly in sucb researcb, and to try to ensuretbat tbe quality of data is bigb.

We believe tbat tbe method reported here offers thepossibility of providing individual feedback in a fortnwbicb reeipients find aeceptable and useful, and in a waywbicb means tbat it can be carried out on a large scale.Given that the data have been gathered, the effort involvedin preparing additional reports is almost independent ofthe number of reports. Preparing 1000 reports involvesvery little more effort tban preparing one.

THE PLACE OF THE COMPUTER

Computers are commonly used as part of tbe process ofaudit and evaluation. Their part has historically been thatof a sophisticated calculator, merely doing the sums.However, we would wish to argue that there should be amuch larger place for them in the overall process of audit.Computers can not only perform numerical calculations;they can also do a substantial amount of symbolic manipu-lation.

Many people would claitn that computers can actuallyreason. With many systems, this appears to be the case,especially when tbey offer sopbisticated justifications fortheir eonclusions. However, we eschew such anthropocen-tric terms as reasoning, thinking, or artificial ititelligence,as they arouse unneeessary emotional reactions, lead tosterile philosophical debates (see Dreyfus & Dreyfus, 1986;Dowie & Elstein, 1988), and are in atiy case unnecessary.We shall restrict ourselves to saying that, properly pro-gramed, eomputers can manipulate symbols. These sym-bols can represent words or concepts, as well as nutnbers.

Knowledge-based sys tems ,̂ ,,;,^^ ;„,, ,,̂One branch of moderti computing, which is rapidlyapproaching tnaturity, is that of knowledge-based systems,a sub-branch of the mote general field of expert systems.

Automating ward evaluations 349

Tbis involves developing computer programs wbieb per-form tasks wbicb, if performed by a buman, would be saidto demonstrate expertise. Tbese programs also performtasks in a way whieli is modelled upon the way in whichhuman experts either say that they undertake such tasks(using conventional knowledge-acquisition methods, suchas interviews) or in the way that they actually do it (byanalysing past decisions), whether or not that is how theysay tbat tbey do it. For a fuller diseussion of sucb systemsand tbe purposes to wbich they have been put, see Parsaye& Chignell, 1988; Quinlan, 1987, 1989; Schorr & Rapo-port, 1989; Smith & Scott, 1991; Rapoport & Smith, 1991;Scott & Klahr, 1992. We stress tbat tbis is a sub-branch,because there are other approaehes which can emulateexpertise (statistical systetns, neural networks) and canoutperform tbe buman expert, but wbicb perform tbe taskin a tborougbly non-buman way.

Knowledge-based systems are intended to emulate tbeexpert, and tbey bring witb tbem certain advantages.1 The programs can be made comprehensible to a non-comp-uting specialist, particularly to tbe domain expert (a nurse, aphysician). Tbe syntax used in tnany expert system shells(tbe programming languages normally used for this pur-pose) is frequently very close to everyday English. Anactual section of a program migbt read:

if claudication is Reportedand night-pain is Reportedand ankle_bracbiaLindex < 0-7and do_feet_feel_cold is Yesand cigarettes_per_week > 60and other._arterial_eonditions are Presenttben possible-problems include 'Iscbaemie ulcer'

We believe tbat most nurses could understand tbis andcould comtnent on its accuracy, note tbings wbicb areincluded wbich should not be included, suggest difl'erentcut-ofi" points (should cigarettes_per_week be eut of!" at 50,ratber tban 60, say), draw attention to factors wbicb mayhave been omitted, to special cases where tbe rule does notapply, suggest breaking one rule up into several rules, etc.

2 The program can explain its chain of inference. As tbesystem is sytnbolic, and is represented inside tbe cotnputeras a grapb or tree, it is possible for it to trace its reasoning,and to display (on request if necessary) tbe input data andintermediate concepts wbieh led it to whatever conclusionit bas drawn. Tbus, when asked why it thinks tbat tberemigbt be an iscbaemie ulcer, it sbould be able to state tbatan examination sbows tbat tbe patient bas cold feet, reportssymptoms wbicb can be interpreted as sbowing intermit-tent claudication, is a beavy smoker, etc., and sbould beable to do it in plain English.

3 Like an expert, they can develop over time. With mostconventional computer programs, changing one line canbring dramatic cbanges in tbe operation of tbe program.With a knowledge-based system it is relatively easy to addnew knowledge, witbout destroying tbe overall logic of thesystem. In tbe coded rule above, for example, it wouldrequire only one line to add a rule wbicb increased tbeebances of an ulcer being isebaemic if tbere was a poplitealpulse but none could be felt at the dorsalis pedis. Similarly,if one wished to have different rules for elderly patients it iseasy to insert a rule or a clause wbich starts,

if patient_age > 60 and . .

Thus, it is easy to start oS witb a system wbicb can dealwitb a lot of common conditions but w bicb migbt flounderwben it encounters rare and unusual eases, and tbengradually add rules to take account of tbese rarer or morecomplex decisions. Tbe system can, in a sense, get smarteras it gets older.

AN EXAN4PLE

Tbe data set for tbe nursing study was voluminous. Foreacb of tbe 654'wards (covering 38 specialties) we bad 49items of background data (staffing levels, bed levels, rotaarratigetnents, levels of qualification, etc.). In additiion,tbere were 34 items (derived frotn the Delphi study) whichwere intended to measure aspeets of ward organization (ofwhich 16 contributed to tbe magnitude ratio scale onprimary tiursing—a ward could score from 0 to 543'2 ontbis scale). Tbe scoring was fixed and was entirely under-taken by tbe eomputer. Tbere were also 5524 free textcotnments to be analysed. Tbe data were stored in tbeParadox relational database. The simpler analyses wereundertaken using tbis tool; more sophisticated statisticalanalyses were undertaken using SPSS-PC + . Some detailsof the tnethods and substance of the research have beenpublished elsewhere (Moseley & Mead, 1992).

One of the major concerns of the principal researcher(and of the customer) was tbat tbere sbould be somefeedback to tbe participating ward sisters. This could haveconsisted of sitnply sending them a copy of the 200-pagereport. Clearly tbis would bave been unsatisfaetory. Manywould have been swamped by tbe sbeer volume of mater-ial, and for tnost of thetn the tnethodological details wouldnot have been relevant to tbeir own situation. However,otie of their major interests would have beeti an assessmentof the standing of their own ward, botb in comparison witbsome absolute standard, and in comparison witb tbestanding of otber wards. Feedback in tbis ease, tben, meantindividual feedback.

Given tbat there was a tnass of data on eaeh w ard, even


to analyse one ward by any conventional means would havebeen a time-consuming process. To do it for all 654 wards(with 54 282 structured answers and 5524 free text com-ments) would have been difficult and would have requireda skilled team, probably working for a substantial period oftime. It would have been difficult to ensure the applicationof consistent standards of judgement and interpretation. Inaddition, there would have heen a serious possibility thatby the time such an analysis was completed, the situationwhich it purported to deseribe would have changed be-yond all recognition. Some better way was needed. It hadto be fast, it had to be accurate, and it had to he open andjustifiable.

A FIRST ATTEMPT AT AUTOMATED FEEDBACK

The computing consultant who had a major interest inknowledge-based and other, expert systems approaches(see Moseley & Cartwright, 1992; Preece & Moseley, 1992)and in data base design was asked, as an experiment, quiteseparate from the main body of the research, to write ashort program which attempted to automate the feedbackprocedure. This reads the questionnaire for a ward and,applying relatively simple rules, generates and prints anevaluative report on that ward. We would stress verystrongly at the outset that it was only a first attempt, andwe would not wish to claim more than that the approachlooks promising enough to merit further investigation.

Firstly, the program itself was designed by a computingspecialist. It is quite possible that if the program had beendesigned by someone whose primary specialism was nurs-ing, the structure and content might have been different.However, the computing consultant's involvement in theproject had given him a grasp of the fundamentals of theterminology and a feel for the domain. In addition, thenursing specialist on the team checked, suggested changesto, and confirmed the structure and wording used. Overall,although, as we report below, initial reaetions were posit-ive, and we believe that the general outline is broadlyuseful, there is probably room for improvement.

Secondly, it was undertaken very rapidly. The programwas initially written purely as a demonstration to the nurseresearcher of the sort of thing which was possible. It wasnot part of the planned research; it developed from whatbegan as an interesting idea. Were the exercise to berepeated as a planned part of the research in the longerterm, rather than as an aside, there are a variety ofcomputer science-techniques (especially natural languagegeneration) which could be used to produce much moresophisticated results.

Thirdly, it was written using, in knowledge-based sys-

tem terms, relatively primitive technology. No specialistexpert system development tool was used. Because the datawere stored in a data base written in Paradox, the programwas written in PAL (the Paradox Application Language).This is not an ideal tool for the job. If one were to use adedicated expert systems tool, the degree of sophisticationin reporting could be considerably enhanced.

Thus, what we are reporting here is an interestingexperiment. However, we think that the results are inter-esting, and sufficiently encouraging to justify a follow upon a large scale.

THE PROGRAM

The program is, in computing terms, very simple. Itanalyses the responses to two questions on each of 11 majoridentified dimensions of primary nursing, and one ques-tion each on five other dimensions. Dimensions includedmajor features such as devolved decision making, account-ability, continuity of care, the named nurse concept (whichturned out to be more complex than we had thought),down to more minor ones such as the presence of visibleevidence of primary nursing (photographs, notice boards,etc.). As there were two questions on most dimensions, award could have answers which approached the ideal type(in Weberian terms) on both question A and B, on A butnot B, on B but not A, or on neither. When there were twoquestions on a dimension, question A was stronger evid-ence for the presence of the dimension on the respondingward than was question B (question A contained the mostcommon phrase which emerged from the Delphi studywhen our experts were defining the relevant dimension,and thus a 'correct' answer on this question would suggestvery strongly that the dimension which it was intended tomeasure was in fact present on the responding ward.)Thus, for most dimensions there were four possibilities.The computer was programmed to prepare cotnments foreach of these four possibilities (or two on some dimen-sions) for each ofthe 16 dimensions, and finally to makesome sort of overall evaluative comment.

Although this is a simple demonstration (with, forexample, no attempt at randomizing the wording to makereports look superficially different one from the other), andit might be tempting to assume that the reports would tendall to look the same, there can in fact be a wide variety ofresponse patterns by nurses, and hence a wide variety ofdifferent evaluative reports, even given the crudeness ofthe attempt. In principle, in our study, we eould empiri-cally have observed anything from one pattern (if all wardshad answered in identical ways) to 654 patterns (if eachward had given a unique pattern of answers, all different in


at least one respect from any other ward). In fact, weobserved 477 different patterns of responses on the majorquestions. Thus there was considerable variability in theanswers actually obtained—and therefore in the reportswhich could be generated. Given that there were 11questions, each of which could be answered in one of fourways, and five questions, each of which could be answeredin one of two ways, in theory there are over 30 millionpossible different reports which could be generated, al-though in practice answers are likely to cluster and toreduce this number somewhat (in our study 27% of wardsproduced patterns of answers identical to at least otie otherward). However, the crucial point is that a very widevariety of different reports can be produced, depetidingupon the answers given by a given ward or individual.

This illustrates a conclusion which always emergeswhen one tries to formalize judgements—they are poten-tially much more complex than one would ever imagine.We give below a typical report on a real ward, although ithas, of course, been made anonymous.

CONFIDENTIAL-WARD SISTER'S REPORTDATE: 12 June 1993The questionnaire which you completed was designed totry to measure 16 dimensions, which are commonly said tocharacterize primary nursing. Together they make up an'ideal type'. The word 'ideal' in this context does not mean'highly desirable'. Rather it represents an abstraction. If award showed evidence of all 16 dimetisions, it would be asdose to the primary nursing model as it would be possibleto get. Of course, in the real world, it would be unusual (ifnot impossible) for a ward to exhibit all 16 dimensions.However, if a ward showed the presence of only a few, oreven none, of these dimensions, it would be difficult toargue that it was practising primary nursing.

It should be noted that a case can be made for severalforms of ward organization, and primary nursing is onlyone of these.

The dimensions which we tried to measure were1 Aecountability, authority, responsibility for a caseload

of patients2 Care delivery centred around individual patient needs3 Case load attachment from admission to discharge4 Continuity of care5 Primary nurse as care giver6 Evidence of a philosophy/value system7 Decentralized decision making8 Care plans and care planning reflect that the primary

nurse is the principal organizer of care9 Changes in ward organization . ; ,

10 Communication pathways indicate that the primarynurse is the principal organizer of care

11 Patient/relative involvement and choice12 Appropriate skill mix13 Evidence of a role change for the nurses involved14 Development of collegiate relationships15 Patients know their nurse16 Visual evidence of a system

HOSPITALSISTERWARDSPECIALTY

St SomewhereNonymous, AnnContinuing CareEMI

Overall Ward OrganizationYour own eategorization of your ward organization fitsfairly closely to the standard model which is normallycalled primary nursing. The general pattern of youranswers supports such a view.Although you are not entirely consistent on the point, youdo seem to be interested in the concept of deeentralizeddecision making.Although the ward has a written philosophy of care, itcould have benefited from a wider involvement in itsdevelopment.Accountability seems not to be a major factor in your viewjudging by your answer to question 16. However, thiscontradicts your answer to question 31.Continuity of care is a major theme in your approach toyour professiotial task. This showed in your answers toquestions 20 and 35.You lay stress on both patients and relatives being awarewho is the individual nurse responsible (even by name).This is consistent with the primary nursing model.The individual nurse is involved when her patients areseen during the ward round. This is the usual practice inwards which are said to be practising primary nursing.Knowledge of the allocation of duties is widely spread andthere is visual evidence of it throughout the ward. Thus,the knowledge is public.

Pat tern of care organization and deliveryYour practice has a strong component of case load attach-ment. This fits in neatly with the generally acceptedprimary nursing model.The idea of care planning by the nurse responsible forgiving care is a major theme in your approach to yourprofessional task.You seem to lay considerable stress upon the idea thatdelivery of care is an individual matter and should not bedetermined entirely by ward routine needs.


You feel it important tbat formal communieation andcontact is cbannelled tbrougb tbe primary nurse forrelatives but not for medical staff.Tbe organization of tbe off-duty rota is one wbicb iseonsistent witb a ward wbicb practises primary nursing.

CommunicationYour answers suggested that communication tbrougb tbeprimary nurse is a major tbeme in your tbinking. Tbis is anelement wbicb manifests itself strongly in your ward.On your ward tbere seems to be a large element ofcoUegiality, i.e. nurses actually read and follow colleagues'care plans, and sister does not cbeck tbeir efficacy, unlessrequested to do so.

Your role as the sister of continuing careAs sister, you see one of your responsibilities as being toensure tbat procedures and policies are followed. Tbislimits tbe freedom of individual nurses. It is tbe eommon-est form of leadersbip.

SummaryOverall your ward bas most of tbe major features ofprimary nursing. In addition, it bas most of tbe minorones. Your numerical score was witbin 20% of tbe max-imum possible.

Evaluation of the feedback

Because tbis form of automated feedback was not part ofour original researcb design, no formal evaluation of it wasplanned. As a result of tbe work reported bere, we are, ofcourse, now in tbe process of undertaking a more formalevaluation. As a precursor to tbis we bave investigatedwbetber sucb a formal evaluation is likely to be wortb-wbile, and bave briefly investigated a number of ways ofassessing tbe usefulness or otberwise of tbis form offeedback to ward sisters and tbeir staff. Eacb sbould beregarded as no more tban a small pilot study. We havesince devised an instrument for assessing tbe usefulness oftbese feedback reports, and a more formal evaluation is inproeess.

1 SINGLE WARD FEEDBACK

A small number of ward sisters (six) was given tbefeedback generated by tbe computer from tbeir own ward'sresponses. Eacb of tbe sisters was told tbat tbey were beinggiven an example of tbe kind of feedback wbicb waspossible from tbe questionnaires completed in tbe larger

Study. Tbey were not told tbat the report was, in fact,about tbeir own ward.

Eacb was asked:• How useful would you find tbis feedback.''• Do you bave any general comments tbat you would like

to make about it.'All of tbe sisters said tbat tbe feedback was useful. One oftbem made a comment suggesting tbat tbe report was anapproximation of tbe praetiee on ber own ward. Tbeverisimilitude of tbe report struck otber respondents aswell.

Examples of unprompted comments received fromotber sisters included 'tbis report is very belpful, and itsounds just like my ward'; 'it could be my ward'. It appearstbat, even witb the simple approacb wbicb we adopted (astructured questionnaire, tbe answers of wbicb are inter-preted according to strictly defined and programmedrules), not only were tbe reports useful, but tbey actuallyseemed to catch the flavour of a ward. We were somewhatsurprised tbat a series of simple rules could capturesufficient information about a ward to stimulate tbe com-ments above, but they appeared to do so.

2 REVIEW OF A SELECTION OF WARDS

A furtber six ward sisters were given a seleetion of reportscontaining feedback. In tbis case, none of tbese were fromtbeir own wards, and they were told tbat tbis was tbe case.Examples included reports on very bigb-scoring (i.e. con-taining many features of primary nursing) wards and ofvery low-scoring wards. Tbe sisters were asked to com-ment on tbe usefulness of tbese reports. Altbougb tbesample was too small to draw firm conclusions, some usefulinsigbts bave been received, and some of tbese will resultin sligbt cbanges in the wording of tbe feedback reports.

Some comments on reports on a high-scoring tvard

'I found it (tbe feedback) belpful because it stronglyreinforces tbe work currently being done on my ward—itgives suggestions for furtber discussion and development.''I liked tbe division of tbe report and tbougbt it lent itselfwell to easy understanding of tbe feedback.''Tbere seems to be general dissatisfaction witb tbe ideatbat only primary nursing is the goal, instead of reeognis-ing tbe value of tbe acbievements and skills at tbat time.Tbe tool allows for this.''I did not fully understand tbe statement "you lay stress onrelatives being aware . . .".''Tbis feedback is the one I would aspire to receive. I foundit helpful.'


Some comments on reports on a medium- to low-scoringward Vi n n lirif. iiju-.^i AM;:; . ' I I M J C J

'On first reading, I felt the individual feedbaek was clear,concise and extremely 'usable'. It was only on subsequentreading that I hegan to feel frustrated hy the conciseness ofthe infortnation (for example, the report states that it doesnot sound like the standard form of primary nursing). Ibegan to wonder what sort of nursing it was.'.'The report was positive even in the giving of f\iirlynegative information. For example, "your ward has nowritten philosophy of care. It is often thought to beworthwhile having one".''No statement was damning in the feedback and I thoughtthis was particularly helpful.''Nurses need to be given rcassuranee that not aehievingprimary nursing is not the end ofthe world — rather theymay reflect on the infortnation given (in the report) and useit to develop their skills and their tnethods of deliveringpatient care.''I found the use of four headings in the feedhack useful.The divisions tnade understanding of the report easier atidgave some pointers for discussion later.''I found the sumtnary quite elating but I wanted itnmedi-ately to know what the tnajor features of primary nursingwere. Therefore, I would have wanted quick access to theresearch' (on which the logic of the report was based).'I found it a useful tool for evaluating progress and wouldhave been pleased to have this information on my ward.'

3 DISCUSSIONS WITH STAl'F

Two ward sisters, with responsihility for 14 pritiiary stafl,were asked to discuss the feedback about their own wardwith those staff. It should be noted that the feedbaek wasgenerated frotn a questionnaire which had been completedby the sister alone.

In almost all cases, the staff agreed that the feedback wasa fair refleetion of the praetice on their ward. However,there was one example in which there was some disagree-ment. It did, interestingly enough, have a positive out-come. In one ward, four primary nurses (out of seven onthe ward) said that they felt that patient and relativeinvolvetnent and choice was not part of their practice,whereas the report (generated from the sister's question-naire alone) had said that such involvement and choice waspart of their practice. Following discussioti among theward team, it becatne evident that, although the sister hadthought that this concept had become well established, infact some of the primary nurses had not yet developed theapproach. This was due to their newness in the role. On

the ward concerned, as a result of the feedhack, a trainingprogramme has been established for these nurses.

This is a typical example of the automated feedhackbeing used to generate discussion among the ward team. Inthis case, it resulted in a change of practice. We shall in duecourse consider whether a case can be made for generatingfeedback on an individual, rather than a ward, level.

Impressions gained from this informal pilotstudy

1 This feedbaek, whether or not it was about their ownwards, has stimulated the sisters concerned to examinetheir own practice.2 In order to maximize the potential usefulness of suchautomated feedhack, additional information may need tohe included in the reports, e.g. an indication of the majorfeatures of primary nursing (although these are listed inthe feedback report, they are not elaborated).3 Some phrases may need to be altered to improve clarity.4 Even though the report draws attention to deficienciesin a given ward's approximation to the primary nursingmodel, it does not do this in a negative way. On thecontrary, the phrases used (even when critical) were foundto he constructive.5 All of the respondents said either that they found ituseful, or that they would have found it useful had it heenabout their own ward.

One might have expected that computer-generated re-ports would appear to sisters as being rather meehanieal,inhuman and stylized. It appears from many of thecomments that this was not often the case. Reactionsincluding words such as 'positive', 'particularly helpful','reassurance', 'elating', 'I would have been pleased to have. . . ', 'helpful' do not suggest that the reports gave anyfeeling of inhutnanity or mechanistic reporting. It wasstriking that none of the sisters spontaneously declaredthat they thought that the feedhack report was in any wayautomated, still less completely eomputer-produced.When asked outright whether they thought that thefeedback report might be eomputer-generated, all of themexpressed surprise at the idea. ffri !• ,;̂ !lur'."it'M. nvjiisq

"il

Conclusion

Our major conclusion is that even a simple attempt atautotnating evaluation is worth undertaking. Clearly, weneed to undertake tnore rigorous evaluation, and to con-sider changes to the nature of the report. In particular, weare currently testing it on a larger sample of wards, andobtaining more formal evaluations of the strengths and


weaknesses of the reports generated. However, overall, inthis small pilot study, the idea appeared to be feasible andpromising.

Naturally, we have some concerns which must be met ifthe approach is to be used in future.• If it were carried out regularly, respondents could

become 'test-sophisticated' and could fake good (or bad).Clearly, one would need to develop an instrument whichminimized this tendency (check questions, banks ofrandomly chosen equivalent questions, false-friendquestions, etc.)

• If it is done mindlessly, it eould produce misleadingresults.

We would note that these concerns apply equally, if notmore so, to non-computerized evaluation, and such evalu-ations are fraught with other social and psychologicalproblems as well.

However, it is clear that it is possible to write a programwhich takes input from ward sisters and can, withouthuman intervention, write a report which respondents findinformative and useful. Still more, it can stimulate discus-sion, debate, and development. It may be that its rigourfocuses attention more clearly than would be the case if theresult was made by a human expert. It obviously over-comes some of the interpersonal problems (anger, jealousy,etc.) which can arise if the report had been made by ahuman being.

We noted earlier that audit or evaluative reports arerarely provided at the individual level. In the current work,it is simple, indeed automatic, to provide feedback at thatlevel. This is useful from more than one point of view.Firstly, it actually gives some feedback at that level, whichwould be an improvement on much current practice.Secondly, if sisters receive feedback, especially feedbackwhich they find helpful, it is more likely that they will bewilling to co-operate in audit, evaluation, or research in thefuture. Thirdly, there is an effect on the researchers. Whenresearchers have feedback in mind, they have to think veryclearly indeed about what conclusions they may or may notdraw from their research, and they have to specify in anovert manner exactly how they would interpret a givenpattern of results. This helps considerably in the produc-tion of usable research.

Perhaps the most surprising element to emerge fromthis attempt to computerize evaluation is that it hasimportant and useful non-computing effects. Teacherscommonly find that explaining their special subject to astudent who is having difficulty in understanding thesubject often clarifies the teacher's own understanding ofthe subject. It is fair to describe a computer as, at best, anot particularly bright student. The process of writing a

program is the equivalent of explaining the subject. As thecomputer has no background knowledge and no common-sense, one has to be particularly precise and accurate.Above all, researchers have to be able to state in advanceexactly how a particular pattern of responses would beinterpreted. We believe that apart from the obvious utilityof rapidly generated, consistent and justifiable reports, thispsychological effect of encouraging clear thought is a majorbenefit in the use of this approach.

Thus, by specifying clear rules for interpreting theanswers to a structured questionnaire, and encoding themin a program, we appear to have been able to produceuseful reports. This offers not only the possibility ofoffering a relatively inexpensive method of providingfeedback at an individual level, but also a feedback which isconsistent and justifiable. It is not mysterious or arbitrary.We believe that this would add an important dimension tothe process of evaluation and audit. Evaluation informa-tion can go directly to the individuals who provided thedata on which it is based. Not only would this improvetheir motivation for assisting in the research and evaluationprocess, but it would also ensure that the feedback goes tothe very people who are likely to make best use of it. Webelieve that the idea is worth pursuing on a larger scale.

References

Dowie J. & Elstein A. (1988) Professional Judgment. CambridgeUniversity Press, Cambridge.

Dreyfus I l.L. & Dreyfus S.r:. (1986) Mind over Machine. Free Press.Luker K.A. & Kenrick M. (1992) An exploratory study of the

sources of influence of the clinical decisions of community nurses.Journal of Advanced Nursing 7, 457-466.

Moseley L.G. & Mead D.M. (1992) The integration of a databaseand statistical program in the analysis of a large-scale survey innursing. Nursing Practice 5(3), 23-28.

Mo.seley L.G. & Cartwright M. (1992) The development of anexpert system for operational use in the .selection of industrialadhesives. Engineering Applications of Artificial lntelligettce 5,319-328.

Parsaye K. & Chignell M. (1988) Expert Systems for Experts. Wiley,Chichester.

Preece A.D. & Moseley L.G. (1992) Empirical study of exportsystems development. Knowledge-Based Systems 5(2), 137-148.

Quinlan J.R. (1989) Applications of Expert Systems. Vols 1 & 2.Addison Wesley, New York.

Rapoport A. & Smith G.R. (1991) Innovative Applications ofArtificial Intelligence, Vol. 2. AAAI Press, Menlo Park, CA.

Scott A.C. & Klahr P. (1992) Innovative Applications of ArtificialIntelligence, Vol. 4. AAAI Press, Menlo Park, CA.

Schorr H. & Rapoport A. (1989) Innovative Applications of ArtificialIntelligence, Vol. 1. AAAI Press, Menlo Park, CA.

Smith G.R. & Scott A.C. (1991) Innovative Applications of ArtificialIntelligence, Vol. 3. AAAI Press, Menlo Park, CA.

automating ward evaluations a tentative first step

Documents