an investigation into the use of a structured clinical operative test for the assessment of a...

7
Eur J Dent Educ 2001; 5: 31–37 Copyright C Munksgaard 2001 Printed in Denmark. All rights reserved ISSN 1396-5883 An investigation into the use of a structured clinical operative test for the assessment of a clinical skill B. J. J. Scott, D. J. P. Evans, J. R. Drummond, P. A. Mossey and D. R. Stirrups Dundee Dental Hospital and School, Park Place, Dundee DD1 4HR, Scotland, UK Aim: This study was designed to investigate the level of agree- ment between a group of assessors observing students under- taking a structured clinical operative test. Method: 3 assessors agreed a series of criteria to assess the performance of undergraduate students in the recording of a dental impression. Guidelines for assessing whether the stu- dents adequately performed in relation to each criteria were also agreed. Following preliminary validation between the assessors, 2 assessors independently scored the performance of each stu- dent by reference to the agreed criteria, and the levels of agree- ment between assessors were compared. The 3 assessors worked in pairs with each other on three groups of students who were in the early stages of their clinical course. Results: A total of 39 clinical dental students were assessed in the recording of a dental impression. The 3 pairs of assessors had satisfactory levels of agreement in the study with similar judgements being made on 90% or more of the 12 criteria as- sessed. Some differences existed between the pairs of as- sessors. Certain criteria were more easily judged than others and this was reflected in the level of agreement seen. For over 90% T HERE HAS been a gradual move towards the devel- opment of competency-based curricula for clin- ical dentistry in undergraduate and postgraduate clin- ical training programmes over the last decade (1). One of the objectives of the undergraduate curriculum is to equip the new graduates with sufficient diagnostic and treatment skills to enable them safely to deliver dental care to their patients. Furthermore, the under- graduate curriculum should act as a starting point in the acquisition of new skills in vocational training, general professional training and specialist training. In a competency-based curriculum, training would be directed to acquiring specific core skills in a wide range of clinical disciplines and these could continue in relation to the development of specialist skills (2). Furthermore, competencies should define minimum attainment and allow for individual and optimal aca- demic development (3). Competency-based pro- grammes are being developed at an undergraduate and postgraduate level in areas such as Prosthodon- tics (4, 5). 31 of the criteria, positive assessments were made by each of the 3 assessors, and although there were minor variations between the pairs of assessors, this may have reflected the ability of the groups of students studied. Conclusion: The study showed that different assessors were generally able to make agreed judgements on performance cri- teria in a structured clinical operative test. In setting up perform- ance assessment it is necessary to have close collaboration be- tween assessors to make clearly defined criteria so that judge- ments are not too subjective. Furthermore, for the assessment of more complex clinical skills, great care is needed in assembling criteria that can be used reproducibly, and sufficient preparation time for the assessors is critical. Key words: dental education; clinical skill; impression recording; performance assessment. c Munksgaard, 2001 Accepted for publication 12 July 2000 Competencies are defined as a set of performance values, which are supported by understanding, and professional values, which are necessary to start the independent practice of dentistry (6). This can define the behaviour expected of beginning independent practitioners (7). Historically, an overall level of per- formance has been used as a marker for independent professional practice, but this is changing with more specific fields being developed as an alternative way to make the assessment (8). Clinical skills might be assessed in a number of ways including clinical skills assessment (9), struc- tured skill testing (9, 10), the clinical logbook (9) and portfolio evaluation (6). Performance analysis is be- ginning to be used in the assessment of operative skills (11). However, the assessment of performance in an objective manner is challenging. Many factors may contribute to the measurement of performance includ- ing the consistency of those who are making the judgement. For example, in a study on final year den- tal students undertaking clinical procedures in re-

Upload: b-j-j-scott

Post on 06-Jul-2016

214 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: An investigation into the use of a structured clinical operative test for the assessment of a clinical skill

Eur J Dent Educ 2001; 5: 31–37 Copyright C Munksgaard 2001Printed in Denmark. All rights reserved

ISSN 1396-5883

An investigation into the use of a structured clinicaloperative test for the assessment of a clinical skill

B. J. J. Scott, D. J. P. Evans, J. R. Drummond, P. A. Mossey and D. R. StirrupsDundee Dental Hospital and School, Park Place, Dundee DD1 4HR, Scotland, UK

Aim: This study was designed to investigate the level of agree-ment between a group of assessors observing students under-taking a structured clinical operative test.Method: 3 assessors agreed a series of criteria to assess theperformance of undergraduate students in the recording of adental impression. Guidelines for assessing whether the stu-dents adequately performed in relation to each criteria were alsoagreed. Following preliminary validation between the assessors,2 assessors independently scored the performance of each stu-dent by reference to the agreed criteria, and the levels of agree-ment between assessors were compared. The 3 assessorsworked in pairs with each other on three groups of students whowere in the early stages of their clinical course.Results: A total of 39 clinical dental students were assessed inthe recording of a dental impression. The 3 pairs of assessorshad satisfactory levels of agreement in the study with similarjudgements being made on 90% or more of the 12 criteria as-sessed. Some differences existed between the pairs of as-sessors. Certain criteria were more easily judged than others andthis was reflected in the level of agreement seen. For over 90%

THERE HAS been a gradual move towards the devel-opment of competency-based curricula for clin-

ical dentistry in undergraduate and postgraduate clin-ical training programmes over the last decade (1). Oneof the objectives of the undergraduate curriculum isto equip the new graduates with sufficient diagnosticand treatment skills to enable them safely to deliverdental care to their patients. Furthermore, the under-graduate curriculum should act as a starting point inthe acquisition of new skills in vocational training,general professional training and specialist training.In a competency-based curriculum, training would bedirected to acquiring specific core skills in a widerange of clinical disciplines and these could continuein relation to the development of specialist skills (2).Furthermore, competencies should define minimumattainment and allow for individual and optimal aca-demic development (3). Competency-based pro-grammes are being developed at an undergraduateand postgraduate level in areas such as Prosthodon-tics (4, 5).

31

of the criteria, positive assessments were made by each of the3 assessors, and although there were minor variations betweenthe pairs of assessors, this may have reflected the ability of thegroups of students studied.Conclusion: The study showed that different assessors weregenerally able to make agreed judgements on performance cri-teria in a structured clinical operative test. In setting up perform-ance assessment it is necessary to have close collaboration be-tween assessors to make clearly defined criteria so that judge-ments are not too subjective. Furthermore, for the assessment ofmore complex clinical skills, great care is needed in assemblingcriteria that can be used reproducibly, and sufficient preparationtime for the assessors is critical.

Key words: dental education; clinical skill; impression recording;performance assessment.

c Munksgaard, 2001Accepted for publication 12 July 2000

Competencies are defined as a set of performancevalues, which are supported by understanding, andprofessional values, which are necessary to start theindependent practice of dentistry (6). This can definethe behaviour expected of beginning independentpractitioners (7). Historically, an overall level of per-formance has been used as a marker for independentprofessional practice, but this is changing with morespecific fields being developed as an alternative wayto make the assessment (8).

Clinical skills might be assessed in a number ofways including clinical skills assessment (9), struc-tured skill testing (9, 10), the clinical logbook (9) andportfolio evaluation (6). Performance analysis is be-ginning to be used in the assessment of operativeskills (11). However, the assessment of performance inan objective manner is challenging. Many factors maycontribute to the measurement of performance includ-ing the consistency of those who are making thejudgement. For example, in a study on final year den-tal students undertaking clinical procedures in re-

Page 2: An investigation into the use of a structured clinical operative test for the assessment of a clinical skill

Scott et al.

storative dentistry, the clinical teachers gave lowerratings than the students’ own self-assessment ratings(12). Furthermore non-standardised clinical taskspresent major problems in achieving objective levelsof assessment since the criteria for performance mayvary.

A structured clinical operative test (SCOT) uses avalidated checklist in which clinical tasks can be as-sessed (9). It allows the opportunity for formativefeedback and, because the criteria are clearly defined,it should allow more reliable assessment. However, asit is usually not possible for a single individual tomake all of the assessments, it is critical that differentassessors are working to agreed standards when as-sessing performance. To our knowledge, this issue hasnot been addressed in the assessment of clinical skills.The aim of the present study was to investigate thelevel of agreement between a group of assessors whoobserved students undertaking a SCOT. The record-ing of a dental impression was chosen for this studyas it is generally regarded as a core skill in clinicaldentistry.

Methods

In a series of meetings, 3 members of clinical staff as-sembled a number of criteria that could be used toassess a clinical skill in a SCOT as well as guidelinesto make the assessment. The core skill chosen to makethe assessment on was the recording of an impressionfrom a dentate subject. The same members of staffacted as the assessors of the SCOTs. To validate them-selves the three assessors observed a number of stu-dents record a dental impression whilst making refer-ence to the agreed criteria. The assessors comparedtheir responses and following discussion the criteria

TABLE 1. The 12 criteria used to assess performance in the recordingof an impression on a dentate subject; the code for each criterion isdisplayed on the left side of the table

AS1 explanation to patient of procedureAS2 patient in correct position and with bib or towelAS3 operator maintained appropriate dialogue with patient

throughout procedureAS4 operator only touched designated work surfacesAS5 procedure for washing and disinfection of impressions

understoodAS6 correct lower tray chosenAS7 impression tray properly prepared for impressionAS8 lower tray optimally loadedAS9 functional movements carried out by patient or assisted by

operatorAS10 reasonable comment made on quality of impressionAS11 impression shows appropriate extension and detailAS12 appropriate prescription to laboratory made

32

were refined (Table 1) along with the guidelines forthe definitive study.

The definitive study was carried out on a total of 39undergraduate dental students who were in the earlystages of their clinical course. At this stage, the stu-dents had completed a phantom head based operativetechniques course and were being introduced to anumber of the clinical disciplines. The students hadreceived a course of instruction and gained clinical ex-perience in the recording of dental impressions. For-mal instruction was composed of a lecture of 45 minduration which covered the clinical aspects of im-pression recording and its relevance to clinical den-tistry. This was followed by clinical demonstrations togroups of approximately ten students on the record-ing of upper and lower impressions. All students thenrecorded a set of upper and lower impressions of eachother, one acting as the operator and one as the pa-tient. The upper and lower impressions were recordedover 2 sessions of approximately 3 h duration.Throughout this period they were closely supervisedby two members of clinical staff and they receivedformative feedback. Feedback included how to com-municate with their patient, the positioning and man-agement of the patient throughout the procedure, thepractice of appropriate cross infection control pro-cedures, making judgements on the quality of the im-pression and the appropriate procedures to be under-taken before the impression went to the laboratory tobe cast. At the same stage of the course, the studentsalso received formal instruction from other clinicalareas in communication skills and cross infection con-trol procedures.

The definitive study was undertaken three weekslater. During this time the students had constructedcustom impression trays (special trays) on the modelscast into the original impressions. They were allowedto take an upper impression immediately before theycarried out the SCOT but no formative feedback wasgiven at this stage. The students worked in the samepairs as they did originally. They were instructed tochoose an appropriately sized stock impression trayand record a lower impression of their dentate partner(patient). Once the impression was produced the stu-dent was questioned by an assessor. 2 assessors ob-served the whole procedure and independentlyscored the students on the twelve agreed criteria. Eachassessor judged whether the operator had reached anacceptable standard for all criteria. Immediately afterthe SCOT the assessors conferred and for each cri-terion (Table 1) an agreed assessment was given toenable the students to be subsequently informed as towhether they had successfully completed the SCOT.

Page 3: An investigation into the use of a structured clinical operative test for the assessment of a clinical skill

Structured clinical operative test

AS1 explanation to patient of procedure.AS2 patient in correct position and with bib or towel.AS3 operator maintained appropriate dialogue with patient

throughout procedure.AS4 operator only touched designated work surfaces.AS5 procedure for washing and disinfection of impressions

understood.

Fig. 1. Bar chart to show the % agreement between the 3 assessor pairs(A & B, A & C, B & C) for the patient management and cross infectioncontrol criteria studied in the SCOT. The codes on the horizontal axisrefer to the criteria shown in the table above.

The 3 assessors worked in pairs. 2 of the assessors(A and B) carried out a majority of their teaching du-ties in Prosthetic Dentistry and the 3rd (C) in Paediat-ric Dentistry. All 3 assessors were experienced clini-cians. Agreement between the assessors on whetherthe criteria reached acceptable standards was studiedusing % agreement, the c2 test and k analysis.

Results

12 criteria were used for assessment of performancefor the impression procedure. These are shown inTable 1. Assessor Pairs A & B worked with 11 stu-dents, A & C with 14 students, and B & C with 14students. A total of 121 criteria were assessed by PairA & B, and 168 criteria by Pairs A & C and B & C.

The agreement between the three groups of as-sessors for the patient management (AS1, AS2, AS3)and cross infection control (AS4, AS5) criteria areshown in Fig. 1. The levels of agreement reflectwhether there was consensus between the assessorson the standard the operator attained. For examplein relation to criteria AS5, there was 100% agreementbetween the assessors A & C. However, not all oper-ators reached the standard, and there was total agree-ment between the assessors as to which operators didachieve the standard (positive assessment) and whichdid not (negative assessment). Similar observations

33

were made in relation to some of the other criteria.There was some variation in the amount of agreementbetween assessors for different criteria. There werevery high levels of agreement between all three pairsof assessors for the communication criteria (AS1 andAS3). There were more variable levels of agreementfor the cross infection control criteria (AS4, AS5), butthe levels of agreement were still high. In over half ofthe assessments made by the assessor pairs there was100% inter-assessor agreement.

The % agreement between the 3 groups of assessorsfor the impression criteria are shown in Fig. 2. For 2of the 7 criteria, there was 100% agreement betweenall three pairs of assessors (AS7, AS12). In almost halfof the assessments made by the assessor pairs, therewas 100% agreement between them. There was morevariation in the % agreement for the impression cri-teria than the management and cross infection criteria.However, in only one case was the level of agreementlower than 70% (Assessors B & C for criterion AS11).

The overall agreement between assessors for theSCOT was calculated by combining the criteria shownin Figs. 1, 2. The levels of agreement are shown inTable 2. There were high levels of % agreement be-tween all 3 groups of assessors with similar judge-ments being made on 90% or more of the twelve cri-teria assessed. The kappa values ranged between 0.46to 0.64. Statistical analysis of all of the criteria studiedshowed no significant differences between agreement

AS6 correct lower tray chosen.AS7 impression tray properly prepared for impression.AS8 lower tray optimally loaded.AS9 functional movements carried out by patient or assisted

by operator.AS10 reasonable comment made on quality of impression.AS11 impression shows appropriate extension and detail.AS12 appropriate prescription to laboratory made.

Fig. 2. Bar chart to show the % agreement between the 3 assessor pairs(A & B, A & C, B & C) for the 7 impression criteria studied in theSCOT. The codes on the horizontal axis refer to the criteria shown inthe table above.

Page 4: An investigation into the use of a structured clinical operative test for the assessment of a clinical skill

Scott et al.

TABLE 2. The % agreement between the assessor pairs (A & B, A &C, B & C) for the 12 criteria studied in the SCOT

Assessor pairs A & B A & C B & C

% agreement 95 94 90k value 0.64 0.58 0.46

TABLE 3. The % of positive assessments for the 12 criteria studiedgiven by each assessor overall and when working with each of the other2 assessors

Assessor

A B C

% positive assessments overall 92 92 90% positive assessments working with A ª 93 91% positive assessments working with B 91 ª 89% Positive assessments working with C 93 90 ª

and disagreement for the three pairs of assessors (c2

test, p.0.05).An analysis of the % of criteria that were positively

scored by each assessor is shown in Table 3. The as-sessors independently gave similar percentages ofpositive assessments in which they judged that theoperator had reached an acceptable standard for thecriteria studied. Table 3 also shows a comparison ofthe positive assessments made by each assessor whenworking with each of the other two assessors. The dif-ferences observed between the percentages of positiveassessments given were generally quite small.

By adding the positive scores for each criterion, adecision was made as to whether the student had car-ried out a satisfactory SCOT. It was agreed betweenthe 3 assessors, that an acceptable standard would beif the operator received a positive assessment on 10 ormore of the criteria. Data on the outcome of the SCOTas judged by the three pairs of assessors are shown inFig. 3. There were very high levels of agreement be-tween Assessors pairs A & B, and A & C, in that over80% of the students should pass. Assessors A & Cwere the only pair that agreed a small proportion ofstudents should fail the assessment. There wasslightly less agreement between Assessor pair B & Cas to the percentage of students who had carried outa satisfactory SCOT. The final agreement on the pro-portion of students that passed the SCOT was basedon the assessors conferring on the criteria they did notagree on and reaching an agreed response. The finalproportion of students who passed and failed theSCOT are also shown in Fig. 3. For Assessor pairs A &B and A & C, the final percentage of students whopassed was similar to that reached by the assessors

34

Fig. 3. Bar charts to show the % of students passed and failed by the3 pairs of assessors for the whole assessment. Top bar chart – assessorpair A & B; middle bar chart – assessor pair A & C; bottom bar chart –assessor pair B & C. For each bar chart, the % of students passed andfailed are shown for both assessors that formed the pair, and for eachassessor only. The agreed bars show the % passed and failed afterconsultation between the pairs of assessors.

independently. For Assessor pair B & C, a greater %of students than assessed independently passed afterdiscussion between them.

Discussion

This study has shown satisfactory levels of agreementbetween groups of assessors observing studentsundertaking a SCOT. When carrying out an assess-ment of performance for a clinical skill it is criticalthat it is carried out as consistently as possible. How-ever, it would not be possible for a large number ofoperators to carry out an identical clinical task on thesame patient. In the present study the group of stu-dents, all of whom acted as patients, were fully den-tate, in good health and were of a similar age range.This gave a form of standardisation that would be dif-ficult to achieve if using a group of patients with vary-

Page 5: An investigation into the use of a structured clinical operative test for the assessment of a clinical skill

Structured clinical operative test

ing patterns of tooth loss, medical conditions thatmight affect management and a wide age range. TheSCOT should be distinguished from ‘‘authentic evalu-ation’’ of a clinical skill in which a judgement can bemade of an operator’s performance in a realistic set-ting (1, 6). The SCOT may best be regarded as a simu-lation in which performance criteria that representsome of the features relevant to authentic perform-ance are assessed. This is particularly appropriate tostudents at the beginning of educational programmes(1). In contrast, authentic evaluation is the assessmentof performance in settings resembling those under-taken after completion of educational courses; port-folio evaluation and test cases would be more suitedto this sort of assessment (1).

For individual criteria in the SCOT, there were gen-erally good levels of agreement between the threepairs of assessors and in many cases agreement was100%. Disagreements between assessors when makingjudgements on criteria are inevitable as clinical per-formance is not easy to standardise. Some criteria aremore easily assessed than others. For certain criteria,100% agreement between pairs of assessors was ob-tained. Furthermore for one quarter of the criteria,there was 100% agreement between all three groupsof assessors. Careful attention to the wording of suchcriteria may have contributed to the high level ofagreement observed. Where there was disagreement,this could often be explained by differences in whatthe assessors may have been expecting. For example,different assessors may have had varying perceptionsas to what they expected when observing functionalmovements being carried out during impression tak-ing (Criterion AS9). The judgement as to whether animpression showed appropriate extension and detailis particularly challenging (Criterion AS11). Despitethis criterion having the most detail in it for the as-sessors to refer to when making their judgement,there were more disagreements for this criterion thanany other. Nevertheless one pair of assessors hadmore than 90% agreement and even the lowest levelwas over 60%.

The % agreement between the three pairs of as-sessors, for all of the criterion that made up the SCOT,were high. k-values indicated moderate to good levelsof agreement between assessors. In no case was therepoor or random agreement between assessor pairs.The reason why the k values indicate only moderateto good levels of agreement, despite agreement levelsbetween assessors of 90% or above is explained by thefact that the student performances were generallyvery good, being around the 90% level (Table 3). It isknown that k values will reflect the proportion of sub-

35

jects (prevalence) in each category (13). The consist-ently high level of performance of the students in thisstudy resulted in more conservative k values.

The k values indicated that the agreement betweenassessor pair B & C was less than the other twogroups of assessors, although this is still consideredto be well in the range of moderate agreement. Thereason for this is not clear. Assessors A & B workedclosely together in the same clinical area whereas as-sessor C worked in another clinical area. However,there was no obvious reason why assessor C hadmore agreement with assessor A than with assessorB. The c2 test indicated that there were no statisticaldifferences between agreement and disagreement forthe 3 groups of assessors. Assuming a level of 95%agreement between the assessors as ideal, and thatthere would be concern if the level of percentageagreement dropped below 85%, the c2 test had apower level of 92%. This means that any differencesbetween assessor pairs would have been very likelyto be detected statistically if they were there. The lackof significant differences might be explained by theamount of preparation that the assessors put in byworking out together how judgements could be madeon the criteria for acceptable performance. It is there-fore important that in any assessment of a clinicalskill, those involved in making the judgment are ap-plying the same standard to assess the criteria makingup the SCOT. This can only be done by close collabor-ation between assessors in advance of the assessment,and having unambiguous criteria.

The assessors generally gave a similar number ofpositive assessments. This meant that the studentswere reaching acceptable levels of achievement in theSCOT. At the time the SCOT was undertaken the stu-dents had received a thorough course of instructionand should have gained sufficient experience. Thiswas reflected in the percentage of positive assess-ments given. If there had been any major differencesin the percentage of positive assessments given by oneassessor when working with either of the remainingassessors this would have been likely to reflect theability levels of different groups of students. In theplanning of the SCOT, it was decided that if the stu-dent achieved positive assessments on 10 or more cri-teria, they had reached a satisfactory level of perform-ance. If they did not reach this level, they received afurther course of instruction and repeated the SCOT ashort time later. In this context there were very similarlevels of agreement between assessors pairs A & Band A & C. For assessors B & C, there was a lowerlevel of agreement but this reflects the fact that thispair of assessors generally had lower levels of agree-

Page 6: An investigation into the use of a structured clinical operative test for the assessment of a clinical skill

Scott et al.

ment for the criteria than the other two pairs. Thiscould be addressed by refining the criteria further togive more objectivity.

The decision that a satisfactory performance hadbeen reached on the basis of positive assessments onten of the twelve criteria, was, in the judgement of theassessors, an acceptable level for a student at this veryearly stage of their clinical training. They would sub-sequently be receiving further clinical instruction andsupervision as they progressed in their course. Whilethese types of judgement are undoubtedly subjective,they were standard for all of the students. Such judge-ments must be made on the basis of planning by theassessors who understand how such evaluations fitwith the objectives of an approved curriculum. In con-trast, if such a SCOT was undertaken at the point ofgraduation when extensive training had been under-taken, the criteria would be the same, but it would beexpected that all criteria would be positive for a stu-dent to be judged to have reached an acceptable levelof performance.

One of the limitations of observing a live SCOT isthat because the procedure is carried out over a shortperiod of time, the assessors have to make rapidjudgements on whether the criteria were met. Thismay have accounted for some of the differences in theagreement between different pairs of examiners. Oneway to approach this might be to video the clinicalskill and the assessors could view this at a later stage.This would allow the assessors to take more time inmaking their judgement. However, video techniqueswould remove the interactive nature of the SCOT andopportunities to give immediate formative feedbackwould be lost. The use of video techniques in SCOTsmay however be a way of assessing clinical treatment,as well as some attitudinal and communication skills,in a summative manner, but this area requires furtherstudy.

A criticism of running even a simple SCOT is thatit is very consuming of time when resources may belimited. However, providing it can be shown that as-sessors are achieving satisfactory levels of agreement,there is a case for having the performance judgementin the SCOT made by one assessor, with different as-sessors looking at different students. There would bea need to ensure that the assessors are well trainedand understand exactly what they are being asked tomake judgements on. This would require furtherstudy. There are similar difficulties in relation to timeand practicality of using this method in the assess-ment of skills in a wide range of clinical disciplines.For this reason it is considered that in the undergrad-uate setting, a SCOT would be best used to assess core

36

skills that would be used across a number of clinicaldisciplines rather than specific to a single one. Clinicalskills such as history taking, the recording of im-pressions and the administration of local anaesthesiacould therefore be appropriately assessed by a SCOT.

Disagreements between assessors might present dif-ficulties in making judgements if the only purpose ofthe SCOT were to see if a student could simply pro-duce a technically good impression with appropriateextension and detail. Whilst the production of an ade-quate impression is clearly what a clinician needs tobe able to achieve for successful practice, training isdirected to acquiring a range of skills as well as beingable to achieve a level of technical excellence. In re-lation to predicting clinical success, it has been sug-gested that the measurement of ability on the finisheddental procedure, both in the clinic and in preclinicaltechnique courses (14, 15), is somewhat imprecise,and that general faculty ratings are just as valid pre-dictors of performance as grading systems. An advan-tage of the SCOT approach is that it is not just thetechnical quality of the end product, but also the pro-cess by which this is achieved, that is being assessed.In addition to assessing the technical procedure, theSCOT embraces management and communicationwith a patient, preparation for a procedure, carryingout a procedure safely, making a critical judgement asto whether the procedure has gone well and planningfor the next stage of clinical management. Assessingonly the quality of the end result itself (i.e., the im-pression) imposes limitations with respect to trainingfor clinical practice. For this reason, it is more appro-priate to carry out a SCOT from looking at the pro-cess, by assessing all criteria that contribute to a clin-ical skill, and it was not felt appropriate to give moreweight to some criteria than others.

To assess performance in a SCOT, there should be agood level of agreement between assessors. Despitecareful collaboration between assessors to define suit-able criteria for making judgements, it is acknowl-edged that any assessment of performance in a clin-ical skill cannot be totally objective. However, in thecontext of professional education, it is desirable thatthere is high inter–subjective agreement between as-sessors making professional judgements. The levels ofagreement found between the assessors in the presentstudy show this and support the use of a SCOT toassess performance of a clinical skill.

The study was carried out on a procedure in whichit was possible to obtain reasonable agreement in theprofessional judgements made by the assessors. Manyskills in clinical dentistry present greater challengesin their assessment than the skill investigated in the

Page 7: An investigation into the use of a structured clinical operative test for the assessment of a clinical skill

Structured clinical operative test

present study. An advantage of using a group of as-sessors in the planning of performance criteria for theSCOT is that it may allow cross fertilization of ideasand concepts across clinical boundaries. The presentstudy showed acceptable levels of agreement betweenassessors, thus supporting the use of a SCOT as avaluable tool in measuring performance in some areasof clinical dentistry. Even in the SCOT carried out inthe present study, further improvements can be madeto the criteria to make it more objective. In any SCOT,especially where more complex clinical skills arebeing assessed, great care will needed to assemble cri-teria that can be reproducibly used and sufficientpreparation time for the assessors is essential.

References

1. Chambers DW, Glassman P. A primer on competency-basedevaluation. J Dent Ed 1997: 61: 651–666.

2. Mossey P. Competency based teaching and assessment indentistry – the context. In: Mossey P, Stirrups D (eds): Clin-ical competencies in dentistry. London: The Medical andDental Education Network, 1998: 7–10.

3. Mossey PA. Core competences in dentistry – exploring theissues. In: Core Competences in Dentistry, UK. Educationfor Health 1998: 11: 99–113.

4. Nimmo A, Knight GW. Patient-centred competency-basededucation in fixed prosthodontics. J Prosthod 1996: 5: 122–128.

5. Nimmo A, Woolsey GD, Arbree NS, Saporito RA, Cooney JP.Defining predoctoral prosthodontic curriculum: a Workshop

37

sponsored by the American College of Prosthodontists andthe Prosthodontic Forum. J Prosthod 1998: 7: 30–34.

6. Chambers DW. Competency based dental education in con-text. Eur J Dent Ed 1998: 2: 8–13.

7. Chambers DW. Competencies: a new view of becoming adentist. J Dent Ed 1994: 58: 342–345.

8. Batchelor P, Albert D. Issues concerning the development ofa competency-based assessment system for dentistry. BrDent J 1998: 185: 141–144.

9. Mossey PA, Newton JP, Stirrups DR. Defining, conferringand assessing the skills of the dentist. Br Dent J 1997: 182:123–125.

10. Davenport ES, Davis JEC, Cushing AM, Holgrove GJ. Aninnovation in the assessment of future dentists. Br Dent J1998: 184: 192–195.

11. Chambers DW, Geissberger M. Toward a competency analy-sis of operative dentistry technique skills. J Dent Ed 1997:10: 795–803.

12. Stacey MA, Morgan MV, Wright C. The effect of clinical tar-gets on productivity and perceptions of clinical competency.J Dent Ed 1998: 62: 409–414.

13. Altman DG. Practical Statistics for medical research. Lon-don, UK: Chapman and Hall, 1991: 403–409.

14. Chambers DW, Leknius C, Woodson R. Predictive validityof instructor judgement in preclinical technique courses. JDent Ed 1997: 61: 736–740.

15. Chambers DW. Faculty Ratings as part of a competency-based evaluation clinic grading system. Eval Health Prof1999: 22: 86–106.

Address:B. J. J. ScottUnit of Clinical Dental SciencesDundee Dental Hospital and SchoolPark Place, Dundee DD1 4HRUK