xerte standard setting undergraduate osce
TRANSCRIPT
XerteXerteStandard SettingStandard Setting
ininPsychiatry Undergraduate Psychiatry Undergraduate
OSCEOSCEDr Ben di MambroDr Ben di Mambro
Clinical Lecturer in PsychiatryClinical Lecturer in Psychiatry
Psychiatry ExaminationPsychiatry Examination 80 EMIs80 EMIs
1 hour 15 minutes1 hour 15 minutes Standard setStandard set
2 OSCE stations2 OSCE stations Each station 10 minutesEach station 10 minutes 3 tasks based on competencies3 tasks based on competencies % given for each task% given for each task
May OSCEMay OSCEStationStation MaxMax MinMin AllAll
AA 60.160.1(14.3)(14.3)
46.446.4(5.6)(5.6)
54.554.5(10.3)(10.3)
BB 60.8 60.8 (6.6)(6.6)
48.248.2(4.4)(4.4)
55.555.5(9.0)(9.0)
CC 66.166.1(8.5)(8.5)
49.049.0(9.7)(9.7)
56.256.2(9.8)(9.8)
DD 67.767.7(12.8)(12.8)
47.247.2(4.1)(4.1)
54.354.3(10.8)(10.8)
Psychiatry OSCEPsychiatry OSCE Changes to the OSCEChanges to the OSCE
Criterion marking schemeCriterion marking scheme Global clinical assessmentGlobal clinical assessment
HonoursHonours PassPass BorderlineBorderline FailFail
2 examiners were placed in each room2 examiners were placed in each room Standard set pass mark for each taskStandard set pass mark for each task
TaskTaskDevelop CAL package:Develop CAL package: Train the examiners in the new Train the examiners in the new
marking schememarking scheme Standardise marking schemeStandardise marking scheme
To allow for multiple sites to To allow for multiple sites to accommodate increasing numbers of accommodate increasing numbers of studentsstudents
Improve inter-rater reliabilityImprove inter-rater reliability
Development of PackageDevelopment of Package Scripted 10x 2-3 minute OSCE Scripted 10x 2-3 minute OSCE
scenariosscenarios Borderline/PassBorderline/Pass Pass/HonoursPass/Honours
2 hours filming in my office2 hours filming in my office 2 lecturers2 lecturers 1 bullied student1 bullied student
Creation of MediaCreation of Media wmv files converted to flv using Flash wmv files converted to flv using Flash
MacromediaMacromedia Images created utilising Flash Images created utilising Flash
MacromediaMacromedia MP3 creation using AudacityMP3 creation using Audacity
XerteXerte Template ‘ratings’ created by Julian Template ‘ratings’ created by Julian
TenneyTenney Media was imported into Xerte Media was imported into Xerte
through the Wizardthrough the Wizard
The PackageThe Package An introduction to the new marking An introduction to the new marking
schemescheme Example mark sheetsExample mark sheets Video of example studentsVideo of example students
Rated on criteriaRated on criteria Global impression givenGlobal impression given User then offered feedbackUser then offered feedback
Using the WizardUsing the Wizard Once Wizard create – efficient Once Wizard create – efficient
method of presenting contentmethod of presenting content Define scenarioDefine scenario Define marking sheet, including number Define marking sheet, including number
of criteriaof criteria Define fail/borderline/pass/honoursDefine fail/borderline/pass/honours Import mediaImport media
Sample of PackageSample of Package
Distribution of PackageDistribution of Package Letters and emails were sent to all Letters and emails were sent to all
examining consultants with following examining consultants with following linklink
www.nottingham.ac.uk/~mczbjd/OSCEwww.nottingham.ac.uk/~mczbjd/OSCE.htm.htm
Package ran for ~20 minutesPackage ran for ~20 minutes
Result of Package Result of Package IntroductionIntroduction
30% examiners accessed the 30% examiners accessed the package prior to the OSCEpackage prior to the OSCE
Remainder underwent abridged Remainder underwent abridged version before the OSCEversion before the OSCE
AnalysisAnalysis ANOVAs undertaken on awarded ANOVAs undertaken on awarded
scores by the different examiners on scores by the different examiners on the same stationthe same station
December OSCEDecember OSCEStationStation MaxMax MinMin AllAll
AA 70.370.3(5.0)(5.0)
53.453.4(8.6)(8.6)
62.362.3(8.6)(8.6)
BB 72.672.6(9.1)(9.1)
52.152.1(8.85)(8.85)
63.763.7(11.5)(11.5)
CC 66.166.1(5.1)(5.1)
51.051.0(7.6)(7.6)
56.956.9(8.6)(8.6)
DD 66.166.1(56.7)(56.7)
56.656.6(5.0)(5.0)
60.760.7(7.4)(7.4)
**p<0.001 *p<0.05**p<0.001 *p<0.05
MayMay DecemberDecember
OSCE 1OSCE 1 54.5**54.5**(10.3)(10.3)
62.3**62.3**(8.6)(8.6)
OSCE 2OSCE 2 56.2**56.2**(9.77)(9.77)
63.7**63.7**(11.5)(11.5)
OSCE 3OSCE 3 55.5**55.5**(9.02)(9.02)
56.9**56.9**(8.6)(8.6)
OSCE 4OSCE 4 54.3**54.3**(10.8)(10.8)
60.760.7(7.4)(7.4)
VariationVariationStationStation MayMay DecDec
AA 13.713.7 16.916.9
BB 12.612.6 20.520.5
CC 17.117.1 15.015.0
DD 20.520.5 10.510.5
LimitationsLimitations Flash Player 7 not installed as standard Flash Player 7 not installed as standard
on healthcare trusts’ computerson healthcare trusts’ computers Minor problems with Xerte - since Minor problems with Xerte - since
resolvedresolved Delay on slow broadband connectionsDelay on slow broadband connections ?Representative videos?Representative videos Any improvement likely to be due to the Any improvement likely to be due to the
new marking schemenew marking scheme
XerteXerteOnce familiar:Once familiar: Easy to useEasy to use Multimedia packages can be create Multimedia packages can be create
in minutesin minutes Easily distributedEasily distributed Much more complicated to create Much more complicated to create
bespoke packagesbespoke packages
FutureFuture Improve IT access in the healthcare Improve IT access in the healthcare
truststrusts Reduce bitrateReduce bitrate Produce improved examples of Produce improved examples of
borderline studentsborderline students Extend the package to attempt to Extend the package to attempt to
standardise the Formative assessmentsstandardise the Formative assessments ?Make available to students?Make available to students