development of viva voce exam to assess clinical...

Development of viva voce exam to assess clinical reasoning in the paraclinical phase of a BVSc program. HelenOwen,SeniorLecturerinVeterinaryPathologyandJus:neGibson,Senior

LecturerinVeterinaryBacteriologyandMycology

DrTamsinBarnesAssociateProfessorRowlandCobboldDrKatrinaGarreGDrStevenKopp

AssociateProfessorJoanneMeersDrDanSchullAssociateProfessorJennySeddonDrFrancesShapterAssociateProfessorRebeccaTraub

Background on Program •  TheBVScatUQisanundergraduate5yearprogram•  First3yearsconsistpredominantlyofpreclinicalscienceswiththefourthandfi?hyearshavingmoreofaclinicalfocus.• Majorcoursesinthirdyear-ayear-longcourseonpathophysiology–stereotypicalwayswhichFssuesrespondtoinjury,prototypediseases-clinicalsigns,clinicalexamanddiagnosFcapproachtobodysystem-courseoninfecFousdisease(parasitology,virology,bacteriologyandmycology).

Development of viva voce exam •  In2012,weweretaskedwithdevelopingavivavoceoralexamasaprogramlevel“hurdle”assessmentattheendofthirdyear.

ObjecFves,toassess:•  applicaFonoftheorytocasescenarios•  diagnosFcclinicalreasoning•  integraFonofmaterialfromboththepathophysiologyandinfecFousdiseasescourses•  professionalismandoralcommunicaFonskills

•  Exposuretooralexamformatbefore5thyearvivaexams

Advantages of oral format

• Goodforassessingproblem-solvingabiliFes.• Canprovideinsightintostudents’cogniFveprocesses.•  Itallowsprobingofthedepthandextentofstudents’knowledgewithfollow-upquesFons• UnclearorambiguousquesFonscanbere-expressedorimmediatelyclarified•  Studentscanbeguidedbackontotherighttrackiftheystray

Advantages of oral format • Allowsjudgementsaboutstudents’interpersonalcompetence-confidence,self-awarenessandaspectsofprofessionalism.•  SimulaFonofclinicalpracFce–studentsneedto“thinkontheirfeet”•  Itimprovesthequalityofstudentlearning–studentshavebeenreportedtostriveforthoroughnessinunderstandinginresponsetotheunpredictabilityandtoavoidembarrassingthemselvesinfrontoftheexaminer

•  Itsuitssomestudents

Disadvantages of oral format • Generatesalotofanxietyinstudentsandexaminers•  Timeconsumingandrequiresalotofenergy•  LackofanonymityandpotenFalforbias–unconsciousbiasbasedoncharismaFcpersonality,welldressed,beingarFculate• Reliability-Inter-case-differentstudentsgetdifferentcasestopreventsharingofinformaFon.

-Inter-rater-Intra-raterreliability

Features of the initial exam •  FormulaicseriesofquesFonsbasedonthediagnosFcprocess-ProvidedwithimageandsomeinformaFon–historyandclinicalexamfindings-  bodysystem(s)-  possiblepathologicalprocesses(highweighFng)–usingpathophysiologyforclinicalreasoning

-  differenFaldiagnoses(lowweighFng!)-  diagnosFcplan(highweighFng)-  Interpretresults,treatment,control,prognosis

• professionalism

Initial exam •  Allstudentsreceivedonecase•  StudentswhofailediniFalexamweregivenaresit•  10minutesperusal,20minutesoral•  2examinersperstudent-increasereliability–consensusmarksused

•  Rubricandmodelanswers-increasereliability

Changes over the years based on examiner and student feedback

•  2cases(usuallyonelargeanimal,onesmall)–increasesreliability• Moreexaminertraining• RemovedhistoryandclinicalexamsecFons•  2-stagepathophysiology• Adjustedrubric-moreemphasisonsystemaFcapproachforpathogenesisanddiagnosFctesFng,forcedtoprioriFseDDxs• Year-longpreparaFonandprovisionofformaFvefeedback.

Youarepresentedwitha1yearolden3remaleDomes3cShortHaircat.Theownerno3cedthatthecatisoffitsfoodand‘quieter’thanusual,thishasbeenthecaseforaboutaweek.

Exam format – written section (30 minutes)

PartA.Descrip:onofcase,presen:ngsigns(images/videos)

Exam format – written section • PartB.OutlinethepathologicalprocesseswhichmaybecausingthepresentaFon•  StudentssFllrotelearningpathogenesisforparFcularclinicalpresentaFonwithoutemphasising/prioriFsingbasedoncaseparFculars• OrstudentsusingpacernrecogniFonandbeingabletodescribeonespecificpathogenesisbutnotbeingabletosuggestalternaFves• Breakdownintotwosteps,inthefirststep,theyneedtodemonstrateasystemaFcknowledgestructurethatincludesallpossiblepathogeneses•  Secondstep,providedwithaddiFonalinformaFonaboutthecaseandneedtousethistoprioriFseDDxs

•  Encouragestudentstoconsiderallpossiblepathogenesesforabroadclinicalsignatthispoint• Mustdemonstratestructure,systemaFcapproach,encouragedtouseschemes

Scheme-inductive knowledge structures

Harasymetal.2008KaohsiungJMedSci,24:7;341-355.

ResearchsuggeststhatexpertproblemsolversaredisFnguishedbythewayintheyorganiseandunderstandtheirknowledge

Scheme-inductive knowledge structures • Mandinetal,UniversityofCalgaryFacultyofMedicine–“ClinicalPresentaFon”curriculum

• “The“hypotheFco-deducFve”strategytradiFonallyusedforPBLshouldbereplacedbyscheme-drivensearchstrategiessothatstudentsdevelopamoreorganisedandlogicalapproachtoproblem-solving”

SchemeshavemulFplepurposes:• ProvidesystemaFcapproachtodecreasetheriskofomidngDDxs

•  wayoforganisingknowledgeformoreefficientunderstanding,serveasascaffoldfornewinformaFon

Thehistory,physicalexaminaFonandlaboratoryinvesFgaFonarealldrivenbythederivedscheme

Paraclinical schemes • Probablydifferentcontenttoclinicalschemes,morerelianceonprinciplesofpathophysiology,lesssignalmentetc?

Nodescanconsistof:•  locaFone.g.bodysystems-cardiovascularversusrespiratorytract;withinbodysystems-upperversuslowerrespiratorytract;•  aformofgeneralpathologicalprocessintheFssuee.g.myocardiFs,hepaFcnecrosis,pulmonaryoedema•  SpecificcausaFveagents(prototypic)•  SomepresentaFonsabitdifferente.g.anaemia,jaundice•  Lotsofschemesprovidedtothestudentsandtheyareencouragedtoimproveonthem

Mnemonics as a system? •  e.g.DAMNITV-DegeneraFve,Anomalous,Metabolicetc•  Studentso?enpreferthisapproach•  tryingtonotbetooprescripFve,flexibilityallowingfordifferentthoughtprocesses,clinicalpresentaFon

Don’tworktoowellinourhands….•  Doesn’trequireexplicitidenFficaFonofbodysystems•  Studentso?endon’tseemtounderstandpathogenesis•  Doesn’tprovidestructureforsystemaFcdiagnosFcapproach• Moreappropriateasamemoryjoggerinclinicalyearsa?erstudentshavedemonstratedpathophysiologyknowledge?•  UsefulaspartofamulFplestrategyapproach?

YouwillbeprovidedwithaddiFonalhistoryorclinicalfeatures•  YouperformaPCVandthereismarkedanaemia.

•  IniFallyweassumethatgoodstudentswouldprioriFseDDxsbasedonallthefeaturesofthecasebutthisrarelyhappened

PartC.Giveaproblemdefini:on•  Subacuteinappetence,depression,jaundiceandanaemia.

Exam format – written section

Exam format – oral section PartD.Bodysystem(s)?• haematopoieFc

• PartE.UsetheaddiFonalinformaFontoidenFfymostlikelypathologicalprocess(es),describepathogenesisandprovideaeFologicalexamplesifappropriate

PartF.List5differenFaldiagnoses

PartG.DiagnosFcplan•  IniFallyfoundthatstudentswouldjusttestandFckoffeachoftheirdifferenFals•  EncouragesystemaFcapproach,useofschemetoguide,mostinformaFonfromfewesttests•  EncourageuseofdiagnosFcstoguidetreatment•  StudentsdorelaFvelypoorlyinthissecFon

Exam format – oral section

Exam format – oral section

Providediagnosis,prognosis,treatmentplan,controlThissecFonalsopoorlydoneingeneral

Exam format – oral section

• PartI–professionalism–useofprofessionallanguage,professionalbehaviouranddress.

Outcome of viva exams •  Thisisamust-passhurdle,alargernumberofstudentsfailthiscomparedtofailingonbasisoftheoryexams(~5-9/120versus~2-3)•  O?enabout10studentsonpass/failline,reluctanttoletthemgointo4thyear–whatshouldourexpectaFonsbeforthesenovicediagnosFcians?• Moststudentswhofailvivaareweakintheoryalsobutnotalways

Preparation for viva exams •  6face-to-facetutorialsdispersedthroughtheyear•  5thyearstudentsactastutorsforsmallgroups•  3rdyearsexaminingeachotherinpairs•  Includeallstepsinprocess,evenatthebeginning•  3on-linecase-basedtutorialswheregenericfeedbackgivenbutstudentscanelecttohavepersonalfeedback•  “diagnosFcwork-shop”•  Viva“hint-sheet”

Lessons we have learnt • WetookalotforegrantediniFally:“goodstudentsshouldautomaFcallydemonstrateasystemaFc/structuredapproach,prioriFseDDxsbasedonparFcularsofacase…”•  NeedtobecompletelyexplicitinexpectaFonsandspecificforeachstepinprocesswhichmeanshavingagoodunderstandingoftheminyourownmind•  NeedtostructurerubricsothatitisnotpossibleforastudenttogetgoodmarkswithoutasystemaFcapproachandbeingabletoprioriFse

•  NeedtostructurepreparaFonmaterialtoencouragegoodhabits•  EncouragepracFceofverbalcommunicaFonasthisisaverydifferentskilltowricen

Conclusions Oralformatbenefits:•  Beingabletoguidestudentsontotherighttrack,test“thinkingonfeet”,probeunderstanding•  FearisapowerfulmoFvatorforembracingnewlearningapproaches

OralformatlimitaFons:•  VariaFonincases–lackofstandardisaFon?•  Inter-raterreliability• MassiveamountofFmeandenergy,difficulttoengageexaminers•  InternaFonalstudentsdisadvantagedbylanguage,culturallyingrainedbehaviour? •  ReallygoodformatforassessingdiagnosFcclinicalreasoning

(oralcommunicaFonskills,professionalism)butdoesgoodoutweighthebad?

Future directions •  Reliability–variaFoninmarksacrosscases,examiners,secFonsofexam•  InvesFgatewhetherpreparaFonandtakingtheseexamshasalasFnginfluenceonstudents’studyhabitsandapproachtocases•  DevelopingmoreefficientmeansofprovidingformaFveassessment•  BecerintegraFonwiththeclinicalyearsoftheprogram–evolvingnatureofclinicalreasoningstrategies,rollingoutofseriesofverFcallyintegratedon-linetutorials•  ObtainbecerunderstandingofapproachesofweakstudentssoastobecerassisttheirpreparaFon

References •  Davis,M.,Karunathilake,I.2005.TheplaceoforalexaminaFonsintoday’sassessmentsystems.MedicalTeacher,27;4:294-297.

•  Harasymetal.2008CurrentTrendsinDevelopingMedicalStudents'CriFcalThinkingAbiliFes.KaohsiungJMedSci,24:7;341-355.

•  Joughin,2010.Ashortguidetooralassessment.LeedsMetropolitanPressinAssociaFonwithUniversityofWollongong.

•  Mandin,H.,Harasym,P.,Eagle,C.andWatanabe,M.1995.Developinga“clinicalpresentaFon”curriculumattheUniversityofCalgary.AcademicMedicine.70;3:186-193.

•  Mandin,H.,Jones,A.,Woloschuk,W.andHarasym,P.1997.Helpingstudentslearntothinklikeexpertswhensolvingclinicalproblems.AcademicMedicine,72;3:173-179.