Upload File

0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

23
!"#$%& ()*)" +#%&,)-()+. /%+0)" 12345 67)+$8%+9 ():8%; .<) =8>)" .)+. Phil Smith Consultant Neurologist and Sub-Dean for Assessments Cardiff

Upload: others

Post on 25-Jan-2022

6 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

!"#$%&'()*)"'+#%&,)-()+.'

/%+0)"'12345'67)+$8%+9'

():8%;'.<)'=8>)"'.)+.'

Phil SmithConsultant Neurologist and Sub-Dean for Assessments

Cardiff

Page 2: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

Learning outcomes

To appreciate the characteristics of high quality SBA questions (statistical and question construction)

To discuss ways to construct better ‘difficult-to-write’ SBAs (sociology, professionalism, ethics)

Page 3: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

Learning outcomes

To appreciate the characteristics of high quality SBA questions (statistical and question construction)

To discuss ways to construct better ‘difficult-to-write’ SBAs (sociology, professionalism, ethics)

Page 4: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

FacilityProportion getting

answer correct

Number of students

30%

70%

Ideal facility is probably 60%

Page 5: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

Discrimination

score on this test upper quartile

lower quartile

score on

this question

Page 6: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

Discriminationscore on

this question

score on this test upper

quartile

lower

quartile

Questions must discriminate in favour of the best candidates

Page 7: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

Distractor efficiencyThree-item questions performed equally well to four-item questions

Tarrant M, Ware J. A comparison of the psychometric properties of three-

and four-option multiple-choice questions in nursing assessments. Nurse

Educ Today 2010; 30: 539-43.

Reducing from 5 items to 4 or 3. Pass rates and reliability remained

constant. Pass rates increased by 1% (for 4-item) and 3% (for 3-item)

questions

Kilgour JM, Tayyaba S. An investigation into the optimal number of

distractors in single-best answer exams. Adv Health Sci Educ Theory Pract.

2016;21:571-85.

It’s fine to have questions that work with only 2 or 3 distractors

Page 8: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

How many questions?

Having enough questions rescues even poor exams

Page 9: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

Learning outcomes

To appreciate the characteristics of high quality SBA questions (statistical and question construction)

To discuss ways to construct better ‘difficult-to-write’ SBAs (sociology, professionalism, ethics)

Page 10: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

Taken as read

• Has a scenario: tests application of knowledge

• Question is positive and expresses uncertainty

• Passes the cover tests: traditional and science

Question construction

Page 11: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

Which of the following is true?

Page 12: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

Science cover test!"#$%&'()%*+,"-(."/012"3'.')(+04',"/'(5.'44"2(4"("60*74&"1(5'."8)*-"204"9:(,)0;'74"-:4;+'<"

/20;2"04"'=(-0.',":40.3"'+';1)*."-0;)*4;*7&>"?2'"7(12*+*3041")'7*)14"(."(6.*)-(+"8*)-(@*."*8"

12'"A*+30"(77()(1:4>"""

B2(1"04"12'"-(0.")*+'"*8"12'"A*+30"(77()(1:4C"

!>";*.1)*+4"(-0.*"(;0,"6)'(5,*/."

D>";*.1)*+4"+070,"6)'(5,*/."

E>";*.1)*+4"7)*1'0."7)*;'440.3"

F>"0.0@(1'4"7)*1'0."(44'-6+&"

G>"0.0@(1'4"7)*1'0."6)'(5,*/."

!.4/')"E<";*.1)*+4"7)*1'0."7)*;'440.3"

!!!!!!""######$$$$$$$%%%&&&&&&'''''''(((((())))%%%%*******++++++,,,,,,"------((((((......""//////000001111112222222""""333333'''''......''''''''))))))((((((

///////22222220000;;;;;22222222"000000444444""''''''========((((((------0000000......'''''''',,,,,,"""::::::44444400000......33333333""""'''''''''++++'''''';;;;;;;11111))))

111111222222''''''"AAAAAA******+++++33333300000""""((((((777777777777(((((())))))(((((((111111:::::::444444>>>>>"""""

((((+++++0000044444'''''',,,,,,,"""///////''''''((((((555555.......'''''444444444444""22222222((((((44444"""((((((""6666660000******777777444444&&&&&&&""11111111((((((

))))))******......."------000000;;;;;;;;))))))******444444;;;;;;;;******777777&&&&&&>>>""????????222222''''''""777777((((((1111112222222******+++++++******3333333330000044444411111

55555''''''......"8888888)))))*******-------""22222200000000444444"999999::::::((((((,,,,,,))))))000000;;;;;;''''''7777744444444""""----::::::4444;;;;;;;;++++++'''''

111111""))))))'''''''7777777****))))))))111111444444"((((((........""((((((666666......********))))))------((((((++++++""888888*******))))))---------((((((@@@@@@@@@*******

''''<<<<<"

****......""******888888""

Page 13: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

!"#$%&'()%*+,"-(."/012"3'.')(+04',"/'(5.'44"2(4"(."(6.*)-(+01&"*8"12'"A*+30"(77()(1:4<"

0,'.@H',"0."("-:4;+'"60*74&>"

B2(1";'++:+()"8:.;@*."04"-*41"+05'+&"1*"6'"(6.*)-(+C"

!>"(-0.*"(;0,"6)'(5,*/."

D>"+070,"6)'(5,*/."

E>"7)*1'0."(44'-6+&"

F>"7)*1'0."6)'(5,*/."

G>"7)*1'0."7)*;'440.3"

!.4/')"G<"7)*1'0."7)*;'440.3

Science cover test

Page 14: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

• Improving reliability

• Improving validity

Writing better SBA questions Beyond the cover test

Page 15: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

Writing better SBA questions

Page 16: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

Table work

• Improving reliability (reproducibility)

• Improving validity (accuracy)

Page 17: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

• Minimising construct-irrelevance

consistent precise additional words relate to an option

• Testwise traps

convergence strategyquestion interdependence

Writing styleImproving reliability

Page 18: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

Testwise candidates

Page 19: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

Testwise candidates

A. CT scan of abdomen and pelvis

B. CT scan of thorax, abdomen and pelvis

C. gallium scan of whole body

D. positron-emission tomography scan of abdomen

E. ultrasound scan of testes and pelvis

What is the best next investigation?

Page 20: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

• Authenticity

realistic scenario

presenting complaint in first line data not summary

• Best options all plausible

in same domain includes best answer

• Best practiceevidence-based practice good practice

non-discriminatory language

Question constructionImproving validity

Page 21: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

Learning outcomes

To appreciate the characteristics of high quality SBA questions (statistical and question construction)

To discuss ways to construct better ‘difficult-to-write’ SBAs (sociology, professionalism, ethics)

Page 22: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

Is it worth it??0I."J!<"K0.."AL<"L;M(;2+(."NE>"GO(+:(@.3"7)*8'440*.(+04-"

0."-',0;(+":.,')3)(,:(1'4":40.3"4'+';1',")'47*.4'"

9:'4@*.4P"H.,0.34"8)*-"(."01'-")'47*.4'"-*,'++0.3"41:,&>"

!"#$"%&'()*$+&,()-./"$QRRSRRPT#>

“Professionalism SBAs show poor psychometrics compared to anatomy questions, relatively poor psychometric properties, and lack associations with other

constructs associated with undergraduate professional behaviour correlation to conscientiousness index.”

Advises not to include as SBAs

Need more care to write

Page 23: 0)’12345’67)+$8%+9’ ():8%;’.)’.)+.’

Difficult-to-write SBAs

• Have a clear learning outcome

• Maintain question-writing discipline

• Ensure single best answer

• Ensure passes cover test


 · $% % '( )*!+,-.*+.,+/0 " ( !"# ! % /# -%/0%0/8 " + * 9 * %8 /) -%#0'%0/# 93 ! * 9 * 8 /7 -%#0'%0/0- 93 ! * 9 * 8 /8 -"60'%/ 93 ! * 9 + %8 /$ 86 -"60'%0 93 !

3.2. - Extremaduraextremambiente.juntaex.es › files › 3_2 SOLUCION ADOPTADA.pdf · 3 8 7 0 0 0 4 3 8 7 0 0 0 4 3 8 8 0 0 0 4 3 8 8 0 0 0 4 3 8 9 0 0 0 4 3 8 9 0 0 0 4 3 9 0 0

- 9 0 8 7 - ? - 4 8 3 5 0 8 / / / + 4 1 0 8 - 3 - 9 0 8 7 ...€¦ · d e f e g g

0 '--' ( ) # *# 1 ! 2 3 4 '5'6# $ 2 7# $8 # *'9 9 : ;3 4 8 *# *'9 0 , */ 9 < - # < = '>, $'/ 9 9 < ?@ # -, < 0 , */ 9 … · ... *- . '*/ 0 '--'" ( ) # *# 1 ! "2 3

repositorio.ipcb.pt · @k " 0 "9 % # ) " " 9 "! " " i ) ) @llhj 0 "9 ) " " ) 8

ung.si · !" $%&'(# ) *+,' -$./$-0.)& 1 9 7 6 1 9 7 8 1 9 8 0 1 9 8 2 1 9 8 4 1 9 8 6 1 9 8 8 1 9 9 0 1 9 9 2 1 9 9 4 1 9 9 6 0 1 0 0 2 0 0 3 0 0 4 0 0 5 0 0 6 0 0 7 0 0 8 0 0

0+0 12134 4( ˛ ˜ 567( 8˚8*’+*)9 , 5/9

Colorado River: Mile 0 to Mile 2 - United States Army ......Sep 09, 2017  · 8 0 8 + 0 0 0 8 0 8 + 0 0 0 9 9 99 9 9 9 98 8 77 77 87 8 8 99 99 89 9 9 9 99 9 8 88 77 6 66 67 79 98 87

3,920> :1 7494.,77D 8;:=?,9? 0@=:?=,9>84??0=> @9/0= ?30 ... · 3,920> :1 7494.,77D 8;:=?,9? 0@=:?=,9>84??0=> @9/0= ?30 ... ... 1;

: 8 ; 3 0 ' % : 9 0 >,

Los Mogotes Peak - Colorado Parks and Wildlife · !i!i") 102") 124 9 3 0 0 92 0 9 1 0 9 0 0 0 8900 86 0 8 5 0 0 8 0 8400 850 870 9000 8 9 0 9 1 9 0 0 0 8 9 0 8 8 0 0 9 7 0 0 9600

Gmina Leszno Mapa ogólna gminy · 2020. 1. 16. · 4 8 4 0 0 0 4 8 6 0 0 0 4 8 8 0 0 0 4 9 0 0 0 0 4 9 2 0 0 0 4 9 4 0 0 0 4 9 6 0 0 0 4 9 8 0 0 0 598000 600000 602000 604000 606000

Fall 2010Q2 0 6 Q3 0 6 Q4 0 6 7 8 0 8 0 8 0 8 Q1 0 9 0 9 0 9 9 0 Transactions Announced $0 $400 $800 $1,200 $1,600 $2,000 Aggregate Value ($ Billions) Transactions Announced Aggregate

IPv6 Rationale and Features › doc › ipv6book-chap1.pdfIPv6 Rationale and Features 5 8 9 9 9 0 0 0 1 9 7 9 6 9 5 9 4 9 3 9 2 0 8 0 9 0 7 0 6 0 5 0 4 0 3 0 2 8 1 9 1 240 Cumulative

6 7 #8(9 :1;-&

F>,A- B,.-1-/0325476$41,.8:9 8:;>KAB 9>8$4S= 0 …jkanno/thesis.pdf9 9,.-%\ ]'%&.0" # ,A41F 9 9 < 9 ,.-¨} :

Legende · 3 9 8 3 9 8 3 9 8 3 9 8 3 9 8 3 9 8 3 9 8 4 0 0 4 0 4 0 0 4 0 0 4 0 0 4 0 0""" """"" """"" """"" " " """"" """ Stadel Weitfeld Staudham Schmalgütl 1 4 1 2 7 6 8

JoséMaríaLarrú 25 añosde ... · CentroAmérica en la AOD mundial 0 10.000 20.000 30.000 40.000 50.000 60.000 8 5 6 8 7 1 8 8 9 9 0 9 9 1 2 9 3 9 9 4 5 9 6 7 8 9 9 0 0 1 0 2 3

$0*11*(-$ 234567$ - monoskop.org€¦ · #$%")%*+,-".&"/+,0(.1"2.)).,0" #$%")%*+,-".&"/+,0(.1"2.)).,0" #c#9)$.*98$'9#$?'8#9:'1.$+*9$8"#$/#;$'98f$@':/8d$("':9.d$+**=d$#1#(89:($'/=$

4 h 8 ? = 8 : 8 : 0 g 5 = 8 o. 0 9 : 8, h 0 9 1 k 8 a : > 1 k 4 ; o 7 0" href="https://vdocuments.mx/-8530-90-4-h-8-8-8-0-g-5-8-o-0-9-8-h-0-9-8530-90.html">! 8530-90, > 4 H 8 ? = 8 : 8 : 0 G 5 = 8 O. 0 9 : 8, H 0 9 ...“ОСТ 8530-90/ГОС… · ! " 8530-90, > 4 h 8 ? = 8 : 8 : 0 g 5 = 8 o. 0 9 : 8, h 0 9 1 k 8 a : > 1 k 4 ; o 7 0

0$/, 34 *56(76# '(8(9:

Fericiti pentru totdeauna - WordPress.com...a 8 9! ! &e a 0 = 8 # )& 8 $ ) " " &0 l & 4 &. 1 7 3 " &% 3 " # ! 0 a # 8 ! 8 ). !! & . 9! 8 # 1 7 ) & 9!

Neighborhood Map - MTA St (4) web.pdfNeighborhood Map ¯ 5 0 1 0 1 9 0 0 8 0 0 2 0 9 0 2 8 1 8 1 0 8 0 1 0 8 0 8 0 1 1 8 3 9 0 6 1 5 0 2 3 8 6 0 8 9 3 6 0 2 9 0 4 8 0 8 7 2 5 8 0 1

;;3 * 0 > )8 ; $ & # @ * $ 7 ( & 9 &" 3> 9> & a 9 * $$ & $ 7 ( ' $" href="https://vdocuments.mx/-6-67-6-83-0-8-7-.html">$ 6 # $$ 67 & $ 6 · &" )8>;;3 * 0 > )8 ; $ & # @ * $ 7 ( & 9 &" 3> 9> & a 9 * $$ & $ 7 ( ' $

0$%)(!#'(!$# - MoMA · ! "#$ %&"&'()*$+ ,(-./ &-$ 01210$0 1!"( "#$ 3(''(,h 1!) %&"$)(-1$/5 61!1/#$0 7-&,1!)/8 9.$"%#$/ &!0 9"*01$/8 9%*':"*-$ 9"*01$/8 ;*-&' 9"*01$/8

After the Crisis: the Economics of the P-C Insurance ...-400-200 0 200 400 600 8 8 8 8 8 8 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 9 9 0 0 0 0 0 0 Monthly Change Employment* Job Losses Since

Nirvana - Smells like teen spirit · 9/15 72 Bridge 3 3 3 0 3 3 0 3 3 3X X X 9 9 7 0 8 8 6 0 9 9 7 0 10 10 8 8 10 10 8 8 9 9 7 8 8 6 9 9 7 14 14 12. 10/15 84

Chi-X Japan Monthly Report...5 Statistics - Chi-Alpha & Chi-Select 0.0% 1.0% 2.0% 3.0% 4.0% 5.0% 0 20 40 60 80 100 120 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 9 9 0 0 0 0 0 0 0 0 0 n Average

C2.a Ordenació sòl urbà 1 8 1 I.2 9 1 8 5 8 0 BI · 2013. 4. 3. · 2 0 0 1 9 8, 6 8 1 9 8, 5 9 1 9 7, 6 7 1 9 7, 6 6 1 9 7, 6 4 1 9 7, 6 4 1 9 6, 6 0 1 9, 5 2 2 0 0, 5 0 2 0 3,

0(#*'-!123"4"54%43"%467%8$9" 1!:';'%47;545$9"!"#$%&'()*+("%,$'+$*)'+-'+.#/0*$("'12+.*"'3%&#+-#"+456+7$-'%&'+8,("'+%*$*0%#66+ 9("*/'+#0./#00'+'$+:+-'+8,($3+%*$*0%

KMITT 10 Sep 2009 · 0 7 0 7 0 7 0 7 0 7 0 8 0 8 0 8 0 8 0 8 0 8 0 8 0 8 0 8 8 0 8 0 9 0 9 0 9 0 9 0 9 0 9 9 0 9 0.00 1.00 Jan - F eb Mar r Ma y n - Ju l-g pOct N o v Dec Jan-Fe b-y

9 9 ! # 9& : +(2 - 6 $% ! & !! ' &( ! ) * ! # ! 9 8 ... · /" * 0! " " 1( ( ))" ' 2 3456 7 2 1 0 8 0 9 " ! ) : " 0

?!) + 9 % 7 0,-( & " /. # 3 1" href="https://vdocuments.mx/-e-nh-9-3-8-1-4-4-0-9-8-1-2-3-b-5-9-.html">? E - NH.gov · 2 & 9 * =,-$. 3 8 1 # / 4 ! 4 & * ' 0 " 9 %)? (+, $ #. 8 1 2 3 / : b 5 !? 9 % 6 8-, a) + ( # " $ 0 ... 6 " $, % = 9 * 0 /. 8 1 4 #- > ?!) + 9 % 7 0,-( & " /. # 3 1

I II * * 2011 5 4 9 9 3 4 1 8 0 2012 0 6 6 5 4 9 4 8 02014 0 7 7 6 9 3 3 8 0 2015 0 1 1 5 9 4 1 8 0 2016 0 3 3 5 5 0 3 8 0 2017 0 2 2 0 8 8 2 8 0 2018 0 3 3 0 5 6 2 8 2 2019 0 0 4

"8-9:)+054-9" href="https://vdocuments.mx/2-00-38-3-5-20-4-9-5-8-58-9-53-802-9-5.html">evolucionar.com.co · 2 ,- ,0+0-3*8- ,- ! 3 #-+);,5 ,- 20-4:-9 ")/5 +8--,58-9 53-8+0)2-9 ")/5 $)2)8059 $-/;80,), $5+0)2 > "8-9:)+054-9