establishing content validity of a novel written examination to assess medical students on the...
TRANSCRIPT
![Page 1: Establishing Content Validity of a Novel Written Examination to Assess Medical Students on the Surgery Clerkship A Berlin MD, A Reinert BA, A Swan-Sein](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e245503460f94b11f85/html5/thumbnails/1.jpg)
Establishing Content Validity of a Novel Written Examination to Assess Medical
Students on the Surgery ClerkshipA Berlin MD, A Reinert BA, A Swan-Sein MEd PhD,
R Nowygrod MD, A Fingeret MD
![Page 2: Establishing Content Validity of a Novel Written Examination to Assess Medical Students on the Surgery Clerkship A Berlin MD, A Reinert BA, A Swan-Sein](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e245503460f94b11f85/html5/thumbnails/2.jpg)
Agenda
• Needs assessment• Exam Development• Exam Design• Methodology• Results• Discussion
![Page 3: Establishing Content Validity of a Novel Written Examination to Assess Medical Students on the Surgery Clerkship A Berlin MD, A Reinert BA, A Swan-Sein](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e245503460f94b11f85/html5/thumbnails/3.jpg)
Needs Assessment• Historical assessment of students– Clinical evaluations– Unstructured oral exam – NBME surgery subject exam
• Goals of new evaluation tool– Objective and reliable– Congruent with educational objectives– Based on ACGME core competencies
![Page 4: Establishing Content Validity of a Novel Written Examination to Assess Medical Students on the Surgery Clerkship A Berlin MD, A Reinert BA, A Swan-Sein](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e245503460f94b11f85/html5/thumbnails/4.jpg)
Exam Design
• Novel Exam: Clinical Skills Exam (CSE)– 20+ clinical scenarios• 5 randomly selected per exam
– Short answer or essay response– Administered over 3 hours– ExamSoft computer software
• Grading – Rubric– Senior residents
![Page 5: Establishing Content Validity of a Novel Written Examination to Assess Medical Students on the Surgery Clerkship A Berlin MD, A Reinert BA, A Swan-Sein](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e245503460f94b11f85/html5/thumbnails/5.jpg)
Example 1
• Identify operation
• Obtain surgical consent
• Determine order
• Describe steps shown
![Page 6: Establishing Content Validity of a Novel Written Examination to Assess Medical Students on the Surgery Clerkship A Berlin MD, A Reinert BA, A Swan-Sein](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e245503460f94b11f85/html5/thumbnails/6.jpg)
Example 2
• Generate differential based on history
• Describe risk factors• Interpret radiographic
study• Identify next step in
management
![Page 7: Establishing Content Validity of a Novel Written Examination to Assess Medical Students on the Surgery Clerkship A Berlin MD, A Reinert BA, A Swan-Sein](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e245503460f94b11f85/html5/thumbnails/7.jpg)
Methodology• July, 2011 - February, 2013– n = 262
• Analysis– Descriptive statistics– Multivariate analysis
• Comparison data for validity assessment – USMLE Step 1 and 2CK– NBME Subject Exams in Medicine and Surgery– Clerkship clinical evaluations – Clerkship order
![Page 8: Establishing Content Validity of a Novel Written Examination to Assess Medical Students on the Surgery Clerkship A Berlin MD, A Reinert BA, A Swan-Sein](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e245503460f94b11f85/html5/thumbnails/8.jpg)
Results: Descriptive Statistics – CSE Content and Score Distribution
![Page 9: Establishing Content Validity of a Novel Written Examination to Assess Medical Students on the Surgery Clerkship A Berlin MD, A Reinert BA, A Swan-Sein](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e245503460f94b11f85/html5/thumbnails/9.jpg)
Results: Trend Analysis – CSE and NBME Surgery Subject Exam
![Page 10: Establishing Content Validity of a Novel Written Examination to Assess Medical Students on the Surgery Clerkship A Berlin MD, A Reinert BA, A Swan-Sein](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e245503460f94b11f85/html5/thumbnails/10.jpg)
Results: Multivariate Statistics - Validity Analysis
CSE NBME
CorrelationLinear
Regression CorrelationLinear
Regressionr p value r p value
NBME Surgery 0.36* 0.82 NBME Medicine 0.01 0.54 0.08 0.20Step 1 0.34* 0.82 0.55* 0.02Step 2 0.44* 0.007 0.56* 0.004Clerk - Clinical Reasoning 0.48* 0.001 0.43* 0.09
Clerk - Fund of Knowledge 0.56* 0.004 0.63* < 0.001
* = p < 0.001
![Page 11: Establishing Content Validity of a Novel Written Examination to Assess Medical Students on the Surgery Clerkship A Berlin MD, A Reinert BA, A Swan-Sein](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e245503460f94b11f85/html5/thumbnails/11.jpg)
Results: Multivariate Statistics – Validity Analysis
CSE NBMEBivariate Analysis
Bivariate Analysis
Logistic Regression
p value p value p value
Clerkship order ANOVA 0.30 < 0.001 0.01
OB/Gyn prior X2 0.34 < 0.001 0.02
Medicine prior X2 0.69 < 0.001 0.03
![Page 12: Establishing Content Validity of a Novel Written Examination to Assess Medical Students on the Surgery Clerkship A Berlin MD, A Reinert BA, A Swan-Sein](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e245503460f94b11f85/html5/thumbnails/12.jpg)
Results: Student Feedback on CSE
![Page 13: Establishing Content Validity of a Novel Written Examination to Assess Medical Students on the Surgery Clerkship A Berlin MD, A Reinert BA, A Swan-Sein](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e245503460f94b11f85/html5/thumbnails/13.jpg)
Results: Student Feedback - Word Cloud
![Page 14: Establishing Content Validity of a Novel Written Examination to Assess Medical Students on the Surgery Clerkship A Berlin MD, A Reinert BA, A Swan-Sein](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e245503460f94b11f85/html5/thumbnails/14.jpg)
Limitations
–Study•Generalizability• Statistical analysis
![Page 15: Establishing Content Validity of a Novel Written Examination to Assess Medical Students on the Surgery Clerkship A Berlin MD, A Reinert BA, A Swan-Sein](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e245503460f94b11f85/html5/thumbnails/15.jpg)
Future Directions
• Future directions– New content– Open access– Computer algorithm based grading– Use as formative assessment mid-clerkship
![Page 16: Establishing Content Validity of a Novel Written Examination to Assess Medical Students on the Surgery Clerkship A Berlin MD, A Reinert BA, A Swan-Sein](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e245503460f94b11f85/html5/thumbnails/16.jpg)
Conclusions
• CSE is a useful adjunct to NBME surgery subject exam– Independent of clerkship order– Course goals and objectives– ACGME core competencies
![Page 17: Establishing Content Validity of a Novel Written Examination to Assess Medical Students on the Surgery Clerkship A Berlin MD, A Reinert BA, A Swan-Sein](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e245503460f94b11f85/html5/thumbnails/17.jpg)
•Questions / Comments