using categories to direct curriculum reform and evaluation

866.429.8889 | 1.954.429.8889 learn.examsoft.com

Thank you for joining. The webinar will begin shortly.

Using Categories to DirectCurriculum Reform & Evaluation

Please pose questions to the presenter through the “Questions” field of the Go To Webinar tool on the right side of your screen.

All questions will be addressed at the conclusion of the presentation.

USING CATEGORIES TO DIRECT CURRICULUM REFORM & EVALUATIONSarah B. McBrien, MSCurriculum & Assessment ManagerCollege of MedicineUniversity of Nebraska Medical Center

Session Objectives• Transform category data into a map of the topics you’re

currently assessing.• Identify gaps and redundancies in a curriculum in order to

better manage content coverage, frequency, and location.• Consider methods for connecting assessment data to

learning objectives data.

Measure Change• Intended or Unintended• Short-Term or Long-TermP

urpo

se

Complexity of Learning

Complexity of the Program

Cha

lleng

es

Frequent & Regular Monitoring

All Students & Subgroups

Sys

tem

atic

A

ppro

ach

When

Who

Opportunities for Category Development

Licensing Bodies

Governing Bodies

Course Objective

s

Program Objectives

Faculty

StudentsProgram Objectives

Opportunities for Category Development

Item Type

ACGME

USMLE Course Objectives

Review Process

Faculty Author

At UNMCItem Usage

Cognitive Level

Instructional Mode

Construction Zone

USMLE Content Outline

Each Examination

Historical Record of Use

Example A: Topic Reporting

2015-2016

07-

Musculoskeletal System %

Points

07.02-Abnormal Processes: Health and Health

Maintenance, Screening, Diagnosis,

Management, Risks,

Prognosis % Points

07.02.01-Infectious,

inflammatory, and

immunologic disorders %

Points

07.02.02-Neoplasms %

Points

07.02.03-Degenerative and metabolic disorders %

Points

07.02.04-Traumatic and mechanical disorders %

Points

07.02.05-Congenital disorders %

Points

07.02.06-Adverse effects of

drugs on the musculoskeletal system %

Points

# Assessments 16 13 2 2 4 8 4 4

# Items 177 118 38 6 13 36 9 6

Group Average 81.70% 85.03% 88.47% 81.83% 91.07% 80.11% 83.69% 73.54%

2015-2016

07-

Musculoskeletal System %

Points


Maintenance, Screening, Diagnosis,

Management, Risks,

Prognosis % Points

07.02.01-Infectious,

inflammatory, and

immunologic disorders %

Points

07.02.02-Neoplasms %

Points


Points


Points

07.02.05-Congenital disorders %

Points

07.02.06-Adverse effects of

drugs on the musculoskeletal system %

Points

# Assessments 16 13 2 2 4 8 4 4

# Items 177 118 38 6 13 36 9 6

Group Average 81.70% 85.03% 88.47% 81.83% 91.07% 80.11% 83.69% 73.54%

07-Musculoskeletal System % Points


Maintenance, Screening, Diagnosis, Management, Risks, Prognosis % Points

07.02.01-Infectious, inflammatory, and

immunologic disorders % Points

07.02.02-Neoplasms % Points


Points


Points07.02.05-Congenital disorders % Points

07.02.06-Adverse effects of drugs on the

musculoskeletal system % Points

# Assessments 16 13 2 2 4 8 4 4

# Items 177 118 38 6 13 36 9 6

Group Average 81.70% 85.03% 88.47% 81.83% 91.07% 80.11% 83.69% 73.54%ID/Rev DifficultyPt. Biserial (Question)

Avg Answer Time Categories

2364 0.000 0.000 - Years Used 2013-2014 2894 0.976 0.203 01:03 Years Used 2013-2014 2014-2015 2015-2016 CPEE Exam 12803 0.984 0.235 01:00 Years Used 2014-2015 2015-2016 CPEE Exam 1

4437 0.938 0.088 00:32 Years Used 2013-2014 2014-2015 2015-2016 MS/Derm/BLS Exam 1







6083 0.984 -0.026 00:51 Years Used 2013-2014 2015-2016 MS/Derm/BLS Exam 1


5976 0.984 0.186 00:51 Years Used 2013-2014 2014-2015 2015-2016 MS/Derm/BLS Exam 15902 0.000 0.000 - Years Used 2014-2015 6089 0.000 0.000 - Years Used 2013-2014 794 0.000 0.000 - Years Used 2013-2014 755 0.818 0.282 00:30 Years Used 2014-2015 2015-2016 Structure Exam 1

Example B: Exam Blueprinting

Principles of Foundational ScienceBiochemistry and molecular biology

Gene expression: DNA structure, replication, exchange, and epigenetics (eg, imprinting, X-activation, DNA methylation)24 items5 Exams (3 in M1 Year, 2 in M2 Year) Average Difficulty Index: 0.845Average Time on Item: 1.12 mins

Example C: Making Predictions

Example D: Gap & Redundancy Analysis

M1 ICE Exams

4 exams, 228 total items45 items come from USMLE Category:Medical ethics and jurisprudence, including issues related to death and dying and palliative care

Medical ethics and jurisprudence, including issues related to death and dying and palliative care

Example E: Curriculum Mapping

ID/Rev DifficultyPt. Biserial (Question) Avg Answer Time

3256 0.589 0.430 01:082672 0.814 0.486 01:023251 0.744 0.308 01:073179 0.891 0.220 00:453258 0.884 0.483 01:193252 0.543 0.227 00:213255 0.899 0.344 00:343261 0.597 0.367 01:403253 0.605 0.461 01:203320 0.736 0.365 00:59Average 0.730

Concluding Messages• Think carefully about how the category feature in

ExamSoft can contribute to your program evaluation efforts.

• You’re never too late. Categories can be attached to an item after an exam administration.

• ExamSoft’s category feature is flexible.• Consider the other systems you have in place and how

they connect to the data in ExamSoft.

Contact Information & Resources• Sarah McBrien: [email protected] 402-559-2505

• UNMC College of Medicine• UNMC Training the Physicians of Tomorrow• USMLE Content Guide• ACGME Competencies Guidebook

http://unmc.edu/com/

http://unmc.edu/com/curriculum/the-future-curriculum/overview.html

http://www.usmle.org/pdfs/usmlecontentoutline.pdf

http://www.acgme.org/Portals/0/MilestonesGuidebook.pdf

using categories to direct curriculum reform and evaluation

Education