student assessment: new ideas and old basics louis pangaro, md professor and vice-chair for...
TRANSCRIPT
![Page 1: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/1.jpg)
Student Assessment:New Ideas and Old Basics
Louis Pangaro, MD
Professor and Vice-chair
for Educational Programs
Department of Internal Medicine
Uniformed Service University of the Health Sciences
26 September 2007
![Page 2: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/2.jpg)
2
“clinical assessment”
• By teachers (house staff and faculty)
• On clinical rotations
• Based on words = descriptors
• Using words = descriptive
• In vivo (vs. in vitro end of course/year)
![Page 3: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/3.jpg)
3
What’s old?
• Suspicion of grades by teachers
• Teachers reluctance to be direct (honest?)
• Belief that grading by teachers is subjective
• Lawsuits about low grades• Jamieson, Guidebook for Clerkship Directors, 2005.
![Page 4: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/4.jpg)
4
What’s basic?
• We have an obligation
• Fairness
• Mentoring – The Hippocratic Oath
• Professionalism– Duty and expertise (Pellegrino)
![Page 5: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/5.jpg)
5
How it looked in 1987
• lack of meaningful comments by evaluators
• insufficient definition of evaluation criteria
• too much inter-observer variability
• late submission of evaluations
• delay in feedback to students
Tonesk X, Buchanan RG. J Med Ed. 1987
![Page 6: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/6.jpg)
6
How it looked in 2001
• “Areas of weakness in current [clinical] evaluation models include psychometric properties associated with the tools, namely their questionable reliability and validity.”
• Turnbull, International Handbook of Research in Medical Education, 2002.
![Page 7: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/7.jpg)
7
How it looks in 2007
• “Constructive criticism is hard to come by..”
• “…Candor is at least as painful to the provider as to the recipient…”
• “…faculty members feel uncomfortable inflicting pain even in a good cause like student improvement.”
• HMS Student Handbook, 2006-2007
![Page 8: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/8.jpg)
8
focus
• The emotional issues for teachers and learners in the grading process.
• Dealing with barriers to candor.
![Page 9: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/9.jpg)
9
• Quality: minimizing unwarranted variation in physician performance.– Wennberg
• Faculty Development: minimizing unwarranted variation in teacher performance.
![Page 10: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/10.jpg)
10
What’s new
• A believe that quality methods can apply to teaching.
• Search for more rigor.
• “Best Evididence Med Ed” (BEBM)
• “Med Ed Research Certificate”
• MPH, MHPE programs
![Page 11: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/11.jpg)
11
Today: describing success in clinical evaluations
![Page 12: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/12.jpg)
12
New focus on evaluation in clincial setting:
• Attention to professional traits (Papadakis)
• Lawsuits by patients
![Page 13: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/13.jpg)
13
Context of new methods
• Objective Structured Clinical Exams
• 3600 assessments
• Portfolios
• Descriptive vocabularies– To get more inter-rater agreement– To calibrate differences between levels of
performance
![Page 14: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/14.jpg)
14
FOCUSED PHYSICAL EXAMINATION
• [Norm] • Exam is generally appropriate in scope
and technique. Identifies major abnormalities and pertinent normal findings, only occasionally missing elements. Exam linked to history. Appropriate for level of training.
• HMS Clerkship Grading Form
![Page 15: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/15.jpg)
15
FOCUSED PHYSICAL EXAMINATION (2)
• [Poor]
• Consistently misses important findings and often does not make appropriate connection between history and physical. Often uses faulty or inappropriate technique. Not organized or thorough.
![Page 16: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/16.jpg)
16
FOCUSED PHYSICAL EXAMINATION (3)
• [Excellent]
• Exam is consistently superior. Uncovers subtle and important findings, incorporating advanced techniques where appropriate. Exceptionally organized and thorough, even on difficult cases.
![Page 17: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/17.jpg)
17
American Board of Internal Med
1 2 3 4 5 6 7 8 9
Minimal acceptable
Average and 2 SDs
![Page 18: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/18.jpg)
18
ACGME Outcomes Project
• New emphasis on outcomes (vs
curriculum)
• “long-term effort designed to
emphasize educational outcome
assessment in residency programs
and in the accreditation process.”
![Page 19: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/19.jpg)
19
ACGME: Outcomes: a cultural shift
• Designing Curriculum is no longer enough
• Results must be demonstrated.
• A shift from process to product.
• Content expertise not enough; need pedagogic expertise
![Page 20: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/20.jpg)
20
ACGME “Competencies”(1999)
• Medical Knowledge
• Interpersonal & communication skills
• Professionalism
• Patient Care
• System-based Practice
• Practice-based learning & Improvement
![Page 21: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/21.jpg)
21
the ACGME has spoken !
![Page 22: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/22.jpg)
22
CanMEDS (2005)
• Medical Expert
• Communicator
• Collaborator
• Professional
• Manager
• Health Advocate
• Scholar
![Page 23: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/23.jpg)
23
“R.I.M.E. Scheme”
– Reporter
– Interpreter
– Manager/Educator
Pangaro, Academic Med, 1999
![Page 24: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/24.jpg)
24
Acceptance (1) Ob-Gyn
“The RIME method is a valid, logistically feasible and acceptable way of assessing medical student clinical performance…
…..minimizes disadvantages of descriptive evaluation, and maximizes the opportunity for accurate observations and helpful feedback.”
APGO UME Taskforce:Espey et al, Am J Ob Gyn, 2007
![Page 25: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/25.jpg)
25
Fairness
Consistency
Expectations
![Page 26: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/26.jpg)
26
Principle 1 : Fairness to society :
valid (not arbitrary) and sensitive to detect marginal performers
to students : know what’s expected, timely feedback
to teachers : know what observations to makeprotected (legally, emotionally)
![Page 27: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/27.jpg)
27
Principle 2 : Consistency
Reliability- Within teacher
- Between teachers- In same rotation or across blocks
- Between sites in same clerkship
- Between disciplines
- suitable for high-stakes decisions
![Page 28: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/28.jpg)
28
strategy
• Simplicity leads to acceptance and use
• Acceptance to consistency• Consistency to fairness
![Page 29: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/29.jpg)
29
Fairness & Consistency
Reliability &Validity =
Stability of measurement and strength of inferences from
observations
![Page 30: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/30.jpg)
30
“ Old” beliefs in our culture
• Grading by teachers is subjective, Examinations are objective
• Measurement > description, Numbers are > words
![Page 31: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/31.jpg)
31
Framing the question more simply:
• What do we expect of students?
• Can we get all teachers to have the same expectations, and apply them consistently?
• [A question of words and of conceptual frameworks]
![Page 32: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/32.jpg)
32
Educational Goals
Curriculum
Evaluation
Feedback / Grading
Depends on expectations
community
![Page 33: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/33.jpg)
33
Teacher/program
Learner
Content – Goals (Patients)
The goal: progressive independence of the learner
after SFDP
![Page 34: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/34.jpg)
34
Frames of reference for expressing goals
1. Analytic
2. Developmental
3. Synthetic
![Page 35: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/35.jpg)
35
goals for education (generic):
• attitudes/behavior• skills• knowledge
“KSA”
![Page 36: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/36.jpg)
36
Bloom’s Taxonomies
Cognitive Domain
Psychomotor Domain
Affective Domain
![Page 37: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/37.jpg)
37
1. Analytic expression of Goals
– “ana-lytic”: takes the learner “apart”
– into domains, categories
– “attitude”, “skills”, “knowledge”
– considered separately
– generic terms
– useful for discrete assessments
![Page 38: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/38.jpg)
38
Curricular Goals: KNOWLEDGE The School of Medicine will ensure that before graduation a student will have
demonstrated, to the satisfaction of the faculty, the following: • The capacity to recognize the limitations in one’s knowledge and clinical skills and
to make a commitment to engage in lifelong learning • Knowledge of the normal structure and function of each of the major organ
systems of the body and the current basic scientific mechanisms operative at the systemic, cellular, and molecular levels
• Knowledge of the various causes (genetic, developmental, metabolic, toxic, microbiologic, immune, psychosocial, neoplastic, traumatic, and degenerative) of illnesses and diseases
• Knowledge of the altered structure and function of the body and its major organ systems that are seen in various illnesses and diseases
• Knowledge of the scientific method in establishing the causation of disease and efficacy of traditional and non-traditional therapies
• Knowledge of health care policy and the economic, psychological, social, and cultural factors that affect health and health care delivery
• Knowledge of the most frequent clinical, laboratory, radiographic, and pathologic manifestations of common as well as life threatening diseases
• Knowledge about relieving pain and ameliorating the suffering of patients • Knowledge of the epidemiology of diseases and the systematic approaches useful
in promoting health • Knowledge of techniques of patient education and counseling in basic lifestyle
changes/prevention • Knowledge of and approaches to reduce the psychological and physical risks and
stresses of the practice of medicine
![Page 39: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/39.jpg)
39
Curricular Goals: SKILLS • The School of Medicine will ensure that before graduation a student will have
demonstrated, to the satisfaction of the faculty, the following: • The ability to obtain an accurate medical history and the ability to perform both a complete
and an organ specific examination, including a mental status examination (See appendix A) • The ability to perform routine technical procedures (See appendix B) • The ability to interpret the results and be aware of the indications, complications, and
limitations of commonly used diagnostic procedures (See appendix C) • The ability to demonstrate knowledge of theories and principles that govern ethical decision
making • The ability to reason deductively and inductively in solving clinical problems • The ability to construct appropriate differential diagnoses and treatment plans for patients
with common conditions, both acute and chronic, including medical, psychiatric, and surgical conditions, and those requiring short- and long-term rehabilitation
• The ability to recognize patients with immediate life threatening conditions regardless of etiology, and to institute appropriate initial therapy
• The ability to recognize and outline an initial course of management for patients with serious conditions requiring critical care
• The ability to communicate effectively, both orally and in writing, with patients, patients’ families, colleagues, and others with whom physicians must exchange information in carrying out their responsibilities
• The ability to select appropriate tests for detecting patients at risk for specific diseases and to determine strategies for responding appropriately
• The ability to retrieve, critically review, and effectively utilize biomedical information from electronic databases and other resources for solving problems and making decisions that are relevant to the care of individuals and populations
• The ability to evaluate the economic, psychosocial, and cultural factors that impact the health of patients and families and to incorporate these into assessment and treatment plan
![Page 40: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/40.jpg)
40
Curricular Goals: ATTITUDES • The School of Medicine will ensure that before graduation
a student will have demonstrated, to the satisfaction of the faculty, the following:
• A commitment to advocate the interests of one’s patients • Compassionate treatment of patients, and respect for their
privacy and dignity • Honesty and integrity and dutifulness in all interactions with
patients, their families, colleagues, and others with whom physicians interact
• An understanding of, and respect for, the roles of other health care professionals, and the need to collaborate with others in caring for patients and promoting health
• A commitment to provide care to patients who are unable to pay and to advocate for access to health care for members of underserved populations
![Page 41: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/41.jpg)
41
2. Developmental Terms:
• “Novice”
• “Advanced learner”
• “Expert”
can identify absence of pulse
can distinguish specific arrythmias
can manage ventricular fibrillation
![Page 42: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/42.jpg)
42
Dreyfus and Dreyfus
Novice Advanced beginner Competent performance Proficient performance Intuitive expert Master
Mind Over Machine (1986)
students
residents
facutly
![Page 43: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/43.jpg)
43
•time-line, progression
included levels of function
•essential for multi-year
training
•although the terms (“novice”,
“master”…) remain generic
Developmental vs. Analytic
![Page 44: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/44.jpg)
44
Goals: ACGME “Competencies”
• Medical Knowledge
• Interpersonal & communication skills
• Professionalism
• Patient Care
• Practice-based learning
• System-based Practice
analytic? developmental?
What about the last three??
![Page 45: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/45.jpg)
45
A cube to encompass competence
Medical Knowledge
Patient Care
Interpersonal skills
Professionalism
Practice-based learning/improv.System-based Practice
UME --> GME CME
clinichospital
school
![Page 46: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/46.jpg)
46
Resistance is futile
![Page 47: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/47.jpg)
47
Educational Goals
Curriculum
Evaluation
Feedback / Grading
Depends on expectations
community
![Page 48: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/48.jpg)
48
• “syn-thetic” -putting the learner back together
• “K S A” are all required, integrated• terms are a bit less generic, more
behavioral
3. the “Synthetic”framework
![Page 49: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/49.jpg)
49
“R.I.M.E. Scheme”
– Reporter
– Interpreter
– Manager/Educator
Pangaro, Academic Med, 1999
![Page 50: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/50.jpg)
50
a framework to classify level of function
• rudimentary reporting: “My patient has
a fever, cough and a bad rash - it’s
vesicular or pustular ….”
• rudimentary interpreting: “I think it
might be due to chicken pox or
herpes.”
![Page 51: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/51.jpg)
51
• rudimentary manager/educator:
“I’d consider a smear of the
fluid and a chest x-ray . …We
might observe or treat with
acyclovir, …but I’m not sure. I’ll
have to look this up.”
![Page 52: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/52.jpg)
52
Alternative model R.I.M.E.
• Framework that is
– developmental
– behavioral•student can visualize, framework for observer
– “synthetic”
![Page 53: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/53.jpg)
53
“RIME”
• Reporter: takes ownership of getting the facts on every patient (“what”?)
• Interpreter: takes ownership of thinking and explaining (“why”?)…
• Manager: takes ownership of planning with patient (“how?)…
• Educator: takes ownership of developing and sharing expertise…
![Page 54: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/54.jpg)
54
Matrix: transition to higher expectations
I II III IV PGY1 PGY2/3 Practice
EDUCATOR
MANAGER
INTERPRETER
REPORTER
I = introduced R = repetition P = proficiency
I R R R P
I R P
I R P
I R P
M
M
M
M
M = mastery in practice
![Page 55: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/55.jpg)
55
Duty and Expertise (Pellegrino)
• Each RIME “level” is a way of asking, does the student fulfill that promise
• Making a diagnosis, not “giving” it.
• “Objective”
• Does it affect teacher ratings?
![Page 56: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/56.jpg)
56
Example: Construct Validity Grade Distributions Univ. of Utah
0
10
20
30
40
50
60
0 0.5 1 1.5 2 2.5 3 3.5 4
Numeric
Low Pass Pass HP HonorsLow Pass Pass HP Honors
Battistone Acad. Med. , 2001
![Page 57: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/57.jpg)
57
Grade Distributions Univ. of Utahafter RIME methods
0
10
20
30
40
50
60
0 0.5 1 1.5 2 2.5 3 3.5 4
RIMENumeric
OO R I M E R I M E
Battistone Acad. Med. , 2001
![Page 58: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/58.jpg)
58
Case using RIME:
W.O., student, “presents” Mrs. Jones:
• 45yo woman with acute lower back pain
• gives detailed description clinical picture suggestive of acute lumbar strain
• through physical examination, – blood pressure 130/80, heart rate 80 – left-sided para-spinal tenderness L2 – L5
![Page 59: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/59.jpg)
59
Case using RIME (continued)
• while student is writing up findings in the patient’s record, you interview and examine patient.
• Mrs. Jones asks: “Doctor, can you take my blood pressure since no one has?”
• at what “RIME” level is this student?
![Page 60: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/60.jpg)
60
Is the student ready for more responsibility – yes or no?
RIME is a razor.
![Page 61: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/61.jpg)
61
Complimentary Approaches
Analytic (domains)
• Attitude
• Skills
• Knowledge
Synthetic (“steps”)• (non-reporter)• Reporter• Interpreter• Manager/
EducatorACGME “Professionalism”
![Page 62: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/62.jpg)
62
The RIME rhythm is familiar:
…….S.0…..
….A…………
….P………….
H&P………….
Assessment..
Plan………….
Reporter
Interpreter
Manager/
Educator
![Page 63: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/63.jpg)
63
X X X X XX X X
Reliable
X X X X XX
X X X X X XX X
Valid
Reliability versus validity
![Page 64: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/64.jpg)
64
reliability
• Computed estimate of whether an assessment tool is testing a single construct.
• A reliability of 0.8 is considered needed for high stakes decisions (80% signal).
![Page 65: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/65.jpg)
65
Typical Reliabilities
exam type alpha
• “shelf”-100 MCQ 0.75 - .8
• Step 1 USMLE 0.9 - .95
• OSCE (6 stations) 0.5 - 0.6
• OSCE (12 stations) 0.7 - 0.9
![Page 66: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/66.jpg)
66
Third-year performance Level = Grade
• (Observer)Low Pass
• Reporter Pass
• Interpreter High Pass• Manager/ Honors
Educator
D 1.0
C 2.0
B 3.0
A 4.0
![Page 67: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/67.jpg)
67
USU Medicine clerkship
(students) Reliability
n = 467 0.83
Roop, Amer J Med, 2001
![Page 68: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/68.jpg)
68
X X X X XX
X X X X X XX X
Valid
validity
What can you infer from your evaluation? Are you measuring what is important?
![Page 69: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/69.jpg)
69
EDP Evaluation System
• RIME Vocabulary
• Formal Evaluation Sessions– Sit down with teachers every few weeks– Noel, J Med Ed, 1987 (detecting marginal
students)– “Frame of reference training”
![Page 70: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/70.jpg)
70
Content Validity: Detecting Deficiencies in Professionalism
% of professionalism domains rated less than acceptable
5.5
11.4
5.7
11.910.2
17.7
0
5
10
15
20
Ambulatory# Ward+
DI
Checksheet DI
Written DI
Eval Session DI
Hemmer, Academic Medicine, 2000.
![Page 71: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/71.jpg)
71
Predictive Validity: Sensitivity in predicting Internship Problems
(PGY1 supervisor surveys)
75
8
53
9
0
10
20
30
40
50
60
70
80
MedicineGPA s Medicine
percent
Low Ratings Bad Comment
USU classes of 86 - 93
Lavin, Academic Medicine 1998
![Page 72: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/72.jpg)
72
Acceptance (2): Medicine
• 93% teachers’ evaluations
• 81% NBME subject examination
• 42 % RIME vocabulary
• 32% OSCE
• 22% direct observation (mini-CEX)
Hemmer, Teach Learn Med, 2007
![Page 73: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/73.jpg)
73
Limitations of our system
• RIME is sometimes taken as developmental scheme.
• The analytic model is very strong, and RIME is used for cognitive growth only.
• Wanting to skip the Evaluation Sessions - they take time!
![Page 74: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/74.jpg)
74
Basics Innovations
• Clinicians are good diagnosticians
• We are mentors for our students
• We promise society duty and expertise
• Descriptors provide patterns and exemplars
• Descriptive frameworks allow behavioral feedback
• Synthetic frameworks make this simpler
![Page 75: Student Assessment: New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed](https://reader035.vdocuments.mx/reader035/viewer/2022062800/56649e025503460f94aecbd3/html5/thumbnails/75.jpg)
75
strategy
• Simplicity leads to acceptance and use
• Acceptance to consistency• Consistency to fairness