direct observation of clinical skills during patient care new insights – reynolds meeting 2012...
TRANSCRIPT
![Page 1: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/1.jpg)
Direct Observation of Clinical Skills During Patient Care
NEW INSIGHTS – REYNOLDS MEETING 2012
Direct Observation Team:J. Kogan, L. Conforti, W. Iobst, E. Holmboe
![Page 2: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/2.jpg)
2
Medical Education Trend 2000- present
Competency BasedEducation
Variable length, defined outcome
Fixed length, variable outcome Structure/Process•Knowledge acquisition•Single subjective measure•Norm referenced evaluation•Evaluation setting removed•Emphasis on summative
Competency Based•Knowledge application•Multiple objective measures•Criterion referenced•Evaluation setting: DO•Emphasis on formative
Caraccio et al 2002
![Page 3: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/3.jpg)
3
In-Training Performance AssessmentAssessment in authentic situations
Learners’ ability to combine knowledge,skills, judgments, attitudes in dealing with realistic problems of professional practice
Assessment in day to day practice Enables assessment of a range of essential
competencies, some of which cannot be validly assessed otherwise
Govaerts MJB et al. Adv Health Sci Edu. 2007;12:239-60
![Page 4: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/4.jpg)
4
Observation to Test AssumptionsDirect observation can test assumptions4 observations needed to detect outliers
Shared responsibility
Detect Outliers Feedback/development
TIME/TASKEarly Late
![Page 5: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/5.jpg)
5
Observation and Safe Patient Care
Importance of appropriate supervisionEntrustment
Trainee performance* X Appropriate level of supervision**
Must = Safe, effective patient-centered care
* a function of level of competence in context
**a function of attending competence in context
![Page 6: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/6.jpg)
6
Types of SupervisionRoutine oversight
Clinical oversight planned in advance (i.e. what we normally do)
Responsive oversight: Clinical activities that occur in response to trainee or
patient specific issues (i.e. you do more than usual)Direct patient care:
When supervisor moves beyond oversight to actively providing care for the patient
Backstage oversight: Clinical oversight which the trainee is not aware of
Kennedy TJT et al. JGIM 2007.22:1080-85.
![Page 7: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/7.jpg)
7
Your Supervision
How do you usually supervise?When do you supervise more closely?How do you change your supervision to ensure
patients get safe, effective, patient-centered care?What did you learn observing that will change how
you supervise going forward?
REMEMBER: SUPERVISION ALSO FOR FEEDBACK
![Page 8: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/8.jpg)
8
Entrustment
“A practitioner has demonstrated the necessary knowledge, skills, and attitudes to be trusted to independently perform this activity.”
Ten Cate O, Scheele F. Acad Med 2007;82:542-7
![Page 9: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/9.jpg)
9
Problems with Performance Assessment
Poor accuracy
Focus on different aspects of clinical performance
Differing expectations about levels of acceptable clinical performance
Rating errors Halo effect/ “Horn” effectLeniency/stringency effect
![Page 10: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/10.jpg)
10
Factors That May Impact Ratings
Minimal impact of demographicsAge, gender, clinical and teaching
experienceFaculty’s own clinical skills may matter
Faculty with higher history and patient satisfaction performance scores provide more stringent ratings.
Kogan JR. et al. Acad Med. 2010;85(10 Suppl):S25-8
![Page 11: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/11.jpg)
11
Factors Influencing Faculty RatingsDifferent frameworks for judgments/ratings
Self-as-reference (predominant) Trainee levelAbsolute standardPracticing physicians
![Page 12: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/12.jpg)
12
Faculty OSCE Clinical Skills
Competency Mean (SD) Range Generaliz-ability
History Taking 65.5% (9.6%) 34% - 79% 0.80
Physical Exam 78.9% (13.6%)
36% - 100% 0.52
Counseling 77.1% (7.8%) 60% - 93% 0.33
Patient Satisfaction1
5.62 (0.48) 4.43 – 6.63 0.60
1On 7-point scale
Kogan JR. et al. Acad Med. 2010;85(10 Suppl):S25-8
N=44
![Page 13: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/13.jpg)
13
Other Factors Influencing Ratings
Factors external to resident performanceEncounter complexity Resident characteristicsInstitutional culture
Emotional impact of constructive feedback
Role of inference
![Page 14: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/14.jpg)
14
Definition of Inference
a. The act or process of deriving logical conclusions from premises known or assumed to be true.b. The act of reasoning from factual knowledge or evidence.
![Page 15: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/15.jpg)
15
Affix meaning
Concrete data
(resident actions)
ConclusionsAssumptions
1.
2.
3.
![Page 16: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/16.jpg)
16
The Problem with InferenceInferences are not recognized
Inferences are rarely validated for accuracy
Inferences can be wrong
![Page 17: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/17.jpg)
17
Types of Inference about ResidentsSkills
Knowledge Competence Work-ethic
Prior experiences Familiarity with scenario
Feelings Comfort Confidence Intentions Ownership
PersonalityCulture
![Page 18: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/18.jpg)
18
High Level Inference
![Page 19: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/19.jpg)
19
Direct Observation: A Conceptual Model
Kogan JR, et al. Med Educ. 2011
![Page 20: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/20.jpg)
20
International Comparative Work Yeates (UK)
Differential salience Criterion uncertainty Information integration
Govaerts (Netherlands) Use of task-specific and person schemas
Substantial inference in person schema Rater idiosyncrasy
Gingerich (Canada) Impact of social models: clusters; person; labels
![Page 21: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/21.jpg)
21
Achieving Accurate, Reliable Ratings
Form is not the magic bullet
Assessment requires faculty trainingSimilar frameworksAgreed upon levels of competenceMove to criterion referenced assessment
![Page 22: Direct Observation of Clinical Skills During Patient Care NEW INSIGHTS – REYNOLDS MEETING 2012 Direct Observation Team: J. Kogan, L. Conforti, W. Iobst,](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56649eca5503460f94bd83af/html5/thumbnails/22.jpg)
22
Questions