Promoting the Development of Clinical Skills throughout the Continuum of Medical Education

University of North Carolina – Chapel Hill School of MedicineNovember 9, 2011

Ann C. Jobe, MD,MSNExecutive Director

Clinical Skills Evaluation Collaboration (CSEC)

Clinical Skills in Practice

• The physician-patient encounter is central to the identity of physicians in the US

• Clinical skills of trainees and young physicians have been described as deficient since at least the 1970’s

• Good evidence supports the diagnostic and therapeutic value of the clinical encounter but…

• …..Technology, fragmented care, reimbursement, and practice culture affect the clinical encounter

Weiner,A. & Nathonson M; JAMA 1976; 236:852-855Verghese, A et al; Annals Int Med 2011;155:550-553

Clinical Skills in Practice

• The clinical encounter is often buried in process measures, such as HEDIS or other guidelines

• The ritual value of the clinical encounter is important, and must be balanced by its documented utility

• The environment determines most of what and how trainees learn about the clinical examination

Weiner,A. & Nathonson M; JAMA 1976; 236:852-855Verghese, A et al; Annals Int Med 2011;155:550-553

COMMUNICATION

• The essence of the patient-physician relationship

• Includes communicating verbally, non-verbally, as well as actions and interactions during a physical examination

Communication

• It is all about COMMUNICATING with patients and families and health professionals

• It is all about improving communication to improve the quality and safety of health care

Why Assess Communication Skills?

• Essential physician competency • (LCME, ACGME, ABMS, USMLE)

• Clinical outcomes require effective

communication

• Public expectations: need for more

information and supportive interactions.

• Quality measures now incorporate

patient-centeredness

Patient-Centered Communication

• Exploring the patient’s illness experience

• Understanding the patient as a whole person

• Picking up on patient cues

• Involvement of the patient in problem definition• Involvement of the patient in decision-making

• (now >50% expect such involvement)

• Finding common ground regarding management• Enhancing the doctor/patient relationship by

being responsive to the patient IOM,2001; Street,2008

Communication Skills

• Prospective study of 80 medical outpatients with new or previously undiagnosed conditions

• Internists asked to list their differential diagnoses and to estimate their confidence in each diagnostic possibility • after the history, • after the physical examination, and • after the laboratory investigation.

Communication Skills

• In 61 of 80 cases (76%), the leading diagnosis after taking the history agreed with the diagnosis accepted at the time the record was reviewed• The physical examination led to the diagnosis in 10 patients (12%)• The laboratory investigation led to the diagnosis in 9 patients

(11%)

• These data support the concept that most diagnoses are made from the medical history

Communication Skills

• Authors suggest that more time should be devoted to improving history-taking skills during clinical training.

Peterson MC, Holbrook JH, Hales D, Smith NL, Staker LV: Contributions of the history, physical examination, and laboratory investigation in making medical diagnoses.

West J Med 1992 Feb; 156:163-165

Communication Skills

• Numerous publications confirm that poor skills in patient communication are associated with:• Lower levels of patient

satisfaction• Higher rates of complaints• Increased risk of

malpractice claims• Poorer health outcomes

High level skills in “bedside medicine” – “clinical skills” • Ability to elicit a patient’s

story/history• Correct use of evidence-based

PE maneuvers in a focused manner based on history

• Ability to synthesize information gathered

• Ability to communicate and negotiate plans for management

are the cornerstone of patient safety and quality of care

Why Does It Matter?

• Initiatives focused on improving clinical skills, especially communication – through teaching and assessment - will be most successful in improving the quality and outcomes of care provided by health professionals

Comprehensive Program

• Overarching Competencies and Objectives

• Map for addressing teaching and assessing throughout the continuum of education• Course content• Assessment methodologies

AAMC Recommendations ForClinical Skills Curricula For Undergraduate

Medical Education(2008)

• Professionalism• The ability to understand the nature of, and demonstrate

professional and ethical behavior in, the act of medical care.

• Patient Engagement and Communication Skills• The ability to engage and communicate with a patient, develop a

student-patient relationship, and communicate with others in the professional setting

• Biomedical Knowledge Application Skills• The ability to apply scientific knowledge and method to clinical

problem solving.

AAMC Recommendations ForClinical Skills Curricula For Undergraduate

Medical Education(2008)

• History Taking• The ability to take a clinical history, both focused and

comprehensive.

• Patient Examination• The ability to perform a mental and physical examination

• Clinical Testing• The ability to select, justify and interpret selected clinical tests and

imaging

• Clinical Procedures• The ability to understand and perform a variety of basic clinical

procedures

AAMC Recommendations ForClinical Skills Curricula For Undergraduate

Medical Education(2008)

• Diagnosis• The ability to diagnose and explain clinical problems in terms of

pathogenesis, to develop basic differential diagnosis, andto learn and demonstrate clinical reasoning and problem identification.

• Clinical Information Management• The ability to record, present, research, critique and manage

clinical information

• Clinical Intervention• The ability to understand and select clinical interventions in the

natural history of disease, including basic preventive, curative and palliative strategies

AAMC Recommendations ForClinical Skills Curricula For Undergraduate

Medical Education(2008)

• Prognosis• The ability to understand and formulate a prognosis about the

future events of an individual’s health and illness basedupon an understanding of the patient, the natural history of disease, and upon known intervention alternatives.

• Personalizing Clinical Care• The ability to provide clinical care within the practical context of a

patient’s age, gender, personal preferences, family, healthliteracy, culture, religious perspective, and their economic circumstances

Core Competencies & Assessment

• Patient Care/Clinical Skills• Students must be able

to provide care that is compassionate, appropriate, and effective for treating health problems and promoting health

Core Competencies & Assessment

• Interpersonal & Communication Skills• Students must

demonstrate interpersonal and communication skills that facilitate effective interactions with patients and their families and other health professionals

Developing a Comprehensive Program

• Types of assessments

• Examinees

• Timing of assessments

• Types of assessments• Formative

• Designed to provide feedback to facilitate acquisition of new skills or improvement of performance

• Part of continuous professional development• Part of performance and quality improvement

• Types of assessments• Summative

• “High stakes”• Associated with an important decision – like

graduation, licensure, certification or credentialing• Utilized to distinguish between those who are

competent and those who are not

• Types of assessments• “Snapshot”

• One time assessment

• Longitudinal• Repeated over various periods of time

• Timing of assessments• At planned intervals for promotion decisions• Ongoing for continuous professional

development and/or performance improvement• One-time “snapshot” for initial licensure • Repeat assessment for license renewal• For credentialing or granting privileges• Review for re-entry into practice

Program Elements

• Depend on PURPOSE of the assessment

and• LEVEL of the

examinee

Assessing Skills and Performance

• What is included in an assessment of skills and performance?

• What are some of the assessment methods and how are they assembled?

• How do the methods perform against the criteria for good assessment?

Miller’s Pyramid for Assessing Clinical

Competence

Does

Knows

Shows How

Knows How

Knowledge

Performance

Competence

Action

Kirkpatrick Criteria

4. ResultsChange in organizational practice

Benefits to patients/clients

5. BehaviorTransfer learning to workplace

Learners apply new knowledge and skills

6. LearningChange attitudes/perceptions

Change knowledge/skills

7. ReactionCustomer satisfaction related to participation in

educational activities

Simulation

• Simulation• Real patients are

replaced with realistic but artificial experiences

• Trainee interacts with the re-creations

• Judgments are made about their performance

Simulation

• Methods can be divided according to how faithful they are to reality• Intermediate fidelity

• Task specific models

• Instructor driven models

• High fidelity • Virtual reality

• Standardized patients (SPs)

Method: Task Specific Models

• Designed around a specific task• Venipuncture model• Animal cadavers

• Usually not automated• Relatively inexpensive

Method: Instructor Driven Models

• Physical representation

• Responses driven by an instructor

• Little feedback

• Moderate cost

Method: Virtual Reality Simulators

• Simple physical representation

• Sensing device that informs computer of user actions

• Computer models realistic reactions• 3D imaging• Haptics

Method: Standardized Patients

• Individuals trained to portray a patient• Scripted and standardized

• USMLE Step 2 CS example• Integrated Clinical Encounter

• Data gathering• SP completing checklists

• Written communication• Doctor rating a patient note

• Communication & Interpersonal skills

• SP Rating

• Spoken English• SP Rating

Ideal Assessment of Communication Skills

• Evidence-based construct• Assessment instrument consists of observable

behaviors• Realistic stimuli

• SPs trained to use instrument reliably

• Appropriate scoring decisions

Putting it Together: Objective Structured Clinical Examination

(OSCE)• Multiple stations

• Each focused on a specific aspect of competence

• Stations might include• Manikins

• SPs

• ECG or X-ray interpretation

• Heart sounds

• Animal cadavers• Anastomosis• Laparoscopic vessel ligation

• Simulators

“In a way the OSCE is not an examination method; rather it is an examination format or framework into which many different types of test methods can be incorporated”

Ian Hart, 2001

Putting it Together: OSCE

• Stations are usually short: 10-15 minutes

• Test is composed of 8-25 stations• Round-robin format

• At a bell, examinees rotate to next station

• Can accommodate as many examinees as stations

• Total score is calculated across all stations

Work-based Methods

• Work-based assessment

• Real patient encounters

• Trainees are observed

• Judgments are made about their performance

“When your work speaks for itself, don't interrupt.”

Henry Kaiser

Work-based Assessment

• Foundation Programme (in the UK)• Two-year program

• Bridge between medical school and advanced training

• Series of clinical placements

• Assessment Purpose• Determine fitness to progress to

the next level• Identify trainees in difficulty• Provide feedback• Establish accountability

• Three methods • Mini-Clinical Evaluation

Exercise (mCEX) • Directly Observed

Procedures (DOPs)• Case-Based Discussion

(CbD)

Mini-Clinical Evaluation Exercise (mCEX)

• Process• List of patient problems

• Trainee picks a patient

• Assessor observes the encounter

• Focused clinical task

• Assessor rates:• Hx, PE, Communication, Clinical

Judgment, Professionalism, Organization/Efficiency

• Assessor provides feedback

• Takes 15-20 minutes

Directly Observed Procedures (DOPs)

• Process• List of procedures• Trainee picks a patient• Assessor observes the

encounter • Procedure

• Assessor rates:• Preparation, Sedation,

Asepsis, Technical skill, etc.

• Assessor provides feedback

• Takes 15-20 minutes

Case-Based Discussion (CbD)

• Process• List of patient problems• Trainee picks 2 case records

• Assessor selects one

• Discussion centered on the trainee’s notes

• Assessor rates:• Diagnosis, Treatment,

Planning, Professionalism, etc.

• Assessor provides feedback

• Takes 15-20 minutes

Putting it Together: Work-based Assessment

• An OSCE “on the hoof”• Multiple encounters are

needed• Captured as feasible

during clinical training

• Multiple examiners are needed

• Encounters can be made to conform loosely to a problem list

• Ongoing, longitudinal assessments

Criteria for Judging an Assessment

• How do simulation and work-based assessment perform against the criteria?• Validity • Reliability• Equivalence• Educational effect• Opportunity for feedback• Feasibility

Validity

• What is validity?• Degree to which the

inferences based on scores are correct

• Does the test measure what it is supposed to measure?

• Simulation • Good content coverage

• Rare conditions• Errors cause no harm

• Good fidelity

• Work-based methods• Excellent content

coverage• Includes difficult to

simulate conditions

• High fidelity

Reliability

• What is reliability?• If an assessment process

is repeated with the same trainees, they should get the same scores

• Physician performance varies considerably from patient to patient• The trainee must be

observed with several patients

• Assessors differ in stringency• The trainee must be

evaluated by different examiners

Equivalence

• What is equivalence?• To compare examinees

they must have taken assessments that are equal in difficulty

• Fairness • Comparable meaning

• Simulation• Different examinees can

be given the same items• Security

• Statistical techniques help with different versions

• Work-based methods• Equivalence is a

problem that can be mitigated but not eliminated

Educational Effect

“Students respect what you inspect.”

• Both simulation and work-based methods signal the importance of working with patients• Drives learning

Opportunity for Feedback

• Feedback is critical to learning • General education (Hattie,

1999)• Meta-analysis of 12 meta-

analyses

• Feedback is among the largest influences on achievement

• Medical education (Veloski et al., 2006)

• Feedback alone is effective in 71% of studies

• Simulation• Amount of feedback varies

by method• Depends on deployment• Lower for instructor driven

methods • Higher for model driven

methods

• Work-based methods• Trainees rarely observed• Provides an excellent

opportunity for feedback following observation

Feasibility

• There are significant resource constraints in most educational programs

• Simulation• Purchase, maintenance,

logistics

• Case development

• SP/Observer training

• Work-based methods• Faculty development

• Logistics

Summary: Assessment of Skills and Performance

• Trainees must ‘show how’ • Simulation

• Can produce equivalent scores

• Work-based methods • Cover more patient problems• Can be more feasible

• Both methods• Require multiple patients and

examiners • Have positive educational

effects• Provide opportunities for

feedback

Finding Opportunities

• Seeking out the “best practices” already in place across the organization

• Disseminating and seeding what is working to other areas

• Finding ways to maximize synergy of work already in place

Opportunities Along the Continuum

• Assessment of team member performance

Opportunities Along the Continuum

• Assessment of outcomes of a team’s performance

Opportunities Along the Continuum

• Assessment of individual team members – using “standardized team members”

Opportunities Along the Continuum

• Assessment of teams composed of members of several health professions

Opportunities Along the Continuum

• Standardized Patient assessments/ OSCEs & simulations for:• Incoming residents• Residents moving into

supervisory roles• Residents at completion of

residency• New medical staff –

credentialing review and privileging

• Individuals who are re-entering practice

Opportunities Along the Continuum

• “Secret Shoppers” -standardized patients in clinical settings assessing clinical skills of:• Residents• Faculty• New medical staff –

credentialing review and privileging

• Individuals upon re-entry into practice

Most Important Consideration

• A Comprehensive Program based on • Well defined Purpose and Goals• Overarching Competencies and

Objectives• A detailed “Map” that covers the

timing and methodologies of assessments across the continuum

• Focused efforts on gaps in teaching and assessment

• A well thought out evaluation of the program

• Providing data and evidence supporting the benefit to patients and improvement in care

Why Does It Matter?

• Initiatives focused on improving clinical skills, especially communication – through teaching and assessment - will be most successful in improving the quality and outcomes of care provided by health professionals

THANK YOU

Let us continue on the journey together –

improving how we care for our

patients