Using Resident Perceptions to Using Resident Perceptions to Improve Educational Quality and Improve Educational Quality and

Accountability Accountability Conference Session:01Conference Session:01

Alice Edler, MD, MPH, MA (Educ) – Faculty Fellow Alice Edler, MD, MPH, MA (Educ) – Faculty Fellow Nancy Piro, PhD, Program Manager/Education SpecialistNancy Piro, PhD, Program Manager/Education Specialist

Ann M. Dohn, MA – Director, GME Ann M. Dohn, MA – Director, GME Designated Institutional Official (DIO) &Designated Institutional Official (DIO) &

Bardia Behravesh, EdD, Bardia Behravesh, EdD, Program Manager/Education SpecialistProgram Manager/Education Specialist

Department of Graduate Medical EducationDepartment of Graduate Medical EducationStanford University Medical CenterStanford University Medical Center

Session ObjectivesSession Objectives

At the end of this session, you will be able At the end of this session, you will be able to…to…1.1. Understand how we tap into trainees’ current Understand how we tap into trainees’ current

perception of the quality of GME at Stanfordperception of the quality of GME at Stanford

2.2. Understand resident perceptions of factors Understand resident perceptions of factors which influence the quality of their which influence the quality of their postgraduate medical educationpostgraduate medical education

3.3. Operationalize the use of survey data for Operationalize the use of survey data for program quality improvementprogram quality improvement

4.4. Be introduced to the use of factor analysis to Be introduced to the use of factor analysis to evaluate survey dataevaluate survey data

Annual GME HouseStaff SurveyAnnual GME HouseStaff Survey“Basics”“Basics”

Annual survey to all HouseStaff

Designed to capture HouseStaff perceptions

Questions are primarily linked to Common Program Requirements

Goal: Early warning and continual program improvement efforts

Setting the StageSetting the Stage

82 Programs (and 82 Program Directors)82 Programs (and 82 Program Directors)

1100 Trainees1100 Trainees

5 Major Affiliate Training Sites5 Major Affiliate Training Sites

Background on our GME Background on our GME HouseStaff SurveyHouseStaff Survey

Do we really want to know what they think Do we really want to know what they think of us?of us?

Why do we survey? We already have the Why do we survey? We already have the ACGME survey and annual program ACGME survey and annual program evaluations.evaluations.

What kind of questions do we ask?What kind of questions do we ask?

What do we do with the responses?What do we do with the responses?

Background on our GME Background on our GME HouseStaff SurveyHouseStaff Survey

Do we really want to know what they think Do we really want to know what they think of us?of us?

YES!!!YES!!!

Background on our GME Background on our GME HouseStaff SurveyHouseStaff Survey

Why do we survey?Why do we survey?– Want residents to be able to speak freely and Want residents to be able to speak freely and

confidentially confidentially – Want to know what is going on across multiple Want to know what is going on across multiple

geographic locations and diverse programsgeographic locations and diverse programs– Surveys- instant and easy access (more Surveys- instant and easy access (more

efficient than Town Halls)efficient than Town Halls)– Conserves Residents’ and GME Staff timeConserves Residents’ and GME Staff time– Tap into perceptions not queried in program Tap into perceptions not queried in program

evaluations or the ACGME surveyevaluations or the ACGME survey

Background on our GME Background on our GME HouseStaff SurveyHouseStaff Survey

What kind of questions do we ask?What kind of questions do we ask?– Information about the ResidentsInformation about the Residents– Overall ExperienceOverall Experience– Program ExperienceProgram Experience– Program FacultyProgram Faculty– Evaluation and Feedback Evaluation and Feedback – Training EnvironmentTraining Environment– Personal WellnessPersonal Wellness– Quality ImprovementQuality Improvement– Communication and Patient PerceptionsCommunication and Patient Perceptions

Background on our GME Background on our GME HouseStaff SurveyHouseStaff Survey

How many questions?How many questions?

Quantitative and Open Ended / CommentsQuantitative and Open Ended / Comments

How do we scale responses?How do we scale responses?

How do we maintain confidentiality?How do we maintain confidentiality?

How do we encourage residents to How do we encourage residents to respond?respond?

What do we do with the responses?What do we do with the responses?– Explore, analyze, and compare the data Explore, analyze, and compare the data

(trend by year, program vs institution).(trend by year, program vs institution).– Prepare summary reports for each programPrepare summary reports for each program

Upload for programs to use the aggregated data Upload for programs to use the aggregated data for their Annual Program Reviews (APRs)for their Annual Program Reviews (APRs)

Background on our GME Background on our GME HouseStaff SurveyHouseStaff Survey

Sample Summary ReportSample Summary Report

Background on our GME Background on our GME HouseStaff SurveyHouseStaff Survey

What do we do with the responses?What do we do with the responses?– Quality ImprovementQuality Improvement– Organizational and Programmatic ChangeOrganizational and Programmatic Change– Data for “Report Cards”Data for “Report Cards”– Early Warning SystemEarly Warning System– Chart ProgressChart Progress

Lessons LearnedLessons Learned

““Charming”Charming”

Non-digital HVAC units (cold residents)Non-digital HVAC units (cold residents)

5 Year Accreditation Cycle vs Extremely 5 Year Accreditation Cycle vs Extremely Unhappy ResidentsUnhappy Residents

Data Mining Data Mining

Exploring the Data for New Ideas…Exploring the Data for New Ideas…– Can residents tell us what elements of Can residents tell us what elements of

education are important to them?education are important to them?

Education Quality Education Quality

Teaching

Curriculum

Educational Leadership

Theories of Educational Quality Theories of Educational Quality

SkeffSkeff

Harris Harris

Cooke, Irby Cooke, Irby

O’Sullivan, Edler O’Sullivan, Edler

ACGME ACGME

? The Post Graduate Learner ?? The Post Graduate Learner ?

Why is the Adult Learner Why is the Adult Learner Important? Important?

All learners are importantAll learners are important– They are “the starting point, They are “the starting point,

the center and the end” of educational the center and the end” of educational activities (Dewey, 1902)activities (Dewey, 1902)

Adults have a lifetime of past educational Adults have a lifetime of past educational experiencesexperiences

Adults are active participants in their own Adults are active participants in their own teaching and learningteaching and learning

Adults are intrinsically motivated Adults are intrinsically motivated

Purpose of Our InvestigationPurpose of Our Investigation

To identify elements of the educational To identify elements of the educational milieu that adult learners, in this case, post milieu that adult learners, in this case, post graduate medical trainees, believe to graduate medical trainees, believe to contribute significantly to the quality of contribute significantly to the quality of their educational experiences.their educational experiences.

Overall Goals Overall Goals

To take these perceptions into account to To take these perceptions into account to help increase ownership of learning, help increase ownership of learning, promote stronger faculty/trainee promote stronger faculty/trainee relationships, and guide educational relationships, and guide educational improvement. improvement.

Methods Methods

Survey model, with questions from:Survey model, with questions from:– Literature reviews Literature reviews – Institutionally relevant themes and accreditation Institutionally relevant themes and accreditation

topics topics

Residents not fellows Residents not fellows 18 Core Residencies 18 Core Residencies AnonymousAnonymousFactor Analysis with Principal Component Factor Analysis with Principal Component AnalysisAnalysis

Core Programs StudiedCore Programs Studied

Anesthesia Anesthesia DermatologyDermatologyEmergency MedicineEmergency MedicineInternal MedicineInternal MedicineNeurosurgeryNeurosurgeryObstetrics and Obstetrics and GynecologyGynecologyOphthalmologyOphthalmologyOrthopedic Surgery Orthopedic Surgery OtolaryngologyOtolaryngology

PathologyPathologyPediatricsPediatricsPhysical Medicine & Physical Medicine & RehabRehabPlastic SurgeryPlastic SurgeryPsychiatry Psychiatry Radiation OncologyRadiation OncologyRadiology Radiology Surgery (General) Surgery (General) UrologyUrology

32 educationally oriented questions 32 educationally oriented questions

were analyzed using :were analyzed using : Bartlett's Test for adequacy of partial Bartlett's Test for adequacy of partial correlations correlations

Determination of Eigen values Determination of Eigen values – All items with Eigen values >0.5 were entered All items with Eigen values >0.5 were entered

into the factor analysis and then rotated via a into the factor analysis and then rotated via a Varimax rotation.Varimax rotation.

Response Rates and Response Rates and Demographic Data Results Demographic Data Results

88.1% Response Rate ( n = 404)88.1% Response Rate ( n = 404)M:F Ratio = 46% : 44%M:F Ratio = 46% : 44%Missing data 13% Missing data 13% PGYPGY– 1 1 17%17%– 22 19%19%– 33 17%17%– 44 19%19%– 55 13%13%– 66 6%6%

Statistical Results Statistical Results

Significant Bartlett's Test Significant Bartlett's Test

15 items with Eigen values >0.515 items with Eigen values >0.5

All factors positively correlated All factors positively correlated

3 principal components identified on Spree 3 principal components identified on Spree plot with 57% of the variance explained plot with 57% of the variance explained

αα = 0.889 = 0.889

Results Results

The spree plot suggested the presence of The spree plot suggested the presence of three principal factors. three principal factors.

Fifty-seven percent of the variance was Fifty-seven percent of the variance was explained by the three factors in this explained by the three factors in this model. model.

Spree Plot Spree Plot

Results of the Factor AnalysisResults of the Factor Analysis

Three factors Three factors emerged from the initial emerged from the initial principal component analysis. principal component analysis. – Extraction was completed at ten iterations.Extraction was completed at ten iterations.– The resultant factor loadings suggested factor The resultant factor loadings suggested factor

one describing one describing curriculum qualitycurriculum quality, factor two , factor two describing describing teaching quality teaching quality and factor three and factor three describing describing quality in educational leadershipquality in educational leadership. .

Rotated Component Matrix

  Component

  1 2 3 4 5

1. In my program I have received sufficient education in responding to cultural differences with patients and colleagues 0.836 0.104 0.143 0.118 0.005

2.In my program I have received sufficient education in applying ethical principles to the practice of medicine 0.809 0.111 0.25 0.02 0.114

3. In my program I have received sufficient education in communicating with non English speaking patients and their families 0.772 0.148 0.022 0.031 0.007

4.In my program I have received sufficient education in improving my communication skills 0.771 0.14 0.059 0.116 0.006

5. In my program I have received sufficient education in improving my teaching skills 0.744 0.14 0.261 0.049 0.071

6. In my program I have received sufficient education in reducing costs while working in the hospital/lab 0.645 0.292 0.041 0.17 0.009

           

7.What % of your faculty are effective teachers 0.189 0.791 0.212 0.025 0.074

8.How many of your faculty show a strong interest in your education 0.154 0.772 0.248 0.125 0.055

9. What % of your faculty regularly attend journal club 0.15 0.764 0.067 0.138 0.022

10. How many of your faculty regularly participate in discussions and conferences 0.168 0.759 0.138 0.119 0.117

11. What % of your faculty spend an adequate amount of time supervising residents 0.179 0.739 0.116 0.007 0.078

           

12. My program director considers my opinion when making decisions 0.12 0.053 0.776 0.002 0.08

13. In my program we are encouraged to ask questions 0.303 0.255 0.65 0.085 0.085

14. My program is organized to meet my educational needs 0.222 0.346 0.58 0.139 0.139

15. In my program I am treated with respect 0.025 0.395 0.58 0.089 0.089

Component 1 1. In my program I have received sufficient education in responding to cultural differences with patients and colleagues 0.836 0.104 0.143 0.118 0.005

2.In my program I have received sufficient education in applying ethical principles to the practice of medicine 0.809 0.111 0.25 0.02 0.114

3. In my program I have received sufficient education in communicating with non English speaking patients and their families 0.772 0.148 0.022 0.031 0.007

4.In my program I have received sufficient education in improving my communication skills 0.771 0.14 0.059 0.116 0.006

5. In my program I have received sufficient education in improving my teaching skills 0.744 0.14 0.261 0.049 0.071

6. In my program I have received sufficient education in reducing costs while working in the hospital/lab 0.645 0.292 0.041 0.17 0.009

Component 2

7.What % of your faculty are effective teachers 0.189 0.791 0.212 0.025 0.074

8.How many of your faculty show a strong interest in your education 0.154 0.772 0.248 0.125 0.055

9. What % of your faculty regularly attend journal club 0.15 0.764 0.067 0.138 0.022

10. How many of your faculty regularly participate in discussions and conferences 0.168 0.759 0.138 0.119 0.117

11. What % of your faculty spend an adequate amount of time supervising residents 0.179 0.739 0.116 0.007 0.078

Component 3

12. My program director considers my opinion when making decisions 0.12 0.053 0.776 0.002 0.08

13. In my program we are encouraged to ask questions 0.303 0.255 0.65 0.085 0.085

14. My program is organized to meet my educational needs 0.222 0.346 0.58 0.139 0.139

15. In my program I am treated with respect 0.025 0.395 0.58 0.089 0.089

Summary of Findings Summary of Findings

Postgraduate medical trainees can identify Postgraduate medical trainees can identify 3 key areas of educational quality.3 key areas of educational quality.

These can be understood as the These can be understood as the constructs of constructs of curriculum, teaching, and curriculum, teaching, and educational leadershipeducational leadership..

Our construct findings were validated with Our construct findings were validated with answers to questions on the ACGME answers to questions on the ACGME survey.survey.

Curriculum ConstructCurriculum Construct

Our findings validated the students’ Our findings validated the students’ understanding of curriculumunderstanding of curriculum– One facet among the highest rated was One facet among the highest rated was

medical ethics instruction medical ethics instruction We had just that year added a mandatory on line We had just that year added a mandatory on line module for medical ethics education to overcome module for medical ethics education to overcome some departmental insufficiencies in this area. some departmental insufficiencies in this area.

– The lowest rated facet was Systems-Based The lowest rated facet was Systems-Based Practice (SBP) Practice (SBP)

This was also a low ranking result in the ACGME This was also a low ranking result in the ACGME survey and our own program directors’ needs survey and our own program directors’ needs assessment assessment

So What?So What?

It is clear that residents can identify It is clear that residents can identify instructional elements and rate their instructional elements and rate their importance and inclusion in their importance and inclusion in their curriculum.curriculum.

We have also demonstrated that learners We have also demonstrated that learners who are reared in a competency based who are reared in a competency based era, regard these competencies as critical era, regard these competencies as critical to their curriculum. to their curriculum.

Teaching Construct Teaching Construct

Our residents identified 3 elements, Our residents identified 3 elements, teaching time both clinically and teaching time both clinically and didactically and, instructor enthusiasm in didactically and, instructor enthusiasm in teaching . teaching .

We believe this demonstrated that the We believe this demonstrated that the teacher and the learner are collaborators teacher and the learner are collaborators in the educational event.in the educational event.

Teaching Construct Teaching Construct

Irby, in his recommendations for medical Irby, in his recommendations for medical teaching in the post Flexner era, teaching in the post Flexner era, suggested we need to suggested we need to “promote “promote relationships with faculty who relationships with faculty who simultaneously support learners…to simultaneously support learners…to create collaborative learning create collaborative learning environments” environments” **

* Irby, DM, Cooke M, Obrien B, Calls for Reform of Medical Education by the Carnegie Foundation for the Advancement of Teaching: 1910-2010

How does this compare How does this compare with the literature?with the literature?

Qualitative studies by educational Qualitative studies by educational researchers list rapport and communication researchers list rapport and communication as the key to excellent teaching. Our findings as the key to excellent teaching. Our findings include the presence of the teacher at formal include the presence of the teacher at formal didactic and conference discussions as a didactic and conference discussions as a portion of this key element.portion of this key element.– Perhaps these formal presentations provide the Perhaps these formal presentations provide the

learner, especially the novice learner, with time to learner, especially the novice learner, with time to digest and reflect on the materials better than digest and reflect on the materials better than “on the fly” teaching during rounds. “on the fly” teaching during rounds.

Educational Leadership Construct

Emphasized – Joint decision making in education– Respectful atmosphere

How does this compare How does this compare with the literature?with the literature?

Our findings, like that of Irby et al. suggested Our findings, like that of Irby et al. suggested that the apprenticeship model of medical that the apprenticeship model of medical teaching need not be abandoned but teaching need not be abandoned but improved with curricular, instructional and improved with curricular, instructional and leadership enhancements. leadership enhancements. Quality GME education is a complex multi-Quality GME education is a complex multi-dimensional endeavor. dimensional endeavor. – Not only must curriculum and instructional Not only must curriculum and instructional

methods be taken into account, but also the methods be taken into account, but also the guidance and quality of leadership. guidance and quality of leadership.

– GME adds an additional layer of complexity.GME adds an additional layer of complexity.

In the professions teacher and learners In the professions teacher and learners must come together over the three must come together over the three

elements: curriculum, instruction and elements: curriculum, instruction and leadership, but an added unique fourth leadership, but an added unique fourth

consideration is omnipresent, consideration is omnipresent, the patientthe patient. .

Overall Conclusions Overall Conclusions

Survey data can be extremely valuable if Survey data can be extremely valuable if you are willing to share the data with your you are willing to share the data with your stakeholders – in a user friendly waystakeholders – in a user friendly way

Be prepared for some surprisesBe prepared for some surprises

Be prepared to act on your findings – Be prepared to act on your findings – develop action plansdevelop action plans

Don’t be afraid to think outside the box Don’t be afraid to think outside the box

Specific Conclusions Specific Conclusions

Quality education is complex and Quality education is complex and multidimensional. multidimensional.

Curriculum and instruction are only 2 key Curriculum and instruction are only 2 key elements, the third is educational leadership.elements, the third is educational leadership.

Our findings have corroborated with findings Our findings have corroborated with findings in the general educational literature. in the general educational literature.

Professional education adds to the Professional education adds to the complexity by a unique, omnipresent 4th complexity by a unique, omnipresent 4th layer: layer: The physician-patient relationThe physician-patient relation..