teaching quality improvement and patient safety to ... · and canmeds10,11 competency frameworks...

Patient Safety

Teaching Quality Improvement and PatientSafety to Trainees: A Systematic ReviewBrian M. Wong, MD, Edward E. Etchells, MD, MSc, Ayelet Kuper, MD, DPhil,Wendy Levinson, MD, and Kaveh G. Shojania, MD

Abstract

PurposeTo systematically review published qualityimprovement (QI) and patient safety (PS)curricula for medical students and/orresidents to (1) determine educationalcontent and teaching methods, (2) assesslearning outcomes achieved, and (3)identify factors promoting or hinderingcurricular implementation.

MethodData sources included Medline (to January2009), EMBASE, HealthSTAR, and articlebibliographies. Studies selected reportedcurricula outlining specific educationalcontent and teaching format. For articleswith an evaluative component, the authorsabstracted methodological features, suchas study design. For all articles, they

conducted a thematic analysis toidentify factors influencing successfulimplementation of the included curricula.

ResultsOf 41 curricula that met the authors’criteria, 14 targeted medical students, 24targeted residents, and 3 targeted both.Common educational content includedcontinuous QI, root cause analysis, andsystems thinking. Among 27 reports thatincluded an evaluation, curricula weregenerally well accepted. Most curriculademonstrated improved knowledge.Thirteen studies (32%) successfullyimplemented local changes in caredelivery, and seven (17%) significantlyimproved target processes of care. Factorsthat affected the successful curricular

implementation included havingsufficient numbers of faculty familiarwith QI and PS content, addressingcompeting educational demands, andensuring learners’ buy-in andenthusiasm. Participants in somecurricula also commented ondiscrepancies between curricularmaterial and local institutional practiceor culture.

ConclusionsQI and PS curricula that target traineesusually improve learners’ knowledge andfrequently result in changes in clinicalprocesses. However, successfullyimplementing such curricula requiresattention to a number of learner, faculty,and organizational factors.

The quality improvement (QI) andpatient safety (PS) movements in recentyears have had important implications

for undergraduate and graduate medicaleducation, including focused attentionon duty hours reductions,1–4 appropriatesupervision of trainees,4,5 andcommunication during resident hand-offs.6 Furthermore, there is a growingimperative to teach QI and PS in medicaleducation.

The Association of American MedicalColleges now endorses the introductionof formal QI education across themedical education continuum, spanningundergraduate, postgraduate, andcontinuing medical education levels.7,8

Both the Accreditation Council forGraduate Medical Education (ACGME)9

and CanMEDS10,11 competencyframeworks define essential physiciancompetencies that relate to quality andsafety. These developments coincide withthe recognition that engagement in QIrepresents an emerging career path forclinicians.12

A previous systematic review of QIeducation for clinicians found that mostcurricula demonstrated improvement inlearners’ knowledge by applying soundadult learning principles.13 However,only 10 of the 39 studies targeted trainees

(of which only 2 involved medicalstudents). Given the increasingrecognition of the need to teach QI tostudents, we systematically reviewedpublished curricula in QI or PSspecifically directed at medical studentsor resident trainees. We sought to (1)describe their educational content andteaching methods used, (2) assess thelearning outcomes achieved, and (3)determine factors that promoted orlimited the successful implementationof these curricula.

Method

Literature search

We searched for relevant English-language studies from January 1, 2000 toJanuary 2009 using electronic literaturedatabases including Medline, EMBASE,and HealthSTAR. We chose January 2000

Dr. Wong is lecturer, Department of Medicine,University of Toronto, and associate scientist, Centerfor Health Services Sciences, Sunnybrook HealthSciences Center, Toronto, Ontario, Canada.

Dr. Etchells is associate professor of medicine,University of Toronto, director, Patient SafetyImprovement Research, Center for Health ServicesSciences, Sunnybrook Health Sciences Center, andassociate director, University of Toronto Center forPatient Safety, Toronto, Ontario, Canada.

Dr. Kuper is assistant professor of medicine,University of Toronto, associate scientist, Center forHealth Services Sciences, and clinician/educatorresearcher, Wilson Center for Research in Education,University Health Network/University of Toronto,Toronto, Ontario, Canada.

Dr. Levinson is professor of medicine and Sir Johnand Lady Eaton Chair of Medicine, University ofToronto, Toronto, Ontario, Canada.

Dr. Shojania is associate professor of medicine,University of Toronto, and director, University of TorontoCenter for Patient Safety, Toronto, Ontario, Canada.

Correspondence should be addressed to Dr. Wong,Department of Medicine, Sunnybrook HealthSciences Center, 2075 Bayview Avenue, Room H466,Toronto, Ontario, Canada M4N 3M5; telephone:(416) 480-6100 ext. 83709; fax: (416) 480-6777;e-mail: [email protected].

Acad Med. 2010;85:1425–1439.doi: 10.1097/ACM.0b013e3181e2d0c6

Supplemental digital content is available for thisarticle. Direct URL citations appear in the printedtext; simply type the URL address into any Webbrowser to access this content. Clickable links tothis material are provided in the HTML text andPDF of this article on the journal’s Web site(www.academicmedicine.org).

Academic Medicine, Vol. 85, No. 9 / September 2010 1425

as the cut-off of the search period inorder to capture articles describingeducational efforts that arose followingthe release of the Institute of Medicine’sreport, To Err Is Human,14 whichinitiated the current widespread interestin QI and PS.

The search strategies combined MedicalSubject Headings terms and text wordsrelated to QI and PS (e.g., medical errors,safety, quality assurance) with thoserelated to undergraduate and graduatemedical education. (We will be glad tosend interested readers our searchstrategy.) We also hand-searchedbibliographies of all included reports andrelevant review articles.

Eligibility criteria

We included any article that, at aminimum, described a curriculum thatexplicitly identified its goal as exposingmedical students or residents to conceptsin QI or PS and that outlined specificteaching methods used to achieve thiseducational goal. We identified conceptsin QI or PS by screening for descriptorsin the article’s title or text, includinggeneral terms such as qualityimprovement, systems learning, systems-based practice, patient safety, as well asspecific topics widely recognized as fallingwithin the domains of QI (i.e.,continuous QI, audit and feedback,process mapping, change management)and PS (i.e., systems thinking, root causeanalysis, human factors engineering,incident reporting, dealing with errors,safety culture, error disclosure).

Studies were excluded if they (1)primarily focused on practicing clinicians(e.g., if the intervention targetedmembers of an academic clinic, some ofwhom happened to be trainees), (2) werepredominantly QI interventions thathappened to be delivered in trainingsettings (e.g., an audit and feedbackintervention delivered in a residentclinic), or (3) described curricula inwhich QI or PS topics were included as aminor component of a larger curriculum(e.g., a single lecture on QI in a broadcurriculum on managed care).

Article review process

Two of us (B.W. and K.S.) independentlyreviewed titles and abstracts to identifyeligible articles. When in doubt, the fulltext of each article was obtained tofinalize article inclusion or exclusion. The

article screening process was followed byindependent abstraction (B.W. abstracteddata from all included articles; K.S. andE.E. carried out duplicate dataabstraction independently) using astructured data entry form.Disagreements at both the articlescreening and data abstraction stageswere resolved by consensus, involving athird reviewer if necessary.

Consistent with the Best EvidenceMedical Education (BEME) reviewprotocol,15 we extracted curriculardescriptors as well as key methodologicalfeatures for those articles that included anevaluative component. We classifiedlearning outcomes using Kirkpatrick’smodel,16 which includes impacts onlearners’ satisfaction (Level 1), changes inlearners’ attitudes (Level 2A), measuresof learners’ knowledge and skills (Level 2B),changes in learners’ behavior (Level 3),changes to clinical processes (Level 4A),and benefits to patients (Level 4B).

We supplemented the structured dataabstraction with a detailed thematicanalysis of each article’s text to identifyfactors that the authors regarded aspromoting or limiting curricularimplementation. One of us (B.W.)established a framework for key factorsthat influenced curricularimplementation based on an initialdetailed reading of all included studies aswell as existing literature on curriculardevelopment and implementation inmedical education more generally.17 Afteriterative review and modification byother investigators, we independentlyapplied the final framework to code eachstudy (B.W. coded all studies; K.S. andA.K. carried out duplicate codingindependently).

Assessment of study quality

For studies with an evaluativecomponent, we assessed the strength ofthe findings using a modified version ofthe BEME protocol.15 The BEME ratingsystem for strength of findings assigns arating of Level 1 when no clearconclusions can be drawn, Level 2 whenresults are ambiguous but exhibit a trend,Level 3 when conclusions can probablybe based on the results, Level 4 whenresults are clear and very likely to be true,and Level 5 when results are unequivocal.Although widely used, the BEMEprotocol does not include explicitfeatures to guide these judgments. We

therefore adopted the protocol to informratings of the strength of study findingsusing considerations of sample size,number of sites, study design,completeness of data, and response rate.

Analysis

We anticipated substantial heterogeneityof study design and reported outcomes,so we chose not to pursue quantitativesynthesis. We summarized educationalcontent, teaching methods, and learningoutcomes (for studies with an evaluativecomponent only) using simpledescriptive statistics. We included themesidentified by the detailed thematicanalysis if they were observed in twoindependent sources. We summarizedthese themes and highlighted keyexcerpts that illustrate these themes todescribe important factors that limited orpromoted implementation of QI and PScurricula.

All data were previously published andpublicly available. Therefore, our studydid not meet criteria for submission tothe local institutional review board forethical approval.

Results

Characteristics of included curricula

Of 953 citations identified by theelectronic search, 41 curricula meteligibility criteria (see Figure 1),18 –58 27(66%) of which provided a curriculardescription along with some form ofevaluation.18 –44 The vast majority (38;93%) of reports came from U.S. trainingprograms; of the remaining three, two(5%) came from Canada47,58 and one(2%) from the United Kingdom.37

Participating learners consisted ofmedical students in 14 studies (34%),residents in 24 (59%), and both in 3(7%). Curricula for residents primarilycame from internal medicine (58%) andfamily medicine (21%) trainingprograms. Twenty-five (61%) of thecurricula for students and residents weremandatory.

Curricular features

The curricula addressed a range of QIand PS content (see Table 1 and Table 2),but the most common topics consisted ofcontinuous QI (21 studies, 51%), rootcause analysis (17 studies, 41%), andsystems thinking (16 studies, 39%). Mostcurricula combined didactic andexperiential learning; detailed case

Patient Safety

Academic Medicine, Vol. 85, No. 9 / September 20101426

discussions and Web-based learning wereless frequently used.

Among curricula targeting medicalstudents, seven targeted preclinicalmedical students and seven targetedclinical medical students. Some curriculawere integrated into one course orrotation, whereas others were delivered asstand-alone sessions. The majority ofthese curricula generally involved fewerthan 10 contact hours, often consisting ofa single session. Five curricula involvedmedical students in QI or PS projects.

QI and PS curricula targeting residentswere similarly brief (approximately 10contact hours) but more often involvedmultiple encounters (i.e., 2–5).Approximately half of the curriculaincorporated their content into existingcore rotations; others occurred as stand-alone sessions or elective rotations. Allcurricula for residents were delivered inclinical settings (e.g., ambulatory clinic orinpatient teaching unit). Residentsparticipated in QI or PS projects in 14(58%) of the curricula.

Study designs and outcomes

Table 2 summarize the outcomes,designs, and main results of the 27studies that included an evaluativecomponent (a more detailed summary ofthe study outcomes is provided in theAppendix, which can be accessed athttp://links.lww.com/ACADMED/A20).The most common design was a simplebefore–after comparison (11; 42%). Five(19%)28,29,34,35,38 evaluations included acontemporaneous control, and two ofthese used a randomized design.29,35 Oneof these randomized, controlled studiesevaluated a curriculum implemented atseven U.S. training programs,29 and theother evaluated programs at 18 U.S.teaching hospitals.38 However, moststudies (24; 92%) came from singlecenters and had methodological concernsthat undermined the results, such as lowresponse rates and small sample sizes(median 41 participants; interquartilerange 20 –106).

Evaluations of curricula targeting medicalstudents primarily measured learners’

knowledge, with a lesser emphasis onbehavior change. Only one medicalstudent curriculum targeted changes inclinical processes.49 Curricula forresidents more commonly involvedresidents in QI projects (14; 58%) andfrequently reported outcomes thatmeasured improvements in process ofcare. Only two studies reported benefitsto patients.24,28

Table 3 reports Kirkpatrick learningoutcomes by training level. The followingsection summarizes each learningoutcome in greater detail.

Learners’ satisfaction

Satisfaction was usually measured on aLikert scale from poor to excellent. Themajority of learners were satisfied withthe QI curricula, consistently rating thecurricula as relevant and useful. Only twostudies reported low satisfaction ratings.One was conducted with first-yearmedical students35 and reported earlytermination of the study due to learners’dissatisfaction with the curriculum. Theother study involved second-year medicalstudents.24 Students participating in thiscurriculum also expressed a number ofconcerns, including skepticism about theproject being an efficient use of time.Among their concerns, 84% of studentsreported dissatisfaction with the chartaudit exercise.

Learners’ attitudes

Learners generally exhibited positiveattitudes prior to exposure to thecurricula. For instance, the majority oflearners already regarded QI and PS asimportant topics relevant to futurepractice. Given these positive baselineattitudes, most curricula reported minimalimpacts on attitudinal outcomes.

Knowledge acquisition

Acquisition of curricular content wasusually assessed using tests of knowledgedesigned by study teams, though somestudies used established assessment toolssuch as the Quality ImprovementKnowledge Assessment Tool.59 With self-assessed knowledge outcomes, learnersgenerally rated their knowledge highlyand improved from baseline. All eightstudies that quantified knowledgeacquisition reported statisticallysignificant improvements.

Behavioral change

The five studies21,26,30,37,43 that reportedbehavioral changes all used self-reported

Figure 1 953 potentially relevant citations identified in literature search: 480 in MEDLINE 450 in EMBASE 20 by Handsearch 3 in HealthSTAR

138 Abstracts retrieved for evaluation

815 Citations excluded; not relevant

97 citations excluded:51 Did not describe a curriculum

25 Did not teach quality improvement or patient safety

7 Learners not medical students or residents 6 Quality improvement or patient safety not the

focus of the curriculum 3 Study unavailable 3 Duplicate report of same curriculum 2 Insufficient description of teaching methods

41 Studies included in review

14 Curriculum description ONLY 27 Curriculum description AND evaluation

Figure 1 Literature search and study selection process for identifying quality improvement andpatient safety curricula published between 2000 and 2008.

Patient Safety


outcomes. Nonetheless, only two studiessuggested any improvements in thebehaviors targeted by the curricula. Onestudy reported that although manystudents had disclosed errors to a peer(71%) or faculty member (46%), only7% had ever used a Web-based reportingsystem highlighted in the curriculum.30

The other study that reported an outcomerelated to learners’ behavior targeteddisclosure of medical errors to patients andfound that only 7% of learners reportedhaving made such a disclosure followingexposure to the curriculum.26

Changes in clinical processes

A number of studies involved chartaudits, modified morbidity and mortalityconferences, or participation in a QIproject. Seven of the 13 studies (54%)reported significant improvements inprocesses of care,20,24,28,31,36,41,43 includingincreased microalbuminuria screening,20

documentation of foot and eyeexaminations,24 and increasedmonofilament testing28 for patients withdiabetes, increased screening for elevatedbody mass index in an ambulatory internalmedicine clinic,36 reduction

in inappropriate telemetry use on aninpatient medical service,43 increaseddischarge dictations with completemedication information,41 and increasedimmunizations in a pediatric clinic.31

Benefits to patients

Two studies measured benefits to patientsin terms of intermediate clinicaloutcomes (serum HbA1c in both cases).In one study, 13 internal medicineresidents performed chart audits onpatients with diabetes and reflected onsolutions to identified problems.28 That

Table 1Features of 41 Quality of Care and Patient Safety Curricula Described in StudiesPublished Between 2000 and January 2009

17 undergraduate curriculastudies*

Characteristic7 preclinical,

no. (%)10 clinical,

no. (%)24 postgraduate,

no. (%)All 41 studies,

no. (%)

Educational setting...................................................................................................................................................................................................................................................................................................................

Classroom/nonclinical setting 7 (100) 5 (50) 11 (46) 23 (56)...................................................................................................................................................................................................................................................................................................................

Ambulatory care 3 (43) 2 (20) 13 (54) 18 (44)...................................................................................................................................................................................................................................................................................................................

Inpatient hospital 0 (0) 0 (0) 7 (29) 7 (17)...................................................................................................................................................................................................................................................................................................................

Mixed clinical setting 0 (0) 0 (0) 3 (13) 3 (7)...................................................................................................................................................................................................................................................................................................................

Distance learning 0 (0) 1 (10) 1 (4) 2 (5)...................................................................................................................................................................................................................................................................................................................

Not stated 0 (0) 2 (20) 0 (0) 2 (5)

Teaching method...................................................................................................................................................................................................................................................................................................................

Didactic lectures 6 (86) 7 (70) 18 (75) 31 (76)...................................................................................................................................................................................................................................................................................................................

Small-group discussion 5 (71) 6 (60) 5 (21) 16 (39)...................................................................................................................................................................................................................................................................................................................

Case discussion 2 (29) 2 (20) 8 (33) 12 (29)...................................................................................................................................................................................................................................................................................................................

Experiential learning 7 (100) 7 (70) 19 (79) 33 (80)...................................................................................................................................................................................................................................................................................................................

Web-based module 2 (29) 1 (10) 3 (13) 6 (15)

Educational content...................................................................................................................................................................................................................................................................................................................

Quality topics 5 (71) 3 (30) 19 (79) 27 (66)...................................................................................................................................................................................................................................................................................................................

Quality of care in general 3 (43) 1 (1) 11 (46) 15 (37)...................................................................................................................................................................................................................................................................................................................

Continuous qualityimprovement (e.g., PDSA)

4 (57) 2 (20) 15 (67) 21 (51)

...................................................................................................................................................................................................................................................................................................................Audit and feedback 2 (29) 1 (10) 4 (17) 7 (17)

...................................................................................................................................................................................................................................................................................................................Process mapping 1 (14) 0 (0) 6 (25) 7 (17)

...................................................................................................................................................................................................................................................................................................................Change management 2 (29) 1 (10) 6 (25) 9 (22)

...................................................................................................................................................................................................................................................................................................................Patient safety topics 4 (57) 9 (90) 16 (63) 29 (70)

...................................................................................................................................................................................................................................................................................................................Patient safety in general 2 (29) 4 (40) 8 (33) 14 (34)

...................................................................................................................................................................................................................................................................................................................Systems thinking 4 (57) 3 (30) 9 (38) 16 (39)

...................................................................................................................................................................................................................................................................................................................Root cause analysis 3 (43) 2 (20) 12 (50) 17 (41)

...................................................................................................................................................................................................................................................................................................................Human factors 0 (0) 1 (10) 2 (8) 3 (7)

...................................................................................................................................................................................................................................................................................................................Error/incident reporting 3 (43) 6 (60) 4 (17) 13 (32)

...................................................................................................................................................................................................................................................................................................................Dealing with errors 0 (0) 3 (30) 0 (0) 3 (7)

...................................................................................................................................................................................................................................................................................................................Safety culture (i.e.,avoiding blame/shame)

0 (0) 3 (30) 2 (8) 5 (12)

...................................................................................................................................................................................................................................................................................................................Disclosure of error 3 (43) 1 (10) 0 (0) 4 (10)

* Three studies that targeted both undergraduate and postgraduate learners were classified by the lowest traininglevel (e.g., undergraduate clinical).

Patient Safety


Table 2Characteristics, Teaching Methods, Educational Content, and Learning Outcomesof 41 Curricula in Quality Improvement or Patient Safety for Trainees Describedin Studies Published Between 2000 and January 2009*

Source (First author,year, referencenumber) Setting Learners Intervention

Teachingmethods Educational content

Learningoutcomes† Main findings

27 studies with a curricular description and an evaluative component

Highest Kirkpatricklearning outcomeachieved �

satisfaction (Level 1),learner attitudes(Level 2A), orknowledge (Level2B)—10 studies

Undergraduatelearners (5 studies)

...................................................................................................................................................................................................................................................................................................................Newell, 200833 Single U.S.

medical school123 third-yearmedical students(surgical clerkship)

2 mandatorysessions (total 3hours)

Large- and small-group discussions

Medical error, coping witherror

Attitudes(Level 2A)

Improvement in attitudes towardmedical errors (increasedawareness of normative medicalerrors from 2% to 21%)

...................................................................................................................................................................................................................................................................................................................Gunderson, 200825 Single U.S.

medical school18 final-yearhealth sciencesstudents (5disciplines)

One 3-hour electivesession

Didactic sessions,experiential role-playing, use ofvideo clips

Disclosing errors, rootcause analysis

Knowledge(Level 2B)

Improvement in observeddisclosure of medical error (2/14failed to include essentialelements of full disclosurecompared with 14/14 before thesession)

...................................................................................................................................................................................................................................................................................................................Moskowitz, 200732 Single U.S.

medical school229 third-yearmedical students

1-day mandatoryinterclerkshipprogram

Plenary sessions,small-groupworkshops, role-playing

Patient safety overview,patient safetyimprovement tools,discussing and reportingmedical errors, clinical QI,legal aspects of patientsafety

Attitudes(Level 2A)Knowledge(Level 2B)

Improvement in self-reportedattitudes and knowledge on 14of 21 questionnaire items

...................................................................................................................................................................................................................................................................................................................Ogrinc, 200735 Single U.S.

medical school39 first-yearmedical students(41 additionalfirst-year medicalstudents assignedto late-intervention groupacted as thecontrol group)

Incorporation of amandatorylongitudinal PBLImodule into anexisting first-yearmedical schoolcourse (four 10-minute overviewsessions);

Small-groupdidactic lectures,practicalapplication ofPBLI methods toimprove personalskills

Knowledge and skills forimproving systems, PDSAcycle, assessment ofsystem performance, howto make changes to asystem

Satisfaction(Level 1)Attitudes(Level 2A)Knowledge(Level 2B)

Low satisfaction rating (30–40out of 100)Increase in QIKAT knowledgescores in intervention group (8.5to 9.3) vs decrease in QIKATscores in control group (8.3 to7.9); P � .05

...................................................................................................................................................................................................................................................................................................................Henley, 200227 Single U.S.

medical school30 third-yearmedical students

Weekly mandatoryQI curriculum (45–60 minutes perweek)

Didactic, video,chart audits

QI theory, audit andfeedback, systemsthinking, effecting change


Moderate satisfaction(50%–60% of students feltteaching was useful)Scored 84% on a 6-item end-of-rotation quiz on QI concepts

Postgraduatelearners (4 studies)

...................................................................................................................................................................................................................................................................................................................Peters, 200838 18 U.S. teaching

hospitals78 internalmedicine residents(PGY2 and PGY3residents)(72 internalmedicine residentsserved as controls)

4-module electiveonline learningcourse (AchievingCompetency Today)

Web-facilitated,self-directed andaction learningand thedevelopment of aQI plan

Systems thinking, PDSAcycle, root cause analysis,effort yield tables


Increase in test of knowledgescores from 55.2 to 59.6compared with 50.2 to 48.3 incontrol group, no significantdifference in attitude change pre-vs postintervention

...................................................................................................................................................................................................................................................................................................................(Continues)

Patient Safety


Table 2(Continued)




Varkey, 200840 1 U.S. teachinghospital

2 preventivemedicine and 7endocrinologyfellows (PGY4 andPGY6)

3-week QI electiveblock

Didactic lecture,small-groupdiscussion, case-based discussion,QI project

QI overview, PDSA cycle,process mapping, patientsafety overview, incidenterror reporting, root causeanalysis, failure modeeffects analysis


High satisfaction (5/9 rated aboveaverage, 4/9 rated superior)Significant increase in learnerQIKAT scores postrotation(11.89/15) compared withprerotation (7.33/15), P � .004Improvement in patientunderstanding of care (11%increase in number of patientswho understood why tests wereordered, 12% increase in thenumber of patients whounderstood recommendedtreatment)

...................................................................................................................................................................................................................................................................................................................Djuricich, 200422 Single U.S.

residencyprogram (internalmedicine andpediatricsprograms)

PGY3 internalmedicine andPGY2 pediatricsresidents (44residents total,split not specified)

3-hour mandatorycurriculum duringambulatory block

Didactic lectures,design of QIproject (althoughactual project notcarried through)

QI overview, PDSA andmodel for improvement


Increase in score on 5-item quizfrom 48% to 89% on pre–posttesting of CQI knowledge

...................................................................................................................................................................................................................................................................................................................Frey, 200323 Single U.S.

residencyprogram (familymedicine)

12 PGY3 familymedicineResidents (6residents from 2separate years)

Longitudinalmandatory teamCQI project

Didacticseminars, CQIproject (practiceguidelineimplementation)

CQI process, audit andfeedback


High overall confidence inknowledge and attitudes (3.5–4.1 out of 5)

Undergraduate andpostgraduatelearners (1 study)

...................................................................................................................................................................................................................................................................................................................Kerfoot, 200729 7 U.S. residencies

(1 emergencymedicine, 1internal medicine,2 Ob/Gyn, 3surgery) and 2medical schools

315 residents(PGY1 to PGY5)and 325 second-and third-yearmedical students

3 Web-basedlearning modules(each taking �30minutes tocomplete)

Interactive Web-based modulesusing audio andvideo clips,multiple-choicequestions,animations

Patient safety overview,error prevention, systemstheory

Satisfaction(Level 1)Knowledge(Level 2B)

High satisfaction rating (4 out of 5)Increase in MCQ test scorescompared with baseline (16%increase from baseline of 58%)Knowledge sustained across 4weeks (1% decay in MCQ testscores)

...................................................................................................................................................................................................................................................................................................................Highest Kirkpatricklearning outcomeachieved � behavior(Level 3) or clinicalprocess change orpatient benefits(Level 4)—17 studies

Undergraduatelearners (4 studies)

...................................................................................................................................................................................................................................................................................................................Patey, 200737 Single UK medical

school110 final-yearmedical students

2 mandatorysessions 3 daysapart (total 5 hours)

Large-grouplectures, small-groupdiscussions,studentpresentation,audio-video casediscussions, role-playing

Understanding medicalerrors, factors influencingadverse events, skillsrequired to deal with error,reporting errors, focusingon cause rather thanculprit

Satisfaction(Level 1)Attitudes(Level 2A)Knowledge(Level 2B)Behavior(Level 3)

High satisfaction rating (4–5 outof 5)Improvement in some self-assessed attitudes andknowledgeMajority planned to reportmedical errors that they make(51 out of 70, 73%)

...................................................................................................................................................................................................................................................................................................................Madigosky, 200630 Single U.S.

medical school92 second-yearmedical students

Integratedmandatorycurriculum intoexisting preclerkshipcourse (10.5contact hours)

Lectures, paneldiscussions,demonstrations,role-playing,learning exercises

Patient safety overview,error reporting, system vshuman approach, safetytools, disclosure, rootcause analysis


High satisfaction rating (72–82 outof 100)Multidirectional changes in self-reported attitudes and knowledgequestionnaire itemsLow impact on behavior—7%reported an error through a formalprocess

...................................................................................................................................................................................................................................................................................................................(Continues)

Patient Safety


Table 2(Continued)




Halbach, 200526 Single U.S.medical school

572 third-yearmedical studentsduring 3 years

4-hour mandatorycurriculum duringfamily medicineclerkship rotation

Lecture, small-group discussion,readings,videotapedsimulation withstandardizedpatient

Discussing/ reportingmedical errors, patientsafety overview


High satisfaction rating (82–94out of 100)High self-reported ratings ofattitudes and knowledgeregarding error disclosure21 of 307 (7%) reported havingdisclosed a medical error to apatient

...................................................................................................................................................................................................................................................................................................................Gould, 200224 Single U.S.

medical school77 second-yearmedical students(plus 893 charts)

Mandatory QIcurriculum integratedinto weeklyambulatory block(total time not stated)

Didactic, small-group discussion,QI project, chartaudit

QI theory, CQI process, QImeasurement, audit andfeedback

Satisfaction(Level 1)Attitudes(Level 2A)Knowledge(Level 2B)Clinicalprocesschange(Level 4A)Patientbenefits(Level 4B)

General dissatisfaction withchart-audit learning experience(16% positive rating)Overall improvement in 27 of 40survey items measuring self-reported attitudes andknowledge toward CQIIncreased rates of foot (51% to70%; P � .001) and eye (27% to38%; P � .001) exams on pre–post chart auditsHbA1c mean value decreasedfrom 7.7% to 7.2% on pre–postchart audits (P � .001)

Postgraduatelearners(13 studies)

...................................................................................................................................................................................................................................................................................................................Oyler, 200836 1 U.S. teaching

hospital34 internalmedicine residents(PGY2)

4 mandatory 90-minute seminars �

2 ambulatory blocks(total 12 hours �

project time)

Didactic lecture,small-groupdiscussion, QIproject, Web-based chart audit

QI overview, PDSA cycle,process mapping, changemanagement

Attitudes(Level 2A)Knowledge(Level 2B)Clinicalprocesschange(Level 4A)

Improvement in self-assessedknowledge (comfort using PDSAcycle increased from 9% to 89%)Improvement in several processesof care (increased documentationof height for BMI screening from11% to 88% (P � .001),decrease in the number of“inaccurate medication lists”from 25% to 9% (P � .001)

...................................................................................................................................................................................................................................................................................................................Voss, 200842 Single U.S.

residencyprogram (internalmedicine)

34 PGY1 andPGY2 residents

Longitudinalmandatory QI andsafety curriculum (7� 3-hour seminars)

Didacticseminars,experientialinvolvement in QIproject

CQI (PDSA), root causeanalysis, systems thinking,human factors, changemanagement, processmapping

Satisfaction(Level 1)Knowledge(Level 2B)Clinicalprocesschange (Level4A)

High satisfaction rating (4.4–4.7out of 5)High self-reported knowledgescores (4.4–4.8 out of 5)Several QI projects implemented(no outcomes reported)

...................................................................................................................................................................................................................................................................................................................Bechtold, 200718 Single U.S.

residencyprogram(department ofinternal medicine)

90 internalmedicine residentsand fellows

Mandatory monthly1-hour-long revisedpatient safety M&Mconference

Large-groupdiscussion ofcases thathighlightimportant healthcare systemsafety issues

Systems thinking, factorsinfluencing adverse events,modified root causeanalysis process


No significant change in 14 of 20survey items related to attitudeand knowledge59% of recommendations forimprovement that were identifiedfrom M&M rounds wereimplemented at 1 year

...................................................................................................................................................................................................................................................................................................................Canal, 200719 Single U.S.

residencyprogram(department ofsurgery)

15 PGY3 surgicalresidents

6-week electivecurriculum duringresearch year(PGY3)

Didactic lectures,design andimplementationof animprovementproject

PDSA cycle within theModel for Improvement


Increase in self-reported attitude(3.7 to 4.4 out of 5) andknowledge (1.9 to 4.6 out of 5)scoresSeveral QI projects implementedto reduce surgical consultationwait times (no outcomesreported)

...................................................................................................................................................................................................................................................................................................................(Continues)

Patient Safety


Table 2(Continued)




Varkey, 200641 Single U.S.academic medicalcenter

5 residents (2preventivemedicine, 1internal medicine,2 family medicine)

4-week electivecurriculum (actualtime commitmentnot specified)

Didactic lectures,small-groupdiscussions andexercises, case-based learning,QI project

QI and patient safetyoverview, process mappingand root cause analysis,PDSA, medical errorreporting

Satisfaction(Level 1)Knowledge(Level 2B)Clinicalprocesschange(Level 4A)

High satisfaction rating (4.1 outof 5)Increase in QIKAT knowledgescores from 2.3 to 3.4 afterinterventionImprovement in medicationreconciliation—increasedcompleteness of dictatedmedication lists from 38% to75% (P value not reported)

...................................................................................................................................................................................................................................................................................................................Coyle, 200521 Single U.S.

residencyprogram(department offamily medicine)

30 family medicineresidents (10 fromeach year)

6 mandatory 1-hourconferences

Didactic lectures,case discussions(small and largegroup)

Patient safety overview,causes of errors, errorreporting, root causeanalysis

Attitudes(Level 2A)Behavior(Level 3)

No change in mean attitude andbehavior scores (medical eventreporting) before and 6 monthsafter education program

...................................................................................................................................................................................................................................................................................................................Holmboe, 200528 Single U.S.

residencyprogram(department ofinternal medicine)

13 PGY2 internalmedicine residents

Weekly half-dayelective for 4 weeks(quality-of-carerotation) �

longitudinal chartaudit

Self-reading,learningexercises, small-group discussions(to discussstrategies toimprove care),chart audit

Patient safety overview(excerpts from IOMreports), self-audit

Attitudes(Level 2A)Clinicalprocesschange(Level 4A)Patientbenefits(Level 4B)

8 of 12 (67%) systems-basedchanges recommended byresidents were carried through at6 monthsIncreased rate of monofilamenttesting (13% vs 1%; P � .02)and ordering of baseline EKG(17% vs 10%; P � .01)Change in pre–post HbA1c of�0.4% in the intervention groupcompared with �0.7% in thecontrol group (P � .001)

...................................................................................................................................................................................................................................................................................................................Tomolo, 200539 Single U.S.

residencyprogram (internalmedicineprogram)

45 internalmedicine residents(PGY1 18%, PGY240%, PGY3 35%,PGY4 7%)

Two 1-hoursessions, and theuse of an“outcomes card”(residents completethese cards tocapture cases whichhighlight importantpatient safetyissues)

Didactic lectures,experientialactivities

Patient safety overview,systems thinking, safetyculture, human factorsengineering


High satisfaction rating (12.3 outof 15)High self-assessment scores forknowledge (48 out of 60)Several organizational practicechanges implemented (nooutcomes measured)

...................................................................................................................................................................................................................................................................................................................Ogrinc, 200434 2 U.S. residency

programs(internalmedicine, andcombined internalmedicineprograms)

11 residents (3PGY2, 7 PGY3, 1PGY4)(22 residentsmatched byspecialty and yearof training servedas controls)

Longitudinal PBLIelective (at least 4weeks, 4–8 hoursper week—timelogs indicated meantime � 120 hours)

Didactic lecturesand experientiallearning (residentimprovementproject)

Foundations of PBLI, PDSAcycle, process and systemschange

Satisfaction(Level 1)Attitudes(Level 2A)Knowledge(Level 2B)Clinicalprocesschange(Level 4A)

High satisfaction rating (4.4–4.7out of 5)Increase in QIKAT knowledgescores from 9.2 to 11.4compared with 8.2 to 8.7 incontrol groupSeveral organizational practicechanges implemented (nooutcomes measured)

...................................................................................................................................................................................................................................................................................................................Weingart, 200443 Single U.S.

residencyprogram(department ofgeneral medicine)

19 internalmedicine residents

QI elective duringambulatory block(20 hours per weekfor 3 weeks)

Didactic lecturesand experientiallearning (residentimprovementproject, QIexercises)

QI and patient safetyoverview, rapid cycleimprovement, root causeanalysis

Satisfaction(Level 1)Attitudes(Level 2A)Knowledge(Level 2B)Behavior(Level 3)Clinicalprocesschange(Level 4A)

High satisfaction rating (71% to87% rating)Positive responder ratings forself-assessed attitudes,knowledge, 56% reported achange in behaviorSeveral organizational practicechanges with positive outcomes(i.e., 62% decrease ininappropriate use of telemetryfor chest pain patients; P valuenot reported)

...................................................................................................................................................................................................................................................................................................................(Continues)

Patient Safety


Table 2(Continued)




Ziegelstein, 200444 Single U.S.residencyprogram (internalmedicineprogram)

44 internalmedicine residents(trainee level notspecified)

Multifacetedintervention (weeklyM&M conference,improvementexercises duringambulatory block andcontinuity clinic), allmandatory

Large-groupdiscussion atM&M rounds—discussion ofcases with focuson systems-practice issues;chart audits

Audit and feedback,systems thinking


High satisfaction rating (76%–92% rating)Improved self-rated scores forknowledge and attitude (1.6 to2.5 out of 5)Organizational practice changeimplemented to improvemammography rates (nooutcomes reported)

...................................................................................................................................................................................................................................................................................................................Coleman, 200320 Single U.S.

residencyprogram (familymedicine)

24 family medicineresidents (PGY1–3)

Longitudinalmandatory QIproject (6months)—withhourly sessions,project time notstated

Didactic sessions,QI project

PDSA, root cause analysis,audit feedback,implementing change

Satisfaction(Level 1)Attitudes(Level 2A)Clinicalprocesschange(Level 4A)

Moderate satisfaction scores(60%–70% rating) for rating ofvalue of interventionOrganizational practice changesresulted from 3 QI projects(increased completion of patientdata summary sheets from 14%to 40% �P � .001; increasedscreening of diabetic patients formicroalbuminuria from 5% to29% �P � .017; increasedmedication list completion from10% to 44% �P � .001)

...................................................................................................................................................................................................................................................................................................................Mohr, 200331 Single U.S.

residencyprogram(pediatriccommunity clinic)

8 residents (of 36senior residentsvoluntarilyrecruited)

Participation in anelective yearlong QIprogram

Didactic lectures,learningexercises,participation in aQI project

Key principles of QI,process mapping,implementing processchanges

Clinicalprocesschange(Level 4A)

Increase in childhoodimmunization rates from 60% to86% (P � .04)

14 studies with a curricular description only

Undergraduatelearners (5studies)

...................................................................................................................................................................................................................................................................................................................Thompson, 200855 Single U.S.

medical schoolFirst-year medicalstudents

5 mandatory weekly2-hour sessions(total 10 hours)

Didactic sessions,small-groupdiscussion,experiential role-playing, audio-video casediscussion

Systems thinking,reporting errors, disclosingerrors, root cause analysis,teamwork andcommunication

...................................................................................................................................................................................................................................................................................................................Varkey, 200756 Single U.S.

medical schoolMedical studentsfrom all 4 years(42 third-yearmedical studentsincluded inevaluation ofknowledge; totalnumber notspecified)

4-year longitudinalcurriculumintegrated intoexisting curriculumwith mandatory andelectivecomponents

Didactic lectures,small-groupsessions, paneldiscussions,simulation, onlinemodules, casediscussions, QIproject

Basic principles of QI andpatient safety, systemsthinking, medical error(and reporting/ disclosure),root cause analysis

...................................................................................................................................................................................................................................................................................................................Gould, 200449 11 U.S. medical

schoolsMedical studentsfrom all 4 years(actual numbernot specified)

Multipleinterventions ofvarying intensityintegrated intoexisting curriculum

Didactic lectures,small-groupdiscussions,learningexercises, chartaudit, Webmodule, QIproject

CQI, PDSA, audit andfeedback, changemanagement, qualitytheory

...................................................................................................................................................................................................................................................................................................................Paulman, 200251 Single U.S.

medical school120 “junior”medical students

Mandatory QIlearning projectintegrated in ruralrotation

Learning exercise(identify problem,collect data todefine problem,designintervention)

CQI process

...................................................................................................................................................................................................................................................................................................................(Continues)

Patient Safety


Table 2(Continued)




Weeks, 200057 Single U.S.medical school

First and second-year medicalschool

Integratedmandatory QIcurriculum (7months) withelective component

Didactic lectures,learningexercises,involvement in QIproject

CQI, systems theory,process mapping

Postgraduatelearners (7 studies)

...................................................................................................................................................................................................................................................................................................................Wong, 200858 Single Canadian

residencyprogram (internalmedicine)

31 PGY1 residents Longitudinalmandatory QIcurriculum (2 �

3.5-hour sessions,plus team-based QIproject—1 hour perweek protectedtime � 10 months)

Didactic lectures,experientialinvolvement in QIproject

QI theory (PDSA), model ofimprovement, processmapping, CQI process

...................................................................................................................................................................................................................................................................................................................Krajewski, 200750 Single U.S.

residencyprogram(radiology)

Radiologyresidents (PGY2and above)

1-month electiveduring radiologytraining program

Didactic lectures,Web-facilitatedself-directedlearning,experientialinvolvement in QIproject

QI and patient safetyoverview, root causeanalysis, changemanagement

...................................................................................................................................................................................................................................................................................................................Rosenfeld, 200552 Single U.S.

residencyprogram(department ofsurgery)

Surgical residents(actual numbernot reported)

Weekly mandatoryM&M conference

Large-groupdiscussion ofcases as theyrelate to theACGME corecompetencies,practice-basedimprovementexercise

Systems-based practice,root cause analysis,practice-basedimprovement

...................................................................................................................................................................................................................................................................................................................Singh, 200554 Single U.S.

residencyprogram(department offamily medicine)

46 family medicineresidents (70%overallparticipation,PGY1 80%, PGY360%)

4-hour mandatoryworkshoppresented duringresidency programorientation series,plus three 1-hoursessions

Didactic lectures,active learning,experientialactivities (i.e.,chart audits)

Patient safety overview,strategies for safetyimprovement, culture ofsafety, behavioral skills forpatient safety, medicationsafety, systems thinking,audit feedback, root causeanalysis

...................................................................................................................................................................................................................................................................................................................Esselman, 200246 Single U.S.

rehabilitationcenter

Physiatry residents(actual numbernot specified)

Monthly mandatoryrehabilitation M&Mconference

Discussion ofcases thathighlightimportantrehabilitationquality and safetyissues

Systems thinking, rootcause

...................................................................................................................................................................................................................................................................................................................Farquhar, 200147 Single Canadian

residency (internalmedicine)

Internal medicineresidents

Quality-of-carecurriculum(mandatory half-dayseminars andmonthly noon-hoursessions)

Didactic sessions,case discussionsto highlightprocess of care

QI theory, how to improvequality, systems thinking

...................................................................................................................................................................................................................................................................................................................Schillinger, 200053 Single U.S.

residency (internalmedicine)

Internal medicineresidents

Mandatory QIproject and seminarseries

Didactic lectures,learning exercise,participation in aQI project

QI theory, processimprovement, outcomesmeasurement

...................................................................................................................................................................................................................................................................................................................(Continues)

Patient Safety


Hstudy reported a decrease in HbA1clevels of 0.4% for patients cared for byparticipants in the intervention groupcompared with an increase of 0.7% in thecontrol group (P � .001). The otherstudy, in which 77 second-year medicalstudents audited charts for patients withdiabetes,24 reported a reduction in HbA1clevels from 7.7% before implementationof the QI curriculum to 7.2% afterwards(P � .001).

Factors that influenced curricularimplementation

Of the 41 included reports, 34 (83%)described factors that influenced

implementation of QI and PS curricula(see Table 4). Commonly cited barriersrelated to learners included competingeducational demands and the level ofinitial buy-in or enthusiasm. For faculty,many reports highlighted the problem ofinadequate numbers of teachers withrequisite expertise, and the timecommitment required for those fewfaculty members (often only one or twoat a given institution with such expertise).Barriers related to the curriculathemselves included achieving theappropriate balance of didactic andexperiential learning and scheduling thecurriculum amidst existing classes and

rotations. Important aspects of thelearning environment included theinstitutional culture with respect toquality and safety, hospital operationalsupport (e.g., some authors noted thepositive impact on trainees of includinghospital executives or faculty role modelsinvolved in local improvement efforts), aswell as the availability of informationsystems that could facilitate QI projectsundertaken by trainees.

Many of the same implementation issuesemerged across all curricula irrespectiveof learner level (i.e., undergraduate orpostgraduate). However, some factorswere more commonly cited as importantfactors only for curricula targetingresidents (e.g., time pressures and theneed for ongoing financial, educational,institutional, and operational support),perhaps because of the greater inclusionof QI or PS projects in curricula forresidents. A barrier unique to curriculathat targeted medical students in thepreclinical years was their perception ofthe unimportance of the materialcompared with traditional clinicalcontent.

Discussion

We identified 41 QI and PS curricula thatspecifically targeted medical students or

Table 2(Continued)




Undergraduate andpostgraduatelearners ( 2 studies)

...................................................................................................................................................................................................................................................................................................................Cosby, 200345 Not stated Emergency

medicine residentsand students

Patient safetycurriculum (topicoutlines andsuggested teachingmethods described)

Didactic sessions,small-groupdiscussion, use ofvideo, casediscussion,modified M&Mrounds, learningexercises

Medical error, safetyculture, models of error,cognitive error, systemsthinking, coping with error

...................................................................................................................................................................................................................................................................................................................Gosbee, 200248 12 U.S. VA

facilitiesResidents (internalmedicine,pediatrics,anesthesia, familymedicine, surgery)and students

5 separate modules(topic outlines andsuggested teachingmethods described)

Didactic andsmall-groupsessions

Patient safety overview,safety culture, humanfactors, root causeanalysis, patient safetyinterventions

* ACGME indicates Accreditation Council for Graduate Medical Education; BMI, body mass index; CQI,continuous quality improvement; EKG, electrocardiogram; HbA1c, hemoglobin A1c; M&M, morbidity andmortality; MCQ, multiple-choice questionnaire; PBLI, practice-based learning and improvement; PDSA, plan-do-study-act; PGY, postgraduate year; QI, quality improvement; QIKAT, quality improvement knowledgeassessment tool.

† Learner outcomes are classified using Kirkpatrick’s16 model, which includes impacts on learners’ satisfaction(Level 1), changes in learner attitudes (Level 2A), measures of learner knowledge and skills (Level 2B), changes inlearner behavior (Level 3), changes to clinical processes (Level 4A), and benefits to patients (Level 4B).

Table 3Kirkpatrick Learning Outcomes, by Trainee Level, of 27 Studies of Quality andSafety Curricula That Had an Evaluation Component, 2000 to January 2009*

Learning outcomes†Undergraduate curricula

(10 studies), no. (%)Postgraduate curricula

(18 studies), no. (%)

Learner satisfaction 7 (70) 7 (39)Learner attitudes 8 (80) 14 (78)Knowledge acquisition 9 (90) 14 (78)Behavioral change 3 (30) 2 (11)Changes in clinical practice 1 (10) 12 (67)Benefits to patients 1 (10) 1 (6)

* Data from a total of 27 studies are reported above; one of these discussed curricula for both medical studentsand residents. Some studies discussed more than one outcome.

† Learner outcomes are classified using Kirkpatrick’s16 model, which includes impacts on learners’ satisfaction(Level 1), changes in learner attitudes (Level 2A), measures of learner knowledge and skills (Level 2B), changes inlearner behavior (Level 3), changes to clinical processes (Level 4A), and benefits to patients (Level 4B).

Patient Safety


Table 4Factors That Influence the Successful Implementation of Quality of Care orPatient Safety Curricula, From 34 Studies Published Between 2000 andJanuary 2009*

Factors Example

Learner factors...................................................................................................................................................................................................................................................................................................................

Level of learner enthusiasm or buy-in toward curriculum

“Until trainees appreciate the clinical relevance of systems-based practice competencies, educationalprograms in this domain may be perceived as unwelcome training requirements.”—Kerfoot et al29

“Medical student demand for �patient safety has helped capture the attention of the Johns HopkinsSchool of Medicine Curriculum Reform Committee …. The committee is considering the best method forincorporating patient safety-related issues into the 4-year medical school curriculum.”—Thompson et al55

...................................................................................................................................................................................................................................................................................................................Competing educational demandsof medical students and residents

“A program must successfully compete with other new technologies, diseases, and treatments, all ofwhich may seem more exciting and pertinent to the developing physician.”—Gould et al24

“Despite a lack of familiarity with QI principles, residents were reluctant to ’sacrifice’ valuablecurricular sessions to learn to use QI tools.”—Coleman et al20

Teacher factors...................................................................................................................................................................................................................................................................................................................

Adequate number of faculty withexpertise in teaching quality andsafety

“Challenges to implementing this curriculum include finding adequate faculty with QIexperience.”—Oyler et al36

“A critical component of this effort is a faculty development initiative that will enhance the ability ofteacher–clinicians in general and hospital medicine to teach residents about quality and safety inhealth care.”—Weingart et al43

...................................................................................................................................................................................................................................................................................................................Involvement of faculty role modelscommitted to patient safety

“An additional factor in the success of our curriculum was the participation of a stable cadre ofcommitted faculty … such faculty role models discuss not only the knowledge and skills required forsafe practice, but also demonstrate the attitudes required.”—Halbach and Sullivan26

...................................................................................................................................................................................................................................................................................................................Faculty recognition and support “An internal grant process helped to focus and support faculty efforts.”—Weingart et al43

...................................................................................................................................................................................................................................................................................................................Level of faculty enthusiasm or buy-in toward curriculum

“Quality improvement is one of the key strategic objectives of the clinic … thus most faculty wereenthusiastic and supportive of integrating QI components into their courses and eager to enhancetheir learning about the subject matter.”—Varkey et al56

...................................................................................................................................................................................................................................................................................................................Time burden on faculty to teachthe curriculum

“An easily imported, ready-made design to overcome the high barrier of creating a program whereboth the director’s time and expertise were limited.”—Peters et al38

Curricular factors...................................................................................................................................................................................................................................................................................................................

Curriculum should combinedidactic and experiential teachingmethods

“Learning must be experience based … by having residents identify a problem, create an aim, studythe work process, measure the processes and outcomes, and recommend improvements, theyapplied PBLI to real situations that were important to them.”—Ogrinc et al34

“The QI elective, now in its fourth year, demonstrates the feasibility and durability of an approachthat balances didactic and experiential learning … the experiential component … provided theresidents with an immediate and relevant ’in-the-trenches’ opportunity that often resulted in atangible contribution to the quality of care.”—Weingart et al43

...................................................................................................................................................................................................................................................................................................................Providing adequate time to carryout curriculum (especially thoseinvolving QI projects)

“The greatest challenge was to identify meaningful projects that could be completed within 3weeks.”—Weingart et al43

“The time-limited nature of the elective limited the resident’s ability to make and followchanges.”—Ogrinc et al34

...................................................................................................................................................................................................................................................................................................................Scheduling of curriculum tooptimize likelihood of completingQI projects

“Many PGY3 residents wished to implement their projects but could not do so because they neededadditional time to complete the projects, yet were near graduation … the curriculum was thereforemoved to the PGY2 year.”—Djuricich et al22

“We believe that providing this curriculum during the research year, when clinical demands are notcompeting, is more likely to produce projects that could come to fruition.”—Canal et al19

...................................................................................................................................................................................................................................................................................................................Integration into existing curriculumlongitudinally and stand-aloneexperiences have both been foundto be effective

“The longitudinal nature of the curriculum helps to ensure its’ sustainability.”—Holmboe et al28

“Although some may argue that this issue �medical errors needs to be integrated throughout themedical school curriculum, evidence indicates that curricular change has little impact on students’perceptions unless there is a concentrated time devoted to unique topics.”—Moskowitz et al32

Learning environment factors...................................................................................................................................................................................................................................................................................................................

Institutional culture regarding QIto support educational efforts

“In order for a program to be successful in adopting this educational intervention … a residencyprogram that supports patient safety curriculum �is essential”—Tomolo et al39

“Fear of tort action and reporting to licensing boards is a barrier to role modeling behaviors ofreporting, investigating systems failures, and disclosing errors to patients”—Madigosky et al30

...................................................................................................................................................................................................................................................................................................................Linking curriculum to hospitalleadership or operational activities

“The greatest success has been achieved by selecting projects that already have organizationalmomentum.”—Schillinger et al53

...................................................................................................................................................................................................................................................................................................................(Continues)

Patient Safety


residents. Concepts of continuous QI,systems thinking, and root cause analysisconstituted the most common topicscovered, and specific projects undertakenoften involved chart audits. Despite theheterogeneity in educational content andteaching methods, most curricula werewell accepted and led to learners’knowledge acquisition. Residentinvolvement in experiential QI projectssuch as chart audits also frequently led tosignificant improvements in processes ofcare.

Few studies demonstrated changes inlearners’ behavior or potential patientbenefits. Although some reports suggestthat educational interventions have thepotential to change behavior or improvehealth outcomes, most studies lack good-quality evidence to support theirfindings.60 There are examples of well-designed continuing medical educationinterventions that are sequenced andmake use of interactive techniques thatlead to changes in learners’ behaviors andhealth outcomes. However, those studiesoften centered on screening, smokingcessation, and communication skills andmay not translate to more complexcurricular content areas, such as QI andPS.61–63 In fact, for QI and PS, improvingpatient outcomes as a result ofeducational efforts represents aparticularly daunting task, given thatintensive, large-scale QI efforts often failto demonstrate improvements in healthoutcomes.64,65 Also for some tools of QIand PS, including ones that commonlyappeared in the curricula we reviewed(e.g., root cause analysis), little empiricevidence guides recommendations onhow to design or use these tools.65

Consequently, even with optimal deliveryof the target educational content, thedegree to which organizational or patientoutcomes might improve remainsunclear.

Our results complement those of asystematic review of educational effortsin QI for clinicians in general13 in thatwell-established adult learning techniques(e.g., experiential learning) wereidentified as key factors for success indelivering curricula in QI and PS.However, our review, which included 34newer reports of curricula specificallytargeting trainees, demonstrated thatresidents’ involvement in QI and PScurricula can lead to meaningfulimprovements in clinical processes, anovel finding compared with those of theprevious review.

Our review also identified importantbarriers and facilitators toimplementation that are likely unique tocurricula in the undergraduate andpostgraduate settings. Many of thestudies identified barriers commonlyencountered with new curricularinitiatives in general.17 For example, mostof the curricula relied on small numbersof faculty members with a personalinterest in QI or PS to teach thecurriculum, often resulting inburdensome time commitments. Manyreports highlighted the need for greaterfaculty development to achieve sufficientnumbers of teachers of QI and PS topicsfor both medical student and residentcurricula. Some curricula addressed theseissues by developing teaching materialsthat circumvented the need to havefaculty experienced in QI or PS.36

Competing educational demands andachievement of learner buy-in alsorepresented major issues for curricula atall levels. However, the only tworeports24,35 that noted these as potentiallyinsurmountable obstacles were ones thattargeted medical students at preclinicalstages. Learners reported significantdissatisfaction with key elements of thecurricula, which suggests that clinicalexperience represents a prerequisite for

appreciating the importance andrelevance of QI or PS concepts.

Curricula that targeted residents mayrequire special consideration, perhapsbecause such curricula more commonlyinvolved the learners in experientialprojects, adding to time pressures andincreasing the need for supportinginfrastructure. Many residents did notcomplete their projects because of timeconstraints. Some programs addressedthis problem by scheduling theircurricula during less busy clinicalrotations or research years.19 Havingadequate personnel, financial, andtechnological resources to supportcurricula involving experiential projectswas also important. For example, studiesthat made use of chart audits requiredadministrative support to retrieve charts.Also, many QI projects depended onefficient availability of clinical datathrough information systems todetermine whether improvementsoccurred.

Finally, a number of studies emphasizedthe importance of a local “safety culture,”substantially enhancing the curricularsuccess when present and undermining itwhen absent. Other curricula that targetnonmedical competencies (e.g.,professionalism) also highlight theimportance of the so-called hiddencurriculum, where there is a discrepancybetween the concepts trainees learn informal educational venues and whattrainees observe when supervised byattending staff in routine clinicalpractice.66 –68 Preparing trainees for thefact that behavior of faculty in routinepractice, design of the delivery systems inwhich they work, and institutionalculture may not conform to acceptedprinciples of QI and PS may reduce thediscomfort reported by participants insome of the curricula we reviewed.

Table 4(Continued)

Factors Example

Financial support to fundeducational efforts and promotechanges from QI projects

“A project on improving communication … was delayed because of the inability to obtain fundingneeded to purchase a wireless telephone.”—Canal et al19

...................................................................................................................................................................................................................................................................................................................Information systems that canprovide easy access to health data

“Access to clinical data is important to plan improvements and to evaluate project successes.”—Voss et al42

* Themes were included if they were identified in at least two independent studies. Of the 41 studies reviewed inthis report, only 34 included any explicit comment about facilitators and barriers.

Patient Safety


Our systematic review had severallimitations. The literature examining theeffectiveness of educational interventionsin QI and PS exhibited substantialheterogeneity in terms of the contentdelivered, educational methods used,learners targeted, and learning outcomesreported. Also, many curricularevaluations involved weak study designs,occurred in single centers, had smallnumbers of learners, and often exhibitedother methodological concerns.Consequently, we did not regardquantitative synthesis as appropriate.

Our thematic textual analysis of all of theincluded curricular reports identified anumber of potentially important factorsthat promote or hinder implementationefforts. However, most of thosereports did not have the identificationof facilitators and barriers toimplementation as their primary aim.Consequently, authors may not haverecognized or reported aspects of thecurricular implementation in a systematicfashion. Moreover, the vast majority ofreports did not comment on the degreeto which curricula had been sustained.

Conclusions

Improving the quality and safety ofpatient care has gained widespreadacceptance as a central activity for thehealth care system. Clinicians will beexpected to have acquired coreconcepts in QI and PS in order to applythem to improve their personalpractices and help support institutionalimprovement efforts. Consequently, aconsensus has emerged that QI and PSshould be broadly taught to trainees,with ACGME9 and CanMEDS10

mandating such education and somestudents actively requesting it.69

Despite this emerging consensus, in2006, few medical schools in the UnitedStates and Canada reported havingexplicit curricula in QI and PS,70

although it is likely that now (2010) there aremore schools with such curricula.

The existing literature indicates thateducational curricula focused on QIand PS are generally well accepted bytrainees, effectively improve knowledgein these domains, and can even lead toimportant improvements in processesof care. Programs undertaking thedevelopment of curricula in QI or PSmust recognize the significant time

pressures and competing educationaldemands for trainees, as well as therequirements for adequate numbers offaculty with appropriate expertise andsupport for their contributions. Tosucceed, these curricula requireengagement of educational andorganizational stakeholders to promoteadoption. Future research must bettercharacterize the learner, faculty, andinstitutional factors that facilitate orhinder the promotion of sustainededucational efforts focused on QI andPS for medical students andpostgraduate trainees.

Funding/Support: Dr. Wong received anhonorarium from the Association of the Faculties ofMedicine of Canada to write an earlier version ofthis review that appeared as a chapter in amonograph entitled “The Future of MedicalEducation in Canada.” Dr. Shojania receives generalsalary support from the Government of CanadaResearch Chairs Program. Neither funding bodyplayed any role in the design and conduct of thestudy; collection, management, analysis, andinterpretation of the data; or preparation, review, orapproval of the systematic review.

Other disclosures: None.

Ethical approval: Not applicable.

Disclaimer: The opinions expressed in this articleare those of the authors alone and do not reflectthe views of the Association of the Faculties ofMedicine of Canada.

Previous presentations: A preliminary version of thereview appears as a chapter in a monograph entitled“The Future of Medical Education in Canada” (http://www.afmc.ca/fmec/activities-env-literature.php),published by the Association of Faculties ofMedicine of Canada.

References1 Fletcher KE, Underwood W 3rd, Davis SQ,

Mangrulkar RS, McMahon LFJ, Saint S.Effects of work hour reduction on residents’lives: A systematic review. JAMA. 2005;294:1088 –1100.

2 Ogden PE, Sibbitt S, Howell M, et al.Complying with ACGME resident duty hoursrestrictions: Restructuring the 80-hourworkweek to enhance education and patientsafety at Texas A&M/Scott & White MemorialHospital. Acad Med. 2006;81:1026–1031.

3 Okie S. An elusive balance—Residents’ workhours and the continuity of care. N EnglJ Med. 2007;356:2665–2667.

4 Shojania KG, Fletcher KE, Saint S.Graduate medical education and patientsafety: A busy—and occasionallyhazardous—intersection. Ann Intern Med.2006;145:592–598.

5 Kennedy TJ, Regehr G, Baker GR, Lingard L.Preserving professional credibility: Groundedtheory study of medical trainees’ requests forclinical support. BMJ. 2009;338:b128.

6 Vidyarthi AR, Arora V, Schnipper JL, WallSD, Wachter RM. Managing discontinuity in

academic medical centers: Strategies for a safeand effective resident sign-out. J Hosp Med.2006;1:257–266.

7 Association of American Medical Colleges.Report X: Contemporary Issues in Medicine:Education in Safe and Effective PrescribingPractices. Medical School Objectives Project.Available at: http://www.aamc.org/meded/msop. Accessed March 24, 2010.

8 Association of American Medical Colleges.Report V: Contemporary Issues in Medicine:Quality of Care. Medical School ObjectivesProject. Available at: http://www.aamc.org/meded/msop. Accessed March 24, 2010.

9 Accreditation Council for Graduate MedicalEducation. ACGME Outcome Project.Common Program Requirements: GeneralCompetencies. Available at: http://www.acgme.org/outcome/comp/compCPRL.asp.Accessed March 24, 2010.

10 Frank JR, Danoff D. The CanMEDS initiative:Implementing an outcomes-based frameworkof physician competencies. Med Teach. 2007;29:642–647.

11 Scheele F, Teunissen P, Van Luijk S, et al.Introducing competency-based postgraduatemedical education in the Netherlands. MedTeach. 2008;30:248 –253.

12 Shojania KG, Levinson W. Clinicians inquality improvement: A new career pathwayin academic medicine. JAMA. 2009;301:766–768.

13 Boonyasai RT, Windish DM, Chakraborti C,Feldman LS, Rubin HR, Bass EB.Effectiveness of teaching qualityimprovement to clinicians: A systematicreview. JAMA. 2007;298:1023–1037.

14 Kohn LT, Corrigan JM, Donaldson MS, eds.To Err Is Human: Building a Safer HealthSystem. Washington, DC: National AcademyPress; 2000.

15 Best Evidence in Medical Education (BEME)Collaboration: Starting a Review. Available at:http://www2.warwick.ac.uk/fac/med/beme/startingreview/. Accessed May 10, 2010.

16 Kirkpatrick DL. Evaluation of training. In:Craig RL, Bittel LR, eds. Training andDevelopment Handbook. New York, NY:McGraw-Hill; 1967:87–112.

17 Hughes MT. Implementation. In: Kern DE,Thomas PA, Howard DM, Bass EB, eds.Curriculum Development for MedicalEducation: A Six-Step Approach. Baltimore,Md: The Johns Hopkins University Press; 1998.

18 Bechtold ML, Scott S, Nelson K, Cox KR,Dellsperger KC, Hall LW. Educational qualityimprovement report: Outcomes from arevised morbidity and mortality format thatemphasised patient safety. Qual Saf HealthCare. 2007;16:422–427.

19 Canal DF, Torbeck L, Djuricich AM.Practice-based learning and improvement:A curriculum in continuous qualityimprovement for surgery residents. ArchSurg. 2007;142:479 –482.

20 Coleman MT, Nasraty S, Ostapchuk M,Wheeler S, Looney S, Rhodes S. Introducingpractice-based learning and improvementACGME core competencies into a familymedicine residency curriculum. Jt Comm JQual Saf. 2003;29:238 –247.

21 Coyle YM, Mercer SQ, Murphy-Cullen CL,Schneider GW, Hynan LS. Effectiveness of agraduate medical education program forimproving medical event reporting attitude andbehavior. Qual Saf Health Care. 2005;14:383–388.

Patient Safety


22 Djuricich AM, Ciccarelli M, Swigonski NL. Acontinuous quality improvement curriculumfor residents: Addressing core competency,improving systems. Acad Med. 2004;79(10suppl):S65–S67.

23 Frey K, Edwards F, Altman K, Spahr N,Gorman RS. The ‘Collaborative Care’curriculum: An educational model addressingkey ACGME core competencies in primary careresidency training. Med Educ. 2003;37:786–789.

24 Gould BE, Grey MR, Huntington CG, et al.Improving patient care outcomes by teachingquality improvement to medical students incommunity-based practices. Acad Med. 2002;77:1011–1018.

25 Gunderson A, Tekian A, Mayer D. Teachinginterprofessional health science studentsmedical error disclosure. Med Educ. 2008;42:531–532.

26 Halbach JL, Sullivan LL. Teaching medicalstudents about medical errors and patientsafety: Evaluation of a required curriculum.Acad Med. 2005;80:600 –606.

27 Henley E. A quality improvement curriculumfor medical students. Jt Comm J QualImprov. 2002;28:42–48.

28 Holmboe ES, Prince L, Green M. Teachingand improving quality of care in a primarycare internal medicine residency clinic. AcadMed. 2005;80:571–577.

29 Kerfoot BP, Conlin PR, Travison T,McMahon GT. Web-based education insystems-based practice: A randomized trial.Arch Intern Med. 2007;167:361–366.

30 Madigosky WS, Headrick LA, Nelson K, CoxKR, Anderson T. Changing and sustainingmedical students’ knowledge, skills, andattitudes about patient safety and medicalfallibility. Acad Med. 2006;81:94 –101.

31 Mohr JJ, Randolph GD, Laughon MM, SchaffE. Integrating improvement competenciesinto residency education: A pilot project froma pediatric continuity clinic. Ambul Pediatr.2003;3:131–136.

32 Moskowitz E, Veloski JJ, Fields SK, Nash DB.Development and evaluation of a 1-dayinterclerkship program for medical studentson medical errors and patient safety. Am JMed Qual. 2007;22:13–17.

33 Newell P, Harris S, Aufses AJ, Ellozy S.Student perceptions of medical errors:Incorporating an explicit professionalismcurriculum in the third-year surgeryclerkship. J Surg Educ. 2008;65:117–119.

34 Ogrinc G, Headrick LA, Morrison LJ, Foster T.Teaching and assessing resident competence inpractice-based learning and improvement.J Gen Intern Med. 2004;19:496–500.

35 Ogrinc G, West A, Eliassen MS, Liuw S,Schiffman J, Cochran N. Integrating practice-based learning and improvement intomedical student learning: Evaluating complexcurricular innovations. Teach Learn Med.2007;19:221–229.

36 Oyler J, Vinci L, Arora V, Johnson J.Teaching internal medicine residents qualityimprovement techniques using the ABIM’spractice improvement modules. J Gen InternMed. 2008;23:927–930.

37 Patey R, Flin R, Cuthbertson BH, et al.Patient safety: Helping medical students

understand error in healthcare. Qual SafHealth Care. 2007;16:256 –259.

38 Peters AS, Kimura J, Ladden MD, March E,Moore GT. A self-instructional model toteach systems-based practice and practice-based learning and improvement. J GenIntern Med. 2008;23:931–936.

39 Tomolo A, Caron A, Perz ML, Fultz T, AronDC. The outcomes card. Development of asystems-based practice educational tool.J Gen Intern Med. 2005;20:769 –771.

40 Varkey P, Karlapudi SP, Bennet KE. Teachingquality improvement: A collaboration projectbetween medicine and engineering. Am JMed Qual. 2008;23:296 –301.

41 Varkey P, Reller MK, Smith A, Ponto J,Osborn M. An experiential interdisciplinaryquality improvement education initiative.Am J Med Qual. 2006;21:317–322.

42 Voss JD, May NB, Schorling JB, et al.Changing conversations: Teaching safety andquality in residency training. Acad Med.2008;83:1080 –1087.

43 Weingart SN, Tess A, Driver J, Aronson MD,Sands K. Creating a quality improvementelective for medical house officers. J GenIntern Med. 2004;19:861–867.

44 Ziegelstein RC, Fiebach NH. “The mirror” and“the village”: A new method for teachingpractice-based learning and improvement andsystems-based practice. Acad Med. 2004;79:83–88.

45 Cosby KS, Croskerry P. Patient safety: Acurriculum for teaching patient safety inemergency medicine. Acad Emerg Med. 2003;10:69 –78.

46 Esselman PC, Dillman-Long J. Morbidity andmanagement conference: An approach toquality improvement in brain injuryrehabilitation. J Head Trauma Rehabil. 2002;17:257–262.

47 Farquhar D, Myers K, Benjamin D. Educationin quality of care in an internal medicineresidency program. Acad Med. 2001;76:562.

48 Gosbee J. A patient safety curriculum forresidents and students: The VA healthcaresystem’s pilot project. ACGME Bulletin.November 2002:2–6.

49 Gould BE, O’Connell MT, Russell MT, PipasCF, McCurdy FA. Teaching qualitymeasurement and improvement, cost-effectiveness, and patient satisfaction inundergraduate medical education: The UME-21experience. Fam Med. 2004;36(suppl):S57–S62.

50 Krajewski K, Siewert B, Yam S, Kressel HY,Kruskal JB. A quality assurance elective forradiology residents. Acad Radiol. 2007;14:239 –245.

51 Paulman P, Medder J. Teaching the qualityimprovement process to junior medicalstudents: The Nebraska experience. FamMed. 2002;34:421–422.

52 Rosenfeld JC. Using the Morbidity andMortality conference to teach and assess theACGME General Competencies. Curr Surg.2005;62:664 –669.

53 Schillinger D, Wheeler M, Fernandez A. Thepopulations and quality improvementseminar for medical residents. Acad Med.2000;75:562–563.

54 Singh R, Naughton B, Taylor JS, et al. Acomprehensive collaborative patient safety

residency curriculum to address the ACGMEcore competencies. Med Educ. 2005;39:1195–1204.

55 Thompson DA, Cowan J, Holzmueller C, WuAW, Bass E, Pronovost P. Planning andimplementing a systems-based patient safetycurriculum in medical education. Am J MedQual. 2008;23:271–278.

56 Varkey P. Educating to improve patient care:Integrating quality improvement into amedical school curriculum. Am J Med Qual.2007;22:112–116.

57 Weeks WB, Robinson JL, Brooks WB,Batalden PB. Using early clinical experiencesto integrate quality-improvement learning intomedical education. Acad Med. 2000;75:81–84.

58 Wong RY, Hollohan K, Roberts M, Hatala R,Ma IW, Kassen BO. A descriptive report of aninnovative curriculum to teach qualityimprovement competencies to internalmedicine residents. Can J Gen Intern Med.2008;3:26 –29.

59 Morrison LJ, Headrick LA, Ogrinc G, FosterT. The quality improvement knowledgeapplication tool: An instrument to assessknowledge application in practice-basedlearning and improvement. J Gen InternMed. 2003;18(suppl 1):250.

60 Marinopoulos SS, Dorman T, RatanawongsaN, et al. Effectiveness of Continuing MedicalEducation. Available at: http://www.ahrq.gov/clinic/tp/cmetp.htm. Accessed March 24, 2010.

61 Belfield C, Thomas H, Bullock A, Eynon R,Wall D. Measuring effectiveness for bestevidence medical education: A discussion.Med Teach. 2001;23:164 –170.

62 Davis D. Does CME work? An analysis of theeffect of educational activities on physicianperformance or health care outcomes. IntJ Psychiatry Med. 1998;28:21–39.

63 Davis D, O’Brien MA, Freemantle N, WolfFM, Mazmanian P, Taylor-Vaisey A. Impactof formal continuing medical education: Doconferences, workshops, rounds, and othertraditional continuing education activitieschange physician behavior or health careoutcomes? JAMA. 1999;282:867–874.

64 Schouten LM, Hulscher ME, van EverdingenJJ, Huijsman R, Grol RP. Evidence for theimpact of quality improvementcollaboratives: Systematic review. BMJ.2008;336:1491–1494.

65 Wu AW, Lipshutz AK, Pronovost PJ.Effectiveness and efficiency of root causeanalysis in medicine. JAMA. 2008;299:685–687.

66 Brainard AH, Brislen HC. Viewpoint:Learning professionalism: A view from thetrenches. Acad Med. 2007;82:1010 –1014.

67 D’Eon M, Lear N, Turner M, Jones C. Perilsof the hidden curriculum revisited. MedTeach. 2007;29:295–296.

68 Gofton W, Regehr G. What we don’t know we areteaching: Unveiling the hidden curriculum. ClinOrthop Relat Res. 2006;449:20–27.

69 Nazem AG. A piece of my mind. Teach ushow. JAMA. 2008;300:2463–2464.

70 Alper E, Rosenberg EI, O’Brien KE, FischerM, Durning SJ. Patient safety education atU.S. and Canadian medical schools: Resultsfrom the 2006 Clerkship Directors in InternalMedicine survey. Acad Med. 2009;84:1672–1676.

Patient Safety


teaching quality improvement and patient safety to ... · and canmeds10,11 competency frameworks...

Documents