An Electronic Medical Record Intervention Increased NursingHome Advance Directive Orders and Documentation

Serge A. Lindner, MD,�w J. Ben Davoren, MD,�w Andrew Vollmer, RN,� Brie Williams, MD,�w

and C. Seth Landefeld, MD�w

OBJECTIVES: To develop an electronic medical recordintervention to improve documentation of patient prefer-ences about life-sustaining care, detail of resuscitation andtreatment-limiting orders, and concordance between theseorders and patient preferences.

DESIGN: Prospective before–after intervention trial.

SETTING: Veterans Affairs nursing home with an elec-tronic medical record for all clinical information, includingclinician orders.

PARTICIPANTS: All 224 nursing home admissions fromMay 1 to October 31, 2004.

MEASUREMENTS: Completion of an advance directivediscussion note by the primary clinician, clinician ordersabout resuscitation and other life-sustaining treatments,and concordance between these orders and documentedpatient preferences.

INTERVENTION: The electronic medical record wasmodified so that an admission order would specify resus-citation status. Additionally, the intervention alerted theprimary clinician to complete a templated advance directivediscussion note for documentation of life-sustaining treat-ment preferences.

RESULTS: Primary clinicians completed an advance direc-tive discussion note for five of 117 (4%) admissionspre-intervention and 67 of 107 (63%) admissions post-intervention (Po.001). In multivariate analysis, theintervention was independently associated with advancedirective discussion note completion (odds ratio 5 42, 95%confidence interval 5 15–120). Of patients who preferreddo-not-resuscitate (DNR) status, a DNR order was writtenfor 86% pre-intervention versus 98% post-intervention(P 5.07); orders to limit other life-sustaining treatmentswere written for 16% and 40%, respectively (P 5.01).

CONCLUSIONS: A targeted electronic medical recordintervention increased completion of advance directive

discussion notes in seriously ill patients. For patients whopreferred DNR status, the intervention also increased thefrequency of DNR orders and of orders to limit other life-sustaining treatments. J Am Geriatr Soc 55:1001–1006,2007.

Key words: advance directives; electronic medical record;computerized physician order entry; quality improve-ment; skilled nursing facility

Clear communication and documentation of patientpreferences are essential to optimal care of patients

who have complex illness or are near the end of life. None-theless, problems with implementation, effectiveness, spec-ificity, and accessibility have plagued documentation ofpatient preferences in advance directive forms.1–9 The ques-tion of whether to remove a feeding tube from TerriSchiavo, a Florida woman in a persistent vegetative state,captured public and political attention during a family bat-tle that might have been avoided had advance directivesbeen documented.10

Efforts to improve documentation of patient prefer-ences in advance directive forms have often been ineffec-tive.4,11,12 Other efforts succeeded in only 5% to 40% ofpatients, required complex interventions that did not in-volve patients and clinicians, and required additional per-sonnel that may not be sustainable or exportable to othersettings.13–15

The purpose of this study was to develop and test anelectronic medical record intervention to increase comple-tion of advance directive discussion notes, the detail of re-suscitation and treatment-limiting orders, and theconcordance between these orders and patient preferences.It was hypothesized that clinicians would write appropriatenotes and orders more frequently as a result of the inter-vention. To increase the potency of the intervention, infor-mation technology designed to fit into routine workflowwas used,16 the intervention was simplified as much as pos-sible, and standard approaches were applied to changingclinician behavior.17,18 The intervention required only atwo-step change in the electronic medical record and

Address correspondence to Serge Lindner, MD, Center For Senior Health,3015 Squalicum Parkway, Ste 100, Bellingham, WA 98225.E-mail: linserge@yahoo.com

DOI: 10.1111/j.1532-5415.2007.01214.x

From the �San Francisco VA Medical Center, San Francisco, California;wUniversity of California, San Francisco, California.

JAGS 55:1001–1006, 2007r 2007, Copyright the AuthorsJournal compilation r 2007, The American Geriatrics Society 0002-8614/07/$15.00

prompted clinicians to document discussion of advance di-rectives and treatment-limiting orders in a standardizedprocess. This process differed from completion of advancedirective forms, rarely undertaken by clinicians, and insteadrelied on enhanced documentation of patient preferences inan advance directive discussion note. The intervention wasdesigned to enhance accessibility and continuity of advancedirective notes and orders across care sites (information onthe Veterans Affairs (VA) Computerized Patient RecordSystem is available at http://www1.va.gov/CPRSdemo/)and to facilitate implementation of the intervention in oth-er facilities with comparable information systems.

METHODS

Participants

All 224 admissions to the San Francisco VA Nursing Homebetween May 1 and October 31, 2004, were studied. The14 clinicians involved in these admissions included ninestaff physicians, three geriatrics fellows, and two nursepractitioners supervised by attending physicians. Therewere no exclusions. The study site has used an electronicmedical record for all clinical information, including clini-cian orders, for more than 10 years.

Intervention

The intervention consisted of two changes to the electronicmedical record. First, to write an admission order, the cli-nician was required to choose between four orders aboutresuscitation in the event of a cardiopulmonary arrest: fullresuscitation, do not resuscitate (DNR) and do not intu-bate, intubate if necessary but do not cardiovert, and cardi-overt if necessary but do not intubate (Figure 1). Theseoptions represented the resuscitation orders that were incommon use at the medical center and were consistent withCalifornia advance directive forms. The clinician wouldthen choose from a drop-down menu of further treatment-limiting orders regarding patient preferences for transferfrom nursing home to emergency or acute hospital care,

intravenous fluid or antibiotic use, and enteral (tube) feed-ing. The order menu was based on the Physician Orders forLife-Sustaining Treatment template used in Oregon andWashington.14,15,19–22 This intervention did not activelytarget the patient–clinician discussion regarding suchgoals of care. It was assumed that clinicians followed VApolicy; an order to withhold resuscitation should reflect thepreference stated by the patient or by the patient’s legalsurrogate.

Second, 24 hours after admission, the electronic med-ical record automatically alerted the primary clinician tocomplete a templated advance directive discussion note fordocumentation of life-sustaining treatment preferences andgoals of care. To ensure that the clinician most familiar withthe patient would complete this note, the primary clinicianreceived the alert even when another clinician admitted thepatient. Clinicians would receive the alert every time theysigned in to the electronic medical record until the note hadbeen completed. The templated note included fields forentry of information on surrogate decision-makers; datesand contents of prior written advance directives; overallvalues, preferences, and goals of care; and expressedwishes regarding resuscitation, artificial nutrition and hy-dration, transfers to acute hospital or emergency care, andautopsy. The intention of the templated note was to provideaccurate, comprehensive, and current informationabout patients’ preferences in the context of the currentadmission.

It was important for the intervention to address treat-ment-limiting orders and patients’ general preferences fortreatment. Resuscitation and other treatment-limiting or-ders are typically referred to only when a patient is in severedistress, such as in a ‘‘code blue’’ situation. These ordersprovide highly useful and specific guidance to clinical staffduring an emergency, but more-subtle questions about gen-eral preferences for treatment are better addressed in a cli-nician’s advance directive discussion note and are mostuseful when referred to during consideration of treatmentoptions.

Figure 1. Intervention part 1Fadmission orders in the electronic medical record, clinician order-entry screen. Nursing Home CareUnit admission order entry required clinicians to specify resuscitation status (arrow 1). The clinician then selected from a menu ofother treatment-limiting orders (not shown). Signed treatment-limiting orders appeared at the top of the orders screen (arrow 2).Advance directive notes could be viewed by clicking on ‘‘Postings,’’ available in every screen in every care site within the medical center(arrow 3).

1002 LINDNER ET AL. JULY 2007–VOL. 55, NO. 7 JAGS

The intervention was designed to enhance accessibilityof treatment-limiting orders and the related advance direc-tive discussion note. Treatment-limiting orders, previouslyburied in a chronological list of other treatment orders andoften written in nonstandardized free text, appeared in anew section at the top of the patient orders list labeled‘‘Code Status’’ (Figure 1). A preexisting mechanism in theelectronic record ensured accessibility of any note titled‘‘Advance Directive’’; by selecting the ‘‘Postings’’ box (Fig-ure 1, upper right-hand corner), it was possible to review alladvance directive notes immediately in every screen in everycare site. The intervention encouraged documentation ofpatient preferences in these uniformly placed and easily ac-cessible notes, rather than the general progress notes thatwere routinely used before the intervention. This interven-tion was integrated into the clinician’s routine work of en-tering notes and orders into the electronic medical recordfor patient admissions. The familiarity of clinicians withoccasional changes in the electronic medical record facil-itated their adoption of the intervention.

Multiple design cycles were performed to arrive at themost effective and streamlined design feasible with theelectronic medical record’s technology. The nursing homedirector helped set the start date for the intervention andcoordinated meetings with the clinical staff to collaborateon any further changes. During this design phase, localnursing home and medical center committees discussed theimplications of the proposed intervention and approved itsuse. Overall, discussion, design, and coordination with theabove parties consumed the majority of our efforts, whereasimplementation of the intervention itself required the briefassistance of a programmer.

MEASUREMENTS

The primary outcome was completion of an advancedirective discussion note by a patient’s primary clinicianduring the admission. To determine whether completion ofan advance directive discussion note indicated the additionof clinically relevant information to the medical record, thetext of these notes was examined. Ninety-four percent of 67advance directive discussion notes written by the primaryclinician documented discussion of patient preferences, and96% were written within 1 month of the admission. Dataon advance directive forms were not available for this study.

The presence of DNR and other treatment-limiting or-ders was also determined, as was concordance betweenDNR orders and patient preferences documented in theadmission or advance directive discussion notes. Examplesof other treatment-limiting orders include those regardingintravenous or enteral feeding or hydration, antibiotics, andtransfer to emergency or acute hospital care.

Sample Size

Assuming that advance directive notes would be completedin 10% of patients in the pre-intervention group, a samplesize of 99 in each group provided 80% power to detect anabsolute difference of 15% between groups (with two-sideda5 0.05) in the proportion of patients with advance direc-tive notes.

Assignment Method

The intervention began on August 3, 2004, and applied toall admissions to the nursing home on or after that date.

Blinding

During the 3-month pre-intervention phase, clinicians wereunaware of the upcoming intervention, although shortlybefore instituting the intervention, the proposed changeswere presented to the affected clinicians, their feedback wassought, and they were oriented in its use. The patientsthemselves did not participate directly in the study and werethus unaware of the intervention.

Statistical Methods

The primary unit of analysis was each nursing homeadmission. The 224 admissions included 153 patients.Forty-nine of these 153 patients were transferred home orto the hospital and later readmitted to the nursing homeduring the study period one or more times, thus accountingfor 71 admissions (32% of the total 224 admissions).Analyses of the first nursing home admission of the 153patients yielded results similar to analyses with all 224admissions and are not reported separately.

Baseline characteristics of the pre- and post-interven-tion groups were compared using two-sample t tests forcontinuous variables and chi-square tests for dichotomousvariables. Logistic regression analysis was then used to de-termine unadjusted and adjusted odds ratios for effect of theintervention on primary and secondary outcomes. Variablesincluded in the adjustment were possible confounders apriori or were different (Po.15) between pre- and post-intervention groups. This permissive P-value threshold of.15 was chosen to ensure that all plausible confounders thathad been measured would be identified. Model checkingincluded sensitivity testing of outliers, as well as Pearsonand Hosmer-Lemeshow tests of fit. A conditional random-effects model was used to take into account clustering ofadmissions according to clinician or patient.

Finally, statistical process control methodology wasused to examine the effect of the intervention over timewhile controlling for observable secular trends. A p-typecontrol chart was constructed showing the percentage ofadmissions with a completed advance directive note foreach week of the study. The chart also displayed an uppercontrol limit for the pre- and post-intervention periods,corresponding to 3 standard deviations above the mean orthe upper limit of a 99.9% confidence interval.

All analyses used Stata software, Intercooled version8.0 (Stata Corp., College Station, TX).

The institutional review board for the University ofCalifornia at San Francisco and the San Francisco VeteransAffairs Medical Center granted approval for this study withwaiver of informed consent.

RESULTS

Participants

The pre-intervention group included 117 admissions of 88patients, and the post-intervention group included 107 ad-missions of 65 patients. Characteristics of patients admittedbefore and after the intervention were similar (Table 1).

ELECTRONIC INTERVENTION INCREASED ADVANCE DIRECTIVES 1003JAGS JULY 2007–VOL. 55, NO. 7

Advance Directive Discussion Note Completion Rates

Completion of an advance directive discussion note bya patient’s primary clinician increased from five of 117 (4%)admissions before the intervention to 67 of 107 (63%) ad-missions during the intervention (Po.001; odds ratio(OR) 5 38, 95% confidence interval (CI) 5 14–100;Table 2). The weekly rate of completion of advancedirective discussion notes was consistently close to zero be-fore the intervention and increased during the intervention(Figure 2). The completion rate of advance directive dis-

cussion notes rose during the intervention to 49% duringthe first month, 68% during the second month, and 71%during the third month (Po.001 for trend.) The interven-tion remained associated with completion of an advancedirective discussion note in hierarchical analyses of admis-sions clustered at the level of the individual clinician(OR 5 73, 95% CI 5 22–246) and the individual patient(OR 5 31.5, 95% CI 5 10–97) and in a multivariate anal-ysis of admissions adjusted for age, race, sex, dementia,religion, admitting service, and DNR status (OR 5 42, 95%CI 5 15–120).

Concordance and Detail of Clinician Orders with PatientPreferences

Review of admission and advance directive notes showedthat patients preferred not to be resuscitated in the event ofcardiopulmonary arrest in 44 of 117 (38%) admissions be-fore the intervention and in 40 of 107 (37%) admissionsafter the intervention (Table 1). Of these 84 admissions, theproportion with a concordant order not to resuscitate in theevent of cardiopulmonary arrest was 86% (kappa statis-tic 5 0.79) before the intervention and 98% (kappa statis-tic 5 0.98) during the intervention (P 5.07; Table 2). Ofpatients with an admission order not to be resuscitated inthe event of cardiopulmonary arrest, all had indicated theirpreference not to be resuscitated according to the admissionand advance directive discussion notes. The proportion ofadmissions with other treatment-limiting orders rose from16% before the intervention to 40% during the interven-tion (P 5.01; Table 2).

DISCUSSION

An electronic medical record intervention with computer-ized clinician order entry increased the rate of cliniciancompletion of advance directive discussion notes from 4%to 63%, with higher completion rates throughout the studyperiod. The intervention was designed to fit smoothly intothe routine admission and daily care processes and to assureaccessibility of advance directive documentation in all sitesof the health system. The intervention also led to 98%concordance between clinician orders and patient prefer-ences not to be resuscitated. For patients whose preferencenot to be resuscitated was documented in the medical re-cord, the intervention increased the proportion with addi-tional treatment-limiting orders. (Documentation of theseorders approximates use of the Physician Orders for Life-Sustaining Treatment form currently used in Washingtonand Oregon.)

The effectiveness of this intervention may be dueto several factors: simultaneous adherence to scientificand quality improvement methods (such as Plan-Do-Study-Act cycles23–25), use of information technologyin a specific site with a limited number of clinicians,engagement of the participating clinicians rather thanancillary staff, and implementation in a receptive group ofclinicians.

This intervention and its effects differed from thoseof previous studies. Although others have focused morenarrowly on the advance directive form itself or on treat-ment-limiting orders, this study addressed clinician docu-mentation of patient preferences in advance directive

Table 1. Patient Characteristics

Characteristic

Pre-Intervention

(May–July 2004;

n 5 117

Admissions)

Post-Intervention

(August–October

2004; n 5 107

Admissions)

P-

Value

Demographic

Age,mean � standarddeviation

71 � 12.3 72 � 12.2 .57

Male, % 89 91 .66

Race or ethnicity, % .11

White 57 70

Black/AfricanAmerican

27 18

Other 16 12

Married, % 39 42 .68

1st-degree relativeinvolved, %

61 61 .99

English-speaking, % 100 97 .19

ReligionFChristian, % 77 65 .04

Education 4thanhigh school, %

53 55 .75

Clinical characteristics, %

Dementia, %� 26 23 .69

Dependent in 41activity of daily living, %w

58 59 .91

Enterally fed, % 3 6.5 .15

Do not resuscitatepreferred, %

38 37 .97

Health services utilization, %

Hospital admission inprevious month

69 72 .65

Emergencydepartment visit inprevious month

44 46 .73

Treatment service .45

Palliative 13 17

Long-term 9 13

Postacute medicine 30 18

Postacute surgery 14 19

Respite care 22 23

Radiation therapy 9 7

Intravenousantibiotics/other

3 3

�Folstein Mini-Mental State Examination score o24 or clinical diagnosis of

moderate-severe dementia.wDressing, eating, ambulating, toileting, and hygiene.

1004 LINDNER ET AL. JULY 2007–VOL. 55, NO. 7 JAGS

discussion notes as well as in completion of treatment-lim-iting orders. First, the improvement from 4% to 63% com-pletion of advance directive discussion notes by clinicianswas greater than in other studies;14 for example, computerreminders increased physician completion of advance di-rective forms from 4% to 15%.13 Second, many interven-tions were time consuming and complicated, and othersinvolved nurses and social workers without involving phy-sicians.13–15 The current intervention was simple, engagedclinicians directly, and used available technologies forwidespread and uniform implementation. In addition, al-though most studies of clinical reminders show ‘‘alert fa-tigue,’’ or decreased effectiveness over time,26,27 the resultsof the current study demonstrated improvement in inter-vention adherence over the 3-month study period.

Methodological Considerations

Several considerations increase confidence in the validity ofthe findings. The prospective design; the timing, rate, and

magnitude of the increase in completion of advance direc-tive discussion notes; the sustained increase in their com-pletion; and the lack of evidence of secular trends oranother cause for their increased completion all support thehypothesis that the intervention increased the completion ofadvance directive discussion notes. Statistical process con-trol, a powerful technique that helps to account for seculartrends in time series analyses,28–33 further increases confi-dence in attributing the increased completion rate to theintervention.

Several limitations in the study should be recognized.First, the prospective design used temporal allocation of theintervention rather than random allocation, and the studywas limited to a single site. Second, before the intervention,patient preferences may have been documented in progressnotes that were not found in the review or in advancedirective forms that were not available for this study.Nonetheless, the intervention increased the completion ofadvance directive discussion notes that were easily identifi-able and, therefore, readily available for clinical use. Third,patient preferences for end-of-life care were determinedfrom clinician documentation in the medical record, andpatient reports of their preferences were not independentlyascertained. Finally, the effects of the intervention on as-pects of the process of care were determined, but patientoutcomes such as resuscitation events and rates of hospitaltransfer or completion of advance directive forms were notassessed. Improved processes of care are nonetheless nec-essary steps in improving outcomes.34

CONCLUSIONS

A targeted electronic medical record intervention greatlyincreased completion of advance directive discussion notesin seriously ill patients. This effect was sustained over3 months, and orders were more often detailed andconcordant with patient preferences to limit treatment.After the intervention, the notes and orders were easilyaccessible across sites of care. Our findings indicate thatthe electronic medical record and quality improvementmethods can together improve communication about end-of-life care.

The intervention evaluated in this study may be one ofmany important processes for improving end-of-life care in

Table 2. Advance Directive Discussion Notes and Treatment-Limiting Orders, Pre- and Post-Intervention

Outcome

Pre-Intervention Post-Intervention Unadjusted Odds

Ratio (95% Confidence Interval)

for Interventionn (%)

All admissions

Admissions 117 (100) 107 (100)

Advance directive discussion note

Completed by primary clinician 5 (4) 67 (63) 37.5 (14.1–99.7)

Admissions of patients preferring DNR

Admissions 44 (100) 40 (100)

DNR order written� 38 (86) 39 (98) 6.2 (0.71–53.4)

Treatment-limiting orders writtenw 7 (16) 16 (40) 3.5 (1.26–9.83)

�Do not resuscitate (DNR), as determined by review of admission and progress notes.wTreatment-limiting orders were orders not to use specific treatments such as transfer to acute hospital, and intravenous or enteral alimentation.

Figure 2. Rate of advance directive discussion note completion,using P-type control chart. The intervention began during Week30. AD 5 advance directive discussion note completed by primaryclinician; UCL1 5 upper control limit before the intervention,which corresponds to the upper bounds of a 99.9% confidenceinterval; UCL2 5 upper control limit after the intervention.

ELECTRONIC INTERVENTION INCREASED ADVANCE DIRECTIVES 1005JAGS JULY 2007–VOL. 55, NO. 7

settings with electronic medical records. Future researchshould examine clinical outcomes such as concordance be-tween care received and care preferred. Following the exam-ple set by states such as Oregon,35 public discourse, legislativechanges, and changes in healthcare processes may all con-tribute to widespread improvements in end-of-life care.

ACKNOWLEDGMENTS

Dr. Lindner was a Fellow and Dr. Landefeld is a SeniorScholar, both in the VA National Quality Scholars Program.Dr. Davoren is Director of Clinical Informatics, and Mr.Vollmer is the senior Clinical Applications Coordinator,both at the San Francisco VA Medical Center. Dr. Williamsis a Fellow in Geriatrics at the University of California, SanFrancisco. Dr. Lindner has presented this work at the Pres-idential Poster Session of the 2005 American Geriatrics So-ciety Annual Meeting.

Financial Disclosure: This work was supported in partby grants from the National Institute on Aging (AG000912,AG000212), the John A. Hartford Foundation, Inc. (2002–0013, 2003–0244), and the Health Services Research andDevelopment Service, Department of Veterans Affairs(Health Services Research Enhancement Award Program).

Author Contributions: Serge Lindner: primary conceptand design, acquisition of subjects and data, analysis andinterpretation of data, and preparation of manuscript. C.Seth Landefeld: concept and design, analysis and interpre-tation of data, and preparation of manuscript. J. Ben Dav-oren and Andrew Vollmer: concept and design, acquisitionof data, and preparation of manuscript. Brie Williams:concept and design, acquisition of data, analysis of data,and preparation of manuscript.

REFERENCES

1. Morrison RS, Olson E, Mertz KR et al. The inaccessibility of advance direc-

tives on transfer from ambulatory to acute care settings. JAMA

1995;274:478–482.

2. Wenger NS, Oye RK, Desbiens NA et al. The stability of DNR orders on

hospital readmission. The SUPPORT Investigators. Study to Understand

Prognoses and Preferences for Outcomes and Risks of Treatments. J Clin Eth-

ics 1996;7:48–54.

3. Baker DW, Einstadter D, Husak S et al. Changes in the use of do-not-resus-

citate orders after implementation of the Patient Self-Determination Act. J Gen

Intern Med 2003;18:343–349.

4. The SUPPORT Principal Investigators. A controlled trial to improve care for

seriously ill hospitalized patients. The study to understand prognoses and

preferences for outcomes and risks of treatments (SUPPORT). JAMA

1995;274:1591–1598.

5. Gregory JJ, Dunn D. Implementation of the Patient Self-Determination Act in

a community hospital. N Engl J Med 1995;92:438–442.

6. OBRA OBRAoPln. Omnibus Budget Reconciliation Act of 1990, 1990.

7. Virmani J, Schneiderman LJ, Kaplan RM. Relationship of advance directives

to physician-patient communication. Arch Intern Med 1994;154:909–913.

8. Lo B. Resolving Ethical Dilemmas: A Guide for Clinicians. Philadelphia, PA:

Lippincott, Williams, & Wilkins, 2000.

9. Greco PJ, Schulman KA, Lavizzo-Mourey R et al. The Patient Self-Determi-

nation Act and the future of advance directives. Ann Intern Med

1991;115:639–643.

10. Annas GJ. ‘‘Culture of life’’ politics at the bedsideFthe case of Terri Schiavo.

N Engl J Med 2005;352:1710–1715.

11. Sulmasy DP, Song KY, Marx ES et al. Strategies to promote the use of advance

directives in a residency outpatient practice. J Gen Intern Med 1996;11:657–

663.

12. Wenger NS, Rosenfeld K. Quality indicators for end-of-life care in vulnerable

elders. Ann Intern Med 2001;135:677–685.

13. Dexter PR, Wolinsky FD, Gramelspacher GP et al. Effectiveness of computer-

generated reminders for increasing discussions about advance directives and

completion of advance directive forms. A randomized, controlled trial. Ann

Intern Med 1998;128:102–110.

14. Morrison RS, Chichin E, Carter J et al. The effect of a social work intervention

to enhance advance care planning documentation in the nursing home. J Am

Geriatr Soc 2005;53:290–294.

15. Brown JB, Beck A, Boles M et al. Practical methods to increase use of advance

medical directives. J Gen Intern Med 1999;14:21–26.

16. Garg AX, Adhikari NK, McDonald H et al. Effects of computerized clinical

decision support systems on practitioner performance and patient outcomes:

A systematic review. JAMA 2005;293:1223–1238.

17. Davis DA, Thomson MA, Oxman AD et al. Changing physician performance.

A systematic review of the effect of continuing medical education strategies.

JAMA 1995;274:700–705.

18. Grimshaw JM, Shirran L, Thomas R et al. Changing provider behavior:

An overview of systematic reviews of interventions. Med Care 2001;39:

II2–45.

19. Lee MA, Brummel-Smith K, Meyer J et al. Physician orders for life-sustaining

treatment (POLST): Outcomes in a PACE program. Program of All-Inclusive

Care for the Elderly. J Am Geriatr Soc 2000;48:1219–1225.

20. Tolle SW, Tilden VP, Nelson CA et al. A prospective study of the efficacy of the

physician order form for life-sustaining treatment. J Am Geriatr Soc

1998;46:1097–1102.

21. Molloy DW, Guyatt GH, Russo R et al. Systematic implementation of an

advance directive program in nursing homes: A randomized controlled trial.

JAMA 2000;283:1437–1444.

22. Cantor MD. Improving advance care planning: Lessons from POLST. Physi-

cian Orders for Life-Sustaining Treatment. J Am Geriatr Soc 2000;48:1343–

1344.

23. Speroff T, O’Connor GT. Study designs for PDSA quality improvement

research. Qual Manag Health Care 2004;13:17–32.

24. Speroff T, James BC, Nelson EC et al. Guidelines for appraisal and publica-

tion of PDSA quality improvement. Qual Manag Health Care 2004;13:

33–39.

25. Batalden PB, Stoltz PK. A framework for the continual improvement of health

care: Building and applying professional and improvement knowledge to test

changes in daily work. Jt Comm J Qual Improv 1993;19:424–447; discussion

448–452.

26. Saleem JJ, Patterson ES, Militello L et al. Exploring barriers and facilitators to

the use of computerized clinical reminders. J Am Med Inform Assoc

2005;12:438–447.

27. Sequist TD, Gandhi TK, Karson AS et al. A randomized trial of electronic

clinical reminders to improve quality of care for diabetes and coronary artery

disease. J Am Med Inform Assoc 2005;12:431–437.

28. Lee KY, McGreevey C. Using comparison charts to assess performance mea-

surement data. Jt Comm J Qual Improv 2002;28:129–138.

29. Lee K, McGreevey C. Using control charts to assess performance measurement

data. Jt Comm J Qual Improv 2002;28:90–101.

30. Benneyan JC. Use and interpretation of statistical quality control charts. Int

J Qual Health Care 1998;10:69–73.

31. Alemi F, Neuhauser D. Time-between control charts for monitoring asthma

attacks. Jt Comm J Qual Saf 2004;30:95–102.

32. Norberg A, Christopher NC, Ramundo ML et al. Contamination rates of

blood cultures obtained by dedicated phlebotomy vs intravenous catheter.

JAMA 2003;289:726–729.

33. Shahian DM, Williamson WA, Svensson LG et al. Applications of statistical

quality control to cardiac surgery. Ann Thorac Surg 1996;62:1351–1358; dis-

cussion 1358–1359.

34. Batalden PB, Nelson EC, Roberts JS. Linking outcomes measurement to con-

tinual improvement: The serial ‘‘V’’ way of thinking about improving clinical

care. Jt Comm J Qual Improv 1994;20:167–180.

35. Teno JM. Looking beyond the ‘‘form’’ to complex interventions needed to

improve end-of-life care. J Am Geriatr Soc 1998;46:1170–1171.

1006 LINDNER ET AL. JULY 2007–VOL. 55, NO. 7 JAGS