changing clinical practice: which interventions work?

Thc .lourna/ of’Continrtiti,q Education it! the Health Prrfi.ssion.s. Volume 13, pp. 273-2XX. Printed in the U.S.A. Copyright 0 1993 The Alliance for Continuing Medical Education and the Society of Medical College Directors of Continuing Medical Education. All rights reserved.

Review Changing Clinical Practice: Which Interventions Work?

ROBYN TAMBLYN, PH.D. RENALDO BATTISTA, M.D., S.C. .D. Department of Medicine Department of Epidemiology and Biostatistics McGill University Montreal, Quebec, H3A 1 A3

Abstract: Changing clinical practice is a complex challenge. A framework is presented for integrating the perspectives of continuing education, technology assessment, and quality assurance. The effectiveness of cwrent interventions is reviewed. We describe what is known about practice determinants, the most important of which are physician age, clinical training, knowledge, skill, personal characteristics, practice setting, arid reimbursement policies. The rehtive importance of these factors varies and seems to be condition-specif c, an important point to be considered in the selection of pi-actice interventions.

Interventions aimed at having an impact on the practice setting (e.g., practice aids or audit programs), 01- on reimbursement policy (e.g., the fee allowed for a service) are more likely to bring about a change in practice than are interventions aimed at changing physician knowledge or skill. This is attributed to the greater relevance of these interventionsfor the individual physician, and to the opportunities provided for practicing them and for feedback. The cost and .feasibility of using the various types cf interventions to bring about practice change hmte to be carefully Haighed against the expected benefit f o r patients. Those seeking to select strategies that will opti- mize the effectiveness of interventions will need to consider the relative importance of physician abilities and attitudes, pi-actice harriers, and economic disincentives to practice change. Key Words: Determinants of clinical practice, changing practice patterns, incentives for change

Introduction Changing clinical practice is a complex endeavor. The sequence of activities that leads to modified practice behavior includes the proper assessment of

27 3

Tumhlyn and Battista

health care technology (defined in its broadest sense), the dissemination of this information to practitioners, appropriate incentives to enhance adoption of new practice behaviors, and the assessment and monitoring of the changes needed to translate new information into clinical practice.

The fields of technology assessment, continuing education, and quality assessment and assurance have evolved independently with minimal inter- action. Each of these complementary fields has separate societies and jour- nals that exist in relative isolation. The purpose of this paper is to bridge this gap and to develop a framework that draws from the collective wisdom ac- cumulated in these fields in order to design comprehensive approaches to changing clinical practice.

We regrouped the factors affecting clinical practice, using Green’s model,’ into predisposing, enabling, and reinforcing factors (see Figure I ). Predisposing juc~tors include such things as the practitioners’ knowledge and skills, as well as the socio-demographic characteristics that predispose them to practice in a certain manner. Enabling fuctors facilitate the initiation of new behavior. Reinforcing Jirctors are those that sustain behavior over time. Enabling and reinforcing factors include elements such as the characteristics of the practice setting, the patient population, and prevailing reimbursement policies.

Determinants of clinical practice are described for each of these groups. Intervention strategies aimed at each of these factors are presented, and their effectiveness is discussed in light of the existing literature. Literature for this review was identified through the McMaster series of annotated bibliogra- phies,’-4 supplemented by an Index Medicus review for the last two years.

Our purpose is to clarify the state of knowledge in this field and to present a comprehensive spectrum of strategies from which decision makers can choose in pursuing a specific change in practice behavior.

Predisposing Factors Determinants Two types of predisposing determinants have been identified. The first

relates to the practitioner’s ability to perform clinically-his or her clinical competence-usually described and measured in terms of its assumed pre- requisites: knowledge, skills, judgment, and attitudes.5.6 The second, over- lapping group of determinants encompasses the practitioner’s socio- demographic characteristics: age, gender, medical training, religious and so- cioeconomic background, and educational activities.

Physician, age, and type and length of clinical training have been consistently associated with the quality of care delivered.7-l9 Better quality of

274

Changing Clinical Practice: Which Interventions Work?

Figure 1 Determinants of Practice Patterns and Related Interventions to Change Practice

CLINICAL PRACTICE

Determinants C I i n ical Sociodemographic Practice Patient Reimbursement Competence Characteristics Setting Population Policy

Interventions knowledge restricVmonitor practice aids knowledge remuneration skill practice standards and expectations criteridamount

feedback method administrative policy

care is provided by younger physicians who have had more clinical training in the area studied. Specialty certification has been associated with better quality of care, in some but not all s tudie~.7-~9~~- '7 Similar associations have been observed for recertification tests of medical While these findings suggest that better medical knowledge is the reason for higher quality of care scores among younger physicians, the relationship is far from straightforward. Relationships between medical knowledge, scores on clinical competence tests, and practice performance are positive, but uniformly weak.21-26 Although medical knowledge can be significantly improved through educational interventions, this improvement does not necessarily result in a commensurate improvement in practice performance (see next sec- tion). Studies in this area are limited, but they suggest that enabling and reinforcing factors have a major influence on the level at which a physician performs in day-to-day practice.

Other physician characteristics, such as gender,27-*8 religious and so- cioeconomic background,2y and attitudes about patient care,3OJ * have been associated with differences in quality and type of services delivered. These influences appear to be condition-specific, being noted especially in such areas of practice as preventive and psychiatric management. The mechanism by which these physician attributes create differences in practice is unclear;

275

Tamhlyn and Battista

they may represent cultural and attitudinal differences that predispose an individual to assume a certain pattern of practice.

Interventions and Their Effectiveness Two types of interventions have been employed to exert influence on

predisposing factors: (a) those that use socio-demographic data to set policies that restrict those physicians who are at greater risk of providing a poor quality of care, and (b) those that are intended to improve physician performance through the manipulation of clinical ability (competence).

Strategies used to reduce or diminish practice by incompetent physicians are listed in Table 1. For the most part, formal evaluation of these strategies is lacking. In the few studies in which CME participation has been evaluated,”32 no relationship has been found between the number of CME cred- its and the quality of care delivered.

The most common methods used to improve physicians’ clinical competence have been aimed at enhancing medical kn0wledge.33-~7 These strategies are shown in Table 1. Short-term knowledge gain has been frequently reported with interventions designed to remedy knowledge deficiencies. Characteristics of interventions that produce the best short-term knowledge gains and the least rapid deterioration in knowledge over short-term follow- up are: active participation, a narrowly defined subject area, opportunities for reinforcement (e.g., self-assessment, group discussion), and the organization of material around clinical problems.

Changes in medical knowledge are not necessarily associated with changes in performance, yet positive changes in performance are observed when interventions to change knowledge are coupled with interventions to re- inforce application in the practice setting (e.g., audit).42147,50-51,68-7*

Performance improvement is striking and consistent when one-to-one con- sultative contact is used to remedy knowledge deficiencies, especially when the consultant is a credible clinical resource.52-53, 57-5*,68-7* A variation on this one-to-one approach is to identify and target interventions at the physician (the educational influential) who provides local professional leadership in the medical community; this approach seems to be a more efficient and equally effective means of changing the practice patterns of local phy~icians.~3

Two strategies that have been used to improve cognitive, psychomotor, and interpersonal skills are workshops and intemships/traineeships.73-85 Short-term improvement has been demonstrated with both approaches,7’-75,77.*~,8*-~5 but some studies have noted deterioration in skills over time.75,77,8*,84 Again, cou- pling skill training with reinforcement in the practice setting has been found effective in reinforcing the desired practice behavi0r.~5

276


Table 1 The Effectiveness of Strategies Used to Change Practice

STRATEGY A C C E P T A N C E ~ S E KNOWLEDGE GAIN PRACTICE CHANGE OUTCOME

Practice Restriction Mandatory CME ? IE IE c?

Recertification ? '1 ? *? Practice Audit Hi-Risk Groups ? ? ? ? Training /Certification ? ? <? #? Requirements for Hospital Privileges Clinical Competence Knowledge Lecture Print Material Journal Data Banks One-to-one Skills Workshops Traineeships

Practice Aids Consultation Algorithms

C ? C IE 7 ? ? ? E #?

E E C <? E E E ,?

ACCEPTANCEhSE QUALITY PERFORMANCE RESOURCE USE OUTCOME

Chart ReminderProtocols E Charting Format E

Standards & Feedback Standard Specification E Chart Audit E Administrative Policy Computerized Records C Restrict Resources ? Form Change ?

E E E E

E IE

E E

C E '?

Patient Population Knowledge Change ? E Remuneration Policy Criteria for Payment ? Amount of Payment ? Method of Payment 9

E ? E

E ? ? <? E ,? 0 ?

? ? E ?

'? ,? E ? E ?

Notes: Effectiveness has been categorized according to four outcomes: physician acceptance/use, change in knowledge, change in day-to-day performance, and change in patient outcome. Effectiveness has been assessed by the authors as being: ( 1 ) not reported (?) (2) contradictory (C) (3) consistently effective in reported studies (E) or (4)consistently ineffective in reported studies (IE). This evaluation schema provides a crude estimate of reported effectiveness. Most strategies are evaluated in a specific area of practice. Differences in their effectiveness may be present when applied to other areas of practice where other determinants of practice are operating.

277


Enabling and Reinforcing Factors Determinants Two characteristics of a physician’s practice setting have been consis-

tently found to be associated with quality of care: type of practice and hospital affiliation. After adjustment for age, physicians in group practice7.””-1’.86-91 and those with teaching affi l iat ion~’~~16--’7,~2-~8 have better quality-of-care scores. W i l l i a m ~ o n , ~ ~ studying drug innovations, found that those in group practice adopted new drugs more quickly than those in solo practice. The reasons for these differences are unclear. One possibility is that physicians with lower levels of ability elect to enter solo practice and are less apt to apply for posts in teaching hospitals. Or, it may be that physicians in group practice or teaching settings may simply benefit from their ex- posure to greater opportunities for collegial input and performance review.

The population of patients served by physicians seems to influence their approach to management and the quality of care rendered. Infrequent expo- sure to specific kinds of patient problems has been associated with lower quality of care and even with higher case fatality r a t e ~ . ~ ~ , ~ 5 , 9 6 , 1 0 0 - ~ ~ ~ The age mix of the patient population has also been noted to influence management styles, with more conservative approaches to health problems being noted among physicians who have a greater proportion of elderly in their practice.28

Finally, there is evidence that economic factors influence the physician’s approach to medical management.86.105 Physicians who work in areas with a high density of doctors and those who report incomes over $100,000 tend to be more aggressive in their management approach and to employ interventions with higher levels of remuneration.lo6 Salaried physicians are more likely than fee-for-service physicians to use interventions with low econoniic returns, such as preventive services.27~107 Physicians who have invested in diagnostic and laboratory equipment are more apt to use tests that require those services.

Interventions and Their Effectiveness Practice Setting Interventions Three categories of interventions that are used to make changes in practice

through modifications in the care setting are practice aids,52.58,108-121 audit programs,I22-137 and administrative policies.138-148 The types of approaches in each of these categories are shown in Table 1.

Certain practice aids have been consistently associated with short-term improvement in practice performance, including ~onsultation,52.58.108-1~~,~ 1 1

particularly when the expert is proactive in providing assistance, practice

278


protocols, and chart reminders.s11112-l14,116-11* Thus far, evaluations of the impact on quality of diagnosis and management created by using practice algorithms or the problem-oriented method of recordkeeping are too limited to draw conclusions about their effectiveness. 1 19-121

Most studies of audit programs-whether they look at the entire chart or focus on diagnostic testing or medication use-show they are effective in changing per f~rmance . I**-~~~ Factors that seem to improve the effectiveness of this type of intervention include: participation in the formulation of audit standards,l2* individualized feedback, 125-1283136-137 comparison of individual results with those of peers,123 and administrative policy to take action on substandard performance. 124

Although positive changes in performance can occur without using interventions to update medical knowledge, the magnitude of improvement seems to be greater when one-to-one consultation, seminars, or print material are used as supplemental means to address knowledge deficiencies.129 There is some evidence that practice change is not sustained after the intervention is withdrawn,I23 suggesting that ongoing surveillance may be necessary with this kind of approach.

Changes in administrative systems and policies have been evaluated for their effectiveness in reducing unnecessary use of health care resources and improving the efficiency of care delivery. Strategies in this group are shown in Table 1. Reductions in admissions, in length of stay, and in use of diagnostic and lab tests have all been reported to occur when policies have been put in place to reduce the access to resources or to require justification for their ~tilization.1~3~38-45 When analyzed, these reductions seem to be ac- companied by an increase in the appropriateness of use,138-14* with most reductions occurring in situations where use is discretionary.142 Only scattered data are available on interventions used to improve the efficiency of care. The provision of summary drug profiles does not appear to diminish the time devoted to discussion of drug-related issues or prescription decisions.145-147 Although computerized records appear to influence performance to some degree, data are not available on their impact on the quality or efficiency of medical management. 148

Patient Population Interventions Although strategies targeted at patient populations may provide an ef-

fective means of changing both physician behavior and patient outcomes, only two studies have reported an evaluation of applying such strategies. Little40 evaluated the effect of simultaneously providing physicians and patients with information on the use of alcohol in pregnancy in an uncontrolled

279


pretest/posttest design, and identified some degree of change (self-reported) in physician counseling advice and in patient information. Lindsay14’ found significant improvement in smoking cessation when a combined physician and patient education approach was used in comparison to patient education alone. In general, the effectiveness of this category of strategies remains rel- atively unexplored.

Remuneration Policy Interventions Although it is recognized that in systems that pay physicians on a fee-

for-service basis, manipulation of the criteria for payment or the actual amount paid for a service are a powerful means of altering practice behavior, such manipulations have rarely been the subject of empirical investiga- tion.150 Success of such manipulations can be measured by showing a predictable change in the frequency with which certain services are delivered after changes in the fee schedule. An example is the study by Brooks, et al.,ss who reported a significant reduction in the use of injections by general practitioners after changing the remuneration criteria for this service.

Different patterns of practice have been observed in the care delivered to equivalent groups of patients by physicians who are salaried versus those paid on a fee-for-service basis.151-155 These differences have given rise to the hypothesis that changing the method of physician payment can be an effective method of bringing about change in practice (e.g., increasing the use of preventive services). Since randomization of physicians to fee payment method would be required to obtain a valid answer to this question, no study to date has addressed the hypothesis.

Discussion In general, interventions that are designed to modify enabling and reinforcing factors seem to be more effective than those aimed at changing the predisposing factors that constitute clinical competence. One reason for the superior effectiveness of efforts focusing on enabling/reinforcing factors is their ability to incorporate basic learning principles in their design. From the educational perspective, three factors are important in maximizing learning gain: perceived relevance, the opportunity to practice, and the provision of corrective feedback.156 It is clear that these principles are more easily in- corporated into strategies that are applied in the practice setting. Chart reminders, for example, appear to work when, and only when, the desired practice is recommended (relevance), when there can be prompt application of the new behavior (practice), and when it can be coupled with an audit or feedback program dealing with the appropriateness of the new behavior (cor-

280


rective feedback). Interventions to change clinical competence, on the other hand, are generally carried out outside of the practice setting and patient con- text, seldom provide opportunity to practice the innovation, and as a result cannot provide feedback on the appropriateness of application of new patient care processes. Novel approaches have been used to improve the relevance, practice, and feedback with such interventions, and these have achieved somewhat better results.37,45-47~67

Although interventions deployed to modify enabling and reinforcing factors seem to be more effective, there is some evidence that attention to all three groups of factors will result in the most positive practice change. If this is the case, intervention design will need to be multifaceted, incorporating elements that address the socio-demographic characteristics and the clinical abilities of the targeted physician group, the obstacles and resources in the practice setting, the demographics of the patient population for whom the change is intended, and finally, an assessment of the economic consequences of a practice change for physicians, patients, and the health care system.

Attention to all these facets will reveal not only the predisposing and enabling factors that need to be addressed, but also the factors within the en- vironment that may prevent or inhibit continued reinforcement of practice change. Since determinants of practice vary in different clinical areas, we suspect that intervention design will also need to be condition specific. For example, attitudes and knowledge deficiencies may be important to address when changing management approaches in the elderly or among patients with terminal illness, whereas economic disincentives may be the most critical barrier to overcome in trying to increase the use preventive services.

Evaluation of potential determinants of practice within each of these groups may improve the likelihood that relevant interventions are designed, that the appropriate group is targeted, and that desired practice change is pro- duced. This issue is important because interventions, particularly those targeted at enabling and reinforcing factors, are difficult and expensive to implement. Therefore, the most important practice determinants should be targeted.

Lastly, the ultimate goal of practice change is either to improve patient outcomes or reduce the cost of care without compromisng health status. Most of the intervention studies have not evaluated patient outcomes but rather have reported on changes in clinical competence, the quality of performance, or resource use.I57 The cost, and the conceptual and method- ological difficulties associated with all of these kinds of studies, no doubt explains the paucity of information.Is8 In light of the relative absence of data

28 1


about these relationships, emphasis should be placed on a careful and thor- ough evaluation of the literature that supports the efficacy of a practice change before engaging in the challenging task of designing interventions to change practice.

Conclusions ‘The approach usually used to change clinical practice is to employ interventions to change physician knowledge and skills. This is not congruent with the increasing complexity of the health care system. Clearly, we need a much broader frame of reference to tackle this important challenge.

The strategies developed to change clinical practice vary in complexity. Strategies targeted at enabling and reinforcing factors, such as organization of the practice setting or the reimbursement policy, are often more effective than those targeted at predisposing factors. However, such approaches are not only more complex but more costly. The decision to choose a specific intervention strategy in a given situation demands a trade-off between perceived effectiveness of the strategy, given the known determinants of the practice behavior to be changed, and the overall cost and difficulty of implementing the strategy.

We have offered a menu of strategies that can be used to change clinical practice. Decision makers will have to decide which ones will be appropriate in specific situations.

References I . Green L, Eriksen MP, Schor EL. Preventive practices by physicians: Behavioral determinants and

potential interventions. In Battista RN, Lawrence RS. eds. Implementing preventive services. Am J Prev Med 1988;4(4) suppl:101-107.

2. Chambers LE, Davis D, Dok C, et. al. The impact of CME. Hamilton, Ont.: McMaster University Publication, 1983.

3. Davis D, McKibbon A, Neufeld VR, Haynes B, Chambers L. The impact of CME: Update for 1985. Hamilton, Ont.: McMaster University Publications, 1986.

4. Davis D. McKibbon A, Neufeld VR, Haynes B, Chambers L. The impact of CME: Update for 1988. Hamilton, Ont.: McMaster University Publications, 1988.

5. American Board of Internal Medicine. Clinical competence in internal medicine. Ann Intern Med 1979;90:402-11.

6. Bloom B. (ed). Taxonomy of educational objectives: Cognitive domain. New York: David MacKay. 1956.

7. McAuley RG, Henderson HW. Results of the peer assessment program of the college of physicians and surgeons of Ontario. CMAJ 1984: 131 557-60.

8 . Butterworth JA, Reppart EH. Auscultatory acumen in the general medical population. JAMA

9. Clute KF. The general practitioner: A study of medical education and practice in Ontario and Nova

10. Evans CE, Haynes RB, Birkett NJ, et al. Does a mailed continuing education program improve

1960;174:32-4.

Scotia. Toronto: Toronto University Press, 1963.

282


physician performance? JAMA 1986:255(4):501-4.

matoses. Arch Dermatol 198 I ; 1 17:620-24.

training and methods of practice. New York: Health Insurance Plan of New York, 1958.

practice. J Med Educ 1956;31(Part 2):31-

1 1, Ramsay DL, Benimoff F. The ability of primary care physicians to recognize the common der-

12. Morehead MA. Quality of medical care provided by family physicians as related to their education,

13. Peterson OL, Andrew LP, Spain RS, Greenberg BG. An analytic study of North Carolina medical

14. Rhee S. Factors determining the quality of physician performance in patient care. Med Care

15. Arnold DJ. Cholecystectomies in Ohio: Results of a survey of Ohio hospitals. Am J Surg

16. Committee on Health Care Resources in the Veterans Administration Assembly of Life Sciences, National Research Council: Health Care for American Veterans. Washington, DC: National Academy of Sciences. 1977.

17. Trussel RE, Morehead MA, Erhlick J. The quantity, quality and costs of medical care secured by a sample of Teamster families in the New York area. New York: Columbia University School of Public Health and Administrative Medicine, 1962.

18. Berwick DM, Thibodeau LA. Receiver operating characteristic analysis of diagnostic skill. Med Care 1983;21(9):876-85.

19. Saywell RM, Studnicki J, Bean JA, Ludke RL. A performance comparison: USMG-FMG house staff physicians. Am J Public Health I980;70( 1):23-28.

20. Norcini JJ, Lipner RS, Benson JA. Webster GD. An analysis of the knowledge base of practicing internists as measured by the 1980 recertification examination. Ann Intern Med 1985;102:385-89.

21. Kroboth FJ, Kapoor W, Brown FH, Karpf M, Levey GS. A comparative trial of the clinical evaluation exercise. Arch Intern Med 1985; 145: 1121-23.

22. Wooliscroft JO, Stross JK, Silva J. Clinical competence certification: a critical appraisal. J Med

23. Norcini JJ, Meskauskas JA, Langdon LO, Webster GD. An evaluation of a computer simulation in the assessment of clinical competence. Evaluation and the Health Prof 1986:9(3):286-90.

24. Maatsch JL, Huang R, Downing SM, Barker D. Predictive validity of medical speciality exami- nations-A final report for Nat'l Cen. Health Serv.Res.(grant#HS 02038-04) Michigan State University: Office of Medical Education Research and Development, 1983.

25. Goran MJ, Williamson JW, Gonnella JS. The validity of patient management problems. J Med Educ

26. Rethans JJE, van Boven CPA. Simulated patients in general practice: A different look at the consultation. Br Med J 1987;294:809-12.

27. Battista RN, Williams JI , McFarlane LA. Determinants of primary care medical practice in adult cancer prevention. Med Care 1986;24(3):216-24.

28. Lomas J, Haynes RB. A taxonomy and critical review of tested strategies for the application ofclin- ical practice recommendations: From official to individual clinical policy. In Battista RN, Lawrence RS, eds. Implementing preventive services. Am J Prev Med 1988;(supp1)4(4):77-94.

1976; 141733-38.

1970;119:714-7.

Educ 1984;59:800-06.

1973;48: 171-7.

29. Hull J. Factors influencing styles of medical practice. Med Care 1979;27(7):718-26. 30. Verhaak PFM. Variations in the diagnosis of psycho-social disorders: A general practice obser-

3 I . Nightengale SD. Risk preference and admitting rates of emergency room physicians. Med Care

32. Lewis CE, Hassanein RS. Continuing medical education: An epidemiological evaluation. N Engl

33. Dohner C, Hamberg RL. Continuing medical education: The CME short course. Northwest Med

vation study. SOC Sci Med 1986;23(6):595-604.

1988;26( 1):84-7.

J Med 1970;282:254-9.

1970;69:327-330.

283


34. MacLeod GK. Continuing medical education: One approach. JAMA 1970;212: 1 171-74. 35. Powers JT, Healy A. Inservice training for physicians servicing handicapped children. Except Child

36. Ahola SJ, Creme JJ, Robinson LA Jr. Effects of physician education on the evaluation of deep vein

37. Brannon BR, Wagstaff DA. Skin cancer/melanoma public and professional education in Hawaii.

38. Jekel JF, Chauncey KJ, Moore NL, et al. The regional educational impact of a renal stone center.

39. Parboosingh IJT. A conjoint medical/nursing diabetes education program. Mobius 1983;3:7- I I . 40. Little RE, Streissguth AP, Guzinski JM. et al. Change in obstetrician advice following a two year

community education program on alcohol use and pregnancy. Am J Obst Gynecol 1983; 146:23-8. 41. Wallace RK, Ford AN, Wallace RW. Geriatric medical education: A project in faculty develop

ment. Am J Geriatric Assoc I983;3 1 : 106-8. 42. Wong ET, McCarron MM. Shaw ST Jr. Ordering of laboratory tests in a teaching hospital: can i t

be improved? JAMA 1983;249:3076-80. 43. Chambers TL. Post-graduate education for part-time doctors. Med Educ 1984; 18: 178-80. 44. Ferguson KJ, Caplan RM, Williamson PS. Factors associated with behavior change in family physi-

cians after CME presentations. J Med Educ 1984;59:662-66. 45. Greenburg LW, Jewitt LS. The impact of two educational techniques on physician knowledge, per-

formance and patient care. J Med Educ 1985;60(5):390-96. 46. Ross AK, Lawton WA. Evaluation of a course for general practitioners on muscles and joints. Br

Med J I984;288:609- 12. 47. White CW, Albanese MA, Brown DD. Caplan RM. The effectiveness of continuing medical edu-

cation in changing the behavior of physicians caring for patients with acute myocardial infarction. Ann Intern Med 1985; 102:686-92.

48. Wylie-Rosett J, Iber F, Lopez A. Professional education in a long term cure facility: Program development in diabetes. Diabetes Care 1985;8:48 1-85.

49. Curry L, Purkis IE. Validity of self-reports of behavior changes by participants after a CME course. J Med Educ 1986;61:579-84.

50. Soumerai SB, Avom J . Economic and policy analysis of university-based drug detailing. Med Care

5 1. Wirtshafter D, Sumners J, Colvin E, et al. Continuing medical education: An assessment of impact on neonatal care in community hospitals. R.I.M.E. Proceedings American Association of Medical Colleges meeting: Chicago, 1979.

52. Avom J, Souremai SB. Improving drug therapy through educational outreach. N Engl J Med

53. Shaffner W, Ray W, Federspiel C, Miller W. Improving antibiotic prescribing in office practice-

54. Lewis CE. The impact of a program for enhancing the competencies of primary care physicians i n

55. Cooper JK, Johnson TP. Medical library support in rural areas. Bull Med Libr Assoc 1983;7 I : 13-

56. Parboosingh J, Lockyer J . Teaching physicians to use a literature retrieval service for clinical in-

57. Craig WM, Uman SJ, Shaw WR, et al. Hospital use of anti-microbial drugs. Ann Intern Med

58. Ray WA, Blazer DC, Shaffner W, Federspiel C, Fink R. Reducing long term diazepam prescrib

59. Levine MD, Scott L, Curran WJ. Ethics rounds in a children’s medical center: Evaluation of a hos-

I982;48:332-36.

thrombosis in a small community hospital. Conn Med 1983;47:392-95.

Adv Cancer Control 1983:497-508.

Yale J Biol Med 1983;53:97-108.

1986;24(4):313-31.

1 Y83;308(24): 1457-63.

A controlled trial of 3 educational methods. JAMA 1983;250(3): 1728-32.

caring for patients with AIDS. J Gen Intern Med 1986;1:287-94.

15.

formation. Ann R Coll Phys Surg Can 1986; 19(4):273-77.

1978;89:793-95.

ing in office practice. JAMA 1986;256( 102536-39.

pital-based program for continuing education in medical ethics. Pediatrics 1977;60:202-8.

284


60. Stross JK, Harlan WR. The dissemination of new medical information. JAMA 1979;241:2622-4. 61. Koutsky LA, Bergner M, Kaetz, SM, Stamm WE. Evaluation of a sexually transmitted diseases

62. Legge DG, Richmond DE. Continuing education and peer review in internal medicine (CEPRIME)

63. Moore S, Marlow RA. Continuing education in nutrition for rural physicians. Fam Med

64. Ives TJ, Frey JJ, Fun SJ, Bentz EJ. Effect of an educational intervention on oral cephalosprin use in primary care. Arch Intern Med. 1987; 147:44-7.

65. Kosecoff J.. Kanouse DE, Rogers WH, et al. Effects of the National Institutes of Health consensus development program on physician practice. JAMA 1987;20:2708-13.

66. Madlon-Kay DJ. Family physician recognition and treatment of severe hypercholesterolemia. J Fam Pract 1987;24:541-6.

67. Love RR, Moorman JE. Continuing medical education with a personalized monthly newsletter. Med Pediatr Oncol 1984; 12309- 12.

68. Jennett PA, Laxdal OE. Hayton RC, et al. The effects of CME upon family physician performance in office practice: A randomized controlled study. Med Educ 1988;22: 139-45.

69. Sommers L. Evaluation of cunetedine usage in a community teaching hospital. QRB 1983;9:220- 23.

70. Martin AR, Marshall AW, Thibideau LA, Dzau V, Beaunwald E. A trial of two strategies to modify the test-ordering behavior of medical residents. N Engl J Med 1980;303: 1330-1336.

71. Sibley J , Sackett D, Neufeld V, Gerrad B, Rudnick K, Fraser W. A randomized trial of continuing medical education. N Engl J Med 1982;306(9):S I 1-5 15.

72. Jennett PA, Wilson TW, Hayton RC, Mainprize GW, Laxdal OE. Desirable behaviors in the office management of hypertension addressed through continuing medical education. Can Pub Health J

73. Stross JK, Hiss RG, Watts CM, Davis W, MacDonald R. Continuing education in pulmonary disease for primary care physicians. Am Rev Respir Dis 1983; 127:739-46.

74. Belkin M, Suchman EA, Levinson B, et al. Mental health training program for the child health conference. Am J Public Health 1965;SS: 1046-56.

75. Eward CK, Li VC, Coates TJ. Increasing physicians' anti-smoking influence by applying an in- expensive feedback technique. J Med Educ 1983;58:408-73.

76. Perera DR, Lo Gerfo JP, Schulenberger E, Ylvisaker JT. Kirz HL. Teaching sigmoidoscopy to primary care physicians: A controlled study of continuing medical education. J Fam Pract

77. Curry L, Putnam RW. Effects of CPR training: Participants' competence and patient outcome.

78. Brundis CD. Development and evaluation of short term training design for health professionals.

79. Milne D. Teaching behavior therapy to psychiatrists with a structured learning format. Med Educ

80. Gall EP, Meredith KE, Stillman PL, et al. The use of trained patient instructors for teaching and as-

8 1. Harris I. Enhancing rural health care delivery through physician continuing education: the mid-ca-

82. Kay W, Mancini ME, Rallis SF. Advanced cardiac life support refresher course using standardized

83. Cask L, McGrath G, Goldberg D, Millar T. Improving the psychiatric skills of established general

84. Curry L, Gass D. Effects of training in cardiopulmonary resuscitation on competence and patient

training program. Sex Transm Dis. 1 9 8 2 9 178-82.

Aust NZ J Med. 1986:16:679-85.

1986;18:104-5.

1989;80(5):359-62.

1983; 16:785-88.

Annual Conference Research in Medical Education 1984:243-48.

Mobius 1986;6( I ):33-48.

1986;20:148-52.

sessing rheumatologic care. Arthritis and Rheumatism 1984;27:557-63.

reer sabbatical program. Annual Conference Research in Medical Education 1983: 15 1-56.

objective based Mega Code testing. Crit Care 1987;15:55-60.

practitioners: Evaluation of group teaching. Med Educ 1987;21:362-8.

outcome. CMAJ 1987;137:491-6.

285

Changing Clinical Practice: Which tnterventions Work?

85. Rodney WM, Albers G. Flexible sigmoidoscopy: Primary care outcomes after two types of continuing medical education. Am J Gastroenterol 1986;81: 133-7.

86. Eisenberg JM. Doctor’s decisions and the cost of medical care. Ann Arbor, Michigan: Health Administration Press Perspectives, 1986.

87. Kahn L, Wirth P, Turner JK. The influence of a change in practice setting on paediatric activity: A case study. Paediatrics 1977;59:69-72.

XX. Brook RH, Williams KN. Evaluation of the New Mexico peer review system. Med Care 1976:l4(Suppl):l.

89. Payne BC, Lyons TF. Method of evaluating and improving personal medical care qua1ity:Episode of illness study and office care study. Ann Arbor:University of Michigan School of Medicine, 1972.

90. Roemer MI, Gartside F. Effect of peer review in medical foundations on qualifications of surgeons. Health Serv Rep 1973;88:808-13.

91, Rosenblatt RA, Moscovice IS. The physician as gatekeeper: Determinants of physicians’ hospi- talization rates. Med Care 1984;22(2): 150-59.

92. Stapleton IF, Zwerneman J. The influence of an intern-resident staff on the quality of private patient care. JAMA 1965; 194: 137-43.

93. Morehead MA, Donaldson R. A study of the quality of hospital care secured by a sample of Teamster families in New York City. New York: Columbia University School of Public Health and Administration, 1964.

94. Peterson OL, Barsamian EM. Medical care chart book (sixth ed.) Ann Arbor, MI: University of Michigan, 1976.

95. Sparling JF. Measuring medical care quality: A comparative study Part I : Hospitals. NY State Med

06. Yankauer A, Allaway NC. An analysis of hospital neonatal mortality rates in New York State. Am

97. Duff RS, Cook CD, Wanerka GR, Rowe DS, Dolan TF. Uses of utilization review to assess the

98. Rosenfeld LS. Quality of medical care in hospitals. Am J Public Health 1957;47:856-60. 99. Williamson PM. The adoption of new drugs by doctors practicing in group and solo practice. SOC

100. Eisele CW, Slee VN, Hoffman RG. Can the practice of internal medicine be evaluated? Ann lntern

101. Graham JB, Paloucek FP. Where should cancer of the cervix be treated? Am J Obstet Gyn

102. Thibodeau LA, Berwick DM. Variation in rates of diagnosis of otitis media. J Med Educ

103. Morehead MA, Donaldson R, Servalli MR. Comparisons between OEO neighborhood health cen- ters and other health providers. Am J Public Health 1971;61:1294-306.

104. Staff of the Stanford Center for Health Care Research. Study of the institutional differences in post-operative mortality. Springfield VA: Assembly of Life, National Academy of Sciences, National Research Council (Contract# PH 43-63-65), 1974.

105. Evans R. Strained Mercy: The economics of Canadian health care. Toronto: Butterworths. 1984. 106. Hemenway D, Fallow D. Testing for physician induced demand with hypothetical cases. Med Care

107. Renaud M, Beauchemin J , LaLonde C, Poirier H, Berthiaume S. Practice settings and prescribing profiles: The simulation of tension headaches to general practitioners working in different practice settings in the Montreal area. Am J Public Health 1980;70( 10):1068-73.

108. Vaughan WP, Waakes TP. Demonstration of the effectiveness of the professional education com- ponent of a comprehensive cancer control project using serial ‘pattern of care’ studies. Prog Canc Control 1983;4:425-32.

J 1962;36(7):62-8..

J Dis Child 1958:95:240-5.

quality of paediatric in patient care. Pediatrics 1972;49: 169-76.

Sci Med 1975;9:233-6.

Med 19S6;44: 144-7.

1963:87:405-8.

1 Y80;5S: 102 1-4.

1985;23(4):344-9.

286


109. Taylor J. Initial experience of the clinical pharmacology and clinical pharmacy interactions in a clinical pharmacokinetics consultation service. J Clin Pharmacol 1979; 19: 1-7.

110. Wasylenki DA, Harrison MK, Britnell J, Hood J. Community-based psychogeriatric service. J Am Geriatr Assoc 1984;32(3):213-18.

1 1 I . Cooper JW. Effect of initiation, termination and reinitiation of consultant phamiacist services in a geriatric long-term care facility. Med Care 1985;23( 1):84-88.

112. Hardwick DF. Structuring complexity of testing: A process-oriented approach to limiting unnecessary laboratory use. Am J Med Tech 1982;48:605-8.

1 13. Bamett GO. Winicoff RN, Morgan MM, Zielstorff RD. A computer-based monitoring system for follow-up of elevated blood pressure. Med Care 1983;2 1 :400-9.

114. Chase C, Vacek P, Shinozaki T, Giard A, Ashikaga T. Medical information management: Improving the transfer of research results to presurgical evaluation. Med Care 1983;21:410-24.

1 15. Achlager DD. A comprehensive patient care system for the family practice. IEEE Comput Applic Med 198255.9.

116. McDonald CJ. Hui SL, Smith DM, Tiemey WM, Cohen SJ, Weinberger M, McCabe P. Reminders to physicians from an introspective computer medical record, 1984.

117. Slenker SE. Cobau CD, Lind DA. Increasing physicians’ and nurses’ compliance with treatment guidelines in a cancer care program. J Med Educ 1985;60:847-54.

I 1 8. Morgan M. Computerized concurrent review of prenatal care. QRB 1978;4:33-6. 119. Karas S. The differential diagnosis simulator. J Am Coll Emerg Phys 1975;4:313-15. 120. Harvey AC. An online Bayesian program for acute abdominal pain. IEEE Comput Applic Med

121. Femow LC, Mackie C. McColl 1. Rendall M. The effect of problem-oriented medical records on

122. h tnam RW. Cuny L. Impact of physician care appraisal on physician behaviour in the office set-

123. Winicoff RN, Coltin KL, Morgan MM, Buxbaum RC, Bamett GO. Improving physician perfor-

124. Kiepart DR. Antibiotic review: The experience of one pharmacy and therapeutic committee. QRB

125. Chokshi AB, Friedman HS, Malach M, Vasavada BC, Bleicher SJ. Impact of peer review in re-

126. Kemer JF, Winn R, Robertson P, Zauber A, Bludingheiser LJ. An MSKCC community hospital

127. Maxwell JA, Cohen RM, Reinhard JD. Effect of a medical care evaluation program on physician

128. Palmer R, Louis T, Hsu L, Rothrock J, Starin R, Thompson M, Wright E. A randomized trial of

129. Payne BC, Lyons TF, Neuhaus E, Kolton M. Dwarschius L. Method of evaluating and improv-

130. Bollish SJ, Kelly WN, Miller DE, Tinimons RG. Establishing an aminoglycoside pharmacokinetic

13 1, Gelbach S, Wilkinson W, Hamrnond W, et al. Improving drug prescribing in primary care prac-

132. De Silva MI, Rissing JP. Peer review of antibiotic use: Positive impact on physicians’ prescrib

133. Berwick DM, Coltin KL. Feedback reduces test use in a health maintenance organization. JAMA

134. Ellis PM, Mellsop GW, Peace KA, Wilson JM. Peer review as an aid to improving the com-

1983:863-67.

clinical management controlled for patient risks. Med Care 1978;26(6):476-87.

ting. Can Med Assoc J 1985; 132: 10259.

mance through peer comparison feedback. Med Care 1984;22(6):527-34.

I979;5:24-27.

duction of permanent pacemaker implantation. JAMA I98 1 ;246:754-57.

colorectal carcinoma audit: A cancer control project. Adv Cancer Control 1983:79-84.

knowledge and performance. J Med Educ 1984;59:33-8.

quality assurance in 16 ambulatory care practices. Med Care 1985;23(6):7S 1-70,

ing ambulatory medical care. Health Serv Res 1984; 19(2):219-45.

monitoring service in a community hospital. Am J Hosp Pharm 1981;38:73-6.

tice. Med Care 1984;22(3): 193-201.

ing habits. QRB October 1985:302-4.

1986;225: 1450-54.

pleteness of psychiatric case notes. Med Educ 1987;2 1 :493-7.

287

135. Wells BG, Guthrie TC, Knott DH, Fink RD. Using a drug audit program to reduce polypharmacy

136. Kroencke K, Pinholt EM. Reducing polypharmacy in the elderly: A controlled trial of physician

137. Manning PR, Lee PV, Clintworth WA, et al. Changing prescribing practices through individual

138. Kunin CM, Dierks JW. A physician-pharmacist-voluntary program to improve prescribing prac-

139. Richman A, Pinsker H. Utilization review of psychiatric inpatient care. Am J Psychiatry

140. Katz E. Savings achieved through cephalosporin surveillance. Am J Hosp Pharm 1978;35: 1.521 -

141. Greenlaw CW. Aminoglycoside serum assays restricted through a pharmacy program. Am J Hosp

142. Durbin WA. Improved antibiotic usage following introduction of a novel prescription system.

143. Brian E. Foundation for medical care protocol for hospital utilization. N Engl J Med 1978; 288:

144. Moleski RJ, Andriole VT. Role of the infectious disease specialist in containing costs of antibi-

145. Johnson RE. Studying the impact of patient drug profiles in an HMO. Med Care 1976; 14:799-807. 146. Johnson RE, Azevedo DJ, Campbell WH, Christensen DB. Examining physicians’ drug order

recording behavior. Med Care 1978; 16:4088-5016. 147. Koepsell TD, Gurtel AL, Diehr PH, et al. The Seattle evaluation of computerized drug profilcs:

Effect of provider actitivites. Med Care 1983;2 1:497-507. 148. Whiting-0-Keefe Q, Simborg D, Epstein W, Warger A. A computerized medical record system

can provide more information than the standard medical record. JAMA 1985;254(9): 185-92. 149. Lindsay-McIntyre E, Wilson D, Best JA, et al. The impact of a continuing education package for

smoking cessation on physicians’ clinical behavior and patient smoking. Proc Ann Res Med Educ

I SO. Battista RN. Innovation and diffusion of health-related technologies. Int J Tech Assessment in

15 I . Contandriopoulos AP. Stimulants economiques et utilisateur des services medicaux. L’Actualite

152. Manning WG, Leibowitz A, Goldberg GA, et al. A controlled trial of the effect of a prepaid group

153. Perkoff GT. Economic versus professional incentives for cost control. N Engl J Med 1982; 307:

154. Marquis MS. Laboratory test ordering by physicians: The effect of reimbursement policies. Santa

155. Pineault R. The effect of prepaid group practice on physicians’ utilization behavior. Med Care

156. Soumerai SB, Avorn J. Principles of academic outreach (academic detailing) to improve clinical

157. Haynes RB, Davis DA, McKibbon A, Tugwell P. A critical appraisal of the efficacy of continu-

158. McAuliffe WE. Studies of process-outcome correlations in medical care evaluations: A critique.

in a state psychiatric hospital. Hosp Commun Psychiatr 1986;37: 1262-5.

feedback. J Amer Geriatric SOC 1990;38:31-6.

continuing education. JAMA 1986;256(2):230-32.

tices. N Engl J Med 1969;280:1442-46.

1973; 130:900-3.

23.

P h m 1979;36:1080-83.

JAMA I98 1 ;246: 1796- 1800.

878-82.

otics in the hospital. Reviews of infectious diseases 1986;8(3):488-93.

1987;26: 14-9.

Health Care 1989;5(2):227-48.

Economique 1980(April-June):264-96.

practice on use of services. N Engl J Med 1984;310:1505-10.

1399- 1401.

Monica, CA: Rand Corporation. 1982. Rand Corporation Pub. R-2901 -HCFA.

1976; 14: 12 1-1 36.

decision-making. JAMA 1990;263(4):549-56.

ing medical education. JAMA 1984;25 1 (l):61-4.

Med Care 1878;26( 1 I ):907-30.

288

changing clinical practice: which interventions work?

Documents