medical audit, continuing medical education and quality assurance

Refer to: Sanazaro PJ: Medical audit, continuing medical educa-tion and quality assurance. West J Med 125:241-252,Sep 1976 Special Article

Medical Audit, Continuing MedicalEducation and Quality AssurancePAUL J. SANAZARO, MD, Berkeley

Medical audit and continuing medical education (CME) are now the mainstaysof quality assurance in hospitals. Audits should address problems that haveserious consequences for patients if proper treatment is not given. The singlemost important step is the selection of, essential or scientific criteria that relateprocess to outcomes. CME does less than commonly believed to improve care.

Today, quality assurance increasingly means a near-guarantee to every pa-tient of appropriate treatment and fewest possible complications. Maintenanceof the public trust rests on a firm commitment of the medical staff and boardto this principle, implemented through an organized program of quality assur-

ance. Under these conditions, medical audit and CME can effectively improvecare by improving physician performance.

TWENTY YEARS AGO, Lembcke systematically de-scribed medical auditing by scientific methods.'Today these methods are widely used and abusedin assessing the technical quality of care. Medicalauditing is done in most hospitals because theJoint Commission on Accreditation of Hospitals(JCAH) and the Professional Standards ReviewOrganization program (PSRO) both require it.2-4Along with this, continuing medical education(CME) is universally endorsed as the principalvehicle for maintaining medical competence.5Considering the enormous amount of time andeffort devoted to these activities nationally andthe public's expectations of benefit from the re-sults, we should critically examine the effective-ness of audit and CME in improving patient care

Dr. Sanazaro is a private consultant in health services researchand development and Clinical Professor of Medicine at the Uni-versity of California, San Francisco.

Reprint requests to: Paul J. Sanazaro, MD, 1126 Grizzly PeakBlvd., Berkeley, CA 94708.

by improving physician performance. At stake isthe medical profession's continuing autonomy inassuring the quality of care.6

Medical AuditThe purpose of auditing is to assure that pa-

tients with specified conditions are receiving thefull benefit of medical care with the least possiblenumber of complications. The techniques for do-ing this are imperfect and are not standardized,despite the seemingly clear-cut methods describedin official publications.278 Being retrospective anddependent entirely on information contained inthe record, auditing can only assess limited aspectsof the technical quality of care. Of central im-portance are the procedures for selecting a topicand adopting objective criteria. The proper ap-proach to these two steps is a necessary conditionfor effective auditing directed to improving physi-cian performance.

THE WESTERN JOURNAL OF MEDICINE 241

ASSURING QUALITY IN HOSPITALS

Selection of Topic

Medical audits should only examine importantareas of care as originally suggested by William-son.9"10 The disease or condition chosen should becurable, controllable or preventable. That is, an

effective treatment or preventive measure existsfor that condition which produces predictableclinical results or outcomes. Also it should be an

established fact that patients suffer serious con-

sequences complications of the disease or ofthe treatment, or both-if that treatment is notgiven properly. Priorities for possible audit sub-jects are decided on the basis of these considera-tions, as well as knowledge or suspicion of a

particular problem. Selecting audit topics in thisway assures the validity of the results and in-creases the likelihood of medical staff commit-ment to taking any needed corrective action.

Secondary requirements in selecting topics are

precision of diagnosis and frequency. Audits can

be best applied to precisely defined primary diag-noses-for example, bacterial pneumonia ofspecified bacterial origin rather than all types andcauses of pijeumonitis, or, diabetes mellitus withketoacidosis. The presence and severity of eachdiagnosis, condition or complication should becapable of objective confirmation, preferably byquantitative data. For example, one audit com-

mittee specified a blood glucose value of 200mg per 100 ml or more plus a serum acetonevalue greater than 1:4 or blood pH less than 7.32,or both, as substantiating the diagnosis of diabetesmellitus plus ketoacidosis. Subdividing a diagnosisor condition into objectively specifiable manifesta-tions, stages of severity or complications facilitatesauditing and eliminates some of the drawbacksin using criteria.'1

The trade-off between importance and fre-quency in selecting a topic is straightforward. Themore serious the consequences for any patientreceiving inadequate or inappropriate treatment,the fewer the cases needed for a worthwhile audit.

Adoption of Criteria

More than anything else, it is the type of criteriaand the method of adopting them that determinethe effectiveness of an audit in documentingwhether medical care in the hospital meets highcontemporary standards. Some physicians stillquesdion the necessity of objective, written cri-

teria, claiming they can readily and accuratelyjudge the quality of care by reading their col-leagues' charts. To a certain extent this is true.

But the implicit criteria of quality which eachphysician carries with him may not be widelyshared. Even physicians within the same specialtycan differ surprisingly in their judgments regard-ing the quality of care as reflected in a particularrecord. Richardson found that as many as 16 to

28 physicians would have to read and judge each

record to be 95 percent certain that care for thatpatient was or was not adequate.'2 It is clearlyimpossible for physicians to devote this mucheffort to such a task, and the alternative now ingeneral use is to specify explicit, written criteria.These criteria enable nonphysician personnel toscreen large numbers of records to identify po-

tential instances of substandard care. Only thoserecords so identified are then subjected to peer

review.Whether the audit can be relied upon for evalu-

ating physician performance and identifying im-portant problems in patient care depends entirelyon the method of choosing criteria. As shown inTable 1, there are only three basic types of cri-teria: statistical, ntormative and scientific. An un-

derstanding of their source and significance forpurposes of quality assurance is necessary formaking effective use of the audit.

TABLE 1.-The Basic Types and Sources of Criteriafor Medical Audits

Type of Criteria Source

Statistical(empirical)

Normative (consensus)Optimal care

(generalconsensus)

Essential (critical)

Scientific (validated)

Regional or national statistics on

length of stay, current practices,complications, mortality

Consensus of physicians on pro-cedures that constitute good medi-cal care for a particular conditionConsensus of experts in a particu-lar disease or condition on effica-cious treatment and achievableclinical results for that conditionClinical research that objectivelyestablishes the efficacy of treat-ment and its clinical results inspecific conditions

242 SEPTEMBER 1976 * 125 * 3

ABBREVIATIONS USED IN TEXTCME=continuing medical educationJCAH=Joint Commission on Accreditation of

HospitalsPAS=Professional Activities StudyPEP=Professional Evaluation ProcedurePSRO= Professional Standards Review

OrganizationQAM=Quality Assurance MonitorRCT=randomized controlled trials


Statistical criteria (also called empirical criteria)are derived from statistics on actual practice.They define what physicians presently do in thecare of their patients. These statistics may comefrom the individual hospital's records or, morecommonly, from hospital data abstracting sys-tems, like Professional Activities Study (PAS) orCalifornia Health Data Corporation. The mostwidely used are the PAS regional statistics onlength of stay.13 According to these, the averagelength of stay is longest in the East (8.3 days)and shortest in the West (6.2 days). Because ofthis, the PSRO program permits regional variationsin setting criteria for length of stay. These differ-ences have never been adequately explained, butthey apparently do not relate to variations inquality of care.When national statistics on hospital care are

based on a large and reasonably representativesample of hospitals, they may be taken as repre-senting average physician performance. Examplesof these are the 25th to 75th percentiles for anumber of procedures as reported by the QualityAssurance Monitor (QAM) of PAS.14 According tothe QAM reports, culture and sensitivity determi-nations are done in 14 to 39 percent of adult pa-tients with pneumonia; in adult patients with acutepyelonephritis, the figures are 43 to 66 percent.If audit committees accept these statistics as ref-erence points for setting their own criteria, theyare equating the existing average level of practice

TABLE 2.-Optimal Care Criteria for Acute UrinaryTract Infection"'

PercentCriterion Observed

History:Urination frequency ......... ............. 14Obstructive symptoms ......... ............ 26Pain . ................................... 71Hematuria .............................. 29Pattern of incontinence ........ ............ 12Chronology .............................. 86Previous urologic disease ....... ........... 56

Physical Examination:Digital rectal and/or pelvic ....... .......... 37Bladder examination ......... ............. 27Kidney area examination ........ ........... 59

Laboratory:Urinalysis with stain sediment or culture ..... 74Urine culture ............................. 70Sensitivity ............................... 49Complete blood count ......... ............ 87Renal function test ........... ............. 42Intravenous pyelogram unless prostatitis ...... 36Antibacterial therapy within one hour ..... ... 68

Average Percent Observed 56

with the desired level of quality. Adopting suchstatistics amounts to endorsing the status quo. Itis doubtful that any specialty or professionalorganization would endorse the national averagesas characterizing quality of care. In short, statisti-cal criteria may be useful in initially assigninglengths of stay, but they are not suitable for audit-ing the technical quality of care.

Normative criteria (or corsensual criteria) rep-resent the judgment of physicians regarding whatought to be done in the care of patients with cer-tain diagnoses. There are two varieties of these.Optimal care criteria (or general consensus) in-corporate the consensus of judgments by physi-cians regarding the elements of good or optimalmedical care for a given condition. Essential cri-teria are developed by experts in the diagnosis andmanagement of the particular diseases or condi-tions being considered.The most widely used normative criteria are

optimal care criteria, first developed by Payne inMichigan.15 These are the ones referred to as"cookbook medicine" or "laundry lists"; theycannot be used to assess the technical quality ofcare. For example, in Table 2 are shown optimalcare criteria for acute urinary tract infection asproposed and agreed upon by a committee ofphysicians.'6 The items represent local consensuson the best care for that condition. The figure tothe right of each item in the table is the percent ofcharts in which that criterion was actually ob-served. Not a single criterion was met in 100 per-cent of charts, and the average for the study was56 percent. Similar results were found when opti-mal care criteria were applied to 20 other diag-noses; an overall average of 71 percent wasobserved. Similar discrepancies between whatphysicians propose as elements of good care andwhat those same physicians actually do in the careof patients were reported by the American Societyof Internal Medicine and by the American Acad-emy of Pediatrics.17"18

This seems to pose a serious dilemma: If com-mittees of physicians develop criteria for goodcare, and if these same physicians actually abideby fewer than 100 percent of their own criteria,then, by definition, does that not mean that theircare is substandard? The fact is, the dilemma is apurely semantic one: there is no way of knowingwhether the care of individual patients is optimalwhen the audit uses so-called optimal care criteria.

There are two specific reasons for this. The firstis that this type of criteria evolved from a study



whose primary concern was the effective use ofhospitals.'5"l'9 They were actually utilization cri-teria, intended to make certain that third partieswould not deny payment for any procedures listedfor particular diagnoses. Consequently, the origi-nal sets of criteria from Michigan contain allprocedures that might be necessary in diagnosingor treating all patients with a particular diagnosis.The criteria are useful to fiscal intermediaries indeciding whether or not to pay for a procedurebecause their only consideration is that the pro-cedure be consistent with the diagnosis. But whensuch lists as shown in Table 2 are used in a medi-cal audit, there is no way of knowing which of thelisted procedures was essential for the appropriatemanagement of a particular patient. That is whyBrook found that only 1 to 2 percent of the recordsin his study contained all the optimal care criterialisted by faculty members as assuring qualitycare.20

The second reason that these criteria are notsuitable for evaluating the care given individualpatients is the tendency to use the audit asan educational vehicle to promote better work-ups and writeups. The criteria often include anumber of symptoms and signs whose presenceor absence "should be recorded." But because notwo physicians arrive at a diagnosis in the sameway, audits which include such educationally-oriented items only show what everyone alreadyknows: There is less in the record than should bethere ideally.The fundamental shortcoming of optimal care

criteria is their lack of relationship to outcomes.20No matter how many or how few of the criteriaare observed, there is little demonstrable relation-ship to the clinical results. In fact, the correla-tions can be negative, meaning that the larger thenumber of criteria met in the care of patients, theless favorable is the result.16 When adherence toa set of criteria cannot be shown to produce goodresults in individual patients, audits based on suchcriteria do little to promote quality of patient care.

In contrast, essential criteria are indispensableto an effective audit. Essential criteria (also calledcritical criteria) apply to almost every patientwith a specified condition because they stipulateelements of care known to produce the desiredclinical results in patients with that condition. Forthis reason, essential criteria enable the medicalstaff to determine whether care of individual pa-tients, and the results of that care, conform to

contemporary high standards. The distinction be-

tween essential criteria and optimal care criteriawas clarified by the efforts of Experimental Medi-cal Care Review Organizations, originally those inAlbemarle County (Virginia), Hawaii and Utahand, more recently, in Southern California."112123The concept is promoted by the JCAH through the"critical management criteria" of its own auditformat.2 Private Initiative in PSRO, a nationalproject supported entirely by the W. K. KelloggFoundation, is currently testing the applicabilityof essential criteria to concurrent monitoring ofcare.6When used in a retrospective audit, essential

criteria specify the objective data or informationneeded to:

* substantiate the diagnosis and the presenceor absence of complications or other conditionswhich influence treatment and prognosis;

* document that each patient received treat-ment of established efficacy, given properly or tothe proper end point;

* document that each patient did not receivecontraindicated treatment;

* document that the expected clinical resultswere achieved in each patient.

Essential criteria apply to "almost every pa-tient" because they permit precise specification ofa condition and incorporate only those elementsof treatment known to be effective in producingthe desired results for that condition. Essentialcriteria are based on the best available scientificevidence of efficacy in diagnosis and treatment.They may also be derived from the application ofa basic principle. For example, the prerequisitefor rational antibiotic therapy of a severe infec-tion is identification of the causative organism,or, in urgent situations, taking the appropriatespecimens before starting or modifying antibiotictreatment.The ideal criteria for an audit are purely scien-

tific criteria derived from results of randomizedclinical trials (RCT), but these are few and farbetween.24 One example is the RCT of length ofstay for patients with uncomplicated myocardialinfarction.25 Findings in this study showed thatthe clinical and functional results in patients inhospital for only 14 days are no different fromthose of patients in hospital for 21 days. Otherexamples of scientific criteria are those based onthe results reported by the Veterans Administra-tion Study Group on Antihypertensive Agents.26Lowering the diastolic blood pressure of certainpatients to specified levels significantly reduced

244 SEPTEMBER 1976 * 125 * 3


TABLE 3.-Professional Standards Review OrganizationDefinitions of the Three Basic Types of Criteria'0

PSRO Ternm Type of Criteria

Norms ... StatisticalStandards . Normative, type unspecifiedCriteria .... Normative: general consensus or essential

Scientific

the incidence of serious or fatal complications inthose patients. The data on reduced morbidityand mortality stand as scientific evidence of effi-cacy of lowering the blood pressure.

Efficacy is what criteria for audits are all about.Scientific study establishes the degree of efficacyor effectiveness of drugs, treatments or operationsin reducing mortality, preventing complicationsor objectively improving the patient's condition.Unfortunately, all this information is not assem-blcd or published in a form that permits auditcommittees to pick out prespecified "scientificcriteria." Instead, clinical experts must be reliedon to identify the relatively few items that con-stitute essential criteria for therapy and the re-sults of therapy.'; 10,2 Experts do this fairly quicklybecause they limit the criteria in their area ofexpertise to those that can be supported by scien-tific evidence.Many commonly used surgical and medical

treatments have not been shown to be effective,and there is a growing challenge to the profes-sion to submit them to clinical trials.24'28 As thisis done, and biomedical and clinical research con-tinue to yield more proven therapies, the scien-tific bases for criteria will grow. Medical auditingwill then expedite the incorporation into hospitalpractice of the effective innovations and modifica-tions derived from sound clinical research.One other consideration now enters discussions

of criteria-namely, recent rulings on standardsof reference in malpractice suits. Historically,courts have admitted expert testimony under theprinciple that a physician's performance shouldbe compared with that of his peers in his owncommunity. But in 1968 the Brune-Belinkoff de-cision established the precedent that competencebe judged by national standards.29 The samestandards apply to physicians in San Franciscoand Baltimore alike. The basis for this ruling isthe recognition that scientific criteria of care aregenerally applicable, without geographic varia-tion. For example drug dosages do not show aregional variation, nor do the results of reducingdiastolic blood pressure.

The adoption of essential or scientific criteriamakes it mandatory that the audit committee de-fine every instance of nonadherence as an im-portant deviation. As proposed by the JCAH, any-thing other than 100 percent adherence to essen-tial treatment, in the absence of an adequatejustification, calls for peer review.2 It is inappro-priate to set arbitrary expected performance levelswhen using essential criteria. However, whencriteria are selected by general consensus, thethreshold for corrective action has to be set atsome mutually agreed upon level below 100 per-cent because there is usually no objectively estab-lished relationship between the processes incor-porated in the criteria and actual patient results.

PSRO Definitions and Guidelines for Criteria

The National Professional Standards ReviewCouncil has adopted different terms for the threetypes of criteria, and the accompanying defini-tions are potentially confusing.30 As shown inTable 3, the Council refers to statistical criteriaas "norms." The QAM Report of PAS has adoptedthis definition and refers to the 25th to 75th per-centiles of its national statistics as "norms."'14 But,as discussed above, statistical criteria are onlyaverages. Although useful to some extent in ana-lyzing length of stay, they cannot be construed asacceptable "norms" for purposes of evaluatingquality of care.

PSRO refers to normative criteria as "standards"but the manual also states that "crit_ria" are de-rived from professional expertise and professionalliterature. If the criteria are derived from expertprofessional judgment, they can be essential cri-teria, based on the best available scientific evi-dence of efficacy. However, "professional judg-ment" can also be the basis of optimal care cri-teria, which are not suitable for evaluating thetechnical quality of care. Similarly, "professionalliterature" may refer to results of soundly con-ducted clinical research (scientific criteria), or toexpert opinion (essential criteria) or to generalconsensus.

These overlapping meanings can mislead somePSRO's into adopting statistical or general con-sensus criteria for their audits. If so, these PSRO'Swould inadvertently be maintaining and reinforc-ing the status quo of care even though their origi-nal intent might have been to raise the quality ofcare.

It is therefore fortunate for the PSRO program



and for medical auditing in general that theAmerican Medical Association's Task Force onGuidelines of Care for PSRO's realized the im-portance of the fundamental distinction betweenoptimal care (general consensus) and essentialcriteria. Its first report was illustrated by severallists of the former but it offered no clear statementon their proper use. Subsequently, the TaskForce did an about-face: it endorsed critical oressential criteria and cogently defined their ra-tionale and proper use.32 Everyone concernedwith medical auditing should set aside time tostudy these two reports side by side. This com-parison will make clear the necessity to use essen-tial or critical criteria in medical care evaluationstudies directed to quality assurance.

"Process" and "Outcome"Critics of "process" criteria still argue that the

only valid basis of assessing medical care is the"outcome." Three things need to be said aboutthis. First, just about everything a physician doesin the care of his patients (process) can haveeither the desired positive effect (that is, a bene-ficial outcome) or a negative effect (that is, adetrimental outcome).33 Second, essential, scien-tific or critical management criteria are processcriteria which are predictive of outcomes that maybe immediately observable or long-term and notapparent for one or more years. Examples of theformer are the rapid clinical recovery from prop-erly managed nonmalignant intestinal obstructionor from diabetes with ketoacidosis. Examples ofthe latter are control of diastolic blood pressurein severe hypertension and internal fixation of ahip fracture. Third, and most pertinent to this dis-cussion, outcomes cannot properly be included inan audit unless they are directly attributable tomedical care (process) received in the hospital.

Viewed in this light, the tiresome argumentsover "process versus outcome" are irrelevant. Anyprocess included in an audit must be related topredictable and objectively definable outcomes,and any outcomes that are examined must bedirectly caused by specified procedures. There-fore, essential criteria for both process and out-come must be specifiable for any audit. In prac-tical terms, most outcome audits address pre-ventable or treatable complications of the diseaseor of its surgical or medical treatment. If suchare found in a higher proportion of patients thanreported in the best available clinical studies, thecauses of the unacceptably high rates can then be

discovered by applying essential (critical manage-ment) criteria to the steps taken for the preventionand management of those complications.

This approach was proposed by Williamsonand subsequently adopted by the JCAH Profes-sional Evaluation Procedure (PEP) program.2'0Even though touted as an outcome audit, PEPmust limit itself to immediate outcomes causedby efficacious medical care received in the hos-pital. If clinical experts cannot specify essentialprocess criteria (that is, no effective treatmentexists), the topic is not suitable for an audit whosepurpose is to evaluate physician performance, nomatter what the analysis of outcomes shows.

With the growing awareness of patient com-pliance as a determinant of proper managementfollowing discharge from hospital, results of pa-tient education are now being proposed as legiti-mate, immediate outcomes of hospital care. Thesewould define how well the patient should knowand understand his or her own condition and hisor her own responsibilities for continuing self-care after discharge. At present, the methods forspecifying and collecting such information in adependable, useful and acceptable manner are notyet sufficiently refined for general adoption. Theimportance of documenting this information inselected situations seems clear.34

Continuing Medical EducationIn the past 20 years, the profession's own initia-

tives and governmental and societal pressures havecombined to firmly establish CME as the thirdmajor segment of medical education for the pur-pose of improving medical competence and medi-cal care.' The State of California has joined otherstates in a growing trend to enact legislation whichrequires participation in CME as a condition forthe reregistration of the medical license.3' All ofthe 22 specialty boards of the American Board ofMedical Specialties have endorsed the principle ofperiodic recertification. Ten have set target datesand two already offer examinations. This is power-ful voluntary peer pressure to engage in self-edu-cation in order to maintain certification. Theoverall situation amounts to mandatory continu-ing education, stemming from the belief of orga-nized medicine and state legislatures that CMEassures better medical care.

This belief has an obvious justification in thefact that it is impossible to remain abreast of newknowledge and techniques without an organizedeffort in continuing self-education. And the litera-

246 SEPTEMBER 1976 * 125 * 3


ture does contain many reports of improvementin medical care attributable to CME or simplyfeedback of information indicating substandardperformance. --10.1516.l 4.36.37 Donabedian has com-piled some of the previously unpublished reportsto this effect. <8 But careful sifting of this literaturefor hard data leads to the surprising discovery thatCME has seldom been reported to change physi-cian behavior promptly or substantially.'3';9-42 Anational symposium in 1975 described the scopeand rationale of CME at length but presented fewdata on its effectiveness.43CME is now generally believed to be more effec-

tive when directed to specific problems in patientcare pinpointed by the audit. This is the well-known bi-cycle model.944'45 Both JCAH and thePSRO program endorse this approach.2"46 But hereagain, published reports raise questions concern-ing the effectiveness of the bi-cycle approach inimproving physician performance. For example,Table 4 lists the results of serial medical audits inthe hospital in which the bi-cycle approach wasfirst developed and applied.36 Note that in twoyears, CME produced approximately 50 percentimprovement in each of three deficiencies foundby audit. One has to wonder how many patientswho passed through that hospital during those twoyears continued to receive the substandard carethat had been identified by the first audit. Onealso has to ask whether the attending staff was

TABLE 4.-Improvement in Specific Deficiencies Fol-lowing Application of Bi-Cycle Model in One Hospital3

1966 1967 1968Deficiency Percent Percent Percent

Complication rate followinghysterectomy .................. 25 21 13

Pathology report inconsistent withacute appendicitis ...... ........ 45 37 19

Inappropriate use of antibiotics .... 70 52 40

TABLE 5.-Initial Physician Performance Index (1968)and Changes Following Two Years of Continuing Medi-

cal Education and Other Efforts to ImprovePerformance (1970-1971)'I

1968 1970 1971Condition Percent Percent Percent

Acute urinary tract infection ......Chronic urinary tract infection .....Chronic cholecystitis .............

Acute cholecystitis ...............Cancer of breast .................

Chronic heart disease .............

Cerebrovascular insufficiency ......

Cerebrovascular accident .........

Average

634368756761525058

675476797964595566

605366897869605665

really motivated to do something about the de-ficiencies; they may not have considered themparticularly important.

Payne has described the results of his two-yeareffort to improve physician performance by CMEand other means.'6 Shown in Table 5 are thechanges in physician performance index for eightdiagnoses in one hospital. The improvements areuneven and not at all impressive. Similar disap-pointing improvement has been described at anacademic center.47What are the possible explanations for these

results which are well below expectation? Ap-parently, some physicians look upon CME as aneducational exercise, somehow divorced fromtheir actions in treating patients. An illustrationof this occurred in a community hospital in whichmedical audit showed that the medical staff failedto properly followup almost 90 percent of majorlaboratory abnormalities.45 Having agreed thatthis was highly unsatisfactory, the staff requestedan educational conference directed to the docu-mented shortcomings. Most of the medical staffparticipated and afterwards enthusiastically ratedit as one of the best CME sessions they had everattended. But as shown in Figure 1, their level ofperformance did not improve following the "suc-cessful" CME conference.A second explanation is that a physician's

knowledge is not necessarily related to his actualperformance. In the study referred to in Table 4,most of the physicians who were prescribing anti-biotics inappropriately in a large proportion oftheir patients showed by written test before theCME effort that they already knew how to useantibiotics appropriately.36 Another report di-rectly compared the level of knowledge and itsapplication.48 In this study, 133 patients wereevaluated by a team of physicians using a proto-col. In 18 patients they diagnosed chronic urinary,tract infection on the basis of positive urine cul-tures. When these same 133 patients were ex-amined in the medical clinic by clinical facultymembers and senior medical students (withneither knowing of the prior evaluation), in onlyeight were positive cultures found. On two ob-jective written tests, the physicians and studentswho missed more than half the diagnoses showedthat they had above-average knowledge of how torecognize and treat chronic urinary infection.There was no consistent relationship between thescores on the test and actual performance inidentifying patients with urinary infections.



In the absence of alternatives, CME must re-main the foundation stone of professional com-mitment to maintaining competence. But weshould face up to some of its serious limitations.Specifically, CME cannot be relied upon to removedeficiencies in performance when the physiciansin question already possess the necessary knowl-edge and simply do not apply it. Also, it is pos-sible that those physicians most in need of CMEare not able to attend conferences designed spe-cifically to assist them in better understanding thereasons why a change in their performance isdesirable. Then there is the observation that manyphysicians who are not providing adequate carenonetheless believe they are keeping up with newdevelopments and feel no need for CME.49 Andfinally, it is a well documented fact that on theaverage, with advancing age, physicians devoteless effort to CME, demonstrate less cognitivelearning and perform at a lower level.41'4549-52Physicians under 40 years of age do more in CMEand achieve better results on tests and in practicethan those past 60, on the average. Taken to-gether, these considerations support the conclu-sion that CME may well be least effective in thosewho most need it.Many physicians now hold such a view. In an

opinion poll of its members by the CaliforniaMedical Association, 53 percent of respondentsfavored the concept of recertification, but only 6percent would base this on "credit for CME."53 In

50 -

g@ 40 -

g 30 -

E:: 20 -c

10 -

Successful CM Econference

I

Dec. 1963 June 1964 Dec. 1964

Figure 1.-Percent of minimum adequate responses toabnormal test results before and after continuing medi-cal education (CME) conference.0

contrast, 39 percent favored objective evaluationof clinical performance as a means of determin-ing continuing competence. Among the profes-sional societies, the American Society of InternalMedicine has adopted the position that assess-ment of physician competence should be based onhis day-to-day performance, not "hours of CME"or scores on a written test.54 But such assessmentis as yet neither technically adequate nor logisti-cally feasible. This brings us full circle, back tothe central question: what can be done to makethe audit and CME more effective in assuring thetechnical quality of care?

Quality Assurance

Despite the rapid growth of medical auditingand CME, criticisms of the adequacy of self-regu-lation in assuring the quality of care persist.55-60Stories in newspapers and magazines highlightpeaks and valleys in the levels of quality. Thecontinuing publicity given total health care ex-penditures and the emerging social and politicalpolicy goal of equity in health care are forcing afundamental change in perspective. No longer isit sufficient that physicians have proper creden-tials and that hospitals be accredited. Even docu-menting that "on the average" the quality of careis "good" is no longer good enough. Qualityassurance is now interpreted increasingly as anear-guarantee that the actual care given everypatient meets the prevailing standards of quality.This is the emerging reality within which we mustjudge the present adequacy of medical auditingand CME.

This position conflicts with commonly statedprinciples that have helped overcome physicianresistance to auditing. For example, hospital-based audit is still promoted in some quarters asan "educational audit" depicting "patterns ofcare" by "monitoring group performance." In theminds of most physicians, the audit is not in-tended and should not be used to examine theperformance of individual practitioners. Reassur-ing as they may be, these premises are losingtheir validity in today's climate of public account-ability. And although the profession has adoptedCME as the best method of improving performance-by responding to "educational needs" found byaudits-an impartial critic would have to con-clude that CME, in and of itself, has too little effecton practice habits to be the mainstay of profes-sional self-regulation of quality.

248 SEPTEMBER 1976 * 125 * 3

.- I I


Factors That Improve Performance

To better understand how medical auditing orCME, or both, can better contribute to qualityassurance, it may be instructive to consider fourcircumstances under which the technical qualityof care has been shown to improve impressively.These observations point to specific steps that canbe taken to raise the level of performance.

In England, surgeons at one hospital wereasked to provide certain items of diagnostic clini-cal information to a team of investigators study-ing computer-assisted diagnosis.6" Examinationof hospital records for the six months precedingthis study showed that 40 percent of patients inwhom there was a diagnosis of acute appendicitiscame to operation with perforation or abscess.During the same period, in almost 30 percent ofthe patients with nonspecific severe abdominalpain in whom laparotomy was carried out, therewere no positive findings at operation. The actualstudy lasted 19 months, during which time thesurgeons provided the requested clinical data oneach patient before deciding whether or not tooperate. As shown in Figure 2, the rates for perfo-ration or abscess and for negative findings onlaparotomies decreased dramatically. The mererequirement that they provide systematic, criticalclinical data apparently enabled the surgeons toimprove their diagnostic skills and surgical judg-ment. The result was that serious, unnecessary

4-c0C)L.0CL Ruptured appendix or abscess

Negative laparotomy fornon-specific severe abdominal pain

_7r]

morbidity was prevented in most patients withacute appendicitis, and patients were spared morethan half the unnecessary exploratory laparoto-mies for undiagnosed severe abdominal pain. Buteven more remarkably, four months after thestudy ended the rates had begun to revert towardthe initial unnecessarily and undesirably highlevels. The failure to sustain the much improvedlevel of quality of care was not explained. It wasestablished that there had been no change inpolicy, procedures, staff or equipment.

Another factor shown to improve performancewas the preparation of a protocol to guide physi-cian's assistants and nurse practitioners in diag-nosing and treating in patients presenting withacute sore throat.62 Unexpectedly, when the proto-col was put into use, the performance of physi-cians also changed greatly in the desired direc-tion, as shown in Table 6. Both the rigorous exer-cise of developing essential process criteria forthe protocol and its continuing availability in-fluenced the physicians to improve their record-ing habits and their treatment patterns substan-tially.

TABLE 6.-Physician Performance Before and AfterIntroduction of Protocol for Evaluation and Management

of Patients with Acute Sore Throat"Before AfterProtocol Protocol

Items of Performance Percent Percent

Essential history recordedrheumatic fever ................. 1 97drug allergy .................... 7 94dysphagia ...................... 8 100

Throat culture performed ...... ..... 79 90Complete blood count ordered ....... 27 14Antibiotics given .................. 57 18

TABLE 7.-Effect of a Computer Alert on PhysicianPerformance'

Proportion of IndicatedChanges Made by

Physicians PhysiciansAlerted by Using Regular

Indicated Change and Reason Printout Records

Reduce aspirin because possiblecause of bleeding ....... ....... 2/5

Reduce triamterene, KCl becauseserum K>5 mEq/liter ..... .... 2/3

Adjust cardiac glycosides andK-wasting diuretics becauseserum K<3.5 mEq/liter ........ 4/8

Increase antihypertensivesbecause of undertreatment ...... 8/14

Reduce thiazides becauseuric acid>9 mg/dl ...... ....... 2/7

Percent of indicated changesmade by physicians ........ 49

1/5

0/6

0/6

0/7

4


Start of 19 months End of 4 months Afterstudy study Study

Figure 2.-Changes in frequency of ruptured appendixor abscess and of negative laparotomies during andafter study of computer-assisted diagnosis.6"


Similar observations were reported when pro-tocols were prepared for determining whethertreatment prescribed in a clinic was insufficient,excessive or dangerous.' One group of cliniciansreceived computer printouts that alerted them tothe possible existence of these situations and sug-gested the appropriate responses, as defined bythe protocols. The changes in treatment made bythese clinicians were compared with those of acontrol group who received no such warningswhile caring for similar patients. As seen in Table7, a much higher proportion of indicated changeswas made by those who received computer alerts.The explanation offered is that these warningsfocused the clinicians' attention on a particularproblem, forcing an explicit decision to continuethe treatment or modify it, if in his judgment thesituation so warranted. In the absence of such awarning system, many of those using the usualrecord apparently overlooked the potential dangersignals. Assuming that the protocols for the sug-gested changes were valid, their use substantiallyimproved the care of patients, increasing thelikelihood of therapeutic benefit and reducing therisks of serious complications.A third documentation of sharp change in staff

performance was provided by Lembcke in hisclassic report.' Conducting a medical audit atregular intervals in a hospital, he analyzed recordsto ascertain the number of unnecessary or un-

180 - Interviews withsurgeons Criteria distributed

150 -

C

1200

c 90 -

0

z60 -

30 -t

Auditbegun

13-week intervals

Figure 3.-Effect of interviews and of distributing criteriaon the number of cases of unnecessary operations.'

justified pelvic operations. The results were re-ported in summary form to a joint liaison com-mittee representing the medical staff, board, andadministration, but the identities of various sur-geons were not given. The "president of the medi-cal staff, assisted by the physician members of thejoint liaison committee, interviewed the surgeonswith poor individual records and sought theircompliance." As shown in Figure 3, there wasprompt and striking reduction in the number ofunnecessary operations. The criteria were thendistributed to the staff near the end of the fourthaudit period, and further improvement occurred.This illustrates the impact of a unified, genuinecommitment of the staff, administration andboard to improve the quality of care, once theprecise deficiency and those responsible for ithave been objectively and systematically identifiedby audits based on essential criteria. This conm-mitment is clearly the sufficient condition fortransforming audits, with or without CME, intopowerful and effective instruments of quality as-surance.A fourth factor known to be associated with a

higher quality of care as reflected in physicianperformance is a teaching environment.64-8 Theprecise mechanisms which lead to better resultshave not been documented, but it seems that theteaching environment sets and maintains high ex-pectations for the level of actual performance.For example, Stapleton observed that physiciansordered critical tests more frequently on the teach-ing than on the nonteaching wards of a hospital.6'In the New York studies, there was no real dif-ference in quality of care attributable to certifica-tion or noncertification of the physicians, butcare was superior in hospitals affiliated with amedical school.65'66One obvious common denominator in the

studies described above is a professionally accept-able form of surveillance or monitoring. Each ofthe techniques for monitoring achieved that whichis impossible for CME: each influenced the physi-cian at the time and place he was actually provid-ing care to an individual patient. Acting directlyor through enhanced awareness and motivation,each of these assisted the physician in applyingmore fully the information and skills that healready possessed.

The Public Trust and Quality Assurance

It is natural for physicians to look at auditingprimarily from the standpoint of its possible im-

250 SEPTEMBER 1976 * 125 * 3


plications for the physicians' status in that hos-pital, rather than its implications for patients'welfare. Yet, a hospital's obligation is to its pa-tients. Its corporate legal responsibility callsfor much more than passive concern over thosepatients about to be admitted to the hospital whowill suffer unnecessary complications or depriva-tion of full benefit unless measures are taken toeliminate the deficiencies identified by audit.29'58

Corporate responsibility requires active andprompt correction of those deficiencies, followedby a suitable form and frequency of monitoringto assure that all patients are actually receivingthe proper treatment and the expected results.2This is the quality assurance cycle. The extent towhich audit and CME will be judged to serve thepublic trust adequately will be in direct proportionto their proven effectiveness in making that cyclea reality.

An Organized Program of Quality AssuranceThe present review of the medical audit indi-

cates that it is not an all-purpose technique forassessing all important aspects of the technicalquality of care. The present state of medicalscience is such that rigorous auditing can only beapplied to a narrow segment of patient care.Therefore the full burden of professional moni-toring should not rest on the medical audit com-mittee(s) alone. Quality assurance requires anorganized program encompassing the activities ofall hospital and staff committees that bear onquality. These include departmental review com-mittees, plus transfusion, infection, tissue, deathand complications, emergency room, pharmacy ordrug, and record committees, in addition to theaudit and utilization review committees.

There are advantages in coordinating all suchquality-related committee work. The individualcommittee reports, combined with the cumulativeresults of audits, add up to a comprehensive docu-mentation of all aspects of the quality of care.That documentation serves as a solid base of duecare by the medical staff and board in regularlyand objectively assessing how well each physicianis discharging his obligations in exercising theclinical privileges granted him by the hospital. Inthis problematic and too often imperfectly andinadequately executed judgment, medical auditscan play a vital role, if designed to yield validcomparative information on staff performance inimportant areas of care. Corrective action, whenindicated, can be initiated promptly with fuller

understanding of the scope and sources of defi-ciencies that are to be remedied. In less urgentsituations, CME, if properly prescribed, may suf-fice. Depending on local resources and interests,other approaches as described above can be initi-ated to raise to standard levels some specificaspect of the quality of care.The continuing upsurge in public demand for

accountability in quality is probably irreversible.In responding, the medical staff and hospital ad-ministration have one unassailable professionalposition: commitment to assuring every patientof full achievable benefit at least possible risk.This commitment is the cardinal prerequisite forquality assurance in its fullest contemporarymeaning. In such settings properly designedmedical audits and CME will be effective in im-proving physician performance.

REFERENCES1. Lembcke PA: Medical auditing by scientific methods, illus-

trated by major female pelvic surgery. JAMA 162:646-655, 19562. The PEP Primer. Professional Evaluation Procedure for

Auditing and Improving Patient Care. Chicago, Illinois, JointCommission on Accreditation of Hospitals, 1974

3. Goran MJ, Roberts JS, Kellogg M, et al: The PSRO hospitalreview system. Med Care 13:No. 4 (supplement), 1975

4. Goran MJ: The future of quality assurance in health care:Next steps from the perspective of the federal government. BullNY Acad Med 52:177-184, 1976

5. Ruhe CHW: Governmental and societal pressures for pro-grams of continuing medical education. Bull NY Acad Med 51:707-718, 1975

6. Sanazaro PJ: Private Initiative in PSRO. N Engl J Med 293:1023-1028, 1975

7. QRB. Quality Review Bulletin 1 :Nos. 1-12, 1974-5 and 2:Nos. 1-2, 1976

8. Educational Patient Care Workshop Program. CaliforniaMedical Association/California Hospital Association, 1975

9. Williamson JW, Alexander M, Miller GE: Priorities in pa-tient care research and continuing medical education. JAMA 204:303-308, 1968

10. Williamson JW: Evaluating quality of patient care: A strat-egy relating outcome and process assessment. JAMA 218:564-569,1971

11. Greenfield S, Lewis CE, Kaplan SH, et al: Peer review bycriteria mapping: Criteria for diabetes mellitus. The use of de-cision-making in chart audit. Ann Int Med 83:761-770, 1975

12. Richardson FM: Peer review of medical care. Med Care 10:29-39, 1972

13. Length of Stay in PAS Hospitals. United States Regional,1972, Ann Arbor, Michigan, Commission on Professional andHospital Activities, 1973, pp 4-5

14. Quality Assurance Monitor: Report Book. Commission onProfessional and Hospital Activities, Ann Arbor, Michigan, Jun1974

15. Payne BC: Continued evolution of a system of medicalcare appraisal. JAMA 201: 536-540, 1967

16. Payne BC, Lyons TF: Methods of Evaluating and Improv-ing Personal Medical Care Quality: Episode of Illness Study. AnnArbor, Michigan, University of Michigan, 1972

17. Hare RL, Barnoon S: Medical care appraisal and qualityassurance in the office practice of internal medicine. San Fran-cisco, American Society of Internal Medicine, Jul 1973, pp 138-141

18. Thompson HC, Osborne CE: Quality assurance of ambu-latory child health care: opinions of practicing physicians aboutproposed criteria. Med Care 14:22-31, 1976

19. Payne BC (ed): Hospital Utilization Review Manual. AnnArbor, Michigan, University of Michigan Department of Post-graduate Medicine, 1968

20. Brook RH, Appel FA: Quality of care assessment. Choosinga method for peer review. N Engl J Med 288:1323-1329, 1973

21. Sanazaro PJ, Goldstein RL, Roberts JS, et al: Research anddevelopment in quality assurance: The experimental medical carereview organization program. N Engl J Med 287:1125-1131, 1972

22. Decker B, Bonner P: Criteria in Peer Review. Cambridge,Mass, Arthur D. Little, Inc., 1973



23. Sanazaro PJ: Medical audit: Experience in the U.S.A. BritMed J 1:271-274, 1974

24. Cochrane AL: Effectiveness and Efficiency: Random Reflec-tions on Health Services. London, The Nuffield Provincial Hos-pitals Trust, 1972

25. Hutter AM, Sidel VW, Shine KI, et al: Early hospital dis-charge after myocardial infarction. N Engl J Med 288:1141-1144,1973

26. Veterans Administration Study Group on AntihypertensiveAgents: Effects of treatment on morbidity in hypertension: Re-sults in patients with diastolic blood pressures averaging 115through 129 mm Hg. JAMA 202:1028-1034, 1967

27. Wagner EH, Greenberg RA, Imrey PB, et al: Influence oftraining and experience on selecting criteria to evaluate medicalcare. N Engl J Med 294:871-876, 1976

28. Advancing the Quality of Health Care: Key Issues andFundamental Principles. A Policy Statement by a Committee ofthe Institute of Medicine. Washington, D.C. National Academy ofSciences, 1974

29. Chayet NL: Corporate responsibility for medical care. Hos-pital Progress 53:32-37, 1972

30. PSRO Program Manual, chap 7. Rockville, Md, Dept ofHealth, Education, and Welfare, 1974

31. PSROs and norms of care-A report by the Task Force onGuidelines of Care: American Medical Association Advisory Com-mittee on PSRO. JAMA 229:166-171, 1974

32. Welch CE: PSRO: Guidelines for criteria of care. JAMA232:47-50, 1975

33. Sanazaro PJ, Williamson JW: Physician perfcrmance andits effects on patients: A classification based on reports by in-ternists, surgeons, pediatricians, and obstetricians. Med Care 8:299-308, 1970

34. Williamson JW, Aronovitch S, Simonson L, et al: Healthaccounting: An outcome-based system of quality assurance:Illustrative application to hypertension. Bull NY Acad Med 51:727-738, 1975

35. Section 2101.5, Business and Professions Code, added bySEC 11, ABlxx. Approved by Governor Sep 23, 1975

36. Brown CR Jr, Uhl HSM: Mandatory continuing education:Sense or nonsense? JAMA 213:1660-1668, 1970

37. Devitt JE: Does continuing medical education by peer re-view really work? Canad Med Ass J 108:1279-1281, 1973

38. Donabedian A: Medical Care Appraisal-Quality and Utili-zation. A Guide to Medical Care Administration-Vol II. NewYork, American Public Health Association, 1969, pp 122-151

39. Lewis CE, Hassanein RS: Continuing medical education-epidemiologic evaluation. N Engl J Med 282:254-259, 1970

40. Roney JG, Roark GM: Continuing Education of Physiciansin Kansas: An Exploratory End-Result Study. Menlo Park, Calif,Stanford Research Institute, Oct 1967

41. Peterson OL, Andrews LP, Spain RS, et al: An analyticalstudy of North Carolina general practice 1953-1954. J Med Educ31:1-165, part 2, 1956

42. Miller GE: Why continuing medical education? Bull NYAcad Med 51:701-706, 1975

43. Symposium on continuing medical education. Bull NY AcadMed 51:701-788, 1975

44. Brown CR Jr, Fleisher DS: The bi-cycle concept-relatingcontinuing education directly to patient care, In Stearns NS,Getchell ME, Gold RA (Eds): Continuing Education in Com-munity Hospitals. Boston, Postgraduate Medical Institute, 1971,pp 88-97

45. Williamson JW, Alexander M, Miller GE: Continuing edu-cation and patient care research: Physician response to screeningtest results. JAMA 201:938-942, 1967

46. Jessee WF, Munier WB, Fielding JF, et al: PSRO: An edu-cational force for improving quality of care. N Engl J Med 292:668-671, 1975

47. Cayton CG, Tanner LA, Riedel DC, et al: Surgical auditusing predetermined weighted criteria. Conn Med 38:117-122, 1974

48. Gonnella JS, Goran MJ, Williamson JW, et al: Evaluationof patient care: An approach. JAMA 214:2040-2043, 1970

49. Clute KF: The General Practitioner: A Study of MedicalEducation and Practice in Ontario and Nova Scotia. Toronto,University of Toronto Press, 1963

50. Meskauskas JA, Webster GD: The American Board of In-ternal Medicine Recertificaticn Examination: Process and Results.Ann Int Med 82:577-581, 1975

51. Youmans JB: Experience with a postgraduate course forpractitioners: Evaluation of results. J Ass Amer Med Coll 10:154-173, 1935

52. Kotre JN, Mann FC, Morris WC, et al: The Michigan phy-sician's use and evaluation of his medical journal. Mich Med 70:11-16, 1971

53. Socioeconomic Report. Bureau of Research and Planning.California Medical Association 16:4, (Mar) 1976

54. American Scciety of Internal Medicine Policy Manual. SanFrancisco, California, 1975, p 52

55. Sherman SR: A realistic look at patient care audit andcontinuing educaticn. Hosp Med Staff 4:14-20, 1975

56. McCleery R, Keelty LT, Lam M, et al: One Life-One Phy-sician: An inquiry into the medical profession's performance inself regulation. Washington, D.C., Public Affairs Press, 1971

57. Worthington W, Silver LH: Regulation of quality of carein hospitals: The need for change. Law and Contemp Prob 35:305-333, 1970

58. Williams KJ: Beyond responsibility: Toward accountability.Hospital Progress 53:44-50, 1972

59. Derbyshire RC: Medical Licensure and Discipline in theUnited States. Baltimore, Md, Johns Hopkins Press, 1969

60. Freidson E: Profession of Medicine. A Study of the Soci-ology of Applied Knowledge. New York, Dodd, Mead & Com-pany, 1970

61. deDombal FT, Leaper DJ, Horrochs JC: Human and com-puter-aided diagnosis of abdominal pain: Further report withemphasis on performance of clinicians. Brit Med J 1:376-380, 1974

62. Grimm RH Jr, Shimoni K, Harlan WR Jr: Evaluation ofpatient-care protocol use by various providers. N Engl J Med292:507-511, 1975

63. McDonald CJ: Use of a computer to detect and respond toclinical events: Its effect on clinical behavior. Ann Int Med 84:162-167, 1976

64. Stapleton JF, Zwerneman JA: The influence of an intern-resident staff on the quality of private patient care. JAMA 194:877-882, 1965

65. Ehrlich J, Morehead M, Trussell RE: The Quantity, Qualityand Costs of Medical and Hospital Care Secured by a Sample ofTeamster Families in the New York Area. New York, ColumbiaUniversity School of Public HIealth and Administrative Medicine,1962

66. Morehead MA, Donaldson RS, Sanderson S, et al: A Studyof the Quality of Hospital Care Secured by a Sample of TeamsterFanilies in New York City. New York, Columbia UniversitySchool of Public Health and Administrative Medicine, 1964

67. Lipworth L, Lee JAH, Morris JN: Case-fatality in teachingand nonteaching hospitals 1956-59. Med Care 1:71-76, 1963

68. Goss MEW: Organizational goals and quality of care: Evi-dence from comparative research on hospitals. J Health and SocBehav 11:255-268, 1970

252 SEPTEMBER 1976 * 125 * 3

medical audit, continuing medical education and quality assurance

Documents