Provider Perceptions in Using Outcomes Data toImprove Clinical PracticeA. Laurie W. Shroyer, PhD, Ira Dauber, MD, Robert H. Jones, MD, Jennifer Daley, MD,Frederick L. Grover, MD, and Karl E. Hammermeister, MDDepartments of Medicine and Surgery, University of Colorado Health Sciences Center, Denver, Colorado; Department of Surgery,Duke University Medical Center, Durham, North Carolina; and Health Services Research and Development, Brockton/WestRoxbury Veterans Affairs Medical Center, West Roxbury, Massachusetts

In this article we report on the perceptions of health careproviders concerning the use of health care outcomes.These perceptions were identified during two afternoonworkshops at the symposium on outcomes data held inKeystone, Colorado, in June 1994. The overall goal ofthese workshops was to explore the potential use ofoutcomes data within the clinical practice setting. Theattitudes and concerns of several key providers formedthe framework for the workshop discussions. Thestrengths and weaknesses associated with using risk­adjusted outcomes data, both to assess and to improve

W e report here on the perceptions of health careproviders identified during two afternoon work­

shops at the National Symposium on Using Outcomes toImprove Clinical Practice: Building on Models from Car­diac Surgery, held in Keystone, Colorado, on June 6 and7, 1994. The overall goal of these workshops was toexplore the strengths and weaknesses of risk-adjustedoutcomes data within the clinical practice setting toimprove the quality of patient care rendered by thepersons using these data.

Material and Methods

Each of the two workshops had unique goals and objec­tives. As many meeting participants had never beenexposed to the study of patient-specific risk factors* andoutcomes-t- measures, the first workshop was designed toprovide a hands-on learning experience using case studymaterials from the Department of Veterans Affairs Con-

Presented at the National Symposium on Using Outcomes Data toImprove Clinical Practice: Building on Models From Cardiac Surgery,Keystone, CO, June 6-7, 1994.

Address reprint requests to Dr Shroyer, Division of Cardiac Research,Denver Veterans Affairs Medical Center, 1055 Clermont St (151), Denver,CO 80014.

* Patient risk factors are the patient baseline characteristics that mayplace the patient either at greater risk of an adverse outcome or at agreater advantage for a positive outcome.

t Outcome may be defined as the "results of care ... [and] can encompassbiological changes in disease, comfort, ability for self-care, physicalfunction and mobility, emotion and intellectual performance, patientsatisfaction and self-perception of health, health knowledge and compli­ance with medical care, and viability of family, job, and social rolefunctioning" [1].

© 1994 by The Society of Thoracic Surgeons

quality of patient care, were debated within the work­shop groups. Overall, conference participants agreed thatthe risk-adjusted outcomes of cardiac care could be usedboth to assess and to improve the quality of patient care.The research to date, however, represents only a first steptoward this goal. The meeting participants challengedthe research community to improve on the measurescurrently available, with the goal of providing cliniciansbetter information to improve the "art" of medicine.

(Ann Thome Surg 1994;58:1877-80)

tinuous Improvement in Cardiac Surgery Study. The goalwas to focus on the details of collecting and analyzingboth patient-specific risk and outcomes data, as well as todiscuss the interpretation of multivariate predictionmodel results, both for estimating the surgical risk forindividual patients and for measuring provider perfor­mance. Specific objectives of this first workshop in­cluded:

1. Providing a general orientation to risk and outcomesdata issues (eg, data quality).

2. Discussing the analytic techniques used (eg, logisticregression) to develop a statistical risk model.

3. Discussing the potential interpretation of predictedoutcomes as determined on the basis of patient riskcompared with observed outcomes for the purpose ofclinical decision-making in terms of both individualpatients and groups of patients and for the purpose ofcomparing performance across providers.

4. Exploring the potential application of risk model in­formation in a clinical practice setting to improvequality of patient care.

The second workshop was designed as an interactivesmall group experience, using a debating techniquecalled "constructive controversy" [2]. The topic of thedebate was the relative strengths and weaknesses ofusing risk-adjusted outcomes measures to assess andimprove the quality!' of patient care [3]. The secondworkshop was intended as a setting for framing a "con-

t Quality of care may be defined as "the degree to which health servicesfor individuals and populations increase the likelihood of desired healthcare outcomes" [3].

0003-4975/94/$7.00

1878 OUTCOMES DATA SHROYER ET ALPROVIDER PERCEPTIONS

ference consensus" as the basis for directing futureresearch and development efforts in this field. Thus, it isour hope that the provider perceptions elicited by theworkshop discussions and summarized here are repre­sentative of provider perceptions regarding the use ofoutcomes data in clinical practice, based on the identifi­cation of common patterns and themes.

Workshop Results

General AttitudesOverall, the care providers exhibited a positive attitudetoward the use of risk-adjusted outcomes measures as ameans to improve the quality of patient care in thefuture. Given the current national health care reformimpetus to uniformly define and measure the quality ofpatient care, the health care providers are experiencingconsiderable external pressure to use these data to effectimprovements in the quality of care. All meeting partic­ipants were agreed on the importance of this evolvingscience. However, concerns were expressed that the datamay be used in a negative, rather than positive, mannerby groups other than clinicians for purposes other than toimprove the quality of clinical practice (eg, governmentpayers using these data to ration or further regulatemedical practice). A certain amount of healthy skepticismexisted in the sense that risk-adjusted outcomes areviewed as an analytic tool still requiring refinement.Current limitations were acknowledged in terms of thequality of the data, the multiple complex statistical meth­ods used to analyze it, and the unproved validity of thesemethods as measures of quality of patient care.

In spite of the technical complexity involved in defin­ing and measuring the quality of care, the meetingparticipants agreed that the care provider team members(in collaboration with health services researchers, biosta­tisticians, administrators, and health policy makers) needto be integrally involved in the design, implementation, andevaluation of the data collection and analysis systemsused to calculate risk-adjusted outcomes of care. A needfor clinician involvement and action was viewed as a toppriority by most of the meeting participants. Clinicians,in particular, felt strongly that the greatest potentialbenefit of outcomes data is to improve clinical practice.Thus, care provider team members should be developinginnovative mechanisms for translating this informationinto changes in the processes of care rendered, such asclinical diagnosis, prognosis, and therapy.

Provider ConcernsThe predominant concern expressed by participants atboth workshops was the potential misinterpretation ofthe risk-adjusted outcomes data by external audiences(not directly involved in providing patient care), includ­ing the general public, administrators, health policy mak­ers, and the news media. Meeting participants agreedthat the best strategies for protecting against potentialmisinterpretation include a combination of educationand open communication. The general consensus wasthat more exposure of these audiences to the concepts

Ann Thorac Surg1994;58:1877-80

and interpretation of risk-adjusted outcomes data wouldlead to a more accurate understanding of the appropriateuses, strengths, and limitations of this information. Fun­damentally, the active use of outcomes data by cliniciansin the clinical practice setting was viewed as the first steptoward addressing this concern. The advantage of havingbetter information for making clinical decisions wasthought to far outweigh the potential disadvantagesstemming from the inaccurate interpretation of the data.In an era of reform, both public accountability (eg,providing outcomes information to consumers and pay­ers) and open communication (eg, education of the newsmedia and other external audiences in the correct inter­pretation of this information) were viewed as mecha­nisms to facilitate positive change.

An insightful concern, raised at several junctures, washow much information patients want and need in orderfor them to make informed choices regarding their med­ical care options. There was a spirited discussion aboutpatients' abilities to understand and interpret risk­adjusted outcomes, either as a decision-making tool tocompare the relative risks and benefits of a given thera­peutic option or as a potential quality-of-care measure toassess provider performance. As this information has notyet been widely used by patients, it was not possible todirectly address this concern. The implications for pa­tient choice were viewed as potentially relevant withinthe context of health care reform.

Another concern expressed by the meeting partici­pants was the lack of uniformity in terminology anddefinitions. Specifically, there is a need to sharpen theterminology used (avoiding generic terms and concepts)and to standardize the definitions of patient risk factors,procedural details, and outcomes measures, in an at­tempt to promote the clear interpretation and compari­son of risk-adjusted outcomes across multiple care set­tings.

A last concern raised was that these new quality toolsare not without costs, costs that accrue to the providers,patients, and payers. A key question is the relativebenefit conferred by these new systems for measuringand monitoring risk-adjusted outcomes in relation to thecosts of collecting these data and administering thesesystems.

Using Outcomes Data in Clinical PracticeThe outcomes findings reported during the conferenceprimarily consisted of risk-adjusted mortality and risk­adjusted major complication rates. Admittedly, thesewell-defined outcomes represent the major thrust ofresearch efforts to-date. In general, the conference par­ticipants felt strongly that this focus on traditional mea­sures of mortality and morbidity was shortsighted. Pa­tient death is a relatively rare event for most types ofhealth care services (eg, outpatient care). Although deathis the most extreme outcome, in some situations (eg,hospice care) it may be an anticipated outcome of thepatient's disease. Additionally, there is no opportunityfor corrective action with regard to the specific individualpatient death. Thus, an expanded outcomes focus on

Ann Thorac Surg1994;58:1877-80

innovative outcomes measures, including other medicaloutcomes measures (eg, change in cardiac functionalstatus), service outcomes (eg, patient satisfaction), andcost, may result in the development of measures morerelevant to both clinicians and patients.

The meeting participants also voiced the opinion that,although great strides had been made in the use ofrisk-adjusted outcomes in the area of cardiac surgery,these models needed to be extended to all areas ofcardiovascular care. Without comparable models for per­cutaneous transluminal coronary angioplasty and themedical therapy of coronary artery disease, clinicianscannot assess the relative risks of different therapeuticalternatives for a given patient or subgroup of patients.Additionally, the potential benefits (eg, event-free sur­vival rates) need to be compared to the relative risks of anadverse outcome when making informed choices regard­ing care. It was agreed that this risk-benefit ratio was the"missing link" in the effective use of outcomes data inclinical practice settings.

The validity of risk-adjusted outcomes data as poten­tial measures of quality of care was questioned as well.Risk-adjusted outcomes are only indirect measures of thequality of care, in that they serve as "cues that promptand motivate the assessment of processes and structuresfor causes that can be remedied" [4]. Although thismeasure has both attributional and face validity, thehypothesis that risk-adjusted outcomes are valid indica­tors of quality of care has not yet been rigorously tested.

Assessing Quality of Patient CareThe consensus was that, as a means to assess the qualityof patient care, the existing risk-adjusted models incardiac surgery can be used only as screening tools todetect either quality problems or excellent performance.Risk-adjusted outcomes of care should be used to stim­ulate further discussion and performance review­complementing but not replacing traditional quality re­view mechanisms.

In comparing provider performance, raw mortalityrates do not account for differences in the severity ofillness and the comorbidity of the patient populationserved [5]. When raw mortality data are reported as ameasure for hospital quality of care, these differences inpatient mix across hospitals have been demonstrated tointroduce significant bias [6]. Without adjustment, vari­ations in referral and admission patterns may be con­strued as differences in the quality of care provided [7].Therefore, the need to control for these other factors hasled to the development of techniques to adjust mortalitymonitors to reflect differing patient risk and other con­founding factors. If the adjustment for baseline patientrisk factors related to disease severity and other con­founding factors is adequate, then differences in risk­adjusted mortality rates may represent actual differencesin the quality of care among providers [8].

In discussing the current approaches used to assess thequality of patient care, a concern was raised that differentdata systems may provide divergent or conflicting re­sults. Different cardiac surgery data systems collect

OUTCOMES DATA SHROYER ET AL 1879PROVIDER PERCEPTIONS

unique risk factor sets and different analytic methods areused to calculate risk model estimates. The goal of estab­lishing a single, uniform national standard of key patientrisk factors and an optimal analytic technique was iden­tified as a top research priority. A national voluntarycommittee chaired by Dr Robert H. Jones, which willexamine the risk factors that have an impact on theshort-term mortality associated with coronary artery by­pass grafting, with participants from all major cardiacsurgery data sets, has recently embarked on this project.

Although the meeting participants agreed that risk­adjusted outcomes provide a relatively objective basis forcomparison across patients and providers, concerns wereraised regarding the "gaming" potential of the lessobjective measures of risk (eg, severity of angina orpriority of operation). Risk-adjusted outcomes were cer­tainly viewed as a considerable improvement over rawoperative mortality rates as a means to assess quality ofpatient care. However, there was a general lack of trust inthe accuracy and reliability of clinician-reported subjec­tive measures of risk used in these statistical estimates.

Improving Quality of Patient CareRisk-adjusted outcomes were identified as the first nec­essary step toward further examining the processes* andstructures+ of the care provided. Several examples werepresented regarding the use of these measures to initiatean internal review process by care provider team mem­bers. Overall, meeting participants agreed that clinician­initiated care improvements were the most importantfuture application of these data.

Another concern repeatedly raised was the potentialadverse impact of the overzealous monitoring of patientoutcomes on the treatment rendered to the high-riskpatient. Estimates of outcomes from risk models, whichdo not include "rare risk factors" or "rare events," maydiscourage care providers from treating these specialpatient groups. For example, active endocarditis is notalways contained in the valve surgery risk models devel­oped to predict the risk of operative mortality becausethe frequency of this event within the valve surgerypatient population is very low. However, every cardiacsurgeon is poignantly aware of the potential for anadverse outcome in this special subgroup of patients.Thus, the quality of overall patient care may be nega­tively affected if clinicians perceive a professional advan­tage to selectively treating only those patients with riskfactors adequately represented within the statistical riskmodels used for outcomes reporting and peer compari­son.

The meeting participants thought that, ideally, risk­adjusted outcomes for cardiac surgery should be linkedto processes and structures of care. Those processes and

• Processes of care may be defined as the"content of care, i.e., how thepatient was moved into, through, and out of the health care system andthe services that were provided during the care episode" [1].

• Structures of care may be defined as "the facilities, equipment, services,and manpower available for care and the credentials and the qualifica­tions of the health professionals involved" [1].

1880 OUTCOMES DATA SHROYER ET ALPROVIDER PERCEPTIONS

structures of care that are found to be associated withfavorable outcomes could then be used to develop uni­form practice and appropriateness guidelines. The De­partment of Veterans Affairs has implemented a study todemonstrate these relationships with regard to the fieldof cardiac surgery.

Finally, it was generally agreed that the effort to obtainrisk-adjusted outcomes as a quality assessment tool en­tails adding considerable cost to an already expensivehealth care system. However, the group viewed this newmarginal expense as relatively small versus the patientcare costs incurred. Thus, it was generally agreed thatrisk-adjusted outcome models are an appropriate invest­ment toward the goal of making patient care morecost-effective.

Summary

There was a high level of participation at the conferenceworkshops and they generated considerable enthusiasmamong the participants. Overall, the participants agreedthat risk-adjusted outcomes of cardiac care can be usedto both assess and improve the quality of patient care.The research to date, however, represents only a first steptoward this goal. The participants identified several crit­icallimitations to the use of risk-adjusted outcomes andchallenged the research community to improve the mea­sures currently available for acquiring and assessingthese data, with the ultimate goal of providing clinicianswith better information for effecting improvements in the"art" of medicine.

Ann Thorae Surg1994;58:1877-80

Doctor Daley is a Senior Research Associate in the CareerDevelopment Award Program of the Department of VeteransAffairs Health Services Research and Career Development Pro­gram.

Supported by the Department of Veterans Affairs Offices ofQuality Management and Clinical Affairs and the VeteransAffairs Health Services Research and Development Service.

We thank the following workshop leaders for their contribu­tions: B. Ned Calonge, MD, Stanley W. Dziuban, Jr, MD, Fred H.Edwards, MD, Edward 1. Hannan, PhD, Lisa I. Iezzoni, MD,JoAnn Lindenfeld, MD, William e. Nugent, MD, GeraldO'Connor, PhD, Larry Seidl, MD, Paul S. Steen, MD, and H.Gareth Tobler, MD.

References

1. Council on Medical Service. Quality of care. JAMA 1988;256:1032-4.

2. Tjosvold D. Constructive controversy, a key strategy forgroups. Personnel 1986;63:39-44.

3. Lohr KN, Schroeder SA. A strategy for quality assurance inMedicare. N Engl J Med 1990;322:707-12.

4. Donabedian A. Quality assessment and assurance: unity ofpurpose, diversity of means. Inquiry 1988;25:173-92.

5. Nugent WC, Shults We. Playing by the numbers: how col­lecting outcomes data changed my life. Ann Thorac Surg 1994;58:1866-70.

6. Greenfield S, Aronow HU, Elashoff RM, Watanabe D. Flawsin mortality data: the hazards of ignoring comorbid disease.JAMA 1988;260:2253-5.

7. Green J, Winfeld N, Sharkey P, Passman LJ. The importanceof severity of illness in assessing hospital mortality. JAMA1990;263:241-6.

8. Dubois RW, Brook RH, Rogers WH. Adjusted hospital deathrates: a potential screen for quality of medical care. Am JPublic Health 1987;77:1162-7.