Transcript
Page 1: Reporting health care performance: learning from the past, prospects for the future

Introduction

In most health systems, information on the quality of care provided by clinicians, hospitals and otherhealth care organizations has traditionally been col-lected for internal quality assurance and has almostalways remained confidential (Schneider & Epstein

1998). However, recent years have witnessed anincreasing volume of data on clinical performancebeing collated externally and being released into thepublic domain. In the USA, where public reportingis most advanced, comparative performance infor-mation, in the form of ‘report cards’, ‘provider pro-files’ and ‘consumer reports’, has been released for

Journal of Evaluation in Clinical Practice, 8, 2, 215–228

© 2002 Blackwell Science 215

CorrespondenceDr Russell MannionCentre for Health EconomicsUniversity of YorkHeslingtonYork YO1 5DDUKE-mail: [email protected]

Keywords: clinical indicators,international lessons, publicdissemination, report cards, US experience

Accepted for publication:23 November 2001

AbstractIn the USA, where public reporting of data on clinical performance is mostadvanced, comparative performance information, in the form of ‘reportcards’, ‘provider profiles’ and ‘physician profiling’, has been published forover a decade. Many other countries are now following a similar route andare seeking to develop comparative data on health care performance.Notwithstanding the idiosyncratic nature of US health care, and the impli-cations this has for the generalizability of findings from the USA to othercountries, it is pertinent to ask what other countries can learn from the USexperience. Based on a series of structured interviews with leading expertson the US health system, this article draws out the key lessons for othercountries as they develop similar policies in this area. This paper highlightsthree concerns that have dominated the development of adequate measuresin the USA, and that require consideration when developing similarschemes elsewhere. Firstly, the need to develop indicators with sound metricproperties – high in validity and meaningfulness, and appropriately risk-adjusted. Secondly, the need to involve all stakeholders in the design ofindicators, and a requirement that those measures be adapted to differentaudiences. Thirdly, a need to understand the needs of end users and toengage with them in partnerships to increase the attention paid to mea-surement. This study concludes that the greatest challenge is posed by thedesire to make comparative performance data more influential in leverag-ing performance improvement. Simply collecting, processing, analysing anddisseminating comparative data is an enormous logistical and resource-intensive task, yet it is insufficient. Any national strategy emphasizing comparative data must grapple with how to engage the serious attention ofthose individuals and organizations to whom change is to be delivered.

Reporting health care performance: learning from the past, prospectsfor the future

Russell Mannion PhD1 and Huw T. O. Davies PhD, HonMFPHM2

1Senior Research Fellow, Centre for Health Economics, University of York, York, UK2Professor of Health Care Policy and Management, University of St Andrews, St Andrews, UK

Page 2: Reporting health care performance: learning from the past, prospects for the future

R. Mannion and H.T.O. Davies

over a decade (Epstein 1998; Davies & Marshall1999). In Europe, Scotland has been at the forefrontof releasing clinical outcome indicators and has dis-seminated such information since 1994 (Mannion &Goddard 2000). More recently, clinical performancedata have been published for hospitals in Englandand Wales as part of the National Health Service(NHS) Performance Assessment Framework. Underthe recently launched NHS Plan, the Commission forHealth Improvement will be given responsibility forpublishing ‘report cards’ on the performance of NHSorganizations (Department of Health 2000). Similarperformance-reporting systems are also being imple-mented in a number of countries, including Canada,New Zealand, Australia and Italy, and in Scandinavia(Peursem & Pratt 2000; Mariotto & Chitarin 1998;Anderson & Noyce 1992; Blais et al. 1999).

The report card movement in the USA is now overa decade old and has grown into a multi-million-dollar industry. Notwithstanding the idiosyncraticnature of US health care, and the implications thishas for the generalizability of findings from the USAto other countries (Davies & Marshall 2000), it is pertinent to ask: what can we learn from this accu-mulated US experience? The development of effec-tive external reporting systems for health care hasbecome a key policy issue in most developed nations;hence, such an analysis has the potential to influencemany national debates. The complexity of the task,potential high costs and considerable concerns overboth ineffective systems (Davies 1998) and dysfunc-tional ones (Smith 1995a, 1995b; Goddard, Mannion& Smith 2000) make it imperative that we attempt to synthesize evidence and learning from as manysources as possible.

Many commentaries and some reports of empiricalwork on the role of report card data have appearedover the years. Arguments about improving clinicalperformance have focused on the relative merits of published data (Anonymous 1993; Marshall et al.2000a, 2000b, 2000c), the role of patients (Lansky1996; Hibbard & Jewett 1997; Hibbard et al. 1997,1998; Lansky 1998), use of clinical indicators inquality initiatives (Thomson, McElroy & Kazandjian1997), the technical shortcomings of health outcomes(Davies & Crombie 1997), their lack of effect (Davies1998) or simple misguidedness (Davies & Lampel1998). Careful review work of empirical studies on

the effects of report cards by Marshall and colleagues(Marshall et al. 2000c) have highlighted a number ofcrucial features:

• despite calls for further relevant information,patients (and their representatives) do not appearto use report card data when making health carechoices;

• health care purchasers and referring doctors alsoseem to be little influenced by these data, and

• there is some limited evidence that health careproviders may utilize such data in internal qualityimprovement activities (Marshall et al. 2000c).

Thus, while reviews of empirical studies haveadvanced our understanding about the impact ofreport cards to some extent, it is our contention thatmuch of the accumulated wisdom of the US experi-ence remains locked up as tacit knowledge in keystakeholders. This work attempts to address this,accessing such knowledge by conducting a series ofin-depth structured interviews with leading expertson the US health system. Our stance is to unlock acritical review of previous US experience (with anemphasis on avoidable problems), and to encouragespeculation as to fruitful future pathways. Our aim is to draw out from the US experience of report cardsthe general lessons that might be used to inform the development of similar reporting initiatives elsewhere.

After explanation of the interviewing strategy,the rest of the paper is devoted to the exploration ofa number of key themes that emerged during the in-depth interviews:

• the role of performance information in the turbulent US health system;

• the major achievements of the report card move-ment in the USA;

• the major problems and challenges that still needto be overcome in relation to monitoring the performance of US health plans and health careproviders;

• the incentive context for performance measures,and the mechanisms by which report cards aremeant to leverage change;

• the debates over the use of outcomes data vis-à-vis process data;

• the unintended and dysfunctional consequencesarising from the publication of performance data,and

216 © 2002 Blackwell Science, Journal of Evaluation in Clinical Practice, 8, 2, 215–228

Page 3: Reporting health care performance: learning from the past, prospects for the future

• the relative balance between trusting clinicians’professionalism and checking up on clinical performance with quantitative report cards.

The final section draws on these discussions tocollate some lessons for other countries who areembarking on national strategies of quality mea-surement in health care.

Interviewing strategy

This article is based mainly on information derivedfrom structured interviews with 18 experts on the UShealth system. This information is then placed in thecontext of an extensive literature. Our intervieweeswere selected using a purposeful sampling frame(Miles & Huberman 1994) to represent a range ofconstituencies and perspectives on performancemeasurement. They comprised leading academics

with an active research interest in this topic, seniorstaff of federal government departments with re-sponsibilities for performance measurement andsenior staff from consumer groups and public- andprivate-sector quality oversight organizations (seeTable 1). The content of the interviews was stand-ardized using a common schedule. On average,interviews lasted 60 minutes, and they were tape-recorded and transcribed prior to analysis. A the-matic analysis of the transcripts was conducted(Denzin & Lincoln 1994).This analysis identified pas-sages of text relating to specific themes and issues,which were then grouped into conceptual categories.To strengthen the validity of the analysis and con-clusions (Kirk & Miller 1992), all interviewees wereinvited to comment on a preliminary draft of thisarticle and the views solicited have been used in thefinal draft.

Reporting health care performance

© 2002 Blackwell Science, Journal of Evaluation in Clinical Practice, 8, 2, 215–228 217

Table 1 US interviewees

Interviewee Post and organization

Robert H. Brook Vice-President and Director of RAND Health, RAND, Santa Monica, CADonald M. Berwick President and Chief Executive Officer, Institute for Health Care Improvement and Clinical Professor

of Pediatrics and Health Care Policy, Harvard Medical SchoolDavid Blumenthal Director of the Institute for Health Policy, Massachusetts General Hospital, Boston, MAMark R. Chassin Professor and Chairman, Department of Health Policy, Mount Sinai School of Medicine, New York

CityPaul D. Cleary Professor of Health Care Policy, Department of Health Care Policy and Social Medicine, Harvard

Medical SchoolShan Cretin RAND, Santa Monica, CAMolla Donaldson Project Director, National Roundtable on Health Care Quality, Institute of Medicine, Washington, DCDavid Eddy Senior Advisor for Health and Social Policy, Kaiser Permanente, Southern CaliforniaEmmett Keeler RAND, Santa Monica, CAMarge Keyes Project Officer, Centre for Quality Measurement and Improvement, Agency for Health Care Policy

and Research, Rockville, MDDavid J. Lansky President of the Foundation for Accountability (FACCT)Jerod Loeb Vice-President of Research and Evaluation, Joint Commission on Accreditation of Health Care

OrganizationsHarold Luft Director, Institute for Health Policy Studies, University of California, San Francisco, CAElizabeth A. McGlynn Director of the Center for Research on Quality in Health Care, RAND, Santa Monica, CAPatricia McTaggart Director of Quality and Performance Management, Center for Medical and State Operation, Health

Care Financing Administration, Washington, DCSandra K. Robinson Consumer Research Co-ordinator, Centre for Quality Measurement and Improvement, Agency for

Health Care Policy and Research, Rockville, MDJoshua Seidman Director, Measure Development, National Committee for Quality Assurance, Washington, DCStephen Shortell Blue Cross of California, Distinguished Professor of Health Policy and Management, University of

California, School of Public Health, Berkeley, CA

Page 4: Reporting health care performance: learning from the past, prospects for the future

R. Mannion and H.T.O. Davies

Health care performance information in the USA

Health care in the USA has undergone seismic shiftsover the past two decades. Of overriding importancehas been the swing away from indemnity healthinsurance tied to fee-for-service reimbursement, witha concomitant rise (and in many areas, a predomi-nance) of managed care arrangements. The keydriver over this period has been a preoccupation withcost-containment. Such intensive tracking of patientsfor billing purposes has led to a system replete withdata, yet one lopsided in its attention to cost issuesover quality concerns. Many initiatives to address thishave focused on improving the availability of quality-related data.

In the USA, the public release of clinical out-comes data dates back to the decision in 1987 by theHealth Care Financing Administrations to publishhospital mortality data (Vladeck et al. 1988). Theseindicators, based on crudely adjusted administra-tive data, were severely criticized for lacking credi-bility and were subsequently abandoned in 1993. Thefirst ‘report card’ based on clinical data specificallydesigned for assessing quality was the New YorkCardiac Surgery report card, first published in 1990(Hannan et al. 1994, 1995), which was closely followed by the Cardiac Surgery Report Card inPennsylvania (Bentley & Nash 1998). Many stategovernments now mandate the collection, analysisand publication of comparative performance in-formation, and websites offering health plan andphysician comparisons are abundant (Davies & Marshall 1999). The two largest accreditors of healthplans in the USA (the National Committee onQuality Assurance and the Joint Commission on theAccreditation of Health Care Organizations) bothnow stipulate the public reporting of performancedata as an accreditation criterion. Similarly, statutorybodies such as Health Care Financing Administration(HCFA) have also recently established a new seriesof quality standards for health plans that includepublic performance reporting through the MedicareCompare Program. All health plans that serveMedicare beneficiaries are now required to reportannually on the technical quality of care measured bythe Health Plan Employer Data and Information Set(HEDIS) and on patient-reported measures of satis-

faction and quality, as measured by the ConsumerAssessment of Health Plans Survey (Epstein 2000).

For all the recent activity around performancereporting, The President’s Commission on ConsumerProtection and Quality in the Health Care Industry(1998) stated that existing information systems werenot adequate. It concluded that the US health careindustry would need to make a significant investmentin such systems in order to provide high-quality dataon the individual and comparative performance ofplans, facilities and practitioners.Thus, one key lessonfrom the US experience is that data produced as aby-product of the health care business do not providea sufficient basis for quality reporting. Although it istempting to utilize existing data, the problems in sodoing (e.g. data quality, data analysis, data validityand data meaningfulness) are legion.

Major achievements of the report card movement

From small and uncertain beginnings, the report cardmovement in the USA has grown into a huge anddiverse set of overlapping activities. These rangefrom local ad hoc data collection, through state-mandated schemes to national federal requirements.The public availability of the Medicare and Medicaiddatabases has fostered significant unregulated activ-ity in the areas of comparative analysis and publica-tion. From our series of interviews, it was clear thatour respondents viewed the report card movement asbeing the primary driver in effecting a major culturalshift in traditional clinical governance relationshipsin the USA. Whereas little over a decade ago moni-toring of clinical performance was viewed as an issueof self-regulation, guarded jealously by the medicalprofession, today external systems of measurement,monitoring and audit are not only accepted but areviewed as an essential component of a high-qualityand accountable health service. According to onerespondent, there has been ‘a sea change in attitudeabout the acceptability of the activity itself’. Anotheron the same theme said ‘it’s been a 10 year processof establishing the legitimacy of the report cardapproach’, while a third commented that ‘just gettingany comparative data across institutions is an impor-tant accomplishment – the major value is that you arecreating precedents’.

218 © 2002 Blackwell Science, Journal of Evaluation in Clinical Practice, 8, 2, 215–228

Page 5: Reporting health care performance: learning from the past, prospects for the future

Despite a lack of evidence from research studiesof the impact of report cards on purchasers, severalof our respondents claimed to detect some shiftsaway from purchasing on cost to ‘purchasing forvalue’. For example, they cited instances of majoremployers using quality data to select health plansand offering bonus payments to high-quality pro-viders. Therefore, there were some expectations thatthe market forces so dominant in US health care maybe refocused by the increasing availability of qualityinformation. Part of the pressure in this direction isthe now-abundant evidence of major variations inthe quality of care (Anderson 1990). Thus, reportcards have served to highlight major disparities inquality and performance between different healthcare organizations and staff. Knowledge of these disparities was seen as bringing pressure to bear onprovider organizations [as highlighted by Marshallet al. in their review (Marshall et al. 2000c)], pur-chasers and large employers. Comments from ourinterviewees included: ‘one of the major achieve-ments is awareness on the consumer side that there’svariation in quality and performance,’ and, ‘by far themajor achievement has been to raise the visibility ofquality and outcomes to the general public’.

The growing awareness of variations in quality, andthe increasing interest in examining these, have inturn led to greater attention to the tools and tech-niques for measuring quality. Routine administrativedata for paying claims is still the basis of many measurement systems, but there have been seriousattempts to improve the reliability, validity andcoding practices, etc., around those sources. ‘There’sa recent very big change in that the people who enterthe International Classification of Disease (ICD)codes recognize for the first time that these things getused for things besides paying claims and are devel-oping coding practices so that these data can be usedfor quality measurement.’ In addition, greater atten-tion has been paid to the development of moresophisticated risk-adjustment and performancemetrics (although most people conceded that theseare currently far from perfect): ‘we have much betterideas about technical concerns such as severityadjustment than existed 15 years ago. So I’d say it’s on the instrumentation side where majorachievements have been made’.

In short, our respondents highlighted that the very

act of collecting and disseminating clinical perfor-mance data had changed the context within whichsuch data were collected. These changes included cultural shifts to bring quality issues to the fore and legitimize its examination, knowledge shifts interms of increasing awareness of quality problemsand quality variation, and technical shifts in terms ofimproved data systems and measurement tools (e.g.refined indicator definitions and more sophisticatedanalytic approaches, including risk adjustment).

Major problems and challenges to be overcome

Notwithstanding the perception that major strideshave been made in technical issues, it was to this areathat our respondents returned when asked about theoutstanding challenges that remain. Many still hadmajor concerns over measurement and interpreta-tion, including data quality, validity, reliability, ac-curacy and risk adjustment. Respondents cited thattoo much quality information was based on routineadministrative data, with a lack of standardizationand inadequate risk adjustment. Such sources makecomparative performance evaluation hazardous andcan lead to both type I (erroneous outlier status) andtype II (false reassurance over performance) errors.Particularly prominent were concerns over riskadjustment as an ‘inexact science’ that can under- orover-adjust for patient risk factors. In any case, riskadjustment can only adjust for known prognosticfactors; indeed, different risk adjustment models cangive rise to very different findings. A considerablebody of published research supports these concerns(Iezzoni et al. 1996; Iezzoni 1997; Hofer & Hayward1996; Davies & Crombie 1997).

A second area of concern lay in the somewhatpatchy participation of health plans and health careproviders in both performance monitoring andquality improvement: ‘there are problems aroundestablishing a mandate or a compulsion to disclosedata. All the systems here are voluntary and there-fore there’s very poor compliance and poor univer-sality’. This was seen as highly undesirable.

In contrast to under-participation by providers/health plans, the arena of quality oversight agencieswas seen as crowded, duplicative and somewhatlacking in co-ordination. The perception was of a

Reporting health care performance

© 2002 Blackwell Science, Journal of Evaluation in Clinical Practice, 8, 2, 215–228 219

Page 6: Reporting health care performance: learning from the past, prospects for the future

R. Mannion and H.T.O. Davies

need to develop an industry-recognized standardthat could be used to compare all health plans and providers. The present diverse and ad hocarrangement of information systems provides data ofvariable quality and usefulness. Respondents saw apotential for synergy arising from standardization of matters such as risk adjustment and coding practices. A national co-ordinating group called the Performance Measurement Co-ordinating Council(PMCC) was established to look into this. ‘ThePMCC is a collaboration between NCQA, theJCAHO and [the AMA’s accreditation programme].The goal is to try to build efficiencies both in thedevelopment of performance measures and also inthe implementation of them so what we are trying todo is build a common clinical logic that underliesmeasure development in different clinical areas.’

After technical issues, participation and co-ordination, the key challenge was seen as promotingthe use of performance measures in bringing aboutchanges in service patterns and health care delivery.One of the main difficulties identified was concernedwith communicating data to different audiences:‘there are problems making report cards useful to thetarget audience and I think that this is the next areathat will require a lot of attention’. A second keyconcern was in finding ways to ensure that the datahad impact: ‘we have very little firm knowledge abouthow these data are used by decision-makers, andwhat knowledge we have says they don’t use themvery much’; ‘we’re nearly totally ignorant about howindividual health consumers are using any healthinformation they are getting’.

In their estimation of the difficulties of communi-cating performance information, our respondentswere accurately reflecting a significant body of em-pirical evidence. US consumers have access to largeamounts of clinical information compared with theircounterparts in most other countries. However, aconsistent research finding in this area suggests thatalthough there is a considerable amount of interestamong the general public in the notion of obtaininginformation on the comparative performance ofhealth care providers, in reality consumers rarelyseek out or act on this type of information (Hibbardet al. 1997; Davies & Marshall 1999). Our interviewsrevealed a number of reasons why this might be thecase:

• Firstly, little effort has generally been made bythose designing report cards to seek input frompotential users. Problems around the compre-hensibility and interpretability of measures areabundant, with many of the reports having a professional focus on data for biomedical and epidemiological purposes. In addition, consumersmay not understand quantitative statistics, andmay be more interested in access- and process-type data rather than health outcomes.

• Secondly, our respondents suggested that manymembers of the public have little confidence inofficial statistics.When making judgements on theperformance of health care providers and plans,they tend to rely on more trusted sources of infor-mation, such as informal and traditional kinshipnetworks (e.g. friends and family).

• Thirdly, performance information is often not ofmuch use to consumers as they often do not havean alternative provider within a reasonable trav-elling distance. Also, for some aspects of healthcare (such as heart attack), patients have little orno time to seek out consumer reports comparinghospitals.

Of course, consumers are not the only audience for quality information: health care providers arealso targeted. Indeed, an empirical study suggeststhat this is the one part of the health system thatmight be most susceptible to using report card data(Marshall et al. 2000c). However, evidence of theimpact on providers of published clinical data issomewhat equivocal and our interviewees’ responsesreflected this. One key concern raised was the dif-ficulty of intermeshing external data with localquality improvement data (Davies 2001). Externalreporting can direct attention to general areas ofconcern but detailed local data is needed to bringabout significant quality improvement.

From our interviews, it is clear that if report cardsare to be relevant to health care stakeholders thenthey need to be designed in such a way that they arecomprehensible and salient to key decision-makers.This will require continued technical improvement,leading to better reliability and validity of the infer-ences drawn from the data. Furthermore, greaterefforts are required to co-ordinate report card activ-ity, including some degree of mandatory participa-tion. Above all, the major challenge lies in finding

220 © 2002 Blackwell Science, Journal of Evaluation in Clinical Practice, 8, 2, 215–228

Page 7: Reporting health care performance: learning from the past, prospects for the future

ways to ensure that the data produced bring aboutappropriate and effective actions by health plans andservice delivery organizations.

The incentive context

An incentive can be defined as a reward or sanc-tion associated with a particular aspect of perfor-mance. Individual rewards or sanctions can takemany forms, including: personal financial rewards;intrinsic rewards (‘a job well done’); peer reputa-tion; career development; additional budget forservice development; time to pursue other activitiessuch as research; reduced levels of inspection, or arequirement to engage in professional developmentor retraining (Dawson et al. 2000). Good incentiveschemes should perform the following functions:reward desired actions; discourage unwanted actions;indicate organizational priorities; secure a betteralignment between individual and organizationalobjectives, and generate organizational benefits thatexceed costs. We explored with our interviewees howdata on clinical performance were incorporated intothe incentive context of US health care.

Our respondents were of the opinion that in theUSA, there are few formal incentives attached toperformance on report cards. In particular, marketincentives were generally seen as weak, as con-sumers, employers and insurers rarely shift customon the basis of quality criteria – they are more influ-enced by cost considerations: ‘the original notionbehind the report cards was market leverage – votingwith your feet – the view that people would leave indroves those health plans, hospitals or doctors whowere not performing that well. I don’t think there ismuch evidence of that happening’. Others viewed themarket as being important not because of actualshifts in market patterns but because of fears thatthese might happen: ‘I think competition is veryimportant – but not in the classical sense by causingpatients to shift in dramatic numbers from oneprovider to another based on a piece of data, as itwon’t work. I think providers are fearful of beingadversely affected, whether or not they are adverselyaffected’. Such fears were seen as potentially impor-tant incentives for change.

The incentives that people did identify seemed tobe around the softer issues such as reputation, status

and professional pride – all of which may beimpacted by an approach that involves ‘naming andshaming’: ‘the incentives are very weak. At the orga-nizational level it is primarily embarrassment’; ‘thevalue of directing them [i.e. report cards] at providersis that regardless of whether individual consumersuse them or not there’s certainly a belief that itmakes providers pay more attention because theydon’t want to look bad’.

Finally, one fundamental concern identified wasthat tying performance measures to sanctions such as accreditation reflects an emphasis on minimumstandards of competence rather than on the pursuitof excellence: ‘accreditation also, but remember thistends to set floors not ceilings – but maybe that’s allthat’s needed’.

In sum, even in the rampantly market-orientedUSA, report card data were not seen as significantdrivers of the health care market. This suggests thatin most other countries, serious consideration needsto be given to: (a) ensuring that incentives are at leastnot misaligned with quality (rather than attemptingto devise significant new incentives tied to perfor-mance), and (b) maximizing the influence of thesofter incentives such as impact on reputation andstatus.

Outcome- versus process-based indicators

One of the most fundamental choices in the designof a clinical performance indicator system is the deci-sion about whether to focus on process or outcomemeasures of care. While a focus on health outcomesis intuitively appealing (because outcomes are linkedmore closely to the ultimate objectives of a healthcare system), there may be legitimate reasons why insome contexts a focus on process is more appropri-ate (Davies & Crombie 1997; Crombie & Davies1998). Our interviewees typically expressed the viewthat an optimal solution would probably comprise a judicious mix of process and outcome measures,tailored to local circumstances. This viewpoint is captured neatly by one interviewee’s response: ‘whatwe need is a good mix of both outcome and processindicators – both serve their own purposes – and thechallenge is to develop reporting strategies that buildon the strengths of each’.The relative advantages anddisadvantages of process and outcomes measures,

Reporting health care performance

© 2002 Blackwell Science, Journal of Evaluation in Clinical Practice, 8, 2, 215–228 221

Page 8: Reporting health care performance: learning from the past, prospects for the future

R. Mannion and H.T.O. Davies

drawn from our interviews and from the literature,are detailed in Table 2.

Potential for dysfunctional consequences

The use of performance indicators, even when notspecifically tied to incentives, can lead to a range of

unintended or indeed dysfunctional consequences.Some of these are described in Table 3. In exploringthis issue with our respondents, a number of concernswere raised. First was a concern over ‘tunnel vision’:

Once you measure something people will divertenergy into it. A good example might be screen-ing rates in breast cancer. In the USA, we’ve

222 © 2002 Blackwell Science, Journal of Evaluation in Clinical Practice, 8, 2, 215–228

Table 2 Comparison of the relative advantages and disadvantages of process (a) and outcome (b) indicators

(a) Process indicators

Advantages Disadvantages

1. Readily measured: utilization of health technologies is 1. Salience: may have little meaning to relatively easily measured without significant patients unless the link to outcomes bias or error. can be explained.

2. Easily interpreted: utilization rates of different technologies 2. Specificity: they are often quite specific to a single can readily be interpreted by reference to the evidence disease or single type of medical care, so base rather than to inter-unit comparisons. process measures across several clinical areas

3. Sample size: compared with outcome indicators, process or aspects of service delivery may be required toindicators can identify significant deficiencies with much represent quality for a particular group of patients.smaller sample sizes. 3. Ossification: may stifle innovation and

4. Unobtrusive: can frequently be assessed unobtrusively the development of new modes of care(e.g. data stored in administrative or medical records). 4. Obsolete: usefulness may dissipate as technology

5. Indicators for action: failures identified in the process of and modes of care changecare provide clear guidance on what must be remedied 5. Adverse behaviour: process indicators are easily to improve health care quality. They are also acted upon manipulated and may give rise to gaming and more quickly than some outcome indicators, which only other adverse behaviourbecome available after a long time has elapsed (when it becomes too late to act on the data).

6. Coverage: can capture aspects of care (such as speed of access and patient experience) that are often valued by patients apart from health outcomes.

(b) Outcome indicators

Advantages Disadvantages

1. Focus: Direct attention towards the patient (rather than the 1. Attribution: may be influenced by many factors that service) and helps nurture a ‘whole-system’ collaboration. are outside the control of a health care organization.

2. Goals: more clearly represent the goals of the NHS. 2. Sample size: require large sample sizes to detect 3. Meaningful: tend to be more meaningful to some of the a statistically significant difference.

potential users of clinical indicators (patient, purchasers). 3. Timing: may take a long period of time to observe.4. Innovation: providers are encouraged to experiment with 4. Interpretation: may be difficult to interpret if the process

new modes of delivery. that produced the outcome occurred far in the past.5. Far-sighted: encourage providers to adopt long-term

strategies such as health promotion, which may realize long-term benefits.

6. Manipulation: are less likely to be manipulated than process indicators – although providers can influence risk-adjusted outcome by exaggerating the severity of patients (upstaging).

Derived and expanded from McGlynn (1998) and Davies et al. (1999).

Page 9: Reporting health care performance: learning from the past, prospects for the future

made mammographic screening rates a veryimportant performance measure in a lot of measurement systems.Well, that would get orga-nizations very interested in mammographyscreening, but it might not do anything toincrease their breast cancer care.

Fears were also raised that an emphasis on mea-sured performance may squeeze out innovativeservice development:

Hard measurements can be very helpful if whatyou are after is standardization and reduction in variation. For innovation, creatively develop-ing new ways to deal with patient needs, youneed softer, narrative, story-telling information.As measurement systems attach a premium to quantitative assessment, they inadvertentlydecrease creativity and innovativeness.

Therefore, experience from the USA confirms thatsome of the concerns over potential dysfunctionalconsequences are legitimate and need serious con-sideration in the design of any national indicatorsystem. Of course, such dysfunctional consequencesmay not be readily predictable, making this a crucial(although difficult) area for detailed research andongoing monitoring.

Balance between checking and trusting

The last two decades have seen significant changes in the balance between a trust in professionalism (e.g. professional self-regulation, confidential peer-review) and demands for external scrutiny (e.g.accreditation and credentialing, public measures ofperformance). Such changes have been exacerbatedby the radical restructuring of health care under the

new demands of managed care. These have led toconcerns over a diminution of trust in doctors andhealth care providers (Mechanic 1996; Mechanic &Schlesinger 1996) and worries about the extent towhich physicians are increasingly having to serve twomasters (Angell 1993; Shortell et al. 1998), as well as other concerns. The rise of report cards is onesignificant facet of these changes that contributes toa changed environment within which doctors work,possibly in a deleterious fashion (Davies & Lampel1998; Davies, Crombie & Mannion 1999).

This rise of external scrutiny reflects an emphasison a different sort of trust: not trust in professionalsto deliver quality improvements, but trust instead in abstract and impersonal systems of checking,verification and audit (Power 1997). Crucial ques-tions then arise as to whether the transfer of trustfrom professionals to systems is either warranted oreffective. In particular, we have to ask where thebalance should lie between trusting and checking(Davies & Mannion 1999a, 1999b; Davies & Rundall2000).

Many of the interviewees were of the opinion thatoversight has increased inexorably in the US healthcare system. Perhaps because of this, there is someevidence that the US medical profession is mountinga rearguard action by seeking to take on a leadershiprole in shaping oversight regimes. Although many in the medical profession are none too happy about the rise of oversight, some exemplary units viewincreased data as an opportunity both to demon-strate their excellence and to improve quality.

Telling quotes included the following:Physicians are certainly frustrated with thecurrent level of oversight, primarily because it’s

Reporting health care performance

© 2002 Blackwell Science, Journal of Evaluation in Clinical Practice, 8, 2, 215–228 223

Table 3 Possible dysfunctional consequences arising from performance indicators

Organizations or individuals may alter their behaviour in response to performance indicators in a variety of undesirable ways:

• Tunnel vision: concentration on those clinical areas measured to the detriment of other important areas.

• Sub-optimization: the pursuit of narrow objectives within a unit or organization at the expense of strategic co-ordinationwith others.

• Myopia: concentration on short-term issues and the neglect of long-term criteria.

• Convergence: an emphasis on not being exposed as an outlier rather than a desire to be outstanding.

• Ossification: a disinclination to experiment with new and innovative approaches for fear of appearing to perform poorly.

• Gaming: altering behaviour to gain strategic advantage.

• Misrepresentation: including creative accounting and fraud.

Adapted from Smith (1995a, 1995b).

Page 10: Reporting health care performance: learning from the past, prospects for the future

R. Mannion and H.T.O. Davies

bad oversight. However, I don’t think there isany likelihood of a shift back towards increasedautonomy as the profession has not demon-strated a successful record of delivering a con-sistently high quality of care.

There’s been many different types of reac-tions and some are very resentful and theirmorale is terrible. And anecdotally you alwayshear of early retirements . . . The reason why Idon’t think the profession has taken on a lead-ership role is that it would require admitting thatthere are quality problems . . . In the past it typically used to avoid accountability by saying‘you don’t need to inspect us – we’re profes-sional, we do it ourselves’, giving the qualityoversight agencies the brush-off.

In terms of the balance between oversight and pro-fessionalism, we’re definitely moving towards over-sight. Though there is a counter-current within theprofession to restore professionalism in the hope offending off increased oversight.

There was much concern over how the balancebetween checking and trusting should be struck.Interviewees acknowledged what gets lost when trustis absent, but nonetheless recognized that the mani-fest evidence of quality problems provides a damningcritique of professionalism alone as the key to high-quality health care. The overriding sense from over a decade’s experience of report cards was that thegenie was out of the bottle and was not going to bestuffed back in.

Performance reporting: lessons for other countries

The extensive US activity in measuring and publish-ing comparative data on clinical performance pro-vides rich experience from which others can learn.One dominant observation is clear: pursuing a quantitative, comparative and public approach tohealth care quality goes some way towards changingthe context within which such measurement takesplace. This alone may be sufficient to merit seriousactivity in this area.

A number of practical lessons for health policiesin those countries developing measured approachesto health care quality also emerged from these

interviews. In articulating these lessons, we are notblind to the vast cultural, economic, organizationaland political differences between countries (Davies& Marshall 2000). Instead, we offer such guidancemerely as a means of structuring local nationaldebates on the development and evolution of per-formance measurement in health care – thoseinvolved in such debates will have to assume theirown responsibility for local contextualization.

Co-ordinated systems with mandatory participation

National systems for performance measurementneed to be co-ordinated carefully to avoid duplica-tion. Participation may need to be made manda-tory, but there should also be scope for differentlevels of involvement (e.g. current UK plans for‘earned autonomy’ – see the NHS Plan athttp://www.nhs.uk/nationalplan). However, althoughparticipation in measurement may need to be orga-nized centrally, the actions following from measure-ment should largely be devolved to the organizationsconcerned:

My theory is that the organizations that aregoing to do best are the ones where there is very strong centralized leadership in terms ofnorms and standards of quality, but where theymaximize local autonomy for making thechange. Provide an infrastructure for measuringquality – but don’t try to control individualbehaviour.

Cost-effective systems

The cost of measurement systems needs to be moni-tored carefully. Trade-offs will be needed betweencheap (probably administrative) data sets and high-quality bespoke data sets built around detailed clinical processes. National systems based aroundinappropriate indicators may be very wasteful:

The main one to watch for me is costs. Becausewe have added billions and billions of dollars of excess cost with very complex and unco-ordinated and silly performance measurementsystems and you are about to waste a lot ofmoney also in the UK if you are not careful.

224 © 2002 Blackwell Science, Journal of Evaluation in Clinical Practice, 8, 2, 215–228

Page 11: Reporting health care performance: learning from the past, prospects for the future

Alignment with incentives

Formal, explicit incentives may have some role toplay when the sorts of behaviour change required are clear and the measurement of success is relativelyunambiguous. However, given the complexity ofmost health care, the main issues with the incentivecontext lie in two areas: the avoidance of disin-centives to performance, and the harnessing of the power of informal incentives (e.g. intrinsic motivation/professional pride).

Avoidance of dysfunctional consequences

The impression of our US experts was that dysfunc-tional consequences such as tunnel vision, ‘short-termism’ and an inhibition of innovation may wellarise as a result of performance monitoring. Therewas also a concern over the potential displacementof intrinsic motivations by external rewards arisingfrom a diminution of trust:

We’re better off on a system in which trust is thedominant theme. If you have a workforce that isnot trusted that would be very bad. It would bebad even in a manufacturing company let alonea complex service industry. So we’ve got to findour way back to trust and what is clear withthese physician groups is that you build trust byimproving, by showing that you are committedto continual improvement – that’s what gener-ates trust in your intentions – it’s a learningissue.

Identifying the presence and impact of perverseincentives and unintended consequences remains an important research goal. Thus, national strategiesthat emphasize external performance monitoringshould be alert to the pitfalls. These strategies needto be designed so as to mitigate potential dysfunc-tional consequences and should have monitoringstructures in place to detect their presence.

Learning not punishing

The health care business is far too complex to be conceptualized adequately as incentive-driven. Thus,our respondents frequently emphasized that external

monitoring should contribute to an ethos of learningrather than a climate of judging or incentivizing:

[We need] evidence-based policy and learning.Applying the principles of continuous qualityimprovement (CQI) to national policy-making.Namely: small trials, continuous learning, feed-back, the PSDA cycle. Plan, Study, Do and Act.It’s a learning health system. It’s a learning-centred approach to implementing health carereform.

Measurement design

Measurement design needs to be an ongoing itera-tive process, but one planned and co-ordinated toprevent the proliferation of inadequate measures orthe unnecessary duplication of effort. Existing datasets will almost certainly limit the inferences that can be drawn and the credibility of the data.Whethersuch deficiencies are fatal flaws or acceptable limita-tions will depend on the role and purpose of the data.Measures that are intended merely to direct internalattention to specific clinical issues may well cope withbeing less robust than those that are used in formalincentive schemes.

Three key concerns have dominated the develop-ment of adequate measures in the USA, and allrequire ongoing work. Firstly, the need to developindicators with sound metric properties: high in reliability, validity and meaningfulness, and appro-priately risk-adjusted. Secondly, the need to involve all stakeholders in the design of indicators, and arequirement that different measures be adapted todifferent audiences.Thirdly, a pressing need to under-stand the needs of end users, and engage with themin partnerships to increase the attention paid to measurement.

Linkages to change

Finally, it is the desire to make comparative per-formance data have greater impact that poses the greatest challenges. It is clear that passive dissemina-tion has little appreciable effect. Simply collecting,processing, analysing, displaying and disseminatingcomparative data is an enormous logistical andresource-intensive task – yet it is insufficient. Any

Reporting health care performance

© 2002 Blackwell Science, Journal of Evaluation in Clinical Practice, 8, 2, 215–228 225

Page 12: Reporting health care performance: learning from the past, prospects for the future

R. Mannion and H.T.O. Davies

national strategy emphasizing comparative data mustgrapple with how to engage the serious attention ofthose individuals and organizations to whom changemust be delivered. Naturally, this involves considera-tions of the incentive context and the dissemina-tion strategy, but it needs to go much further. Forexample, there is a need to educate and support end-users in utilizing the data, and dissemination strate-gies may need to be extended to enable infiltrationof informal networks and new ways of reachingfront-line staff (instead of relying on hierarchical dis-tribution) (Goddard et al. 2000). It is in these areasthat we have the most work to do and the least experience to guide us.

Acknowledgements

The authors would like to thank the interviewees,who without exception readily gave up their valuabletime to share with us their considerable knowledgeand experience of the US health system. We wouldalso like to thank Maria Goddard and Peter Smithfor comments on an earlier draft of this paper.

Russell Mannion acknowledges receipt of supportfrom the UK Department of Health in the form ofresearch funding for a programme of work at theCentre for Health Economics, University of York.Huw Davies was Harkness Fellow in Health CarePolicy at the University of California, San Francisco,during the development of this project. Therefore,this work was also supported by the CommonwealthFund, a New York City-based private independentfoundation. Huw Davies is sincerely grateful to theFund and the Institute for Health Policy Studies atthe University of California (San Francisco) for theopportunities afforded him by the Harkness Fellow-ship. However, the views presented here are those ofthe authors and not necessarily those any of thefunding bodies.

References

Anderson T.F. (1990) The Challenges of Medical PracticeVariations. Macmillan, Basingstoke.

Anderson B.G. & Noyce J.A. (1992) Clinical indicators andtheir role in management. Australian Clinical Review12, 15–21.

Angell M. (1993) The doctor as double agent. KennedyInstitute of Ethics Journal 3, 279–286.

Anonymous (1993) Dicing with death rates [editorial].Lancet 341, 1183–1184.

Bentley J.M. & Nash D.B. (1998) How Pennsylvania hos-pitals have responded to publicly released reports oncoronary artery bypass graft surgery. Joint CommissionJournal on Quality Improvement 24, 40–49.

Blais R., Larouche D., Pineault Boyle P. & Dube S. (1999)What do managers and physicians prefer as indicators ofclinical outcome? In Managing Quality: Strategic Issuesin Health Care Management (eds H.T.O. Davies, M.Malek, A. Neilson & M. Tavakoli), pp. 173–184. Ashgate,Aldershot.

Crombie I.K. & Davies H.T.O. (1998) Beyond health out-comes: the advantages of measuring process. Journal ofEvaluation in Clinical Practice 4, 31–38.

Davies H.T.O. (1998) Performance management usinghealth outcomes: in search of instrumentality. Journal ofEvaluation in Clinical Practice 4, 150–153.

Davies H.T.O. (2001) Public release of performance dataand quality improvement: internal responses to externaldata by US health care providers. Quality in Health Care10, 104–110.

Davies H.T.O. & Crombie I.K. (1997) Interpreting healthoutcomes. Journal of Evaluation in Clinical Practice3 (3), 187–200.

Davies H.T.O., Crombie I.K. & Mannion R. (1999) Perfor-mance indicators in health care: guiding lights or wreck-ers’ lanterns? In Managing Quality: Strategic Issues in Health Care Management (eds H.T.O. Davies, M.Malek, A. Neilson & M. Tavakoli), pp. 215–230. Ashgate,Aldershot.

Davies H.T.O. & Lampel J. (1998) Trust in performanceindicators. Quality in Health Care 7, 159–162.

Davies H.T.O. & Mannion R. (1999a) Clinical governance:striking a balance between checking and trusting. InReforming Markets in Health Care – an Economic Perspective (ed. P.C. Smith), pp. 246–267. Open Univer-sity Press, Milton Keynes.

Davies H.T.O. & Mannion R. (1999b) The rise of oversightand the decline of mutuality. Public Money and Man-agement 19 (2), 55–59.

Davies H.T.O. & Marshall M.N. (1999) Public disclosure ofperformance data. Does the public get what the publicwants? Lancet 353, 1639–1640.

Davies H.T.O. & Marshall M.N. (2000) UK and US healthcare systems: divided by more than a common language.Lancet 355, 336.

Davies H.T.O. & Rundall T. (2000) Managing patient trust in managed care. Milbank Quarterly 78 (4), 609–624.

Dawson D., Goddard M., Gravelle H. & Smith P. (2000)Designing Incentives in the NHS. Paper prepared for

226 © 2002 Blackwell Science, Journal of Evaluation in Clinical Practice, 8, 2, 215–228

Page 13: Reporting health care performance: learning from the past, prospects for the future

the UK Government Performance and ProductivityModernisation Action Team.

Denzin N. & Lincoln Y. (1994) Handbook of QualitativeResearch. Sage Publications, London.

Department of Health (2000) The NHS Plan: a Plan forInvestment, a Plan for Reform. HMSO, London.

Epstein A. (1998) Rolling down the runway: the challengesahead for quality report cards. Journal of the AmericanMedical Association 279, 1691–1696.

Epstein A. (2000) Public release of performance data: aprogress report from the front. Journal of the AmericanMedical Association 14, 184–186.

Goddard M., Mannion R. & Smith P. (2000) Enhancing performance in health care: a theoretical perspective onagency and the role of information. Health Economics9, 95–107.

Hannan E.L., Kilburn H., Racz M., Shields E. & ChassinM.R. (1994) Improving the outcomes of coronary arterybypass surgery in New York State. Journal of the American Medical Association 271, 761–766.

Hannan E.L., Siu A.L., Kumar D., Kilburn H. & ChassinM.R. (1995) The decline in coronary artery bypass graftsurgery mortality in New York state. Journal of the American Medical Association 273, 209–213.

Hibbard J.H. & Jewett J.J. (1997) Will quality report cardshelp consumers? Health Affairs 16, 218–228.

Hibbard J.H., Jewett J.J., Engelmann S. & Tusler M. (1998)Can Medicare beneficiaries make informed choices?Health Affairs 17, 181–193.

Hibbard J.H., Jewett J.J., Legnini M.W. & Tusler M. (1997)Choosing a health plan: do large employers use the data?Health Affairs 16, 172–180.

Hofer T.P. & Hayward R.A. (1996) Identifying poor-qualityhospitals: can hospital mortality rates detect quality pro-blems for medical diagnoses? Medical Care 34, 737–753.

Iezzoni L.I. (1997) The risks of risk adjustment. Journal ofthe American Medical Association 278 (19), 1600–1607.

Iezzoni L.I., Shwartz M., Ash A.S., Hughes J.S., Daley J. &Mackiernan Y.D. (1996) Severity measurement methodsand judging hospital death rates for pneumonia. MedicalCare 34 (1), 11–28.

Kirk J. & Miller D. (1992) Reliability and Validity in Qualitative Research. Sage Publications, London.

Lansky D. (1996) The facts about FACCT (Foundation forAccountability). Clinical Performance and QualityHealth Care 4 (4), 211–212.

Lansky D. (1998) Measuring what matters to the public.Health Affairs 17, 40–41.

Mannion R. & Goddard M. (2000) The Impact of Perfor-mance Measurement in the NHS, Report 1. EmpiricalAnalysis of the Impact of Public Dissemination of theScottish Clinical Resource and Audit Group Data.

Report prepared for the Department of Health. Centrefor Health Economics, University of York, York.

Mariotto A. & Chitarin M. (1998) External evaluation ofhealth care in Italy. Quality in Health Care 7, 168–169.

Marshall M.N., Shekelle P.G., Leatherman S. & Brook R.H.(2000a) Public disclosure of performance data: learningfrom the US experience. Quality in Health Care 9, 53–57.

Marshall M.N., Shekelle P., Brook R. & Leatherman S.(2000b) Public reporting of performance: lessons fromthe USA. Journal of Health Services Research and Policy5, 1–2.

Marshall M.N., Shekelle P.G., Leatherman S. & Brook R.H.(2000c) What do we expect to gain from the publicrelease of performance data? A review of the evidence.Journal of the American Medical Association 283, 1866–1874.

McGlynn E. (1998) Choosing and evaluating clinical per-formance measures. The Joint Commission Journal onQuality Improvement 24 (9), 470–479.

Mechanic D. (1996) Changing medical organization andthe erosion of trust. Milbank Quarterly 74, 171–189.

Mechanic D. & Schlesinger M. (1996) The impact ofmanaged care on patients’ trust in medical care and theirphysicians. Journal of the American Medical Association275, 1693–1697.

Miles M. & Huberman A. (1994) Qualitative Data Analy-sis. Sage Publications, London.

Peursem K. & Pratt M. (2000) Accountability for healthsector performance: what does the New Zealand publicwant? International Journal of Business PerformanceManagement 2, 11–19.

Power M. (1997) The Audit Society: Rituals of Verification.Oxford University Press, Oxford.

Schneider E.C. & Epstein A.M. (1998) Use of public performance reports: a survey of patients undergoingcardiac surgery. Journal of the American Medical Asso-ciation 279 (20), 1638–1642.

Shortell S.M., Waters T.M., Clarke K.W.B. & Budetti P.P.(1998) Physicians as double agents: maintaining trust in an era of multiple accountabilities. Journal of theAmerican Medical Association 280, 1102–1108.

Smith P. (1995a) On the unintended consequences of pub-lishing performance data in the public sector. Interna-tional Journal of Public Administration 18, 277–310.

Smith P. (1995b) Outcome-related performance indicatorsand organizational control in the public sector. In Per-formance Measurement and Evaluation (eds J. Holloway,J. Lewis & G. Mallory), pp. 192–216. Sage Publications,London.

The President’s Advisory Commission on Consumer Protection and Quality in the Health Care Industry

Reporting health care performance

© 2002 Blackwell Science, Journal of Evaluation in Clinical Practice, 8, 2, 215–228 227

Page 14: Reporting health care performance: learning from the past, prospects for the future

R. Mannion and H.T.O. Davies

(1998) Quality First: Better Health Care for All Ameri-cans. US Government Printing Office, Washington,DC.

Thomson R.G., McElroy H. & Kazandjian V.A. (1997)Maryland Hospital Quality Indicator Project in the

United Kingdom: an approach for promoting continuousquality improvement. Quality in Health Care 6, 49–55.

Vladeck B.C., Goodwin E.J., Myers L.P. & Sinisi M. (1988)Consumers and hospital use: the HCFA ‘death list’.Health Affairs 7, 122–125.

228 © 2002 Blackwell Science, Journal of Evaluation in Clinical Practice, 8, 2, 215–228


Top Related