Journal of Substance Abuse Treatment 42 (2012) 102–107

Brief article

Comparing alternative specifications of quality measures: Access topharmacotherapy for alcohol use disorders

Sara Fernandes-Taylor, (PhD)⁎, Alex H.S. Harris, (MS, PhD)

Center for Health Care Evaluation, VA Palo Alto Health Care System and Stanford University School of Medicine, Menlo Park, CA 94025, USA

Received 31 January 2011; received in revised form 25 May 2011; accepted 5 July 2011

Abstract

Availability and consideration of pharmacotherapy for the treatment of alcohol use disorders (AUD) are now consensus standards forevidence-based treatment. This study compares three competing specifications of the proportion of patients with AUD receiving approvedmedications. We examined how altering the numerator and denominator definitions affects observed rates of pharmacotherapy use andfacilities' percentile ranks. Using pharmacy and administrative data from the Veterans Health Administration (VHA), three measures ofpharmacotherapy receipt were calculated for 129 VHA facilities. Difference in measure specifications alters the overall estimates ofpharmacotherapy receipt but unevenly across facilities, with some experiencing no change in percentile rank and others decreasing orincreasing by over a quartile. The results demonstrate that the quality measures are not interchangeable, and the choice of which version toimplement is of high consequence for some facilities. Published by Elsevier Inc.

Keywords: Alcohol use disorders; Pharmacotherapy; Quality measurement; Veterans

1. Introduction

The U.S. Food and Drug Administration (FDA) hasapproved four medications for the treatment of alcoholdependence—disulfiram, oral naltrexone, acamprosate, andextended release injectable naltrexone. Clinical evidence hasmounted, generally supporting the efficacy of thesepharmacotherapies, especially acamprosate and both formu-lations of naltrexone. The efficacy of naltrexone is supportedby the Combined Pharmacotherapies and Behavioral In-terventions for Alcohol Dependence (COMBINE) trial(Anton et al., 2006), the largest completed treatment trialof alcohol dependence. In addition, a Cochrane Reviewexamining 50 randomized controlled trials (N = 7,793) foundthat naltrexone reduced the risk of heavy drinking to 83% ofthat in the control groups and decreased total drinking days

Funding/Support: This study was partially supported by the VA Officeof Research and Development, Health Services Research and DevelopmentService (IIR-07-092).

⁎ Corresponding author. Tel.: +1 310 963 6258.E-mail address: sarataylorrohe@gmail.com (S. Fernandes-Taylor).

0740-5472/11/$ – see front matter. Published by Elsevier Inc.doi:10.1016/j.jsat.2011.07.005

almost 4% (Rosner, Hackl-Herrwerth, Leucht, Vecchi, et al.,2010). Although the COMBINE trial did not find support forthe use of acamprosate, the Cochrane Review on the use ofacamprosate for alcohol dependence reviewed 24 random-ized controlled trials (n = 6,915) and found that it reduced therisk of drinking to 86% of that in the control groups with asignificant increase in the duration of abstinence (Rosner,Hackl-Herrwerth, Leucht, Lehert, et al., 2010).

Thus, the availability and consideration of thesemedications are now among the consensus standards forevidence-based treatment for alcohol dependence and havebeen approved by the American Psychiatric Association/Physician Consortium for Performance Improvement/National Committee for Quality Assurance (2008). TheVeterans Health Administration (VHA) Uniform MentalHealth Services Benefits Handbook and VA/DoD ClinicalPractice Guidelines for the Management of Substance UseDisorders also clearly support the availability andactive consideration of these pharmacologic treatments(Department of Veterans Affairs, 2008).

Despite general consensus that these medications shouldbe available and considered for patients with alcohol

103S. Fernandes-Taylor, A.H.S. Harris / Journal of Substance Abuse Treatment 42 (2012) 102–107

dependence, the use of these medications generally remainslow and highly variable within many health care systems.(Harris, Kivlahan, Bowe, & Humphreys, 2010; Mark, Kassed,Vandivort-Warren, Levit, & Kranzler, 2009). Many barriers togreater use have been documented, including incongruencewith addiction treatment program philosophy, lack ofinformation and low perceived efficacy among providers,concerns about side effects, lack of patient demand, formularyrestrictions, and high copayments (Horgan, Reif, Hodgkin,Garnick, & Merrick, 2008; Mark et al., 2003).

One important response to the perceived underutilizationand variability of access to pharmacotherapy for alcohol usedisorders (AUD) has been the development of qualitymeasures in this domain. The Washington Circle, a group ofnational experts in the substance abuse policy, research, andperformance management, is in the process of developingand testing performance measures for the pharmacologictreatment of alcohol dependence using commonly availablein pharmacy and claims data (Thomas et al., 2011). The VAProgram Evaluation and Resource Center is engaged in aparallel process.

The VHA has invested resources in several areas ofperformance measurement using both financial and compet-itive incentives. The Office of Quality and Performance forthe VHA system coordinates national performance monitor-ing efforts and works with regional quality management.Regional directors receive a 10% salary bonus by meetingcertain performance benchmarks (Kerr & Fleming, 2007)and create corresponding pay-for-performance incentiveswithin facilities in their region. In addition, facilities'performance results are disseminated on a VHA intranetsite. Besides the measures that are included in executiveperformance contracts, mental health and substance disorderevaluation centers monitor various access and processmetrics to identify gaps in services and target implementa-tion efforts. Performance measures are largely based onevidence-based clinical practice guidelines created jointly byVHA and the Department of Defense and undergo a rigorousreview process before recommendation for adoption to theUndersecretary for Health. A description of the VHA'sperformance monitoring and measurement efforts is avail-able in Humphreys, Harris, and Kivlahan (2009).

Current VHA performance monitoring efforts related toSubstance Use Disorders (SUD) care include (a) percentageof outpatients screened for alcohol misuse, (b) percentage ofpatients who screen positive for alcohol misuse receivingbrief alcohol counseling, (c) percentage of tobacco usersreceiving cessation counseling, medication, or programreferral in the past 12 months, and (d) percentage of patientsbeginning new episodes of SUD specialty care meetingcertain criteria (Humphreys et al., 2009). The pharmacother-apy measures reviewed here represent potential futuremeasures of the quality of SUD care that keep pace withthe evolving standard of care, are consistent with measuresproposed within VHA and the Washington Circle, and aretractable using existing data available in VHA records.

As is often the case in the development of qualitymeasures, data to operationalize the standard of care, in thiscase consideration of pharmacotherapy, do not exist in auseful form. Whether pharmacotherapy is considered maynot be recorded in the medical record or may be mentionedonly in the text of progress notes. Although a CommonProcedural Terminology code for counseling regardingpsychosocial and pharmacologic treatment options foralcohol dependence (4320F) exists, it was only used 12times in VHA in 2010. Consequently, it is difficult todetermine for patients with AUD who do not receivepharmacotherapy if active consideration of pharmacothera-py occurred either by the clinician or patient. In addition,from a quality measurement perspective, it is problematicthat the proportion of patients that should actually receivethese medications, based on clinical judgments, medicalcontraindications, and patient preferences, has not yetbeen determined.

However, it may be possible to use the proportion ofdiagnosed or treatment-seeking patients receiving pharma-cotherapy as a rough proxy for availability and consider-ation. If this proportion is very low or zero at a facility, onemight conclude that the medications are either not availableor are not being routinely considered. Using this generalstrategy, although with different specifications, the Wash-ington Circle, the VA Program Evaluation and ResourceCenter, and others have proposed very similar core measuresof pharmacotherapy access, roughly described as theproportion of patients with a relevant AUD diagnosis orservice during the measurement year who also receive atleast one prescription for any FDA-approved medication foralcohol dependence during the same measurement year. Aswith most quality measure development efforts, manyseemingly small decisions need to be made that might affectthe validity or performance of the measure in unknown ways.

This study compares three competing specifications of thesame general domain—the proportion of AUD patientsreceiving pharmacotherapy. The questions of interest are (a)Do observed rates of pharmacotherapy use vary between themeasures? and (b) Do the facility percentile ranks changefrom measure to measure? If the three candidate measures donot change estimates of pharmacotherapy rates or relativefacility rank, then the simplest and easiest version should bechosen. However, if overall performance or relativeperformance varies substantially between measures, thenchoosing between them becomes more complicated.

2. Materials and methods

Patients with either alcohol abuse or dependencediagnoses were included, even though the medications areapproved only for alcohol dependence. Although theprimary impetus for this inclusion is the proposed elimina-tion of the abuse–dependence distinction in the upcomingDiagnostic and Statistical Manual of Mental Disorders,

104 S. Fernandes-Taylor, A.H.S. Harris / Journal of Substance Abuse Treatment 42 (2012) 102–107

Fifth Edition revision (www.dsm5.org), preliminary analysisand existing research also indicate that many patients carryboth diagnoses and many patients receive these medicationswith only an abuse diagnosis (Slutske et al., 1998; Harriset al., 2010). We included “in remission” diagnosesbecause patients in treatment are sometimes incorrectlydiagnosed as being in full remission. To identify allpatients who received an alcohol abuse or dependencediagnosis (International Classification of Diseases, NinthRevision, Clinical Modification codes 303.9× or 305.0×) infiscal years (FYs) 2007–2009 in 129 major VHA facilities(STA3N, the parent station identifier, which indicates theVA hospital or parent station of a branch at which thepatient was treated), we used the VHA National PatientCare Database. We used the VHA Decision SupportSystem inpatient and outpatient pharmacy benefits files todetermine the number of patients with AUD in each facilitythat filled at least one prescription for any of the four FDA-approved medications. The institutional review board ofStanford University and research committee of VA PaloAlto Health Care System approved the study protocol.

The three candidate quality measures are described inTable 1. Versions of Measure 1 are currently underconsideration in the VHA and Washington Circle (Thomaset al., 2011). Because the measure includes a single FY, theauthors speculated that a change in the period specified forthe measure would either generate few differences in facilityperformance (and lend significant credence to the robustnessof the measure) or generate differences in facility perfor-mance from measure to measure and thus qualify the resultsof the measure. Subsequently, the authors constructed twoalternative specifications of the measure.

The denominator of the first measure is the number ofpatients diagnosed with AUD in FY2009, and thenumerator is the number of patients in the denominatorwho received at least one prescription for any of the FDA-approved medications in FY2009. In other words, whatproportion of the patients with an AUD diagnosis this yearalso got medications this year? The denominator of thesecond measure (the same as that of the first measure) isthe number of patients who had an AUD diagnosis inFY2009. However, the numerator of the second measure isnumber of patients in the denominator who received atleast one prescription for any of the FDA-approvedmedications any time from FY2007 through FY2009. Inother words, what proportion of the patients with an AUD

Table 1Three calculations of the AUD performance measure

Measure 1 No: of patients with pharmacotherapy trial in 2009No: of patients with AUD diagnosis in FY2009

Measure 2 No: of patients with pharmacotherapy trial in last 2 yearsNo: of patients with AUD diagnosis in FY2009

Measure 3 No: of patients with pharmacotherapy trial in FY2009No: of patients with AUD diagnosis in FY2009without previous pharmacotherapy trial

diagnosis this year got an approved medication sometimeduring the past 3 years? This approach gives facilitiescredit for previous medication trials. The third measure hasthe same numerator as the first measure—the number ofpatients with AUD who received at least one prescriptionfor any of the FDA-approved medications in FY2009.However, the denominator is restricted to patients who didnot receive trials of medications in the previous 2 years. Inother words, what proportion of the patients with an AUDdiagnosis this year got medications for the first time thisyear? This version attempts to capture information oninitial trails of pharmacotherapy.

The analysis undertaken is a descriptive comparison ofthe three measures. The three measures are first comparedbased on their overall rates of pharmacotherapy across the129 VA facilities to illustrate how the absolute rate ofprescribing changes for each facility based on the measureused. The three measures are then compared in a pairwisefashion by fractional rank percent to examine how mucheach facility's relative percentile rank changes based on themeasure used. The differences in facilities' percentile rank,defined as the percentage of facilities that falls below a givenfacility in terms of prescribing rate, are quantified. Finally,interquintile movement by measure is examined to explorepossible implications for facilities in an incentive-basedpayment system.

3. Results

The three measures of pharmacotherapy receipt werecalculated for 129 VHA facilities, the results of which aredepicted in Fig. 1. All three measures show notable variationin the rates of pharmacotherapy between, but also within,facilities. Measure 1 has an overall facility-level mean of3.2%, with a minimum of 0% and a maximum of 11.6% ofpatients receiving pharmacotherapy for AUD. By givingfacilities credit for pharmacotherapy trials in the last 2 years(Measure 2), the facility-level mean rate of AUD pharma-cotherapy jumps to 5.1%, with a minimum of 0% and amaximum of 18.2%. Although Measure 2 is consistentlygreater than Measure 1 for each facility, the two measures donot track perfectly. Rather, within-facility differencesbetween Measures 1 and 2 vary from 0% to 6.6% dependingon how many patients in the denominator receivedpharmacotherapy in the previous 2 years. In contrast,excluding patients with a previous pharmacotherapy trialfrom the denominator (Measure 3) systematically decreasesthe rates of AUD pharmacotherapy. Measure 3 has a facility-level mean of 2.0%, with a minimum of 0% and a maximumof 7.5%. Like Measures 1 and 2, Measures 1 and 3 do nottrack perfectly, although deviations are somewhat smaller,ranging from 0% to 4.1%.

Generally, VHA facilities exhibit a range in the provisionof pharmacotherapy for AUD, with some facilities providingalmost no pharmacotherapy to patients with AUD and others

Fig. 1. Rate of VHA facilitates administering pharmacotherapy for AUD inFY2009 by measure (N = 129).

105S. Fernandes-Taylor, A.H.S. Harris / Journal of Substance Abuse Treatment 42 (2012) 102–107

prescribing pharmacotherapy for more than 10% of patients.The within-facility variation between measures is particu-larly notable. Some facilities vary widely in their rates ofpharmacotherapy provision based on the measure used.Other facilities' performance is less influenced by theparticular measure specifications.

These discrepancies have corresponding implications forthe facility-level percentile rank within the VHA. Thefacility's percentile rank is essential to monitoring facilityprogress/performance, to distributing performance-basedrewards (if any), and to any prestige that comes withdemonstrated excellence in performance measures. As aresult, differences in the percentile rank resulting fromdifferences in how the performance measure is calculatedwarrant attention. Fig. 2 shows the scatter plot of theassociation between facilities' percentile ranks for eachmeasure and the corresponding histogram of variation inindividual facilities' ranks. Most differences in rank betweenMeasures 1 and 2 are less than 10 percentile points, but somefacilities drop as much as 24.0 percentile points betweenMeasures 1 and 2 or increase their rank as much as 14.7percentile points. The spread is even greater betweenMeasures 2 and 3, where fewer percentile rank differencescluster around zero and the minimum and maximumpercentile rank differences between measures are −27.1and 29.5, respectively. Finally, Measures 1 and 3 areassociated in a manner similar to Measures 1 and 2; althoughmost facilities experience small variation in their percentilerank, some facilities experience significant movement intheir percentile ranking with a minimum of −20.2 and amaximum of 24.8.

Because many incentive-based quality initiatives arebased on a benchmark or a rougher metric than percentilerank, the facility-level changes from one measure to anotherwere also analyzed by quintile. As one would expect,interquintile movement occurs on the borders of eachquintile. Fifteen facilities moved down a quintile fromMeasure 1 to Measure 2, and 15 facilities moved up aquintile. Between Measures 1 and 3, 14 facilities moved upa quintile, and 14 moved down a quintile. Twenty-fivefacilities moved up a quintile from Measure 1 to Measure 3,

and 1 facility moved up two quintiles. Twenty-one facilitiesmoved down a quintile from Measure 1 to Measure 3, with 3facilities moving down two quintiles. These differencesbetween measures are reflected in the scatter plots of Fig. 2.The interquintile movement is notable given that, even usingthis rougher metric, between 22% and 40% of facilities'quintile membership are affected by changing the measure.

4. Discussion

Overall, variations in the way three performancemeasures for pharmacotherapy for AUD are specified havea noteworthy impacts on overall measured performancelevels and, perhaps more importantly, facilities' percentilerank within VHA. At the extremes, facilities' percentile rankcan change by more than a quartile depending on whetherthey (a) receive credit for diagnoses and pharmacotherapy inthe current FY with no regard for previous years, (b) receivecredit for patients who previously received a pharmacother-apy trial, or (c) receive credit only for diagnoses andpharmacotherapy trials in the current FY. Therefore, thesemeasures are not interchangeable and appear to tap subtlydistinct domains on which facility performance varies.Therefore, the choice of which version to adopt andimplement is of high consequence.

Although these analyses make clear that these measuresare not equivalent, only experts and stakeholders can judgewhich of the measures most closely maps onto the intendedprocesses of care. Issues of feasibility must also beconsidered; constructing measures from several years ofadministrative data is more complex and expensive thanmeasures that rely on data from a single year. In addition, itcan be argued that even patients who received trials ofmedications in previous years should be considered for andperhaps receive medications as long as their AUD diagnosisis active. Ultimately, these data can only assist qualitymeasure developers and stakeholders by showing that thesemeasures are meaningfully different but cannot indicatewhich to choose.

This work has several noteworthy limitations. First,treatment, diagnostic, and prescribing patterns in VHAprobably differ from other systems. The differences betweenthese measures may be greater or less when evaluatedoutside of the VHA context. We are also using one-timereceipt of any pharmacotherapy as a proxy for availabilityand consideration of these medications. Moreover, we didnot examine whether patients given pharmacotherapy forAUD received a sufficient course of therapy or whetherpatients took their medication as directed. In addition, we donot have data on VHA patients who filled AUD medicationprescriptions at non-VHA pharmacies. However, this isunlikely given that VHA prescription copayments tend to belower than pharmacy copayments elsewhere. There is a smallpossibility that an individual patient may not have received adiagnosis of alcohol abuse/dependence in 2007–2008, and

min=-24.03

max=14.73

min=-27.13

max=29.46

min=-20.16

max=24.81

Fig. 2. Comparison of facilities' percentile rank by measure.

106 S. Fernandes-Taylor, A.H.S. Harris / Journal of Substance Abuse Treatment 42 (2012) 102–107

the medication trial would therefore have come before thediagnosis was confirmed. However, pharmacotherapy isFDA-approved only for those patients that carry an abuse ordependence diagnosis, and the performance measuresnecessarily reflect this prescribing criterion. The possibilityalso remains that the style of practice at a given facility mayhave changed over time due to provider turnover. Given thatthe facility-level variable used is the parent station ratherthan the servicing facility, this is unlikely.

On the whole, this study sheds light on how variations inAUD pharmacotherapy performance measurement cangenerate considerable changes in how facilities rank inquality of care relative to their peers. Furthermore, thisanalysis generally demonstrates the need for careful pilottesting of performance measures, demonstrated links todesired outcomes, and a considerable evidence base beforeimplementation and attachment to financial incentives.Moving forward, AUD pharmacotherapy performance

107S. Fernandes-Taylor, A.H.S. Harris / Journal of Substance Abuse Treatment 42 (2012) 102–107

measures using administrative data should be justifiedcarefully, used in concert with other performance measures,and published alongside their strengths and drawbacks toavoid misrepresenting the quality of care in facilities.

Acknowledgments

The views expressed herein are the authors' and not thoseof the Department of Veteran Affairs.

References

American Psychiatric Association (APA) / Physician Consortium forPerformance Improvement® (PCPI) / National Committee for QualityAssurance (NCQA). (2008). Substance Use Disorders PhysicianPerformance Measurement Set. from http://www.ama-assn.org/ama1/pub/upload/mm/370/sud_ws_final.pdf.

Anton, R. F., O'Malley, S. S., Ciraulo, D. A., Cisler, R. A., Couper, D.,Donovan, D. M., Gastfriend, D. R., Hosking, J. D., Johnson, B. A.,LoCastro, J. S., Longabough, R., Mason, B. J., Miller, W. R., Pettinati,H. M., Randall, C. L., Swift, R., Weiss, R. D., Williams, L. D., &Zweben, A. (2006). Combined pharmacotherapies and behavioralinterventions for alcohol dependence: The COMBINE study: Arandomized controlled trial. JAMA, 295, 2003–2017.

Department of Veterans Affairs, V. H. A. (2008). VHA handbook 1160.01:Uniform mental health services in VA medical centers and clinics.Washington, DC: Author.

Harris, A. H., Kivlahan, D. R., Bowe, T., & Humphreys, K. N. (2010).Pharmacotherapy of alcohol use disorders in the Veterans HealthAdministration. Psychiatric Services, 61, 392–398.

Horgan, C. M., Reif, S., Hodgkin, D., Garnick, D. W., & Merrick, E. L.(2008). Availability of addiction medications in private health plans.Journal of Substance Abuse Treatment, 34, 147–156.

Humphreys, K., Harris, A. H. S., & Kivlahan, D. R. (2009). Performancemonitoring of substance use disorder interventions in the VeteransHealth Administration. The American Journal of Drug and AlcoholAbuse, 35, 123–127.

Kerr, E. A., & Fleming, B. (2007). Making performance indicators work:Experiences of the US Veterans Health Administration. BMJ, 335,971–973.

Mark, T. L., Kassed, C. A., Vandivort-Warren, R., Levit, K. R., & Kranzler,H. R. (2009). Alcohol and opioid dependence medications: Prescriptiontrends, overall and by physician specialty. Drug Alcohol Depend, 99,345–349.

Mark, T. L., Kranzler, H. R., Poole, V. H., Hagen, C. A., McLeod, C., &Crosse, S. (2003). Barriers to the use of medications to treat alcoholism.American Journal on Addictions, 12, 281–294.

Rosner, S., Hackl-Herrwerth, A., Leucht, S., Lehert, P., Vecchi, S., &Sokya, M. (2010). Acamprosate for alcohol dependence. CochraneDatabase of Systematic Reviews, 9, CD004332, doi:10.1002/14651858.CD004332.pub2.

Rosner, S., Hackl-Herrwerth, A., Leucht, S., Vecchi, S., Srisurapanont, M., &Sokya, M. (2010). Opioid antagonists for alcohol dependence. CochraneDatabase of Systematic Reviews, 12, CD001867, doi:10.1002/14651858.CD001867.pub3.

Slutske, W. S., True, W. R., Scherrer, J. F., et al. (1998). Long-termreliability and validity of alcoholism diagnoses and symptoms in alarge national telephone interview survey. Alcoholism: Clinical andExperimental Research, 22, 553–558.

Thomas, C. P., Garnick, D. W., Horgan, C. M., McCorry, F., Gmyrek, A.,Chalk, M., et al. (2011). Advancing performance measures for use ofmedications in substance abuse treatment. Journal of Substance AbuseTreatment, 40, 35–43.