1

Massachusetts Quality e-Measure Validation Study

(MQeVS)

Eric C. Schneider, MD, MScBrigham and Women’s Hospital

Harvard School of Public Health

Sponsor: AHRQ (R18 HS017048)

2

Performance Measurement: Fundamental to Improvement

Performance Visibility

Performance Rewards

Peers Patients Public Purchasers

PerformanceFeedback

Market Share Payments &Penalties

“Report Cards” “P4P”

3

Quality Measurement and Reporting: 1997

• Limited public demand

• Few standardized quality measures

• Few organizations engaged

• Few physicians aware

• Public disclosure rare

• Few patients aware

4

2007: Dramatic Increase in Standardized Measures

• Standard-setting organizations– National Quality Forum (NQF)– Ambulatory Quality Alliance (AQA)

• Engagement of Medical Profession– Physician Consortium for Performance Improvement

• AHRQ’s National Quality Measures Clearinghouse: (www.qualitymeasures.ahrq.gov)– Access - 22 measures – Outcome - 204 measures – Patient Experience - 298 measures – Population Health - 33 measures – Process - 636 measures– Structure - 44 measures – Use of Services - 33 measures

5

Implementation of Measurement and Reporting Remains Controversial

Measurement programs based on Administrative dataHybrid method (supplemental med record

review) Patient/enrollee surveys

Skepticism about validity of performance results

Burdensome, costly data collection

6

Colorectal Cancer Screening: Results by Method

Health Plan

Admin Hybrid Survey

A 35.9 35.9 53.0

B 38.6 53.5 69.7

C 47.2 52.7 55.9

D 27.3 38.9 62.1

E 44.4 45.5 66.2

Schneider et al, Under Review

7

Envisioning the EHR forPerformance Measurement

• Detailed, structured clinical data

• Unobtrusive data collection

• Performance data aggregated across care settings to enable sophisticated measures (e.g. care coordination, safety)

• Performance results at physician group rather than health plan level

Schneider et al, Enhancing performance measurement: NCQA’s Roadmap for a Health Information Framework. JAMA 1999;282:1184

8

Studies of Performance Measurement Using EHR and HIE

• Single institution

• Single care setting

• Single IT platform

• Limited number of performance measures

• Local, rather than national standards

9

Massachusetts e-Health Collaborative (MAeHC)

• 2004 Demonstration Project – $50 million from Blue Cross Blue Shield of MA– Universal EHR adoption in 3 MA communities– Intra- and inter-community data exchange (HIE)

• Integrated performance measurement/reporting – Massachusetts Health Quality Partners (MHQP)– AQA ambulatory performance measure starter set (26

measures)– Quality Data Warehouse

• Receives HIE data as needed to calculate measure results

10Courtesy of Micky Tripathi

11

MA Quality e-Measure Validation Study (MQeVS)

To compare a quality measurement method using structured, coded EHR data with…

1) a “hybrid method” involving a combination of aggregated claims data and medical record review.

2) a “claims-only method” based on a novel database that aggregates claims data from commercial health plans and Medicare.

12

MQeVS Highlights

• Implementation– MAeHC Pilot (www.maehc.org)– MHQP (www.mhqp.org)

• Evaluation– Community ambulatory practices “similar to” U.S.– Broad clinical and research expertise among research

partners• Harvard School of Public Health• MHQP • Partners Healthcare• Harvard Medical School• Center for Survey Research (U Mass, Boston)

13

MQeVS Sample

• Two Specific Aims assure that study addresses broad populations and measures– Aim 1: 900 patients with EHR-HIE data,

patient survey, medical record review, health plan administrative data

– Aim 2: All “measure eligible” patients with EHR-HIE data and health plan administrative data

14

HSPH (PI)Analysis

Team

SurveyTeam (CSR) Patients

Med Record Review Team

(Partners)

EHR VENDORS

PhysicianOffices

CSC/QDW

Medical Record Copies

Pre-notification

Re-identifiedStudy Cohort:Pt Contact Info

Study Consent, Survey, and Med Record Review Consent

Data Collection Protocol: Privacy/Confidentiality

Medical Record Requests

Medical Record Consent Cohort (Study IDs)

Completed Surveys and Consents

Med Record Extract (Aim 1)

Survey Data Extract(Aim 1) Opt-out

Step

EHR Quality Measure Extracts (Aims 1 & 2)

De-identified Study Cohort List (Aim 1)

MHQP

Claims Data Extract (Aims 1&2)

EHR Data

Mass E-Health Collaborative

#1

#2

#9 #10

#8

#5

#7

#6

#3

#4

15

Analysis: Availability of Data for Measure Components

Five key steps comprise a quality measure calculation algorithm, determining whether a patient meets criteria …1. for inclusion in the preliminary denominator

2. to be excluded from the preliminary denominator

3. for membership in the final denominator (after exclusions)

4. for membership in the measure numerator

5. for selection for both the numerator and denominator

16

Quality Measures: Deconstructing Data Needs

E=exclusion criteria; D=denominator inclusion; N=numerator inclusion; Var=varies

Age/Sex

Den Time

Window

Num Time

Window

EncData

DxData

Rx Data

ProcData

TestTest

Result

Colorectal Screening

D 2 yr 10 yr N E N N

Beta-blocker after MI

D 1 yr 7 d D D,E N

HbA1C Control

D 1 yr 1 yr D,E D,N

N

Eye exam D 1 yr 1 yr D,N D E N,E E

17

Data Adequacy Assessment

(1) Data Present, Event Confirmed: a lipid lowering medication is recorded in the patient’s data;

(2) Data Present, Event Not Confirmed: there is no lipid lowering medication, but data indicate that the patient is taking other medications;

(3) Data Not Present: there are no data regarding medications making it uncertain whether the patient is taking a lipid lowering medication.

18

AnalysisAvailability of Inclusion Criteria Data for

Colonoscopy?

Through EHR Data Method

Yes No

Through Hybrid Method

Yes a b

No c d

Where: Availability through the EHR = (a+c) / (a+b+c+d) = 92% And: Availability through Hybrid method = (a+b) / (a+b+c+d) = 98%

19

Challenges

• Logistical– HIE implementation– Data sharing (privacy/confidentiality)

• Analytic– Lack of a “gold standard”– Complex correlation among data sources– Identifying and interpreting “missing” data– Small sample sizes for some measures

20

Opportunities

• NQF-endorsed national standard measure set

• Measure set creates need for broad range of data types

• Direct comparison of HIE to two widely-used measurement methods

• “Deconstruction” of measure components identifies data problems that may affect future performance measures

21

“Crossing the Quality Chasm?”