a historical look at the udsmr® program evaluation · pdf file* weight assignment modeled...

© 2015 Uniform Data System for Medical Rehabilitation, a division of UB Foundation Activities, Inc. FIM, UDS Central, UDSMR, and the UDSMR logo are trademarks of Uniform Data System for Medical Rehabilitation, a division of UB Foundation Activities, Inc.

A Historical Look at the UDSMR® Program Evaluation Model

Troy Hillman, Manager of Analytical Services Group Sarah Mullin, MS, Data Analyst Uniform Data System for Medical Rehabilitation

Overview

• PEM background • PEM calculations • Historical outlook on PEM scores and percentiles • Historical outlook on top 10% • Possible improvements • How to interpret your PEM score • How to improve your PEM score

2

Background of the PEM

• Why do we need a performance evaluation model? • UDSMR’s subscribers requested one • Need to constantly demonstrate the value of

acute inpatient rehabilitation • Need to identify and learn from “high

performers” and/or “high improvers” • CMS initiated and publicly reports on

performance • IOM has recommended to Congress that all

healthcare providers be compensated on a value-based performance (P4P) basis • Comments sought from IRF providers in

2009 proposed rule 3


• Stipulations of the UDSMR® PEM: 1. Patterned after an accepted model

• HQID Premier Acute Hospital Initiative (sponsored by CMS)

2. Based on a composite measure • Indicator(s) of effectiveness, efficiency, and

quality (safety) 3. Easy to create and maintain

• No new data collection • Indicators captured routinely and accepted

by the field 4. Auditable (scoring patterns, etiologic diagnosis

coding, etc.) 4


• The result was a composite measure of effectiveness, efficiency, and quality that utilizes IRF-PAI data already collected as part of the IRF PPS process

• Indicators were created at both the case level and the facility level, and the model utilizes CMG-based expectations to account for variations in impairment distribution, patient severity, or both

• The UDSMR® PEM has been utilized in over 70% of all IRFs in the nation since 2007

5

Inclusion/Exclusion Criteria

• Inclusions: • Only PPS facilities • Only facilities that have at least thirty cases

during the year • Only facilities that have at least one case in

each quarter • Exclusions:

• Expired cases

6

Presenter

Presentation Notes

Typically 93% of IRFs in the UDSMR® database are used in the PEM each year.

How Is the PEM Calculated?

• Case-level indicators: • Discharge FIM® total

• Level of functional independence at discharge

• FIM® change • Functional improvement / reduction in

burden of care • LOS efficiency

• Rate of functional improvement over time • Facility-level indicators:

• % discharged to the community • % discharged to acute care

7

Case-Level Indicators

• Case-level indicators (discharge FIM® total, FIM® change, and LOS efficiency) are calculated at the patient level for each patient • Actual case performance is measured against

expected performance (impairment and severity-adjusted [CMG-adjusted] benchmark)

• Credit (1 point) is given if the patient’s actual score meets or exceeds the average score for cases within the patient’s CMG

8

Presenter

Presentation Notes

Truncation—Discharge FIM® Total and FIM® Change are truncated+1. LOS efficiency is rounded to the nearest 100th.


9

The facility actual column for each indicator identifies the number of discharged patients who met or exceeded the CMG benchmark

The facility target column for each indicator identifies the number of possible points that could be obtained and equals the number of patients discharged


10

The facility subscore column for each indicator indicates the percentage of patients who met or exceeded the CMG benchmark

The composite subscore is the sum of actual points divided by the sum of potential points (i.e., the percentage of all CMG targets achieved)

Facility-Level Indicators

• Facility-level indicators: • % discharged to the community • % discharged to acute care

• These indicators measure a facility’s performance against impairment- and severity-adjusted (CMG-adjusted) benchmarks

11


12

The facility actual value for each indicator is the actual percentage of patients discharged to each setting

The facility targets are the CMG-adjusted expected discharge rates


13

The facility subscore for: • % discharge to community is the actual percentage

divided by the CMG-adjusted expected percentage • % discharge to acute care is

100%−𝑎𝑎𝑎𝑎𝑎𝑎 %

100%−𝐶𝐶𝐶 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎 %∗ 100 =100−10.1

100−10= 0.999 ∗ 100 = 99.9%

Weighting for PEM Calculation

• Weighting* for final PEM score: • Facility subscores are weighted as follows for

the case-level and facility-level indicators: • 60% case-level indicator composite

• Discharge FIM® total, FIM® change, and LOS efficiency

• 30% discharge to community • 10% discharge to acute care

• Higher credit for lower D/C to acute care

• Lower credit (penalty) for higher D/C to acute care

* Weight assignment modeled after HQI Demonstration Project 14

15

Facility PEM Score: Sum of the weighted indicator subscores

How Is the PEM Score Calculated?

2014 PEM Score Distribution

16

How Is the Percentile Rank Calculated?

17

Facility Percentile Rank: • The facility-specific composite scores are then ranked from

lowest to highest, and each facility in the UDSMR® database is assigned a percentile rank from 1 to 100 relative to the other IRF subscribers in the database

• The percentile rank affords each facility a sense of its performance in comparison to those of the other IRFs that qualified for inclusion in the PEM that year

Basic Rules of Thumb for Interpreting Your PEM Report 1. Start with discharge to acute care %

• If your facility’s actual value is higher than the expected value, ask yourself these questions: • How has this affected your facility’s

discharge-to-community percentage? • How does this affect case-level indicators?

• Acute care discharges typically have lower-than-expected outcomes, and a higher-than-expected rate is bound to affect case-level indicators

18

Basic Rules of Thumb for Interpreting Your PEM Report 2. Look at discharge to community %

• If your facility’s actual value is less than the expected value, ask yourself: • How does this affect case-level indicators?

• Because discharges to other settings typically have lower-than-expected outcomes, a lower-than-expected rate is bound to affect case-level indicators

19

Basic Rules of Thumb for Interpreting Your PEM Report 3. Analyze case-level indicators

• Are any of the facility subscores (% of target) less than 50%? • This means that less than 50% of your cases

are meeting or exceeding their CMG-specific expectation for that variable

20

PEM Targets

• We are frequently asked how to identify the value(s) used for each of the targets utilized in the PEM

• For the 2015 PEM, UDSMR published these targets for subscribers in the Annual Program Evaluation Model (PEM) Reports section of the UDS Central™ website

21

PEM Scores vs. National Percentiles for Annual PEM Reports (2010–2014)

22

Year 0% 25% 50% 75% 100% 2010 44.2 65.9 71.3 76.7 95.9 2011 43.9 64.7 70.7 76.4 97.2 2012 44.8 64.6 70.7 76.4 96.6 2013 40.6 63.9 70.0 76.1 94.4

2014 36.1 62.4 68.7 76.6 93.2

What Does It Mean to Be in the Top 10%? • If your facility is in the top 10%, its percentile rank

is between 90 and 100 • UDSMR considers all facilities that rank in the top

10% of the database as top program performers, recognizing their delivery of quality patient care that is effective, efficient, timely, and patient-centered

23

Minimum PEM Score Required for Top 10% Designation

24

The PEM score required to achieve the top 10% has increased since 2007 and, in recent years, has begun to level off. This indicates that facilities as a whole have increased the quality of their outcomes over time.

80.2 80.2

80.8 80.8

81.6

82.1 81.7 81.8

79.0

79.5

80.0

80.5

81.0

81.5

82.0

82.5

2007 2008 2009 2010 2011 2012 2013 2014

Discharge FIM® Total Distribution: Top 10% PEM Facilities (2010–2014)

25

Dis

char

ge F

IM® S

ubsc

ore

FIM® Change Distribution: Top 10% PEM Facilities (2010–2014)

26

FIM

® C

hang

e Su

bsco

re

LOS Efficiency Distribution: Top 10% PEM Facilities (2010–2014)

27

LOS

Effic

ienc

y Su

bsco

re

28

Dis

char

ge to

Com

mun

ity S

ubsc

ore

D/C to Community Distribution: Top 10% PEM Facilities (2010–2014)

D/C to Acute Care Distribution: Top 10% PEM Facilities (2010–2014)

29

Dis

char

ge to

Acu

te S

ubsc

ore

Setting Realistic Goals

• Not every facility can be in the top 10%, but every facility can improve

• Improvement comes from setting goals appropriate for your facility

Possibility for Improvement:

30

Category 2011

(N=767) 2012

(N=761) 2013

(N=752) 2014

(N=752) % of facilities that improved 51.4% 48.2% 50.5% 50.5% % of facilities that improved their percentile rank by at least 10%

25.2% 22.7% 23.1% 21.1%

Maximum rank improvement 58 66 64 74 Maximum score improvement 17.1 18.3 22.2 20.6

How to Identify Opportunities for Improving Your PEM Score Step 1: Determine the accuracy of your CMGs • PEM expectations are based on CMGs • CMGs are based on IGCs and admission FIM®

ratings • Therefore, the validity of your comparison to

expectation is highly dependent on impairment group coding and FIM® rating accuracy

• Utilize the UDSMR® report set to identify any possible rating inaccuracies • Etiologic Diagnosis by Impairment Group Code

Listing • Scoring Report • Frequency of FIM® Ratings Report

31

How to Identify Opportunities for Improving Your PEM Score Step 2: Determine which indicator is lagging • For case-level indicators:

• Generate an on-demand report for the indicator • Can you identify RICs or CMGs whose

average value is less than the expected national average?

• Can you use the case listing to identify patients who may not have met or exceeded expectations?

• Can you identify characteristics of groups/patients that create less-than-expected outcomes?

32


• Generate an on-demand report for the indicator

33


• Generate an on-demand report for the indicator

34

How to Identify Opportunities for Improving Your PEM Score Step 2: Determine which indicator is lagging • For the case-level indicators discharge FIM® total

and FIM® change: • Generate the Scoring Report

• Can you identify FIM® items whose average values are different from the expected national averages?

35


and FIM® change: • Generate the Frequency of FIM® Ratings

Report • Can you identify patterns for FIM® items

where a certain level is rated more or less frequently than the national expectation?

36


and FIM® change: • Are you utilizing the Informatics tab for

management to national expectations?

37

Current Case FIM® Scores by RCMG

How to Identify Opportunities for Improving Your PEM Score Step 2: Determine which indicator is lagging • For the case-level indicator LOS efficiency:

• Determine whether less-than-expected results are created by: • FIM® change • LOS • Other factors

38

How to Identify Opportunities for Improving Your PEM Score Step 2: Determine which indicator is lagging • For facility-level indicators:

• % discharge to community: • What barriers prevented the patient from

being discharged to the community? • % discharge to acute care:

• What are the characteristics of patients discharged to acute care? • Are some of these instances

preventable?

39

How to Identify Opportunities for Improving Your PEM Score Step 2: Determine which indicator is lagging • Typical issues identified:

• Outcomes for acute care discharges, short stays, and early transfers • Are your patients likely to be discharged to

acute care or other settings prior to the completion of rehabilitation services?

• FIM® ratings • Are you using an interdisciplinary approach

and documenting all episodes? • Are you using the twenty-four-hour period

that yields the highest overall discharge FIM® total?

40

How to Identify Opportunities for Improving Your PEM Score Step 3: Improve your processes • Are specific policies and procedures affecting your

outcomes? • Pre-admission screening/admission practices • Interdisciplinary assessments • Safety guidelines • Discharge planning

41

How to Identify Opportunities for Improving Your PEM Score Step 4: Schedule a PEM consultation • If you are looking for an in-depth interpretation of

your PEM Report, we offer an hour-long PEM consultation • A clinician and a statistician from UDSMR will

conduct a detailed review of your facility’s reports, identifying areas for improvement

• The clinician and the statistician will conduct an hour-long conference call to share findings with your facility

• If you want more information or would like to schedule a consultation, contact Carole Stickels at 716-817-7815

42

Thank You! Any Questions?

• Troy Hillman: • [email protected] • 716-817-7869

• Sarah Mullin: • [email protected] • 716-817-7867

• Analytical Services Group: • [email protected] • 716-817-7870

43

a historical look at the udsmr® program evaluation · pdf file* weight assignment modeled...

Documents