1 research designs for evaluating disease management program effectiveness ariel linden, dr.p.h.,...

1

Research Designs For Evaluating Disease

Management Program Effectiveness

Ariel Linden, Dr.P.H., M.S.President, Linden Consulting Group

Disease Management ColloquiumJune 27-30, 2004

2

What’s the Plan?

Discuss “threats to validity”

Provide methods to reduce those threats using currently-used evaluation designs

Offer additional designs that may be suitable alternatives or supplements to the current methods used to assess DM

program effectiveness

3

VALIDITY

Selection Bias

Loss to Attrition

MaturationBenefit

Design

Unit Cost IncreasesRegression to the

MeanCase-mix

Treatment Interference

Access

New Technology

Hawthorne Effect

Reimbursement

Seasonality

Secular Trends

Measurement Error

4

Selection Bias

Definition: Participants are not representative of the population from which they were drawn:

Motivation

Severity or acuteness of symptoms

Specifically targeted for enrollment

5

Selection Bias (cont’)

Fix #1: Randomization

How: Distributes the “Observable” and “Unobservable” variation

equally between both groups

Limitations: costly, difficult to implement, intent to treat, not always possible

6

Pretest-posttest Control Group:R O1 X O2

R O3 O4

Solomon 4-Group Design: R O X O

R O O

R X O

R O

Selection Bias (cont’)

7

Selection Bias (cont’)

Fix #2: Standardized Rates

How: Direct/indirect adjustment enables comparisons over time or across populations by weighting frequency of events

Limitations: does not control for “unobservable” variation

8

Age-adjusted Program Results

Age Group

Pre-Progra

m(rate/1000)

r X PProgram

(rate/1000)r X P

Proportion (P)of Population

20 – 29 7.3 0.9 10.2 1.2 0.1189

30 – 39 65.2 5.7 79.9 6.9 0.0868

40 – 49 190.8 13.4 173.6 12.2 0.0703

50 – 59 277.9 21.3 226.1 17.4 0.0768

60 - 69 408.4 25.2 287.8 17.7 0.0616

70 - 79 475.8 17.7 368.8 13.8 0.0373

80 + 422.2 8.4 356.0 7.0 0.0198Adjusted

rate 92.6 76.2

9

Tenure-adjusted Program ResultsBaseline Group Compared to

inflation-adjusted…

2003 prevalent group’s 2003 claims

2003 prevalent group’s 2004 claims plus 2004 incident group assumed to have cost 2003 prevalent group’s claims in 2003

Baseline Group Compared to inflation-adjusted…

2002 prevalent group’s 2003 claims

2003 prevalent group’s 2004 claims

2003 Newly incident members actual claims, 2003

2004 Newly incident members actual claims, 2004

10

Selection Bias (cont’)

Fix #3: Propensity Scoring

What?: Logistic regression score for likelihood of being in intervention

How: Controls for “Observable” variation

Limitations: does not control for “unobservable” variation

11

1st Year CHF Program Results

Intervention

(N=94)

Control Group(N=4606)

P(T<=t)two-tail

Age 77.4 76.6 NS

% Female 0.51 0.56 NS

% Portland 0.17 0.69 p<0.0001

Pre-Hospitalization 1.13 0.5 p<0.0001

Pre-ED 0.7 0.4 p=0.003

Pre-Costs $18,287 $8,974 p<0.0001

Post-Hospitalization 0.59 0.87 p=0.008

Post-ED 0.57 0.58 NS

Post-Costs $11,874 $16,036 p=0.005

12

1st Year CHF Program Results Admits

0.59

0.87

1.13

0.5

0

0.2

0.4

0.6

0.8

1

1.2

Pre-Hospital Admits Post-Hospital Admits

Hos

pita

liza

tion

Rat

e

Intervention Group (N=94) Control Group (N=4606)

13

1st Year CHF Program Results

ER Visits

0.57

0.7

0.58

0.4

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Pre-ED Visits Post-ED Visits

ED

Vis

it R

ate

Intervention Group (N=94) Concurrent Control Group (N=4606)

14

1st Year CHF Program ResultsCosts

$16,036

$11,874

$18,287

$8,974

$0$2,000$4,000$6,000$8,000

$10,000$12,000$14,000$16,000$18,000$20,000

Pre-Costs Post-Costs

Ave

rage

Cos

t

Intervention Group (N=94) Control Group (N=4606)

15

1st Year CHF Program Results

Propensity Scoring MethodCases

(N=94)

Matched Controls(N=94)

P(T<=t)two-tail

Propensity Score 0.061 0.062 NS

Age 77.4 78.2 NS

% Female 0.51 0.51 NS

% Portland 0.17 0.17 NS

Pre-Hospitalization 1.13 1.09 NS

Pre-ED 0.70 0.67 NS

Pre-Costs $18,287 $17,001 NS

Post-Hospitalization 0.59 1.17 0.005

Post-ED 0.57 0.77 0.026

Post-Costs $11,874 $24,085 0.003

16

1st Year CHF Program Results

Propensity Scoring Method - Admits

0.59

1.171.131.09

0

0.2

0.4

0.6

0.8

1

1.2

1.4

Pre-Hospital Admits Post-Hospital Admits

Hos

pita

liza

tion

Rat

e

cases (N=94) matched controls (N=94)

17

1st Year CHF Program Results

Propensity Scoring Method – ED Visits

0.57

0.70.77

0.67

00.10.20.30.40.50.60.70.80.9

Pre-ED Visits Post-ED Visits

ED

Vis

it R

ate

cases (N=94) matched controls (N=94)

18

1st Year CHF Program Results

Propensity Scoring Method – Costs

$24,085

$11,874

$18,287$17,001

$0

$5,000

$10,000

$15,000

$20,000

$25,000

$30,000

Pre-Costs Post-Costs

Ave

rage

Cos

t

cases (N=94) matched controls (N=94)

19

Regression to the Mean

Definition: After the first of two related measurements has been made, the second is expected to be closer to the mean than the first.

20

Regression to the MeanCAD

First quintile

35%

Second quintile

19%

Third quintile

13%

Fourth quintile

16%

Fifth quintile

17%

First quintile

11% Second quintile

18%

Third quintile

21%

Fourth quintile

23%

Fifth quintile

27%

Where the 1st Quintile (N=749) Went In Year 2 Where the 5th Quintile (N=748) Went In Year 2

21

Regression to the MeanCHF

First quintile

35%

Second quintile

18%

Third quintile

15%

Fourth quintile

17%

Fifth quintile

15%

First quintile

7%Second quintile

14%

Third quintile

19%Fourth quintile

23%

Fifth quintile

37%

Where the 1st Quintile (N=523) Went In Year 2 Where the 5th Quintile (N=537) Went In Year 2

22

Regression to the Mean (cont’)

Fix #1: Increase length of measurement periods

How: Controls for movement toward the mean across periods

Limitations: periods may not be long enough, availability of historic data

23

Regression to the Mean (cont’)

Currently-Used Method

BaselineMeasurement

Year

1st ContractMeasurement

Year

2nd ContractMeasurement

Year

Compare to baseline

Compare to baseline

Claims run-out periods

24

Regression to the Mean (cont’)

Valid Method (from Lewis presentation)

Historic 2-YearPeriod

BaselineMeasurement

Year

1st ContractMeasurement

Year

2 year “freeze” period + measurement

2 year “freeze” period + measurement

25

Regression to the Mean (cont’)

Fix #2: Time Series Analysis

How: Controls for movement across many periods (preferably > 50 observations)

Limitations: availability of historic data, change in collection methods

26

Measurement Error

Definition: Measurements of the same quantity on the same group of subjects will not always elicit the same results. This may be because of natural variation in the subject (or group), variation in the measurement process, or both (random vs. systematic error).

27

Measurement Error (cont’)

Fix #1: Use all suitables in the analysis (to adjust for the “zeroes”)

Fix #2: Use identical data methods pre and post (like unit claims-to-claims comparison)

Fix #3: Use utilization and quality measures instead of cost.

28

Alternative Designs

Survival Analysis

Time Series Analysis

Time-dependent Regression

29

Survival Analysis

Features:

Time to event analysis – longitudinal

Censoring

Allows for varying enrollment points

30

Survival Analysis

Commencement ofDM Intervention

Improvement inclinical markers

First PatientContact

Reduction in Utilizationand Costs

??

??

??

31

Survival Analysis

Time (Months)

3020100

Pro

ba

bili

ty o

f Ho

spita

liza

tion

.6

.5

.4

.3

.2

.1

0.0

PRA Value

High Risk

High-censored

Low Risk

Low-censored

32

Time Series Analysis

Features:

Longitudinal analysis

Serial Dependency (autocorrelation)

Does not require explanatory variables

Controls for trend and seasonality

Can be used for forecasting

33

Time Series Analysis (cont’)

0

10

20

30

40

50

60

Month

Adm

its P

TM

PY

Admits (Actual)

SES

DES

ARIMA (1,0,0)

Historical PeriodBaseline Period

SES (15.5%)DES (19.7%)

ARIMA (16.0%)

1st Measurement Period

SES (3.6%)

34

Time-dependent Regression

Combines important elements of other models to create a new method, including variables such as:

Program tenure (censuring)

Seasonality (important for Medicare)

Can be used for forecasting

35

0

50

100

150

200

250

300

350

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Adm

its/1

000

Admits/1000 (Tenure) Admits/1000 (Month)

Simulated hospital admissions per thousand members based on program tenure and month-of-year (months 1-12 represent Jan – Dec of program year 1, and months 13-24 represent Jan – Dec of program year 2).

36

Conclusions

Identify potential threats to validity before determining evaluation method

Choose outcome variables that mitigate measurement bias (e.g. all identified members vs those with costs)

There is no panacea! Use more than one design to validate results.

37

How does this presentation differ from what you just saw?

• Lewis approach is the only valid pre-post population-based design in use today

• But valid = accurate. “Valid” just means adjustment for systematic error

• These methods reduce chances of non-systematic error to increase accuracy

38

1. Linden A, Adams J, Roberts N. An assessment of the total population approach for evaluating disease management program effectiveness. Disease Management 2003;6(2): 93-102.

2. Linden A, Adams J, Roberts N. Using propensity scores to construct comparable control groups for disease management program evaluation. Disease Management and Health Outcomes Journal (in print).

3. Linden A, Adams J, Roberts N. Evaluating disease management program effectiveness: An introduction to time series analysis. Disease Management 2003;6(4):243-255.

4. Linden A, Adams J, Roberts N. Evaluating disease management program effectiveness: An introduction to survival analysis. Disease Management 2004;7(2):XX-XX.

References (1)

39

5. Linden A, Adams J, Roberts N. Evaluation methods in disease management: determining program effectiveness. Position Paper for the Disease Management Association of America (DMAA). October 2003.

6. Linden A, Adams J, Roberts N. Using an empirical method for establishing clinical outcome targets in disease management programs. Disease Management. 2004;7(2):93-101.

7. Linden A, Roberts N. Disease management interventions: What’s in the black box? Disease Management. 2004;7(4):XX-XX.

8. Linden A, Adams J, Roberts N. Evaluating disease management program effectiveness: An introduction to the bootstrap technique. Disease Management and Health Outcomes Journal (under review).

References (2)

40

9. Linden A, Adams J, Roberts N. Generalizability of disease management program results: getting from here to there. Managed Care Interface 2004;(July):38-45.

10. Linden A, Roberts N, Keck K. The complete “how to” guide for selecting

a disease management vendor. Disease Management. 2003;6(1):21-26.

11. Linden A, Adams J, Roberts N. Evaluating disease management program effectiveness adjusting for enrollment (tenure) and seasonality. Research in Healthcare Financial Management. 2004;9(1): XX-XX.

12. Linden A, Adams J, Roberts N. Strengthening the case for disease management effectiveness: unhiding the hidden bias. J Clin Outcomes Manage (under review).

References (3)

41

Software for DM Analyses

The analyses in this presentation used XLStat for Excel. This is an Excel add-in, similar to the data analysis package that comes built-in to the program.

Therefore, users familiar with Excel will find this program easy to use without much instruction.

42

Questions?

Ariel Linden, DrPH, MSariellinden@yahoo.com