1 research designs for evaluating disease management program effectiveness ariel linden, dr.p.h.,...

42
1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management Colloquium June 27-30, 2004

Upload: aiden-hawkins

Post on 26-Mar-2015

218 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

1

Research Designs For Evaluating Disease

Management Program Effectiveness

Ariel Linden, Dr.P.H., M.S.President, Linden Consulting Group

Disease Management ColloquiumJune 27-30, 2004

Page 2: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

2

What’s the Plan?

Discuss “threats to validity”

Provide methods to reduce those threats using currently-used evaluation designs

Offer additional designs that may be suitable alternatives or supplements to the current methods used to assess DM

program effectiveness

Page 3: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

3

VALIDITY

Selection Bias

Loss to Attrition

MaturationBenefit

Design

Unit Cost IncreasesRegression to the

MeanCase-mix

Treatment Interference

Access

New Technology

Hawthorne Effect

Reimbursement

Seasonality

Secular Trends

Measurement Error

Page 4: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

4

Selection Bias

Definition: Participants are not representative of the population from which they were drawn:

Motivation

Severity or acuteness of symptoms

Specifically targeted for enrollment

Page 5: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

5

Selection Bias (cont’)

Fix #1: Randomization

How: Distributes the “Observable” and “Unobservable” variation

equally between both groups

Limitations: costly, difficult to implement, intent to treat, not always possible

Page 6: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

6

Pretest-posttest Control Group:R O1 X O2

R O3 O4

Solomon 4-Group Design: R O X O

R O O

R X O

R O

Selection Bias (cont’)

Page 7: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

7

Selection Bias (cont’)

Fix #2: Standardized Rates

How: Direct/indirect adjustment enables comparisons over time or across populations by weighting frequency of events

Limitations: does not control for “unobservable” variation

Page 8: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

8

Age-adjusted Program Results

Age Group

Pre-Progra

m(rate/1000)

r X PProgram

(rate/1000)r X P

Proportion (P)of Population

20 – 29 7.3 0.9 10.2 1.2 0.1189

30 – 39 65.2 5.7 79.9 6.9 0.0868

40 – 49 190.8 13.4 173.6 12.2 0.0703

50 – 59 277.9 21.3 226.1 17.4 0.0768

60 - 69 408.4 25.2 287.8 17.7 0.0616

70 - 79 475.8 17.7 368.8 13.8 0.0373

80 + 422.2 8.4 356.0 7.0 0.0198Adjusted

rate 92.6 76.2

Page 9: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

9

Tenure-adjusted Program ResultsBaseline Group Compared to

inflation-adjusted…

2003 prevalent group’s 2003 claims

2003 prevalent group’s 2004 claims plus 2004 incident group assumed to have cost 2003 prevalent group’s claims in 2003

Baseline Group Compared to inflation-adjusted…

2002 prevalent group’s 2003 claims

2003 prevalent group’s 2004 claims

2003 Newly incident members actual claims, 2003

2004 Newly incident members actual claims, 2004

Page 10: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

10

Selection Bias (cont’)

Fix #3: Propensity Scoring

What?: Logistic regression score for likelihood of being in intervention

How: Controls for “Observable” variation

Limitations: does not control for “unobservable” variation

Page 11: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

11

1st Year CHF Program Results

Intervention

(N=94)

Control Group(N=4606)

P(T<=t)two-tail

Age 77.4 76.6 NS

% Female 0.51 0.56 NS

% Portland 0.17 0.69 p<0.0001

Pre-Hospitalization 1.13 0.5 p<0.0001

Pre-ED 0.7 0.4 p=0.003

Pre-Costs $18,287 $8,974 p<0.0001

Post-Hospitalization 0.59 0.87 p=0.008

Post-ED 0.57 0.58 NS

Post-Costs $11,874 $16,036 p=0.005

Page 12: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

12

1st Year CHF Program Results Admits

0.59

0.87

1.13

0.5

0

0.2

0.4

0.6

0.8

1

1.2

Pre-Hospital Admits Post-Hospital Admits

Hos

pita

liza

tion

Rat

e

Intervention Group (N=94) Control Group (N=4606)

Page 13: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

13

1st Year CHF Program Results

ER Visits

0.57

0.7

0.58

0.4

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Pre-ED Visits Post-ED Visits

ED

Vis

it R

ate

Intervention Group (N=94) Concurrent Control Group (N=4606)

Page 14: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

14

1st Year CHF Program ResultsCosts

$16,036

$11,874

$18,287

$8,974

$0$2,000$4,000$6,000$8,000

$10,000$12,000$14,000$16,000$18,000$20,000

Pre-Costs Post-Costs

Ave

rage

Cos

t

Intervention Group (N=94) Control Group (N=4606)

Page 15: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

15

1st Year CHF Program Results

Propensity Scoring MethodCases

(N=94)

Matched Controls(N=94)

P(T<=t)two-tail

Propensity Score 0.061 0.062 NS

Age 77.4 78.2 NS

% Female 0.51 0.51 NS

% Portland 0.17 0.17 NS

Pre-Hospitalization 1.13 1.09 NS

Pre-ED 0.70 0.67 NS

Pre-Costs $18,287 $17,001 NS

Post-Hospitalization 0.59 1.17 0.005

Post-ED 0.57 0.77 0.026

Post-Costs $11,874 $24,085 0.003

Page 16: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

16

1st Year CHF Program Results

Propensity Scoring Method - Admits

0.59

1.171.131.09

0

0.2

0.4

0.6

0.8

1

1.2

1.4

Pre-Hospital Admits Post-Hospital Admits

Hos

pita

liza

tion

Rat

e

cases (N=94) matched controls (N=94)

Page 17: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

17

1st Year CHF Program Results

Propensity Scoring Method – ED Visits

0.57

0.70.77

0.67

00.10.20.30.40.50.60.70.80.9

Pre-ED Visits Post-ED Visits

ED

Vis

it R

ate

cases (N=94) matched controls (N=94)

Page 18: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

18

1st Year CHF Program Results

Propensity Scoring Method – Costs

$24,085

$11,874

$18,287$17,001

$0

$5,000

$10,000

$15,000

$20,000

$25,000

$30,000

Pre-Costs Post-Costs

Ave

rage

Cos

t

cases (N=94) matched controls (N=94)

Page 19: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

19

Regression to the Mean

Definition: After the first of two related measurements has been made, the second is expected to be closer to the mean than the first.

Page 20: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

20

Regression to the MeanCAD

First quintile

35%

Second quintile

19%

Third quintile

13%

Fourth quintile

16%

Fifth quintile

17%

First quintile

11% Second quintile

18%

Third quintile

21%

Fourth quintile

23%

Fifth quintile

27%

Where the 1st Quintile (N=749) Went In Year 2 Where the 5th Quintile (N=748) Went In Year 2

Page 21: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

21

Regression to the MeanCHF

First quintile

35%

Second quintile

18%

Third quintile

15%

Fourth quintile

17%

Fifth quintile

15%

First quintile

7%Second quintile

14%

Third quintile

19%Fourth quintile

23%

Fifth quintile

37%

Where the 1st Quintile (N=523) Went In Year 2 Where the 5th Quintile (N=537) Went In Year 2

Page 22: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

22

Regression to the Mean (cont’)

Fix #1: Increase length of measurement periods

How: Controls for movement toward the mean across periods

Limitations: periods may not be long enough, availability of historic data

Page 23: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

23

Regression to the Mean (cont’)

Currently-Used Method

BaselineMeasurement

Year

1st ContractMeasurement

Year

2nd ContractMeasurement

Year

Compare to baseline

Compare to baseline

Claims run-out periods

Page 24: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

24

Regression to the Mean (cont’)

Valid Method (from Lewis presentation)

Historic 2-YearPeriod

BaselineMeasurement

Year

1st ContractMeasurement

Year

2 year “freeze” period + measurement

2 year “freeze” period + measurement

Page 25: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

25

Regression to the Mean (cont’)

Fix #2: Time Series Analysis

How: Controls for movement across many periods (preferably > 50 observations)

Limitations: availability of historic data, change in collection methods

Page 26: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

26

Measurement Error

Definition: Measurements of the same quantity on the same group of subjects will not always elicit the same results. This may be because of natural variation in the subject (or group), variation in the measurement process, or both (random vs. systematic error).

Page 27: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

27

Measurement Error (cont’)

Fix #1: Use all suitables in the analysis (to adjust for the “zeroes”)

Fix #2: Use identical data methods pre and post (like unit claims-to-claims comparison)

Fix #3: Use utilization and quality measures instead of cost.

Page 28: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

28

Alternative Designs

Survival Analysis

Time Series Analysis

Time-dependent Regression

Page 29: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

29

Survival Analysis

Features:

Time to event analysis – longitudinal

Censoring

Allows for varying enrollment points

Page 30: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

30

Survival Analysis

Commencement ofDM Intervention

Improvement inclinical markers

First PatientContact

Reduction in Utilizationand Costs

??

??

??

Page 31: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

31

Survival Analysis

Time (Months)

3020100

Pro

ba

bili

ty o

f Ho

spita

liza

tion

.6

.5

.4

.3

.2

.1

0.0

PRA Value

High Risk

High-censored

Low Risk

Low-censored

Page 32: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

32

Time Series Analysis

Features:

Longitudinal analysis

Serial Dependency (autocorrelation)

Does not require explanatory variables

Controls for trend and seasonality

Can be used for forecasting

Page 33: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

33

Time Series Analysis (cont’)

0

10

20

30

40

50

60

Month

Adm

its P

TM

PY

Admits (Actual)

SES

DES

ARIMA (1,0,0)

Historical PeriodBaseline Period

SES (15.5%)DES (19.7%)

ARIMA (16.0%)

1st Measurement Period

SES (3.6%)

Page 34: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

34

Time-dependent Regression

Combines important elements of other models to create a new method, including variables such as:

Program tenure (censuring)

Seasonality (important for Medicare)

Can be used for forecasting

Page 35: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

35

0

50

100

150

200

250

300

350

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Adm

its/1

000

Admits/1000 (Tenure) Admits/1000 (Month)

Simulated hospital admissions per thousand members based on program tenure and month-of-year (months 1-12 represent Jan – Dec of program year 1, and months 13-24 represent Jan – Dec of program year 2).

Page 36: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

36

Conclusions

Identify potential threats to validity before determining evaluation method

Choose outcome variables that mitigate measurement bias (e.g. all identified members vs those with costs)

There is no panacea! Use more than one design to validate results.

Page 37: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

37

How does this presentation differ from what you just saw?

• Lewis approach is the only valid pre-post population-based design in use today

• But valid = accurate. “Valid” just means adjustment for systematic error

• These methods reduce chances of non-systematic error to increase accuracy

Page 38: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

38

1. Linden A, Adams J, Roberts N. An assessment of the total population approach for evaluating disease management program effectiveness. Disease Management 2003;6(2): 93-102.

2. Linden A, Adams J, Roberts N. Using propensity scores to construct comparable control groups for disease management program evaluation. Disease Management and Health Outcomes Journal (in print).

3. Linden A, Adams J, Roberts N. Evaluating disease management program effectiveness: An introduction to time series analysis. Disease Management 2003;6(4):243-255.

4. Linden A, Adams J, Roberts N. Evaluating disease management program effectiveness: An introduction to survival analysis. Disease Management 2004;7(2):XX-XX.

References (1)

Page 39: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

39

5. Linden A, Adams J, Roberts N. Evaluation methods in disease management: determining program effectiveness. Position Paper for the Disease Management Association of America (DMAA). October 2003.

6. Linden A, Adams J, Roberts N. Using an empirical method for establishing clinical outcome targets in disease management programs. Disease Management. 2004;7(2):93-101.

7. Linden A, Roberts N. Disease management interventions: What’s in the black box? Disease Management. 2004;7(4):XX-XX.

8. Linden A, Adams J, Roberts N. Evaluating disease management program effectiveness: An introduction to the bootstrap technique. Disease Management and Health Outcomes Journal (under review).

References (2)

Page 40: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

40

9. Linden A, Adams J, Roberts N. Generalizability of disease management program results: getting from here to there. Managed Care Interface 2004;(July):38-45.

10. Linden A, Roberts N, Keck K. The complete “how to” guide for selecting

a disease management vendor. Disease Management. 2003;6(1):21-26.

11. Linden A, Adams J, Roberts N. Evaluating disease management program effectiveness adjusting for enrollment (tenure) and seasonality. Research in Healthcare Financial Management. 2004;9(1): XX-XX.

12. Linden A, Adams J, Roberts N. Strengthening the case for disease management effectiveness: unhiding the hidden bias. J Clin Outcomes Manage (under review).

References (3)

Page 41: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

41

Software for DM Analyses

The analyses in this presentation used XLStat for Excel. This is an Excel add-in, similar to the data analysis package that comes built-in to the program.

Therefore, users familiar with Excel will find this program easy to use without much instruction.

Page 42: 1 Research Designs For Evaluating Disease Management Program Effectiveness Ariel Linden, Dr.P.H., M.S. President, Linden Consulting Group Disease Management

42

Questions?

Ariel Linden, DrPH, [email protected]