1 research designs for evaluating disease management program effectiveness ariel linden, dr.p.h.,...
TRANSCRIPT
1
Research Designs For Evaluating Disease
Management Program Effectiveness
Ariel Linden, Dr.P.H., M.S.President, Linden Consulting Group
Disease Management ColloquiumJune 27-30, 2004
2
What’s the Plan?
Discuss “threats to validity”
Provide methods to reduce those threats using currently-used evaluation designs
Offer additional designs that may be suitable alternatives or supplements to the current methods used to assess DM
program effectiveness
3
VALIDITY
Selection Bias
Loss to Attrition
MaturationBenefit
Design
Unit Cost IncreasesRegression to the
MeanCase-mix
Treatment Interference
Access
New Technology
Hawthorne Effect
Reimbursement
Seasonality
Secular Trends
Measurement Error
4
Selection Bias
Definition: Participants are not representative of the population from which they were drawn:
Motivation
Severity or acuteness of symptoms
Specifically targeted for enrollment
5
Selection Bias (cont’)
Fix #1: Randomization
How: Distributes the “Observable” and “Unobservable” variation
equally between both groups
Limitations: costly, difficult to implement, intent to treat, not always possible
6
Pretest-posttest Control Group:R O1 X O2
R O3 O4
Solomon 4-Group Design: R O X O
R O O
R X O
R O
Selection Bias (cont’)
7
Selection Bias (cont’)
Fix #2: Standardized Rates
How: Direct/indirect adjustment enables comparisons over time or across populations by weighting frequency of events
Limitations: does not control for “unobservable” variation
8
Age-adjusted Program Results
Age Group
Pre-Progra
m(rate/1000)
r X PProgram
(rate/1000)r X P
Proportion (P)of Population
20 – 29 7.3 0.9 10.2 1.2 0.1189
30 – 39 65.2 5.7 79.9 6.9 0.0868
40 – 49 190.8 13.4 173.6 12.2 0.0703
50 – 59 277.9 21.3 226.1 17.4 0.0768
60 - 69 408.4 25.2 287.8 17.7 0.0616
70 - 79 475.8 17.7 368.8 13.8 0.0373
80 + 422.2 8.4 356.0 7.0 0.0198Adjusted
rate 92.6 76.2
9
Tenure-adjusted Program ResultsBaseline Group Compared to
inflation-adjusted…
2003 prevalent group’s 2003 claims
2003 prevalent group’s 2004 claims plus 2004 incident group assumed to have cost 2003 prevalent group’s claims in 2003
Baseline Group Compared to inflation-adjusted…
2002 prevalent group’s 2003 claims
2003 prevalent group’s 2004 claims
2003 Newly incident members actual claims, 2003
2004 Newly incident members actual claims, 2004
10
Selection Bias (cont’)
Fix #3: Propensity Scoring
What?: Logistic regression score for likelihood of being in intervention
How: Controls for “Observable” variation
Limitations: does not control for “unobservable” variation
11
1st Year CHF Program Results
Intervention
(N=94)
Control Group(N=4606)
P(T<=t)two-tail
Age 77.4 76.6 NS
% Female 0.51 0.56 NS
% Portland 0.17 0.69 p<0.0001
Pre-Hospitalization 1.13 0.5 p<0.0001
Pre-ED 0.7 0.4 p=0.003
Pre-Costs $18,287 $8,974 p<0.0001
Post-Hospitalization 0.59 0.87 p=0.008
Post-ED 0.57 0.58 NS
Post-Costs $11,874 $16,036 p=0.005
12
1st Year CHF Program Results Admits
0.59
0.87
1.13
0.5
0
0.2
0.4
0.6
0.8
1
1.2
Pre-Hospital Admits Post-Hospital Admits
Hos
pita
liza
tion
Rat
e
Intervention Group (N=94) Control Group (N=4606)
13
1st Year CHF Program Results
ER Visits
0.57
0.7
0.58
0.4
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Pre-ED Visits Post-ED Visits
ED
Vis
it R
ate
Intervention Group (N=94) Concurrent Control Group (N=4606)
14
1st Year CHF Program ResultsCosts
$16,036
$11,874
$18,287
$8,974
$0$2,000$4,000$6,000$8,000
$10,000$12,000$14,000$16,000$18,000$20,000
Pre-Costs Post-Costs
Ave
rage
Cos
t
Intervention Group (N=94) Control Group (N=4606)
15
1st Year CHF Program Results
Propensity Scoring MethodCases
(N=94)
Matched Controls(N=94)
P(T<=t)two-tail
Propensity Score 0.061 0.062 NS
Age 77.4 78.2 NS
% Female 0.51 0.51 NS
% Portland 0.17 0.17 NS
Pre-Hospitalization 1.13 1.09 NS
Pre-ED 0.70 0.67 NS
Pre-Costs $18,287 $17,001 NS
Post-Hospitalization 0.59 1.17 0.005
Post-ED 0.57 0.77 0.026
Post-Costs $11,874 $24,085 0.003
16
1st Year CHF Program Results
Propensity Scoring Method - Admits
0.59
1.171.131.09
0
0.2
0.4
0.6
0.8
1
1.2
1.4
Pre-Hospital Admits Post-Hospital Admits
Hos
pita
liza
tion
Rat
e
cases (N=94) matched controls (N=94)
17
1st Year CHF Program Results
Propensity Scoring Method – ED Visits
0.57
0.70.77
0.67
00.10.20.30.40.50.60.70.80.9
Pre-ED Visits Post-ED Visits
ED
Vis
it R
ate
cases (N=94) matched controls (N=94)
18
1st Year CHF Program Results
Propensity Scoring Method – Costs
$24,085
$11,874
$18,287$17,001
$0
$5,000
$10,000
$15,000
$20,000
$25,000
$30,000
Pre-Costs Post-Costs
Ave
rage
Cos
t
cases (N=94) matched controls (N=94)
19
Regression to the Mean
Definition: After the first of two related measurements has been made, the second is expected to be closer to the mean than the first.
20
Regression to the MeanCAD
First quintile
35%
Second quintile
19%
Third quintile
13%
Fourth quintile
16%
Fifth quintile
17%
First quintile
11% Second quintile
18%
Third quintile
21%
Fourth quintile
23%
Fifth quintile
27%
Where the 1st Quintile (N=749) Went In Year 2 Where the 5th Quintile (N=748) Went In Year 2
21
Regression to the MeanCHF
First quintile
35%
Second quintile
18%
Third quintile
15%
Fourth quintile
17%
Fifth quintile
15%
First quintile
7%Second quintile
14%
Third quintile
19%Fourth quintile
23%
Fifth quintile
37%
Where the 1st Quintile (N=523) Went In Year 2 Where the 5th Quintile (N=537) Went In Year 2
22
Regression to the Mean (cont’)
Fix #1: Increase length of measurement periods
How: Controls for movement toward the mean across periods
Limitations: periods may not be long enough, availability of historic data
23
Regression to the Mean (cont’)
Currently-Used Method
BaselineMeasurement
Year
1st ContractMeasurement
Year
2nd ContractMeasurement
Year
Compare to baseline
Compare to baseline
Claims run-out periods
24
Regression to the Mean (cont’)
Valid Method (from Lewis presentation)
Historic 2-YearPeriod
BaselineMeasurement
Year
1st ContractMeasurement
Year
2 year “freeze” period + measurement
2 year “freeze” period + measurement
25
Regression to the Mean (cont’)
Fix #2: Time Series Analysis
How: Controls for movement across many periods (preferably > 50 observations)
Limitations: availability of historic data, change in collection methods
26
Measurement Error
Definition: Measurements of the same quantity on the same group of subjects will not always elicit the same results. This may be because of natural variation in the subject (or group), variation in the measurement process, or both (random vs. systematic error).
27
Measurement Error (cont’)
Fix #1: Use all suitables in the analysis (to adjust for the “zeroes”)
Fix #2: Use identical data methods pre and post (like unit claims-to-claims comparison)
Fix #3: Use utilization and quality measures instead of cost.
28
Alternative Designs
Survival Analysis
Time Series Analysis
Time-dependent Regression
29
Survival Analysis
Features:
Time to event analysis – longitudinal
Censoring
Allows for varying enrollment points
30
Survival Analysis
Commencement ofDM Intervention
Improvement inclinical markers
First PatientContact
Reduction in Utilizationand Costs
??
??
??
31
Survival Analysis
Time (Months)
3020100
Pro
ba
bili
ty o
f Ho
spita
liza
tion
.6
.5
.4
.3
.2
.1
0.0
PRA Value
High Risk
High-censored
Low Risk
Low-censored
32
Time Series Analysis
Features:
Longitudinal analysis
Serial Dependency (autocorrelation)
Does not require explanatory variables
Controls for trend and seasonality
Can be used for forecasting
33
Time Series Analysis (cont’)
0
10
20
30
40
50
60
Month
Adm
its P
TM
PY
Admits (Actual)
SES
DES
ARIMA (1,0,0)
Historical PeriodBaseline Period
SES (15.5%)DES (19.7%)
ARIMA (16.0%)
1st Measurement Period
SES (3.6%)
34
Time-dependent Regression
Combines important elements of other models to create a new method, including variables such as:
Program tenure (censuring)
Seasonality (important for Medicare)
Can be used for forecasting
35
0
50
100
150
200
250
300
350
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Adm
its/1
000
Admits/1000 (Tenure) Admits/1000 (Month)
Simulated hospital admissions per thousand members based on program tenure and month-of-year (months 1-12 represent Jan – Dec of program year 1, and months 13-24 represent Jan – Dec of program year 2).
36
Conclusions
Identify potential threats to validity before determining evaluation method
Choose outcome variables that mitigate measurement bias (e.g. all identified members vs those with costs)
There is no panacea! Use more than one design to validate results.
37
How does this presentation differ from what you just saw?
• Lewis approach is the only valid pre-post population-based design in use today
• But valid = accurate. “Valid” just means adjustment for systematic error
• These methods reduce chances of non-systematic error to increase accuracy
38
1. Linden A, Adams J, Roberts N. An assessment of the total population approach for evaluating disease management program effectiveness. Disease Management 2003;6(2): 93-102.
2. Linden A, Adams J, Roberts N. Using propensity scores to construct comparable control groups for disease management program evaluation. Disease Management and Health Outcomes Journal (in print).
3. Linden A, Adams J, Roberts N. Evaluating disease management program effectiveness: An introduction to time series analysis. Disease Management 2003;6(4):243-255.
4. Linden A, Adams J, Roberts N. Evaluating disease management program effectiveness: An introduction to survival analysis. Disease Management 2004;7(2):XX-XX.
References (1)
39
5. Linden A, Adams J, Roberts N. Evaluation methods in disease management: determining program effectiveness. Position Paper for the Disease Management Association of America (DMAA). October 2003.
6. Linden A, Adams J, Roberts N. Using an empirical method for establishing clinical outcome targets in disease management programs. Disease Management. 2004;7(2):93-101.
7. Linden A, Roberts N. Disease management interventions: What’s in the black box? Disease Management. 2004;7(4):XX-XX.
8. Linden A, Adams J, Roberts N. Evaluating disease management program effectiveness: An introduction to the bootstrap technique. Disease Management and Health Outcomes Journal (under review).
References (2)
40
9. Linden A, Adams J, Roberts N. Generalizability of disease management program results: getting from here to there. Managed Care Interface 2004;(July):38-45.
10. Linden A, Roberts N, Keck K. The complete “how to” guide for selecting
a disease management vendor. Disease Management. 2003;6(1):21-26.
11. Linden A, Adams J, Roberts N. Evaluating disease management program effectiveness adjusting for enrollment (tenure) and seasonality. Research in Healthcare Financial Management. 2004;9(1): XX-XX.
12. Linden A, Adams J, Roberts N. Strengthening the case for disease management effectiveness: unhiding the hidden bias. J Clin Outcomes Manage (under review).
References (3)
41
Software for DM Analyses
The analyses in this presentation used XLStat for Excel. This is an Excel add-in, similar to the data analysis package that comes built-in to the program.
Therefore, users familiar with Excel will find this program easy to use without much instruction.