Validation / citations

Validation

Expert review of model structure Expert review of basic code implementation Reproduce original inputs Correctly respond to changes in parameters Consistent with other analyses Consistent across versions

Documentation specifications

Validation requires adequate documentation to… Describe clinical model Describe programming algorithms Describe inputs (sources / values /

data quality) Describe validation strategy

Citations

Basic rationale and structure -- Matchar et al., Annals of Internal Medicine (1997)

Bootstrapped sensitivity analysis -- Parmigiani et al., Medical Decision Making (1997)

Application to acute stroke trial -- Samsa et al., Journal of Clinical Epidemiology (in press)

Outline

Rationale for modeling Stroke model described Applying the SPM to a

randomized trial (*) Extensions

Applying the SPM to an RCT

Basic idea: After the short follow-up period typical of an RCT, we are likely to observe that the intervention induces small shifts in the distribution of morality / disability (measured at the conclusion of follow-up). Modeling can be used to account for the

effects of these shifts on long-term outcomes.

Disability measure

Disability could be measured by the Rankin Scale (RS), Barthel Index (BI), or a number of other instruments.

For concreteness, we use the RS with 5 categories. The analysis easily extends to other scales and/or cut-points.

Risk (hazard) ratios for disability on subsequent outcomes*

RS Utility h(IS) h(MI) h(DT) Cost 0-1 .80 1.00 1.00 1.00 1.00 2 .65 1.05 1.07 1.11 1.27 3 .50 1.12 1.14 1.27 1.94 4 .35 1.24 1.26 1.71 3.98 5 .20 1.30 1.31 2.37 6.01

*As derived by an expert panel, supported by published and unpublished literature

Rationale The utilities for RS of 0-1 and 5 were obtained

from the PORT’s patient survey.

• Other values obtained by interpolation• Consistent with unpublished RCT data

Increased hazard of death results from sequelae of inactivity such as aspiration pneumonia, etc.

Daily costs greatly increase when disability level implies high likelihood of institutionalization.

SPM results by Rankin*

RS Survival QALY Cost 0-1 10.60 6.07 143,820 2 9.99 4.70 167,602 3 9.28 3.37 217,039 4 8.22 2.13 328,895 5 7.09 1.09 403,911 *Beginning 6 months after IS, for patients

with mean age 70 years

Thought experiment -- typical RCT results (6 month follow-up)

Intervention Placebo QALY 0.25 0.23 Cost 27,000 24,000

That is, little difference in QALY in absolute terms. For cost, we assume equivalent utilization plus the cost of the drug.

Typical RCT results (cont’d)

RS Intervention Placebo 0-131% 25% 221% 15% 311% 10% 46% 10% 511% 15% Died20% 25%

Typical results (cont’d)

The previous results describe a “break-through drug”, but the ICER based upon 6 months of follow-up is only (27,000-24,000) / (0.25-0.23) = 150,000 $/QALY.

Combining short- and long-term outcomes

For each patient, we assign long-term outcomes (cost and QALYs) based upon RS at 6 months. For example, total costs then become 6-month costs (observed) + expected long-term costs (simulated).

Long-term QALYs by group

Intervention: (.31)(6.07) + (.21)(4.70) + (.11)(3.37) + (.06)(2.13) + (.11)(1.09) + (.20)(0) = 3.49 QALY

Placebo: (.25)(6.07) + (.15)(4.70) + (.10)(3.37) + (.10)(2.13) + (.15)(1.09) + (.25)(0) = 2.94 QALY– Identical multipliers -- groups only differ in

proportion of patients in each RS category– Same weighted average calculations for cost

Comprehensive CEA

Intervention QALY = 0.25+3.49 = 3.74 Intervention cost = 27,000+167,818 =

194,818 Placebo QALY = 0.23+2.94 = 3.17 Placebo cost = 24,000 + 176,275 = 200,275 ICER = (194,818-200,275)/(3.74-3.17) = -

9,573 $/QALY

Conclusion

By considering its long-term effects, the intervention moves from “not cost effective” to “cost saving,” even if the price of the drug is increased substantially. This result holds across a wide range of sensitivity analyses.

Outline

Rationale for modeling Stroke model described Applying the SPM to a

randomized trial Extensions (*)

Extensions

International applications Planning trials

International applications

Basic idea: Use the same SPM but change the inputs.

In theory, every component of the input data (e.g., natural history, effect of covariates, effect of interventions, QOL, cost) could differ by nation.

In practice, the biggest differences will involve utilization patterns (and thus cost).

Comment

The size of the task depends upon the difficulty in estimating the relevant parameters. This, in turn, depends what we are willing to

assume versus what will require additional data collection.

Once new estimates have been derived, inserting them into the SPM is

straightforward.

International costs

Costs are affected by Different utilization patternsDifferent unit pricesDifferent payment mechanisms /

perspectives (affecting which components of cost to include in the calculation)

Recommendation

Begin with US data on utilization and unit prices. Then use expert judgement (supported by the

literature as available) to posit any changes in utilization patterns.

Attach country-specific unit prices. Limit the analysis to the cost categories and time

periods of interest to decision-makers in that country.

Planning trials

The SPM can help to determine the (minimum) clinically important difference in short-term disability.

This, in turn, would be the basis for the trial’s sample size calculations.

Observation

Many stroke trials have been under-powered.

That is, the size of the effect which is significant from the perspective of public health / CEA is much smaller than that suggested by clinical intuition (Samsa et al, Journal of Clinical Epidemiology, in press).

Observation

Many RCTs of acute stroke treatments have had other sources of statistical inefficiency -- for example, pertaining to the choice of target population, the choice of measure, and so forth.

Summation

Because most benefits of stroke treatments accrue in the long-term, modeling is necessary.

The SPM is a well-validated model which can be helpful in both the planning and analysis of trials.

Planned extensions include international practice patterns and user-friendly software.