1 subgroup reporting in the general medical literature: do investigators misinterpret their own...
Post on 14-Dec-2015
213 Views
Preview:
TRANSCRIPT
1
Subgroup Reporting in the General Medical Literature: Do Investigators Misinterpret Their
Own Findings?
Erik Fernandez y Garcia, MD MPHUniversity of California, Davis
Co-authors: Hien Nguyen, MD (UCD); Naihua Duan, PhD (Columbia University); Nicole
Bloser Gabler, MHA MPH (UCD); Diana Liao MPH(UCLA); Richard L. Kravitz, MD MSPH
(UCD)
Supported by a grant from Pfizer.
2
RATIONALE
• Randomized-controlled trials (RCTs) yield an average treatment effect.
• In a trial, treatments may have different net benefits and harms for different patients (heterogeneity of treatment effects, or HTE).
• Examination of the HTE is important for optimizing treatment for individual patients.
• Especially critical in era of increasing population diversity and health disparities.
3
RATIONALE
• Subgroup analysis (SGA) in RCTs is one way of investigating HTE.
• The usefulness of SGA is hampered by the problems of
– insufficient power (false negative)
– multiple testing (false positive)
4
RATIONALE
• Under-use: SGA not performed in studies with sufficient power and theoretical rationale to anticipate helpful results.
• Over-use: SGA performed in studies which were underpowered or lacked theoretical rationale to anticipate helpful results.
• Misuse: SGA was performed in the appropriate setting but with inappropriate methodology and/or interpretation.
5
STUDY QUESTIONS
We sought to specifically investigate the potential misuse of SGA by asking:
1) How often are HTE analyses and corresponding covariates prespecified and were the reasons (if any) primarily substantive or statistical?
2) What was the objective evidence for or against the presence of HTE?
3) How did authors interpret their own HTE-related findings, and to what extent did their interpretations match the objective evidence?
6
METHODS
• Design: Systematic Review• Population: JAMA, BMJ, Lancet, NEJM, Annals• Probability Sample:
– Odd months in 1994, 1999, 2004– Initial search: N = 4,863 articles – After additional random sampling and
exclusions, N = 319 clinical trials– Final sample: 87 of 319 trials (27%) reporting
test for HTE
7
Covariates Examined in HTE analyses
Prespecified Not Prespecified
All covariates Some
Substantive Statistical
Number of Covariates with Rationale
None
Types of Reasons
Coding of Covariates
8
Defining Clinicostatistical Divergence
Clinicostatistical Divergence: Clinically meaningful and statistically significant differences between subgroup effects and average effect (coded as “none” “weak” “moderate” “strong”)
– Clinical Divergence (CD): Was the ratio measure of effect in any subgroup at least 25% greater or smaller than in the sample as a whole?
– Statistical Significance (SS): Was a test for interaction associated with a p value of less than or equal to 0.10?
9
Coding Clinicostatistical Divergence
* Denotes absence of data
Strength of Evidence
CD
SS
Prespecified Covariate?
Strong Yes Yes Yes Moderate Yes Yes No Weak Yes No Weak Yes * Weak No Yes Weak * Yes None No No None * No None No * Unable to Classify * *
10
Coding Authors’ Interpretations
• Evidence for HTE sufficient to support different treatment recommendations in one or more subgroups
• Evidence for HTE insufficient to support different treatment recommendations but sufficient to warrant further systematic research
• Evidence for HTE was possibly present but insufficient to warrant further research
• Definite evidence against HTE
• No interpretation of HTE results
11
87 RCTs
Prespecified Not Prespecified
All covariates Some
Substantive Statistical
Trials by Number of Covariates with Rationale
None
Types of Reasons
RESULTS
53 (61%) 34 (39%)
17 (32%) 12 (23%) 24 (45%)
22 (76%) 7 (24%)
Strength of Evidence Author’s Interpretation Unable None Weak Moderate Strong Total Supports Differential Treatment
0 2 5 1 2 10
Warrants Further Research
10 6 9 5 1 31
Possibly Present
0 0 0 0 1 1
Evidence Against
6 2 8 2 2 20
No Interpretation
13 6 3 3 0 25
Total 29 16 25 11 6 87
Strength of Evidence Author’s Interpretation Unable None Weak Moderate Strong Total Supports Differential Treatment
0 2 5 1 2 10
Warrants Further Research
10 6 9 5 1 31
Possibly Present
0 0 0 0 1 1
Evidence Against
6 2 8 2 2 20
No Interpretation
13 6 3 3 0 25
Total 29 16 25 11 6 87
29/87 = 33%
Strength of Evidence Author’s Interpretation Unable None Weak Moderate Strong Total Supports Differential Treatment
2 5 1 2
Warrants Further Research
6 9 5 1
Possibly Present
0 0 0 1
Evidence Against
2 8 2 2
No Interpretation
6 3 3 0
Total 16 25 11 6 58
17/58 = 29%
Strength of Evidence Author’s Interpretation Unable None Weak Moderate Strong Total Supports Differential Treatment
0 2 5 1 2 10
Warrants Further Research
10 6 9 5 1 31
Possibly Present
0 0 0 0 1 1
Evidence Against
6 2 8 2 2 20
No Interpretation
13 6 3 3 0 25
Total 29 16 25 11 6 87
25/87 = 29%
Strength of Evidence Author’s Interpretation Unable None Weak Moderate Strong Total Supports Differential Treatment
0 2 5 1 2 10
Warrants Further Research
10 6 9 5 1 31
Possibly Present
0 0 0 0 1 1
Evidence Against
6 2 8 2 2 20
No Interpretation
Total 62
31/62 = 50%10/62 = 16%31/62 = 50%
Overstated = 27% Understated = 9%
Strength of Evidence Author’s Interpretation Unable None Weak Moderate Strong Total Supports Differential Treatment
0 2 5 1 2 10
Warrants Further Research
10 6 9 5 1 31
Possibly Present
0 0 0 0 1 1
Evidence Against
6 2 8 2 2 20
No Interpretation
13 6 3 3 0 25
Total 29 16 25 11 6 87
18
LIMITATIONS
• Limited number of journals, years, trials reviewed
• Data potentially incomplete– HTE analyses performed but not published
– HTE analyses performed and published in secondary journals
19
CONCLUSIONS
• Analysis and reporting of HTE incomplete
– Prespecification inconsistent
– Rationale incomplete
– Effect measures and p-values (or CIs) incompletely reported
• When reported, objective evidence for clinicostatistical divergence found in approximately 1/3 of trials
• Authors frequently misinterpret own findings (in both directions)
20
IMPLICATIONS• Researchers:
– Ensure that SGA are prespecified with a priori rationales for covariate inclusion, or clearly labeled as exploratory
– Include a statistical test for interaction or heterogeneity in the analyses
– Report all results from SGA (including p values for HTE tests and effect measures with confidence intervals), even if not significant
21
IMPLICATIONS
• Journal Editors:– Ensure authors report SGA-associated data
when SGA is performed – Ensure that authors’ discussion includes
interpretation of SGA performed and limitations of such analyses
• Readers/Clinicians:– Weigh the authors’ interpretations and
recommendations in light of the objective evidence presented prior to changing practice or implementing recommendations
top related