summarising the evidence from animal models of neurological disease: publication bias, poor internal...
TRANSCRIPT
Summarising the evidence from animal models of neurological disease: Publication bias, poor internal validity, and (perhaps) some efficacy
Malcolm MacleodUniversity of Edinburgh
• Lots of drugs seem to work in animal models of stroke
• Some of these drugs have been around for ages, and are probably safe
• Rather than spend billions developing a novel drug, why not test something we already know works in animals?
Where we started …
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
What do we already know works in animals?
• Lots of individual promising studies
• Little in the way of research synthesis
So …• Review the field systematically• Pick a target and synthesise the evidence
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
1026
1026 interventions in experimental stroke
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
1026603
1026 interventions in experimental stroke
Tested in focal ischaemia
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
1026883374
1026 interventions in experimental stroke
Effective in focal ischaemia
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
1026883550
97 18
1026 interventions in experimental stroke
Tested in clinical trial
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
1026883550
97 171 3
1026 interventions in experimental stroke
Effective in clinical trial
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
Outline
• Review the internal validity of animal studies
• Review the external validity of animal studies
• Explore the potential uses of this approach in mechanisms research
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
Methodological approach
• Written review protocol – hypothesis, inclusion and exclusion criteria, analyses prespecified
• Standardised search strategy – 3 online databases, conference abstracts, dual screening
• Data extraction to bespoke database/analysis tool
• Random effects weighted mean difference stratified meta-analysis or meta-regression
• Publication bias by Funnel plot, Egger regression, Trim and Fill
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
There is nothing new under the sun …“…you will meet with several observations and experiments
which, though communicated for true by candid authors or undistrusted eye-witnesses, or perhaps recommended by your own experience, may, upon further trial, disappoint your expectation, either not at all succeeding, or at least varying much from what you expected”
Robert Boyle (1693)Concerning the Unsuccessfulness of Experiments
Internal validity …
• the extent to which an experiment accurately describes what happened in that model system
• may be confounded bySelection Bias (Randomisation)Performance Bias (Allocation Concealment)Detection Bias (Blinded outcome assessment)Attrition bias (Reporting drop-outs/ ITT
analysis)False positive report bias (Adequate sample sizes)
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
NXY-059 in animal stroke models9 publications, 29 experiments, 408 animals 44% (35-53%) improvement in outcome
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
A widespread problem …
Dopamine Agonists in 6-OH model of Parkinson’s Disease
Various interventions in 435 experiments (7170 animals) in EAE
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
Internal validity and Journal Impact Factors• Regression model offered:
– Effect size– Variance– Randomisation– Allocation concealment– Blinded assessment of outcome– Sample size calculation– Statement of possible conflict of interest
• Univariate:– Allocation concealment– Blinded assessment of outcome– Statement of possible conflict of interest
• Multivariate: IF = 3.7 + 2.4(Conflict of Interest Statement) + 1.2(Allocation Concealment)
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
External validity
• Has the intervention been tested under a range of circumstances similar to those which might be encountered in clinical practice?
• Are the data you know about representative of all data? (what is the likelihood of publication bias)
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
Hypertension in studies of NXY-059 in experimental stroke
Hypertension:– 7% of animal studies– 77% of patients in the
(neutral) SAINT II study
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
Hypertension in studies of tPA in experimental stroke
Comorbidity
“Normal” BP
Effi
cacy
-2%25%
Infarct Volume:– 113 publications– 212 experiments– 3301 animals– Improved outcome by 24% (20-28)
Hypertension:– 9% of animal studies– Specifically exclusion criterion in (positive)
NINDS study
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
The importance of Time to Treatment• Both tPA and tirilazad appear to work in animals
• tPA works in humans but tirilazad doesn’t
• Time to treatment: tPA:– Animals – median 90 minutes– Clinical trial – median 90 minutes
• Time to treatment: tirilazad– Animals – median 10 minutes– Clinical trial - >3 hrs for >75% of patients
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
Publication bias in experimental stroke
• Only 11/525 publications (2.2%) reported no significant treatment effects
• Trim and Fill suggested ~16% (214/1573) of experiments remain unpublished
• Best estimate of magnitude of problem – Observed efficacy31.3% (29.7-32.8)– Adjusted efficacy 23.8% (22.2-25.5)
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
Publication bias
Randomisation Co-morbidity
bias
Reported efficacy
24%32% 18% 4%
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
There are multiple drivers of bias
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
Developing a lexicon for mechanisms research
• The interval validity of findings– Are they the product of bias?
• The validity of research summaries– Have the authors of “pivotal” reviews considered all relevant data?
• The external validity of findings– Do these findings hold only in limited highly controlled situations, or only one species, or are they generally applicable across biology?
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
The added value of a systematic approach …
• Identifying, and explaining, heterogeneity
• Developing a systematic evidence base for understanding biological pathways
• Providing the components for mathematical models predicting the response to interventions
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
The role of Th17 cells in the development of EAE
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
Immunisation EAETh17
Th1
CD4+
- = +
- 6
= 3
+ 6 1
EAE
Th
1
- = +
- 11 1
= 1
+ 2 2
EAE
Th
17
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
Summary
• The internal and external validity of animal experiments is limited
• Conclusions from such experiments are confounded by many biases
• There is no reason to believe that more “pathophysiological” experiments are any better
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary
Future directions
• Qualitative systematic research• Quantitative systematic research– Study quality issues– Evidence based pathways– Strategic research targeting
Introduction Translation –Internal Validity
Translation –External Validity
Mechanisms Research
Summary