CAMARADES: Bringing evidence to translational medicine
Study Quality and Publication Bias in Experimental Stroke
CAMARADES: Bringing evidence to translational medicine
Presenter Disclosure Information
FINANCIAL DISCLOSURE:None
Malcolm Macleod PhD FRCPStudy Quality and Publication Bias in Experimental Stroke
UNLABELED/UNAPPROVED USES DISCLOSURE:NoneCONFLICT OF INTEREST DISCLOSURE:President, EuroHYP
CAMARADES: Bringing evidence to translational medicine
1026
1026 interventions in experimental stroke
Tested in experimentsO’Collins et al, 2006
CAMARADES: Bringing evidence to translational medicine
1026603
1026 interventions in experimental stroke
Tested in focal ischaemiaO’Collins et al, 2006
CAMARADES: Bringing evidence to translational medicine
1026883374
1026 interventions in experimental stroke
Effective in focal ischaemiaO’Collins et al, 2006
CAMARADES: Bringing evidence to translational medicine
1026883550
97 18
1026 interventions in experimental stroke
Tested in clinical trialO’Collins et al, 2006
CAMARADES: Bringing evidence to translational medicine
1026883550
97 171 3
1026 interventions in experimental stroke
Effective in clinical trialO’Collins et al, 2006
CAMARADES: Bringing evidence to translational medicine
Control half dose full dose
Infa
rct V
olum
e
0
50
100
150
200
250
300
10-120 M 10-60 M
CAMARADES: Bringing evidence to translational medicine
What is translational medicine?
From: Institute of Translational Medicine, University of Texas Medical Branch
CAMARADES: Bringing evidence to translational medicine
Animal data in stroke
• There are huge amounts of often confusing data
• Systematic review can help to make sense of it
• If you select extreme bits of the evidence you can “prove” either harm or substantial benefit
• However, if you have a precise and highly significant overall effect, then it is probably real
Hypothermia: a systematic search identified 277 experiments in
3353 animals
Bett
er
Wors
e
Van der Worp et al, 2007
CAMARADES: Bringing evidence to translational medicine
CAMARADES: Bringing evidence to translational medicine
Potential sources of bias in animal studies
• Internal validity
• External validity– Publication bias – Are the models we use good models?
• Co-morbidities
Problem Solution
Selection Bias Randomisation
Performance Bias Allocation Concealment
Detection Bias Blinded outcome assessment
Attrition bias Reporting drop-outs/ ITT analysis
CAMARADES: Bringing evidence to translational medicine
Internal ValidityRandomisation and blinding in studies of hypothermia in experimental stroke
• Infarct Volume– 101 publications– 222 experiments– 3256 animals– Improved outcome by
43.5% (40.1-47.0) Blinded outcome
assessment
Yes No
Effi
cacy
47%39%
Randomisation
47%37%
Effi
cacy
Van der Worp et al, 2007
CAMARADES: Bringing evidence to translational medicine
Internal Validity NXY-059
• Infarct Volume– 11 publications
– 29 experiments
– 408 animals
– Improved outcome by 44% (35-53%)
Macleod et al, 2008
CAMARADES: Bringing evidence to translational medicine
The File Drawer problemPublication bias
0worse better
CAMARADES: Bringing evidence to translational medicine
BetterWorse
Pre
cisi
on
• All outcomes– 29 publications– 109 experiments– 1596 animals– Improved outcome by 31% (27-35%)
External ValidityPublication Bias for FK506
Macleod et al, 2005
CAMARADES: Bringing evidence to translational medicine
Publication bias – G-CSF
England T et al 2009
CAMARADES: Bringing evidence to translational medicine
Publication bias in experimental stroke
• Only 11/525 publications (2.2%) reported no significant treatment effects
• Trim and Fill suggested ~16% (214/1573) of experiments remain unpublished
• Best estimate of magnitude of problem – Observed efficacy 31.3% (29.7-32.8)– Adjusted efficacy 23.8% (22.2-25.5)
CAMARADES: Bringing evidence to translational medicine
Publication bias
Randomisation
Co-morbidity
bias
How much efficacy is left?
CAMARADES: Bringing evidence to translational medicine
There are many other sources of bias (Sacket, 1979)
• Tidying up bias• Repeated peaks bias• Significance bias• Bogus control bias• Hot stuff bias• Selective reporting bias
CAMARADES: Bringing evidence to translational medicine
Current performance against key quality items
RandomisationBlinded Outcome
AssessmentSample Size calculation
Stroke 36% 29% 3%
CAMARADES: Bringing evidence to translational medicine
How does stroke compare?
RandomisationBlinded Outcome
AssessmentSample Size calculation
Stroke 36% 29% 3%
MND 31% 20% <1%
AD 15% 25% 0%
PD 12% 15% 0%
EAE 8% 15% <1%
Glioma 14% 0% 0%
CAMARADES: Bringing evidence to translational medicine
Internal validity in PD models
Blinded outcome assessment Composite quality
Rooke et al, submitted
CAMARADES: Bringing evidence to translational medicine
Estimates of affinity at cannabinoid receptors
McPartland et al 2007
CAMARADES: Bringing evidence to translational medicine
AnimalStudies
Clinical Trial
Systematic Review
AndMeta-analysis
CAMARADES: Bringing evidence to translational medicine
Conclusions
• We could, and should, do a lot better• Other diseases could do much much better• Like any other process, the processes of
translational medicine can be studied, systematically and methodically