1
Unreliable researchTrouble at the lab
Patrik Ahlberg Magnus Åhberg
Syeda Rabab NaqviThawatachart Chulapakorn
Group Members:
3
Outline:Introduction
Understanding Insignificance
Not Even Wrong
Blame the Ref
Harder to Clone than you wish
Making the Paymaster´s Care
Personal Reflections
4
Introduction
• Irreproducibility is on rampage• Self corrections are not working IRL
“There is no cost to getting things wrong, The cost is not getting them published.”
Brian Nosek, University of Virginia
5
Understanding Insignificance
• False Positives + False Negatives = Problem
• We are not using statistics
• Not possible with power of 0.8
6
Not even wrong
• Research is commonly not thought through or not well executed
• Examples:– The ”pentaquark” saga (experiment not properly
blinded)– Risks with computer model ”tuning”– Longevity associated genetic variations due to
inproper handling of research samples
7
Blame the ref
• Peer reviewers are not detecting errors– John Bohannon´s made-up paper was published in
157 out of 304 journals– Fiona Godlee yielded similar results when sending
an 8-error paper to the 200 reviewers of BMJ• Errors are explained by incompetence rather
than fraud• The replication mechanism for scientific self-
correction is not functioning well
8
Harder to Clone than you wish :
Replication is hard to meet our standard
To find errors in the publications , we need to replicate the work of others, but process is not according to our wish.
Replication
Lack of interest by Journals and Academic Researchers
9
Lack of Interest
Journals thirsty of novelty, show little interest in it.
Academic researchers mostly spend time on work which is
more likely to enhance their careers.
Replication is Hard
Original methods and data required
Unpublished Research
Clinical trials are very costly to rerun
Software may be different during the replication
Failure of replication due to tacit knowledge
Expermenter’s regress
10Finally, we would say that…
Making the paymasters careSituation Paymasters Researcher
Problem - Prefer pioneer- Deny failed result
- Paper machine- Bias discussion- Blame the
previous errorPurpose - Emphasize
replication as same as others
- Don’t serious the failed result (Hard)
- Develop technical and statistical skill
- Honest result- Enforce standard
(Journal)
Quality is more important than Quantity
11
Patriks work
• Unethical publishing cancels this problem– No reproducibility possible– Small feasibility tests
• The system has given up on the system– Seniors warning juniors– Reliability comes from people not publications
12
Own reflections
• The possibility to ”tune” computer models until the desired results appear relates to my own research
• It is easy to percieve patterns and get results that are expected and might not even exist
• Responsibility as a reviewer:– When do I accept a reviewer invitation?– Do I expect to find errors? What am I looking for
as a reviewer?
13
Own ReflectionsIf some one tries to repeat my experiment, he must have Access to original methods and data Awareness of simulation procedure Similar samples for study
14
Related to my (future) research
• Replication– Same or not? Why?
• Scrutiny– Procedure– Measurement
• Bias discussion• Raw data• Log book!!!
15