woolcock institute 20 mar 2012

[email protected] of Psychology

@ceptional

The broadest problem in science:Our publishing system

http://www.slideshare.net/holcombea/

1

Opacity, Sluggishness, Redundancy, Redundancy, & Redundancy!

from Kravitz &

Baker (2011)

Inefficient

2

The File-Drawer Problem

unpublished results

files

•Little career incentive to publish a non-replication or a replication

•Very difficult to publish a non-replication or replication

•Most journals only publish papers that “make a novel contribution”

•Reviewers/editors tend to hold non-replicating manuscript to higher standard than original.

•Bem example

http://www.flickr.com/photos/nickperez/2569423078 t. magnum

Incomplete

3

The File-Drawer Problem

Tower of unpublished

resultsfiles

Corollary 4: The greater the flexibility in designs, definitions, outcomes, and analytical modes ina scientific field, the less likely the research findings are to be true. Flexibility increases the potential for transforming what would be “negative” results into “positive” results.

Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.

http://www.flickr.com/photos/nickperez/2569423078 t. magnum

“In summary, while we agree with Ioannidis that most

research findings are false...”

Incomplete

4

Barriers to publishing replications and failed-replications

• No glory in publishing a replication

• Few journals publish replications

• usually uphill battle even with those that do

• The wrath of the original researcher

5

File-drawer fixes

• Journals that don’t reject replications for being uninteresting or unimportant

• Pre-registration of study designs and analysis methods

• Brief reporting of replications

✔•◦

✔

•◦ ◦

◦◦

6

• problems: incentives

http://psychfiledrawer.org/view_article_list.php

7


Pashler, Spellman,

Holcombe& Kang (2011)

DETAILS page: http://psychfiledrawer.org/replication.php?attempt=MTU%3D

8


http://psychfiledrawer.org/view_article_list.php

9

File-drawer fixes

• Journals that don’t reject replications for being uninteresting or unimportant

• Pre-registration of study designs and analysis methods

• Brief reporting of replications

✔•◦

✔

•◦ ◦

◦◦

10

ROIM- peer Review of Intro & Methods1. Authors plan a replication study

2. They submit an introduction and methods section

3. It is sent to reviewers, including the targeted author

4. The editor decides whether to accept/reject, based on:

1. Reviewer comments regarding the proposed protocol

2. Importance of the study, judged by argument in the introduction, number of citations of original, reviewer comments

5. The Intro, Method and analysis plan, and reviewer comments are posted on the journal website

6. When the results come in, the authors write a conventional results and discussion section and that together with the raw data are posted, yielding the complete publication

1. some sort of minimal peer review needed for that. What exactly?

✔••

More efficient, more complete

11

ROIM- peer Review of Intro & Methods

• Original author sort-of signed off on it, so can’t complain / hate the replication authors as much.

• Good way to start for a new PhD project, anyone planning to build on some already-published results

• Reduce the incentive to publish flashy, headline-grabbing but unreliable studies?

✔••

12

When a new paperappears, readers often spot logical flaws,experimental weaknesses, questionableassumptions or alternative interpretations.Yet individual criticisms may not beconsidered important enough to warrantpublication. Even major criticisms areunlikely to appear until months or yearslater, and are often overlooked in thehaystacks of the literature.

post-publication peer review

Incomplete

13

I did a spot check of fifty of those citing articles to see if any had noted problems with the paper: only one of them did so. The others repeated the authors’ conclusions

The article in question was published in the Proceedings of the National Academy of Sciences, and at the time of writing has had 270 citations.

http://deevybee.blogspot.com.au/2012/03/time-for-neuroimaging-to-clean-up-its.html


The people who DON’T notice the problems with a paper cite it, the people who DO, don’t. So people not in the area never find out how flawed a paper is.

14


Absence of it means we don’t have many indicators of quality of individual articles.

Reinforces reliance on poor measures like journal impact factor.

15

I have a dream.. our papers will be judged not by the impact factor of their journal, but by the quality of their content

16

I have a dream that one day,

our papers will be judged not by the impact factor of their journal, but by the quality of their content

Impact Factor announced (4.3)

from Peter Binfield’s talk

17


• Original author sort-of signed off on it, so can’t complain / hate the replication authors as much.

• Good way to start for a new PhD project, anyone planning to build on some already-published results

• Reduce the incentive to publish flashy, headline-grabbing but unreliable studies?

• How to incorporate post-publication commentary?

✔••

18


• What publisher to publish it?

19

Academic knowledge is boxed in by expensive journals.

Scientist meets publisher: the video

http://www.youtube.com/watch?v=GMIY_4t-DR0

20

•Academia marginalising itself

•More impact!

•We do our research to benefit anyone interested, not some exclusive club

•Many scholars, doctors, patients, engineers, policymakers (and esp. in poor countries/small universities) can’t get access

The academic community

is only hurting itself, and

its long term public

support, by keeping its

knowledge behind high

subscription walls -

Andrew Carr

The countries we work with can’t afford journals; they’re already paying an arm and a leg for textbooks -Sir John Daniel

Why Open Access?• Free• Full-text• Online access

21

Open Access “Hulk”

OA HULK WANTS TO KNOW WHO TO OCCUPY!

ELSEVIER!? ACS!? HARPERCOLLINS!?

YOU NAME IT, OA HULK WILL OCCUPY AND SMASH!

22

Experimental Brain Research $13,670Journal of Radioanalytic and Nuclear Chemistry $19,826

Journal of Mathematical Sciences $17,880Journal of Materials Science $16,699

Institutional Subscription Cost, 2012

Claudio Aspesi at http://poynder.blogspot.com.au/2012/03/open-access-brick-by-brick.html

$3983 USD per article for Elsevier $1350 USD per article for PLoS ONE

23

Claudio Aspesi at http://poynder.blogspot.com.au/2012/03/open-access-brick-by-brick.html

$3983 USD per article for Elsevier $1350 USD per article for PLoS ONE

24

•Not all closed-access publishers profiteer

•The real enemy is us!

Ryan Orr http://www.flickr.com/photos/ryan_orr/615470501

25

•Deposit your manuscripts in the university repository (http://ses.library.usyd.edu.au/

•Even with closed journals, you often have the right to deposit your final version (e.g. Word document before typeset by publisher)

•Best for university and its funders if research outputs open access; indeed it’s mandated by:

•NIH, Wellcome Trust

•Princeton, Harvard

•Queensland University of Technology

•Support open-access publishing models

•PLoS, BioMed Central, eLife

•I don’t recommend paying for open-access “choice” in closed corporate-published journals

•Support innovations that address problems of inefficiency and incompleteness

Let me know if you want to help open Sydney Uni

How to help

26

Comprehensive solution? Open Science

•As data comes in, uploaded automatically to web

•Electronic lab notebook

•Papers written via open collaborative documents on the web

27

Wikipedia(vs.(Academic(Papers(12/21

Mat Todd

28

Mat Todd

This stimulated 20 comments (from 11 different people) and four private e-mails (via the website).

Industry suffers less from such metrics, but it is nevertheless surprising that industry were so heavily involved in this project. For example, of the roughly 100 comments since January 2010 on The Synaptic Leap website, around 60 came from readers not involved in the kernel project at Sydney, and of those approximately 42 came from industry, 16 from academia. Besides the input described above to the resolution experiments, a different company contributed samples of PZQ enantiomers isolated by chromatography for analytical purposes, and another company is currently determining the phase diagram of PZQamine. Why would companies choose to be involved, particularly in a project in neglected tropical diseases where there is little profit margin and no new intellectual property available? One can appeal to human nature — we see a problem we can help solve, and we find it impossible to resist stepping in, p

29