Ideals and Norms in

Scholarship

Paul Groth

Elsevier Labs

April 20, 2015

Warning:

my perspective

not comprehensive

still thinking

Responsible Science, Volume I: Ensuring the Integrity of the

Research Process. 1992

Panel on Scientific Responsibility and the

Conduct of Research, National Academy of

Sciences, National Academy of

Engineering, Institute of Medicine

Smarter people have thought more:

Ideal

“a principle to be aimed at”

Ideal

“a principle to be aimed at”

Norm

“a standard or pattern, esp. of social behavior, that is typical or expected of a group”

WHY NOW?

Reproducibility Crisis

“Amgen, recently dispatched 100 Amgen

scientists to replicate 53 landmark experiments

in cancer—the kind of experiments that lead

pharmaceutical companies to sink millions of

dollars to turn the results into a drug. In March

Begley published the results:They failed to

replicate 47 of them.”

“Good Scientist! You Get a Badge” Slate Aug 14. 2012

Original Paper: http://doi.org/10.1038/483531a

Figure 1. Papers published and retracted per year since 1973.

Steen RG, Casadevall A, Fang FC (2013) Why Has the Number of Scientific Retractions Increased?. PLoS ONE 8(7): e68397.

doi:10.1371/journal.pone.0068397

http://127.0.0.1:8081/plosone/article?id=info:doi/10.1371/journal.pone.0068397

IS IT A PROBLEM FOR

COMPUTER SCIENCE?

http://reproducibility.cs.arizona.edu/v2/in

dex.html

http://cs.brown.edu/~sk/Memos/Examining-Reproducibility/

IDEAL

The R Dimensions

Research Objects facilitate research that is

reproducible, repeatable, replicable, reusable,

referenceable, retrievable, reviewable, replayable,

re-interpretable, reprocessable, recomposable,

reconstructable, repurposable, reliable,

respectful, reputable, revealable, recoverable,

restorable, reparable, refreshable?”

@dder 14 April 2014

sci method

access

understand

new use

social

curation

Research Object

Principles

WHY NOT?

Cost of documentation

http://www.indoition.com/en/services/costs-

prices-software-documentation.htm

People try to use the science only if

the science looks important

Why? – it’s expensive to r* science – see the next slides

24Yolanda GilUSC Information Sciences Institute gil@isi.edu

Measuring Time Savings with

“Reproducibility Maps” [Garijo et al PLOS CB12]

2 months of effort in reproducing published method (in PLoS’10)

Authors expertise was required

Comparison of ligand binding sites

Comparison of dissimilar protein structures

Graph network generation

Molecular Docking

Work with D. Garijo of UPM and P. Bourne of UCSD

“The Reproducibility Project: Cancer Biology has been funded through a

$1.3mm grant from the Laura and John Arnold Foundation,”

http://validation.scienceexchange.com/#/cancer-

biology

NORMS

Remember: science is a social system

Publishing is central

• “But Ivan Oransky, founder of the blog

Retraction Watch, says data-sharing isn’t

enough. The incentive structure in science

remains a problem, because there is too

much emphasis on getting published in top

journals,…”

http://www.washingtonpost.com/national/health-science/the-new-scientific-revolution-reproducibility-at-last/2015/01/27/ed5f2076-9546-11e4-927a-

4fa2638cd1b0_story.html

Publishers Guidance

• Committee on Publishing Ethics

– http://publicationethics.org

– How to handle authorship

• http://publicationethics.org/files/u2/2003pdf12.pdf

• Society of Scholarly Publishing

– Ethical principles of scholarly publishing

Guidance for Authors

• Conflict of Interest

• Confidentiality

• Privileged Information/Transparency

• Reporting standards and research conduct

• Data Access and Retention

• Originality and Plagiarism

• Multiple, Redundant or Concurrent Publication

• Acknowledgement of Sources

• Authorship of the Paper

• Fundamental errors in published works

http://www.stm-assoc.org/2013_05_21_STM_Ethical_Principles_for_Scholarly_Publication.pdf

Two norms sum it up

• Be transparent*

• Embrace the iteration inherent in science

* If we encourage transparency, well then we should be constructive

Co-rex-ions

http://whatsinjohnsfreezer.com/2014/05/10/co-rex-ions/

“My message from that

experience is to get out

in front of problems like

this, as an author.”

http://ivory.idyll.org/blog/replication-i.html

Registered Reports

• “conducting the peer review prior to data

collection and analysis”

“Science’s incremental, self-correcting

process can be carried out quite effectively

by publishing new papers that correct and

improve on old ones, rather than dismantling

the older papers themselves.”

http://whatsinjohnsfreezer.com/2014/05/10/

co-rex-ions/

Constructive criticism

“I like to think of the Braintrust

as Pixar’s version of peer

review, a forum that ensures we

raise our game – not by being

perspective but by offering

candor and deep analysis”

-- Ed Catmull p. 93

Conclusion

• Scientists are people too

• Ideal as a guiding light

• Discussion

– What are other norms?

– Is this a correct characterization of ideals?