Breaking the phenotyping barrier

and the dire need for identification

Anita Bandrowski, SciCrunch / NIF / RRID

Phenotyping data comes from authors Curators read

the articlesAlign free text to community

ontologiesDeposit

aligned data to databases

Step 1 is identification

But, quality of published research is in question

Solving quality: mandates and solutions

NIH NOT-OD-16-011Journals RRIDs

Societies FASEB GuidelinesNon-Profits TOP Guidelines

*Reproducibility of science is key*

What is reproducible science?

Step 1: Material

s

Step 2: Method

s

Step 1: In

Practice

How common is this?

Papers are currently poor at identifying the

simplest part of the paper, the materials used

Vasilevsky 2013

But the author knows what was used! This author got back to me within 2 hours with the stock number of this mouse

Left open annotation with this stock number from JAX, but will others find this?

How can we get better data?• 2009: LAMHDI meeting – project hatched • 2011: Meeting with Society for Neuroscience, Journal of Neuroscience full editorial board presenting the

problem and results of text mining study• 2012: Society for Neuroscience – defined the problem for Editors of top Neuroscience journals; sponsored

by INCF• 2013: NIH Meeting - brought the editors back to define the solution; 2 day workshop sponsored by NIDA

and INCF, several IC directors in attendance• 2013: Society for Neuroscience – mainly publishers, defined the timeline of starting the project• 2014: Neuroscience Information Framework – built scicrunch.org/resources based on NIF technologies and

members of the OHSU team populated web pages / instructions etc.• 2014: Project starts with Journal of Neuroscience, Neuroinformatics, F1000, Brain and Behavior and

Journal of Comparative Neurology taking a strong lead• 2015: Paper describing how RRIDs are used by authors of the first 100 papers is co-published in 4 journals• 2016: integration with Hypothes.is tool gives curators an easy way to verify RRIDs, sci-score gives

authors an easier way to detect what is a resource

Journals key solution

Journals will not change

Journals in aggregate

Funders role

Project Management Key

ID based tracking

Technology innovation

Inclusion of stock center data Repository Name StatusAmbystoma Genetic Stock Center IncludedA Resource Center for Tetrahymena thermophila IncludedDevelopment of Validated Drosophila in vivo RNAi Models of Human Diseases IncludedWormBase IncludedJAX IncludedRGD IncludedMGI IncludedZebrafish International Resource Center IncludedBloomington Drosophila Stock Center at Indiana University IncludedMutant Mouse Resource and Research Center Included

MMRRC at University of California, Davis IncludedZFIN IncludedCaenorhabditis Genetics Center IncludedThe Mouse Mutant Resource (MMR) IncludedCre Driver Strain Resources IncludedDrosophila Genomics Resource Center IncludedMutant Mouse Regional Resource Center at University of North Carolina IncludedMutant Mouse Resource and Research Center at the University of Missouri IncludedNational Swine Resource and Research Center IncludedNational Xenopus Resource Center IncludedRat Resource and Research Center IncludedSperm Stem Cell Libraries for Biological Research. IncludedThe Special Mouse Strain Resource (SMSR) at The Jackson Laboratory IncludedXiphophorus Genetic Stock Center IncludedAnimal Model Resources for Cystic Fibrosis No repsonseNational Gnotobiotic Rodent Resource Center No repsonseAdult Mesenchymal Stem Cell Resource Stocks not availableGene Library Resource for the Sea Urchin S. purpuratus Stocks not availablePrimate Embryo Gene Expression Resource Stocks not availableResearch Resources for Model Amphibians Stocks not availableViper Resource Center (VRC) at Texas A&M University-Kingsville (TAMUK) Stocks not availableWormGuides Stocks not available

Data in SciCrunch is used by authors to identify organisms, authors publish papers and curators and scientists then find clear simple citations to resources

Copy/PastePublish

Resources are identifiable!

Bandrowski et al, 2015

Which repositories are being cited?• This data set is from 980

papers, 499 organisms noted • Curators filled in resources that

are missing RRIDs in a small number of cases (eg., registered reports for cancer reproducibility studies in eLife), most are RRIDs asserted by authors

• Pie chart shows total number of organism annotations by species

• Most citations are to repositories that have been included for the longest time (eg JAX, RGD, BDSC)

• NXR – frog is a recently joined repository, 1st paper is out

Data can be found herehttps://docs.google.com/spreadsheets/d/1VOUml95YoxGQnG0hjTwOKHf6vbVb6hmNb53L2bixLGM/edit?usp=sharing

JAX MGI BDSC RGD ZFIN MMRRC CGC ZIRC RBRC EM KOMP NCIMR NXR0

50

100

150

200

250

223

104

55 48

18 179 9 6 3 2 1 1

Fly; 55; 11% Fish; 27; 5%

Worm; 9; 2%

Rat; 48; 10%Mouse

; 359; 72%

Frog; 1; 0%

Most popularRRID:IMSR_JAX:000664 27RRID:RGD_70508 9RRID:ZIRC_ZL1 7RRID:RGD_737903 5RRID:RGD_734476 5RRID:IMSR_JAX:012569 5RRID:RGD_737929 3RRID:RGD_737891 3RRID:IMSR_JAX:007677 4RRID:IMSR_JAX:006410 4RRID:IMSR_JAX:005628 4RRID:IMSR_JAX:000671 4RRID:ZFIN_ZDB-GENO-030619-2 3RRID:IMSR_MMRRC:000230 3

Recent Journals/PublisherJuly 2016

Aug 2016 Sept/Oct 2016

11 papers

Why is this working?How: Make it easy for authors to

include a unique identifier for each research resource that is used

At publicationInstructions to authors

*Instructions to reviewers*Direct contact from editors

We get 97% accuracy from authors (~1000 papers verified, ~9000

RRIDs)

Bandrowski et al 2015F1000 doi: 10.12688/f1000research.6555.2, Journal of Comparative Neurology (doi: 10.1002/cne.23913),

Brain and Behavior (doi: 10.1002/brb3.417) and NeuroInformatics (doi: 10.1007/s12021-015-9284-3).

Make compliance easy and traceable for journals

Attention from NIH, societiesJournals listen

201425 journals signed on5 executed effectively

2016139 journals represented

10 execute effectively*many more coming*

Next Steps ….we are at 1%, how to get to 100?

Fly Fish Worm Rat Mouse Frog0

0.2

0.4

0.6

0.8

1

1.2

1.41.31233595800525

0.913705583756345

0.627615062761506

0.122019421424577

1.22042425890672

0.909090909090909

RRIDs as a Percent of total papers 2015/2016 (Organism search term) Outreach: We need more journals / authors; we

need support from the stock centers

Peer pressure: We need to stress the new NIH guidelines, thank you NIH!

Tools: We need to have tools that make the process easier for journals

SciBot – finds RRIDs in papers (curation tool)Hypothes.is – display annotations on papers – W3C compliant annotationsSci-Score.com – finds sentences that should have RRIDs

*estimate of the number of papers per organism in 2015/16 was based on key words in PubMed representing the main model organism (mouse, rat etc)

To put the proper citation format onto your repository!To help increase awareness of RRIDs (blogs, webinar, newsletters, twitter #RRID)

To add / ask for RRIDs (authors / reviewers)To bring your journal (editors)

Comments / Complaints: abandrowski@ucsd.edu

6 papers since late April 2016