the quality of method reporting in

Bio-Health Informatics GroupSchool of Computer Science

The Quality of Method Reporting in Biology

ExperimentsRobert Stevens


University of [email protected]


Why is good method reporting important?

Results are determined by method

Understanding method helps understanding the results

Are the results valid?

Can these two sets of results be compared?

Can these results be reproduced?


The origin of our work

inhibitors of iNO synthase

Experiment [1]

Experiment [2]

+

+

=[…] the treatment of infected C57BL/6 mice with inhibitors of NO synthase increased parasitaemia and mortality.

+

iNO synthase-knockout

=[…] blood parasitaemia and survival rates are similar between wild-type and iNOS-deficient mice.


Our Approach

What should be reported in a parasite host experiment?

Select papers for a range of parasite types

Code against what should be reported

Generate some metrics


Investigate experiments on a range of organisms

Leishmania

Schistosoma

Toxoplasma

Plasmodium

Trypanosoma Trichuris Mycobacterium


A checklist for Trypanosoma experiments

Identify the species of the parasite

Describe the age of the animal

Describe the ratio -parasites per cell- used

Describe the way of inoculationIn vivo

Describe the supplements / antibiotics used

Expe

rimen

tal

cond

ition

sHo

stPa

rasit

e

In vitro

The full checklist is here: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0101131


Finally 23 papers on Trypanosomiasis and 10 papers selected for each of the other species.

Selecting papers


Compliance of parasite reporting with checklist


Compliance of host reporting with checklist


Compliance of experiment reporting with checklist


Summary of assessing Trypanosoma experiments


Time

Literature survey using text mining

From 1994 to 2014 15,311 full-text articles F-measure ~0.92

Sex and Age matter in biomedical research

* Copy Right Disclaimer *The images posted on this slide are available in various places on the Internet, and believed to be public domain. Copyright Fair Use Act (title 17, U.S. Code). Under Section 107 of the Copyright Act 1976, allowance is made for “fair use” for purposes such as criticism, comment, news reporting, teaching, scholarship, and research. Fair use is a use permitted by copyright statute that might otherwise be infringing.


Our Approach

Sex and age are two important factors widely mis-reported

PubMed Query: ("Mice"[Mesh] AND (mouse[ti] OR mice[ti])) AND (Journal Article[ptyp] NOT Review[ptyp] AND ("1994/01/01"[PDAT] : "2014/12/31"[PDAT]) AND "animals"[MeSH Terms:noexp] AND English[lang]) AND pubmed pmc[sb]

Look for papers doing experiments on mice with full text open access articles

Use text-mining to look for sex and age reporting

Factor results according to disease and biological mechanism area

Do the metrics


Pie-chart (a) showing an overview of the reporting and non-reporting (none) of sex only, age, or both sex and age in a set of 15,311 studies published between 1994 and 2014 by stating the number and percentage of articles in each portion.

The chronological change of the reporting and non-reporting is displayed both in a stacked area plot (b) and a scatter plot after normalisation [per articles/year] (c).

Reporting sex and/or age in mouse-model experiments


Bar-chart (d) showing the number and percentage of articles reporting/not reporting of sex or age. The chronological change of the reporting and non-reporting of sex (e) is displayed in scatter plots after normalisation [per articles/year].

Reporting sex and/or age in mouse-model experiments


The distribution is presented in stacked bar charts that illustrate the percentage of the reporting and non-reporting of sex; stating the number of articles corresponding to each percentage inside the stacks.

Distribution of reporting of the sex in mouse model of a group of diseases


The reporting of sex was assessed for each disease by the topic of research whether genetics (a), immunology (b), physiopathology (c), or therapy (d).

This analysis was performed in the set of 14,225 articles published from 2001, when the US Institute of Medicine reported that sex matters in disease and therapy.

Distribution of reporting of the sex in mouse model of a group of diseases by research approach


The graph shows the reporting in particular diseases. All these diseases that are among the most frequently reported causes of death world-wide or commonly used models.

Distribution of reporting of the sex in mouse model of diseases


Why is reporting often poor?

Space allowed in papers The “expert” function Specificity of checklists Historic lack of awareness of the importance of some

factors Method as a second class citizen in reporting Editors: “A recent survey indicates that almost half of

journal editors who responded (n = 67) consider completeness of reporting to be one of the top three factors when making a publication decision. In the same survey, however, 18% of the editors who were interviewed perceived endorsement of a reporting guideline as burdensome” - Larissa Shamseer, Systematic Reviews 2012, 1:24.


Acknowledgements

Oscar Florez-Vargas did the majority of the work

Andy Brass and Robert Stevens supervise Oscar’s Ph.D.

Sheena Cruickshank collaborates on parasitology

Michael Bramhall works on parasite based mouse models of colitis

George Karystianis2, and Goran Nenadic collaborated on the text-mining


doi: 10.1371/journal.pone.0101131. eCollection 2014

doi: 10.1097/MIB.0000000000000369

Publications