Matthew Cockerill, Co-Founder, Riffyn Incwww.riffyn.com

What can be done to improve reproducibility?

The future of scientific scholarly communication, Royal Society, 5th May 2015

Reproducibility issues begin upstream of publication

• Biomedical research still has more in common with a “blacksmith shop” than a reliable modern process engineered for reliability

• Essential methodological knowledge is implicit and undocumented

• To make things work “spend time in the relevant lab”

• Technology transfer too often fails• If we want published results to be more

reliable, we need to to change the approach to methodology

Reproducibility issues begin upstream of publication

• Biomedical research still has more in common with a “blacksmith shop” than a reliable modern process engineered for reliability

• Essential methodological knowledge is implicit and undocumented

• To make things work “spend time in the relevant lab”

• Technology transfer too often fails• If we want published results to be

more reliable, we need to to change the approach to methodology

In industry, the costs of haziness are clear

Certainly it is a problem that we don’t write down our

processes - people change what they know, not what

necessarily matters. - Head of Process

Development of a biopharma company

When we first transferred our manufacturing process to full-

scale operations, our yields dropped 90%.

- CEO of a diagnostics company

We lost four months tracking-down a problem at the demo

scale. Root cause was a parameter change that a new

employee thought did not matter.

- VP Process Development of an algae company

Below: Process design sent to manufacturing

Industrial R&D seeks to improve processes

Improve

Design

Analyse

Measure

Industrial R&D seeks to improve processes

Improve

Design

Analyse

Measure

Noise and irreproducibility gets in the way

Insights from manufacturing

Quality-oriented approachDesign• Unambiguous CAD files• Manage complexity via

modularity

Standardize & automate • Reduce variability in execution

Continuous improvement• Use analytics to identify and

resolve remaining causes of noise

Change of mindset • Make metrics transparent to

entire organization is focused on quality

Insights from manufacturing

How can this benefit R&D?• It means you can build reliably on your own results, and those of others• But needs to be more flexible – in R&D processes change all the time!

Quality-oriented approachDesign• Unambiguous CAD files• Manage complexity via

modularity

Standardize & automate • Reduce variability in execution

Continuous improvement• Use analytics to identify and

resolve remaining causes of noise

Change of mindset • Make metrics transparent so

entire organization is focused on quality

Examples of application in real world R&D

Reducing noise allows more rapid progress

Reproduced from Gardner TS, Trends in Biotechnology, March 2013, Vol. 31, No. 3

Data from product development at leading biotech firm

Quality improvement cuts error by 6X. R&D productivity doubles overnight.Fe

rmen

tatio

n pr

oces

s yi

eld

Date

trend lines

30% relative error 5% relative error

Analytics to track down root cause of noiseH

ow it

wor

ks

Define process inputs, outputs and critical variables1

Measure inputs and outputs2

Stra

in H

T sc

reen

ing

scor

e

Tray #

Analyse and improve3

How do we generalize to basic research?

New generation of lab software tools

Make experiments more reproducible via:• Cloud-based software• CAD approach to experimental design • Integrated data-stream capture and

analysis• Sharing and versioning of methodology

via new publication outlets with github-like approaches

Parallel from cloud computing

• Traditional ad hoc approach to configuring and managing IT systems unreliable and unscalable

• Configuration management systems now use automated ‘recipes’ which reproducibly deliver a fully-configured virtual server which behaves the same every time 

Reusable workflows are well established in computational science

We need to move this to the lab bench

Taverna Kepler

Galaxy

New approaches are starting to bring similar automation to the lab bench

So what does the future look like?

Standards for structured, interoperable data

Software and tools for lab automation/analytics

Hubs supporting sharing of data and methodology

An open ecosystem for reproducible science

How will we make it happen?

Incentives to drive adoption of new approach

In industry• Measurable improvements to R&D productivity

create powerful financial incentives

In academia • Tools will gain adoption if they make

researchers life easier and more productive• Endorsement and encouragement from funders• Requirement from participating journals for

sharing of experimental process descriptions• Metrics that reward the sharing of reusable

protocols

Cultural shift won’t be easy, but it is needed

Change in the lab requires corresponding change in publishing