Thoughts on reproducibility

atul.butte@ucsf.edu @atulbutte

Atul Butte, MD, PhDDirector, Institute for Computational

Health SciencesUniversity of California, San Francisco

Conflicts of Interest• Scientific founder and

advisory board membership– Genstruct– NuMedii– Personalis– Carmenta

• Honoraria for talks– Lilly– Pfizer– Siemens– Bristol Myers Squibb– AstraZeneca– Roche– Genentech– Warburg Pincus

• Past or present consultancy– Lilly– Johnson and Johnson– Roche– NuMedii– Genstruct– Tercica

– Ecoeos– Helix– Ansh Labs– Prevendia– Samsung– Assay Depot– Regeneron– Verinata– Pathway Diagnostics– Geisinger Health– Covance– Wilson Sonsini Goodrich & Rosati – 10X Genomics– Medgenics– GNS Healthcare– Gerson Lehman Group– Coatue Management

• Corporate Relationships– Northrop Grumman– Aptalis– Allergan– Astellas

– Thomson Reuters– Intel– SAP– SV Angel– Progenity– Illumina

• Speakers’ bureau– None

• Companies started by students– Carmenta– Serendipity– NuMedii– Stimulomics– NunaHealth– Praedicat– MyTime– Flipora

bit.ly/genedata

bit.ly/geobrca

bit.ly/immport

The next big open data: clinical trials data

immport.org

Public microarray data led to• Novel receptor in type 2 diabetes• Mouse knockout model• Human serum companion diagnostic• Prototype drug for type 2 diabetes

Integrating publicly-available data yielded CD44 as a new targetable receptor for type 2 diabetes

Use contract research services for the parts of research you don’t do well

Share successful, failed, and so-so data• Rituximab in ANCA-Associated Vasculitis (RAVE) trial of new approach to

the induction of remission• But even though rituximab was found to be non-inferior than

cyclophosphamide, which drug is the right one to use?

The five lessons I’ve learned in the past 18 years

• Be careful asking for standard procedures, or you will risk over-fitting those standard models

• Make data and computational methods publicly available, no exceptions for clinical or clinical trials data

• Wisdom of the crowd approach: trust what you see in common across experiments

• Outsource the problematic parts to experts or companies• Failed, successful, and so-so studies should all be shared