meta’omic analysis with metaphlan, humann, and lefse curtis huttenhower 08-08-13 harvard school of...

Meta’omic Analysis withMetaPhlAn, HUMAnN, and LEfSe

Curtis Huttenhower

08-08-13Harvard School of Public HealthDepartment of Biostatistics

2

Some setup notes

• Slides with green titles or text include instructions not needed today, but useful for your own analyses

• Keep an eye out for red warnings of particular importance

• Command lines and program/file names appear in a monospaced font.

3

Getting some HMP data

• Go to http://hmpdacc.org Click “Get Data”

4

Getting some HMP data

• Check out what’s available

Click “HMIWGS”

5

Getting some HMP data

• Check out what’s available

Click on your favorite body site

6

Getting some HMP data

• Check out what’s availableDon’t click on anything!

7

Getting some (prepped) HMP data

• Connect to the server instead and run:– for S in `ls /class/stamps-shared/chuttenh/7*.fasta`;

do ln -s $S; done

• These are subsamples of six HMP files:– SRS014459.tar.bz2 763577454-SRS014459-Stool.fasta– SRS014464.tar.bz2 763577454-SRS014464-Anterior_nares.fasta– SRS014470.tar.bz2 763577454-SRS014470-Tongue_dorsum.fasta– SRS014472.tar.bz2 763577454-SRS014472-Buccal_mucosa.fasta– SRS014476.tar.bz2 763577454-SRS014476-Supragingival_plaque.fasta– SRS014494.tar.bz2 763577454-SRS014494-Posterior_fornix.fasta

• All six shotgunned body sites from– One subject, first visit– Subsampled to 20,000 reads

8

Who’s there: MetaPhlAn

Gene X

X is a core gene for clade Y X is a unique marker gene for clade Y

9

Who’s there: MetaPhlAn

• Go to http://huttenhower.sph.harvard.edu/metaphlan

Scroll down

10

Who’s there: MetaPhlAn

• You could download MetaPhlAn by clicking here

11

Who’s there: MetaPhlAn

• But don’t! Instead, we’ve downloaded MetaPhlAn already for you by clicking here

12

Who’s there: MetaPhlAn

• And saving just the file metaphlan.py in stamps-software/bin

13

Who’s there: MetaPhlAn

• You could download the bowtie2 database here

14

From the command line...

• But don’t! Instead, get it by:– for S in `ls /class/stamps-shared/chuttenh/*.bt2`; do ln -s $S; done

• To see what you can do, run:– module load stamps– metaphlan.py -h | less– Use the arrow keys to move up and down,

q to quit back to the prompt

15

Who’s there: MetaPhlAn

16

Who’s there: MetaPhlAn

• For future reference: these extra options aren’t necessary if you download the whole “default” MetaPhlAn package– metaphlan.py your_input.fasta > your_output.txt

• To launch your first analysis, run:– metaphlan.py --bowtie2db mpa 763577454-SRS014459-Stool.fasta > 763577454-SRS014459-Stool.txt

17

Who’s there: MetaPhlAn

• What did you just do?• Two new output files:

– 763577454-SRS014459-Stool.fasta.bowtie2out.txt

• Contains a mapping of reads to MetaPhlAn markers– 763577454-SRS014459-Stool.txt

• Contains taxonomic abundances as percentages

18

Who’s there: MetaPhlAn

• less 763577454-SRS014459-Stool.fasta.bowtie2out.txt

19

Who’s there: MetaPhlAn

• less 763577454-SRS014459-Stool.txt

20

Who’s there: MetaPhlAn

• Now finish the job:– metaphlan.py --bowtie2db mpa763577454-SRS014464-Anterior_nares.fasta > 763577454-SRS014464-Anterior_nares.txt

– ...

• Note that you can use the up arrow key to make your life easier!

21

Who’s there: MetaPhlAn

• Let’s make a single table containing all six samples:– mkdir tmp– mv *.bowtie2out.txt tmp– merge_tables.py -l -d *.txt | zero.py > 763577454.tsv

• You can look at this file using less– Note 1: The arguments less -x4 -S will help– Note 2: You can set this “permanently” using export LESS="-x4 -S"

22

Who’s there: MetaPhlAn

• But it’s easier using MeV; go to http://www.tm4.org/mev.html

Click “Download”

23

An interlude: MeV

• Don’t forget to transfer your 763577454.tsv file locally for viewing using scp

• Unzip, launch MeV, and select File/Load data

24

An interlude: MeV

• Click “Browse” to your TSV file, then– Tell MeV it’s a two-color array– Uncheck “Load annotation”– Click on the upper-leftmost data value

25

An interlude: MeV

• “Load” your data, then make is visible by:– Display/Set Color Scale Limits– Choose Single Gradient, min 0, max 10

26

An interlude: MeV

• Finally, to play around a bit:– Display/Set Element Size/whatever you’d like– Clustering/Hierarchical Clustering– Optimize both gene and sample order– And select Manhattan Distance (imperfect!)

27

An interlude: MeV

• If you’d like, you can– Display/Sample-Column Labels/Abbr. Names

28

An interlude: MeV

• MeV is a tool; imperfect, but convenient– You should likely include just “leaf” nodes

• Species, whose names start include “s__”• You can filter your file using:cat 763577454.tsv | grep -E '(Stool)|(s__)' > 763577454_species.tsv

– You can, but might not want to, z-score normalize• Adjust Data/Gene-Row Adjustments/Normalize Genes-Rows

• Many other tools built in – experiment!

29

What they’re doing: HUMAnN

• Back to the task at hand; you could download HUMAnN at: http://huttenhower.sph.harvard.edu/humann

Click here

30

What they’re doing: HUMAnN

• ...but instead we’ve already downloaded it• Expand HUMAnN (no install!)

– tar -xzf /class/stamps-shared/chuttenh/sources/humann-0.98.tar.gz

• Set up a link to the KEGG reference DB:– ln -s /class/stamps-shared/chuttenh/kegg.reduced.udb

• And although you would normally download USEARCH from here:– http://www.drive5.com/usearch/download.html

We’re going to use it preinstalled instead

31

What they’re doing: HUMAnN

• If we weren’t all running this, you’d need to:– Get KEGG – used to be free, now it’s not!

• Fortunately, we have a HUMAnN-compatible distributable version; contact me...

– Index it for USEARCH:• usearch6 -makeudb_usearch kegg.reduced.fasta-output kegg.reduced.udb

• This takes a minute or two, so we’ve precomputed it; thus, forge ahead...

32

What they’re doing: HUMAnN

• Did you notice that we didn’t QC our data at all?– MetaPhlAn is very robust to junk sequence– HUMAnN is pretty robust, but not quite as much

• We’ve already run a standard metagenomic QC:– Quality trim by removing bad bases (typically Q ~15)– Length filter to remove short sequences (typically <75%)

33

What they’re doing: HUMAnN

• Must start from FASTQ files to do this• Quality trim by removing bad bases:

– TrimBWAstyle.py < your_data.fastq > your_trimmed_data.fastq

• Length filter by removing short sequences:– 75% of original length is standard (thus 75nt from 100nt reads)– remove_bad_seqs.py 75 < your_trimmed_data.fastq >

your_filtered_data.fastq

• Now convert your FASTQ to a FASTA:– fastq2fasta.py < your_filtered_data.fastq >

your_filtered_data.fasta

• Some final caveats:– If you’re using paired end reads, match filters!– See my course homeworks at http://huttenhower.sph.harvard.edu/bio508– Aren’t you glad you’re not doing this today?

34

What they’re doing: HUMAnN

• Enter the humann directory– cd humann-0.98

• Run your first translated BLAST search:– usearch6.0.192_i86linux32 -usearch_local ../763577454-SRS014459-Stool.fasta -db ../kegg.reduced.udb -id 0.8 -blast6out input/763577454-SRS014459-Stool.txt

• What did you just do?– less input/763577454-SRS014459-Stool.txt– Recall BLAST’s tab-delimited output headers:

• qseqid sseqid pident length mismatch gapopen qstart qend sstart send evalue bitscore

• Rinse and repeat for the remaining samples

35

What they’re doing: HUMAnN

• Normally you’d need to install SCons from:– http://www.scons.org

• Instead, we’ll use it preinstalled as well, so...

GO!– /class/stamps-software/scons/bin/scons

• You should see a bunch of text scroll by– Note: you can run scons -j8 to parallelize tasks

36

What they’re doing: HUMAnN

• After a minute or two, you should see:

37

What they’re doing: HUMAnN

• This has created four main files:– Two each for pathways (big) and modules (small)– Two each for coverage and relative abundance

• Each is tab-delimited text with one column per sample

• All four are in the output directory:– output/04a-hit-keg-mpt-cop-nul-nve-nve-xpe.txt

• Coverage (a) of pathways (t)

– output/04a-hit-keg-mpm-cop-nul-nve-nve-xpe.txt• Coverage (a) of modules (m)

– output/04b-hit-keg-mpt-cop-nul-nve-nve.txt• Abundance (b) of pathways (t)

– output/04b-hit-keg-mpm-cop-nul-nve-nve.txt• Abundance (b) of modules (m)

• I almost always just use 04b-mpm (module abundances)

38

What they’re doing: HUMAnN

• Let’s take a look:– less output/04b-hit-keg-mpm-cop-nul-nve-nve.txt

39

What they’re doing: HUMAnN

• That’s ugly; it gets much better in Excel– Note: this is very sparse since we’re using a small subset of KEGG– Note: the mock community demo data is included on the right

40

What they’re doing: HUMAnN

• And there’s nothing stopping us from using MeV– Or R, or QIIME, or anything that’ll read tab-delimited text

41

What matters: LEfSe

• All of these analyses give you tables– 16S OTUs– MetaPhlAn Species– HUMAnN Modules (or pathways or genes)

• If you know something about your samples...– Case/control– Different habitats– Different time points

• How can you identify features that change?

42

What matters: LEfSe

• Let’s get all of the HMP species data:http://hmpdacc.org/resources/data_browser.php

Click “HMSMCP”

43

What matters: LEfSe

• Download the MetaPhlAn table for all 700 samples

Right click this irritatingly tiny icon

44

Downloading from the command line

• Instead of saving this, download it by:– Right-click to copy the URL– Run wget <paste URL here>– Note: curl –O <URL> works just as well

45

What matters: LEfSe

• Make sure this file is in your home directory, and expand it:– bunzip2 HMP.ab.txt.bz2

• Look at the result– less HMP.ab.txt

• IMPORTANT!!!– This file’s too big to analyze directly today– ln -s /class/stamps-shared/chuttenh/HMP.ab.filtered.txt

• This is great – tons of data, but no metadata– HUMAnN to the rescue– From the humann-0.98 directory:

python src/metadata.py input/hmp_metadata.dat < ../HMP.ab.filtered.txt > ../HMP.ab.filtered.metadata.tsv

• NOW take a look again

46

What matters: LEfSe

• Let’s modify this file to be LEfSe-compatible• Open it up in Excel

47

What matters: LEfSe

• Delete all of the metadata rows except:– RANDSID and STSite– Save it as tab-delimited text: HMP.ab.filtered.metadata.txt

48

What matters: LEfSe

• Visit LEfSe at: http://huttenhower.sph.harvard.edu/lefse

Click here

49

What matters: LEfSe

• Then upload your formatted table– After you upload, wait for the progress meter to turn green!

1. Click here

2. Then here

3. Then here

4. Then watch here

50

What matters: LEfSe

• Then tell LEfSe about your metadata:

1. Click here

2. Then select STSite

3. Then select RANDSID

4. Then here

51

What matters: LEfSe

• Then select LDA=4, “One-against-all,” and run LEfSe!– You can change other default statistical parameters if desired

1. Click here

4. Then GO!

3. Then here(finds differences in at

least one condition rather than in all conditions)

2. Then here(finds only very

extreme differences)

52

What matters: LEfSe

• You can plot the results as a bar plot– Again, lots of graphical parameters to modify if desired

1. Click here

2. Then here

53

What matters: LEfSe

• In Galaxy, view a result by clicking on its “eye” Click here

54

What matters: LEfSe

55

What matters: LEfSe

• You can plot the results as a cladogram– Lots and lots of graphical parameters to modify if desired

1. Click here

2. Then here

56

What matters: LEfSe

57

What matters: LEfSe

• Finally, you can see the raw data for individual biomarkers– These are generated as a zip file of individual plots

1. Click here

2. Then here

58

What matters: LEfSe

• In Galaxy, download a result by clicking on its “disk”

Click here

Then here

59

What matters: LEfSe

Strep. mitis

Veillonellaceae

Actinobacteria

60

Summary

• MetaPhlAn– Raw metagenomic reads in– Tab-delimited species relative abundances out

• HUMAnN– Quality-controlled metagenomic reads in– Tab-delimited gene, module, and pathway

relative abundances out• LEfSe

– Tab-delimited, stratified relative abundances in– Significantly differentially abundant features out

Ramnik XavierHarry SokolDan Knights

Moran Yassour

Thanks!

61

Nicola Segata Levi Waldron

Human Microbiome ProjectOwen WhiteJoe PetrosinoGeorge WeinstockKaren NelsonLita ProctorDirk Gevers

Kat Huang

Bruce Birren Mark DalyDoyle Ward Ashlee Earl

http://huttenhower.sph.harvard.edu

Joseph Moon

Felix Wong

Tim TickleXochi Morgan

DanielaBoernigen

Rob KnightJesse ZaneveldGreg Caporaso

Mark SilverbergBoyko Kabakchiev

Andrea Tyler

EmmaSchwager

Jim Kaminski

Brian Palmer

Eric Franzosa

Boyu Ren

Ren Lu Koji Yasuda

Sahar AbubuckerBrandi Cantarel

Alyx SchubertMathangi Thiagarajan

Beltran Rodriguez-MuellerErica SodergrenAnthony FodorMarty Blaser

Jacques RavelPat Schloss

Makedonka MitrevaYuzhen YeMihai Pop

Larry ForneyBarbara Methe

Jacques Izard

Katherine Lemon

Wendy GarrettMichelle Rooks

Bruce SandsRuth Ley

Omry Koren

Rob BeikoMorgan Langille

Jeroen RaesKaroline Faust

Interested? We’re recruiting just about everything!