efficient clustering of large est data sets on parallel computers

Efficient Clustering of Large EST Data Sets on Parallel Computers

CECS 694-04 Bioinformatics Journal Club

September 17, 2003

Nucleic Acids Research, 2003, 31(11), 2963-2974Presented by Elizabeth Cha

2

Problem Statement

We are given an EST database from a single species, where multiple EST sequences may belong to the same gene.

We want to find an efficient algorithm to cluster EST sequences, so that all EST sequences in a cluster belong to a single gene. (It’s possible to have more than one cluster for a gene.)

3

Efficient Algorithm Considerations

Memory efficiency to reduce the memory required to linear in the size of input

Computational efficiency without sacrificing the quality of clustering

Reduction of run-time of clustering large EST data sets by parallel processing (e.g. MPI)

4

EST database (dbEST)

Expressed Sequence Tag (EST) representations provide a dynamic view of genome content and expression > 5 million human ESTs > 3.5 million mouse ESTs

Reference information: dbEST

(ncbi.nlm.nih.gov/dbEST/dbEST_summary.html)

5

What is EST?

A unique DNA sequence derived from a cDNA library.

The length of EST is around 200 ~ 500 nucleotides long.

ESTs are generated by sequencing either one of both ends of an expressed gene.

The EST can be mapped, by a combination of genetic mapping procedures, to a unique locus in the genome and serves to identify that gene locus.

6

An overview of the process of protein synthesis

Image adopted by http://ncbi.nlm.nih.gov/About/primer/est.html

7

An overview of how ESTs are generated.

Image adopted from ncbi.nlm.nih.gov/About/primer/est.html

8

Current Problems in dbEST

Imposing size of EST database Low sequence quality Highly similar (but distinct) gene family

members Chimeric cDNA clones Retained introns and alternatively spliced

transcripts Incomplete gene coverage Other limitations

9

Types of alternative splicing

Skipped exons

Retained introns

Alternative donor or acceptor site

Image adopted from Trends in Genetics, 2002, 18(1), 53-57

10

How to solve the problems

Remove the redundancy by clustering ESTs representing the same native transcripts

Current software for clustering ESTs UniGene STACK (Sequence Tag Alignment and

Consensus Knowledgebase) HGI (Human Gene Index) TIGR Assembler CAP3 Phrap

11

Goals of clustering ESTs

Each cluster represents a distinct gene, including all alternative transcript isoforms derived from the same gene (e.g. UniGene).

Each cluster is deemed to represent a distinct mRNA transcript (e.g. CAP3, TIGR Assembler, Phrap).

ESTs and first categorized by their RNA source and are subsequently clustered separately for each source sample (e.g. STACK).

12

Ideas to get evidential gene or transcript

1. Pairwise sequence alignment with dynamic programming algorithm

2. Fast identification of promising pairs with good quality overlap

3. Report pairs based on maximal common substrings

14

PaCE (Parallel Clustering of ESTs)

A software program for EST clustering on parallel computers

2 reasons for this combination enables clustering and assembly of large-scale EST data sets Memory requirement: grows linearly in the size of

input The input size is reduced from the complete set of

ESTs to the size of the biggest cluster

15

EST Clustering

Given:

ESTs drawn from multiple mRNAs

Partition:

The ESTs into clusters such that ESTs from the same gene are put together in a distinct cluster

16

EST Clustering (Cont’d)

Image adopted from the Summer Lecture of Dr. Aluru’s of Iowa State Univ.

17

EST Clustering Algorithm

Initially, treat each EST as a cluster by itself

If two ESTs from two different clusters show significant overlap, merge the clusters

Output the clusters once finished

18

EST Clustering (Cont’d)

Merging Clusters

Successful overlap results in:


19

Determining Overlaps

Compute only lower and upper rectangles Do banded dynamic programming


20

Maximum Common Substring

Given:

a set of strings Find:

Pairs of strings that have a maximal common substring ≥ a threshold φ


21

Organization of PaCE

1. Build a distributed representation of the GST data structure in parallel

2. Use a single processor to handle maintaining and updating the EST clusters

22

Generalized Suffix Tree (GST)

A GST for a set of n sequences is a suffix tree constructed using all suffixes of the n sequences.

23

Basic Concept of Suffix Tree

A suffix tree is a data structure that exposes the internal structure of a string in a deeper way than does the fundamental preprocessing.

24

Definition of Suffix Tree

1. A suffix tree T for an m-character string S is a rooted directed tree with exactly m leaves numbered 1 to m.

2. Each internal node has at least 2 children and each edge is labeled with a nonempty substring of S.

3. No 2 edges out of a node can have edge-labels beginning with the same character.

25

Definition of Suffix Tree (Cont’d)

4. Key feature:

for any leaf i, the concatenation of the edge-labels on the path from the root to leaf i exactly spells out the suffix of S that starts at position i.

26

Ukkonen’s Algorithm to Construct a Suffix Tree

Construct tree I1

(It is just the single edge labeled by character S(1))for i = 1 to m-1 dobegin {phase i+1}

for j = 1 to i+1begin {extension j}Find the end of the path from the root labeled S[j..i] in the current tree.If needed, extend that path by adding character S(i+1).end;

end;

27

Suffix Tree

Construct a suffix tree of sequence gaac

28

Suffix Tree (Cont’d)

Image adopted from article (1999) Nucleic Acids Research, 27, 2369-2376

29

Main idea to use GST data structure


• Maximal Common Substring

30

Parallel Clustering

A master-slave paradigm is used. Master processor:

maintains and updates the clusters Slave processors:

1. Generate pairs as demanded by the master processor

2. Perform pairwise alignments of the pairs dispatched by the master processor

Data structure for maintaining the clusters: union-find algorithm

31

Software availability

PaCE is freely available for non-profit, academic use.

To request source code and executables Contact information : [email protected]

32

Quality Assessment

Benchmark data set:

Arabidopsis thaliana 168,200 ESTs Small genome (114.5 Mb / 125 Mb

total) has been sequenced in year 2000

Reference information:

http://www.arabidopsis.org/info/aboutarabidopsis.html

33

Achievements of PaCE

Reduce the worst-case memory requirement from quadratic to linear

Generate promising pairs in decreasing order of maximal common substring length and cluster the ESTs such that the number of pairwise alignments is reduced by an order of magnitude without affecting the quality of clustering

Reduce the number of duplicates generated for each promising pairs

34

Future Research

Extend PaCE to do assembly and build consensus sequences in parallel

Incorporate quality values available to ESTs as part of input

Ensure quality clustering and assembly

35

System used to implement

IBM xSeries cluster 30 dual-processor nodes 1.26 GHz Intel Pentium III processors connected by Myrinet 2.25 GB memory at each node 512 MB of RAM

36

Quality Assessment of PaCE and CAP3

efficient clustering of large est data sets on parallel computers

Documents