ibm_workflow.ppt

IBM T. J. Watson Research Center

1 Aug 2005

A Virtual Data Language and System for

Scientific Workflow Specification and Execution

Yong ZhaoDepartment of Computer Science

University of [email protected]

Ian Foster, Mike Wilde (ANL, UChicago), Jens Voeckler (UChicago)


1 Aug 2005

2

Context

• GriPhyN (Grid Physics Network)– Petascale Data Grid Infrastructure for Data

Intensive Sciences• ATLAS (A Toroidal LHC Apparatus)

• CMS (Compact Muon Solenoid)

• SDSS (Sloan Digital Sky Survey)

• LIGO (Laser Interferometer Gravitational-wave Observatory)

• Collaboration with USC/ISI


1 Aug 2005

3

Overview

• Characteristics of Scientific workflow

• Virtual Data Concept

• Virtual Data Language, System, and Portal

• Scientific Applications

• XDTM (XML Dataset Typing and Mapping)

• Dataset Iteration

• Summary


1 Aug 2005

4

Scientific Workflow

• Large amount of data – Petabytes

• Long running – weeks to months

• Heterogeneous distributed resources– Diverse computation environments– Various analysis tools– Data storages, formats, transport protocols

• Community-wide collaboration

• Dynamic sharing relations and policies


1 Aug 2005

5

CMS Grid Hierarchy

Tier2 Center

Online System

CERN Computer Center > 20

TIPS

USA CenterFrance Center

Italy Center UK Center

InstituteInstituteInstitute

Workstations,other portals

100MB~1.5GB/sec

2.5-10 Gbits/sec

0.1-1 Gbits/sec

Bunch crossing per 25 ns100 triggers per second~1 MByte per event

Physics data cache

10 ~ 40 Gbits/sec

Tier2 CenterTier2 Center

0.6-2.5 Gbits/sec

Tier 0

Tier 1

Tier 3

Tier 4

Experiment2500 Physists, 40 countries

10s of Petabytes/Yr by 2008

InstituteInstituteInstituteInstituteInstituteInstitute

Tier 2


1 Aug 2005

6

SDSS Pipeline


1 Aug 2005

7

Challenges for Scientists

• Finding needle in a haystack– Gain control over data– Focus on science, rather than execution details– E-Logbook/workspace

• Transition from try-outs to production– Compose, modify workflow with ease– Partial results, steering

• Reproducibility, validability, audit-trail• Improve Throughput


1 Aug 2005

8

Virtual Data Concept (1)

• Motivated by next generation data-intensive applications– Enormous quantities of data, petabyte-scale– The need to discover, access, explore, and analyze

diverse distributed data sources– The need to organize, archive, schedule, and

explain scientific workflows– The need to share data products and the resources

needed to produce and store them– Usability and productivity


1 Aug 2005

9


• Capture and manage information about relationships among– Data (of distributed locations and widely varying

representations)– Programs (& their inputs, outputs, prerequisites, constraints)– Computations (& execution environments)

• Apply this information to, e.g.– Discovery: data and program discovery– Explanation: provenance (data reproduction and validation) – Workflow management: structured paradigm for organizing,

locating, specifying and requesting data– Planning and scheduling– Performance optimization


1 Aug 2005

10


• Location Transparency– Data are requested without knowledge of the physical data

locations.

– Replica Location Service – Replica Selection Service [PDCS03]

• Materialization Transparency– Data are requested without regard to whether they already

exist or must be computed.

– Recipes for data derivation

• Physical Representation Transparency


1 Aug 2005

11

Virtual Data Grid for Data-Intensive Sciences

GridOperations

simulation data

discovery

ScienceReview

Data Grid

storageelement

replica locationservice

storageelement

storageelement

Dat

aT

ran

spo

rt Sto

rage

Reso

urce

Mg

mt

virtualdata

catalogvirtual data

index

virtualdata

catalog

virtualdata

catalog

Computing Grid

workflowplanner

request plannerworkflowexecutor

(DAGman)

request executor(Condor-G,

GRAM)

requestpredictor

(Prophesy)

Grid Monitor

ProductionManager

Researcher

planning

discovery

com

po

sition

sim

ula

tio

n

anal

ysis

sharing

raw d

ata

detector

derivatio

n


1 Aug 2005

12

Virtual Data System

• Represent, query and automate data derivation [SSDBM02].

Data Grid Resources(distributed execution

and data management)

VDL Interpreter(manipulate derivations

and transformations)

Virtual Data Catalog(implements Virtual Data Schema)

Virtual DataApplications

Virtual Data Language(definition and query)

Task Graphs(compute and data

movement tasks, withdependencies)

Virtual Data System

GriPhyN VDT

(Replica Catalog,Condor, DAGMan,MyProxy,Globus Toolkit)


1 Aug 2005

13

VDS Flow of Work


1 Aug 2005

14

Virtual Data Language

• Definitions - declarative specification of– Transformation (TR) :

• Function definition – name and formal parameters– Derivation (DV) :

• Function call – actual parameters and datasets

• Provenance– Invocation

• Metadata– Annotations about definitions

• Query– All the above


1 Aug 2005

15

VDL Schema (1)


1 Aug 2005

16

VDL Schema (2)


1 Aug 2005

17

Compound Transformation

TR diamond( inout fa, inout fb, inout fc, output fd, none p1, none p2 ){ call generate(

f=(output)fa, p1=p1 );

call process( f1=(input)fa, f2=(output)fb, name="LEFT", p2=p2 );

call process( f1=(input)fa, f2=(output)fc, name="RIGHT", p2=p2 );

call combine( f1=(input)fb, f2=(input)fc, f3=(output)fd );

}

G

P

C

P

fd

fa

fb fc


1 Aug 2005

18

Abstract Workflow

• Specified in DAX (DAG XML description)

• Workflow activities independent of Grid resources

• Allows sharing of abstract descriptions

• Graph editing/rewriting, workflow refinement

• Patterns compound workflow


1 Aug 2005

19

Concrete Workflow

• Concretize abstract workflow into executable form– Logical datasets to physical (RLS)

– Logical transformation to physical (TC)

– Make vs. Build

– Grid site selection (MDS)

– Data movement

– Data publication Late binding, just-in-time scheduling

– Policy

– Optimization


1 Aug 2005

20

Workflow Execution

• Condor-G and DAGMan– Logging, job status monitoring, persistence, fault recovery– Checkpointing, migration

• Planner– Shell Planner:

• shell scripts– Pegasus (ISI) :

• Condor DAG and submit files• Partitioning, clumping (group multiple small jobs)

– Euryale• Condor DAG and submit files • Template-based, one on one mapping• Just-in-time scheduling• Throttling

– Others


1 Aug 2005

21

Sloan Digital Sky Survey

• Map one-quarter of the entire sky

• Determine the positions and absolute brightness of more than 100 million celestial objects.

• Measure the distance to a million of the nearest galaxies, and to 100,000 quasars.

• 40 terabyte of data.


1 Aug 2005

22

The 2.5-Meter Telescope

• Located at Apache Point Observatory, Sunspot, New Mexico, 9200 feet above sea level. The night sky is among the darkest in the united states.


1 Aug 2005

23

SDSS Cluster Finding

• MaxBCG: Maximum likelihood determination of the Brightest Cluster Galaxy.

• Galaxies in a cluster follow certain patterns.• Walk through 5-D space: RA, Dec, g-r, r-i, i

Sky Area 7000 square degree

Storage 1540 GB (raw)

Computation 7000 CPU hours

(PIII 500MHZ, 1G RAM)


1 Aug 2005

24

Virtual Data in SDSS Galaxy Cluster Analysis

1

10

100

1000

10000

100000

1 10 100

Num

ber

of C

lust

ers

Number of Galaxies

Galaxy clustersize distribution

DAGSloan Data

Collaboration with Jim Annis, Steve Kent at Fermilab [SC02]


1 Aug 2005

25

Grid Resources Involved

• More than 7 stripes, 14,000 fields processed, 60,000 clustered identified.

University of Wisconsinat Milwaukee

296 CPUs @ 1GHz100 Mb/s Ethernet1000 GB disk

40 CPUs @ 1 Ghz100 Mb/s Ethernet80 GB disk

110 CPUs @ 300- 450 Mhz10Mb/s and 100 Mb/s Ethernet360GB disk

University of Chicago University of Wisconsinat Madison

WAN600 Mb/s

50 Mb/s

400 CPUs @ 1GHz10 Mb/s and 100 Mb/s Ethernet~500 GB disk

University ofFlorida

100 Mb/s600Mb/s


1 Aug 2005

26

Execution Log for 1200 Fields


1 Aug 2005

27

mass = 200decay = WWstability = 1event = 8

mass = 200decay = WWstability = 1plot = 1

mass = 200decay = WWplot = 1

mass = 200decay = WWevent = 8

mass = 200decay = WWstability = 1

mass = 200decay = WWstability = 3

mass = 200

mass = 200decay = WW

mass = 200decay = ZZ

mass = 200decay = bb

mass = 200plot = 1

mass = 200event = 8

A virtual space of simulated data is created for futureuse by scientists...

Work withRick Cavanaugh andDimitri Bourilkov, et al.University of Florida [CHEP2003]

Virtual Data in CMS Analysis


1 Aug 2005

28

Virtual Data in Genome Analysis


1 Aug 2005

29

Chiron Virtual Data Portal

• Integrated, interactive problem solving environment [MGC2004] [CCPE2005]

• Provides both user-level and service-level functionalities for– Virtual Data Discovery, Composition, and Integration

– User Management

– Job Submission

– Workflow Construction, Visualization, Execution

– Grid Resource Management

• Virtual Data Educator


1 Aug 2005

30

Chiron Virtual Data Portal

Web Browser

Tomcat Web Server

Java ServerPages

Virtual Data API & Services

GrapherLocal

PlannerGrid

Planner

Condor-GDAGMan

MDS

RLS

Grid Site

SE/GridFTP

CE/Condor

Grid Site

SE/GridFTP

CE/PBS

Grid Site

SE/GridFTP

CE/Other

VirtualData

Catalog

Chiron System Architecture


1 Aug 2005

31

Virtual Data in fMRI Analysis

3a.h

align_warp/1

3a.i

3a.s.h

softmean/9

3a.s.i

3a.w

reslice/2

4a.h

align_warp/3

4a.i

4a.s.h 4a.s.i

4a.w

reslice/4

5a.h

align_warp/5

5a.i

5a.s.h 5a.s.i

5a.w

reslice/6

6a.h

align_warp/7

6a.i

6a.s.h 6a.s.i

6a.w

reslice/8

ref.h ref.i

atlas.h atlas.i

slicer/10 slicer/12 slicer/14

atlas_x.jpg

atlas_x.ppm

convert/11

atlas_y.jpg

atlas_y.ppm

convert/13

atlas_z.jpg

atlas_z.ppm

convert/15

Collaboration with Jed Dobson, Dartmouth


1 Aug 2005

32

• Using Grid virtual data tools and methods to transform and enrich science learning and education

• It’s an experiment to give students the means to:– discover and apply datasets, algorithms, and data analysis methods– collaborate by developing new ones and sharing results and

observations– learn data analysis methods that will ready and excite them for a

scientific career– and in later steps, may actually use the Grid!

• Educational researchers evaluate the effectiveness of such an endeavor

• Grid specialists explore interface designs that enhance accessibility to virtual data and Grid resources

QuarkNet/Trillium Collaboration


1 Aug 2005

33

Virtual Data in Science Education

Collaboration with Marge Bardeen, Tom Jordan, Liz Quigg, Eric Gilbert, Paul Nepywoda, Fermilab [CCGRID2005] [FGCS 2005]


1 Aug 2005

34

Quarknet Shower Study

minime.combined

Sort/4

minime.event

EventPlot/7

minime.event.plot.png minime.event.plot

minime.events

EventChoice/6

minime.sorted

Search/5

minime1.thresh

Combine/3

minime1

ThresholdTimes/1

minime2.thresh

minime2

ThresholdTimes/2

minimeloc


1 Aug 2005

35

Search by Metadata


1 Aug 2005

36

Web Service Integration

• WS described/imported as TR

• Dynamic invocation

• XSLT as glue


1 Aug 2005

37

“Messy” Scientific Data

• Heterogeneous storage format– Logically identical dataset can be stored in

• Textual File (e.g. CSV)

• Binary format (e.g. CDF)

• Spreadsheet

• Database table

• Metadata encoded in directory and file names– A fMRI volume is composed of an image file and a header

file with the same prefix.

• Format dependency hinders workflow reuse[SIGMOD-SWF 2005]


1 Aug 2005

38

Typing System

• Separate logical structure from physical representation

• Type checking, conversion

• Discovery by types

• Workflow composition

• Dynamic discovery

• Dataset selection and iteration


1 Aug 2005

39

Virtual Data Schema

Transformation

type-signature=prog1( in type1 X, out type2 Y)

Dataset

name=footype=type2

Derivation

type-signature=prog1( in type1 fnn, out type2 foo)

instanceof

Invocation

when=10amtime=20 secslocn=U.Chicago

invocationof

Reads/writes/creates/deletes

Replica

locn=U.Chicagophysicalreplica of

Reads/writes/creates/deletes

Type

name=type2repres=<...>

Containsarguments of

instanceof

Metadata

describes describes

[CIDR 2003]


1 Aug 2005

40

XDTM

• XML Dataset Typing and Mapping• Logical structure described by XML Schema

– Primitive scalar types: int, float, string, date …– Complex types

• Mapping descriptor– How dataset elements are mapped to physical

representations– External parameters (e. g. location)

• XPath for dataset selectionJoint work with Luc Moreau, Southampton, UK [EGC 2005]


1 Aug 2005

41

XDTM Based VDL

type Image Any;type Header Any;type Volume {

Image img; Header hdr;

}type Run { Volume v[ ]; }type Anat Volume;type Subject {

Anat anat; Run run [ ]; Run snrun [ ];

}type Warp Any;type NormAnat {

Anat aVol; Warp aWarp; Volume nHires;

}

airsn_subject( // Main function on “Subject” Subject s, Volume atlas, Air ashrink, Air fshrink ) { NormAnat a = anatomical(s.anat, atlas, ashrink); Run r, snr; Foreach r in s.run {

snr = functional ( r, a, fshrink );s.snrun[ name(r) ] = snr;

}}

Part of fMRI AIRSN (Spatial Normalization) Workflow


1 Aug 2005

42

Dataset Iteration

• Common in scientific applications

• Apply a transformation to each sub-elementreorientRun

reorientRun

reslice_warpRun

random_select

alignlinearRun

resliceRun

softmean

alignlinear

combinewarp

strictmean

gsmoothRun

binarize

reorient/01

reorient/02

reslice_warp/22

alignlinear/03 alignlinear/07alignlinear/11

reorient/05

reorient/06

reslice_warp/23

reorient/09

reorient/10

reslice_warp/24

reorient/25

reorient/51

reslice_warp/26

reorient/27

reorient/52

reslice_warp/28

reorient/29

reorient/53

reslice_warp/30

reorient/31

reorient/54

reslice_warp/32

reorient/33

reorient/55

reslice_warp/34

reorient/35

reorient/56

reslice_warp/36

reorient/37

reorient/57

reslice_warp/38

reslice/04 reslice/08reslice/12

gsmooth/41

strictmean/39

gsmooth/42gsmooth/43gsmooth/44 gsmooth/45 gsmooth/46 gsmooth/47 gsmooth/48 gsmooth/49 gsmooth/50

softmean/13

alignlinear/17

combinewarp/21

binarize/40

reorient

reorient

alignlinear

reslice

softmean

alignlinear

combine_warp

reslice_warp

strictmean

binarize

gsmooth


1 Aug 2005

43

Summary

• Concept– Location, existence, representation transparency

• Technology– Virtual data language and system

• Application– Physics, biology, neuroscience, education

• Work in progress– Type system, XDTM


1 Aug 2005

44

For more information

• GriPhyN– http://www.griphyn.org

• GriPhyN documents– http://www.griphyn.org/documents

• VDS– http://www.griphyn.org/vds

• Publications– http://www.cs.uchicago.edu/~yongzh

ibm_workflow.ppt

Technology

data petabytes

data focus

data planning

data products

data derivationssdbm02

virtual data grid

data grid resources

physical data locations