what’s new in work queue
DESCRIPTION
What’s New in Work Queue. Michael Albrecht, University of Notre Dame CCL Workshop, June 2012. Overview. New “Batch Job” Systems Moab / Cluster MPI Queue Work Queue Enhancements Hierarchical Work Queue. Batch Job Systems. Abstraction Layer Library for generic task submission - PowerPoint PPT PresentationTRANSCRIPT
What’s New in Work What’s New in Work QueueQueue
Michael Albrecht, University of Notre Dame
CCL Workshop, June 2012
2
Overview
New “Batch Job” SystemsMoab / ClusterMPI Queue
Work Queue EnhancementsHierarchical Work Queue
3
Batch Job Systems
Abstraction Layer Library for generic task submission Used by Makeflow
4
Moab & other Clusters
Similar to “SGE” Explicitly support Moab scheduler Support other, similar schedulers
Set name, submit and remove commands
moab
5
MPI Queue
Workflow
W WW WW W W W W
W W W W W
6
MPI Queue
Workflow
W W W W W
W W W W F
7
MPI Queue
Enables arbitrary computation on “MPI-only” clusters
Ranks 1-N talk to Rank 0, which acts as a “foreman”
Assumes shared parallel filesystem for cluster
API very similar to Work QueueFully supported by Batch Job/Makeflow
Work Queue
9
Work Queue is Wonderful
Easily harness 100’s-1000’s of cores
Combine multiple resources for one project
Dynamically scale computational resources
PrivateCluster
CampusCondor
Pool
PublicCloud
Provider
SharedSGE
Cluster
Makefile
Makeflow
Local Files and Programs
sge_submit_workers
W
W
W
ssh
WW
WW
W
Wv
W
condor_submit_workers
W
W
W
10
Work Queue has Limits
Bandwidth, file size, and computation length constrain potential number of workers
11
Work Queue has Limits
Natural parallelism of a workflow leaves extra workers idle
T1 T2 T3 T4
T0
T5
???W5W1 W2 W3 W4
12
Work Queue has Limits
Synchronous transfer leaves network resources idle, increases dispatch time
Add more indirection!
“All problems in computer science can be solved by another layer of indirection”
-David Wheeler
“…except for the problem of too many layers of indirection”
-Kevlin Henney
14
Hierarchical Work Queue
Master(Makeflow)
Worker Worker Worker Worker Worker Worker Worker Worker
15
Hierarchical Work Queue
Master(Makeflow)
Worker Worker Worker Worker Worker Worker Worker Worker
Foreman Foreman
16
Hierarchical Work Queue
Master(Makeflow)
Worker Worker Worker Worker Worker Worker Worker Worker
Foreman Foreman
Shared FS
17
Hierarchical Work Queue
Master(Makeflow)
Worker Worker Worker Worker Worker Worker Worker Worker
Foreman ForemanShared FS Shared FS
18
Shared FS
Hierarchical Work Queue
Master(Makeflow)
Worker Worker Worker Worker Worker Worker Worker Worker
Foreman ForemanShared FS Shared FS
Foreman Foreman
19
Hierarchical Work Queue
19
http://www.nd.edu/~cclhttp://www.nd.edu/~ccl
Coming
Soon!