job submission condor, globus , java cog kit
DESCRIPTION
Job Submission Condor, Globus , Java CoG Kit. Young Suk Moon. Workflow. The computerized facilitation or automation of a business process, in whole or part. W i T = ( G r T , G s T , Q u T , W m T ) W i = Workflow instantiation G r = Grid resources G s = Grid services - PowerPoint PPT PresentationTRANSCRIPT
Job SubmissionCondor, Globus, Java CoG Kit
Young Suk Moon
Workflow
• The computerized facilitation or automation of a business process, in whole or part.
• WiT = ( Gr
T, GsT, Qu
T, WmT )
– Wi = Workflow instantiation
– Gr = Grid resources
– Gs = Grid services
– Qu = Quality expectations from the user
– Wm = Workflow model.
http://grid.rit.edu 2
von Laszewski, G. (2006). Java CoG Kit Workflow Concepts. Journal of Grid Computing.
Condor
• High throughput computing system– Job management– Scheduling policy– Priority scheme– Resource monitoring / management
• Checkpointing– Fault tolerance– Migration
3http://grid.rit.edu
Condor
• Kernel architecture
Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356.
4http://grid.rit.edu
Condor
• Job submission in a condor pool
Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356.
5http://grid.rit.edu
Condor
• Gateway flocking
Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356.
6http://grid.rit.edu
Condor-G
• Interacting with GRAM• Disadvantage – the agent does not know
which resource is available in the foreign system
Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356.
7http://grid.rit.edu
Condor – Gliding In
• Step 1 – submit Condor servers as jobs
Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356.
8http://grid.rit.edu
Condor – Gliding In
• Step 2 – submitted servers form an ad-hoc Condor pool with the matchmaker
Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356.
9http://grid.rit.edu
Condor – Gliding In
• Step 3 – use the Condor pool as normal
Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356.
10http://grid.rit.edu
Condor - Problem Solvers• Master-Worker
– Master process assigns jobs to worker processes
– Suitable for a large problem with no dependency
• DAGMan (The Directed Acyclic Graph Manager)– For multiple jobs with
dependancy
Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356.
A Directed Acyclic Graph
11http://grid.rit.edu
Condor Universes
• Define execution environments– Standard (default) : checkpointing & remote
system calls– Vanila : run programs which cannot be re-linked– PVM : run Parallel Virtual Machine programs– MPI : run MPICH programs– Globus : use a Globus system– Java : run on JVM– Scheduler : executes jobs without waiting to be
matched to resources. Used for DAGMan
12http://grid.rit.edu
Condor
• Example submit description file– submit.helloExecutable = helloUniverse = standard Output = hello.out Log = hello.logQueue
• To submit the job– > condor_submit submit.hello
13http://grid.rit.edu
Globus GRAM
• Grid Resource Allocation and Management• Program execution requirements
– State– User executables– Staging of input and output– Streaming output– Control– Scheduler– Monitoring
14http://grid.rit.edu
Globus GRAM
• State transition diagram for GRAM jobs
Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4.
15http://grid.rit.edu
Globus GRAM
• Client APIs– Four portTypes: Managed Job Factory, Managed
Job, Delegation Factory, Delegation
Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4
operations
interfaces
resources
16http://grid.rit.edu
Globus GRAM
• GT4 Container– Hosts Web Services
Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4
17http://grid.rit.edu
Globus GRAM
• GRAM implementation structure
Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4
18http://grid.rit.edu
Globus GRAM
• Job submission% globusrun-ws -submit -job-command /bin/program argSubmitting job…DoneJob ID: uuid:c51fe35a-4fa3-11d9-9cfc-000874404099Termination time: 12/17/2004 20:47 GMTCurrent job state: ActiveCurrent job state: CleanUpCurrent job state: DoneDestroying job...Done.
Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4
19http://grid.rit.edu
Globus GRAM• Job submission with job description files
% globusrun-ws -submit -job-description-file touch.xml -factory-epr-file gram.epr#touch.xml <job> <executable>/bin/program</executable> <argument>arg</argument> </job>#gram.epr <factoryEndpoint xmlns:gram="http://www.globus.org/namespaces/2004/10/gram/job"
xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/03/addressing"> <wsa:Address> https://viz-login.isi.edu:9000/wsrf/services/ManagedJobFactoryService </wsa:Address> <wsa:ReferenceProperties> <gram:ResourceID>PBS</gram:ResourceID> </wsa:ReferenceProperties> </factoryEndpoint>
Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4
20http://grid.rit.edu
Java CoG Kit
• Java Commodity Grid Toolkit• Maps Grid functionality into a commodity
framework (Java)• Extends the use of Globus• Purpose
– To use Grid services more efficiently by using / developing Java classes
21http://grid.rit.edu
Java CoG Kit• Workflow management layer
– It supports multiple Grid middleware
Grid Middleware + CoG Kit providersGT2, SSH, GT4 WS-RF, WebDAV
Job submission file transfer & authenticationWorkflow Abstractions, Queue/Set Abstractions
Task & workflow managementKarajan (GridAnt, ant)
Workflow monitoringWorkflow Portlet & visualizer/Editor
ApplicationNano Materials, Bio-Informatics, etc.
22http://grid.rit.edu
Java CoG Kit - Job Submission (GUI)
http://www.mcs.anl.gov/~laszewsk/www/www-cogkits-org/talks/Commodity_Grid_Kits_Cog.pdf23http://grid.rit.edu
Java CoG Kit – Command Line
• Job execution– globusrun
• File transfer– globus-url-copy
• GASS (Global Access to Secondary Storage)– globus-gass-server– globus-gass-server-shutdown
24http://grid.rit.edu
Java CoG Kit - Shell
http://www.mcs.anl.gov/~laszewsk/www/www-cogkits-org/talks/Commodity_Grid_Kits_Cog.pdf 25http://grid.rit.edu
Referances
• Thain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience, 17(2-4), 323-356.
• http://www.cs.wisc.edu/condor/manual/v6.4/2_4Road_map_Running.html• http://www.cs.wisc.edu/condor/quick-start.html• Foster, I. (2005). A Globus Primer: Describing Globus Toolkit 4• von Laszewski, G. (2006). Java CoG Kit Workflow Concepts. Journal of Grid Computing.• http://www.mcs.anl.gov/~laszewsk/www/www-cogkits-org/talks/Commodity_Grid_Kits_Cog.pdf• von Laszewski, G., Foster, I., Gawor, J., & Lane, P. (2001). A Java Commodity Grid Kit. Concurrency and
Computation: Practice and Experience, 13(8-9), 643-662.
26http://grid.rit.edu