using the nsf teragrid for parametric sweep cms applications jeffrey p. gardner edward walker...
TRANSCRIPT
Using the NSF TeraGrid for Parametric Sweep CMS Applications
Jeffrey P. GardnerJeffrey P. GardnerEdward Walker Edward Walker Vladimir LitvinVladimir Litvin
Pittsburgh Supercomputing Pittsburgh Supercomputing Center Center
Texas Advanced Computing Texas Advanced Computing Center Center
California Institute of California Institute of TechnologyTechnology
The NSF TeraGrid• Links nine resource provider sites via a high speed
10/30 Gbps network backbone, providing, in aggregate, 20 teraflops of computing power, 1 petabyte of storage capacity, and high-end facilities for remote visualization of computing results.
• Compute resources composed of a heterogeneous mix of clusters with different architectures and operating systems, running different workload management systems, with different local configuration, specifying different queues, with different job submission and run-time limits.
Requirements
• Schedulers at each Teragrid site– Should be able to gracefully handle ~1000 single-
processor jobs at a time
• Metascheduler (Teragrid doesn’t have metascheduler)– distributes jobs across the Teragrid– Metascheduler must be able to handle ~100,000
jobs
Solution: PBS/LSF?
• In theory, existing Teragrid schedulers like PBS or LSF should provide the answer.
• In practice, this does not work.– Teragrid nodes are multiprocessor
– Only 1 PBS job per node
– Teragrid machines frequently restrict the number of jobs a single user may run
– Asking for many processors at once communicates your actual resource requirements.
Solution: Clever shell scripts?
• We could submit a single PBS job that uses many processors.– Now we have a reasonable number of PBS jobs.– Scheduling priority would reflect our actual
resource usage.
• This still has problems.– Each job takes a different amount of time to run:
we are using resources inefficiently.
Metacheduler Solution: Condor-G?
• Condor-G will schedule an arbitrarily large number of jobs across multiple grid resources using Globus.
• However, 1 serial Condor-G job = 1 PBS job, so we are left with the same PBS limitations as before:– Teragrid nodes are multiprocessor
– Only 1 PBS job per node
– Teragrid machines frequently restrict the number of jobs a single user may run.
The Real Solution: Condor + GridShell
• The real solution is to submit one large PBS on each Teragrid node, then use a private scheduler to manage serial work units within the PBS job.
Vocabulary: JOB: (n) a thing that is submitted via Globus or PBS WORK UNIT: (n) An independent unit of work (usually serial), such as the analysis of a single run
PBS Job
PE
PEPE
PEPE
PE
Serial Work Units PrivateScheduler
The Real Solution: Condor + GridShell
• The real solution is to submit one large PBS on each Teragrid node, then use a private scheduler to manage serial work units within the PBS job.
Vocabulary: JOB: (n) a thing that is submitted via Globus or PBS WORK UNIT: (n) An independent unit of work (usually serial), such as the analysis of a single run
PBS Job
PE
PEPE
PEPE
PE
Serial Work Units PrivateScheduler
CondorCondor
GridShellGridShell
Features: GridShell/Condor on the TeraGrid
• Automatically throttles condor_startd job submissions based on configurable local site policies.
• Tolerates and recovers in the presence of transient faults in the WAN and login nodes.
• Balances condor_startd jobs between sites based on queue wait times.
• Shunts condor_startd jobs off temporary faulty sites.• Automatically renews grid proxy credentials across
sites within the command environment. • Current users: Caltech CMS and NVO. SUBMITTED
70,000 jobs through clusters on TeraGrid to date.
Why GridShell/Condor?
• scalability - the actual parametric job submission is done directly to the compute nodes and not through the front-end node of the cluster;
• fault-tolerance –agents at a front-end node of a cluster maintain the condor_startd submissions locally, allowing transient WAN outages and periodic front-end node reboots to be resolved independently, in isolation from the rest of the system; and
• usability – the entire Condor submission, monitor, and control infrastructure is leveraged as a common job management environment for the user.
GridShell/Condor Process Architecture
COMPUTE NODES
CLIENT HOST FRONT-END NODE
#! /bin/gtcsh# setup condor configurationsetenv _GRID_WATCH_HOST Xsetenv _GRID_WATCH_PORT Ysetenv _GRID_THROTTLE Kcondor_startd on N procs in -1 instances
MASTER AGENT
SUBMISSION AGENT
STARTER
CONDOR_STARTD
STARTER
CONDOR_STARTD
USER SITE CHECKER
CONDOR CENTRAL MANAGER
EVENT AGENT
User
GRAMping
COMM AGENT
GRIDSHELLSESSION
SCREEN
Condor Overview
• Condor was first designed as a CPU cycle harvester for workstations sitting on people’s desks.
• Condor is designed to schedule large numbers of jobs across a distributed, heterogeneous and dynamic set of computational resources.
Advantages of Condor• Condor is flexible
– Resources can be any mix of architectures– Resources do not need a common filesystem– Resources do not need common user accounting
• Condor is dynamic– Resources can disappear and reappear
• Condor is fault-tolerant– Jobs are automatically migrated to new resources if
existing one become unavailable.
Central Manager
collector
Condor Daemon Layout (very simplified)
Submission Machine
schedd
Execution Machine
startdStartd sends system specifications (ClassAds) and system status to Central Manager
negotiator
(To simplify this example, the functions of the Negotiator are combined with the Collector)
Condor Daemon Layout (very simplified)
Central Manager
collector
Submission Machine
schedd
Execution Machine
startd
Schedd sends job info to Central Manager
User submits Condor job
Central Manager
collector
Condor Daemon Layout (very simplified)
Submission Machine
schedd
Execution Machine
startd
Central Manager uses information to match Schedd jobs to available Startds
Condor Daemon Layout (very simplified)
Submission Machine
schedd
Execution Machine
startd
Schedd sends job to Startd on assigned execution node
Central Manager
collector
“Personal” Condor on a Teragrid Platform
• Condor daemons can be run as a normal user.
• Condor “GlideIn”™ ability supports the ability to launch condor_startd’s on nodes within an LSF or PBS job.
“Personal” Condor on a Teragrid Platform(Condor runs with normal user permissions)
Central Manager
collector
Submission Machine
schedd
Execution PE
startd
Execution PE
startd
Execution PE
startd
Login Node
PBS Job on Compute Nodes- GlideIn
GridShell Overview• Allows users to interact with distributed grid computing
resources from a simple shell-like interface.• extends TCSH version 6.12 to incorporates grid-enabled
features:– parallel inter-script message-passing and synchronization– output redirection to remote files– parametric sweep
GridShell Examples
Redirecting the standard output of a command to a remote file location using GlobusFTP:
a.out > gsiftp://tg-login.ncsa.teragrid.org/data
Message passing between 2 parallel tasks:
if ( $_GRID_TASKID == 0) then echo "hello" > task_1
else Set msg=`cat < task_0`
endif
Executing 256 instances of a job:a.out on 256 procs
Merging GridShell with Condor
• Use GridShell to launch Condor GlideIn jobs at multiple grid sites
• All Condor GlideIn jobs report back to a central collector
• This converts the entire Teragrid into your own personal Condor pool!
Merging GridShell with Condor
Login Node
Gridshell event monitor
User starts GridShell Session at TACCUser starts GridShell Session at TACC
TACC (IA32)
Purdue (IA32)UC (IA32)
GridShell process
Merging GridShell with Condor
Login Node
Gridshell event monitor
Login Node
Gridshell event monitor
Login Node
Gridshell event monitor
GridShell session starts event monitor on remote login nodes via GlobusGridShell session starts event monitor on remote login nodes via Globus
TACC (IA32)
Purdue (IA32)UC (IA32)
GridShell process
Condor process
Merging GridShell with Condor
Login Node
collectorschedd
Gridshell event monitor
Login Node
Gridshell event monitor
Login Node
Gridshell event monitor
Local event monitor starts condor daemons on login nodeLocal event monitor starts condor daemons on login node
TACC (IA32)
Purdue (IA32)UC (IA32)
GridShell process
Condor process
Login Node
collectorschedd
Gridshell event monitor
PBS Job
Login Node
Gridshell event monitor
PBS Job
Login Node
Gridshell event monitor
PBS Job
gtcsh-ex
UC (IA32)TACC (IA32)
Purdue (IA32)
All event monitors submit PBS/LSF jobs.These jobs start GridShell gtcsh-exec on all processors
All event monitors submit PBS/LSF jobs.These jobs start GridShell gtcsh-exec on all processors
gtcsh-ex gtcsh-ex
Master gtcsh-exec
GridShell process
Condor process
gtcsh-ex gtcsh-ex gtcsh-ex
Master gtcsh-exec
gtcsh-ex gtcsh-ex gtcsh-ex
Master gtcsh-exec
Login Node
collectorschedd
Gridshell event monitor
PBS Job
Login Node
Gridshell event monitor
PBS Job
Login Node
Gridshell event monitor
PBS Job
gtcsh-ex
UC (IA32)TACC (IA32)
Purdue (IA32)
gtcsh-exec on each processor starts a Condor startd.Heartbeat is maintained between all gtcsh-exec processes
gtcsh-exec on each processor starts a Condor startd.Heartbeat is maintained between all gtcsh-exec processes
gtcsh-ex gtcsh-ex
Master gtcsh-exec
GridShell process
Condor process
gtcsh-ex gtcsh-ex gtcsh-ex
Master gtcsh-exec
gtcsh-ex gtcsh-ex gtcsh-ex
Master gtcsh-exec
startdstartd startd
startdstartd startd
startdstartd startd
“Heartbeat”
Login Node
collectorschedd
Gridshell event monitor
PBS Job
Login Node
Gridshell event monitor
PBS Job
Login Node
Gridshell event monitor
PBS Job
gtcsh-ex
UC (IA32)TACC (IA32)
Purdue (IA32)
gtcsh-exec on each processor starts a Condor startdgtcsh-exec on each processor starts a Condor startd
gtcsh-ex gtcsh-ex
Master gtcsh-exec
GridShell process
Condor process
gtcsh-ex gtcsh-ex gtcsh-ex
Master gtcsh-exec
gtcsh-ex gtcsh-ex gtcsh-ex
Master gtcsh-exec
startdstartd startd
startdstartd startd
startdstartd startd
Login Node
collectorschedd
Gridshell event monitor
PBS Job
Login Node
Gridshell event monitor
PBS Job
Login Node
Gridshell event monitor
PBS Job
gtcsh-ex
UC (IA32)TACC (IA32)
Purdue (IA32)
Condor schedd distributes Condor jobs to compute nodesCondor schedd distributes Condor jobs to compute nodes
gtcsh-ex gtcsh-ex
Master gtcsh-exec
GridShell process
Condor process
gtcsh-ex gtcsh-ex gtcsh-ex
Master gtcsh-exec
gtcsh-ex gtcsh-ex gtcsh-ex
Master gtcsh-exec
startdstartd startd
startdstartd startd
startdstartd startd
CMS production chain (preparations)
• First of all we are installing all CMS software on clusters and pileup at lonestar (we don’t have enough disk space to set up pileup everywhere)
• We are defining the name of the clusters because now we have only one cluster where digitization with pileup and final analysis must be done – lonestar– Copy both the condor_config.master.template and
condor_config.glidein.template to our own private directory. Then in our login script (.cshrc or .bashrc), set the environment variable _GRID_TEMPLATE_DIR to point to this directory
– Add the unique definition of the cluster in condor_config.glidein.template
CMS production chain (preparations)
• Define specific variables in .cshrc which will be different from cluster to cluster:
• $WORKDIR - where temporary subdir will be created and everything will be run
• $WWORKDIR - where temporary subdir will be created and everything will be run, as it is originally defined in OS. Usually it is the same with $WORKDIR, but sometimes it is differ. It is needed for FCrenamePFN to get rid of the absolute paths before tarring files and moving them out.
• $OSCAR_SRC - main OSCAR source directory where "eval `scram runtime -csh ` must be done to set everything up.
• $ORCA_SRC - main ORCA source directory where "eval `scram runtime -csh ` must be done to set everything up.
• $TOPRELEASE - where all releases of CMS software were installed.• $VCAL - place where VCAL shower library was placed• $DETGEOMFILE - place where cms133.rz is placed• $PILEUP - where are the pileup data and PoolFileCatalog_pileup.xml
resided
Start GridShell Session
Write a simple GridShell configuration script:# vo.conf:# A GridShell config scripttg-login.uc.teragrid.orgtg-login.purdue.teragrid.org
Start GridShell session. This submits PBS jobs at each site and starts local Condor daemons
% vo-login –n 4:16 –H ~/vo.conf –G –W 600Spawning on tg-login.uc.teragrid.org
Spawning on tg-login.purdue.teragrid.org
waiting for VO participants to callback...
###########Done.
-n 4:16 Start 4 PBS job per Teragrid machine, each with 16 processors-W 600 Each PBS job has a wallclock limit of 600 minutes-H vo.conf Configuration file
CMS production chain
• One main preparation script which massaging the scripts and putting all run numbers etc in place (walltime is in secs):– ./createCondorSubmit.csh <walltime_oscar_init>
<walltime_oscar_job> <walltime_orca_init> <walltime_orca_job> <walltime_tt>
– This script has all info hardcoded inside – number of events per run, dataset and owner names, condor submission parameters, etc. So, if you need to check this out and change the parameters – this is the place to look at.
– It invokes ./createCondorJob.csh which is making preparations for single job with specified run number.
CMS production chain• The main part of createCondorJob.csh:
./makeDAG.pl $titles $counter >>! $TOPDIR/work/${titles}.dag
./makeOscarInitCDR.pl $titles $counter $OSCAR_OWNER $OSCAR_DATASET $wall_time_oscar_init >! $TOPDIR/work/${titles}_oscar_init_$counter.cdr
./makeOscarJobCDR.pl $titles $counter $OSCAR_OWNER $OSCAR_DATASET $wall_time_oscar_job $PYTHIA_DATASET_NAME >! $TOPDIR/work/${titles}_oscar_job_$counter.cdr
./makeOrcaInitCDR.pl $titles $counter $OSCAR_OWNER $OSCAR_DATASET $ORCA_OWNER $ORCA_DATASET $wall_time_orca_init >! $TOPDIR/work/${titles}_orca_init_$counter.cdr
./makeOrcaJobCDR.pl $titles $counter $OSCAR_OWNER $OSCAR_DATASET $ORCA_OWNER $ORCA_DATASET $wall_time_orca_job >! $TOPDIR/work/${titles}_orca_job_$counter.cdr
./makeTTCDR.pl $titles $counter $ORCA_OWNER $ORCA_DATASET $wall_time_tt >! $TOPDIR/work/${titles}_tt_$counter.cdr
CMS production chain
• DAG template
JOB OSCAR_INIT /home/tg456083/Prod2005/aajj/work/aajj_oscar_init.cdr JOB OSCAR_JOB /home/tg456083/Prod2005/aajj/work/aajj_oscar_job.cdr JOB ORCA_INIT /home/tg456083/Prod2005/aajj/work/aajj_orca_init.cdr JOB ORCA_JOB /home/tg456083/Prod2005/aajj/work/aajj_orca_job.cdr JOB TT /home/tg456083/Prod2005/aajj/work/aajj_tt.cdr PARENT OSCAR_INIT CHILD OSCAR_JOB PARENT OSCAR_JOB CHILD ORCA_INIT PARENT ORCA_INIT CHILD ORCA_JOB PARENT ORCA_JOB CHILD TT RETRY OSCAR_INIT 0 RETRY OSCAR_JOB 0 RETRY ORCA_INIT 0 RETRY ORCA_JOB 0
CMS production chain• CDR template # Common job definitions"
Executable = /home/tg456083/Prod2005/aajj/startup/orca_job_wrapper
Getenv = True
Notification = Never
universe = vanilla
Error = /home/tg456083/Prod2005/aajj/work/err_orca_job.err
Output = /home/tg456083/Prod2005/aajj/work/out_orca_job.out
Log = /home/tg456083/Prod2005/aajj/logs/log_orca_job.log
should_transfer_files = TRUE
when_to_transfer_output = ON_EXIT
arguments = $titles $counter $orca_owner $orca_dataset
requirements = TARGET.TimeToLive > _WALLTIME_ORCA_JOB_
Requirements = TARGET.Project == “TACC_RESOURCES"
Queue
CMS production chain
• *_wrapper is doing all job (orca_job_wrapper e.g.):– Checking all setup variables– Transferring files from previous step to the area
where this job will be executed– Running the job– Attaching run and making FixColl – Moving data out
CMS production chain• Submit the whole DAG to condor# Filename: /home/tg456083/Prod2005/aajj/work/aajj.dag.condor.sub
# Generated by condor_submit_dag /home/tg456083/Prod2005/aajj/work/aajj.dag
universe = scheduler
executable = /home/tg456083/condor2//bin/condor_dagman
getenv = True
output = /home/tg456083/Prod2005/aajj/work/aajj.dag.lib.out
error = /home/tg456083/Prod2005/aajj/work/aajj.dag.lib.out
log = /home/tg456083/Prod2005/aajj/work/aajj.dag.dagman.log
remove_kill_sig = SIGUSR1
arguments = -f -l . -Debug 3 -Lockfile /home/tg456083/Prod2005/aajj/work/aajj.dag.lock Condorlog /home/tg456083/Prod2005/aajj/logs/personal_aajj.log –Dag /home/tg456083/Prod2005/aajj/work/aajj.dag -Rescue /home/tg456083/Prod2005/aajj
/work/aajj.dag.rescue -MaxJobs 200 -MaxPre 200 -MaxPost 200 -NoPostFail
environment
=_CONDOR_DAGMAN_LOG=/home/tg456083/Prod2005/aajj/work/aajj.dag.dagman.out;_CONDOR_MAX_DAGMAN_LOG=0
queue
Conclusion• We have used GridShell to turn the TeraGrid into our
own personal Condor pool• We can submit Condor jobs, and Condor will schedule
these jobs across multiple TeraGrid site• TeraGrid sites do not need to share architecture or
queuing systems• GridShell also allows us to use TeraGrid protocols to
transfer our input and output data• All of this fits into existing Teragrid software.• We can run CMS production chain through this system
– 40000 were done– Will be demonstrated at SC2005 in Seattle in November
2005