loadleveler 3.3

IBM Systems & Technology Group

LoadLeveler 3.3

Dr. Roland Kunz, IT [email protected]

l


LoadLeveler V3.3 (GA 4/05)

LoadLeveler deliverables Highlights Scheduler Improvements

Preemption using Backfill Scheduler

Advance Reservation

Job Launch Performance Improvements

Additional Policy Control and Usability Enhancements

Grid interaction


Hardware OS GA

LL 3.2 Intel 32 bit (xSeries nodes and blades)

RHEL 3.0 5/2004

LL 3.2 pSeries (pSeries nodes and JS20s)

SLES 8 5/2004

LL 3.2.1 Intel 32 bit

SLES 9 12/2004

LL 3.2.1pSeries

SLES 9 12/2004

LL 3.2.1 Opterons

SLES 9RHEL 3.0

12/2004

Linux Deliverables in 2004


Hardware OS

xSeries – Intel (32 bit and EM64T) RHEL 3.0RHEL 4.0 SLES 9

xSeries – AMD RHEL 3.0RHEL 4.0 SLES 9

pSeries SLES 9RHEL 4.0

LL V3.3 on Linux – GA 8/05


Click here to type page titleCommunication through daemons

LoadL_master

LoadL_negotiator

LoadL_schedd GSMonitor

LoadL_startd

LoadL_kbdd


Click here to type page title

Programm

program starts

Checkpoints

Program terminated (by user, admin or accident)

Planned ending of program

Restart

Real ending of program


Click here to type page titleBackfill scheduler


Advance Reservation


Advance Reservation

Overview of Advance Reservation

Administrator Externals

User Externals

Misc.


Overview Advance Reservation == Reservation (Terms used interchangeably)

Satisfy Grid Computing Customer Requirements to Reserve Resources for Jobs

Can Reserve Computing Resources (Nodes) forWorkloadMaintenance…

For BACKFILL Scheduler Only


Reservation Request

Reservation: A set of nodes reserved for a period of time

Unique Reservation ID: c94n04.2.rAdd creation time to make it truly unique

Start Time: 03/23/2005 10:00

Duration: 120 Minutes

A List of Reserved Nodes: c94n04 c94n03


Reservation Owner and Group

Owner: Peterusually the user who made the reservationcan be changed by LoadLeveler administratorscan run jobs in the reservationcan modify or cancel the reservationcan allow other users to use the reservation

Group Owner: Research_GroupLoadLeveler group, not AIX/Linux groupFor quota checking only


Reservations

c94n04.3.r

c94n03.5.r

c94n04.5.r

Reservations Do Not Overlap

timetime

nodesnodes


Setup State of the Reservation

To get reserved nodes ready :

Preempt Running Jobs if any on the Reserved Nodes

ƒ Use DEFAULT_PREEMPTION_METHOD

Check Status of Reserved Nodes

Send email to the owner and LoadLeveler administrators if

ƒ A node is not usableƒ A non-preemptable job is running


Active State of the Reservation

Schedule Jobs to Run

No System Preemption in Reservation

Manual Preemption by llpreempt is allowed

Jobs should not require resources other than those on reserved machines.


Administrator Externals


Admin Control – User/Group Must Setup Quotas:

max_reservations for User and Group stanzas

Example 1 : Every user can make a reservation:default: type = usermax_reservations = 1

Example 2: A group of users can make 10 reservations total

res_group: type = groupmax_reservations = 10include_users = carol dave alex rich


Config Control - RESERVATION_MIN_ADVANCE_TIME

Default: Reservation Can Start As Soon As PossibleRESERVATION_MIN_ADVANCE_TIME = 0No advance time is required

To Require a Reservation Be Made One Day In Advance:RESERVATION_MIN_ADVANCE_TIME = 1440


AccountingAccounting

To Collect Reservation History:To Collect Reservation History:

Set Set ACCT = A_RES in LoadL_config to record in LoadL_config to record reservation usage data reservation usage data

TheThe RESERVATION_HISTORY keyword in keyword in LoadL_config can be used to define the name of a LoadL_config can be used to define the name of a file containing the local history of reservationsfile containing the local history of reservations

llacctmrg -R command can be used to merge command can be used to merge reservation history files, similar to job historyreservation history files, similar to job history


User Externals


Common Questions

How Many Reservations One Can Make and in Which Groups

How Long the Reservation Duration Can Be

Can Jobs Expected to Run Beyond Reservation End Time be Allowed to Run

How Far In Advance a Reservation Needs to be Made

Floating Consumable Resources Should Not Be Used by Jobs in a Reservation


User Commands

Use llmkres to Make a New Reservation

Use llchres to Modify an Existing Reservation

Use llrmres to Remove an Existing Reservation

Use llbind to Bind Jobs to an Existing Reservation

Use LL_RES_ID=<Reservation ID> llsubmit to submit Jobs to an Existing Reservation

Use llqres to Query Reservations

Use llq -l to Check Whether a Job Step is Bound to a Reservation


llmkres Examples Reserve 2 nodes at 2pm today for 60 minutes

llmkres -t 14:00 -d 60 -n 2

Reserve a Specific Node c94n04llmkres -t 03/23/2005 14:00 -d 120 -h c94n04

Reserve All Available Nodesllmkres -t 09:00 -d 60 -h all

Reserve Nodes to Run Job Step c94n04.3.0llmkres -t 14:00 -d 60 -j c94n04.3.0


llchres Examples To Move the Start Time of Reservation c94n04.2.r Earlier by 60 Minutes

llchres -t -60 -R c94n04.2.r

To Add a Reserved Node:llchres -n +1 -R c94n04.2.r

To Remove a Specific Reserved Node:llchres -h -c94n04 -R c94n04.2.r


llrmres Examples

To Remove Reservation c94n04.2.r llrmres -R c94n04.2.r

To Remove All Reservations:llrmres -R all

Note: regular users can only remove all of their own reservations

LoadLeveler administrators can remove all reservations


llbind and llsubmit Examples

To Submit a job to Reservation c94n04.2.r LL_RES_ID=c94n04.2.r llsubmit weather.cmd

To Bind an Idle Job Step to Reservation c94n04.2.r llbind -R c94n04.2.r c94n03.1.0

To Unbind Job Steps from Reservation c94n03.1.r llbind –r c94n03.1.r c94n04.1


API

Reservation APIsSample: /usr/lpp/LoadL/full/samples/llres/res.c

RESERVATIONS query in Data Access APIs/usr/lpp/LoadL/full/samples/lldata_access/qres.c

See LoadLeveler U&A Guide for More Information


Backfill Preemption


Preemption with Backfill Scheduler

Similar concept as in Gang Scheduler

Options to terminate job in addition to suspend

Support for AIX and Linux–Suspend not supported on Linux

Configuration keyword PREEMPTION_SUPPORT– Applicable to Gang Scheduler only ; default for gang scheduler is FULL

– When using the backfill scheduler, either do not set it or set it to NONE

Configuration keyword DEFAULT_PREEMPT_METHOD–Default is suspend

–Must be set for Linux


Preemption Methods

Configuration Keyword (for Backfill Scheduler Only) -

DEFAULT_PREEMPT_METHOD = rm | sh | su | vc | uh

Preemption Methods–Remove (rm)

–System Hold (sh)

–Suspend (su)

–Vacate (vc)

–User Hold (uh)


Preemption Commands and API

llpreempt command

–llpreempt -? | -H | -v | [-q] [-r | -m method] { [-u userlist] [-h hostlist] | [joblist] }

–-r option only for jobs preempted by suspend

–llhold -r to resume jobs preempted by system hold and user hold

–Jobs preempted by remove must be resubmitted

–Jobs preempted by vacate will restart when resources available

ll_preempt_jobs API

–int ll_preempt_jobs (int version, void *errObj, LL_preempt_param **param);

–Replaces ll_preempt API


llmodify and llq

llmodify command

–llmodify [-?] | [-H] | [-v] | [-q] | {-x <execution_factor> | -c <consumable_cpus> | -m <consumable_real_memory> | -W <wclimit_add_min> | -C <job_class> | -a <account_no> | -s <q_sysprio> | -p {preempt|nopreempt} } <jobstep>

llq command

– llq –l from Central Manager

Preemptable: yes

– llq –s

This job step is scheduled to run but is waiting for the following job steps to be preempted:


Misc. Enhancements


Accounting

Correlating AIX and LoadLeveler Accounting Records–Find all AIX accounting records for all processes in LoadLeveler job

–Add unique identifier in both AIX and LoadLeveler accounting

–LoadLeveler invokes setsubproj() to set accounting key in AIX

–Supported in AIX 5.2I and AIX 5.3

–AIX library dynamically loaded: /usr/lib/libaacct.a

–Use llsummary –l to find accounting key in LoadLeveler history file Job Step Id: c188f2n07.ppd.pok.ibm.com.1.0 Step Name: 0 Queue Date: Wed Feb 9 15:12:15 EST 2005 Job Accounting Key: 4758737585449234063 …


Modifying System Priority

Modifying job priorities– llmodify [-?] | [-H] | [-v] | [-q] | {-x <execution_factor> | -c <consumable_cpus> | -m <consumable_real_memory> | -W <wclimit_add_min> | -C <job_class> | -a <account_no> | -s <q_sysprio> | -p {preempt|nopreempt} } <jobstep>

– LL_MODIFY_SYSPRIO enum with the ll_modify API

– New priority is fixed

– llq –l from Central Manager

System Priority: -1560 q_sysprio: 97 Previous q_sysprio: -1560


Running LoadLeveler jobs on the IBM Grid Toolbox (Globus)- LoadLeveler GAR to deploy (llgrid.gar)- GAR contains loadleveler.pm, globus-gram-loadleveler-provider,

rips-loadleveler-provider.xml, mjs-ll-server-deploy.wsdd, server-

deploy.wsdd, deploy/loadleveler-preDeploy.sh.

The IBM Grid Toolbox must be installed and gars /mmjfs.gar and gars/gram-rips.gar must be deployed before deploying LoadLeveler GAR.

Log in as ibmgrid that is the owner ID of the Grid Toolbox. export GLOBUS_LOCATION=/opt/IBMGrid Copy llgrid.gar into $GLOBUS_LOCATION/gars directory.

cp /usr/lpp/LoadL/full/lib/llgrid.gar $GLOBUS_LOCATION/gars/llgrid.gar As ibmgrid, run the following commands to deploy the llgrid.gar: cd $GLOBUS_LOCATION . igt-setenv.sh

Integration in a Grid environment


igt-deploy-gar gars/llgrid.gar

After deploying two new files exist:

$GLOBUS_LOCATION/lib/perl/Globus/GRAM/JobManager/loadleveler.pm

$GLOBUS_LOCATION/etc/globus-gram-loadleveler-provider.

LoadlevelerManagedJobFactoryService service name added in $GLOBUS_LOCATION/local-server-config.wsdd MasterLoadlevelerManagedJobFactoryService service added in$GLOBUS_LOCATION/AppServer/installedApps/DefaultNode/IBMGrid.ear/ogsa.war/WEB-INF/server-config.wsdd.LoadLevelerInformation service data provider enabled in $GLOBUS_LOCATION/AppServer/installedApps/DefaultNode/IBMGrid.ear/ogsa.war/WEB-INF/etc/rips-service-config.xml.

Integration in a Grid environment


Remove handler="jobDataHandler" in

$GLOBUS_LOCATION/AppServer/installedApps/DefaultNode/IBMGrid.ear/ogsa.war/WEB-INF/etc/rips-service-config.xml if this xml file contains handler="jobDataHandler".

Integration in a Grid


Questions

loadleveler 3.3

Documents