october 19, 2005charm++ workshop, 2005 1 faucets tutorial presented by esteban pauli and greg koenig...

October 19, 2005 Charm++ Workshop, 2005 1

Faucets Tutorial

Presented by Esteban Pauli and Greg Koenig

Parallel Programming Lab, UIUC

Charm++ Workshop, 2005 2October 19, 2005

Outline

System Overview Cluster Scheduler Meta Scheduler Writing a Scheduling Strategy


Outline



Current Situation

Cluster

Cluster

UserUser

Cluster

Cluster

User

User

Where should I submit my job?


Faucets System

User

User

UserCluster

Cluster

Cluster

Faucets System

I’ll submit my job to Faucets!!!


Faucets System

User Faucets System

Cluster

Cluster

Cluster

1. User submits job to Faucets system

Job requirements


Faucets System

User Faucets System

Cluster

Cluster

Cluster

2. Faucets forwards requests to clusters meeting minimum requirements

Job requirements

Job require

ments


Faucets System

User Faucets System

Cluster

Cluster

Cluster

3. Clusters analyze job specs and return bids

Bid

Bid

Bids


Faucets System

User Faucets System

Cluster

Cluster

Cluster

4. User selects winner

Winner selected Winner selected


Faucets System

User Faucets System

Cluster

Cluster

Cluster

5. Winner runs job, user monitors progress


Outline



Cluster

Cluster Daemon

Scheduler

System Architecture

Central Server

Database

User

User


Why a New Scheduler?

Current schedulers try to maximize throughput – good for showing cluster is busy, but can be bad for users

Users worry about deadlines, priorities, fairness, etc. Profit centers worry about profits Need good interface between cluster scheduler and

meta scheduler Want scheduler that can leverage run-time system

(Charm++ checkpoint/restart, shrink/expand)


Scheduler

Database

Request Server

Job Monitor

Cluster Monitor

Strategy

Job scheduler

Scheduler Design

UserCluster


Installation

Install MySQL database Install Charm++, MPI ./configure; (edit config file); make; make install

sh sqlwriter.sh | mysql –user=root –p

Set up node list ./startScheduler

Done!!!


Job Submission

ufrun (uni-processor), frun (Charm++), frun-mpi (mpi) used for interactive jobs

ufsub, fsub, fsub-mpi used for batch jobs Options: +n, +p, +ppn, -stdout, … See manual for full details


Job Control

Job monitoring: fjobsOptions: -user <name>, -full, idSample output:UserId JobId # Nodes Head Node Status Name Time Remaining

------ ----- ------- --------- ------ ---- --------------

fuser 63503 1 arch017.c RUNNING my_exe 0:06:39:28

Job deletion: fkillOptions: -u <name>, id


Outline



Disclaimer

Current code base developed as a proof of concept

Code is not yet production quality Code works, but has not been tested thoroughly Code has some security issues

Use at your own risk, and please report bugs Code will be updated within the next year


Cluster

Cluster Daemon

Scheduler

System Architecture

Central Server

Database

User

User


Central Server

Responsible for keeping all information about users and clusters in system

Responsible for forwarding users’ job requests to clusters

Responsible for dispute arbitration Responsible for keeping account balances


Central Server: Database

Cluster table: contains information about confederated clusters

mysql> create table Cluster (

domainName text not null,

port int not null,

status text not null,

acctId int not null );


Central Server: Database (cont.)

User table: contains information about registered users

mysql> create table users (

userid text not null,

password text not null,

localCluster text not null,

acctId int not null );



Account table: keeps account balances for clusters and users

mysql> create table accounts (

clusterid text not null,

acctId int not null,

balance int not null,

pbalance int not null);



Job table: keeps track of all running and completed jobs

mysql> create table Jobs (

JobID text not null,

User text not null,

Status text not null,

ClusterID text not null);


Central Server: Installation

Compilation & Configuration: cd faucets; makeEdit faucets/cs/db.propertiesGet and install JDBC

Running the central server: cd faucets java -cp .:/path/to/mm.mysql-2.0.8-bin.jar TheServer

As users and clusters join, update DB


Cluster

Cluster Daemon

Scheduler

System Architecture

Central Server

Database

User

User


Cluster Daemon

Purpose: provide interface between central server and cluster scheduler

No user intervention Installation: cd faucets; make Usage:

cd faucets java -cp .:./common/TB.jar cd.ClusterDaemon

<CS_hostname> <CS_port> /tmp/


Cluster

Cluster Daemon

Scheduler

System Architecture

Central Server

Database

User

User


Command Client

Installation: cd faucets/cc; make Some common commands

Job submissionjava cc.FaucetCLI <central_server_DNS> <port> <application name> [-input file1,file2,...,filen] [<args>]

Retrieving output filesjava FaucetCLI GetFile <job-id> <file-name>


Faucets GUI


Cluster

Cluster Daemon

Scheduler

System Architecture

Central Server

Database

User

User


Outline



Faucets Scheduling Framework

In many ways Faucets can be thought of simply as a framework for creating cluster scheduling solutions

Any Faucets deployment has some scheduling objective that it tries to achieve Traditional FIFO scheduling On-demand scheduling – driven by workloads and the

priorities that users have to access resources Resource bartering – driven on an economic basis

(“where can this job be run for the least cost?”)


Schedule Strategies

The scheduling method used by Faucets can readily be changed by writing a scheduling strategy

The scheduling strategy can do interesting things that take advantage of features in lower level runtime systems (e.g., Charm++ Adaptive Jobs that shrink/expand to better utilize cluster)

Scheduling strategy code is a C++ class Implement to reflect the scheduling method Recompile the cluster scheduler executable


Schedule Strategy Examples

PriorityFIFOStrategy Jobs have assigned priorities Jobs of a given priority are scheduled FIFO Jobs of a higher priority can preempt jobs of a lower priority

LimitFIFOStrategy Jobs are scheduled FIFO The number of short/long jobs that a given user may be running

simultaneously is limited to prevent resource domination GanttChartStrategy

Jobs are scheduled by arranging them on a Gantt Chart Predictions can be made about whether a given job can be

completed before a user-specified deadline


Implementing a Strategy (1)

To implement a new strategy Inherit from SchedulingStrategy base class Implement four methods

class PriorityFIFOStrategy : public SchedulingStrategy{ public: PriorityFIFOStrategy (int nodes);

float is_available (Job *job, Job *wait_queue, Job *run_queue); void allocate_nodes (Job *wait_queue, Job *run_queue); void addjob (char *username, int num_procs); void removejob (char *username, int num_procs);};



float is_available (Job *job, Job *wait_queue, Job *run_queue);

Method parameters job – a pointer to an incoming job wait_queue – a pointer to the queue of all waiting jobs run_queue – a pointer to the queue of all running jobs

Method returns a float 0.0 to 1.0 – the utilization of the cluster if the incoming job is accepted

for execution (0% to 100%) -1.0 - the incoming job cannot be accepted for immediate execution

This method is used (indirectly) by the Cluster Daemon to make the choice of target cluster when doing metascheduling



void allocate_nodes (Job *wait_queue, Job *run_queue);

Method parameters wait_queue – a pointer to the queue of all waiting jobs run_queue – a pointer to the queue of all running jobs

Method examines each Job object in the wait_queue and updates it by allocating available cluster nodes to the job so the scheduler can launch it For Charm++ jobs, normally try to allocate max_nodes first and

then min_nodes after that to fulfill the request



void addjob (char *username, int num_procs);void removejob (char *username, int num_procs);

Method parameters username – the user submitting a new job into the cluster num_procs – the number of processors the new job is allocated

Method can be used to enforce some characteristic about the number of processors allocated to any given user (e.g., limit the total number of processors that any given user is allocated)


Instantiating a Strategy

The scheduling strategy is instantiated in the Scheduler class constructor located in Scheduler.C

strategy = new PriorityFIFOStrategy (num_nodes);

The total number of nodes in the cluster is provided to the strategy’s constructor

That’s it! The scheduler code calls into your custom Strategy class whenever it needs to make a decision about whether a new job can be scheduled, to allocate nodes to the job, etc.


Questions?


Thanks

october 19, 2005charm++ workshop, 2005 1 faucets tutorial presented by esteban pauli and greg koenig...

Documents

uiuccharm workshop

shrinkexpandcharm workshop

detailscharm workshop

idcharm workshop

bidsbidbidbidscharm

yearcharm workshop

null charm workshop

faucets systemcharm