Grid Scheduling through Service-Level Agreement
Karl CzajkowskiThe Globus Project
http://www.globus.org/
2
Overview
Introduction to Grid Environments The Resource Management Problem
– Cross-domain applications
– Resource owner goals vs. application goals An Open Architecture to Manage Resources
– Service-Level Agreement (SLA)
– GRAM and Managed Services Related and Ongoing Work
3
Grid Resource Environment
Distributed users and resources Variable resource status Variable grouping and connectivity Decentralized scheduling/policy
RR
RR
R
R
?
?
R
RR
R
R R
R
R R?
?R
R
R
R R
dispersed users
VO-A VO-B
network
4
Social/Policy Conflicts
Application Goals– Users: deadlines and availability goals
– Applications: need coordinated resources Localized Resource Owner Goals
– Policies towards users
– Optimization goals Community Goals Emerge As:
– An aggregate user/application?
– A virtual resource? Both!
5
Data-Intensive Example
Concurrent resource requirements– Large scale storage, computing, network, graphics
Datapath involves autonomous domains
Reduction
TCP/IP
RenderingSorting Transport
... ... ... ...
Receive Buffer
Parallel I/O
6
Early Co-Allocation in Grids
SF-Express (1997-8)– Real-time simulation
– 12+ supercomputers, 1400 processors Required advance reservation
– Brokered by telephone!
– Globus DUROC software to sync startup
– Over 45 minutes to recover from failure In use today in MPICH-G2 (MPI library)
7
Traditional Scheduling
Closed-System Model– Presumption of global owner/authority
– Sandboxed applications with no interactions
– “Toss job over the fence and wait” Utilization as Primary Metric
– Deep batch queues allow tighter packing
– No incentives for matching user schedule Sub-cultures Counter Site Policies
– Users learn tricks for “gaming” their site
8
An Open Negotiation Model
Resources in a Global Context– Advertisement and negotiation
– Normalized remote client interface
– Resource maintains autonomy Users or Agents Bridge Resources
– Drive task submission and provisioning
– Coordinate acts across domains Community-based Mediation
– Coordination for collective interest
9
Community Scheduling Example
Individual users– Require service
– Have application goals
Community schedulers– Broker service
– Aggregate scheduling
Individual resources– Provide service
– Have policy autonomy
– Serve above clients
R2 R3 R4 R5 R6R1
S1 S2
J1 J2 J3 J4 J5 J6 J7
T/BSLA
RSLA
10
Negotiation Phases Discovery
– “What resources are relevant to interest?”– Finds service providers
Monitoring– “What’s happening to them now?”– Compare service providers
Service-Level Agreement– “Will they provide what I need?”– The core Resource Management problem– Process can iterate due to adaptation
11
Service-Level Agreement
Three kinds of SLA– Task submission (do something)
– Resource reservation (pre-agreement)
– Lazy task/resource Binding (apply resv.) Simple protocol for negotiating SLAs
– Basic 2-party negotiation> Support for basic offer/accept pattern
> Optional counter-offer patterns
> Variable commitment phase for stricter promises
– Client may maintain multiple 2-party SLAs
12
Many Types of Service
Must support service heterogeneity– Resources
> Hardware: disks, CPU, memory, networks, display…
> Logical: accounts, services…
> Capabilities: space, throughput…
– Tasks> Data: stored file, data read/write
> Compute: execution, suspended/swapped job
SLAs bear embedded term languages– Isolate domain-specific details
13
Domain Extension: File Transfer
Single goal– Reliable deadline transfer
Specialized scheduler– Brokers basic services
– Synthesizes new service> Fault-handling logic
Distributed resources– Storage space
– Storage bandwidth
– Network bandwidth
R1 R3R2
J1
S1
J3J2
T/BSLA
RSLA
14
Technical Challenges
Complex Security Requirements Global Scalability
– Similar ideals to Internet
– Interoperable infrastructure
– Policy-configurable for social needs Permanence or “Evolve in Place”
– Cannot take World off-line for service
– Over time: upgrade, extend, adapt
– Accept heterogeneity
15
GRAM2 GRAM2 GRAM2
Job CPU Disk
Application
Domain-specific SLA
Incremental SLAs
Information Service
Localresourcemanagers
SLAimplementation
Planner
Concrete SLA
Coordinator
Monitor& Discover
GRAM Architecture
16
WS-Agreement
New standardization effort Generalizes GRAM ideas
– Service-oriented architecture
– Resource becomes Service Provider
– Tasks become Negotiated Services
– SLAs presented as Agreement services Still supports extensible domain terms
17
WS-Agreement Entities
Policy
Agreement
Ops:setTerminationTime(limits)findServiceData(query)...
SDEs:
status query
(negotiate)
Consumer
Terms Related
Application Service
StatusAgrmts.
(monitor)
(invoke)
Agreement Provider
Application Service Provider
?InitiatorAgreement
18
WS-Agreement Adds Management
Policy
Consumer 1 Application Service Provider 1
Appl. Service 1
(invoke)
Agreement Provider 1
(negotiate)
(monitor)
Initiator 1Agreement AgreementFactory
Agreement 1
S.T. R.A.
Agreement 2
S.T. R.A.
19
Virtualized Providers
Policy
AgreementFactory
AgreementFactory
Policy
Policy
Agreement 3
S.T. R.A.
Consumer 1
Agreement 1
S.T. R.A.
Agreement 2
S.T. R.A.
Application Service Provider 1
Agreement Provider 1
Application Service Provider 2
Agreement Provider 2
Appl. Service 1
Appl. Service 2
(negotiate)
(monitor)(d
epen
ds o
n)(composes)
(invoke)
(manage)
Initiator 1Agreement
(Consumer 2)
(Agreement Initiator 2)
20
Agreement-based Jobs
Agreement represents “queue entry”– Commitment with job parameters etc.
Agreement Provider– i.e. Job scheduler/Queuing system
– Management interface to service provider Service Provider
– i.e. scheduled resource (compute nodes) Service is the Job computation
21
Advance Reservation for Jobs Schedule-based commitment of service
– Requires schedule based SLA terms Optional Pre-Agreement (RSLA)
– Agreement to facilitate future Job Agreement
– Characterizes virtual resource needed for Job
– May not need full job terms Job Agreement almost as usual
– May exploit Pre-Agreement> Reference existing promise of resource schedule
– May get schedule commitment in one shot> Directly include schedule terms
> (Can think of as atomic advance reserve/claim)
22
Need for Complex Description 128 physical nodes
– Physical topology> Interconnect
> RAM, disk size
– Subject of RSLA
Single MPI job– Subject of TSLA
– May reference RSLAs
Quality requirements– Real-time parameters
> CPU, disk performance
– Subject of BSLA
net prog
mpi
128 node
ratio
2 cpu
size
ram
size rate
disk
23
MDS Resource Models (History)
OS info
OS info
CPUsdev group=
cpu 0dev=
CPU info CPU info
cpu 1dev=
diskdev group= netdev group=
eth0dev=/scratch1dev=VMdev=dev=RAM
software=OS
hn=hostname
dev group=memory
CPU info RAM info
VM infoCPU info NET info
DISK info
NET infoDISK info
NET infoDISK infoVM infoRAM info
VM info
RAM info
CPU info
CPU info
provider
aggregation
logical resource
24
Future Models
Service behavioral descriptions– Unified service term model
– Capture user/application requirements
– Capture provider capabilities Core meta-language
– Facilitates planner/decision designs
– Extends with domain concepts
– Extensible negotiability mark-up> Capture range of negotiability for variable terms
> Capture importance of terms (required/optional)
> Capture cost of options (fees/penalties)
25
SLA Types in Depth Resource SLA (RSLA), i.e. reservation
– A promise of resource availability> Client must utilize promise in subsequent SLAs
Task SLA (TSLA), i.e. execution– A promise to perform a task
> Complex task requirements
> May reference an RSLA (implicit binding)
Binding SLA (BSLA), i.e. claim– Binds a resource capability to a TSLA
> May reference an RSLA (otherwise obtain implicitly)
> May be created lazily to provision the task
26
Resource Lifecycle S0: Start with no SLAs S1: Create SLAs
– TSLA or RSLA
S2: Bind task/resource– Explicit BSLA
– Implicit provider schedule
S3: Active task– Resource consumption
Backtrack to S0– On task completion
– On expiration
– On failure
S0
S1
S2
S3
RSLA TSLA
setdeathagree
async
BSLA
active
27
Incremental Negotiation
RSLA: reserve resources for future use
SLAsUser goal Resource state
RSLA TSLA
BSLA
RSLA
BSLA
Resources change state due to SLAs and scheduler decisions
TSLA: submit task to scheduler BSLA: bind reservation to task
28
Linking SLAs for Complex Case
Dependent SLAs nest intrinsically– BSLA2 defined in terms of RSLA2 and TSLA4
Chained SLAs simplify negotiation– Optionally link destruction/reclamation
TSLA1
RSLA1
BSLA1
TSLA2
TSLA3
Sta
ge o
ut
Sta
ge in
time
BSLA2
RSLA2
TSLA4
Net
30 GB for /scratch/tmpuser1/foo/* files
Complex job
50 GB in /scratch filesystem
account tmpuser1
29
Related Work
Academic Contemporaries– Condor Matchmaking
– Economy-based Scheduling
– Work-flow Planning Commercial Scheduler Examples
– Many examples for traditional sites> Several generalized for “the enterprise”
– Platform Computing> LSF scaled to lots of jobs
> MultiCluster for site-to-site resource sharing
– IBM eWLM> Goal-based provisioning of transactional flows
30
Condor Matchmaking
At heart: a scheduling algorithm Heuristics for pairing job with resource
– Match symmetric “Classified Ads”
– Great for bulk/commodity matching Closed system view
– Subsumes resource through “lease”
– Sandboxed job environment
– Favor vertical integration over generality
– Tuned high-throughput system
32
Future Work
SLA interaction with policy– SLA negotiation subject to policy
> One SLA affects another, e.g. RSLA subdivision
> One client “more important” than another
– SLA implemented by low-level policies> Domain-specific SLA maps to resource SLAs
> Resource SLAs map to resource control mechanisms
Resource characterization– Advertisement of resources: options, cost
– Interoperable capability languages
33
Conclusion
Generic SLA management– Compositional for complex scenarios
– Extensible for unique requirements
– Requires work on Grid service modeling> To describe jobs, resource requirements, etc.
Enhancement to proven architectures– Encompasses GRAM+GARA
Evolution of the Globus Toolkit RM– GRAM evolving since 1997
– WS-Agreement standard in progress
