ResourceManagementandSchedulingResourceManagementandScheduling

MechanismsinGridComputingMechanismsinGridComputing

EdgarMagaaPerdomo

UniversitatPolitcnicadeCatalunyaNetworkManagementGroup

Barcelona,Spain

emagana@nmg.upc.eduhttp://nmg.upc.es/~emagana/

UNIVERSITATPOLITCNICADECATALUNYA

2

OutlineOutline

Introduction

ResourceManagementinGridComputing

SchedulinginGridComputing

OurApproach

PolicybasedManagementSystem

HeuristicMethodologies

Conclusions

3

Introduction:GridComputing(I)Introduction:GridComputing(I)

GRIDComputing:

GridSystemsareinterconnectedcollectionsofgeographicallydistributedandheterogeneousresourcesharnessedtogetherto satisfy different computational needs by great variety of

users.Exploiting underutilized computational resources (reducing

costs)

Allowing very large computation through virtual parallelmachines(speedup)

The essential mode to implement the Grid is through VirtualOrganizations(VO).

4

Introduction:GridComputing(II)Introduction:GridComputing(II)

Twopointsofview: Usersview:Ananalogytopowergrid GridDevelopersview:Accountmanagement,

resourcemanagement,applicationscheduling,andapplication/developmentenvironments;allpartsshouldworkinasecureway

andmore

GridComputingTaxonomy: Scale:Intragrid,Intergrid;cluster,campusgrid,global

grid Application/DevelopmentEnvironment:

computational/datagrid,informationgrid,andknowledgegrid

5

NextGenerationGridComputingTendencies:

Services should be provided to users regardless ofnetwork technology, administrative domain oroperativeplatform.

Effective access to large amount of computing,networkandstorageresources,reducingprocurement,deployment,maintenanceandoperationalcost.

Network Performance: Faulttolerance, Reliability,Scalability,FlexibilityandPersistence.

ChangesinGridServices:WSRF

Introduction:GridComputing(III)Introduction:GridComputing(III)

6

MainProblems:

Swift and dynamic allocation and reservation ofcomputationalresources

Allocationofnetworkresourcesperservice

Configurationofresourcesonfly

Deployment of distributed services in heterogeneousandmultidomainnetworks

CommunicationmechanismswithGrid/WebServices

Introduction:GridComputing(IV)Introduction:GridComputing(IV)

7

ResourceManagementResourceManagement

andand

SchedulingMechanismsSchedulingMechanisms

GridComputingGridComputing

8

ResourceManagementinGrid(I)ResourceManagementinGrid(I)

BasicRequirements:Membersshouldbetrustfulandtrustworthy.Sharingisconditional.Shouldbesecure.Sharingshouldbeabletochangedynamically.Discoveryandregisteringofresources.Canbepeertopeerorclient/server.Sameresourcemaybeusedindifferentways.Welldefinedarchitectureandprotocols.

9

Gridisaffectedbycontinuousinnovations:SchemasconversiontechnologiesAllowdatatomovebetweendifferentsystems

technologiesIntercommunicationbetweendifferentnetwork

domainsServicesonthenetneedspecificresources

requirementsGridneedtohandleresourcesinmoredynamicwayGridServiceswillrequiretocoordinationand

orchestrationofresourcesatruntimeQoSAware

ResourceManagementinGrid(II)ResourceManagementinGrid(II)

10

ResourceManagementArchitecture:

agent

SuperScheduler

JobQueuing

resource

Collection/monitoring

ResourceInformation

Service

agent

agent

LocalSchedulerqueuing

LocalSchedulerqueuing

LocalSchedulerqueuing

resource

resource

ResourceManagementinGrid(III)ResourceManagementinGrid(III)

11

ResourceManagementChallenges:

Satisfactory endtoend performance throughmultipledomains

Availabilityofcomputationalresources

Handle of conflicts between common resourcesdemand

Faulttolerance

Interdomaincompatibility(P2P)

ResourceManagementinGrid(IV)ResourceManagementinGrid(IV)

12

DifficultiesinResourceManagement:AutonomyHeterogeneityDynamics

GridResources:ComputingPowerDiskSpaceMemorySpaceNetworkBandwidthSoftware,etc.

ResourceManagementinGrid(V)ResourceManagementinGrid(V)

13

StagesofGridResourcesManagement:

Phase1:ResourceDiscovery: Findavailableresources

Phase2:SystemsSelection: Allocatetheresources

Phase3:JobExecution Runthejob Logtheresourceusage Releasetheresources Chargetheuser

ResourceManagementinGrid(VI)ResourceManagementinGrid(VI)

Scheduling

(JobShop)

14

SchedulingProblem(I)SchedulingProblem(I)

Afinitesetofnjobs

Eachjobconsistsofachainofoperations

Afinitesetofmmachines

Eachmachinecanhandleatmostoneoperationatatime

Each operation needs to be processed during anuninterruptedperiodofagivenlengthonagivenmachine

Purpose is to find a schedule, that is, an allocation of theoperations to time intervals to machines, that has minimallength

FormalDefinition:

15

SchedulingProblem(II)SchedulingProblem(II)

N i= [R1 R2 R3 Rm ]G=[

N1N2N3N n

]G=[

R11 R12 R13 R1mR21 R22 R23 R2mR31 R32 R33 R3m Rn1 Rn2 Rn3 Rnm

]

AGrid(G)consistsofnNodes:

ANode(N)consistsofmResources(R):

16

TraditionalScheduling(I)TraditionalScheduling(I)

ComplexityAnalysis:

BinomialAnalysis:

ComplexityEquation

17

ApproachesTraditionalScheduling(II)

AlgorithmAnalysis:

18

ApproachesTraditionalScheduling(III)

MostCommonAlgorithmsComplexTimes:

1*1063.0003,01.000

250.0001.3492,7500

40.0004602,3200

1.26765*103010.0002002,0100

1.12590*10152.500851,750

1,048,576400261,320

1024100101,010

32253,50,75

21001

2nn2n*log(n)log(n)n

19

OurApproach(I)

MainTarget:

DevelopmentofAlgorithmsforSchedulingComputationalResourcesbelongingtotheGridComputingInfrastructure

basedonHeuristicMethodologies

20

OurApproach(II)

ComplementaryTargets:

To implement the resulting algorithm into the environment of aPolicyBasedGridNetworkManagementArchitecture

ToguaranteecertainQualityofService(QoS)

To offer a rapid and costeffective access to large amounts ofcomputing,memory,software,networkandstorageresourcesacrossmultipledomains

Scheduling of resources regardless of network technology,operativeplatformoradministrativedomain

EvaluatetheperformanceoftheResourceManagementSystem

21

PolicyBasedManagementApproach

Definition:Policiesarerulesofthetype:ifthen

Automatedandflexiblewayofexpressingthesystembehavior

Thesystembehaviorcanbedynamicallyadaptedaccordingtobusinessneeds

Naturallyfittedtospecifythebusinessobjectives

TechnologyindependentandStandardizedbytheIETF

E. Magaa and J. Serrat, "QoS Aware PolicyBased Management Architecture for Service Grids", 14thIEEE International Workshops on Enabling Technologies: Infrastructures for Collaborative Enterprises(WETICE/ETNGRID2005).LinkopingUniversity,Sweden,June1315,2005.

22

IETFPolicyframeworkandarchitecture:

PolicyBasedManagementSystem

PolicyRepository

PolicyDecisionPoint

PolicyConsole

PolicyEnforcementPoint

Use

rIn

terf

ace

Ser

ver

Age

nts

RepositoryProtocol

RepositoryProtocol

PolicyProtocol

FunctionsofthePDP Interpretpolicy Detectpolicyconflicts Receivepolicydecision

requestsfromPEPs Determiningwhichpolicyis

relevant Sendpolicyelementsto

PEPsFunctionsofthePEP

Metering(auditingofpolicycompliance)

ApplyingactionsaccordingtoPDPdecisions

23

PBMSComponentsArchitecture

24

PBMSHighLevelFunctionality

1.ServiceLevelAgreement(SLA)

GridInfrastructureProviders(GIP)andGridServicesConsumers(GSC)

QoSLevels:DiamondGoldSilverBronze

25

PBMSHighLevelFunctionality

2.GridServiceRequirementsNLPolicy

SLAServiceRequirementsWebServicesResourceFrameworkRequirements

ServiceDescriptorWSResourcePropertiesDocument GridServiceInstancebyUDDIRegistry

InterDomainCommunications(P2P)

3.NLPolicyAnalysis

PolicyManagertoPDPManagerResourcesAvailabilitybyResourceManagerResourcesStatusbyMonitoringSystem

26

PBMSHighLevelFunctionality

4.SelectionofGridTargetNodes

Resource Manager contacts several times withMonitoringSystem

Resource Manager computes a set of available nodesofferingtheirresources

5.GridNodesConfiguration

QoSPDPcreatestheELpoliciesQoS PEP enforces the EL policies in the specific

nodesService PDP PEP respectively, keep the

functionality of the service according the servicerequirements

27

ApproachesHeuristicMethodologies

SimulatedAnnealing

TabuSearch

EvolutionaryAlgorithms

28

ApproachesHeuristicMethodologies(I)

SimulatedAnnealing:

Ithastheabilitytoavoidbecomingtrappedatlocalminima.

Thealgorithmemploysarandomsearch.

Inputandassessinitialsolution

EstimateinitialTemperature

GenerateNewSolution

AssessNewSolution

AdjustTemperature

UpdatesStoresStop

AcceptNewSolution

TerminateSearch

yes

no

29

TabuSearch:

Metastrategyforguidingknownheuristicstoovercomelocaloptimality.

Iterativetechniquewhichexploresasetofproblemssolutions(X).

Movesfromsolutionstoanothersolutions.

Thesemovesareperforme