resource brokering on complex grids eurogrid and grip presented by john brooke esnw october 3/4...
TRANSCRIPT
Resource Brokering on Complex Resource Brokering on Complex GridsGrids
EUROGRID and GRIP EUROGRID and GRIP
Presented by John Brooke ESNW
October 3/4UK/Japan N+N
January 17, 2003
ContentsContents
Aims of the resource broker
Functionality of the ancestral broker EUROGRID broker
Interoperability architecture (UNICORE-Globus)
Towards a resource description ontology
Relation to the GGF and OGSA
OrganizationFirewalls
Users
VirtualOrganisationBrokers
SystemBrokers
ComputeResources
Brokers as VOs
Client
Broker Servic
e
Broker Servic
e
Broker Servic
e
Broker Servic
e
Broker
Service
Broker
Service
Client
Client
Client
Client
Broker Servic
e
Resource
Resource
Resource
Resource
Resource
Resource
Broker
Service
VO Layer
Specialist Layer
Site Layer
Replication
Federated Brokering
Interoperability for BrokeringInteroperability for Brokering
We want to broker on Grids controlled by either UNICORE or Globus. In GRIP we developed two methods
1. Bifurcation, separate “sub-brokers” for a Globus or a Unicore Grid. This is achieved and is extensible to a limited extent.
2. Constructing an extendable resource broker utilising a Grid Resource Ontology to handle mappings of resource terms.
1
Ancestral EUROGRID Broker
The API allows two levels of operation:Resource Checking: Static requirements, capability and
capacity.QoS Checking: Performance vs cost. Tickets can be issued
as a “guarantee”.Protocol can be used symmetrically by Broker.
User
BrokerNJS
ExecutionNJS
ExecutionNJS
1 CheckQoS
2 CheckQoS
2 CheckQoS
3 CheckQoS_Outcome
3 CheckQoS_Outcome
4 CheckQoS_Outcome
Brokering for Workflows
BezierSGI Onyx @ Manchester
Vtk + VizServer
DiracSGI Onyx @ QMUL
LB3D with RealityGridSteering API
LaptopSHU Conference Centre
UNICOREGateway and NJS
Manchester
Fir
ew
al
l
SGI OpenGL VizServer
Simulation
Data
VizServer clientSteering GUI The Mind Electric GLUE web service hosting environment with OGSA extensionsSingle sign-on using UK e-Science digital certificates
UNICOREGateway and NJS
QMUL
Steering (XML)
Interoperable Broker – Method 1
1.1. The Network Job Supervisor (NJS) delegates the The Network Job Supervisor (NJS) delegates the Resource Check to the Broker at the Vsite.Resource Check to the Broker at the Vsite.
2.2. The UNICORE brokering track utilises the IDB The UNICORE brokering track utilises the IDB exactly as for the ancestral broker.exactly as for the ancestral broker.
3.3. The Globus track uses a translator of the QoS check The Globus track uses a translator of the QoS check object. The translation service is extendable.object. The translation service is extendable.
4.4. The results of the translation are used to drive the The results of the translation are used to drive the LDAP search and the Globus broker then utilises LDAP search and the Globus broker then utilises MDS to perform this.MDS to perform this.
UNICORE NJS 4.0 gave much greater power and UNICORE NJS 4.0 gave much greater power and flexibility in brokering for complex workflows.flexibility in brokering for complex workflows.
NJSNJS
BrokerBroker
Unicore BrokerUnicore Broker Globus BrokerGlobus Broker
IDBIDB TranslatorTranslator FilterFilter
Basic TranslatorBasic Translator MDS(GRIIS/GRIS)MDS(GRIIS/GRIS)
Delegates resource checkDelegates resource check
LookupLookupresourcesresources
Delegates translationDelegates translation Uses to drive Uses to drive LDAP searchLDAP search
PerformsPerformsLDAP searchLDAP search
DiagramDiagramOf Broker Of Broker ArchitectureArchitecture
Architecture – Method 1Architecture – Method 1
Pros and ConsPros and Cons
•A nice feature of Method 1 is that no alteration needs to be made to the client side of UNICORE, thus no alteration for application plugins or “expert” brokers
•Also no alterations need to be made to Globus.
•However the UNICORE description of Grid resources is very different from the MDS-2 description. MDS-2 does not publish software resource and user environment, Unicore does not check dynamic resource, e.g. machine loading.
•The need for resource description translation is thus highlighted.
NJSNJS
BrokerBroker
Unicore BrokerUnicore Broker Globus BrokerGlobus Broker
IDBIDBTranslatorTranslator
FilterFilterOntology engineOntology engine
Resource DiscoveryResource DiscoveryServiceService
Delegates resource checkDelegates resource check
LookupLookupresourcesresources
Delegates Delegates translationtranslation Uses to drive Uses to drive
MDS searchMDS search
HierarchicalHierarchicalGrid SearchGrid Search
DiagramDiagramOf Broker Of Broker ArchitectureArchitecture
Architecture – Method 2Architecture – Method 2
FilterFilter
Uses to Uses to Drive MDSDrive MDSSearchSearch
HierarchicalHierarchicalGrid SearchGrid Search
OtherOtherBrokersBrokers
Resource DiscoveryResource DiscoveryServiceService
OntologiesOntologies
•Defines knowledge domain and allows reasoning on this domain.
•If we can create a Grid Resource Ontology, creation of specialist translation classes from basic Grid translator becomes possible.
•IDB at sites can be created via ontology, it contains site specific information which the clients job specification cannot do.
•So brokers take client request formulated in RR space, at each site use translator to convert to RR space, offers come back with capability and QoS.
BrokerNJS
IDB
TSI/Host GT3
Gateway
Client
Broker
NJS
Host Host
Gateway
Client
R-GMANJS NJS
Host
Site-Wide BrokeringNormal EUROGRID/GRIP Brokering
Local Brokering Configurations
Persistent Virtual
Environments
Metascheduling
Service
Workflow Manager
Clients
Broker
Chargeable Schedulable GridServices
Other Brokers
Site Feedback Policy
Manager
Resource Usage
Monitor
Banking
Services
Brokering and OGSA Services
1
Relevant GGF workRelevant GGF work
•Grid Protocol Architecture-RG : Core Grid Functions
•Grid Resource Allocation Agreement Protocol-WG :advanced reservation, co-allocation
•CIM-based schema-WG : successor to LDAP
•GESA-WG looking at economic issues of scheduling
The recently-formed Semantic Grid RG is very interested in the Grid Resource Ontology idea.
1
Points for DiscussionPoints for Discussion
•What is the relationship between brokering and scheduling?
•How to deal with legacy (not Grid-aware) schedulers?
•How to relate the ontologies from the application side (Resource Requestor) to the service provision side (Resource Provider)?
•How does a broker estimate upper and lower bounds for turnaround time?
•How far does the broker trust information from the service provider. Should it monitor running workflows?
January 17, 2003 GRIP First Project Review 1
RR space
RP space
RP space
RP spaceRR space
request
request
Request referral
sync
Figure 1: Request from RR space at A mapped into resource providers at B and C, with C forwarding a request formulated in RR space to RP space at D. B and C synchronize at end of workflow before results returned to the initiator A.
AB
CD
RR and RP SpacesRR and RP Spaces