university of catania, engineering faculty department of computer science and telecommunications...
TRANSCRIPT
University of Catania, Engineering FacultyUniversity of Catania, Engineering Faculty
Department of Computer Science and TelecommunicationsDepartment of Computer Science and Telecommunications
Report about the Development of Report about the Development of
Scheduling Algorithm in the Resource BrokerScheduling Algorithm in the Resource Broker
S. Cavalieri and S. MonforteS. Cavalieri and S. Monforte
IntroductionIntroduction
State of the State of the implementatiimplementationon
Design IssuesDesign Issues
One ClassAd One ClassAd One LDAP One LDAP EntryEntry
All in OneAll in One
Final Final RemarksRemarks
IntroductionIntroduction
State of the State of the implementatiimplementationon
Design IssuesDesign Issues
One ClassAd One ClassAd One LDAP One LDAP EntryEntry
All in OneAll in One
Final Final RemarksRemarks
IntroductionIntroduction
State of the State of the implementatiimplementationon
Design IssuesDesign Issues
One ClassAd One ClassAd One LDAP One LDAP EntryEntry
All in OneAll in One
Final Final RemarksRemarks
Current GRIS/GIIS Information ArchitectureCurrent GRIS/GIIS Information Architecture
Information Services (i.e. grid-info-host, globus-version etc.) on GRISes publish Information Services (i.e. grid-info-host, globus-version etc.) on GRISes publish LDIF objects representing their resources in LDAP directories.LDIF objects representing their resources in LDAP directories.
None of these informations are sent to the GIIS.None of these informations are sent to the GIIS. The GIIS maintains a directory (grid-info-site-regdir) as an index of registered The GIIS maintains a directory (grid-info-site-regdir) as an index of registered
GRISes (reg.GRISes (reg.<port><port>..<hn><hn>.conf).conf)
GIISGIIS
GRIS gridct1.ct.infn.itGRIS gridct1.ct.infn.it
GRIS gridct2.ct.infn.itGRIS gridct2.ct.infn.it
GRIS gridct3.ct.infn.itGRIS gridct3.ct.infn.it
globus-versionglobus-version
grid-info-hostgrid-info-host
globus-gram-schedulerglobus-gram-scheduler
globus-versionglobus-version
grid-info-hostgrid-info-host
globus-gram-schedulerglobus-gram-scheduler
globus-versionglobus-version
grid-info-hostgrid-info-host
globus-gram-schedulerglobus-gram-scheduler
grid-info-site-regdirgrid-info-site-regdir
reg.2163.gridct1.ct.infn.it.confreg.2163.gridct1.ct.infn.it.conf
reg.2163.gridct2.ct.infn.it.confreg.2163.gridct2.ct.infn.it.conf
reg.2163gridct3.ct.infn.it.confreg.2163gridct3.ct.infn.it.conf
Design IssuesDesign Issues
One ClassAd One ClassAd One LDAP One LDAP EntryEntry
All in OneAll in One
Final Final RemarksRemarks
The Resource BrokerThe Resource Broker
Implementation of the scheduling functions performed by Implementation of the scheduling functions performed by the Resource Broker, is based on the following architecture:the Resource Broker, is based on the following architecture:
GIISGIIS
GRISGRIS
GRISGRIS
GRISGRIS
Resource Resource BrokerBroker
Resource Resource BrokerBroker
Resources Published Resources Published in LDAP treesin LDAP trees
ClassAd Description of ClassAd Description of GRISes ResourcesGRISes Resources
JDLJDLJDLJDL
ClassAd Description of ClassAd Description of Job RequirementsJob Requirements
Best Matching GRISBest Matching GRIS
IntroductionIntroduction
State of the State of the implementatiimplementationon
IntroductionIntroduction
State of the State of the ImplementatiImplementationon
Design IssuesDesign Issues
One ClassAd One ClassAd One LDAP One LDAP EntryEntry
All in OneAll in One
Final Final RemarksRemarks
Implemented Functions Implemented Functions
Search of all the available resources. Through the analysis Search of all the available resources. Through the analysis of the content of the directory “grid-info-site-regdir” where of the content of the directory “grid-info-site-regdir” where GIIS maintains the list of GRIS referring to it, information GIIS maintains the list of GRIS referring to it, information useful to localisation and access to each GRIS is obtained.useful to localisation and access to each GRIS is obtained.
For each GRIS that can be reached from the GIIS, For each GRIS that can be reached from the GIIS, information published are retrieved through LDAP.information published are retrieved through LDAP.
Collected resources are converted in terms of ClassAds Collected resources are converted in terms of ClassAds and maintained by an ad hoc data structure. and maintained by an ad hoc data structure.
IntroductionIntroduction
State of the State of the ImplementatiImplementationon
Design IssuesDesign Issues
One ClassAd One ClassAd One LDAP One LDAP EntryEntry
All in OneAll in One
Final Final RemarksRemarks
Implemented Functions Implemented Functions
Two objects have been developed in order to provide an Two objects have been developed in order to provide an abstraction over the resources published in a LDAP abstraction over the resources published in a LDAP directory tree of a GRIS as well as regarding the directory tree of a GRIS as well as regarding the architecture designed for connecting GRISes and GIIS. architecture designed for connecting GRISes and GIIS.
A GRISResourceWrapper is instantiated for each GRIS A GRISResourceWrapper is instantiated for each GRIS supplying the GIISResourceWrapper with a ClassAd View supplying the GIISResourceWrapper with a ClassAd View of its resources.of its resources.
The GIISResourceWrapper provides for a coherent and The GIISResourceWrapper provides for a coherent and easy to query view over the entire site resources, ready to easy to query view over the entire site resources, ready to be used for matching.be used for matching.
IntroductionIntroduction
State of the State of the ImplementatiImplementationon
Design IssuesDesign Issues
One ClassAd One ClassAd One LDAP One LDAP EntryEntry
All in OneAll in One
Final Final RemarksRemarks
Design IssuesDesign Issues
What is the best design for the GRISResourceWrapper ?What is the best design for the GRISResourceWrapper ?
How many ClassAds should be mantained for each How many ClassAds should be mantained for each GRISResourceWrapper ?GRISResourceWrapper ?
Which attributes should be mantained ? Which attributes should be mantained ?
How and when a GRISResourceWrapper should be How and when a GRISResourceWrapper should be updated ? updated ?
IntroductionIntroduction
State of the State of the ImplementatiImplementationon
Design IssuesDesign Issues
One ClassAd One ClassAd One LDAP One LDAP EntryEntry
All in OneAll in One
Final Final RemarksRemarks
What is the best design for the What is the best design for the GRISResourceWrapper ?GRISResourceWrapper ?
Three Solutions:Three Solutions:
One ClassAd, One LDAP EntryOne ClassAd, One LDAP Entry
One ClassAd for all the LDAP EntriesOne ClassAd for all the LDAP Entries
Nested ClassAd (not discussed in the presentation)Nested ClassAd (not discussed in the presentation)
IntroductionIntroduction
State of the State of the ImplementatiImplementationon
Design IssuesDesign Issues
One ClassAd One ClassAd One LDAP One LDAP EntryEntry
All in OneAll in One
Final Final RemarksRemarks
One ClassAd, One LDAP EntryOne ClassAd, One LDAP Entry… …
A GRISResourceWrapper maintains a collection of ClassAds, one for A GRISResourceWrapper maintains a collection of ClassAds, one for each entry of the LDAP directory, which contains the resources each entry of the LDAP directory, which contains the resources published by a globus information service.published by a globus information service.
[ [ ……
schedulertype=”condor”schedulertype=”condor”
……]]
[ [ ……
schedulertype=”condor”schedulertype=”condor”
……]]
globus-gram-schedulerglobus-gram-scheduler
grid-info-hostgrid-info-host
[ …[ …
ostype=”linux”ostype=”linux”
……] ]
[ …[ …
ostype=”linux”ostype=”linux”
……] ]
globus-versionglobus-version
[ …[ …
releaseDate=2000/04/11releaseDate=2000/04/11
……] ]
[ …[ …
releaseDate=2000/04/11releaseDate=2000/04/11
……] ] ClassAd View of ClassAd View of ResourcesResources
IntroductionIntroduction
State of the State of the ImplementatiImplementationon
Design IssuesDesign Issues
One ClassAd One ClassAd One LDAP One LDAP EntryEntry
All in OneAll in One
Final Final RemarksRemarks
One ClassAd, One LDAP EntryOne ClassAd, One LDAP Entry… …
The ClassAd expression of the job requirements may contain attributes The ClassAd expression of the job requirements may contain attributes published by different information services, thus owned by different published by different information services, thus owned by different ClassAd of the collection the GRISResourceWrapper stores. ClassAd of the collection the GRISResourceWrapper stores.
[… […
requirements= requirements= other.ostype==”linux” && other.ostype==”linux” && other.schedulertype==”condorother.schedulertype==”condor
……]]
[… […
requirements= requirements= other.ostype==”linux” && other.ostype==”linux” && other.schedulertype==”condorother.schedulertype==”condor
……]]
JobAdJobAd
[ [ ……
schedulertype=”condor”schedulertype=”condor”
……]]
[ [ ……
schedulertype=”condor”schedulertype=”condor”
……]]
[ …[ …
ostype=”linux”ostype=”linux”
……] ]
[ …[ …
ostype=”linux”ostype=”linux”
……] ]
[ …[ …
releaseDate=2000/04/11releaseDate=2000/04/11
……] ]
[ …[ …
releaseDate=2000/04/11releaseDate=2000/04/11
……] ]
GRISGRIS
ResourceAdsResourceAds
IntroductionIntroduction
State of the State of the ImplementatiImplementationon
Design IssuesDesign Issues
One ClassAd One ClassAd One LDAP One LDAP EntryEntry
All in OneAll in One
Final Final RemarksRemarks
One ClassAd, One LDAP EntryOne ClassAd, One LDAP Entry… …
Since the MatchClassAd specialization performs a bilateral match Since the MatchClassAd specialization performs a bilateral match between two and only two ClassAds, it will return FALSE for every between two and only two ClassAds, it will return FALSE for every ClassAd describing the GRIS resources.ClassAd describing the GRIS resources.
GIISGIIS
Resource Resource BrokerBroker
Resource Resource BrokerBroker ????
[… […
requirements= requirements= other.ostype==”linux” && other.ostype==”linux” && other.schedulertype==”condorother.schedulertype==”condor
……]]
[… […
requirements= requirements= other.ostype==”linux” && other.ostype==”linux” && other.schedulertype==”condorother.schedulertype==”condor
……]]
JobAdJobAd
[ [ ……
schedulertype=”condor”schedulertype=”condor”
……]]
[ [ ……
schedulertype=”condor”schedulertype=”condor”
……]]
[ …[ …
ostype=”linux”ostype=”linux”
……] ]
[ …[ …
ostype=”linux”ostype=”linux”
……] ]
[ …[ …
releaseDate=2000/04/11releaseDate=2000/04/11
……] ]
[ …[ …
releaseDate=2000/04/11releaseDate=2000/04/11
……] ]
GRISGRIS
ResourceAdsResourceAds
No Such GRISNo Such GRISNo Such GRISNo Such GRIS
IntroductionIntroduction
State of the State of the ImplementatiImplementationon
Design IssuesDesign Issues
One ClassAd One ClassAd One LDAP One LDAP EntryEntry
All in OneAll in One
Final Final RemarksRemarks
All in OneAll in One… …
The second approach is based on the use of just one ClassAd describing The second approach is based on the use of just one ClassAd describing all the resources published by a given GRIS. all the resources published by a given GRIS.
[ [ ……
releaseDate=2000/04/11releaseDate=2000/04/11
schedulertype=”condor”schedulertype=”condor”
ostype=”linux”ostype=”linux”
……]]
[ [ ……
releaseDate=2000/04/11releaseDate=2000/04/11
schedulertype=”condor”schedulertype=”condor”
ostype=”linux”ostype=”linux”
……]]
ResourceAdResourceAd
globus-versionglobus-version
grid-info-hostgrid-info-host
globus-gram-schedulerglobus-gram-scheduler
IntroductionIntroduction
State of the State of the ImplementatiImplementationon
Design IssuesDesign Issues
One ClassAd One ClassAd One LDAP One LDAP EntryEntry
All in OneAll in One
Final Final RemarksRemarks
All in OneAll in One… …
On the other hand, it should be kept in mind that two different entries of On the other hand, it should be kept in mind that two different entries of the LDAP directory might contain one or more attributes with the same the LDAP directory might contain one or more attributes with the same name, but different values.name, but different values.
Therefore, any new value for an attribute already inserted in the Therefore, any new value for an attribute already inserted in the ClassAd must appended, thus creating a list of values. ClassAd must appended, thus creating a list of values.
grid-info-hostgrid-info-host
globus-gram-schedulerglobus-gram-schedulerCONDORCONDOR
[ [ ……releaseDate=2000/04/11releaseDate=2000/04/11schedulertype={”condor”,”lsf”}schedulertype={”condor”,”lsf”}ostype=”linux”ostype=”linux”……]]
[ [ ……releaseDate=2000/04/11releaseDate=2000/04/11schedulertype={”condor”,”lsf”}schedulertype={”condor”,”lsf”}ostype=”linux”ostype=”linux”……]]
ResourceAdResourceAd
globus-versionglobus-version
globus-gram-schedulerglobus-gram-schedulerLSFLSF
IntroductionIntroduction
State of the State of the ImplementatiImplementationon
Design IssuesDesign Issues
One ClassAd One ClassAd One LDAP One LDAP EntryEntry
All in OneAll in One
Final Final RemarksRemarks
All in OneAll in One… …
Nevertheless,Nevertheless, the match between the ResourceAd and the the match between the ResourceAd and the JobAd does not work yet. JobAd does not work yet.
GIISGIIS
Resource Resource BrokerBroker
Resource Resource BrokerBroker ????
GRISGRIS
No Such GRISNo Such GRISNo Such GRISNo Such GRIS
[… […
requirements= requirements= other.ostype==”linux” && other.ostype==”linux” && other.schedulertype==”condorother.schedulertype==”condor
……]]
[… […
requirements= requirements= other.ostype==”linux” && other.ostype==”linux” && other.schedulertype==”condorother.schedulertype==”condor
……]]
JobAdJobAd
ResourceAdResourceAd
[ [ ……releaseDate=2000/04/11releaseDate=2000/04/11schedulertype={”condor”,”lsf”}schedulertype={”condor”,”lsf”}ostype=”linux”ostype=”linux”……]]
[ [ ……releaseDate=2000/04/11releaseDate=2000/04/11schedulertype={”condor”,”lsf”}schedulertype={”condor”,”lsf”}ostype=”linux”ostype=”linux”……]]
IntroductionIntroduction
State of the State of the ImplementatiImplementationon
Design Issues Design Issues
One ClassAd One ClassAd One LDAP One LDAP EntryEntry
All in OneAll in One
Final Final RemarksRemarks
All in OneAll in One… …
The problem is within the description of the job requirements in The problem is within the description of the job requirements in ClassAd, that is, the operator == does not work when applied to ClassAd, that is, the operator == does not work when applied to an attribute whose value is a list. The member function must be an attribute whose value is a list. The member function must be rather used.rather used.
GIISGIIS
Resource Resource BrokerBroker
Resource Resource BrokerBroker ????
GRISGRIS
Match OkMatch OkMatch OkMatch Ok
[… […
requirements= other.ostype==”linux” && requirements= other.ostype==”linux” &&
Member(“condor”,other.schedulertype)Member(“condor”,other.schedulertype)
……]]
[… […
requirements= other.ostype==”linux” && requirements= other.ostype==”linux” &&
Member(“condor”,other.schedulertype)Member(“condor”,other.schedulertype)
……]]
JobAdJobAd
ResourceAdResourceAd
[ [ ……releaseDate=2000/04/11releaseDate=2000/04/11schedulertype={”condor”,”lsf”}schedulertype={”condor”,”lsf”}ostype=”linux”ostype=”linux”……]]
[ [ ……releaseDate=2000/04/11releaseDate=2000/04/11schedulertype={”condor”,”lsf”}schedulertype={”condor”,”lsf”}ostype=”linux”ostype=”linux”……]]
It should be pointed out that the JDL user must know which attribute It should be pointed out that the JDL user must know which attribute requires the member function for a feasible matching. requires the member function for a feasible matching.
IntroductionIntroduction
State of the State of the ImplementatiImplementationon
Design IssuesDesign Issues
One ClassAd One ClassAd One LDAP One LDAP EntryEntry
All in OneAll in One
Final Final RemarksRemarks
Final RemarksFinal Remarks In the author’s point of view the best solution is to use only one In the author’s point of view the best solution is to use only one
ClassAd representing a subset of the more meaningful attributes, ClassAd representing a subset of the more meaningful attributes, rather than the entire attributes published.rather than the entire attributes published.
Problem to be discussed: what information is needed for Problem to be discussed: what information is needed for the Broker and who has to publish itthe Broker and who has to publish it