teragrid information services december 1, 2006 jp navarro gig software integration
TRANSCRIPT
TeraGrid Information ServicesDecember 1, 2006
JP NavarroGIG Software Integration
December 1, 2006 TeraGrid Information Services Page 2
Outline
• Motivation• Deployed Infrastructure• Phase 0 - thru June 2006• Phase 1 - User Portal, Docs, …• Phase 2 - Kit Registration• Futures - What would you like?
December 1, 2006 TeraGrid Information Services Page 3
Motivation
The TeraGrid, like any other grid, manages metadata about the organizations, individuals, and infrastructure that make up a grid.
Effectively managing this metadata is a critical element of the TeraGrid's infrastructure, software, and services strategy.
The TeraGrid’s Information Services aims to:
• Provide standards based grid access to TeraGrid information
• Accessible thru programmatic interfaces
• Initially focused on local RP infrastructure metadata
• Initially based on Globus 4 MDS4 services
Information services will be the common Grid interface thru which a resource can publish it’s existence and provide information about its capabilities.
December 1, 2006 TeraGrid Information Services Page 4
Use Cases / Consumers
• Meta-schedulers and co-schedulers
• User portal
• Gateways
• TeraGrid documentation
• Other Grids
• Applications
• Application hosting environments
December 1, 2006 TeraGrid Information Services Page 5
Deployed Infrastructure Overview
mds.teragrid.org
RP
Resource1 MDS
Resource2 MDS
Resource3 MDS
RP
Resource1 MDS
RP
Resource1 MDS
Resource2 MDS
Other MDSs
WebMDS
December 1, 2006 TeraGrid Information Services Page 6
Deployed Infrastructure Details
• Tooling– MDS4 from CTSS v3 Globus 4.0.1 -r3
– Is a WSRF service that runs in a GT4 container
• RP Configuration– RPs responsible for their data
– RPs are pre-configured to register with global mds.teragrid.org
– Using a combination of push/registration and pull
• Central (aggregating) Configuration– Aggregating mds.teragrid.org has all the information from all RPs
– A public anonymous access service on port 8443https://mds.teragrid.org:8443/wsrf/services/DefaultIndexService
– A secure TeraGrid authenticated service on port 8448https://mds.teragrid.org:8448/wsrf/services/DefaultIndexService
– WebMDS views of some of the data, follow links at:http://mds.teragrid.org/
December 1, 2006 TeraGrid Information Services Page 7
Phase 0
• Focused on hypothetical meta-scheduling needs• Delivery was targeted for June 2006, when CTSS v3 went production• Significant MDS team design and development effort,
including new MDS core capabilities to load arbitrary XML data• Published data:
– queue details– cluster and sub-cluster– file-system– individual compute node details
• Data sources:– Monitoring data from Nagios, Ganglia, and ClueMon– Local resource managers– Raw XML files for sites not running these 3 monitoring systems
• Moderate administrator configuration• Background material:
– http://www.teragridforum.org/mediawiki/images/9/9a/Mds4-TG-overview.pdf
– http://www.teragridforum.org/mediawiki/images/5/5b/Mds4-TG-march2006.pdf
December 1, 2006 TeraGrid Information Services Page 8
Phase 1
• Overview– Effort started in June/July 2006
– Phased approach focused on specific real consumers
– Lead by DIVS AD Kelly Gaither w/ SI technical assistance
– Expected to go production late in Q4 ‘06 or early Q1 ‘07
• Primary focus in delivering information to the User Portal– Scheduling Load– Queue Contents (restricted access)
• Motivation for Secure/Authenticated MDS4 services• Status
– Initial information provider deployed at TACC and UC/ANL– Actively in development by user portal– Soon will publish a formal change management document
•With a TeraGrid wide testing and production plan
• Other goals for phase 1– Create information services documentation repository– Engage TeraGrid groups in Metadata and Information Services discussions– Trying to configure some Phase 0 providers to verify they work
December 1, 2006 TeraGrid Information Services Page 9
Phase 2
• Overview– Effort started in Q4 2006– Lead by SI co-AD JP Navarro– Expected to go production mid to late Q1 ‘07
• Primary focus in delivering CTSS v4 kit registration information
• Required component of the CTSS v4 Core kit– http://www.teragridforum.org/mediawiki/index.php?title=CTSS_-
_Coordinated_TeraGrid_Software_and_Services
• Actively in development– Mach-up of kit registration– Design document– WebMDS views
• Will likely include kit specific registration information– Software registration (SoftEnv keys)– Services registration– Other kit specific attributes
December 1, 2006 TeraGrid Information Services Page 10
Futures
• Community Software Registration– Software installed by the community on TeraGrid Resources
• Community Service Registration– Hosted community services that register themselves– On RP resources that are willing to host services– Application Hosting Environment support
• Arbitrary XML publishing– Application, user, or job level
• Key consumers– Gateways– Meta-schedulers, co-schedulers, workflow systems– User portal– User documentation
December 1, 2006 TeraGrid Information Services Page 11
Useful links
• TeraGrid Information Services Pagehttp://www.teragridforum.org/mediawiki/index.php?title=Information_Services
•TeraGrid MDS4 Implementation Plan Pagehttp://www.teragridforum.org/mediawiki/index.php?title=MDS4_Implementation_Plan
•TeraGrid MetaData Cataloghttp://www.teragridforum.org/mediawiki/index.php?title=Metadata_Catalog
•Metadata, catalogs, and information services [email protected]
•Globus MDS4 documentationhttp://www.globus.org/toolkit/docs/4.0/info/