Grid Interoperability Through Standards Dr. Alistair Dunlop
Project Manager
EU project: RIO31844-OMII-EUROPE
The OMII-Europe Standardisation process
Globus
OMII-UK
CROWN
Components
Components
IN
OUT
Re-engineering
Compliance testing and QA
Benchmarking
Component Integration
Repository
Evaluation Infrastructure
Identifying new components
Standards Implementation
EU project: RIO31844-OMII-EUROPE
Session Outline
• Overall architecture and components in OMII-Europe (Morris Riedel)
• Standards and implementations from OMII-Europe– SAML (Valerio Venturi) – BES/JSDL (Stephen Crouch)– RUS and UR (Gilbert Netzer)– GLUE II (Sergio Andreozzi)– DAIS, DMI and ByteIO (Neil Chue Hong)
• Future services
EU project: RIO31844-OMII-EUROPE
Future Work: 2008-2010
• Common implementations of specification and services for:– Data storage services
• Common data service layer building on DAIS, ByteIO, DMI and GSM
– Service discovery – Grid monitoring and service mangement– Grid billing and pricing– Grid visualisation and steering– Grid Authorisation
Architecture and SecurityOpen Standards used in real interoperability scenarios and use cases
Morris Riedel, Forschungszentrum Juelich (FZJ), Jülich Supercomputing Centre (JSC), Germany
Leader Infrastructure Integration (Interoperability) ActivityOpen Grid Forum 21, Seattle, 17th October 2007
EU project: RIO31844-OMII-EUROPE
Outline
• Motivation: e-Infrastructure Islands in Europe• OMII-Europe in Context• Common Vision: Interoperability Highway• Interoperability scenarios / use cases• Infrastructure to test interoperability scenarios• Interoperability scenario: WISDOM• Conclusion
EU project: RIO31844-OMII-EUROPE
• DEISA Grid (Supercomputing/HPC community)– Non WS-based UNICORE 5: Proprietary jobs (AJO/UPL)– No Virtual Organization Membership Service (VOMS), Full X.509– Suitable for massively parallel scientific jobs (MPI, …)
• EGEE Grid (mainly HEP community + others)– Non WS-based gLite: Proprietary jobs (JDL)– Proxy-based X.509 security, but proprietary VOMS support– Suitable for embarrassingly parallel scientific jobs
• Both Grids are currently not technical interoperable– Scientists cannot use one middleware to access both– Both Middleware’s had so far less adoption of open standards
e-Infrastructure Islands in EuropeDEISA [2]
EGEE [3]
EU project: RIO31844-OMII-EUROPE
OMII – Europe in ContextEC e-Infrastructures [4]
EU project: RIO31844-OMII-EUROPE
GridMiddlewares
EmergingOpen
Standards
others
End-usersvia clients& portals
„Interoperability highway“based on open standards
GOAL: Transparencyof Grids for end-users
GridResources
others
Common Vision: Interoperability Highway
UNICORE [5] gLite [6] GT [7] CROWN [8]
EU project: RIO31844-OMII-EUROPE
Infrastructure to test interoperability scenarios
Next steps in RED/ORANGE SAML- based Attribute Authority (AA) VOMS
gets central role & middleware independent
Venturi et al. [9]
EU project: RIO31844-OMII-EUROPE
• OMII – Europe implements open standards…– OGSA-BES, OGSA-RUS, WS-DAIS (OGSA-DAI), SAML (VOMS)…
• E.g: OGSA - Basic Execution Services (OGSA-BES)– In real deployments is not a ‘vanilla OGSA-BES interface’ available– Same exact “client” works not directly with gLite & UNICORE
• E.g. different security models: X.509 Proxies vs. full X.509 certificates• Solution in OMII-Europe: Agree on a Common Security Profile
• Grid interoperability: use components together…– e.g. OGSA-BES job submission secured via VOMS AuthZ
• Interoperability scenarios / use cases: e-Infrastructure use cases requiring resources in more than one Grid!
Interoperability Scenarios / Use cases
EU project: RIO31844-OMII-EUROPE
• The following interoperability scenarios have been discussed with members of the respective projects
• These scenarios demonstrate the required technical interoperability between e-Infrastructures in future that can be provided by OMII-Europe
• However, the negotiation for getting computational time on resources within these e-Infrastructures is still required by the respective projects
Disclaimer
EU project: RIO31844-OMII-EUROPE
• WISDOM (Wide In Silicio Docking on Malaria)– WISDOM aims at developing new drugs for Malaria– WISDOM uses EGEE for large scale in silicio docking
• Comp. method for prediction of whether one molecule will bind to another
– Using AutoDock and FlexX software provided via gLite in EGEE– Output is a list of best chemical compounds (potential drugs)
• That is not the final solution, only a potential list of drugs
• Refine best compound list via molecular dynamics(MD)– Fast MD computations use highly scalable AMBER in DEISA
• AMBER (Assisted Model Building with Energy Refinement) , version 9
• Goal: Accelerate drug discovery using EGEE + DEISA
Interoperability Scenario: WISDOM (1)
WISDOM [1]
EU project: RIO31844-OMII-EUROPE
Interoperability Scenario: WISDOM (2)
EU project: RIO31844-OMII-EUROPE
Conclusion
• Solutions for European e-Infrastructure Islands– Adopt components from OMII-Europe to gain interoperability
• Why OMII – Europe components…– Components going through quality assurance steps– Components adopt standards whereever possible– Components tested in real interoperability scenarios (e.g. WISDOM)
• Not only interesting for European e-Infrastructures– Beneficial for all Grids that adopt Globus, gLite, and UNICORE
• E.g. National German Grid Initiative (D-Grid)
• Continuing work in the open standards working groups!– „Interoperability highway…“ realize the „true global Grid“
EU project: RIO31844-OMII-EUROPE
References
• [1] WISDOM Project, http://wisdom.eu-egee.fr/
• [2] DEISA Project, http://www.deisa.org
• [3] EGEE Project, http://public.eu-egee.org/
• [4] European e-Infrastructures, http://www.beliefproject.org/cookbook/cookbook-intro/view
• [5] European UNICORE Grid middleware, http://www.unicore.eu
• [6] European gLite Grid middleware, http://glite.web.cern.ch/glite/
• [7] US Globus Toolkit, http://www.globus.org/
• [8] CROWN, http://www.crown.org.cn/en/
• [9] Using SAML-based VOMS for Authorization within Web Services-based UNICORE Grids, Venturi et al., UNICORE Summit 2007 @ EuroPar 2007
• [10] Improving Quantum Computing Simulations, http://www.deisa.org/applications/projects2005-2006/iqcs.php
EU project: RIO31844-OMII-EUROPE
Acknowledgements
• Open Middleware Infrastructure Institute for Europe
– OMII – Europe project under EC grant RIO31844-OMII-EUROPE, duration May 2006 - April 2008
• Jülich Supercomputing Centre (JSC)of Forschungszentrum Jülich (FZJ) in the HELMHOLTZ association
Forschungszentrum Jülichin der Helmholtz-Gesellschaft
EU project: RIO31844-OMII-EUROPE
IGIIW @ e-Science 2007
International Grid Interoperability & Interoperation
Workshop
in conjunction with
e-Science 2007, Bangalore, India
http://www.omii-europe.org/OMII-Europe/igiiw2007.html
EU project: RIO31844-OMII-EUROPE
• IQCS (Improving Quantum Computater Simulation)– One project of DECI (DEISA Extreme Computing Initiative)
• Evaluation period/test runs of IQCS code in EGEE– IQCS scientists use currently their own desktop machines or some
spare resources within institutes to develop/evaluate/test their code– That‘s slow and the management doing it personally is cumbersome
• This can be significantly improved using EGEE clusters!
• Final IQCS code production runs within DEISA– Leverage massive amount of CPUs/memory available on HPC
resources
• Goal: Accelerate code development with EGEE + DEISA
Optional: Interoperability Scenario: DECI IQCSDEISA DECI IQCS [10]
EU project: RIO31844-OMII-EUROPE
Optional: Interoperability appraoch in OMII-Europe
e-InfrastructureInteroperability by using more
than one technology!
Venturi et al. [9]
EU project: RIO31844-OMII-EUROPE
Questions for JRA3 – Task 2
Morris Riedel
JRA 3 Team
EU project: RIO31844-OMII-EUROPE
Attribute Exchange Profile
• Problem statement– Interoperable, pluggable, replaceable attribute services– Easier aggregation of attributes coming from difference sources
• OGSA Architecture envision the use of authorization policies coming from different stakeholders
• Solution– Standard interface for requesting attribute assertions
EU project: RIO31844-OMII-EUROPE
Attribute Exchange Profile
• Problem statement– Attributes assertions understood across middleware boundaries
• Solution– Agreed format for attribute assertions
EU project: RIO31844-OMII-EUROPE
Status of the specification
• The specification is being worked on inside the OGF OGSA Authorization Working Group– Basically profiles existing specification (OASIS, WS-I) for use
with Grid Services• Bases are solid
– OASIS SAML V2.0 set of specifications• SAML V2.0 Deployment Profiles for X.509 Subjects
• Profile presented on Monday• Those interested should join the OGS Authorization WG
http://www.ogf.org/gf/group_info/view.php?group=ogsa-authz-wg
EU project: RIO31844-OMII-EUROPE
Specification Implementers
• VOMS– gLite and OMII-Europe developed, used in EGEE,
OSG, Naregi– Implementing the service
• UNICORE– Implementing the client part of the profile in OMII-
Europe• Part of the 6.2 release planned for early 2008
– Implementing the service in the chemomentum project• GridShib
EU project: RIO31844-OMII-EUROPE
Implementation status
• OMII-Europe is re-engineering the Virtual Organization Membership Service (VOMS) to meet the specification
• VOMS is developed also in EGEE• VOMS is an Attribute Authority focused on VO management
– It allows users to register to a VO and being assigned attributes such as groups and roles
– The attributes are used by Grid Services to drive authorization decisions
• VOMS is already used in EGEE, OSG and Naregi– Plugin for the globus Authorization framework
• OMII-Europe is also adding VOMS support to UNICORE– UNICORE is implementing the client part of the profile
EU project: RIO31844-OMII-EUROPE
Implementation Status
• Alpha version of the VOMS SAML Service available since May 2007 for integration testing purposes– Demonstrated at OGF 20 in Manchester in conjunction with
UNICORE OGSA-BES• Beta version available early November
– Then into the QA process of OMII-Europe, that comprises standard compliance testing
• VOMS SAML Fed back to gLite early 2008• OMII-Europe support for VOMS
– Available from version 6.2 early 2008
EU project: RIO31844-OMII-EUROPE
Implementation Plans
• Undergoing integration with BES services re-engineered by OMII-Europe– UNICORE OGSA BES already done, demonstrated at OGF 20– gLite CREAM BES nearly done– May be demonstrated at SC07
• Planned integration with other OMII-Europe re-engineered components– RUS services: SGAS, gLite DGAS, UNICORE OGSA RUS– OGSA DAI
• Integration with BES services is a proof of concept of the first addressed issue– Availability of attributes across middleware boundaries
EU project: RIO31844-OMII-EUROPE
Further plans
• Shibboleth 2 on its way out– GridShib will implement the same interface
• We plan to test integration scenarios– Shibboleth attributes concern Home Institution
Affiliation– VOMS attributes concern Virtual Organization
• This would be proof of concept of the second issue addressed– Easy aggregation of attributes coming from different
sources
Job Submission: OGSA-BES & JSDLMoreno Marzolla ([email protected]), Stephen Crouch SOTON
EU project: RIO31844-OMII-EUROPE
Contents
• The problem of incompatible job submission systems
• The solution: interoperability through standards– OGSA-BES– JSDL
• Standards status and limitations• Implementations of standards within OMII-Europe
– Computation endpoints– Meta-scheduler
• Future interactions with other OMII-Europe activities
EU project: RIO31844-OMII-EUROPE
Setting the Scene: Grid Job Submission
• User problem:– User (e.g. researcher) has a problem they need to solve computationally– They do not have local access to appropriate resources in terms of
computation and/or applications• A non-local provider:
– Has the required computation/application resources– Wishes to allow them to be shared by appropriate users
• Grid job submission provides the technological ‘glue’ to allow users to utilise the provider’s computational/application resources
UserJob
Comp/AppResources
Job Submission
EU project: RIO31844-OMII-EUROPE
The Problem
• Different job submission systems often incompatible in terms of– Interface– Job description– Security (more general problem)
• Overhead in terms of– Use – learning curve– Infrastructure support – multiple client installations– Application development – knowledge of multiple APIs
• Interoperation solutions expensive in terms of development & maintenance
ClientApp
Comp/AppResources
Job Submission A
Client (A)
Client (B)
Job Submission B
Comp/AppResources
…
… …Desc A
Desc B
EU project: RIO31844-OMII-EUROPE
Solution: Interoperability through Standards
• Two key standards for two key elements:– The Basic Execution Service web service interface (OGSA-BES)
• 1 instance of job container within OGSA-EMS (Execution Mngt Service)• Handles basic job lifecycle management• Defines simple (but extendable) job state model
– Pending, running, cancelled, failed or finished– The Job Submission Description Language (JSDL)
• Specify job executable, data staging and resource requirements
ClientApp
Comp/AppResources
Job Sub. A
JobSub. B
Comp/AppResources
BESClient
JSDL BESI/F
BESI/F
JSDL
… …
…
EU project: RIO31844-OMII-EUROPE
Summary of OGSA-BES
BES-Management Port-type
StopAcceptingNewActivities Request that the BES stop accepting new activities
StartAcceptingNewActivities Request that the BES start accepting new activities
BES-Factory Port-type
CreateActivity Request the creation of a new activity
GetActivityStatuses Request the status of a set of activities
TerminateActivities Request that a set of activities be terminated
GetActivityDocuments Request the JSDL documents for a set of activities
GetFactoryAttributesDocument Request XML document containing BES properties
BES-Activity Port-type (optional)
GetStatus Request the status of an activity
Terminate Request that an activity be terminated
GetDocument Request the JSDL document for an activity
GetActivityAttributesDocument Request XML document containing activity properties
EU project: RIO31844-OMII-EUROPE
JSDL Example I<JobDefinition xmlns="http://schemas.ggf.org/jsdl/2005/11/jsdl"> <JobDescription> <JobIdentification> <JobName>cat job</JobName> <Description>cat job description</Description> <JobAnnotation>no annotation</JobAnnotation> <JobProject>an example project</JobProject> </JobIdentification> <Application> <POSIXApplication xmlns="http://schemas.ggf.org/jsdl/2005/11/ jsdl-posix"> <Executable>/bin/cat</Executable> <Argument>dir1/file1.txt</Argument> <Output>stdout.txt</Output> <Error>stderr.txt</Error> </POSIXApplication> </Application>
EU project: RIO31844-OMII-EUROPE
JSDL Example II <DataStaging> <FileName>dir1/file1.txt</FileName> <CreationFlag>overwrite</CreationFlag> <Source> <URI>http://server.domain/helloworld.txt</URI> </Source> </DataStaging> <DataStaging> <FileName>stdout.txt</FileName> <CreationFlag>overwrite</CreationFlag> <DeleteOnTermination>true</DeleteOnTermination> <Target> <URI>ftp://anonymous:[email protected]:45521/public/output/stdout.txt</URI> </Target> </DataStaging> </JobDescription></JobDefinition>
EU project: RIO31844-OMII-EUROPE
Standards Status & Limitations
• Both JSDL & OGSA-BES have reached version 1.0 (recommendation)
• Future support being considered– Parametric jobs – being considered as JSDL
extension– Collections of jobs with dependencies
• Neither addresses security– Fundamental requirement for interoperability in real
deployments– Mutually compatible security model (JRA3/Security,
JRA1/VOMS)
EU project: RIO31844-OMII-EUROPE
Implementations: Endpoint Supportfor BES & JSDL
• BES 1.0/JSDL 1.0 support developed in OMII-Europe for– UNICORE 6.1 (Jan/Feb 2008):
• BES/JSDL support in vendor middleware
– gLite 3.1:• CREAM-BES component based on CREAM-CE
• Available through OMII-Europe repository in Dec 2007
– Globus 4.2:• Some BES/JSDL support exists within vendor middleware
• Work progressing – target release date TBD
• Support for BES within other job submission implementations:– OMII-UK, Beihang, Microsoft, University of Virginia, Platform
Computing
EU project: RIO31844-OMII-EUROPE
Implementations: Meta-schedulingsupport for BES & JSDL
• BES/JSDL 1.0 support in CROWN– Progressing towards support for BES 1.0
• CROWN Meta-scheduler hosted within Globus 4.2:– Accept BES submissions, delegate to BES endpoints based on
policy– Release targeted March 2007, available from OMII Europe
repository
Res.Job
Sub. ABESI/F
Res.Job
Sub. BBESI/F
ClientApp
BESClient
JSDL
JSDL
MetaScheduler
BES I/FJSDL
EU project: RIO31844-OMII-EUROPE
Future Interactions with OMII-Europe Activities
• Security (JRA3)– Support common security profile
• Compliance testing (SA2) w.r.t. OGSA-BES/JSDL– Test endpoints– Meta-scheduler ‘plugs in’ to compliance test
• Benchmarking (JRA4)• Training (NA3)
Accounting in OMII-EuropeGilbert Netzer (KTH)
EU project: RIO31844-OMII-EUROPE
Outline
• Accounting Standards– Overview– Usage Record Format– Resource Usage Service
• OMII-Europe Accounting Activity– Goals– Used Standards– Schedule– OMII-Europe Accounting Scenarios
• Questions
EU project: RIO31844-OMII-EUROPE
Benefits of Standardization
• Many (batch) systems are alike– All batch systems collect similar metrics– However units, data format and semantics differ
• Large systems are heterogeneous– Standards help in building these systems– Off-the-shelf components can be plugged together
• Standards give guidance for implementations– Define parts of the requirements– Can help defining the scope of an application
• Promote sharing of knowledge
EU project: RIO31844-OMII-EUROPE
OGF Standards for Accounting
• Usage Record Format (URF, GFD-R-P. 98)– Recommends a document format for usage information– Does not address transport of information– Proposed Recommendation status– Interoperability workshop scheduled for OGF22
• Resource Usage Service (RUS, GWD)– Specifies means to transport usage information– Uses the URF as data format– Working Draft stage– Meetings were held at OGF19, 20 and 21
EU project: RIO31844-OMII-EUROPE
Accounting Standards Usage Scenario
Usage Records
Pro
du
ce
s
URStorage
RUS
Resource
SensorMetering Component
`
Grid User
Publish
Submit Job
Submission Endpoint
Queue
Accounting System
Ch
ec
k
Qu
ota
Usage Record Format (UR)
Resource Usage Service
(RUS)
EU project: RIO31844-OMII-EUROPE
OMII-Europe Goals
• Make existing accounting infrastructure interoperable• Use OGF standards as vehicle:
– Usage Record Format (UR)– Resource Usage Service (RUS)
• Address the three middlewares of the project:– Distributed Grid Accounting System (DGAS)– SweGird Accounting System (SGAS)– UNICORE-RUS
• First step, focus on the existing basics only• Augment existing systems to allow easy transition
EU project: RIO31844-OMII-EUROPE
OMII-Europe Schedule
• Alpha versions demonstrated at OGF20 (Manchester)– DGAS client uploads/downloads from SGAS server– UNICORE demonstrated using RUS for monitoring
• Beta versions November-December 2007– Will have both server and client components for all
three middlewares– Will be available for download from repository
• Final versions for general public release– Will be ready in March 2008
• Software will be released via OMII-Europe Repository– We also try to feed SW back to the MW distributions
EU project: RIO31844-OMII-EUROPE
Alpha Demonstration - OGF20
OMII-Europe added value (RUS)
UNICORE gLite DGAS Globus SGAS
End User Applications
Services
Resources
LLView DGAS Clients SGAS Clients
UNICORE RUS LUTS
UNICORE UR Generator DGAS Sensor
UNICORE Alpha at German eScience 2007
EU project: RIO31844-OMII-EUROPE
OMII-Europe Accounting Scenario
OMII-Europe added value (RUS)
UNICORE gLite DGAS Globus SGAS
End User Applications
Services
Resources
LLView DGAS Clients SGAS Clients
UNICORE RUS
HLR RUS Service LUTS
UNICORE UR Generator DGAS Sensor JARM
LegacyInteraction
EU project: RIO31844-OMII-EUROPE
Standards in OMII-Europe Accounting
• Phase I (Current Project)– Uses RUS draft version 17
• Is available• Has passed public comment phase
– Uses late URF draft• This is used by RUS draft 17• Different namespace than GWD-R.P. 98• Minor differences (e.g. new GlobalUserName)
– RUS is work in progress!• Phase II (Next Project, 2008-2010)
– Upgrade to final RUS&URF recommendation
EU project: RIO31844-OMII-EUROPE
Usage Record Format
• XML Document Format for Usage Information• Approach
– Defines many elements for different metrics– Most elements optional, implementation picks– Extension framework for further properties
• Limitations– Only computational jobs considered in version 1– One record per job, no summarization– VO membership information was not considered– Limited extension framework, only string content
EU project: RIO31844-OMII-EUROPE
Anatomy of a UsageRecord <JobUsageRecord xmlns="http://schema.ogf.org/urf/2003/09/urf" xmlns:urf="http://schema.ogf.org/urf/2003/09/urf"> <RecordIdentity urf:recordId="http://www.emsl.pnl.gov/mscf/colony/PBS.1234.0" urf:createTime="2003-08-13T18:56:56Z" /> <JobIdentity> <LocalJobId>PBS.1234.0</LocalJobId> </JobIdentity> <UserIdentity> <LocalUserId>scottmo</LocalUserId> </UserIdentity> <Charge>2870</Charge> <Status>completed</Status> <Memory urf:storageUnit="MB">1234</Memory> <ServiceLevel urf:type="QOS">BottomFeeder</ServiceLevel> <Processors>4</Processors> <ProjectName>mscfops</ProjectName> <MachineName>Colony</MachineName> <WallDuration>PT1S</WallDuration> <StartTime>2003-08-13T17:34:50Z</StartTime> <EndTime>2003-08-13T18:37:38Z</EndTime> <NodeCount>2</NodeCount> <Queue>batch</Queue> <Resource urf:description="quoteId">1435</Resource> <Resource urf:description="application">NWChem</Resource> <Resource urf:description="executable">nwchem_linux</Resource></JobUsageRecord>
Unique ID &TimestampReference to Job
Reference to User
Usage Information
Extensions
EU project: RIO31844-OMII-EUROPE
Anatomy of the Resource Usage Service
• Defines a web-service interface (draft 17):– Store (insert) Usage Records– Extract information using XPath queries
• Whole UsageRecords wrapped with audit information• Only the ID assigned by the RUS (not RecordId!)
– Modify stored Usage Records• Replace whole UR with new UR• Modify UR using XUpdate expression• Atomically increment numeric property of a UR
– Delete stored Usage Records by ID or XPath query– Get local constraints (mandatory elements)
EU project: RIO31844-OMII-EUROPE
Limitations of RUS
• Extract always returns results in one message– Problems with large queries (memory, timeouts)– Planned to support enumerations
• No summarization possible– Cannot return e.g. total jobs this month– Planned for Advanced RUS specification
• Not aware of VOs in standard way– Can use UR extensions, standardization in URF v. 2
• No federation support– Need to always query whole set, no diffs.
EU project: RIO31844-OMII-EUROPE
OMII-Europe's Contribution in Accounting
• Usage Record Working Group– Plan to supply UR-data to interop WS at OGF22
• Resource Usage Service Working Group– 2 of 3 co-chairs come from OMII-Europe– Will provide three independent implementations– Will provide implementation experience
EU project: RIO31844-OMII-EUROPE
References
• OMII-Europe Website:http://www.omii-europe.org
• UR-WG GridForgehttp://forge.ogf.org/sf/projects/ur-wg
• RUS-WG GridForgehttp://forge.ogf.org/sf/projects/rus-wg
GLUE 2: Common Information Model for Grid resources
Sergio Andreozzi, INFN
17 Oct 2007, Seattle, WA, USA
EU project: RIO31844-OMII-EUROPE
Problem Statement
• How do we describe resources shared in Grid systems in order to enable:
– Resource awareness
– Resource discoverability
– Resource requirements expression
– Resource basic monitoring
EU project: RIO31844-OMII-EUROPE
Use Case 1
• I want to run my job on an execution environment characterized by:
– OS• Linux, Distribution X, version Y
– CPU Archicture • IA64
– Available software packages: • S1, S2
EU project: RIO31844-OMII-EUROPE
Use Case 2
• I want to know
– how many job slots are used by members of the VO A
– what is the global available storage space for the users of VO B
EU project: RIO31844-OMII-EUROPE
Focus on Computing Resources
• OGSA-BES and JSDL are already considered by OMII-Europe
• They lack a common description of Grid resources suitable for discovery, monitoring and scheduling
• Many descriptions exist– e.g.: GLUE Schema, NorduGrid
Schema
• Barrier to interoperable Grid systems
EU project: RIO31844-OMII-EUROPE
How Are We Contributing
• Co-leading the OGF GLUE WG which purpose is to define the next-generation information model for Grid resoruces as a community standard
• OGF GLUE WG Goals:– define a use case document collecting use cases from different
Grid projects/infrastructures– define a conceptual model defining the abstract schema GLUE
2.0 satisfying the collected use cases. – develop reference implementations
• Timeline: go public comments by OGF 22 (Jan 2008)
EU project: RIO31844-OMII-EUROPE
Relationship to other OGF WGs
JSDL
GLUE
Reference Model
SAGA
GSMOGSA Res.Mgt.
Used to express requirements inCommon service descripton
for discovery API
Used to describe exposed resources
Should fit into the picture
Should fit into the picture
BES
Used to describe exposed resources
EU project: RIO31844-OMII-EUROPE
How Are We Contributing
• Rendering of the GLUE conceptual model as XML Schema and inclusion into the OMII-Europe repository– Other renderings like LDIF are expected
• implementation of information providers – for BES implementations developed in the context of OMII-
Europe– leveraging DMTF standards
• Inclusion of this capability into the OMII-Europe Repository
EU project: RIO31844-OMII-EUROPE
Activity Status
• OGF Documents– Specification and Use Cases drafts available– http://forge.ogf.org/sf/docman/do/listDocuments/projects.glue-wg/docman.root.drafts
• Rendering as XML Schema– First version to be issued in Nov 2007 based on OGF21 draft
• Information providers– Prototype framework based on OpenPegasus to be released in
Nov 2007
• Production Quality full implementation by Apr 2008
EU project: RIO31844-OMII-EUROPE
Adoption and Related Efforts
• The output of this activity will be adopted by the OMII-Europe middleware providers
• The software will be publicly accessible from the OMII-Europe repository and will be advertised for larger adoption
• Other projects will make their own implementation of the GLUE spec (e.g. ARC/NorduGrid)
Data Specs: DAIS, OGSA ByteIO, OGSA-DMINeil Chue Hong
OMII-UK
EU project: RIO31844-OMII-EUROPE
Overview
• Data specifications and OMII-Europe– Database Access and Integration Service (DAIS)– OGSA ByteIO (ByteIO)– OGSA Data Movement Interface (OGSA-DMI)
• What is the problem• What is the status of the specification• What is the status of the implementations
EU project: RIO31844-OMII-EUROPE
Uniform access to structured data
• Each database vendor typically supports different methods of access and different data models– difficult to federate
• DAIS provides a uniform interface for data access that allows– location independence– heterogeneity– composability
• It also presents interfaces that allow virtualisation of database resources, and result sets
EU project: RIO31844-OMII-EUROPE
DAIS Specifications
WS-DAI
WS-DAIR WS-DAIX WS-DAIO
Sets general pattern for DAIS realisations
Currently defined
Draft at GGF14
RDF
In discussion
EU project: RIO31844-OMII-EUROPE
Data Service Composition
GDSClient
GDSClient
Client
1 Operation
GDSClient
2
DB
Operation
Operation
OperationDB
4
Operation
Operation
Operation
DB
GDS
GDS
GDS
3
Operation
Operation
Operation
DB
GDS
GDS
GDS
Client5
Operation
Operation
Operation
DB
GDS
GDS
GDS
EU project: RIO31844-OMII-EUROPE
Complex Scenario
GFactory G Q ES F
GFactory G Q ES F
GFactory G Q ES F
N 2
N 1
N3
GC lie n tGG D S
GG D S
G D Q
G D T
G D Q S
N 0G D S
GFactory G Q ES F
N4
p erform (Q u ery)1
cre a te S e rv ice
cre a te S e rv ice2
cre ate S e rvi ce
2
2
GG D S G Q ES 2
G D T
GG D S G Q ES 3
G D T
GG D S G Q ES 1
G D T
GG D S G Q ES 1
G D T
p erform (Q u ery S u b p la n )
p erform (Q u ery S u b p la n )
perform(Q
uer ySu bpl an)
3
s eq u en t ial_ s can
red u ce (p r o tein ID ,s eq u en ce )
s eq u en t ial_ s can ( ter m = 8 3 7 2 )
red u ce (p r o tein ID )
h as h _ jo in(p .p r o tein ID = t.p r o tein ID )
3
o p erat io n _ callb la s t(p .s eq u en ce)
red u ce (p .p r o tein ID , b la s t)
o p erat io n _ callb la s t(p .s eq u en ce)
red u ce (p .p r o tein ID , b la s t)
3
W e b S e rvi ce s (B L A S T)
resu lts
resu lts
resu lts
4
1144
EU project: RIO31844-OMII-EUROPE
DAIS
• The OGSA-DAI project has been heavily involved in standards process– implementation of DAIS proposed recommendation in
progress• Work in OMII-Europe has concentrated on
– interoperability across Globus, UNICORE and gLite– integration with VOMS security
• Other implementations of DAIS from– Ohio State University– EGEE (DAIS interface to AMGA)– IBM
EU project: RIO31844-OMII-EUROPE
DMI Scenario
EU project: RIO31844-OMII-EUROPE
User moves data from A B
• Does not care how it is done• Does not want to:
– Deal with different protocols– Deal with different security models– Learn new interface for different data types
• Handle structured & un-structured data– Agnostic about the data semantics
77
EU project: RIO31844-OMII-EUROPE
ByteIO: POSIX like
• A concise, standard way of interacting with grid data resources as POSIX-like files– interfaces defined for random access and streamable
• With other specs, allows a file and directory-like view of a Grid and many other simplified views of data– shell, file explorer, database result sets, pipes
• ByteIO Spec out of public comment with four implementations – UNICORE implementation by FZJ– OGSA-DAI implementation by EPCC– Clean room implementation by FLE– Genesis II implementation by UVa
EU project: RIO31844-OMII-EUROPE
Staging files in Job Submission
• HPC Profile enables interoperable job submission systems
• OGSA-DMI provides standard way of staging files in and out
• OGSA ByteIO allows part of a large remote file to be staged in without transferring the whole file
EU project: RIO31844-OMII-EUROPE
Summary• OMII-Europe is a 24 Month EU funded project with 16 partners to establish grid
infrastructure interoperability through implementing a set of agreed open standards on all middleware platforms
• OMII-Europe is implementing a common Job submission system, Accounting service, Database service, Virtual Organisation service, Information Service and Security model for major middleware platforms. This will allow identically specified jobs to be run, managed and migrated to different middleware platforms thus enabling “the Global Grid”
• Initial versions of BES, VOMS and security service have already enabled UNICORE and EGEE resources to be used by the same job
• A complete set of fully interoperable services will be available in spring 2008 with early versions of some services available already
• Users can try interoperability on the OMII-Europe evaluation infrastructure, or obtain services for installation on their own resources from the OMII-Europe repository
• We anticipate OMII-Europe services to be integrated into standard middleware distributions as well as deployed on large scale e-infrastructures such as EGEE and DEISA
• OMII-Europe will request continuing funding in the September EU call to support the existing services and provide further services in the areas of data and grid management
EU project: RIO31844-OMII-EUROPE
Further Information
• http://www.omii-europe.org
Q & A