ILDG Middleware Status

Presented By: Bálint Joó, Jlab, USAWorking Group Members:

G. Beckett (EPCC, UK) T. Boku (CCS Tsukuba, Japan)

D. Byrne (EPCC, UK) P. Coddington (CSSM, Australia)

M. Ernst (DESY, Germany) M.Sato (CCS Tsukuba, Japan, Convener)

J. Simone (Fermilab, USA), W. Watson (Jlab, USA, Convener)

Contents

● Reminder of Our Task

● Reminder of Our Architecture

● Summary of Current Discussions

● Future plans

Our Mission

● We develop standard interfaces that facilitate the interoperation of a grid of grids

● Local grids

– Control Local Policies (Eg Security)

– Control Local Grid Implementations● Choice of software components

Our Architecture

● Simple veneer over local implementations

– Minimally intrusive (wrapper?)

– Not too burdensome to implement

● Language and platform neutral if possible

● Our choice: SOAP Web Services

Our Architecture: Components

Replica CatalogueWeb Service

(RC)

Metadata CatalogueWeb Service

(MDC)

Security Infrastructure(Identity management

access controlencryption)

SRM service (data management)

XML ILDG service

description file

Logical File Name (LFN, formerly GFN)

ILDG Service File

● Publishes Service Endpoints

● Each collaboration currently has entries for MDC and RC

● Started filling this out at MW Meeting in Tsukuba

● http://www.lqcd.org/ildg/Services.xml

● http://www.lqcd.org/ildg/Services.xsd

● Entries for LatFor & CSSM

Logical File Names (LFN)

● Changed to LFN from GFN to fit better with MDWG usage

● Unique and Persistent name for files

● MWWG mandated URI format

lfn://<ILDG Collaboration>/<Collaboration Specific Path>

● Collaboration indexes service file

● Collaborations control their own path

MDC Web Service

● Maps metadata to LFNs

● Provides “Virtual XML Database”

● Prototype WSDL definition by M. Sato

● Production MDC Web Services deployed by LatFor & CSSM

● Interface definition in flux

– Raw queries – variable return types

– Targeted Queries – return LFNs

Replica Catalog Service

● Maps (data) LFNs to URL

● Intention was to map to SRM URL

– SRM component deals with transfer

● SRM requirement temporarily relaxed – RC can return any URL

● No standard download protocol

● Client must have appropriate tool

● Prototype implementation at Jlab

SRM Service

● SRM definition v3 almost ready

– Fewer backward incompatible changes than v1->v2 transition.

– Migration should be simple(r)

● Popular dCache storage management software still uses v1.x (Fermilab, DESY) but Jlab uses v2. v1 and v2 incompatible

● ILDG uses v2 SRM, but SRM use is not mandated currently

Security Infrastructure

● Currently quite nebulous

● Previously defined 'ILDG Group Files'

– XML Documents containing user certificates

– Not very grid-like, not best match with LatFor preference/implementation

● Currently considering VOMS service

● Security not explicit in current interfaces

WG Organization

● Had very nice and productive meeting in Tsukuba in Oct 2005. Minutes on ILDG web site.

● Instituted more frequent teleconference calls – first of these was on Dec 5. Minutes on ILDG web site. Next call Jan 23, 2006.

● Middleware BLOG set up on ILDG web site to aid reporting progress.

Plans

● Finish up discussions and make a decision on using VOMS by end of February

● US basic service up with some data by end of January.

● EPCC to convert MDC GUI browser to work with Web Services

● Interoperability testing Feb-May 2006.

● Interoperable system by June 2006.

Current Discussions

● MDC

– Need to fix interface● Small inconsistencies between WSDL

from LatFor and Japan – to be fixed● Clarify semantic meaning of query

functions – Raw queries – return results appropriate to

query

– LFN queries – return markovChainUR or dataLFN

– Need document retreival queries ?

<?xml version=”1.0” ?><gaugeConfiguration ...> <management/> <implementation/> <algorithm/> <precision> <markovStep> <markovChainURI>lfn://myCollab/myEnsemble</markovChainURI> <series>1</series> <update>010170</update> <avePlaquette>0.53380336E+00</avePlaquette> <dataLFN>lfn://myCollab/myFilename</dataLFN> </markovStep></gaugeConfiguration>

Raw Query:

number(/gaugeConfiguration/markovStep [dataLFN="lfn://myCollab/myFilename"]/avePlaquette)

Returns: 0.53380336

Raw Query:

/gaugeConfiguration/markovStep [markovChainURI="lfn://myCollab/myEnsemble"]/dataLFN

Returns: <dataLFN > lfn://myCollab/myFilename </ dataLFN >

XPath funcion

XPath Number Type

XML Tags

Raw Queries:

<?xml version=”1.0” ?><gaugeConfiguration ...> <management/> <implementation/> <algorithm/> <precision> <markovStep> <markovChainURI>lfn://myCollab/myEnsemble</markovChainURI> <series>1</series> <update>010170</update> <avePlaquette>0.53380336E+00</avePlaquette> <dataLFN>lfn://myCollab/myFilename</dataLFN> </markovStep></gaugeConfiguration>

LFN Query: /gaugeConfiguration/markovStep[update="010170"]

Returns: { lfn://myCollab/myFilename }

LFN Query: /gaugeConfiguration/markovStep [markovChainURI=”lfn://myCollab/myEnsemble”]

Returns: { lfn://myCollab/myFilename, lfn://myCollab/myFilename2,... }

LFN Queries:

Single Return Type: Array of LFNs for all queries

Comparison

● Raw Queries

– Powerful: Need Only 1 Query function

● Can use XPath functions eg: to count matches

– Query returns different types● Eg: numbers, strings, XML Snippets

– Can use to implement LFN Queries

– Also good for Web interfaces, testing

Comparison● Domain Specific (eg LFN) Queries

– Less Flexible● Can't execute arbitrary XPath functions

– More logic in clients

– More complex● Need more functions:

– Query/Get Config XML , Query/Get Ensemble XML

– Tailored to Particular Use Cases● Eg: Command Line Tools (find &

download)

– Don't need XML database necessarily

Security Infrastructure

● Can use SSL to encrypt communications with services

● Can have certificate based authorization in Web servers, Application servers, and even some databases

● Use firewalls to secure back ends

● Credential forwarding for service aggregator portals may need thought

Service (MDC/RC)

Client

Certificate Exchange

Encrypted Connection

Client Encrypted Connection

Certificate Exchange

Aggregate Service(MDC/RC)

US Service(MDC/RC)

LatFor Service(MDC/RC)

UK Service(MDC/RC)

AustralianService

(MDC/RC)

JapaneseService

(MDC/RC)

Certificate forwarding? Proxying? How to maintain secure connection??

Actual Database

Actual Database

Actual Database

Actual Database

Actual Database

Firewall

Clear

Thought Needed

VOMS: Virtual Organisation Management Service

● Technology for identity and role management

● Distributed and decentralized

● Part of LCG, OSG and gLite

● LatFor moving to production use

● To be trialled in UK and Japan

● In US, Fermilab will use infrastructure on collaborative project with DESY

VOMS● Interest in VOMS expressed by

Australia

● WG to come to a decision on standardising use of VOMS by end of February 2006

– Personal worry: This is choosing software rather than defining an interface.

– Pragmatism: LatFor requires users' certificate in their VOMS server. Will eventually hold everyone's certificate

LCG Software Components

● Michael recommended use of LCG software:

– Rich in features, presented as RPMs through software repositories, with installation tools (apt-get, yaim)

– Can install just components we need

– Can use any Red Hat Enterpise Linux like system (Scientific Linux (SL), RHEL, Fedora Core) CERN supports SL.

LCG Components

– Large User Community (LHC, OSG)

– Used by LatFor (File Catalog – LFC, and VOMS)

– In US Fermilab will have infrastructure

● BUT

– Don't want to force particular technology/software. That's not our mandate !

Conclusions

● Components coming along – individual collaborations will present own status reports

● MDC interface solidifying currently

● Most groups have received funding/effort

● Most groups have local infrastructures and are now working on ILDG issues

● I have a LOT of work ahead of me.

US Middleware status● Effort recently revitalised with arrival

of Bálint at the Jlab.

● General Web Service Infrastructure:

– Apache Web Server

– Tomcat Application Server

– Axis Web Services Framework

– X509 Certificates, SSL encryption

● Prototype Data: MILC Datasets from NERSC Gauge Connection

US Middleware Status

● RC Prototype ready for some time

– Need secure connection and certificate based authentication for production use.

● RC needs data in it to be useful.

– In discussion with MILC to choose suitable LFNs for data in NERSC archive. Recent helpful changes in metadata schema facilitate things.

US Middleware Status

● In analysis stage for MDC

– Plan: use eXist native XML database● Pure Java● Integrates well with Tomcat and Axis ● Comes with Web Service API● Sophisticated XML capabilities:

– Validation against Schemata, XQuery, XUpdate

● Expect very thin ILDG layer on top

US Middleware Status

● Progress with ILDG File Format

– Conversion routine from NERSC format to ILDG format exists within MILC code and QDP++ using QIO library.

– MILC Code can read QDP++ files and those provided by Dirk Pleiter (LatFor)

– Dirk could read files produced by QDP++ and MILC

US Middleware Status

● File format (cont'd)

– Only minor discrepancy● QIO uses LIME type: ildg-data-lfn● Dirk's code & spec give: ildg-data-LFN● Dirk suggests case insensitivity in codes

– Backward compatible solution for all sides.

– Carleton willing to change QIO to use case insensitive checking of LIME types.