Download - Experiences of the Grid… Gavin McCance University of Glasgow NeSC Meeting, 24 October 2001
Experiences of the Grid…
Gavin McCanceUniversity of GlasgowNeSC Meeting, 24 October 2001
NeSC 24 October 2001
Gavin McCance, University of Glasgow 2/38
Background
Experimental Particle Physics background Analysing the structure of matter
…Fortran (19)77 !
Working in ‘Grid’-like areas since January this year
NeSC 24 October 2001
Gavin McCance, University of Glasgow 3/38
GridPP 20+ institutes…
NeSC 24 October 2001
Gavin McCance, University of Glasgow 4/38
…GridPP
£17M 3-year projectWorking in collaboration with EU DataGrid project
Middleware productionIntegration of middleware technologies into HEP experimentsValidation of Grid Software
NeSC 24 October 2001
Gavin McCance, University of Glasgow 5/38
…GridPP
Initial GridPP testbed underway
A personal snapshot of activities on the grid… Middleware activities we’re
involved in Some examples
Technologies we’re using Issues with integration of ‘Grid’
with particle physics experiments
NeSC 24 October 2001
Gavin McCance, University of Glasgow 6/38
Middleware
What is middleware…???
Application programs – local gridopen()
Data access specifics – HPSS, CastorJob submission specifics – PBS, LSF
Specific security procedures
Grid middleware Layered API’s.
Transparent security. Transparent data access.
Intelligent use ofdistributed resources.
NeSC 24 October 2001
Gavin McCance, University of Glasgow 7/38
Middleware Activities
GridPP ~mirrors EU DataGrid:Workload Management What jobs go where?
Data Management (*) Where’s the (best) data?
Information Services (*) What’s the state of everything?
NeSC 24 October 2001
Gavin McCance, University of Glasgow 8/38
…Middleware Activities
Fabric Management Interfaces to underlying systems
Mass Storage Management How to get the data to/from the fabric e.g.
Implementing ‘file-save()’ APIs for different mass storage systems
Security Crops up everywhere … transparent to
applications
NeSC 24 October 2001
Gavin McCance, University of Glasgow 9/38
Data Management
Data ReplicationTransparent and Secure Data AccessMeta Data StorageQuery Optimisation
NeSC 24 October 2001
Gavin McCance, University of Glasgow 10/38
Example problem: Data Replication
Problems if data exist only in one place Multiple accesses to the same data
overload network! Petabytes! Funding constraints! e.g. CERN can’t store
all of the data required
Make Replica! But need to keep track of all the files and their various replica! Need replica catalogue!
NeSC 24 October 2001
Gavin McCance, University of Glasgow 11/38
…Catalogues
Examples solutions:Have a globally unique Logical File Name (LFN) mapping to multiple physical instances of the file (PFNs).
Replica selection required Choose the ‘best’ / ‘nearest’ / ‘fastest’
Cost modelling… how time expensive to transfer files X’ from A to B
File-1
File-1
File-1
File-1
Paris
Glasgow
Chicago
LFN
NeSC 24 October 2001
Gavin McCance, University of Glasgow 12/38
…Data Replication
Grid Data Mirroring Package C, C++, JAVA, command-line APIs
Replication issues: File transfer…
Synchronisation / consistency models Basic middleware doesn’t enforce any
policy
Scalable architectures
NeSC 24 October 2001
Gavin McCance, University of Glasgow 13/38
…GDMP
File transfer uses GridFTP Existing IETF-approved (?RFC?) ftp additions + the standard grid security (GSI)
Registers new files in replica catalogue E.g. interfaced to the existing Globus
Replica Catalogue
Basic replica manager functionality to maintain consistency of replica sets
NeSC 24 October 2001
Gavin McCance, University of Glasgow 14/38
…Implementation issues
Structure not imposed by the middleware software itself… But … must think about scalable
implementations
E.g. a RC may exist on each storage element responsible for its own filesCERN Root RC
CERN RCUK RCINFN RC
Queries will propagate down until replica information is found…
NeSC 24 October 2001
Gavin McCance, University of Glasgow 15/38
…Longer term problems
Query / Replica Optimisation Grid can make / delete replica
Eg. Many people in Glasgow & Edinburgh access the ATLAS Higgs dataset ‘A1’…
Autonomously make new replica in / near Scotland based on historical information
Grid might re-cluster data
A2
A3
A1 B1 B3
B2
GlasgowParis
A2
A3
A1
B1
B3
B2
GlasgowParis
NeSC 24 October 2001
Gavin McCance, University of Glasgow 16/38
…longer termreal Grid...real Grid...
…simulated Grid provides testing arena for more
adventurous ideas!
…simulated Grid provides testing arena for more
adventurous ideas!
MONARC simulation tool
NeSC 24 October 2001
Gavin McCance, University of Glasgow 17/38
…Integration of middleware
Many iterations of requirements and use-cases with end-users… meetings…Middleware solutions must be scalable and useable by a variety of end users HEP, Biological, Earth sciences, Astro
Always looking for common elements E.g. replica / meta-data catalogues… data
transport… security…
NeSC 24 October 2001
Gavin McCance, University of Glasgow 18/38
…examples of common interfaces:generic meta-data catalogue tools
SQL Database Service: Problem: many relational databases,
diverse security, diverse wire protocols …Solution: Build on existing wire protocols: XML
transported over HTTP(S) Grid standard security framework (GSI)
NeSC 24 October 2001
Gavin McCance, University of Glasgow 19/38
..examples
Leverage open-source technology JAVA servlet based (Apache Tomcat
engine) JDBC drivers Utilises Oracle’s XSQL servlet (open
source)
Security over HTTPS with Grid-standard GSI mechanism
NeSC 24 October 2001
Gavin McCance, University of Glasgow 20/38
…examples
Allows any HTTP compliant system e.g. Web-browsers / standard C++ HTTP libraries to access any relational database…
= SQL Database Service (Spitfire)
OraclePostgreSQL
+
PKI Security
+
Standard communicationprotocols
(XML over HTTPS)
NeSC 24 October 2001
Gavin McCance, University of Glasgow 21/38
Global Grid Forum
Global Grid Forum meetings GGF1: Amsterdam meeting in April
2001
Helps define aspects common to all Grid-like projects. E.g. architectures, ‘grid’ protocols
As example… Grid Monitoring Architecture (GMA)
NeSC 24 October 2001
Gavin McCance, University of Glasgow 22/38
Information Services - GMA
One Implementation of the GMA Globus MDS, currently based on (Open)LDAP
Hierarchical directory like structure Very fast for information retrieval if you
already know the query designed into structure.
Bad for complex or ranged queries
NeSC 24 October 2001
Gavin McCance, University of Glasgow 23/38
..complementary implementation
Producer
Producer API
Register,re-register,publish
Relational Database
Consumer
Querying API
Query
ProducerServlet
RegistryServlet
SchemaServlet
subscribestream
Implementation of GMARelational queries in SQL format
NeSC 24 October 2001
Gavin McCance, University of Glasgow 24/38
…relational GMA
Information is transferred in generic SQL format…‘Producers’ of information register themselves…‘Consumers’ construct (possibly complex) SQL query and are streamed query results directly from Producers.
NeSC 24 October 2001
Gavin McCance, University of Glasgow 25/38
…implementation
Again, uses JAVA servlets Tomcat servlet engine
Again, communication with servlet is over standard HTTP.
All the internal parts communicate via HTTP and XML modular design, easily replaceable…
NeSC 24 October 2001
Gavin McCance, University of Glasgow 26/38
Useful Tools…
JAVA… nicely platform independentUML Universal(?) Modelling Language Architecture and API’s ‘should be’
defined in this…!
CASE tools Together Control Centre
NeSC 24 October 2001
Gavin McCance, University of Glasgow 27/38
…useful tools
Globus toolkit Both the original and its java
implementation (CoG)
My experience of CoG so far is generally good…! Easy GSI authentication, Globus file
transfer, Globus job submission, MDS interface
NeSC 24 October 2001
Gavin McCance, University of Glasgow 28/38
Testbeds
For GridPP, primary testbeds are the HEP experiment ones
CERN LHC (EU DataGrid WP8) US experiments, e.g. Fermilab, SLAC
First software release now!! Integration team ‘show-and-tell’ at CERN
end of this month…
NeSC 24 October 2001
Gavin McCance, University of Glasgow 29/38
...testbed work
Grid software packaged for release to experiments!
Primarily packaged using RPM
For end of October release, supported platforms are:Linux (and Solaris on a best effort basis)
NeSC 24 October 2001
Gavin McCance, University of Glasgow 30/38
..Globus installation
Generally found the Globus software installation OK! Successfully deployed on a number of batch
systems in UK Experience fed back into eScience Centres
Difficulties were setting up and recognising each countries’ Certificate Authorities (CAs) Tricky legal implications to resolve!
NeSC 24 October 2001
Gavin McCance, University of Glasgow 31/38
Testbed work so far…
UK Certificate Authority set-up… Many institutes already on testbed
Grid Status and Network monitoring demonstrator available soon
Networking status information provided by GridPP and DataGrid networking groups!
NeSC 24 October 2001
Gavin McCance, University of Glasgow 32/38
…testbed work so far
Successful tests within ATLAS (and others) of some middleware products E.g. Large file transfers between UK,
Italy, US and CERN
Further tests planned with new release!
NeSC 24 October 2001
Gavin McCance, University of Glasgow 33/38
…experimental integration
Work to do… Taking the kit and trying to integrate it into
the experiments’ software frameworks
ATLAS/LHCb software
framework (GAUDI)
Grid middleware
GANGA framework
Make Grid Services transparently available to
ATLAS and LHCbprograms
NeSC 24 October 2001
Gavin McCance, University of Glasgow 34/38
Grid validation
Preliminary tests of basic middleware has been successful
Now we have opportunity to see how it performs and scales with real datasets and real experimental users
NeSC 24 October 2001
Gavin McCance, University of Glasgow 35/38
Overall experience
Middleware development is fun! Several good products have already been
delivered
Re-using industry standard components and protocols where they exist LDAP, SQL, HTTP(S), XML, SOAP PKI security Open Source…!
NeSC 24 October 2001
Gavin McCance, University of Glasgow 36/38
…overall
Middleware being built using a variety of languages… JAVA, C++, C, Python
APIs should be available for all JAVA, C++, C and command line… web access(?)
NeSC 24 October 2001
Gavin McCance, University of Glasgow 37/38
…overall
NeSC 24 October 2001
Gavin McCance, University of Glasgow 38/38
…finally
Preliminary grid software architectures have been definedBasic middleware has been deliveredLarge scale validation underway NOW!!
A excellent base to build on! Plenty still to do!