n° 1

LCG

EDG Data Management Catalogs in LCGJames CaseyLCG Fellow, IT-DB Group, CERN

james.casey@cern.ch

n° 2

LCG

Talk Outline

Overview of Data Management components in EDG 2.0

EDG catalogs

Architecture and features

Implementation details

Deployment choices for LCG

POOL as a client of EDG

Conclusions

n° 3

LCG

StorageElement

Data Management: Basic Functionality

Replica ManagerReplica Location

Service

Replica Metadata Catalog

StorageElement

Files have replicas stored at many Grid sites on Storage Elements.

Each file has a unique GUID.Locations corresponding to the GUID are kept in the Replica Location Service.

Users may assign aliases to the GUIDs. These are kept in the Replica Metadata Catalog.

The Replica Manager provides atomicity for file operations, assuring consistency of SE and catalog contents.

n° 4

LCG

StorageElement

Interactions with EDG 2.0 Components

Replica ManagerReplica Location

Service

Replica Optimization Service

Replica Metadata Catalog

SEMonitor

Network Monitor

Information Service

Resource Broker

User Interface orWorker Node

StorageElement

Virtual OrganizationMembership Service

Applications and users interface to data through the Replica Manager either directly or through the Resource Broker.

n° 5

LCGEDG Grid Catalogs (1/2)

Replica Location Service (RLS)

Local Replica Catalog (LRC) Stores GUID to Physical File Name (PFN) mappings Stores attributes on PFNs Many Local Replica Catalogs in Grid

One per Storage Element (per VO) Tested to 1.5M entries

Replica Location Index (RLI) Allow fast lookup of which sites store GUID -> PFN mappings for a given GUID Many Replica Location Index in the Grid

Normally one per Site (per VO), which indexes all LRCs in the Grid Being deployed as part of EDG 2.1 in July

In the process of integration into Replica Manager, POOL, EDG Job Scheduler

Tested to 10M entries in an RLI

n° 6

LCGEDG Grid Catalogs (2/2)

Replica Metadata Catalog (RMC) Stores Logical File Name (LFN) to GUID mappings – user-defined aliases

Stores attributes on LFNs and GUIDs

One Replica Metadata Catalog in Grid (per VO) Single point of synchronization – current assumption in EDG model bottleneck ? - move to replicated distributed database

No Application Metadata Catalog provided But Replica Metadata Catalog has support for small level of application

metadata – O(10)

RMC usage not as well understood as Replica Location Service Architectural changes likely

Use cases required

n° 7

LCGTypical Location of Services in LCG-1

ReplicaLocation

Index

LocalReplicaCatalog

StorageElement

CNAF

ReplicaLocation

Index

LocalReplicaCatalog

StorageElement

RAL

ReplicaLocation

Index

LocalReplicaCatalog

StorageElement

CERN

ReplicaLocation

Index

LocalReplicaCatalog

StorageElement

IN2P3

ReplicaMetadataCatalog

StorageElement

n° 8

LCGCatalog Implementation Details

Catalogs implemented in Java as Web Services, and hosted in a J2EE application server

Uses Tomcat4 or Oracle 9iAS for application server

Uses Jakarta Axis for Web Services container

Java and C++ client APIs currently provided using Jakarta Axis (Java) and gSoap (C++)

Catalog data stored in a Relational Database Runs with either Oracle 9i or MySQL

Catalog APIs exposed as a Web Service using WSDL Easy to write a new client if we don’t support your language right now

Vendor neutral approach taken to allow different deployment options

n° 9

LCGQuality of Service

Quality of Service depends upon both the server software and architecture used as well as the software components deployed on it

Features required for high Quality of Service High Availability

Manageability

Monitoring

Backup and Recovery with defined Service Level Agreements

Approach Use vendor solutions for availability and manageability where available

Use common IT-DB solutions for monitoring and recovery

Components architected to allow easy deployment in high-availability environment

A variety of solutions with different characteristics are possible

n° 10

LCGTradeoffs in different solutions

Manageability

Ava

ilabi

lity

Single InstanceMySQL/Tomcat

ClusteredOracle 9i/Tomcat

ClusteredOracle 9i/9iAS

Single InstanceOracle 9i/9iAS

n° 11

LCGCurrent Deployment Plans

EDG All sites use MySQL/Tomcat single instance solution

LCG-1 CERN deploys LRC/RLI/RMC on Oracle 9iAS/Oracle 9i single instance Tier-1 sites invited to use either Oracle 9iAS/Oracle or Tomcat4/MySQL single

instance for their LRC/RLIs

CERN IT-DB working on “easy-install” packaging of Oracle Oracle sees ease of install as a high priority for Oracle 10i

release date - Nov 2003

Allow deployment of an Oracle based solution without requiring a lot of Oracle expertise

Testing of components for high-availability solution in progress Based on Oracle 9i Plan to be available for year-end 2003

n° 12

LCGSystem Architecture – High Availability

Standard n-tier architecture

Front end application layer load-balancer

Oracle 9iAS Web Cache

Cluster of stateless application servers

Oracle 9iAS J2EE container

Clustered database nodes Oracle 9i/RAC

Shared SAN storage Fibre Channel storage

Inte

rnal LA

N

Sto

rage N

etw

ork

Exte

rnal L

AN

n° 13

LCGPOOL and the Grid

Good match in terms of architecture for usage of EDG Catalogs A POOL FileID is a GUID in our architecture

Combines features from Replica Metadata Catalog (LFN aliases, GUID attributes) and Local Replica Catalog (GUID to PFN mappings)

EDG provides a C++ library to POOL which provides all functions required to implement a POOL File Catalog

Catalogs deployed for POOL release 1.0 – May 2003 rlstest.cern.ch

Pre-production quality service

Catalogs deployed for LCG-1 – July 2003 rlscms.cern.ch, rlsatlas.cern.ch, rlslhcb.cern.ch, rlsalice.cern.ch

Production quality service

n° 14

LCGSummary & Conclusions

New data management Architecture deployed as part of EDG 2.0

Good match with requirements of POOL File Catalog

Focus on manageability and scalability aspects

Can’t wait for OGSA – production deploying by September 2003

Design will allow evolution into OGSA

POOL and LCG acting as good “real” customers for EDG data management

Validate our components can act outside of an EDG context

LCG-1 will provide hard targets to meet in terms of scalability and reliability

Good testing ground

n° 15

LCG

Questions ?

n° 16

LCG

Oracle at Tier-1 Sites

n° 17

LCGWP2 Deployment

A farm node running Red Hat Enterprise Linux and Oracle9iAS Runs Java middleware for LRC, RLI, RLS-service etc.

A disk server running Red Hat Enterprise Linux and Oracle9i Stores GUID <-> PFN mappings Data volume for LCG 1 small (~105 – 106 entries, each < 1KB) Query / lookup rate low (~1 every 3 seconds)

Projection to 2008: 100 – 1000Hz; 109 entries

Site responsible for acquiring and installing h/w and RHEL $349 for ‘basic edition’ http://www.redhat.com/software/rhel/es/

CERN will provide distribution kits of Oracle s/w for RHEL together with automatic installation scripts and documentation

n° 18

LCGSupport Issues

CERN will recommend Oracle books & Training

technet.oracle.com – excellent source of information Free access, but registration required

Support calls to Oracle via ‘metalink’ (Web) Can provide read-only access to small number of administrators

(e.g. Tier1 sites) Already very useful for solving problems

Further escalation channeled through CERN-IT-DB As is done now for other CERN users

n° 19

LCGMonitoring & Backup

CERN is moving to Oracle Enterprise Manager (OEM) and Recovery Manager (RMAN) for monitoring and backup respectively

Goal is to have common database setup, monitoring and backup strategy across all servers

Can provide example scripts, guidelines etc, but monitoring, backup & recovery is clearly responsibility of local site

n° 20

LCGOracle 10i

Will be announced in September

Many new features requested by CERN Native floats & doubles, ULDB, greatly simplified installation

(database cloning, no client install etc.), machine independent transportable tablespaces(!), etc

For clients, just copy 2 files to LD_LIBRARY_PATH

For servers, database cloning should simplify and speedup deployment

n° 21

LCGConclusion

Oracle licensing for LCG is a solved issue

Distribution kits are ready

Documentation is ready

We are ready to start working with Tier1 sites on the deployment on Oracle of WP2 services

n° 22

LCGOther Distribution Kits

From Oracle 10i: ship client libraries e.g. with POOL? Potential clients include conditionsDB, POOL with RDBMS backend

etc.

Server kit for other applications, e.g. local conditionsDB, local copy of COMPASS event metadata etc.

Full distribution Will require local experienced DBAs

n° 23

LCG

Misc.

n° 24

LCGThe POOL Project

POOL is the LCG Persistency Framework Pool of persistent objects for LHC

Consists of Several Components Storage Service File Catalog Object-level Collections Object Cache

Pool had several File Catalog implementations, for different usage patterns and user requirements

XML Catalog Native MySQL Catalog

POOL wanted a Grid-aware catalog – December 2002 Looked at EDG Catalogs as a possible solution