WLCG Grid Deployment Board, CERN 9 July 2008

Proposal for a Dynamic Megatable

Flavia DonnoCERN/IT

Tier-2 Sites & CapacitiesTier-2 Sites & Capacities A little difficult to determine exact resources installed in the Tier2s; must be improved

(Ian Bird, March LCG OB)

WLCG Grid Deployment Board, CERN 9 July 2008 3

The requirementsThe requirements

The goal Provide the management with information about installed capacity (per VO). Provide the management with information about resource usage (per VO).

This info can also be used by VO operations people in order to "monitor" their usage of the resources

Main focus Storage resources Computing resources

Dynamic view Information should be retrieved as dynamically as possible in order to be reliable The information will be made available through the WLCG accounting system Use the information system as source

WLCG Grid Deployment Board, CERN 9 July 2008 4

The documentationThe documentation

The technical specifications are availablein the CCRC twiki in the SSWG section:https://twiki.cern.ch/twiki/bin/view/LCG/WLCGCommonComputingReadinessChallenges#Storage_Solution_Working_Group_S

Storage only: The documentation available reports on the conclusions reached during

focused meetings with developers and information providers Specific solutions found to cover dCache internal specialized buffers and

avoid double counting

The document has been agreed by storage developers, storage information providers, data management developers, …

WLCG Grid Deployment Board, CERN 9 July 2008 5

The Storage ResourcesThe Storage Resources Current accounting harvests data from BDII

… but many errors in data. combination of Glue1.2 and 1.3

Need to improve information providers and configuration. Existing Reports

Current status (last hour) Monthly Report (like manual Tier1 reports).

WLCG Grid Deployment Board, CERN 9 July 2008 6

The Storage ResourcesThe Storage Resources Use GlueSA class

It describes a Storage Area: logical view of a portion of space (can include disks and tapes) Online resources only for the moment GlueSAs do not overlap. Shared spaces can be represented with one GlueSA with multiple

GlueSAAccessControlBaseRule

A GlueSA can be published for unreserved space as well In this case Total=Reserved Size (see later)

GlueSA can have one or more VOInfo associated objects

VOInfoTag is optional The VOInfo object can be used to publish the path associated to a VO

using the associated GlueSA (in case of shared SAs)

WLCG Grid Deployment Board, CERN 9 July 2008 7

The Storage ResourcesThe Storage Resources GlueSA Sizes

GlueSAReservedOnlineSize Space physically allocated to a VO (or a set of VOs) – Installed capacity

GlueSATotalOnlineSize (in GB = 109 bytes) Total Online Space available at a given moment (it does not account for

broken disk servers, draining pools, etc.) GlueSAUsedOnlineSize (in GB = 109 bytes)

Size occupied by files that are not candidates for garbage collection – Resources usage

For Castor this is equal to GlueSATotalOnlineSize for T1D0 (they differ for T0D1)

GlueSAFreeOnlineSize (in GB = 109 bytes) Free = Total - Used

GlueSACapability Key/value pair to publish agreed characteristics of Storage Area, i.e.

access=read, network=WAN-LAN, etc. Experiments can use this field to select dCache pools

Dynamic information providers for CASTOR by J. Jensen Corrections needed to comply with agreement Packaging and distribution effort through the CASTOR CVS First installation foreseen at CERN Availability: end of July 2008?

Dynamic information providers for dCache available by R. Trompert Need to verify with dCache developers that pinned files usage info is available, and

other details 2-3 weeks to implement the proposal The changes will be reflected also in the new official dCache information providers

by P. Millar

WLCG Grid Deployment Board, CERN 9 July 2008 8

Storage: Status and plansStorage: Status and plans

Dynamic information providers for DPM by Michel Jouvin The proposal has been already implemented Installed at Edinburgh – Testing phase started

Information already available for StoRM by developers Probable minor additions for VOInfoPath

Need to work with OSG and NorduGrid

WLCG Grid Deployment Board, CERN 9 July 2008 9

Storage: Status and plansStorage: Status and plans

WLCG Grid Deployment Board, CERN 9 July 2008 10

The Computing ResourcesThe Computing Resources Pledges expressed in terms of KSI2000 per federation in current Megatable Use available information published in BDII about installed capacity

Red=deprecated

WLCG Grid Deployment Board, CERN 9 July 2008 11

The Computing ResourcesThe Computing Resources Ongoing work to make published information about installed capacity more reliable (see presentation of Steve Traylen at

GDB) Current tools publish always 1 GlueCluster and 1 GlueSubCluster

when a GlueCE is published (problematic CPU counting) Difference between SubCluster PhysicalCPUs and

GlueCETotalCPUs Is Subcluster PhysicalCPUs coherent with SubCluster Host

BenchmarkSI00? When SubCluster PhysicalCPUs=0 consider GlueCETotalCPUs:

Number of slots per queue. Compatible with Subcluster Host BenchmarkSI00? Is this number correct (extracted from batch system)?

VOView TotalCPUs deprecated. Difficult to guess VO shares.

WLCG Grid Deployment Board, CERN 9 July 2008 12

Computing Resources ExampleComputing Resources Example

Need to interpret results Refinement and cleaning (discrepancy between SubClusters

and GlueCE, CPUs vs. cores, HEP only, etc.) Improve information providers (Glue 2.0 helps)

Double-countedResources

Computing resources First procedures in place. Looking for better technologies to make

results available. We are currently in the process of understanding and refining

information

Work with sites Check that needs are covered by Glue 2.0 Help automate the publishing process Work with OSG and NorduGrid Provide a working prototype for APEL by the end of August 2008 Give feedback to GDB and MB Help producing monthly reports

WLCG Grid Deployment Board, CERN 9 July 2008 13

Computing: Status and plansComputing: Status and plans

WLCG Grid Deployment Board, CERN 9 July 2008

Thank You