lcg storage accounting john gordon cclrc – ral lcg grid deployment board september 2006

LCG Storage Accounting

John GordonCCLRC – RAL

LCG Grid Deployment BoardSeptember 2006

LCG

John Gordon – GDB 6 September 2006

Since July

Following my update session at the July GDB, There has been some progress but only limited deployment

LCG

John Gordon – GDB 6 September 2006

Status A Storage Accounting schema has been defined A single storage accounting sensor at Edinburgh

queries the BDII once per day to gather information on current storage usage from all UK sites

This sensor populates an R-GMA archiver with the gathered information

StorageRecords table The information is aggregated per VO/Site/Month as

in APEL Value stored is average over the number of

measurements Sum(measurements)/N(measurements)

APEL-like visualisation interface has been developed Grids, countries, regions Only tables for country and region implemented so far http://goc02.grid-support.ac.uk/accountingDisplay/

LCG

John Gordon – GDB 6 September 2006

StorageRecords Schemargma> describe StorageRecords+------------------+--------------+-------------+-------------+| Column name | Type | Primary key | Can be NULL |+------------------+--------------+-------------+-------------+| RecordIdentity | VARCHAR(255) | Yes | No || ResourceIdentity | VARCHAR(255) | No | Yes || Grid | VARCHAR(50) | No | Yes || ExecutingSite | VARCHAR(50) | No | Yes || VO | VARCHAR(50) | No | Yes || SpaceUsed | INTEGER | No | Yes || SpaceAvailable | INTEGER | No | Yes || Total | INTEGER | No | Yes || Unit | VARCHAR(50) | No | Yes || SEArchitecture | VARCHAR(50) | No | Yes || Type | VARCHAR(50) | No | Yes || EventDate | DATE | No | No || EventTime | TIME | No | No || MeasurementDate | DATE | No | No || MeasurementTime | TIME | No | No |+------------------+--------------+-------------+-------------+

LCG

John Gordon – GDB 6 September 2006

StorageRecords Examples+------------------+--------------+-------------+----------------------------------------+| Column name | Type | Pkey | NULL?| Example |+------------------+--------------+-------------+----------------------------------------+| RecordIdentity | VARCHAR(255) | Yes | No | dcache.gridpp.rl.ac.uk atlas 2006-08-13

20:00:17 | | ResourceIdentity | VARCHAR(255) | No | Yes | dcache.gridpp.rl.ac.uk || Grid | VARCHAR(50) | No | Yes | EGEE || ExecutingSite | VARCHAR(50) | No | Yes | RAL-LCG2 |

| VO | VARCHAR(50) | No | Yes | atlas || SpaceUsed | INTEGER | No | Yes | 9717308 || SpaceAvailable | INTEGER | No | Yes | 687036 || Total | INTEGER | No | Yes | 10404344 || Unit | VARCHAR(50) | No | Yes | MB | | SEArchitecture | VARCHAR(50) | No | Yes | multidisk |

| Type | VARCHAR(50) | No | Yes | permanent || EventDate | DATE | No | No | 2006-08-13 || EventTime | TIME | No | No | 20:00:17 || MeasurementDate | DATE | No | No | 2006-08-13 |

| MeasurementTime | TIME | No | No | 19:00:17 |+------------------+--------------+-------------+-----------------------------------------+

LCG

John Gordon – GDB 6 September 2006

Information Providers

The Castor GIP is under test Local BDII so not included here

A replacement DPM GIP is under development

A dCache GIP has been developed Used by RAL T1 and T2 Not sure if it will be officially released. Chimera later this year will allow better info

There is a GIP for Classic SE but propose not to account classic SEs as Grid

storage should be SRM-aware

Deployment of these has been later than planned

LCG

John Gordon – GDB 6 September 2006

dCache (1)+---------------------+--------+-----------+----------------+---------+| ExecutingSite | VO | SpaceUsed | SpaceAvailable | Total |+---------------------+--------+-----------+----------------+---------+| UKI-SOUTHGRID-RALPP | cdf | 0 | 33349 | 33349 || UKI-SOUTHGRID-RALPP | cms | 166576 | 1167395 | 1333971 || UKI-SOUTHGRID-RALPP | esr | 0 | 33349 | 33349 || UKI-SOUTHGRID-RALPP | ilc | 53 | 33296 | 33349 || UKI-SOUTHGRID-RALPP | ops | 0 | 16674 | 16674 || UKI-SOUTHGRID-RALPP | t2k | 0 | 33349 | 33349 || UKI-SOUTHGRID-RALPP | hone | 12759 | 270709 | 283468 || UKI-SOUTHGRID-RALPP | lhcb | 0 | 1333971 | 1333971 || UKI-SOUTHGRID-RALPP | na48 | 0 | 33349 | 33349 || UKI-SOUTHGRID-RALPP | zeus | 3535 | 279933 | 283468 || UKI-SOUTHGRID-RALPP | alice | 0 | 16674 | 16674 || UKI-SOUTHGRID-RALPP | atlas | 523340 | 810631 | 1333971 || UKI-SOUTHGRID-RALPP | babar | 211547 | 1122424 | 1333971 || UKI-SOUTHGRID-RALPP | cedar | 0 | 16674 | 16674 || UKI-SOUTHGRID-RALPP | dteam | 173520 | -140172 | 33348 || UKI-SOUTHGRID-RALPP | dzero | 0 | 266794 | 266794 || UKI-SOUTHGRID-RALPP | magic | 0 | 16674 | 16674 || UKI-SOUTHGRID-RALPP | minos | 0 | 33349 | 33349 || UKI-SOUTHGRID-RALPP | pheno | 2120 | 31228 | 33348 || UKI-SOUTHGRID-RALPP | ralpp | 0 | 66698 | 66698 || UKI-SOUTHGRID-RALPP | biomed | 0 | 33349 | 33349 || UKI-SOUTHGRID-RALPP | fusion | 0 | 33349 | 33349 || UKI-SOUTHGRID-RALPP | geant4 | 0 | 16674 | 16674 || UKI-SOUTHGRID-RALPP | planck | 0 | 16674 | 16674 |+---------------------+--------+-----------+----------------+---------+

LCG

John Gordon – GDB 6 September 2006

dCache (2)+----------------+--------+-----------+----------------+---------+| ExecutingSite | VO | SpaceUsed | SpaceAvailable | Total |+----------------+--------+-----------+----------------+---------+| UKI-LT2-IC-HEP | cms | 4177738 | 3568092 | 7745830 || UKI-LT2-IC-HEP | dteam | 1560864 | 1197938 | 2758802 || UKI-LT2-IC-HEP | atlas | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | alice | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | lhcb | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | sixt | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | zeus | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | na48 | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | calice | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | ilc | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | biomed | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | babar | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | dzero | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | hone | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | pheno | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | t2k | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | geant4 | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | gin | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | ltwo | 0 | 0 | 0 || UKI-LT2-IC-HEP | ops | 108198 | 116196 | 224394 |+----------------+--------+-----------+----------------+---------+

LCG

John Gordon – GDB 6 September 2006

Multiple SRM+---------------+--------+-----------+----------------+-----------+----------+| ExecutingSite | VO | SpaceUsed | SpaceAvailable | Type | Total |+---------------+--------+-----------+----------------+-----------+----------+| RAL-LCG2 | cdf | 0 | 0 | permanent | 0 || RAL-LCG2 | cms | 12993009 | 5301774 | permanent | 18294783 || RAL-LCG2 | esr | 15 | 102386 | permanent | 102401 || RAL-LCG2 | ilc | 498 | 101903 | permanent | 102401 || RAL-LCG2 | ops | 0 | 2050039 | permanent | 2050039 || RAL-LCG2 | t2k | 0 | 102401 | permanent | 102401 || RAL-LCG2 | hone | 312936 | 732185 | permanent | 1045121 || RAL-LCG2 | lhcb | 6482263 | 7302824 | permanent | 13785087 || RAL-LCG2 | zeus | 48913 | 53488 | permanent | 102401 || RAL-LCG2 | alice | 0 | 2012159 | permanent | 2012159 || RAL-LCG2 | atlas | 11457691 | 1342813 | permanent | 12800504 || RAL-LCG2 | babar | 30284 | 72117 | permanent | 102401 || RAL-LCG2 | cedar | 0 | 102401 | permanent | 102401 || RAL-LCG2 | dteam | 128390 | 1921649 | permanent | 2050039 || RAL-LCG2 | dzero | 0 | 0 | permanent | 0 || RAL-LCG2 | magic | 0 | 102401 | permanent | 102401 || RAL-LCG2 | minos | 182 | 2114379 | permanent | 2114561 || RAL-LCG2 | pheno | 29713 | 72688 | permanent | 102401 || RAL-LCG2 | biomed | 581 | 101820 | permanent | 102401 || RAL-LCG2 | cms | 0 | 19531 | permanent | 19531 || RAL-LCG2 | fusion | 0 | 102401 | permanent | 102401 || RAL-LCG2 | geant4 | 0 | 102401 | permanent | 102401 || RAL-LCG2 | lhcb | 0 | 19531 | permanent | 19531 || RAL-LCG2 | babar | 0 | 19531 | permanent | 19531 || RAL-LCG2 | cms | 41943040 | 0 | permanent | 41943040 || RAL-LCG2 | dteam | 0 | 19531 | permanent | 19531 || RAL-LCG2 | ops | 524288 | 0 | permanent | 524288 || RAL-LCG2 | lhcb | 20971520 | 0 | permanent | 20971520 || RAL-LCG2 | alice | 2097152 | 0 | permanent | 2097152 || RAL-LCG2 | atlas | 8388608 | 0 | permanent | 8388608 || RAL-LCG2 | dteam | 524288 | 0 | permanent | 524288 || RAL-LCG2 | minos | 10485760 | 0 | permanent | 10485760 |+---------------+--------+-----------+----------------+-----------+----------+

Tape SRM

LCG

John Gordon – GDB 6 September 2006

LCG

John Gordon – GDB 6 September 2006

LCG

John Gordon – GDB 6 September 2006

LCG

John Gordon – GDB 6 September 2006

Problems

Old information providers publish info per Storage Area, not per VO

Cannot add up all SA/VO to get site storage

Published numbers GB not NB Just changing to TB

Tape not well understood

LCG

John Gordon – GDB 6 September 2006

Old dCache GIP

+----------------------+--------+-----------+----------------+---------+| ExecutingSite | VO | SpaceUsed | SpaceAvailable | Total |+----------------------+--------+-----------+----------------+---------+| UKI-SOUTHGRID-OX-HEP | atlas | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | alice | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | lhcb | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | cms | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | dteam | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | sixt | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | babar | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | zeus | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | cdf | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | biomed | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | pheno | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | ilc | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | hone | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | dzero | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | oxg | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | ops | 781250 | 2304687 | 3085937 |+----------------------+--------+-----------+----------------+---------+

LCG

John Gordon – GDB 6 September 2006

Issues (1) GLUE publishes space used and space

available. For MoUs etc we are interested in space allocated There have been discussions about changes to

GLUE for Storage (at the SRM workshop?) or derive allocated=used+available at client or

server

Haven’t solved the double counting of available space in a StorageArea shared between VOs

But we are not really interested in available space for accounting

But this is important if we use it as above

Don’t have an SRM yet for the storage on WNs Reporting tape needs more thought

LCG

John Gordon – GDB 6 September 2006

Issues (2)

We can’t do user-level accounting via the BDII

We can probably support a FQAN owning a GlueSA if GLUE allows this in future

We can support future storagetypes as long as they are published in GLUE

But not differentiate between raw data and ESD (eg)

Current storage information in BDII isn’t that reliable

Start a programme of checking

LCG

John Gordon – GDB 6 September 2006

Status

So if we currently have a schema, a repository, collection, and visualisation

Then we have a framework for storage accounting

…but little good data.

LCG

John Gordon – GDB 6 September 2006

Plan

Continue developing the prototype portal Graphical views Storage(time, VO, site), used, allocated, available

Populate it with data from sites running the new GIPs

Check the correctness of the data published Change to TB

Ask for comments and feedback

LCG

John Gordon – GDB 6 September 2006

Can I have comments and feedback, please?