lcg storage accounting john gordon cclrc – ral lcg grid deployment board september 2006
TRANSCRIPT
LCG Storage Accounting
John GordonCCLRC – RAL
LCG Grid Deployment BoardSeptember 2006
LCG
John Gordon – GDB 6 September 2006
Since July
Following my update session at the July GDB, There has been some progress but only limited deployment
LCG
John Gordon – GDB 6 September 2006
Status A Storage Accounting schema has been defined A single storage accounting sensor at Edinburgh
queries the BDII once per day to gather information on current storage usage from all UK sites
This sensor populates an R-GMA archiver with the gathered information
StorageRecords table The information is aggregated per VO/Site/Month as
in APEL Value stored is average over the number of
measurements Sum(measurements)/N(measurements)
APEL-like visualisation interface has been developed Grids, countries, regions Only tables for country and region implemented so far http://goc02.grid-support.ac.uk/accountingDisplay/
LCG
John Gordon – GDB 6 September 2006
StorageRecords Schemargma> describe StorageRecords+------------------+--------------+-------------+-------------+| Column name | Type | Primary key | Can be NULL |+------------------+--------------+-------------+-------------+| RecordIdentity | VARCHAR(255) | Yes | No || ResourceIdentity | VARCHAR(255) | No | Yes || Grid | VARCHAR(50) | No | Yes || ExecutingSite | VARCHAR(50) | No | Yes || VO | VARCHAR(50) | No | Yes || SpaceUsed | INTEGER | No | Yes || SpaceAvailable | INTEGER | No | Yes || Total | INTEGER | No | Yes || Unit | VARCHAR(50) | No | Yes || SEArchitecture | VARCHAR(50) | No | Yes || Type | VARCHAR(50) | No | Yes || EventDate | DATE | No | No || EventTime | TIME | No | No || MeasurementDate | DATE | No | No || MeasurementTime | TIME | No | No |+------------------+--------------+-------------+-------------+
LCG
John Gordon – GDB 6 September 2006
StorageRecords Examples+------------------+--------------+-------------+----------------------------------------+| Column name | Type | Pkey | NULL?| Example |+------------------+--------------+-------------+----------------------------------------+| RecordIdentity | VARCHAR(255) | Yes | No | dcache.gridpp.rl.ac.uk atlas 2006-08-13
20:00:17 | | ResourceIdentity | VARCHAR(255) | No | Yes | dcache.gridpp.rl.ac.uk || Grid | VARCHAR(50) | No | Yes | EGEE || ExecutingSite | VARCHAR(50) | No | Yes | RAL-LCG2 |
| VO | VARCHAR(50) | No | Yes | atlas || SpaceUsed | INTEGER | No | Yes | 9717308 || SpaceAvailable | INTEGER | No | Yes | 687036 || Total | INTEGER | No | Yes | 10404344 || Unit | VARCHAR(50) | No | Yes | MB | | SEArchitecture | VARCHAR(50) | No | Yes | multidisk |
| Type | VARCHAR(50) | No | Yes | permanent || EventDate | DATE | No | No | 2006-08-13 || EventTime | TIME | No | No | 20:00:17 || MeasurementDate | DATE | No | No | 2006-08-13 |
| MeasurementTime | TIME | No | No | 19:00:17 |+------------------+--------------+-------------+-----------------------------------------+
LCG
John Gordon – GDB 6 September 2006
Information Providers
The Castor GIP is under test Local BDII so not included here
A replacement DPM GIP is under development
A dCache GIP has been developed Used by RAL T1 and T2 Not sure if it will be officially released. Chimera later this year will allow better info
There is a GIP for Classic SE but propose not to account classic SEs as Grid
storage should be SRM-aware
Deployment of these has been later than planned
LCG
John Gordon – GDB 6 September 2006
dCache (1)+---------------------+--------+-----------+----------------+---------+| ExecutingSite | VO | SpaceUsed | SpaceAvailable | Total |+---------------------+--------+-----------+----------------+---------+| UKI-SOUTHGRID-RALPP | cdf | 0 | 33349 | 33349 || UKI-SOUTHGRID-RALPP | cms | 166576 | 1167395 | 1333971 || UKI-SOUTHGRID-RALPP | esr | 0 | 33349 | 33349 || UKI-SOUTHGRID-RALPP | ilc | 53 | 33296 | 33349 || UKI-SOUTHGRID-RALPP | ops | 0 | 16674 | 16674 || UKI-SOUTHGRID-RALPP | t2k | 0 | 33349 | 33349 || UKI-SOUTHGRID-RALPP | hone | 12759 | 270709 | 283468 || UKI-SOUTHGRID-RALPP | lhcb | 0 | 1333971 | 1333971 || UKI-SOUTHGRID-RALPP | na48 | 0 | 33349 | 33349 || UKI-SOUTHGRID-RALPP | zeus | 3535 | 279933 | 283468 || UKI-SOUTHGRID-RALPP | alice | 0 | 16674 | 16674 || UKI-SOUTHGRID-RALPP | atlas | 523340 | 810631 | 1333971 || UKI-SOUTHGRID-RALPP | babar | 211547 | 1122424 | 1333971 || UKI-SOUTHGRID-RALPP | cedar | 0 | 16674 | 16674 || UKI-SOUTHGRID-RALPP | dteam | 173520 | -140172 | 33348 || UKI-SOUTHGRID-RALPP | dzero | 0 | 266794 | 266794 || UKI-SOUTHGRID-RALPP | magic | 0 | 16674 | 16674 || UKI-SOUTHGRID-RALPP | minos | 0 | 33349 | 33349 || UKI-SOUTHGRID-RALPP | pheno | 2120 | 31228 | 33348 || UKI-SOUTHGRID-RALPP | ralpp | 0 | 66698 | 66698 || UKI-SOUTHGRID-RALPP | biomed | 0 | 33349 | 33349 || UKI-SOUTHGRID-RALPP | fusion | 0 | 33349 | 33349 || UKI-SOUTHGRID-RALPP | geant4 | 0 | 16674 | 16674 || UKI-SOUTHGRID-RALPP | planck | 0 | 16674 | 16674 |+---------------------+--------+-----------+----------------+---------+
LCG
John Gordon – GDB 6 September 2006
dCache (2)+----------------+--------+-----------+----------------+---------+| ExecutingSite | VO | SpaceUsed | SpaceAvailable | Total |+----------------+--------+-----------+----------------+---------+| UKI-LT2-IC-HEP | cms | 4177738 | 3568092 | 7745830 || UKI-LT2-IC-HEP | dteam | 1560864 | 1197938 | 2758802 || UKI-LT2-IC-HEP | atlas | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | alice | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | lhcb | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | sixt | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | zeus | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | na48 | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | calice | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | ilc | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | biomed | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | babar | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | dzero | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | hone | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | pheno | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | t2k | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | geant4 | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | gin | 108198 | 116196 | 224394 || UKI-LT2-IC-HEP | ltwo | 0 | 0 | 0 || UKI-LT2-IC-HEP | ops | 108198 | 116196 | 224394 |+----------------+--------+-----------+----------------+---------+
LCG
John Gordon – GDB 6 September 2006
Multiple SRM+---------------+--------+-----------+----------------+-----------+----------+| ExecutingSite | VO | SpaceUsed | SpaceAvailable | Type | Total |+---------------+--------+-----------+----------------+-----------+----------+| RAL-LCG2 | cdf | 0 | 0 | permanent | 0 || RAL-LCG2 | cms | 12993009 | 5301774 | permanent | 18294783 || RAL-LCG2 | esr | 15 | 102386 | permanent | 102401 || RAL-LCG2 | ilc | 498 | 101903 | permanent | 102401 || RAL-LCG2 | ops | 0 | 2050039 | permanent | 2050039 || RAL-LCG2 | t2k | 0 | 102401 | permanent | 102401 || RAL-LCG2 | hone | 312936 | 732185 | permanent | 1045121 || RAL-LCG2 | lhcb | 6482263 | 7302824 | permanent | 13785087 || RAL-LCG2 | zeus | 48913 | 53488 | permanent | 102401 || RAL-LCG2 | alice | 0 | 2012159 | permanent | 2012159 || RAL-LCG2 | atlas | 11457691 | 1342813 | permanent | 12800504 || RAL-LCG2 | babar | 30284 | 72117 | permanent | 102401 || RAL-LCG2 | cedar | 0 | 102401 | permanent | 102401 || RAL-LCG2 | dteam | 128390 | 1921649 | permanent | 2050039 || RAL-LCG2 | dzero | 0 | 0 | permanent | 0 || RAL-LCG2 | magic | 0 | 102401 | permanent | 102401 || RAL-LCG2 | minos | 182 | 2114379 | permanent | 2114561 || RAL-LCG2 | pheno | 29713 | 72688 | permanent | 102401 || RAL-LCG2 | biomed | 581 | 101820 | permanent | 102401 || RAL-LCG2 | cms | 0 | 19531 | permanent | 19531 || RAL-LCG2 | fusion | 0 | 102401 | permanent | 102401 || RAL-LCG2 | geant4 | 0 | 102401 | permanent | 102401 || RAL-LCG2 | lhcb | 0 | 19531 | permanent | 19531 || RAL-LCG2 | babar | 0 | 19531 | permanent | 19531 || RAL-LCG2 | cms | 41943040 | 0 | permanent | 41943040 || RAL-LCG2 | dteam | 0 | 19531 | permanent | 19531 || RAL-LCG2 | ops | 524288 | 0 | permanent | 524288 || RAL-LCG2 | lhcb | 20971520 | 0 | permanent | 20971520 || RAL-LCG2 | alice | 2097152 | 0 | permanent | 2097152 || RAL-LCG2 | atlas | 8388608 | 0 | permanent | 8388608 || RAL-LCG2 | dteam | 524288 | 0 | permanent | 524288 || RAL-LCG2 | minos | 10485760 | 0 | permanent | 10485760 |+---------------+--------+-----------+----------------+-----------+----------+
Tape SRM
LCG
John Gordon – GDB 6 September 2006
LCG
John Gordon – GDB 6 September 2006
LCG
John Gordon – GDB 6 September 2006
LCG
John Gordon – GDB 6 September 2006
Problems
Old information providers publish info per Storage Area, not per VO
Cannot add up all SA/VO to get site storage
Published numbers GB not NB Just changing to TB
Tape not well understood
LCG
John Gordon – GDB 6 September 2006
Old dCache GIP
+----------------------+--------+-----------+----------------+---------+| ExecutingSite | VO | SpaceUsed | SpaceAvailable | Total |+----------------------+--------+-----------+----------------+---------+| UKI-SOUTHGRID-OX-HEP | atlas | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | alice | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | lhcb | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | cms | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | dteam | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | sixt | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | babar | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | zeus | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | cdf | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | biomed | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | pheno | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | ilc | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | hone | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | dzero | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | oxg | 781250 | 2304687 | 3085937 || UKI-SOUTHGRID-OX-HEP | ops | 781250 | 2304687 | 3085937 |+----------------------+--------+-----------+----------------+---------+
LCG
John Gordon – GDB 6 September 2006
Issues (1) GLUE publishes space used and space
available. For MoUs etc we are interested in space allocated There have been discussions about changes to
GLUE for Storage (at the SRM workshop?) or derive allocated=used+available at client or
server
Haven’t solved the double counting of available space in a StorageArea shared between VOs
But we are not really interested in available space for accounting
But this is important if we use it as above
Don’t have an SRM yet for the storage on WNs Reporting tape needs more thought
LCG
John Gordon – GDB 6 September 2006
Issues (2)
We can’t do user-level accounting via the BDII
We can probably support a FQAN owning a GlueSA if GLUE allows this in future
We can support future storagetypes as long as they are published in GLUE
But not differentiate between raw data and ESD (eg)
Current storage information in BDII isn’t that reliable
Start a programme of checking
LCG
John Gordon – GDB 6 September 2006
Status
So if we currently have a schema, a repository, collection, and visualisation
Then we have a framework for storage accounting
…but little good data.
LCG
John Gordon – GDB 6 September 2006
Plan
Continue developing the prototype portal Graphical views Storage(time, VO, site), used, allocated, available
Populate it with data from sites running the new GIPs
Check the correctness of the data published Change to TB
Ask for comments and feedback
LCG
John Gordon – GDB 6 September 2006
Can I have comments and feedback, please?