wlcg transfers dashboard a unified monitoring tool for heterogeneous data transfers . alexandre...
DESCRIPTION
WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers . Alexandre Beche < [email protected]>. Outlines. Mission History of data transfer monitoring tools Data volume and retention policy Main improvements Future work Application usage. Mission. - PowerPoint PPT PresentationTRANSCRIPT
IT-SDC : Support for Distributed Computing
WLCG Transfers Dashboard A unified monitoring tool for
heterogeneous data transfers.
Alexandre Beche<[email protected]>
2IT-SDC
Outlines Mission History of data transfer monitoring tools Data volume and retention policy Main improvements Future work Application usage
12 February 2014Alexandre Beche – GDB
3IT-SDC
ATLASFTS + XRootD
CMSFTS + XRootD
LHCbFTS
ALICEXRootD
Mission
12 February 2014Alexandre Beche – GDB
4IT-SDC
ATLASFTS + XRootD
CMSFTS + XRootD
LHCbFTS
ALICEXRootD
Mission
12 February 2014Alexandre Beche – GDB
5IT-SDC
WLCG Transfers Dashboard User Interface
12 February 2014Alexandre Beche – GDB
6IT-SDC
History of data transfers monitoring tools
WEBAPI / UI
WLCG
FTS
WLCGTransfers
Dashboard
12 February 2014Alexandre Beche – GDB
7IT-SDC
History of data transfers monitoring tools
WEBAPI / UI
WLCG
FAX
AAA
FTS
WLCGTransfers
Dashboard
XRootDIntegration
12 February 2014Alexandre Beche – GDB
8IT-SDC
History of data transfers monitoring tools
WEBAPI / UI
WEBAPI/UI
WEBAPI/UI
WLCG FAX AAA
FAX
AAA
FTS
WLCGTransfers
Dashboard
XRootDIntegration
XRootDDetailed
Dashboard
12 February 2014Alexandre Beche – GDB
9IT-SDC
History of data transfers monitoring tools
WEBAPI / UI
WEBAPI/UI
WEBAPI/UI
WLCG FAX AAA
FAX
EOS
AAA
EOS
FTS
WLCGTransfers
Dashboard
XRootDIntegration
XRootDDetailed
Dashboard
EOSIntegration
12 February 2014Alexandre Beche – GDB
10IT-SDC
History of data transfers monitoring tools
WEBAPI / UI
WEBAPI/UI
WEBAPI/UI
WLCG FAX AAA
FAX
EOS
AAA
EOS
FTS
ALIC
EWLCG
TransfersDashboard
XRootDIntegration
XRootDDetailed
Dashboard
EOSIntegration
ALICEIntegration
12 February 2014Alexandre Beche – GDB
11IT-SDC
WLCG Transfers DashboardFederated approach
WEBAPI / UI
WEBAPI/UI
WEBAPI/UI
FTS FAX AAA
FAX
EOS
AAA
EOS
FTS
ALIC
E
WEBAPI
ALICE
12 February 2014Alexandre Beche – GDB
12IT-SDC
WLCG Transfers DashboardFederated approach
WEBAPI / UI
WEBAPI/UI
WEBAPI/UI
FTS FAX AAA
FAX
EOS
AAA
EOS
FTS
ALIC
E
WEBAPI
ALICE
WLCG Transfers DashboardAPI / UI
12 February 2014Alexandre Beche – GDB
13IT-SDC
Data Volumesince June 2012
Data are split into 3 schemas FTS, XRootD (AAA and FAX) High optimization (Partition, no Global indexes, …)
12 February 2014Alexandre Beche – GDB
14IT-SDC
Data Volumesince June 2012
# ROWS (million) Space (GB)FTS 648 (+7.2/day) 274 (+3/day)
XRootD (incl. local accesses) 2,283 (+4.2/day) 636 ( +1.2/day)
Data are split into 3 schemas FTS, XRootD (AAA and FAX) High optimization (Partition, no Global indexes, …)
12 February 2014Alexandre Beche – GDB
15IT-SDC
Data Volumesince June 2012
# ROWS (million) Space (GB)FTS 648 (+7.2/day) 274 (+3/day)
XRootD (incl. local accesses) 2,283 (+4.2/day) 636 ( +1.2/day)
Data are split into 3 schemas FTS, XRootD (AAA and FAX) High optimization (Partition, no Global indexes, …)
Snapshot on Tuesday 11 Feb 2014 Shows the usage of the DB at this specific time No conclusion can be drawn on technology usage
See next slide on data retention policy
12 February 2014Alexandre Beche – GDB
16IT-SDC
Data retention policy Two kind of data:
Raw: Kept for 3 months for FTS Kept indefinitely for AAA and FAX
Statistics: Different granularity (10 minutes, daily) Less than 50GB of data Kept indefinitely
12 February 2014Alexandre Beche – GDB
17IT-SDC
Main improvements
ALICE traffic is now included No more duplication at the raw data
level WLCG view is an aggregation over APIs
FTS Dashboard is much faster Does not have the XRootD overhead
Long time period can be queried See next slide
12 February 2014Alexandre Beche – GDB
Alexandre Beche 18IT-SDC 27 May 2013
Activity since June 2012
Impossible with the old architecture
19IT-SDC
Future work Improve the integration of
technology specific dashboards
12 February 2014Alexandre Beche – GDB
20IT-SDC
Future work Integration of the map view
Already available for AAA, FAX and FTS
12 February 2014Alexandre Beche – GDB
21IT-SDC
Application usageincluding WLCG Home page
12 February 2014Alexandre Beche – GDB
22IT-SDC
Summary Completely new architecture
Greater flexibility Easier integration of heterogeneous data
Optimized technology-specific dashboards Aggregated together inside a top level UI
Thanks for your attention
12 February 2014Alexandre Beche – GDB
23IT-SDC
Useful links
AAA Dashboard http://dashb-cms-xrootd-transfers.cern.ch
FAX Dashboard: http://dashb-atlas-xrootd-transfers.cern.ch
FTS Dashboard: http://dashb-fts-transfers.cern.ch
WLCG Transfers Dashboard: http://dashb-wlcg-transfers.cern.ch
12 February 2014Alexandre Beche – GDB