DDM FAX Dashboard status and future Luca Magnoni IT/SDC 2 nd June 2014 News Alexandre left Thanks for all the work! New people take over: Luca Magnoni Marian Babik Domenico Giordano (continues Data Popularity) Cristovao Cordeiro Collaboration from Dubna and Brunel university 2nd June 2014 DDM FAX dashboard / status update 2 The XRootD Dashboard project Dashboard project as common ground for monitoring miscellaneous data transfer services XRootD, FTS, etc. Same architecture and code /different instances: ATLAS FAX, CMS AAA, (LHCb) 2nd June 2014 DDM FAX dashboard / status update 3 Architecture (for XRootD ) 2nd June 2014 DDM FAX dashboard / status update 4 FAX dashboard dashb-atlas-xrootd-transfers.cern.ch Twiki: https://twiki.cern.ch/twiki/bin/view/LCG/WLCGDataTransferMonitoring https://twiki.cern.ch/twiki/bin/view/LCG/WLCGDataTransferMonitoring Jira: https://its.cern.ch/jira/browse/DASHB/component/15024 https://its.cern.ch/jira/browse/DASHB/component/15024 Mailing List: 2nd June 2014 DDM FAX dashboard / status update 5 Open Issue / Solution Issue: Topology resolution Incorrect accounting for sites on the same domain (e.g. in2p3.fr) Solution: improve host/domain to site mapping and rely on server-provided site-name Work already started (Cristovao) Servers have to report site-name correctly 2nd June 2014 DDM FAX dashboard / status update 6 Open Issue/Solution II Issue: EOS data missing EOS data already collected Statistics generation is the bottleneck PL/SQL procedure latency proportional to raw data volume Not matter which statistic method used Solution: promising results on new storage/processing technology HDFS/MapReduce Scale horizontally ready for other transfer services/protocols In the framework of the WLCG Analytics project : https://twiki.cern.ch/twiki/bin/view/LCG/WLCGMonDataAnalytics https://twiki.cern.ch/twiki/bin/view/LCG/WLCGMonDataAnalytics Temporary fix: Try to optimize the PL/SQL procedure, if possible 2nd June 2014 DDM FAX dashboard / status update 7 Open issue/Solution III Issue: Transfer accounting for for Multi- VO sites Static mapping between XRootD server/Gled collector/broker topic Solution: GLED-mix with multi VO filtering in autumn Others: missing concepts of failures from XRootD reporting 2nd June 2014 DDM FAX dashboard / status update 8 On data consistency Comparison between Monalisa and DDM FAX dashboard was done for the last week ( thanks Igor, from Dubna ) very good overall data consistency (< 1-5% diff) few exceptions, with major discrepancies Site not reporting on f-stream detailed result herehere Comparison can be easily automated we are working on a comparison web tool for systematic validation 2nd June 2014 DDM FAX dashboard / status update 9 On the raw data access No plan to support programmatic way to access raw data Planning to offer snapshot for data mining Files DB dump 2nd June 2014 DDM FAX dashboard / status update 10 Now, Your turn! What you would like to get from the tool? Missing data? If it is on the reporting side, we cannot do much! If you experience issues, let us know 2nd June 2014 DDM FAX dashboard / status update 11