worldwide lhc computing grid status and progress report ... · worldwide lhc computing grid status...

Worldwide LHC Computing Grid

Status and Progress Report

October 2014 – March 2015

24 April 2015

Table of Contents

Services and Applications WLCG Services ......................................................................................... 3 Grid Deployment Board ............................................................................. 6 Applications Area ....................................................................................... 8

Experiments ALICE ...................................................................................................... 11 ATLAS ..................................................................................................... 12 CMS......................................................................................................... 13 LHCb ....................................................................................................... 15

WLCG Tier-1 Sites Reliability .............................................................................. 17 Tier-2 Sites Reliability .............................................................................. 25

WLCG – Status and Progress Report 2015H1 (October 2014 – March 2015) 1

WLCG Service Report October 2014 – March 2015

Maria Dimou, Andrea Sciabà

April 2015

The Argus support issues were discussed in two dedicated meetings and a community is now forming, with tangible results: a failure observed with ARGUS-PAP was solved in March. GGUS was agreed as perfectly suitable for issue tracing.

The end of support for lcg-utils was announced in November, being replaced by the gfal-util clients.

Several services at CERN have been decommissioned in this period, including the gLite WMS, the SLC5-based interactive and batch resources, the AFS-based User Interface installation, Savannah and VOMRS.

VOMS-Admin has been successfully commissioned as a replacement for Grid user registration, with only minor remaining issues being followed up in GGUS. Similarly, the new VOMS servers (with SHA-2 compliant host certificates) have been validated against all experiment workflows.

The Tier-0 critical services were updated for Run2 and presented at the Operations Coordination and Management Board meetings.

A number of vulnerabilities and bugs were observed and quickly patched following a good information dissemination to the WLCG sites (Shellshock in October, a RHEL6.6 kernel Fuse bug in December, and GNU C lib in February).

The WLCG Operations working groups and task forces made significant progress in several areas: the Network and Transfer Metrics WG on infrastructure monitoring tools and procedures; the Middleware Readiness WG by verifying multiple versions of DPM, dCache, StoRM at multiple sites using the ATLAS and CMS workflows; the Squid Monitoring TF by documenting configuration options; the gLExec TF by continuing Panda testing; the IPv6 TF by expanding in number of sites; the Multicore deployment TF by disseminating information on APEL publishing.

Between October and November, the experiments successfully performed several production and deployment campaigns. ATLAS completed the migration to Rucio and the DC14 production, CMS ran the PHYS14 and Run2 MC productions and LHCb ran two stripping campaigns.

A rich agenda was prepared for the April WLCG Workshop in Okinawa, including current operations and plans for HL-LHC.

A WLCG Site survey was launched in November and collected valuable input from about one hundred WLCG sites. Preliminary results were presented at the March GDB and MB meetings and the implications for the future evolution of WLCG operations will be discussed at the WLCG workshop.

Summary of Main Service Incidents An overview of the main incidents for which a Service Incident Report (SIR) was produced is provided below. N.B. SIRs are reported by calendar quarter: for consistency and to facilitate the comparison with previous numbers, Figures 1 and 2 still provide the breakdown per quarter rather than for the full semester.

The full list of SIRs is available at https://twiki.cern.ch/twiki/bin/view/LCG/WLCGServiceIncidents


https://twiki.cern.ch/twiki/bin/view/LCG/WLCGServiceIncidents

Site Date(s) Duration Service Area Summary

SARA 15/1/15 25 days dCache Storage Data loss that resulted from hardware failure in a dCache pool node

IN2P3-CC 26/11/14 1.6 hours VO-boxes Network A switch connecting a whole rack to the rest of the site broke, causing various services (all VO-boxes) to become unusable

CERN 14/10/14 4 hours CASTOR Storage All CMS transfers from CASTOR to EOS got blocked due to an outage of the SRM service

CERN 11/10/14 5 hours CASTOR Storage All CMS transfers from CASTOR to EOS got blocked due to an outage of the SRM service

KIT 30/9/14 - Tapes Storage 424 files on 28 tapes from 2011 were found affected by a bug in TSM that caused an EndOfData marker to be written at the wrong place

Table 1 - Summary of SIRs in Q4 2014 and Q1 2015

Figure 1 - Service Incidents by Area


Figure 2 - Time to Resolution

Summary and Conclusions Smooth operations, also during the Christmas shutdown and the experiment challenges/campaigns. Several products were decommissioned/migrated successfully. Good progress towards operational cost reduction thanks to the feedback received via the site survey results.


GDB Report October 2014 – March 2015

Michel Jouvin

April 2015

Introduction There were six GDB meetings held during this period, one of them outside CERN. The agendas show the topics covered, the slides presented and associated papers and the meeting summaries. They can be viewed at http://indico.cern.ch/categoryDisplay.py?categId=3l181. GDB remained lively with a lot of topical discussions, several of them with a dedicated pre-GDB meeting (5 during this period)

Pre-GDBs covered several new technologies: - Volunteer computing - HTCondor

Some of them were dedicated to follow up discussions on cloud and data management/storage issues.

To improve discussion and action follow-up, meeting summaries are produced after each meeting and a consolidated list of actions decided is updated after each meeting. All this material is publically available at https://twiki.cern.ch/twiki/bin/view/LCG/WLCGGDBDocs.

During GDB, we got reports of two grid infrastructure providers for WLCG: EGI, OSG.

Main Topics New generation of HS benchmark: the current plan, build a new HEP-SPEC benchmark based on the upcoming SPEC14 suite, has not materialized yet as the SPEC14 suite release has been delayed. It is now expected during autumn 2015. In the meantime, some work was done to understand the real discrepancies between HEP-SPEC06 and the experiment applications: it turned out that most problems reported were linked to misusage of the benchmark (wrong compiler or compiler options).

IPv6: due to the IPv4 address exhaustion, several WLCG sites confirmed that they will operate resources (WNs) reachable only by IPv6 in the near future. Since several years, a joint HEPiX/WLCG WG has been driving an effort to get the WLCG infrastructure ready for IPv6 with both sites and experiments. Since last autumn, we have seen a significant progress in both application and site readiness. Most of the applications, including experiment frameworks, are now IPv6 ready (http://hepix-ipv6.web.cern.ch/wlcg-applications) or with a coming release that will bring it. More than 10 sites are now participating in the transfer test bed, recently migrated to FTS3. And a good number of sites have concrete short term plans, even though 40% of them have no plan yet.

Batch systems: many European WLCG sites rely on Torque/MAUI, the free batch system promoted by the grid middleware originally. MAUI being unmaintained, there has been for quite some time a concern about possible security vulnerabilities that may lead to shutting down a significant part of WLCG resources. The GDB has been charged by WLCG MB to organize the discussion and exchange of experience around alternatives to Torque/MAUI. After the successful migration of some sites, HTCondor has emerged as the main alternative with a real European community building up.

Clouds: the work of the WG created early 2013 is continuing with a best-effort participation of its members. This WG explores if it is feasible to replace a CREAM CE by a cloud infrastructure to provision computing resources. Significant progress has been made on possible approaches to dynamic sharing of clouds between several user communities with at least two complementary approaches: an external component to the cloud or a cloud scheduler. Accounting is the other hot


http://indico.cern.ch/categoryDisplay.py?categId=3l181

https://twiki.cern.ch/twiki/bin/view/LCG/WLCGGDBDocs

http://hepix-ipv6.web.cern.ch/wlcg-applications

topic for cloud adoption in WLCG, in particular to report on the usage of pledged resources. Work to be done on the accounting infrastructure is in progress and is expected to be done by the summer. A TF has been set-up in January 2015 to do the gap analysis and propose a work plan: the result is expected in spring 2015.


Applications Area Report October 2014 – March 2015

Benedikt Hegner

April 2015

ROOT A second major release for ROOT version 6.02 was made available at the end of September 2014, followed by three more bug fix releases since then. Compared to 6.00, the ROOT team managed with 6.02 to reduce the memory requirements and the startup time up to a level that was adequate for the LHC experiments to decide its adoption for the 2015 run. This has been possible with the help of all the LHC experiments and in particular LHCb, CMS and ATLAS that continuously tested the new version in their nightly integration builds. The ROOT team also improved its overall usability while adding only very few new features. A large number of memory and performance improvements are still ahead, resulting in future versions to be even more lean and mean. The next major release is scheduled for the end of May 2015.

Several follow-up versions have been released for the 5.34 branch to fix a number of issues detected by end-users and the LHC experiments. In general we have established that a new patch release is produced every month. This provides experiments with fixes in a timely manner, which is very important at this period with the preparation of the new physics run in LHC.

Over the period of reference 476 new issues (bugs, feature requests) have been created and 416 issues have been resolved.

Simulation The new release 10.1 of Geant4 was announced on 5th December 2014, according to schedule. Geant4 10.1 introduces explicit use of C++11 types for multi-threading (for thread-local data and atomics) in conjunction with the CLHEP version 2.2 series, with revised use of random number engines directly from CLHEP. This features is activated only on compilers fully supporting C++11.

A new memory allocation strategy for navigation-histories data is introduced, aiming to reuse memory previously allocated and considerably reduce overall memory churn due to blind copy of vector collections. In this release, an option allows an installation to use the Unified Solids Library primitives to replace the original Geant4 solids, enabling a build against an external installation of the USolids library.

Among the various features introduced in the EM physics module, to note the introduction of a new tool class and updated set of UI commands, aiming to provide easier access to definition of EM-specific parameters both in sequential and multi-threaded modes. All internal physics tables are shared between threads, thus reducing contribution of electromagnetic physics to total memory per thread.

The Fritiof (FTF) hadronic model has been significantly improved, resulting in closer agreement with thin-target data. Revisions applied in string-fragmentation affect the meson production and in particular the pi-zero fraction of showers. The fragmentation part of the Quark-Gluon-String (QGS) model has been improved significantly; the hadronic showers produced with QGS-based physics-lists become wider and longer, with lower energy response, making them in closer agreement with those produced by FTF. The INCLXX model has been extended up to 15-20 GeV incident energy.

Release 10.1 largely consolidates the multi-threading technology introduced in the release 10 series. In 10.1, the memory space required per thread is reduced to less than half, and an overall speedup up to 10% is measured in several use-cases. The detailed list of fixes can be found in the release notes, http://cern.ch/geant4/support/ReleaseNotes4.10.1.html.


https://cern.ch/aidasoft/USolids

http://cern.ch/geant4/support/ReleaseNotes4.10.1.html

New patch releases of Geant4 for the 10.0 series, Geant4-10.0.p03 and Geant4-10.0.p04, have been also released in October and recently in March respectively. The patches include mainly fixes/improvements to multi-threading, back-porting fixes introduced in the last release. The detailed list of fixes can be found in the release notes, http://cern.ch/geant4/support/Patch4.10.0-3.txt and http://cern.ch/geant4/support/Patch4.10.0-4.txt.

Validation of Geant4-10.0.p04 has been made by CMS, planning to start production this year with it; CMS is already adopting MT builds and planning to use 10.1 for the current development.

A patch release for the 9.6 series, Geant4-9.6.p04 has also been released last January, providing a collection of back-ported fixes from the 10 series and addressing in good part issues reported by ATLAS. All the details can be found in the release notes, http://cern.ch/geant4/support/Patch4.9.6-4.txt.

CORAL and COOL Three new releases of the 3.0 branches of CORAL and COOL have been prepared for ATLAS, based on successive patch releases of ROOT 6.02. All changes to the code bases have also been backported to new releases of the CORAL 2.4 and COOL 2.9 branches, based on ROOT5. The main change has been the implementation of a PyCool patch to avoid crashes in PyROOT when C++ exceptions are thrown: this issue is expected to be fixed with the upcoming move of ROOT6 from JIT to ORCJIT, but a temporary workaround had to be added as this issue became a blocker for ATLAS and its priority was escalated. A second major improvement has been the integration of the CORAL and COOL test suites, still based on CMT, with the new nightly build infrastructure, now based on cmake. Other notable changes include the transfer of responsibility to ATLAS for the deployment of Oracle client-side configuration files, the integration of CoralServer tests with more recent versions of the ATLAS HLT software, bug fixes for the Frontier backend of CORAL and the porting to CERN CentOS7 and gcc49.

The 3.0 branch of COOL is also being tested in the nightly build infrastructure against the ROOT6 master branch, which includes the move from JIT to ORCJIT and a major reimplementation of PyROOT. This change is expected to bring many improvements and simplifications in PyCool, but progress is presently prevented by a blocking bug in PyROOT.

The migration of the CORAL server test infrastructure and CORAL/COOL test nodes from quattor to Puppet has been completed. The CORAL and COOL test suites have been adapted accordingly.

CernVM file system The main activity of the past months has been the preparation of version 2.1.20, which was finally tagged on 10th February 2015. The standard validation procedure, involving the main Stratum-1 sites, and is due to be completed by 27th March 2015. This release contains several consolidation fixes and some new functionality, namely: automatic garbage collection, S3 backend, new structure for key and config file distribution. Given the all the community is by now on version 2.1, upgrade to this version is expected to be smooth. Version 2.1.20 also works on SL 7.

CernVM appliance Consolidation of CernVM 3.3, based on SL 6.5, has continued; 10 security hotfix patches have been released following the related Scientific Linux patch releases. Version CernVM 3.4, based on SL 6.6, is under preparation and expected to be available during Q2 2015. Work has started to support SL 7; a first public version is expected by end of Q3 2015. Prototyping of integration with Docker containers has also started.


http://cern.ch/geant4/support/Patch4.10.0-3.txt




Data Preservation / Open Data The main activity in this context has been the support given to all the experiments participating to the Open Data pilot project for encapsulated images based on CernVM and the bootloader technology; the project has been launched at the end of October 2014.

CernVM Users Workshop The workshop, which was held at CERN on 5th and 6th March 2015, was well received with attendance peaks of about 80 people. The main outcomes, in terms of new requirements, were: the need to strengthen the support for HPC resources, implying dedicated solutions for CernVM-FS; a request for better disclosure and documentation of the procedure to create images and operating systems templates in CernVM-FS with the bootloader technology; the need for access control or sensitive content protection in CernVM-FS. The integration of these requests in the development plans in under discussion.

SPI In the mentioned period, successive ROOT5 versions 5.34.22 and 5.34.24 have been included in the major releases LCG_70 and LCG_71 together with updated COOL and CORAL versions. In addition, ROOT5.34.25 has been included in a minor LCG_72a release created to upgrade the version of fastjet to 3.1.1, following an ATLAS request. The ROOT6 versions 6.02.02, 6.02.03, 6.02.04 and 6.02.05 have been included in LCG_71roo6, LCG_72root6, LCG_73root6 and LCG_74root6 respectively, together with the new CORAL and COOL versions prepared for ATLAS. Regarding ROOT6 releases, Python and pylint versions have been ungraded in LCG_73root6 and Eigen 3.2.4 has been provided in LCG_74root6.

In terms of the platforms and compilers, slc6 for gcc48, gcc49 (both in debug and release types) on 64 bit have been the common setup for these releases. LCG_70, LCG_71 and LCG_71root6 have been the last releases created also in 32 bit as required by ATLAS. In this period the automatization of LCG release builds using the new Jenkins infrastructure has been completed.

In terms of regular nightly builds, a new slot dev3 has been setup together with the already existing dev2 and dev4 slots, including the ROOT master version. CentOS7 regular builds in dev3 and Ubuntu14.04 builds in both dev3 and dev4 have been also added in this period. The compiler versions gcc4.8.1 and gcc4.9.1 for CentOs7 have been released to AFS. In addition, the latest clang3.6 version has been also provided in this area.

As of 19th February, Savannah has been fully deprecated providing exclusively static pages.

In terms of outside collaboration, the participants of the Librarians and Integrators Meeting initiated the creation of a new Packaging Working Group within the HEP Software Foundation.


ALICE Report October 2014 – March 2015

Predrag Buncic

April 2015

Data processing, analysis and preparation for Run 2

During the reporting period ALICE has finished the reprocessing of 2010 p+p data (final pass3) and of the majority of 2012 p+p and p+A data (final pass2 and pass 3). In total, 900 million events were reconstructed in 2010 and about 2 billion events were reconstructed in 2012. The procedure included full offline recalibration and processing, similar to the Run 1, and all data was processed with the same code, which was one of the main parameters of the exercise. The new reconstruction was fully evaluated through detector QA and standard physics QA. If there is sufficient time before the collisions start in the summer, we plan to reprocess also the 2011 p+p data.

The general-purpose MC associated to the 2010 reconstruction was completed in December 2014, profiting from the excellent running conditions on the Grid during the Christmas break. The same type of MC production for the 2012 data reprocessing is being prepared and will start imminently. Specific signal-embedding productions are being prepared by the Physics Working Groups and will be run according to the analysis plan set by the ALICE Physics coordination.

In addition to that, a set of specific MC requests have been completed, amounting to 60 production cycles and about 2 billion events. A large-scale MC production with the full Run 2 detector setup and the new gas mixture in TPC was also completed. The Geant4 tests and integration are progressing with new versions of G4 being continuously rolled in production and every G4 simulation being evaluated for performance.

The planned AliRoot split was completed into AliRoot core, which includes the simulation, reconstruction and steering part, and AliPhysics, which contains the Physics Working Groups analysis software. The release cycles have been correspondingly adjusted, with AliRoot core being released with low frequency and after a rigorous unit testing, while AliPhysics is released daily and being used for organized (trains) and individual user analysis.

The development of the code to be used in HLT for calibration and data compression is completed and validated on the test HLT cluster. The porting on the production cluster is ongoing, together with the integration of the production cluster for Grid use.

The Grid utilization in the accounting period was exceptionally good. Between November 2014 and February 2015, ALICE was able to use opportunistically resources about 10% higher than the standard allocation. The planned site upgrades were done with minimal disruption of overall Grid QoS. The CPU/Wall efficiency remained constant, at about 85% across all tiers.

The ALICE Run 2 commissioning began in September 2014 and complete set of shifts with continuous data taking started in January 2015. The offline processing of the collected data with cosmics and calibration triggers is ongoing, in parallel with the data taking. This data is used to evaluate the working state of the ALICE reconstruction code, the detectors status as well as for initial alignment. As of the writing of this report, more than 100 million cosmics trigger have been collected, reconstructed and available for analysis.


ATLAS Report October 2014 – March 2015

Eric Lancon, Simone Campana

April 2015

The ATLAS experiment completed the commissioning of the new Workload Management system (Prodsys-2) and the new Data Management system (Rucio) in the last three months of 2014. Both Prodsys-2 and Rucio went in production on 1st December 2014 after a period of time when they operated in parallel to the old systems. The migration was not meant to be transparent and therefore ATLAS stopped production activities for a few days and had a slow ramp up in December, to monitor the behavior of the newly commissioned tools in production. The Monte Carlo Production activity was therefore lower than usual in the end of the year, while data analysis (not affected by the migration to Prodsys-2) could ramp up and utilize more spare cycles.

In December and January ATLAS validated the offline software to be used for simulation in Run 2. Massive event generation and detector full simulation started in February, with the goal to produce the Monte Carlo sample at the running condition of Run 2 in in 2015. This will consist in more than one billion events, which will be reconstructed in early spring. To accommodate this large workload, since the middle of February ATLAS is concurrently utilizing an average of 130k CPU cores, where 30% cover the analysis use case and the rest are dedicated to production. The production activity is carried on both in single-core and multi-core job slots (in approximately equal ratio), depending on the workflow.

To accommodate for the newly launched Monte Carlo production and prepare the storage for Run 2 data taking, ATLAS carried on massive data reorganization, leveraging the Data Lifetime model defined in autumn 2014 and carefully rolled out in production afterwards. For the first time we cleaned data from tape storage, deleting all the simulated data with expired lifetime and the ESD data sets from CERN storage. This allowed freeing up 10.8 PB of tape storage at Tier1s and 10.5 PB at CERN. For disk storages we identified 4.7 PB of files at Tier1s and Tier2s that could be marked eligible for deletion in case space was needed. Some physics groups started archiving on tape the derived data utilized for published papers (and cleaning the corresponding files from disk) where the SUSY group played an important role, archiving 500 TB of data and releasing the corresponding disk space to the centrally managed pool. Finally, 3.1 PB of data has been redistributed around most Grid sites, to balance the storage utilization with more flexible boundaries between Tier1s and Tier2s.

The focus of the next months will be on the startup of data taking. In particular, from the offline software perspective, ATLAS will concentrate in finalizing the reconstruction software version to be used at Tier0 and for Monte Carlo samples. From the distributed computing area, more functionality of Rucio (flexible space management, support for multi protocols) and Prodsys-2 (internal merging, support for complex workflows) will be brought in production. The main activity will of course concentrate on the first pass processing of the LHC data and the reconstruction of the Monte Carlo samples being simulated.


CMS Report October 2014 – March 2015

Maria Girone, David Lange

April 2015

The software and computing groups have focused on completing the ambitious development program undertaken during LS1, finalizing software, commissioning of the data federation for production and analysis workflows, completing the scale testing of the Tier0 and Tier1 infrastructure for distributed prompt-reconstruction, while preparing for Run 2 operations.

Since the last RRB, the computing group has been finalizing the deployment of the remaining components of the Run 2 system. Areas of recent developments include multicore workflow deployment at Tier0 and Tier1s, deployment of the new generation of the distributed analysis tool (CRAB3), the transition from CRAB2 to CRAB3 by the CMS user community, the commissioning of the HLT farm for data reprocessing during LHC machine technical stops and winter shutdown, and finally the deployment of a central Condor pool for easier prioritization of analysis and production workflows.

CRAB3 brings important improvements in terms of faster client, improved automatic resubmissions and centralized handling of the user outputs. CRAB3 is now our default analysis software, and has been adopted by a substantial fraction of analysis users over the last several months.

The commissioning of the HLT for production work during shutdown and machine development periods has made a huge resource available to CMS computing. The HLT farm is similar in size to the total Tier-1s, and even with only a few months of dedicated use it allowed computing to avoid requesting a further 25% increase in Tier-1 processing capacity. This commissioning activity represents a savings of millions of Swiss francs in 2015.

Furthermore the techniques used to make the HLT available directly impacted the Tier0 commissioning on the CERN Agile Infrastructure and improved tools for opening other opportunistic sites. CMS is working in 2015 on developing the capability to use more opportunistic computing as an affordable way to improve contingency.

Another computing accomplishment with significant impact has been the final deployment of the data federation. An Xrootd based data federation called AAA has been deployed at CERN, all Tier1s, and most of the Tier2s allowing nearly all of the CMS stored on disk to be accessible over wide area access. This provides a fall back channel to mitigate local file access problems, it allows workflows to be shared across sites by serving the data remotely, and it provides much better access to samples for CMS users. AAA is targeted at 20% of the total access and scaling tests throughout 2014 met these goals.

In 2014 computing has deployed a system for dynamic data placement. The program began with a monitoring activity to identify how well the disks were utilized in terms of accessing stored samples. In response to the monitoring plots a series of automated subscription, replication, and cleanup scripts have been deployed. The ultimate goal is to optimize the utilization of the disk storage resources, an important component of the overall computing budget, and to reduce the future projected increase in disk by using what we have more efficiently.

The software group continues to produce improved software releases to support the development of the major fronts of CMS. The two areas of primary development are Run 2 and Phase-II upgrades, where the reconstruction software continues to evolve.

The configuration for CMS simulation of Run 2 is now finalized, and has been essentially unchanged since the fall. The one still evolving component is the generator configuration for Run 2. CMS has used recent months to validate the new generator versions using a data driven approach. Now that this work has reached its conclusion, the Run 2 GEN-SIM sample production is now well underway.


The Run 2 reconstruction application continues to evolve to be ready for the new challenges of Run 2. We have continued to improve both the physics performance, particularly in preparation for 25 ns LHC operations, and technical performance. The software release for data taking is currently in its final preparation phases. Recent accomplishments include the adoption of ROOT6 and continued adoption of the multi-threaded framework within the simulation and reconstruction areas.

The CMSSW_6_2_X_SLHC release cycle continues to be the basis of simulation studies to support the preparation of the CMS HL-LHC Technical Proposal. This release supports both Phase-I and Phase-II detector studies. The Phase-II simulation configurations for the Technical Proposal are now finalized, including recent work on the forward calorimetric systems, forward region shielding and support materials. There is a continued focus on the challenges of developing robust reconstruction algorithms against the very high pileup conditions, up to an average pileup of 200 interactions per bunch crossing. Large Monte Carlo sample generation campaigns have continued as the software evolves. We have made large improvements in both the operational and reconstruction quality of these very challenging applications.

Computing resource usage for production activities in recent months has been dominated by the “PHYS14” exercise and samples for the Technical Proposal. The slightly under pledge usage of the CMS computing cycles is expected since the final software for Run 2 is still under development. Once it is completed, the Monte Carlo sample production will fully utilize resources until the start of Run 2 and beyond.

Deployment testing continues on numerous fronts as development is completed and resources are available. Among our recent accomplishments is testing of the CMSSW multithreaded application on both the Tier0 and Tier1 resources. This is an initial demonstration that the infrastructure deployed for this purpose can be used efficiency and effectively.


LHCb Report October 2014 – March 2015

Marco Cattaneo

April 2015

Real data processing The full re-stripping of the 2011-2012 dataset, intended to be the legacy dataset for Run 1, began at the end of November, several weeks later than planned. The delay in commissioning this stripping was due to its increased complexity: it contains twice more lines than previous strippings and includes the migration to microDST format for a majority of lines, in preparation of Run 2. The input data were pre-staged from tape using FTS3. This helped to fix a number of issues in FTS3 and meant that, once the stripping campaign started, it could be completed smoothly in 6 weeks as planned. Most sites were very effective in staging at very high rate, except one that is limiting itself to the pledged bandwidth. Unfortunately, just as the campaign was ending, a major storage failure at one of the Tier1s caused the loss of a small fraction of intermediate files that had not yet been collected into the final output files. The lost files were scattered in the whole data-taking period and recovering them would have meant re-running one third of the whole stripping; it was decided not to perform this recovery due to the human effort involved. The loss represents between 0.03% and 1% of the total luminosity, depending on the stream. Measures have been taken to reduce the operational impact of such an incident in future.

Montecarlo simulation Montecarlo production for the analysis of the 2011 and 2012 data has taken place on demand throughout the period. A version with a preliminary 2015 geometry was used in the autumn for studies of the 2015 HLT bandwidth division. The first production version of the 2015 simulation (Sim09) has been released at the end of the period and will be used in April to prepare the analyses of the measurements to be made on early 2015 data.

Considerable effort has gone into developing strategies for fast simulation. One of these, signal particle gun, is very useful for certain studies and is ten times faster to produce than full simulation; it has been made available as an alternative generator for Sim09.

User analysis During the period under review, analysis jobs accounted for 15.7% of the CPU used on the Grid.

CPU optimisation During this semester we have made major efforts to optimise the LHCb reconstruction code, focusing on the HLT reconstruction, with the aim of running in the HLT a reconstruction as close as possible to offline. The optimisation effort has led to a 30% speed up of the HLT reconstruction, without loss of physics performance, which also benefits the offline reconstruction.

Data Management After the end of the re-stripping campaign, LHCb commissioned and put in production the Dirac File Catalog (DFC), a catalog of files and replicas on the GRID, which replaces the corresponding WLCG service (LFC) that is due to be discontinued. The DFC contains major performance improvements relative to the LFC and has been designed for the load expected at the end of Run 2.

The work on data popularity has continued, concentrating on the development of metrics to predict which datasets will be unpopular in future, to allow for a reduction in the number of replicas kept on disk. Around 1 PB of disk can be recovered by applying these metrics.


Plots The first plot shows the number of running LHCb jobs by job type as a function of time. CPU usage continues to be dominated by simulation, but the impact of the re-stripping campaign is clearly visible. Resources pledged to LHCb correspond to about 27000 concurrent jobs.

The second plot is a graphical representation of the CPU time spent in the HLT algorithms: each rectangle represents an algorithm, whose area is proportional to the number of CPU cycles spent in that algorithm. The top picture is taken before optimisation, the bottom after optimisation, at the same scale.


WLCG Sites Reliability

ALICE March 2015

From Oct-2014

Target for each site is 97.0%

Colors: Green > Target Orage > 90% of Target Red < 90% of TargetAvailability Algorithm: CREAM-CE + ARC-CE

Average of 8 best sites

(not always the same sites)

Month Reliability

Oct-2014 100%

Nov-2014 100%

Dec-2014 100%

Jan-2015 100%

Feb-2015 100%

Mar-2015 100%

Average of ALL Tier-0 and

Tier-1 sites

Month Reliability

Oct-2014 100%

Nov-2014 99%

Dec-2014 100%

Jan-2015 100%

Feb-2015 100%

Mar-2015 99%

Detailed Monthly Site Reliability

Site Oct-2014 Nov-2014 Dec-2014 Jan-2015 Feb-2015 Mar-2015

CERN 99% 100% 100% 100% 100% 100%

CCIN2P3 100% 100% 100% 100% 100% 97%

CNAF 100% 100% 100% 100% 100% 100%

FZK 100% 100% 100% 100% 100% 99%

KISTI_GSDC 100% 100% 100% 100% 100% 100%

NIKHEF 100% 100% 100% 100% 100% 98%

RAL 100% 100% 99% 100% 100% 100%

RRC_KI_T1 100% 100% 100% 100% 100% 100%

SARA 100% 95% 100% 100% 99% 100%

Target 97% 97% 97% 97% 97% 97%



ATLAS March 2015

From Oct-2014


Colors: Green > Target Orage > 90% of Target Red < 90% of TargetAvailability Algorithm: (OSG-CE + CREAM-CE + ARC-CE) * (all SRMv2 + all OSG-SRMv2)



Month Reliability

Oct-2014 99%

Nov-2014 99%

Dec-2014 100%

Jan-2015 100%

Feb-2015 100%

Mar-2015 99%


Tier-1 sites

Month Reliability

Oct-2014 98%

Nov-2014 95%

Dec-2014 98%

Jan-2015 99%

Feb-2015 98%

Mar-2015 97%



CERN-PROD 100% 98% 100% 99% 97% 91%

BNL-ATLAS 98% 99% 99% 98% 100% 99%

FZK-LCG2 97% 90% 95% 100% 100% 91%

IN2P3-CC 100% 99% 100% 100% 100% 96%

INFN-T1 99% 99% 99% 99% 99% 96%

NDGF-T1 97% 100% 93% 93% 100% 99%

NIKHEF-ELPROD 96% 99% 99% 95% 99% 98%

RAL-LCG2 97% 99% 94% 100% 99% 99%

RRC-KI-T1 100% 61% 100% 100% 91% 99%

SARA-MATRIX 100% 95% 100% 100% 97% 96%

TRIUMF-LCG2 99% 99% 100% 100% 100% 100%

Taiwan-LCG2 98% 91% 99% 99% 94% 100%

pic 99% 100% 100% 100% 100% 98%

Target 97% 97% 97% 97% 97% 97%



CMS March 2015

From Oct-2014


Colors: Green > Target Orage > 90% of Target Red < 90% of TargetAvailability Algorithm: (OSG-CE + CREAM-CE + ARC-CE) * (all SRMv2 + all OSG-SRMv2)



Month Reliability

Oct-2014 99%

Nov-2014 100%

Dec-2014 98%

Jan-2015 100%

Feb-2015 99%

Mar-2015 97%


Tier-1 sites

Month Reliability

Oct-2014 99%

Nov-2014 100%

Dec-2014 98%

Jan-2015 99%

Feb-2015 88%

Mar-2015 86%



T0_CH_CERN 94% 100% 98% 100% 98% 91%

T1_DE_KIT 100% 100% 97% 100% 100% 93%

T1_ES_PIC 100% 100% 98% 100% 99% 98%

T1_FR_CCIN2P3 99% 98% 96% 93% 96% 99%

T1_IT_CNAF 99% 100% 98% 100% 98% 98%

T1_RU_JINR 99% 100% 99% 100% 99% 97%

T1_TW_ASGC 100% 100% 100% 100% N/A N/A

T1_UK_RAL 97% 98% 95% 100% 98% 99%

T1_US_FNAL 100% 100% 100% 100% 100% 100%

Target 97% 97% 97% 97% 97% 97%



LHCB March 2015

From Oct-2014


Colors: Green > Target Orage > 90% of Target Red < 90% of TargetAvailability Algorithm: (CREAM-CE + ARC-CE) * all SRMv2



Month Reliability

Oct-2014 100%

Nov-2014 100%

Dec-2014 99%

Jan-2015 100%

Feb-2015 99%

Mar-2015 100%


Tier-1 sites

Month Reliability

Oct-2014 99%

Nov-2014 99%

Dec-2014 98%

Jan-2015 99%

Feb-2015 99%

Mar-2015 99%



LCG.CERN.ch 99% 96% 90% 100% 100% 100%

LCG.CNAF.it 98% 100% 96% 100% 97% 100%

LCG.GRIDKA.de 100% 100% 100% 100% 100% 97%

LCG.IN2P3.fr 99% 100% 100% 100% 95% 100%

LCG.NIKHEF.nl 99% 100% 100% 100% 100% 98%

LCG.PIC.es 100% 100% 100% 100% 100% 100%

LCG.RAL.uk 99% 100% 96% 93% 99% 99%

LCG.RRCKI.ru 100% 100% 100% 99% 100% 100%

LCG.SARA.nl 100% 95% 100% 100% 99% 100%

Target 97% 97% 97% 97% 97% 97%


Availability of WLCG Tier-0 + Tier-1 Sites

ALICE Oct-2014 - Mar-2015

Target Availability for each site is 97.0%. Target for 8 best sites is 98.0%

Availability Algorithm: CREAM-CE + ARC-CE

CERN Avail: 99% Unkn: 1% CCIN2P3 Avail: 98% Unkn: 1% CNAF Avail: 100% Unkn: 0% FZK Avail: 100% Unkn: 0%

KISTI_GSDC Avail: 97% Unkn: 1% NIKHEF Avail: 100% Unkn: 0% RAL Avail: 100% Unkn: 0% RRC_KI_T1 Avail: 100% Unkn: 0%

SARA Avail: 97% Unkn: 2%



ATLAS Oct-2014 - Mar-2015


Availability Algorithm: (OSG-CE + CREAM-CE + ARC-CE) * (all SRMv2 + all OSG-SRMv2)

CERN-PROD Avail: 98% Unkn: 1% BNL-ATLAS Avail: 99% Unkn: 0% FZK-LCG2 Avail: 95% Unkn: 0% IN2P3-CC Avail: 97% Unkn: 1%

INFN-T1 Avail: 98% Unkn: 0% NDGF-T1 Avail: 97% Unkn: 0% NIKHEF-ELPROD Avail: 93% Unkn: 1% RAL-LCG2 Avail: 98% Unkn: 0%

RRC-KI-T1 Avail: 85% Unkn: 0% SARA-MATRIX Avail: 96% Unkn: 2% TRIUMF-LCG2 Avail: 99% Unkn: 0% Taiwan-LCG2 Avail: 96% Unkn: 1%

pic Avail: 99% Unkn: 0%



CMS Oct-2014 - Mar-2015



T0_CH_CERN Avail: 96% Unkn: 1% T1_DE_KIT Avail: 98% Unkn: 1% T1_ES_PIC Avail: 98% Unkn: 1% T1_FR_CCIN2P3 Avail: 95% Unkn: 2%

T1_IT_CNAF Avail: 99% Unkn: 1% T1_RU_JINR Avail: 99% Unkn: 0% T1_TW_ASGC Avail: 92% Unkn: -82% T1_UK_RAL Avail: 98% Unkn: 1%

T1_US_FNAL Avail: 100% Unkn: 1%



LHCB Oct-2014 - Mar-2015


Availability Algorithm: (CREAM-CE + ARC-CE) * all SRMv2

LCG.CERN.ch Avail: 97% Unkn: 2% LCG.CNAF.it Avail: 99% Unkn: 1% LCG.GRIDKA.de Avail: 100% Unkn: 0% LCG.IN2P3.fr Avail: 94% Unkn: 11%

LCG.NIKHEF.nl Avail: 100% Unkn: 0% LCG.PIC.es Avail: 99% Unkn: 0% LCG.RAL.uk Avail: 97% Unkn: 1% LCG.RRCKI.ru Avail: 98% Unkn: 1%

LCG.SARA.nl Avail: 98% Unkn: 0%


March 2015 **Tier-2 Availability and Reliablity Report

ALICE

Federation Summary - Sorted by Name *

Color coding: N/A <30% <60% <90% >=90%

Availability Algorithm: CREAM-CE + ARC-CE

Federation Availability Reliability Federation Availability Reliability

CZ-Prague-T2 93% 93% PL-TIER2-WLCG 87% 87%

FR-GRIF 94% 94% RO-LCG 98% 98%

FR-IN2P3-IPHC 100% 100% RU-RDIG 100% 100%

FR-IN2P3-LPC 100% 100% SK-Tier2-Federation 83% 83%

FR-IN2P3-LPSC 100% 100% T2-LATINAMERICA 99% 100%

FR-IN2P3-SUBATECH 100% 100% TH-Tier2 99% 99%

HU-HGCC-T2 100% 100% UA-Tier2-Federation 33% 33%

IN-DAE-KOLKATA-TIER2 97% 97% UK-SouthGrid 100% 100%

IT-INFN-T2 95% 95%

* Federation Details are available at http://wlcg-sam.cern.ch/reports/2015/201503/wlcg/WLCG_All_Sites_ALICE_Mar2015.pdf** Tier-2 availability and reliability reports for previous months are available at http://wlcg-sam.cern.ch/reports



ATLAS


Color coding: N/A <30% <60% <90% >=90%



AT-HEPHY-VIENNA-UIBK 97% 97% JP-Tokyo-ATLAS-T2 94% 94%

AU-ATLAS 100% 100% PL-TIER2-WLCG 82% 82%

CA-EAST-T2 92% 92% PT-LIP-LCG-Tier2 87% 88%

CA-WEST-T2 87% 88% RO-LCG 96% 99%

CH-CHIPP-CSCS 93% 94% RU-RDIG 98% 98%

CN-IHEP 100% 100% SE-SNIC-T2 92% 92%

CZ-Prague-T2 97% 97% SI-SiGNET 99% 99%

DE-DESY-GOE-ATLAS-T2 99% 99% SK-Tier2-Federation 93% 93%

DE-DESY-LHCB 91% 91% T2-LATINAMERICA 99% 99%

DE-DESY-RWTH-CMS-T2 99% 99% TR-Tier2-federation 94% 94%

DE-FREIBURGWUPPERTAL 98% 100% TW-FTT-T2 100% 100%

DE-MCAT 94% 94% UK-London-Tier2 75% 77%

ES-ATLAS-T2 96% 98% UK-NorthGrid 98% 98%

FR-GRIF 99% 99% UK-ScotGrid 98% 98%

FR-IN2P3-CC-T2 93% 100% UK-SouthGrid 87% 87%

FR-IN2P3-CPPM 100% 100% US-AGLT2 86% 86%

FR-IN2P3-LAPP 95% 95% US-MWT2 90% 96%

FR-IN2P3-LPC 100% 100% US-NET2 48% 48%

FR-IN2P3-LPSC 100% 100% US-SWT2 98% 99%

IL-HEPTier-2 92% 92% US-WT2 99% 99%

IT-INFN-T2 91% 97%

* Federation Details are available at http://wlcg-sam.cern.ch/reports/2015/201503/wlcg/WLCG_All_Sites_ATLAS_Mar2015.pdf** Tier-2 availability and reliability reports for previous months are available at http://wlcg-sam.cern.ch/reports



CMS


Color coding: N/A <30% <60% <90% >=90%



AT-HEPHY-VIENNA-UIBK 87% 87% PK-CMS-T2 100% 100%

BE-TIER2 93% 93% PL-TIER2-WLCG N/A N/A

BR-SP-SPRACE 98% 98% PT-LIP-LCG-Tier2 87% 89%

CERN-PROD 82% 83% RU-RDIG 84% 84%

CH-CHIPP-CSCS 96% 98% T2-LATINAMERICA 83% 88%

CN-IHEP 97% 97% T2_US_Caltech 96% 96%

DE-DESY-RWTH-CMS-T2 96% 96% T2_US_Florida 93% 93%

EE-NICPB 88% 88% T2_US_MIT 94% 94%

ES-CMS-T2 100% 100% T2_US_Nebraska 81% 81%

FI-HIP-T2 99% 99% T2_US_Purdue 100% 100%

FR-GRIF 99% 99% T2_US_UCSD 94% 95%

FR-IN2P3-CC-T2 92% 99% T2_US_Wisconsin 95% 95%

FR-IN2P3-IPHC 99% 99% TH-Tier2 82% 82%

GR-Ioannina-HEP 98% 98% TR-Tier2-federation 78% 78%

HU-HGCC-T2 98% 98% UA-Tier2-Federation 72% 93%

IN-INDIACMS-TIFR 88% 90% UK-London-Tier2 95% 99%

IT-INFN-T2 93% 94% UK-SouthGrid 97% 97%

KR-KNU-T2 98% 98%

* Federation Details are available at http://wlcg-sam.cern.ch/reports/2015/201503/wlcg/WLCG_All_Sites_CMS_Mar2015.pdf** Tier-2 availability and reliability reports for previous months are available at http://wlcg-sam.cern.ch/reports



LHCB


Color coding: N/A <30% <60% <90% >=90%

Availability Algorithm: (CREAM-CE + ARC-CE) * all SRMv2


CH-CHIPP-CSCS 98% 100% IT-INFN-T2 93% 96%

DE-DESY-LHCB 100% 100% PL-TIER2-WLCG 88% 88%

DE-DESY-RWTH-CMS-T2 100% 100% RO-LCG 90% 90%

ES-LHCb-T2 98% 98% RU-RDIG 87% 87%

FR-GRIF 100% 100% T2-LATINAMERICA 100% 100%

FR-IN2P3-CPPM 100% 100% UK-London-Tier2 98% 98%

FR-IN2P3-LAPP 100% 100% UK-NorthGrid 98% 99%

FR-IN2P3-LPC 100% 100% UK-ScotGrid 99% 99%

IL-HEPTier-2 60% 60% UK-SouthGrid 100% 100%

* Federation Details are available at http://wlcg-sam.cern.ch/reports/2015/201503/wlcg/WLCG_All_Sites_LHCB_Mar2015.pdf** Tier-2 availability and reliability reports for previous months are available at http://wlcg-sam.cern.ch/reports


worldwide lhc computing grid status and progress report ... · worldwide lhc computing grid status...

Documents