b. hegner, p. mato, p. mendez cern, ph-sft group 1 st development@cern forum cern 28-sep-2015 the...

B. Hegner, P. Mato, P. Mendez

CERN, PH-SFT Group

1ST DEVELOPMENT@CERN FORUMCERN 28-SEP-2015

THE QUALITY AND TESTING INFRASTRUCTURE OF PH-SFT

Quality and Testing PH-SFT infrastructure 2

• Introduction• The LCG build system

• Tools and core elements• Test, nightly and release infrastructures

• Scheduler and Dashboard• Summary and future plans

28/09/15

OUTLOOK


• Development and maintenance of common scientific software for the experiments in collaboration with the PH and IT departments and external HEP institutes

• Areas covered by the group:• Simulation: Geant4, GeantV• Frameworks: ROOT• Distributed systems: CERN VM, CERNVMFS

28/09/15

PH-SFT GROUP


Goals of the project: Provision of consistent

releases of external and LCG application PKGs to both experimental and theoretical communities

Provide daily tests and previews of incoming releases to users

28/09/15

THE PH-SFT PACKAGES AND RELEASES PROJECT WITHIN THE LHC SOFTWARE

STACK

Applications

Event

Experiments specific Packages

External packages non-HEP specific

Detector Calib

Simul. Data

Mngmt. Distrib.

Analysis

Core Libraries

Quality and Testing PH-SFT infrastructure 528/09/15

OUR BUILD PACKAGES AND COMBINATIONS

Externals (~140) MC Generators (~50) Group’s projects

Python, GSL, Boost… PythiaHerwigHepmcanalysis …

ROOTGEANT4About 10 Grid packages

FTS, WMS, DPM…

Compilers Architectures Operating systems

Build types

• gcc4.8.4, gcc4.9.3, gcc5.1.0, gcc5.2.0

• clang35, 37• Icc15• Native

• Intel• arm64

• SLC6• CentOS7• CentOS6• Ubuntu12, 14• Mac 10.8,

10.9, 10.10

• Release• Debug

Packages

Combinations Packages and combinations agreed with experiments


1. Build results reproducibility 2. Manageable handling of package dependencies 3. End-systems independent installation procedures; reallocation4. Proper quality assurance5. Scalability of the infrastructure: increasing number of

compilers, architectures 6. Fast improvement/feedback cycleOverall signifi cant number of challenges common to many projects

that could be commonly managed

HEP Software Foundation (HSF) project

Promote and facilitate the creation, maintenance and lifecycle management of high quality common software of value to HEP and beyond

Forum for collaboration between software developers Identification of community goals and deliverables and ease community

contributions28/09/15

THE PROJECT CHALLENGES


THE CORE ELEMENTS

Toolset.cmake

# Application Area ProjectsLCG_AA_project(COOL 3_0_4)LCG_AA_project(CORAL 3_0_4)LCG_AA_project(RELAX RELAX_1_4_2)LCG_AA_project(ROOT 6.04.02)

# ExternalsLCG_external_package(4suite 1.0.2p1 )LCG_external_package(AIDA 3.2.1 ) LCG_external_package(backports 3.4.0.2 )


THE CORE ELEMENTS

AFS

Toolset.cmake

LCGCMAKE

Sources download

Build instructions

(Cmake, make)

http://

Branch (release number, nightly)ArchitectureCompiler Build typeOperating System

GITLAB repository


THE CORE ELEMENTS

AFS

Toolset.cmake

LCGCMAKE

Packages download

Machinery (Cmake, make)

http://

Packages/rpm repository

Dependencies text (json)

Installation scripts

AFS/CVMFS

<prefix>/<package_name>/<version>_<hash>/<platform_tag>/…


THE CORE ELEMENTS

Toolset.cmake

LCGCMAKE

Packages download

Machinery (Cmake, make)

Packages/rpm repository

Dependencies text (json)

Installation scripts

AFS/CVMFS


Cdash

pub

licatio

n

Release summary web page


INFORMATION ACCESS FOR EXPERIMENTS

AFS/CVMFS


Cdash

pub

licatio

n

Experim

en

ts

Release summary web page


Build and testing infrastructure CMAKE based:• Open-source system dedicated to build, test

and install software• Build specifications included in platform-

independent list fi les • Generation of CMakeFile that drives the full

build• Modular and scalable system

• Available in both AFS and CVMFS

SFT tool based on cmake: LCGCMAKE• Based on “ExternalProject” CMAKE module

implementation• Enable the build of external software

sources• Easy way to control package dependencies

• Repository recently migrated to GITLAB28/09/15

BUILDING FROM SOURCES


LCGPackage_Add( Python URL http://service-spi.web.cern.ch/service-spi/external/tarFiles/Python-${Python_native_version}.tgz CONFIGURE_COMMAND <SOURCE_DIR>/configure --prefix=<INSTALL_DIR> --enable-unicode=ucs4 ${Python_config_options} CPPFLAGS=-I${sqlite_home}/include INSTALL_COMMAND ${MAKE} -j1 install COMMAND ${CMAKE_SOURCE_DIR}/externals/Python_postinstall.sh <INSTALL_DIR> DEPENDS ${Python_deps} sqlite)

• Modular fi le system based structure separated in blocks• Few lines sufficient to describe the build steps per

package• Dependencies with other packages explicitly included

28/09/15

LCGCMAKE PACKAGE BUILD DESCRIPTION


In big build systems cmake provides a full control of PKGs dependencies Build Configuration outputs contains these dependencies

28/09/15

DEPENDENCIES CONTROL


• Common installation tree in all builds for binaries<prefix>/<package_name>/<version>_<hash>/<platform_tag>/…

• hash Calculated from the list of package dependencies and the version

• Platform_tag combination of architecture, OS, compiler version and build type: x86_64-slc6-gcc49-opt…

• RPM installation in both AFS and CVMFS areas• Maintenance of all installed RPMs in AFS• Management of installed RPMs through a specific DB

• Package relocation end-system independent

28/09/15

AFS/CVMFS DEPLOYMENT


1. Build results reproducibility 2. Manageable handling of package dependencies 3. End-systems independent installation procedures;

reallocation4. Proper quality assurance5. Scalability of the infrastructure: increasing number

of compilers, architectures 6. Fast improvement/feedback cycle

28/09/15

BACK TO SLIDE 6 - CHALLENGES


RELEASE PROCEDURES

Experimental: (daily and on demand)

AFS

CVMFS

3 Nightly builds(Daily)

• Dev2: ROOT 6.02• Dev3: ROOT MASTER• Dev4: ROOT 6.04

Releases LCG_XX

• Set of testing and nightly chains defined to ensure the quality of the products provided in releases

• Nightly branches copied to AFS to ensure a fast access to experiments

• Use of these external packages on the experimental builds and releases


Cross-platform continuous integration and delivery open-source

application

• Read access to all CERN members with NICE account (SSO protected) and reachable outside CERN

• Test instance available to try new versions and plug-ins

• Characteristics of our system:• Ca. 300 cores available for nightly

and releases• 10 big projects including a total of

~100 diff erent jobs

• Replacement of the previous tool (Electric Commander) expected before the end of this year (2 weeks time)

28/09/15

BUILD ORCHESTRATION: JENKINS


Build results sent from Jenkins to cdash at the end of the build Enable the fast identification of errors and causes

28/09/15

BUILD VISUALIZATION: CDASH


LCG RELEASES PUBLICATION

Using the information provided at the end of the build configuration


1. Build results reproducibility 2. Manageable handling of package dependencies 3. End-systems independent installation procedures;

reallocation4. Proper quality assurance5. Scalability of the infrastructure: increasing number

of compilers, architectures 6. Fast improvement/feedback cycle

28/09/15

BACK TO SLIDE 6 - CHALLENGES


SUMMARY OF OUR BUILDS

Compilers OS Release type

BUILD CHAINS

gcc48

gcc49

gcc51

clang35

icc15

native

CentOS7

SLC6

mac1010

mac109

ubuntu14

Release Debug

Experimental

✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

dev2 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

dev3 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

dev4 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Releases ✔ ✔ ✔ ✔ ✔ ✔• Clang37 to be included soon• Already working in testing of gcc5.2.0


SUMMARY OF THE TOOLS

• Platform independent

• PKGs Dependences handling

• End-system installation independent

• Easy “configure-build-install” approach

• Escalation and automation scheduling procedures

• Easy access to build and test results

• Fast feedback of experiments

• Lcgcmake repository

• Facilitate specific branch procedures for nightly and releases

• Enable developers collaboration

CMAKE/LCGCMAKE

JENKINS CDASH GIT


• PH-SFT has established a robust build and test infrastructure to provide experiments with ca. 200 packages on a regular basis

• CMAKE based• Quality focused release procedures

• Experimental nightly (3) Releases chain

• Automation and escalation infrastructure:• Based on Jenkins• Test and production instances available

• Results visualization and access enabled for experiments• CDASH, Releases publication web• Installation in AFS and CVMFS

In a near future:• Consolidation of the group projects into Jenkins• Improvement of the Jenkins instance• Fostering of CVMFS usage including results of the nightly builds• Inclusion of new architectures and compilers• lcgcmake documentation

28/09/15

SUMMARY AND FUTURE PLANS


PH-SFT web page http://ph-dep-sft.web.cern.ch

HSF web page http://hep-software-foundation.web.cern.ch

GITLAB repository for LCGCMAKE https://gitlab.cern.ch/sft/lcgcmake.git

LCG Releases publication http://lcgsoft.cern.ch/lcgsoft

PH-SFT Jenkins instance https://phsft-jenkins.cern.ch

Cdash instance https://cdash.cern.ch

28/09/15

LIST OF POINTERS

b. hegner, p. mato, p. mendez cern, ph-sft group 1 st development@cern forum cern 28-sep-2015 the...

Documents

testing infrastructure

phsft group1st development

sftintroductionthe lcg

2p1 lcg

cern forum cern

releases project

lcg application pkgs

core elementstest