bockjoo kim u of florida

1CMS Software Installation, Bockjoo Kim, 23 Oct. 2008, T3 Workshop, Fermilab

CMS Commissioning and First Data

Stan DurkinThe Ohio State Universityfor the CMS CollaborationBockjoo Kim

U of Florida

Bockjoo KimU of Florida

CMS Software Installation



Stan DurkinThe Ohio State Universityfor the CMS Collaboration

CMSSW Installation Sites on OSG

T3

T3

T3

T3

T3

T3

T3

T3

T3

T2

T2

T2

T2

T2

T2

T2

T2

T2

9 T39 T2

(Caltech)

(FIU)




CMSSW on US T3 Sites(as of 10/21/08)Sitename CE (OSG) CPU slots CMSSW

T3_US_Buffalo?? u2-grid.ccr.buffalo.edu/fork 2112 2_1_10, …

T3_US_Colorado heposg01.colorado.edu/? 16 2_1_10, …

T3_US_Cornell nys1.cac.cornell.edu/fork 120 2_1_9, …

T3_US_FIT uscms1.fltech-grid3.fit.edu/fork 160 2_1_10, …

T3_US_FIU ?/?

T3_US_FNALLPC ?/?

T3_US_Kansas ?/?

T3_US_Minnesota ?/?

T3_US_Olemiss?? umiss001.hep.olemiss.edu/fork 42 2_1_10, …

T3_US_Princeton ?/?

T3_US_Rutgers ?/?

T3_US_TTU antaeus.hpcc.ttu.edu/fork 252 2_1_10, …

T3_US_UCLA saxon.hosted.ats.ucla.edu/fork 44 2_1_10, …

T3_US_UCR top.ucr.edu/condor 40 2_1_10, …

T3_US_UIowa ?/?

T3_US_UMD ?/?

T3_US_Vanderbilt vampire.accre.vanderbilt.edu/fork

1408 2_1_10, …




Centralized CMS Installation

CMS requires CMSSW to be installed centrally for T2

For T3, one can install it locally or centrally

non-CMS OSG sites, it can be done centrally

All installations are uniform ($OSG_APP/cmssoft/cms/$SCRAM_ARCH/cms/cmssw/CMSSW_X_Y_Z)

Central Installation on OSG done by me

LCG/EGEE counter part (serveral people) is doing this on LCG/EGEE




CMS Software Life Cycle

1. Software Development

2. Release Build

3. APT Packaging

4. Tagging Release in XML for Publication

5. Release Announcement

6. Deployment on Grids/Local

7. Release Deprecation Pre-Announcement

8. Tagging for Release Deprecation Release

9. Release Deprecation Announcement

10. Remove Deprecated Release




Features of Software Deployment ToolFeatures of Software Deployment Tool

Condor-G Job Submission with Customized Installation/Verfication Scripts

Central Run and Bookkeeping DB

Possible Install via Grid Proxy Based Portal: Different DN Can Install Different Site Simul.

Cronized Installation In Parallel per Release

Production CMS Soft Release on OSG T2/T3

Twiki Pages : https://twiki.cern.ch/twiki/bin/view/CMS/CMSSoftDeployOSG

Portal : https://dev01.ihepa.ufl.edu:8443/csdogrid/csdogrid/




Consideration For InstallationConsideration For Installation

Different Linux Flavors

64-bit at 32-bit mode

Network Insulation (non-CMS sites)

Missing Tools on WN : apt-get, rpm, rpmbuild (non-CMS sites)

Different Shared File Systems (Lustre, AFS, etc)

Dedicated Slot : Needs to be done before any other CMS job starts

Disk Space Issue : ~50GB required typically Many files ==> Installation time gets longer

Keeping many releases ==> Deprecation becomes terrible and should be done on time




APT and File SystemsAPT and File Systems

APT (RPM tool) uses lots of locks

At least, four different FS accros OSG sites : NFS, AFS, GPFS, lustre

GPFS, lustre, other not well-known FS requires special treatment for locks using local FS ( not many exotic FS, though)

Many files ==> Installation time gets longer

Keeping many releases ==> Deprecation becomes very important ==> (64-bit OS/8GB required in the worst case)




DeprecationDeprecation

Mostly same as installation

Most time consuming part is find dependency and dependency bookkeeping

NFS stale file handle and GPFS : ‘rm -rf’

Recycle dependency calculation from one site

Deprecation is also automated and cronized




CMSCMS Software Packaging Tool Software Packaging Tool

CMS employs APT packaging for software distribution

CMS provides packaging tools for initial setup and update

RPMs are installed in a non-root area

CMS provides release publication : this allows deployment people for immediate deployment




AutomatedAutomated CMS CMS software Deploymentsoftware Deployment

Well-established OSG CMS sites needs software deployment promptly

CRON is used for automation

List of well-established OSG CMS sites

Database(DB) for bookkeeping

Scripts checks and executes: New release that needs to be deployed Deployment status from DB Deployment job submission/resubmission Installation job self-monitoring Email notification Repeatition of all if necessary




Implementation of the Deployment ToolImplementation of the Deployment Tool

Local Scripts

A Remote OSG Site

Application Area

Info. Publication

CMS Grid Users

OSG Software

GridCat

GridCat Client

CMS Pkg Tool

RPMs

CMS Pkg Tool RPMs CMS APT Repository

Tool Design

OSG CMS

X509 Web Portal

MySQL DB

Deploy Script

Condor-G Job

Execution Script

CRON Site Catalog

Site Availability

Site List

DB

Update

DB Check DB Check

Execution Script

Job Status Email




Grid Portal Screen ShotGrid Portal Screen Shot




SAM and SW Installation MonitoringSAM and SW Installation Monitoring

Related with SW InstallationRelated with SW Installation

SAM monitors SW InstallationsSAM monitors SW Installations




List of Problems and SolutionsList of Problems and Solutions

RPM version mismatch -> Rebuild RPM DB

rpm-wrapper error 88 -> insufficient disk rpm-wrapper error 92 -> permission problem“Could not get lock” -> FileSystem, use local disk

memory alloc (4byte)-> Remove releases

memory alloc (8byte) -> Use 64-bit apt-get




Statistics of CMS SW DeploymentStatistics of CMS SW Deployment

Installed

Releases

2006

0_4_1 TO 1_2_0

2007

1_2_0 TO 1_7_5

2008

1_6_10 TO 2_1_10

27 50 41

118

Sites OSG T2, T3, and Opportune

33 sites

Installations

(ReleasesxSites)

2006 2007 2008

126 294 297

717

Removals

(ReleasesxSites)

2006 2007 2008

15 325 684

1024

Years 2006 - 2008




Timing and FilesTiming and Files Site Tier Time to Install* FS No of Files Size (GB)

Caltech 2 54 min XFS 1.7 x 106 57

Florida 2 868 min NFS/lustre 1.3 x 106 48

MIT 2 66 min NFS 1.5 x 106 57

Nebraska 2 57 min NFS 1.5 x 106 54

Purdue 2 52 min NFS 1.6 x 106 96

SPRACE 2 100 min NFS 1.7 x 106 53

UCSD 2 98 min NFS 1.6 x 106 60

UERJ 2 51 min NFS 1.4 x 106 52

Wisconsin 2 71 min AFS 1.9 x 106 61

TTU 3 80 min lustre 1.3 x 106 45

UCR 3 56 min NFS? 1.5 x 106 54

Mississippi 3 186 min NFS? 1.6 x 106 55

UCLA 3 48 min NFS? 1.6 x 106 55

Vanderbilt 3 78 min GPFS 1.2 x 106 47

Buffalo 3 45 min NFS? 1.1 x 106 43

FIT 3 763 min NFS? 1.1 x 106 41

Cornell 3 - - - -

FIU 3 - - - -




SummarySummary

More than 717(1024) installations/removal for CMS software have been deployed on OSG, 2006-2008

Automated installation works quite efficiently with almost no problem theses days

Recently, most installation problem comes from rpm db limitation and can be fixed via 64-bit apt-get

T3 sites are not required to deploy CMSSW centrally. At the moment, there are 9 sites under the regular/central deploy list.

If other T3 sites wish to be included, please let me know

bockjoo kim u of florida

Documents