yannick patois – cvs and autobuild tools at ccin2p3 – hepix - october, 24 2002 - n° 1 cvs setup...

Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1

CVS setup at CC-IN2P3 and Datagrid edg-build tools

CVS management, centralized code checking and automatic building of rpm packages from CVS

Yannick Patois

E-mail [email protected]


Table Of Content

CVS service at CC-IN2P3 (Loïc Tortay)

How it started

Authorized user access in read/write setup

Anonymous access setup

CVSweb

Datagrid CVS tools (edg-build)

General idea

Code checking

RPM building

Package publishing

RPM distribution management


CVS service at CC-IN2P3 - starting point

To share code development among local users:

CVS repository on AFS

But a project (Datagrid) needed a CVS repository open to more people

Lab policy wouldn't allow local login for everyone

Needed another solution

Once created and tested for Datagrid, this CVS service was extender to other experiments: cvs.in2p3.fr was created.

The presented setup has been set in place by Loïc Tortay at CC-IN2P3.

CVS.IN2P3.FR

Dedicated authentication mechanism (not part of the generic users account policy)

Offer open CVS services to experiements.


CVS at CC-IN2P3 - Read/Write access setup

Authentication by ssh keys

Interactive login is disabled

CVS operations are performed through a specific pseudo-shell that only allow CVS operations to be executed.

rcvssh : remote CVS shell

Build from 'anoncvssh' (OpenBSD project)

Authentication managed above (by sshd)

The shell ensure that the command and arguments are in a predefined set of allowed commands (cvs commands in our case)

Can chroot the user to a subtree (anonymous access)

Verbose login facility


CVS setup at CC-IN2P3 - Anonymous access

Anonymous access is done on a chrooted cvs server

Use a different port than the authenticated access mechanism

The dedicated ssh daemon uses specifics configurations parameters (allow password authentication, allow empty password, only one account allowed)

Picture taken from http://www.winternet.com/~mikelr/ with kind courtesy of Mike Reed.


CVS setup at CC-IN2P3 - CVSweb and other tools

CVSweb setup

Module based configuration

Web access aliased cvs.in2p3.fr/experiment

Apache access rights can be managed by experiment

CVS commit info on mailing list


CVS Services at CCIN2P3

cvs@in2p3 Service just started a few weeks ago

A few experiments (EROS, SUPERNOVAE, ..) uses it

Datagrid CVS repository

Around 120 accounts

360Mb of sources

A few hundreds connexions per day (more than one thousand if accounting for automatic connexion -like autobuild)


Autobuild tools: The Datagrid collaboration

European grid project

A Grid for HEP Biomedical applications (genomic

and imaging) Earth study (ESA)

Constraint

Distributed development

Several dependent modules (around 30)


The Datagrid Software Repository The Datagrid Software

Repository

Hosted at CC-IN2P3 (Lyon)CNRS - France

Main services: Source code management for

developers (CVS) Packages distribution (RPMS) The autobuild an check system

(automatic building of packages) Packages distribution

Management

http://datagrid.in2p3.fr/autobuild/rh6.2/http://datagrid.in2p3.fr/autobuild/rh6.2/


Autobuild system - generalities

Autobuild

Around 3000 SLOC of python

One external Java program called

A lot of os.system() calls to various unix utilities (from mkdir to cvs)

Context

Run on RH6.2 and RH7.2, have been tested on Solaris

CVS and package repository machine is remote (btw it's an AIX machine).

Publication Machine IBM-AIX

CVS repositoryPackages (rpm) repository

Publication Machine IBM-AIX

CVS repositoryPackages (rpm) repository

Build Machine RedHat Linux

Building, package managing

Build Machine RedHat Linux

Building, package managing CVS checkoutScp


Autobuild system : what it does

Statistical informations about the project

RPMS number/size

SLOCs for every edg code

Auto-documentation pages

Processed user doc (LaTeX files, ...)

Autogenerated code doc (Doxygen...)

Build platform informations

Installed rpms, environments...

CVS module processing for each module

Conformance check README and INSTALL files CVS tags

Build autotools make make doc make check make install make dist make rpm

Publish build RPMs


Autobuild : result sample


Edg-release rpm lists


Autobuild present and futur state

Autobuild program (called edg-build) can be downloaded from http://datagrid.in2p3.fr

The code is most likely still very specific to Datagrid own needs, but is improving.

Current evolution plan

Build on demand

Email to developers on build failure

Package managing capability (partially implemented)


Creating RPM lists from dependencies I (idea)

Problem : Managing huge lists of RPM (edg-release) can be difficult.

Idea : Creating them from the RPM inter-dependenciesPractical implementation

Datagrid RPMS Striped RPMS (only headers)

Custom “everything” RPM database

Build Machine

Local copyLocal copy

RPM task(virtual rpm pkgs)

RPM task(virtual rpm pkgs)

Full RPM listFull RPM list

--whatprovide--whatrequire

Customlist RPM database

Rpm -Va

ValidationValidation


Creating RPM List from Dependencies II (example)

Redhat-base-edg-0.0.1-1.i386.rpmredhat-kernel-edg-0.0.1-1.i386.rpm

task-build-wp6-0.0.1-1.i386.rpm

Redhat-base-edg-0.0.1-1.i386.rpmredhat-kernel-edg-0.0.1-1.i386.rpm

task-build-wp6-0.0.1-1.i386.rpm

Name: task-build-wp6[...]

# This is for edg-exampleRequires: libtool

Requires: automakeRequires: autoconf

Requires: makeRequires: c++

Requires: doxygenRequires: sgml-toolsRequires: tetex-dvipsRequires: ghostscript

[...]Full RPM ListFull RPM List

yannick patois – cvs and autobuild tools at ccin2p3 – hepix - october, 24 2002 - n° 1 cvs setup...

Documents