software project status torre wenaus, bnl/cern us atlas software manager doe/nsf review of the us...

Software Project Status

Torre Wenaus, BNL/CERN

US ATLAS Software Manager

DOE/NSF Review of the US ATLAS Physics and Computing Project

January 15, 2003

Agency Review, January 15, 2003 Slide 2 Torre Wenaus, BNL/CERN

U.S. ATLAS Software Project Overview

Control framework and architecture Software Project Leader, principal development role, ATLAS

LCG applications area liaison Databases and data management

Database Leader, primary ATLAS expertise on ROOT/relational baseline

Software support for development and analysis Software librarian, quality, development tools, … Automated build/testing system adopted by Int’l ATLAS

Subsystem software roles complementing hardware responsibilities Muon system software coordinator, LAr DB coordinator

Scope commensurate with U.S. in ATLAS: ~20% of overall effort Commensurate representation on steering group

Strong role and participation in LCG common effort

Recent developments in green


U.S. ATLAS Software Organization

William WillisProject Manager

John HuthAssociate Project Manager,

Computing and Phy sics WBS 2

James ShankDeputy

External Advisory Group

Ian HinchliffeManager, Phy sics

WBS 2.1

Torre WenausManager, Sof tware

WBS 2.2

Bruce GibbardManager, Facilities

WBS 2.3

D. QuarrieControl/Framework

2.2.1.1,2.2.1.2

David MalonData Management

2.2.1.3

S. RajagopalanEv ent Model

2.2.1.4

J. ShankDetector Specif ic

2.2.2, 2.2.2.1

F. MerrittTraining

2.2.5L. Vacavant

Pixel/SCT2.2.2.2

F. LuehringTRT

2.2.2.3

S. RajagopalanLiquid Argon Calorimeter

2.2.2.4

T. LeCompteTilecal2.2.2.5

S. GoldfarbMuons2.2.2.6

S. GonzalezTrigger/DAQ

2.2.2.7

Subsy stems

Core Sof tware

R. BakerTier 1 Facility

R. GardnerDistributed ITInf rastructure

Facilities

TBNCollaborativ e

Tools2.2.3

R. BakerDeputy

T. WenausSof tware Support

Coordinator2.2.4

A. UndrusSof tware Librarian

2.2.4.1

Computing CoordinationBoard

Phy sics Manager, IB Conv ener, co-chairs


U.S. ATLAS - ATLAS Coordination

US roles in Int’l ATLAS software:US roles in Int’l ATLAS software:

D. Quarrie (LBNL), Chief Architect => D. Quarrie (LBNL), Chief Architect => Software Project LeaderSoftware Project Leader

D. Malon (ANL), Database CoordinatorD. Malon (ANL), Database Coordinator

P. Nevski (BNL), Geant3 Simulation P. Nevski (BNL), Geant3 Simulation Coordinator, Simulation production leadCoordinator, Simulation production lead

C. Tull (LBNL), EDG WP8 LiaisonC. Tull (LBNL), EDG WP8 Liaison

H. Ma (BNL), Raw Data CoordinatorH. Ma (BNL), Raw Data Coordinator

T. Wenaus (BNL), Planning OfficerT. Wenaus (BNL), Planning Officer

US International

See task matrix

William WillisU.S. ATLAS Project

Manager

Peter JenniATLAS

Spokesperson

John HuthAssociate PM

NormanMcCubbinSoftware

Coordiinator

I. HinchliffePhysics SM

F. GianottiPhysics

Coordinator

T. WenausSoftware SM

B. GibbardFacilities SM

D. QuarrieFramework

D. MalonDatabase

SubsystemSoftware

T. WenausPlanning Officer

D. QuarrieChief Archectect

D.MalonDatabase

A. PutzerNCB

L. PeriniATLAS GRID

R. GardnerDistributed Computing

SubsystemSoftware


ATLAS Subsystem/Task Matrix

Offline

Coordinator

Reconstruction Simulation Database

Chair N. McCubbin =>

D. Barberis

D. Rousseau A. Dell’Acqua D. Malon

Inner Detector D. Barberis D. Rousseau F. Luehring S. Bentvelsen /

D. Calvet

Liquid Argon J. Collot S. Rajagopalan M. Leltchouk H. Ma

Tile Calorimeter A. Solodkov F. Merritt V.Tsulaya T. LeCompte

Muon J.Shank J.F. Laporte A. Rimoldi S. Goldfarb

LVL 2 Trigger/

Trigger DAQ

S. George S. Tapprogge M. Weilers A. Amorim /

F. Touchard

Event Filter V. Vercesi F. Touchard

Computing Steering Group members/attendees: 4 of 19 Computing Steering Group members/attendees: 4 of 19 from US (Malon, Quarrie, Shank, Wenaus)from US (Malon, Quarrie, Shank, Wenaus)

(Likely to be reorganized)(Likely to be reorganized)

Physics Coordinator: F.Gianotti

Chief Architect: D.Quarrie


Project Planning Status

U.S./Int’l ATLAS WBS and schedule fully unified US/Int’l software planning currently covered by same person (TW)

Synergies with the US ATLAS Software Manager job outweigh the added burden of the ATLAS Planning Officer role

No ‘coordination layer’ between US and Int’l ATLAS planning Possible only because of how the ATLAS Planning Officer role is

currently (narrowly) scoped: schedule and WBS As pointed out by an ATLAS computing review in March, ATLAS

would benefit from a full FTE Planning Officer Reorganization expected soon under the new ATLAS computing

management Probably a new position combining Planning Officer with Release

Coordinator Will be important that the new person is effective and well

connected with the US – it would be strongly in the U.S. self-interest to have it be a U.S. person


ATLAS Computing Planning

US led a comprehensive review and update of ATLAS computing schedule in the spring

Milestone count increased by 50% to 600; many others updated Milestones and planning coordinated around DC schedule Reasonably comprehensive and detailed through 2002

New round underway now to flesh out 2003 schedule US core activity scheduling in reasonable shape

Long term milestones reworked to reflect LHC schedule, LCG Centered around escalating data challenges

Computing management reorganization should help planning Stronger decision making Project office (recommendation to new management) cf. recommendations to new management from computing working

group, Sep 2002 (on web)


Major Milestones

One DC per year until startup


Summary Major Milestones

0 1 2 3 4 5 6 71 Tbyte database prototype (Done)Release of Athena pre-alpha version (Done)Athena alpha release (Done)Geant3 digi data available (Done)Athena beta release (Done)Athena accepted (ARC concluded) (Done)Athena Lund release (Done)Event store architecture design document (Done)DC0 production release (Done)Decide on database product (Done)DC0 Completed - continuity test (Done)Full validation of Geant4 physics (Done)DC1 Completed DelayComputing TDR Finished (Align with LCG) DelayDC2 Completed (followed by annual DCs) DelayDC3 Completed (Exercise LCG-3) NewPhysics readiness report completed DelayDC4 Completed NewFull chain in real environment (DC5) Delay

Green: Done Gray: Original date Blue: Current date


Software Support, Quality Control

New releases are available in the US typically ~1-2 days after CERN Provided in AFS for use throughout the US

Librarian receives help requests and queries from ~25 people in the US US-developed nightly build facility used throughout ATLAS

Central tool in the day to day work of developers and the release process Recently expanded as framework for progressively integrating more quality

control and testing Testing at component, package and application level Code checking to be integrated

CERN support functions partially transferred to new ATLAS librarian BNL-based nightlies resumed

Much more stable build environment than CERN at the moment Use timely, robust nightlies to promote usage of the Tier 1 for development

System will probably be adopted by the LCG Applications Area


Software Support, Quality Control (2)

Testing integrated into automated builds Unit tests, package tests, integration/system tests

ATLAS (finally!) established a dedicated support team (SIT) for software infrastructure, testing, release management etc.

U.S. represented by the U.S. ATLAS librarian, an active team member

SIT needs a dedicated leader (currently the rotating, and overloaded, release manager heads SIT)

Provides a much needed context for U.S. support and QA efforts pacman (Boston U) for remote software installation

Adopted by grid projects for VDT, and a central tool in US grid testbed work


Grid Software

Development is being managed as an integral part of the software effort Grid software activities tightly integrated into ongoing core software

program, for maximal relevance and return Grid project programs consistent with this have been developed

And has been successful e.g. Distributed data manager tool (Magda) we developed was

adopted ATLAS-wide for data management in the DCs Grid goals, schedules integrated with ATLAS (particularly DC)

program However we do suffer some program distortion

e.g. priority tradeoffs between near term and long term needs affected by grid funding: emphasis on the latter sometimes at the expense of the former

e.g. basic persistency needs vs. distributed data management


FY03 Software FTEs by Category

5.25

0.80

4.401.50

1.00

3.27

1.10

0.33

0.25

Arch/Frame

Event

DB

Offl Infr

Prod

Dist SW

Support/QA

Analysis

Mgmt

Total 17.9. Total effort, all funding sources.


FY03 Core Software Personnel

Institute Project funded FTE Total FTE Other support Activity

Saul Youssef BU 0.00 1.00 GriVDGL grid

Steve Eckmann ANL 0.50 0.50 DB

Pending full time hire ANL 0.50 0.50 DB

David Malon ANL 1.00 1.00 DB

Sasha Vanyashin ANL 1.00 1.00 DB

Jerry Gieraltowski ANL 0.00 1.00 PPDG, base grid

David Adams BNL 0.50 1.00 PPDG event, grid

Wensheng Deng BNL 0.00 1.00 PPDG grid

Valeri Fine BNL 0.60 0.60 DB

Yuri Fisyak BNL 0.70 0.70 reco infra, prod

Hong Ma BNL 0.00 0.40 base event, DB

Pavel Nevski BNL 0.90 0.90 simu infra, prod

Victor Perevoztchikov BNL 0.30 0.30 DB

Srini Rajagopalan BNL 0.00 0.40 base event

Alex Undrus BNL 1.00 1.00 support

Torre Wenaus BNL 0.00 0.25 base mgmt

Paolo Calafiura LBNL 1.00 1.00 frame

Charles Leggett LBNL 1.00 1.00 frame

Massimo Marino LBNL 1.00 1.00 frame

David Quarrie LBNL 1.00 1.00 arch, frame

Craig Tull LBNL 0.00 0.25 base frame

Wim Lavrijsen LBNL 0.50 1.00 base frame

Joe Boudreau U Pitt 0.00 0.40 base geom model

Kaushik De UTA 0.00 0.70 base prod

Total 11.50 17.90

Green: new people

since last November


Software Activity in FY03

FTE breakdown by category and funding source

0.00

1.00

2.00

3.00

4.00

5.00

6.00

Arch/Frame Event DB Offl Infr Prod Dist SW Support/QA Analysis

GriVDGL

PPDG

Institute

Project


Effort Level Changes

ANL/Chicago – loss of .5 FTE in DB Ed Frank departure; no resources in FY03 to replace Another .5 FTE lost via departure; replacement soon with full-time hire

BNL – cancelled 1 FTE new hire in data management Insufficient funding in the project and the base program to sustain the bare-bones

plan Results in transfer of DB effort to grid (PPDG) effort – because the latter pays the

bills, even if it distorts our program towards lesser priorities LBNL – stable project-supported FTE count in architecture/framework

But loss of base support is threatening effort level and deliverables Grid funding being sought to ameliorate

DB effort hard-hit; somewhat ameliorated by common project Because the work is now in the context of a broad common project, US can still

sustain our major role in ATLAS DB A material example of common effort translating into savings (even if we wouldn’t

have chosen to structure the savings this way!)


Personnel Priorities for FY02, FY03

This is how we are doing, relative to goals… Sustain LBNL (4.5FTE) and ANL (3FTE) support

This we are doing so far. Add FY02, FY03 1FTE increments at BNL to reach 3FTEs in FY03

Failed in 02; BNL hire cancelled. Recover to 3 FTEs in FY03 Restore the .5FTE lost at UC to ANL

No resources in FY03. Expect to recover this in FY04. Establish sustained presence at CERN.

No resources, despite being a high priority We rely on labs to continue base program and other lab support to

sustain existing complement of developers And needed base program support is not there. Lab base programs

are being hammered…


Software Funding in FY02 and FY03

0

2

4

6

8

10

12

Project Institute PPDG GriVDGL

Univ

LBNL

BNL

ANL

0.0

2.0

4.0

6.0

8.0

10.0

12.0

Project Institute PPDG GriVDGL

Univ

LBNL

BNL

ANL

Funded FTEs

FY02

FY03

Drop in base support at labs in FY03 exceeds increase in project support


General and longer term priorities

These are reflected in the software request in the research program proposal (and go back as far as our original Jan 2000 project plan)

Priorities in order:1. Sustain existing ANL, BNL, LBNL efforts2. Complete the ramp of the lab based core developer FTEs to the long-

planned levels ANL 3.5 FTEs, LBNL 4.5 FTEs, BNL 4 FTEs

3. Establish, over and above these lab levels, presence at CERN of three core developer FTEs

In addition to lab people located at CERN Close local coupling to ATLAS users, LCG, Software Leader

4. Establish effort at the core-subsystem interface – sited mainly at universities and possibly CERN – to

support translation of core developments into established software employed by end users

better support US leadership roles with developer effort capable of translating decisions into established solutions


FY04 Software Project Costs

FTEs $ANL 3 666000 3 developersBNL 4 706568 3 developers, 1 librarianLBNL 4.5 1044000 4.5 developersTotal 11.5 2416568

ANL 0.5 111000 Restore ANL to 3.5 FTEsBNL 1 185432 Complete ramp to 4 developer FTEsTotal 12 2713000

Univ @ CERN 3 520000 Planned CERN complementTotal 15 3233000 of core developers (new hires)

Univ @ US 2.2 377000 US university based new hiresTotal 3610000 working at core/subsystem interface

Best guess for FY04 support is $3.61M 1 + 2 + 3 + 4 (2 FTEs)

Priority

1

2

3

(Dollar figures are estimates)

4


SW Funding Profile Comparisons

0.00

1.00

2.00

3.00

4.00

5.00

6.00

FY01 FY02 FY03 FY04 FY05 FY06 FY07 FY08

M$

Nov 2000 guidelineJan 2000 PMPNov 2001 guidelineMid 2002 'bare bonesJan 2003 GAP profile


Project-funded FTEs based on GAP profile

0

5

10

15

20

25

FY02 FY03 FY04 FY05 FY06 FY07 FY08

Univ @US

Univ @CERN

LBNL

BNL

ANL


US Software Project Effort

FTEs in principal activities by fiscal year and WBS category for project-supported people.

0

2

4

6

8

10

12

14

16

FY01 FY02` FY03 FY04 FY05 FY06

Analysis tools

Offline infra/prod

SW Support

DB/Data mgmt

Frame/Arch/Event

With conservative assumptions about project support (labs only)


Total US Core Software Effort

FTEs by fiscal year and WBS category. Total effort from all sources.

0

2

4

6

8

10

12

14

16

18

20

FY01 FY02 FY03 FY04 FY05 FY06

/us/sw/prod/ops

/us/sw/prod/tools

/us/sw/support/qa

/us/sw/support/help

/us/sw/support/librarian

/us/sw/simrec/general

/us/sw/core/distsw

/us/sw/core/analysis

/us/sw/core/db

/us/sw/core/event

/us/sw/core/frame

/us/sw/core/arch

With conservative assumptions about project support (labs only)


US ATLAS Software Manager (L2SM)

By my estimate a ~.3FTE role (a long-ago estimate which I think is accurate for someone ‘up to speed’ in the job)

My total ‘US time’ is 25%; I addressed the deficiency by Taking on a deputy, David Adams, for both L2SM and BNL PAS

group leader roles PAS deputy role is much more substantive, because the greatest

impact of my LCG job is on that role, not the L2SM Dropping PPDG liaison role and involvement in US grid Dropping technical work on Magda

But, the time I have is still not enough to the job fully I am all for turning over the job to someone else under the right

conditions, and a very good scenario looks close to being settled With other L2 reorganization taking place, grid aspects will no longer

be part of the job, which should make the time required significantly less than .3FTE


Concluding Remarks

No strategic changes; program is working, but stressed by funding

US has consolidated the leading roles in our targeted core software areas

Involved with new LCG common efforts in all our core areas

Architecture/framework effort level being sustained so far

And is delivering the baseline core software of ATLAS

Database/data mgmt effort reduced but so far preserving key technical expertise

Delivered current ATLAS baseline event store, and strong POOL participant

Cannot tolerate further reduction in a key strategic US core area

US major contributor to software infrastructure and QA in ATLAS

Recent emphasis: improve QA, and make the US development and production environment as effective as possible

Soft support from the project and base programs while the emphasis on grids grows: program distortion away from real priorities is a danger

software project status torre wenaus, bnl/cern us atlas software manager doe/nsf review of the us...

Documents

intl atlas software

atlas computing review

us atlas physics

intl atlas wbs

intl atlas planningpossible

us atlas software manager

primary atlas expertise

wenaus bnl