tony doyle [email protected] gridpp2 – project specification, gridpp9 collaboration...

Tony [email protected]

“GridPP2 – Project Specification”, GridPP9 Collaboration Meeting, Edinburgh, 4 February 2004

Tony Doyle - University of Glasgow

OutlineOutline

• Background… A Year in

the Life

• ARDA first thoughts

• Patchwork Quilting

• PMB High Level View

• EGEE Grid Operations

• GridPP2 Deployment Board

• GridPP2 User Board

• Who reports where?

• ARDA second thoughts

• We are legion..

• Reporting: General Points

• Proposed Structure

• Where does the work get done?

• Areas on Green Alert

• Areas on Amber Alert• Outcome?


4. Background…4. Background…GridPP2 steps towards project spec. GridPP2 steps towards project spec.

0. PPARC Call (February 2003)1.GridPP2 Proposal (30/5/03)2. Feedback from Reviewers3. Responses from the PMB See http://www.gridpp.ac.uk/docs/gridpp2/

4. Projects Peer Review Panel (14-15/7/03)5. Grid Steering Committee (28-29/7/03)See http://www.gridpp.ac.uk/talks/GridPP_GSC_030728.ppt

6. Tender Document (11/11/03)See http://www.gridpp.ac.uk/cb/doc/GridPP2_Tender.doc7. Science Committee (17-18/11/03)8. PPARC (25/11/03)9. CB (4/12/03) 10.PRSC (16/1/04). PRSC summary11.PMB (26/1/04). Work area effort tables12. GridPP9 Collaboration Meeting (4-5/2/04)13.OC (18/2/04)… GridPP2 presentation. (unlucky for some..)

2003 2004 2005 2006 2007 2008

Middleware Deployment Operations

ResourceProvider

UserCommunity

EDG

LCG

LCG

EGEE

EGEE

HEP/BIOMED/…

Prototype-1EDG 1.x

(GT2, Condor-G)

Prototype-2LCG-1 (EDG-2)

Prototype-3LCG-2 Releases

Production-1LCG-3

Production-2 Production-3

Migrate to OGSA/F. Follow GGF Standards


3. Background…3. Background…A Year in the Life of GlobusA Year in the Life of Globus


2. Background… 2. Background… UK e-Science DevelopmentUK e-Science Development

1. The UK e-Science Production Grid will have significant overlap with the GridPP2 Production Grid – Mutual benefit to engagement– e.g. convergence talks organised prior to

Christmas– GT2-based approach

2. OMII developments will be based upon the Web Services Resource Framework– The OMII delivery timescales are in 2005 onwards– Incorporates e.g. OGSA-DAI – Limited overlap with our proposed programme


1. Background… 1. Background… LCG DevelopmentLCG Development

1. AliEn provided a pre-Grid implementation

2. ARDA provides a framework for PP application middleware – We can and should (and do)

participate here– Identify your development within

this structure


ARDA thoughts..ARDA thoughts.. Ian Stokes-Rees Ian Stokes-Rees

• “ARDA” is the LCG middleware – framework to deliver a distributed analysis environment to the experiments.

• We need to look at the decomposition and contribute coherently to this

• There are existing products which need to be used as building blocks here, e.g.– “GANGA”=Grid Job

Submission Wizard– “DIRAC”=Distributed MC

Production Service– “Spitfire” = Metadata Service– “R-GMA” = Information Service

http://www-pnp.physics.ox.ac.uk/~stokes/arda/ARDA-views-ijstokes-20040122.pdf


Patchwork Quilting Patchwork Quilting the GridPP2 HourGlassthe GridPP2 HourGlass

Tender Document Outcome

Tier-1 Tier-1

Applications Applications

Middle/Sec/NetMiddle/Sec/Net

Tier-2 Tier-2

+ 1 Portal+ 1 PhenoGrid

-0.5 BaBar

-1 LHC Deployment

+1 LHC Deployment-1 Network-0.5 Workload

-1 Data Management

+1 Data Management

III. Grid Middleware

e.g. Information Services

I. Experiment Layer

e.g. Portals

II. Application Middleware

e.g. Metadata

IV. Facilities and Fabrics

e.g. Storage Services


SysMan Experiments

Developers

Deployment

Board

User

Board

Dissemination

Team

Developer

Team

GridPP2 PMB GridPP2 PMB High Level ViewHigh Level View

Project Management

Board

(PMB)

Collaboration Board

(CB)

Production Grid Deployment

Board

(DB)

User Board

(UB)

Project Management

Board

(PMB)

Collaboration Board

(CB)


Board

(DB)

User Board

(UB)

“Opinion

Formers”

PMB


PMB

User

Board

Developer

Team

GridPP2GridPP2Organisational View? (~75 FTEs)Organisational View? (~75 FTEs)

Deployment

Board

Dissemination

Team


Grid and Regional Grid and Regional OperationsOperations

• Summary Equation:GOC > ROC (= ROC+CIC)– UK has a leading role.. – Grid Operations Centre

a.k.a. "International Rescue”

– But if its that exciting by the end of GridPP2, then we will have failed..

• Regional Operations Centre– Core Operational Tasks – Monitor infrastructure, components and

services– Troubleshooting– Verification of new sites joining Grid– Acceptance tests of new middleware

releases– Verify suppliers are meeting SLA– Performance tuning and optimisation– Publishing use figures and accounts– Grid information services – Monitoring services – Resource brokering – Allocation and scheduling services – Replica data catalogues – Authorisation services – Accounting services


EGEE Grid OperationsEGEE Grid Operations

EGEEGridPPJISCCore UK

Production Manager

Grid Support Centre5 FTE

Core Grid Coordinator1 FTE

MiddleWare Specialist Support6 FTE

Deployment Team8 FTE

Grid Operations Centre3 FTE

Security OfficerRAL

Data and Storage ManagementGlasgow, Bristol, Edinburgh

VO Management and ServicesNorth (0.5 FTE)

Workload Management ServicesLondon

Network ManagementLondon (0.5 FTE)

2 Tier1 DeploymentRAL

4 Tier 2 UK CoordinatorsLondonGrid,NorthGridScotGrid, SouthGrid

1 Tier2 CoordinatorIreland

Applications ExpertRAL

+1

+1

+

Technical Editor

~ 23 FTE (EU + non-EU funded)

The characters are the same whichever “OS”


SysMan

GridPP2 Deployment Board GridPP2 Deployment Board Foundation ViewpointFoundation Viewpoint

Production ManagerGrid Support Centre

5 FTECore Grid Coordinator

1 FTEMiddleWare Specialist Support

6 FTEDeployment Team

8 FTEGrid Operations Centre

3 FTE

• Summary:The Production Manager & DB have a

major role in (re-)defining and implementing Production Planning. Refs: GridPP-PMB-24-Production

GridPP-PMB-20-Management





Deployment

Board


SysMan

GridPP2 Deployment Board: GridPP2 Deployment Board: Two RolesTwo Roles

The role of Production Manager is to:1. lead the Production Grid Team;

2. sit on the Deployment Board;

3. ensure the formulation and implementation of procedures necessary to ensure timely deployment;

4. report to the Deployment Board on progress and issues;

5. liaise with all the Tier centres on the technical implementation of the production grid.

The role of the Deployment Board Chair is to:1. convene and chair Deployment Board meetings;

2. sit on the Project Management Board;

3. ensure that the Deployment Board are kept fully aware of the objectives of the project and the priorities identified by the PMB;

4. ensure that the Project Management Board is full aware of technical progress and any issues that arise;

5. ensure that progress toward deliverables is monitored against the project schedule on a quarterly basis;

6. ensure that the use of GridPP resources is monitored and reported;

7. report to the Project Manager on matters relating to any project deliverables and milestones.

Deployment

Board


SysMan

GridPP2 Deployment Board GridPP2 Deployment Board Terms of ReferenceTerms of Reference

• The Deployment Board is the body at which technical developments and problems wider than a single Working Group or Tier Centre are discussed and solved.

• The Deployment Board meets approximately quarterly to:1. determine and execute the technical project plan necessary to implement the strategy of

the Project Management Board on the deployment of a production quality grid for Particle Physics;

2. report to the Project Management Board, through its chair, on progress towards this goal;

3. ensure that problems and ideas with implications beyond individual working groups are discussed and solved in the most efficient manner;

4. provide a forum in which all constituent parts of the project can exchange information on technical issues of mutual interest;

5. manage the deployment of GridPP2 hardware resources in the UK, ensuring they are efficiently and effectively used to deliver the Particle Physics Grid with the minimum of duplication;

6. deal with matters of coordination and liaison between UK Tier1 and Tier2 centres;

7. ensure that technical developments elsewhere, particularly at CERN, are efficiently integrated into the project.

Deployment

Board


User

Board

Experiments

• The User Board ensures that the requirements for the major Particle Physics experiments that will use the Grid are properly taken into account in the strategy of the PMB.

• The User Board meets approximately quarterly to:1. ensure that the GridPP2 project remains science driven and that the

project develops to meet the needs of the UK Particle Physics community;

2. keep an overview of both technical and experimental software developments, ensuring these developments are compatible;

3. facilitate close working relationships between the UK experimental collaborations to identify common areas for the development of Grid-enabled experimental software;

4. overview the major hardware purchases within GridPP2 to ensure compatibility with the needs of the experimental collaborations and between collaborations;

5. promote the use of the developing Grid structures for a wide variety of computing needs in the whole of the UK Particle Physics community.

GridPP2 User Board Terms of ReferenceGridPP2 User Board Terms of Reference

I. Experiment Layer

e.g. Portals


e.g. Metadata


• The PMB is the central part of the management structure. • The Project Management Board meets approximately

weekly to:1. determine the detailed objectives of the project;

2. approve the allocation of resources to meet these objectives;

3. ensure timely progress is made towards meeting the objectives;

4. report on progress to the Oversight Committee;

5. inform the community of progress, formally via the Collaboration Board;

6. establish an identity for GridPP2 reflecting the set of GridPP deliverables;

7. ensure that GridPP2 is visible externally and is recognised

as a flagship project.

GridPP2 Project Management BoardGridPP2 Project Management Board Terms of Reference Terms of Reference



I. Experiment Layer

e.g. Portals


e.g. Metadata



PMB

UserBoard

DeploymentBoard

PMB


Who reports where?Who reports where?

Developers

Developer

Team

Dissemination

Officer“Opinion

Formers”

Project Management

Board

(PMB)

Collaboration Board

(CB)


Board

(DB)

User Board

(UB)

Project Management

Board

(PMB)

Collaboration Board

(CB)


Board

(DB)

User Board

(UB)

Representatives

• Formal roles are (pre-)defined in the proposal

• GridPP-PMB-20-Management


GAG

LCG

PEB

Alice ATLAS CMS LHCb

Generic Middleware

Resource providers

RequirementsGuidelines

Service specs

Integration: workers’ forum

ARDA – “A Realisation of ARDA – “A Realisation of Distributed Analysis” Distributed Analysis”

LCG AAprojects


ARDA: many facetsARDA: many facets


Application InterfacesApplication Interfaces [FTE] [FTE]

• Applications– 18.5 FTEs: ongoing

programme of work can continue

– Difficult to involve experiment activity not already engaged within GridPP

• Project needs to build on cross-experiment collaboration

•Encourage new joint developments across experiments

•New areas are PhenoGrid and generic portal work

GridPP2 Work Area FTE

ATLAS/LHCb (GANGA) 2.0

ATLAS 2.5

BaBar 2.0

CDF/D0 (SAM) 2.0

CDF 1.0

CMS 3.0

D0 1.0

LHCb 2.0

Portal 1.0

UKQCD 1.0

PhenoGrid 1.0

Total 18.5


Middleware, Security and Network Middleware, Security and Network [FTE][FTE]

• Information Services [4+4 FTE] and Networking [2+1.5+1.5 FTE]: strategic roles within EGEE• Security expands to meet reqts.• Storage and Workload Management continue• No further configuration management development

• programme defined by – mission criticality (experiment requirements

driven) – International/UK-wide lead – leverage of EGEE, UK core and LCG developments

• Redefine (EDG-based) UK Work Packages into Development Groups

• Metadata area requires group defined in association with applications area

• General emphasis on deployment/support rather than development per se

SecurityMiddleware

Networking

GridPP2 Work Area

PPARC funding Other funding

Metadata 1.0 0.0

Storage Management 2.0 0.0

Workload Management 1.0 4.0*

Security 3.5 2.0*

Information & Monitoring 4.0 4.0

Network Sector 2.0 3.0

LHC Applications 1.0 0.0

Totals 14.5 9.0 -13.0


Tier-2 Services [FTE]Tier-2 Services [FTE]

• Four Regional Tier-2 Centres • London: Brunel, Imperial College, QMUL, RHUL, UCL.• SouthGrid: Birmingham, Bristol, Cambridge, Oxford, RAL PPD.• NorthGrid: CCLRC Daresbury, Lancaster, Liverpool, Manchester, Sheffield.• ScotGrid: Durham, Edinburgh, Glasgow.

• Hardware provided by Institutes

• GridPP provides added manpower GridPP2 Work Area FTE

Security 1.0

Resource Broker 1.0

Network 0.5

Data Management 1.5

Storage Management 1.5

VO Management 0.5

Operations 8.5

Total 14.5 [+4.0]


Tier-1/A Services [FTE]Tier-1/A Services [FTE]

• High quality data services

• National and International Role• UK Focus for International Grid development

• Highest single priority within GridPP2

Total Manpower

CPU 2.0

Disk 1.5

Tape 2.5

Core Services 2.0

Operations 2.5

Networking 0.5

Deployment 2.0

Experiments 2.0

Management 1.5

Total 16.5


In LCG In LCG ContextContext

Proposed Proposed StructureStructure

ARDA

Ex

pm

tsEG

EE LCG

Deployment Board

Tier1/Tier2,Testbeds,

Rollout

Servicespecification& provision

User Board

Requirements

ApplicationDevelopment

Userfeedback

Metadata

Workload

Network

Security

Info. Mon.

PMB

CB

Storage


GridPP2 User Board: GridPP2 User Board: Two Roles??Two Roles??

The role of Applications Manager(?) is to:1. lead the Application Development Team;

2. sit on the User Board;

3. ensure the formulation and implementation of application middleware necessary to ensure timely deployment;

4. report to the User Board on progress and issues;

5. liaise with all the application experts on the technical implementation of the production grid.

The role of the User Board Chair is to:1. The role of the User Board Chair is to:

2. convene and chair User Board meetings;

3. sit on the Project Management Board;

4. inform the User board of issues from the PMB;

5. solicit input from users and potential users;

6. inform the PMB and DB of user requirements and issues;

7. ensure that progress towards deliverables in the application interface area is monitored on a quarterly basis;

8. report to the Project Manager on matters relating to any project deliverables and milestones defined within his or her purview.

User

Board

Experiments


GeneralGeneral Points Points

1. All post-holders report back at 3-monthly intervals (using similar reporting mechanisms to those used for the majority of GridPP posts). Reporting through either the:1. User Board; or

2. Deployment Board

2. All GridPP2 posts will end on Aug 31st 2007. Existing posts that are renewed will commence at the end-date of the present grant.

3. A review of posts will be made prior to this end point, in order to determine their effectiveness within the GridPP2 programme.


Where does the work get done? Where does the work get done? Areas on Green AlertAreas on Green Alert

• “Approved subject to minor/moderate changes”: timeline described by Steve…

• Groups bridge across the applications/middleware/support boundaries

• Active subgroups linking to EGEE:1. Information Services

– RAL• LCG developments (GridPP2/LCG/EGEE joint programmes?)

1. “Security” inc. VO management Group• Manchester, RAL [DK, AM]

2. “Workload Management” • IC [DC]

• Deployment groups1. “Data Replication Group” (support, small, but necessary)

• RAL, Glasgow, Bristol…2. Others..

• These can also be task-force like areas addressing specific

issues.. Workshop organised by RM


Areas on Amber AlertAreas on Amber Alert

• First discussions on “not yet approved” and “approved subject to major changes” areas

• We need to develop a detailed plan taking on board PRSC input not defined a priori 1. “Metadata Group”

• Glasgow, Oxford, IC/Bristol? ++? [TD]

2. “Storage Management Group”• RAL, Edinburgh, Bristol [JG]

3. “Networking Group”• UCL, DL, M’cr [PC]

• New deliverables in “1-2 months”…


Outcome?Outcome?

• We need to specify management/technical roles in advance of GridPP2 to ensure we are successful in developing/deploying the Production Grid

• By the end of this GridPP2 specification process we will be able to:

– Continue the work started in GridPP1 (by many people)– Define the GridPP2 Project Map tasks and metrics– Appoint additional people in recognised areas to contribute to agreed

tasks/roles– Contribute fully to EGEE/LCG– Start to go from Prototype to Production

• Anticipate new management/technical posts and holders are defined before GridPP10 meeting at CERN in June

• Beyond this, we anticipate a transition after GridPP2 in the PPARC Road Map where these areas are supported within the Particle Physics Programme

• We are masters of our own destiny..

tony doyle [email protected] gridpp2 – project specification, gridpp9 collaboration...

Documents

glasgow grid

tony doyle university

grid middleware

production grid team

storage services slide

fte grid operations

fte deployment team

structure slide