“How to Connect Campus Grids to the NGS”

Neil GeddesDirector, GOSC

The UK's National Grid Service is a project to deploy and operate a grid infrastructure for computing and data

access across the UK. This development will be a cornerstone of the development of the UK's "e-

Infrastructure" over the coming decade. The goals, current status and plans for the National Grid Service and the Operations Support Centre will be described.

http://www.ngs.ac.ukhttp://www.grid-support.ac.uk

Outline

• Overview of GOSC and NGS• Services & Getting Access• Joining the NGS

– How and why

• The Future– Roadmap for the future

• Summary

GOSC

The Grid Operations Support Centre is a distributed “virtual centre” providing deployment and operations support for the NGS and the wider UK e-Science programme.

- started October 2004

GOSC Roles

UK Grid Services National Services

Authentication, authorization, certificate management, VO management, security, network monitoring, help desk + support centre.

support@grid-support.ac.uk NGS Services

Job submission, simple registry, data transfer, data access and integration, resource brokering, monitoring and accounting, grid management services, workflow, notification, operations centre.

NGS core-node Services CPU, (meta-) data storage, key software

Services coordinated with others (eg OMII, NeSC, LCG, EGEE): Integration testing, compatibility & Validation Tests, User Management,

training

Administration: Security Policies and acceptable use conditions SLA’s, SLD’s Coordinate deployment and Operations

The National Grid Service:Towards the UK's

e-infrastructure

• The UK's National Grid Service is a project to deploy and operate a grid infrastructure for computing and data access across the UK. – Learn what it means– Learn how to do it

• This development will be a cornerstone of the development of the UK's "e-Infrastructure" over the coming decade

NGS “Today”

Projectse-Mineralse-MaterialsOrbital Dynamics of GalaxiesBioinformatics (using BLAST) GEODISE projectUKQCD Singlet meson projectCensus data analysis MIAKT projecte-HTPX project.RealityGrid (chemistry)

Users LeedsOxfordUCLCardiffSouthamptonImperialLiverpoolSheffieldCambridgeEdinburghQUBBBSRCCCLRC.Nottingham…

Interfaces

OGSI::LiteOGSI::Lite

If you need something else, please say!

Country providing resourcesCountry anticipating joining

In EGEE: 113 sites, 30 countries >10,000 cpu ~5 PB storage

Includes non-EGEE sites:• 9 countries• 18 sites

EGEE Resources: Feb 2005

GOSC Management Board - NGS StatusEGEE Third Conference, Athens, 19.04.2005 8

Enabling Grids for E-sciencE

INFSO-RI-508833

Applications

• HEP Applications• Biomed Applications

– imaging, drug discover– mri simulation– protein sequence analyis

• Generic Applications– Earth Observation,

Seismology, Hydrology, Climate, Geosciences

– Computational Chemistry– Astrophysics

• Applications “behind the corner” – R-DIG– BioDCV

0

2000

4000

6000

8000

10000

12000

6/24

/2004

7/1/

2004

7/8/

2004

7/15

/2004

7/22

/2004

7/29

/2004

8/5/

2004

8/12

/2004

8/19

/2004

8/26

/2004

9/2/

2004

9/9/

2004

9/16

/2004

9/23

/2004

9/30

/2004

10/7/

2004

10/14

/200

4

10/21

/200

4

10/28

/200

4

11/4/

2004

11/11

/200

4

11/18

/200

4

11/25

/200

4

12/2/

2004

12/9/

2004

12/16

/200

4

12/23

/200

4

12/30

/200

4

1/6/

2005

1/13

/2005

1/20

/2005

1/27

/2005

2/3/

2005

2/10

/2005

2/17

/2005

2/24

/2005

3/3/

2005

3/10

/2005

3/17

/2005

3/24

/2005

3/31

/2005

Jo

bs

/da

y

LCG/CondorGLCG/OriginalNorduGridGrid3

NGS core nodes:

Need UK e-Science certificate (1-2 days)

Apply through NGS web site (1-2 weeks)

http://www.ngs.ac.uk

Service and Access

What we provide and

how to get permission

NGS Core Services

• Globus Toolkit version 2• Job submission, File transfer, Shell

• Storage Resource Broker• Oracle (9i)• OGSA-DAI• Certificate Authority• Information Services (MDS/GIIS)• MyProxy server• Integration tests and database• Cluster monitoring• LCG-VO

In testing:• VOMS• EDG Resource Broker• Portal(s)

Gaining Access

NGS Partner Sites• Data nodes at RAL + Manchester• Compute nodes at Oxford + Leeds• Compute nodes at Cardiff +

Bristol• Free at point of use• Apply through NGS web site• Accept terms and conditions of

use• Light-weight peer review

– 1-2 weeks

• To do: project or VO-based application and registration

National HPC services

• Must apply separately to research councils

• Digital certificate and Conventional (username/ password) access supported

Affiliate Sites• Access approved projects/VO’s

• All access is through digital X.509 certificates • From UK e-Science CA or recognized peer

Joining the NGS

How, Whyand

The Vision Thing

How to JoinResource providers join the NGS by• Defining level of service commitments through

SLDs• Adopting NGS acceptable use and security

policies• Run compatible middleware

– as defined by NGS Minimum Software Stack– and verified by compliance test suite

• Support monitoring and accounting

Two levels of membership1. Affiliation

• a.k.a. connect to NGS2. Partnership

How to Join - 11. Ask

• Grid Support Centre, Stephen Pickles• Process currently limited only by available efforthttp://www.ngs.ac.uk

How to Join - 21. Ask

• Grid Support Centre, Stephen Pickles• Process currently limited only by available effort

2. Get assigned a buddy• added to rollout mailing list• read the joining guidehttp://www.ngs.ac.uk/man/documents/NGS_Partner_joining_procedure_0.4.pdf

How to Join - 31. Ask

• Grid Support Centre, Stephen Pickles• Process currently limited only by available effort

2. Get assigned a buddy• added to rollout mailing list• read the joining guide

3. Install the minimum software stack• On the resource, or on a gatewayhttp://www.ngs.ac.uk/man/documents/NGS_Minimum_software_stack_0.7.pdf

How to Join - 41. Ask

• Grid Support Centre, Stephen Pickles• Process currently limited only by available effort

2. Get assigned a buddy• added to rollout mailing list• read the joining guide

3. Install the minimum software stack• On the resource, or on a gateway

4. Pass monitoring tests for 7 days

How to Join - 51. Ask

• Grid Support Centre, Stephen Pickles• Process currently limited only by available effort

2. Get assigned a buddy• added to rollout mailing list• read the joining guide

3. Install the minimum software stack• On the resource, or on a gateway

4. Pass monitoring tests for 7 days5. Agree to security and acceptable use policies

• security contact + part of campus security• developing operational security with GridPP, EGEE, OSG

How to Join - 61. Ask

• Grid Support Centre, Stephen Pickles• Process currently limited only by available effort

2. Get assigned a buddy• added to rollout mailing list• read the joining guide

3. Install the minimum software stack• On the resource, or on a gateway

4. Pass monitoring tests for 7 days5. Agree to security and acceptable use policies6. Define a Service Level

How to Join - 71. Ask

• Grid Support Centre, Stephen Pickles• Process currently limited only by available effort

2. Get assigned a buddy• added to rollout mailing list• read the joining guide

3. Install the minimum software stack• On the resource, or on a gateway

4. Pass monitoring tests for 7 days5. Agree to security and acceptable use policies6. Define a Service Level7. Set up service level monitoring

How to Join - 81. Ask

• Grid Support Centre, Stephen Pickles• Process currently limited only by available effort

2. Get assigned a buddy• added to rollout mailing list• read the joining guide

3. Install the minimum software stack• On the resource, or on a gateway

4. Pass monitoring tests for 7 days5. Agree to security and acceptable use policies6. Define a Service Level7. Set up service level monitoring8. Approval by GOSC Board

– Get representation on Technical Board– Process took 6-12 months for core sites– Process took 3-6 months for Cardiff +Bristol– Process should take less than 3 months for Lancaster

Why the NGS

• Common tools, procedures and interfaces– Reduce total cost of ownership for providers– Lower threshold for users

• Early adopter system for UK research grids– technology evaluation (in production)– technology choices– pool expertise– drive interface standards and requirements

• both a voice and a target

– etc.

Why Join• (your) Users increasingly want resources as services and not as

complicated bits of kit– common interfaces across a range of facilities

• Funders of regional and national facilities want common interfaces to lower barriers to access– e.g. Hector

• By joining you leverage the national expertise in running these services– technical advice and support– security procedures and incident response– tools to help monitor and patch

• All of the above required in any TCO calculation– Get it at lower cost by joining the NGS

• Members get a say in the technical decisions …

The Future

More Visionand

Vision Meets Reality

UK e-Infrastructure

LHC

ISIS TS2

HPCx + HECtoR

Users get common access, tools, information, Nationally supported services, through NGS

Integratedinternationally

VRE, VLE, IE

Regional and Campus grids

Community Grids

Maintaining Compatibility

• Operating a production grid means valuing robustness and reliability over fashion.

– bug uncovered in OpenSSH in October 2004• NGS cares about:

– alignment/compatibility with leading international Grid efforts– special requirements of UK e-Science community– easy migration/upgrade paths– proven robustness/reliability– based on standards or standards-track specifications

• NGS cannot support everything• Everyone wants service-oriented grids

– but still settling out: WS-I, WS-I+, OGSI, WSRF, GT3, GT4, gLite• Caution over OGSI/WSRF has led to wide convergence on GT2 for

production grids and hence some inter-Grid compatibility– but there are potentially divergent forces at work

• Significant changes to NGS Minimum Software Stack will require approval by NGS Management Board on conservative time scales

Strategic Framework

• GOSC/NGS UK e-Science project– support other UK (e-)science projects

• International Compatibility – EGEE

• European infrastructure (and possible funding)• LHC at most UK universities

– only user group who want to build the grid– GridPP committed to common w/s plan in 2005

• GEANT

– Others• TeraGrid – US cyberinfrastructure $$$ (unlikely to pay us)• Open Science Grid – will develop compatibility with LCG• RoW e.g. China

– Want use other software, but must be EGEE compatible– Also driven by user requirements – Sets framework for relationship with OMII and others

• Other factors– JISC and Shibboleth

Process for Moving Forward

1. New developments evaluated by ETF– must have some longer term support likely

2. User requests treated on case by case basis3. NGS Technical Board consider against needs

• user demand• new functionality• improved functionality• improved security/performace/managability

4. Proposal brought to GOSC Board • Prepared by GOSC “executive” • N.Geddes, S.Pickles, A.Richards, S.Newhouse

Current Roadmap

• April 2005 Extensions of NGS membership

• July 2005 First assessment of NGS WS infrastructure Outline plans for 2005/2006 NGS AAA devt.Migration and/or interoperability of GT2/GT4

• Oct 2005 Deployment of Resource Brokering and VO mgmtClient (compute element) compatibility with EGEE Decision on support for GT4 infrastructure

• Jan 2006 Evaluation of Shibboleth interoperability solutions

• April 2006 Approval of Shibboleth interoperability plans

• July 2006 Interoperability with Shibboleth infrastructure

• Oct 2006 Technology refresh for core NGS nodes

The beginning …

– Backup slides to follow …

© 2004 Electronic Data Systems Corporation. All rights reserved.

Step 2: Consolidate (Server, Network, Storage, etc)Step 2: Consolidate (Server, Network, Storage, etc)

Step 5: Utility ServiceStep 5: Utility Service

Step 1: Migrate & Manage (Regional Facilities)Step 1: Migrate & Manage (Regional Facilities)

Agility Agility DriversDrivers

• Standards

• Visibility

• Quality

• Security

• Efficiency

Step 4: Virtual Service SuiteStep 4: Virtual Service Suite

Step 3: Automated Operations & Managed StorageStep 3: Automated Operations & Managed Storage

Step 6: GridStep 6: Grid

Reduce Risk

Improve Utilisation

Improve Scalability,

Service Quality/Levels, Productivity &

more

Reduce TCO

EDS is Transforming Clients to Agile Enterprise – Virtualised Computing PlatformEDS is Transforming Clients to Agile Enterprise – Virtualised Computing Platform

EDS Services Transition Roadmap

The GOSC BoardDirector, GOSC (Chair) Neil GeddesTechnical Director, GOSC Stephen PicklesCollaborating Institutions

CCLRC Prof. Ken PeachLeeds Prof. Peter DewOxford Prof. Paul JeffreysManchester Mr. Terry HewittEdinburgh/NeSC Prof. Malcolm AtkinsonUKERNA Dr. Bob Day London College tbd

ETF Chair Dr. Stephen NewhouseGridPP Project Leader Prof. Tony DoyleOMII Director Dr. Alistair DunlopEGEE UK+I Federation Leader Dr. Robin MiddletonHEC Liaison Mr. Hugh Pilcher-Clayton

Also invitede-Science User Board Chair. Prof. Jeremy Frey Director, e-Science Core Programme Dr. Anne Trefethen