tier-1 status andrew sansum gridpp18 21 march 2007

Tier-1 Status

Andrew SansumGRIDPP18

21 March 2007

Staff Changes

• Steve Traylen left in September• Three new Tier-1 staff

– Lex Holt (Fabric Team) – James Thorne (Fabric Team)– James Adams (Fabric Team)

• One EGEE funded post to operate a PPS (and work on integration with NGS):– Marian Klein

Team Organisation

Grid Services

Grid/Support RossConduracheHodgesKlein (PPS)Vacancy

Fabric(H/W and OS)

Bly (team leader)WheelerHoltThorneWhite (OS support)Adams (HW support)

CASTORSW/Robot

Corney (GL)Strong (Service Manager)Folkes (HW Manager)deWittJensenKrukKetleyBonnet2.5 FTE effort

Machine Room operations

Networking Support

Database Support (Brown)

Project Management (Sansum/Gordon/(Kelsey))

Hardware Deployment - CPU

• 64 Dual core/dual CPU Intel Woodcrest 5130 systems delivered in November (about 550 KSI2K)

• Completed acceptance tests over Christmas and into production mid January

• CPU farm capacity now (approximately):– 600 systems– 1250 cores– 1500 KSI2K

Hardware Deployment - Disk

• 2006 was a difficult year with deployment hold-ups:– March 2006 delivery: 21 servers, Areca RAID

controller – 24*400GB WD (RE2) drives. Available: January 2007

– November 2006 delivery: 47 servers, 3Ware RAID controller – 16*500GB WD (RE2). Accepted February 2007 (but still deploying to CASTOR)

– January 2007 delivery:39 servers, 3Ware RAID controller – 16*500GB WD (RE2). Accepted March 2007. Ready to deploy to CASTOR

Disk Deployment - Issues

• March 2006 (Clustervision) delivery:– Originally delivered with 400GB WD400YR drives– Many drive ejects under normal load test (had worked OK when we

tested in January).– Drive specification found to have changed – compatibility problems with

RAID controller (despite drive being listed as compatible)– Various firmware fixes tried – improvements but not fixed.– August 2006 WD offer to replace for 500YS drive– September 2006 – load test of new configuration begin to show

occasional (but unacceptably frequent) drive ejects (different problem).– Major diagnostic effort by Western Digital – Clustervision also trying

various fixes lots of theories – vibration, EM noise, protocol incompatability – various fixes tried (slow as failure rate quite low)..

– Fault hard to trace, various theories and fixes tried but eventually traced (early December) to faulty firmware.

– Firmware updated and load test shows problem fixed (mid Dec). Load test completes in early January and deployment begins.

Disk Deployment - Cause

• Western digital working at 2 sites – logic analysers on SATA interconnect.

• Eventually fault traced to a “missing return” in the firmware:– If drive head stays too long in one place it

repositions to allow lubricant to migrate.– Only shows up under certain work patterns– No return following reposition and 8 seconds

later controller ejects drive

Disk Deployment

#Servers

Capacity (TB)

2006 57 179

Jan 2007 21 190

Feb 2007 47 238

March 2007

39 197

Total 138 750

Hardware Deployment - Tape

• SL8500 tape robot upgraded to 10000 slots in August 2006.

• GRIDPP buy 3 additional T10K tape drives in February 2007 (now 6 drives owned by GRIDPP)

• Further purchase of 350TB tape media in February 2007.

• Total Tape capacity now 850-900TB (but not all immediately allocated – some to assist CASTOR migration – some needed for CASTOR operations.

Hardware Deployment - Network

• 10GB line from CERN available in August 2006• RAL was scheduled to attach to Thames Valley Network

(TVN) at 10GB by November 2006:– Change of plan in November – I/O rates from Tier-1 already

visible to UKERNA. Decide to connect T1 by 10Gb resilient connection direct into SJ5 core (planned mid Q1)

– Connection delayed but now scheduled for end of March

• GRIDPP load tests identify several issues at RAL firewall. These resolved but plan is now to bypass the firewall for SRM traffic from SJ5.

• A number of internal Tier-1 topology changes while we have enhanced LAN backbone to 10Gb in preparation to SJ5

RALSite

55105530

4 x 5530

RouterA

OPNRouter

3 x 5510+ 5530

6 x 5510+ 5530

ADSCaches

CPUs +Disks

CPUs +Disks

CPUs +Disks

CPUs +Disks

10Gb/s

10Gb/sto CERN

N x 1Gb/s

10Gb/s

5 x 5510+ 5530

2 x 5510+ 5530

RALTier 2

Tier 1

Oraclesystems

1Gb/s to SJ4

Tier-1 LAN

New Machine Room•Tender underway, planned completion: August 2008•800M**2 can accommodate 300 racks + 5 robots•2.3MW Power/Cooling capacity (some UPS)•Office accommodation for all E-Science staff•Combined Heat and Power Generation (CHP) on site•Not all for GRIDPP (but you get most)!

Tier-1 Capacity delivered to WLCG

(2006)

Asia Pacific4%

BNL18%

CERN11%

FNAL5%

FZK6% IN2P3

8%

INFN-T115%

PIC5%

RAL17%

SARA/NIKHEF10%

Others1%

Last 12 months CPU Occupancy

+260 KSI2KMay 2006

+550 KSI2KJanuary 2007

Recent CPU Occupancy (4 weeks)

Air-conditioning Work (300KSI2K offline)

CPU Efficiencies

CPU Efficiencies

CMS merge jobs – hang on CASTOR

ATLAS/LHCB jobs hanging on dCache

Babar jobs running slow – reason unknown

3D Service

• Used by ATLAS and LHCB to distribute conditions data by Oracle streams

• RAL one of 5 sites who deployed a production service during Phase I.

• Small SAN cluster – 4 nodes, 1 Fibre channel RAID array.

• RAL takes a leading role in the project.

Reliability

• Reliability matters to the experiments.– Use the SAM monitoring to identify priority areas – Also worrying about job loss rates

• Priority at RAL to improve reliability:– Fix the faults that degrade our SAM availability– New exception monitoring and automation system

based on Nagios

• Reliability is improving, but work feels like an endless treadmill. Fix one fault and find a new one.

Reliability - CE

• Split PBS server and CE long time ago• Split CE and local BDII• Site BDII times out on CE info provider

– CPU usage very high on CE info provider “starved”– Upgraded CE to 2 cores.

• Site BDII still times out on CE info provider – CE system disk I/O bound– Reduce load (changed backups etc)– Finally replaced system drive with faster model.

CE Load

Job Scheduling

• Sam Jobs failing to be scheduled by MAUI– SAM tests running under operations VO, but share gid with

dteam. dteam has used all resource – thus MAUI starts no more jobs

– Change scheduling and favour ops VO (Long term plan to split ops and dteam)

• PBS server hanging after communications problems – Job stuck in pending state jams whole batch system (no job

starts – site unavailable!)– Auto detect state of pending jobs and hold – remaining jobs

start and availability good– But now held jobs impact ETT and we receive less work from RB

– have to delete held jobs

Jobs de-queued at CE

• Jobs reach the CE and are successfully submitted to the scheduler but shortly afterwards CE decides to de-queue the job.– Only impacts SAM monitoring occasionally – May be impacting users more than SAM but

we cannot tell from our logs– Logged a GGUS ticket but no resolution

RB

• RB running very busy for extended periods during the summer:– Second RB (rb02) added early November but no

transparent way of advertising. Needs UIs to manually configure (see GRIDPP wiki).

• Jobs found to abort on rb01 linked to size of database– Database needed cleaning (was over 8GB)

• Job cancels may (but not reproducibly) break RB (RB may go 100% CPU bound) – no fix to this ticket.

RB Load

rb02 deployed

Drained to fix hardware

rb02 High CPU Load

Top Level BDII

• Top level BDII not reliably responding to queries– Query rate too high – UK sites failing SAM tests for extended periods

• Upgraded BDII to two servers on DNS round robin– Sites occasionally fail SAM test

• Upgraded BDII to 3 servers (last Friday)– Hope problem fixed – please report timeouts.

FTS

• Reasonably reliable service– Based on a single server– Monitoring and automation to watch for

problems

• At next upgrade (soon) move from single server to two pairs:– One pair will handle transfer agents– One pair will handle web front end.

dCache

• Problems with gridftp doors hanging– Partly helped by changes to network tuning– But still impacts SAM tests (and experiments).

Decide to move SAM CE replica-manager test from dCache to CASTOR (cynical manoeuvre to help SAM)

• Had hoped this month’s upgrade to version 1.7 would resolve problem– Didn’t help– Have now upgraded all gridftp doors to Java 1.5. No

hangs since upgrade last Thursday.

SAM Availability

RAL-LCG2 Availability/Reliability

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

May-06 Jun-06 Jul-06 Aug-06 Sep-06 Oct-06 Nov-06 Dec-06 Jan-07 Feb-07

Available

Old Reliability

New Reliability

Target

Average

Best 8

CASTOR• Autumn 2005/Winter 2005:

– Decide to migrate tape service to CASTOR– Decision that CASTOR will eventually replace dCache for disk pool management - CASTOR2

deployment starts• Spring/Summer 2006: Major effort to deploy and understand CASTOR

– Difficult to establish a stable pre-production service– Upgrades extremely difficult to make work – test service down for weeks at a time

following upgrade or patching.• September 2006:

– Originally planned we have full production service – Eventually – after heroic effort CASTOR team establish a pre-production service for CSA06

• October 2006– But we don’t have any disk – have to – BIG THANK YOU PPD!– CASTOR performs well in CSA06

• November/December work on CASTOR upgrade but eventually fail to upgrade• January 2007 declare CASTOR service as production quality• Feb/March 2007:

– Continue work with CMS as they expand range of tasks expected of CASTOR – significant load related operational issues identified (eg CMS merge jobs cause LSF meltdown).

– Start work with Atlas/LHCB and MINOS to migrate to CASTOR

CASTOR Layout

ralsrma ralsrmb ralsrmc ralsrmd ralsrme ralsrmf

D1T0

cmswanout

D0T1 prdD0T1 tmpD0T1 CMSwanin

cmsFarmRead lhcbD1T0 atlasD1T0prod

atlasD1T0usr atlasD1T1 atlasD0T1test atlasD1T0test

SRM 1

Disk

Pools

service classes

CMS

Phedex Rate to CASTOR (RAL Destination)

Phedex Rate to CASTOR RAL Source

SL4 and gLite

• Preparing to migrate some batch workers to SL4 for experiment testing.

• Some gLite testing (on SL3) already underway but becoming increasingly nervous about risks associated with late deployment of forthcoming SL4 gLite release

Grid Only

• Long standing milestone that Tier-1 will offer a “Grid Only” service by the end of August 2007.

• Discussed at January UB. Considerable discussion WRT what “Grid Only” means.

• Basic target confirmed by Tier-1 board but details still to be fixed WRT exactly what remains as needed.

Conclusions

• Last year was a tough year but we have eventually made good progress.– A lot of problems encountered– A lot accomplished

• This year focus will be on:– Establishing a stable CASTOR service that meets the

needs of the experiments– Deploying required releases of SL4/gLite– meeting (hopefully exceeding) availability targets– Hardware ramp up as we move towards GRIDPP3