pc farms & central data recording

Eu

rop

ean

Lab

orat

ory

for

Par

ticl

e P

hys

ics

PC Farms & Central Data Recording

ATLAS Trigger/DAQ Workshop

Chamonix, October 20, 1998

Frédéric HEMMER - CERN/IT

CE

RN

- E

uro

pea

n L

abor

ator

y fo

r P

arti

cle

Ph

ysic

s

Http://wwwinfo.cern.ch/pdp/pc/EF/ATLAS-TDAQ-1098

ATLAS T/DAQ Workshop-Chamonix 20 Oct. 98 Frédéric Hemmer CERN/IT 2Eu

rop

ean

Lab

orat

ory

for

Par

ticl

e P

hys

ics

Overview

• NA48 Data Recording

• NA45 Data Recording in Objectivity

• NA57 Data Recording in HPSS

• Summary


rop

ean

Lab

orat

ory

for

Par

ticl

e P

hys

ics

NA48 Central Data Recording

Cisco 5505

3Com

3900

FDDI

Fast Ethernet

Fast Ethernet

XLNT Gbit

FDDI

HiPPI

GigaRouter

3Com 9300Gigabit Ethernet

HiPPI

CS/22.5 TB Disk space

SUN E450500 GB Disk space

Event BuilderOnline PC Farm

Sub detectorVME crates

7 KM

OfflinePC Farm


rop

ean

Lab

orat

ory

for

Par

ticl

e P

hys

ics

NA 48 Data Recording in 98• May September 1998• Raw Data on Tape

– 68 TB (1450 tapes, mainly 50 GB tapes)

– 12.5 TB Selected Reconstructed Data

– Total with 97 data : 96 TB

• Average Data Rate : 18 MB/s (peaks @ 23 MB/s)• CDR system can do 40-50 MB/s; limitation is CPU

Time available• Data recorded as files (4 million)


rop

ean

Lab

orat

ory

for

Par

ticl

e P

hys

ics

NA48 On Line Farm

• 11 Subdetector PC’s (dual PII-266, 128 MB)

• 8 Event Building PC’s (dual PII-266, 128 MB, 18 GB SCSI)

• 4 CDR routing PC’s (dual PII-266, 64 MB, FDDI)

• All running Linux

• Software event building in the interburst gap

• Optional Software Filter (tags data)

• Send data to computer center (local disk buffers : 144 GB , 2 hours)

• On CS/2 : L3 Filtering and tape writing


rop

ean

Lab

orat

ory

for

Par

ticl

e P

hys

ics

NA48 Plans for 1999

Fast Ethernet

Gigabit Ethernet

HiPPI

4 * SUN E4504.5 TB Disk space

EventBuilder

Sub detectorVME crates

7 KM

3Com

3900

HiPPI3Com 9300

Gigabit Ethernet

Fast Ethernet

Cisco 5505

On/OfflinePC Farm


rop

ean

Lab

orat

ory

for

Par

ticl

e P

hys

ics

NA45 Data Recording

Fast Ethernet

Gigabit Ethernet

HiPPI

2 * SUN E450500 GB Disk space

Event BuilderOn Line PC Farm

Sub detector VME crates

7 KM

3Com

3900

HiPPI

Gigabit Ethernet

Fast Ethernet

SCI

3Com 3900

3Com 9300

NA48

PCSF


rop

ean

Lab

orat

ory

for

Par

ticl

e P

hys

ics

NA45 Raw Data recording in Objectivity

• October 98 ; November 98• Estimated bandwidth : 15 MB/s• Processes translate Raw Data format to Objectivity• Database files (1.5 GB) are closed, then written on tape• Steering done using a set of perl scripts on the disk

servers• On line filtering/reconstruction/calibration possible• Farm is running Windows NT• Reconstruction can use PCSF


rop

ean

Lab

orat

ory

for

Par

ticl

e P

hys

ics

PCSF Configuration (1)

• Server running NT 4.0 Server SP3– 1 dual capable Ppro @ 200 MHz, 96 MB, with 9 GB

data disk (with mirroring). LSF central queues.

• Server running NT Terminal Server Beta 2– 1 dual Ppro @ 200 MHz, 128 MB, with 4 GB data

disk. Runs IIS 3.0 and is accessible from outside CERN. It also host the asp’s for Web access

• Servers running NT 4.0 Workstation SP3– 9 dual Ppro’s @ 200 MHz, 64 MB, 2*4GB

– 25 dual PII’s @ 300 MHz, 128 MB, 2*4GB

All equipped with boot proms


rop

ean

Lab

orat

ory

for

Par

ticl

e P

hys

ics

PCSF Configuration (2)

• Machines interconnected with 4 3com 3000 100BaseT switch

• Display/Keyboard/Mouse connected to a Raritan multiplexor

• PC Duo for remote admin access There were problems with other products• All running LSF 3.0.

LSF 3.2 does not work, support weak• Completely integrated with NICE


rop

ean

Lab

orat

ory

for

Par

ticl

e P

hys

ics

Racking evolution

1997 1998


rop

ean

Lab

orat

ory

for

Par

ticl

e P

hys

ics

HPSS test system

Barracuda-9, 96GB

SSA, 80GB

2 x 3590drives

STK powderhorn silos

1 x Redwood drive

NA57

~8 km

DEC alpha 4100

IBM RS6000/G40

IBM RS6000/F50

RZ29, 20GB

MotorolaMPPC2604running AIX 4.1

Barracuda-9, 18GB

IBM 3494 robot

FDDIEthernet

FDDI


rop

ean

Lab

orat

ory

for

Par

ticl

e P

hys

ics

HPSS Tests results (NA57)• Disk to disk transfers without tape migration:

– sustained: 5 - 6MB/s (1GB files) 4-5MB/s (200MB files)– average: ~5.5MB/s– peak: 6.5MB/s

• With disk to tape migration:– sustained: 2 - 3MB/s (200MB files)– average: 3 - 5MB/s– peak: 5.5 - 6.5MB/s

• Disk @ DAQ limited to 7 MB/s read !


rop

ean

Lab

orat

ory

for

Par

ticl

e P

hys

ics

Production system

RS6000/F50, 2CPU, 512MB

2 * DEC Alpha 41004 CPU, 512Mb

IBM 3494 robot

MPPC2604

Barracuda-9, 18GB

3900 switch

Gbit Ethernet

STK Powderhorn silos

Redwood

100BaseT

Com

pute

r ce

nter

3590

NA57

FDDIHiPPI

100BaseTGbit Ethernet

~120 GBmirrored

~120 GBmirrored

RS6000/F50, 2CPU, 256MB


rop

ean

Lab

orat

ory

for

Par

ticl

e P

hys

ics

Current & Future Data ratesYear Experiments Bandwidth

MB/sRaw DataTB/year

ProcessingSPECInt95

1990-2000

LEP 0.5 1 100

1997-2000

SPS 15-20 30-70 500

2000-2008

SPS 35 300 2000

2004- LHC 100-1000 3000 50000


rop

ean

Lab

orat

ory

for

Par

ticl

e P

hys

ics

Summary

• On line PC farms are being used to record data at sensible rates (Linux)

• Off line PC farms are being used for reconstruction/filtering/analysis (Linux/NT)

• New paradigms of recording data are explored (Objectivity/HPSS)

• Still a lot to do on scalable farm management, global steering, CDR monitoring, etc..

pc farms & central data recording

Documents