pc farms & central data recording
Post on 02-Jan-2016
33 Views
Preview:
DESCRIPTION
TRANSCRIPT
Eu
rop
ean
Lab
orat
ory
for
Par
ticl
e P
hys
ics
PC Farms & Central Data Recording
ATLAS Trigger/DAQ Workshop
Chamonix, October 20, 1998
Frédéric HEMMER - CERN/IT
CE
RN
- E
uro
pea
n L
abor
ator
y fo
r P
arti
cle
Ph
ysic
s
Http://wwwinfo.cern.ch/pdp/pc/EF/ATLAS-TDAQ-1098
ATLAS T/DAQ Workshop-Chamonix 20 Oct. 98 Frédéric Hemmer CERN/IT 2Eu
rop
ean
Lab
orat
ory
for
Par
ticl
e P
hys
ics
Overview
• NA48 Data Recording
• NA45 Data Recording in Objectivity
• NA57 Data Recording in HPSS
• Summary
ATLAS T/DAQ Workshop-Chamonix 20 Oct. 98 Frédéric Hemmer CERN/IT 3Eu
rop
ean
Lab
orat
ory
for
Par
ticl
e P
hys
ics
NA48 Central Data Recording
Cisco 5505
3Com
3900
FDDI
Fast Ethernet
Fast Ethernet
XLNT Gbit
FDDI
HiPPI
GigaRouter
3Com 9300Gigabit Ethernet
HiPPI
CS/22.5 TB Disk space
SUN E450500 GB Disk space
Event BuilderOnline PC Farm
Sub detectorVME crates
7 KM
OfflinePC Farm
ATLAS T/DAQ Workshop-Chamonix 20 Oct. 98 Frédéric Hemmer CERN/IT 4Eu
rop
ean
Lab
orat
ory
for
Par
ticl
e P
hys
ics
NA 48 Data Recording in 98• May September 1998• Raw Data on Tape
– 68 TB (1450 tapes, mainly 50 GB tapes)
– 12.5 TB Selected Reconstructed Data
– Total with 97 data : 96 TB
• Average Data Rate : 18 MB/s (peaks @ 23 MB/s)• CDR system can do 40-50 MB/s; limitation is CPU
Time available• Data recorded as files (4 million)
ATLAS T/DAQ Workshop-Chamonix 20 Oct. 98 Frédéric Hemmer CERN/IT 5Eu
rop
ean
Lab
orat
ory
for
Par
ticl
e P
hys
ics
NA48 On Line Farm
• 11 Subdetector PC’s (dual PII-266, 128 MB)
• 8 Event Building PC’s (dual PII-266, 128 MB, 18 GB SCSI)
• 4 CDR routing PC’s (dual PII-266, 64 MB, FDDI)
• All running Linux
• Software event building in the interburst gap
• Optional Software Filter (tags data)
• Send data to computer center (local disk buffers : 144 GB , 2 hours)
• On CS/2 : L3 Filtering and tape writing
ATLAS T/DAQ Workshop-Chamonix 20 Oct. 98 Frédéric Hemmer CERN/IT 6Eu
rop
ean
Lab
orat
ory
for
Par
ticl
e P
hys
ics
NA48 Plans for 1999
Fast Ethernet
Gigabit Ethernet
HiPPI
4 * SUN E4504.5 TB Disk space
EventBuilder
Sub detectorVME crates
7 KM
3Com
3900
HiPPI3Com 9300
Gigabit Ethernet
Fast Ethernet
Cisco 5505
On/OfflinePC Farm
ATLAS T/DAQ Workshop-Chamonix 20 Oct. 98 Frédéric Hemmer CERN/IT 7Eu
rop
ean
Lab
orat
ory
for
Par
ticl
e P
hys
ics
NA45 Data Recording
Fast Ethernet
Gigabit Ethernet
HiPPI
2 * SUN E450500 GB Disk space
Event BuilderOn Line PC Farm
Sub detector VME crates
7 KM
3Com
3900
HiPPI
Gigabit Ethernet
Fast Ethernet
SCI
3Com 3900
3Com 9300
NA48
PCSF
ATLAS T/DAQ Workshop-Chamonix 20 Oct. 98 Frédéric Hemmer CERN/IT 8Eu
rop
ean
Lab
orat
ory
for
Par
ticl
e P
hys
ics
NA45 Raw Data recording in Objectivity
• October 98 ; November 98• Estimated bandwidth : 15 MB/s• Processes translate Raw Data format to Objectivity• Database files (1.5 GB) are closed, then written on tape• Steering done using a set of perl scripts on the disk
servers• On line filtering/reconstruction/calibration possible• Farm is running Windows NT• Reconstruction can use PCSF
ATLAS T/DAQ Workshop-Chamonix 20 Oct. 98 Frédéric Hemmer CERN/IT 9Eu
rop
ean
Lab
orat
ory
for
Par
ticl
e P
hys
ics
PCSF Configuration (1)
• Server running NT 4.0 Server SP3– 1 dual capable Ppro @ 200 MHz, 96 MB, with 9 GB
data disk (with mirroring). LSF central queues.
• Server running NT Terminal Server Beta 2– 1 dual Ppro @ 200 MHz, 128 MB, with 4 GB data
disk. Runs IIS 3.0 and is accessible from outside CERN. It also host the asp’s for Web access
• Servers running NT 4.0 Workstation SP3– 9 dual Ppro’s @ 200 MHz, 64 MB, 2*4GB
– 25 dual PII’s @ 300 MHz, 128 MB, 2*4GB
All equipped with boot proms
ATLAS T/DAQ Workshop-Chamonix 20 Oct. 98 Frédéric Hemmer CERN/IT 10Eu
rop
ean
Lab
orat
ory
for
Par
ticl
e P
hys
ics
PCSF Configuration (2)
• Machines interconnected with 4 3com 3000 100BaseT switch
• Display/Keyboard/Mouse connected to a Raritan multiplexor
• PC Duo for remote admin access There were problems with other products• All running LSF 3.0.
LSF 3.2 does not work, support weak• Completely integrated with NICE
ATLAS T/DAQ Workshop-Chamonix 20 Oct. 98 Frédéric Hemmer CERN/IT 11Eu
rop
ean
Lab
orat
ory
for
Par
ticl
e P
hys
ics
Racking evolution
1997 1998
ATLAS T/DAQ Workshop-Chamonix 20 Oct. 98 Frédéric Hemmer CERN/IT 12Eu
rop
ean
Lab
orat
ory
for
Par
ticl
e P
hys
ics
HPSS test system
Barracuda-9, 96GB
SSA, 80GB
2 x 3590drives
STK powderhorn silos
1 x Redwood drive
NA57
~8 km
DEC alpha 4100
IBM RS6000/G40
IBM RS6000/F50
RZ29, 20GB
MotorolaMPPC2604running AIX 4.1
Barracuda-9, 18GB
IBM 3494 robot
FDDIEthernet
FDDI
ATLAS T/DAQ Workshop-Chamonix 20 Oct. 98 Frédéric Hemmer CERN/IT 13Eu
rop
ean
Lab
orat
ory
for
Par
ticl
e P
hys
ics
HPSS Tests results (NA57)• Disk to disk transfers without tape migration:
– sustained: 5 - 6MB/s (1GB files) 4-5MB/s (200MB files)– average: ~5.5MB/s– peak: 6.5MB/s
• With disk to tape migration:– sustained: 2 - 3MB/s (200MB files)– average: 3 - 5MB/s– peak: 5.5 - 6.5MB/s
• Disk @ DAQ limited to 7 MB/s read !
ATLAS T/DAQ Workshop-Chamonix 20 Oct. 98 Frédéric Hemmer CERN/IT 14Eu
rop
ean
Lab
orat
ory
for
Par
ticl
e P
hys
ics
Production system
RS6000/F50, 2CPU, 512MB
2 * DEC Alpha 41004 CPU, 512Mb
IBM 3494 robot
MPPC2604
Barracuda-9, 18GB
3900 switch
Gbit Ethernet
STK Powderhorn silos
Redwood
100BaseT
Com
pute
r ce
nter
3590
NA57
FDDIHiPPI
100BaseTGbit Ethernet
~120 GBmirrored
~120 GBmirrored
RS6000/F50, 2CPU, 256MB
ATLAS T/DAQ Workshop-Chamonix 20 Oct. 98 Frédéric Hemmer CERN/IT 15Eu
rop
ean
Lab
orat
ory
for
Par
ticl
e P
hys
ics
Current & Future Data ratesYear Experiments Bandwidth
MB/sRaw DataTB/year
ProcessingSPECInt95
1990-2000
LEP 0.5 1 100
1997-2000
SPS 15-20 30-70 500
2000-2008
SPS 35 300 2000
2004- LHC 100-1000 3000 50000
ATLAS T/DAQ Workshop-Chamonix 20 Oct. 98 Frédéric Hemmer CERN/IT 16Eu
rop
ean
Lab
orat
ory
for
Par
ticl
e P
hys
ics
Summary
• On line PC farms are being used to record data at sensible rates (Linux)
• Off line PC farms are being used for reconstruction/filtering/analysis (Linux/NT)
• New paradigms of recording data are explored (Objectivity/HPSS)
• Still a lot to do on scalable farm management, global steering, CDR monitoring, etc..
top related