2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
The DataFlow of the ATLAS Trigger and Data Acquisition
System
Giovanna LehmannOn Behalf of the ATLAS
Trigger/DAQ DataFlow Subsystem
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
Outline
ATLAS Interaction rates and event sizesThe Trigger/DAQ architecture
The DataFlowROS
• Design & Performance
LVL2 dataflow• Design & Performance
Event Builder• Design & Performance
Conclusions & Outlook
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
CERN Accelerators’ Complex
•Colliding particles: protons•Center of mass Energy: 14 TeV•Bunch crossing rate: 40 MHz•Interaction rate: 109 Hz•Event size: 1-2 Mbytes
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
DATAFLOW
EB
120 GB/s
H
L
T
L1
DET ROD
LVL2
Trigger DAQARCHITECTURE
2.5
s
~ 10 ms
Calo MuTrChOther detectors
SFI
SFO
EFN
FE Pipelines
ROIB
L2P
L2SV
L2N
Event FilterEFP
EFPEFP
EFP
RoI
RoI data = 2%
RoI requests
Lvl2 acc= ~2 kHz
~ sec
Lvl1 acc = 75 kHz
40 MHz
~3 GB/s
EFacc= ~0.2 kHz
EB req/clears
EBN
ROSROB
IOManager
data
DFM
RRM
40 MHz
75 kHz
~2 kHz
~ 200 Hz
120 GB/s
~ 300 MB/s
~3+3 GB/s
O(100) L2P
O(1000) EFP
1628 ROLs
O(100) ROS
O(100) SFI
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
The ROS
Receive & buffer event fragments from the 1628 detector ROLs Up to 160 MB/s per ROL
Send selected event fragments on requestROI requests : high rate, low data volume
• Rate: LVL1 rate (75 kHz), volume: ~2% of ROLs
EB requests : low rate, high data volume• Rate: ~3% of LVL1 rate (~2 kHz), volume:
complete event data
Provide fragment sampling for data monitoring
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
ROS High Level Components
Data requests
Event Fragments
Monitoring data
Control/Configuration
RODsRODsROS subsystemL2 & EBL2 & EB
Online SWOnline SW
I/O Manager(SW process)
RobIn(custom module)
Local Controller(SW process)
500x
150x
150x
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
Test Setup: ROS performance
ROS implemented on a 2 GHz PC, with 4 PCI busses (64 bit/66 MHz)
3 RobIn emulators on PCI On-board “local” bus limited to 266MB/s Each simulates 4 input channels 12 ROLs per
ROS
I/O to/from L2 & EB emulator Connected to the ROS through a GE switch Sends ROI/EB requests and clears to the ROS Receives data fragments back Uses TCP as communication protocol
(maximum possible overhead for message passing)
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
0
20
40
60
80
100
120
140
0 1 2 3 4 5 6 7
ROI Volume [% of full event]
Max
. LV
L1
Rat
e [k
Hz]
2 GHz PC; 3% EB requests
ROS Performance
ATLAS baseline conditions
(from paper model: contains safety factor 4 with respect to physics simulation )
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
The LVL2 Dataflow
Receive RoI information from LVL18 ROLs @ LVL1 rate (75 kHz)
Form a LVL1 result recordBuild 1 record out of 8 @ LVL1 rate
Retrieve RoI data from ROSs2 % of full Event (~30 kB)
Forward the LVL2 decision to the EB@ LVL2 accept rate (rejects are
grouped) Forward the LVL2 decision record to the
EB@ LVL2 accept rate (~2 kHz)
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
RoI req./data
Decisions
Control/Configuration
EBEBLVL2 subsystemLVL1LVL1
Online SWOnline SW
RoIBuilder(custom module)
L2SV(SW process)
L2PU(SW process)
pROS(SW process)
L2 recordROSROS
RoI information
1x 10x
200-500x
1x
LVL2 High Level Components
DC Controller(SW process)
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
Performance of RoI Builder, L2SV and pROS
Performance of RoI BuilderCustom built 12U VME prototype has
achieved required performance Performance of each LVL2 supervisor
Measured to be ~30 kHz on a 2.4 GHz dual CPU PC
Is insensitive to the number of L2PUs Performance of pROS
Not a demanding applicationRequirement to receive <10 kB at LVL2
accept rate (~3 kHz) and forward them to the EB is largely satisfied.
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
Test Setup: Performance of L2PU
ROS emulators used to send data over Gbit Ethernet.
RoI data collection takes always a small fraction of the time requested by the LVL2 event processing (~10 ms).
From a dataflow point of view << 100 L2PUs could sustain already the LVL1 rate.
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
The Event Builder
Receive LVL2 decisions @ LVL2 accept rate (~2 kHz; rejects are
grouped) Request data from ROS and pROS Build complete events
Depending on ROS implementation merge 140-1600 fragments into one.
~70 MB/s at every SFI Distribute clears to ROS and pROS
@ rate < LvL2 accept rate Forward complete events to EF
~70 MB/s at every SFI Provide fragment sampling for data monitoring
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
EB req./data
LVL2 Decisions
Control/Configuration
L2L2EB subsystempROSpROS
Online SWOnline SW
DFM(SW process)
SFI(SW process)
Clears
ROSROS
1x
50x
EB High Level Components
EFEF
Monitoring
DC Controller(SW process)
10x
Complete Event
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
Test Setup: EB Performance
DFM
16x
ROS em
... Switch
SFI
SFI
... 8x
ROS em
• SFI applications were run on 2.4 GHz dual CPU PCs
Many ROSs to many SFIs
ROS Emulators: ALTEON programmable GE NICs Raw ethernet communication protocol Simulating n sources
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
EB Performance
0
50
100
150
200
250
300
350
400
0 2 4 6 8 10
# of SFIs
EB
ra
te H
z
Limit of 16 ROS emulators for single frame messages
EB rate with 8 SFIs ~ 350Hz (17% of ATLAS EB rate)
8 ROLs/ROS
Flow Control
1 ROL/ROS
No Flow Control
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
Conclusions & Outlook
All elements of the DataFlow system have shown that they can satisfy the ATLAS requirements already with the present implementations and with today’s technology.
From now on emphasis will be put on the performance of the integrated DataFlow system.
Testbeds are being setup to measure its behaviour and the first results are encouraging.
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
Spares
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
0
50
100
150
200
250
300
350
400
0 2 4 6 8 10 12
ROI Volume [% of full event]
Max
. L
VL
1 R
ate
[kH
z]
EB rate =1% of LVL1 rate
EB rate = 2% of LVL1 rate
EB rate = 3% of LVL1 rate
EB rate = 4% of LVL1 rate
ATLAS baseline conditions (from paper model)
Results of Test 1 (no I/O to LVL2 & EB)
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
Test Setup 2: Scaling of LVL2 Network
From a dataflow point of view a few L2PUs sustain already a large fraction of the LVL1 rate.
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
Test Setup 1: Performance of the DFM
Tester:L2SV + n SFIs
DFM
LVL2 decision(group)
SFI EOE
DFM_Decision
Clears (group 300)
Tester Application emulates L2SV and many SFIs
DFM handling full I/O as for real ATLAS
DFM exposed to full input message rate from tester
DFM sending to non existing destinations (Connectionless protocol used)
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
DFM Performance
ATLAS event building rate
raw ethernet frames
udp
Test on a 2.2 GHz dual CPU PC:
Rate= function of CPU clock ->
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
Test Setup 1: SFI Performance
DFM 1 Gbit/s Ethernet
16x
ROS em
... EFSwitch
ROS em• SFI application was run on
a 2.4 GHz dual CPU PC
Many ROSs to 1 SFI
ROS Emulators: ALTEON programmable GE NICs Raw ethernet communication protocol Simulating n sources
SFI
2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California
Giovanna Lehmann - CERN EP/ATD
SFI Performance
0
10
20
30
40
50
60
0 2 4 6 8 10#ROLs/ROS
EB
ra
te H
z95 MB/s – IO limited
#ROLs/ROS
EB only
With output to EFCPU limited (2.4 GHz
CPU)
Reaching I/O limit at 95 MB/s otherwise CPU limited
35% performance gain with at least 8 ROLs/ROS
Will approach I/O limit for 1 ROL/ROS with faster CPU