particle physics & astrophysics representing: mark freytag gunther haller ryan herbst michael...
TRANSCRIPT
Particle Physics & Astrophysics
Representing:Mark FreytagGunther HallerRyan HerbstMichael HufferChris O’GradyAmedeo PerazzoLeonid SapozhnikovEric Siskind
LCWS 2008 Data Acquisition LCWS 2008 Data Acquisition and Global Detector Networkand Global Detector Network
SiD ATCA DAQ System
Matt Weaver, [email protected] National Accelerator Laboratory
November 18, 2008
Talk based largely on Huffer, et al., “A new proposal for the construction of high speed,
massively parallel, ATCA based Data Acquisition Systems,” IEEE 2008 IEEE 2008 Nuclear Science Symposium.Nuclear Science Symposium.
Represents development work done at SLAC for the following projects:
PetaCache, LCLS, LSST
Particle Physics & Astrophysics
2
OutlineOutline
• Modular DAQ design to address new projects
• ATCA demonstration boards– Reconfigurable Cluster Element– Cluster Interconnect Module
• Example application to SiD
Particle Physics & Astrophysics
3
Abstract HEP Data Acquisition SystemAbstract HEP Data Acquisition System
– Analog shaping– Digitization– Transmission– Trigger pipeline buffering– Charge injection
Front-End Electronics
DataflowDetector
Back-End Systems
– High-Level Triggering– Event-Building– Archival– Data Quality Assurance
– Feature Extraction– Compression– Re-formatting– Event buffering– Aggregation– Reception & Transmission– Protocol conversion
emphasis is on this component
Particle Physics & Astrophysics
4
A A clustercluster of 12 elements of 12 elements
To back-end systems To back-end systems
ClusterInterconnect
Elements
From Front-End systems
switching fabric
switching fabric
Particle Physics & Astrophysics
5
Typical (5 slot) ATCA crateTypical (5 slot) ATCA crate
fans
“back-end” I/O
“front-end” I/O
“interconnect” board
Power supplies
“element” board Shelf
manager
Front
Rear
Particle Physics & Astrophysics
6
(Reconfigurable) Cluster Element (RCE) (Reconfigurable) Cluster Element (RCE)
• Bundled software:– bootstrap loader– Open Source kernel (RTEMS)
• POSIX compliant interfaces• standard I/P network stack
– exception handling support
MGTs
Core
DSP tilesCombinatoric logic
Resources
Processor450 MHZ PPC-405
512 MByte RLD-II
Boot Options Memory Subsystem Configuration
data
128 MByte Flash
Data Exchange Interface (DEI)
instruction
reset & bootstrap
options
• Class libraries (C++) provide:– DEI support– configuration interface
• Bundled software:– GNU cross-development
environment (C & C++)– remote (network) GDB
debugger– network console
Particle Physics & Astrophysics
7
ResourcesResources
• Multi-Gigabit Transceivers (MGTs)– up to 24 channels of:
• SER/DES• input/output buffering• clock recovery• 8b/10b encoder/decoder• 64b/66b encoder/decoder
– each channel can operate up to 6.5 gb/s– channels may be bound together for greater aggregate speed
• Combinatoric logic• gates• flip-flops (block RAM)• I/O pins
• DSP support– contains up 192 Multiple-Accumulate-Add (MAC) units
Particle Physics & Astrophysics
8
RCE board + RTM (Block diagram)RCE board + RTM (Block diagram)
MFD RCE
flashmemory
RCE
slice0
slice1
slice2
slice3
slice7
slice6
slice5
slice4
MFD
Fiber-optic transceiversZone2
Payload RTM
Zone3
Zone3
E0
E1
E0
E1
Basenetwork
Fabricnetwork
Particle Physics & Astrophysics
9
RCE board + RTMRCE board + RTM
Media Carrier with flash
Media Slice controller
RCE
Zone 1(power)
Zone 2
Zone 3
transceivers
RTM
Particle Physics & Astrophysics
10
Front-end Interface ExampleFront-end Interface Example
Transceiver
RegisterInterface
Clock/Reset generator
TransmitInterface
GT11CLK
MGT
Reference clock
PGPprotocol
coreMGT
GT11CLK
TriggerInterface
reset request
input[31:0]
address[23:0]
request
R/W
ACK
status
output[31:0]
context
opcode
trigger
enable
SOF
EOF
error
byte/word
data[15:0]
clock
reset
FEE logic Front-End Board
FPGA
POR
Particle Physics & Astrophysics
11
Cluster Interconnect board + RTM (Block diagram)Cluster Interconnect board + RTM (Block diagram)
MFD CI
Q2
P2
1-GE
10-GE XFP
XFP
Q0
Q1 Q3
XFP
XFP
XFP
XFP
XFP
XFP
XFP
Payload RTM
P3
P3
10-GE1-GE
10-GE XFP
10-GE
(fabric)
(base)
base
fabric
(fabric)
(base)
Particle Physics & Astrophysics
12
Cluster Interconnect board + RTMCluster Interconnect board + RTM
Zone 3
Zone 1
10 GE switch
XFP
1G Ethernet
MgmtProcessor
XFPRTM
Particle Physics & Astrophysics
13
SiD Data AcquisitionSiD Data Acquisition
• EMCal example
– ~100M channels– 4bytes/hit x 4hits/train (KPiX)– readout in 100ms (1/2 of inter-train spacing)– EMCal total readout data rate ~ 128Gbps
Particle Physics & Astrophysics
14
SiD DAQ continued…SiD DAQ continued…
to Cluster Element
FPGA
Protocol interface
One ATCA crate
EMCal(super?-) concentrators
FPGA
8 Gbytes/sec
500 Mbytes/sec
16 elements
8 RCE boards
2 CI boards
1.6 Gbytes/train
ClusterElement
ClusterInterconnect
Particle Physics & Astrophysics
15
StatusStatus
• SLAC is positioning itself for the production of a new generation of DAQ systems– strategy is based on the idea of modular building blocks
• inexpensive computational elements (the RCE)• interconnect mechanism (the CI)• industry standard packaging (ATCA)
• Architecture is now relatively mature– both demonstration boards (& their corresponding RTMs) are functional
• RTEMS ported & operating• network stack fully tested and functional
– performance and scaling have been demonstrated to meet expectations• Current focus is on applications using these concepts…
– LSST DAQ– LCLS DAQ– PetaCache project
• Application of same boards to SiD DAQ could yield a compact yet scalable system.
Particle Physics & Astrophysics
16