a scalable gigabit data acquisition system for calorimeters for linear collider
DESCRIPTION
A scalable gigabit data acquisition system for calorimeters for linear collider. Grant ANR-2010-0429-01. GASTALDI Franck. On behalf of the electronic & software team. Introduction: ILC detectors. Method: Imaging calorimetry ~100 10 6 channels/detectors Issues: Integration - PowerPoint PPT PresentationTRANSCRIPT
TIPP 2014 @ Amsterdam 1Monday, June 2 2014
A scalable gigabit data acquisition system for
calorimeters for linear colliderGASTALDI Franck
Grant ANR-2010-0429-01
On behalf of the electronic & software team
TIPP 2014 @ Amsterdam 2
Introduction: ILC detectors
• Method: Imaging calorimetry ~100 106 channels/detectors• Issues:
– Integration– Power consumption
• Ideas:– Detectors prototypes
• Power pulsing (1% duty cycle ~25µW/ch) allowed due to the beam structure (5 Hz spills)
– Switched on during > ~1ms of ILC bunch train and data acquisition– Bias currents shut down between bunch trains
– Data acquisition and control • A single cable for everything• Scalable architecture• Reliable protocols & simplicity
Monday, June 2 2014
TIPP 2014 @ Amsterdam 3
Introduction: ‘generic’ DAQ
• In most cases, detectors and associated readout systems are designed, tested and approved before DAQ effort is undertaken
• Our idea for this project is to design as a ‘generic’, scalable, and a self contained system, build around commercial components where possible.
• This DAQ is then configured towards multiple ‘use-cases’. ILC calorimetry might not be the only
customer
Remark : This work follows a R&D from Univ.College London, Manchester Univ and Cambridge Univ that continued at LLR-Ecole Polytechnique / IN2P3-CNRS
Monday, June 2 2014
TIPP 2014 @ Amsterdam 4
Calorimeter DAQ: overview
Monday, June 2 2014
Machineclock
DIFs SlabsDAQ2 PC
DCC(optiona
l)
Clock& Control
Digital (Config, Control, Data)Clock & Sync
GDCC
Network card
GDCC×7
⋮×8
Optic GigE or copperDebug USB
×n layers
:DCC
(optional)
⋮×8
Slabs = detector unit : detector with integrated front-end electronics and sensorsDIFs: Detector InterFace, servicing the detector unitGDCC: Giga-Data-Concentrator-Card: Concentrates data, fanin/fanout for clock and control dataCCC: Clock & Control card: Fanout of clock and fast controlsDCC: Data concentrator Card: optionnal extra level of data concentration
50 Mb/s
50 Mb/s
50 Mb/s
50 Mb/s
50 Mb/s
50 Mb/s
1 Gb/s
TIPP 2014 @ Amsterdam 5
Calorimeter DAQ: Serial Link (cont’d)
HDMI connectors between DIF-DCC-GDCC-CCC- Commercial standard for consumer electronics- High-bandwith connection at low cost
3 twisted pairs + 2 optional• Reference clock (50 MHz), fan-out from CCC• Data in (fast control, slow-control)• Data out (slow control, data readout)
Monday, June 2 2014
TIPP 2014 @ Amsterdam 6
DAQ: The DIF card• The DIF concept is generic in firmware, running on detector specific
hardware– Based on low cost FPGA– Compact (73mm x 50 mm)– Control up to 10K channels
• Functionalities are simple– VFE chip management (power pulsing, SC, DAQ) with a common
interface– Local storage of SC data (Flash Ram)
Monday, June 2 2014
Architecture of the DIF FPGA
TIPP 2014 @ Amsterdam 7
• Format : VME 6U (chassis with only J1 connector used for power distribution)• Format shared in 2 part (1/3 – 2/3)
1/3 is the mezzanine with the HDMI connectionsReliability of mezzanine by a specific Samtec connector (SEAM and SEAF series: 160 pins)Until 28 differential signals and 19 single ended
2/3 is the GDCC “heart” with the main functionalitiesBased around a Xilinx Spartan XC6SLX75 + Marvell component
• USB is used to an extra access to the GDCC (debug for example)
DAQ: The GDCC card
Monday, June 2 2014
7 x DIFs HDMI CCC HDMI
RJ45 & sfp fiber VME USB
Main part
Mezzanine part
TIPP 2014 @ Amsterdam 8
DAQ: the GDCC card (cont’d)Functionalities:• Aggregate data from many DIF links and send it to the PC over Gigabit Ethernet link
– The PHY layers is made by a specific component MARVELL88E1111• Signaling between the DIF and the GDCC is made by 5 differential LVDS pairs in HDMI cable • Extract packet from the PC and execute the command sent (R/W register, DIF configuration
packet, fast command)• Encapsulate data from DIF in Ethernet frame and send them to the PC
Monday, June 2 2014
Gemac
lithe
Homemade
(from xilinx reference design)
Totally free
Main
Interface
(based on several FSM
And few Xilinx reference design)
DIFsLinks
(Protocol fsmser-des8b/10b)
E
T
H
E
R
N
E
T
T
O
D
I
F
CCC interface
MARVELL
component
FPGA
mclk
trig
Single architecture of GDCC card
TIPP 2014 @ Amsterdam 9
DAQ: The GDCC card (cont’d)
Dst MAC Src MAC Ethernet Type
GDCC_type GDCC_modifier GDCC_pktID GDCC_dataLength GDCC_Data PAD CRC32
6 Bytes 6 Bytes 2 Bytes 2 Bytes 2 Bytes 2 Bytes 2 Bytes Variable Min size Eth 4 Bytes
Monday, June 2 2014
GDCC frame to the PC is based on standard Ethernet format
3 kinds of frameFast-command with a special Ethernet type 0x809 (GDCC DIF)Control data with a special Ethernet type 0x810 (GDCC DIF)Read-out data with the Ethernet type 0x811 (DIF GDCC)
GDCC Header Content of the DIF structure
Data
Example of sending data from GDCC to DIFData are sampled on rising-edge of clock
CLK
DIF SOF DIF EOF
Register packet from DIF
TIPP 2014 @ Amsterdam 10
DAQ: GDCC improvement• Put in place an UDP interface
– Simple and fast protocol– Easy to be implemented in hardware– Does not require big resources
Monday, June 2 2014
Header ETH Data
Header IP Data
Header UDP Data
@MACdest
@MACsrc
Linktype
Portsrc
Portdest Length Checksum
Version &length Service Frame
length ID Flag Offset Time oflife Protocole Checksum IP
srcIP
dest
RX
Reveive
Header_Filter
Frame_isValid
HeaderUDP (3)
HeaderIP (2)
HeaderETH (1)
REGs
Flags
GOOD FRM
BAD FRM
Send
BufferQueuePacket switch
FIFO9x1k
FIFO9x1k
DATA_OUT
Header generator
Header RX
Header TX
Checksumheader, frame
LengthIP, UDP
MEMORISE source adressIP, MAC, Port
TX
DATAs+
Nb count words
DATA_IN(UDP frame)
DATA_IN(ETH frame)
Ethernet frame structure with UDP Header
Architecture of UDP bloc
Currently under the first tests and after 3 days of sending a command to the DIF to read some registers (~ 3.106 times), there is no error.
TIPP 2014 @ Amsterdam 11
DAQ: The DCC card (optional)
• VME format• VME only used for the card
power supply• 1 HDMI connection for the
GDCC• Until 8 connections for the
DIFs• Identical data rate at the input
and output (50 Mb/s)Advantage:
This card can be connected or disconnected in DAQ chain without modification of behavior.
Monday, June 2 2014
Architecture of DCC
TIPP 2014 @ Amsterdam 12
Calicoes Software The Acquisition chain
• Ecal dedicated software suite• Based on the Pyrame
framework (LLR)– Based on XML language– Allow to prototype rapidly a on-line
system• Multi-media distribution(files,
sockets and shared memories) • Online event-building
Monday, June 2 2014
Acquisition chain: software architecture
TIPP 2014 @ Amsterdam 13
Calicoes Software The Control-Command
• Highly modular and distributed• Control the Ecal electronics
but also the peripheral devices (Power supply, pulse generator,…)
• Provides a high level state machine for final user
• Scripting language (Python)• Good stability
Monday, June 2 2014
Global control-command architecture
TIPP 2014 @ Amsterdam 14
The system: beam testThis DAQ has been used on the SiW-Ecal technical prototype for two years
It has been used successfully for 4 test beams at DESYTypical setup is : (~2.5K Channels)
10 layers of detection, 10 DIFs, 2 GDCC
Monday, June 2 2014
DAQ chassis SLAB structure
S/N > 14
250 GBytes of data have been generatedThis system has been validated for 10 Hz of spill frequency (ILC requirement is 5 Hz)
Exemple of event display1e- (5GeV)5 W plates between layers
TIPP 2014 @ Amsterdam 15
Conclusion• The aim who was to develop a DAQ system generic in nature, using commercial
components where possible has been in most part attained• The tests had shown the ability of the DAQ to take a lot of data (~250 GB)• During the last beam test, 120000 configurations have been injected in the system in
three weeks. It remained stable during all this time.• Currently, we improve our system with the implantation of a UDP block on GDCC.
Next stepConnect the DAQ to a real calorimeter system
– 16 ASICs per ASU (under test today), will be up to 160 ASICs per layer
Perspectives for ECAL(ILC)With this actual configuration and for 100M channels ECAL for example the setup will be:12500 DCC, 2000 GDCC and 200 PCFor reducing the number of card, the main work must be done on front end modules for easiness of integration
Monday, June 2 2014
TIPP 2014 @ Amsterdam 16Monday, June 2 2014
Thank youfor
your attention
TIPP 2014 @ Amsterdam 17
Back up
Monday, June 2 2014
TIPP 2014 @ Amsterdam 18
Time line
Monday, June 2 2014
Physics prototype
Technological prototype
ILD ?
Proof of concept
• Linearity• Resolution• Sensors• Very front-end
Feasibility of design options
• Compactness• Granularity• Front-end• Power pulsing• Long SLAB
Construction
• Integration• Environement• Services• Industrialization• Tooling• Project org.
S/N ~ 7.5
~24 X0, 20 cm thick~2500 m2 active detectors~100M readout channels
2004-200830 layers4000 channels
1500 channels/dm3 4000 channels/dm3 4000..10000 channels/dm3
S/N ~ 15
TIPP 2014 @ Amsterdam 19
Slab details
Monday, June 2 2014
ASUS with 16 Asics (180 x 180 mm) 1 Si Wafer with 256 pixels of 5X5 mm2 and thickness of 325 µm
Battery charger applicationAVX BestCap BZ01After regulator
360 mm
190 mm
180 mm70 m
m
Slab overview
TIPP 2014 @ Amsterdam 20
DIF cardSlow control and read-out• Sent from the DAQ/control PC as a raw Ethernet frame• Passed to/from the DIF via GDCC/DCC with the
following structure (protocol)
Monday, June 2 2014
Internally decoded frame (test pin)
DIF input:Standard packet
DIF output:Here: read out of 13x16b status registers (Reshaped into GDCC frame)
IDLE
SOF dataheader EOF
Exemple of a decoding frame at the DIF level
Exemple of a fast decodind command at the DIF level
TIPP 2014 @ Amsterdam 21
GDCC: some plots
Monday, June 2 2014
Example of data readout
Example of Result of eyes diagram and jitter on data readout
Example of readout packet spied by wireshark
Trigger = start spill
DIF SOF
RJ ~23 ps DJ ~166ps eye width ~19.75ns
0xfcff = start spill
0xfdff = start chipdata
Data from DIF
TIPP 2014 @ Amsterdam 22
CCC card
• Supplied by University of Cambridge in 2009
• Synchronize all sub-systems upon pre-spill warning
• Until 8 HDMI connection• Distribute asynchronous fast
trigger and/or busy signals• Capable to run stand-alone for
distribute clock (50 MHz) and spill from an external trigger
Monday, June 2 2014