dune daq meeting felix+dpm - indico.fnal.gov · tech. proposal assumes 4 x dpm per apa. aim for 1 x...
TRANSCRIPT
DUNE DAQ Meeting
Felix+DPM
David Cussans30th July 2018
Introduction• Investigate ”FPGA-centric” approach for DUNE DAQ
• (TP/IDR baseline. Alternative is CPU/GPU centric )
• Provide information for TDR
• Need result Q3/Q4 2018
• Show plausibility of PCIe based ”FPGA-centric” approach.
• Filtering and hit-finding for SP-TPC done in processing FPGA
• Most firmware blocks agnostic to location of this FPGA ( in separate crate / incorporated with Felix )
• Technical Proposal (IDR) baseline solution has 10-second buffer in CPU memory. Test idea of buffer connected to FPGA.
• Solution must be suitable for all sub-systems: SP-TPC, DP-TPC, DP-PDS, SP-PDS
• Optional daughter-board? Different Felix versions?
David Cussans | DAQ meeting, 30th July 182
Generic Hardware Layout• 10 second buffer in DRAM• Collect enough data to decide if SN• Look back before trigger fires
• Solid-state storage for > 100s of data if SN trigger• Tech. proposal assumes 4 x DPM per APA. Aim for 1 x DPM per APA• Initial firmware results show compression , filtering can be fitted into less FPGA resource
ToBack-end
DataSelection
Dia
gram
–Ba
bak
Abi
TriggerCommand
David Cussans | DAQ meeting, 30th July 18
Sync Cmd
Time/Sync Cmd
(1/4 APA)
DPM-Felix Links• Data reception, timing reception, clocking on
Felix.• FIFO-like links DPMßàFelix?
David Cussans | DAQ Firmware meeting, RAL 31st May 18
DPM-Felix Links – WIB Data• Max density probably ~ 1 Felix/DPM per APA
• Limited by SSDs
• à 10 x 9.6 Gbit/s links into Felix, 10 x 9.6Gbit/s to DPM
David Cussans | DAQ Firmware meeting, RAL 31st May 18
DPM-Felix – Trigger Primitives• Assume dominated by 39Ar (107 Bq/10kT) , 16-
byte primitives• 67k primitives/s , 8.5 Mbit/s
David Cussans | DAQ Firmware meeting, RAL 31st May 18
DPM-Felix – Triggered Data• 10-sec buffer in FPGA
• Read out variable length, depending on commands from data selection.
• Messages to buffer controller ~ 67 kHz ( assume we select all 39Ar )• Each block of data ~ 2.4 kBits ( 100us, 12 bit , decompressed before transmission )• à 160 Mbit/s
David Cussans | DAQ Firmware meeting, RAL 31st May 18
DPM-Felix – SN Data• Send capture/readback commands over same
links as triggered data• Add one additional link
David Cussans | DAQ Firmware meeting, RAL 31st May 18
DPM-Felix – Setup/Control• DPM will need setup, control and monitoring
• Best as register access (?)• Protocol?• One additional bidirectional link
David Cussans | DAQ Firmware meeting, RAL 31st May 18
DPM-Felix – IPMI• DPM from xTCA ecosystem.
• Needs IPMI control• Physical layer = I2C
• Firmware/software• Not onerous (probably) for group with previous experience• May be possible to disable IPMI. ( Need to confirm )
David Cussans | DAQ Firmware meeting, RAL 31st May 18
DPM-Felix – Timing• Do we need a link to the timing/synchronization
system?• 250MBit/s signal in Single-Phase timing system
compatible with timing system hardware in Felix
David Cussans | DAQ Firmware meeting, RAL 31st May 18
DPM-Felix – Summary of Links• WIB Data à DPM: 10 links• Triggered data à Felix: < 10 links • Trigger primitives: 1 link• Supernova data: 1 link• Setup/control/monitoring: 1 link• Total: 13 bidirectional links, <= 10Gbit/s
• Links will need protocol
• IPMI: I2C ( 2 wires )
David Cussans | DAQ Firmware meeting, RAL 31st May 18
Firmware Blocks
David Cussans | DAQ meeting, 30th July 1813
• Pre hit-finding filtering:• RAL: Kostas Manoulopolous . See
https://indico.bnl.gov/event/4854/contributions/22958/attachments/19346/25193/NoiseFiltering.pdf
• Hit-finding:• Bristol: Kumar Kunal, Jim Brooke. See
https://indico.bnl.gov/event/4854/contributions/22892/attachments/19347/25194/JimBrooke-FPGAHitFinder-BNLWorkshop.pdf
• Buffer management:• UCL: Erdem Motuk, Ryan Nichols. See
https://indico.bnl.gov/event/4854/contributions/22957/attachments/19387/25252/buffer_management_dune20072018.pdf
• Framework for evaluation of firmware blocks in hardware:
• Bristol, RAL: Dave Newbold, Kostas Monoulopolous. See https://indico.bnl.gov/event/4854/contributions/22956/attachments/19340/25184/DAQ_FPGA_180720.pdf
Firmware Blocks
David Cussans | DAQ meeting, 30th July 1814
• Collaboration welcome in all areas • Especially welcome in:
• Compression –• Use lighter-weight algorithm than APE compression
implemented for protoDUNE• E.g. fixed encoding of the difference of samples• Loss in compression w.r.t. APE, but much lower resource usage
and quicker to decompress.• Storage to SSD
• Current assumption: Handled by Zync ARM core. Needs to be implemented and tested.
• Could buy IP to write from programmable logic ( get moregates per $/£/CHF/Euro for Kintex etc. than Zync ). Would need to be integrated, tested.
DUNE Specific Firmware for Felix• Transmit data to DPM
• Decoding WIB data done in Felix.• What format to send to DPM? Just a copy?
• Interfaces that could all look the same ( AXI4-Stream ?)
• Transmit trigger primitives• Receive trigger commands (including SN capture)• Transmit data in trigger buffer requested by trigger
command• Transmit (at low bandwidth ) SN data on request
David Cussans | DAQ Firmware meeting, RAL 31st May 18
Felix + DPM hardware plans• BNL will develop a variant of the Felix that can host a
DPM• Will also contain some other features ( e.g. new FPGA family,
removing need for external PCIe bridge)• Ready ~ October/November.
• DPM scheduled to be ready before modified Felix• Design a carrier board that can host a DPM + miniPod optics (
UCL ).• Based on SLAC carrier board(?)• Use with existing, unmodified, Felix.
• Need urgent work on establishing a way of exchanging design information about DPM, Felix.• FNAL Redmine not open to all collaborators. Pay for private
Github, Bitbucket repo?• Before DPM arrives – test with Eval board coupled to
existing Felix.• E.g. ZCU102 with two dual QSFP FMCs. ( 16 bidirectional links
total )
David Cussans | DAQ Firmware meeting, RAL 31st May 18
Hardware for TDR – Eval Boards• Eval boards• UnmodifiedFelix
Hardware for TDR – DPM + Carrier• DPM on simple carrier• Unmodified Carrier
Summary• Plan in place to construct hardware test-platform of Felix+DPM
on time-scale of TDR:
• Eval board ßà unmodified Felix at first
• Then DPM + carrier ß à unmodified Felix
• Then DPM + modified Felix
• This option ready in time for TDR, most data will be gathered with previous hardware.
• Plan in place to develop algorithm firmware
• Compression, storage to SSD still open
• Need to develop plan for DUNE specific Felix firmware.
David Cussans | DAQ Firmware meeting, RAL 31st May 1819