Online Triggers and DAQopenlab V

Niko Neufeld, CERN/PH

CERN openlab major review Oct. 2014

openlab V lightning DAQ N. Neufeld 2

Data acquisition and online challenges - recap

Online data filtering and processing(quasi-) realtime data reduction for high-rate detectors

High bandwidth networking for data acquisitionTerabit/s networks

Data-transfer and storage

openlab V lightning DAQ N. Neufeld 3

Evolution of the LHCb trigger-DAQLHCb Run2 LHCb Run 3

Max. inst. luminosity 4 x 10^32 2 x 10^33

Event-size (mean – zero-suppressed) [kB] ~ 60 (L0 accepted) ~ 100

Event-building rate [MHz] 1 40

# read-out boards ~ 330 400 - 500

link speed from detector [Gbit/s] 1.6 4.5

output data-rate / read-out board [Gbit/s] 4 100

# detector-links / readout-board up to 24 up to 48

# farm-nodes ~ 1600 1000 - 4000

# links 100 Gbit/s (from event-builder PCs) n/a 400 - 500

final output rate to tape [kHz] 12 20 - 100

openlab V lightning DAQ N. Neufeld 4

Detector front-end electronics

Eventbuilder network

Eventbuilder PCs + software LLT

Eventfilter Farm~ 80 subfarms

UX8

5BPo

int 8

surf

ace

subfarm switch

TFC500

6 x 100 Gbit/s

subfarm switch

Online storage

Clock & fast com

mands

8800Versatile Link

throttle from PCIe40

Clock & fast commands

6 x 100 Gbit/s

Online ArchitectureEC

S

5

Event-size [kB] Rate [kHz] Bandwidth [Gb/s] Year [CE]ALICE 20000 50 8000 2019ATLAS 4000 200 6400 2023

CMS 4000 1000 32000 2023LHCb 100 40000 32000 2019

Future data rates @ LHC (trigger stage 2)

openlab V lightning DAQ N. Neufeld

40000 kHz == collision rate LHCb abandons Level 1 for an all-software triggerO(100) Tbit/s networks required

openlab V lightning DAQ N. Neufeld 6

Project HTCC (High Throughput Computing Collaboration)

Partners: CERN, intelGoal apply intel technologies to Online computing challenges

Work-packages1. Interconnect for DAQ (Stormlake)2. Knights Landing for software triggers (HLT and L1.5)3. Reconfigurable Logic (Xeon/FPGA)

Use LHCb use-case as concrete example problems, but try to keep solutions and reports as widely applicable as possible

openlab V lightning DAQ N. Neufeld 7

Original “Use-cases” in Online (from openlab V white paper) 1) “Level-1” using (more/all) COTS hardware ✔2) “Data Acquisition” ✔3) “High Level Trigger” ✔ 4) “Controls”5) “Online data processing” ✔6) “Exporting data”

✔ == covered in HTCC

Additional material

Use-case #1 The first level trigger

openlab V lightning DAQ N. Neufeld 10

Calorimeter data

11

Finding Muons (2d view)

openlab V lightning DAQ N. Neufeld

Level 1 Trigger todayThe Level 1 Trigger is implemented in hardware: FPGAs and ASICs difficult / expensive to upgrade or change, maintenance by experts onlyDecision time: ~ a small number of microsecondsIt uses simple, hardware-friendly signatures looses interesting collisionsEach sub-detector has its own solution, only the uplink is standardized

openlab V lightning DAQ N. Neufeld12

openlab V lightning DAQ N. Neufeld 13

Level 1 challenge

Can we do this in software? Using GPGPUs / XeonPhis?We need low and near– deterministic latency Need an efficient interface to detector-hardware: CPU/FPGA hybrid?

Use-case #2 and #6Data Acquisition and Export fast data transport

Data Acquisition (generic example)

openlab V lightning DAQ N. Neufeld15

custom radiation- hard link from the detector

DAQ (“event-building”) links – some LAN (10/40/100 GbE / InfiniBand)

Links into compute-units: typically 1/10 Gbit/s

Detector

DAQ network

100 m rock

Readout Units

Compute Units

Every Readout Unit has a piece of the collision dataAll pieces must be brought together into a single compute unitThe Compute Unit runs the software filtering (High Level Trigger – HLT)

10000 x

~ 1000 x

~ 3000 x

openlab V lightning DAQ N. Neufeld 16

Data acquisition challenge

Transport large amount of data (multiple Terabit/s @ LHC) reliably and cost-effectivelyIntegrate the network closely and efficiently with compute resources (be they classical CPU or “many-core”)Multiple network technologies should seamlessly co-exist in the same integrated fabric (“the right link for the right task”), end-to-end solution from online processing to scientist laptop (e.g. light-sources)

Use-case #3 and #5Online Data processing

openlab V lightning DAQ N. Neufeld 18

Pattern finding - tracks

19

Same in 2 dimensions

openlab V lightning DAQ N. Neufeld

Can be much more complicated: lots of tracks / rings, curved / spiral trajectories, spurious measurements and various other imperfections

LHC High Level Trigger: Key Figures

Existing code base: 5 MLOC of mostly C++Almost all algorithms are single-threaded (only few exceptions)Currently processing time on a X5650 per event: several 10 ms / process (hyper-thread)Currently between 100k and 1 million events per second are filtered online in each of the 4 LHC experiments

openlab V lightning DAQ N. Neufeld20

21

Online Data processing challenge

Make the code-base ready for multi/many-core (this is not Online specific!)Optimize the online processing compute in terms of cost, power, cooling Find the best architecture integrating “standard servers”, many-core systems and a high-bandwidth network

openlab V lightning DAQ N. Neufeld