cern, john erik sloper october 18, 20151 the atlas trigger and data acquisition: a brief overview of...

CERN, John Erik SloperApril 21, 2023 1

The ATLAS Trigger and Data Acquisition:The ATLAS Trigger and Data Acquisition:a brief overview of concept, design and realization a brief overview of concept, design and realization

John Erik SloperJohn Erik SloperATLAS TDAQ groupATLAS TDAQ groupCERN - Physics Dept.CERN - Physics Dept.

CERN, John Erik Sloper

OverviewOverview

IntroductionChallenges & RequirementsATLAS TDAQ Architecture

• Readout and LVL1

• LVL2 Triggering & Region of interest

• Event Building & Event filtering

Current status of installation

April 21, 2023 2


IntroductionIntroduction

Me: • John Erik Sloper, originally from Bergen, Norway

• With the TDAQ group for 3½ years

• Computer science background, currently enrolled at university of Warwick, Coventry for a PhD in engineering

Today:• Overview of Trigger and Data AcQuisition (TDAQ)

• Practical view point

• Using the real ATLAS TDAQ system as an example• We will go through the entire architecture of the TDAQ system from readout to

storage

• A brief overview of the status of the installation

April 21, 2023 3


Data acquisition and Data acquisition and triggeringtriggering

Data acquisition• Gathering

• Receiving data from all read-out links for the entire detector

• Processing• “Building the events” – Collecting all data that correspond to a single event• Serving the triggering system with data

• Storing• Transporting the data to mass storage

Triggering• The trigger has the job of selecting the bunch-

crossings of interest for physics analysis, i.e. those containing interactions of interest

• Tells the data acquisition system what should be used for further processing

April 21, 2023 4

CERN, John Erik Sloper 5

What is an “event” anyway?What is an “event” anyway?

In high-energy particle colliders (e.g. Tevatron, HERA, LHC), the particles in the counter-rotating beams are bunched • Bunches cross at regular intervals• Interactions only occur during the bunch-crossings

In this presentation “event” refers to the record of all the products of a given bunch-crossing• The term “event” is not uniquely defined!• Some people use the term “event” for the products of

a single interaction between the incident particles• People sometimes unwittingly use “event”

interchangeably to mean different things!


Trigger menusTrigger menus

Typically, trigger systems select events according to a “trigger menu”, i.e. a list of selection criteria• An event is selected by the trigger if one or more of the

criteria are met• Different criteria may correspond to different signatures

for the same physics process• Redundant selections lead to high selection efficiency and allow the efficiency of the

trigger to be measured from the data

• Different criteria may reflect the wish to concurrently select events for a wide range of physics studies

• HEP “experiments” — especially those with large general-purpose “detectors” (detector systems) — are really experimental facilities

• The menu has to cover the physics channels to be studied, plus additional event samples required to complete the analysis:

• Measure backgrounds, check the detector calibration and alignment, etc.


OverviewOverview






April 21, 2023 8


ATLAS TDAQ ATLAS TDAQ

April 21, 2023 9

Weight: 7000 t 44 m

22 m


Particle multiplicity Particle multiplicity

• = rapidity = log(tg/2) (longitudinal dimension)• uch = no. charged particles / unit-• nch = no. charged particles / interaction• Nch = total no. charged particles / BC• Ntot = total no. particles / BC

nch = uch x h= 6 x 7 = 42

Nch = nch x 23 = ~ 900

Ntot = Nch x 1.5 = ~ 1400

The LHC flushes each detector with ~1400 particles every 25 ns(p-p operation)

7.5 m

… still much more complex than a LEP event


Higgs -> 4

The challengeThe challenge

How to extract this… … from this …

+30 MinBias

… at a rate of ONE every 10 thousands billionsand without knowing where to look for:

the Higgs could be anywhere up to ~1 Tev or even nowhere…


Global requirementsGlobal requirements

No. overlapping events/25 ns 23

No. particles in ATLAS/25 ns 1400

Data throughput• At detectors (40 MHz) (equivalent to)

TB/s

• --> LVL1 Accepts O(100) GB/s

• --> Mass storage O(100) MB/s


Immediate observationsImmediate observations

Very high rate• New data every 25 ns – virtually impossible to make

real time decisions at this rate.• Not even time for signals to propagate through electronics

Amount of data• TB/s

• Can obviously not be stored directly. No hardware or networks exist (or at least not affordable!) that can handle this amount of data

• Even if we could, analyzing all the data would be extremely time consuming

April 21, 2023 13

The TDAQ system must reduce the amount of data by several order of magnitudes


OverviewOverview




• Event Building

• Event filtering


April 21, 2023 14


22 m

Global viewGlobal view

The ATLAS TDAQ architecture is based on a three-level trigger hierarchy

• Level 1• Level 2• Even filter

It uses a Level 2 selection mechanism based on a subset of event data -> Region-of-Interest • This reduces the amount of data needed to do

lvl2 filtering• Therefore, there is a much reduced demand on

dataflow power• Note that ATLAS differs from CMS on this point

April 21, 2023 15

Weight: 7000 t


Multi-level triggersMulti-level triggers

Multi-level triggers provide• Rapid rejection of high-rate backgrounds without

incurring (much) dead-time• Fast first-level trigger (custom electronics)• Needs high efficiency, but rejection power can be comparatively modest

• High overall rejection power to reduce output to mass storage to affordable rate

• Progressive reduction in rate after each stage of selection allows use of more and more complex algorithms at affordable cost

• Final stages of selection, running on computer farms, can use comparatively very complex (and hence slow) algorithms to achieve the required overall rejection power

16Nick Ellis, Seminar, Lecce, October 2007


OverviewOverview






April 21, 2023 17


LVL1

Trigger DAQ

2.5

s

Lvl1 acc

40 MHz

Calo MuTrCh Other detectors

ARCHITECTURE(Functional elements and their connections)

FE Pipelines


LVL1 selection criteriaLVL1 selection criteria

Features that distinguish new physics from the bulk of the cross-section for Standard Model processes at hadron colliders are:

• In general, the presence of high-pT particles (or jets)• e.g. these may be the products of the decays of new heavy particles• In contrast, most of the particles produced in minimum-bias interactions are soft

(pT ~ 1 GeV or less)

• More specifically, the presence of high-pT leptons (e, ), photons and/or neutrinos

• e.g. the products (directly or indirectly) of new heavy particles• These give a clean signature c.f. low-pT hadrons in minimum-bias case, especially

if they are “isolated” (i.e. not inside jets)

• The presence of known heavy particles• e.g. W and Z bosons may be produced in Higgs particle decays• Leptonic W and Z decays give a very clean signature

• Also interesting for physics analysis and detector studies


LVL1 signatures and LVL1 signatures and backgroundsbackgrounds

LVL1 triggers therefore search for• High-pT muons

• Identified beyond calorimeters; need pT cut to control rate from , K+ , as well as semi-leptonic beauty and charm decays

• High-pT photons• Identified as narrow EM calorimeter clusters; need cut on ET; cuts on isolation

and hadronic-energy veto reduce strongly rates from high-pT jets

• High-pT electrons• Same as photon (matching track in required in subsequent selection)

• High-pT taus (decaying to hadrons)• Identified as narrow cluster in EM+hadronic calorimeters

• High-pT jets• Identified as cluster in EM+hadronic calorimeter — need to cut at very high pT to

control rate (jets are dominant high-pT process)

• Large missing ET or total scalar ET


Level1 - Only Calorimeters and MuonsLevel1 - Only Calorimeters and Muons


Weight: 7000

t44 m

22 m

Lev

el-1

lat

ency

Lev

el-1

lat

ency

Interactions every 25 ns …• In 25 ns particles travel 7.5

m

• Cable length ~100 meters …

• In 25 ns signals travel 5 m

Total Level-1 latency = (TOF+cables+processing+distribution) = 2.5 sec

For 2.5 sec, all signals must be stored in electronics pipelines

(there are 108 channels!)


Global Architecture - Level 1Global Architecture - Level 1

Level 1 Trigger

Calorimetry

Muon Tr Ch FE pipelines

DETECTOR

Inner Tracker Calorimeters

Muon Trigger

Muon tracker

LHC Beam Xings

Central Trigger Processor

Level-

1 A

ccep

t

Data Acquisition

…

…

……

…

……

…

……

…

……

…


LVL1

DE

T RO

Trigger DAQ

2.5

s

Read-Out DriversLvl1 acc

40 MHz


ROD ROD ROD

Read-Out Links

Read-Out BuffersROB ROB ROB

Event data


FE Pipelines


AT

LA

S

AT

LA

S -

A

To

roid

al L

hc

Ap

par

atu

S

-

- A

To

roid

al L

hc

Ap

par

atu

S

-

T

rig

. ra

te &

Ev.

siz

eT

rig

. ra

te &

Ev.

siz

e

LVL1 trigger rate (high lum) = 40 kHzTotal event size = 1.5 MBTotal no. ROLs = 1600

Design system for 75 kHz --> 120 GB/s

(upgradeable to 100 kHz, 150 GB/s)

Muon Channels

No. ROLs

Fragment size -

kB

MDT 3.7x105 192 0.8

CSC 6.7x104 32 0.2

RPC 3.5x105 32 0.38

TGC 4.4x105 16 0.38

InDet Channels

No. ROLs

Fragment size -

kB

Pixels 1.4x108 120 0.5

SCT 6.2x106 92 1.1

TRT 3.7x105 232 1.2

Calo Channels

No. ROLs

Fragment size -

kB

LAr 1.8x105 764 0.75

Tile 104 64 0.75

Trig Channels

No. ROLs

Fragment size -

kB

LVL1 56 1.2

LVL1 trigger ratesSelection 21033 cm-2s-

1 1034 cm-2s-1

MU20 (20) 0.8 4.0

2MU6 0.2 1.0

EM25I (30) 12.0 22.0

2EM15I (20) 4.0 5.0

J200 (290) 0.2 0.2

3J90 (130) 0.2 0.2

4J65 (90) 0.2 0.2

J60 + xE60 (100+100) 0.4 0.5

TAU25I + xE30 (60+60) 2.0 1.0

MU10 + EM15I 0.1 0.4

Others (pre-scales, calib, …)

5.0 5.0

Total ~ 25 ~ 40

Rates in kHz No safety factor included!


OverviewOverview






April 21, 2023 27


LVL1

DE

T RO

Trigger DAQ


2.5

s


FE Pipelines

Read-Out Drivers

Read-Out Buffers

ROIB

RoI BuilderR

eg

ion

of

Inte

rest

Lvl1 acc = 75 kHz

40 MHz

Read-Out Links

ROD ROD ROD

ROB ROB ROB

120 GB/sEvent data


The Level-1 selection is dominated by local signatures• Based on coarse granularity (calo, mu trig chamb), w/out access to

inner tracking

• Important further rejection can be gained with local analysis of full detector data

Region of Interest - Why?Region of Interest - Why?

The geographical addresses of interesting signatures identified by the LVL1 (Regions of Interest)

• Allow access to local data of each relevant detector

• Sequentially

Typically, there is 1-2 RoI per event accepted by LVL1

• <RoIs/ev> = ~1.6

The resulting total amount of RoI data is minimal

• a few % of the Level-1 throughput


There is a simple correspondence

# region <-> ROB number(s)

(for each detector)

-> for each RoI the list of ROBs with the corresponding data from each detector is quickly identified (LVL2 processors)

This mechanism provides a powerful and economic way to add an important rejection factor before full Event Building

4 RoI addresses

--> the ATLAS RoI-based Level-2 trigger

… ~ one order of magnitude smaller ReadOut network …

… at the cost of a higher control traffic …

RoI mechanism - ImplementationRoI mechanism - Implementation

Note that this example is atypical; the average number of RoIs/ev is ~1.6


ROS

LVL1

DE

T RO

LVL2

Trigger DAQ

2.5

s


FE Pipelines

Read-Out Drivers

Read-Out Sub-systemsROIB

L2P

L2SV

L2N

RoI Builder

L2 SupervisorL2 N/work

L2 Proc Unit

RoI

RoI data

RoI requests

Lvl2 acc

Lvl1 acc = 75 kHz

40 MHz

Read-Out Buffers

Read-Out Links

ROD ROD ROD

ROB ROB ROB


There is typically only 1-2 RoI / event

Only the RoI Builder and ROB input see the full LVL1 rate(same simple point-to-point link)

120 GB/s


LVL2 TraficLVL2 Trafic

ReadoutBuffers

LVL2 Supervisor

L2 Processors

L2 Network

Detector FrontendLevel 1Trigger

Controls

decision

decision


L2 & EB networks L2 & EB networks - Full system on surface (SDX15)- Full system on surface (SDX15)

L2Switch Cross

Switch

ROS systemsROS systems

144

…

~80

USA15

SDX15

L2PUL2PU

L2SV

L2PU

GbEthernet


Level-2 Trigger

Three parameters characterise the RoI-based Level-2 trigger:

the amount of data required the overall time budget in L2P

the rejection factor

Level-2 Trigger

Three parameters characterise the RoI-based Level-2 trigger:

the amount of data required : 1-2% of totalthe overall time budget in L2P : 10 ms average

the rejection factor : x 30


OverviewOverview






April 21, 2023 35


ROS

LVL1

DE

T RO

LVL2

Trigger DAQ

2.5

s

~ 10 ms


FE Pipelines

Read-Out Drivers

Read-Out Sub-systems

ROIB

L2P

L2SV

L2N

RoI Builder


L2 Proc Unit

RoI

RoI data = 1-2%

RoI requests

Lvl2 acc = ~2 kHz

Lvl1 acc = 75 kHz

40 MHz

120 GB/s

Read-Out Buffers

Read-Out Links

ROD ROD ROD

ROB ROB ROB


EBSFI

EBN

Sub-Farm Input

Event Building N/work

Event Builder

Event Filter N/workEFN

Dataflow ManagerDFM


Event BuildingEvent Building

ReadoutSystems

DataFlow Manager

Event Filter

Builder Networks

Detector FrontendLevel 1 +2 Trigger

Controls


H

L

T

DATAFLOW

ROS

LVL1

DE

T RO

LVL2

Trigger DAQ

2.5

s


FE Pipelines

Read-Out Drivers

Read-Out Sub-systems

ROIB

L2P

L2SV

L2N

RoI Builder


L2 Proc Unit

RoI

~ 40 ms

RoI data = 1-2%

RoI requests

Lvl2 acc = ~2 kHz

Sub-Farm OutputSFO

Lvl1 acc = 75 kHz

40 MHz

Event FilterEFP

EFPEFP

EFP

~ 4 s

Event FilterProcessors

120 GB/s

~4

GB

/s

EFacc = ~0.2 kHz

Read-Out Buffers

Read-Out Links

ROD ROD ROD

ROB ROB ROB


EBSFI

EBN

Sub-Farm Input

Event Building N/work

Event Builder

Event Filter N/workEFN

Dataflow ManagerDFM

40 MHz

75 kHz

~2 kHz

~ 200 Hz

120 GB/s

~ 300 MB/s-> 1 CD / sec

~2+4 GB/s

10’s TB/s(equivalent)

100 000 CDs/s


L2 & EB networks L2 & EB networks - Full system on surface (SDX15)- Full system on surface (SDX15)

L2Switch

EBSwitchCross

Switch

ROS systemsROS systems

144 144

…

~80~110

USA15

SDX15

L2PUL2PU

L2SV

DFM

L2PU

SFI

Event Filte

r

GbE switches of this size can be bought today SFO

GbEthernet


OverviewOverview



• LVL2 Triggering

• Event Building

• Event filtering


April 21, 2023 40


TDAQ ARCHITECTURE

H

L

T

DATAFLOW

ROS

LVL1

DE

T RO

LVL2

2.5

s

~ 10 ms

ROIB

L2P

L2SV

L2N

SFO

Event FilterEFP

EFPEFP

EFP

~ sec

ROD ROD ROD

ROB ROB ROB

EB

SFI

EBN

EFN

DFM


Trigger / DAQ architecture

UX15

Read-Out

Drivers(RODs) First-

leveltrigger

Dedicated linksVME

Data of events acceptedby first-level trigger

Read-OutSubsystems

(ROSs)

1600Read-OutLinks

~150PCs

Event data pushed @ ≤ 100 kHz, 1600 fragments of ~ 1 kByte each


Read-Out System (ROS)Read-Out System (ROS) 153 ROSs installed in USA15 (completed in August 2006)

• 149 = 100% foreseen for “core” detectors + 4 hot spares • All fully equipped with ROBINs

PC properties• 4U, 19" rack mountable PC • Motherboard: Supermicro X6DHE-XB• CPU: Uni-proc 3.4 GHz Xeon• RAM: 512 MB

• Redundant power supplyNetwork booted (no local hard disk)Remote management via IPMI

• Network: • 2 GbE onboard (1 for control network)• 4 GbE on PCI-Express card

1 for LVL2 + 1 for event building

• Spares:• 12 full systems in pre-series as hot spares• 27 PC purchased (7 for new detectors)

ROBINs (Read-Out Buffer Inputs)• Input : 3 ROLs (S-Link I/F) Output : 1 PCI I/F and 1 GbE I/F (for upgrade if needed)

Buffer memory : 64 Mbytes (~32k fragments per ROL)

• ~700 ROBINs installed and spares• + 70 ROBINs ordered (new detectors and more spares)

•

System is complete : no further purchasing/procurement foreseen


UX15

Read-Out


leveltrigger

Dedicated linksVME


Read-OutSubsystems

(ROSs)

USA15 1600Read-OutLinks

~150PCs



USA15

SDX1

Reg

ions

Of I

nter

est

LVL2Super-visor

Gig

abit

Eth

erne

t

Eve

nt d

ata

requ

ests

Del

ete

com

man

ds

Req

uest

ed e

vent

dat

a

Network switches

Event data pulled:partial events @ ≤ 100 kHz, full events @ ~ 3 kHz

RoIBuilder

Second-leveltrigger

pROS

~ 500

stores LVL2output

LVL2 farm

DataFlowManager

EventFilter(EF)

~1800

Network switches

~100 EventBuilderSubFarm

Inputs

(SFIs)


Central SwitchesCentral Switches

• Data network- Event builder traffic- LVL2 traffic

• Force10 E1200 - 1 Chassis + blades for 1 and 10 GbE as required

For July 2008 : 14 blades in 2 chassis~700 GbE ports@line

speedFinal system : extra blades

for final system

• Back-end network- To Event Filter and SFO

• Force10 E600 - 1 Chassis + blades for 1 GbE

For July 2008 : blades as requiredfollowing EF evolution

Final system : extra bladesfor final system

• Core online/Control network- Run Control - Databases- Monitoring samplers

• Force10 E600 - 1 Chassis + blades for 1 GbE

For July 2008 : 2 chassis + blades following full system evolution

Final system : extra blades for final system


HLT FarmsHLT Farms• Final size for max L1 rate (TDR)

~ 500 PCs for L2~ 1800 PCs for EF

(multi-core technology -> same no. boxes, more applications)

• Recently decided:~ 900 of above will be XPUs

connected to both L2 and EF networks

• 161 XPU PCs installed• 130 x 8 cores

• CPU: 2 x Intel E5320 quad-core 1.86 GHz • RAM: 1 GB / core, i.e. 8 GB

+ 31 x 4 cores

• Cold-swappable power supplyNetwork booted - Local hard disk Remote management via IPMI

• Network:

• 2 GbE onboard, 1 for control network, 1 for data network

• VLAN for the connection to both data and back-end networks

For July 2008 : total of 9 L2 + 27 EF racks as from TDR for 2007 run (1 rack = 31 PCs)

Final system : total of 17 L2 + 62 EF racks of which 28 (of 79) racks with XPU connection


DAQ/HLT in SDX1DAQ/HLT in SDX1

+~90 racks (mostly empty…)+~90 racks (mostly empty…)


USA15

SDX1

Gig

abit

Eth

erne

t

Eve

nt d

ata

requ

ests

Del

ete

com

man

ds

Req

uest

ed e

vent

dat

a

Reg

ions

Of I

nter

est

LVL2Super-visor

Network switches

Second-leveltrigger

pROS

~ 500

stores LVL2output

LVL2 farm

UX15

Read-Out


leveltrigger

Dedicated linksVME


Read-OutSubsystems

(ROSs)

USA15 1600Read-OutLinks

RoIBuilder

~150PCs


Timing Trigger Control (TTC)

DataFlowManager

EventFilter(EF)

~1800

Network switches

Event data pulled:partial events @ ≤ 100 kHz, full events @ ~ 3 kHz

SDX1 dual-CPU nodesCERN computer centre Event rate

~ 200 HzData storage

6Local

StorageSubFarmOutputs

(SFOs)

~100 EventBuilderSubFarm

Inputs

(SFIs)



SubFarm OutputSubFarm Output• 6 SFO PCs installed• 5+1 hot spare

• 5U, 19" rack mountable PC • Motherboard: Intel • CPU: 2 x Intel E5320 dual-core 2.0 GHz • RAM: 4 GB

• Quadruple cold-swappable power supplyNetwork bootedLocal hard disk

recovery from crashespre-load events for testing (e.g. FDR)

Remote management via IPMI

• Disks:• 3 SATA Raid controllers

• 24 disks x 500 GB = 12 TB

• Network: • 2 GbE onboard

1 for control and IPMI, 1 for data• 3 GbE on PCIe card for data

System is complete, no further purchasing foreseen required b/width (300MB/s) already availableto facilitate detector commissioning andcalibrations in early phase


CABLING!CABLING!


Weight: 7000 t 44 m

22 m TDAQ System operational at Point-1

Routinely used for

Debugging and standalone commissioning of all detectors after installation

TDAQ Technical Runs - use physics selection algorithms in the HLT farms on simulated data pre-loaded in the Read-Out System

Commissioning Runs of integrated ATLAS - take cosmic data through full TDAQ chain (up to Tier-0) after final detectors integration

CERN, John Erik SloperApril 21, 2023 53CHEP 07 B. Gorini - Integration of the Trigger and Data Acquisition Systems in ATLAS

Run ControlRun ControlRun ControlRun Control

7

The M4 Commissioning Run (Aug 23 - Sep 03, 2007)Essentially all detectors integrated

Full DAQ chain (up to Tier0)Tile+RPC+TGC triggering on cosmics

Shifters and systems readiness

Trigger status

Run Control tree

Event rate


Combined Cosmic run in June Combined Cosmic run in June 2007 2007 (M3)(M3)

59

In June we had a 14 day combined cosmic run. Ran with no magnetic field.Included following systems:

Muons – RPC (~1/32) , MDT (~1/16), TGC (~1/36)

Calorimeters – EM (LAr )(~50%) & Hadronic (Tile) (~75%)

Tracking – Transition Radiation Tracker (TRT) (~6/32 of the barrel of the final system)

Only systems missing are the Silicon strips and pixels and the muon system CSCs


Conclusions:Conclusions:we have started to operate ATLAS at full-system scalewe have started to operate ATLAS at full-system scale

We seem to be in business,the ATLAS TDAQ system is doing its

jobat least so far…

… but the real test is still to come ...


Status & ConclusionsStatus & Conclusions

The ATLAS TDAQ system has a three-level trigger hierarchy,making use of the Region-of-Interest mechanism

--> important reduction of data movement

The system design is complete, but is open to:Optimization of the I/O at the Read-Out System level

Optimization of the deployment of the LVL2 and Event Builder networks

The architecture has been validated via deployment of full systems:

On test bed prototypesAt the ATLAS H8 test beam

And via detailed modeling to extrapolate to full size


Thanks for listening!Thanks for listening!

Big thanks to:

Livio Mapelli, Giovanna Lehmann-Miotto and

Nick Ellis

for lending slides and helping with this presentation

April 21, 2023 62

cern, john erik sloper october 18, 20151 the atlas trigger and data acquisition: a brief overview of...

Documents