Download - High Level Triggering
High Level Triggering
Fred Wickens
2
High Level Triggering (HLT)
• Introduction to triggering and HLT systems– What is Triggering– What is High Level Triggering – Why do we need it
• Case study of ATLAS HLT (+ some comparisons with other experiments)
• Summary
3
Why do we Trigger and why multi-level• Over the years experiments have focussed on rarer processes
– Need large statistics of these rare events– DAQ system (and off-line analysis capability) under increasing
strain• limiting useful event statistics
• Aim of the trigger is to record just the events of interest– i.e. Trigger selects the events we wish to study
• Originally - only read-out the detector if Trigger satisfied– Larger detectors and slow serial read-out => large dead-time – Also increasingly difficult to select the interesting events
• Introduced: Multi-level triggers and parallel read-out– At each level apply increasingly complex algorithms to obtain better
event selection/background rejection• These have:
– Led to major reduction in Dead-time – which was the major issue– Managed growth in data rates – this remains the major issue
4
Summary of ATLAS Data Flow Rates
• From detectors > 1014 Bytes/sec
• After Level-1 accept ~ 1011 Bytes/sec
• Into event builder ~ 109 Bytes/sec
• Onto permanent storage ~ 108 Bytes/sec
~ 1015 Bytes/year
5
The evolution of DAQ systems
6
TDAQ Comparisons
7
Level 1• Time: few microseconds• Hardware based
– Using fast detectors + fast algorithms – Reduced granularity and precision
• calorimeter energy sums• tracking by masks
• During Level-1 decision time store event data in front-end electronics – at LHC use pipeline - as collision rate shorter than
Level-1 decision time• For details of Level-1 see Dave Newbold talk
8
High Level Trigger - Levels 2 + 3• Level-2 : Few milliseconds (10-100)
– Partial events received via high-speed network– Specialised algorithms
• 3-D, fine grain calorimetry• tracking, matching• Topology
• Level-3 : Up to a few seconds– Full or partial event reconstruction
• after event building (collection of all data from all detectors)• Level-2 + Level-3
– Processor farm with Linux server PC’s– Each event allocated to a single processor, large farm of
processors to handle rate
9
Summary of Introduction• For many physics analyses, aim is to obtain as high
statistics as possible for a given process– We cannot afford to handle or store all of the data a detector
can produce!• The Trigger
– selects the most interesting events from the myriad of events seen
• I.e. Obtain better use of limited output band-width• Throw away less interesting events• Keep all of the good events(or as many as possible)
– must get it right• any good events thrown away are lost for ever!
• High level Trigger allows:– More complex selection algorithms– Use of all detectors and full granularity full precision data
Case study of the ATLAS HLT system
Concentrate on issues relevant forATLAS (CMS very similar issues), but
try to address some more general points
11
Starting points for any Trigger system
• physics programme for the experiment– what are you trying to measure
• accelerator parameters– what rates and structures
• detector and trigger performance– what data is available– what trigger resources do we have to use it
• Particularly network b/w + cpu performance
12
7 TeV Interesting events are buried in a seaof soft interactions
Higgs production
High energy QCD jet production
Physics at the LHC
B physics
top physics
13
The LHC and ATLAS/CMS• LHC has
– Design luminosity 1034 cm-2s-1 • 2010: 1027 – 2x1032 ; 2011: up to 3.6x1033 ; 2012: up to 6x1033
– Design bunch separation 25 ns (bunch length ~1 ns)• Currently running with 50 ns
• This results in– ~ 23 interactions / bunch crossing (Already exceeded!)
• ~ 80 charged particles (mainly soft pions) / interaction • ~2000 charged particles / bunch crossing
• Total interaction rate 109 sec-1
– b-physics fraction ~ 10-3 106 sec-1
– t-physics fraction ~ 10-8 10 sec-1
– Higgs fraction ~ 10-11 10-2 sec-1
14
Physics programme• Higgs signal extraction important - but very difficult • There is lots of other interesting physics
– B physics and CP violation– quarks, gluons and QCD– top quarks– SUSY– ‘new’ physics
• Programme evolving with: luminosity and HLT capacity– i.e. Balance between
• high PT programme (Higgs etc.)• b-physics programme (CP measurements)• searches for new physics
15
Trigger strategy at LHC• To avoid being overwhelmed use signatures with
small backgrounds– Leptons– High mass resonances– Heavy quarks
• The trigger selection looks for events with: – Isolated leptons and photons, – -, central- and forward-jets – Events with high ET
– Events with missing ET
16
Objects Physics signatures
Electron 1e>25, 2e>15 GeV Higgs (SM, MSSM), new gauge bosons, extra dimensions, SUSY, W, top
Photon 1γ>60, 2γ>20 GeV Higgs (SM, MSSM), extra dimensions, SUSY
Muon 1μ>20, 2μ>10 GeV Higgs (SM, MSSM), new gauge bosons, extra dimensions, SUSY, W, top
Jet 1j>360, 3j>150, 4j>100 GeV SUSY, compositeness, resonances
Jet >60 + ETmiss >60 GeV SUSY, exotics
Tau >30 + ETmiss >40 GeV Extended Higgs models, SUSY
Example Physics signatures
17
ARCHITECTURE
40 MHzTrigger DAQ
~1 PB/s(equivalent)
~ 200 Hz ~ 300 MB/sPhysics
Three logical levelsLVL1 - Fastest:Only Calo and
MuHardwired
LVL2 - Local:LVL1 refinement
+track
associationLVL3 - Full
event:“Offline” analysis
~2.5 ms
~40 ms
~4 sec.
Hierarchical data-flow
On-detector electronics:
Pipelines
Event fragments buffered in
parallel
Full event in processor farm
18
Selected (inclusive) signatures
Process Level-1 Level-2H0 2 em, ET>20 GeV 2 , ET>20 GeV
H0 Z Z* + – + – 2 em, ET>20 GeV2 µ, pT>6 GeV1 em, ET>30 GeV1 µ, pT>20 GeV
2 e, ET>20 GeV2 µ, ET>6 GeV, I1 e, ET>30 GeV1 µ, ET>20 GeV, I
Z+–+X 2 em, ET>20 GeV2 µ, pT>6 GeV1 em, ET>30 GeV1 µ, pT>20 GeV
2 e, ET>20 GeV2 µ, ET>6 GeV, I1 e, ET>30 GeV1 µ, ET>20 GeV, I
t t leptons+jets 1 em, ET>30 GeV1 µ, pT>20 GeV
1 e, ET>30 GeV1 µ, ET>20 GeV, I
W', Z' jets 1 jet, ET>150 GeV 1 jet, ET>300 GeVSUSY jets 1 jet, ET>150 GeV
ETmiss
3 jet, ET>150 GeVET
miss
19
Central TriggerProcessor
Region-of-Interest Unit(Level-1/Level-2)
Level-2 TriggerFront-end Systems
Calorimeter TriggerProcessor
MuonTrigger
Processor
µ
Subtriggerinformation
Timing, trigger andcontrol distribution
JetET e /
Calorimeters Muon Detectors
Trigger design – Level-1• Level-1
– sets the context for the HLT– reduces triggers to ~75 kHz
• Limited detector data– Calo + Muon only– Reduced granularity
• Trigger on inclusive signatures
• muons; • em/tau/jet calo clusters;
missing and sum ET
• Hardware trigger– Programmable thresholds– CTP selection based on
multiplicities and thresholds
20
Level-1 Selection• The Level-1 trigger
– an “or” of a large number of inclusive signals – set to match the current physics priorities and beam
conditions• Precision of cuts at Level-1 is generally limited• Adjust the overall Level-1 accept rate (and the
relative frequency of different triggers) by– Adjusting thresholds – Pre-scaling (e.g. only accept every 10th trigger of a
particular type) higher rate triggers• Can be used to include a low rate of calibration events
• Menu can be changed at the start of run – Pre-scale factors may change during the course of a run
21
Trigger design - HLT strategy
• Level 2– confirm Level 1, some inclusive, some semi-
inclusive,some simple topology triggers, vertex reconstruction(e.g. two particle mass cuts to select Zs)
• Level 3– confirm Level 2, more refined topology selection,
near off-line code
22
Trigger design - Level-2• Level-2 reduce triggers to ~4 kHz (was ~2 kHz)
– Note CMS does not have a physically separate Level-2 trigger, but the HLT processors include a first stage of Level-2 algorithms
• Level-2 trigger has a short time budget – ATLAS ~40 milli-sec average
• Note for Level-1 the time budget is a hard limit for every event, for the High Level Trigger it is the average that matters, so OK for a small fraction of events to take times much longer than this average
• Full detector data is available, but to minimise resources needed:– Limit the data accessed– Only unpack detector data when it is needed– Use information from Level-1 to guide the process– Analysis proceeds in steps - can reject event after each step– Use custom algorithms
23
Regions of Interest
• The Level-1 selection is dominated by local signatures (I.e. within Region of Interest - RoI)– Based on coarse granularity
data from calo and mu only• Typically, there are
1-2 RoI/event• ATLAS uses RoI’s to reduce
network b/w and processing power required
24
Trigger design - Level-2 - cont’d• Processing scheme
– extract features from sub-detectors in each RoI – combine features from one RoI into object – combine objects to test event topology
• Precision of Level-2 cuts– Limited (although better than at Level-1)– Emphasis is on very fast algorithms with
reasonable accuracy• Do not include many corrections which may be applied
off-line– Calibrations and alignment available for trigger not
as precise as ones available for off-line
25
ARCHITECTURE
H
L
T
40 MHz
75 kHz
~4 kHz
~ 400 Hz
40 MHz
RoI data = 1-2%~2 GB/s
FE Pipelines2.5 ms
LVL1 accept
Read-Out DriversROD ROD ROD
LVL1 2.5 ms
CalorimeterTrigger
MuonTrigger
Event BuilderEB
~6 GB/s
ROS Read-Out Sub-systems
Read-Out BuffersROB ROB ROB
120 GB/s Read-Out Links
Calo MuTrCh Other detectors ~ 1 PB/s
Event FilterEFP
EFPEFP
~ 1 sec
EFN
~6 GB/s
~ 600 MB/s
~ 600 MB/s
Trigger DAQ
LVL2 ~ 10 ms
L2P
L2SV
L2NL2PL2P
ROIB
LVL2 accept
RoI requests
RoI’s
26
CMS Event Building• CMS perform Event Building after Level-1• Simplifies the architecture, but places much
higher demand on technology:– Network traffic
~100 GB/s– 1st stage use
Myrinet – 2nd stage has
8 GbE slices
27
t i m
e
e30i e30i +Signature
ecand ecand+Signature
e e +Signature
e30 e30+Signature
EM20i EM20i+Level1 seed
Cluster shape
Cluster shape STEP 1
Iso–lation
Iso–lationSTEP 4
pt>30GeV
pt>30GeVSTEP
3
trackfinding
trackfindingSTEP 2
HLT Strategy: Validate step-by-step Check intermediate signatures Reject as early as possible
Sequential/modular approach facilitates early rejection
LVL1 triggers on two isolated e/m clusters with pT>20GeV(possible signature: Z–>ee)
Example for Two electron trigger
28
Trigger design - Event Filter / Level-3
• Event Filter reduce triggers to ~400 Hz – (was ~200 Hz)
• Event Filter budget ~ 4 sec average• Full event detector data is available, but to
minimise resources needed:– Only unpack detector data when it is needed– Use information from Level-2 to guide the process– Analysis proceeds in steps with possibility to reject
event after each step– Use optimised off-line algorithms
29
Execution of a Trigger Chain
match?
L2 calorim.
L2 tracking
cluster?
track?
Level 2 seeded by Level 1• Fast reconstruction
algorithms • Reconstruction within RoI
Electromagneticclusters
EM ROI
Level1:Region of Interest is found and position in EM calorimeter is passed to Level 2
E.F.calorim.
E.F.tracking
track?
e/ OK?
e/ reconst.
Ev.Filter seeded by Level 2• Offline reconstruction
algorithms • Refined alignment and
calibration
30
e/γ Trigger
• pT≈3-20 GeV: b/c/tau decays, SUSY
• pT≈20-100 GeV: W/Z/top/Higgs• pT>100 GeV: exotics
• Level 1: local ET maximum in ΔηxΔφ = 0.2x0.2 with possible isolation cut
• Level 2: fast tracking and calorimeter clustering – use shower shape variables plus track-cluster matching
• Event Filter: high precision offline algorithms wrapped for online running
L1 EM triggerpT > 5GeV
31
• Discriminate against hadronic showers based on shower shape variables
• Use fine granularity of LAr calorimeter
• Resolution improved in Event Filter with respect to Level 2
R E37cells
E77cells
32
80% acceptance due to support structures etc.
Muon Trigger• Low PT: J/Y, Uand B-physics• High PT: H/Z/W/τ μ, SUSY, exotics➝
• Level 1: look for coincidence hits in muon trigger chambers – Resistive Plate Chambers (barrel) and
Thin Gap Chambers (endcap)– pT resolved from coincidence hits in look-up
table
• Level 2: refine Level 1 candidate with precision hits from Muon Drift Tubes (MDT) and combine with inner detector track
• Event Filter: use offline algorithms and precision; complementary algorithm does inside-out tracking and muon reconstruction
33
The Trigger Menu• Collection of trigger signatures• In LHC GPD’s menus there can be 100’s of algorithm
chains – defining which objects, thresholds and algorithms, etc should be used
• Selections set to match the current physics priorities and beam conditions within the bandwidth and rates allowed by the TDAQ system
• Includes calibration & monitoring chains• Principal mechanisms to adjust the accept rate (and
the relative frequency of different triggers)– Adjusting thresholds – Pre-scaling (e.g. only accept every 10th trigger of a
particular type) higher rate triggers• Can be used to include a low rate of calibration events
34
L1 trigger items and estimated rates at 10^31 cm−2 s−1 for jets
Jet ET spectrum at 10^31 cm−2 s−1 before (dashed) and after (solid) pre-scaling at L1
Example use of thresholds/prescales at Level-1
35
Trigger Menu cont’d• Basic Menu is defined at the start of a run
– Pre-scale factors can be changed during the course of a run• Adjust triggers to match current luminosity• Turn triggers on/off
36
Trigger Evolution in ATLAS
37
Matching problem• Ideally
– off-line algorithms select all the physics channel and no background
– trigger algorithms select all the physics accepted by the off-line selection (and no background)
• In practice, neither of these happen– Need to optimise the combined
selection• For this reason many trigger studies quote trigger efficiency wrt
events which pass off-line selection– BUT remember off-line can change algorithm, re-process and
recalibrate at a later stage• So, make sure on-line algorithm selection is well known, controlled
and monitored
Background
Physics channel
Off-line
On-line
38
Other issues for the Trigger• Optimisation of cuts
– Balance background rejection vs efficiency• Efficiency and Monitoring
– In general need high trigger efficiency– Also for many analyses need a well known efficiency
• Monitor efficiency by various means– Overlapping triggers– Pre-scaled samples of triggers in tagging mode (pass-through)
• Final detector calibration and alignment constants not available for the trigger– keep as up-to-date as possible– allow for the lower precision in the trigger cuts
• Code used in trigger needs to be fast + very robust– low memory leaks, low crash rate
39
Summary
• High-level triggers allow complex selection procedures to be applied as the data is taken– Thus allow large samples of rare events to be recorded
• The trigger stages - in the ATLAS example– Level 1 uses inclusive signatures (mu’s; em/tau/jet; missing and
sum ET)– Level 2 refines Level 1 selection, adds simple topology triggers,
vertex reconstruction, etc– Level 3 refines Level 2 adds more refined topology selection
• Trigger menus need to be defined, taking into account:– Physics priorities, beam conditions, HLT resources
• Include items for monitoring trigger efficiency and calibration• Try to match trigger cuts to off-line selection• Trigger efficiency should be as high as possible and well
monitored • Must get it right - events thrown away are lost for ever!• Triggering closely linked to physics analyses – so enjoy!
40
ATLAS 2e2μ candidate with m2e2μ= 124.3 GeVpT (e+, e-, μ-, μ+)= 41.5, 26.5, 24.7, 18.3 GeVm (e+e-)= 76.8 GeV, m(μ+μ-) = 45.7 GeV
41
4242
Excluded at 95% CL
Putting all channels together combined constraintsHγγ, H ττ H WW(*) lνlνH ZZ(*) 4l, H ZZ llννH ZZ llqq, H WWlνqqW/ZH lbb+X not included
Excluded at 99% CL
Expected if no signal
112.7 < mH < 115.5 GeV 131 <mH < 453 GeV, except 237-251 GeV
124.6-520 GeV
133 <mH < 230 GeV, 260 < mH < 437 GeV
LEP ATLAS+CMSCombination
ATLAStoday
43
44
45
Additional Foils
46
ATLAS HLT HardwareEach rack of HLT (XPU) processors contains- ~30 HLT PC’s (PC’s very similar to Tier-0/1 compute nodes)- 2 Gigabit Ethernet Switches- a dedicated Local File ServerFinal system will contain ~2300 PC’s
47SDX1|2nd floor|Rows 3 & 2
CFS nodes
UPS for CFS
LFS nodes
XPUs
48
Price to pay for the high luminosity: larger-than-expected pile-up
Z μμ
Period A: up to end August
Period B:Sept-Oct
Pile-up = number of interactions per crossing Tails up to ~20 comparable to design luminosity (50 ns operation; several machine parameters pushed beyond design)
LHC figures used over the last 20 years:~ 2 (20) events/crossing at L=1033 (1034)
Challenging for trigger, computing resources, reconstruction of physics objects (in particular ET
miss, soft jets, ..) Precise modeling of both in-time and out-of-time pile-up in simulation is essential
Event with 20 reconstructed vertices(ellipses have 20 σ size for visibility reasons)
49
Naming ConventionFirst Level Trigger (LVL1) Signatures in
capitals e.g. LVL1 HLT type
EMe electron
g photon
MU mu muon
HA tau tau
FJ fj forward jet
JE je jet energy
JT jt jet
TM xe missing energy
HLT in lower case:
name
threshold
isolated
mu 20 i _ passEF
EF in tagging mode
name
threshold
isolated
MU 20 I
New in 13.0.30: • Threshold is cut value applied• previously was ~95% effic. point.
• More details : see :https://twiki.cern.ch/twiki/bin/view/Atlas/TriggerPhysicsMenu
50
What is a minimum bias event ?
- event accepted with the only requirement being activity in the detector with minimal pT threshold [100 MeV] (zero bias events have no requirements) - e.g. Scintillators at L1 + (> 40 SCT S.P. or > 900 Pixel clusters) at L2
- a miminum bias event is most likely to be either: - a low pT (soft) non-diffractive event - a soft single-diffractive event - a soft double diffractive event(some people do not include the diffractive events in the definition !)
- it is characterised by: - having no high pT objects : jets; leptons; photons - being isotropic - see low pT tracks at all phi in a tracking detector - see uniform energy deposits in calorimeter as function of rapidity - these events occur in 99.999% of collisions. So if any given crossing has two interactions and one of them has been triggered due to a high pT component then the likelihood is that the accompanying event will be a dull minimum bias event.
51
Phys.Lett.B 688, Issue 1, 2010
LHC collision rate (nb=4)
LHC collision rate (nb=2)
• Soft QCD studies• Provide control trigger on p-p collisions;
discriminate against beam-related backgrounds (using signal time)
• Minimum Bias Scintillators (MBTS) installed in each end-cap;
• Example: MBTS_1 – at least 1 hit in MBTS
• Also check nr. of hits in Inner Detector in Level-2
Minimum Bias Trigger
Minbias Trigger Scintillator: 32 sectors on LAr cryostatMain trigger for initial running coverage 2.1 to 3.8
52
Hadronic Tau Trigger• W/Z ➝ , SM &MSSM Higgs, SUSY, Exotics
• Level 1: start from hadronic cluster – local maximum in ΔηxΔφ = 0.2x0.2 – possible to apply isolation
• Level 2: track and calorimeter information are combined – narrow cluster with few matching tracks
• Event Filter: 3D cluster reconstruction suppresses noise; offline ID algorithms and calibration used
• Typical background rejection factor of ≈5-10 from Level 2+Event Filter – Right: fake rate for loose tau trigger with pT > 12
GeV – aka tau12_loose– MC is Pythia with no LHC-specific tuning
53
Jet Trigger• QCD multijet production, top, SUSY,
generic BSM searches
• Level 1: look for local maximum in ET in calorimeter towers of ΔηxΔφ = 0.4x0.4 to 0.8x0.8
• Level 2: simplified cone clustering algorithm (3 iterations max) on calorimeter cells
• Event Filter: anti-kT algorithm on calorimeter cells; currently running in transparent mode (no rejection)
Note in preparation
54
Missing ET Trigger• SUSY, Higgs• Level 1: ET
miss and ET calculated from all calorimeter towers
• Level 2: only muon corrections possible (at present)
• Event Filter: re-calculate from calorimeter cells and reconstructed muons
Level 15 GeV threshold
Level 120 GeV threshold
55
Example Level-1 Menu for 2x10^33
Level-1 signature Output Rate (Hz)
EM25i 12000
2EM15i 4000
MU20 800
2MU6 200
J200 200
3J90 200
4J65 200
J60 + XE60 400
TAU25i + XE30 2000
MU10 + EM15i 100
Others (pre-scaled, exclusive, monitor, calibration) 5000
Total ~25000
56
End of pp trigger operations in 2010
Trigger group Trigger chain Rate[Hz]
Single-muon EF_mu13_tight 24Di-muon EF_2mu6 28
Single-electron EF_e15_medium 38
Di-electron EF_2e10_loose 2.4Single-photon EF_g40_loose 9
Di-photon EF_2g15_loose 2.1Single jet EF_L1J95_NoAlg 11
MET EF_xe40_noMu 6Single-tau EF_tau84_loose 6.8
Di-tau EF_2tau29_loose1 2.6Trigger Report
Run 167607 - record peak luminosity 2.1x1032cm-2s-1
For a given threshold tighten selection
Loose->medium->tightNon-isolation->isolation
Go higher in pT
Trigger evolution in 2010
L1 output 35kHz, L2 output 5kHz, EF output 400Hz
57
3
58
Other issues for the Trigger – cont’d• For details of the current ideas on ATLAS Menu evolution see
– https://twiki.cern.ch/twiki/bin/view/Atlas/TriggerPhysicsMenu• Gives details of menu since Startup and for 2011
• Corresponding information for CMS is at– https://twiki.cern.ch/twiki/bin/view/CMS/TriggerMenuDevelopment
• The expected performance of ATLAS for different physics channels (including the effect of the trigger) is documented in http://arxiv.org/abs/0901.0512 (beware - nearly 2000 pages)
59
ATLAS works!
Top-pair candidate - e-mu + 2b-tag
60
CMS works!