architectural design and supervisor roi arif mailov [email protected] new since imou 1 jena...

M. Abolins, US ATLAS Internal Review, BNL, Aug. 4, 1998.

TLASU.S. ATLAS

Architectural Design and Supervisor RoIBuilder

• T/DAQ Organization

• The Pilot Project

• Supervisor & RoI Builder


TLASU.S. ATLAS

Major T/DAQ Milestones

• LVL1 Technical Design Report June 1998

• Technical Progress Report and Work Plan June 1998 for DAQ, EF, LVL2 and DCS

• Technical Proposal on DAQ and HLT Dec. 1999

• Technical Design Report on DAQ and HLT June 2001

• LVL1 Trigger Construction Completed Dec. 2002(excluding on detector electronics)

• Stand-alone DCS System Completed Jan. 2003

• DAQ and HLT Construction Completed Dec. 2003(reduced processing power)

• Integration of detectors with T/DAQ/DCS Dec. 2004completed


TLASU.S. ATLAS

T/DAQ IB Members and ResponsibilitiesInstitute Representative Email address L1 L2 DAQ/EF DCSArgonne National Lab Robert BLAIR [email protected] 1 1Bern Klaus PRETZL [email protected] NEW SINCE IMOU 1Birmingham John GARVEY [email protected] 1Bucharest Mihai CAPRINI [email protected] 1CERN Livio MAPELLI [email protected] 1 1 1 1Copenhagen, NBI John Renner HANSEN [email protected] 1 1Cracow Zbigniew HAJDUK [email protected] 1Edinburgh Owen Boyle [email protected] NEW SINCE IMOU 1 1Geneva Allan CLARK [email protected] 1Genova Paulo MORETTINI [email protected] 1 1Heidelberg Eike-Erik KLUGE [email protected] 1Innsbruck Dietmar KUHN [email protected] 1 1Istanbul Arif MAILOV [email protected] NEW SINCE IMOU 1Jena Andreas REINSCH [email protected] 0.5 0.5JINR, Dubna Vladislav KOTOV [email protected] 1KEK Hiroyuki IWASAKI [email protected] 1 1Kobe Hiroshi TAKEDA [email protected] NEW SINCE IMOU 1Kyoto University Hiroshi SAKAMOTO [email protected] NEW SINCE IMOU 1Lecce Ornella PALAMARA [email protected] 1Lisbon Amelia TEIXEIRA MAIO [email protected] 1Liverpool Robert DOBINSON [email protected] 1 1London, QMW Eric EISENHANDLER [email protected] 1London, RHBNC John STRONG [email protected] 1 1London, UCL Peter CLARKE [email protected] 1 1Mainz Lutz KOPKE [email protected] 1 1Manchester Richard HUGHES JONES [email protected] 1 1Mannheim Reinhard MAENNER [email protected] 1 1Marseille Francois ETIENNE [email protected] 1Michigan State Maris A. ABOLINS [email protected] 1 1Moscow State Serguei SIVOKLOKOV [email protected] 1Nagasaki Yasushi NAGASAKA [email protected] NEW SINCE IMOU 1Naples Sergio PATRICELLI [email protected] NEW SINCE IMOU 1NIKHEF Joseph VERMEULEN [email protected] 1 1 1Novosibirsk Iouri MERZLIAKOV [email protected] 1Pavia Valerio VERCESI [email protected] 1Prague AS & CU Maria SMIZANSKA [email protected] 1Protvino Serguei KOPIKOV [email protected] 1RAL Charles Norman GEE [email protected] 1 1 1Rome I Emilio PETROLO [email protected] 1 1Rome II Rinaldo SANTONICO [email protected] 1Saclay Jiri BYSTRICKY [email protected] 1 1Sheffield Chris BOOTH [email protected] NEW SINCE IMOU ? 1Shinshu Tohru TAKESHITA [email protected] NEW SINCE IMOU 1St Petersburg Iouri RIABOV [email protected] 1 1Stockholm Sten HELLMAN [email protected] 1Technion Simon ROBINS [email protected] NEW SINCE IMOU 1 1Tel Aviv Halina ABRAMOWICZ [email protected] 1 1Tokyo ICEPP Tomio KOBAYASHI [email protected] NEW SINCE IMOU 1Tokyo MU Ryosuke HAMATSU [email protected] NEW SINCE IMOU 1UCI Andrew James LANKFORD [email protected] 1 1Udine Fabrizio Scuri [email protected] NEW SINCE IMOU 1Weizmann Lorne LEVINSON [email protected] 1 1Wisconsin Sau Lan WU [email protected] 1 1


TLASU.S. ATLAS

National CommitmentsLVL 1 LVL 2 DAQ / DCS to ta l

tr igger tr igger evt. filter

Armenia 0Australia 0Aust r ia 3 0 0 3 0 0Azerbaijan 0Belarus 0Brazi l 0Canada 0Czech Republic 4 0 4 0Denmark 5 0 0 5 0 0 1 0 0 0Finland 0France IN2P3 0France CEA 1 3 5 0 2 5 5 0 3 9 0 0Georgia 0Germany BMBF 2 9 1 0 1 0 2 5 7 7 0 4 7 0 5Greece 0I s r a e l 3 1 5 6 0 3 7 5I t a l y 2 6 0 0 9 0 0 2 4 0 0 5 9 0 0Japan 3 0 2 5 1 5 0 0 4 5 2 5Morocco 0Netherlands 3 2 5 3 2 5 2 0 0 8 5 0Norway 0Poland 1 5 0 1 5 0Portugal 3 0 0 3 0 0Romania 0Russia + JINR 4 5 4 5Slovak Republic 0Slovenia 0Spain 0Sweden 6 0 0 6 0 0Switzer land 4 0 0 0 4 0 0 0Turkey 1 5 0 1 5 0United Kingdom 3 7 2 0 1 1 5 0 1 0 7 0 5 9 4 0US DoE + NSF 3 2 9 0 6 7 5 3 9 6 5CERN 2 5 0 0 1 5 0 0 4 4 0 0 1 6 0 0 1 0 0 0 0

total 1 5 6 7 0 1 0 2 9 0 1 8 9 8 5 1 8 0 0 4 6 7 4 5


TLASU.S. ATLAS

Level 1 Trigger Participating Institutes

Germany Heidelberg Mainz

Italy Naples Rome I Rome II

Israel Haifa Tel Aviv Weizmann

Japan KEK Kobe Kyoto UE

Shinshu Tokyo ICEPP Tokyo MU

Sweden Stockholm

UK Birmingham London QMW RAL

CERN


TLASU.S. ATLAS

Commitments to LVL1• Calorimeter Trigger Processor (7640 kCHF, Dec. 2002)

Birmingham, London QMW, RAL (49%); Heidelberg, Mainz (38%);Stockholm (8%); not covered (5%)

• Muon Trigger Processor for RPC System (2605 kCHF, Dec. 2002)Naples, Rome I & II (100%)

• Muon Trigger Processor for TGC System (3340 kCHF, Dec. 2002)KEK, Kobe, Kyoto, Shinshu, Tokyo ICEPP & MU (91%); Haifa,Tel Aviv, Weizmann (9%)

• Muon Central Trigger Processor (675 kCHF, Dec. 2002)CERN (100%)

• Central Trigger Processor (535 kCHF, Dec. 2002)CERN (100%)

• TTC Distribution Backbone (1310 kCHF, Dec. 2002)CERN (100%)


TLASU.S. ATLAS

Level 2 Participating Institutes

Austria Innsbruck

Chech RepublicPrague AS & CU

Denmark Copenhagen

France CEA/Saclay

Germany Mannheim

Italy Genova Lecce RomeI

Israel Haifa Tel Aviv Weizmann

Netherlands NIKHEF

Poland Cracow


TLASU.S. ATLAS

LVL2 Participating Institutes Cont.

Russia Moscow SU

UK EdinburghLondon UCL

LiverpoolManchester

London RHBNCRAL

US ArgonneU of Wisconsin

UC Irvine Michigan State U

CERN


TLASU.S. ATLAS

Commitments to LVL2

• LVL2 Calorimeter Trigger (1870 kCHF, Dec. 2003)Argonne, Michigan State (45%); Mannheim (11%); Saclay (45%)

• LVL2 Muon Trigger (1290 kCHF, Dec. 2003)CERN (16%); Genova, Lecce, Rome I & II (53%); Mannheim (28%);Haifa, Tel Aviv, Weizmann (5%)

• LVL2 tracking trigger (4595 kCHF, Dec. 2003)CERN (22%); Copenhagen (11%); Cracow (3%); Mannheim (10%);London RHBNC & UCL, Manchester, RAL (18%); NIKHEF (7%);Prague AS & CU (1%); UC Irvine, Wisconsin (28%)

• LVL2 global trigger (1590 kCHF, Dec. 2003)Argonne, Michigan State (18%); CERN (19%); Genova, Lecce, Rome I(14%); Liverpool, Manchester, RAL (19%); Saclay (31%)

• LVL2 Supervisor/RoI Builder (845 kCHF, Dec. 2003)Argonne, Michigan State (100%)


TLASU.S. ATLAS

Data Acquisition and Event Filter

Austria Innsbruck

Denmark Copenhagen

France CEA Saclay

Germany Mainz

Italy Pavia Udine

Netherlands NIKHEF

Portugal

Romania Bucharest

Russia NovosibirskJINR

Protvino St. Petersburg NPI


TLASU.S. ATLAS

Data Acquisition and Event Filter cont.

Switzerland Bern Geneva

Turkey Ankara Istanbul

UK EdinburghLondon UCSheffield

LiverpoolManchester

London RHBNCRAL

US ArgonneWisconsin

UC Irvine Michigan State

CERN


TLASU.S. ATLAS

Commitments to DAQ/EF

• DAQ Readout (including readout buffers, local DAQ and interfaces to othersystems) (9275 kCHF excluding Common Project items, Dec 2003)Argonne, Michigan State, UC Irvine, Wisconsin (7%); Bern, Geneva (18%);CERN (22%); Copenhagen (5%); Edinburg, Liverpool, London RHBNC &UCL, Manchester, RAL, Sheffield (12%); Pavia, Udine (12%); Ankara,Istanbul (1%); NIKHEF (3%); Saclay (20%)

• Event Builder (3710 kCHF, Dec 2003)Bern, Geneva (32%); CERN (27%);KEK, Nagasaki (27%); Saclay (18%)

• Event Filter and Back-End DAQ (5675 kCHF excluding Common ProjectItems, Dec. 2003)Bern, Geneva (19%); CERN (25%); Pavia, Udine (23%); Innsbruck (5%);Ankara, Istanbul (1%); JINR (1%); KEK, Nagasaki (9%); Lisbon (5%); Mainz(14%); NIKHEF (1%)


TLASU.S. ATLAS

Detector Control System1720 kCHF, excluding Common Project Items, Jan. 2003

• Netherlands: NIKHEF 12%

• CERN 93%

• Russia St. Petersburg NPI manpowercontributiononly


TLASU.S. ATLAS

Pilot Project Definition

• Guidelines– 12 - 18 month program starting in March ‘98

– Coherent and complementary activities focused on main issues

– Integrate and coordinate present and future work of LVL2 group

• Inputs– URD

– Lessons from Demo Program

• Goals– Demonstrate full LVL2 functionality

– Select technology candidates

– Define architecture and prepare integration with DAQ/EF


TLASU.S. ATLAS

Critical Issues for LVL2

• Data Collection from ROB’s– RoI Mapping

– Preprocessing

• Data flow and control protocols– Data Transfer: via Supervisor or Processor

– Separate or integrated control network

– Processor allocation strategy

• Architectural Considerations– Boundary between LVL2 and EF

– Can LVL2 and EF use same network technology?


TLASU.S. ATLAS

Pilot Project Timeline

Technicalchoices

Demonstrators

Final spec.

TPR

96 97 98 99 00 01

Pilot project

High LevelTriggerPrototype &

Design phase

TP TDR

Architecturechoice

Technology candidates

DAQ-1

•Functional Components•Software testbeds•System integration

HLT

I`NTEGRATION


TLASU.S. ATLAS

Pilot Project Activities Matrix

Menus & strategiesEvaluation of:- SW options

- Performance requirements

Inputs to ROB URDMultiROB

ProcessorsCo-processors

NetworksTechnology watch

SoftwareTestbeds

FunctionalComponents

Processors, i/f’s & Networks

Pilot Project

Training samplesAlgorithm benchmarks

Framework

ROI BuilderROI distributors Paper

Computer

URDCost

New ideas

Reference Application

SystemDesign

Small Full sliceLarger commercial

Network

Modelling IntegrationSupervisor ROB Complex


TLASU.S. ATLAS

ATM Testbed Workplan• Organizer Maris Abolins, Bob Blair, Patrick Le Du

• Need A large ATM switch is needed to study system performance.

• Full functionality should be demonstrated, including

• the operation and performance of LVL2 trigger algorithms.

• Objectives Establish a 64-port ATM testbed for system performance studies

• Approach Construct a large ATM testbed with commercial components

• (PCs and VME SBCs).

• Install 16 ATM ports in a 64-port FORE switch at Saclay.

• Install a 16-port FORE switch at ANL/MSU.

• Import U.S. material to Saclay to complete a 32-port system.

• Transport to CERN to install a 32-64 port testbed

• combining components from the ATM and Ethernet testbeds.

• Duration June 98 - Dec 99

• Deliverables Saclay testbed with 16 ATM ports Jun 98

• U.S. testbed with 16 ATM ports Jun 98

• Milestones U.S. testbed operational Jun 98

• Combined 32-port testbed at Saclay Dec 98

• Combined 32-64 port testbed at CERN Mar 99

• Dependencies Availability of RD31 ATM modules and components.

• Availability of Ethernet testbed components.

• Agreement with DAQ/EF group for additional components.

• Partners ANL, CERN, MSU, RHBNC, Rome, Saclay

• Risks No risk. Commercial components.

• Status/date 27/4/98 Saclay testbed with 12 ports equipped.

• 27/4/98 U.S. testbed: FORE switch and compnonents ordered.

• 27/4/98 RD31 ATM switch available Jan-June 1999.


TLASU.S. ATLAS

Workplan to Integrate RoI Builder and Multi-supervisor

• Organizer John Dawson

• Objectives Demonstrate event routing of Level 1 information

• and CPU scaling of supervisor

• Approach Construct routing hardware using S-link input

• and implementing hardware event assembly for ROI builder.

• Measure performance versus number of supervisor CPU's

• for scaling.

• Duration June - Dec 1998

• Deliverables Hardware router for ROI builder Nov 98

• Supervisor scaling document Dec 98

• Milestones All orders placed for components Apr 98

• LVL1/LVL2 Interface Fully Specified Apr 98

• Preliminary Design of RoI Builder May 98

• US testbed assembled at ANL Jun 98

• US testbed fully operational Jul 98

• ANL testbed meeting Jul 98

• S-Link Performance Measurements Aug 98

• RoI Builder Design Complete Aug 98

• Scaling Tests of Supervisor Start Sep 98

• RoI Builder Prototype Ready Oct 98

• Supervisor Scaling tests complete Nov 98

• RoI Builder to Saclay Dec 98

• Partners ANL & MSU


TLASU.S. ATLAS

Summary of Pilot Project Milestones

• LVL1/LVL2 Interface Fully Specified

• Preliminary Design of Pilot RoI Builder Complete

• Users Requirement Document (URD) form and content defined

• Supervisor Emulator Design Ready

• Start S-Link Performance Measurements

• U.S. Pilot Testbed Operational

• Final Design of Pilot RoI Builder Complete

• Start Scaling Tests of Pilot Supervisor

• Supervisor Emulator Written

• S-Link Performance Measurements Complete

• System Design Document (SDD) for 1st baseline design ready

• New Cost Estimate Ready

• Generic models for selected test setups ready in Simdaq

• Start Integration of Reference Software

• Pilot RoI Builder Complete

• Draft 0,1 of SDD Ready

• Draft 0.1 of Design Costing Document (DCD) Ready

• Supervisor Scaling Tests Finished

• WEB document on event & control records LVL1 to LVL2 &LVL2 to LVL3 first draft ready

• 15-Apr-98

• 15-May-98

• 15-Jun-98

• 15-Jul-98

• 15-Jul-98

• 15-Jul-98

• 15-Aug-98

• 15-Sep-98

• 15-Sep-98

• 15-Sep-98

• 15-Sep-98

• 15-Sep-98

• 1-Oct-98

• 15-Oct-98

• 15-Oct-98

• 31-Oct-98

• 31-Oct-98

• 1-Nov-98

• 15-Nov-98


TLASU.S. ATLAS

Pilot Project Milestones - continued

• LVL2 TTC record and interface to LVL1 Trigger Throttle Defined

• Pilot RoI Builder & Switch shipped to Saclay

• Reference Software Integration Complete

• Pilot Ethernet SRB ships to CERN

• Details of LVL1 Throttle Specified

• SRB Interface to LVL2 and Event Filter Defined

• URD Updated

• Proposal for Common Supervisor for LVL2 and EF complete

• URD Version 1.0 Review Complete

• Final Baseline Design Complete

• Costing of Final Baseline Design Approved

• Comparison of performance of selected test setups with models complete

• Version 1.0 of SDD Ready

• Version 1.0 of DCD Ready

• Documentation of modeling of selected test setups and comparison withactual performance complete

• 30-Nov-98

• 30-Nov-98

• 15-Dec-99

• 15-Jan-99

• 1-Feb-99

• 1-Mar-99

• 1-Apr-99

• 1-Jun-99

• 15-Jun-99

• 15-Jun-99

• 30-Jun-99

• 30-Jun-99

• 1-Jul-99

• 15-Jul-99

• 1-Oct-99


TLASU.S. ATLAS

U.S. ATM Testbed


TLASU.S. ATLAS

U.S. ATM Switch and Hub


TLASU.S. ATLAS

Supervisor and RoI Builder

major contributions by

Bob Blair, John Dawson, Yuri Ermoline andJim Schlereth


TLASU.S. ATLAS

LVL1/LVL2 Layout

CPULVL2

NW i/f

CPULVL2

NW i/f

CPULVL2

NW i/f

CPULVL2

NW i/f

CPULVL2

NW i/f

CPULVL2

NW i/f

NetWork

RO

BIN

RO

BIN

RO

BIN

RO

BIN

NW

i/f

Con

trol

ler

RO

BIN

RO

BIN

RO

BIN

RO

BIN

NW

i/f

Con

trol

ler

CPUROI

NW i/f

Input Router

In FIFO

CPUROI

NW i/f

In FIFO

CPUROI

NW i/f

In FIFO

CPUROI

NW i/f

In FIFO

LVL2

i/f

Muo

nToC

TP

Inte

rfac

e

LVL2

i/f

RoI

Col

lect

or

RoI

Col

lect

or

RoI

Col

lect

or

RoI

Col

lect

or

L1A

e/γ jetsτ ETCT

P

Calorimeter Trigger

LVL1

LVL2

- R

oI B

uild

er/S

uper

viso

r

ROB ComplexROB Complex ROB Complex

LVL2 CPU Farm

LVL1 information:

• RoI generation algorithms• Trigger tower mapping to

modules/crates• RoI read-out organisation

in modules/crates• RoI address encoding


TLASU.S. ATLAS

LVL1 RoI Readout

• Assumptions

– One Readout Link per LVL1 Crate --> 12 Links

– Same links through whole system (calorimeter, muon, CTC)

– Zero-suppressed data is transferred• Questions

– Link Technology

– Data Formats

LVL1 ROI Read-out

RoI Read-out

RO

C

RoI Read-out

RO

C

RoI Read-out

RO

C

e/γ, τ jets ET,ET, µ, CTP


TLASU.S. ATLAS

LVL1 Calorimeter Data Formats

• Item: Length Comments: (bits):

• * Start of Event (Header) 32

• * L1ID 32 24 bits

• * BCID 32 12 bits

• * RoI data

• RoI Type 4 e.g. 1 = e/g RoI eta-phiindex 12 0.1 x 0.1 RoI Thresholdset 8

• * Error Word 1 32


• * End of Event (Trailer) 32

Word #

1

Content

2

3

4

N+4

N - number of RoIs

Header

Trailer

BCID

RoI data

Error Word 2

L1ID 00

00000

Error Word 1

N+5

N+6


TLASU.S. ATLAS

LVL1 Muon Data Format

• * Start of Event (Header) 32

• * L1ID 32 24 bits

• * BCID 32 12 bits

• * RoI data (16 candidatesmaximum)

• Sector number 8 224 sectors Sub-sector number 5

• Pt value 3



• * End of Event (Trailer) 32

Word #

1

Content

2

3

4

N+4

N - number of muon RoIs

Header

Trailer

BCID

RoI data

Error Word 2

L1ID 00

00000

Error Word 1

N+5

N+6


TLASU.S. ATLAS

Supervisor & RoI Builder

12 S-Linksfrom LVL1

RoI

Pro

cess

or

RoI

Pro

cess

or

RoI

Pro

cess

or

RoI

Pro

cess

or

FPGA card

In 9U VME

Add Processorsas needed

S-Link

Event is assembled in cardand fed to RoI Processorsin turn.

VME interface permitsemulation of LVL1 input.

To Network

RoI Builder


TLASU.S. ATLAS

Important Questions

• Is S-Link suitable? Tests in progress.

• Are current processors adequate for 100 kHz?

• Is the design robust and fault tolerant?

• Do we need an event size cutoff? Yes - 32 RoI’s.

• Is a “busy” from LVL2 necessary and how will itbe implemented? Yes...

• How will we test the device? Do we need a LVL1emulator? Design in progress.

architectural design and supervisor roi arif mailov [email protected] new since imou 1 jena...

Documents