lhc cms detector upgrade project us cms trigger upgrade proposal wesley smith, u. wisconsin fermilab...

LHC CMSDetectorUpgrade

Project

Fermilab Director's Review -- US CMS Trigger Upgrade 1

US CMS Trigger Upgrade Proposal

Wesley Smith, U. WisconsinFermilab Directors’ Review16 July 2013

Wesley Smith, 16 July 2013


Project

Fermilab Director's Review -- US CMS Trigger Upgrade 2Wesley Smith, 16 July 2013

L1 Trigger Upgrades: Detectors

ElectromagneticCalorimeter

Hadron Calorimeter

MuonDetectors

401.04


Project


Trigger TDR Status, Overall Goals

Overview of Trigger Upgrade Design

Muon Trigger Upgrade

Calorimeter Trigger Upgrade

Overview of Cost & Schedule


Outline


ProjectL1 Upgrade TDR

Final presentation to LHCC was Tuesday June 11th

Public version of the TDR here:https://cds.cern.ch/record/1556311

Given approval, although still awaiting final formal LHCC write-up

Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 4


Project

Trigger rates are driven by the increase in luminosity, the center-of-mass energy, & by higher Pile-Up

Detector readout systems limit Level-1 trigger output rate to 100 kHz, latency to 4 μsec. LHC beams collide every 25 ns.

Mitigate by improving e/γ isolation, τ id, μ pT resolution, μ isolation, jets with PU subtraction & L1 menu sophistication

Increase system flexibility with high bandwidth optical links and large FPGAs, and standardize on μTCA telecom standard

Build and commission upgrade in parallel with current trigger system to safeguard physics, decouple from LHC schedule

Overall design: Two-layer calorimeter trigger with tower-level precision and PU subtraction Integrated muon trigger combining all CSC, DT & RPC systems in track-finding


L1 Trigger Upgrade – overview



ProjectFinal Overall Upgrade Scheme

Fermilab Director's Review -- US CMS Trigger Upgrade 6Wesley Smith, 16 July 2013


Project


Muon Trigger Transition


Install fully paralleland higher bandwidthoptical path for CSC New CSC MPC mezzanine

cards installed in cavern during LS1

Alleviates bottleneck and send all segments from each CSC (robustness to PU and collimated signals)

Build up new Track Finder in 2015 and commission in parallel, ready by 2016

Endcap hardware design is proposed by CMS to be used for all muon regions: endcap, barrel, and overlap

Old systemNew system


Project


Muon Trigger WBS Detail


→Calorimeter

Trigger


Project


401.04.03 EMU Trigger UpgradeCavern Counting Room

μTCA: Advanced Mezzanine Cards fromTelecommunications Computing Architecture(commercial telco, see backup slide)


401.04.03.03MPC-EMUTFOptical Fibers

401.04.03.02Muon Port CardMezzanine

401.04.03.04Endcap MuonTrack-Finder

401.04.03.05EMUTF Infrastructure

401.04.03.06: Muon Sorter

Trig

ge

r M

oth

erB

oa

rd

Da

ta M

oth

erB

oa

rd

Clo

ck&

Co

ntr

ol

Sector Processors

Clo

ck &

Co

ntr

ol

Mu

on

So

rte

r


Project


401.04.03.02 Muon Port Card Upgrade(Rice)

x 60 +20 spares

Use the existing MPC main board

Backplane interface to TMB remains unchanged

3 original optical links are still available

New mezzanine card with new FPGA and new links


(60 needed + 15 spares + 5 test setups = 80)

http://bonner-ntserver.rice.edu/cms/mpcs6.jpg


Project

11

401.04.03.04: EMU Track-Finder Upgrade(U. Florida)

Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade

Muon Track Finder processor: MTF7 13 needed + 4 spares + 3 test setups = 20 Optimized for maximum input from muon detectors

(84 input links, 24 output links) Dual card with large capacity for RAM (~1GB) to be used for pT

assignment in track finding

Current prototype based on Virtex 6 FPGAis undergoing final tests (RLDRAM memory access)

Virtex 7 FPGA design is ~75% done, expected late 2013

Back: Core FPGA card with PT LUT mezzanine

Front: Optics card


Project


401.04.03.05: EMUTF Infrastructure(U. Florida)

Not US*

*Processing the resulting tracks with RPC or CSC hits is a US responsibility


401.04.02.04Endcap MuonTrack-Finder

401.04.02.06: Muon Sorter


Project


Provide immediate CSC Trigger improvement in 2015

Provide optical path to new μTCA Global Muon Trigger from old CSCTF from each endcap

Allows independent upgrade of individual endcaps

Add isolation to endcap muons from old CSCTF with a new mezzanine card on the old muon sorter to send an optical path to 2015 calorimeter trigger initial upgrade

Analogous to MPC upgrade:


401.04.03.06: Intermediate Sorter Upgrade(Rice)


Project


Upgrade Cal. Trig. Layer 1 & 2 Hardware

US:

UK:


Vienna:


Project


Calorimeter Trigger WBS Detail


←MuonTrigger

M&S paid byCMS-France


Project


Upgraded HCAL Readout & Trigger Electronics (μHTR) is after split that sends data to old electronics (HTR)

μHTR supplies TP to upgrade cal. Trigger, HTR continues to send to present cal. Trigger

ECAL Trigger Concentrator Cards (TCC) are not upgraded, instead their mezzanines with trigger serial links are replaced (built by Lisbon):

Optical Serial Link Board oSLB replaces single copper link to current Regional Calorimeter Trigger (RCT – U. Wisc.) with two optical links (WBS 401.04.04.06)

One optical link to optical Receiver Module (oRM) on RCT (WBS 401.04.04.05)o Replaces copper Receiver Module

One optical link to input of Upgraded Calorimeter Trigger: CTP7 Card

574 oSLB’s & oRM’s with Fibers installed in 2014 French Contribution to Calorimeter Trigger (no US Upgrade Funds)

Presently oSLB & oRM in final prototype stage oSLB:

oRM:


HCAL & ECAL Trigger Primitives


Project


Calo. Trigger Upgrade in Parallel: Split inputs from ECAL & HCAL

HCALEnergy(HTR)

ECALEnergy(TCC)

Regional Calo Trigger

Global Calo Trigger

EMcandidates

Regionenergies

HFEnergy(μHTR)

HCALEnergy(μHTR)

Layer 1Calo Trigger

Layer 2 Calo TriggerC

urre

nt L

1 T

rigge

r S

yste

m

Upg

rade

L1

Trig

ger

Sys

tem

oSLB

oRM

Install optical SLB and optical RM mezzanines during LS1 Install HCAL optical splitters during LS1 or YETS Install HCAL backend μHTR cards for input to new trigger Install HCAL frontend electronics after LS2 (finer long. segmentation)


US

UK

US

UK

HCAL OpticalSplitters

Stage HCAL uHTR & assoc. Cal. Trigger Cards


Project


401.04.04.02: Calorimeter Trigger ProcessorVirtex-6 Prototype Board (CTP-6) – U. Wisconsin

Back End FPGAXC6VHX250T/XC6VHX380T

Front End FPGAXC6VHX250T/XC6VHX380T

Avago AFBR-810B Tx Module

4X Avago AFBR-820B Rx Module

MMC Circuitry

JTAG/USB ConsoleInterface Mezzanine

Power Modules

Dual SDRAM for dedicated DAQ and TCP/IP buffering

12x Multi Gig Backplane Connections


36 total + 8 spares + 2 test setups = 46


Project


401.04.04.02: Calorimeter Trigger Processor Final Version: Virtex-7 (CTP-7) – U. Wisconsin

• Replace 2 Virtex-6s with a Virtex 7 for processing+ZYNQ for embedded TCP/IP endpoint

• 30A, 1V power module for FPGA logic core

• 3x CXP Pluggable modules for 36 Tx + 36 Rx 10G optical links

• 2x AFBR-820 modules for 24 Rx 10G optical links

• Simpler design to execute than the CTP-6

• 36 total + 8 spares + 2 test setups = 46

Virtex-7 VX690T FPGA

ZYNQXC7Z03

0EPP

1.5V

S

up

ply

2.5V

S

up

ply

3.3V

S

up

ply

1V

30A

S

up

ply CXP Module

12Tx + 12 Rx

CXP Module12Tx + 12Rx

CXP Module12Tx + 12 Rx

12XRx

12XRx

(CTP-6 CAD View)



Project


401.04.04.03: Crosspoint IO Card: provides crate interconnections – U. Wisconsin

Controller (MMC and link mgmt)

4X Avago AFBR-79EQDZ QSFP+ ModulePositions

4x4 Lane Bidirectional Multi Gig Backplane Connections

Backplane Rx/Tx

Redriver ICs (top and

bottom sides)


(2/crate x 3 crates = 6 + spares)


Project


401.04.04: CTP Infrastructure:Vadatech VT894 Crate Test Setup

BU

AM

C13

Vad

atec

h

MC

H UW

CT

P-6

UW

CT

P-6

TTC Downlink

UW

Au

x


(Final system: 3 crates w/ 12 CTP7 ea. + 2 test setups + spare = 6)

U. Wisconsin designed backplane with dense card interconnectsmanufactured & installed in commercial Vadatech VT892 Crate available in Vadatech Catalog


Project


Goal (Original): Provide readout of original RCT descoped during construction project Present readout through GCT input buffer not workable with trigger evolution Uses connection to a single CTP6 prototype or CTP7 card for DAQ readout

optical Receiver Summary Card (oRSC) paid by DOE Nuclear VME Slave Interface Card Fits in current RCT Crates (1 per crate) 18 Cards in 2015 System Receives RCT Jet Sum Card Output

to GCT on Copper “SCSI” Data Cables Provides direct optical input to GCT

o Bypassing old optical conversion cards Planned use for Heavy Ion Triggers Prototype under test

Multiple optical outputs provide: Inputs for upgrade calorimeter trigger &

parallel operation of old & new trigger wherever ECAL & HCAL electronics notavailable in 2015


Calorimeter Trigger Risk Mitigation


Project

Fermilab Director's Review -- US CMS Trigger Upgrade

Evolution: Partial Upgrade in 2015

Wesley Smith, 16 July 2013 23

Layer 2 Calo Trigger

HCALenergy

Regional Calo Trigger

Global Calo Trigger

HFenergy

oRSC

Layer 2Calo Trigger

oRM

ECALenergy

Layer 1 Processors

oSLB ECAL & HF ClustersHalf-tower position

4x4 E+H Clusters2x1 E+H Clusters

2-bit region ID

Layer 2 Processors

MuonJets & Sums

eGammaTau

Heavy IonSpare

RCT RO + Testing

GCT fallback and Minimal Stage-1 fallback remains

available

With just a fraction of final cards or even prototypes, derive many benefits

of full upgrade, incl. muon isolation

Uses reprogrammed RCT clusters with improved algorithms in Layer-2 Also, brings in all the power of finer grain HF using “Slice Test” of Layer-1 & 2

Improved PU subtraction, Isolation Calculation (e/γ/τ/μ) & Half-tower Position Resolution

US Items: oRSCs, oSLB/oRM commissioning CTP Layer 1 Processors RCT RO+Testing CTP

UK Items: MP7 Layer 2 Processors

36 mHTRs

2 CTP7s 9 CTP7s

Add to bring in finer grain EM clusters

2 CTP7s

4 MP7s


Project


CTP7 M&S: no HB/HE Backend until 2016 Delay purchase of 34 CTP7 cards until FY15 Use 12 CTP7 Cards w/prototypes as additional spares & for test setups. Connected as shown on previous slide

EMU CSCTF M&S: one endcap in 2015 & other in 2016 Commission & test one endcap thoroughly so 2nd can be integrated

quickly Delay spares & use prototypes for spares, test setups Compatible with Global Muon Trigger using different Endcap inputs

Stages and Content:


Trigger Staging


Project


Working prototypes for all major cards exist & have passed tests Production quotes exist for all critical parts with confirmed delivery upon order

Upgrade is based upon common μTCA hardware platform also used by HCAL & other CMS systems

Common hardware components across subsystems: Virtex-7 FPGA, multi-gigabit optical links

Many handles to facilitate testing – test-pattern injection, spy-buffer readout as well as test stands

Upgrade trigger project replaces existing trigger hardware built by the same team → interfaces well understood

Same team wrote software & firmware for existing trigger system → scope & requirements well understood

Requirements are frozen in approved Trigger TDR

System is commissioned & operated in parallel with existing triggerWesley Smith, 16 July 2013

Technical Feasibility


Project


Trigger L2 Manager: Wesley Smith

Deputies:Darin Acosta,

Sergo Jindariani

Muon Trigger L3 Manager:

Darin Acosta

Muon Port Card & Sorter:

Paul Padley

Muon Track-Finder:Ivan Furic

Calorimeter Trigger L3 Manager:

Sridhara Dasu

CERN Integration:Pam Klabbers


Trigger Project Organization


Project


W.S. (U. Wisconsin) CMS Trigger Project Manager 1994-2007, Trigger Coordinator 2007 – 2012 US CMS L2 Trigger Project Manager 1998 – present

Darin Acosta (U. Florida) CMS Trigger Project Manager 2012-13 EMU Track-Finder, US CMS Trigger, 1998- present

Sridhara Dasu (U. Wisconsin) US CMS L3 Manager for Calorimeter Trigger, 1998 – present Author of original & upgrade cal. trig. Algorithms 1994 – present

Pam Klabbers (U. Wisconsin) CMS Regional Calorimeter Trigger Operations Manager (more than a decade on RCT project) CMS Deputy Trigger Technical Coordinator

Ivan Furic (U. Florida) CMS CSC muon trigger maintenance & operations 2010-11 CMS CSCTF Upgrade leader

Paul Padley (Rice U.) US CMS EMU Project Manager 2006-2012 EMU Port Card, Clock Card, Sorter, 1998 – presentWesley Smith, 16 July 2013

Trigger Management CMS Experience


Project


Tom Gorski (U. Wisconsin) – Calorimeter Trigger Over a decade of engineering on the CMS Calorimeter Trigger

Alex Madorsky (U. Florida) – Muon Track-Finder Over a decade of engineering on CMS Trigger, EMU, CSCTF

Mike Matveev (Rice U.) – Muon Port Cards & Sorter Over a decade of engineering on CMS Trigger, EMU, Port Cards,

Sorter

Mathias Blake (U. Wisconsin) – Cal. Trig. Firmware New to project 5 years Applications Engineer at Xilinx, 2 years FPGA Engineer at DRS

Technologies, 3 years FPGA Design Engineer at Dolby Labs


CMS Trigger Engineering Experience


Project


M&S: Parts based on quotes & prototype purchases Ordering times based on experience with prototype orders Board cost & delivery times based on prototype boards Assembly cost & delivery times based on prototype boards

EDIA: Labor Hours based on experience with corresponding tasks done

for prototypes and on experience in delivering the original CMS trigger system (done by the same people & groups).

Labor costs based on actual labor rates of all of the individuals who are now working on the project

Contingency: Follows US CMS Contingency rules


C&S Methodology


Project

30Fermilab Director's Review -- US CMS Trigger Upgrade

Base Cost & Contingency in AY K$

Fermilab Labor M & S Univ. Labor Contingency

Base Cost Contingency Base Cost Contingency Base Cost Contingency $% on

base costTPC

401.04.02 Trigger Management 371 0% 371

401.04.03 MUON Trigger 1,242 391 1,524 604 995 36% 3,761

401.04.04 Calorimeter Trigger 1,251 355 1,113 445 800 34% 3,164

Grand Total 2,865 746 2,637 1,049 1,795 33% 7,297


401.04 Cost Estimate with Contingency


Project


Schedule/Milestones


The project completes at the end of FY16

FY14: Launch production of trigger electronics 1QFY14 Deliver on-detector muon trigger electronics to CERN: 3QFY14 Deliver Stage 1 calorimeter layer 1 & muon track-finder electronics to CERN:

4QFY14

FY15: Test operation of trigger calorimeter and muon trigger electronics*: 2QFY15

(*except muon sorter) Deliver Stage 2 calorimeter layer 1 & muon track-finder electronics to CERN:

2QFY15 Deliver Muon Sorter electronics to CERN: 3QFY15 Test Operation of Muon Sorter electronics: 4QFY15

FY16: Test high luminosity capability of endcap muon trigger electronics: 4QFY16 Test high luminosity capability of calorimeter trigger electronics: 4QFY16


Project


Base Cost = AY $K(No Contingency)

FY13 FY14 FY15 FY16 FY17 FY18 FY19

University Labor $ K $798 K $908 K $931 K $ K $ K $ K

Material & Services $22 K $1,384 K $1,174 K $285 K $ K $ K $ K

Fermilab Labor $ K $ K $ K $ K $ K $ K $ K

Cumulative Total $22 K $2,204 K $4,286 K $5,502 K $5,502 K $5,502 K $5,502 K

1,000

2,000

3,000

4,000

5,000

6,000

500

1,000

1,500

2,000

2,500

Cu

mu

lativ

e C

osts

K$

An

nu

al C

ost

s K

$

University Labor

Material & Services

Fermilab Labor

Cumulative Total


401.04: Trigger Resource Profile


Project


Senior Engineer becomes unavailable Hire new engineer, subcontract to consulting firm, use FNAL engineer

Funding is delayed Commission with prototypes and/or fewer production boards

Software or Firmware does not meet requirements Hire extra expert effort to recover schedule and help personnel

Boards are delayed (design, manufacture or testing) Hire extra effort to speed up testing schedule

Vendor non-performance Acquire spending authority to use alternative vendors (while original funds are being

unencumbered).

Input or output electronics (non-trigger) delayed Built in capabilities of trigger electronics provide signals for their own inputs & outputs

Overall Risk Mitigation Strategy: Continue to provide fully operational current trigger system in parallel with upgrade

commissioning Partial operation of the upgrade systems provide tangible benefits from 2015 onwardsWesley Smith, 16 July 2013

Overall Trigger Risks


Project


Safety: follows procedures in CMS-doc-11587, FESHM L2 Manager (W.S.) responsible for applying ISM to trigger upgrade.

o Under direction of US CMS Project Management Fermilab employees and CMS collaborators working at Fermilab & CERN receive

training in accordance with Fermilab & CERN policies. Personnel conducting work at collaborating institutions are trained in accordance with the policies of those institutions.

Regular US CMS Upgrade Mgm’t meetings review work activity, schedule, costs, hazards, risks and controls.

Quality Assurance: follows procedures in CMS-doc-11584 Regularly evaluate achievement relative to performance requirements & appropriately

validate or update performance requirements & expectations to ensure quality. QA: Equipment inspections & verifications; Software code inspections, verifications, &

validations; Design reviews; Baseline change reviews; Work planning; and Self-assessments.

Graded Approach: Apply appropriate level of analysis, controls, and documentation commensurate with

the potential to have an environmental, safety, health, radiological, or quality impact. Four ESH&Q Risk levels are defined & documented in CMS-doc-11584.


Trigger ESH&Q


Project


C&S understood since prototypes for all major cards have been built & have passed tests

C&S based on experience of the same team that built original CMS calorimeter & muon trigger systems

Hardware is based upon common μTCA hardware platform also used by HCAL & other CMS systems

Same team wrote software & firmware for existing trigger system → scope & requirements well understood

Requirements are frozen in approved Trigger TDR

Schedule is robust because it always provides a working trigger at each stage with parallel operation of old & new trigger systems, commissioning & testing of new system during data-taking (with data) & evolution to final system while providing immediate availability of improved trigger in 2015Wesley Smith, 16 July 2013

Cost & Schedule Understanding


Project


Develop a plan prior to the CD-1 review for procurement of components that need to be installed prior to the end of LS1.

These components are the Muon Port Card Mezzanines and MPC-EMUTF Optical Fibers that must be installed in the collision hall during LS1.

These components are being moved to the operations program so they can be purchased in 2013.

Incorporate the missing information identified within comment section into the CDR prior to the CD-1 review.

This has been done and the CDR has been revised


Response to Recommendationsfrom May Review


Project


Trigger Upgrade is technically feasible since based on working prototypes, & built by the same team that built original CMS Calorimeter & Muon triggers.

Trigger Upgrade can be accomplished within Cost & Schedule since based on working prototypes and experience in building the original CMS Calorimeter & Muon triggers.

Risks are understood and familiar due to team experience – risk is mitigated by parallel operation of existing and upgraded calorimeter systems

Plan to complete prototyping and begin production when approved.Wesley Smith, 16 July 2013

Conclusions


Project


BACKUP



ProjectCalorimeter Trigger Components


Type Name Purpose

TCC Trigger Concentrator Card Existing ECAL trigger primitives (TP) (FR)

oSLB Optical Serial Link Board Transmit ECAL TPs to RCT & CTP (PT)

mHTR HCAL Trigger & DAQ Board Create and transmit HCAL TPs

RCT Regional Calorimeter Trigger Existing trigger system

oRM Optical Receiver Modules Receive ECAL TPs on RCT

oRSC Optical Regional Summary Card Convert and transmit RCT output

CTP Compact Trigger Processor Large FPGA w/ optical & backplane

CIOX Compact IO cross-point switch Intra-crate sharing switch

MP7 Multipurpose Processor Large FPGA w/ only optical links (UK)


ProjectTMT Calorimeter Trigger Upgrade


Time-multiplexed architecture, calo data from an entire event processed by a single processor


Project


Crate GUp to 12TriggerPath

Modules

Trig Path 1(MP7)

2x288-fiber patch panel (2x24×12-fiber optical ribbons in & out)

Crate B Crate C

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CIO

-U

CIO

-L

6 x 8-fiber ribbons for Lateral Network(@ 4.8 Gbps)

Trig Path 2(MP7)

Trig Path 3(MP7)

Trig Path 4(MP7)

Trig Path 5(MP7)

Trig Path 6(MP7)

HCALECALHCALECAL HCALECAL

Each 1st-Level CTP7 Drives 2 fibers out to 6-12 Trigger Paths @ 9.6 Gbps

Each Trigger Path MP7 rcvs 48 total 1st-Level fibers @ 9.6 Gbps (2/region)

To GT To GT

CT

P

CT

P

CT

P

CT

P

CIO

-L

CT

P

CT

P

CIO

-U

CIO

-L

CIO

-U

Crate A

Crate G

Pipelined Calorimeter Trigger Upgrade

Calorimeter Trigger Processor (CTP) CardCrosspoint I/O (CIO) Card


Each Layer 2 processor calculates suite of trigger paths

Layer 1 sends dedicated clusters to Layer 2

lhc cms detector upgrade project us cms trigger upgrade proposal wesley smith, u. wisconsin fermilab...

Documents

us cms trigger upgrade7install

current trigger system

layer calorimeter trigger

trigger output rate

commission upgrade

lhc beams

lhc scheduleoverall

muon regions