btev availability issues we don’t have problems any more, we have issues... 21 aug 2001 m. haney,...

BTeV Availability Issues

We don’t have problems any more,we have issues...

21 Aug 2001

M. Haney, University of Illinois

m-haney@uiuc.edu 21aug01

2

Early on, high energy physicists found too many particles...

’

a0

f0

a1

h0

h1

f’2

f1

a2

f2

fj

a4

f4

K*

K

K1

K2

K*2

K3

K*3

K*4

Ds

D1

D*2

D*

D

D2B*

B

Bs

Bc

c

c

“the secret is to bang the rocks together…” Douglas Adams

m-haney@uiuc.edu 21aug01

3

which led to the Standard Model:

quarks & leptons

quark charge

up

down

charm

strange

top

bottom

+ 2/3

- 1/3

-1

0

e

e

lepton charge

(truth)

(beauty)

m-haney@uiuc.edu 21aug01

4

quarks & anti-quarks

proton neutron

+ D0 Bs

u

d

c

s

t

b

+ 2/3

- 1/3

- 2/3

+ 1/3

u

d

c

s

t

b

Now we can easily build any ofof the particles we have discovered(and predict some we haven't)

(baryons)

(mesons)

anti-proton

Rule: Quarks cannot exist alone,but only in 2’s and 3’s

m-haney@uiuc.edu 21aug01

5

How to build a Particle Accelerator

Protons are also quite easy: Ionize hydrogen

+

e +

e

+

e

+

e

+e+

e

+

e

+

e

ENERGIZER

-+

Start with some particles (usually electrons or protons)

Electrons are easy: Just heat up a filament ee

eeeee eeeeeee ee

ENERGIZER

+-

m-haney@uiuc.edu 21aug01

6

• if you match the frequency correctly,

• the particle will “surf” the RF wave...

Linear Accelerator (concept)

+

-+

+

-

+

-

m-haney@uiuc.edu 21aug01

7

Linear Accelerator (practice)

• LINAC at Fermilab(0.4 GeV)

m-haney@uiuc.edu 21aug01

8

Vacuum tube (beam pipe)

Synchrotrons

bending magnet

Accelerating section

Focussingmagnets

m-haney@uiuc.edu 21aug01

9

Fermi National Accelerator Lab

• The ring is about 4 miles around...

m-haney@uiuc.edu 21aug01

10

Two for the price of one

1000 GeV-1000 GeV +

2000 GeV

• accelerate protons andantiprotons– same ring– opposite

charge,+ direction

• bang themtogether...

m-haney@uiuc.edu 21aug01

11

4 Detector• The typical arrangement

is to wrap the detectoraround the interactionpoint:

CDF at Fermivertex(later…)

m-haney@uiuc.edu 21aug01

12

BTeV is rather different...

protons antiprotons

m-haney@uiuc.edu 21aug01

13

Too much physics!

• Let’s talk about computational requirements:– every 132ns, two “clouds of quarks”

cross paths...– on the average, there will be 2 “events”

(collisions)– 200 KBytes of data are produced per event

m-haney@uiuc.edu 21aug01

14

BTeV Data Path

About 1 TByte of (distributed) buffer memory...

m-haney@uiuc.edu 21aug01

15

Why have a Trigger?• Compression is not enough

– To get from 1.5 TBytes/sec from the detector,– to 200 MBytes/sec to tape, you need to:

• identify/reject “boring” events• identify/accept/save-to-tape “interesting” events• keep extremely good records on every action!

– All of the analysis is statistical– Bad statistics = Mission failure

– Losing an event is survivable• Not knowing that you lost it, or why, is death

m-haney@uiuc.edu 21aug01

16

The BTeV Pixel Trigger• (there is also a Muon Trigger;

there may be others...)– 2 KBytes/crossing (from Pixel Detector)

• every 132ns...– Fast, pipelined FPGA(s) to sort,

match up pairs and triplets between planes– DSP(s) to find tracks, compute vertex(s)

• time available– <1 ms, from crossing to trigger “opinion”

m-haney@uiuc.edu 21aug01

17

Muon DSP Farm

L1Buf (basis)

L1Buf (basis)

Muon Preprocessor

Segment Preprocessor L1Buf (cooked)

MTSM

Detector

Front End Board (DCB)

Crossing Switch

Pixel Preprocessor(FPGAs)

Segment Preprocessor(FPGAs)

Pixel DSP Farm(joint track+vertex)

PTSM

RegionalControl

andMonitoring

BTeV Run Control

GL1

L2/L3(Linux Farm)

Opinions

Requested/AssignedCrossing data

1 Highway

GLSM

L1Buf

L1Buf (raw)

L1Buf (basis)

L1Buf (details)

L1Buf (details)

L1Buf (basis)

Accept/Reject Decisions

Resource Mgr

/c/spot

Others...

m-haney@uiuc.edu 21aug01

18

Trigger Path Issues• direct (optical) link, front-end to FPGA-part

– geographic specificity• Many-to-many interconnect, from FPGAs to DSPs

– work assignments handed out, one crossing to each DSP

– all FPGAs contribute to the “one crossing”• hence many-to-(one-of)-many

– of)-many ~ 2500 DSPs (!)

– no inter-DSP connection (needed…)

m-haney@uiuc.edu 21aug01

19

Trigger Path Issues (page 2)

• DSPs to GL1– each DSP offers one “opinion” per crossing

• multiple triggers = multiple opinions– every 132ns

• GL1 decides– accept/reject

• based on the opinions received for that crossing– 7.6 million decisions per second

m-haney@uiuc.edu 21aug01

20

Trigger Path Issues (page 3)

• Resource Manager– conveys GL1 decisions to L1 Buffers,

• “push” (as opposed to “pull”) architecture

• L1 Buffers deliver data to appointed L2 processor– Pentium-class Linux machine(s)

• again, many-to-(one-of)-many– of)-many ~ 2000 Linux machines

m-haney@uiuc.edu 21aug01

21

Trigger Path Issues (page last)

• L2 Processor is L3 Processor– sufficiently interesting events

receive additional (L3) processing– events that satisfy L3 are recorded to “tape”

• or saved on disk, or burned to CD, or ...

m-haney@uiuc.edu 21aug01

22

Muon DSP Farm

L1Buf (basis)

L1Buf (basis)

Muon Preprocessor

Segment Preprocessor L1Buf (cooked)

MTSM

Detector

Front End Board (DCB)

Crossing Switch

Pixel Preprocessor(FPGAs)

Segment Preprocessor(FPGAs)

Pixel DSP Farm(joint track+vertex)

PTSM

RegionalControl

andMonitoring

BTeV Run Control

GL1

L2/L3(Linux Farm)

Opinions

Requested/AssignedCrossing data

1 Highway

GLSM

L1Buf

L1Buf (raw)

L1Buf (basis)

L1Buf (details)

L1Buf (details)

L1Buf (basis)

Accept/Reject Decisions

Resource Mgr

/c/spot

Others...

m-haney@uiuc.edu 21aug01

23

Pixel (Muon) Trigger Supervisor Monitor (P/M)TSM

• Control Functions:– Initialization

• sets the maximum bandwidth necessary– can not take all day...

– Command Parsing and Distribution• to subordinates, from RunControl

– Error Response • autonomous, “local” (regional)• not to be confused with higher-level driven Error

Handling, which appear as “Commands”

m-haney@uiuc.edu 21aug01

24

(P/M)TSM (continued)

• Monitor Functions– Error Message Collection and Reporting

• organized, formatted, sent higher up– Hardware and Software Status

• not unlike Error Messages...– Status and Data Histogramming

• utilizes remaining (P/M)TSM system bandwidth

m-haney@uiuc.edu 21aug01

25

EtherNet,for example

ARCNet,for example

Close-up: FPGATrigger Supervisor and Monitor

Preprocessorfor example...

_TSM

Regional Controland

Monitoring

BTeV Run Controland database(s), etc.

_TSM

Local Controland

Monitoring

FPGA

JTAG,programming

and debug/c/spot run

Local Config(Flash)

Regional copyof config data,

and history

FPGA

StandaloneOperationalCapability

ControlInfluence

MonitoringAwareness

Fire DetectPower, Cooling Monitoringand Control

m-haney@uiuc.edu 21aug01

26

EtherNet,for example

ARCNet,for example

Close-up: DSPTrigger Supervisor and Monitor

DSP

_TSM

Regional Controland

Monitoring

BTeV Run Controland database(s), etc.

_TSM

Local Controland

Monitoring

JTAG,programming,

debug, and monitoringrun(spot);

run;

Local Config(Flash)

Regional copyof config data,

and history

DSP

StandaloneOperationalCapability

ControlInfluence

MonitoringAwareness

Fire DetectPower, Cooling Monitoringand Control

Host PortInterface

DMAin

BSPout

FPGA

m-haney@uiuc.edu 21aug01

27

Factoids• TMS320C67x DSP

– ~70 Kbyte internal RAM, ~1200 MIP– fixed/floating point

• TMS320C64x DSP– ~1 Mbyte internal RAM, ~3x faster… – fixed point only

• times 2500 devices (!)– Sony Playstation 2.5K ?

m-haney@uiuc.edu 21aug01

28

More Factoids• Host Port Interface (TI DSP only)

– almost-direct access into the DSP• peek, poke• uses DMA (like) resources…

– (concept not unique to TI)• DMA

– crossing data in• Buffered Serial Port(s)

– opinions out; dual, ~75Mbps (C6x)

m-haney@uiuc.edu 21aug01

29

DSP/BIOS (Texas Instruments)• based on SPOX

– “the oldest DSP RTOS…”• scalable real-time kernel

– small footprint (< 2Kw)– preemptive multithreading

• scheduling (tasks, soft/hard interrupts)• synchronization (between tasks, interrupts)

– hardware abstraction– extensible

m-haney@uiuc.edu 21aug01

30

DSP/BIOS (continued)

– real-time analysis• real-time instrumentation

– explicit API calls– (controllably) implicit, at context changes

• host-target communications– statistics gathering

• same as above– host data channels

• binds kernel I/O objects to host files

m-haney@uiuc.edu 21aug01

31

RTDX - Real Time Data Exchange

• utilizes JTAG chain (and emulation bits)• target (kernel) component

– moves messages to/from DSP/BIOS queues from/to JTAG-accessible buffer

– “real time” target to host (?)• host component

– data visualization and analysis tools– data to target “not” real-time… (?)

m-haney@uiuc.edu 21aug01

32

RTDX (continued)• fundamental (TI) model

– one PC running CCS (or app with CCS API calls)

– small number (4’ish) of DSPs on one JTAG chain

– JTAG/emulator “pod” connects PC to chain• (2 “extra” emulation bits, in addition to

TDI/TDO/TMS/TCLK)

• BTeV challenge– not buying 500 PC’s to support 2000 DSPs…

m-haney@uiuc.edu 21aug01

33

BTeV Conclusion(s)• fierce data rates• massive parallelism• heterogeneous control-in-depth

• limited loss-of-data (or performance)– tolerable

• any loss of understanding– unacceptable

m-haney@uiuc.edu 21aug01

34

BTeV TimeTable• detailed technical design report

– required Feb 2002• R&D for TDR, now

• to be operational in 2006

• http://www-btev.fnal.gov/btev.html