![Page 1: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/1.jpg)
BTeV Availability Issues
We don’t have problems any more,we have issues...
21 Aug 2001
M. Haney, University of Illinois
![Page 2: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/2.jpg)
[email protected] 21aug01
2
Early on, high energy physicists found too many particles...
’
a0
f0
a1
h0
h1
f’2
f1
a2
f2
fj
a4
f4
K*
K
K1
K2
K*2
K3
K*3
K*4
Ds
D1
D*2
D*
D
D2B*
B
Bs
Bc
c
c
“the secret is to bang the rocks together…” Douglas Adams
![Page 3: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/3.jpg)
[email protected] 21aug01
3
which led to the Standard Model:
quarks & leptons
quark charge
up
down
charm
strange
top
bottom
+ 2/3
- 1/3
-1
0
e
e
lepton charge
(truth)
(beauty)
![Page 4: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/4.jpg)
[email protected] 21aug01
4
quarks & anti-quarks
proton neutron
+ D0 Bs
u
d
c
s
t
b
+ 2/3
- 1/3
- 2/3
+ 1/3
u
d
c
s
t
b
Now we can easily build any ofof the particles we have discovered(and predict some we haven't)
(baryons)
(mesons)
anti-proton
Rule: Quarks cannot exist alone,but only in 2’s and 3’s
![Page 5: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/5.jpg)
[email protected] 21aug01
5
How to build a Particle Accelerator
Protons are also quite easy: Ionize hydrogen
+
e +
e
+
e
+
e
+e+
e
+
e
+
e
ENERGIZER
-+
Start with some particles (usually electrons or protons)
Electrons are easy: Just heat up a filament ee
eeeee eeeeeee ee
ENERGIZER
+-
![Page 6: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/6.jpg)
[email protected] 21aug01
6
• if you match the frequency correctly,
• the particle will “surf” the RF wave...
Linear Accelerator (concept)
+
-+
+
-
+
-
![Page 8: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/8.jpg)
[email protected] 21aug01
8
Vacuum tube (beam pipe)
Synchrotrons
bending magnet
Accelerating section
Focussingmagnets
![Page 10: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/10.jpg)
[email protected] 21aug01
10
Two for the price of one
1000 GeV-1000 GeV +
2000 GeV
• accelerate protons andantiprotons– same ring– opposite
charge,+ direction
• bang themtogether...
![Page 11: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/11.jpg)
[email protected] 21aug01
11
4 Detector• The typical arrangement
is to wrap the detectoraround the interactionpoint:
CDF at Fermivertex(later…)
![Page 13: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/13.jpg)
[email protected] 21aug01
13
Too much physics!
• Let’s talk about computational requirements:– every 132ns, two “clouds of quarks”
cross paths...– on the average, there will be 2 “events”
(collisions)– 200 KBytes of data are produced per event
![Page 15: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/15.jpg)
[email protected] 21aug01
15
Why have a Trigger?• Compression is not enough
– To get from 1.5 TBytes/sec from the detector,– to 200 MBytes/sec to tape, you need to:
• identify/reject “boring” events• identify/accept/save-to-tape “interesting” events• keep extremely good records on every action!
– All of the analysis is statistical– Bad statistics = Mission failure
– Losing an event is survivable• Not knowing that you lost it, or why, is death
![Page 16: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/16.jpg)
[email protected] 21aug01
16
The BTeV Pixel Trigger• (there is also a Muon Trigger;
there may be others...)– 2 KBytes/crossing (from Pixel Detector)
• every 132ns...– Fast, pipelined FPGA(s) to sort,
match up pairs and triplets between planes– DSP(s) to find tracks, compute vertex(s)
• time available– <1 ms, from crossing to trigger “opinion”
![Page 17: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/17.jpg)
[email protected] 21aug01
17
Muon DSP Farm
L1Buf (basis)
L1Buf (basis)
Muon Preprocessor
Segment Preprocessor L1Buf (cooked)
MTSM
Detector
Front End Board (DCB)
Crossing Switch
Pixel Preprocessor(FPGAs)
Segment Preprocessor(FPGAs)
Pixel DSP Farm(joint track+vertex)
PTSM
RegionalControl
andMonitoring
BTeV Run Control
GL1
L2/L3(Linux Farm)
Opinions
Requested/AssignedCrossing data
1 Highway
GLSM
L1Buf
L1Buf (raw)
L1Buf (basis)
L1Buf (details)
L1Buf (details)
L1Buf (basis)
Accept/Reject Decisions
Resource Mgr
/c/spot
Others...
![Page 18: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/18.jpg)
[email protected] 21aug01
18
Trigger Path Issues• direct (optical) link, front-end to FPGA-part
– geographic specificity• Many-to-many interconnect, from FPGAs to DSPs
– work assignments handed out, one crossing to each DSP
– all FPGAs contribute to the “one crossing”• hence many-to-(one-of)-many
– of)-many ~ 2500 DSPs (!)
– no inter-DSP connection (needed…)
![Page 19: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/19.jpg)
[email protected] 21aug01
19
Trigger Path Issues (page 2)
• DSPs to GL1– each DSP offers one “opinion” per crossing
• multiple triggers = multiple opinions– every 132ns
• GL1 decides– accept/reject
• based on the opinions received for that crossing– 7.6 million decisions per second
![Page 20: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/20.jpg)
[email protected] 21aug01
20
Trigger Path Issues (page 3)
• Resource Manager– conveys GL1 decisions to L1 Buffers,
• “push” (as opposed to “pull”) architecture
• L1 Buffers deliver data to appointed L2 processor– Pentium-class Linux machine(s)
• again, many-to-(one-of)-many– of)-many ~ 2000 Linux machines
![Page 21: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/21.jpg)
[email protected] 21aug01
21
Trigger Path Issues (page last)
• L2 Processor is L3 Processor– sufficiently interesting events
receive additional (L3) processing– events that satisfy L3 are recorded to “tape”
• or saved on disk, or burned to CD, or ...
![Page 22: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/22.jpg)
[email protected] 21aug01
22
Muon DSP Farm
L1Buf (basis)
L1Buf (basis)
Muon Preprocessor
Segment Preprocessor L1Buf (cooked)
MTSM
Detector
Front End Board (DCB)
Crossing Switch
Pixel Preprocessor(FPGAs)
Segment Preprocessor(FPGAs)
Pixel DSP Farm(joint track+vertex)
PTSM
RegionalControl
andMonitoring
BTeV Run Control
GL1
L2/L3(Linux Farm)
Opinions
Requested/AssignedCrossing data
1 Highway
GLSM
L1Buf
L1Buf (raw)
L1Buf (basis)
L1Buf (details)
L1Buf (details)
L1Buf (basis)
Accept/Reject Decisions
Resource Mgr
/c/spot
Others...
![Page 23: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/23.jpg)
[email protected] 21aug01
23
Pixel (Muon) Trigger Supervisor Monitor (P/M)TSM
• Control Functions:– Initialization
• sets the maximum bandwidth necessary– can not take all day...
– Command Parsing and Distribution• to subordinates, from RunControl
– Error Response • autonomous, “local” (regional)• not to be confused with higher-level driven Error
Handling, which appear as “Commands”
![Page 24: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/24.jpg)
[email protected] 21aug01
24
(P/M)TSM (continued)
• Monitor Functions– Error Message Collection and Reporting
• organized, formatted, sent higher up– Hardware and Software Status
• not unlike Error Messages...– Status and Data Histogramming
• utilizes remaining (P/M)TSM system bandwidth
![Page 25: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/25.jpg)
[email protected] 21aug01
25
EtherNet,for example
ARCNet,for example
Close-up: FPGATrigger Supervisor and Monitor
Preprocessorfor example...
_TSM
Regional Controland
Monitoring
BTeV Run Controland database(s), etc.
_TSM
Local Controland
Monitoring
FPGA
JTAG,programming
and debug/c/spot run
Local Config(Flash)
Regional copyof config data,
and history
FPGA
StandaloneOperationalCapability
ControlInfluence
MonitoringAwareness
Fire DetectPower, Cooling Monitoringand Control
![Page 26: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/26.jpg)
[email protected] 21aug01
26
EtherNet,for example
ARCNet,for example
Close-up: DSPTrigger Supervisor and Monitor
DSP
_TSM
Regional Controland
Monitoring
BTeV Run Controland database(s), etc.
_TSM
Local Controland
Monitoring
JTAG,programming,
debug, and monitoringrun(spot);
run;
Local Config(Flash)
Regional copyof config data,
and history
DSP
StandaloneOperationalCapability
ControlInfluence
MonitoringAwareness
Fire DetectPower, Cooling Monitoringand Control
Host PortInterface
DMAin
BSPout
FPGA
![Page 27: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/27.jpg)
[email protected] 21aug01
27
Factoids• TMS320C67x DSP
– ~70 Kbyte internal RAM, ~1200 MIP– fixed/floating point
• TMS320C64x DSP– ~1 Mbyte internal RAM, ~3x faster… – fixed point only
• times 2500 devices (!)– Sony Playstation 2.5K ?
![Page 28: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/28.jpg)
[email protected] 21aug01
28
More Factoids• Host Port Interface (TI DSP only)
– almost-direct access into the DSP• peek, poke• uses DMA (like) resources…
– (concept not unique to TI)• DMA
– crossing data in• Buffered Serial Port(s)
– opinions out; dual, ~75Mbps (C6x)
![Page 29: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/29.jpg)
[email protected] 21aug01
29
DSP/BIOS (Texas Instruments)• based on SPOX
– “the oldest DSP RTOS…”• scalable real-time kernel
– small footprint (< 2Kw)– preemptive multithreading
• scheduling (tasks, soft/hard interrupts)• synchronization (between tasks, interrupts)
– hardware abstraction– extensible
![Page 30: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/30.jpg)
[email protected] 21aug01
30
DSP/BIOS (continued)
– real-time analysis• real-time instrumentation
– explicit API calls– (controllably) implicit, at context changes
• host-target communications– statistics gathering
• same as above– host data channels
• binds kernel I/O objects to host files
![Page 31: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/31.jpg)
[email protected] 21aug01
31
RTDX - Real Time Data Exchange
• utilizes JTAG chain (and emulation bits)• target (kernel) component
– moves messages to/from DSP/BIOS queues from/to JTAG-accessible buffer
– “real time” target to host (?)• host component
– data visualization and analysis tools– data to target “not” real-time… (?)
![Page 32: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/32.jpg)
[email protected] 21aug01
32
RTDX (continued)• fundamental (TI) model
– one PC running CCS (or app with CCS API calls)
– small number (4’ish) of DSPs on one JTAG chain
– JTAG/emulator “pod” connects PC to chain• (2 “extra” emulation bits, in addition to
TDI/TDO/TMS/TCLK)
• BTeV challenge– not buying 500 PC’s to support 2000 DSPs…
![Page 33: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/33.jpg)
[email protected] 21aug01
33
BTeV Conclusion(s)• fierce data rates• massive parallelism• heterogeneous control-in-depth
• limited loss-of-data (or performance)– tolerable
• any loss of understanding– unacceptable
![Page 34: BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1ad07f8b9ab0599714a3/html5/thumbnails/34.jpg)
[email protected] 21aug01
34
BTeV TimeTable• detailed technical design report
– required Feb 2002• R&D for TDR, now
• to be operational in 2006
• http://www-btev.fnal.gov/btev.html