current fed tester status

04/22/23 James Leaver

Current FED Tester Status


Software Status• FED Tester software is in a

fairly refined state

• Universal Toolbox class enables full calibration, configuration and initialisation of system with a single function call

• Toolbox configures and provides access to:

– FED Testers

– FED

– FED Kit

– Frame data generator

– FED event wrapper which transparently enables readout though Slink or VME

• Forms the core of several testing programs, which could easily be used at RAL

• Entire FED Tester Software package: 31,563 physical lines of code so far…

TrimDAC

Calibration FTE AOH

Calibration FED Timing

Calibration

Calibrator

Configurator

Allow User To

Make Changes Initialise Device

Descriptions

Initialiser

Toolbox

Provide Access

To Devices

Provide Access

To Descriptions

Initialise

Devices


TrimDAC & AOH Calibration

• TrimDAC calibration:

– FED Tester output disabled

– FED TrimDACs calibrated (Fed9UDevice method)

– TrimDAC values increased to shift baseline below FED ADC range – ensures efficient use of ADC range when optical input from FED Testers is present

• AOH calibration:

– FED Testers output alternate digital high / digital low signals to each FED channel

– FED Tester AOH bias currents and gains are adjusted for each FED channel to optimise use of FED input ADC range


TrimDAC & AOH Calibration: Results

1000

800

600

400

200

0

AD

C v

alue

806040200

Channel number

5% of ADC range

95% of ADC range

FED ADC range (digital low → digital high) post TrimDAC and AOH calibration:


FED Timing Calibration

• FED Testers send tick marks to FED

• All FED coarse and fine delay settings are scanned through

• Appropriate FED delay settings for each FED channel are found, to ensure optimal data sampling points


FED Timing Calibration: Results

1000

800

600

400

200

0

AD

C V

alue

560540520500480

channel 0

1000

800

600

400

200

0A

DC

Val

ue560540520500480

channel 1

1000

800

600

400

200

0

AD

C V

alu

e

560540520500480

channel 2

1000

800

600

400

200

0

AD

C V

alue

560540520500480

channel 3

1000

800

600

400

200

0

AD

C V

alue

560540520500480

channel 4

1000

800

600

400

200

0

AD

C V

alue

560540520500480

channel 5

1000

800

600

400

200

0

AD

C V

alu

e

560540520500480

channel 6

1000

800

600

400

200

0

AD

C V

alue

560540520500480

channel 7

1000

800

600

400

200

0

AD

C V

alue

560540520500480

channel 8

1000

800

600

400

200

0

AD

C V

alue

560540520500480

channel 9

1000

800

600

400

200

0

AD

C V

alu

e

560540520500480

channel 10

1000

800

600

400

200

0

AD

C V

alue

560540520500480

channel 11

Positions of calibrated sampling points on a tick mark for FE Unit 0:


FED Inter-Channel Crosstalk

• Wanted to measure the effects of crosstalk between FED channels

• Will show two sets of results here:

– ‘Worst Case’:

• 11 of 12 FE Unit channels carry a ‘noise’ signal, look at crosstalk on remaining channels

– ‘Nearest (& Next Nearest) Neighbour’:

• 1 of 12 FE Unit channels carry a ‘noise’ signal, look at crosstalk on neighbouring channels


Crosstalk: Worst Case: Setup

• Sent typical frame to FE Unit channel 5

• Sent similar frame to other FE Unit channels, but added a delay of 10 clock cycles

– (Repeated for each FE Unit)

• Used FED Testers to simultaneously phase shift all frames relative to FED sampling point, from 0 to ~25 ns in ~100 ps steps

– By capturing Scope Mode data from the FED at each phase step, can build up a high resolution image of signal seen by FED


Crosstalk: Worst Case: Results

700

600

500

400

300

AD

C C

ount

s

34003200300028002600240022002000

Time (ns [approx.])

Data: Channel 5 Channel 6

Simple Crosstalk Measurement (offset_frame)

********************* Gener al Set t i ngs ********************* *** Cur r ent FED set t i ngs: FED - Read Rout e = FED9U_ROUTE_SLI NK64 FED - DAQ Mode = FED9U_MODE_SCOPE FED - Scope Lengt h = 512 FED - Test Regi st er = NORMAL *** Cur r ent FTE set t i ngs: FTE - Tr i gger St at e = REPETI TI VE FTE - L1A Per i od = 39900 FTE - L1A Low Rat e Per i od = 39900 FTE - L1A Random Rat e = 164 FTE - Mi ni mum Tr i gger Spaci ng = 36 *** Cur r ent FTE FM set t i ngs: FTE FM - Cl ust er Defi ni t i on = MULTI _STRI P FTE FM - Cl ust er Wi dt hs Fi l e = FTeFr ameMaker Fi l es/ FTe_Cl uWi dt hDi st r _Si mul at i on. t xt FTE FM - Gaussi an Noi se Level = 0 ###################################################################### I ni t i al set t i ngs: *** Fr ame Number i ng: Fr ame 0 = ' Level ' f r ame Fr ame 1 = ' Pul se' f r ame Phase shi f t f r ame 0 : TRUE Phase shi f t f r ame 1 : TRUE Fr ame 1 f r ame del ay = 10 Fr ame 1 pedest al val ue = 1200 Fr ame 1 hi t val ue = 2000 Fr ame 1 hi t posi t i on = 60 Fr ame 1 hi t wi dt h = 1 Fr ame t o FE Uni t channel mappi ng: Channel : 0 1 2 3 4 5 6 7 8 9 10 11 Fr ame: 1 1 1 1 1 0 1 1 1 1 1 1 ######################################################################

FE Unit 0: A view of the overlapping frame digital headers - All channels apart from 5 see the blue input signal


Crosstalk: Worst Case: Results(2)

370

360

350

340

AD

C C

ount

s

29002850280027502700Time (ns [approx.])


Simple Crosstalk Measurement (offset_frame)

********************* Gener al Set t i ngs ********************* *** Cur r ent FED set t i ngs: FED - Read Rout e = FED9U_ROUTE_SLI NK64 FED - DAQ Mode = FED9U_MODE_SCOPE FED - Scope Lengt h = 512 FED - Test Regi st er = NORMAL *** Cur r ent FTE set t i ngs: FTE - Tr i gger St at e = REPETI TI VE FTE - L1A Per i od = 39900 FTE - L1A Low Rat e Per i od = 39900 FTE - L1A Random Rat e = 164 FTE - Mi ni mum Tr i gger Spaci ng = 36 *** Cur r ent FTE FM set t i ngs: FTE FM - Cl ust er Defi ni t i on = MULTI _STRI P FTE FM - Cl ust er Wi dt hs Fi l e = FTeFr ameMaker Fi l es/ FTe_Cl uWi dt hDi st r _Si mul at i on. t xt FTE FM - Gaussi an Noi se Level = 0 ###################################################################### I ni t i al set t i ngs: *** Fr ame Number i ng: Fr ame 0 = ' Level ' f r ame Fr ame 1 = ' Pul se' f r ame Phase shi f t f r ame 0 : TRUE Phase shi f t f r ame 1 : TRUE Fr ame 1 f r ame del ay = 10 Fr ame 1 pedest al val ue = 1200 Fr ame 1 hi t val ue = 2000 Fr ame 1 hi t posi t i on = 60 Fr ame 1 hi t wi dt h = 1 Fr ame t o FE Uni t channel mappi ng: Channel : 0 1 2 3 4 5 6 7 8 9 10 11 Fr ame: 1 1 1 1 1 0 1 1 1 1 1 1 ######################################################################

Level shift of ~4 ADC counts

when blue frame goes low (power issue?)

Level shift of ~4 ADC counts

when blue frame goes low (power issue?)

Spikes of ~10 ADC counts in amplitude

Spikes of ~10 ADC counts in amplitude


Crosstalk: Nearest Neighbour: Setup

• Send typical empty frame to all FE Unit channels apart from 5

• Send a frame to FE Unit channel 5 which has a 2-strip wide ‘pulse’ in the centre of the payload

– (Repeated for each FE Unit)

• Use FED Testers to phase shift frames sent to channel 5 (keep constant phase for other channels) – build up high resolution signal images as before

• Repeat with 3 pulse heights:

– Pulse 1: Pedestal to digital high

– Pulse 2: Pedestal to (2/3) * (digital high)

– Pulse 3: Pedestal to (1/3) * (digital high)


Crosstalk: Nearest Neighbour: Results

367

366

365

364

363

362

361

360

359

358

357

356

AD

C C

ount

s

57005680566056405620560055805560Time (ns [approx.])


Simple Crosstalk Measurement (2400Pulse_Ch5)

420

419

418

417

416

415

414

413

412

411

AD

C C

ount

s

5680566056405620560055805560

Time (ns [approx.])



Pu

lse

1: N

eare

st N

eig

hb

ou

r

Pulse 1: Next Nearest Neighbour

• FE Unit 2: All channels apart from 5 see the blue input signal

• With maximum ‘noise’ pulse height:– Nearest neighbour crosstalk

amplitude: ~3.5 ADC counts

– Next nearest neighbour crosstalk amplitude: negligible


Crosstalk: Nearest Neighbour: Results(2)

700

600

500

400

300

AD

C C

ount

s

12x103

1086420Time (ns [approx.])



366

365

364

363

362

361

360

359

358

357

356

AD

C C

ount

s

57505700565056005550Time (ns [approx.])



700

600

500

400

300

AD

C C

ount

s

12x103

1086420Time (ns [approx.])



366

365

364

363

362

361

360

359

AD

C C

ount

s

5700565056005550Time (ns [approx.])



Pu

lse

2: N

eare

st N

eig

hb

ou

r

Pu

lse

3: N

eare

st N

eig

hb

ou

r


FED Channel Noise

• Simple noise measurement:

– Disabled FED Tester output and set appropriate FED TrimDAC / OptoRx values (constant across all channels)

– Captured a Scope Mode event for each FED channel (Scope Length = 1020)

– Found mean and standard deviation of signal at each channel:

450

400

350

Mea

n of

Rec

eive

d S

igna

l (A

DC

Cou

nts)

806040200

FED Channel

Average Signal Vs FED Channel

FTE opt i cal out put s : DI SABLEDFED DAQ Mode = FED9U_MODE_SCOPEFED Scope Lengt h = 1020FED Test Regi st er = NORMALFED Opt o Rx i nput off set = 0x5FED Opt o Rx out put off set = 0x3FED Tr i m DAQ off set = 0xa0

0.80

0.75

0.70

0.65S.D

. of R

ecei

ved

Sig

nal (

AD

C C

ount

s)

806040200

FED Channel

Standard Deviation of Signal Vs FED Channel

FTE opt i cal out put s : DI SABLEDFED DAQ Mode = FED9U_MODE_SCOPEFED Scope Lengt h = 1020FED Test Regi st er = NORMALFED Opt o Rx i nput off set = 0x5FED Opt o Rx out put off set = 0x3FED Tr i m DAQ off set = 0xa0


FED Hit Check

• Wanted to check that the numbers / locations of hits input to the FED match the numbers / locations of hits output by the FED in Zero Suppressed Mode.


FED Hit Check: Setup• Ran with 100 kHz random triggers, FED in Zero Suppressed Mode

• Used randomly generated events with simulated CMS cluster distribution (2% Tracker occupancy)

• Set FED strip high & low thresholds to 50

• Set the hit height in our randomly generated frames to vary (almost) between pedestal and digital high

– i.e. from just below the FED strip threshold to near the top of the FED ADC range

• Generated 1024 events (FED Tester capacity), read them out through the FED, generated another 1024 events, repeated 100 times

• Compared every input hit ADC value with every output hit ADC value; expect a linear scatter graph of non-zero values if FED detects every input hit correctly


FED Hit Check: Results

250

200

150

100

50

0

Out

put H

it V

alue

(F

ED

AD

C C

ount

s)

40003000200010000Input Hit Value (FTE ADC Counts)

500

400

300

200

100

0

Nu

mb

er o

f Sa

mp

les

Hit Value Output by FED Vs Hit Value Input by FTE

FED Channel 0: Graph contains data from ~5 x 105 hits


FED Hit Check: Results(2)

220

210

200

190

180

170

160

150

140

130

120

Out

put

Hit

Val

ue (

FE

D A

DC

Cou

nts)

16001560152014801440140013601320128012401200

Input Hit Value (FTE ADC Counts)

500

400

300

200

100

0

Nu

mb

er o

f Sa

mp

les


Close-up of linear region



174

172

170

168

166

164

162

160

158

Out

put

Hit

Val

ue (

FE

D A

DC

Cou

nts)

142014151410140514001395139013851380137513701365136013551350


500

400

300

200

100

0

Num

ber of Sam

ples


Extreme close-up of linear region



60

55

50

45

40

35

30

25

20

15

10

5

0

Out

put H

it V

alue

(F

ED

AD

C C

ount

s)


FED High & Low Strip Threshold = 50500

400

300

200

100

0

Nu

mb

er o

f Sa

mp

les


A close-up of the region around the FED strip threshold:The FED appears to detect a small fraction of hits that occur below the strip threshold!



250

200

150

100

50

0

Out

put H

it V

alue

(F

ED

AD

C C

ount

s)


FED Chan 0


A scatter plot of the same data shows the effect more clearly:

‘Hits’ below FED threshold

‘Hits’ below FED threshold

‘Hits’ detected when input

signal is zero

‘Hits’ detected when input

signal is zero


FED Hit Check: Unexpected Non-Zero ADC Values

Cluster 1

Event Data

Cluster 2

Noise

Why does the FED return non-zero ADC values for strips that do not contain valid hits?

→ Due to data packaging format:

i.e. 1 Cluster

FED Strip Threshold

1 Value Below Threshold

2 clusters separated by 1 strip:

• Transmitted as 1 cluster

• Read out noise level at the strip in-between, instead of zero

• Non-zero ADC value = ‘false hit’

OR

1 cluster with 1 strip below threshold:

• Transmitted as 1 cluster

• Read out below-threshold hit level, instead of zero

• Non-zero ADC value = ‘invalid hit’


FED Hit Check: Unexpected Non-Zero ADC Values(2)

• The non-zero ADC values transmitted due to data formatting might be intentional…

– Or perhaps the Fed9UEvent class could filter them out with a threshold cut…

• To work around the issue, the FED Hit Check was repeated using randomly generated frames containing only single strip clusters, separated by more than 1 strip

– Reduced Tracker occupancy to 1% to account for increased data volume


FED Hit Check: Single Hit Results

250

200

150

100

50

0

Out

put H

it V

alue

(F

ED

AD

C C

ount

s)


FED Chan 0


All invalid ‘hits’ have vanished

All invalid ‘hits’ have vanished

Result: FED is correctly identifying all input hits


FED Hit Check: Single Hit Results(2)

250

200

150

100

50

0

Out

put

Hit

Val

ue (

FE

D A

DC

Cou

nts)

2500200015001000500


200

150

100

50

Nu

mb

er o

f Sa

mp

les


Results from all 96 FED channels:


FED Efficiency Test

• Wanted to test that FED operates correctly with:

– 100 kHz random triggers

– High Tracker occupancy

– Slink readout

– FED throttling system in place

• Also wanted to test FED efficiency (fraction of events lost vs. Tracker occupancy) predictions made by Emlyn

– However, the assumptions used in earlier predictions have changed:

• Emlyn used 2 bytes per hit (strip position, ADC value)

• Now have cluster finding (cluster position, cluster width, ADC values)

• Emlyn used back-to-back frames

– Could reproduce Emlyn’s conditions, with fixed 2 strip wide clusters and back-to-back frames…

– … But our goal is to replicate CMS, so more realistic conditions were used (FED efficiency prediction should still be reasonably accurate)


FED Efficiency: Buffering in ZS Mode

• Large buffers: 80 MB/s FE-BE link dominates

• Occupancy reaches ~9% before events are lost

Front-End @ 140 kHz (b2b frames) Back-End @ 140 kHz (b2b frames)

• Large buffers: slink dominates

• Events lost when:

– Occupancy ~1.4% at 100 MB/s

– Occupancy ~2.8% at 200 MB/s

Emlyn’s predictions:

→ → Slink data rate determines FED efficiencySlink data rate determines FED efficiency


FED Efficiency: Experimental Setup

• Random 100 kHz triggers

• FED in Zero Suppressed mode

• Readout rate through Slink limited to maximum of 200 MB/s

• Sent randomly generated frames with simulated CMS cluster distribution

• Increased simulated Tracker occupancy of generated frames and recorded fraction of events vetoed by FED


0.5

0.4

0.3

0.2

0.1

0.0Nor

mal

ised

fre

quen

cy

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Cluster width (strips)

FED Efficiency: Cluster Distribution

Mean = 2.96 strips per cluster

Average size of hit = 1.91 bytes

(Emlyn’s average hit size = 2 bytes)

Mean = 2.96 strips per cluster

Average size of hit = 1.91 bytes

(Emlyn’s average hit size = 2 bytes)

Simulated cluster distribution was generated from the plots in Figure 4 of Ian Tomalin’s CMS-IN 2005/025 Note:

Note: Would clusters of this size really exist?


FED Efficiency: Results

3

4

5

678

0.01

2

3

4

5

678

0.1

2

3

4

5

6

Fra

ctio

n o

f L

1A

s V

eto

ed

by

FE

D

1086420Tracker Occupancy (%)

200

180

160

140

120

100

80

Re

ad

ou

t Ra

te (M

B / s)

Data: L1As vetoed Readout rate

L1As Vetoed Vs Tracker Occupancy (200_sim1)

********************* Gener al Set t i ngs ********************* *** Cur r ent FED set t i ngs: FED - Read Rout e = FED9U_ROUTE_SLI NK64 FED - DAQ Mode = FED9U_MODE_ZERO_SUPPRESSED FED - Scope Lengt h = 6 FED - Test Regi st er = NORMAL *** Cur r ent FTE set t i ngs: FTE - Tr i gger St at e = RANDOM FTE - L1A Per i od = 400 FTE - L1A Low Rat e Per i od = 40000 FTE - L1A Random Rat e = 164 FTE - Mi ni mum Tr i gger Spaci ng = 36 *** Cur r ent FTE FM set t i ngs: FTE FM - Cl ust er Defi ni t i on = MULTI _STRI P FTE FM - Cl ust er Wi dt hs Fi l e = FTeFr ameMaker Fi l es\ FTe_Cl uWi dt hDi st r _Si mul at i on. t xt FTE FM - Gaussi an Noi se Level = 0 ################################################################################ I ni t i al set t i ngs: Number of t r i gger s t o send per t est = 10000000 I ni t i al Tr acker occupancy per cent = 0 Tr acker occupancy st ep = 0. 05 Number of Tr acker occupancy per cent ages t o t est = 400 Maxi mum r eadout r at e ( MB/ s) = 200 ################################################################################

Events lost when occupancy exceeds ~2.8% with 200 MB/s readout rate


FED Efficiency: CRC Errors

600

500

400

300

200

100

0

Num

ber

of E

vent

s w

ith E

rror

s

1086420Tracker Occupancy (%)

Data: WC errors Event number errors Event length errors CRC errors

Numbers of Errors Vs Tracker Occupancy (200_sim1)

********************* Gener al Set t i ngs ********************* *** Cur r ent FED set t i ngs: FED - Read Rout e = FED9U_ROUTE_SLI NK64 FED - DAQ Mode = FED9U_MODE_ZERO_SUPPRESSED FED - Scope Lengt h = 6 FED - Test Regi st er = NORMAL *** Cur r ent FTE set t i ngs: FTE - Tr i gger St at e = RANDOM FTE - L1A Per i od = 400 FTE - L1A Low Rat e Per i od = 40000 FTE - L1A Random Rat e = 164 FTE - Mi ni mum Tr i gger Spaci ng = 36 *** Cur r ent FTE FM set t i ngs: FTE FM - Cl ust er Defi ni t i on = MULTI _STRI P FTE FM - Cl ust er Wi dt hs Fi l e = FTeFr ameMaker Fi l es\ FTe_Cl uWi dt hDi st r _Si mul at i on. t xt FTE FM - Gaussi an Noi se Level = 0 ################################################################################ I ni t i al set t i ngs: Number of t r i gger s t o send per t est = 10000000 I ni t i al Tr acker occupancy per cent = 0 Tr acker occupancy st ep = 0. 05 Number of Tr acker occupancy per cent ages t o t est = 400 Maxi mum r eadout r at e ( MB/ s) = 200 ################################################################################

FED Efficiency Test yields unexplained distribution of CRC errors at high Tracker occupancies…


FED Efficiency: Theoretical Vs Measured Data Rates

• Theoretical data rate from CMS Note 2002/047:

– TIB1 ‘Full’ FED (180 of 192 APVs)

– Zero Suppressed, 100 kHz triggers, Occupancy of 2.8 - 2.9%

→ Data rate = 109.5 MB/s109.5 MB/s

• Measured data rate at occupancy of 2.9% = 217.8 MB/s217.8 MB/s

• Reason for discrepancy?

– Current ‘formatting’ information (non-data) = 816 bytes / event

– CMS Note ‘formatting’ information (non-data) = 112 bytes / event

– Assume current average data padding = 32 bytes / event

– Scale by FED fill factor of 180 / 192

• ‘Corrected’ measured data rate = 139 MB/s139 MB/s


FED Efficiency: Theoretical Vs Measured Data Rates(2)

• ‘Corrected’ measured data rate = 139 MB/s139 MB/s

• Can use average cluster width of generated frames to estimate expected data rate:

→ 132 MB/s 132 MB/s (good agreement)

• Calculate average of 1.55 bytes / hit for Ian’s data:

– Implies large number of clusters with 4 or more hits

– TIB1 cluster distribution in CMS Note must be different to that used in measurement - should account for remaining discrepancy…

• Conclusion:

– A true data rate comparison would require more accurate cluster distribution data for the frame generator

– Results show some agreement with Emlyn’s prediction, but cannot demonstrate FEDs performance in final system until we have accurate cluster distribution information and the FED Zero Suppressed ‘Lite’ mode

• i.e. Current data rates are significantly higher than those expected at CMS


Summary

• FED Crosstalk:FED Crosstalk:

– Unpleasant effects when a single frame arrives at the FED out of sync with the others…

– …But crosstalk caused by ‘hit’ features is small and should only impact nearest neighbours

• FED Channel Noise:FED Channel Noise:

– Average noise is less than 1 ADC count, with no significant variation from channel to channel

• FED Hit Check:FED Hit Check:

– FED correctly identifies hits in Zero Suppressed mode (for a 2% Tracker occupancy)

• FED Efficiency:FED Efficiency:

– FED vetoes triggers at high data rates in a similar manner to that predicted by Emlyn, but Zero Suppressed ‘Lite’ mode required for genuine CMS performance characterisation

– Measured data rates show reasonable agreement with theory, but a more accurate cluster distribution is required


Unresolved FED Problems…

• FED still randomly produces a small fraction of events with CRC errors (~1 in 1-10 million events, depending upon Tracker occupancy)

– Possibly a timing issue due to transition to new FPGA tools…?


What Do We Test Now?

• Still need to investigate:

– Pedestal variations

– Pipeline addresses

– Use of TTC

– etc.

• Need to prioritise remaining tests, and check that all important areas are included…

current fed tester status

Documents