ian lazarus

Time Stamping Overview

Ian Lazarus

(STFC Daresbury)

Overview of talk:

• Introduction:– What is Time Stamping?

– Why do we do it?– How do we do it?

– Problems to consider.

– Examples of Time Stamping

• Future work

What is Time Stamping?

• Associate time label with data from a channel, a sub-system or a system.

• Fixed time relationship to moment when channel fired or event generated

• Permits the use of these time stamps to correlate data within detectors, between detectors, between systems.

• Simplest form is to correlate multiple free running systems (c.f. 1991 RDT experiments at Daresbury NSF)

• Note: Hardware triggers can be used too if necessary but many benefits of TS are lost if data is thrown away in this way.

Why do we do Time Stamping?

• Eliminates dead time (but not pileup)• Allows software triggering which hugely increases flexibility:

– Software can look back/forward in time (time=buffer size)

– Can take 100% of low rate in delayed coincidence with high rate(minimal dead time losses)

– Can correlate multiple systems without cross-triggering cables.

– Can easily do multiplicity triggers in systems with huge channelcounts

– Eliminate huge number of trigger connections (no problems from cables, ground/isolation)

– Overcome latency in pipelined processing (digital or analogue)

• Summary: Better data selection

How do we do Time Stamping?

• Need stable clock, lined up everywhere (adjust delay/phase)

• Need local TS logic in channel/sub-system/system

• Need a regular pulse to “timeframe” data (and lack of data)

• Need a powerful processing system and lots of memory

• Can be used with analogue or digital signal processing.

• Can be used alongside conventional hardware triggered readout, but with limited results.

Clk

Sync

Insert timeframe in data

and check local TS count now.

27 28 29 30 31 32 33 34TS

Count

Problems to consider when Time Stamping?

• Need timestamps in ASICs (c.f. N-XYTER and STFC CZT ASIC, CALICE MAPS, planned R3B Si ASIC. Others?)

• Latency- doesn’t matter until we make connections to ancillary detectors-cross-system triggers with analogue triggers

• Buffer edge effects (time/channels)

• Time alignment (setup) is critical – sub ns for AGATA PSA

– for simple coincidence must be better than half a clock across all systems

– for ToF must be few tens of ps (or calibrated)

• ToF not easy- TS is only as accurate as the clock after alignment and jitter errors.

– For ps ToF timing we still need to use TACs

Clk

Discrim

TS Register

TS

Count27 28 29 30 31 32 33 34

29

Discrim

Clk

TAC Start TAC Stop

Problems to consider when Time Stamping?

• Need timestamps in ASICs (c.f. N-XYTER and STFC CZT ASIC, CALICE MAPS, planned R3B Si ASIC. Others?)

• Latency- doesn’t matter until we make connections to ancillary detectors-cross-system triggers with analogue triggers

• Buffer edge effects (time/channels)

• Time alignment (setup) is critical – sub ns for AGATA PSA

– for simple coincidence must be better than half a clock across all systems

– for ToF must be few tens of ps (or calibrated)

• ToF not easy- TS is only as accurate as the clock after alignment and jitter errors.

– For ps ToF timing we still need to use TACs

– Should also calibrate the ToF channels for clock alignment errors

• Bandwidth for readout of high rate beam tracking (or else use links to feed in lists of interesting timestamp ranges to local readout)

Examples of NP Time Stamping Systems

• GREAT TDR (UK/Finland, Jyvaskala)- working

• CENTRUM (GANIL)- working• CENTRUM plus ATOM (GANIL) under design

• AGATA GTS system (Europe) designed; being built

• BUTIS (GSI/FAIR) Built (local distribution being specified)

• AIDA (DeSpec)• n-Xyter ASIC

Time Stamping example 1 -GREAT TDR

Rack includes

• 4 NIM and 2 CAMAC

for front end Amps, CFDs.

• A VME crate (data merger, metronome and

pattern unit)

• 2 VXI crates containing

between 12 and 15 32

channel cards (up to

480 channels).

Single stream of

time ordered data

to event builder.

Online

Analysis

Event

Builder

Tape

server

Tape

Time-ordered input data stream

Time of flight (5us)

start

Recoil data from focal plane

Ge data from target array

Time window

Coinc window (1us)

Selected data (6us)

GREAT data selection

Time Stamping example 2 -CENTRUM

OUT

GANIL

CLRC

TX 1-7RX

RX

TX2

TX1

RTAG LTAG1

TX5TX4

TX3

TX7TX6

LTAG2 LTAG3

LTAG4 LTAG5

LTAG7LTAG6

IN

CKEVT

IN

TEST2

DSP

IN

VXI-C

IN

IN

+ -ecl

TEST1

TST

SCLK

IN

AUX.

CENTRUM

•Time Stamps Events (not channels)

•Correlates

EXOGAM with VAMOS and TIARA

•ATOM upgrade adds channel timestamps

Time Stamping example 3 -AGATA

GTS ROOT

Ge

DIGITISER

PP

SEG

PP

Core

FADC FADC

DATADATA

Clk &

commands

Trig Req

“synch”clksynch

clksynch

Data Data

d15 = Synch or Inhibit

Test

Pulser

Test

pulse

Ge

GTS MEZZ

PP

SEG

PP

CoreDATA

DATA

Clk &

commands

Trig Req

“synch”clksynch

clksynch

Data Data

Test

pulse

PP core sends

special trig req

to GTS mezz

when it gets

d15 (10 or

20ns wide)

GTS MEZZ

MGT MGT

DIGITISER

FADC FADC


Test

Pulser

MGT MGT

GTS ROOT

Ge

DIGITISER

PP

SEG

PP

Core

FADC FADC

DATADATA

Clk &

commands

Trig Req

“synch”clksynch

clksynch

Data Data


Test

Pulser

Test

pulse

Ge

GTS MEZZ

PP

SEG

PP

CoreDATA

DATA

Clk &

commands

Trig Req

“synch”clksynch

clksynch

Data Data

Test

pulse

PP core sends

special trig req

to GTS mezz

when it gets

d15 (10 or

20ns wide)

GTS MEZZ

MGT MGT

DIGITISER

FADC FADC


Test

Pulser

MGT MGT

Alignment process: step 1

RocketIO

NODE MEZZANINE

RocketIO

TDC

Start Stop 1 Stop 2

TDC BOARD

ROOT MEZZANINE

Pulse

Generator

2

121

OSOSOSUO

TTTT ≅−=

UOT

The target is determining the uplink time (Tu) without RocketIO MGTs. In this case Tu is about the half of the loop time.


CHT

RocketIO

NODE MEZZANINE

RocketIO

TDC

Start Stop 1 Stop 2

TDC BOARD

ROOT MEZZANINE

Pulse Generator

2

111

OSSUOSCH

TTTTT −=−≅

In this case RocketIO Transceivers are enabled only for downlink channel.


FPGA

RocketIO Transceiver

RECCLK

DATA

DATA

RX FIFO

FILTER

TX FIFO

DELAY

FIFO

DLL

SFP

DATA

DRECCLK

FINE DELAY

(0÷2047 - 10 ps/step)

COARSE DELAY

(0÷255 - 10 ns/step)

USER CLOCK

USER DATA

UPSTREAM

DATA

TOWARD

ROOT NODE

Once the channels

latencies are known,

shorter latencies can be increased in each user

node. The result is that

each node will be

aligned to the longest

node latency.

Systematic Errors

L = 1 [m]

N 1 N 2 N 3

L = 1 [m]

N 1 N 2 N 3

Estimated Latency Error (without MGTs)

0,00

0,05

0,10

0,15

0,20

0,25

0,30

0,35

0,40

0,45

-1,7 -1,6 -1,6 -1,5 -1,4 -1,3 -1,2 -1,2 -1,1 -1,0 -0,9 -0,8 -0,7 -0,7 -0,6

Latency error [ns]

Relative Frequency

node 1

node 2

node 3

Node Latency (without MGTs)

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

16,1 16,3 16,4 16,6 16,7 16,9 17,1 17,2 17,4 17,6 17,7 17,9 18,1 18,2 18,4 18,6

Latency [ns]

Relative Frequency

L1

L2

L3

EL1

EL2

EL3

Time Stamping example 4 –NUSTAR (BUTIS)

Si Ladder

DSSD

ASIC

FEE

Card

Handles

16 ASICs,

each 64ch

= 1024ch.

Total approx

100 FEE cardsBUTISBUTIS

Fan-out

NUSTAR

DABC

DAQR3B

input

code

module

Gbit Ethernet fibre(s)

ASIC

Fibre

Ethernet

R3B conceptual design

for Si tracker FEE and DAQ

Draft 1 April 2008

fibre

links

either fibre

or HDMI linksSlow

Control

Front End Electronics

Data output stage

standard format and

output medium e.g.

10G Ethernet fibre

Correlate by timestamp

Clock and

Timestamp

BUTIS

Common

Clocks

10/200MHz

<100ps/km

Slow Control

Common

database

loaded into

local

controllers

over Ethernet DetectorDetector

HV etc.

Time Stamping example 5 –AIDA (DeSpec)

Mechanics diagrams

Daughte

r card

connecto

r

16 channel

ASIC

AD9252FADC

AD9252

FADC

16 bit ADC

Sin

gle

todiffn

.

16

1

I2C controls

N x 2 discriminator outputs

Readout controls

Clocks

Reset

10 LVDS20 pins

10 LVDS

20 pins

4

Daughte

r card

connecto

r

16 channelASIC

AD9252

FADC

AD9252FADC

16 bit ADC

Sin

gle

todiffn

.

16

1

I2C controls


Readout controls

ClocksReset

10 LVDS

20 pins

10 LVDS20 pins

4

Daughte

r card

connecto

r

16 channel

ASIC

AD9252

FADC

AD9252

FADC

16 bit ADC

Sin

gle

todiffn

.

16

1

I2C controls


Readout controls

Clocks

Reset

10 LVDS

20 pins

10 LVDS

20 pins

4

Daughte

r card

connecto

r

16 channel

ASIC

AD9252

FADC

AD9252

FADC

16 bit ADC

Sin

gle

todiffn

.

16

1

I2C controls


Readout controls

Clocks

Reset

10 LVDS

20 pins

10 LVDS

20 pins

4

Xilinx XC5VFX70T-?FF1136

640 I/O

344 used for 4 ASICS

ContainsPowerPC processor

Readout State Machine

64 channels of

Energy/Timing/Waveform

4 channels of ASIC readout

Timestamp logic

TEMACPHY

64MBSDRAM

FLASH forboot load of

LINUX

RJ45

Timestampexternalinterface

Consoleinterface

Oscillators

AIDA - FEE (64) - ASIC to DAQ block diagram

High speed Discr OR

JTAG

Gbit Ethernet

BuTiS interface

Water cooled metal

Water cooled metal

FPGA & FADCs

FPGA & FADCs

Water cooled metal

Water cooled metal

View A

Kapton Cable toto detector

RJ45Gigabit

RJ45Gigabit

HDMIClock,SYNC,ResetPower, Disc_out

HDMIClock,SYNC,Reset

Power

JTAG

JTAG

Connector board View A

One of 4

One of 64

Microblaze running

LINUX

Max rate = ( 60K x 8 )

+

( 10k x 2024 )

=>

20,720,000

bytes/sec

Include edge events

=>61,200,000

bytes/sec

SharedRAM

Max rate 60 K events/sec

Shared

RAM

Shared

RAM

Shared

RAM




State

machine

Q

ADC

ASIC

16

channels

Timestamp

16 Discriminators

Sync/Pse/Res

To D

igita

l

Max rate 30K events/sec - 16 bit data

State

machine

Q

1024 x

(48 + 64 + 3)

Energy MWD

Leading Edge

discriminator

Slow in Fast out Waveform RAM

"Circulating Buffer"

Timestamp

Sync/Pse/Res

1024 long (20uS ) Floating pointtime vernier

calc.

Q

10 x 36Energy

DMA

Memory

manager

64MByte

DDR

SDRAM

32 bit

FIFO3072 x 16

VHDL block diagram

Patrick Coleman-Smith STFC, Daresbury Laboratory

Time Stamping example 6

– n-xyter

Time Stamping- future work in Hi/Despec

• House-training BUTIS?– Simple interface module– Keep down the costs per FEE card

• Connect AGATA DAQ– exchange timestamps

• Could we make a common TS port?– c.f. AIDA for DeSpec

• Common hardware interface, e.g. HDMI

• Make signals available to FPGA (change VHDL)

ian lazarus

Documents