frank lemke dpg frühjahrstagung 2010 time synchronization and measurements of a hierarchical daq...

Frank LemkeDPG Frühjahrstagung 2010

Time synchronization and measurements of a Time synchronization and measurements of a hierarchical DAQ networkhierarchical DAQ network

DPG Conference Bonn 2010

Session: HK 70.3

University of Heidelberg

Frank Lemke, Sebastian Manz and Wenxue Gao

19.03.2010

2Frank LemkeDPG Frühjahrstagung 2010

OutlineOutline

• Data Acquisition Topology

• DAQ Features

• DAQ Components- Readout Controller- Data Combiner Board- Active Buffer Board

• Conclusion & Outlook


Data Acquisition Topology Data Acquisition Topology

ABB

DCB

ROC

ROC

…

FEB

FEB

FEB

FEB

DCB

ROC

ROC

…FEB

FEB

FEB

FEB

Com

puting Cluster (F

LES

)……

Detector S

ystem

Optical connectionNon-Optical connection

FEB = Frontend BoardROC = Readout ControllerDCB = Data Combiner BoardABB = Active Buffer Board


CBM Optical Link ProtocolCBM Optical Link Protocol

• CBM protocol- Three traffic classes- Delivers specific adapted

performance & features

• Generic link protocol- Minimized interface- Modularized- Efficient interface

synchronization- Transparent

16

16

ctrl2send_startctr l2send_end

ctrl2send

data2send_startdata2send_end

data2send

data2send_stop

ctrl2send_stop

4

4

16

16

dlm2send_vadlm2send_type

dlm_rec_typedlm_rec_va

data_recdata_rec_startdata_rec_end

ctr l_recctrl_rec_startctr l_rec_end 125 MHz ctrl_rec_stop

data_rec_stop

link_active

clk

CBMProtocolModule

CoreModule

link_clk

res_n

(2,5 GBit Link)

DLM - Receive

Control

DataMessage

DataMessage

ControlMessage

DLM - Send

MessageSend

Send

Receive

Receive

crc_error_send

crc_error_rec

User Network


ROC ROCROCROC ROCROC

Clock & Synchronization NetClock & Synchronization Net

ROC ROC ROC ROCROC

Frontend to Detector System

…… …… ………

DCB DCB DCB

Synchronization and clock distribution over optical link

Backend to Computing Cluster (FLES)

… …

JC

JC = Jitter cleaning and clock recovery

ROC ROC ROCROCROCROC ROCROC

JC JC JC JC JC

JC JC JC

Clock &Sync. Gen.Tree structured connection


Jitter Cleaner ModuleJitter Cleaner Module

• Requirements- Internal deterministic

implementation- Recovered clock usage- Peak-to-peak below 40ps

• Results- Extender connected with

mezzanine- National LMK03000 Family- MMCX to transmit clock- Bit precise synchronisation

V 1.0

V 1.1


Readout Controller BoardReadout Controller Board

• Functions- Interfacing the front-end

electronics- Timestamp expansion- Data transport via optics

(or Ethernet)

• Board development- SysCore 1: Virtex 4 FX20- SysCore 2: better suited

connectors- SysCore 3 (planned):

mother- and daughter-board, Spartan6? V 1.0

V 2.0

(exemplary)


ROC – Functional DiagramROC – Functional Diagram

• Virtex-4 FX20• Ethernet, MGTs

for communication• DDR SDRAM• Actel Flash-FPGA

for configuration• SD-Card and

Flash• Many User-I/O

pins

(picture from http://cbm-wiki.gsi.de)


ROC – Modular Firmware DesignROC – Modular Firmware Design

??

ReadoutLogic Transport

Logic

nXYTERROC

nXYTERROC

FEETROC

FEETROC

EthernetEthernet

MGTsMGTs

Bus

FIFO AB

BA

BB

DriverDriver

DABC

GPIO

© N.Abel


Data Combiner BoardData Combiner Board

• Functions- Data switching and concentration

multiple ROCs => ABB- 6 SFPs- 250MHz low jitter

programmable clock oscillator- Interface for jitter cleaner

mezzanine

• Board development- DCB V1.0: Virtex-4 FX100 - DCB V1.3: Improvements on

MGT reference clock insertion

V 1.0

V 1.3


Receive clk

Transmit clk

Functional DiagramFunctional Diagram

DCB V1.3

XV6-fx100

JC

ABBSFP

SFP

SFPSFP

SFP

SFP

SFP DLMLP

LP

LP

LP

LP

• Special deterministic latency message handling• Erroneous detector control message retransmission• Efficient switching mechanism• Well defined clocking scheme

ROCs


Hardware DevelopmentsHardware Developments

• Protocol development• Protocol test and verification • Clock distribution with jitter cleaning• Synchronization via deterministic latency messages• Crossbar and switching logic


Active Buffer BoardActive Buffer Board

• Functions- Buffering the incoming hit packets- Hits reorganization- Transfer events to the host node- PCI Express 4-lane- DMA upstream: DAQ data packets

from buffer to the host node

• Board development- ABB-1: Virtex 4 FX60- ABB-2 (AVNET): Virtex 5 LX110T- ABB-3 (ML605): Virtex 6 LX240T

ABB-1

ABB-3


ABB Diagram and Measurement

• Control messages invoke the data flow from ROC via DCB to ABB

• Hit packets are first buffered in the FIFO• Host software starts DMA to transfer the buffered hits to

the host memory• Measured bandwidth of 224 MB/s for one optical link


Software Developments

• Driver under Linux 2.6• Performance emphasized• Well-layered architecture• ABBDaemon masks driver details to the users

- Enables access from multiple users- Uses UNIX IPC mechanism

• Provided test programs- RAM memory- One-hole memory / FIFO- CTL messages


Test SetupTest Setup


Conclusion & OutlookConclusion & Outlook

• Prototype systems working and in use

• Virtex6 adaption and development for all boards• ROC

- Modular hardware (mother/daughterboard)

• DCB- Protocol HDL optimizations

• ABB- Event Building - Flexible strategies for hit dispatching- Unified test program for more efficient debugging

18Frank LemkeDPG Frühjahrstagung 2010

Questions ?

Thank you for your attention !

frank lemke dpg frühjahrstagung 2010 time synchronization and measurements of a hierarchical daq...

Documents