The Application of DAQ-Middleware

to the J-PARC E16 Experiment

E Hamada1, M Ikeno1, D Kawama2, Y Morino1, W Nakai3, 2, Y Obara3,

K Ozawa1, H Sendai1, T N Takahashi4, M M Tanaka1, S Yokkaichi2

1High Energy Accelerator Research Organization (KEK)2RIKEN Nishina Center

3The University of Tokyo4Research Center for Nuclear Physics, Osaka University

2015/4/14 CHEP2015 1

Track 1 Session: #3(Data acquisition and electronics)April 14, 2015 18:00 – Village Center

Outline

1. DAQ-Middlewareoverview and features

2. DAQ System at E16 Experimentrequirements and architecture

3. DAQ System Performancemethod and result

4. Summary and Future Plan

2015/4/14 CHEP2015 2

1. DAQ-Middleware

2015/4/14 CHEP2015 3

What DAQ-Middleware Is

• A framework for network based DAQ software– Easy to use, configure and develop

• Target– Medium-scale experiments– Test benches (sensors, electronics)

2015/4/14 CHEP2015 4

DAQ-Middleware Architecture

• Develop DAQ system by configuring DAQ components

2015/4/14 CHEP2015 5

DispatcherGatherer

・・・

Read-out-modules

PC

PC

XML

System Configuration

DaqOperator

Logger

Monitor

Command / Status

software unit that achieve DAQ functions

DAQ Component

2015/4/14 CHEP2015 6

InPort OutPort

Service Port

(command/status)

Logics

(for data handling)

Data

InPort OutPort

Service Port

(command/status)

(for data handling)

Data+ =DAQ-Middleware

provides users writes

Logics

DAQ Component Features

2015/4/14 CHEP2015 7

READOUTREADOUT

READOUTREADOUT

READOUTREADOUT

READOUTREADOUT

Flexibility

Reusability

Scalability

logic

Ring Buffer

・・・

Users can flexibly change DAQ component combination.

Users can improve performance by adding new PCs and deploying DAQ components.

Users can use a DAQ component in various DAQ system.

DAQ component has a ring buffer. Users do not need to implement a buffer.

List of DAQ-Middleware Users

2015/4/14 CHEP2015 8

DAQ-Middleware Working

J-PARC MLFNeutron beam lines

• Experiments [19 experiments]- Material and Life Science

- J-PARC MLF Neutron/Muon

- DAQ system of Depth-resolved XMCD (KEK PF)

- Elementary Particle/Nuclear Physics

- J-PARCE16 Experiment

- CANDLES

- SuperNEMO (planning)

• Test benches for sensors and electronics [9 test benches]–Liquid Argon TPC

–SOI Pixel Detector

–ILC CCD Vertex

–ADC-SiTCP

・ 2D gaseous detector with readout ASIC using printing technologies

and so on….

2. DAQ System

at E16 Experiment

2015/4/14 CHEP2015 9

Requirements

2015/4/14 CHEP2015 10

660MBdata

receive

660MBdata

receive

notreceive

1spill

notreceive

Trigger rate fluctuates due to beam rate variation.Event size per one event is almost constant.

Estimation of data transfer to DAQ PCs

Data rate per spill 660MB/spill

Trigger rate (average) 1kHz

(max) 2kHz

Instantaneous data rate

(average) 330MB/s

(max) 660MB/s

System Architecture

2015/4/14 CHEP2015 11

Function ・ Event build・ Analyze data & show the result

Read-out-module Network switch Storage PC

Network switch Monitor PCHDD

HDD

HDD

Function ・ Read data・ Save data on each HDD・ Send data of a part of event to Monitor PC

・・・・・・

DAQ Component Configuration

2015/4/14 CHEP2015 12

Storage PC Monitor PC

・・・

Gatherer

Merger Dispatcher

Logger

Filter

Merger Eventbuilder Monitor

Component Function

Gatherer Read data

Merger Receive data from multiple Gatherers

Dispatcher Send data to Logger and Filter

Logger Save data on each HDD

Filter Send data of a part of event to Merger of Monitor PC

Component Function

Merger Receive data from multiple Filters

Eventbuilder Event build

Monitor Analyze data & show the result

DAQ Component Features

2015/4/14 CHEP2015 13

READOUTREADOUT

READOUTREADOUT

READOUTREADOUT

READOUTREADOUT

Flexibility

Reusability

Scalability

logic

Ring Buffer

・・・

Users can flexibly change DAQ component combination.

Users can improve performance by adding new PCs and deploying DAQ components.

Users can use a DAQ component in various DAQ system.

DAQ component has a ring buffer. Users do not need to implement a buffer.

If data volume increase by detector upgrade, we add Storage PC.

Advantage of DAQ Component Features

2015/4/14 CHEP2015 14

Flexibility & Scalability

Reusability

Ring BufferUsing DAQ component which is prepared, we saved time and effort for development.

We can use no event time effectively.

Sample Dispatcher, LoggerGatherer, Monitor

DAQ system

receive

process

receive

process

3. DAQ System Performance

2015/4/14 CHEP2015 15

Purpose & Method

2015/4/14 CHEP2015 16

Emulator Storage PCNetwork switch Monitor

PCHDD

HDD

Emulator

Emulator

EmulatorEmulator

Emulator

EmulatorEmulator

Emulator

test data

send data once every 10 events

Purpose

Evaluation of a total throughput using a DAQ-Middleware for the J-PARC E16 experiment

• Emulators run with 1 cycle per 6 seconds. During one cycle, emulators send test data for 2 seconds and do not send test data for 4 seconds.

• Monitor PC shows data value of a part of data regularly.

• We changed the number of emulators to change transfer data volume and measured processing data speed.

Method

Result

The result met the requirements of the E16 experiment!

2015/4/14 CHEP2015 17

requirement throughput(Trigger rate is always max [2kHz]. )

HDD write limit

requirement throughput(Trigger rate is always average [1kHz]. )

During sending test data,one event data size per one emulator is 14kB.

2kHz

1kHz

4. Summary and Future Plan

2015/4/14 CHEP2015 18

Summary and Future Plan

• DAQ-Middleware is a framework for network based DAQ software.

• DAQ component has following features.– Flexibility– Scalability– Reusability– Ring Buffer

• We have developed DAQ system for E16 experiment by using DAQ-Middleware.• The requirements from E16 experiment have been

met.• In the future, we are going to connect DAQ system to

read out module, and evaluate the system.

2015/4/14 CHEP2015 19

Backup

2015/4/14 CHEP2015 20

J-PARC E16 Experiment DAQ system overview

2015/4/14 CHEP2015 21

Evaluation Environment

2015/4/14 CHEP2015 22

HDD

HDD

PC1

PC2PC3

CPU Intel(R) Xeon(R) X5650 @ 2.67GHz 6Cores

Memory 24GB

Network 1Gbps x 5

OS Scientific Linux 6.4

HDD Hitachi HDS724040ALE64TB

PC1CPU Intel(R) Xeon(R)

CPU E5-2640 0 @ 2.50GHz 6Cores

Memory 32GB

Network 1Gbps x 5

OS Scientific Linux 6.6

HDD Hitachi HDS724040ALE64TB

PC2CPU Intel(R) Xeon(R)

CPU E3-1220 v3 @ 3.10GHz

Memory 8GB

Network 2Gbps

OS Scientific Linux 6.6

PC3

CiscoCatalyst 2960G

HDD write speed check

2015/4/14 CHEP2015 23

SRS (Scalable Readout System)

2015/4/14 CHEP2015 24

SRS is general purpose multi-channel readout system.User can choose variety of frontend chips.

FECADC

FEC

Front-End Card

FEC

Front-End Card

FEC

Front-End Card

Eurocrate

Front-End ASIC

Front-End ASIC

Front-End ASIC

SRU

Evaluation by using SRS

2015/4/14 CHEP2015 25

SRSADC/EFC

ADC/FEC send test data.Maximum transfer speed of test data is 1Gbps + 1Gbps.We configured that DAQ PC could process all of data.

Storage PCNetwork switch Monitor PC

HDD

HDD

test data

send data once every 10 events

Storage PC Performance

2015/4/14 CHEP2015 26

Emulator Storage PC

Test PC

emulatoremulator

emulatoremulator

CPU Intel(R) Xeon(R) X5650 @ 2.67GHz 6CoresMemory 24GBNetwork 1Gbps x 10OS Scientific Linux 6.4SSD Intel SSD520Series 240GB

only receive data

Throughput of 1 Storage PC is 1000MB/spill.

write data on SSD

test datatrigger rate = 2kHz1 event size = 14kB

DAQ Component & Configuration Example

2015/4/14 CHEP2015 27

InPort OutPort

Service Port(command/status)

Logics (for data handling)

Data

InPort OutPort

Service Port(command/status)

Logics (for data handling)

Data+ =DAQ-Middleware provides user writes

Monitor

DaqOperator

Gatherer

Online Monitor only

Examples of DAQ component combination

Logger

DaqOperator

Gatherer

Logging only

Dispatcher

Logger

Monitor

DaqOperator

Gatherer

Online Monitor & Logging