the effelsberg direct digitization project

The Effelsberg Direct Digitization Project

Radionet Workshop,

September 2020

Amit Bansod, Jan Behrend, Ewan Barr, Niklas Esser, Mathias Heininger, Sakthi Priya Sathyanarayanan, Tobias Winchen, Jason Wu

Contact: twinchen@mpifr-bonn.mpg.de

2

Direct Digitization

Current and future receivers with integrated digitizer/packetizer unit

UDP Ethernet output (40GbE / 100 GbE/ ...) Increased sensitivity Improved stability RFI mitigation techniques Reduction of hardware complexity ...

Increase science / € Reduced maintenance COTS components (network, computing) Adaptability to new / rare science cases:

» real time transients with machine learning,

» detection of extreme energy particles

3

Data Transport

Standard Ethernet data backbone (Infiniband, RoCE)

Data streamed in multicast groups, client subscribe to group

RobustnessScalability

SPEAD2 Protocol (VDIF)

Digitizer / Packetizer

Data Broker

DataClient

DataClient

DataClient

DataClient

DataClient

4

Processing Clients

COTS Server:MKRECV / MKSEND programs for

network traffics (w. ibverbs)

72+ Gbit/s w. < 0.1% loss PSRDADA ring bufferPSRDADACPP framework

https://github.com/MPIfR-BDG/psrdada_cpp

Processing on GPUs w. CUDA

FGPA processing boards: (ROACH2/SKARAB)

PCIe mounted FPGAs under investigation

Data Client

MK

RE

CV

MK

SE

ND

/V

DIF

SE

ND

AP

P

DA

DA

DA

DA

5

Client Control via KATCP

FPGA

Digitizer / Packetizer

FPGA ControllerFPGA Controller

KATCP Server(Python)

- configure / start / stop

Data Client

MK

RE

CV

MK

SE

ND

/V

DIF

SE

ND

AP

P

DA

DA

DA

DA

Docker Container

KATCP Server(Python)

- configure / start / stop

FPGA ControllerDigitizer/Packetizer Controller

KATCP Server(Python)

- configure / start / stop

All clients encapsulated inside docker containerFlexibilityReproducibility

6

Uniform State Machine for all ComponentsE

rro

r

Idle

Streaming Ready

Set

Running

Pa

nic

Configured

Measurement Stop

Measurement Prepare

Measurement Start

Configure

Deconfigure

Intermediate states during execution (Configuring, capture starting, …)Intermediate states may also result in error state

Capture Start

7

Control Framework: MPIKAT

Abstract Pipeline Implements server with

default katcp commands Provides standard sets

of sensors Unified commandline args System to check provided config

options

Utility Library: Ip address handling Subprocess management Management of dada buffer Numa inspection ...

Source Repository:

https://gitlab.mpcdf.mpg.de/mpifr-bdg/mpikat

Abstract Interface (EDDPipeline)

SpectrometerPipeine

MasterController

SkarabController

PulsarPipeline

...

8

Master Controller

Err

or

Idle

Streaming Ready

Set

Running

Pan

ic

Configured

Unprovisioned ProvisionDeprovision Extends state machine:

`unprovisioned’ Provisioning:

Boot up of individual container on per observation basis

Connect container to requested data streams

Relay commands to individual components

9

Provisioning

?

All container in central docker registry Container launch + build using ansible

(execution manually or by master controller) Semi-automatized allocation of resources:

Why not kubernetes?Not (yet?) NUMA awareRDMA capabilities unclearOverhead vs. Performance ?Manual control of components may be necessaryAnsible used to set up core system

- hosts: gpu_server[0] roles: - pulsar_pipeline- hosts: gpu_server[1] roles: - spectrometer_pipeline

https://gitlab.mpcdf.mpg.de/mpifr-bdg/edd_provisioningRepository:

10

Telescope Connection

MasterController

Eff. FitsWriterInterface

Eff. ScpiInterface

Eff. StatusServer

State DB(Redis)

DockerRegistry

Effelsberg Configuration

Digitizer / PacketizerHigh Speed Data

Observation Data

Control

Mosty telescope – agnostic architecture

© N

. Ta

cken

, M

PIfR

EDD backend

Telescope Control

11

User Frontend

12

Internal Monitoring

Central Logging Monitoring Dashboards

13

Currently Available Data Products

Spectroscopy:High resolution FFT Spectrometer (up to 32M Channels, 2 GHz Input

Bandwidth)» Gated, allows for high speed noise diode switching

Full Stokes Spectrometer (up to 512k Channels, 1 GHz Input Bandwidth))» Gated, allows for high speed noise diode switching

PFB on GPU (6 GHz Bandwidth)PFB on FPGA (Skarab, Roach2)

Basic VLBI Mode2 (4) bit downconversion of baseband + VDIF output

Pulsar PipelinesPulsar timingPulsar search

...

14

Conclusion

EDD: A versatile backend for new generation direct digitization receivers using modern software architecture

Commissioning observations with Effelsberg next week To be deployed at

TNRT 40m,MPG-SKA Prototype Dish ...

Software FOSS:https://gitlab.mpcdf.mpg.de/mpifr-bdg

Contact: twinchen@mpifr-bonn.mpg.de