Event Management. EMU

Graham Heyes April 2008

Overview

Background

Requirements

Solution

Status

Background

CODA is a data acquisition framework for all halls.

Version 1 of CODA developed in early 1990’s.Used in detector tests and hall-C startup.

Version 2 of CODA based on lessons learned and targeted at hall-B.

Version 3 is based on lessons learned from earlier versions and is targeted at 12 GeV operation.

CODA 2 data acquisition systemROC digitizes individual events, EB merges data streams, ET gives access to data for online analysis, ER records data to a file.

CODA version 2 limitations

ScalabilityUp to 32 front ROCs.

One EB.

One ER.

ThroughputNo pipeline readout, event-by-event trigger.

Single data path.

Single event building thread.

12 GeV Requirements

ScalabilitySingle crate testbed -> multi-hundred crate system.

RatesPipeline readout.

Cosmic ray tests -> 200 KHz+ event rates.

Hundreds of megabytes/S to storage.

Lead time -don’t want to lock down some technologies too early.

Step back and look at the problem

DAQ system is a chain or mesh of components.

Each component has four core functions:

It can be controlled (and by implication be monitored).

It reads data from somewhere.

It processes that data in some way.

It sends the processed data somewhere.

Introducing The EMU

A framework for processing data:Controlled and monitored by experiment control.

Configured by XML file or by message from control system.

Reports to outside world via cMsg messages.

Dynamically loads software modules that do all the hard work.

Manages data transport.

Manages processing threads.

Manages metadata such as event count, run number and user defined parameters.

Data transport

The EMU software defines an abstract layer that interfaces to the software/hardware used to move events.

This allows the DAQ to be prototyped without forcing an early choice of data transport technology.

MetadataXML configuration.

XML converted into a hierarchy of “data objects”.

Metadata accessed by a simple API, similar to xpath.

Metadata can be converted back into an XML string.

Example:

getValue(“component/status/run_number”);

setValue(“component/status/state”,”active”);

In practice these are static methods of the Configurer class. So the syntax is:

Configurer.setValue etc

XML dump of metadata

Module types

The way EMUs manage data is chosen at runtime by loadable modules. Initially there will be five.

ReadInput from network output to an event queue.

SendInput from an event queue output to the network

StoreInput from an event queue output to a file, manages file naming.

Build

Input from one or several event queues, process then output to a queue.

Level 3 trigger interface

Input from an event queue, events passed to L3 farm, returning events put on output event queue.

Loadable modules

Modules written in C, C++ or Java.

Modules interface to framework in three ways:

Metadata, see previous slide.

Command mailbox.

Status object.

A thread watches for commands appearing in the mailbox.

One or more worker threads process the data.

EMU manages threads.

Example 1From Left to right: EMUs in first column handle a subset of ROCs. EMUs in second column build full events. EMUs in third column interface with L3. EMUs in fourth column store to disk.

Example 2Similar to example 1 except a second crossbar allows data to be stored according to event type.

Example 3Here ROCs send odd numbered blocks of events to EMU1 and even numbered ones to EMU2. The issue with this design is that each EMU in column A must be able to read from EVERY ROC. With CODA 2 it was found that context switching latency between threads reading data was the limiting factor in the number of open connections. This may no longer be the case.

Status

First prototype EMU:Written in C and C++.

Modules were “compiled in”.

Simulated Example 1 (slide 13) at a rate of 20,000 blocks per second. Small blocks to test latency of handling blocks. Factor of 10 better than needed.

Second prototype Outer framework rewritten in Java.

XML parsing and metadata handling working.

Modules are loadable.

To Do

Rework read and send modules from prototype 1 to fit new framework.

Write a “real” event building module that modifies event format etc.

Write the event storage module.

Test a small system.

Test a large system.

The real McCoy

The metadata objects “know” how to display themselves using Java.

The component name, the manifold and modules appear as a result of pressing the configure button.

The status area is generated by the EMU software.

Messages in lower window go to cMsg.

These messages logged from the loaded module.

Remember for next slide.

try {Configurer.newContainer("component", "status");Configurer.newValue("component/status", "state", "");Configurer.newValue("component/status", "events", "0");Configurer.newValue("component/status", "data_count", "0");Configurer.newValue("component/status", "run_number", "0");Configurer.newValue("component/status", "run_type", "unknown");Configurer.newValue("component/status", "run_start_time", "00:00:00");Configurer.newValue("component/status", "run_end_time", "00:00:00");

} catch (DataNotFoundException noContainer) {

if (run_state != old) {Logger.info("State Change", run_state.toString());try {

Configurer.setValue("component/status/state", run_state.toString());} catch (DataNotFoundException e) {// This is almost impossible but catch anyway

Logger.error("CODAComponent thread failed to set state");}

}

Creating the status metadata

Updating the state

Conclusion

Control and configuration is a large part of the problem.

Start with a platform independent framework with well defined interfaces.

Implement something that works.

Re-implement modules and underlying data transport as needed.