Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg - DPG 2005 – HK 41.2 1

New Test Results for the ALICE

High Level Trigger

Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg - DPG 2005 – HK 41.2 2

ALICE High Level Trigger Overview● ALICE: LHC heavy-ion experiment● Up to 15,000 particles per event● HLT input data rate: Up to 25 GB/s● HLT output to storage: 1.2 GB/s● Online reconstruction in several steps

● ADC values → 3D coordinates → track segments → tracks● No latency requirements● Large commodity PC cluster

● Hierarchy levels (HL) to match physical detector layout

1, 2, 123, 255,100, 30, 5, 1, 4,3, 2, 3, 4, 5, 3, 4,60, 130, 30, 5,....................

HL2HL1

HL0

HL0-HL2

HL3

Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg - DPG 2005 – HK 41.2 3

ALICE High Level Trigger Overview

Software framework to transport data through cluster● Requirements: Flexible, efficient, fault-tolerant● Components that communicate via defined interface● Interface does not copy large event data

– Only descriptors– Lowers CPU load for data transport

● Components to define flow of data in cluster● Component templates for application specific functionality● Components can be plugged together to create systems

w. arbitrary functionality● Data-driven architecture

● Components receive data from other components● Process received data● Forward output data to next component

Input

Processing

Output

Input

Processing

Output

Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg - DPG 2005 – HK 41.2 4

Interface Test Results

Benchmarks of Publisher/Subscriber Component Interface● Only interface benchmarked● No shared memory access

● Pentium 4 family influenced by L1 cache size (only 8kB)

● Maximum rate: >110 kHz (Dual Opteron)● CPU overhead f. full event round-trip: <18 μs

Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg - DPG 2005 – HK 41.2 5

HLT in TPC Sector Beamtest

Goals of HLT participation in TPC testbeam– Test of DAQ-HLT interface – Gain real world operation experience

PCI Card

PCI Card

DAQ PC 1

PCI Card

DAQ PC 2

Fast Ethernet

DAQ PC 3

3x 250 GB disk

Optical

Link

Optical

Link

Optical Link

Detector Prototype

Optical Link

Optical Link

Mass

Storage

1 MB/s

HLT PC 1

PCI Card

(RORC)

HLT PC 3

HLT PC 2

PCI Card

(RORC)

PCI Card

(HLT-Out)

Gigabit Ethernet

10 MB/s

Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg - DPG 2005 – HK 41.2 6

TPC Sector Beamtest HLT Software

● EventRateSubscriber measures event receiving rate● StorageWriter writes data to local disc

Optical LinkHLT PC 1

RORCPublisher

SubscriberBridgeHead

HLT PC 3

PublisherBridgeHead

PublisherBridgeHead

EventMerger

StorageWriter

HLT-OutSubscriber

EventRateSubscriber

HDD

HLT PC 2RORC

Publisher

SubscriberBridgeHead

StorageWriter

HDD

Optical Link

Optical Link

Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg - DPG 2005 – HK 41.2 7

HLT in Readout of TRD Beamtest● No optical link interface for TRD prototype● Use HLT output card for DAQ connection● Adapt HLT input component to TRD prototype interface

readout PCI card (ACEX card)

● TRDFormatter checks data, adds header f. optical link● Optional local output via StorageWriter

Detector Prototype

HLT PC

PCI Card

LVDS Link ACEXPublisher

HLT-Out

Monitoring

PCI Card

DAQ PC

HDD

TRDFormatter

Optical Link

StorageWriter

Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg - DPG 2005 – HK 41.2 8

HLT TaskManager Control System● System to control large number of

components in final HLT● Fault-tolerant

● No single-points-of-failure● Hierarchical, master-slave● Flexible configuration● XML and Python based● Interface libraries to communicate

with controlled programsCore

Configuration

Engine

XMLConfigurat

ionFile

Interface

Library

ControlledPrograms

Program State & ActionEngine

EmbeddedPython

Interpreter

TaskManager

ProgramInterfaceEngine

Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg - DPG 2005 – HK 41.2 9

HLT Control System

MasterTaskMana

ger

SlaveTaskMana

ger1

SlaveTaskMana

ger2

RORCPublish

er

StorageWriter

Subscriber

BridgeHead

PublisherBridgeHead

PublisherBridgeHead

HLT-OutSubscrib

er

Event-Rate

Subscriber

– Status & Command flow is black

HLT PC 2

HLT PC 3

EventMerger

SlaveTaskMana

ger0

RORCPublish

er

StorageWriter

HLT PC 1

Subscriber

BridgeHead

– Data flow is yellow

System used to control HLT in TPC Beamtest

Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg - DPG 2005 – HK 41.2 10

Cape Town

Heidelberg

Bergen

Dubna

Tromsø

Global Processing Test● Proof-of-concept global online

grid system● High Level Trigger system

distributed globally● Not just distributed on one cluster● Distributed on five clusters● Bergen & Tromsø,Norway ; Dubna,

Russia ; Heidelberg, Germany ; Cape Town, South Africa

● Test run for more than 15h

● Framework designed for operation in cluster also works globally

Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg - DPG 2005 – HK 41.2 11

Global Processing Test

Setup & procesing steps used

Bergen,Norway

Simulated TPC

Clusterfinding3 PCs

Heidelberg, Germany

Simulated TPCTracking

2 PCs

Tromsø,Norway

Simulated TPC

Clusterfinding2 PCs

Dubna, RussiaSimulated

TPCClusterfinding

1 PC

Capetown, South Africa

Simulated DiMuonTracking + Global

Event Merging7 PCs

Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg - DPG 2005 – HK 41.2 12

Conclusion● ALICE High Level Trigger Data Transport Software has

reached mature state● Performance on current CPUs enough for use in HLT● Stable● Can also operate in globally distributed online grid-like

system

● Used successfully in two beamtest– TPC beamtest for interface tests– TRD beamtest used for production data readout

● Current/future development concentrates on higher layers, e.g. control system