!chaos seminary 12_2012

C.

Control system based on aHighlyAbstracted andOpenStructure

seminar 2012

http://chaos.infn.it

Thursday, January 10, 13



What is!CHAOS (Control system based on Highly Abstracted and Open Structure), is an experiment of CSN 5(technological research experiments) at the LNF and Roma-TV section.

The project has been appreciated by educational world and the University of

UniversitàTorVergata LNF - Roma2

Universitàdi Cagliari


Objective

!CHAOS has been designed to widely scale and for manage large apparatus with different needs in term of storage technology and speed for data acquisition


Objective

new topology in the CS that permit to integrate the triggered DAQ.

define a new way to access realtime data

data acquisition, storage and indexing fully scalable


Objective

• 99% of adaptation time of instrument to !CHOAS done by framework

• simplify the creation of the user interfaces

• simplify the creation of the control algorithms basing it on an abstraction of the I/O channels or full dataset not on physical device

A CPP Framework and services that permit to have:


6

!CHAOS Simplify Vision


Implementation

Impl

emen

tatio

nImplem

entation

Control'sAlgorithm

User InterfaceInstrument's Driver

Implementation

Data Cloud

Live Data

History Data

network

DirectoryServerCloud

CHAOS Boundary

User specialized Boundary

Simplify Vision


Implementation

Impl

emen

tatio

nImplem

entation

Control'sAlgorithm


Implementation

Data Cloud

Live Data

History Data

network


CHAOS Boundary


Simplify Vision

Abstract RPC Driver

AbstractEvent Drivernetwork

AbstractDirect I/O


Implementation

Impl

emen

tatio

nImplem

entation

Control'sAlgorithm


Implementation

Data Cloud

Live Data

History Data

network


CHAOS Boundary


Simplify Vision

Abstract RPC Driver

AbstractEvent Drivernetwork

AbstractDirect I/O

Implementation

Data Cloud

Live Data

History Data

network

Implementation

Data Cloud

Live Data

History Data


8

!CHAOS Real Model


Live / History domain

Live / History domain

Data Boundary

Common Toolkit

User Interface Toolkit

Information

ChaosDirectory

Nodes

Common Toolkit

Control Unit Toolkit

Co

mm

on

To

olk

it

Exe

cuti

on

Un

it T

oo

lkit

RPC - Event - Direct I/O

RPC - Event - Direct I/O

RP

C -

Eve

nt

- D

ire

ct I

/O

HT

TP

- A

PI

Node Information

Node Activity

Security

CQL Proxy Nodes

Indexer Nodes

Management Nodes

Abstraction Boundary

Driver for sensor and instruments

User Interface for monitoring and controls

ControlsAlgorithm

User Customization


10

Toolkit and Services

!CHAOS is a set of toolkit and services that will permit the realization of an control system


11

!CHAOS Toolkit and Services

Common Toolkit

CU Toolkit

UI Toolkit

EU Toolkit

DirectoryServices

CQL Proxy

Indexer

Managment

Live and History Data


12

!CHAOS Toolkit


12

!CHAOS Toolkit

implements the fundamental software's layer to RPC, I/O, Event and other’s utility code and for logic

Common Toolkit

abstracts the !CHAOS resources to the device drivers developers.

abstracts the CHAOS resources to the developers that need to develop control and computing algorithms

abstracts the CHAOS resources to the developers that need to develop control and monitoring user interface

CU Toolkit

UI Toolkit

EU Toolkit


!CHAOS Services


!CHAOS Services

DirectoryServices

gives services via RPC (or with other higher transport layers) for get device structured information, system topology

CQL Proxy

Indexer

Management

give the access to insert and query operation on live and history data to other CHOAS node using CHOAS Query

performs the indexing of the new inserted data

performs the management of the datas chunk saved on the storage (erase, move,


14

!CHOAS Scalability and Fault-

Some node needs to be scaled for performance reason or for fault-tolerant:• Directory Nodes• Live and History nodes


15


Server P1

Server P2

Server PN

Device 1Driver

Framework

Server P1

Server P1

Server P2

Server PN

Client ReaderDriver

Framework

Server P2

X

XX

Clients that need to talk to scaled services, has more than one ip address for every service


15


Server P1

Server P2

Server PN

Device 1Driver

Framework

Server P1

Server P1

Server P2

Server PN

Client ReaderDriver

Framework

Server P2

X

XX

Every ip address has a priority, the client, as first address, uses the one with highest priority


15


Server P1

Server P2

Server PN

Device 1Driver

Framework

Server P1

Server P1

Server P2

Server PN

Client ReaderDriver

Framework

Server P2

X

XX

when a server goes down, another ip will be used


16


the re-balance occurs when a server with a highest priority, previously goes down or stopped for maintenance, becomes

up and running

Server P1

Server P2

Server PN

Device 1Driver

Framework

Server P1

Server P1

Server P2

Server PN

Client ReaderDriver

Framework

Server P2

use at TS1

use atTS1

ok


16


Server P1

Server P2

Server PN

Device 1Driver

Framework

Server P1

Server P1

Server P2

Server PN

Client ReaderDriver

Framework

Server P2

use at TS1

use atTS1

ok

the new server will be used starting from a pre-determinated time stamp (TS1)


16


Server P1

Server P2

Server PN

Device 1Driver

Framework

Server P1

Server P1

Server P2

Server PN

Client ReaderDriver

Framework

Server P2

use at TS1

use atTS1

ok

all the operations and the calculation of TS1 are coordinated by one of the directory nodes


11/dic/2011 17

!CHOAS Communication


18

!CHOAS has three different communication system:

RPC, used for command (messages that need to be securely received and recorded and for message that need a response [ asynchronous ])

Events, used for inform to a wide range of nodes, that something has happened (device heartbeat, alert on device channel, received command, etc..)

Communication Systems


19

Every !CHAOS node has at last one client and server instance

of any of the three system

Communication Systems


09/dic/2011 20

•RPC layer is implemented as a plug-ins System, it's is abstracted to internal CHAOS framework so we con change it

• used to send important messages that need to be sure of it's delivering status

• can be used to receive asynchronous responses

• the set command for input channel on control unit is done with RPC

RPC System


21

RPC System

Node A Node B

Messagge forwarding

ack [sync]Node A Node B

Response forwarding

ack [sync]

Message Answer [optional]


21

RPC Message Flow• request message is constructed with BSON, serialized and sent to RPC end point

• if the requester want a response it put answer-code and sender-address in BSON pack,and automatically RPC System wait for the answer

• the request and response involves the forwarding of the ack message, from the receiver to the sender in

RPC System

Node A Node B

Messagge forwarding

ack [sync]Node A Node B

Response forwarding

ack [sync]

Message Answer [optional]


09/dic/2011 22

• Event is implemented as a plug-ins System it's is abstracted to internal CHAOS framework so we con change it

• there are four type of event:

• Alert (highest priority)

• Instrument

• Command

• Custom

• all information are packed into 1024 byte and transported by protocols that support multicast (UDP, PGM[http://en.wikipedia.org/wiki/Pragmatic_General_Multicast])

Event System


http://en.wikipedia.org/wiki/Pragmatic_General_Multicast%5D




09/dic/2011 23

Some numbers about event data

• Every event type has 2^16 sub type

• Every sub type can have up to 2^16 priority level

• Every subtype can have up to 255 byte of information

Event System


09/dic/2011 24

• Direct I/O is implemented as a plug-ins system

• used to make insert or select into the live and the history data

• can be used as stream for receive result data of the query or read the stored raw data file of the devices

Direct I/O System


11/dic/2011 25

!CHAOS Abstraction of the


26

!CHAOS use a structured serialization for data description and serialization

• BSON (http://bsonspec.org/) has been chosen, very similar to JSON but it is binary

• serialization is wrapped into a CPP class so it can be changed infuture

• it is used in:•Hardware dataset description's and acquired data packaging (“channel desc” = value)

•RPC message•Chaos Query Language

Abstraction


http://bsonspec.org

http://bsonspec.org

11/dic/2011 27

•Hardware attributes and I/O channels are described into the DATASET

•Each attribute is defined by•Name•Description•Type (kind+basic type ex: VOLT32, CUSTOM_STR, etc...)•Cardinality (single or array)•Flow (Input, Output, Bidirectional)•Range

Hardware Dataset


11/dic/2011 28

VOLT_CHANNEL_

RESET

DATASETname:VOLT_CHANNEL_1type:VOLT32flow:outputrange:1-20card: 1

name:VOLT_CHANNEL_2type:VOLT32flow:outputrange:1-20card: 1

name: RESETtype: BYTEflow: inputcard: 1

VOLT_CHANNEL_

Hardware Dataset


11/dic/2011 29

Hardware DATASET

The JSON visualization of BSON DS representation

{“device_id”: DEVICE_ID

“device_ds”: { "name" : "VOLT_CHANNEL_1", "desc" : "Output volt...", "attr_dir" : 1, "type" : 4, "cardinality" : 1 ! }

....

}

VOLT_CHANNEL_

RESET

VOLT_CHANNEL_


11/dic/2011 30

!CHAOS Unit toolkit


11/dic/2011

CUToolkit help the Control Unit development

Developer need to extend only one class to create a CHAOS driver, the “AbstractControlUnit”

AbstractControlUnit expose all the APIs needed for interacting with !CHOAS

CU Toolkit


11/dic/2011

Control Unit

AbstractControlUnit is an abstract cpp class, that force developer to implement some method

Common Toolkit



11/dic/2011

•defineActionAndDataset

•init•run•stop•deinit

AbstractControlUnit

Control Unit

AbstractControlUnit is an abstract cpp class, that force developer to implement some method

Common Toolkit



Common Toolkit


CUToolkit featuremulti-threading messaging dispatcher; every CU has it’s own thread for dispatching messages after it has been received by RPC System

BSONPACK

sent from

CommonToolkcpp method:

setDatasetAttribute


Common Toolkit



BSONPACK the RPC dispatcher

find destination

sent from


setDatasetAttribute


Common Toolkit




find destination

CU Message Queue is filled

sent from


setDatasetAttribute


Common Toolkit




find destination

BSONPACK

CU Message Queue is filled

BSONPACK

BSONPACK

sent from

Pack is sent to CUToolkit

scheduler

CommonToolk AbstractControlU

cpp method: setDatasetAttribute


11/dic/2011 34

!CHAOS UI Toollkit


UIToolkit helps the user interface development

Gives API that help to develop: monitoring processes, device controller etc...

Gives an high performance queue using "locking free" design. The queue can manage quantizations , over time, for the dataset data acquisition

CU Toolkit

UIToolkit

Live Acquisition Thread Pool

Absr

actIO

Drive

r

Absr

actIO

Drive

r

DeviceControl DeviceControl

InterThread Cache Logic

Network


11/dic/2011 36

!CHAOS EU Toolkit


helps the development of distributed algorithms (Execution Unit) that can be used to compute or automate the devices control

The EU can have, in input the device channels or all the datasset description, so it work on a "Class" of device not directly on one device

EU Toolkit


EU Toolkit

Input Parameter Output Parameter

ExecutionQueue

Result Queue

Specialized Execution Unit

Abstract Execution Unit

Input Param Pack Output Result Pack

scheme of flow of input or output of an EU


EU Toolkitexception could be concatenated creating a sequence of any combination of:

• Computing algorithm• Control algorithm


ExecutionQueue

Result Queue



Input Param Pack Output Result Pack


ExecutionQueue

Result Queue



Input Param Pack


11/dic/2011 40

!CHAOS Data Management


41

Data Management

!CHOAS has two different management system for the data apparatus controlled environment data

Live DataHistory Data


11/dic/2011 42

!CHAOS Live Datalive

cachedev_1

push data

ui_1

ui_2

eu_1


43

Live Data

The system of the Live Data, is ideated for the sharing of the real time state of the instruments (CU) to other nodes


44

Live Data

livecachedev_1

push data

ui_1

ui_2

eu_1

Every instrument pushes the data of his I/O channels into the shared cached through the controller CU


44

Live Data1. every node can read data from the central cache at any

time[polling]

2. the nodes can register to central cache for receive update[push]

livecachedev_1

push data

ui_1

ui_2

eu_1


45

Live Data

livecachedev_1

push data

ui_1

ui_2

eu_1

livecachedev_2

push data

scalability & reliability

All data of all instruments are distributed across many cache

server


46

Live Data

livecachedev_1

push data

ui_1

ui_2

eu_1

livecachedev_2

push data

X

scalability & reliability

with the algorithm previously shown, when one server goes down, all client that push's, or read, data on that server, use another server, with highest

priority


47

Live Data

The usage of the two modes permits to manage, more efficiently, the bandwidth of the flows of the real time data (read and write)

not all nodes need realtime data at the same speed:• the mode (1) is for monitoring and control• the mode (2) is for control algorithm that can’t lose packages


11/dic/2011 48

!CHAOS History Data


49

Live & History Data

Storage Driver

Index Driver

Com

mon Layer

Abstract History Logic

Proxy CQL Interpreter

Cache D

river

Cache Logic

ChaosQL

the live data management is merged into history data management into the ChaosQL node with the Cache Driver


50

History Data

The System for History Data has been designed to efficiently store the serialized data pack, manage different type of index on demand, and managing data file


History Data

Storage Driver

Index Driver

Common LayerAbst

ract

His

tory

Log

ic

CQ

L In

terp

rete

r

Storage Driver

Index Driver

Common LayerAbst

ract

His

tory

Log

ic

Man

agem

ent I

ndex

Log

ic

Storage Driver

Index Driver

Common LayerAbst

ract

His

tory

Log

ic

Stor

age

Man

agem

ent L

ogic

It is realized by three different services and two

ChaosQL Indexer Management

Storage Driver Index Driver

Storage Driver Index Driver


52

Chaos Query Language

ChaosQL implement the following operation on data• push history/live data per device• create index on device::attribute::{rule}• delete index on device::attribute• retrieve data with logic operation on attribute and index


53

History Data

Storage Driver

Index Driver

Common LayerAbst

ract

His

tory

Log

ic

CQ

L In

terp

rete

r

ChaosQL

Manages the insert and select operations by parsing the Chaos Query Language

Insert - as fast as possible stores the data into the storage system

select - decodes the query, asks at the indexer the position in the storage of the data that match the


54

History Data

All nodes, using the ChaosQL Driver (that uses direct I/O) can query the history and live data.

the results are received in asynchronous mode

Storage Driver

Index Driver

Comm

on Layer



UI or CU or EU

UI-CU-EU Toolkit

Chaos Common

Query Re

sult

CQL I/O Driver

Direct I/O Driver


11/dic/2011

Index

Storage

CommonTK

CUTK

CQL Proxy

CQL Proxy

CQL

Prox

y

Indexer Cluster

Management


11/dic/2011

Index

Storage

CommonTK

CUTK

CQL Proxy

CQL Proxy

Push Data

CQL

Prox

y

Indexer Cluster

Management


11/dic/2011

Index

Storage

CommonTK

CUTK

CQL Proxy

CQL Proxy

Push Data

CQL

Prox

y

Indexer Cluster

Store Data

Management


11/dic/2011

Index

Storage

CommonTK

CUTK

CQL Proxy

CQL Proxy

Push Data

CQL

Prox

y

Indexer Cluster

Store Data

Launch Index

Management


11/dic/2011

Index

Storage

CommonTK

CUTK

CQL Proxy

CQL Proxy

Push Data

CQL

Prox

y

Indexer Cluster

Store Data

Launch Index

get data pack

Management


11/dic/2011

Index

Storage

CommonTK

CUTK

CQL Proxy

CQL Proxy

Push Data

CQL

Prox

y

Indexer Cluster

Store Data

Launch Index

get data pack

Store Index

Management


11/dic/2011

Index

Storage

CommonTK

CUTK

CQL Proxy

CQL Proxy

CQL

Prox

y

Indexer Cluster

Management


11/dic/2011

Index

Storage

CommonTK

CUTK

CQL Proxy

CQL Proxy

CommonTK

EUTK

UITK

CommonTK

CQL

Prox

y

Indexer Cluster

Management


11/dic/2011

Index

Storage

CommonTK

CUTK

CQL Proxy

CQL Proxy

CommonTK

EUTK

UITK

CommonTK

CQL

Prox

y

Query Live Data

Indexer Cluster

Management


11/dic/2011

Index

Storage

CommonTK

CUTK

CQL Proxy

CQL Proxy

CommonTK

EUTK

UITK

CommonTK

CQL

Prox

y

Query Live Data

Query History

Data

Indexer Cluster

Management


11/dic/2011

Index

Storage

CommonTK

CUTK

CQL Proxy

CQL Proxy

CommonTK

EUTK

UITK

CommonTK

CQL

Prox

y

Query Live Data

Query History

Data

Indexer Cluster

send search on

idx

execute search on idx

DB

Management


11/dic/2011

Index

Storage

CommonTK

CUTK

CQL Proxy

CQL Proxy

CommonTK

EUTK

UITK

CommonTK

CQL

Prox

y

Query Live Data

Query History

Data

Indexer Cluster

send search on

idx


DB

get pack at location

Management


11/dic/2011

Index

Storage

CommonTK

CUTK

CQL Proxy

CQL Proxy

CommonTK

EUTK

UITK

CommonTK

CQL

Prox

y

Query Live Data

Query History

Data

Indexer Cluster

send search on

idx


DB

get pack at location

Receive History

Data

Management


57

History Data

the query can be splitted in more than one time period from the receiving proxy.

The splitted query are sent to other proxy that will process the sub query

Storage Driver

Index Driver

Comm

on Layer



UI or CU or EU

UI-CU-EU Toolkit

Query

Resu

lt

Storage Driver

Index Driver

Comm

on Layer



Storage Driver

Index Driver

Comm

on Layer



Result

CQLPackOriginal

CQLPack1

CQLPack2

CQLPack3

Sub Query

Result

QueryChannel

ChaosCommon

CQL I/O Driver

Direct I/O Driver


58

History Data

Executes the necessary operations to index the data saved on the storage system

two type of index:• TMI - Time Machine index• DSI - Dataset Index

Storage Driver

Index Driver

Common LayerAbst

ract

His

tory

Log

ic

Man

agem

ent I

ndex

Log

ic

Indexer


59

History Data

TMI - Time Machine index•This index permits to catalog every data pack for a

device for the timestamp. • all the data pack are recorded and associated to their

timestamp• this index is managed by system


60

History Data

DSI - Dataset Index•This index permits to catalog attributes of the dataset

for a determinated instrument• there will be possibility of customize the index

algorithm• this index is created on demand and on specific time

periods


61

History Data

Executes management operations on files that store the data packs

some data pack could be subject to ageing

the ageing state will determine the

Storage Driver

Index Driver

Common LayerAbst

ract

His

tory

Log

ic

Stor

age

Man

agem

ent L

ogic

Management


23/02/12INFN-Vitrociset meeting

Directory server

CU Toolkit

CU

modules (init)

CU

modules (config)

CU

modules (set)

CU

modules (de-init)

Control UnitC

U m

odules (stop)

infrastructure

Control Unit

CU

modules (init)

CU

modules (config)

CU

modules (set)

CU

modules (de-init)

CU

modules (stop)

CU-LV interface

distributedHistory

Data service

distributedLive Dataservice

I/O m

odules

I/O Unit Exec Unit

I/O m

odules

I/O m

odules

EU m

odules

EU m

odules

EU m

odules

Common Toolkit

C User Interface LV User Interface MatLab UI

User Interface Toolkit 20%

Common Toolkit

Project Status


23/02/12INFN-Vitrociset meeting

infrastructure

developments’ details

LV User Interface: prototype in preparation (shlib for LV)

Live Data: single-instance (no-distributed) prototype ready

RPC client/server: prototype ready (msgpack and zmq impl)

History Data service: performance test (mongodb) ongoing (UniTV)

Directory services: DS simulator ready; DS under developmentscalability & availability: test facility in preparation

UI Toolkit: prototype ready

Common Toolkit: prototype ready

CU Toolkit: prototype ready

CU modules: prototype ready (C++)

CU-LV Interface prototypes& modules: almost ready

QT User Interface: prototype ready (Qt)

Event client/server: prototype ready (boost asio impl)Direct I/O c/s: todo

C wrapper UI Toolkit: prototype ready


64

...closing

in the next year all development force will be focused to realize the system for the history data and for the directory service, a prototype for each system will be available for 2013 2Q


65

thanks for the time

CHOAS GroupClaudio Bisegni, Luciano Catani,

Giampiero Di Pirro, Luca Foggetta, Mazziteli Giovanni, Matteo Mara,

Antonello Paoletti, Stecchi Alessandro, Federico Zani

thesis'sFlaminio Antonucci (tor Vergata

Ingengeria), Andrea Capozzi (Tor Vergata Ingengeria), Francesco

Iesu (Cagliari Infromatica),


!chaos seminary 12_2012

Technology