Diamon/Laser Design proposal

Marek Misiowiec BE/CO/APMay 2010

Core OverviewMulti-tier, distributed, fail-safe system processing diagnostic data collected repeatedly from a variety of sources. Raw data ofdifferent shapes is transformed into unified formats and furtheranalyzed on a middle layer according to well-defined rules.

Actions may be taken, in form of alarms or monitoring informationdelivered to clients. Processed data is made available as JAPCparameters or for dedicated GUIs. It is stored in long term archive

too. Downward communication with the source devices is possible.2

devs

apps DAQ

raw data

unifieddata

SRV

clients

client view

Core Overview Diamon/Laser core responsibilities:

• data acquisition for various sources – push & pull modes• communication through layers, publishing• data transformation, processing, analysis, conditioning• failover, database separation, scaling

complementary functionalities:• rules engine• logging, archiving, configuration updates

non-core:• definitions & rules, data provider’s workflow• GUIs, dedicated client applications

3

Importance of a changeNowadays: upgrades mess

• stale sources, legacy constraints, dusty components meeting only part of the requirements

• partial data, simplistic thresholds, no conditioning monitoring based on alarms - Diamon on Laser

• misuse of the system, overhead• mediocre scaling• badly-behaving sources/clients cannot be detached/isolated

legacy solutions hiding in Laser world• SonicMQ, OC4J, EJBs, Hibernate, Sleepycat

...and no standard CO components in place

4

Importance of a change

We would like to: widely use CO solutions and help to establish others worth it better drive interactions with the bottom and upper tiers consolidate logics in system core be able to draw more conclusions on collected data sustain upgrades, updates, dependency changes

5

Transition

6

JMSNotifier

oc4j-diamon

AlarmArchive

Agent Messages

Checks surveillance

Alarms, filters non registered

Alarms

Result Messages

Sends update/Requests config

CLIC Agents

Central Agents

CMWJMXWIE TWIDO SNMP

PING

jdaemon

Configuration

Agentsto be

configured

configures

commands

commands

Router

CCC Consoles

LASER

ALARMS

YAMI protocol

JMS

Transition

7

JMSNotifier

oc4j-diamon

AlarmArchive

Agent Messages

Checks surveillance

Alarms, filters non registered

Alarms

Result Messages

Sends update/Requests config

CLIC Agents

Central Agents

CMWJMXWIE TWIDO SNMP

PING

jdaemon

Configuration

Agentsto be

configured

configures

commands

commands

Router

CCC Consoles

LASER

ALARMS

YAMI protocol

JMS

OC4J, EJBs, Hibernate,

BerkeleyDB

OC4J, EJBs, Hibernate,

BerkeleyDB

secondSonicMQ

firstSonicMQ

In-house DB

In-house DB

SNMP, CMW, variety of data

formats

directly to devices

not enough data

YAMI specific protocol

Transition

8

JMS

spring AlarmArchive

Agent Messages

Result Messages

Sends update/Requests config

CLIC Agents

Central Agents

CMWJMXWIE TWIDO SNMP

PING

jdaemon

Configuration

Agentsto be

configured

configures

commands

commands

CCC Consoles

YAMI protocol

JMS

LASERPROXY

LASERCMW

Transition

9

spring

AlarmArchive

Result MessagesSends update/

Requests config

CLIC Agents

Central Agents

CMWJMXWIE TWIDO SNMP

PING

jdaemon

Configuration

configures

commands

CCC Consoles

YAMI protocol

JMS

LASERPROXY

LASERCMW

Transition

10

spring

AlarmArchive

Result Messages

Data Acquisition Layer

CLIC AGENTS

CMWJMXWIE TWIDO SNMP

PINGConfiguration

commands

CCC Consoles

JMS

LASERPROXY

LASERCMW

Rules, thresholds, distribution, analysis, publishing...

JAPC

other applications

Technical outcomeThen: unified view of complete data sound construction on CO materials

• OC4J -> Spring, Hibernate -> JDBC Templates, EJBs -> POJOsSonicMQ -> ActiveMQ, multitude of data formats -> JAPCin-house database -> CCDB + Logging

• wider choice of middleware: CMW, YAMI, JMS focusing logics in one place

• rules, thresholds, alarm generation• conditioning: machine mode, working hours, state of subsystems

driving the whole process • detachable sources, clients• less restart/reboot downtime• convergence with similar efforts

11

information exchange

12

japc-ext-yami japc-ext-cmw japc-ext-snmp jms-to-japc proxy japc-ext-?

Core Services

rules thresholds

alarm generator

externalsconditions

publisher w-flow

consoles

client middleware: JMSclientinter

actions

other apps

JAPCprovider

ConfigDB

archiverArchive

DB

Core

DAQ [1] DAQ [2] DAQ [n]

tackleproblematic

sources

Clicagents CMW

srcsLaserWie, Twi

?other

sources

source middleware protocols

Available technologies

Building blocks: CO standard components eg. JAPC CO less standard, but demanded eg. japc-ext-yami Laser dev code base, ready to reuse eg. caching, distribution TIM being refurbished and struggling with similar future other CO projects: INCA

13

Lessons learned from Laser eradication of legacy concepts & technologies pays off to scale well, we need to distribute processing power

• a supervisor over multiple workers• dividing is difficult

separation of concerns single processing unit as a set of loosely coupled actions

• patterns: chain of responsibilities, observable• upward flow: source input -> processing -> publishing

cater for database outages through local caching JMS+RMI as a middleware for all parties involved

• more advanced features of JMS channel help ...but no JAPC concepts were introduced at all

14

Device/property model to clarify the current incoherent view strong incentives:

• speak one language• let everyone fit (Logging)• common tools, well-established in CO world

migration of legacy sources• expose properties• eradicate laser-source as it is

interaction with devices through JAPC calls domain entities:

• Alarm, Alarms, Diagnostic,...

15

JAPC extensions to unify view on a variety of devices/applications

available now:• CMW (japc-ext-cmwrda) and JMS (remote) as most popular

we need some more:• YAMI through japc-ext-yami is already available,

used in new Clic agent• SNMP through japc-ext-snmp under development

16

Bottom tierExperience gained in: current Diamon and Laser

• eg. CMW Alarm Monitor as a huge data acquisition process LHC Concentrators , similar problems INCA data acqusition layer

Progress:

17

source type current planned/done

Clic Agents JMS, YAMI japc-ext-yami

CMW Alarm Monitor Laser gateway JAPC deviceLaser sources C/Java laser-source Laser-to-JAPC proxyWiener, Twido SNMP central agents japc-ext-snmpPing central agent JAPC device

Already developed JAPC extensions current version of YAMI meets our requirements Diamon Clic agent

• fully YAMI-based, JAPC values CMW Alarm Monitor refurbished

• 6000 subscriptions to GM and FESA devices• will be incorporated into new core

proxy between Laser production alarms and JAPC values• for the legacy sources

18

Open questionsStill to think over: best components and collaborations to choose level of distribution and scaling shared memory, distributed cache storage: short-term, long-term, Logging DB rules handling: homemade, proprietary ...and many others

19