1

MONITORING AND DIAGNOSTIC OF

MIDDLEWARE SERVICES

Felix Ehm

28th June 2012

2

Introduction

Problem: how can I detect a failure in my systems ?

What is the reason? Host, Network ? Add machine monitoring

Is my program running correctly ? ?

28th June 2012

3

Introduction

Problem: how can I detect a failure in my systems ?

Gain control by exposing process internal information to enable constant monitoring for pre-failure recognition.

JMX for Java Processes CMWAdmin for CMW servers CMX for C/C++ general services Tracing/Central Logging System

28th June 2012 Gina Gorgogianni, CMX Feedback mechanism

4

The Java Management Extension

Java Standard to expose process internal information Inspect data (remotely) via JConsole/Jvisualvm Many monitoring systems support this

28th June 2012

Example for JMS Broker

5

The CMWAdmin GUI

Java GUI to inspect CMW-enabled process Browse and watch information from one server Uses CMW middleware to access data

28th June 2012

CMWAdmin

CMW Servers list from the Directory

Server

6

The CMX Library

A general solution to allow exposure of internal metrics for C/C++ programs.

• Idea origins from JMX: Why can’t we have something like this for C/C++?

• Requirements • Small memory footprint• Non-blocking calls• Metrics: floats & strings

• Project started in 201228th June 2012

9

Architecture

High Level 2 lightweight APIs with non-blocking operations to

○ Update : registers, exposes & updates metrics○ Read : retrieves information for metrics / process

No dependencies

Low Level Main Segment:

table containing the registered processesProcess Segments:

structures containing information on metrics

28th June 2012

Shared Memory

11

CMX Library Characteristics

Very small footprint: 140 KB in memory usage Easy non-blocking API: 10 core functions in total Supports floats and string data types Incorporated input from real-time experts CMX Library is ready for preproduction No dependencies on external libraries Future: Deployment for all cmw servers

But also applicable for other C/C++ projects

28th June 2012

12

Constant Monitoring

28th June 2012

Host Health Process up/down Process service endpoint ok?

E.g. HTTP Server : is wget successful ? Process does what it is supposed to do

13

Constant Monitoring

DIAMON as CO in-house solutionReads metrics and applies rulesEasy to extend though pluggable architectureProvides history of metricsProvides replay functionality

In case of problem detectionDisplays it to OperatorsSends notification via SMS/Mail

28th June 2012

Controls config

JMX

DIAMON

DAQs

CMW CMX

DAQsDAQs

14

The DIAMON Synoptic Viewer

28th June 2012

15

The DIAMON Console

28th June 2012

16

Diamon

View History Data on metrics

28th June 2012

17

The Central Tracing/Logging System

I need more information than just numbers to diagnose a problem!

Log events are helpful Find the point where the program crashes/fails Access to (past) events is required

Problems Frontends are diskless Multi-layer systems implies watching many sources at the same time You get quickly drawn in the amount of information

CMW Project was initiated June 2011 Target: Collect log events from CMW servers for better diagnostic

(n.b. log events = info, debug, error, warning, etc.) Replace previous system

28th June 2012

18

The Central Tracing/Logging System

28th June 2012

Finding/Debugging a problem becomes cumbersome!

Collecting and unifying tracing messages in one central place

Easy correlation of events among many services

?DB

Tracing Server

Equipment Specialist/ Developer

The Tracing/Log GUI

Finding/Debugging a problem becomes cumbersome!

Collecting and unifying tracing messages in one central place

Easy correlation of events among many services

?DB

Tracing ServerAvail. Log Instances

Message Panel

Filter

Incoming log events

Record to File

20

The CMW Tracing Package

C++ client libraryVery lightweight Supports TCP + UDPFile + syslog + STOMP appenderIntegrated with CMW componentsLog level can be changed during runtime

JAVA client library Based on log4jVery easy to integrate with existing JAVA services

28th June 2012

21

The CMW Tracing Package

The Server

Modules○ Converters to accept message○ Broker to distribute data○ FileWriter and Database Writer○ Registry keeping discovered sources

Can be deployed as all-in-one process or separate processes○ Scale horizontally and vertically

28th June 2012

The CMW Tracing Package

C/C++ & Java Libraries for Log Events C/C++ Library for Config Messages

Server Accepts events coming via UDP or TCPStores events in database and filesSends events to multiple receivers

User Interface(s)“online” : Java GUI, Linux Console (web console)“offline” : Database viewer based

UDP based

The CMW Tracing Service

Nearly all CMW services send log eventsProxies, RDA servers, JMS, …

Great help for identifying problems Easy to extend to other protocols

Performance~100M Messages/day6% stored in the DB100% stored additionally

in FilesSystem does very well

Low netw

ork and CP

U load

24

The CMW Tracing Service

Collects also other information than log events What is done, where and when by whom?

Software upgrades / installationsProcess restarts eventsConfiguration changes

28th June 2012

25

Summary

Gain control by exposing process internal information to enable constant monitoring for pre-failure recognition

JMX for Java Processes CMWAdmin for CMW servers CMX for C/C++ general services Tracing/Central Logging System

But: try also to monitor the system as the user sees itJMS : send test message and measure speed

> 100ms = WARNING

28th June 2012

DIAMON

DIAMON

2628th June 2012