JMS – HIGH LEVEL MESSAGING

Felix EhmCERN BE-CO

Content

Introduction

JMS in the Controls System

Deployment and Operation

Conclusion

Introduction

Introduction to JMS – The API

Java Messaging ServiceMessaging API defined in 2002Decouples Source and DestinationTopic / Queue Concepts for named Destinations

Introduction to JMS – A Broker

Technology originated from the financial sector “Few” sources and large set of readers

A Broker is the central entityAccepts messages from 1..* publisher and

re-distributes to 0..* subscribersProvides Quality of Service for Messages

○ Persistency, Transactions, Buffering, Expiration Policies, Slow Client Handling

Design to distribute data reliably and scalableLoad balancing / fail-over / embedded

deployment scenarios possible

Broker

Introduction to JMS - A Broker

Many broker implementations availableProprietary

IBM WebsphereMQ, SonicMQ, TIBCO, RedHat MRG, …

OpenSourceApache ActiveMQ, Apache Apollo, RabbitMQ, HornetMQ, …

Broker may be implemented in another language than Java (e.g. Qpid & RabbitMQ)

Communication protocol is non-standardBrokers of different vendors are not interoperableAMQP & STOMP address this issue

Introduction to JMS - Request/Reply

“Direct communication” via Broker using Request – Reply mechanism

Due to intermediate Broker two hops are required

Requestor Replier

BrokerRequest

MSG

ReplyMSG

Request Queue

Temporary Reply Queue

creates

sends

sends

Introduction to JMS - Selectors

Filter messages from one Destination usingMessages SelectorsEach message can be enriched

with AttributesSubscriber receives filtered

messages via defined selector

Broker

Subscriber

selector:BUILDING ≠ A

HEADERSCYCLE=ATIME=9845

BODY

JMS

Message

BUILDING=A

BUILDING=B

BUILDING=C

Topic/Queue

Introduction to JMS - Persistency

JMS defines persistent and non-persistent messages

Broker stores message to disk before acknowledging to sender

Avoids loss of data when broker crashes Drawbacks

Higher disk loadSlower message processing

Sender

1. send

3. ack 2. store

JMS in the Controls System

CERN’s Accelerator Complex

CERN’s Accelerator Complex

JMS in the Controls System Purpose

Reliable transport of data between (Java) ProcessesHigh level controls applications + middle tier servers

○ Alarm System○ Data Rendering Services○ (Control) GUIs○ Logging Services○ Beam Security System(s)○ Access Systems, ….

Today vital part of Beam Operations Product choice:

No JMS - No Beam !

JMS in the Controls System - History

HistorySonicMQ since 2001 for CERN’s Alarm

and CO Monitoring systemOpenSource ActiveMQ since 2005

○ Middle tier servers update GUIs○ Middle tier to Middle tier

Phased out in 2013

JMS for Accelerator Controls Early days only Java clients via JAPC

japc-ext-remote contains JMS communication

Today multi-language clientsJAPC clients remainJava using ActiveMQ client libraries directlyC++/Python using STOMP

Deployment and Operation

29 production brokers Single or broker cluster

Large spectrum of usage patternsfrom few topics to 40K topicsfrom few subscribers to >23 K subscribersfrom couple msg / minute to 10K msg / secfrom few bytes/msg to 3 Mbyte/msg

Deployment and Operation

Deployment and Operation

Some numbers : 300 Applications 4’400 Connections 40’000 Subscriptions 85’000 Topics 8 Million msg/h IN,

3.5M msg/h OUT

Archived Uptime in 2012: 99.998%

Not all data which is produced is also

consumed

Deployment and Operation Distribute Load

Interconnect two BrokersConsumers/Producer can

choose any BrokerSharing of work

Smooth Upgrades possibleClient library version != Broker version OK

DrawbacksConsumer might not get all messages in case of

network errorRouting overhead

The STOMProtocol

The STOMProtocol

Simple Text Oriented Message Protocol Defines messaging API + wire format Many open source clients libraries available

Python, C/C++, Java, Perl, Ruby, PHP, Erlang, … Targets the messaging interoperability among

language, platforms and brokers Supported by many brokers Very easy to use

Example to send STOMP in Python

mysocket = socket.socket(socket.AF_INET,socket.SOCK_STREAM)mysocket.connect( ("jms-co-dev", 61680) )

text = '''SEND\n\destination:/queue/CERN.DEPLOYMENT\n\HOST:%s\n\INSTALLPATH:%s\n\PRODUCT:%s\n\TIME:%s\n\persistent:false\n\n\x00''‘

mysocket.sendall(text)mysocket.sendall("DISCONNECT\x00")mysocket.close()

The AMQProtocol

The AMQProtocol

Advanced Message Queue Protocol Defines messaging API + wire format Next evolution after JMS

Consortium of RedHat, Credit Suisse, Microsoft, JPMorgen, Barclays, VMWare, Cisco, …

Released as Version 1.0More and more brokers implement it

Direction for the future ?

Conclusion JMS is vital part of Accelerator Controls System

Performs very well for publish - subscribe scenarios

ActiveMQ was a good choiceRunning reliably Low maintenance and flexible deploymentsProvides many metrics for diagnostic and monitoring

Multi-language applications can use service

Central (redundant) service

Result Example

Messages received/sec over time

time

AV

G M

essa

ges

/ se

c

Result Example Histogram of delayed message distribution

Delay time [msec]

# M

essa

ges

Result Example

Number of “late” (>300ms) messages for test duration

time

Del

ayed

Mes

sage

s

Message Delay Summary Measuring delayed Message

~ 99.32% of messages delivered < 300msRest < 1 sec

Doubling numbers of messages (~9500 msg/sec)100% between 5 and 10 secondsFurther studies for improvement needed

40% less CPU load thanks to optimization

Result Example CPU load

time

% C

PU

lo

ad

Consumer connects

Consumer disconnects

Broker Memory Consumption

Initial Memory is proportional toConnectionsSubscriptions

0 5000 10000 20000 30000 400000

50

100

150

200

250

300

350

400

Heap Memory Usage Live Threads

0 10000 20000 40000 60000 800000

100

200

300

400

500

600

700

800

Heap Memory Usage Live Threads

1000 Subscription / Connection 500 Subscriptions / Connection