london jbug - connecting applications everywhere with jboss a-mq
DESCRIPTION
Presentation by Rob Davies (Red Hat) delivered at the London JBoss USer Group event on the 10th of July 2013. See the video here: http://youtu.be/VrJyKLAXVTg As part of the FuseSource acquisition, Red Hat now supports Apache ActiveMQ as the recently released Red Hat JBoss A-MQ product. ActiveMQ is the most widely used message-oriented middleware that uses messaging to connect remote applications written in Java, C/C++, Python, Perl, Ruby, and many other languages. ActiveMQ is standards based and supports messaging protocols such as AMQP 1.0, WebSockets, Stomp, OpenWire, and MQTT. In this session, Rob Davies will discuss the product’s features and functionality, and demonstrate connectivity from a microprocessor to web sockets using JBoss A-MQ.TRANSCRIPT
1 redhat Rob Davies
Connec(ng Applica(ons Everywhere with JBoss A-‐MQ
Rob Davies, Technical Director, Fuse Engineering, Red Hat July 2013
2 redhat Rob Davies 2
Rob Davies Technical Director, Red Hat -
h"p://www.redhat.com ■ Software projects:
■ Apache ActiveMQ, ■ Apache Camel ■ Apache ServiceMix
■ On Expert Group for JSR 343: JMS 2.0 ■ Co-‐author of AcCveMQ in AcCon:
3 redhat Rob Davies
History of Fuse open source Development
2004 2005 2006 2007 2008 2009 2010 2011 2012
ActiveMQ created May 2004
The CodeHaus
ServiceMix created May 2005
Camel created March 2007
IONA acquire April 2007
Karaf created
Progress acquire October 2008
FuseSource created October 2010
The Apache Software Foundation
LogicBlaze Formed May 2005
CXF August 2006
Fabric created February 2011
Fuse Forge And ASF
Red Hat acquire Sept 2012
4 redhat Rob Davies
Why use Message-‐Oriented Middleware? • Robustness to change • Time Independence • LocaCon Independence • Hide Latency • Scalability • Event driven • Simplicity • Configurable Quality of
Service • PlaRorm and Language
IntegraCon • Fault tolerant
5 redhat Rob Davies 5
What is Apache ActiveMQ ?
• Top Level Apache Software Foundation Project
• Wildly popular, high performance, reliable message broker
• Connects to nearly everything
• Native Java, C/C++, .Net,
• AMQP 1.0, MQTT 3.1, STOMP (1.0-1.2) and OpenWire
• STOMP enables Ruby, JS, Perl, Python, PHP, ActionScript …
• Embedded and standalone deployment options
• The key component of JBoss A-MQ!
6 redhat Rob Davies
Messaging: The Basics
7 redhat Rob Davies
Message Channels and RouIng
• Message Channels • Named communica(on between interested par(es • JMS calls them ‘Des(na(ons’
• Can “tune-‐in” to mul(ple channels using wildcards • Can fine-‐tune message consump(on with selectors • Can route a message based on content
8 redhat Rob Davies
Message Channels = JMS DesInaIons
Producer
Broker Consumer
Consumer
Consumer
Destination WIDGET
Destination ORDER
9 redhat Rob Davies
Point-‐to-‐Point Channel: JMS Queues
Producer
Consumer
Consumer
Consumer
Queue
10 redhat Rob Davies
Publish/Subscribe Channel: JMS Topics
Producer
Consumer
Consumer
Consumer
Topic
11 redhat Rob Davies
Message RouIng : Selectors
DestinationProducer
Consumercolor='blue'
Consumercolor='red'
12 redhat Rob Davies
Message RouIng : DesInaIon Wildcards
Topic:BAR.BEER
Consumertopic:BAR.>
Topic:BAR.WINE
Producer
13 redhat Rob Davies
Messaging: Features every Message broker should have
But some don’t…
14 redhat Rob Davies
Exclusive Consumers
Producer
Consumer
Consumer
Consumer
Queue
15 redhat Rob Davies
Message Groups
• Like Exclusive Consumers – but in parallel • Guaranteed ordering of related messages across a
Queue • But – load balancing of messages across mul(ple
consumers • All messages with the same JMSXGroupID go to the
same consumer • How you group messages is down to the applica(on’s
producer • To explicitly close a group, set the JMSXGroupSeq to
-‐1
15
16 redhat Rob Davies
JBoss A-‐MQ – Synchronous sends
Message Broker
Message Producer
Consumer
Consumer
Consumer
17 redhat Rob Davies
JBoss A-‐MQ – Synchronous sends
Set alwaysSyncSend on the ActiveMQConnectionFactory
You can set: sendTimeout on the ActiveMQMessageProducer
Synchronous send: JMS Client
producer.send(session.createTextMessage("Hello"), new AsyncCallback() { public void onSuccess() {} public void onException(JMSException exception) { exception.printStackTrace(); }});
Performance Tip – use callbacks …
18 redhat Rob Davies
Deploying JBoss A-‐MQ
• AcIveMQ Can run standalone or embedded • As a standalone, or part of a highly available message
broker cluster • Embedded – easy to use an en(re broker in JUnit tests (no
need to Mock) • In Tomcat, deployed as a war • In JEE Server – either co-‐locate – or use client with JCA
• Ac(veMQ distribu(ons contain a rar for this purpose
18
19 redhat Rob Davies
JBoss A-MQ: Message Storage
20 redhat Rob Davies
Message Delivery Mode
Messages can be persisted – allowing for decoupled applicaIons
For Queues, any message delivered as PERSISTENT must be stored on long term storage before being delivered to a consumer
For Topics, a message delivered as PERSISTENT will only be stored if there exists a durable subscriber
NON-‐PERSISTENT messages may also be stored on disk if the memory limits of the broker have been reached
20
21 redhat Rob Davies
Message stores supported by JBoss A-‐MQ
■ SQL (JDBC) ■ SQL (JDBC) with journal ■ AMQ Message Store ■ Kaha ■ KahaDB ■ LevelDB
21
Slow but popular
Don’t use (deprecated) The “current” default
The “best yet”
22 redhat Rob Davies
LevelDB Store vs KahaDB
• Fewer index entries per message than KahaDB • Faster recovery when a broker restarts • LevelDB index out-‐perform Btree index at sequen(al access . • LevelDB indexes support concurrent read access. • Pauseless data log file garbage collec(on cycles. • Fewer IOPS to load stored messages. • It exposes it's status via JMX for monitoring
23 redhat Rob Davies
AcIveMQ: LevelDB Store
24 redhat Rob Davies
JBoss A-MQ: Protocols
25 redhat Rob Davies
OpenWire
• Advantages: • Fast – op(mized for
Ac(veMQ • client failover • automa(c reconnect • Client load balancing • Flow control • Many advanced features
25
Disadvantages:
• Not a recognized standard
• Only Java, C/C++/.Net
http://activemq.apache.org/openwire.html
26 redhat Rob Davies
MQTT h`p://mq`.org
Advantages: • M2M/”Internet of Things”
transport • Proposed as an OASIS
standard • Extremely light weight • Growing support from
vendors and OS products • WebSphereMQ • Ac(veMQ + Apollo • Mosquilo • RabbitMQ
26
Disadvantages:
• 3.1 does not support Queues
• Advanced features not standard
• Flow control
• Failover etc.
27 redhat Rob Davies
AMQP – see www.amqp.org
Advantages: • AMQP 1.0 OASIS standard • Proposed as an OASIS
standard • Commodi(zes the Broker
27
Disadvantages:
• One size doesn’t really fit all
• Currently no plans for IBM or Tibco to adopt it
28 redhat Rob Davies
STOMP – h`p://stomp.github.com
Advantages: • Easy to use text based
protocol • Can use telnet as a client • Defacto standard:
• Ac(veMQ + Apollo • HornetQ • RabbitMQ • PocoMQ • StompServer • OpenMQ • Many more …
28
Disadvantages:
• Not as fast as binary formats
29 redhat Rob Davies
But there’s more: WebSockets STOMP is a natural wire protocol for WebSockets And so is MQTT!
29
ActiveMQ
Client(Browser)
WebSocketConnector
StompConverter
ActiveMQ(Backend)
TCP(WebSocket)
HTTP(S)Request/Response
30 redhat Rob Davies
JBoss A-MQ: Enterprise Features – Failover …
31 redhat Rob Davies
High Availability
ActiveMQ Cluster
ActiveMQ Client
Supported by both Java and C++ OpenWire clients
32 redhat Rob Davies
JBoss A-MQ: Enterprise Features – High Availability …
33 redhat Rob Davies
• JDBC Master/Slave • Shared File System Master/Slave • Replicated LevelDB Master/Slave
Master Slave Configurations
34 redhat Rob Davies
Broker Slave
JDBC Master/Slave
Broker Master
ActiveMQ Client
Broker Slave
Database
35 redhat Rob Davies
• Extreme reliability – but not as fast • Recommended if already using an enterprise database • No restric(on on number of slaves • Simple configura(on • Configurable lockKeepAlivePeriod
JDBC Master Slave
36 redhat Rob Davies
Broker Slave
Shared File System Master/Slave
Broker Master
ActiveMQ Client
Broker Slave
File System
Larry’s Removals
37 redhat Rob Davies
• Recommended if you have a SAN, or DRDB • No restric(on on number of slaves • Simple configura(on • Ensure file locking works – and (mes out – NFSv4 good!
Shared File System M/S
38 redhat Rob Davies
Local File System Local File System Local File System
Broker Slave
Broker Master
Broker Slave
ZooKeeper Cluster
Replicated LevelDB Master Slave
39 redhat Rob Davies
ZooKeeper Cluster
Local File System Local File System Local File System
Broker Slave
Broker Master
Client
Broker Slave
Replicated LevelDB Master Slave
Broker Master
40 redhat Rob Davies
• Requires a HA ZooKeeper Cluster • No Single Point of Failure • Dynamic number of slaves • Simple configura(on • Sync or Async Replica(on
Replicated LevelDB Master Slave
41 redhat Rob Davies
• Link Brokers together • Use Store and Forward • Are uni-‐direc(onal by default • All Des(na(ons are global
Network of Brokers
42 redhat Rob Davies
Apache AcIveMQ – Broker Topologies
Store and Forward
ActiveMQ Local broker
ActiveMQ Remote broker
43 redhat Rob Davies
Apache AcIveMQ – Broker Topologies
Bi-directional network
ActiveMQ Local broker
ActiveMQ Remote broker
44 redhat Rob Davies
<networkConnectors> <networkConnector name="backoffice" uri="static://(tcp://backoffice:61617)" duplex="true"> </networkConnector> </networkConnectors>
Bi-directional Config
45 redhat Rob Davies
<networkConnectors> <networkConnector uri="static:(tcp://remote:61617) "/> <dynamicallyIncludedDestinations> <queue physicalName=”free.food.>"/> <queue physicalName=”free.beer.>"/> <topic physicalName=”cricket.scores.>"/> </dynamicallyIncludedDestinations> </networkConnectors>
Reducing Crosstalk
46 redhat Rob Davies
Broker Networks (Store and Forward)
47 redhat Rob Davies
Network of Brokers : Geographically Dispersed
48 redhat Rob Davies
Network of Brokers : Network with Master/Slave
49 redhat Rob Davies
Scaling Networks of Brokers
• Brokers share rou(ng informa(on across networks • All des(na(ons are considered Global • This is really convenient • Though it starts to get problema(c with 1000’s of
brokers • However – we can do filtering to shape the traffic
across networks • But this involves a lot of manual configura(on
49
50 redhat Rob Davies
• Brokers share rou(ng informa(on across networks • All des(na(ons are considered Global
• This is really convenient
• Though it starts to get problema(c with 1000’s of brokers
• However – we can do filtering to shape the traffic across networks
• But this involves a lot of manual configura(on
50
Time to talk about Apache Camel …
Scaling Networks of Brokers
51 redhat Rob Davies
AcIveMQ with embedded Camel
Service
Consumer
ConsumerActiveMQ Broker
Flexible and is faster J
52 redhat Rob Davies
Scaling Networks – use Apache Camel • Allows for non-‐chaly networks to be established
• Rou(ng informa(on can be externalized to the broker
• Successfully used in produc(on for 1000’s of brokers today
52
53 redhat Rob Davies
More problems with Large Deployments
54 redhat Rob Davies
Problems – Deploying and maintenance
Main problems • Installing brokers on mul(ple hosts
o ssh, untar, set directories and environment
• Sesng configura(on manually for every broker o copying xml config, tweaking, tes(ng
• Upda(ng configura(on across cluster • Upgrading brokers
It’s a tedious and error-prone process
55 redhat Rob Davies
Problems – TradiIonal best-‐pracIce Ips
• Keep XML as a template and configure node-‐specific details through proper(es
• Keep configura(on in SVC system (git, svn, …)
• Keep configura(on separate from installa(on for easier upgrades
56 redhat Rob Davies
Problems -‐ Clients
• Topology is very “sta(c” • Clients need to be aware of topology • Clients need to know broker loca(ons • Changes are not easy as clients need to be updated • Adding new resources (brokers) requires client updates • Not suitable for “cloud” deployments
Fuse Fabric makes deployments more “elastic”
57 redhat Rob Davies
Fuse Fabric to the rescue!
58 redhat Rob Davies
Fuse Fabric – Key Features
Fuse IDE
FMC
JBoss A-MQ Cluster
Camel Endpoints JBoss Fuse
Runtime Registry
• Supports Hybrid deployments • Distributed Configura(on • Run(me registry • Centralized Management
59 redhat Rob Davies
FuseMQ features
• mq-‐base profile • Defines OSGi features and bundles to be installed • Defines basic broker sesngs
• mq-‐create command • Helper command for crea(ng brokers • Creates an new profile based on mq-‐base • Op(onally creates new containers • Assigns the profile to containers (essen(ally starts the broker)
FuseMQ = ActiveMQ deployed in Apache Karaf +
60 redhat Rob Davies
JBoss A-‐MQ A small-footprint, high-performance, open source messaging platform
61 redhat Rob Davies
JBoss A-‐MQ – Value ProposiIon
Value Proposition: – Easier to configure, monitor,
manage and maintain: Small IT footprint
– Multiple deployment options – centralized, distributed, embedded: Small IT footprint
– Proven and robust platform: Based on popular Apache ActiveMQ, Standards-based reliable messaging & High-performance
– Multiple connectivity options: Multi-standards and multi-platform support
– Real-time and reliable integration: Supports messaging paradigms: Pub/Sub, Point-2-Point, Store and forward
New in 6.0: AMQP 1.0 support (Tech Preview)
JBoss Developer Studio to include Fuse IDE
JBoss Operations Network + Fabric Management Console
Rebranding, repackaging
62 redhat Rob Davies
JBoss A-‐MQ -‐ Benefits
Breadth of connecIvity – cross-‐language client support and mul(-‐protocol support provides wide range of connec(vity from browser-‐based clients to mobile devices
Proven reliability and support – proven track record of suppor(ng mission-‐cri(cal applica(ons, and backed by enterprise-‐class services and tools
Small IT footprint – high-‐performance and reliable messaging with small footprint and minimal maintenance
Multiple deployment options – Flexible configuration allows multiple deployment options
No license fees – try before you buy, and deploy widely without incurring exorbitant costs
63 redhat Rob Davies
So where is JBoss A-MQ used ?
64 redhat Rob Davies
Smart Grids …
64
65 redhat Rob Davies
Next GeneraIon Air Traffic Control …
65
NextGen Applications
FTI IP Backbone
En Route Controllers
Terminal Controllers
Non-FAA Users (e.g., Airlines, DoD DHS, etc.) FAA
Command Center
SWIM Enterprise Infrastructure
66 redhat Rob Davies
Well not exactly in space – but close
66
67 redhat Rob Davies
And helping run the biggest science experiment – ever!
67
68 redhat Rob Davies
Btw -‐ Why is IntegraIon and Messaging Important ?
Head Office Enterprises Need to know Everything!
69 redhat Rob Davies
There are more things to Integrate with!
70 redhat Rob Davies
The Internet of Things
• Uniquely iden(fiable objects and their virtual representa(ons in an internet-‐like structure
• Originally defined by Kevin Ashton, co-‐founder of Auto-‐ID center at MIT
• Today its meaning has evolved to encompass a world where physical objects are integrated into the informa(on network, and where physical objects par(cipate in business processes.
71 redhat Rob Davies
Machine to Machine
• M2M involves collec(on and transmission of event level data (used to be called telemetry) from intelligent devices
• Machines can interact over the net – and/or alached networks – by wired and wireless networks
• Payloads are typically minimal (temperature, pressure, loca(on, metering etc.)
72 redhat Rob Davies
Internet of Things
Time
Connected Devices on the internet
2010 2000 2020
1 Billion
50 Billion
10 Billion
Ubiquitous posiConing – locaCng people and everyday objects
RFID tags for Inventory control
Advanced sensors, Enterprises fully connected: integraCon of everything
Geo-‐located devices Microcontrollers internet enabled
Vending machines, lone workers, POS, vehicles, planes, tolls, power systems, transport, telemedicine, home appliances, power systems, etc.
73 redhat Rob Davies
M2M Examples Smart Energy –
• Smart Grid uses smart meters for monitoring energy uses by premises for billing, substaCon and transmission line monitoring, energy producCon (renewables)
Inventory Control –
• smart labels and RFID tags – connected scanners pass informaCon upstream to servers for monitoring
In-‐Vehicle Systems
• Maintenance informaCon, global posiConing, ‘black-‐box’ (speed and driving habits), wireless payments for tolls/gas
Environmental Monitoring
• Weather sta(ons (temperature, air-‐pressure), ocean monitors, earth movements, volcanic ac(vity crop yields etc.
Livestock monitoring
• Cow herds, monitored for opCmum Cme for milking, sheep herds for locaCon
74 redhat Rob Davies
M2M Topology
Gateway
Hadoop
Servers
Analytics
Gateway
Internet
Mesh Network
Mobile wireless
75 redhat Rob Davies
Power ConsumpIon! • Balery life for mobiles, power consump(on for smart
devices is an important considera(on.
• Facebook use MQTT for real-‐(me updates – efficiency was one of the things considered
• MQTT is more efficient at establishing connec(ons, sends informa(on more reliably, and consumes less power to transmit data than HTTP.
76 redhat Rob Davies
Deluge of Data • Billions of devices send informa(on will require storage
and processing of terabytes or even petabytes of data.
• Big Data Inges(on: Processing of vast amounts of data business generates to make good decisions is only half the problem. Inges(ng that data from all areas of the enterprise will require TBs of data to be ingested in highly reliable and scalable way.
77 redhat Rob Davies
Need an Eco system of technologies
Dev
• Integra(on is gesng harder • Tradi(onal message brokers are key
to connect “everything” • Mul(ple protocols need to be
supported
• Camel can provide integra(on at the edges – and the data center
• Technologies like Fuse Fabric provide centralized provisioning and management
• Need to mix protocols, brokers and “message routers” to scale
FuseMQ Clusters
Arrival Airport 1
78 redhat Rob Davies
Demo Time!
79 redhat Rob Davies
Messaging through the Internet of Things
ActiveMQ
AMQP/MQTT routers
AMQP/MQTT routers
AMQP/MQTT routers
MQTT
MQTT-‐S
AMQP
AMQP
AMQP
MQTT
MQTT
websockets
80 redhat Rob Davies
Simplified demo
Java MQTT Client
AMQP MQTT
JBoss A-MQ MQTT/AMQP Gateway
JBoss A-MQ AMQP/OpenWire Broker JMS OpenWire Client
81 redhat Rob Davies
What is Arduino ?
• Open Source Hardware microcontroller • Cheap and easily available. • Open Source Souware. • Widespread: many projects. • Extra HW (shields) available (e.g. Ethernet shield)
82 redhat Rob Davies
USB
7-12 v
3 v GRD 5 v Analog Input Pins
Digital Input/Output Pins Pins with ~ are PWM [Analog Output]
GRD Transmitter/Receiver Serial Connection
Microcontroller ATmega328 Operating Voltage 5V Input Voltage (recommended)7-12V Input Voltage (limits)6-20V Digital I/O Pins 14 (of which 6 provide PWM output) Analog Input Pins 6 DC Current per I/O Pin 40 mA DC Current for 3.3V Pin 50 mA
83 redhat Rob Davies
Arduino Programming
• 3 types of memory on Arduino • Flash memory (program space) – 32 Kb • SRAM – used by sketches – 2 Kb • EPROM – long term memory – 1 Kb
• Programming language based on Wiring: C/C++ library designed to make input/output easier. Programs are called sketches.
• Arduino has a simple IDE, available on Windows, Linux and Mac
84 redhat Rob Davies
Arduino MQTT to WebSockets
Arduino
AMQP MQTT
JBoss A-MQ MQTT/AMQP Gateway
JBoss A-MQ AMQP/WebSockets Broker HTML 5/WebSockets
85 redhat Rob Davies
Useful Resources:
hlp://ac(vemq.apache.org hlp://camel.apache.org hlp://fuse.fusesource.org/fabric/docs/overview.html hlp://fuse.fusesource.org/mq/docs/mq-‐fabric.html hlps://github.com/fusesource/fuseide hlp://hawt.io hlp://www.arduino.cc/
Messaging and Integration�
Developer tooling �
Management�
Configuration and provisioning �
Arduino �
86 redhat Rob Davies
Any Ques-ons ?