scaling mqtt - webinar with elastic beam
TRANSCRIPT
WHAT IS MQTT?
➤ IoT Messaging Protocol
➤ Minimal Overhead
➤ Publish / Subscribe
➤ Easy
➤ Binary
➤ Data Agnostic
➤ Designed for reliable communication over unreliable channels
USE CASES
➤ Push Communication
➤ Unreliable communication channels (e.g. mobile)
➤ Constrained Devices
➤ Low Bandwidth and High Latency environments
➤ Communication from backend to IoT device
PROTOCOL DETAILS
➤ MQTT requires TCP/IP
➤ IPv4 and IPv6 supported
➤ Persistent TCP connections
➤ Heartbeat mechanism
➤ Clean & Persistent Sessions
➤ QoS levels for message delivery guarantees
➤ TLS in the mix for professional deployments
MQTT
TCP
IP
5-7
ISO/OSI
Layer
4
ISO/OSI
Layer
3
ISO/OSI
Layer
CHALLENGE 1:
HUGE AMOUNT OF TCP CONNECTIONS
➤ MQTT uses persistent TCP connections
➤ Number of concurrent TCP connections limited (hard to scale > 1-2 Million on a single machine)
➤ “Reconnect Storms”
CHALLENGE 2:
SECURITY
➤ TLS for secure communication
➤ TLS handshake is resource intensive
➤ Brokers accessible over the Internet
➤ Scalable backend systems for authentication & authorization needed
CHALLENGE 3:
MQTT IS STATEFUL
➤ MQTT allows sessions
➤ Subscriptions, Queued Messages, Message Flow remembered for offline client
➤ Retained Messages for every topic possible (dynamic topics!)
CHALLENGE 4:
HIGH AVAILABILITY
➤ MQTT Brokers are a single point of failure
➤ Cloud Environments WILL fail
➤ Bridged Brokers (brokers that are chained via MQTT) do not support session migration
IoT Devices
elastic beam
MQTT with TLS 1.2
elastic beam
MQTT BrokerCluster
OtherEnterprise Applications
APIProxyCluster
Application Servers
Databases Microservices
Datacenter
MQTT over Websocketswith TLS 1.2
HIVEMQ OVERVIEW
➤ Enterprise MQTT Broker
➤ Elastic Clustering
➤ Linearly scalable
➤ Best-in-class Performance
➤ Open Source Plugin System
➤ Scalable horizontally & vertically
➤ > 1.000.000 MQTT connections per broker possible
HIVEMQ CLUSTER ADVANTAGES
➤ Elimination of the Single Point of Failure
➤ Sophisticated message distribution across cluster nodes
➤ Clients can connect to any node to resume their sessions
➤ Linearly scalable
➤ High Availability due to resilience and fault-tolerance
➤ Zero-Downtime Upgrades
FALLACIES OF DISTRIBUTED COMPUTING
➤ The network is reliable.
➤ Latency is zero.
➤ Bandwidth is infinite.
➤ The network is secure.
➤ Topology doesn't change.
➤ There is one administrator.
➤ Transport cost is zero.
➤ The network is homogeneous.
“The Eight Fallacies of Distributed Computing”
Peter Deutsch
THE CAP THEOREM
➤ Applicable to all distributed systems with persistent state
➤ “Pick two”
➤ You can’t trade Partition Tolerance (see Fallacies of Distributed computing)
➤ CP would render the whole broker cluster unusable in case no consensus is achieved
➤ HiveMQ is AP and has self-healing mechanisms for reconciliation => The broker cluster is always available and never drops client connections
Image Credits: Benjamin Erb
http://berb.github.io/diploma-thesis/original/061_challenge.html
HIVEMQ CLUSTER DETAILS
➤ Masterless Architecture
➤ Highly Resilient
➤ Always Available (favors AP in the CAP theorem)
➤ Elastic. You can add and remove nodes at runtime
➤ Network partitions are supported
➤ Data distribution across cluster nodes
➤ Configurable Replica Count
➤ Self-Healing mechanisms when merging network partitions
Broker 2 Broker 3
Broker 1
MQTT Broker Cluster