throughput high concurrency & using mqtt and kafka
Post on 14-Feb-2022
7 Views
Preview:
TRANSCRIPT
High Concurrency & Throughput Using MQTT and Kafka
by Evan Purnama (Co-Founder, CTO Qiscus Pte Ltd)
Introduction
What is Qiscus?
We are a startup company that help other companies to enable conversations.
What is Qiscus?
What is Qiscus?
What is Qiscus?
Future of Technology
Future of Technology
Future of Technology
All Of Those Translate to ...
Huge Concurrent Connections
Huge Number of Messages Being
Processed
MQTT (Message Queuing Telemetry Transport)
- Created at 1999 by IBM.- Optimised for constrained environment.- Being used for communicate in rural/offshore area.
Pub Sub
MQTT
- Broker and Client (Publisher & Subscriber). - Highly Scalable Architecture- Broker become Single Point of Failure (SPoF).
- Both publishers and subscribers are very simple- Lightweight ( ~ few bytes overhead per message)- Very little resources on client side
- MQTT Broker is complex while the client is thin- Suitable for constrained environment
MQTT
- Reliable- QoS 0, 1, 2- Last Will & Testament Messages
- Secure- Username & Password- TLS
Components of MQTT
- Publisher and Subscriber- Topics- Message- QoS- Broker- etc.
Topics
- “Address” for client side to Publish or Subscribe.- Patterns :
- foo/abc/bar- foo/+/bar- foo/#
- Creating topic is cheap.- Use topic for specific event instead of general event.
Broker
- Open Source Based- Mosquitto- EMQ- Mosca- VerneMQ
- Managed Service MQTT Broker
Scaling Goal
Scaling Goals
- Connect as many connected users/devices as possible.- Handle as many throughput as possible
Scaling Goals
- Cloud based (AWS, GCP, etc)- Horizontal Scaling
- reduce cost- plan for future- less impact downtime
Concurrent Connections
Publisher Publisher Publisher Subscriber Subscriber Subscriber
Single MQTTBroker
Scaling Concurrent Connections
Publisher Publisher Publisher Subscriber Subscriber Subscriber
MQTTBroker 2
MQTTBroker 1
Scaling Concurrent Connections
Publisher Publisher Publisher Subscriber Subscriber Subscriber
MQTTBroker 2
MQTTBroker 1
Shared Memory Data
Scaling Concurrent Connections
Publisher Publisher Publisher Subscriber Subscriber Subscriber
MQTTBroker 2
MQTTBroker 1
Scaling Concurrent Connections
Load Balancing
- Which broker to connect to?- DNS Load Balancing.
High Concurrent Connections
Publisher Publisher Publisher Subscriber Subscriber Subscriber
MQTTBroker 2
MQTTBroker 1
DNS Load Balancer
Message Processing
Message Processing
Message Processing
Publisher
Publisher
Publisher
Publisher
Publisher
ClusterBroker MQTT
Subscriber
Subscriber
DB
Elastic Search
Push Notif, etc
Message Processing
- Sending to multiple destinations from the same instance that receiving very huge throughput is not a scalable approach.
- Adding more component means need to add to all of the subscribers.- What if one of the component suddenly become unavailable?
- Data lost?- Subscriber need to replay?
Kafka
- Unified, high-throughput, low-latency platform for handling real-time data feeds.
- Highly scalable stream, with throughput up to ~200k-300k for single node.
- Originally from LinkedIn, to solve Log Aggregation problems.- Zoo-Keeper- Publisher, Consumer, Topic, Partition concept.- Very scalable server-to-server messaging system.
Kafka
Scalable Message Processing
Publisher
Publisher
Publisher
Publisher
Publisher
ClusterBroker MQTT
DB
Elastic Search
Push Notif, etc
Subscriber
Subscriber
ClusterKafka
Notes :
- This strategy is only suitable for burst- need to make sure that the “consumer” able to catch-up to process
the input stream.- Kafka becoming Single Point of Failure (SPoF) on message processing,
be careful with this. Make sure to setup properly and make it High Availability (HA).
Thank You
References :
- https://blog.qiscus.com/handle-10-million-concurrent-user-connections-5-millions-messages-per-minute-throughput/
- https://www.youtube.com/watch?v=WFfciQ1U_lU
- https://www.csoonline.com/article/3346082/what-is-shadow-iot-how-to-mitigate-the-risk.html
- https://www.fastcompany.com/4011195/this-ibm-smart-bus-comes-with-a-watson-powered-assistant
- https://youngster.id/featured/vendor-telekomunikasi-berperan-penting-dalam-melindungi-smart-home/
top related