improvements in rabbitmq
TRANSCRIPT
![Page 1: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/1.jpg)
Improvements in RabbitMQ Performance and Future Directions
Alvaro Videla - RabbitMQ
![Page 2: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/2.jpg)
@old_sound
github.com/videlalvaro
![Page 4: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/4.jpg)
What is RabbitMQ
![Page 5: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/5.jpg)
RabbitMQ
![Page 6: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/6.jpg)
RabbitMQ
![Page 7: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/7.jpg)
RabbitMQ• Multi Protocol Messaging Server
![Page 8: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/8.jpg)
• Multi Protocol Messaging Server• Open Source (MPL)
RabbitMQ
![Page 9: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/9.jpg)
• Multi Protocol Messaging Server• Open Source (MPL)• Polyglot
RabbitMQ
![Page 10: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/10.jpg)
• Multi Protocol Messaging Server• Open Source (MPL)• Polyglot• Written in Erlang/OTP
RabbitMQ
![Page 11: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/11.jpg)
Multi Protocol
http://bit.ly/rmq-protocols
![Page 12: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/12.jpg)
http://www.rabbitmq.com/community-plugins.html
Community Plugins
![Page 13: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/13.jpg)
Polyglot
![Page 14: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/14.jpg)
Polyglot• Java• node.js• Erlang• PHP• Ruby• .Net• Haskell
![Page 15: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/15.jpg)
Polyglot
Even COBOL!!!11
![Page 16: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/16.jpg)
Some users of RabbitMQ
![Page 17: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/17.jpg)
• Indeed.com
• Telefonica• Mercado Libre
• NHS• Mozilla
Some users of RabbitMQ
![Page 18: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/18.jpg)
The New York Times on
This architecture - Fabrik - has dozens of RabbitMQ instances spread across 6 AWS zones in Oregon and Dublin.
Upon launch today, the system autoscaled to ~500,000 users. Connection times remained flat at ~200ms.
http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2014-January/032943.html
![Page 19: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/19.jpg)
http://www.rabbitmq.com/download.html
Unix - Mac - Windows
![Page 20: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/20.jpg)
Messaging with RabbitMQ
A demo with the RabbitMQ Simulator
https://github.com/RabbitMQSimulator/RabbitMQSimulator
![Page 22: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/22.jpg)
RabbitMQ Simulator
![Page 23: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/23.jpg)
Performance Improvements
![Page 24: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/24.jpg)
The Problem
![Page 25: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/25.jpg)
The Problem
./runjava.sh com.rabbitmq.examples.PerfTest \-e exchange_name -t topic -u queue_name \-f persistent -r 5000 -s 1000 -x1 -y0
![Page 26: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/26.jpg)
The Problem
Publish 5000 1kb msgs/sec without consumers
Be able to sustain publish rate during one hour
![Page 27: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/27.jpg)
The Problem
Big throughput drop around 1 minute into the benchmark
![Page 28: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/28.jpg)
The Problem
As soon as Queue Paging kicked inPerformance Dropped
![Page 29: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/29.jpg)
Culprits
• queue_index flush journal
• messages published into the queue index when paging
• memory leaks related to refc binaries
![Page 30: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/30.jpg)
Flushing the QI Journal
• Sparse Array by Message SeqIds
• Publishes, Delivers and Acks
• Queue Index is written in Append Only Segments
• RabbitMQ keeps an in memory Journal of ops to prevent going to disk all the time
![Page 31: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/31.jpg)
Flushing the QI Journal
queue_index_max_journal_entries = 65536
![Page 32: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/32.jpg)
Flushing the QI Journal
entry_to_segment/3 called 65536 times
![Page 33: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/33.jpg)
Flushing the QI Journal
entry_to_segment/3 called 65536 times
Appending data to a binary and then writing that to disk
![Page 34: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/34.jpg)
Solution
• Keep a Sparse Array as a cache of “entry to segments”
• Flush said array to disk whenever the Journal needs to be flushed
![Page 35: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/35.jpg)
Solution
array:to_list(EntriesToSegment).
![Page 36: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/36.jpg)
Solution
array:to_list(EntriesToSegment).
What happens with blank elements?
![Page 37: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/37.jpg)
Solution
array:new([{default, []}, fixed, {size, ?SEGMENT_ENTRY_COUNT}]).
![Page 38: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/38.jpg)
Messages Publishes into the Queue Index
![Page 39: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/39.jpg)
queue_index_embed_msgs_below = 4096
Messages Publishes into the Queue Index
![Page 40: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/40.jpg)
Messages Publishes into the Queue Index
• Implemented Batch Publishing into the Queue Index
• Batch Handling of Delivers and Acks
• Flush Publishes, Delivers and Acks cache every 20.000 messages or when paging finished.
![Page 41: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/41.jpg)
Queue Paging
Alphas -> Betas -> Deltas
![Page 42: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/42.jpg)
binary memory leak• Related to Paging
• discovered with recon
• forced queue process garbage collection after paging
http://ferd.github.io/recon/index.html
![Page 43: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/43.jpg)
The Problem II
![Page 44: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/44.jpg)
The Problem II
Mirror Queue Synchronisation
“is too slow”
![Page 45: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/45.jpg)
The Problem II
Initial tests showed that it took 60 seconds to sync 1MM messages
![Page 46: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/46.jpg)
The Problem II
Original implementation was synching one message at a time
![Page 47: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/47.jpg)
Solution II
Why not implement batch publishing at the queue level?
![Page 48: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/48.jpg)
Batch Publishing• Implemented for Backing Queue Implementations
• rabbit_variable_queue
• rabbit_priority_queue
• rabbit_mirror_queue_master
• rabbit_mirror_queue_slave
![Page 49: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/49.jpg)
Solution II
Initial tests showed that it took 6 seconds to sync 1MM messages
![Page 50: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/50.jpg)
The Problem III
Queues that support offline consumers for 24 hours or more
![Page 51: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/51.jpg)
Solution III• Lazy-Queue concept
• Disable in-memory message cache
• Only load messages in RAM if consumers are present
• Implement Lazy Queues for all Backing Queue implementations
![Page 52: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/52.jpg)
Lazy Queues
https://github.com/rabbitmq/rabbitmq-server/issues/351
Initial tests show stable throughput comparable to “default” queues with persistent messages
![Page 53: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/53.jpg)
Lazy Queues
https://github.com/rabbitmq/rabbitmq-server/issues/351
10MM of 1KB use 1.5MB of RAM vs
1.2GB of RAM for default queues
![Page 54: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/54.jpg)
Lazy Queues
https://github.com/rabbitmq/rabbitmq-server/issues/351
40MB of RAM when a consumer is active
![Page 55: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/55.jpg)
Other Improvements
• moved to erlang.mk
• moving to Ranch/Cowboy and related libraries
• improved management UI performance
• improved rabbitmqctl performance
![Page 56: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/56.jpg)
Other Improvements
• pluggable hashing algorithm for passwords
• queue master location strategies
![Page 57: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/57.jpg)
Future
• add Raft based queue mirroring
• add Raft based cluster formation
• Re-thing rabbitmqctl to allow for extensions
• Tools to recover data from disk
![Page 58: Improvements in RabbitMQ](https://reader033.vdocuments.mx/reader033/viewer/2022042600/58a4462d1a28ab41618b6989/html5/thumbnails/58.jpg)
Thanks
Alvaro Videla - RabbitMQ