Event processing with Node.js and RabbitMQ

by Zlatko Duric

Disclaimer (AKA about me)

● Experience– Node.js? Lots

– NoSQL? Lots.

– Messaging? Little

● Backend Developer at ThinxNet.com

(http://thinxnet.com/#jobs)

Connected cars?

● Hardware device● Various sensors – lots of data● “Cloud” processing

Ready to serve. How?

● Event collection and processing● Webapp, mobile app● Backoffice and support apps, notification services etc.● Reporting, third party software integration etc● REST approach:

– Lots of requests

– Parsing of data is blocking

– Avoid choking devices

– Communication overhead

– Error management

What service? Microservice!

● Redis PUBSUB● ZeroMQ, MQTT, AMQP● RabbitMQ, ActiveMQ

About RabbitMQ

● Messaging broker based on AMQP protocol ● Platform/language independent● Extensible (Erlang)● Persistent message queues● Node.js drivers (node-amqp, rabbit.js...)

Hello world

Exchanges and topics

● Publisher and subscriber. Simple as that.● Get message, ack or reject (nack) it.● Subscribers listen on a queue● Publishers talk to an exchange● Exchange routes to queues based on topics

Example

Channels and connections

● 1 node process = 1 connection● Many channels, share same connection

PUBSUB

● Simple PUBSUB, as used elsewhere● All listeners get the same message● Suboptimal

Example

PUSH-PULL/WORKER

● Round-robin distribution● Worker socket.ack() is explicit (rabbit.js

driver)● Prefetch sets the number of msgs received● Problem: concurrent/sequential

modification?

Example

Content-based routing

● Erlang extensions? Not standard.● App-side:

– Keep internal mini-queue

– Do not ack a msg if it's not first on local queue

– Hold internal messages based on internal “topic”, ie “user.<ID>.save” - hold msgs for that user

– Item.update() instead of item.save()

● Gets nasty fast, should minimize these scenarios

Example

REST API with MQ

● REQUEST/REPLY for immediate responses● Other patterns for direct access

– REST API just forwards the message

– replies will go by other route

● Pub/sub is easy!

Example

Error handling

● Uncaught exceptions are covered● Retry logic is simplified – push incomplete

messages to a 'fixer' queue, requeue failed jobs or when no resources

● Dedicated error handling

Example

Error handling

● Uncaught exceptions are covered● Retry logic is simplified – push incomplete

messages to a 'fixer' queue, requeue failed jobs or when no resources

● Dedicated error handling

Testing

● No good known mocks, please help● RabbitMQ always on on Jenkins machine● Test handlers directly

Buzzwords index

● Cloud● Connected car● Microservice● Patterns

:)

Info sources

● https://www.rabbitmq.com/● http://www.squaremobius.net/rabbit.js/● http://www.manning.com/videla/

Note to self

● Thank the audience for patience● Invite audience for a discussion and questions