messaging - rabbitmq, azure (service bus), docker and azure functions
TRANSCRIPT
![Page 1: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/1.jpg)
Message System Fundamentals
By John Staveley27/07/2017
https://uk.linkedin.com/in/johnstaveley/@johnstaveley
![Page 2: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/2.jpg)
Who am I?
John StaveleyContract Mvc.net developerOrganiser @LeedsSharpUser of messaging systems for the last 4 years (MSMQ,
Rabbit, Azure Service Bus etc)
![Page 3: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/3.jpg)
Overview
Why use messaging systems?What are they? Benefits?Demos (Azure Storage Queues, Azure Service Bus,
RabbitMQ)Azure FunctionsInfrastructure as code.Net Core and DockerOther Messaging implementationshttps://github.com/johnstaveley/MessagingPresentationSlides on SlideShare
![Page 4: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/4.jpg)
Use cases and benefits
Distributed workloads Increased reliability (cf fallacies of distributed networking) Synchronous vs Asynchronous processes Loosely coupled systems (Microservices) The promise of fast reads and fast writes LinkedIn, CompareTheMarket.com, Facebook
![Page 5: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/5.jpg)
Demo: Azure Storage Queues
What is Azure? What is Serverless / FaaS - Azure functions? What are Azure Queues? Message patterns - One way/competing consumers Demo
![Page 6: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/6.jpg)
What is Azure?
Second largest cloud provider in the world IaaS (Windows or Linux VMs), PaaS, FaaS Data storage, Notifications, Load Balancing, Active
Directory, Application insights SQL Azure, DocumentDb, HDInsight, Azure data lakes,
Azure machine learning, Cognitive services ARM Templates Message Queues
![Page 7: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/7.jpg)
Azure Functions (1)
Serverless – paradigm shift FaaS = Function as a Service Azure function = Event + Code Triggers – Schedule, Http, Queues, Storage Bindings – connecting to data Programmed in JavaScript, C#, Python, PHP, Powershell Functions are stateless Try it for free - monthly grant of 1,000,000 executions and
400,000 GB-s Execution limited to 5 minutes
![Page 8: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/8.jpg)
Azure Functions (2)
Monitoring for free Try online: https://functions.azure.com/try Can be edited inside of the Azure portal Can be edited and deployed from Visual Studio
![Page 9: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/9.jpg)
Azure Queuing technologies
Storage Queues- Supports Get/Put/Peek- Can store over 80Gb in queue- No ordering guarantee- Cheap (100x cheaper than Azure Service Bus)
Service Bus Queues- Can support message exchange patterns e.g. Pub/Sub,Topics- Doesn't require polling- 80Gb queue limit- AMQP- Apps can support at-most-once delivery- Partitions to overcome broker limitation
Event Hubs- Can handle huge scale or event streaming needs- AMQP or HTTP
![Page 10: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/10.jpg)
Messaging pattern – One way
One-way
S Q C
Message Broker
![Page 11: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/11.jpg)
Azure Storage Queue Demo
Workload: Calculate factors of numbers90 = 1, 2, 3, 5, 6, 9, 10, 15, 18, 30, 45, 90
Source RestAPI Queue
FactorCalc Queue Consumer
![Page 12: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/12.jpg)
Azure Queues Demo
![Page 13: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/13.jpg)
Azure Function auto-scaling
Scales using heuristics
Source RestAPI Queue
FactorCalc Queue Consumer
![Page 14: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/14.jpg)
Messaging – Competing Consumers
Competing Consumers (a.k.a Worker queues)
S Q
C
CMessage Broker
![Page 15: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/15.jpg)
Types of messaging pattern
One-way Competing consumers RPC Pub-Sub Routing Topics Scatter-Gather
![Page 16: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/16.jpg)
Messaging – Pub sub
Publish-subscribe
S
Q C
CQ
E
Message Broker
![Page 17: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/17.jpg)
Messaging – Routing
Routing
S
Q C
CQ
E
Message Broker
Large
Medium
Small
![Page 18: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/18.jpg)
Messaging – RPC
Remote Procedure Call
S
Q
C
Q
Message Broker
Request queue
Response queue
![Page 19: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/19.jpg)
Messaging – Scatter-Gather
Scatter-Gather
S
Q C
CQE
Message Broker
Dynamic response queue
Q
![Page 20: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/20.jpg)
Demo: Azure Service Bus
Infrastructure as code: What are ARM templates? Azure Service Bus Topics message pattern Demo
![Page 21: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/21.jpg)
ARM Templates (1)
Azure Resource Manager templates Infrastructure as code JSON Schema
https://github.com/Azure/azure-quickstart-templates
![Page 22: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/22.jpg)
Azure Service Bus
Queues → Single Consumer Topics & Subscriptions → Multiple consumers Patterns– Topics– Routing– Pub/Sub
![Page 23: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/23.jpg)
Messaging – Topics
Topics
S
Q C
CQ
E
Message Broker
*.small.*
high.*.*
Q C
High.small.mail
Low.large.post
Low.small.post
![Page 24: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/24.jpg)
Azure Service Bus Demo
S
All Products C
COrange, White, Yellow
E
Azure Service Bus
Large C
All Products
SubscriptionsTopics
Expensive
Light Colour
Large
Cost > 80 C
Products:Colour.Size.Cost
![Page 25: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/25.jpg)
Azure Service Bus Demo
![Page 26: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/26.jpg)
Break!
![Page 27: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/27.jpg)
Demo: RabbitMQ in Docker
Introduction to Docker Introduction to RabbitMQ Setup .Net core on docker Routing message pattern Demonstration of RabbitMQ Management UI Creation of Exchange and Queue using UI and powershell Running rabbitmq, producer, receiver in docker
![Page 28: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/28.jpg)
Infrastructure
Infrastructure: How we used to do it– Windows image, Packer, Boxstarter, Chocolatey, Vagrant = 6Gb
What is Docker?– Lightweight containers– Run on a host operating system (either Linux or Windows)– Inside of VirtualBox or HyperV host– Extensive support - lots of images available on docker hub– Very fast to setup and start images– Docker is scriptable
What is kitematic?
![Page 29: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/29.jpg)
RabbitMQ
AMQP – Advance Message Queuing Protocol Built on Erlang Easy to setup - runs on Linux, Windows and containers Exchanges Queues Routing Key Virtual Hosts Web based Management UI EasyNetQ (https://www.infoq.com/presentations/RabbitMQ-NET-EasyNetQ) Powershell management
![Page 30: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/30.jpg)
Docker and .Net Core
.Net Core runs on Linux Choose the right NuGet package Build .Net core to run inside docker:
– Build a container image to build the C# solution (DockerFile.build)– Create an instance of the build container– Copy the .Net core build output from the docker build container– Use the .Net core build output to create images with .Net core (DockerFile)– Run the containers (docker run or docker-compose)
![Page 31: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/31.jpg)
Messaging – Routing
Routing
S
Q C
CQ
E
Message Broker
Large
Medium
Small
![Page 32: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/32.jpg)
RabbitMQ Demo
S
Q C
E
Large Products
Products:Colour.Size.Cost
*
Large
![Page 33: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/33.jpg)
Rabbit MQ Demo
![Page 34: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/34.jpg)
… and now!
Reliability patternsWhat is a service bus?Messaging jargonTypes of messaging systemsHow to select a messaging system?Event sourcing
![Page 35: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/35.jpg)
Reliability patterns (1)
S Q C
Message Broker
What could possibly go wrong?
![Page 36: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/36.jpg)
Reliability patterns (2)
Acknowledgement levelsConnection failures -> Client automatically reconnect, use heartbeatsFailure at broker/queue -> Persist to diskMessaging hardware failure -> Clustering (e.g. RabbitMQ and MSMQ)Producer failure -> Ack will not be received so need Idempotent messagingBad Config (Message not routed) -> Set ack to confirm message routedEasyNetQ - Error handling, connection management, publisher confirms for reliable messaginghttps://www.rabbitmq.com/reliability.html
![Page 37: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/37.jpg)
What is a service bus?
Layer on top of messaging systems– Handles Retries, transaction management
nServiceBus and MassTransit Sagas (aka transactions across messaging)
![Page 38: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/38.jpg)
Messaging fundamentals
Brokered vs non-brokered Persisted vs non-persisted Dead letter queue
![Page 39: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/39.jpg)
Messaging systems
Azure Storage Queue Azure Service Bus RabbitMQ MSMQ ZeroMQ WebSphereMQ / IBM MQ Kafka Azure Event Hubs AWS Simple Queue Service
![Page 40: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/40.jpg)
ZeroMQ
Concurrency framework using socketsFaster than RabbitMQ, but you have to work harder for itUsed by Cisco, AT&T, NASA, Spotify, Microsoft
![Page 41: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/41.jpg)
MSMQ
Store and forward: can do point to point or fan outBrokerless, but for reliability you need a cluster plus MSDTCLimited to windowsUsed by Microsoft, nServiceBus
![Page 42: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/42.jpg)
Kafka
Distributed streaming platform (but can do queuing and pub/sub)Used by thousands of companies like LinkedIn, Yahoo, Twitter, Netflix, Spotify, Pinterest, Uber, Paypal etc etcStreams data efficiently in real time to a replicated commodity cluster running LinuxCan form ecosystems with Hadoop, Spark etcApache projectAppend Onlyhttps://kafka.apache.org/uses
![Page 43: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/43.jpg)
How do I select a messaging system?
Performance Messaging patterns Brokered/Non-brokered Platform Other features
− Duplicate detection− Can it be used with a Service Bus?
Ecosystem
![Page 44: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/44.jpg)
Event Sourcing
Append Only – Store the history of events Don't hold state, build up state from the history of events Useful when holding a log of everything is useful EventStore LMAX architecture:
https://martinfowler.com/articles/lmax.html
![Page 45: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/45.jpg)
Further Resources
Pluralsight₋ https://app.pluralsight.com/library/courses/rabbitmq-by-example₋ https://app.pluralsight.com/library/courses/rabbitmq-dotnet-developers₋ https://app.pluralsight.com/library/courses/message-queue-
fundamentals-dotnet₋ https://app.pluralsight.com/library/courses/docker-web-development
SyncFusion RabbitMQ succinctly bookhttps://github.com/Azure/azure-quickstart-templates
![Page 46: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/46.jpg)
Summary
Demos of Messaging patterns– One Way– Competing Consumer– Topics– Routing
Messaging systems– Azure Storage Queues– Azure Service Bus– RabbitMQ
Hipster tech– FaaS – Build auto-scaling apps without the hassle of any infrastructure– ARM Templates – Infrastructure as code– Docker – fast setup of infrastructure
![Page 47: Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions](https://reader030.vdocuments.mx/reader030/viewer/2022012306/5a6576747f8b9aa84f8b4a3f/html5/thumbnails/47.jpg)
Any Questions?
@johnstaveley