How to build more reliable, robust and scalable distributed
systems
Lars-Erik KindbladSenior Consultant
Twitter: @kindbladE-mail: [email protected]
The Book Shop
Place order should:1. Save the order in a database
2. Charge the customer’s credit card
3. Send a confirmation e-mail to the customer
The typical way – request/response
Also called synchronous remote procedure call
OrderController
Place order
Browser
1. Add order
2. Charge credit card
3. Send mail
PaymentService
NotificationService
OrderService
Problem #1 – the order is lost
Order is lostUser will receive an error
User might leave – lost sale User might retry
OrderController
Place order
Browser
Error page
1. Add order
2. Charge credit card
3. Send mail
PaymentService
NotificationService
OrderServiceError
Failures:Network
WebserviceDatabase++
Exception
Problem #2 – no transactional management
Order is storedPayment is processedE-mail is not sentUser receives an error and might retry:
Order might be stored twice in the database Credit card might be charged multiple times
OrderController
Place order
Browser
Error page
1. Add order
2. Charge credit card
3. Send mail
ExceptionPaymentService
NotificationService
OrderService
Error
Solution – one-way messaging
Asynchronous
OrderController
Place order
Browser
Add PlaceOrdermessage
Queue
Queue
PlaceOrder message 4
PlaceOrder message 3
PlaceOrder message 2
PlaceOrder message 1
First-In
First-Out
The message:
The message must also be processed
Queue
Worker
1. Connect 2. Receive message PlaceOrder
3. PlaceOrderMessage Handler
PlaceOrder 4. Add order
5. Charge credit card
6. Send mail
PaymentService
NotificationService
OrderService
Benefits
Very fast on the frontend – put the message on the queueThe order is never lostAutomatically retries during errors
Queue
Worker
1. Connect 2. Receive messagePlaceOrder
3. PlaceOrderMessage Handler
PlaceOrderError
4. Rollback. Put the message back on the queueand retry
We still have poor transactional management
Order is createdCredit card is chargedE-mail sending failsThe message is put back on the queue and will be retried
Order is duplicated and credit card will be charged twice
Handle PlaceOrder1. Add order
2. Charge credit card
3. Send mail
ExceptionError
PaymentService
NotificationService
OrderService
Solution
Split into many messages - one message per transactional boundary
OrderController
Place order
Browser
1. Add PlaceOrder message
Queue
PlaceOrder message
2. Handle PlaceOrder Add PayOrder message
OrderService
PayOrder message
Add SendMail3. Handle PayOrder
PaymentService
4. Handle SendMailSendMail message
NotificationService
If this fails the message is put back on the queue and will
be retried
All the services must be idempotent to be
100% reliable
Scaling
Request/Response Needs to process unpredictable many requests
Messaging Needs to store unpredictable number of messages The workers only process a predictable number of messages, even during peaks
Scale up Concurrently process multiple messages
Scale out Use multiple workers
Messaging challenges
1. ID generation The order ID is first available in the worker
2. Eventual consistency The PlaceOrder message might not have been picked up yet
NServiceBus
Lightweight messaging framework for .NETGreat choice for implementing one-way messaging + publish/subscribeOpen Source but not freeAvailable at http://particular.net/
The information contained in this presentation is proprietary.© 2012 Capgemini. All rights reserved.
www.capgemini.com
About Capgemini
With more than 120,000 people in 40 countries, Capgemini is one of the world's foremost providers of consulting, technology and outsourcing services. The Group reported 2011 global revenues of EUR 9.7 billion.Together with its clients, Capgemini creates and delivers business and technology solutions that fit their needs and drive the results they want. A deeply multicultural organization, Capgemini has developed its own way of working, the Collaborative Business ExperienceTM, and draws on Rightshore ®, its worldwide delivery model.
Rightshore® is a trademark belonging to Capgemini