how to build more reliable, robust and scalable distributed systems

Download How to build more reliable, robust and scalable distributed systems

Post on 19-Dec-2014




5 download

Embed Size (px)


Slides from my lightning talk at Teknologihuset in Oslo on how to build more reliable, robust and scalable distributed systems.


  • 1. How to build more reliable, robust and scalable distributed systems Lars-Erik Kindblad Senior Consultant Twitter: @kindblad E-mail:

2. The Book Shop Place order should: 1. Save the order in a database 2. Charge the customers credit card 3. Send a confirmation e-mail to the customer 3. The typical way request/responsePlace orderBrowser1. Add orderOrderServiceOrderController 2. Charge credit cardPaymentService 3. Send mailNotificationServiceAlso called synchronous remote procedure call 4. Problem #1 the order is lostFailures: Network Webservice Database++Place order 1. Add orderBrowserOrderController Error pageErrorOrderServiceException 2. Charge credit cardPaymentService 3. Send mailNotificationService Order is lost User will receive an error User might leave lost sale User might retry 5. Problem #2 no transactional management Place order 1. Add orderBrowserOrderServiceOrderController Error page Exception2. Charge credit cardPaymentService 3. Send mailError Order is stored Payment is processed E-mail is not sent User receives an error and might retry: Order might be stored twice in the database Credit card might be charged multiple timesNotificationService 6. Solution one-way messagingAdd PlaceOrder messagePlace orderBrowserOrderControllerQueueAsynchronousThe message:PlaceOrder message 4 First-In Queue PlaceOrder message 3 PlaceOrder message 2 PlaceOrder message 1 First-Out 7. The message must also be processed Queue1. Connect2. Receive message PlaceOrderPlaceOrderWorker3. PlaceOrder Message Handler4. Add orderOrderService 5. Charge credit cardPaymentService 6. Send mailNotificationService 8. Benefits Very fast on the frontend put the message on the queue The order is never lost Automatically retries during errors 4. Rollback. Put the message back on the queue and retryQueue1. Connect2. Receive message PlaceOrder PlaceOrderWorker3. PlaceOrder Error Message Handler 9. We still have poor transactional management 1. Add orderHandle PlaceOrderOrderService 2. Charge credit cardPaymentService 3. Send mail ExceptionErrorNotificationService Order is created Credit card is charged E-mail sending fails The message is put back on the queue and will be retried Order is duplicated and credit card will be charged twice 10. Solution Split into many messages - one message per transactional boundary Place orderOrderControllerBrowser1. Add PlaceOrder message All the services must be idempotent to be 100% reliablePlaceOrder message2. Handle PlaceOrderAdd PayOrder messagePayOrder messageQueueOrderService3. Handle PayOrder Add SendMailSendMail message4. Handle SendMailNotificationServiceIf this fails the message is put back on the queue and will be retriedPaymentService 11. Scaling Request/Response Needs to process unpredictable many requests Messaging Needs to store unpredictable number of messages The workers only process a predictable number of messages, even during peaks Scale up Concurrently process multiple messages Scale out Use multiple workers 12. Messaging challenges 1. ID generation The order ID is first available in the worker2. Eventual consistency The PlaceOrder message might not have been picked up yet 13. NServiceBus Lightweight messaging framework for .NET Great choice for implementing one-way messaging + publish/subscribe Open Source but not free Available at 14. About Capgemini With more than 120,000 people in 40 countries, Capgemini is one of the world's foremost providers of consulting, technology and outsourcing services. The Group reported 2011 global revenues of EUR 9.7 billion. Together with its clients, Capgemini creates and delivers business and technology solutions that fit their needs and drive the results they want. A deeply multicultural organization, Capgemini has developed its own way of working, the Collaborative Business ExperienceTM, and draws on Rightshore , its worldwide delivery model. Rightshore is a trademark belonging to Capgeminiwww.capgemini.comThe information contained in this presentation is proprietary. 2012 Capgemini. All rights reserved.