3 common pitfalls in microservice integration and how to avoid … · circuit breaker photo by...

3 Common Pitfalls inMicroservice Integration andHow to Avoid Them

@berndruecker

Distributed systems

https://www.infoworld.com/article/3254777/application-development/3-common-pitfalls-of-microservices-integrationand-how-to-avoid-them.html

SomeService

SomeService

SomeService

SomeService

SomeService

SomeService

SomeService

Failure will happen. Accept it!

But keep it local! Be resilient.

Let‘s start with a simple example

CreditCard

Payment

REST

Live hacking

https://github.com/berndruecker/flowing-retail

Circuit Breaker

Photo by CITYEDV, available under Creative Commons CC0 1.0 license.

Fail fastis important

Fail fastis important

but not enough!

Photo by Tookapic, available under Creative Commons CC0 1.0 license.

„There was an errorwhile sending your

boarding pass“

Check-in

Web-UI

Me

Current situation

Check-in

Barcode Generator

Web-UI

Me

Output Mgmt

Current situation

Check-in

Barcode Generator

Web-UI

Me

Output Mgmt

Current situation

Check-in

Barcode Generator

Web-UI

Me

Output Mgmt

Current situation

Circuit breaker

Check-in

Barcode Generator

Web-UI

Me

Output Mgmt

Current situation – the bad part

Check-in

Barcode Generator

Web-UI

Me

Output Mgmt

Current situation – the bad part

Check-in

Barcode Generator

Web-UI

Me

Output Mgmt

Current situation – the bad part

StatefulRetry

We are having some technicaldifficulties and cannot present you

your boarding pass right away.

But we do actively retry ourselves, so lean back, relax and we will send it

on time.

Check-in

Barcode Generator

Web-UI

Me

Output Mgmt

Possible situation – much better!

Check-in

Barcode Generator

Web-UI

Me

Output Mgmt

Possible situation – much better!

StatefulRetry

Persist thing(Entity, Document, Actor, …)

State machine orworkflow engine

Typicalconcerns

DIY = effort, accidentalcomplexity

Complex, proprietary, heavyweight, slow,

don‘t scale,developer adverse

Scheduling, Versioning, operating, visibility, scalability, …

Handling State

Warning:Contains Opinion

Berlin, Germany

http://berndruecker.io/mail@berndruecker.io@berndruecker

Bernd RueckerCo-founder and Chief Technologist ofCamunda

Workflow engines, state machines

It is

relevantin modern

architectures

CADENCE

Silicon valleyhas recognized

Workflow engines, state machines

CADENCEWorkflow engines, state machines

CADENCE

also at scaleWorkflow engines, state machines

CADENCE

for todays demo

Workflow engines, state machines

Live hacking

https://github.com/berndruecker/flowing-retail

Payment

Now you have a state machine!

CreditCardREST

Manigfold architecture options

https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

Service A Service B

WF-Engine WF-Engine

Spring Boot Spring Boot

Manigfold architecture options

https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

Infrastructure

Service A Service B

WF-Engine WF-Engine

Wildfly WildFly

WARWAR

Manigfold architecture options

https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

Service A

WF-Engine

C#Client

Service B

WF-Engine

Node.jsClient

has to implement

Retryhas to implement

Idempotency

Client Service Provider

Don‘t worry, it will happen safely –even if you loose connection.

Feel free to reload this page any time!

Photo by pixabay, available under Creative Commons CC0 1.0 license.

Requirement: Idempotency of services!

Photo by pixabay, available under Creative Commons CC0 1.0 license.

Requirement: Idempotency of services!

Photo by Chr.Späth, available under Public Domain.

Make every service idempotent!

CreditCard

Payment

Charge Credit CardcardNumber

amount

Charge Credit CardcardNumber

amounttransactionId

Not idempotent

Idempotent

charge

Distributed systems introduce complexity you have to tackle!

CreditCard

PaymentREST

Distributedsystems

It is impossible todifferentiate certain

failure scenarios.

Independant ofcommunication style!

Service Provider

Client

Distributed systems introduce complexity you have to tackle!

CreditCard

PaymentREST

Distributed systems introduce complexity you have to tackle!

CreditCard

PaymentREST

Cancel

charge

Executableand mature Easy to

understand*

ISO Standardwidespread

*(for Biz, Dev, and Ops)

BPMNBusiness Process

Model and Notation

ProperOperations

Visibility + Context

Challenges of asynchronicity

Check-in

Barcode Generator

Web-UI

Me

Output Mgmt

Asynchronous communication

You need tomonitor timeouts

Check-in

Barcode Generator

Web-UI

Me

Output Mgmt

Remember…

The failurenever leavesthis scope!

Workflow…

Workflow…

Who uses a message bus?

Who has no problemsoperating a message bus?

Dead messages | No context | Inaccesible payload | Hard to redeliver | Home-grown message hospitals | …

Manigfold architecture options

https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

Manigfold architecture options

https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

Manigfold architecture options

https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

DistributedTransactionsDistributed

Transactions

Transactions…

Do A

Do B

+ All ornothing

@berndruecker

Distributedsystems

Pat Helland

Grown-Ups Don’t Use Distributed Transactions

“

Distributed Systems Guru Worked at Amazon, Microsoft & Salesforce

Distributed transactions using compensation *

Compensation

„Lost in transaction“

https://vimeo.com/289508460

@berndrueckerhttps://berndruecker.io/lost-in-transaction/

has to implement

Timeout, Retry, Compensation

has to offer

Compensationhas to implement

Idempotency

Client Service Provider

has to implement

Timeout, Retry, Compensation

has to offer

Compensationhas to implement

Idempotency

Client Service ProviderDon‘t forgetabout state

Event-driven example also available

InventoryPaymentOrder ShippingCheckout Monitor

https://github.com/berndruecker/flowing-retail/

Human Tasks

H2 H2

# Be aware of complexity of distributed systems# Know strategies and tools to handle ite.g. Circuit breaker (Hystrix > Resilience4J)

Workflow engine for stateful retry, waiting, timeout and compensation (Camunda)

Thank you!

mail@berndruecker.io@berndruecker

https://berndruecker.io

https://medium.com/berndruecker

https://github.com/berndruecker

https://www.infoq.com/articles/events-workflow-automation

Contact:

Slides:

Blog:

Code:

https://www.infoworld.com/article/3254777/application-development/3-common-pitfalls-of-microservices-integrationand-how-to-avoid-them.html

https://thenewstack.io/5-workflow-automation-use-cases-you-might-not-have-considered/