the hardest part of microservices: your data - christian posta, red hat

Full slide deck here: h"p://bit.ly/ceposta-hardest-part

Twitter: @christianposta

Blog: http://blog.christianposta.com

Email: christian@redhat.com

Christian Posta Principal Architect – Red Hat

•  Author “Microservices for Java Developers”

•  Committer/contributor Apache Camel, Apache ActiveMQ,

Fabric8.io, Apache Kafka, Debezium.io, et. al.

•  Worked with large Microservices, web-scale, unicorn company

•  Blogger, speaker about DevOps, integration, and microservices

PeopletrytocopyNet,lix,buttheycanonlycopywhattheysee.Theycopytheresults,nottheprocess.

AdrianCockcro,,formerChiefCloudArchitect,Ne6lix

“Microservices” is about optimizing … for speed.

Howdoesyourcompanygofast?

Managedependencies.

Wait.Whatisdata?

Whatisone“thing”?

Bookcheckout/purchase TitleSearch

RecommendaBons

WeeklyreporBng

Focusondomainmodels,notdatamodels

•  Break things into smaller,

understandable models

•  Surround a model and its

“context” with an explicit

boundary

•  Implement the model in code

or get a new model

•  Explicitly map between

different contexts

•  Model transactional

boundaries as aggregates

SBckwiththeseconveniencesaslongasyoucan.Seriously.

But...

•  Load/sizeistoogreattofitononebox

•  Modules/usecaseshavedifferentread/writecharacterisBcs

•  Queries/joinsaregeOngtoocomplex

•  Securityissues

•  LotsofconflicBngchangestothemodel/schema

•  Needdenormalized,opBmizedindexingengines

•  Wewanttoexplicitlyreducedependenciesondatabetween

ourservices

Fromhereonout,whatwe’resayingis“thankyouoldwork-horsedatabase,we’vegotitfromhere”…

“Amicroservicehasitsowndatabase”

We’renowbuildingafull-fledgeddata-centricdistributedsystem.Somethingstoremember…

Planforfailures.

BuildconceptsofBme,delay,

network,andfailuresintothe

designasafirst-classciBzen.

h"ps://secure.phabricator.com/book/phabcontrib/arBcle/n_plus_one/

h"ps://secure.phabricator.com/book/phabcontrib/arBcle/n_plus_one/

getBulkHats()

getBulkHatsForCatsExcept()

wellReallyIJustWantCertainHats()

justExecuteThisSqlForMe()

Weneed“consistency”.Butweexpectfailures.ThisisstarBngtosoundlikeCAP…

Consistencymodels…

h"ps://en.wikipedia.org/wiki/Consistency_model

•  Strict consistency (Linearizability) •  Sequential consistency •  Causal consistency •  Processor consistency •  PRAM consistency (FIFO) •  Bounded staleness consistency •  Monotonic read consistency •  Monotonic write consistency •  Read your writes consistency •  Eventual consistency

ReplicatedDataConsistencyExplainedthroughBaseball(DougTerry)

h"ps://www.microsoa.com/en-us/research/publicaBon/replicated-data-consistency-explained-through-baseball/

•  What consistency model do you need, depending on what role you’re playing?

•  What consistency model are you willing to pay for? •  Official score keeper? (Linearizability or RMW) •  Umpire? (Linearizability) •  Sports writer? (Bounded staleness, Eventual

consistency) •  Radio updates? (Monotonic read, Bounded staleness) •  Statistician (Bounded staleness) •  Friends in the pub (Eventual consistency)

MaybewecanusearelaxedconsistencymodelforsomeofthosepreviouslymenBonedusecases…

...andsolvefordata-sharingissueswhiletakinginto

accountthenetworkandfailures.

ExampleusingsequenBalconsistency…

Whatwe’vedoneisgoneoffandbuiltadatasystemattheapplicaBonlayer.

Andthisiswhattheinternetcompaniesdidalso.(someevenopensource!!)

•  Yelp – MySQL Streamer

https://github.com/Yelp/mysql_streamer

•  LinkedIn – Databus

https://github.com/linkedin/databus

•  Zendesk – Maxwell

https://github.com/zendesk/maxwell

Meetdebezium.io

Meetdebezium.io

Twitter: @christianposta

Blog: http://blog.christianposta.com

Email: christian@redhat.com

Thanks for listening! Time for demo?

Full slide deck here: h"p://bit.ly/ceposta-hardest-part