CQRS + ESDDD

• Intro!

• CAP

• CQRS

• Event Driven

• Event Sourcing / Event Store

• Appendix

performance trouble?

scalability trouble?

big large data?

high frequency data?

complex business domain?

performance problems

if your system is slow for a single user

scalability problems

if your system is fast for a single user but slow under heavy load

system level performance

maximal throughput

acceptable latency

system level requirements

• fast response time = high availability

• slow response time = low availability

• instant (secure) consistency = ACID (Atomicity, Consistency, Isolation und Durability)

• time window consistency = BASE (Basically Available, Soft state, Eventual consistency)

common mistakes• The network is fail-safe

• The latency time is equal to zero

• The data throughput is infinite

• The network is secure

• The network topology will not change

• There is always only one network administrator

• The cost of data transport can be calculated as zero,

• The network is homogeneous

• Intro

• CAP!

• CQRS

• Event Driven

• Event Sourcing / Event Store

• Appendix

You can only pick 2

At a given point in time!

Consistency

Availability

Partition tolerance

CAP

– Brewer

„…a distributed system cannot satisfy all three of these guarantees at the same time…“

Centralized SystemsIn a centralized system (RDBMS etc.) we don’t have network partitions e.g. P in CAP

you GET BOTH:

Consistency

Availability

Distributed SystemsIn a distributed system we have network partitions e.g. P in CAP

You can get to ONLY PICK ONE:

Consistency

Availability

CAP in practice• … there are only 2 types of systems

• CP!

• AP!

• … there is only one choice. In case of existing network partitions, what do you prefer?

1. Consistency

2. Availability

BASE Eventual consistency

Consistency of data is seen as a state which is eventually attained.

CA CP

AP

Where are you?

SOFTWARE SYSTEM

valid correct solid

data driveneasy & fast

flexible modern

task driven

different needssystem level

different needs

UI usability

Data-Driven Task-based

• Reporting / Search? • State Changes? • Business behaviors and rules? • Algorithms / Calculations? • ORM + rich domain model is a

anti pattern! • The reading of an object ended

very often in charge of the entire database.

Data Modeldifferent needs

Storage

• When do you need ACID? • When Eventually

Consistent better fit? • Should Reporting / Search

strictly consistent?

Scaling out READS to a RDBMS is hard (sharing/replication) Scaling out WRITES to a RDBMS is impossible!

different needs

–Greg Young 2008

„A single model cannot be appropriate for reporting, searching and transactional

behavior.“

So… one fits all?

CA CP

AP

needsmultiple systems Data Administration Bank Service / ATM NoSQL

Web-/Mobile End-Users DNS Web Caching

single system RDBMS Data Analytics

CA

CPAP

Sometimes!

Do you really need a RDBMS?

But many times we don’t!

application building (today)

ACID <vs.> EVENTUAL CONSISTENCY

CUD <vs.> BEHAVIORAL DOMAIN

READ <vs.> SEARCH / REPORTS

DATA DRIVEN <vs.> TASK BASED

Think about your data: Different data need different guarantees

• Intro

• CAP

• CQRS!

• Event Driven

• Event Sourcing / Event Store

• Appendix

CQRScommand query responsibility segregation

conceptual

strictly segregates the responsibility

independent optimization and specialization

asymmetrical execution of commands & queries

„Do your domain experts speak with only four verbs (Create Read Update Delete)?“

CRUD & OOP

CQSBertrand Meyer's

split interfaces into

state changes Commands

side-effect free Queries

simple pattern

• side-effect-free query/read access

• autonomous commands represent behavioral state changes

–Greg Young 2008

„A single model cannot be appropriate for reporting, searching and transactional behavior.“

That’s it!

CRUD <—> CQRS

CRUD <—> CQRS

conceptual

@Microsoft

CQRS

side-effect free processing of queries

execution of commands represent changes in the state of the software system

scaling benefits

host the two services (read / write) differently

we can host the read service on 25 servers

write service on one server

read side benefits

• top-down driven and optimized view data model

• denormalized schema - no joins

• easy queries (select * from DB)

• no mapping

• no ORM mismatches

write side benefits

• behavioral driven

• exposes only functionality

• no read access

• (event driven) state machine

• forms a consistency boundary

• Intro

• CAP

• CQRS!

• Event Driven !

• Event Sourcing / Event Store

• Appendix

Event-Driven

@Microsoft

–Eric Evans, 2009

„It’s really become clear to me in the last couple of years that we need a new building block and

that is the Domain Events.“

Events

–Greg Young, 2008

„State transitions are an important part of our problem space and should be modeled within

our domain“

*syntax notescommands

events

domain events

business events

notifications

Messages

events

something that has happened in the past

CustomerHandled CargoShipped

commands

something that should do named in imperative

ChangeCustomerLocale MakeCustomerPreferred

storing structure

events & commands• uniquely identifiable • self contained • pure data structure - no behavior • Observable • Time relevant • non-blocking - async execution • conceptual immutable e.g. new, deep copy,

clone • events represent serialized function calls • conceptual duality between function calls and

messages

flow

flow in a nutshell• Write side receive Commands and publish

Events

• All state changes are represented by Business Events

• Read side is updated as a result of the published Events

• All Queries go directly to the Reporting, the Business-Logic is not involved

flow

Common benefits• Focus to Business

• allow to manage business and technology risks separately

• Technology agnostic, platform neutral and easy externals integration

• Fully encapsulated domain that only exposes behavior

• Queries do not use the business model

• No object-relational (ORM) impedance mismatch

• Scalable and Flexible and Testable

• Reduced complexity and simplify system architecture

benefits Event-Driven

• Explicit error/failure modeling

• Easy integration with external systems

• Location Transparency

• Reactive Behavior

• Bullet-proof auditing and historical tracing

Event Aggregator

• publish / subscribe pattern • observe events • dispatch / invoke functions sync or async

Command-Handler

• execute functions by command (type) • integration point for cross-cutting concerns (logging,

transactions, validating, authorization)

Event-Handler

• execute functions by event (type) • view or data model generation • schema denormalizer

trade-offs• initially feels complex

• Consistency Boundaries (isn’t a top level architecture)

• Monitoring / Heartbeats

• Stale Data

• Collaborative Domains

• Data Integration

• Modeling with time

• Manage idempotency and event de-duplication

• Intro

• CAP

• CQRS

• Event Driven

• Event Sourcing / Event Store!

• Appendix

Event Sourcing

Event SourcingStoring Deltas

Event Sourcingreplaying events

Event StoreEvent Store

Event Sourcerecord

meta data time stamp

payload identity version

append only log

replay

Produce

Consume #1

#5

#9

Event Sourcing

Every state change is materialized in an Event All Events are sent to an EventProcessor EventProcessor stores all events in an Event Log System can be reset and Event Log replayed Many different EventListeners can be added to EventProcessor (or listen directly on the Event Log) No need for ORM, just persist the Events

Event Sourcing

recover state

1. EventProcessor record & replay all events in the context of a consistency boundary

2. projections - loop over a series of events and aggregate to a state

f(state, event) → state

Event ProcessorRestoring State from Deltas

• Event Processor are tracking events as to what has changed

• Current state is constructed by replaying events

• Data is not persisted in a structure but as a series of transactions

• No ORM is needed

Projections

Query language of Event Streams

Filter, Map, Reduce over Event Streams

Projection is about deriving current state from the stream of events

Given the stream of events, we can transform them to any structural representation

• Recording all state changes • is a business transaction history • Event History + Time Machine • Replay summary of total records is the last state • any regeneration of models • projections as a useful event stream query language • time related analysis and statistics (forecastings)

Event Sourcingbenefits

Event Sourcingbenefits

• No object-relational impedance mismatch • time ordered series of business transactions • Support future ways of looking at data • Performance and scalability • Testability • Reconstruct production scenarios • Long-lived processes, tracking and auditability

• Intro

• CAP

• CQRS

• Event Driven

• Event Sourcing / Event Store

• Appendix

Thanks Questions?

appendix• messaging pattern

• idempotency and de-duplication

Publish / Subscribe

Messaging Pattern

Store and Forward

Messaging Pattern

Request-Reply

Messaging Pattern

Point-to-Point

Messaging Pattern

f(x) = f(f(x))request twice means the same outcome

Idempotency

Idempotency

• natural idempotent operations are mostly easy

• nicely for state machines with alter state without side-effects

• turn the light ON - duplicate calls generate some result - the light is on

• not naturally idempotent operations are hard

• only process ones that I see are new

• Balance += Amount - increases balance again and again

Idempotency

• de-duplicated messages by query a log store • check incoming message by an identifier agains a log store

and reject if a identifier match • check against a correlation identifier (e.g. ETags,

TransactionID) • check against a computed HASH of the complete message

(like GIT commits)

Idempotency

• using a buffer

• time throttling