start small bigger biggest

Start Small Bigger BiggestThe Event Driven Organisation

Based on the Lambda Architecture

Who Is That?

Daan Gerits

BigData.be co-organizer

daan.gerits@gmail.com

@daangerits

Who Is That?

BigData.be

Belgian BigData Group

Founded in juli 2011

+200 members

http://www.bigdata.be

http://www.meetup.com/bigdatabe

@bigdatabe

Agenda

TransCo

The Event Driven OrganisationEventsViewsIntegration

TransCo

Meet TransCoInternational Transport

Track & TraceFreelance DriversCustom SoftwareEnterprise Service Bus

TransCo

DepartmentsITFinanceLegalLogisticsSalesCommunicationsMarketing...

TransCo

A customer requests a quote

A server goes down

A website visitor clicks on a link

A financial transaction is booked

A delivery truck pings its GPS coördinates

TransCo

All these have a similar thing:

They Generate Events

Events

ITFinanceLegalLogisticsSalesCommunications...

Events

Events

Events used to manipulate their master data

Events

Today, events ARE their master data

TransCo

Isn't this a big data presentation?

Hang on, We'll get to that

Events

An event isTime basedRecord linkedField orientedIncrementalImmutable

Events

A new customer is addedADD [ts=1,id=123,field="name",value="Daan Gerits"]

ADD [ts=1,id=123,field="address",value="MyStreet 40"]

A customer is updatedADD [ts=13,id=123,field="address",value="AnotherStreet 1"]

Eventually a customer is removedADD [ts=43,id=123,field="address",value=]

Events

Notice I only created new records

No UpdatesNo Deletes

CRUD becomes CR

Events

Oh no, I shouldn't have done that!DELETE [ts=43,id=123,field="address"]

TransCo

Discover market trends

Application and server statistics

Direct Marketing

Financial validation

Tracking Dashboard

Views

View = Fx All Data

Views

View =

Different View TypesReal-timeRelationalReportingApplicationAPI...

Views

Views areFull data modelsBased on all dataFocussed

Views

Integration

How do I integrate with my existing infrastructure?

Views can also be stored in- RDBMS- Indexing Engine- Files- DataWarehouse- ...

But these views are read-only!If you want read-write use the API

Integration

Big Data

So what has all this to do with Big Data?

Only appending eventsneed to store a lot of data

=> HDFS, HBase, ...

need batch processing=> MapReduce, Pig, Hive, ...

need real-time processing=> Storm, S4, ...

Big Data

Allows trendingdriver statistics for his whole career

Allows state regenerationthe state of all master data on februari 12, 2005

Is human-error-proofremove the last event on this record

Scales very well

Event Driven Organisation

By storing events

By generating focussed views

Event Driven Organisation

Thank you!Daan Gerits

@daangeritsdaan.gerits@gmail.com

Backup SlidesGenerating Views

Views

Realtime Views

Views

Latency

Views

Generating takes time=> View is always x hours old

View Ready

Start Generating + + + ++

Views

How to solve this?Store events in 2 locations

Master Data SetBuffer DB

Buffer DB has - data of last x hours- data, not events

Views

Speed View

API

Batch View

Views

Batch ViewsUse MapReduce

Speed ViewsUse Storm or something similar