lambda architecture 2.0 for reactive ab testing
TRANSCRIPT
Lambda Architecture 2.0 for Data-Driven Business
Team Trieu Nguyen - http://nguyentantrieu.info Truc Le - https://www.linkedin.com/pub/le-kien-truc/31/379/938
Data-driven + Lambda Architecture = growing business2
mc2ads.com - Fast Data Labs
Key questions for us today ?1. What if the business is not driven by data?
2. What and why is Lambda Architecture?3. What problems did it solve for us?
Workshop with case study: Improving “Flappy bird” with A/B Testing Tool and Lambda Architecture 2.0
Red bird Blue birdWhich bird could let you down soon ?OK, let’s play the Game ! Design it better with data
VS
Data-driven game design for Flappy Bird !
What if the business is not driven by data?
Refer: http://www.nytimes.com/2011/04/24/business/24unboxed.html
What if the business is not driven by data?
How ?Data-Driven Business
1970s 1990s 2000s 2010s
Data Management Technology and Trends● Netty.io● Apache Storm● Apache Kafka● Apache Spark● RFX● ...
● Hadoop Ecosystem● NoSQL Ecosystem● ...
● Oracle● MySQL● PostgreSQL● ...
Why is Lambda Architecture 2.0 ?It helps to organize your data infrastructure into understandable structure and react quickly to context changes
“Vision Without Execution Is Just Hallucination”
Ok, cool ideas, but how we build it ? Our
Our
We are here
Our goals
1. Understand the big picture2. See the reality3. Do actions to make it happenOk! Let’s make “Flappy bird” into “Happy bird” !
What is Lambda Architecture 2.0 ?It’s just the architecture for data-driven business
for reacting to fast data
for data mining and machine learning on Big Data
for observable data
for SQL querying (SQL is true lambda language !?)
Case study:Improving “Flappy bird” with A/B Testing Tool and Lambda Architecture 2.0
● Short introduction about A/B testing● Setup full open source technology stack● Run example code with Java and Python
Our goal: Applying data-driven in Game Design
forecasting
Make correct decision
How? One of basic principle is “Test our theory”
From observable solutions, test them all to find the best one ! More at http://en.wikipedia.org/wiki/A/B_testing
1. Working with A/B testing tool (using Abba framework)2. Let’s play Flappy Bird 2.0 !3. Collecting data → store data as stream (Kafka)4. Stream processing → real-time view processing (RFX)5. Batch processing → sampling AB Test (Spark) 6. Query processing → finding facts from experiment
(SQL over Phoenix / HBase)7. Collecting feedback data → Game Design Report
Steps
For simple demo, we use Abba, a simple A/B testing self-hosted framework
Why is reactive view in Lambda Architecture 2.0 ?UX is the key for successful product development, so we must react to bad UX quickly (with data)
Technology stack ( 5D model )1) Data collector (I/O networking)
● Netty for event log collector and HTTP server (lambda2)2) Data persistence (aka: data storage)
● Kafka for distributed message storage (Apache Kafka)● HBase for scalable big table
3) Data processing● RFX with fast data processing (RFX framework)● Python for data sampling in A/B test experiments● Rx(Java/JS) for reacting to data experiment (reactivex)
4) Data analysis● measures of uncertainty(Python Dempster-Shafer theory)
5) Data ad-hoc reporting● SQL over Phoenix / HBase ( http://phoenix.apache.org )
