exstreamlycheap final slides
TRANSCRIPT
ExStreamlyCheap.club
hello!
I am Emmanuel AwaInsight Data Engineering Fellow 2016 Spring cohort.
You can find me at @awaemma
content.
1. project motivation.
2. Engineering solution.
3. Challenges.
4. Takeaways
1.
Project Motivation
...what did I spend three weeks working on?
“I have learnt to seek my
happiness in limiting my desires, rather than attempting to satisfy
them.~ John Stuart Mill
Big concept1. Real time scalable deals serving
platform2. Insights and searching capabilities3. Maximizing time and profit.
sample queries
� User’s holistic view - real time trends visualization.
sample queries
� One highly scalable search platform: price or discount options.
sample queries
� Engineer User purchase interaction…
sample queries
� … and reaction.
2.
Engineering Solution
...Finding the right tools for the job.
data source - sqoot api
� Rich Merchant Info.○ Location based queries.
data source - sqoot api
� Scaled to all categories ○ ~ 11 million served.○ Over 80 categories.
pipeline - λ architecture
Ingestion Batch / Speed layer
Serving layer
Queries
batch pipeline HybridStreaming
API Interaction
Async Hybrid Distributed Query
Engine
restful api
QueriesServing LayerBatch Layer
Ingestion Layer
Real Time pipeline
Async Hybrid Distributed Query
Enginerestful api
QueriesServing LayerSpeed Layer
Ingestion Layer
Engineered user data
~ 200k events / min
3.
Challenges
...now it was easy, right?
project challenges.
BAD API DESIGN
1. Pagination.
2. Max #100 per page.
3. Dynamic api without firehose or sockets.
4. Duplicate deals with sync api calls.
ROBUST PLUGINS.
PyKafka vs Kafka-Python
1. Balanced consumer.
2. Topic to Partition assignment - HASH PARTITIONING.
GENERAL ENGR. CONSTRAINTS.
1. Design and architecture choice.
2. Tools deep dive - Tweak source code.
3. Constant Cassandra crashes. - Real time writes.
4. DevOps
4.
Takeaways
...some lessons learned.
PROPER DB INDEXES
� Partition and Clustering keys
CREATE TABLE trend_with_price PRIMARY KEY (price, discount)) WITH CLUSTERING ORDER BY (discount DESC);
� Secondary indexes
CREATE INDEX trend_with_price_category_idx ON trend_with_price (category);
secret to answering the
questions?
KAFKA CONSUMPTION OFFSETS
� Topic Offsets
Set the right start offset per partition
secret to answering the
questions?
secret to answering the
questions?
that’s all
folks!
About me
- Masters in CS
- 2 ½ yrs SE in Travel
- Nigerian - Hobbyist Photographer
BACKUP SLIDES
Benchmarking exercises
Elasticsearch and cassandra
METRICS - 20 GB of dirty data
1. I/O - Read and Writes.2. EC2 four (4) m4.xlarge
clusters
CONSIDERATIONS.
1. ElasticSearch vs Cassandra.
2. ElasticSearch on Cassandra.
GENERAL PROCESS FLOW.1. Read dirty python dictionary
from DB.2. Parse and process3. Write back to DB.4. Profile process.
ElasticSearch Advantages
1. Good for preserving data indexes.2. Great for more reads than writes.3. Analytics and text search.
Cassandra Advantages.
1. Good for fast writes.2. Preserving data schemas.3. Uptime critical and Time series data.
benchmarking
pipeline
Async Hybrid Distributed Query
Engine
restful api
QueriesServing LayerBatch Layer
Ingestion Layer
BIGGEST PROJECT
CHALLENGE - api constraintsAPI Pagination and max per page:
http://api.sqoot.com/v2/deals?api_key=xxxxxx;category_slug=home_goods;page=1;per_page=100
Freezing time for real-time non-fire-hose data source is hard
1. Three queries done at the same time. 2. Not fun – Inconsistent. 3. Application depends largely on total counts.
Page #1 loads first time Page #1 refresh in millisecs Page #2 loads
Async distributed query engine - (Async DQE)
1. First Stage Master
Producer (FSM)
2. Intermediate Hybrid
Consumer-Producer
3. Final Stage Consumer
THE END...