keystone event processing pipeline on a dockerized microservices architecture
TRANSCRIPT
![Page 1: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/1.jpg)
Keystone Event Processing Pipeline
Zhenzhong Xu
on a Dockerized Microservices Architecture
![Page 2: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/2.jpg)
Real-time Data Infrastructure – Netflix
Cloud Infrastructure - Microsoft
About Me
![Page 3: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/3.jpg)
About Netflix
● 83M+ Subscribers
● 125M+ Streaming Hours / Day
● > 1/3 Peak NA Internet Traffic
● Thousands of Device Types
● Many Tens of Thousands of VMs
● 3 Active-Active Regions Across the World
![Page 4: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/4.jpg)
Observe
Orient
Decide
Act CD
![Page 5: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/5.jpg)
Observe
Orient
Decide
Act
Innovation
CD
![Page 6: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/6.jpg)
Observe
Orient
Decide
Act
Innovation
Big Data
CD
![Page 7: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/7.jpg)
Observe
Orient
Decide
Act
Innovation
Big Data
Culture
CD
![Page 8: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/8.jpg)
Observe
Orient
Decide
Act
Innovation
Big Data
Culture
Cloud
CD
![Page 9: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/9.jpg)
Microservices Ecosystem
![Page 10: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/10.jpg)
Why a Event Processing Platform in Netflix?
● 500+ Billion events generated per day
● 1+T events processed per day
○ >1 PB
○ 4M – 16M / sec
○ 13GB - 43GB /sec
● Message Payload: 3 kb - 10mb
![Page 11: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/11.jpg)
Data Driven Culture
● Realtime System Failure Detection
● A/B Testing
● Recommendation Algorithm
● Fraud Detection
● Distributed Tracing
● Log Quering
![Page 12: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/12.jpg)
Paved Road in a Microservices Ecosystem
Microservices produces events
Storage service, and Batch/Stream
Processing services
Event Processing
Pipeline
![Page 13: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/13.jpg)
Paved Road in a Microservices Ecosystem
![Page 14: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/14.jpg)
Supports Batch & Streaming
![Page 15: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/15.jpg)
Evolution of Netflix Keystone Pipeline
![Page 16: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/16.jpg)
In the Old Days ...
EMR
EventProducers
![Page 17: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/17.jpg)
About a year ago
EventProducer
Druid
Stream Consumers
EMR
ConsumerKafka
Suro Router
EventProducer
Suro
Kafka
SuroProxy
![Page 18: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/18.jpg)
Today
Stream Consumers
SamzaRouter
EMR
FrontingKafka
ConsumerKafka
Control Plane
EventProducer
KS P
roxy
Self Service UI
![Page 19: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/19.jpg)
Event flowKeystone Pipeline As a Service
![Page 20: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/20.jpg)
Stream Consumers
SamzaRouter
EMR
FrontingKafka
EventProducer
ConsumerKafka
Control Plane
Self Service UI
![Page 21: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/21.jpg)
Stream Consumers
SamzaRouter
EMR
FrontingKafka
EventProducer
ConsumerKafka
Control Plane
Self Service UI
![Page 22: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/22.jpg)
Stream Consumers
SamzaRouter
EMR
FrontingKafka
EventProducer
ConsumerKafka
Control Plane
Self Service UI
![Page 23: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/23.jpg)
Stream Consumers
SamzaRouter
EMR
FrontingKafka
EventProducer
ConsumerKafka
Control Plane
Self Service UI
![Page 24: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/24.jpg)
Stream Consumers
SamzaRouter
EMR
FrontingKafka
EventProducer
ConsumerKafka
Control Plane
Self Service UI
![Page 25: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/25.jpg)
What exactly is Keystone?
![Page 26: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/26.jpg)
Keystone is ...
… a collection of microservices & components
Stream Processing
ServiceElastic
Pub/Sub Queue
Producer API
Control Plane
Consumer API
Self Service UI
![Page 27: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/27.jpg)
Keystone is ...… a single self-contained logical service
Event Processing
Pipeline
![Page 28: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/28.jpg)
Keystone is ...… an self-scaling, multi-tenancy service that embraces CI/CD
![Page 29: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/29.jpg)
Keystone is ...… a self healing, cloud failure tolerant service, guarantees at-least-once delivery semantics
![Page 30: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/30.jpg)
Let’s drill down ...
For the purposes of this talk, we’ll focus on...
Stream Processing
Service
Elastic Pub/Sub Queue
Producer SDK
Control Plane
Consumer SDK
Self Service UI
![Page 31: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/31.jpg)
Overview
![Page 32: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/32.jpg)
Self Service UI
![Page 33: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/33.jpg)
Routing Infrastructure
![Page 34: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/34.jpg)
Routing Infrastructure
EC2 InstancesZookeeper(Instance Id assignment)
JobJob
Job
ksnode
Checkpointing Cluster
Server Group (Cluster)Store logs
in S3
![Page 35: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/35.jpg)
Routing Infrastructure
+
CheckpointingCluster
+
0.9.1Go
C language
![Page 36: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/36.jpg)
Control Plane
Custom Cluster Orchestration and Scheduling Layer
![Page 37: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/37.jpg)
Control Plane
• Decides container resources
• Schedules container placements
• Orchestrates cluster deployments
![Page 38: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/38.jpg)
Design Decisions?Distributed System is all about
trade-offs.
![Page 39: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/39.jpg)
Container
● Process Isolation● Fast Startup
![Page 40: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/40.jpg)
Service Protocol
● Declarative
● Idempotent
● Reconciliation
![Page 41: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/41.jpg)
State Management
● Stateless vs Stateful service
● Single source of truth
![Page 42: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/42.jpg)
Scaling
● Self Scaling
● Partition boundary
● Idempotent operations
● Immutable server deployments
![Page 43: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/43.jpg)
Delivery Semantics
● At-most-once
● At-least-once (best effort)
● Exactly-once
![Page 44: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/44.jpg)
At-least-once under failure condition
● Checkpointing mechanism
● Optimize for writes
● Occasional reads
![Page 45: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/45.jpg)
Multi-tenancy
● Isolation
● Heterogenous
● Cluster fragmentation
![Page 46: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/46.jpg)
Failure Recovery
• Back pressure
• Network blip
• Container level failure
• Instance level failure
• Zone level failure
• Cluster level failure - Kafka-Kong
• Regional failure - Chaos-Kong
![Page 47: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/47.jpg)
Stream Processing Engine
• Discovery integration
• Custom wire format integration
• Samza: Per partition serialized process loop
• Samza: Simple payload transformation
• Plugable abstraction
![Page 48: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/48.jpg)
Current Scale - Routing Service
● 14,000 + docker containers
● 1,400 + EC2 C3-4XL instances
● 3 regions
![Page 49: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/49.jpg)
Future Improvements
● Integrate with more sophisticated orchestration/scheduling/cluster management ecosystem
● Unlock value in real-time unbounded data streams
● Data Discovery● Data Silos
![Page 50: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/50.jpg)
Questions?
![Page 51: Keystone event processing pipeline on a dockerized microservices architecture](https://reader036.vdocuments.mx/reader036/viewer/2022081604/5873bb491a28abbc788b55d5/html5/thumbnails/51.jpg)