keystone - leverage big data 2016
TRANSCRIPT
The Netflix way to deal
with real-time data
How we built a 1t/day stream processing cloud platform in a year
What should I expect
� Keystone Season 1 - Who, What, How and Why � Keystone Season 2 - Preview Trailer
hello!
I am Peter BakasI lead the Real-Time Data Infrastructure team @ Netflix
You can find me at @peter_bakas
Daily Averages
� 700B unique events ingested � 1T events processed� 1.4 PB
By the numbers
Peak
� 1T unique events ingested/day� 12.5M/sec� 35 GB/sec
Q4 2015 - Keystone
InternalRouting Service
EMR
FrontingKafka
EventProducer
ConsumerKafka
Control PlaneHTTP
PROXY
Stream Consumers
Q4 2015 - Keystone
InternalRouting Service
EMR
FrontingKafka
EventProducer
ConsumerKafka
Control PlaneHTTP
PROXY
Stream Consumers
Keystone Kafka Footprint
Fronting Kafka Consumer Kafka
Number of Clusters 24 8
Number of Instances 3000+ 900+
Retention Period 8 to 24 hrs 2 to 4 hrs
Q4 2015 - Keystone
InternalRouting Service
EMR
FrontingKafka
EventProducer
ConsumerKafka
Control PlaneHTTP
PROXY
Stream Consumers
Keystone Internal Routing Service Footprint
S3 ElasticSearch Consumer Kafka
Number of containers 7000 1500 4500
“It may well be the most
important document to ever come out of the Valley
Sheryl Sandberg, COO @ Facebook
Lessons learned
� There are times when things can go wrong… and no turning back � Reduce complexity� Minimize blast radius� Find a way to start over fresh
Failover
� Cold standby Kafka cluster with different instance type � Different ZooKeeper cluster with no state� Fully automated
Place your screenshot hereTime is of the essence
Failover as fast as
5 minutes
Fully Automated
Failover
125,000,000 hrs/day
That’s a lot of hours!
37 %of North America internet traffic @ peak!
81,000,000 members
and a lot of members
Keystone
� Unified event publishing, collection, routing for batch and stream processing
� 85% of data volume
Keystone Messaging
Ad-hoc Messaging
� 15% of data volume
Consumers weary of
� Complexity of self-managed infrastructure
� Multiple runtimes across different platforms
Keystone Stream Processing
Consumers want
� Simple unified model/API/UI/system
Credits
Special thanks to all the people who made and released these awesome resources for free:� Presentation template by SlidesCarnival� Photographs by Unsplash