#netflixeverywhere global architecture - qcon london · [email protected]. bringing movies &...
TRANSCRIPT
![Page 1: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/1.jpg)
Josh Evans - Director of Operations Engineering
March, 2016
#NetflixEverywhereGlobal Architecture
![Page 2: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/2.jpg)
December 24th, 2012
![Page 3: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/3.jpg)
Disappointment
![Page 4: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/4.jpg)
Outrage
![Page 5: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/5.jpg)
Withdrawal
![Page 6: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/6.jpg)
December 24th, 2012
![Page 7: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/7.jpg)
![Page 8: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/8.jpg)
Failure is inevitable
![Page 9: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/9.jpg)
Never fail the same way twice
Failure-Driven Architecture
![Page 10: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/10.jpg)
#NetflixEverywhere
Failure-Driven Architecture
Never fail the same way twice
![Page 11: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/11.jpg)
• Introductions
• Failure-Driven Architecture
• Taking It Global
#NetflixEverywhere
Our Talk Today
![Page 12: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/12.jpg)
• Introductions• Failure-Driven Architecture
• Taking It Global
#NetflixEverywhere
Our Talk Today
![Page 13: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/13.jpg)
1999 – 2009• Ecommerce (DVD Streaming)
2009 – 2013• Playback Services (Activate, Manifests, DRM)
2013 - present• Operations Engineering
– CD, RTA, Chaos, Performance
Josh Evans – Director of Operations Engineering
![Page 14: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/14.jpg)
Bringing movies & TV shows from all over the world to people all over the world
• Streaming, on demand, subscription• Global & regional licensing• Hollywood, independent, international
• Striving for global ubiquity
![Page 15: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/15.jpg)
2007
• Jan – Windows
2008
• May – Roku
• Oct – LG, Samsung Blu-ray
• Oct – Apple Mac
• Nov – XBox 360
2009
• Jun – LG DTV
• Nov –Sony PS3 (disc)
• Nov – Sony Bravia– DTV & Blu-ray
Device Ubiquity
![Page 16: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/16.jpg)
2011
• May – Android
• Nov – First e-readers– Kindle Fire, Nook
Device Ubiquity2010
• Mar – Nintendo Wii (disc)
• Apr – Apple iPad
• Aug – Apple iPhone
• Sep – Apple TV
• Oct – Sony PS3 (no disc)
• Oct – Nintendo Wii (no disc)
• Nov – Windows Phone 7
![Page 17: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/17.jpg)
![Page 18: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/18.jpg)
![Page 19: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/19.jpg)
2010 - Canada
2011 - Latin America
2012 - UK, Ireland, Nordics
2013 – Netherlands
2014 - Austria, Belgium, France, Germany, Luxembourg, Switzerland
2015 - Australia, New Zealand, Japan, Spain, Italy, Portugal
Geographic Ubiquity
![Page 20: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/20.jpg)
• English• Spanish (Latin American)• Portuguese (Brazilian)• Dutch• French• German• Japanese• Spanish (Castilian)• Italian• Portuguese (European)
Language Ubiquity - Subs, Dubs, UI
![Page 21: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/21.jpg)
75,000,000
![Page 22: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/22.jpg)
• Introductions
• Failure-Driven Architecture• Taking It Global
#NetflixEverywhere
Our Talk Today
![Page 23: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/23.jpg)
August 2008
![Page 24: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/24.jpg)
• No automation, virtualization, standardization• Manual, error prone, slow• Big iron & monoliths
DC22009
![Page 25: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/25.jpg)
Undifferentiated Heavy Lifting
![Page 26: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/26.jpg)
US-East-1
Amazon Web Services2010
• Scale & elasticity• Virtual, programmable• Global footprint
![Page 27: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/27.jpg)
• Micro-services
• Database
• Cache
• Traffic
Architectural Pillars
![Page 28: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/28.jpg)
• Micro-services
• Database
• Cache
• Traffic
Architectural Pillars
![Page 29: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/29.jpg)
![Page 30: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/30.jpg)
![Page 31: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/31.jpg)
![Page 32: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/32.jpg)
FITFault-Injection
Test Framework
Micro-service Failure
![Page 33: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/33.jpg)
• Micro-services
• Database
• Cache
• Traffic
Architectural Pillars
![Page 34: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/34.jpg)
NoSQL but…
• Not web scale
• Throttling
Modest scale• 100s of play starts / second• 10,000s of requests / second• 10s of billions of records
SimpleDB
![Page 35: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/35.jpg)
• Micro-services
• Database
• Cache
• Traffic
Architectural Pillars
![Page 36: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/36.jpg)
Ephemeral Volatile memCache (EVCache)
Clustered memcached optimized for Netflix use cases
![Page 37: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/37.jpg)
EVCache Server
Memcached
Prana (Sidecar)
Monitoring & Other Processes
Eureka
Client Application
Client Library
EVCache Client
Shards, consistent hashing
TTLs & LRU
EVCache Architecture
![Page 38: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/38.jpg)
Zone A
Client Application
Client Library
EVCache Client
Zone B
Client Application
Client Library
EVCache Client
Zone C
Client Application
Client Library
EVCache Client
. . .. . .. . .
Reads
![Page 39: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/39.jpg)
Zone A Zone B Zone C
. . .. . .. . .
Writes
Client Application
Client Library
EVCache Client
Client Application
Client Library
EVCache Client
Client Application
Client Library
EVCache Client
![Page 40: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/40.jpg)
1. Read from cache2. On cache miss call service3. Service calls DB & responds4. Service updates cache
Client Application
Client Library
EVCache Client Service Client
S S S S. . .
DB DB DB DB. . .
. . .
Fronting Micro-services
. . .
![Page 41: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/41.jpg)
Linear Scaling
• 30 million requests/sec
• 2 trillion requests per day globally
• Hundreds of billions of objects
• Tens of thousands of memcached instances
• Milliseconds of latency per request
![Page 42: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/42.jpg)
US-East-1
Canada
International Expansion2011
US
Latin America
![Page 43: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/43.jpg)
US-East-1 EU-West-1
Cloud Islands2012
![Page 44: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/44.jpg)
• Micro-services
• Database
• Cache
• Traffic
Architectural Pillars
![Page 45: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/45.jpg)
US-East-1 EU-West-1
UK/IE, Nordics, Netherlands
Latin America
DNS Geo Mapping
Canada
US
![Page 46: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/46.jpg)
• Micro-services
• Database!
• Caching
• Traffic
Architectural Pillars
![Page 47: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/47.jpg)
Why Cassandra?• NoSQL at scale
• Open source
• Multi-region
• Multi-directional
• CAP Choices
– Availability
– Partition tolerance
– Eventual consistency*
Scalable, Durable, Global
![Page 48: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/48.jpg)
Single Region, Multiple AZs
1. Client writes to any node
2. Coordinator replicates to nodes
3. Nodes ack to coordinator
4. Coordinator acks to client
5. Write to commit log
Zone A
Zone B
Zone C
Zone B
Zone C
Client
Zone A
• Hinted handoff to offline nodes
Local Quorum(Typical)
100ms
![Page 49: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/49.jpg)
Not quite fast enough
![Page 50: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/50.jpg)
December 24th, 2012
![Page 51: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/51.jpg)
US-East-1US-West-2 EU-West-1
IsthmusSpring 2013
Survive a regional ELB outage
![Page 52: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/52.jpg)
AZ1 AZ2 AZ3
US-EAST-1 ELBs
Zuul
Data Data Data
Geo-locatedstate/province
AZ1 AZ2 AZ3
US-WEST-2 ELBs
Zuul
Data Data Data
Americas Internet Traffic
Eastern NA +
LatAm Traffic
• Zuul routes locally or remotely
• Eureka - multi-region aware
Isthmus
![Page 53: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/53.jpg)
US-East-1US-West-2 EU-West-1
Active-Active2013 - 2014
Survive a large-scale regional service outage
![Page 54: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/54.jpg)
Active-Active Data Replication
![Page 55: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/55.jpg)
Region BRegion A
Zone A
Zone B
Zone C
Zone B
Zone C
Zone A
Zone A
Zone B
Zone C
Zone C
Client Client
Zone AZone B
Multi-Region Writes
500ms
Bi-directional
Nightly compare & repair
Local Quorum(Typical)
![Page 56: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/56.jpg)
EVCache Replication
Repl Writer
SQS
Application
Client
EVCache Replication
Repl Writer
1. Set or delete
2. send metadata
3. poll msg
6. set or delete
Application
Client
SQS
7. read
EVCache Cross-Region Replication
Region BRegion A
![Page 57: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/57.jpg)
Active-Active Traffic Management
![Page 58: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/58.jpg)
ELB US-West-2 ELB US-East-1 ELB EU-West-1
DNS api-global.netflix.com UltraDNS
Route53
![Page 59: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/59.jpg)
DNS api-global.netflix.com • Remove state from geo bucket
ELB US-West-2 ELB US-East-1 ELB EU-West-1
![Page 60: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/60.jpg)
api-global.netflix.comDNS • Remove state from geo bucket• Add state to geo bucket • Log event
• For each end point
ELB US-West-2 ELB US-East-1 ELB EU-West-1
![Page 61: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/61.jpg)
api-global.netflix.com
api-global.us-west-2.prodaa.neflix.com
api-global.us-east-1.prodaa.neflix.com
api-global.eu-west-1.prodaa.neflix.com
ELB ELB ELB
Shim
![Page 62: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/62.jpg)
api-global.netflix.com
api-global.us-west-2.prodaa.neflix.com
api-global.us-east-1.prodaa.neflix.com
api-global.eu-west-1.prodaa.neflix.com
ELB ELB ELB
Shim
![Page 63: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/63.jpg)
api-global.netflix.com
api-global.us-west-2.prodaa.neflix.com
api-global.us-east-1.prodaa.neflix.com
api-global.eu-west-1.prodaa.neflix.com
ELB ELB ELB
Shim
![Page 64: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/64.jpg)
Active-Active Failover
![Page 65: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/65.jpg)
![Page 66: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/66.jpg)
• Introductions
• Failure-Driven Architecture
• Taking It Global
#NetflixEverywhere
Our Talk Today
![Page 67: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/67.jpg)
January 6th, 2016
![Page 68: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/68.jpg)
Geographic Ubiquity
![Page 69: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/69.jpg)
Before Global• English• Spanish (Latin American)• Portuguese (Brazilian)• Dutch• French• German• Japanese• Spanish (Castilian)• Italian• Portuguese (European)
Global• Chinese
• Korean
• Arabic
Language Ubiquity
![Page 70: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/70.jpg)
![Page 71: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/71.jpg)
![Page 72: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/72.jpg)
March 18th, 2016
Daredevil Season 2
All episodes, all devices, all countries
Simultaneously
Content Ubiquity
![Page 73: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/73.jpg)
Ubiquitous, Resilient Architecture
![Page 74: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/74.jpg)
US-East-1US-West-2 EU-West-1
Reliably and efficiently serve any customer from any region
Netflix Global 2015
![Page 75: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/75.jpg)
US-East-1US-West-2 EU-West-1
![Page 76: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/76.jpg)
US-East-1US-West-2 EU-West-1
![Page 77: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/77.jpg)
Ubiquitous Data
![Page 78: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/78.jpg)
EVCache Replication
Repl Writer
Application
Client
Kafka
SQS
• High latency
• Read once
Kafka
• Low latency
• Multiple readers
• > 1M replications/sec
![Page 79: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/79.jpg)
US Ring US Ring
EU Ring
EU-West-1US-East-1
1. Extend US ring to EU region & run repairs
Client2. Dual Write
3. Forklift
![Page 80: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/80.jpg)
EU-West-1US-West-2
Global Ring
Global Ring
US-East-1
Global Ring
![Page 81: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/81.jpg)
Ubiquitous Traffic Management
![Page 82: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/82.jpg)
us-east-1-na
• East US
• East CA
• MX
us-west-2
• APAC
• West US
• West CA
eu-west-1
• Europe
• Mid East
• Africa
us-east-1-sa
• LatAm
• Not MX
Virtual DNS Regions
![Page 83: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/83.jpg)
• Fixed virtual modules
• Origin tier
• Standardized names
api-global.netflix.com
api-global.us-west-2.prodaa.neflix.com
api-global.us-east-1-sa.prodaa.neflix.com
api-global.us-east-1-na.prodaa.neflix.com
api-global.eu-west-1.prodaa.netflix.com
api-global.us-west-2.origin.prodaa.neflix.com
api-global.us-east-1.origin.prodaa.neflix.com
api-global.eu-west-1.origin.prodaa.neflix.com
ELB ELB ELB
Virtual
Origin
DNS Tiers
![Page 84: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/84.jpg)
api-global.netflix.com
api-global.us-west-2.prodaa.neflix.com
api-global.us-east-1-sa.prodaa.neflix.com
api-global.us-east-1-na.prodaa.neflix.com
api-global.eu-west-1.prodaa.netflix.com
api-global.us-west-2.origin.prodaa.neflix.com
api-global.us-east-1.origin.prodaa.neflix.com
api-global.eu-west-1.origin.prodaa.neflix.com
ELB ELB ELB
Virtual
Origin
Split Failover
![Page 85: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/85.jpg)
api-global.netflix.com
api-global.us-west-2.prodaa.neflix.com
api-global.us-east-1-sa.prodaa.neflix.com
api-global.us-east-1-na.prodaa.neflix.com
api-global.eu-west-1.prodaa.neflix.com
api-global.us-west-2.origin.prodaa.neflix.com
api-global.us-east-1.origin.prodaa.neflix.com
api-global.eu-west-1.origin.prodaa.neflix.com
ELB ELB ELB
Virtual
Origin
Split Failover
![Page 86: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/86.jpg)
api-global.netflix.com
api-global.us-west-2.prodaa.neflix.com
api-global.us-east-1-sa.prodaa.neflix.com
api-global.us-east-1-na.prodaa.neflix.com
api-global.eu-west-1.prodaa.netflix.com
api-global.us-west-2.origin.prodaa.neflix.com
api-global.us-east-1.origin.prodaa.neflix.com
api-global.eu-west-1.origin.prodaa.neflix.com
ELB ELB ELB
Virtual
Origin
Cascading Failover
![Page 87: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/87.jpg)
api-global.netflix.com
api-global.us-west-2.prodaa.neflix.com
api-global.us-east-1-sa.prodaa.neflix.com
api-global.us-east-1-na.prodaa.neflix.com
api-global.eu-west-1.prodaa.netflix.com
api-global.us-west-2.origin.prodaa.neflix.com
api-global.us-east-1.origin.prodaa.neflix.com
api-global.eu-west-1.origin.prodaa.neflix.com
ELB ELB ELB
Virtual
Origin
Cascading Failover
![Page 88: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/88.jpg)
api-global.netflix.com
api-global.us-west-2.prodaa.neflix.com
api-global.us-east-1-sa.prodaa.neflix.com
api-global.us-east-1-na.prodaa.neflix.com
api-global.eu-west-1.prodaa.netflix.com
api-global.us-west-2.origin.prodaa.neflix.com
api-global.us-east-1.origin.prodaa.neflix.com
api-global.eu-west-1.origin.prodaa.neflix.com
ELB ELB ELB
Virtual
Origin
Cascading Failover
![Page 89: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/89.jpg)
api-global.netflix.com
api-global.us-west-2.prodaa.neflix.com
api-global.us-east-1-sa.prodaa.neflix.com
api-global.us-east-1-na.prodaa.neflix.com
api-global.eu-west-1.prodaa.netflix.com
api-global.us-west-2.origin.prodaa.neflix.com
api-global.us-east-1.origin.prodaa.neflix.com
api-global.eu-west-1.origin.prodaa.neflix.com
ELB ELB ELB
Virtual
Origin
Cascading Failover
![Page 90: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/90.jpg)
![Page 91: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/91.jpg)
x x
Multi-region Failover
![Page 92: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/92.jpg)
January 6th, 2016
![Page 93: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/93.jpg)
“Going global is just like having a baby.”- Reed Hastings, Netflix CEO
![Page 94: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/94.jpg)
What’s next?• Global latency• Edge computing• ML-based monitoring• Self-healing systems• Capacity utilization• Fast, autonomous traffic• Integrate DB & caching
#NetflixEverywhere
![Page 95: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/95.jpg)
Takeaways
![Page 96: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/96.jpg)
Never fail the same way twice
Christmas Eve2012
Today
![Page 97: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/97.jpg)
Know your resiliency patterns
Pattern Properties
DC SPoF, infrastructure heavy lifting
Cloud (one region) Multiple DCs, one control plane
Islands Regional containment
Isthmus Regional ELB bypass
Active-active Regional failover
Global Ubiquity, resiliency, efficiency
![Page 98: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/98.jpg)
Invest in architectural pillars
• Micro-services
• Database
• Caching
• Traffic
![Page 99: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/99.jpg)
#NetflixEverywhere
Think globally, act locally
![Page 100: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/100.jpg)
netflix.github.io
![Page 101: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/101.jpg)
netflix.github.io
![Page 102: #NetflixEverywhere Global Architecture - QCon London · jevans@netflix.com. Bringing movies & TV shows from all over the ... EVCache Architecture. Zone A Client Application Client](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b03b5237f8b9a2d518c9069/html5/thumbnails/102.jpg)
Netflix Tech Blog
techblog.netflix.com