migrating to cassandra in the cloud, the netflix way to cassandra in the cloud, the netflix way...
TRANSCRIPT
![Page 1: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/1.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Migrating to Cassandra in the Cloud, the Netflix Way
Jason Brown - @jasobrown
Senior Software Engineer, Netflix
![Page 2: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/2.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Tech History, 1998-2008
In the beginning, there was the webapp and a single database in a single datacenter
Then Netflix grew, and grew, and grew More databases, all linked Database links, PL/SQL, materialized views Multi-Master Replication (MMR)
2
![Page 3: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/3.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
History, 2008
Then it melted down (Aug 2008) Hardware driver “one in a billion chance”
Couldn’t ship DVDs for ~5 days
3
![Page 4: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/4.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
History, 2009
Time to rethink everything Abandon our datacenter Ditch the monolithic webapp Migrate SPOF database
On-demand streaming was becoming the thing!
4
![Page 5: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/5.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
History, 2010
SimpleDB Managed by Amazon Got us started with NoSQL in the cloud Problems:
High latency, rate limiting (throttling) (no) auto-sharding, no backups We were running at 10x the intended capacity
5
![Page 6: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/6.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
History, 2011-now
Cassandra Similar to SImpleDB, but with limits removed Dynamo-style, master-less system Great multi-datacenter support Written in Java
6
![Page 7: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/7.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Cassandra in 3 minutes
Dynamo and Big Table papers CAP Theorem: AP Eventually consistent
7
![Page 8: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/8.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Distributed hash table Each node takes range on ring
Replication RF = how many copies to keep Within datacenter (shard across nodes) Across datacenters (full data set)
Peer-to-peer Gossip Failure detection
8
![Page 9: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/9.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Anti-Entropy protocols Hinted handoff Read repair Node repair
9
![Page 10: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/10.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Writes mmap’d commit log files Mutations buffered in memtable Flushed to immutable files (sstable)
Reads Check all sstables for key Bloom filter used for IO optimization
10
![Page 11: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/11.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Compactions Several styles, but basically reduce the
number of sstables by merging data Eliminate expired/deleted data I/O & CPU intensive
11
![Page 12: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/12.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Tunable Consistency 1, 2, Quorum (local, each), All
Columnar, Key-Value storage Schema-less Denormalization (wide rows)
Cassandra Query Language (CQL)
12
![Page 13: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/13.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Cassandra at Netflix
![Page 14: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/14.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
By the numbers…
Production clusters > 65
Production nodes > 2000
Multi-region clusters > 40
Most regions used 4 (three clusters)
Total data 300 TB Largest cluster size 144 nodes (three clusters) Max reads/writes 300k rps, 1.2m wps
14
![Page 15: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/15.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Example 1
Subscriber data Wide row implementation Row key = customer id Column for each attribute about a subscriber
id, name, subscription plan details, holds, …
15
![Page 16: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/16.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Example 2
Movie Ratings New ratings initially stored flat (per subscriber) Recurring job to aggregate into JSON blob Reads grab JSON blob + new writes
16
![Page 17: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/17.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Example 3
Edge Services scripts Versioned, executable Groovy scripts Inefficient query read _entire_ table Created inverted index to get IDs
Roll-your-own indices in Cassandra
17
![Page 18: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/18.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Life in the cloud
![Page 19: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/19.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Cloud life
AWS is our home Availability Zones (AZ) Regions
All Cassandra clusters run in multiple AZs within each region
Some clusters on multiple regions
19
![Page 20: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/20.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Resiliency
We use Cassandra replication factor = 3 / region Stripe data across AZs
20
ABC
BCD
CDE
DEF
EFA
FAB Availability
Zone 1
Availability Zone 2
Availability Zone 3
1 2
3 4
5
6
1 4
2 5
3 6
![Page 21: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/21.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Priam
Co-process that runs next to Cassandra on every node
Manages Backup/restore Cassandra bootstrap / token assignment Centralized configuration management
github.com/netflix/priam
21
![Page 22: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/22.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Priam – AWS Cluster Management
Assigns tokens to nodes at instance launch Stores token->node assignments in external
datastore Reference implementation uses SimpleDB Netflix uses another Cassandra cluster
Stripes neighbor node token in different availability zones
Supports multi-region Cassandra deployments
22
![Page 23: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/23.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Priam, Backup and Restore
Automate Cassandra snapshots Nightly backups
Copy all Cassandra artifacts to S3 Snappy compression Throttled, multi-part upload Data imported to our BI system (Aegisthus)
Restore (full or partial) Common for prod -> test refresh
23
![Page 24: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/24.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Secondary backup
Stand alone app for disaster recovery Copies all previous day’s artifacts: Secondary Amazon account Secondary cloud provider
Longer TTL than primary backup
24
![Page 25: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/25.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Java client libraries
Hector https://github.com/hector-client/hector
Astyanax Developed by Netflix https://github.com/netflix/astyanax
Cassandra Java Driver Developed by DataStax https://github.com/datastax/java-driver
25
![Page 26: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/26.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Astyanax
Clean object model Cassandra node discovery Node quarantine Request failover/retry JMX monitoring Connection pooling Futures execution for timing out long-running
queries
26
![Page 27: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/27.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Astyanax, Round Robin pool
Round Robin uses coordinator node to determine a node with data
27
ABC
BCD
CDE
DEF
EFA
FAB Service app astyanax
![Page 28: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/28.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Astyanax, Token-Aware pool
Token aware knows where the data resides Great for point reads
28
ABC
BCD
CDE
DEF
EFA
FAB AB service astyanax
![Page 29: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/29.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Challenges of running Cassandra in the cloud
![Page 30: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/30.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Building for/Deploying in AWS
Every application change is a new: Build artifact (war, tar, library) AMI (Amazon Machine Image) Auto Scale Group Instance Launch
No hand-modified configurations in production
30
![Page 31: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/31.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
… but Cassandra is different
We can’t drop/launch new instances endlessly We have state!
Updates are done in place on existing instances Bootstrapping new nodes not free
We turn over instances at a slow rate
31
![Page 32: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/32.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Misbehaving nodes
Node death happens Just ‘disappears’ Disk failure
EBS wonkiness Currently, root mount is EBS-mounted
32
![Page 33: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/33.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Networking
Network flapping causes Cassandra to see peers as UP, DOWN, UP, DOWN, UP ….
Causes Weird latency statistics request failure if partitioning is really bad Anti-entropy (hints) This is good – system as a whole did not fail
33
![Page 34: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/34.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
hi1.4xlarge
SSDs in the cloud!� Welp, not sooo fast Higher IOPs, better latencies Not quite like SSD on bare metal
34
![Page 35: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/35.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
BI integration
Didn’t want BI to run monster distributed queries on prod clusters
Instead, they: grab the nightly backups Export to JSON Import to Hadoop Find new data Import to Hive, Teradata, etc.
35
![Page 36: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/36.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Developer education
Data model re-think Denormalization Client query tuning is different Eventual consistency
36
![Page 37: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/37.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Takeaways
Netflix is making all of our components distributed and fault tolerant as we grow domestically and internationally.
Cassandra is a core piece of our cloud infrastructure.
Netflix is open sourcing it’s cloud platform, including Cassandra support (Astyanax, Priam, more to come).
37
![Page 38: Migrating to Cassandra in the Cloud, the Netflix Way to Cassandra in the Cloud, the Netflix Way Jason Brown - @jasobrown Senior Software Engineer, Netflix 2013 Storage Developer Conference](https://reader031.vdocuments.mx/reader031/viewer/2022022516/5b00a38f7f8b9a89598cf0b4/html5/thumbnails/38.jpg)
2013 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved.
Thank you!