Outside The Box With Apache Cassandra

Eric Evanseevans@rackspace.com

@jericevans

Palemetto Open Source Software ConferenceApril 16, 2010

Cassandra is...

A massively scalable, decentralized, structured data store (akadatabase).

Outline

1 Background

2 Project History

3 Description

4 Case Studies

5 Roadmap

The Digital Universe

Consolidation

Old Guard

Vertical Scaling Sucks

CAP Theorem (aka Brewer’s Theorem)

Distributed systems cannot provide all three of:

• Consistency

• Availability

• Partition Tolerance

Influential Papers

Dynamo: Amazon’s Highly Available Key-value Store 1

• Voldemort

• Riak

Bigtable: A Distributed Storage System for Structured Data 2

• Hypertable

• HBase

1http:

//www.allthingsdistributed.com/2007/10/amazons_dynamo.html2http://labs.google.com/papers/bigtable-osdi06.pdf

Outline

1 Background

2 Project History

3 Description

4 Case Studies

5 Roadmap

• 7 new committers added

• Dozens of contributors

• 200+ (!) people on IRC

• Hundreds of closed issues (bugs, features, etc)

• 4 major releases; a number of stable point releases

• Graduation to TLP

Outline

1 Background

2 Project History

3 Description

4 Case Studies

5 Roadmap

Cassandra is...

• O(1) DHT

• Eventual consistency

• Tunable trade-offs, consistency vs. availability

But...

• Values are structured, indexed

• Columns / column families

• Slicing w/ predicates (queries)

Column families

Supercolumn families

Client API

• Thrift (12 different languages!)3

• High-level client libraries• Ruby• Perl• Python (Twisted too)• Scala• Java• PHP• Grails• C++

3http://incubator.apache.org/thrift

Querying

• get(): retrieve by column name

• multiget(): by column name for a set of keys

• get slice(): by column name, or a range of names• returning columns• returning super columns

• multiget slice(): a subset of columns for a set of keys

• get count: number of columns or sub-columns

• get range slice(): subset of columns for a range of keys

Updating

• insert(): add/update column (by key)

• batch insert(): add/update multiple columns (by key)

• remove(): remove a column

• batch mutate(): like batch insert() but can also delete(new for 0.6, deprecates batch insert())

Column comparators

• TimeUUID

• LexicalUUID

• UTF8

• Long

• Bytes

• ...

Consistency

CAP Theorem: choose any two of Consistency, Availability, orPartition tolerance.

• Zero

• One

• Quorum ((N / 2) + 1)

• All

About writes...

• Atomic within a column family

• Any node

• Always writeable (hinted hand-off)

• Fast

Writes

About reads...

• Any node

• Read repair

• Key cache

• Record cache

Reads

Outline

1 Background

2 Project History

3 Description

4 Case Studies

5 Roadmap

Case 1: Digg

Digg is a social news site that allows people to discover and sharecontent from anywhere on the Internet by submitting stories andlinks, and voting and commenting on submitted stories and links.

Ranked 98th by Alexa.com.

Digg

Problem

• Terabytes of data; high transaction rate (reads dominated)

• Multiple clusters; heavily sharded

• Management nightmare (high effort, error prone)

• Unsatisfied availability requirements (geographic isolation)

Solution

• Currently production on ”Green Badges”

• Cassandra as primary data store RSN

• Datacenter and rack-aware replication

Case 2: Twitter

Twitter is a social networking and microblogging service thatenables its users to send and read tweets, text-based posts of up to140 characters.

Ranked 12th by Alexa.com.

Twitter

MySQL

• Terabytes of data, ˜1,000,000 ops/s

• Calls for heavy sharding, light replication

• Schema changes are very difficult, (if possible at all)

• Manual sharding is very high effort

• Automated sharding and replication is Hard

Case 3: Facebook

Facebook is a social networking site where users can create aprofile, add friends, and send them messages. Users can also joingroups organized by location or other points of common interest.

Ranked #2 by Alexa.com.

Inbox Search

• 100 TB

• 160 nodes

• 1/2 billion writes per day (2yr old number?)

Outline

1 Background

2 Project History

3 Description

4 Case Studies

5 Roadmap

0.6

• batch mutate command

• authentication (basic)

• new consistency level, ANY

• fat client

• mmapped i/o reads (default on 64bit jvm)

• improved write concurrency (HH)

• networking optimizations

• row caching

• improved management tools

• per-keyspace replication factor

0.7

• more efficient compactions (row sizes bigger than memory)

• easier (dynamic?) column family changes

• SSTable versioning

• SSTable compression

• support for column family truncation

• improved configuration handling

• remove key range command

• even more improved management tools

• vector clocks w/ server-side conflict resolution

Questions?