cassandra @formspring

Cassandra @Formspring

Tuesday, August 30, 2011

Cassandra @Formspring

Yet another Knight, Dragon and Princess Story.


Formspring helps people find out more about each other by sharing interesting & personal responses


25M users


3.5B responses


Follow[ing|ers]


Our situation before Cassandra


MySQL + Memcache


MySQL + Memcache

• Flat MySQL table


MySQL + Memcache


• Indexes (Followers vs Following)


MySQL + Memcache


• Indexes (Followers vs Following)

• Memcache


Memcache

• Stored as JSON list


Memcache

• Stored as JSON list

• Can get out of sync


MySQL

• Hundreds of millions of rows


MySQL

• Hundreds of millions of rows

• m2.4xl EC2 instance


It all started with a Feature.


QOTD


QOTD

• Feb: 35K Followers


QOTD


• Mid-Feb: 100K


QOTD


• Mid-Feb: 100K

• Growing about 30K a week.


QOTD


• Mid-Feb: 100K


• 200K: closer to Memcache 1MB limit.


QOTD


• Mid-Feb: 100K


• 200K: closer to Memcache 1MB limit.

• (with gz compaction . . .)


Then we added more features . . .


Temporary solution


Redis

• Key : JSON blob of IDs


Redis

• Key : JSON blob of IDs

• ~2-3s / insert


Frontend Ask

MySQLMemCache


OutboxQW

Frontend Ask

MySQLMemCache


OutboxQW

Frontend Ask

SELECT account_idFROM follow WHERE following = 1234 MySQL

MemCache


OutboxQW

Frontend Ask

MySQLMemCache

OH SHIT


OutboxQW

Frontend Ask

OH SHIT

MySQLMemCache


OutboxQW

Frontend Ask

Gazillion messages

OH SHIT

MySQLMemCache


OutboxQW

Frontend Ask

Gazillion messagesInboxQWInboxQWInboxQWInboxQWInboxQWInboxQWInboxQWInboxQWInboxQWInboxQWInboxQWInboxQWInboxQWInboxQWInboxQWInboxQWInboxQWs

OH SHIT

MySQLMemCache


OutboxQW

Frontend Ask

Gazillion messagesInboxQWInboxQWInboxQWInboxQWInboxQWInboxQWInboxQWInboxQWInboxQWInboxQW

Inbox Object

InboxQWInboxQWInboxQWInboxQWInboxQWInboxQWInboxQWs

OH SHIT

MySQLMemCache


OutboxQW

Frontend Ask


Inbox Object


*HANGING*OH SHIT

MySQLMemCache


OutboxQW

Frontend Ask


Inbox Object


*OOM*OH SHIT

MySQLMemCache


OutboxQW

Frontend Ask


Inbox Object


OutboxQW

OutboxQW

OutboxQW

OutboxQW

OutboxQW

OutboxQW

OH SHIT

MySQLMemCache


OutboxQW

Frontend Ask


Inbox Object


OutboxQW

OutboxQW

OutboxQW

OutboxQW

OutboxQW

OH SHIT

MySQLMemCacheOOM


We need a solution


Fast


Step 1: Research


Our first shot with Cassandra


All Responses


Explain our stack here


Index of responses to questions on the site.


Explain our stack here


Ask Followers


How to scale ?


Build a Social Graph


Social Graph



• Python



• Python

• Cassandra (Obviously..)



• Python


• PyCassa



• Python


• PyCassa

• Thrift


4 types of actions

• Following


4 types of actions

• Following

• Followers


4 types of actions

• Following

• Followers

• Blocks


4 types of actions

• Following

• Followers

• Blocks

• Blockers


Thrift


Cassandra


Python + Pycassa


Python + Pycassa

Column Families


Python + Pycassa

Get list of Follow[ers/ing]


Python + Pycassa

More Queries


Python + Pycassa

Follow user


Switch to Production


Simultaneous writes to MySQL and Cassandra


Cassandra MySQL

Read x

Write x x

Switch to Production


Idempotence is your friend.


Migration from MySQL to Cassandra


Load batches of user info into Cassandra.


It’s OK if it fails.


Remember idempotence is your

friend.


Adjust your capacity


From 4 instancesto

a 12 instances


New Ask Followers Infrastructure


Frontend Ask

CassandraCluster


OutboxQW

Frontend Ask

CassandraCluster


OutboxQW

Frontend Ask

CassandraCluster

IterativeKW


OutboxQW

Frontend Ask

CassandraCluster

IterativeKW

get_followers(user_id, 0, N)


OutboxQW

Frontend Ask

CassandraCluster

IterativeKW

[‘1234, ‘1238’, ..., ‘1999’]


OutboxQW

Frontend Ask

CassandraCluster

IterativeKW

[‘1234, ‘1238’, ..., ‘1999’]

N Messages


OutboxQW

Frontend Ask

CassandraCluster

IterativeKW

[‘1234, ‘1238’, ..., ‘1999’]

InboxQW

N Messages


OutboxQW

Frontend Ask

CassandraCluster

IterativeKW

[‘1234, ‘1238’, ..., ‘1999’]

InboxQW

N Inbox Object


OutboxQW

Frontend Ask

CassandraCluster

IterativeKW

InboxQW

N Inbox Object

get_followers(user_id, N, N+N)


Count Following

14K/min hits at peak time


Count Followers

40K/min hits at peak time


Reduce our #instances

InboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWs

InboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsOutboxQWs

12x 12x


InboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWs

12x

InboxQWsInboxQWsInboxQWsInboxQWsOutboxQWs

4x



InboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWsInboxQWs

Inbox+

Iterator QW

InboxQWsInboxQWsInboxQWsInboxQWsOutboxQWs

12x 4x



Counts for free with 0.8.1


Counting . . .


Counting is hard

Let’s go Shopping.


Counting

• Fast


Counting

• Fast

• Not-accurate


Counting

• Fast

• Not-accurate

• Faking “Read-repair”


Counting


Ghetto Count


Cassandra allowed us:

• To create new features.




• To scale.




• To scale.

• To save money.




• To scale.

• To save money.

• To sleep. (Almost there)




• To scale.

• To save money.

• To sleep. (Almost there)

• Lots of fun :)


Come work with us !


cassandra @formspring

Technology

mem mysql cachetuesday

mysql hundreds of millions

followers mid

mem qwmysqlcachetuesday

json listtuesday

json blob of idstuesday

redis key

mb limit