cassandra summit 2014: cassandra at instagram 2014

69
CASSANDRA @ INSTAGRAM 2014 Rick Branson, Software Engineer, Backend Engineering @rbranson Cassandra Summit September 11th, 2014

Upload: planet-cassandra

Post on 05-Dec-2014

2.112 views

Category:

Technology


3 download

DESCRIPTION

Presenter: Rick Branson, Infrastructure Engineer at Instagram As Instagram has scaled to over 200 million users, so has our use of Cassandra. We've built new features and rebuilt old on Cassandra, and it's become an extremely mission-critical foundation of our production infrastructure. Rick will deliver a refresh of our use cases and go deep on the technical challenges we faced during our expansion.

TRANSCRIPT

Page 1: Cassandra Summit 2014: Cassandra at Instagram 2014

CASSANDRA @ INSTAGRAM 2014Rick Branson, Software Engineer, Backend Engineering

@rbranson Cassandra Summit

September 11th, 2014

Page 2: Cassandra Summit 2014: Cassandra at Instagram 2014

2013 in a Nutshell

• Moving some log-style data from Redis to Cassandra

• Saving $, better availability, more elasticity

• Some problems, but we worked through them

Page 3: Cassandra Summit 2014: Cassandra at Instagram 2014

Overview

• "The Trouble with Redis"

• A whole new category of functionality

• Useful consumer advice

Page 4: Cassandra Summit 2014: Cassandra at Instagram 2014

"THE TROUBLE WITH REDIS"

Page 5: Cassandra Summit 2014: Cassandra at Instagram 2014

REDIS: THE GÜD PARTS

Page 6: Cassandra Summit 2014: Cassandra at Instagram 2014

The Network PartIt's a heap on the network with a great text-based protocol.

Page 7: Cassandra Summit 2014: Cassandra at Instagram 2014

Extremely EfficientIt does a lot with every ounce of CPU

Page 8: Cassandra Summit 2014: Cassandra at Instagram 2014

StableVery much so. Sets the bar for the vast NoSQL space.

Page 9: Cassandra Summit 2014: Cassandra at Instagram 2014

Useful, Rich TypesYou won't find anywhere else.

Page 10: Cassandra Summit 2014: Cassandra at Instagram 2014

REDIS: ZDARKSIDE

Page 11: Cassandra Summit 2014: Cassandra at Instagram 2014

What's in there? Hello?

Page 12: Cassandra Summit 2014: Cassandra at Instagram 2014

Snapshots (BGSAVE)

Page 13: Cassandra Summit 2014: Cassandra at Instagram 2014

Single-Threaded (!)

Page 14: Cassandra Summit 2014: Cassandra at Instagram 2014

Memory Allocator

Page 15: Cassandra Summit 2014: Cassandra at Instagram 2014

(Lack of) Durability

Page 16: Cassandra Summit 2014: Cassandra at Instagram 2014

Unsafe Replication

Page 17: Cassandra Summit 2014: Cassandra at Instagram 2014

THE PHILOSOPHY

Page 18: Cassandra Summit 2014: Cassandra at Instagram 2014

CACHE OR DATABASE? Neither.

Page 19: Cassandra Summit 2014: Cassandra at Instagram 2014

Memory Cliff

Page 20: Cassandra Summit 2014: Cassandra at Instagram 2014
Page 21: Cassandra Summit 2014: Cassandra at Instagram 2014

Unlikely to Change :(

Page 22: Cassandra Summit 2014: Cassandra at Instagram 2014
Page 23: Cassandra Summit 2014: Cassandra at Instagram 2014

ZOOKEEPER???

Page 24: Cassandra Summit 2014: Cassandra at Instagram 2014

ZOZO: *VERY* few writes, extremely high volume, no-

latency reads without hotspots.

Page 25: Cassandra Summit 2014: Cassandra at Instagram 2014

zozo agent

/var/cache/zozo

Consumer Process

Consumer Process

Consumer Process

WATCH /zozo/files

WriterUPDATE /zozo/files/knobs

ZooKeeper Ensemble

Page 26: Cassandra Summit 2014: Cassandra at Instagram 2014

Examples

• "Knobs" key/integer live configuration

• Small (<250K) User Lists

• Security Data Sets (i.e. Blocked IPs)

Page 27: Cassandra Summit 2014: Cassandra at Instagram 2014

JSON sucks.

Page 28: Cassandra Summit 2014: Cassandra at Instagram 2014

Stay tuned.

Page 29: Cassandra Summit 2014: Cassandra at Instagram 2014

zozo postmortem

• Agent is dead simple, never needs updating

• ZooKeeper resists both machine & human errors

• Multi-region strategy coming

Page 30: Cassandra Summit 2014: Cassandra at Instagram 2014

REDIS2CASSANDRA (omg, finally)

Page 31: Cassandra Summit 2014: Cassandra at Instagram 2014

Some things we moved

• Ad Serving Metadata

• Account Suggestion Metadata

• Churn Tracking

• Follow Requests

Page 32: Cassandra Summit 2014: Cassandra at Instagram 2014

Not a drop-in.

Page 33: Cassandra Summit 2014: Cassandra at Instagram 2014

A B

source target

"User A has requested to follow User B"

Key Column ColumnA B ...

Key Column Column

B A ...

Outbound Requests

Inbound Requests

Page 34: Cassandra Summit 2014: Cassandra at Instagram 2014

INSTAGRAM DIRECT

Page 35: Cassandra Summit 2014: Cassandra at Instagram 2014

Y U CASSANDRA?

• Elasticity + Scalability

• Write Availability

• Storage Efficiency

• Good Data Model Fit

Page 36: Cassandra Summit 2014: Cassandra at Instagram 2014

WE CHOSE THRIFT.

Page 37: Cassandra Summit 2014: Cassandra at Instagram 2014

fan-out-on-write

• Caching doesn't work

• Copy of full data for each recipient

• Atomic batches super useful

Page 38: Cassandra Summit 2014: Cassandra at Instagram 2014

#nocache

Page 39: Cassandra Summit 2014: Cassandra at Instagram 2014
Page 40: Cassandra Summit 2014: Cassandra at Instagram 2014

CompositeType(IntegerType(reversed=true), IntegerType, IntegerType)

postid

typecode

entryid

Page 41: Cassandra Summit 2014: Cassandra at Instagram 2014

POST_HEADER => creator_id: 12938 media_type: 1 asset_path: "/objects/23904801..." recipients: [28910, 208192]

(SEEN, 28910) (SEEN, 208192) (LIKE, 208192)

(COMMENT, 9302...) => [19238, "George wishes he ha..."]

(COMMENT, 3499...) => [28910, "Weird"]

Page 42: Cassandra Summit 2014: Cassandra at Instagram 2014

IMMUTABLE (mostly) LOG

Page 43: Cassandra Summit 2014: Cassandra at Instagram 2014

FILTER-ON-READ

Page 44: Cassandra Summit 2014: Cassandra at Instagram 2014

CQL3, please. no? ok. :(

Page 45: Cassandra Summit 2014: Cassandra at Instagram 2014

THE PENDING QUEUE

Page 46: Cassandra Summit 2014: Cassandra at Instagram 2014
Page 47: Cassandra Summit 2014: Cassandra at Instagram 2014
Page 48: Cassandra Summit 2014: Cassandra at Instagram 2014

Key Column ColumnB [A's PostID] ...

C [A's PostID] ...

D [A's PostID] ...

A C

sender

recipients

D

B

DFirst read, delete pending

Page 49: Cassandra Summit 2014: Cassandra at Instagram 2014

2000000 (POINTER, 1000000)1500000 (POST_HEADER, 0) => ...

(SEEN, 902394)(SEEN, 94083)(LIKE, 902394)

1000000 (POST_HEADER, 0) => ...(BUMPED, 0) => 2000000(SEEN, 1298333)(SEEN, 848183)...

Page 50: Cassandra Summit 2014: Cassandra at Instagram 2014
Page 51: Cassandra Summit 2014: Cassandra at Instagram 2014

Justin's Pending

25000

24000

23000

22000

21000

20000

19000

...

1234

Justin's Inbox

...

(1234, POST..)

(1234, LIKE..)

(1234, SEEN..)

(1234, SEEN..)

(1010, POST..)

(1010, SEEN..)

(1010, COMM..)

...

Inbox

...

(1234, POST..)

(1234, LIKE..)

(1234, SEEN..)

(1234, SEEN..)

(1010, POST..)

(1010, SEEN..)

(1010, COMM..)

...

Inbox

...

(1234, POST..)

(1234, LIKE..)

(1234, SEEN..)

(1234, SEEN..)

(1010, POST..)

(1010, SEEN..)

(1010, COMM..)

...

Inbox

...

(1234, POST..)

(1234, LIKE..)

(1234, SEEN..)

(1234, SEEN..)

(1010, POST..)

(1010, SEEN..)

(1010, COMM..)

...

(1234, EXCLUDE, <JUSTIN>)

Page 52: Cassandra Summit 2014: Cassandra at Instagram 2014
Page 53: Cassandra Summit 2014: Cassandra at Instagram 2014

Rick's Unseen

849300

738939

649019

641192

629012

610392

483201

348931

Rick's Unseen

849300

738939

649019

641192

629012

610392

483201

348931

Rick's Unseen

849300

738939

649019

641192

629012

610392

483201

348931

8

Page 54: Cassandra Summit 2014: Cassandra at Instagram 2014

DEPLOYMENT

• 60 x hi1.4xlarge EC2 instances

• 2X our highest estimates

• Shrunk cluster after we knew load

• Peace of mind

• Data models held together

Page 55: Cassandra Summit 2014: Cassandra at Instagram 2014

We ran C* in production.We learned some things.

Page 56: Cassandra Summit 2014: Cassandra at Instagram 2014

young gc fun!

Page 57: Cassandra Summit 2014: Cassandra at Instagram 2014
Page 58: Cassandra Summit 2014: Cassandra at Instagram 2014

Yung GC has problems

• Reads = young gen garbage

• Double-collection bug in JDK 1.7+

• Best practices were for nodes with more writes

Page 59: Cassandra Summit 2014: Cassandra at Instagram 2014

now we run: 10G Young Gen 20G Heap

<blink>DEAR GOD

ALMIGHTY!!!</blink>

Page 60: Cassandra Summit 2014: Cassandra at Instagram 2014

This won't work out of the box.

Page 61: Cassandra Summit 2014: Cassandra at Instagram 2014

-XX:+CMSScavengeBeforeRemark

Page 62: Cassandra Summit 2014: Cassandra at Instagram 2014

-XX:CMSMaxAbortablePrecleanTime=60000

Page 63: Cassandra Summit 2014: Cassandra at Instagram 2014

-XX:CMSWaitDuration=30000

Page 64: Cassandra Summit 2014: Cassandra at Instagram 2014

-XX:+CMSEdenChunksRecordAlways -XX:+CMSParallelInitialMarkEnabled

Page 65: Cassandra Summit 2014: Cassandra at Instagram 2014

20-core Ivy Bridge 144GB RAM PCIe Flash

15,000 reads/sec 2,000 writes/sec

Page 66: Cassandra Summit 2014: Cassandra at Instagram 2014
Page 67: Cassandra Summit 2014: Cassandra at Instagram 2014

TWO ISSUES

• Counters. Don't use them.

• Hundreds of thousands / millions of tombstones.

Page 68: Cassandra Summit 2014: Cassandra at Instagram 2014

Async Storage Port Handler

READ 12349081

Client

RPC Thread Pool

Queue

ReadStage Thread Pool

Queue

GET 12349081

Page 69: Cassandra Summit 2014: Cassandra at Instagram 2014

THANK YOU.