CMB

An Open Source Alternative for Queuing and Notification

Cloud Services

Ryan KingComcast Silicon Valley

OpenStack Summit – San Diego 2012

Intro

SummaryA team inside Comcast open sourced a compatible version of Amazon SNS and SQS on top of Cassandra & Redis

Who are we?• Comcast Silicon Valley

– Consumer-facing internet products for Comcast

– Web, Mobile, TV

• Silicon Valley

– Product Development & Innovation Center

CMB

CMB – Message Bus

CNS – Notification Service

CQS – Queue Service

But, wait.. Why build your own???

Comcast building Private Cloud

(… using OpenStack!)

To power next-gen services

(… including TV!)

Latency extremely important

Every millisecond counts

Needs to be cost effective

(… for 50MM users)

What else is out there?

None of those could meet our requirements

Requirements• Compatible with AWS

• Support Active-Active Multi-Data-Center

• Horizontal Scalability

• Guaranteed Delivery

• Very Low Latency

NOT Required1. Order is NOT guaranteed

2. Duplicate messages CAN happen

2. Duplicate messages CAN happen

First, the Queue

CMB – Message Bus

CNS – Notification Service

CQS – Queue Service

What is Simple Queue Service?

(from a Popular Cloud Service Provider)

“Simple Queue Service offers a reliable, highly scalable, hosted queue for storing messages as they travel between computers”

Simple Queue Service

Main SQS Methods

• CreateQueue

• SendMessage

• ReceiveMessage

• DeleteMessage

Our implementation:

CQS

CQS APICQS API

Lists

Redis

Hashtables

Cassandra

Payload cache

API

Cassandra Persistence• Queue sharded over N=100 rows

–Avoids wide rows (> 500K items)

–Reduces “churn” (Tombstones)

–Distributes queue among nodes

CQS APICQS API

API

Lists

Redis

Hashtables

Payload cache

Redis Caching

• Message IDs: List

• Visibility: Hashtable

• Payload cache: Set

CQS Notes• Minimal duplication and good ordering

– When things are happy..

• Cassandra tombstones degrade message performance

– Normally reads => Redis

CQS Performance

0

10

20

30

40

50

60

70

80

90

10 50 75 100 250 500 1000

Late

ncy

(m

s)

Msg/Sec

Why use Cassandra?• very low latency

• very high write throughput

• extremely high availability (across multi-dc)

• Near linear horizontal scalability

Why add Redis?• Queue metadata:

– changes frequently

– not durable

• Needed extremely low latency reads

• Reduces “churn” in Cassandra

Next, Notifications

CMB – Message Bus

CNS – Notification Service

CQS – Queue Service

What is Simple Notification Service?

(from a Popular Cloud Services Provider … in Seattle… )

“Simple Notification Service is a web service that makes it easy to set up, operate, and send notifications from the cloud”

Simple Notification Service

Main SNS Methods

•CreateTopic

•Subscribe

•Publish

Our Implementation:

CNS

Scalability• Publish Requests

=> API servers

• # subscribers => Delivery Producers

• Publishing to endpoints => Delivery Consumers

• Overall => CQS & Cassandra

0

200

400

600

800

1000

1200

500 1000 2000 4000 8000

Late

ncy

(m

s)

Throughput (Msgs/Sec)

Throughput Scalability

6 workers

3 workers

0

500

1000

1500

2000

2500

3000

100 250 500 750 1000 1250 1500

Late

ncy

(m

s)

# of Subscribers per Topic

Subscriber Scalability

3 workers

6 workers

What are we using this for?

Now Open Source!github.com/Comcast/cmb

Next steps• Does this fit into OpenStack?• Get Feedback from the community (you!)

• Questions?

• ryan@sv.comcast.com• @ryanking• PS - We’re hiring @ Comcast SV!