CMB
An Open Source Alternative for Queuing and Notification
Cloud Services
Ryan KingComcast Silicon Valley
OpenStack Summit – San Diego 2012
Intro
SummaryA team inside Comcast open sourced a compatible version of Amazon SNS and SQS on top of Cassandra & Redis
Who are we?• Comcast Silicon Valley
– Consumer-facing internet products for Comcast
– Web, Mobile, TV
• Silicon Valley
– Product Development & Innovation Center
CMB
CMB – Message Bus
CNS – Notification Service
CQS – Queue Service
But, wait.. Why build your own???
Comcast building Private Cloud
(… using OpenStack!)
To power next-gen services
(… including TV!)
Latency extremely important
Every millisecond counts
Needs to be cost effective
(… for 50MM users)
What else is out there?
None of those could meet our requirements
Requirements• Compatible with AWS
• Support Active-Active Multi-Data-Center
• Horizontal Scalability
• Guaranteed Delivery
• Very Low Latency
NOT Required1. Order is NOT guaranteed
2. Duplicate messages CAN happen
2. Duplicate messages CAN happen
First, the Queue
CMB – Message Bus
CNS – Notification Service
CQS – Queue Service
What is Simple Queue Service?
(from a Popular Cloud Service Provider)
“Simple Queue Service offers a reliable, highly scalable, hosted queue for storing messages as they travel between computers”
Simple Queue Service
Main SQS Methods
• CreateQueue
• SendMessage
• ReceiveMessage
• DeleteMessage
Our implementation:
CQS
CQS APICQS API
Lists
Redis
Hashtables
Cassandra
Payload cache
API
Cassandra Persistence• Queue sharded over N=100 rows
–Avoids wide rows (> 500K items)
–Reduces “churn” (Tombstones)
–Distributes queue among nodes
CQS APICQS API
API
Lists
Redis
Hashtables
Payload cache
Redis Caching
• Message IDs: List
• Visibility: Hashtable
• Payload cache: Set
CQS Notes• Minimal duplication and good ordering
– When things are happy..
• Cassandra tombstones degrade message performance
– Normally reads => Redis
CQS Performance
0
10
20
30
40
50
60
70
80
90
10 50 75 100 250 500 1000
Late
ncy
(m
s)
Msg/Sec
Why use Cassandra?• very low latency
• very high write throughput
• extremely high availability (across multi-dc)
• Near linear horizontal scalability
Why add Redis?• Queue metadata:
– changes frequently
– not durable
• Needed extremely low latency reads
• Reduces “churn” in Cassandra
Next, Notifications
CMB – Message Bus
CNS – Notification Service
CQS – Queue Service
What is Simple Notification Service?
(from a Popular Cloud Services Provider … in Seattle… )
“Simple Notification Service is a web service that makes it easy to set up, operate, and send notifications from the cloud”
Simple Notification Service
Main SNS Methods
•CreateTopic
•Subscribe
•Publish
Our Implementation:
CNS
Scalability• Publish Requests
=> API servers
• # subscribers => Delivery Producers
• Publishing to endpoints => Delivery Consumers
• Overall => CQS & Cassandra
0
200
400
600
800
1000
1200
500 1000 2000 4000 8000
Late
ncy
(m
s)
Throughput (Msgs/Sec)
Throughput Scalability
6 workers
3 workers
0
500
1000
1500
2000
2500
3000
100 250 500 750 1000 1250 1500
Late
ncy
(m
s)
# of Subscribers per Topic
Subscriber Scalability
3 workers
6 workers
What are we using this for?
Now Open Source!github.com/Comcast/cmb
Next steps• Does this fit into OpenStack?• Get Feedback from the community (you!)
• Questions?
• [email protected]• @ryanking• PS - We’re hiring @ Comcast SV!
