[rakuten techconf2014] [b-1] performance at scale
DESCRIPTION
Rakuten Technology Conference 2014 "Performance at scale" Amir Ish-Shalom (Viber)TRANSCRIPT
Performance at ScaleOctober 2014
Amir Ish-ShalomSenior Solutions Architect
About Viber
The Viber service• Free, cross platform text messaging• Free, cross platform VoIP calls
(voice and video)• Photo, video and location sharing• Stickers and Emoticons• Group communication platform
(up to 100 participants)• Push To Talk
Monetization
Viber out Sticker market
Simplicity and User Experience• No registration needed
• User ID = your mobile number• Automatic friends detection
(no add a friend)
• Always on. No battery impact• 32 languages• Multiple devices experience:
Mobile, Tablets and Desktop
Viber in numbers
Viber in numbers• Over 400 million users• 1 million added daily• Billions of messages every day• Billions of talking minutes every month
Viber Growth
20
13
-12
20
13
-11
20
13
-10
20
13
-09
20
13
-08
20
13
-07
20
13
-06
20
13
-05
20
13
-04
20
13
-03
20
13
-02
20
13
-01
20
12
-12
20
12
-11
20
12
-10
20
12
-09
20
12
-08
20
12
-07
20
12
-06
20
12
-05
20
12
-04
20
12
-03
20
12
-02
20
12
-01
20
11
-12
20
11
-11
20
11
-10
20
11
-09
20
11
-08
20
11
-07
20
11
-06
20
11
-05
20
11
-04
20
11
-03
20
11
-02
In 2013 vs. 2012 there was:• Over 3x growth in talking minutes• Over 5x growth in messages• Over 12x growth in group messages
Viber DB Architecture
Viber DB Architecture – 1st Generation
In-housein-memory DB
Application ServersViberClients
Viber DB Architecture – 2nd Generation
Redis Cache MongoDB Cluster
Redis Cluster
Application ServersViberClients
Redis Sharder
Redis Sharder
2nd generation DB architecture advantages• Got us through the first few years of extreme growth• Never lost data from MongoDB• Redis performance
2nd generation DB architecture problems• MongoDB performance• MongoDB does not scale well with many application servers• Redis – In-memory database with no sharding• Redis Sharder – Not manageable and robust enough
3rd generation DB architecture requirements• High performance• Large data sets• Scalable• Robust• Backed-up• Always on• Easy to monitor• Prefer single DB solution
Solution:
Viber DB Architecture – 3rd Generation
Application ServersViberClients
CouchbaseBackupClusterCouchbase
Clusters
XDCR
Migrating from 2nd to 3rd generation DB’s• Migrate a live system• Zero downtime• No data loss• Consistent data
How did we migrate?• Stage 1: Add new CB cluster in parallel to existing cluster
Only delete keys from CB• Stage 2: Read only from MongoDB
Write/Delete to both CB & MongoDB• Stage 3: Background process that copies all data from
MongoDB to CB (if it doesn’t exist)• Stage 4: Validate data (both DB’s should be identical)• Stage 5: Read only from CB
Write/Delete from both CB & MongoDB• Stage 6: Remove MongoDB and use only CB
Couchbase Cluster with 60 nodes
Migration
Back-end servers• Over 500 application servers• 2nd generation DB architecture:
• MongoDB – 1 cluster with 150 servers (master + 2 slaves)• Redis – 3 clusters with a total of 144 servers (master + 1 slave)
• 3rd generation DB architecture:• 7 Couchbase clusters (up to 60 nodes each)• 1 – 2 replicas, XDCR & external backup• Total of less than 200 Couchbase servers
Increased performance using fewer DB servers!
Interesting facts1. We have recently doubled our Couchbase instance sizes to
cope with increased usage2. Total of ~1.5 million DB operations per second3. Bigger clusters don’t necessarily do more ops4. Highest performing clusters have 100% of their data in
memory
Couchbase Cluster with 12 nodes & XDCR
Questions?
Thank you