libon cassandra summiteu2014

Billion Records from SQL to Cassandra, lessons learned DuyHai Doan Brice Dutheil

#CassandraSummit @doanduyhai @BriceDutheil

Who are we ?

Brice Dutheil

Mockito Java Track Lead @ Devoxx France Independant contractor @ Libon (Orange-Vallée)

DuyHai Doan

Achilles Cassandra Technical Advocate Former Java Developer @ Libon

2


Agenda •  Libon context

•  Migration strategy

•  Business code migration

•  Data Modeling

•  Take Away

3


Libon Context


What is Libon ? •  Messaging app

•  VOIP (out)

•  Custom voicemail & greetings

•  SMS/chat/file transfer

•  Contacts matching

5


Contact Matching

6

Libon User


Contact Matching

7

Libon User Friend


Contact Matching

8

Libon User Friend

Contact matching


Contact Matching

9

Libon User Friend

Accept link


Project Context •  Application grew over the years

10


Project Context •  Application grew over the years

•  Already using Cassandra to handle events

•  messaging / file sharing / SMS / notifications

•  Cassandra R/W latencies ≈ 0,4 ms

•  server response time under 10 ms

11


Project Context •  About contacts …

12



•  stored as relational model in RDBMS (Oracle)

13




•  1 user ≈ 300 contacts

14





•  with millions users ☞ billions of contacts to handle

15





•  with millions users ☞ billions of contacts to handle

•  query latency unpredictable

16

#CassandraSummit @doanduyhai @BriceDutheil 17


Fixing the problem •  Tune the RDBMS

18



•  indices

19



•  indices

•  partitioning

20



•  indices

•  partitioning

•  less joins, simplified relational model

21



•  indices

•  partitioning


•  hardware capacity increased

22



•  indices

•  partitioning



That worked

23



•  indices

•  partitioning



That worked but …

24


Back-end application

RDBMS Cassandra

25


Next Challenges •  High Availability (DB failure, site failure …)

26



•  Predictable performance at scale

27



•  Predictable performance at scale

•  Going to multi data-centers

28


Going for Cassandra •  Denormalize (if possible …)

29



•  Know your business ☞ know your queries

30




•  Linear scaling out

31




•  Linear scaling out

•  Consistent performance

32


Data Migration Strategy


Objectives •  No downtime

34



•  No concurrency corner-cases

35




•  Safe rollback possible

36




•  Safe rollback possible

•  Replay-ability & resume-ability

37


Strategy •  3 phases

38



•  Write contacts to both data stores

39




•  Old contacts migration

40




•  Old contacts migration

•  Switch to Cassandra …

•  … and deprecate SQL

41


Migration Phase 1 Back end server

· ·

·

SQL SQL SQL

C*

C*

C* C*

C*

Write

contactUUID

42

contactId … contactUUID 129363 123e4567-

e89b-12d3… 834849

contacId(long) + contactUUID



· ·

·

SQL SQL SQL

C*

C*

C* C*

C*

Read

43


Migration Phase 2

SQL SQL SQL

C*

C*

C* C*

C*

For each batch of users SELECT * FROM contacts WHERE user_id = … AND contact_uuid IS NULL

•  On live production, migrate old contacts

44

Old contacts created before phase 1


Migration Phase 2

SQL SQL SQL

C*

C*

C* C*

C*

For each batch of users SELECT * FROM contacts WHERE user_id = … AND contact_uuid IS NULL

Logged batches of INSERT INTO contacts(..) VALUES(…) USING TIMESTAMP now() - 1 week

•  On live production, migrate old contacts

45

Old contacts created before phase 1


Migration Phase 2

USING TIMESTAMP now() - 1 week 😳

46


Migration Phase 2 •  During data migration …

47



•  … concurrent writes from the migration batch …

48



•  … concurrent writes from the migration batch …

•  … and updates from production for the same contact

49


Migration Phase 2

contact_uuid name (now -1 week) … name (now) …

Johny … Johnny …

Insert from batch (to the past)

Update from production

50


Migration Phase 2

contact_uuid name (now -1 week) … name (now) …

Johny … Johnny …

Future reads pick the most up-to-date value

51


Migration Phase 2

"Write to the Past… to save the Future"

Libon – 2014/10/08

52



· ·

·

SQL SQL SQL

C*

C*

C* C*

C*

Write

❌ 53


Business Code Refactoring


Code Inventory •  Written for RDBMS

55



•  Lots of joins (no surprise)

56




•  Designed around transactions

57




•  Designed around transactions

•  Spring @Transactional everywhere

58


Code Inventory cont. •  Entities go through Services & Repositories

59

Repositories

Services

ContactEntity


Code Inventory cont. •  Hibernate is auto-magic

60


Code Inventory cont. •  Hibernate is auto-magic

•  lazy loading

•  1st level cache

•  N+1 select

61

Repositories

Services

ContactEntity


Which options ? •  Throw existing code …

•  … and re-design from scratch for Cassandra

62


Which options ? •  Throw existing code …

•  … and re-design from scratch for Cassandra No way !

63


Code Quality •  Existing business code has…

•  … ≈ 3500 unit tests

64


Code Quality •  Existing business code has…

•  … ≈ 3500 unit tests

•  and ≈600+ integration tests

65


Code Quality •  We are TDD aficionados …

66


Code Quality •  We are TDD aficionados …

•  … and we love our code coverage

67


Code Quality

"The code coverage is one of your most

valuable technical asset" Libon – since beginning

68


Repositories

Services

Refactoring Strategy

ContactMatchingService ContactService ContactSync

ContactEntity

n 1 n n

69


Repositories

Services


ContactMatchingService ContactService

ContactNoSQLEntity

ContactSync

ContactEntity

n 1 n n

70

Proxy


Repositories

Services


ContactMatchingService ContactService

ContactNoSQLEntity

ContactSync

ContactEntity

n 1 n n

Denorm2 … DenormN Denorm1

71

Proxy


Refactoring Strategy •  Use CQRS

•  ContactReadRepository

•  ContactWriteRepository

•  ContactUpdateRepository

•  ContactDeleteRepository

72


Refactoring Strategy •  ContactReadRepository

•  direct sequential read

•  no joins

•  1 read ≈ 1 SELECT

73


Refactoring Strategy •  ContactWriteRepository

•  write to all denormalized tables

•  using CQL logged batches

•  use TTLs

74


Refactoring Strategy •  ContactUpdateRepository

•  read-before-write most of the time 😟

•  rare updates ☞ acceptable perf penalty

75


Refactoring Strategy •  ContactDeleteRepository

•  delete

•  update contact modification date

76


Outcome •  5 months of 2 men work

77



•  Many iterations to fix bugs (thanks to IT)

78




•  Lots of performance benchmarks using Gatling

79


Gatling Output

80





☞ data model & code validation

81





☞ data model & code validation

•  … we are almost there for production

82


Data Model


Denormalization, the good •  Support fast reads

•  1 read ≈ 1 SELECT

•  Worthy because mostly read, few updates

84


Denormalization, the bad •  Updating mutable data can be nightmare

•  Data model bound by existing client-facing API

•  Update paths very error-prone without tests

85


Data model in detail

Contacts_by_id

Contacts_by_identifiers

Contacts_in_profiles

Contacts_by_modification_date

Contacts_by_firstname_lastname

Contacts_linked_user

86



Contacts_by_id






87

user_id always component

of partition key


Scalable design

88

n1

n2

n3

n4

n5

n6

n7

n8

A

B

C

D

E

F

G

H

user_id1

user_id2

user_id3

user_id4

user_id5


Scalable design

89

n1

n2

n3

n4

n5

n6

n7

n8

A

B

C

D

E

F

G

H

user_id1 user_id2

user_id3

user_id4

user_id5


Bloom filters in action

90

•  For some tables, partition key = (user_id, contact_id)

☞ fast look-up, leverages Bloom filters

☞ touches 1 SSTable most of the time



Contacts_by_id






91

Wide partition Bucketed


A "queue" story

92

•  contacts_by_modification_date

•  queue-like pattern 😭


A "queue" story

93

•  contacts_by_modification_date

•  queue-like pattern 😭

☞ buckets to the rescue

user_id:2014-12 date35 date12 … … date47

… … … …

user_id:2014-11 date11 date12 … … date34

… … … …


Data model summary •  7 tables for denormalization

94



•  Normalize some tables because rare access

95



•  Normalize some tables because rare access

•  Read-before write in most update scenarios 😟

96


Notes on contact_id •  In SQL, auto-generated long using sequence

•  In Cassandra, auto-generated timeuuid

97


Notes on contact_id •  How to store both types ?

98



•  As text ? ☞ easy solution …

99



•  As text ? ☞ easy solution …

•  … but waste of space !

•  because encoded as UTF-8 or ASCII in Cassandra

100


Notes on contact_id •  Long ☞ 8 bytes

•  Long as text(UTF-8: 1 byte) ☞ "digits count" bytes

101


Notes on contact_id •  UUID ☞ 16 bytes

•  32 hex chars + 4 hyphens = 36 chars

•  UUID as text(UTF-8: 1 byte) ☞ 36 bytes

•  Bytes overhead = 36 – 16 = 20 bytes

102


Notes on contact_id •  20 bytes wasted per contact uuid

103



•  × 7 denormalizations = 140 bytes per contact uuid

104



•  × 7 denormalizations = 140 bytes per contact uuid

•  × 109 contacts = 140 GB wasted

😠 105

not even counting replication factor …


Notes on contact_id •  ☞ just save contact id as byte[ ]

106



•  Achilles @TypeTransformer for automatic conversion (see later)

107



•  Achilles @TypeTransformer for automatic conversion (see later)

•  Use blobAsBigInt( ) or blobAsUUID( ) to view data

108


Achilles •  Advanced "object mapper"

•  Fluent API

•  Tons of features

•  TDD friendly

109


Achilles •  Dirty checking, why is it important ?

110



•  1 contact ≈ 8 mutable fields

111




•  × 7 denormalizations = 56 update combinations …

112




•  × 7 denormalizations = 56 update combinations …

•  and not even counting multiple fields updates …

113


Achilles •  Are you going to manually generate 56+ prepared

statements for all possible updates ?

114


Achilles •  Are you going to manually generate 56+ prepared

statements for all possible updates ?

•  Or just use dynamic plain string statements and get some perf penalty ?

115


Achilles •  Dirty check in action

//No read-before-write ContactEntity proxy = manager.forUpdate(ContactEntity.class, contactId); proxy.setFirstName(…); proxy.setLastName(…); //type-safe updates proxy.setAddress(…);

manager.update(proxy);

116


Achilles

117

Empty Entity

DirtyMap

Proxy Setters interception

PrimaryKey


Achilles •  Dynamic statements generation

UPDATE contacts SET firstname=?, lastname=?,address=? WHERE contact_id=?

118

prepared statements are cached, of course


Achilles •  Insert strategy, what is it ?

119


Achilles •  Simple INSERT prepared statement

INSERT INTO contacts(contact_id,name,age,address,gender,avatar,…) VALUES(?, ?, ?, ? … ?);

120


Achilles •  Runtime values binding

•  some columns are optional

preparedStatement.bind(49374,’John DOE’,33, null, null, …, null);

121


Achilles

Wait … are you saying inserting null in CQL???

😳

122


Achilles

Inserting null ≡ creating tombstones

123


Achilles

Inserting null ≡ creating tombstones × 7 denormalizations

124


Achilles

Inserting null ≡ creating tombstones × 7 denormalizations

× billions of contacts created

😱 125

not even counting replication factor …


Achilles

@Entity(table = "contacts_by_id ») @Strategy(insert = InsertStrategy.NOT_NULL_FIELDS) public class ContactById {

}

126

•  Simple annotation


Achilles •  Runtime dynamic INSERT statement

INSERT INTO contacts(contact_id, name, age, address,) VALUES(:contact_id, :name, :age, :address);

127

prepared statements are cached, of course


Achilles

@PartitionKey @Column(name = "contact_id") @TypeTransformer(valueCodecClass = ContactIdToBytes.class) private ContactId contactId;

128

•  Remember the contactId ⇄ byte[ ] conversion ?

BYOC ☞ Bring Your Own Codec


Achilles public interface Codec<FROM, TO> { Class<FROM> sourceType(); Class<TO> targetType(); TO encode(FROM fromJava) FROM decode(TO fromCassandra); }

129


Achilles

130

2014-12-01 14:25:20,554 Bound statement : [INSERT INTO contacts.contacts_by_modification_date(user_id,month_bucket,modification_date,...) VALUES (:user_id,:month_bucket,:modification_date,...) USING TTL :ttl;] with CONSISTENCY LEVEL [LOCAL_QUORUM] 2014-12-01 14:25:20,554 bound values : [222130151, 2014-12, e13d0d50-7965-11e4-af38-90b11c2549e0, ...]

2014-12-01 14:25:20,701 Bound statement : [SELECT birthday,middlename,avatar_size,... FROM contacts.contacts_by_modification_date WHERE user_id=:user_id AND month_bucket=:month_bucket AND (modification_date)>=(:modification_date) ORDER BY modification_date ASC;] with CONSISTENCY LEVEL [LOCAL_QUORUM] 2014-12-01 14:25:20,701 bound values : [222130151, 2014-10, be6bc010-6109-11e4-b385-000038377ead]

•  Dynamic logging in action


Achilles

131

•  Dynamic logging

•  runtime activation

•  no need to recompile/re-deploy

•  save us hours of debugging

•  TRACE log level ☞ query tracing


Take Away


Conditions for success •  Data modeling is crucial

133



•  Double-run strategy & timestamp trick FTW

134




•  Data type conversion can be tricky

135





•  Benchmark !

136





•  Benchmark !

•  Mindset shifts for the team

137

Thank You

libon cassandra summiteu2014

Technology

project context application

rdbms oracle13

rdbms indices19

rdbms indices partitioning20

relational model21

contact matching6libon

messaging app voip

ms server response time