crdt sets: theory & practice - cse.chalmers.se · consistency there must exist a total order on...
TRANSCRIPT
![Page 1: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/1.jpg)
CRDT Sets: Theory & Practice
Russell Brown Basho Technologies
![Page 2: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/2.jpg)
What?
• Why we need CRDTs
• What’s a CRDT, anyway?
• To a general CRDT set
![Page 3: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/3.jpg)
What?
• Riak Set Data Type
• Delta-Sets
• “Big”sets
![Page 4: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/4.jpg)
This project is funded by the European Union,
7th Research Framework Programme, ICT call 10,
grant agreement n°609551.
![Page 5: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/5.jpg)
Why CRDTs?
![Page 6: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/6.jpg)
Scale Up
$$$Big Iron (still fails)
![Page 7: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/7.jpg)
Scale Out
Commodity Servers Distributed Systems
Multi Datacenter
![Page 8: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/8.jpg)
DISTRIBUTED DATABASE
![Page 9: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/9.jpg)
Trade Off
![Page 10: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/10.jpg)
CAP
![Page 11: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/11.jpg)
C A
http://aphyr.com/posts/288-the-network-is-reliable
![Page 12: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/12.jpg)
C A
![Page 13: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/13.jpg)
C A
![Page 14: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/14.jpg)
C AP P
![Page 15: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/15.jpg)
ConsistencyThere must exist a total order on all operations
such that each operation looks as if it were completed at a single instant. This is equivalent to requiring requests of
the distributed shared memory to act as if they were executing on a single node, responding to operations one at a time.
--Gilbert & Lynch
![Page 16: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/16.jpg)
One important property of an atomic read/write shared memory is that
any read operation that begins after a write operation completes must return that value, or the result of a later write operation. This is the consistency guarantee that generally provides
the easiest model for users to understand, and is most convenient for those attempting to design a client application that uses the distributed service
--Gilbert & Lynch
Consistency
![Page 17: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/17.jpg)
https://aphyr.com/posts/313-strong-consistency-models
![Page 18: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/18.jpg)
Replica A Replica B Replica C
Client X Client Y
PUT “sue”PUT “bob”
NO!!!! :(
Consistent
![Page 19: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/19.jpg)
AvailabilityAny non-failing node can respond to any
request
--Gilbert & Lynch
![Page 20: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/20.jpg)
Replica A Replica B Replica C
Client X Client Y
PUT “sue”PUT “bob”
NO!!!! :(
Consistent
![Page 21: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/21.jpg)
Consensus for a total order of events
![Page 22: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/22.jpg)
Requires a quorum
![Page 23: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/23.jpg)
Coordination waits
![Page 24: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/24.jpg)
Replica A Replica B Replica C
Client X Client Y
PUT “sue”PUT “bob”
Consistent
![Page 25: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/25.jpg)
Client X put “BOB”
Client Y put “SUE”
Events put in a TOTAL ORDER
![Page 26: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/26.jpg)
Eventual Consistency
Eventual consistency is a consistency model used in distributed computing that informally guarantees that, if no new updates are made to a given data item, eventually all accesses to that item will return the last updated value.
--Wikipedia
![Page 27: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/27.jpg)
https://aphyr.com/posts/313-strong-consistency-models
![Page 28: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/28.jpg)
Replica A Replica B Replica C
Client X Client Y
PUT “sue”
C’
PUT “bob”
A’ B’
Available
![Page 29: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/29.jpg)
Optimistic replication (and logical clocks)
![Page 30: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/30.jpg)
Reconcile concurrency on read
![Page 31: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/31.jpg)
No coordination for lower latency
![Page 32: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/32.jpg)
Replica A Replica B Replica C
Client X Client Y
PUT “sue”PUT “bob”
Low Latency
[c1] “sue”
[c1] “sue”[a1] “bob”
![Page 33: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/33.jpg)
Problem?
![Page 34: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/34.jpg)
ConsistencyThis is the consistency guarantee that generally provides the easiest model for users to understand, and is most convenient for those attempting to design a client application that uses the distributed service
--Gilbert & Lynch
![Page 35: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/35.jpg)
Conflict!
Replica A Replica B Replica C
ClientGET
“Bob”
“Bob”
“Sue”
![Page 36: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/36.jpg)
Eventual Consistency
Eventual consistency is a consistency model used in distributed computing that informally guarantees that, if no new updates are made to a given data item, eventually all accesses to that item will return the last updated value.
--Wikipedia
![Page 37: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/37.jpg)
Last Updated Value?
![Page 38: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/38.jpg)
Convergence
![Page 39: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/39.jpg)
Availability is great - what’s my data?
![Page 40: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/40.jpg)
It depends
![Page 41: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/41.jpg)
Last Write Wins!
Replica A Replica B Replica C
ClientGET
“Bob” ts=1234”
“Bob” ts=1234
“Sue” ts=1235
![Page 42: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/42.jpg)
Physics Problem
4,148 km 14 ms Light 21 ms fibre
SF NY
PUT “bob” 1394382600000
PUT “sue” 1394382600020
![Page 43: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/43.jpg)
Last Write Wins
Replica A Replica B Replica C
Client
“Sue”
![Page 44: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/44.jpg)
LWW - A Lossy Total Order
![Page 45: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/45.jpg)
Conflict!
Replica A Replica B Replica C
ClientGET
“Bob” [{a,1}]
“Bob”[{a,1}]
“Sue” [{c, 1}]
![Page 46: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/46.jpg)
Multi-Value
Replica A Replica B Replica C
Client
[“Bob”, “Sue”] [{a,1}, {c, 1}]
![Page 47: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/47.jpg)
MVR - A Partial Order
![Page 48: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/48.jpg)
happens before
concurrent ——— divergent
convergent
Logical Clocks
![Page 49: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/49.jpg)
Summary
• Distributed systems for scale/fault tolerance/perf
• CAP trade-off
• Eventual consistency - concurrent writes
![Page 50: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/50.jpg)
![Page 51: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/51.jpg)
Semantic Resolution
![Page 52: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/52.jpg)
DynamoThe Shopping Cart
![Page 53: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/53.jpg)
A B
HAIRDRYER
![Page 54: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/54.jpg)
A B
HAIRDRYER
![Page 55: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/55.jpg)
A B
PENCIL CASE
HAIRDRYER
![Page 56: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/56.jpg)
A B
PENCIL CASEHAIRDRYER
![Page 57: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/57.jpg)
A B
[HAIRDRYER], [PENCIL CASE]
![Page 58: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/58.jpg)
ConvergeSet Union of Values
Simples, right?
![Page 59: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/59.jpg)
Set Union? “Anomaly” Reappear
Removes?
![Page 60: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/60.jpg)
Google F1“We have a lot of experience with eventual consistency systems at Google.”
“We find developers spend a significant fraction of their time building extremely complex and error-prone mechanisms to cope with eventual consistency”
![Page 61: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/61.jpg)
Google F1“Designing applications to cope with concurrency anomalies in their data is very error-prone, time-consuming, and ultimately not worth the performance gains.”
![Page 62: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/62.jpg)
http://www.infoq.com/articles/key-lessons-learned-from-transition-to-nosql
“…writing merge functions was likely to confuse the hell out of all our developers and slow down
development…”
![Page 63: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/63.jpg)
Ad Hoc
![Page 64: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/64.jpg)
![Page 65: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/65.jpg)
What’s a CRDT?
![Page 66: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/66.jpg)
State Based CRDT Convergent
![Page 67: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/67.jpg)
Join Semi-lattice
![Page 68: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/68.jpg)
Join Semi-latticePartially ordered set; Bottom; least upper bound
⊥ ⨆𝐒 ⟩⟨ , ,
![Page 69: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/69.jpg)
Associativity: (X⨆Y)⨆Z = X⨆(Y⨆Z)
Join Semi-lattice
![Page 70: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/70.jpg)
Commutativity: X⨆Y = Y⨆X Join Semi-lattice
![Page 71: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/71.jpg)
Idempotent: X⨆X = X
Join Semi-lattice
![Page 72: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/72.jpg)
Objects grow over time; merge computes LUB
Join Semi-lattice
![Page 73: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/73.jpg)
Examples
Join Semi-lattice
![Page 74: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/74.jpg)
b a c
a, b a, c
a, b, c
Set; merge function: union.
b, c
![Page 75: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/75.jpg)
3 5 7
5 7
7
Increasing natural; merge function: max.
![Page 76: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/76.jpg)
F F T
F T
T
Booleans; merge function: or.
![Page 77: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/77.jpg)
Deterministic
MergeIdempotentAssociative
Commutative
![Page 78: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/78.jpg)
Principled Merge
![Page 79: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/79.jpg)
Reusable Data Types
Defined Semantics
![Page 80: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/80.jpg)
Evolution of a CRDT Set
![Page 81: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/81.jpg)
http://www.infoq.com/articles/key-lessons-learned-from-transition-to-nosql
“…after some analysis we found that much of our data could be modelled
within sets so by leveraging CRDT’s our developers don't have to worry about
writing bespoke merge functions for 95% of carefully selected use cases…”
![Page 82: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/82.jpg)
Evolution of a CRDT Set
![Page 83: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/83.jpg)
Evolution of a Set
G-SET
![Page 84: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/84.jpg)
Evolution of a Set
G-SET
![Page 85: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/85.jpg)
Bob
Shelly
![Page 86: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/86.jpg)
Bob
Pete
Shelly
![Page 87: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/87.jpg)
Bob
Pete
Shelly
Anna
Joe
![Page 88: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/88.jpg)
Bob
Pete
Shelly
Anna
Joe
Reece
Pete
Shelly
Alex
![Page 89: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/89.jpg)
Bob
Pete
Shelly
Anna
Joe
Reece
Pete
Shelly
Alex
⨆
![Page 90: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/90.jpg)
Bob
Pete
Shelly
Anna
Joe
Reece
Alex
![Page 91: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/91.jpg)
Removes?
![Page 92: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/92.jpg)
Evolution of a Set
G-SET2P-SET
![Page 93: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/93.jpg)
Adds Removes
Bob
Pete
Shelly
Bob
Pete
Shelly
Anna
Shelly
![Page 94: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/94.jpg)
Adds Removes
Bob
Pete
Shelly
Bob
Pete
Shelly
Anna
Anna=
![Page 95: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/95.jpg)
Value /= Structure
![Page 96: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/96.jpg)
Adds Removes
Bob
Pete
Shelly
Bob
Pete
Shelly
Anna
Anna=
![Page 97: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/97.jpg)
I changed my mind!
![Page 98: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/98.jpg)
CRDT Sets
answers the question of "what is in the set?" when presented with siblings:
[x,y,z] | [w,x,y]
![Page 99: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/99.jpg)
CRDT Sets
is w not added by A or removed by A? is z not added be B or removed by B?
[x,y,z] | [w,x,y]
![Page 100: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/100.jpg)
Evolution of a Set
U-SET
![Page 101: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/101.jpg)
2
3
Bob
Pete
1 Shelly
4 Anna
6
7
Jack
Jed
5 Shelly
⨆
![Page 102: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/102.jpg)
2
3
Bob
Pete
1,5 Shelly
4 Anna
6
7
Jack
Jed
![Page 103: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/103.jpg)
Evolution of a Set
U-SETOR-SET
![Page 104: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/104.jpg)
Adds
2
3
Removes
Bob
Pete
1 Shelly
2
3
Bob
Pete
1 Shelly
4 Anna
![Page 105: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/105.jpg)
Adds
2
3
Removes
Bob
Pete
1 Shelly
2
3
Bob
Pete
1 Shelly
4 Anna
5 Shelly
![Page 106: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/106.jpg)
Adds
2
3
Bob
Pete
1 Shelly
Replica A
![Page 107: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/107.jpg)
Adds
2
3
Bob
Pete
1 Shelly
Replica A
Remove
Removes
2
3
Bob
Pete
1 Shelly
![Page 108: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/108.jpg)
Adds
Replica B
4 Anna
5 Shelly
![Page 109: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/109.jpg)
Adds
2
3
Removes
Bob
Pete
1 Shelly
2
3
Bob
Pete
1 Shelly
⨆ Adds
4 Anna
5 Shelly
![Page 110: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/110.jpg)
Adds
2
3
Removes
Bob
Pete
1 Shelly
2
3
Bob
Pete
1 Shelly
4 Anna
5 Shelly
Anna=Shelly
![Page 111: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/111.jpg)
ObservedRemove
![Page 112: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/112.jpg)
Semantics
AddWins
![Page 113: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/113.jpg)
Evolution of a Set
U-SETOR-SET
![Page 114: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/114.jpg)
Adds
2
3
Removes
Bob
Pete
1 Shelly
2
3
Bob
Pete
1 Shelly
4 Anna
5 Shelly
Anna=Shelly
4 Anna
5 Shelly
[]
![Page 115: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/115.jpg)
![Page 116: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/116.jpg)
![Page 117: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/117.jpg)
Evolution of a Set
U-SETOR-SET
OR-SWOT
![Page 118: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/118.jpg)
[{a, 1}]
{a, 1} Shelly
![Page 119: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/119.jpg)
[{a, 1}] [{a, 1}
{a, 1} Shelly {a, 1} Shelly
![Page 120: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/120.jpg)
[{a, 1}] [{a, 1}, {b, 3}]
{a, 1} Shelly
{b, 1}
{b, 2}
{b, 3}
Bob
Pete
Phil
{a, 1} Shelly
![Page 121: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/121.jpg)
[{a, 1}, {b,3}] [{a, 1}, {b, 3}]
{a, 1} Shelly
{b, 1}
{b, 2}
{b, 3}
Bob
Pete
Phil
{a, 1} Shelly
{b, 1}
{b, 2}
{b, 3}
Bob
Pete
Phil
![Page 122: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/122.jpg)
[{a, 2}, {b, 3}]
{b, 1}
{b, 3}
[{a, 1}, {b, 4}]
Bob
Pete
{a, 1} Shelly
{b, 1}
{b, 2}
{b, 3}
Bob
Pete
Phil
{a, 1} Shelly
{a, 2} Anna {b, 4} John
![Page 123: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/123.jpg)
[{a, 2}, {b, 3}]
{b, 1}
{b, 3}
[{a, 2}, {b, 4}]
Bob
Pete
{a, 1} Shelly
{b, 1}
{a, 2}
{b, 3}
Bob
Pete
Anna
{a, 1} Shelly
{a, 2} Anna
{b, 4} John
![Page 124: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/124.jpg)
[{a, 2}, {b, 4}]
{b, 1}
{a, 2}
{b, 3}
Bob
Pete
Anna
{a, 1} Shelly
{b, 4} John
= Bob
Pete
Anna
Shelly
John
![Page 125: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/125.jpg)
CRDT Sets
a semantic of “Add-Wins” via
“Observed Remove”
![Page 126: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/126.jpg)
CRDTs
• Principled Merge
• Data Types with Defined Semantic
• Fine Grained Causality
• minimal representation
![Page 127: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/127.jpg)
CRDTs IRL
![Page 128: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/128.jpg)
Replica A Replica B Replica C
Client X Client Y
PUT “sue”
C’
PUT “bob”
A’ B’
Available
![Page 129: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/129.jpg)
Replica A Replica B Replica C
Client X Client Y
PUT “sue”PUT “bob”
Low Latency
[c1] “sue”
[c1] “sue”[a1] “bob”
![Page 130: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/130.jpg)
Conflict!
Replica A Replica B Replica C
ClientGET
“Bob”
“Bob”
“Sue”
![Page 131: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/131.jpg)
A B
[HAIRDRYER], [PENCIL CASE]
![Page 132: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/132.jpg)
{“key”: “value”}
![Page 133: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/133.jpg)
Sets
C1 C2
RIAK
GET Friends GET Friends
![Page 134: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/134.jpg)
State To Client State
C1 C2
RIAK
[]
[]
![Page 135: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/135.jpg)
State To Client
C1
RIAK
PUT [Rita][Rita]
![Page 136: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/136.jpg)
State To Client
C2
RIAK
PUT [Sue]{[Rita], [Sue]}
![Page 137: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/137.jpg)
State To Client
C1
RIAK
PUT [Rita, Bob]
{[Sue], [Rita, Bob]}
![Page 138: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/138.jpg)
State To Client
C2
RIAK
PUT [Rita, Sue, Bob]
![Page 139: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/139.jpg)
Problem?• Requires Read Your Own Writes consistency
• Client must manage Actors in set’s logical clock
• Client ensures invariants
• Serial actor, total order of events
• Read and Send all Data to Add/Remove an Element??
![Page 140: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/140.jpg)
Operations!
C1
RIAK
Add “Bob”
![Page 141: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/141.jpg)
Operations!
C1
RIAK
Remove “Sue”
![Page 142: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/142.jpg)
Operations! Observed Remove
C1
RIAK
Remove “Sue” with Context
![Page 143: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/143.jpg)
Operations!
C1
RIAK
GET Friends -> [Bob, Rita, Sue] [{a,1}, {b, 3}]
![Page 144: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/144.jpg)
Operations! Observed-Remove
C1
RIAK
Remove “Sue” with Context [{a,1}, {b, 3}]
![Page 145: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/145.jpg)
Riak 2.0Riak Data Types
Riak_DT CRDTs
![Page 146: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/146.jpg)
[{a, 2}, {b, 2}]
{b, 1}
{a, 2}
[{a, 1}, {b, 3}]
Y
Z
{a, 1} X
{b, 1} Y
{a, 1} X⊔ =
[{a, 2}, {b, 3}]
{b, 1} Y
{a, 1} X
{a, 2} Z
{b, 2} W
{b, 3} A {b, 3} A
![Page 147: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/147.jpg)
Sets in Riak
An optimized conflict-free replicated set Annette Bieniusa et al
http://arxiv.org/abs/1210.3368
![Page 148: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/148.jpg)
Sets in Riak
![Page 149: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/149.jpg)
Sets in Riak
![Page 150: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/150.jpg)
Sets in Riak
PHOTO © 2011 J. RONALD LEE, CC ATTRIBUTION 3.0. https://www.flickr.com/photos/jronaldlee/5566380424
![Page 151: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/151.jpg)
Teach Riak about CRDTs
• API Boundary
• Syntactic merge riak_object:merge
• Version Vector merge and sibling storage
• CRDT == no siblings
![Page 152: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/152.jpg)
Problem?
![Page 153: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/153.jpg)
Use Case• bet365 million pound customer
• Use CRDT sets for open bet tracking
• Partition Riak Sets
• Performance - write speed
• Size - cardinality
![Page 154: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/154.jpg)
Use Case• NHS England
• Use CRDT sets for mailbox app
• Truncate/archive older messages
• Performance - write speed
• Size - cardinality
![Page 155: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/155.jpg)
Problem?
• Poor Write speed • Can’t have “big” sets
![Page 156: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/156.jpg)
Sets in Riak
![Page 157: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/157.jpg)
Sets in Riak
10k sets, 100k elements, 50 workers - write
![Page 158: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/158.jpg)
Sets in Riak• read at replica
• deserialise
• mutate
• serialise
• write
![Page 159: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/159.jpg)
Sets In Riak
• replicate FULL STATE
• (read, deserialise? merge? serialise?, write?)
• ? riak_object.vv
• Accidental Optimisation
![Page 160: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/160.jpg)
Every time we change the set we read and write the
whole set!
![Page 161: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/161.jpg)
Delta-Sets• Only replicate the Delta - the change
• The delta is element + causal tag
• Can be “Joined” like full state
• Idempotent/Associate/Commutative
• Efficient State-based CRDTs by Delta-Mutation - Paulo Sérgio Almeida et al
![Page 162: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/162.jpg)
Delta-Sets in Riak• Still read whole set to generate delta
• Still read whole set to merge delta - in fact MUST
• (read, deserialise! merge! serialise! write!)
• Database - disk i/o is THE thing
• Delta is always concurrent/sibling
• Save on the network, pay on the disk
![Page 163: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/163.jpg)
Sets in Riak
Small : riak object 1MB limit
![Page 164: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/164.jpg)
Disconnect
• Paper - minimal model to express innovation
• A set Actors, each a replica
• A single CRDT in memory
• Reads are R=1
![Page 165: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/165.jpg)
Disconnect• Riak - A real world industrial database product
• Many Keys, many CRDTs
• Durably stored on Disk - serialisation
• Clients act remotely on State
• One Key, One Set O(n)
![Page 166: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/166.jpg)
Problem Summary
• Join is expensive
• Serialisation/Deserialisation dance wasteful
• Disk i/o matters to a database!
![Page 167: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/167.jpg)
Bigsets: Make writes faster
and sets bigger
![Page 168: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/168.jpg)
Bigset Design: Overview
![Page 169: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/169.jpg)
Bigset Design: Overview
![Page 170: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/170.jpg)
Bigset Design: Overview
• Decomposed
• A clock, and some elements
• each gets a key in leveldb
![Page 171: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/171.jpg)
Initial Results
![Page 172: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/172.jpg)
Initial Results
10k sets, 100k elements, 50 workers - write
![Page 173: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/173.jpg)
10k sets, 100k elements, 50 workers - write
![Page 174: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/174.jpg)
One small change
• Thinking from the bottom up
• Thinking about the disk and the database
• NOT a theoretical model
![Page 175: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/175.jpg)
Bigset Design: write• read clock
• increment
• assign dot to element
• store clock+element
• replicate delta
![Page 176: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/176.jpg)
Bigset Design: write
• read clock
• if seen dot, ignore
• else add dot to clock
• store clock+element
![Page 177: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/177.jpg)
Bigset Design: Clock
• Base VV [{actor, counter}]
• “dot-cloud” [{actor, [counter]}]
![Page 178: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/178.jpg)
Bigset Design: Clock Gaps?
A CB
Add “x” Add “y” Add “z”
![Page 179: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/179.jpg)
Bigset Design: Clock Gaps?
A CB
{a, 1} {b, 1} {c, 1}x zy
![Page 180: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/180.jpg)
Bigset Design: Clock Gaps?
A CB
{a, 1} {b, 1} {c, 1}x zy
x{a,1}->x
![Page 181: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/181.jpg)
Bigset Design: Clock Gaps?
A CB
{a, 1} {b, 1} {c, 1}x zy
x{b,1}->y
![Page 182: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/182.jpg)
Bigset Design: Clock Gaps?
A CB
{a, 1} {b, 1} {c, 1}x
x
A
{a, 1}
B
{b, 1}
![Page 183: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/183.jpg)
Bigset Design: Clock Gaps?
A CB
{a, 1} {b, 1} {c, 1}x
x
A
{a, 1}
B
{b, 1}
Add “n” Add “o” Add “p”
![Page 184: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/184.jpg)
Bigset Design: Clock Gaps?
A CB
{b, 1} {c, 1}x
x
A
{a, 1}
B
{b, 1}
Add “n” Add “o” Add “p”
{a, 1}
{a, 2} {b, 2} {c, 2}
![Page 185: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/185.jpg)
Bigset Design: Clock Gaps?
A CB
{b, 1} {c, 1}x
x
A
{a, 1}
B
{b, 1}{a, 1}
{a, 2} {b, 2} {c, 2}
{a,2}->n
![Page 186: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/186.jpg)
Bigset Design: Clock Gaps?
A CB
{b, 1} {c, 1}x
x
A
{a, 1}
B
{b, 1}{a, 1}
{a, 2} {b, 2} {c, 2}
{c,2}->p
![Page 187: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/187.jpg)
Bigset Design: Clock Gaps?
A CB
{b, 1} {c, 1}x
x
A
{a, 1}
B
{b, 1}{a, 1}
{a, 2}{b, 2} {c, 2}
A
{a, 2} {a, 2}
C
{c, 2}
C
{c, 2}
B
{b, 2}
![Page 188: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/188.jpg)
Bigset Design: Clock Gaps?
C
{c, 1}
{a, 2} {c, 2}
A B
{b, 2}
{a, 7}
{b, 3}
{b, 200}
![Page 189: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/189.jpg)
Bigset Design: elements• <<Set, Element, Actor, Cnt>> so Actor,Cnt make a
dot
• Times/Space trade off for concurrent elements
• Ordered by Set, Element, Actor, Cnt
• c++ key comparator for leveldb
• No serialisation - fast writes
![Page 190: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/190.jpg)
Bigset Design: End Key
• <<Set, $z>>
• Sorts last
![Page 191: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/191.jpg)
Bigset Design:Sorting
• Clock first, then elements, the end key
• For each set the keys are contiguous
![Page 192: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/192.jpg)
Reads?
![Page 193: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/193.jpg)
Initial Read Results
![Page 194: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/194.jpg)
10k sets, 100k elements, 20 workers - read
![Page 195: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/195.jpg)
Bigset Design: Read
• Iterate over many keys
• Leveldb iterate -> erlang fold
![Page 196: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/196.jpg)
Bigset Design: Read• “Streaming Fold” over Set (start-to-key-end key in a
buffer)
• Configurable chunks, say 100k elements
• Stream keys in batches to read_fsm - back pressure
• Read fsm incremental ORSWOT merge over R replicas
• stream results to client
![Page 197: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/197.jpg)
Bigset Designclock
element-N
end_key
set-tombstone
element-1
![Page 198: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/198.jpg)
Bigset Design: read
Client X
read fsm
block1
replica A replica C
block1
blockN
block1
blockN
incremental merge
blockN
![Page 199: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/199.jpg)
Reads Today
10k sets, 100k elements, 20 workers - read
![Page 200: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/200.jpg)
Reads Today
10k sets, 100k elements, 20 workers - read
![Page 201: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/201.jpg)
Full Set Read or Queries?
• Decomposed design enables queries
• Is Member
• subset queries - per vnode is c++
• Range queries SORTED!
• Pagination
![Page 202: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/202.jpg)
Is Member(X)
Client X
read fsm
is_member(<<bob>>).
replica A replica C
read(<<bob>>)
![Page 203: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/203.jpg)
Is Member(X)Replica A
{ReplicaA, clock} [{a, 3}, {b,3}]
{<<bob>>, a, 1} <<>>
{<<bob>>, b, 2} <<>>
{<<shelly>>, b, 1} <<>>
{ReplicaA, end_key} <<>>
read clock
{<<anne>>, a, 3} <<>>
Seek <<bob>>
![Page 204: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/204.jpg)
Is Member(X)Replica A
{ReplicaA, clock} [{a, 3}, {b,3}]
{<<bob>>, a, 1} <<>>
{<<bob>>, b, 2} <<>>
{<<shelly>>, b, 1} <<>>
{ReplicaA, end_key} <<>>
read clock
{<<anne>>, a, 3} <<>>
Seek <<bob>>
![Page 205: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/205.jpg)
Is Member(X)Replica A
[{a, 3}, {b,3}]
<<bob>> [{a,1}, {b,2}]
read fsm
<<bob>> Set
Replica C
[{a,3}, {c, 1}]
<<bob>> [{c, 1}]
![Page 206: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/206.jpg)
Is Member(X)
⊔=[{a, 3}, {b,3}]
<<bob>> [{a,1}, {b,2}]
[{a,3}, {c, 1}]
<<bob>> [{c, 1}]
==<<bob>> [{b,2}, {c,1}]
Read FSM
{true, [{b,2}, {c, 1}]}Client X
![Page 207: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/207.jpg)
Removes
• Observed-Remove - context
• Requires _some kind_ of read
• cheap membership check
![Page 208: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/208.jpg)
Sets in Riak
• Adds are removes!
• Action-at-a-distance!
• Clients are NOT replicas
![Page 209: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/209.jpg)
Adds are removes
[{a, 1}, {b, 4}, {c,1}]
{b, 1}
{b, 2}
{b, 3}
Bob
Pete
Phil
{a, 1}, {c, 1}
Shelly
{b, 4} John
Add “Shelly”
replica b
![Page 210: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/210.jpg)
Adds are removes
[{a, 1}, {b, 5}, {c,1}]
{b, 1}
{b, 2}
{b, 3}
Bob
Pete
Phil {a, 1} {c, 1}
Shelly
{b, 4} John
Not concurrent - Seen
replica b
{b, 5} Shelly
![Page 211: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/211.jpg)
Action-at-a-Distance
[{a, 1}, {b, 4}, {c, 1}
{b, 1}
{b, 2}
{b, 3}
Bob
Pete
Phil
{b, 4} John
replica b
{a, 1} {c, 1}
Shelly Client XAdd “Shelly”
![Page 212: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/212.jpg)
Action-at-a-Distance
[{a, 1}, {b, 4}, {c, 1}
{b, 1}
{b, 2}
{b, 3}
Bob
Pete
Phil {a, 1} {c, 1}
Shelly
{b, 4} John
Are adds removes? Concurrent?
Seen?
replica b
{a, 1} {c, 1}
Shelly
?
![Page 213: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/213.jpg)
Contexts & Consistency
• add X no ctx
• empty ctx - always concurrent (safe!)
• local ctx - non-deterministic (could remove all, some, none other X)
• depends on handling vnode’s state
![Page 214: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/214.jpg)
Contexts & Consistency
• add X + per element ctx
• only removes observed X regardless of handling vnode
• mmmm, deterministic outcome
![Page 215: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/215.jpg)
Bigset
• Adds are removes!
• They need a context
• Cheap is_member(X)
![Page 216: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/216.jpg)
Elided Complications• Hand-Off
• 1-way Full state merge
• Without reading keys on receiving side!
• Event Set Maths
• Anti-Entropy
• Read Repair
• Multi-Data-Center
![Page 217: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/217.jpg)
• MUST read full set at sender
• Read full state at receiver?
• Would be bad
Bigset Handoff
![Page 218: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/218.jpg)
• Sender sends keys
• Keys receiver hasn’t seen - store
• key receiver has seen - ignore
• Only read clock!
Bigset Handoff
![Page 219: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/219.jpg)
• Sender doesn’t send keys it has removed
• key receiver never saw - merge clocks
• Just read clock!
• key receiver saw - add to set-tombstone
Bigset Handoff
![Page 220: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/220.jpg)
• (Compact) Set of causal tags of removed keys
• Stored like on disk in a key
Set-Tombstone
![Page 221: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/221.jpg)
• handoff receiver state
• On sender clock_key
• C=clock, T=bigset_clock:fresh()
• for each key received add dot to tracker
Bigset Handoff
![Page 222: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/222.jpg)
• On end_key
• C - T = Removed Dots
• ToRemove = Receiver Clock intersected with Removed Dots
• Receiver Tombstone + ToRemove
Bigset Handoff
![Page 223: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/223.jpg)
Compaction• set-tombstone - logical clock/set of dots
• leveldb compaction
• if dot < set-tombstone discard
• set-tombstone = set-tombstone - dot
• tell vnode after compaction “remove dots [dot()] from set-tombstone”
![Page 224: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/224.jpg)
Next?• Production level code for Riak 2.x
• Causal Consistency
• More types - “Big”Maps
• Tables of Maps or Sets
• SQL over Eventual Consistency
![Page 225: CRDT Sets: Theory & Practice - cse.chalmers.se · Consistency There must exist a total order on all operations such that each operation looks as if it were completed at a single instant](https://reader035.vdocuments.mx/reader035/viewer/2022070904/5f721d816ac771462f47c5cc/html5/thumbnails/225.jpg)
Summary• Eventual Consistency buys you low latency &
availability
• Conflicts can be hard for application developers
• CRDTs help
• A little engineering goes a long way
• Decomposition brings complications too