intro to riak · 2019. 5. 7. · riak kv client apis request coordination riak core get put delete...
TRANSCRIPT
![Page 1: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/1.jpg)
INTRO TO RIAK
![Page 2: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/2.jpg)
Riak Overview
![Page 3: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/3.jpg)
Riak
Distributed
![Page 4: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/4.jpg)
Riak
Distributed, replicated, highly available
![Page 5: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/5.jpg)
Riak
Distributed, highly available, eventually consistent
![Page 6: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/6.jpg)
Riak
Distributed, highly available, eventually consistent, key-Value Database
![Page 7: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/7.jpg)
Riak
Distributed, highly available, eventually consistent, key-Value Database
Mainly written in Erlang!
![Page 8: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/8.jpg)
Riak• Modelled after Amazon Dynamo*
• see annotated version of Dynamo paper with comparisons to Riak: http://docs.basho.com/riak/latest/references/dynamo/
*https://dl.acm.org/citation.cfm?id=1294281
![Page 9: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/9.jpg)
Amazon Dynamo
• SOSP 2007
• Latency - 100ms of latency cost them 1% in sales.
• “not novel” - synthesis of last 40 years dist-sys research
• Real world application of CS
![Page 10: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/10.jpg)
Riak• A database • Key-Value (like a hash table) • NoSql • Distributed - Fault Tolerant • Favours (write) Availability over
Consistency
![Page 11: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/11.jpg)
KEY-VALUE STORE
• Simple operations - GET, PUT, DELETE
• Value is opaque (mostly), with metadata
• Extras, e.g.
• Secondary Indexes (2i)
• MapReduce
• CRDTs/Search/Time Series etc etc
![Page 12: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/12.jpg)
FAULT TOLERANT
• All nodes participate equally - no single point of failure (SPOF)
• All data is replicated
• Cluster transparently survives...
• node failure
• network partitions
• Built on Erlang/OTP (designed for FT)
![Page 13: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/13.jpg)
Riak - Write Available
• Unable to write mean lost dollars • Amazon Shopping Cart
• Low Latency matters more than Consistency
![Page 14: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/14.jpg)
Riak Overview
![Page 15: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/15.jpg)
{“key”: “value”}
![Page 16: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/16.jpg)
Distributed
![Page 17: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/17.jpg)
Homogenous
![Page 18: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/18.jpg)
The ring
![Page 19: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/19.jpg)
The Ring
• Membership
• Ownership
• Routing
• Abstract and Concrete in Riak
![Page 20: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/20.jpg)
Riak Overview The Ring
![Page 21: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/21.jpg)
• 160-bit integer keyspace
• divided into fixed number of evenly-sized partitions/ranges
• partitions are claimed by nodes in the cluster
• replicas go to the N partitions following the key
node 0
node 1
node 2
node 3
hash(“users/clay-davis”)
N=3
The Ring - Consistent
Hashing
![Page 22: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/22.jpg)
VNODES
supervisor process !
basic unit of concurrency !
Process “knows” its range
![Page 23: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/23.jpg)
VNODES
A Key/Value database !
local storage !
bitcask/leveldb
![Page 24: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/24.jpg)
ROUTING
routing table !
mapping of ranges/vnodes to nodes
!
![Page 25: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/25.jpg)
GOSSIP
GOSSIP !
The ring is shared via epidemic gossip
![Page 26: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/26.jpg)
PRIMARY PREFERENCE LIST (preflist)
{SHA1(key)
![Page 27: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/27.jpg)
node 0
node 1
node 2
node 3+
hash(key)
![Page 28: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/28.jpg)
!
Replication
node 0
node 1
node 2
node 3
Replicas are stored N - 1 contiguous partitions
hash(“cities/london”)
![Page 29: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/29.jpg)
AvailabilityAny non-failing node can respond to any
request!!
--Gilbert & Lynch
![Page 30: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/30.jpg)
Fault Tolerance
node 0
node 1
node 2
node 3
Replicas are stored N - 1 contiguous partitions
node 2offline
put(“cities/london”)
![Page 31: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/31.jpg)
Fault Tolerance
node 0
node 1
node 2
node 3
Replicas are stored N - 1 contiguous partitions
node 2offline
put(œcities/london’)
FALLBACK “SECONDARY”
node 2HINTED HANDOFF
![Page 32: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/32.jpg)
node 0
node 1
node 2
node 3+
hash(key)
![Page 33: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/33.jpg)
node 0
node 1
node 2
node 3-
hash(key)
![Page 34: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/34.jpg)
node 0
node 1
node 2
node 3-
hash(key)
OWNERSHIPHANDOFF
![Page 35: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/35.jpg)
node 0
node 1
node 2
hash(key)
![Page 36: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/36.jpg)
Consistent Hashing
![Page 37: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/37.jpg)
Balanced Scaling
![Page 38: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/38.jpg)
Read Repair
![Page 39: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/39.jpg)
Replication with Sloppy Quorum
![Page 40: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/40.jpg)
One-Hop Request Routing
![Page 41: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/41.jpg)
The Ring - Summary• Membership - nodes in the cluster
• Ownership - claim of vnodes
• Routing - vnodes to nodes
• Vnodes - processes & databases
• Handoff - primaries/fallbacks, ownership change
![Page 42: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/42.jpg)
The Ring - Summary
• Automatic failure detection (heartbeats)
• Automatic fallback data storage
• Automatic healing - hinted handoff
• Automatic ownership transfer - handoff
![Page 43: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/43.jpg)
RIAK ARCHITECTUREErlang/OTP Runtime
Riak KV
Client APIs
Request Coordination
Riak Core
get put delete map-reduce
HTTP Protocol Buffers
Erlang local client
membershipconsistent hashing handoff
node-livenessgossip
buckets
vnodes
storage backend
Workers
vnode master
![Page 44: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/44.jpg)
Eventual Consistency
Eventual consistency is a consistency model used in distributed computing that informally guarantees that, if no new updates are made to a given data item, eventually all accesses to that item will return the last updated value. !!--Wikipedia!
![Page 45: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/45.jpg)
Riak Overview N, R, W, PR, PW etc
![Page 46: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/46.jpg)
REQUEST QUORUMS
• Every request contacts all replicas of key
• N - number of replicas (default 3)
• R - read quorum
• W - write quorum
• Quorum:The quantity of replicas that must respond to a read or write request before it is considered successful. (default 2 - Calculated as: floor(n_val / 2) + 1 )
![Page 47: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/47.jpg)
Quora: For Consistency• How many Replicas must respond: 1, quorum, all?
• Strict Quorum: Only Primaries are contacted
• Sloppy Quorum: Fallbacks are contacted
• Fast Writes? W=1
• Fast Reads? R=1
• Read Your Own Writes? PW+PR>N
![Page 48: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/48.jpg)
Replica A Replica B Replica C
Client X Client Y
PUT “sue”
C’
PUT “bob”
NO!!!! :(
Strict
![Page 49: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/49.jpg)
Replica A Replica B Replica C
Client X Client Y
PUT “sue”
C’
PUT “bob”
A’ B’
Sloppy
![Page 50: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/50.jpg)
ANATOMY OF A REQUESTget(“user_id”)
Get Handler (FSM)
clientRiak
hash(“user_id”)== 10, 11, 12
get(“user_id”)Coordinating node
Cluster
6 7 8 9 10 11 12 13 14 15 16
The Ring
R=2
v1 v2
v1 v2
v2
![Page 51: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/51.jpg)
READ REPAIR
v2v2
get(“user_id”)
Get Handler (FSM)
clientRiak
Coordinating nodeCluster
6 7 8 9 10 11 12 13 14 15 16
R=2 v1 v2
v2
v1
v2v1v1 v2v2
![Page 52: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/52.jpg)
Version Vectors - Logical Clocks
• Happens before or causal relationship
• vector of pairs {Actor, Counter}
• Each Actor updates own entry only
• Each Object has own Version Vector
• Concurrent Updates Detected
![Page 53: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/53.jpg)
![Page 54: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/54.jpg)
Summary• Distributed key Value
• Homogenous nodes
• Ring for membership, routing, ownership
• vnodes for datamanagement
• FSMs for read/write logic
• Always available for writes
![Page 55: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/55.jpg)
Trade Off
![Page 56: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/56.jpg)
CAP
![Page 57: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/57.jpg)
C Ahttp://aphyr.com/posts/288-the-network-
is-reliable
![Page 58: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/58.jpg)
C A
![Page 59: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/59.jpg)
C A
![Page 60: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/60.jpg)
C APEL
![Page 61: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/61.jpg)
Conflict!
![Page 62: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/62.jpg)
Replica A Replica B Replica C
Client X Client Y
PUT “sue”
C’
PUT “bob”
NO!!!! :(
CP
![Page 63: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/63.jpg)
Replica A Replica B Replica C
Client X Client Y
PUT “sue”
C’
PUT “bob”
A’ B’
AP
![Page 64: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/64.jpg)
Conflict!
Replica A Replica B Replica C
ClientGET
“Bob”
“Bob”
“Sue”
![Page 65: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/65.jpg)
Eventual Consistency
Eventual consistency is a consistency model used in distributed computing that informally guarantees that, if no new updates are made to a given data item, eventually all accesses to that item will return the last updated value. !!--Wikipedia!
![Page 66: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/66.jpg)
LAST UPDATED VALUE?
• Last by time?
• What about concurrent operations?
• What is the “last updated value?”
![Page 67: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/67.jpg)
Last Write Wins!
Replica A Replica B Replica C
ClientGET
“Bob” ts=1234”
“Bob” ts=1234
“Sue” ts=1235
![Page 68: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/68.jpg)
Last Write Wins
Replica A Replica B Replica C
Client
“Sue”
![Page 69: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/69.jpg)
Multi-Value
Replica A Replica B Replica C
Client
[“Bob”, “Sue”] [{a,1}, {c, 1}]
![Page 70: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/70.jpg)
Semantic Resolution
![Page 71: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/71.jpg)
DynamoThe Shopping Cart
![Page 72: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/72.jpg)
A B
HAIRDRYER
![Page 73: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/73.jpg)
A B
HAIRDRYER
![Page 74: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/74.jpg)
A B
PENCIL CASE
HAIRDRYER
![Page 75: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/75.jpg)
A B
PENCIL CASEHAIRDRYER
![Page 76: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/76.jpg)
A B
[HAIRDRYER], [PENCIL CASE]
![Page 77: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/77.jpg)
Semantic Resolution
![Page 78: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/78.jpg)
MergeSet Union of Values
Simple, right?
![Page 79: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/79.jpg)
Conflicting Writes• Version Vector Detects Concurrency
• Multi-Value Register
• Application “merges” to a single value
• Semantic Resolution
• Tells Riak “value is X”
![Page 80: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/80.jpg)
Riak Part 1 Summary
• Consistent Hashing and the ring
• Availability over Consistency
• Trade-Off - Great for Business and Ops
• Can be challenging for developers
![Page 81: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/81.jpg)
Hands On
• vagrant up
• vagrant ssh
• cd lecture/riak
![Page 82: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/82.jpg)
Hands On• Create a bucket
• Create some values
• Pass the vclock
• Create some siblings!
• Data Modelling
![Page 83: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/83.jpg)
Why CRDTs?
![Page 84: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/84.jpg)
Conflict!
![Page 85: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/85.jpg)
Replica A Replica B Replica C
Client X Client Y
PUT “sue”
C’
PUT “bob”
A’ B’
AP
![Page 86: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/86.jpg)
Conflict!
Replica A Replica B Replica C
ClientGET
“Bob”
“Bob”
“Sue”
![Page 87: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/87.jpg)
![Page 88: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/88.jpg)
Google F1“Designing applications to cope with concurrency anomalies in their data is very error-prone, time-consuming, and ultimately not worth the performance gains.”
![Page 89: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/89.jpg)
http://www.infoq.com/articles/key-lessons-learned-from-transition-to-nosql
“…writing merge functions was likely to confuse the hell out of all our developers and slow down
development…”
![Page 90: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/90.jpg)
Set Union? “Anomaly” Reappear
Removes?
![Page 91: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/91.jpg)
Absence How can you tell if X is missing from A but present in B because A hasn’t yet seen the addition, or if A has removed it already?
![Page 92: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/92.jpg)
![Page 93: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/93.jpg)
http://www.infoq.com/articles/key-lessons-learned-from-transition-to-nosql
“…after some analysis we found that much of our data could be modelled
within sets so by leveraging CRDT’s our developers don't have to worry about
writing bespoke merge functions for 95% of carefully selected use cases…”
![Page 94: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/94.jpg)
CRDT Sets
answers the question of "what is in the set?" when presented with siblings:
!
[x,y,z] | [w,x,y]
![Page 95: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/95.jpg)
CRDT Sets
is w not added by A or removed by A? is z not added be B or removed by B?
!
[x,y,z] | [w,x,y]
![Page 96: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/96.jpg)
CRDT Sets
a semantic of “Add-Wins” via
“Observed Remove”
![Page 97: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/97.jpg)
This project is funded by the European Union,
7th Research Framework Programme, ICT call 10,
grant agreement n°609551.
![Page 98: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/98.jpg)
Hands On
• CRDTs Sets
![Page 99: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/99.jpg)
Data Types• Counters
• Sets
• Booleans
• Maps
• compose all the above (recursively)
![Page 100: INTRO TO RIAK · 2019. 5. 7. · Riak KV Client APIs Request Coordination Riak Core get put delete map-reduce HTTP Protocol Buffers Erlang local client membership consistent hashing](https://reader036.vdocuments.mx/reader036/viewer/2022063022/5fe71ee95d48755a6a0c9b55/html5/thumbnails/100.jpg)
Riak - Summary• Always Write Available Key-Value Database
• Self-Healing fault tolerance
• Eventually Consistent
• Be aware of trade-offs and use cases
• CRDTs simplify data modelling
• Research on databases is ACTIVE and INTERESTING