stream-based data synchronization

Stream Based Data Synchronization

Klemen Verdnik

1. Introduction

1.1 Who am I, What I Do?• low-level programming enthusiast

(audio and video DSP routines, tight loop optimizations)

• embedded systems (graphic EQ with DSP, fleet management, mobile payment)

• familiar with iOS SDK since 2008

• vox.io (web / mobile sip, xmpp)

• layer.com (messaging)

• obsession with synchronization protocols

http://vox.io

http://layer.com

2. Data Synchronization

2.1 What is Data Synchronization?• Having data

consistency across two or more networked entities

2.1.1 Example

Toggle Switch App

2.1.2 How to Design the System?• Simple server

Toggle Switch App

2.1.2 How to Design the System?• Simple server • Simple client

Toggle Switch App

2.1.2 How to Design the System?• Simple server • Simple client • Simple data

structure

{ lightsOn: true }

Toggle Switch App

2.1.2 How to Design the System?

2.2 Other Use Cases• E-mail (IMAP, POP) • Messaging (iMessages, Hangouts) • Photo sharing (Photo Stream, Google Photos) • File sharing (Dropbox, iCloud Drive) • Online text editors / spreadsheet editors (Google Docs) • Multiplayer Games (Minecraft)

2.3 Types of Data Synchronization• File

synchronization


synchronization • Text / document

synchronization


synchronization • Text / document

synchronization • Data model

synchronization

2.4 Approaches to Data Synchronization

2.4.1 Absolute Synchronization (copying)

• Copying (wholesale transfer) is ok when dealing with small data-sets

(e.g. refreshing weather forecast, RSVP list ...)


• Figuring out differences between previously fetched data-sets costs CPU and memory

O(n ⋁ m)



O(n ⋁ m)

Dan Alex Blake Emily George Caroline You



O(n ⋁ m)

Dan Alex Blake Emily George Caroline You

Dan Alex Emily George Caroline You

≠

2.4.2 Relative Synchronization (changes)

• Getting data up-to-date with changesinstead of full data sets.

(a.k.a. deltas)

2.4 What are Deltas?Delta encoding is a way to describe differences

between two datasets.

2.5.1 How to Encode Deltas?insert ― adds new values to dataset

update ― updates existing values in dataset

delete ― deletes existing values from dataset

+

✔

-

• Three primitive operations

2.5.1 How to Encode Deltas?0000000: 4749 4654 2B31 0d00 0d00 9100 00b6 6257 GIFT+1........bW 0000010: 0804 0456 2c27 e5aa 7f21 f904 0000 0000 ...V,'...!...... 0000020: 002c 0000 0000 0d00 0d00 0002 318c 8f29 .,..........1..) 0000030: 3000 7986 944f 8823 260d 0feb b620 0b03 0.y..O.#&.... .. 0000040: 2e97 e1a4 0f79 920c 60a5 28e5 c452 abc6 .....y..`.(..R..

[ { type: "update", offset: 0x03, values: [ 0x38, 0x39, 0x61 ] }, { type: "insert", offset: 0x50, values: [ 0xCE, 0xE1, 0x50, 0x96, 0x89, 0x48, 0x9D, 0x02, 0x43, 0x62, 0x8D, 0x98, 0x28, 0x00, 0x00, 0x3B ] } ]

• Example on how to encode binary data changes

2.5.1 How to Encode Deltas?0000000: 4749 4638 3961 0d00 0d00 9100 00b6 6257 GIF89a........bW 0000010: 0804 0456 2c27 e5aa 7f21 f904 0000 0000 ...V,'...!...... 0000020: 002c 0000 0000 0d00 0d00 0002 318c 8f29 .,..........1..) 0000030: 3000 7986 944f 8823 260d 0feb b620 0b03 0.y..O.#&.... .. 0000040: 2e97 e1a4 0f79 920c 60a5 28e5 c452 abc6 .....y..`.(..R..



2.5.1 How to Encode Deltas?0000000: 4749 4638 3961 0d00 0d00 9100 00b6 6257 GIF89a........bW 0000010: 0804 0456 2c27 e5aa 7f21 f904 0000 0000 ...V,'...!...... 0000020: 002c 0000 0000 0d00 0d00 0002 318c 8f29 .,..........1..) 0000030: 3000 7986 944f 8823 260d 0feb b620 0b03 0.y..O.#&.... .. 0000040: 2e97 e1a4 0f79 920c 60a5 28e5 c452 abc6 .....y..`.(..R.. 0000050: cee1 5096 8948 9d02 4362 8d98 2800 003b ..P..H..Cb..(..;



2.5.1 How to Encode Deltas?

• Example on how to encode text changes

83: // 84: // Toggles the private ivar `_lightSwitchState` boolean, updates the 85: // background image, plays a sound and transmits the change over network. 86: // 87: func toggleAndSendLightSwitchState() { 88: self.lightSwitchState = !self.lightSwitchState > 89: self.lightSwitchClient.sendLightSwitchState(self.lightSwitchState) 90: }


• Example on how to encode text changes (diff patch)

83: // 84: // Toggles the private ivar `_lightSwitchState` boolean, updates the 85: // background image, plays a sound and transmits the change over network. 86: // 87: func toggleAndSendLightSwitchState() { 88: self.lightSwitchState = !self.lightSwitchState > 89: self.lightSwitchClient?.sendLightSwitchState(self.lightSwitchState) 90: }

--- 89: self.lightSwitchClient.sendLightSwitchState(self.lightSwitchState) +++ 89: self.lightSwitchClient?.sendLightSwitchState(self.lightSwitchState)


• Example on how to encode text changes (insert operation)

83: // 84: // Toggles the private ivar `_lightSwitchState` boolean, updates the 85: // background image, plays a sound and transmits the change over network. 86: // 87: func toggleAndSendLightSwitchState() { 88: self.lightSwitchState = !self.lightSwitchState > 89: self.lightSwitchClient?.sendLightSwitchState(self.lightSwitchState) 90: }

{ type: "insert", offset: 2781, values: [ "?" ] }


• Example on how to encode custom data model changes

{ guests: [ "Alex", "Blake", "Caroline", "Dan", "Emily", "George" ] }


• Example on how to encode custom data model changes

{ guests: [ "Alex", "Blake", "Caroline", "Dan", "Emily", "George" ] }

{ type: "delete", guest: [ "Blake" ] }

{ guests: [ "Alex", "Caroline", "Dan", "Emily", "George" ] }

3. Stream Based Synchronization

3.1 The Motivation

• Minimum data redundancy

3.1 The Motivation

• Speed / minimum bandwidth

3.1 The Motivation

• Fast writes = good concurrency characteristics

3.1 The Motivation

• Distributability and scalability

3.1 The Motivation

• Offline support

3.2 Stream of Mutations

• Clients with an open connection receive a live stream of events from the server

3.2 Stream of Mutations

• Example "To Do" app

3.2.1 Example (To-do List App Data-model)

• Live synchronized list of to-do tasks

public struct Todo { public class List: NSObject { private let tasks: Array<Task> = [] } }



• Task element consists of: checkbox, label and color public struct Todo {

public class List: NSObject { private let tasks: Array<Task> = [] } }

public struct Todo { public class Task: NSObject { public private(set) var identifier: NSUUID public private(set) var completed: Bool public private(set) var title: String public private(set) var label: ColorLabel public enum ColorLabel: UInt8 { case None = 0, Red, Orange, Yellow, Green, Turquoise, Blue, Purple, Pink } } }



• Task element consists of: checkbox, label and color

• Tasks can be added, edited and removed

public struct Todo { public class List: NSObject { private let tasks: Array<Task> = [] public func create(title: String, label: Task.ColorLabel) public func update(identifier: NSUUID, completed: Bool?, title: String?, label: Task.ColorLabel?) public func remove(identifier: NSUUID) } }

public struct Todo { public class Task: NSObject { public private(set) var identifier: NSUUID public private(set) var completed: Bool public private(set) var title: String public private(set) var label: ColorLabel public enum ColorLabel: UInt8 { case None = 0, Red, Orange, Yellow, Green, Turquoise, Blue, Purple, Pink } } }



• Task element consists of: checkbox, label and color

• Tasks can be added, edited and removed

3.2.2 Example (To-do List Sync Data-model)

• Todo.List user actions turn into events (!)



• Simple concrete objects describing changes

public struct Sync { public class Event: NSObject { public enum Type: UInt8 { case Insert = 0, Update, Delete }

public private(set) var type: Type public private(set) var identifier: NSUUID public private(set) var completed: Bool? public private(set) var title: String? public private(set) var label: Int? } }




• Serializable

public struct Sync { public class Event: NSObject, Serializable { public enum Type: UInt8 { case Insert = 0, Update, Delete }

public private(set) var type: Type public private(set) var identifier: NSUUID public private(set) var completed: Bool? public private(set) var title: String? public private(set) var label: Int? } }

public protocol Serializable: class { init(fromDictionary dictionary: Dictionary<String, AnyObject>) func toDictionary() -> Dictionary<String, AnyObject> }




• Serializable


• Creating new task

{ // serialized event structure type: 0, // 0 = Insert identifier: "cb55ceec-b9ae-4bd9-8783-7dbf3e9cb2cd", // client generated id completed: false, // an incomplete task title: "Buy Milk", // task description label: 0 // color tag }


• Editing an existing task

{ // event structure type: 1, // 1 = Update identifier: "cb55ceec-b9ae-4bd9-8783-7dbf3e9cb2cd", // reference to task completed: true // new state }

3.2.3 Example (To-do List Sync and Transport)

• Receive live serialized Events from the server. public protocol TransportDelegate: class {

func transport(transport: Transport, didReceiveObject object: Serializable) func transportDidConnect(transport: Transport) func transportDidDisconnect(transport: Transport) }

public struct Sync { public class Client: NSObject, TransportDelegate { public private(set) var stream: Stream = Stream() public private(set) var transport: Transport public private(set) var todoList: Todo.List public private(set) var publishedEvents: Array<Event> = []

private func publish(event: Event) -> Bool public func transport(transport: Transport, didReceiveObject object: Serializable) } }


• Receive live serialized Events from the server.

• Send serialized Events to server.

public protocol TransportDelegate: class { func transport(transport: Transport, didReceiveObject object: Serializable) func transportDidConnect(transport: Transport) func transportDidDisconnect(transport: Transport) }

public struct Sync { public class Client: NSObject, TransportDelegate { public private(set) var stream: Stream = Stream() public private(set) var transport: Transport public private(set) var todoList: Todo.List public private(set) var publishedEvents: Array<Event> = []

private func publish(event: Event) -> Bool public func transport(transport: Transport, didReceiveObject object: Serializable) } }


• Receive live serialized Events from the server.

• Send serialized Events to server.

3.3 Let the Streaming Begin

• Data consistent, as long as clients remain connected


• Missing out on events puts the client out-of-sync


• Missing out on events puts the client out-of-sync

{ // event structure type: 1, identifier: "cb55ceec-b9ae-4bd9-8783-7dbf3e9cb2cd", completed: true }

3.3 Let the Streaming Begin• Data consistent, as long as clients remain

connected • Missing out on events puts the client out-of-sync • Clients can recover from out-of-sync state • Server's responsibility beside broadcasting should

also be preserving the events

3.4 Persistent Stream

3.4 Persistent Stream• Think of it as a linear magnetic tape, or as a storage

with a WORM behavior

• Append only

• Immutable events

• Journal of all the events that have happened

3.4 Persistent Stream

• Always copy all the events? (too expensive)

• Integrity check by hashing events? (only detects mismatches)

How does a client know if it's got all the events?

3.5 Event Discovery

3.5 Event Discovery

• Sequencing Events on server

3.5 Event Discovery

• Sequencing Events on server

public struct Sync { public class Event: NSObject { public private(set) var seq: Int? public private(set) var type: Type public private(set) var identifier: NSUUID? public private(set) var completed: Bool? public private(set) var title: String? public private(set) var label: Int? } }

3.5 Event Discovery

• Sequencing Events on server • Sequence is a linear function f(x)=x reproducible on client

3.5 Event Discovery

• Client only needs to know the seq value of the last event f(x<12)=x

3.5 Event Discovery


• Figuring out missing events by subtracting the set of seqs

3.5 Event Discovery// Seq values pulled from all events the client has. // [ 0, 1, 2, 10, 11, 12 ] let seqsOfEvents: Set = events.map({ $0.seq })

// Calculated sequence ranging from 0 to 12. // [ 0, 1, 2, 3 ... 12 ] let seqsOfAllEvents: Set = [Int](0...12)

// Diffed set of seq values. // [ 3, 4, 5, 6, 7, 8, 9 ] let seqsOfMissingEvents: Set = seqOfAllEvents.subtract(seqOfEvents)


• Figuring out missing events by subtracting the set of seqs

4. Event and Model ReconciliationOutbound and inbound reconciliation

4.1 Outbound Reconciliation

• Turning user actions (model changes) into Events



public class List: NSObject { public func create(title: String, label: Task.ColorLabel) -> Sync.Event public func update(identifier: NSUUID, completed: Bool?, title: String?, label: Task.ColorLabel?) -> Sync.Event? public func remove(identifier: NSUUID) -> Sync.Event? }



let todoList = List() let event = todoList.create("Buy Milk", label: Task.ColorLabel.None) print("event: '\(event)", event)

// event: { // type: 0, // 0 = Insert // identifier: "cb55ceec-b9ae-4bd9-8783-7dbf3e9cb2cd", // client generated id // completed: false, // an incomplete task // title: "Buy milk", // task description // label: 0 // task without a label // }


• Publishing events

let todoList = List() let event = todoList.create("Buy Milk", label: Task.ColorLabel.None) print("event: '\(event)", event)

// event: { // type: 0, // 0 = Insert // identifier: "cb55ceec-b9ae-4bd9-8783-7dbf3e9cb2cd", // client generated id // completed: false, // an incomplete task // title: "Buy milk", // task description // label: 0 // task without a label // }

// Sends the event to the stream over the network. self.syncClient.publish(event)

4.2 Inbound Reconciliation

• Apply Events onto the model

4.2 Inbound Reconciliation

• Apply Events onto the model

public class List: NSObject { private func apply(event: Sync.Event) -> Bool { switch event.type { case .Insert: // Task creation let task = Task(identifier: event.identifier, completed: event.completed!, title: event.title!, label: Task.ColorLabel(rawValue: event.label!)!) self.tasks.append(task) case .Update: // Task updates let task = self.task(event.identifier) if task == nil { return false } task!.update(event.completed!, title: event.title!, label: Task.ColorLabel(rawValue: event.label!)!) case .Delete: // Task removal if !self.removeTask(event.identifier) { return false } } return true }

}

4.3 Offline Support

• Events generated offline have to be published eventually

4.3 Offline Support

• Queue generated events; drain queue for publication

✔

4.3 Offline Support

• Generating redundant events while offline

4.4 Reducing the Edit Distance

• Events describing the same mutation


• Causes stream pollution • Increases the edit distance


1. Insert Event merges withUpdate Events → single Insert Event

2. Update Event merge withthe rest of Update Events → single Update Event

3. Last Update Event defines final state.

4. Delete Event clobbers other Event types

Simple set of rules when queueing:

4.4 Reducing the Edit Distancepublic struct Sync { public class Event: NSObject, Serializable { var mergedEvents = Array<Event>() for oldEvent in events.reverse() { if oldEvent.identifier != self.identifier { // Event not mergeable, due to the identifier mismatch. mergedEvents.append(oldEvent) continue } else if self.type == Type.Delete { // Rule #4 self.reset() self.type = Type.Delete } else if self.type == Type.Update && (oldEvent.type == Type.Insert || oldEvent.type == Type.Update) { // Rule #1, #2, #3 self.completed = self.completed ?? oldEvent.completed self.title = self.title ?? oldEvent.title self.label = self.label ?? oldEvent.label } } mergedEvents.append(self) return mergedEvents } }

4.5 Conflict Resolution• Concurrent systems experience conflicts when two

or more nodes (clients) work on the same resource at the same time.

4.5 Conflict Resolution• Concurrent systems experience conflicts when two

or more nodes (clients) work on the same resource at the same time.

• Example: a client deletes a Todo Task before another client tries to mutate it.

4.5 Conflict ResolutionPossible conflict resolutions:

• Bring the deleted task back (last writer wins)

• Deleted task stays deleted (first writer wins)

• Ask the User what to do? (requires user interaction)

5. Order of Events

5. Order of EventsEvent sequence dictates the order they were written to stream this puts the Events in total order

• Task objects will be in the exact same order, defined bythe Event.seq

• Task mutations will be applied in the same manner on all clients

5. Order of Events (total order)

• Queued events must be published in batches

5.1 Total Order (sequential writes)

• Synchronized sequential writes block other clients from writing

5.1 Total Order (sequential writes)

• Synchronized sequential writes block other clients from writing - violates our fast concurrent writes requirement

serial writes

concurrent writes

5.1 Total Order (offline support)

• Both clients online


• Left client loses connection


• Offline client adds more To-do tasks to the list


• Online client also adds a Todo task to the list


• Left client comes back online ― events generated offline get published and fall at the end (higher seq values)

5.2 Causal OrderCauses must precede their effects - effects come after causes,

and never before

5.2 Causal OrderCauses must precede their effects - effects come after causes,

and never before

cause

effect

5.2 Causal Order• Generated Event is an effect caused by user taking action /

responding to the UI.

• Events should be reconciled in the same order as they were generated by clients.

• Events should be applied onto the app model under the same conditions as it was when author generated the events.

• Total order cannot guarantee Events will be written to stream in the same order they were generated.

5.2.1 Order Based on TimestampsClient B's events are written before Client A's, even though Client A generated them first.

Encoding local time with events.

5.2.1 Order Based on Timestamps

Sorting events based on the embedded timestamp.

5.2.1 Order Based on Timestamps

5.2.1 Order Based on Timestamps• No guarantee

time will be the same on all devices

• Clock skew

• Manual override

5.2.2 Version Vectors• Reconstructing Events' order as it was perceived by the author

based on happened-before information.

• Provides causality-tracking basic principle in some optimistic (lazy) replication algorithms.

• Allows the client to operate independently from the server.

• When all clients eventually publish their events, it brings other online clients into a consistent state eventual consistency.

5.2.2 Version VectorsHow to encode happened-before information?

public struct Sync { public class Event: NSObject { public private(set) var seq: Int? public private(set) var type: Type public private(set) var identifier: NSUUID? public private(set) var completed: Bool? public private(set) var title: String? public private(set) var label: Int? } }

5.2.2 Version Vectors

1. Information of what's the last seen event - event.seq

How to encode happened-before information?

public struct Sync { public class Event: NSObject { public private(set) var seq: Int? public private(set) var precedingSeq: Int public private(set) var type: Type public private(set) var identifier: NSUUID? public private(set) var completed: Bool? public private(set) var title: String? public private(set) var label: Int? } }



2. Keep unpublished events in order - event.clientSeq

How to encode happened-before information?

public struct Sync { public class Event: NSObject { public private(set) var seq: Int? public private(set) var precedingSeq: Int public private(set) var clientSeq: Int public private(set) var type: Type public private(set) var identifier: NSUUID? public private(set) var completed: Bool? public private(set) var title: String? public private(set) var label: Int? } }



2. Keep unpublished events in order - event.clientSeq

3. Order

How to encode happened-before information?public struct Sync { public class Event: NSObject { /// Event sorting closure static public let causalOrder = { (e1: Event, e2: Event) -> Bool in if e1.precedingSeq == e2.precedingSeq { return e1.clientSeq < e2.clientSeq } return e1.precedingSeq < e2.precedingSeq }

public private(set) var seq: Int? public private(set) var precedingSeq: Int public private(set) var clientSeq: Int public private(set) var type: Type public private(set) var identifier: NSUUID? public private(set) var completed: Bool? public private(set) var title: String? public private(set) var label: Int? } }


public struct Sync { public class Event: NSObject, Serializable { var mergedEvents = Array<Event>() for oldEvent in events.sort(Event.causalOrder) { if oldEvent.identifier != self.identifier { // etc...

public struct Todo { public class List: NSObject, ModelReconciler { public func apply(events: Array<Sync.Event>) -> Bool { for event in events.sort(Sync.Event.causalOrder) { let success = self.apply(event) // etc...

Minor adjustment in outbound / inbound reconciliation:


• Newly published events generated offline are ordered by their causality.


• Concurrent writes - no need for batched writes anymore, due to clientSeq.


• Concurrent writes - events can be written with undetermined order; order can be reconstructed on clients

serial writes

concurrent writes

6. Advantages

6. Advantages• Shared source - minimal redundancy

6. Advantages• Shared source - minimal redundancy • Lightweight data structure - fast delivery

6. Advantages• Shared source - minimal redundancy • Lightweight data structure - fast delivery • Minimal server logic (low CPU)

6. Advantages• Shared source - minimal redundancy • Lightweight data structure - fast delivery • Minimal server logic (low CPU) • Short writes - high concurrency

6. Advantages• Shared source - minimal redundancy • Lightweight data structure - fast delivery • Minimal server logic (low CPU) • Short writes - high concurrency • Scalable / distributable

6. Advantages• Shared source - minimal redundancy • Lightweight data structure - fast delivery • Minimal server logic (low CPU) • Short writes - high concurrency • Scalable / distributable • Offline support

7. Disadvantages

7. Disadvantages• Server simplicity = client complexity

7. Disadvantages• Server simplicity = client complexity • Rogue clients = stream pollution

7. Disadvantages• Server simplicity = client complexity • Rogue clients = stream pollution • Clients must read full stream

7. Disadvantages• Server simplicity = client complexity • Rogue clients = stream pollution • Clients must read full stream • Partial sync difficult to implement

END_OF_STREAMquestions?

[email protected]/chipxsd

@chipxsd

stream-based data synchronization

Engineering