differential sync and json patch @ springone2gx 2014
DESCRIPTION
Craig Walls and I presented a new approach to client-server communication at SpringOne2GX 2014. Abstract: The world of client-server has changed. The traditional application of REST is no longer the best fit. We're depolying applications into a world where users expect responsive UIs, on all their devices, even while disconnected. We're deploying into a world where connection latency, mobile radio usage and battery life have become primary concerns. Differential Synchronization (DS) is an algorithm that syncs data across N parties, even in the face of dropped connections, offline devices, etc. It makes more efficient use of connections by batching and sending only changes, in both directions, from client to server and from server to client. We’ll look at how it can be used with JSON Patch to synchronize application data between clients and servers over HTTP Patch, WebSocket, and STOMP, and how it can be integrated into the Spring ecosystem.TRANSCRIPT
![Page 1: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/1.jpg)
© 2014 SpringOne 2GX. All rights reserved. Do not distribute without permission.
Efficient Client Communication with Differential Synchronization and JSON Patch
By Brian Cavalier and Craig Walls
![Page 2: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/2.jpg)
What’s the problem with REST?
![Page 3: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/3.jpg)
Nothing, REST is awesome
![Page 4: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/4.jpg)
What’s the problem with how we typically use REST in practice?
![Page 5: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/5.jpg)
Agenda
• Motivation • Differential Synchronization (DS) • JSON Patch • DS + JSON Patch • DS w/Spring and JavaScript • The future
![Page 6: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/6.jpg)
What’s the problem with how we typically use REST in practice?
![Page 7: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/7.jpg)
Isn’t “typical REST” good enough?
• “Typical REST”: 1 request per entity per operation type • create 2 entities, update 3, delete 1 = 6 requests
• Expensive for mobile: connection latency, battery, data ($$$) • Doesn’t exploit return payload
![Page 8: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/8.jpg)
Motivation: Goals
• More efficient data transfer • More efficient use of mobile radios, networks and batteries • Take advantage of WebSocket & messaging • Data synchronization with multiple clients • Offline / disconnected
![Page 9: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/9.jpg)
How can we get data from a Spring back-end all the way to
the pixels more efficiently?
![Page 10: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/10.jpg)
Differential Synchronization
![Page 11: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/11.jpg)
Differential Sync
• Algorithm for syncing N copies of a document • Potentially supports any kind of document/data
• As long as diff & patch algorithms are available • Text, DOM, JSON
• Nice properties • Efficient: Transfer only differences • Symmetrical: same algorithm at each node • Recoverable: disconnected/offline clients, lost messages
• Published by Neil Fraser in 2009 • https://neil.fraser.name/writing/sync/
![Page 12: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/12.jpg)
Differential Sync
doc docshadow
diff
diff
patch
patch
live edits
live edits
![Page 13: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/13.jpg)
Differential Sync
doc shadow docshadow
diff
diff
patch
patch
live edits
live edits
![Page 14: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/14.jpg)
Differential Sync
• Cooperative synchronization loop • Distributed or local • Uses diff & patch algorithms
![Page 15: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/15.jpg)
JSON Patch
![Page 16: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/16.jpg)
JSON Patch
• JSON-formatted Patch for structured documents • RFC 6902, plus related JSON Pointer RFC 6901
• https://tools.ietf.org/html/rfc6902 • https://tools.ietf.org/html/rfc6901
• Suitable for sending via HTTP Patch • Defines operations, format, algorithm, and mime type
• application/json-patch+json • Can coexist w/handlers at same url via Content-Type routing • Does not define diff algorithm • Sensible requirement: patch(diff(a, b), a) === b
![Page 17: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/17.jpg)
JSON Patch
json![{"value":"a"},{"value":"b"},{"value":"c"}]!
+ patch!
[{"op":"add","path":"/3","value":{"value":"d"}},!{“op":"remove","path":"/1"}]!
= new json!
[{"value":"a"},{"value":"c"},{"value":"d"}]
![Page 18: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/18.jpg)
JSON Patch
json!{"name":{"first":"Brian","last":"Cavalier"},"occupation":"JavaScript
Engineer"}!
+ patch!
[{"op":"replace","path":"/occupation","value":"JavaScript Ranger"}]!
= new json!
{"name":{"first":"Brian","last":"Cavalier"},"occupation":"JavaScript Ranger"}
![Page 19: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/19.jpg)
JSON Patch
• Moves the operation type inside request payload • create 2 entities, update 3, delete 1 = 1 request
• Moves the identifier inside request payload • Potentially patch many entity types in a single request
• Patches are atomic • If any part of a patch fails, whole patch must fail (as per RFC) • Think: data integrity
![Page 20: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/20.jpg)
“Typical REST”
POST /todos {“description”: “Try JSON Patch”, “complete”: false} PUT /todos/1 {“description”: “…”, “complete”: true} PATCH /todos/2 {“complete”: true} DELETE /todos/3
![Page 21: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/21.jpg)
JSON Patch
PATCH /todos [ {“op”: “add”, “path”: “-“, “value”: { “description”: “Try JSON Patch”, “complete”: true}}, {“op”: “replace”, “path”: “/1”, “value”: { “description”: “…”, “complete”: true}}, {“op”: “replace”, “path”: “/2/complete”, “value”: true}}, {“op”: “remove”, “path”: “/3”}} ]
![Page 22: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/22.jpg)
“Typical REST”
POST /person { “name”: “Brian” } PUT /todos/1 { “description”: “…”, “complete”: true } PATCH /meetings/2 { “agenda”: “…” } DELETE /todos/3
![Page 23: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/23.jpg)
Pie in the sky JSON Patch
PATCH / [ {“op”: “add”, “path”: “/person/-“, “value”: { “name”: “Brian” }}, {“op”: “replace”, “path”: “/todos/1”, “value”: { “description”: “…”, “complete”: true }}, {“op”: “replace”, “path”: “/meetings/2/agenda”, “value”: “…” }}, {“op”: “remove”, “path”: “/todos/3” }} ]
![Page 24: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/24.jpg)
JSON Patch
• Reduce requests • “Typical” REST:
• # Requests = Entity type x Operation type • JSON Patch + HTTP Patch
• # Requests = 1 • Reduce payload size: Transfer only deltas
![Page 25: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/25.jpg)
Demo
![Page 26: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/26.jpg)
Introducing jiff.js
• JavaScript library for JSON diff & patch • https://github.com/cujojs/jiff
• Diffs arbitrary JSON or JavaScript object graphs • inc. objects containing arrays containing objects containing…
• Patches arbitrary JSON or JavaScript object graphs atomically • Supports advanced features: inverse patches, rebase, contextual
(“smart”) patching
![Page 27: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/27.jpg)
jiff.js
var jiff = require(‘jiff’); var rest = require(‘rest’); !var changedData = jiff.clone(data); !// … data changes via user interactions !var patch = jiff.diff(data, changedData); !rest({ method: ‘PATCH’, entity: patch });
![Page 28: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/28.jpg)
Hmmmm …. Differential synchronization requires diff and patch algorithms
JSON Patch defines a patch format and algorithm
![Page 29: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/29.jpg)
Differential Sync + JSON Patch What if we put these two things together to synchronize
structured data?
![Page 30: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/30.jpg)
Differential Sync
data model shadow data
modelshadow
diff
diff
patch
patch
changeschanges
JSON Patch
JSON Patch
Spring ServerClient (web browser, phone, etc.)
![Page 31: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/31.jpg)
Demo
![Page 32: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/32.jpg)
Differential Synchronization in Server-Side Spring
![Page 33: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/33.jpg)
Applying JSON Patch to the Java-based Domain
![Page 34: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/34.jpg)
Path to SpEL
JSON Patch Path SpEL
/0 [0]
/complete .complete
/1/description [1].description
/clients/3/address/zip .clients[3].address.zip
![Page 35: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/35.jpg)
Challenges with JSON Patch and Java
• How do you “remove” or “move” a property? • How do you “move” a list item to a different index? • How do you “add” a list item to a specific index? • How do you avoid saving the entire list when patching a list? • How do you delete an item as the result of a “delete” op? • What if a “remove” is only intended for a particular view?
![Page 36: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/36.jpg)
Introducing Spring Sync
• GitHub: http://github.com/spring-projects/spring-sync • Maven/Gradle: org.springframework.sync:spring-sync:0.5.0.BUILD-
SNAPSHOT • JsonPatch
• Applies a JSON Patch to a Java object graph
• DiffSync • Applies Differential Synchronization algorithm (leveraging JsonPatch)
• DiffSyncController • Handles PATCH requests for “application/json-patch+json” • Returns a JSON Patch to update client
• @EnableDifferentialSynchronization
![Page 37: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/37.jpg)
Enabling Spring Sync
@Configuration @EnableDifferentialSynchronization public class DiffSyncConfig extends DiffSyncConfigurerAdapter { ! @Autowired private PagingAndSortingRepository<Todo, Long> repo; @Override public void addPersistenceCallbacks( PersistenceCallbackRegistry registry) { ! registry.addPersistenceCallback( new JpaPersistenceCallback<Todo>(repo, Todo.class)); ! } }
![Page 38: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/38.jpg)
Differential Synchronization in JavaScript
![Page 39: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/39.jpg)
Network
Differential Sync in JavaScript
DOM User Interface
JavaScript objects, arrays, etc
Spring Server
patchpatch
patch patch
![Page 40: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/40.jpg)
Differential Sync in JavaScript
• Synchronize from the Spring data model to the pixels • Decouple change frequency from communication frequency:
• fast sync = responsive, but network/resource intense • slow sync = slower UI updates, but less network/resource
intense • Current Status
• Incubator JavaScript implementation based on cujojs/jiff.js
![Page 41: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/41.jpg)
When you have a system based on patches, you can do some interesting things
![Page 42: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/42.jpg)
Streaming changes w/WebSocket
42
![Page 43: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/43.jpg)
Demo
![Page 44: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/44.jpg)
Patch Algebra
44
![Page 45: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/45.jpg)
Patch Algebra
• Inverse patches • think Undo & Redo with no application specific logic
• Merge and Rebase • Apply parallel changes from multiple parties without locking
• jiff.js supports inverse, rebase
![Page 46: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/46.jpg)
Demo
![Page 47: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/47.jpg)
Challenges
• DS in an Entity-oriented world • What constitutes a document?
• Each participant must maintain a shadow copy of the document • Lists and arrays are tricky • Conflict resolution (not a big deal in practice?)
• Hypothesis: Conflicts no more likely to occur than REST • DS in itself does not solve conflict resolution, but neither does
REST
![Page 48: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/48.jpg)
Today
• Spring Sync • http://github.com/spring-projects/spring-sync • org.springframework.sync:spring-sync:0.5.0.BUILD-SNAPSHOT
• JSON Patch + diff in JavaScript • https://github.com/cujojs/jiff
• Experimental Spring support for Differential Sync and JSON Patch over HTTP Patch
• https://github.com/royclarkson/spring-rest-todos
![Page 49: Differential Sync and JSON Patch @ SpringOne2GX 2014](https://reader033.vdocuments.mx/reader033/viewer/2022042713/547e7fb1b4af9fbe158b57ca/html5/thumbnails/49.jpg)
The Future
• Continue evolution of Spring Sync and JavaScript DS implementations
• Further Integrate with Spring ecosystem • Messaging & WebSocket • Smart patching
• Offline/disconnected client support • Guidance (when to use it, how to tune it, etc.)