webinar- couchbase mobile deep dive
TRANSCRIPT
©2009 Helder da Rocha
Architecture of Couchbase Lite and & Sync Gateway
• Document data store Mul?-‐Version Concurrency Control (MVCC)
• Replica=on protocol Conflict handling
• Querying/filtering Map/reduce views
• Client APIs REST
Na?ve
Document Store
Document Database
• Persistent map from IDs → documents
• where documents are JSON objects
• …plus aLachments Extra set of proper?es mapping name → {binary data, MIME type}
• …with mul=ple revisions
Mul=-‐Version Concurrency Control
• Each document has a revision history
• Each revision’s ID has a genera=on number and a digest e.g. 12-‐c8c993737bf3e800b0a3d2e864c2039d
• Dele=ons are just special revisions “_deleted”: true
• Every doc update must specify its parent revision ID
Sequence Index
• Every revision added gets a sequence number
• Sequence numbers are assigned consecu=vely
• Local to a single database
• Documents are indexed by sequence number Abstract ?meline of the database’s history
Allows asking “what changed since sequence ____?”
Replica=on
What Replica=on Does
• Transfers changed documents from a source to a target database
• Incremental
• Asynchronous
• Peer-‐to-‐peer
• Semi-‐op=mized Doesn’t send any revision that the target already has
Doesn’t send any adachment version that the target already has
But does send en?re JSON docs & adachments (no delta compression)
Replica=on Conflicts
• Conflicts are inevitable, per the CAP Theorem
• Replica=ng conflic=ng revision histories produces a tree
• The document model allows for conflicts A conflict is a valid state of a document
• Conflict resolu=on is the app’s job Only the app needs to care about resolving conflicts
App can use any logic it wants to pick winners or merge changes
Resolu?on creates new revisions that are replicated
Jens — This is too
confusing. We really
need a diagram
here! Thx.
Replica=on Conflicts Cause Branches
1-‐abcd 2-‐c124
1-‐abcd 2-‐c124 3-‐beef 1-‐abcd 2-‐c124 3-‐fade
1-‐abcd 2-‐c124
3-‐beef
3-‐fade
4-‐dead
4-‐feed
Resolving a Conflict
1-‐abcd 2-‐c124
3-‐beef
3-‐fade
Replica=on Conflicts
• Conflicts are inevitable, per the CAP Theorem
• Replica=ng conflic=ng revision histories produces a tree
• The document model allows for conflicts A conflict is a valid state of a document
• Conflict resolu=on is the app’s job Only the app needs to care about resolving conflicts
App can use any logic it wants to pick winners or merge changes
Resolu?on creates new revisions that are replicated
Replica=on and Peer-‐to-‐Peer
• It’s more than just push/pull to a server
• Any two databases can replicate if they can connect Couchbase Lite’s “Listener” component allows connec?ons
• Arbitrary topologies (trees, meshes) are supported
Replica=on Protocol
• Extension of REST API
• One ac=ve partner (client), one passive (server)
• Ac=ve partner can push or pull
• State (checkpoint) is stored redundantly on both sides
Pull Algorithm
• Verify local & remote checkpoints
• Request the changes since checkpointed remote sequence → array of {sequence, doc-‐id, rev-‐id} tuples
• Determine which document revisions do not exist locally
• Download leaf revision(s) of each such document Including revision history
Skipping unmodified adachments
• Update the checkpoints
• In con1nuous mode, keep monitoring remote changes
Push Algorithm
• Verify local & remote checkpoints
• Send IDs of revisions made since checkpoint’s local sequence Server responds with the subset that it doesn’t have
• Upload the leaf revision(s) of each such document Including revision history
Skipping unmodified adachments
• Update checkpoints
• In con1nuous mode, keep monitoring local changes
Note to self: verify against https://github.com/couchbase/couchbase-lite-ios/wiki/Replication-Algorithm
Filtered Replica=on
• Implemented in Couchbase Lite, CouchDB, PouchDB, etc.
• App-‐defined filter func=on can accept or skip each outgoing revision
• Always runs on source side of replica=on Push: on client side
Pull: on server side
• Simple, but not super scaleable
Channel-‐Based Replica=on
• Implemented by Couchbase Sync Gateway
• Documents are tagged with channel IDs
• User accounts have access to sets of channels
• Documents are indexed by channel
• Pull replica=ons efficiently look up docs in available channels
Tell people to watch
the Sync Gateway
talk next week for
more details!
Map/Reduce Views
Anatomy of a View
• A view belongs to a database, has a: Name
Map func?on
Reduce func?on (op?onal)
Persistent index
Checkpoint sequence
• Views exist in: Couchbase Lite
Couchbase Server
not Sync Gateway
Also CouchDB, PouchDB, etc.FYI: Last week’s iOS
webinar had better
slides explaining this
stuff.
Indexing a View
• For each document updated since checkpoint sequence: 1. Remove all index rows previously generated by that document
2. Call map func?on with current document proper?es as parameter
3. Each ?me emit() is called, add {key, value, docID} tuple to the index
• Save the current sequence number as the checkpoint
• The keys in the index are sorted as JSON
Querying a View
• Is its checkpoint sequence the database’s latest sequence? If not, update the index (as per previous slide)
This can be suppressed by sejng query’s IndexUpdateMode
• Fetch a range of {key, value, docID} tuples from the index Honoring startKey, endKey, skip, limit, reverse
Client APIs
REST API — The Classic
• Originally defined by Apache CouchDB
• Supported by Sync Gateway (with slight differences) Server side of replica?on
API for app servers
• Supported by Couchbase Lite Listener For JavaScript app code (e.g. PhoneGap)
Server side of peer-‐to-‐peer replica?on
REST API
• Document access (CRUD): PUT / GET / DELETE
/dbname/docid
• Query a view: POST /dbname/_design/name/_view/name
• Replicate: POST /dbname/_replicate
Na=ve APIs by Plalorm
• Objec=ve-‐C: iOS, Mac OS X
• Java: Android, Linux
• C#: Windows, Windows Mobile, Mono BETA
• Common API design across languages We have a single carefully designed language-‐neutral API spec
Matching class names, method names, seman?cs
Slight differences, to remain idioma?c
Point them to the
“Development with
Android / iOS”
webinars.
Na=ve API: Key Classes
• Manager
• Database
• Document
• View
• Query
• Replica=on
Oops, that’s all I wrote. Stall for time — maybe answer some questions?
THE END
www.couchbase.com/webinars
developer.couchbase.com/mobile/