Introduction toCouchbase Sync Gateway

Andrew Reslan | Senior Software Engineer, Couchbase

Why Sync Gateway?

Why Sync Gateway?

Couchbase Server ?

Why Sync Gateway?

?

What Sync Gateway Is

• Intermediary between mobile apps and Couchbase server

• Communicates data by replication- Apps can locally store a data set from your server- Or, apps can create and share their own data sets

• Facilitates bidirectional, multi-master sync

Intermediary Between Mobile and Server

• Speaks mobile replication protocol

• Stores documents in Couchbase Server

Lets Mobile Apps Store Data Locally

• Authenticates users

• Authorizes document access

• Routes documents to users

Facilitates Bidirectional Sync

• Tracks document sync metadata- Bucket history- Document revisions- Supports conflicts via revision trees

• Validates document updates

Installing and Configuring

Installing Sync Gateway

• It’s just a single binary!

• Several options:- Download from www.couchbase.com/download- Build from github.com/couchbase/sync_gateway- Install AMI package on AWS

Starting Sync Gateway

$ sync_gateway

15:54:25.167157 ==== Couchbase Sync Gateway/() ====15:54:25.167650 Configured Go to use all 8 CPUs; setenv GOMAXPROCS to override this15:54:25.167679 Opening db /sync_gateway as bucket "sync_gateway", pool "default", server <walrus:>15:54:25.168257 Opening Walrus database sync_gateway on <walrus:>15:54:25.170553 Using default sync function 'channel(doc.channels)' for database "sync_gateway"15:54:25.170565 Starting profile server on 15:54:25.170570 Starting admin server on 127.0.0.1:498515:54:25.174619 Starting server on :4984 ...

Sync URL: http://hostname:4984/sync_gateway

Playing with the REST API

$ curl http://localhost:4985/sync_gateway/{“committed_update_seq”:0, “compact_running":false, “db_name":"sync_gateway", “disk_format_version":0, “instance_start_time":1411944865170109, “purge_seq":0, "update_seq":0}

$ curl http://localhost:4984/sync_gateway{"error":"Unauthorized","reason":"Login required”}

$ curl :4985/sync_gateway/_user/GUEST{“name”:"GUEST","all_channels":[],"disabled":true}

Configuring

$ sync_gateway --helpUsage of bin/sync_gateway: -adminInterface="127.0.0.1:4985": Address to bind admin interface to -bucket="sync_gateway": Name of bucket -configServer="": URL of server that can return database configs -dbname="": Name of CouchDB database (defaults to name of bucket) -deploymentID="": Customer/project identifier for stats reporting -interface=":4984": Address to bind to -log="": Log keywords, comma separated^C

$ sync_gateway myConfigFile.json

Configuration Files

{"databases": {

"my_cool_app": {"server": "http://localhost:8091","bucket": "sync_gateway","users": {

"GUEST": {"disabled": false, "admin_channels": ["*"] }}

}}

}

Couchbase Server URL

Enable guest

access

Access to all documents

No-auth account

Public database

name

Creating the sync_gateway Bucket

Starting the Gateway

$ curl http://localhost:4984/my_cool_app/{“committed_update_seq”:0, “compact_running":false, “db_name":"my_cool_app", “disk_format_version":0, “instance_start_time":1411944865170109, “purge_seq":0, "update_seq":0}

$

$ sync_gateway myConfigFile.json17:14:52.845218 Enabling logging: [HTTP+]17:14:52.845635 ==== Couchbase Sync Gateway/() ====17:14:52.845730 Configured Go to use all 8 CPUs; setenv GOMAXPROCS to override this17:14:52.845757 Opening db /my_cool_app as bucket "sync_gateway", pool "default", server <http://localhost:8091>17:14:52.846316 Opening Couchbase database sync_gateway on <http://localhost:8091>17:14:52.908102 Starting admin server on 127.0.0.1:498517:14:52.911783 Starting server on :4984 ...

Meanwhile, In The Client Code…

// The remote database URL to sync with.#define kServerDbURL @"http://example.com/my_cool_app/"

NSURL* serverDbURL = [NSURL URLWithString: kServerDbURL];_pull = [database createPullReplication: serverDbURL];_push = [database createPushReplication: serverDbURL];_pull.continuous = _push.continuous = YES;[_push start];[_pull start];

Progress Report

• Mission accomplished:- The Sync Gateway is running- Backed by a new Couchbase Server bucket- Client apps can replicate with it

• TBD:- User authentication- Document authorization- Using existing data sets

User Authentication

User Accounts

• Sync Gateway manages mobile user accounts- User database- Admin REST API

• Authentication Mechanisms- HTTP Basic- Session cookie- Facebook- Custom (via app server)

User Admin API

• /dbname/_user/username- Create user: POST or PUT- Retrieve user: GET- Update user: PUT- Delete user: DELETE

User Attributes

• ID- Immutable

• Password- Write-only- Securely hashed using bcrypt

• Access privileges- Channels- Role membership

Channels and Sync Functions

What Are Channels?

• Tags attached to documents

• Message queues of document updates

• All of the above

Sync Function

• A JavaScript function

• Given in the Gateway config file

• Input: New & old document revisions

• Capabilities:- Enforce document validity (schema)- Enforce specific user ID or role membership- Tag document with channels- Grant users access to channels

Document Routing via Channels

App's Sync Function

publ

ic

proj

2

proj1

proj2

public

sales_Q1

jens

Jens’ phone

Anil’s phone

Proj2Press

Release

function(doc, oldDoc) {channel(doc.projectID);if (doc.accessLevel <

1) channel(“public”);...}

Let's Write a Sync Function!

function(doc, oldDoc) {

New document contents

Previous document contents

{"_id": "ACA9083F", "owner": "alice", "readers": ["bob",…], "category": "riddle","text": "Why is a…"}

{"_id": "ACA9083F", "owner": "alice", "readers": ["bob",…], "category": "joke"}

Enforcing Document Validity

function(doc, oldDoc) {if (!doc.owner)

throw({forbidden: "Missing owner"});if (oldDoc && doc.owner != oldDoc.owner)

throw({forbidden: "Owner changed"});

{"_id": "ACA9083F", "owner": "alice", "readers": ["bob",…], "category": "riddle", "text": "Why is a…"}

"Every document must have an owner.""The owner property must be immutable."

Disclaimer: Slightly simplified(not considering deletions)

Enforcing User Identity

function(doc, oldDoc) {if (!doc.owner)

throw({forbidden: "Missing owner"});if (oldDoc && doc.owner != oldDoc.owner)

throw({forbidden: "Owner changed"});

requireUser(doc.owner);requireRole("creators");

"The owner is the user who created the document.""Only those with the 'creator' role can create documents."

{"_id": "ACA9083F", "owner": "alice", "readers": ["bob",…], "category": "riddle", "text": "Why is a…"}

Tagging Documents With Channels

function(doc, oldDoc) {if (!doc.owner)

throw({forbidden: "Missing owner"});if (oldDoc && doc.owner != oldDoc.owner)

throw({forbidden: "Owner changed"});

requireUser(doc.owner);requireRole("creators");

channel(doc.category);

"The document will be tagged by its category."

{"_id": "ACA9083F", "owner": "alice", "readers": ["bob",…], "category": "riddle", "text": "Why is a…"}

Granting User Access To Channels

function(doc, oldDoc) {if (!doc.owner)

throw({forbidden: "Missing owner"});if (oldDoc && doc.owner != oldDoc.owner)

throw({forbidden: "Owner changed"});

requireUser(doc.owner);requireRole("creators");

channel(doc.category);channel("readers_" + doc._id)access(doc.readers, "readers_" + doc._id);

} "Only users in the 'readers' list can view the document."

{"_id": "ACA9083F", "owner": "alice", "readers": ["bob",…], "category": "riddle", "text": "Why is a…"}

Progress Report

• Mission accomplished:- The Sync Gateway is running- Backed by a new Couchbase Server bucket- Client apps can replicate with it- User authentication- Document authorization

• TBD:- Using existing data sets

Using Existing Database Buckets

What If You Already Have a Bucket?

• Loyal Couchbase customer!

• Existing data set

• Now want to share data withmobile clients

What If You Already Have a Bucket?

• Problem: Sync Gateway needs toown its bucket- Replication metadata- User accounts- etc.

• Do not point Sync Gateway at youroriginal bucket!- It'll add internal fields to your docs- It'll be confused when you modify docs

Bucket Shadowing To The Rescue

• Solution: Create a Shadow Bucket for the Gateway- Create a new empty bucket- Point the Gateway's config at it- Tell the config to shadow your original bucket

Bucket Shadowing Configuration

{"databases": {

"my_cool_app": {"server": "http://localhost:8091","bucket": "sync_gateway","shadow": {

"server": "http://localhost:8091","bucket": "megacorp_database"

}}

}}

Original existing bucket

New bucket for Gateway

Progress Report

• Mission accomplished:- The Sync Gateway is running- Backed by a new Couchbase Server bucket- Client apps can replicate with it- User authentication- Document authorization- Using existing data sets

✅