couchbase singapore meetup #2: why developing with couchbase is easy !!

Why Developing with Couchbase is Easy

Karthik Babu SekarSolutionsArchitect,WorldWideTechnicalFieldOperations@Couchbase

- LinkedIn - https://www.linkedin.com/in/karthikbabusekar- Twitter- https://twitter.com/SKBhere

- E-mail – [email protected]

Change is happening: the way we engage has

evolved

Technology is Further Driving Digital Innovation

Internet of

ThingsBig

Data

MobileCloud

Personalized Feature-rich Quick Anywhere Anytime

Customers now expect exceptional experiences

Agility to adapt to changes in demand

Responsive, context-aware customer experiences

Multi-channel, multi-device engagement

Ability to support a large and growing number of users

What businesses need to succeed in this new world

Buttheserequirementscannotbemetjustattheapplicationtier:

theyneedtobeappliedtothedatabase

©2017 Couchbase Inc. 7

HistoryofCouchbase

The Rise of the System of Engagement Database

8©2017 Couchbase Inc.

System of Record

1980’s

CRM ERP ECM

System of Record

1995

CACHE

CRM ERP ECM

NoSQLSystem of Record

2005

CACHE

CRM ERP ECM

TODAY

NoSQLSystem of

Engagement

MOBILEQUERY

INDEX

SEARCH REPLICATION

ANALYTICS

DATA CACHE

Database requirements to power engaging experiences

9

Adaptable. Flexible schema

Responsive. Consistent

High-Performance

Scale as needed. Able to handle any demand (operations, queries, data)

Secure.Device to cloud,

auditable, pluggable

Global.Data locality,

Geo-distributed, Multi-Data Center

Always on. Fault Tolerant. Able to continue operations

under failure conditions

Mobile experience. Offline, sync’d,

secure, low latency, personalized

Intelligent.Full Text Search, Real

Time Analytics, Big Data enabled

©2017 Couchbase Inc.

Couchbase Data Platform

Unified Administration

Unified Programming Interface

Data Query Index SearchMobileReplication Analytics

{N1QL}

The System of Engagement Database

Developing with Couchbase

Couchbase supports a wide range of frameworks, languages, platforms and infrastructure choices.

Frameworks Languages Platforms Infrastructure

MOBILE MOBILE


Languages and Interfaces for Couchbase

§ Official SDKs– Java– .NET– Node.js– Python

§ For each of these we have– Full Document support– Interoperability– Common Programming Model

Others: Erlang, Perl, TCL, Clojure, Scala

- PHP- C- Go- Ruby

JDBC and ODBC

CouchbaseServerArchitectureOverview


Service-Based Architecture

§ Multi-Dimensional Scalability (MDS) – Optionally Scale each service independently:– Data– Index– Query

MDS is the architecture that enables independent

scaling of data, query, and indexing workloads while

being managed as one cluster.


Full Cluster Architecture

15

STORAGE

Couchbase Server 1

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster Manager

Cluster Manager

Managed Cache

Storage

Data Service

Index Service

Query Service STORAGE

Couchbase Server 2

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster Manager

Cluster Manager

Managed Cache

Storage

Data Service

Index Service


Couchbase Server 3

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster Manager

Cluster Manager

Managed Cache

Storage

Data Service

Index Service


Couchbase Server 4

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster Manager

Cluster Manager

Managed Cache

Storage

Data Service

Index Service


Couchbase Server 5

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster Manager

Cluster Manager

Managed Cache

Storage

Data Service

Index Service


Couchbase Server 6

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster Manager

Cluster Manager

Managed Cache

Storage

Data Service

Index Service

Query Service


Couchbase Data Access

• Everything is built on top of Key Value

• A Document store is a special case of Key-Value

• Views provide aggregation and real-time analytics through incremental map-reduce

• Global Secondary Indexes provide low latency/high throughput indexes

• N1QL is a language that provides a powerful and expressive way of accessing documents


Cluster

Bucket

CRUDView

QueryN1QL Query

Interacting with Data


Couchbase SDKs

§ What does it mean to be a Couchbase SDK?

Cluster

Bucket

CRUD

FunctionalGive the application developer a concurrent API for basic (k-v) or document management

APIget()insert()replace()upsert()remove()


The Document

§ Documents are integral to the SDKs.§ All SDKs support JSON format§ In addition: Serialized objects, Unquoted Strings, Binary pass-through§ A Document contains:

19

Property Description

ID The bucket-unique identifier

Content The value that is stored

Expiry An expiration time

CAS Check-and-Set identifier


Bucket API – Modifying Documents

§ insert() the document if it does not exist§ replace() the document if it does exist§ upsert() the document (insert or replace)§ remove() the document

§ append() data to the document§ prepend() data to the document

§ counter() for increment/decrement type operations


Bucket API – Retrieving Documents

§ get() the document§ getFromReplica() if the active is not available

§ getAndTouch() to load the document and reset the expiry§ getAndLock() to load the document and write-lock it


Unified API – DML, CAS Example1. Two Clients retrieve the same document "XYZ"2. Client A retrieves it first.3. Client B then retrieves XYZ. Both clients will have the same CAS value for document XYZ4. Client B tries to perform an update to document XYZ. The update succeeds as the CAS value

was unchanged from when Client B initially retrieved the document. Once the update succeeds, the CAS value for XYZ changes.

5. Client A then tries to perform an update on XYZ immediately after Client B. The update will fail as Client A's CAS value is out of date. When Client B updated XYZ, the CAS value changed.

©2014 Couchbase, Inc.


Durability Requirements – Mutations

§ By default, a write operation is successful when the data has been written to the

memory of the node with the active vBucket.

– Eventually (but pretty quickly), the data will also be written to persistent storage and to replica

vBuckets

§ persistTo – write operation is successful when data has been written to persistent

storage

– Can be set to require writes to persistent storage for replicas, too

– Using persistTo will slow performance

§ replicateTo – write operation is successful when data has been written to both the

active vBucket and replica vBuckets

– If more than one replica, can set the number of replicas that must be written for success


Bucket API – Querying

§ query() is possible for§ Views§ N1QL

§ Streams N response rows as they arrive from the server§ Loads more than one Document based on Criteria§ Typically used to satisfy secondary and advanced querying use cases

©2017 Couchbase Inc. 25©2016 Couchbase Inc. 25

Bootstrapping your application - Service List

25

• Dynamic Distributed Services

• Dynamic Configuration Updates – No additional

work from the developer

• Fault Tolerant/Durable Connectivity (we are paranoid)

Client Connectivity Characteristics:


Key/Value

§ Fastest way to access & mutate documents§ Access by Document ID only§ Includes subdocument manipulation§ Efficient binary protocol

Latency Flexibility


Key/Value – API DML Methods (Create/Delete/Update)

• insert-Insert a document or binary key/value. Fails if the item exists. • upsert-Stores a document or binary key/value to the bucket, or updates if a

document exists. • replace-Replaces a document or binary key/value in a bucket. Fails if the

item doesn’t exist. • remove-Deletes an item from the bucket. Fails if the item doesn’t exist• append/prepend-Appends or prepends in place the value of a binary k/v

item. Does NOT work with documents • touch-Updates the ttl of a documet.• getAndTouch-Retrieves a document or binary key/value and updates the

expiry of the item at the same time.• counter-Increments or decrements a key's numeric value.


Key/Value – API DML Methods (Retrieval)

• get-Retrieves a document or binary key/value.• getAndLock-Lock the document or binary key/value on the server and

retrieve it. When an document is locked, its CAS changes and subsequent operations on the document (without providing the current CAS) will fail until the lock is no longer held.

• getReplica-Get a document binary key/value from a replica server in your cluster.

• unlock-Unlock a previously locked document or binary key/value on within a bucket.


Key/Value – Durability Methods

§ By default, a write operation is successful when the data has been written to the

memory of the node with the active vBucket.

– Eventually (but pretty quickly), the data will also be written to persistent storage and to replica vBuckets

persistTo – A write operation is

successful when data has been

written to persistent storage.• Can be set to require writes to persistent

storage for replicas, too

• Using persistTo will slow performance

replicateTo – A write operation is

successful when data has been written

to both the active vBucket and replica

vBuckets.• If more than one replica, can set the number of

replicas that must be written to for success


Key/Value - Sub-Document Operations

§ Document Mutations:• Atomic Operate on individual

fields• Identical syntax behavior to

regular bucket methods (upsert, insert, get, replace)

• Support for JSON fragments.• Support for Arrays with

uniqueness guarantees and ordinal placement (front/back)


Views

§ Distributed Map-Reduce§ Great for Aggregations (fast)§ Spatial Support

Latency Flexibility


N1QL

§ Very flexible, but get your index right§ Global Index instead of Map/Reduce§ Similar to SQL

Latency Flexibility


Full Text Search

§ Very Flexible, but more concepts to learn initially§ Inverted Index§ Still Developer Preview§ Similar to Elasticsearch

Latency Flexibility


Show me some code ...

ü Create connection

ü Open bucket

ü Create JSON object

ü Create doc from object, assign key, and "upsert"• if exists, update• if not, insert

ü Get the document by key, and display

// Connect to localhostCluster cluster = CouchbaseCluster.create();

// Open a bucket connection

Bucket bucket = cluster.openBucket("customers");

// Create a document

JsonObject user = JsonObject.empty().put("first", "Walter")

.put("last", "White")

.put("job", "chemistry teacher")

.put("age", 50);

// Store a document

JsonDocument stored =bucket.upsert(JsonDocument.create("walter", user));

// Get the documentJsonDocument walter = bucket.get("walter");

System.out.println("Found: " + walter.getString(“first”));("first"));

§ Connect, create, and retrieve a document (Java)

What’s New and Improved in 4.6.0


Agenda

Cross Datacenter Replication (XDCR)

– Timestamp-based Conflict Resolution

Security

– Secret Management and support for Pluggable Authentication Modules (PAM)

Tools [Developer Preview]

– New CBImport and CBExport utilities

Search [Developer Preview 2] (FTS)

– New Sort and Document “type” features

Query (N1QL)

– New String functions, Date functions, Array functions and Object functions

– Performance Improvements


Cross Datacenter Replication (XDCR)

Timestamp-based Conflict Resolution


XDCR Conflict Resolution Modes

● Revision-based Conflict Resolution [Default]

Current XDCR conflict resolution uses the revision ID (part of document metadata) as the

first field to resolve conflicts between two writes across clusters. Revision IDs keep track

of the number of mutations to a key, thus the current XDCR conflict resolution can be

best characterized as “the most updates wins”.

● Timestamp-based Conflict Resolution [New]

Timestamp-based conflict resolution uses the hybrid logical clock (part of

document metadata) as the first field to resolve conflicts between two writes across

clusters. Timestamp has both physical time (NTP) and a logical counter, thus the new

XDCR conflict resolution is also known as Last Write Wins (LWW) and is best

characterized as “the most recent update wins”.


What is Hybrid Logical Clock?

§ Hybrid Logical Clock is combination of physical time and logical counter.

§ Hybrid Logical Clock is represented as 64 bit integer

– First 48 bit – physical time

– Last 16 bit - logical counter

§ Hybrid Logical Clock is stored in CAS


XDCR Use Cases Supported for Conflict Resolution

● Unidirectional Replication

○ Hot spare/ Disaster Recovery

● Bidirectional Replication

○ Datacenter Locality

● High Availability

○ Cross-Cluster FailOver and FailBack


Security

Secret Management


Secret Management in 4.6

Secret management provides encryption of system secrets• Zero-knowledge secret management system driven by user specified

master password• Allows online password rotation without application downtime

PBKDF2 Encrypt/Decrypt

Data Key

Administrator passwordBucket passwordErlang cookieE-mail password (alerts)

Master password

Non-persisted key


Setting Master Password & Rotating Data Key

Setting Master Password- Via REST APIPOST - /node/contorller/changeMasterPassword- Via CLIcouchbase-cli master-password -c 192.168.0.1:8091 -u Administrator -p password --new-password

Rotating Data Key- Via REST APIPOST - /node/controller/rotateDataKey- Via CLIcouchbase-cli master-password -c 192.168.0.1:8091 -u Administrator -p password --rotate-data-key


Pluggable Authentication Modules (PAM)

▪ Allows UNIX local accounts to authenticate as Couchbase administrators▪ Pluggable authentication architecture that is policy driven

Centralized Management

Centralized and synchronize administrator account management using UNIX user management services

Security Policy Enforcement

Allows configuration of strong security policies such as strong password requirements


Tools

CBImport & CBExport


CBImport

- ‘cbimport' is a new tool to import data into a Couchbase cluster from different data sources. In Spock,

CBImports supports importing data from CSV file or JSON documents.

- Each supported format is a sub-command of the cbimport tool

- ‘cbimport csv’: Imports data into Couchbase from a CSV file

- ‘cbimport json’: Imports data into Couchbase from a JSON file

- ‘cbimport' tool also supports custom key-generation for each document in the imported file. Key

generation is done with a combination of pre-existing fields in a document and custom generator

functions supplied by cbimport.

- ‘cbimport' tool supports importing JSON document of different structure

- JSON object on each line,

- files that contain a JSON list (that is array) where each element is a document,

- and the Couchbase Samples files


CBExport

- ‘cbexport' is a new tool to export data from a Couchbase cluster. In Spock, CBExports supports exporting

JSON data to a file.

- Each supported format is a sub-command of the cbexport tool

- ‘cbimport json’: Exports JSON data from Couchbase to a file

- ‘cbexport' tool supports exporting JSON document of different structure

- JSON object on each line,

- files that contain a JSON list (that is array) where each element is a document,

- and the Couchbase Samples files


Search

Full Text Search (FTS)


Custom Sort Ordering

Sort search results your way § Before 4.6, all results sorted by score§ Now, sort on any indexed field, id, or score in ascending / descending

order§ Control how to handle fields with multiple or missing values

{ ... "sort": [

"country",{ "by" : "field", "field" : "reviews.ratings.Overall", "mode" : "max", "missing" : "last" },{ "by" : "field", "field" : "reviews.ratings.Location", "mode" : "max", "missing" : "last" },"-_score”

]}


Type Mapping By ID

Type mapping = specifying different indexing rules for different doc typesPre-4.6, type defined by a type attribute in the JSON doc bodyNow, doc types can also be determined by parsing doc key – prefix or regular expression


Query


New Date Functions

▪ CLOCK_LOCAL()– Returns the local time at the server.

▪ CLOCK_UTC()– Returns the Coordinated Universal Time

▪ CLOCK_TZ()– Returns the time in the specified timezone

▪ DATE_FORMAT_STR(expr, fmt)– This function converts a given date string parameter to the specified format


Couchbase Mobile 1.4


Couchbase Mobile: The Complete Mobile Database Solution

Lightweight embedded NoSQL database with full CRUD and query

functionality.

Secure web gateway with sync,

REST, stream, batch and event APIs for syncing data over

the web.

Highly scalable, Highly available,

High performance NoSQL database

servers.

Couchbase Lite Sync Gateway Couchbase Server

EMBEDDED DATABASE SYNCHRONIZATION DATABASE SERVER


Couchbase Lite Overview

§Cross-platform support for all major operating systems and platforms

§Built native from the ground up

§500kb for most platforms

§256-bit AES full database encryption

55

Couchbase LiteEmbedded Database


Sync Gateway Overview§Synchronize data between Couchbase Lite &Couchbase Server

§Pluggable Authentication

§Fine grained user and role based access control

§Elastically scalable in real-time

56

Sync GatewaySynchronization


Security

§Encryption of Local Database

–AES-256 / PBKDF2

•HTTPS / TLS•Separate Read & Write Permission

with Fine Grain Document Control• Pluggable Authentication

Couchbase LiteEmbedded Database

Sync GatewaySynchronization


Load Balancer

CRUD , Security &

Seq. Number

HA DataStorage &

Channel Index

Couchbase Mobile - Before

20,000 MaxConcurrent


Problems – Single Global Sequence Number

Updating a single key at scale causes too many problems


Bucket = Database

vB

Data buckets (default)

vB

1 ….. 1024

Virtual buckets or

(shards / files)


Solution - DCP (Database Change Protocol)Cluster data replication via sequential streaming feed from each vBucket (shard/file) on each node

called “DCP”


Sync Gateway Accelerator (Enterprise Only)Couchbase Mobile v1.2+

Accelerator listens to DCP to create k/v channel indexes & uses DCP sequence numbers

Same vLAN


Horizontally Scale All Three Parts in v1.2+

LBx,000,000+Concurrent

CRUD & Security

HA DataStorage

Channel Index &Seq. Number

Questions?