couchbase sydney meetup #1 couchbase architecture and scalability

Couchbase Sydney Meetup

Couchbase Architecture and ScalabilityKarthik Babu Sekar

SolutionsArchitect,WWFieldOperations@Couchbase- LinkedIn - https://www.linkedin.com/in/karthikbabusekar

- Twitter- https://twitter.com/SKBhere- E-mail – [email protected]

©2016 Couchbase Inc.©2015 Couchbase Inc.

History of Couchbase

2


Couchbase provides a complete Data Management solution

3

Highavailabilitycache

Key-valuestore

Documentdatabase

Embeddeddatabase Syncmanagement

Multi-purpose capabilities support a broad range of apps and use cases

Enterprises often start with cache, then broaden usage to other apps and use cases


Couchbase meets today’s & tomorrow’s requirements

4

Flexible data model

Consistent performance at scale

High availability

Easy,affordablescalability

24x365


Common Use-Cases

5

360DegreeCustomerView

ProfileManagement

Catalog FraudDetection

ContentManagement

InternetofThings

DigitalCommunication

RealTimeBigData

MobileApplications

Personalization


What makes Couchbase unique?

6

Performance & scalability leaderSubmillisecondlatencywithhighthroughput;memory-centricarchitecture

Multi-purpose

Enterprise class Web Console

Easytodeploy&manage;integratedAdminConsole,single-clickclusterexpansion&rebalance

Cache,keyvaluestore,documentdatabase,andlocal/mobiledatabaseinsingleplatform

Always-on availability

Datareplicationacrossnodes,clusters,anddatacenters

Enterprises choose Couchbase for several key advantages

24x365


§ Consolidatedcacheanddatabase

§ Tunememoryrequiredbasedonapplicationrequirements

Multi-purpose database supports many uses

77

Tunable built-in cache

Flexible schemas with JSON

Couchbase Lite

§ RepresentdatawithvaryingschemasusingJSONontheserveroronthedevice

§ IndexandquerydatawithJavascript views

§ LightweightembeddedDBforalwaysavailableapps

§ SyncGatewaysyncsdataseamlesslywithCouchbaseServer


Couchbase delivers always-on availability

8

High Availability

Disaster Recovery

Backup&Restore

§ In-memoryreplicationwithmanualorautomaticfailover

§ Rack-zoneawarenesstominimizedataunavailability

§ Memory-to-memorycrossclusterreplicationacrossdatacentersorgeos

§ Active-activetopologywithbi-directionalsetup

§ FullbackuporIncrementalbackupwithonlinerestore

§ Deltanodecatch-upsforfasterrecoveryafterfailures

24x365


Simplified administration for exceptional ease of use

Online upgrades and operations

Built-in enterprise class admin console

Restful APIs

§ Onlinesoftware,hardwareandDBupgrades

§ Indexing,compaction,rebalance,backup&restore

§ Performalladministrativetaskswiththeclickofabutton

§ Monitorstatusofthesystemvisualatclusterlevel,databaselevel,serverlevel

§ AlladminoperationsavailableviaUI,RESTAPIsorCLIcommands

§ IntegratethirdpartymonitoringtoolseasilyusingREST


Couchbase leads in performance and scalability

Auto Sharding Memory-memory XDCR

MultiDimensionalScaling

§ Nomanualsharding

§ Databasemanagesdatamovementtoscaleout– nottheuser

§ Market’sonlymemory-to-memorydatabasereplicationacrossclustersandgeos

§ Providesdisasterrecover/datalocality

§ Hugelysimplifiesmanagementofclusters

§ Easytoscaleclustersbyaddinganynumberofnodes

Service-BasedArchitecture

Multi-Dimensional Scalability (MDS) – Optionally Scale each service independently:• Data• Index• Query

MDS is the architecture that enables independent

scaling of data, query, and indexing workloads while

being managed as one cluster.

©2015 Couchbase Inc.

Full Cluster Architecture

12

STORAGE

Couchbase Server 1

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster ManagerClusterManager

ManagedCache

Storage

DataService

IndexService

QueryService STORAGE

Couchbase Server 2

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


ManagedCache

Storage

DataService

IndexService


Couchbase Server 3

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


ManagedCache

Storage

DataService

IndexService


Couchbase Server 4

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


ManagedCache

Storage

DataService

IndexService


Couchbase Server 5

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


ManagedCache

Storage

DataService

IndexService


Couchbase Server 6

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


ManagedCache

Storage

DataService

IndexService

QueryService

©2015 Couchbase Inc.

Full Cluster Architecture

13

STORAGE

Couchbase Server 1

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


ManagedCache

Storage

DataService

IndexService


Couchbase Server 2

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


ManagedCache

Storage

DataService

IndexService


Couchbase Server 3

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


ManagedCache

Storage

DataService

IndexService


Couchbase Server 4

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


ManagedCache

Storage

DataService

IndexService


Couchbase Server 5

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


ManagedCache

Storage

DataService

IndexService


Couchbase Server 6

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


ManagedCache

Storage

DataService

IndexService

QueryService

Couchbase Architecture: Single Node

14

High level architecture - Single Node

CouchbaseServernodesareidentical

Twocorecomponentsü ClusterManagerü DataManager

Threeindependentservicesü DataServiceü IndexServiceü QueryService

CouchbaseServerNode

DataManager ClusterManager

ManagementRESTAPIWebUI

Erlang/OTP

Node/Cluster

Coordination

DataService

Storage

ManagedCache

Get/Set

MapReduceViews

QueryService

QueryEngine

QueryPlanCache

Storage

IndexService

IndexingEngine

ManagedCache

Storage

Cluster ManagerManagementandcoordination

Responsibleforü UI/RESTinterfaceü Configuration/Administrationü Coordinatingclusterrebalancingü Processmonitoringü Statisticsü No dataprocessing

Erlang/OTPcodebaseü DevelopedbyEricssonü Telco-switchgradeperformance

CouchbaseServerNode

ClusterManager


Erlang/OTP

Node/Cluster

Coordination

DataManager

DataService

Storage

ManagedCache

Get/Set

Map-ReduceViews

QueryService

QueryEngine

QueryPlanCache

Storage

IndexService

IndexingEngine

ManagedCache

Storage

:8091 AdminConsole/API

Node Manager

Copyright © 2015 Couchbase, Inc.

Threesupportedworkloads

DataServiceü Get/Setofindividualdocumentsü Distributedsecondaryindexes(Views)IndexServiceü Globalsecondaryindexesü IndexmaintenanceandprovisionQueryServiceü N1QL(SQLsupersetforJSON)ü Executionplans,indexcoordination

C/C++andGocodebasesü Veryfastandmemoryefficient

evenwithlargememoryfootprint CouchbaseServerNode

ClusterManager


Erlang/OTP

Node/Cluster

Coordination

DataManager

DataService

Storage

ManagedCache

Get/Set

Map-ReduceViews

QueryService

QueryEngine

QueryPlanCache

Storage

IndexService

IndexingEngine

ManagedCache

Storage

Singlenodetype/codebaseü Allservicesavailableoneverynodeü Servicesenabledindependently

Allowsclustertobeconfiguredforü Varyinghardwarecapacitiespernodeü Applicationworkloadtuning

CouchbaseServerNode

ClusterManager


Erlang/OTP

Node/Cluster

Coordination

DataManager

QueryService

DataService

IndexService

IndexService

IndexingEngine

ManagedCache

Storage

QueryService

QueryEngine

QueryPlanCache

Storage

DataService

Storage

ManagedCache

Get/Set

Map-ReduceViews

Multi-Dimensional Scalability

Copyright © 2015 Couchbase, Inc.

©2015CouchbaseInc. 19

CouchbaseDataAccess

• Everything is built on top of Key Value

• A Document store is a special case of Key-Value

• Views provide aggregation and real-time analytics through incremental map-reduce

• Global Secondary Indexes provide low latency/high throughput indexes

• N1QL is a language that provides a powerful and expressive way of accessing documents

Key Differentiators - N1QL

N1QL is a Next-generation NoSQL query language for JSON.

20

§ JOINS§ Powerful Extensions for JSON § ODBC/JDBC drivers available

Introduction to N1QL

SQL query language for documents

• SELECT * FROM WHERE/LIKE/JOIN/GROUP/etc, CREATE INDEX

• Extended for JSON to support nested and hierarchical data structures

• Support for map-reduce views and Global Secondary Indexes

• Query (DQL), Manipulation (DML), Description (DDL)

• ODBC/JDBC drivers

SQL 4 documents - examples

2222

Listingproductsinacategory Findingthemostpopularproductsinacategory

GlobalSecondaryIndexes

NewStorageOptionsforGlobalSecondaryIndex1. Memory-OptimizedGlobalSecondaryIndexes(MOI)

• OptimizedforMemory:smallmemoryfootprint,optimizedforlowestlatencyqueries• FasterIndexing:freshindexesunderheavymutationswithlockfreeindexmaintenance• PredictablelowlatencywithQueryandKeyBasedOperations,eveninpresenceoflarge

numberofindexes

1. StandardGlobalSecondaryIndexes• OptimizedforStorage+Memory:efficientspilltodiskwithForestDBasthestorage

engine• AdvancedIOPerformance: newcircularwritesthatminimizecompactionoverheadof

appendonlywrites

©2016CouchbaseInc.

Mobile

Search

CacheOLAP

Data Warehouse

Hadoop

RDBMS

©2016CouchbaseInc.

Search

OLAP

Data Warehouse

Hadoop

{N1QL}

©2016CouchbaseInc.

OLAP

Data Warehouse

Hadoop

{N1QL}

©2016CouchbaseInc.

INTRODUCING SEARCH

Structured Query AND SearchOne platform.

|

Search Feelinglucky?

©2016CouchbaseInc.

OLAP

Data Warehouse

Hadoop

{N1QL}

©2016CouchbaseInc.

Data Warehouse

Hadoop

{N1QL}

©2016CouchbaseInc.

INTRODUCING REALTIME ANALYTICS

You may also like these products

Thank you for your order!Order #: 3423943

Paid Partner Content

Operations & Analytics. One platform.

©2016CouchbaseInc.

OLAP

Data Warehouse

Hadoop

{N1QL}

©2016CouchbaseInc.

INTRODUCING REALTIME ANALYTICS

You may also like these products

Thank you for your order!Order #: 3423943

Paid Partner Content

Operations & Analytics. One platform.

©2016CouchbaseInc.

THE DATA PLATFORM FOR THE DIGITAL ECONOMY

Unified Adminstration

Unified Programming Interface

Data Query Index SearchMobileReplication Analytics

{N1QL}

XDCR: Cross Data Center Replication

Application can access both clusters (master – master)Scales out linearlyDifferent from intra-cluster replication (“CP” versus “AP”)

XDCR: Flexible topologies

One-one, one-many, many-oneDifferently sized and resourced clusters supported

Couchbase Architecture: Mobile

43


Couchbase Mobile Overview


Always Available and Fast

45


Native Support

46


Couchbase Mobile Solution

47

Key Take-aways

Couchbase Mobile = Couchbase Lite + Couchbase Sync Gateway + Couchbase Server

Couchbase Lite: Only NoSQL database for mobile devices (phones, tables, embedded systems)

Couchbase Sync Gateway: Offline/Online synchronization

48


Backup and Restore with Couchbase§ Zer0-downtimebackupandrestore

§ Built-inutilities:cbbackup /cbrestore§ Full,differentialandcumulativebackupavailableper-bucket.§ Restorefromanypoint,toanybucketortopology

Demo & Q&A

couchbase sydney meetup #1 couchbase architecture and scalability

Technology