overview of couchbase server architecture – couchbase connect 2016

©2016 Couchbase Inc. 1

Couchbase Server Architecture

©2016 Couchbase Inc. 2©2016 Couchbase Inc.

Look Who’s Talking

David HaikneyDirector of Engineering,

EMEA

Dave RigbyTechnical Lead, Data Service


What To Expect

•Whistle-Stop Tour with demos•Key Architectural Concepts•Exploration of each Service•Where to hear more @ Connect!


Key Concepts

©2016 Couchbase Inc.©2016 Couchbase Inc.

Concept 1 – Your Choice of Services

STORAGE

Couchbase Server 1

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster ManagerCluster Manager

Data Service

Index Service

Query Service

Full Text Search



STORAGE

Couchbase Server 1

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Data Service

Index Service

Query Service

STORAGE

Couchbase Server 2

Managed Cache


STORAGE

Couchbase Server 3

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


STORAGE

Couchbase Server 4

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


STORAGE

Couchbase Server 5

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


STORAGE

Couchbase Server 6

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Full Text Search



STORAGE

Couchbase Server 1

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Data Service STORAGE

Couchbase Server 2

Managed Cache

Cluster Manager


Couchbase Server 3

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster Manager


Couchbase Server 4

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster Manager

Index Service

STORAGE

Couchbase Server 5

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster Manager

Index Service

STORAGE

Couchbase Server 6

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster Manager

Query Service

FTS Service


Find Out More!

Preparing for success: the top deployment 10 do's and don'ts

• Today 11:00 am - 11:50 am

• Operations Track• Great America 1

Karthik SekarSolutions Architect,

Couchbase

Ian McCloyTechnical Support Manager,

EMEA, Couchbase


Find Out More!

Picking the right API for the right job

• Wednesday 2:00 pm - 2:50 pm

• Development Track• Great America K Michael Nitschinger

JVM Engineer, Couchbase


Application

Concept 2 – Automatic Sharding

Cluster

Key = connect2016

HASH(“connect2016”)

Server A Server CServer B

VB Server

0 A

1 B

2 C

…

27 B

1023 C

{…

}


Application

Concept 2 – Automatic Sharding

Cluster

Key = connect2016

HASH(“connect2016”)

Server A Server CServer B

VB Server

0 A

1 B

2 C

…

27 D

1023 C

{…

}

Server D


Concept 3 – Exceptional Data Streaming

• Database Change Protocol (DCP)

• High Performance / In-Memory

• De-Duplication• Ordered, predictable and

consistent• Restartable

Data Service

Index Service

Map/Reduce

DCP

Cross Data Center Cluster

Hadoop

Client/AppNotifications

(future)

Backup/Export XDCR

Full Text Search

DataReplication


STORAGE

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Data Service

Index Service

Query Service

Full Text Search


Cluster Manager


The Cluster Manager

STORAGE

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Data Service

Index Service

Query Service

Full Text Search

The control plane of the server:• Cluster membership

• Status & health monitoring

• Service layout• Data placement

• Rebalance• Failover

• Authentication• Admin APIs

Implemented in Erlang


The Cluster Manager

The control plane of the server:• Cluster membership

• Status & health monitoring

• Service layout• Data placement

• Rebalance• Failover

• Authentication• Admin APIs

Implemented in Erlang

Generic distributed facilities• Global configuration (gossip replication)• Distributed node discovery

Generic local facilities

Master services• Cluster-wide

operations • Data placement • Rebalance• Auto-failover

Admin Console & REST API

Per-node services • Heartbeats• Babysitter

Bucket services• DCP management• Stats collection & monitoring

Authentication

• Local configuration store• Logging and Other Services• Vclock, uuid• Work queue

• Misc


Data Service


The Data Service

The (low-level) data plane of the server: • Key/Value access• Map/Reduce Views [*]

KV-Engine:

• Evolution of memcached; adding persistence, replication, enhanced data access APIs

• Asynchronous networking supports ~10K clients.

• Memory-centric architecture.• Disk IO performed via background threads.

C++11

STORAGE

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Data Service

Index Service

Query Service

Full Text Search


Data Service Architecture

Database Engine (ep-engine)

Dispatcher Request / Response

Scheduler

Reader IO

Aux IONon IO

…

KV-Bucket

…

KV-EngineItem Pager

Expiry Pager

Checkpoint Manager

ManagedCache

Partition Hash Tables

(Active and Replica)

Flusher

Batch Reader

Partition Hash Tables

(Active and Replica) Writer IO

DCP Streaming Authentication


Journey of a K/V operation


Journey of a K/V operation

STORAGE

Couchbase Server 1

Managed Cache

Cluster Manager

Data Service

(1x replica)STORAGE

Couchbase Server 0

Managed Cache

Cluster Manager

Data Service

(1x replica)

SDK Application

DCP Replication

'tweets::Dhaikney'-> SD.upsert('hashtags', 'Connect2016')


Indexing Service


Index Service

Three “indexing” services:• Incremental Map / Reduce Views

• Javascript map() & reduce() functions applied to all mutations• Supports geo-spatial views• Co-located with Data service

• Global Secondary Indexes (GSI)• Full-Text Search – more later

GSI: efficient indexes for secondary lookups and ad-hoc query processing

STORAGE

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Data Service

Index Service

Query Service

Full Text Search


Data Service

Projector & Router

Indexing Service

Bucket 1

DCP Stream

Projector and Router: 1 Projector and Router per node1 stream of changes per buckets per supervisor

Bucket 2


Data Service

Projector & Router

Indexing Service

Query ServiceIndex Service

SupervisorIndex maintenance &

Scan coordinator

Index#2Index#1

Query Processorcbq-engine

Bucket 1

DCP StreamIndex#4Index#3

Bucket#2

Bucket#1

Projector and Router: 1 Projector and Router per node1 stream of changes per buckets per supervisor

ForestDBStorage Engine

Supervisor1 Supervisor per nodeMany indexes per Supervisor

Bucket 2

Memory-Optimized

Indexes


Find Out More!

Memory-optimized indexes: how they work

Today, 4:00 pm - 4:50 pm• Great America 2

Sarath LakshmanSenior Software Engineer, Couchbase


Query Service


Query Service

Client

FetchParse Plan Join FilterPre-Aggregate

Offset Limit ProjectSortAggregateScan

Query Service

Index Service

Data Service


Find Out More!

How to understand and use the query optimizer

Today, 11:00 am - 11:50 am• Great America 2

Keshav MurthyDirector, Query Development, Couchbase


Find Out More!

Going beyond SELECT, top 10 ways to use N1QL

Wednesday 10:00 am - 10:50 am

• Great America KPrasad VarakurProduct Manager, Couchbase


Full Text Search


Full Text Search

“Googling for your JSON documents”

• Index Fields or Documents

• Lexical Analysis and Stemming

• Flexible Query Capabilities

• Available and Scalable

STORAGE

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Data Service

Index Service

Query Service

Full Text Search


Full Text Search

“Googling for your JSON documents”

• Index Fields or Documents

• Lexical Analysis and Stemming

• Flexible Query Capabilities

• Available and Scalable

Config{ Node Definitions: … Index Definitions: … Plan Indexes: …}

Planner Janitor PIndex

bleve

DCPFeed

REST Interface

Manager

DCPFeed

PIndex

bleve


Find Out More!

Full-text search: how it works and what it can do

Wednesday, 1:00 pm - 1:50 pm

Great America 2 Steve YenCo-Founder, Couchbase

Marty SchochSenior Software Engineer,

Couchbase


New! Analytics Service


New: Analytics Service!

STORAGE

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Data Service

Index Service

Query Service

Full Text Search

Analytics Service

Sneak Peak: Couchbase AnalyticsWednesday, 2pm –

2:50pm• Great America 2 Till Westman, Senior Director,

Server Development, Couchbase


Summary

• The architecture of a next generation database• The addition of new features whilst retaining our principles:

• Performance• Scalability• Availability • Ease of use

• More technical meat available at Connect• Ask questions and get involved!


Thank You!