deep dive into cluster manager in couchbase server 4.0: couchbase connect 2015

DEEP DIVE INTO CLUSTER MANAGER IN 4.0

Dave FinlaySnr Director Server Engineering, Couchbase

©2015 Couchbase Inc. 2

Cluster Manager

What is this “Cluster Manager” of which you speak?


First: We Need a Cluster!

machine 1 machine 2 machine 3

Ethernet


The “Depth Gauge”

cluster

node

os process

erlang process

The “depth gauge” indicates the current

zoom-level as we dive into Cluster

Manager


The “Depth Gauge”

cluster

node

os process

erlang process


Back to the Cluster


Ethernet

Couchbase Node

Couchbase Node

Couchbase Node


Back to the Cluster


Ethernet

node 1 node 2 node 3

This is the Cluster

Manager


What Does the Cluster Manager Do?


What Does the Cluster Manager Do?

manage services

manage data topology

manage replications

provide metadata

service

auth / security REST API UIstats,

monitoring, simple alerting


Anatomy of a Node

machine 1

babysitter

query

indexe

r

mem

cach

ed

ns-se

rver

xdcr

vie

w -

en

gin

e

oth

er …


Anatomy of a Node

machine 1

babysitter

query

indexe

r

mem

cach

ed

ns-se

rver

xdcr

vie

w -

en

gin

e

oth

er …

The Cluster Manager is babysitter

and ns-server


Anatomy of a Node

machine 1

babysitter

mem

cach

ed

ns-se

rver

Memcached serves key-

value memcached requests & is deployed on

“data service” nodes


Anatomy of a Node

machine 1

babysitter

mem

cach

ed

ns-se

rver

xdcr

vie

w -

en

gin

e

oth

er …

XDCR, view-engine and some other

services also deployed on “data nodes”


Anatomy of a Node

machine 1

babysitter

mem

cach

ed

ns-se

rver

xdcr

vie

w -

en

gin

e

oth

er …

XDCR and view-engine

were previously part of ns-

server. They’ve been carved out for

improved robustness


Anatomy of a Node

machine 1

babysitter

indexe

r

mem

cach

ed

ns-se

rver

Indexer is deployed on nodes with

the “indexing service”

xdcr

vie

w -

en

gin

e

oth

er …


Anatomy of a Node

machine 1

babysitter

query

indexe

r

mem

cach

ed

ns-se

rver

Query is deployed on

“query nodes” and serves N1QL

requests xdcr

vie

w -

en

gin

e

oth

er …


Oh no, it’s the Chaos Monkey!

machine 1

babysitter

query

indexe

r

mem

cach

ed

ns-se

rver

xdcr

vie

w -

en

gin

e

oth

er …

xdcr



machine 1

babysitter

query

indexe

r

mem

cach

ed

ns-se

rver

xdcr

vie

w -

en

gin

e

oth

er …

xdcr


Inside ns-server

per-node-&-bucket services

generic distributed facilities

generic local facilities

vclock, uuid, work queue, events, misclogging (ALE)

distributed node discovery

master-only services

REST admin

config gossip replication

local config store

per-node services



A Little Bit About Erlang

Developed in the 80’s for use in switches Such as this – the

famous AXD 301 switch which

boasted ridiculous reliability

(Even allowing for hyperbole, it was pretty reliable)


What Cool About Erlang?

According to Four-fold Increase in Productivity and Quality— Industrial-Strength Functional Programmingin Telecom-Class Products, http://www.erlang.se/publications/Ulf_Wiger.pdf

http://www.erlang.se/publications/Ulf_Wiger.pdf

http://www.erlang.se/publications/Ulf_Wiger.pdf


Key Feature of Erlang #1: Processes & Messages

process

Active, “lightweight processes”

Supports “actor model” style of programming

Processes have a mailbox for incoming messages



process Pxyz

process Q

Values are immutable (no shared state)

Communication asynchronous

P ! x,P ! y,P ! z.



aprocess Q

x y z

process P

“Selective receive”

receive y -> Q ! aend.



process Q

x z

process P

a



process Q

x z

a

bprocess P


1

gen_server

Key Feature of Erlang #2: OTP

21process

Such as gen_server, a generic, stateful,

startable and stoppable server

Framework that provides useful higher-level

constructs built from Erlang primitives


Key Feature of Erlang #2: OTP

supervisor

gen_server3

gen_server2

gen_server1

Also such as supervisor, a process

responsible for starting, restarting, stopping

and monitoring its child processes


gen_server2

gen_server2


supervisor

gen_server3

gen_server1


Inside ns-server

ns_serverThe ns_server ‘application’.

Applications describe the bundle of code that is started and stopped

together.


Inside ns-server

ns_server ns_server_cluster_sup

The application is responsible for starting

the top level supervisor:

ns_server_cluster_sup

Supervises all supervisors and by extension all Erlang

processes in ns_server


Inside ns-server


ns_config_sup ns_clusterother

services…ns_server_nodes_sup

Main supervisor for the node outside of

config mgmt

Everything related to managing config: e.g.

storing, updating change

notifications

Performs node join / leave requests

Other stuff we won’t focus on

today


Inside ns-server


ns_config_sup ns_clusterother

services…ns_server_nodes_sup

ns_server_sup otherservices…Controls main

part of supervision hierarchy


Inside ns-server


ns_config_sup ns_cluster otherservices

…ns_server_nodes_sup

ns_server_sup otherservices…


Inside ns-server

ns_server_sup

mb_master ns_node_disco_sup menelaus_sup

Responsible for master election

Node discovery supervisor

Supervisor for REST API and web UI app

serverns_orchestrator

mb_master_sup


Inside ns-server

ns_server_sup

ns_bucket_supother

processes


Inside ns-server

ns_bucket_sup

single_bucket_kv_supother

processes

ns_memcached_sup

ns_memcached

ns_memcached mediates

communication with memcached

process and tracks its health


Inside ns-server

ns_bucket_sup


processes

ns_memcached_supreplicationmanager

dcp_sup

dcp_replicator-node1


ns_memcached

dcp_sup and the replicators

manage inbound (consumer) DCP

replication streams


Inside ns-server

ns_bucket_sup


processes

ns_memcached_supreplicationmanager

dcp_sup



janitor_agent_sup

janitoragent

ns_memcached

The janitor agent keeps things nice

and clean and provides remote communication

façade for bucket management


NS-Server in Action: Rebalance

ns_rebalancer

idlerebalancing

ns_orchestratormenelaus_web

node master node

rebalance

rebalancing

User clicks “Rebalance”

button

State -> “rebalancing”

Orchestrator starts

rebalancer


ns_single_vbucket_mover1

ns_single_vbucket_mover2


ns_rebalancer ns_bucket_mover

master node

Rebalance computes new vbucket map & starts the

bucket mover

Bucket mover in turn starts

single vbucket movers


janitoragent


master nodeto-node for vbucket

dcp_replicator

dcp_supjanitoragent

beginns_single_vbucketmover

from-node for vbucket

Single vbucket mover sends

request to new node to begin

replicating

memcachedto-node

memcachedfrom-node


janitoragent



dcp_replicator

dcp_sup


wait

ns_single_vbucketmover

memcachedto-node

memcachedfrom-node

janitoragent

Then mover waits until the backfill is

done and replication is up-

to-date


ns_single_vbucketmover

janitoragent



dcp_sup


memcachedto-node

janitoragent

dcp_replicator

memcachedfrom-node

Lastly mover converts the

backfill replication stream to a “takeover”

stream

takeover


Inside ns-server


generic distributed facilities

generic local facilities

vclock, uuid, work queue, events, misclogging (ALE)

distributed node discovery

master-only services

REST admin

config gossip replication

local config store

per-node services



Anatomy of a Node

machine 1

babysitter

query

indexe

r

mem

cach

ed

ns-se

rver

xdcr

vie

w -

en

gin

e

oth

er …


Back to the Cluster


Ethernet

node 1 node 2 node 3

Get Started Today Couchbase Server 4.0 & N1QL

couchbase.com/beta

deep dive into cluster manager in couchbase server 4.0: couchbase connect 2015

Technology

node machine

cluster machine

node services

ethernet node

cluster manager

view engine

n1ql requests xdcr viewengine

query nodes