Einführung inCouchbase Server 2.0

• Developer Advocate at Couchbase, Inc.

• Maintainer of the Couchbase Java SDK

• Speaking at Conferences and Meetups

• Living and Working in Vienna, Austria

{“about”: “me”}

1. Introduction to Couchbase Server 2.0

2. Cluster Management

3. Operations & Document Design

4. Introduction to Views

Agenda

Couchbase Server 2.0

Big Release in December 2012

Couchbase Open Source Project

• One of the leading NoSQL database projects focused on distributed database technology and surrounding ecosystem

• Supports both key-value and document-oriented use cases

• All components are available under the Apache 2.0 Public License

• Obtained as packaged software in both enterprise and community editions.

Easy Scalabili

ty

Consistent High

Performance

Always On

24x365

Grow cluster without application changes, without downtime with a single click

Consistent sub-millisecond read and write response times

with consistent high throughput

No downtime for software upgrades, hardware maintenance, etc.

JSONJSONJSON

JSONJSON

PERFORMANCE

Flexible Data Model

JSON document model with no fixed schema.

Core Principles

New in 2.0

JSON support Indexing and Querying

Cross data center replicationIncremental Map Reduce

JSONJSONJSON

JSONJSON

Couchbase Server 2.0 Architecture

Hea

rtbe

at

Proc

ess

mon

itor

Glo

bal s

ingl

eton

sup

ervi

sor

Confi

gura

tion

man

ager

on each node

Reba

lanc

e or

ches

trat

or

Nod

e he

alth

mon

itor

one per cluster

vBuc

ket s

tate

and

repl

icati

on m

anag

er

httpRE

ST m

anag

emen

t API

/Web

UI

HTTP8091

Erlang port mapper4369

Distributed Erlang21100 - 21199

Erlang/OTP

storage interface

Couchbase EP Engine

11210Memcapable 2.0

Moxi

11211Memcapable 1.0

Memcached

New Persistence Layer

8092Query API

Que

ry E

ngin

e

Data Manager Cluster Manager

33 2

Single node - Couchbase Write Operation

Managed Cache

Dis

k Q

ueue

Disk

Replication Queue

App Server

Couchbase Server Node

Doc 1Doc 1

Doc 1

To other node

GET

Doc

1

33 2

Single node - Couchbase Read Operation

Dis

k Q

ueue

Replication Queue

App Server

Doc 1

Doc 1Doc 1

Managed Cache

Disk

To other node

Couchbase Server Node

COUCHBASE SERVER CLUSTER

Basic Operation

• Docs distributed evenly across servers

• Each server stores both active and replica docsOnly one server active at a time

• Client library provides app with simple interface to database

• Cluster map provides map to which server doc is onApp never needs to know

• App reads, writes, updates docs

• Multiple app servers can access same document at same time

User Configured Replica Count = 1

READ/WRITE/UPDATE

ACTIVE

Doc 5

Doc 2

Doc

Doc

Doc

SERVER 1

ACTIVE

Doc 4

Doc 7

Doc

Doc

Doc

SERVER 2

Doc 8

ACTIVE

Doc 1

Doc 2

Doc

Doc

Doc

REPLICA

Doc 4

Doc 1

Doc 8

Doc

Doc

Doc

REPLICA

Doc 6

Doc 3

Doc 2

Doc

Doc

Doc

REPLICA

Doc 7

Doc 9

Doc 5

Doc

Doc

Doc

SERVER 3

Doc 6

APP SERVER 1

COUCHBASE Client Library

CLUSTER MAP

COUCHBASE Client Library

CLUSTER MAP

APP SERVER 2

Doc 9

Add Nodes to Cluster

• Two servers addedOne-click operation

• Docs automatically rebalanced across clusterEven distribution of docsMinimum doc movement

• Cluster map updated

• App database calls now distributed over larger number of servers

REPLICA

ACTIVE

Doc 5

Doc 2

Doc

Doc

Doc 4

Doc 1

Doc

Doc

SERVER 1

REPLICA

ACTIVE

Doc 4

Doc 7

Doc

Doc

Doc 6

Doc 3

Doc

Doc

SERVER 2

REPLICA

ACTIVE

Doc 1

Doc 2

Doc

Doc

Doc 7

Doc 9

Doc

Doc

SERVER 3 SERVER 4 SERVER 5

REPLICA

ACTIVE

REPLICA

ACTIVE

Doc

Doc 8 Doc

Doc 9 Doc

Doc 2 Doc

Doc 8 Doc

Doc 5 Doc

Doc 6

READ/WRITE/UPDATE READ/WRITE/UPDATE

APP SERVER 1

COUCHBASE Client Library

CLUSTER MAP

COUCHBASE Client Library

CLUSTER MAP

APP SERVER 2

COUCHBASE SERVER CLUSTER

User Configured Replica Count = 1

Fail Over Node

REPLICA

ACTIVE

Doc 5

Doc 2

Doc

Doc

Doc 4

Doc 1

Doc

Doc

SERVER 1

REPLICA

ACTIVE

Doc 4

Doc 7

Doc

Doc

Doc 6

Doc 3

Doc

Doc

SERVER 2

REPLICA

ACTIVE

Doc 1

Doc 2

Doc

Doc

Doc 7

Doc 9

Doc

Doc

SERVER 3 SERVER 4 SERVER 5

REPLICA

ACTIVE

REPLICA

ACTIVE

Doc 9

Doc 8

Doc Doc 6 Doc

Doc

Doc 5 Doc

Doc 2

Doc 8 Doc

Doc

• App servers accessing docs

• Requests to Server 3 fail

• Cluster detects server failedPromotes replicas of docs to activeUpdates cluster map

• Requests for docs now go to appropriate server

• Typically rebalance would follow

Doc

Doc 1 Doc 3

APP SERVER 1

COUCHBASE Client Library

CLUSTER MAP

COUCHBASE Client Library

CLUSTER MAP

APP SERVER 2

User Configured Replica Count = 1

COUCHBASE SERVER CLUSTER

Q & A

Operations & Document Design

Fundamentals

• Every Document has a Key assigned to it

• Keys­ must be max. 255 UTF-8 chars long­ must be unique in a bucket (“database”)­ are completely under the control of the application

• Values­ can be any binary blob (but bonus points when JSON!)­ can be up to 20MB in size

Basic Store & Retrieve Operations

• get(key)Retrieve a document

• set(key, value)Store a document or replace if it exists

• add(key, value)Store a document and fail if it exists

• replace(key, value)Replace a document and fail if it does not exist

Lots of other Operations

• View operations

• Atomic Counters (increment, decrement)

• Append/Prepend

• CAS (Compare and Set)­ Optimistic Locking

• “Get with Lock”­ Write Lock on Objects

• Bulk Operations­ Saves network overhead

• Stats

Couchbase 2.0 Bonus Points: JSON

• JSON is a lightweight format to represent document structure in a language-independent manner.

• If JSON documents are stored, the View engine can be used.

• Allows to build secondary indexes on your datasets.

• Makes it possible to ask questions like“Give me all user documents by lastname”.

The BIG mental adjustment

• In SQL, we tend to avoid hitting the database as much as possible.­ Use JOINs and let the DB optimizer figure out what to do

• In Couchbase, get’s and set‘s are so fast they are trivial, not bottlenecks, this is hard for many people to accept at first; Multiple get statements are commonplace, don’t avoid it!

meta{“id”: “u::michael@couchbase.com”,“rev”: “1-0002bce0000000000”,“flags”: 0,“expiration”: 0,“type”: “json”}

document{“uid”: 123456,“firstname”: “Michael”,“lastname”: “Nitschinger”,“age”: 25,“favorite_colors”: [“blue”, “black”],“email”: “michael.nitschinger@...”}

Meta­Information­Including Key

All Keys Unique and Kept in RAM

Document­Value

Most Recent In Ram And Persisted To Disk

Document Structure with JSON

Views & Indexes

• Views can cover a few different use cases­ Simple secondary indexes (the most common)­ Complex secondary, tertiary and composite indexes­ Aggregation functions (reduction)

•Example: count the number of North American Ales

­ Organizing related data

• Built using Map/Reduce­ Map function creates the index (b-tree)­ Reduce function summarizes (reduces) information­ Written using superfast Javascript (V8)

A View in Action

Developing with Couchbase

Aggregate View of Data

http://martinfowler.com/bliki/AggregateOrientedDatabase.html

What is an aggregate?

When­you­need­to­retrieve­data­from­RDBMS,­you­are­"aggregating"­or­"denormalizing"­the­data­for­your­application­through­queries­with­joins,­where­clauses­and­order­by­clauses.

In­Document­Databases,­instead­of­breaking­data­into­tables­and­foreign­keys,­you­store­the­aggregate­data­together­in­JSON­document(s).

Store and Retrieve Aggregates

• Easier to Distribute Data• More Flexibility• Reduced Latency

order::1001{uid: ji22jd,customer: Ann,line_items: [

{ sku: 0321293533, quan: 3, unit_price: 48.0 },{ sku: 0321601912, quan: 1, unit_price: 39.0 },{ sku: 0131495054, quan: 1, unit_price: 51.0 }

],payment: { type: Amex, expiry: 04/2001,

last5: 12345 }}

Relational vs Document Model

31

Relational­data­modelHighly-structured table organization with rigidly-defined data formats and record structure.

C1 C2 C3 C4

Document­data­modelCollection of complex documents witharbitrary, nested data formats andvarying “record” format.

JSONJSON

JSON

{

}

SQL Normalized Tables

32

Addresses

1 DEN 30303CO

2 MV 94040CA

3 CHI 60609IL

Users

KEY First ZIP_IDLast

4 NY 10010NY

1 Jasdeep 2Jaitla

2 Joe 2Smith

3 Ali 2Dodson

4 John 3Doe

ZIP_ID CITY ZIPSTATE

To­get­information­about­specific­user,­you­perform­a­join­across­two­tables­

foreign key

SELECT­*­FROM­Users­u­INNER­JOIN­Addresses­a­ON­u.zip_id­=­a.zip_id­WHERE­key=1

Documents are Aggregates

33

+

Addresses

1 DEN 30303CO

2

3 CHI 60609IL

4 NY 10010NY

ZIP_ID CITY ZIPSTATE

Users

KEY First ZIP_IDLast

2

2 Joe 2Smith

3 Ali 2Dodson

4 John 3Doe

All­data­in­a­single­document

 ­{­­­­“ID”:­1,­­­­“First”:­“Jasdeep”,­­­­“Last”:­“Jaitla”,­­­­“ZIP”:­“94103”,­­­­“CITY”:­“SF”,­­­­“STATE”:­“CA”­­}

JSON

=

couchbase.get(“user::1”)

Document Data is an Aggregate

1 Jasdeep Jaitla

94103CASF

1 Jasdeep Jaitla

Document Keys

Keys have to be unique in a Bucket (database) which acts as a namespace.

Keys­tend­to­have­three­categories: human-readable, randomly generated, and hybrid

Human­readable­keys­use­unique­descriptors:­email addresses, social media id’s, phone numbers, blog post titles, usernames, sku’s, etc.

www.couchbase.com/develop

PythonRuby

Go Clojure

Official SDKs

Community & Libraries

Couchbase Views

Indexing Architecture

33 2Managed Cache Disk Q

ueue

Disk

Replication Queue

App Server

Couchbase Server Node

Doc­1Doc­1

Doc­1

To other node

View Engine

Doc­1

Doc Updated in RAM Cache First

Indexer Updates Indexes After On Disk, in Batches

All Documents & Updates Pass Through View Engine

Buckets >> Design Documents >> Views

Couchbase Bucket

Design Document 1 Design Document 2

View ViewViewViewView

Indexers Are Allocated Per Design Doc

All Updated at Same TimeAll Updated at Same TimeAll Updated at Same Time

Can Only Access Data in the Bucket NamespaceCan Only Access Data in the Bucket Namespace

Map() Function => Index

function(doc, meta) {emit(doc.username, doc.email)

}indexed key output value(s)create row

json doc doc metadata

Every­Document­passes­through­View­Map()­functions

Map

Single Element Keys (Text Key)

function(doc, meta) {emit(doc.email, null)

}text key

Map

doc.email meta.id

abba@couchbase.com u::1

jasdeep@couchbase.com u::2

zorro@couchbase.com u::3

Compound Keys (Array)

function(doc, meta) {emit(dateToArray(doc.timestamp), 1)

} array key

Array­Based­Index­Keys­get­sorted­as­Strings,­but­can­be­grouped­by­array­elements

Map

dateToArray(doc.timestamp) value

[2012,10,9,18,45] 1

[2012,9,26,11,15] 1

[2012,8,13,2,12] 1

Reduce Values (doc.abv) with _stats

add _stats built-in reduction

Result Set - Brewery ID’s by Beer

brewery_id

document key (of the beer)

alcohol by volume (abv)

Q & A

Thank you!michael.nitschinger@couchbase.com

@daschl

Get Couchbase Server at http://www.couchbase.com/download