couchbase 101 - installation of couchbase
DESCRIPTION
Learn about the Architecture of Couchbase and why it’s so fast, installation of Couchbase on various platforms, setup and configuration parameters, monitoring and the Admin console, and scaling Couchbase. What will be covered during this training: Couchbase Architecture Setup parameters and options Admin Console Monitoring Graphs and what they mean Buckets RAM/CPU/IO Writers Scaling, Clustering and Rebalancing Couchbase in the Cloud Using AMI installations References to Docs, Communities, etc.TRANSCRIPT
Tuesday, October 8, 13
Technical Evangelist
twi0er: @scalabl3email: [email protected]
Jasdeep Jaitla
Couchbase 101 -‐ The Server
Tuesday, October 8, 13
Technical Evangelist
twi0er: @scalabl3email: [email protected]
Jasdeep Jaitla
Couchbase 101 -‐ The Server
Tuesday, October 8, 13
Tuesday, October 8, 13
ARCHITECTURE
Tuesday, October 8, 13
Evolution from memcached
•Founders were key contributors to memcached
•Evolved into Membase, a distributed and persisted key-‐value store
•Evolved into Couchbase Document Store with JSON support and Map-‐Reduce Indexes, ElasOc Search IntegraOon, and Cross-‐Data Center ReplicaOon
Tuesday, October 8, 13
Couchbase Server Core Principles
Easy Scalability
Consistent High Performance
Always On 24x365
Grow cluster without applica9on changes, without down9me with a single click
Consistent sub-‐millisecond read and write response 9mes with consistent high throughput
No down9me for soBware upgrades, hardware maintenance, etc.
Flexible Data Model
JSON Anywhere document model with no fixed schema.
JSONJSONJSON
JSONJSON
PERFORMANCE
Tuesday, October 8, 13
Couchbase Server 2.0 Architecture
Heartbeat
Process m
onito
r
Glob
al singleton supe
rviso
r
Confi
gura9o
n manager
on each node
Rebalance orchestrator
Nod
e he
alth m
onito
r
one per cluster
vBucket state and
replica9
on m
anager
h"pRE
ST m
anagem
ent A
PI/W
eb UI
HTTP8091
Erlang port mapper4369
Distributed Erlang21100 -‐ 21199
Erlang/OTP
storage interface
Couchbase EP Engine
11210Memcapable 2.0
Moxi
11211Memcapable 1.0
Memcached
New Persistence Layer
8092Query API
Que
ry Engine
Data Manager Cluster Manager
Tuesday, October 8, 13
Couchbase Server 2.0 Architecture
New Persistence Layer
storage interface
Couchbase EP Engine
11210Memcapable 2.0
Moxi
11211Memcapable 1.0
Object-‐level Cache
Disk Persistence
8092Query API
Que
ry Engine
HTTP8091
Erlang port mapper4369
Distributed Erlang21100 -‐ 21199
Heartbeat
Process m
onito
r
Glob
al singleton supe
rviso
r
Confi
gura9o
n manager
on each node
Rebalance orchestrator
Nod
e he
alth m
onito
r
one per cluster
vBucket state and
replica9
on m
anager
h"p
REST m
anagem
ent A
PI/W
eb UI
Erlang/OTP
Server/Cluster Management & Communica6on
(Erlang)
RAM Cache, Indexing & Persistence Management
(C)
Tuesday, October 8, 13
Couchbase OrganizaOon
• Couchbase operates like a Key-‐Value Document Store
• Key is a UTF-‐8 string up to 256 Bytes
• Values can be:
-‐ Simple Datatypes: strings, numbers, dateOme, boolean, and binary data can be stored -‐-‐ they are stored as Base64 encoded strings
-‐ Complex Datatypes: dicOonaries/hashes, arrays/lists, can be stored in JSON format (simple lists can be string based with delimiter)
-‐ JSON is a special class of string with a specific format for encoding simple and complex data structures
• Schema is unenforced and implicit, schema changes are programmaOc, done online, and can vary from Document to Document
Tuesday, October 8, 13
Metadata and Documentsmeta{
“id”: “u::[email protected]”,“rev”: “1-‐0002bce0000000000”,“flags”: 0,“expiraUon”: 0,“type”: “json”
}
document{
“uid”: 123456,“firstname”: “jasdeep”,“lastname”: “Jaitla”,“age”: 22,“favorite_colors”: [“blue”, “black”],“email”: “[email protected]”
}
Meta Informa6on Including Key (id)
All Keys Unique and Kept in RAM
Document Value
Most Recent In RAM And Persisted To Disk
Tuesday, October 8, 13
Retrieval Operations
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
Application Server
Replica Couchbase Cluster Machine
Tuesday, October 8, 13
Retrieval Operations
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
Application Server
Replica Couchbase Cluster Machine
get
Tuesday, October 8, 13
Retrieval Operations
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
Application Server
Replica Couchbase Cluster Machine
get
Tuesday, October 8, 13
Storage Operations
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
Application Server
Replica Couchbase Cluster Machine
Tuesday, October 8, 13
Storage Operations
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
Application Server
Replica Couchbase Cluster Machine
set/add/replace
Tuesday, October 8, 13
Storage Operations
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
Application Server
Replica Couchbase Cluster Machine
set/add/replace
Tuesday, October 8, 13
Storage Operations
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
Application Server
Replica Couchbase Cluster Machine
set/add/replace
Tuesday, October 8, 13
Storage Operations
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
Application Server
Replica Couchbase Cluster Machine
set/add/replace
Tuesday, October 8, 13
Consistency
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
Application Server
Replica Couchbase Cluster Machine
Tuesday, October 8, 13
Consistency
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
Application Server
Replica Couchbase Cluster Machine
get
Tuesday, October 8, 13
Ejection, NRU, Cache Miss
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
Application Server
Replica Couchbase Cluster Machine
Tuesday, October 8, 13
Ejection, NRU, Cache Miss
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
Application Server
Replica Couchbase Cluster Machine
set/add/replaceset/add/replaceset/add/replace
Tuesday, October 8, 13
Ejection, NRU, Cache Miss
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
Application Server
Replica Couchbase Cluster Machine
FULL (90%)
NRU Documents Ejected
Tuesday, October 8, 13
Ejection, NRU, Cache Miss
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
Application Server
Replica Couchbase Cluster Machine
Tuesday, October 8, 13
Ejection, NRU, Cache Miss
Couchbase Server
EP EngineRAM Cache
Disk Write Queue
Replication Queue
Application Server
Replica Couchbase Cluster Machine
get
Non-‐Resident Document
("Cache Miss")
Tuesday, October 8, 13
Clients Connect Directly to Couchbase Nodes
Application Servers
MAP
1024
8 GB RAMPartitions
3 IO Workers
Tuesday, October 8, 13
Clients Connect Directly to Couchbase Nodes
Application Servers
MAP
1024
8 GB RAMPartitions
3 IO Workers
Tuesday, October 8, 13
Clients Connect Directly to Couchbase Nodes
Application Servers
MAP
MAP
MAP
1024
8 GB RAMPartitions
3 IO Workers
Tuesday, October 8, 13
Key Hash-Partitioning
Application Servers
MAP
MAP
MAP
1024
8 GB RAMPartitions
3 IO Workers
ClientHashFunc6on("[email protected]") => Par66on[0..1023] {25}ClusterMap[P(25)] => [x.x.x.x] => IP of Server Responsible for Par66on 25
Tuesday, October 8, 13
Application Servers8 GB RAM
3 IO Workers
8 GB RAM
3 IO Workers
Horizontal Scale-Rebalance
TOTAL
16 GB RAM
6 IO Workers
1024
Partitions
1024
Partitions
TOTAL
8 GB RAM
3 IO Workers
1024
Partitions
Tuesday, October 8, 13
Application Servers8 GB RAM
3 IO Workers
8 GB RAM
3 IO Workers
Horizontal Scale-Rebalance
TOTAL
16 GB RAM
6 IO Workers
1024
Partitions1024
Partitions
512
Partitions
512
Partitions
Tuesday, October 8, 13
Horizontal Scale-Rebalance
TOTAL
32 GB RAM
12 IO Workers
1024
Partitions
Application Servers
8 GB RAM
3 IO Workers
8 GB RAM
3 IO Workers
8 GB RAM
3 IO Workers
8 GB RAM
3 IO Workers
512
Partitions
512
Partitions
TOTAL
16 GB RAM
6 IO Workers
1024
Partitions
MAP
MAP MAP
MAP
Tuesday, October 8, 13
Horizontal Scale-Rebalance
TOTAL
32 GB RAM
12 IO Workers
1024
Partitions
Application Servers
8 GB RAM
3 IO Workers
8 GB RAM
3 IO Workers
8 GB RAM
3 IO Workers
8 GB RAM
3 IO Workers
512
Partitions
512
Partitions
256
Partitions
256
Partitions
256
Partitions
256
Partitions
TOTAL
16 GB RAM
6 IO Workers
1024
Partitions
MAP
MAP
MAP
MAP
MAP
MAP
MAP
MAP
Tuesday, October 8, 13
Horizontal Scale-Rebalance
TOTAL
32 GB RAM
12 IO Workers
1024
Partitions
Application Servers
8 GB RAM
3 IO Workers
8 GB RAM
3 IO Workers
8 GB RAM
3 IO Workers
8 GB RAM
3 IO Workers
512
Partitions
512
Partitions
256
Partitions
256
Partitions
256
Partitions
256
Partitions
MAP
MAP
MAP
MAP
MAP
MAP
MAP
MAP
Tuesday, October 8, 13
All Metadata for All Documents
(64 bytes + Key Length)
Document Values(NRU Ejected if RAM Quota
Used > 90%)
Also Leave RAM For OS:
[Filesystem Cache >> Views]
Document Indexing
Monitoring
XDCR
Recommended:
minimum 4 Cores+ 1 core per design document+ 1 core per XDCR replicated
bucket
Persisted Documents
All Indexes for Design Documents/Views
Append-‐Only Disk Format& CompacUon
Performance:
MulUple EBS Volumes High IOPS Raid 0 on Amazon
RAM, CPU and IO Guidelines
RAM CPU Disk IO
Tuesday, October 8, 13
Binary Socket Operations (11210/1)• get (key)
– Retrieve a document
• set (key, value)
– Store a document, overwrites if exists
• add (key, value)
– Store a document, error/exception if exists
• replace (key, value)
– Store a document, error/exception if doesn’t exist
• incr (key)
– Create/Increment Atomic Counter
• decr (key)
– Decrement Atomic Counter
• cas (key, value, cas)
– Compare and swap, set document only if it hasn’t changed (Optimistic Lock)
Tuesday, October 8, 13
HTTP Operations (8092)
• View Querying
– Range Queries
– Index-Key Match Queries
– Set Match Queries
– Aggregate Reduces
– Group Level + Grouping Queries
Tuesday, October 8, 13
Tuesday, October 8, 13
INSTALL SERVER
Tuesday, October 8, 13
Download Couchbasewww.couchbase.com/download
Tuesday, October 8, 13
Install on Amazon via AMI
Tuesday, October 8, 13
Install on Amazon via AMI
Tuesday, October 8, 13
Run Couchbase and Open Browserlocalhost:8091 or ec2-‐xx-‐xx-‐xx-‐xx.compute-‐1.amazonaws.com:8091
Tuesday, October 8, 13
Setup: Step 1 - Storage Paths & RAM Maximum
Tuesday, October 8, 13
Setup: Step 1 - Storage Paths & RAM Maximum
Tuesday, October 8, 13
Setup: Step 2 - Sample Buckets
Tuesday, October 8, 13
Setup: Step 3 - Configure Default Bucket
Tuesday, October 8, 13
Setup: Step 3 - Configure Default Bucket
Tuesday, October 8, 13
Setup: Step 4 - Register & Agree
Tuesday, October 8, 13
Setup: Step 5 - Admin password
Tuesday, October 8, 13
Setup: Complete
Tuesday, October 8, 13
Tuesday, October 8, 13
SETUP SDK
Tuesday, October 8, 13
Supported SDK'swww.couchbase.com/communiUes
• Each supported SDK page has instrucOons for setup
• PHP, Ruby, NodeJS and Python clients are wrappers around libcouchbase C library, so libcouchbase must be installed first
• For other community clients, click on "All Clients" on lei nav, scroll down the page and you can see clients for Go, Erlang, Clojure, TCL, other nodejs and Perl.
Tuesday, October 8, 13
Installing Libcouchbase
• Mac Tips before Libcouchbase & SDK Install
• Make sure you have XCode & Command Line Tools Installed
• Install Homebrew if you don't have it already
• Do a $ brew update, $ brew upgrade and $ brew doctor to be sure you're up to date
www.couchbase.com/communiUes/c/gekng-‐started
Tuesday, October 8, 13
Installing Libcouchbase
• Mac Via Homebrew
• $ brew install libcouchbase
• PC-‐Windows
• Download appropriate Zip from website
• Redhat/CentOS
• wget the yum repositories
• $ sudo yum install -‐y libcouchbase2-‐libevent libcouchbase-‐devel
• Ubuntu
• wget ubuntu repositories
• $ sudo apt-‐get install libcouchbase2-‐libevent libcouchbase-‐dev
www.couchbase.com/communiUes/c/gekng-‐started
Tuesday, October 8, 13
More On Development
Training: Couchbase 102 -‐ Developingwww.couchbase.com/webinars
Tuesday, October 8, 13
Tuesday, October 8, 13
SERVER MONITORING
Tuesday, October 8, 13
Store and Retrieve Operations
Tuesday, October 8, 13
Store and Retrieve Operations
Tuesday, October 8, 13
Store and Retrieve Operations
Storage Ops/sRetrieve Ops/sRetrieve off DiskTotal Ops/s
Tuesday, October 8, 13
Store and Retrieve Operations
Storage Ops/sRetrieve Ops/sRetrieve off DiskTotal Ops/s
Item CountDelete Ops/s CAS Ops/s % Documents in RAM
Tuesday, October 8, 13
Store and Retrieve Operations
Tuesday, October 8, 13
Store and Retrieve Operations
RAM FULL + Very High Velocity Writes!
Tuesday, October 8, 13
Store and Retrieve Operations
RAM FULL + Very High Velocity Writes!
RAM Used for Metadata + Data
Tuesday, October 8, 13
Store and Retrieve Operations
RAM FULL + Very High Velocity Writes!
Begin EjecUon of Replicas from RAM
Begin EjecUon of AcUve from RAM
RAM Used for Metadata + Data
Tuesday, October 8, 13
Store and Retrieve Operations
Tuesday, October 8, 13
Store and Retrieve Operations
Disk Creates Disk Updates Disk Queue SizeDisk Reads (Cache Miss)
Tuesday, October 8, 13
Store and Retrieve Operations
Disk Creates Disk Updates Disk Queue SizeDisk Reads (Cache Miss)
Total Disk UsageDocs Data Size (Compressed) Docs Size on Disk FragmentaUon
(from Appends)
Tuesday, October 8, 13
Store and Retrieve Operations
Disk Queue Size
Tuesday, October 8, 13
Store and Retrieve Operations
Disk Queue SizeDisk Queue Size(Need More IO!)
Tuesday, October 8, 13
Partition (vbucket) Details
Tuesday, October 8, 13
Partition (vbucket) Details
Tuesday, October 8, 13
Partition (vbucket) DetailsAcUve ParUUons
Tuesday, October 8, 13
Partition (vbucket) DetailsAcUve ParUUons Replica ParUUons
Tuesday, October 8, 13
Partition (vbucket) DetailsAcUve ParUUons Replica ParUUons Pending ParUUons
(Rebalance)
Tuesday, October 8, 13
Partition (vbucket) Details
Tuesday, October 8, 13
Partition (vbucket) Details
Item Count
Tuesday, October 8, 13
Partition (vbucket) Details
% Documents in RAM
Item Count
Tuesday, October 8, 13
Partition (vbucket) Details
% Documents in RAM
CreaUons
Item Count
Tuesday, October 8, 13
Partition (vbucket) Details
EjecUons (RAM Full)
% Documents in RAM
CreaUons
Item Count
Tuesday, October 8, 13
Partition (vbucket) Details
Documents
EjecUons (RAM Full)
% Documents in RAM
CreaUons
Item Count
Tuesday, October 8, 13
Partition (vbucket) Details
Metadata
Documents
EjecUons (RAM Full)
% Documents in RAM
CreaUons
Item Count
Tuesday, October 8, 13
Partition (vbucket) Details
Tuesday, October 8, 13
Partition (vbucket) Details
Note: 64 Par66ons on Mac OS X! This is simply to avoid altering default file descriptor limits,
don't XDCR with other Opera6ng Systems
Tuesday, October 8, 13
Disk IO Information
Tuesday, October 8, 13
Disk IO Information
AcUve ParUUons
Tuesday, October 8, 13
Disk IO Information
AcUve ParUUons Replica ParUUons
Tuesday, October 8, 13
Disk IO Information
AcUve ParUUons Replica ParUUons Pending ParUUons (Rebalance)
Tuesday, October 8, 13
Disk IO Information
Tuesday, October 8, 13
Disk IO Information
Queue Size
Tuesday, October 8, 13
Disk IO Information
Queue Fill Rate
Queue Size
Tuesday, October 8, 13
Disk IO Information
Queue Fill Rate
Queue Drain Rate
Queue Size
Tuesday, October 8, 13
Disk IO Information
Average Age of Queue Items
Queue Fill Rate
Queue Drain Rate
Queue Size
Tuesday, October 8, 13
Tuesday, October 8, 13
QUICK DEMO
Tuesday, October 8, 13
Tuesday, October 8, 13
Q & A
Tuesday, October 8, 13
Focused on Beginning DevelopmentThis Thursday!
www.couchbase.com/webinars
Tuesday, October 8, 13
COUCHBASE 102
Focused on Beginning DevelopmentThis Thursday!
www.couchbase.com/webinars
Tuesday, October 8, 13