what's new in couchbase server 4.0 – couchbase live new york 2015

What’s New InCouchbase Server 4.0

Chin Hong | VP Product Management, Couchbase

©2015 Couchbase Inc. 2

Simplified Development

Connected Bigdata Experience

Improved Performance

Simplified Security Compliance

Improved HA & DR

Improved Provisioning

Simplified Familiar and Flexible Query with N1QL

Integrated BI with ODBC/JDBC

Spatial Queries for Location Aware Applications

New Frameworks and Languages - LINQ, - Spring, - Go

Surround Big-data - Spark SQL- Spark Streams- Kafka, - Sqoop, - Elastic, - SOLR

Global Secondary Indexes for Snappy Queries

Independent Scaling with Multi-dimensional Scaling

Faster Reporting and Interactive Analytics with Views Queries

Enhanced Storage Performance with ForestDB and Bloom Filters

Integrated Enterprise Identity Management with LDAP Integration

Security Forensics with Admin Auditing

Improved Data Protection with Lower latency XDCR

High Performance Global Data Distribution with XDCR Filtering

Deployment with High Performance Containers: Docker

Expanded Public and Private Cloud Support- AWS,- Google,- Azure,- Joyent,- Cisco,- Verizon

New Enterprise Platforms- SUSE - Oracle Ent.Linux

13

2

2 55

4

N1QL

©2015 Couchbase Inc. 4

What is N1QL?

4

Flexibility of JSON

Power of SQL

©2015 Couchbase Inc. 5

How Does One Evaluate a Query Language?

ExpressiveDeclarativeEcosystem

©2015 Couchbase Inc. 6

N1QL is Expressive: E-Commerce Examples

SELECT product.name, SUM(items.count) AS unitsSold FROM purchases UNNEST purchases.lineItems AS items JOIN product ON KEYS items.product GROUP BY product ORDER BY unitsSold DESC LIMIT 10

SELECT SUBSTR(purchases.purchasedAt, 0, 7) AS month, ROUND(SUM(product.unitPrice * items.count)/1000000, 3) revenueMillionFROM purchases UNNEST purchases.lineItems AS items JOIN product ON KEYS items.product GROUP BY SUBSTR(purchases.purchasedAt, 0, 7) ORDER BY month

©2015 Couchbase Inc. 7

N1QL is Expressive: Image Processing Example

SELECT m.image AS gallery_image, s.image AS source_image, s.x, s.y, m.r AS gallery_r, m.g AS gallery_g, m.b AS gallery_b, s.r AS source_r, s.g AS source_g, s.b AS source_b,FROM SourcePixels sJOIN GalleryMeans m ON KEY s.rgb

Source: http://www.metanautix.com/tr01

Learn more about N1QL @ query.couchbase.com/tutorial

©2015 Couchbase Inc. 8

N1QL is Declarative: What vs. How

You specify WHATCouchbase Server figures out HOW

©2015 Couchbase Inc. 9

N1QL is Declarative: Top 10 Selling ProductsSELECT product.name, SUM(items.count) AS unitsSold FROM purchases UNNEST purchases.lineItems AS items JOIN product ON KEYS items.product GROUP BY product ORDER BY unitsSold DESC LIMIT 10

WHAT

HOW

©2015 Couchbase Inc. 10

N1QL Connects To Your Enterprise Ecosystem

Standards-based drivers

Integrations, partnerships

ODBC / JDBC

App

CB Node

ODBC / JDBC

ETL

ODBC / JDBC

BI

ODBC / JDBC

Visualization

CB Node CB Node

©2015 Couchbase Inc. 11

Sessions on N1QL Introducing N1QL: New SQL Based Query Language for JSON

1:15 pm – 1:55 pm, Grand Ballroom N1QL and SDK Support for Java, .NET, and Node.js

3:50 pm – 4:30 pm, The Library Deep Dive into N1QL with Global Secondary Indexes

4:35 pm – 5:15 pm, Grand Ballroom Agile Data Modeling at DIRECTV

2:00 pm – 2:40 pm, Stillman Evolution of Couchbase at CenterEdge: From Cache to Query

3:50 pm – 4:30 pm, Stillman Nielsen’s Interactive Data Analytics with Couchbase N1QL

4:35 pm – 5:15 pm, Stillman Introduction to BI with Couchbase Server using Tableau, Informatica, Excel

and more4:35 pm – 5:15 pm, The Library

Indexing

©2015 Couchbase Inc. 13

Why Do We Need Indexes?

Indexes are used to quickly look up objects

meeting user-specified criteria without having to search over every

object in the database.

©2015 Couchbase Inc. 14

What Are the Different Types of Indexes?

Primary IndexesSecondary Indexes

Spatial IndexesText Indexes

©2015 Couchbase Inc. 15

Index Types … Let’s Use the Library Analogy

Source: http://www.montel.com/en/markets/library/

©2015 Couchbase Inc. 16

Secondary IndexesLocal secondary indexes – Map/Reduce views in 3.0 Co-located with data Higher write performance Lower read performance: scatter-gather Scaling bottleneck with a high number of indexes or data

nodes

Global secondary indexes (GSI) – New in 4.0 Higher query performance Async writes to a large number of global indexes Independently scaled and partitioned Isolated from Key-Value operations

©2015 Couchbase Inc. 17

Box on Points Box Intersects Multi Dimensions Geo + Others

Spatial Indexes Experimental in 3.0, now GA in 4.0

– Performance and stability improvements GeoJSON output Bounding-box and range queries on multiple dimensions

©2015 Couchbase Inc. 18

Text Indexes (Developer Preview)

…

wise

…

engineer……

…

…, akay1980, …

…, akay1980, …

engineers

engineer Exact

Match

Inverted Index

Apply the sametext analysisat search timethat we usedat index time.

©2015 Couchbase Inc. 19

Sessions on Indexes Understanding Views in Couchbase Server

11:45 am – 12:25 pm, Screening Room Deep Dive into N1QL with Global Secondary Indexes

4:35 pm – 5:15 pm, Grand Ballroom Developer Preview of Full Text Search

labs.couchbase.com/cbft

Multi-Dimensional Scaling

©2015 Couchbase Inc. 21

Workload and ScalingHorizontal scaling Partitions a dataset onto one or more homogenous nodes Each node runs the same mixed workloads Re-partition dataset with additional hardware capacity

Node 8

Index Service

Couchbase Cluster

Query Service

Data Service

Node 1

©2015 Couchbase Inc. 22

Multi-Dimensional ScalingIsolated Service for minimized interference Independent “zones” for query, index, and data services

Minimize indexing and query overhead on core key-value operations.

Index Service

Couchbase Cluster

Query Service Data Service

Node 1 Node 8

©2015 Couchbase Inc. 23

Multi-Dimensional ScalingIndependent scalability for the best computational capacity per service

Couchbase Cluster

node1 Node 8 Node 9

Data Service

Index ServiceQuery Service

More storage for indexing: scale up index service nodes.

More cores for query processing:scale up query service nodes.

Node 1

©2015 Couchbase Inc. 24

Session on Multi-Dimensional Scaling Deep Dive into N1QL with Global Secondary Indexes

4:35 pm – 5:15 pm, Grand Ballroom

ForestDB: Next-Generation Storage Engine

©2015 Couchbase Inc. 26

Common Storage Structures – B+-Tree

26

Reads Writes EnginesB+-Tree Good read performance if the

fan-out is high (“short” tree) for small fixed-length keys. Read performance degrades for variable-length keys.

Update-In-Place results in random writes and bad write latency.Append-Only file improves write performance but requires periodic compaction.

BerkeleyDB, InnoDB, Couchstore (append-only), WiredTiger B+

04/26

…

…

…Key

Value (or Pointer)

longer keys

©2015 Couchbase Inc. 27

Common Storage Structures – LSM-Tree

27

Reads Writes EnginesLSM-Tree

Reads may have to traverse multiple trees – typically worse than B+ tree.

WAL(Write-Ahead Log) improves writes within-memory trees that are appended to the end of the log.

LevelDB, RocksDB, Cassandra, WiredTiger LSM

…

In-memory

Sequential log

flush/merge merge

C1 tree C2 tree

merge

Capacity increases exponentially

©2015 Couchbase Inc. 28

Fast Storage – ForestDB

Fundamentally rethinking the requirements for fast storage Compact and efficient storage for a variety of data Unified engine that performs well for various workloads Unified engine that scales from small devices to large

servers Optimized for new SSD storage technology

28

ForestDB provides a compact index structure

that significantly reduces the disk I/Oand write amplification.

©2015 Couchbase Inc. 29

ForestDBForestDB Evaluations: LevelDB, RocksDB

29

Throughput

UP TO

FASTER6X

Future Proof

FITTED

FORSSD

Efficiency

UP TO

5X

COMPACT

©2015 Couchbase Inc. 30

Trie (prefix tree) for which the nodes are B+-Trees– HB+-Trie was originally presented at ACM SIGMOD 2011 Programming

Contest, by Jung-Sang Ahn who works at Couchbase(http://db.csail.mit.edu/sigmod11contest/sigmod_2011_contest_poster_jungsang_ahn.pdf)

HB+-Trie (Hierarchical B+-Trie)

©2015 Couchbase Inc. 31

Sessions on ForestDB Next Generation Storage Engine: ForestDB

5:20 pm – 6:00 pm, Grand Ballroom

Security

©2015 Couchbase Inc. 33

Previously… In 2.2 In 2.5 In 3.0 New in 4.0

SASL AuthN with Bucket PasswordsAdmin UserSecure Build

Platform

Read-Only User

Easy Admin Password

Reset

Non-Root User

Deployments

Secure Communication for XDCR

Encrypted Client- Server

Communication

Encrypted Admin Access

Access LogData-at-Rest Encryption

• Simplified compliance with admin auditing

• External identity management for admins using LDAP

Couchbase Security Features

In a fewslides ..

33

©2015 Couchbase Inc. 34

External Identity Management Using LDAP

34

Centralized identity management Define multiple read-only admins and full-admins Centralized security policy management for

admin accounts for stronger passwords, password rotation, and auto lockouts

Individual accountability; simplified compliance Define UIDs in LDAP, and map UIDs to the read-

only/full admin role in Couchbase Comprehensive audit trails with LDAP UIDs in

audit records

©2015 Couchbase Inc. 35

Admin Auditing in Couchbase

35

Rich audit events Over 25+ different, detailed admin audit events Auditing for tools including backup

Configurable auditing Configurable file target Support for time-based log rotation and audit

filtering

Easy integration JSON format allows for easy integration with

downstream systems using flume, logstash, and syslogd

©2015 Couchbase Inc. 36

Auditing a successful login

36

{ "timestamp":"2015-02-20T08:48:49.408-08:00", "id":8192, "name":"login success", "description":"Successful login to couchbase cluster", "role":"admin", "real_userid": { "source":"ns_server", "user":"bjones” }, "sessionid":"0fd0b5305d1561ca2b10f9d795819b2e", "remote":{"ip":"172.23.107.165", "port":59383}}

WHEN

WHO

WHAT

HOW

©2015 Couchbase Inc. 37

Sessions on Security Securing Your Couchbase Server Deployment

2:45 pm –3:25 pm, Grand Ballroom Transparent Data Encryption in Couchbase using Vormetric

3:50 pm –4:30 pm, Grand Ballroom

©2015 Couchbase Inc. 38

Simplified Development

Connected Bigdata Experience

Improved Performance

Simplified Security Compliance

Improved HA & DR

Improved Provisioning

Simplified Familiar and Flexible Query with N1QL

Integrated BI with ODBC/JDBC

Spatial Queries for Location Aware Applications

New Frameworks and Languages - LINQ, - Spring, - Go

Surround Big-data - Spark SQL- Spark Streams- Kafka, - Sqoop, - Elastic, - SOLR

Global Secondary Indexes for Snappy Queries

Independent Scaling with Multi-dimensional Scaling

Faster Reporting and Interactive Analytics with Views Queries

Enhanced Storage Performance with ForestDB and Bloom Filters

Integrated Enterprise Identity Management with LDAP Integration

Security Forensics with Admin Auditing

Improved Data Protection with Lower latency XDCR

High Performance Global Data Distribution with XDCR Filtering

Deployment with High Performance Containers: Docker

Expanded Public and Private Cloud Support- AWS,- Google,- Azure,- Joyent,- Cisco,- Verizon

New Enterprise Platforms- SUSE - Oracle Ent.Linux

13

2

2 55

4

Get Started with Couchbase Server 4.0: developer.couchbase.com/server

Get Trained on Couchbase: training.couchbase.com