a n1ql for every query: couchbase connect 2014

A N1QL for Every QuerySQL and More for a Document Database

Gerald Sangudi | Chief Architect, Couchbase

@sangudi

Rich data

Language

Architecture

Ecosystem

Mission

Roadmap

Resources

Demo

Q & A

Agenda — Previewing N1QL

©2014 Couchbase, Inc. 2

Rich Data

A Basic Data Model

©2014 Couchbase, Inc.

4

Tables, schemas, constraints

Prevailing model for 30+ years

A Rich Data Model


5

Person

Name

DOB

Billing

Connections

PurchasesProduct

Structure and richness of real-world data

Drivers: Big Data, agility

Enabler: JSON — convenient, standard

Comparing the Data Models


Basic Data Model Rich Data Model

Technology Relational Document

Building blocks

Flat tables Rich objects

Developer impact

Real world must be translated

Objects must be assembled

Data must be uniform

Change must be an exception

Real world can be reflected

Objects can be represented

Data can be uniform or varied

Change can be a norm

Query language SQL N1QL — SQL and more

Language

N1QL Defined


8

JSON: Rich DataSQL: Expressive Power N1: Nested Algebra

N1QL: Query Language for Rich Data

Systematic language design

Established theory & model

SQL++ benchmark & presentation

Features from SQL — Reads

©2014 Couchbase, Inc. 9*Upcoming

Reading Data

SELECT Projection

DISTINCT De-duplication

FROM Sourcing

JOIN INNER, LEFT OUTER

WHERE Filtering

GROUP BY Aggregation — HAVING, MIN, MAX, SUM, AVG, COUNT [ DISTINCT ]

ORDER BY Sorting

LIMIT, OFFSET Paging

UNION*, INTERSECT*, EXCEPT* Set operators

EXPLAIN Analyzing and tuning query execution plans

Features from SQL — Writes


*Upcoming

Writing Data

CREATE INDEX Create a Couchbase view index or secondary index*

DROP INDEX Drop an index

INSERT* INSERT using VALUES or SELECT

UPDATE* UPDATE…WHERE, with document-level atomicity

DELETE* DELETE FROM…WHERE, with document-level atomicity

MERGE* MERGE with INSERT or UPDATE, depending on match

Features from SQL — Expressions


*Upcoming

Expressions

Literals Primitives [ 0, ‘hello’, TRUE ]

NULL

Operators

Arithmetic [ +, -, *, /, % ]

Logical [ AND, OR, NOT ]

Comparison [ <, <=, =, !=, >=, >, BETWEEN, IS NULL ]

Pattern matching [ LIKE ]

Conditional [ CASE ]

Scalar functions

Numeric [ trigonometric, ROUND, TRUNC, … ]

String [ UPPER, LOWER, TRIM, SUBSTR, … ]

Date [ string and numeric dates, NOW, date arithmetic*, … ]

Aggregate functions MIN, MAX, SUM, AVG, COUNT [ DISTINCT ]

Subqueries* Subqueries are full expressions

Features for Rich Data — Nested Model


Nested Data

Multi-valued attributes Arrays as attributes

Nested objects Multi-level nesting of objects; path navigation

NEST Collecting second term into nested array [ INNER, LEFT OUTER ]

UNNEST Flattening nested array [ INNER, LEFT OUTER ]

Collection operators Ranging, filtering, predicate, lookup, and slicing operators

Collection functions Sort, Reverse, Distinct, Append, Concatenate, Contains…

Collection aggregation* MIN, MAX, SUM, AVG, COUNT [ DISTINCT ]

Deep traversal* Finding or collecting of matching elements WITHIN any depth

Array and deep update* UPDATE of matching elements IN arrays and WITHIN any depth

Construction Dynamic construction of objects, arrays, and their combinations

*Upcoming

Heterogeneous Data

Heterogeneous inputs Input data of varying structure and type

Heterogeneous results Results of varying structure and type

Raw projection Support for non-tuple, non-object results

Schema inference* Inference of result structure and types, based on result expressions

Missing attributes IS [ NOT ] MISSING and other operators and functions

Attribute removal* UPDATE…UNSET to remove attributes

Total ordering* Comparison and ordering for all values, across all types

Implicit types* Type-encoding functions + expression indexes*

For example, range scans on dates

Closure JSON in, JSON out

Features for Rich Data — Heterogeneity

©2014 Couchbase, Inc. 13*Upcoming

Features for Distributed Data


*Upcoming

Distributed Data

Key-value access KEYS available in SELECT, UPDATE, DELETE

KEYS used in JOIN, NEST, subqueries*, MERGE*

Document metadata META function to access ID, TTL, CAS, FLAGS…

LIMIT on UPDATE & DELETE* LIMIT available in write statements

Architecture

Topology


Client SDK

App

Index

CB Node

xDBC

App

Query

Manager

Data

Index

CB Node

Query

Manager

Data

Index

CB Node

Query

Manager

Data

Index

CB Node

Query

Manager

Data

Index

CB Node

Query

Manager

Data

Index

CB Node

Query

Manager

Data

Homogeneous node image (same bits)

Node roles & deployment flexibility

Independent resourcing & scaling

Query throughput & availability

Query Execution


Index

Client

Data

FetchScanParse Plan Join Filter

Pre-Aggregate

Offset Limit Project

Data-parallel — Query is N data streams over N cores*

Memory-based

Pluggable architecture — datastore, index…

Request Response

*Upcoming

SortAggregate

ProjectorIndex

KeysRouter

Index Topology

Query Interface

Indexer

Indexer

Indexer

Indexer

Write Interfaces

Distributed

Document Store

Write Workload

DCP Mutation Stream

Timer

Incremental

Update

Index

Scan

Timestamp

Timestamp

Re-balancerTopology

Changes

Coordinator

Query Statistics

Usage Statistics

Secondary Indexing

Ecosystem


xDBC

App

CB Node

xDBC

ETL

xDBC

BI

xDBC

Visualization

CB Node CB Node

Standards-based drivers

Simba partnership & presentation

Direction

Mission


Power for Big Data Applications

Ability to index and query rich data at scale

Power of Couchbase extended to querying

Joy for Developers

Freedom, flexibility, alignment, productivity

Best of both — SQL and rich data

Value for Customers

Query technology for a new era

Enabler for applications of the future

Roadmap


DP1 - Start

DP2 - Fixes

DP3 - Joins

DP4 - Language

Alpha - Integration

Beta - PerformanceGA - Production

9/13

11/13

3/14

Q4 ‘14

H1 ‘15

CB Connect

10/14

Resources


query.couchbase.com

Downloads

Tutorial

Documentation

Blogs

Webinars

Forum

@N1QL

Follow us

Engage

Thank you!

@sangudi

PS — Visit sessions and download Couchbase 3.0

a n1ql for every query: couchbase connect 2014

Data & Analytics