introducing n1ql: new sql-based query language for json – couchbase live new york 2015

Introducing N1QL: New SQL-Based Query Language

for JSON

Keshav Murthy| @rkeshavmurthyDirector, Query | Couchbase

Team @N1QL

NoSQL

NoRDBMSNoRelationalNoSchema

NoTransactions

NoACID

NoConstraintsNoTriggers

NoScaleUp

#First get all the customerscustomerlist = self.customer.find({},{"c_id": 1, "c_name":1 })

#initialize the total listtotlist = set()

#For each customer in customer list#Create a pipeline to get total order PER CUSTOMER#Get the SUM of ol_amount for the customer#Add the customerid, customer name and total to the totlist (list of tuples)for cl in customerlist: pipe = [{'$unwind': "$order_line"}, {'$group':{'_id':cl, 'total':{'$sum':'$ol_amount'}}}] csum = db.orders.aggregate(pipeline=pipe) totlist.add(cl["c_id"], cl["c_name"], csum["total"])

#sort this full list by the total in the reverse order.data.sort(key=lambda tup: tup[2], reverse=True)

Application Developers Started Writing Complex

Code for Queries…With Serious

Performance Issues

©2015 Couchbase Inc. 5

SQL


SQLResultS

et


{ "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" }, { "type" : "master", "cardnum" : "6274-2842-2847-3909", "expiry" : "2019-03" } ], "Connections" : [ { "CustId" : "XYZ987", "Name" : "Joe Smith" }, { "CustId" : "PQR823", "Name" : "Dylan Smith" } { "CustId" : "PQR823", "Name" : "Dylan Smith" } ], "Purchases" : [ { "id":12, item: "mac", "amt": 2823.52 } { "id":19, item: "ipad2", "amt": 623.52 } ]}

LoyaltyInfo ResultDocuments

PricingInfo

CUSTOMER


N1QL examplesSELECT d.C_ZIP, SUM(ORDLINE.OL_QUANTITY) AS TOTALQTYFROM CUSTOMER d UNNEST ORDERS as CUSTORDERS UNNEST CUSTORDERS.ORDER_LINE AS ORDLINEWHERE d.C_STATE = ”NY”GROUP BY d.C_ZIPORDER BY TOTALQTY DESC;

INSERT INTO CUSTOMER("PQR847", {"C_ID":4723, "Name":"Joe"});

UPDATE CUSTOMER c SET c.STATE=“CA”, c.C_ZIP = 94501 WHERE c.ID = 4723;

Data in JSON


Data Comes from the Real World


Properties of Real-World Data

Rich structure– Attributes, Sub-structure

Relationships– To other data

Value evolution– Data is updated

Structure evolution– Data is reshaped

Customer

Name

DOB

Billing

Connections

Purchases

Jane Smith

Jan-30-1990


Modeling Customer Data in Relational World

Billing

Connections

Purchases

Contacts

Customer

Rich structure Normalize & JOIN

Queries Relationships

JOINS and Constraints Value evolution

Additional ROWS, JOINS Structure evolution

ALTER TABLE Application Downtime Application Migration Application Versioning


Using JSON For Real World Data

CustomerID

Name DOB

CBL2015 Jane Smith 1990-01-30

Table: Customer{ "Name" : "Jane Smith", "DOB" : "1990-01-30”}

Customer DocumentKey: CBL2015

The primary (CustomerID) becomes the DocumentKey

Column name-Column value become KEY-VALUE pair.

{ "Name" : { "fname": "Jane ", "lname": "Smith” } "DOB" : "1990-01-30”}

OR


Using JSON to store Data

CustomerID

Name DOB


Table: Customer

{ "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" } ]}


CustomerID

Type Cardnum

Expiry

CBL2015 visa 5827… 2019-03

Table: Billing

Rich Structure & Relationships– Billing Information is stored as a sub-document– There could be more than a single credit card. So, use an array.



CustomerID

Name DOB


Table: Customer

{ "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" }, { "type" : "master", "cardnum" : "6274-2542-5847-3949", "expiry" : "2018-12" } ]}


CustomerID

Type Cardnum

Expiry

CBL2015 visa 5827… 2019-03

CBL2015 master

6274… 2018-12

Table: Billing

Value evolution Simply add additional array

element Relationships are implied



CustomerID

ConnId Name

CBL2015 XYZ987 Joe Smith

CBL2015 SKR007 Sam Smith

Table: Connections{ "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" }, { "type" : "master", "cardnum" : "6274-2542-5847-3949", "expiry" : "2018-12" } ], "Connections" : [ { "ConnId" : "XYZ987", "Name" : "Joe Smith" }, { "ConnId" : ”SKR007", "Name" : ”Sam Smith" }}


Structure evolution Simply add new key-value pairs No downtime to add new KV

pairs Applications can validate data Structure evolution over time.

Relations via Reference


Using JSON to store Data{ "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" }, { "type" : "master", "cardnum" : "6274-2842-2847-3909", "expiry" : "2019-03" } ], "Connections" : [ { "CustId" : "XYZ987", "Name" : "Joe Smith" }, { "CustId" : "PQR823", "Name" : "Dylan Smith" } { "CustId" : "PQR823", "Name" : "Dylan Smith" } ], "Purchases" : [ { "id":12, item: "mac", "amt": 2823.52 } { "id":19, item: "ipad2", "amt": 623.52 } ]}



Models for Representing Data

Data Concern Relational Model JSON Document Model (NoSQL)

Rich Structure Multiple flat tables Constant assembly /

disassembly Documents No assembly required!

Relationships Represented Queried (SQL)

Represented Queried? Not until

now…Value Evolution Data can be updated Data can be updated

Structure Evolution

Uniform and rigid Manual change

(disruptive) Flexible Dynamic change

The N1QL Language


The N1QL Language

GoalGive developers and enterprises an expressive, powerful, and complete language for querying, transforming, and manipulating JSON data.


The N1QL LanguageQuery Features SQL N1QL

Statements SELECT, INSERT, UPDATE, DELETE, MERGE

SELECT, INSERT, UPDATE, DELETE, MERGE

Query Operations

Select, Join, Project, Subqueries

Strict Schema Strict Type checking

Select, Join, Project, Subqueries

Nest & Unnest Look Ma! No Type Mismatch

Errors! JSON keys act as columns

Schema Predetermined Columns Fully addressable JSON Flexible document structure

Data Types SQL Data types Strings, Integer, Float,

Boolean, arrays, objects Scalar conversion Functions

Query Processing

INPUT: Rows OUPUT: Rows

INPUT: JSON OUTPUT: JSON


SELECT Statement

SELECT [ DISTINCT ] …FROM … JOIN …WHERE …GROUP BY … HAVING …ORDER BY …LIMIT …OFFSET …( UNION | INTERSECT | EXCEPT ) [ ALL ] …


SELECT StatementSELECT customers.id, customers.NAME.lastname, customers.NAME.firstname Sum(orderline.amount) FROM orders UNNEST orders.lineitems AS orderline JOIN customers ON KEYS orders.custid WHERE customers.state = 'NY'GROUP BY customers.id, customers.NAME.lastname HAVING sum(orderline.amount) > 10000 ORDER BY sum(orderline.amount) DESC

• Dotted sub-document reference

• Names are CASE-SENSITIVE

• UNNEST to flatten the arrays

JOINS with Document KEY of customers


SELECT StatementSELECT * FROM ( SELECT a, b, c FROM cust WHERE x = 1 ORDER BY x LIMIT 100 OFFSET 0 UNION ALL SELECT a, b, c FROM cust WHERE y = 2 ORDER BY x LIMIT 100 OFFSET 0) AS newtab LEFT OUTER JOIN contacts ON KEYS newtab.c.contactidORDER BY a, b, cLIMIT 10 OFFSET 0


SELECT Statement Highlights Querying across relationships

– JOINs– Subqueries

Aggregation– MIN, MAX– SUM, COUNT, AVG, ARRAY_AGG [ DISTINCT ]

Combining result sets using set operators– UNION, UNION ALL, INTERSECT, EXCEPT


Data Modification Statements

UPDATE … SET … WHERE … DELETE FROM … WHERE … INSERT INTO … ( KEY, VALUE ) VALUES

… INSERT INTO … ( KEY …, VALUE … )

SELECT … MERGE INTO … USING … ON …

WHEN [ NOT ] MATCHED THEN …Note: Couchbase Server provides per-document

atomicity.



INSERT INTO ORDERS (KEY, VALUE) VALUES ("1.ABC.X382", {"O_ID":482, "O_D_ID":3, "O_W_ID":4)};

UPDATE ORDERS SET O_CARRIER_ID = ”ABC987” WHERE O_ID = 482 AND O_D_ID = 3 AND O_W_ID = 4

DELETE FROM NEW_ORDER WHERE NO_D_ID = 291 AND NO_W_ID = 3482 AND NO_O_ID = 2483



INSERT INTO ORDERS (KEY, VALUE) VALUES ("1.ABC.X382", {"O_ID":482, "O_D_ID":3, "O_W_ID":4)}),VALUES ("1.ABC.X383", {"O_ID":422, "O_D_ID":3, "O_W_ID":4)}),VALUES ("1.ABC.X384", {"O_ID”:782, "O_D_ID":3, "O_W_ID":4)}),VALUES ("1.ABC.X352", {"O_ID”:282, "O_D_ID":3, "O_W_ID":4)}),VALUES ("1.ABC.X362", {"O_ID”:582, "O_D_ID":3, "O_W_ID":4)}),VALUES ("1.ABC.X322", {"O_ID”:682, "O_D_ID":3, "O_W_ID":4)}),VALUES ("1.ABC.X389", {"O_ID”:982, "O_D_ID":3, "O_W_ID":4)}),VALUES ("1.ABC.X392", {"O_ID”:182, "O_D_ID":3, "O_W_ID":4)});


Index Statements

CREATE INDEX ON … DROP INDEX … EXPLAIN …Highlights Functional indexes

– on any data expression

Partial indexes

©2014 Couchbase, Inc. ©2015 Couchbase Inc. 30

N1QL Expressions from SQL

30

Literals• Primitives [ 0, ‘hello’, TRUE ]• NULL

Operators

• Arithmetic [ +, -, *, /, % ]• Logical [ AND, OR, NOT ]• Comparison [ <, <=, =, !=, >=, >, BETWEEN, IS NULL ]• Pattern matching [ LIKE ]• Conditional [ CASE ]

Scalar functions

• Numeric [ trigonometric, ROUND, TRUNC, … ]• String [ UPPER, LOWER, TRIM, SUBSTR, … ]• Date [ string and numeric dates, NOW, date arithmetic, …

]Aggregate functions • MIN, MAX, SUM, AVG, COUNT [ DISTINCT ]

Subqueries • Subqueries are full expressions

N1QL Extensions to SQL


N1QL Query Operators [ 1 of 2 ]

USE KEYS …– Direct primary key lookup bypassing index scans– Ideal for hash-distributed datastore– Available in SELECT, UPDATE, DELETE

JOIN … ON KEYS …– Nested loop JOIN using key relationships– Ideal for hash-distributed datastore


N1QL Query Operators [ 2 of 2 ]

NEST– Special JOIN that embeds external child documents

under their parent– Ideal for JSON encapsulation

UNNEST– Flattening JOIN that surfaces nested objects as top-

level documents– Ideal for decomposing JSON hierarchies

JOIN, NEST, and UNNEST can be chained in any

combination

©2014 Couchbase, Inc. ©2015 Couchbase Inc. 34

N1QL Expressions for JSON

34

Ranging over collections

• WHERE ANY c IN children SATISFIES c.age > 10 END• WHERE EVERY r IN ratings SATISFIES r > 3 END

Mapping with filtering • ARRAY c.name FOR c IN children WHEN c.age > 10 END

Deep traversal, SET, and UNSET

• WHERE ANY node WITHIN request SATISFIES node.type = “xyz” END

• UPDATE doc UNSET c.field1 FOR c WITHIN doc END

Dynamic Construction

• SELECT { “a”: expr1, “b”: expr2 } AS obj1, name FROM … // Dynamic object

• SELECT [ a, b ] FROM … // Dynamic array

Nested traversal • SELECT x.y.z, a[0] FROM a.b.c …

IS [ NOT ] MISSING • WHERE name IS MISSING


N1QL Data Types from JSON

N1QL supports all JSON data types Numbers Strings Booleans Null Arrays Objects


N1QL Data Type HandlingNon-JSON data types MISSING BinaryData type handling Date functions for string and numeric

encodings Total ordering across all data types

– Well defined semantics for ORDER BY and comparison operators

Defined expression semantics for all input data types– No type mismatch errors

The Query Servicein Couchbase Server 4.0


Couchbase Server Cluster Architecture

38

STORAGE

Couchbase Server 1

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster ManagerCluster Manager

Managed CacheStorage

Data Service

Index Service

Query Service

STORAGE

Couchbase Server 2

Managed Cache


Data Service

Index Service

Query Service

STORAGE

Couchbase Server 3

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Data Service

Index Service

Query Service

STORAGE

Couchbase Server 4

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Data Service

Index Service

Query Service

STORAGE

Couchbase Server 5

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Data Service

Index Service

Query Service

STORAGE

Couchbase Server 6

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Data Service

Index Service

Query Service







Couchbase Server Cluster Service Deployment

39

STORAGE

Couchbase Server 1

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache



Data Servi

ce STORAGE

Couchbase Server 2

Managed Cache


Data Servi

ce STORAGE

Couchbase Server 3

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Data Servi

ce STORAGE

Couchbase Server 4

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Query

Service

STORAGE

Couchbase Server 5

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Query

Service

STORAGE

Couchbase Server 6

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Index

Service


Managed CacheStorage Storage

STORAGE

Couchbase Server 6

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Index

Service

StorageManaged Cache

Managed Cache


N1QL: Query Execution Flow

Clients

1. Submit the query over REST API

8. Query result

2. Parse, Analyze, create Plan

7. Evaluate: Documents to results

3. Scan Request;

index filters6. Fetch the documents

Index

Service

Query

Service Data

Service

4. Get qualified doc keys

5. Fetch Request, doc keys

SELECT c_id, c_first, c_last, c_max FROM CUSTOMER WHERE c_id = 49165;

{ "c_first": "Joe", "c_id": 49165, "c_last": "Montana", "c_max" : 50000}


Query Execution

Client

FetchParse Plan Join FilterPre-Aggregate

Offset Limit ProjectSortAggregateScan

Query ServiceInde

x Servi

ce

Data Servi

ceData-parallel — Query latency scales with coresIn-memory, streamedPluggable architecture — datastore, indexer, client, …

Query

Resultset

Benefits of N1QLfor your Application and

Enterprise


Benefits for your Application and Enterprise Model your Data Cleanly

– Model once, query– Use both relationships and embedding

Query your Data with Flexibility– Query across structure, relationships, and datasets– Query across change and heterogeneity

Develop Rich Applications with Agility– Deliver features that transform, combine, and aggregate data– Use your favorite clients, frameworks, and interfaces

Integrate your Applications and Data– Leverage ecosystem: Connectivity, Analytics, Cloud and Packaged Apps

Community and Participation


Community and Participation

Enterprise and Community Editions of Couchbase 4.0 Community Ecosystem

– Build complementary tools, products, and drivers– Build and integrate via our open APIs


Community and Participation in Action

Couchbase N1QL Query Runner– by Warren Postma, Beta customer– 30-minute quick start with Python’s Tkinter GUI package

Getting Started


Getting Started

Test drive Couchbase Server 4.0 Now!Visit query.couchbase.comPlay with the online tutorialAsk us questions

– Couchbase forums, Stack Overflow, @N1QL

N1QL Sessionsat Couchbase Live

NewYork 2015


2:00pmLevel 2

Developer

3:50pmLevel 2

Architecture

3:50pmLevel 2

Architecture

4:35pmLevel 2

Developer

4:35pmLevel3

Architecture

Sampling of N1QL Sessions by Couchbase

Data Modeling with Couchbase Server

Evolution of Couchbase at CenterEdge: From Cache to Query

Nielsen’s Interactive Data Analytics with Couchbase N1QL

Deep Dive into N1QL with Global Secondary Indexes

N1QL and SDK Support for Java, .NET, and Node.js

Thank you.Keshav Murthy, Couchbase

@rkeshavmurthyTeam @N1QL

introducing n1ql: new sql-based query language for json – couchbase live new york 2015

Software