introducing n1ql: couchbase connect 2015
TRANSCRIPT
INTRODUCING N1QL:QUERY WITHOUT COMPROMISEGerald Sangudi | @sangudiChief Architect, Query | CouchbaseTeam @N1QL
©2015 Couchbase Inc. 2
Agenda
The Nature of Data
Why Bring SQL to NoSQL
The N1QL Language
The Query Service
Benefits of N1QL for your Application and
Enterprise
Community and Participation
Getting Started
Q & A
The Nature of Data
©2015 Couchbase Inc. 4
Data Comes from the Real World
©2015 Couchbase Inc. 5
Properties of Real-World Data
Rich structure Attributes, Sub-structure
Relationships To other data
Value evolution Data is updated
Structure evolution Data is reshaped
Person
Name
DOB
Billing
Connections
Purchases
Jane Smith
Jan-30-1990
So Why Bring SQL to NoSQL?
©2015 Couchbase Inc. 7
Models for Representing Data
Data Concern Relational Model JSON Document Model (NoSQL)
Rich Structure Multiple flat tables Constant assembly /
disassembly
Documents No assembly required!
Relationships Represented Queried (SQL)
Represented Queried? Not until
now…
Value Evolution
Data can be updated Data can be updated
Structure Evolution
Uniform and rigid Manual change
(disruptive)
Flexible Dynamic change
©2015 Couchbase Inc. 8
For the Best of Both, Bring SQL to NoSQL
From JSON Rich structure, no assembly
Schema flexibility
From SQL General, proven, and pervasive query capabilities Querying across relationships
The Result: N1QL: SQL for JSON
The N1QL Language
©2015 Couchbase Inc. 10
The N1QL Language
Goal
Give developers and enterprises an
expressive, powerful, and complete
language for querying, transforming,
and manipulating JSON data.
N1QL Features from SQL
©2015 Couchbase Inc. 12
SELECT Statement
SELECT [ DISTINCT ] …
FROM … JOIN …
WHERE …
GROUP BY … HAVING …
ORDER BY …
LIMIT …
OFFSET …
( UNION | INTERSECT | EXCEPT )
[ ALL ] …
©2015 Couchbase Inc. 13
SELECT Statement Highlights
Querying across relationships JOINs
Subqueries
Aggregation
MIN, MAX
( SUM, COUNT, AVG, ARRAY_AGG ) [ DISTINCT ]
Combining result sets using set
operators
( UNION, INTERSECT, EXCEPT ) [ DISTINCT ]
©2015 Couchbase Inc. 14
Data Modification Statements
UPDATE … SET … WHERE …
DELETE FROM … WHERE …
INSERT INTO … ( KEY, VALUE ) VALUES …
INSERT INTO … ( KEY …, VALUE … ) SELECT …
MERGE INTO … USING … ON …
WHEN [ NOT ] MATCHED THEN …
Note: Couchbase Server provides per-
document atomicity.
©2015 Couchbase Inc. 15
Index Statements
CREATE INDEX ON …
DROP INDEX …
EXPLAIN …
Highlights
Functional indexes
on any data expression
Partial indexes
©2014 Couchbase, Inc. ©2015 Couchbase Inc. 16
N1QL Expressions from SQL
16
Literals• Primitives [ 0, ‘hello’, TRUE ]
• NULL
Operators
• Arithmetic [ +, -, *, /, % ]
• Logical [ AND, OR, NOT ]
• Comparison [ <, <=, =, !=, >=, >, BETWEEN, IS NULL ]
• Pattern matching [ LIKE ]
• Conditional [ CASE ]
Scalar functions
• Numeric [ trigonometric, ROUND, TRUNC, … ]
• String [ UPPER, LOWER, TRIM, SUBSTR, … ]
• Date [ string and numeric dates, NOW, date arithmetic, …
]
Aggregate functions
• MIN, MAX, SUM, AVG, COUNT [ DISTINCT ]
Subqueries • Subqueries are full expressions
N1QL Extensions to SQL
©2015 Couchbase Inc. 18
N1QL Query Operators [ 1 of 2 ]
USE KEYS …
Direct primary key lookup bypassing index scans
Ideal for hash-distributed datastore
Available in SELECT, UPDATE, DELETE
JOIN … ON KEYS …
Nested loop JOIN using key relationships
Ideal for hash-distributed datastore
©2015 Couchbase Inc. 19
N1QL Query Operators [ 2 of 2 ]
NEST Special JOIN that embeds external child documents
under their parent Ideal for JSON encapsulation
UNNEST
Flattening JOIN that surfaces nested objects as top-
level documents
Ideal for decomposing JSON hierarchies
JOIN, NEST, and UNNEST can be chained in any
combination
©2014 Couchbase, Inc. ©2015 Couchbase Inc. 20
N1QL Expressions for JSON
20
Ranging over collections
• WHERE ANY c IN children SATISFIES c.age > 10 END
• WHERE EVERY r IN ratings SATISFIES r > 3 END
Mapping with filtering • ARRAY c.name FOR c IN children WHEN c.age > 10 END
Deep traversal, SET, and UNSET
• WHERE ANY node WITHIN request SATISFIES node.type = “xyz”
END
• UPDATE doc UNSET c.field1 FOR c WITHIN doc END
Dynamic Construction
• SELECT { “a”: expr1, “b”: expr2 } AS obj1, name FROM … //
Dynamic object
• SELECT [ a, b ] FROM … // Dynamic array
Nested traversal • SELECT x.y.z, a[0] FROM a.b.c …
IS [ NOT ] MISSING • WHERE name IS MISSING
©2015 Couchbase Inc. 21
N1QL Data Types from JSON
N1QL supports all JSON data types
Numbers
Strings
Booleans
Null
Arrays
Objects
©2015 Couchbase Inc. 22
N1QL Data Type Handling
Non-JSON data types
MISSING
Binary
Data type handling
Date functions for string and numeric
encodings
Total ordering across all data types
Well defined semantics for ORDER BY and comparison
operators
Defined expression semantics for all input data
types
No type mismatch errors
The Query Servicein Couchbase Server 4.0
©2015 Couchbase Inc. 24
Topology of Couchbase Server 4.0
Client SDK
App
Index
CB Node
ODBC / JDBC
App
Query
Manager
Data
Index
CB Node
Query
Manager
Data
Index
CB Node
Query
Manager
Data
Index
CB Node
Query
Manager
Data
Index
CB Node
Query
Manager
Data
Index
CB Node
Query
Manager
Data
Multi-dimensional scaling
Query throughput scales with nodes
Multiple connectivity options
ODBC / JDBC
BI Tool
©2015 Couchbase Inc. 25
Query Engine
Query Processor
Inside a Query Engine
Service Listener
Parser Optimizer
Data Stores
Execution Engine
Couchbase Server
Auth DataIndexers
GSIView
s
Others…
8093/18903
File systemData Service
Index Service
......
Cluster Manager
Bucket#1
Bucket#2
Index#2
Index#1
Admin Listener
©2015 Couchbase Inc. 26
Query Request Processing
Index Services
Clients
Data Services
(1) Request (7) Response
Query Services
(3) Plan: AST to execution plan
(6) Evaluate: Documents to results
(4) Scan: Attributes to keys (5) Fetch: Keys to documents
- Prepared
statements
- Streaming, no
cursors (2) Parse: Query to AST
©2015 Couchbase Inc. 27
Query Execution
Client
FetchParse Plan Join FilterPre-Aggregate
Offset Limit ProjectSortAggregateScan
Query ServiceInde
x Servi
ce
Data Servi
ce
Data-parallel — Query latency scales with cores
In-memory, streamed
Pluggable architecture — datastore, indexer, client, …
Benefits of N1QLfor your Application and Enterprise
©2015 Couchbase Inc. 29
Benefits for your Application and Enterprise Model your Data Cleanly
Model once, query Use both relationships and embedding
Query your Data with Flexibility Query across structure, relationships, and datasets Query across change and heterogeneity
Develop Rich Applications with Agility Deliver features that transform, combine, and aggregate data Use your favorite clients, frameworks, and interfaces
Integrate your Applications and Data Leverage ecosystem: Connectivity, Analytics, Cloud and Packaged Apps
Community and Participation
©2015 Couchbase Inc. 31
Community and Participation
Enterprise and Community Beta Programs
Community Beta Work with CB engineers and influence the product Showcase your use case
Community Ecosystem Build complementary tools, products, and drivers Build and integrate via our open APIs
©2015 Couchbase Inc. 32
Community and Participation in Action
Couchbase N1QL Query Runner by Warren Postma, Beta customer 30-minute quick start with Python’s Tkinter GUI package
Getting Started
©2015 Couchbase Inc. 34
Getting Started
Test drive Couchbase Server 4.0 Beta
Visit query.couchbase.com
Play with the online tutorial
Ask us questions Couchbase forums, Stack Overflow, @N1QL
N1QL Sessionsat Couchbase Connect ‘15
©2015 Couchbase Inc. 36
N1QL Customer Sessions
Wed June 3 Thurs June 4
9:30amKeynote
9:45amKeynote
3:45pmDeveloper
1:45pmDeveloper
©2015 Couchbase Inc. 37
N1QL Partner Sessions
Wed June 3 Thurs June 4
1:00pmBig Data
1:45pmBig Data
3:45pmBig Data
3:45pmBig Data
4:30pmBig Data
©2015 Couchbase Inc. 38
Wed June 3 Thurs June 4
1:45, 3:45pmArchitecture
1:00pmDeveloper
2:30pmArchitecture
1:45pmArchitecture
4:30pmDeveloper
4:30pmBig Data
Sampling of N1QL Sessions by Couchbase
Indexing:Intro & Deep Dive
N1QL: Internals & Power FeaturesN1QL in Dev SDKs:Java, .NET, Node.js
Migrating: MySQL to N1QL
Tuning Query Performance
Big DataQuery Landscape
Q & A
Thank you.
Gerald Sangudi, Couchbase@sangudi
©2015 Couchbase Inc. 41