migrating mysql queries to n1ql: couchbase connect 2015

MIGRATING RDBMS QUERIES TO N1QLMIGRATING RDBMS QUERIES TO N1QL

Parag Agarwal, CouchbaseRaju Suravarjjala, Couchbase

©2015 Couchbase Inc.

AgendaAgendaMotivation of MigrationFactors in migration

Data modelData typeKeys and IndexesQuery statementsCRUD operations

SummaryDemo


Motivation for Migration to CouchbaseMotivation for Migration to CouchbaseFlexible/Dynamic SchemaSupport for JSON and Binary dataHierarchical dataExtension of SQLHighly ScalableHighly Available


RDBMS (e.g., MySQL): How to migrate to Couchbase?RDBMS (e.g., MySQL): How to migrate to Couchbase?

order_no (pk) cust_id (fk)

purchase_dt item_id (fk)

o1 1234 2015-01-01 00:00:00 a1234

cust_id (pk)

Name Address phone_no

c1 Adam Smith xxx 4087378383

SALES

item_id (pk) desc manufacture_dt

i1 shirt 2014-01-01 00:00:00

ORDER

CUSTOMER

ITEM

SELECT * FROM ORDERSELECT * FROM CUSTOMERSELECT * FROM CUSTOMERCREATE PRIMARY INDEX P_IDX ON ORDERCREATE INDEX IDX ON ORDER(Name)

SELECT *FROM ORDER oINNER JOINCUSTOMER cON o.cust_id = c. o.cust_id

SELECT *FROM ORDER oWHERE o.cust_id =(SELECT cust_id from CUSTOMER)


RDBMS (e.g., MySQL) : Factors in migrationRDBMS (e.g., MySQL) : Factors in migration

CRUDOps

Keys

Data Types

Data Model

Indexes

SQLstatemen

ts

Data ModelData Model


DATA MODEL: Map RDBMS to CouchbaseDATA MODEL: Map RDBMS to Couchbase



o1 1234 2015-01-01 00:00:00 a1234

cust_id (pk)



SALES


i1 shirt 2014-01-01 00:00:00

ORDER

CUSTOMER

ITEM

How to map RDBMS database to Couchbase bucket(s)/JSON documents?


DATA MODEL: Map RDBMS to CouchbaseDATA MODEL: Map RDBMS to Couchbase

RDBMS Couchbase

Database Bucket(s)

Table Bucket(s)/keyspaces

Row JSON Document

Fixed Schema Flexible Schema


Map Tables to Buckets: Normalized DataMap Tables to Buckets: Normalized Data



o1 c1 2015-01-01 00:00:00 i1

cust_id (pk)



SALES


i1 shirt 2014-01-01 00:00:00

ORDER

CUSTOMER

ITEM

BUCKET: order“o1”: {

“cust_id”:”c1”,“purchase_dt”:

“2015-01-01 00:00:00”,

“item_id”:”i1” }

BUCKET: customer“c1”: {

“Name”: “Adam Smith”,“Address”: “XXX”,“phone_no”: 4087378383

}

BUCKET: item“i1”: {

“desc”: “shirt”,“manufacture_dt”:

“2015-01-01 00:00:00” }


Map Database to a Bucket: Normalized DataMap Database to a Bucket: Normalized Data



o1 c1 2015-01-01 00:00:00 i1

cust_id (pk)



SALES


i1 shirt 2014-01-01 00:00:00

ORDER

CUSTOMER

ITEM

BUCKET: sales“o1”: {

“order”:{“cust_id”:”c1”,“purchase_dt”:“2015-01-01

00:00:00”,“item_id”:”i1”

}}“c1”: {

”customer”:{ “Name”: “Adam

Smith”, “Address”: “XXX”, “phone_no”:

4087378383 }}“i1”: {

“item”:{“desc”: “shirt”,“manufacture_dt”:

“2015-01-01 00:00:00” }}


Map Database(s) to a Bucket: Normalized DataMap Database(s) to a Bucket: Normalized Data



o1 c1 2015-01-01 00:00:00 i1

cust_id (pk)



SALES


i1 shirt 2014-01-01 00:00:00

ORDER

CUSTOMER

ITEM

BUCKET: Default“o1”: {

“sales”:{ “order”:{ “cust_id”:”c1”, “purchase_dt”: “2015-01-01

00:00:00”, “item_id”:”i1”

}}}“c1”: {“sales”:{ “customer”:{ “Name”: “Adam


4087378383 }}}“i1”: {

“sales”:{ “item”:{ “desc”: “shirt”, “manufacture_dt”:

“2015-01-01 00:00:00” }}}


Map Database(s) to a Bucket: Denormalized DataMap Database(s) to a Bucket: Denormalized Data



o1 c1 2015-01-01 00:00:00 i1

cust_id (pk)



SALES


i1 shirt 2014-01-01 00:00:00

ORDER

CUSTOMER

ITEM


“sales”:{ “order”:{ “purchase_dt”:

“2015-01-01 00:00:00” “customer”: {

“Name”: “Adam Smith”, “Address”: “XXX”, “phone_no”:

4087378383 },

“item”:{ “desc”: “shirt”, “manufacture_dt”:

“2015-01-01 00:00:00” } } }

Data TypeData Type


DATA TYPES MAPPINGDATA TYPES MAPPING

Data Feature/Type RDBMS Couchbase

Case Sensitivity Yes/No Yes (fields only)/No

Numbers supported supported

String supported supported

boolean supported as tinyint supported

datetime supported supported as string in JSON

Spatial data supported supported

MISSING not supported supported

NULL supported supported

Objects and Arrays not supported supported

Blobs supported supported


DATA TYPES : MISSING in JSONDATA TYPES : MISSING in JSONMISSINGTo track if a given field is present in a JSON document

SELECT * from customer where name IS MISSING

use to check presence of (name, value) pairBucket: default

“1”:{“name”:“dhdh”

} “2”:{

“value”:“xxxx” }

Results in output [{“value”:“xxxx”}]


FOUR VALUE LOGIC: Impact of MISSINGFOUR VALUE LOGIC: Impact of MISSING

A B A OR B A AND B

TRUE NULL TRUE NULL

FALSE NULL FALSE NULL

TRUE MISSING TRUE MISSING

FALSE MISSING TRUE MISSING

NULL MISSING NULL MISSING

NULL NULL NULL NULL

MISSING MISSING MISSING MISSING

backward compatible with RDBMS


FOUR VALUE LOGIC: Impact of MISSINGFOUR VALUE LOGIC: Impact of MISSING

A NOT A

TRUE FALSE

FALSE TRUE

MISSING MISSING

NULL NULL

backward compatible with RDBMS

KEYS AND INDEXINGKEYS AND INDEXING


KEYS : PRIMARY KEYKEYS : PRIMARY KEY



o1 c1 2015-01-01 00:00:00 i1

cust_id (pk)



SALES


i1 shirt 2014-01-01 00:00:00

ORDER

CUSTOMER

ITEM



00:00:00”, “item_id”:”i1”



4087378383 }}}“i1”: {

“sales”:{ “item”:{ “desc”: “shirt”, “manufucture_dt”:

”2015-01-01 00:00:00” }}}

primary is key of (key, value) pair

primary/surrogate key


KEYS : FOREIGN KEYKEYS : FOREIGN KEY



o1 c1 2015-01-01 00:00:00 i1

cust_id (pk)



SALES


i1 shirt 2014-01-01 00:00:00

ORDER

CUSTOMER

ITEM



00:00:00”, “item_id”:”i1”



4087378383 }}}“i1”: {

“sales”:{ “item”:{ “desc”: “shirt”, “manufucture_dt”:

”2015-01-01 00:00:00” }}}

primary

secondary

secondary


IndexingIndexing

Indexing provided using views (existed prior to 4.0 Beta) global secondary indexes (gsi)

Support for primary indexprovided by viewsprovided by gsi

Support for secondary indexprovided by viewsprovided by gsi

Support for hints (views/gsi), similar to MySQL


Indexes : Primary IndexesIndexes : Primary Indexes

Primary indexes are built only using key of (key,value) pairGSI CREATE Example: CREATE primary index on customer using gsiVIEW CREATE Example: CREATE primary index on customer using viewAlias CREATE Example: CREATE primary index pk_index1 on customer using gsiVIEW Drop Example: DROP primary index on customer using viewGSI Drop Example: DROP primary index on customer using gsi


Indexes : Secondary IndexesIndexes : Secondary Indexes

Simple: CREATE INDEX idx on customer(cust_id) using

GSI(does not throw error if cust_id does not exist)Composite: CREATE INDEX idx on customer(cust_id, Name) using GSI

Where condition: CREATE INDEX idx on customer(Name) WHERE Name LIKE ‘a%’ using GSI

Computed: CREATE INDEX idx on employee(First_name || “ “ ||

Last_Name) using GSIDrop index:

DROP INDEX customer.idx USING GSI

SQL StatementsSQL Statements


Operators: Searching in JSON (NESTED DATA)Operators: Searching in JSON (NESTED DATA)


“sales”:{ “order”:{ “purchase_dt”:

“2015-01-01 00:00:00” “customer”: {

“Name”: “Adam Smith”, “Address”: “XXX”, “phone_no”:

4087378383 },

“item”:{ “desc”: “shirt”, “manufacture_dt”:

“2015-01-01 00:00:00”, “store_id”:[11,22,33]

} } }

Example: FieldSELECTc.NameFROMsales.order.customer c

Example: Array Element SELECTi.store_id[1] str_idFROMsales.order.item i

Example: All fieldsSELECTo.customer.*FROMsales.order o

[ { “Name”: “Adam Smith”, “Address”: “XXX”, “phone_no”: 4087378383 }]

[ { “str_id”:22 }]

[ { “Name”: “Adam Smith” }]

INPUT: (KEY, VALUE) N1QL JSON OUTPUT


Operators: Searching in JSON (NON-EXISTING FIELDS)Operators: Searching in JSON (NON-EXISTING FIELDS)

IS MISSING

SELECT *FROM sales.customerWHERE Name IS MISSING

example json{ “id”:”233”}

No Errors for missing fields

SELECT *FROM sales.customerWHERE field= ‘xxx’

example json{ “name”:NULL}

New featureFlexible Schema


Operators: Searching in JSON (NON-EXISTING VALUES)Operators: Searching in JSON (NON-EXISTING VALUES)

IS VALUED

SELECT *FROM sales.customerWHERE Name IS VALUED

example json{ “name”:”Adam Smith”, “id”:””}

IS NULL

SELECT *FROM sales.customerWHERE Name IS NULL

example json{ “name”:NULL}

same as sql

New feature


JOINS: LEFT [OUTER], INNERJOINS: LEFT [OUTER], INNER



o1 c1 2015-01-01 00:00:00 i1

cust_id (pk)



SALES


i1 shirt 2014-01-01 00:00:00

ORDER

CUSTOMER

ITEM


“order”{“cust_id”:”c1”,“purchase_dt”:“2015-01-01

00:00:00”,“item_id”:”i1”

}}“c1”: {

“customer”:{ “Name”: “Adam Smit

h”, “Address”: “XXX”, “phone_no”:

4087378383 }}“i1”: {

“item”:{“desc”: “shirt”,“manufucture_dt”: ”2015-01-01 00:00:0

0” }}

ON KEYS outer_keyspace.field matches with primary key of inner_keyspace

SQLSELECT *FROMORDER oINNER JOINCUSTOMER cON o.cust_id = c.cust_idINNER JOINITEM iON o.item_id = i.item_id

N1QLSELECT *FROMsales.orderINNER JOINsales.customerON KEYS sales.order.cust_idINNER JOINsales.item ON KEYS sales.order.item_id


JOINS: OUTPUT OF JOIN - NESTED DATAJOINS: OUTPUT OF JOIN - NESTED DATA


”order”{“cust_id”:”c1”,“purchase_dt”:“2015-01-01

00:00:00”,“item_id”:”i1”

}}“c1”: {

”customer”:{ “Name”: “Adam Smit


4087378383 }}“i1”: {

“item”:{“desc”: “shirt”,“manufacture_dt”: “2015-01-01 00:00:00”

}}

[ {

“order”:{ “purchase_dt”:

”2015-01-01 00:00:00”

}, “customer”: {

“Name”: “Adam Smith”,

“Address”: “XXX”, “phone_no”:

4087378383 },

“item”:{ “desc”: “shirt”,

“manufacture_dt”: “2015-01-01

00:00:00” } } ]

N1QLSELECT *FROMsales.orderINNER JOINsales.customerON KEYS sales.order.cust_idINNER JOINsales.item ON KEYS sales.order.item_id

OUTPUT JSONINPUT : (KEY, VALUE )


NEST : CREATE NESTED JSONNEST : CREATE NESTED JSON


“order”{“cust_id”:“c1”,“purchase_dt”:“2015-01-01

00:00:00”,“item_id”:“i1”

}}“c1”: {

“customer”:{ “Name”: “Adam Smit


4087378383 }}“i1”: {

“item”:{“desc”: “shirt”,“manufucture_dt”: “2015-01-01 00:00:00”

}}

N1QLSELECT sales.order.*,sales.item.*FROMsales.orderNESTsales.itemON KEYS sales.order.item_id

[ {


“2015-01-01 00:00:00”

}, “item”:[

{ “desc”: “shirt”,

“manufucture_dt”: “2015-01-01

00:00:00” }

] } ]

NESTED JSONINPUT: (KEY, VALUE)


UNNEST: REMOVE NESTING FROM JSONUNNEST: REMOVE NESTING FROM JSON

N1QLSELECT oi.purchase_id,c.customer_name,i.descFROMsales.order oUNNESTorder.customer cUNNESTorder.item i

NESTING REMOVED JSON

BUCKET: sales “o1”: {


“2015-01-01 00:00:00”

}, “customer”: [ {

“Name”: “Adam Smith”,

“Address”: “XXX”, “phone_no”:

4087378383 } ],

“item”:[ { “desc”: “shirt”,

“manufucture_dt”: “2015-01-01

00:00:00” }

] }

[ {

“purchase_dt”:“2015-01-01

00:00:00” “Name”: “Adam

Smith”, “desc”: “shirt” } ]

INPUT : (KEY, VALUE)


USE KEYS: Improving Full Scan EfficiencyUSE KEYS: Improving Full Scan Efficiency

Example: One valueSELECT * FROM customer USE KEYS “c1”Example: ArraySELECT * FROM customer USE KEYS [“c1”, “c2”]

Using primary keys of particular(s) value for a given bucket

Example: One valueSELECT * FROM sales.order oWHERE o.cust_id = (SELECT meta(c).idFROMsales.customer cUSE KEYS “c1”)

Example: One valueSELECT * FROM sales.order oUSE KEYS “o1”INNER JOINsales.customer cON KEYS o.cust_id


FUNCTIONS: Comparing DATE TIME Fields and ValuesFUNCTIONS: Comparing DATE TIME Fields and Values

Operations on date time fields require date time functions in N1QLSQLSELECT *FROM orderWHEREpurchase_dt > “1980-01-01 00:00:00”

N1QLSELECT *FROM orderWHEREMILLIS(purchase_dt) > MILLIS(‘1980-01-01 00:00:00’)

SQLSELECT *FROM orderWHEREpurchase_dt > Now()

N1QLSELECT *FROM orderWHEREMILLIS(purchase_dt) > NOW_MILLIS()

CRUD OPERATIONSCRUD OPERATIONS


CRUD OPERATIONS : INSERT/UPSERTCRUD OPERATIONS : INSERT/UPSERT

SQL

INSERT INTOCustomer(ID, NAME) VALUES(“c1”, “Adam Smith”)

N1QL

INSERT INTOCUSTOMER(KEY, VALUE) VALUES(“c1”, { “Name”: “Adam Smith”}) Returning *

SQL

INSERT INTOCustomer(ID, NAME) VALUES(“c1”, “Adam Smith”)on duplicate key update

INSERT

N1QL

UPSERT INTOCUSTOMER(KEY, VALUE) VALUES(“c1”, { “Name”: “Adam Smith”}) Returning *

UPSERT

not present in MySQL

present in MySQL

supported with format change

supported


CRUD OPERATIONS : UPDATE/DELETECRUD OPERATIONS : UPDATE/DELETE

SQL

UPDATECustomerSETName=“Adam Smith”WHERE ID=“ddd”

N1QL

UPDATECustomer eSETe.FIELD=“Adam Smith”

same as sql

SQL

DELETEFROM CUSTOMER ewheree.ID = “c1”

N1QLDELETEFROM CUSTOMER ewhere e.cust_id = “c1”

same as sql

UPDATE

DELETE

BACKWARD COMPATIBLE


CRUD OPERATIONS : UPDATE/DELETECRUD OPERATIONS : UPDATE/DELETE

SQL

UPDATECustomerSETName=“Adam Smith”WHERE ID=“ddd”

N1QL

UPDATECustomer eUSE KEYS “c1”SETe.FIELD=“Adam Smith”Returning *

different format

SQL

DELETEFROM CUSTOMER ewheree.ID = “c1”

N1QLDELETEFROM CUSTOMER eUSE KEYS “c1”Returning *

different format

UPDATE

DELETE

NEW FEATURE: Returning, USE KEYS

SummarySummary


SummarySummary

Data Model: Move from fixed schema to Flexible schemaData Type: Data types are backward compatible, new data

types( MISSING, Arrays, Objects)Keys and Indexes: backward compatible with two index

services (views/gsi)SQL Statements: JOINs, NEST, UNNEST, Datetime functions,

new Operators for new data types (MISSING/Arrays/Objects)CRUD Operations: INSERT/DELETE/UPDATE are backward

compatible with new operations UPSERT

DemoDemo


DemoDemo

Database used: World DatabaseMySQL Vs Couchbase 4.0 Beta

Migration of dataMigration tool: http://www.talend.com/Create Primary IndexesRun queries to use primary indexesExplain on queries

http://www.talend.com/

Thank you.Thank you.

migrating mysql queries to n1ql: couchbase connect 2015

Technology