migrating mysql queries to n1ql: couchbase connect 2015
Post on 26-Jul-2015
258 Views
Preview:
TRANSCRIPT
MIGRATING RDBMS QUERIES TO N1QLMIGRATING RDBMS QUERIES TO N1QL
Parag Agarwal, CouchbaseRaju Suravarjjala, Couchbase
©2015 Couchbase Inc.
AgendaAgendaMotivation of MigrationFactors in migration
Data modelData typeKeys and IndexesQuery statementsCRUD operations
SummaryDemo
©2015 Couchbase Inc.
Motivation for Migration to CouchbaseMotivation for Migration to CouchbaseFlexible/Dynamic SchemaSupport for JSON and Binary dataHierarchical dataExtension of SQLHighly ScalableHighly Available
©2015 Couchbase Inc.
RDBMS (e.g., MySQL): How to migrate to Couchbase?RDBMS (e.g., MySQL): How to migrate to Couchbase?
order_no (pk) cust_id (fk)
purchase_dt item_id (fk)
o1 1234 2015-01-01 00:00:00 a1234
cust_id (pk)
Name Address phone_no
c1 Adam Smith xxx 4087378383
SALES
item_id (pk) desc manufacture_dt
i1 shirt 2014-01-01 00:00:00
ORDER
CUSTOMER
ITEM
SELECT * FROM ORDERSELECT * FROM CUSTOMERSELECT * FROM CUSTOMERCREATE PRIMARY INDEX P_IDX ON ORDERCREATE INDEX IDX ON ORDER(Name)
SELECT *FROM ORDER oINNER JOINCUSTOMER cON o.cust_id = c. o.cust_id
SELECT *FROM ORDER oWHERE o.cust_id =(SELECT cust_id from CUSTOMER)
©2015 Couchbase Inc.
RDBMS (e.g., MySQL) : Factors in migrationRDBMS (e.g., MySQL) : Factors in migration
CRUDOps
Keys
Data Types
Data Model
Indexes
SQLstatemen
ts
Data ModelData Model
©2015 Couchbase Inc.
DATA MODEL: Map RDBMS to CouchbaseDATA MODEL: Map RDBMS to Couchbase
order_no (pk) cust_id (fk)
purchase_dt item_id (fk)
o1 1234 2015-01-01 00:00:00 a1234
cust_id (pk)
Name Address phone_no
c1 Adam Smith xxx 4087378383
SALES
item_id (pk) desc manufacture_dt
i1 shirt 2014-01-01 00:00:00
ORDER
CUSTOMER
ITEM
How to map RDBMS database to Couchbase bucket(s)/JSON documents?
©2015 Couchbase Inc.
DATA MODEL: Map RDBMS to CouchbaseDATA MODEL: Map RDBMS to Couchbase
RDBMS Couchbase
Database Bucket(s)
Table Bucket(s)/keyspaces
Row JSON Document
Fixed Schema Flexible Schema
©2015 Couchbase Inc.
Map Tables to Buckets: Normalized DataMap Tables to Buckets: Normalized Data
order_no (pk) cust_id (fk)
purchase_dt item_id (fk)
o1 c1 2015-01-01 00:00:00 i1
cust_id (pk)
Name Address phone_no
c1 Adam Smith xxx 4087378383
SALES
item_id (pk) desc manufacture_dt
i1 shirt 2014-01-01 00:00:00
ORDER
CUSTOMER
ITEM
BUCKET: order“o1”: {
“cust_id”:”c1”,“purchase_dt”:
“2015-01-01 00:00:00”,
“item_id”:”i1” }
BUCKET: customer“c1”: {
“Name”: “Adam Smith”,“Address”: “XXX”,“phone_no”: 4087378383
}
BUCKET: item“i1”: {
“desc”: “shirt”,“manufacture_dt”:
“2015-01-01 00:00:00” }
©2015 Couchbase Inc.
Map Database to a Bucket: Normalized DataMap Database to a Bucket: Normalized Data
order_no (pk) cust_id (fk)
purchase_dt item_id (fk)
o1 c1 2015-01-01 00:00:00 i1
cust_id (pk)
Name Address phone_no
c1 Adam Smith xxx 4087378383
SALES
item_id (pk) desc manufacture_dt
i1 shirt 2014-01-01 00:00:00
ORDER
CUSTOMER
ITEM
BUCKET: sales“o1”: {
“order”:{“cust_id”:”c1”,“purchase_dt”:“2015-01-01
00:00:00”,“item_id”:”i1”
}}“c1”: {
”customer”:{ “Name”: “Adam
Smith”, “Address”: “XXX”, “phone_no”:
4087378383 }}“i1”: {
“item”:{“desc”: “shirt”,“manufacture_dt”:
“2015-01-01 00:00:00” }}
©2015 Couchbase Inc.
Map Database(s) to a Bucket: Normalized DataMap Database(s) to a Bucket: Normalized Data
order_no (pk) cust_id (fk)
purchase_dt item_id (fk)
o1 c1 2015-01-01 00:00:00 i1
cust_id (pk)
Name Address phone_no
c1 Adam Smith xxx 4087378383
SALES
item_id (pk) desc manufacture_dt
i1 shirt 2014-01-01 00:00:00
ORDER
CUSTOMER
ITEM
BUCKET: Default“o1”: {
“sales”:{ “order”:{ “cust_id”:”c1”, “purchase_dt”: “2015-01-01
00:00:00”, “item_id”:”i1”
}}}“c1”: {“sales”:{ “customer”:{ “Name”: “Adam
Smith”, “Address”: “XXX”, “phone_no”:
4087378383 }}}“i1”: {
“sales”:{ “item”:{ “desc”: “shirt”, “manufacture_dt”:
“2015-01-01 00:00:00” }}}
©2015 Couchbase Inc.
Map Database(s) to a Bucket: Denormalized DataMap Database(s) to a Bucket: Denormalized Data
order_no (pk) cust_id (fk)
purchase_dt item_id (fk)
o1 c1 2015-01-01 00:00:00 i1
cust_id (pk)
Name Address phone_no
c1 Adam Smith xxx 4087378383
SALES
item_id (pk) desc manufacture_dt
i1 shirt 2014-01-01 00:00:00
ORDER
CUSTOMER
ITEM
BUCKET: Default“o1”: {
“sales”:{ “order”:{ “purchase_dt”:
“2015-01-01 00:00:00” “customer”: {
“Name”: “Adam Smith”, “Address”: “XXX”, “phone_no”:
4087378383 },
“item”:{ “desc”: “shirt”, “manufacture_dt”:
“2015-01-01 00:00:00” } } }
Data TypeData Type
©2015 Couchbase Inc.
DATA TYPES MAPPINGDATA TYPES MAPPING
Data Feature/Type RDBMS Couchbase
Case Sensitivity Yes/No Yes (fields only)/No
Numbers supported supported
String supported supported
boolean supported as tinyint supported
datetime supported supported as string in JSON
Spatial data supported supported
MISSING not supported supported
NULL supported supported
Objects and Arrays not supported supported
Blobs supported supported
©2015 Couchbase Inc.
DATA TYPES : MISSING in JSONDATA TYPES : MISSING in JSONMISSINGTo track if a given field is present in a JSON document
SELECT * from customer where name IS MISSING
use to check presence of (name, value) pairBucket: default
“1”:{“name”:“dhdh”
} “2”:{
“value”:“xxxx” }
Results in output [{“value”:“xxxx”}]
©2015 Couchbase Inc.
FOUR VALUE LOGIC: Impact of MISSINGFOUR VALUE LOGIC: Impact of MISSING
A B A OR B A AND B
TRUE NULL TRUE NULL
FALSE NULL FALSE NULL
TRUE MISSING TRUE MISSING
FALSE MISSING TRUE MISSING
NULL MISSING NULL MISSING
NULL NULL NULL NULL
MISSING MISSING MISSING MISSING
backward compatible with RDBMS
©2015 Couchbase Inc.
FOUR VALUE LOGIC: Impact of MISSINGFOUR VALUE LOGIC: Impact of MISSING
A NOT A
TRUE FALSE
FALSE TRUE
MISSING MISSING
NULL NULL
backward compatible with RDBMS
KEYS AND INDEXINGKEYS AND INDEXING
©2015 Couchbase Inc.
KEYS : PRIMARY KEYKEYS : PRIMARY KEY
order_no (pk) cust_id (fk)
purchase_dt item_id (fk)
o1 c1 2015-01-01 00:00:00 i1
cust_id (pk)
Name Address phone_no
c1 Adam Smith xxx 4087378383
SALES
item_id (pk) desc manufacture_dt
i1 shirt 2014-01-01 00:00:00
ORDER
CUSTOMER
ITEM
BUCKET: Default“o1”: {
“sales”:{ “order”:{ “cust_id”:”c1”, “purchase_dt”: “2015-01-01
00:00:00”, “item_id”:”i1”
}}}“c1”: {“sales”:{ “customer”:{ “Name”: “Adam
Smith”, “Address”: “XXX”, “phone_no”:
4087378383 }}}“i1”: {
“sales”:{ “item”:{ “desc”: “shirt”, “manufucture_dt”:
”2015-01-01 00:00:00” }}}
primary is key of (key, value) pair
primary/surrogate key
©2015 Couchbase Inc.
KEYS : FOREIGN KEYKEYS : FOREIGN KEY
order_no (pk) cust_id (fk)
purchase_dt item_id (fk)
o1 c1 2015-01-01 00:00:00 i1
cust_id (pk)
Name Address phone_no
c1 Adam Smith xxx 4087378383
SALES
item_id (pk) desc manufacture_dt
i1 shirt 2014-01-01 00:00:00
ORDER
CUSTOMER
ITEM
BUCKET: Default“o1”: {
“sales”:{ “order”:{ “cust_id”:”c1”, “purchase_dt”: “2015-01-01
00:00:00”, “item_id”:”i1”
}}}“c1”: {“sales”:{ “customer”:{ “Name”: “Adam
Smith”, “Address”: “XXX”, “phone_no”:
4087378383 }}}“i1”: {
“sales”:{ “item”:{ “desc”: “shirt”, “manufucture_dt”:
”2015-01-01 00:00:00” }}}
primary
secondary
secondary
©2015 Couchbase Inc.
IndexingIndexing
Indexing provided using views (existed prior to 4.0 Beta) global secondary indexes (gsi)
Support for primary indexprovided by viewsprovided by gsi
Support for secondary indexprovided by viewsprovided by gsi
Support for hints (views/gsi), similar to MySQL
©2015 Couchbase Inc.
Indexes : Primary IndexesIndexes : Primary Indexes
Primary indexes are built only using key of (key,value) pairGSI CREATE Example: CREATE primary index on customer using gsiVIEW CREATE Example: CREATE primary index on customer using viewAlias CREATE Example: CREATE primary index pk_index1 on customer using gsiVIEW Drop Example: DROP primary index on customer using viewGSI Drop Example: DROP primary index on customer using gsi
©2015 Couchbase Inc.
Indexes : Secondary IndexesIndexes : Secondary Indexes
Simple: CREATE INDEX idx on customer(cust_id) using
GSI(does not throw error if cust_id does not exist)Composite: CREATE INDEX idx on customer(cust_id, Name) using GSI
Where condition: CREATE INDEX idx on customer(Name) WHERE Name LIKE ‘a%’ using GSI
Computed: CREATE INDEX idx on employee(First_name || “ “ ||
Last_Name) using GSIDrop index:
DROP INDEX customer.idx USING GSI
SQL StatementsSQL Statements
©2015 Couchbase Inc.
Operators: Searching in JSON (NESTED DATA)Operators: Searching in JSON (NESTED DATA)
BUCKET: Default“o1”: {
“sales”:{ “order”:{ “purchase_dt”:
“2015-01-01 00:00:00” “customer”: {
“Name”: “Adam Smith”, “Address”: “XXX”, “phone_no”:
4087378383 },
“item”:{ “desc”: “shirt”, “manufacture_dt”:
“2015-01-01 00:00:00”, “store_id”:[11,22,33]
} } }
Example: FieldSELECTc.NameFROMsales.order.customer c
Example: Array Element SELECTi.store_id[1] str_idFROMsales.order.item i
Example: All fieldsSELECTo.customer.*FROMsales.order o
[ { “Name”: “Adam Smith”, “Address”: “XXX”, “phone_no”: 4087378383 }]
[ { “str_id”:22 }]
[ { “Name”: “Adam Smith” }]
INPUT: (KEY, VALUE) N1QL JSON OUTPUT
©2015 Couchbase Inc.
Operators: Searching in JSON (NON-EXISTING FIELDS)Operators: Searching in JSON (NON-EXISTING FIELDS)
IS MISSING
SELECT *FROM sales.customerWHERE Name IS MISSING
example json{ “id”:”233”}
No Errors for missing fields
SELECT *FROM sales.customerWHERE field= ‘xxx’
example json{ “name”:NULL}
New featureFlexible Schema
©2015 Couchbase Inc.
Operators: Searching in JSON (NON-EXISTING VALUES)Operators: Searching in JSON (NON-EXISTING VALUES)
IS VALUED
SELECT *FROM sales.customerWHERE Name IS VALUED
example json{ “name”:”Adam Smith”, “id”:””}
IS NULL
SELECT *FROM sales.customerWHERE Name IS NULL
example json{ “name”:NULL}
same as sql
New feature
©2015 Couchbase Inc.
JOINS: LEFT [OUTER], INNERJOINS: LEFT [OUTER], INNER
order_no (pk) cust_id (fk)
purchase_dt item_id (fk)
o1 c1 2015-01-01 00:00:00 i1
cust_id (pk)
Name Address phone_no
c1 Adam Smith xxx 4087378383
SALES
item_id (pk) desc manufacture_dt
i1 shirt 2014-01-01 00:00:00
ORDER
CUSTOMER
ITEM
BUCKET: sales“o1”: {
“order”{“cust_id”:”c1”,“purchase_dt”:“2015-01-01
00:00:00”,“item_id”:”i1”
}}“c1”: {
“customer”:{ “Name”: “Adam Smit
h”, “Address”: “XXX”, “phone_no”:
4087378383 }}“i1”: {
“item”:{“desc”: “shirt”,“manufucture_dt”: ”2015-01-01 00:00:0
0” }}
ON KEYS outer_keyspace.field matches with primary key of inner_keyspace
SQLSELECT *FROMORDER oINNER JOINCUSTOMER cON o.cust_id = c.cust_idINNER JOINITEM iON o.item_id = i.item_id
N1QLSELECT *FROMsales.orderINNER JOINsales.customerON KEYS sales.order.cust_idINNER JOINsales.item ON KEYS sales.order.item_id
©2015 Couchbase Inc.
JOINS: OUTPUT OF JOIN - NESTED DATAJOINS: OUTPUT OF JOIN - NESTED DATA
BUCKET: sales“o1”: {
”order”{“cust_id”:”c1”,“purchase_dt”:“2015-01-01
00:00:00”,“item_id”:”i1”
}}“c1”: {
”customer”:{ “Name”: “Adam Smit
h”, “Address”: “XXX”, “phone_no”:
4087378383 }}“i1”: {
“item”:{“desc”: “shirt”,“manufacture_dt”: “2015-01-01 00:00:00”
}}
[ {
“order”:{ “purchase_dt”:
”2015-01-01 00:00:00”
}, “customer”: {
“Name”: “Adam Smith”,
“Address”: “XXX”, “phone_no”:
4087378383 },
“item”:{ “desc”: “shirt”,
“manufacture_dt”: “2015-01-01
00:00:00” } } ]
N1QLSELECT *FROMsales.orderINNER JOINsales.customerON KEYS sales.order.cust_idINNER JOINsales.item ON KEYS sales.order.item_id
OUTPUT JSONINPUT : (KEY, VALUE )
©2015 Couchbase Inc.
NEST : CREATE NESTED JSONNEST : CREATE NESTED JSON
BUCKET: sales“o1”: {
“order”{“cust_id”:“c1”,“purchase_dt”:“2015-01-01
00:00:00”,“item_id”:“i1”
}}“c1”: {
“customer”:{ “Name”: “Adam Smit
h”, “Address”: “XXX”, “phone_no”:
4087378383 }}“i1”: {
“item”:{“desc”: “shirt”,“manufucture_dt”: “2015-01-01 00:00:00”
}}
N1QLSELECT sales.order.*,sales.item.*FROMsales.orderNESTsales.itemON KEYS sales.order.item_id
[ {
“order”:{ “purchase_dt”:
“2015-01-01 00:00:00”
}, “item”:[
{ “desc”: “shirt”,
“manufucture_dt”: “2015-01-01
00:00:00” }
] } ]
NESTED JSONINPUT: (KEY, VALUE)
©2015 Couchbase Inc.
UNNEST: REMOVE NESTING FROM JSONUNNEST: REMOVE NESTING FROM JSON
N1QLSELECT oi.purchase_id,c.customer_name,i.descFROMsales.order oUNNESTorder.customer cUNNESTorder.item i
NESTING REMOVED JSON
BUCKET: sales “o1”: {
“order”:{ “purchase_dt”:
“2015-01-01 00:00:00”
}, “customer”: [ {
“Name”: “Adam Smith”,
“Address”: “XXX”, “phone_no”:
4087378383 } ],
“item”:[ { “desc”: “shirt”,
“manufucture_dt”: “2015-01-01
00:00:00” }
] }
[ {
“purchase_dt”:“2015-01-01
00:00:00” “Name”: “Adam
Smith”, “desc”: “shirt” } ]
INPUT : (KEY, VALUE)
©2015 Couchbase Inc.
USE KEYS: Improving Full Scan EfficiencyUSE KEYS: Improving Full Scan Efficiency
Example: One valueSELECT * FROM customer USE KEYS “c1”Example: ArraySELECT * FROM customer USE KEYS [“c1”, “c2”]
Using primary keys of particular(s) value for a given bucket
Example: One valueSELECT * FROM sales.order oWHERE o.cust_id = (SELECT meta(c).idFROMsales.customer cUSE KEYS “c1”)
Example: One valueSELECT * FROM sales.order oUSE KEYS “o1”INNER JOINsales.customer cON KEYS o.cust_id
©2015 Couchbase Inc.
FUNCTIONS: Comparing DATE TIME Fields and ValuesFUNCTIONS: Comparing DATE TIME Fields and Values
Operations on date time fields require date time functions in N1QLSQLSELECT *FROM orderWHEREpurchase_dt > “1980-01-01 00:00:00”
N1QLSELECT *FROM orderWHEREMILLIS(purchase_dt) > MILLIS(‘1980-01-01 00:00:00’)
SQLSELECT *FROM orderWHEREpurchase_dt > Now()
N1QLSELECT *FROM orderWHEREMILLIS(purchase_dt) > NOW_MILLIS()
CRUD OPERATIONSCRUD OPERATIONS
©2015 Couchbase Inc.
CRUD OPERATIONS : INSERT/UPSERTCRUD OPERATIONS : INSERT/UPSERT
SQL
INSERT INTOCustomer(ID, NAME) VALUES(“c1”, “Adam Smith”)
N1QL
INSERT INTOCUSTOMER(KEY, VALUE) VALUES(“c1”, { “Name”: “Adam Smith”}) Returning *
SQL
INSERT INTOCustomer(ID, NAME) VALUES(“c1”, “Adam Smith”)on duplicate key update
INSERT
N1QL
UPSERT INTOCUSTOMER(KEY, VALUE) VALUES(“c1”, { “Name”: “Adam Smith”}) Returning *
UPSERT
not present in MySQL
present in MySQL
supported with format change
supported
©2015 Couchbase Inc.
CRUD OPERATIONS : UPDATE/DELETECRUD OPERATIONS : UPDATE/DELETE
SQL
UPDATECustomerSETName=“Adam Smith”WHERE ID=“ddd”
N1QL
UPDATECustomer eSETe.FIELD=“Adam Smith”
same as sql
SQL
DELETEFROM CUSTOMER ewheree.ID = “c1”
N1QLDELETEFROM CUSTOMER ewhere e.cust_id = “c1”
same as sql
UPDATE
DELETE
BACKWARD COMPATIBLE
©2015 Couchbase Inc.
CRUD OPERATIONS : UPDATE/DELETECRUD OPERATIONS : UPDATE/DELETE
SQL
UPDATECustomerSETName=“Adam Smith”WHERE ID=“ddd”
N1QL
UPDATECustomer eUSE KEYS “c1”SETe.FIELD=“Adam Smith”Returning *
different format
SQL
DELETEFROM CUSTOMER ewheree.ID = “c1”
N1QLDELETEFROM CUSTOMER eUSE KEYS “c1”Returning *
different format
UPDATE
DELETE
NEW FEATURE: Returning, USE KEYS
SummarySummary
©2015 Couchbase Inc.
SummarySummary
Data Model: Move from fixed schema to Flexible schemaData Type: Data types are backward compatible, new data
types( MISSING, Arrays, Objects)Keys and Indexes: backward compatible with two index
services (views/gsi)SQL Statements: JOINs, NEST, UNNEST, Datetime functions,
new Operators for new data types (MISSING/Arrays/Objects)CRUD Operations: INSERT/DELETE/UPDATE are backward
compatible with new operations UPSERT
DemoDemo
©2015 Couchbase Inc.
DemoDemo
Database used: World DatabaseMySQL Vs Couchbase 4.0 Beta
Migration of dataMigration tool: http://www.talend.com/Create Primary IndexesRun queries to use primary indexesExplain on queries
Thank you.Thank you.
top related