tugdual grall - from sql to nosql in less than 40 min - nosql matters paris 2015
TRANSCRIPT
![Page 1: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/1.jpg)
![Page 3: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/3.jpg)
Tugdual “Tug” Grall • MongoDB
– Technical Evangelist • Couchbase
– Technical Evangelist • eXo
– CTO • Oracle
– Developer/Product Manager – Mainly Java/SOA
• Developer in consulting firms
{“about” : “me”}
• Web – @tgrall – http://blog.grallandco.com – tgrall
• NantesJUG cofounder
• Pet Project – http://www.resultri.com
![Page 4: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/4.jpg)
Why Migrate At All?
![Page 5: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/5.jpg)
Understand Your Pain(s)
Existing solution must be struggling to deliver 2 or more of the following capabilities:
• High performance (1000’s – millions ops / sec)
• Need dynamic schema with rich shapes and rich querying
• Need truly agile software lifecycle and quick time to market for new features
• Geospatial querying
• Need for effortless replication across multiple data centers, even globally
• Need to deploy rapidly and scale on demand
• 99.999% uptime (<10 mins / yr)
• Deploy over commodity computing and storage architectures
• Point in Time recovery
![Page 6: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/6.jpg)
Migration Difficulty Varies By Architecture
Moving to NosSQL is not the same as migrating from one RDBMS to another. To be successful, you must address your overall design and technology stack, not just schema design.
![Page 7: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/7.jpg)
The Stack: The Obvious
RDBMS
JDBC
SQL / ResultSet
ORM
POJOs
Assume there will be many changes at this level: • Schema • Stored Procedure Rewrite • Ops management • Backup & Restore • Test Environment setup
Apps
Storage Layer
![Page 8: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/8.jpg)
Don’t Forget the Storage
Most RDBMS are deployed over SAN. MongoDB works on SAN, too – but value may exist in switching to locally attached storage
RDBMS
JDBC
SQL / ResultSet
ORM
POJOs
Apps
Storage Layer
![Page 9: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/9.jpg)
Less Obvious But Important
Opportunities may exist to increase platform value:
• Convergence of HA and DR • Read-only use of secondaries • Schema • Ops management • Backup & Restore • Test Environment setup
RDBMS
JDBC
SQL / ResultSet
ORM
POJOs
Apps
Storage Layer
![Page 10: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/10.jpg)
O/JDBC is about Rectangles
MongoDB uses different drivers, so different • Data shape APIs • Connection pooling • Write durability And most importantly • No multi-document TX RDBMS
JDBC
SQL / ResultSet
ORM
POJOs
Apps
Storage Layer
![Page 11: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/11.jpg)
NoSQL means… well… No SQL
MongoDB doesn’t use SQL nor does it return data in rectangular form where each field is a scalar And most importantly • No JOINs in the database
RDBMS
JDBC
SQL / ResultSet
ORM
POJOs
Apps
Storage Layer
![Page 12: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/12.jpg)
Goodbye, ORM
ORMs are designed to move rectangles of often repeating columns into POJOs. This is unnecessary in MongoDB.
RDBMS
JDBC
SQL / ResultSet
ORM
POJOs
Apps
Storage Layer
![Page 13: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/13.jpg)
The Tail (might) Wag The Dog
Common POJO mistakes: • Mimic underlying relational
design for ease of ORM integration
• Carrying fields like “id” which
violate object / containing domain design
• Lack of testability without a
persistor RDBMS
JDBC
SQL / ResultSet
ORM
POJOs
Apps
Storage Layer
![Page 14: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/14.jpg)
Migrate Or Rewrite: Cost/Benefit Analysis
Migration Approach
RDBMS
JDBC
SQL / ResultSet
ORM
POJOs
Apps
Rewrite Approach
Con
stan
t mar
gina
l cos
t C
onsi
sten
t and
cle
an d
esig
n
Incr
easi
ng m
argi
nal c
ost
Dec
reas
ing
valu
e of
m
igra
tion
vs. r
ewrit
e $
$
$
$ Storage Layer
![Page 15: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/15.jpg)
Sample Migration Investment “Calculator”
Design Aspect Difficulty Include Two-phase XA commit to external systems (e.g. queues) -5
More than 100 tables most of which are critical -3 ✔
Extensive, complex use of ORMs -3
Hundreds of SQL driven BI reports -2
Compartmentalized dynamic SQL generation +2 ✔
Core logic code (POJOs) free of persistence bits +2 ✔
Need to save and fetch BLOB data +2
Need to save and query third party data that can change +4
Fully factored DAL incl. query parameterization +4
Desire to simplify persistence design +4
SCORE +1
If score is less than 0, significant investment may be required to produce desired migration value
![Page 16: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/16.jpg)
Migration Spectrum
• Small number of tables (20) • Complex data shapes stored in BLOBs • Millions or billions of items • Frequent (monthly) change in data shapes • Well-constructed software stack with DAL
• POJO or apps directly constructing and executing SQL
• Hundreds of tables • Slow growth • Extensive SQL-based BI reporting
GOOD
REWRITE INSTEAD
![Page 17: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/17.jpg)
What Are People Going to Do Differently
![Page 18: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/18.jpg)
Everyone Needs To Change A Bit
• Line of business • Solution Architects • Developers • Data Architects • DBAs • System Administrators • Security
![Page 19: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/19.jpg)
…especially these guys
• Line of business • Solution Architects • Developers • Data Architects • DBAs • System Administrators • Security
![Page 20: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/20.jpg)
Data Architect’s View: Data Modeling
RDBMS MongoDB
{ name: { last: "Dunham”, first: “Justin” }, department : "Marketing", pets: [ “dog”, “cat” ], title : “Manager", locationCode: “NYC23”, benefits : [ { type : "Health", plan : “Plus" }, { type : "Dental", plan : "Standard”, optin: true } ] }
![Page 21: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/21.jpg)
An Example
![Page 22: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/22.jpg)
Product Catalog
![Page 23: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/23.jpg)
Entity Attributes Values prodID property value
1 length/weight -3
1 barrel dia 2 5/8
1 type composite
1 certification BBCOR
… 5 size 12
5 position infield
5 pattern B212
5 material leather
5 color black
… 8 color white
8 cover leather
8 core cork
prodID Category Model Name Brand Country Price
1 Bat B1403E Air Elite RIP-IT USA $399.99
2 Bat B1403 Prototype RIP-IT USA $199.99
3 Bat MCB1B One Marucci Imported $199.99
4 Bat BB14S1 S1 Easton China $399.99
5 Glove WTA2000BBB2
12 A2000 Wilson Vietnam $299.99
6 Glove PRO112PT HOH Pro Rawlings China $229.99
7 Baseball DICRLLB1PBG Little League Rawlings China $4.99
8 Baseball ROML MLB Rawlings China $6.99
![Page 25: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/25.jpg)
Bulk Migration
![Page 26: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/26.jpg)
From The Factory: mongoimport $ head -‐1 customers.json { "name": { "last": "Dunham", "first": "Jus3n" }, "department" : "Marke3ng", "pets": [ "dog", "cat" ] , "hire": {"$date": "2012-‐12-‐14T00:00:00Z"} ,"3tle" : "Manager", "loca3onCode": "NYC23" , "benefits" : [ { "type":"Health", "plan":"Plus" }, { "type" : "Dental", "plan" : "Standard", "op3n": true }]} $ mongoimport -‐-‐db test -‐-‐collec8on customers –drop < customers.json connected to: 127.0.0.1 2014-‐11-‐26T08:36:47.509-‐0800 imported 1000 objects $ mongo MongoDB shell version: 2.6.5 connec3ng to: test Ø db.customers.findOne() {
"_id" : ObjectId("548f5c2da40d2829f0ed8be9"), "name" : { "last" : "Dunham”, “first" : "Jus3n” }, "department" : "Marke3ng", "pets" : [ "dog”"cat”], "hire" : ISODate("2012-‐12-‐14T00:00:00Z"), "3tle" : "Manager", "loca3onCode" : "NYC23", "benefits" : [ { "type" : "Health", "plan" : "Plus" },{ "type" : "Dental", "plan" : "Standard", "op3n" : true } ]
}
![Page 27: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/27.jpg)
Traditional vendor ETL
Source Database ETL
![Page 28: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/28.jpg)
Many other options
• Community Tools • Build your One • ….
![Page 30: Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015](https://reader031.vdocuments.mx/reader031/viewer/2022032216/55a644451a28ab9d518b491a/html5/thumbnails/30.jpg)
Thank you
mongodb.com