scalable data services with nosql (mongodb)linuxtag.org/2012/fileadmin/ · “tl;dr“ license?...
TRANSCRIPT
![Page 1: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/1.jpg)
Scalable Data Serviceswith mongoDB
![Page 2: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/2.jpg)
High PerformanceHigh Availability
![Page 3: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/3.jpg)
for...
Managers
Architects
Developers (Web)
Admins
![Page 4: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/4.jpg)
Data ...
![Page 5: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/5.jpg)
http://www.flickr.com/photos/annarbor/4350618884/in/set-72157623414542180/
![Page 6: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/6.jpg)
Data Services ...
![Page 7: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/7.jpg)
services data for...
Dynamic Web Sites
Mobile Applications
![Page 8: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/8.jpg)
data services are driven by...
Databases (on)
Remote Servers
![Page 9: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/9.jpg)
mongo ...
![Page 10: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/10.jpg)
mongo what?
{
“type“: “db“,
“name“: “mongo“
}
![Page 11: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/11.jpg)
mongo key features?
{
“tables“: false,
“sql“: false,
“documents“: true,
“json“: true,
“buzzword_bingo_compatible“: true
}
![Page 12: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/12.jpg)
more mongo features?
{
“implemented_in“: “C++“,
“has_replication“: true,
“has_sharding“: true,
“stores_files“: true,
“commercial_support“: true
}
![Page 13: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/13.jpg)
still more mongo features?
{
“documentation“: “excellent“,
“speed“: “blazing_fast“,
“level_to_start_with“: “low“,
“stores_files“: true,
“location_search“: true
}
![Page 14: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/14.jpg)
mongo suited for?
{
“reads“: “a great many“
“writes“: “just a few“
}
![Page 15: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/15.jpg)
mongo not suited for?
{
“accuracy“: “must have“,
“consistency“: “must have“,
“internal_references“: “many“
}
![Page 16: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/16.jpg)
language access?
[
“php“,
“ruby“,
“java“,
“python“
]
![Page 17: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/17.jpg)
more language access?
[
“perl“,
“javascript“,
“scala“,
“erlang“
]
![Page 18: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/18.jpg)
still more language access?
“tl;dr“
![Page 19: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/19.jpg)
license?
{
“core“: “AGPL“
“drivers“: “Apache“
}
![Page 20: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/20.jpg)
No SQL ...
![Page 21: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/21.jpg)
tables?
PERSONS:
id, firstname, lastname
1, "Christian", "Hartmann"
ADDRESSES:
street, zip, city, person_id
"Katzlerstraße 9", 10829, "Berlin", 1
URLS:
url, person_id
“http://hartmann-it-design.de“, 1
“mailto:[email protected]", 1
![Page 22: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/22.jpg)
documents!
{
"firstname": "Christian", "lastname": "Hartmann",
"postal": {
"street": "Katzlerstraße 9", "zip": 10829, "city": "Berlin"
},
"urls": [
"http://hartmann-it-design.de",
"mailto:[email protected]",
],
"rev": 1.0
}
![Page 23: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/23.jpg)
storing files ...
![Page 24: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/24.jpg)
storing files in database?
yes!
small files
and large files
and even huge files
and even across multiple servers
“GridFS“
![Page 25: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/25.jpg)
GridFS?
{
“name“: “GridFS“,
“type“: “collection“
}
> (no) mount --type gridfs
![Page 26: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/26.jpg)
Scalability ...
![Page 27: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/27.jpg)
need for scalability
mass of data
high performance
![Page 28: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/28.jpg)
scalability factors
amount of data
performance
![Page 29: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/29.jpg)
replication (HA)
server 1: A-X
server 2: A-X
server 3: A-X
automatic failover
automatic desaster recovering
with or without master
multi data center
![Page 30: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/30.jpg)
sharding (HP)
server 1: A-F
server 2: H-K
server 3: L-P
server 4: Q-T
server 5: U-X
![Page 31: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/31.jpg)
for (SQL) Developers ...
![Page 32: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/32.jpg)
SQL?
no!
sorry
no sql here
![Page 33: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/33.jpg)
sql mappings...
ALTER TABLE users ADD ...
oops .. n/a (there is no alter table)
![Page 34: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/34.jpg)
more sql mappings...
SELECT * FROM users WHERE age=33
db.users.find({age:33})
![Page 35: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/35.jpg)
still more sql mappings...
SELECT order_id FROM orders o, order_line_items li WHERE li.order_id=o.order_id AND li.sku=12345
db.orders.find({"items.sku":12345},{_id:1})
![Page 36: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/36.jpg)
syntactic shugar ...
db.users.find( { name: /^[JY]oe|^[JY]ö/i } )
![Page 37: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/37.jpg)
more syntactic shugar ...
db.users.find( {age: {'$exists': false}} ).count()
![Page 38: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/38.jpg)
can't stop with syntactic shugar ...
db.users.find( { homes : { $size: 2 } } )
![Page 39: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/39.jpg)
still more - it's annoying i know ...
db.meta.find( { name: { $notin : [ “foo“,“bar“ ] }} )
![Page 40: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/40.jpg)
map & reduce?
available
no need for (if distributed storage reasoned)
![Page 41: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/41.jpg)
for Admins ...
![Page 42: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/42.jpg)
replication (master slave)
server 1: A-X (master)
server 2: A-X (slave)
[/etc/mongodb.conf]
master = true
# or
slave = true
source = master.example.com
![Page 43: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/43.jpg)
replication (replica set)
server 1: (primary)
server 2: (secondary)
server 3: (recovering)
replSet = my_first_repl_set
![Page 44: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/44.jpg)
replication (replica set) cont...
> var config = {
_id : “my_first_repl_set“,
members : [
{_id : 0, host : “server1“ },
{_id : 1, host : “server2“ },
{_id : 1, host : “server3“ }
]
}
> rs.initiate(config)
![Page 45: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/45.jpg)
sharding (components)
shards (preferable replicated)
config servers (preferable replicated)
routing servers
sharding key[s]
chunks
![Page 46: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/46.jpg)
![Page 47: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/47.jpg)
sharding (simple config)
> db.runCommand( { “enablesharding“ : "test" } )
> db.runCommand( {
“shardcollection“ : "test.users",
“key“ : { “name“ : 1 }
} )
![Page 48: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/48.jpg)
sharding (chunk config)
{
"ns" : "test.users",
"min" : { "name" : "A" },
"max" : { "name" : "F" },
"shard" : "shard1"
}
![Page 49: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/49.jpg)
GUI ? ...
![Page 50: Scalable Data Services with NoSQL (mongoDB)linuxtag.org/2012/fileadmin/ · “tl;dr“ license? {“core“: “AGPL“ ... sql mappings... ALTER TABLE users ADD ... oops .. n/a (there](https://reader030.vdocuments.mx/reader030/viewer/2022041000/5ea07ca627e6750f1248b051/html5/thumbnails/50.jpg)