Code JoeD gets you a 25% discount off the list price

Back to Basics 2016 : Webinar 2

Your First MongoDB Application

Joe DrumgooleDirector of Developer Advocacy, EMEA

MongoDB@jdrumgoole#back2basics

V1.0

4

Summary of Part 1

• Why NoSQL exists• The types of NoSQL database• The key features of MongoDB• Data durability in MongoDB – Replica Sets• Scalability in MongoDB - Sharding

5

Agenda

• Database concepts• Installing MongoDB• Building a basic blogging application• Adding an index• Query optimization with explain

6

Concepts

Relational MongoDBDatabase DatabaseTable CollectionRow DocumentIndex IndexJoin LookupForeign Key ReferenceMulti-table transaction Single document transaction

7

Installing MongoDB

$ curl -O https://fastdl.mongodb.org/osx/mongodb-osx-x86_64-3.2.6.tgz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 60.9M 100 60.9M 0 0 2730k 0 0:00:22 0:00:22 --:--:-- 1589k$ tar xzvf mongodb-osx-x86_64-3.2.6.tgz x mongodb-osx-x86_64-3.2.6/READMEx mongodb-osx-x86_64-3.2.6/THIRD-PARTY-NOTICESx mongodb-osx-x86_64-3.2.6/MPL-2x mongodb-osx-x86_64-3.2.6/GNU-AGPL-3.0x mongodb-osx-x86_64-3.2.6/bin/mongodumpx mongodb-osx-x86_64-3.2.6/bin/mongorestorex mongodb-osx-x86_64-3.2.6/bin/mongoexportx mongodb-osx-x86_64-3.2.6/bin/mongoimportx mongodb-osx-x86_64-3.2.6/bin/mongostatx mongodb-osx-x86_64-3.2.6/bin/mongotopx mongodb-osx-x86_64-3.2.6/bin/bsondumpx mongodb-osx-x86_64-3.2.6/bin/mongofilesx mongodb-osx-x86_64-3.2.6/bin/mongooplogx mongodb-osx-x86_64-3.2.6/bin/mongoperfx mongodb-osx-x86_64-3.2.6/bin/mongosniffx mongodb-osx-x86_64-3.2.6/bin/mongodx mongodb-osx-x86_64-3.2.6/bin/mongosx mongodb-osx-x86_64-3.2.6/bin/mongo$ ln -s mongodb-osx-x86_64-3.2.6 mongodb

8

Running MongodJD10Gen:mongodb jdrumgoole$ ./bin/mongod --dbpath /data/b2b2016-05-23T19:21:07.767+0100 I CONTROL [initandlisten] MongoDB starting : pid=49209 port=27017 dbpath=/data/b2b 64-bit host=JD10Gen.local2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] db version v3.2.62016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] git version: 05552b562c7a0b3143a729aaa0838e558dc49b252016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] allocator: system2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] modules: none2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] build environment:2016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] distarch: x86_642016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] target_arch: x86_642016-05-23T19:21:07.768+0100 I CONTROL [initandlisten] options: { storage: { dbPath: "/data/b2b" } }2016-05-23T19:21:07.769+0100 I - [initandlisten] Detected data files in /data/b2b created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.2016-05-23T19:21:07.769+0100 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=4G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),2016-05-23T19:21:08.837+0100 I CONTROL [initandlisten] 2016-05-23T19:21:08.838+0100 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 10002016-05-23T19:21:08.840+0100 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker2016-05-23T19:21:08.840+0100 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/data/b2b/diagnostic.data'2016-05-23T19:21:08.841+0100 I NETWORK [initandlisten] waiting for connections on port 270172016-05-23T19:21:09.148+0100 I NETWORK [initandlisten] connection accepted from 127.0.0.1:59213 #1 (1 connection now open)

9

Connecting Via The Shell$ ./bin/mongoMongoDB shell version: 3.2.6connecting to: testServer has startup warnings: 2016-05-17T11:46:03.516+0100 I CONTROL [initandlisten] 2016-05-17T11:46:03.516+0100 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000>

10

Inserting your first record> show databaseslocal 0.000GB> use testswitched to db test> show databaseslocal 0.000GB> db.demo.insert( { "key" : "value" } )WriteResult({ "nInserted" : 1 })> show databaseslocal 0.000GBtest 0.000GB> show collectionsdemo> db.demo.findOne(){ "_id" : ObjectId("573af7085ee4be80385332a6"), "key" : "value" }>

11

Object ID

573af7085ee4be80385332a6TS------ID----PID-Count-

12

A Simple Blog Application

• Lets create a blogging application with:– Articles– Users– Comments

13

Typical Entity Relation Diagram

14

In MongoDB we can build organically> use blogswitched to db blog> db.users.insert( { "username" : "jdrumgoole", "password" : "top secret", "lang" : "EN" } )WriteResult({ "nInserted" : 1 })> db.users.findOne(){

"_id" : ObjectId("573afff65ee4be80385332a7"),"username" : "jdrumgoole","password" : "top secret","lang" : "EN"

}

15

How do we do this in a program?'''Created on 17 May 2016

@author: jdrumgoole'''import pymongo ## client defaults to localhost and port 27017. eg MongoClient('localhost', 27017)client = pymongo.MongoClient()blogDatabase = client[ "blog" ]usersCollection = blogDatabase[ "users" ]

usersCollection.insert_one( { "username" : "jdrumgoole", "password" : "top secret", "lang" : "EN" })

user = usersCollection.find_one()

print( user )

16

Next up Articles

…articlesCollection = blogDatabase[ "articles" ]

author = "jdrumgoole"

article = { "title" : "This is my first post", "body" : "The is the longer body text for my blog post. We can add lots of text here.", "author" : author, "tags" : [ "joe", "general", "Ireland", "admin" ] }

## Lets check if our author exists#

if usersCollection.find_one( { "username" : author }) : articlesCollection.insert_one( article )else: raise ValueError( "Author %s does not exist" % author )

17

Create a new type of article## Lets add a new type of article with a posting date and a section#author = "jdrumgoole"title = "This is a post on MongoDB" newPost = { "title" : title, "body" : "MongoDB is the worlds most popular NoSQL database. It is a document database", "author" : author, "tags" : [ "joe", "mongodb", "Ireland" ], "section" : "technology", "postDate" : datetime.datetime.now(), }

## Lets check if our author exists#

if usersCollection.find_one( { "username" : author }) : articlesCollection.insert_one( newPost )

18

Make a lot of articles 1import pymongoimport stringimport datetimeimport random def randomString( size, letters = string.letters ): return "".join( [random.choice( letters ) for _ in xrange( size )] )

client = pymongo.MongoClient()

def makeArticle( count, author, timestamp ): return { "_id" : count, "title" : randomString( 20 ), "body" : randomString( 80 ), "author" : author, "postdate" : timestamp }

def makeUser( username ): return { "username" : username, "password" : randomString( 10 ) , "karma" : random.randint( 0, 500 ), "lang" : "EN" }

19

Make a lot of articles 2blogDatabase = client[ "blog" ]usersCollection = blogDatabase[ "users" ]articlesCollection = blogDatabase[ "articles" ]

bulkUsers = usersCollection.initialize_ordered_bulk_op()bulkArticles = articlesCollection.initialize_ordered_bulk_op()

ts = datetime.datetime.now()

for i in range( 1000000 ) : #username = randomString( 10, string.ascii_uppercase ) + "_" + str( i ) username = "USER_" + str( i ) bulkUsers.insert( makeUser( username ) ) ts = ts + datetime.timedelta( seconds = 1 ) bulkArticles.insert( makeArticle( i, username, ts )) if ( i % 500 == 0 ) : bulkUsers.execute() bulkArticles.execute() bulkUsers = usersCollection.initialize_ordered_bulk_op() bulkArticles = articlesCollection.initialize_ordered_bulk_op() bulkUsers.execute()bulkArticles.execute()

20

Find a User> db.users.findOne(){

"_id" : ObjectId("5742da5bb26a88bc00e941ac"),"username" : "FLFZQLSRWZ_0","lang" : "EN","password" : "vTlILbGWLt","karma" : 448

}> db.users.find( { "username" : "VHXDAUUFJW_45" } ).pretty(){

"_id" : ObjectId("5742da5bb26a88bc00e94206"),"username" : "VHXDAUUFJW_45","lang" : "EN","password" : "GmRLnCeKVp","karma" : 284

}

21

Find Users with high Karma> db.users.find( { "karma" : { $gte : 450 }} ).pretty(){

"_id" : ObjectId("5742da5bb26a88bc00e941ae"),"username" : "JALLFRKBWD_1","lang" : "EN","password" : "bCSKSKvUeb","karma" : 487

}{

"_id" : ObjectId("5742da5bb26a88bc00e941e4"),"username" : "OTKWJJBNBU_28","lang" : "EN","password" : "HAWpiATCBN","karma" : 473

}{

…

22

Using projection > db.users.find( { "karma" : { $gte : 450 }}, { "_id" : 0, username : 1, karma : 1 } ){ "username" : "JALLFRKBWD_1", "karma" : 487 }{ "username" : "OTKWJJBNBU_28", "karma" : 473 }{ "username" : "RVVHLKTWHU_31", "karma" : 493 }{ "username" : "JBNESEOOEP_48", "karma" : 464 }{ "username" : "VSTBDZLKQQ_51", "karma" : 487 }{ "username" : "UKYDTQJCLO_61", "karma" : 493 }{ "username" : "HZFZZMZHYB_106", "karma" : 493 }{ "username" : "AAYLPJJNHO_113", "karma" : 455 }{ "username" : "CXZZMHLBXE_128", "karma" : 460 }{ "username" : "KKJXBACBVN_134", "karma" : 460 }{ "username" : "PTNTIBGAJV_165", "karma" : 461 }{ "username" : "PVLCQJIGDY_169", "karma" : 463 }

23

Update an Article to Add Comments 1> db.articles.find( { "_id" : 19 } ).pretty(){

"_id" : 19,"body" :

"nTzOofOcnHKkJxpjKAyqTTnKZMFzzkWFeXtBRuEKsctuGBgWIrEBrYdvFIVHJWaXLUTVUXblOZZgUqWu",

"postdate" : ISODate("2016-05-23T12:02:46.830Z"),"author" : "ASWTOMMABN_19","title" : "CPMaqHtAdRwLXhlUvsej"

} > db.articles.update( { _id : 18 }, { $set : { comments : [] }} )WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })

24

Update an article to add Comments 2> db.articles.find( { _id :18 } ).pretty(){

"_id" : 18,"body" :

"KmwFSIMQGcIsRNTDBFPuclwcVJkoMcrIPwTiSZDYyatoKzeQiKvJkiVSrndXqrALVIYZxGpaMjucgXUV",

"postdate" : ISODate("2016-05-23T16:04:39.497Z"),"author" : "USER_18","title" : "wTLreIEyPfovEkBhJZZe","comments" : [ ]

}>

25

Update an Article to Add Comments 3> db.articles.update( { _id : 18 }, { $push : { comments : { username : "joe", comment : "hey first post" }}} )WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })

> db.articles.find( { _id :18 } ).pretty(){

"_id" : 18,"body" : "KmwFSIMQGcIsRNTDBFPuclwcVJkoMcrIPwTiSZDYyatoKzeQiKvJkiVSrndXqrALVIYZxGpaMjucgXUV","postdate" : ISODate("2016-05-23T16:04:39.497Z"),"author" : "USER_18","title" : "wTLreIEyPfovEkBhJZZe","comments" : [{"username" : "joe","comment" : "hey first post"}]

}>

26

Delete an Article

> db.articles.remove( { "_id" : 25 } )WriteResult({ "nRemoved" : 1 })> db.articles.remove( { "_id" : 25 } )WriteResult({ "nRemoved" : 0 })> db.articles.remove( { "_id" : { $lte : 5 }} )WriteResult({ "nRemoved" : 6 })

• Deletion leaves holes• Dropping a collection is cheaper than deleting a large collection

element by element

27

A Quick Look at Users and Articles Again> db.users.findOne(){

"_id" : ObjectId("57431c07b26a88bf060e10cb"),"username" : "USER_0","lang" : "EN","password" : "kGIxPxqKGJ","karma" : 266

}> db.articles.findOne(){

"_id" : 0,"body" :

"hvJLnrrfZQurmtjPfUWbMhaQWbNjXLzjpuGLZjsxHXbUycmJVZTeOZesTnZtojThrebRcUoiYwivjpwG","postdate" : ISODate("2016-05-23T16:04:39.246Z"),"author" : "USER_0","title" : "gpNIoPxpfTAxWjzAVoTJ"

}>

28

Find a User> db.users.find( { "username" : "ABOXHWKBYS_199" } ).explain(){

"queryPlanner" : {"plannerVersion" : 1,"namespace" : "blog.users","indexFilterSet" : false,"parsedQuery" : {

"username" : {"$eq" : "ABOXHWKBYS_199"

}},"winningPlan" : {

"stage" : "COLLSCAN","filter" : {

"username" : {"$eq" : "ABOXHWKBYS_199"

}},"direction" : "forward"

},"rejectedPlans" : [ ]

},"serverInfo" : {

"host" : "JD10Gen.local","port" : 27017,"version" : "3.2.6","gitVersion" : "05552b562c7a0b3143a729aaa0838e558dc49b25"

},"ok" : 1

}

29

Find a User – Execution Stats> db.users.find( {"username" : "USER_999999" } ).explain( "executionStats" ).executionStats{

"executionSuccess" : true,"nReturned" : 1,"executionTimeMillis" : 433,"totalKeysExamined" : 0,"totalDocsExamined" : 1000000,"executionStages" : {

"stage" : "COLLSCAN","filter" : {

"username" : {"$eq" : "USER_999999"

}},"nReturned" : 1,"executionTimeMillisEstimate" : 330,"works" : 1000002,"advanced" : 1,"needTime" : 1000000,"needYield" : 0,"saveState" : 7812,"restoreState" : 7812,"isEOF" : 1,"invalidates" : 0,"direction" : "forward","docsExamined" : 1000000

30

We need an index

> db.users.createIndex( { username : 1 } ){

"createdCollectionAutomatically" : false,"numIndexesBefore" : 1,"numIndexesAfter" : 2,"ok" : 1

}>

31

Indexes Overview

• Parameters– Background : Create an index in the background as opposed to locking the database– Unique : All keys in the collection must be unique. Duplicate key insertions will be

rejected with an error.– Name : explicitly name an index. Otherwise the index name is autogenerated from the

index field.• Deleting an Index

– db.users.dropIndex({ “username” : 1 })• Get All the Indexes on a collection

– db.users.getIndexes()

32

Query Plan Execution Stages

• COLLSCAN : for a collection scan• IXSCAN : for scanning index keys• FETCH : for retrieving documents• SHARD_MERGE : for merging results from shards

33

Add an Index> db.users.find( {"username" : "USER_999999”} ).explain("executionStats”).executionStats{

"executionSuccess" : true,"nReturned" : 1,"executionTimeMillis" : 0,"totalKeysExamined" : 1,"totalDocsExamined" : 1,

…

34

Execution Stage"executionStages" : {

"stage" : "FETCH","nReturned" : 1,"executionTimeMillisEstimate" : 0,"docsExamined" : 1,,"inputStage" : {

"stage" : "IXSCAN","nReturned" : 1,"executionTimeMillisEstimate" : 0,"keyPattern" : {

"username" : 1},"indexName" : "username_1","isMultiKey" : false,"isUnique" : false,"isSparse" : false,"isPartial" : false,"indexVersion" : 1,"direction" : "forward","indexBounds" : {

"username" : ["[\"USER_999999\", \"USER_999999\"]"

]},"keysExamined" : 1,"seenInvalidated" : 0

}}

}

35

Drivers and Frameworks

Morphia

MEAN Stack

36

What we have learned

• How to create a database and a collection• How to insert content into that collection• How to query the collection• How to update a document in place• How to delete a document• How to check the efficiency of an operation• How to add an index• How to check an index is being used in an operation

37

Next Webinar : Thinking in Documents

• Rather than normalisation we look at a hybrid approach to schema that provides a coherent mapping between objects in your application and objects in your database.

• Then we will optimise that schema for query purposes based on the pattern of queries we expect to see.

• Finally we will show how dynamic schema and schema validation allow you to extend your schema in a controlled fashion

14-June-2016, 14:00 GMT.

Q&A