2014 03-12-fr schema design and app architecture-2
DESCRIPTION
TRANSCRIPT
![Page 1: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/1.jpg)
Tugdual Grall (@tgrall) Alain Hélaïli (@AlainHelaili)
#MongoDBBasics @MongoDB
Construire une application avec MongoDBDesign du schéma et architecture applicative
![Page 2: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/2.jpg)
Agenda
• Travailler avec des documents
• Fonctionnalités de l’application
• Design du schéma
• Architecture de ‘myCMS’ et exemples de code
• Q&A
![Page 3: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/3.jpg)
Vocabulaire
RDBMS MongoDB
Database ➜ Database
Table ➜ Collection
Row ➜ Document
Index ➜ Index
Join ➜ Embedded Document
Foreign Key ➜ Reference
![Page 4: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/4.jpg)
Modélisation des données
![Page 5: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/5.jpg)
Exemple de document
{ ‘_id’ : ObjectId(..), ‘title’ : ‘Schema design in MongoDB’, ‘author’ : ‘mattbates’,‘text’ : ‘Data in MongoDB has a flexible
schema..’,‘date’ : ISODate(..),‘tags’ : [‘MongoDB’, ‘schema’],‘comments’ : [ { ‘text ‘ : ‘Really useful..’, ts:
ISODate(..) } ]}
![Page 6: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/6.jpg)
Fonctionnalités de ‘myCMS’
• Différents types d’articles et catégories.
• Les utilisateurs peuvent s’enregistrer, se
connecter/déconnecter, et éditer leur profil.
• Les utilisateurs peuvent poster des articles et
effectuer des commentaires sur ces articles.
• Des statistiques d’utilisation sont collectées et
analysées – publications, visualisations, interactions –
pour le site et le back-office (analytics).
![Page 7: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/7.jpg)
Entités de ‘myCMS’
• Articles • Differents types – blogs, galleries, enquêtes• Multimedia embarqué (images, videos)• Tags
• Utilisateurs• Profils
• Interactions• Commentaires• Vues
![Page 8: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/8.jpg)
Typical (relational) ERD
![Page 9: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/9.jpg)
# Python dictionary (or object)
>>> article = { ‘title’ : ‘Schema design in MongoDB’,
‘author’ : ‘mattbates’,
‘section’ : ‘schema’,
‘slug’ : ‘schema-design-in-mongodb’,
‘text’ : ‘Data in MongoDB has a flexible
schema..’,
‘date’ : datetime.datetime.utcnow(),
‘tags’ : [‘MongoDB’, ‘schema’] }
>>> db[‘articles’].insert(article)
Design du schéma… en code
![Page 10: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/10.jpg)
>>> img_data = Binary(open(‘article_img.jpg’).read())
>>> article = { ‘title’ : ‘Schema design in MongoDB’,
‘author’ : ‘mattbates’,
‘section’ : ‘schema’,
‘slug’ : ‘schema-design-in-mongodb’,
‘text’ : ‘Data in MongoDB has a flexible
schema..’,
‘date’ : datetime.datetime.utcnow(),
‘tags’ : [‘MongoDB’, ‘schema’],
‘headline_img’ : {
‘img’ : img_data,
‘caption’ : ‘A sample document at the shell’
}}
>>> db[‘articles’].insert(article)
Ajoutons une image
![Page 11: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/11.jpg)
>>> article = { ‘title’ : ‘Favourite web application framework’,
‘author’ : ‘mattbates’,
‘section’ : ‘web-dev’,
‘slug’ : ‘web-app-frameworks’,
‘gallery’ : [
{ ‘img_url’ : ‘http://x.com/45rty’, ‘caption’ :
‘Flask’, ..},
..
]
‘date’ : datetime.datetime.utcnow(),
‘tags’ : [‘MongoDB’, ‘schema’],
}
>>> db[‘articles’].insert(article)
Et differents types d’articles
![Page 12: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/12.jpg)
>>> user= { ‘user’ : ‘mattbates’,
‘email’ : ‘[email protected]’,
‘password’ : ‘xxxxxxxxxx’,
‘joined’ : datetime.datetime.utcnow()
‘location’ : { ‘city’ : ‘London’ },
}
>>> db[‘users’].insert(user)
Utilisateurs et profils
![Page 13: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/13.jpg)
Modélisation des commentaires (1)
• Deux collections – articles et comments• Référence (i.e. foreign key) pour les relier• MAIS.. N+1 requêtes pour récupérer articles et
commentaires
{‘_id’ : ObjectId(..),‘title’ : ‘Schema design in
MongoDB’, ‘author’ : ‘mattbates’,
‘date’ : ISODate(..),‘tags’ : [‘MongoDB’,
‘schema’], ‘section’ : ‘schema’, ‘slug’ : ‘schema-design-in-mongodb’, ‘comments’ : [ ObjectId(..), …]}
{ ‘_id’ : ObjectId(..), ‘article_id’ : 1,
‘text’ : ‘A great article, helped me understand schema design’, ‘date’ : ISODate(..),, ‘author’ : ‘johnsmith’}
![Page 14: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/14.jpg)
Modélisation des commentaires (2)• Une seule collection
articles–commentaires embarqués dans les documents article
• Pros• Requête unique, design
optimisé pour la lecture• Localité (disk, shard)
• Cons• Tableau de commentaires
non borné; taille des documents va croitre (rappel : limite 16MB)
{‘_id’ : ObjectId(..),‘title’ : ‘Schema design in
MongoDB’, ‘author’ : ‘mattbates’,
‘date’ : ISODate(..),‘tags’ : [‘MongoDB’,
‘schema’], … ‘comments’ : [
{ ‘text’ : ‘A great article, helped me understand schema design’, ‘date’ : ISODate(..),
‘author’ : ‘johnsmith’
},…
]}
![Page 15: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/15.jpg)
Modélisation des commentaires (3)
• Autre option: hybride de (2) et (3), embarquer top x commentaires (e.g. par date, popularité) dans le document article • Tableau de commentaires de taille fixe (2.4
feature)• Tous les autres commentaires sont déversés
dans une collection ‘comments’ par lots• Pros– Taille des documents plus stable– moins de
déplacements– Basé sur une seule requête dans la plupart des accès– Historique complet des commentaires disponible via
requêtage/agrégation
![Page 16: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/16.jpg)
Modélisation des commentaires (3){
‘_id’ : ObjectId(..),‘title’ : ‘Schema design in
MongoDB’, ‘author’ : ‘mattbates’,
‘date’ : ISODate(..),‘tags’ : [‘MongoDB’, ‘schema’],
…‘comments_count’: 45,
‘comments_pages’ : 1 ‘comments’ : [
{ ‘text’ : ‘A great article, helped me understand schema design’, ‘date’ : ISODate(..),
‘author’ : ‘johnsmith’},…
]}
Ajout d’un compteur de commentaires• Elimine les
comptages lors de la lecture
Tableau de commentaires de taille fixe• 10 plus récents• Triés par date lors
de l’insertion
![Page 17: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/17.jpg)
Modélisation des commentaires (3){
‘_id’ : ObjectId(..), ‘article_id’ : ObjectId(..), ‘page’ : 1, ‘count’ : 42 ‘comments’ : [
{ ‘text’ : ‘A great article, helped me understand schema design’, ‘date’ : ISODate(..), ‘author’ : ‘johnsmith’ }, …}
Document ‘lot de commentaires’ contenant jusqu’à 100 commentaires
Tableau de 100 commentaires
![Page 18: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/18.jpg)
Modélisation des interactions
• Interactions– Article vus– Commentaires– (Social media sharing)
• Besoins– Séries temporelles– Pré-agrégations pour préparer l’analytique
![Page 19: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/19.jpg)
Modélisation des interactions
• Document par article par jour
– ‘bucketing’
• Compteur journalier et sous-
document par heure pour les
interactions
• Tableau borné (24 heures)
• Requête unitaire, prêt à être
graphé
{‘_id’ : ObjectId(..),
‘article_id’ : ObjectId(..), ‘section’ : ‘schema’,
‘date’ : ISODate(..),‘daily’: { ‘views’ : 45,
‘comments’ : 150 } ‘hours’ : { 0 : { ‘views’ : 10 }, 1 : { ‘views’ : 2 }, … 23 : { ‘comments’ : 14, ‘views’ : 10 } }}
![Page 20: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/20.jpg)
JSON and RESTful API
Client-sideJSON
(eg AngularJS, (BSON)
Real applications are not built at a shell – let’s build a
RESTful API.
Pymongo driver
Python web app
HTTP(S) REST
Examples to follow: Python RESTful API using Flask
microframework
![Page 21: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/21.jpg)
myCMS REST endpointsMethod URI Action
GET /articles Retrieve all articles
GET /articles-by-tag/[tag] Retrieve all articles by tag
GET /articles/[article_id] Retrieve a specific article by article_id
POST /articles Add a new article
GET /articles/[article_id]/comments
Retrieve all article comments by article_id
POST /articles/[article_id]/comments
Add a new comment to an article.
POST /users Register a user user
GET /users/[username] Retrieve user’s profile
PUT /users/[username] Update a user’s profile
![Page 22: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/22.jpg)
$ git clone http://www.github.com/mattbates/mycms-mongodb
$ cd mycms-mongodb
$ virtualenv venv
$ source venv/bin/activate
$ pip install –r requirements.txt
($ deactivate)
Getting started with the skeleton code
![Page 23: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/23.jpg)
@app.route('/cms/api/v1.0/articles', methods=['GET'])
def get_articles():
"""Retrieves all articles in the collection
sorted by date
"""
# query all articles and return a cursor sorted by date
cur = db['articles'].find().sort({'date':-1})
if not cur:
abort(400)
# iterate the cursor and add docs to a dict
articles = [article for article in cur]
return jsonify({'articles' : json.dumps(articles, default=json_util.default)})
RESTful API methods in Python + Flask
![Page 24: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/24.jpg)
@app.route('/cms/api/v1.0/articles/<string:article_id>/comments', methods = ['POST'])
def add_comment(article_id):
"""Adds a comment to the specified article and a
bucket, as well as updating a view counter
"”” …
# push the comment to the latest bucket and $inc the count
page = db['comments'].find_and_modify(
{ 'article_id' : ObjectId(article_id),
'page' : comments_pages},
{ '$inc' : { 'count' :1 },
'$push' : {
'comments' : comment } },
fields= {'count':1},
upsert=True,
new=True)
RESTful API methods in Python + Flask
![Page 25: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/25.jpg)
# $inc the page count if bucket size (100) is exceeded
if page['count'] > 100:
db.articles.update(
{ '_id' : article_id,
'comments_pages': article['comments_pages'] },
{ '$inc': { 'comments_pages': 1 } } )
# let's also add to the article itself
# most recent 10 comments only
res = db['articles'].update(
{'_id' : ObjectId(article_id)},
{'$push' : {'comments' : { '$each' : [comment],
'$sort' : {’date' : 1 },
'$slice' : -10}},
'$inc' : {'comment_count' : 1}})
…
RESTful API methods in Python + Flask
![Page 26: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/26.jpg)
def add_interaction(article_id, type):
"""Record the interaction (view/comment) for the
specified article into the daily bucket and
update an hourly counter
"""
ts = datetime.datetime.utcnow()
# $inc daily and hourly view counters in day/article stats bucket
# note the unacknowledged w=0 write concern for performance
db['interactions'].update(
{ 'article_id' : ObjectId(article_id),
'date' : datetime.datetime(ts.year, ts.month, ts.day)},
{ '$inc' : {
'daily.views' : 1,
'hourly.{}.{}'.format(type, ts.hour) : 1
}},
upsert=True,
w=0)
RESTful API methods in Python + Flask
![Page 27: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/27.jpg)
$ curl -i http://localhost:5000/cms/api/v1.0/articles
HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 20
Server: Werkzeug/0.9.4 Python/2.7.6
Date: Sat, 01 Feb 2014 09:52:57 GMT
{
"articles": "[{\"author\": \"mattbates\", \"title\": \"Schema design in MongoDB\", \"text\": \"Data in MongoDB has a flexible schema..\", \"tags\": [\"MongoDB\", \"schema\"], \"date\": {\"$date\": 1391293347408}, \"_id\": {\"$oid\": \"52ed73a30bd031362b3c6bb3\"}}]"
}
Testing the API – retrieve articles
![Page 28: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/28.jpg)
$ curl -H "Content-Type: application/json" -X POST -d '{"text":"An interesting article and a great read."}' http://localhost:5000/cms/api/v1.0/articles/52ed73a30bd031362b3c6bb3/comments
{
"comment": "{\"date\": {\"$date\": 1391639269724}, \"text\": \"An interesting article and a great read.\"}”
}
Testing the API – comment on an article
![Page 29: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/29.jpg)
Schema iteration
New feature in the backlog?
Documents have dynamic schema so we just iterate the object schema.
>>> user = { ‘username’ : ‘matt’,
‘first’ : ‘Matt’,
‘last’ : ‘Bates’,
‘preferences’ : { ‘opt_out’ : True } }
>>> user..save(user)
![Page 30: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/30.jpg)
Scale out with sharding
![Page 31: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/31.jpg)
![Page 32: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/32.jpg)
![Page 33: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/33.jpg)
Résumé
• Document avec schéma flexible et possiblité d’embarquer des structures de données riches et complexes
• Différentes stratégies pour assurer la performance
• Design du schéma s’appuie sur les modes d’accès – pas sur les modes de stockage
• Références pour plus de flexibilité
• Garder en tête la distribution horizontale (shard key)
![Page 34: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/34.jpg)
Further reading
• ‘myCMS’ skeleton source code:http://www.github.com/mattbates/mycms-mongodb
• Use case - metadata and asset management:http://docs.mongodb.org/ecosystem/use-cases/metadata-and-asset-management/
• Use case - storing comments:http://docs.mongodb.org/ecosystem/use-cases/storing-comments/
![Page 35: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/35.jpg)
Prochaine Session– 26 Mars
• Interactions avec la base de données• Langage de requêtes (find & update)• Interactions entre l’application et la base
• Exemples de code
![Page 36: 2014 03-12-fr schema design and app architecture-2](https://reader036.vdocuments.mx/reader036/viewer/2022081602/549e7309ac795910768b477b/html5/thumbnails/36.jpg)
#MongoDBBasics
Merci
Q&A avec l’équipe