nosql e python rupy 2012
DESCRIPTION
http://www.youtube.com/watch?v=EGDv8jctVqw Introdução ao MongoDB, Redis e Cassandra através de Python. Quais as características principais de um banco orientado a documentos, chave-valor e colunar. Que vantagens esses bancos possuem em relação a um banco relacional tradicional. No final farei uma aplicação que persiste dados do Twitter e Facebook nos três bancos mencionados.TRANSCRIPT
![Page 1: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/1.jpg)
NoSQL e Python @fmasanori
![Page 2: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/2.jpg)
• Graduado IME-USP e Mestrado ITA
• ed e lint Cobra Tecnologia
• Software Express
• Credicard Mastercard
• PriceWaterhouseCoopers
• ITAU BankBoston
• Docente FATEC SJC (amo dar aulas): – Business Intelligence/Data Warehouse
– NoSQL databases
• Interesses: Python, NoSQL, Google Technology, Facebook
@fmasanori
![Page 3: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/3.jpg)
![Page 4: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/4.jpg)
• Python?
• NoSQL?
• pymongo?
• redis-py?
• pycassa?
Conhecendo os assistentes
![Page 5: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/5.jpg)
• Como comecei com NoSQL
• LucidDB: my first NoSQL experiment
• Flexible Schema
• Horizontal Scalability
• MongoDB (document)
• Redis (key-value)
• Cassandra (columnar)
• Demo
Agenda
![Page 6: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/6.jpg)
Como comecei com NoSQL
![Page 7: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/7.jpg)
• Me ensinaram que tudo deveria ser normalizado...
• E que o mundo era SQL
• Mas um dia tive problemas:
– Analytics
– Twitter and Facebook data persistence
One Simple Idea
![Page 8: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/8.jpg)
• 3 years ago
• Benchmark LucidDB x MySQL for Business Intelligence applications
• LucidDB: columnar NoSQL for BI
• Analytic applications:
– Read only
– Massive aggregations
– Denormalization
LucidDB
![Page 9: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/9.jpg)
Queries
• Which movie genres generate the most revenue? (Q1)
• How is our revenue evolving over time? (Q2)
• At which time of day do costumers place the most orders? (Q3)
• How effective are the promotions we launch? (Q4)
Queries
![Page 10: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/10.jpg)
Average time (ms)
0
5000
10000
15000
20000
25000
30000
35000
Q1 Q2 Q3 Q4
LucidDB
MySQL
![Page 11: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/11.jpg)
JSON persistence
![Page 12: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/12.jpg)
menina: aaaaaaaaaaaaaa eu NAO GOSTO DELE
amiga: e vc ficará rica , e irá dividir sua riqueza cm sua amg aq
*u*
menina: kkkkkk como assim eu fikar rica pq?
amiga: pq ele é inteligente , e vai arranjar um ótimo emprego , e
ganhar mt dinheiro rs
menina: kkkkk ele é inteligente como assim ??
amiga: AQUELE VIADO PASSOU NO CASDINHO, E EU NÃO !
Sentiment Analysis (Twitter)
![Page 13: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/13.jpg)
• Sabe o que é ter vontade de acordar cedo no domingo para estudar?!?
• Bom, eu também não sei explicar, mas é o que estou sentindo no momento .-.
• Bora Casdinho!
Sentiment Analysis (Facebook)
![Page 14: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/14.jpg)
Social Data Change Fast
![Page 15: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/15.jpg)
“Life begins at end of your
comfort zone” Neale Donald Walsch
Slide EuroPython 2012 Python, Flask e MongoDB depois
de 20 anos com .NET
![Page 16: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/16.jpg)
“It’s better to think of NoSQL as a
movement rather
than a technology” Martin Fowler
![Page 17: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/17.jpg)
NoSQL vale a pena
![Page 18: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/18.jpg)
Empregabilidade
![Page 19: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/19.jpg)
Empregabilidade
![Page 20: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/20.jpg)
Empregabilidade
![Page 21: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/21.jpg)
Dynamic Schema
O que as empresas esperam de um NoSQL?
![Page 22: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/22.jpg)
Flexible Schema
O que as empresas esperam de um NoSQL?
![Page 23: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/23.jpg)
Relational DB
![Page 24: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/24.jpg)
Martin Fowler
Relational DB
![Page 25: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/25.jpg)
• The difference between the relational model and the in-memory data structures (Fowler)
• Limitations of relational DB: values only in tuples
• We need a more complex structure than a set of tuples as a data unit
Impedance mismatch
![Page 26: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/26.jpg)
Costumers Orders
Martin Fowler
Aggregate Model
![Page 27: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/27.jpg)
Costumers
Martin Fowler
Aggregate Model
![Page 28: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/28.jpg)
Scalability
![Page 29: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/29.jpg)
If I had asked people what they wanted, they would have said faster
horses. Henry Ford.
![Page 30: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/30.jpg)
• Escalabilidade para leituras num banco relacional: trabalhoso
• Escalabilidade para escritas num banco relacional: MUITO, MUITO, trabalhoso
• (se conseguir seu BD deixou de ser relacional...)
Escalabilidade vertical
![Page 31: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/31.jpg)
Escalabilidade vertical
![Page 32: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/32.jpg)
Escalabilidade horizontal
![Page 33: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/33.jpg)
Escalabilidade horizontal
![Page 34: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/34.jpg)
• Bigtable: A distributed storage system for structured data, 2006
• Dynamo: Amazon’s highly available key-value store, 2007
Dois artigos famosos
![Page 35: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/35.jpg)
Apache Hadoop
![Page 36: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/36.jpg)
Not Only SQL
![Page 37: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/37.jpg)
There is no best NoSQL
![Page 38: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/38.jpg)
• JSON oriented • Autosharding • Full featured indexes • Great community
![Page 39: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/39.jpg)
Dowload and install:
http://www.mongodb.org/downloads
Criar c:/data/db
Servidor:
bin/mongod
Cliente:
bin/mongo
Cliente pymongo:
http://api.mongodb.org/python/current/
import pymongo
Curso completo MongoDB com Python:
http://education.10gen.com/
![Page 40: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/40.jpg)
• Data Structure Server • In-memory dataset • Persistence options
![Page 41: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/41.jpg)
• CPU performance 175X (96) • Os discos continuam iguais • Memory 1M faster • 1M = $1 (2000) x 1 cent (2010)
![Page 42: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/42.jpg)
http://redis.io/download
https://github.com/MSOpenTech/redis
(for Windows)
Servidor:
redis-server
Cliente:
redis-cli
Cliente redis-py:
https://github.com/andymccurdy/redis-py
import redis
![Page 43: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/43.jpg)
Cassandra: • Linearly Scalable • Column based • CQL
![Page 44: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/44.jpg)
Martin Fowler
![Page 45: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/45.jpg)
MySQL:
~300ms write
~350ms read
Cassandra:
~0.12ms write
~15ms read
Cassandra vs MySQL (50GB)
![Page 46: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/46.jpg)
![Page 47: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/47.jpg)
(Timo Elliott, SAP)
Relational DB
![Page 48: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/48.jpg)
Columnar DB
(Timo Elliott, SAP)
![Page 49: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/49.jpg)
Download DataStax Community Edition
Servidor:
cassandra
Cliente:
cassandra-cli –h localhost –p 9160
Conflitos:
cassandra.bat (windows)
/etc/cassandra/cassandra-env.sh (ubuntu)
Cliente pycassa:
https://github.com/pycassa/pycassa/downloads
import pycassa
![Page 50: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/50.jpg)
Talk is cheap.
Show me the code. Linus Torvalds
![Page 51: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/51.jpg)
MongoDB Server
![Page 52: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/52.jpg)
Cassandra Server
![Page 53: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/53.jpg)
Cassandra Server
![Page 54: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/54.jpg)
Redis Server
![Page 55: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/55.jpg)
Obs.: não aparece o token de autenticação
Twitter and Facebook data
![Page 56: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/56.jpg)
Pymongo
![Page 57: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/57.jpg)
Redis-py
![Page 58: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/58.jpg)
Pycassa
![Page 59: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/59.jpg)
Pymongo response
![Page 60: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/60.jpg)
Redis-py response
![Page 61: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/61.jpg)
Pycassa response
![Page 62: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/62.jpg)
Bigliografia
![Page 63: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/63.jpg)
Bigliografia
![Page 64: NoSQL e Python RuPy 2012](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5552dbe2b4c905014c8b4a3b/html5/thumbnails/64.jpg)
Bigliografia
Google IO 2012 - SQL vs NoSQL Battle of the Backends