no sql landscape_nosqltips
TRANSCRIPT
The NoSQL Landscape
Objective – Reasonable understanding of the non-relational or NoSQL data stores and how they relate to RDBMS databases we are all used to working with.
About Me
Chief Architect – youwho.com
Former dot com CTO
NoSql advocate
nosqltips.blogspot.com
@nosqltips on twitter
Agenda
What is NoSQL?
Landscape
Vocabulary and concepts
CAP Theorem
SQL vs NoSQL comparison
Overview of each type w/ examples
Question and Answer
Vocabulary
CAP Theorem – consistency, availability, partitioning
ACID – Atomic, Consistent, Isolated, Durable
BASE – Basically Available, Soft state, Eventually consistent
RDF – Resource Description Framework
Sharding – Partitioning, distributed
Web Scale – Google, Twitter, Facebook, etc
CAP Tuning
NRW
N: Number of Data Copies
R: Read Quorum
W: Write Quorum
Hard Consistency – RDBMS
Soft Consistency – No Guarantees
Eventual Consistency – Most NoSQL
Cap Tuning Chart
NRW Outcome
N=3 Magic Number of Data Replicas
W=N R=1 Read Optimized – Strong Consistency.
W=1 R=N Write Optimized – Strong Consistency.
W+R > N Strong Consistency on Read and Write.
W+R <= N Weak Eventual Consistency. Read may not see the latest Data.
N > W > 1 Eventual Consistency - Most NoSQL data stores live here.
Eventual Consistency
All replicas have same data – eventually
Milliseconds to seconds
Not all applications are compatible
Various ways to ensure latest data
Vector Clocks, Read Repair, Gossiping
Application determines correct data
Comparison
SQL
Prefers big-box, self redundant
Keep things from breaking
Solidly in CA land
P is difficult and expensive
Query by SQL
Stored procedures
NoSQL
Prefers commodity hardware, distributed
Assume things break or are broken
Mostly AP, some tunable
P generally easy
Custom API, SQLish
Map/Reduce
Comparison
SQL
ACID transactions
Advanced indexing
Foreign key support
Strong lock support
Schema centric
API – usually JPA or JDBC
Strong access control
NoSQL
BASE transactions
Key only to Advanced
Usually none
Usually none
Usually schema-less
Depends on implementation
Usually none
Comparison
SQL
Complex disk store, random access
Easy for dev with JPA/Hibernate/SQL
Multi-platform
General purpose
Strong commercial support
Great tool support
NoSQL
Usually append only, 1 seek, 1 read
Puts more work on application dev
Favors Linux/Unix
More special purpose
Strong to no commercial support
Not so much
Column Stores
Data stored by column instead of row
Schema-less
Non-relational, data is de-normalized
Column format stores sparse data efficiently
Column families cannot change
10,000+ columns by 100 million+ rows
Easy sharding (partitioning)
Usually not ACID compliant
Column stores
BigTable – Google, 2006 paper
Hadoop/HBase – Part of Apache Hadoop
Cassandra – Facebook, LAN/WAN replication
Hypertable – Pluggable DFS, HQL
Vertica – Full SQL implementation
Amazon SimpleDB – Cloud store
Document Stores
CAP tunable
Either key/value or bucket/key/value
Easy/Auto sharding - Consistent hashing
Usually ACID compliant
Not SQL compliant, maybe custom query
Easy implementation via map or custom api
Document stores
Amazon – Dynamo and S3 (cloud based)
Riak – CAP tunable, built in map/reduce
CouchDB – ACID, REST api
MongoDB – Indexing, query support
Voldemort – Java, pluggable serialization
MySQL – Key access, denormalize schema, kill indexes
Memory Stores
Mostly in the CA realm
P can be tough depending on implementation
Some are distributed, some local only
Usually key-value stores
Many are disk backed, append only files
Designed for very high-speed access
Memory stores
CouchBase – Membase + CouchDb
Memcached – Local map
Coherence – Commercial Oracle, distributed
Redis – Supports hash, list, set, and sorted set, data structure server
Tokyo/Kyoto Cabinet – disk backed map
Infinispan – JSR-107 jcache impl
Scalaris – Erlang, strong consistency
Graph/Triple Store
Model relationships well, bi-directional
Node/edges – edges can be weighted or not
RDF Triple – subject -> predicate -> object, w3c standard for semantic web
Many implement SPARQL, object api
Sharding can difficult because of graph nature
Schema-less – nodes, edges, properties
Fast set operations
Graph/Triple Stores
Neo4j – ACID transactions, object API
Alegrograph – Reference impl of SPARQL
Bigdata – dynamic sharding
Trinity – Microsoft research
Infinite Graph – Distributed, cross-platform
FlockDb – Twitter, fast set operations
Infogrid – Object based, REST api
Interesting Integrations
Lucene - Document Store with Search as Query Language
SOLR and Elastic Search – Scalable Lucene
Riak Search – Elang impl of Lucene APIs
Solandra – Lucene on Cassandra backend
Couchdb-lucene – Integration
DistributedLucene – Lucene on Hadoop
Neo4j – Full Text Search on Graph Store
Worth Mentioning
Configuration Dbs – ZooKeeper, Doozer
Distributed configuration, locks, synchronization
Used to make other apps scalable
XML Dbs – eXist, BaseX, Xindice
XML only, Xquery, Xpath, ACID, GUI support
non-distributed
Case Study - HBase
Apache – part of Hadoop/HDFS
Requires ZooKeeper
Java based
Runs well on Amazon EC2
Excellent language support
Supports REST interface
HBase continued
Map/Reduce via Hadoop
Schema-less, column families fixed
Nearly unlimited columns and rows
HBQL – partial sql + JDBC support
Some ACID support, atomicity, durability
Integration with Hive for data warehousing, ad-hoc query support - HiveQL
Case Study - Riak
Data Model – Bucket/Key/Value
Value has MIME type, byte[]
Value supports one-way Links, basic graph
Erlang, Protocol Buffers, REST interfaces
Pre/Post Commit Hooks
CAP Tunable per bucket
Map/Reduce – Erlang and Javascript
Riak Continued
Vector Clocks
Read repair for R < N
Peer-to-Peer, Nothing Shared Architecture
Replication across data centers
Pluggable storage
API for Most Languages + REST
Commercial Support
Case Study - Redis
Supports hash, list, set, and sorted set
Fast set operations
Atomic updates
Everything stored in memory
Persistence to disk – periodic save, append only file, can be compacted
Good API support, JDBC subset driver
Redis Continued
Master – slave replication, read scalability, redundancy, slave can sync to disk
Can swap out values, keys must be in memory
Can be used as pub/sub messaging system
Can send multiple commands in single request
Built to be extremely fast
Supports very high speed atomic counters
Case Study - Neo4j
Java based – cross platform
ACID transactions
Durable persistence
Handle billions of nodes/edges single machine
Supports bulk data loading
Good language support
Neo4j Continued
Spatial index support
RDF triples/OWL/SPARQL support
Replication and HA – commercial version
Object oriented API
Sharding at client level
Dual open source and commercial license
Resources
fallabs.com/tokyocabinet
fallabs.com/kyotocabinet
redis.io
www.membase.org
neo4j.org
en.wikipedia.org/wiki/Triplestore
en.wikipedia.org/wiki/Graph_theory
research.microsoft.com/en-us/projects/trinity
Resources
www.jboss.org/infinispan
basho.com
nosqlpedia.com/wiki/Consistency_models_in_nonrelational_dbs
www.hypertable.org
project-voldemort.com
www.allthingsdistributed.com/2007/10/amazons_dynamo.html
Resources
nosql-database.org
couchdb.apache.org
engineering.twitter.com/2010/05/introducing-flockdb.html
infinitegraph.com
nosql-database.org
http://www.w3.org/TR/rdf-concepts/