riak intro with azure
DESCRIPTION
This slide deck is the part of the talk, generally centered around the topics and details of the Riak Architecture & related material. It currently doesn't have the Azure sample commands or other elements around that, as it is the live part of the presentation. I'll likely add these parts in the future though.TRANSCRIPT
&
Thursday, March 7, 13
#WHOISAdron Hall | @adron | Coder, Messenger, Recon
Thursday, March 7, 13
芭蕉
Thursday, March 7, 13
Thursday, March 7, 13
Thursday, March 7, 13
Distributed, masterless, highly-available key/value store
Thursday, March 7, 13
Horizontal Scalability
Fault-Tolerance
Low-latency
Ops Friendliness
Predictability
High-Availability
DESIGN GOALS
Thursday, March 7, 13
When to use Riak...
Thursday, March 7, 13
Metadata Users/Profiles
Object StorageSession Storage
Sensor Data Logging Systems
Record SystemsNotification Systems
RIAK USE CASES
Thursday, March 7, 13
IN PRODUCTION AT
And 1000s more...
Thursday, March 7, 13
DATA MODEL
Thursday, March 7, 13
{“Key”:“Value”}
• Values are stored against keys
• Key/Value + Metadata = Object
• Fundamental Unit of Replication
• Any Datatype will work
• Record to disk in binary format
Thursday, March 7, 13
<<BUCKET>>/<<KEY>>
• Virtual Namespace
• Bucket + Keys = Object Address
• Buckets have properties
• Objects in bucket inherit properties
• No relationships between buckets
Thursday, March 7, 13
DATA ACCESS
Thursday, March 7, 13
INTERFACES
HTTP API - Via a little piece of magic called Webmachine
Protocol Buffers API - Thanks, Google!
Largely-faithful REST implementation
Compact, binary protocol
Thursday, March 7, 13
CLIENT LIBSPython
Ruby
PHP
OCaml
Java
Perl
Erlang
Node.js
C/C++
Haskell
Clojure
Scala
Go Dart
.NET
And more. Supported by either Basho or our community.
Thursday, March 7, 13
RIAK GIVES YOU [FOUR] WAYS TO STORE, RETRIEVE, AND QUERY
DATA
Thursday, March 7, 13
123456789101112131415161718
CRUD
// PUTPUT /buckets/bucket/keys/key // User-‐defined key
POST /buckets/bucket/keys/key // Riak-‐defined key
DELETE /buckets/bucket/keys/key
GET /buckets/bucket/keys/key
// GET
// DELETE
Thursday, March 7, 13
MapReduce
Distributed processing system using Riak Pipe
Efficient for targeted queries over known key range
Write jobs in Erlang or JS. (Erlang more performant)
Thursday, March 7, 13
Secondary Indexing (2i)
riak_object
riak_object
X-Riak-Index-email_bin
X-Riak-Index-value_int
“42”
Tag objects with custom metadata on PUT...Exact match and range queries...No multi-index queries yet...
Pagination is on its way...
Thursday, March 7, 13
Riak Search
Store and index documents (JSON, text, XML, etc)
Current Riak Search supports subset of Solr API
Next iteration (Yokozuna; in beta)will implement distributed Solr on Riak. It will be sexy.
Looking for beta testers to help harden Yokozuna
Thursday, March 7, 13
ARCHITECTURE
The scaleability and ease of operation goals inform architectural decisions.
These come with tradeoffs.
Consistent Hashing
Virtual Nodes
Append-only storage
Handoff/Rebalancing
Vector Clocks
Active Anti-Entropy*
Thursday, March 7, 13
Consistent Hashing
Location of data in the Riak ring is determined based on hash of bucket + key.
Provides even distribution of storage and query load
Trades off advantages gained from locality
- e.g. Range queries and aggregates
Thursday, March 7, 13
Consistent Hashing
Thursday, March 7, 13
Virtual Nodes
Unit of addressing and concurrency in Riak
Each physical host manage many vnodes Partition count / physical machines = vnodes/machine*
Decouples physical assets from data distribution. This provides:
- simplicity in cluster sizing - failure isolation
Thursday, March 7, 13
Handoff/Rebalancing
Mechanisms for data rebalancing
When nodes join/leave cluster, handoff and rebalancing manage the date shuffling dynamically
Trades off speed of convergence vs. effects on cluster performance
- causes disk & network load
Thursday, March 7, 13
Vector Clocks
VCs used to rectify object consistency at READ time.
Lots of knobs to turn; well-documented
Trades off space, speed, and complexity for safety
- will store all sibling objects until resolved- can lead to object size issues
Thursday, March 7, 13
Append-Only Storage
Riak provides a pluggable backend interface. (Write your own; we’ll probably hire you...)
Bitcask, LevelDB are most-heavily used. Both are append - only
Provides crash safety and speed.
Trade off: periodic compaction/merge ops
Thursday, March 7, 13
RIAK 1.3 (AKA “new hotness”)
Active Anti Entropy MapReduce ImprovementsIPv6 Support
Riaknostic included by default
Much more Riak Control improvements
Full release notes: https://github.com/basho/riak/blob/1.3/RELEASE-NOTES.md
Thursday, March 7, 13
FUTURE WORK*(1.4 and beyond)
(* all code subject to ship early, late, or not at all)
Dynamic Ring SizeYokozunaCRDTs/Data TypesRiak Object
Consistency2i ImprovementsRiak Pipe workMuch more
Thursday, March 7, 13
S3-API compatible and supports per-tenant reporting for billing and metering use cases. Additional APIs on the way.
Multi-tenant cloud storage software for public and private clouds.
Designed to provide simple, available, distributed cloud storage at any scale.
Stores files of arbitrary size. Under the hood stores 1MB chunks along side a manifest.
Stateless proxy (CS) does chunking. Riak does distribution, storage, etc.
Thursday, March 7, 13
Data transfer is unidirectional (source -> sink). Bidirectional synchronization can be achieved by configuring a pair of connections between clusters.
Extends Riak's capabilities with:- multi-datacenter replication- SNMP Configuration- JMX-Monitoring- 24x7 support from Basho Engineers
One cluster acts as a "source cluster". The source cluster replicates its data to one or more "sink clusters" using either real-time or full sync.
Thursday, March 7, 13
RIAK COMMUNITYMailing List - 1300 developers
IRC - 200+ people every day yelling about software
GitHub - 1000s of watchers; 200+ contributors to all projects
Meetups - 10 Countries, 23 Cities, 3700+ Members & growing fast!
Deployments - 1000s in production.
Thursday, March 7, 13
May 13-14th in New York Cityricon.io/east.html
Talks, hacking, partiesDedicated to the future of Riak and distributed systems in production
REGISTER NOW!
https://ricon-east-2013.eventbrite.com/?discount=lovevnodes
Thursday, March 7, 13
GETTING STARTEDDownloads - http://docs.basho.com/riak/latest/downloads/Docs - http://docs.basho.comRiak Source Code - github.com/basho/riak
All Basho source Code - github.com/basho/Riak Mailing List - http://bit.ly/riak-list
Email or Tweet me @adron or [email protected]
Thursday, March 7, 13
Let’s Talk UI & CLI - Demo Things
Thursday, March 7, 13
#WHOISAdron Hall | @adron | Coder, Messenger, Recon
Thursday, March 7, 13