how you can benefit from using redis - ramirez
DESCRIPTION
Slides from Javier Ramirez talk @ codemotion roma 2014TRANSCRIPT
![Page 1: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/1.jpg)
javier ramirez@supercoco9
How you can benefit from using
Redis
![Page 2: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/2.jpg)
![Page 3: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/3.jpg)
![Page 4: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/4.jpg)
![Page 5: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/5.jpg)
![Page 6: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/6.jpg)
![Page 7: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/7.jpg)
I was Lois Lane
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 8: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/8.jpg)
redis has super powers
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 9: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/9.jpg)
myth: the bottleneck
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 10: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/10.jpg)
redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop,mset -P 16 -q
On my laptop:
SET: 513610 requests per secondGET: 641436 requests per secondLPUSH: 845665 requests per secondLPOP: 783392 requests per secondMSET (10 keys): 89988 requests per second
On a small digital ocean server ($5/month)
SET: 227816 requests per secondGET: 228258 requests per secondLPUSH: 251098 requests per secondLPOP: 251572 requests per secondMSET (10 keys): 43918 requests per second
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 11: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/11.jpg)
http://redis.io
open source, BSD licensed, advanced key-value store.
It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 12: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/12.jpg)
started in 2009 by Salvatore Sanfilippo @antirez
109 contributors at
https://github.com/antirez/redis
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 13: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/13.jpg)
Redis makes you think
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 14: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/14.jpg)
Redis data typesStrings
Hashes
Lists
Sets
Sorted Sets
HyperLogLogjavier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 15: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/15.jpg)
basic commands
Setting/getting/deleting/expiring
Increment/decrement
Pushing/popping
Checking membership, size...
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 16: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/16.jpg)
much cooler commandsSetting only if exists
Blocking pop
(Blocking) pop from one list, push to another
Get/set string ranges (and bit operations)
Set intersections/diffs (and store)
Pub/subjavier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 17: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/17.jpg)
A chat in 6 lines of code
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
r=Redis.new driver: :hiredisr.subscribe “chatrooms:42” do |on|
on.message do |topic, msg|puts “#{topic}: #{msg}”
endend
r.publish “chatrooms:42”, “Hello Codemotion”
![Page 18: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/18.jpg)
atomicity
single threaded, so no concurrency problems
transactions and lua scripts to run multiple operations atomically
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 19: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/19.jpg)
Scripting with luaYou can use Lua for scripting Redis when you need to atomically execute a sequence of commands in which the output of a command is used as input for another
It reduces the need to use complex lock mechanisms and simplifies dependencies between clients
You can even extend the functionality of Redis by using Lua scripts
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 20: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/20.jpg)
Redis keeps everything in memory all the time
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 21: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/21.jpg)
Persistance: RDB Compact binary formatSaves snapshots every few minutesGood for backups and synchronizing
If Redis crashes, a few minutes worth of data will be lost
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 22: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/22.jpg)
Durability: AOF
Log text formatConfigurable durabilityLarge file, can slow down startup
If Redis crashes, typically one second of data could be lost
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 23: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/23.jpg)
High Availability & scaling out
one master, several read-only slaves
sharding
Twemproxy & redis clusterjavier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 24: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/24.jpg)
what's being used for
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 25: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/25.jpg)
Every time line (800 tweets per user) is on redis
5000 writes per second avg300K reads per second
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 26: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/26.jpg)
user info from gizmoduck(memcached)
user id tweet id metadata
write API (from browser or client app)
rpushx to Redis
tweet info from tweetypie (memcached + mysql)
your twitter timeline
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
fanout (flockDB)one per follower
![Page 27: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/27.jpg)
World Of Warcraft
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
Blizzard is quite secretive about it
Known to have at least 8 servers with 64Gb each to serve user avatars
Known to have seen 1.000.000 users concurrently in the Asian version of the game
![Page 28: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/28.jpg)
pinterest object graph
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
per userRedis SortedSet, with timestamp as the score to
store the users followed explicitlystore the users followed implicitlystore the user’s explicit followersstore the user’s implicit followersstore boards followed explicitly
Redis set to store boards unfollowed explicitly
per boardRedis Hash to store a board’s explicit followersRedis Set to store a board’s explicit unfollowers
![Page 29: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/29.jpg)
stack overflowThree level cache:
local cache (no persistence)sessions, and pending view count updates
site cache hot question id lists, user acceptance rates...
global cache (separate Redis DB)Inboxes, API usage quotas...
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 30: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/30.jpg)
youporn
Most data is found in hashes with ordered sets used to know what data to show.
zInterStore on: videos:filters:released, Videos:filters:orientation:straight,Videos:filters:categories:{category_id}, Videos:ordering:rating
Then perform a zRange to get the pages we want and get the list of video_ids back.
Then start a pipeline and get all the videos from hashes.
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 31: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/31.jpg)
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
snapchatRedis cluster of 216 master and 216 slaves.Over 400MM messages per day.Running on Google App Engine.No need to have an operations team.
![Page 32: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/32.jpg)
githubGit as a service internally.
Redis is used for storing routing info matching user repositories to server names
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 33: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/33.jpg)
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
hipchatRedis for caching.
Information like which users are in which rooms, presence information, who is online...
Redis to balance XMPP, so it doesn’t matter which XMPP server you connect to.
![Page 34: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/34.jpg)
when things go wrong
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 35: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/35.jpg)
the instagram case
moving from redis to cassandra: 75% savings on servers
lesson learnt: know when redis is not appropriate
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 36: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/36.jpg)
the twilio case
credit card hell
lesson learnt: know what you are doing. Don't change config on the fly
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 37: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/37.jpg)
everybody fails, sometimes
Salvatore Sanfilippo lost his own blog*because he forgotto configure persistance
* he recovered it in 30 minutes
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 38: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/38.jpg)
Numbers can be scary
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
Twitter: 2TB RAM for Redis on its own servers
Blizzard: At least 8 servers with 64Gb each
Pinterest: 110 Redis instances
Instagram: 1.2 TB of data
![Page 39: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/39.jpg)
you can go a long waybefore havingscary numbers
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 40: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/40.jpg)
how teowaki is using redis
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 41: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/41.jpg)
Abusing queues
keep track of every activity in the system, even if you don't need them all right now:- every page view- every API request- every time a record is created/updated/deleted
benefits:- highly decoupled system- easier to divide into services- you can add behaviour without changing your app
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 42: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/42.jpg)
intermediate cache* As a very fast lightweight storage for analytics data before sending them to our google bigquery based solution
* As a cache for attributes frequently looked up in join tables(names, nicknames, guids, delegated or included model names...)
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 43: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/43.jpg)
Some of our uses of LuaExpiring attributes inside a Redis hash
Inserting notifications into a list only if there are notpending notifications from the same user for the same scope
Paginating a list by an attribute
Manipulating JSON directly at the Redis layer
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 44: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/44.jpg)
counters
Atomic counters can be safely invoked concurrently from anywhere, so you can implement “like” features, global sequences or usage monitoring systems in highly concurrent applications for free.
You can share your counters with any other internal application and still be sure they won't collide.
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 45: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/45.jpg)
Temporary dataRedis allows us to self expire keys after a time has passed. You can use this mechanism for a simple cache
If you operate on a key with expiration time, you can change its value and still keep the expiration going. By combining a decrementing counter with an expiration time, implementing usage quotas is trivial
Also, you can inspect which keys you have in your server efficiently using SCAN commands
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 46: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/46.jpg)
nginx + lua + redisapache + mruby + redis
Multiple levels of cache by using Redis on the webserver/middleware layer
http://wiki.nginx.org/HttpRedis
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 47: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/47.jpg)
summarizing* Redis is more powerful than it seems
* Very fast, easy to use, simple, good documentation
* In-memory data structures, distributed, shared and persisted
* Good as data store, intermediate data store, cache or queue
* Lots of use cases, both in huge and smaller systems
You should probably use it a lot more
javier ramirez @supercoco9 https://teowaki.com Codemotion 2014
![Page 48: How you can benefit from using Redis - Ramirez](https://reader033.vdocuments.mx/reader033/viewer/2022052619/55514642b4c905f2288b4d20/html5/thumbnails/48.jpg)
Find related links at
https://teowaki.com/teams/javier-community/link-categories/redis
Grazie.
If you enjoyed this talk, please sign up for https://teowaki.com
Javier Ramírez@supercoco9
Codemotion 2014