redis: need for speed

of 159/159
REDIS: Need for speed @elena_kolevska Elena Kolevska 1 About Our Company REDIS: NEED FOR SPEED Elena Kolevska

Post on 22-Jan-2018

256 views

Category:

Software

1 download

Embed Size (px)

TRANSCRIPT

  1. 1. REDIS: Need for speed @elena_kolevska Elena Kolevska 1 About Our Company REDIS:NEEDFORSPEED Elena Kolevska
  2. 2. 2
  3. 3. Hello World! A free spirit, traveller, capoeirista, mum, music lover, wannabe entrepreneur, proud geek. Elena Kolevska Software engineer We'regoingtotalkaboutRedis: Things you didn't know How much faster does it get? 2
  4. 4. HOW MUCH FASTER DOES IT GET?
  5. 5. REDIS: Need for speed @elena_kolevska Elena Kolevska 4 How much faster does it get? OUR SCENARIO: TWITTER ANALYZER DASHBOARD 751 TOTAL TWEETS 751Main stream 120sub-stream foo 107sub-stream bar ...Foo Bar Baz 0 30 60 90 120 86 107 120
  6. 6. REDIS: Need for speed @elena_kolevska Elena Kolevska 4 How much faster does it get? OUR SCENARIO: TWITTER ANALYZER DASHBOARD 751 TOTAL TWEETS 751Main stream 120sub-stream foo 107sub-stream bar ...Foo Bar Baz 0 30 60 90 120 86 107 120 Total tweets
  7. 7. REDIS: Need for speed @elena_kolevska Elena Kolevska 4 How much faster does it get? OUR SCENARIO: TWITTER ANALYZER DASHBOARD 751 TOTAL TWEETS 751Main stream 120sub-stream foo 107sub-stream bar ...Foo Bar Baz 0 30 60 90 120 86 107 120 Total tweets Main Stream
  8. 8. REDIS: Need for speed @elena_kolevska Elena Kolevska 4 How much faster does it get? OUR SCENARIO: TWITTER ANALYZER DASHBOARD 751 TOTAL TWEETS 751Main stream 120sub-stream foo 107sub-stream bar ... Substream:keyword 1 Foo Bar Baz 0 30 60 90 120 86 107 120 Total tweets Main Stream
  9. 9. REDIS: Need for speed @elena_kolevska Elena Kolevska 4 How much faster does it get? OUR SCENARIO: TWITTER ANALYZER DASHBOARD 751 TOTAL TWEETS 751Main stream 120sub-stream foo 107sub-stream bar ... Substream:keyword 1 Substream:keyword 2 Foo Bar Baz 0 30 60 90 120 86 107 120 Total tweets Main Stream
  10. 10. REDIS: Need for speed @elena_kolevska Elena Kolevska 4 How much faster does it get? OUR SCENARIO: TWITTER ANALYZER DASHBOARD 751 TOTAL TWEETS 751Main stream 120sub-stream foo 107sub-stream bar ... Substream:keyword 1 Substream:keyword 2 Foo Bar Baz 0 30 60 90 120 86 107 120 Total tweets Scores Main Stream
  11. 11. REDIS: Need for speed @elena_kolevska Elena Kolevska 5 How much faster does it get? THE PROCESS: This is a tweet about the words foo and bar
  12. 12. REDIS: Need for speed @elena_kolevska Elena Kolevska 5 How much faster does it get? THE PROCESS: This is a tweet about the words foo and bar Send tweet to main stream Main stream
  13. 13. REDIS: Need for speed @elena_kolevska Elena Kolevska 5 How much faster does it get? THE PROCESS: This is a tweet about the words foo and bar Send tweet to main stream Main stream Increase counter of total tweets Total tweets counter
  14. 14. REDIS: Need for speed @elena_kolevska Elena Kolevska 5 How much faster does it get? THE PROCESS: This is a tweet about the words foo and bar foo Send tweet to appropriate sub-stream Stream foo Stream barbar Send tweet to main stream Main stream Increase counter of total tweets Total tweets counter
  15. 15. REDIS: Need for speed @elena_kolevska Elena Kolevska 5 How much faster does it get? THE PROCESS: This is a tweet about the words foo and bar foo Send tweet to appropriate sub-stream Stream foo Stream barbar Increase counters for streams foo/bar Counter foo Counter bar Send tweet to main stream Main stream Increase counter of total tweets Total tweets counter
  16. 16. REDIS: Need for speed @elena_kolevska Elena Kolevska 6 How much faster does it get? THE STEPS: Save the complete tweet Get all monitored keywords Check tweet for any keywords Increase counter for total tweets Add tweet to main feed Increase counter for found keywords Add tweet to sub-feed
  17. 17. REDIS: Need for speed @elena_kolevska Elena Kolevska 7 How much faster does it get? Redis is TCP server using client/server model. A request to the server is accomplished in two steps: The client sends a query to the server, and reads from the socket, usually in a blocking way, for the server response The server processes the command and sends the response back to the client Client Server
  18. 18. REDIS: Need for speed @elena_kolevska Elena Kolevska 7 How much faster does it get? Redis is TCP server using client/server model. A request to the server is accomplished in two steps: The client sends a query to the server, and reads from the socket, usually in a blocking way, for the server response The server processes the command and sends the response back to the client Client Server Networking link
  19. 19. REDIS: Need for speed @elena_kolevska Elena Kolevska 7 How much faster does it get? Redis is TCP server using client/server model. A request to the server is accomplished in two steps: The client sends a query to the server, and reads from the socket, usually in a blocking way, for the server response The server processes the command and sends the response back to the client Client Server
  20. 20. REDIS: Need for speed @elena_kolevska Elena Kolevska 7 How much faster does it get? Redis is TCP server using client/server model. A request to the server is accomplished in two steps: The client sends a query to the server, and reads from the socket, usually in a blocking way, for the server response The server processes the command and sends the response back to the client Client Server Request from client to server
  21. 21. REDIS: Need for speed @elena_kolevska Elena Kolevska 7 How much faster does it get? Redis is TCP server using client/server model. A request to the server is accomplished in two steps: The client sends a query to the server, and reads from the socket, usually in a blocking way, for the server response The server processes the command and sends the response back to the client Client Server Request from client to server Response from server to client
  22. 22. REDIS: Need for speed @elena_kolevska Elena Kolevska 8 How much faster does it get? Redis is TCP server using client/server model. A request to the server is accomplished in two steps: The client sends a query to the server, and reads from the socket, usually in a blocking way, for the server response The server processes the command and sends the response back to the client Client Server Request from client to server Response from server to client RTT: Round Trip Time
  23. 23. REDIS: Need for speed @elena_kolevska Elena Kolevska 9 How much faster does it get? THE STEPS: SET request to the server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score
  24. 24. REDIS: Need for speed @elena_kolevska Elena Kolevska 9 How much faster does it get? THE STEPS: SET request to the server 1 Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score
  25. 25. REDIS: Need for speed @elena_kolevska Elena Kolevska 10 How much faster does it get? THE STEPS: Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 1
  26. 26. REDIS: Need for speed @elena_kolevska Elena Kolevska 10 How much faster does it get? THE STEPS: LPUSH Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 1
  27. 27. REDIS: Need for speed @elena_kolevska Elena Kolevska 10 How much faster does it get? THE STEPS: LPUSH Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 2
  28. 28. REDIS: Need for speed @elena_kolevska Elena Kolevska 10 How much faster does it get? THE STEPS: LPUSH Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score + LTRIM 2
  29. 29. REDIS: Need for speed @elena_kolevska Elena Kolevska 10 How much faster does it get? THE STEPS: LPUSH Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score + LTRIM 3
  30. 30. REDIS: Need for speed @elena_kolevska Elena Kolevska 11 How much faster does it get? THE STEPS: Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 3
  31. 31. REDIS: Need for speed @elena_kolevska Elena Kolevska 11 How much faster does it get? THE STEPS: INCR Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 3
  32. 32. REDIS: Need for speed @elena_kolevska Elena Kolevska 11 How much faster does it get? THE STEPS: INCR Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 4
  33. 33. REDIS: Need for speed @elena_kolevska Elena Kolevska 12 How much faster does it get? THE STEPS: Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 4
  34. 34. REDIS: Need for speed @elena_kolevska Elena Kolevska 12 How much faster does it get? THE STEPS: SMEMBERS Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 4
  35. 35. REDIS: Need for speed @elena_kolevska Elena Kolevska 12 How much faster does it get? THE STEPS: SMEMBERS Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 5
  36. 36. REDIS: Need for speed @elena_kolevska Elena Kolevska 13 How much faster does it get? THE STEPS: Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 5
  37. 37. REDIS: Need for speed @elena_kolevska Elena Kolevska 13 How much faster does it get? THE STEPS: Logic on client side Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 5
  38. 38. REDIS: Need for speed @elena_kolevska Elena Kolevska 14 How much faster does it get? THE STEPS: Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 5
  39. 39. REDIS: Need for speed @elena_kolevska Elena Kolevska 14 How much faster does it get? THE STEPS: INCR Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 5
  40. 40. REDIS: Need for speed @elena_kolevska Elena Kolevska 14 How much faster does it get? THE STEPS: INCR Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 6
  41. 41. REDIS: Need for speed @elena_kolevska Elena Kolevska 15 How much faster does it get? THE STEPS: Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 6
  42. 42. REDIS: Need for speed @elena_kolevska Elena Kolevska 15 How much faster does it get? THE STEPS: LPUSH Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 6
  43. 43. REDIS: Need for speed @elena_kolevska Elena Kolevska 15 How much faster does it get? THE STEPS: LPUSH Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 7
  44. 44. REDIS: Need for speed @elena_kolevska Elena Kolevska 15 How much faster does it get? THE STEPS: LPUSH Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 7 + LTRIM
  45. 45. REDIS: Need for speed @elena_kolevska Elena Kolevska 15 How much faster does it get? THE STEPS: LPUSH Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score + LTRIM 8
  46. 46. REDIS: Need for speed @elena_kolevska Elena Kolevska 16 How much faster does it get? THE STEPS: Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 8
  47. 47. REDIS: Need for speed @elena_kolevska Elena Kolevska 16 How much faster does it get? THE STEPS: ZINCRBY Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 8
  48. 48. REDIS: Need for speed @elena_kolevska Elena Kolevska 16 How much faster does it get? THE STEPS: ZINCRBY Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 9
  49. 49. REDIS: Need for speed @elena_kolevska Elena Kolevska 17 How much faster does it get? THE STEPS: Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 9
  50. 50. REDIS: Need for speed @elena_kolevska Elena Kolevska 17 How much faster does it get? THE STEPS: Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 9 } + 4 requests for every found keyword
  51. 51. REDIS: Need for speed @elena_kolevska Elena Kolevska 17 How much faster does it get? THE STEPS: Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score 13? } + 4 requests for every found keyword
  52. 52. REDIS: Need for speed @elena_kolevska Elena Kolevska 17 How much faster does it get? THE STEPS: Number of requests to server Save the tweet Add tweet to main feed Increase counter for total tweets Get all monitored keywords Check tweet for any keywords Increase counter for found keywords Add tweet to sub-feed Update score } + 4 requests for every found keyword 17?
  53. 53. LEVEL1: PIPELINING
  54. 54. REDIS: Need for speed @elena_kolevska Elena Kolevska 19 How much faster does it get? Using the PRedis library for PHP smembers("keywords"); $responses = $client->pipeline(function ($pipe) use ($keywords) { $pipe->set(...); // Save the tweet $pipe->lpush(...); // Save the tweet to main feed $pipe->ltrim(...); // Trim the main feed to X tweets so it doesn't grow forever foreach ($keywords as $keyword) { $pipe->incr(...); // Increase the counter for the keyword $pipe->lpush(...); // Add to the subbed $pipe->ltrim(...); // Trim the stream to X tweets $pipe->zincrby(...); // Update score } });
  55. 55. REDIS: Need for speed @elena_kolevska Elena Kolevska 19 How much faster does it get? Using the PRedis library for PHP smembers("keywords"); $responses = $client->pipeline(function ($pipe) use ($keywords) { $pipe->set(...); // Save the tweet $pipe->lpush(...); // Save the tweet to main feed $pipe->ltrim(...); // Trim the main feed to X tweets so it doesn't grow forever foreach ($keywords as $keyword) { $pipe->incr(...); // Increase the counter for the keyword $pipe->lpush(...); // Add to the subbed $pipe->ltrim(...); // Trim the stream to X tweets $pipe->zincrby(...); // Update score } });
  56. 56. REDIS: Need for speed @elena_kolevska Elena Kolevska 19 How much faster does it get? Using the PRedis library for PHP 1 Number of requests to serversmembers("keywords"); $responses = $client->pipeline(function ($pipe) use ($keywords) { $pipe->set(...); // Save the tweet $pipe->lpush(...); // Save the tweet to main feed $pipe->ltrim(...); // Trim the main feed to X tweets so it doesn't grow forever foreach ($keywords as $keyword) { $pipe->incr(...); // Increase the counter for the keyword $pipe->lpush(...); // Add to the subbed $pipe->ltrim(...); // Trim the stream to X tweets $pipe->zincrby(...); // Update score } });
  57. 57. REDIS: Need for speed @elena_kolevska Elena Kolevska 20 How much faster does it get? Using the PRedis library for PHP 2 Number of requests to serversmembers("keywords"); $responses = $client->pipeline(function ($pipe) use ($keywords) { $pipe->set(...); // Save the tweet $pipe->lpush(...); // Save the tweet to main feed $pipe->ltrim(...); // Trim the main feed to X tweets so it doesn't grow forever foreach ($keywords as $keyword) { $pipe->incr(...); // Increase the counter for the keyword $pipe->lpush(...); // Add to the subbed $pipe->ltrim(...); // Trim the stream to X tweets $pipe->zincrby(...); // Update score } });
  58. 58. LEVEL2: LUA SCRIPTING
  59. 59. REDIS: Need for speed @elena_kolevska Elena Kolevska 22 How much faster does it get? Create a hello.lua le: return "Hello" > redis-cli --eval /path/to/script/hello.lua "Hello"
  60. 60. REDIS: Need for speed @elena_kolevska Elena Kolevska 22 How much faster does it get? Create a hello.lua le: return "Hello" > redis-cli --eval /path/to/script/hello.lua "Hello"
  61. 61. REDIS: Need for speed @elena_kolevska Elena Kolevska 23 How much faster does it get? hello.lua: local tweet = ARGV[1] local key_tweet = KEYS[1] local key_keywords = KEYS[2] local key_total_tweets_count = KEYS[3] local key_scores = KEYS[4] local key_main_feed = KEYS[5] redis.call("SET", key_tweet, tweet) -- Save the tweet redis.call("INCR", key_total_tweets_count) -- Increase the total tweet count redis.call("LPUSH", key_main_feed, tweet) -- Push the tweet to the main feed redis.call("LTRIM", key_main_feed, 0, 100) -- Trim the main feed local keywords = redis.call("SMEMBERS", key_keywords) -- Get the keywords for i, name in ipairs(keywords) do if string.find(tweet, name) then local substream_name = "sub_feed:" .. name redis.call("LPUSH", substream_name, tweet) -- Push the tweet to the sub feed redis.call("LTRIM", substream_name, 0, 100) -- Trim the sub feed redis.call("ZINCRBY", key_scores, 1, name) -- Increment the score for the keyword in the leaderboard end end return "OK"
  62. 62. REDIS: Need for speed @elena_kolevska Elena Kolevska 24 How much faster does it get? > redis-cli --eval hello.lua tweet_123 keywords ... , "A tweet about the words foo and bar" "OK"
  63. 63. REDIS: Need for speed @elena_kolevska Elena Kolevska 24 How much faster does it get? > redis-cli --eval hello.lua tweet_123 keywords ... , "A tweet about the words foo and bar" "OK" 1! Number of requests to server
  64. 64. REDIS: Need for speed @elena_kolevska Elena Kolevska 25 How much faster does it get? SCRIPT Evaluates a script cached on the server side by its SHA1 digest. The command is otherwise identical to EVAL. > redis-cli SCRIPT LOAD "$(cat hello.lua)" "6d52847f03028ab1d4620b60dd6ef4a14c8727d7" > redis-cli evalsha 6d52847f03028ab1d4620b60dd6ef4a14c8727d7 5> tweet_123 keywords ... "A tweet about the words foo and bar" "OK" EVALSHA Load a script into the scripts cache, without executing it. Returns a SHA-1 digest of the script.
  65. 65. REDIS: Need for speed @elena_kolevska Elena Kolevska 26 How much faster does it get? CONS: While your script is running, everything else is on hold Trickier to maintain PROS: Speed Atomicity Logic moves out of your app layer Logic moves out of your app layer
  66. 66. WARP DRIVE: REDIS MODULES
  67. 67. REDIS: Need for speed @elena_kolevska Elena Kolevska 28 How much faster does it get? What are Redis modules? Dynamically loaded libraries Written in C Almost as fast as the Redis core Let you extend Redis commands, create new data structures, access data almost as fast as native Redis commands Add-ons to Redis Coming in version 4.0, currently in Beta (RC)
  68. 68. REDIS: Need for speed @elena_kolevska Elena Kolevska 29 How much faster does it get? Layers of the Modules API: Low-level: Close to native access to core data structures High-level: Client-like access to core and modules' commands
  69. 69. REDIS: Need for speed @elena_kolevska Elena Kolevska 30 How much faster does it get? 0 0.35 0.7 1.05 1.4 Python Lua M:HighLevel M:LowLevel 1.2 1.25 1.05 0.1 Time needed for summing 1 000 000 Sorted Set Scores Statistics from Itamar Haber @ redislabs
  70. 70. REDIS: Need for speed @elena_kolevska Elena Kolevska 31 How much faster does it get?
  71. 71. REDIS: Need for speed @elena_kolevska Elena Kolevska 32 How much faster does it get?
  72. 72. REDIS: Need for speed @elena_kolevska Elena Kolevska 33 How much faster does it get?
  73. 73. REDIS: Need for speed @elena_kolevska Elena Kolevska 34 How much faster does it get? High level API RedisModule_Call(ctx,"INCR","sc",argv[1],"10"); c -- Null terminated C string pointer. b -- C buer, two arguments needed: C string pointer and size_t length. s -- RedisModuleString as received in argv or by other Redis module APIs returning a RedisModuleString object. l -- Long long integer. v -- Array of RedisModuleString objects. ! -- This modier just tells the function to replicate the command to slaves and AOF. It is ignored from the point of view of arguments parsing.
  74. 74. REDIS: Need for speed @elena_kolevska Elena Kolevska 34 How much faster does it get? High level API RedisModule_Call(ctx,"INCR","sc",argv[1],"10"); The context object c -- Null terminated C string pointer. b -- C buer, two arguments needed: C string pointer and size_t length. s -- RedisModuleString as received in argv or by other Redis module APIs returning a RedisModuleString object. l -- Long long integer. v -- Array of RedisModuleString objects. ! -- This modier just tells the function to replicate the command to slaves and AOF. It is ignored from the point of view of arguments parsing.
  75. 75. REDIS: Need for speed @elena_kolevska Elena Kolevska 34 How much faster does it get? High level API RedisModule_Call(ctx,"INCR","sc",argv[1],"10"); Command c -- Null terminated C string pointer. b -- C buer, two arguments needed: C string pointer and size_t length. s -- RedisModuleString as received in argv or by other Redis module APIs returning a RedisModuleString object. l -- Long long integer. v -- Array of RedisModuleString objects. ! -- This modier just tells the function to replicate the command to slaves and AOF. It is ignored from the point of view of arguments parsing.
  76. 76. REDIS: Need for speed @elena_kolevska Elena Kolevska 34 How much faster does it get? High level API RedisModule_Call(ctx,"INCR","sc",argv[1],"10"); Arguments c -- Null terminated C string pointer. b -- C buer, two arguments needed: C string pointer and size_t length. s -- RedisModuleString as received in argv or by other Redis module APIs returning a RedisModuleString object. l -- Long long integer. v -- Array of RedisModuleString objects. ! -- This modier just tells the function to replicate the command to slaves and AOF. It is ignored from the point of view of arguments parsing.
  77. 77. REDIS: Need for speed @elena_kolevska Elena Kolevska 34 How much faster does it get? High level API RedisModule_Call(ctx,"INCR","sc",argv[1],"10"); Format specier c -- Null terminated C string pointer. b -- C buer, two arguments needed: C string pointer and size_t length. s -- RedisModuleString as received in argv or by other Redis module APIs returning a RedisModuleString object. l -- Long long integer. v -- Array of RedisModuleString objects. ! -- This modier just tells the function to replicate the command to slaves and AOF. It is ignored from the point of view of arguments parsing.
  78. 78. REDIS: Need for speed @elena_kolevska Elena Kolevska 35 How much faster does it get? High level API RedisModule_Call(ctx,"INCR","sc",argv[1],"10"); c -- Null terminated C string pointer. b -- C buer, two arguments needed: C string pointer and size_t length. s -- RedisModuleString as received in argv or by other Redis module APIs returning a RedisModuleString object. l -- Long long integer. v -- Array of RedisModuleString objects. ! -- This modier just tells the function to replicate the command to slaves and AOF. It is ignored from the point of view of arguments parsing.
  79. 79. REDIS: Need for speed @elena_kolevska Elena Kolevska 35 How much faster does it get? High level API RedisModule_Call(ctx,"INCR","sc",argv[1],"10"); Format specier c -- Null terminated C string pointer. b -- C buer, two arguments needed: C string pointer and size_t length. s -- RedisModuleString as received in argv or by other Redis module APIs returning a RedisModuleString object. l -- Long long integer. v -- Array of RedisModuleString objects. ! -- This modier just tells the function to replicate the command to slaves and AOF. It is ignored from the point of view of arguments parsing.
  80. 80. REDIS: Need for speed @elena_kolevska Elena Kolevska 36 How much faster does it get? High level API RedisModule_Call(ctx,"INCR","sc",argv[1],"10"); c -- Null terminated C string pointer. b -- C buer, two arguments needed: C string pointer and size_t length. s -- RedisModuleString as received in argv or by other Redis module APIs returning a RedisModuleString object. l -- Long long integer. v -- Array of RedisModuleString objects. ! -- This modier just tells the function to replicate the command to slaves and AOF. It is ignored from the point of view of arguments parsing.
  81. 81. REDIS: Need for speed @elena_kolevska Elena Kolevska 36 How much faster does it get? High level API RedisModule_Call(ctx,"INCR","sc",argv[1],"10"); Format specier c -- Null terminated C string pointer. b -- C buer, two arguments needed: C string pointer and size_t length. s -- RedisModuleString as received in argv or by other Redis module APIs returning a RedisModuleString object. l -- Long long integer. v -- Array of RedisModuleString objects. ! -- This modier just tells the function to replicate the command to slaves and AOF. It is ignored from the point of view of arguments parsing.
  82. 82. REDIS: Need for speed @elena_kolevska Elena Kolevska 37 How much faster does it get? Anatomy of a module #include "redismodule.h" #include int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "tweet_processor", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "tweet.process", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; }
  83. 83. REDIS: Need for speed @elena_kolevska Elena Kolevska 37 How much faster does it get? Anatomy of a module #include "redismodule.h" #include int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "tweet_processor", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "tweet.process", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } Denitions of the API
  84. 84. REDIS: Need for speed @elena_kolevska Elena Kolevska 37 How much faster does it get? Anatomy of a module #include "redismodule.h" #include int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "tweet_processor", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "tweet.process", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } Denitions of the API
  85. 85. REDIS: Need for speed @elena_kolevska Elena Kolevska 37 How much faster does it get? Anatomy of a module #include "redismodule.h" #include int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "tweet_processor", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "tweet.process", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } Denitions of the API Called whenever the module is loaded Must be present in each Redis module
  86. 86. REDIS: Need for speed @elena_kolevska Elena Kolevska 37 How much faster does it get? Anatomy of a module #include "redismodule.h" #include int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "tweet_processor", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "tweet.process", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } Denitions of the API Register the module or error out
  87. 87. REDIS: Need for speed @elena_kolevska Elena Kolevska 37 How much faster does it get? Anatomy of a module #include "redismodule.h" #include int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "tweet_processor", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "tweet.process", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } Denitions of the API Register the command or error out
  88. 88. REDIS: Need for speed @elena_kolevska Elena Kolevska 37 How much faster does it get? Anatomy of a module #include "redismodule.h" #include int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "tweet_processor", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "tweet.process", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } Denitions of the API Command name
  89. 89. REDIS: Need for speed @elena_kolevska Elena Kolevska 37 How much faster does it get? Anatomy of a module #include "redismodule.h" #include int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "tweet_processor", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "tweet.process", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } Denitions of the API Function name
  90. 90. REDIS: Need for speed @elena_kolevska Elena Kolevska 38 How much faster does it get? Anatomy of a module #include "redismodule.h" #include int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "process_tweet", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "process_tweet.rand", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; }
  91. 91. REDIS: Need for speed @elena_kolevska Elena Kolevska 38 How much faster does it get? Anatomy of a module #include "redismodule.h" #include int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "process_tweet", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "process_tweet.rand", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } The logic of the module
  92. 92. REDIS: Need for speed @elena_kolevska Elena Kolevska 39 How much faster does it get? Anatomy of a module #include "redismodule.h" #include int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "process_tweet", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "process_tweet.rand", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // We need EXACTLY 5 arguments if (argc != 5) return RedisModule_WrongArity(ctx); RedisModule_AutoMemory(ctx); RedisModuleCallReply *reply; reply = RedisModule_Call(ctx, "INCR", "s", argv[3]); RedisModule_ReplyWithCallReply(ctx, reply); return REDISMODULE_OK; }
  93. 93. REDIS: Need for speed @elena_kolevska Elena Kolevska 39 How much faster does it get? Anatomy of a module #include "redismodule.h" #include int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "process_tweet", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "process_tweet.rand", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // We need EXACTLY 5 arguments if (argc != 5) return RedisModule_WrongArity(ctx); RedisModule_AutoMemory(ctx); RedisModuleCallReply *reply; reply = RedisModule_Call(ctx, "INCR", "s", argv[3]); RedisModule_ReplyWithCallReply(ctx, reply); return REDISMODULE_OK; } Enable automatic memory management
  94. 94. REDIS: Need for speed @elena_kolevska Elena Kolevska 39 How much faster does it get? Anatomy of a module #include "redismodule.h" #include int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "process_tweet", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "process_tweet.rand", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // We need EXACTLY 5 arguments if (argc != 5) return RedisModule_WrongArity(ctx); RedisModule_AutoMemory(ctx); RedisModuleCallReply *reply; reply = RedisModule_Call(ctx, "INCR", "s", argv[3]); RedisModule_ReplyWithCallReply(ctx, reply); return REDISMODULE_OK; } Call Redis commands
  95. 95. REDIS: Need for speed @elena_kolevska Elena Kolevska 39 How much faster does it get? Anatomy of a module #include "redismodule.h" #include int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // ... } int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // Register the module or error out if (RedisModule_Init(ctx, "process_tweet", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; // Register the command or error out if (RedisModule_CreateCommand(ctx, "process_tweet.rand", ProcessTweet_RedisCommand, "readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { // We need EXACTLY 5 arguments if (argc != 5) return RedisModule_WrongArity(ctx); RedisModule_AutoMemory(ctx); RedisModuleCallReply *reply; reply = RedisModule_Call(ctx, "INCR", "s", argv[3]); RedisModule_ReplyWithCallReply(ctx, reply); return REDISMODULE_OK; } Reply with a call object
  96. 96. REDIS: Need for speed @elena_kolevska Elena Kolevska 40 How much faster does it get? Low level API RedisModule_AutoMemory(ctx); RedisModuleKey *key; key = RedisModule_OpenKey(ctx,argv[1],REDISMODULE_READ|REDISMODULE_WRITE); if (RedisModule_KeyType(key) == REDISMODULE_KEYTYPE_EMPTY) { RedisModule_StringSet(key,argv[2]); } RedisModule_CloseKey(key);
  97. 97. REDIS: Need for speed @elena_kolevska Elena Kolevska 40 How much faster does it get? Low level API RedisModule_AutoMemory(ctx); RedisModuleKey *key; key = RedisModule_OpenKey(ctx,argv[1],REDISMODULE_READ|REDISMODULE_WRITE); if (RedisModule_KeyType(key) == REDISMODULE_KEYTYPE_EMPTY) { RedisModule_StringSet(key,argv[2]); } RedisModule_CloseKey(key); Open a key. Return a key pointer
  98. 98. REDIS: Need for speed @elena_kolevska Elena Kolevska 40 How much faster does it get? Low level API RedisModule_AutoMemory(ctx); RedisModuleKey *key; key = RedisModule_OpenKey(ctx,argv[1],REDISMODULE_READ|REDISMODULE_WRITE); if (RedisModule_KeyType(key) == REDISMODULE_KEYTYPE_EMPTY) { RedisModule_StringSet(key,argv[2]); } RedisModule_CloseKey(key); Checks the type of the key
  99. 99. REDIS: Need for speed @elena_kolevska Elena Kolevska 40 How much faster does it get? Low level API RedisModule_AutoMemory(ctx); RedisModuleKey *key; key = RedisModule_OpenKey(ctx,argv[1],REDISMODULE_READ|REDISMODULE_WRITE); if (RedisModule_KeyType(key) == REDISMODULE_KEYTYPE_EMPTY) { RedisModule_StringSet(key,argv[2]); } RedisModule_CloseKey(key); Set a string value for a key
  100. 100. REDIS: Need for speed @elena_kolevska Elena Kolevska 40 How much faster does it get? Low level API RedisModule_AutoMemory(ctx); RedisModuleKey *key; key = RedisModule_OpenKey(ctx,argv[1],REDISMODULE_READ|REDISMODULE_WRITE); if (RedisModule_KeyType(key) == REDISMODULE_KEYTYPE_EMPTY) { RedisModule_StringSet(key,argv[2]); } RedisModule_CloseKey(key); Close the key
  101. 101. REDIS: Need for speed @elena_kolevska Elena Kolevska 41 How much faster does it get? Compiling $ gcc -fPIC -std=gnu99 -c -o process_tweet process_tweet.c $ ld -o process_tweet.so process_tweet.o -shared -Bsymbolic -lc On Linux: $ gcc -dynamic -fno-common -std=gnu99 -c -o process_tweet.o process_tweet.c $ ld -o process_tweet.so process_tweet.o -bundle -undefined dynamic_lookup -lc On OSX: Loading ./redis-unstable/src/redis-server --loadmodule ./process_tweet.so
  102. 102. REDIS: Need for speed @elena_kolevska Elena Kolevska 42 How much faster does it get? Calling the command 127.0.01:6379> tweet.process tweet_id:42 "This is my tweet about the words foo and bar" ... Our command
  103. 103. THINGS YOU DIDN'T KNOW
  104. 104. THINGS YOU probably DIDN'T KNOW
  105. 105. REDIS: Need for speed @elena_kolevska Elena Kolevska 45 Big O Notation: O(1) commands ZSCORE LPOP, BLPOP, LPUSHX, BRPOPLPUSH EXISTS GET, GETSET, HGET, GETBIT GETRANGE, SETRANGE HLEN, HSTRLEN LLEN PFADD, PFCOUNT SISMEMBER APPEND Things you didn't know
  106. 106. REDIS: Need for speed @elena_kolevska Elena Kolevska 45 Big O Notation: O(1) commands ZSCORE LPOP, BLPOP, LPUSHX, BRPOPLPUSH EXISTS GET, GETSET, HGET, GETBIT GETRANGE, SETRANGE HLEN, HSTRLEN LLEN PFADD, PFCOUNT SISMEMBER APPEND Things you didn't know O(N), but can be considered O(1) for short substrings
  107. 107. REDIS: Need for speed @elena_kolevska Elena Kolevska 45 Big O Notation: O(1) commands ZSCORE LPOP, BLPOP, LPUSHX, BRPOPLPUSH EXISTS GET, GETSET, HGET, GETBIT GETRANGE, SETRANGE HLEN, HSTRLEN LLEN PFADD, PFCOUNT SISMEMBER APPEND Things you didn't know
  108. 108. REDIS: Need for speed @elena_kolevska Elena Kolevska 46 Things you didn't know Expiration and eviction Keys are expired in two ways: Active Passive 1 3 2 Test 20 random keys from the set of keys with an associated expire 10 times per second Delete all the keys found expired If more than 25% of keys were expired, start again from step 1
  109. 109. REDIS: Need for speed @elena_kolevska Elena Kolevska 47 Pub Sub PublisherSubscriber A Subscriber B Things you didn't know
  110. 110. REDIS: Need for speed @elena_kolevska Elena Kolevska 47 Pub Sub 127.0.0.1:6379> subscribe bgwebsummit PublisherSubscriber A Subscriber B Things you didn't know
  111. 111. REDIS: Need for speed @elena_kolevska Elena Kolevska 47 Pub Sub 127.0.0.1:6379> subscribe bgwebsummit Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "bgwebsummit" 3) (integer) 1 PublisherSubscriber A Subscriber B Things you didn't know
  112. 112. REDIS: Need for speed @elena_kolevska Elena Kolevska 47 Pub Sub 127.0.0.1:6379> subscribe bgwebsummit Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "bgwebsummit" 3) (integer) 1 127.0.0.1:6379> subscribe bgwebsummit soa PublisherSubscriber A Subscriber B Things you didn't know
  113. 113. REDIS: Need for speed @elena_kolevska Elena Kolevska 47 Pub Sub 127.0.0.1:6379> subscribe bgwebsummit Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "bgwebsummit" 3) (integer) 1 127.0.0.1:6379> subscribe bgwebsummit soa Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "bgwebsummit" 3) (integer) 1 1) "subscribe" 2) "soa" 3) (integer) 2 PublisherSubscriber A Subscriber B Things you didn't know
  114. 114. REDIS: Need for speed @elena_kolevska Elena Kolevska 47 Pub Sub 127.0.0.1:6379> subscribe bgwebsummit Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "bgwebsummit" 3) (integer) 1 127.0.0.1:6379> subscribe bgwebsummit soa Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "bgwebsummit" 3) (integer) 1 1) "subscribe" 2) "soa" 3) (integer) 2 PublisherSubscriber A Subscriber B 127.0.0.1:6379> pubsub channels Things you didn't know
  115. 115. REDIS: Need for speed @elena_kolevska Elena Kolevska 47 Pub Sub 127.0.0.1:6379> subscribe bgwebsummit Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "bgwebsummit" 3) (integer) 1 127.0.0.1:6379> subscribe bgwebsummit soa Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "bgwebsummit" 3) (integer) 1 1) "subscribe" 2) "soa" 3) (integer) 2 PublisherSubscriber A Subscriber B 127.0.0.1:6379> pubsub channels 1) "soa" 2) "bgwebsummit" Things you didn't know
  116. 116. REDIS: Need for speed @elena_kolevska Elena Kolevska 47 Pub Sub 127.0.0.1:6379> subscribe bgwebsummit Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "bgwebsummit" 3) (integer) 1 127.0.0.1:6379> publish bgwebsummit Hello 127.0.0.1:6379> subscribe bgwebsummit soa Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "bgwebsummit" 3) (integer) 1 1) "subscribe" 2) "soa" 3) (integer) 2 PublisherSubscriber A Subscriber B 127.0.0.1:6379> pubsub channels 1) "soa" 2) "bgwebsummit" Things you didn't know
  117. 117. REDIS: Need for speed @elena_kolevska Elena Kolevska 47 Pub Sub 127.0.0.1:6379> subscribe bgwebsummit Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "bgwebsummit" 3) (integer) 1 127.0.0.1:6379> publish bgwebsummit Hello (integer) 1 127.0.0.1:6379> subscribe bgwebsummit soa Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "bgwebsummit" 3) (integer) 1 1) "subscribe" 2) "soa" 3) (integer) 2 PublisherSubscriber A Subscriber B 127.0.0.1:6379> pubsub channels 1) "soa" 2) "bgwebsummit" Things you didn't know
  118. 118. REDIS: Need for speed @elena_kolevska Elena Kolevska 47 Pub Sub 127.0.0.1:6379> subscribe bgwebsummit Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "bgwebsummit" 3) (integer) 1 127.0.0.1:6379> publish bgwebsummit Hello (integer) 1 1) "message" 2) "bgwebsummit" 3) "Hello" 127.0.0.1:6379> subscribe bgwebsummit soa Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "bgwebsummit" 3) (integer) 1 1) "subscribe" 2) "soa" 3) (integer) 2 PublisherSubscriber A Subscriber B 127.0.0.1:6379> pubsub channels 1) "soa" 2) "bgwebsummit" Things you didn't know
  119. 119. REDIS: Need for speed @elena_kolevska Elena Kolevska 47 Pub Sub 127.0.0.1:6379> subscribe bgwebsummit Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "bgwebsummit" 3) (integer) 1 127.0.0.1:6379> publish bgwebsummit Hello (integer) 1 1) "message" 2) "bgwebsummit" 3) "Hello" 127.0.0.1:6379> subscribe bgwebsummit soa Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "bgwebsummit" 3) (integer) 1 1) "subscribe" 2) "soa" 3) (integer) 2 1) "message" 2) "bgwebsummit" 3) "Hello" PublisherSubscriber A Subscriber B 127.0.0.1:6379> pubsub channels 1) "soa" 2) "bgwebsummit" Things you didn't know
  120. 120. REDIS: Need for speed @elena_kolevska Elena Kolevska 47 Pub Sub 127.0.0.1:6379> subscribe bgwebsummit Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "bgwebsummit" 3) (integer) 1 127.0.0.1:6379> publish bgwebsummit Hello (integer) 1 1) "message" 2) "bgwebsummit" 3) "Hello" 127.0.0.1:6379> subscribe bgwebsummit soa Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "bgwebsummit" 3) (integer) 1 1) "subscribe" 2) "soa" 3) (integer) 2 1) "message" 2) "bgwebsummit" 3) "Hello" PublisherSubscriber A Subscriber B 127.0.0.1:6379> publish soa Zdraveite 127.0.0.1:6379> pubsub channels 1) "soa" 2) "bgwebsummit" Things you didn't know
  121. 121. REDIS: Need for speed @elena_kolevska Elena Kolevska 47 Pub Sub 127.0.0.1:6379> subscribe bgwebsummit Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "bgwebsummit" 3) (integer) 1 127.0.0.1:6379> publish bgwebsummit Hello (integer) 1 1) "message" 2) "bgwebsummit" 3) "Hello" 127.0.0.1:6379> subscribe bgwebsummit soa Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "bgwebsummit" 3) (integer) 1 1) "subscribe" 2) "soa" 3) (integer) 2 1) "message" 2) "bgwebsummit" 3) "Hello" PublisherSubscriber A Subscriber B 127.0.0.1:6379> publish soa Zdraveite 127.0.0.1:6379> pubsub channels 1) "soa" 2) "bgwebsummit" 1) "message" 2) "soa" 3) "Zdraveite" Things you didn't know
  122. 122. REDIS: Need for speed @elena_kolevska Elena Kolevska 48 Things you didn't know Pub Sub PSUBSCRIBE can be used to subscribe to all channels matching a pattern Once a Redis client enters the subscriber mode, it can no longer perform any operation other than subscribe to more channels or unsubscribe from the subscribed ones. An additional connection will be needed in order to publish messages. Publishing on db 10, will be heard by a subscriber on db 1
  123. 123. REDIS: Need for speed @elena_kolevska Elena Kolevska 49 Persistence Persistence options: RDB (Point-in-time snapshots) AOF (Append only le) Combination of AOF and RDB No persistence Things you didn't know
  124. 124. REDIS: Need for speed @elena_kolevska Elena Kolevska 50 Geo API Search a GeoSet for members within radius distance from given coordinates Return the distance in unit (meters by default) between elem1 and elem2 in a GeoSet Things you didn't know Storing and querying named pairs of longitude/latitude coordinates into Redis keys Sorted sets with GeoHashing
  125. 125. REDIS: Need for speed @elena_kolevska Elena Kolevska 51 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/services redis-cli get bar > ~/bar.txt cat /commands.txt| redis-cli Things you didn't know
  126. 126. REDIS: Need for speed @elena_kolevska Elena Kolevska 51 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/services redis-cli get bar > ~/bar.txt > redis-cli --bigkeys cat /commands.txt| redis-cli Things you didn't know
  127. 127. REDIS: Need for speed @elena_kolevska Elena Kolevska 51 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/services redis-cli get bar > ~/bar.txt > redis-cli --bigkeys # Scanning the entire keyspace to nd biggest keys as well as # average sizes per key type. You can use -i 0.1 to sleep 0.1 sec # per 100 SCAN commands (not usually needed). [00.00%] Biggest string found so far 'foo1' with 10285 bytes [00.00%] Biggest zset found so far 'foo2' with 1 members [00.00%] Biggest string found so far 'foo4' with 100095 bytes [00.00%] Biggest set found so far 'foo5' with 1 members [12.66%] Biggest string found so far 'foo5' with 13505361 bytes [25.32%] Biggest list found so far 'foo6' with 10001 items -------- summary ------- Sampled 79 keys in the keyspace! Total key length in bytes is 3243 (avg len 41.05) Biggest string found 'foo5' has 13505361 bytes Biggest list found 'foo6' has 10001 items Biggest set found 'foo5' has 12 members Biggest zset found 'foo7' has 8 members 50 strings with 13946784 bytes (63.29% of keys, avg size 278935.68) 3 lists with 10007 items (03.80% of keys, avg size 3335.67) 3 sets with 3 members (03.80% of keys, avg size 1.00) 0 hashs with 0 elds (00.00% of keys, avg size 0.00) 23 zsets with 80 members (29.11% of keys, avg size 3.48) cat /commands.txt| redis-cli Things you didn't know
  128. 128. REDIS: Need for speed @elena_kolevska Elena Kolevska 52 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/services redis-cli get bar > ~/bar.txt cat /commands.txt| redis-cli Things you didn't know
  129. 129. REDIS: Need for speed @elena_kolevska Elena Kolevska 52 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/services redis-cli get bar > ~/bar.txt > redis-cli monitor cat /commands.txt| redis-cli Things you didn't know
  130. 130. REDIS: Need for speed @elena_kolevska Elena Kolevska 52 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/services redis-cli get bar > ~/bar.txt OK 1491054587.455533 [0 127.0.0.1:35532] "zadd" "foo" "1" "one" 1491054591.768172 [0 127.0.0.1:35532] "zadd" "foo" "2" "two" 1491054605.088297 [0 127.0.0.1:35532] "get" "bar" 1491054609.336546 [0 127.0.0.1:35532] "del" "bar" 1491054618.424296 [0 127.0.0.1:35532] "set" "foobar" "here I am" 1491054669.240038 [0 127.0.0.1:35532] "zcount" "foo" "-inf" "+inf" > redis-cli monitor cat /commands.txt| redis-cli Things you didn't know
  131. 131. REDIS: Need for speed @elena_kolevska Elena Kolevska 53 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/services redis-cli get bar > ~/bar.txt cat /commands.txt| redis-cli Things you didn't know
  132. 132. REDIS: Need for speed @elena_kolevska Elena Kolevska 53 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/services redis-cli get bar > ~/bar.txt > redis-cli --scan --pattern '*foo*' cat /commands.txt| redis-cli Things you didn't know
  133. 133. REDIS: Need for speed @elena_kolevska Elena Kolevska 53 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/services redis-cli get bar > ~/bar.txt key-foo12 key-foo23 foo24 foobarbaz foobarbaz43 foo:bar:12 > redis-cli --scan --pattern '*foo*' cat /commands.txt| redis-cli Things you didn't know
  134. 134. REDIS: Need for speed @elena_kolevska Elena Kolevska 54 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/services redis-cli get bar > ~/bar.txt cat /commands.txt| redis-cli Things you didn't know
  135. 135. REDIS: Need for speed @elena_kolevska Elena Kolevska 54 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/services redis-cli get bar > ~/bar.txt > redis-cli --stat cat /commands.txt| redis-cli Things you didn't know
  136. 136. REDIS: Need for speed @elena_kolevska Elena Kolevska 54 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/services redis-cli get bar > ~/bar.txt ------- data ------ -------------------------------------- load ----------------------------------- - child - keys mem clients blocked requests connections 506 1015.00K 1 0 24 (+0) 7 506 1015.00K 1 0 25 (+1) 7 506 3.40M 51 0 60461 (+60436) 57 506 3.40M 51 0 146425 (+85964) 107 507 3.40M 51 0 233844 (+87419) 157 507 3.40M 51 0 321715 (+87871) 207 508 3.40M 51 0 408642 (+86927) 257 508 3.40M 51 0 497038 (+88396) 257 > redis-cli --stat cat /commands.txt| redis-cli Things you didn't know
  137. 137. REDIS: Need for speed @elena_kolevska Elena Kolevska 55 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/foo.txt redis-cli get bar > ~/bar.txt cat /commands.txt| redis-cli Things you didn't know
  138. 138. REDIS: Need for speed @elena_kolevska Elena Kolevska 55 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/foo.txt redis-cli get bar > ~/bar.txt > cat /etc/foo.txt cat /commands.txt| redis-cli Things you didn't know
  139. 139. REDIS: Need for speed @elena_kolevska Elena Kolevska 55 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/foo.txt redis-cli get bar > ~/bar.txt A long long text In multiple lines > cat /etc/foo.txt cat /commands.txt| redis-cli Things you didn't know
  140. 140. REDIS: Need for speed @elena_kolevska Elena Kolevska 55 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/foo.txt redis-cli get bar > ~/bar.txt A long long text In multiple lines > cat /etc/foo.txt > redis-cli -x set foo < /etc/foo.txt cat /commands.txt| redis-cli Things you didn't know
  141. 141. REDIS: Need for speed @elena_kolevska Elena Kolevska 55 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/foo.txt redis-cli get bar > ~/bar.txt A long long text In multiple lines > cat /etc/foo.txt > redis-cli -x set foo < /etc/foo.txt OK cat /commands.txt| redis-cli Things you didn't know
  142. 142. REDIS: Need for speed @elena_kolevska Elena Kolevska 55 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/foo.txt redis-cli get bar > ~/bar.txt A long long text In multiple lines > cat /etc/foo.txt > redis-cli -x set foo < /etc/foo.txt OK > redis-cli 127.0.0.1:6379> cat /commands.txt| redis-cli Things you didn't know
  143. 143. REDIS: Need for speed @elena_kolevska Elena Kolevska 55 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/foo.txt redis-cli get bar > ~/bar.txt A long long text In multiple lines > cat /etc/foo.txt > redis-cli -x set foo < /etc/foo.txt OK > redis-cli 127.0.0.1:6379> get foo cat /commands.txt| redis-cli Things you didn't know
  144. 144. REDIS: Need for speed @elena_kolevska Elena Kolevska 55 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/foo.txt redis-cli get bar > ~/bar.txt A long long text In multiple lines > cat /etc/foo.txt > redis-cli -x set foo < /etc/foo.txt OK > redis-cli 127.0.0.1:6379> "A long long textnIn multiple linesn" get foo cat /commands.txt| redis-cli Things you didn't know
  145. 145. REDIS: Need for speed @elena_kolevska Elena Kolevska 56 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/foo.txt redis-cli get foo > ~/bar.txt A long long text In multiple lines > cat /etc/foo.txt > redis-cli -x set foo < /etc/foo.txt OK > redis-cli 127.0.0.1:6379> "A long long textnIn multiple linesn" get foo cat /commands.txt| redis-cli Things you didn't know
  146. 146. REDIS: Need for speed @elena_kolevska Elena Kolevska 56 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/foo.txt redis-cli get foo > ~/bar.txt A long long text In multiple lines > cat /etc/foo.txt > redis-cli -x set foo < /etc/foo.txt OK > redis-cli 127.0.0.1:6379> "A long long textnIn multiple linesn" get foo 127.0.0.1:6379> set foo "A long long text" cat /commands.txt| redis-cli Things you didn't know
  147. 147. REDIS: Need for speed @elena_kolevska Elena Kolevska 56 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/foo.txt redis-cli get foo > ~/bar.txt A long long text In multiple lines > cat /etc/foo.txt > redis-cli -x set foo < /etc/foo.txt OK > redis-cli 127.0.0.1:6379> "A long long textnIn multiple linesn" get foo 127.0.0.1:6379> set foo "A long long text" > cat /etc/bar.txt cat /commands.txt| redis-cli Things you didn't know
  148. 148. REDIS: Need for speed @elena_kolevska Elena Kolevska 56 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/foo.txt redis-cli get foo > ~/bar.txt A long long text In multiple lines > cat /etc/foo.txt > redis-cli -x set foo < /etc/foo.txt OK > redis-cli 127.0.0.1:6379> "A long long textnIn multiple linesn" get foo 127.0.0.1:6379> set foo "A long long text" A long long text > cat /etc/bar.txt cat /commands.txt| redis-cli Things you didn't know
  149. 149. REDIS: Need for speed @elena_kolevska Elena Kolevska 57 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/foo.txt redis-cli get foo > ~/bar.txt cat /commands.txt| redis-cli Things you didn't know
  150. 150. REDIS: Need for speed @elena_kolevska Elena Kolevska 57 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/foo.txt redis-cli get foo > ~/bar.txt > cat /commands.txt cat /commands.txt| redis-cli Things you didn't know
  151. 151. REDIS: Need for speed @elena_kolevska Elena Kolevska 57 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/foo.txt redis-cli get foo > ~/bar.txt set foo 100 incr foo append foo xxx get foo > cat /commands.txt cat /commands.txt| redis-cli Things you didn't know
  152. 152. REDIS: Need for speed @elena_kolevska Elena Kolevska 57 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/foo.txt redis-cli get foo > ~/bar.txt set foo 100 incr foo append foo xxx get foo > cat /commands.txt > cat /commands.txt| redis-cli cat /commands.txt| redis-cli Things you didn't know
  153. 153. REDIS: Need for speed @elena_kolevska Elena Kolevska 57 redis-cli redis-cli --bigkeys redis-cli monitor redis-cli --scan --pattern '*foo*' redis-cli --stat redis-cli -x set foo < /etc/foo.txt redis-cli get foo > ~/bar.txt set foo 100 incr foo append foo xxx get foo > cat /commands.txt OK (integer) 101 (integer) 6 "101xxx" > cat /commands.txt| redis-cli cat /commands.txt| redis-cli Things you didn't know
  154. 154. REDIS: Need for speed @elena_kolevska Elena Kolevska 58 Tips and best practices Thankyou! @elena_kolevska