redis everywhere - php london
TRANSCRIPT
![Page 1: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/1.jpg)
REDIS EVERYWHERE
Ricard Clau (@ricardclau)PHP London - 1st August 2013
![Page 2: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/2.jpg)
HELLO WORLD
•Ricard Clau, born and grown up in Barcelona
• Software engineer at Hailo
• Symfony2 lover and PHP believer
•Open-source contributor, sometimes I give talks
•Twitter @ricardclau / Gmail [email protected]
![Page 3: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/3.jpg)
WHAT IS REDIS?
•REmote DIctionary Server
• Created in 2009 by Salvatore Sanfilipo (@antirez)
•Open source
• Advanced in-memory key-value data-structure server
• Stop thinking SQL, think back to structures!
![Page 4: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/4.jpg)
TO ME REDIS IS LIKE...
![Page 5: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/5.jpg)
AGENDA
•REDIS introduction
•Getting started with PHP / Symfony2: sessions, caching
•Cookbooks: queues, statistics, leaderboards, users online, friends online, locking
•War Stories from a 7.5M DAU project
• Final thoughts
![Page 6: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/6.jpg)
INTRODUCTION TO REDISAnd some demystifications
![Page 7: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/7.jpg)
DATA TYPES
• Strings up to 512 Mb
• Lists of elements sorted by insertion order (max 2^32 - 1)
• Sets unordered collection of elements supporting unions, intersections and differences (max 2^32 - 1)
•Hashes key-value pairs (max 2^32 - 1)
• Sorted sets automatically ordered by score (max 2^32 - 1)
![Page 8: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/8.jpg)
ONLY IN-MEMORY?
• Your data needs to fit in your memory
• It has configurable persistance: RDB snapshots, AOF persistence logs, combine both or no persistance at all
• Think like memcached on steroids with persistance
• http://redis.io/topics/persistence
• “Memory is the new disk, disk is the new tape”
![Page 9: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/9.jpg)
INTERESTING FEATURES
•Master-slave replication
• Pipelining to improve performance
•Transactions (kind of with MULTI ... EXEC)
• Publisher / Subscriber
• Scripting with LUA (~stored procedures)
• Predictable performance (check commands complexity)
![Page 10: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/10.jpg)
SHARDING DATA
• Redis cluster will be available in 3.0 (delayed every year...)
• Project Twemproxy can help sharding memcached / redis
•Client-side partitioning is enough for most cases
•Range partitioning VS hash partitioning
• Presharding can help scaling horizontally
![Page 11: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/11.jpg)
HARDWARE TIPS
• Redis is single-threaded, so a 2 core machine is enough
• If using EC2, maybe m1.large (7.5Gb) is the most suitable, but if it is not enough you might consider any m2 memory optimized type
•CPU is rarely the bottleneck with Redis
• Read carefully the doc to maximize performance!
![Page 12: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/12.jpg)
GETTING STARTED WITH PHPIt is easier than you may think!
![Page 13: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/13.jpg)
PHP CLIENTS
• http://redis.io/clients
• Predis (PHP) vs phpredis (PHP extension)
• Predis is very mature, actively maintained, feature complete, extendable, composer friendly and supports all Redis versions
• Phpredis is faster, but not backwards compatible
•Network latency is the biggest performance killer
![Page 14: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/14.jpg)
REDIS-CLI AND MONITOR
•Redis-cli to connect to a Redis instance
•With Monitor we can watch live activity!
![Page 15: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/15.jpg)
SESSIONS, CACHING
• Super-easy to implement with commercial frameworks
• Commands used: GET <session-id>, SETEX <session-id> <expires> <serialized-data>
•Much better performance than PDO and also persistent!
• Caching and temp-data storage work exactly equal
• Be careful with temp-data storage and memory usage!
![Page 16: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/16.jpg)
SYMFONY2 INTEGRATION
• There must be a bundle for that...
• https://github.com/snc/SncRedisBundle
•Out of the box: Session management, Monolog handler, Swiftmailer spooling, Doctrine caching
• Full integration with Symfony2 profiler
![Page 17: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/17.jpg)
REDIS EVERYWHERE
![Page 18: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/18.jpg)
SOME COOKBOOKSAnd comparisons with other technologies
![Page 19: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/19.jpg)
QUEUE PROCESSING
• Using Lists and LPUSH / RPOP instructions
•We can have different languages accessing data
•We used that to send OpenGraph requests to Facebook (REALLY slow API) with Python daemons
• ~80k messages/minute processed with 1 m1.large EC2
• If intensive, consider using RabbitMQ, ActiveMQ, ZeroMQ...
![Page 20: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/20.jpg)
REAL-TIME ANALYTICS
•We can use hashes to group many stats under one key
•Most of the times we have counters: HINCRBY “stats" <key> <increment>
•HMGET “stats” <key1> <key2> ... to retrieve values and HMSET “stats” “stat1” <value1> “stat2” <value2> to set them
•HGETALL to get the full hash
![Page 21: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/21.jpg)
LEADERBOARDS
• Super-easy with Redis, heavy consuming with other systems
• Each board with a single key, using sorted sets
• ZINCRBY “rankXXX” <increment> <userId>
• Top N ZREVRANGE “rankXXX” 0 N [WITHSCORES]
•We stored several milions of members under 1 key
![Page 22: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/22.jpg)
WHO IS ONLINE? (I)
•We can use SETS to achieve it!
•One set for every minute, SADD <minute> <userId>Time +1m +2m +3m
2
2
7
2
3
1
4 1
1
13
5
SUNION
2
13
5
7
4
![Page 23: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/23.jpg)
WHO IS ONLINE? (II)
•Online users == everyone active in the last N minutes
• SUNION <now> <now-1> <now-2> ...
• SUNIONSTORE “online” <now> <now-1> <now-2> ...
•Number of users: SCARD “online”
•Obtain online users with SMEMBERS “online”
![Page 24: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/24.jpg)
FRIENDS ONLINE?
• If we store user´s friends in a SET with key friends-<user-id>
• Friends online: SINTERSTORE “friends-<userid>-online” “online” “friends-<userid>”
• Imagine how you would do it without Redis!
7
13
5
2
4
15
93
10
7
1
ON
LINE
FRIE
ND
S-12
SINTE
R
FRIENDS-12-ONLINE
![Page 25: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/25.jpg)
DISTRIBUTED LOCKING (I)
• Problem in heavy-write applications: 2 almost concurrent requests trying to update same records
• SELECT FOR UPDATE? Easy to create deadlocks!
timeline
R1 R2
R1read
R2save
R2read
R1save
R1 updatesare lost!!!!!
![Page 26: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/26.jpg)
DISTRIBUTED LOCKING (II)
• Locking can be solved with Redis: SETNX returns true only if key does not exist.
• PseudoCode: try SETNX <lock> <time+expires>, if false check if existing lock has expired and overwrite if so. Otherwise, sleep some milliseconds and try to acquire again. Remember to release lock at the end!
• 8 m1.xlarge instances to lock 7.5M Daily Active Users
• Alternatives: Apache Zookeeper, perhaps better for scaling
![Page 27: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/27.jpg)
WAR STORIEShttp://apps.facebook.com/dragoncity/
(~7.5M DAU Facebook Game)
![Page 28: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/28.jpg)
PROGRESSIVELY
•DragonCity had ~2M DAU when we introduced Redis
• First usages: Queues PHP-Python and Temporary storage
• Introduced Locking with Redis -> End of deadlocks
• Used for temporary contests involving rankings
• Session tracking, cheat detection and many others
• About 7.5M DAU and similar amount of servers
![Page 29: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/29.jpg)
REDIS IS SUPER-FAST
• Your network gets killed much before Redis is down
• Always increase your ulimit values in Redis servers
•We had Gigabit network connectivity between machines but it was not enough to handle the data sent and retrieved
• Redis was just at 20% of CPU
•Do some heavy-load tests before launching!
![Page 30: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/30.jpg)
SURPRISES WITH MEMORY
• Redis documentation: “Sets of binary-safe strings”so, we changed values in sets from being just numbers to number + :<char> to add some info to values
•We expected ~20% memory increase but in was 500%
• https://github.com/antirez/redis/blob/unstable/src/t_set.c#L55
• If value can be represented as long, it is stored more efficiently
• http://alonso-vidales.blogspot.co.uk/2013/06/not-all-redis-values-are-strings.html
![Page 31: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/31.jpg)
FINAL THOUGHTSWhen and when not to consider Redis
![Page 32: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/32.jpg)
REDIS IS PERFECT FOR...
• Intensive read-write data applications
•Temporary stored data
•Data that fits in memory
• Problems that fit Redis built-in data types
• Predictability in performance needed (all commands complexity documented)
![Page 33: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/33.jpg)
BUT IS NOT SUITABLE WHEN..
• Big data sets, archive data
• Relational data (RDBMS are absolutely fine to scale)
•We don´t know how we will access data
•Reporting applications (no where clauses)
• ALWAYS choose the right tool for the job!
![Page 34: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/34.jpg)
SPECIAL THANKS
• Ronny López (@ronnylt) & Alonso Vidales (@alonsovidales)
• All backend and systems engineers at @socialpoint
•Of course, to all of you for coming the 1st of August!
• Check our open-source project Redtrine, PR welcome: https://github.com/redtrine/redtrine
![Page 35: Redis everywhere - PHP London](https://reader033.vdocuments.mx/reader033/viewer/2022042607/554f5aa2b4c905b9508b537d/html5/thumbnails/35.jpg)
QUESTIONS?
•Twitter: @ricardclau
• E-mail: [email protected]
•Github: https://github.com/ricardclau
•Blog about PHP and Symfony2: http://www.ricardclau.com
•Hailo is hiring! If you are interested, talk to us or apply!