speed up your symfony2 application and build awesome features with redis

SPEED UP YOUR SYMFONY2 APPLICATION AND BUILD

AWESOME FEATURES WITH REDIS

HELLO WORLD

• Ricard Clau, born and grown up in Barcelona

• Server engineer at Another Place Productions

• Symfony2 lover and PHP believer (sometimes…)

• Open-source contributor, sometimes I give talks

• Twitter @ricardclau / Gmail ricard.clau@gmail.com

AGENDA

• REDIS introduction, data types and commands

• Getting started with Redis in PHP / Symfony2

• Options out of the box in SncRedisBundle

• Cookbooks implemented in Real World Applications

• Sharding data and the new Redis Cluster

INTRODUCTION TO REDISAnd some demystifications

WHAT IS REDIS?

• REmote DIctionary Server

• Created in 2009 by Salvatore Sanfilipo (@antirez)

• Open source (https://github.com/antirez/redis)

• Advanced in-memory key-value data-structure server

• Part of the NoSQL movement, think back to structures!

TO ME REDIS IS LIKE...

ONLY IN-MEMORY?

• Your data needs to fit in your memory

• It has configurable persistence: RDB snapshots, AOF persistence logs, combine both or no persistence at all

• Think like memcached on steroids with persistence

• http://redis.io/topics/persistence

• “Memory is the new disk, disk is the new tape”

INTERESTING FEATURES

• Master-slave replication

• Pipelining to improve performance

• Transactions (kind of with MULTI ... EXEC)

• Scripting with LUA (~stored procedures)

• Iterators for Keyspaces and SETS and HASHES (>= 2.8)

!

COMMANDShttp://redis.io/commands

NO QUERIES

• We communicate with Redis via commands

• No indexes, no schemas, just structures under a key

• Very good documentation and sandbox

• All commands have their complexity documented

DATA TYPES

• Strings (binary-safe) up to 512 Mb

• Lists of elements sorted by insertion order (max 2^32 - 1)

• Sets supporting unions, intersections and diffs (max 2^32 - 1)

• Hashes key-value pairs (max 2^32 - 1)

• Sorted sets automatically ordered by score (max 2^32 - 1)

• HyperLogLog to compute cardinality of BIG sets (2^64)

STRINGS COMMANDS

• GET, SET, STRLEN, APPEND

• GETRANGE, SETRANGE

• Bitmap operations: GETBIT, SETBIT, BITCOUNT

• Counter operations: INCR, DECR, INCRBY, DECRBY

• If the value is a number, Redis tries to store it efficiently

LISTS COMMANDS

• Order of insertion matters

• Easy to implement Queues and Stacks

• RPUSH, LPUSH, RPOP, LPOP

• Blocking versions BLPOP, BRPOP, BRPOPLPUSH with timeout

Key S1 S2 S3 SN…

SETS COMMANDS

• Collection of unique unordered elements

• SCARD (Size), SADD, SREM, SISMEMBER, SMEMBERS

• Intersections, Unions and Diffs: SINTER, SUNION, SDIFF, SINTERSTORE, SUNIONSTORE, SDIFFSTORE

• Iterators with SSCAN (Redis >= 2.8)

KeyS1

S5

S3

S2S4

HASHES COMMANDS

• Many key-value pairs under the same key

• HSET, HGET, HGETALL, HLEN, HEXISTS

• Counters HINCR, HINCRBY, HINCRBYFLOAT

• Iterators with HSCAN (Redis >= 2.8)

KeyK1 V1

K2 V2

K3 V3

K4 V4

SORTED SETS COMMANDS

• Collection of unique elements ordered by score

• ZCARD (Size), ZADD, ZREM, ZSCORE

• Ranking: ZRANGE, ZREVRANGE (optional WITHSCORES)

• Iterators with ZSCAN (Redis >= 2.8)

Key S2 - 20S3 - 15

S4 - 1000S1 - 0

HYPERLOGLOG COMMANDS

• Counting unique things in massive data sets using a very small amount of memory

• PFADD, PFCOUNT, PFMERGE

• HyperLogLog has an standard error of 0.81%

• PF prefix in honor of Philippe Flajolet

GETTING STARTEDEasy steps

PHP CLIENTS

• Predis (PHP) vs phpredis (PHP extension)

• Predis is very mature, actively maintained, feature complete, extendable, composer friendly and supports all Redis versions

• Phpredis is faster, but not always backwards compatible

• Network latency is the biggest performance killer

• http://redis.io/clients

EASY STEPS (I)

$ composer require snc/redis-bundle 1.1.*

$ composer require predis/predis 0.8.*

or install the phpredis extension https://github.com/nicolasff/phpredis

EASY STEPS (II)

Add the bundle to the Symfony2 kernel

EASY STEPS (III)

Add some magic to your config.yml

EASY STEPS (IV)

You can now use Redis as a Symfony2 service! !

In your controller extending Symfony\Bundle\FrameworkBundle\Controller\Controller.php

EASY STEPS (V)Or inject it to your services via your services.xml

REDIS-CLI AND MONITOR

• Redis-cli to connect to a Redis instance

• Experiment with http://redis.io/commands

• With Monitor we can watch live activity!

SNCREDISBUNDLESo many options out of the box!

CLIENTS EVERYWHERE

PHP SESSION SUPPORT!

TTL is the session lifetime, Redis expires it for you Session Locking implementation (not optimal)

Added not so many months ago

LOCKING PROBLEM• Problem in heavy-write applications: 2 almost concurrent

requests trying to update same records

• Concurrent AJAX requests for the same user

timeline

R1 R2

R1read

R2save

R2read

R1save

R1 updates are lost!!!!!

DOCTRINE CACHINGImplemented with SET / SETEX / GET

Be careful with expirations!

No expiring Possibly better

use APC

entities data (annotations / xml / yaml)

DQL parsing

RecordSets Setting TTL explicitly

MONOLOG (I)Implemented with LISTS using RPUSH

Be careful with memory!

MONOLOG (II)And add a new monolog handler using the service

Level recommended at least warning

SWIFTMAILER SPOOLImplemented with LISTS using RPUSH

Serialized Swift_Mime_Messages in the list

COOKBOOKSFrom Real World applications

REDIS AS A QUEUE

• Using Lists and LPUSH / RPOP instructions

• We can have different languages accessing data

• I used this to send OpenGraph requests to Facebook (REALLY slow API) with Python daemons

• If you need some advanced message queuing features check RabbitMQ (@cakper last month´s talk) and others

REAL-TIME DASHBOARDS

• We can use hashes to group many stats under one key

• Most of the times we have counters: HINCRBY “stats" <key> <increment>

• HMGET “stats” <key1> <key2> ... to retrieve values and HMSET “stats” “stat1” <value1> “stat2” <value2> to set them

• HGETALL to get the full hash

LEADERBOARDS

• Super-easy with Redis, heavy consuming with other systems

• Each board with a single key, using sorted sets

• ZINCRBY “rankXXX” <increment> <userId>

• Top N ZREVRANGE “rankXXX” 0 N [WITHSCORES]

• You can store several millions of members under 1 key

WHO IS ONLINE? (I)

• We can use SETS to achieve it!

• One set for every minute, SADD <minute> <userId>Time +1m +2m +3m

2

2

7

2

3

1

4 1

1

13

5

SUNION

2

13

5

7

4

WHO IS ONLINE? (II)

• Online users == everyone active in the last N minutes

• SUNION <now> <now-1> <now-2> ...

• SUNIONSTORE “online” <now> <now-1> <now-2> ...

• Number of users: SCARD “online”

• Obtain online users with SMEMBERS “online”

FRIENDS ONLINE?

• If we store user´s friends in a SET with key friends-<user-id>

• Friends online: SINTERSTORE “friends-<userid>-online” “online” “friends-<userid>”

!

!

!

• Imagine how you would do it without Redis!

7

13

5

2

4

15

93

10

7

1

ON

LINE

FRIE

ND

S-12

SINTE

R

FRIENDS-12-ONLINE

SHARDING / CLUSTERINGThis is when it gets tricky…

NEEDS TO FIT IN MEMORY

• Redis Cluster will be available in 3.0 (currently RC1)

• Proxy servers: Twemproxy (also for memcached)

• Client-side partitioning (supported in many clients)

• Sharding strategies (range vs hash partitioning)

• Presharding can help scaling horizontally

CONFIGS WARNING!!

The default behaviour is RandomDistributionStrategy! PHPRedis has a RedisArray class for client-side sharding

but this is not supported yet in the bundle!

RANGE PARTITIONINGMay work in some cases, not a silver bullet

1-99999 100000-199999

200000-299999 300000-399999

id % 4 == 0 id % 4 == 1

id % 4 == 2 id % 4 == 3

Easy to predict scale Load not properly distributed

Load distributed ok Fairly easy to reshard

HASH PARTITIONINGBetter distribution... but still issues If few keys, it also becomes useless

hash(key) % 4 == 0 hash(key) % 4 == 1

hash(key) % 4 == 2 hash(key) % 4 == 3

Distribution also depending on hash algorithm Complicated resharding

HASH CRC16 CRC32 MD5 SHA1

MURMUR3 !

DIST MODULUS KETAMA

PRESHARDINGMaybe overengineered

Many Redis instances in one initial server

!When needed, just split it into more servers and you

don´t need resharding!!! !

Be careful configuring memory limits

REDIS CLUSTER

• Shards the dataset among N nodes

• Has a responsive failover in order to survive certain failures (this was partially covered already with Sentinel)

• Ability to reshard keys internally

• Neither CP nor AP, eventually consistent, not very resistant to network partitions in some scenarios

• Many disagree with some of the decisions and tradeoffs

FINAL THOUGHTSWhen and when not to consider Redis

REDIS IS PERFECT FOR...

• Intensive read-write data applications

• Temporary stored data

• Data that fits in memory

• Problems that fit Redis built-in data types

• Predictability in performance needed

BUT IS NOT SUITABLE WHEN..

• Big data sets, archive data

• Relational data (RDBMS are absolutely fine to scale)

• We don´t know how we will access data

• Reporting applications (no where clauses)

• It gets tricky for distributed applications (replication, clustering)

• ALWAYS choose the right tool for the job!

SPECIAL THANKS

• Ronny López (@ronnylt) & Alonso Vidales (@alonsovidales)

• Salvatore Sanfilippo (@antirez)

• Henrik Westphal (https://github.com/snc)

• Danielle Alessandri (https://github.com/nrk)

• Nicolas Favre-Felix (https://github.com/nicolasff)

• Of course, to all of you for being here today!

QUESTIONS?

• Twitter: @ricardclau

• E-mail: ricard.clau@gmail.com

• Github: https://github.com/ricardclau

• Blog about PHP and Symfony2: http://www.ricardclau.com