raffi krikorian, twitter timelines at scale
DESCRIPTION
Charla de Raffi Krikorian, el VP de Ingeniería de Twitter, sobre la infraestructura de los Timelines de twitterTRANSCRIPT
timelines at scale
@ra!qcon sf 2012
Pull Push
Targeted twitter.comhome_timeline API
User / Site StreamsMobile Push (SMS, etc.)
Queried Search API Track / Follow Streams
the challenge⇢> 150M world wide active users
⇢> 300K QPS for timelines
⇢naïve timeline “materialization” can be slow
Timeline Service
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Redis
Timeline Service
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Redis
Social Graph Service
Timeline Service
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Redis
Social Graph Service
insert⇢keyed o"
“recipient”
⇢pipelined 4k “destinations” at a time
⇢replicated
Timeline Service
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Redis
using redis⇢native list
structure
Tweet ID BitsUser ID
8 bytes 4 bytes8 bytes
Timeline Service
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Redis
using redis⇢native list
structure
⇢RPUSHX to only add to cached timelines
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID Tweet ID
Tweet ID
Tweet ID
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Timeline Service
Redis
Timeline Service
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Redis
TweetyPieGizmoduck
Pull Push
Targeted twitter.comhome_timeline API
User / Site StreamsMobile Push (SMS, etc.)
Queried Search API Track / Follow Streams
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Timeline Service
Redis
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
HadoopSe
arch
Inde
x
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybird
blender⇢queries one
replica of all indexes
⇢merges & ranks results
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
HadoopSe
arch
Inde
x
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybird
Write API Redis
Redis
Redis
Write API Earlybird
Earlybird
Earlybird
API
Cach
e
Read API Redis
Redis
Redis
Read API Earlybird
Earlybird
EarlybirdAPI
Cach
e⇢O(n) write
⇢O(1) write
⇢O(1) read
⇢O(n) read
the challenge (part #2)⇢fanout can be really slow!
⇢ ...especially for high follower counts
@barackobama23 million followers
31 million followers
@katyperry28 million followers
@justinbieber28 million followers
@ra!0.019 million followers
@ladygaga
there are over400 million tweetsa day
a second4600 tweets
0.2 msa tweet≈
Write API
Ingester Fanout
Sear
ch In
dex
RedisEarlybird
EarlybirdRedis
RedisRedis
Tim
elin
e C
ache
search index ⇢[‘hello’,‘world’]
fanout index ⇢[@danadanger, ...]
User Intent Query Expansion
“Hello, world” “Hello” AND “world”
@ra!’s home timeline home_timeline:ra!
User Intent Query Expansion
“Hello, world” “Hello” AND “world”
@ra!’s home timelineuser_timeline:nelson
ORuser_timeline:danadanger
User Intent Query Expansion
“Hello, world” “Hello” AND “world”
@ra!’s home timeline home_timeline:ra!
User Intent Query Expansion
“Hello, world” “Hello” AND “world”
@ra!’s home timelinehome_timeline:ra!
ORuser_timeline:taylorswift13
Bat
ch C
ompu
te
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
Asynchronous Path
Query Path
Bat
ch C
ompu
te
Hadoop
Synchronous Path
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
Synchronous Path
Query Path
Bat
ch C
ompu
te
Hadoop
Asynchronous Path
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
Asynchronous Path
Synchronous Path
Bat
ch C
ompu
te
Hadoop
Query Path
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
timeline query statistics⇢>150m active users worldwide
⇢>300k qps poll-based timelines @ 1ms p50 / 4ms p99
⇢>30k qps search-based timelines
tweet input⇢~400m tweets per day
⇢~5K/sec daily average
⇢~7K/sec daily peak
⇢>12K/sec during large events
timeline delivery statistics⇢30b deliveries / day (~21m / min)
⇢3.5 seconds @ p50 to deliver to 1m
⇢~300k deliveries / sec
thanks!