q con shanghai2013-[jains krums]-[real-time-delivery-archiecture]

@raffi qcon shanghai 2013

real-time delivery architecture


http://twitpic.com/135xa - There’s a plane in the Hudson. I’m on a ferry going to pick up the people. Crazy.!15 Jan 09 Janis Krums @jkrums


what are the goals?⇢evolve from being solely a web stack


LOGICPRESENTATIONROUTING

T-Bird

T-Flock + Haplo

Flock(s)

Gizmoduck

Monorail

STORAGE & RETRIEVAL


PRESENTATIONROUTING

T-Bird

T-Flock + Haplo

Flock(s)

Gizmoduck

Monorail

STORAGE & RETRIEVAL

LOGIC


ROUTING

T-Bird

T-Flock + Haplo

Flock(s)

Gizmoduck

Monorail

STORAGE & RETRIEVAL

LOGICPRESENTATION


T-Bird

T-Flock + Haplo

Flock(s)

Gizmoduck

Monorail

STORAGE & RETRIEVAL



T-Flock + Haplo

Flock(s)

Gizmoduck

Monorail

STORAGE & RETRIEVAL


T-Bird


Flock(s)

Gizmoduck

Monorail

STORAGE & RETRIEVAL


T-Bird

T-Flock + Haplo


Flock(s)

Monorail

STORAGE & RETRIEVAL


T-Bird

T-Flock + Haplo

Darkwing


Monorail

STORAGE & RETRIEVAL


T-Bird

T-Flock + Haplo

Flock(s)

Darkwing


STORAGE & RETRIEVAL


Monorail

T-Bird

T-Flock + Haplo

Flock(s)

Darkwing


what are the goals?⇢evolve from being solely a web stack

⇢ isolate responsibilities and concerns

⇢site speed and reliability

⇢developer innovation speed


Pull Push

Targeted twitter.com home_timeline API

User / Site Streams Mobile Push (SMS, etc.)

Queried Search API Track / Follow Streams


Write API

Fanout

Tim

elin

e C

ache

Redis

Ingester

Sear

ch C

ache

Redis

Push

Com

pute

HTTP Push

Bat

ch C

ompu

te

Hadoop



Ingester

Write API



Write API



Tim

elin

e C

ache

RedisRedis

Redis

Ingester

Sear

ch C

ache

RedisRedisEarlybird

Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

Hadoop

Write API

Fanout



Ingester

Sear

ch C

ache

Push

Com

pute

Bat

ch C

ompu

te

Hadoop

Write API

Fanout



RedisRedis

Redis

Timeline Service

Ingester

Sear

ch C

ache

RedisRedisEarlybird

Blender

Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

Hadoop

Write API

Fanout

Tim

elin

e C

ache



RedisRedis

Timeline Service

Ingester

Sear

ch C

ache

RedisRedisEarlybird

Blender

Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

Hadoop

Write API

Fanout

Redis

Tim

elin

e C

ache



Timeline Service

Ingester

Sear

ch C

ache

RedisRedisEarlybird

Blender

Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

Hadoop

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

Redis



Timeline Service

Ingester

Sear

ch C

ache

RedisRedisEarlybird

Blender

Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

Hadoop

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

Redis

Social Graph Service



Timeline Service

Ingester

Sear

ch C

ache

RedisRedisEarlybird

Blender

Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

Hadoop

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

Redis


insert⇢keyed off

“recipient”

⇢pipelined 4k “destinations” at a time

⇢replicated @raffi qcon shanghai 2013


Timeline Service

Ingester

Sear

ch C

ache

RedisRedisEarlybird

Blender

Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

Hadoop

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

Redis

using redis⇢native list

structure

⇢RPUSHX to only add to cached timelines

Tweet ID BitsUser ID

8 bytes 4 bytes8 bytes



Timeline Service

Ingester

Sear

ch C

ache

RedisRedisEarlybird

Blender

Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

Hadoop

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

Redis


structure













Timeline Service

Ingester

Sear

ch C

ache

RedisRedisEarlybird

Blender

Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

Hadoop

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

Redis


structure











Tweet ID BitsUser ID Tweet ID

Tweet ID

Tweet ID


Timeline Service

Ingester

Sear

ch C

ache

RedisRedisEarlybird

Blender

Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

Hadoop

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

Redis



Ingester

Sear

ch C

ache

RedisRedisEarlybird

Blender

Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

Hadoop

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

Timeline Service

Redis



Pull Push





Ingester

Sear

ch C

ache

RedisRedisEarlybird

Blender

Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

Hadoop

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

Timeline Service

Redis



Sear

ch C

ache

RedisRedisEarlybird

Blender

Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

Hadoop

Timeline Service

Ingester

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

Redis


EarlybirdRedis

Earlybird

Blender

Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

Hadoop

Timeline Service

Ingester

Sear

ch In

dex

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

Redis


RedisEarlybird

Blender

Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

Hadoop

Timeline Service

Ingester

Sear

ch In

dex

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

Redis


Blender

Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

HadoopSe

arch

Inde

xRedis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

RedisEarlybird


Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

HadoopSe

arch

Inde

x

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

RedisEarlybird


Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

HadoopSe

arch

Inde

x

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

RedisEarlybird

blender⇢queries one

replica of all indexes

⇢merges & ranks results


Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

HadoopSe

arch

Inde

x

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

RedisEarlybird


Pull Push





Push

Com

pute

HTTP Push

Mobile Push B

atch

Com

pute

HadoopSe

arch

Inde

x

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

RedisEarlybird



HTTP Push

Mobile Push B

atch

Com

pute

Hadoop

Push

Com

pute

Sear

ch In

dex

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

RedisEarlybird



Mobile Push B

atch

Com

pute

Hadoop

Push

Com

pute

HTTP Push

Sear

ch In

dex

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

RedisEarlybird



⇢maintains persistent connections with end clients

⇢processes tweet & social graph events

⇢event-based “router”

http push / hosebird


Hosebird

Hosebird

Hosebird Firehose

User Streams

Track / Follow

Write API

Text


Hosebird

Hosebird Firehose

User Streams

Track / Follow

HosebirdWrite API


Hosebird

Hosebird Firehose

User Streams

Track / Follow

HosebirdWrite API

event propagation⇢write API sends all events into hosebird;

sees content creation events, social graph changes, etc.

⇢different queues for public tweets, protected tweets, social events, etc.


Firehose

User Streams

Track / Follow

Hosebird

Hosebird

Hosebird

Write API


Firehose

User Streams

Track / Follow

Hosebird

Hosebird

Hosebird

Write API

event cascading⇢bandwidth management

⇢simultaneous connection management (~1m long lived & open connections to this cluster)


User Streams

Track / Follow

Firehose

Hosebird

Hosebird

Hosebird

Write API


User Streams

Track / Follow

Firehose

Hosebird

Hosebird

Hosebird

Write API

firehose⇢edge machine simply outputs the public

tweet queue

⇢only allow a limited number of firehoses per hosebird box for bandwidth management


User Streams

Firehose

Track / FollowHosebird

Hosebird

Hosebird

Write API


User Streams

Firehose

Track / FollowHosebird

Hosebird

Hosebird

Write API

track / follow⇢simple query based on tweet content

⇢keeps list of terms / users of interest

⇢parses public tweets at the edge, and if term matches a token, or user is of interest, then route


Firehose

Track / Follow

User Streams

Hosebird

Hosebird

Hosebird

Write API


Firehose

Track / Follow

User Streams

Hosebird

Hosebird

Hosebird

Write API

user streams⇢replicate home timeline experience

⇢upon login, obtain “following” list

⇢keep cached following list coherent by seeing social graph updates

⇢route tweet if from a followed user


Mobile Push B

atch

Com

pute

Hadoop

Push

Com

pute

HTTP Push

Sear

ch In

dex

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

RedisEarlybird



Pull Push





Mobile Push B

atch

Com

pute

Hadoop

Push

Com

pute

HTTP Push

Sear

ch In

dex

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

RedisEarlybird



Bat

ch C

ompu

te

Hadoop

Push

Com

pute

HTTP Push

Sear

ch In

dex

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache

RedisEarlybirdMobile Push



Bat

ch C

ompu

te

Hadoop

Push

Com

pute

HTTP Push

Sear

ch In

dex

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache



Pull Push





Bat

ch C

ompu

te

Hadoop

Push

Com

pute

HTTP Push

Sear

ch In

dex

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache




Hadoop

Bat

ch C

ompu

te

Push

Com

pute

HTTP Push

Sear

ch In

dex

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache




Bat

ch C

ompu

te

Hadoop

Push

Com

pute

HTTP Push

Sear

ch In

dex

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache




Asynchronous Path

Query Path

Bat

ch C

ompu

te

Hadoop

Synchronous Path

Push

Com

pute

HTTP Push

Sear

ch In

dex

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache




Synchronous Path

Query Path

Bat

ch C

ompu

te

Hadoop

Asynchronous Path

Push

Com

pute

HTTP Push

Sear

ch In

dex

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache




Asynchronous Path

Synchronous Path

Bat

ch C

ompu

te

Hadoop

Query Path

Push

Com

pute

HTTP Push

Sear

ch In

dex

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache




Bat

ch C

ompu

te

Hadoop

Push

Com

pute

HTTP Push

Sear

ch In

dex

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache




Writ

e Pa

th

Bat

ch C

ompu

te

Hadoop

Push

Com

pute

HTTP Push

Sear

ch In

dex

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache


Rea

d Pa

th



Rea

d Pa

th

Bat

ch C

ompu

te

Hadoop

Push

Com

pute

HTTP Push

Sear

ch In

dex

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache


Writ

e Pa

th



Bat

ch C

ompu

te

Hadoop

Push

Com

pute

HTTP Push

Sear

ch In

dex

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache




things we’re trying...


Write API

Ingester Fanout

Sear

ch In

dex

RedisEarlybird

EarlybirdRedis

RedisRedis



Ingester Fanout

Tim

elin

e C

ache

Sear

ch In

dex

RedisRedis

RedisRedisEarlybird

Earlybird

Write API



Tim

elin

e C

ache

Sear

ch In

dex

RedisRedis

RedisRedisEarlybird

Earlybird

Write API

Ingester Fanout



RedisRedis

RedisRedisEarlybird

Earlybird

Write API

Ingester Fanout

Sear

ch In

dex

Tim

elin

e C

ache



Write API

Ingester Fanout

Sear

ch In

dex

RedisEarlybird

EarlybirdRedis

RedisRedis

Tim

elin

e C

ache



Write API

Ingester Fanout

Sear

ch In

dex

RedisEarlybird

EarlybirdRedis

RedisRedis

Tim

elin

e C

ache

search index ⇢[‘hello’,‘world’]

fanout index ⇢[@danadanger, ...]



User Intent Query Expansion

“Hello, world” “Hello” AND “world”

@raffi’s home timeline home_timeline:raffi


Write API

Ingester Fanout

Sear

ch In

dex

RedisEarlybird

EarlybirdRedis

RedisRedis

Tim

elin

e C

ache

fan-in⇢O(1) write

⇢O(n) read

fan-out⇢O(n) write

⇢O(1) read




@raffi’s home timeline home_timeline:raffi




@raffi’s home timelinehome_timeline:raffi

OR user_timeline:taylorswift13


streaming compute⇢continuous computation

⇢driven by the events that come into twitter

⇢generalizing the push mechanism


Bat

ch C

ompu

te

Hadoop

Push

Com

pute

HTTP Push

Sear

ch In

dex

Blender

Redis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Tim

elin

e C

ache



timeline query statistics⇢>150m active users worldwide

⇢300k qps poll-based timelines @ 1ms p50 / 4ms p99

⇢30k qps search-based timelines


tweet input⇢~340m tweets per day

⇢~4K/sec daily average

⇢~6K/sec daily peak

⇢>10K/sec during large events


followingfollowed by


timeline delivery statistics⇢26b deliveries / day (~18m / min)

⇢3.5 seconds @ p50 to deliver to 1m

⇢~300k deliveries / sec

thanks!

q con shanghai2013-[jains krums]-[real-time-delivery-archiecture]

Technology

ra qcon

follow streams

haplo monorail gizmoduckflocks

haplo monorail darkwingflocks

concerns site speed

com135xa theres

janis krums

delivery architecture