q con shanghai2013-[jains krums]-[real-time-delivery-archiecture]

@raffi qcon shanghai 2013

real-time delivery architecture

http://twitpic.com/135xa - There’s a plane in the Hudson. I’m on a ferry going to pick up the people. Crazy.!15 Jan 09 Janis Krums @jkrums

what are the goals?⇢evolve from being solely a web stack

LOGICPRESENTATIONROUTING

T-Bird

T-Flock + Haplo

Flock(s)

Gizmoduck

Monorail

STORAGE & RETRIEVAL

PRESENTATIONROUTING

T-Bird

T-Flock + Haplo

Flock(s)

Gizmoduck

Monorail

STORAGE & RETRIEVAL

ROUTING

T-Bird

T-Flock + Haplo

Flock(s)

Gizmoduck

Monorail

STORAGE & RETRIEVAL

LOGICPRESENTATION

T-Bird

T-Flock + Haplo

Flock(s)

Gizmoduck

Monorail

STORAGE & RETRIEVAL

T-Flock + Haplo

Flock(s)

Gizmoduck

Monorail

STORAGE & RETRIEVAL

T-Bird

Flock(s)

Gizmoduck

Monorail

STORAGE & RETRIEVAL

T-Bird

T-Flock + Haplo

Flock(s)

Monorail

STORAGE & RETRIEVAL

T-Bird

T-Flock + Haplo

Darkwing

Monorail

STORAGE & RETRIEVAL

T-Bird

T-Flock + Haplo

Flock(s)

Darkwing

STORAGE & RETRIEVAL

Monorail

T-Bird

T-Flock + Haplo

Flock(s)

Darkwing

STORAGE & RETRIEVAL

Monorail

T-Bird

T-Flock + Haplo

Flock(s)

Darkwing

what are the goals?⇢evolve from being solely a web stack

⇢ isolate responsibilities and concerns

⇢site speed and reliability

⇢developer innovation speed

Pull Push

Targeted twitter.com home_timeline API

User / Site Streams Mobile Push (SMS, etc.)

Queried Search API Track / Follow Streams

Pull Push

Write API

Fanout

Ingester

HTTP Push

Hadoop

Ingester

Write API

RedisRedis

Ingester

RedisRedisEarlybird

HTTP Push

Mobile Push B

Hadoop

Write API

Fanout

Ingester

Hadoop

Write API

Fanout

RedisRedis

Timeline Service

Ingester

RedisRedisEarlybird

Blender

HTTP Push

Mobile Push B

Hadoop

Write API

Fanout

RedisRedis

Timeline Service

Ingester

RedisRedisEarlybird

Blender

HTTP Push

Mobile Push B

Hadoop

Write API

Fanout

RedisRedis

Timeline Service

Ingester

RedisRedisEarlybird

Blender

HTTP Push

Mobile Push B

Hadoop

Write API

Fanout

Timeline Service

Ingester

RedisRedisEarlybird

Blender

HTTP Push

Mobile Push B

Hadoop

Write API

Fanout

RedisRedis

Timeline Service

Ingester

RedisRedisEarlybird

Blender

HTTP Push

Mobile Push B

Hadoop

Write API

Fanout

RedisRedis

Timeline Service

Ingester

RedisRedisEarlybird

Blender

HTTP Push

Mobile Push B

Hadoop

Write API

Fanout

RedisRedis

Social Graph Service

Timeline Service

Ingester

RedisRedisEarlybird

Blender

HTTP Push

Mobile Push B

Hadoop

Write API

Fanout

RedisRedis

insert⇢keyed off

“recipient”

⇢pipelined 4k “destinations” at a time

⇢replicated @raffi qcon shanghai 2013

Timeline Service

Ingester

RedisRedisEarlybird

Blender

HTTP Push

Mobile Push B

Hadoop

Write API

Fanout

RedisRedis

using redis⇢native list

structure

⇢RPUSHX to only add to cached timelines

Tweet ID BitsUser ID

8 bytes 4 bytes8 bytes

Timeline Service

Ingester

RedisRedisEarlybird

Blender

HTTP Push

Mobile Push B

Hadoop

Write API

Fanout

RedisRedis

structure

Timeline Service

Ingester

RedisRedisEarlybird

Blender

HTTP Push

Mobile Push B

Hadoop

Write API

Fanout

RedisRedis

structure

Tweet ID BitsUser ID Tweet ID

Tweet ID

Timeline Service

Ingester

RedisRedisEarlybird

Blender

HTTP Push

Mobile Push B

Hadoop

Write API

Fanout

RedisRedis

Ingester

RedisRedisEarlybird

Blender

HTTP Push

Mobile Push B

Hadoop

Write API

Fanout

RedisRedis

Timeline Service

Ingester

RedisRedisEarlybird

Blender

HTTP Push

Mobile Push B

Hadoop

Write API

Fanout

RedisRedis

Timeline Service

Ingester

RedisRedisEarlybird

Blender

HTTP Push

Mobile Push B

Hadoop

Write API

Fanout

RedisRedis

Timeline Service

Pull Push

Ingester

RedisRedisEarlybird

Blender

HTTP Push

Mobile Push B

Hadoop

Write API

Fanout

RedisRedis

Timeline Service

RedisRedisEarlybird

Blender

HTTP Push

Mobile Push B

Hadoop

Timeline Service

Ingester

Write API

Fanout

RedisRedis

RedisRedisEarlybird

Blender

HTTP Push

Mobile Push B

Hadoop

Timeline Service

Ingester

Write API

Fanout

RedisRedis

EarlybirdRedis

Earlybird

Blender

HTTP Push

Mobile Push B

Hadoop

Timeline Service

Ingester

Write API

Fanout

RedisRedis

RedisEarlybird

Blender

HTTP Push

Mobile Push B

Hadoop

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

RedisEarlybird

Blender

HTTP Push

Mobile Push B

Hadoop

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Blender

HTTP Push

Mobile Push B

HadoopSe

xRedis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

RedisEarlybird

Blender

HTTP Push

Mobile Push B

HadoopSe

xRedis

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

RedisEarlybird

HTTP Push

Mobile Push B

HadoopSe

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

RedisEarlybird

HTTP Push

Mobile Push B

HadoopSe

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

RedisEarlybird

blender⇢queries one

replica of all indexes

⇢merges & ranks results

HTTP Push

Mobile Push B

HadoopSe

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

RedisEarlybird

HTTP Push

Mobile Push B

HadoopSe

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

RedisEarlybird

HTTP Push

Mobile Push B

HadoopSe

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

RedisEarlybird

HTTP Push

Mobile Push B

HadoopSe

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

RedisEarlybird

Pull Push

HTTP Push

Mobile Push B

HadoopSe

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

RedisEarlybird

HTTP Push

Mobile Push B

Hadoop

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

RedisEarlybird

Mobile Push B

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

RedisEarlybird

Mobile Push B

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

RedisEarlybird

⇢maintains persistent connections with end clients

⇢processes tweet & social graph events

⇢event-based “router”

http push / hosebird

Hosebird

Hosebird Firehose

User Streams

Track / Follow

Write API

Hosebird

Hosebird Firehose

User Streams

Track / Follow

HosebirdWrite API

Hosebird

Hosebird Firehose

User Streams

Track / Follow

HosebirdWrite API

event propagation⇢write API sends all events into hosebird;

sees content creation events, social graph changes, etc.

⇢different queues for public tweets, protected tweets, social events, etc.

Firehose

User Streams

Track / Follow

Hosebird

Write API

Firehose

User Streams

Track / Follow

Hosebird

Write API

event cascading⇢bandwidth management

⇢simultaneous connection management (~1m long lived & open connections to this cluster)

User Streams

Track / Follow

Firehose

Hosebird

Write API

User Streams

Track / Follow

Firehose

Hosebird

Write API

firehose⇢edge machine simply outputs the public

tweet queue

⇢only allow a limited number of firehoses per hosebird box for bandwidth management

User Streams

Firehose

Track / FollowHosebird

Hosebird

Write API

User Streams

Firehose

Track / FollowHosebird

Hosebird

Write API

track / follow⇢simple query based on tweet content

⇢keeps list of terms / users of interest

⇢parses public tweets at the edge, and if term matches a token, or user is of interest, then route

Firehose

Track / Follow

User Streams

Hosebird

Write API

Firehose

Track / Follow

User Streams

Hosebird

Write API

user streams⇢replicate home timeline experience

⇢upon login, obtain “following” list

⇢keep cached following list coherent by seeing social graph updates

⇢route tweet if from a followed user

Mobile Push B

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

RedisEarlybird

Mobile Push B

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

RedisEarlybird

Mobile Push B

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

RedisEarlybird

Pull Push

Mobile Push B

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

RedisEarlybird

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

RedisEarlybirdMobile Push

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Pull Push

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Asynchronous Path

Query Path

Hadoop

Synchronous Path

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Synchronous Path

Query Path

Hadoop

Asynchronous Path

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Asynchronous Path

Synchronous Path

Hadoop

Query Path

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

things we’re trying...

Write API

Ingester Fanout

RedisEarlybird

EarlybirdRedis

RedisRedis

Ingester Fanout

RedisRedis

RedisRedisEarlybird

Earlybird

Write API

Ingester Fanout

RedisRedis

RedisRedisEarlybird

Earlybird

Write API

RedisRedis

RedisRedisEarlybird

Earlybird

Write API

Ingester Fanout

RedisRedis

RedisRedisEarlybird

Earlybird

Write API

Ingester Fanout

RedisRedis

RedisRedisEarlybird

Earlybird

Write API

Ingester Fanout

Write API

Ingester Fanout

RedisEarlybird

EarlybirdRedis

RedisRedis

Write API

Ingester Fanout

RedisEarlybird

EarlybirdRedis

RedisRedis

search index ⇢[‘hello’,‘world’]

fanout index ⇢[@danadanger, ...]

User Intent Query Expansion

“Hello, world” “Hello” AND “world”

@raffi’s home timeline home_timeline:raffi

Write API

Ingester Fanout

RedisEarlybird

EarlybirdRedis

RedisRedis

fan-in⇢O(1) write

⇢O(n) read

fan-out⇢O(n) write

⇢O(1) read

@raffi’s home timelinehome_timeline:raffi

OR user_timeline:taylorswift13

streaming compute⇢continuous computation

⇢driven by the events that come into twitter

⇢generalizing the push mechanism

Hadoop

HTTP Push

Blender

Timeline Service

Ingester

Earlybird

Write API

Fanout

RedisRedis

timeline query statistics⇢>150m active users worldwide

⇢300k qps poll-based timelines @ 1ms p50 / 4ms p99

⇢30k qps search-based timelines

tweet input⇢~340m tweets per day

⇢~4K/sec daily average

⇢~6K/sec daily peak

⇢>10K/sec during large events

followingfollowed by

timeline delivery statistics⇢26b deliveries / day (~18m / min)

⇢3.5 seconds @ p50 to deliver to 1m

⇢~300k deliveries / sec

thanks!

q con shanghai2013-[jains krums]-[real-time-delivery-archiecture]

ra qcon

follow streams

haplo monorail gizmoduckflocks

haplo monorail darkwingflocks

concerns site speed

com135xa theres

janis krums

delivery architecture

Technology

collection eupalinos série archiecture et urbanisme t...

club visits -...

qcon shanghai2013-davidko-如何利用 kanban让 scrum...

collection eupalinos série archiecture et urbanisme t le...

2013/10: q con shanghai2013-davidko-如何利用 kanban让...

for school officials...n jan. 15, 2009, janis krums used his...

indice curso online de revit archiecture autodesk...

software archiecture lecture10

managerial economics- etics and organizational archiecture -...

software archiecture lecture05

your host: alex...

q con shanghai2013-赵永明-ats与cdn实践

q con shanghai2013-黄慧攀-又拍云cdn技术探秘

chapter 1 a first introduction to uppaal - aalborg...

january 2010 cmu profiles in rchitecture · clark county...

oracle supercluster—secure private cloud archiecture ›...

software archiecture lecture07

q con shanghai2013-罗婷-performance methodology

q con shanghai2013-[黄舒泉]-[intel it openstack practice]

software archiecture lecture06