how twitter works (arsen kostenko technology stream)
Post on 14-Feb-2017
186 Views
Preview:
TRANSCRIPT
HOW TWITTER WORKS
The Challenge
… and more!
How Twitter Works
Tweets
How Twitter Works
Just 140 characters???
#oscars
#worldcup
Twitter Architecture - How Twitter Works
Tweetsauthor
mention
actions
card
counts
Twitter Architecture - How Twitter Works
Let’s Draw Twitter’s Architecture!
clients routing API logic storage
Twitter Architecture - How Twitter Works
Twitter Architecture: The Old Days (~2010)
clients routing logic storage
MySQLCluster
TweetDeck
Twitter for iOS
Twitter for Android
twitter.com
Twitter API Clients
API
The Monorail
(Ruby)The Monorail
(Ruby)The Monorail
(Ruby)
Twitter Architecture - How Twitter Works
Twitter Architecture: The Old Days (~2010)
Twitter Architecture - How Twitter Works
Twitter Architecture: Microservices
clients routing logic storage
Macaw-SwiftMacaw-Swift
User StoreGizmoduck(User Service)
Web API
TFE (Twitter Front End)
Tweetypie (Tweet Service)
Timeline Service
Social Graph Service
Tweet Store
Macaw-Tweets
Search Service
Direct Messages Service
Macaw-Users
Macaw-Search
Macaw-DM
Graph Store
Tweet Graph Store
Direct Messages Store
TweetDeck
Twitter for iOS
Twitter for Android
twitter.com
Twitter API Clients
Distributed Key-Value Store
API
Macaw-SwiftMacaw-SwiftTweet API
Macaw-SwiftMacaw-SwiftUsers API
Macaw-SwiftMacaw-SwiftSearch API
Macaw-SwiftMacaw-SwiftMessages API
Macaw-SwiftMacaw-SwiftTweet Service
Tweetypie (Tweet Service)
Macaw-SwiftMacaw-SwiftUser Service
Tweetypie (Tweet Service)
Macaw-SwiftMacaw-SwiftTimeline Service
Tweetypie (Tweet Service)
Macaw-SwiftMacaw-SwiftSocial Graph Service
Tweetypie (Tweet Service)
Macaw-SwiftMacaw-SwiftSearch Service
Tweetypie (Tweet Service)
Macaw-SwiftMacaw-SwiftDirect Messages Service
Macaw-UsersMacaw-SwiftMacaw-SwiftTimeline API
Twitter Architecture - How Twitter Works
Twitter Architecture: Microservices
10-100x performance
hundreds of million tweets/day
143,200 peak tweets/sec
Twitter Architecture - How Twitter Works
Twitter Clients
clients routing logic storage
Manhattan (Distributed DB)
Gizmoduck(User Service)
Web API Tweetypie (Tweet Service)
Timeline Service
Social Graph Service
TBird (Tweet Store)
Tweets API
Search Service
Direct Messages Service
Users API
Search API
Messages API
Flock (Graph Store)
TFlock (Tweet Graph Store)
DMBird (Direct Msg Store)
TFE (Twitter Front End)
HTTPS
TweetDeck
Twitter for iOS
Twitter for Android
twitter.com
Twitter API Clients
Manhattan (Distributed DB)
API
Timeline API
Twitter Architecture - How Twitter Works
Twitter Front End (TFE)
clients routing logic storage
Manhattan (Distributed DB)
Gizmoduck(User Service)
TFE (Twitter Front End)
Tweetypie (Tweet Service)
Timeline Service
Social Graph Service
TBird (Tweet Store)
Search Service
Direct Messages Service
Flock (Graph Store)
TFlock (Tweet Graph Store)
DMBird (Direct Msg Store)
HTTPS HTTP
TweetDeck
Twitter for iOS
Twitter for Android
twitter.com
Twitter API Clients
Manhattan (Distributed DB)
API
Web API
Tweets API
Users API
Search API
Messages API
Timeline API
Twitter Architecture - How Twitter Works
Presentation Layer Services
clients routing logic storage
Manhattan (Distributed DB)
Gizmoduck(User Service)
Tweetypie (Tweet Service)
Timeline Service
Social Graph Service
TBird (Tweet Store)
Search Service
Direct Messages Service
Flock (Graph Store)
TFlock (Tweet Graph Store)
DMBird (Direct Msg Store)
HTTP
TweetDeck
Twitter for iOS
Twitter for Android
twitter.com
Twitter API Clients
HTTPS
Web API
Tweets API
Users API
Search API
Messages API
TFE (Twitter Front End)
Thrift
Manhattan (Distributed DB)
API
Timeline API
Twitter Architecture - How Twitter Works
Application Logic Services
clients routing logic storage
Manhattan (Distributed DB)
TBird (Tweet Store)
Flock (Graph Store)
TFlock (Tweet Graph Store)
DMBird (Direct Msg Store)
HTTP
TweetDeck
Twitter for iOS
Twitter for Android
twitter.com
Twitter API Clients
HTTPS
TFE (Twitter Front End)
Thrift
Web API
Tweets API
Users API
Search API
Messages API
Gizmoduck(User Service)
Tweetypie (Tweet Service)
Timeline Service
Social Graph Service
Search Service
Direct Messages Service
Manhattan (Distributed DB)
API
Timeline API
Twitter Architecture - How Twitter Works
Storage Services
clients routing logic storage
HTTP
TweetDeck
Twitter for iOS
Twitter for Android
twitter.com
Twitter API Clients
HTTPS
TFE (Twitter Front End)
Thrift
Gizmoduck(User Service)
Tweetypie (Tweet Service)
Timeline Service
Social Graph Service
Search Service
Direct Messages Service
Manhattan (Distributed DB)
TBird (Tweet Store)
Flock (Graph Store)
TFlock (Tweet Graph Store)
DMBird (Direct Msg Store)
Manhattan (Distributed DB)
API
Web API
Tweets API
Users API
Search API
Messages API
Timeline API
Twitter Architecture - How Twitter Works
Write Path Example: Tweet
clients routing logic storage
HTTP
TweetDeck
Twitter for iOS
Twitter for Android
Twitter API Clients
HTTPS
TFE (Twitter Front End)
Thrift
Manhattan (Distributed DB)
TBird (Tweet Store)
Flock (Graph Store)
TFlock (Tweet Graph Store)
DMBird (Direct Msg Store)
Manhattan (Distributed DB)
twitter.com
API
Gizmoduck(User Service)
Tweetypie (Tweet Service)
Timeline Service
Social Graph Service
Search Service
Direct Messages Service
Web API
Tweets API
Users API
Search API
Messages API
Timeline API
Twitter Architecture - How Twitter Works
Write Path Example: Tweet
clients routing logic storage
HTTP
TweetDeck
Twitter for iOS
Twitter API Clients
HTTPS Thrift
Manhattan (Distributed DB)
TBird (Tweet Store)
Flock (Graph Store)
TFlock (Tweet Graph Store)
DMBird (Direct Msg Store)
Manhattan (Distributed DB)
TFE (Twitter Front End)
API
Twitter for Android
twitter.com
Gizmoduck(User Service)
Tweetypie (Tweet Service)
Timeline Service
Social Graph Service
Search Service
Direct Messages Service
Web API
Tweets API
Users API
Search API
Messages API
Timeline API
Twitter Architecture - How Twitter Works
Write Path Example: Tweet
clients routing logic storage
HTTP
TweetDeck
Twitter for iOS
Twitter API Clients
HTTPS Thrift
Manhattan (Distributed DB)
TBird (Tweet Store)
Flock (Graph Store)
TFlock (Tweet Graph Store)
DMBird (Direct Msg Store)
Manhattan (Distributed DB)
TFE (Twitter Front End)
API
Twitter for Android
twitter.com
Gizmoduck(User Service)
Tweetypie (Tweet Service)
Timeline Service
Social Graph Service
Search Service
Direct Messages Service
Web API
Tweets API
Users API
Search API
Messages API
Timeline API
Twitter Architecture - How Twitter Works
Write Path Example: Tweet
clients routing logic storage
HTTP
TweetDeck
Twitter for iOS
Twitter API Clients
HTTPS Thrift
Direct Messages Service
Manhattan (Distributed DB)
Flock (Graph Store)
TFlock (Tweet Graph Store)
DMBird (Direct Msg Store)
Manhattan (Distributed DB)
TFE (Twitter Front End)
Gizmoduck(User Service)
Tweetypie (Tweet Service)
Timeline Service
Social Graph Service
Search Service
API
Twitter for Android
twitter.com
TBird (Tweet Store)
Web API
Tweets API
Users API
Search API
Messages API
Timeline API
Twitter Architecture - How Twitter Works
Write Path Example: Tweet
clients routing logic storage
HTTP
TweetDeck
Twitter for iOS
Twitter API Clients
HTTPS Thrift
Direct Messages Service
Manhattan (Distributed DB)
TBird (Tweet Store)
Flock (Graph Store)
TFlock (Tweet Graph Store)
DMBird (Direct Msg Store)
Manhattan (Distributed DB)
TFE (Twitter Front End)
Gizmoduck(User Service)
Tweetypie (Tweet Service)
Timeline Service
Social Graph Service
Search Service
API
Twitter for Android
twitter.com
Web API
Tweets API
Users API
Search API
Messages API
Timeline API
Twitter Architecture - How Twitter Works
Building a TimelineConsider user timelines for three different Twitter users
Twitter Architecture - How Twitter Works
Building a TimelineIf you follow these three users...
time
Twitter Architecture - How Twitter Works
Building a Timeline… we can build your Twitter timeline as a list of tweets in reverse chronological order:
time
Although we can also rely on other pieces of information to create tweet timelines that are more relevant for our users.
Twitter Architecture - How Twitter Works
Building a Timeline
clients routing logic storage
TweetDeck
Twitter for iOS
Twitter for Android
Twitter API Clients
twitter.com TFE (Twitter Front End)
Gizmoduck(User Service)
Tweetypie (Tweet Service)
Timeline Service
Social Graph Service
Search Service
Direct Messages Service
Manhattan (Distributed DB)
TBird (Tweet Store)
Flock (Graph Store)
TFlock (Tweet Graph Store)
DMBird (Direct Msg Store)
Manhattan (Distributed DB)
API
Web API
Tweets API
Users API
Search API
Messages API
Timeline API
Twitter Architecture - How Twitter Works
Building a Timeline
logic storage
Tweetypie (Tweet Service)
Timeline Servicec
Manhattan (Distributed DB)
TBird (Tweet Store)
Flock (Graph Store)
TFlock (Tweet Graph Store)
DMBird (Direct Msg Store)
Haplo (Redis cluster)
API
Web API
Tweets API
Users API
Search API
Messages API
Timeline API
FanoutService
TWITTER SERVICE STACK
How Twitter Works
Service StackMost services at Twitter run on the following stack:
Hardware
Twitter Service Stack - How Twitter Works
Distributed Resource Manager
Job Scheduler
Runtime
RPC Framework
Operating System
Network Server
REST Framework
Hardware
Mesos
Aurora
Scala + JVM
Finagle
Linux
TwitterServer
Finatra
Mesos and Aurora
Distributed resource manager
Twitter Service Stack - How Twitter Works
Hardware
OS
Hypervisor
OS
Hardware
OS
Mesos
Job
Job
Job
Job
Job
Job
OS
Job
OS
Job
OS
Job
OS
Job
Job scheduler
Scala, Finagle and Finatra
Twitter Service Stack - How Twitter Works
An object-oriented, functional programming language
Compiles to Java bytecode
Used by most production services at Twitter
An extensible RPC system for high-concurrency clients and
servers
Supports HTTP, Thrift, and other protocols
A framework for fast and testable HTTP services
#oneteamMike Cvet @mikecvet
Moses Nakamura @mnnakamura
Saša Gargenta @sasa
Anna Sulkina @asulkina
Ricardo Cervera-Navarro @ricardoc
John Zeman @zemanJulie Dilger @JulieDil
Nisha Antony @nantony345
Questions?
top related