![Page 1: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/1.jpg)
Functional Programming at Facebook
Chris Piro, Eugene LetuchyCommercial Users of Functional Programming (CUFP)Edinburgh, Scotland4 September 2009
![Page 2: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/2.jpg)
! Facebook and Chat
" Chat architecture
# Erlang strengths
$ Setbacks
% What has worked
Agenda
![Page 3: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/3.jpg)
![Page 4: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/4.jpg)
The Facebook Environment
![Page 5: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/5.jpg)
The Facebook Environment
▪ The web site
▪ More than 250 million active users▪ More than 3.5 billion minutes are spent on Facebook each day
![Page 6: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/6.jpg)
The Facebook Environment
▪ The web site
▪ More than 250 million active users▪ More than 3.5 billion minutes are spent on Facebook each day
▪ The engineering team
▪ Fast iteration: code gets out to production within a week▪ Polyglot programming: interoperability with Thrift▪ Practical: high-leverage tools win
![Page 7: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/7.jpg)
Using FP at Facebook
![Page 8: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/8.jpg)
Using FP at Facebook▪ Erlang
▪ Chat backend (channel servers)▪ Chat Jabber interface (ejabberd) ▪ AIM presence: a JSONP validator
![Page 9: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/9.jpg)
Using FP at Facebook▪ Erlang
▪ Chat backend (channel servers)▪ Chat Jabber interface (ejabberd) ▪ AIM presence: a JSONP validator
▪ Haskell
▪ lex-pass: PHP parse transforms▪ Lambdabot▪ textbook: command line Facebook API client▪ Thrift binding
![Page 10: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/10.jpg)
Thrift
![Page 11: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/11.jpg)
▪ An efficient, cross-language serialization and RPC framework
Thrift
![Page 12: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/12.jpg)
▪ An efficient, cross-language serialization and RPC framework▪ Write interoperable servers and clients
Thrift
![Page 13: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/13.jpg)
▪ An efficient, cross-language serialization and RPC framework▪ Write interoperable servers and clients▪ Includes library and code generator for each language
Thrift
![Page 14: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/14.jpg)
▪ An efficient, cross-language serialization and RPC framework▪ Write interoperable servers and clients▪ Includes library and code generator for each language▪ Servers define interfaces with an IDL
Thrift
struct UserProfile { 1: i32 uid, 2: string name, 3: string blurb}
service UserStorage { void store(1: UserProfile user), UserProfile retrieve(1: i32 uid)}
![Page 15: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/15.jpg)
▪ An efficient, cross-language serialization and RPC framework▪ Write interoperable servers and clients▪ Includes library and code generator for each language▪ Servers define interfaces with an IDL▪ Many supported languages
Thrift
struct UserProfile { 1: i32 uid, 2: string name, 3: string blurb}
service UserStorage { void store(1: UserProfile user), UserProfile retrieve(1: i32 uid)}
C++
C#
Erlang
Haskell
Java
Objective C
OCaml
Perl
PHP
Python
Ruby
Squeakr
HTML
XSD
![Page 16: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/16.jpg)
Facebook Chat
![Page 17: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/17.jpg)
Motivation
![Page 18: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/18.jpg)
MotivationWhy does Facebook need Chat?
▪ Inbox, Wall, Comments are asynchronous, slow
▪ Real-time conversation
▪ Unique advantages:
▪ List of friends for free
▪ Integrated Facebook content
▪ No install required
![Page 19: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/19.jpg)
Timeline
![Page 20: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/20.jpg)
Timeline
▪ Jan 2007: Chat prototyped at Hackathon
![Page 21: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/21.jpg)
Timeline
▪ Jan 2007: Chat prototyped at Hackathon
▪ Fall 2007: Chat becomes a “real” project
▪ 4 engineers, 0.5 designer
![Page 22: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/22.jpg)
Timeline
▪ Jan 2007: Chat prototyped at Hackathon
▪ Fall 2007: Chat becomes a “real” project
▪ 4 engineers, 0.5 designer
▪ Winter 2007-08: Code, code, code (learn Erlang)
![Page 23: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/23.jpg)
Timeline
▪ Jan 2007: Chat prototyped at Hackathon
▪ Fall 2007: Chat becomes a “real” project
▪ 4 engineers, 0.5 designer
▪ Winter 2007-08: Code, code, code (learn Erlang)
▪ Feb 2008: “Dark launch” testing begins
▪ Simulates load on the Erlang servers ... they hold up
![Page 24: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/24.jpg)
Timeline
▪ Jan 2007: Chat prototyped at Hackathon
▪ Fall 2007: Chat becomes a “real” project
▪ 4 engineers, 0.5 designer
▪ Winter 2007-08: Code, code, code (learn Erlang)
▪ Feb 2008: “Dark launch” testing begins
▪ Simulates load on the Erlang servers ... they hold up
▪ Apr 6, 2008: First user message sent:
![Page 25: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/25.jpg)
Timeline
▪ Jan 2007: Chat prototyped at Hackathon
▪ Fall 2007: Chat becomes a “real” project
▪ 4 engineers, 0.5 designer
▪ Winter 2007-08: Code, code, code (learn Erlang)
▪ Feb 2008: “Dark launch” testing begins
▪ Simulates load on the Erlang servers ... they hold up
▪ Apr 6, 2008: First user message sent: “msn chat?”
![Page 26: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/26.jpg)
Timeline
▪ Jan 2007: Chat prototyped at Hackathon
▪ Fall 2007: Chat becomes a “real” project
▪ 4 engineers, 0.5 designer
▪ Winter 2007-08: Code, code, code (learn Erlang)
▪ Feb 2008: “Dark launch” testing begins
▪ Simulates load on the Erlang servers ... they hold up
▪ Apr 6, 2008: First user message sent:
▪ Apr 23, 2008: 100% rollout (Facebook has 70M users at the time)
“msn chat?”
![Page 27: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/27.jpg)
Chat today
![Page 28: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/28.jpg)
Chat today
▪ 1+ billion user messages / day
▪ 10+ million active channels at peak
▪ 1+ GB traffic at peak
▪ 100+ channel machines
![Page 29: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/29.jpg)
Chat today
▪ 1+ billion user messages / day
▪ 10+ million active channels at peak
▪ 1+ GB traffic at peak
▪ 100+ channel machines
▪ Work load has increased 10x while machines not even 3x
![Page 30: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/30.jpg)
Chat Architecture
![Page 31: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/31.jpg)
Channel servers (Erlang)
![Page 32: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/32.jpg)
Channel servers (Erlang)
![Page 33: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/33.jpg)
Channel servers (Erlang)
![Page 34: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/34.jpg)
Channel servers (Erlang)Architectural overview
▪ Web, Jabber tiers authenticate, deliver messages
![Page 35: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/35.jpg)
Channel servers (Erlang)Architectural overview
▪ Web, Jabber tiers authenticate, deliver messages
▪ One message queue per user (channel)
![Page 36: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/36.jpg)
Channel servers (Erlang)Architectural overview
▪ Web, Jabber tiers authenticate, deliver messages
▪ One message queue per user (channel)
▪ Timing, idleness information
![Page 37: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/37.jpg)
Channel servers (Erlang)Architectural overview
▪ Web, Jabber tiers authenticate, deliver messages
▪ One message queue per user (channel)
▪ Timing, idleness information
▪ HTTP long poll to simulate push (Comet)
▪ Server replies when a message is ready
▪ One active request per browser tab
![Page 38: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/38.jpg)
Channel servers (Erlang)Architectural overview
▪ Web, Jabber tiers authenticate, deliver messages
▪ One message queue per user (channel)
▪ Timing, idleness information
▪ HTTP long poll to simulate push (Comet)
▪ Server replies when a message is ready
▪ One active request per browser tab
▪ User ID space partitioned statically (division of labor)
![Page 39: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/39.jpg)
Channel servers (Erlang)Architectural overview
▪ Web, Jabber tiers authenticate, deliver messages
▪ One message queue per user (channel)
▪ Timing, idleness information
▪ HTTP long poll to simulate push (Comet)
▪ Server replies when a message is ready
▪ One active request per browser tab
▪ User ID space partitioned statically (division of labor)
▪ Each partition served by a cluster of machines (availability)
![Page 40: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/40.jpg)
Erlang strengths
![Page 41: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/41.jpg)
Concurrency
![Page 42: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/42.jpg)
Concurrency
▪ Cheap parallelism at massive scale
![Page 43: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/43.jpg)
Concurrency
▪ Cheap parallelism at massive scale
▪ Simplifies modeling concurrent interactions
▪ Chat users are independent and concurrent▪ Mapping onto traditional OS threads is unnatural
![Page 44: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/44.jpg)
Concurrency
▪ Cheap parallelism at massive scale
▪ Simplifies modeling concurrent interactions
▪ Chat users are independent and concurrent▪ Mapping onto traditional OS threads is unnatural
▪ Locality of reference
![Page 45: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/45.jpg)
Concurrency
▪ Cheap parallelism at massive scale
▪ Simplifies modeling concurrent interactions
▪ Chat users are independent and concurrent▪ Mapping onto traditional OS threads is unnatural
▪ Locality of reference
▪ Bonus: carries over to non-Erlang concurrent programming
![Page 46: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/46.jpg)
Distribution
![Page 47: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/47.jpg)
Distribution
▪ Connected network of nodes
![Page 48: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/48.jpg)
Distribution
▪ Connected network of nodes
▪ Remote processes look like local processes
▪ Any node in a channel server cluster can route requests▪ Naive load balancing
![Page 49: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/49.jpg)
Distribution
▪ Connected network of nodes
▪ Remote processes look like local processes
▪ Any node in a channel server cluster can route requests▪ Naive load balancing
▪ Distributed Erlang works out-of-the-box (all nodes are trusted)
![Page 50: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/50.jpg)
Fault Isolation
![Page 51: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/51.jpg)
Fault Isolation
▪ Bugs in the initial versions of Chat:
▪ Process leaks in the Thrift bindings▪ Unintended multicasting of messages▪ Bad return state for presence aggregators
![Page 52: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/52.jpg)
Fault Isolation
▪ Bugs in the initial versions of Chat:
▪ Process leaks in the Thrift bindings▪ Unintended multicasting of messages▪ Bad return state for presence aggregators
▪ (Horrible) bugs don’t kill a mostly functional system:
▪ C/C++ segfault takes down the OS process and your server state▪ Erlang badmatch takes down an Erlang process
▪ ... and notifies linked processes
![Page 53: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/53.jpg)
Error logging (crash reports)
![Page 54: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/54.jpg)
Error logging (crash reports)
▪ Any proc_lib-compliant process generates crash reports
![Page 55: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/55.jpg)
Error logging (crash reports)
▪ Any proc_lib-compliant process generates crash reports
▪ Error reports can be handled out-of-band (not where generated)
![Page 56: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/56.jpg)
Error logging (crash reports)
▪ Any proc_lib-compliant process generates crash reports
▪ Error reports can be handled out-of-band (not where generated)
▪ Stacktraces point the way to bugs (functional languages win big here)
![Page 57: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/57.jpg)
Error logging (crash reports)
▪ Any proc_lib-compliant process generates crash reports
▪ Error reports can be handled out-of-band (not where generated)
▪ Stacktraces point the way to bugs (functional languages win big here)
▪ Writing error_log handlers is simple:
▪ gen_event behavior▪ Allows for massaging of the crash and error messages (binaries!)▪ Thrift client in the error log
![Page 58: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/58.jpg)
Error logging (crash reports)
▪ Any proc_lib-compliant process generates crash reports
▪ Error reports can be handled out-of-band (not where generated)
▪ Stacktraces point the way to bugs (functional languages win big here)
▪ Writing error_log handlers is simple:
▪ gen_event behavior▪ Allows for massaging of the crash and error messages (binaries!)▪ Thrift client in the error log
▪ WARNING: excessive error logging can OOM the Erlang node!
![Page 59: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/59.jpg)
Hot code swapping
![Page 60: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/60.jpg)
Hot code swapping
▪ Restart-free upgrades are awesome (!)
▪ Pushing new functional code for Chat takes ~20 seconds▪ No state is lost
![Page 61: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/61.jpg)
Hot code swapping
▪ Restart-free upgrades are awesome (!)
▪ Pushing new functional code for Chat takes ~20 seconds▪ No state is lost
▪ Test on a running system
![Page 62: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/62.jpg)
Hot code swapping
▪ Restart-free upgrades are awesome (!)
▪ Pushing new functional code for Chat takes ~20 seconds▪ No state is lost
▪ Test on a running system
▪ Provides a safety net ... rolling back bad code is easy
![Page 63: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/63.jpg)
Hot code swapping
▪ Restart-free upgrades are awesome (!)
▪ Pushing new functional code for Chat takes ~20 seconds▪ No state is lost
▪ Test on a running system
▪ Provides a safety net ... rolling back bad code is easy
▪ NOTE: we don’t use the OTP release/upgrade strategies
![Page 64: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/64.jpg)
Monitoring and Error Recovery
![Page 65: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/65.jpg)
Monitoring and Error Recovery
▪ Supervision hierarchies
▪ Organize (and control) processes▪ Systematize restarts and error recovery▪ Extended supervisor with a “directory” type
▪ one_for_one with string -> child pid map
![Page 66: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/66.jpg)
Monitoring and Error Recovery
▪ Supervision hierarchies
▪ Organize (and control) processes▪ Systematize restarts and error recovery▪ Extended supervisor with a “directory” type
▪ one_for_one with string -> child pid map
▪ net_kernel (Distributed Erlang)
▪ sends nodedown, nodeup messages▪ any process can subscribe
![Page 67: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/67.jpg)
Monitoring and Error Recovery
▪ Supervision hierarchies
▪ Organize (and control) processes▪ Systematize restarts and error recovery▪ Extended supervisor with a “directory” type
▪ one_for_one with string -> child pid map
▪ net_kernel (Distributed Erlang)
▪ sends nodedown, nodeup messages▪ any process can subscribe
▪ heart: monitors and restarts the OS process
![Page 68: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/68.jpg)
Hibernation
![Page 69: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/69.jpg)
Hibernation
▪ Drastically shrink memory usage with erlang:hibernate/3
▪ Throws away the call stack, minimizes heap▪ Enters a wait state for new messages▪ “Jumps” into a passed-in function for a received message
![Page 70: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/70.jpg)
Hibernation
▪ Drastically shrink memory usage with erlang:hibernate/3
▪ Throws away the call stack, minimizes heap▪ Enters a wait state for new messages▪ “Jumps” into a passed-in function for a received message
▪ Perfect for a long-running, idling HTTP request handler
![Page 71: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/71.jpg)
Hibernation
▪ Drastically shrink memory usage with erlang:hibernate/3
▪ Throws away the call stack, minimizes heap▪ Enters a wait state for new messages▪ “Jumps” into a passed-in function for a received message
▪ Perfect for a long-running, idling HTTP request handler
▪ But ... not compatible with gen_server:call (and gen_server:reply)
▪ gen_server:call has its own receive() loop▪ hibernate() doesn’t support an explicit timeout▪ gen_hibernate: a few hours and a look at gen.erl
![Page 72: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/72.jpg)
hipe_bifs
![Page 73: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/73.jpg)
hipe_bifs Cheating single assignment
▪ Erlang is opinionated:
▪ Destructive assignment is hard because it should be
![Page 74: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/74.jpg)
hipe_bifs Cheating single assignment
▪ Erlang is opinionated:
▪ Destructive assignment is hard because it should be
▪ hipe_bifs:bytearray_*(): manipulate references to mutable arrays (!)
▪ Necessary for aggregating Chat users’ presence▪ Same in-memory format as presence servers (C++)▪ Don’t tell anyone!
![Page 75: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/75.jpg)
Setbacks
![Page 76: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/76.jpg)
“What’s Erlang?”
![Page 77: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/77.jpg)
“What’s Erlang?”
▪ Lack of Erlang educational resources (at start of 2007)
▪ Few industry-focused English-language resources▪ Few blogs (outside of Yariv’s and Joel Reymont’s)▪ U.S. Erlang community limited in number and visibility
![Page 78: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/78.jpg)
“What’s Erlang?”
▪ Lack of Erlang educational resources (at start of 2007)
▪ Few industry-focused English-language resources▪ Few blogs (outside of Yariv’s and Joel Reymont’s)▪ U.S. Erlang community limited in number and visibility
▪ Engineers are uncomfortable with FP
▪ Universities have very conservative curricula▪ FP : academia, AI :: ‘normal programming’ : industry▪ “If you want to succeed, learn C++ and Java”, not “use the right tool for
the job”▪ Not similar to rest of the codebase, not hiring specifically for FP
![Page 79: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/79.jpg)
Institutional pressures
![Page 80: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/80.jpg)
Institutional pressures
▪ Hard to get others to join the effort
![Page 81: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/81.jpg)
Institutional pressures
▪ Hard to get others to join the effort
▪ Can’t reuse specialized infrastructure
▪ PHP-, C++-centric tools▪ Chat deploy process is a one-off
![Page 82: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/82.jpg)
Institutional pressures
▪ Hard to get others to join the effort
▪ Can’t reuse specialized infrastructure
▪ PHP-, C++-centric tools▪ Chat deploy process is a one-off
▪ Divides department into us vs. them
▪ We’re “the Erlang guys”▪ Sole responsibility for fixing bugs▪ Less time for us to evangelize and innovate elsewhere
![Page 83: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/83.jpg)
Institutional pressures
▪ Hard to get others to join the effort
▪ Can’t reuse specialized infrastructure
▪ PHP-, C++-centric tools▪ Chat deploy process is a one-off
▪ Divides department into us vs. them
▪ We’re “the Erlang guys”▪ Sole responsibility for fixing bugs▪ Less time for us to evangelize and innovate elsewhere
▪ (Seemingly) contrary to “move fast” value
![Page 84: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/84.jpg)
What has worked
![Page 85: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/85.jpg)
What has worked
![Page 86: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/86.jpg)
What has worked
▪ Use FP from the beginning
![Page 87: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/87.jpg)
What has worked
▪ Use FP from the beginning
▪ Outline language strengths, give evidence
![Page 88: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/88.jpg)
What has worked
▪ Use FP from the beginning
▪ Outline language strengths, give evidence
▪ Internal tech talks
![Page 89: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/89.jpg)
What has worked
▪ Use FP from the beginning
▪ Outline language strengths, give evidence
▪ Internal tech talks
▪ ICFP Programming Contest: give the FP people an excuse!
![Page 90: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/90.jpg)
What has worked
▪ Use FP from the beginning
▪ Outline language strengths, give evidence
▪ Internal tech talks
▪ ICFP Programming Contest: give the FP people an excuse!
▪ Language independence with Thrift
![Page 91: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/91.jpg)
What has worked
▪ Use FP from the beginning
▪ Outline language strengths, give evidence
▪ Internal tech talks
▪ ICFP Programming Contest: give the FP people an excuse!
▪ Language independence with Thrift
▪ “The right tool for the job”
![Page 92: Functional Programming at Facebook - CUFPcufp.org/archive/2009/slides/PiroLetuchy.pdf · 2014-05-11 · The Facebook Environment The web site More than 250 million active users More](https://reader036.vdocuments.mx/reader036/viewer/2022070711/5ec9814a840bd01f0a0e2435/html5/thumbnails/92.jpg)
(c) 2009 Facebook, Inc. or its licensors. "Facebook" is a registered trademark of Facebook, Inc.. All rights reserved. 1.0