![Page 1: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/1.jpg)
BitLove
Noob to Productionin two months...
Saturday, 28 July, 12
![Page 2: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/2.jpg)
BitLove
jamesgolick.coman blog.
Saturday, 28 July, 12
![Page 3: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/3.jpg)
BitLove
github.com/jamesgolickcode.
Saturday, 28 July, 12
![Page 4: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/4.jpg)
BitLove
twitter.com/jamesgolickshit talk
Saturday, 28 July, 12
![Page 5: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/5.jpg)
BitLove
bitlove.cocompany.
Saturday, 28 July, 12
![Page 6: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/6.jpg)
BitLove
fetlife.comproduct.
Saturday, 28 July, 12
![Page 7: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/7.jpg)
BitLove
fetlife.com
• ~ 375 million pageviews / mo
• ~ 900 web req/s peak
• > 1 million users
• Mandatory, site-wide SSL
• profitable (*gasp*)
Saturday, 28 July, 12
![Page 8: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/8.jpg)
BitLove
Noob to Productionin two months...
Saturday, 28 July, 12
![Page 9: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/9.jpg)
BitLove
Requirements
• IM chat exposed over web UI, similar to Facebook chat.
• Authentication through our main web app.
• Rosters managed by and pulled from the web app.
• Security: only friends on site can chat with each other.
Saturday, 28 July, 12
![Page 10: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/10.jpg)
xmpp
Saturday, 28 July, 12
![Page 11: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/11.jpg)
BitLove
ejabberd
Saturday, 28 July, 12
![Page 12: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/12.jpg)
BitLove
erlanghttp://www.erlang.org/download/getting_started-5.4.pdf
Saturday, 28 July, 12
![Page 13: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/13.jpg)
-module(ejabberd_auth_fetlife).
-export([start/1, set_password/3, check_password/3, check_password/5, try_register/3, dirty_get_registered_users/0, get_vh_registered_users/1, get_vh_registered_users/2, get_vh_registered_users_number/1, get_vh_registered_users_number/2, get_password/2, get_password_s/2, is_user_exists/2, remove_user/2, remove_user/3, plain_password_required/0 ]).
Saturday, 28 July, 12
![Page 14: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/14.jpg)
-module(ejabberd_auth_fetlife).
-export([start/1, set_password/3, check_password/3, check_password/5, try_register/3, dirty_get_registered_users/0, get_vh_registered_users/1, get_vh_registered_users/2, get_vh_registered_users_number/1, get_vh_registered_users_number/2, get_password/2, get_password_s/2, is_user_exists/2, remove_user/2, remove_user/3, plain_password_required/0 ]).
Saturday, 28 July, 12
![Page 15: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/15.jpg)
-module(mod_roster_fetlife).
-export([start/2, stop/1, process_iq/3, process_local_iq/3, get_user_roster/2, get_subscription_lists/3, get_in_pending_subscriptions/3, in_subscription/6, out_subscription/4, set_items/3, remove_user/2, get_jid_info/4, item_to_xml/1, webadmin_page/3, webadmin_user/4, get_versioning_feature/2, roster_versioning_enabled/1, roster_version/2]).
Saturday, 28 July, 12
![Page 16: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/16.jpg)
-module(mod_filter_fetlife).
filter_packet(drop) -> drop;filter_packet(Packet = {#jid{user=FromUser}, #jid{user=FromUser},_}) -> Packet;filter_packet(Packet = {#jid{user=""}, _, _}) -> Packet;filter_packet(Packet = {#jid{user=[]}, _, _}) -> Packet;filter_packet(Packet = {#jid{user=FromUser}, #jid{lserver=LServer,server=Server,user=ToUser}, _}) -> case get_friendship_status(FromUser, ToUser) of true -> Packet; false -> drop end.
Saturday, 28 July, 12
![Page 17: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/17.jpg)
IM Bridge
Saturday, 28 July, 12
![Page 18: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/18.jpg)
BitLove
IM Bridge
• Written in erlang.
• Bridge between web and ejabberd.
• Speaks jsonp to web and XMPP protocol to ejabberd.
• Keeps buddy list / conversation state in memory and relays updates to front end.
Saturday, 28 July, 12
![Page 19: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/19.jpg)
BitLove
IM Bridge
• Multiplexes XMPP session across multiple browser tabs.
• Erlang XMPP client sorta works ish.
• Prototype deployed to ~20k alpha testers.
Saturday, 28 July, 12
![Page 20: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/20.jpg)
BitLove
ejabberd revisited
• 59kLoC of erlang.
• Completely incomprehensible logging.
• ejabberd_c2s.erl has several functions that are more than 2 screenfuls on my Macbook Air.
• Fuck ejabberd.
Saturday, 28 July, 12
![Page 21: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/21.jpg)
BitLove
IM Bridge.erl v2
• What if we wrote our own presence implementation?
• How hard could it possibly be to write a router?
• UI state persistence.
• Conversation history persistence.
Saturday, 28 July, 12
![Page 22: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/22.jpg)
BitLove
IM Bridge.erl v2
• Written, deployed, and fully rolled out to 1.5M users in about a month.
• Maxing out around 50k sessions with 200k open tabs.
• ~10k “packets” per second.
• 20 messages per second.
• Running on one machine.
Saturday, 28 July, 12
![Page 23: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/23.jpg)
BitLove
Hardware Specs
• Dual E5520s.
• 12GB Memory.
• 2 x Micron 100GB gen3 SSDs in RAID1.
• CentOS 5.4.
• Linux 2.6.39.
Saturday, 28 July, 12
![Page 24: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/24.jpg)
BitLove
Optimizations
• Remove JSON calls in favour of speaking directly to the production database.
• Remove local roster cache.
• Gutted the router to increase efficiency.
• ~50 deployments in 3 weeks.
• 0 process restarts.
Saturday, 28 July, 12
![Page 25: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/25.jpg)
BitLove
Major Bug
• Tons of sessions timing out at once.
• Guess: they’re all waiting for some resource.
• Deadlock?
• Chat still appears working despite all this because of supervision.
Saturday, 28 July, 12
![Page 26: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/26.jpg)
BitLove
Major Bug
• Need more data.
• Catch timeout exceptions from key gen_server:call invocation and log stack traces.
• Lager trace files.
Saturday, 28 July, 12
![Page 27: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/27.jpg)
lager:debug([{user_id, User#user.id}], "Something incredibly relevant to your debugging efforts happened!", []).
Saturday, 28 July, 12
![Page 28: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/28.jpg)
lager:debug([{user_id, User#user.id}], "Something incredibly relevant to your debugging efforts happened!", []).
Saturday, 28 July, 12
![Page 29: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/29.jpg)
lager:trace_file("/var/log/myservice/user-1.debug.log", [{user_id, 1}], debug).
Saturday, 28 July, 12
![Page 30: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/30.jpg)
lager:trace_file("/var/log/myservice/user-1.debug.log", [{user_id, 1}], debug).
Saturday, 28 July, 12
![Page 31: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/31.jpg)
lager:trace_file("/var/log/myservice/user-1.debug.log", [{user_id, 1}], debug).
Saturday, 28 July, 12
![Page 32: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/32.jpg)
lager:trace_file("/var/log/myservice/user-1.debug.log", [{user_id, 1}], debug).
Saturday, 28 July, 12
![Page 33: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/33.jpg)
BitLove
Major Bug
• Turns out all the dead sessions were waiting on MySQL queries.
• Open MySQL driver sources and immediately puke all over myself.
• Looks like the MySQL driver’s pool only maxes out at one connection unless you specifically issue mysql:connect/8.
Saturday, 28 July, 12
![Page 34: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/34.jpg)
BitLove
Major Bug
• Fork of MySQL driver that makes it more OTP-ish @ jamesgolick/erlang-mysql-driver.
• Uses poolboy for connection pooling.
• Bug fixed.
Saturday, 28 July, 12
![Page 35: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/35.jpg)
BitLove
Conclusions• Code reloading and supervision actually
work.
• BEAM actually does make operations easier.
• But that doesn’t mean that all erlang software is easy to operate.
• Erlang OSS world is a bit of a mess.
• BitLove is standardizing on Erlang.
Saturday, 28 July, 12
![Page 36: Noob to Production - erlang-factory.com · BitLove erlang Saturday, 28 July, 12](https://reader031.vdocuments.mx/reader031/viewer/2022022613/5b9d23e809d3f2df1f8c4d37/html5/thumbnails/36.jpg)
The End.
Saturday, 28 July, 12