behind the scenes of a grown-up web application
DESCRIPTION
Behind the scenes of a grown-up web application - YAPC::EU 2014, Sofia, BulgariaTRANSCRIPT
Behind the scenes of a grown-up web application Kerstin Puschke YAPC::EU 2014
Who am I
Contact • http://www.kpuschke.eu
• twitter: @titanoboa42
• https://www.xing.com/profile/Kerstin_Puschke
software engineer at XING Hamburg • xing.com
social network for business professionals
• about 14M users
• debian, perl, javascript, ruby on rails, mysql, redis, riak, . . .
We’re hiring!
http://corporate.xing.com/english/company/careers-at-xing/
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 2
Outline
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 3
• Technology & Architecture • programming languages, databases, storage,...
• applications, APIs & asynchronous communication
• Development Process and Tools • development and test environments
• version control
• continous integration, deployment
How it all began
10 years ago - Perl & MySQL
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 4
The camel has been growing
Today - Perl & MySQL & more
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 5
xing.com
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 6
perl
rails
rails
rails
mobile
...
Backend
• Perl
• Ruby on Rails
• Scala
• Hadoop
• Elastic Search
• Insect / Arachnid
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 7
Frontend
• HTML
• Javascript • jquery
• backbone
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 8
Storage, Databases & Cache
• MySQL
• Redis
• Memcache
• Swift / Open Stack
• Riak
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 9
Perl „core“
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 10
perl „core“
rails
rails
rails
mobile
...
Rails
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 11
perl „core“
rails jobs
rails ...
rails events
mobile
rails ...
rails ~ 20 apps
Communication
• REST API
• RabbitMQ AMQP • plus Beetle
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 12
perl „core“
rails jobs
rails public API
rails events
mobile
rails ...
The camel has been growing...
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 13
Dissolving Profile
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 14
perl „core“
rails
rails (public API)
rails
mobile
rails profile frontend
Shadow calls
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 15
old app
new app user request
worker amqp message
shadow call
Still at work
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 16
Perl App
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 17
• ~600k lines of code, ~300k lines of perl code
Perl App
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 18
• ~30M external requests / day
Perl App
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 19
• ~360M internal REST requests / day, 6k / second
Perl App
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 20
DC1
DC2
2 x 6 application server
2 x 18 API Server
2 x 4 worker server
Taking care
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 21
Office Locations
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 22
Hamburg, Munich, Vienna, Barcelona
~520 people from ~28 countries, ~80 engineers
Provisioning Development VMs
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 23
• Vagrant & Chef
Version Control
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 24
• Github >1800 repositories
Continuous Integration & Deployment
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 25
• Jenkins CI ~30 projects, ~250 build jobs
• weekly to continuous deployment
Monitoring
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 26
• logjam
Monitoring
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 27
• graphite
Monitoring
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 28
• cube
famous last words
die "WTF?"; # should not be possible
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 29
The professional network www.xing.com
Thank you for your attention!
Who am I
Contact • http://www.kpuschke.eu
• twitter: @titanoboa42
• https://www.xing.com/profile/Kerstin_Puschke
software engineer at XING Hamburg • xing.com
social network for business professionals
• about 14M users
• debian, perl, javascript, ruby on rails, mysql, redis, riak, . . .
We’re hiring!
http://corporate.xing.com/english/company/careers-at-xing/
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 31
Credits
Map • The map on slide 21 was created as a screenshot from
http://umap.openstreetmap.fr
• Map background credits Outdoors (OSM) Tiles © Gravitystorm / map data OpenStreetMap
August 14 // Behind the scenes of a grown-up web application // Kerstin Puschke // YAPC::EU 2014 32