symfonycon madrid 2014 - rock solid deployment of symfony apps

Photo: Alvaro Videla

Pablo Godel @pgodel

Rock Solid Deployment of Symfony

Applications

Photo by @old_sound

https://joind.in/12960

Hi, I am Pablo.

Hi, I am Pablo. !

@pgodel

A GUIDING LIGHT FOR STUDENT SUCCESS

Symfony2 + AngularJS + much more!

A GUIDING LIGHT FOR STUDENT SUCCESS

We are Hiring!

A GUIDING LIGHT FOR STUDENT SUCCESS

Deployment

?

Deployment

Software deployment is all of the activities that !make a software system available for use.

http://en.wikipedia.org/wiki/Software_deployment

Deployment

A very important part of the application life-cycle

Deployment

A very important critical part of the application life-cycle

Deployment

It should not be an !after thought

Deployment

It should be predictable

Deployment

The more you do it the better it goes

Deployment in the PaaS era

1. Open an account

2. Create app

3. Push code

4. Voilà!

composer.json !!

Procfile

$ heroku create!

$ git push heroku master!

$ heroku ps:scale web=5!

That’s it! ;-)

Questions?

Slides: http://slideshare.net/pgodel Twitter: @pgodel

But…

Deployment for the rest of us

Deployment Goals

One-click deploys

Deployment Goals

One-click deploys

Deployment Goals

Continuous deploys

Deployment Goals

Continuous deploys

Deployment Goals

Anytime & Anywhere

Deployment Goals

Anytime & Anywhere

Deployment Goals

Anyone

Deployment Goals

Anyone

Deployment Goals

Reliable

Deployment Goals

Reliable

Deployment Goals

Rollbacks

Deployment Goals

Rollbacks

Deployment Goals

No downtime

Deployment Goals

No downtime

Deployment Goals

Reusable

Deployment Goals

Reusable

Deployment Goals

Reusable

Deployment Goals

Scalable

Deployment Goals

Scalable

Deployment Goals

Deployment Facts

Deployment starts with the developer

Deployment Fact #1

Deployment starts with the developer

• Development environment must be as close as possible to productions servers!

Deployment Fact #1

Deployment starts with the developer

• Development environment must be as close as possible to productions servers!

• Setup test/qa/staging servers!

Deployment Fact #1

Deployment starts with the developer

• Development environment must be as close as possible to productions servers!

• Setup test/qa/staging servers!• Use Vagrant to manage VMs!

Deployment Fact #1

Deployment starts with the developer

• Development environment must be as close as possible to productions servers!

• Setup test/qa/staging servers!• Use Vagrant to manage VMs!• Use Containers with Docker!

Deployment Fact #1

Deployment starts with the developer

• Development environment must be as close as possible to productions servers!

• Setup test/qa/staging servers!• Use Vagrant to manage VMs!• Use Containers with Docker!• Automate your development environment setup

(Ansible, Homebrew Cask)

Deployment Fact #1

Success depends on server OS setup

Deployment Fact #2

Success depends on server OS setup

• Automate installation/configuration with Ansible/Puppet/Chef !

Deployment Fact #2

• Automate installation/configuration with Ansible/Puppet/Chef !

• Create OS packages for 3rd party software

Deployment Fact #2

Success depends on server OS setup

• Automate installation/configuration with Ansible/Puppet/Chef !

• Create OS packages for 3rd party software!• Setup your own package (rpm/deb) repositories

Deployment Fact #2

Success depends on server OS setup

Monitoring is uptime

Deployment Fact #3

Monitoring is uptime

• Use tools to know what is going on with your servers (Ganglia, Cacti, Zabbix, etc.)!

Deployment Fact #3

Monitoring is uptime

• Use tools to know what is going on with your servers (Ganglia, Cacti, Zabbix, etc.)!

Deployment Fact #3

Monitoring is uptime

• Use tools to know what is going on with your servers (Ganglia, Cacti, Zabbix, etc.)!

• Add metrics to your app (Graphite, StatsD, New Relic)!

Deployment Fact #3

Monitoring is uptime

• Use tools to know what is going on with your servers (Ganglia, Cacti, Zabbix, etc.)!

• Add metrics to your app (Graphite, StatsD, New Relic)!

• Use your logs wisely (Logstash, Hecka, Kibana)

Deployment Fact #3

Deployment Methodologies

Deployment Methodologies

• VIM-style

Deployment Methodologies

• VIM-style!• FTP uploads

Deployment Methodologies

• VIM-style!• FTP uploads

Deployment Methodologies

• VIM-style!• FTP uploads!• Rsync

Deployment Methodologies

• VIM-style!• FTP uploads!• Rsync!• Source control (GIT, SVN)

Deployment Methodologies

• VIM-style!• FTP uploads!• Rsync!• Source control (GIT, SVN)!• Build tools (Ant, Phing, Jenkins)!

Deployment Methodologies

• VIM-style!• FTP uploads!• Rsync!• Source control (GIT, SVN)!• Build tools (Ant, Phing, Jenkins)!• Package based (RPM, DEB, …) !

Deployment Methodologies

• VIM-style!• FTP uploads!• Rsync!• Source control (GIT, SVN)!• Build tools (Ant, Phing, Jenkins)!• Package based (RPM, DEB, …) !• Specialized tools (Capifony, Deployer, …)!

Deployment Methodologies

• VIM-style!• FTP uploads!• Rsync!• Source control (GIT, SVN)!• Build tools (Ant, Phing, Jenkins)!• Package based (RPM, DEB, …) !• Specialized tools (Capifony, Deployer, …)!• IT automation tools (Ansible, Puppet, Chef, …)

Deployment: Steps overview

Deployment: First time

• Copy files to server(s)!• Set server-side configurations!• Load DB fixtures!• Process and install assets!• Warm up cache!• Enable site

Deployment: Subsequent times

• Copy files to server(s)!• Update server-side configurations!• Update DB (migrations)!• Process and install assets!• Warm up cache!• Enable new version

We can automate!

Deployment Challenges

Challenge: Secure

Deployment Challenges

• use ssh based connections!

Challenge: Secure

Deployment Challenges

• use ssh based connections!• don’t store passwords on source control

Challenge: Secure

Deployment Challenges

• use ssh based connections!• don’t store passwords on source control!• use environment variables for sensitive data

Challenge: Secure

Deployment Challenges

Challenge: Static assets

Deployment Challenges

Challenge: Static assets

• Minimize/combine JS and CSS files!

Deployment Challenges

Challenge: Static assets

• Minimize/combine JS and CSS files!• Enable web server compression!

Deployment Challenges

Challenge: Static assets

• Minimize/combine JS and CSS files!• Enable web server compression!• Add versioning to static assets links (code.js?v=1)!

Deployment Challenges

Challenge: Static assets

• Minimize/combine JS and CSS files!• Enable web server compression!• Add versioning to static assets links (code.js?v=1)!• Run utilities locally or in a staging server, create

artifact, deploy result!

Deployment Challenges

Challenge: Static assets

• Minimize/combine JS and CSS files!• Enable web server compression!• Add versioning to static assets links (code.js?v=1)!• Run utilities locally or in a staging server, create

artifact, deploy result!• Serve static files from subdomain / CDN

Deployment Challenges

Challenge: File permission conflicts

Deployment Challenges

Challenge: File permission conflicts

• Run Apache/PHP with same user!

Deployment Challenges

Challenge: File permission conflicts

• Run Apache/PHP with same user!• Use php-fpm instead of mod_php!

Deployment Challenges

Challenge: File permission conflicts

• Run Apache/PHP with same user!• Use php-fpm instead of mod_php!• Create “deploy” user and add web server to the

group

Deployment Challenges

Challenge: File permission conflicts

• Run Apache/PHP with same user!• Use php-fpm instead of mod_php!• Create “deploy” user and add web server to the

group!• Use setfacl to give write access to multiple users!

Deployment Challenges

Challenge: File permission conflicts

• Run Apache/PHP with same user!• Use php-fpm instead of mod_php!• Create “deploy” user and add web server to the

group!• Use setfacl to give write access to multiple users!• Use external services for writing files and create

readonly installations

Deployment Challenges

Common Pitfalls

• Case sensitive filesystems!

Common Pitfalls

• Case sensitive filesystems!!

src/Acme/ClassName !!= !src/Acme/Classname.php

Common Pitfalls

• Case sensitive filesystems!!

src/Acme/ClassName !!= !src/Acme/Classname.php

Common Pitfalls

• Case sensitive filesystems!• Configuration differences!

Common Pitfalls

• Case sensitive filesystems!• Configuration differences!• Outdated 3rd party software

Common Pitfalls

• Case sensitive filesystems!• Configuration differences!• Outdated 3rd party software!• Github down

Common Pitfalls

• Case sensitive filesystems!• Configuration differences!• Outdated 3rd party software!• Github down

$ git daemon --base-path=/git/repo/path/ --export-all!!

$ git clone git://127.0.0.1/repo!

http://ozmm.org/posts/when_github_goes_down.html

Common Pitfalls

• Case sensitive filesystems!• Configuration differences!• Outdated 3rd party software!• Github down

Common Pitfalls

Deployment Examples

Simplest continuous deployment ever!

<?php !!exec(‘/usr/bin/env -i HOME=/var/www git pull’);!echo “All done!”;

hook.php

Deployment Examples

<?php !!exec(‘/usr/bin/env -i HOME=/var/www git pull’);!echo “All done!”;

hook.php

screenshot

Deployment Examples

GitHub hook

Simplest continuous deployment ever!

Capistrano!!

+ !!

Capifony

Deployment Examples

Capistrano / Capifony

• Ruby based!• Very extensible!• Large number of extensions!• Simple client side installation

Deployment Examples

• Ruby based!• Very extensible!• Large number of extensions!• Simple client side installation

$ gem install capifony!

Deployment Examples

Capistrano / Capifony

set :application, "myapp" # Application name set :deploy_to, "/var/www/myapp" !set :user, "deployer" set :use_sudo, false # sudo isn't required !set :deploy_via, :remote_cache set :repository, "git@github.com:user/repo.git" !role :web, "server.example.com", “server2.example.com”

Deployment Examples

Capistrano / Capifony

$ cap deploy:setup

Deployment Examples

Capistrano / Capifony

!

|-- releases!`-- shared! |-- logs! `-- uploads!

Deployment Examples

Capistrano / Capifony

$ cap deploy!$ cap deploy:migrations!$ cap deploy:rollback!

Deployment Examples

Capistrano / Capifony

!

|-- current (symlink to releases/20130112)!|-- releases!| `-- 20130112!`-- shared! |-- logs! `-- uploads!

Deployment Examples

Capistrano / Capifony

Deployment Examples

Deploying with

Deployment Examples!

!

Define hosts in inventory file[webservers] foo.example.com bar.example.comwww[01:50].example.com ![dbservers] one.example.com two.example.com three.example.com

Deployment Examples!

!

Define tasks / actions in playbook--- - hosts: webservers vars: http_port: 80 max_clients: 200 remote_user: root tasks: - name: ensure apache is at the latest version yum: pkg=httpd state=latest - name: write the apache config file template: src=/srv/httpd.j2 dest=/etc/httpd.conf notify: - restart apache - name: ensure apache is running service: name=httpd state=started

Deployment Examples!

!

Reuse tasks with Roles- hosts: webservers roles: - { role: servergrove.symfony2, symfony2_project_root: /var/www/vhosts/example.com/, symfony2_project_name: demo, symfony2_project_branch: master , symfony2_project_release: 1 }

Deployment Examples!

!

Run process

$ ansible-playbook -i inventories/servers \ playbook.yml -v

Deployment Examples!

!

Run process

Deployment: Other options

• Fabric (Python) • Idephix (PHP) • Magellanes (PHP) • Deployer (PHP) • Laravel / envoy (PHP) • Rocketeer (PHP)

Deployment Complement Tools

Packaging: fpm

https://github.com/jordansissel/fpm

Build packages for multiple platforms (deb, rpm, etc) with great ease and sanity.

$ fpm -s dir -t rpm -n "myapp" -v 1.0 /var/www/myapp !$ fpm -s dir -t deb -a all -n myapp -v 1.0 /etc/apache2/conf.d/my.conf /var/www/myapp

Deployment Complement Tools

App Metrics: StatsD & Graphite

Deployment Complement Tools

App Metrics: liuggio/statsd-php-client

Deployment Complement Tools

$ composer require liuggio/statsd-php-client

App Metrics: liuggio/statsd-php-client

Deployment Complement Tools

$sender = new SocketSender(/*'localhost', 8126, 'udp'*/); !$client = new StatsdClient($sender); $factory = new StatsdDataFactory('\Liuggio\StatsdClient\Entity\StatsdData'); !// create the data with the factory $data[] = $factory->timing('usageTime', 100); $data[] = $factory->increment('visitor'); $data[] = $factory->decrement('click'); $data[] = $factory->gauge('gaugor', 333); $data[] = $factory->set('uniques', 765); !// send the data as array or directly as object $client->send($data);

App Metrics: liuggio/statsd-php-client

Deployment Complement Tools

$data[] = $factory->timing('usageTime', 100);!!

// send the data as array or directly as object $client->send($data);

App Metrics: liuggio/statsd-php-client

Deployment Complement Tools

$data[] = $factory->increment('orders');!!

// send the data as array or directly as object $client->send($data);

App Metrics: liuggio/statsd-php-client

Deployment Complement Tools

$data[] = $factory->gauge('memory', 333);!!

// send the data as array or directly as object $client->send($data);

App Metrics: StatsD & Graphite

Deployment Complement Tools

App

StatsD Graphite

Grafana

App App

Deployment Complement Tools

App Metrics: Grafana

Deployment Complement Tools

App Metrics: Grafana

Logging: LogstashShip logs from any source, parse them, get the right timestamp, index them, and search them

Deployment Complement Tools

input { file { path => “/var/log/apache2/access_log” } } output { elasticsearch { host => localhost } }

Configure Apache to log jsonLogFormat "{ \"@timestamp\": \"%{%Y-%m-%dT%H:%M:%S%z}t\", \"@fields\": { \"client\": \"%a\", \"duration_usec\": %D, \"status\": %s, \"request\": \"%U%q\", \"method\": \"%m\", \"referrer\": \"%{Referer}i\" } }" logstash_json!!!# Write our 'logstash_json' logs to logs/access_json.log!CustomLog logs/access_json.log logstash_json!

Deployment Complement Tools

Logging: Logstash

Configure Apache to log jsonLogFormat "{ \"@timestamp\": \"%{%Y-%m-%dT%H:%M:%S%z}t\", \"@fields\": { \"client\": \"%a\", \"duration_usec\": %D, \"status\": %s, \"request\": \"%U%q\", \"method\": \"%m\", \"referrer\": \"%{Referer}i\" } }" logstash_json!!!# Write our 'logstash_json' logs to logs/access_json.log!CustomLog logs/access_json.log logstash_json!

{ "@timestamp": "2012-08-22T14:35:19-0700", "client": "127.0.0.1", "duration_usec": 532, "status": 404, "request": "/favicon.ico", "method": "GET", "referrer": "-" }!

Result

Deployment Complement Tools

Logging: Logstash

Logging: KibanaKibana is a user friendly way to view, search and visualize your log data

Deployment Complement Tools

Logging: KibanaKibana is a user friendly way to view, search and visualize your log data

Deployment Complement Tools

Logging: Logstash + ElasticSearch + Kibana

Deployment Complement Tools

App / Logstash

Redis Logstash filter/processor

ElasticSearch Kibana

App / Logstash App / Logstash

Deployment Summary

Developer Tests

build artifact

Ansible Production

GitHub

Jenkins QA deploy

Integration Tests

Notifications

Deployment Summary

Developer Tests

build artifact

Ansible Production

GitHub

Jenkins QA deploy

Integration Tests

Notifications

Deployment Summary

Developer Tests

build artifact

Ansible Production

GitHub

Jenkins QA deploy

Integration Tests

Notifications

Deployment Summary

Developer Tests

build artifact

Ansible Production

GitHub

Jenkins QA deploy

Integration Tests

Notifications

Deployment Summary

Developer Tests

build artifact

Ansible Production

GitHub

Jenkins QA deploy

Integration Tests

Notifications

Deployment Summary

Developer Tests

build artifact

Ansible Production

GitHub

Jenkins QA deploy

Integration Tests

Notifications

Deployment Summary

Developer Tests

build artifact

Ansible Production

GitHub

Jenkins QA deploy

Integration Tests

Notifications

Deployment Summary

Developer Tests

build artifact

Ansible Production

GitHub

Jenkins QA deploy

Integration Tests

Notifications

Deployment Summary

Developer Tests

build artifact

Ansible Production

GitHub

Jenkins QA deploy

Integration Tests

Notifications

Deployment Summary

Developer Tests

build artifact

Ansible Production

GitHub

Jenkins QA deploy

Integration Tests

Notifications

Deployment Summary

Developer Tests

build artifact

Ansible Production

GitHub

Jenkins QA deploy

Integration Tests

Notifications

Deployment Summary

Deployment Future

Deployment Future

Containers

Deployment Future

Docker

Web App

Linux

Apache / PHP

Deployment Future

DockerHost

Docker client

CVM

Web Server

DB Server

App Server

C C

DockerHost

Docker client

C

VM

Web Server

DB Server

App Server

C C

Deployment Future

DockerHost

Docker client

C

VM

Web Server

DB Server

App Server

C C

Deployment Future

DockerHost

Docker client

C

VM

Web Server

DB Server

App Server

C C

Host

C

VM

Web Server

DB Server

App Server

C C

Deployment Future

DockerHost

Docker client

C

VM

Web Server

DB Server

App Server

C C

Host

C

VM

Web Server

DB Server

App Server

C C

Host

C

Web Server

DB Server

App Server

C C

Docker Kernel

Deployment Future

Final Thoughts

Stop using FTP

Plan early

Practice

Monitor

AUTOMATE!

Now, we did finish! !

Questions?

Feedback Please! !

!

http://slideshare.net/pgodel @pgodel

https://joind.in/12960

Feedback Please! !

!

Thank you!

http://slideshare.net/pgodel @pgodel

https://joind.in/12960