symfonycon madrid 2014 - rock solid deployment of symfony apps

Photo: Alvaro Videla

Pablo Godel @pgodel

Rock Solid Deployment of Symfony

Applications

Photo by @old_sound

https://joind.in/12960


Hi, I am Pablo.

Hi, I am Pablo. !

@pgodel

A GUIDING LIGHT FOR STUDENT SUCCESS

Symfony2 + AngularJS + much more!


We are Hiring!


Deployment

?

Deployment

Software deployment is all of the activities that !make a software system available for use.

http://en.wikipedia.org/wiki/Software_deployment

http://en.wikipedia.org/wiki/Software_system

http://en.wikipedia.org/wiki/Software_deployment

Deployment

A very important part of the application life-cycle

Deployment

A very important critical part of the application life-cycle

Deployment

It should not be an !after thought

Deployment

It should be predictable

Deployment

The more you do it the better it goes

Deployment in the PaaS era

1. Open an account

2. Create app

3. Push code

4. Voilà!

composer.json !!

Procfile

$ heroku create!

$ git push heroku master!

$ heroku ps:scale web=5!

That’s it! ;-)

Questions?

Slides: http://slideshare.net/pgodel Twitter: @pgodel

http://slideshare.net/pgodel

But…

Deployment for the rest of us

Deployment Goals

One-click deploys

Deployment Goals

Continuous deploys

Deployment Goals

Anytime & Anywhere

Deployment Goals

Anyone

Deployment Goals

Reliable

Deployment Goals

Rollbacks

Deployment Goals

No downtime

Deployment Goals

Reusable

Deployment Goals

Scalable

Deployment Goals

Deployment Facts

Deployment starts with the developer

Deployment Fact #1


• Development environment must be as close as possible to productions servers!

Deployment Fact #1



• Setup test/qa/staging servers!

Deployment Fact #1



• Setup test/qa/staging servers!• Use Vagrant to manage VMs!

Deployment Fact #1



• Setup test/qa/staging servers!• Use Vagrant to manage VMs!• Use Containers with Docker!

Deployment Fact #1



• Setup test/qa/staging servers!• Use Vagrant to manage VMs!• Use Containers with Docker!• Automate your development environment setup

(Ansible, Homebrew Cask)

Deployment Fact #1

Success depends on server OS setup

Deployment Fact #2


• Automate installation/configuration with Ansible/Puppet/Chef !

Deployment Fact #2


• Create OS packages for 3rd party software

Deployment Fact #2



• Create OS packages for 3rd party software!• Setup your own package (rpm/deb) repositories

Deployment Fact #2


Monitoring is uptime

Deployment Fact #3


• Use tools to know what is going on with your servers (Ganglia, Cacti, Zabbix, etc.)!

Deployment Fact #3



• Add metrics to your app (Graphite, StatsD, New Relic)!

Deployment Fact #3



• Add metrics to your app (Graphite, StatsD, New Relic)!

• Use your logs wisely (Logstash, Hecka, Kibana)

Deployment Fact #3

Deployment Methodologies


• VIM-style


• VIM-style!• FTP uploads


• VIM-style!• FTP uploads!• Rsync


• VIM-style!• FTP uploads!• Rsync!• Source control (GIT, SVN)


• VIM-style!• FTP uploads!• Rsync!• Source control (GIT, SVN)!• Build tools (Ant, Phing, Jenkins)!


• VIM-style!• FTP uploads!• Rsync!• Source control (GIT, SVN)!• Build tools (Ant, Phing, Jenkins)!• Package based (RPM, DEB, …) !


• VIM-style!• FTP uploads!• Rsync!• Source control (GIT, SVN)!• Build tools (Ant, Phing, Jenkins)!• Package based (RPM, DEB, …) !• Specialized tools (Capifony, Deployer, …)!


• VIM-style!• FTP uploads!• Rsync!• Source control (GIT, SVN)!• Build tools (Ant, Phing, Jenkins)!• Package based (RPM, DEB, …) !• Specialized tools (Capifony, Deployer, …)!• IT automation tools (Ansible, Puppet, Chef, …)

Deployment: Steps overview

Deployment: First time

• Copy files to server(s)!• Set server-side configurations!• Load DB fixtures!• Process and install assets!• Warm up cache!• Enable site

Deployment: Subsequent times

• Copy files to server(s)!• Update server-side configurations!• Update DB (migrations)!• Process and install assets!• Warm up cache!• Enable new version

We can automate!

Deployment Challenges

Challenge: Secure


• use ssh based connections!

Challenge: Secure


• use ssh based connections!• don’t store passwords on source control

Challenge: Secure


• use ssh based connections!• don’t store passwords on source control!• use environment variables for sensitive data

Challenge: Secure


Challenge: Static assets



• Minimize/combine JS and CSS files!



• Minimize/combine JS and CSS files!• Enable web server compression!



• Minimize/combine JS and CSS files!• Enable web server compression!• Add versioning to static assets links (code.js?v=1)!



• Minimize/combine JS and CSS files!• Enable web server compression!• Add versioning to static assets links (code.js?v=1)!• Run utilities locally or in a staging server, create

artifact, deploy result!



• Minimize/combine JS and CSS files!• Enable web server compression!• Add versioning to static assets links (code.js?v=1)!• Run utilities locally or in a staging server, create

artifact, deploy result!• Serve static files from subdomain / CDN


Challenge: File permission conflicts



• Run Apache/PHP with same user!



• Run Apache/PHP with same user!• Use php-fpm instead of mod_php!



• Run Apache/PHP with same user!• Use php-fpm instead of mod_php!• Create “deploy” user and add web server to the

group




group!• Use setfacl to give write access to multiple users!




group!• Use setfacl to give write access to multiple users!• Use external services for writing files and create

readonly installations


Common Pitfalls

• Case sensitive filesystems!

Common Pitfalls

• Case sensitive filesystems!!

src/Acme/ClassName !!= !src/Acme/Classname.php

Common Pitfalls

• Case sensitive filesystems!• Configuration differences!

Common Pitfalls

• Case sensitive filesystems!• Configuration differences!• Outdated 3rd party software

Common Pitfalls

• Case sensitive filesystems!• Configuration differences!• Outdated 3rd party software!• Github down

Common Pitfalls


$ git daemon --base-path=/git/repo/path/ --export-all!!

$ git clone git://127.0.0.1/repo!

http://ozmm.org/posts/when_github_goes_down.html

Common Pitfalls

git://127.0.0.1/repo

http://ozmm.org/posts/when_github_goes_down.html


Common Pitfalls

Deployment Examples

Simplest continuous deployment ever!

<?php !!exec(‘/usr/bin/env -i HOME=/var/www git pull’);!echo “All done!”;

hook.php

Deployment Examples

<?php !!exec(‘/usr/bin/env -i HOME=/var/www git pull’);!echo “All done!”;

hook.php

screenshot

Deployment Examples

GitHub hook

Simplest continuous deployment ever!

Capistrano!!

+ !!

Capifony

Deployment Examples

Capistrano / Capifony

• Ruby based!• Very extensible!• Large number of extensions!• Simple client side installation

Deployment Examples

• Ruby based!• Very extensible!• Large number of extensions!• Simple client side installation

$ gem install capifony!

Deployment Examples


set :application, "myapp" # Application name set :deploy_to, "/var/www/myapp" !set :user, "deployer" set :use_sudo, false # sudo isn't required !set :deploy_via, :remote_cache set :repository, "[email protected]:user/repo.git" !role :web, "server.example.com", “server2.example.com”

Deployment Examples


mailto:[email protected]

$ cap deploy:setup

Deployment Examples


!

|-- releases!`-- shared! |-- logs! `-- uploads!

Deployment Examples


$ cap deploy!$ cap deploy:migrations!$ cap deploy:rollback!

Deployment Examples


!

|-- current (symlink to releases/20130112)!|-- releases!| `-- 20130112!`-- shared! |-- logs! `-- uploads!

Deployment Examples


Deployment Examples

Deploying with

Deployment Examples!

!

Define hosts in inventory file[webservers] foo.example.com bar.example.comwww[01:50].example.com ![dbservers] one.example.com two.example.com three.example.com

http://bar.example.com

http://example.com

http://three.example.com


!

Define tasks / actions in playbook--- - hosts: webservers vars: http_port: 80 max_clients: 200 remote_user: root tasks: - name: ensure apache is at the latest version yum: pkg=httpd state=latest - name: write the apache config file template: src=/srv/httpd.j2 dest=/etc/httpd.conf notify: - restart apache - name: ensure apache is running service: name=httpd state=started


!

Reuse tasks with Roles- hosts: webservers roles: - { role: servergrove.symfony2, symfony2_project_root: /var/www/vhosts/example.com/, symfony2_project_name: demo, symfony2_project_branch: master , symfony2_project_release: 1 }


!

Run process

$ ansible-playbook -i inventories/servers \ playbook.yml -v


!

Run process

Deployment: Other options

• Fabric (Python) • Idephix (PHP) • Magellanes (PHP) • Deployer (PHP) • Laravel / envoy (PHP) • Rocketeer (PHP)

Deployment Complement Tools

Packaging: fpm

https://github.com/jordansissel/fpm

Build packages for multiple platforms (deb, rpm, etc) with great ease and sanity.

$ fpm -s dir -t rpm -n "myapp" -v 1.0 /var/www/myapp !$ fpm -s dir -t deb -a all -n myapp -v 1.0 /etc/apache2/conf.d/my.conf /var/www/myapp


https://github.com/jordansissel/fpm

App Metrics: StatsD & Graphite


App Metrics: liuggio/statsd-php-client


$ composer require liuggio/statsd-php-client



$sender = new SocketSender(/*'localhost', 8126, 'udp'*/); !$client = new StatsdClient($sender); $factory = new StatsdDataFactory('\Liuggio\StatsdClient\Entity\StatsdData'); !// create the data with the factory $data[] = $factory->timing('usageTime', 100); $data[] = $factory->increment('visitor'); $data[] = $factory->decrement('click'); $data[] = $factory->gauge('gaugor', 333); $data[] = $factory->set('uniques', 765); !// send the data as array or directly as object $client->send($data);



$data[] = $factory->timing('usageTime', 100);!!

// send the data as array or directly as object $client->send($data);



$data[] = $factory->increment('orders');!!




$data[] = $factory->gauge('memory', 333);!!


App Metrics: StatsD & Graphite


App

StatsD Graphite

Grafana

App App


App Metrics: Grafana

Logging: LogstashShip logs from any source, parse them, get the right timestamp, index them, and search them


input { file { path => “/var/log/apache2/access_log” } } output { elasticsearch { host => localhost } }

Configure Apache to log jsonLogFormat "{ \"@timestamp\": \"%{%Y-%m-%dT%H:%M:%S%z}t\", \"@fields\": { \"client\": \"%a\", \"duration_usec\": %D, \"status\": %s, \"request\": \"%U%q\", \"method\": \"%m\", \"referrer\": \"%{Referer}i\" } }" logstash_json!!!# Write our 'logstash_json' logs to logs/access_json.log!CustomLog logs/access_json.log logstash_json!


Logging: Logstash

Configure Apache to log jsonLogFormat "{ \"@timestamp\": \"%{%Y-%m-%dT%H:%M:%S%z}t\", \"@fields\": { \"client\": \"%a\", \"duration_usec\": %D, \"status\": %s, \"request\": \"%U%q\", \"method\": \"%m\", \"referrer\": \"%{Referer}i\" } }" logstash_json!!!# Write our 'logstash_json' logs to logs/access_json.log!CustomLog logs/access_json.log logstash_json!

{ "@timestamp": "2012-08-22T14:35:19-0700", "client": "127.0.0.1", "duration_usec": 532, "status": 404, "request": "/favicon.ico", "method": "GET", "referrer": "-" }!

Result


Logging: Logstash

Logging: KibanaKibana is a user friendly way to view, search and visualize your log data


Logging: Logstash + ElasticSearch + Kibana


App / Logstash

Redis Logstash filter/processor

ElasticSearch Kibana

App / Logstash App / Logstash

Deployment Summary

Developer Tests

build artifact

Ansible Production

GitHub

Jenkins QA deploy

Integration Tests

Notifications

Deployment Summary

Deployment Future

Deployment Future

Containers

Deployment Future

Docker

Web App

Linux

Apache / PHP

Deployment Future

DockerHost

Docker client

CVM

Web Server

DB Server

App Server

C C

DockerHost

Docker client

C

VM

Web Server

DB Server

App Server

C C

Deployment Future

DockerHost

Docker client

C

VM

Web Server

DB Server

App Server

C C

Host

C

VM

Web Server

DB Server

App Server

C C

Deployment Future

DockerHost

Docker client

C

VM

Web Server

DB Server

App Server

C C

Host

C

VM

Web Server

DB Server

App Server

C C

Host

C

Web Server

DB Server

App Server

C C

Docker Kernel

Deployment Future

Final Thoughts

Stop using FTP

Plan early

Practice

Monitor

AUTOMATE!

Now, we did finish! !

Questions?

Feedback Please! !

!

http://slideshare.net/pgodel @pgodel



Feedback Please! !

!

Thank you!

http://slideshare.net/pgodel @pgodel



symfonycon madrid 2014 - rock solid deployment of symfony apps

Technology