let's code our infrastructure!

Let’s Code our Infrastructure

Who I am● Oswald De Riemaecker

● Continuous S.A. CEO & Co-Founder o Agile Delivery specialist o Editor of the continuousphp PaaS

● « Agilist », Cloud Architect

Agile Culture● Improvement in the development team

o Better collaboration o Continuous improvement o All skills / No silos o Testing Framework and continuous integration

● Improvement in Agile product management o Iterative development o Features driven o Maximizing Value o MVP/Lean Startup

DevOps Culture● Agile + Operation

● 3 main pillars: o Infrastructure as Code (IaC) o Continuous Delivery / Deployment o Culture of cooperation

Infrastructure as Code● Coding machines resources of the infrastructure

● Coding and versioning the infrastructure

● Testing both resources and infrastructure

● On demand rebuilding machines and infrastructure

OrganizationMy Infrastructure Your Infrastructure Their Infrastructure

Local

EnvironmentsDevelopment Testing Staging Production

Roles

Load Balancer

Application Servers

Indexes Servers

Databases Servers

Resources● A resource is a part of the system and its configuration

Apache MySQL Solr

Complexity● Maintaining infrastructure is complex ● It lives and evolves, so you have to

o Add new resources o Update resources o Apply Security Patches o Change/Update OS o ...

Tools● The answer to this:

Chef● Chef is an open-source tool ● Management of automation systems and infrastructure ● Chef is based on Ruby

Chef● Nodes represent the servers within your infrastructure

● Nodes can be: o physical servers o virtual servers o instances in the cloud

● Each node belongs to: o an organization o an environment o has one or more role(s)

Node

NodeNodeNode

Node Node

Chef Client / Server

Node

NodeNodeNode

Node Node

Chef Server

chef-client

Resources

Chef Cookbooks● The cookbooks contain artifacts of our resources

Recipes

Templates

Files

Attributes State of the resources


Recipes

Templates

Files


Installation / configuration of resources


Recipes

Templates

Files



Configuration templates


Recipes

Templates

Files



Configuration templates

Configuration/script files

Provisioning

Role Provisioning

Node Role:

Database

Chef Server Hey, I am role “Database”

and my environment is “Development”

Chef clientRecipes

Templates

Attributes

Files

Cookbook Recipes

Templates

Attributes

Files

Cookbook

Infrastructure Example● Web Server - Apache 2.4/PHP-5.5 ● Database Server - MySQL ● Index Server - Solr Web

Server

Solr

MySQL

The Chef Community

Apache Cookbook Analysis● Support many platforms ● Support different Apache versions ● Attributes:

o dir, docroot_dir o log_dir, error_log, access_log o default_modules o version o ...

Cookbook support and dependencies

Recipes

Templates

Attributs

Files

Metadata

Dependancies

The attributes

Recipes

Templates

Attributes

Files

Metadata

The recipes

Recipes

Templates

Attributes

Files

Metadata

cd recipes ls -l

default.rb iptables.rb logrotate.rb mpm_prefork.rb mpm_worker.rb mod_php5.rb mod_*.rb

The Templates

Recipes

Templates

Attributes

Files

Metadata

The Files

Recipes

Templates

Attributes

Files

Metadata

cd files/default/ ls -l

apache2_module_conf_generate.pl

Let’s code our cookbooks

Role: Web Server / Env: Production

Apache / php-5.5

Recipes

Templates

Attributes

Files

Metadata

Apache2

Recipes

Templates

Attributes

Files

Metadata

iptables

Recipes

Templates

Attributes

Files

Metadata

logrotate

PRODUCTION: Replacement AttributesNode Role: WebServer

RO L E

Chef Development Kit● Installation of chef-dk

o Berkself o Knife o ChefSpec o Test-Kitchen o Serverspec o Foodcritic

● knife cookbook create mon_org-webserver

Creating our web server cookbook

Adding our dependencies

Dépendance(s)

● berks init

Initialization with Berkshelf

● berks install

Downloading dependencies

● vi attributes/defaults.rb

The Attributes

The Recipes

Testing our cookbook● kitchen list

Testing our cookbook● kitchen verify default-ubuntu-1404

● knife cookbook upload mon_org-webserver

Versioning and uploading our cookbook

Repeating the exercise for database and indexes roles

Provisioning our SandboxDevelopment

Role: Sandbox / Environment: Dev

mon_org-webserver

mon_org-index

mon_org-database

Composer NodeJs

Recipes

Templates

Attributs

Files

Metadata

Recipes

Templates

Attributs

Files

Metadata

Recipes

Templates

Attributs

Files

Metadata

Recipes

Templates

Attributs

Files

Metadata

Recipes

Templates

Attributs

Files

Metadata

mon_org-webserver mon_org-Index mon_org-database composer nodejs

DEVELOPMENT: Replacement of AttributsNode Role: Sandbox

RO L E

Creating our sandbox cookbook● knife cookbook create mon_org-sanbox

Adding dependencies

Dependancies

Downloading dependencies

The Recipes● vi recipes/default.rb

● vi attributes/default.rb

The attributes

Attributs

Testing our cookbook

● Cookbooks by role of our infrastructure ● Specific Cookbook for our development environment

o uses the cookbooks of our infrastructure roles

● We can reliably reproduce our infrastructure roles, it is:

o Coded o Versioned o Tested o Deployed

Provisioning Summary

Let’s create our environments

Local

Creating our development environmentDevelopment

Vagrant● Create and set up development environments

● Providers (Drivers) o Virtualbox/VMWare o Docker o Instance EC2

● Provisioning with: o Chef Client/Solo o Docker o Puppet Agent o …

Vagrantfile

Vagrant up

Immutable Infrastructure

Immutable Infrastructure

● Benefits: o Saving time when supplying o No more maintenance of the servers => Replacement o Static machine Image pre-built and tested o Easy rollback to a previous version

V1

V2

V3

Data

Packer● Packer

o Builders (Amazon AMI, Microsoft Azure, DigitalOcean, Docker, Google Compute Engine, …)

o Provisioners (Shell script, Chef client, Ansible, Puppet, Salt, …) o Post-processor (Docker-push, Vagrant, Atlas, …)

Packer Config

Builders

Packer Config

Provisioners

● packer build packer.json

Creating our image

Creating our image

Repeating the exercise for the index role

Creating our Production/Staging environment

AWS CloudFormation● Easy way to create and manage an AWS resource

● CloudFormation takes care of dependencies

● Once the resources are deployed, you can edit and update in a predictable and controlled manner

● AWS CloudFormation supports the following resources:

o Auto Scaling/Elastic Load Balancing o VPC o EC2 o RDS o IAM o …

VPC

Private Subnet

Public Subnet

JSON

Autoscale

Web Web Web

Autoscale

Index Index Index

Cloud Formation Structure

Creating our web server stack

● Machines Provisioning

● Immutable Infrastructure o Static Machine Images

● Using Machine Images in AWS Cloud Formation

● Cloud Formation environment parameters

● We can now use our artifacts to build our different environments

Infrastructure Summary

Orchestrating our infrastructureTesting

Development

Vagrant

Feature branch

develop branch

revision branch

Staging

Production

CI/CD

prepare test package



Questions?

Thank you!

oswald_odr

[email protected]

mailto:[email protected]

let's code our infrastructure!

Technology