devops, common use cases, architectures, best practices

DevOps

Daniel Zoltak & Shiva N

Use Cases, Architectures & Best Practices

Agenda

• Culture

• Technology

• Your experiences

• Breakdown the barriers • Work as one team end to end • Support business and IT agility • Collaboration & Communication • Treat Infrastructure as code • Automate everything • Test everything • Measure & monitor everything

Agenda…

Culture

Technology

Competing Forces

Business

Development Operations

Build it faster Keep it stable

Change is inevitable

• Change is the root cause of most outages

• This leaves two approaches: -  Make it a big deal

-  Small iterative non events

“Do painful things more frequently, so you can make it less painful…”

- Adrian Cockcroft, Architect, Netflix

Deploying More Frequently Lowers Risk

Smaller Effort “Minimized Risk”

Frequent Release Events: “Agile Methodology”

Time

Cha

nge

Rare Release Events: “Waterfall Methodology”

Larger Effort “Increased Risk”

Time

Cha

nge

Knowledge is power

Requirements Gathering Release Development

and Check In Test and QA

Some Learning

Lots of Learning

Minimal Learning

Cycle Time

“How long would it take your organization to deploy a change that involves just one single line of code?” “Do you do this on a repeatable, reliable basis?”

- Mary & Tom Poppendieck, Lean Software Development

Waterfall

Analyze Design Build Test Deploy

Agile Sprint

Test Deploy

Agile-er

A D B A D B

Sprint Sprint

A D B

Sprint

A D B T D

Sprint Sprint

A D B T D A D B T D

Agile-er Dev QA Ops

Dev QA Ops

Dev/QA/Ops Dev/QA/Ops Dev/QA/Ops

What is it?

•  A philosophy? Cultural change? Paradigm shift ?

•  Alignment of development and IT operations with better communication and collaboration ?

•  Improvement in software deployment ?

•  Breaking down the barriers between development and IT operations ?

•  Akin to Agile software development applied to infrastructure and IT operations

•  Set of tools and processes •  It’s all of the above!

Code

Wal l

Developer IT Operations

DevOps

Building Teams

• Make DevOps everyone’s job - No silo/walled off DevOps teams

• Encourage developers to participate openly in the automation of operations code

• Embolden operations participation in testing and automation of application code

•  “Take pride” in how fast and frequent you deploy

DevOps Team Anti Patterns

Dev Ops

Separate Silos

Dev Ops DevOps

Separate DevOps Silo

Dev Ops DevOps

We don’t need Ops http://www.slideshare.net/matthewskelton/devops-patterns-team-topologies

DevOps is a Re-Org! - Adrian Cockcroft, Architect, Netflix

RE

DevOps Team Patterns

Dev Ops

Collaboration Dev Ops

Embedded

Dev Ops

DevOps-as-a-service

DevOps Dev Ops DevOps

Temporary DevOps team

http://www.slideshare.net/matthewskelton/devops-patterns-team-topologies

Business Case Requirements Use Case Features Plan Go to market

Business

Design Code Refactor Unit Test Bug Fix Deliver

Developers (application)

Provision Configure Orchestrate Deploy Report Monitor

IT Operations (infrastructure)

Agile Development

DevOps •  Continuous Integration •  Continuous Deployment •  IT Automation •  Application Management

•  Iterative development •  Scrum, sprints, stories •  Velocity

Business Agility

IT Agility

Business and IT agility

Collaboration and Communication

Standups Pairing

Rotation Visibility

Everyone knows what production looks like.

Building a DevOps Culture

• Encourage a fail fast, learn quick mindset • Foster innovation and accountability • Promote open and honest sharing of lessons

learned • Build trust across organizational boundaries

- Include all the stakeholders

• Expect that it will be a work-in-progress

The DevOps Stack

The DevOps Stack

Continuous Deployment

Delivery Pipelines

Deployment Automation

Continuous Integration

Automated Testing

Configuration Management

Agi

le

Com

mun

icat

ion

•  Infrastructure as code

•  IT Automation

• Continuous Integration - Application

§  Compile, test, optimize (code coverage)

-  Infrastructure

§  Logical, valid, secure

DevOps Practices

• Continuous Deployment - Application - Rollout & Rollback

• Version control integration

• Application and Infrastructure version management

• Monitoring and logging

DevOps Practices

Continuous Integration

• Every developer check-in initiates a build -  Builds are fully automated

-  Tests validate every check-in

-  Always check-in to the trunk

•  Instant feedback for developers - The build must remain fast

• Minimizes the impact of broken builds - Bugs don’t accumulate

• Drives the culture of small, frequent releases

Continuous Integration Tools

Automated Testing

• Robust, automated testing drives confidence in the entire process

•  Testing is not viewed as a “phase” of a project - Testing starts on day one

• View failures as successes • Don’t overlook testing the infrastructure

- Are you ready for the Simian Army?

•  Testing is everyone’s responsibility

Configuration Management

• Everything must go into version control

•  Implement an effective strategy for branching and merging

• Don’t let tools get in the way

Deployment Automation •  Invest in streamlining and unifying all aspects of the

deployment process •  An application’s environment and configuration should be

managed through code - Remove any dependencies on human intervention

• Every manual step represents an opportunity for future failure - Console tweaks = future bugs

• Value repeatability - How fast can you completely rebuild your environment from scratch?

One Step Environment Creation

• Require a single build process for all environments - Try to build on horizontal concepts

• Expect the environments to be constantly evolving - Fragile automation becomes fragile environments

•  “The longer you wait to have a common environment build process, the harder it is to create one”

Version Control

Build/ Compile

Code Dev

Unit Test App Code

IT Ops

DR Env

Test Env

Prod Env

Dev Env

Application Write App

Code

Infrastructure

CloudFormation

tar, war, zip yum, rpm Deploy

App

Package Applicatio

n

Deploy application only

Deploy infrastructure only

AMI

Build AMIs

Validate Template

s

Write Infra Code

Deploy Infras

Automate Deployme

nt

Artifact Repository

Continuous Integration / Deployment & Automation

Delivery Pipelines

Commit Unit Test

System Test

•  Defines the automated lifecycle of delivery •  Enables collaboration between the groups •  Provides everyone with visibility into the flow of change

Commit Unit Test

System Test

UAT Staging Prod Commit Unit Test

System Test

Continuous Deployment

• Code that makes it through the pipeline goes straight to production

• Requires continuous integration and delivery

• Relies heavily on small, constantly tested releases

• Often a stretch goal for many organizations

Continuous Deployment vs Delivery

• Deploy in-place - Deploy all at once (Service outage)

-  Rolling updates

Deployment Approaches

Demo

• Blue-Green Deployment - Discrete environment

- Multiple environments from branches

- Support A/B testing

- “Rolling DNS”

• Alternate Blue-Green (Red-Black?) deployment - Attach and detach Auto Scaling groups from Elastic Load balancers

- Avoid messing with DNS

Deployment Approaches

Blue-Green Deployment

Amazon Route 53

EC2 Instances

ELB

100%

DynamoDB MySQL RDS

Instance ElastiCache Cache Node

• Stand up duplicate infrastructure and slowly cut traffic over to it - Shift via DNS - Simplifies feature testing - Simplifies roll back

• As we shift more traffic over, let auto-scaling grow/shrink our instances - Shut down the old when no traffic there

Blue-Green Deployment

Amazon Route 53

EC2 Instances

ELB

EC2 Instances

ELB

90% 10%

DynamoDB MySQL RDS Instance

ElastiCache Cache Node

• Stand up duplicate infrastructure and slowly cut traffic over to it - Shift via DNS - Simplifies feature testing - Simplifies roll back

• As we shift more traffic over, let auto-scaling grow/shrink our instances - Shut down the old when no traffic there

Blue-Green Deployment

Amazon Route 53

EC2 Instances

ELB

EC2 Instances

ELB

0% 100%

DynamoDB MySQL RDS Instance

ElastiCache Cache Node

• Stand up duplicate infrastructure and slowly cut traffic over to it - Shift via DNS - Simplifies feature testing - Simplifies roll back

• As we shift more traffic over, let auto-scaling grow/shrink our instances - Shut down the old when no traffic there

Blue-Green Deployment

• Stand up duplicate infrastructure and slowly cut traffic over to it - Shift via DNS - Simplifies feature testing - Simplifies roll back

• As we shift more traffic over, let auto-scaling grow/shrink our instances - Shut down the old when no traffic there

Amazon Route 53

EC2 Instances

ELB

EC2 Instances

ELB

0% 100%

DynamoDB MySQL RDS Instance

ElastiCache Cache Node

ASG V1

ELB

Red-Black Deployment

RDS

Red-Black Deployment

ELB

ASG V1 ASG V2

UAT

RDS

Red-Black Deployment

ELB

ASG V1 ASG V2

RDS

Red-Black Deployment

ELB

RDS

ASG V2

Agile Everywhere

• Agile tenets are core to all dimensions of the DevOps lifecycle

- Small, iterative releases

- Quality is everyone’s job

- Customer focused

- Fast, constant feedback

- Done means “released”

Automation Everything

• Automation is vital to smaller, more frequent releases • All steps that add human intervention also add

fragility - “Small compromises can undermine your success”

• Use knowledge gathered from frequent, automated releases to constantly evolve the process

If you have to login to production, your DevOps-fu is sub optimal

• Scalability (anything manual is not scalable) • Reliability • Reproduction/Duplication

• Environment consistency • Auditability/Record Keeping

• Security • Governance

Infrastructure as code – why?

"WebServer": { "Type": "AWS::EC2::Instance", "Metadata" : { "AWS::CloudFormation::Init" : { "config" : { "packages" : { "yum" : { "httpd" : [], "php" : [], "php-mysql" : [], "php-gd" : [], "php-xml" : [], "php-mbstring" : [], "mysql" : [] } }, "sources" : { "/var/www/html" : "http://ftp.drupal.org/files/projects/drupal-7.8.tar.gz", "/home/ec2-user" : "http://ftp.drupal.org/files/projects/drush-7.x-4.5.tar.gz" },

AWS CloudFormation

template

Here’s some infrastructure as Code

Describing Infrastructure with Code

Developers &

Operations Internal Git CI Server

Pre-commit Hook

Testing Environment Subnet

CI Workers

Dev Environment VPC Subnet

DEV WEB ELB Dev Stack

Tier 1 Dev Stack

Tier 2

Dev MySQL DB Instance

DEV APP ELB

VPN TUNNEL

VPN facing VPC Subnet

Internet Gateway

VPN Endpoint

Dev Admin Instance

NAT Instance

Amazon S3

Amazon DynamoDB

Amazon SQS

Amazon CloudFront

Amazon Route 53

This entire infrastructure stack can be constructed, configured, and deployed with code:

Infrastructure Code

The code that describes infrastructure should inherit the same values applied to application code

• Not JUST revision control • Make use of bug tracking/ticketing systems • Peer reviews of changes before they happen • Establish infrastructure code patterns/designs • Test infrastructure changes like code changes

Declarative Approach to: • Provisioning • Configuration • Orchestration • Reporting

Elastic Beanstalk

CloudFormation

OpsWorks

Automation and configuration management

Automation Tools AWS Tools

•  CloudFormation •  Elastic Beanstalk •  OpsWorks

Third-Party Tools •  Chef •  Puppet

Monitoring & Metrics : The DevOps Pulse

• Track every resource possible • Alert on services, availability, and response times • Capture, learn, and improve

• Share access with everyone on the team • Plan metrics and monitoring into the DevOps lifecycle

If it moves graph it, or it didn’t happen

Monitoring/Logging Tools •  Logging

- Logstash Ø  Check out ELK!

-  Sumo Logic -  Splunk -  CloudWatch logs

•  Metrics - CloudWatch - Ganglia - StatsD + Graphite + Grafana - NewRelic

-  Monitoring - Nagios - Munin - Sensu - Hyperion

Discussion

Thank You