1

Serve Meals, Not Ingredients

John Sweitzerjohn_sweitzer@thirdwaveinsights.com@tkdsweitz

Christine Draperchristine_draper@thirdwaveinsights.com@CristineDraper

@3rdwaveinsightshttp://www.thirdwaveinsights.com

2

Topology

Meals are topologies, nodes are key ingredients

Nodes

Software on a Node

Connections

MealIngredients

3

Emulating topologies in Chef starts with grouping nodes

Topology JSON

Minimize collisionsTopology-as-codeOptimized for ChefFaster iterations

{ "name": "inttest1", “provisioning” : { }, "services": [ { "name": "appserver-elb", "type": "load_balancer" } ], "nodes": [ { "name": "appserver" }, { "name": "dbserver" } ]}

topo.json

Chef Server

Environments

Data Bags

Cookbooks

Nodes

Chef Server

Nodes Node

Environments

Data Bags

Cookbooks

● Ways to emulate:– Environments – Tags – Node attributes– Data bags

knife-topo

4

Multi-* Enterprise - This may not be you, exactly, but...

● Multiple lines of business running business systems

● Multiple deployment across application pipeline

● Technology Adoption Model

First team First wave Broad adoption Full adoption

Asset Tracker Inventory

CustomerOrders

Human Resources

Application pipeline

Test Stage DeliveryDevelop

5

Jake's charter

More meetingsLess autonomy/controlEndless debates/escalationsTedious data gathering

Reduce costShare skillsShare content

Why increase consistency?

INCREASE CONSISTENCY ACROSS TEAMS WITHOUT DECREASING THEIR AGILITY

6

The WORK (and it is work) to increase consistency...

Gather

Organize

Decide

Change

Monitor

Waste Innovation

7

Organize: Topology-Blueprint-Inventory Pattern

Blu

eprin

tsP

ipel

ine

Topo

logi

es

Test Stage DeliverDevelop

Inve

ntor

y

Optimized Arrangements

Particular Deployment

Automate.Insights

Organizational Flow

Bill of Materials

An IT business system is a collection of IT resources delivering a business function

● Blueprints evolve

● Topologies are specialized

● Visibility encourages collaboration

8

Decide: What is expected to be common in all deployments?

Software Version Default recipe

mongodb 2.6.5 coolco_mongodb::default

nodeapp 1.0 coolco_nodeapp::deploy_from_package

nodejs 0.10.29 coolco_nodejs::default

Recipe Inventory

apt

chef_client::config

coolco_handlers::default

coolco_mongodb::default

coolco_nodeapp::deploy_from_package

coolco_nodejs::default

Software Attribute Value

mongodb xxx yyy

xxx yyy

xxx yyy

Software Attribute Value

nodeapp nodeapp.name asset_tracker

nodeapp.user jake

nodeapp.dir /home/automation

RunlistNode attributes

9

Decide: Look across blueprints...

Automate. Insights

● Node Types● Software and Version● Attribute Values● Recipes● Cookbooks

Software attribute value by blueprint

10

Change – All does not need to change at once

Blu

eprin

tsP

ipel

ine

Topo

logi

es

Test Stage DeliverDevelop

Inve

ntor

y

Optimized Arrangements

Particular Deployment

Organizational Flow

Bill of Materials

Chef Server

Actual IT Resources

Environments

Data Bags

CookbooksN

odes

Roles

● Prime● Synchronize● Local Override

Sta

ged

Ch

ang

es

nodejs 0.10.28 to 0.10.30

11

Gathering and Changing details in Chef Server

knife topo

export YourTopology

node1 node2 node3 …

> YourTopology.jsonTopology

JSONChef Server

knife-topo

Automate.Insights

knife topo

import YourTopology

knife topo

create YourTopology

Create blueprints from nodesAlign topologies with blueprint

Propagate changes to Chef server

Gather

Organize

Decide

Change

Monitor

12

Fred's Scenario

Meanwhile Shadow IT has sprung up in various lines of business, using a range of providers

Fred's one of the technical leads who's been successful bringing consistency across CoolCo's

core business systems...

CoolCo wants Fred to help the Lines of Business adopt Chef and become more consistent in their

delivery, without impacting their agility

13

Fred meets with one of the lines of business to understand what they have...

appserver AMI

appserverauto scaling groupElastic Load

Balancer(appserver-

elb)

dbserver

AWS cloud

Launch Configuration

db url

protocol & port

instanceprotocol & port

Resources": { "elbappserverelb": { "Type": "AWS::ElasticLoadBalancing::LoadBalancer", "Properties": { "AvailabilityZones": [ "us-west-2c", "us-west-2b", "us-west-2a" ], "Listeners": [ { "InstancePort": "3001", "LoadBalancerPort": "3001", "Protocol": "HTTP", "InstanceProtocol": "HTTP" } ]} },...

Create multiple test topologies rapidly & consistentlyDelete topologies when done

CloudFormation Template

CloudFormation Stacks

CloudFormation

AWS Icons: http://aws.amazon.com/architecture/icons/

Hook-Em... a recently-developed customer conversion system

14

… and what their challenges are

app+

nodejs

mongodb

application & middleware

"DBServer": { "Type" : "AWS::EC2::Instance", "Properties" : { "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [ "#!/bin/bash\n", "yum update -y aws-cfn-bootstrap\n", <snip 20 lines> "yum -y install mongo-10gen-server > /tmp/yum-mongo.log 2>&1\n", <snip 50 lines> "/etc/init.d/mongod start > /tmp/mongod-start.log 2>&1\n", <snip 20 lines> "/opt/aws/bin/cfn-signal -e 0 -r \"Mongo setup complete\" '", { "Ref" : "WaitHandleMongo" }, "'\n" ] ] } }, ... } ... }}}

It's hard to configure & update the softwareOur admin who created our templates has left...

15

A shared goal emerges...

Deployer Interface that makes it easy to customize and update the topology

Topology-as-codeDeployment lifecycle actionsFull stack provisioning

Reuse organization cookbooks & practices

Chef Server

Git

TopologyJSON

chef-provisioning

...Now to make it happen!

ProvisioningRecipes

Convert from existing templates

16

Chef Server

What are the steps?For the Hook'Em business system:

1) Set up Hook'Em System Inventory in Automate.Insights

2) Create a Blueprint based on the existing CloudFormation template

3) Create Topologies and provision them using generated recipes

app +nodejs

mongodbapplication &

middleware

infrastructure

TopologyJSON

chef-provisioning

Automate. Insights

Provisioning recipes

● System inventory● Existing template

17

1 – Setup Hook'Em System Inventory

Software Version Default recipe

mongodb 2.6.5 coolco_mongodb::default

nodeapp 1.0 coolco_nodeapp::deploy_from_package

nodejs 0.10.29 coolco_nodejs::default

Recipe Inventory

apt

chef_client::config

coolco_handlers::default

coolco_mongodb::default

coolco_nodeapp::deploy_from_package

coolco_nodejs::default

Software Attribute Value

mongodb xxx yyy

xxx yyy

xxx yyy

Software Attribute Value

nodeapp nodeapp.name hookem

nodeapp.user automation

nodeapp.dir /home/automation

RunlistNode attributes

Fred primes the System Inventory in Automate.Insights with CoolCo's standard recipes and recommended attribute values for the software in the HookEm System.

System Inventory

Automate. Insights

mongodb

nodeappnodejsBlueprint

18

2 - Create a Blueprint for Hook'Em

Topology JSON Automate.

Insights

The Hook'Em team converts the existing CloudFormation template using topo-provision, and uses it and the System Inventory to create a Blueprint and Topologies in Automate.Insights

topo-provision

CloudFormationtemplate

Software inventory

Full-StackTopology JSON

19

3 - Generate recipes & provision the full stack topology

chef-provisioning can handle network resources too!

app +nodejs

mongodbapplication &

middleware

infrastructure

Full StackTopology JSON

topo-provision

Chef-Provisioning recipes

chef-provisioning

with_driver “aws”

load_balancer ... do ... end

machine … do … end

aws_launch_config … do ... end

aws_autoscaling_group … do ... end

● Deploy● Undeploy

load_balancer "appserver-elb" do load_balancer_options({ :listeners=> [{ :port=>3001, :protocol=>"HTTP", :instance_port=>3001, :instance_protocol=>"HTTP" }], :availability_zones=> ["us-west-2c", "us-west-2a", "us-west-2b"]})end

machine "dbserver" do run_list ["recipe[coolco::db]"] attribute 'topo', {"node_type"=>"dbserver", "name"=>"test"} end

deploy.rb:Supports deployment lifecycle actionsSensitive to resource relationships

Using topo-provision, the Hook'Em team generate topology provisioning recipes that enable chef-provisioning to provision the full stack.

20

Summary of transformation

Infrastructure templates

Stack create and delete

Customization of parameters

Full stack Topology JSONConversion from existing templates

Topology deploy and un-deployIntegrated provisioning and configuration management

Customization of attributes & run listLine-of-sight across topologiesTool assistance for change propagation

TopologyTemplate

Deployment

Deployer Interface

As-Is To-Be

Practices Reuse organization cookbooksBenefit from Chef test tools

CloudFormation Template

CloudFormation Stacks

CloudFormation

Topology JSON

ChefAutomate.Insights

21

A Cloud-neutral topology template has wider value to the Chef community

TopologyJSON

chef-provisioning

ProvisioningRecipesConvert from other

formatsIncrease portability

CloudFormation

Terraform

OASIS TOSCA

Home-grownIntegrate with multiple tools, from Chef and others

Adoption of a Chef-oriented topology template that is not tied to one Cloud Provider helps increase portability and tool integration going forward.

Deploy to multiple providers

This is a vision with proof points for CloudFormation, AWS and Vagrant.

Test Kitchen

kitchen.yml

Automate. Insights

22

Thank you!

Tools:http://www.thirdwaveinsights.com/automateinsightshttps://github.com/christinedraper/topo-provisionhttps://github.com/christinedraper/knife-topo

Find Out MoreSpeakers:john_sweitzer@thirdwaveinsights.com@tkdsweitzchristine_draper@thirdwaveinsights.com@CristineDraper