November 13, 2014 | Las Vegas, NV

Jonathan Quail / Cliff McCollum, FillZ.com

Story background…

https://www.flickr.com/photos/stevecorey/12844122413

Debian CentOS Amazon

Linux

https://www.flickr.com/photos/bdesham/2432400623

https://www.flickr.com/photos/beigephotos/5334612

https://www.flickr.com/photos/colorhome/4033895230

Monitoring

Provisioning

Logging

Deployment Modeling

Configuration management Auto Scaling Access management

Resource management

github.com/aws/opsworks-cookbooks

supermarket.getchef.com

Hot

tip!

Hot

tip!

• Application servers

• Monitoring servers

• Graphite/dashboards

• Splunk cluster

– fillz-app::setup, fillz-app::configure, fillz-app::deploy

– fillz-base::setup, fillz-base::configure

Setup Configure Deploy Undeploy Shutdown

$ easy_deploy.py deploy –-application=myapp instances –-hosts=host1,host2 \

-–stack-name=my-test-stack --comment=“Deploying myapp to host1/host2”

$ easy_deploy.py deploy –-application=myapp all –-exclude-hosts=host1,host2 \

-–layer-name=backend-servers -–stack-name=my-test-stack \

--comment=“Deploying myapp to rest of the stack”

$ easy_deploy.py deploy –-application=myapp rolling –-layer-name=apiservers \

-–stack-name=my-api-stack --comment=“Deploying myapp behind ELB servers”

Code is available from the Github repo

By permission of Leigh Rubin and Creators Syndicate, Inc.

all_hosts = {}

all_hostgroups = {}

# Gets all (online only) instances in OpsWorks stack

all_instances = search(:node, 'role:*')

all_instances.each do |instance|

node_hostgroups = {}

# add a hostgroup for each layer the instance is in

instance['opsworks']['layers'].each do |layer_id, layer_info|

node_hostgroups[layer_id] = layer_info['name']

end

# add a hostgroup for each availability zone

node_hostgroups[instance['availability_zone']] =

instance['availability_zone']

all_hosts[instance['hostname']] = {

:hostgroups => node_hostgroups,

:private_ip => instance['private_ip']

}

all_hostgroups = all_hostgroups.merge(node_hostgroups)

end

template "/etc/nagios/conf.d/hostgroups.cfg" do

source 'hostgroups.cfg.erb'

owner 'nagios'

group 'nagios'

mode 00640

variables(

:hostgroups => all_hostgroups

)

notifies :reload, 'service[nagios]'

backup 0

end

template "/etc/nagios/conf.d/hosts.cfg" do

source 'hosts.cfg.erb'

owner 'nagios'

group 'nagios'

mode 00640

variables(

:hosts => all_hosts

)

notifies :reload, 'service[nagios]'

backup 0

end

Hot

tip!

<infomercial>

</infomercial>

Wait…Remember that deployment tool? If you call in

the next 20 minutes…it does patching too.

No “4 easy payments of $29.95” this part is open source too

$ easy_deploy.py update --amazon-linux-release=2014.09 --allow-reboots \

instances –-hosts=host1,host2 -–stack-name=my-test-stack \

--comment=“Upgrading host1/host2 to 2014.09 release”

$ easy_deploy.py update all –-exclude-hosts=host1 \

-–layer-name=backend-servers -–stack-name=my-test-stack \

--comment=“Patching layer for ++openssl vulnerability”

$ easy_deploy.py update --allow-reboots rolling \

-–layer-name=apiservers –-stack-name=my-api-stack \

--comment=“Patching api behind ELB servers and reboot for kernel”

“<stack> <name>”

“ bastion-dev fillz-graphite”

“ bastion-dev localhost:8000:fillz-graphite:80”

https://www.flickr.com/photos/wfryer/716965391

/var/log/aws/opsworks/opsworks-agent.log

/var/log/aws/opsworks/opsworks-agent.keep_alive.log

/var/log/aws/opsworks/opsworks-agent.process_command.log

http://blogs.aws.amazon.com/application-

management/post/TxTX72HFKVS9W9/Using-Amazon-CloudWatch-

Logs-with-AWS-OpsWorks

BEFORE AFTER

Time to deploy a new

server2 days + 2 hours 25 minutes

Time to deploy a new

software release2 hours 2 minutes

After-hours alerts 6 / week < 1 / week

BEFORE AFTER

Scheduled downtime per

month5 hours 0 minutes

Software deployment rate 2 / month 2 / week

http://github.com/aws/opsworks-cookbooks

http://supermarket.getchef.com

http://blogs.aws.amazon.com/application-

management/post/TxZX2UI4YSBW3T/Customizing-AWS-OpsWorks-with-Attributes

http://blogs.aws.amazon.com/application-

management/post/TxTX72HFKVS9W9/Using-Amazon-CloudWatch-Logs-with-AWS-

OpsWorks

https://github.com/awslabs/reinvent2014-scalable-site-management

http://bit.ly/awsevals