(ent207) creating a culture of cost management in your organization | aws re:invent 2014
DESCRIPTION
As your organization increases its AWS usage, budget owners and users demand new levels of cost visibility. This session explains how scaled organizations control and optimize their AWS spending as usage increases across multiple product teams. Attendees walk away with a strategy for driving cost effective behavior across the entire organization, from engineering to finance. Sponsored by Cloudability. Topics include: - Maintaining cost oversight while giving autonomy to individual teams - Allocating costs across dozens or hundreds of accounts or applications - Creating accountability around spending habits and waste - Tying cloud spending to the bottom lineTRANSCRIPT
![Page 1: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/1.jpg)
© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
November 12, 2014 | Las Vegas
ENT207
Creating a Culture of Cost Management
in Your Organization
J.R. Storment, Cloudability
![Page 2: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/2.jpg)
0
45
90
135
180
225
De
v/T
est
PO
C
Ap
p #
1
Ap
p #
3
Ap
p #
2
Exec team starts
really caringL
oa
d te
st
A story of growth
![Page 3: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/3.jpg)
Why does this need to be cultural?
![Page 4: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/4.jpg)
The cloud is no longer one person’s responsibility…
Engineers Finance Operations Capacity Execs
![Page 5: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/5.jpg)
How do you build the culture?• Put data in the hands of the people
• Enact policies and evangelize best practices
• Incentivize good behavior
![Page 6: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/6.jpg)
Rolling out a cost management program
• Stage I: Cost visibility for all (emails, alerts, dashboards)
• Stage II: Allocating costs to departments (tags, linked accounts)
• Stage III: Efficiently using only what you need (underutilized, rightsizing)
• Stage IV: Saving by lowering hourly costs (Reserved and Spot Instances)
• Stage V: Tying spending to the bottom line (unit cost)
![Page 7: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/7.jpg)
Visibility
Allocation Efficiency
Savings
Unit Cost
![Page 8: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/8.jpg)
I. Cost visibilityVisibility
![Page 9: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/9.jpg)
Two types of people
Incur costs Watch costs
![Page 10: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/10.jpg)
Two types of people
Incur costs Watch costs
![Page 11: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/11.jpg)
Two types of people
Incur costs Watch costs
![Page 12: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/12.jpg)
Two types of people
Incur costs Watch costs
![Page 13: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/13.jpg)
Tips for cost visibility
• Get each stakeholder the spending fundamentals daily
• Let each teams see other teams’ spending habits
• Create broadly available dashboards
![Page 14: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/14.jpg)
![Page 15: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/15.jpg)
Case study: Enterprise with 100s of apps
"Our goal is to put control of AWS resources in each product
team’s hands. We want to get out of the way but at the same
time we still need oversight to stop overspending.”
- Split the detailed billing reports into 750 different views broken
down by BUs/applications/environments/etc.
- Daily spending updates
800 users get daily emails with just their spending
- Oversight for reporting and control
Execs/finance see the total spending in a simple weekly digest
![Page 16: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/16.jpg)
II. Allocation
Allocation
![Page 17: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/17.jpg)
First, get everyone involved
๏ Multiple major business units
๏ Lots of products or cost
centers
๏ Thousands of shared AWS
resources
๏ Distributed teams using AWS
![Page 18: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/18.jpg)
Tools for splitting up resources
• Tags are highly flexible, but 100% coverage is difficult due to compliance
• Linked accounts offer clean chargeback but limit reporting options
![Page 19: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/19.jpg)
![Page 20: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/20.jpg)
Pro-tips: Allocating costs
• Get consensus on the taxonomy (but let Finance drive)
• Define 2–3 mandatory tags like “project” or
“environment”
• Consider a “tag or terminate” rule to enforce compliance
![Page 21: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/21.jpg)
- Split the accounts based on product and environment
- SAP data as source of truth for grouping metadata:
Cost center, product group, business unit, etc
- Chargeback aligned to finance’s view of the business,
not just the operational linked account structure
Case study: The enterprise with 500 linked accounts,
no oversight, hundreds of influencers
![Page 22: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/22.jpg)
III. Efficiency
Efficiency
![Page 23: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/23.jpg)
168hours in a
week
![Page 24: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/24.jpg)
nights &
weekends
108
![Page 25: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/25.jpg)
Don’t run the cloud like a data center:
65% of the hours in a month are nights and
weekends
![Page 26: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/26.jpg)
Three easy wins for Amazon EC2 efficiency…
![Page 27: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/27.jpg)
Low CPU, low bandwidth, low disk I/O, >1 day old | This example: Save $1,682 per week
Turn off underutilized instances
![Page 28: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/28.jpg)
Schedule dev/test downtime
![Page 29: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/29.jpg)
Find M1s that could be upgraded
1. Newer instance families are faster & cheaper
but have smaller SSDs
2. Look for instances that have Amazon Elastic Block Store
(EBS) devices attached and low ephemeral disk access
3. Move M1 to M3 to save $0.03-$0.10 an hour
Source: http://aws.amazon.com/ec2/pricing/
![Page 30: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/30.jpg)
Get alerts when thresholds are crossed
![Page 31: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/31.jpg)
Tips for encouraging efficient behavior
• Automate weekly waste reporting to each team
• Gamify clean-up by creating a visible leaderboard
• Do a monthly whole-company waste review
For more check out:
“MegaRun: Behind the 156,000 Core HPC Run”
on Friday at 1pm
![Page 32: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/32.jpg)
Case study: High traffic site with decentralized ops
avg hourly node cost ($)
x
avg node uptime (%)
x
sq rt inverse of CPU util (%)
x
avg node running life (hrs)
x
avg daily unique nodes^1.3
Lots of developers using dev/test instances
and forgetting to turn them off
Used Cloudability API data to normalize
dev/test environment “health”
Developed an algorithm to determine
health/sickness of any dev/test
environment
Midge’s law
![Page 33: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/33.jpg)
IV. SavingsSavings
![Page 34: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/34.jpg)
Spot vs. Reserved Instances?
Reservations require no engineering,
enabling finance to autonomously reduce costs
![Page 35: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/35.jpg)
Don’t fear the reservation
• Prices drop, but not fast enough to erode savings
• With usage over 50% three-year RIs always wonSource: http://blog.cloudability.com/cloud-cost-war-shouldnt-stop-buying-reserved-instances/
![Page 36: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/36.jpg)
Buy based on current hourly data
Don’t wait to buy reservations until you’ve completed the
perfect analysis—you’ll waste more money than you save
![Page 37: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/37.jpg)
Tips for reservation efficiency
•Appoint a person to own reservation purchases—
often a tech-minded finance person
•Start with an immediate small and uncontroversial buy
•Buy iteratively on a schedule that you hold sacred
![Page 38: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/38.jpg)
Case study: Buying for 12,500+ daily instances
25th of the month:
Central team drives decisions with minimal input from teams
26th of the month:
Modifications made each month ahead of purchase
28th of the month:
Purchase based on the current month’s utilization data
![Page 39: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/39.jpg)
For more, check out:
Iterating Your Way to 95% Reserved Instances
(ENT 203)
Today at 4:30pm
![Page 40: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/40.jpg)
V. Unit cost
Unit cost
![Page 41: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/41.jpg)
Why does unit cost matter?
0
45
90
135
180
225
De
v/T
est
PO
C
Ap
p #
1
Ap
p #
3
Ap
p #
2
Exec team starts
really caring
Hic
cu
p
![Page 42: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/42.jpg)
An increasing bill isn’t necessarily a bad thing
0
45
90
135
180
225
0
45
90
135
180
225
The bill still
goes up!Exec team starts
really caring
Sta
ge
III
Efficiency
Sta
ge
IV
RIs
![Page 43: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/43.jpg)
@cloudabilit
y
Unit cost is
cost per X
![Page 44: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/44.jpg)
@cloudabilit
y
cost per subscriber
cost per pageview
cost per customer
cost per api call
![Page 45: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/45.jpg)
0
30
60
90
120
150
Focus on reducing unit cost, even as total cost grows
Unit cost Total cost
![Page 46: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/46.jpg)
@cloudabilit
y
Unit cost =
total cost / business
metric
![Page 47: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/47.jpg)
@cloudabilit
y
$1000 / 1000 customers
=
$1 unit cost
![Page 48: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/48.jpg)
Case study: Scaled web business pushing unit cost
• Product owners track unit cost daily tying spending to revenue
• Operations examines infrastructure changes in terms of unit cost impact
• Finance reports monthly & quarterly on margin impacts based on unit cost
• VPs set goals on unit cost and worry less about setting arbitrary total cost
goals
Shortcut: Track your AWS bill as a percentage of revenue
![Page 49: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/49.jpg)
Getting started on your culture
![Page 50: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/50.jpg)
Things to do right after re:InventI. Visibility: Give stakeholders a daily view into spending
II. Allocation: Put together a taxonomy with your finance team and start
splitting linked accounts and tagging
III.Efficiency: Look at underutilized instances (low CPU + BW + disk) and M1
to M3 candidates (disk access + attached storage)
IV.Hourly savings: Make an immediate small and uncontroversial RI buy
V.Unit cost: Determine your top-line business metric, divide costs by them
![Page 51: (ENT207) Creating a Culture of Cost Management in Your Organization | AWS re:Invent 2014](https://reader033.vdocuments.mx/reader033/viewer/2022060121/55935ddb1a28ab59648b46a3/html5/thumbnails/51.jpg)
Start a cost management group
to share best practices within your company
and encourage accountability