aws billing deep dive (dmg203) | aws re:invent 2013
Post on 26-Jun-2015
11.273 Views
Preview:
DESCRIPTION
TRANSCRIPT
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
DMG203 - AWS Billing Deep Dive Billing Console and Detailed Billing Reports
Serge Hairanian, Vadim Jelezniakov, AWS Billing
November 13, 2013
Agenda Billing Console Billing Reports
Cost Allocation Report
Detailed Billing Report w/
Resources & Tags
Detailed Billing Report
Monthly Report 5 levels of visibility
Agenda
Q&A at the end
The Billing Console
The Billing Console • Part of the AWS
Management Console – Accessibility – Visibility
• More responsive • New features • New look
Billing Console: Dashboard
Current Month’s Spend • Previous month comparison • Include/exclude subscriptions
- Reserved Instances - Premium Support charges
Visualized breakdown • Over 30 AWS services • See your top 4 services
by spend
Notification Center • Status of Billing Alerts
• Alarm on spending threshold
• Issues with payment methods • Outstanding charges • Free Tier eligibility • General notices regarding
your account or bill
Fully localized in 8 languages • English • German • Spanish • French • Japanese • Portuguese • Korean • Chinese
Mobile Support • Dashboard looks nice on mobile devices
• Quick view on how you are trending
Drilling down: Bills
Bills • View current & past bills • Download monthly report (csv) • Download invoices (pdf) • Product breakdown
Linked Account Activity • Totals by linked account • Drill down into each account
Consolidated Billing • Receive a single bill for all
charges incurred across all linked accounts
• Share RI discounts • Combine tiering benefits
• View & manage linked accounts
• Add additional accounts
Payer Bill Accounts 1-4
Account 1 Regular Bill
Account 3 Regular Bill
Account 2 Regular Bill
Account 4 Regular Bill
Share RI Discounts Combine Tiering
Payment
Payment History • List successful transactions • Charges and refunds match
your credit card statement • Central place to download
all invoices (pdf)
Payments Due • Amount Past Due
• Charges past net terms • Amount Due
• Charges within net terms • Outstanding Credit
• Invoice customers • Credits need to be applied to
future charges • Pay Now
• Available for failed credit card charges
Checkout Experience • Ability to add/edit cards during checkout
Billing Preferences
Billing Preferences • Invoices via email • Billing Alerts • Billing Reports • Cost Allocation Report Tags
Billing Reports • Delivered to your S3 bucket • Bucket requires specific
permissions - Sample policy
Bucket Permissions • Bucket policy generator • Allows our reporting engine to
- Confirm your bucket’s permissions - And also write to that bucket
• We won’t write to overly permissive buckets
• It is recommend to use the exact policy generated
Billing Reports • Invalid bucket name or
bucket policy
Billing Reports • Valid bucket • Enabled specific reports
Report Types • Monthly report
- Grouped by product - Monthly grain
• Detailed billing report - Grouped by product - Hourly grain
• Cost allocation report - Grouped by tag key - Monthly grain
• DBR with resources and tags - Grouped by resourceId - Hourly grain - Also has tags on each row
Manage Report Tags • Filter tags by included/excluded • Search tags by key name • Active tags
- Cost allocation report grouping - Included in your detailed billing
report with resources and tags
Finding your reports • Located in your Amazon S3 bucket
– Monthly report • <AWS account number>-aws-billing-csv-yyyy-mm.csv
– Detailed billing report • <AWS account number>-aws-billing-detailed-line-items-yyyy-mm.csv.zip
– Cost allocation report • <AWS account number>-aws-cost-allocation-yyyy-mm.csv
– DBR with resources and tags • <AWS account number>-aws-billing-detailed-line-items-with-resources-and-tags-yyyy-mm.csv.zip
Detailed Billing Reports (DBRs)
Goals
1. Eliminate ambiguity about DBRs – What are the four billing reports? – What’s inside of each report?
2. Enable you to answer cost questions – Using Monthly, Detailed, Cost Allocation and Detailed w/
Resources and Tags Reports
The Four Billing Reports
Cost Allocation Report
Detailed Billing Report w/ Resources
and Tags
Detailed Billing Report
Monthly Report
Hou
rly
Mon
thly
Advanced Data Basic Data
Joe: from developer to software tycoon
Joe: individual developer
• What is my total monthly spend?
• Which services am I spending my money on? Do I need to?
Monthly Report
Hou
rly
Advanced Data
Monthly Report
Mon
thly
Basic Data
Monthly Report
Monthly Report
Mon
thly
Basic Data • CSV format • Monthly granularity • Aggregated by product • Same data as your Invoice
<AWS account #>-aws-billing-csv-yyyy-mm.csv
Example Monthly Report RateId ItemDescription ProductCode UsageType Operation Usage Quantity Rate CostBeforeTax
805167 $0.01 per 10,000 GET and all other requests AmazonS3 Requests-Tier2 47 $0.01 $0.47800029 $0.000 per GB - data transfer in per month AWSDataTransfer DataTransfer-In-Bytes 0.00471905 $0.00 $0.00796850 $0.06 per 100,000 Amazon SNS API Requests per month thereafter AmazonSNS Requests-Tier1 6 $0.06 $0.36799999 $0.090 per GB - next 40 TB / month data transfer out AWSDataTransfer DataTransfer-Out-Bytes 0.00053194 $0.09 $0.00794470 $0.00 per request - first 1,000,000 requests AmazonEC2 CW:Requests 9878 $0.00 $0.00805187 $0.110 per GB - next 49 TB / month of storage used AmazonS3 TimedStorage-ByteHrs 0.00003376 $0.11 $0.00805207 $0.01 per 1,000 PUT, COPY, POST, or LIST requests AmazonS3 Requests-Tier1 40 $0.01 $0.40796889 First 100,000 Amazon SNS API Requests per month are free AmazonSNS USW2-Requests-Tier1 2 $0.00 $0.00795828 $0.10 per GB-month of provisioned storage AmazonEC2 EBS:VolumeUsage 67.86110956 $0.10 $6.79
18427 $0.06 per High-CPU Medium Instance (c1.medium) instance-hour AmazonEC2 BoxUsage:c1.medium RunInstances 359 $0.06 $21.54799999 $0.090 per GB - next 40 TB / month data transfer out AWSDataTransfer DataTransfer-Out-Bytes 0.47127815 $0.09 $0.04
18463 $0.03 per Small Instance (m1.small) instance-hour AmazonEC2 BoxUsage RunInstances 1436 $0.03 $43.08793970 $0.125 per GB-Month of snapshot data stored AmazonEC2 EBS:SnapshotUsage 0.54326496 $0.13 $0.07800029 $0.000 per GB - data transfer in per month AWSDataTransfer DataTransfer-In-Bytes 0.16240232 $0.00 $0.00805167 $0.01 per 10,000 GET and all other requests AmazonS3 Requests-Tier2 561311 $0.01 $0.56794470 $0.00 per request - first 1,000,000 requests AmazonEC2 CW:Requests 660 $0.00 $0.00805187 $0.110 per GB - next 49 TB / month of storage used AmazonS3 TimedStorage-ByteHrs 1.7335875 $0.11 $0.19805207 $0.01 per 1,000 PUT, COPY, POST, or LIST requests AmazonS3 Requests-Tier1 193442 $0.01 $1.93
18427 $0.06 per High-CPU Medium Instance (c1.medium) instance-hour AmazonEC2 BoxUsage:c1.medium RunInstances 1436 $0.06 $86.16794470 $0.00 per request - first 1,000,000 requests AmazonEC2 CW:Requests 932 $0.00 $0.00809956 $0.015 per Small EMR Instance (m1.small) Instance-hour ElasticMapReduce BoxUsage 1432 $0.02 $21.48
18463 $0.03 per Small Instance (m1.small) instance-hour AmazonEC2 BoxUsage RunInstances 1436 $0.03 $43.08809792 $0.12 per Extra Large EMR Instance (m1.xlarge) instance-hour ElasticMapReduce BoxUsage:m1.xlarge 404 $0.12 $48.48795287 $0.025 per LoadBalancer-hour (or partial hour) AmazonEC2 LoadBalancerUsage LoadBalancing 359 $0.03 $8.98793970 $0.125 per GB-Month of snapshot data stored AmazonEC2 EBS:SnapshotUsage 82.45148478 $0.13 $10.31794668 $0.520 per M1 Standard Extra Large (m1.xlarge) Linux/UNIX instance-hour (or part AmazonEC2 BoxUsage:m1.xlarge RunInstances 470 $0.52 $244.40805207 $0.01 per 1,000 PUT, COPY, POST, or LIST requests AmazonS3 Requests-Tier1 1693140 $0.01 $16.93809803 $0.03 per High-CPU Medium EMR Instance (c1.medium) instance-hour ElasticMapReduce BoxUsage:c1.medium 1436 $0.03 $43.08794128 $0.008 per GB Data Processed by the LoadBalancer AmazonEC2 DataProcessing-Bytes LoadBalancing 0.00005498 $0.01 $0.00799999 $0.090 per GB - next 40 TB / month data transfer out AWSDataTransfer DataTransfer-Out-Bytes 324.4072553 $0.09 $29.20
Usa
ge Q
uant
ity
Rat
e
Cos
t
Line Item Pr
oduc
t
Usa
ge T
ype
Ope
ratio
n
Example Monthly Report: Line Item
Line Item
Example Monthly Report: Line Item
Example Monthly Report: Line Item
Monthly Report: Demo
Monthly Report
Mon
thly
Basic Data
• Enabling Billing Reports • Analyzing Monthly Cost
Monthly Report: Considerations
Monthly Report
Mon
thly
Basic Data • Changing Rates result in
more Line Items, e.g. – AWS lowers cost – You purchase Reserved Capacity
• This is true for all reports! – I.e. two rates during the month
will result in 2x line items
Joe: Start-Up
Joe: Start-Up
• How much am I spending hour-by-hour?
• Are my developers turning things off?
Detailed Billing Report
Detailed Billing Report H
ourly
M
onth
ly
Basic Data
Detailed Billing Report
• CSV format • Hourly granularity • Aggregated by product • x720 or x744 more records
than Monthly Report
<AWS account #>-aws-billing-detailed-line-items-yyyy-mm.csv.zip
Detailed Billing Report H
ourly
Basic Data
Usa
ge Q
uant
ity
Blen
ded
Rat
e
Blen
ded
Cos
t
Line Item
Prod
uct
Usa
ge T
ype
Ope
ratio
n
Example Detailed Billing Report
Usa
ge Q
uant
ity
Blen
ded
Rat
e
Blen
ded
Cos
t
Line Item
Prod
uct
Usa
ge T
ype
Ope
ratio
n
Same as Monthly Report
Usa
ge Q
uant
ity
Blen
ded
Rat
e
Blen
ded
Cos
t
Line Item
Prod
uct
Usa
ge T
ype
Ope
ratio
n
Same as Monthly Report + UnBlended Rate and Cost + Hour-by-Hour
+ R
eser
ved
Inst
ance
Fla
g
+ Av
aila
bility
Zon
e
Detailed Billing Report: Demo
Analyzing Cost Hour-by-Hour Detailed Billing Report Hou
rly
Basic Data
Detailed Billing Report: Considerations
• Lots of data – use PowerPivot or import into your Data Warehouse
• Sum of allocated cost is not always equal to total on Invoice. This is due to rounding differences b/w blended and unblended cost
Detailed Billing Report Hou
rly
Basic Data
Joe: Mid-Size Company
Joe: Mid-Size Company • What is my cost by
department? • How do I do charge-backs?
• Use optional features:
– Consolidated Billing – Tagging
Company Account (“Payer” Account)
Financial Systems
Ordering Systems
Fulfillment Systems Website
Catalog Workflow Order Pipeline
Application (Tag) =
Cost Center (Tag) =
Department Accounts (“Linked” Accounts)
General Ledger Inventory 3 way
match Proxy fleet
2222 3333 6600
Stack (Tag) = Prod Alpha Beta Gamma
TAGS
1111 7784
Overview: Consolidated Billing & Tagging
Demo: Setting Up Consolidated Billing Demo: Tagging Your Resources
Optional Features: Consolidated Billing & Tagging
Now Back to Joe
• What is my cost by department?
• How do I do charge-backs?
Cost Allocation Report
Hou
rly
Mon
thly
Cost Allocation Report
Advanced Data Basic Data
Cost Allocation Report
Cost Allocation Report
Mon
thly
Advanced Data • CSV format • Monthly granularity • Aggregated by product first • Then aggregated by tags
<AWS account #>-aws-cost-allocation-yyyy-mm.csv
Remember? Monthly Report: RateId ItemDescription ProductCode UsageType Operation Usage Quantity Rate CostBeforeTax
805167 $0.01 per 10,000 GET and all other requests AmazonS3 Requests-Tier2 47 $0.01 $0.47800029 $0.000 per GB - data transfer in per month AWSDataTransfer DataTransfer-In-Bytes 0.00471905 $0.00 $0.00796850 $0.06 per 100,000 Amazon SNS API Requests per month thereafter AmazonSNS Requests-Tier1 6 $0.06 $0.36799999 $0.090 per GB - next 40 TB / month data transfer out AWSDataTransfer DataTransfer-Out-Bytes 0.00053194 $0.09 $0.00794470 $0.00 per request - first 1,000,000 requests AmazonEC2 CW:Requests 9878 $0.00 $0.00805187 $0.110 per GB - next 49 TB / month of storage used AmazonS3 TimedStorage-ByteHrs 0.00003376 $0.11 $0.00805207 $0.01 per 1,000 PUT, COPY, POST, or LIST requests AmazonS3 Requests-Tier1 40 $0.01 $0.40796889 First 100,000 Amazon SNS API Requests per month are free AmazonSNS USW2-Requests-Tier1 2 $0.00 $0.00795828 $0.10 per GB-month of provisioned storage AmazonEC2 EBS:VolumeUsage 67.86110956 $0.10 $6.79
18427 $0.06 per High-CPU Medium Instance (c1.medium) instance-hour AmazonEC2 BoxUsage:c1.medium RunInstances 359 $0.06 $21.54799999 $0.090 per GB - next 40 TB / month data transfer out AWSDataTransfer DataTransfer-Out-Bytes 0.47127815 $0.09 $0.04
18463 $0.03 per Small Instance (m1.small) instance-hour AmazonEC2 BoxUsage RunInstances 1436 $0.03 $43.08793970 $0.125 per GB-Month of snapshot data stored AmazonEC2 EBS:SnapshotUsage 0.54326496 $0.13 $0.07800029 $0.000 per GB - data transfer in per month AWSDataTransfer DataTransfer-In-Bytes 0.16240232 $0.00 $0.00805167 $0.01 per 10,000 GET and all other requests AmazonS3 Requests-Tier2 561311 $0.01 $0.56794470 $0.00 per request - first 1,000,000 requests AmazonEC2 CW:Requests 660 $0.00 $0.00805187 $0.110 per GB - next 49 TB / month of storage used AmazonS3 TimedStorage-ByteHrs 1.7335875 $0.11 $0.19805207 $0.01 per 1,000 PUT, COPY, POST, or LIST requests AmazonS3 Requests-Tier1 193442 $0.01 $1.93
18427 $0.06 per High-CPU Medium Instance (c1.medium) instance-hour AmazonEC2 BoxUsage:c1.medium RunInstances 1436 $0.06 $86.16794470 $0.00 per request - first 1,000,000 requests AmazonEC2 CW:Requests 932 $0.00 $0.00809956 $0.015 per Small EMR Instance (m1.small) Instance-hour ElasticMapReduce BoxUsage 1432 $0.02 $21.48
18463 $0.03 per Small Instance (m1.small) instance-hour AmazonEC2 BoxUsage RunInstances 1436 $0.03 $43.08809792 $0.12 per Extra Large EMR Instance (m1.xlarge) instance-hour ElasticMapReduce BoxUsage:m1.xlarge 404 $0.12 $48.48795287 $0.025 per LoadBalancer-hour (or partial hour) AmazonEC2 LoadBalancerUsage LoadBalancing 359 $0.03 $8.98793970 $0.125 per GB-Month of snapshot data stored AmazonEC2 EBS:SnapshotUsage 82.45148478 $0.13 $10.31794668 $0.520 per M1 Standard Extra Large (m1.xlarge) Linux/UNIX instance-hour (or part AmazonEC2 BoxUsage:m1.xlarge RunInstances 470 $0.52 $244.40805207 $0.01 per 1,000 PUT, COPY, POST, or LIST requests AmazonS3 Requests-Tier1 1693140 $0.01 $16.93809803 $0.03 per High-CPU Medium EMR Instance (c1.medium) instance-hour ElasticMapReduce BoxUsage:c1.medium 1436 $0.03 $43.08794128 $0.008 per GB Data Processed by the LoadBalancer AmazonEC2 DataProcessing-Bytes LoadBalancing 0.00005498 $0.01 $0.00799999 $0.090 per GB - next 40 TB / month data transfer out AWSDataTransfer DataTransfer-Out-Bytes 324.4072553 $0.09 $29.20
Usa
ge Q
uant
ity
Rat
e
Cos
t
Line Item Pr
oduc
t
Usa
ge T
ype
Ope
ratio
n
Usa
ge Q
uant
ity
Rat
e
Cos
t
Line Item
Prod
uct
Usa
ge T
ype
Ope
ratio
n
Example Cost Allocation Report
Usa
ge Q
uant
ity
Rat
e
Cos
t
Line Item
Prod
uct
Usa
ge T
ype
Ope
ratio
n
+Your Tags Same as Monthly Report
+ Av
aila
bility
Zon
e
Cost Allocation Report: Demo
Calculating Charge-Backs Cost Allocation Report
Mon
thly
Advanced Data
Charge-Backs: End Result
Cost Center AWS Product Code Cost Before Tax Tax Total Cost
Charge-Backs: Start with Cost Allocation Report
Charge-Backs: Insert Pivot Table
Defaults are usually OK
Charge-Backs: Choose Fields Rows Labels: user:Cost Center (tag) ProductCode ItemDescription Values: Sum of CostBeforeTax Sum of TaxAmount Sum of TotalCost
Charge-Backs: Finishing Touches Rename columns Filter out blank Cost Centers (Unallocated Cost)
Charge-Backs: End Result
Joe: Enterprise
Joe: Enterprise
• How much capacity do I need to reserve?
• I need to know how much each resource cost me!
DBR w/ Resources and Tags
Detailed Billing Report w/ Resources
and Tags Hou
rly
Mon
thly
Advanced Data Basic Data
DBR w/ Resources and Tags
Detailed Billing Report w/ Resources
and Tags Hou
rly
Advanced Data • CSV format • By resource • Hourly granularity • Has tags • Most detailed – lots of data!
<AWS account #>-aws-billing-detailed-line-items-with-resources-and-tags-yyyy-mm.csv.zip
Usa
ge Q
uant
ity
Blen
ded
Rat
e
Blen
ded
Cos
t
Prod
uct
Usa
ge T
ype
Ope
ratio
n
+ R
eser
ved
Inst
ance
Fla
g
+ Av
aila
bility
Zon
e
UnB
lend
ed R
ate
UnB
lend
ed C
ost
Remember? Detailed Billing Report:
Hour-by-Hour
Usa
ge Q
uant
ity
Blen
ded
Rat
e
Blen
ded
Cos
t
Prod
uct
Usa
ge T
ype
Ope
ratio
n
+ R
eser
ved
Inst
ance
Fla
g
+ Av
aila
bility
Zon
e
UnB
lend
ed R
ate
UnB
lend
ed C
ost Hour-by-Hour
Same as Detailed Billing Report + Your Tags
+ AW
S R
esou
rce
ID
DBR w/ Resources & Tags: Considerations
Detailed Billing Report w/ Resources
and Tags Hou
rly
Advanced Data
• Lots of data – pull into your
Data Warehouse to do detailed analysis
• Any change in Rates (e.g. linking/unlinking, RI subscriptions) produces even more data
Recap: Billing Reports Two Basic Reports Monthly Report
– Monthly granularity – Product level aggregation
Detailed Billing Report – Hourly granularity – Product level aggregation – x720 or x744 more lines
than Monthly Report!
Two Advanced Reports Cost Allocation Report
– Monthly granularity – Product, tag key aggregation
Detailed w/ Resources & Tags – Hourly granularity – Grouped by resource – Has tags – Lots and lots of data!
Resources “Decoding Your AWS Bill”, re:Invent 2012 http://bit.ly/aws-reinvent2012-billing
AWS Billing FAQ http://aws.amazon.com/billing/faqs/
AWS Billing Documentation http://bit.ly/aws-billing-docs
Please give us your feedback on this presentation
As a thank you, we will select prize winners daily for completed surveys!
DMG203
Appendix A How Billing Reports are generated
Billing reports: how are they generated?
Billing reports: how are they generated?
Aggregate Metering Records
Calculate Line Items
Compose Reports
What is Usage Record?
• Product • Usage Type • Operation (opt) • Quantity
EC2, S3, … Micro Instance, TimedStorage, … RunInstances, DataTransfer, … 1, 0.007, …
Billing reports: how are they generated?
Aggregate Metering Records
Calculate Line Items
Compose Reports
• Defines how much a unit of usage costs – For each AWS product (more than 30), and – For each unique usage type and operation, and – For each AWS region (9 of them)
• Examples – $0.004 per 10,000 requests (S3) in us-east-1 – $0.020 per Micro Instance instance-hour (EC2) in us-
west-2
What’s Rate?
Billing reports: how are they generated?
Aggregate Metering Records
Calculate Line Items
Compose Reports
• You define key/value pairs (up to 10 pairs per resource)
• Enable them under Manage Tags in Console!
• Tags become columns in Cost Allocation and DBR w/ Resources and Tags Reports
What are Tags?
Billing reports: how are they generated?
Aggregate Metering Records
Calculate Line Items
Compose Reports
Line Items: Two Steps to Calculate • Step 1: Count Usage
– For each unique rate (Product, Usage, Operation, Region)
• Step 2: Calculate Lowest Cost Bill – Apply Free Tier Rates – Apply Reserved Capacity Rates – Apply On-Demand Rates
Appendix B Blended vs Unblended Cost
Linked Account 1 purchased 2 RI’s with $0.00 usage rate Linked Accounts 2 and 3 run On Demand usage
Linked Account
Product Name Usage Type Operation Item Description Usage
Quantity Blended
Rate Blended
Cost
Payer Line Item AmazonEC2 BoxUsage:m1.large RunInstances $0.156 per m1.large 2396 0.156 373.77
Payer Line Item AmazonEC2 BoxUsage:m1.large RunInstances $0.00 per m1.large 1488 0 0
Linked Account Line Item
LA 1 AmazonEC2 BoxUsage:m1.large RunInstances $0.00 per m1.large 1488 0.096233 143.194704
Linked Account Line Item
LA 2 AmazonEC2 BoxUsage:m1.large RunInstances $0.00 per m1.large 1645 0.096233 158.303285
Linked Account Line Item
LA 3 AmazonEC2 BoxUsage:m1.large RunInstances $0.00 per m1.large 751 0.096233 72.270983
Blended rate
Linked Account
Product Name Usage Type Operation Item Description Usage
Quantity UnBlended
Rate UnBlended
Cost
Payer Line Item AmazonEC2 BoxUsage:m1.large RunInstances $0.156 per m1.large 2396 0.156 373.77
Payer Line Item AmazonEC2 BoxUsage:m1.large RunInstances $0.00 per m1.large 1488 0 0
Linked Account Line Item
LA 1 AmazonEC2 BoxUsage:m1.large RunInstances $0.00 per m1.large 1488 0 0
Linked Account Line Item
LA 2 AmazonEC2 BoxUsage:m1.large RunInstances $0.00 per m1.large 1645 0.156 256.62
Linked Account Line Item
LA 3 AmazonEC2 BoxUsage:m1.large RunInstances $0.00 per m1.large 751 0.156 117.15
Fixed Reserved Instance rate
On Demand rate
Linked Account 1 purchased 2 RI’s with $0.00 usage rate Linked Accounts 2 and 3 run On Demand usage
Linked Account Product Name Usage Type Usage
Quantity Blended
Rate Blended Cost UnBlended Rate
UnBlended Cost
LA 1 AmazonEC2 BoxUsage:m1.large 1488 0.096233 143.194704 0 0
LA 2 AmazonEC2 BoxUsage:m1.large 1645 0.096233 158.303285 0.156 256.62
LA 3 AmazonEC2 BoxUsage:m1.large 751 0.096233 72.270983 0.156 117.15
Linked Account 1 purchased 2 RI’s with $0.00 usage rate Linked Accounts 2 and 3 run On Demand usage
top related