understanding aws storage...
TRANSCRIPT
© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc. © 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
Understanding AWS Storage Options
Ian Massingham, Technical Evangelist
30 April 2014
@IanMmmm
What’s in this talk?
• Scalable storage
• Inexpensive archive storage
• Persistent direct attached storage
• Turn-key gateway solution
• Customer presentation: SoundCloud
We are constantly producing more data
From all types of industries
#1 Object Storage
●○○
AMAZON S3 SIMPLE STORAGE SERVICE
99.999999999% Durability
Trillions Of Unique Customer Objects
Q4 2006
Q1 2007
Q2 2007
Q3 2007
Q4 2007
Q1 2008
Q2 2008
Q3 2008
Q4 2008
Q1 2009
Q2 2009
Q3 2009
Q4 2009
Q1 2010
Q2 2010
Q3 2010
Q4 2010
Q1 2011
Q2 2011
Q3 2011
Q4 2011
Q1 2012
Q2 2012
Q3 2012
Q4 2012
Q1 2013
Q2 2013
Q3 2013
1.5 Million+ peak transactions per second
Storage Tiers: Buckets + Unlimited Objects
Reduced Redundancy Option 99.99% saves ~20%
Spotify adds over 20,000 tracks a day - RRS
Amazon S3 Website: Static Content
Amazon S3 Continuous Cost Reduction
• 16 price reductions since launch
• TCO: On-premises vs. Amazon S3
– Can be challenging for some customers
– We can help!
1 PB raw storage
800 TB usable storage
600 TB allocated storage
400 TB written application storage
Amazon S3:
Only actual
usage is
charged
RAW Storage On-Premises vs.
Cloud Storage
Use Amazon S3 When You Need
• Unlimited storage capacity
• High durability
• Storage for backups
• Single origin store with delivery via Amazon CloudFront
AMAZON GLACIER LOW-COST ARCHIVING SERVICE
1¢ per GB / month
$120 per TB / year
99.999999999% Durability
3-5 Hours Data retrieval
STORAGE COSTS
VS
RETRIEVAL COSTS
Use Amazon Glacier When You Need
• Inexpensive or long-term archiving
• Unlimited storage capacity
• No tape museums
• No tech refresh
• High durability
Amazon S3 / Amazon Glacier Integration POLICY-BASED ARCHIVING SERVICE
Lifecycle Rule
Archive Recovery Process with Tape
+ Days or Weeks
Archive Recovery Process with AWS &
Amazon Glacier
$$
Hours
Amazon
Glacier
Amazon S3 Amazon EC2
/ HPC
Amazon
CloudFront
Generating
Business
Value
Use Amazon S3 and Amazon Glacier When You Need
• HSM in the cloud
• Archive data from Amazon S3/RRS to Amazon Glacier by policy
• Delete data from Amazon Glacier by policy
#2 Block Storage
●●○
AMAZON EBS ELASTIC BLOCK STORAGE
Ephemeral Storage
10 GB 1TB
IOPS Provisioned
4000
IOPS
Amazon EBS Snapshots
Use Amazon EBS When You Need
• Long-term persistent storage
• Frequent data changes
• Block storage for your databases – Provisioned IOPS volumes
• Filesystem for an instance NTFS, ExtFS, RAID, LVM…
• Access to raw, unformatted block-level storage
#3 Sync Volumes
●●●
AWS STORAGE GATEWAY
AWS Storage Gateway
What is AWS Storage Gateway?
• Integrates on-premise IT environments with cloud storage for departmental and remote office backup and DR
• Uses a virtual appliance that sits in customer datacenter
• Exposes compatible iSCSI interface on front end
• Stores primary data in Amazon S3, or on-premise with data backed up to Amazon S3 as Amazon EBS snapshots
Solution Overview
Run apps in the cloud using your uploaded data –
HPC/Hadoop/Analytics
2. DR
1. Offsite Backup
3. Data Mirroring
AWS Storage Gateway works with your existing backup
application and moves your data into Amazon S3 as Amazon
EBS snapshots
Run AWS Storage Gateway in Amazon EC2 and access
snapshots up to 32 TB in size
AWS Storage Gateway – Cached volumes help you create
storage volumes in the cloud and keep most recent
accessed data locally [Reduced SAN footprint for file shares] 4. Dept. File Share
Virtual Tape Library version of AWS Storage Gateway helps
customers move virtual tape data from on premises to
Amazon S3 and then to Amazon Glacier (Virtual Tape Shelf) 5. Archive/Glacier
IT’S ALL ABOUT
CHOICE PERFORMANCE-ORIENTED
COST-ORIENTED
Oliver Hookins
Backend Engineer, Media Publish/Delivery Team
SoundCloud and AWS AWS Summit Berlin 2014
title, date, 01 of 10
What is SoundCloud?
“YouTu
be
of
Audio”
Subtitle
12 hours of audio
every minute
music / podcasts
baby’s first words
environmental noise
you name it
title, date, 01 of 10
title, date, 01 of 10
title, date, 01 of 10
Subtitle
petabytes of storage hundreds of millions of files
hundreds of millions of
listeners
fastest performance possible
widest platform coverage title, date, 01 of 10
Subtitle
Horizontal Scaling Hands-Off Infrastructure
HTTP Interfaces S3 to Glacier Transition Policies
Reliability Great SDKs
title, date, 01 of 10
Thank you!
https://soundcloud.com/oliver-hookins
We’re Hiring! - https://soundcloud.com/jobs
© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc. © 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
Understanding AWS Storage Options
Ian Massingham, Technical Evangelist
30 April 2014
@IanMmmm