understanding aws storage options (stg101) | aws re:invent 2013
DESCRIPTION
With AWS you can choose the right storage service for the right use case. Given the myriad of choices, from object storage to block storage, this session will profile details and examples of some of the choices available to you, with details on real world deployments from customers using Amazon Simple Storage Service (Amazon S3), Amazon Elastic Block Store (Amazon EBS), Amazon Glacier and AWS Storage Gateway. In addition, this session will also cover all the new AWS storage features introduced in the last 12 months.TRANSCRIPT
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
Choosing the Right Data Storage Solution
Joe Lyons – Sr. Manager Global Storage
November 13, 2013
AWS Storage Options
• Scalable Storage
• Inexpensive Archive Storage
• Persistent Direct Attached Storage
• Turn-key gateway solution
• Customer Presentation: iMemories
We are constantly producing more data
From all types of industries
#1 Object Storage
●○○○○
AMAZON S3 SIMPLE STORAGE SERVICE
99.999999999% Durability
Trillions Of Unique Customer Objects
Q4 2006
Q1 2007
Q2 2007
Q3 2007
Q4 2007
Q1 2008
Q2 2008
Q3 2008
Q4 2008
Q1 2009
Q2 2009
Q3 2009
Q4 2009
Q1 2010
Q2 2010
Q3 2010
Q4 2010
Q1 2011
Q2 2011
Q3 2011
Q4 2011
Q1 2012
Q2 2012
Q3 2012
Q4 2012
Q1 2013
Q2 2013
Q3 2013
1.5 Million+ Peak Transactions Per Second
Storage Tiers: Buckets + Unlimited Objects
Reduced Redundancy Option 99.99% saves ~20%
Spotify adds over 20,000 tracks a day - RRS
Amazon S3 Website: Static Content
Amazon S3 continuous cost reduction
• 16 Price reductions since launch
• TCO: On-premises vs. Amazon S3
– Can be challenging for some customers
– We can help!
1PB Raw Storage
800TB Usable Storage
600TB Allocated Storage
400TB Written Application Storage
Amazon S3 -
Only actual
usage is
charged
RAW Storage at On perm vs. Cloud
Storage
Use Amazon S3 When You Need
• Unlimited storage capacity
• High durability
• Storage for backups
• Single Origin Store with Delivery via Amazon CloudFront
AMAZON GLACIER LOW-COST ARCHIVING SERVICE
1c Per GB / Month
$120 Per TB / Year
99.999999999% Durability
3-5 Hours Data Retrieval
STORAGE COSTS
VS
RETRIEVAL COSTS
Use Amazon Glacier When You Need
• Inexpensive/Long-term archiving
• Unlimited storage capacity
• Eliminated Tape Museums
• Eliminate Tech Refresh
• High durability
Amazon S3 / Amazon Glacier Integration POLICY-BASED ARCHIVING SERVICE
Lifecycle Rule
Archive Recovery Process with Tape
+ Days or Weeks
Archive Recovery Process with AWS &
Amazon Glacier
$$
Hours
Glacier S3 EC2
/HPC
CloudFront Generating
Business
Value
Use Amazon S3 and Amazon Glacier When You Need
• HSM in the cloud
• Archive data from Amazon S3/RRS to Amazon Glacier by policy
• Delete data from Amazon Glacier by policy
#2 Block Storage
●●○○○
#3 SYNC VOLUMES
●●●○○
AMAZON EBS ELASTIC BLOCK STORAGE
Ephemeral Storage
10 GB 1TB
IOPS Provisioned
4000
IOPS
Amazon EBS Snapshots
Use Amazon EBS When You Need
• Long-term persistent storage
• Data changes frequently
• Block storage for your databases – Provisioned IOPS volumes
• Filesystem for an instance NTFS, ExtFS, RAID, LVM…
• Access to raw, unformatted block-level storage
AWS STORAGE GATEWAY
AWS Storage Gateway
What is AWS Storage Gateway?
45 Amazon Confidential
Integrates on-prem IT environments with Cloud storage for departmental and remote office backup and DR
Utilizes a virtual appliance that sits in customer datacenter
Exposes compatible iSCSI interface on front end
Stores primary data on-AWS in Amazon S3 or on-premise with data backed-up to Amazon S3 as Amazon EBS snapshots
Solution Overview
Run apps in the cloud using your uploaded data –
HPC/Hadoop/Analytics
2. DR
1. Offsite Backup
3. Data Mirroring
AWS Storage Gateway works with your existing backup
application and moves your data into Amazon S3 as EBS
snapshots
Run AWS Storage Gateway in Amazon EC2 and access
snapshots up to 32 TB in size
AWS Storage Gateway – Cached Volumes will help you
create Storage Volumes in cloud and keep most recent
accessed data locally [Reduce SAN footprint for File Shares] 4. Dept. File Share
VTL version of AWS Storage Gateway would help customer
move data in the form of Virtual Tapes from on premise to
Amazon S3 (VTL – Virtual Tape Library) and then to Amazon
Glacier (VTS – Virtual Tape Shelf)
5. Archive/Glacier
IT’S ALL ABOUT
CHOICE PERFORMANCE-ORIENTED
COST-ORIENTED
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
Understanding AWS Storage Options
Tamara Monson, iMemories, Inc.
November 13, 2013
Our Story: History • iMemories was founded in 2006
• Cloud was in its infancy – AWS officially launched in 2006: Amazon
S3
• Handling physical assets: – VHS, Betamax, MiniDV tapes
– 16mm, 8mm film reels
– Photos, slides
• Using on-premises equipment: – VCRs, decks
– Film projectors
– Photo and slide scanners
• Focus was digitization for web/online viewing, DVD creation, archiving
2006
2008
20
10
2012
2013
iMemories' Growth
Our Story: Success • Growing infrastructure
• Reliability not as high as desired
• Less-than-ideal customer experience
• Tripled full-time resources to manage
• Costs for static, infrequently accessed assets much higher than for more-frequently accessed assets
• Rate of growth driving need for infrastructure elasticity
2006
2008
20
10
2012
2013
iMemories' Growth
8TB
0.75PB
1.5PB
Our Story: Need
Use Case
• Video and photo
converted and digital
source files
• Short-term local copy
• Access infrequently
– Disc creation
– Additional encodings
Requirements
• Durable, redundant,
secure
• Keep forever
• No or low operational
costs
• No new hardware
• Off-premises
Evaluating Options: Calculating TCO
• Operations
– Percentage FTE
– Facilities and power
– Maintenance
• Hardware
– Life span
– Number of copies
– Raw GB vs. usable GB vs.
consumed GB
0
0.2
0.4
0.6
0.8
1
1.2
Time
Raw Usable Consumed
Evaluating Options: Comparisons
Amazon S3 Cloud Files
Cloud Amazon Glacier
iMemories Evolution
Architecture 1.0
• All storage and processing on-premises (circa 2006)
• Centered on physical assets, devices, servers
• Expensive: scale by adding more
• Non-ideal customer experience
Architecture 2.x
• Solve scale challenges
• Reduce costs
• Best possible customer
experience
Storage 1.0
• On-premises
• ~1.5 PB capacity
• Limitations of scale:
– Add new hardware
– Space and power
File Mgmt
iMemories Scottsdale Data center
Digitization,
Encoding
Servers Storage Volume
Storage 2.0: Amazon Glacier
AWS Cloud
File Mgmt
iMemories Scottsdale Data Center
Digitization,
Encoding
Servers Storage Volume
Glacier
Manager
Regions
Vaults
Amazon SQS Amazon SNS
Amazon Glacier
Manager
- Interfaces with Amazon SQS
- Tracks restore/submit request
state
- Processes Submit requests in
real-time
- Processes Restore requests
once/hour
- 5% of consumed / 720 hours
- peak hourly rate
- Distribute request across 1-hr
windows
- Uses 10 MB chunks
- Performs audits 3x/week
- One discrepancy
- Zero lost files on Glacier (!!!)
- Generates internal activity reports
Glacier Manager
File Management
Digitization,
Encoding
Servers
2.x: Move to AWS
Regions
Elastic Load
Balancing Video
Files
Availability Zones
Streaming
Servers
Streaming
Server
v-ec2-east1.imemories.com
v-ec2-west2.imemories.com
File Mgmt
iMemories Scottsdale Data Center
Digitization,
Encoding
Servers
Storage
Volume
S3 Manager
AWS Cloud
2.x: Move to AWS
Region
Elastic Load
Balancing
Video
Files Upload
Web App
File Mgmt
iMemories Scottsdale Data Center
API
AWS Cloud
Amazon
SWF
u-ec2.imemories.com Video
Processing
Group
In Closing
• iMemories Story – History
– Success
– Need
• Evaluating Storage Options – Define use case and solution requirements
– Understand TCO
– Design and build for Amazon Glacier’s 4-hr download SLA, 5% restore cap
• iMemories Evolution – Amazon Glacier is first full production integration
– Migrating more functionality to cloud
– Will always have an on-premises component
Enabling families to share memories in endless ways.
http://www.imemories.com
Please give us your feedback on this
presentation
As a thank you, we will select prize
winners daily for completed surveys!
STG101