sizing up the aws storage options -...
TRANSCRIPT
EDITOR’S NOTE STUDY THE CLASSES OF AWS STORAGE
NEARLINE STORAGE COMES IN FROM THE COLD
AWS PROVIDES A RANGE OF COLD STORAGE OPTIONS
Sizing Up the AWS Storage OptionsCloud storage can solve an organization’s data problems, but cost, latency requirements, storage classes and other factors need to be taken into account.
HOME
EDITOR’S NOTE
STUDY THE CLASSES
OF AWS STORAGE
NEARLINE STORAGE
COMES IN FROM
THE COLD
AWS PROVIDES
A RANGE OF COLD
STORAGE OPTIONS
SIZING UP THE AWS STORAGE OPTIONS2
EDITOR’SNOTE
Cloud Storage Questions to Ask
As cloud computing emerged, business decision makers needed to determine just how comfortable they were with putting important data beyond the walls of a data center to which they had physical access. They needed to work through questions like: Is our business ready for the cloud? And is the cloud ready for us?
Some of those same questions remain, even as cloud becomes an increasingly impor-tant part of many businesses. This handbook approaches the issues with a focus on cloud storage.
Cloud expert Ofir Nachmani provides a detailed overview of the multiple storage options available to AWS users. He breaks down how object storage differs from block storage and when an archival storage option might make the most sense.
Those differences are important, Nachmani notes, and sorting through them will likely matter in whether your cloud storage effort
meets performance expectations and falls within your budget. Included in his assessment are points of comparison between storage ser-vices offered by AWS, Google Cloud Platform and Microsoft Azure.
Also, this handbook offers suggestions for how to choose a cloud storage provider. IT ana-lyst Mike Matchett shares pointers on what to look for when evaluating nearline storage options and how to avoid common missteps in selecting a provider.
Lastly, TechTarget’s Stephen Bigelow writes about cold storage in AWS. He offers tips on using the AWS Direct Connect and AWS Import/Export Snowball services and how to do so at minimal expense. n
Phil Sweeney Senior Managing Editor
Data Center and Virtualization GroupTechTarget
HOME
EDITOR’S NOTE
STUDY THE CLASSES
OF AWS STORAGE
NEARLINE STORAGE
COMES IN FROM
THE COLD
AWS PROVIDES
A RANGE OF COLD
STORAGE OPTIONS
SIZING UP THE AWS STORAGE OPTIONS3
STORAGE CLASSES
Study the Classes of AWS Storage
Without a solid understanding of the storage options AWS offers, it will be difficult to ensure your choice meets application and performance requirements. You’ll need to iden-tify features that strike an appropriate balance between performance and cost. In general, AWS storage can be divided into three classes:
n block device storage (Amazon Elastic Block Store);
n object storage (Amazon Simple Storage Service, or S3); and
n archival storage (Glacier).
Each storage type offers certain capabilities, so it is worth breaking down the distinctions.
AMAZON ELASTIC BLOCK STORE
Amazon’s EBS is a low-latency, persistent stor-age offering. Because it is highly available in
the same availability zone, a component failure doesn’t lead to data loss. Amazon EBS volumes are used with Amazon Elastic Compute Cloud instances.
Amazon EBS volumes are offered with solid state drives (SSD) and hard disk drives (HDD). Within those types are two key options to consider.
SSD offers low-latency results, durability and consistent I/O performance. The performance of SSD volumes is measured by their IOPS (I/O operations per second). To meet further per-formance versus cost requirements, Amazon offers Elastic Block Store SSD volumes in two types:
EBS Provisioned IOPS SSD. These volumes are used for applications that are highly sensitive to latency. For these apps, performance and consistency are crucial. An EBS Provisioned IOPS SSD volume allows you to set a preferred
HOME
EDITOR’S NOTE
STUDY THE CLASSES
OF AWS STORAGE
NEARLINE STORAGE
COMES IN FROM
THE COLD
AWS PROVIDES
A RANGE OF COLD
STORAGE OPTIONS
SIZING UP THE AWS STORAGE OPTIONS4
STORAGE CLASSES
IOPS level and ensures that you always achieve at least 90% of that level. The size of an EBS Provisioned IOPS volume can fall between 4 GB and 16 TB with up to 20,000 IOPS per volume. Each gigabyte of the volume provides around 30 IOPS. Typical uses of EBS Provisioned IOPS SSD volumes would be I/O intensive databases and critical business applications.
EBS General Purpose SSD. These volumes, referred to as GP2, fit the majority of use cases. Each gigabyte of storage provides up to 3 IOPS, and the size of GP2 volumes can range between 1 GB and 16 TB. Each GP2 volume comes up with baseline performance IOPS, which allows it to burst its performance up to 3,000 IOPS for 30 minutes, ensuring a fast boot-up cycle. The typical uses for EBS General Purpose SSD volumes are system boot volume or root vol-ume, low-latency interactive apps and transac-tional workloads.
HDD volumes, meanwhile, are low-cost, low-latency magnetic volumes. In these, per-formance is measured in megabytes per second rather than IOPS.
Throughput Optimized HDD. These low-latency magnetic storage volumes are used by sequen-tial I/O workloads. The volume size for this can vary from 500 GB to 16 TB with a maxi-mum throughput of 500 megabytes per second (MBps). Uses for EBS Throughput Optimized HDD volumes include big data tasks, log pro-cessing and data warehouses.
Cold HDD. These volumes are designed to handle sequential data that doesn’t need to be instantly available. They are useful for when performance is not a big concern. Cold HDD cannot be used for bootable drives. Volume size varies from 500 GB to 16 TB with a maximum throughput of 250 MBps. Typically, EBS Cold HDD volumes would be used for data that’s accessed infrequently.
AMAZON SIMPLE STORAGE SERVICE
S3 is a highly available, durable, secure object storage service offered by AWS. It can be accessed via API calls from anywhere in the world. There is no upfront storage commit-ment, and S3 and can be integrated with other
HOME
EDITOR’S NOTE
STUDY THE CLASSES
OF AWS STORAGE
NEARLINE STORAGE
COMES IN FROM
THE COLD
AWS PROVIDES
A RANGE OF COLD
STORAGE OPTIONS
SIZING UP THE AWS STORAGE OPTIONS5
STORAGE CLASSES
Amazon services. Features include cross-region replication, lifecycle rules, event notifications, versioning, encryptions and flexible storage options.
Amazon offers storage options within S3. Amazon S3 Standard provides availability, durability and security with low-latency deliv-
ery for frequently accessed objects. The dura-bility of the objects is set to 9.999999999% and a yearlong availability of 99.99%. It can support encryption of data in transit and at rest. The typical uses for S3 Standard Storage are simple (static) websites, distribution of rich content (such as video) and big data analytics.
Amazon S3 Standard Infrequent Access pro-vides the same performance, availability and durability as S3 Standard, but it also sets up the right combination of cost and performance for objects that may not be used frequently. It works in storage cost and per-gigabyte retrieval
cost. This class applies to objects and can stay in the same bucket along with S3 Standard objects. S3 Standard Infrequent Access is often used for backups and data stored for disaster recovery.
AMAZON GLACIER
Amazon Glacier is a low-cost storage option. It provides durable, secure, long-term storage that’s well-suited for archiving and backup.
Glacier is best deployed where it’s accept-able for data retrieval times to be hours long. The service works well with Amazon S3, where automatic lifecycle policies allow an organiza-tion to store data.
The typical uses for Amazon Glacier are long-term backups and logs. It can also be used to store compliance-related data, which in cer-tain cases needs to be kept for years.
COMPARING STORAGE OPTIONS
AWS isn’t alone in cloud storage. Rival public cloud providers Microsoft Azure and Google Cloud Platform (GCP) also offer multiple
Amazon S3 Standard can support encryption of data in transit and at rest.
HOME
EDITOR’S NOTE
STUDY THE CLASSES
OF AWS STORAGE
NEARLINE STORAGE
COMES IN FROM
THE COLD
AWS PROVIDES
A RANGE OF COLD
STORAGE OPTIONS
SIZING UP THE AWS STORAGE OPTIONS6
STORAGE CLASSES
storage options. There are some key points of comparison between these services.
With object storage, AWS S3 offers users multiple options and touts availability of 99.99%. Azure’s Blob service offers Blob Zone Redundant Storage, Locally Redundant Storage and Cool Blob options. It assures availability at 99.9%. GCP’s Cloud Storage service offers Standard and Durable Reduced Availability object storage options. It lists availability at 99.9%.
As for block storage, Amazon’s EBS offers SSD and HDD options. Maximum disk size is 16 TB.
Azure Disks is Microsoft’s SSD block stor-age service, for which customers pay based on what they use, regardless of the size pro-visioned. Azure gives users the ability to map shared drives. The number of disks that can be attached to a server depends on VM size. GCP’s Persistent Disks block storage service includes SSD and HDD options. Maximum disk size is 64 TB.
With archive storage, AWS Glacier is a low-cost option ($0.007 per GB per month) with a retrieval time of four hours. GCP’s Nearline
service is low-cost ($0.01 per GB per month), and its retrieval time is measured in seconds. Azure does not have a comparable offering in the archival storage category.
Be sure to look at your organization’s storage selection from multiple perspectives. Specifi-cally, look at the following:
Storage Size. Size plays a vital role in perfor-mance. For block storage options across multiple cloud providers, you can see how IOPS or throughput of a volume or disk linearly increases with the storage size of the disk.
Access Frequency. Each cloud provider has multiple storage options based on your access frequency. If you have data that’s infrequently accessed, you can use different storage classes to save money while achieving the same avail-ability, durability and security.
Performance. If you are looking for predictable, low-latency, consistent performance levels for your applications, you can customize your storage options to fit those requirements.
HOME
EDITOR’S NOTE
STUDY THE CLASSES
OF AWS STORAGE
NEARLINE STORAGE
COMES IN FROM
THE COLD
AWS PROVIDES
A RANGE OF COLD
STORAGE OPTIONS
SIZING UP THE AWS STORAGE OPTIONS7
STORAGE CLASSES
Availability. Consider your data availability requirements. For example, if you are willing to compromise on availability, you should find a low-cost storage service.
As long as you have your business and appli-cation requirements laid out correctly, you’ll
likely have several good storage options avail-able. These choices will let you juggle perfor-mance, cost, durability and consistency. These storage options can be combined to achieve practical storage solutions for your workloads, allowing you to transform from one storage class to another. —Ofir Nachmani
HOME
EDITOR’S NOTE
STUDY THE CLASSES
OF AWS STORAGE
NEARLINE STORAGE
COMES IN FROM
THE COLD
AWS PROVIDES
A RANGE OF COLD
STORAGE OPTIONS
SIZING UP THE AWS STORAGE OPTIONS8
NEARLINE STORAGE
Nearline Storage Comes in From the Cold
It was just a few years ago that many CIOs and IT directors were highly skeptical about public nearline cloud storage. Security, gover-nance and cost were big concerns. Bandwidth, migration, access control and provider stabil-ity also impeded the long-predicted journey to the cloud. Most concerns have since been addressed to the point where some new genera-tions of on-premises storage arrays are directly tiering to public cloud storage.
Cloud storage has evolved. It’s now to the point where it can be used in production, not just as a remote backup archive. Network latencies still prevent I/O-hungry data center workloads from using the cloud as primary storage, but hybrid architectures that can move workloads to the cloud using VMs and con-tainers are becoming more the norm than the
exception. Still, many data centers are not yet using nearline cloud storage for purposes other than cold storage.
To nail down good use cases, look at the various tiers of storage each cloud service pro-vider offers. You should consider each tier as a possible plug-in resource in your architecture. Ask vendors how simple—and costly—it is to transfer data between cloud storage services directly, as this can make it easier to shift data around should needs change.
Be sure to understand the costs involved in both storing data over time and access-ing it when necessary. Even though there may be access costs, they might be well within expected budgets. And be sure to pay close attention to access latencies; backups that take hours to recall may not provide satisfactory
Cloud storage has evolved. It’s now to the point where it can be used in production, not just as a remote backup archive.
HOME
EDITOR’S NOTE
STUDY THE CLASSES
OF AWS STORAGE
NEARLINE STORAGE
COMES IN FROM
THE COLD
AWS PROVIDES
A RANGE OF COLD
STORAGE OPTIONS
SIZING UP THE AWS STORAGE OPTIONS9
NEARLINE STORAGE
levels of business continuity.With AWS and competing providers lower-
ing prices, nearline cloud storage users should position themselves to take advantage of the lowest costs available. It isn’t easy to migrate massive amounts of data from one provider to another, nor is it necessarily cheap. This results in some friction-based lock-in, which is something to be aware of when committing large repositories to any one service provider.
UNITING CLOUD WITH ARRAYS
New storage arrays are going beyond the premises with built-in native cloud tiers. For example, Microsoft’s StorSimple cloud storage service for inactive data is linked with Azure.
If you plan to use nearline cloud storage with an array, there are a number of capabilities to evaluate:
Native cloud tiering. The array can use the public cloud as a colder tier of storage.
Snapshots to the cloud. The array can direct and
recall snapshots to remote systems, including cloud storage targets.
Processing in the cloud. Look for ways in which the cloud-hosted data can still be actively used, such as for analytics, forensics or other cloud-side processing.
Recovery in the cloud. Look for service provid-ers that make it easy to recover storage backed up to the cloud. This enables disaster recovery as a service, in which your DR site is the public cloud.
Content distribution and global access. Cloud storage can also serve to replicate and propa-gate data around the globe for local access or as remote office/branch office primary storage for sharing large design files, videos and follow-the-sun business processing.
WHAT A PROVIDER PROVIDES
You will also want to look for service providers that work closely with your storage-portfolio vendors—or at least the product roadmaps
HOME
EDITOR’S NOTE
STUDY THE CLASSES
OF AWS STORAGE
NEARLINE STORAGE
COMES IN FROM
THE COLD
AWS PROVIDES
A RANGE OF COLD
STORAGE OPTIONS
SIZING UP THE AWS STORAGE OPTIONS10
NEARLINE STORAGE
anticipating your next storage upgrade. You don’t want to be the one troubleshooting if there are problems between on-premises arrays and cloud storage tiers.
Many providers, such as IBM and Oracle, even offer virtual storage area networks out of their public clouds, not just virtual storage
endpoints or volumes. These virtual SANs can often be used and operated like on-premises storage arrays. This approach helps data center admins directly extend existing storage prac-tices while taking advantage of the cloud infra-structure’s elasticity and utility pricing.
—Mike Matchett
HOME
EDITOR’S NOTE
STUDY THE CLASSES
OF AWS STORAGE
NEARLINE STORAGE
COMES IN FROM
THE COLD
AWS PROVIDES
A RANGE OF COLD
STORAGE OPTIONS
SIZING UP THE AWS STORAGE OPTIONS11
COLD STORAGE OPTIONS
AWS Provides a Range of Cold Storage Options
Customers can choose from several archival choices for AWS cold storage, a capa-bility that enables an organization to move data to and from the cloud without a significant strain on resources.
Dedicated high-speed network connections, such as AWS Direct Connect, can establish a connection between a data center and an AWS cloud facility. The Ethernet connection bypasses the public internet, avoiding bottle-necks that often reduce bandwidth and lower data transfer performance. AWS Direct Con-nect services are available in 1 gigabit Ethernet (GbE) or 10 GbE speeds; IT teams can aggregate multiple links for additional bandwidth.
THE APPLIANCE OPTION
The AWS Import/Export Snowball is another way to transfer huge volumes of data. It uses a portable out-of-band storage appliance. An
administrator requests a large transfer job, and AWS ships a storage appliance. The admin then connects the appliance to the network and transfers the desired data, which is then encrypted. The company returns the device to AWS, which moves the data to storage using Amazon’s internal network.
The process sounds old-fashioned, but this method can be faster and less expensive than moving data over the internet, depending on a company’s network connection speeds and the amount of data it needs to move. AWS Import/Export Snowball can be a good option for data volumes as low as 60 TB over a 1 GbE con-nection. Also, the appliance might be a better alternative when restoring massive backups from cloud storage to the local data center.
Neither AWS Direct Connect nor AWS Import/Export Snowball directly moves data to Amazon Glacier. These methods transfer data to Amazon Simple Storage Service (S3).
HOME
EDITOR’S NOTE
STUDY THE CLASSES
OF AWS STORAGE
NEARLINE STORAGE
COMES IN FROM
THE COLD
AWS PROVIDES
A RANGE OF COLD
STORAGE OPTIONS
SIZING UP THE AWS STORAGE OPTIONS12
COLD STORAGE OPTIONS
Admins then move the data offline to a Glacier vault.
Other alternatives to accelerate data transfers between a business and AWS include Amazon Kinesis Firehose, which addresses multiple streaming data sources. Amazon S3 Transfer Acceleration handles recurring storage jobs with incremental changes—usually over long distances—and gateways to cache data locally. Still, AWS Direct Connect and AWS Import/Export Snowball are probably the most desirable options for use with AWS cold storage.
WATCH THOSE COSTS
There are also some cost considerations for using AWS cold storage. For example, each object that Glacier archives requires a company to store some data in S3 to maintain metadata; this can be as much as 40 KB per object. While
that isn’t much, the total can add up for a large number of objects. When an IT team plans to retain those objects in Glacier for months or even years, storing Glacier object metadata in S3 can add unexpected costs to the archives. It can be helpful to compress many small data objects into a single consolidated .tar or .zip file before uploading and moving them to AWS cold storage.
AWS imposes an early deletion fee if an admin deletes or changes an object within three months of creating the archive. There’s no charge for changes made after three months. Amazon charges to move each object to Glacier, which means you may find it expensive to shift a large number of objects.
IT teams can incur other costs when restor-ing data to S3 from Glacier. AWS usually charges for restorations based on the highest rate encountered in GB per hour.
—Stephen J. Bigelow
HOME
EDITOR’S NOTE
STUDY THE CLASSES
OF AWS STORAGE
NEARLINE STORAGE
COMES IN FROM
THE COLD
AWS PROVIDES
A RANGE OF COLD
STORAGE OPTIONS
SIZING UP THE AWS STORAGE OPTIONS13
ABOUT THE
AUTHORS
STEPHEN J. BIGELOW is senior technology editor in TechTarget’s Data Center and Virtualization Media Group. He has more than 20 years of experience in technology writing and is the author of several books.
MIKE MATCHETT is a senior IT analyst and consultant at Taneja Group, which evaluates technologies in the data storage and server markets.
OFIR NACHMANI is a business technology advisor, blogger and lecturer. His IamOnDemand.com blog is a guide for technology startups and developers in the world of cloud computing.
Sizing Up the AWS Storage Options is a SearchAWS.com e-publication.
Margie Semilof | Editorial Director
Phil Sweeney | Senior Managing Editor
Dan Cagen | Associate Features Editor
Linda Koury | Director of Online Design
Megan Cassello | Graphic Designer
Moriah Sargent | Managing Editor, E-Products
Rebecca Kitchens | Publisher [email protected]
TechTarget 275 Grove Street, Newton, MA 02466
www.techtarget.com
© 2016 TechTarget Inc. No part of this publication may be transmitted or repro-duced in any form or by any means without written permission from the pub-lisher. TechTarget reprints are available through The YGS Group.
About TechTarget: TechTarget publishes media for information technology pro-fessionals. More than 100 focused websites enable quick access to a deep store of news, advice and analysis about the technologies, products and processes crucial to your job. Our live and virtual events give you direct access to independent ex-pert commentary and advice. At IT Knowledge Exchange, our social community, you can get advice and share solutions with peers and experts.
COVER: FOTOLIA
STAY CONNECTED!
Follow @SearchAWS today.