data storage for the long haul: compliance and archive

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Henry Zhang, Senior Product Manager, Amazon Glacier

April 2016

Data Storage for the Long Haul

Compliance and Archive

AWS storage maturity

Amazon EFS

File

Amazon EBSAmazon EC2

Instance Store

Block

Amazon S3 Amazon Glacier

Object

Data Transfer

AWS Direct

Connect

AWS

Snowball

ISV

Connectors

Amazon

Kinesis

Firehose

Amazon S3

Transfer

Acceleration

AWS Storage

Gateway

Audio archives – SoundCloud

• World’s leading social sound platform

• Audio files transcoded and stored in multiple formats

• Stores PBs of data

• Transcoded files served from Amazon S3

• Originals moved to Amazon Glacier for long-term retention

• Media distribution backbone (Ve.nue platform)

• OTT broadcast service

• PBs of media assets

• Assets to be archived and retained for decades

Video archives

Patient data – Philips Healthcare

• HealthSuite digital platform powered by AWS

• 15 petabytes of patient data

• Archived for decades (beyond the lifetime of patients)

• Uses AWS HIPAA eligible services in the BAA

Public sector – King County

• Most populous county in Washington state

• Replace tape solution for backup from 17 agencies

• Meet compliance requirement

• Saved $1MM in first year, no more tape refresh or

management churn

Archive:

Data retained for the long term,

for compliance or potential

future reference

Data archiving needs are growing everywhere

• Media assets, 4K, 8K

• Health care / life sciences

• Financial services

• Regulated industries

• Oil and gas / geospatial

• Digital preservation

• Long-term backups

• Logs

Traditional archiving approaches

• Storage arrays / disk arrays

• Tape silos / tape libraries

• Tape drives (LTO-X / DLT / etc.)

• Virtual tape libraries (VTLs)

• Tape out / vaulting

• Specialized software and

personnel

How can AWS help with your archival?

Metered usage:

Pay as you go

No capital investment

No commitment

No risky capacity planning

Avoid risks of physical

media handling

Control your

geographic locality for

performance and

compliance

Archive Options – Storage Tiers and Data Lifecycle

Object Storage Options

S3 Standard

Active data Archive dataInfrequently accessed data

S3 Standard - Infrequent

Access

Amazon Glacier

Milliseconds 3-5 hoursMilliseconds

$0.03/GB/mo $0.007/GB/mo$0.0125/GB/mo

A Closer Look: S3-IA and Amazon Glacier

S3 - IA

• Same durability and throughput as S3 Standard

• Instant access

• $0.01/GB on each data retrieval

Amazon Glacier

• Same 11 9s durability as S3 Standard

• 3-5 hour data retrieval latency

• Suitable for cold archive such as offsite tapes

S3 Standard - Infrequent

Access

Amazon Glacier

- Transition Standard to Standard-IA

- Transition Standard-IA to Amazon Glacier

- Expiration lifecycle policy

- Versioning support

Data lifecycle management

T T+3 days T+5 days T+ 15 days T + 25 days T + 30 days T + 60 days T + 90 days T + 150 days T + 250 days T + 365 days

Data access frequency over time

Setup lifecycle policy

Transition older videos to Standard-IA

Archive to S3-IA after 30 days

Lifecycle policy

Standard Storage -> Standard-IA

<LifecycleConfiguration>

<Rule>

<ID>sample-rule</ID>

<Prefix>documents/</Prefix>

<Status>Enabled</Status>

<Transition>

<Days>30</Days>

<StorageClass>STANDARD-IA</StorageClass>

</Transition>

<Transition>

<Days>365</Days>

<StorageClass>GLACIER</StorageClass>

</Transition>

</Rule>

</LifecycleConfiguration>

Archive to Amazon Glacier after 365 days

Lifecycle policy

Standard Storage -> Standard-IA

<LifecycleConfiguration>

<Rule>

<ID>sample-rule</ID>

<Prefix>documents/</Prefix>

<Status>Enabled</Status>

<Transition>

<Days>30</Days>

<StorageClass>STANDARD-IA</StorageClass>

</Transition>

<Transition>

<Days>365</Days>

<StorageClass>GLACIER</StorageClass>

</Transition>

</Rule>

</LifecycleConfiguration>

Standard-IA Storage -> Amazon Glacier

Save money on storage

58% saving over S3 Standard

44% saving over S3 Standard-IA

* Assumes the highest public pricing tier

Example backup software integration

• CommVault – Native Integration

with Amazon S3 and

Amazon Glacier

• Deduplication and encryption

• Single console management

Amazon S3 Amazon Glacier

Compliance Use Case 1 – Regulatory Retention

Amazon Glacier Vault Lock allows you to easily

set compliance controls on individual vaults and

enforce them via a lockable policy

Time-based retention

MFA authentication

Controls govern all

records in a Vault

Immutable policy

Two-step locking

Compliance storage with Vault Lock

Vault Lock for compliance storage

• Non-overwrite, non-erasable records

• Time-based retention with “ArchiveAgeInDays” control

• Policy lockdown (strong governance)

• Legal hold with vault-level tags

• Configure optional designated third-party access and grant

temporary access

Amazon Glacier received a third-party assessment

from Cohasset Associates on how Amazon Glacier

with Vault Lock can be used to meet the requirements

of SEC Rule 17a-4(f) and CFTC 1.31(b)-(c).

Example control: 1 year record retention

Example control: 1 year record retention

Vault Lock: Two-step locking

Legal hold with vault-level tags

Example control: Legal hold

Vault Lock in the Amazon Glacier console

Vault Lock in the Amazon Glacier console

Vault Lock in the Amazon Glacier console

Vault Lock in the Amazon Glacier console

Vault Lock in the Amazon Glacier console

Vault Lock in the Amazon Glacier console

Vault Lock in the Amazon Glacier console

Vault Lock in the Amazon Glacier console

Vault Lock in the Amazon Glacier console

Vault Lock in the Amazon Glacier console

Vault Lock in the Amazon Glacier console

Vault Lock in the Amazon Glacier console

Vault Lock in the Amazon Glacier console

Vault Lock in the Amazon Glacier console

Vault Lock in the Amazon Glacier console

Vault Lock in the Amazon Glacier console

Vault Lock in the Amazon Glacier console

Compliance Use Case 2 – Auditing and Alerts

Audit logging with AWS CloudTrail

• Amaozn S3 and Amazon Glacier can log

API calls for audit via CloudTrail

• Enable CloudTrail in the AWS console

and designate your log bucket

• S3 logs bucket-level activities; object

activities supported via event notification

• Amazon Glacier logs all APIs calls for

vault and archives

Access policy for a storage container

• Control access to a storage container in a single location

– S3 bucket or Amazon Glacier vault access policy

– Grant/revoke access to internal business units/teams

– “Marketing_Vault” has a distinct access policy from “DevOps_Vault”

• Easily manage cross-account access for your business partner

– Simply add a section for your business partner in the same policy

– Cross-account activities (API calls) also show up in CloudTrail logs

Amazon S3 event notifications

Events

SNS topic

SQS

queue

Lambda

function

• Notification when objects are

created via PUT, POST, Copy, or

Multipart Upload, DELETE

• Filtering on prefixes and suffixes

for all types of notifications

Request specific notifications

Request notifications on specific

PUT APIs

Request notifications on specific

DELETE APIs

s3:ObjectCreated:*

s3:ObjectCreated:Put

s3:ObjectCreated:Post

s3:ObjectCreated:Copy

s3:ObjectCreated:CompleteMultipartUpload

s3:ObjectRemoved:*

s3:ObjectRemoved:Delete

s3:ObjectRemoved:DeleteMarkerCreated

Compliance Use Case 3 – Geographic Redundancy

Remote replicas managed

by separate AWS accounts

Secure

Distribute data to regional

customers

Lower Latency

Store hundreds of

miles apart

Compliance

Amazon S3 cross-region replicationAutomated, fast, and reliable asynchronous replication of data across AWS regions

• Usual charges for

storage, requests, and

inter-region data transfer

for the replicated copy of

data

• Replicate into Standard-IA

or Amazon Glacier

Cost

HEAD operation on a source

object to determine replication

status

• Replicated objects will not be

re-replicated

• Use Amazon S3 COPY to

replicate existing objects

Replication status

DELETE without object

version ID• Marker replicated

DELETE specific object

version ID• Marker NOT replicated

Delete operation

Cross-region replication: Details

Object ACL updates are

replicated

• Objects with Amazon-

managed encryption key

replicated

• AWS KMS encryption not

replicated

Access control

Versioning with cross-region replication

A

B

Vid1- v2

Vid1- v1

Key: A/vid1 Key: B/vid1

Vid1- v2

Vid1- v1

Vid1- v3Vid1- v3

Vid1- v4Vid1- v4

A

Cross-region replication with lifecycle archiving

S3

Bucket A

Amazon Glacier

S3

Bucket B

AWS Import/Export Snowball

• Accelerate PBs with AWS-

provided appliances

• 80 TB model, global availability

AWS Storage Gateway

• Instant hybrid cloud

• Up to 120 MB/s cloud upload rate

(4x improvement), and

Data ingestion into AWS storage services

Amazon Kinesis Firehose

• Ingest data streams directly into

AWS data stores

AWS Direct Connect

• COLO to AWS

ISV Connectors

• CommVault

• VERITAS

• etcetera

Amazon S3 Transfer Acceleration

• Move data up to 300% faster

using AWS’s private network

What is AWS Snowball? Petabyte scale data transport

E-ink shipping

label

Ruggedized

case

“8.5G Impact”

All data encrypted

end-to-end50TB or 80 TB

10G network

Rain & dust

resistant

Tamper-resistant

case & electronics

Pricing

Dimension Price

Usage Charge per Job $250.00

Extra Day Charge (First 10 days* are free) $15.00

Data Transfer In $0.00/GB

Data Transfer Out $0.02/GB

Shipping** Varies

Amazon S3 Charges Standard storage and request

fees apply

* Starts one day after the appliance is delivered to you. The first day the appliance is received at your site and the last day the appliance is shipped out are also free

and not included in the 10-day free usage time.

** Shipping charges are based on your shipment destination and the shipping option (e.g., overnight, 2-day) you choose.

Transfer 1 PB with 13 devices

in parallel in 1 week!