2016 utah cloud summit: aws s3

© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Raul Frias, Solutions Architect

January 2016

Amazon S3

New for 2015…

Cross-region replication

Amazon S3 Standard-IA

AWS CloudTrail support for Amazon

S3

Amazon CloudWatch metrics for Amazon

S3

VPC endpoint for Amazon S3

Amazon S3 bucket limit increase

Event notifications

Read-after-write consistency in all regions

Video sharing serviceVidShare

• Allow friends and families to record, edit and share video clips

• Great user experience with low latency

• Create thumbnails to playback videos

• Millions of customers

Video sharing serviceVidShare Roadmap:

Launch

• Scale: Millions of users• Focus on US market• Create thumbnails for users to

playback videos

V2 Optimize on cost

V3 Expand globally

V4 Enterprise enablement

Video sharing service

VidShare Design

• Use event notifications to trigger an AWS Lambda function

• Add thumbnails and metadata to Amazon DynamoDB

• Playback videos

VidShare

- Thumbnail- Update Index- Update WebApp

Video sharing service

Event

MetadataThumbnaillogs

VidShare

VidShare Design

• Use event notifications to trigger an AWS Lambda function

• Add thumbnails and metadata to Amazon DynamoDB

• Playback videos

Amazon S3 event notifications

Events

SNS topic

SQS queue

Lambda function

• Notification when objects are created via PUT, POST, Copy, or Multipart Upload, DELETE

• Filtering on prefixes and suffixes for all types of notifications

Fast IntegratedSimple

Setup delete notifications – config and execute

Launch


playback videos

V2 Optimize on cost

V3 Expand globally


Optimizing VidShare

• 1 PB of storage and growing• No impact on user experience• Minimal code change

VidShare Roadmap:

TTTTTTTTTTTTT+3 daysT+3 daysT+3 daysT+3 daysT+3 daysT+3 daysT+3 daysT+3 daysT+3 daysT+3 daysT+3 daysT+3 daysT+5 daysT+5 daysT+5 daysT+5 daysT+5 daysT+5 daysT+5 daysT+5 daysT+5 daysT+5 daysT+5 daysT+5 daysT+ 15 daysT+ 15 daysT+ 15 daysT+ 15 daysT+ 15 daysT+ 15 daysT+ 15 daysT+ 15 daysT+ 15 daysT+ 15 daysT+ 15 daysT+ 15 daysT + 25 daysT + 25 daysT + 25 daysT + 25 daysT + 25 daysT + 25 daysT + 25 daysT + 25 daysT + 25 daysT + 25 daysT + 25 daysT + 25 daysT + 30 daysT + 30 daysT + 30 daysT + 30 daysT + 30 daysT + 30 daysT + 30 daysT + 30 daysT + 30 daysT + 30 daysT + 30 daysT + 30 daysT + 60 daysT + 60 daysT + 60 daysT + 60 daysT + 60 daysT + 60 daysT + 60 daysT + 60 daysT + 60 daysT + 60 daysT + 60 daysT + 60 daysT + 60 daysT + 90 daysT + 90 daysT + 90 daysT + 90 daysT + 90 daysT + 90 daysT + 90 daysT + 90 daysT + 90 daysT + 90 daysT + 90 daysT + 90 daysT + 90 daysT + 150 daysT + 150 daysT + 150 daysT + 150 daysT + 150 daysT + 150 daysT + 150 daysT + 150 daysT + 150 daysT + 150 daysT + 150 daysT + 150 daysT + 150 daysT + 250 daysT + 250 daysT + 250 daysT + 250 daysT + 250 daysT + 250 daysT + 250 daysT + 250 daysT + 250 daysT + 250 daysT + 250 daysT + 250 daysT + 250 daysT + 365 daysT + 365 daysT + 365 daysT + 365 daysT + 365 daysT + 365 daysT + 365 daysT + 365 daysT + 365 daysT + 365 daysT + 365 daysT + 365 daysT + 365 days

Acc

ess

Freq

uenc

y

Choice of storage classes on Amazon S3

Standard

Active data Archive dataInfrequently accessed data

Standard - Infrequent Access Amazon Glacier

11 9’s of Durability

Standard-Infrequent Access storageInfrequently accessed data

Designed for 99.9% availability

Durable AvailableSame throughput as

Amazon S3 Standard storage

High performance

• Server-side encryption• Use your encryption keys• KMS managed encryption keys

Secure• Lifecycle management• Versioning • Event notifications• Metrics

Integrated• No impact on user

experience• Simple REST API• Single bucket

Easy to use

- Transition Standard to Standard-IA- Transition Standard-IA to Amazon Glacier

storage- Expiration lifecycle policy- Versioning support

Standard-Infrequent Access storage

Integrated with lifecycle

Integrated: Lifecycle management

Save money on VidShare

Lifecycle policy

Standard Storage -> Standard-IA

<LifecycleConfiguration> <Rule> <ID>sample-rule</ID> <Prefix>documents/</Prefix> <Status>Enabled</Status> <Transition> <Days>30</Days>

<StorageClass>STANDARD-IA</StorageClass> </Transition> <Transition> <Days>365</Days>

<StorageClass>GLACIER</StorageClass> </Transition> </Rule> </LifecycleConfiguration>


Lifecycle Policy

Standard Storage -> Standard-IA

<LifecycleConfiguration> <Rule> <ID>sample-rule</ID> <Prefix>documents/</Prefix> <Status>Enabled</Status> <Transition> <Days>30</Days>

<StorageClass>STANDARD-IA</StorageClass> </Transition> <Transition> <Days>365</Days>

<StorageClass>GLACIER</StorageClass> </Transition> </Rule> </LifecycleConfiguration>

Standard-IA Storage -> Amazon Glacier

Setup lifecycle policy

Transition older videos to Standard-IA


• 1 PB of storage and growing• 1 PB for S3 Standard = $28,816/- per month for storage cost

• Videos older than 30 days are watched less than once a month on average

1 PB for S3 Standard-IA = $17,600/- per month for storage costSaving 39% on storage

* Assumes the highest public pricing tier

VidShare is global!• Expand to global users• Share videos with families

across the globe• New user base in Japan• Need low latency for global

users

VidShare

Launch


playback videos

V2 Optimize on cost

V3 Expand globally


Roadmap:

Remote replicas managed by separate AWS accounts

SecureDistribute data to regional customers

Lower Latency

Store hundreds of miles apart

Compliance

Amazon S3 cross-region replicationAutomated, fast, and reliable asynchronous replication of data across AWS regions

VidShare v3.0 – Global expansion

VidShare for enterprises!• Enterprise customers want to record

meetings and share videos• Logically isolated section of the cloud• All users within the Amazon Virtual

Private Cloud (Amazon VPC)• Restricted Internet access

VidShare

Launch


playback videos

V2 Optimize on cost

V3 Expand globally


Roadmap:

Using Amazon S3 with VPC endpoints – Previously…

mybucket

InternetPUT S3

PU

T S3NAT\

Internet Gateway

• Public IP on EC2 instances and IGW

• Private IP on EC2 instances and NAT

Amazon S3 VPC endpointsAccess Amazon S3 from your Amazon VPC using VPC endpoints

VP

C

End

poin

tPUT S3

VPC Policy

mybucket

Internet


Using Amazon S3 VPC endpoints

Improved throughput from VPC resources to Amazon S3

High availability

High performanceHigh availability

Reduce cost by avoiding expensive NAT, Internet gateways

Lower cost

Simple to setup, no need to manage NATs and Internet gateways

Simple

Improved security, no need to route traffic through the internet

Secure


Amazon S3 VPC endpoints

Using Amazon S3 VPC endpoints

• Control access to buckets from specific Amazon VPC endpoints, or specific VPCs

• Control which VPCs or VPC endpoints have access to your S3 buckets by using S3 bucket policies

Audit logs Amazon S3Demonstrate compliance, improve security

Log Amazon S3 API using AWS CloudTrail

Track bucket-level operations• Creation and deletion of buckets• Changes to access control, lifecycle policy,

cross region replications policy etc.

Integrated with Amazon CloudWatch• Alarm if a specific API called

Configure once per AWS Account• Track multiple services with AWS CloudTrail

Amazon S3 storage metricsUnderstand your Amazon S3 buckets

Amazon CloudWatch metrics for Amazon S3

Bucket-level metrics include:• Total bytes for Standard storage

• Total bytes for Standard-IA storage

• Total bytes for Reduced-Redundancy storage

• Total number of objects for a given S3 bucket

Alarm on S3 metrics• Set thresholds for alarms

Daily metrics• Metrics emitted daily, after midnight GMT

Thank you!

2016 utah cloud summit: aws s3

Technology