(stg202) aws import/export snowball: large-scale data ingest into aws

40
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Colin Lazier, General Manager - AWS Import/Export & Amazon Glacier October 2015 Introduction to AWS Import/Export Snowball Accelerating Large-Scale Data Ingest into AWS Storage Services STG202

Upload: amazon-web-services

Post on 07-Jan-2017

2.208 views

Category:

Technology


0 download

TRANSCRIPT

© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Colin Lazier, General Manager - AWS Import/Export & Amazon Glacier

October 2015

Introduction to

AWS Import/Export SnowballAccelerating Large-Scale Data Ingest

into AWS Storage Services

STG202

What to expect from the session

• What is AWS Import/Export Snowball?

• How does it work?

• What are the two most commonly asked Snowball

questions?

• How fast is that truck full of drives?

• How is my data transported securely?

• Where can it go?

• What does it cost?

• Demo

What is Snowball? Petabyte scale data transport

E-ink shipping

label

Ruggedized

case

“8.5G Impact”

All data encrypted

end-to-end50 TB

10G network

Rain & dust

resistant

Tamper-resistant

case & electronics

How does it work?

Question 1:

Question 1: You have heard of networks, right?

Question 1: You have heard of networks, right?

• Yes

Question 1: You have heard of networks, right?

• Yes

• However, for certain use cases networks are:

Too slow

Inelastic

Too expensive

Question 1: You have heard of networks, right?

• Yes

• However, for certain use cases networks are:

Too slow

Inelastic

Too expensive

Bandwidth is growing

more slowly than

storage volumes

Question 1: You have heard of networks, right?

• Snowball

Fast

Elastic

Cost effective

10G network

50TB capacity

Question 1: You have heard of networks, right?

• Snowball

Fast

Elastic

Cost effective

10G network

50TB capacity

Question 1: You have heard of networks, right?

• Snowball

• Parallelize as needed

5X Snowballs

50G & 250TB

Question 2:

Question 2: Can I drop it?

Question 2: Can I drop it?

• No (please don’t)

Question 2: Can I drop it?

• No (please don’t)

• Snowball is its own box

• Has had many drop tests already

• Can handle 8.5G impacts

• Designed for shipping

Ok, let’s drop it

How fast is that truck full of drives?

• Less than 1 day to transfer 50TB via a 10G connection

with Snowball, less than 1 week including shipping

• Number of days to transfer 50TB via the internet at typical

utilizations

Internet Connection Speed

Utilization 1Gbps 500Mbps 300Mbps 150Mbps

25% 19 38 63 126

50% 9 19 32 63

75% 6 13 21 42

How fast is that truck full of drives?

• Less than 1 day to transfer 250TB via 5x10G connections

with 5 Snowballs, less than 1 week including shipping

• Number of days to transfer 250TB via the Internet at

typical utilizations

Internet Connection Speed

Utilization 1Gbps 500Mbps 300Mbps 150Mbps

25% 95 190 316 632

50% 47 95 158 316

75% 32 63 105 211

How is my data transported securely?

• 256-bit encryption

• Data is encrypted by the

Snowball client before it reaches

the Snowball appliance

• Keys are managed by KMS and

are never sent to the Snowball

How is my data transported securely?

• Strong chain of custody

• Tamper-resistant case

• Tamper-resistant electronics

(TPM)

• Each Snowball is erased

according to NIST 800-88 media

sanitization guidelines between

every job

From Where can it go?

Import to Amazon S3, Export coming soon

Amazon

EMRAmazon Redshift

Amazon S3

Amazon Glacier

Amazon DynamoDBAmazon RDSAmazon EBS

US-EAST-1

(Virginia)

US-WEST-2

(Oregon)

What does it cost?

• $200 / job plus shipping

• Includes 10 days to fill the device at your site

• $15/day after the tenth day on site

• Standard Amazon S3 charges apply

• $0.03/GB to transfer data out

• $0.00/GB to transfer data in

What does it cost?

Example 1:

• 40TB loaded on to 1 Snowball

• 2 days at your site

• $200 plus shipping

Example 2:

• 30TB loaded on to 1 Snowball

• 12 days at your site

• $200 + 2*$15/day = $230 plus shipping

What does it cost?

Example 3:

• 140TB loaded on to 3 Snowballs

• 8 days at your site

• 3 * $200 = $600 plus shipping

Example 4:

• 30TB exported on to 1 Snowball

• 8 days at your site

• $200 + 30TB * $0.03/GB = $1,121.60 plus shipping

Demo

Overview

Step 1: Create a job

Step 1: Create a job

Step 1: Create a job

Step 1: Create a job

Step 2: Install client & await arrival

Step 3: Plug in your Snowball appliance

Step 4: Get your credentials

Step 5: Connect client to the Snowball appliance

From a terminal:

snowball start -i 192.0.2.0 -m

/Downloads/JID2EXAMPLE-0c40-49a7-9f53-

916aEXAMPLE81-manifest.bin -u 12345-abcde-12345-

ABCDE-123

snowball start -i [Snowball IP Address] -m

[Path/to/manifest/file] -u [29 character unlock code]

Step 6: Copy your data

From terminal:

snowball cp --recursive /Logs/April

s3://MyBucket/Logs/April

snowball cp [options] [path/to/data/source]

s3://[path/to/data/destination]

Step 7: Unplug your Snowball appliance & notify UPS

Step 7: Wait for import to complete

Thank you!

Remember to complete

your evaluations!