hybrid architectures for database

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Hybrid architectures for database backups & file migrations

S T G 3 1 3

Jeff Bartley

Solutions Architect, Hybrid Cloud Storage and Transfer

Amazon Web Services

Peter Levett

Storage Specialist Solutions Architect

Amazon Web Services

Agenda

AWS Storage Gateway overview

Scenario 1: SQL Server backup to AWS

Scenario 2: NFS migration with AWS DataSync and AWS Storage Gateway

Start working

Related sessions

STG201 - Leadership session: Storage state of the union

STG225 - Getting started with hybrid file storage using File Gateway

STG226 - Hands-on with hybrid block storage using a volume gateway

STG217 - Shift your tape backups to AWS to save time and money

STG305 - Build hybrid storage architectures with AWS Storage Gateway

STG336 - Using hybrid cloud storage to close a data center and migrate

STG208 - Backup-and-restore and disaster-recovery solutions with AWS

STG204 - Get data to AWS: How to choose and use data migration services

STG224 - Getting started with AWS DataSync

STG220 - How to move 700 TB over the wire to AWS with AWS DataSync

STG316 - Get hands-on & learn best practices for AWS data migrations

AWS Storage Gateway

Provides on-premises access to virtually unlimited cloud storage . . .

. . . regardless of cloud adoption stage

Move on-premises backups

to the cloud

Provide low-latency access

for on-premises applications

to cloud data

Shift on-premises storage to

cloud-backed file shares

AWS CloudCustomer premise

Files

(NFS/SMB)

Volumes

(iSCSI)

Tapes

(iSCSI VTL)

AWS Storage Gateway

Integrated with AWS Identity and Access Management

(IAM), AWS Key Management Service (AWS KMS),

AWS CloudTrail, Amazon CloudWatch services

Amazon S3

Glacier

Amazon Simple

Storage Service

(Amazon S3)

Amazon Elastic

Block Store

(Amazon EBS)

AWS Storage Gateway

Configuration: VMware ESXi, Microsoft Hyper-V,

Amazon Elastic Compute Cloud (Amazon EC2),

hardware appliance

AWS Backup

Amazon S3

Glacier Deep

Archive

Storage Gateway serviceStorage Gateway

HTTPS

Move on-premises backups to the cloud

iSCSI VTL

AWS Cloud

File

gateway

Volume

gateway

Tape

gateway

AWS

Storage

Gateway

managed

service

Database/

application

server

Application

server

Backup

server

iSCSI

NFS/SMB

Tape library

(Amazon S3)Tape archive

(S3 Glacier/GDA)

Amazon S3 Amazon EBSAWS Backup

HTTPS

HTTPS

HTTPS

On premises

Any S3 storage class

Lifecycle

Amazon S3

Eject

Maintain your backup workflows while reducing your backup infrastructure on-premises

File Gateway for on-premises backupMove database and file backups into the cloud and free up on-premises storage capacity

Features

NFS/SMB protocol support; mount shares directly

on database and application servers

Files stored durably in Amazon S3, lifecycle to

any S3 storage class

Local cache for accessing recent backups

Windows ACL support to control access to

backup files

Support for S3 object lock

Bandwidth optimized; only changes are

transferred

Reduce on-premises storage for backups

Easily integrates with SAP, SQL Server,

Oracle, HDFS, and other applications

Restore backups on-premises or in the

cloud on Amazon EC2 or Amazon RDS

Benefits

AWS Cloud

HTTPSFile

gateway

NFS/SMBDatabase/

application

server


On premises

Amazon S3

Lifecycle

Shift on-premises storage to cloud-backed file sharesAccess virtually unlimited, highly durable cloud storage using common file protocols

Features Benefits

Supports NFS and SMB protocols; no application

changes required

Files stored durably in Amazon S3, lifecycle to

any S3 storage class

SMB shares integrate with Microsoft Active

Directory

Amazon CloudWatch events for automated

workflows

Reduce costs by moving storage to Amazon

S3 while still accessing from on premises

Virtually unlimited cloud storage; no more

running out of capacity

Eliminate expensive hardware refresh cycles

Files stored as native S3 objects for further

processing in AWS

AWS Cloud

HTTPSFile

gateway

NFS/SMBApplication


On premises

Amazon S3

Lifecycle

NAS storage

Low-latency access for on-premises applications to cloud data Access files quickly from distributed locations and scale capacity as needed

Features Benefits

Generate data in the cloud or ingest from on

premises using AWS DataSync or AWS Snowball

Up to 16 TB local cache per gateway

Fully managed gateway cache provides low-

latency access to data

Refresh cache at the bucket or prefix level

Access cloud storage from any on-premises

location

Process data in the cloud and refresh

gateway cache for up-to-date results

Data stored cost-effectively and centrally in

the cloud

AWS Cloud

Application

NFS/SMB

Cache refresh

HTTPS

Cache refresh

HTTPS

Application

NFS/SMB

On-premises

File gateway

On-premises

File gatewayIn-cloud processing

AWS

DataSync

AWS

Snowball

2019 feature releases

YTD 2019 Just launched

Protocols &

services

• Tape gateway supports archiving existing and

new tapes to Amazon S3 Glacier deep archive

• AWS PrivateLink support for all gateway types

• FGW adds SMB ACLs and additional encryption

options for enhanced security

• S3 object lock support to enable WORM

workloads

Management

& automation

features

• Hardware appliance available in the EU

• Selective cache refresh to enable content

distribution

• TGW support for Amazon S3 Glacier deep archive

• Tag-on-create, tag-based access control

• High availability for VMware deployments across

all gateway types

• Improved Amazon CloudWatch logging and

metrics for all gateway types

• Gateway software update scheduling for all

gateway types

Limits• 5× read perf. increase for file gateway (to 4.8

Gbps)

• 3× perf. increase for tape gateway (to 2.7 Gbps)

Regions • ARN, BAH, HKG, GovCloud (PDT), GRU (TGW)

Learn more

aws.amazon.com/storagegateway/

• Whitepapers

• Webinars

• Demos & videos

• Developer resources

• Recorded AWS re:Invent

Sessions

• Product documentation

• Pricing

• FAQs

• Customer case studies

Workshop scenario 1: MS SQL backup to Amazon S3

AWS Cloud

Region (eu-west-1)

Availability Zone

SQL instance

Domain controller

instance

AWS Storage Gateway

instance

CacheShare

(smb)

AWS Cloud

Region (eu-central-1)

Amazon CloudWatchAWS Storage

Gateway

Amazon S3 Glacier

Amazon S3: Infrequent access

Backup

bucket

Expire/delete backup after 7 years

Lifecycle after 30 days

Amazon SQS

Bastion Amazon RDS

instance


AWS Cloud

Region (eu-west-1)

Availability Zone

SQL instance

Domain controller

instance

AWS Storage Gateway

instance

AWS Cloud


1. Deploy VPC, domain controller,

SQL Server, and Storage

Gateway (simulate on-premises

in eu-west-1)

2. Active gateway in eu-central-1

AWS Storage

Gateway


AWS Cloud

Region (eu-west-1)

Availability Zone

SQL instance

Domain controller

instance

AWS Storage Gateway

instance

Share

(smb)

AWS Cloud


Backup

bucket

1. Create S3 bucket

2. Attach gateway to AD

domain

3. Create SMB share attached

to new bucket

4. Test access to bucket from

SQL Server


AWS Cloud

Region (eu-west-1)

Availability Zone

SQL instance

Domain controller

instance

AWS Storage Gateway

instance

CacheShare

(smb)

AWS Cloud



Gateway

Backup

bucket

Amazon SQS

1. Create SQS queue

2. Create Amazon CloudWatch rule that triggers Amazon SQS


AWS Cloud

Region (eu-west-1)

Availability Zone

SQL instance

Domain controller

instance

AWS Storage Gateway

instance

CacheShare

(smb)

AWS Cloud



Gateway

Backup

bucket

Amazon SQS

1. Run backup process


AWS Cloud

Region (eu-west-1)

Availability Zone

SQL instance

Domain controller

instance

AWS Storage Gateway

instance

CacheShare

(smb)

AWS Cloud


Backup

bucket

Domain controller

instance

Amazon RDS

instance

1. Provision Bastion host & RDS instance

2. On-premises backup to RDS

Migrate on-premises NFS server data to Amazon S3, validate

access to Amazon S3 from File Gateway, and shut down the on-

premises NFS server

Workshop goal

Workshop scenario 2

Application

server

NFS

server

NFS

In-cloud Region (eu-central-1)On-premises Region (eu-west-1)

DataSync

agent

File gateway

NFS

S3 bucket

DataSync

managed service

HTTPS

TLSNFS

• Resources deployed using CloudFormation templates

• Connect to the application server, mount NFS, and verify files

Deploy resources in 2 regions and verify data on the NFS server

Workshop scenario 2: Module 1

Application

server

NFS

server

NFS


DataSync

agent

File gateway

S3 bucket

Activate the DataSync agent, create source and destination locations, trigger the task to copy files to S3

Copy file data to Amazon S3 using AWS DataSync


Application

server

NFS

server

NFS


DataSync

agent

File gateway

S3 bucket

DataSync

managed service

TLSNFS

Source

location

Destination

location

Activate the file gateway, create an NFS share, mount the share

on the application server

Access the S3 bucket on premises using File Gateway


Application

server

NFS

server

NFS


DataSync

agent

File gateway

NFS

S3 bucket

DataSync

managed service

HTTPS

TLSNFS

Rerun the DataSync task to copy remaining files from the NFS

server to the S3 bucket

Perform an incremental transfer to get remaining data


Application

server

NFS

server

NFS


DataSync

agent

File gateway

NFS

S3 bucket

DataSync

managed service

HTTPS

TLSNFS

Unmount the NFS server on the application server, all traffic

now goes to File Gateway

Cutover to File Gateway and shut down the NFS server


Application

server

NFS

server


DataSync

agent

File gateway

NFS

S3 bucketHTTPS

Go to https://dashboard.eventengine.run

Click on the AWS Consolebutton

Launch the AWS Console

(opens a new tab)

Click on the Readme for next steps


Visit aws.amazon.com/training/path-storage/

Classroom offerings, like Architecting on AWS, feature AWS expert instructors and hands-on activities

45+ free digital courses cover topics related to cloud storage, including:

Learn storage with AWS Training and Certification

• Amazon S3

• AWS Storage Gateway

• Amazon S3 Glacier

• Amazon Elastic File System

(Amazon EFS)

• Amazon Elastic Block Store

(Amazon EBS)

Resources created by the experts at AWS to help you build cloud storage skills

Thank you!


hybrid architectures for database

Documents