(bac404) deploying high availability and disaster recovery architectures with aws | aws re:invent...

November 12, 2014 | Las Vegas, NV

Vikram Garlapati

• Higher costs for disaster recovery

sites or data centers

• Costs for storage, backup, archival

and retrieval tools, and processes

• Capacity planning, procuring, and

deploying can be challenging

• Very low total cost of ownership

• Scalable storage services and

consistent tools and processes

across various AWS regions and

Availability Zones.

Backup and

restore

Pilot light

Warm standby

Multi-site

On-site infrastructure

S3 Bucket

Over the Internet

AWS Import/Export

or

AWS Storage Gateway

AWS region

Corporate data center

Using AWS Direct Connect

Availability Zone

Amazon Elastic

Compute Cloud (EC2)

AmazonS3 bucket

Backup and

restore

Availability Zone

AWS region

AMI

Instance

AmazonS3 bucket

Pre-bundled with OS and applications

Instance quickly

provisioned from AMI

Data copied from objects

in S3

Data Volume

Amazon EC2

On-site infrastructure

AWS

Import/Export

or

AWS Storage

Gateway

Using AWS Direct

Connect

Backup and

restore

Pilot light

www.example.com

Data Mirroring/ Replication

Not Running

Databa

se

Server

DataVolume


Web

Application Server

Master

Database Server

Pilot light

Not Running

Database

Server

DataVolume


Web

Application Server

MasterDatabase

Server

www.example.com

Application

Server

Web Server

Server

Warm standby

Data Mirroring/ Replication

Application Data Source

Cut Over

Elastic Load

Balancer

Active Production

Amazon Route 53

www.example.com


DataVolume

ApplicationServer

Slave DatabaseServer

Web

AWS region

Web

Application Server

Master

Database Server

Non Production Traffic

Warm standby

Elastic Load

Balancer

Active Production

Amazon Route 53

www.example.com


DataVolume

ApplicationServer

Slave DatabaseServer

Web

AWS region

Web

Application Server

Master

Database Server

AWS

database licenses

low infrastructure

budgets

variable loads.

99.99%

0-2 minutes

0-15 minutes

Backup and restore

Pilot light

Warm standby

Multi-site

awsdrdemo.com

http://www.awsdrdemo.com/

Amazon Route 53

Elastic Load Balancing

Auto Scaling

Redundant nodes / clustering

Redundant nodes

Deploy Multi-AZ

Deploy multi-region

Infrastructure

Foundation

Services

Regions Availability Zones

Storage(Object, Block and Archive)

NetworkingSecurity &

Access Control

Platform

Services

Databases

Relational

NoSQL

Caching

Analytics

Hadoop

Real-time

Data warehouse

App Services

Queuing

Orchestration

App streaming

Transcoding

Email

Search

Deployment & Management

Containers

Dev/ops Tools

Resource

Templates

Mobile Services

Identity

Sync

Mobile

Analytics

Notifications

Enterprise

ApplicationsVirtual Desktops

Data Workflows

Usage

Tracking

Monitoring

and Logs

Compute(VMs, Auto-scaling and Load Balancing)

CDN and Points of Presence

Collaboration and Sharing

Oregon North Virginia

Oregon

Availability Zone #1 Availability Zone #2

Public subnet

Private subnet

Public subnet

Private subnet

Northern Virginia


Public subnet

Private subnet

Public subnet

Private subnet

10.0.0.0/16 192.168.0.0/16

Oregon


Public subnet

Private subnet

Public subnet

Private subnet

Northern Virginia


Public subnet

Private subnet

Public subnet

Private subnet

VPN

Instance

A

VPN

Instance

B

Secondary

IPsec

Primary IPsec

Oregon


Public subnet

Private subnet

Public subnet

Private subnet

Northern Virginia


Public subnet

Private subnet

Public subnet

Private subnet

VPN

Instance

A

VPN

Instance

B

Secondary

IPsec

G2

GPU

enabled

M3

General

purpose

Memory

optimized

R3

CR1M2

Storage and I/O

optimized

C3

Compute

optimized

CC2C1

I2

HI1

HS1

CG1M1

© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

Web, app, and database servers

Northern Virginia


RDPGW

Web

/ app

WSFC2

Oregon

Availability Zone #2

www.awsdrdemo.com

Northern Virginia


RDP

GW / NAT

WSFC2

Oregon


RDP

GW / NATRDP

GW / NAT

Northern Virginia


RDPGW

Web

/ app

WSFC2

Oregon


AD1 AD2 AD3

Northern Virginia


RDPGW

WSFC2

Oregon


Web

/ app

Web

/ app

Web & App

AMI

Northern Virginia


Web

/ app

Oregon


AD1 AD2 AD3SQL1 SQL2 SQL3

ASYNCData

Replication



www.awsdrdemo.com

RDP

GW

RDP

GWRDP

GW

Web

/ app Web

/ app

Web

/ app

Web

/ app

Web

/ appWeb

/ app

ADSQL2

AD1 SQL1 AD2

Sync Data Replication

Northern Virginia


RDP

GW

RDPGW

Web

/ app

Oregon


RDP

GW

AD1 AD2 AD3

Web

/ app

Web

/ app

SQL1 SQL2 SQL3

www.awsdrdemo.com

Async Data Replication

VPN

Northern Virginia


APP

failover.awsdrdemo.com

http://www.awsdrdemo.com/

Check live status at http://status.awsdrdemo.com/ha

http://status.awsdrdemo.com/ha

Check live status at http://status.awsdrdemo.com/dr

http://failover.awsdrdemo.com/dr

HttpWebResponse webresponse;

string uri = http://169.254.169.254/latest/meta-data/placement/availability-zone;

string retValue;

/* Create webrequest for the uri */

HttpWebRequest webrequest = (HttpWebRequest)WebRequest.Create(uri);

/* Capture response and read the stream */

webresponse = (HttpWebResponse)webrequest.GetResponse();

Encoding enc = System.Text.Encoding.GetEncoding(1252);

StreamReader loResponseStream = new

StreamReader(webresponse.GetResponseStream(), enc);

retValue = loResponseStream.ReadToEnd();

http://169.254.169.254/latest/meta-data/placement/availability-zone

"AWSTemplateFormatVersion" : "2010-09-09",

"Description" : "AWS CloudFormation Template ELBWithEC2Instances: Create a load balanced, Auto Scaled sample website where th e instances are locked down to only accept traffic from the load balancer. This script creates an Auto Scaling group behind a load balancer with a simple health check. The web site is available on port 80, however, the instances can be configured to listen on any port (8888 by default).",

"Parameters" : {

"KeyPairName" : {

"Description" : "Name of an existing Amazon EC2 key pair for SSH access",

"Type" : "String",

"Default" : "kamalkey"

},

"InstanceType" : {

"Description" : "WebServer EC2 instance type",

"Type" : "String",

"Default" : "m3.medium",

"AllowedValues" : [ "t1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m2.xlarge","m2.2xlarge","m2.4xlarge","m3.medium","m3.large","m3.xlarge","m3.2xlarge","c1.medium","c1.xlarge"],

"ConstraintDescription" : "must be a valid EC2 instance type."

},

"WebServerPort" : {

"Description" : "TCP/IP port of the web server",

"Type" : "String",

"Default" : "80"

},

"HostedZoneId" : {

"Type" : "String",

"Description" : "The Record Set's Hosted Zone Id for the existing hosted zone",

"Default" : "Z1M58G0W56PQJA"

}

},

"Mappings" : {

"AWSInstanceType2Arch" : {

"t1.micro" : { "Arch" : "64" },

"m1.small" : { "Arch" : "64" },

"m1.medium" : { "Arch" : "64" },

"m1.large" : { "Arch" : "64" },

"m1.xlarge" : { "Arch" : "64" },

"m2.xlarge" : { "Arch" : "64" },

"m2.2xlarge" : { "Arch" : "64" },

"m2.4xlarge" : { "Arch" : "64" },

"m3.medium" : { "Arch" : "64" },

"m3.large" : { "Arch" : "64" },

"m3.xlarge" : { "Arch" : "64" },

"m3.2xlarge" : { "Arch" : "64" },

"c1.medium" : { "Arch" : "64" },

"c1.xlarge" : { "Arch" : "64" }

},

"AWSRegionArch2AMI" : {

"us-east-1":{"32" : "ami-06ef6b6e", "64" : "ami-06ef6b6e" }

}

},

"Resources" : {

"WebServerGroup" : {

"Type" : "AWS::AutoScaling::AutoScalingGroup ",

"Properties" : {

"AvailabilityZones" : [ "us-east-1c","us-east-1d"],

"LaunchConfigurationName" : { "Ref" : "LaunchConfig" },

"MinSize" : "2",

"MaxSize" : "2",

"LoadBalancerNames" : [ { "Ref" : "ElasticLoadBalancer" }],

"VPCZoneIdentifier" : ["subnet-d8f2edf0","subnet -2fc76658"],

"Tags" :[ {

"Key" : "Name",

"Value" : "DR-DEMOAPPWEBSERVER",

"PropagateAtLaunch" : true

}]

}

},

"LaunchConfig" : {

"Type" : "AWS::AutoScaling::LaunchConfiguration",

"Properties" : {

"ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" },

{ "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" },

"Arch" ] } ] },

"UserData" : { "Fn::Base64" : { "Ref" : "WebServerPort" }},

"SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" } ],

"InstanceType" : { "Ref" : "InstanceType" },

"KeyName" : { "Ref" : "KeyPairName" },

"AssociatePublicIpAddress" : "false"

}

},

"ElasticLoadBalancer" : {

"Type" : "AWS::ElasticLoadBalancing::LoadBalancer ",

"Properties" : {

"SecurityGroups" : [ { "Ref" : "LoadBalancerSecurityGroup" } ],

"Subnets" : ["subnet-90cdd6b8","subnet-6024f917"],

"Listeners" : [ {

"LoadBalancerPort" : "80",

"InstancePort" : { "Ref" : "WebServerPort" },

"Protocol" : "HTTP"

} ],

"HealthCheck" : {

"Target" : { "Fn::Join" : [ "", ["HTTP:", { "Ref" : "WebServerPort" }, "/index.html"]]},

"HealthyThreshold" : "2",

"UnhealthyThreshold" : "10",

"Interval" : "10",

"Timeout" : "3"

}

}

},

"LoadBalancerSecurityGroup" : {

"Type" : "AWS::EC2::SecurityGroup",

"Properties" : {

"GroupDescription" : "Enable HTTP access on port 80",

"VpcId" : "vpc-0eef506b",

"SecurityGroupIngress" : [ {

"IpProtocol" : "tcp",

"FromPort" : "80",

"ToPort" : "80",

"CidrIp" : "0.0.0.0/0"

} ],

"SecurityGroupEgress" : [ {


"FromPort" : { "Ref" : "WebServerPort" },

"ToPort" : { "Ref" : "WebServerPort" },

"CidrIp" : "0.0.0.0/0"

} ]

}

},

"myDNS" : {

"Type" : "AWS::Route53::RecordSetGroup",

"Properties" : {

"HostedZoneName" : "awsdrdemo.com.",

"Comment" : "Zone apex alias targeted to myELB LoadBalancer.",

"RecordSets" : [

{

"Name" : "www.awsdrdemo.com.",

"Type" : "A",

"AliasTarget" : {

"HostedZoneId" : { "Fn::GetAtt" : ["ElasticLoadBalancer", "CanonicalHostedZoneNameID"] },

"DNSName" : { "Fn::GetAtt" : ["ElasticLoadBalancer","CanonicalHostedZoneName"] }

}

}

]

}

},

"InstanceSecurityGroup" : {


"Properties" : {

"GroupDescription" : "Enable RDP access and HTTP access on the inbound port",




"FromPort" : { "Ref" : "WebServerPort" },

"ToPort" : { "Ref" : "WebServerPort" },

"CidrIp" : "0.0.0.0/0"

},

{


"FromPort" : "3389",

"ToPort" : "3389",

"CidrIp" : "0.0.0.0/0"

} ]

}

}

},

"Outputs" : {

"URL" : {

"Description" : "URL of the website",

"Value" : { "Fn::Join" : [ "", [ "http://", { "Fn::GetAtt" : [ "ElasticLoadBalancer", "DNSName" ]}]]}

}

}

}

Create ELB

Create Auto Scaling group

Launch instances

Route 53 recordset updates

Status notifications

# Retrieve ELB Zone Id from existing Route53 zone

zoneid=$(aws --output text route53 list-resource-record-sets --hosted-zone-id $hostedzoneid --start-record-name $domainname --start-record-type A --max-items 1 | grep ALIASTARGET | awk {'print $4'})

# Retrieve ELB DNS name from existing Route53 zone

dns=$(aws --output text route53 list-resource-record-sets --hosted-zone-id $hostedzoneid --start-record-name $domainname--start-record-type A --max-items 1 | grep ALIASTARGET | awk {'print $2'})

# Remove existing Alias from Route53 hosted zone

aws --region us-west-1 route53 change-resource-record-sets --hosted-zone-id $hostedzoneid --change-batch "{ \"Changes\": [ { \"Action\": \"DELETE\", \"ResourceRecordSet\": { \"Name\": \"$domainname\", \"Type\": \"A\", \"AliasTarget\": { \"HostedZoneId\": \"$zoneid\", \"DNSName\": \"$dns\", \"EvaluateTargetHealth\": false } } } ]}"

# Launch DR stack using CloudFormation script

launchedstackid=$(aws --region $snsregion --output text cloudformation create-stack --stack-name $stackname --template-body file:///usr/local/bin/ELBWithEC2Instances.template --notification-arns $snsarn --parameters ParameterKey="HostedZoneId",ParameterValue="$hostedzoneid")

# Check the CloudFormation status

do cmd=$(aws cloudformation describe-stacks --region $snsregion --output text --stack-name $stackname --max-items 1 | grep CREATE_COMPLETE)

"Parameters" : {

"KeyPairName" : {

"Description" : "Name of an existing Amazon EC2 key pair for SSH access",

"Type" : "String",

"Default" : “drkey"

},

"InstanceType" : {

"Description" : "WebServer EC2 instance type",

"Type" : "String",

"Default" : "m3.medium",

"AllowedValues" : [ "t1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m2.xlarge","m2.2xlarge","m2.4xlarge","m3.medium","m3.large","m3.xlarge","m3.2xlarge","c1.medium","c1.xlarge"],

"ConstraintDescription" : "must be a valid EC2 instance type."

},

"WebServerPort" : {

"Description" : "TCP/IP port of the web server",

"Type" : "String",

"Default" : "80"

},

"HostedZoneId" : {

"Type" : "String",

"Description" : "The Record Set's Hosted Zone Id for the existing hosted zone",

"Default" : "Z1M58G0W56PQJA"

}

},

"Resources" : {

"WebServerGroup" : {

"Type" : "AWS::AutoScaling::AutoScalingGroup",

"Properties" : {

"AvailabilityZones" : [ "us-east-1c","us-east-1d"],

"LaunchConfigurationName" : { "Ref" : "LaunchConfig" },

"MinSize" : "2",

"MaxSize" : "2",

"LoadBalancerNames" : [ { "Ref" : "ElasticLoadBalancer" }],

"VPCZoneIdentifier" : ["subnet-d8f2edf0","subnet-2fc76658"],

"Tags" :[ {

"Key" : "Name",

"Value" : "DR-DEMOAPPWEBSERVER",

"PropagateAtLaunch" : true

}]}},

"LaunchConfig" : {

"Type" : "AWS::AutoScaling::LaunchConfiguration",

"Properties" : {

"ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" }, { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" },"Arch" ] } ] },

"UserData" : { "Fn::Base64" : { "Ref" : "WebServerPort" }},

"SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" } ],

"InstanceType" : { "Ref" : "InstanceType" },

"KeyName" : { "Ref" : "KeyPairName" },

"AssociatePublicIpAddress" : "false"

}

"Resources" : {

"ElasticLoadBalancer" : {

"Type" : "AWS::ElasticLoadBalancing::LoadBalancer",

"Properties" : {

"SecurityGroups" : [ { "Ref" : "LoadBalancerSecurityGroup" } ],

"Subnets" : ["subnet-90cdd6b8","subnet-6024f917"],

"Listeners" : [ {

"LoadBalancerPort" : "80",

"InstancePort" : { "Ref" : "WebServerPort" },

"Protocol" : "HTTP”} ],

"HealthCheck" : {

"Target" : { "Fn::Join" : [ "", ["HTTP:", { "Ref" : "WebServerPort" }, "/index.html"]]},

"HealthyThreshold" : "2",

"UnhealthyThreshold" : "10",

"Interval" : "10",

"Timeout" : "3”}}},

"LoadBalancerSecurityGroup" : {


"Properties" : {

"GroupDescription" : "Enable HTTP access on port 80",




"FromPort" : "80",

"ToPort" : "80",

"CidrIp" : "0.0.0.0/0"

} ]

"myDNS" : {

"Type" : "AWS::Route53::RecordSetGroup",

"Properties" : {

"HostedZoneName" : "awsdrdemo.com.",

"Comment" : "Zone apex alias targeted to myELB LoadBalancer.",

"RecordSets" : [

{

"Name" : "www.awsdrdemo.com.",

"Type" : "A",

"AliasTarget" : {

"HostedZoneId" : { "Fn::GetAtt" : ["ElasticLoadBalancer", "CanonicalHostedZoneNameID"] },

"DNSName" : { "Fn::GetAtt" : ["ElasticLoadBalancer","CanonicalHostedZoneName"] }

}

}

]

BufferedReader in = new BufferedReader(new InputStreamReader(request.getInputStream();

while((line = in.readLine()) != null) {

jsonBody+=line; }

if(request.getHeader("x-amz-sns-message-type").equalsIgnoreCase("SubscriptionConfirmation")){

System.out.println("Processing Subscription Message !");

String subscribeURL = SNSJSONParser.parseJSONString(jsonBody, "SubscribeURL");try {

sendGet(subscribeURL);System.out.println("Send Confirmation GET to subscription URL");

} catch (Exception e) {

e.printStackTrace();

System.err.println("Error confirming subscription via GET request..");}

}else{

System.out.println("Processing Notification Message !");

String snsMessage = SNSJSONParser.parseJSONString(jsonBody, "Message");s

String resourceStatusReason = getSubString(snsMessage, "ResourceStatusReason");

message+="<li>" + resourceId + " :: " + resourceStatus + " :: " + resourceStatusReason + "</li>”;}else{

message+= "<li>" + snsMessage + "</li>";

}

}

(99.99%)

(0-2 minutes)

0-15 minutes)

http://bit.ly/awsevals

http://bit.ly/awsevals


www.awsdrdemo.com

RDP

GW

RDP

GWRDP

GW

Web

/ app Web

/ app

Web

/ app

Web

/ app

Web

/ appWeb

/ app

AD

sql2

AD2AD1

SQL1

Northern Virginia


RDP

GW

RDPGWRDP

GW

Web

/ app

Oregon


RDP

GW

AD1 AD2 AD3

Web

/ app

Web

/ app

SQL1 SQL2 SQL3

www.awsdrdemo.com

Web

/ appWeb

/ app

(bac404) deploying high availability and disaster recovery architectures with aws | aws re:invent...

Technology