cloud computing with aws

Tim BixlerFederal Solutions Architecture Manager & Principal Solutions Architect

Worldwide Public Sector October 11, 2012

Cloud Computing With AWS An Overview

background

…get into cloud computing?

How did Amazon…

?

Consumer Business

Tens of millions of active customer

accounts

Eight countries:US, UK, Germany,

Japan, France, Canada, China, Italy

SellerBusiness

Sell on Amazon websites

Use Amazon technology for your own retail website

Leverage Amazon’s massive fulfillment

center network

IT InfrastructureBusiness

Cloud computing infrastructure for hosting web-scale

solutions

Hundreds of thousands of

registered customers in over 190 countries

Over 10 years in the making

Enablement of sellers on Amazon

Internal need for scalable deployment environment

Early forays proved developers were hungry for more

AWS Mission

Enable businesses and developers to use web services* to build scalable,

sophisticated applications.

*What people now call “the cloud”

Utility computing

On demand Pay as you go

Uniform Available

Utility

Infrastructure


Uniform Available

Utility computing

Compute

Storage

SecurityScaling

Database

NetworkingMonitoring

Messaging

Workflow

DNS

Load Balancing

BackupCDN


Uniform Available

Utility computing

No Up-Front Capital Expense

Up-Front On-Premise Costs

Physical Space

Cabling

Power

Cooling

Networking

Racks

Servers

Storage

Certification

Labor

On-Premise Variable Cloud Computing Costs

$0to Get Started

no long-term contracts

Cloud Computing

Traditional IT

capacity

Elastic capacity

Capacity

TimeYour IT needs

On and Off Fast Growth

Variable peaks Predictable peaks

Elastic capacity

On and Off Fast Growth

Predictable peaksVariable peaks

WASTE

CUSTOMER DISSATISFACTION

Elastic capacity

Fast GrowthOn and Off

Predictable peaksVariable peaks

Elastic capacity

From one compute instance…

…to thousands

Num

ber

of E

C2

Inst

ance

s

Day 1 Day 3 Day 4 Day 5 Day 7 Day 8 Day 9Day 6Day 2

Case Study

40 servers to 5000 in 3 days

EC2 scaled to peak of 5000 instances

“Techcrunched”

Launch of Facebook modification

Steady state of ~40 instances

Each day AWS adds the equivalent server capacity to power Amazon when it was a global, $2.76B enterprise

(circa 2000)

2012

632011

82

2010

612009

48

2008

24

2007

9 Amazon FPSRed Hat EC2

SimpleDBCloudFrontEBSAvailability ZonesElastic IPs

Relational Database ServiceVirtual Private Cloud

Elastic Map ReduceAuto Scaling

Reserved InstancesElastic Load Balancer

Simple Notification ServiceRoute 53RDS Multi-AZSingapore RegionIdentity Access ManagementCluster Instances

Elastic BeanstalkSimple Email ServiceCloudFormationRDS for OracleElastiCache

DynamoDBSimple Workflow

CloudSearchStorage Gateway

Route 53 Latency Based Routing

number of released features, sample services described

Government Customers

System Integrators Independent Software Vendors

Large Partner Ecosystem

Operating Systems Languages & Libraries Certified Applications

AWS is Open and Flexible

AWS: Services Drill-downs

AWS Platform

Your Applications

Foundation Services

ComputeAmazon EC2Auto Scale

StorageAmazon S3

Amazon EBSAmazon StorageGateway

DatabaseAmazon RDS

Amazon SimpleDBAmazon ElastiCacheAmazon DynamoDB

NetworkingAmazon VPC

Elastic Load BalancingAmazon Route 53

AWS Direct Connect

Management & Administration

Application Platform Services

Content DistributionAmazon CloudFront

Application SvcsSimple Workflow Service

CloudSearchAmazon SNS, SQS, SES

Parallel ProcessingElastic MapReduce

Libraries & SDKsJava, PHP, Python,

Ruby, .NET

Identity & AccessAWS IAM

Identity FederationConsolidated Billing

Web InterfaceManagement Console

MonitoringAmazon CloudWatch

Deployment & AutomationAWS Elastic BeanstalkAWS CloudFormation

AWS Global InfrastructureRegions

Availability ZonesEdge Locations

AWS Global Infrastructure

AWS Regions (8)

AWS Edge Locations (33)

US West(Northern California)

US East(Northern Virginia)

EU(Ireland)

Asia Pacific

(Singapore)

Asia Pacific(Tokyo)

GovCloud(US ITAR Region)

US West(Oregon)

South America(Sao Paulo)

AWS Regions & Availability Zones

Customer Decides Where Applications and Data ResideNote: Conceptual drawing only. The number of Availability Zones may vary.

Built to Enterprise & Gov Standards

Security & Compliance Resources

• Security & Compliance Center: http://aws.amazon.com/security

• Security Overview & Best Practices

• AWS Risk & Compliance Whitepaper

• Creating HIPAA Compliant Applications

Hardware, Software & Network

• Systematic change management

• Phased updates deployment

• Safe storage decommission

• Automated monitoring and self-audit

• Advanced network protection systems

Certifications and Accreditations• ISO 27001

• SSAE 16 / ISAE 3402 / SOC1 (formerly U.S. standard SAS-70 Type II)

• FISMA Moderate & DIACAP Controls; ITAR region

• HIPAA applications certified on AWS

• Payment Card Industry (PCI) Data Security Standard (DSS) Level 1

Physical

• Datacenters in nondescript facilities

• Physical access strictly controlled

• Must pass two-factor authentication at least twice for floor access

• Physical access logged and audited

Foundation Services

Your Applications

Foundation Services


StorageAmazon S3


DatabaseAmazon RDS




AWS Direct Connect








Ruby, .NET








Compute

Auto Scaling

Virtual Servers

Compute

EC2 Instances = Virtual Servers

• Resizable compute capacity in 14 instance types

• Reduces the time required to obtain and boot new server instances to minutes or seconds

• Scale capacity as your computing requirements change

• Pay only for capacity that you actually use

• Choose Linux or Windows

• Deploy across Regions and Availability Zones for reliability

• Flexible networking (NAT/classic, VPC, Elastic IPs)

• Support for virtual network interfaces that can be attached to EC2 instances in your VPC

Amazon Elastic Compute Cloud (Amazon EC2)

Compute

Amazon Elastic Compute Cloud (Amazon EC2) 12864

32

16

8

4

2

1

1 2 4 8 16 32 64 128EC2 Compute Units (HP)

Mem

ory

(GB)

Small 1.7 GB, 32-Bit1 EC2 Compute Unit1 virtual core$0.08/0.115

Micro 613 MB Up to 2 ECUs (for short bursts)$0.02/0.03

Large 7.5 GB4 EC2 Compute Units 2 virtual cores$0.32/0.46

Extra Large 15 GB 8 EC2 Compute Units4 virtual cores $0.64/0.92

Hi-Mem XL 17.1 GB6.5 EC2 Compute Units 2 virtual cores $0.45/0.57

Hi-Mem 2XL 34.2 GB13 EC2 Compute Units4 virtual cores $0.90/1.14

Hi-Mem 4XL 68.4 GB26 EC2 Compute Units 8 virtual cores $1.80/2.28

High-CPU Med 1.7 GB 5 EC2 Compute Units 2 virtual cores $0.165/0.285

High-CPU XL 7 GB 20 EC2 Compute Units8 virtual cores $0.66/1.14

Cluster GPU 4XL 22 GB 33.5 EC2 Compute Units, 2 x NVIDIA Tesla “Fermi” M2050 GPUs$2.10/2.60

Cluster Compute 4XL 23 GB 33.5 EC2 Compute Units$1.30/1.61

Cluster Compute 8XL 60.5 GB 88 EC2 Compute Units$2.40/2.97

Medium 3.75 GB2 EC2 Compute Units 1 virtual cores$0.16/0.23

High I/O 4XL 60.5 GB35 EC2 Compute Units$3.10/3.58

Compute

• Client Defined Business Rules

• Scale your Amazon EC2 capacity automatically once you define the conditions (may be 1000’s of servers)

• Can scale up just a little…doesn’t need to be massive number of servers (may be simply 2 servers)

• Well suited for applications that experience variability in usage

• Set minimum and maximum scaling policies

• Alternate Use is for Fault Tolerance

Auto Scaling

Storage

S3

EBS

Import/Export

Glacier

Storage Gateway

so new we don’t have an icon!

Storage

Web-scale Internet Storage

• A “Bucket” is equivalent to a “folder”

• Able to store unlimited number of Objects in a Bucket

• Objects from 1B-5 TB; no bucket size limit

• Highly available storage for the Internet (object store)

• HTTP/S endpoint to store and retrieve any amount of data, at any time, from anywhere on the web

• Highly scalable, reliable, fast, and inexpensive

• Over 1 trillion objects stored

• Peak requests 750,000+ per second

• Ideal Use Cases:

• Static web content – often used with CloudFront CDN

• Source and output storage for large-scale “Big Data” analytics

• Backup, archival, and DR storage that is always “live”

Simple Storage Service (S3)

Q4 2006

Q4 2007

Q4 2008

Q4 2009

Q4 2010

Q4 2011

Q2 20120.000

250.000

500.000

750.000

1000.000

1 Trillion

750,000+ peak transactions per second

Objects in S3

Storage

EBS Volumes = Virtual Disks

• Use for persistent storage

• Can use to create RAID configuration for a server

• Off-instance block storage that persists independently

• Storage volumes for use with Amazon EC2 instances – create, attach, backup, restore and delete

• Can be attached to a running Amazon EC2 instance and exposed as a block device for raw or formatted (filesystem) access

• Volumes behave like unformatted block devices for Linux or Windows instances

• Ideas use cases:

• OS Boot device / root file system; secondary volumes/filesystems

• Typical basis for database storage

• Raw block devices for RAID, some databases

Elastic Block Store (EBS)

Storage

• A low-cost storage service for data archiving and backup

• $0.01 per GB / Month

• Optimized for data that is infrequently accessed

• Retrieval times measured in hours not days or weeks

• Annual durability of 99.999999999% for an archive

• AES 256 data at rest encryption

• Data stored as archives within a vault. Vaults are located within a specific AWS region

• Archives can be up-to 40 TB in size

AWS Glacier

Storage

• Accelerates moving large amounts of data into and out of S3 or EBS

• Transfers your data directly onto and off of USB or SATA storage devices shipped to AWS with manifest file

• Final copy uses high-speed datacenter network

AWS Import/Export

Storage

• Storage gateway service connects an on-premise software appliance with cloud-based storage

• On-premises software appliance solution to store data on Amazon S3’s storage infrastructure

• Exposes standard iSCSI interface to on-premises applications, while maintaining low-latency data access

• Data in Amazon S3 stored as Amazon EBS snapshots for local & EC2-based recovery

• Use Cases

• Backup/Restore on-premise data

• Set up a test/dev environment with production data

• Migrating applications to the cloud

• On-premise DR/COOP to AWS

AWS Storage Gateway

Database

SimpleDB

DynamoDB

RDSRDS

ElastiCache

Database

• Fully managed NoSQL database.

• Eliminates the administrative burden of data modeling, index maintenance, and performance tuning.

• Durability and high-availability - stores data on Solid State Drives (SSDs) and replicates it synchronously across multiple AWS Availability Zones in an AWS Region.

• Scalability - With AWS Console, you can grow your DynamoDB table from 10 to 100,000 writes per sec.

• See video: http://www.youtube.com/watch?v=oz-7wJJ9HZ0

DynamoDB

http://www.youtube.com/watch?v=oz-7wJJ9HZ0

http://www.youtube.com/watch?v=oz-7wJJ9HZ0

RDS

Database

• Fully-managed, tuned MySQL, Oracle 11g, or MS SQL databases

• Cost-efficient and resizable capacity

• Manages time-consuming database admin tasks

• Code, applications, and tools you already use today work seamlessly

• Automatically patches the database software and backs up your database

• Flexible Licensing: BYOL or License Include

Amazon Relational Database Service (RDS)

Database

• Fully-managed, distributed, in-memory cache

• Memcached compliant cache cluster on-demand

• Manages patching, cache node failure detection and recovery

• Simple APIs calls to grow and shrink the cache cluster

• Seamlessly caches in front of SimpleDB or RDS instances

• Integrated with CloudWatch and SNS for monitoring and alerts

Amazon ElastiCache

Database

• Core database functions of data indexing and querying of text data

• No schema, automatic indexing

• Eliminates the administrative burden of data modeling, index maintenance, and performance tuning

• Real-time lookup and simple querying of structured data

• Use cases:

• Metadata storage -- often used in conjunction with S3

• Structured, fine-grained data needing query

• Data needing flexible schema

Amazon SimpleDB

Networking

ELB VPCRoute 53

Networking

• Supports the routing and load balancing of HTTP, HTTPS and generic TCP traffic to EC2 instances

• Supports health checks to ensure detect and remove failing instances

• Dynamically grows and shrinks required resources based on traffic

• Seamlessly integrates with Auto-scaling to add and remove instances based on scaling activities

• Single CNAME provides stable entry point for DNS configuration

Amazon Elastic Load Balancing

Networking

• Route end users to Internet applications

• Answers DNS queries with low latency by using a global network of DNS servers

• Latency based routing to closest AWS endpoint (e.g. EC2 instances, Elastic IPs or ELBs)

• Deep integration with other AWS services (ELB, EC2 NAT/EIP, etc.)

Amazon Route 53

Networking

• Secure and seamless bridge between a company’s existing private network and the AWS cloud

• Connect existing infrastructure to a set of isolated AWS compute resources via a Virtual Private Network (VPN) connection

• Bring your own address space and extend existing management capabilities

Amazon Virtual Private Cloud (VPC)

Networking


Your Applications

Foundation Services


StorageAmazon S3


DatabaseAmazon RDS




AWS Direct Connect








Ruby, .NET








Content Delivery

• Web service for content delivery

• Distribute content to end users with low latency, high data transfer speeds, and no commitments

• Delivers your content using a global network of 33 edge locations

• Supports download, streaming, live streaming, and dynamic content

• Key features: RTMP Streaming, HTTPS Delivery, Private Content for HTTP & Streaming, Programmatic Invalidation, Detailed Logs for HTTP & Streaming, Default Root Object

• Use Cases: Video and Rich Media, Online Gaming, Interactive Agencies, Software Downloads, Static Websites

• Static web content that must be delivered to global user base at Highest bandwidth / Lowest latency / Lowest cost

Amazon CloudFront

Application Services

SNS

SQS

SES

SWF

CloudSearch


• Set up, operate, and send notifications

• Publish messages from an application and immediately deliver them to subscribers or other applications

Amazon Simple Notification Service (SNS)


• Hosted queue for storing messages as they travel between computers

• Move data between distributed components of their applications

Amazon Simple Queue Service (SQS)


• Bulk and transactional email-sending service

• Eliminates the hassle of email server management, network configuration, and meeting rigorous Internet Service Provider (ISP) standards

• Provides a built-in feedback loop, which includes notifications of bounce backs, failed and successful delivery attempts, and spam complaints

Amazon Simple Email Service (SES, beta)


• Easily manage workflows, including state, decisions, executions, tasks and logging

• Coordinate processing steps across distributed systems

• Ensure tasks are executed reliably, in order, and without duplication

• Simple API calls that can be executed from code written in any language and run on your EC2 instances, or any of your machines located anywhere in the world that can access the Internet

Amazon Simple Workflow Service (SWF)


• Fully-managed search service

• Integrate fast and highly scalable search functionality into applications

• Scales automatically: with increases in searchable data or as query rate changes

• AWS manages hardware provisioning, data partitioning, and software patches

Amazon CloudSearch (beta)

Parallel Processing

• Managed Hadoop 0.20.205 infrastructure

• Reduces complexity of Hadoop management

• Handles node provisioning, customization, and shutdown

• Tunes Hadoop to your hardware and network

• Provides tools to debug and monitor your Hadoop clusters

• Provides tight integration with AWS services

• Optimized for Amazon Simple Storage Service (S3)

• EC2 integration with automatic re-provisioning on node failure

• Cluster monitoring/alarming through CloudWatch

• Leverages significant operational experience

• Monitor thousands of clusters per day

• Use cases span from University students to Fortune 50

Amazon Elastic MapReduce (EMR)

Libraries & SDKs

• Your choice of programming language (Java, PHP, Python, Ruby, .NET) and mobile platform (Android, iOS)

• The Developer Centers contains sample code, documentation, tools, and additional resources to help you build applications on Amazon Web Services.

• http://aws.amazon.com/java/

• http://aws.amazon.com/mobile/

• http://aws.amazon.com/php/

• http://aws.amazon.com/python/

• http://aws.amazon.com/ruby/

• http://aws.amazon.com/net/

http://aws.amazon.com/java/

http://aws.amazon.com/java/

http://aws.amazon.com/mobile/

http://aws.amazon.com/mobile/

http://aws.amazon.com/php/

http://aws.amazon.com/php/

http://aws.amazon.com/python/

http://aws.amazon.com/python/

http://aws.amazon.com/ruby/

http://aws.amazon.com/ruby/

http://aws.amazon.com/net/

http://aws.amazon.com/net/


Your Applications

Foundation Services


StorageAmazon S3


DatabaseAmazon RDS




AWS Direct Connect








Ruby, .NET








Web Console

On-demand, Self Service Management Access

Identity & Access Management

• IAM enables customers to create and manage users in AWS’s identity system

• Identity Federation with local directory is an option for enterprises

• Very familiar security model

• Users, groups, permissions

• Allows customers to

• Create users

• Assign individual passwords, access keys, multi-factor authentication devices

• Grant fine-grained permissions

• Optionally grant them access to the AWS Console

• Organize users in groups

Consolidated Billing with IAM

• Allows you to get one bill for multiple accounts

• You can easily track each account's costs and download the cost data in CSV format

• You may be able to reduce costs by combining usage from all the accounts to qualify for volume pricing discounts

Deployment and Management

• Simply upload your application (Java, NET, and PHP)

• Automatically handles the deployment details of capacity provisioning, load balancing, auto-scaling, and application health monitoring

• Retain full control over the AWS resources powering your application

AWS Elastic Beanstalk (beta)


• Create templates of stack of resources

• Deploy stack from template with runtime parameters

• Templates are simple JSON formatted text files

• CloudFormer supports generating templates from running environments

AWS CloudFormation

"Resources" : { "Ec2Instance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" } ], "ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]}, "Tags" : [{ "Key" : "MyTag", "Value" : "TagValue" }] } },


• Visibility into resource utilization, operational performance, and overall demand patterns

• Metrics such as CPU utilization, disk reads and writes, and network traffic

• Accessible via the AWS Management Console, web service APIs or Command Line Tools

• Add custom metrics of your own

• Alarms (which tie into auto-scaling, SNS, SQS, etc.)

• Billing Alerts to help manage charges on AWS bill

Amazon CloudWatch

Your Applications

Your Applications

Foundation Services


StorageAmazon S3


DatabaseAmazon RDS




AWS Direct Connect








Ruby, .NET








AWS: Big Data/HPC

A scalable compute platform

• Researchers and scientists want:– A platform that can scale– Offers choice at run time– Can be automated to run complex workflows– Don’t want to be bothered about the muck of

managing infrastructure• AWS provides Just-in-Time infrastructure

So you can go from one instance…

…to thousands

Small 1.7 GB, 1 ECU1 virtual core

Large 7.5 GB4 ECUs2 virtual cores

Extra Large 15 GB 8 ECUs4 virtual cores

Hi-Mem XL 17.1 GB6.5 ECUs2 virtual cores

Hi-Mem 2XL 34.2 GB13 ECUs4 virtual cores


High-CPU Med 1.7 GB 5 ECUs 2 virtual cores

High-CPU XL 7 GB 20 ECUs8 virtual cores

Micro 613 MB Up to 2 ECUs (for short bursts)

Cluster GPU 4XL 22 GB 33.5 ECUs8 Nehalem virtual cores 2 x NVIDIA Tesla “Fermi” M2050 GPUs

Cluster Compute 4XL 23 GB 33.5 ECUs 8 Nehalem virtual cores

Cluster Compute 8XL 60.5 GB 88 ECUs 8 core 2 x Intel Xeon

Medium 3.75 GB2 ECUs1 virtual cores

…and choose the EC2 instance type…

Small 1.7 GB, 1 ECU1 virtual core

Large 7.5 GB4 ECUs2 virtual cores

Extra Large 15 GB 8 ECUs4 virtual cores

Hi-Mem XL 17.1 GB6.5 ECUs2 virtual cores



High-CPU Med 1.7 GB 5 ECUs 2 virtual cores

High-CPU XL 7 GB 20 ECUs8 virtual cores

Micro 613 MB Up to 2 ECUs (for short bursts)

Cluster GPU 4XL 22 GB 33.5 ECUs8 Nehalem virtual cores 2 x NVIDIA Tesla “Fermi” M2050 GPUs

Cluster Compute 4XL 23 GB 33.5 ECUs 8 Nehalem virtual cores

Cluster Compute 8XL 60.5 GB 88 ECUs 8 core 2 x Intel Xeon

Medium 3.75 GB2 ECUs1 virtual cores

…and choose the EC2 instance type… Parallel workloads, high performance

computing

Processor intensive workloads, encoding,

modelingLow resource requirement applications

Average applications and workloads

Memory intensive applications, in-

memory computations

GPUs for Molecular Dynamics

GPU compute instancesIntel® Xeon® X5570 processors

2 x NVIDIA Tesla “Fermi” M2050 GPUs

I/O Performance: Very High (10 Gigabit Ethernet)

Cluster GPU

33.5 EC2 Compute Units

20GB RAM

2x NVIDIA GPU @ >400 Cores Each

CC2 Instance Cluster

240 TFLOPSMaking it the 72nd fastest

supercomputer in the world

Yours for $2554/hr – on demand

(Test performed Nov 2011, benchmark published June 2012 Top500 list. #42 on that list)

ec2-run-instances ami-b232d0db --instance-count 3 --availability-zone eu-west-1a --instance-type m1.small

A cluster that you can automate, control, auto-scale…

CLI, API and Console

Scripted configurations as-create-auto-scaling-group MyGroup --launch-configuration

MyConfig--availability-zones eu-west-

1a--min-size 2--max-size 200

Handle long running processes across many nodes and task

steps with Simple Workflow

Task A

Task B

(Auto-

scaling)Task C

2

3

1

…and coordinate workloads and task clusters in

Bid on unused Amazon EC2 capacity

Optimize cost with spot prices

Leverage Spot instances in workflows1 days worth of effort

resulted in 50% savings in cost

Harvard Medical SchoolThe Laboratory of Personal Medicine

Run EC2 clusters to analyze entire genomes

“The AWS solution is stable, robust, flexible, and low cost. It has everything to recommend it.”Dr. Peter Tonellato, LPM, Center for Biomedical Informatics, Harvard Medical School

Computation drug analysis

Estimated computation time 12.55 years

51,132 Core AWS ClusterCompleted in 3 hoursCosting $4828.85 / hr

Data Management

Data Ingestion

• AWS Import/Export– Move large amounts of data into and outside AWS– Data Migration, Content Distribution, DR, etc.

• AWS Direct Connect– Secure private link to AWS– 1Gbps, 10Gbps connectivity– You can also co-locate hardware in AWS DX locations

• Bandwidth Optimization Solutions– Commercial providers – Aspera, Riverbed, Attunity, etc.– Open Source – Tsunami UDP, Globus Online

AWS Direct Connect

AWS Import/Export

Data Collection

Relational Database

Service

Fully managed database

(MySQL, Oracle, MSSQL)

DynamoDB

NoSQL, Schemaless,

Provisioned throughput

database

S3

Object datastore up to

5TB per object

99.999999999%

durability

Fully managed SQL, NoSQL and object storage

Data Archival

• Announcing Amazon Glacier– Meet your regulatory requirements– Long term archival– 11 9’s of durability as S3 standard– All data encrypted using Server Side Encryption– Starting at $0.01/GB/month

“Every day our genome sequencers produce terabytes of data. As our company moves into the clinical space, we face a legal requirement to archive patient data for years that would drastically raise the cost of storage. Thanks to Amazon Glacier’s secure and scalable solution, we will be able to provide cost-effective, long-term storage and thereby eliminate a barrier to providing whole genome sequencing for medical treatment of cancer and other genetic diseases.” - Keith Raffel, Senior Vice President and Chief Commercial Officer, Complete Genomics

Share your data• Share Amazon Machine Images (AMIs)

– Share installations of your software packages and tools with collaborators so that they can duplicate your set up using EBS snapshots

– Collaborate by sharing your images with partners and customers

• Share architecture templates– Share the collection of resources required to run your pipeline with

collaborators by using CloudFormation templates

• Share data– Decouple your compute from data and share storage buckets with

collaborators– Create Requester Pays buckets so the charges associated with accessing

data are paid by the requesters

AWS Public Data Sets• A centralized repository of public datasets • Seamless integration with cloud based applications• No charge to the community• Some of the datasets available today:

– 1000 Genomes Project– Ensembl– GenBank– Illumina – Jay Flateley Human Genome Dataset– YRI Trio Dataset– The Cannabis Sativa Genome– UniGene– Influenza Virrus– PubChem

• Tell us what else you’d like for us to host …

Putting Foundation Services Together

deployment model: availability

1. Use multiple availability zones

2. Use RDS with replicas and standby

3. Use auto-scaling groups

4. Use Elastic Load Balancing

5. Use Route53 to host DNS zones

Three Services: Better Together

Utilization

Latency

Metrics

CloudWatch

Elastic Load Balancer

Auto Scaling

Use at regional levelCombined with autoscaling will balance requests and resource

capacity across availability zones

Within VPCUse to loadbalance between

application tiers within an availability zone

Instance migrations

Easily move instances from dev environments to test environments

by moving between ELBs

Leverage SLAImprove application reliability with Route 53’s SLA on requests served

Weighted routingPerform A/B analysis, and staged application roll-outs by moving a

portion of traffic to new infrastructure

Control TTLs and updatesTake absolute control of DNS

updates for more decisive system updates

Scale databases without admin overhead

Choose instance size for databases and scale up over time

Add high availability from management console

Create master-slave configurations and read-replicas. AWS takes care of the failover and recreation of a new

slave in event of master DB loss

Elastic Load Balancing Route 53 RDS

Dynamically scale resources & control costs

Only provision the resources that are required with scale up and cool down policies that match demand

Auto-scaling

Architect to use cloud strengths

Services not software

Less time managing and installing software

More time focused on mission applications

Your technology skillsUse AWS services

let AWS do the heavy lifting

+=

Relational Database ServiceDatabase-as-a-Service

No need to install or manage database instances

Scalable and fault tolerant configurations

DynamoDBProvisioned throughput NoSQL database

Fast, predictable performance

Fully distributed, fault tolerant architecture


Use RDS for databases

Use DynamoDB for high performance key-

value DB

Amazon SQS

Processing

task/processing

trigger

Processing results

Amazon SQSReliable, highly scalable, queue service

for storing messages as they travel

between instances


Task A

Task B

(Auto-scaling)

Task C

2

3

1

Simple WorkflowReliably coordinate processing steps

across applications

Integrate AWS and non-AWS resources

Manage distributed state in complex

systems

Push inter-process workflows into the cloud with SWF

Reliable message queuing without

additional software

Cloud SearchElastic search engine based upon

Amazon A9 search engine

Fully managed service with sophisticated

feature set

Scales automatically

DocumentServer

Results

SearchServer

Don’t install search software, use CloudSearch


Process large volumes of data cost effectively

with EMR

Elastic MapReduceElastic Hadoop cluster

Integrates with S3 & DynamoDB

Leverage Hive & Pig analytics scripts

Integrates with instance types such as

spot

Customer Case Studies

Case

Study

Challenge

Because of the latency of data transmission from and to Mars,

during a 2 hour window, it took mission planners 90 minutes to

process telemetry data from the Mars Rover, 20 mins to decide

where to move the Rover to, and 10 mins to up load the data.

Solution

NASA-JPL, loading their custom software application on EC2, was

able to horizontally scale the number of virtual machines

supporting the data processing.

Benefit

Reduced data processing time from 90 minutes to 15 minutes using

parallel processing

Increased mission planning time, resulting in high quality scientific

observations

NASA - Mission Data Processing

Case

Study

NASA - Mission Data Processing

Daily Mars Rover Data Processing Window

Process UploadPlan

Pre-cloud:

Process UploadPlan

Cloud:

Increase available mission planning time from 15 minutes to 105 minutes!

Case

Study

“We were able to reduce our DNS costs by ninety-three percent, which in tandem allowed us to

shorten our time-to-live (TTLs) for easier, timelier management of

DNS records.”

Nathan ButlerThe Newsweek/Daily Beast Company

Case

Study

Challenge

Recovery and Transparency Board needed a platform for their

website that was scalable, secure, could be quickly deployed, and

saved tax payer money

Solution

RATB chose a FISMA-compliant cloud computing solution based on

Amazon Web Services

deployed applications:

Microsoft SharePoint for web Content Management

Business Objection SAP for BI

Benefit• Avoided Capital expense, and added capacity to

scale up and down based on demand • Saved $750k per year in first year and additional

dollars from existing solution

RECOVERY.GOV – Website/App Hosting

“By migrating to the public cloud, the Recovery

Board is in position to leverage many advantages

including the ability keep the site up as millions of

Americans help report potential fraud, waste, and

abuse. The Board expects savings of about

$750,000 during its current budget cycle and

significantly more savings in the long-term.”

- Vivek Kundra, CIO, United

States

Security and Information: A Reprise

Foundation Services

Compute Storage Database Networking

AWS Global Infrastructure Regions

Availability Zones

Edge Locations

Client-side Data Encryption & Data Integrity Authentication

Server-side Encryption (File System and/or Data)

Network Traffic Protection(Encryption/Integrity/Identity)

Platform, Applications, Identity & Access Management

Operating System, Network & Firewall Configuration

Customer Data

Amaz

onCu

stom

erShared responsibility

• SAS-70 Type II• ISO 27001/ 2 Certification• Payment Card Industry (PCI) Data Security Standard (DSS)• FISMA Compliant Controls• DIACAP Compliant Controls• FedRAMP Compliant Controls• HIPAA and ITAR Compliant

Shared responsibility

• Facilities

• Physical Security

• Physical Infrastructure

• Network Infrastructure

• Virtualization Infrastructure

• Operating System

• Application

• Security Groups

• Network ACLs

• Network Configuration

• Account Management

AWS Customer

Examples of Customer Responsibilities

• Apply Your Information Management Program - that integrates Information Assurance

• Standardize Machine Images – create gold copy images for production deployment/to launch new instances

• Build and test in a sandbox environment – work out the bugs, figure out how to break it, architect to be resilient

• Do the same stuff you do in-house – quarterly patch management, IDS/IPS, logging, tripwire, etc.

• Conduct a Risk Assessment - to determine level of security controls you require

• Role Based Access Controls – restrict access to system components based upon need to know

• Use Encryption – for data in transit, for data at rest, filesystem

• Key Management – rotate keys used to access your resources (AWS does not hold these…you do)

• Setup Monitoring/Alerting – collect metrics and enable alerting for when events occur

• Vulnerability Scans – allowed via a permission process (else we’ll kill/block the source of scans)

• Prepare for Failure – create backups, store data in more than one location, test backups, have a contingency system ready

Build upon AWS features

Single TenantPhysical Nodes

Run your virtualized operating systems and apps in a “single tenant per physical node” model within the

AWS infrastructure

Instance firewallsFirewall control on instances via

Security Groups

CLIs and APIsInstantly audit your entire AWS

infrastructure from scriptable APIs – generate an on-demand IT inventory enabled by programmatic nature of

AWS

Subnet controlCreate low level networking

constraints for resource access, such as public and private subnets, internet gateways and NATs

Bastion hostsOnly allow access for management

of production resources from a bastion host. Turn off when not

needed

Dedicated Instances Security Groups VPC

Private connections to VPCSecured access to resources in AWS over software or hardware VPN and

dedicated network links

Direct Connect & VPN

AWS system entitlements

RolesAccount

Administrators Developers Applications

Bob

Kevin

Tomcat

Jim Brad

Mark

Susan

Reporting

Console

Multi-factor authentication

Groups

AWS Multi-Factor Authentication

• Helps prevent access based on unauthorized knowledge of your e-mail address and password

• Additional protection for account information

• Works with master account and IAM users

• Integrated into

• AWS Management Console

• Key pages on the AWS Portal

• S3 (Secure Delete)

• Virtual MFA (using OATH standard)

Account Management/Isolation

End User 4

End User 3

Cons

olid

ated

Bill

ing

Iden

tity

& A

cces

s M

anag

emen

t

End User 1

End User 2

End User 5

Linked AccountCustomer 1

End User 3

End User 1

End User 2

End User 3

End User 2

End User 1

End User 4

Reseller User 3

Reseller User 1

Reseller User 2

Reseller User 4

Linked AccountReseller Internal

Use



Payor Account

End User Group

AWS GovCloud – Who can Use?

• US Government/State/Local Clients & organizations conducting work on their behalf

• AWS will screen customers prior to providing access to the AWS GovCloud (US). Customers must be:

• U.S. Persons

• Not subject to export restrictions

• Agree to comply with U.S. export control laws and regulations, including the International Traffic In Arms Regulations

Useful Resources & Links

• Architecture Center: http://aws.amazon.com/architecture

• Security Center: http://aws.amazon.com/security

• Whitepapers: http://aws.amazon.com/whitepapers

• Resources: http://aws.amazon.com/resources

• Case Studies: http://aws.amazon.com/solutions/case-studies

• Solution Providers: http://aws.amazon.com/solutions/global-solution-providers/

http://aws.amazon.com/architecture

http://aws.amazon.com/architecture

http://aws.amazon.com/security

http://aws.amazon.com/security

http://aws.amazon.com/whitepapers

http://aws.amazon.com/whitepapers

http://aws.amazon.com/resources

http://aws.amazon.com/resources

http://aws.amazon.com/solutions/case-studies

http://aws.amazon.com/solutions/case-studies

http://aws.amazon.com/solutions/global-solution-providers/

http://aws.amazon.com/solutions/global-solution-providers/

Thank you

[email protected]

cloud computing with aws

Documents

day aws

power amazon

instances day

utility computing

elastic capacity waste

elastic capacity traditional

aws edge locations

aws missionenable businesses