zürich - amazon web services€¦ · security groups • restrict access to instances by: –port...
TRANSCRIPT
Introduction: Rinon Belegu
Rinon Belegu
Digicomp (Technical-Lead AWS and Veeam)Legendary IT (Owner)
Certification:
AWS Mentor, AWS Trainer, Microsoft Certified Trainer, Veeam Trainer
Cloud experience:
Buildup diffrent Cloud-Solutions 2007 – 2012 (Switzerland)
Implementation of Private-,Public- and Hybrid-Cloud [email protected]
Matthias Imsand
Founder Amanox Solutions (CTO)
Dipl. Ing. FH Informatik
AWS Instructor
Cloud & Linux Module Teacher @ FFHS
Introduction: Matthias ImsandDatacenter and Cloud Solutions
Cloud JourneyHow to successfully move to the cloud
ENABLING
- Trainings
- Workshops
- Seminars
- Labs
ASSESSMENT
- Analysis
- Potential, Readiness
- Strategy
- KPI Definition
- Architecture
PROOF OF CONCEPT
- Pilot projects
- Awareness programs
- Test installation
- Lesson learned
DATA MIGRATION
- Storage concept
- Security
- Migration planning
- Identity / Accounting
APPLICATION MIGRATION
- Forklift / Re-architecting
- Containerization
- Automation
- CI / CD
OPERATION
- Supported by Amanox
- Managed by Amanox
- Operated by Amanox
Amazon History
1994: Jeff Bezos incorporated the
company.
1995: Amazon.com launched its
online bookstore.
2005: Amazon
Publishing was
launched.
2006: Amazon
Web Services (AWS)
was launched.
2007: Kindle was
launched.
2011: Amazon
Fresh was launched.
2012: Amazon Game Studios was launched.
2013: Amazon Art was
launched.
2014: Amazon Prime
Now was launched.
2015: Amazon Home
Services and Amazon
Echo were launched.
Amazon Web Services (AWS)
ComputeMessaging
Mobile
App Services
Database
Networking
Development and
Management Tools
Payments
VPCOn-Demand Workforce
Analytics Content Delivery
Storage
Enable businesses and developers to
use web services to build scalable,
sophisticated applications.
2010
61
516
1,017
159
2012 2014 2016
AWS has been continually expanding its services to support virtually any cloud workload,
and it now has more than 90 services that range from compute, storage, networking,
database, analytics, application services, deployment, management, developer, mobile,
Internet of Things (IoT), Artificial Intelligence (AI), security, hybrid and enterprise
applications. AWS has launched a total of 1,017 new features and/or services year to
date* - for a total of 2,913 new features and/or services since inception in 2006.
AWS Pace of Innovation
1,950
AWS Direct
Connect
AWS Elastic Beanstalk
AWS GovCloud (US)
AWS CloudTrail
AWS CloudHSM
Amazon WorkSpaces
Amazon Kinesis
Amazon
AppStream
Amazon SNS
AWS Identity and Access
Management
Amazon Route 53
AWS Import/Export
Amazon SWF
Amazon Redshift
Amazon DynamoDB
Amazon CloudSearch
AWS Data
Pipeline
AWS Certificate Manager
AWS KMS
AWS Config
Amazon RDS
for Aurora
Amazon WorkDocs
AWS
Directory
Service
AWS CodeCommit
AWS CodePipeline
AWS Service
Catalog
Amazon CloudWatch Logs
Amazon EFS
Amazon API
Gateway
Amazon Machine
Learning
AWS Device Farm
AWS WAF
Elasticsearch Service
Amazon QuickSight
AWS Import/Export
Amazon RDS for MariaDB
Amazon Inspector
AWS IoT
Amazon EC2 Container
Registry
Amazon
ElastiCache
AWS
CloudFormation
Amazon Mobile
Analytics
AWS Mobile Hub
AWS Storage GatewayAWS OpsWorks
Amazon Elastic Transcoder
Amazon SES
Amazon EC2
Container Service
Amazon Cognito
AWS CodeDeploy
Amazon Glacier
Amazon WorkMail
AWS Lambda
Services and Features(February 1, 2016)
2,420
AWS Direct
Connect
AWS Elastic Beanstalk
AWS GovCloud (US)
AWS CloudTrail
AWS CloudHSM
Amazon WorkSpaces
Amazon Kinesis
Amazon
AppStream
Amazon SNS
AWS Identity and Access
Management
Amazon Route 53
AWS Import/Export
Amazon SWF
Amazon Redshift
Amazon DynamoDB
Amazon CloudSearch
AWS Data
Pipeline
AWS Certificate Manager
AWS KMS
AWS Config
Amazon RDS
for Aurora
Amazon WorkDocs
AWS
Directory
Service
AWS CodeCommit
AWS CodePipeline
AWS Service
Catalog
Amazon CloudWatch Logs
Amazon EFS
Amazon API
Gateway
Amazon Machine
Learning
AWS Device Farm
AWS WAF
Elasticsearch Service
Amazon QuickSight
AWS Import/Export
Amazon RDS for MariaDB
Amazon Inspector
AWS IoT
Amazon EC2 Container
Registry
Amazon
ElastiCache
AWS
CloudFormation
Amazon Mobile
Analytics
AWS Mobile Hub
AWS Storage GatewayAWS OpsWorks
Amazon Elastic Transcoder
Amazon SES
Amazon EC2
Container Service
Amazon Cognito
AWS CodeDeploy
Amazon Glacier
Amazon WorkMail
AWS Lambda
Services and Features(August 1, 2016)
2,913
AWS Direct
Connect
AWS Elastic Beanstalk
AWS GovCloud (US)
AWS CloudTrail
AWS CloudHSM
Amazon WorkSpaces
Amazon Kinesis
Amazon
AppStream
Amazon SNS
AWS Identity and Access
Management
Amazon Route 53
AWS Import/Export
Amazon SWF
Amazon Redshift
Amazon DynamoDB
Amazon CloudSearch
AWS Data
Pipeline
AWS Certificate Manager
AWS KMS
AWS Config
Amazon RDS
for Aurora
Amazon WorkDocs
AWS
Directory
Service
AWS CodeCommit
AWS CodePipeline
AWS Service
Catalog
Amazon CloudWatch Logs
Amazon EFS
Amazon API
Gateway
Amazon Machine
Learning
AWS Device Farm
AWS WAF
Elasticsearch Service
Amazon QuickSight
AWS Import/Export
Amazon RDS for MariaDB
Amazon Inspector
AWS IoT
Amazon EC2 Container
Registry
Amazon
ElastiCache
AWS
CloudFormation
Amazon Mobile
Analytics
AWS Mobile Hub
AWS Storage GatewayAWS OpsWorks
Amazon Elastic Transcoder
Amazon SES
Amazon EC2
Container Service
Amazon Cognito
AWS CodeDeploy
Amazon Glacier
Amazon WorkMail
AWS Lambda
Services and Features(January 1, 2017)
AWS Direct
Connect
AWS Elastic Beanstalk
Schema Conversion Tool
AWS Shield EFS
WorkSpaces
Amazon Lumberyard
Amazon
Pinpoint
AWS IoT
AWS Managed ServicesAmazon Route 53
AWS Import/Export
AWS OpsWorks for
Chef Automate
Redshift
Dynamo DB
Amazon Polly
AWS
Snowball
AWS Organizations
Device Farm
Amazon Config
Amazon RDS
for Aurora
WorkDocs
AWS
Snowball Edge
CodeCommit
AWS CodePipeline
AWS Service Catalog
CloudWatch Logs
Amazon Lex
AWS Greengrass
Amazon EC2
Systems Manager
AWS WAF
Amazon Appstream 2.0
Amazon
Athena
AWS Glue
Amazon Lightsail
Amazon Rekognition
AWS Step Functions
AWS Discovery
Services
AWS Certificate
Manager
Amazon
ElastiCache
Mobile
Analytics
AWS Mobile Hub
AWS Storage GatewayAWS OpsWorks
AWS Batch
Amazon Inspector
EC2
Container Service
Amazon Cognito
AWS CodeDeploy
AWS Personal Health Dashboard
AWS Snowmobile
Lambda
* As of 1 August 2017
AWS Codebuild
AWS X-Ray
Amazon QuickSight
Amazon Kinesis Firehose
Amazon
Workmail
Amazon Inspector
Machine Learning
3,567Services and Features
(August 1, 2017)
AWS Positioned as a Leader in the Gartner Magic Quadrant
for Cloud Infrastructure as a Service, Worldwide*
AWS is positioned
highest in execution
and furthest in
vision within the
Leaders Quadrant
*Gartner, Magic Quadrant for Cloud Infrastructure as a Service, Worldwide, Leong, Lydia, Petri, Gregor, Gill, Bob, Dorosh, Mike, August 32016
This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from AWS :
http://www.gartner.com/doc/reprints?id=1-2G2O5FC&ct=150519&st=sb
Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications
consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability
or fitness for a particular purpose.
AWS Core Infrastructure and Services
Security
Network
Security
Network
Security Groups NACLs Access Mgmt
VPCVPCEC2 “Classic”
“Public
”
ELBOn-Demand
Provision
Traditional Infrastructure Amazon Web Services
Servers
AMI Amazon EC2 InstancesOn-Premises Servers
SecuritySecurity Groups Network ACLs AWS IAMFirewalls ACLs Administrators
Storage and
DatabaseRDBMSDAS SAN NAS Amazon
EBSAmazon
EFSAmazon
S3
Amazon
RDS
NetworkingVPCELBRouter Network Pipeline Switch
Infrastructure Regions Edge LocationsAvailability Zones
Foundation
Services
Compute(Virtual, Auto-scaling and
Load Balancing)
Networking
ApplicationsVirtual
DesktopsCollaboration and Sharing
Platform
Services
AWS Cloud Computing
Databases
Relational
NoSQL
Caching
Analytics
Cluster
Computing
Real-time
Data
Warehouse
Data
Workflows
App Services
Queuing
Orchestration
App Streaming
Transcoding
Search
Deployment and
Management
Containers
Dev/ops Tools
Resource Templates
Usage Tracking
Monitoring and Logs
Mobile Services
Identity
Sync
Mobile Analytics
Notifications
Storage(Object, Block and Archive)
AWS Regions
OREGON
N. CALIFORNIA
AWS GOVCLOUD
OHIO
N. VIRGINIA
MONTREAL
SÃO PAULO
LONDON
IRELAND
FRANKFURT
BEIJING
SEOUL
TOKYO
NINGXIA (Coming soon)
INDIA
SINGAPORE
SYDNEY
PARIS (Coming soon)
AZ - A AZ - B
Region
AZ - C
AWS Availability Zones
• Each Availability Zone is:– Made up of one or more data
centers.
– Designed for fault isolation.
– Interconnected with other Availability Zones using high-speed private links.
• You choose your Availability Zones.
• AWS recommends replicating across AZs for resiliency.
Availability Zone
Ways to access AWS
AWS Management ConsoleEasy-to-use graphical interface that supports majority of Amazon Web Services.
Command Line Interface (CLI)Access to services via discrete commands that can be issued from a Linux
command line, Linux shell script, Windows cmd prompt, Windows batch file,
or Windows PowerShell.
Software Development Kits (SDKs)Launch EC2 instances, configure networks, etc. from most major programming
languages (Python, Ruby, .NET, Java, etc.).
Cloud Deployment Models
Data Center Cloud
• Upfront capital expense
• Provision hardware and staff for
normal operations and disaster
recovery (DR)
• Limited experimentation and
reusability
• Available when needed
• Build up, tear down and reuse with
ease
• Reduced cost and planning for DR,
storage redundancy
• More independence, innovation
within the company
Hybrid Model
• Connect data center and cloud resources
Amazon Virtual Private Cloud (VPC)
• Provision a private, isolated virtual network on
the AWS cloud.
• Have complete control over your virtual
networking environment.Amazon
VPC
Amazon VPC Example
Virtual Private Cloud
AWS Cloud
Public Subnet Private Subnet VPN Only Subnet
DB ServerWeb Server
Customer
Network
R
Internet
App ServerVPC NAT
Gateway
Internet
Gateway
Web Server App Server DB Server
Virtual
Private
Gateway
Amazon Elastic Compute Cloud (EC2)
• Resizable compute capacity
• Complete control of your computing resources
• Reduced time required to obtain and boot new
server instancesAmazon
EC2
Launching an Amazon EC2 Instance
1. Determine the AWS Region in which you want to launch the
Amazon EC2 instance.
2. Launch an Amazon EC2 instance from a pre-configured Amazon
Machine Image (AMI).
3. Choose an instance type based on CPU, memory, storage, and
network requirements.
4. Configure network, IP address, security groups, storage volume,
tags, and key pair.
1) Determine the AWS Region
Determine the right region for
your services, applications, and
data based on these factors.
Proximity to
customers (latency)
Data governance,
legal requirements
Services available
within the region
Costs (vary by region)
2) Launch
Select an AMI based on:
• Region
• Operating system
• Architecture (32-bit or 64-bit)
• Launch permissions
• Storage for the root device
AMI
Instances
Instance
Launch
instances of any
type
Host computer
Host computer
3) Choose an instance type
M4
General
purpose
Compute
optimized
C4
Storage and I/O
optimized
I3
GPU- or FPGA-
enabled
Memory
optimized
D2
M3
X1
I2 HS1
C3
T2
R4
R3
F1
P2
G2
4) Configure
Network placement
and addressing
Block Storage
Ephemeral or EBS
AMI
Tenancy
Instance
Server Role Security groups
Key pairs
User data
Security Groups
• Restrict access to instances by:
– Port range
– IP range
– Security group or resource ID
• Instances can be associated with
multiple security groups.
• Allow data ingress and egress.
• Can be added/modified after launch.
Remote Access 22
Web
Servers
DB
NAT
Web Traffic 80
port 3306
Remote Access 22
User Data
Are supplied to initialize instances automatically and can be a• Linux script
• Windows batch or PowerShell scripts
Can install any software package, such as • Web servers
• Database servers
• Configuration management tools
Are executed by• Cloud-init on Linux
• EC2Config service on Windows
Runs once per instance-id by default
Amazon Simple Storage Service (S3)
• Storage for the Internet
• Natively online, HTTP access
• Storage that allows you to store and retrieve any
amount of data, any time, from anywhere on the
web
• Highly scalable, reliable, fast and durable
Amazon S3
Amazon S3 Facts
• Can store an unlimited number of objects in a bucket
• Objects can be up to 5 TB; no bucket size limit
• Designed for 99.999999999% durability and 99.99% availability of objects over a given year
• Can use HTTP/S endpoints to store and retrieve any amount of data, at any time, from anywhere on the web
• Is highly scalable, reliable, fast, and inexpensive
• Can use optional server-side encryption using AWS or customer-managed provided client-side encryption
• Auditing is provided by access logs
• Provides standards-based REST and SOAP interfaces
AWS Storage Options: Block vs. Object Storage
Block Storage
Change one block (piece of the file)
that contains the character
Object Storage
Entire file must be updated
What if you want to change one character in a 1-GB file?
Common Use Scenarios
• Storage and backup
• Application file hosting
• Media hosting
• Software delivery
• Store AMIs and snapshots
Amazon S3 Concepts
To upload your data (photos, videos, documents, etc.):
1. Create a bucket in one of the AWS Regions.
2. Upload any number of objects to the bucket.
Amazon S3
[bucket name]
Preview2.mp4
Tokyo Region
(ap-northeast-1)
Bucket
Object
https://s3-ap-northeast-1.amazonaws.com/[bucket name]/Preview2.mp4
https://s3-ap-northeast-1.amazonaws.com/[bucket name]/
Region code Bucket name
Key
Amazon S3 Security
• You can control access to buckets and objects
with:– Access Control Lists (ACLs)
– Bucket policies
– Identity and Access Management (IAM) policies
• You can upload or download data to Amazon S3
via SSL encrypted endpoints.
• You can encrypt data using AWS SDKs.
Amazon S3 Versioning
• Protects from accidental overwrites and deletes with no performance penalty.
• Generates a new version with every upload.
• Allows easily retrieval of deleted objects or roll back to previous versions.
• Three states of an Amazon S3 bucket– Un-versioned (default)
– Versioning-enabled
– Versioning-suspendedVersioning Enabled
Key: photo.gif
ID: 121212
Key: photo.gif
ID: 111111
Amazon Glacier
• Long term low-cost archiving service
• Optimal for infrequently accessed data
• Designed for 99.999999999% durability
• Three to five hours’ retrieval time
• Less than $0.01 per GB/month (depending on
region)
Amazon
Glacier
S3 Lifecycle Policies
Amazon S3 lifecycle policies allow you to delete
or move objects based on age.
Amazon S3
Standard
Amazon S3
Standard -
Infrequent
Access
Delete
30 Days 60 Days 365 Days
Preview2.mp4 Preview2.mp4 Preview2.mp4
S3 Storage Class
Standard Standard - Infrequent
Access
Reduced Redundancy
StorageGlacier
Durability 99.999999999% 99.999999999% 99.99%99.999999999%
Availability 99.99% 99.9% 99.99% N/A
First Byte Latency ms ms ms 3-5h
Lifecycle Management
PoliciesYes Yes Yes Yes
Knowledge Check
Q: What AWS service would help support your web application by hosting static assets and storing user uploaded images and video off-instance?
Q: How would an EC2 instance find its private and public IP addresses?
Q: You want to deploy a new version of your web application. How do you trigger the user data to run again and update your app?
Q: True or False: S3 limits the total amount you can store.
False (There is a 5TB limit per object)
Retrieve the instance metadata. http://169.254.169.254/latest/meta-data/
Amazon S3
You don't. By default, user data is run once, when the instance first
boots.
AWS Shared Responsibility Model
AWS Foundation Services
Compute Storage Database Networking
AWS Global Infrastructure Regions
Availability ZonesEdge Locations
Client-side Data Encryption
Server-side Data Encryption
Network Traffic Protection
Platform, Applications, Identity, and Access Management
Operating System, Network, and Firewall Configuration
Customer Applications & ContentC
ust
om
ers
Customers are
responsible for
security IN the cloud
AWS is responsible
for the security OF
the cloud
Physical Security
• 24/7 trained security staff
• AWS data centers in nondescript
and undisclosed facilities
• Two-factor authentication for
authorized staff
• Authorization for data center
access
Hardware, Software, and Network
• Automated change-control
process
• Bastion servers that record
all access attempts
• Firewall and other boundary
devices
• AWS monitoring tools
Certifications and Accreditations
ISO 9001, ISO 27001, ISO 27017, ISO 27018, IRAP (Australia), MLPS Level 3 (China),
MTCS Tier 3 Certification (Singapore) and more …
SSL Endpoints
VPC
Secure Transmission
Use secure endpoints
to establish secure
communication
sessions (HTTPS).
Instance Firewalls
Use security groups
to configure firewall
rules for instances.
SSL Endpoints Security Groups
Network Control
Use public and
private subnets,
NAT, and VPN
support in your
virtual private cloud
to create low-level
networking
constraints for
resource access.
SSL Endpoints
Security Groups
SSL Endpoints Security Groups
Instance Firewalls
Use security groups
to configure firewall
rules for instances.
VPC
Secure Transmission
Use secure endpoints
to establish secure
communication
sessions (HTTPS).
Network Control
Use public and
private subnets,
NAT, and VPN
support in your
virtual private cloud
to create low-level
networking
constraints for
resource access.
HTTPPorts 80 and 443 only
open to the Internet
SSH/RDPEngineering staff have SSH/RDP
access to Bastion Host
AWS Multi-Tier Security Groups
Bastion
All other internet ports blocked by default
Amazon Virtual Private Cloud (VPC)
VPCSSL Endpoints Security Groups
Network Control
Use public and
private subnets,
NAT, and VPN
support in your
virtual private cloud
to create low-level
networking
constraints for
resource access.
Instance Firewalls
Use security groups
to configure firewall
rules for instances.
Secure Transmission
Use secure endpoints
to establish secure
communication
sessions (HTTPS).
AWS Identity and Access Management (IAM)
AWS IAM
3
Manage federated users
and their permissions
2
Manage AWS IAM roles
and their permissions
1
Manage AWS IAM users
and their access
AWS IAM Authentication
• Authentication
• AWS CLI or SDK API– Access Key and Secret Key
Access Key ID: AKIAIOSFODNN7EXAMPLE
Secret Access Key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Java Python .NET
AWS SDK & APIAWS CLI
IAM User
AWS IAM Authorization
Authorization
• Policies:
– Are JSON documents to describe
permissions.
– Are assigned to users, groups or
roles.
IAM User IAM Group
IAM Roles
AWS IAM Policy Elements
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1453690971587",
"Action": [
"ec2:Describe*",
"ec2:StartInstances",
"ec2:StopInstances”
],
"Effect": "Allow",
"Resource": "*",
"Condition": {
"IpAddress": {
"aws:SourceIp": "54.64.34.65/32”
}
}
},
{
"Sid": "Stmt1453690998327",
"Action": [
"s3:GetObject*”
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::example_bucket/*”
}
]
}
IAM Policy
AWS IAM Roles
• An IAM role uses a policy.
• An IAM role has no associated credentials.
• IAM users, applications, and services may
assume IAM roles.
IAM Roles
AWS IAM Policy Assignment
IAM UserIAM Group
IAM Roles
Assigned Assigned
Assig
ned
IAM Policy
IAM User
Assumed Assumed
AWS Resources
Example: Application Access to AWS Resources
• Python application hosted on an Amazon EC2
Instance needs to interact with Amazon S3.
• AWS credentials are required:– Option 1: Store AWS Credentials on the Amazon EC2 instance.
– Option 2: Securely distribute AWS credentials to AWS Services
and Applications.
IAM Roles
AWS IAM Roles - Instance Profiles
Amazon EC2
App &EC2 MetaData Service
http://169.254.169.254/latest/meta-data/iam/security-credentials/rolename
Amazon S3
1
2
3
4
Create Instance
Sele
ct
IAM
Ro
le
Ap
plic
atio
n in
tera
cts
with
S3
AWS IAM Roles – Assume Role
IAM Restricted Policy
IAM User A-1
AWS Account A
IAM Admin RoleIAM Admin Policy
Assigned
Assume
Assigned
1
2
IAM User B-1
AWS Account B
Amazon S3
Assume
4
Access
53
Access
1
Temporary Security Credentials (AWS STS)
Use Cases
• Cross account access
• Federation
• Mobile Users
• Key rotation for Amazon EC2-based apps
Session
Access Key ID
Secret Access Key
Session Token
Expiration
Temporary Security Credentials
15 minutes to 36 hours
AWS IAM Authentication and Authorization
• Authentication
– AWS Management Console
• User Name and Password
– AWS CLI or SDK API
• Access Key and Secret Key
• Authorization
– Policies
IAM User IAM Group
IAM Roles
AWS IAM Best Practices
• Delete AWS account (root) access keys.
• Create individual IAM users.
• Use groups to assign permissions to IAM users.
• Grant least privilege.
• Configure a strong password policy.
• Enable MFA for privileged users.
AWS IAM Best Practices (cont.)
• Use roles for applications that run on Amazon EC2 instances.
• Delegate by using roles instead of by sharing credentials.
• Rotate credentials regularly.
• Remove unnecessary users and credentials.
• Use policy conditions for extra security.
• Monitor activity in your AWS account.
AWS CloudTrail
• Records AWS API calls for accounts.
• Delivers log files with information to an Amazon
S3 bucket.
• Makes calls using the AWS Management
Console, AWS SDKs, AWS CLI and higher-level
AWS services.
AWS CloudTrail Amazon S3 Bucket
Logs
Continuous Monitoring
• 24/7 monitoring to detect incidents.
• Industry-standard diagnostic procedures to drive
resolution during business-impacting events.
• Preventative maintenance for continued
operability of equipment.
SQL and NoSQL Databases
SQL NoSQL
Data Storage Rows and Columns Key-Value
Schemas Fixed Dynamic
Querying Using SQL Focused on collection of
documents
Scalability Vertical Horizontal
ISBN Title Author Format
9182932465265 Cloud Computing
Concepts
Wilson,
Joe
Paperback
3142536475869 The Database
Guru
Gomez,
Maria
eBook
SQL NoSQL
{ISBN: 9182932465265,Title: “Cloud Computing Concepts”,Author: “Wilson, Joe”,Format: “Paperback”
}
Data Storage Considerations
• No one size fits all.
• Analyze your data requirements by considering:– Data formats
– Data size
– Query frequency
– Data access speed
– Data retention period
AWS Managed Database Services
Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment and Administration
Networking
Amazon DynamoDB
Amazon ElastiCache
Amazon RDS
Amazon Redshift
AWS Database Migration Service
Amazon Relational Database Service (RDS)
• Cost-efficient and resizable capacity
• Manages time-consuming database
administration tasks
• Access to the full capabilities of Amazon
Aurora, MySQL, MariaDB, Microsoft SQL
Server, Oracle, and PostgreSQL databases
Amazon
RDS
Amazon RDS
• Simple and fast to deploy
• Manages common database administrative tasks
• Compatible with your applications
• Fast, predictable performance
• Simple and fast to scale
• Secure
• Cost-effective
DB Instances
• DB Instances are the basic building blocks
of Amazon RDS.
• They are an isolated database
environment in the cloud.
• They can contain multiple user-created
databases.
How Amazon RDS Backups Work
• Automatic Backups:– Restore your database to a
point in time.
– Are enabled by default.
– Let you choose a retention
period up to 35 days.
• Manual Snapshots:– Let you build a new
database instance from a
snapshot.
– Are initiated by the user.
– Persist until the user deletes
them.
– Are stored in Amazon S3.
Cross-Region Snapshots
• Are a copy of a database snapshot stored in a different AWS Region.
• Provide a backup for disaster recovery.
• Can be used as a basefor migration to a different region.
Amazon RDS Security
• Run your DB instance in an Amazon VPC.
• Use IAM policies to grant access to Amazon RDS resources.
• Use security groups.
• Use Secure Socket Layer (SSL) connections with DB instances (Amazon Aurora, Oracle, MySQL, MariaDB, PostgreSQL, Microsoft SQL Server).
• Use Amazon RDS encryption to secure your RDS instances and snapshots at rest.
• Use network encryption and transparent data encryption (TDE) with Oracle DB and Microsoft SQL Server instances.
• Use the security features of your DB engine to control access to your DB instance.
A Simple Application Architecture
Amazon RDS database
instance
Amazon EC2
Application Servers
Elastic Load Balancing
load balancer instance
DB snapshots in
Amazon S3
Multi-AZ RDS Deployment
• With Multi-AZ operation, your database is
synchronously replicated to another
Availability Zone in the same AWS Region.
• Failover to the standby automatically occurs in
case of master database failure.
• Planned maintenance is applied first to standby
databases.
A Resilient, Durable Application Architecture
Amazon RDS database instances:
Master and Multi-AZ standby
Application, in Amazon
EC2 instances
Elastic Load Balancing
load balancer instance
DB snapshots in
Amazon S3
Amazon RDS Best Practices
• Monitor your memory, CPU, and storage usage.
• Use Multi-AZ deployments to automatically provision and maintain a synchronous standby in a different Availability Zone.
• Enable automatic backups.
• Set the backup window to occur during the daily low in WriteIOPS.
• To increase the I/O capacity of a DB instance:– Migrate to a DB instance class with high I/O capacity.
– Convert from standard storage to provisioned IOPS storage and use a DB instance class optimized for provisioned IOPS.
– Provision additional throughput capacity (if using provisioned IOPS storage).
• If your client application is caching the DNS data of your DB instances, set a TTL of less than 30 seconds.
• Test failover for your DB instance.
Amazon DynamoDB
• Allows you to store any amount of data with no limits.
• Provides fast, predictable performance using SSDs.
• Allows you to easily provision and change the request capacity needed for each table.
• Is a fully managed, NoSQL database service.
Amazon
DynamoDB
Provisioned Throughput
• You specify how much provisioned throughput
capacity you need for reads and writes.
• Amazon DynamoDB allocates the necessary
machine resources to meet your needs.
Supported Operations
• Query:– Query a table using the partition key and an optional sort key filter.
– If the table has a secondary index, query using its key.
– It is the most efficient way to retrieve items from a table or secondary index.
• Scan:– You can scan a table or secondary index.
– Scan reads every item – slower than querying.
• You can use conditional expressions in both Query and Scan operations.
Simple Application Architecture
Elastic Load
Balancing Amazon EC2
app instances
Clients
Amazon
DynamoDB
Business logic
Amazon RDS and Amazon DynamoDBFactors Relational (Amazon RDS) NoSQL (Amazon DynamoDB)
Application
Type
• Existing database apps
• Business process–centric apps
• New web-scale applications
• Large number of small writes and
reads
Application
Characteristics
• Relational data models,
transactions
• Complex queries, joins, and
updates
• Simple data models, transactions
• Range queries, simple updates
ScalingApplication or DBA–architected
(clustering, partitions, sharding)
Seamless, on-demand scaling based
on application requirements
QoS
• Performance–depends on data
model, indexing, query, and
storage optimization
• Reliability and availability
• Durability
• Performance–Automatically
optimized by the system
• Reliability and availability
• Durability
Database ConsiderationsIf You Need Consider Using
A relational database
service with minimal
administration
Amazon RDS
• Choice of Amazon Aurora, MySQL, MariaDB, Microsoft
SQL Server, Oracle, or PostgreSQL database engines
• Scale compute and storage
• Multi-AZ availability
A fast, highly scalable
NoSQL database
service
Amazon DynamoDB
• Extremely fast performance
• Seamless scalability and reliability
• Low cost
A database you can
manage on your own
Your choice of AMIs on Amazon EC2
and Amazon EBS that provide scale compute and
storage, complete control over instances, and more.
November Traffic to Amazon.com
Provisioned capacity
November
76%
24%
The challenge is to efficiently
‘guess’ the unknown quantity of
how much compute capacity you
need.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Enable Scalability (1 of 2)
• Ensure that your architecture can handle changes in demand.
A key advantage of a cloud-based
infrastructure is how quickly you can
respond to changes in resource needs.
Anti-pattern
App servers
at full
capacity
Admin
launches
new server
New server
takes time
to launch
Users
prevented
from access
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Enable Scalability (2 of 2)
Best practice
App servers
at alarm
threshold
Auto
Scaling is
alerted and
scales out
New server is
ready before
capacity is
reached
Users never
experience
interruption in
accessibility
Ensure that your architecture can handle changes in demand.
A key advantage of a cloud-based
infrastructure is how quickly you can
respond to changes in resource needs.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Vertical vs. Horizontal Scaling
• Vertical scaling
Scale up and down– Change in the specifications of
instances (more CPU, memory,
etc.)
Horizontal scaling
Scale in and out
• Change in the number of
instances (Add and remove
instances as needed)
small xlarge
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Triad of Services
Latency
Utilization
CloudWatchAuto Scaling
Elastic Load
Balancing
Auto Scaling group
Execute AS
Policy
Elastic Load Balancing
• Distributes traffic across multiple EC2 instances,
in multiple Availability Zones
• Supports health checks to detect unhealthy
Amazon EC2 instances
• Supports the routing and load balancing of
HTTP, HTTPS, SSL, and TCP traffic to Amazon
EC2 instances
Elastic Load
Balancing
Classic Load Balancer - How It Works
Register
instances
with your
load
balancer.
Availability Zone A Availability Zone B
load balancer
X
Target Group /mobile
Application Load Balancer – How It Works
Register instances as
targets in a target
group, and route
traffic to a target
group.
load balancer
Listener ListenerRule Rule Rule
Target Group Target Group /api
Target Target Target Target Target Target Target
Health
Check
Health
Check
Health
Check
ELB - Features
• Sticky Sessions
• Connection Draining
• Cross Zone Loadbalancing
• SSL Termination
• IPv6 Support
• Request Tracing (Header Injection)
• WAF Integration
• AWS Shield Integration (DDoS Protection)
Load Balancer Comparison
Classic Load Balancer
benefits include support
for:• EC2-Classic.
• VPC.
• TCP and SSL listeners.
• Sticky sessions.
• OSI Layer 4(network protocol level)
ALB benefits include
support for:• Path-based routing.
• Routing requests to multiple
services on a single EC2 instance.
• Containerized applications.
• Monitoring the health of each
service independently.
• OSI Layer 7(application level)
Amazon CloudWatch
• A monitoring service for AWS cloud resources and the applications you run on AWS
• Visibility into resource utilization, operational performance, and overall demand patterns
• Custom application-specific metrics of your own
• Accessible via AWS Management Console, APIs, SDK, or CLI
Amazon
CloudWatch
Amazon CloudWatch Architecture
AWS resources
that support
CloudWatch
Amazon
CloudWatch
Amazon
CloudWatch
Alarm
SNS Email
Notification
Auto Scaling
Available
Statistics
Statistics
Consumer
AWS Management
Console
CloudWatch Metrics
CPUUtilization
StatusCheckFailed
Custom
Application-
Specific Metrics
PageViewCount
CloudWatch Alarm Examples
Amazon
EC2
Amazon
RDS
If CPU utilization is > 60% for 5 minutes…
If number of simultaneous connections is > 10 for one minute…
If number of healthy hosts is < 5 for 10 minutes… Amazon
ELB
CloudWatch Alarms and Actions
CloudWatch alarms:
Measure a single
metric and perform
one or more actions
Stop, terminate, reboot, or
recover an Amazon EC2
instance
Scale an Auto Scaling
group in or out
Send message to Amazon
Simple Notification Service
(SNS)
Auto Scaling
• Scale your Amazon EC2 capacity
automatically
• Well-suited for applications that experience
variability in usage
• Available at no additional charge Auto
Scaling
Do Not Guess About Resource Needs
Build a flexible system that will react to changes in customer
demand and manage costs dynamically.
Availability Zone
Auto Scaling group
Availability Zone
Alarm
CloudWatch
Launch Configurations
• A launch configuration is a template that an Auto Scaling group uses to launch EC2 instances.
• When you create a launch configuration, you can specify:– AMI ID
– Instance type
– Key pair
– Security groups
– Block device mapping
– User data
Auto Scaling Groups
• Contain a collection of EC2 instances that share similar
characteristics.
• Instances in an Auto Scaling group are treated as a logical
grouping for the purpose of instance scaling and management.
Auto Scaling group
Minimum size
Desired capacity
Maximum size
Scale out as needed
Scaling Actions
• Dynamic Scaling– You can create a scaling policy that uses CloudWatch alarms to determine:
• When your Auto Scaling group should scale out.
• When your Auto Scaling group should scale in.
– You can use alarms to monitor:
• Any of the metrics that AWS services send to Amazon CloudWatch.
• Your own custom metrics.
• Manual Scaling– Scheduled Actions
– API Calls
Auto Scaling Basic Lifecycle
instances
Auto Scaling group
Scale Out
Amazon CloudWatchScheduled Event
Scale In
Amazon CloudWatchScheduled Event
Launch
Instance
Attach to Group
Detach from
GroupTerminate
Instance X
Triad of Services
Latency
Utilization
CloudWatchAuto Scaling
Elastic Load
Balancing
Auto Scaling group
Execute AS
Policy
AWS Trusted Advisor
• Best practice and recommendation engine.
• Provides AWS customers with performance and
security recommendations in four categories:
– Cost optimization
– Security
– Fault tolerance
– Performance improvement.
AWS Trusted
Advisor
Cost Optimization
• Amazon EC2 Reserved Instance Optimization
• Low-utilization Amazon EC2 Instances
• Idle load balancers
• Underutilized Amazon EBS volumes
• Unassociated Elastic IP addresses
• Amazon RDS idle DB instances
Security
• Security groups
• AWS IAM use
• Amazon S3 bucket permissions
• MFA on Root Account
• AWS IAM password policy
• Amazon RDS security group access risk
Fault Tolerance
• Amazon EBS Snapshots
• Load balancer optimization
• Auto Scaling Group Resources
• Amazon RDS Multi-AZ
• Amazon Route 53 name server delegations
• ELB connection draining
Performance Improvement
• High-utilization Amazon EC2 instances
• Service limits
• Large number of rules in EC2 security group
• Over-utilized Amazon EBS magnetic volumes
• Amazon EC2 to EBS throughput optimization
• Amazon CloudFront alternate domain names
The Challenges of Cloud Deployments
– Updating live servers.
– Rollouts across multiple
geographical locations.
– Ability to manage a rollback.
– Debugging deployments.
– Managing dependencies on
systems and subsystems.
Technologies for Automated, Repeatable Deployments
Custom Scripts and Applications
• Use AWS CLI or API to automate deployments in a variety of languages.
• Userdata
AWS CloudFormation
• Use a simple, declarative domain-specific language (DSL) to build a template file that creates and deletes a collection of resources together as a single unit (a stack).
AWS OpsWorks
• Use a simple, declarative domain-specific language (DSL) to create AWS resources.
What is AWS CloudFormation?
– Declarative programming language for deploying
AWS resources.
– Supports many AWS services.
– Create, update, and delete a set of resources as a
single unit (stack).
– Infrastructure as Code.
– Free of Charge.
What Does Infrastructure as Code Mean?
• Techniques, practices, and tools from software
development applied to creating reusable,
maintainable, extensible and testable infrastructure.
CloudFormation: Infrastructure as Code
Allows you to launch, configure, and connect AWS resources with JavaScript Object Notation (JSON) and YAML-formatted templates
Template StackAWS CloudFormation
Engine
• JSON-formatted file
describing the resources
to be created
• Treat it as source code:
put it in your repository
• YAML-formatted template
support
• AWS service component
• Interprets AWS
CloudFormation template
into stacks of AWS
resources
• A collection of resources
created by AWS
CloudFormation
• Tracked and reviewable in
the AWS Management
Console
• Cross stack references
Benefits of Treating Infrastructure as Code
template
Development
Repeatability
Production
Reusability
Auto Scaling
group
Load
balancer
Auto Scaling
group
Load
balancer
Auto Scaling
group
Load
balancer
Auto Scaling
group
Load
balancer
Benefits of Treating Infrastructure as Code
template
Development Production
Maintainability, Consistency, and Parallelization
Template
updated
Auto Scaling
group
Load
balancer
security group
Auto Scaling
group
Load
balancer
security group
Auto Scaling
group
Load
balancer
security group
Auto Scaling
group
Load
balancer
security group
Knowledge Check 1
• How does Auto Scaling scale instances?– Scale up and down, or
– Scale in and out?
Answer:
Scale in and out. In other words, change the quantity of
instances in the Auto Scaling group.
Knowledge Check 2
• True or False: “Memory Utilization” is a basic
monitoring metric of CloudWatch.
Answer:
• False. It is a custom metric and has to be
implemented by using CloudWatch Logs
Knowledge Check 3
• You have configured a CloudWatch alarm to trigger when
CPU rises above 60%. CPU is currently at 80%. What is
the status of the alarm?
– OK
– ALARM
– INSUFFICIENT DATA
Answer:
• Alarm, but only if the period condition has also
been met (i.e. above 60% for one minute).
Knowledge Check 4
• Can you deploy Configuration Files with
CloudFormation?
Answer:
Yes, by using CloudFormation:Init.
Further Information
• Official AWS Events– Transformation Day
– AWSomeDay
• Meetups
• Digicomp Trainings
• Amanox Events– Bootcamp
• DevOps
• Microservices and Docker