how to migrate your startup to aws
Post on 15-Jan-2017
669 Views
Preview:
TRANSCRIPT
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Andreas Chatzakis, AWS Solutions Architecture
Bob Gregory, Application Architect, Made.com
7th July 2016
Migrating your Startup to AWS
What to expect from this session
• Advice for startups migrating to AWS
• A practical checklist
• Relevant AWS services
• A real example from Made.com
Services you can use before migrating to AWS
Amazon Cognito
Amazon Mobile Analytics
AWS Device Farm
Amazon SNS
Mobile
Storage & CDN BI & Machine Learning
DNS
Amazon S3
Amazon CloudFront
Amazon QuickSight
Amazon Redshift
Amazon Machine Learning
Amazon Kinesis
Amazon EMR
Amazon Route 53
Amazon SES
Amazon WorkMail
Proof of Concept
• Do this early
• Will answer many questions
• Identify gaps and dependencies
• Estimate cost
Start with DEV&TEST!
Accelerate your migration with Training
Intro Videos and Labs
• Free introductory training
Online Labs
• Hands-on practice in a sandbox
environment.
Instructor Led Training
• Architecting, Developing, Operating, Big
Data, Security
https://aws.amazon.com/training/
Extend your team with AWS Support
• 3 plans to match your needs
• Developer Support
• Business Support
• Enterprise Support
• AWS Infrastructure Event Management
Migration approaches
• Big Bang
• Default choice for most startups
• Phased
• Driven by technical complexity
or expiration of old infrastructure investments
• Per application
• Per layer
Connectivity
• VPN
• AWS Direct Connect
• Existing provider on AWS (verify region)
3 stage journey
Lift & Shift• Move platform
• Security
• High Availability
Optimise• Automate
• Optimize cost
• Infinite scalability
Transform• Decouple
• Managed Services
• Serverless
Preparing for an application migration
• Establish objectives
• Enumerate application components
• Document dependencies
• Think about licensing
• Licence-included pricing (RDS,EC2)
• AWS Marketplace
• BYOL
• Map to AWS
On-premises infrastructure mapped to AWS
Technology On-premises AWS
Network VPN, MPLS Amazon VPC, AWS Direct Connect
Storage DAS, SAN, NAS, SSDAmazon EBS, Amazon S3, Amazon EC2 instance storage,
Amazon EFS
Compute Hardware, virtualization Amazon EC2, Amazon ECS, AWS Lambda
Content delivery Third-party CDN Amazon CloudFront
DatabasesMS SQL Server, MySQL, Oracle, DB2,
PostgreSQL, MongoDB,. …
Amazon RDS, Amazon DynamoDB, Amazon ElastiCache,
DB software on Amazon EC2
Load balancing Hardware and software load balancers Elastic Load Balancing, software load balancers
Scaling & cluster
management
Hardware and software clustering
toolsAuto Scaling, software clustering solutions
DNS BIND, Windows Server, third party Amazon Route 53, third-party DNS software on Amazon EC2
On-premises infrastructure mapped to AWS
Technology On-premises AWS
Analytics & data warehouseHadoop, Vertica, Cassandra, specialized
hardware and software Amazon EMR, Amazon Redshift, software on Amazon EC2
Messaging and workflow RabbitMQ, ActiveMQ, Kafka, …Amazon SQS, Amazon SNS, Amazon SWF,
software on Amazon EC2
Caching Redis, Memcached, … Amazon ElastiCache, Memcached, SAP Hana
Archiving Tape library, off-site data storage Amazon S3, Amazon Glacier
Email Email software Amazon SES
Identity, authoritzation, &
authenticationAD/ADFS, LDAP, SAML, third party…
AWS Identity and Access Management/AWS STS,
Amazon Cognito, AWS Directory Service, AD & LDAP on
Amazon EC2
Deployment & configuration
management
Chef, Puppet, Salt, Ansible, PowerShell
DSC
AWS CloudFormation, AWS OpsWorks, AWS Elastic Beanstalk,
AWS CodeDeploy, Amazon ECS
Management and
monitoringCA, BMC, Rightscale
Amazon CloudWatch, AWS Config, AWS CloudTrail,
AWS Trusted Advisor
Simplified cutoff process
Your Data Center
Web Layer
Database
Layer
Load Balancing
Firewalls etc
DNS
Simplified cutoff process
AWS region Your Data Center
Web Layer
Database
Layer
Load Balancing
Firewalls etc
DNS
Simplified cutoff process
AWS region
Private
Connection
Your Data Center
Web Layer
Database
Layer
Load Balancing
Firewalls etc
DNS
Simplified cutoff process
AWS region
Web
Layer
Private
Connection
Your Data Center
Web Layer
Database
Layer
Load Balancing
Firewalls etc
DNS
Elastic Load
Balancer
Simplified cutoff process
AWS region
Web
Layer
Private
Connection
Your Data Center
Web Layer
Database
Layer
Load Balancing
Firewalls etc
DNS
Elastic Load
Balancer
Amazon
RDS
Simplified cutoff process
AWS region
Web
Layer
Private
Connection
Your Data Center
Web Layer
Database
Layer
Load Balancing
Firewalls etc
DNS
Elastic Load
Balancer
Amazon
RDSDB replication
Simplified cutoff process
AWS region
Web
Layer
Private
Connection
Your Data Center
Web Layer
Load Balancing
Firewalls etc
DNS
Elastic Load
Balancer
Amazon
RDS
Database
LayerDB replication
Simplified cutoff process
AWS region
Web
Layer
Private
Connection
Your Data Center
Web Layer
Load Balancing
Firewalls etc
DNS
Elastic Load
Balancer
Amazon
RDS
Database
LayerDB replication
Simplified cutoff process
AWS region
Web
Layer
DNS
Elastic Load
Balancer
Amazon
RDS
Cutoff Readiness
• Review Security best practices
• Pen testing• https://aws.amazon.com/security/penetration-testing/
• Load testing
• Review AWS Service Limits
• AWS Business Support
• A tested Migration process
• A tested Roll-Back process
What could possibly go wrong…
• Hard coded IP addresses or host names
• Incorrectly sized AWS resources
• Auto Scaling ramp up period
• High DNS TTL slowing cut over and rollback
• Bandwidth from traditional infra to AWS
• IP address overlap with old network
• Third-party whitelisted IP addresses
• Email limits (request increase/use Amazon SES)
Migrating the application servers
• VM Import
• Rebuild
• Configuration management (Chef, Puppet…)
• Docker and the EC2 Container Service (ECS)
• AWS Elastic Beanstalk, AWS Opsworks
• Infrastructure as Code and AWS CloudFormation
Elastic Beanstalk
Alert
Log
Mon
Ap
p
AZ
EL
B
http://your-app.elasticbeanstalk.com
Database Migration options
Dump/Restore
• Small database
• Source DB not supporting replication
Native Replication
• Homogeneous migration
• No transformations
AWS Database Migration Service
• Heterogeneous migrations
• Transformations
• Easy to setup
Migrating data into AWS
• File transfer using S/FTP, SCP, 3rd party tools
• Point on-premises backup to S3
• AWS Storage Gateway for asynchronous backup
• AWS Import/Export: Disk or Snowball
• Database backup tools
• Database replication tools
• AWS Direct Connect 100 Mbps to 10 Gbps
DNS Migration
example.com
Third-party monitoring
System monitoring
Internal DNS
Public DNS
Route 53 public zones
Route 53 private zones
Route 53 health checks
example.com
Bulk transfer domains
1. Export DNS to Route 53
2. Delegate to Route 53
3. Transfer domains to Route 53
Order matters for availability!
Optimize
AWS Trusted Advisor
Free with Business or Enterprise Support
Align Resources with Demand
Use Reserved Instances
MADE.COM / AWS
How do I migrate my application to the cloud?
Bob Gregory
Introduction
Hi everyone, I’m Bob@made.com and I am an Application Architect.
I want to talk about things to do before you migrate, then some things to do
early in migration, and finish by talking about how we are managing cost.
MADE.COM : great design direct from the makers
Right now we’re moving from this:
Magento
OpenERP
XML-RPC
To this:
Magento
Availability /
Inventory
Batch
Allocation
Warehouse
Integration
PIM
Procurement
Refunds
Returns
Async
Events
Last year we moved our hosting away from:
● Traditional managed service
● 4 big hypervisors (20 core, 320GB RAM)
● 19TB SAN (SAS/SSD auto-tiered)
● All replicated in DR site
● VMWare running about 65 VMs (dev / test / prod)
● Running Magento, OpenERP and Unboxed
...to AWS, because:
● We like being connected to the Internet
● To automate infrastructure build (and get things done faster)
● Autoscaling (vs overprovisioning)
● It’s cool. And this is a better reason than it sounds.
Start with something simple
You are going to break things. A big-bang approach will hurt.
Start with something simple
If you have no users, you will have no complaints.
Consider starting with Greenfield systems.
Start with something simple
How will you integrate your cloud deployments with on-premise?
Loose coupling will save your bacon.
Consider starting with Public HTTP services.
Consider starting with Asynchronous messaging.
Start with something simple
It’s easier to handle an outage that only affects you personally.
Consider starting with Development Infrastructure.
Before you start
Treat manual configuration as a bug.
Automate all the things.
Before you start
Cloudformation is okay but can be unwieldy.
Consider Terraform if you like JSON.
Consider Ansible if you like extensibility.
Before you start
We had several iterations of our network layout.
It is hard to change your entire topology.
Keep it simple.
We use 9 subnets.
Address space is cheap.
2 Route tables.
Is this internal or external?
Managed NAT instances
Let AWS worry about it.
Before you start
The hardest things to run are stateful services.
Amazon do lots of the hard work for you.
Consider S3.
Consider RDS.
Consider Elasticache.
Consider Elastic File System.
Before you start
It is much simpler to manage identical machines.
Treat servers as cattle not pets.
Consider Docker.
Before you start
Docker has been a major factor in our success.
Developers can think at an application-stack level.
Teams deliver a working package configuration and a source-
controlled application configuration along with their code-base.
Before you start
Keep it simple.
You don’t need to deploy Kubernetes on top of Mesos
+ =
Before you start
There is no charge for running extra accounts.
Set up Consolidated Billing and run multiple AWS accounts.
We use
A hub-and-spoke model.
We run a vpn in Management.
VPC Peering
to access other environments
Separate Production.
This allows easier access
control
A separate account for
testing automation
We use
A Bastion account for IAM.
Like sudo for AWS.
Two-factor authentication
to access other environments.
Role-based access.
Developers are only allowed
to destroy test machines.
http://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html
AWSpocalypse
A 30 second Ansible tutorial:
Please can I have exactly one
instance with the Name
“my-instance”?
A 30 second Ansible tutorial:
Please can I have exactly one
instance ?
Don’t be this guy
It’s called best practice for a reason
So you’ve deployed your first system.
You’ve chosen your automation tooling
You’ve pushed data into stateful backing services
You’ve containerised your codebase and its configuration
So you’ve deployed your first system.
You have prevented AWSpocalypse.
Before you deploy your second service
Instances come and go. In order to manage your environment you
will need telemetry.
Monitor all the things.
Create Health Dashboards for each system you deploy.
Before you deploy your second service
Cloudwatch is okay.
It’s easy to set up and
free for basic
monitoring.
We useCollectd
To gather metrics
Riemann
To process and alert
InfluxDB
To store metrics
Grafana
To make pretty
pictures
Before you deploy your second service
Instances come and go. In order to manage your environment you
will need log shipping.
Log all the things.
Developers shouldn’t need to ssh to instances just to view logs.
We useRsyslog
To gather and process
logs
Logstash
To route logs into indexes
Elasticsearch
To store logs
Kibana
To view logs
This is expensive in engineering time
You must implement monitoring, but there are quicker ways to get
going.
Consider SAAS.
Time to move the big things
Database migration is the worst part (except for all the other worst
parts).
Amazon’s Database Migration Service is essentially magic.
Find it in the RDS tab of the AWS Console.
We used
HAProxy on the old and
new systems.
To switch over, we
configured HAProxy OLD to
route traffic to HAProxy
NEW.
To fail back, we could
configure HAProxy NEW to
route to HAProxy OLD.
So where are we now?
We prioritised agility over cash.
Each service has redundancy, and
shares no infrastructure with
other systems.
This makes it simple for developers
to deploy their stacks.
So where are we now?
We prioritised agility over cash.
Each service has redundancy, and
shares no infrastructure with
other systems.
This makes it really expensive.
How are we reducing costs?
Not everything needs to be up all the time
Consider Turning off test environments overnight.
Consider Scaling down database instances.
Consider Auto-scaling Groups.
How are we reducing costs?
On-demand instances are the expensive option.
Consider Reserved Instances for RDS, Elasticache, anything
where you have some well-known capacity.
How are we reducing costs?
On-demand instances are the expensive option.
Consider Spot Instances for automated tests and batch
processing.
How are we reducing costs?
CloudHealth has useful reports on instance over-sizing;
unattached volumes; and violations of best practice.
They charge a fixed percentage of your bill, so you are never too
small to consider using them.
How are we reducing costs?
The next step is Container Scheduling.
By abstracting away the ec2 instances, we can retain agility while
deploying multiple systems to each instance.
By using servers more efficiently we can cut our bill.
How are we reducing costs?
Container scheduling requires
Great monitoring of service performance and health.
Log shipping from your containers.
Service discovery (we’re using consul)
A fundamental rethink of the way you architect systems.
It has taken us approximately a year to reach this point.
Questions?
Please remember to rate this
session under My Agenda on
awssummit.london
top related