aws re:invent 2016: workshop: stretching scalability: doing more with amazon aurora (dat322)
TRANSCRIPT
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amit Sharma, Partner Solutions ArchitectKD Singh, Partner Solutions Architect
Scott Ward, Partner Solutions Architect
November 30, 2016
Workshop: Stretching Scalability: Doing More with Amazon Aurora
DAT322
What to Expect from the Session
1. Amazon Aurora 1. Introduction2. Features overview 3. Use cases
2. Workshop scenario go-through
3. Team presentations
Relational databases
Fully managed and secure
Fast, predictable performance
Simple and fast to scale
Low cost, pay for what you useAmazon
RDS
Amazon Aurora
Why did AWS build Amazon Aurora?
Speed and availability of high-end commercial databases
Simplicity and cost-effectiveness of open source databases
Drop-in compatibility with MySQL
Simple pay-as-you-go pricing
Delivered as a managed service
Database architectures in the last 30 years
Even when you scale it out, you’re still replicating the same stack
SQL
Transactions
Caching
Logging
SQL
Transactions
Caching
Logging
ShardingCoupled at the application layer
Application
SQL
Transactions
Caching
Logging
SQL
Transactions
Caching
Logging
Shared nothingCoupled at the SQL layer
Application
SQL
Transactions
Caching
Logging
SQL
Transactions
Caching
Logging
Shared diskCoupled at the caching and
storage layer
Storage
Application
A service-oriented architecture applied to the database
Moved the logging and storage layer into a multi-tenant, scale-out, database-optimized storage service
Integrated with other AWS services such as Amazon EC2, Amazon VPC, Amazon DynamoDB, Amazon SWF, and Amazon Route 53 for control plane operations
Integrated with Amazon S3 for continuous backup with 99.999999999% durability
Control planeData plane
Amazon DynamoDB
Amazon SWF
Amazon Route 53
Logging + Storage
SQL
Transactions
Caching
Amazon S3
1
2
3
Rapid adoption of Amazon Aurora
Fastest growing service
in AWS history
Aurora customer adoption
Expedia: Online travel marketplace Real-time business intelligence and analytics on
a growing corpus of online travel marketplace data.
Current Microsoft SQL Server–based architecture is too expensive. Performance degrades as data volume grows.
Cassandra with Solr index requires large memory footprint and hundreds of nodes, adding cost.
Aurora benefits: Aurora meets scale and performance
requirements with much lower cost. 25,000 inserts/sec with peak up to 70,000. 30 ms
average response time for write and 17 ms for read, with 1 month of data.
World’s leading online travel company, with a portfolio that includes 150+ travel sites in 70 countries.
Features Overview
Aurora features
- Migration- Performance- HA and failover- Backup- Security- Cost
Migration to Aurora is easy
Migration from MySQL
Source database on RDS Snapshot migration: One-click
migration from RDS MySQL 5.6 to Aurora
Source database external or on Amazon EC2 Use native MySQL migration tools
Backup to S3 using Percona XtraBackup, restore from S3
DBSnapshot
One-clickMigration
RDS MySQL Master/Slave New Aurora
Cluster
RDS Snapshot Migration
MySQL Backup to Aurora via S3
Source MySQL Database Target Aurora
Database
Amazon S3
Use multi-part upload or S3 Transfer Acceleration for faster uploads
AWS Import/Export Snowball
Migrate from Oracle and SQL ServerMove your tables, views, stored procedures, and data manipulation language (DML) to MySQL, MariaDB, and Amazon Aurora
Know exactly where manual edits are neededDownload at aws.amazon.com/dms
AWSSchema Conversion
Tool
Know exactly where manual edits are needed
Start your first migration in 10 minutes or lessKeep your apps running during the migrationReplicate within, to, or from Amazon EC2 or RDSMove data to the same or different database engine
AWSDatabase Migration
Service
Customerpremises
Application users
AWS
Internet
VPN
Start a replication instance
Connect to source and target databases
Select tables, schemas, or databases
Let AWS Database Migration Service create tables, load data, and keep them in sync
Switch applications over to the target at your convenience
Keep your apps running during the migration
AWSDatabase Migration
Service
“When we ran Alfresco’s workload on Aurora, we were blown away to find that Aurora was 10x faster than our MySQL environment,” said John Newton, founder and CTO of Alfresco. “Speed matters in our business, and Aurora has been faster, cheaper, and considerably easier to use than MySQL.”
Amazon Aurora is fast
Aurora requires fewer I/Os
Binlog Data Double-write bufferLog records FRM files, metadata
T Y P E O F W R I T E S
EBS mirrorEBS mirror
AZ 1 AZ 2
Amazon S3
MYSQL WITH STANDBY
SEQUENTIALWRITE
SEQUENTIALWRITE
EBSAmazon Elastic
Block Store (EBS)
PrimaryInstance
StandbyInstance
AZ 1 AZ 3
PrimaryInstance
Amazon S3
AZ 2
ReplicaInstance
AMAZON AURORA
ASYNC4/6 QUORUM
DISTRIBUTED WRITES
Amazon Aurora Performance Benefits
Concurrent Connections
Table Count
Size of Data
Read Replica Count
Amazon Aurora is highly available
Amazon Aurora is highly available
Highly available storage
• Six copies of data across three Availability Zones
• Latency-tolerant quorum system for read/write
• Up to 15 replicas with low replication lag
Survivable caches
• Cache remains warm in the event of a database restart
• Lets you resume fully loaded operations much faster
Instant crash recovery
• Underlying storage replays redo records on demand as part of a disk read
• Parallel, distributed, asynchronous
AZ 1 AZ 2 AZ 3
Amazon S3
SQL
Transactions
CachingT0
Backup of Amazon Aurora
Amazon S3
AZ 1 AZ 2 AZ 3
Aurora primary instance
Cluster volume spans 3 AZs
Aurora Replica Aurora Replica
Continuous
Automatic
Incremental
Manual Snapshots
Restoration of Data in Amazon Aurora
Amazon S3 Manual Snapshots
AmazonAurora
Point in timerestore
AmazonAurora
AmazonAurora
Restore Snapshot
Restore to a new cluster
Recover an existing cluster
High availability with Aurora Replicas
Amazon S3
AZ 1 AZ 2 AZ 3
Aurora primary instance
Cluster volume spans 3 AZs
Aurora Replica Aurora Replica
db.r3.8xlarge db.r3.2xlargePriority: tier-1
db.r3.8xlargePriority: tier-0
High availability with Aurora Replicas
Amazon S3
AZ 1 AZ 2 AZ 3
Aurora Primary instance
Cluster volume spans 3 AZs
Aurora Replica Aurora primaryinstance
db.r3.8xlarge db.r3.2xlargePriority: tier-1
db.r3.8xlarge
Balanced Read Replica Access
AZ 1 AZ 2 AZ 3
Aurora Primary instance
Aurora Replica Aurora Replica
Read Replica Endpoint
Faster, more predictable failover
ApprunningFailure detection DNS propagation
Recovery Recovery
DBfailure
MySQL
Apprunning
Failure detection DNS propagation
Recovery
DBfailure
AURORA WITH MARIADB DRIVER
1 5 – 2 0 s e c .
3 – 2 0 s e c .
ALTER SYSTEM CRASH [{INSTANCE | DISPATCHER | NODE}]
ALTER SYSTEM SIMULATE percent_failure DISK failure_type IN
[DISK index | NODE index] FOR INTERVAL interval
ALTER SYSTEM SIMULATE percent_failure NETWORK failure_type
[TO {ALL | read_replica | availability_zone}] FOR INTERVAL interval
Simulate failures using SQL
To cause the failure of a component at the database node:
To simulate the failure of disks:
To simulate the failure of networking:
Choose cross-region read replicas for faster disaster recovery and enhanced data locality
Promote a read replica to a master for faster recovery in the event of disaster
Bring data close to your customer’s applications in different regions
Promote to a master for easy migration
Amazon Aurora is easy to use
“Amazon Aurora’s new user-friendly monitoring interface made it easy to diagnose and address issues. Its performance, reliability, and monitoring really shows Amazon Aurora is an enterprise-grade AWS database.” —Mohamad Reza, information systems officer at United Nations
Simplify storage management
• Automatic storage scaling up to 64 TB—no performance impact
• Continuous, incremental backups to Amazon S3
• Instantly create user snapshots—no performance impact
• Automatic restriping, mirror repair, hot spot management, encryption
Up to 64 TB of storage—auto-incremented in 10 GB units
up to 64 TB
Simplify monitoring with AWS Management ConsoleAmazon CloudWatch metrics for RDS
CPU utilization Storage Memory 50+ system/OS metrics 1–60 second granularity DB connections Selects per second Latency (read and write) Cache hit ratio Replica lag
CloudWatch alarms Similar to on-premises custom
monitoring tools
Lambda Integration
AmazonAurora
Asynchronous Lambda Function
Built-in trigger
Email Notification
Custom ActionFor ex. – Row inserted in a specific tableOrA delete operation was performed
Amazon S3Data Lake
Data Load from S3 into Aurora
Multiple Data Sources
AmazonAurora
Move data from Amazon Aurora using Data Pipeline
AmazonAurora
AWS Data Pipeline
AmazonS3 Amazon Machine
Learning
Amazon EMR
Amazon Redshift
AnalyticsETL….
Model Training
Data Warehousing
Amazon Aurora is secure
Simplify data security
Encryption to secure data at rest • AES-256; hardware accelerated• All blocks on disk and in Amazon S3 are encrypted• Key management by using AWS KMS
SSL to secure data in transit
Network isolation by using Amazon VPC by default
No direct access to nodes
Supports industry standard security and data protection certifications
Storage
SQL
Transactions
Caching
Amazon S3
Application
Amazon Aurora saves you money
Simple pricingNo licensesNo lock-inPay only for what you use
Discounts44% with a 1-year Reserved Instance63% with a 3-year Reserved Instance
vCPU Mem Hourly Price
db.r3.large 2 15.25 $0.29
db.r3.xlarge 4 30.5 $0.58
db.r3.2xlarge 8 61 $1.16
db.r3.4xlarge 16 122 $2.32
db.r3.8xlarge 32 244 $4.64
• Storage consumed, up to 64 TB, is $0.10/GB-month• IOs consumed are billed at $0.20 per million I/O• Prices are for US East (N. Virginia) region
Enterprise grade, open source pricing
Cost of ownership: Aurora vs. MySQLAurora configuration hourly cost
Instance cost: $13.92/hrStorage cost: $4.43/hr
Total cost: $18.35/hr
Primaryr3.8XL
Replicar3.8XL
ReplicaR3.8XL
Storage/6 TB
$4.64/hr $4.64/hr $4.64/hr
$4.43/hr
*At a macro level, Aurora saves over 50% in storage cost compared to RDS MySQL.
21.6%Savings
No idle standby instance
Single shared storage volume
No PIOPS—pay for use IO
Reduction in overall IOP
Cost of ownership: Aurora vs. MySQLFurther opportunity for saving
Instance cost: $6.96/hrStorage cost: $4.43/hr
Total cost: $11.39/hrStorage IOPS assumptions:1. Average IOPS is 50% of maximum IOPS2. 50% savings from shipping logs vs. full pages
51.3%Savings
Primaryr3.8XL
Replicar3.8XL
Replicar3.8XL
Storage/6 TB
$2.32/hr $2.32/hr $2.32 hr
$4.43/hr
r3.4XL r3.4XL r3.4XL
Use smaller instance size
Pay-as-you-go storage
Cost of Storage
Other databases
Amazon Aurora
$$$ = Provisioned Storage
$ = Consumed Storage
t2.medium Support
Dev & Test Pilot Simulations
Workshop
What will you learn from the workshop?
1. Different scenarios to apply Amazon Aurora knowledge 2. Understand the strengths of Amazon Aurora3. Patterns and anti-patterns of where to use Amazon
Aurora4. Key integrations of Amazon Aurora with other services
What is expected from you -
1. Create a team of six members each2. Come up with a solution for the given scenario 3. Present the solution – 10 minutes 4. Questions
Thank you!
Remember to complete your evaluations!
Related Sessions
DAT301 - Amazon Aurora Best Practices: Getting the Best Out of Your Databases
DAT323 - Workshop: Converting your Oracle or SQL Server database to an open source Aurora or Postgres database using the Schema Conversion Tool and Database Migration Service
DAT303 - Deep Dive on Amazon Aurora