aws re:invent 2016: workshop: stretching scalability: doing more with amazon aurora (dat322)

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Amit Sharma, Partner Solutions ArchitectKD Singh, Partner Solutions Architect

Scott Ward, Partner Solutions Architect

November 30, 2016

Workshop: Stretching Scalability: Doing More with Amazon Aurora

DAT322

What to Expect from the Session

1. Amazon Aurora 1. Introduction2. Features overview 3. Use cases

2. Workshop scenario go-through

3. Team presentations

Relational databases

Fully managed and secure

Fast, predictable performance

Simple and fast to scale

Low cost, pay for what you useAmazon

RDS

Amazon Aurora

Why did AWS build Amazon Aurora?

Speed and availability of high-end commercial databases

Simplicity and cost-effectiveness of open source databases

Drop-in compatibility with MySQL

Simple pay-as-you-go pricing

Delivered as a managed service

Database architectures in the last 30 years

Even when you scale it out, you’re still replicating the same stack

SQL

Transactions

Caching

Logging

SQL

Transactions

Caching

Logging

ShardingCoupled at the application layer

Application

SQL

Transactions

Caching

Logging

SQL

Transactions

Caching

Logging

Shared nothingCoupled at the SQL layer

Application

SQL

Transactions

Caching

Logging

SQL

Transactions

Caching

Logging

Shared diskCoupled at the caching and

storage layer

Storage

Application

A service-oriented architecture applied to the database

Moved the logging and storage layer into a multi-tenant, scale-out, database-optimized storage service

Integrated with other AWS services such as Amazon EC2, Amazon VPC, Amazon DynamoDB, Amazon SWF, and Amazon Route 53 for control plane operations

Integrated with Amazon S3 for continuous backup with 99.999999999% durability

Control planeData plane

Amazon DynamoDB

Amazon SWF

Amazon Route 53

Logging + Storage

SQL

Transactions

Caching

Amazon S3

1

2

3

Rapid adoption of Amazon Aurora

Fastest growing service

in AWS history

Aurora customer adoption

Expedia: Online travel marketplace Real-time business intelligence and analytics on

a growing corpus of online travel marketplace data.

Current Microsoft SQL Server–based architecture is too expensive. Performance degrades as data volume grows.

Cassandra with Solr index requires large memory footprint and hundreds of nodes, adding cost.

Aurora benefits:  Aurora meets scale and performance

requirements with much lower cost. 25,000 inserts/sec with peak up to 70,000. 30 ms

average response time for write and 17 ms for read, with 1 month of data.

World’s leading online travel company, with a portfolio that includes 150+ travel sites in 70 countries.

Features Overview

Aurora features

- Migration- Performance- HA and failover- Backup- Security- Cost

Migration to Aurora is easy

Migration from MySQL

Source database on RDS Snapshot migration: One-click

migration from RDS MySQL 5.6 to Aurora

Source database external or on Amazon EC2 Use native MySQL migration tools

Backup to S3 using Percona XtraBackup, restore from S3

DBSnapshot

One-clickMigration

RDS MySQL Master/Slave New Aurora

Cluster

RDS Snapshot Migration

MySQL Backup to Aurora via S3

Source MySQL Database Target Aurora

Database

Amazon S3

Use multi-part upload or S3 Transfer Acceleration for faster uploads

AWS Import/Export Snowball

Migrate from Oracle and SQL ServerMove your tables, views, stored procedures, and data manipulation language (DML) to MySQL, MariaDB, and Amazon Aurora

Know exactly where manual edits are neededDownload at aws.amazon.com/dms

AWSSchema Conversion

Tool

Know exactly where manual edits are needed

Start your first migration in 10 minutes or lessKeep your apps running during the migrationReplicate within, to, or from Amazon EC2 or RDSMove data to the same or different database engine

AWSDatabase Migration

Service

Customerpremises

Application users

AWS

Internet

VPN

Start a replication instance

Connect to source and target databases

Select tables, schemas, or databases

Let AWS Database Migration Service create tables, load data, and keep them in sync

Switch applications over to the target at your convenience

Keep your apps running during the migration

AWSDatabase Migration

Service

“When we ran Alfresco’s workload on Aurora, we were blown away to find that Aurora was 10x faster than our MySQL environment,” said John Newton, founder and CTO of Alfresco. “Speed matters in our business, and Aurora has been faster, cheaper, and considerably easier to use than MySQL.”

Amazon Aurora is fast

Aurora requires fewer I/Os

Binlog Data Double-write bufferLog records FRM files, metadata

T Y P E O F W R I T E S

EBS mirrorEBS mirror

AZ 1 AZ 2

Amazon S3

MYSQL WITH STANDBY

SEQUENTIALWRITE

SEQUENTIALWRITE

EBSAmazon Elastic

Block Store (EBS)

PrimaryInstance

StandbyInstance

AZ 1 AZ 3

PrimaryInstance

Amazon S3

AZ 2

ReplicaInstance

AMAZON AURORA

ASYNC4/6 QUORUM

DISTRIBUTED WRITES

Amazon Aurora Performance Benefits

Concurrent Connections

Table Count

Size of Data

Read Replica Count

Amazon Aurora is highly available

Amazon Aurora is highly available

Highly available storage

• Six copies of data across three Availability Zones

• Latency-tolerant quorum system for read/write

• Up to 15 replicas with low replication lag

Survivable caches

• Cache remains warm in the event of a database restart

• Lets you resume fully loaded operations much faster

Instant crash recovery

• Underlying storage replays redo records on demand as part of a disk read

• Parallel, distributed, asynchronous

AZ 1 AZ 2 AZ 3

Amazon S3

SQL

Transactions

CachingT0

Backup of Amazon Aurora

Amazon S3

AZ 1 AZ 2 AZ 3

Aurora primary instance

Cluster volume spans 3 AZs

Aurora Replica Aurora Replica

Continuous

Automatic

Incremental

Manual Snapshots

Restoration of Data in Amazon Aurora

Amazon S3 Manual Snapshots

AmazonAurora

Point in timerestore

AmazonAurora

AmazonAurora

Restore Snapshot

Restore to a new cluster

Recover an existing cluster

High availability with Aurora Replicas

Amazon S3

AZ 1 AZ 2 AZ 3

Aurora primary instance

Cluster volume spans 3 AZs

Aurora Replica Aurora Replica

db.r3.8xlarge db.r3.2xlargePriority: tier-1

db.r3.8xlargePriority: tier-0

High availability with Aurora Replicas

Amazon S3

AZ 1 AZ 2 AZ 3

Aurora Primary instance

Cluster volume spans 3 AZs

Aurora Replica Aurora primaryinstance

db.r3.8xlarge db.r3.2xlargePriority: tier-1

db.r3.8xlarge

Balanced Read Replica Access

AZ 1 AZ 2 AZ 3

Aurora Primary instance

Aurora Replica Aurora Replica

Read Replica Endpoint

Faster, more predictable failover

ApprunningFailure detection DNS propagation

Recovery Recovery

DBfailure

MySQL

Apprunning

Failure detection DNS propagation

Recovery

DBfailure

AURORA WITH MARIADB DRIVER

1 5 – 2 0 s e c .

3 – 2 0 s e c .

ALTER SYSTEM CRASH [{INSTANCE | DISPATCHER | NODE}] 

ALTER SYSTEM SIMULATE percent_failure DISK failure_type IN

[DISK index | NODE index] FOR INTERVAL interval

ALTER SYSTEM SIMULATE percent_failure NETWORK failure_type

[TO {ALL | read_replica | availability_zone}] FOR INTERVAL interval

Simulate failures using SQL

To cause the failure of a component at the database node:

To simulate the failure of disks:

To simulate the failure of networking:

Choose cross-region read replicas for faster disaster recovery and enhanced data locality

Promote a read replica to a master for faster recovery in the event of disaster

Bring data close to your customer’s applications in different regions

Promote to a master for easy migration

Amazon Aurora is easy to use

“Amazon Aurora’s new user-friendly monitoring interface made it easy to diagnose and address issues. Its performance, reliability, and monitoring really shows Amazon Aurora is an enterprise-grade AWS database.” —Mohamad Reza, information systems officer at United Nations

Simplify storage management

• Automatic storage scaling up to 64 TB—no performance impact

• Continuous, incremental backups to Amazon S3

• Instantly create user snapshots—no performance impact

• Automatic restriping, mirror repair, hot spot management, encryption

Up to 64 TB of storage—auto-incremented in 10 GB units

up to 64 TB

Simplify monitoring with AWS Management ConsoleAmazon CloudWatch metrics for RDS

CPU utilization Storage Memory 50+ system/OS metrics 1–60 second granularity DB connections Selects per second Latency (read and write) Cache hit ratio Replica lag

CloudWatch alarms Similar to on-premises custom

monitoring tools

Lambda Integration

AmazonAurora

Asynchronous Lambda Function

Built-in trigger

Email Notification

Custom ActionFor ex. – Row inserted in a specific tableOrA delete operation was performed

Amazon S3Data Lake

Data Load from S3 into Aurora

Multiple Data Sources

AmazonAurora

Move data from Amazon Aurora using Data Pipeline

AmazonAurora

AWS Data Pipeline

AmazonS3 Amazon Machine

Learning

Amazon EMR

Amazon Redshift

AnalyticsETL….

Model Training

Data Warehousing

Amazon Aurora is secure

Simplify data security

Encryption to secure data at rest • AES-256; hardware accelerated• All blocks on disk and in Amazon S3 are encrypted• Key management by using AWS KMS

SSL to secure data in transit

Network isolation by using Amazon VPC by default

No direct access to nodes

Supports industry standard security and data protection certifications

Storage

SQL

Transactions

Caching

Amazon S3

Application

Amazon Aurora saves you money

Simple pricingNo licensesNo lock-inPay only for what you use

Discounts44% with a 1-year Reserved Instance63% with a 3-year Reserved Instance

vCPU Mem Hourly Price

db.r3.large 2 15.25 $0.29

db.r3.xlarge 4 30.5 $0.58

db.r3.2xlarge 8 61 $1.16

db.r3.4xlarge 16 122 $2.32

db.r3.8xlarge 32 244 $4.64

• Storage consumed, up to 64 TB, is $0.10/GB-month• IOs consumed are billed at $0.20 per million I/O• Prices are for US East (N. Virginia) region

Enterprise grade, open source pricing

Cost of ownership: Aurora vs. MySQLAurora configuration hourly cost

Instance cost: $13.92/hrStorage cost: $4.43/hr

Total cost: $18.35/hr

Primaryr3.8XL

Replicar3.8XL

ReplicaR3.8XL

Storage/6 TB

$4.64/hr $4.64/hr $4.64/hr

$4.43/hr

*At a macro level, Aurora saves over 50% in storage cost compared to RDS MySQL.

21.6%Savings

No idle standby instance

Single shared storage volume

No PIOPS—pay for use IO

Reduction in overall IOP

Cost of ownership: Aurora vs. MySQLFurther opportunity for saving

Instance cost: $6.96/hrStorage cost: $4.43/hr

Total cost: $11.39/hrStorage IOPS assumptions:1. Average IOPS is 50% of maximum IOPS2. 50% savings from shipping logs vs. full pages

51.3%Savings

Primaryr3.8XL

Replicar3.8XL

Replicar3.8XL

Storage/6 TB

$2.32/hr $2.32/hr $2.32 hr

$4.43/hr

r3.4XL r3.4XL r3.4XL

Use smaller instance size

Pay-as-you-go storage

Cost of Storage

Other databases

Amazon Aurora

$$$ = Provisioned Storage

$ = Consumed Storage

t2.medium Support

Dev & Test Pilot Simulations

Workshop

What will you learn from the workshop?

1. Different scenarios to apply Amazon Aurora knowledge 2. Understand the strengths of Amazon Aurora3. Patterns and anti-patterns of where to use Amazon

Aurora4. Key integrations of Amazon Aurora with other services

What is expected from you -

1. Create a team of six members each2. Come up with a solution for the given scenario 3. Present the solution – 10 minutes 4. Questions

Thank you!

Remember to complete your evaluations!

Related Sessions

DAT301 - Amazon Aurora Best Practices: Getting the Best Out of Your Databases

DAT323 - Workshop: Converting your Oracle or SQL Server database to an open source Aurora or Postgres database using the Schema Conversion Tool and Database Migration Service

DAT303 - Deep Dive on Amazon Aurora