amazon dynamodb: a serverless nosql database for scale and

© 2020, Amazon Web Services, Inc. or its Affiliates.

Amazon DynamoDB: a

serverless NoSQL Database

for scale and performance


Venkat Govindarajan

Sr. Database Specialist SA

Robert McCauley

Sr. DynamoDB Specialist SA

© 2020, Amazon Web Services, Inc. or its Affiliates. © 2020, Amazon Web Services, Inc. or its Affiliates.

Agenda

• Relational Databases

• Why NoSQL and DynamoDB?

• Adaptive Capacity

• Managing Throughput

• New Responsibilities

• Microservice Architecture with DynamoDB

• Global Tables & DEMO

• Data Access Functions

• NoSQL Workbench DEMO

• Migrating to DynamoDB


The NoSQL Way


Database evolution – A paradigm shiftLE

VE

L O

F A

BS

TR

AC

TIO

N

FOCUS ON BUSINESS LOGIC

Managed Databases

Easier to setup and launch

Automated patching and backups

Scale with read replicas


Database evolution – A paradigm shiftLE

VE

L O

F A

BS

TR

AC

TIO

N

FOCUS ON BUSINESS LOGIC

Predictable fast performance

Fault tolerance built-in

Pay for value

Easy access via API

SERVERLESS NoSQL


Purpose-builtThe right tool for the right

job


Purpose-built databases


FINRA uses different AWS databases for different jobs

Market data search:

• Massive data volume

• Need quick lookups for searches

Amazon Aurora Amazon RDS

AmazonDynamoDB

Amazon DocumentDB

Amazon Neptune

Amazon ElastiCache

Session state:

• In-memory store for sub-millisecond site rendering

Data collection

Fraud detection

Relational data:

• Referential integrity

• Primary transactional database


Wait, what’s wrong with my existing relational DB?

• Monolith

• Hard to scale


Monolith Issue #1 – Shared Everything


Monolith Issue #2 – buffer cache

Buffer Cache – 64 GB

Data – 50 GBData – 500 GB

SELECT * FROM TableWHERE Column15 = "xyz"

Year 1: here you go!

Year 2: please wait..…


Monolith Issue #3 – nested queries

Data

SELECT * FROM (

SELECT * FROM (

SELECT * …

))

Temp Space


Monolith Issue #4 –stored procedures

Stored Procedures

Data

IF Column7 > :xTHEN

UPDATE..ELSE

INSERT INTO

Data Statistics

Compiled Procedures


Hard to Scale – JOINs are taxing

customers products reviewscarts

Cart.JSON


Hard to Scale – Sharding your data

Cust 234

Node1

Node2

Node3

??

Node4

• Different sizes

• Different

velocities

• Expanding and

migrating data

is difficult


Hard to Scale – Heavy proprietary connections• Connection count

maximum

• Install custom

drivers (~50mb)

• Custom ports

• Custom DB user

directory

• Best for fixed app

servers or desktops

Connection

Manager

C1

C2

C3

Function.

JS


monolithic application

+ DBA army

Lesson learned: decompose for agility

microservices

+ 2 pizza teams

Dev el o p m en t t r a n s fo r m a t i o n a t A m a z o n


Purpose built data stores, improved resiliency, API

communication


Why NoSQL?


Scaling Databases

Traditional SQL NoSQL

DB

Scale up

DB

P1

DB

Pn

DB

P2

DB

P3

Scale out to many shards

There is a way to design data that’s horizontally scalable.

DBDB


Wri

te C

ap

ac

ity

Un

its

/ se

c

Super Bowl Peak Consumed Capacity

Provisioned Capacity

Global-Scale Events: Elastic is the New Normal


Developer-friendly – Document PUT & GET to an API

Cart.JSON API


Why DynamoDB?


Amazon DynamoDB

Document or key-value Scales to any workloadFully managed NoSQL

Access control Event-driven programmingFast and consistent


Amazon DynamoDB - Features

NoSQL Workbench Point-in-time recovery

Time To Live (TTL)

DynamoDB local

CloudWatch

Contributor InsightsEncryption at rest DynamoDB Accelerator

(DAX)

On-demand backup


Amazon DynamoDB - Features

99.999% SLADynamoDB Streams Global Tables

Adaptive CapacityAuto Scaling

SLA

ACID

Transactions


Server 1

T1.p1

Table 1 Table 2 Table 3

Server N

T1.pn

You work with tables…

1K WCU/ 3K RCU

up to 10 GB

DynamoDB does the rest under the hood…


Storage – Serverless like S3

MyTableMyTable

Create a Table

Put "Items"

Pay for size stored

No storage provisioning needed


Horizontal scaling with DynamoDB

Workload:

data volume, reads, writes

DynamoDB resources:

storage, read, and write capacity


Adaptive Capacity


The uneven access problem common to NoSQL

Across data and time


Scaling NoSQL databases

Most NoSQL databases DynamoDB

DB

Servers and clusters DynamoDB: partitions

DBDB

DBDB

DBDB

DBDB

DB

DB

P1

DB

P1

DB

P1

DB

P1

DB

P1

DB

P1DB

P1

DB

P1

DB

P1

DB

P1

DB

P1

DB

P1DB

P1

DB

P1

DB

P1

DB

P1

DB

P1

DB

P1DB

P1

DB

P1

DB

P1

DB

P1

DB

P1

DB

P1DBP25

DBP26

DBP27

DBP28

DBP29

DBP30

Basic premise: There is a way to shard data that’s horizontally scalable.


Adaptive Capacity - Core Functions

• Dynamic partitioning

• High-traffic item isolation

• Throughput boosting


T1

Server 1

p.A

Dynamic Partitioning – Storage Growth


Server 1

T1

Server 2

A B



Server 1

T1

Server 2 Server 3

A B C



Server 1

T1

Server 4 Server 2 Server 3

A D B C



High-traffic item isolation


Server 1

T1

Server 4 Server 2 Server 3

A D B C

Adaptive capacity throughput boosting

AA


Read/Write Capacity Modes


Provisioned and On-

demand Capacity Modes


Read Capacity Unit

Item size Strongly consistent reads

4 KB 1 1 RCU

Item size Eventually consistent reads

Item size Transactional reads

4 KB 2 1 RCU

4 KB 1 2 RCU


Read Capacity Unit - Examples

Item size Strongly consistent reads

8 KB * (1 RCU / 4 KB) * (1) = 2 RCU

Conversion ratio

Item size Eventually consistent reads

8 KB * (1 RCU / 4 KB) * (1 / 2) = 1 RCU

Conversion ratio

Item size Transactional reads

8 KB * (1 RCU / 4 KB) * (2) = 4 RCU

Conversion ratio


Write Capacity Unit

Item size Standard writes

1 KB 1 1 WCU

Item size Transactional writes

1 KB 1 2 WCU


Write Capacity Unit - Examples

Item size Standard writes

3 KB * (1 WCU / 1 KB) * (1) = 3 WCU

Conversion ratio

Item size Transactional writes

3 KB * (1 WCU / 1 KB) * (2) = 6 WCU

Conversion ratio


• Add any number of items to a table

• Item max size = 400 KB

• Scaling is achieved through partitioning

• Partition max capacity = (3000 RCU) + (1000 WCU)

• Global Secondary Index (GSI)

• GSI has its own capacity settings beyond the base table

Throughput Considerations


Provisioned Mode -

Auto Scaling


Auto Scaling


DynamoDB on-demand capacity mode

Start at zero

No limit

Features

• No capacity planning, provisioning, or

reservations—simply make API calls

• Pay only for the reads and writes you

perform

Key benefits

• Eliminates tradeoffs of

overprovisioning or underprovisioning

• Instantly accommodates your workload

as traffic ramps up or down


On-demand scaling properties

Base throughput

• Up to 4,000 write request units: 4,000 writes per second

• Up to 12,000 read request units: 24,000 eventually consistent reads

per second

• Any linear combination of the two

Maximum throughput

Unlimited!

Pay per request: Use nothing, pay nothing


Maximum Scaling Capabilities:Up to twice your previous peak

DynamoDB request rate

Maximum throughput


On-demand tables do not “scale down”Maximum throughput

Maximum throughput


Use provisioned mode

• Steady workloads

• Gradual ramps

• Events with known traffic

Use on-demand mode

• Unpredictable workloads

• Frequently idle workloads

• Events with unknown traffic

• “Set it and forget it”

Consider your tolerance for operational overheadand overprovisioning


Post-launch Switchover: Conversion after load stabilization


Change

Capacit

y Mode


Amazon CloudWatch Contributor Insights for DynamoDB


New Roles and

Responsibilities


New Roles & Responsibilities

DBA → Cloud DBA

Developer += Data Architect


New Roles – Cloud DBA

Monitor CloudWatch:

• Read Throttles

• Write Throtttles

• Returned Bytes

• Request Latency (AVG)

• User Errors

• System Errors



Move to or from On Demand

mode

Perform any Backups &

Restores

Enable Point In Time Recovery

Monitor Global Table

Replication



Manage user access via IAM

or Cognito

Define IAM Policies for

types of users

Define Encryption Key via

KMS service (optional)


New Roles – Data Architect

Model:

Table Keys

NoSQL Workbench for

DynamoDB

CustomerA Order1#Line1 Yoga Mat Qty: 3

CustomerA Order1#Line2 Kettlebell Qty: 2

CustomerA Order2#Line1 Jump Rope Qty: 4



Part.

Key

Sort

Key



Model :

Global Secondary Indexes

Re-groups Items on an

alternate key

Kettlebell Order1#Line2 CustomerA Qty: 2

Kettlebell Order2#Line3 CustomerB Qty: 7

Yoga Mat Order1#Line1 CustomerC Qty: 3

Yoga Mat Order2#Line2 CustomerA Qty: 2

Jump Rope Order2#Line1 CustomerA Qty: 4

GSI

SKey

GSI

PKey



DENORMALIZE!

Duplicate data to avoid

looups and JOINs

Peach 2020-06-10 Qty: 2

Customer:

{name: “Patty”,

street: “123 Main”,

City:”Portland”}

Peach 2020-06-12 Qty: 5

Customer:

{name: “Timmy”,

street: “456 Nook”,

City:”San Diego”}

Peach 2020-07-25 Qty: 4

Customer:

{name: “Sam”,

street: “789 Elm”,

City:”Providence”}



Define Conventions:

Overload different Item

types to store complete

Item Collections

Turnip 2020-06-10 Qty: 2

Customer:

{name: “Patty”,

street: “123 Main”,

City:”Portland”}

Turnip 2020-06-12 Qty: 5

Customer:

{name: “Timmy”,

street: “456 Nook”,

City:”San Diego”}

Turnip 2020-07-25 Qty: 4

Customer:

{name: “Sam”,

street: “789 Elm”,

City:”Providence”}

Turnip DESCRIPTION

Turnip Description: This turnip is a

fast growing, crisp, fine-grained

turnip that keeps its mild sweet

flavor yet grows larger than…



Define Data Types and Item Lifecycle

Format DynamoDB Type Bytes Stored

Unix Epoch Date

32 bit Int

NUMBER:

15899886456

STRING:

158998864510

ISO 8601 StringSTRING:

2020-05-20T14:26:59Z20

TTL


Implementing Microservice

Architectures

with Amazon DynamoDB and AWS

Lambda


Example

AWS

Lambda

Amazon

DynamoDBAmazon API

Gateway

Amazon

DynamoDB

Streams

Lambda

Amazon

S3

Amazon

ES

Amazon

Athena

DynamoDB

Query sideOnline Transaction

Processing (OLTP) side

A serverless app


Building Blocks

AWS

Lambda

Amazon

DynamoDB

Amazon API

Gateway

1. API Gateway + Lambda + DynamoDB: core building block

2. DynamoDB Streams + AWS Lambda: building block for reliable event delivery

Amazon

DynamoDB

Streams

LambdaDynamoDB


DynamoDB Streams and Lambda

Partition A

Partition B

Partition C

Updates GetRecords

DynamoDB

table

DynamoDB Streams

Lambda

function

DynamoDB Streams + AWS Lambda = reliable “exactly once” event delivery


Example: Transactions and Querying

• An ecommerce app

• Relational vs. nonrelational data models

• Transactions in a distributed microservices architecture

• Enabling querying in a distributed microservices architecture


Example: Ecommerce App Using Microservices

Shopping cart

service

Product

catalog

service

Account

service

Lambda

API

Gateway

DynamoDBDB1 DB2 DB3


Shopping Cart Data Model

CART

CartId: 1,Version: 3

CART_ITEM

ItemId:2, Qty: 1,

CartId: 1

Relational database

CART

CartID:2,

Version: 3,

Items: [

{ID: 2, Qty: 1},

{ID: 5, Qty: 2}]

Nonrelational

database

Normalized schema:

multiple tables

Denormalized schema:

aggregates in a single table


Data Consistency on Cart Updates

Example: add/remove shopping cart items

1.get cart => vread = Version2.update cart:

IF Version = vreadadd/remove cart items++Version

ELSE go back to Step 1.

Use ConditionExpression

in DynamoDB

Optimistic concurrency control (OCC)


Updating Cart: Data Consistency Using OCC

CART

CartID:2,

Version: 3,

CartItems: [

{ID: 2, Qty: 1},{ID: 5, Qty: 2}]

✓ Use conditions to implement OCC, ensuring data

consistency

✓ GetItem call can eventually be consistent

1. Get the cart: GetItem

{ "TableName": “Cart”,

"Key": {“CartId”: {“N”: “2”}}

}

2. Update the cart: conditional PutItem (or UpdateItem)

{ "TableName": “Cart”,

"Item": {

“CartID”: {“N”: “2”},

“Version”: {“N”:”4”},

“CartItems”: {…}

},

"ConditionExpression": “Version = :v1”,

"ExpressionAttributeValues": {":v1": {“N”: “3”}}

}


Global Tables


Enhancements to Global Tables

Features

• Add regions on the fly

• Convert single-region tables to global

• Reduced write capacity consumption

Key benefits

• Flexible regional footprint

• Save on replicated write costs

Replica (N. America)

Replica (Europe)

Replica (Asia)

Global table


High Level Architecture

Region

Region 1

Region

Region 2

StreamReader


Multi-Primary Replication

Region

Region 1

Region

Region 2

StreamReader

StreamReader


Multi-Region Multi-Primary Replication

Region

Region 1

StreamReader

Region

Region 3

StreamReader

Region

Region 2

StreamReader


Amazon DynamoDBI M P O R T A N T 2 0 2 0 L A U N C H E S A T A G L A N C E

Export to Amazon S3

A native feature of DynamoDB

that allows you to export

DynamoDB table data to Amazon

Simple Storage Service

(Amazon S3) – no code required.

Build materialized views that

combine and replicate data across

multiple data stores without you

having to write custom code.

AWS Glue Elastic Views

PartiQL

Stream your DynamoDB data to

Kinesis Data Streams to take

advantage of longer retention

time and other Kinesis services.

Change data capture for Amazon Kinesis Data Streams

PartiQL


Backup Export to Amazon S3

Amazon

S3

Your

Table

Point

In

Time

Recovery

Compressed objects

• JSON format

• ION format


Analyze Exported Data with Athena, QuickSight, Sagemaker,

etc.

Amazon

S3

Amazon

Athena

SQL query

Your

Table


How Disney+ scales globally on DynamoDB

Disney+ on DynamoDB

Bookmark

global table

Amazon Kinesis

Data Streams

Bookmark

writer

Content API

Telemetry

service


Data Access Functions


Data Access Functions – Show cost per read

Request Parameter "Return Consumed Capacity"

Evaluate efficiency of Filters


Querying Data – next & previous

Peach 2020-06-20 Qty: 3

Peach 2020-06-21 Qty: 1

Cherry 2020-06-10 Qty: 2

Cherry 2020-06-12 Qty: 5

Cherry 2020-07-25 Qty: 4

Coconut 2020-05-28 Qty: 2



Pear 2020-06-13 Qty: 1

Pear 2020-06-15 Qty: 4

GET-ITEM to return a single item

QUERY can return a single next item

Key Condition

Expression

PKey = “Cherry”

SKey > “2020-06-12”

Limit 1

Scan Forward TRUE


Data Access Functions – Writes

Carefully structuring Items unlocks powerful Query patterns


CustomerA Order1#Line1#RETURN RETURN Yoga Mat Qty: 3

CustomerA Order2#Line1 Jump Rope Qty: 4



CustomerA STATUS#HOLD HOLD


StatusDate

DONE_2014-10-02

IN_PROGRESS_2014-10-08

IN_PROGRESS_2014-10-03

PENDING_2014-09-30

PENDING_2014-10-03

Status

DONE

IN_PROGRESS

IN_PROGRESS

PENDING

PENDING

Date

2018-10-02

2018-10-08

2018-10-03

2018-10-03

2018-09-30

+ =

Data Access Functions – Writes

Combine values together for composite sort key

Query with Sort Key Expression: BEGINS_WITH('PENDING')


A client-side application that helps you build scalable, high-performance data models

Simplifies query development and testing

A rich GUI-based tool that helps you visualize data models and perform DynamoDB operations

Available for Windows, macOS, and Linux

NoSQL WorkbenchEnterprise ready

Data modeler

Visualizer

Operation builder

102


Demo: Generating Data Access Functions with

the NoSQL Workbench


Migrating to DynamoDB


Existing Tables to Single Table Design

customers

products reviews

carts DynamoDB

Single Table

Put-Item

BatchWriteItem

CREATE VIEW

SQL UNION ALL


Single Table Design- Generic Keys, Overload Items, GSIs

Cart1 Peach Peach CustomerA … …

Cart1 Coconut Coconut CustomerA … …

CustomerA CustomerA CustomerA … …

CustomerB CustomerB CustomerB … …

Peach Order2#Line3 Peach … …

Review3 Review3 Peach CustomerC … …

PKey SKey


Existing Table Data prepared by View SQL

customers

products reviews

carts Single Table

VIEW


Dual Writes to Both Systems

customers

products reviews

cartsSingle Table

Application

verification


One table or many?

Denormalize and overload the table to

enable item collections and GSI queries

Avoid third normal form, foreign key

constraints, and JOINS

Consider one table per micro service

Use multiple tables as needed!


DynamoDBFast and flexible NoSQL database service for any scale

Performance at scale

• Handles millions of requests per second

• Delivers single-digit-millisecond latency

• Automated global replication

• New advanced streaming with Amazon Kinesis Data Streams for DynamoDB

Enterprise ready

• ACID transactions

• Encryption at rest

• Continuous backups (PITR), and on-demand backup and restore

• NoSQL Workbench

• Export table data to S3

• PartiQL (a SQL-compatible query language) support

No servers to manage

• Maintenance free

• Auto scaling

• On-demand capacity mode

• Change data capture for integration with AWS Lambda, Amazon Redshift, Amazon Elasticsearch Service

110


Amazon DynamoDB: a

serverless NoSQL Database

for scale and performance


@dynamodb

bit.ly/ddbefficiency

amazon-dynamodb-labs.com

Thank you for attending!

amazon dynamodb: a serverless nosql database for scale and

Documents