© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Sean Shriver, AWS DynamoDB Solutions Architect

November 2017

Deep Dive: Amazon DynamoDB

Dating Website Serverless IoT

o DAX

o GSIs

o TTL

o Streams

o DAX

Getting Started

o Developer Resources

Amazon DynamoDB

o Foundations

o Tables

o Indexes

o Partitioning

New Features

o TTL

o Tagging

o VPC Endpoints

o Auto Scaling

o DAX

Plan

Dynamo whitepaper

NoSQL foundations

0000 {“Texas”}

0001 {“Illinois”}

0002 {“Oregon”}

T

XW

A

I

L

Key

Column

0000-0000-0000-0001

Game Heroes

Version 3.4

CRC ADE4

Key Value Graph Document Column-family

Dynamo:Amazon’s Highly Available

Key-value

Store

January 2012Fall 2007 June 2009

Meetup235 2nd St

San Francisco

Scaling relational vs. non-relational databases

Traditional SQL NoSQL

DB

DB

Scale up

DB

Host

1

DB

Host

n

DB

Host

2

DB

Host

3

Scale out to many shards

(DynamoDB: partitions)

Scaling NoSQL

- Good sharding (partitioning) scheme affords even

distribution of both data and workload, as they grow

- Key concept: partition key

- Ideal scaling conditions:

1. Partition key is from a high cardinality set (that grows)

2. Requests are evenly spread over the key space

3. Requests are evenly spread over time

What (some) customers store in NoSQL DBs

Market Orders Tokenization

(PHI, Credit Cards)

Chat MessagesUser Profiles

(Mobile)

IoT Sensor Data

(& device status!)

File MetadataSocial Media Feeds

Use case: DataXu’s attribution engine

Meta

Amazon

EMR

JobAmazon

Cloud

Watch

DynamoDBAWS Data

Pipeline

3rd

Party

S3

Buckets

1st

Party

AWS Direct

Connect

Amazon

VPC

Amazon

EC2

Amazon

RDS

Amazon SNS

AWS IAM

“Attribution" is the marketing term for the allocation of credit to individual

advertisements that eventually lead to a desired outcome (e.g., purchase).

Use case: DataXu’s attribution engine

Meta

Amazon

EMR

JobAmazon

Cloud

Watch

DynamoDBAWS Data

Pipeline

3rd

Party

S3

Buckets

1st

Party

AWS Direct

Connect

Amazon

VPC

Amazon

EC2

Amazon

RDS

Amazon SNS

AWS IAM

“Attribution" is the marketing term for the allocation of credit to individual

advertisements that eventually lead to a desired outcame (e.g. purchase).

Technical challenges

Amazon EC2

InstancesEBS Volumes

CloudWatch

MetricsNotifications

Scaling new AZs,

new Regions

Highly available Consistent, single digit

millisecond latency

at any scale

Fully managed

Secure

Integrates with AWS Lambda,

Amazon Redshift, and more.

Amazon DynamoDB

Elastic is the new normal

Write Capacity Units

Read Capacity Units

Consum

ed C

apacity U

nits

>200% increase from baseline>300% increase from baseline

Time

Scaling high-velocity use cases with DynamoDB

Ad Tech Gaming MobileIoT Web

Partition Key

Mandatory

Key-value access pattern

Determines data distribution

Optional

Model 1:N relationships

Enables rich query capabilities

DynamoDB Table

A1

(partition key)

A2

(sort key)

A3 A4 A7

A1

(partition key)

A2

(sort key)

A6 A4 A5

A1

(partition key)

A2

(sort key)

A1

(partition key)

A2

(sort key)

A3 A4 A5

SortKey

Table

Items

10 GB max per

partition key, i.e.

LSIs limit the # of

sort keys!

A1

(partition key)

A3

(sort key)

A2 A4 A5

A1

(partition key)

A4

(sort key)

A2 A3 A5

A1

(partition key)

A5

(sort key)

A2 A3 A4

• Alternate sort key

attribute

• Index is local to a

partition key

Local Secondary Indexes

RCUs/WCUs

provisioned

separately for GSIs

INCLUDE A2

A

L

L

KEYS_ONLY

A3

(partition key)

A1

(table key)

A2 A4 A7

A3

(partition key)

A1

(table key)

A3

(partition key)

A1

(table key)

A2

• Alternate partition

(+sort) key

• Index is across all

table partition keys

• Can be added or

removed anytime

A3

(partition key)

A1

(table key)

A2 A4 A7

A3

(partition key)

A1

(table key)

A2

A3

(partition key)

A1

(table key)

Global Secondary Indexes

Data types

Type DynamoDB Type

String String

Integer, Float Number

Timestamp Number or String

Blob Binary

Boolean Bool

Null Null

List List

SetSet of String, Number,

or Binary

Map Map

Table creation options

PartitionKey, Type:

SortKey, Type:

Provisioned Reads:

Provisioned Writes:

LSI Schema GSI Schema

AttributeName [S,N,B]

AttributeName [S,N,B]

1+

1+

Provisioned Reads: 1+

Provisioned Writes: 1+

TableName

Op

tion

al

Re

qu

ired

CreateTable

String,

Number,

Binary ONLY

Per Second

Unique to

Account and

Region

Optionally:

TTL and Auto Scaling

Provisioned capacity

Provisioned capacity

Read Capacity Unit (RCU)1 RCU returns 4KB of data for strongly

consistent reads, or double the data

at the same cost for eventually

consistent reads

Capacity is per second, rounded up to the next

whole number

Write Capacity Unit (WCU)1 WCU writes 1KB of data, and each

item consumes 1 WCU minimum

Horizontal Sharding

Host 1 Host 99 Host n

~Each new host brings compute, storage and network bandwidth~

CustomerOrdersTable

Cu

sto

merO

rders

Tab

le

OrderId: 1

CustomerId: 1

ASIN: [B00X4WHP5E]

Partitioning

00

55

AA

FF

Hash(1) = 7B

CustomerOrdersTable

OrderId: 2

CustomerId: 4

ASIN: [B00OQVZDJM]

OrderId: 3

CustomerId: 3

ASIN: [B00U3FPN4U]

Partition A

33.33 % Keyspace

33.33 % Provisioned Capacity

Partition B

33.33 % Keyspace

33.33 % Provisioned Capacity

Partition C

33.33 % Keyspace

33.33 % Provisioned Capacity

Hash.MIN = 0

Hash.MAX = FF

Ke

ysp

ace

Hash(2) = 48

Hash(3) = CD

Cu

sto

merO

rders

Tab

le

00

55

AA

FF

Partition A

33.33 % Keyspace

33.33 % Provisioned Capacity

Partition B

33.33 % Keyspace

33.33 % Provisioned Capacity

Partition C

33.33 % Keyspace

33.33 % Provisioned Capacity

Hash.MIN = 0

Hash.MAX = FF

Ke

ysp

ace

Time

Partition A

33.33 % Keyspace

33.33 % Provisioned Capacity

Partition B

33.33 % Keyspace

33.33 % Provisioned Capacity

Partition D

Partition E

16.66 %

16.66 %

16.66 %

16.66 %

Partition split due to partition size00

55

AA

FF

Partition A

33.33 % Keyspace

33.33 % Provisioned Capacity

Partition B

33.33 % Keyspace

33.33 % Provisioned Capacity

Partition C

33.33 % Keyspace

33.33 % Provisioned Capacity

Time

Partition A

Partition C

16.66 %

16.66 %

16.66 %

16.66 %

Partition splits due to capacity increase

16.66 %

16.66 %

16.66 %

16.66 %

16.66 %

16.66 %

16.66 %

16.66 %

Partition B

Partition D

Partition E

Partition F

The desired size of a partition

is 10GB* and when a

partition surpasses this it can

split*=subject to change

Split for partition size

The desired capacity of a

partition is expressed as:

3w + 1r < 3000 *

Where w = WCU & r = RCU*=subject to change

Split for provisioned capacity

Partitioning

Partition A

1000 RCUs

100 WCUs

Partition C

1000 RCUs

100 WCUs

Host A Host C

Availability Zone A

Partition A

1000 RCUs

100 WCUs

Partition C

1000 RCUs

100 WCUs

Host E Host G

Availability Zone B

Partition A

1000 RCUs

100 WCUs

Partition C

1000 RCUs

100 WCUs

Host H Host J

Availability Zone C

CustomerOrdersTable

54:∞00:0 54:∞00:0 54:∞00:0FF:∞AA:0 FF:∞AA:0 FF:∞AA:0

Data is replicated to

three Availability Zones

by design

3-way replication

OrderId: 1

CustomerId: 1

ASIN: [B00X4WHP5E]

Hash(1) = 7B

Partition B

1000 RCUs

100 WCUs

Host B Host F Host I

Partition B

1000 RCUs

100 WCUs

Partition B

1000 RCUs

100 WCUs

A9:∞55:0 A9:∞55:0 A9:∞55:0

Partitioning

DynamoDB Streams

Partition A

Partition B

Partition C

Ordered stream of item

changes

Exactly once, strictly

ordered by key

Highly durable, scalable

24 hour retention

Sub-second latency

Compatible with Kinesis

Client Library

DynamoDB Streams

1

Shards have a lineage and

automatically close after time or

when the associated DynamoDB

partition splits

2

3Updates

KCL

Worker

Amazon

Kinesis Client

Library

Application

KCL

Worker

KCL

Worker

GetRecords

Amazon DynamoDB

TableDynamoDB Streams

Stream

Shards

TTL

job

Time-To-Live (TTL)

Amazon DynamoDB

Table

CustomerActiveOrder

OrderId: 1

CustomerId: 1

MyTTL:

1492641900

DynamoDB Streams

Stream

Amazon Kinesis

Amazon Redshift

An epoch timestamp marking when an

item can be deleted by a background

process, without consuming any

provisioned capacity

Time-To-Live

Removes data that is no longer relevant

Time-To-Live (TTL)

TTL items

identifiable in

DynamoDB Streams

Configuration protected by AWS

Identity and Access Management

(IAM), auditable with AWS

CloudTrail

Eventual deletion,

free to use

Cost allocation tagging

• Track costs: AWS bills broken down by tags

in detailed monthly bills and Cost Explorer

• Flexible: Add customizable tags to tables,

indexes and DAX clusters

Features

Key Benefits

• Transparency: know exactly how much

your DynamoDB resources cost

• Consistent: report of spend across AWS

services

DynamoDB Auto Scaling

Specify: 1) Target capacity in percent 2) Upper and lower bound

DynamoDB Auto Scaling

Amazon

DynamoDB

msμs

DAXYour App

in VPC

Amazon DynamoDB Accelerator (DAX)

DynamoDB in the VPC

Availability Zone #1 Availability Zone #2

Private Subnet Private Subnet

VPC endpoint

web

app

server

security

groupsecurity

group

oMicroseconds latency in-memory cache

oMillions of requests per second

oFully managed, highly available

oRole based access control

oNo IGW or VPC endpoint required

DAX

oDynamoDB-in-the-VPC

o IAM resource policy

restricted

VPC Endpoints

AWS Lambda

security

groupsecurity

group

DAX

web

app

server

DAX

DynamoDB Accelerator (DAX)

Private IP, Client-side

Discovery

Supports AWS Java and NodeJS SDK,

with more AWS SDKs to come

Cluster based, Multi-AZ Separate Query and

Item cache

Elements of even access in NoSQL

1) Time

2) SPACE

DynamoDB key choice

To get the most out of DynamoDB throughput, create tables where the

partition key has a large number of distinct values, and values are requested

fairly uniformly, as randomly as possible.

Amazon DynamoDB Developer Guide

Burst capacity is built-in

0

400

800

1200

1600

Ca

pa

cit

y U

nit

s

Time

Provisioned Consumed

“Save up” unused capacity

Consume saved up capacity

Burst: 300 seconds

(1200 × 300 = 360k CU)

DynamoDB “saves” 300

seconds of unused capacity

per partition

Burst capacity may not be sufficient

0

400

800

1200

1600

Cap

ac

ity U

nit

s

Time

Provisioned Consumed Attempted

Throttled requests

Don’t completely depend on burst capacity… provision sufficient throughput

Burst: 300 seconds

(1200 × 300 = 360k CU)

Hot shards

Host 1 Host 2 Host 3

Hot key!

AZ A AZ B AZ C

Throttling

Occurs if sustained throughput goes beyond provisioned throughput per partition

• Possible causes

• Non-uniform workloads

• Hot keys/hot partitions

• Very large items

• Mixing hot data with cold data

• Remedy: Use TTL or a table per time period

- Disable retries, write your own retry code, and log all throttled or returned

keys

Design Patterns

Online dating website running on AWS

Users have people they like, and conversely

people who like them

Hourly batch job matches users

Data stored in Likes and Matches tables

Dating Website

DESIGN PATTERNS:

DynamoDB Accelerator and GSIs

Schema Design Part 1

GSI_Otheruser_id_other

(Partition key)

user_id_self

(sort key)

Requirements:

1. Get all people I like

2. Get all people that like me

3. Expire likes after 90 days

LIKES|

Likesuser_id_self

(Partition key)

user_id_other

(sort key)

MyTTL

(TTL attribute)

… Attribute N

Schema Design Part 2

Matchesevent_id

(Partition key)

timestamp

(sort key)

UserIdLeft

(GSI left)

UserIdRight

(GSI right)

Attribute N

GSI LeftUserIdLeft

(Partition key)

event_id

(Table key)

timestamp

(Table Key)

UserIdRight

GSI RightUserIdRight

(Partition key)

event_id

(Table key)

timestamp

(Table Key)

UserIdLeft

Requirements:

1.Get my matchesMATCHES|

Table Keys

Matchmaking

LIKES

Requirements:

1.Get all new likes every hour

2.For each like, get the other user’s likes

3.Store matches in matches table

Partition 1

Partition …

Partition NAvailability Zone

Public Subnet

match

making

server

security group

Auto Scaling group

Matchmaking

LIKES

Requirements:

1.Get all new likes every hour

2.For each like, get the other user’s likes

3.Store matches in matches table

Partition 1

Partition …

Partition NAvailability Zone

Public Subnet

match

making

server

security group

Auto Scaling group

THROTTLE!

Matchmaking Requirements:

1.Get all new likes every hour

2.For each like, get the other user’s likes

3.Store matches in matches table

1.Key choice: High key cardinality

2.Uniform access: access is evenly spread over the key-space

3.Time: requests arrive evenly spaced in time

Even Access:

Matchmaking

LIKES

Requirements:

1.Get all new likes every hour

2.For each like, get the other user’s likes

3.Store matches in matches table

Partition 1

Partition …

Partition NAvailability Zone

Public Subnet

match

making

server

security group

Auto Scaling group

0. Write like to like table, then query by user id to warm

cache, then queue for batch processing

security group

DAX

Takeaways:

Keep DAX warm by querying after writing

Use GSIs for many to many relationships

Dating Website

DESIGN PATTERNS:

DynamoDB Accelerator and GSIs

Amazon DynamoDB

DESIGN PATTERNS:

TTL, DynamoDB Streams, and DAX

Single DynamoDB table for storing sensor data

Tiered storage to remove archive old events to S3

Data stored in data table

Serverless IoT

Schema Design

DataDeviceId

(Partition key)

EventEpoch

(sort key)

MyTTL

(TTL attribute)

… Attribute N

Requirements:

1.Get all events for a device

2.Archive old events after 90 daysDATA|

UserDevicesUserId

(Partition key)

DeviceId

(sort key)

Attribute 1 … Attribute N

Requirements:

1.Get all devices for a userUSERDEVICES|

References

DATA

DeviceId: 1

EventEpoch: 1492641900

MyTTL: 1492736400 Expiry

AWS Lambda

Amazon S3

Bucket

Amazon DynamoDB Amazon DynamoDB Streams

Single DynamoDB table for storing sensor data

Tiered storage to remove archive old events to S3

Data stored in data table

USERDEVICES

Serverless

IoT

Serverless IoT

DATA

Partition A Partition B Partition DPartition C

Throttling

Noisy sensor produces data at

a rate several times greater

than others

Data

00

3F

BF

FF

Partition A25.0 % Keyspace

25.0 % Provisioned Capacity

Partition B25.0 % Keyspace25.0 % Provisioned Capacity

Partition D

25.0 % Keyspace25.0 % Provisioned Capacity

Hash.MIN = 0

Hash.MAX = FF

Ke

ysp

ace

Partition C25.0 % Keyspace25.0 % Provisioned Capacity

7F

Data

00

3F

BF

FF

Partition A25.0 % Keyspace

25.0 % Provisioned Capacity

Partition B25.0 % Keyspace25.0 % Provisioned Capacity

Partition D

25.0 % Keyspace25.0 % Provisioned Capacity

Hash.MIN = 0

Hash.MAX = FF

Ke

ysp

ace

Partition C25.0 % Keyspace25.0 % Provisioned Capacity

7F1.Key choice: High key cardinality2.Uniform access: access is evenly spread

over the key-space3.Time: requests arrive evenly spaced in

time

Even Access:

Serverless IoT

Requirements:

1. Single DynamoDB table for storing sensor data

2. Tiered storage to remove archive old events to

S3

3. Data stored in data table

0. Capable of dynamically sharding to overcome

throttling

Schema Design

ShardDeviceId

(Partition key)

ShardCount

Requirements:

1. Get shard count for given device

2. Always grow the count of shardsSHARD|

Requirements:

1. Get all events for a device

2. Archive old events after 90 daysData |

DataDeviceId

(Partition key)

EventEpoch

(sort key)

MyTTL

(TTL attribute)

… Attribute N

A sharding scheme where the number of

shards is not predefined, and will grow over

time but never contract. Contrast with a fixed

shard count

Naïve Sharding

Range: 0..1,000

DATA

DeviceId_ShardId: 1_3

EventEpoch: 1492641900

MyTTL: 1492736400

SHAR

DDeviceId: 1

ShardCount: 10

1.

2.

Serverless IoT: Naïve Sharding

Request path:

1.Read ShardCount from Shard table

2.Write to a random shard

3.If throttled, review shard count

Expiry

Serverless IoT

DATA

Partition A Partition B Partition DPartition C

Pick a random shard to write data to

DeviceId_ShardId: 1_Rand(0,10)

EventEpoch: 1492641900

MyTTL: 1492736400

2.

?

SHAR

DDeviceId: 1

ShardCount: 10

1.

DATA

DeviceId: 1

EventEpoch: 1492641900

MyTTL: 1492736400 Expiry

AWS Lambda

Amazon S3

Bucket

Amazon DynamoDB

Streams

Single DynamoDB table for storing sensor data

Tiered storage to remove archive old events to S3

Data stored in data table

Capable of dynamically sharding to overcome throttling

USERDEVICES

Serverless

IoT

SHAR

DDeviceId: 1

ShardCount: 10

DAX

+

Amazon Kinesis

Firehose

DESIGN PATTERNS:

TTL, DynamoDB Streams, and DAX

Takeaways:

Use naïve write sharding to dynamically expand shards

Use DAX for hot reads, especially from Lambda

Use TTL to create tiered storage

Serverless IoT

Getting started?

DynamoDB Local Document SDKsDynamoDB

Developer Resources

https://aws.amazon.com/dynamodb/developer-resources/

Thank you!