(sdd407) amazon dynamodb: data modeling and scaling best practices | aws re:invent 2014

143
November 13, 2014 | Las Vegas, NV David Yanacek, Amazon DynamoDB

Upload: amazon-web-services

Post on 30-Jun-2015

1.105 views

Category:

Technology


0 download

DESCRIPTION

Amazon DynamoDB is a fully managed, highly scalable distributed database service. In this technical talk, we show you how to use DynamoDB to build high-scale applications like social gaming, chat, and voting. We show you how to use building blocks such as secondary indexes, conditional writes, consistent reads, and batch operations to build the higher-level functionality such as multi-item atomic writes and join queries. We also discuss best practices such as index projections, item sharding, and parallel scan for maximum scalability.

TRANSCRIPT

Page 1: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

November 13, 2014 | Las Vegas, NV

David Yanacek, Amazon DynamoDB

Page 2: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 3: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 4: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 5: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Cache read-heavy items

Page 6: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 7: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Product A Product B

Shopper

ProductCatalog Table

Page 8: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Product A Product B

Shopper

SELECT Id, Description, ...FROM ProductCatalogWHERE Id=“Product A”

ProductCatalog Table

Page 9: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Need to scale

for the sale

Administrator (You)

ProductCatalog Table

Page 10: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Administrator (You)

Provision 5000 read capacity units

ProductCatalog Table

Page 11: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Administrator (You)

2500 read capacity units (each)

Provision 5000 read capacity units

Partition 1 Partition 2

ProductCatalog Table

Page 12: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Administrator (You)

(no sharing)

Partition 1 Partition 2

ProductCatalog Table

Provision 5000 read capacity units

Page 13: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Partition 1 Partition 2

ProductCatalog Table

Page 14: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Administrator (You)

ProductCatalog Table

Provision 100,000 read capacity units

Partition 1

2000 RCU

Partition K

2000 RCU

Partition M

2000 RCU

Partition 50

2000 RCU

Page 15: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Product A Product B

Shoppers

Partition 1

2000 RCU

Partition K

2000 RCU

Partition M

2000 RCU

Partition 50

2000 RCU

ProductCatalog Table

Page 16: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Partition 1

2000 RCU

Partition K

2000 RCU

Partition M

2000 RCU

Partition 50

2000 RCU

Product A Product B

Shoppers

ProductCatalog Table

Page 17: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Partition 1

2000 RCU

Partition K

2000 RCU

Partition M

2000 RCU

Partition 50

2000 RCU

Product A Product B

Shoppers

ProductCatalog Table

Page 18: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Partition 1

2000 RCU

Partition K

2000 RCU

Partition M

2000 RCU

Partition 50

2000 RCU

Product A Product B

Shoppers

ProductCatalog Table

Page 19: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 20: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Size Throughput

Page 21: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

# 𝑜𝑓 𝑃𝑎𝑟𝑡𝑖𝑡𝑖𝑜𝑛𝑠 =𝑇𝑎𝑏𝑙𝑒 𝑆𝑖𝑧𝑒 𝑖𝑛 𝑏𝑦𝑡𝑒𝑠

10 𝐺𝐵(𝑓𝑜𝑟 𝑠𝑖𝑧𝑒)

Page 22: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

# 𝑜𝑓 𝑃𝑎𝑟𝑡𝑖𝑡𝑖𝑜𝑛𝑠(𝑓𝑜𝑟 𝑡ℎ𝑟𝑜𝑢𝑔ℎ𝑝𝑢𝑡)

=𝑅𝐶𝑈𝑓𝑜𝑟 𝑟𝑒𝑎𝑑𝑠

3000 𝑅𝐶𝑈+

𝑊𝐶𝑈𝑓𝑜𝑟 𝑤𝑟𝑖𝑡𝑒𝑠

1000𝑊𝐶𝑈

Page 23: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

(𝑓𝑜𝑟 𝑡ℎ𝑟𝑜𝑢𝑔ℎ𝑝𝑢𝑡)(𝑓𝑜𝑟 𝑠𝑖𝑧𝑒)(𝑡𝑜𝑡𝑎𝑙)

Page 24: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

100,000𝑅𝐶𝑈50𝑃𝑎𝑟𝑡𝑖𝑡𝑖𝑜𝑛𝑠

≈ 𝟐𝟎𝟎𝟎 𝑟𝑒𝑎𝑑 𝑐𝑎𝑝𝑎𝑐𝑖𝑡𝑦 𝑢𝑛𝑖𝑡𝑠 𝑝𝑒𝑟 𝑝𝑎𝑟𝑡𝑖𝑡𝑖𝑜𝑛

Partition 1

2000 RCU

Partition K

2000 RCU

Partition M

2000 RCU

Partition 50

2000 RCU

ProductCatalog Table

Page 25: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Space Time

Page 26: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Part

itio

n

Time

Heat

Page 27: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 28: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

0

400

800

1200

1600

Cap

ac

ity U

nit

s

Time

Provisioned Consumed

“save up” unused capacity

consume saved up capacity

Page 29: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

0

400

800

1200

1600

Cap

ac

ity U

nit

s

Time

Provisioned Consumed Attempted

Burst capacity: 300 seconds

(1200 × 300 = 3600 CU)

Throttled requests

Page 30: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 31: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Partition 1

2000 RCU

Partition K

2000 RCU

Partition M

2000 RCU

Partition 50

2000 RCU

Product A Product B

Shoppers

ProductCatalog Table

Page 32: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 33: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Req

uests

Per

Seco

nd

Item Primary Key

Request Distribution Per Hash Key

DynamoDB Requests

Page 34: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Req

uests

Per

Seco

nd

Item Primary Key

Request Distribution Per Hash Key

DynamoDB Requests Cache Hits

Page 35: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Partition 1 Partition 2

ProductCatalog Table

User

DynamoDB

User

SELECT Id, Description, ...

FROM ProductCatalog

WHERE Id="POPULAR_PRODUCT"

Page 36: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 37: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Substitute indexes for filters

Page 38: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 39: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

GameId Date Host Opponent Status

d9bl3 2014-10-02 David Alice DONE

72f49 2014-09-30 Alice Bob PENDING

o2pnb 2014-10-08 Bob Carol IN_PROGRESS

b932s 2014-10-03 Carol Bob PENDING

ef9ca 2014-10-03 David Bob IN_PROGRESS

Games Table

Page 40: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

GameId Date Host Opponent Status

d9bl3 2014-10-02 David Alice DONE

72f49 2014-09-30 Alice Bob PENDING

o2pnb 2014-10-08 Bob Carol IN_PROGRESS

b932s 2014-10-03 Carol Bob PENDING

ef9ca 2014-10-03 David Bob IN_PROGRESS

Games Table

Page 41: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

IN_PROGRESS DONEPENDING

Page 42: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 43: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Secondary Index

Bob

Opponent Status GameId Date Host

Alice DONE d9bl3 2014-10-02 David

Carol IN_PROGRESS o2pnb 2014-10-08 Bob

Bob IN_PROGRESS ef9ca 2014-10-03 David

Bob PENDING 72f49 2014-10-03 Alice

Bob PENDING b932s 2014-09-30 Carol

Page 44: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Secondary Index

Opponent Status GameId Date Host

Alice DONE d9bl3 2014-10-02 David

Carol IN_PROGRESS o2pnb 2014-10-08 Bob

Bob IN_PROGRESS ef9ca 2014-10-03 David

Bob PENDING 72f49 2014-10-03 Alice

Bob PENDING b932s 2014-09-30 Carol

Bob

Page 45: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Secondary Index

Opponent Status GameId Date Host

Alice DONE d9bl3 2014-10-02 David

Carol IN_PROGRESS o2pnb 2014-10-08 Bob

Bob IN_PROGRESS ef9ca 2014-10-03 David

Bob PENDING 72f49 2014-10-03 Alice

Bob PENDING b932s 2014-09-30 Carol

Bob

SELECT * FROM Game

WHERE Opponent='Bob'

AND Status='PENDING'

Page 46: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Bob

SELECT * FROM Game

WHERE Opponent='Bob'

AND Status='PENDING'

ORDER BY Date?

Opponent Status GameId Date Host

Alice DONE d9bl3 2014-10-02 David

Carol IN_PROGRESS o2pnb 2014-10-08 Bob

Bob IN_PROGRESS ef9ca 2014-10-03 David

Bob PENDING 72f49 2014-10-03 Alice

Bob PENDING b932s 2014-09-30 Carol

Secondary Index

Page 47: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

SELECT * FROM Game

WHERE Opponent='Bob'

AND Status='IN_PROGRESS'

ORDER BY Date?

(hash)

(range)

(?)

Page 48: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Secondary Index

Opponent Date GameId Status Host

Alice 2014-10-02 d9bl3 DONE David

Carol 2014-10-08 o2pnb IN_PROGRESS Bob

Bob 2014-09-30 72f49 PENDING Alice

Bob 2014-10-03 b932s PENDING Carol

Bob 2014-10-03 ef9ca IN_PROGRESS David

Bob

Page 49: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Secondary Index

Bob

Opponent Date GameId Status Host

Alice 2014-10-02 d9bl3 DONE David

Carol 2014-10-08 o2pnb IN_PROGRESS Bob

Bob 2014-09-30 72f49 PENDING Alice

Bob 2014-10-03 b932s PENDING Carol

Bob 2014-10-03 ef9ca IN_PROGRESS David

SELECT * FROM Game

WHERE Opponent='Bob'

ORDER BY Date DESC

FILTER ON Status='PENDING'

(filtered out)

Page 50: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 51: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Bob

Page 52: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Bob

Page 53: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 54: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

StatusDate

DONE_2014-10-02

IN_PROGRESS_2014-10-08

IN_PROGRESS_2014-10-03

PENDING_2014-09-30

PENDING_2014-10-03

Status

DONE

IN_PROGRESS

IN_PROGRESS

PENDING

PENDING

Date

2014-10-02

2014-10-08

2014-10-03

2014-10-03

2014-09-30

Page 55: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Secondary Index

Opponent StatusDate GameId Host

Alice DONE_2014-10-02 d9bl3 David

Carol IN_PROGRESS_2014-10-08 o2pnb Bob

Bob IN_PROGRESS_2014-10-03 ef9ca David

Bob PENDING_2014-09-30 72f49 Alice

Bob PENDING_2014-10-03 b932s Carol

Page 56: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Opponent StatusDate GameId Host

Alice DONE_2014-10-02 d9bl3 David

Carol IN_PROGRESS_2014-10-08 o2pnb Bob

Bob IN_PROGRESS_2014-10-03 ef9ca David

Bob PENDING_2014-09-30 72f49 Alice

Bob PENDING_2014-10-03 b932s Carol

Secondary Index

Bob

SELECT * FROM Game

WHERE Opponent='Bob'

AND StatusDate STARTS_WITH 'PENDING'

Page 57: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Status + Date

Page 58: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Distribute large items

Page 59: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 60: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Messages

Table

Messages app

Page 61: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

David

Recipient Date Sender Message

David 2014-10-02 Bob …

… 48 more messages for David …

David 2014-10-03 Alice …

Alice 2014-09-28 Bob …

Alice 2014-10-01 Carol …

(Many more messages)

MessagesTable

Page 62: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Recipient Date Sender Message

David 2014-10-02 Bob …

… 48 more messages for David …

David 2014-10-03 Alice …

Alice 2014-09-28 Bob …

Alice 2014-10-01 Carol …

(Many more messages)

David

MessagesTable

SELECT *

FROM Messages

WHERE Recipient='David'

LIMIT 50

ORDER BY Date DESC

Page 63: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Recipient Date Sender Message

David 2014-10-02 Bob …

… 48 more messages for David …

David 2014-10-03 Alice …

Alice 2014-09-28 Bob …

Alice 2014-10-01 Carol …

(Many more messages)

David

MessagesTable

Cost of query?

SELECT *

FROM Messages

WHERE Recipient='David'

LIMIT 50

ORDER BY Date DESC

Page 64: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Recipient Date Sender Message

David 2014-10-02 Bob …

… 48 more messages for David …

David 2014-10-03 Alice …

Alice 2014-09-28 Bob …

Alice 2014-10-01 Carol …

(Many more messages)

MessagesTable

Average item size = 256 KB

Large message bodies

Attachments?

David

SELECT *

FROM Messages

WHERE Recipient='David'

LIMIT 50

ORDER BY Date DESC

Page 65: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Recipient Date Sender Message

David 2014-10-02 Bob …

… 48 more messages for David …

David 2014-10-03 Alice …

Alice 2014-09-28 Bob …

Alice 2014-10-01 Carol …

(Many more messages)

David

MessagesTable

50 items × 256 KB each

SELECT *

FROM Messages

WHERE Recipient='David'

LIMIT 50

ORDER BY Date DESC

Page 66: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Items evaluated by query

Average item size

Conversion ratio

Eventually consistent reads

Page 67: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Messages

Table

Inbox

Table

Messages app

Page 68: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Recipient Date Sender Subject MsgId

David 2014-10-02 Bob Hi!… afed

David 2014-10-03 Alice RE: The… 3kf8

Alice 2014-09-28 Bob FW: Ok… 9d2b

Alice 2014-10-01 Carol Hi!... ct7r

Inbox Table Messages Table

MsgId Body

9d2b …

3kf8 …

ct7r …

afed …

(Bulk message data)(Message metadata, indexed by recipient)

Page 69: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Recipient Date Sender Subject MsgId

David 2014-10-02 Bob Hi!… afed

David 2014-10-03 Alice RE: The… 3kf8

Alice 2014-09-28 Bob FW: Ok… 9d2b

Alice 2014-10-01 Carol Hi!... ct7r

Inbox Table Messages Table

MsgId Body

9d2b …

3kf8 …

ct7r …

afed …

David1. Query Inbox: 1 RCU

(50 sequential items at 128 bytes)

Page 70: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Recipient Date Sender Subject MsgId

David 2014-10-02 Bob Hi!… afed

David 2014-10-03 Alice RE: The… 3kf8

Alice 2014-09-28 Bob FW: Ok… 9d2b

Alice 2014-10-01 Carol Hi!... ct7r

Inbox Table Messages Table

MsgId Body

9d2b …

3kf8 …

ct7r …

afed …

David1. Query Inbox: 1 RCU

2. BatchGetItem Messages: 1600 RCU

(50 separate items at 256 KB)

(50 sequential items at 128 bytes)

Page 71: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Messages

Table

Inbox

Table

David

PutItem

{

MsgId: 123,

Body: ...

}

PutItem

{

Recipient: Steve,

Date: 2014-10-23,

MsgId: 123,

...

}

Page 72: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Messages

Table

Inbox

Table

David

PutItem

{

MsgId: 123,

Body: ...

}

PutItem

{

Recipient: Steve,

Date: 2014-10-23,

MsgId: 123,

...

}

Page 73: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

David

PutItem

{

MsgId: 123,

Body: ...,

Recipient: Steve,

Sender: David,

Date: 2014-10-23,

...

}

Inbox

Global secondary

index

Messages

Table

Page 74: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 75: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Inbox

Global secondary

index

Messages

Table

David

Outbox

Global secondary

index

Coming soon:

Page 76: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Inbox

Messages

Page 77: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

De-normalize schemas

Design for retries

Page 78: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 79: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User HighScore

Chuck 20

Alice 1000

Bob 470

Dave 850

Erin 580

Friends Table HighScores Table

User Friend

Bob Alice

Bob Chuck

Alice Bob

Alice Chuck

Alice Erin

Page 80: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User HighScore

Chuck 20

Alice 1000

Bob 470

Dave 850

Erin 580

Friends Table HighScores Table

User Friend

Bob Alice

Bob Chuck

Alice Bob

Alice Chuck

Alice Erin

Alice

Page 81: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User HighScore

Chuck 20

Alice 1000

Bob 470

Dave 850

Erin 580

Friends Table HighScores Table

User Friend

Bob Alice

Bob Chuck

Alice Bob

Alice Chuck

Alice Erin

1. Query 2. BatchGetItemAlice

Page 82: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 83: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User Friend H.S.

Bob Alice 1000

Bob Chuck 20

Alice Bob 470

Alice Chuck 20

Alice Erin 580

User HighScore

Chuck 20

Alice 1000

Bob 470

Dave 850

Erin 580Table

FriendScores Table HighScores Table

Page 84: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User Friend H.S.

Bob Alice 1000

Bob Chuck 20

Alice Bob 470

Alice Chuck 20

Alice Erin 580

User HighScore

Chuck 20

Alice 1000

Bob 470

Dave 850

Erin 580

User H.S. Friend

Bob 20 Chuck

Bob 1000 Alice

Alice 20 Chuck

Alice 470 Bob

Alice 580 Erin

Table Local Secondary Index

FriendScores Table HighScores Table

Page 85: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User Friend H.S.

Bob Alice 1000

Bob Chuck 20

Alice Bob 470

Alice Chuck 20

Alice Erin 580

User HighScore

Chuck 20

Alice 1000

Bob 470

Dave 850

Erin 580

User H.S. Friend

Bob 20 Chuck

Bob 1000 Alice

Alice 20 Chuck

Alice 470 Bob

Alice 580 Erin

Table Local Secondary Index

FriendScores Table HighScores Table

Query

FROM FriendScores-Index

WHERE User=Alice, DESC, LIMIT 10Alice

Page 86: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

ScoresFriends

Page 87: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User HighScore

Chuck 30

Alice 1000

Bob 470

Dave 850

Erin 580

User Friend H.S.

Bob Alice 1000

Bob Chuck 30

Alice Bob 470

Alice Chuck 30

Alice Erin 580

FriendScores Table HighScores Table

1. UpdateItem

2. UpdateItem(s)

Chuck

Page 88: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User HighScore

Chuck 30

Alice 1000

Bob 470

Dave 850

Erin 580

User Friend H.S.

Bob Alice 1000

Bob Chuck 30

Alice Bob 470

Alice Chuck 20

Alice Erin 580

FriendScores Table HighScores Table

1. UpdateItem

2. UpdateItem(s)

Chuck

Page 89: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User H.S. Updating

Chuck 20

Alice 1000

Bob 470

Dave 850

Erin 580

User Friend H.S.

Bob Alice 1000

Bob Chuck 20

Alice Bob 470

Alice Chuck 20

Alice Erin 580

FriendScores Table HighScores Table

true while updating the high score

Page 90: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User H.S. Updating

Chuck 30 true

Alice 1000

Bob 470

Dave 850

Erin 580

User Friend H.S.

Bob Alice 1000

Bob Chuck 20

Alice Bob 470

Alice Chuck 20

Alice Erin 580

FriendScores Table HighScores Table

1. UpdateItemChuck

Page 91: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User H.S. Updating

Chuck 30 true

Alice 1000

Bob 470

Dave 850

Erin 580

User Friend H.S.

Bob Alice 1000

Bob Chuck 30

Alice Bob 470

Alice Chuck 20

Alice Erin 580

FriendScores Table HighScores TableChuck

2. UpdateItem(s)

Page 92: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User H.S. Updating

Chuck 30 true

Alice 1000

Bob 470

Dave 850

Erin 580

User Friend H.S.

Bob Alice 1000

Bob Chuck 30

Alice Bob 470

Alice Chuck 20

Alice Erin 580

FriendScores Table HighScores TableChuck

3. GetItem

Page 93: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User H.S. Updating

Chuck 30 true

Alice 1000

Bob 470

Dave 850

Erin 580

User Friend H.S.

Bob Alice 1000

Bob Chuck 30

Alice Bob 470

Alice Chuck 30

Alice Erin 580

FriendScores Table HighScores TableChuck

4. UpdateItem(s)

Page 94: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User H.S. Updating

Chuck 30

Alice 1000

Bob 470

Dave 850

Erin 580

User Friend H.S.

Bob Alice 1000

Bob Chuck 30

Alice Bob 470

Alice Chuck 30

Alice Erin 580

FriendScores Table HighScores Table

5. UpdateItemChuck

Page 95: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 96: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

PutItemRequest acceptBob;

PutItemRequest acknowledgeChuck;

Transaction t = txManager.newTransaction();

t.putItem(acceptBob);

t.putItem(acknowledgeChuck);

t.commit();

Page 97: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Friends TableTransactions

Table

Transaction Images

Table

Transaction

Client

Bob

Page 98: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 99: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 100: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Your app

FriendScores

Table

HighScores

Table

Page 101: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Your app

FriendScores

Table

HighScores

Table

Page 102: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Your

Kinesis-enabled

app

Your app

FriendScores

Table

HighScores

DynamoDB

Stream

HighScores

Table

Page 103: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User H.S.

Chuck 20

Alice 100

HighScores Table

Chuck

Seq User Old New

1 Chuck 10 20

2 Alice 80 90

3 Alice 90 100

HighScores Stream

Page 104: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User H.S.

Chuck 30

Alice 100

HighScores Table

Chuck

Seq User Old New

1 Chuck 10 20

2 Alice 80 90

3 Alice 90 100

HighScores Stream

Page 105: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User H.S.

Chuck 30

Alice 100

HighScores Table

Chuck

Seq User Old New

1 Chuck 10 20

2 Alice 80 90

3 Alice 90 100

4 Chuck 20 30

HighScores Stream

Page 106: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User H.S.

Chuck 30

Alice 100

HighScores Table

Chuck

Seq User Old New

1 Chuck 10 20

2 Alice 80 90

3 Alice 90 100

4 Chuck 20 30

HighScores Stream

Page 107: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User H.S.

Chuck 30

Alice 100

HighScores Table

User Friend H.S.

Bob Chuck 20

Alice Chuck 20

FriendScores Table

Friend User

Chuck Alice

Chuck Bob

Table Global Secondary Index

Seq User Old New

1 Chuck 10 20

2 Alice 80 90

3 Alice 90 100

4 Chuck 20 30

HighScores Stream

Your

Kinesis-enabled

app

Page 108: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User H.S.

Chuck 30

Alice 100

HighScores Table

User Friend H.S.

Bob Chuck 20

Alice Chuck 20

FriendScores Table

Friend User

Chuck Alice

Chuck Bob

Table Global Secondary Index

Seq User Old New

1 Chuck 10 20

2 Alice 80 90

3 Alice 90 100

4 Chuck 20 30

HighScores Stream

Your

Kinesis-enabled

app

Page 109: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User H.S.

Chuck 30

Alice 100

HighScores Table

User Friend H.S.

Bob Chuck 30

Alice Chuck 30

FriendScores Table

Friend User

Chuck Alice

Chuck Bob

Table Global Secondary Index

Seq User Old New

1 Chuck 10 20

2 Alice 80 90

3 Alice 90 100

4 Chuck 20 30

HighScores Stream

Your

Kinesis-enabled

app

Page 110: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

User H.S.

Chuck 30

Alice 100

HighScores Table

User Friend H.S.

Bob Chuck 30

Alice Chuck 30

FriendScores Table

Friend User

Chuck Alice

Chuck Bob

Table Global Secondary Index

Seq User Old New

1 Chuck 10 20

2 Alice 80 90

3 Alice 90 100

4 Chuck 20 30

HighScores Stream

Your

Kinesis-enabled

app

Streams

Checkpoint

Table

Page 111: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 112: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Shard write-heavy items

Design for retries

Page 113: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Candidate A

Votes: 20

Candidate B

Votes: 30

Voter

Votes Table

Page 114: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Candidate A

Votes: 21

Candidate B

Votes: 30

Votes Table

UPDATE VotesSET Votes = Votes + 1WHERE Id=“Candidate A”Voter

(aka atomic increment)

Page 115: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Need to scale

for the election

Administrator (You)

Votes Table

Page 116: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Administrator (You)

Provision 200,000 write capacity units

Partition 1

1000 WCU

Partition K

1000 WCU

Partition M

1000 WCU

Partition N

1000 WCU

Votes Table

Page 117: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Candidate A Candidate B

Voters

Partition 1

1000 WCU

Partition K

1000 WCU

Partition M

1000 WCU

Partition N

1000 WCU

Votes Table

Page 118: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Partition 1

1000 WCU

Partition K

1000 WCU

Partition M

1000 WCU

Partition N

1000 WCU

Votes Table

Candidate A Candidate B

Voters

Page 119: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Candidate A

Candidate A_2

Candidate A_1

Candidate A_3

Candidate A_4Candidate A_7

Candidate A_6 Candidate A_8

Candidate A_5

Page 120: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Candidate A_2

Candidate B_1

Candidate B_2

Candidate B_3

Candidate B_5

Candidate B_4

Candidate B_7

Candidate B_6

Candidate A_1

Candidate A_3

Candidate A_4Candidate A_7 Candidate B_8

Candidate A_6 Candidate A_8

Candidate A_5

Voter

Votes Table

Page 121: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Candidate A_2

Candidate B_1

Candidate B_2

Candidate B_3

Candidate B_5

Candidate B_4

Candidate B_7

Candidate B_6

Candidate A_1

Candidate A_3

Candidate A_4Candidate A_7 Candidate B_8

UpdateItem: “CandidateA_” + rand(0, 10)

ADD 1 to Votes

Candidate A_6 Candidate A_8

Candidate A_5

Voter

Votes Table

Page 122: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Votes Table

Candidate A_2

Candidate B_1

Candidate B_2

Candidate B_3

Candidate B_5

Candidate B_4

Candidate B_7

Candidate B_6

Candidate A_1

Candidate A_3

Candidate A_4

Candidate A_5

Candidate A_6 Candidate A_8

Candidate A_7 Candidate B_8

Periodic

Process

Candidate A

Total: 2.5M

1. Sum

2. Store Voter

Page 123: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 124: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 125: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

UserId Candidate Date

Alice A 2013-10-02

Bob B 2013-10-02

Eve B 2013-10-02

Chuck A 2013-10-02

RawVotes Table

Voter

Page 126: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

UserId Candidate Date

Alice A 2013-10-02

Bob B 2013-10-02

Eve B 2013-10-02

Chuck A 2013-10-02

RawVotes Table

Segment Votes

A_1 23

B_2 12

B_1 14

A_2 25

AggregateVotes Table

Voter

1. Record vote and de-dupe, retry 2. Increment candidate counter

Page 127: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

UserId Candidate Date

Alice A 2013-10-02

Bob B 2013-10-02

Eve B 2013-10-02

Chuck A 2013-10-02

RawVotes Table

Segment Votes

A_1 23

B_2 12

B_1 14

A_2 25

AggregateVotes Table

Voter

Page 128: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

AggregateVotes

Table

Voters

RawVotes Table

Voting app

Page 129: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

AggregateVotes

Table

Amazon

Redshift Amazon EMR

Your

Kinesis-enabled

app

Voters RawVotes TableVoting app RawVotes

DynamoDB

Stream

Page 130: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

AggregateVotes

Table

Amazon

Redshift Amazon EMR

Your

Kinesis-enabled

app

Voters RawVotes TableVoting app RawVotes

DynamoDB

Stream

Page 131: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

AggregateVotes

Table

Amazon

Redshift Amazon EMR

Your

Kinesis-enabled

app

Voters RawVotes TableVoting app RawVotes

DynamoDB

Stream

Page 132: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

AggregateVotes

Table

Amazon

Redshift Amazon EMR

Your

Kinesis-enabled

app

Voters RawVotes TableVoting app RawVotes

DynamoDB

Stream

Page 133: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

AggregateVotes

Table

Amazon

Redshift Amazon EMR

Your

Kinesis-enabled

app

Voters RawVotes TableVoting app RawVotes

DynamoDB

Stream

Page 134: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 135: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 136: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 137: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 138: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Status + Date

Page 139: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

Inbox

Messages

Page 140: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

ScoresFriends

Page 141: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 142: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Page 143: (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014

http://bit.ly/awsevals