aws webcast - data modeling and best practices for scaling your application with amazon dynamodb
DESCRIPTION
Amazon DynamoDB is a fully managed, highly scalable distributed database service. In this technical talk, we will deep dive on how to: • Use DynamoDB to build high-scale applications like social gaming, chat, and voting. • Model these applications using DynamoDB, including how to use building blocks such as conditional writes, consistent reads, and batch operations to build the higher-level functionality such as multi-item atomic writes and join queries. • Incorporate best practices such as index projections, item sharding, and parallel scan for maximum scalabilityTRANSCRIPT
Peter-Mark Verwoerd
Data Modeling and Best Practices in DynamoDB
Solutions Architect
Traditional Database Architecture
App/Web Tier
Client Tier
RDBMS
One Database for All Workloads
• key-value access
• complex queries
• transactions
• analytics
App/Web Tier
Client Tier
RDBMS
Cloud Data Tier Architecture
App/Web Tier
Client Tier
Data Tier
Search Cache Blob Store
RDBMS NoSQL Data Warehouse
Workload Driven Data Store Selection
Data Tier
Search Cache Blob Store
RDBMS NoSQL Data Warehouse
logging analytics
key/value simple query
rich search
transaction processing
hot reads
AWS Services for the Data Tier
Data Tier
Amazon DynamoDB
Amazon RDS
Amazon ElastiCache
Amazon S3
Amazon Redshift
Amazon CloudSearch
logging analytics
key/value simple query
rich search
transaction processing
hot reads
Plan
• Basics
– Basic Game State
– Save Games
– Social Gaming
• Advanced
– Social Gaming
– Voting
– Global Leaderboard
Plan
• Basics
– Basic Game State (conditional writes)
– Save Games (hash + range)
– Social Gaming (secondary indexes)
• Advanced
– Social Gaming (transactions)
– Voting (write sharding)
– Global Leaderboard (scatter-gather query)
Basic Game State
conditional writes
Basics
Tic Tac Toe
Basics
Tic Tac Toe
Alice Bob
DynamoDB
Your App
Basics
Tic Tac Toe Table
Game Table
Id Players O State IsTie Winner Data
abecd [ Alice, Bob ] Alice DONE 1 …
fbdcc [ Alice, Bob ] Alice DONE Alice …
dbace [ Alice, Bob ] Alice STARTED …
Basics
Tic Tac Toe Table
Game Table
Id Players O State IsTie Winner Data
abecd [ Alice, Bob ] Alice DONE 1 …
fbdcc [ Alice, Bob ] Alice DONE Alice …
dbace [ Alice, Bob ] Alice STARTED …
Basics
Id Players O State IsTie Winner Data
abecd [ Alice, Bob ] Alice DONE 1 …
fbdcc [ Alice, Bob ] Alice DONE Alice …
dbace [ Alice, Bob ] Alice STARTED …
Tic Tac Toe Table
Item
Basics
Id Players O State IsTie Winner Data
abecd [ Alice, Bob ] Alice DONE 1 …
fbdcc [ Alice, Bob ] Alice DONE Alice …
dbace [ Alice, Bob ] Alice STARTED …
Tic Tac Toe Table
Attribute
Basics
Id Players O State IsTie Winner Data
abecd [ Alice, Bob ] Alice DONE 1 …
fbdcc [ Alice, Bob ] Alice DONE Alice …
dbace [ Alice, Bob ] Alice STARTED …
Tic Tac Toe Table
Primary Key
Basics
Id Players O State IsTie Winner Data
abecd [ Alice, Bob ] Alice DONE 1 …
fbdcc [ Alice, Bob ] Alice DONE Alice …
dbace [ Alice, Bob ] Alice STARTED …
Tic Tac Toe Table
Set String Number Binary
Basics
Tic Tac Toe Table
{
"Data" : [ [ "X", null, "O" ],
[ null, "O", null],
[ "O", null, "X" ]
]
}
Id Players O State IsTie Winner Data
abecd [ Alice, Bob ] Alice DONE 1 …
fbdcc [ Alice, Bob ] Alice DONE Alice …
dbace [ Alice, Bob ] Alice STARTED …
Basics
State Transitions with Conditional Writes
DynamoDB
Alice Bob
Basics
State Transitions with Conditional Writes
DynamoDB
UpdateItem: Top-Right = O Turn = Bob
Alice Bob
Basics
State Transitions with Conditional Writes
DynamoDB
UpdateItem: Top-Left = X Turn = Alice
Alice Bob
Basics
State Transitions with Conditional Writes
Alice Bob (1)
DynamoDB
Bob (2) Bob (3)
Basics
State Transitions with Conditional Writes
Alice Bob (1)
DynamoDB
Bob (2) Bob (3)
Basics
State Transitions with Conditional Writes
Alice Bob (1)
DynamoDB
Bob (2) Bob (3)
Basics
State Transitions with Conditional Writes
Bob (1)
DynamoDB
Bob (2) Bob (3)
State : STARTED,
Turn : Bob,
Top-Right : O
Basics
State Transitions with Conditional Writes
Bob (1)
DynamoDB
Bob (2) Bob (3)
Update:
Turn : Alice
Top-Left : X
Update:
Turn : Alice
Low-Right : X
Update:
Turn : Alice
Mid : X
State : STARTED,
Turn : Bob,
Top-Right : O
Basics
State Transitions with Conditional Writes
Bob (1)
DynamoDB
Bob (2) Bob (3)
Update:
Turn : Alice
Top-Left : X
Update:
Turn : Alice
Low-Right : X
Update:
Turn : Alice
Mid : X
State : STARTED,
Turn : Alice,
Top-Right : O,
Top-Left : X,
Mid: X,
Low-Right: X
Basics
Conditional Writes
• Apply an update only if values are as expected
• Otherwise reject the write
Basics
Conditional Writes
{
Id : abecd,
Players : [ Alice, Bob ],
State : STARTED,
Turn : Bob,
Top-Right: O
}
Game Item Updates: {
Turn : Alice,
Top-Left: X
}
Expected: {
Turn : Bob,
Top-Left : null,
State : STARTED
}
UpdateItem Id=abecd
Basics
State Transitions with Conditional Writes
Bob (1)
DynamoDB
Bob (2) Bob (3)
Update:
Turn : Alice
Top-Left : X
Expect:
Turn : Bob
Top-Left : null
State : STARTED,
Turn : Bob,
Top-Right : O
Update:
Turn : Alice
Low-Right : X
Expect:
Turn : Bob
Low-Right : null
Update:
Turn : Alice
Mid : X
Expect:
Turn : Bob
Mid : null
Basics
State Transitions with Conditional Writes
Bob (1)
DynamoDB
Bob (2) Bob (3)
State : STARTED,
Turn : Bob,
Top-Right : O
Update:
Turn : Alice
Top-Left : X
Expect:
Turn : Bob
Top-Left : null
Update:
Turn : Alice
Low-Right : X
Expect:
Turn : Bob
Low-Right : null
Update:
Turn : Alice
Mid : X
Expect:
Turn : Bob
Mid : null
Basics
State Transitions with Conditional Writes
Bob (1)
DynamoDB
Bob (2) Bob (3)
State : STARTED,
Turn : Alice,
Top-Right : O,
Top-Left : X
Update:
Turn : Alice
Top-Left : X
Expect:
Turn : Bob
Top-Left : null
Update:
Turn : Alice
Low-Right : X
Expect:
Turn : Bob
Low-Right : null
Update:
Turn : Alice
Mid : X
Expect:
Turn : Bob
Mid : null
Basics
Save Games
hash + range
Save Games
Save Games
Save Games
Primary Key Schemas
Id Players O State IsTie Winner Data
abecd [ Alice, Bob ] Alice DONE 1 …
fbdcc [ Alice, Bob ] Alice DONE Alice …
dbace [ Alice, Bob ] Alice STARTED …
Primary Key
Hash Key Schema
Primary Key Schemas
Id Turn Players Turn State IsTie Winner Data
abecd 0 [ Alice, Bob ] Alice STARTED …
abecd 1 [ Alice, Bob ] Bob STARTED …
abecd 2 [ Alice, Bob ] Alice STARTED …
abecd 3 [ Alice, Bob ] Bob STARTED …
abecd 4 [ Alice, Bob ] Alice DONE Alice …
dbace 0 [ Alice, Bob ] Bob STARTED
dbace 1 [ Alice, Bob ] Alice STARTED …
Primary Key
Hash and Range Key Schema
Save Games
Primary Key Schemas
Id Turn Players Turn State IsTie Winner Data
abecd 0 [ Alice, Bob ] Alice STARTED …
abecd 1 [ Alice, Bob ] Bob STARTED …
abecd 2 [ Alice, Bob ] Alice STARTED …
abecd 3 [ Alice, Bob ] Bob STARTED …
abecd 4 [ Alice, Bob ] Alice DONE Alice …
dbace 0 [ Alice, Bob ] Bob STARTED
dbace 1 [ Alice, Bob ] Alice STARTED …
Primary Key
Save Games
Primary Key Schemas
• Hash-only
– Key/value lookups only
• Hash and Range
– Given a hash key value, query for items by range key
– Items are sorted by range key within each hash key
Save Games
Primary Key Schemas
Id Turn Players Turn State IsTie Winner Data
abecd 0 [ Alice, Bob ] Alice STARTED …
abecd 1 [ Alice, Bob ] Bob STARTED …
abecd 2 [ Alice, Bob ] Alice STARTED …
abecd 3 [ Alice, Bob ] Bob STARTED …
abecd 4 [ Alice, Bob ] Alice DONE Alice …
dbace 0 [ Alice, Bob ] Bob STARTED
dbace 1 [ Alice, Bob ] Alice STARTED …
Primary Key
Query WHERE Id=abecd, ORDER BY Turn DESC, LIMIT 2
Save Games
Social Gaming
local secondary indexes
Social Gaming
Social Gaming
• Host games
• Invite friends to play
• Find friends’ games to play
• See history of games
Social Gaming
Social Gaming
HostedGame Table
Hash: UserId Range: GameId Attributes: OpponentId, Date, (rest of game state)
UserId GameId Date OpponentId …
Carol e23f5a 2013-10-08 Charlie …
Alice d4e2dc 2013-10-01 Bob …
Alice e9cba3 2013-09-27 Bob …
Alice f6a3bd 2013-10-08
Social Gaming
Social Gaming
• Host games
• Invite friends to play
• Find friends’ games to play
• See history of games
Social Gaming
Social Gaming: find recent games
UserId GameId Date OpponentId …
Carol e23f5a 2013-10-08 Charlie …
Alice d4e2dc 2013-10-01 Bob …
Alice e9cba3 2013-09-27 Bob …
Alice f6a3bd 2013-10-08
Query UserId=Alice
Social Gaming
Query cost
• Provisioned Throughput: Work / sec allowed on your table
• Capacity Units: Amount of provisioned throughput consumed by an
operation
Social Gaming
Query cost
UserId GameId Date OpponentId …
Carol e23f5a 2013-10-08 Charlie …
Alice d4e2dc 2013-10-01 Bob …
Alice e9cba3 2013-09-27 Bob …
Alice f6a3bd 2013-10-08
(1 item = 600 bytes)
(397 more games for Alice)
Social Gaming
Query cost
UserId GameId Date OpponentId …
Carol e23f5a 2013-10-08 Charlie …
Alice d4e2dc 2013-10-01 Bob …
Alice e9cba3 2013-09-27 Bob …
Alice f6a3bd 2013-10-08
(1 item = 600 bytes)
(397 more games for Alice)
400 X 600 / 1024 / 4 = 60 Read Capacity Units
(bytes per item) (KB per byte)
(KB per Read Capacity Unit) (Items evaluated by Query)
Social Gaming
Local Secondary Indexes
• An alternate range key on a table
UserId GameId Date
Carol e23f5a 2013-10-08
Alice d4e2dc 2013-10-01
Alice e9cba3 2013-09-27
Alice f6a3bd 2013-10-01
UserId Date GameId
Carol 2013-10-08 e23f5a
Alice 2013-09-27 e9cba3
Alice 2013-10-01 d4e2dc
Alice 2013-10-01 f6a3bd
HostedGame Table LocalSecondaryIndex on Date
Social Gaming
Query cost on Local Secondary Indexes
UserId Date GameId …
Carol 2013-10-08 e23f5a …
Alice (397 older games)
Alice 2013-09-27 e9cba3 …
Alice 2013-10-01 d4e2dc …
Alice 2013-10-01 f6a3bd …
Query for the 10 most recent games
Social Gaming
Query cost on Local Secondary Indexes
UserId Date GameId …
Carol 2013-10-08 e23f5a …
Alice (397 older games)
Alice 2013-09-27 e9cba3 …
Alice 2013-10-01 d4e2dc …
Alice 2013-10-01 f6a3bd …
10 X 600 / 1024 / 4 = 2 Read Capacity Units
(bytes per item) (KB per byte)
(KB per Read Capacity Unit) (Items evaluated by Query)
Query for the 10 most recent games
Social Gaming
Example Local Secondary Indexes
• Find 10 recent matches between Alice and Bob
Social Gaming
Example Local Secondary Indexes
• Find 10 recent matches between Alice and Bob
– Hash: UserId
– Range: OpponentId + Date
Social Gaming
Query WHERE UserId=Alice AND OpponentAndDate STARTS_WITH “Bob-” LIMIT 10 DESC
More example Local Secondary Indexes
• Find a host’s matches without an opponent
Social Gaming
More example Local Secondary Indexes
• Find a host’s matches without an opponent
– Hash: UserId
– Range: UnmatchedDate
(sparse index)
Social Gaming
Query WHERE UserId=Alice LIMIT 10 DESC
Local Secondary Index Projections
• Choose what attributes are copied into the index
– ALL, SPECIFIC, KEYS
• Substantially cheaper to Query only projection
• Project the attributes that your use case requires
• Can make writes cheaper too
Social Gaming
Write cost for Local Secondary Index
• Insert new item
– 1 additional write
• Setting index range key to / from null
– 1 additional write
• Updating a projected attribute
– 1 additional write
• Updating a non-projected attribute
– 0 additional writes
• Updating the index range key
– 2 additional writes
Social Gaming
Read cost for Query of non-projected attributes
• Regular Query cost
+
• Single-item Get cost for each evaluated item
Social Gaming
Example Local Secondary Index Projections
• Query Alice’s 10 most recent Games
Social Gaming
UserId GameId Date OpponentId …
Carol e23f5a 2013-10-08 Charlie …
Alice d4e2dc 2013-10-01 Bob …
Alice e9cba3 2013-09-27 Bob …
Alice f6a3bd 2013-10-08
Example Local Secondary Index Projections
• Query Alice’s 10 most recent Games
– Opponent, Winner, (UserId, GameId, Date)
– Projected item size from 600 bytes to 40 bytes
• Write cost:
– 1 Write Capacity Unit for insert, opponent joining, and completion
– 0 Write Capacity Units for other state transitions
Social Gaming
Social Gaming
transactions
Social Gaming
Social Gaming: Friends
• Query who you are friends with
• Ask to be friends with someone
• Acknowledge (or decline) friend request
Social Gaming
Social Gaming: Friends
Friends Table
Hash: UserId Range: FriendId Attributes: Status, Date, etc
UserId FriendId Status Date …
Alice Bob FRIENDS 2013-08-20 …
Bob Alice FRIENDS 2013-08-20 …
Bob Chuck INCOMING 2013-10-08 …
Chuck Bob SENT 2013-10-08 …
Social Gaming
Becoming Friends: Multi-item Atomic Writes
UserId FriendId Status
Alice Bob FRIENDS
Bob Alice FRIENDS
Bob Chuck INCOMING
Chuck Bob SENT
Bob
A friend request!
Social Gaming
Becoming Friends: Multi-item Atomic Writes
UserId FriendId Status
Alice Bob FRIENDS
Bob Alice FRIENDS
Bob Chuck INCOMING
Chuck Bob SENT
Bob
1. Update Bob/Chuck record 2. Update Chuck/Bob record
Social Gaming
Becoming Friends: Multi-item Atomic Writes
UserId FriendId Status
Alice Bob FRIENDS
Bob Alice FRIENDS
Bob Chuck FRIENDS
Chuck Bob SENT
Bob
UpdateItem Status=FRIENDS
1. Update Bob/Chuck record 2. Update Chuck/Bob record
Social Gaming
Becoming Friends: Multi-item Atomic Writes
UserId FriendId Status
Alice Bob FRIENDS
Bob Alice FRIENDS
Bob Chuck FRIENDS
Chuck Bob FRIENDS
Bob
UpdateItem Status=FRIENDS
1. Update Bob/Chuck record 2. Update Chuck/Bob record
Social Gaming
When things go wrong
Social Gaming
Becoming Friends: When things go wrong
UserId FriendId Status
Alice Bob FRIENDS
Bob Alice FRIENDS
Bob Chuck INCOMING
Chuck Bob SENT
Bob
A friend request!
1. Update Bob/Chuck record 2. Update Chuck/Bob record
Social Gaming
Becoming Friends: When things go wrong
UserId FriendId Status
Alice Bob FRIENDS
Bob Alice FRIENDS
Bob Chuck FRIENDS
Chuck Bob SENT
Bob
UpdateItem Status=FRIENDS
1. Update Bob/Chuck record 2. Update Chuck/Bob record
Social Gaming
Becoming Friends: When things go wrong
UserId FriendId Status
Alice Bob FRIENDS
Bob Alice FRIENDS
Bob Chuck FRIENDS
Chuck Bob SENT
Bob
UpdateItem Status=FRIENDS
1. Update Bob/Chuck record 2. Update Chuck/Bob record
Social Gaming
Multi-item transaction in DynamoDB
• Scan for “stuck” transactions
• Use the Client Transactions Library on the AWS SDK for Java
• Roll your own scheme
Social Gaming
Replayable state machines
INCOMING ACCEPTING FRIENDS
SENT FRIENDS SENDING
Bob/ Chuck
Chuck/ Bob
Social Gaming
Client Transactions Library
Friends Table Transactions Table
Transaction Images Table
Transaction Client
Bob
Social Gaming
Client Transactions Usage
• Low contention only
• Don’t mix Tx Client writes with normal writes
• No Query support
• Expensive, slower
• But, easy to use
Social Gaming
Specialized Transactions
UserId FriendId Status V
Alice Bob FRIENDS 3
Bob Alice FRIENDS 3
Bob Chuck INCOMING 2
Chuck Bob SENT 2
Id Status V1 V2
Bob
1. Read items 2. Write to Tx table 3. Apply writes 4. Delete from Tx table
Transactions Table
Social Gaming
A friend request!
Specialized Transactions
UserId FriendId Status V
Alice Bob FRIENDS 3
Bob Alice FRIENDS 3
Bob Chuck INCOMING 2
Chuck Bob SENT 2
Id Status V1 V2
Bob BatchGetItem
1. Read items 2. Write to Tx table 3. Apply writes 4. Delete from Tx table
Transactions Table
Social Gaming
Specialized Transactions
UserId FriendId Status V
Alice Bob FRIENDS 3
Bob Alice FRIENDS 3
Bob Chuck INCOMING 2
Chuck Bob SENT 2
Id Status V1 V2
Bob-Chuck Bob: FRIENDS Chuck: FRIENDS
2 2
Bob PutItem, Expect not exists
1. Read items 2. Write to Tx table 3. Apply writes 4. Delete from Tx table
Transactions Table
Social Gaming
Specialized Transactions
UserId FriendId Status V
Alice Bob FRIENDS 3
Bob Alice FRIENDS 3
Bob Chuck FRIENDS 3
Chuck Bob FRIENDS 3
Id Status V1 V2
Bob-Chuck Bob: FRIENDS Chuck: FRIENDS
2 2
Bob UpdateItem, Expect V=Vprev
1. Read items 2. Write to Tx table 3. Apply writes 4. Delete from Tx table
Transactions Table
Social Gaming
Specialized Transactions
UserId FriendId Status V
Alice Bob FRIENDS 3
Bob Alice FRIENDS 3
Bob Chuck FRIENDS 3
Chuck Bob FRIENDS 3
Id Status V1 V2
Bob DeleteItem, Expect V1=V1prev,
V2=V2prev,
1. Read items 2. Write to Tx table 3. Apply writes 4. Delete from Tx table
Transactions Table
Social Gaming
When things go wrong
Social Gaming
Specialized Transactions
UserId FriendId Status V
Alice Bob FRIENDS 3
Bob Alice FRIENDS 3
Bob Chuck INCOMING 2
Chuck Bob SENT 2
Id Status V1 V2
Bob BatchGetItem
1. Read items 2. Write to Tx table 3. Apply writes 4. Delete from Tx table
Transactions Table
Social Gaming
Specialized Transactions
UserId FriendId Status V
Alice Bob FRIENDS 3
Bob Alice FRIENDS 3
Bob Chuck INCOMING 2
Chuck Bob SENT 2
Id Status V1 V2
Bob-Chuck Bob: FRIENDS Chuck: FRIENDS
2 2
Bob PutItem, Expect not exists
1. Read items 2. Write to Tx table 3. Apply writes 4. Delete from Tx table
Transactions Table
Social Gaming
Specialized Transactions
UserId FriendId Status V
Alice Bob FRIENDS 3
Bob Alice FRIENDS 3
Bob Chuck FRIENDS 3
Chuck Bob SENT 2
Id Status V1 V2
Bob-Chuck Bob: FRIENDS Chuck: FRIENDS
2 2
Bob UpdateItem, Expect V=Vprev
1. Read items 2. Write to Tx table 3. Apply writes 4. Delete from Tx table
Transactions Table
Social Gaming
Social Gaming
Specialized Transactions
UserId FriendId Status V
Alice Bob FRIENDS 3
Bob Alice FRIENDS 3
Bob Chuck FRIENDS 3
Chuck Bob SENT 2
Id Status V1 V2
Bob-Chuck Bob: FRIENDS Chuck: FRIENDS
2 2
Sweeper Scan
1. Scan for stuck Tx 2. Apply writes 3. Delete from Tx table
Transactions Table
Social Gaming
Specialized Transactions
UserId FriendId Status V
Alice Bob FRIENDS 3
Bob Alice FRIENDS 3
Bob Chuck FRIENDS 3
Chuck Bob FRIENDS 3
Id Status V1 V2
Bob-Chuck Bob: FRIENDS Chuck: FRIENDS
2 2
UpdateItem, Expect V=Vprev
Transactions Table
Sweeper
1. Scan for stuck Tx 2. Apply writes 3. Delete from Tx table
Social Gaming
Specialized Transactions
UserId FriendId Status V
Alice Bob FRIENDS 3
Bob Alice FRIENDS 3
Bob Chuck FRIENDS 3
Chuck Bob FRIENDS 3
Id Status V1 V2
DeleteItem, Expect V1=V1prev,
V2=V2prev,
Transactions Table
Sweeper
1. Scan for stuck Tx 2. Apply writes 3. Delete from Tx table
Social Gaming
Transaction advice
• Lock items before modifying
– Including items that don’t exist yet
• Don’t stomp on future writes (use versions)
• Sweep for stuck transactions
• Avoid deadlock
Social Gaming
High-Throughput Voting
write sharding
Voting
Voting
Votes Table
Voter
Candidate A Votes: 20
Candidate B Votes: 30
Voting
Voting
Votes Table
Voter
Candidate A Votes: 21
Candidate B Votes: 30
UpdateItem ADD 1 to “Candidate A” (aka Atomic Increment)
Voting
Scaling on DynamoDB
Votes Table
You Need to scale
for the election
Voting
Scaling on DynamoDB
Votes Table
You
Provision 1200 Write Capacity Units
Voting
Scaling on DynamoDB
Votes Table
Partition 1 Partition 2
You
600 Write Capacity Units (each)
Provision 1200 Write Capacity Units
Voting
Scaling on DynamoDB
Votes Table
Partition 1 Partition 2
You
(no sharing)
Provision 1200 Write Capacity Units
Voting
Scaling on DynamoDB
Votes Table
You
Provision 200,000 Write Capacity Units
Partition 1 (600 WCU)
Partition K (600 WCU)
Partition M (600 WCU)
Partition N (600 WCU)
Voting
Scaling bottlenecks
Votes Table
Partition 1 (600 WCU)
Candidate A
Partition K (600 WCU)
Partition M (600 WCU)
Partition N (600 WCU)
Candidate B
Voters
Voting
Scaling bottlenecks
Votes Table
Partition 1 (600 WCU)
Candidate A
Partition K (600 WCU)
Partition M (600 WCU)
Partition N (600 WCU)
Candidate B
Voters
Voting
Best Practice: Uniform Workloads
“To achieve the full amount of request throughput you have
provisioned for a table, keep your workload spread evenly across the hash
key values.” – DynamoDB Developer Guide
Voting
Scaling on DynamoDB
Votes Table
Candidate A_2
Candidate B_1
Candidate B_2
Candidate B_3
Candidate B_5
Candidate B_4
Candidate B_7
Candidate B_6
Candidate A_1
Candidate A_3
Candidate A_4 Candidate A_7 Candidate B_8
Voter
Candidate A_6 Candidate A_8
Candidate A_5
Voting
Scaling on DynamoDB
Votes Table
Candidate A_2
Candidate B_1
Candidate B_2
Candidate B_3
Candidate B_5
Candidate B_4
Candidate B_7
Candidate B_6
Candidate A_1
Candidate A_3
Candidate A_4 Candidate A_7 Candidate B_8
Voter
UpdateItem: “CandidateA_” + rand(0, 10) ADD 1 to Votes
Candidate A_6 Candidate A_8
Candidate A_5
Voting
Scaling on DynamoDB
Votes Table
Candidate A_2
Candidate B_1
Candidate B_2
Candidate B_3
Candidate B_5
Candidate B_4
Candidate B_7
Candidate B_6
Candidate A_1
Candidate A_3
Candidate A_4
Candidate A_5
Candidate A_6 Candidate A_8
Candidate A_7 Candidate B_8
Periodic Process
Candidate A Total: 2.5M
1. Sum
2. Store Voter
Voting
Scaling Reads
Votes Table
Partition 1 (600 WCU)
Candidate A_Total Votes: 2.5M
Partition K (600 WCU)
Partition M (600 WCU)
Partition N (600 WCU)
Candidate B_Total Votes: 2.1M
Voters
Voting
Scaling Reads
Votes Table
Partition 1 (600 WCU)
Candidate A_Total Votes: 2.5M
Partition K (600 WCU)
Partition M (600 WCU)
Partition N (600 WCU)
Candidate B_Total Votes: 2.1M
Voters
Voting
Global Leaderboard
scatter-gather
Leaderboards
Game-Wide Leaderboard
• Find the top 10 scores game-wide
HighScore User
1000 Alice
850 Dave
580 Erin
470 Bob
30 Chuck
Leaderboards
Game-Wide Leaderboard
• Find the top 10 scores game-wide
HighScore User
1000 Alice
850 Dave
580 Erin
470 Bob
30 Chuck
Table Schemas must begin with a Hash Key
Leaderboards
Game-Wide Leaderboard
• Find the top 10 scores game-wide
Cannot be Queried the way we want
User HighScore
Chuck 20
Alice 1000
Bob 470
Dave 850
Erin 580
Leaderboards
Game-Wide Leaderboard
• Use a constant Hash key?
Constant HighScore-User
1 0001000-Alice
1 0000850-Dave
1 0000580-Erin
1 0000470-Bob
1 0000030-Chuck Zero-pad strings for sort stability
Leaderboards
Game-Wide Leaderboard
• Use a constant Hash key?
Constant HighScore-User
1 0001000-Alice
1 0000850-Dave
1 0000580-Erin
1 0000470-Bob
1 0000030-Chuck
Extremely non-uniform workload
Leaderboards
Scatter-Gather Leading Range Key
HighScores Table
Shard HighScore-User
1 0001000-Alice
1 0000850-Dave
1 0000580-Erin
Shard HighScore-User
3 0000900-Dan
3 0000850-Wendy
3 0000080-Trent
Shard HighScore-User
2 0000980-Eve
2 0000600-Frank
2 0000581-Trent
Shard HighScore-User
4 0000500-Merlin
4 0000350-Carole
4 0000280-Paul
Shard HighScore-User
5 0000999-Oscar
5 0000700-Craig
5 0000030-Chuck
Leaderboards
Scatter-Gather Leading Range Key
HighScores Table
Shard HighScore-User
1 0001000-Alice
1 0000850-Dave
1 0000580-Erin
Shard HighScore-User
3 0000900-Dan
3 0000850-Wendy
3 0000080-Trent
Shard HighScore-User
2 0000980-Eve
2 0000600-Frank
2 0000581-Trent
Shard HighScore-User
4 0000500-Merlin
4 0000350-Carole
4 0000280-Paul
Shard HighScore-User
5 0000999-Oscar
5 0000700-Craig
5 0000030-Chuck
1. Periodically Query each Shard DESC, LIMIT N
Leaderboards
Scatter-Gather Leading Range Key
HighScores Table
Shard HighScore-User
1 0001000-Alice
1 0000850-Dave
1 0000580-Erin
Shard HighScore-User
3 0000900-Dan
3 0000850-Wendy
3 0000080-Trent
Shard HighScore-User
2 0000980-Eve
2 0000600-Frank
2 0000581-Trent
Shard HighScore-User
4 0000500-Merlin
4 0000350-Carole
4 0000280-Paul
Shard HighScore-User
5 0000999-Oscar
5 0000700-Craig
5 0000030-Chuck
2. Keep only the top N, Store somewhere
HighScore User
1000 Alice
999 Oscar
Leaderboards
Scatter-Gather Leading Range Key
HighScores Table
Shard HighScore-User
1 0001000-Alice
1 0000850-Dave
1 0000580-Erin
Shard HighScore-User
3 0000900-Dan
3 0000850-Wendy
3 0000080-Trent
Shard HighScore-User
2 0000980-Eve
2 0000600-Frank
2 0000581-Trent
Shard HighScore-User
4 0000500-Merlin
4 0000350-Carole
4 0000280-Paul
Shard HighScore-User
5 0000999-Oscar
5 0000700-Craig
5 0000030-Chuck
Store the Shard id by User for high score updates
User Shard
Alice 1
Oscar 5
Carole 4
Leaderboards
Recap
• Basics
– Basic Game State (conditional writes)
– Save Games (hash + range)
– Social Gaming (secondary indexes)
• Advanced
– Social Gaming (transactions)
– Replication (cross-region, cross-data-store)
– Global Leaderboard (scatter-gather query)
Thanks!
aws.amazon.com/dynamodb/resources