(sdd424) simplifying scalable distributed applications using dynamodb streams | aws re:invent 2014

57
SDD424 Simplifying Scalable Distributed Applications Using DynamoDB Streams

Upload: amazon-web-services

Post on 30-Jun-2015

994 views

Category:

Technology


0 download

DESCRIPTION

Dynamo Streams provides a stream of all the updates done to your DynamoDB table. It is a simple but extremely powerful primitive which will enable developers to easily build solutions like cross-region replication, and to host additional materialized views, for instance an ElasticSearch index, on top of DynamoDB tables. In this session we will dive deep into details of Dynamo Streams, and how customers can leverage Dynamo Streams to build custom solutions and to extend the functionality of DynamoDB. We will give a demo of an example application built on top of Dynamo Streams to demonstrate the power and simplicity of Dynamo Streams.

TRANSCRIPT

Page 1: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

SDD424

Simplifying Scalable Distributed

Applications Using DynamoDB Streams

Page 2: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

• Why DynamoDB Streams

• What DynamoDB Streams

• How

Demo@akshatvig

Page 3: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

use cases

Onlinegaming

Ad tech

Live voting

Socialmedia

Mobilemessaging

Backup & restore

@akshatvig

Page 4: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

replication

@akshatvig

Page 5: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

backups

@akshatvig

Page 6: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

views

@akshatvig

Page 7: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Reactors – the new triggers

@akshatvig

Page 8: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Logs

@akshatvig

Page 9: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Op: PUT

John

Tokyo

Op: UPDATE

John

Pluto

Op: UPDATE

John

Mars

Logs and databases

Name Destination

TokyoMars

@akshatvig

Page 10: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

is truth

construct state

@akshatvig

Page 11: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Logs are powerful.

Atomicity, consistency, and durability

Replication

Point-in-time restores

Materialized views

Auditing

And much more...@akshatvig

Page 12: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

What is DynamoDB Streams?

It is a stream of updates

Scales with your table

DynamoDB StreamsDynamoDB

@akshatvig

Page 13: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Social network application

Comments and notifications

@akshatvig

Page 14: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Take 1

Comments Table

@akshatvig

Page 15: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

post

Take 2

Comments TableProcessor

@akshatvig

Page 16: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Take 3

Comments Table

@akshatvig

Page 17: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Global users

@akshatvig

Page 18: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Cross region library

@akshatvig

Page 19: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

groups

@akshatvig

Page 20: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Cross region replication

post

@akshatvig

Page 21: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Cross region library

Horizontal scaling : Workers

Load balancing

@akshatvig

Page 22: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Cross region library

Fault tolerant

Checkpointing

@akshatvig

Page 23: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Cross region replication

Shard 1

Partition

1Shard 2

Shard 3

Partition

2

KCL

Worker

KCL

Worker

KCL

Worker

@akshatvig

Page 24: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Cross region replication

Partition 1

Partition 2

Partition 3

Partition 4

Partition 5

Shard 1

Shard 2

Shard 3

Shard 4

KCL

Worker

KCL

Worker

KCL

Worker

KCL

Worker

@akshatvig

Page 25: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Cross region replication library

@akshatvig

Page 26: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Cross region replication library

@akshatvig

Page 27: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

DynamoDB Streams and

DynamoDB Connectors simplify

cross region replication!

@akshatvig

Page 28: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Materialized view

Extending DynamoDB capabilities

@akshatvig

Page 29: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Cross region

StreamsStreams

Cross region

@akshatvig

Page 30: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Data Tier

Amazon DynamoDB

Amazon RDS

Amazon ElastiCache

Amazon S3

Amazon CloudSearch

Amazon Redshift

archiverich search

read replicas

hot reads

analyticscomplex queries& transactions

purpose

@akshatvig

Page 31: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Features of

@akshatvig

Page 32: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

View Type Destination

Old Image – Before update Name = John, Destination = Mars

New Image – After update Name = John, Destination = Pluto

Old and New Images Name = John, Destination = Mars

Name = John, Destination = Pluto

Keys Only Name = John

types

@akshatvig

Page 33: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Asynchronous

i=A

i=A

ack

@akshatvig

Page 34: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Exactly once

i=Ai=Bi=B

i=B

@akshatvig

Page 35: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Strictly ordered records

i=Ai=B

i=C

i=C

@akshatvig

Page 36: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Durability & high availability

High throughput consensus protocol

Replicated across multiple AZs

@akshatvig

Page 37: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Managed

@akshatvig

Page 38: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Elasticity

Adjusts

@akshatvig

Page 39: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Performance

Sub-second latency

@akshatvig

Page 40: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Retention period

Records available for 24 hours

@akshatvig

Page 41: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

DynamoDB Local

Desktop installable

Development & testing

Publicly available - http://bit.ly/1yt1r9q

Now supports DynamoDB Streams @akshatvig

Page 42: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Consuming AWSDynamoDBstreamsAdapterClient adapterClient =

new AWSDynamoDBstreamsAdapterClient(streamsCredentials, .. );..

AmazonDynamoDBClient dynamoDBClient =new AmazonDynamoDBClient(dynamoDBCredentials, ..);

..

KinesisClientLibConfiguration workerConfig =new KinesisClientLibConfiguration (.., streamId,

streamsCredentials, ..)

.withMaxRecords(100)

.withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON);

Worker worker =new Worker(recordProcessorFactory, workerConfig, adapterClient,

dynamoDBClient, ..);

Thread t = new Thread(worker);t.start();

@akshatvig

Page 43: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Consuming AWSDynamoDBstreamsAdapterClient adapterClient =

new AWSDynamoDBstreamsAdapterClient(streamsCredentials, .. );..

AmazonDynamoDBClient dynamoDBClient =new AmazonDynamoDBClient(dynamoDBCredentials, ..);

..

KinesisClientLibConfiguration workerConfig =new KinesisClientLibConfiguration (.., streamId,

streamsCredentials, ..)

.withMaxRecords(100)

.withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON);

Worker worker =new Worker(recordProcessorFactory, workerConfig, adapterClient,

dynamoDBClient, ..);

Thread t = new Thread(worker);t.start();

@akshatvig

Page 44: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Consuming AWSDynamoDBstreamsAdapterClient adapterClient =

new AWSDynamoDBstreamsAdapterClient(streamsCredentials, .. );..

AmazonDynamoDBClient dynamoDBClient =new AmazonDynamoDBClient(dynamoDBCredentials, ..);

..

KinesisClientLibConfiguration workerConfig =new KinesisClientLibConfiguration (.., streamId,

streamsCredentials, ..)

.withMaxRecords(100)

.withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON);

Worker worker =new Worker(recordProcessorFactory, workerConfig, adapterClient,

dynamoDBClient, ..);

Thread t = new Thread(worker);t.start();

@akshatvig

Page 45: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Consuming AWSDynamoDBstreamsAdapterClient adapterClient =

new AWSDynamoDBstreamsAdapterClient(streamsCredentials, .. );..

AmazonDynamoDBClient dynamoDBClient =new AmazonDynamoDBClient(dynamoDBCredentials, ..);

..

KinesisClientLibConfiguration workerConfig =new KinesisClientLibConfiguration (.., streamId,

streamsCredentials, ..)

.withMaxRecords(100)

.withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON);

Worker worker =new Worker(recordProcessorFactory, workerConfig, adapterClient,

dynamoDBClient, ..);

Thread t = new Thread(worker);t.start();

@akshatvig

Page 46: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Processingpublic class StreamsRecordProcessor implements IRecordProcessor {..@Overridepublic void processRecords(List<Record> records,.. ) {

for(Record record : records) {

if (record instanceof RecordAdapter) {

Record ddbStreamRecord = ((RecordAdapter)record).getInternalObject();

switch(ddbStreamRecord.getEventName()) {case "INSERT" : case "MODIFY" :

DemoHelper.putItem(dynamoDBClient, tableName,

ddbStreamRecord.getDynamodb().getNewImage());break;

... @akshatvig

Page 47: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Processingpublic class StreamsRecordProcessor implements IRecordProcessor {..@Overridepublic void processRecords(List<Record> records,.. ) {

for(Record record : records) {

if (record instanceof RecordAdapter) {

Record ddbStreamRecord = ((RecordAdapter)record).getInternalObject();

switch(ddbStreamRecord.getEventName()) {case "INSERT" : case "MODIFY" :

DemoHelper.putItem(dynamoDBClient, tableName,

ddbStreamRecord.getDynamodb().getNewImage());break;

... @akshatvig

Page 48: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Processingpublic class StreamsRecordProcessor implements IRecordProcessor {..@Overridepublic void processRecords(List<Record> records,.. ) {

for(Record record : records) {

if (record instanceof RecordAdapter) {

Record ddbStreamRecord = ((RecordAdapter)record).getInternalObject();

switch(ddbStreamRecord.getEventName()) {case "INSERT" : case "MODIFY" :

DemoHelper.putItem(dynamoDBClient, tableName,ddbStreamRecord.getDynamodb().getNewImage());

break;...

@akshatvig

Page 49: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Processingpublic class StreamsRecordProcessor implements IRecordProcessor {..@Overridepublic void processRecords(List<Record> records,.. ) {

for(Record record : records) {

if (record instanceof RecordAdapter) {

Record ddbStreamRecord = ((RecordAdapter)record).getInternalObject();

switch(ddbStreamRecord.getEventName()) {case "INSERT" : case "MODIFY" :

DemoHelper.putItem(dynamoDBClient, tableName,ddbStreamRecord.getDynamodb().getNewImage());

break;...

@akshatvig

Page 50: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Cross region replication@akshatvig

Page 51: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014
Page 52: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

DynamoDB Reactors

Trigger Lambda functions

Example – Validate address, send notifications

Console support

@akshatvig

Page 53: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Preview available

Temporary endpoints: N. Virginia & Ireland

Available until global launch

Register for preview at http://amzn.to/11dh9M0

@akshatvig

Page 54: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Preview console

TBD: Console snapshot

Page 55: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

@akshatvig

Page 56: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Landing page http://amzn.to/1xodVzQ

@akshatvig

Page 57: (SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Please give us your feedback on this session.

Complete session evaluations and earn re:Invent swag.

http://bit.ly/awsevals