dynamodb at hasoffers

DynamoDB at HasOffers

• Attribution for web & mobile marketing• Tons of data• Ad-hoc analysis• Near real-time

18 months in production

207 billion items

87 TB of data

DynamoDB at HasOffers

Tracking sessions

Deduplication

Attribution

Use Cases

Tracking Sessions

Session Data

Use Case: Tracking Sessions

671c6a9a-bed0-4cec-a465-613cca8c9832

• Impression• Click

Use Case: Tracking Sessions

• Ideal for NoSQL

• KV lookups

• Scalability

• Availability

Deduplication

Use Case: Event Deduplication671c6a9a-bed0-4cec-a465-613cca8c9832

ebe0efef-aa71-471c-b3f8-7b66b458bd79

ebe0efef-aa71-471c-b3f8-7b66b458bd79

d149e52c-d2b3-49bc-b6af-f304b5fab57f

DEDUPE!

(Idempotency)

Use Case: Event Deduplication

Hash uuidRange <none>

Write capacity = X, Read capacity = 1

Conditional puts

Use Case: Event Deduplication

DEDUPE!

def is_duplicate( dynamo_tbl, uuid ):

row_data = {‘host’: socket.gethostname(), ‘created’: int(time.time())}

row = dynamo_tbl.new_item( hash_key=uuid, attrs=row_data )

try: row.put( expected_value={‘host’: False} )

except DynamoDBConditionalCheckFailedError: return True

# not a duplicate return False

Use Case: Event Deduplication

DEDUPE!

• Real-time dedupe• Conditional put• Writes only; no read• Extra debug info

{uuid: d149e52c-d2b3-49bc-b6af-f304b5fab57f, host: server03.hasoffers.com, created: 1398884990}

Why NoSQL SaaS?

• Upsize and downsize trivial in DynamoDB

• Infrequent on own cluster = limits operational experience

Cluster Resizing

Cluster Software Updates

0.7 1.0

• Production depends on NoSQL solution

being available.

• Software version updates somewhere

between nail-biting and terrifying

Dynamo Operations

Per table R/W capacity

Capacity units

Exceed = throttling

DynamoDB Capacity

DynamoDB Throttling

CloudWatch = 5 minutes

DynamoDB = 1 second

DynamoDB Throttling

• Double capacity on table

• Root cause from log analysis– capacity usage

– key hot-spotting

• Fix root cause

• Reduce capacity on table

Closing Thoughts

• Compelling offering

• Simple operations

• Avoid hammer syndrome

• Very happy

Closing Thoughts

Thank you!

niek@hasoffers.com