Instrumenting your Application Stack in a

Dynamically Scaling Environment

Mike Fiedler, Datadog

November 15, 2013

The Past

• Server uptime

• Intertwined applications

• SPOFs

The Present

• SOA

• Scale or Fail

• Data-driven decisions

Modern application stack

Failure is inevitable

Design to tolerate failure

Things that produce metrics

business/service Services

• application

• OS

• instances

API-based

• Amazon S3, S*S

• Amazon RDS, Amazon Redshift

• etc.

Users

• RUM

The Desire

CPU comparison (hypervisor)

CPU comparison (Agent)

My Stack

• HAProxy

• Apache HTTP

• Bottle.py – Custom python webapp

• MongoDB

• Amazon EC2

• Chef

• Datadog

• Users

Tracking database operations

Moving up the stack…

Even higher…

Using metadata for more flavor

Replace an instance, keep the metrics

Enter…

Aggregating numbers

Comparing numbers

My App

from bottle import route

import pymongo

import json

db = pymongo.Connection('mongodb://localhost').test

@route('/insert/:name')

def insert(name):

doc = {'name': name}

db.words.update(doc, {"$inc":{"count": 1}}, upsert=True)

return json.dumps(doc, default=default)

Add 2 lines and you win!

from bottle import route

import pymongo

import json

from statsd import statsd

db = pymongo.Connection('mongodb://localhost').test

@route('/insert/:name')

@statsd.increment('fullstack.insert')

def insert(name):

doc = {'name': name}

db.words.update(doc, {"$inc":{"count": 1}}, upsert=True)

return json.dumps(doc, default=default)

Profit!

Correlation

Don’t be caught unaware

Question Time!

We are sincerely eager to hear

your feedback on this

presentation and on re:Invent.

Please fill out an evaluation form

when you have a chance.