observable node.js applications - enterprisejs

Observable Node.js Applications at Scale

Yunong Xiao, Senior Node.js Engineer, UI Platform, Netflix

@yunongx, yunong@netflix.com

July 2015EnterpriseJS

Node.js @ Netflix

Production is War

What’s Wrong?

Increased Errors Increased Latency

Memory Leak

Software is Complex

Drunk Man Anti-Method

DATA

“It is a capital mistake to theorize before one has DATA. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.”

Sherlock Holmes

-A Scandal in Bohemia

Observability From the Ground Up

Hard Harder Insane

Development ProductionProduction w/

Direct Customer Impact

Dev === Prod

node-bunyan

https://github.com/trentm/node-bunyan

node-bunyan

Streaming JSON logging format

Streaming JSON

Unix Philosophy

grep(1), cut(1), wc(1), awk(1), perl(1), jq(1), json(1), daggr(1)

Query Logs in Real Time

$ cat http.log | grep audit| bunyan -0 --strict -c 'this.res.statusCode === 200' | json -ga

‘req.timers["fetch-api"]' req.url | awk '{print $1 " " substr($2,0,7)}' | daggr -k 2 -v 1 quantize

Illegible!

bunyan(1) CLI

{ "name": "demo", "hostname": "Yunongs-MacBook-Pro.local", "pid": 35919, "level": 30, "authInfo": { "username": "yunong", "password": "xxxxxx" }, "msg": "got request", "time": "2015-07-07T21:46:01.317Z", "v": 0 }

Features

• Lightweight API

• Log levels: trace, debug, info, warn, error, fatal

• Extensible Streams interface.

• Custom object rendering with serializers.

HTTP Request

• Make API requests

• Persist state to file system

• Query database

• Update caches

Async HTTP Request

• Make API requests

• Persist state to file system

• Query database

• Update caches

node-vasync

• Observable Async workflow

• https://github.com/davepacheco/node-vasync

async f() state

failed f()

successful f()

pending f()

finished f() # of errors

Example

cb() not invoked

Results

function three pending

How Do I See this in Prod?

• Logs

• Core Dumps

• REPL

• HTTP API

Observable Node.js HTTP

• Observability

• Metrics

• Bunyan integration

restify http://restify.com

http://github.com/restify/node-restify

Audit Logs

Audit Logsstatus code request ID (UUID)URL

request headers

response headers

individual handler timers

req latency

Too many logs!

Request Capture Stream

• Captures all log statements at trace level in memory.

• Dump all logs for a particular request only on error.

Scoped Child Loggers

Logging

• Native Bunyan integration

• Request capture stream

• Audit logs

• Scoped child loggers

What about Data Processing?

Examples

Find all logs for a specific request.

$ grep $uuid restify.log

Examples

Count the # of non-200 responses.

$ grep restify-audit restify.log | bunyan -c ‘this.res.statusCode !== 200` -0 | wc -l

Examples

Show all requests that took longer than 200ms.

$ grep restify-audit restify.log | bunyan -c ‘this.latency > 200` -0 | wc -l

Advanced ExampleRequest latency distribution by URL

restify + Bunyan• Streaming JSON.

• Processing using Unix tools is easy.

• Helpful tools:

• Unix: cut(1), wc(1), grep(1), awk(1), perl(1), json(1), jq(1)…

• JSON: https://github.com/trentm/json (npm install -g json)

• daggr: https://github.com/joyent/daggr (npm install -g daggr)

Increased Errors Increased Latency

restifyCore Contributors at Netflix

@stinkydofu, @mjr578, @yunongx

restifyInterested? Contribute to restify!

restify.com github.com/restify/node-restify

distributed processing

• elasticsearch • spark • hive

bunyanrestify

audit logs

req id req headers

req latency

handler latenciesURL

res codeErrors

res headers

Processing w/ Unix tools

• grep(1) • awk(1) • cut(1) • wc(1) • sort(1) • json(1) • bunyan(1) • …

req scoped logs application specific

context

req capture stream dumps all logs on error

DTrace*req latencyhandler

latencies

* Where available

Production Observability

Observable Toolkit• vasync: https://github.com/davepacheco/node-vasync

• Observable async operations

• bunyan: https://github.com/trentm/node-bunyan

• Streaming JSON logs

• restify: restify.com

• Observable REST applications

• Unix Tools

• Easily process JSON logs

One more thing…

DTrace

Thanks

• Questions? We’re hiring!

• yunong@netflix.com

• @yunongx

• http://restify.com

July 2015EnterprisJS