Metrics Insight with Riemann!Ali Asad Lotia | ali.asad.lotia@gmail.com | @aalotia !!

Clarification!

• Riemann usage at Beamly is WIP

Platform visibility• Dashboards – Plots

– Summary values (min, max, mean)

• Alerts

Dashboard Limitations• Dashboard overload

• Hard to be proactive

• Mostly reactive

Alerting challenges• Service interdependencies

• Multiple services + multiple metrics

• Alerting before outage

If only…• A tool to gain insight from the constant

stream of metrics

• Analyse incoming metrics in near real time

Well, there is…• http://riemann.io/

• Started by Aphyr (Kyle Kingsbury)

• Named for Bernhard Riemann – http://en.wikipedia.org/wiki/Bernhard_Riemann

Riemann

http://riemann.io/concepts.html

Requirements• Java 7 or greater

• Sufficient CPU and RAM

• Curiosity

• Patience

Concepts• Events

• Streams

• Index

• Event manipulation (via functions)

EventStruct/Dictionary/Map

{:host "foo"! :service "My test service"! :state "good"! :time 1414687710! :description "A test that I am running"! :tags ["foo", "bar"]! :metric 1.0! :ttl 900}

Stream

http://riemann.io/concepts.html

Configuration• Written in clojure – Pro: Power of a programming language

– Con: Power of a programming language

• Updatable on the fly

Beamly metrics flow

Sensu

Server

Server

Server

Server

Graphite

Riemann

Accepting graphite metrics!(let [host "0.0.0.0"! port 2003]! (graphite-server {:host host! :port port})! (tcp-server {:host host})! (udp-server {:host host! :max-size 65000})! (repl-server {:host "127.0.0.1"})! (ws-server {:host host}))!

Transforming graphite metrics!!(defn process-graphite-metric-name! "Split the graphite metric name and set the event map values! accordingly. Currently a somewhat naive function since it assumes! the format of the graphite metric key."! [{:keys [service metric time]}]! (let [[component source environment region aws-host metric-details]! (cs/split service #"\." 6)! host-ip! (extract-host-ip-from-aws-hostname aws-host)! new-service! (cs/join "." (vector component source metric-details))]! {:service new-service! :environment environment! :region region! :host host-ip! :component component! :metric (or metric 0)! :time time! :state "ok"! :description service! :tags ["from-sensu" component]}))!

Indexing select events(let [index (default :ttl 900 (index))]! (streams! (where (= (:component event) "comp-a")! #(info (format "received event from component %s" (:component %)) %)! index)! (where (= (:component event) "comp-b")! (with :ttl 300 index! #(info (format "received event from component %s" (:component %)) %)))! (where (= (:component event) "comp-c")! (with :ttl 300 index! #(info (format "received event from component %s" (:component %)) %)))! (where (and (not (nil? service)) (service #"\.sys\.swapused$") (> metric 0))! (with :state "swapping"! #(info (format "swap used on component %s" (:component %)) %)! index))))!

Output• Email

• Pagerduty

• Graphite, InfluxDB

• custom plugin

Riemann Dashboardhttp://riemann.io/dashboard.html !Can query riemann index

Gotchas• Stateful

• Dashboard docs sparse

• Scaling possible by cascading to other Riemann instances

Resources• Website: http://riemann.io

• IRC: #riemann on freenode

• Riemann Users Google Group https://groups.google.com/forum/#!forum/riemann-users

• Guardian riemann-config repo on Github https://github.com/guardian/riemann-config

Thank you• Questions?

• @aalotia

• ali.asad.lotia@gmail.com

LONDON !Drury House

34-43 Russell Street London

WC2B 5HA

NEW YORK CITY !84 Wooster Street

Suite 703 New York NY 10012

SYDNEY !22-36 Mountain Street

Suite 1.10 Ultimo

NSW 2007

© Beamly 2014. All Rights Reserved.