client side monitoring with prometheus

Client-side Prometheus Metrics@tom_wilkie, CNCFCon Berlin April 2017

The Problem

A rich-client web app

Weave Cloud

~28k LOC

What we had:

What we wanted:

Prometheus Aggregating Push Gateway

JS Prometheus client library

Rich Client Web App

https://github.com/weaveworks/prom-aggregation-gateway

https://github.com/weaveworks/promjs

weaveworks/promjs

Failure modes

1. Failed to load JS quickly enough, or was invalid

2. JS hits a bug when rendering a view

3. High client <-> sever latency

Case 1: Cannot load JS assets

^^ special handling code directly in the index.html ^^

Case 1: Cannot load JS assets (continued)

timer cancelled after ~5s

Case 2: Error rendering page

a higher order react class is then used to wrap the views

Case 3: High client <-> sever latency

Case 3: High client <-> sever latency (II)

• “Counters” are actually more like gauges, with partial aggregation on the client (I’ll explain later)

• Client pushes to server every 15s

• Opensource: https://github.com/weaveworks/promjs

• All credit to Jordan Pellizzari @ Weaveworks

weaveworks/prom-aggregation-gateway

The Pushgateway never forgets series pushed to it and will expose them to Prometheus forever…

The latter point is especially relevant when multiple instances of a job differentiate their metrics in the

Pushgateway via an instance label or similar.

https://prometheus.io/docs/practices/pushing/

Prometheus

Aggregating Push Gateway

Aggregated in memory

/metrics

weaveworks/prom-aggregation-gateway

~300 LOC

+

Bunch of similar projects• Webdriver Exporter by Matt Bostock from Cloudflare

• Looks awesome, but we didn’t see this when we started our project… • https://github.com/mattbostock/webdriver_exporter

• Prometheus User Metrics by Riley Eynon-Lynch from Peardesk • New project, similar aims • https://github.com/peardeck/prometheus-user-metrics

• Torch by Outbrain • https://github.com/outbrain/torch

• Prometheus at JustWatch by Dominik Schulz from JustWatch • Describes the same problem, couldn’t find any code through • https://speakerdeck.com/dominikschulz/prometheus-at-justwatch

Demo

Thank you!Questions?

We’re hiring!London BerlinSan Francisco

jobs@weave.works