client side monitoring with prometheus
TRANSCRIPT
Client-side Prometheus Metrics@tom_wilkie, CNCFCon Berlin April 2017
The Problem
A rich-client web app
Weave Cloud
~28k LOC
What we had:
What we wanted:
Prometheus Aggregating Push Gateway
JS Prometheus client library
Rich Client Web App
https://github.com/weaveworks/prom-aggregation-gateway
https://github.com/weaveworks/promjs
weaveworks/promjs
Failure modes
1. Failed to load JS quickly enough, or was invalid
2. JS hits a bug when rendering a view
3. High client <-> sever latency
Case 1: Cannot load JS assets
^^ special handling code directly in the index.html ^^
Case 1: Cannot load JS assets (continued)
timer cancelled after ~5s
Case 2: Error rendering page
a higher order react class is then used to wrap the views
Case 3: High client <-> sever latency
Case 3: High client <-> sever latency (II)
• “Counters” are actually more like gauges, with partial aggregation on the client (I’ll explain later)
• Client pushes to server every 15s
• Opensource: https://github.com/weaveworks/promjs
• All credit to Jordan Pellizzari @ Weaveworks
weaveworks/prom-aggregation-gateway
The Pushgateway never forgets series pushed to it and will expose them to Prometheus forever…
The latter point is especially relevant when multiple instances of a job differentiate their metrics in the
Pushgateway via an instance label or similar.
https://prometheus.io/docs/practices/pushing/
Prometheus
Aggregating Push Gateway
Aggregated in memory
/metrics
weaveworks/prom-aggregation-gateway
~300 LOC
+
Bunch of similar projects• Webdriver Exporter by Matt Bostock from Cloudflare
• Looks awesome, but we didn’t see this when we started our project… • https://github.com/mattbostock/webdriver_exporter
• Prometheus User Metrics by Riley Eynon-Lynch from Peardesk • New project, similar aims • https://github.com/peardeck/prometheus-user-metrics
• Torch by Outbrain • https://github.com/outbrain/torch
• Prometheus at JustWatch by Dominik Schulz from JustWatch • Describes the same problem, couldn’t find any code through • https://speakerdeck.com/dominikschulz/prometheus-at-justwatch
Demo
Thank you!Questions?
