better monitoring for python: inclusive monitoring with prometheus (pycon ireland 2015, lightning...

Better Monitoring for PythonInclusive Monitoring with Prometheus

Monitoring is often a Problem, not a SolutionThemes common among companies I talk to:

● Monitoring tools are limited, both technically and conceptually● Tools don’t scale well and are unwieldy to manage● Operational practices don’t align with the business

Result: Engineers continuously woken up for non-issues, get fatigued

Fundamental Challenge is Limited Visibility

PrometheusInspired by Google’s Borgmon monitoring system.

Started in 2012 by ex-Googlers working in Soundcloud as an open source project.

Mainly written in Go. Publically launched in early 2015.

100+ companies using it including Digital Ocean, GoPro, Apple, Red Hat and Google.

Services have Internals

Monitor the Internals

Monitor as a Service, not as Machines

Inclusive MonitoringDon’t monitor just at the edges:

● Instrument client libraries● Instrument server libraries (e.g. HTTP/RPC)● Instrument business logic

Library authors get information about usage.

Application developers get monitoring of common components for free.

Dashboards and alerting can be provided out of the box, customised for your organisation!

Let’s Talk Codepip install prometheus_client

from prometheus_client import Summary, start_http_serverREQUEST_DURATION = Summary('request_duration_seconds', 'Request duration in seconds')

@REQUEST_DURATION.time()def my_handler(request): pass // Your code here

start_http_server(8000)

Multiple Dimensions (No Evil Twins Please)from prometheus_client import CounterREQUESTS = Summary('requests_total', 'Total requests', ['method'])

def my_handler(request): REQUESTS.labels(request.method).inc() pass // Your code here

Exceptional Circumstances In Progressfrom prometheus_client import Counter, GaugeEXCEPTIONS = Counter('exceptions_total', 'Total exceptions')IN_PROGRESS = Gauge('inprogress_requests', 'In progress')

@EXCEPTIONS.count_exceptions()@IN_PROGRESS.track_inprogress()def my_handler(request): pass // Your code here

Getting Data Outfrom prometheus_client import start_http_server

if __name__ == '__main__': start_http_server(8080)

Also possible with Django, Twisted etc.

Oh Noes, My Vendor Lock In!from prometheus_client.bridge.graphite import GraphiteBridgegb = GraphiteBridge(('graphite.your.org', 2003))

gb.start(10.0) # Push every 10 seconds

Exposition to Prometheus doesn’t use a special API, can be hooked into whatever monitoring system you have.

Instrument once, work with everything!

Running PrometheusGrab binary from http://www.robustperception.io/prometheus-nightly-binaries/

Put config in prometheus.yml:

scrape_configs: - job_name: python target_groups: targets: [localhost:8000]Run it:

./prometheus

Dashboards

Prometheus is as Powerful as PythonCan slide, dice, aggregate and do math to produce your alerts

If it’s computable, Prometheus can compute it*

Alert on what matters, not on what your monitoring limit you to!

* Halting, tractability and sanity not guaranteed

Links!

Blog Posts: www.robustperception.io/tag/prometheus

Demo: demo.robustperception.io

Python Client: https://github.com/prometheus/client_python

Project Website: prometheus.io