prometheus data explore your in grafana - overview · in grafana david kaltschmidt @davkals promcon...

33
Explore your Prometheus data in Grafana David Kaltschmidt @davkals PromCon 2018

Upload: others

Post on 11-Aug-2021

7 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Explore your Prometheus datain Grafana

David Kaltschmidt@davkals

PromCon 2018

Page 2: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

I’m not Carl

Carl is having a

great time on parental leave

Page 3: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

I’m David

All things UX at Grafana Labs

If you click and are stuck, reach out to [email protected]

Twitter: @davkals

Page 4: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Grafana

From

Dashboarding solution

ToObservability platform

Page 5: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Instant query queries only the last datapoint

best displayed in table panel

Page 6: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Table row merge on labels

Page 7: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Step alignment

“Jumpy” rate graphs

Align start/end parameters to step

Page 8: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Heatmappanel

released in 5.1

Query: rate(foo_metric_bucket[10m])Legend format: {{le}}Format as: Heatmap

Page 9: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Variable expansion in template variables

Summer intern project (Tobias, tak!!)

New built-in variable: $__range

Dashboard-range-based variables

Page 10: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Template variable expansion

Page 11: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Use range-based template variables in queries

Page 12: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Provisioning API

Define data sources and

dashboards in files

Auto-reload on change

Allows version control of files

http://docs.grafana.org/administration/provisioning/

Page 13: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Grafana is now fully CI’ed

Test new features that are in master:

docker run -d --name=grafana -p 3000:3000 grafana/grafana:latest

https://hub.docker.com/r/grafana/grafana/

Page 14: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Prometheus adoption

Page 15: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

2.8K

2016

36K PromCon

Page 16: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

2.8K 16K

2016 2017

36K 92KPromCon

Page 17: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

2.8K 16K ?K

2016 2017 2018

36K 92K 186KPromCon

Page 18: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

2.8K 16K 54K

2016 2017 2018

36K 92K 186KPromCon

Page 19: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

New: Explore UI (Alpha)

Page 20: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Motivation

Dashboard UI is made for

building dashboards, not query iteration.

“I just want to quickly…”

Page 21: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Motivation

once panel is found, difficult to

interact with

overwhelming style and display options

Page 22: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Motivation

Incident response currently happens elsewhere!

User story:As an SRE, I want to investigate a certain

aspect or incident of my infrastructure. I’m

already looking at the right dashboard panel

and have formed a hypothesis. From here I

want to quickly run variations of the panel’s

query to verify my hypothesis. The panel I

started from should stay unaffected.

Page 23: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Explore UI wireframes

rate(http_requests_total[5m])

GRAPH TABLE BOTH Last 1 hour, Refresh: 10s

RUN

1 - rate(http_requests_total[5m])

. . .

rate(http_requests_total[5m])

1 - rate(http_requests_total[5m])

4.2s

3.2s

rate(http_requests_total[5m])

GRAPH TABLE BOTH Last 1 hour, Refresh: 10s

RUN

1 - rate(http_requests_total[5m])

. . .

rate(http_requests_total[5m])

1 - rate(http_requests_total[5m])

4.2s

3.2s

First tab Second tab 3rd tab My tab ╳

Page 24: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Demo: http://localhost:3333/explore

Page 25: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Enable Explore UI

Behind feature flag. To enable, edit Grafana config ini file

[explore]

enabled = true

Set up a datasource that supports Explore, e.g., Prometheus.

Will be released in 6.0

Page 26: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

What we’re working on

Page 27: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Explore UI needs to be refined

still behind feature flag, feedback welcome: @davkals or [email protected]

Prometheus metric metadata from HELP line in the exposition

support for completion forfoo_metric / on(|) group_left(|) bar_metric

design doc for explore components for other datasources

bring query field over to panel editor

Page 28: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

MultiStat panel https://github.com/grafana/grafana/pull/12620

Page 29: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Dashboard management

Improve Grafana’s integration with Mixins (see Tom Wilkie’s talk)

Git integration to save changes back

Defaults for panel configs

Page 30: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

One more thing...

Page 31: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Logging is coming to Grafana

Page 32: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Tak for listening

UX feedback [email protected]@davkals

Page 33: Prometheus data Explore your in Grafana - Overview · in Grafana David Kaltschmidt @davkals PromCon 2018. I’m not Carl Carl is having a great time on parental leave. I’m David

Tak for listening

UX feedback [email protected]@davkals