practical monitoring with prometheus and grafana · practical monitoring with prometheus and...
TRANSCRIPT
Abstract
Prometheus is an open source monitoring and alerting toolkit.
In this session we'll review the Prometheus architecture andvarious tools and walk you through erecting an endtoendmonitoring and alerting infrastructure with the Prometheus stack.
In addition to Prometheus, we'll use:
Consul for automatic service discovery
Grafana for data visualisation
Practical monitoring with Prometheus and Grafana | OSCON 2018
0
Session OverviewThe session will cover the following topics:
The Prometheus daemon and metric exporters
Common metric exporters (MySQL, memcached, Apache, andNginx)
Writing custom exporters to instrument your web app's metrics
Leveraging Consul for auto detection and configuration ofservices
Deploying and configuring AlertManager
Deploying Grafana and generating different graphs and reports
Practical monitoring with Prometheus and Grafana | OSCON 2018
1
Prometheus main featuresA multidimensional data model with time series data identifiedby metric name and key/value pairs
A flexible query language to leverage this dimensionality
No reliance on distributed storage; single server nodes areautonomous
Time series collection happens via a pull model over HTTP
Targets are discovered via service discovery or staticconfiguration
Pushing time series is supported via an intermediary gateway
Multiple modes of graphing and dashboarding support
Practical monitoring with Prometheus and Grafana | OSCON 2018
2
Prometheus main componentsThe Prometheus server which scrapes and stores time seriesdata
Specialpurpose exporters for services (HAProxy, MySQL,memcached, Apache, etc)
Client libraries for instrumenting application code
The Alertmanager
A push gateway for supporting shortlived jobs (optional)
Practical monitoring with Prometheus and Grafana | OSCON 2018
3
Prometheus data model
Prometheus fundamentally stores all data as time series: streamsof timestamped values belonging to the same metric and the sameset of labeled dimensions.
Besides stored time series, Prometheus may generate temporaryderived time series as the result of queries.
Practical monitoring with Prometheus and Grafana | OSCON 2018
4
Architecture overview
Practical monitoring with Prometheus and Grafana | OSCON 2018
5
Prometheus monitoring
In Prometheus terms, the main monitoring service is referred to asthe Prometheus Server and the services Prometheus monitors arecalled targets. A target can be a host, a network equipment or a specific service.
Typically, the Prometheus server/daemon collects metrics fromtargets by making HTTP[s] requests to the Prometheus exporters.In Prometheus terms, that process is called scarping.
Practical monitoring with Prometheus and Grafana | OSCON 2018
6
Prometheus exporters
An Exporter is a piece of software that fetches metrics from a givensystem and exports them in a format that the Prometheus servercan understand.
There are a number of libraries that can be used to write customexporters but there are also many existing FOSS exporters, writtenand maintained by the Prometheus team, third party vendors andcommunity members.
If you need to monitor a popular FOSS system, chances are you'llfind an exporter has already been written to get the job done.
Practical monitoring with Prometheus and Grafana | OSCON 2018
7
Prometheus monitoring your appIn order to monitor your application, you'll need to write code thatretrieves the desired metrics and exports it in a format Prometheuscan interpret.Prometheus offers several official client libraries that can make thetask easier:
Go
Java or Scala
Python
Ruby
Additional third party clients are also available. For a full list, see:https://prometheus.io/docs/instrumenting/clientlibs/
Practical monitoring with Prometheus and Grafana | OSCON 2018
8
Consul service auto discoveryService discovery uses a registry to keep a realtime list ofservices, their location, and their health.The registry can then be used to discover the location of upstreamservices and probe/connect to them directly.
This allows you to scale up/down and gracefully handle failurepoints.
In this demo, we'll see how Consul can be used in correlation withPrometheus so that targets are automatically discovered andscraped without having to modify Prometheus' configuration.
Practical monitoring with Prometheus and Grafana | OSCON 2018
9
Prometheus AlertManager
The Promtheus Alertmanager handles alerts sent by clientapplications such as the Prometheus server.
It takes care of deduplicating, grouping, and routing them to thecorrect receiver integration such as email, PagerDuty, orOpsGenie.
AlertManager is also capable of silencing and inhibition of alerts.In this context, inhibition means suppressing notifications forcertain alerts if certain other alerts are already firing.
Practical monitoring with Prometheus and Grafana | OSCON 2018
10
Grafana reporting and datavisualisation
Grafana is an open source platform for analytics and monitoring.
It allows you to query, visualise, alert on and understand yourmetrics.Grafana can connect to many different data sources including,though not limited to: MySQL, ElasticSearch and of course,Prometheus:)
In this session, we'll learn how to create, explore, and sharedashboards.
Practical monitoring with Prometheus and Grafana | OSCON 2018
11
Thank you && Questions
Appendix Useful ResourcesPrometheusAlertManagerConsulGrafanaPrometheus exportersPrometheus client libs
Practical monitoring with Prometheus and Grafana | OSCON 2018
