computer monitoring with prometheus & grafana€¦ · computer monitoring with prometheus &...
TRANSCRIPT
![Page 1: Computer Monitoring with Prometheus & Grafana€¦ · Computer Monitoring with Prometheus & Grafana Brad Davis, EnikoeBihari, Eliana Purcell HPC Academy 2018 August 14, 2018. 2 LLNL-PRES-xxxxxx](https://reader037.vdocuments.mx/reader037/viewer/2022100223/5f07426b7e708231d41c19e9/html5/thumbnails/1.jpg)
LLNL-PRES-XXXXXX
This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC
Computer Monitoring with Prometheus & Grafana
Brad Davis, Enikoe Bihari, Eliana PurcellHPC Academy 2018
August 14, 2018
![Page 2: Computer Monitoring with Prometheus & Grafana€¦ · Computer Monitoring with Prometheus & Grafana Brad Davis, EnikoeBihari, Eliana Purcell HPC Academy 2018 August 14, 2018. 2 LLNL-PRES-xxxxxx](https://reader037.vdocuments.mx/reader037/viewer/2022100223/5f07426b7e708231d41c19e9/html5/thumbnails/2.jpg)
2LLNL-PRES-xxxxxx
Prometheus
Computer monitoring database software that stores data received from various exporters
![Page 3: Computer Monitoring with Prometheus & Grafana€¦ · Computer Monitoring with Prometheus & Grafana Brad Davis, EnikoeBihari, Eliana Purcell HPC Academy 2018 August 14, 2018. 2 LLNL-PRES-xxxxxx](https://reader037.vdocuments.mx/reader037/viewer/2022100223/5f07426b7e708231d41c19e9/html5/thumbnails/3.jpg)
3LLNL-PRES-xxxxxx
Prometheus Configuration
![Page 4: Computer Monitoring with Prometheus & Grafana€¦ · Computer Monitoring with Prometheus & Grafana Brad Davis, EnikoeBihari, Eliana Purcell HPC Academy 2018 August 14, 2018. 2 LLNL-PRES-xxxxxx](https://reader037.vdocuments.mx/reader037/viewer/2022100223/5f07426b7e708231d41c19e9/html5/thumbnails/4.jpg)
4LLNL-PRES-xxxxxx
Exporters
Scrape various data from the nodes— Node Exporter— IPMI Exporter— Blackbox
Exporter
![Page 5: Computer Monitoring with Prometheus & Grafana€¦ · Computer Monitoring with Prometheus & Grafana Brad Davis, EnikoeBihari, Eliana Purcell HPC Academy 2018 August 14, 2018. 2 LLNL-PRES-xxxxxx](https://reader037.vdocuments.mx/reader037/viewer/2022100223/5f07426b7e708231d41c19e9/html5/thumbnails/5.jpg)
5LLNL-PRES-xxxxxx
Grafana
Grafana is a versatile UI
Displays the data from Prometheus on dashboards
![Page 6: Computer Monitoring with Prometheus & Grafana€¦ · Computer Monitoring with Prometheus & Grafana Brad Davis, EnikoeBihari, Eliana Purcell HPC Academy 2018 August 14, 2018. 2 LLNL-PRES-xxxxxx](https://reader037.vdocuments.mx/reader037/viewer/2022100223/5f07426b7e708231d41c19e9/html5/thumbnails/6.jpg)
6LLNL-PRES-xxxxxx
Main Dashboard
![Page 7: Computer Monitoring with Prometheus & Grafana€¦ · Computer Monitoring with Prometheus & Grafana Brad Davis, EnikoeBihari, Eliana Purcell HPC Academy 2018 August 14, 2018. 2 LLNL-PRES-xxxxxx](https://reader037.vdocuments.mx/reader037/viewer/2022100223/5f07426b7e708231d41c19e9/html5/thumbnails/7.jpg)
7LLNL-PRES-xxxxxx
Queries
![Page 8: Computer Monitoring with Prometheus & Grafana€¦ · Computer Monitoring with Prometheus & Grafana Brad Davis, EnikoeBihari, Eliana Purcell HPC Academy 2018 August 14, 2018. 2 LLNL-PRES-xxxxxx](https://reader037.vdocuments.mx/reader037/viewer/2022100223/5f07426b7e708231d41c19e9/html5/thumbnails/8.jpg)
8LLNL-PRES-xxxxxx
Drilldowns
![Page 9: Computer Monitoring with Prometheus & Grafana€¦ · Computer Monitoring with Prometheus & Grafana Brad Davis, EnikoeBihari, Eliana Purcell HPC Academy 2018 August 14, 2018. 2 LLNL-PRES-xxxxxx](https://reader037.vdocuments.mx/reader037/viewer/2022100223/5f07426b7e708231d41c19e9/html5/thumbnails/9.jpg)
9LLNL-PRES-xxxxxx
Alertmanager Configuration
Alertmanager groups and sends alerts when certain metrics reach a threshold
Sends alerts over email and Slack to different groups depending on the type of alert
![Page 10: Computer Monitoring with Prometheus & Grafana€¦ · Computer Monitoring with Prometheus & Grafana Brad Davis, EnikoeBihari, Eliana Purcell HPC Academy 2018 August 14, 2018. 2 LLNL-PRES-xxxxxx](https://reader037.vdocuments.mx/reader037/viewer/2022100223/5f07426b7e708231d41c19e9/html5/thumbnails/10.jpg)
10LLNL-PRES-xxxxxx
Alert Examples
![Page 11: Computer Monitoring with Prometheus & Grafana€¦ · Computer Monitoring with Prometheus & Grafana Brad Davis, EnikoeBihari, Eliana Purcell HPC Academy 2018 August 14, 2018. 2 LLNL-PRES-xxxxxx](https://reader037.vdocuments.mx/reader037/viewer/2022100223/5f07426b7e708231d41c19e9/html5/thumbnails/11.jpg)
11LLNL-PRES-xxxxxx
Scalability With Ansible
Automated the installation and configuration of Prometheus, Grafana, etc. with Ansible
Allows for scalability for future use
![Page 12: Computer Monitoring with Prometheus & Grafana€¦ · Computer Monitoring with Prometheus & Grafana Brad Davis, EnikoeBihari, Eliana Purcell HPC Academy 2018 August 14, 2018. 2 LLNL-PRES-xxxxxx](https://reader037.vdocuments.mx/reader037/viewer/2022100223/5f07426b7e708231d41c19e9/html5/thumbnails/12.jpg)
12LLNL-PRES-xxxxxx
Questions?
![Page 13: Computer Monitoring with Prometheus & Grafana€¦ · Computer Monitoring with Prometheus & Grafana Brad Davis, EnikoeBihari, Eliana Purcell HPC Academy 2018 August 14, 2018. 2 LLNL-PRES-xxxxxx](https://reader037.vdocuments.mx/reader037/viewer/2022100223/5f07426b7e708231d41c19e9/html5/thumbnails/13.jpg)
DisclaimerThis document was prepared as an account of work sponsored by an agency of the United States government. Neither the United States government nor Lawrence Livermore National Security, LLC, nor any of their employees makes any warranty, expressed or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States government or Lawrence Livermore National Security, LLC. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States government or Lawrence Livermore National Security, LLC, and shall not be used for advertising or product endorsement purposes.