Elasticsearch, Nginx monitoringwithPrometheusandGrafana

2016/08/18Wataru Yukawa(@wyukawa)PrometheusLondonmeetup

WhoamI?

• DataEngineeratLINE• LINEmakesamessagingapplicationofthesamename,inadditiontootherrelatedservices

• ItisthemostpopularmessagingplatforminJapan

• IlikePremierLeague

Agenda

• browserlogcollectionsystem• monitorElasticsearch/Nginx

browser log collection system• FrontendengineerdevelopsJavaScriptbutit’sdifficulttodebugbecausetherearemanykindofbrowsers/devices.

• SowecreatebrowserlogcollectionsystemlikeGoogleAnalytics by using web beacon

• NginxlogsaresenttoESbyfluentd• Fluentd isanOSSlogcollectorlikelogstash,flume writteninruby.

Whatmiddlewareshouldbemonitored?

• Nginx• Elasticsearch• Fluentd– IwilltalkinPromCon2016

Nginxmonitoring• Usengx_http_stub_status_module

#monitoringlocation/___nginx_status{stub_statuson;allow127.0.0.1;denyall;

}

$curl-shttp://localhost/___nginx_statusActiveconnections:2serveracceptshandledrequests3010423010424242850Reading:0Writing:1Waiting:1

lifecycleofconnection

https://www.datadoghq.com/blog/how-to-monitor-nginx/

Nginxmonitoring• Usenginx_exporter– nginx_exporter usesngx_http_stub_status_moduleresult

#HELPnginx_connections_currentNumberofconnectionscurrentlyprocessedbynginx#TYPEnginx_connections_currentgaugenginx_connections_current{state="active"}2nginx_connections_current{state="reading"}0nginx_connections_current{state="waiting"}1nginx_connections_current{state="writing"}1#HELPnginx_connections_processed_totalNumberofconnectionsprocessedbynginx#TYPEnginx_connections_processed_totalcounternginx_connections_processed_total{stage="accepted"}300966nginx_connections_processed_total{stage="any"}4.241939e+06nginx_connections_processed_total{stage="handled"}300966

nginxdashboard

requestconnection

irate(nginx_connections_processed_total{...,stage="any"}[1m])

Elasticsearch monitoring

• Useelasticsearch_exporter• UseMarvel

Elasticsearch dashboard

javaheap

https://github.com/elastic/elasticsearch/issues/18635

Myfeeling

• Prometheus’squeryisreallypowerfulespeciallywhenmetrictypeiscounter

• Wecanuserate/iratefunction.• nginx_exporter and elasticsearch_exporter areconvienient– easytosetup

• PrometheusandGrafana areaperfectcombination

References

• https://github.com/discordianfish/nginx_exporter

• https://github.com/ewr/elasticsearch_exporter