monitoring cassandra with riemann
DESCRIPTION
Monitor your Cassandra cluster using RiemannTRANSCRIPT
Real-time Monitoring with Riemann
Patricia Gorla@patriciagorlaCassandra Consultantwww.thelastpickle.com
OSCON
About Us
• Work with clients to deliver and improve Apache Cassandra services
• Apache Cassandra committer, Datastax MVP, Hector maintainer, Apache Usergrid committer
• Based in New Zealand & USA
Overview
Setting up Riemann
Simple Monitoring
Aggregate Monitoring
Monitoring a cluster
database Compactions Pending!Data Size!Read/Write Requests!Read/Write Latency!CF SSTable Size
database Compactions Pending!Data Size!Read/Write Requests!Read/Write Latency!CF SSTable Size
Read/Write Request Timeouts!Dropped Messages!Hints Stored
networking
database Compactions Pending!Data Size!Read/Write Requests!Read/Write Latency!CF SSTable Size
Read/Write Request Timeouts!Dropped Messages!Hints Stored
networking
JVM CMS Collection Time!ParNew Collection Time
org.apache.cassandra.metrics
• Codahale metrics library first introduced in CASSANDRA-3671 (v 1.1)
• Pluggable reporters introduced in CASSANDRA-4430 (v 2.0.2)
• Complete description in Apache documentation
Dropped MessagesCache
Commit LogCompactionConnection
Client RequestStorage
Riemann• Real-time
copyright Kyle Kingsbury, used with permission
Riemann• Real-time
• Streams are successions of filters
copyright Kyle Kingsbury, used with permission
Riemann• Real-time
• Streams are successions of filters
• Expressive programming language
copyright Kyle Kingsbury, used with permission
Monitoring Ecosystem
Download package from riemann.io
/etc/riemann/riemann.config
Modified version of https://github.com/addthis/metrics-reporter-config
Specify the metrics to send from Cassandra
cassandra-env.sh
Starting Cassandra
INFO [2014-07-15 21:44:50,712] pool-1-thread-1 - riemann.config - average event #riemann.codec.Event{ :host x.x.x.x, :service org.apache.cassandra.metrics ClientRequest Write Latency .95, :state “ok”, :description nil, :metric 19514.037499999988, :tags nil, :time 1405482287, :ttl nil }
INFO [2014-07-15 21:44:50,712] pool-1-thread-1 - riemann.config - average event #riemann.codec.Event{ :host x.x.x.x, :service org.apache.cassandra.metrics ClientRequest Write Latency .95, :state “ok”, :description nil, :metric 19514.037499999988, :tags nil, :time 1405482287, :ttl nil }
INFO [2014-07-15 21:44:50,712] pool-1-thread-1 - riemann.config - average event #riemann.codec.Event{ :host x.x.x.x, :service org.apache.cassandra.metrics ClientRequest Write Latency .95, :state “ok”, :description nil, :metric 19514.037499999988, :tags nil, :time 1405482287, :ttl nil }
INFO [2014-07-15 21:44:50,712] pool-1-thread-1 - riemann.config - average event #riemann.codec.Event{ :host x.x.x.x, :service org.apache.cassandra.metrics ClientRequest Write Latency .95, :state “ok”, :description nil, :metric 19514.037499999988, :tags nil, :time 1405482287, :ttl nil }
INFO [2014-07-15 21:44:50,712] pool-1-thread-1 - riemann.config - average event #riemann.codec.Event{ :host x.x.x.x, :service org.apache.cassandra.metrics ClientRequest Write Latency .95, :state “ok”, :description nil, :metric 19514.037499999988, :tags nil, :time 1405482287, :ttl nil }
package name class module
INFO [2014-07-15 21:44:50,712] pool-1-thread-1 - riemann.config - average event #riemann.codec.Event{ :host x.x.x.x, :service org.apache.cassandra.metrics ClientRequest Write Latency .95, :state “ok”, :description nil, :metric 19514.037499999988, :tags nil, :time 1405482287, :ttl nil }
/etc/riemann/riemann.config
• Flexible
• Simple to set up
• Keyboard shortcuts
riemann-dash
Edit graphs with keyboard shortcuts
Mean 95th Percentile Latency across all nodes
Mean 95th Percentile Latency across all nodes
Can also use fixed-event-window for less regular occurrences
Mean 95th Percentile Latency across all nodes++
Mean 95th Percentile Latency across all nodes++
Mean 95th Percentile Latency across all nodes++
(avg)
Chart in riemann-dash
(avg)
Chart in riemann-dash
(avg)
Detecting threshold breaches
Generalise which metrics to monitor
Call the custom function in the (streams …) section
Patricia Gorla@patriciagorlaCassandra Consultantwww.thelastpickle.com
OSCONQ&A