cassandra summit 2014: diagnosing problems in production

©2013 DataStax Confidential. Do not distribute without consent.

Jon Haddad, Technical Evangelist, DatastaxBlake Eggleston, Software Developer, Datastax

Diagnosing Problems in Production

1

Preventative Measures• Opscenter •Metrics Integration •Munin •Monit •Nagios / Icinga • Graphite / Statsd (application level) • Variety of 3rd party monitoring services

Narrow Down the Problem•Weird consistency issues - NTP? • Problems with Streaming / Repair - version conflicts • Slow queries • Compaction • Histograms • Tracing

•Nodes flapping / failing • Dig into system metrics • JVM GC issues

Compaction• Compaction merges SSTables • Too much compaction? • Opscenter provides insight into compaction cluster wide • nodetool • compactionhistory • getcompactionthroughput

• Leveled vs Size Tiered • Leveled on SSD + Read Heavy • Size tiered on Spinning rust • Size tiered is great for write heavy time series workloads

System Utilities• iostat • disk level statistics

• htop • process overview

• iftop & netstat • network utilities

• dstat • all the above in 1 tool

• strace • …for the hardcore

Histograms• proxyhistograms • High level read and write times • Includes network latency

• cfhistograms <keyspace> <table> • reports stats for single table on a single

node • Used to identify tables with

performance problems

Query Tracing

JVM GC Overview•What is garbage collection? • Manual vs automatic memory management

• Generational garbage collection (ParNew & CMS) • New Generation • Old Generation

New Generation•New objects are created in the new gen •Minor GC • Occurs when new gen fills up • Stop the world • Dead objects are removed • Live objects are promoted into old gen • Removing objects is fast, promoting objects is slow

Old Generation• Objects are promoted to new gen from old gen •Major GC • Old generations fills up some percentage. • Mostly concurrent • 2 short stop the world pauses

• Full GC • Occurs when old gen fills up or objects can’t be promoted • Stop the world • Collects all generations • These are bad!

GC Profiling• Opscenter gc stats • Look for correlations between gc spikes

and read/write latency

• Cassandra GC Logging • Can be activated in cassandra-env.sh

• jstat • prints gc activity

GC Profiling•What to look out for: • Long, multi-second pauses • Caused by Full GCs. Old gen is filling up faster than the concurrent GC can keep up with

it. Typically means garbage is being promoted out of the new gen too soon • Long minor GC • Many of the objects in the new gen are being promoted to the old gen. • Most commonly caused by new gen being too big • Sometimes caused by objects being promoted prematurely

©2013 DataStax Confidential. Do not distribute without consent. 13