cassandra metrics

Monitor EverythingBy: Chris

Lohfink

Cassandra Summit 2014

About Me

#CassandraSummit 2014

● Sr. Engineer at Pythiano Lead of Cassandra Practice

● Remote in Minnesota● DataStax MVP for Cassandra ‘14● Interests

o Java, Clojure, Python devo Data scienceo Hobbyist electronics

About Pythian

Pythian is a global data outsourcing and consulting company that specializes in optimizing and managing mission-critical data systems.

Pythian blends the world’s leading data experts with advanced, secure service delivery processes to create the industry’s best standard of care for its clients.

Since its inception, Pythian has managed some of the world’s largest, most business-critical data infrastructures.

10,000Pythian currently manages more than 10,000

systems.

350Pythian currently employs more than 350 people

in 25 countries worldwide.

1997Pythian was founded in 1997

About Cassandra

● No Single Point of Failure

● Fault Tolerant

● Awesome properties for an operations team who does not want to get up at 3am

About Cassandra

● Nothing should be set up and forgotten about● Easy to do with Cassandra though

o Fault tolerance on properly configured setup handles single node being down or having temp performance issues

o No back pressure on writes until there is a lot of trouble

Utilize the fault tolerance buffer

● Need to observe and react to current issues● Predict future issues● Divide this into two approaches

o Proactiveo Reactive

Proactive

● Daily & Weekly checkups to prevent, and predict problemso Capacityo Performance bottleneckso Data Modeling issues

Reactive

● Something about best laid plans…o Hardware failureso Bugso Malicious or Non-Malicious users

● Alarms, Pager Duty

Common element

● Data is neededo form alertso find anomalieso trendingo debugging

Metrics

● Window to the applicationo Bridge the gap - Coda Hale

Cassandra Environment

OpsCenter Logs

JMX CPU, Disk, Network

Nodetool JVM, GC

SOURCES

Gathering Metrics

Metricsbut of course…

Without context, the data is just pretty graphs

● Java Management Extensions● Complex… very engineered● Resources represented as objects with

attributes and operations● Used for monitoring or as input

● The annoying gateway to metrics ○ Poor tooling - requires java○ Slow, Memory Leaks○ Historically and currently frustrating for ops (pre 2.0.8)

1024-65535

Init connection to port 7199 Reply with hostname:port

for RMI connection

Client (You)

Gets new hostname:port, drops old connection and attempts to connect

Connected!

Cassandra

● Visualo jconsoleo visualvm

● Command lineo jmxtermo jmxsh

● MX4J● Jolokia

[domain]:[key]=[value],[key2]=[value2]...

com.pythian:site=blog,type=views,target=post1

JMX Domains

org.apache.cassandra.● db● internal● net● request

org.apache.cassandra.metrics● db● internal● net● request

JMX Beans

org.apache.cassandra.metrics :type=

● Cache● Client● ClientRequest● ClientRequestMetrics● ColumnFamily● CommitLog● Compaction

● DroppedMessage● FileCache● Keyspace● Storage● ThreadPools

org.apache.cassandra.metrics

type=*, scope=*, name=*,

type=ThreadPools, path=*, scope=*, name=*,

type=ColumnFamily, keyspace=*, scope=*, name=*,

type=Keyspace, keyspace=*, name=*,

Metrics

● Toolkit called metrics for metricso By Coda Hale @ Yammer

● Easy to use● Easy to read (if you know java)● Popular

Types of Metrics

● Gaugeo instantaneous value

● Countero number that can be incremented & decremented

● Metero rate of events over time (1/5/15 min moving avg)

● Histogramo representation of statistical distribution

50, 75, 95, 98, 99, 99.9 percentile average, median, min, max, standard deviation

● Timero rate of events (meter)o histogram of duration

75th percentile is 683 MICROSECONDS (75% took 683us or less)

One minute rate is 13,915 calls per SECOND

● Overwhelming at first● Hard to tell what they mean without the source● Moves around a lot● Fortunately there is nodetool

Nodetool

● JMX command line wrapper● Many options● Operations and diagnostic procedures● For reactive analysis

o ad hoc, spot checks

Nodetool tpstats

nodetool tpstatsorg.apache.cassandra.metrics:type=ThreadPools,path={internal|request|transport},scope={*},name= {ActiveTasks|PendingTasks|CompletedTasks|CurrentlyBlockedTasks|TotalBlockedTasks}Pool Name Active Pending Completed Blocked All time blocked

ReadStage 0 0 113702 0 0RequestResponseStage 0 0 0 0 0MutationStage 0 0 164503 0 0...InternalResponseStage 0 0 0 0 0HintedHandoff 0 0 0 0 0

Message type DroppedRANGE_SLICE 0READ_REPAIR 0...REQUEST_RESPONSE 0COUNTER_MUTATION 0

Staged Event Driven Architecture

● Decomposes complex event system● Set of stages (thread pools)● Queue between each● Shares a lot of pros cons as SOA

Threads

ReadStage

est RequestResponse

Threads

ReadRepairStage

Threads

MessagingService

Node 2

Node 1 Node 1

= Task

● Possible to overrun the processing capabilities of a stage that is not in the requests feedback loop (i.e. ReadRepairStage)

Nodetool tpstats

RequestResponse

Threads

Nodetool tpstats

RequestResponse

Threads

Nodetool tpstats

nodetool tpstatsorg.apache.cassandra.metrics:type=ThreadPools,path={internal|request|transport},scope={*},name= {ActiveTasks|PendingTasks|CompletedTasks|CurrentlyBlockedTasks|TotalBlockedTasks}

More at:http://www.evidencebasedit.com/guide-to-cassandra-thread-pools

Nodetool cfhistograms

nodetool cfhistograms {keyspace} {table}org.apache.cassandra.metrics:type=ColumnFamily,keyspace={keyspace},scope={table}

SSTables per Read1 sstables: 985542 sstables: 4534

Write Latency (microseconds)No Data

Read Latency (microseconds) 10 us: 2 12 us: 17 14 us: 96 17 us: 208 20 us: 677 24 us: 3081 29 us: 4552 35 us: 3559

Read Write Path mile high overview

Memtable SSTable

Writes Reads

Memtable SSTable

Writes Reads

Memtable SSTable

Writes Reads

Memtable SSTable

Writes Reads

Memtable SSTable

Writes Reads

SSTables per Read1 sstables: 985542 sstables: 4534

Write Latency (microseconds)No Data

Read Latency (microseconds) 10 us: 2 12 us: 17 14 us: 96 17 us: 208 20 us: 677 24 us: 3081 29 us: 4552 35 us: 3559

Nodetool cfhistograms 1.1

Offset SSTables Write Latency Read Latency Row Size Column Count1 3579 0 0 0 02 0 0 0 0 0. . .35 0 0 0 0 042 0 0 27 0 050 0 0 187 0 060 0 10 460 0 072 0 200 689 0 086 0 663 552 0 0103 0 796 367 0 0124 0 297 736 0 0149 0 265 243 0 0179 0 460 263 0 0. . .25109160 0 0 0 0 0

https://gist.github.com/clohfink/6068003

Nodetool cfhistograms 2.1

Keyspace/Table histogramsPercentile SSTables Write Latency Read Latency Partition Size Cell Count (micros) (micros) (bytes) 50% 1.00 10.00 524.00 310 575% 1.00 11.75 888.00 310 595% 1.00 15.00 4843.75 310 598% 1.00 17.00 9658.90 310 599% 1.00 19.00 12306.47 310 5Min 0.00 0.00 68.00 30 0Max 2.00 219386.00 45383.00 310 5

Nodetool cfstats

nodetool cfstats {-i} {keyspace}.{table}org.apache.cassandra.metrics:type=ColumnFamily,keyspace={keyspace},scope={table}

Keyspace: Keyspace1 Read Count: 11207 Read Latency: 0.047931114482020164 ms. Write Count: 17598 Write Latency: 0.053502954881236506 ms. Pending Tasks: 0 Table: Standard1 SSTable count: 3 Space used (live), bytes: 9088955 Space used (total), bytes: 9088955 Space used by snapshots (total), bytes: 0 SSTable Compression Ratio: 0.3672150946 Memtable cell count: 0 Memtable data size, bytes: 0 Memtable switch count: 3

Local read count: 11207Local read latency: 0.048 msLocal write count: 17598Local write latency: 0.054 msPending tasks: 0Bloom filter false positives: 0Bloom filter false ratio: 0.00000Bloom filter space used, bytes: 11688Compacted partition minimum bytes: 1110Compacted partition maximum bytes: 126934Compacted partition mean bytes: 2730Average live cells per slice: 0.0Average tombstones per slice: 0.0

Nodetool cfstats

Keyspace: Keyspace1 Read Count: 11207 Read Latency: 0.047931114482020164 ms. Write Count: 17598 Write Latency: 0.053502954881236506 ms. Pending Tasks: 0 Table: Standard1 SSTable count: 3 SSTables in each level: [14/4, 1, 0, …, 0] Space used (live), bytes: 9088955 Space used (total), bytes: 9088955 Space used by snapshots (total), bytes: 0 SSTable Compression Ratio: 0.3672150946 Memtable cell count: 0 Memtable data size, bytes: 0 Memtable switch count: 3

Nodetool cfstats

Nodetool proxyhistograms

nodetool proxyhistogramsorg.apache.cassandra.metrics:type=ClientRequest,scope={Read|Write|RangeSlice},name=Latency

$ nodetool proxyhistogramsproxy histograms

Read Latency (microseconds)61214 us: 1

Write Latency (microseconds) 103 us: 22 124 us: 142 149 us: 297 179 us: 1190 215 us: 1823 258 us: 2091

Nodetool

Much more!!

http://www.datastax.com/documentation/cassandra/2.0/cassandra/tools/toolsNodetool_r.html

Reporting Interface

Default Addons Community

JMX Ganglia Cassandra StatsD NewRelic Splunk

Console

Graphite

Cloudwatch Kafka Riemann TempDB

Csv Munin Riak InfluxDB Sematext

Slf4j MongoDB OpenTSDB Librato … MORE

Reporting Interface

● Configurable with yamlo console, csv, ganglia, graphite, riemann

● Create reporter with premain agento compiling new jar with manifesto add to classpatho add javaagent in cassandra-env.sh

Garbage Collection

● Death, Taxes, and a stop the world GC● Common issue to all JVM based applications

Garbage Collection

Enable gc logging● Virtually no overhead● Can be very helpful in diagnosing

performance issues

Garbage Collection

JVM_OPTS="$JVM_OPTS -XX:+PrintGCDetails" JVM_OPTS="$JVM_OPTS -XX:+PrintGCDateStamps" JVM_OPTS="$JVM_OPTS -XX:+PrintHeapAtGC" JVM_OPTS="$JVM_OPTS -XX:+PrintTenuringDistribution" JVM_OPTS="$JVM_OPTS -XX:+PrintGCApplicationStoppedTime" JVM_OPTS="$JVM_OPTS -XX:+PrintPromotionFailure" JVM_OPTS="$JVM_OPTS -XX:PrintFLSStatistics=1"

JVM_OPTS="$JVM_OPTS -Xloggc:/var/log/cassandra/gc-`date +%s`.log" JVM_OPTS="$JVM_OPTS -Xloggc:/var/log/cassandra/gc.log" JVM_OPTS="$JVM_OPTS -XX:+UseGCLogFileRotation" JVM_OPTS="$JVM_OPTS -XX:NumberOfGCLogFiles=10" JVM_OPTS="$JVM_OPTS -XX:GCLogFileSize=10M"

Garbage Collection

JVM_OPTS="$JVM_OPTS -XX:+PrintGCDetails" JVM_OPTS="$JVM_OPTS -XX:+PrintGCDateStamps" JVM_OPTS="$JVM_OPTS -XX:+PrintHeapAtGC" JVM_OPTS="$JVM_OPTS -XX:+PrintTenuringDistribution" JVM_OPTS="$JVM_OPTS -XX:+PrintGCApplicationStoppedTime" JVM_OPTS="$JVM_OPTS -XX:+PrintPromotionFailure" JVM_OPTS="$JVM_OPTS -XX:PrintFLSStatistics=1"

JVM_OPTS="$JVM_OPTS -Xloggc:/var/log/cassandra/gc-`date +%s`.log" JVM_OPTS="$JVM_OPTS -Xloggc:/var/log/cassandra/gc.log" JVM_OPTS="$JVM_OPTS -XX:+UseGCLogFileRotation" JVM_OPTS="$JVM_OPTS -XX:NumberOfGCLogFiles=10" JVM_OPTS="$JVM_OPTS -XX:GCLogFileSize=10M"

Garbage Collection

Could be its own talk

Honorable mentions:● https://github.com/chewiebug/GCViewer● http://jworks.idv.tw/GcWeb/ ● Python, R, Octave

Logging

/var/log/cassandra/system.log o provides a rolling logo log4j

/var/log/cassandra/output.log o captured standard error and standard outo truncated on restart

System Logso syslog, dmesg, etc

OS Metrics

Shout-out:

http://www.brendangregg.com/linuxperf.html

● Heapo GC logso JMX

● Threadso jvmtopo Jstack (+htop)o kill -3o JMX

And Everything

Questions

cassandra metrics

Technology

pythian blends

metrics toolkit

metrics window

types of metrics

jmx beansorg

jmx domainsorg

jmx overwhelming

message type droppedrange