Bill Au

CBS Interactive

http://www.apache.org/~billa/oscon2012/

Troubleshooting Slow or Hung Java Applications

2

Agenda

» Tools of the Trade

» Case Studies

2

3

Tools – Out Of The Box

»  jinfo, jmap, jstack, jstat

» JConsole, VisualVM – Monitor CPU usage by using JTop plug-in ($JAVA_HOME/

demo/management/JTop)

»  java.lang.management –  Platform MXBeans: runtime, operating system, class loading,

compilation, garbage collection, memory, thread –  Sample code in $JAVA_HOME/demo/management –  JDK demo and sample was part of JDK download for

jdk6u31 or older, separate download for u32+

3

4

Tools – java.lang.management

» Sample code: import java.lang.management.*;

ThreadMXBean tb=ManagementFactory.getThreadMXBean();

if (tb.isThreadCpuTimeSupported()) {

long[] tids=tb.getAllThreadIds();

for (long tid : tids) {

System.out.println(“cputime:”+tb.getThreadCpuTime(tid));

System.out.println(“usertime:”+tb.getThreadUserTime(tid));

}

}

4

5

Tools

» Garbage collection log –  -Xloggc:<filename> –  Timestamp and duration of GC events –  Sizes before and after GC events – More options to explore: − -XX:+PrintGCDateStamps − -XX:+PrintGCDetails − -XX:+PrintTenuringDistribution − -XX:+PrintGCApplicationsStoppedTime

–  Some GC flags can be set dynamically by jinfo/JConsole/VisualVM: PrintGC, PrintGCDetails, PrintGCTimeStamps

» visualgc – GUI for garbage collection monitoring

5

6

Tools

» Thread dump –  kill -3 (or kill –SIGHUP) –  JConsole, VisualVM –  ThreadMXBean –  jstack (-F)

» Consecutive thread dumps –  Stop motion animation

6

7

Hung or Slow App

» Long GC pauses and/or high GC overhead –  Basic GC tuning − max heap size: -Xmx − type of collector used

−  -XX:+UserSerialGC −  -XX:+UseParallelGC/-XX:+UseParallelOldGC −  -XX:+UseConcMarkSweepGC −  -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC (jdk6u14+)

–  Advanced GC tuning − young generation size: -Xmn

–  Baseline, tune, measure one change at a time

» Demo: HPjmeter – garbage collection analysis (and more…)

7

8

Hung or Slow App

» Deadlock –  Take thread dump to run the JVM’s deadlock detector

» Looping Threads – Monitor CPU times of running threads − ThreadMXBean − JTop plug-in for JConsole/VisualVM

–  Inspect stack trace of suspicious threads

8

9

Slow or Hung App

» Blocked threads –  Look for blocked threads waiting on the same condition –  Identify bottleneck for resource contention –  Inspect code for incautious and/or improper synchronization

» Demo: samurai – thread dump analysis tool

» Stuck threads –  Look for runnable threads with the same stack trace across

consecutive thread dumps –  Some common causes: − Blocking for a network call without specifying connect and/or

read timeout (default may be no timeout) − Long execution time for calls to backend servers

9

10

Links

» Presentation http://www.apache.org/~billa/oscon2012

» HPjmeter https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=HPJMETER

» samurai http://yusuke.homeip.net/samurai/en/index.html https://github.com/yusuke/samurai

10