Tuning WebSphere Application Server ND 7.0

Royal Cyber Inc.

Application server stops responding• Server crash• Hung process• Out of memory condition

Performance degradation• Check if the process ID is continually changing

o Indicates the application server is probably crashing and being restartedCore filesJavacore filesVerboseGC logsHeap dump

JVM related problems

The tuning process is iterative in nature. To find the optimal configuration several tests and evaluation should be performed

Tuning Methodology

IBM Monitoring and Diagnostic Tools for Java - Garbage Collection and Memory Visualizer

JVM Analysis Tools

IBM Pattern Modeling and Analysis Tool for Java Garbage Collector

JVM Analysis Tools

WebSphere process hang detection stepsa) Once a hang is suspected, obtain a thread dump or javacoreb) For a typical hang, collect three dumps at a few minutes interval

To see if anything is moving within the process (but slowly)

a) Examine the thread dumps with Thread Analyzer or by handHang detection toolsThreadMonitor architecture was created to monitor thread pools withinThreadAnalyzerGUI-based tool Gathers and analyzes thread dumps from a WebSphere Application ServerProvides recommendations based on analysis

Troubleshooting server hang issues

ThreadAnalyzer

ThreadAnalyzer: Analysis

ThreadAnalyzer: Multiple dump analysis

ThreadAnalyzer: Overall thread analysis

SymptomsProcess terminated with Java exception or native signal

Usual causesOut of memory exceptionCall stack overflowUnexpected exception (for example, out of disk space) Optimizer failure (for example, JiT)Bad JNI call or library problemSegmentation violations while executing native code

How to troubleshoot crashes

Dump Analyzer: Analysis completed

Java virtual machine error Not enough memory to allocate an object can be caused by the following:The Java heap is too smallMemory is available in the heap, but it is fragmented (for JDK 1.4.2 and earlier)Memory leak in the Java codeNot enough space in the native memory

How to troubleshoot Out Of Memory error?

Administrative Console TPV Graph

PMAT: Summary page

PMAT: Analysis and recommendations

PMAT: Chart view

Each task performed by the server runs on a thread obtained from one of WebSphere Application

Server’s many thread pools

Three of the most commonly used (and tuned) thread pools within the application server are:

• Web container: Used when requests come in over HTTP

• Default: Used when requests come in for a message driven bean or if a particular transport

chain has not been defined to a specific thread pool.

• ORB: Used when remote requests come in over RMI/IIOP for an enterprise bean from an

EJB application client, remote EJB interface, or another application server

Thread pool size

A good practice is to use 5 threads per server CPU core for the default thread pool10 threads per server CPU for the ORB and Web container thread pools. For a machine with up to 4 CPUs, the default settings are usually a good start for most applications

There could be situations where the thread pool size might need to be increased to account for slow I/O or long running back-end connections

Default Thread Pool Setting

The IBM Tivoli® Performance Viewer is a tool embedded in the administrative console that lets you view the PMI (Performance Monitoring Infrastructure) data associated with almost any server component.The viewer provides advice to help tune systems for optimal performance and recommends alternatives to inefficient settings.

Monitoring and Tuning of Thread Pools

Each time an application attempts to access a back-end store (such as a database), it requires

resources to create, maintain, and release a connection to that data store.

PMI Metrics before tuning connection pool

Monitoring and Tuning Connection Pool

PMI Metrics after tuning connection pool

Monitoring and Tuning Connection Pool

• Data source statement cache size specifies the number of prepared JDBC statements that can be

cached per connection.

• Review the application code (or an SQL trace gathered from the database or database driver) for

all unique prepared statements, and ensure the cache size is larger than that value.

• Iteratively increase the cache size and run the application under peak steady state load until the

PMI metrics report no more cache discards.

Monitoring and Tuning Data source statement cache size

• PMI metrics after tuning the data source statement cache

Monitoring and Tuning Data source statement cache size

Five steps for all messaging issues• Check that the ME is in a started state

Debug steps for Websphere Default Messaging

Check SystemOut.log for the specific exception we are trying to debug.

Check the FFDCs for further details on the specific exception we are trying to debug.

Check the configuration.

Trace and engage IBM support.