oracle weblogic diagnostics & perfomance tuning
DESCRIPTION
extract of a held seminar in 2013TRANSCRIPT
Oracle WebLogic
Troubleshoot, Diagnostics &
Performance tuning
Michel Schildmeijer, 13 juni 2013
Agenda
17:10-18:00
• What does Performance include?
• Performance hints, tips & tricks
• Questions
Agenda
18:00
• Refreshments
My personalia
• Michel Schildmeijer, 46 years, living in Amsterdam
• Oracle Fusion Middleware specialist
���� WebLogic 6 to 12, OSB, Tuxedo, SOA
• Since April 2012 part of Qualogy
• Part of Qualogy’s Exalogic Squad team
Troubleshoot methods
• System errors, messages lost, bad performance
• Where do I begin??
• What method should I follow?
WebLogic Troubleshoot overview
• Using these troubleshooting methods helps you to:
• A structured way of detecting issues
• Get behind the cause of the problems that occur
• Methods or patterns to follow:
• Use of Operating system tools
• Use of JDK tools
• Use of WebLogic Server tools
• Most likely you will often use a combi-
nation
WebLogic Troubleshoot overview
• O/S tools:
• sar: system activity
• mpstat: per-processor statistics
• vmstat: virtual memory statistics
• netstat: network statistics
• iostat: Input/Output statistics
• top
WebLogic Troubleshoot overview
• Start with a simple check – it seems so obvious
but:
• Is there enough diskspace left
• Is there enough RAM available
• Does the system do a lot of paging
WebLogic Troubleshoot overview
• On Operating System:• See if WebLogic's operating system Java process is running
• Unix: ps -ef | grep -v ' weblogic.Name' | grep weblogic.Name –color
���� to see which server instances are running on that host
• Simple O/S tools like top, nmon, sar can help you on the way
• Check the contents of the process's system-out/err logs• Usually in the WebLogic Server log location or redirected to the
Server Logs
• Check the contents of the WebLogic
server and domain logs• Located at: <domain>/servers/
<server>/logs
WebLogic Troubleshoot overview
• WebLogic Server generic:
• WebLogic Diagnostics Framework
• Logs
• WLST
• JVM specific :
• JRockit Mission Control,
• Jstat,
• JConsole
Tools to troubleshoot
Command-line tools
Oracle Fusion Middleware basis
• FMW basis: de Applicatie Server
• Wat is een Applicatie Server?
• Wat doet een Applicatie Server?
• Welke Applicatie Server gebruikt Oracle?
• Operating system basic HotSpot JDK tools:• jps -l
• jinfo <pid>
• jmap <pid>
• jstat -<option><pid>
• pid is gathered using ps –ef …. Or
using jps
Tools to investigate
• jps –l
WebLogic Troubleshoot overview
Oracle Fusion Middleware basis
• FMW basis: de Applicatie Server
• Wat is een Applicatie Server?
• Wat doet een Applicatie Server?
• Welke Applicatie Server gebruikt Oracle?
• Operating system basic JRockit JDK tools:• jps -l
• jrcmd <pid> --MissonControl Command line
• pid is gathered using ps –ef …. Or
using jps
Tools to investigate
Oracle Fusion Middleware basis
• FMW basis: de Applicatie Server
• Wat is een Applicatie Server?
• Wat doet een Applicatie Server?
• Welke Applicatie Server gebruikt Oracle?
• jrcmd
• jrcmd <pid> -l | grep jrockit.gc.latest.heapSize
Tools to investigate
Oracle Fusion Middleware basis
• FMW basis: de Applicatie Server
• Wat is een Applicatie Server?
• Wat doet een Applicatie Server?
• Welke Applicatie Server gebruikt Oracle?
• Force a dump by:
• kill -3 <pid> … stack in STDOUT
• Usually in
<domain>/servers/logs/<serverinstance>/<serverinstance.out>
Tools to investigate
WebLogic troubleshoot methods
• Keep it as simple as it is :
• In the AdminConsole
• WLST • connect('weblogic', 'weblogic', 't3://localhost:700 1')
serverRuntime()print cmo.getHealthState()
• Or in clustered modedomainConfig()
serverList=cmo.getServers();
domainRuntime()
cd('/ServerLifeCycleRuntimes/<server>')
cmo.getState()
WebLogic troubleshoot methods
• Just a simple example, but parameterize your scripts
WebLogic troubleshoot methods
• A threaddump is a snapshot of the state of a WebLogic Server Instance
• Different ways to take a threaddump:
• kill -3 <pid> ���� the O/S way
• Using the JDK tools
• WLST(WebLogic Scripting Tool)cd ('Servers')
ls()
cd ('AdminServer')
ls()
threadDump()
WebLogic taking threaddumps
• In the AdminConsole
WebLogic taking threaddumps
• Thread Dumps can give you:
• View of the state of application server threads at that
instant in time
• Information about potential issues such as
• Hot spots within code which seem to be called often
• Portions of code where the application seems to be hung
• Locking and thread synchronization issues in an application
The use of taking threaddumps
• Take several dumps to investigate various states
• Threads high in stack can be treated as idle, mostly
waiting for some work to do
• Blocked or Wait on socketAccept() ����listener
thread waiting for socket connections
The use of taking threaddumps
• JVM process consists Java and JVM internal threads
• JVM threads: Garbage Collect, Optimizer and Finalizer
• Each thread has a stack to store runtime data
• Overflow will happen if this stack will store more than its size
• Each O/S has its default stacksize
• Can be extended with -Xss
Understanding Java threads
• Are meant to synchronize between threads
• Lock contention is a thread waiting for another to
be released
• Fat locks: Have a history of contention (several
threads try to take the lock simultaneously), or
waiting for notification
• Thin locks: Without any contention.
• Recursive locks: A lock on a
thread several times without having been released.
Understanding Locks
• WebLogic specific:
• Ignore threads in ExecuteThread.waitForRequest()
• Muxer and processSocket threads are mainly for
houskeeping
• A Muxer which is blocked should not be waiting for a
lock by a non Muxer
The use of taking threaddumps
• Hanging threads will be diagnosed as STUCK
• Default time to get STUCK is 600 secs
• Thread has not completed it’s work
The use of taking threaddumps
• Logging services are important for diagnostics
• WebLogic produces output by using a Message
Catalog
• Using the standard java.util.logging
• Can be extended using log4j modules
• All kinds of events, like startup/shutdown,
appl.deployments, subsystem failures
are written to logs
WebLogic Log and Debug options
WebLogic Log and Debug options
WebLogic Log and debug methods
• Several severity levels like:
• TRACE, DEBUG,INFO ..
• Important to set the proper level
• For FMW specific components, ODL-HANDLER can
be used
WebLogic Log and Debug options
• Subsystem logs can be from:
• HTTP
• Transaction Manager
• JDBC
• JMS server
• WebLogic Auditing provider
WebLogic Log and Debug options
• Several Subsystem can be debugged :
• HTTP
• JTA
• JDBC
• JMS server
• Security/SSL
Seems obvious, but only use it
when it’s neccessary
WebLogic debug options
• JDBC debug
• Setting debug options in the
AdminConsole
WebLogic debug options
• OpenSource Log engine from Apache
• Enable the LogMBean.isLog4jLoggingEnabled
attribute
• In WLST: cmo.setLog4jLoggingEnabled(true)
• Add WebLogic-specific Log4j classes,
WL_HOME/server/lib/wllog4j.jar and the log4j.jar
file to the server CLASSPATH
WebLogic Log4j
• Enable log4j in AdminConsole
WebLogic Log4j
• Or use –D options in startup script-Dweblogic.debug.DebugJDBCSQL=true
Prints information about all JDBC methods
invoked, arguments and return values, and thrown exceptions
Depending what item you want to debug: connection leaks,
queries
WebLogic debug options
• Or use WLST
serverConfig() or domainConfig()
debug = getMBean('/Servers/mserver/ServerDebug/mserver')
debug.setDebugJDBCInternal(true)
WebLogic debug options
• As an extension on the default logging handlers
• Comes with FMW installation, not WebLogic default
• Logging view and config in Fusion Middleware Control
• Can be set to several trace levels
• Output in <servername>-diagnostic.log
Oracle diagnostic Logging (ODL)
• Wrong or missing drivers in Classpath
• Connection errors by:
• Usernames/passwords
• Connection leaks by bad apps
• Set Inactive timeout to a value (def is 0)
• Insufficient connection parameters
• Max Capacity, Reserve Time Out(-1)
• Invalid JDBC URLS• weblogic.common.ResourceException:
Could not create pool connection.
The DBMS driver exception wa…
WebLogic JDBC common errors
WebLogic and JVM diagnostics
based on JRockit JDK
The JVM
• JVM is a byte code interpreter
• JVM heap is the area of memory used by
the JVM
• A Java application runs a piece of code
• Code enters the JVM ( JRockit, HotSpot)
• It enters several stages:
• Operations
• Data structure
• Transformation ( to bytecode)
JVM from code to machine
Java code
JVMJVM
Operatons
Data
Structure
Transformation
110110100
• Crash can occur during
• Compile process• Cause: invalid native code
• Internal Code – Garbage Collect ( Segmentation Fault)• Cause: GC Fault leading into bad memory access
• Long running GC can result in paging an CPU overload
• Application code
• Like 3rd Party drivers
• (JDBC, MQ libraries)
• Software modules
JVM from code to machine
• Runtime behaviour:
o JVM runs as an OS process
o Execute work through threading model
(concurrency)
o Garbage Collector
• Each Weblogic Server (instance) runs inside a
separate JVM (instance).
JVM from code to machine
• What to look for in java thread dumps:
• Low Memory Detector
• (Daemon thread to detect and report low memory conditions)
• CompilerThreadn
• (Daemon thread/s to compile & optimize byte code)
• Signal Dispatcher
• (Daemon thread to respond OS signals)
JVM troubleshoot
• Finalizer
• Daemon thread to handle objects in finalizer queue
• Reference Handler
• Daemon thread to handle objects in reference queue
• VM Thread
• Main thread from JVM
• GC task thread#n
• Thread/s to perform garbage collection
JVM troubleshoot
• JRockit
• JRockit Mission Control
• Grinder
• HotSpot
• JConsole
• JVisualVM
JVM tools
JRockit Mission Control
• JRockit Mission Control provides:
• Console
• Live JVM statistics
• Graphs and metrics about local or remote systems
• Basic JMX interaction with MBeans
• Runtime data
• An in-depth look at live metrics, like memory, CPU and stack trace on
specific threads
• Profile methods or exceptions
and execute commands
JRockit Mission Control
• Memory Leak Detector• Shows overall trend
• The growth rate of each type (class) is shown, rather than just the percentage of the heap they take up.
• No common classes from showing at the top (like char[] arrays)• Visual representation of hot objects in the heap
• By type• See how each class is referenced by other classes
• By instance• Can also see how each instance of each class is referenced
• Shows where objects are allocated• See stack trace of each method holding
an instance and what percentage of instances are held by each method.
JRockit Mission Control
WebLogic Diagnostics Framework
• Framework for monitoring and diagnostics
• Consolidated access for MBean data,
instrumentation, logging, debugging,
WLDF
• Source for monitoring data
• Server & application instrumentation notifications
based on the output : log, email, SNMP, JMX, JMS
• Request dyeing and tracing
• Data archiving for later access
• WLDF Dashboard
WLDF
• Key Features
• Gathering input from JMX MBeans
• WebLogic subsystems and application code can be
instrumented to provide events and watches
• Notifications can be defined for events and watches
• Console extension for monitoring of current and
historical data
• Data archiving
WLDF
• Can be configured on filesystem or Database
• Configure with Data retirement
WLDF
• Create Diagnostic Module
• Collect Metrics. A collection includes:
• The MBean type to query
• The specific MBean instance names to query (all instances,
• by default)
• The MBean attributes to collect (all attributes, by default)
• How often to gather data
• Create Watches & Notifications
WLDF
WebLogic Perfomance
• Performance : How system’s response time and throughput are affected by adding load.
• Capacity : Maximum threshold a system is under a given set of conditions.
• Scalability : how does a system responds to increasing load by adding additional resources.
• Key indicators are :• Response time
• Throughput
Performance Terms and Definitions
• Locate and minimize bottlenecks in Your System
• Gather info about Application(s)• Concurrent number of users and interfaces.
• Number and size of app requests.
• Amount of data and its consistency.
• Determining your target CPU utilization
• Tune your Database(s) ( client or repos)
• Tune WebLogic Server Performance Parameters
• Tune Your JVM
• Tune the Operating System
• Tuning WebLogic Persistent Store
Performance objectives
• Investigate and analyze!
• Look at the following
Tune WebLogic Server
• Enable by default
• Will use a O/S threadpool instead from “borrowing” from WebLogic def exec.Q
• Native muxers use platform-specific native binaries to read data from sockets
• Non native or Java sockets :
• Uses pure Java to read data from sockets.
• Blocks until there is data to be read
from a socket(can cause issues
on server level)
Native I/O and muxers
• Enable by default
• Will use a O/S threadpool instead from “borrowing” from WebLogic def exec.Q
• Native muxers use platform-specific native binaries to read data from sockets
• Non native or Java sockets :
• Uses pure Java to read data from sockets.
• Blocks until there is data to be read
from a socket(can cause issues
on server level)
Native I/O and muxers
• Represents the number of simultaneous
operations performed by apps that use the execute
queue.
WebLogic Thread Count
WebLogic Socket Muxer
Application
WebLogic Kernel assigns
Execute Req to
ExecuteThread 1
Execute Request
ExecuteThread 1 handles
the request (for example
executes a JDBC request)
WebLogic Kernel handles
self tuning ThreadPool,
Workmanagers, allocations
The app now handles the
thread itself and gives back
result
• Do not set this too high; can impact you
performance
• Default contains 15 threads
• Can be changed in config.xml
• Leave this default
WebLogic Thread Count
• Default All apps use all execute queues
• Configuring multiple execute queues can provide
additional control for applications.
• By using multiple execute queues, selected
applications have access to a fixed number of
execute threads, regardless of the load on
WebLogic Server.
Applications and Execute Queues
• To use user-defined execute queues include the use81-style-execute-queues sub-element in the config.xml file
• Validation of config.xml will fail, set
• setDomainEnv.sh-Dweblogic.configuration.schemaValidationEnabled=fa lse
• config.xml<server>
<name>WLS12cExaSvr1</name>……..>
<use81-style-execute-queues>true</use81-style-execu te-queues>
How to enable
• To use user-defined execute queues include the use81-style-execute-queues sub-element in the config.xml file
• Validation of config.xml will fail, set
• setDomainEnv.sh-Dweblogic.configuration.schemaValidationEnabled=fa lse
• config.xml<server>
<name>WLS12cExaSvr1</name>
……..>
How to enable
<use81-style-execute-queues>true</use81-style-execute-queues>
• ThreadPoolPercentSocketReaders
• Default set to 33 ( range is 1-99)
• Allocating threads as readers can increase
accepting requests
Threads as SocketReaders
• Piece of memory for read and write data to sockets
• A pool of chunks is maintained
• For applications with large amounts of data
• Parameters are:• weblogic.Chunksize—Size of a chunk (in bytes). Set to the network's maximum transfer
unit (MTU), after subtracting from the value any Ethernet or TCP header sizes
• weblogic.utils.io.chunkpoolsize—Sets the maximum size of the chunk pool. Monitor the CPU profile or use a memory/ heap profiler for call stacks invoking the constructor weblogic.utils.io.Chunk.
• weblogic.PartitionSize—Sets the number of pool partitions used (default is 4).
Partitioning the thread pool spreads the potential
for contention over more than one partition
To be set in the setDomainEnv.sh
WebLogic ChunkSize
• Specifies how many TCP connections can be
buffered. These are on the TCP stack but not
yet accepted by an app.
• Default is 50
• Maximum depends on O/S
WebLogic Backlog
• When to tune?• When getting “connection refused” in high throughput
apps ( SOA composites)
• If tuning, increase by 25% of it’s default until errors
dissapear
WebLogic Backlog
• Prepared Statement Cache
• Keeps compiled SQL statements in memory
• Set to LRU
• A data source with 10 connections deployed on 2 servers, with the Statement Cache to 10 (default), 200 cursors can be used on the database for these cached statements
• MaxCapacity
• Max of physical connections
• Db drivers can limit connections
• Number should equal number of client sessions
JDBC Connectionpool tuning
• In some cases it takes ages after startup to
continue
• Generate a stackup
Tuning Startup
• This happens when using default seed
generator:
• Use of dev/urandom during startup in stead of
random
• Specifiy in startup properties -
Djava.security.egd=file:///dev/urandom
• Or in $JAVA_HOME/jre/lib/security/java.securitysecurerandom.source=file:/dev/./urandom
Tuning Startup
• Most aspects of JVM tuning relate to:
• Sufficient memory heap
• An efficient garbage collection scheme
• Choice between two Unix threading models: green
and native threads
Tuning the JVM
• Most important to focus on:• Garbage collection scheme
• Memory heap• The Java heap is a runtime repository for live objects, dead
objects and free memory.
• The JVM heap size determines how often and how long the VM spends collecting garbage
• Large heap size :• full garbage collection slower and less frequent
• Small heap size • full garbage collection is faster
and frequency is more.
Tuning the JVM
• Goal is:
• Minimize Garbage Collect
• Set heapsize that GC does no happen all the time
Tuning the JVM
• Investigate with –verbosegc option
• How often does GC run
• How long does it take?
• Full garbage collection should not take longer than 3 to 5
seconds. Lower heap if major GC time is greater.
Tuning the JVM
• Set Xms and Xmx parameters to specify initial
and Max HeapSize
• Xmx is the Java Object Heap, not the System
Heap –> which is larger
• Setting these values equal will cause no
overhead of heapmanagement
• PermSize and MaxPermSize
For stored classes, methods
Tuning the JVM
Oracle Fusion Middleware basis
• FMW basis: de Applicatie Server
• Wat is een Applicatie Server?
• Wat doet een Applicatie Server?
• Welke Applicatie Server gebruikt Oracle?
Enterprise Management@@@
Let’s keep in touch!!
@MNEMONIC01
@Qualogy_news
@QAFE
Oracle Fusion Middleware basis
• FMW basis: de Applicatie Server
• Wat is een Applicatie Server?
• Wat doet een Applicatie Server?
• Welke Applicatie Server gebruikt Oracle?
Enterprise Management@@@
Let’s keep in touch!!
nl.linkedin.com/in/mschldmr
Oracle Fusion Middleware basis
• FMW basis: de Applicatie Server
• Wat is een Applicatie Server?
• Wat doet een Applicatie Server?
• Welke Applicatie Server gebruikt Oracle?
Enterprise Management@@@h
Let’s keep in touch!!
http://www.qualogy.com