double the performance of oracle soa suite 11g? absolutely!

Double the Performance of Oracle SOA Suite 11g? Absolutely!

April 10th, 2013

12:15-12:45

Mile High Ballroom 3C

Session ID #187

Ahmed Aboulnaga Technical Director

Raastech, Inc.

of 44 © Raastech, Inc. 2013 | All rights reserved.

1. Introduction

2. Summary

3. Monitoring Performance

4. Tuning

5. Resources

Agenda


INTRODUCTION


Ahmed Aboulnaga

15 years of Oracle middleware experience

Focus on Oracle Fusion Middleware, with emphasis on SOA,

portal, and identity management technologies

Implementations for defense, commercial, federal,

international, and global customers

Author of “Oracle SOA Suite 11g Administrator’s Handbook”

OCE (SOA Architect / SOA Foundations) and OCA (Application Server)

President of WMOUG/EGOUG

Oracle ACE

Blogger at blog.raastech.com

About Me


SUMMARY


Summary

JVM heap size is rarely a concern if:

• You size it appropriately

• You are running on PS3 (11.1.1.4) or higher

Disk I/O is never a concern on the midtier… the database is

a different story

SWAP is rarely a concern, but keep your eyes on it

CPU utilization is usually never a problem unless the

AdminServer acts up


Summary

Upgrading to PS3 (11.1.1.4) addresses memory instability issues

Moving from Sun JDK to JRockit results in a 32% boost

Optimizing the audit level makes a huge difference, resulting in

a 46% to 92% performance improvement

Increasing the Mediator worker threads results in a 30%

performance improvement for async services

Mediator sequential routing rules may improve performance

from 4% to 509%

Don’t forget to tune the BPEL Service Engine

Consider parallel garbage collection, but don't have statistics


MONITORING PERFORMANCE


Instance Performance

Timestamps on the console display up to seconds

Timestamps on the console are “last update time”


SQL Queries

SQL queries to the dehydration store are the best way to get

performance related metrics


DMS Spy Servlet displays instant Dynamic Monitoring

Service (DMS) related metrics

Navigate to http://<host>:<soaport>/dms/Spy

http://docs.oracle.com/cd/E15586_01/core.1111/e10108/monitor.htm#CFAHIAIB

The DMS Spy Servlet


Java Heap

Navigate to Home > Monitoring Dashboard

Once sized appropriately, usually not a problem


Data Sources

Navigate to Home > Data Sources


root@soahost1:/root> df –m

Filesystem 1M-blocks Used Available Use% Mounted on

/dev/sda8 996 451 494 48% /

/dev/sda9 815881 697454 76314 91% /u01

/dev/sda7 996 36 909 4% /home

/dev/sda5 1984 138 1744 8% /tmp

/dev/sda3 1984 283 1598 16% /var

/dev/sda2 5950 3842 1802 69% /usr

/dev/sda1 99 12 83 13% /boot

tmpfs 8023 0 8023 0% /dev/shm

Disk Space

Often an underlooked area


The vmstat command easily outputs CPU, memory, and I/O

statistics

Do not rely on Linux’s reporting of available memory, and

best to look at SWAP space usage

Why Linux reports 100% memory usage all the time:

http://blog.raastech.com/2008/01/why-linux-reports-100-memory-usage-all.html

Check CPU, RAM, and I/O

root@soahost1:/root> vmstat -S m

procs -------memory--------- --swap-- ---io-- --system-- ----cpu-----

r b swpd free buff cache si so bi bo in cs us sy id wa st

0 0 0 59 402 15055 0 0 2 16 0 0 2 2 96 1 0


System log files can reveal resource issues:

Check OS Resources

root@soahost1:/root> cat /var/log/messages

Aug 31 20:53:22 uslx286 sshd[22480]: fatal:

setresuid 10000: Resource temporarily unavailable

root@soahost1:/root> ps -A | wc -l

297

root@soahost1:/root> lsof | wc -l

6064

Too many open files can exhaust system resources:

Too many running processes can exhaust system resources:


TUNING


Tuning the Java Virtual Machine (JVM)

What?

Switch from Sun JDK to JRockit JDK

64-bit, to avoid 3GB (Linux) or 4GB (Windows) limitation

Why?

32% overall performance improvement

Forget what other people say about benefits of Sun JDK


Tuning the Java Virtual Machine (JVM)

How? $MW_HOME/wlserver_10.3/common/bin/commEnv.sh

OLD: JAVA_HOME="/u01/app/oracle/jdk1.6.0_26"

NEW: JAVA_HOME="/u01/app/oracle/jrockit1.6.0_29"

OLD: JAVA_VENDOR=Sun

NEW: JAVA_VENDOR=Oracle

$MW_HOME/user_projects/domains/[Domain]/bin/setDomainEnv.sh

OLD: BEA_JAVA_HOME=""

NEW: BEA_AVA_HOME="/u01/app/oracle/jrockit1.6.0_29"

OLD: SUN_JAVA_HOME="/u01/app/oracle/jdk1.6.0_26"

NEW: SUN_JAVA_HOME=""

$MW_HOME/user_projects/domains/[Domain]/bin/setSOADomainEnv.sh

OLD: PORT_MEM_ARGS="-Xms768m -Xmx1536m"

NEW: PORT_MEM_ARGS="-Xms1536m -Xmx1536m -Xgcprio:throughput

-XXtlasize:min=16k,preferred=128k,wasteLimit=8k"


Tuning the Java Transaction API (JTA)

What?

JTA provides begin/commit/rollback functionality for apps

The default value is 30 seconds

Why?

Want to avoid errors such as

Unexpected exception while enlisting XAConnection

java.sql.SQLException: Transaction rolled back:

Transaction timed out after 301 seconds


Tuning the Java Transaction API (JTA)

How?

Navigate to Services > JTA

Set Timeout Seconds from 120 to 300


Tuning the Log Level

What?

Standard SOA Suite logs are located in:

$MW_HOME/user_projects/domains/[Domain]/

servers/[soa_server]/logs

Why?

Excessive logging results in considerably more disk space

usage and also impacts performance


Tuning the Log Level

How?

Right-click on soa-infra, navigate to Logs > Log Configuration

Select the dropdown View With Persistent Log Level State

Log Levels - oracle.bpm ERROR: 1 (SEVERE)

Log Levels - oracle.integration ERROR: 1 (SEVERE)

Log Levels - oracle.sdp ERROR: 1 (SEVERE)

Log Levels - oracle.sdpinternal ERROR: 1 (SEVERE)

Log Levels - oracle.soa ERROR: 1 (SEVERE)

Log Levels - oracle.sysman ERROR: 1 (SEVERE)

Log Levels - oracle.wsm ERROR: 1 (SEVERE)


Tuning the Audit Store Logging

What?

Disable Capture Composite Instance State and

Payload Validation should be disabled

Why?

24% performance improvement

Because must query the COMPOSITE_INSTANCE,

CUBE_INSTANCE, and MEDIATOR_INSTANCE tables


Tuning the Audit Store Logging

How?

Right-click on soa-infra, navigate to SOA Administration >

Common Properties

Uncheck Capture Composite Instance State

Uncheck Payload Validation


Tuning the Audit Level

What?

Reduce the audit levels to Production

Why?

Reducing the audit level from Development to Production

results in a 46% performance improvement

Changing the audit level from Production to Off results in a

further 61% performance improvement, but Off is not

recommended


Tuning the Audit Level

How?


Common Properties

Set Audit Level to Production


Tuning the Audit Configuration Policy

What?

Set the audit configuration to Deferred instead of the default

Immediate setting

Why?

Perhaps the largest composite instance performance gain

can be attained by setting the audit policy to deferred, which

results in all audit data being committed to the database

asynchronously


Tuning the Audit Configuration Policy

How?


Common Properties > More SOA Infra Advanced

Configuration Properties

Click on Audit Config


Tuning the BPEL Service Engine

What?

There are many BPEL Service Engine properties that can be

tuned, but we will go through some of the key ones

Why?

Trust me!


Tuning the BPEL Service Engine

Parameter Name Default Tuned Remarks

Audit Trail 50000 20000 Messages with values greater than this threshold are

saved in the AUDIT_DETAILS table instead of

AUDIT_TRAIL.

Dispatcher System

Threads

2 10 Cleaning and housekeeping work for the engine.

Dispatcher Engine

Threads

30 100 Responsible for picking and processing instances. Set

higher if a lot of processes with many dehydration

points.

Dispatcher Invoke

Threads

20 70 Instantiates new instances in the engine. Tune higher

value if there are multiple branches/flows for faster

processing. Basically throttles the load setting of your

BPEL Engine.

AuditStorePolicy SyncLocal

Transaction

AsyncsyncL

ocal

Transaction

Will change the audit data storage strategy to save audit

data asynchronously, using a separate local transaction.

SyncMaxWaitTime 45 300 Determines the timeout for request-response

operations.


Tuning the Mediator Service Engine

What?

There are many Mediator Service Engine properties that can

be tuned, but we will go through some of the key ones

Why?

To improve overall throughput

Increasing the Mediator worker threads can result in a 30%

performance improvement for async services


Tuning the Mediator Service Engine

How?


Mediator Properties

Parameter Name Default Tuned Remarks

Metrics Level Enabled Disabled Unless DMS metrics collection is

explicitly required.

Parallel Worker Threads 4 20 Small vs. large environments.

Parallel Maximum

Rows Retrieved

200 600 Small vs. large environments.

Parallel Locker Thread Sleep 2 1 Bad parameter. Always set to 1.


Implementing BPEL In Memory Persistence

What?

Set inMemoryOptimization in your BPEL processes

Why?

To reduce the number of times the BPEL process writes to

the dehydration store

Useful for certain high performance use cases


Tuning BPEL Persistence

How?

Add the following to composite.xml to stop in-flight

instances from writing to the database:

<BPELProcess src="YourProcess.bpel" id="YourProcess">

<configurations>

<property name="inMemoryOptimization">true</property>

<property name="completionPersistPolicy">faulted</property>

<property name="completionPersistLevel">all</property>

</configurations>

</BPELProcess>


Implementing HTTP Timeouts

What?

Set HTTP timeouts at design time

Why?

To reduce consumption of threads in the event of poor

performing external services


Tuning HTTP Timeouts

How?

Add the following two properties in the reference section of

composite.xml:

<reference name="Register">

<binding.ws port="http://tempuri.org/#wsdl.endpoint

location="RegisterWS.svc.wsdl">

<property name="oracle.webservices.httpReadTimeout"

type="xs:string" many="false">10000</property>

<property name="oracle.webservices.httpConnTimeout"

type="xs:string" many="false">10000</property>

</binding.ws>

</reference>

Can be manipulated at runtime


Using Mediator Sequential Routing Rules

What?

‘Sequential’ Mediator routing rules are superior in

performance compared to ‘parallel’

This is because of the poor design that Oracle has chosen

for avoiding thread starvation

Why?

Implementing sequential routing rules may improve

performance of Mediator anywhere from 4% to 509%


100 messages 98 messages

Wait 1 second

No messages here

Wait 1 second

No messages here

Wait 1 second

No messages here

99 messages


Using Mediator Sequential Routing Rules

How?

Edit all *.mplan files for your Mediator operations and

change the executionType as follows:

OLD: executionType="direct"

NEW: executionType="queued"


RESOURCES


Oracle SOA Suite 11g Administrator’s

Handbook

http://www.packtpub.com/oracle-soa-suite-11g-

administrators-handbook/book

Chapter 4: Tuning Oracle SOA Suite

11g for Optimum Performance

“Highly recommended, a tour de

force.”

~Mark Nelson, Oracle A-Team

Book

http://redstack.wordpress.com/2012/10/28/a-review-of-oracle-soa-suite-11g-administrators-handbook/


Performance tuning Oracle SOA Suite 11g

http://blog.raastech.com/2011/04/performance-tuning-oracle-soa-

suite-11g.html

Blog


Ahmed Aboulnaga

Technical Director

[email protected]

Contact Information

double the performance of oracle soa suite 11g? absolutely!

Technology

summary slide

monitoring performance

problem slide

introduction slide

underlooked area slide

update time slide

resources agenda slide

dms spy servlet slide