dealing with application performance issues: where to start?

Dealing with Application Performance

Issues: Where to Start?

Ahmed Alomari

Agenda

• Where to Start?

• Application Tier

• Network Performance

• Database Tier

• Tuning Workflow

• Application Tuning

• Q & A

Where to Start?

Where to Start?

• Performance Tuning is not rocket science.

➢Common Sense and Logic.

➢Understanding of how the components work.

➢Functional understanding of the use case.

• Focus on the underlying business flow rather than the technical issue.

• Understand the exact use case.

• Is the user performing an online txn, running a report, or a waiting for a concurrent request or Workflow to complete?

• Is there an external service involved (web service callout)?

• Apply your understanding of the technology and code path with the tuning best practices which you will learn.

Where to Start?

➢ First do not panic?

➢ Second, understand the exact flow which is slow?

• Is the entire system slow?

• Users might report the entire system is slow because the txns which they perform are slow.

• It is actually rare that the “entire system” is slow unless a general infrastructure issue exists which impacts the entire system.

➢ Find out exactly which processes and txns are slow.

• Obtain the detailed navigation steps.

Where to Start?

➢ Find out exactly which processes and txns are slow.

• Obtain the detailed navigation steps.

• Find out exactly when the performance issue started?

• Was it always slow since day 1?

• Was it performing fine earlier and just started recently?

• Was the performance issue always present, but an increase in data volume or concurrency highlighted the performance issue?

• Is it a performance issue or abnormal exception (white screen)?

Where to Start?

➢ After identifying the relevant process/transaction:

• Determine the impact on the business?• Business throughput/backlog?

• Is it impacting end users?

• Are the users waiting for response or the user requests timing out?

• Are these customer front-facing end users or back office users?

• Is it impacting crticial jobs involved in the transactional workflows?

• If not critical, does the job or transaction need to run during peak windows?

• Is the job still needed?

• Is it running too frequently?

Where to Start?

➢ “Nothing Changed!”

➢ If the performance issue started recently, then it is more likely that “not aware of the change which occurred.”

➢ Infrastructure Changes

➢ Security updates

➢ Configuration changes

➢ Statistics or data distribution changed

➢ “Subtle” changes.

➢Profile/trace the problematic transactions to determine the root cause of the change.

Performance Tuning

• Where do I start?

➢ Identify the relevant areas and focus on those areas of interest.

• For example, if the user says the Sales Order form find window is slow, looking at Concurrent Manager queues is unlikely to yield benefit.

• If a job is slow, obtain the details including the request id.

• Check the history of the job to determine typical runtimes and compare parameters being submitted for the slow requests with those of the requests which are quick.

Performance Tuning


➢ Identify the relevant areas and focus on those areas of interest.

• If the user is performing an online txn using a form or OA Framework based web page, obtain the exact navigation steps.

• Tracing can be done at the user level and for the specific pages which are slow in order to determine which step is slow.

Performance Tuning


• If a particular SQL statement is slow?

• Find the source/module of the SQL statement and check the history of the performance of the SQL including for possible execution plan changes.

• Did the execution plan change?

• Is the execution plan the same, but the elapsed time increased?

• Parameter value differences?

• Data volume changes?

• Period end data case?

• Identify the initiating business process or respective transaction which triggers the SQL to be executed.

Application Tier

R12.2 – Application Architecture

R12.2 - Dual File System Setup

High Level Tuning Guidelines

Shared Application Tier FS

• Ensure optimal performance of NAS

device and the shared application

filesystem (NFS)

– Runtime and patching activities perform a

large amount of file I/O activity

• copies/compilation/executable generation

• Request log and output files (runtime)

• UI requests (OHS and JVM logs)

– Fsclone and patching performance will be

impacted considerably due to slow NAS.

Shared Application Tier FS

• Ensure optimal performance of NAS

device and the shared application

filesystem (NFS)

– Utilize NFS V4 for application file system

mounts

• Ensure NAS device supports V4

• nas-app-prod:/app01 on /app01 type nfs4

(rw,relatime,vers=4.0,rsize=64k,wsize=64k,

namlen=255,soft,nordirplus,

local_lock=none)


• Review BIOS Settings for Application

tier hosts and DB tier hosts– Disable Node interleaving

– Number of cores per processor=ALL

– System Profile → Maximum Performance

• Disable power management mode

– Enable Hardware controlled Turbo/Bursting

– Hyperthreading (HT)

• Enabled for Application Servers

• Disabled for Database Servers


• Review Settings for Application tier

hosts and DB tier hosts

– Disable CPU throttling/governors

– cpupower frequency-info

– lscpu


• VMs

– Review reservation settings for memory and CPU for

production.


• Monitor Entropy pool to ensure sufficient availability

– If Entropy values are low or exhausted, the JVM can hang

– Java processes will appear to be hung with threads blocked waiting

on random key generation.

– applmgr> cat /proc/sys/kernel/random/entropy_avail

165

– applmgr> cat /proc/sys/kernel/random/entropy_avail

3520

– Might need to switch to /dev/urandom or use options to increase

Entropy availability.

Application Tier – Web Server• Use a hardware load balancer if available (LBR)

Application Tier - LBR• Hardware load balancer

– Select round-robin balancing method

– Enable cookie persistence

– Leverage hardware SSL acceleration

• Eliminates Apache SSL overhead

• Ensure keep alive is enabled

– KeepAlive ON (httpd.conf)

– If using Apache based SSL, remove nokeepalive primitive (ssl.conf)

• SetEnvIf User-Agent "MSIE" nokeepalive ssl-unclean-shutdown

• Enable X-Forwarded-For header

– Captures actual end user Client IP

– Configure Oracle HTTP Server (OHS) to log value to the access log.

Application Tier – Web Server

• Review access log file

– Check for frequent downloads or timestamp checks of:

• images, javascript, or style sheets.

• JAR files

– http codes

• 200 (request for document)

• 304 (timestamp check)

• 404 (document not found)

– Log entry format

• <IP address> <date/time> <command> <URL> <status code>

<bytes> <elapsed time>


• Review access log file

– Log entry format

• <IP address> <date/time> <command> <URL> <status code> <bytes> <elapsed time>

– Add %T in LogFormat option in OHS config

• 12.1.3: $IAS_CONFIG_HOME/Apache/Apache/conf/httpd.conf:

• 12.2: ${IAS_ORACLE_HOME}/instances/${ORACLE_INSTANCE_NAME}/config/${COMPONENT_TYPE}/${COMPONENT_NAME}

– LogFormat "%h %l %u %t \"%r\" %>s %b %T \"%{Referer}i\" \"%{User-Agent}i\" combined

– Records elapsed time in access log file.

– Helps determine page response time.


– Review OHS access log file

112.30.35.34 - - [26/Oct/2020:13:36:56 -0500] "GET /OA_HTML/OA.jsp?OAFunc=OANEWHOMEPAGE HTTP/1.1" 200 130018 15

112.30.35.34 - - [26/Oct/2020 13:37:30 -0500] "GET /OA_HTML/cabo/oajsLibs/oafcoreR122.js HTTP/1.1" 200 66728

112.30.35.34 - - [26/Oct/2020:13:37:30 -0500] "GET /OA_HTML/cabo/oajsLibs/oaGestureHandler.js HTTP/1.1" 200 22791

112.30.35.34 - - [26/Oct/2020:13:37:30 -0500] "GET /OA_HTML/cabo/styles/cache/alta-desktopEXTN-custom-11_1_1_7_0-

en-gecko-windows.css HTTP/1.1" 200 136174 5

112.30.35.34 - - [26/Oct/2020:13:37:30 -0500] "GET /OA_HTML/cabo/oajsLibs/tableUtils.js HTTP/1.1" 200 156324

112.30.35.34 - - [26/Oct/2020:13:37:31 -0500] "GET /OA_HTML/cabo/oajsLibs/popup.js HTTP/1.1" 200 66174

112.30.35.34 - - [26/Oct/2020:13:37:31 -0500] "GET /OA_HTML/cabo/oajsLibs/oaGestureLib.js HTTP/1.1" 200 42813

112.30.35.34 - - [26/Oct/2020:13:37:31 -0500] "GET /OA_HTML/cabo/oajsLibs/mainMenuTree.js HTTP/1.1" 200 44123

112.30.35.34 - - [26/Oct/2020:13:37:30 -0500] "GET /OA_HTML/cabo/jsLibs/Common12_2_6_0_0.js HTTP/1.1" 200 215878

112.30.35.34 - - [26/Oct/2020:13:37:31 -0500] "GET /OA_HTML/cabo/oajsLibs/richTable.js HTTP/1.1" 200 71007

112.30.35.34 - - [26/Oct/2020:13:37:31 -0500] "GET /OA_HTML/cabo/oajsLibs/OAFSlideoutMenu.js HTTP/1.1" 200 59330

112.30.35.34 - - [26/Oct/2020:13:37:31 -0500] "GET /OA_HTML/cabo/oajsLibs/flexibleLayout.js HTTP/1.1" 200 3985

112.30.35.34 - - [26/Oct/2020:13:37:31 -0500] "GET /OA_HTML/cabo/oajsLibs/FNDWFWorklistService.js HTTP/1.1" 200

27137

112.30.35.34 - - [26/Oct/2020:13:37:31 -0500] "GET /OA_HTML/cabo/oajsLibs/oafb.js HTTP/1.1" 200 1278

Elapsed Time

(seconds)Bytes Sent

JVM Tuning

Application Tier - JVM

• Java Heaps

Reference: https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/sizing.html

• Java Heaps (-Xmx/-Xms)

– -Xmx (maximum heap size)

– -Xms (minimum heap size at startup)

• Heap segment is typically memory mapped.

– Young Generation

• Eden (Nursery)

– NewRatio or NewSize/MaxNewSize

• Survivor Space

– Can be tuned via SurvivorRatio

– Tenured Generation (Old Generation)

– Permanent Generation / Metaspace (JDK 8)

• PermSize and MaxPermSize / MetaspaceSize and MaxMetaspaceSize

• Code Cache (classes)

• Constants

• Intern strings (moved to main heap in 7)



• JDK 7.0 Features

➢ Invokedynamic instruction for dynamic languages.

➢ Garbage-First (G1) Garbage Collector.

➢ Improved Class loading

➢ Parallel ClassLoader

➢ try with block

➢ java.lang.AutoCloseable

➢ Tiered Compilation

➢ -XX:+TieredCompilation

➢ Compressed Ordinary Object Pointers

➢ -XX:+UseCompressedOops

➢ NUMA Aware Collectors

➢ -XX:+UseNUMA

➢ Switch statement can reference String Objects as expressions.

➢ ThreadLocalRandom

➢ Improve performance of random number generators.


• Ensure you are running a current version of the JDK as

well as current patch set.

JDK

Version

Current

Patchset

MetaLink Note Applications

Version

7.0 Update 281

(31856324)

1467892.1 (12.1)

1530033.1 (12.2)

12.1.3 /12.2.X

8.0 Update 271

(31856303)

393931.1 (R12) JRE


➢ jvisualvm (JVM monitoring tool)


➢ jmc (Java Mission Control)


➢ jmc (Java Mission Control) – Flight Recorder

Application Tier - JVM• Review verbose GC data

– How often garbage collection is occurring, type of GC, and length of

GC (elapsed time).

263888.839: [GC [PSYoungGen: 378131K->9061K(393280K)] 1175648K->807594K(1212480K), 0.0249310 secs] [Times:

user=0.19 sys=0.01, real=0.02 secs]



264062.982: [Full GC[Unloading class sun.reflect.GeneratedMethodAccessor443]

[PSYoungGen: 2107K->0K(392768K)] [PSOldGen: 805666K->247375K(819200K)] 807774K->247375K(1211968K) [PSPermGen:

54677K->53972K(71680K)], 2.6600860 secs] [Times: user=2.63 sys=0.03, real=2.66 secs]






New Space

GC

Elapsed Time

Size Before

GCSize After

GC

Perm. Gen

Type of GC

Application Tier – Session Timeout

• Web Server Session duration/timeout

– Impacts overall footprint of the JVM.

• Set the Web Server session timeout to 30 minutes

– 12.1: orion-web.xml

• <session-timeout>30</session-timeout>

• s_sesstimeout

• s_oc4j_sesstimeout

– 12.2: profile “ICX:Session Timeout”

– Larger timeout values increase overall JVM memory

footprint.

– Ensure the users are trained to use the Logout or Home global

links when completing their transactions.

• Ensure users do not use the browser close (“x”) link.

– Logging out gracefully releases the memory and corresponding

resources (i.e. connections, etc..)

• Avoids memory leaks and timeout based invalidation.

Application Tier – Web ApplicationsApplication Tier – Session Timeout

Application Tier -- WebLogic

• Monitor Long Running and Stuck Threads• Stuck Thread Max Time

• Stuck Thread Timer Interval

Application Tier -- WebLogic

• Monitor Long Running and Stuck Threads• Stuck Thread Max Time

• Stuck Thread Timer Interval

• Stuck threads can also be due to long running SQL

statements• Thread stack will show the thread inside an active JDBC call.

• Long running tasks, integration or web service calls can

also show up as Stuck threads.

Application Tier – Connection Pool

• 12.2 WebLogic Admin Console

– Data Source Monitoring Statistics

Application Tier – Connection Pool

• Monitor the number of JDBC connections for both

APPS and APPLSYSPUB to check for leaks or

excessive connection usage.

• General rule of thumb is that the total number of

connections should not exceed 2*<peak number of

users>.

– Keep in mind that each JVM will create ~10 connections for

background threads and bootstrapping at JVM startup time.

• Query GV$SESSION and group by module to

determine which modules are consuming the

connections.

Concurrent Manager

Application Tier – Concurrent Manager

• Avoid enabling an excessive number of standard or specialized managers.

• Use specialization rules and work shifts to bind specific jobs to specific time windows.

– Helps avoid scheduling resource intensive batch requests during peak activity.

• Tune the sleep time for queues with a large number of target processes.

– Define dedicated queues for short and frequent requests and increase the sleep times for managers which do not require near real-time job execution.


• Utilize Parallel Concurrent Processing (PCP) to leverage the

Cluster (RAC).

– Environment variable APPLDCP=ON (context variable s_appldcp)

– Configure Multiple Concurrent Manager Instances

– Each CM Instance points to a specific DB instance (in the RAC

Cluster).

• Set s_cp_twotask to the respective DB instance TNS entry on each CM

Instance.

– TWO_TASK is then set correctly on each CM Instance.


• Parallel Concurrent Processing (PCP)

DB node2 DB node3 DB node4DB node1

Private

Interconnect

Public Network

CM node1 CM node2 CM node3

Shared Disk

CM node4


• Employ Application affinity for concurrent manager jobs to

minimize Interconnect traffic and maximize the performance of

the concurrent request.

– Order Import request runs on Standard Manager1 which connects

to DB Node1

– Workflow Background Engine which processes Order related Item

types (OEOH and OEOL) should also be scheduled on the same

node to minimize RAC InterConnect traffic.


• Application Affinity

– Use Specialization Rules via Inclusion/Exclusion at the Application level to implement affinity.

– Organize jobs first by Application, and then by dependency based on your knowledge of the Application usage.

– Example:

• Order Import (Order Management)

• Pick Release (Inventory)

• Since one of the first major items of work of Pick release is to fetch the “pickable” order lines, co-locating Pick Release with Order Import or Sales Order Creation minimizes Interconnect traffic.


• Workflow Background Engines (FNDWFBG)

– Specify Item Type in order to minimize contention as well as RAC

traffic

• Example:

– Process Order related Item types (OEOH and OEOL) on DB Instance 1

– Process Service Request related item type (SERVEREQ) on DB Instance 2

• 12.2: RAC Support for Background Engines

– New Program

– Workflow Background Process for RAC (FNDWFBGRAC)

– Allows Workflows to run on specific instances

– Partition runtime WF tables by INST_ID

– Implement custom VPD policy to filter by instance id

Network Performance

Network Performance

• Ensure the Application server hosts and Data server hosts are

connected to the same network switch

– Minimizes network latency between Application tier and Database

tier.

• Validate via traceroute that a direct route is being used.

• Latency should be less than 1 ms. Validate via a ping test.

{apptier-host}-> ping -s 2048 -c 5 proddb

PING proddb (10.67.12.45) 2048(2076) bytes of data.

2056 bytes from proddb (10.67.12.45): icmp_seq=0 ttl=255 time=0.307 ms





Network Performance

• Determine the number of hops and network latency for the

end users.

C:\>tracert apptier1

Tracing route to apptier1 [10.12.85.23]

over a maximum of 30 hops:

1 42 ms 37 ms 36 ms 10.64.23.2

2 42 ms 49 ms 56 ms 192.10.236.141

3 127 ms 131 ms 133 ms 192.10.245.177

4 140 ms 134 ms 143 ms 192.10.245.178

5 152 ms 132 ms 133 ms 10.12.85.1

6 159 ms 134 ms 138 ms apptier1

C:\> ping apptier1 -n 5 -l 512

Pinging apptier1 [10.12.85.23] with 512 bytes of data:

Reply from 10.12.85.23: bytes=512 time=148ms TTL=58





Database Tier

Database Tier

• Use large/huge pages for the SGA

– Ensures the SGA is pinned in memory

– Shares page table entries for SGA

mappings

– Reduces kernel memory utilization and

paging

Database Tier

• Use large/huge pages for the SGA

• Linux:

–set use_large_pages=ONLY

–Use 2 MB pages

–vm.nr_hugepages=<# of pages>

–memlock (/etc/security/limits.conf)

• AIX:

–set lock_sga=TRUE

– Use 16 MB pages (vmo options)

Database Tier

• Ensure Storage frame is optimized

– Monitor latency and IOPS

– Identify and tune top I/O consumers

– High LGWR latency will impact overall

database performance due to slow

commits.

– Use ASM for DB files.

– Tune kernel OS settings to eliminate AIO

timeouts.

Database Tier

• DB parameters:– *._high_priority_processes='LMS*|LGWR|VKTM|LM*|LCK0|GCR*|DIAG'

– *._use_single_log_writer='TRUE’

– Disable Adaptive Plans

• _optimizer_adaptive_plans=FALSE

• optimizer_adaptive_features=FALSE

– Do not set db_file_multiblock_read_count explicitly (use AUTO mode)

– Disable Cache Advisory once SGA sizes are finalized

• db_cache_advice=OFF

• Set minimum DB cache sizes (via spfile) to avoid frequent resizing:– db_cache_size

– shared_pool_size

– shared_pool_reserved_size=10% of shared pool size

– large_pool_size=2G

– java_pool_size=2G

Database Tier

• Redo log file sizes

– 8 GB for large databases

– 4 GB for medium size databases

• If storage level mirroring is in place

– Use a single redo log member per group

– Improves overall latency of LGWR

– Reduces # writes

• Monitor LGWR latency during peak windows.

Database Tier

• Disable default nightly and weekend

maintenance windows

– Batch cycles generally run through night and

weekend.

– Maintenance window throttles CPU for batch

cycles.

– Database sessions will wait on “resmgr:cpu

quantum” event

– Check the value of resource_manager_plan

Database Tier

• Review AWR or Top Activity from peak windows

– Top 5 Foreground Events by Time

– Load Profile

– Top SQL sections

– Top Activity (EM Page)

– Review I/O waits and average I/O service times.

Database Tier• Space Management

– Adopt the Oracle Applications Tablespace Model (OATM)

• Organizes tablespaces by category rather than by product.

• Co-locates large transaction based objects.

• Use AUTOALLOCATE mode for all Apps tablespaces.

• Refer to Support IDs 404954.1, 2064005.1.

– Ensure large transaction tables are placed in locally managed tablespaces with auto segment management enabled (ASSM).

• Use AUTOALLOCATE for Apps tablespaces.

• If UNIFORM extent size is being used, use an extent size of at least 10 MB for the large tables. Larger extent sizes such as 128MB can be used as well provided the very large tables are located in a dedicated tablespace.

– Ensure tablespace quotas are set to unlimited

• Avoids frequent update of TS quota information.

Database Tier

• Space Management

– Monitor frequency of waits for space management related events such as “enq: HW – contention”.

– Monitor execution frequency of space management related SQL statements.

select type#,blocks,extents,minexts,maxexts,extsize,extpct,

user#,iniexts,NVL(lists,65535),NVL(groups,65535),

cachehint,hwmincr, NVL(spare1,0),NVL(scanhint,0)

from seg$

where ts#=:1 and

file#=:2 and

block#=:3

select file# from file$ where ts#=:1

Database Tier

• Undo Tablespace

– Disable Auto Undo retention

• _undo_autotune = FALSE

– Tune undo_retention

• Start with a reasonable setting such as 12 hours and adjust as necessary.

• Value should be based on transaction lengths, not maximum query length.

– Query gv$UNDOSTAT or review AWR undo statistics section to determine undo usage as well as undo traffic.

– Monitor EXPIRED/UNEXPIRED extents via dba_undo_extents.

Database Tier

• Gathering Statistics

– Utilize the FND_STATS interface or equivalent Concurrent Programs

• (Gather Schema Statistics, Gather Table Statistics).

– Use the Gather Auto Option

• Gather Statistics on tables which have changed as per the modification threshold.

• Specify ‘GATHER AUTO’ for the options parameter

– fnd_stats.gather_schema_stats (. . .

.,options=>'GATHER AUTO‘);

– Ensure statistics are reflective of current data distribution.

Database Tier

• Gathering Statistics – Gather Auto Option

Database Tier

• Dictionary Statistics

– Gather stats on the dictionary

– After upgrade or decent app patching activity

• exec dbms_stats.gather_schema_stats

('SYS',degree=>30,estimate_percent=>100,cascade=

>TRUE);

• Fixed Object Statistics

– Execute during a peak window to ensure in-memory

objects reflect peak usage statistics

– exec

dbms_stats.GATHER_FIXED_OBJECTS_STATS;

Database Tier

• Statistics

– Review tables with zero statistics

• May need to delete the statistics for certain global

temporary tables so that the optimizer uses dynamic

sampling

• Can cause sub-optimal execution plans for those tables

which are populated due to zero stats.

• Some interface programs provide profiles to gather

statistics inline as part of the import program.

– AR: AutoInvoice Gather Statistics

– CST: Cost update- Gather Statistics

– EGO: Stats Collection Threshold

Database Tier - RAC

• Enable Jumbo Frames for the Interconnect

– Increase MTU to ~9000

• Minimizes number of packets needed to transfer

data blocks.

• Since Oracle Applications requires an 8K block

size, jumbo frames reduces the number of packets

overall.

$ /sbin/ifconfig bond1

bond1 Link encap:Ethernet HWaddr 00:1E:4A:06:4E:8A

. . . . . .

UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

$ ifconfig bond1 mtu 9000

Database Tier - RAC

• Monitor the Interconnect network traffic

and latencies via AWR.

Database Tier - RAC

• Use Services in place of port/host/sid.

– Increases availability

– Leverages Server side load balancing

• Use VIPs in the listener.ora and tnsnames.ora including the client

hosts.

– Improves failover time

– Avoids long waits upon failover due to physical host TCP timeouts

• Ensure PX requests do not span instances

– Set the database initialization parameter parallel_force_local=TRUE

Database Tier - RAC

• Review Top SQL sections of the AWR Report including the Cluster

Wait section.

– Tune SQLs causing a large amount of InterConnect traffic.

• Leverage Application Affinity (if applicable).

Database Tier - RAC

• Monitor CPU utilization on DB server nodes

– High and sustained server CPU utilization could induce or increase GC

related waits due to the delay of pushing and consuming messages.

Event Waits Time(s) Avg Wait(ms) % Total Call Time Wait Class

CPU time 569,114 28.5

gc cr grant 2-way 23,770,122 466,282 20 23.4 Cluster

gc current block 2-way 9,205,706 186,398 20 9.3 Cluster

gc buffer busy 12,701,689 158,452 12 7.9 Cluster

db file sequential read 61,525,660 139,660 2 7.0 User I/O

Top 5 Timed Events

Tuning Workflow

Tuning Workflow

• Workflow

– Background Engines (FNDWFBG)

• Background Engines which run frequently such as every 5 or 10 minutes

should disable the timeout and stuck process checks.

– Process Timeout=> No

– Process Stuck=> No

• Configure a separate background engine which runs a few times per day

to check for timed-out and stuck processes.

– Configure separate Background Engines by Item Type

• Orders

• POs

• Service Requests

Tuning Workflow

• Workflow

– Disable retention on the Workflow Queues

• Improves overall performance.

• Stablizes queue sizes.

select name,queue_table,retention

from dba_queues

where owner='APPLSYS' and retention != ‘0’;

select 'exec dbms_aqadm.alter_queue (queue_name=>''' || name ||

''',retention_time=>0);'

from dba_queues

where owner='APPLSYS' and

retention != '0'

order by name;

Tuning Workflow

• Workflow

– Ensure underlying documents are being closed properly such as

Orders, Purchase Orders, etc..

– Monitor Workflow runtime tables to ensure footprint is stable and

completed workflows are being properly purged.

– Schedule the purge program “Purge Obsolete Workflow Runtime

Data” to run based on the volume of Workflow data being generated.

• Specify an Item Type

• Commit Frequency => 1000

– Review the need to rebuild the Workflow runtime tables following a

mass purge.

– Rebuild indexes as needed as repeated purges will create holes in

indexes.

Tuning Workflow

• Workflow

– Agent Listeners

• Avoid specifying a correlation ID when launching the agent listeners as

dequeue by message ID is more efficient.

• If dequeue by correlation is needed for certain critical messages,

configure a dedicated listener to dequeue by correlation.

• Monitor performance of Agent Listeners.

• Monitor log file of Agent Listeners

– Review errors and exceptions.

• If dequeuing by message ID, poorly performing subscriptions can impact

entire queue performance and can result in queue pile-up.

Tuning Workflow

• Ensure Workflow processes are streamlined

• Eliminate unnecessary sub-processes and flatten the workflow.

• Significantly reduces amount of Workflow records generated.

• Example: Order Management:

– Line Flow –Generic: Performance

• Remove branches and activities which do not apply and will never be

used.

• Minimize attribute list and default values.

• Avoid using timers inside processes. Use status and/or result codes in

place of timers.

– Purchase Orders

• Ensure profile “PO: Set Debug Workflow ON” to No.

• Ensure Purchase Orders are closed (i.e. finally closed).

• Set “PO:Workflow Processing Mode” to Background if possible to

improve document creation performance.

Tuning Workflow

• Deferred Queue

– Ensure deferred queue is not abused or mis-used to mask poorly

performing subscriptions.

– Avoid flooding the queue and use bulk events if possible.

– Ensure the underlying tables of the WF_DEFERRED queue are as slim as

possible.

– Rebuild the WF_DEFERRED queue (if needed)

• Refer to Support ID 286394.1: “How to rebuild the WF_DEFERRED queue.”

Tuning Workflow

• Agent Listeners and Mailer

– Set Maximum Error Count:

• Max Error Count = 10K

– Tune Thread Count as per transaction rates

• Inbound Thread Count

– Consider creating dedicated Listeners for critical item types.

• Specify Correlation ID.

Application Tuning

Tuning the Application

• Order Management

– Use the Streamlined Workflows

• Eliminates unnecessary sub-processes and flattens the workflow.

• Significantly reduces amount of Workflow records generated.

• Line Flow –Generic: Performance

Line Flow: Generic,

Performance

Line Flow: Generic



– Use Quick Sales Order Form (OEXOETEL)

• Allows pricing and tax calculations to be deferred.



– Availability Check (ATP)

• Set the profile MRP: Calculate Supply Demand to No.

• Avoids inserting into SD temp tables

– Review and tune pricing setup to ensure optimal pricing engine

performance.

– Avoid creating triggers on OE_ORDER_HEADERS_ALL or

OE_ORDER_LINES_ALL for customizations.

• Use relevant business event or customize the Workflow via a custom

activity to insert integration point.

– Run the program Purge ATP Temp Tables-MSCATPPURG during

maintenance windows with Number of hours zero (i.e. truncate

mode)

• MRP_ATP_SCHEDULE_TEMP

• MRP_ATP_DETAILS_TEMP

Invoicing – Auto Invoice

• Ensure the table RA_INTERFACE_LINES_ALL

has representative statistics, and lock the

statistics.

• Create index on the standard flex fields (order#

and line id) on the interface table.

• Set the profile AR: AutoInvoice Gather

Statistics to No to avoid gathering statistics for

each and every run.

• Avoids invalidating cursors for online users.

Invoicing – Auto Invoice

• Tune the number of parallel workers

(RAXTRX) spawned by the master

program Autoinvoice Master Program

(RAXMTR).• 10K-20K lines per worker as a baseline and

adjust as needed.

• Review tax setup to ensure tax processing is

efficient.• Several performance patches related to tax

processing, and some are adapter specific.

Configurator

• Review Configurator rules to ensure

page load time is optimal• Opening the configurator UI from the Sales

Order form or Quoting UI should take 1-2

seconds ideally.

• Review Configurator Extensions and

profile them to determine how long each

CX is taking.

Configurator

• Configure the Configurator caches.

www.mous.us

THANK YOU

SAVE THE DATE

• OATUG Forum LiveDecember 7-11, 2020

Virtual Event

https://forum.oatug.org

• MOUS 2021October 27, 2021

Schoolcraft College -VisTaTech Center,

18600 Haggerty Rd, Livonia, MI

https://forum.oatug.org/

WWW.MOUS.US

THANK YOU

SURVEYS

• Session SurveysPlease complete the session survey for this

session using the Zoom session survey link.

The survey link will be provided via email

once the webinar is closed.

http://www.mous.us/

Q & A -- Thank you for attending

World Class Managed Services

Application Performance Management

Leader in Oracle Performance Tuning

Application Development and Integrations

Upgrades and Cloud Implementations

[email protected] Ph: (925) 924-0400

mailto:[email protected]

dealing with application performance issues: where to start?

Documents