inspiring value chain...
TRANSCRIPT
INSPIRING VALUE CHAIN ADVANTAGE
2013 OTM SIG CONFERENCE Performance Tuning/Monitoring
Alex Chang
July 30, 2013
Agenda
Page 2 INSPIRING VALUE CHAIN ADVANTAGE
• General guidelines – Effective tuning goal – Tuning session – Tuning life cycle
• Common tools • Case studies & Common tuning problems • Q&A
Effective Tuning Goal • Specific • Measurable • Achievable • Cost effective
3 INSPIRING VALUE CHAIN ADVANTAGE
Tuning Session • Define the problem and state the goal • Collect current performance statistics • Consider common performance errors • Build a trial solution • Implement and measure the change • Decide: ” Did the solution meet the goal? ”
– No. Then go back to step 3 and repeat – Yes. Then create a new baseline
4 INSPIRING VALUE CHAIN ADVANTAGE
Tuning Life Cycle • An OTM life cycle can be divided into different phases:
– Functional and technical design and implementation – Testing – Deployment – Production: Troubleshooting and tuning
• Use a test instance to determine whether the solution eliminated the bottleneck – Migration, upgrade, and environment changes
• Any change of OS, database and OTM requires thorough testing to avoid performance issues.
5 INSPIRING VALUE CHAIN ADVANTAGE
Tuning Life Cycle • Functional and technical design and implementation
– Proactive (make it better, so it will not break) • Test scenarios • Measure results • Find the problem areas • Resolve the problem
– Whenever possible, you should start tuning at this level. With a good design, many tuning problems do not occur
– It is less expensive to correct bugs and performance problems early in the life cycle than later
6 INSPIRING VALUE CHAIN ADVANTAGE
Tuning Life Cycle • Testing:
– More realistic test data – The test case should exercise the application functions, expected loads,
and stress tests of improbable loads – Check physical layout – Monitor for resource contention – Test for resource exhaustion – Incorporate the business requirements for recovery time and availability as
much as possible
7 INSPIRING VALUE CHAIN ADVANTAGE
Tuning During the Life Cycle • Deployment:
– When OTM is initially deployed, the performance expectations are often different than reality
– Generate regular performance reports and save them as baselines – Monitor growth and performance
• Compare new performance reports to the previous reports • Tune before the performance degrades to an unacceptable level
8 INSPIRING VALUE CHAIN ADVANTAGE
Design and Implementation • Production
– You need to know: • What has changed? • Were is the baseline?
9 INSPIRING VALUE CHAIN ADVANTAGE
Design and Implementation • Migration, Upgrade, and Environment Changes
– Create a test system. – Capture a representative workload – Execute the workload against the test system. – Collect statistics. – Reset the test system, make a change. – Execute the workload. – Capture statistics and compare.
10 INSPIRING VALUE CHAIN ADVANTAGE
Common Tools • OTM diagnostics servlets • Database monitoring tools
– AWR – ADDM
• OS monitoring tools – SAR/TOP
• JVM tools – Console logs – Thread dumps – Heap dumps
• Network monitoring
11 INSPIRING VALUE CHAIN ADVANTAGE
OTM Diagnostic Servlets • Event diagnostics
– http://localhost/GC3/glog.webserver.event.EventDiagServlet
• Cache diagnostics – http://localhost/GC3/glog.webserver.cache.CacheDiagServlet
• Object lock diagnostics – http://localhost/GC3/glog.webserver.synch.object.ObjectLockDiagServlet
• Mediator diagnostics – http://localhost/GC3/glog.webserver.mediator.MediatorDiagServlet
• Threads – http://localhost/GC3/glog.webserver.thread.ThreadDiagServlet
12 INSPIRING VALUE CHAIN ADVANTAGE
Event Diagnostics
13 INSPIRING VALUE CHAIN ADVANTAGE
Event Diagnostics
14 INSPIRING VALUE CHAIN ADVANTAGE
Cache Diagnostics
15 INSPIRING VALUE CHAIN ADVANTAGE
Object Lock Diagnostics
16 INSPIRING VALUE CHAIN ADVANTAGE
Oracle Enterprise Manager
17 INSPIRING VALUE CHAIN ADVANTAGE
Oracle Enterprise Manager
18 INSPIRING VALUE CHAIN ADVANTAGE
Oracle Enterprise Manager
19 INSPIRING VALUE CHAIN ADVANTAGE
Case Study 1: Design Phase • Business requirement: The client wants to manage its fleet resources
for its operational units in OTM as follows. – Driver: Available, Intransit, Unavailable – Power unit: Available, Intransit, Unavailable – Equipment: Intransit Empty, Intransit Loaded, Total intransit, Assigned, Out
of Service, Loaded Available, Empty Available…etc.
20 INSPIRING VALUE CHAIN ADVANTAGE
Case Study 1 : Design Phase • Potential approach – Location Asset Bucket Inventory
– Standard built-in inventory control and use SKU tables – Buckets are setup by resource type (i.e. 48’ Trailer,48’ Flatbed…etc.) – 41 buckets proposed
21 INSPIRING VALUE CHAIN ADVANTAGE
Case Study 1: Design Phase
22 INSPIRING VALUE CHAIN ADVANTAGE
Case Study 1: Design Phase • Location asset bucket inventory
– Pre-setup buckets for all operational units would result in the number of bucket records equal to # locations x bucket counts ( i.e. 100,000 locations x 41 buckets = 4.1 Million records)
– Daily shipments : 500,000 shipments – 1,500,000 insert and update per day – Serious performance bottleneck
23 INSPIRING VALUE CHAIN ADVANTAGE
Case Study 1 : Design Phase • Pros, Cons and Data Use Analysis
24 INSPIRING VALUE CHAIN ADVANTAGE
Common Tuning Problems • Logging
– The most frequent cause of performance problems – Single thread on the log file – each thread must wait its turn to write to the
log file – Create user logs for debugging and disable these when not in use
25 INSPIRING VALUE CHAIN ADVANTAGE
Bulk Plan – No Logging • Bulk plan 500 orders without logging
– Start time : 23:38:58 – End time: 23:39:45 – Duration: 47 seconds
26 INSPIRING VALUE CHAIN ADVANTAGE
Bulk Plan – Logging • Bulk plan 500 orders with logging
– Start time : 00:11:04 – End time: 00:12:22 – Duration: 78 seconds
27 INSPIRING VALUE CHAIN ADVANTAGE
Bulk Plan – No Logging • Performance improved by 65%!!! • These are simple test scenarios with very small data volume in my
database. In reality, performance improve much more. • Minimize logging!! • NO Detail logs!! • Disable logs for all domains!!
28 INSPIRING VALUE CHAIN ADVANTAGE
Bulk Plan – No Logging • Use Logging Overview to check
29 INSPIRING VALUE CHAIN ADVANTAGE
Common Tuning Problems • Business Number Generator (BNG) contention
– Business number generator assigns serialized business numbers to relevant business objects.
– Single thread – Eliminate unnecessary use of BNG
30 INSPIRING VALUE CHAIN ADVANTAGE
BNG Impact • Created 500 order releases without BNG
– Used Integration layer to control order release keys
31 INSPIRING VALUE CHAIN ADVANTAGE
BNG Impact • Created 500 order releases without BNG
– Start time: 01:17:40 – End time: 01:18:19 – Duration: 29 seconds
32 INSPIRING VALUE CHAIN ADVANTAGE
BNG Impact • Created 500 order releases with BNG
– BNG assigned unique keys to each order
33 INSPIRING VALUE CHAIN ADVANTAGE
BNG Impact • Created 500 order releases with BNG
– Start time: 01:15:44 – End time: 01:16:27 – Duration: 43 seconds – Spent 48% more time!!
34 INSPIRING VALUE CHAIN ADVANTAGE
Common Tuning Problems • Inefficient or high-load SQL statements
– Long parse time – Excessive I/O (physical reads and writes) – Excessive buffer gets – Excessive CPU time – Excessive waits
35 INSPIRING VALUE CHAIN ADVANTAGE
Top SQL Reports
36 INSPIRING VALUE CHAIN ADVANTAGE
Explain Plan
37 INSPIRING VALUE CHAIN ADVANTAGE
Common Tuning Problems • Scalability
– Minimize the size of the object_lock table – Clear as frequently as possible – No need to keep data for more than one day even shorter
38 INSPIRING VALUE CHAIN ADVANTAGE
Common Tuning Problems • Scalability – Object Lock Cleanup
39 INSPIRING VALUE CHAIN ADVANTAGE
Common Tuning Problems • Bulk Plan
– Query phase • Spend extra time to fine tune SQL queries • Keep data volume in mind
– Planning phase • Itinerary, rate offering and rate record need to be as specific as possible • Design them with zero or minimal overlap with each other • Try not use “global” accessorial codes
40 INSPIRING VALUE CHAIN ADVANTAGE
Common Tuning Problems • Shipment Created Phase
– Bulk Plan 500 orders/2 shipment built – No agent listens to shipment created event – Duration: 58 seconds
41 INSPIRING VALUE CHAIN ADVANTAGE
Common Tuning Problems • Bulk Plan 500 orders/2 shipment built
– An agent listens to shipment created event – Duration: 74 seconds – 27% more time
42 INSPIRING VALUE CHAIN ADVANTAGE
Common Tuning Problems • Bulk Plan 500 orders/2 shipment built
– Created a new process – Duration: 65 seconds – 12% more time
43 INSPIRING VALUE CHAIN ADVANTAGE
Common Tuning Problems • Bulk Plan
– Shipment created phase • Agents that are triggered on shipment creation can prolong the bulk plan
significantly!! • Create new process if possible • Potential data lock or contention issue if an order are in multiple shipments or
multi-leg shipments
44 INSPIRING VALUE CHAIN ADVANTAGE
Case Study 2 - Migration • Bulk plan performance issue after OTM upgraded to 6.3
– Before upgrade • 10 minutes for a couple hundred orders
– After upgrade • 3 hrs
45 INSPIRING VALUE CHAIN ADVANTAGE
Case Study 2 - Migration • Cause: high volume of gets and low hit ratios
– glog.cache.REAccessorialCache – glog.cache.RERateGeoCache – glog.cache.TPackagedItemCache – glog.cache.TRateGeoSeqCache
• Cache size issue
46 INSPIRING VALUE CHAIN ADVANTAGE
Common Tuning Problems • Agent
– Avoid one event triggers multiple agents and each agent checks saved conditions
• Data contention: For example, one agent set status and another agent sends integration.
• Use parent-child agent approach
47 INSPIRING VALUE CHAIN ADVANTAGE
Common Tuning Problems • Agent
– Avoid sequential agents • Agent A triggers agent b and agent b triggers agent c. • Consolidate into one agent
– Avoid overuse direct sql update, assign variable and saved query • Consider using PL/SQL to consolidate them
48 INSPIRING VALUE CHAIN ADVANTAGE
Q&A
49 INSPIRING VALUE CHAIN ADVANTAGE