1 copyright © 2012, oracle and/or its affiliates. all rights reserved

42
1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8

Upload: lynhan

Post on 01-Jan-2017

216 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Insert Information Protection Policy Classification from Slide 8

Page 2: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

2 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Insert Information Protection Policy Classification from Slide 8

Triage and Diagnostics for E-Business Suite SystemsDave Suri, Project Lead, Applications PerformanceGustavo Jimenez, Senior Development Manager, ATG

Page 3: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

3 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Agenda

• Architecture• Patches/Certified Browsers/JDK versions• JVM Tuning and Tools• Forms Tools (strace/FRD)• Java Concurrent Program Performance• Case studies

Page 4: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

4 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Architecture Overview

• The applications tier connects to the database via JDBC connections• The application connections are pooled, so a database session is used to

service requests from multiple users throughout the lifetime of the connection

• Java concurrent programs run in their own JVM - each has a special configuration.

• Forms runs in Socket mode (Forms Listener) or Servlet mode (HTTP/s to the JVM)

• Apache HTTP forwards requests to OC4J through mod_oc4j. OC4J routes them to the right oacore JVM while OPMN helps keep the uptime high with death detection.

Page 5: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

5 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Architecture Overview - R12 OA-Framework Flow

Browser

UIX/HTML Page

`

Apache

httpd

2

1

Mod_oc4j

3

Listener

OC4JGroup

5 OC4J:oacore OC4J:oafm

4

7,8

7,87,8

Browser

UIX/HTML Page

`

Apache

httpd

2

1

Mod_oc4j

3

Listener

OC4JGroup

5 OC4J:oacore OC4J:oafm

4

7,8

7,87,8

Page 6: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

6 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Architecture Overview - R12 OA-Framework Flow

B ro w s e r

U IX /H T M L P a g e

O C 4 J G ro u p

`

A p a c h e

h ttp d

2

1

M o d _ o c 4 j

3

L iste ne r

O C 4 J G ro u p

5 O C 4 J :o a c o re O C 4 J :o a fm

4

O C 4 J :o a c o re

A O L /JR F .js p 6

7

8

7 ,8

7 ,87 ,8

F N D /S S OA O L /J

S S O

8

8

9

d a ta b a s e

B ro w s e r

U IX /H T M L P a g e

O C 4 J G ro u p

`

A p a c h e

h ttp d

2

1

M o d _ o c 4 j

3

L iste ne r

O C 4 J G ro u p

5 O C 4 J :o a c o re O C 4 J :o a fm

4

O C 4 J :o a c o re

A O L /JR F .js p 6

7

8

7 ,8

7 ,87 ,8

F N D /S S OA O L /J

S S O

8

8

9

d a ta b a s e

Page 7: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

7 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Architecture Overview - R12 OA-Framework Flow

B ro w s e r

U IX /H T M L P a g e

`

O C 4 J G ro u p

O C 4 J :o a c o re

O A F

O A .js p

U IX B C 4 J M D S 1 2

1 3

1 9 -2 2

1 6 -1 8

1 4 ,1 5

d a ta b a s e

2 3

B ro w s e r

U IX /H T M L P a g e

`

O C 4 J G ro u p

O C 4 J :o a c o re

O A F

O A .js p

U IX B C 4 J M D S 1 2

1 3

1 9 -2 2

1 6 -1 8

1 4 ,1 5

d a ta b a s e

2 3

Page 8: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

8 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Architecture Overview – Forms Servlet Mode• The Forms Listener Servlet is a Java

servlet runs inside the OC4J container– The listener servlet delivers the

ability to run Forms applications over HTTP

– The listener servlet manages the creation of Forms Runtime process (frmweb) for each client and manages any further communications between the client and its designated runtime process.

– The main difference is the traffic goes through HTTP/HTTPS and leverages load balancing/security capabilities of the MT.

– Does not require a special port to be open

Page 9: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

9 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Architecture Overview – Forms Socket Mode• Forms Direct Socket

Connection– Client requests forms application– OHS receives requests & sends

the request to the forms listener– Listener contacts a forms runtime

engine (frmsrv/f60srvm)– frmsrv/f60srvm spawns a runtime

process, creates connection to the database & passes the forms params (menu, sid, form name)

– forms runtime communicates with the database & gets the forms logic (pl/sql,etc).

– forms service sends applet & the response back to client

– forms is running, page is rendered and direct connection to runtime process

Page 10: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

10 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Agenda

• Architecture• Patches/Certified Browsers/JDK versions• JVM Tuning and Tools• Forms Tools (strace/FRD)• Java Concurrent Program Performance• Case studies

Page 11: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

11 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Patches, Browsers & Versions

• Document ID 244040.1 “Recommended Performance Patches for the Oracle E-Business Suite”– Recommended performance patches for all the modules and technology stack components

are consolidated in this note

• Apply latest Release Update Packs– Release 12.1.3 - Patch 9239090

• Document ID 1080973.1 “Oracle E-Business Suite Release 12.1.3 Readme”– Release 12.1 - Patch 7303030

• Document ID 752619.1 “Oracle E-Business Suite Release 12.1.1 Maintenance Pack Installation Instructions”

– Release 12.0 - RUP6/7 Patch 6728000 / 7577660 • Document ID 743368.1 “Oracle E-Business Suite Release 12.1.1 Maintenance Pack

Installation Instructions”

Page 12: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

12 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Patches, Browsers & Versions

• Upgrade to the latest certified technology stack– EBS 12.1 is certified with 11gR2, Oracle Apps Server 10gR3, JRE 6.0. – Document ID

561580.1

• Document ID 380482.1 “Oracle E-Business Suite Release 12 Technology Stack Documentation Roadmap”– Central document for installation, apps, database, security and other resources.

• OA Framework Applications– Recommended Patches for Applications: note 275880.1 (Framework Roadmap)

• Oracle E-Business Suite Recommended Performance Patches– Document id: 244040.1 “Oracle E-Business Suite Recommended Performance Patches”

• For a list of certifications, use Document ID 561580.1 “Oracle E-Business Suite Release 12.1 Release Content Documents”

Page 13: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

13 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Patches, Browsers & Versions

• Browsers, OAF Pages in R12.x– Firefox 3.6 is supported in 11i, R12– Windows 7 / Internet Explorer 8 / Firefox 3.6 supported in 11i/R12– Oracle E-Business Suite 12.1.1 or higher– R12.0 => use R12.ATG_PF.A.DELTA.6 & Forms 10.1.2.3 or higher– 11.5.10 => CU2, 11i.ATG_PF.H Rollup 7 with Document ID 783600.1– Document ID 1103276.1, 389422.1 (R12), 285218.1 (11i)

• R12.1.2 have ajax customizable homepage for increased performance• R12.1.3 available as patch #9239090

Page 14: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

14 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Agenda

• Architecture• Patches/Certified Browsers/JDK versions• JVM Tuning and Tools• Forms Tools (strace/FRD)• Java Concurrent Program Performance• Case studies

Page 15: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

15 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools

• Response Time/JVM Usage• OutOfMemoryErrors• Deadlocks• Some great tools

– Jstat : monitor JVM– Eclipse MAT : analyze heap dumps– IBM Thread Dump Analyzer : analyze thread dumps– Visual VM : profile, monitor, dump the JVM– Jconsole : similar to Visual VM except for profiling– Jrockit Mission Control : flight recorder, profiler, dump– Thread Dump Analyzer

Page 16: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

16 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and ToolsResponse Time/JVM Usage• First, rule out any SQL issues.

– Document ID 357597.1 “How To Generate A SQL Trace In OA Framework For Oracle Applications”

• Monitor Request Response Time– Modify $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf

• LogFormat "%h %T

• Logs: $LOG_HOME/ora/10.1.3/Apache/access_log*

• If there are no database-related issues, take a look into the JVM– Techniques you can use

• Thread dumps – Send to Oracle Support for analysis• GC logs for sizing, Profilers - Send to Oracle Support for analysis

Page 17: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

17 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and ToolsResponse Time/JVM Usage• GC tuning

– Review the frequency of collections, especially major collections (i.e. Full GC) in the logs for healthy and unhealthy activity

• Use jstat tool with the –gc option to review FCGT count• Important columns to pay attention to are survivor spaces, eden

and old gen– Jstat column details (observe the grey highlighted columns)

• Timestamp, S0C, S1C, S0U, S1U, EC, EU, OC, OU, PC, PU, YGC, YGCT, FGC, FGCT, GCT

• 933.9,59904.0,57728.0,0.0,40504.5,230784.0,0.0,699072.0, 666307.1,131072.0,119088.8,31,2.054,1,0.000,2.054

Page 18: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

18 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and ToolsResponse Time/JVM Usage• GC tuning

– Bigger heaps => Full GC will take longer– Longer GCs => users may experience pauses– Increase the number of JVMs used may provide the ability to handle

more users with faster GC times.• Each JVM has a smaller Xmx,Xms memory footprint so GCs will

be faster!– Also try to change the JVM collectors used and track the performance

with Jconsole– Enable the ParallelGC collector with ParallelGCThreads = 2

Page 19: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

19 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and ToolsResponse Time/JVM Usage• GC tuning

– Unhealthy JVM with FullGC activity• 898753.288: [Full GC 825983K->743065K(913024K), 15.8442715 secs]

898772.243: [Full GC 825983K->750548K(913024K), 13.3512933 secs] 898798.124: [Full GC 825983K->731902K(913024K), 13.3466676 secs] 898832.082: [Full GC 825983K->738500K(913024K), 13.1591655 secs] 898857.239: [Full GC 825973K->728250K(913024K), 16.4536985 secs]

– Healthy JVM with FullGC activity• 902130.499: [GC 829181K->716211K(935296K), 0.2895464 secs]

902220.498: [GC 833715K->721542K(935296K), 0.3210096 secs] 902400.049: [GC 839046K->725265K(936768K), 0.3448279 secs] 902400.394: [Full GC 725265K->652064K(936768K), 8.5214844 secs] 902455.778: [GC 771040K->661918K(935296K), 0.1618026 secs]

Page 20: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

20 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools

• GC tuning– Unhealthy JVM with FullGC activity

• 898753.288: [Full GC 825983K->743065K(913024K), 15.8442715 secs] 898772.243: [Full GC 825983K->750548K(913024K), 13.3512933 secs] 898798.124: [Full GC 825983K->731902K(913024K), 13.3466676 secs] 898832.082: [Full GC 825983K->738500K(913024K), 13.1591655 secs] 898857.239: [Full GC 825973K->728250K(913024K), 16.4536985 secs]

– Healthy JVM with FullGC activity• 902130.499: [GC 829181K->716211K(935296K), 0.2895464 secs]

902220.498: [GC 833715K->721542K(935296K), 0.3210096 secs] 902400.049: [GC 839046K->725265K(936768K), 0.3448279 secs] 902400.394: [Full GC 725265K->652064K(936768K), 8.5214844 secs] 902455.778: [GC 771040K->661918K(935296K), 0.1618026 secs]

Full GC too frequent!!!

Infrequent and shorter times!

Response Time/JVM Usage

Page 21: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

21 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and ToolsOut of Memory Errors• Common causes of OutOfMemoryError

– Sizing Issues – Heap is undersized for expected load– Memory Leak – Some objects not cleaned up

• Strong reached references, connection objects, etc.– Memory Hemorrhage

• Reading large data sets consuming vast amounts of memory in any period of time

• Connection leaks leading to vast memory/stack allocations• SQL Statements/caching may lead to excess memory used in

JDBC libraries

Page 22: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

22 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and ToolsOut of Memory Errors• Out of Memory Heap Dump Example

– This will be present in the $LOG_HOME/ora/10.1.3/opmn/default_group~oacore~default_group~1.log

• 10067.492: [Full GC 118783K->118783K(124928K), 1.2127460 secs]

• 10068.706: [Full GC 118783K->118783K(124928K), 1.2166770 secs]

• 10069.924: [Full GC 118783K->117977K(124928K), 1.2416330 secs]

• java.lang.OutOfMemoryError: GC overhead limit exceeded

• Dumping heap to java_pid8515.hprof ...

• Heap dump file created [149076109 bytes in 10.240 secs]

• Heap Location

cd $INST_TOP/../../../apps/tech_st/10.1.3/j2ee/home

ls -l *hprof

– -rw------- 1 oraperf dba 149076109 Jan 26 11:11 java_pid8515.hprof

Page 23: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

23 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools

Page 24: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

24 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and ToolsJVM Tools• JDK 6 – Jconsole

– Enable heap dumps to be collected from the attached JVM– Goto java.sun.management | HotSpotDiagnostic | Operations |

dumpHeap• P0 : absolute file name (directory + filename)• P1: leave to true so that only reachable objects are dumped

• Use any binary hprof heap analyzer to analyze the heap dump and see which objects take up the most space

• JDK 6u7 and above has VisualVM– Does thread/heap dumps, profiling, graphs

Page 25: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

25 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and ToolsJVM Tools• Jstat

– Shows the JVM heap internals– Shows Timestamps and Capacity/Utilization for

• Survivor Spaces – where live objects become old enough to copy

• Eden – new objects are created here• Old Generation – long lived objects are kept here• Young Generation – composed of eden + 2 survivor spaces• Perm Generation – native heap and other non-heap objects

– Shows the Full GC and Young GC collection count– To get the data from PID nnnnn every 60s for 2 hours

• jstat -gc -t nnnnn 60000 120 > jstat_gcutil_pid_31041 &

Page 26: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

26 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

JVM Tuning and Tools

Page 27: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

27 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Agenda

• Architecture• Patches/Certified Browsers/JDK versions• JVM Tuning and Tools• Forms Tools (strace/FRD)• Java Concurrent Program Performance• Case studies

Page 28: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

28 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Forms Tools (strace/FRD)

• Forms– Next, use strace (linux), trace (solaris) to attach to the pid:

• strace -o formsR12_frmweb.log -p 22874• strace -o formsR12_frmsrv.log -p 17347

– The logs will show the activity of the forms server as well as the listener while the user is performing their actions.

– FRD, Tracing, Logging for Forms• Document ID 438652.1• FRD gives clues into forms code execution, parameters,

variables, item values– Along with strace, FRD can provide valuable clues into researching

issues.

Page 29: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

29 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Agenda

• Architecture• Patches/Certified Browsers/JDK versions• JVM Tuning and Tools• Forms Tools (strace/FRD)• Java Concurrent Program Performance• Case studies

Page 30: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

30 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Java Concurrent Program Performance

• Observe the concurrent program log file after adding –verbose:gc to see if the JVM is healthy or is it suffering from low memory

• Increase the Memory when you encounter OOME in the logs– location: Concurrent -> Program -> Define

• Remember it runs in a separate JVM from oacore• All the regular java support utilities will work (jstat, eclipse mat, tda, etc)• Can specify regular JVM commands in the define field to observe the JCP

behaviour during the run

Page 31: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

31 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Agenda

• Architecture• Patches/Certified Browsers/JDK versions• JVM Tuning and Tools• Forms Tools (strace/FRD)• Java Concurrent Program Performance• Case studies

Page 32: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

32 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Case Studies

• Issue– Slow Performance with Rules Enabled vs Rules Disabled

• Tool Used– IBM Thread Dump Analyzer

• Analyzes thread dumps• Can compare multiple thread dumps• Shows different thread states with different colors• Primary area of concentration is : AJPRequestHandler-

HTTPThreadGroup-*• These thread groups are the Apache/AJP Requests coming to oacore

• Solution– Thread dumps showed JVM waiting for DB to send results. This eventually lead to the creation

of a cache to improve Rules Enabled performance

Oracle Advanced Product Catalog

Page 33: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

33 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Case Studies

Page 34: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

34 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Case Studies

• Issue– Error saving a service request

• Tools Used– Forms FRD

• FRD showed a NULL being passed to a variable named ‘control.mfg_org_id’– Executing USER_EXIT Built-in: – In Argument 0 - Type: String Value: FND FFLEX PRE-UPDATE CSXSRISR, 17,

Built-In, Entry, -1424992664, USER_EXIT – Analyzed the CSXSRISR_LIBRARY.Init() method to find the NULL– Analyzed the SR_CONTACTS.PRE_UPDATE to find how the NULL was passed

to the control.mfg_org_id

• Solution– Product team notified of the issue and a test case was given to isolate the affected code

Saving a Oracle Service Request

Page 35: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

35 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Case Studies

• Issue– Out of Memory error experienced when running a Java Concurrent Program

• Tool Used– Eclipse MAT

• Analyzed heap dump • Showed which package consumes significant percentage of the heap• Helps derive clues and questions as to what is happening in the tech

stack layers• Can see what objects are occupying space and their values

• Solution– The Java Concurrent Program’s max JVM memory was increased to 1GB to allow the program

to finish successfully

Java Concurrent Program

Page 36: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

36 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Case Studies

Page 37: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

37 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Case Studies

Page 38: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

38 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Case Studies

Page 39: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

39 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Patches 244040.1 Oracle E-Business Suite Recommended Performance Patches

561580.1 Oracle E-Business Suite Release 12.1 Release Content Documents

Roadmap 380482.1 Oracle E-Business Suite Release 12 Technology Stack Documentation Roadmap

275880.1 Oracle Application Framework Release 11i Documentation Road Map

Browsers 1103276.1 Is Windows 7 Certified With Oracle E-Business Suite?

389422.1 Recommended Browsers for Oracle E-Business Suite Release 12

285218.1 Recommended Browsers for Oracle E-Business Suite 11i

Setup 362851.1 Guidelines to setup the JVM in Apps E-Business Suite 11i and R12

Diagnosis 357597.1 How To Generate A SQL Trace In OA Framework For Oracle Applications

1160285.1 R12: Forms Runtime Diagnostics (FRD), Tracing And Logging For Forms In Oracle Applications

ATG 438652.1 Application Technology Group (ATG) Product Information Center

Useful Documents

Page 40: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

40 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Q&A

Page 41: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

41 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Page 42: 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved

42 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Insert Information Protection Policy Classification from Slide 8