peoplesoft instrumentation for oracle rdbms david kurtz go-faster consultancy ltd....
TRANSCRIPT
PeopleSoft Instrumentation for Oracle RDBMS
David KurtzGo-Faster Consultancy Ltd.
www.go-faster.co.uk
Who Am I?
• Oracle Database Specialist– Independent consultant
• Performance tuning– PeopleSoft ERP– Oracle RDBMS
• Book– www.psftdba.com
• OakTable•
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 2
Instrumentation
Data! Data! Data! I can't make bricks without clay.
The Adventures of Sherlock Holmes, Arthur Conan-Doyle
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 3
Instrumentation
• PeopleTools– SQL Trace– Batch Timings– Performance Monitor
• Oracle– Dynamic Performance
Views (V$)– SQL Trace– Automatic Workload
Repository (AWR)– Active Session History
(ASH)
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 4
©2013 www.go-faster.co.uk 5
ASH: A Brief Overview
• Samples active sessions every second• Circular buffer in memory
– v$active_Session_history– It should hold about 1 hour of data
• 1 in 10 samples stored in database– DBA_HIST_ACTIVE_SESS_HISTORY– Flushed out during AWR snapshot
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 7
What does ASH retain?
• Most of the columns are on v$session– Session
• Session ID and serial, query coordinator– Wait
• event id, name and parameters– SQL
• SQL_ID, plan hash, opcode, plan line– Object
• object, file, block and row numbers– Application
• module, action …
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 8
ASH –v- SQL*Trace
• ASH– Licensed (Diagnostics)– Always there– Real Time– No marginal overhead– Who is blocking me?– Statistical data– SQL ID– Plan if captured by AWR– Estimate of duration
• Per wait event• Per Plan Operation 11g
• SQL*Trace– Free– Enable, File, Profile– Reactive– Run-time overhead– Being Blocked– Every SQL & event– SQL HASH– Actual execution plan– Exact duration
• Operations in Plan
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 9
ASH –v- SQL*Trace
• ASH can be used to resolve many of your performance issues.
• Sometimes, you will still need SQL*Trace
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 10
Application Instrumentation
• It is essential to be able to match – database sessions– application processes
• DBMS_APPLICATION_INFO– set_module, set_action– Calls in application
• Introduced in PeopleTools 8.50-8.52.– Appears in SQL*Trace and ASH
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
CLIENT_INFO
– PeopleTools 7.53– Every application server service calls
• DBMS_APPLICATION_INFO.SET_CLIENT_INFO• V$SESSION.CLIENT_INFO• Operator ID, …
– Inserts into Audit tables can be done by database trigger.• Trigger uses READ_CLIENT_INFO to obtain PeopleSoft
OPRID
– Not copied to ASHUKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 11
CLIENT_ID
• PeopleTools 8.52• Also set to PeopleSoft OPRID.• Client_ID is copied to ASH
– So you can identify the PS operator
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 12
PIA Instrumentation
• PeopleTools 8.50
• Module = Component• Action = Page
– Or XYZZY for search dialogue
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 13
IB Instrumentation
• PeopleTools 8.50
• Module = Service• Action = Queue
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 14
Application Engine
• PeopleTools 8.50
• Module = ‘PSAE’• Action = Program
– This is not good– See http://
blog.psftdba.com/2009/03/using-oracle-enterprise-manager-grid.html
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 15
Application Engine
• PeopleTools 8.52
• Module = ‘PSAE.’||AE program name||session id– session id is stored in
PSPRCSQUE.SESSIONID_NUM
• Action = Program.Section.Step.Actions
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 16
Cobol / SQR
• No instrumentation• DIY with trigger
– on Process Scheduler request table– http://
www.go-faster.co.uk/scripts.htm#psftapi.sql
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 17
OEM
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 18
©2013 www.go-faster.co.uk 19UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
ASH Report in OEM
• You can run ASH reports via EM– This example is for a specific module
Enable Trace by Module/Action
• Like setting a watchpoint in a debugger.• See http://
blog.psftdba.com/2012/09/oracle-sql-tracing-by-module-action.html
• Can set SQL trace for a specific component/page.
• But you can’t predict the value of module for AE so may need to go back to the trigger based solution.UKOUG Apps 2013 - PeopleSoft
Instrumentation for Oracle©2013 www.go-faster.co.uk 20
Ability to Profile SQL duration by Process/Process Instance
PROGRAM MODULE ASH_SECS-------------------- -------------------------------- ----------PSQRYSRV PSQRYSRV 633520oracle oracle 328630rman backup archivelog 236050PSAESRV GL_JEDIT 203120rman backup incr datafile 132840FSPCCURR FSPCCURR 119350PSAPPSRV VCHR_EXPRESS 111520... ----------sum 2982340
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 21
Profile of Whole System
WITH x AS (SELECT /*+leading(x h) use_nl(h)*/ CASE WHEN h.module IS NULL THEN REGEXP_SUBSTR(h.program, '[^@]+',1,1) WHEN h.module LIKE 'PSAE.%' THEN REGEXP_SUBSTR(h.module, '[^.]+',1,2) WHEN h.module LIKE '%.%' THEN REGEXP_SUBSTR(h.module, '[^.]+',1,1) WHEN h.module LIKE '%@%' THEN REGEXP_SUBSTR(h.module, '[^@]+',1,1) ELSE h.module END AS module, REGEXP_SUBSTR(h.program, '[^.@]+',1,1) program, h.eventFROM dba_hist_snapshot x, dba_hist_active_sess_history hWHERE h.SNAP_id = X.SNAP_idAND h.dbid = x.dbidAND h.instance_number = x.instance_numberAND x.end_interval_time >= TO_DATE('20130730', 'YYYYMMDD')AND x.begin_interval_time <= TO_DATE('20130807', 'YYYYMMDD')AND h.sample_time BETWEEN TO_DATE('20130730','YYYYMMDD') AND TO_DATE('20130807','YYYYMMDD'))SELECT program, module, sum(10) ash_SecsFROM x GROUP BY program, moduleORDER BY ash_secs DESC
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 22
ASH Data for period of interest
Snapshots for Period of interest
Logic to trim module name at @ or .
Batch Timings by Exec Time
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 23
ASH Profile a Single Process
Process SQL Plan Exec Instance SQL_ID Hash Value ACTION secs ASH_SECS--------- ------------- ---------- -------------------------------- ----- ---------- 51159995 fpj0dmac7rw20 207497280 AR_POSTING.HSS_PYMT.ITEMS.S 2797 410 51159995 2qnfa4afzvj91 416825695 AR_POSTING.HS_PYMNT.ITEMS2.S 2797 320 51159995 1u37a5hzzc2z6 768682620 AR_POSTING.HS_PYMNT.ITEMS.S 2797 270 51159995 grhj2f0dcbd70 2964999827 AR_POSTING.HSS_DSO.BACK_OUT.S 2797 190 51159995 a6nrbg7sbx401 721249526 AR_POSTING.HSS_PYMT.UPD_STAT.S 2797 170 51159995 49yzw3xa50r1t 2665694782 AR_POSTING.HS_USER.INSERTS.S 2797 160 51159995 0wcx4syn7tjk2 1678621177 AR_POSTING.HS_USER.UPDATES.S 2797 150 51159995 44sjax92d6kqa 3752318859 AR_POSTING.HS_DSO.ADJ_BAL.S 2797 130 51159995 cm5xjajkfjvkg 3814124734 AR_POSTING.HSS_USR1.INSERTS.S 2797 130 51159995 djur79tkd6uum 2504040148 AR_POSTING.HS_USER.UPD_ACTV.S 2797 130 51159995 cb1cankq47ky9 362952569 AR_POSTING.HSS_USR1.UPDATES.S 2797 130 51159995 1awa6cr95ug3t 3137593480 AR_POSTING.HS_PYMNT.UPD_STAT.S 2797 110 51159995 03p8ga7kazdg4 2504040148 AR_POSTING.HSS_USER.UPD_ACTV.S 2797 100 51159995 5yrur2nv3nn2p 2663443920 AR_POSTING.HSS_DSO.NO_ACTIV.S 2797 80
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 24
How profile a process (i)
WITH x AS ( SELECT /*+LEADING(r q x h)*/ h.sql_id, h.sql_plan_hash_value, h.action , SUM(10) ash_secs , 10*count(t.sql_id) awr_secs FROM sysadm.psprcsrqst r , sysadm.psprcsque q , dbA_hist_snapshot x , dba_hist_active_Sess_history h LEFT OUTER JOIN dba_hist_sqltext t ON t.sql_id = h.sql_id WHERE x.END_INTERVAL_TIME >= r.begindttm AND x.begin_interval_time <= r.enddttm AND h.sample_time BETWEEN r.begindttm AND r.enddttm AND h.SNAP_id = X.SNAP_id AND h.dbid = x.dbid AND h.instance_number = x.instance_number AND q.prcsinstance = r.prcsinstance AND h.module = 'PSAE.'||r.prcsname||'.'||q.sessionidnum AND r.prcsname = 'AR_UPDATE2' AND r.prcsinstance = 51159995 GROUP BY h.sql_id, h.sql_plan_hash_value, h.action) …
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 25
Process Request TableOne Row per Process
Process Request Queue TableOne Row per Process
Specific Process Request
ASH data for period for which process was running
Hint to guarantee sensible join order, and partition
elimination
Filter ASH data by module
How profile a process (ii)
…), y AS ( SELECT ROW_NUMBER() OVER (PARTITION BY x.sql_plan_hash_value order by x.awr_secs desc) as ranking , x.sql_id, x.sql_plan_hash_value, x.action , SUM(x.ash_secs) OVER (PARTITION BY x.sql_plan_hash_value) tot_ash_secs , SUM(x.awr_secs) OVER (PARTITION BY x.sql_plan_hash_value) tot_awr_secs , COUNT(distinct sql_id) OVER (PARTITION BY x.sql_plan_hash_value) sql_ids FROM x)Select * from ywhere y.ranking = 1AND tot_ash_secs > 30ORDER BY tot_ash_secs DESC/
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 26
Ranking by SQL Plan Hash Value
DB Time for Execution Plan
AWR secs 0 if SQL not captured in ASH repository
Profile a Single Process
Process SQL Plan Exec Instance SQL_ID Hash Value ACTION secs ASH_SECS--------- ------------- ---------- -------------------------------- ----- ---------- 51159995 fpj0dmac7rw20 207497280 AR_POSTING.HSS_PYMT.ITEMS.S 2797 410 51159995 2qnfa4afzvj91 416825695 AR_POSTING.HS_PYMNT.ITEMS2.S 2797 320 51159995 1u37a5hzzc2z6 768682620 AR_POSTING.HS_PYMNT.ITEMS.S 2797 270 51159995 grhj2f0dcbd70 2964999827 AR_POSTING.HSS_DSO.BACK_OUT.S 2797 190 51159995 a6nrbg7sbx401 721249526 AR_POSTING.HSS_PYMT.UPD_STAT.S 2797 170 51159995 49yzw3xa50r1t 2665694782 AR_POSTING.HS_USER.INSERTS.S 2797 160 51159995 0wcx4syn7tjk2 1678621177 AR_POSTING.HS_USER.UPDATES.S 2797 150 51159995 44sjax92d6kqa 3752318859 AR_POSTING.HS_DSO.ADJ_BAL.S 2797 130 51159995 cm5xjajkfjvkg 3814124734 AR_POSTING.HSS_USR1.INSERTS.S 2797 130 51159995 djur79tkd6uum 2504040148 AR_POSTING.HS_USER.UPD_ACTV.S 2797 130 51159995 cb1cankq47ky9 362952569 AR_POSTING.HSS_USR1.UPDATES.S 2797 130 51159995 1awa6cr95ug3t 3137593480 AR_POSTING.HS_PYMNT.UPD_STAT.S 2797 110 51159995 03p8ga7kazdg4 2504040148 AR_POSTING.HSS_USER.UPD_ACTV.S 2797 100 51159995 5yrur2nv3nn2p 2663443920 AR_POSTING.HSS_DSO.NO_ACTIV.S 2797 80
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 27
Now investigate the SQL
• Extract SQL from AWR repository with the dbms_xplan package– Obtaining and Interpreting Execution Plans usi
ng DBMS_XPLAN • http://
www.go-faster.co.uk/ukougpres.htm#Intro_DBMS_XPLAN
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 28
SELECT * FROM table( dbms_xplan.display_awr( 'duu7n4uzsthxz',
768682620,NULL,'ADVANCED'));
SQL_ID duu7n4uzsthxz
--------------------
INSERT INTO PS_HS_CLSITM_TMP4 (…)
SELECT 51164249 , I.BUSINESS_UNIT ,…
FROM PS_ITEM I
,PS_ITEM_ACTIVITY A
,PS_POST_REQ_TMP4 R
WHERE R.PROCESS_INSTANCE = 51164249
AND R.PROCESS_P_HIST ='Y'
--
AND I.BUSINESS_UNIT = R.BUSINESS_UNIT
AND I.ITEM_STATUS = 'C'
AND I.HIST_STATUS = 'N'
--
AND A.BUSINESS_UNIT = I.BUSINESS_UNIT
AND A.CUST_ID = I.CUST_ID
AND A.ITEM = I.ITEM
AND A.ITEM_LINE = I.ITEM_LINE
AND A.ENTRY_TYPE = I.ENTRY_TYPE
AND A.ENTRY_REASON = I.ENTRY_REASON
AND A.ITEM_SEQ_NUM = (
SELECT MIN(ITEM_SEQ_NUM)
FROM PS_ITEM_ACTIVITY X
WHERE X.BUSINESS_UNIT = I.BUSINESS_UNIT
AND X.CUST_ID = I.CUST_ID
AND X.ITEM = I.ITEM
AND X.ITEM_LINE = I.ITEM_LINE
AND X.ENTRY_TYPE = I.ENTRY_TYPE
AND X.ENTRY_REASON = I.ENTRY_REASON)
AND A.ENTRY_AMT_BASE > 0
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 29
Execution Plan
Plan hash value: 768682620 -----------------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |-----------------------------------------------------------------------------------------------------------| 0 | INSERT STATEMENT | | | | | 222K(100)| || 1 | FILTER | | | | | | || 2 | HASH JOIN | | 23719 | 3613K| | 111K (1)| 00:22:19 || 3 | TABLE ACCESS BY INDEX ROWID | PS_POST_REQ_TMP4 | 8 | 112 | | 2 (0)| 00:00:01 || 4 | INDEX RANGE SCAN | PS_POST_REQ_TMP4 | 8 | | | 1 (0)| 00:00:01 || 5 | HASH JOIN | | 23719 | 3289K| 11M| 111K (1)| 00:22:19 || 6 | TABLE ACCESS BY INDEX ROWID| PS_ITEM | 118K| 9944K| | 30295 (1)| 00:06:04 || 7 | INDEX RANGE SCAN | PSKITEM | 280K| | | 812 (1)| 00:00:10 || 8 | TABLE ACCESS FULL | PS_ITEM_ACTIVITY | 3480K| 185M| | 67704 (1)| 00:13:33 || 9 | SORT AGGREGATE | | 1 | 42 | | | || 10 | TABLE ACCESS BY INDEX ROWID | PS_ITEM_ACTIVITY | 1 | 42 | | 5 (0)| 00:00:01 || 11 | INDEX RANGE SCAN | PS_ITEM_ACTIVITY | 1 | | | 4 (0)| 00:00:01 |-----------------------------------------------------------------------------------------------------------
Query Block Name / Object Alias (identified by operation id):-------------------------------------------------------------
1 - SEL$1 3 - SEL$1 / R@SEL$1 4 - SEL$1 / R@SEL$1 6 - SEL$1 / I@SEL$1 7 - SEL$1 / I@SEL$1 8 - SEL$1 / A@SEL$1 9 - SEL$2 10 - SEL$2 / X@SEL$2 11 - SEL$2 / X@SEL$2
Outline Data-------------
/*+ BEGIN_OUTLINE_DATA IGNORE_OPTIM_EMBEDDED_HINTS OPTIMIZER_FEATURES_ENABLE('10.2.0.4') OPT_PARAM('_unnest_subquery' 'false') OPT_PARAM('optimizer_dynamic_sampling' 4) OPT_PARAM('_optimizer_cost_based_transformation' 'off') ALL_ROWS OUTLINE_LEAF(@"SEL$2") OUTLINE_LEAF(@"SEL$1") OUTLINE_LEAF(@"INS$1") FULL(@"INS$1" "PS_HS_CLSITM_TMP4"@"INS$1") INDEX_RS_ASC(@"SEL$1" "I"@"SEL$1" ("PS_ITEM"."HIST_STATUS")) FULL(@"SEL$1" "A"@"SEL$1") INDEX_RS_ASC(@"SEL$1" "R"@"SEL$1" ("PS_POST_REQ_TMP4"."PROCESS_INSTANCE" "PS_POST_REQ_TMP4"."BUSINESS_UNIT")) LEADING(@"SEL$1" "I"@"SEL$1" "A"@"SEL$1" "R"@"SEL$1") USE_HASH(@"SEL$1" "A"@"SEL$1") USE_HASH(@"SEL$1" "R"@"SEL$1") SWAP_JOIN_INPUTS(@"SEL$1" "R"@"SEL$1") INDEX_RS_ASC(@"SEL$2" "X"@"SEL$2" ("PS_ITEM_ACTIVITY"."BUSINESS_UNIT" "PS_ITEM_ACTIVITY"."CUST_ID" "PS_ITEM_ACTIVITY"."ITEM" "PS_ITEM_ACTIVITY"."ITEM_LINE" "PS_ITEM_ACTIVITY"."ITEM_SEQ_NUM")) END_OUTLINE_DATA */
Note----- - dynamic sampling used for this statement
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 30
Limitations
• SQL may not be captured by AWR– SQL inside a loop– With bind variables – Without ReUseStatement
• (binds become literals)
• Increase AWR snapshot frequency– 15 minutes instead of an hour– Top N SQL threshold
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 31
Conclusion
• Instrumentation permits you to profile DB performance with ASH
• Proactively address performance issues
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 32
QUESTIONS?Nullius in verba
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 33
You know my methods,
apply them.A Study in Scarlet, Arthur Conan-Doyle
UKOUG Apps 2013 - PeopleSoft Instrumentation for Oracle
©2013 www.go-faster.co.uk 34