db2 accounting reporting

Understanding DB2 for z/OS Accounting Information

John Campbell & Florence DuboisIBM DB2 for z/OS Development

© 2012 IBM Corporation

[email protected]@uk.ibm.com

Acknowledgements

• Many thanks to all people who contributed to this presentation, in particular:

– Namik Hrle

– Akira Shibamiya

– John Tobler

– Miao Zheng


Agenda

• DB2 traces: sources of key performance indicators

• Where is a DB2 transaction spending its time?

• CPU vs. suspension time

• Most common reasons for large suspension times

• SQL / Locking / BP / GBP accounting counters

• Tuning hints and tips as we go along


•

DB2 Accounting vs. Statistics Data

• DB2 statistics trace records

– Contain information about the activity of the entire DB2 subsystem (including the DB2 system address spaces, such as db2MSTR, db2DBM1, db2IRLM, db2DIST

• DB2 accounting trace records

– Contain information about the activity performed by an individual DB2 thread or ‘transaction’ (or x threads in case of roll-up accounting)


(or x threads in case of roll-up accounting)

– The focus of this presentations is here

Synchronous/ Asynchronous Agents

• Main synchronous agent

– Used to execute the SQL statement

– Usually a TCB that originates in the user’s address space

– In case of DDF, it is an (independent enclave) SRB instead of a TCB, which originates in the DDF address space

– In case of parallelism, the TCB (SRB for DDF) is helped by synchronous SRBs

• Asynchronous agent


• Asynchronous agent

– Normally performs system or ancillary functions

– Usually an SRB, but it can also be a TCB

– The asynchronous CPU load is typically 10% of the synchronous load in most systems

• Can be higher in DDF and/or data sharing

• More important to see if the ratio changes than looking at absolute numbers

• Can indicate a change in workload -Access path change - Maintenance causing regression

TSO attachCall attachUtilities attach (*)

RRS attach (incl. JCC T2)IMS attachCICS attach

Use TCB

Synchronous Agent

Different DB2 Attachments

• Different attachments used by DB2


CICS attach

DDF “attach”

All use XMEM services

Uses SRB

(*) Except with zIIP offload during IX build phase and RUNSTATS (V10)

Synchronous Agent

TCB

SQL

How Work Enters DB2 – Allied Thread


USER (allied) DBM1 MSTR IRLM

SRB

SQL

How Work Enters DB2 – Distributed Work


DIST (distributed)

DBM1 MSTR IRLM

TCBAsync I/O Physical

loggerDeadlock detection

SQL

Asynchronous ’Agents’ in DB2


USER (allied)

DBM1 MSTR IRLM

Systemcheckpoint Lock

resumeOpen/close

DIST (distributed)

SRB

SQL

Auth

Synchronous/ Asynchronous Agents CPU times

• The synchronous CPU load is recorded in accounting

• The asynchronous CPU load is recorded in statistics

– If a thread has to wait for the async. agent to complete its work, this elapsed time is typically recorded in an accounting class 3 wait time counter

• Important note


• Important note

– Since DDF work doesn’t have an agent outside the DB2 address spaces to charge for the work, the CPU time used by synchronous agents for DDF work also shows up in DB2 statistics (as DDF enclave SRB time). So be aware of double accounting!

CPU TIMES (from Statistics ) v10 TCB TIME PREEMPT SRB NONPREEMPT SRB TOTAL TIME PREEMPT IIP SRB

------------------------------- -------------- ------ --------- --------------- --------------- ------------ ---

SYSTEM SERVICES ADDRESS SPACE 8:05.013248 21:20.445149 12.782106 29:38.240503 N/A

DATABASE SERVICES ADDRESS SPACE 21:51.008289 13:35.088803 16.280806 35:42.377897 3 :19:20.488131

IRLM 0.320224 0.000000 22.664271 22.984495 N/A

DDF ADDRESS SPACE 2:12.462002 2:50:01.556367 2:15.574735 2:54:29.593103 2:37:50.897412

TOTAL 32:08.803762 3 :24:57.090319 3:07.301917 4:00:13.195998 5 :57:11.385543

TCBAsync I/O Physical

loggerDeadlock detection

(From Accounting )

Sync and Async Work in DB2 Stats and Accounting


USER (allied)

DBM1 MSTR IRLM

logger

Systemcheckpoint

detection

Lock resumeOpen/

close

DIST

SRB

SQL

SQL

Auth

AVERAGE APPL(CL.1) DB2 (CL.2)

------------ ---------- ----------

ELAPSED TIME 0.225604 0.099221

NONNESTED 0.220121 0.097390

STORED PROC 0.005483 0.001831

CP CPU TIME 0.002241 0.002066

AGENT 0.002241 0.002066

NONNESTED 0.001787 0.001699

STORED PRC 0.000454 0.000366

IIP CPU TIME 0.002082 0.001987

STORED PROC N/A N/A

How CPU Time is Reported (Acctg vs. Stats)

Application DBM1 MSTR IRLM

Accounting Statistics

TSQL processing Scans

Synchronous I/O Lock requests

Open, Close

Extend

ArchivingError checkingAllocate archive dsn


TCB

SRB

Synchronous I/O

Global lock requests*

Buffer updates

Lock requests

Logical logging

GBP reads*

The same as in the TCB case, but only in enclave pre-emptible SRB mode.

Reported in TCB instrumentation.

Extend

Preformat

Asynchronous I/OMemory managementReal-time StatisticsCastout*P-lock negotiation*Prefetch in CF*SYSLGRNX updates*GBP checkpoints*

BSDS processing

Physical loggingCheckpointsBackoutsDeallocationUpdate commit

GBP writes at commit*Global unlock at commit*

Error checking

Management

Deadlock detection

Lock resume

Global lock conflictresolution*

(*) data sharing specific

Allocate archive dsn

Response Time Factors

presentation network application logic network database processing


General transaction flow

Response Time Scope

TRANSACTION

PROGRAM A

STATEMENT 1STATEMENT 2

.

.

PLAN

DBRM A (*)


.

.


PROGRAM B

.


.

.

PACKAGE B

.


.

.

Application space DB2 (*) No longer possible in V10

Response Time: DB2 View

thread allocation

1st SQL

2nd SQL

DB2 can collect data during the entire thread life: activity time

It includes time spent in:

• Presentation layer (GUI)• Application logic• Network


nth SQL

thread deallocation

Out of DB2 In DB2

For all practical purposes, this is the DB2 Accounting Class 1 elapsed time

• Network• DB2 processing

activitytime

DB2 Accounting Data

• DB2 Accounting trace

– DB2 command -STA TRA(ACCTG) …. Or at DB2 start-up SMFACCT DSNZPARM

– Typically written to SMF (as SMF type 101 record)

• IFCIDs

– IFCID 3 – plan level info

– IFCID 239 – package level info

• Since V8, IFCID 239 is written as a separate record, but cut at the same time


• Since V8, IFCID 239 is written as a separate record, but cut at the same time

• Accounting Trace classes

– Class 1 – Total time (elapsed and CPU) – IFCID 3

– Class 2 – Time in DB2 (elapsed and CPU) – adds info to IFCID3

– Class 3 – Suspension time in DB2 – adds info to IFCID 3

– Class 7 – Package/DBRM info (similar to class 2)

– Class 8 – Package/DBRM info (similar to class 3)

– Class 10 – SQL/Locking/BP info at package/DBRM level

A DB2 Accounting Data is Written at:

• Thread deallocation (including abends)

• (re)Signon in case of thread reuse by RRS, IMS, CICS

– (many options in CICS)

• Commit for RRS threads using accounting-interval COMMIT on signon / authsignon / context signon

• Commit when CMTSTAT=INACTIVE and provided connection can go inactive

– Things that prevent a connection from going inactive:


– Things that prevent a connection from going inactive:

• Touched a package that uses KEEPDYNAMIC(YES)

• Active DGTT (not explicitly or implicitly dropped)

• Open cursor with hold

• Held LOB locator

• If only reason for not going inactive is the use of KEEPDYNAMIC(YES), still cut accounting record (and reset enclave)

– To please our SAP friends

DB2 Accounting Data is Written at: -2

• When using rollup accounting for DRDA and RRS

– Every ACCUMACC times you would have normally written an accounting record

– If gets stale (not enough occurrences of the rollup “group”)

– If rollup blocks are using too much storage

– QWACRINV indicates reason why acctg record was written

• Rollup accounting blessing and curse

– Reduces the number of accounting records


– Reduces the number of accounting records

– Can do rollup based on different “group by” criteria

– Some accounting information not available in rollup records(some enhancements in V10)

– Makes problem shooting more complicated (see later)

When is an accounting record written?

• When the thread de-allocates – normal case

1st

SQLPgm end

1st

SQLPgm end

PgmStart

PgmStart


Time

Thread create

StatsAcctg

Thread de-allocate

Thread create

Thread de-allocate

StatsAcctg

When is an accounting record written? - 2

• Commit does not create an accounting record, except

– For inactive connections in DRDA (coming up)

– RRS using accounting-interval commit (and no open held cursors)

1st

SQL CommitPgm end

PgmStart CommitCommit


Time

Thread create

Thread deallocate

StatsAcctg


• When a thread is reused

– Applies to IMS and CICS

– (There are additional options in CICS – see CICS Transaction Server - DB2 Guide)

1st

SQL CommitPgm end

PgmStart

New tran1st SQL Commit

New tran1st SQL


Time

Create Thread + Signon

Thread de-allocate

Signon Signon

StatsAcctg StatsAcctg StatsAcctg

Accounting pitfall

• When a thread is reused but there is a long time between transaction end and the next transaction doing signon

1st

SQL CommitPgm end

PgmStart

New tran1st SQLRegion idle

CLASS 1 time reported


Time

Create Thread + Signon

Thread de-allocate

Signon

CLASS 1 time reported

StatsAcctg StatsAcctg


• When the connection goes inactive

– Distributed thread

– CMTSTAT=INACTIVE

1st

SQL Commit

Pgm endPgm

Start

New tran1st

SQL Commit

New tran1st

SQL


Time

AcquireDBAT

Conn.ends

Conn.inactive

Conn.inactive

AcquireDBAT

AcquireDBAT

StatsAcctg StatsAcctg StatsAcctg


• When using rollup accounting (DDF and RRS)

– E.g. ACCUMACC=3, ACCUMUID=2 (‘group by’ transaction name)

– Values will be the totals of all 3 transactions

1st

SQL Commit

Pgm endPgm

Start

New tran1st

SQL Commit

New tran1st

SQLCommit


Time

AcquireDBAT

Conn.ends

Conn.inactive

Conn.inactive

AcquireDBAT

AcquireDBAT

Conn.inactive

AcquireDBAT

AcctgAcctg AcctgStatsAcctg

Accounting Class 1 Data

thread allocation

1st SQL

2nd SQL

For local applications, DB2 provides an accumulative timer that includes both CPU spent in application and CPU spent in DB2.

activity

Class 1elapsedand CPU


thread deallocation

Out of DB2 In DB2

DB2.

Activity time is very close to Class 1 elapsed time.

For local applications, Class 1 (elapsed – CPU) time can point to application inefficiencies.

activitytime


thread allocation

1st SQL

2nd SQL

Class 2 timers are accumulative

Time spent out of DB2Class 1 elapsed – Class 2 elapsed




2nd SQL

thread deallocation

Time spent waiting in DB2Class 2 elapsed – Class 2 CPU

Class 7 is used for package/DBRM scope reporting. It is equivalent to Class 2

Out of DB2 In DB2


thread allocation

1st SQL

2nd SQL

Class 8 is used for package level reporting. It is equivalent to Class



Class 3Suspensions


2 SQL

thread deallocation

It is equivalent to Class 3.

Out of DB2 In DB2

in Appl in DB2 IRLM

Class 1 Class 2 (in DB2) 1st SQL..

SQL ..

SQL ..

...Wait for lock

(Creating Thread)

DB2 Times Terminology - Accounting Class 1,2,3


AgentAgent, non-nested ET

(Terminating Thread)

...Wait for I/O

Agent, non-nested CPU

Class 3(susp time)

...Commit

DB2 Times Terminology –Class 1&2 Elapsed Time with Nested Activity

in Appl in DB2 in UDFElapsed Time

Class 1 Class 2 (in DB2)

Trigger1

1st SQL ..

SQL ..

UDF

SQL ..

...UDF

(Creating Thread)


AgentTrigger

UDF

Agent, non nestedTrigger

UDF

Trigger2 SQL ..

(Terminating Thread)

DB2 Accounting Report Class 1-2-3 SectionsTIMES/EVENTS APPL(CL.1) DB2 (CL.2) IFI (CL.5) CLASS 3 SUSPENSIONS ELAPSED TIME EVENTS------------ ---------- ---------- ---------- --------- ----------- ------------ --------ELAPSED TIME 1:42.494131:42.49413 0.111129 N/P LOCK/LATCH(DB2+IRLM) 0 .000000 0

NONNESTED 0.208342 0.0571680.057168 N/A SYNCHRON. I/O 0.000000 0STORED PROC 0.000000 0.000000 N/A DATABASE I/O 0.000000 0UDF 1:42.238001:42.23800 0.0061650.006165 N/A LOG WRITE I/O 0.000000 0TRIGGER 0.0477970.047797 0.0477970.047797 N/A OTHER READ I/O 0.000000 0

OTHER WRTE I/O 0.000000 0CP CPU TIME 28.858601 0.029797 N/P SER.TASK SWTCH 0.020768 1

AGENT 28.858601 0.029797 N/A UPDATE COMMIT 0.020768 1NONNESTED 0.016157 0.005022 N/P OPEN/CLOSE 0.000000 0STORED PRC 0.000000 0.000000 N/A SYSLGRNG REC 0.000000 0UDF 28.822345 0.004677 N/A EXT/DEL/DEF 0.000000 0TRIGGER 0.020098 0.020098 N/A OTHER SERVICE 0.000000 0

PAR.TASKS 0.000000 0.000000 N/A ARC.LOG(QUIES) 0.000000 0LOG READ 0.000000 0

SECP CPU 0.000000 N/A N/A DRAIN LOCK 0.000000 0


SECP CPU 0.000000 N/A N/A DRAIN LOCK 0.000000 0CLAIM RELEASE 0.000000 0

SE CPU TIME 0.000000 0.000000 N/A PAGE LATCH 0.000000 0NONNESTED 0.000000 0.000000 N/A NOTIFY MSGS 0.000000 0STORED PROC 0.000000 0.000000 N/A GLOBAL CONTENTION 0.000000 0UDF 0.000000 0.000000 N/A COMMIT PH1 WRITE I/O 0.000000 0TRIGGER 0.000000 0.000000 N/A ASYNCH CF REQUESTS 0.000000 0

TCP/IP LOB 0.000000 0PAR.TASKS 0.000000 0.000000 N/A TOTAL CLASS 3 0.020768 1

SUSPEND TIME 27.525561 0.020768 N/A AGENT N/A 0.020768 N/A PAR.TASKS N/A 0.000000 N/A STORED PROC 0.000000 N/A N/A UDF 27.52556127.525561 N/A N/A

NOT ACCOUNT. N/A 0.060564 N/A DB2 ENT/EXIT N/A 104 N/A EN/EX-STPROC N/A 0 N/A EN/EX-UDF N/A 0 N/A DCAPT.DESCR. N/A N/A N/P LOG EXTRACT. N/A N/A N/P

DB2 Accounting Report Class 1-2 Sections

TIMES/EVENTS APPL(CL.1) DB2 (CL.2) TIMES/EVENTS APPL(CL.1) DB2 (CL.2) ------------ ---------- ---------- ------------ ------- --- ----------ELAPSED TIME 1:42.494131:42.49413 0.111129 ... ... ...

NONNESTED 0.208342 0.0571680.057168STORED PROC 0.000000 0.000000 SUSPEND TIME 27.525561 0.020768 UDF 1:42.238001:42.23800 0.0061650.006165 AGENT N/A 0.020768 TRIGGER 0.0477970.047797 0.0477970.047797 PAR.TASKS N/A 0.000000

STORED PROC 0.000000 N/A CP CPU TIME 28.858601 0.029797 UDF 27.52556127.525561 N/A

AGENT 28.858601 0.029797


AGENT 28.858601 0.029797 NONNESTED 0.016157 0.005022 NOT ACCOUNT. N/A 0.060564 STORED PRC 0.000000 0.000000 UDF 28.822345 0.004677 TRIGGER 0.020098 0.020098

PAR.TASKS 0.000000 0.000000 ... ... ...

Reporting of Nested Activity

• In DB2 accounting data, you can distinguish between

– Non-nested activity

– Nested activity (cannot distinguish between the level of nesting – that requires performance traces)

• Trigger

• Stored Procedure (SP)

• User Defined Function (UDF)

•


• For SP and UDF that run in a WLM-managed AS, DB2 can distinguish between Class 1 (program logic - COBOL code) and class 2 (executing SQL statements)

• For triggers, SQL PL SP and non-fenced UDFs, CL1=CL2

Reporting of Nested Activity - 2

• Only actual work done by SP is reported as nested activity at the plan level

– That is work done by the CALL sp

– When using result set cursors (WITH RETURN), typically only the OPEN cursor is done by the SP and is reported as nested activity

– The FETCHing is done by the calling application and is reported as non-nested time at the plan level


• Package level data shows the FETCHes (as well as CPU/ET time incurred) as if they were done by the SP package (that performed the OPEN cursor)

in Appl in DB2 In SPElapsed Time

Class 1 Class 2 (in DB2) 1st SQL ..

CALL sp SP

OPEN curs WITH RETURN

SQL ..

(Creating Thread)

Nested Activity for WITH RETURN processing

SP end


Agent

SP

Agent, non nested

SP

(TerminatingThread)

SP endFETCH

FETCH

FETCH

DB2 Accounting Class 1-2 (Plan Level) Sections

TIMES/EVENTS APPL(CL.1) DB2 (CL.2)------------ ---------- ----------ELAPSED TIME 0.941479 0.608886

NONNESTED 0.731353 0.4021540.402154STORED PROC 0.2101260.210126 0.2067320.206732UDF 0.000000 0.000000TRIGGER 0.000000 0.000000

CP CPU TIME 0.249935 0.192192AGENT 0.249935 0.192192

NONNESTED 0.248412 0.191915

SQL DML TOTAL-------- --------SELECT 0INSERT 0

ROWS 0UPDATE 0

ROWS 0MERGE 0


NONNESTED 0.248412 0.191915STORED PRC 0.001523 0.000277UDF 0.000000 0.000000TRIGGER 0.000000 0.000000

PAR.TASKS 0.000000 0.000000... ... ...

SUSPEND TIME 0.000323 0.000000AGENT N/A 0.000000PAR.TASKS N/A 0.000000STORED PROC 0.000323 N/AUDF 0.000000 N/A

NOT ACCOUNT. N/A 0.416693 ... ... ...

MERGE 0DELETE 0

ROWS 0

DESCRIBE 0DESC.TBL 0PREPARE 0OPEN 1FETCH 5204

ROWS 5203CLOSE 1

DML-ALL 5206

DB2 Accounting Report Class 7-8-10 (Package Level)

• Package level data is being reported as if the FETCHes are executed by the SP

MRSBMCBM (main) TOTAL MRSBMS (SP) TOTAL------------------ -------- ------------------ ------- -SELECT 0 SELECT 0INSERT 0 INSERT 0UPDATE 0 UPDATE 0DELETE 0 DELETE 0


DESCRIBE 0 DESCRIBE 0PREPARE 0 PREPARE 0OPEN 0 OPEN 1FETCH 0 FETCH 5204CLOSE 0 CLOSE 1

LOCK TABLE 0 LOCK TABLE 0CALL 1 CALL 0

DB2 Accounting Report Class 7-8 (Package Level)MRSBMCBM (main) VALUE MRSBMCBM TIMES ------------------ ------------------ --------------- --- ------------TYPE PACKAGE ELAPSED T IME - CL7 0.151778 LOCATION STLEC1 CP CPU TI ME 0.083859 COLLECTION ID BARTCOB AGENT 0.083859 PROGRAM NAME MRSBMCBM PAR.TAS KS 0.000000 CONSISTENCY TOKEN 1906C50F153EE340 SE CPU TI ME 0.000000 ACTIVITY TYPE NONNESTED SUSPENSION-CL8 0.000000 ACTIVITY NAME 'BLANK' AGENT 0.000000 SCHEMA NAME 'BLANK' PAR.TAS KS 0.000000

MRSBMS (SP) VALUE MRSBMS TIMES ------------------ ------------------ ------------------ ------------


------------------ ------------------ ------------------ ------------TYPE PACKAGE ELAPSED T IME - CL7 0.4960330.496033LOCATION STLEC1 CP CPU TI ME 0.139170 COLLECTION ID BARTCOB AGENT 0.139170 PROGRAM NAME MRSBMS PAR.TAS KS 0.000000 CONSISTENCY TOKEN 1906C50417D039C3 SE CPU TI ME 0.000000 ACTIVITY TYPE STORED PROC SUSPENSION-CL8 0.000000 ACTIVITY NAME MRSBMS AGENT 0.000000 SCHEMA NAME SYSPROC PAR.TAS KS 0.000000 SUCC AUTH CHECK NO NOT ACCOU NTED 0.356863

Greater than Nested time at plan level - includes time for FETCH operations

Analyzing Acctg Data: Top-Down Analysis

• Accounting report (not trace) by connection type most useful for initial analysis

• For OMPE users:

ACCOUNTING REPORT LAYOUT(LONG) ORDER(CONNTYPE)

EXCLUDE(PACKAGE(*))


• To group by thread connection type such as TSO, CICS, DB2CALL, RRS, IMS, DRDA, UTILITY, etc. for the period of interest

• Also combine with STATISTICS REPORT LAYOUT(LONG) for the corresponding period

Analyzing Thread Activity Time

Thread activity time = Class 1 elapsed

Elapsed time spent out of DB2 = Class 1 elapsed – Class 2 elapsed(in application, network, idle..)

Elapsed time spent in DB2 = Class 2 elapsed

Processing time = Class 2 CPU (GCP and zIIP)


Waiting time = Class 2 elapsed – Class 2 CPU

Suspended time = Class 3 suspension time

Not accounted time = Waiting time – Suspended time

What if Activity Time is Split as …

• Application logic inefficiency

– Often combined with Class 2 CPU << Class 1 CPU

• Network problems

• User think (coffee?) time

• Class 2 not active all the time

In DB2

Out of DB2


• Class 2 not active all the time

• Potential pitfalls

– Thread reuse and long time between consecutive transactions (allied threads)

– Long running DDF threads when CMTSTAT=ACTIVE

What if Activity Time is Split as …

• What’s the actual number (not just the %)

• Is this a DB2 intensive process anyway?

•

In DB2

Out of DB2


• Need to analyze time distribution in DB2 : Where is the time really spent inside DB2?

Time in DB2 (Class 2 Time)

• Burning CPU cycles (class 2 cpu time)

– Non-nested or nested

– On a general CP or zIIP

• Suspended in DB2 for a known event (class 3 suspension time)

– Many different types of suspensions (21 in V10)

• In DB2 but don’t know what it is we are doing



– Mostly things that happen outside of DB2’s control, such as

• Lack of CPU or paging

– Should be a small %

Accounting CPU time reporting

• CP CPU time

– CPU time used on general purpose engine

• SE CPU time

– Any CPU used on a specialty engine (zIIP or zAAP)

– Not included in GP CPU time

• Sub-capacity machines

– SE CPU time is reported as if had run on general CP


– SE CPU time is reported as if had run on general CP

– Can result in total CPU= (CP + SE CPU) > ET for CPU intensive work

• SECP CPU

– Could have run on specialty engine but didn’t

– No longer reported in V10

Accounting CPU time reportingCONNTYPE: DRDA

AVERAGE APPL(CL.1) DB2 (CL.2)------------ ---------- ----------ELAPSED TIME 2.840103 1.913457

NONNESTED 2.839901 1.913339STORED PROC 0.000202 0.000118UDF 0.000000 0.000000TRIGGER 0.000000 0.000000

CP CPU TIME 0.260398 0.251655AGENT 0.260398 0.251655

NONNESTED 0.260326 0.251584


NONNESTED 0.260326 0.251584STORED PRC 0.000072 0.000071UDF 0.000000 0.000000TRIGGER 0.000000 0.000000

PAR.TASKS 0.000000 0.000000

SECP CPU 0.000726 N/A

SE CPU TIME 0.261583 0.246227NONNESTED 0.261583 0.246227STORED PROC 0.000000 0.000000UDF 0.000000 0.000000TRIGGER 0.000000 0.000000

PAR.TASKS 0.000000 0.000000

What if In DB2 Time is Split as …

• Turn off expensive traces

• Inefficient access paths

− Run EXPLAIN

− Check transaction profile− #SQL stmts− #locks

ProcessingWaiting


• What is the largest contributor

− Class 3 and 8 analysis

− #locks− #getpages

Processing

Waiting

Time in DB2 (Class 2 time)





– Many different types of suspensions (21 in V10)




– Mostly things that happen outside of DB2’s control, like

– Lack of CPU or paging



• Class 3 accounting strongly recommended

• Provides information about “known” DB2 suspensions. For suspensions, DB2 reports:

– The number of suspensions

– The total time it was suspended for those

• Always look at both #suspensions and suspension time

• Double accounting is avoided


• Double accounting is avoided

• Negligible overhead except when high internal DB2 latch contention, e.g. over 10,000/sec

Sync.EU Switch

Locking

I/O C

lass

3 S

uspe

nsio

n T

ypes

Synchronous read/write & log write ��

Synchronous read/write ��

Log write ��

Other agents’ read ��

Other agents’ write ��

Force-at-commit database writes (LOG NO LOBs only) ��

TCP/IP LOB XML ��

IRLM lock/latch & DB2 internal latch ��

IRLM lock/latch ��

DB2 internal latch ��

Page latch ��

Drain lock ��

Claim release ��

Synchronous Execution Unit Switch total ��

Open/Close ��

Define/Extend/Delete ��

V5V5 V6V6 V7,8V7,8 V9V9 V10V10


Archiving

Scheduling

Data Sharing

Cla

ss 3

Sus

pens

ion

Typ

es

SYSLGRNX recording ��

Commit ��

Other services ��

Archive Log command ��

[Archive] log read ��

Stored procedures ��

UDFs ��

Global locks total ��

Parent L-locks ��

Child L-locks ��

Other L-locks ��

Page set/Partition P-locks ��

Page P-locks ��

Other P-locks ��

Sending Notify messages ��

Asynchronous coupling facility request completion ��

Lock/Latch Suspensions

• Thread can be suspended for:

– Lock contention

• includes preformatting (X’09’)

– IRLM latch contention

– DB2 internal latch contention

• Page latch contentions are reported in a separate timer

•wa

UPDATE

applicationaddress space

DB2address space

lock suspend

lock unavailable


• DB2 reports:

– Accumulated time thread was suspended

– Total number of suspensions

• Time/events is an average wait due to a lock/latch conflict. The number of true lock suspensions (from the Locking block) is often a better divisor.

ait

lock resume

buffer update

Large Lock/Latch Suspensions – Large number?

Number of IRLM lock suspensions high?− Application problems− Incompatible workload mix− Page/row level locking− Inefficient preformatting

�Use PAV�Check ICF catalog performance

Number of DB2 internal latch suspensions high?− LC06: Index tree P-lock latch contention caused most

likely by splits of GBP-dependent index pages− LC11: Generating Identity Column− LC14: BP LRU chain – Isolate small highly accessed

tables into separate pools− LC19: Log – Speed up log I/O− LC24: EDM LRU or BM latch− LC25: EDM Pool hash chain


Waits forlock/latch suspensions

In DB2 Time

Number of IRLM latch suspensions high?− If more than 10% of all IRLM requests− IRLM Trace on− Low IRLM dispatching priority− Frequent IRLM Query requests (e.g. -DIS DATABASE

LOCKS, or MODIFY irlmproc, STATUS)− Low deadlock detection cycle and many lock

suspensions

− LC25: EDM Pool hash chain− LC32 Storage Manager Pool Header

Large Lock/Latch Suspensions – High average time?

Time per suspension large? • Lock suspensions

– Non-committing transactions

– Concurrent DDL

• RELEASE(DEALLOCATE)

– For detail analysis, use performance trace classes 6 and 7

Waits forlock/latch suspensions

In DB2 Time


classes 6 and 7

• DB2 latch suspensions

– Designed to be short duration

– Can be a symptom of another problem

– For detail analysis, use performance trace class 11

suspensions

Notes

• Lock/Latch wait = Lock wait + IRLM latch wait + Internal DB2 latch wait

– In the rare case of over 10,000 per second, disabling class 3 may significantly bring down class 1 and 2 cpu time

– Look at IRLM lock suspension and IRLM latch suspension counter to get a better idea which one of the 3 is likely to be the culprit


which one of the 3 is likely to be the culprit

• DB2 10

– Separate out into IRLM lock/latch and DB2 latch

– More efficient DB2 latch resume logic

• Less chance to get resumed and suspended again without ever obtaining the latch in between

Synchronous I/O Suspensions Large?

Number of suspensions high− Excessive Getpages− Buffer pools contention− Disorganized data/indexes

SynchronousI/O Waits

In DB2 Time


Time per suspension large − Make sure there are enough I/O resources− Use RMF to verify− For detailed analysis in DB2,

use performance class 4

Synchronous Database I/O Suspensions

• A common timer for:

– Synchronous database reads

– Synchronous database writes

• A separate timer for synchronous log writes

• Use BP statistics to find out how

getpage missUPDATE


DB2address space

read begin


• Use BP statistics to find out how many of the suspensions should be attributed to synchronous reads vs. writes

• Avg I/O wait time in Acctg rpt =0.028539 sec/12.41 events = 0.002 sec / sync DB IO

wait

read end

buffer update

Tuning the # Sync I/O suspensions

• If many database I/O waits, check

– Excessive Getpages because of a bad access path

• Use Explain to tune access path

– Buffer pools contention

• Other applications ruining our hit ratio

• BP too small

• (V)DWQT setting


•• Look at BP statistics

– Disorganized data/indexes

• Check statistics

– Minimize index splits

– Zero freespace for data for minimum read and write I/O in heavy Insert or APPEND YES

Tuning I/O Wait Time

• If high database I/O wait, check

– Enough I/O resources ? (Use RMF to verify)

– DASD contention

– High IOSQ time – use PAV

– Faster I/O devices (use SSD)

– Control Unit cache misses

– CPU contention


– CPU contention

– Inappropriate I/O priority

• Use I/O striping where appropriate

– Less effective with newer I/O devices

– Still beneficial for non-saturated I/O configuration with infrequent commit and large amount of data to be read or written in one I/O, e.g. list prefetch, heavy write

Synchronous Log write I/O Suspensions

Force log buffers to diskCommit


DB2address space

• “Normally” only forced write to active log at commit time

• All log buffers that have not been written are written at commit

• Most common cases are listed on next foil

write begin


wait

write begin

write end

No more log buffers to write

What triggers sync log write I/Os ..

• For transactions that use 2PC, Phase 1 of 2-phase commit

• Synchronous write (to disk or GBP)

– Log write ahead protocol

• Running out of log output buffers

• Identity columns/sequences that are using NO CACHE (or low cache value) or ORDER is used (in data sharing)

– Hidden SPRMNLGF zparm to avoid especially if identity columns are used


– Hidden SPRMNLGF zparm to avoid especially if identity columns are used

• V10 up to 40% reduction in (dual) log I/O wait because of parallel log write I/O’s

What triggers sync log write I/Os ..

• Index split in data sharing

– Reduce index page splits by

• Non-zero freespace (FREEPAGE, PCTFREE) for random insert

• PCTFREE 0 for sequential insert

• Larger index page size in V9

– Trade-off with possibly higher index page latch contention and P-lock contention

• Asymmetric index page split in V9

•


• P-lock negotiation

• IMMEDWRITE(YES) for GBP-dependent objects

• If GBP-dependent, when taking (a completely empty) index page off the IX chain

• If GBP-dependent, new (reuse after mass delete or brand-new) page used in segmented + UTS

– PK83735/PK94122 (V9) and V8 PM17813/17817 (V8) should address this

Wait for read i/o by another application agent or prefetch engine

Suspensions for Other Agents’ Reads

wa

getpage miss

FETCH


DB2address space

prefetch begin

Number of suspensions high?− Excessive GetPages

�Use Explain to tune access path− Buffer pools contention

�Look at BP statistics

In DB2 Time

Wait forotheragents’reads

What if …


ait

first block of pagesFETCH

getpage hit

getpage hit

�Look at BP statistics− Disorganized data/indexes

�Check statistics

Time per suspension large?− Use RMF and, for detailed analysis

performance class 4 to check on: �DASD contention�Control Unit cache misses

�Check SEQCACHE=YES�Consider parallelism/partitioning�Inappropriate I/O priority

Suspensions for Other Agents’ Writes

UPDATE


DB2address space

Number of suspensions high?− Too frequent checkpoints

�Check DB statistics, LOGLOAD− Too low (vertical) deferred write

In DB2 Time

Wait forotheragents’ writesWhat if …

buffer update

UPDATE


wait

Page being written

page written

− Too low (vertical) deferred write threshold for high re-reference access pattern�Check BP statistics

Time per suspension large?− DASD contention− CU cache misses− Inappropriate I/O priority

buffer update deferredwrite start

UPDATE

Notes

• Other write I/O wait = wait for write i/o by another application agent or write engine, may include some time waiting for log write-ahead

• When updating a buffer being written, first agent waits for OTHER WRITE I/O. All others wait for PAGE LATCH contention.


Synchronous EU Switch Suspensions

w

SQL


DB2address space

switch to a TCB/SRB

any of these tasks

• ‘Update’ commits− Single-phase commit− Phase 2 of two-phase commit− Abort− Deallocation

• Dataspace Manager tasks

A separate counter for:


wait

task completed

− Delete/Define/Extend data set− Reset data set− VSAM catalog access

• OPEN/CLOSE data set− Not include HSM RECALL of data

set

• SYSLGRNX recording− And down-level detection

• Other services tasks− DDF application requester

considerations

Large Suspensions for Synchronous EU Switches?

In DB2 Time

Waits forSynchronous

• Check log efficiency− DB statistics, RMF

• Check VSAM catalog performance


SynchronousExecution UnitSwitches

• PCLOSET or PCLOSEN too low

• Check if DSMAX is too low

Service Task Wait - Commit Related Timers

• Main components of commit processing

– Synchronous log writes

– Cursors, locking, etc. clean-up

– Force-at-commit synchronous database writes, or GBP writes in data sharing

Class 3 Timer Read-only commit Single-phase commit Two-phase commit


1 Included in wait for synchronous EU switch for 'update' commit.2 These are log writes that happen in phase 1 of two-phase commit.

Log writes that happen in phase 2 are included in wait for synchronous EU switch for 'update' commit.3 Applies to LOBs with LOG(NO) only. These database writes happen in phase 1 of two-phase commit.

Wait for log writes no no1 yes2

Wait for synchronous EUswitch for update commit

no yes yes

Wait for force-at-commitdatabase writes

no no1 yes3

Service Task Wait - 2

• Dataset [Extend]/Define/Delete

• Aka Dataspace manager work

– Extend for preformatting, takes 0.02 to 1sec each time depending on device type and alloc unit/size

– x'09' lock wait in async preformat since V7 so no longer shows up here

– Counting pieces of a segmented TS (at open time)

– Dataset Delete/Define when Stogroup is used


– Dataset Delete/Define when Stogroup is used

– IFCID 258 contains DBID/OBID for Dataset Extend

– IFCID 92 contains Define Cluster statement for Define/Delete

Service Task Wait - 3

• Dataset Open/Close

– Up to 20 tasks for parallel Open in V8, 40 in V9

– Typically no class3 acctg wait for dataset close as it is asynchronous

– V10 Access DB Mode(Open) – parallel data set open

– PM19528 V9/10 introduces IFCID370-371

• Others


• Others

– Time waiting for server in TCP/IP requestor accounting

– VSAM catalog update

– Parallel query cleanup

– And many others which are typically infrequent

66

Commonly Observed OTHER SERVICE TASKS

• Parallel query cleanup x’14’/x’77’ or x’84’

– Shows up in repeated execution of short-running queries in parallelism

– Use higher SPRMPTH ZPARM value to replace this (default=120(ms))

– Of course, for a long-running query, parallel query with degree ANY can result in many times reduction in elapsed time

• VSAM catalog update x’0A’/x’94’


• VSAM catalog update x’0A’/x’94’

• DB2 for z/OS application requestor using TCP/IP waiting for server response x’1B’/x’8F’

67

How to find which service task invoked

• Performance trace of IFCID 46 thru 50

– IFCID 46 for service task switch

• 47 and 48 for Begin and End of SRB service task

• 49 and 50 for Begin and End of TCB service task

– Shows all nested service tasks

• Or use IFCID 170 and 171 for less output


• Or use IFCID 170 and 171 for less output

– Shows one (outermost) service task for each event in class 3 accounting

Notes

• Correlation name of system agent (SYSOPR planname) in performance trace:

– eg. 021.OPNL for SYSLGRNX update in pageset open

• Chapter 9 of DB2 Diagnosis Guide and Reference contains a list of system agent correlation identifiers as well as a list of Resource Manager id's displayed in IFCID 46, 47, 49, and 170.


Commonly observed service tasks

• Commonly observed service tasks and RMID/FC identifiers as shown in IFCID 46, 47, 49, and 170

– Update Commit x’03’/x’49’

– Dataset Open x’0A’/x’59’

– Dataset Close x’0A’/x’4F’


– SYSLGRNX Update x’15’/x’44’

– Dataset Extend x’12’/x’65’

– Dataset Delete x’12’/x’6A’

– Dataset Define x’12’/x’68’

– Dataset Reset x’12’/x’6C’

– Dataset Count x’12’/x’88’

Notes

• FC = Function Code

• RMID = Resource Manager ID

• If a long time gap between IFCID 46 and 47 (begin SRB task) or 49 (begin TCB task), then it is possible that we are waiting for a service task to become available.


Log Read Suspensions

w

ROLLBACK


DB2address space

read active or archive log

Wait forotheragents’ writes

data not in log buffer

What if … Wait forlog reads

Number of suspensions high?− Find out why the transactions rollback

(most rollbacks are done under EU-switch - those don’t show up here)

− Keep to a minimum


wait

archive log

data retrieved

backout

− Keep to a minimum

Time per suspension high?− Increase the number and size of

active log data sets

• In V8 only archive log read wait was tracked• Starting in V9 active log prefetch read wait time also included• Vnext will also include sync. active log read

(currently NOT ACCOUNT time)

Archive Log Mode (Quiesce) Suspensions

UPDATE


DB2address space

Wait forotheragents’ writesWhat if …

Waits forquiesce

commit

buffer update

COMMITARCHIVE LOGMODE(QUIESCE)issued


wait

suspend

resume

Number of suspensions high?− Too low QUIESCE system parameter

or command TIME option

Time per suspension high?− Long units of recovery− Inappropriate timing for quiesce process

buffer update

commit

UPDATE

issued

archivebegins

Page Latch Suspensions

w

FETCH


DB2address space

page latch suspend


getpage hit,but page latched Most often a symptom of something else

Often during high insert activity−Space map page

What if …

Wait forpage latchsuspensions


ait

page latch resume

−Space map page�Partitioning�MEMBER CLUSTER

−Data page�Randomize inserts

−Use IFCID 226, 227 for detail analysis

Page Latch Contention

• Also consider

– Higher VDWT to possibly reduce write frequency

– Faster active log and database write I/O device

– If spacemap page,

• TRACKMOD NO instead of default YES

– However, no ‘real’ incremental image copy if TRACKMOD NO

• Concurrent inserts to different partitions


• Concurrent inserts to different partitions

• Classic partitioned rather than segmented or UTS in V9

– If index page,

• V9 RANDOM instead of ASC/DESC

– Trade-off with higher data Getpage, I/O, lock requests

• V9 index page: smaller if random key insert, bigger if sequential key insert

Global Lock Suspensions

wa

UPDATE


DB2address space

lock suspend


lock unavailable – held by another system

• Thread can be suspended for:− Global L-parent lock− Global L-child (row/page) lock− Other global L-lock− Global pageset P-lock− Global page P-lock

− Data− IX leaf pages− SM pages


ait

lock resume

buffer update

− SM pages− Other P-locks

• DB2 reports:− Accumulated time spent suspended− Total number of suspensions− Sum of all IRLM, XES, and False

contentions in Data Sharing Locking section


What if …Wait forglobal locksuspensions

High Global Lock Contention Events/Time


• Keep CF utilization below 50% for availability, response time, CPU time• If most time or most events are “OTHER” P-lock or L-lock

– -STA TRA (P) C(6) … for a short time to get detailed info

High Global lock contention events/time

• To minimize P-lock contention/negotiation

– First find out from GBP statistics what is contributing to P-lock contention

– If spacemap page P-lock and/or data page P-lock, consider using MEMBER CLUSTER table space

– V8/9/10 Partitioned table space

–


– V10 UTS

– High spacemap page latch contention possible when the latch is held waiting for data spacemap page P-lock contention to resolve.


• P-lock contention/negotiation - continued

– Consider setting VDWQT=0 to free page P-locks as soon as possible

– If index tree P-lock (high index splits), consider

– Freespace tuning (PCTFREE FREEPAGE)

– Minimize index key size especially for unique or semi-unique multi-column index key

– V8 non-padded index if large VARCHAR(s) in index key

– V9 bigger index page size and asymmetric page split


–

– If data page P-lock contention on small tables with LOCKSIZE ROW, consider MAXROWS=1 and LOCKSIZE PAGE to ‘simulate’ row locking and/or reduce SM page freespace update


• Can also affect other counters

– Update Commit service task wait P-lock unlock for each updated page

– Lock/Latch wait can increase because of

– Possibly higher IRLM latch contention

– “Mixing” local/global lock waits (next foil)


“Mixing” Global/Local Lock Suspensions

Acquire lock

Lock

TRAN AX-Lock page 5

TRAN BX-Lock page 5

lock suspend

TRAN CX-Lock page 5

Release lock

TRAN ACommit

TRAN BX-Lock page 5

TRAN BCommit

Lock

TRAN CX-Lock page 5

Time Member 2


Lockresume

Time Member 1

lock suspend(Wait behind TRAN B)Lock suspend

LockresumeRelease lock

Global lock wait

Local lock wait

• Even though part of the wait time for tran C it is waiting for a global lock held by tran A, all suspension time is reported as local because at resume time, the lock that was released was local to this member

Time in DB2 (Class 2 time)





– Many different types of suspensions ( 21 in V10)




– Mostly things that happen outside of DB2’s control, like

– Lack of CPU or paging


What Is Left?

• Processor wait

– Wrong dispatching priorities

• NO DB2 work should run in discretionary

– Overloaded processor

• Use RMF for details

• General tuning

• Processor upgrade

•

Suspensions

In DB2 Time

Processing


• Paging

– Overcommitted real storage

• Use RMF for details

• MAXKEEPD too large

• Check Buffer pools, EDM pool, Sort, number of threads, …

• Real storage upgrade

Not Accounted

What Is Left? - 2

• OEM monitors

• Wait time for completion of parallel tasks

• System-managed duplexing lock requests

• HSM dataset recall (since PQ67298)

• Waiting for requests to be returned from

Suspensions

In DB2 Time

Processing


• Waiting for requests to be returned from SNA DB2 Server

• Portion of OPEN processing

– Accompanied with the appropriate synchronous EU switch suspensions

Not Accounted

Notes

• Most common are:

– CPU wait under high CPU utilization, especially with lower dispatching priority

– Excessive detailed online tracing with vendor tools

• Other causes are much less frequent and widely variedSome events not being captured by DB2, but more events are being captured in newer DB2 versions

– See online support document:


– See online support document: http://www.ibm.com/support/docview.wss?rs=64&context=SSEPEK&uid=swg21045823

Effect of lack of CPU resources on CL3/8 timers

wa

getpage missUPDATE


DB2address space

• “Clock” on the wait time can only be stopped after the thread is dispatched again

• If you have to wait for the CPU the timer keeps “ticking”

Start I/O

Start clock


ait

tI

me

Stop clock

buffer update

suspend

I/O interrupt

resume

I/O time

CPU wait

Accounting for DDF Work

• Processing "in DB2" (Class 2) should (almost) be the same regardless of connection type

• Time in DB2 server is:

– Class 2 nonnested elapsed time + Class 1 stored procedure, UDF, and trigger ET+Nonnested (Class 1 CPU - Class 2 CPU) +Nonnested (class 1 SE CPU - class2 SE CPU)

•


• Time outside of DB2 server is total Class 1 elapsed less previous calculation

• Active thread accounting records are created at thread deallocation (disconnect)

• Inactive thread accounting records are created at DBAT inactive (look for DBAT inactive)

DDF Accounting at Plan Level

TIMES/EVENTS APPL(CL.1) DB2 (CL.2)------------ ---------- ----------ELAPSED TIME 3:10.24012 3.387696

NONNESTED 3:10.24012 3.387696STORED PROC 0.000000 0.000000UDF 0.000000 0.000000TRIGGER 0.000000 0.000000

CP CPU TIME 1.020178 0.980084AGENT 0.979296 0.939458

NONNESTED 0.979296 0.939458STORED PRC 0.000000 0.000000UDF 0.000000 0.000000

• Time in DB2 = class 2 nonnested ET + class 1 nested activity ET

[stored procedure + UDF + Trigger]+ nonnested (class 1 CP CPU - class 2 CP CPU) + nonnested (class 1 SE CPU - class2 SE CPU)

• Time in DB2 = 3.387696 + (0.000000 + 0.000000 + 0.000000)+ (0.979296 - 0.939458 )


UDF 0.000000 0.000000TRIGGER 0.000000 0.000000

PAR.TASKS 0.040882 0.040626

SECP CPU 0.000000 N/A

SE CPU TIME 0.008714 0.008714NONNESTED 0.008714 0.008714STORED PROC 0.000000 0.000000UDF 0.000000 0.000000TRIGGER 0.000000 0.000000

PAR.TASKS 0.000000 0.000000

SUSPEND TIME 0.000000 1.440549.. .. ..

+ (0.979296 - 0.939458 )+ (0.008714 - 0.008714)= 3.427534

• Time outside DB2 = c lass 1 ET - Time in DB2

• Time outside of DB2 =3:10.24012 – 3.427534 = 3:06.8126

• Note: Be careful with SE CPU time when running on a ‘knee-capped’ machine – using normalized CPU times

DDF Accounting at Plan Level -2

SQL DML TOTAL

---- DISTRIBUTED ACTIVITY -------------------------- ---------------------------------------------------REQUESTER : DDF1 ROLLBCK(1) RECEIVED: 0 THREADS INDOUBT : 0 PRODUCT ID : DB2 SQL RECEIV ED : 601 ROWS SENT : 46515 PRODUCT VERSION : V9 R1 M5 MESSAGES SENT : 603 BLOCKS SENT : 340 METHOD : DRDA PROTOCOL MESSAGES R ECEIVED : 604 CONVERSAT.INITIATED: 1 COMMITS(1) RECEIVED: 0 BYTES SENT : 2087111 NBR RLUP THREADS : N/ A

BYTES RECEIVED : 809497

COMMIT(2) RECEIVED : 1 COMMIT(2) RESP.SENT: 1 PREPARE RECEIVED : 1 BACKOUT(2) RECEIVED: 0 BACKOUT(2) RESP.SENT: 0 LAST AGENT RECV. : 0 COMMIT(2) PERFORMED: 0 BACKOUT(2) PERFORMED: 0 MESSAGES IN BUFFER : 4651 5 TRANSACTIONS RECV. : 1 FORGET SENT : 0


SQL DML TOTAL-------- --------SELECT 0INSERT 226

ROWS 282UPDATE 30

ROWS 84MERGE 0DELETE 0

ROWS 0

DESCRIBE 0DESC.TBL 0PREPARE 144OPEN 142FETCH 199

ROWS 46515CLOSE 1

DML-ALL 742

• Blocking is crucial for performance: Rows areput into blocks which are then sent out in messages

• Note # rows in DML operations for multi-row operations (implicit or explicit)

Acctg Reporting when (CPU) Parallelism is used

• In DB2 accounting data you can distinguish between

– Agent (parent)

– Parallel task

• # acctg records depends on ZPARM PTASKROL (Opaque ZPARM)

– Yes (Default)

• One acctg record for the parent task, and one acctg record for all // activity

• When using sysplex //, coordinating member is same as above, and each assisting member


• When using sysplex //, coordinating member is same as above, and each assisting member cuts one acctg record per // group (so can be more than one per query/tran)

– No

• Each parallel task cuts a separate acctg record (regardless of type of CPU //)

– Normally your friendly monitor takes care of ‘rolling up’ this information

OMPE Accounting Report/Trace for a Parallel Query

TIMES/EVENTS APPL(CL.1) DB2 (CL.2) IFI (CL.5) CLASS 3 SUSPENSIONS ELAPSED TIME EVENTS------------ ---------- ---------- ---------- --------- ----------- ------------ --------ELAPSED TIME 1:27.10913 1:27.10502 N/P LOCK/LATCH(DB2+IRLM) 0.067109 634

NONNESTED 1:27.10913 1:27.10502 N/A IRLM LOCK+LATCH N/A N/ASTORED PROC 0.000000 0.000000 N/A DB2 LATCH N/A N/AUDF 0.000000 0.000000 N/A SYNCHRON. I/O 0.016820 47TRIGGER 0.000000 0.000000 N/A DATABASE I/O 0.016820 47

LOG WRITE I/O 0.000000 0CP CPU TIME 24.253183 24.252075 N/P OTHER READ I/O 10:38.762530 98546

AGENT 0.002148 0.001053 N/A OTHER WRTE I/O 0.000000 0NONNESTED 0.002148 0.001053 N/P S ER.TASK SWTCH 0.319902 22

Susp. time of parent and ALL child tasks


NONNESTED 0.002148 0.001053 N/P S ER.TASK SWTCH 0.319902 22STORED PRC 0.000000 0.000000 N/A UPDATE COMMIT 0.000000 0UDF 0.000000 0.000000 N/A OPEN/CLOSE 0.318962 10TRIGGER 0.000000 0.000000 N/A SYSLGRNG REC 0.000823 10

PAR.TASKS 24.251035 24.251022 N/A EXT/DEL/DEF 0.000000 0OTHER SERVICE 0.000117 2

.. .. .. .. .. .. .. ..

.. .. .. .. .. .. .. ..

SUSPEND TIME 0.000000 10:39.1664 AGENT N/A 0.000117 PAR.TASKS N/A 10:39.1662 STORED PROC 0.000000 N/A UDF 0.000000 N/A

NOT ACCOUNT. N/A 1:27.10385

NOT ACCT time calculated based on parent task only

Sum of CPU time of all // tasks

CPU time of parent task

Note: CPU time can exceed ET when using // for CPU intensive queries

Understanding the Transaction ‘Profile’

• To get a better feel for type of work performed by a transaction/job, always at least check:

– # and type of SQL statements (S/I/U/D)

– The # and type of lock request (local and global (when data sharing)


– BP activity and GBP activity(when data sharing)

SQL Info in DB2 Accounting at Plan level

• PK62161 (V8/9) Counters added for multi-row operations

• PM24812 (V8/V9/V10)Counters will include only rows that are updated, deleted or inserted by direct user action

SQL DML AVERAGE TOTAL ------------- ---------- ----------SELECT 5.02 1217636 INSERT 0.84 203878

ROWS 1.69 409044 UPDATE 0.87 211103

ROWS 1.01 244029 MERGE 0.00 0 DELETE 0.10 24085

ROWS 0.10 24085


• DB2 10 ‘DRDA bundled fetch’ may reduce #FETCH counter, but #ROWS-FETCH should be accurate

ROWS 0.10 24085

DESCRIBE 0.00 0 DESC.TBL 0.00 0 PREPARE 0.00 0 OPEN 3.14 762136 FETCH 29.10 7053106

ROWS 26.13 6333609 CLOSE 2.94 711718

DML-ALL 42.02 10183662

Locking Info in Accounting at Plan Level

• Extra info about MAX PG/ROW LOCKS HELD compared to stats

LOCKING AVERAGE TOTAL--------------------- -------- --------TIMEOUTS 0.00 0DEADLOCKS 0.00 0ESCAL.(SHARED) 0.00 0ESCAL.(EXCLUS) 1.00 4MAX PG/ROW LOCKS HELD 5004.00 5004LOCK REQUEST 5042.00 20168


LOCK REQUEST 5042.00 20168UNLOCK REQUEST 34.00 136QUERY REQUEST 0.00 0CHANGE REQUEST 3.00 12OTHER REQUEST 0.00 0TOTAL SUSPENSIONS 1.75 7

LOCK SUSPENSIONS 0.00 0IRLM LATCH SUSPENS. 1.75 7OTHER SUSPENS. 0.00 0

Locking Info in Accounting at Plan Level - 2

• TIMEOUTS

– The number of times lock suspension ultimately resulted in a timeout

• DEADLOCKS

– The number of times lock suspension ultimately resulted in a deadlock. This happens when two or more application processes each hold locks on resources that the others need and without which they cannot proceed

• ESCAL.(SHARED/EXCLUSIVE)


• ESCAL.(SHARED/EXCLUSIVE)

– Number of times the LOCKS PER TABLE(SPACE) parameter on the panel DSNTIPJ was exceeded and the lock size was promoted from a page lock to a table space lock for this thread. For example, an application updates most pages in a table without issuing commits

• Further investigation

– Typically start with statistics trace class 3 info + syslog + db2MSTR log

– Zoom in with Locking traces – perf. trace class 6,7 and SQL trace – perf. trace class 3 –STA TRA(P) C(1,2,3,6,7) DEST(xxx) should do


• MAX PG/ROW LOCKS HELD from Accounting trace is a useful indicator of commit frequency

– Page or row locks only

– AVERAGE is for average of MAX, TOTAL is for max of MAX (of Accounting records)

• So if transaction A had max. locks of 10 and transaction B had 20, then

– AVERAGE (avg. of max.) = 15

– TOTAL (max. of max.) = 20


– In general, try to issue Commit to keep max. locks held below 100

LOCKING AVERAGE TOTAL--------------------- -------- --------...MAX PG/ROW LOCKS HELD 5004.00 5004


• Good Lock avoidance can significantly improve performance

– Quick check in acctg data can provide some clues whether or not lock avoidance is ok

– Lock avoidance may not be working effectively if Unlock request/commit is high

• >5/commit and > 1/3 of Lock request

• E.g. 50 Lock and 40 Unlock requests per commit

– Check table on “Lock avoidance factors” in PMTG to see under which conditions DB2 is able to use lock avoidance techniques


• Using ISO(CS) CURRENTDATA(NO) – default now – gives DB2 most opportunities to avoid having to take locks

• Check the IRLM lock and IRLM latch suspensions in the locking section against the L/L CL3 suspensions

Data Sharing Locking Section

DATA SHARING AVERAGE TOTAL------------------- -------- --------GLOBAL CONT RATE(%) 0.48 N/AFALSE CONT RATE(%) 0.00 N/AP/L-LOCKS XES(%) 43.83 N/ALOCK REQ - PLOCKS 0.67 270412UNLOCK REQ - PLOCKS 0.63 252966CHANGE REQ - PLOCKS 0.01 2550

• Global Contention should be less than 3-5% of XES IRLM Requests– (IRLM + XES + False SUSP) /

((IRLM + XES + False SUSP) + (LOCK+CHANGE+UNLOCK +CONV/ XES))


LOCK REQ - XES 5.23 2101826UNLOCK REQ - XES 0.89 358098CHANGE REQ - XES 0.04 16744SUSPENDS - IRLM 0.03 11906SUSPENDS - XES 0.00 0CONVERSIONS- XES 0.02 7204FALSE CONTENTIONS 0.00 0INCOMPATIBLE LOCKS 0.00 0NOTIFY MSGS SENT 0.01 2278

• False contention less than 1-3% of sum of all XES IRLM Requests

• P/L LOCKS XES% – Gives an idea how many of the locks

requests are propagated to XES (“degree” of sharing)

Local vs. Data Sharing locks

LOCKING TOTAL DATA SHARING TOT AL------------------- -------- ------------ --------TIMEOUTS 0 P/L-LOCKS(%) 5DEADLOCKS 0 P-LOCK REQ 193024ESCAL.(SHAR) 0 P-UNLOCK REQ 1916 24ESCAL.(EXCL) 0 P-CHANGE REQ 1 01MAX PG/ROW LCK HELD 91 LOCK - XES 739773LOCK REQUEST 14175179 UNLOCK-XES 193684UNLOCK REQST 138996 CHANGE - XES 101


UNLOCK REQST 138996 CHANGE - XES 101QUERY REQST 0 SUSP - IRLM 19 6CHANGE REQST 0 SUSP - XES 5 7OTHER REQST 0 CONV - XES 31 7TOTAL SUSPENSIONS 12337 FALSE CONT 7 89

LOCK SUSPENS 3 INCOMP.LOCK 0IRLM LATCH SUSPENS 12334 NOTIFY SENT 0OTHER SUSPENS 0

At last – Finally – RejoiceRequires PK85543 (DB2) PK85159(IRLM) OA12164(XES)

Local vs. Data Sharing locks - 2

• LOCK REQUESTS are L-lock requests

• P-LOCK REQ are the number of P-lock requests

– P-locks are ALWAYS sent to XES so they are included in the LOCK - XES counter

• LOCK – XES are the number of lock requests propagated to XES. This includes

– L-locks from LOCK REQUEST that need to be propagated (because of the interest) and

– P-LOCK REQ that are always propagated to XES


– P-LOCK REQ that are always propagated to XES

– Only those that are Synchronously propagated to XES.

• With current maintenance, we can now distinguish between suspended because of false contention vs. suspended because of XES heuristic conversions

BP Info in Accounting at Plan Level

• Not as many BP counters in DB2 accounting compared to DB2 statistics

BP0 BPOOL ACTIVITY AVERAGE TOTAL--------------------- -------- --------BPOOL HIT RATIO (%) 92.22 N/AGETPAGES 711.64 12605326BUFFER UPDATES 226.48 4011708SYNCHRONOUS WRITE 0.00 0


SYNCHRONOUS WRITE 0.00 0SYNCHRONOUS READ 49.71 880538SEQ. PREFETCH REQS 0.03 597LIST PREFETCH REQS 0.19 3313DYN. PREFETCH REQS 1.11 19595PAGES READ ASYNCHR. 5.67 100352

BP Info in Accounting at Plan Level - 2

• #Getpages is a good indicator of the amount of work that needs to be done by the transaction

– Likely to change if access path changes, or object gets more disorganized

• #SYNC READs are important as you wait for those (check against the CL3 data)

• A lot of prefetch activity can be sign of access path problem


• A lot of prefetch activity can be sign of access path problem

• BP info

– At the plan level – info for each BP

– At the package level – all BP together and only if accounting class 10 is active

GBP info

• GBP tuning is mostly done via stats info (not acctg)

• Check GBP hit/miss ratio for application

• Check application page P-lock activity

– # Page P-lock requests

GROUP BP0 AVERAGE TOTAL--------------------- -------- --------GBP-DEPEND GETPAGES 14707.80 2191462READ(XI)-DATA RETUR 0.32 47READ(XI)-NO DATA RT 0.00 0READ(NF)-DATA RETUR 101.45 15116READ(NF)-NO DATA RT 313.67 46737PREFETCH PAGES READ 9.48 1413CLEAN PAGES WRITTEN 0.00 0UNREGISTER PAGE 19.46 2899ASYNCH GBP REQUESTS 9.31 1387EXPLICIT X-INVALID 0.00 0


– # Page P-lock requests

– # Page P-lock suspensions

– Type of page P-lock SM, data index leaf

• There is no GBP information at the package level

ASYNCH SEC-GBP REQ 0.00 0PG P-LOCK LOCK REQ 497.81 74174

SPACE MAP PAGES 17.58 2620DATA PAGES 0.80 119INDEX LEAF PAGES 479.43 71435

PG P-LOCK UNLOCK REQ 146.15 21776PG P-LOCK LOCK SUSP 1.17 175

SPACE MAP PAGES 0.58 87DATA PAGES 0.05 8INDEX LEAF PAGES 0.54 80

WRITE AND REGISTER 176.83 26347WRITE & REGISTER MULT 17.73 2642CHANGED PAGES WRITTEN 304.57 45381

GBP Read Tuning

– Local BP search -> GBP search -> DASD I/O

GROUP BP14 QUANTITY /SECOND /THREAD /COMMIT

----------------------------- -------- ------- ------- -------

...

SYN.READ(XI)-DATA RETURNED 1932.00 0.09 0.01 0.00

SYN.READ(XI)-NO DATA RETURN 39281.6K 1823.66 236.31 79.22

SYN.READ(NF)-DATA RETURNED 22837.00 1.06 0.14 0.05

SYN.READ(NF)-NO DATA RETURN 6955.8K 322.93 41.85 14.03


– Local BP search -> GBP search -> DASD I/O

– SYN.READ(NF) = Local Buffer Pool miss

– SYN.READ(XI) = Local Buffer Pool hit but cross-invalidated buffer

– Most data should be found in GBP >> if not, GBP may be too small or pages have been removed because of directory entry reclaims

Sync.Read(XI) miss ratio should be < 10%ROTROT

TOTAL SYN.READ(XI) = SYN.READ(XI)-DATA RETURNED + SYN.READ(XI)-NO DATA RETURNSync.Read(XI) miss ratio = SYN.READ(XI)-NO DATA RETURN / TOTAL SYN.READ(XI)

DSQGESQM VALUE DSQGESQM TIMES DSQGESQM AVER AGE TIME AVG.EV TIME/EVENT

------------------ ------------------ --------------- --- ------------ ------------------ ------------ ------ ------------

TYPE PACKAGE ELAP-CL7 TIME-AVG 2:37.547703 LOCK/LATCH 0.017786 42.47 0.000419

CP CPU TIME 2:21.224699 IRLM LOCK+LATC H 0.000335 0.20 0.001641

LOCATION USTSYIPT0D AGENT 2:21.224699 DB2 LATCH 0.017451 42.27 0.000413

COLLECTION ID Q PAR.TAS KS 0.000000 SYNCHRONOUS I/O 0.634727 698.49 0.000909

PROGRAM NAME DSQGESQM SE CPU TIME 0.000000 OTHER READ I/O 0.494961 404.7 5 0.001223

SUSPENSION-CL8 1.278264 OTHER WRITE I/O 0.026058 9.56 0.002726

ACTIVITY TYPE NONNESTED AGENT 1.278 264 SERV.TASK SWITCH 0.104705 16.77 0.006242

ACTIVITY NAME 'BLANK' PAR.TAS KS 0.000000 ARCH.LOG(QUIESCE) 0.000000 0.00 N/C

SCHEMA NAME 'BLANK' NOT ACCOU NTED 15.044740 ARCHIVE LOG READ 0.000000 0.00 N/C

SUCC AUTH CHECK 0 AVG.DB2 E NTRY/EXIT 112.91 DRAIN LOCK 0.000000 0.00 N/C

OCCURRENCES 93 DB2 ENTRY/EXIT 10501 CLAIM REL EASE 0.000000 0.00 N/C

NBR OF ALLOCATIONS 649 PAGE LAT CH 0.000027 0.05 0.000505

SQL STMT - AVERAGE 13.94 CP CPU SU 5066356.40 NOTIFY MESSAGES 0.000000 0.00 N/C

SQL STMT - TOTAL 1296 AGENT 5066356.40 GLOBAL CONTENTION 0 .000000 0.00 N/C

NBR RLUP THREADS 93 PAR.TASKS 0.00 TCP/IP LO B XML 0.000000 0.00 N/C

SE CPU SU 0.00 TOTAL CL8 SUSPE NS. 1.278264 1.2K 0.0 01091

Package Level accounting information


SE CPU SU 0.00 TOTAL CL8 SUSPE NS. 1.278264 1.2K 0.0 01091

DSQGESQM AVERAGE TIME AV.EVENT DSQGESQM AVERAGE TIME AV.EVENT

------------------------- ------------ -------- ------ ------------------- ------------ --------

GLOBAL CONTENTION L-LOCKS 0.000000 0.00 GLOBAL CONTENTION P-LOCKS 0.000000 0.00

PARENT (DB,TS,TAB,PART) 0.000000 0.00 PAGESET/PARTITION 0.000000 0.00

CHILD (PAGE,ROW) 0.000000 0.00 PAGE 0.000000 0.00

OTHER 0.000000 0.00 OTHER 0.000000 0.00 CL8 info

OCCURRENCES - # IFCID 239 entries for this package NBR OF ALLOCATIONS - # times switched to the packageNBR RLUP THREADS (new in V10)package level detail info with rollup (not class10) - no longer *ROLLUP*[In OMPE averages are based on # NBR RLUP THREADS (= #OCC with no rollup) ]

• Also SE CPU time at package level

DSQGESQM AVERAGE TOTAL DSQGESQM AVERAGE TOTAL DSQGESQM AVERAGE TOTAL

------------------ -------- -------- ------------------- -------- -------- --------------------- -------- --------

SELECT 0.00 0 BPOOL HIT RATIO (%) 99.25 N/A TIMEOUTS 0.00 0

INSERT 0.12 11 GETPAGES 4185.6K 389262K DEADLOCKS 0.00 0

UPDATE 0.04 4 BUFFER UPDATES 890.5K 82816786 ESCAL.(SHARED) 0.00 0

DELETE 0.06 6 SYNCHRONOUS WRITE 0.05 5 ESCAL.(EXCLUS) 0.04 4

SYNCHRONOUS READ 698.41 64952 MAX PG/ROW LOCKS HELD 477.02 44363

DESCRIBE 0.00 0 SEQ. PREFETCH REQS 2036.97 189438 LOCK REQUEST 1904.25 177095

PREPARE 0.00 0 LIST PREFETCH REQS 4.84 450 UNLOCK REQUEST 636.86 59228

OPEN 0.00 0 DYN. PREFETCH REQS 1976.46 183811 QUERY REQUEST 0.00 0

FETCH 0.00 0 PAGES READ ASYNCHR. 30603.12 2846090 CHANGE REQUEST 224.92 20918

CLOSE 0.00 0 OTHER REQUEST 0.00 0

SQL BP totals Locking

Package level Accounting - acctg CL10


CLOSE 0.00 0 OTHER REQUEST 0.00 0

TOTAL SUSPENSIONS 0.20 19

LOCK TABLE 0.00 0 LOCK SUSPENSIONS 0.00 0

CALL 0.00 0 IRLM LATCH SUSPENS. 0.20 19

OTHER SUSPENS. 0.00 0

• Only available when accounting trace class 10 is active • Not available when rollup accounting is in effect (in plan for Vnext)• BP info is not at BP level, but all BP together• Note that sum SQL here is not same as SQL STMT TOTAL

(CL10 section does not include things like SET stmts, so expect <= )

DB2 V8 Enhancements

• Better control over basic accounting intervals

– KEEPDYNAMIC YES will not prevent completing the accounting interval at commit boundary

– At the same time change of WLM enclave is enabled for the new transaction

• Accounting roll-up (for RRS and DDF work)

– Addressing the issue about the large volume of accounting data

– Aggregating accounting data for the same values of the client-side identifiers (end


– Aggregating accounting data for the same values of the client-side identifiers (end user ID, end user transaction/application name, end user workstation name)

• zparm ACCUMUID controls which combination of IDs is used

• zparm ACCUMACC controls when the aggregated accounting records are externalized

DB2 V10 Enhancements

• Better accounting rollup (for RRS and DDF work)

– Still don’t like it

– Packages not all lumped together (up to 24 packages will be rolled up separately – if more rest is rolled up into a single package)

– DDF info (up to 4 unique QLACs will be rolled up separately – if more the rest is rolled up into a single one)


• Compression of SMF records (SMFCOMP=YES ZPARM)

– Should be enough to avoid a need to use ACCUMACC

– Small overhead (1%), great compression ratio (60-80%)

• Separating DB2 internal latch and IRLM lock/latch suspensions (see before)

DB2 V10 Enhancements

• Aggregate accounting statistics

– Introduced by APAR PM62797 (8/12)

– (OMPE APAR PM72949 to support this in stats reports)

– High level info accounting information per connection type

– New statistics class 9 will write IFCID369 (if acctg trace class 1 is also active)

– Written at 1 min (stats) interval

– Basically class 1,2,3 time info per connection type


– Basically class 1,2,3 time info per connection type

• Fields from DSNDQWAC and DSNDQWAX

– ‘Draft’ sample OMPE report

CONNTYPE CL1 ELAPSED CL1 CPU CL1 SE CPU CL 2 ELAPSED CL2 CPU CL2 SE CPU CL3 SUSP CL 2 NOT ACC THREADS-------- ------------ ------------ ------------ ------- ----- ------------ ------------ ------------ ---------- -- --------BATCH 1:14.326062 8.882753 0.000000 1 :14.234073 8.879979 0.000000 1:05.997069 0.000000 3.00CICS N/P N/P N/P N/P N/P N/P N/P N/P 0.00DDF 3:20.927303 9.873839 8.240478 55.917303 8.697064 6.742272 39.852088 7.368151 15895.00IMS N/P N/P N/P N/P N/P N/P N/P N/P 0.00RRSAF 1:04.091521 0.143129 0.000000 0.050749 0.044200 0.000000 0.000062 0.006487 10.00UTILITY 21.317271 0.712988 0.031434 1.224435 0.189245 0.031434 0.987360 0.047829 72.00

© Copyright IBM Corporation 2012. All rights reser ved.U.S. Government Users Restricted Rights - Use, dupli cation or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

THE INFORMATION CONTAINED IN THIS PRESENTATION IS P ROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND AC CURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION,

Disclaimer

© 2012 IBM Corporation110

PRESENTATION, IT IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RES PONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, NOR SHALL HAVE THE EFFECT OF, CREATING ANY WARRANTIES OR REPRESENTATIONS FROM IBM (OR ITS SUPPLIERS OR LICEN SORS), OR ALTERING THE TERMS AND CONDITIONS OF ANY AGREEMENT OR LICENSE GOVERNING THE USE OF IBM P RODUCTS AND/OR SOFTWARE

IBM, the IBM logo, ibm.com, and DB2 are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml

db2 accounting reporting

Data & Analytics