analyzing performance using statistics and accounting reports · (class 1 se cpu - class2 se cpu)...

Analyzing performance using

Statistics & Accounting reports

CRISTIAN MOLARO [email protected]

Guide Share Europe - DB2 Leuven, Belgium

March 2016

Analyzing performance using Statistics and Accounting reports | Introduction to the DB2 instrumentation

Agenda

1 Introduction to the DB2 instrumentation

2 DB2 traces

3 Understanding and working with reports

4 Using Statistics & Accounting information

5 Summary

Cristian Molaro - [email protected] 2014 2 / 89


About the author

Cristian Molaro, MConsulting, Belgium• Independent DB2 specialist and an IBM Gold Consultant• Recognized by IBM as an Information Champion in 2009, 2010, 2011,

2012, 2013, and 2014• Recognized by IBM as "TOP" EMEA Consultant in 2011 / 2013• Co-author of 9 IBM Redbooks related to DB2, including:

I DB2 10 for z/OS Performance TopicsI DB2 11 for z/OS Technical overviewI DB2 11 for z/OS Performance TopicsI Holder of the merit badge “Platinum IBM Redbook Author”

• 12 books published about DB2 for z/OS and DB2 for LUW



Disclaimer

PLEASE BE AWARE THAT THE ACTUAL PROGRAMMING TECHNIQUES,ALGORITHMS AND ALL NUMERICAL PARAMETERS USED IN EXAMPLES GIVEN INTHIS PRESENTATION ARE SUBJECT TO CHANGE AT SOME FUTURE DATE EITHERBY A NEW VERSION OF DB2, A NEW RELEASE, A SMALL PROGRAMMINGENHANCEMENT (SPE) OR A PROGRAMMING TEMPORARY FIX (PTF).THE INFORMATION CONTAINED IN THIS PRESENTATION HAS NOT BEENSUBMITTED TO ANY FORMAL REVIEW AND IS DISTRIBUTED ON AN “AS IS” BASISWITHOUT ANY WARRANTY EITHER EXPRESS OR IMPLIED. THE USE OF THISINFORMATION OR THE IMPLEMENTATION OF ANY OF THESE TECHNIQUES IS ACUSTOMER RESPONSIBILITY AND DEPENDS ON THE CUSTOMER’S ABILITY TOEVALUATE AND INTEGRATE THEM INTO THE CUSTOMER’S OPERATIONALENVIRONMENT. WHILE EACH ITEM MAY HAVE BEEN REVIEWED FOR ACCURACY INA SPECIFIC SITUATION, THERE IS NO GUARANTEE THAT THE SAME OR SIMILARRESULTS WILL BE OBTAINED ELSEWHERE. CUSTOMERS ATTEMPTING TO ADAPTTHESE TECHNIQUES TO THEIR OWN ENVIRONMENTS DO SO AT THEIR OWN RISK.DB2 IS A TRADEMARK OF INTERNATIONAL BUSINESS MACHINE CORPORATION.THIS PRESENTATION USES MANY TERMS THAT ARE TRADEMARKS. WHEREVERWE ARE AWARE OF TRADEMARKS THE NAME HAS BEEN SPELLED IN CAPITALS.



Agenda


2 DB2 traces



5 Summary



What release of DB2 are you running in Production?

1 DB2 11 for z/OS2 DB2 10 for z/OS3 DB2 9 for z/OS (End of support 27-Jun-2014)4 DB2 8 for z/OS (End of support 30-Apr-2012)5 DB2 7 for z/OS (End of support 30-Jun-2008)6 Older DB2 version?



DB2 Accounting vs DB2 Statistics Data

DB2 statistics trace records• Contain information about the activity of the entire DB2 subsystem• Including the DB2 system address spaces

I Such as MSTR, DBM1, IRLM, DIST

DB2 accounting trace records• Contain information about the activity performed by a DB2 thread or

‘transaction’I The scope is defined by the “accounting interval”I A record can contain info about several threads in case of roll-up



How CPU time is reported : Accounting vs Statistics



DB2 Allied thread

Allied threads: threads connected to DB2 from TSO, batch, IMS,CICS, CAF, or RRSAF



DB2 distributed workload

DBATs: (Distributed database access threads) threads that areconnected through a network



Asynchronous agents in DB2

Asynchronous agents



Synchronous and Asynchronous DB2 Agents

The main synchronous agent is usually a TCB that originates in theuser’s address spaceIt is used to execute the SQL statement

• DDF: it is an (independent enclave) SRB instead of a TCB, whichoriginates in the DDF address space

• Parallelism: the TCB (SRB for DDF) is helped by synchronous SRBs

The asynchronous agent normally performs system or ancillaryfunctionsThe asynchronous CPU load is typically 10% of the synchronous loadin most systems

• It can be higher in DDF and/or data sharing• Can indicate a change in workload or problems introduced by

maintenance



Synchronous and Asynchronous Agents CPU times

The synchronous CPU load is recorded in accountingThe asynchronous CPU load is recorded in statistics

• If a thread has to wait for the async. agent to complete its work, thiselapsed time is typically recorded in an accounting class 3 wait timecounter

About DDF• DDF work does not have an agent outside the DB2 address spaces to

charge for the work• The CPU time used by synchronous agents for DDF work also shows

up in DB2 statisticsI DDF enclave SRB time

• Carefull with double accounting



DB2 Events and its context

IMPORTANTOut of its context, the information about the occurence of many eventsmay be useless... you need to put all in perspective!

The performance interval:



Performance Quick winsInvestigate common DB2 performance bottlenecks

• CPU• I/O: Synchronous I/O, Logging• Concurrency: Locking• Real Storage

I Not enough Virtual storage, DBM1 constraint BTB

20-80 rule :• The Pareto principle applies very well to DB2 performance Roughly

80% of the effects come from 20% of the causes• Example: Top x CPU consumers Top x more frequently executed

transactions Top x more active End Users


Analyzing performance using Statistics and Accounting reports | DB2 traces

Agenda


2 DB2 traces



5 Summary



DB2 Performance Oriented traces

ACCOUNTINGSTATISTICSMONITORPERFORMANCE (manually started)Start traces using system parameters:

• SMFACCT: define if and what Accounting traces to send to SMFI NO, YES, list, *

• SMFSTAT: define if and what Statistics traces to send to SMFI NO, YES, list, *

• STATIME: specifies the time between statistics collectionsI Set to 1

Other traces• AUDIT• GLOBAL



Accounting classes

In DB2 Time distribution is of capital importance for performancetuning

CLASS 1 contains most of the accounting data but NO time distribution



Statistics classes

System-wide traceStart with DB2 by setting the zParm SMFSTAT

• SMFSTAT =YES start classes 1, 3, 4, 5, and 6



Performance classes



What classes to start?

Continuous performance monitoring• ACCOUNTING CLASS 1, 2, and 3 destination SMF• STATISTICS CLASS 1, 3, 4, 5 and 6 destination SMF

Package information• Start Accounting Class 7 and 8 (recommended)• Detailed Accounting information for packages: Class 10

I Warning: overhead!I Use when needed

Detailed performance monitoring• Keep cost of detailed monitoring low!

TIPUse zParms to automatically start DB2 traces



Trace overhead

Typical trace costs• Accounting Class 1,2,3,7,8 + Statistics 1,3,4,5,6

I ~ 2 to 5%• Audit

I ~ < 5%• Performance

I Range between 20 to 100%• Global

I ~ up to 100% or more. . .

Performance and Global traces can be quite resource intensiveEnable only the minimal trace and audit classes that you need

• You can enable more detailed traces only when you encounter specificperformance problems

Beware of possible overhead introduced by Performance monitorsCristian Molaro - [email protected] 2014 22 / 89


START TRACE COMMAND

START TRACE example>>-START TRACE--(-+-PERFM---+-)--+-------------------------+----> +-ACCTG---+ | .-,-------. | +-STAT----+ | V | | +-AUDIT---+ '-DEST--(----+-GTF-+-+--)-' '-MONITOR-' +-SMF-+ +-SRV-+ +-OPn-+ '-OPX-'

>--+----------------------+--+---------------------+------------> '-| constraint block |-' '-| filtering block |-'

>--+------+--+-----------------+--+---------------------+------>< '-RMID-' '-COMMENT(string)-' | .-LOCAL-. | '-SCOPE-(-+-GROUP-+-)-'



The constraint and filtering blocks

Optional limits on the kinds of data that are collected by the traceThe filtering options are exclusionary equivalents to the correspondingconstraint optionsOnly the CLASS constraint option can be specified for starting astatistics traceA START TRACE command can contain multiple constraint optionsExamples

• PLAN( plan-name , . . . ) or XPLAN( plan-name , . . . )I Default is PLAN( * )

• AUTHID( authorization-id , . . . ) or XAUTHID( authorization-id , . . . )• USERID or XUSERID• CONNID or XCONNID• CORRID or XCORRID



Minimize CPU impact of DB2 traces

TIPUse trace qualifiers, specially for PERFORMANCE traces

BUT: with AUTHID and PLAN, related indirect functions like systemtasks are not recorded, i.e. prefetchCan use eXclude filtersUse Destination GTF for PERFORMANCE traces

• Default destination for PERFORMANCE traces• Allow immediate analysis Requires a running GTF STC• Warning: external dataset wraps around!

I Check dataset header

TIP

Use DEST(GTF) for high volume monitoring like PERFORMANCEIFCIDs



Start / stop traces

Example -START TRACE(MON) CLASS(30) IFCID(316,317,318) DEST(SMF)

DSNW130I -DBAT MON TRACE STARTED, ASSIGNED TRACE NUMBER 04DSN9022I -DBAT DSNWVCM1 '-START TRACE' NORMAL COMPLETION

-STOP TRACE(MON) TNO(04)

To start only those IFCIDs specified in the IFCID option, use traceclasses 30, 31 or 32Classes 30,31 and 32 have no predefined IFCIDsIf you do not specify the IFCID option, only those IFCIDs contained inthe activated trace classes are started

TIP: Classes 30, 31 and 32 are reserved for local use



Too much SMF information

If SMF cannot receive and store SMF records: data loss• Too much SMF records created by DB2• I/O problems delaying SMF offload

Reported in IFCID 1: MSTR stats Message in console

MSTR MESSAGEIEE366I NO SMF DATA SETS AVAILABLE--DATA BEING BUFFERED*IEE986E SMF HAS USED25% OF AVAILABLE BUFFER SPACE*IEE986E SMF HAS USED31% OF AVAILABLE BUFFER SPACE...*IEE986E SMF HAS USED93% OF AVAILABLE BUFFER SPACE*IEE986E SMF HAS USED100% OF AVAILABLE BUFFER SPACE*IEE979W SMF DATA LOST - NO BUFFER SPACE AVAILABLEDSNW133I -DBAT DSNWVSMF - TRACE DATA LOST, SMF NOT ACCESSIBLERC=28...DSNW123I -DBAT DSNWVSMF - TRACE RECORDING HAS BEEN RESUMED ON SMF



Reducing SMF volume with Accounting Rollup

zParm ACCUMACC controls whether and when DB2 accounting datais accumulated for DDF and RRSAF threads

• ACCUMACC=NO, default no effect• ACCUMACC = n, n defines the accumulation interval

zParm ACCUMUID defines the aggregation criteria• Value from 0 to 17• ACCUMUID=1, End user ID• Can be changed online

Applies only to DDF and RRSAF activity

STATISTICS REPORT ACCOUNTING ROLLUP QUANTITY /SECOND /THREAD /COMMIT --------------------------- -------- ------- ------- ------- ROLLUP THRESH RECS WRITTEN 0.00 0.00 0.00 0.00 STORAGE THRESH RECS WRITTEN 0.00 0.00 0.00 0.00 STALEN THRESH RECS WRITTEN 0.00 0.00 0.00 0.00 RECS UNQUALIFIED FOR ROLLUP 0.00 0.00 0.00 0.00



DB2 10: SMF compression

New in DB2 10Applies to all types of workloadsControlled by new system parameter SMFCOMP

• OFF (default): SMF trace records are not compressed• ON: Trace records written to SMF are compressed

The z/OS compression service CSRCESRV compresses everythingafter the SMF headerData Sharing environment: SMFCOMP has member scopePerformance measurements

• Minimal overhead; ~ 1% with accounting class 1, 2, 3, 7, 8, 10• Significant compression rate of 60% to 80%

TIPUse SMFCOMP instead of accounting roll-up



DB2 10 Accounting Enhancements

Separation of DB2 internal latch and IRLM lock/latch suspensionsBetter ACCUMACC accounting rollup for RRS and DDF work

• Reported at Plan level prior to DB2 10• Packages not all aggregated together• DDF Location information

Compression of SMF records• SMFCOMP=YES• Should be enough to avoid a need to use ACCUMACC• Small overhead with great compression ratio

Aggregate accounting statistics• High level info accounting information per connection type



Aggregated accounting statistics

Aggregated accounting data in Statistics• High level info accounting information per connection type• Basically class 1, 2,3 time info per connection type• Created at statistics interval: 1 minute

IFCID 369• Started with START TRACE command for STATISTICS CLASS(9)

STATISTICS REPORT LONGCPU TIMES TCB TIME PREEMPT SRB NONPREEMPT SRB CP CPU TIME PREEMPT IIP SRB CP CPU /COMMIT------------------------------- --------------- --------------- --------------- --------------- --------------- --------------SYSTEM SERVICES ADDRESS SPACE 19.529808 0.250513 3.999053 23.779374 17.988831 0.006812DATABASE SERVICES ADDRESS SPACE 1.508536 0.149497 0.039359 1.697391 6.389113 0.000486IRLM 0.000066 0.000000 2.256474 2.256540 0.000000 0.000646DDF ADDRESS SPACE 0.141870 1.262984 0.022711 1.427565 0.701908 0.000409

TOTAL 21.180280 1.662993 6.317596 29.160870 25.079851 0.008353

CONNTYPE CL1 ELAPSED CL1 CPU CL1 SE CPU CL2 ELAPSED CL2 CPU CL2 SE CPU CL3 SUSP CL2 NOT ACC QUANTITY-------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ --------BATCH 5:45.700412 17.102508 0.000000 4:32.481540 14.523955 0.000000 18.522110 3:59.443103 126.00CICS 3:35.843506 2.622503 0.000000 1:20.432247 2.010811 0.000000 17.077670 1:01.343766 144.00DDF 2.967722 0.431580 0.606128 2.042811 0.380759 0.526230 0.438922 1.223130 2826.00IMS N/P N/P N/P N/P N/P N/P N/P N/P 0.00RRSAF 1:01:02.9662 42.948526 0.000000 16:07.659323 39.667408 0.000000 3:09.608421 12:20.389787 282.00UTILITY 5:09.584462 6.927362 5.508104 2:23.883903 2.295560 5.508104 22.666686 1:58.921657 11.00,


Analyzing performance using Statistics and Accounting reports | Understanding and working with reports

Agenda


2 DB2 traces



5 Summary



DB2 monitors vs. DB2 reports

DB2 monitors• Very usefull for real-time problem investigation• Available at almost every data center

I Provides online monitoring informationI Generally provides a short term history view of the activityI Can introduce performance overhead by starting DB2 tracesI Too agressive online monitoring can show as CLASS 2 NOT

ACCOUNTED TIME

DB2 reports• Very usefull for in-depth analysis of• Application performance: accounting reports

I System level performance: statistics reportsI On paper or dataset

• Sometimes can be build based on online monitor colected data



Creating DB2 reports

Use case example• Daily accounting and statistics reports

I STATISTICS LONG REPORTI ACCOUNTING LONG REPORT

• Created in batch at 00:00• Stored in GDG for easier management

I Keep a large number of generations based on your needsI Example: rolling 90 days reports

• Space utilizationI Statistics: very lowI Accounting: depends on the details. Can be big.



The DB2 performance database

Very usefull for trend analysis and problem investigationQuickly access to historic dataUse SQL for querying accounting and statistics reports

• Sometimes easier access to data than using paper version of report

Can read data directly into a spreadsheet



OMPE input data for spreadsheets

New: OMPE report batch can be used to generate CSV filesCSV can be read in spreadsheet



The DB2 Accounting trace

The DB2 accounting trace provides the following types of information• Start and stop times• Number of commits and aborts• The number of times certain SQL statements are issued• Number of buffer pool requests• Counts of certain locking events• Processor resources consumed• Thread wait times for various events• RID pool processing• Distributed processing• Resource limit facility statistics

IMPORTANTAccounting records are cut at the accouting interval



Accounting Top Down analysisStart looking at highest level and then increase level of details

• CONNECTION TYPE is a good starting point

OMPE Command ExampleACCOUNTING REPORT LAYOUT(LONG) ORDER(CONNTYPE) EXCLUDE(PACKAGE(*))STATISTICS REPORT LAYOUT(LONG)

Then investigate the CONNECTION creating the problem• Plans / Transactions / AuthIDs

Then move to Package levelAccounting cannot provide STATEMENT level details

• Monitor• Performance traces

I CLASS(3) or IFCID 316-317-318Cristian Molaro - [email protected] 2014 38 / 89


DB2 accounting elapsed times

Class 1 elapsed time• Shows the duration of the accounting interval

I Depends on the application infrastructure

• Includes time spent in DB2 and in the application• Also referred to as application time

Class 2 elapsed time• It counts only the time spent in the DB2 address space during the

accounting interval• It represents the sum of the times from any entry into DB2 until the

corresponding exit from DB2• Also referred to as the time spent in DB2

Class 3 elapsed time• Wait time including I/O wait time, lock and latch wait time

Not Accounted: non identified time spent in DB2Cristian Molaro - [email protected] 2014 39 / 89


Thread activity time analysis



DB2 Accounting Classes 1,2,3 (or 1,7,8)



Class 3 suspension time



When the DB2 Accounting Data is Written ?

Thread deallocation (including abends)(Re)signon in case of thread reuse by RRS,IMS,CICSAt commit for RRS threads using accounting-interval COMMIT onsignon/ auth signon / context signonAt commit for DDF threads when CMTSTAT=INACTIVE and theDBAT thread goes inactive

• Things that prevent a DBAT thread from going inactive:I Touched a package that uses Keepdynamic(yes)I Active DGTT (not explicitly or implicitly dropped)I Open cursor with holdI Held LOB locator

If only reason for DBAT thread not going inactive is the use ofKEEPDYNAMIC(YES)

• Still cut accounting record (and reset enclave)• For applications like SAP



The DB2 accounting interval

Thread de-allocation



The DB2 accounting interval and COMMIT

COMMIT does not create an accounting record, except for:• Inactive connections in DRDA• RRS using accounting-interval COMMIT (and no open held cursors)



The DB2 accounting interval and thread reuse

An accounting record is created at thread reuse• Applies to IMS and CICS



Accounting interval anomalies with thread reuse

Anormally long Elapsed Time in DB2 Accounting• Thread reuse and idle time until record cut



Accounting interval and distributed threads

Accounting record cut when the connection goes inactive• CMTSTAT = INACTIVE



Accounting rollup

Accounting rollup accumulates accounting information• DDF and RRS only



How to read an Accounting report?

Start by inspecting the Accouting Report summary block:

OMPE ACCOUNTING REPORT LONG ELAPSED TIME DISTRIBUTION CLASS 2 TIME DISTRIBUTION ---------------------------------------------------------------- ---------------------------------------------------------------- APPL !====================================> 72% CPU !====> 9% DB2 !==> 4% SECPU ! SUSP !===========> 23% NOTACC !===> 7% SUSP !==========================================> 84%

Not in DB2 time = Class 1 – Class 2



In DB2 Time = Accounting CLASS 2 & 3 / 7 & 8

Where goes the time in DB2?• CPU• Wait• Unknown



DB2 Class 2 Not Accounted TimeDB2 Class 2 Not Accounted Time = DB2 Class 2 Elapsed time - DB2

Class 2 CPU time - DB2 Class 3 suspension time

It represents time that DB2 is unable to account forAlso known as Not Accounted Time or Not Accounted in DB2

IMPORTANT: It is not the same as Suspension Time!

What is an acceptable value?• Non CPU constraint environments: ~ 10 %• CPU constraints environments? A lot higher.

ACCOUNTING LONG REPORTELAPSED TIME DISTRIBUTION CLASS 2 TIME DISTRIBUTION ---------------------------------------------------------------- --------------------------------------------------------------APPL | CPU | DB2 |================================================> 97% SECPU | SUSP |=> 3% NOTACC |================================================> 97% SUSP |=> 3%



DB2 Class 2 Not Accounted Time

DB2 Class 2 Not Accounted TimeDB2 Class 2 Elapsed time - DB2 Class 2 CPU time - DB2 Class 3suspension time

Most common reason: waiting for CPU• High overall CPU utilization• DB2 running with Low-priority• How to fix:

I Review WLM policyI Running in a WLM Ressource Group? Capping?I Reduce CPU or increase CPU capacity

Too much detailed online tracing in performance monitorHigh MVS pagingHSM RecallFull list: http://goo.gl/xkTd43



WLM Resource Groups

Describes the amount of CPU available to a specific service classYou can use a RG to

• Limit (capping) the amount of CPU capacity available to a SCI May dramatically increase DB2 Not Accounted time

• Guarantee some minimum CPU capacity to the service classesThere are three types of WLM Resource Groups:

• Type 1: Capacity is specified in CPU SU per second• Type 2: Capacity is specified as a percentage of the LPAR• Type 3: Capacity is specified as a percentage of a single CP

----------------------------------------------------------------------------------------------------------------------Formats DEFAULT ALERTS CPU IO PERF PROC STG USER WLM ----------------------------------------------------------------------------------------------------------------------Cmd Jobname | ASIDD Workload Srvclass Srvp Resgroup Server Quiesce CSA E-CSA SQA E-SQA C-Stg Omvs IODelta CPUDelta CRIS01 285 TSO TSO 1 NO NO 136 2.25K 96 368 2.84K 0.000000 PERF10 288 TSO TSO 1 NO NO 136 2.25K 96 368 2.84K 0.000000 CRISB101 348 BATCH BATCHMED 2 RG10MSU NO NO 8.39K 192 8.58K 1399 0.212473 TOTO10 355 TSO TSO 1 NO NO 136 2.25K 96 368 2.84K 0.000000 MACOS2 361 TSO TSO 1 NO NO 136 6.25K 96 368 6.84K 0.000000 CRISREO 367 BATCH BATCHHI 2 RG30MSU NO NO 144 144 1855 0.015792*********************************************************** End of Data **********************************************



Not accounted time and parallelism

With query or utility parallelism• Not Accounted only for the originating task



Accounting Distributed workloadsTime in DB2: Class 2 nonnested ET + Class 1 nestedactivity ET [stored procedure + UDF + Trigger] +nonnested (Class 1 CP CPU - Class 2 CP CPU) + Nonnested(Class 1 SE CPU - Class2 SE CPU)

ACCOUNTING REPORT LONG AVERAGE APPL(CL.1) DB2 (CL.2) IFI (CL.5) CLASS 3 SUSPENSIONS AVERAGE TIME AV.EVENT HIGHLIGHTS ------------ ---------- ---------- ---------- -------------------- ------------ -------- -------------------------- ELAPSED TIME 0.592949 0.309976 N/P LOCK/LATCH(DB2+IRLM) 0.000347 0.59 #OCCURRENCES : 199624 NONNESTED 0.592408 0.309913 N/A IRLM LOCK+LATCH 0.000263 0.00 #ALLIEDS : 0 STORED PROC 0.000541 0.000064 N/A DB2 LATCH 0.000084 0.59 #ALLIEDS DISTRIB: 0 UDF 0.000000 0.000000 N/A SYNCHRON. I/O 0.124586 84.59 #DBATS : 199624 TRIGGER 0.000000 0.000000 N/A DATABASE I/O 0.122127 84.43 #DBATS DISTRIB. : 0 LOG WRITE I/O 0.002459 0.16 #NO PROGRAM DATA: 3 CP CPU TIME 0.010708 0.010665 N/P OTHER READ I/O 0.141491 35.70 #NORMAL TERMINAT: 199624 AGENT 0.010708 0.010665 N/A OTHER WRTE I/O 0.000452 0.02 #DDFRRSAF ROLLUP: 0 NONNESTED 0.010695 0.010653 N/P SER.TASK SWTCH 0.000799 0.07 #ABNORMAL TERMIN: 0 STORED PRC 0.000013 0.000012 N/A UPDATE COMMIT 0.000011 0.06 #CP/X PARALLEL. : 0 UDF 0.000000 0.000000 N/A OPEN/CLOSE 0.000008 0.00 #IO PARALLELISM : 0 TRIGGER 0.000000 0.000000 N/A SYSLGRNG REC 0.000060 0.00 #INCREMENT. BIND: 0 PAR.TASKS 0.000000 0.000000 N/A EXT/DEL/DEF 0.000085 0.00 #COMMITS : 188367 OTHER SERVICE 0.000635 0.00 #ROLLBACKS : 11572 SECP CPU 0.000676 N/A N/A ARC.LOG(QUIES) 0.000000 0.00 #SVPT REQUESTS : 0 LOG READ 0.000000 0.00 #SVPT RELEASE : 0 SE CPU TIME 0.014287 0.014232 N/A DRAIN LOCK 0.000000 0.00 #SVPT ROLLBACK : 0 NONNESTED 0.014287 0.014232 N/A CLAIM RELEASE 0.000000 0.00 MAX SQL CASC LVL: 1 STORED PROC 0.000000 0.000000 N/A PAGE LATCH 0.000004 0.00 UPDATE/COMMIT : 0.15 UDF 0.000000 0.000000 N/A NOTIFY MSGS 0.000000 0.00 SYNCH I/O AVG. : 0.001473 TRIGGER 0.000000 0.000000 N/A GLOBAL CONTENTION 0.000000 0.00 COMMIT PH1 WRITE I/O 0.000000 0.00



Block fetchDB2 uses a block fetch to group the rows that a query retrieves intoas large a "block" of rowsThen transmits the block over the network

• By sending multiple rows in a block, DB2 avoids sending a message forevery row

ACCOUNTING REPORT LONG ---- DISTRIBUTED ACTIVITY ----------------------------------------------------------------------------------------------------- REQUESTER : ::10.123.0.11 #COMMIT(1) RECEIVED: 3825 MESSAGES SENT : 11.16 ROWS SENT : 1641.06 PRODUCT ID : DB2 #ROLLBK(1) RECEIVED: 7666 MESSAGES RECEIVED: 11.16 BLOCKS SENT : 8.16 METHOD : DRDA PROTOCOL SQL RECEIVED : 8.41 BYTES SENT : 220958.47 #DDF ACCESSES: 15316 CONV.INITIATED : 0.50 BYTES RECEIVED : 1174.92 #RLUP THREADS: 15316 #THREADS INDOUBT : 0 #COMMIT(2) RECEIVED: N/A TRANSACTIONS RECV. : N/A #PREPARE RECEIVED: N/A MSG.IN BUFFER: N/A #BCKOUT(2) RECEIVED: N/A #COMMIT(2) RES.SENT: N/A #LAST AGENT RECV.: N/A #FORGET SENT : N/A #COMMIT(2) PERFORM.: N/A #BACKOUT(2)RES.SENT: N/A #BACKOUT(2)PERFORM.: N/A

Blocking is crucial for distributed performanceTIP: The OPTIMIZE for x ROWS clause is used to control bothaccess path selection and DRDA blocking



Enabling Block fetch for distributed applications

DB2 can use different types of block fetch• Limited block fetch• Continuous block fetch

Block fetch is used only with cursors that do not update or delete dataTo ensure that DB2 uses block fetch

• Add the FOR FETCH ONLY or FOR READ ONLY clause

CURRENTDATA for remote access• CURRENTDATA YES: turns off block fetching for ambiguous cursors• The data returned is current with the source object• Turning on block fetch offers best performance

IMPORTANT

BIND applications with ISOLATION(CS) and CURRENTDATA(NO)


Analyzing performance using Statistics and Accounting reports | Using Statistics & Accounting information

Agenda


2 DB2 traces



5 Summary



RID failures

RID pool used for record identifier (RID) processing• Enforcing unique keys for multi-row updates• List prefetch, including single index list prefetch access paths• Multiple index access paths• Hybrid joins

All concurrent work shares the RID poolMAXRBLK controls the maximum size of the RID pool

• Limit is 10,000 MB.RID overflow scenarios

• Concurrent queries each consuming shared RID pool• Single query requesting > 25% of table or hitting RID pool limit

DB2 9 will fallback to Tablespace ScanDB2 10 will continue by writing new RIDs to workfile

• Avoid the performance problems of falling back to TS-Scan• Has no effect on access path selection



RID failures in reports

STATISTICS report----RID LIST----------------- -----QTY /MINUTE /THREAD /COMMRID CURRENT BLOCKS ALLOCATE 52RID HIGH BLOCKS ALLOCATED 2438RID TERM OVER RDS LIMIT 2065 1.44 0.05 0.0RID TERM TOO MANY CONCURR 0 0.00 0.00 0.0RID TERM - NO STORAGE 0 0.00 0.00 0.0RID TERM - OVER DM LIMIT 0 0.00 0.00 0.0

ACCOUNTING REPORT LONG CONNTYPE: DRDA DATA CAPTURE AVERAGE TOTAL DATA SHARING AVERAGE TOTAL QUERY PARALLELISM AVERAGE TOTAL ----------------- -------- -------- ------------------- -------- -------- ---------------------------- -------- -------- IFI CALLS MADE N/P N/P GLOBAL CONT RATE(%) N/C N/A MAXIMUM MEMBERS USED N/A 0 RECORDS CAPTURED N/P N/P FALSE CONT RATE(%) N/C N/A MAXIMUM DEGREE N/A 0 ...... LOGGING AVERAGE TOTAL ROWID AVERAGE TOTAL RID LIST AVERAGE TOTAL ------------------- -------- -------- ------------- -------- -------- ------------------- -------- -------- LOG RECORDS WRITTEN 84.77 16921873 DIRECT ACCESS 0.00 0 USED 192.94 38516147 TOT BYTES WRITTEN 48211.94 9624259K INDEX USED 0.00 0 FAIL-NO STORAGE 0.00 0 LOG RECORD SIZE 568.75 N/A TS SCAN USED 0.00 0 FAIL-LIMIT EXCEEDED 0.00 0



Fixing the problemOften caused by:

• Statistics inadequacy• Use of LIKE, range predicates with HOST variables

Often solved by Runstats + REBIND• Optionally: OPTIMIZE FOR 1 ROW



Dynamic Statement Cache

Can help to avoid significant cost of a Dynamic Stmt full prepare2 levels of caching

• Global Dynamic Statement Cache• Local Dynamic Statement Cache

GDSC: Global Dynamic Stmt Cache• Enabled by System Parameter CACHEDYN=YES• No prepared STMT is kept in storage across COMMIT

LDSC: Local Dynamic Stmt Cache• BIND option KEEPDYNAMIC(YES)• Keeps STMT in thread across COMMIT

I Review MAXKEEPDI Should not be the default, but good for SAPI Distributed access: prevents DBAT to become inactive



Dynamic statement cache

STATISTICS report EDM POOL QUANTITY /SECOND /THREAD /COMMIT --------------------------- -------- ------- ------- -------...... DBD REQUESTS 6176.4K 71.54 118.72 2.50 DBD NOT FOUND 9255.00 0.11 0.18 0.00 DBD HIT RATIO (%) 99.85 N/A N/A N/A CT REQUESTS 28409.00 0.33 0.55 0.01 CT NOT FOUND 0.00 0.00 0.00 0.00 CT HIT RATIO (%) 100.00 N/A N/A N/A PT REQUESTS 400.8K 4.64 7.70 0.16 PT NOT FOUND 0.00 0.00 0.00 0.00 PT HIT RATIO (%) 100.00 N/A N/A N/A PKG SEARCH NOT FOUND 98.00 0.00 0.00 0.00 PKG SEARCH NOT FOUND INSERT 0.00 0.00 0.00 0.00 PKG SEARCH NOT FOUND DELETE 0.00 0.00 0.00 0.00 STATEMENTS IN GLOBAL CACHE 8130.51 N/A N/A N/A

What is the problem? EDM POOL QUANTITY /SECOND /THREAD /COMMIT --------------------------- -------- ------- ------- -------... STATEMENTS IN GLOBAL CACHE 0.00 N/A N/A N/A



CACHEDYN subsystem parameter

Controls whether prepared, dynamic SQL statements are to be cachedYES: Highly recommended for environments with dynamic sql

EDM pool size calculationDSNTIPC INSTALL DB2 - CLIST CALCULATIONS - PANEL 1===> You can update the DSMAX, EDMPOOL STATEMENT CACHE (if CACHE DYNAMIC is YES), EDM DBD CACHE, SORT POOL, and RID POOL sizes if necessary. Calculated Override 1 DSMAX - MAXIMUM OPEN DATA SETS = 20000 (1-200000) 2 DSNT485I EDM STATEMENT CACHE = 122880 K K 3 DSNT485I EDM DBD CACHE = 40960 K K 4 DSNT485I EDM SKELETON POOL SIZE = 81920 K K 5 DSNT485I EDM LIMIT BELOW THE BAR = 0 K K 6 DSNT485I BUFFER POOL SIZE = 109 M 7 DSNT485I SORT POOL SIZE = 10000 K K 8 DSNT485I MAX IN-MEMORY SORT SIZE = 1000 K K 9 DSNT485I RID POOL SIZE = 400000 K K10 DSNT485I DATA SET STORAGE SIZE = 26000 K11 DSNT485I CODE STORAGE SIZE = 38200 K12 DSNT485I WORKING STORAGE SIZE = 45024 K13 DSNT486I TOTAL MAIN STORAGE = 617 M M14 DSNT487I TOTAL STORAGE BELOW 16M = 1036 K WITH SWA ABOVE 16M LINE15 DSNT438I IRLM LOCK MAXIMUM SPACE = 2160 M, AVAILABLE = 2160 M

PRESS: ENTER to continue RETURN to exit HELP for more information



Storage ABOVE 2 GB bar

STATISTICS REPORT LONG (DB2 10) DBM1 STORAGE ABOVE 2 GB QUANTITY -------------------------------------------- ------------------ GETMAINED STORAGE (MB) 888.04 FIXED STORAGE (MB) 71.26 VARIABLE STORAGE (MB) 154.76 COMPRESSION DICTIONARY (MB) 196.81 IN USE EDM DBD POOL (MB) 124.49 IN USE EDM STATEMENT POOL (MB) 140.02 IN USE EDM RDS POOL (MB) N/A IN USE EDM SKELETON POOL (MB) 14.25 STAR JOIN MEMORY POOL (MB) N/A STORAGE MANAGER CONTROL BLOCKS (MB) 11.29 VIRTUAL BUFFER POOLS (MB) 2116.57 VIRTUAL POOL CONTROL BLOCKS (MB) 72.66 CASTOUT BUFFERS (MB) 0.00 SHARED GETMAINED STORAGE (MB) 5.70 SHARED FIXED STORAGE (MB) 8.02 RID POOL (MB) 47.00 SHARED VARIABLE STORAGE (MB) 1134.28 TOTAL AGENT LOCAL STORAGE (MB) 944.49 TOTAL AGENT SYSTEM STORAGE (MB) 95.61 TOTAL AGENT NON-SYSTEM STORAGE (MB) 848.88 DYNAMIC STMT CACHE CNTL BLKS (MB) 162.90 THREAD COPIES OF CACHED SQL STMTS (MB) N/A IN USE STORAGE (MB) 0.37 STATEMENTS COUNT 9.16 HWM FOR ALLOCATED STATEMENTS (MB) 7.58 STATEMENT COUNT AT HWM 440.00 DATE AT HWM 05/16/13 TIME AT HWM 04:01:40.11 THREAD PLAN AND PACKAGE STORAGE (MB) 34.06 ...



Monitoring performance

STATISTICS REPORT LONG DYNAMIC SQL STMT QUANTITY /SECOND /THREAD /COMMIT SUBSYSTEM SERVICES QUANTITY /SECOND /THREAD /COMMIT --------------------------- -------- ------- ------- ------- --------------------------- -------- ------- ------- ------- PREPARE REQUESTS 6573.4K 76.13 126.35 2.67 IDENTIFY 20495.00 0.24 0.39 0.01 FULL PREPARES 252.8K 2.93 4.86 0.10 CREATE THREAD 52025.00 0.60 1.00 0.02 SHORT PREPARES 6324.4K 73.25 121.57 2.56 SIGNON 66.00 0.00 0.00 0.00 GLOBAL CACHE HIT RATIO (%) 96.16 N/A N/A N/A TERMINATE 72595.00 0.84 1.40 0.03 ROLLBACK 2008.00 0.02 0.04 0.00 IMPLICIT PREPARES 5440.1K 63.01 104.57 2.21 PREPARES AVOIDED 0.00 0.00 0.00 0.00 COMMIT PHASE 1 0.00 0.00 0.00 0.00 CACHE LIMIT EXCEEDED 0.00 0.00 0.00 0.00 COMMIT PHASE 2 1796.7K 20.81 34.54 0.73 PREP STMT PURGED 35985.00 0.42 0.69 0.01 READ ONLY COMMIT 7489.00 0.09 0.14 0.00 LOCAL CACHE HIT RATIO (%) 0.00 N/A N/A N/A UNITS OF RECOVERY INDOUBT 0.00 0.00 0.00 0.00 UNITS OF REC.INDBT RESOLVED 0.00 0.00 0.00 0.00 CSWL - STMTS PARSED 4.00 4.00 4.00 4.00 SYNCHS(SINGLE PHASE COMMIT) 162.3K 1.88 3.12 0.07 CSWL - LITS REPLACED 2.00 2.00 2.00 2.00 QUEUED AT CREATE THREAD 0.00 0.00 0.00 0.00 CSWL - MATCHES FOUND 0.00 0.00 0.00 0.00 SUBSYSTEM ALLIED MEMORY EOT 4918.00 0.06 0.09 0.00 CSWL - DUPLS CREATED 1.00 1.00 1.00 1.00 SUBSYSTEM ALLIED MEMORY EOM 0.00 0.00 0.00 0.00 SYSTEM EVENT CHECKPOINT 315.00 0.00 0.01 0.00 HIGH WATER MARK IDBACK 313.00 0.00 0.01 0.00 HIGH WATER MARK IDFORE 168.00 0.00 0.00 0.00 HIGH WATER MARK CTHREAD 329.00 0.00 0.01 0.00

Expected values:• GDSC Hit ratio > 90-95%• LDSC Hit ratio > 70%



DB2 10 Literal replacement

Dynamic SQL with literals can be re-used in the DSCLiterals replaced with &

ExampleCache unfriendly SQL:

SELECT COL1, COL2, COL3 FROM CRIS.TBL1 WHERE ACCOUNT_NUMBER = 123;

Cache friendly SQL:

SELECT COL1, COL2, COL3 FROM CRIS.TBL1 WHERE ACCOUNT_NUMBER = &;



DB2 Literal replacement

How to enable:• CONCENTRATE STATEMENTS WITH LITERALS in PREPARE

ATTRIBUTES clause• Set LITERALREPLACEMENT in the ODBC initialization file• Set the keyword enableLiteralReplacement=’YES’ in the JCC Driver

Performance• Biggest performance gain for repeated SQL with different literals• Using parameter marker still provides best performance

ACCOUNTING REPORT LONG AVERAGE SU CLASS 1 CLASS 2 DYNAMIC SQL STMT AVERAGE TOTAL MISCELLANEOUS AVERAGE TOTAL ------------ -------------- -------------- -------------------- -------- -------- -------------------- -------- -------- CP CPU 582.46 580.12 REOPTIMIZATION 0.00 0 MAX STO LOB VAL (KB) 0.00 0 AGENT 582.46 580.12 NOT FOUND IN CACHE 0.70 138988 MAX STO XML VAL (KB) 0.01 1214 NONNESTED 581.77 579.50 FOUND IN CACHE 0.31 61900 STORED PRC 0.69 0.62 IMPLICIT PREPARES 0.00 0 UDF 0.00 0.00 PREPARES AVOIDED 0.00 0 TRIGGER 0.00 0.00 CACHE_LIMIT_EXCEEDED 0.00 0 PAR.TASKS 0.00 0.00 PREP_STMT_PURGED 0.12 24948 CSWL - STMTS PARSED 0.00 0 SECP CPU 36.72 N/A CSWL - LITS REPLACED 0.00 0 CSWL - MATCHES FOUND 0.00 0 SE CPU 777.55 774.52 CSWL - DUPLS CREATED 0.00 0



I/O suspension

Synchronous I/O suspensions: the most common reason for excessivewait times and long response timeAccounting report

• A single counter for Synchronous reads and writes, see BP info

Synchronous buffer pool reads• One or few consecutive pages are retrieved• Requested pages are not consecutive• Prefetch disabled via VPSEQT=0• Sequential Prefetch threshold (90% of VPSIZE) reached prefetch pages

stolen before the application gets to them

Synchronous buffer pool writes:• Immediate Write threshold (97.5 % of VPSIZE) reached more than two

checkpoints have passed without the page being written• Write engine not available• In data sharing case, an index split forces log write for non-leaf pages



Synchronous I/O Suspensions



Investigating I/O issues

High number of short suspensions• Excessive GetPages

I Use Explain to tune access path• Buffer pools contention

I Look at BP statistics• Disorganized data/indexes

I Check statistics and reorg / rebuild

Low number of long suspensions, i.e. large time per suspension• Investigate I/O response time• DASD contention PAV Disk replication, PPRC or SDRF• Control Unit cache misses CPU contention

IMPORTANT

First step is to investigate origin of high Synch I/O suspension: Highnumber of short suspensions? Low number of long suspensions?



Elongated suspension time with lack of CPU



Synchronous Log write I/O SuspensionsAll log buffers that have not been written are written at commitFor transactions that use 2 Phase Commit, Phase 1 of 2-phasecommitSynchronous write (to disk or GBP)

• Log write ahead protocolRunning out of log output buffersIdentity columns/sequences that are using NO CACHE (or low cachevalue) or ORDER is used (in data sharing)Index split in data sharing

• Reduce index page splits byI Non-zero freespace (FREEPAGE, PCTFREE) for random insert

(PCTFREE 0 for sequential insert)I Larger index page sizeI Asymmetric index page split

DB2 10 introduces up to 40% reduction in (dual) log I/O waitbecause of parallel log write I/O’s



Open / Close DS activity

Big data leads to more partitioningPartitioning increases the # of open datasetsDSMAX specifies the maximum number data sets that can be openat one time

STATISTICS REPORT OPEN/CLOSE ACTIVITY QUANTITY /SECOND /THREAD /COMMIT LOG ACTIVITY QUANTITY /SECOND /THREAD /COMMIT --------------------------- -------- ------- ------- ------- --------------------------- -------- ------- ------- ------- OPEN DATASETS - HWM 17060.00 N/A N/A N/A READS SATISFIED-OUTPUT BUFF 167.2M 1936.34 3213.52 67.80 OPEN DATASETS 17008.84 N/A N/A N/A READS SATISFIED-OUTP.BUF(%) 94.58 DS NOT IN USE,NOT CLOSE-HWM 17004.00 N/A N/A N/A READS SATISFIED-ACTIVE LOG 9584.2K 111.01 184.22 3.89 DS NOT IN USE,NOT CLOSED 16910.18 N/A N/A N/A READS SATISFIED-ACTV.LOG(%) 5.42 IN USE DATA SETS 98.66 N/A N/A N/A READS SATISFIED-ARCHIVE LOG 0.00 0.00 0.00 0.00 READS SATISFIED-ARCH.LOG(%) 0.00 DSETS CLOSED-THRESH.REACHED 0.00 0.00 0.00 0.00 TAPE VOLUME CONTENTION WAIT 0.00 0.00 0.00 0.00 DSETS CONVERTED R/W -> R/O 15831.00 0.18 0.30 0.01 READ DELAYED-UNAVAIL.RESOUR 0.00 0.00 0.00 0.00 ARCHIVE LOG READ ALLOCATION 0.00 0.00 0.00 0.00



To REBIND or not To REBIND?

Reasons for REBIND• To activate features that provide performance• Better scalability• Potentially better access path

Strongly recommended at DB2 migrationConcerns

• Potential performance degradation due to access path changes• Concurrency issues

I Breaking-in into packagesI Timeout

PLAN MANAGEMENT• Provides a safer REBIND strategy

Challenge: to identify packages with performance degradation• Difficult if working with many packages



DB2 PLAN MANAGEMENT: OVERVIEWREBIND . . . PLANMGMT(EXTENDED)

REBIND SWITCH(ORIGINAL) vs. SWITCH(PREVIOUS)



What DB2 11 performance feature shows after REBIND?

Feature Trigger

General code optimizations NoneCustomized code generation for sort NoneAccess against table with large # of partitions NoneREL(DEALLOCATE) optimization NoneDB2 latch reductions NoneBuffer Pool enhancements NoneMore zIIP processing None (enough zIIP capacity)DDF Sync Receive enhancements None (TCP/IP APAR)Data sharing performance improvements NoneCustomized code generation for column processing REBINDDecompression Improvement REBINDSQL Access Path Iimprovements REBIND



DB2 PLAN MANAGEMENT and APREUSE

Allows a package to reuse access paths for static SQLAPREUSE(NO) = DefaultAPREUSE(ERROR)

• Effectively operates at the package level• RC=8, Package processing ends

APREUSE(WARN) = NEW in DB2 11• Effectively operates at the statement level: like HINTs

I Access paths kept on all statements that accept the HINTI “Fresh” access paths for statements on which the HINT failedI All packages rebound successfully

• Some limitations apply: Consider the feature as being using HINTsI Will not always work: reported 1% to 5% “failure” ratio

APREUSE allows to obtain DB2 11 updated runtime structures withoutchanging access path



Mass REBIND using PLAN MANAGEMENT

DB2 user in process of migrating to DB2 10Focus is on CPU reduction: need REBIND for out-of-the-box savingsStrategy

• Mass REBIND of packages with PLAN MANAGEMENT• Identify performance degradation using accounting information• REBIND SWITCH problem packages• Investigate and fix root cause of performance degradation

REBIND command syntax exampleREBIND PACKAGE(PDB2PK.PCKGCE0) - OWNER(PDB2) - QUALIFIER(PDB2) - EXPLAIN(YES) - REOPT(NONE) - PLANMGMT(EXTENDED)



Finding performance degradation problemsCompare Total CPU and Average CPU per package

• Easy if using a Performance Waredhouse database

PGM TOTCPU_OLD AVGCPU_OLD TOTCPU_NEW AVGCPU_NEW-------- ----------- ----------- ----------- ----------- A2IF04 2.092 0.00036 2184.000 0.44067 CCB820 0.039 0.03935 0.360 0.18004CGOC28IG 0.223 0.01719 0.681 0.11360 SCPOIN6 0.632 0.05268 12.162 3.0405 SCTRID 0.393 0.00289 29.817 0.28670 VGSG15 0.043 0.04321 0.225 0.22501VGSGI9PN 0.075 0.07583 0.457 0.45756 VSAC37 7.746 0.04120 58.847 0.37482

Switch to original package when needed

REBIND SWITCH exampleREBIND PACKAGE(PDB2PK.A2IF04) - SWITCH(ORIGINAL)



Disabled SPROCs

SPROCs improve performance of column processingSPROCS may get invalidated

• Migration to a new release of DB2 without REBIND

DB2 has to build SPROCs dynamically at execution timeTypical CPU performance impact in 0 to 10% range

• Monitor BYPASS COL value in DB2 Statistics• IFCID 224 identifies plans and packages that need rebinding to

re-enable SPROCs

STATISTICS REPORT LONG MISCELLANEOUS VALUE ------------------------ ------------------ HIGH LOG RBA 0000E9E947F0D6ED BYPASS COL 58091.00 MAX SQL CASCADING LEVEL 0.00 MAX STOR LOB VALUES (MB) 14.00 MAX STOR XML VALUES (MB) 0.00



Finding performance degradations using scatter plots

Scatter plot Displays values for two variables for a set of data usingcartesian coordinates



DB2 Accounting scatter plot



Big performance degradation

Recommendation: REBIND SWITCH impacted packages as soon aspractical



Overall view of performance changes

Recommendation:• Correlate average changes with total CPU utilization• Investigate performance degradation

I REBIND SWITCH when neededCristian Molaro - [email protected] 2014 86 / 89

Analyzing performance using Statistics and Accounting reports | Summary

This was our Agenda


2 DB2 traces



5 Summary


Any questions?

analyzing performance using statistics and accounting reports · (class 1 se cpu - class2 se cpu)...

Documents