everything a db2 for z/os application developer needs to ...• stored procedure –sqlpl, ... index...
TRANSCRIPT
© 2016 IBM Corporation
Everything a DB2 for z/OS Application Developer needs to
know about the DB2 engine in 3.6 million milliseconds…
Randy Ebersole
DB2 Technical Specialist
DB2 Lab Services, IBM Analytics
#ibmiod
Agenda
• The Basics
• The DB2 Engine
• DB2 SQL Processing
• Stored Procedures
• DB2 Logging
#ibmiod
The Basics
• We assume the application wants to access data in DB2 for z/OS
• Regardless of the “driver” of the request for data, an SQL statement will execute
o Drivers:
• CICS – Programs that contain SQL
• DDF – Dynamic SQL
• Batch – Wide range of SQL processing
• Stored Procedure – SQLPL, COBOL, JAVA, contains SQL for data access
• RESTful API
– New with DB2 12
#ibmiod
DB2 Objects
Nonpartitioned Tablespace
Table T1 Table T2
Index Space
Index X1
Index Space
Index X2
Partitioned
Tablespace
Table
Part 1
Index Space
Part 2
Part 3
Partitioning
Index Part 1
Part 2
Part 3
Volume
4 Volume
25
Volume
6
Volume 3
Volume
2Volume
1
Storage
Group
G2
Storage
Group G1
Database A
Database B
#ibmiod
DB2 Engine
DBM1 - Data
MSTR - Recovery
IRLM – Locking
CICS
Application
DDF
SELECT COL1, COL2, COL3, COL4
FROM ACCT_TABLE
WHERE ACCT_NBR = ?
SQL
Data Returned
#ibmiod
DB2 Engine Components
• Application
o SQL processing, Synch I/O, Buffer updates, Lock requests, Logical logging, Global lock requests, GBP reads
• DDF - DIST
o Distributed Transaction Manager, Connection and DBAT management
• SYSTEM SERVICES - MSTR
o Archiving, BSDS processing, Physical log write/log prefetch, Update commit, Checkpoints, Backouts, Thread deallocation, incl. page P-lock unlock
• DATABASE SERVICES – DBM1
o Dataset Open/Close, Preformat Extend, Deferred write, Prefetch read, Parallel child tasks, DBM1 Full/System Contraction, Castout, P-lock negotiation, AsyncGBP write, GBP checkpoints, IRLM and XES global contention, Notify Exit, Delete Name*
• IRLM
o Error checking Management, Deadlock detection, Local IRLM lock/latch contention, P-lock negotiation, Async XES request*
6
#ibmiod
Program Execution
• Program enablement
o Getting the DB2 “pieces” in place for execution
o Some differences depending on access method
• Dynamic SQL, Stored Procedures, package use
• Thread Processing
o Allocation, Deallocation, REUSE
• SQL Processing
o Access or “UPDATE” of the data
#ibmiod
DB2 Response Time Flow
CICS/IMSTSODDF
Elapsed Time
1st SQL call
End of Call
2nd SQL call
End of Call
Class 2 Time
Class 2Time
Class 1Time
ProgramStart
ProgramElapsedTime
ProgramEnd
End UserResponseTime
Enter
Receive
“Network
“network”
#ibmiod
Performance Evaluation Basics
• Elapsed Time
o Waiting for what ?
• Locks
• I/O
• Other
• CPU
o SQL Workload driven
• Access path related
#ibmiod
SQL Access Path Impact
• A chosen access path drives the cost of execution
o Pattern of access drives Getpage activity
• Index pages
• Data pages
o Prefetch usage/volume
o Understanding predicate filtering
• Knowledge of application data
#ibmiod
DB2 Catalog Tables
DB2 Optimizer
CPU Cost Traversing Rows and Pages
Applying Predicates(RDS vs. DM)
Cost of sorting
"Least Cost Access Path"
I/O Cost TS Scan vs. Index Scan
Join Sequence Test
Index Scan (MATCHCOLS,
CLUSTERRATIOF)
Work File Cost
=
Access Path Selection
Valid SQL DB2 Internal Cost
Formulas
#ibmiod
Page Access Review
Leaf Pages
Data Pages
Index
. . .rows
Root Page
Non-leaf Pages
#ibmiod
SQL Considerations
SELECT COL1, COL2, COL3, COL4
FROM ACCT_TABLE
WHERE ACCT_NBR = ?
SELECT COL1, COL2, COL3, COL4, COL5, COL6, COL7
FROM ACCT_TABLE
WHERE ACCT_PAY_DATE BETWEEN ? AND ?
AND STATUS IN (?,?,?)
ACCT_TABLE
IX1 ACCT_NBR
IX2 ACCT_PAY_DATE
ACCT_INFO
#ibmiod
Page Access for SQL
SELECT COL1, COL2, COL3, COL4
FROM ACCT_TABLE
WHERE ACCT_NBR = ?
• Index access via a unique key
• Data page access required
• 3 Index getpages
• One data page access
#ibmiod
Page Access for SQL
SELECT COL1, COL2, COL3, COL4, COL5, COL6, COL7
FROM ACCT_TABLE
WHERE ACCT_PAY_DATE BETWEEN ? AND ?
AND STATUS IN (?,?,?)
• Index access – scan of some number of non-leaf page and leaf pages
• Data page access required
o To qualify STATUS +
o To retrieve qualified rows
• Getpage Estimate ?
#ibmiod
Consideration for DB2 SORT
• RDS SORT invoked for SQL Requirements
o ORDER BY
o GROUP BY
o DISTINCT
o UNION (WITHOUT ALL)
o Some SUBQUERIES
SELECT COL1, COL2, COL3, COL4, COL5, COL6, COL7
FROM ACCT_TABLE
WHERE ACCT_PAY_DATE BETWEEN ? AND ?
AND STATUS IN (?,?,?)
ORDER BY STATUS
• Special Sort Object
o Like a Tablespace
• GETPAGE Requests
• I/O
o Bufferpool Use
#ibmiod
Common Issues related to predicates
• SQL Predicate Related
• ....AND ACCT_TYPE = ?
• ACCT_TYPE is not in index
• Data page/ row filtering required to process SQL
• Stage 2 Predicate Usage
• COL BETWEEN COL1 AND COL2
• T1.COL1 = T1.COL2
• Stage 1 not indexable predicates(evaluated at stage 1 processing, during index screening, or after data page access)
• COL <> value
• COL NOT BETWEEN valA AND valB
• COL NOT IN (list)
#ibmiod
Common Issues related to data
• Date range check
• WHERE DATECOL BETWEEN ? AND ?
• Search
• WHERE LASTNAME LIKE ?
• And there is no management of how the host variable is populated
• Example - 'S%'
#ibmiod
Row Processing
Row 1 Row 2 Row 3
Row 6 Row 9 Row 12
Row 26 Row 44 Row 60
Row 2
Row 26
Row 60
Page 1
Row 13 Row 27 Row 32
Row 65 Row 91 Row 112
Row 226 Row 474 Row 690SQL
Row 27
Row 32
Row 690
Page 2
Row 19 Row 29 Row 39
Row 365 Row 591 Row 712
Row 726 Row 874 Row 990
Row 19
Row 29
Row 365
Page 3
SELECT COL1, COL2, COL3, COL4
FROM ACCT_TABLE
WHERE ACCT_NBR = ?Examined Processed
#ibmiod
SQL Call
Buffer Pool Processing
BP1
• GETPAGE FOR
o DATA PAGES
o INDEX PAGES
• NOT FOUND ?
o I/O WAIT
DASD
BP7 DASD
DB2 Sort work
#ibmiod
TABLE T1 TABLE T2
Composite
Composite
TABLE T3
OUTER TABLESor
COMPOSITE TABLES
INNER TABLESor
NEW TABLES
Join Processing
#ibmiod
Index Structure
Root
Page
Non-Leaf
PageNon-Leaf
Page
Non-Leaf
PageNon-Leaf
Page
Non-Leaf
Page
Leaf
Page
Leaf
PageLeaf
PageLeaf
Page
Leaf
PageLeaf
Page
Leaf
PageLeaf
Page
Offset Key1
Key3...rid
Key1...rid1
Key2...rid
Key Value RID
RID = RID Prefix (1 byte) + RID (4 bytes)Offset Key2Offset Key3
#ibmiod
Indexes
• Indexes are not always the answer
o Take additional space
o All indexes changed on INSERT and DELETE
• 3 - 5 + % DB2 CPU overhead
o UPDATE may change index
• Indexed columns should not be in SET statement
o MERGE can experience the additional cost
#ibmiod
Workload Considerations
• Complexities of transactions are a major consideration
o SQL processing on common objects
o SQL volume
• Impact to I/O patterns
• Locking
• other
• Outside influences
o CPU
o I/O workload
o Other data store access
#ibmiod
What do we know about the data ?
• About the Data
• Total Rows, Total Pages
• Data is for a year, a month, for last 20 years
• Reorg Frequency
• About the Access
• Common, very specific data retrieval
• Searches(wild card – LIKE )
• UPDATE/INSERT/DELETE
#ibmiod
SMF – Basics
• SMF (System management facilities) collects and records system and job-related information
• SMF collects data, writes to SMF datasets, with a periodic Dump to Dump datasets for ongoing use
• For DB2:
o Record Type 100 – Statistics - DB2 writes record type 100 to record data collected at event monitoring points
o Record Type 101 – Accounting - DB2 writes record type 101 to account for resources during a transaction
o Record Type 102 - DB2 writes record type 102 to record performance information (Audit also)
#ibmiod
• Accounting class detail has information that provides details related to application programs, usually counts and times:
o Start and stop times
o Commits and aborts
o SQL statements are issued
o Bufferpool requests
o Locking events
o Processor resources consumed
o Thread wait times for various events
o RID pool processing
o Distributed processing
o Resource limit facility statistics
Accounting Information
#ibmiod
DB2 Times – Page 1
AVERAGE APPL(CL.1) DB2 (CL.2)----------------------------------------------------------------ELAPSED TIME 5:22.25677 4:13.82966NONNESTED 5:22.25677 4:13.82966STORED PROC 0.000000 0.000000UDF 0.000000 0.000000TRIGGER 0.000000 0.000000
CP CPU TIME 1:19.53273 1:17.35850AGENT 1:16.91287 1:14.73884NONNESTED 1:16.91287 1:14.73884STORED PRC 0.000000 0.000000UDF 0.000000 0.000000TRIGGER 0.000000 0.000000PAR.TASKS 2.619862 2.619665
SECP CPU 0.000000 N/A
#ibmiod
DB2 Times – Page 2
AVERAGE APPL(CL.1) DB2 (CL.2)----------------------------------------------------------------SE CPU TIME 5.724770 5.724770NONNESTED 0.000000 0.000000STORED PROC 0.000000 0.000000UDF 0.000000 0.000000TRIGGER 0.000000 0.000000
PAR.TASKS 5.724770 5.724770
SUSPEND TIME 0.000000 2:45.71139AGENT N/A 2:12.25768PAR.TASKS N/A 33.453715STORED PROC 0.000000 N/AUDF 0.000000 N/A
NOT ACCOUNT. N/A 41.108370DB2 ENT/EXIT N/A 671802.22
#ibmiod
DB2 Stored Procedures
• Allows for defined access to DB2
• External Stored Procedures
o available languages on the z/OS server are: COBOL, PL/I, C, C++, Assembler, REXX, and JAVA
o Looks like other SQL applications. Including use of:
• static or dynamic SQL statements, IFI calls, and DB2 commands
o Except for REXX, pre-compile, compile, link, and bind the host source program to create the appropriate load modules and packages
o Definition and program logic are two separate components
o Runs through Workload Manager address spaces
• Native Stored Procedures
o Coded in SQL/PL
o Definition contains the SQL logic
o Execution path is more efficient
• NO WILM path, runs internal to the DB2 engine
#ibmiod
Locking
• A process is suspended when:
o A lock request is requested on an object that is already locked by another concurrent process and The two locks are not compatible
o The suspended process temporarily stops running. Incoming lock requests are queued in the order of arrival
• The suspended process resumes running when:
o All concurrent processes that hold incompatible locks release them
• The suspended time is the lock wait time
o The requesting process times out or deadlocks and the process resumes to deal with an error condition.
#ibmiod
• What’s in The DB2 Log
o Unit of recovery
o Checkpoint data
o Database page set control records
o Other miscellaneous stuff!
• Unit of Recovery Data
o Type of activity (Insert, Update, Delete)
o Before and after images of rows/columns
o Redo and undo records
• How DB2 handles rolling back a change
DB2 Logging
#ibmiod
• Change activity level and tracking
• Report activity affecting DB2 catalog objects
• Display INSERT, UPDATE, and DELETE activity
• Application RI reporting
• Auditing of DB2 update, insert, delete activity
• Recovery of DB2 data
• Replication of DB2 activity
• DDL tracking and reporting
o Auditing Catalog Changes
• Translate the activity to
o GRANT, REVOKE
o CREATE, ALTER, DROP
HOW TO LEVERAGE THE DB2 LOG
#ibmiod
Summary
• Think about the SQL and the processing and what it will need to do
o Consider the data characteristics
o It’s Not Magic
• Understand the performance possibilities
o Even at a high level
o What is Normal (or not !)
#ibmiod
RANDY EBERSOLE
Everything a DB2 for z/OS Application Developer needs to know about
the DB2 engine in 3.6 million milliseconds…