db2 v10 1. to include or not include? 2. where did all the...

Chad Reiber BMC Software Inc. BWDB2UG – September 10, 2014

DB2 V10 1. To Include or Not Include? 2. Where did all the LOBs go?

© Copyright 9/8/2014 BMC Software, Inc 2

BWDB2UG – DB2 V10 Questions

To Include or Not Include - Overview of Index Include

Columns - Storage - Benefit or Lack of Benefit - Getting to the question

Include or Not Include

Where did all the LOBs Go? - How to make LOBS - Inline - Best way to implement - How to find Candidates

Include or Not Include That is the Question ?


Reasons for defining an index

Enforce uniqueness

Ensure physical clustering of table data

Provide quicker access to the data - Matching column access - Index screening

Index access without matching columns - Index only access - Other index methods


Index only example – DB2 V9

SELECT A.EMPNO, A.PROJNO, B.FIRSTNME, B.MIDINIT, B.LASTNAME

FROM RDADMB.EMPPROJACT A, RDADMB.EMP B

WHERE A.EMPNO = B.EMPNO

AND B.EMPNO = ‘005025’

TABLE INDEX KEYS

EMP XEMP1 EMPNO

EMPPROJACT XEMPPROJACT1 PROJNO.ACTNO.

EMSTDATE.EMPNO

TABLE INDEX KEYS

EMP XEMP3 EMPNO.LASTNAME.

FIRSTNME.MIDINIT

UNIQUENESS

INDEX ONLY

TABLE INDEX KEYS

EMPPROJACT XEMPPROJACT2 EMPNO MATCHING


DB2 10 Index with INCLUDE columns

Allows you to add non-key columns to a unique index

Does not - Affect uniqueness - Enforce Referential Integrity constraint

So What is the Point? - Saves Resources by eliminating extraneous indexes - Reduce Index Overhead (INSERT, DELETE) - Increase INSERT performance - Those darn NPIs and DB2 Utilities - Potential provide Query Performance by expanding existing indexes

Downside – could degrade some current queries

CREATE UNIQUE INDEX RDADMB.XEMP1

ON RDADMB.EMP(EMPNO ASC)

INCLUDE(LASTNAME, FIRSTNME, MIDINIT)


Restrictions

Index must be UNIQUE

Index can’t be - System defined catalog index - Index controlled partitioning index

Partitioned index on table controlled partitioned space OK - Auxiliary index - Index on a foreign key - XML index - Index that has an expression - Hash overflow index

INCLUDE column can’t be LOB or DECFLOAT

Can’t ALTER ADD key if index has INCLUDE columns


Alter an existing index

Alter add each column - ALTER INDEX RDADMB.XEMP1 ADD INCLUDE COLUMN(LASTNAME) - ALTER INDEX RDADMB.XEMP1 ADD INCLUDE COLUMN(FIRSTNME) - ALTER INDEX RDADMB.XEMP1 ADD INCLUDE COLUMN(MIDINIT)

COMMIT - Index will be in rebuild pending

REORG or REBUILD

RUNSTATS

REBIND


Catalog changes

Catalog - SYSIBM.SYSKEYS.ORDERING

A – ascending D – descending R – random Blank – INCLUDE column

- SYSIBM.SYSINDEXES.UNIQUE_COUNT New column 0 – no INCLUDE columns > 0 – Number of columns used for uniqueness


Physical page changes

Index header page - HPGIFLGS2.HPGIINCLCOLIDX – yes if index has INCLUDE columns - HPGIOLTH – maximum ordered key length - HPGIORDCOLS – number of columns used for uniqueness

Leaf page

LASTNAME FIRSTNME MIDINIT

XEMP1

XEMP3

EMPNO RID

LASTNAME FIRSTNME MIDINIT EMPNO RID


Determine what indexes can be combined

SQL provided in DB2 10 for z/OS Technical Overview - Finds indexes that share the leading columns of a unique index - Returns a list of columns, but you still need to review the list for

restrictions Does not consider column ordering Does not consider UNIQUE WHERE NOT NULL Does not consider partitioning

There are a number of queries available - Added some to the notes section

End Result: • List of Candidate Indexes for INCLUDE

• Based on existing Indexes • Also want to view tables where NPIs were avoided


Positive Reason #1 - Better performing INSERT Processing - NPIs and Utility Processing - Overhead of maintaining indexes, INSERTS and DELETES

Positive Reason #2 - Got a UNIQUE Index only for uniqueness – No Brainer - (do we really know that?)

Negative Reason #1

Hurt existing Query Access - Larger Indexes

Get Pages Index Levels Organization More Sync I/Os Hybrid Joins

- Increased Elapsed and CPU Time

Then comes the balancing act …


Determine index usage

Want to look at the existing indexes - UNIQUE Index - Index to be “INCLUDED”

How often are these indexes used - SYSINDEXSPACESTATS.LASTUSED - Accounting statistics - Performance Database

What SQL statements are using these indexes? - SYSPACKDEP – static SQL - DSN_STATEMENT_CACHE_TABLE – dynamic SQL - Accounting statistics - Performance Database

What is the INSERT and DELETE activity? - SYSINDEXSPACESTATS.STATSINSERTS - SYSINDEXSPACESTATS.STATSDELETE - Accounting statistics


Questions this work will answer

Is this object heavily inserted / deleted - Where the “extra” index could be hurting performance

How exposed are we with existing queries using unique index - Not only how many queries - How often they run

What happens if you are limiting NPIs do we have an opportunity to include columns and improve some queries


Lets get to work …

What do we need? - SQL that runs against those tables / indexes - Baseline of Access Paths - What-if we have a new INCLUDE INDEX

COMPARE !!! - Start with the critical ones

How do we do that? … - Development

Build the new indexes in development Get Statistics Correct

- Clone the environment - Virtual Indexes

Available in V9 DSN_VIRTUAL_INDEX table


Example using Virtual Indexes

CR_ORDERS

CRORDDX1

CRORDDX2

UNIQUE INDEX Columns PRIMARY_KEY_A PRIMARY_KEY_B ORDER_NBR

INDEX added for query performance Columns ORDER_AMT

CRINCIDX

UNIQUE INDEX Columns PRIMARY_KEY_A PRIMARY_KEY_B ORDER_NBR INCLUDE (ORDER_AMT)


What I did …

Collected all the SQL Statements for CR_ORDER table

Built a Baseline of Access Paths

Updated the DSN_VIRTUAL_INDEXES table - DROP both existing indexes CRORDDX1 and CRORDDX2 - Added my new index CRINCIDX

Re-Explained the CR_ORDER table

Compare the Results


Two Quick Examples

SELECT A.PRIMARY_KEY_A, A.PRIMARY_KEY_B, B.ORDER_NBR, B.ORDER_AMT

FROM CR_INDIVIDUAL A, CR_ORDERS B

WHERE A.CUST_PHONE_NBR LIKE '201%'

AND A.PRIMARY_KEY_A = B.PRIMARY_KEY_A

AND A.PRIMARY_KEY_B = B.PRIMARY_KEY_B

UNION ALL

SELECT A.PRIMARY_KEY_A, A.PRIMARY_KEY_B, ' ', 0

FROM CR_INDIVIDUAL A

WHERE NOT EXISTS

(SELECT 1

FROM CR_ORDERS B

WHERE A.PRIMARY_KEY_A = B.PRIMARY_KEY_A

AND A.PRIMARY_KEY_B = B.PRIMARY_KEY_B

)

AND A.CUST_PHONE_NBR LIKE '201%'

UNION ALL

SELECT ' ', ' ', B.ORDER_NBR, B.ORDER_AMT

FROM CR_ORDERS B

WHERE NOT EXISTS

(SELECT 1

FROM CR_INDIVIDUAL A

WHERE B.PRIMARY_KEY_A = A.PRIMARY_KEY_A

AND B.PRIMARY_KEY_B = A.PRIMARY_KEY_B

AND A.CUST_PHONE_NBR LIKE '201%' )

ORDER BY 3


Good use of INCLUDE INDEX

STMT EXEC WEIGHTED WEIGHTED DIFF STMTNO COST COUNT COST DIFF IN

------ ------- ------- -------- -------- -------

461 849 1 849 -8127 PATH

461 8976 1 8976 INDEX

EXPL SRC LABL SSID SSID NAME

---- ---- ---- --------

XD01 DJJ1 DJJ1 CRINXPK

XS00 DJJ1 DJJ1 CRINXPK

QB PL M TB AC MT I NEW COMP P C MX LCK

LABL BL NO E NO TY CL TABLE INDEX O UJOG UJOG F F SQ MOD

---- -- -- - -- -- -- ------------ ------------ -- ---- ---- - - -- ---

XD01 1 1 3 0 0 N NNNN NNYN 0

XD01 2 1 0 1 R 0 CR_INDIVIDUA N NNNN NNNN S 0 IS

XD01 2 2 4 2 I 2 CR_ORDERS CRINCIDX N NNNN NNNN L 0 IS

XD01 3 1 0 3 R 0 CR_INDIVIDUA N NNNN NNNN S 0 IS

XD01 4 1 0 4 I 2 CR_ORDERS CRINCIDX Y NNNN NNNN 0 N

XD01 5 1 0 5 I 0 CR_ORDERS CRINCIDX N NNNN NNNN S 0 N

XD01 6 1 0 6 I 2 CR_INDIVIDUA CRINDDX1 N NNNN NNNN 0 IS

XS00 1 1 3 0 0 N NNNN NNYN 0

XS00 2 1 0 1 R 0 CR_INDIVIDUA N NNNN NNNN S 0 IS

XS00 2 2 4 2 I 2 CR_ORDERS CRORDDX1 N NNNN NNNN L 0 IS

XS00 3 1 0 3 R 0 CR_INDIVIDUA N NNNN NNNN S 0 IS

XS00 4 1 0 4 I 2 CR_ORDERS CRORDDX1 Y NNNN NNNN 0 IS

XS00 5 1 0 5 R 0 CR_ORDERS N NNNN NNNN S 0 IS

XS00 6 1 0 6 I 2 CR_INDIVIDUA CRINDDX1 N NNNN NNNN 0 IS


Good use of INCLUDE INDEX

EXPL SRC LABL SSID SSID NAME

---- ---- ---- -------

XD01 DJJ1 DJJ1 CRBMDPK

XS00 DJJ1 DJJ1 CRBMDPK

STMT EXEC WEIGHTED WEIGHTED DIFF STMTNO COST COUNT COST DIFF IN

------ ------- ------- -------- -------- ----------

504 849 1 849 11 PATH INDEX

504 838 1 838


Summary

Proceed cautiously

INCLUDE columns can save resources

Can possibly degrade performance

Be sure to measure all the affects - We looked at explain only in this presentation - The real determination is the actual performance of the queries

Where did all the LOBS Go?


LOBs have been available since DB2 Version ?

LOBs allowed storage of data in a separate space

The number one reason was VARCHAR and the limit of 32K

Version 10 gave us INLINE LOBs

Discuss how to determine the best INLINE LENGTH value.

Pinpoint the new V10 features that affect INLINE LOBs (ZPARMS,utilities)

Explore how INLINE LOBs effect processing regarding DASD savings and processing savings.


A short history of LOBs

V6 - EBCDIC & ASCII

V7 - UNICODE, ability to use LOAD & UNLOAD - LOBS materialized using data spaces - CHECK LOB – identify structural defects in LOB TS and invalid LOB

values - CHECK DATA – validate consistency between base and auxiliary

tables

V8 - LOBs materialized using DBM1 storage above 2GB - ZPARMS to limit storage allocation for LOB materialization

LOBVALA – size per user LOBVALS – size per system

- Auto generation of ROWID column and base table - XMB2CLOB function – convert XML value into a CLOB


LOBs in V9

- LOAD/UNLOAD - ability to use file reference - ZPARM MAXOFILR – control max number of file references

concurrently open - REORG removes embedded free space, attempts to make LOB

pages contiguous and uses shadow data sets - REORG SHRLEVEL REFERENCE – longer availability during reorg - Avoidance of LOB locking - LOB lock is no longer required to

serialize the consistency between the value of the LOB and the column of the base row for an uncommitted read, resulting in fewer locks and less locking overhead

- Automatic creation of objects – omit IN clause in CREATE TABLE - FETCH CONTINUE – way for applications to read LOB data without

having to materialize the entire LOB in memory or use LOB LOCATORs

- CHECK LOB – SHRLEVEL CHANGE - CHECK DATA – SHRLEVEL CHANGE


LOBs in V10

REORG enhancements - Allow SHRLEVEL CHANGE - Permit rows to move between partitions - Add AUX YES syntax

INLINE LOBs introduced

Eliminate LOB materialization within DB2

Alter AUX tablespace and index page size

DEFINE NO on AUX tablespace and index

Spanned records in LOAD and UNLOAD

Alter maximum length of LOB column


CREATE TABLE BMCTABLE (KEYCOL1 INTEGER ,CHARCOL1 CHAR(20) ,CHARLOB CLOB(1M))

OUTLINE LOB

INLINE LOB BASE TABLESPACE

KEYCOL1 CHARCOL1

AUX TABLESPACE CHARLOB

BASE TABLESPACE KEYCOL1

CHARCOL1 CHARLOB

CREATE TABLE BMCTABLE (KEYCOL1 INTEGER ,CHARCOL1 CHAR(20) ,CHARLOB CLOB(1M) INLINE LENGTH(2000))

CHARLOB contains 1024 bytes of data


CREATE TABLE BMCTABLE (KEYCOL1 INTEGER ,CHARCOL1 CHAR(20) ,CHARLOB CLOB(1M) INLINE LENGTH( 2000))

OUTLINE LOB INLINE LOB


CHARCOL1


(2200 bytes)


CHARCOL1 CHARLOB (2000 bytes)


(200 bytes)

CHARLOB contains 2200 bytes of data

OUT-OF-LINE

LOB


How do you create an INLINE LOB?

Universal Table Space only

Must be Reordered Row Format

Define No on AUX table space

INLINE LOB columns can be used in index on expression - SUBSTR only, INLINE portion only - Start and end values must be constants

If you use TYPE to define INLINE LOB, you cannot use LOB column in index on expression and you cannot specify a default value


How do you create an INLINE LOB?

CREATE TYPE BMC.BMC_CHARLOB AS CLOB(1M) CCSID ASCII INLINE LENGTH 32680;

CREATE TABLE BMC.TABLE1

(KEYCOL1 INTEGER

,CHARLOB BMC.BMC_CHARLOB


(KEYCOL1 INTEGER

,BINLOB BLOB(1M) INLINE LENGTH (32680)

WITH DEFAULT CAST(‘I AM TRYING TO STAY AWAKE’AS BLOB)


(KEYCOL1 INTEGER

,DBCHARLOB DBCLOB(1M) INLINE LENGTH (16340)


ZPARM that effects INLINE LOB

LOB_INLINE_LENGTH - Determines inline length value if none specified in DDL - Default value is 0 (outline LOB).

Good Default Value - Valid values 0-32680 (bytes)

If this value is greater than lob length in the DDL, DB2 will use the lob length as the inline length

If the total length of the table columns plus the inline length exceed a page size, DB2 will issue a -670 error

-670, ERROR: THE RECORD LENGTH OF

THE TABLE EXCEEDS THE PAGE SIZE

LIMIT


What are the advantages of using INLINE LOBS?

Performance

INLINE LOB CPU - Avoid accessing both AUX TS (and index) to get data - INDEX on Expression can be used to enable LOB data to provide

more direct access

INLINE LOB I/O - Synchronous I/Os to the AUX index and LOB table space are avoided

for fully INLINE LOB Outline LOBS – one row per page Inline LOBS – many rows per page (Less I/Os)

– Sequential and Dynamic Prefetch is available

DASD Savings - More than one LOB per page

Two LOBs cannot share a page on a LOB Tablespace - Inline LOBs can be compressed


What is a good value for INLINE LENGTH?

Factors that affect the decision - LOB column frequency of reference

Rarely Referenced ? Frequently Referenced, result in fewer I/O and possible DASD savings.

- Page size of base tablespace Determines number of rows that fit on a page Might need to change Page Size when going Inline LOB

- Compressibility of LOB data LOB Tablespaces can not be compressed Increase the number of rows on a page CLOBs are best compression candidates

- Search requirements on LOB data Ability to use Index on Expression

- VARCHAR?


Distribution of LOB column sizes

Need to consider size of the row and size of the page

When it make sense to convert a LOB to INLINE LOB? - How often is the LOB accessed?

Frequency? - What is the size of the LOBs stored in the table?

Can the Inline LOB be defined < 32,680 ? A high percentage of LOBs that can fit in the INLINE LOB the better If there are many OUT-OF-LINE LOBS then you are incurring the extra

I/O to get the rest of the LOB data - What is a good length to defined?


Frequency

How can we determine ? - SQL Statements

DB2 Catalog Dynamic Statement Cache Performance Database DB2 Tools

BMC Performance Advisor Database

STMT_

STATISTICS

BMC Performance Advisor Database

STMT_TEXT

DAILY_STMT_STATISTICS WEEKLY_STMT_STATISTICS MONTHLY_STMT_STATISTICS


Inline LOB – Distribution of Data / Actual LOB Sizes

Statistics on the distribution of LOB column lengths are not maintained in the DB2 catalog. - IBM RedBook DB2 V10 Performance Topics provides sample SQL

query Read every row of the LOB object

DB2 Tools - BMC DASD Manager

Collects LOB specific data with “BMCSTATS” BMC Replacement for RUNSTATS


SQL query to display LOB column sizes

Query uses LENGTH function to pull the actual length of the LOB column for each row

The end result is a report that shows how many LOB rows fit into each multiple of 4000 bytes in length.

WITH LOB_DIST_TABLE (LOB_LENGTH, LOB_COUNT) AS(SELECT

LOBCOL_LENGTH, COUNT(*)

FROM(SELECT ((LENGTH(lob column name) / 4000) + 1) * 4000

AS LOBCOL_LENGTH FROM table name) LOB_COL_LENGTH_TABLE GROUP

BY LOBCOL_LENGTH)

SELECT '04000', SUM(LOB_COUNT) FROM LOB_DIST_TABLE WHERE

LOB_LENGTH <= 4000 UNION

SELECT '08000', SUM(LOB_COUNT) FROM LOB_DIST_TABLE WHERE

LOB_LENGTH <= 8000 UNION …..


Information about current OUTLINE LOB

LOB Column Name

Avg LOB Size

From the zPARM Inline Lgth

Max LOB Length


Display number of INLINE LOB columns

If you want to find out how many INLINE LOBs you have in a specific database or schema, you can issue an SQL statement like this

SELECT SUBSTR(TBCREATOR,1,15) AS TBCREATOR,

SUBSTR(TBNAME,1,15) AS TBNAME,

SUBSTR(NAME,1,15) AS COLNAME,

COLTYPE, LENGTH

FROM SYSIBM.SYSCOLUMNS

WHERE TBCREATOR LIKE 'SYS%'

AND COLTYPE IN ('BLOB','CLOB','DBCLOB') ;

AND LENGTH > 4;


Alters available for INLINE LOBs

Alter Add INLINE LENGTH to existing LOB column - Space left in AREO status after alter - Cannot create index on expression until after a reorg

Alter increase existing INLINE LENGTH - Space left in AREO status after alter - Newly inserted rows will have new inline length - Cannot increment to length greater than LOB column length

Alter decrease existing INLINE LENGTH - Space left in REORP status after alter - Cannot decrement if space has index on expression - Cannot decrement if new inline length is less than default

Alter add LOB column with INLINE LENGTH to existing table


DB2 utility support for INLINE LOBs

Reorg Plus - If a LOB has OUT-OF-LINE LOBs, base and auxiliary objects should be

reorganized in the same step. Use AUX YES and SHRLEVEL REFERENCE.

- With AUX Yes you must use TEMPLATE for copy datasets - UNLOAD ONLY and UNLOAD EXTERNAL are not available for tables

with INLINE LOB

UNLOAD/LOAD with spanned records - Don’t need LOB File Reference Variables - UNLOAD EXTERNAL is invalid for INLINE LOBs


BMC Experiment – objects and data

Three object types

CLOB(2000) (outline lob)

CLOB(2000) INLINE LENGTH 1600 (no overflow, index

on expression)

CLOB(2000) INLINE LENGTH 1000 (overflow, index on

expression)

Used different page sizes -- 4K, 32K, 32K compressed

Columns in table: LOB, integer, timestamp

500,000 rows

LOB data is 1500 bytes

Measured EXCP in MSTR, DBM1, DIST, IRLM


Dasd (in tracks)

0

10000

20000

30000

40000

50000

60000

70000

Outline LOB Inline LOB Inline LOB w/OV

4K

32K

32C


LOAD EXCP

0

200000

400000

600000

800000

1000000

1200000

1400000


Load 4K

Load 32K

Load 32C


DELETE EXCP

0

2000

4000

6000

8000

10000

12000

14000

16000

18000


4K

32K

32C


INSERT EXCP

0

500000

1000000

1500000

2000000

2500000

3000000

3500000


4K

32K

32C


Summary – Inline LOBs

There is no definitive answer as to whether using INLINE LOBs will benefit your shop. It depends on your data and usage trends.

General INLINE Tuning Practices - Choose INLINE LENGTH according to LOB Distribution - Consider increasing the page size of base table (larger rows better

INLINE Percentage) - Compression on Base Table (need good compression) - Buffer Pool Tuning

db2 v10 1. to include or not include? 2. where did all the...

Documents