boosting sql performance with indexing technology sql performance with indexing technology ... two...

© 2015 IBM Corporation© 2016 IBM Corporation

Boosting SQL Performance with Indexing Technology

Rob [email protected] - DB2 for i

IBM Training

© 2016 IBM CorporationIBM Power Systems2

in·dex

Something that serves to guide, point out, or otherwise facilitate reference.

NEUGC 2016

NEUGC 2016

1

IBM Training


Indexing

IBM Training


Index Types

Two types of indexing technologies are supported

– Radix Index. The default unless you explicitly say otherwise

– Encoded Vector Index

Each type of index has specific uses and advantages

– Radix is the best general purpose index

Respective indexing technologies complement each other

NEUGC 2016

NEUGC 2016

2

IBM Training


Index Essence

Indexes:

Are ‘helper’ objects

– In SQL they exist simply to improve performance (try referencing them in a FROM clause)

Are used for implementation and statistics

Are maintained automatically by the database as the table rows come and go

– Only contain active rows (no deleted rows!)

Contain data (the key) and a row reference (RRN)

Are most often used in conjunction with table access

– They assist in determining the desired rows of the table, then the rows are retrieved from the table

Can be used without having to access the table – index only access

IBM Training


Index Usage - Probe vs. Scan

Indexes can be probed or scanned– Probe – using the leading (contiguous) key columns for fast lookups

• If more than one key is used, equal selection is required for all but the last key

– Scan can occur on any key column

Probe and scan can be used together

– Probe to a subset of entries, then scan each of those

A Radix index can provide ordering of data (ORDER BY)

– Can leverage this ‘side effect’ with either probe or scan

Indexes are cooperative

– More than one index can be used for a table in a given query

• Requires use of RRN list or bitmap of RRNs

– Mixing and matching Radix and EVI supported

NEUGC 2016

NEUGC 2016

3

IBM Training


CUSTOMER ORDER ITEM

001 B507 AB-2700

001 B607 CD-2000

002 B100 XY-1005

002 B102 AZ-5000

003 B709 HH-6500

004 B043 HH-6500

Index Key Columns (CUSTOMER, ORDER, ITEM)

• Probe (key positioning)with leading, n contiguouskey columns11+21+2+3

• Scan (test)with any otherkey columns232+3

...WHERE ORDER = ‘B102‘ AND CUSTOMER = 002

...WHERE ITEM = ‘HH-6500’

…WHERE ORDER = ‘B102‘ OR CUSTOMER = 002

Using Indexes - Probe v Scan

Why is the last WHERE example a scan??

IBM Training


SELECT *FROM EMPLOYEEWHERE STATE = ‘MINNESOTA'AND WORKDEPT IN ( 'B01', C01, 'E01')

IntermediateRRN list


FinalRRN list

AND(Merge)

State Workdept

State Workdepts

351015

10001005100730013050

371027

10001010203530014100

310

10003001

Two Radix - ANDing Example

Represents all the local selection

RadixRadix

NEUGC 2016

NEUGC 2016

4

IBM Training


Encoded Vector Index (EVI)

Index for delivering fast data access in analytical and reportingenvironments– Used to produce dynamic bitmaps and RRN lists

– Fast access to statistics to improve query optimizer decision making

– Can also include non-key summary data (SUM, COUNT)

Not a “tree” structure. Contains two parts:1. Symbol table of unique key values. Each unique value is encoded

(assigned) to a number

2. Vector with one entry per row in the underlying table. Each row contains the associated key value’s assigned number

Can only be created through an SQL interface or System i Navigator GUI

CREATE ENCODED VECTOR INDEXSchemaName/IndexName ON SchemaName/TableName(ColumnName)INCLUDE ( SUM(ColumnName));

IBM Training


SELECT *FROM EMPLOYEEWHERE STATE = ‘MINNESOTA'AND WORKDEPT IN ( 'B01', C01, 'E01')



EVI

FinalRRN list

AND(Merge)

State Workdept

State Workdepts

EVI

351015

10001005100730013050

371027

10001010203530014100

310

10003001

Two EVIs - ANDing Example


NEUGC 2016

NEUGC 2016

5

IBM Training


001100001...1...

000100000...1...

0

0

1

0

0

0

0

0

1

.

.

.

1

.

.

.

SELECT *FROM EMPLOYEEWHERE STATE = ‘IOWA'OR WORKDEPT IN ( 'B01', C01, 'E01')

IntermediateBitmap

IntermediateBitmap

RadixEVI

FinalBitmap

OR(Merge)

State Workdept

State Workdepts


Radix + EVI Usage - ORing Example

IBM Training

© 2016 IBM CorporationIBM Power Systems

Search / Scansymbol table

for key(s) and counts

CREATE ENCODED VECTOR INDEX idx2 ON employee (state) INCLUDE (SUM(x), SUM(y), COUNT(*))

SELECT COUNT(*) FROM EMPLOYEE WHERE STATE = ‘Wisconsin’;SELECT COUNT(DISTINCT STATE) FROM EMPLOYEE;SELECT STATE, SUM(x), SUM(y) FROM EMPLOYEE GROUP BY STATE;

EVI Index for Aggregate Answers

Symbol Table

Key Value Code CountInclude

Sum(x)

Include

Sum(y)

Arizona 1 5000 1500 2005

Arkansas 2 7300 3200 450

…

Wisconsin 49 340 575 1200

Wyoming 50 2760 210 0 NEUGC 2016

NEUGC 2016

6

IBM Training


cardinality The number of elements in a set

•High cardinality = large number of distinct values

•Low cardinality = small number of distinct values

In general…

•A radix index is best when accessing a small set of rows and the key cardinality is high

•An encoded vector index is best when accessing a subset of rows and the key cardinality is low

•Understanding the data and query is essential

The right index for the job

IBM Training


Creating Indexes – How?

CREATE INDEX SQL statement

CREATE INDEX MY_IX on MY_TABLE (KEY1, KEY2)

CREATE ENCODED VECTOR INDEX SQL statement

CREATE ENCODED VECTOR INDEX MY_EVI on MY_TABLE (KEY1)

System i Navigator – Database graphical interface

CRTPF and CRTLF CL commands (under the covers)

– Keyed access path within the physical file or logical file

– Join logical file

– Caution: Do not use this (native) approach for SQL! Illustration purposes only

Primary Key, Foreign Key and Unique Key Constraints have indexes created automatically under the covers

NEUGC 2016

NEUGC 2016

7

IBM Training


Creating Indexes through iNav

IBM Training


Derived Keys

Creation of indexes with derived keys via SQL supported

CREATE INDEX ORDERPRIORITYUPPER ON T1(UPPER(ORDERPRIORITY) AS UORDERPRIORITY ASC);

CREATE ENCODED VECTOR INDEX YEARQTR ON T1(YEAR(ORDERDATE) AS ORDYEAR ASC, QUARTER(ORDERDATE)AS ORDQTR ASC);

CREATE INDEX TOTALEXTENDEDPRICE ON T1(QUANTITY * EXTENDEDPRICE AS TOTEXTPRICE ASC);

NEUGC 2016

NEUGC 2016

8

IBM Training


When to create a Derived Index?

Derived indexes can be useful for – Case insensitive searches

– Data extracted from a column e.g. SUBSTR, YEAR, MONTH…

– Derive Common Grouping columns e.g. YEAR(ORDERDATE)

– Results of operations e.g. COL1+COL2 , QTY * COST

– Might be useful to allow index only access in more cases

• Including the INCLUDE support for EVIs

Could replace some logical files with SQL indexes for use by RLA native, high level language programs

– Modernize those objects

– Big logical page size (8K v 64K)

Query Optimization(using indexes)

NEUGC 2016

NEUGC 2016

9

IBM Training


Cost based optimization dictates that the fastest access method for a given table will vary based upon selectivity of the query

Number of rows searched / accessed

Few Many

ResponseTime Table Scan

Low

High

Probe

Clustered,Skip Seq

Data Access Methods

IBM Training


Optimizing indexes will generally follow this (simplified) strategy:

Gather list of indexes for statistics and costing implementation methods

Consider the indexes based on how the index can be used

Local selectionJoiningGroupingOrderingIndex only access

One index may be useful for statistics, and another useful for implementation

Strategy for Query Optimization

If an index is missing that could be useful for the query, the optimizer will often

Advise it! NEUGC 2016

NEUGC 2016

10

IBM Training


Advised IndexCatalog

SYSIXADV

SQE Plan Cache

Query Optimization

SQL requestDetailed

DB Monitor Data

VisualExplain

Query Optimization Feedback

SQE Plan Cache

Snapshots

IBM Training


Index Advice from the Query Optimizer

Index advice collected into a system file (catalog)

– Catalog (file) QSYS2/SYSIXADV• Each iASP has its own SYSIXADV catalog

– View through System i Navigator

Index advice also stored for each query and viewable through Visual Explain

Both Radix and EVI indexes advised (Radix advised more frequently)

Advice based on all parts of the query (selection, join, grouping, ordering)

Multiple indexes can be advised for the same query

Advice is limited to columns (fields) only. No derivations/expressions

Temporary Index creation also provides some insight

Advise can be ‘Condensed’ into a smaller, more general purpose set of indexesNEUGC 2016

NEUGC 2016

11

IBM Training


System Wide Index Advisor

IBM Training


Index Advisor – System-wide

NEUGC 2016

NEUGC 2016

12

IBM Training


Too Much Advice?

Index Advisor - Condenser

Queries:

…WHERE YEAR = 2008 AND QUARTER = 1 AND COLOR = ‘BLUE;…WHERE YEAR = 2008 AND QUARTER = 1; …WHERE YEAR = 2008;

Index advice by query:

YEAR, QUARTER, COLORYEAR, QUARTERYEAR

Condensed advice:

YEAR, QUARTER, COLOR

Can be used for all three queries

IBM Training


Index Advisor - Condenser

NEUGC 2016

NEUGC 2016

13

IBM Training


MTIs

IBM Training


Autonomic (Temporary) Index Creation

Optimizer can have the DB Engine create a maintained temporary index (MTI)

As the name implies, MTIs are maintained as table rows change

Temporary indexes consume temporary storage

Temporary indexes are usually a good indicator that a permanent index should be created

NEUGC 2016

NEUGC 2016

14

IBM Training


Index Evaluator (Show Existing Indexes)

Meta data for Indexes…

IBM Training


Visual Explain

NEUGC 2016

NEUGC 2016

15

IBM Training


IBM Training


NEUGC 2016

NEUGC 2016

16

IBM Training


Run SQL ScriptsA JDBC client that supports dynamic SQL

IBM Training


NEUGC 2016

NEUGC 2016

17

IBM Training


IBM Training


NEUGC 2016

NEUGC 2016

18

IBM Training


IBM Training


Making Choices

NEUGC 2016

NEUGC 2016

19

IBM Training


Nothing, let the system handle it?

Create all advised indexes?

Monitor, analyze, and tune important tables and queries?

IBM Training


The goals of creating indexes are:

1. Provide the optimizer implementation choices, based on the selectivity of the query

• Probe

• Scan

• Index only access

2. Provide the optimizer the statistics needed to understand the data, based on the query

DB2 for i

In all cases, the point is to

Improve Performance!NEUGC 2016

NEUGC 2016

20

IBM Training


Proactive method• Analyze the data model, application and SQL requests

• Add Database Primary Key, Unique Key and Referential Integrity constraints are helpful

Reactive method• Rely on optimizer feedback and actual implementation methods

• Rely on SQE’s ability to auto tune using temporary indexes

Understand the data being queried• Column selectivity

• Column cardinality

The Process of Identifying Indexes

IBM Training


Radix Indexes

• Common local selection columns• Join columns• Local selection columns + join columns• Local selection columns + grouping columns• Local selection columns + ordering columns

Note: Columns used with equal conditions are first in key list

Indexing Strategy

Minimum

NEUGC 2016

NEUGC 2016

21

IBM Training


Other Radix Index usage considerations

• Index only access – index contains all columns needed by the query(s)

• Avoid deleted records

• Replace an MTI

Indexing Strategy

IBM Training


NEUGC 2016

NEUGC 2016

22

IBM Training


Indexing Strategy - Examples

-- Query 1SELECTA.CUSTOMER_NO, A.ORDER_DATE, A.QUANTITYFROM ORDERS AWHERE A.CUSTOMER_NO = 0112358;

CREATE INDEX ORDERS_IX1 ON ORDERS (CUSTOMER_NO);

-- Query 2SELECTA.CUSTOMER_NO, A.ORDER_DATE, A.QUANTITYFROM ORDERS AWHERE A.CUSTOMER_NO = 0112358AND A.ITEM_ID = ‘ABC123YXZ’;

CREATE INDEX ORDERS_IX2 ON ORDERS (CUSTOMER_NO, ITEM_ID);

IBM Training



-- Query 3SELECTA.CUSTOMER_NO, A.CUSTOMER, A.ORDER_DATEFROM ORDERS AWHERE A.CUSTOMER_NO IN (0112358, 1321345, 5891442)AND A.ORDER_DATE > ‘2005/06/30’ORDER BY A.ORDER_DATE;

CREATE INDEX ORDERS_IX3a ON ORDERS (CUSTOMER_NO, ORDER_DATE);

CREATE INDEX ORDERS_IX3b ON ORDERS (ORDER_DATE, CUSTOMER_NO);

Q: Which is better?

-- Query 4SELECTA.CUSTOMER_NO, A.CUSTOMER, A.ORDER_DATEFROM ORDERS AWHERE A.CUSTOMER_NO = 0112358OR A.ORDER_DATE = ‘2005/06/30’;

CREATE INDEX ORDERS_IX4 ON ORDERS (CUSTOMER_NO);

CREATE ENCODED VECTOR INDEX ORDERS_EVI4 ON ORDERS (ORDER_DATE);

Q: Why not just create one index? NEUGC 2016

NEUGC 2016

23

IBM Training



-- Query 5SELECTA.CUSTOMER_NO, B.CUSTOMER, A.ORDER_DATE, A.QUANTITYFROM ORDERS A,

CUSTOMERS B,ITEMS C

WHERE A.CUSTKEY = B.CUSTKEYAND A.ITEMKEY = C.ITEMKEYAND A.CUSTOMER_NO = 0112358;

CREATE INDEX ORDERS_IX5a ON ORDERS (CUSTOMER_NO, CUSTKEY);

CREATE INDEX ORDERS_IX5b ON ORDERS (CUSTOMER_NO, ITEMKEY);

CREATE INDEX CUSTOMERS_IX5 ON CUSTOMERS (CUSTKEY);

CREATE INDEX ITEMS_IX5 ON ITEMS (ITEMKEY);

Q: Are there other useful index combinations?

IBM Training



-- Query 6SELECTYEAR(A.ORDER_DATE),SUM(A.QUANTITY), COUNT(*)FROM ORDERS AGROUP BY YEAR(A.ORDER_DATE);

CREATE ENCODED VECTOR INDEX ORDERS_IX6A ON ORDERS (YEAR(ORDER_DATE))INCLUDE (SUM(QUANTITY), COUNT(*));

NEUGC 2016

NEUGC 2016

24

IBM Training



-- Query 7

SELECT YEAR(A.ORDER_DATE),QUARTER(A.ORDER_DATE),

MONTH(ORDER_DATE), SUM(A.QUANTITY), COUNT(*)

FROM ORDERS A

WHERE QUARTER(A.ORDER_DATE) = 4

GROUP BY YEAR(A.ORDER_DATE), QUARTER(A.ORDER_DATE),

MONTH(ORDER_DATE)

ORDER BY YEAR(A.ORDER_DATE),QUARTER(A.ORDER_DATE),

MONTH(ORDER_DATE),

CREATE ENCODED VECTOR INDEX ORDERS_IX6A

ON ORDERS (YEAR(ORDER_DATE), QUARTER(ORDER_DATE), MONTH(ORDER_DATE) )

INCLUDE (SUM(QUANTITY), COUNT(*));

IBM Training


Practice Examples

select *from orderswhere shipdate = ‘2014-10-01’;

select *from orderswhere orderdate = ‘2014-10-01’and quantity = 0;

select *from orders o,

customers cwhere o.custkey = c.custkey;

NEUGC 2016

NEUGC 2016

25

IBM Training


Practice Examples

select *from orders o,

customers cwhere o.custkey = c.custkeyand c.state = ‘MN’;

select *from orders oinner join customers c

on o.custkey = c.custkey;

select *from orders oinner join customers c

on o.custkey = c.custkeyand o.orderkey = 112358;

IBM Training


Practice Examples

select *from customerswhere country in ('CANADA', 'FRANCE', ‘GERMANY');

select *from customerswhere upper(customer) = ‘JOE COOL'and country in ('CANADA', 'FRANCE', ‘GERMANY');

select *from customerswhere country = 'FRANCE'or continent = 'EUROPE‘and rating = ‘BEST’;

NEUGC 2016

NEUGC 2016

26

IBM Training


Practice Examples

select count(*)from customersgroup by country;

select *from orderswhere orderdate > '2014-09-01‘and shipdate < '2014-09-30';

select *from orderswhere orderpriority = '1-URGENT'and orderdate > '2014-09-01‘and shipdate < '2014-09-30';

select count(*)from customerswhere continent = ‘ASIA’group by country;

IBM Training


Practice Examples

update ordersset shipmode = ‘AIR’where orderkey = 112358;

update orders oset status = ‘BACK ORDERED’where o.partkey in

(select p.partkeyfrom parts pwhere inventory_amount = 0);

NEUGC 2016

NEUGC 2016

27

IBM Training


For best query performance, create the appropriate indexes

Eliminating table scans and temporary data structures will more than make up for index maintenance overhead

Consider the number of indexes when doing high volume changes

– Drop indexes when inserting into any empty table

– Consider dropping and recreating indexes when adding, changing, or deleting more than 40% of the rows

Consider parallel index maintenance for INSERTs and parallel index builds

– DB2 SMP feature installed and enabled

– Use SMP to create indexes in parallel

– Especially useful when (INSERT + INDEX CREATION) < (INSERT + INDEX MAINT)

To Index or Not to Index?

IBM Training


Summary

View creating indexes as an opportunity rather than a chore!

Take advantage of the integrated indexes provided by the query optimizer.

Manage your indexes, creating those that demonstrably improve performance and dropping those that are just taking up space.

Modernize your DDS logical file indexes to take advantage of larger maximum sizes and larger logical page sizes.

The right set of indexes will pay for themselves in terms of user satisfaction and system capacity!

NEUGC 2016

NEUGC 2016

28

IBM Training


Lab Services IBM DB2 for i Team

Database modernization

DB2 Web Query

Database architecture and design

DB2 SQL performance analysis and tuning

Data warehousing and Business Intelligence

DB2 for i education and training

Contact: Mike Cain [email protected]

IBM Systems and Technology Group

Rochester, MN USA

Are you experiencing performance problems?

Are you using SQL?

Are you getting the most out of DB2 for i?

Need help?

IBM Training


Thank you!

NEUGC 2016

NEUGC 2016

29

IBM Training


This document was developed for IBM offerings in the United States as of the date of publication. IBM may not make these offerings available in other countries, and the information is subject to change without notice. Consult your local IBM business contact for information on the IBM offerings available in your area.

Information in this document concerning non-IBM products was obtained from the suppliers of these products or other public sources. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give you any license to these patents. Send license inquires, in writing, to IBM Director of Licensing, IBM Corporation, New Castle Drive, Armonk, NY 10504-1785 USA.

All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

The information contained in this document has not been submitted to any formal IBM test and is provided "AS IS" with no warranties or guarantees either expressed or implied.

All examples cited or described in this document are presented as illustrations of the manner in which some IBM products can be used and the results that may be achieved. Actual environmental costs and performance characteristics will vary depending on individual client configurations and conditions.

IBM Global Financing offerings are provided through IBM Credit Corporation in the United States and other IBM subsidiaries and divisions worldwide to qualified commercial and government clients. Rates are based on a client's credit rating, financing terms, offering type, equipment type and options, and may vary by country. Other restrictions may apply. Rates and offerings are subject to change, extension or withdrawal without notice.

IBM is not responsible for printing errors in this document that result in pricing or information inaccuracies.

All prices shown are IBM's United States suggested list prices and are subject to change without notice; reseller prices may vary.

IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.

Any performance data contained in this document was determined in a controlled environment. Actual results may vary significantly and are dependent on many factors including system hardware configuration and software design and configuration. Some measurements quoted in this document may have been made on development-level systems. There is no guarantee these measurements will be the same on generally-available systems. Some measurements quoted in this document may have been estimated through extrapolation. Users of this document should verify the applicable data for their specific environment.

Special notices

IBM Training


IBM, the IBM logo, ibm.com AIX, AIX (logo), AIX 6 (logo), AS/400, BladeCenter, Blue Gene, ClusterProven, DB2, ESCON, i5/OS, i5/OS (logo), IBM Business Partner (logo), IntelliStation, LoadLeveler, Lotus, Lotus Notes, Notes, Operating System/400, OS/400, PartnerLink, PartnerWorld, PowerPC, pSeries, Rational, RISC System/6000, RS/6000, THINK, Tivoli, Tivoli (logo), Tivoli Management Environment, WebSphere, xSeries, z/OS, zSeries, AIX 5L, Chiphopper, Chipkill, Cloudscape, DB2 Universal Database, DS4000, DS6000, DS8000, EnergyScale, Enterprise Workload Manager, General Purpose File System, , GPFS, HACMP, HACMP/6000, HASM, IBM Systems Director Active Energy Manager, iSeries, Micro-Partitioning, POWER, PowerExecutive, PowerVM, PowerVM (logo), PowerHA, Power Architecture, Power Everywhere, Power Family, POWER Hypervisor, Power Systems, Power Systems (logo), Power Systems Software, Power Systems Software (logo), POWER2, POWER3, POWER4, POWER4+, POWER5, POWER5+, POWER6, POWER6+, System i, System p, System p5, System Storage, System z, Tivoli Enterprise, TME 10, Workload Partitions Manager and X-Architecture are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml

The Power Architecture and Power.org wordmarks and the Power and Power.org logos and related marks are trademarks and service marks licensed by Power.org.UNIX is a registered trademark of The Open Group in the United States, other countries or both. Linux is a registered trademark of Linus Torvalds in the United States, other countries or both.Microsoft, Windows and the Windows logo are registered trademarks of Microsoft Corporation in the United States, other countries or both.Intel, Itanium, Pentium are registered trademarks and Xeon is a trademark of Intel Corporation or its subsidiaries in the United States, other countries or both.AMD Opteron is a trademark of Advanced Micro Devices, Inc.Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States, other countries or both. TPC-C and TPC-H are trademarks of the Transaction Performance Processing Council (TPPC).SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPEC OMP, SPECviewperf, SPECapc, SPEChpc, SPECjvm, SPECmail, SPECimap and SPECsfs are trademarks of the Standard Performance Evaluation Corp (SPEC).NetBench is a registered trademark of Ziff Davis Media in the United States, other countries or both.AltiVec is a trademark of Freescale Semiconductor, Inc.Cell Broadband Engine is a trademark of Sony Computer Entertainment Inc.InfiniBand, InfiniBand Trade Association and the InfiniBand design marks are trademarks and/or service marks of the InfiniBand Trade Association. Other company, product and service names may be trademarks or service marks of others.

Special notices (cont.)

NEUGC 2016

NEUGC 2016

30

boosting sql performance with indexing technology sql performance with indexing technology ... two...

Documents