oracle database in-memory and the query optimizer

2014 © Trivadis

BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA

2014 © Trivadis

Oracle Database In-Memory and

the Query Optimizer UKOUG Tech14, Liverpool (GB)

Christian Antognini

11 December 2014 Oracle Database In-Memory and the Query Optimizer

1

2014 © Trivadis


2

@ChrisAntognini

■ Senior principal consultant, trainer and partner

at Trivadis in Zurich (CH)

■ [email protected]

■ http://antognini.ch

■ Focus: get the most out of Oracle Database

■ Logical and physical database design

■ Query optimizer

■ Application performance management

■ Author of Troubleshooting Oracle Performance (Apress, 2008/2014)

■ OakTable Network, Oracle ACE Director

2014 © Trivadis

1. In-Memory Column Store

2. In-Memory Aggregation


3

AGENDA

2014 © Trivadis


In-Memory Column Store

4

2014 © Trivadis


5

Architecture Overview

2014 © Trivadis

■ Data is populated into the IMCS in chunks called IM Column Unit (IMCU)


6

IM Column Unit

… C1 C2 C3 C1 C2 C3 C1 C2 C3

2014 © Trivadis

■ They are similar to Exadata storage indexes

■ Do not store information about NULL values

■ They are automatically created and maintained for each column inside the IMCS

■ They are only stored in memory

■ They keep track of the minimum and maximum values in each IMCU

■ Their aim is to avoid accessing IMCUs that do not contain relevant data


7

IM Storage Indexes

2014 © Trivadis

■ The cost model has been enhanced to be IM aware

■ It requires IM statistics

■ The total cost is a combination of both non-IM costs and IM costs

■ Enhancements are disabled if OPTIMIZER_INMEMORY_AWARE = FALSE or

OPTIMIZER_FEATURES_ENABLE < 12.1.0.2

■ IMCS is not disabled

■ Costs are based on the former cost model


8

IM-Aware Query Optimizer

2014 © Trivadis

■ The following statistics are used by the query optimizer:

■ Number of IMCUs

■ Number of blocks with data into the IMCUs

■ Number of rows stored into the IMCUs

■ Fraction of the rows stored into the IMCUs

- Value between 0 and 1

■ Number of rows stored into the transaction journal

■ IM statistics are computed on the fly during hard parses

■ A cursors is not invalidated when the IM statistics it is based on changes


9

IM Statistics

2014 © Trivadis

■ Two operations are related to IM scans:

■ TABLE ACCESS INMEMORY FULL

■ MAT_VIEW ACCESS INMEMORY FULL

■ Also used in case of Exadata storage

■ IM scans are selected by the query optimizer regardless of whether data is

actually stored into the IMCS

■ The current behavior seems buggy to me


10

IM Scans

2014 © Trivadis

■ Pruning based on storage indexes

■ Pruning based on compression metadata

■ Dictionary-based compression algorithms store metadata that contain a list of distinct

values for each column within IMCU

■ Evaluation against compressed data is possible

■ Except for CAPACITY LOW/HIGH

■ Evaluation takes advantage of SIMD

■ Execution plans provide a new type of predicate: INMEMORY

■ Also used in case of predicates offloaded to Exadata storage


11

Predicate Evaluation Taking Place During IM Scans

2014 © Trivadis

■ The query optimizer can generate predicates that are evaluated during IM scans

■ The generated predicates are used as explicit predicates


12

Generation of Implied Predicates

2014 © Trivadis


13

IM Scan Example

SELECT * FROM t WHERE id = 1 OR id > 1 AND n < 42

-------------------------------------------

| Id | Operation | Name |

-------------------------------------------

| 0 | SELECT STATEMENT | |

|* 1 | TABLE ACCESS INMEMORY FULL| T |

-------------------------------------------

1 - inmemory("ID">=1 AND ("N"<42 AND "ID">1 OR "ID"=1))

filter("N"<42 AND "ID">1 OR "ID"=1)

Implied predicate

2014 © Trivadis

■ The INMEMORY_QUERY initialization parameter enables or disables IM scans

at the session or system level

■ IM scans can also be controlled through the (NO_)INMEMORY hints

■ The hints override the INMEMORY_QUERY parameter


14

Controlling IM Scans

2014 © Trivadis

■ Based on Bloom filters

■ Similar concept than parallel joins in previous versions, but executed serially

■ Filter created by the left input

■ Filter used by the right input

■ Only available for hash joins

■ Only the probe input have to be “IM-enabled”

■ The join is used to filter the false positives

■ IM joins can be controlled through the (NO_)PX_JOIN_FILTER hints


15

IM Joins

2014 © Trivadis


16

IM Join Example

SELECT * FROM t1, t2 WHERE t1.id = t2.id AND t1.n < 1E3 ------------------------------------------------ | Id | Operation | Name | ------------------------------------------------ | 0 | SELECT STATEMENT | | |* 1 | HASH JOIN | | | 2 | JOIN FILTER CREATE | :BF0000 | |* 3 | TABLE ACCESS FULL | T1 | | 4 | JOIN FILTER USE | :BF0000 | |* 5 | TABLE ACCESS INMEMORY FULL| T2 | ------------------------------------------------ 1 - access("T1"."ID"="T2"."ID") 3 - filter("T1"."N"<1E3) 5 - inmemory(SYS_OP_BLOOM_FILTER(:BF0000,"T2"."ID")) filter(SYS_OP_BLOOM_FILTER(:BF0000,"T2"."ID"))

2014 © Trivadis

■ Table expansion is a cost-based transformation introduced in 11.2 to leverage

partially unusable indexes

■ It has been made IM aware

■ Useful in case not all partitions are “IM-enabled”

■ It can be controlled through the (NO_)EXPAND_TABLE hints


17

IM Table Expansion

2014 © Trivadis


18

IM Table Expansion Example

SELECT count(d) FROM t WHERE d BETWEEN :b1 AND :b2

---------------------------------------------------------------------------

| Operation | Name | Pstart| Pstop |

---------------------------------------------------------------------------

| SELECT STATEMENT | | | |

| SORT AGGREGATE | | | |

| VIEW | VW_TE_1 | | |

| UNION-ALL | | | |

| PARTITION RANGE SINGLE | | 11 | 11 |

| TABLE ACCESS INMEMORY FULL | T | 11 | 11 |

| PARTITION RANGE SINGLE | | 12 | 12 |

| TABLE ACCESS BY LOCAL INDEX ROWID BATCHED| T | 12 | 12 |

| INDEX RANGE SCAN | I | 12 | 12 |

---------------------------------------------------------------------------

The IM scan is only

enabled for the

partition 4

2014 © Trivadis

■ The plan hash value is independent from the kind of FTS

■ TABLE ACCESS FULL

■ TABLE ACCESS STORAGE FULL

■ TABLE ACCESS INMEMORY FULL

■ Features like SQL plan baselines do not break when IMCS is enabled

■ The “best” execution plan can change


19

Plan Hash Value

2014 © Trivadis

■ _INMEMORY_QUERY_SCAN controls IM scans (TRUE)

■ _OPTIMIZER_INMEMORY_ACCESS_PATH controls whether the query

optimizer costs for IM (TRUE)

■ _OPTIMIZER_INMEMORY_BLOOM_FILTER controls IM joins (TRUE)

■ _OPTIMIZER_INMEMORY_GEN_PUSHABLE_PREDS controls the generation

of implied predicates (TRUE)

■ _OPTIMIZER_INMEMORY_TABLE_EXPANSION controls IM table expansion

(TRUE)


20

Undocumented Parameters

2014 © Trivadis


In-Memory Aggregation

21

2014 © Trivadis

■ To optimize queries against a star schema, the query optimizer should do the

following:

■ Start evaluating each dimension table that has restrictions on it

■ Assemble a list with the resulting dimension keys

■ Use this list to extract the matching rows from the fact table

■ This approach cannot be implemented with regular joins

■ The query optimizer can join only two data sets at one time

■ Joining two dimension tables leads to a Cartesian product

■ To solve this problem, Oracle Database implements two query transformations:

■ Star transformation

■ Vector transformation (new in 12.1.0.2 – requires the In-Memory option)


22

The Star Schema Challenge

2014 © Trivadis

■ In the documentation Oracle refers to it as In-Memory Aggregation

■ It isn’t only about aggregation

■ It requires the In-Memory option because it takes advantage of some of its

features

■ INMEMORY_SIZE must be greater than 0

■ The query optimizer considers it when

■ equijoins are used

■ an aggregate function is applied to a column of the fact table

■ the query contains a GROUP BY clause

- ROLLUP, CUBE and GROUPING SETS are not yet supported


24

Vector Transformation (1)

2014 © Trivadis

■ It’s a cost-based query transformation

■ It can be controlled through the (NO_)VECTOR_TRANSFORM hints

■ As with the star transformation, sometimes is not possible to force it

■ It introduces new row source operations

■ KEY VECTOR CREATE

■ KEY VECTOR USE

■ VECTOR GROUP BY


25

Vector Transformation (2)

2014 © Trivadis

1. For every dimension table with a filter on it the following operations take place

■ Access table and filter data

■ Create key vector

■ Aggregate data

■ Create temporary table

2. Access the fact table through a FTS and filter data by applying the key vectors

■ On Exadata the filter is not yet offloaded

3. Aggregate data with either a vector or hash aggregation

4. Join data from the fact table to temporary tables

5. Join dimension tables without a filter on them


26

Vector Transformation – Processing Steps

2014 © Trivadis


27

Example

SELECT c.customer_class, sum(o.order_total)

FROM orders o, customers c, warehouses w

WHERE o.customer_id = c.customer_id

AND o.warehouse_id = w.warehouse_id

AND c.dob BETWEEN :b1 AND :b2

AND w.warehouse_name BETWEEN :b3 AND :b4

GROUP BY c.customer_class

customers orders warehouses

2014 © Trivadis


28

Execution Plan – Access Dimension Tables, Create Key Vectors,

and Create Temporary Tables

-------------------------------------------------------------------------

| Id | Operation | Name |

-------------------------------------------------------------------------

| 0 | SELECT STATEMENT | |

| 1 | TEMP TABLE TRANSFORMATION | |

| 2 | LOAD AS SELECT | SYS_TEMP_0FD9DE69C_28565764 |

| 3 | VECTOR GROUP BY | |

| 4 | KEY VECTOR CREATE BUFFERED | :KV0000 |

|* 5 | TABLE ACCESS STORAGE FULL | CUSTOMERS |

| 6 | LOAD AS SELECT | SYS_TEMP_0FD9DE69D_28565764 |

| 7 | VECTOR GROUP BY | |

| 8 | HASH GROUP BY | |

| 9 | KEY VECTOR CREATE BUFFERED | :KV0001 |

|* 10 | TABLE ACCESS STORAGE FULL | WAREHOUSES |

...

2014 © Trivadis


29

Execution Plan – Access Fact Table by Applying Key Vectors, and

Join Temporary Tables

... | 11 | HASH GROUP BY | | |* 12 | HASH JOIN | | | 13 | MERGE JOIN CARTESIAN | | | 14 | TABLE ACCESS STORAGE FULL | SYS_TEMP_0FD9DE69D_28565764 | | 15 | BUFFER SORT | | | 16 | TABLE ACCESS STORAGE FULL | SYS_TEMP_0FD9DE69C_28565764 | | 17 | VIEW | VW_VT_72AE2D8F | | 18 | VECTOR GROUP BY | | | 19 | HASH GROUP BY | | | 20 | KEY VECTOR USE | :KV0001 | | 21 | KEY VECTOR USE | :KV0000 | |* 22 | TABLE ACCESS STORAGE FULL| ORDERS | ------------------------------------------------------------------------- ... 22 - filter(SYS_OP_KEY_VECTOR_FILTER("O"."CUSTOMER_ID",:KV0000) AND SYS_OP_KEY_VECTOR_FILTER("O"."WAREHOUSE_ID",:KV0001))

2014 © Trivadis

Core Messages

■ In-Memory Column Store

■ The query optimizer is IM aware

■ In-Memory Aggregation

■ Very interesting concept, complements the

star transformation


30

2014 © Trivadis

Questions and answers ...

2014 © Trivadis

BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA

Christian Antognini

Principal Senior Consultant

[email protected]


31

oracle database in-memory and the query optimizer

Technology

trivadis oracle database

imaware query optimizer

oracle database logical

query optimizer ukoug

trivadis data

memory aggregation

memory column store

trivadis pruning