oracle exadata performance: latest improvements and less known features

blog.tanelpoder.com 1 © 2015 Tanel Poder

Exadata Performance: Latest Improvements and Less Known Features

Tanel Poder


Intro: About me

•  Tanel Põder •  Consultant, Trainer, Speaker, Computer Performance Geek •  17+ years…

•  Oracle Database Performance •  Exadata Performance •  Hadoop & Linux Performance

•  Director of R&D at Accenture Enkitec Group •  We rock! J

Expert Oracle Exadata book

(with Kerry Osborne and Randy Johnson of Enkitec)


Prerequisites

•  This is a random list of recent Exadata improvements and less known features

•  You already should know Exadata fundamentals (what is a smart scan, flash cache, storage index etc)


Agenda

1.   Oracle 12c SQL Monitoring Improvements for Exadata 2.   Storage Index Improvements

•  IO pruning with Bloom filters •  Min/Max IO pruning

3.   Recent Flash Cache Improvements •  Columnar Flash Caching

4.   Metrics Improvements •  Storage Cell Metrics in AWR

5.   Q&A

6.   Download all my scripts, tools from: •  blog.tanelpoder.com/files/


Latest Cellsrv Documenta^on?

•  Stored in storage cell filesystem: •  /opt/oracle/cell/doc/doc/

•  Also available from MOS as a patch:


SQL Monitoring Improvements


Exadata Metrics in SQL Monitoring reports (11g)


SQL Monitoring Improvements in Oracle 12c

•  Execu^on plan line level addi^onal metrics in the Other column!!!



HASH JOIN row source

JOIN FILTER CREATE

row source

Only one row was used for building the filter, resul^ng

in one bit set



TABLE ACCESS FULL on orders

TABLE ACCESS FULL on customers Only 115kB was

returned out of 760MB of smart IO issued. All IO from

Flash Cache

Storage Indexes helped to skip 424 MB of IOs


v$sql_monitor_statname in Oracle 12c SQL> @sqlmon_sn

NAME DESCRIPTION---------------------------------------- -------------------------------------------------------------------------Build Size Size of the build input in bytes Build Row Count Number of rows for the buildFan-out Number of partitions used to split both inputs Slot Size Size of an in-memory hash-join slot Total Build Partitions Total number of build partitionsTotal Cached Partitions Total number of build partitions left in-memory before probing Multi-pass Partition Pairs Total number of partition pairs processed multi-pass Total Spilled Probe Rows Total number of rows from the probe spilled to disk (excluding buffering)Bitmap Size Size of the bitmap in bytes...Folded Bitmap Size Folded size of the bitmap in bytesBits Set Number of bits set in the bitmapTotal Creator Rows Total number of rows from the creator side Total user Rows Total number of rows from the user sideTotal Filtered Rows Total number of rows filtered by the bloom filterTimes hash func called Number of times hash func is calledSlow metadata bytes Size of slow metadata in bytesEligible bytes Total bytes eligible for offloadFiltered bytes Total bytes returned after offloadSI saved bytes Total bytes saved by storage indexFlash cache bytes Total bytes fetched from flash cachePartial flash cache and disk bytes Total bytes where IO was partially fetched from flash cache and diskCell passthru IO bytes Total cell passthru IO bytesBlock IO bytes Total block IO bytes Slow metadata bytes Size of slow metadata in bytes

This is just a glance, the script produces more output of available plan

line level metrics


Flash Cache Improvements


Oracle 11.2.0.4 Cell Flash Cache Write Hit Metrics

SQL> @ses2 "select sid from v$session where program like '%DBW0%'" flash

SID NAME VALUE---------- ---------------------------------------------------------------- ---------- 296 cell writes to flash cache 50522 296 cell overwrites in flash cache 43998 296 cell flash cache read hits 36

SQL> @ses2 "select sid from v$session where program like '%DBW0%'" optimized

SID NAME VALUE---------- ---------------------------------------------------------------- ---------- 296 physical read requests optimized 36 296 physical read total bytes optimized 491520 296 physical write requests optimized 25565 296 physical write total bytes optimized 279920640

Previously the flash write hits were also accumulated under the cell flash

read hits metric!

The cell write hits (of IO requests) metrics can be 2-‐3x higher than DB write requests due to ASM mirroring (each cell

reports its own hits back)

hop://blog.tanelpoder.com/2013/12/04/cell-‐flash-‐cache-‐read-‐hits-‐vs-‐cell-‐writes-‐to-‐flash-‐cache-‐sta^s^cs-‐on-‐exadata/


Comments on Flash Cache (pre-‐11.2.3.3.0 cellsrv)

•  ALTER TABLE t STORAGE (CELL_FLASH_CACHE KEEP); •  DEFAULT •  KEEP •  NONE

•  Smart Scans bypass the Flash Cache regardless of I/O size, unless CELL_FLASH_CACHE = KEEP •  (even if the table data is already cached there) •  The Flash Cache I/O path is a FIFO, no IORM for Flash Cache before 11.2.3.3.0

•  With DEFAULT, buffered I/O operaôns will cache data of small IOs •  Small I/O < 128kB •  This includes TEMP IO requests smaller than 128kB! •  Flash cache populaôn is done asynchronously by separate jobs in cellsrv

The KEEP setng works at the table level (not par^ôn level). Kept segments get

automa^cally unkept auer 24 hrs of no access: ( _cell_object_expira9on_hours = 24 )


Automa^c expiry of KEEP objects from Flash Cache

CellCLI> LIST FLASHCACHECONTENT WHERE dbUniqueName='DEMO' AND hoursToExpiration < 24 DETAIL; cachedKeepSize: 23345496064 cachedSize: 23364829184 cachedWriteSize: 327680 dbID: 1538629110 dbUniqueName: DEMO hitCount: 14116 hoursToExpiration: 14 missCount: 35955 objectNumber: 88330 tableSpaceNumber: 27

CellCLI> LIST FLASHCACHECONTENT WHERE dbUniqueName='DEMO' AND objectNumber = 88330 DETAIL; cachedKeepSize: 24099815424 cachedSize: 24119345152 cachedWriteSize: 327680 dbID: 1538629110 dbUniqueName: DEMO hitCount: 20797 hoursToExpiration: 24 missCount: 36700 objectNumber: 88330 tableSpaceNumber: 27

The KEEP object hasn't been accessed for a while (~10 hours

since last access)

The ^mer back to 24 hours auer a simple COUNT(*) query

on the table


Flash Cache on Cellsrv 11.2.3.3.0+

•  Automa^c – even the CELL_FLASH_CACHE DEFAULT objects can be Smart Scanned from Flash Cache

•  IORM now throoles Flash Cache IOs on the flash disks too •  Previously only IOs on hard disks and disks carved out of flash were IORM-‐ed •  Reduces commit latency as Smart Flash (Redo) Logs are on the same devices


Cellsrv 12.1.2.1.0 (January 2015)

•  Lots of improvements, but let's look into columnar flash cache •  Independent dual format caching in storage cell flash •  Smart Scans use Columnar Flash Cache (if all required columns are cached) •  Block IO uses tradi^onal Flash Cache •  Conceptual overview tradi^onal vs. columnar Flash Cache

CU 1 col 1

col 2

col 3

col 3

col 4

col 4

col 5

col 5

col 6

CU 2 col 1

col 2

col 3

col 3

col 4

col 4

col 5

col 5

col 6

CU x col 1

col 2

col 3

col 3

col 4

col 4

col 5

col 5

col 6

CU 1 .. x col 1 col 2

col 3

col 4

col 5

col 6

col 2 col 2

col 3 col 3

col 4 col 4

col 5 col 5

col 5 col 6

compact data – less flash IO for a column scan


Columnar Flash Cache metrics – 1

SQL> @snapper all 5 1 1234

-------------------------------------------------------------------------------------------- TYPE, STATISTIC , DELTA, HDELTA/SEC-------------------------------------------------------------------------------------------- STAT, physical read IO requests , 6416, 1.26k STAT, physical read bytes , 6717366272, 1.32G STAT, cell physical IO bytes eligible for predicate offload , 6717366272, 1.32G STAT, cell physical IO interconnect bytes returned by smart scan, 445177024, 87.68M STAT, cell physical IO bytes saved by columnar cache , 5128519680, 1.01G STAT, cell blocks processed by cache layer , 250366, 49.31k STAT, cell blocks processed by txn layer , 250366, 49.31k STAT, cell blocks processed by data layer , 50327, 9.91k STAT, cell flash cache read hits , 6416, 1.26k WAIT, enq: KO - fast object checkpoint , 747, 147.12us WAIT, cell smart table scan , 315077, 62.05ms WAIT, SQL*Net message to client , 8, 1.58us WAIT, SQL*Net message from client , 2668093, 525.48ms WAIT, events in waitclass Other , 1136, 223.73us

Stats snap 1, end=2015-02-11 10:57:34, seconds=5.1


Columnar Flash Cache metrics -‐ 2


Lis^ng Columnar Flash Cache usage in cell

CellCLI> LIST METRICCURRENT FC_COL_BY_USED DETAIL; name: FC_COL_BY_USED alertState: normal collectionTime: 2015-02-10T23:24:44-06:00 metricObjectName: FLASHCACHE metricType: Instantaneous metricValue: 2,059 MB objectType: FLASHCACHE

CellCLI> LIST FLASHCACHECONTENT WHERE dbUniqueName = 'DW' AND objectNumber = 101821 DETAIL; cachedKeepSize: 0 cachedSize: 4418445312 cachedWriteSize: 18243584 columnarCacheSize: 2159017984 columnarKeepSize: 0 dbID: 1996233620 dbUniqueName: DW hitCount: 48176 missCount: 5238 objectNumber: 101821 tableSpaceNumber: 8

"Number of megabytes used in Columnar FlashCache"

You can experiment with it using _enable_columnar_cache

session parameter


Storage Index Improvements


Skipping IOs via Bloom Filters + Storage Indexes

•  hops://blog.tanelpoder.com/2014/05/17/combining-‐bloom-‐filter-‐offloading-‐and-‐storage-‐indexes-‐on-‐exadata/

SQL> SELECT 2 * 3 FROM 4 soe.customers c 5 , soe.orders o 6 WHERE 7 o.customer_id = c.customer_id 8 AND c.cust_email = '[email protected]' 9 /

We have a direct filter only on the CUSTOMERS table, but a

Bloom filter constructed from it may allow to skip IOs on the

ORDERS table too!

1.  Remember the join column's MIN/MAX in the driving row-‐source 2.  Push the MIN/MAX of the join column to storage with Bloom filter 3.  Compare the MIN/MAX value to the storage index to skip IOs on the joined

table


Skipping IOs for MIN/MAX calcula^on

•  If both projected and filtering column are in storage indexes… •  …skip storage regions where it's known no bigger MAX (or min) values exist •  Works if you don't actually need to access the values (COUNT, SUM, AVG…) •  Controlled by _cell_storidx_minmax_enabled parameter

SELECT MAX(cust_postal_code) FROM customers_nocache WHERE cust_id > 1234;


What else?

•  MOS Note 1947476.1 •  Exadata 12.1.2.1.0 release and patch (17885582)

•  Sparse Grid Disks •  Snapshot Databases for Dev/Test (Thin Cloning in Exadata!) •  V$ASM_DISKGROUP_SPARSE, V$ASM_DISK_SPARSE, etc

•  Limi^ng Flash Cache space usage per DB •  IORM interdatabase resource plan

•  12c AWR reports contain storage cell performance data! •  V$CELL_DB, DBA_HIST_CELL_DB and many others


Further Reading – Smart Scans

•  My Exadata blog entries: •  hop://blog.tanelpoder.com/category/exadata/

•  Frits Hoogland's Exadata ar^cles: •  hop://fritshoogland.wordpress.com/category/oracle/exadata/

•  The Expert Oracle Exadata book •  We are working on a 2nd edi^on of this book (Andy,Karl,Mar^n,Frits)


Thank You!

Ques^ons?

[email protected] blog.tanelpoder.com

@tanelpoder

www.enkitec.com

oracle exadata performance: latest improvements and less known features

Technology

tanel poder exadata

tanel poder v

tanel poder intro

tanel poder prerequisites

tanel poder agenda

number of partitions

c sql monitoring improvements

known features tanel