exadata performance

5/23/2018 Exadata performance

1/35

5/23/2018 Exadata performance

2/35

Exadata Performance Debugging

Biswaroop Biswal / Ranga Sarvabhouman

5/23/2018 Exadata performance

3/35

Agenda

What basic information one must know about Exadata I/O related performance:

- Check if cells are IO bound

- Flash Cache

- Smart Scan

Check if cells are IO bound

Check if db nodes are CPU or memory bound

Check if smart scan works as expected

Check if HCC/Partitioning/DBFS are used

If none is true, go back to database performance tuning and planning

5/23/2018 Exadata performance

4/35

Check if cells are IO bound?

Differentiate between slowness vs. full utilization

Use OSW iostat and/or CellDisk metrics to compute total HDD and FLASHthroughput (MBPS) and IOPS

Refer to Exadata DBM data sheet for peak numbers:

Watch out for high latency if IOs ever approach peak numbers

High latency does NOTmean slow disks each IO takes long primarily due to time waiting in disk queue

IO latency can be >100ms (note disks are not slow!)

IO latency depends on disk queue length so can be varied based ondifferent workloads

Be aware that max MBPS and max IOPS can not be reachedsimultaneously

How to evaluate mixed workload?

Examine disk utilization - is it close to 100%?

Run calibrate if needed (requires cells being quiesced)

5/23/2018 Exadata performance

5/35

What happens when cells are IO bound

Consider SAME (Stripe And Mirror Everywhere)

when any disk or cell is maxed out, performance will be

throttled by that disk/cell even with workload parallelization

Be aware of potential performance disparity between system and

data disks

System disks not only have user data but also have cells own file

systems

System disks may run slower than data disks

More pronounced on High Capacity 2TB drives due to lowerIOPS capacity when compared with High Performance

600GB drives

5/23/2018 Exadata performance

6/35

Exadata Flash Cache - overview

Know your Flash: Flash storage entities and relationships:

5/23/2018 Exadata performance

7/35

When/How and what to measure in Flash cache:

When to measure:

- Missing SLA- Poor performance across the environment.

- AWR reports

- Users screaming

How to measure:- V$SYSSTAT/V$SQL/V$SESSION_WAIT/V$SESSION_EVENT and V$SySTEM_EVENT

- Exadata Storage Servers using views

- Exadata Storage Servers using metrics (cellcli and dcli commands)

- Exadata Smart flash log with metrics

What to measure:- Effective use of flash

- Percentage of I/O requests satisfied by flash

- Number of objects kept on flash

- Size of objects kept on flash

5/23/2018 Exadata performance

8/35

Smart Flash logwhat to look for?

Smart Flash Logging affects log file parallel write time, notlog file synctime.

Check the AWR report for high log file parallel write times; there should bevery few log file parallel write waits > 32 ms, and no waits > 0.5 seconds.This can be verified in the following sections of an AWR report:

Wait Event Histogram

Wait Event Histogram Detail (64 msec to 2 sec)

If a cell has a non-zero value for FL_IO_W_SKIP_BUSY, then this means that

the hard disks which contain the database log files (or their mirroredcopies) are not performing well. This can due to throughput or heavy load onthe database. It can be resolved by expanding your system to more cells.

If a cell has a non-zero value for FL_ACTUAL_OUTLIERS andFL_EFFICIENCY_PERCENTAGE is not 100%, then this means that flash disksare not performing well. This can be due to load or performance issue. If it isload issue then try to reduce the load or replace the flash disk.

Besides hard disk and flash disk performance, there are other factors whichcan affect redo log write latencies:

Examine database nodes to make sure that LGWR is not experiencingscheduling hiccups due to factors such as swapping.

Check whether the network is impacting the performance.

5/23/2018 Exadata performance

9/35

Smart Scanworks as expected & how to debug

Symptoms In AWR report you would see these 2 wait events

Cell table smart scan

Cell index smart scan

V$viewsv$sysstat and v$cell_state, statistics that you need to look for (also found systemstatitiscs section of AWR report):

Cell physical IO bytes eligible for predicate offload

Cell physical IO bytes saved by storage index

Cell physical IO bytes send directly to DB node to balance CPU usage

Cell physical IO interconnect bytes returned by smart scan

%cell num smart%

PREDIO (v$cell_state)

Oswatcher logs

Reasons Smart scan has less filtering/no filtering

Suboptimal storage index usage

Less/No filtering due to quarantine/ time zone upgrade

Less filtering due to CPU pass through

Other reasons Cell is CPU bound or IO bound

Suboptimal flash usage

5/23/2018 Exadata performance

10/35

How to identify long running transaction that

causes sub optimal Storage Index usage?

Purpose: Long running transaction prevents min active scn from progressing,

there by causing scans to not use storage index.

Steps Get global min active scn by setting system event to 55703, level 1; the

min active scn is printed in alert log once every 3 minutes. Unset theevent after you get the min active scn. Convert the min scn from hexto decimals.

Use scn_to_timestamp to compare scn from alert log and current_scn,if there is differ by good amount then continue.

Query the X$KTUXE to get the slot number, undo segment number

where the column KTUXESTA is not like INACTIVE. Query v$process to obtain the instance ID, process ID.

Now either you can kill the process or use pstack to obtain moreinformation.

5/23/2018 Exadata performance

11/35

Less/No Filtering due to quarantines.

Presence of SQL or DB quarantines result in smart scan not being used.

Look at following v$cell_state statistics to see if filtering is not happening

due to quarantines

Smart IO not used to SQL Step or DB Quarantine.

Smart IO not used to Disk Region Quarantine. Use dcli or cellcli on cells to check for quarantines, for eg: , list quarantine

where QuarantineType=Database in cellcli prompt.

Quarantines are removed automatically when cell software is upgraded or

use cellcli drop quarantine to remove manually.

5/23/2018 Exadata performance

12/35

Less/No filtering due to timezone upgrade; CPU

passthrough

Look at v$cell_state statistics to see if smart scan is nothappening due to timezone upgrade:

Smart IO uses passthru as timezone file is unavailable. OR

Select value from v$sysstat where name = cell num smart

IO sessions using passthru mode due to timezone;Smart scans will take place after timezone file is made available.

Reasons for CPU passthrough (storage is CPU bound) More scan queries is running on the storage server resulting in high

CPU usage.

Suboptimal Storage Index usage results in more physical IO beingperformed. Which results in more filtering on storage server whichincreases CPU usage.

Scans on encrypted tables increases storage CPU usage.

5/23/2018 Exadata performance

13/35

I/O Resource Management Plans

5/23/2018 Exadata performance

14/35

I/O Resource Management Plans : Example

5/23/2018 Exadata performance

15/35

I/O Resource Management Plans : Example

5/23/2018 Exadata performance

16/35

I/O Resource Management Plans : Example

5/23/2018 Exadata performance

17/35

Setting the IORM Objective

Available IORM objective settings:

basicIORM does not enforce user-defined plans.

IORM protects against extreme latencies for small I/O requests.

Maximum throughput is maintained.

low_latencyMinimizes latency by limiting the number of concurrent I/O requests

Useful for critical OLTP workloadsPerformance of high-throughput workloads may suffer

high_throughputMaximizes throughput by not limiting concurrent I/O requests

Useful for DSS and data warehouse workloads

Performance of latency-critical workloads may suffer

balancedBalances low disk latency and high throughput

Useful for mixed workloads

autoIORM decides the best objective setting based on active plans and workloads

5/23/2018 Exadata performance

18/35

Intradatabase Plan : Example

5/23/2018 Exadata performance

19/35

Interdatabase Plan : Example

5/23/2018 Exadata performance

20/35

Using Share-Based Allocation in the

Interdatabase Plan

Commencing with Exadata Storage Server software release 11.2.3.1.0, I/O allocations in the

Interdatabase plan can be expressed as shares rather than using the level and allocation attributesshown on the previous page. Each share is a value between 1 and 32, with 1 being the lowest share, and

32 being the highest share. The share value represents the relative importance of each database rather

than specifying an IO allocation percentage.

Share-