exadata performance

Download Exadata performance

Post on 13-Oct-2015




10 download

Embed Size (px)


To learn what and how to proceed when faced with performance problems on Exadata


  • 5/23/2018 Exadata performance


  • 5/23/2018 Exadata performance


    Exadata Performance Debugging

    Biswaroop Biswal / Ranga Sarvabhouman

  • 5/23/2018 Exadata performance



    What basic information one must know about Exadata I/O related performance:

    - Check if cells are IO bound

    - Flash Cache

    - Smart Scan

    Check if cells are IO bound

    Check if db nodes are CPU or memory bound

    Check if smart scan works as expected

    Check if HCC/Partitioning/DBFS are used

    If none is true, go back to database performance tuning and planning

  • 5/23/2018 Exadata performance


    Check if cells are IO bound?

    Differentiate between slowness vs. full utilization

    Use OSW iostat and/or CellDisk metrics to compute total HDD and FLASHthroughput (MBPS) and IOPS

    Refer to Exadata DBM data sheet for peak numbers:

    Watch out for high latency if IOs ever approach peak numbers

    High latency does NOTmean slow disks each IO takes long primarily due to time waiting in disk queue

    IO latency can be >100ms (note disks are not slow!)

    IO latency depends on disk queue length so can be varied based ondifferent workloads

    Be aware that max MBPS and max IOPS can not be reachedsimultaneously

    How to evaluate mixed workload?

    Examine disk utilization - is it close to 100%?

    Run calibrate if needed (requires cells being quiesced)

  • 5/23/2018 Exadata performance


    What happens when cells are IO bound

    Consider SAME (Stripe And Mirror Everywhere)

    when any disk or cell is maxed out, performance will be

    throttled by that disk/cell even with workload parallelization

    Be aware of potential performance disparity between system and

    data disks

    System disks not only have user data but also have cells own file


    System disks may run slower than data disks

    More pronounced on High Capacity 2TB drives due to lowerIOPS capacity when compared with High Performance

    600GB drives

  • 5/23/2018 Exadata performance


    Exadata Flash Cache - overview

    Know your Flash: Flash storage entities and relationships:

  • 5/23/2018 Exadata performance


    When/How and what to measure in Flash cache:

    When to measure:

    - Missing SLA- Poor performance across the environment.

    - AWR reports

    - Users screaming


    - Exadata Storage Servers using views

    - Exadata Storage Servers using metrics (cellcli and dcli commands)

    - Exadata Smart flash log with metrics

    What to measure:- Effective use of flash

    - Percentage of I/O requests satisfied by flash

    - Number of objects kept on flash

    - Size of objects kept on flash

  • 5/23/2018 Exadata performance


    Smart Flash logwhat to look for?

    Smart Flash Logging affects log file parallel write time, notlog file synctime.

    Check the AWR report for high log file parallel write times; there should bevery few log file parallel write waits > 32 ms, and no waits > 0.5 seconds.This can be verified in the following sections of an AWR report:

    Wait Event Histogram

    Wait Event Histogram Detail (64 msec to 2 sec)

    If a cell has a non-zero value for FL_IO_W_SKIP_BUSY, then this means that

    the hard disks which contain the database log files (or their mirroredcopies) are not performing well. This can due to throughput or heavy load onthe database. It can be resolved by expanding your system to more cells.

    If a cell has a non-zero value for FL_ACTUAL_OUTLIERS andFL_EFFICIENCY_PERCENTAGE is not 100%, then this means that flash disksare not performing well. This can be due to load or performance issue. If it isload issue then try to reduce the load or replace the flash disk.

    Besides hard disk and flash disk performance, there are other factors whichcan affect redo log write latencies:

    Examine database nodes to make sure that LGWR is not experiencingscheduling hiccups due to factors such as swapping.

    Check whether the network is impacting the performance.

  • 5/23/2018 Exadata performance


    Smart Scanworks as expected & how to debug

    Symptoms In AWR report you would see these 2 wait events

    Cell table smart scan

    Cell index smart scan

    V$viewsv$sysstat and v$cell_state, statistics that you need to look for (also found systemstatitiscs section of AWR report):

    Cell physical IO bytes eligible for predicate offload

    Cell physical IO bytes saved by storage index

    Cell physical IO bytes send directly to DB node to balance CPU usage

    Cell physical IO interconnect bytes returned by smart scan

    %cell num smart%

    PREDIO (v$cell_state)

    Oswatcher logs

    Reasons Smart scan has less filtering/no filtering

    Suboptimal storage index usage

    Less/No filtering due to quarantine/ time zone upgrade

    Less filtering due to CPU pass through

    Other reasons Cell is CPU bound or IO bound

    Suboptimal flash usage

  • 5/23/2018 Exadata performance


    How to identify long running transaction that

    causes sub optimal Storage Index usage?

    Purpose: Long running transaction prevents min active scn from progressing,

    there by causing scans to not use storage index.

    Steps Get global min active scn by setting system event to 55703, level 1; the

    min active scn is printed in alert log once every 3 minutes. Unset theevent after you get the min active scn. Convert the min scn from hexto decimals.

    Use scn_to_timestamp to compare scn from alert log and current_scn,if there is differ by good amount then continue.

    Query the X$KTUXE to get the slot number, undo segment number

    where the column KTUXESTA is not like INACTIVE. Query v$process to obtain the instance ID, process ID.

    Now either you can kill the process or use pstack to obtain moreinformation.

  • 5/23/2018 Exadata performance


    Less/No Filtering due to quarantines.

    Presence of SQL or DB quarantines result in smart scan not being used.

    Look at following v$cell_state statistics to see if filtering is not happening

    due to quarantines

    Smart IO not used to SQL Step or DB Quarantine.

    Smart IO not used to Disk Region Quarantine. Use dcli or cellcli on cells to check for quarantines, for eg: , list quarantine

    where QuarantineType=Database in cellcli prompt.

    Quarantines are removed automatically when cell software is upgraded or

    use cellcli drop quarantine to remove manually.

  • 5/23/2018 Exadata performance


    Less/No filtering due to timezone upgrade; CPU


    Look at v$cell_state statistics to see if smart scan is nothappening due to timezone upgrade:

    Smart IO uses passthru as timezone file is unavailable. OR

    Select value from v$sysstat where name = cell num smart

    IO sessions using passthru mode due to timezone;Smart scans will take place after timezone file is made available.

    Reasons for CPU passthrough (storage is CPU bound) More scan queries is running on the storage server resulting in high

    CPU usage.

    Suboptimal Storage Index usage results in more physical IO beingperformed. Which results in more filtering on storage server whichincreases CPU usage.

    Scans on encrypted tables increases storage CPU usage.

  • 5/23/2018 Exadata performance


    I/O Resource Management Plans

  • 5/23/2018 Exadata performance


    I/O Resource Management Plans : Example

  • 5/23/2018 Exadata performance


    I/O Resource Management Plans : Example

  • 5/23/2018 Exadata performance


    I/O Resource Management Plans : Example

  • 5/23/2018 Exadata performance


    Setting the IORM Objective

    Available IORM objective settings:

    basicIORM does not enforce user-defined plans.

    IORM protects against extreme latencies for small I/O requests.

    Maximum throughput is maintained.

    low_latencyMinimizes latency by limiting the number of concurrent I/O requests

    Useful for critical OLTP workloadsPerformance of high-throughput workloads may suffer

    high_throughputMaximizes throughput by not limiting concurrent I/O requests

    Useful for DSS and data warehouse workloads

    Performance of latency-critical workloads may suffer

    balancedBalances low disk latency and high throughput

    Useful for mixed workloads

    autoIORM decides the best objective setting based on active plans and workloads

  • 5/23/2018 Exadata performance


    Intradatabase Plan : Example

  • 5/23/2018 Exadata performance


    Interdatabase Plan : Example

  • 5/23/2018 Exadata performance


    Using Share-Based Allocation in the

    Interdatabase Plan

    Commencing with Exadata Storage Server software release, I/O allocations in the

    Interdatabase plan can be expressed as shares rather than using the level and allocation attributesshown on the previous page. Each share is a value between 1 and 32, with 1 being the lowest share, and

    32 being the highest share. The share value represents the relative importance of each database rather

    than specifying an IO allocation percentage.



View more >