exadata performance
Post on 13-Oct-2015
133 views
Embed Size (px)
DESCRIPTION
To learn what and how to proceed when faced with performance problems on ExadataTRANSCRIPT
5/23/2018 Exadata performance
1/35
5/23/2018 Exadata performance
2/35
Exadata Performance Debugging
Biswaroop Biswal / Ranga Sarvabhouman
5/23/2018 Exadata performance
3/35
Agenda
What basic information one must know about Exadata I/O related performance:
- Check if cells are IO bound
- Flash Cache
- Smart Scan
Check if cells are IO bound
Check if db nodes are CPU or memory bound
Check if smart scan works as expected
Check if HCC/Partitioning/DBFS are used
If none is true, go back to database performance tuning and planning
5/23/2018 Exadata performance
4/35
Check if cells are IO bound?
Differentiate between slowness vs. full utilization
Use OSW iostat and/or CellDisk metrics to compute total HDD and FLASHthroughput (MBPS) and IOPS
Refer to Exadata DBM data sheet for peak numbers:
Watch out for high latency if IOs ever approach peak numbers
High latency does NOTmean slow disks each IO takes long primarily due to time waiting in disk queue
IO latency can be >100ms (note disks are not slow!)
IO latency depends on disk queue length so can be varied based ondifferent workloads
Be aware that max MBPS and max IOPS can not be reachedsimultaneously
How to evaluate mixed workload?
Examine disk utilization - is it close to 100%?
Run calibrate if needed (requires cells being quiesced)
http://www.oracle.com/ocom/groups/public/@otn/documents/webcontent/175280.pdfhttp://www.oracle.com/ocom/groups/public/@otn/documents/webcontent/175280.pdf5/23/2018 Exadata performance
5/35
What happens when cells are IO bound
Consider SAME (Stripe And Mirror Everywhere)
when any disk or cell is maxed out, performance will be
throttled by that disk/cell even with workload parallelization
Be aware of potential performance disparity between system and
data disks
System disks not only have user data but also have cells own file
systems
System disks may run slower than data disks
More pronounced on High Capacity 2TB drives due to lowerIOPS capacity when compared with High Performance
600GB drives
5/23/2018 Exadata performance
6/35
Exadata Flash Cache - overview
Know your Flash: Flash storage entities and relationships:
5/23/2018 Exadata performance
7/35
When/How and what to measure in Flash cache:
When to measure:
- Missing SLA- Poor performance across the environment.
- AWR reports
- Users screaming
How to measure:- V$SYSSTAT/V$SQL/V$SESSION_WAIT/V$SESSION_EVENT and V$SySTEM_EVENT
- Exadata Storage Servers using views
- Exadata Storage Servers using metrics (cellcli and dcli commands)
- Exadata Smart flash log with metrics
What to measure:- Effective use of flash
- Percentage of I/O requests satisfied by flash
- Number of objects kept on flash
- Size of objects kept on flash
5/23/2018 Exadata performance
8/35
Smart Flash logwhat to look for?
Smart Flash Logging affects log file parallel write time, notlog file synctime.
Check the AWR report for high log file parallel write times; there should bevery few log file parallel write waits > 32 ms, and no waits > 0.5 seconds.This can be verified in the following sections of an AWR report:
Wait Event Histogram
Wait Event Histogram Detail (64 msec to 2 sec)
If a cell has a non-zero value for FL_IO_W_SKIP_BUSY, then this means that
the hard disks which contain the database log files (or their mirroredcopies) are not performing well. This can due to throughput or heavy load onthe database. It can be resolved by expanding your system to more cells.
If a cell has a non-zero value for FL_ACTUAL_OUTLIERS andFL_EFFICIENCY_PERCENTAGE is not 100%, then this means that flash disksare not performing well. This can be due to load or performance issue. If it isload issue then try to reduce the load or replace the flash disk.
Besides hard disk and flash disk performance, there are other factors whichcan affect redo log write latencies:
Examine database nodes to make sure that LGWR is not experiencingscheduling hiccups due to factors such as swapping.
Check whether the network is impacting the performance.
5/23/2018 Exadata performance
9/35
Smart Scanworks as expected & how to debug
Symptoms In AWR report you would see these 2 wait events
Cell table smart scan
Cell index smart scan
V$viewsv$sysstat and v$cell_state, statistics that you need to look for (also found systemstatitiscs section of AWR report):
Cell physical IO bytes eligible for predicate offload
Cell physical IO bytes saved by storage index
Cell physical IO bytes send directly to DB node to balance CPU usage
Cell physical IO interconnect bytes returned by smart scan
%cell num smart%
PREDIO (v$cell_state)
Oswatcher logs
Reasons Smart scan has less filtering/no filtering
Suboptimal storage index usage
Less/No filtering due to quarantine/ time zone upgrade
Less filtering due to CPU pass through
Other reasons Cell is CPU bound or IO bound
Suboptimal flash usage
5/23/2018 Exadata performance
10/35
How to identify long running transaction that
causes sub optimal Storage Index usage?
Purpose: Long running transaction prevents min active scn from progressing,
there by causing scans to not use storage index.
Steps Get global min active scn by setting system event to 55703, level 1; the
min active scn is printed in alert log once every 3 minutes. Unset theevent after you get the min active scn. Convert the min scn from hexto decimals.
Use scn_to_timestamp to compare scn from alert log and current_scn,if there is differ by good amount then continue.
Query the X$KTUXE to get the slot number, undo segment number
where the column KTUXESTA is not like INACTIVE. Query v$process to obtain the instance ID, process ID.
Now either you can kill the process or use pstack to obtain moreinformation.
5/23/2018 Exadata performance
11/35
Less/No Filtering due to quarantines.
Presence of SQL or DB quarantines result in smart scan not being used.
Look at following v$cell_state statistics to see if filtering is not happening
due to quarantines
Smart IO not used to SQL Step or DB Quarantine.
Smart IO not used to Disk Region Quarantine. Use dcli or cellcli on cells to check for quarantines, for eg: , list quarantine
where QuarantineType=Database in cellcli prompt.
Quarantines are removed automatically when cell software is upgraded or
use cellcli drop quarantine to remove manually.
5/23/2018 Exadata performance
12/35
Less/No filtering due to timezone upgrade; CPU
passthrough
Look at v$cell_state statistics to see if smart scan is nothappening due to timezone upgrade:
Smart IO uses passthru as timezone file is unavailable. OR
Select value from v$sysstat where name = cell num smart
IO sessions using passthru mode due to timezone;Smart scans will take place after timezone file is made available.
Reasons for CPU passthrough (storage is CPU bound) More scan queries is running on the storage server resulting in high
CPU usage.
Suboptimal Storage Index usage results in more physical IO beingperformed. Which results in more filtering on storage server whichincreases CPU usage.
Scans on encrypted tables increases storage CPU usage.
5/23/2018 Exadata performance
13/35
I/O Resource Management Plans
5/23/2018 Exadata performance
14/35
I/O Resource Management Plans : Example
5/23/2018 Exadata performance
15/35
I/O Resource Management Plans : Example
5/23/2018 Exadata performance
16/35
I/O Resource Management Plans : Example
5/23/2018 Exadata performance
17/35
Setting the IORM Objective
Available IORM objective settings:
basicIORM does not enforce user-defined plans.
IORM protects against extreme latencies for small I/O requests.
Maximum throughput is maintained.
low_latencyMinimizes latency by limiting the number of concurrent I/O requests
Useful for critical OLTP workloadsPerformance of high-throughput workloads may suffer
high_throughputMaximizes throughput by not limiting concurrent I/O requests
Useful for DSS and data warehouse workloads
Performance of latency-critical workloads may suffer
balancedBalances low disk latency and high throughput
Useful for mixed workloads
autoIORM decides the best objective setting based on active plans and workloads
5/23/2018 Exadata performance
18/35
Intradatabase Plan : Example
5/23/2018 Exadata performance
19/35
Interdatabase Plan : Example
5/23/2018 Exadata performance
20/35
Using Share-Based Allocation in the
Interdatabase Plan
Commencing with Exadata Storage Server software release 11.2.3.1.0, I/O allocations in the
Interdatabase plan can be expressed as shares rather than using the level and allocation attributesshown on the previous page. Each share is a value between 1 and 32, with 1 being the lowest share, and
32 being the highest share. The share value represents the relative importance of each database rather
than specifying an IO allocation percentage.
Share-
Recommended