using awr for io subsystem analysis

Using AWR For IO Subsystem Analysis

Mike Ault, Oracle GuruApril, 2011

The World’s Fastest Storage® for over thirty years!

The World’s Fastest Storage®Texas Memory Systems, Inc.

Michael R. Ault, Oracle Guru- Nuclear Navy 6 years- Nuclear Chemist/Programmer 10 years - Kennedy Western University Graduate- Bachelors Degree Computer Science- Certified in all Oracle Versions Since 6- Oracle DBA, author, since 1990

A Texas Memory Systems Presentation


Books by Michael R. Ault


Free Statspack/AWR Analysis

Sponsored by Texas Memory Systems

-Looks for IO bottlenecks and other configuration issues.

-Straightforward tuning advice

StatspackAnalyzer.com


Know Your Load Type!• Online Transaction Processing

– Few reads– Many writes– Many small transactions– Look for redo/undo and sequential read issues

• Decision Support/Data Warehouse– Many reads– Few writes (other then possible temp)– Few transactions– Look for sort/workarea and scattered read issues

• Mixed or Hybrid


Load Profile 1Load Profile~~~~~~~~~~~~ Per Second Per Transaction --------------- --------------- Redo size: 32,222,638.27 107,697,649.70 Logical reads: 368,069.30 1,230,197.16 Block changes: 177,853.38 594,438.94 Physical reads: 398.81 1,332.94Physical writes: 1,404.91 4,695.64 User calls: 5.06 16.92 Parses: 28.29 94.55 Hard parses: 0.00 0.00 Sorts: 3.18 10.62 Logons: 0.24 0.80 Executes: 28.58 95.52 Transactions: 0.30


Load Profile 2

Load Profile Per Second Per Transaction Per Exec Per Call~~~~~~~~~~~~ --------------- --------------- ---------- ---------- DB Time(s): 2.3 7.1 0.63 1.05 DB CPU(s): 0.3 0.9 0.07 0.13 Redo size: 800.5 2,461.8 Logical reads: 6,307.6 19,396.7 Block changes: 3.6 10.9 Physical reads: 2,704.9 8,317.8 Physical writes: 86.9 267.3 User calls: 2.2 6.8 Parses: 2.0 6.1 Hard parses: 0.0 0.1W/A MB processed: 932,965.4 2,868,990.9 Logons: 0.1 0.2 Executes: 3.7 11.3 Rollbacks: 0.1 0.3 Transactions: 0.3


Top 5 Waits Section• Critical to look closely at this section• Use highest wait times to guide investigation

– DB FILE type waits – physical IO– BUFFER type waits – Logical IO– LOG type waits – Redo related– PX – Parallel Query– GC – Global Cache (RAC related)– Undo – Undo or rollback segment related


Top 5 Waits Section

Top 5 Timed Foreground Events~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Avg wait % DBEvent Waits Time(s) (ms) time Wait Class------------------------------ ------------ ----------- ------ ------ ----------db file sequential read 465,020 3,969 9 47.4 User I/ODB CPU 995 11.9db file parallel read 2,251 322 143 3.8 User I/Odb file scattered read 15,268 153 10 1.8 User I/Ogc current block 2-way 108,739 116 1 1.4 Cluster


DB File Type Waits• DB File Sequential Reads – memory starvation, non-selective indexes• DB File Scattered Reads – full table scans, insufficient indexing• Direct Path Writes – Appends, data loads• Direct Path Reads – Parallel slaves used to retrieve data• DB File Parallel Writes – Backup and partition use• DB File Parallel Reads – Partition use• DB File Single Write – File header writes, excessive data files• Direct path read temp – Temp file activity (sorts, hashes, temp tables,

bitmaps)• Direct path write temp – Temp file activity (sorts, hashes, temp tables,

bitmaps)


Log Type Waits• log file parallel write – Look for log file contention• log buffer space – Look at increasing log buffer size• log file switch (checkpoint incomplete) – May

indicate excessive db files or slow IO subsystem• log file switch (archiving needed) – Indicates archive

files are written too slowly• log file switch completion – May need more log files

per thread• log file sync – Could indicate excessive commits


Undo Events

• undo segment extension – If excessive, tune undo

• latch: In memory undo latch – If excessive could be bug, check for your version, may have to turn off in memory undo

• wait for a undo record – Usually only during recovery of large transactions, look at turning off parallel undo recovery.


What Next?

• Determine wait events of concern• Drill down to specific sections of report for

deeper analysis• Use custom scripts, ADDM and Ash to

investigate issues


Drilling Down to IO Issues

• Operating System Statistics – Look at comparison between Busy, Idle and IOWait

• Events – See what events other than top five are IO related

• SQL Area – Look at SQL statements with high physical IO

• Tablespace Area – Review IO characteristics of tablespaces

• Datafile Area – Review IO characteristics of datafiles


Operating System Statistics

• What you get depends on OS• Some may not give IO timings


Operating System StatisticsOperating System Statistics DB/Inst: AULTDB/aultdb1 Snaps: 91-92-> *TIME statistic values are diffed. All others display actual values. End Value is displayed if different-> ordered by statistic type (CPU Use, Virtual Memory, Hardware Config), NameStatistic Value End Value------------------------- ---------------------- ----------------BUSY_TIME 126,029IDLE_TIME 597,505IOWAIT_TIME 327,861NICE_TIME 766SYS_TIME 48,452USER_TIME 76,784LOAD 0 3PHYSICAL_MEMORY_BYTES 3,189,190,656NUM_CPUS 2NUM_CPU_CORES 1NUM_CPU_SOCKETS 1GLOBAL_RECEIVE_SIZE_MAX 4,194,304GLOBAL_SEND_SIZE_MAX 262,144TCP_RECEIVE_SIZE_DEFAULT 87,380TCP_RECEIVE_SIZE_MAX 1,048,576TCP_RECEIVE_SIZE_MIN 4,096TCP_SEND_SIZE_DEFAULT 65,536TCP_SEND_SIZE_MAX 1,048,576TCP_SEND_SIZE_MIN 4,096


Foreground Wait events

• Foreground=user processes• Usually most important• Usual source for top 5 wait events• 2 sections – classes and events• Classes are the rolled up sums for waits


Foreground Wait Classes

Foreground Wait Class DB/Inst: AULTDB/aultdb1 Snaps: 91-92-> s - second, ms - millisecond - 1000th of a second-> ordered by wait time desc, waits desc-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0-> Captured Time accounts for 68.9% of Total DB time 8,371.33 (s)-> Total FG Wait Time: 4,770.85 (s) DB CPU time: 995.13 (s) Avg %Time Total Wait waitWait Class Waits -outs Time (s) (ms) %DB time-------------------- ---------------- ----- ---------------- -------- ---------User I/O 518,267 0 4,449 9 53.1DB CPU 995 11.9Cluster 188,753 9 173 1 2.1Other 3,806,446 100 146 0 1.7Concurrency 1,854 2 2 1 0.0Commit 15 0 1 39 0.0Application 740 0 0 0 0.0System I/O 40 0 0 3 0.0Network 6,970 0 0 0 0.0Configuration 0 0 0.0 -------------------------------------------------------------


Foreground Wait Events

Foreground Wait Events DB/Inst: AULTDB/aultdb1 Snaps: 91-92-> s - second, ms - millisecond - 1000th of a second-> Only events with Total Wait Time (s) >= .001 are shown-> ordered by wait time desc, waits desc (idle events last)-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0 Avg %Time Total Wait wait Waits % DBEvent Waits -outs Time (s) (ms) /txn time-------------------------- ------------ ----- ---------- ------- -------- ------db file sequential read 465,020 0 3,969 9 395.8 47.4db file parallel read 2,251 0 322 143 1.9 3.8db file scattered read 15,268 0 153 10 13.0 1.8gc current block 2-way 108,739 11 116 1 92.5 1.4PX Deq: reap credit 3,247,703 100 107 0 2,764.0 1.3gc cr grant 2-way 57,265 7 28 0 48.7 .3gc cr multi block request 22,451 6 23 1 19.1 .3enq: BF - allocation conte 14 93 14 983 0.0 .2PX qref latch 555,843 100 9 0 473.1 .1IPC send completion sync 1,070 52 8 8 0.9 .1gc remaster 22 0 5 221 0.0 .1


Background Wait Events

• SMON, PMON, DBWR, LMON, LMS, etc process waits

• Usually not a big contributor• Types of waits should correspond to

foreground waits


Background Wait Events

Background Wait Events DB/Inst: AULTDB/aultdb1 Snaps: 91-92-> ordered by wait time desc, waits desc (idle events last)-> Only events with Total Wait Time (s) >= .001 are shown-> %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0 Avg %Time Total Wait wait Waits % bgEvent Waits -outs Time (s) (ms) /txn time-------------------------- ------------ ----- ---------- ------- -------- ------control file sequential re 8,336 0 72 9 7.1 33.5control file parallel writ 1,287 0 31 24 1.1 14.5db file parallel write 792 0 11 14 0.7 5.3log file parallel write 701 0 11 15 0.6 4.9events in waitclass Other 44,191 98 5 0 37.6 2.5library cache pin 449 0 2 4 0.4 .8db file sequential read 221 0 2 7 0.2 .8gc cr multi block request 1,915 0 2 1 1.6 .7os thread startup 19 0 1 56 0.0 .5gc cr block 2-way 246 0 0 1 0.2 .2db file scattered read 18 0 0 12 0.0 .1db file parallel read 3 0 0 59 0.0 .1gc current grant 2-way 98 0 0 1 0.1 .1


Wait event Histograms

• Allows you to see wait time distributions• Not available in earlier versions of Statspack• Shows distribution of waits by time buckets


Wait Event HistogramsWait Event Histogram DB/Inst: AULTDB/aultdb1 Snaps: 91-92-> Units for Total Waits column: K is 1000, M is 1000000, G is 1000000000-> % of Waits: value of .0 indicates value was <.05%. Value of null is truly 0-> % of Waits: column heading of <=1s is truly <1024ms, >1s is truly >=1024ms-> Ordered by Event (idle events last) % of Waits ----------------------------------------------- TotalEvent Waits <1ms <2ms <4ms <8ms <16ms <32ms <=1s >1s-------------------------- ----- ----- ----- ----- ----- ----- ----- ----- -----control file parallel writ 1287 59.0 24.1 16.9control file sequential re 9147 23.4 21.3 23.3 22.3 6.8 2.9 .0db file parallel read 2256 .3 1.0 7.4 32.6 56.8 1.9db file parallel write 792 .5 .8 4.2 28.7 50.0 8.8 7.1db file scattered read 15K .4 2.7 31.5 59.2 5.8 .5db file sequential read 465K .0 .6 2.2 49.5 45.0 2.3 .4gc cr grant 2-way 50K 87.2 11.1 1.3 .3 .2 .0gc cr multi block request 24K 59.0 36.8 3.0 .5 .6 .0gc current block 2-way 84K 6.5 87.7 5.2 .3 .2 .0library cache lock 488 82.8 10.9 4.9 1.0 .2 .2library cache pin 4371 77.6 11.1 7.4 3.1 .6 .0gcs remote message 274K 28.5 15.4 9.9 11.6 7.5 5.8 21.4ges remote message 53K 11.4 3.3 2.7 1.9 1.8 2.1 76.8 -------------------------------------------------------------


Service Related Statistics

• New in 10g and above• A service is a grouping of processes• Users may be grouped in SYS$USER• Application logins (single user) may be

grouped with that user name• Used to confirm IO issues, show what users

are dominating IO


Service Statistics

Service Statistics DB/Inst: AULTDB/aultdb1 Snaps: 91-92-> ordered by DB Time Physical LogicalService Name DB Time (s) DB CPU (s) Reads (K) Reads (K)---------------------------- ------------ ------------ ------------ ------------aultdb 8,344 981 9,769 22,715SYS$USERS 23 12 1 56SYS$BACKGROUND 1 0 1 17aultdbXDB 0 0 0 0 -------------------------------------------------------------


Service Wait Class Statistics

Service Wait Class Stats DB/Inst: AULTDB/aultdb1 Snaps: 91-92-> Wait Class info for services in the Service Statistics section.-> Total Waits and Time Waited displayed for the following wait classes: User I/O, Concurrency, Administrative, Network-> Time Waited (Wt Time) in secondsService Name---------------------------------------------------------------- User I/O User I/O Concurcy Concurcy Admin Admin Network NetworkTotal Wts Wt Time Total Wts Wt Time Total Wts Wt Time Total Wts Wt Time--------- --------- --------- --------- --------- --------- --------- ---------aultdb 517710 4446 234 1 0 0 5828 0SYS$USERS 555 3 1615 1 0 0 1140 0SYS$BACKGROUND 350 3 3486 4 0 0 0 0 -------------------------------------------------------------



• Shows breakdown by AWR period• Only shown if the span of the report brackets

other AWR collections• Can be used to pinpoint peak activity times for

further analysis



-------------------------------------------------------------Operating System Statistics - Detail DB/Inst: AULTDB/aultdb1 Snaps: 91-92Snap Time Load %busy %user %sys %idle %iowait--------------- -------- -------- -------- -------- -------- --------04-Aug 12:00:15 0.4 N/A N/A N/A N/A N/A04-Aug 13:00:28 3.0 17.4 10.6 6.7 45.3 82.6 -------------------------------------------------------------


The SQL Sections

• Total Elapsed Time – IO waits• Total CPU Time – Sorting, hashing• Total Buffer Gets – High logical IO• Total Disk Reads – High physical IO• Total Executions – May indicate loop issues• Total Parse Calls – Memory issues• Total Sharable Memory - Informational• Total Version Count – May indicate unsafe bind

variables• Total Cluster Wait Time – Indicates physical issues

(RPB, block size)


AWR SQL Disk Reads SectionSQL ordered by Reads DB/Inst: TEST/test Snaps: 29415-29416-> Total Disk Reads: 1,736,422-> Captured SQL account for 93.6% of Total

Reads CPU ElapsedPhysical Reads Executions per Exec %Total Time (s) Time (s) SQL Id-------------- ----------- ------------- ------ -------- --------- ------------- 268,145 5 53,629.0 15.4 16.42 1214.21 7k0dau5j6m6mwselect * from ( SELECT c.TRANSACTION_NUMBER trx , c.CARD_BRANCH_CODE CARD_BRANCH_CODE , c.TERMINAL_ID_ACTUAL TERMINAL_ID_ACTUAL , c.BANK_CODE BANK_CODE, c.MERCHANT_ID MERCHANT_ID , c.Product_Type Product_Type , c.AUX_SERIAL_NUMBER AUX_SERIAL_NUMBER , c.STATUS_CODE STATUS_CODE , c.TRANSACTION_TYPE TRANSACT

246,073 4 61,518.3 14.2 14.56 798.11 adf3kr30mg56wselect count(*) aa from ISWITCH_TRANSACTIONS c Where c.Added_Date between :1 and :2 and c.status_code ='1' AND c.TERMINAL_ID_ACTUAL IN (select INSTANCE_ID from ISWITCH_CHANNEL_INSTANCE i where i.BRANCH_CODE = '012' ) and c.TERMINAL_ID_ACTUAL ='A0000012' and c.Transaction_Type ='102' and c.Product_Type ='ATM'


Instance StatisticsInstance Activity Stats DB/Inst: ORCL/orcl Snaps: 29415-29416

Statistic Total per Second per Trans-------------------------------- -------- ----------- -------------DBWR checkpoint buffers written 2,764 0.8 0.7DBWR checkpoints 36 0.0 0.0DBWR object drop buffers written 0 0.0 0.0DBWR parallel query checkpoint b 118 0.0 0.0DBWR revisited being-written buf 0 0.0 0.0DBWR tablespace checkpoint buffe 0 0.0 0.0DBWR thread checkpoint buffers w 0 0.0 0.0DBWR transaction table writes 84 0.0 0.0DBWR undo block writes 612 0.2 0.2summed dirty queue length 1,747 0.5 0.4


Instance Statistics

Statistic Total per Second per Trans-------------------------------- -------- ----------- -------------change write time 68 0.0 0.0cluster key scan block gets 10,664 3.0 2.6cluster key scans 2,363 0.7 0.6index fast full scans (direct re 0 0.0 0.0index fast full scans (full) 787 0.2 0.2index fast full scans (rowid ran 0 0.0 0.0index fetch by key 93,700 26.0 22.4index scans kdiixs1 71,615 19.9 17.2leaf node 90-10 splits 20 0.0 0.0leaf node splits 67 0.0 0.0lob reads 49 0.0 0.0lob writes 157 0.0 0.0lob writes unaligned 157 0.0 0.0


Instance StatisticsStatistic Total per Second per Trans------------------------------- -------------- ----------- -----------physical read IO requests 1,306,076 362.7 312.7physical read bytes 14,224,769,024 3,950,124.3 3,405,498.9physical read total IO requests 1,318,016 366.0 315.5physical read total bytes 14,446,929,920 4,011,816.9 3,458,685.6physical read total multi block 66,857 18.6 16.0physical reads 1,736,422 482.2 415.7physical reads cache 1,634,630 453.9 391.3physical reads cache prefetch 344,579 95.7 82.5physical reads direct 101,792 28.3 24.4physical reads direct (lob) 0 0.0 0.0physical reads direct temporary 0 0.0 0.0physical reads prefetch warmup 0 0.0 0.0


Instance Statistics

Statistic Total per Second per Trans-------------------------------- ----------- ---------- -------------physical write IO requests 4,208 1.2 1.0physical write bytes 42,860,544 11,902.1 10,261.1physical write total IO requests 26,028 7.2 6.2physical write total bytes 238,408,192 66,204.4 57,076.4physical write total multi block 12,240 3.4 2.9physical writes 5,232 1.5 1.3physical writes direct 247 0.1 0.1physical writes direct (lob) 0 0.0 0.0physical writes direct temporary 0 0.0 0.0physical writes from cache 4,985 1.4 1.2physical writes non checkpoint 3,862 1.1 0.9


Instance statisticsStatistic Total per Second per Trans-------------------------------- -------- ----------- -------------redo synch time 739 0.2 0.2redo synch writes 9,341 2.6 2.2redo write time 803 0.2 0.2redo writes 6,159 1.7 1.5


Instance Statistics

Statistic Total per Second per Trans-------------------------------- ----------- ----------- -------------table fetch by rowid 4,701,301 1,305.5 1,125.5table fetch continued row 71 0.0 0.0table scan blocks gotten 3,411,330 947.3 816.7table scan rows gotten 108,225,337 30,053.5 25,909.8table scans (cache partitions) 0 0.0 0.0table scans (direct read) 3,055 0.9 0.7table scans (long tables) 3,087 0.9 0.7table scans (rowid ranges) 3,055 0.9 0.7table scans (short tables) 30,677 8.5 7.3user I/O wait time 585,938 162.7 140.3sorts (disk) 0 0.0 0.0workarea executions – multipass 0 0.0 0.0workarea executions - onepass 0 0.0 0.0 ------------------------------------------------------------


Instance Thread Activity

• Shows how often your logs are switching• Rule of thumb says 4 per hour. • Whatever works for your application and

doesn’t result in waits is good.


Instance Thread Activity

Instance Activity Stats - Thread ActivityDB/Inst: TEST/test Snaps: 29415-2941-> Statistics identified by '(derived)' come from sources other than SYSSTAT

Statistic Total per Hour-------------------------------- ------------------ ---------log switches (derived) 6 6.00-------------------------------------------------------------


Tablespace IO

• Shows reads, reads/sec, read latency, avg blocks read, writes, writes/sec, (no write latency), bufer busy waits, avg wait for BBW

• Sum of reads+writes divided by clock time for report should come close to total physical reads+total physical writes on a per second basis, however, I have seen it differ!

• Tells you what tablesapces are doing the most IO• Datafile IO (not shown) breaks it out by data file


Tablespace IO-> ordered by IOs (Reads + Writes) desc

Tablespace------------------------------ Av Av Av Av Buffer Av Buf Reads Reads/s Rd(ms) Blks/Rd Writes Writes/s Waits Wt(ms)-------------- ------- ------ ------- ------------ -------- ---------- ------DEVDATA 1,173,883 326 3.7 1.3 2,734 1 323,173 2.8IBKDATA 98,513 27 5.7 1.4 264 0 1 10.0DEVINDEX 15,225 4 0.5 1.0 125 0 0 0.0RETAILDATA 9,218 3 8.7 1.0 202 0 0 0.0SYSTEM 5,594 2 3.3 1.5 144 0 0 0.0SYSAUX 919 0 5.0 1.1 344 0 0 0.0UNDOTBS1 86 0 4.9 1.0 221 0 0 0.0TEMP_NEW 16 0 0.6 1.0 0 0 0 0.0


Buffer Space advisory

• Shows how much phsyical IO will be reduced by increasing )or decreasing) buffer size

• A majority of this IO will be single block reads• Use to determine if increasing cache will reduce

physical IO• However, if buffer busy waits are predominately disk

blocks and majority waits are sequential read, usually memory starvation is occurring regardless of what this says


Buffer Space Advisory Est Phys Size for Size Buffers for Read EstimatedP Est (M) Factor Estimate Factor Physical Reads--- -------- ------ ---------------- ------ ------------------D 192 .3 23,796 1.1 438,875,348D 256 .4 31,728 1.1 431,906,549D 320 .5 39,660 1.0 426,776,467D 384 .5 47,592 1.0 422,102,165D 448 .6 55,524 1.0 419,074,429D 512 .7 63,456 1.0 416,407,215D 576 .8 71,388 1.0 414,312,661D 640 .9 79,320 1.0 412,509,975D 704 1.0 87,252 1.0 410,854,050D 768 1.1 95,184 1.0 408,796,940D 832 1.2 103,116 1.0 407,142,199D 896 1.3 111,048 1.0 405,649,002D 960 1.4 118,980 1.0 404,148,251D 1,024 1.5 126,912 1.0 402,745,613D 1,088 1.5 134,844 1.0 401,506,706D 1,152 1.6 142,776 1.0 400,276,499D 1,216 1.7 150,708 1.0 399,110,953D 1,280 1.8 158,640 0.9 389,800,510


PGA Aggregate Target

• Controls temp usage• Sorts, hashes, GT tables, bitmap work• Excessive temp IO shows

PGA_AGGREGATE_TARGET issues• sorts(disk), single and multipass operations

show problems• Note: some sort operations (shared server

connections, backup up, etc) still use sort area size and hash_area_size


PGA_AGGREGATE_TARGET> PGA cache hit % - percentage of W/A (WorkArea) data processed only in-memory

PGA Cache Hit % W/A MB Processed Extra W/A MB Read/Written--------------- ------------------ -------------------------- 99.8 845,324 1,607 -------------------------------------------------------------


PGA_AGGREGATE_TARGETPGA Aggr Target Histogram DB/Inst: TEST/test Snaps: 24080-24107-> Optimal Executions are purely in-memory operations

Low HighOptimal Optimal Total Execs Optimal Execs 1-Pass Execs M-Pass Execs------- ------- -------------- -------------- ------------ ------------ 2K 4K 8,446,124 8,446,124 0 0 64K 128K 28,112 28,112 0 0 128K 256K 43,917 43,917 0 0 256K 512K 173,347 173,347 0 0 512K 1024K 643,107 643,107 0 0 1M 2M 24,595 24,595 0 0 2M 4M 15,934 15,912 22 0 4M 8M 20,446 20,436 10 0 8M 16M 3,848 3,846 2 0 16M 32M 1,671 1,668 3 0 32M 64M 1,173 1,172 1 0 64M 128M 76 74 2 0 128M 256M 378 378 0 0 512M 1024M 2 0 2 0 -------------------------------------------------------------


PGA_AGGREGATE_TARGETPGA Target Size W/A MB W/A MB Read/ Cache Overallo Estd Est (MB) Factr Processed Written to Disk Hit % Count Time---------- ------- ---------------- ---------------- ------ -------- ------- 7,872 0.1 35,936,407.5 32,945.6 100.0 0 5.2E+10 15,744 0.3 35,936,407.5 31,937.3 100.0 0 5.2E+10 31,488 0.5 35,936,407.5 31,937.3 100.0 0 5.2E+10 47,232 0.8 35,936,407.5 31,937.3 100.0 0 5.2E+10 62,976 1.0 35,936,407.5 31,937.3 100.0 0 5.2E+10 75,571 1.2 35,936,407.5 5,941.2 100.0 0 5.2E+10 88,166 1.4 35,936,407.5 5,941.2 100.0 0 5.2E+10 100,762 1.6 35,936,407.5 5,941.2 100.0 0 5.2E+10 113,357 1.8 35,936,407.5 5,941.2 100.0 0 5.2E+10 125,952 2.0 35,936,407.5 5,941.2 100.0 0 5.2E+10 188,928 3.0 35,936,407.5 5,941.2 100.0 0 5.2E+10 251,904 4.0 35,936,407.5 5,941.2 100.0 0 5.2E+10 -------------------------------------------------------------


Streams Pool

• If you get spills to disk increase streams poolStreams Pool Advisory DB/Inst: TMSP/tmsp Snap: 24107 Size for Size Est Spill Est Spill Est Unspill Est Unspill Est (MB) Factor Count Time (s) Count Time (s)---------- --------- ----------- ----------- ----------- ----------- 1,536 0.8 0 0 0 0 1,792 0.9 0 0 0 0 2,048 1.0 0 0 0 0 2,304 1.1 0 0 0 0 2,560 1.3 0 0 0 0 2,816 1.4 0 0 0 0 3,072 1.5 0 0 0 0 3,328 1.6 0 0 0 0 3,584 1.8 0 0 0 0 3,840 1.9 0 0 0 0 4,096 2.0 0 0 0 0 4,352 2.1 0 0 0 0 4,608 2.3 0 0 0 0 4,864 2.4 0 0 0 0 5,120 2.5 0 0 0 0 -------------------------------------------------------------


Buffer Wait Statistics

• Tells you exactly where buffer busy waits are coming from

• Usually will be from disk blocks• Excessive disk block waits + db file sequential

reads means memory starvation• By themselves could indicate insufficient or

congested db writer processes


Buffer Wait StatisticsBuffer Wait Statistics DB/Inst: TEST/test Snaps: 24080-24107-> ordered by wait time desc, waits desc

Class Waits Total Wait Time (s) Avg Time (ms)------------------ ----------- ------------------- --------------data block 2,859,144 29,615 10segment header 1,173 25 211st level bmb 1,204 20 162nd level bmb 992 18 19undo header 15,332 14 1undo block 19,509 6 0file header block 2 0 30 -------------------------------------------------------------


UNDO (rollback) Segments

• Self adjusting• Starts with10 whose size is based on the size

of tablespace• When ratio of transactions to transactions per

rollback segment is > number of current undo segments, more are created

• They are never dropped


UNDO (Rollback) segmentsUndo Num Undo Number of Max Qry Max Tx Min/Max STO/ uS/uR/uU/ TS# Blocks (K) Transactions Len (s) Concurcy TR (mins) OOS eS/eR/eU---- ---------- --------------- -------- -------- --------- ----- -------------- 127 690.1 2,408,751 9,232 133 420/420 0/0 0/0/0/0/0/0 -------------------------------------------------------------Undo Segment Stats DB/Inst: TEST/test Snaps: 24080-24107-> Most recent 35 Undostat rows, ordered by Time desc Num Undo Number of Max Qry Max Tx Tun Ret STO/ uS/uR/uU/End Time Blocks Transactions Len (s) Concy (mins) OOS eS/eR/eU------------ ----------- ------------ ------- ------- ------- ----- ------------27-Jan 23:55 1,057 2,119 2,033 44 420 0/0 0/0/0/0/0/027-Jan 23:45 1,652 5,535 1,431 80 420 0/0 0/0/0/0/0/027-Jan 23:35 965 3,234 830 66 420 0/0 0/0/0/0/0/027-Jan 23:25 1,021 4,856 945 65 420 0/0 0/0/0/0/0/027-Jan 23:15 566 3,781 2,267 55 420 0/0 0/0/0/0/0/027-Jan 23:05 6,914 30,730 1,665 70 420 0/0 0/0/0/0/0/027-Jan 22:55 1,561 4,536 1,064 53 420 0/0 0/0/0/0/0/027-Jan 22:45 543 2,085 940 48 420 0/0 0/0/0/0/0/027-Jan 22:35 1,111 6,020 338 71 420 0/0 0/0/0/0/0/027-Jan 22:25 950 3,195 1,365 43 420 0/0 0/0/0/0/0/027-Jan 22:15 1,043 3,859 763 62 420 0/0 0/0/0/0/0/027-Jan 22:05 3,699 16,955 937 86 420 0/0 0/0/0/0/0/027-Jan 21:55 976 3,732 361 96 420 0/0 0/0/0/0/0/027-Jan 21:45 891 3,018 937 98 420 0/0 0/0/0/0/0/027-Jan 21:35 1,619 3,165 335 86 420 0/0 0/0/0/0/0/027-Jan 21:25 5,174 13,627 1,534 76 420 0/0 0/0/0/0/0/0


Segment Statistics

• Tells by type of action what segments are seeing most activity

• Tune objects by adding indexes, changing rows per block (RAC), tuning object related SQL


Segment StatisticsSegments by Physical Reads DB/Inst: TEST/test Snaps: 24080-24107-> Total Physical Reads: 27,349,451-> Captured Segments account for 53.8% of Total

Tablespace Subobject Obj. PhysicalOwner Name Object Name Name Type Reads %Total------- ---------- -------------------- ---------- ----- ------------ -------DNE4 MEP_D01 MODE_EXECUTION_POINT TABLE 3,350,928 12.25FMS FMS_DATA ACCTNG_EVENT TABLE 2,720,574 9.95DNE4 SHIPMENT_D SHIPMENT_PLAN TABLE 1,046,173 3.83ISS ISS_DATA IB_SHIPMENT_STATUS_H TABLE 852,046 3.12DNE4 TMS_AUDIT TEMP_SQL TABLE 552,676 2.02 -------------------------------------------------------------


Segment StatisticsSegments by Direct Physical Reads DB/Inst: TEST/test Snaps: 24080-24107-> Total Direct Physical Reads: 6,048,316-> Captured Segments account for 77.9% of Total

Tablespace Subobject Obj. DirectOwner Name Object Name Name Type Reads %Total------ ---------- -------------------- ---------- ----- ------------ -------FMS FMS_DATA ACCTNG_EVENT TABLE 2,720,470 44.98DNE4 MEP_D01 MODE_EXECUTION_POINT TABLE 1,468,961 24.29FMS FMS_DATA FIN_EVENT TABLE 521,600 8.62FAM FAM_D SYS_LOB0000148435C00 LOB 360 .01SYS SYSAUX SYS_LOB0000008958C00 LOB 19 .00 -------------------------------------------------------------


Segment StatisticsSegments by Physical Writes DB/Inst: TEST/test Snaps: 24080-24107-> Total Physical Writes: 5,084,963-> Captured Segments account for 36.9% of Total

Tablespace Subobject Obj. PhysicalOwner Name Object Name Name Type Writes %Total---------- ---------- -------------------- ---------- ----- ------------ -------ISD ISD_INDX ISD_SUPPORT_I03 INDEX 139,666 2.75ISD ISD_INDX ISD_SUPPORT_I02 INDEX 134,617 2.65DNE4 SHIPMENT_I SP_SUPP_INFO_I05 INDEX 113,698 2.24DNE4 SHIPMENT_I SP_SUPP_INFO_I04 INDEX 112,988 2.22DNE4_AUDIT TMS_AUDIT WORKFLOW_UNIT_AUDIT_ INDEX 109,555 2.15 -------------------------------------------------------------


Segment StatisticsSegments by Direct Physical Writes DB/Inst: TEST/test Snaps: 24080-24107-> Total Direct Physical Writes: 231,265-> Captured Segments account for 9.8% of Total

Tablespace Subobject Obj. DirectOwner Name Object Name Name Type Writes %Total---------- ---------- -------------------- ---------- ----- ------------ -------DNE4 DNE4_CLOB SYS_LOB0000050106C00 LOB 17,136 7.41DNE4_AUDIT TMS_AUDIT SYS_LOB0000126341C00 LOB 4,274 1.85FAM FAM_D SYS_LOB0000148435C00 LOB 719 .31SYS SYSAUX WRH$_ACTIVE_SESSION_ 0822_24013 TABLE 239 .10DNE4_AUDIT TMS_AUDIT SYS_LOB0000126341C00 LOB 184 .08 -------------------------------------------------------------


Segments by Table Scans DB/Inst: TMSP/tmsp Snaps: 24080-24107-> Total Table Scans: 6,618-> Captured Segments account for 2.5% of Total

Tablespace Subobject Obj. TableOwner Name Object Name Name Type Scans %Total---------- ---------- -------------------- ---------- ----- ------------ -------DNE4 INDEX01 GP_CITY_I4 INDEX 41 .62DNE4 RATE_INFO_ RATE_I3 INDEX 40 .60DNE4 MEP_D01 MODE_EXECUTION_POINT TABLE 25 .38FMS FMS_DATA ACCTNG_EVENT TABLE 10 .15DNE4 DNE4_D02 ME_SERVICE_GROUP TABLE 6 .09 -------------------------------------------------------------


Initialization Parameters

• Only a couple deal with IO• If the IO parameters are incorrect, may cost up

to 30% of IO performance


Initialization ParametersParameter Name Begin value ----------------------------- ----------------------------control_files /TEST/ctl01/control01.ctl, /TEST/db_2k_cache_size 268435456db_block_size 32768db_cache_size 15032385536db_file_multiblock_read_count 16db_files 5000db_keep_cache_size 268435456db_recovery_file_dest /archive/TESTdb_recovery_file_dest_size 751619276800log_archive_dest /archive/TESTlog_archive_format tmsp_%t_%s_%r.arclog_archive_max_processes 4log_buffer 262144000memory_max_target 68719476736memory_target 68719476736streams_pool_size 2147483648undo_management AUTOundo_retention 25200undo_tablespace UNDO_TMS


What is Missing?

• Disk_async_io – set to async• Filesystemio_options – set to setall• SORT_AREA_SIZE – take care of thos non-PGAT

sorts• Since they are using MEMORY_MAX_SIZE and

MEMORY_TARGET other items not set (PGAT)• Why should they be?


Why is IO Important?

• Most performance issues can be tracked to IO issues

• IO Latency is the most critical issue• Modern databases rapidly tax the most

complex disk based system


Oracle and QueriesWhere does latency matter?

MemorySGA & PGA

OracleProcesses

Tables & Indexes

Logs

Reads - Cache miss

Foreground Waits:

DB file sequential read

DB file scattered read

5 ms

User’s Query

Storage latency

READS


Oracle and Insert/update/deleteWhere does latency matter?

MemorySGA & PGA

OracleProcesses

Tables & Indexes

LogsLGWR

(foreground)

DBWR (background)

Users Insert

Commit

LOG WRITES


RamSan - Reads

MemorySGA & PGA

OracleProcesses

Logs

Reads - Cache miss

Foreground Waits:

DB file sequential read

DB file scattered read

0.25 ms

User’s Query

SSD latency


RamSan – WritesInsert, Update, Delete

MemorySGA & PGA

OracleProcesses

LGWR

SSD

(foreground)

DBWR (background)

Disk speed

Users Insert

Commit


Thank You!Mike Ault

[email protected]://www.statspackanalyzer.com/

using awr for io subsystem analysis

Technology

memory sga

log file switch

total wait time

wait time desc

operating system statistics

segment statistics segments

cache missforeground waits

buffer busy waits