Introducing The Oracle Disk Manager

Kevin Closson

Sr. Staff Software Engineer

VERITAS Software Corporation

Topics

Oracle9i Oracle Disk Manager (ODM)• What it is• How it operates• Performance• Getting From Here to There • Key Messages

More Information

Q & A

Key Messages

ODM is Oracle’s idea • It is how they want to do IO• They encourage customers to use filesystem files

ODM is not just a IO performance package

ODM support raw partitions mostly to reduce the pains of a migration

Usage of ODM today will not hinder Real Application Clusters deployment in the future• Veritas is soon to release the VERITAS Cluster File System

product which will support ODM• OMF• Ease of Administration for Real Application Clusters

ODM History

1999 20012000Pre-1999

Oracle andVERITAS sign

ODM agreement

VERITASreleases DBEdition 3.0

w/ODMsupport

Oracle developsinitial spec

of ODMOracle 9i

ships

Othervendors

offerODM?

Prototyping &early developmentestablish ODM’s

viability

Feature Overview

File IO Feature

File Management Features

File Identification Feature

Oracle File IO Basics

Oracle Server generates a very complex IO mix. Oracle IO operations include:• Scattered Reads

• Shadow Processes, MTS• PQO index range scans, etc

• Single-Block Reads• Shadow Processes, MTS, PQO.

• Direct Reads• Shadow Processes. Reads from TEMP segments, NOCACHE hint, etc• PQO Scan Processes

• Single-Block Asynchronous Writes • DBWR processes. Write batches bound for many files, need to be

asynchronous for performance’s sake• Direct Writes

• Shadow Processes. Writes to TEMP segments, direct path loads, etc• PQO writes

• Large Asynchronous Writes• lgwr writes - if redo buffer is larger than the port-level max supported IO,

LGWR will issue multiple async writes to flush the buffer

File I/O Features

Oracle without ODM• The Oracle Server resorts to many different library routines to

support the wide variety of IO operations. • Example:pwrite(),pread(),async_write(),readv(),read(),write() lio_listio(),kaio()

• Asynchronous DBWR page flushing requires two calls: one to issuethe I/O and another to poll for completed IO.

• Many of the Oracle IO operations, such as asynchronous IO, are not supported on UFS files.

Oracle with ODM• With ODM, Oracle needs only a single library routine

• odm_io()• odm_io() supports ALL Oracle file IO operations on ALL files (Raw

or VxFS)!• Gathered writes (DBWR) and LGWR asynchronous writes occur with a

single call to odm_io() without regard for file type (VxFS or RAW) or number of target files

File Management Features

File Creation Feature

File Resizing Feature

ODM File Creation Feature

ODM Includes features that enable more effective Oracle file creation

Without ODM, failed attempts to add files to a database will result in an unused file to be cleaned up from outside Oracle.

With ODM, files are no longer created with traditional open() or creat()• Files will be created with odm_create() and then initialized or

filled.

• If Oracle is happy with the file it calls odm_commit()

• If Oracle is not happy with the file, Oracle calls odm_abort(). The file will be completely cleaned up from within ODM.

With ODM, VxFS files are laid out with contiguous disk blocks. • This enhances table and index scan throughput

File Resizing Features

ODM offers enhanced support for tablespaceautoextend feature• This is a key requirement for the Oracle

Managed Files (OMF) feature • The space is added with odm_resize() • The space that is added is comprised of contiguous disk

blocks• This is not adding a datafile, but instead adding

extents to the existing datafile.• Oracle doesn’t autoextend raw partitions • With Non-ODM Filesystems, the added portion will not be

contiguous• UFS-based OMF configured databases will be extremely

fragmented from a physical disk block perspective

File Identification Feature

Large databases require too many kernel resources. Most notably per-process and system-wide file descriptor slots

With Oracle Dedicated Processes (Shadow Processes), large numbers of users combined with large numbers of datafiles becomes an issue.• Example: 1000 users on 500 datafiles will cost 500,000 kernel file

descriptor table entries• Note, the math is not really based on “Users”. Each Shadow process may

need to open every datafile. • A “User” may actually have many shadow processes depending upon the

application.

Kernel file descriptor usage is protected with locking. Performing a high rate of implicit/explicit file open/close operations will cause undesirable amount of kernel-mode CPU cycle wastage.• File open/close internals use the same kernel locking infrastructure.

File Identification Feature (cont.)

• Process birth/death causes implicit file open/close

• Oracle shadow processes open each datafile - but only at first access

• Example. A Shadow Process that connects to perform a table scan for a report will open every datafile the tablespace is comprised of.

File Identification Feature (cont)

With ODM, Oracle no longer uses file descriptors. Instead, ODM Identifiers are used.

ODM Identifiers are shareable from process to process within the node

Oracle caches ODM Identifiers in the SGA at instance startup• DBWR performs initial odm_identify() on all datafiles• CKPT identifies the control files and caches the information• LGWR identifies the redo logs and caches the information

ODM Identifier usage reduces Kernel overhead• Calls to exec(),and exit() no longer cause implicit contention on

Kernel open file table locks• Explicit contention is eliminated ( open() / close() )

S.A.M.E.

Special Application - S.A.M.E

Stripe And Mirror EverythingOracle’s recommended physical layout methodology• You’re going to be hearing a lot about this

Large tables and indexes typically comprised of many partitions, therefore many datafiles.

Deploying a database in accordance with S.A.M.E. methodology on raw partitions is extremely cumbersome.• Combining S.A.M.E methodology with large numbers of

datafiles creates incredibly complex logical volume layouts

Implementing SAME - Example100 Oracle data files

striped across 100 disks

then mirrored

SYSTEM.dbf EMP1.dbf EMP2.dbf EMP3.dbf ... roll1.dbf ORDERS1.dbf ORDERS2.dbf ORDERS3.dbf ...roll2.dbf CUST1.dbf CUST2.dbf CUST3.dbf …TEMP1.dbf TEMP2.dbf TEMP3.dbf ...

20,300 Volume Manager Objects

Implementing SAME with ODM -Example

Under ODM, 100 Oracle data files…

… striped across 100 physical disks.

… could reside in one VxFS filesystem on one logical VxVM volume…

SYSTEM.dbf EMP1.dbf EMP2.dbf EMP3.dbf ... roll1.dbf ORDERS1.dbf ORDERS2.dbf ORDERS3.dbf ...roll2.dbf CUST1.dbf CUST2.dbf CUST3.dbf …TEMP1.dbf TEMP2.dbf TEMP3.dbf ...

… then mirrored

… only 203 Volume Manager Objects

Performance Study

ODM Performance Goal

Brief Workload Description

Test Environment

Observations

ODM Performance Goal

“At least as good as Raw Partitions”

• A familiar tune

• VERITAS has done this before with QIO

ODM offers more than improved performance• It is very difficult to measure improved management!

Performance StudyWorkload Description

ODM performance study based upon an OLTP workload with real world characteristics, a diverse workload analysis• Adhoc Component• Real Users ( EXEC SQL CONNECT)• Dynamic User Count• Pro*C, not OCI• Some Dynamic SQL• Some Table Scans

Test Environment

Sun Microsystems Ultra Enterprise 10000 (A Domain)

• 12 Processors

• 12 GB Main Memory

• FCAL JBOD Disk attached

Preliminary performance results• Oracle9i Beta

1000

1500

2000

2500

3000

3500

4000

QUERYINTENSIVE

REPORTING MIXED OLTP UPDATE INTENSIVE

NONODM RAW

ODM

ODM vs NONODM ThroughputVarious Workloads

Tra

nsac

tions

per

Sec

ond

ODM Improves System Utilization

2500

3500

4500

5500

SMTX/sec MutexOps/sec

Locking Improvements, Reporting Workload.

NONODM

ODM VxFS

Smtx -5%, Mutex Ops -9%

ODM Improves System Utilization

20000250003000035000400004500050000

Sys Calls CSW

System Call and Context Switch Statistics Mixed OLTP Workload

NONODMODM VxFS

SysCalls -9%, Context Switches -6%

ODM Helps Reduce Oracle Contention

25000

35000

45000

55000

Latch Sleeps

Oracle Latch Sleeps. Mixed OLTP Workload.

NONODMODM VxFS

Latch Sleeps -10%

Oracle Helps Reduce Oracle Contention

2500

3500

4500

5500Buffer Busy

Waits

Buffer Busy Waits. Mixed OLTP Workload

NONODM

ODM VxFS

Buffer Busy Waits -7%

ODM Significantly Reduces Database Writer Overhead

250

450

650Seconds CPU

Database Writer Processor Consumption. Mixed OLTP - 30 Minute Test Interval

NONODM

ODM VxFS

Database Writer Processes summed CPU consumption -39%

ODM Significantly Reduces Log Writer Overhead

100

150

200

Seconds CPU

Log Writer Processor Consumption. Mixed OLTP - 30 Minute Test Interval

NONODM

ODM VxFS

Log Writer processor consumption -19%

Getting From Here to There

ODM is an Oracle9i feature• New Oracle9i databases can just commence with ODM• Oracle8 and Oracle8i databases come in many forms:

• UFS• Upgrade to Oracle9i first

• Work on migration to VxFS with ODM

• RAW• Upgrade to Oracle9i first• Enable ODM (ODM on Raw works nicely)• Work on migration to VxFS with ODM

• VxFS• Upgrade to Oracle9i• Enable ODM (placebo really)• Convert files to ODM (optional, but best to make them contiguous)

• VxFS with QIO• Upgrade to Oracle9i• Enable ODM• Convert files to ODM

• Oracle9i will support any mix-match of data file type. This is a significant aid to migration!

More InformationSee us here todaySee us on the web http://www.veritas.com/oracleOracle paper on S.A.M.E.• http://technet.oracle.com/deploy/availability/pdf/

oow2000_sane.pdf

Berkeley TPC paper• http://sunsite.berkeley.edu/Dienst/UI/2.0/Describe/

ncstrl.ucb/CSD-99-1070

QIO performance paper• http://www.oracle.com/support/library/news/

veritas_lss.html

The End

Thank you for your time.