configuring sun storage 7000 unified storage systems for ...hosteddocs.ittoolbox.com/8207213.pdf ·...

CONFIGURING SUN™ STORAGE 7000 UNIFIED STORAGE SYSTEMSFOR ORACLE® DATABASES

Jeffrey T. Wright, Application Integration EngineeringSridhar Ranganathan, Application Integration Engineering

Sun BluePrints™ Online

Part No 820-7213-10Revision 1.0, 1/23/09

Sun Microsystems, Inc.

Table of Contents

Configuring Sun™ Storage 7000 Unified Storage Systems for ORACLE® Databases . .1

Overview of the Sun Storage 7000 Unified Storage Systems. . . . . . . . . . . . . . . . . . . . 2

Ideal Application Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Configuring Unified Storage Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Basic I/O Service Time Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Test Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

128kB Skip-Sequential Read . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

128kB Skip-Sequential Write . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

8kB Random Read. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

8kB 75 Percent Read/25 Percent Write/100 Percent Miss . . . . . . . . . . . . . . . . . . 13

Estimating System Performance with Cache Hit and Miss I/O . . . . . . . . . . . . . . . . . 16

Data Access Characteristics Using Oracle Database Workloads . . . . . . . . . . . . . . . . 19

Environment for Oracle Database Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Oracle Database Tests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Oracle Database Testing Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Implementation Guidelines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Choosing a Unified Storage System Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Write-Optimized Flash Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Read-Optimized Flash Devices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Access Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Project and Share Configuration for Snapshot and Replication . . . . . . . . . . . . . . 34

General Database Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Detailed DTrace Analytics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

About the Authors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

Ordering Sun Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

Accessing Sun Documentation Online . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

1 Configuring Sun Storage 7000 Systems for Oracle Databases Sun Microsystems, Inc.

Configuring Sun™ Storage 7000 Unified Storage Systems for ORACLE® Databases

In planning storage solutions for Oracle database applications, Sun Storage 7000

Unified Storage Systems offer a range of performance and scalability options, and can

be configured to meet specific application requirements for capacity, performance, and

reliability. Because these systems offer such tremendous configuration flexibility,

storage architects face a number of decisions in designing optimal configurations to

meet Oracle database storage requirements.

To help the system planner understand how to accurately match a Unified Storage

System configuration to specific Oracle data access requirements, this article provides

results for two sets of I/O-related tests: testing of fundamental system I/O service times

and testing under a series of Oracle database workloads. This Sun BluePrints™ article

reviews configuration criteria and also discusses some important implementation

guidelines. The article addresses these topics:

• “Overview of the Sun Storage 7000 Unified Storage Systems” on page 2 highlights

storage system architecture and features, comparing and contrasting models

within the product family.

• “Basic I/O Service Time Measurements” on page 5 provides test results that

demonstrate general I/O performance characteristics. I/O service time

characteristics can help storage architects understand how Sun Storage 7000

Unified Storage System configurations can effectively support different types of

I/O workloads.

• “Estimating System Performance with Cache Hit and Miss I/O” on page 16 gives a

method for estimating average I/O response times based on the application’s

working set and projected rates of I/O requests that are serviced from cache, DRAM,

read-optimized flash, and disk media.

• “Data Access Characteristics Using Oracle Database Workloads” on page 19

presents test results for Oracle workloads that represent different types of

Oracle applications.

• “Implementation Guidelines” on page 32 reviews some configuration decisions

and criteria for selecting specific models and features according to Oracle

database workload characteristics.

This article is intended for storage architects who have a thorough understanding of

data storage systems and basic skills to interpret test results derived from Oracle and

operating system performance analysis tools.


In this article, all test results reflect a single client accessing a single share on a Sun

Storage 7000 Unified Storage System — all response time and throughput metrics

represent the single client/single share case. In practice, horizontal scaling of multiple

clients accessing multiple shares over multiple interfaces can yield higher system

throughput. Because such scaling depends strongly on specific system

implementations, this article does not specifically address optimizing performance

using multiple clients and multiple shares. Although this article focuses on testing with

Oracle database workloads, the results presented here can be extended to other

database software if the expected database workload is similar in composition.

Overview of the Sun Storage 7000 Unified Storage SystemsThe Sun Storage 7000 Unified Storage Systems are low-cost, fully-functional network

attached storage (NAS) storage devices designed around these core technologies:

• A general-purpose server as the NAS head

• A general-purpose operating system — the OpenSolaris™ Operating System and the

ZFS file system

• A large and adaptive two-tiered cache based on DRAM and flash memory devices

The Sun Storage 7000 Unified Storage Systems do not rely on proprietary operating

systems, controller hardware, or a particular vendor's disk, but instead incorporate an

open-source operating system and commodity hardware. Thus customers can leverage

industry-standard solutions in place of expensive, proprietary disk controllers. ZFS,

which is included in the OpenSolaris operating system, enables Hybrid Storage Pools

that provide data placement, data protection, and data services such as RAID, error

correction, and system management. These capabilities help to insulate applications

from failures in the underlying storage hardware.

Since Sun Storage 7000 systems are based on an open storage system architecture, they

offer significant cost savings while providing enterprise-class data services, good

scalability, and superior performance. They feature a common, easy-to-use

management interface, along with a comprehensive analytics environment to help

isolate and resolve issues. The systems support NFS, CIFS, and iSCSI data access

protocols, mirrored and parity-based data protection, local point-in-time (PIT) copy,

remote replication, data checksum, data compression, and data reconstruction.

To meet varied needs for capacity, reliability, performance, and price, the product

family includes three different models — the Sun Storage 7110, 7210, and 7410

systems. When appropriate data processing and data storage resources are configured

in these models, the systems can support nearly arbitrary data access requirements for

Oracle Online Transaction Processing (OLTP) applications, Decision Support Systems

(DSS), and mixed OLTP/DSS systems. Table 1 summarizes product features, comparing

tested models in the product family.


Table 1. Sun Storage 7000 Series Family Configurations

a The initial release offers 288 TB in usable capacity. A software upgrade will enable the system’sfull 576 TB capacity to be utilized.b A single system includes a 2U system plus a 4U expansion array. Clustered system includes two2U systems plus a 4U expansion array

Sun Storage 7000 systems combine general-purpose hardware and software

components in a design that simplifies hardware integration and enables

comprehensive features. Of note, a large and sophisticated adaptive resource cache

automatically identifies and places active data on high-performance DRAM (ARC)1 or on

read-optimized flash (L2ARC) on Sun Storage 7210 and 7410 systems. This capability

keeps data access times low for frequently and recently accessed data while storing

large amounts of infrequently accessed data on less expensive, high-density SATA disk

media. Because the cache provides exceptionally high cache hit rates for practical

business applications, it reduces the need to access data from spinning media.

Because of these storage systems’ tiered design, configuration decisions differ from

those for solutions that use traditional disk-based storage architectures. Specifically,

Sun Storage 7000 system performance depends strongly on configured flash, DRAM,

and CPU resources instead of the number of physical drives in the storage system.

Consequently, to accurately predict I/O performance under a particular application

load, the designer must understand how cache can service the application's active data

set, and match application-level data access times and data access requirements to

configurations of cache, disk, network, and CPU resources. This article provides

guidelines and test results to guide storage architects in the task of configuring these

systems to achieve optimal storage performance for Oracle database workloads.

1 The Unified Storage System ARC is based on the work “ARC: A Self-Tuning, Low OverheadReplacement Cache”, Nimrod Megiddo and Dharmendra S. Modha, Proceedings of FAST ’03: 2ndUSENIX Conference on File and Storage Technologies, 2003.

System Model

Maximum Storage Capacity

Space (Rack Units)

NAS Head Processor(s)

NAS Head Main Memory

Write-Optimized Solid State Drive (SSD)

Read-Optimized Solid State Drive (SSD)

Cluster Option

Sun Storage 7110 system

2 TB (16 x 2.5" SAS disks)

2U Quad-Core AMD Opteron

8GB N N N


44 TB (48 x 3.5" SATA II disks)

4U Two Quad-core AMD Opteron

Up to 64GB Up to 36 GB N N


576 TBa (576 x 3.5" SATA II disks)

6U for single system, or 8U for clustered configurationb

Up to Four Quad-Core AMD Opteron

Up to 128GB

Up to 288 GB for cluster configuration

Up to 600 GB Y


Ideal Application ProfilesWhile nearly any application profile can be supported with an appropriate

configuration, Sun Storage 7000 Unified Storage Systems are ideal for consolidating

many small and lightly accessed database servers on a single storage device. In the

case of database consolidation, the system’s large cache becomes a shareable resource

among all small client systems, and the system automatically migrates active data to

high-performance storage media and avoids over-provisioning media for inactive

clients. Thus the architecture reduces system cost without compromising operations.

The following application characteristics also help to determine whether a Sun Storage

7000 system can accelerate application performance:

• The application performs sufficient I/O operations per transaction to generate a

distribution of cache hit and cache miss I/O

• The application tolerates a bi-modal or tri-modal distribution of I/O response times

(i.e., it exhibits a range of I/O response times because of the distribution of hits and

misses in bi-modal configurations with DRAM and disk media or tri-modal

configurations with flash, DRAM, and disk media)

For example, an application that executes one read per transaction will have

inconsistent transaction response times because there is no way to guarantee whether

or not the I/O will be serviced from cache. On the other hand, an application that

executes 100 reads per transaction is more likely to have an I/O distribution that reflects

the engineered cache hit rate for the system. Likewise, if a user must perform a single

read operation before making a decision, then response times will be inconsistent —

but if the user can batch process data such that multiple I/O operations are performed

at each step, then response times will tend to be more constant.

Configuring Unified Storage Systems

Sun Storage 7000 Unified Storage Systems allow storage architects to control data

placement to minimize storage system costs and meet service level agreements (SLA).

Designing an optimal storage solution starts with quantifying data access requirements

in terms of protection, access time, access rate, and capacity. Once these requirements

have been defined, a Unified Storage System architecture can be created to meet these

requirements along with a pre-determined amount of headroom for growth.

In the context of configuring storage systems for Oracle database applications, the

following guidelines can help to estimate an appropriate architecture for arbitrary data

access requirements:

• Network interface configuration

– 12,000 IOPS (8kB) or 110 MPBS per 1Gb NIC port

– 1000 MBPS maximum per system

• CPU configuration

– One CPU core per 50 MBPS of client-side data traffic

– 16 CPU core maximum


• DRAM configuration

– Four to eight times the combined database buffer cache of all active clients

– 128GB maximum

– Maximum IOPS: 12,000 per 1Gb network interfaces

– Typical read response time: 0.3 ms

• Read-optimized flash device configuration (on the Sun Storage 7410 system only)

– One device per 16-32 GB of DRAM

– Six devices maximum

– One device per 2500 IOPS; 15,000 IOPS maximum

– Typical read response time: 5 ms

• Write-optimized flash device configuration (on Sun Storage 7210 and 7410

systems only)

– Eight devices maximum

– Striped option: one per 9,000 IOPS or 100 MBPS of client-side synchronous writes

– Mirrored option: two per 9,000 IOPS or 100 MBPS of client-side

synchronous writes

– Typical write response time: 0.3 - 0.4 ms

• Storage configuration

– Data protection: mirrored

– Physical drive count for performance: estimated back-end IOPS / 50 IOPS

per drive

– Physical drive count for capacity: estimated capacity / (0.5 * drive capacity)

– Typical read response time: 45 ms

Basic I/O Service Time Measurements

Basic I/O service time characteristics can help storage architects understand how a

storage system can effectively support an arbitrary database workload. For the Sun

Storage 7000 Unified Storage Systems, it is helpful to analyze four basic workloads:

• 128kB Skip-Sequential Read

The 128kB skip-sequential read workload measures read bandwidth between the

application server and tested Sun Storage 7000 system. The read could be from ARC,

L2ARC, or physical drives. 128kB sequential read throughput is useful for planning

storage systems for Data Warehouse applications, batch processing, and

backup operations.

• 128kB Skip-Sequential Write

The 128kB skip-sequential write workload uses synchronous I/O to measure write

bandwidth between the application server and either the back-end drives or the

write-optimized flash of the tested storage system. 128kB skip-sequential write

throughput is useful for planning database load, database recovery, and disk

sort operations.


• 8kB Random Read

8kB random read workloads demonstrate the controller's ability to use storage

media, including how hard the controller can effectively push the media and how

much time it takes the controller to process I/O to and from the media. (The

controller in the Sun Storage 7000 systems is the system server.) 8kB random I/O

rate and response time data is useful when planning solutions for Online

Transaction Processing (OLTP) applications.

• 8kB 75 Percent Read/25 Percent Write/100 Percent Miss

This workload demonstrates the controller's ability to handle a mixed workload,

including how long it takes the controller to process I/O to and from the media

when there are no cache hits. Writes are executed per synchronous I/O semantics.

Test Environment

All tests were conducted under the following client system configuration:

• Access protocol: NFSv3 over 1 Gb Ethernet

• NFS mount options: rw, bg, hard, nointr, rsize=32768, wsize=32768, proto=tcp,

vers=3, noac, forcedirectio

• Ethernet interface MTU: 1500

• Operating system: Solaris 10 OS x64 update 5

• Hardware platform: Sun Fire™ V40Z and Sun Fire 4100 Series servers

Table 2 details the hardware and software configuration for each tested Sun Storage

7000 Unified Storage System. The tested systems represent only a subset of available

configuration options. For data access requirements beyond those met by the tested

configurations, larger or smaller configurations can be deployed.

Table 2. Tested Storage System Configurations

Option Sun Storage 7110 System

Sun Storage 7210 System


CPU core count 4 8 8

Network interface 1x1Gb Ethernet 1x1Gb Ethernet 3x1Gb Ethernet

DRAM 8 64 16

Write-optimized flash 0 1 2

Read-optimized flash 0 0 1

Disk drives 16x10k RPM SAS 34x7200 RPM SATA

47x7200 RPM SATA

Data protection mirror mirror mirror

Share record size 8kB 8kB 8kB

Access protocol NFSv3 NFSv3 NFSv3


128kB Skip-Sequential ReadAs Figure 1 shows, all Sun Storage 7000 Unified Storage System models can service a

single process executing 128kB skip-sequential reads at 40 MBPS. However, the

scalability of throughput as the number of active I/O requests increases depends on the

number of CPUs and the number and type of network interfaces available in the

storage system. In this test, the Sun Storage 7110 and 7210 systems have equal

network configurations but the Sun Storage 7210 system has twice the CPU resources.

Consequently, throughput scales linearly on the Sun Storage 7210 system until the

network interface saturates, while throughput on the Sun Storage 7110 system shows

signs of non-linear scaling with only two outstanding I/O requests. In both cases, eight

outstanding I/O requests in the queue are sufficient to saturate the 1Gb network

interface. Configured with three network interfaces, the Sun Storage 7410 system

shows linear throughput scaling up to four outstanding I/O requests and saturation at

170 MBPS with eight outstanding I/O requests.

This behavior shows a practical limit for a single client accessing a single file system on

the Sun Storage 7410 system. Near linear throughput scaling occurs for two clients

accessing separate file systems through separate network interfaces on the Sun Storage

7410 system.

Figure 1. Throughput Versus Queue Length: 128kB Skip-Sequential Read.

128kB Skip-Sequential Write

As shown by the test results in Figure 2, Sun Storage 7210 and 7410 systems can service

a single process executing synchronous 128kB skip-sequential writes at 35 MBPS, while

the Sun Storage 7110 system can support only 5 MBPS. This difference is due to the Sun

0

50

100

150

200




32168421Queue Length

MB

PS


Storage 7110 system’s lack of write-optimized flash for synchronous write processing.

Consequently, the Sun Storage 7110 system must wait for the back-end disk drives to

flush I/O from the drive cache to the physical spindle before acknowledging that the

write is complete to the application. Non-synchronous I/O does not exhibit this

behavior, but in the case of Oracle databases, an assumption of synchronous writes

should be used for I/O planning.

Throughput limits on the Sun Storage 7110 system show suitable use cases in the

context of Oracle databases. For applications that can tolerate 5-15 MBPS of

synchronous write processing, the Sun Storage 7110 system provides a low-cost storage

solution. For applications that require higher throughput levels for synchronous write

processing, the Sun Storage 7210 and 7410 systems configured with write-optimized

flash devices are more appropriate building blocks.

Figure 2. Throughput Versus Queue Length: 128kB Skip-Sequential Write.

The throughput of Sun Storage 7210 and 7410 systems is limited by network bandwidth

and the latency of the write-optimized flash device. In the tested configurations, 16

outstanding I/O requests are sufficient to saturate a single network interface on the

Sun Storage 7210 system. In the case of the Sun Storage 7410 system, a second network

interface and a second write-optimized flash device is configured, and the saturation

characteristics reflect the interplay of the network interface limits and the limits of the

write-optimized flash device.

0

20

40

60

80

100

120


MBP

S





From the perspective of capacity planning for database load and index build operations

on the Sun Storage 7210 and 7410 systems, Figure 2 shows that four outstanding I/O

requests to a single share over a single 1Gb network link with one solid state write

processing device can process synchronous writes at 80 MBPS. As with 128kB sequential

reads, near-linear throughput scaling can be expected for two servers accessing two

shares over two network interfaces.

8kB Random Read

Because of the advanced cache architecture of the Sun Storage 7000 Unified Storage

Systems, 8kB I/O tests are run in cache hit and cache miss test cases and the combined

results of these tests are used to estimate average I/O response times for systems

running with arbitrary cache hit rates. 8kB random read miss and hit testing shows how

the Sun Storage 7000 Unified Storage Systems can respond to I/O requests for data

stored on disk and in cache. From the perspective of planning for Oracle database

applications, test results for 8kB read hit and 8kB read miss workloads can help system

planners to create solutions that meet arbitrary db file sequential read wait times

at arbitrary database I/O rates.

8kB Read Hit

The 8kB read hit test results demonstrate the best-case scenario for read performance

from the storage systems because the cache-friendly workload ensures that all data is

read from the ARC. Applications that have small active data sets compared to the size of

the cache can assume cache-hit I/O for the majority of I/O operations. In practice, not

all application I/O may be serviced from cache, so data access characteristics from the

storage target may be an average of both hit and miss I/O.

Figure 3 and Figure 4 show 8kB read hit performance. Figure 3 demonstrates

throughput as a function of queue length to show planners how the data access

characteristics an individual client can expect from the storage system. Figure 4

demonstrates I/O response time as a function of I/O rate for purposes of

general planning.

As these figures show, there are three important features when planning for read-hit

I/O workloads on the Sun Storage 7000 Unified Storage Systems:

• Latency in the lightly loaded case is 0.3 ms on all platforms

• Throughput scales linearly until the network bandwidth is saturated

• The network is the most significant factor affecting throughput


Consequently, a single host or client with a single outstanding I/O can expect 3000 IOPS

or 27 MBPS for 8kB transfers from the ARC. As the number of outstanding I/O

operations to the storage target increases, the total throughput increases until the

network saturates. Beyond this point, as described by simple queuing theory, each I/O

request observes an increase in I/O response time and a decrease in I/O throughput.

Figure 3. I/O Access Rate Versus Queue Length: 8kB Read Hit.

Figure 4. Response Time Versus Workload: 8kB Read Hit.

0

5000

10000

15000

20000

25000

2561286432168421Queue Length

IOPS




0 5000 10000 15000 20000 250000.0

0.5

1.0

1.5

2.0

2.5

3.0


Sun Storage 7210 SystemSun Storage 7110 System

IOPS

Serv

ice

Tim

e (m

s)


8kB Read Miss

The 8kB read miss test results demonstrate nominal read performance from the storage

systems because the cache-hostile workload in this test helps to ensure that nearly all

data is read from spinning media. Applications that have large active data sets

compared to the size of the cache should assume cache-miss I/O behavior for the

majority of I/O operations. In practice, some applications may benefit from cache hits

and data access characteristics will be an average of both hit and miss I/O.

Figure 5, Figure 6, and Figure 7 show three different ways to analyze 8kB read miss

performance. Figure 5 demonstrates system throughput as a function of the number of

outstanding I/O requests. For workloads that feature a known amount of active I/O

requests, system planners can use Figure 5 to estimate how much throughput they can

expect from the system and what the throughput would be for a given client.

Figure 5. I/O Access Rate Versus Queue Length: 8kB Read Miss.

Figure 6 shows client-observed I/O response time2 as a function of the total I/O rate per

drive. This chart is useful for understanding the number of drives required to meet a

specific I/O response time at an arbitrary level of system throughput. The data is

especially relevant because of the wide range of drive configuration options available

on the Sun Storage 7410 system.

2 I/O response is defined by the time for the I/O system call (e.g., pread64) to complete; this timereflects the asvc_t metric reported by iostat.

0

500

1000

1500

2000

2500

3000

3500

2561286432168421Queue Length

IOPS





Figure 6. Response Time Versus Workload per Drive: 8kB Read Miss.

Figure 7 shows I/O service times as a function of the total I/O rate and applies most

directly to Sun Storage 7110 and 7210 systems.

Figure 7. Response Time Versus Total Workload: 8kB Read Miss.

0 50 100 150 2000

10

20

30

40

50


Sun Storage 7210 SystemSun Storage 7110 System

IOPS per Active Drive

Serv

ice

Tim

e (m

s)

0 500 1000 1500 2000 2500 30000

10

20

30

40

50




IOPS

Serv

ice

Tim

e (m

s)


Read miss performance depends on drive speed and drive count. Due to the higher

drive speed, the Sun Storage 7110 system has lower I/O response times at a fixed I/O

rate per drive and higher I/O throughput for small number of active I/O requests. In the

case of the Sun Storage 7110 system, read-miss response time ranges from 12-30 ms

over a range of 10-125 IOPS per drive or 80-1500 total IOPS. Beyond 125 IOPS per drive or

1500 total IOPS, response time increases exponentially and total system throughput

saturates. For a single active I/O request, a 12 ms response time provides a single-

threaded throughput of 83 IOPS.

Compared to the Sun Storage 7110 system, the Sun Storage 7210 and 7410 systems

have longer response times and lower throughput per I/O request. This effect is due to

the slower drive speed on these models. However, the Sun Storage 7210 and 7410

systems scale to a higher total throughput for large numbers of outstanding I/O

requests. Read miss response times range from 25 ms in lightly loaded cases to 50 ms at

60 IOPS per drive. Beyond 60 IOPS per drive, the physical drive performance saturates

and I/O response time increases exponentially.

For planning purposes, the 44 drives available in Sun Storage 7210 configurations can

practically support up to 2600 IOPS, while the Sun Storage 7410 system can scale read

miss performance up to the maximum number of available drives. For example, a

mirrored 80 drive configuration in a Sun Storage 7410 system can support up to 5800

read miss IOPS at a 50 ms I/O response time.

8kB 75 Percent Read/25 Percent Write/100 Percent Miss

The 8kB 75 percent read/25 percent write/100 percent miss workload represents a mix

of read and synchronous write I/O operations. This workload tests how well the storage

device's I/O processing algorithm can balance the needs of front-end application I/O

with the capabilities of the back-end drives and the data protection algorithm. There

are three important results from this test to consider in the context of planning storage

solutions for Oracle database applications:

• Front-end read response time versus front-end I/O rate (Figure 8)

• Front-end write response time versus front-end I/O rate (Figure 9)

• Total throughput as a function of queue length (Figure 10)

Read response time (Figure 8) can be used to set expectations for how a user accessing

the database will see response times for lookup operations that miss the database

cache and must be serviced by spinning media. As the workload increases and system

throughput approaches saturation, a gradual increase in the read response time

translates to a gradual degradation in the end user's experience. Database operations

can be dramatically influenced by write response time (Figure 9) because write

response time directly affects database commit operations. For this reason system


planners should ensure that workloads are kept within a range that provides consistent

write response times. For example, Figure 9 shows the tested Sun Storage 7210 system

can produce consistent write response times up to 2500 IOPS.

Figure 8 and Figure 9 show read and write response times versus total I/O rate. The Sun

Storage 7110 system (with 12 active 10,000 RPM SAS drives and no write-optimized

flash devices) shows shorter read response times in the lightly loaded case with less

total throughput. In contrast, in the saturated case, the Sun Storage 7210 system

(configured with 44 active 7,200 RPM SATA drives and one write-optimized flash device)

scales to more total throughput but with a reduced throughput per active I/O request.

The performance of the Sun Storage 7410 system (configured with 32 active 7,200 RPM

SATA drives and two write-optimized flash devices) falls between that of the Sun

Storage 7110 and 7210 systems. Practical Sun Storage 7410 system configurations can

be scaled to 80 active 7,200 RPM SATA drives to achieve both increased throughput

and capacity.

Figure 8. Read Response Time Versus Workload: 8kB 75% Read/25% Write/

100% Miss.

0 500 1000 1500 2000 2500 30000

10

20

30

40

50




IOPS

Serv

ice

Tim

e (m

s)


Figure 9. Write Response Time Versus Workload: 8kB 75% Read/25% Write/

100% Miss.

Figure 10. I/O Access Rate Versus Queue Length: 8kB 75% Read/25% Write/

100% Miss.

0 500 1000 1500 2000 2500 30000

10

20

30

40

50




IOPS

Serv

ice

Tim

e (m

s)

0

500

1000

1500

2000

2500

3000


IOPS





Estimating System Performance with Cache Hit and Miss I/OThis section presents a simple model to predict average I/O response time for Sun

Storage 7210 and 7410 systems based on the percentage of I/O requests serviced from

DRAM (primary ARC), read-optimized flash (secondary L2ARC), or spinning disk. The

storage architect can use this information to design solutions to meet arbitrary

performance requirements using the following process:

1. Quantify the access rate and average access time requirements for the active

working set. (The I/O access rate is typically specified in terms of IOPS or

throughput (MBPS), and the access time defined in terms of read/write

response times.)

2. Estimate the size of the active working set for the targeted applications.

3. Identify the cache hit rate that supports access time requirements for the active

working set.

4. Configure DRAM and flash resources to help maintain the appropriate cache

hit rate.

5. Configure CPU and network resources to meet the data access rate requirements.

6. Configure physical disk drives to meet the total capacity and cache miss

performance requirements.

In a Sun Storage 7210 or 7410 system configured without read-optimized flash devices,

the average I/O response time reflects the time for I/O operations to be serviced from

cache and spinning disk. Thus a simple formula predicts the average I/O response time

Tave as a function of cache hit rate x:

Tave(x) = Tcache * x + Tdisk * (1 - x)

Tave is the sum of the time it takes to service an I/O from cache (Tcache) times the

cache hit probability, plus the time it takes to service an I/O from disk (Tdisk):

Tcache and Tdisk can be estimated from Figure 4 and Figure 6 by observing the 100%

hit and 100% miss cases. From the data, Tcache equals 0.3 ms and Tdisk equals 45 ms.

These results assume that there is enough network bandwidth to service I/O operations

in the event of a hit and enough disk bandwidth to service I/O operations in the event

of a miss. To plan storage solutions based on these heuristics, the system should run

below 10,000 IOPS per 1Gb network interface and 50 IOPS per physical disk drive. Under

these conditions the following formula estimates average I/O response time (in

milliseconds):

Tave(x) = 0.3 * x + 45 * (1 – x)

Table 3 shows estimated response times based on the ARC cache hit rate.


Table 3. Estimated Response Time Versus ARC Hit Rate

For Sun Storage 7410 system configurations with read-optimized flash devices, the

average read response time can be estimated by adding a Tflash parameter that

reflects the response time of the read-optimized flash device, and by including the

probability of hits in the flash cache. In this model, the primary cache hit rate is x, the

secondary flash cache hit rate is y, and a miss I/O operation occurs after a miss in the

secondary cache:

Tave(x,y) = Tcache * x + Tflash * y + Tdisk * (1 - y)

where

x + y ≤ 1

Tflash for the Sun Storage 7410 system has been measured as 5 ms when running flash

devices at 2,500 IOPS. Assuming less than 10,000 IOPS per 1Gb network interface and

less than 50 IOPS per physical spindle, the following formula estimates the average I/O

response time Tave as a function of the primary and secondary cache hit rates x and y:

Tave(x,) = 0.3 *x + 5 * y + 45 * (1 – y)

Table 4 lists estimated response times based on ARC and L2ARC cache hit rates.

ARC Hit Rate Average Read Response Time

100 0.3

95 2.5

90 4.8

80 9

70 14

60 18

50 22

40 27

30 32

20 36

10 41

0 45


Table 4. Estimated Response Times Versus ARC and L2ARC Hit Rates

ARC Hit Rate

L2ARC Hit Rate

Read Response Time (ms)

ARC Hit Rate

L2ARCH Hit Rate

Read Response Time (ms)

0 100 5 30 40 16

90 9 30 20

80 13 20 24

70 17 10 28

60 21 0 32

50 25 40 60 3.1

40 29 50 7.1

30 33 40 11

20 37 30 15

10 41 20 19

0 45 10 23

10 90 4.5 0 27

80 8.5 50 50 2.7

70 13 40 6.7

60 17 30 11

50 21 20 15

40 25 10 19

30 29 0 23

20 33 60 40 2.2

10 37 30 6.2

0 41 20 10

20 80 4.1 10 14

70 8.1 0 18

60 12 70 30 1.7

50 16 20 5.7

40 20 0 9.2

30 28 0 14

10 32 80 20 1.2

0 36 10 5.2

30 70 3.6 90 10 0.8

60 7.6 0 4.8

50 12 100 0 0.3


Data Access Characteristics Using Oracle Database WorkloadsThe previous section estimates response times based on rates at which I/O requests are

serviced from the primary ARC, secondary L2ARC, or disk. These estimates can help to

provide a starting point for planning solutions that meet specific data access rate and

access time requirements for Oracle database applications. To help storage architects to

more accurately optimize Sun Storage 7000 system configurations for Oracle

transaction processing systems, this section presents the results of testing against

actual Oracle database systems. By analyzing these results, the architect can better

understand how to apply the I/O service times and response time calculations in the

previous sections to predict complex database application behavior.

Storage performance was measured for five types of database activity: database load,

index build, OLTP processing, parallel table scan, and mixed OLTP/parallel table scan.

The results help to set appropriate expectations for practical system response time and

throughput, and demonstrate and quantify variations in performance that are often

inherent with complex Oracle database workloads.

The test results represent performance of a single Oracle instance running from a single

share on the storage target. In practice, a Unified Storage System can support an

increased level of throughput with minimal increases in response times when multiple

servers access multiple shares.

Environment for Oracle Database Testing

In the Oracle test environment, little tuning was done to the application, the database

instance, or the operating system. Thus the results represent conservative estimates for

system performance. In practice, careful system tuning can produce increases in

throughput and/or decreases in response time. However, system tuning

recommendations are generally application-specific and are beyond the scope of

this article.

Database Server Configuration

Figure 11 depicts the test configuration. The database servers used were either dual-

core dual-socket 3.1 GHz processors (Sun Fire 4100 servers) or dual-core 8-socket 2.4GHz

processors (Sun Fire V40z servers). Both servers were over-configured with CPU

resources compared to storage subsystem resources, so the processor configuration did

not significantly affect system throughput. Each system had 16-32 GB of RAM with 4 GB

reserved for the database buffer cache. Client and storage connections were made over


alternate 1Gb Ethernet interfaces. With the exception of parallel table scan tests, the

network interfaces had more bandwidth than that required by the application, so

performance results reflect storage subsystem response time for specific I/O workloads.

Figure 11. Test Configuration for Oracle Database Testing.

The test operating system was Solaris 10 OS Update 5. The default MTU (Maximum

Transmission Unit) of 1500 was used for all network interfaces. The kernel was tuned

per standard Oracle recommendations for a 12 GB SGA and up to 1 MB I/O transfers

as follows:

The TCP stack was tuned with ndd as follows:

Database Instance Configuration

The tested Oracle database was configured to emulate a small-scale system that would

be a good candidate for consolidation on a Unified Storage System. At the database

level, no specific cache tuning (such as reserving a “keep” pool for data that is known to

benefit from caching) was implemented. A single file system mount point was used for

all database data files. Oracle Managed Files (OMF) was used for file management, a

separate share was created on the same project as the database data for the database

flash recovery area, and a separate share was created for the archived redo logs. The

OracleDatabase Server

(Sun Fire 4100 Server or Sun Fire V40z Server)

Sun Storage 7110System



System Under Test

Database Client(Sun Fire X4600 Server)

set noexec_user_stack=1set shmsys:shminfo_shmmax=12704901885

ndd -set /dev/tcp tcp_conn_req_max_q 16384ndd -set /dev/tcp tcp_conn_req_max_q0 16384ndd -set /dev/tcp tcp_xmit_hiwat 131072ndd -set /dev/tcp tcp_recv_hiwat 131072ndd -set /dev/tcp tcp_naglim_def 1


database ran in archive log mode, and copies of the online redo log and control file

were multiplexed between the db_create_file_dest and the db_recovery_file_dest.

The server parameter file, init.ora, contained the following entries:

Database Application and Tables

Each storage architecture was tested using an order-entry system with the following

specifications:

• Capacity: 200 GB production data

• Transaction profile: 30 reads and 10 writes per transaction

• Peak workload: 180 transactions per second

• Response time: new-order transaction completes in < 3.0 seconds

The test application was based on a classical order entry system. The application had

nine tables (item, warehouse, stock, district, customer, history, orders, order line, and

new order), and executed new-order, stock-level, payment, and order-status

transactions. To prevent this implementation from creating an unrealistically simple

workload for the storage subsystem, no special partitioning techniques were used to

improve query execution.

log_archive_dest_1='LOCATION=/fishworks/oraarch'log_archive_format=%t_%s_%r.dbfdb_block_size=8192db_cache_size=4294967296db_file_multiblock_read_count=16cursor_sharing=forceopen_cursors=300db_domain=""db_name=benchbackground_dump_dest=/export/home/oracle/admin/bench/bdumpcore_dump_dest=/export/home/oracle/admin/bench/cdumpuser_dump_dest=/export/home/oracle/admin/bench/udumpdb_create_file_dest=/fishworks/oradatadb_files=1024db_recovery_file_dest=/fishworks/orafradb_recovery_file_dest_size=214748364800job_queue_processes=10compatible=10.2.0.1.0java_pool_size=0large_pool_size=0shared_pool_size=536870912processes=1024sessions=1131log_buffer=104857600audit_file_dest=/export/home/oracle/admin/bench/adumpremote_login_passwordfile=EXCLUSIVEpga_aggregate_target=1707081728undo_management=AUTOundo_tablespace=UNDOTBS1


Database Test Approach

Each storage subsystem configuration was characterized by its ability to support five

use cases: database load, index build, OLTP response time as a function of workload,

parallel table scan, and mixed OLTP/scan. The following accounting tools were used to

analyze each use case:

• Application: wall clock time, new order rate, and new order response time

• Operating system: vmstat and iostat (sar -d and sar -u)

• Database: Oracle STATSPACK

Oracle Database Tests

Database Load Test

The database load test subjects the storage system to deep queues (50-150 active I/O

requests) of large-block synchronous write I/O operations. Writes to the online redo log

pose the most important storage-based bottleneck in this test. Consequently, system

throughput reflects the total database I/O throughput based on the throughput

available to multiplexed copies of the online redo log. From the perspective of the

Oracle STATSPACK report, log file sync is the most important wait event. This event is

influenced by write response time and the operating system’s scheduling of Oracle's

log writer process (LGWR), so the test results reflect the combined limits of the

database server and the storage subsystem target.

The database preload test is executed as follows:

1. Begin without tables, indexes, or constraints defined.

2. Build tables with pre-allocated extents and all table constraints disabled.

3. Start iostat, vmstat, and create a STATSPACK snapshot; start wall clock.

4. Use multiple (40) processes to simultaneously load each table.

5. When all processes complete, stop vmstat, iostat, and wall clock, and take a

STATSPACK snapshot.

6. Create STATSPACK report; record vmstat, iostat, and wall clock results.

Index Build Test

The index build test performs full table scan and disk sort operations. This test drives

large-block sequential I/O to the storage subsystem and shows how efficiently a storage

system processes sequential transfers. Storage subsystem response time is a critical

factor in the execution time for this test. The index build test is executed as follows:


2. Use multiple processes to simultaneously build indexes.

3. When all processes complete, stop vmstat, iostat, and clock, and take a

STATSPACK snapshot.

4. Create STATSPACK report; record vmstat, iostat, and wall clock results.


OLTP Workload Versus Response Time Test

The OLTP workload versus response time test demonstrates the storage subsystem's

ability to service the random, high-concurrency, and correlated I/O workload associated

with online transaction processing systems. Since the data set is large with respect to

the host and storage array cache sizes, this test also quantifies media access

performance and how efficiently the storage subsystem uses the physical media. The

OLTP workload versus response time test is executed as follows:

1. Start iostat, vmstat, and create a STATSPACK snapshot.

2. Connect 100 clients to the database with a specified think time to control the work-

load (think time ranges from 0 to 12 seconds).

3. Measure transaction rate and transaction response time over a specified interval

(10 minutes) to determine an average rate and response time.

4. Stop iostat and vmstat, and create a STATSPACK snapshot.

5. Generate STATSPACK report.

Parallel Table Scan Test

The parallel table scan issues a parallel query hint for a table scan operation against a

stock table in the order-entry system as follows:

The count of the records is divided by the total execution time to calculate throughput

in terms of rows per second. The process to execute the parallel table scan is as follows:


2. Issue select statement to measure the total number of rows and the sum of a

non-indexed column.

3. When the scan completes, stop vmstat, iostat, and clock, and take a

STATSPACK snapshot.

4. Create STATSPACK report; record vmstat, iostat, and calculated rows per second.

Oracle Database Testing Results

Database Load

For a single database running from a single file system, a single 1Gb network interface

and a single write-optimized flash device provide sufficient bandwidth for write

processing. Table 5 lists the results from measuring Sun Storage 7210 and 7410 systems

with one and two write-optimized flash devices on 8kB and 64kB shares.

SELECT /*+PARALLEL( stock 16 )*/ COUNT( s_quantity ), SUM( s_quantity )INTO count, sumFROM stock;


Table 5. Database Load Results

The STATSPACK information reflects the average throughput over the entire load

process, which includes small and large numbers of clients pushing information into

the database. Consequently, the STATSPACK metrics do not accurately capture peak

system load. The iostat results reflect the combined work of writing the redo log

members and reading and writing the archived redo log. The iostat results cover a

short time period compared to the load process and consequently show a more

accurate indication of peak throughput. Write response time (defined as the active

queue length divided by the average service time) ranged from 0.20-0.36 ms per

32kB I/O.

The Unified Storage System’s cache architecture is designed and optimized to provide a

large amount of throughput for a large number of clients. As a result, buffering and

flushing data can result in decreased throughput and increased response time for

individual clients. In practice, periodic flushing on a 5-30 second time scale can cause

increased response time and reduced throughput for the client-side application during

the flush (which typically lasts 1-3 seconds). The results given here reflect average

throughput with this activity occurring. When quantifying performance requirements in

service level agreements (SLAs), storage architects should take into account that a

batch load process can incur increased latency because of more frequent data flushes.

Index Build

The index build test subjects the storage subsystem target to a large-block mixed

read/write workload typical of disk sort operations. In this test, the data set is much

larger than the buffer cache on the database server, so sort operations must be paged

to temporary space allocated on the storage target. Ten different indexes are built in

parallel, and each index is built with the PARALLEL( DEGREE 8 ) clause. The top Oracle

STATSPACK wait events reported during the index build test are typically a direct path

read or db file scattered read.

Metric Result Measured by

Average transaction rate 36-43 TPS STATSPACK

Average redo generation rate 14-16 MBPS STATSPACK

Log file parallel write wait time 140-160 ms STATSPACK

Average physical writes per second 1800-2100 STATSPACK

Peak write throughput 3900-4400 IOPS and 75-89 MBPS iostat

Peak read throughput 300-700 IOPS and 10-22 MBPS iostat

Queue length and service time 80-110 active and 22-29 ms iostat


For a single database running from a single file system, a single 1Gb network interface

and a single write-optimized flash device provide sufficient bandwidth for write

processing. Since this workload is a scan and sort operation, the presence of read-cache

devices does not strongly affect system throughput because the throughput of the

spinning media is sufficient to saturate other system resources. For example,

configurations with 16GB of DRAM perform comparably to systems with 64GB of DRAM,

and systems with read-optimized flash devices perform comparably to systems without

read-optimized flash devices. Share record size, however, strongly affects system

throughput — an index build throughput for systems with a 64kB record size is 40-75%

greater than systems with an 8kB record size. Table 6 list results measured on Sun

Storage 7210 and 7410 systems using 64kB and 8kB record sizes.

Table 6. Index Build Results

The STATSPACK information represents the average throughput over the entire build

process, which includes small and large numbers of clients pushing information into

the database. Consequently, the STATSPACK information does not accurately capture

peak load on the system. The iostat results reflect the combined work of read and

writing from and to the storage target, and since the iostat results cover a short time

period compared to the load process, they show a more accurate indication of peak

throughput. I/O response time (defined as the active queue length divided by the

average service time) was 6.9 ms per I/O in the case of the 8kB record size, and

3.7 - 5.1 ms in the case of the 64kB record size.

Metric Result Measured by

Average blocks per read 16 STATSPACK

Average tablespace reads per second with 8kB record size 310-330 STATSPACK

Average tablespace reads per second with 64kB record size 460-540 STATSPACK

Tablespace read response time with 8kB record size 29-51 ms STATSPACK

Tablespace read response time with 64kB record size 17-29 ms STATSPACK

Peak write throughput with 8kB record size

450-510 IOPS and 13-16 MBPS iostat

Peak write throughput with 64kB record size

350-600 IOPS and 11-17 MBPS iostat

Peak read throughput with 8kB record size

2700-3300 IOPS and 86-105 MBPS

iostat

Peak read throughput with 64kB record size

3200-3400 IOPS and 100-110 MBPS

iostat


OLTP Processing

Characterized by 8kB I/O operations to small isolated working sets that evolve with

time, OLTP processing highlights the effectiveness of the Sun Storage 7000 systems’

cache architecture. In cases where the active working data set is cached in DRAM or

read-optimized flash devices, access time to that data is extremely fast compared to

traditional storage systems. In practical cases, a subset of the active working set

typically resides in cache and the remainder of the data is served from disk media.

Consequently, the response time for a given I/O request is either bi-modal (in the case

of a system configured with DRAM and spinning media), or tri-modal (in the case of a

system configured with read-optimized flash in addition to DRAM and spinning media).

The I/O distribution, however, is not easily captured in an average measurement using

application-level or operating-system level tools such as STATSPACK or iostat, so

average measurements must be interpreted in the context of the underlying

distribution. From a planning perspective, this means that the storage architect should

define SLAs that provide an appropriate allocation for both cache hit and cache

miss I/O.

The test environment consists of a 200GB order entry system, and the Oracle instance

maintains a 4GB buffer cache that produces a 90-95% cache hit ratio. As reported by the

STATSPACK Load Profile, the physical I/O for the transaction profile is

10-30 reads per transaction and 7-15 writes per transaction. The lightly loaded cases

typically have fewer reads and more writes per transaction, and the heavier loaded

cases have more reads and fewer writes per transaction.

The most significant factors affecting storage system performance are the ratio of cache

hit to cache miss I/O and the share record size. For systems where more I/O is served

from cache, I/O response time is shorter and the variation in average I/O response time

is less than in systems where fewer I/O requests are served from cache. To simplify

analysis and discussion, reads from either the primary ARC cache stored in DRAM or the

secondary L2ARC cache stored on read-optimized flash are treated as a cache hit. Misses

in the L2ARC are treated as a cache miss. This process overstates I/O response time as a

function of cache hit for systems with low hit rates in the primary ARC cache, but it is

accurate for systems with higher hit rates in the primary ARC compared to the

secondary L2ARC. To capture the impact of caching, cache statistics are gathered

though DTrace Analytics capabilities that are built into the Sun Storage 7000 Unified

Storage Systems.


Figure 12 shows database-reported read response times (db file sequential read)

versus the cache hit rate for 8kB and 64kB record sizes on 1-tier and 2-tier systems

running with a cache hit range of 50-100 percent. As Figure 12 shows, high cache hit

rates result in shorter response times and less response time variation, and lower cache

hit rates exhibit longer response times and greater response time variation.

Figure 12. Database Read Response Time Versus Cache Hit Rate.

Note that the I/O response time in the 100% cache hit case is greater than predictions

made in the earlier section (see “Estimating System Performance with Cache Hit and

Miss I/O” on page 16). This is a consequence of the model’s simplicity compared to the

complexity of the actual database workload and the possibility of a 100% cache hit

condition coming from a combination of DRAM and read-optimized flash resources.

For planning purposes, the results suggest a practical limit of 1.25 ms for 100% cache

hit systems, a 5ms response time for 90% cache hit systems, and an increase of 0.34 ms

for each cache hit percentage point under 90%. The linear regression predicts a 35ms

response time in the 0% cache hit case. This is a shorter response time than that

predicted from the model presented earlier and is a consequence of the disk drives

being lightly loaded (e.g., <30 IOPS/drive) during the test.

Figure 13 shows the cache hit rate on the storage target as a function of the total I/O

rate from the database perspective. The first case (1-Tier, 64GB) represents a large

single-tiered configuration on a Sun Storage 7210 system with DRAM cache only. The

second case (2-Tier 16GB/100GB) represents a small primary cache on DRAM and a large

secondary cache using read-optimized flash on a Sun Storage 7410 system. The active

40 60 80 1000

5

10

15

20

25

30

Read Response Time

Cache Hit Percent

DB

File

Seq

uent

ial R

ead

(ms)

Linear Regression (Read Response Time)


data set size is the same in both cases. The results show that a larger primary cache can

support more cache-hit I/O under light workloads, and that a small primary cache with

a larger secondary cache is more effective under heavy workload conditions.

This effect is driven by two factors:

• The working set size becomes larger as more clients drive the database harder and

the smaller total cache size in a single tier system cannot maintain the entire working

set in cache

• The time to warm up the second tier cache becomes shorter as the total workload on

the system increases

From a planning perspective, lightly loaded systems benefit more from a large primary

cache tier and no secondary cache, and heavily loaded systems benefit from a large

secondary cache and a smaller primary cache.

Figure 13. Cache Hit Rate as a Function of I/O Rate.

Write response time characteristics as a function of workload play an important role in

database performance. The OLTP system under test has more read requests than write

requests per transaction (30 reads per 10 writes), so log write response time to the

write-optimized flash device is not the most significant factor in system performance.

Notwithstanding, write response time can play a significant role in performance for

lightly threaded applications, or for applications that perform few or no additional I/O

0 1000 2000 3000 4000 5000 6000 7000 800040

60

80

100

2-Tier (16GB/100GB)1-Tier (64GB)

Database I/O Rate(rps + wps + 2*tps)

Cach

e H

it P

erce

ntag

e

Linear Regression (1-Tier)Linear Regression (2-Tier)


operations per transaction. In these cases, it is useful to understand log write response

time as a function of the database transaction rate in the context of small

OLTP transactions.

Figure 14 shows the variation in log write response time versus the database-reported

transaction rate for systems running with one or two write-optimized flash devices and

a single 1Gb network interface. In lightly loaded cases, log writes are serviced in 1 ms,

and response time falls below 4 ms for transaction rates below 100 transactions per

second (TPS). Beyond 100 TPS, larger variations in log write response times require

more careful planning and tuning to optimize the system for the more

demanding workload.

Figure 14. Log Write Response Time Versus Transaction Rate.

Figure 15 shows how 8kB and 64kB record sizes affect OLTP throughput and transaction

response time for an Oracle database formatted with an 8kB database block size. The

results show that the 8kB record size delivers 80% more throughput at a 300 ms

transaction response time, with a 60% reduction in response time at lower throughput

levels. For systems running pure OLTP workloads, the 8kB record size is the most

appropriate implementation. However, for systems running heterogeneous workloads

with concurrent OLTP and scan operations, the 64kB record size offers a compelling

trade-off to balance available throughput for both workloads.

0 50 100 150 2000

5

10

15

20

Log Write Response Time

Database Transaction Rate (TPS)

Log

File

Par

alle

l Wri

te (

ms)

Linear Regression (Log Write Response Time)


Figure 15. Application Transaction Response Time Versus Application Transaction

Rate for 8kB and 64kB Record Sizes.

Parallel Table Scan

Parallel table scan operations provide guidelines to help plan deployments that support

Decision Support Systems (DSS). In combination with basic I/O service times and index

build data described previously, the bandwidth measurements from the parallel table

scan tests provide the storage architect with guidelines for the number of parallel I/O

streams necessary to extract full system throughput, as well as estimating how much

throughput can be expected.

As with other tests, this test case reflects a single database read from a single share

over a single 1Gb network interface. The table scan operates on the 60 GB stock table of

an order entry system. The average row size in the table is 300 bytes, and the table has

200 million rows. The table is subject to updates but not appends, so the scan reflects

reading of a fixed working set size. The operating system tool iostat measures the scan

I/O rate for the NFS share. The database-reported physical read rate is measured with

STATSPACK, and the user-level application records the number of rows per

second scanned.

Per I/O latency and network interface bandwidth are the critical factors affecting

performance for isolated parallel table scans. In this workload, for both 8kB and 64kB

shares, throughput for the scan was as follows:

• Application throughput: 270k-300k rows/second

• Database throughput: 9900-11000 reads/second

0 1000 2000 3000 4000 5000 60000.0

0.1

0.2

0.3

0.4

0.5

64kB Record Size

8kB Record Size

New Orders per Minute (1/minute)

New

Ord

er R

espo

nse

Tim

e (s

ec)


• iostat throughput: 3300-3600 reads/second, 86-110 MBPS read, 50 active I/O,

15-20 ms (asvc_t)

These results set reasonable expectations for table scan throughput over a single 1Gb

link. Practical DSS systems can benefit from horizontal throughput scaling over multiple

shares accessed on multiple links. In a horizontal scaling model, per I/O latency will

remain constant or increase as workload increases. Consequently, the degree of

parallelism used for the scan must be increased to engage all of the horizontally

scaled resources.

Mixed OLTP Processing with Parallel Table Scan

Heterogeneous data processing — where an OLTP processing workload runs

concurrently with DSS processing — presents a more complex workload to the storage

subsystem. Such a mix requires 8kB random lookups that are sensitive to latency as

well as 128kB lookups that are sensitive to bandwidth. Due to the disparate nature of

I/O requirements between these two workloads, a configuration that supports a mixed

workload must balance performance for both applications at the expense of optimal

performance for either application.

A mixed processing workload requires the selection of appropriate client-side workload

levels. This test takes the approach of running the OLTP workload at 50% throughput to

represent off-hours processing, and running the DSS workload at full throughput to

represent a business requirement that mandates completing DSS processing as quickly

as possible. Assessing the performance of mixed workload processing can be done by

comparing response times and system throughput in the homogenous case to the

response time and throughput of the heterogeneous case. For the OLTP workload, the

application-level transaction rate and application-level transaction response time

provide good metrics because they closely indicate a change in the end user’s

experience. For the DSS component (represented by a parallel full table scan), the

application-level scan rate (in terms of rows per second) and the total time to scan the

data set capture how the mixed workload affects timing windows that represent low-

production OLTP workload periods.

In the context of Unified Storage Systems, two design criteria are critical for planning

for mixed workloads: the amount of DRAM supporting the primary ARC cache and the

the share record size. The amount of DRAM strongly influences OLTP processing, while

the share record size has a strong effect on throughput for the table scan operation

running concurrently with OLTP processing.

For 64kB and 8kB record sizes OLTP processing throughput drops 10-15% during table

scan operations. In the system configured with 64GB of DRAM supporting the primary

ARC cache and no L2ARC cache, OLTP transaction response time increases 50% during

the table scan. In the system configured with 16GB of ARC cache and 100GB of L2ARC


cache, OLTP transaction response time increases 100% during the scan. These data

show that the larger ARC cache provides greater benefit for mixed processing than a

smaller ARC cache and a larger L2ARC cache.

The size of the primary ARC cache and secondary L2ARC cache weakly affects

throughput for the table scan operation — however, the share record size strongly

affects throughput for the table scan operation that runs concurrently with OLTP

processing. In the case of the 64kB share record size, table scan throughput drops 25%

when OLTP processing is added to the scan operation. In the other case with an 8kB

record size, table scan throughput drops 50% when OLTP processing is added to the

table scan operation. This data shows that significant performance gains can be made

for scan throughput if the system is designed to support OLTP processing with a 64kB

share record size.

Implementation Guidelines This section provides an overview of implementation details that should be considered

when deploying Oracle databases on Sun Storage 7000 Unified Storage Systems.

Important topics include the choice of Unified Storage System model, data protection

on the storage pool, write-optimized and read-optimized flash devices, access protocol,

and project and share configuration when using replication and snapshot.

Choosing a Unified Storage System Model

A critical configuration decision involves selecting a particular model in the family of

Sun Storage 7000 Unified Storage Systems — the Sun Storage 7110, 7210, or 7410

systems. Of the three models, the Sun Storage 7110 and 7210 systems cannot be

clustered, but the Sun Storage 7410 system can be deployed in a clustered

configuration. For this reason, the Sun Storage 7410 platform is mandatory for

applications that require a highly available NAS head. For applications that can tolerate

outages, the Sun Storage 7110 and 7210 can be used instead. The Sun Storage 7110

system, however, does not support write-optimized flash devices. Consequently,

synchronous writes from the database require back-end media access to be complete

before the write can be acknowledged to the database instance. While this does not

impede read throughput, it limits write throughput for typical database workloads to

<20 MBPS and write latency for typical database workloads to >15 ms. For data access

requirements that fit in this range, the Sun Storage 7110 system offers a small and low-

cost platform. For applications that require solid-state write acceleration, the Sun

Storage 7210 system is a more appropriate choice.

Write-Optimized Flash Devices

Write-optimized flash devices provide solid-state acceleration for synchronous write

requests, and they are available on both the Sun Storage 7210 and 7410 platforms.

Write-optimized flash devices may be deployed in two configurations: striped and


mirrored. In the striped configuration, data on the flash device is protected by a second

copy in DRAM on the NAS head. In the event of a loss of the write-optimized flash

device, there is a short time (less than 60 seconds) where writes that have been

acknowledged to the host are not stored on persistent media. Following the failure, all

writes are acknowledged after being written to spinning media, and the system will run

with reduced synchronous write throughput and increased synchronous write response

times. For applications with requirements met by these availability and performance

features, the striped configuration of the write-optimized flash device offers a low-cost

and high-performance solution. For applications that require redundant copies of data

on alternate persistent storage, or applications that cannot tolerate the performance

degradation associated with completely losing all write-optimized flash devices, then a

mirrored configuration of the write-optimized flash device is appropriate.

Read-Optimized Flash Devices

Read-optimized flash devices are available on the Sun Storage 7410 platform and offer a

low-cost extension of the storage device's cache resources. Slower in speed and larger

in size than comparably priced DRAM, read-optimized flash devices are a compelling

storage technology since they afford a compromise between the high cost of DRAM and

the long access time of spinning media. Technical optimizations of read-optimized flash

restrict share record sizes to less than 16kB — therefore, systems using read-optimized

flash should be designed around the performance of 8kB-16kB record sizes. Since the

read-optimized flash resides in the NAS head and is not replicated to an alternate NAS

head, the loss of a NAS head in a cluster causes the loss of any performance gain due to

the read-optimized flash devices. When defining service level agreements, the storage

architect should take this point into consideration.

Access Protocol

The Unified Storage System supports NFS, CIFS, and iSCSI access protocols, and any of

these protocols may be used to support an Oracle database. Careful optimizations in

the NFS stack, however, result in the highest levels of performance over NFSv3 protocol.

For applications that require CIFS or iSCSI protocol, system throughput may drop or

system response time may increase compared to NFS depending on the specific

workload. Furthermore, the iSCSI implementation provides a volatile cache option that

is disabled by default. This option should be left disabled for general purpose

applications, clustered applications, and applications that do not manage synchronous

writes through the SCSI SYNCHRONIZE CACHE command.


Project and Share Configuration for Snapshot and ReplicationThe Sun Storage 7000 Unified Storage Systems provide a wide range of configurations

to best match storage technology to business requirements. Several implementation

choices are important for Oracle databases depending on the requirements for remote

replication, snapshot, and horizontal scaling over multiple network interfaces or

storage devices for increased throughput.

On the Unified Storage Systems, replication is accomplished by periodic replication of

project snapshots. Consequently, replication is asynchronous from the perspective of

the application, and write-order is maintained only at the project level. To use

automatic replication to protect an Oracle database, the entire database must be

contained within the same project. Manual replication (controlled through scripts) can

be coordinated with Oracle hot backup to replicate databases that span

multiple projects.

Snapshot can be executed at the project or share level. When combined with hot-

backup at the database level, databases that span multiple projects can be backed up

with snapshot. For iSCSI implementations with Oracle ASM, however, hot backup is not

available because Oracle ASM does not have a hot backup mode. In this case, snapshot

backup can protect the database if the database resides on a single iSCSI LUN and

snapshot is executed against the LUN.

General Database Layout

Although manual data placement can yield better throughput for specific workloads, a

stripe and mirror everything (SAME) approach is an effective solution for general-

purpose Oracle databases. The Unified Storage System, in addition to automatically

caching frequently and recently accessed data on high-performance DRAM and read-

optimized flash resources, implements a SAME architecture over the physical drive

resources for all data when mirrored data protection is selected as a configuration

option. Consequently, the logical layout of database data on Unified Storage System

shares can be designed arbitrarily to optimize other business processes, such as

backup or redeployment, without compromising the performance benefits of a

SAME architecture.

Although optimal deployment depends on the requirements for a specific system, the

following guidelines provide a good starting point for Oracle implementations on

Unified Storage Systems:

• Segregate different databases to different projects

• Store all databases files for a specific database in the same project

• Segregate online redo log files and control files, production data files, temporary

files, and recovery files to separate shares within the same project


• Segregate production data files to separate shares within the same project if the data

files are subject to significantly different workloads or requirements

• Fine-tune share settings for recordsize, compression, checksumming, and security

based on the requirements for the share

Storing all database files in the same project streamlines replication and snapshot

backup implementations because write order is preserved within a specific project.

Segregating data files from temp files, log files, control files, and recovery files allows

for both flexibility and simplicity in rollback, restore, and deployment of alternate

systems using snapshot and clone features. Also, segregating specific database

components to separate file systems within the same project allows for optimal tuning

of share recordsize, compression, checksumming, and security features to match the

specific requirements of the component.

However, SAME may not be the right fit in all implementations. For example, SAME

may not provide an optimal storage architecture for databases that store data with

disparate value and diverse data access requirements. In these cases, segregating the

database to specific storage tiers that can support different access requirements can

help to provide a cost-effective solution that still meets business and operational

requirements. In a tiered storage architecture, SAME is used over all elements in a

specific storage tier, and multiple storage tiers support the database.

Detailed DTrace Analytics

Once Sun Storage 7000 Unified Storage Systems are deployed, storage architects and

administrators may find it necessary to troubleshoot application performance issues.

DTrace Analytics is a dynamic tracing framework that helps organizations simplify the

task of identifying causes behind intermittent or sustained application performance

problems. With Analytics, administrators and application developers can instrument a

live operating system kernel and running applications without rebooting the kernel,

recompiling, or even restarting applications. Instrumentation can be activated as

needed, without adding any overhead when tracing is turned off.

For Sun Storage 7000 Unified Storage Systems, Analytics provides real-time

observability for the entire system, including:

• NFS V3 and V4

• CIFS

• iSCSI

• ZFS

• CPU

• Memory utilization

• Networking


The in-depth Analytics feature helps organizations to fine-tune deployed Sun Storage

7000 Unified Storage Systems. For instance, Analytics can help administrators

understand and optimize workloads in real time, aiding them in:

• Understanding the benefits of write-optimized and read-optimized flash devices for

specific storage workloads

• Understanding when CPU, memory, and networking are creating bottlenecks

• Understanding the read/write/metadata mix of particular workloads

For more information on Analytics, see the white paper entitled “Architected for Open,

Simple, and Scalable Enterprise Storage: Sun Storage 7110, 7210, and 7410 Unified

Storage Systems.”

SummarySun Storage 7000 Unified Storage Systems offer incredible flexibility with respect to

storage system configuration. Storage architects can design systems to optimize data

placement, taking advantage of various combinations of system DRAM, flash, and

spinning disk media. Understanding the characteristics of Oracle database applications

can help system planners define data access requirements for data protection, access

times, access rates, and storage capacities. After these requirements are quantified, the

storage architect can more accurately match the specific configuration of the Unified

Storage System. The test results and decision criteria presented here can help the

architect in developing an optimal storage system configuration.

About the Authors Jeff Wright works in the Application Integration Engineering (AIE) Group in Sun's

Systems Organization. Jeff remains obsessed with performance topics involving Oracle

and data storage technologies from product conception to final delivery. His recent

focus has been on improving the timeliness and accuracy of storage solutions that Sun

proposes and delivers to customers. Prior to joining Sun in 1997 through the StorageTek

acquisition, Jeff worked as systems test engineer for disk storage products. His early

career experiences in automotive manufacturing, software development, and

experimental physics shaped his unique approach and perspectives on assessing

storage performance and designing storage architectures for database systems.

Sridhar Ranganathan also works in the Application Integration Engineering Group in

Sun's Systems Organization. His group's charter is to continuously improve the

competitive standing of Sun open storage products for key independent software

vendor (ISV) applications. Sridhar works on feature, functionality, and performance

tests involving Oracle database management systems in conjunction with traditional

storage and open storage technologies. Prior to joining Sun in 2000, his early career

experiences were in software development and database administration in the

manufacturing and banking industries.


References

Bitar, Roger. “Deploying Hybrid Storage Pools With Flash Technology and the Solaris ZFS

File System,” Sun BluePrints Online, October 2008. To access this article online, go to

http://wikis.sun.com/display/BluePrints/Deploying+Hybrid+

Storage+Pools+With+Flash+Technology+and+the+Solaris+ZFS+

File+System

Shapiro, Mike. “An Economical Approach to Maximizing Data Availability: Sun Storage

7000 Unified Storage Systems,” Sun BluePrints Online, November 2008. To access this

article online, go to http://wikis.sun.com/display/BluePrints/

An+Economic+Approach+to+Maximizing+Data+Availability

Wright, Jeffrey T. “Balancing System Cost and Data Value With Sun StorageTek Tiered

Storage Systems,” Sun BluePrints Online, February 2008. To access this article online,

go to http://wikis.sun.com/display/BluePrints/

Balancing+System+Cost+and+Data+Value+With+Sun+StorageTek+Ti

ered+Storage+Systems

Sun Microsystems. “Architected for Open, Simple, and Scalable Enterprise Storage: Sun

Storage 7110, 7210, and 7410 Unified Storage Systems White Paper,” November 2008.

To access this white paper online, go to http://www.sun.com/offers/

details/Unified_Storage_Systems_Architecture.html

Sun Storage and Oracle Partnership: http://sun.com/storagetek/

partnerships/oracle

Sun Unified Storage Systems: http://www.sun.com/storage/

disk_systems/unified_storage

Ordering Sun DocumentsThe SunDocsSM program provides more than 250 manuals from Sun Microsystems, Inc. If

you live in the United States, Canada, Europe, or Japan, you can purchase

documentation sets or individual manuals through this program.

Accessing Sun Documentation Online The docs.sun.com web site enables you to access Sun technical documentation

online. You can browse the docs.sun.com archive or search for a specific book title

or subject. The URL is http://docs.sun.com/

To reference Sun BluePrints Online articles, visit the Sun BluePrints Online Web site at:

http://www.sun.com/blueprints/online.html

Configuring Sun™ Storage 7000 Unified Storage Systems for ORACLE® Databases On the Web sun.com

Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, CA 95054 USA Phone 1-650-960-1300 or 1-800-555-9SUN (9786) Web sun.com

© 2009 Sun Microsystems, Inc. All rights reserved. Sun, Sun Microsystems, the Sun logo, Sun BluePrints, Sun Fire, Solaris, and OpenSolaris are trademarks or registered trademarks of Sun Microsystems, Inc. or its

subsidiaries in the United States and other countries. ORACLE is a registered trademark of Oracle Corporation. Information subject to change without notice. Information subject to change without notice.

Printed in USA 01/09