emc proven high performance solution for oracle rac on vnx · pdf filewhite paper emc...

White Paper

EMC Solutions

Abstract

This solution demonstrates the benefits of deploying EMC® XtremCache™ and EMC VNX® to increase IOPS and reduce latency for OLTP databases, and of deploying VNX for Data Warehouse workloads using Oracle RAC. It provides scalability, high performance, and ease of use for mission-critical business demands.

March 2014

EMC PROVEN HIGH PERFORMANCE SOLUTION FOR ORACLE RAC ON VNX EMC VNX8000, EMC XtremSF, EMC XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition

Optimum IOPS for Oracle OLTP workloads Optimum throughput for Oracle data warehouse workloads

EMC Proven High Performance Solution for Oracle RAC on EMC VNX8000, XtremSF, XtremCache, Red Hat Enterprise Linux,

Oracle Database Enterprise Edition White Paper

2

Copyright © 2014 EMC Corporation. All Rights Reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

EMC2, EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries. All other trademarks used herein are the property of their respective owners.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

Part Number H12859

3 EMC Proven High Performance Solution for Oracle RAC on VMAX EMC VNX8000, XtremSF, XtremCache, Red Hat Enterprise Linux,


Table of contents

Executive summary ............................................................................................................................... 5

Business challenge .......................................................................................................................... 5

Technology solution: open architecture components ....................................................................... 5

Open standards benefits ............................................................................................................. 6

Operational advantages .............................................................................................................. 6

Solution overview ............................................................................................................................ 6

Key results and recommendations ................................................................................................... 7

Introduction.......................................................................................................................................... 9

Purpose ........................................................................................................................................... 9

Scope .............................................................................................................................................. 9

Audience ......................................................................................................................................... 9

Acronyms ......................................................................................................................................... 9

Technology overview .......................................................................................................................... 11

EMC proven high performance solution for Oracle RAC on VNX ....................................................... 11

Architecture diagram ................................................................................................................. 11

Server layer .................................................................................................................................... 11

Server software ......................................................................................................................... 12

Network layer ................................................................................................................................. 14

Storage layer .................................................................................................................................. 14

Storage software ....................................................................................................................... 15

EMC FAST Suite .............................................................................................................................. 15

EMC FAST Cache ........................................................................................................................ 15

EMC FAST VP ............................................................................................................................. 16

Oracle Database layer .................................................................................................................... 17

Storage virtual provisioning design ........................................................................................... 17

Drive type....................................................................................................................................... 17

ASM disk group configuration for OLTP database ...................................................................... 17

ASM disk group configuration for DW database ......................................................................... 18

Enable HugePage ...................................................................................................................... 18

Performance tests .............................................................................................................................. 19

Overview ........................................................................................................................................ 19

Test objectives ............................................................................................................................... 19

OLTP database and workload profile ......................................................................................... 20

DW database and workload profile ............................................................................................ 21

SLOB OLTP workload tests ................................................................................................................. 22

Overview ........................................................................................................................................ 22



4

Test objectives ............................................................................................................................... 23

Query-only test scenarios and test results ...................................................................................... 23

Update-only test scenarios and test results .................................................................................... 27

Data warehouse query workload test.................................................................................................. 33

Overview ........................................................................................................................................ 33

Test objective ................................................................................................................................. 33

Test scenarios and test results ....................................................................................................... 33

Data Warehouse data loading test ...................................................................................................... 37

Overview ........................................................................................................................................ 37

Test objective ................................................................................................................................. 37

Test scenarios and test results ....................................................................................................... 37

Conclusion ......................................................................................................................................... 40

Summary ....................................................................................................................................... 40

Findings ......................................................................................................................................... 40

Core advantages ....................................................................................................................... 40

OLTP test results........................................................................................................................ 40

DW test results .......................................................................................................................... 41

References.......................................................................................................................................... 42

EMC documentation ....................................................................................................................... 42

Oracle documentation.................................................................................................................... 42

Appendix: Configuring XtremCache devices ....................................................................................... 43



Executive summary

Customers require an open, scalable, tiered, highly available and high performance infrastructure to run their critical Oracle database systems. Their IT organizations must strive for better performance and increased efficiency in their Oracle infrastructure and Oracle database and storage administration operations. To achieve these goals, they need to:

Reduce capital expenditures and operational expenditures by deploying an open, non-lock-in technology.

Consolidate many Oracle databases (Oracle database versions 10gR1 to 11gR2 and Oracle 12c) and database workloads, including online transaction processing (OLTP) and Data Warehouse, to maximize the efficiency of the data center infrastructure.

Deliver maximum performance while effectively utilizing the existing storage arrays and computing infrastructure for Oracle databases.

Maintain high performance levels and provide predictable throughput to deliver the quality of service required in these Oracle mixed-workload environments.

EMC proven high performance solution for Oracle Real Application Clusters (RAC) on EMC® VNX® is an open architecture that incorporates a choice of Intel servers with EMC server-side flash storage (EMC XtremSF™) and EMC VNX storage arrays.

The solution uses optimal servers to balance performance, scalability and Oracle license costs. The use of EMC Xtrem™ technologies, including XtremSF cards and EMC XtremCache™ software in the servers, provides distinct performance and operational advantages over equivalent systems that do not contain server-side flash technologies.

To accelerate an Oracle RAC environment, XtremCache:

Features an ultra-performance tier: XtremCache accelerates any application that benefits from low-latency, high bandwidth physical read I/O, meaning that:

The most frequently accessed data resides on database server flash.

The data is as close to the Oracle Database server CPU as any storage model will allow.

Cooperates with Oracle Clusterware: Oracle Clusterware is the final authority on all node membership information in an Oracle RAC deployment.

Offers optimized performance: VNX arrays do not need to handle read IOPS once the data is read into XtremCache, leaving more bandwidth for handling writes. EMC Fully Automated System Tiering (FAST™) enables automatic data placement as data goes from hot to cooler usage.

Improves performance with FAST Suite: Enabling the FAST Suite feature in VNX including FAST VP and FAST Cache improves the performance when the workload increases and the working set grows beyond the total capacity of the XtremCache.

Business challenge

Technology solution: open architecture components



6

Delivers high performance levels: This solution delivers high performance for mixed workloads in Oracle database environments. The Key results and recommendations section provides details.

Open standards benefits

This solution is based on open standards. Advantages resulting from the open standards commitment include the following:

A “flash everywhere” architecture, which produces the optimal use of flash from the Oracle database server to the EMC storage platform

Mitigation of I/O bottlenecks to deliver maximum Oracle read performance, enabling the VNX to serve more I/Os for other applications

Flexible adaptation to existing and future customer needs and open industry standards

Lower capital investment and operational expense without the need to use a specific vendor

Operational advantages

EMC open architecture not only supports different releases of Oracle Database software (10g, 11g, and 12c), but it also allows different databases to run concurrently.

Open architecture and flexible adaptation means that application modification is not required for database deployment for this solution. This lessens the potential impact to business operations and systemic data flow throughout the enterprise.

The purpose of the solution is to build an EMC high performance solution for Oracle RAC on a VNX infrastructure based on an open architecture and demonstrate the following capabilities of the infrastructure:

High performance and flexibility

Low operational costs

Reduced risk

This white paper validates the performance of the solution and provides guidelines to build similar solutions.

Solution overview



The EMC Proven high performance solution for Oracle RAC on VNX solution has several core advantages:

Uses EMC technology enablers in the reference architecture:

EMC VNX8000™ with FAST Suite enabled

XtremSF PCIe flash card

XtremCache caching software

This solution provides a foundation that can be scaled in a flexible, predictable, and nearly linear way. It uses additional server resources, including CPUs and memory, HBA ports, and front-end ports, to provide higher IOPS and throughput based on the configuration in this solution.

Delivers the highest performance for mixed Oracle database workload environments: The solution has demonstrated sustained metrics of over 3.8 million read IOPS with latency of less than 1 millisecond for OLTP, and Data Warehouse workloads with a sustained query throughput of 44 GB/s and a data load rate of 21 TB/hr. This impressive performance is achieved by utilizing optimal open components at the computer, network and storage layers. Details are listed in the next four tables.

Although XtremCache is a write-through cache, the Update workload can still be accelerated by EMC XtremCache. The data blocks that need to be read into buffer cache can be served from XtremCache when there are read hits during the test cycle. Meanwhile, the dirty blocks that have been flushed out of the database buffer cache pass through XtremCache and are directly written to the back-end VNX array. These results are shown in Table 1.

Table 1. IOPS test results when XtremCache is enabled and FAST Suite disabled

Workload type

Performance statistics

One node

Two nodes

Four nodes

Eight nodes

Read only workload

IOPS 454,157 892,430 1,858,382 3,883,371

Response time (ms) 0.5 0.80 0.75 0.68

Update transaction workload

Write 27,501 54,027 77,312 95,475

Read 26,608 52,375 74,759 92,107

Aggregate IOPS 54,109 106,402 152,071 187,582

DBWR latency(ms) 1.78 1.44 1.02 1.23

Read response time (ms)

0.38 0.21 0.18 0.17

Redo throughput (MB/s)

21 41 59 73

LGWR latency (ms) 0.75 0.88 0.97 1.10

Key results and recommendations



8

In the scenario illustrated by Table 2, we generated a larger workload to grow the working set beyond the total capacity of the XtremCache, causing more read misses in the XtremCache.

Table 2. IOPS test results with heavier workload when XtremCache and FAST Suite enabled

Workload on 8-node

FAST Cache

Read only workload UPDATE transaction workload

IOPS Latency (ms)

Read IOPS

Write IOPS

Aggregate IOPS

Redo size (MB/s)

Baseline No 2,396,780 1.02 48,140 49,442 97,582 38

FAST VP (20 SSD)

No 2,517,820 1.25 60,356 62,014 122,370 47

Yes 2,808,816 1.04 92,537 95,032 187,569 72

FAST VP (40 SSD)

No 2,784,765 0.97 78,049 80,160 158,209 61

Yes 2,905,339 0.97 114,237 117,334 231,571 89

FAST VP (80 SSD)

No 29,028,41 1.06 117,990 121,416 239,406 93

Yes 3,072,801 0.94 133,906 137,893 271,799 105

In the scenario illustrated in Table 3, we generated DW query workload with XtremCache first disabled and then enabled, to validate the performance improvement achieved through the use of XtremCache. We also disabled XtremCache and tested the data loading throughput.

Table 3. Query and data loading throughput test results when XtremCache is enabled or disabled, and FAST Suite is disabled

In the scenario illustrated in Table 4, we doubled the workload to grow the working set beyond the total capacity of the XtremCache, causing more read misses on the XtremCache.

Table 4. Query results with workload when XtremCache and FAST VP are enabled

Workload type

XtremCache 1 node 2 nodes 4 nodes 8 nodes

Query throughput (GB/s)

Disabled 4.58 7.74 9.13 11.22

Enabled 6.08 11.12 22.97 44.30

Data loading throughput (TB/Hour)

Disabled 2.51 4.96 10. 19 20.7

workload on 8-node Baseline FAST VP

20 SSD 40 SSD 80 SSD

Query throughput (GB/s) 19.90 24.86 29.68 37.80



Introduction

The purpose of this white paper is to describe an EMC high performance solution for Oracle RAC on VNX storage arrays based on an open architecture and demonstrate the following capabilities of the infrastructure:

High performance and flexibility

Low operational costs

Reduced risk

This paper validates the performance of the solution and provides guidelines for building similar solutions.

This white paper serves the following purposes:

Introduces the key solution technologies

Describes the solution architecture and design

Describes the solution test scenarios and presents the results of performance testing

Identifies the key business benefits of the solution

This white paper is intended for chief information officers (CIOs), data center directors, Oracle DBAs, storage administrators, system administrators, technical managers, and any others involved in evaluating, acquiring, managing, operating, or designing Oracle database environments.

Error! Reference source not found. lists acronyms used in this white paper.

Table 5. Acronyms

Term Definition

AWR Automatic Workload Repository

ASM Automatic Storage Management

DBWR Database writer process

DML Data Manipulation Language

FAST Fully Automatic Storage Tiering

LGWR Log writer process

PCIe Peripheral Component Interconnect Express

PGA Program Global Area

RAC Real Application Clusters

SATA Serial Advanced Technology Attachment

SGA System Global Area

Purpose

Scope

Audience

Acronyms



10

Term Definition

SLOB Silly Little Oracle Benchmark

SSD Solid State Disks

PL/SQL Procedure language/Structure Query Language



Technology overview

The EMC proven high performance solution for Oracle RAC on VNX includes the following layers of components:

Server—Cisco UCS C240 M3, Red Hat Enterprise Linux 6.3

Network—Cisco Director –MDS 9506

Storage—EMC storage and software:

EMC VNX8000

EMC XtremSF–Server PCIe flash card and its corresponding driver and firmware

EMC XtremCache–Cache software for server-side flash cache

Database—Eight-node Oracle 11gR2 RAC deployment

Architecture diagram

Figure 1 depicts the architecture of this solution. We deployed an eight-node SLOB (Silly Little Oracle Benchmark) RAC database for OLTP workload test first. After the SLOB test was finished, we replaced the SLOB database with an eight-node DW RAC database for a DW workload test on the same eight-node cluster.

Figure 1. Solution architecture

The server layer of the solution contains eight Cisco UCS C240 M3 servers that use a total of 128 cores, 2.56 TB RAM, and 11 TB XtremSF flash PCIe cards. The server layer is an enterprise-class rack server designed for performance and expandability. As part of the solution, it enables a high-performing, consolidated approach to an Oracle infrastructure, resulting in deployment flexibility without the need for application modification.

EMC proven high performance solution for Oracle RAC on VNX

Server layer



12

Features and benefits include the following:

Hardened protection for virtual and cloud environments, as part of the Intel Xeon processor E5-2600 product family

Fully integrated quad-port gigabit Ethernet

EMC XtremSF flash storage technology EMC XtremSF is an advanced flash storage technology deployed in a server and is designed to deliver unprecedented performance acceleration by reducing latency and increasing I/O throughput. It allows applications to access data in the most efficient manner possible. Residing on the server PCIe interconnect bus, XtremSF reduces application response time from milliseconds to microseconds by performing I/O operations at the server side.

Server software

Table 6 describes the various software components (versions tested are shown below) at the server layer in the solution.

Table 6. Server software

Server software Configuration Description

Red Hat Enterprise Linux 6.3 Operating system for database servers

Oracle Grid Infrastructure 11g Release 2

Enterprise Edition 11.2.0.3

Software provides Clusterware and ASM storage volume management

Oracle Database 11g Release 2

Enterprise Edition 11.2.0.3

Database software

EMC XtremCache software

2.0.1 Software for server-side flash cache

Red Hat Enterprise Linux Red Hat Enterprise Linux includes enhancements and new capabilities that provide rich functionality, especially the developer tools, virtualization features, security, scalability, file systems, and storage. Red Hat Enterprise Linux is a versatile platform that can be deployed on physical systems, as a guest on the major hypervisors, or in the cloud. It supports all leading hardware architectures with compatibility across releases.

Oracle Grid Infrastructure and Database 11g Release 2 Oracle Database 11gR2 is available in a variety of editions tailored to meet the business and IT needs of an organization. This solution utilizes Oracle Database 11gR2 Enterprise Edition (EE). Oracle Database 11g R2 EE delivers industry-leading performance, scalability, security, and reliability on a choice of clustered or single servers running Windows, Linux, or UNIX. It supports advanced features, either included or as extra-cost options. Oracle Grid Infrastructure provides system support for an Oracle database including clustering, volume management, file system and automatic restart capabilities.



EMC XtremCache technology EMC XtremCache and XtremeSF work together to reduce latency and accelerate throughput to dramatically improve application performance without compromising data consistency in the storage array.

In this solution, two 700 GB EMC XtremSF flash cards are used in each RAC node. One XtremCache cache device is created from one XtremSF card, which means that there are two 700 GB cache devices configured on each Oracle RAC node.

XtremCache accelerates reads and protects data by using a write-through cache policy to the networked storage to deliver persistent high availability, integrity, and disaster recovery.

XtremCache coupled with array-based EMC FAST software provides the most efficient and intelligent I/O path from the application to the underlying storage array. The result is a networked infrastructure that is dynamically optimized for performance, intelligence, and protection for both physical and virtual environments.

Benefits of XtremCache include the following:

Provides performance acceleration for read-intensive workloads.

Enables accelerated performance with the protection of the back-end, networked storage array.

Provides an intelligent path for the I/O and ensures that the right data is in the XtremCache of the servers at the right time.

Uses minimal CPU and memory resources from the server by offloading flash and wear-level management onto the XtremSF PCIe flash card.

Works in both physical and virtual environments.

Provides better data protection. Because XtremCache is a write-through cache, it does not compromise data consistency in the storage array, even if the cards fail in the middle of I/O processing.

Requires no warm up for database instances reboot. (A server reboot, however, requires a cache warm-up.)

Works for any kind of I/O; for example, any applications and any database platforms.

Is supported on various operating systems and server platforms.

Allows customers flexibility in choice of cache capacity on the cards.

Supports Oracle RAC database, even RAC databases “stretched” with EMC VPLEX.

As XtremCache is installed in more servers in the environment, more I/O processing is offloaded from the storage array to the XtremCache configured on the servers. This provides a highly scalable performance model in the storage environment. For more information, refer to:

Introduction to EMC XtremCache for Oracle Real Application Clusters listed in References



14

Introduction to EMC XtremCache for Oracle Real Application Clusters video listed in References.

EMC XtremCache configuration XtremCache supports Oracle RAC using a distributed cache coherency algorithm. XtremCache automatically recognizes the presence of Oracle RAC and switches operation to clustering mode.

All active Oracle RAC nodes must have XtremCache installed for the distributed cache feature to become available. EMC recommends using XtremCache with Oracle RAC to cache LUNs holding data files and TEMP files. EMC does not recommend caching redo logs, archives, or Clusterware files.

Appendix: Configuring XtremCache devices provides steps for configuring XtremCache devices.

The switch component level is made up of two Cisco MDS 9506 director-class SAN switches, configured to produce 108 GB/s active bandwidth. The Cisco MDS 9506 is designed for deployment in storage networks supporting virtualized data centers and enterprise clouds. It combines high performance and low total cost of ownership, a core architectural requirement at all levels of the VNX performance block. Nexus 5.2(8) software is used in the solution.

The Cisco MDS 9506 also offers these benefits:

Highly available scalability through a combination of nondisruptive software upgrades, stateful process failover, and full redundancy of all core components

Optimal platform for accelerated, intelligent storage applications such as EMC replication and backup, data migration, and storage media encryption

Virtual machine transparency and end-to-end visibility from the virtual machine down to the EMC storage, enabling scalable, mobile virtual machines

The EMC proven high performance solution for Oracle RAC on VNX includes the following storage components:

– One VNX8000

– EMC PowerPath®

Table 7 lists the VNX8000 components used in the solution.

Table 7. VNX8000 configuration

Component Quantity Configuration

Storage array 1 2 storage processors, each with 128GB Memory (cache size 46 GB)

24 ports on each storage processors with 8 GB FC

Network layer

Storage layer



Component Quantity Configuration

10K SAS drives 300G 220 192 for Data Pool,

57.6 TB Raw,25.7 TB usable(RAID10 configured)

16 for Redo Pool,

4.8 TB Raw,2.1 TB usable(RAID10 configured)

4 for Vault drive, 8 for hot spare

Flash drives SLC 100G 33 32 for Fast cache (RAID 1 configured)

3.2 TB Raw, 1.4 TB usable fast cache capacity

1 for hot spare

Flash drives MLC 200G 83 80 for Data Pool

16 TB Raw, 12.8 TB usable (RAID5 configured)

3 for Vault drive

Storage software

Table 8 lists the software used in the solution at storage layer.

Table 8. Storage software

Storage software Configuration Description

EMC VNX OE for block 05.33.000.5.034

VNX operating environment

EMC Unisphere® management software

1.3.1 VNX management software

EMC PowerPath 5.7 SP1 Multipathing and load balancing software

EMC FAST Cache

FAST Cache uses flash drives to add an extra layer of high-speed cache between DRAM cache and rotating disk drives, thereby creating a faster medium for storing frequently accessed data. FAST Cache is an extendable, read/write cache. It boosts application performance by ensuring that the most active data is served from high-performing flash drives and can reside on this faster medium for as long as necessary.

FAST Cache is most effective when application workloads exhibit high data activity skew. This is where a subset of data is responsible for most of the dataset activities. FAST Cache is more effective when the primary block reads and writes are small and fit within the 64 K FAST Cache track. The storage system is able to take advantage of such data skew by dynamically placing data according to its activity. For those applications whose datasets exhibit a high degree of skewing, FAST Cache can be assigned to concentrate a high percentage of application IOPS on flash capacity.

In this solution, we enabled FAST Cache using thirty-two 100 GB SSDs with RAID 1/0 protection.

EMC FAST Suite



16

EMC FAST VP

Like FAST Cache, FAST VP works best on datasets that exhibit a high degree of skew. FAST VP is very flexible and supports several tiered configurations, such as single tiered, multitiered, with or without a flash tier, and FAST Cache support. Adding a flash tier can locate hot data on flash storage in 256 MB slices.

Use FAST VP to aggressively reduce TCO and to increase performance. Service a target workload that requires a large number of performance tier drives with a mix of tiers and a much lower drive count. In some cases, you can achieve an almost two-thirds reduction in drive count.

You can use FAST VP in combination with FAST Cache. A common strategy is to use FAST VP to gain TCO benefits while using FAST Cache to boost overall system performance. There are other use cases in which it makes sense to use FAST VP for both purposes. This paper discusses considerations for an optimal deployment of these technologies.

FAST VP configuration involves two types of components—storage pools and tiering policies.

Storage pools are the framework that allows FAST VP to take full advantage of different drive types. A pool is similar to a RAID group, which is a physical collection of drives on which logical units (LUNs) are created. Pools can contain a few drives or hundreds of drives, whereas RAID groups are limited to 16 drives. Because of the large number of drives supported in a pool, pool-based provisioning spreads workloads over more resources and requires minimal planning and management efforts. Pools can be homogeneous or heterogeneous. Homogeneous pools have a single drive type (Flash, SAS, or NL-SAS), whereas heterogeneous pools have different drive types.

FAST VP is a completely automated feature that implements a set of user-defined tiering polices to ensure the best performance for various business needs. FAST VP tiering policies, determine how new allocations and ongoing

relocations should be applied within a storage pool. The relocations are done for each set of logically ordered blocks, called slices. FAST VP uses an algorithm to make data relocation decisions based on the activity level of each slice. It ranks the order of data relocation across all LUNs within each separate pool. The system uses this information in combination with the per LUN tiering policy to create a candidate list for data movement. A storage tier is made up of one or more virtual pools. To be a member of a tier, a virtual pool must contain only data devices that match the technology type and RAID protection type of the tier.

Note: Oracle redo logs have a very predictable sequential write workload, and this type of activity does not benefit significantly from up-tiering to SSD. Exclude these logs from any FAST policy, or pin them to a 10k rpm or 15k rpm drive tier so that FAST VP will not include them in its analysis.

When using FAST VP, there is no need to match the Logical Volume Manager (LVM) stripe depth with the Virtual Provisioning thin device extent.



Because Oracle typically accesses data either by random single-block read/write operations (usually 8 KB in size) or by sequentially reading large portions of data, FAST VP movements have no impact on the ASM AU size or on data access.

In Oracle 11g R2, Oracle ASM and Oracle Clusterware have been integrated into the Oracle Grid Infrastructure. Oracle Automatic Storage Management Cluster File System (ACFS) extends ASM functionality to act as a general-purpose cluster file system. In the solution, we use ASM to store the database files and Oracle ACFS to store the comma-separated values (CSV) files for the Data Warehouse data loading test.

Storage virtual provisioning design

EMC Virtual Provisioning™ automatically stripes data across all data devices in a virtual pool and balances the workload across storage devices.

Table 9 shows the RAID selections and number of spindles for each virtual pool. In this solution, Oracle data files and redo log files are located on thin devices using RAID 5 protection and all physical spindles for the best performance and capacity. The flash tier is used when FAST VP is enabled on VNX during the workload test.

Table 9. Virtual pool design on VNX8000

Virtual pool

Tier RAID protection

Drive type

Physical drive size

Number of active drives Item

Data_Pool1 Performance RAID 1/0 (4+4)

SAS 10K

300 GB 48 DATA, CRS

Extreme Performance

RAID 5 (4+1)

SSD 200 GB 5/10/20


SAS 10K

300 GB 48 DATA, CRS

Extreme Performance

RAID 5 (4+1)

SSD 200 GB 5/10/20


SAS 10K

300 GB 48 DATA, CRS

Extreme Performance

RAID 5 (4+1)

SSD 200 GB 5/10/20


SAS 10K

300 GB 48 DATA, CRS

Extreme Performance

RAID 5 (4+1)

SSD 200 GB 5/10/20

Log_Pool Performance RAID 1/0 (4+4)

SAS 10K

300 GB 16 REDO

ASM disk group configuration for OLTP database

Table 10 details the RAC database’s ASM disk group design. For the OLTP database, we used three ASM disk groups to store the relevant database files, including data

Oracle Database layer



18

files, control files, online redo log files, and temporary files. Default settings are used for ASM disk groups.

Table 10. ASM disk group design for OLTP databases

Item LUN size (GB) Number of LUNs ASM disk group name

CRS 10 2 +CRS

DATA 1024 24 +DATA

REDO 64 4 +REDO

ASM disk group configuration for DW database

Table 11 details the ASM disk group design for a data warehouse database. Four ASM disk groups store the relevant database files, including data files, online redo log files, control file, temporary file and CSV files (used for data loading).

Table 11. ASM disk group design for DW database

Item LUN size (GB)

Number of LUNs

AU_Size (MB)

Striping ASM disk group name

DATA 1024 24 16 Fine-grain +DATA

REDO 64 4 1 Fine-grain +REDO

CSV 512 2 1 Fine-grain +CSV

CRS 10 2 1 Coarse +CRS

Enable HugePage

HugePages is crucial for better Oracle database performance on Linux if you have a large amount of RAM and SGA. If the database SGA is large (more than 8 GB), you need to configure HugePages.

The advantages of enabling HugePages include:

Larger page size and fewer pages

Better overall memory performance

No swapping

No kswapd operations

The database initialization parameter, USE_LARGE_PAGES, was introduced in Oracle 11.2.0.2 and later versions to manage HugePages for the database. From version 11.2.0.3, setting USE_LARGE_PAGES to AUTO causes the oradism process to try to reconfigure the Linux kernel to increase the number of HugePages to match the database requirements.

See My Oracle Support: USE_LARGE_PAGES To Enable HugePages In 11.2 [ID 1392497.1] for more details.



Performance tests

The test environment of the solution consisted of two main workloads in order to characterize both OLTP and Data Warehouse (DW) systems. We1 created an eight-node Oracle 11gR2 RAC database for the OLTP workload testing. When we finished the OLTP test, we destroyed the OLTP database and created another DW database for DW workload testing on the same cluster environment.

We used the SLOB (Silly Little Oracle Benchmark) with XtremCache enabled to generate random physical read/write I/O, which is a typical I/O pattern seen in Oracle OLTP database environments. We also tested the performance improvement by enabling FAST Cache and FAST VP configured with different numbers of SSDs when running SLOB workload.

On the DW database, we used a DSS-like toolkit to generate the workload. There are two types of DW workload: data query and data loading. During the generation of the DW query workload, we enabled XtremCache and configured FAST VP with different numbers of SSDs.

The system (including the server side and the array side) I/O performance metrics (IOPS and latency) were gathered primarily from the Automatic Workload Repository (AWR) report. In addition, we gathered metrics for I/O throughput at the server/database and storage levels.

The objectives of our tests were to demonstrate the following:

Sustained server flash and storage array IOPS for Oracle OLTP database workload

Sustained query throughput in GB/s as well as data loading throughput in TB/hour for an Oracle data warehouse workload

During the test, the database was in no-archive-log mode to achieve maximum performance. The test scenarios are listed in Table 12.

Table 12. Test scenarios

Test scenarios XtremCache FAST Cache FAST VP Notes

OLTP with query only

Yes No No Node scalability test

Yes

No No Heavier workload running on 8-node Oracle database as the baseline

No Yes Workload running on 8-node Oracle database, enabling FAST VP configured with different number of SSD to validate the performance improvement

1 In this document, "we" refers to the EMC Solutions engineering team that validated the solution.1

Overview

Test objectives



20

Test scenarios XtremCache FAST Cache FAST VP Notes

Yes Yes Workload running on 8-node Oracle database, enabling FAST Cache and FAST VP configured with different number of SSD to validate the performance improvement

OLTP with update only

Yes No No Node scalability test

Yes No No Heavier workload running on 8-node as the baseline

No Yes Workload running on 8-node, enabling FAST VP configured with different numbers of SSDs to validate the performance improvement

Yes Yes Workload running on 8-node, enabling FAST Cache and FAST VP configured with different numbers of SSDs to validate the performance improvement

DW query No No No Node scalability test

Yes No No Node scalability test to validate the performance improvement when XtremCache enabled

Yes No No Heavier workload running on 8-node Oracle database as the baseline

No Yes Workload running on 8-node Oracle database, enabling FAST VP configured with different numbers of SSDs to validate the performance improvement

DW data loading

No No No Node scalability test

The OLTP and DW workload profiles used in these tests are shown below.

OLTP database and workload profile

Table 13 describes each OLTP database workload profile for the solution. We used the SLOB toolkit to generate an OLTP database and drive the OLTP-like workloads, including the query-only and update-only workloads required for the solution.



Table 13. Database workload profile for each OLTP database

Profile characteristic Details

Database type OLTP

Database size 16 TB

Oracle Database 11gR2 8-node RAC database on ASM

Instance configuration SGA size for each instance: 16 GB

Note: Considering that larger database cache size would buffer more data, we configured a very small buffer cache to generate a stable and high physical I/O workload.

Data block size 8 KB

Workload profile OLTP-like workload drive by SLOB

Network connectivity 8 Gb FC for SAN

10 GbE for IP

DW database and workload profile

Table 14 details the database and workload profile for the solution. We used a DSS-like toolkit to generate a data warehouse database and deliver the DSS workloads, including the query and data loading workloads required for the solution.

Table 14. Database and workload profile for DW database

Profile characteristic Details

Database type Data warehouse

Database size 24 TB

Oracle Database 11gR2 8-node RAC on ASM

Workload profile DSS-like workload

Data block size 16 KB

Data load source External flat files on Oracle ACFS used for external tables

Network connectivity 8 Gb FC for SAN

10 GbE for IP



22

SLOB OLTP workload tests

The solution characterizes the Oracle OLTP database performance on VNX arrays with EMC XtremSF cards installed on database nodes. An eight-node Oracle 11gR2 RAC database was deployed for the OLTP workload test. We used SLOB to generate the workload because it is the preferred workload generator for driving maximum physical random I/O from a database platform.

SLOB is a SQL-driven Oracle database I/O generator, instead of a synthetic I/O generator. SLOB uniquely drives massive physical I/O using minimal host CPU resources, and it specifically targets the Oracle I/O subsystem. SLOB performs all of its physical I/O buffering in the Oracle SGA; no physical I/O buffering in the Oracle PGA is performed. SLOB possesses the following characteristics:

Supports testing Oracle logical read (SGA buffer gets) scaling

Supports testing physical, random single-block reads (db file sequential read/db file parallel read)

Supports testing random single block writes (db file parallel write)

Supports testing extreme REDO logging I/O

Consists of simple PL/SQL

Is entirely free of all application contention

We used SLOB to generate OLTP-like workload on an eight-node Oracle RAC database to demonstrate sustained server flash and storage array IOPS. The database performance metrics including IOPS and I/O latency were gathered primarily from the AWR report, and the ratios of I/O served from the XtremCache. In addition, we gathered metrics for I/O throughput at the server/database and storage levels.

Notes: Benchmark results are highly dependent upon workload, specific application requirements, and system design and implementation. Relative system performance will vary as a result of these and other factors. Therefore, the solution test workloads should not be used as a substitute for a specific customer application benchmark when critical capacity planning and/or product evaluation decisions are contemplated. All performance data contained in this report was obtained in a rigorously controlled environment. Results obtained in other operating environments may vary significantly. EMC Corporation does not warrant or represent that a user can or will achieve similar performance expressed in transactions per minute.

Overview



The tests measured the following performance levels:

Physical I/O scalability along with the scaling of the number of concurrent SLOB zero-think-time sessions (simulated concurrent users) and the number of RAC nodes. Multiple concurrent sessions (reader sessions) executing similar query SQL statements were run to validate a read-only workload, and multiple concurrent sessions (writer sessions) executing similar UPDATE SQL statements were run to validate the physical read/write workload.

After the node scalability test, we validated the performance improvement with FAST Suite including FAST VP (with different number of 200 GB SSD) and FAST Cache (with 32 * 100GB SSD). During the FAST Suite test, we built up a new baseline with a heavier workload and a larger active dataset. This simulated a case in which the amount of hot data exceeded the total capacity of XtremSF cards in the RAC database hosts, so part of the hot data missed at XtremCache had to be accessed from the back-end storage array.

Baseline: XtremCache enabled with FAST Suite disabled.

FAST VP: XtremCache and FAST VP configured with twenty, forty, and eighty 200GB SSDs.

FAST Cache and FAST VP: XtremCache and FAST Cache configured with 32 100GB SSD, together with FAST VP configured with twenty, forty, and eighty 200GB SSD.

In query-only testing, we conducted two scenarios: node scalability test and FAST Suite test. When we measured the query-only workload performance results, we captured the statistics using Oracle Automatic Workload Repository (AWR) RAC reports. We observed the physical read IO requests value in the AWR report to assess read IOPS statistics. Query average response time was calculated from the db file parallel read and db file sequential read record in the Top Timed Events section of the AWR report, as shown in 0.

Figure 2. AWR RAC report snippet for read I/O response time calculation

We used the following logic to calculate the I/O latency:

For the db file sequential read event:

The total wait time is T1 which is 78,386.31 seconds, as shown in Error! Reference source not found..

The total number of waits is N1 which is 135,062,955, as shown in Error! Reference source not found..

For the db file parallel read event:

The total wait time is T2 which is 38,061.96 seconds, as shown in Error! Reference source not found..

Test objectives

Query-only test scenarios and test results



24

The total number of waits is N2 which is 36,359,301, as shown in Error! Reference source not found..

The average read response time is (T1+T2) / (N1+N2), which is (78,386.31+38,061.96) * 1,000/ (135,062,955 + 36,359,301), as shown in Error! Reference source not found.. The average response time is 0.68 ms.

Node scalability test - scenario

In the node scalability test, XtremCache was enabled, with FAST Cache and FAST VP disabled. Then we gradually increased the number of Oracle RAC database nodes and the number of concurrent users, with each user running similar OLTP queries simultaneously.

When we added a RAC node, we also added resources, including CPU power and XtremSF cards. With the addition of each new host, we tested the system again by running the similar SLOB workload. For this test, workloads were running simultaneously from all the RAC nodes added.

The test process included the following steps:

1. Run the query-only workload with 64 concurrent simulated users (zero-think-time sessions) on the first node of an eight-node RAC database using SLOB.

2. Add the second node into the system, then run the workload with 64 concurrent users on each node; that is, with a total of 128 concurrent users running simultaneously on two nodes.

3. Add two additional nodes into the system, then run the workload with 64 concurrent users on each node separately; that is, with a total of 256 concurrent users running simultaneously on four nodes.

4. Add four additional nodes into the system, then run the workload with 64 concurrent users on each node separately; that is, with a total of 512 concurrent users running simultaneously on eight nodes.

Node scalability test - results

Error! Reference source not found. depicts the physical read IOPS increase with the scaling of RAC database nodes.

Table 15. Scaling of nodes and resulting increases in IOPS

Error! Reference source not found. shows that the IOPS increase when the number of RAC nodes scales, while the average response time remains under one millisecond.

Metrics 1 node 2 nodes 4 nodes 8 nodes

IOPS 454,157 892,430 1,858,382 3,883,371

Average response time (ms) 0.50 0.80 0.75 0.68



Figure 3. Query only IOPS scaling along with RAC node scaling

As Error! Reference source not found. shows, we achieved a total of 3,883,371 read IOPS and an average latency of 0.68 milliseconds with the eight-node RAC database when running 64 concurrent sessions executing similar query SQL statements on each node.

The IOPS increased near linearly with each additional RAC instance added into the database. For example, the total IOPS of four database nodes reached 1,858,382. After we added another four database nodes for a total of eight database nodes, the IOPS almost doubled to 3,883,371.

The read-hit ratio for XtremCache was about 98 percent for each cache device during the test. Two percent of the I/Os were served from the storage array. The statistics can be monitored with the following command:

vfcmt display -cache_dev <device>

FAST Suite test - scenario

After the node scalability test, we enabled FAST Suite including FAST VP (with different number of 200 GB SSD) and FAST Cache (with 32 * 100GB SSD), while kept XtremCache enabled, and ran SLOB query-only workload to validate the performance improvement. During the test we increased the active data set to simulate that the amount of active data was larger than the total capacity of XtremSF cards configured on the database hosts. This caused part of the read I/O missed at XtremCache to be served from the back-end VNX array.

We recorded the performance statistics until the workload was stable. The test steps were:

1. Run 512 concurrent users on an eight-node RAC database with FAST VP and FAST Cache disabled to get the baseline.

2. Enable FAST VP configured with 20 SSD, and run 512 concurrent users on the eight-node RAC database.

3. Enable FAST Cache and keep FAST VP with 20 SSD enabled, then run 512 concurrent users on the eight-node RAC database.



26

4. Disable FAST Cache but enable FAST VP configured with 40 SSD, and run 512 concurrent users on the eight-node RAC database.

5. Enable FAST Cache and keep FAST VP with 40 SSD enabled, then run 512 concurrent users on the eight-node RAC database.

6. Disable FAST Cache but enable FAST VP configured with 80 SSD, and run 512 concurrent users on the eight-node RAC database.

7. Enable FAST Cache and keep FAST VP with 80 SSD, and run 512 concurrent users on the eight-node RAC database.

Note: During the FAST Suite test XtremCache was always enabled on the database hosts.

FAST Suite test – results

Table 16 shows that physical read IOPS increases as the number of SSDs increases in the flash tier of FAST VP and when the FAST Cache is enabled.

Table 16. Query-only IOPS and latency with different FAST Suite configurations

Figure 4 shows that IOPS increase as the number of SSDs increases in the flash tier of FAST VP and when the FAST Cache is enabled.

Figure 4. Query-only IOPS increased under different FAST Suite configurations

8-node workload FAST Cache Disabled FAST Cache Enabled

Read IOPS Read latency (ms)

Read IOPS Read latency (ms)

Baseline 2,396,780 1.02 N/A N/A

FAST VP 20 SSD 2,517,820 1.25 2,808,816 1.04

40 SSD 2,784,765 0.97 2,905,339 0.97

80 SSD 2,902,841 1.06 3,072,801 0.94



As Table 16 and Figure 4 show, we achieved a total of 2,784,765 read IOPS and an average latency of 0.97 milliseconds when running 512 concurrent users on an eight-node RAC database with 40 SSD drives configured in the flash tier of FAST VP—a 16.19% increase in IOPS compared with the baseline. When we further enabled FAST Cache, the physical read IOPS increased to 2,905,339, which achieved a 21.22% increase in IOPS, compared with the baseline. From the results, the IOPS increased nearly linearly as more SSD drives were added to the flash tier of FAST VP. And the physical read IOPS further increased as FAST Cache was enabled. Also the latency was stable at about one millisecond for all the tests.

In update-only testing, we conducted two scenarios: node scalability test and FAST Suite test. When we measured the update-only workload performance results, we captured the statistics using AWR reports. We read the physical write IO requests row in the AWR report for the average write IOPS statistics. Because the write workload is generated by UPDATE statements, as described previously, we also collected physical read IO requests from the AWR report for the average read IOPS statistics that were incurred by the write transaction.

As shown in Error! Reference source not found., we calculated the write average response time by dividing the Total Wait Time (s) by the Waits of the db file parallel write record in the Top Timed Events section of the AWR report. Meanwhile, we calculated LGWR latency by dividing the Total Wait Time (s) by the Waits of the log file parallel write record in the Top Timed Events section of the AWR report.

Taking the following AWR snippet for example, the total wait time of the db file parallel write wait event is 2,338.59 seconds, which is 2,338,590 ms, and the number of waits is 1,898,993; thus, the write average response time can be calculated as 2,338,590 / 1,898,993= 1.23 ms. The LGWR latency is 703.43*1,000/ 640,338 = 1.10 ms.

Figure 5. UPDATE only write average response time measurement from the RAC AWR report


During the node scalability test for update only test, XtremCache was enabled and FAST Suite was disabled. We gradually increased the number of RAC database nodes and ran multiple concurrent sessions with each session running similar UPDATE SQL against the RAC database.

We decreased the buffer cache for each RAC instance to push a consistent write I/O workload to the back-end storage. The write workload was driven by the UPDATE SQL statement. Generally, it incurs the following operations sequentially:

1. Reads the data blocks that needs to be updated into the buffer cache.

2. Updates the rows in the data blocks.

Update-only test scenarios and test results



28

3. Commits the updated rows and trigger LGWR flushing redo entries to online log files.

While the SQL UPDATE workload is running, the background DBWR process flushes the dirty blocks out of the buffer cache into the data files. Considering that we used a very small buffer cache, the data blocks were read into the buffer cache and written out of the buffer cache soon after the rows were updated. Thus, the execution of each UPDATE operation caused physical reads, which were accelerated by the EMC XtremCache (when the modified data was found in XtremCache) or the back-end VNX array (when the modified data was not found in XtremCache).

When the updated data blocks were written out of the buffer cache by the DBWR process, as being a write-through cache for XtremCache, the data blocks were written to the back-end VNX array. After the application got an acknowledgement from the back-end array, the application I/O request was complete. When database writes happen, the data is written to XtremCache in parallel (while the data is being sent to the VNX array). The VNX storage array delivers persistent high availability, integrity, and disaster recovery.


1. Run the update-only workload with 24 concurrent users on one RAC node.

2. Add one additional node into the RAC database, then run the workload with 24 concurrent users; that is, with 12 concurrent users running simultaneously on each node.

3. Add two additional nodes into the RAC database, then run the workload with 24 concurrent users; that is, with 6 concurrent users running simultaneously on each node.

4. Add four additional nodes into the RAC database, then run the workload with 24 concurrent users; that is, with 3 concurrent users running simultaneously on each node.

Node scalability test - results

Error! Reference source not found. and Error! Reference source not found. show that the peak disk array write IOPS increase as the number of RAC nodes scales.

Table 17. Scaling of RAC nodes and resulting increases in peak disk array IOPS

Item 1 node 2 nodes 4 nodes 8 nodes

Write 27,501 54,027 77,312 95,475

Read 26,608 52,375 74,759 92,107

Aggregate 54,109 106,402 152,071 187,582

DBWR latency(ms)

1.78 1.44 1.02 1.23

Read response time (ms)

0.38 0.21 0.18 0.17



Item 1 node 2 nodes 4 nodes 8 nodes

redo size (MB/s)

21 41 59 73

LGWR latency (ms)

0.75 0.88 0.97 1.10

Error! Reference source not found. shows the update only read/write IOPS for RAC node scaling while average response time remains under two milliseconds.

Figure 6. Update-only IOPS scaling along with RAC node scaling

During an UPDATE transaction, the back-end VNX storage array only needs to handle the write I/O activities, because the read I/O activities have been cached and accelerated by XtremCache. Therefore, the solution can scale to accommodate a very heavy transaction workload, as confirmed in testing.

As shown in Error! Reference source not found. and Error! Reference source not found., when running 24 concurrent sessions on an eight-node RAC database while executing similar Update SQL statements, we achieved 187,582 aggregate IOPS including 95,475 write IOPS and 92,107 read IOPS, which were part of the write transaction. The average latency of writes was 1.23 milliseconds. Because we used a very small buffer cache to generate a high physical write I/O workload, almost no data was cached in the database server.

The IOPS increased nearly linearly when additional RAC nodes were added into the RAC database. For example, the aggregate IOPS were 54,109 when running a SLOB update-only workload on one RAC node, and this increased to 106,402 when another RAC node was added to the database.

Redo size is also a key metric used to measure the transaction processing capability. As demonstrated through testing, the workload on one node generated 21



30

MB/second redo entries, and almost doubled to 41 MB/second with the workload running on two nodes. When we ran workload on four nodes the redo throughput is almost doubled again to 59 MB/second. The transaction capability scaled linearly with the node scaling.

FAST Suite test - scenario

After the scalability test, we enabled FAST Suite including FAST VP (with a different number of 200 GB SSDs) and FAST Cache (with 32 * 100GB SSDs), while keeping XtremCache enabled, and ran the SLOB update-only workload against the system. During the test, we increased the active data set that needed to be updated to simulate that part of the I/O was not served from the XtremCache. This forced part of the physical read I/O to the back-end VNX array. We recorded the performance statistics until the workload was stable. The test steps were:

1. Run 384 concurrent users on an eight-node RAC database with FAST Suite disabled to get the baseline.

2. Enable FAST VP with 20 SSD and run 384 concurrent users on an eight-node RAC database.

3. Enable FAST Cache and keep FAST VP with 20 SSD enabled, and ran 384 concurrent users on an eight-node RAC database.

4. Disable FAST Cache but enable FAST VP with 40 SSD, and run 384 concurrent users on an eight-node RAC database.

5. Enable FAST Cache and enable FAST VP with 40 SSD, and run 384 concurrent users on an eight-node RAC database.

6. Disable FAST Cache but enable FAST VP with 80 SSD, and run 384 concurrent users on an eight-node RAC database.

7. Enable FAST Cache and enable FAST VP with 80 SSD, and run 384 concurrent users on an eight-node RAC database.

Note: During the FAST Suite test, XtremCache was always enabled on the database hosts.



FAST Suite test - results

Table 18 shows that IOPS increase as the number of SSDs increases in the flash tier of FAST VP, and as FAST Cache is enabled.

Table 18. Update-only workload with XtremCache and FAST Suite enabled

Figure 7 shows that aggregate read/write IOPS increased as the number of SSDs increased in the flash tier of FAST VP and when FAST Cache was enabled.

Figure 7. Update-only aggregate read/write IOPS with different FAST Suite configurations As Table 18 and Figure 7 show, we achieved a total of 78,904 read IOPS and 80,160 write IOPS. That is 158,209 aggregate IOPS with 384 concurrent users on an eight-node RAC database with FAST VP configured with 40 SSD drives on the flash tier. This

8-node workload

FAST Cache

Write IOPS

DBWR latency (ms)

Read IOPS

Read latency (ms)

Aggregate IOPS

Redo size (MB/s)

Baseline No 49,442 3.90 48,140 1.21 97,582 38

FAST VP

20 SSD

No 62,014 4.00 60,356 1.40 122,370 47

Yes 95,032 2.49 92,537 0.52 187,569 72

40 SSD

No 80,160 2.40 78,049 0.63 158,209 61

Yes 117,334 1.58 114,237 0.31 231,571 89

80 SSD

No 121,416 2.90 117,990 0.93 239,406 93

Yes 137,893 2.58 133,906 1.52 271,799 105



32

represents a 62.13 percent aggregate IOPS increase, compared with the baseline; By further enabling the FAST Cache based on that FAST VP configuration, the aggregate IOPS increased to 231,571, achieving a 137.31 percent increase in the aggregate IOPS, compared with the baseline.



Data warehouse query workload test

The DSS-like toolkit provides an Oracle data warehouse test workload to test and validate the performance of typical Oracle data warehouse workloads on the EMC VNX8000 storage platform.

The schema in the kit has 12 tables including two fact tables—sales and returns. The remaining ten tables act as dimension tables. The two main fact tables are range-partitioned by date and sub-partitioned by hash on their join key. The database is an eight-node RAC 24TB database. Multiple concurrent users run a series of typical queries against the database to simulate the real-world-like reporting including joins and sorts. As the result, the workload generates the I/O pattern of direct path read, which simulates the large I/O request and bypassing the buffer cache. The throughput is measured during the test.

The objectives of the tests were:

Test physical I/O throughput scalability along with the scaling of the number of concurrent zero-think-time sessions (simulated concurrent users) and the number of RAC nodes when XtremCache on the database nodes were disabled. The concurrent users were generated by the DSS-like workload toolkit, with each user running similar queries to validate the throughput of the solution. Then we enabled XtremCache and ran a similar workload with the scaling of concurrent sessions and the number of RAC nodes, to validate the performance improvement.

After the node scalability test, we validated the performance improvement with FAST VP by configuring with a different number of SSDs. During the FAST VP test, we built up a new baseline with a heavier workload by running more concurrent users and accessing a larger active dataset for each user. This simulated that the amount of hot data exceeded the total capacity of XtremSF cards in the RAC database hosts; therefore, part of the hot data overflowed from XtremSF and had to be accessed from the back-end storage array.

Baseline: XtremCache enabled, FAST VP disabled.

FAST VP: XtremCache enabled and FAST VP configured with twenty, forty, and eighty 200GB SSDs.

In DW query testing, we conducted two scenarios: node scalability test and FAST VP test. When we measured the query workload performance results, we captured statistics using AWR reports. We read the physical read total bytes row in the AWR report for the query throughput (GB/s).


In the node scalability test, XtremCache was disabled on the database hosts and FAST VP disabled on the back-end storage. We gradually increased the number of Oracle RAC database nodes and the number of concurrent users, with each user running similar DW queries simultaneously. After that, more additional concurrent users were added into the test and then the performance scalability was measured.

Overview

Test objective

Test scenarios and test results



34


1. Run the DW query workload with 80 users on one RAC node using the scripts in the DSS-like toolkit.

2. Add one additional RAC node into the system, then run the workload with 80 users on each node; that is, with a total of 160 concurrent users running simultaneously.

3. Add two additional RAC nodes into the system, then run the workload with 80 users on each of the four nodes separately; that is, with a total of 320 concurrent users running simultaneously.

4. Add four additional RAC nodes into the system to get eight servers, then run the workload with 80 user on each node separately; that is, with a total of 640 concurrent users running simultaneously.

We enabled XtremCache on each database host and kept FAST VP disabled on the storage, and then ran the similar DW workload to validate the performance improvement by following the same test process as shown above.

Note: When we enabled XtremCache and added the cache cards, we increased the page size to 64 KB and increased the maximum I/O size to 128 KB to cache the large I/O size data in XtremSF cards. We also set db_file_multiblock_read_count to 8 to make one I/O size 128 KB (db_block_size is 16 KB) so the data could be cached by XtremCache.

Node scalability test – results

Error! Reference source not found. and Error! Reference source not found. depict how throughput increased as the RAC nodes scaled.

Table 19. Scaling of nodes and resulting increases in IOPS

Throughput (GB/s) 1 node 2 nodes 4 nodes 8 nodes

XtremCache disabled 4.58 7.74 9.13 11.22

XtremCache enabled 6.08 11.12 22.97 44.30



Figure 8. Query throughput scaling along with node scaling

As the blue line shows in Figure 10, when XtremCache was disabled and all the active data was accessed from the backend VNX array, the average throughput scaled nearly linearly as the nodes increased. After XtremCache was enabled, the throughput increased quickly as the RAC nodes scaled. For example, the throughput was 11.12 GB/s when running workload on two RAC nodes, and nearly doubled to 22.97 GB/s when four RAC nodes joined the system. Throughput nearly doubled again to 44.30 GB/s when the workload ran on eight nodes.

FAST VP test - scenario

After the node scalability test, we enabled FAST VP configured with different numbers of SSDs, keeping XtremCache enabled, and then ran the DW query-only workload on 8 nodes to validate the performance improvement. During the test we intentionally increased the active data set to simulate that the amount of active data set which was larger than the total capacity of XtremSF cards on the database hosts; therefore, part of the read I/O was missed at XtremCache and was served from the back-end VNX array.

We recorded the performance statistics until the workload was stable. The test steps were:

1. Run 1440 concurrent users on an eight-node RAC database with FAST VP disabled to get the baseline.

2. Enable FAST VP configured with 20 SSDs, and run 1440 concurrent users on the eight-node RAC database.





36

FAST VP test - results

Table 20 shows that the solution’s query throughput increased as the number of SSDs increased in the flash tier of FAST VP.

Table 20. Query throughput with different FAST Suite configurations

Figure 9 shows that query throughput increased as the number of SSDs increased in the flash tier of FAST VP.

Figure 9. Query throughput increased under different FAST VP configurations

As Table 20 and Figure 9 show, we achieved the query throughput with an average of 19.90 GB/s when running 1024 concurrent users on an eight-node RAC database with FAST VP disabled, which is the baseline.

When we enabled FAST VP with 20 SSDs, the total query throughput increased to 24.86 GB/s, which is about a 24.92 percent increase compared with the baseline. The throughput increased nearly linearly to 29.68 GB/s after adding 20 SSDs in FAST VP, a 39.34 percent increase as compared with the baseline. When the number of SSDs increased to 80, throughput reached 37.80 GB/s. According to these test results, the query throughput increased nearly linearly as more SSD drives were added to the flash tier of FAST VP.

8-node workload Query throughput ( GB/s )

Baseline 19.90

FAST VP 20 SSD 24.86

40 SSD 29.68

80 SSD 37.80



Data Warehouse data loading test

Modern enterprise data warehouses (EDWs) require large, frequent data loads periodically throughout the day. The 24x7 nature of the EDW no longer allows a long window of data loading for DBAs. Therefore, it is important to simulate the impact of data extract, transform, and load (ETL) processes on the performance of the database.

This test scenario demonstrates the ETL processes on the production database and records the performance data, especially the throughput (physical write total terabytes per hour), during the ETL load.

We used Oracle external tables as the source for the data loading, which use the ORACLE_LOADER access driver to load data from external tables to internal tables. The raw data comes from CSV flat files.

This test scenario shows the throughput scalability when data is loaded from external tables on the Oracle ACFS file system into the database.

The objective of the test was to show data loading throughput scalability of the VNX8000 when the sessions running the data loading were scaling out the RAC nodes. During the data loading test, FAST VP was disabled on the storage array, and XtremCache was disabled on each database node. Each session ran a similar ETL workload by loading CSV flat files into the database.

Test scenario

This solution scenario demonstrated performance scalability on the VNX8000 by loading data from external tables into the database. The test process included the following steps:

1. Run five concurrent users on one RAC node to load data from an external table. Each session loaded one CSV file with a size of 120 GB. The CSV file was located on the Oracle ACFS file system. The external table was created as follows:

create table sales_ext (

id integer,

…)

organization external(

type oracle_loader

default directory EXT_DIR

access parameters (fields terminated by "|")

location ('sales.csv'))

parallel reject limit unlimited;

The data was loaded from the external table as follows:

alter session enable parallel dml;

alter table sales parallel;

alter table sales_ext parallel;

insert into /*+ append */ sales select * from sales_ext;

Overview

Test objective

Test scenarios and test results



38

Note: The table “sales” had the same structure as the table “sales_ext.” The data was loaded directly with the “append” hint, and multiple parallel slaves were used for data loading.

The data in CSV files was cached in Oracle cache, and no physical read I/O was incurred during the loading process. All the I/O activities to the back-end storage were written requests.

2. Add one additional RAC node into the system, then run the same data loading workload as the first step; that is, with a total of 10 concurrent users running simultaneously.

3. Add two additional RAC nodes into the system, then run the same data loading workload with five users on each of the four servers; that is, with a total of 20 concurrent users running simultaneously.

4. Add four additional RAC nodes into the system, then run the workload with five users on each of the eight servers; that is, with a total of 40 concurrent users running simultaneously.

Test results

We read the throughput (TB/hour) from the physical write total bytes column on the system statistics section in the AWR report. Table 21 and Figure 10 show how throughput increased when the RAC nodes were scaled.

Table 21. Throughput (TB/hour) increasing with additional RAC nodes

1 node 2 nodes 4 nodes 8 nodes

2.51 4.96 10. 19 20.7

Figure 10. Data loading throughput scaling



The average throughput increased nearly linearly when the second RAC node was added into the workload, and the number of data loading users doubled. For example, running five users to load data from an external table against one RAC node achieved a throughput of 2.51 TB/hour. This increased to 4.96 TB/hour when 10 users were running in two RAC nodes with five users on each node. The throughput increased similarly when the additional RAC nodes were added into the environment.

Higher throughput can be achieved if additional nodes are added into the environment, including CPUs, HBA ports, front-end ports, and other resources.



40

Conclusion

Implementing the EMC proven high performance solution for Oracle RAC on VNX with innovative and proven products like EMC VNX systems enabled by FAST Suite gives customers choices within an open infrastructure. The solution integrates easily into existing data center operations while taking advantage of new technologies such as XtremCache, which now fully supports Oracle RAC. Customers can efficiently utilize resources through virtualization, database, and applications consolidation. Customers can also independently scale capacity and processing capability without the limitations imposed by a single-purpose appliance.

As the customer undergoes changes from any level, such as applications, databases, and non-database software, this stack is open to align with the shifting technical demands imposed by the business needs. This solution keeps the balance between OLTP and DW workloads while maintaining the protection and resiliency of the data. This adaptability to change and the ability to apply the technology where it is needed protects the capital investment, and remains fluid as the requirements change, without sacrificing any of the other data center operations.

Core advantages

The solution includes the following core advantages:

Delivers the highest performance for mixed Oracle workload environments. EMC proven high performance solution for Oracle RAC on VNX has demonstrated

Sustained metrics over 3.8 million IOPS

Latency of less than 1 millisecond

Executing multi-workload OLTP and Data Warehouse workloads

Sustained query throughput of 44 GB/s (XtremCache enabled)

Data load rate of 21 TB/hr (XtremCache and FAST Suite disabled)

This impressive performance is achieved by utilizing open best-in-class components at the computer, network, and storage layers.

Uses the following EMC technology enablers in the reference architecture:

EMC VNX8000 with FAST Suite enabled

XtremSF

XtremCache

Provides full support for EMC Performance Boost, high availability, continuous availability, and replication technologies

OLTP test results

OLTP test results demonstrated that the solution:

Increases IOPS for OLTP workloads without FAST Suite. During RAC nodes scaling:

Read IOPS increase linearly as RAC nodes scale up from one to eight.

Summary

Findings



Aggregate read/write IOPS for UPDATE transaction workloads increase linearly as RAC nodes scale up from one to eight.

With FAST Suite enabled and heavier workload running against the system, IOPS increase for OLTP workloads:

The read IOPS increased when FAST VP with 20 SSDs was enabled, and increased even more when FAST Cache was also enabled. When FAST VP with 80 SSDs was enabled with FAST Cache, read IOPS increased to 3,072,801.

The aggregate read/write IOPS for a write workload increased when FAST VP with 20 SSDs was enabled, and increased even more when FAST Cache was also enabled. When FAST VP with 80 SSDs was enabled with FAST Cache, aggregate read/write IOPS increased to 271,799.

DW test results

From the DW test results, average query and data loading throughput for DW workload increased during RAC nodes scaling as follows:

The average query throughput increased when additional RAC nodes were added with XtremCache disabled and enabled on RAC database hosts while FAST VP disabled on VNX.

With FAST VP enabled and XtremCache enabled, and heavier workload running against the system, the throughput for DW query workloads increased using FAST VP with 20 SSDs. It increased even more using FAST VP with 80 SSDs enabled.

The average throughput of the data loading increased linearly along with the addition of the RAC nodes, with XtremCache disabled on RAC database hosts and the FAST suite disabled on VNX. The throughput increased when running four RAC nodes, and increased even more with eight RAC nodes.

This solution is offered as a foundation that can be scaled in a flexible, predictable, and nearly linear way, by adding additional node resources including CPUs and memory, HBA ports, and front-end ports to provide higher IOPS and throughput based on the configuration described in this white paper.



42

References

The following documents provide additional and relevant information. Access to these documents depends on your login credentials. If you do not have access to a document, contact your EMC representative.

EMC Infrastructure for High Performance Microsoft and Oracle Database Systems

Introduction to EMC XtremCache

EMC XtremCache Data Sheet

In addition, XtremCache documentation is available at EMC Online Support: https://support.emc.com/products/25208_XtremCache-Cache/Documentation/

A video entitled Introduction to EMC XtremCache for Oracle Real Application Clusters is available at: https://community.emc.com/videos/6740

For additional information, see the following documents:

Oracle Grid Infrastructure Installation Guide 11g Release 2(11.2) for Linux

Oracle Database Installation Guide 11g Release 2 (11.2) for Linux.

My Oracle Support: USE_LARGE_PAGES To Enable HugePages In 11.2 [ID 1392497.1]

EMC documentation

Oracle documentation

https://support.emc.com/products/25208_XtremSW-Cache/Documentation/

https://community.emc.com/videos/6740



Appendix: Configuring XtremCache devices

Use the following steps to configure XtremCache devices:

1. Create two cache devices on two 700 GB EMC XtremSF flash cards using these commands:

vfcmt add -cache_device /dev/rssda

vfcmt add -cache_device /dev/rssdb

2. Because source devices are not automatically assigned to cache devices, after cache devices are created, add all of the database source LUNs (data LUNs only) to one of cache devices using this command:

vfcmt add -source_device /dev/emcpowerXX

3. Because there are two cache devices, after all source LUNs are added to one cache device, use this command to move half of the LUNs to another cache device to ensure that the workload on each cache device is balanced:

vfcmt migrate -source_dev /dev/emcpowerXX -existing_cache_dev /dev/rssda -new_cache_dev /dev/rssdb

4. After all source devices are added, use the following command to validate the status of XtremCache:

vfcmt display -all

emc proven high performance solution for oracle rac on vnx · pdf filewhite paper emc...

Documents