emc backup and recovery for oracle 11g oltp backup and recovery for oracle 11g oltp. enabled by emc...

EMC Backup and Recovery for Oracle 11g OLTP

Enabled by EMC CLARiiON,

EMC Data Domain, EMC NetWorker, and Oracle Recovery Manager

using Fibre Channel

Proven Solution Guide

Copyright © 2010 EMC Corporation. All rights reserved. Published June, 2010 EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. Benchmark results are highly dependent upon workload, specific application requirements, and system design and implementation. Relative system performance will vary as a result of these and other factors. Therefore, this workload should not be used as a substitute for a specific customer application benchmark when critical capacity planning and/or product evaluation decisions are contemplated. All performance data contained in this report was obtained in a rigorously controlled environment. Results obtained in other operating environments may vary significantly. EMC Corporation does not warrant or represent that a user can or will achieve similar performance expressed in transactions per minute. No warranty of system performance or price/performance is expressed or implied in this document. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All other trademarks used herein are the property of their respective owners. Part number: H6834.1

Table of Contents

EMC Backup and Recovery for Oracle 11g OLTP Enabled by EMC CLARiiON, EMC Data Domain, EMC NetWorker, and Oracle Recovery Manager using Fibre Channel Proven Solution Guide

3

Table of Contents Chapter 1: About this Document ............................................................................................... 4

Overview ............................................................................................................................ 4 Audience and purpose ....................................................................................................... 5 Business challenge ............................................................................................................ 6 Technology solution ........................................................................................................... 6 Objectives .......................................................................................................................... 7 Reference Architecture ...................................................................................................... 8 Validated environment profile ............................................................................................. 9 Hardware and software resources ..................................................................................... 9 Prerequisites and supporting documentation ................................................................... 11 Terminology ..................................................................................................................... 12

Chapter 2: Use Case Components ......................................................................................... 13 Chapter 3: Storage Design ...................................................................................................... 17

Overview .......................................................................................................................... 17 CLARiiON storage design and configuration ................................................................... 18 Data Domain .................................................................................................................... 23 SAN topology ................................................................................................................... 25

Chapter 4: Oracle Database Design ....................................................................................... 28 Overview .......................................................................................................................... 28

Chapter 5: Installation and Configuration ................................................................................ 33 Overview .......................................................................................................................... 33 Navisphere ....................................................................................................................... 34 PowerPath ........................................................................................................................ 37 Install Oracle clusterware ................................................................................................. 41 Data Domain .................................................................................................................... 47 NetWorker ........................................................................................................................ 51 Multiplexing ...................................................................................................................... 57

Chapter 6: Testing and Validation ........................................................................................... 59 Overview .......................................................................................................................... 59 Section A: Test results summary and resulting recommendations .................................. 60

Chapter 7: Conclusion ............................................................................................................. 76 Overview .......................................................................................................................... 76

Appendix A: Scripts ................................................................................................................. 78

Chapter 1: About this Document

Overview

Introduction to about this document

This Proven Solution Guide summarizes a series of best practices that were discovered, validated, or otherwise encountered during the validation of the EMC Data Domain® Backup and Recovery for an Oracle 11g OLTP environment enabled by EMC® CLARiiON®, EMC Data Domain, EMC NetWorker®, and Oracle Recovery Manager. EMC's commitment to consistently maintain and improve quality is led by the Total Customer Experience (TCE) program, which is driven by Six Sigma methodologies. As a result, EMC has built Customer Integration Labs in its Global Solutions Centers to reflect real-world deployments in which TCE use cases are developed and executed. These use cases provide EMC with an insight into the challenges currently facing its customers.

Use case definition

A use case reflects a defined set of tests that validates the reference architecture for a customer environment. This validated architecture can then be used as a reference point for a Proven Solution.

Contents The content of this chapter includes the following topics.

Topic See Page

Audience and purpose 5

Business challenge 6

Technology solution 6

Objectives 7

Reference Architecture 8

Validated environment profile 9

Hardware and software resources 9

Prerequisites and supporting documentation 11

Terminology 12


Audience and purpose

Audience The intended audience for the Proven Solution Guide is

• internal EMC personnel • EMC partners, and • customers

Purpose The purpose of this proven solution for deduplication is to define a working

infrastructure for an Oracle RAC environment with an Oracle 1 TB OLTP database on CLARiiON storage infrastructure using a Data Domain appliance to show the following: • Benefits of deduplication in an Oracle OLTP environment. • Backup and restore times for a 1 TB OLTP database, using EMC NetWorker,

Recovery Manager (RMAN) backups, and EMC Data Domain with inline deduplication.

• Utilization of SnapView clones to facilitate backup and restore, with minimum impact to production and ensuring business continuity during backup and recovery.

This document provides a specification for the customer environment (storage configurations, design, sizing, software and hardware, and so on) that constitutes an enterprise Oracle 11g RAC backup and recovery solution in an Oracle OLTP environment, deployed on the CX4-960. In addition, this use case provides information on: • Building an enterprise Oracle 11g RAC environment on an EMC CLARiiON

CX4-960. • Identifying the steps required to design and implement an enterprise-level Oracle

11g RAC solution around EMC software and hardware. • Deployment of a Data Domain DD880 appliance as a Virtual Tape Library.


5


Business challenge

Overview Today's IT is being challenged by the business to solve the following pain points

around the backup and recovery of the business’ critical data:

• Protect the business information as an asset of the business’ defined recovery point objective (RPO - amount of data to recover) and recovery time objective (RTO - time to recover)

• Efficient use of both infrastructure and people to support the business • Difficulties around the backing up of large enterprise-critical systems – multi-

terabyte systems Exponential data growth, changing regulatory requirements, and increasingly complex IT infrastructure all have a major impact on data managers’ data protection schemes. RTO continue to decrease while the precision of the RPO increases. In other words, IT managers must be able to recover from a given failure quicker than ever and with less data loss. It is not uncommon for organizations to routinely exceed their backup window or even have a backup window that takes up most of the day. Such long backup operations leave little margin for error and any disruption can place some of the data at risk of loss. Such operations also mean that a guaranteed RPO cannot be met. Because of the demands generated by data growth and the RTO/RPO requirements in Oracle database environments, it is critical that robust, reliable, and tested backup and recovery processes are in place. Backup and recovery of Oracle databases are a vital part of IT data protection strategies. To meet these backup and recovery challenges enterprises need proven solution architectures that encompass the best of what EMC and Oracle can offer.

Technology solution

Overview This solution describes a backup and recovery environment of an Oracle 11g OLTP

database. The database was deployed on a CLARiiON CX4-960 using Oracle ASM in a two-node RAC configuration. Backup and recovery was implemented using RMAN, EMC NetWorker, and an EMC Data Domain DD880 appliance. The backup process was offloaded to a NetWorker proxy node using Navisphere® SnapView™ clones to minimize the impact to the production environment. The DD880 appliance enabled an 84 percent saving on the storage capacity required by the backup process. Reference the “Storage saving % after 5 weeks of backup cycle” chart in the “Testing and Validation” section.


6


The following table describes the key components and their configuration details

within this environment.

Component Configuration Software

CLARiiON CX4-960

Four BE 4 Gb FC ports, eight FE 4 Gb FC ports per storage processor, nine DAEs with five 146 GB, and 130 x 300 GB disk drives

FLARE® 04.29.000.5.003

Data Domain DD880

Four 4 Gb FC ports - VTL SAN connectivity, two SAS HBAs - disk connectivity, three EOS disk shelves with 48 disks

DDOS 4.7.1.3

Oracle 11g OLTP database system

1 TB Oracle 11g OLTP database using ASM on a two-node RAC

Oracle 11g Database/Cluster/ ASM versions 11.1.0.7

NetWorker NetWorker Management Console, storage nodes, clients

NetWorker 7.6 NMO 5.0

Objectives

This document provides guidelines on how to configure and set up an Oracle 11g

OLTP database with Data Domain deduplication storage systems. The solution demonstrates the benefits of deduplication in an Oracle backup environment. The backup schedule used level 0 (full backups), level 1 (incremental differential backups), and Oracle Block Change Tracking (BCT). This document is not intended to be a comprehensive guide to every aspect of an Enterprise Oracle 11g solution. The information that is to be obtained and reported from this document is described in the following list:

• Installation and build of the infrastructure • Configure and test CLARiiON storage • Configure the Oracle 11g environment • Configure a Data Domain virtual tape library • Configure NetWorker


7


Reference Architecture

Corresponding Reference Architecture

This use case has a corresponding Reference Architecture document that is available on Powerlink® and EMC.com. Refer to EMC Backup and Recovery for Oracle 11g OLTP Enabled by EMC CLARiiON, EMC Data Domain, EMC NetWorker, and Oracle Recovery Manager using Fibre Channel Reference Architecture for details. If you do not have access to the following content, contact your EMC representative.

Reference Architecture diagram

The following diagram depicts the overall physical architecture of the use case.


8


Validated environment profile

Profile characteristics

The use case was validated with the following environment profile.

Profile characteristic Value

Database characteristic OLTP Benchmark profile Swingbench OrderEntry - TPC-C-like

benchmark Response time < 10 ms Read / Write ratio 70 / 30 Database scale A Swingbench load that keeps the system

running within agreed performance limits Size of databases 1 TB Number of databases 1 Array drives: size and speed 300 GB 15k rpm

Hardware and software resources

Hardware The hardware used to validate the use case is listed below.

Equipment Quantity Configuration

Storage array 1 CLARiiON CX4-960: • Nine DAEs • Five 146 GB FC drives • 130 x 300 GB FC drives

SAN 2 4 Gb capable FC switch, 64 port

Oracle database node 2 Four Quad-Core Xeon E7330 processors, 2.4 GHz, 6 MB, 1066 FSB, 32 GB RAM. Two 73 GB 10k internal disks. Four 4 Gb Emulex LP11002E HBAs.

Proxy node (mount host) 1 Four Quad-Core Xeon E7330 processors, 2.4 GHz, 6 MB, 1066 FSB, 32 GB RAM. Two 73 GB 10k internal disks. Four 4 Gb Emulex LP11002E HBAs.

Navisphere management server NetWorker server

1 Two Quad-Core processors, 1.86 GHz, 16 GB RAM. Two 4 Gb Emulex LP11002E HBAs.

Network 2 Cisco Catalyst 3750G


9


Software The software used to validate the use case is listed below.

Software Version Comment RedHat Linux 5.3 OS for database nodes Microsoft Windows 2003 SP2 OS for Navisphere

Management Server Oracle Database/Cluster/ASM 11g Release 1 (11.1.0.7.0) Database/cluster

software/volume management Oracle ASMLib 2.0 Support library for ASM Swingbench 2.3 OLTP database benchmark Orion 10.2 Orion is the Oracle I/O

Numbers Calibration Tool designed to simulate Oracle I/O workloads

FLARE operating environment 04.29.000.5.003 Navisphere Management Suite Includes:

• Access Logix™ • Navisphere Agent

Navisphere Analyzer 6.29.0.6.34 SnapView 6.29.0.6.34.1 PowerPath® 5.3 Multipathing software DDOS 4.7.1.3 Data Domain OS NetWorker 7.6 Back up and recover suite NetWorker Module for Oracle 5.0 NetWorker Oracle integration Cisco IOS 12.2 Network Fabric OS 6.2.0g SAN


10


Prerequisites and supporting documentation

Technology It is assumed the reader has a general knowledge of:

• EMC CLARiiON • EMC Data Domain • EMC NetWorker • Oracle Database • Red Hat Linux

Supporting documents

The following documents, located on Powerlink.com, provide additional, relevant information. Access to these documents is based on your login credentials. If you do not have access to the following content, contact your EMC representative.

• EMC CLARiiON CX4-960 Setup Guide • EMC Navisphere Manager Help (html) • EMC PowerPath Product Guide • EMC CLARiiON Database Storage Solution: Oracle 10g/11g with CLARiiON

Storage Replication Consistency • EMC CLARiiON Server Support Products for Linux Servers Installation

Guide • EMC Support Matrix • Data Domain OS Initial Configuration Guide • Data Domain OS Administration Guide • NetWorker Installation Guide • NetWorker Administration Guide • NetWorker Module for Oracle Administration Guide • NetWorker Module for Oracle Installation Guide

Third-party documents

The following documents are available on third-party websites.

• Oracle Database Installation Guide 11g Release 1 (11.1) for Linux • Oracle Database Backup and Recovery User's Guide • Orion: Oracle I/O Numbers Calibration Tool


11


Terminology

Terms and definitions

This section defines the terms used in this document.

Term Definition

ASM Automatic Storage Management

BCT Block Change Tracking

BE Back End

DAE Disk Array Enclosure

DBCA Database Configuration Assistant

FE Front End

NMO NetWorker Module for Oracle

RAC Real Application Cluster

RPO Recovery Point Objective

RTO Recovery Time Objective

SAS Serial Attached SCSI

SISL Streams Informed Segment Layout

VTL Virtual Tape Library


12

Chapter 2: Use Case Components


Introduction This section briefly describes the key solutions components. For details on all of the

components that make up the solution architecture, refer to the hardware and software sections.

CLARiiON CX4-960

The EMC CLARiiON CX4 model 960 enables you to handle the most data-intensive workloads and large consolidation projects. CLARiiON CX4-960 delivers innovative technologies such as Flash drives, Virtual Provisioning™, a 64-bit operating system, and multi-core processors. The CX4 new flexible I/O module design, UltraFlex™ technology, delivers an easily customizable storage system. Additional connection ports can be added to expand connection paths from servers to the CLARiiON. The CX4-960 can be populated with up to six I/O modules per storage processor. The CX4-960 also uses a new generation of storage processor CPUs, memory, and PCI bus architecture.

EMC Data Domain DD880

EMC Data Domain deduplication storage systems dramatically reduce the amount of disk storage needed to retain and protect enterprise data. By identifying redundant data as it is being stored, Data Domain systems provide a storage footprint that is five to 30 times smaller, on average, than the original dataset. Backup data can then be efficiently replicated and retrieved over existing networks for streamlined disaster recovery and consolidated tape operations. The Data Domain DD880 is the industry’s highest throughput, most cost-effective, and scalable deduplication storage solution for disk backup and network-based disaster recovery (DR). The high-throughput inline deduplication data rate of the DD880 is enabled by the Data Domain Streams Informed Segment Layout (SISL) scaling architecture. The level of throughput is achieved by a CPU-centric approach to deduplication, which minimizes the number of disk spindles required.

Navisphere Management Suite

The Navisphere Management Suite of integrated software tools allows you to manage, discover, monitor, and configure EMC CLARiiON systems as well as control all platform replication applications from an easy-to-use, secure, web-based management console. Navisphere Management Suite enables you to access and manage all CLARiiON advanced software functionality—including EMC Navisphere Quality of Service Manager, Navisphere Analyzer, SnapView, SAN Copy™, and MirrorView™. When used with other EMC storage management software, you gain storage resource, SAN, and replication management functionality—for greater efficiency and control over CLARiiON storage infrastructure.


13


EMC PowerPath

EMC PowerPath® is a server-resident software that enhances performance and application availability. PowerPath works with the storage system to intelligently manage I/O paths, and supports multiple paths to a logical device. In this solution, PowerPath manages I/O paths and provides: • Automatic failover in the event of a hardware failure. PowerPath automatically

detects path failure and redirects I/O to another path. • Dynamic multipath load balancing. PowerPath intelligently distributes I/O requests

to a logical device across all available paths, thus improving I/O performance and reducing management time and downtime by eliminating the need to configure paths statically across logical devices.

PowerPath enables customers to standardize on a single multipathing solution across their entire environment.

EMC NetWorker EMC NetWorker software comprises a high-capacity, easy-to-use data storage

management solution that protects and helps to manage data across an entire network. NetWorker simplifies the storage management process and reduces the administrative burden by automating and centralizing data storage operations.

NetWorker Module for Oracle NMO provides the capability to integrate database and file system backups, to relieve the burden of backup from the database administrator while allowing the administrator to retain control of the restore process. NMO includes the following features:

• Automatic database storage management through automated scheduling, autochanger support, electronic tape labeling, and tracking.

• Support for backup to a centralized backup server. • High performance through support for multiple, concurrent high-speed

backup devices. Together with the NetWorker server, NMO augments the backup and recovery system provided by the Oracle server and provides a storage management solution that addresses the need for cross-platform support of enterprise applications.

EMC SnapView SnapView is a storage-system-based software application that allows you to create a

copy of a LUN by using either clones or snapshots. A clone is an actual copy of a LUN and takes time to create, depending on the size of the source LUN. A snapshot is a virtual point-in-time copy of a LUN and takes only seconds to create. SnapView has the following important benefits: • It allows full access to a point-in-time copy of your production data with modest

impact on performance and without modifying the actual production data. • For decision support or revision testing, it provides a coherent, readable and

writable copy of real production data. • For backup, it practically eliminates the time that production data spends offline or

in hot backup mode, and it offloads the backup overhead from the production server to another server.


14


• It provides a consistent replica across a set of LUNs. You can do this by performing a consistent fracture, which is a fracture of more than one clone at the same time, or a fracture that you create when starting a session in consistent mode.

• It provides instantaneous data recovery if the source LUN becomes corrupt. You can perform a recovery operation on a clone by initiating a reverse synchronization and on a snapshot session by initiating a rollback operation.

Oracle Database 11g Enterprise Edition

Oracle Database 11g Enterprise Edition delivers industry-leading performance, scalability, security, and reliability on a choice of clustered or single servers running Windows, Linux, and UNIX. It provides comprehensive features easily managing the most demanding transaction processing, business intelligence, and content management applications. Oracle Database 11g Enterprise Edition comes with a wide range of options to help grow your business and meet users' performance, security, and availability service level expectations.

Oracle Database 11g RAC Oracle Real Application Clusters (RAC) is an optional feature of Oracle Database 11g Enterprise Edition. Oracle RAC supports the transparent deployment of a single database across a cluster of servers, providing fault tolerance from hardware failures or planned outages. If a node in the cluster fails, Oracle continues running on the remaining nodes. If more processing power is needed, new nodes can be added to the cluster providing horizontal scaling. Oracle RAC supports mainstream business applications of all kinds. This includes Online Transaction Processing (OLTP) and Decision Support System (DSS).

Oracle ASM Oracle Automatic Storage Management (ASM) is an integrated database filesystem and disk manager. It can reduce the complexity of managing the storage for the database. The ASM filesystem and volume management capabilities are built into the Oracle database kernel. In addition to providing performance and reliability benefits, ASM can also increase database availability because disks can be added or removed without shutting down the database. ASM automatically rebalances the database files across an ASM diskgroup after disks have been added or removed.

Oracle ASMLib ASMLib is a support library for the ASM feature of Oracle Database. It is an add-on module that simplifies the management and discovery of ASM disks. The ASMLib provides an alternative to the standard operating system interface for ASM to identify and access block devices. ASMLib is composed of the actual ASMLib library, which is loaded by Oracle at Oracle startup, and a kernel driver that is loaded into the OS kernel at system boot. The kernel driver is specific to the OS kernel.

ASMCMD The asmcmd utility can be used by Oracle database administrators to query and manage their ASM systems. ASM-related information can be retrieved easily for


15


diagnosing and debugging purposes.

Oracle Recovery Manager Oracle Recovery Manager (RMAN) is a command-line and Enterprise Manager-based tool for backing up and recovering an Oracle database. It provides block-level corruption detection during backup and restore. RMAN optimizes performance and space consumption during backup with file multiplexing and backup set compression, and integrates with Oracle Secure Backup and third-party media management products for tape backup.

Oracle Block Change Tracking This database option causes Oracle to track datafile blocks affected by each database update. The tracking information is stored in a block change tracking file. When block change tracking is enabled, RMAN uses the record of changed blocks from the change tracking file to improve incremental backup performance by only reading those blocks known to have changed, instead of reading datafiles in their entirety.

Swingbench Swingbench is a publicly available load generator (and benchmark tool) designed to

stress test Oracle databases. Swingbench consists of a load generator, a coordinator, and a cluster overview. The software enables a load to be generated and the transactions/response times to be charted. Swingbench is provided with four benchmarks: • OrderEntry – TPC-C-like workload. • Calling Circle – Telco-based self-service workload. • Stress Test – Performs simple insert/update/delete/select operations. • DSS – A DSS workload, based on the Oracle Sales History schema. The Swingbench workload used in this testing was Order Entry. The Order Entry (PL/SQL) workload models the classic order entry stress test. It has a profile similar to the TPC-C benchmark. It models an online order entry system, with users being required to log in before purchasing goods.


16

Chapter 3: Storage Design


Overview

Introduction to storage design

The environment consists of a two-node Oracle 11g RAC cluster that accesses a single production database. Each cluster node resides on its own server, which is a typical Oracle RAC configuration. The two RAC nodes communicate with each other through a dedicated private network that includes a Cisco Catalyst 3750G-48TS switch. This cluster interconnection synchronizes cache across various database instances between user requests. A Fibre Channel SAN is provided by two Brocade 4900 switches. EMC PowerPath is used in this solution and works with the storage system to intelligently manage I/O paths. In this solution, for each server, PowerPath manages four active I/O paths to each device and four passive I/O paths to each device.

Contents This chapter contains the following topics:

Topic See Page

CLARiiON storage design and configuration 18

Data Domain 23

SAN topology 25


17


CLARiiON storage design and configuration

Design CLARiiON CX4-960 uses UltraFlex technology to provide array connectivity. This

approach is extremely flexible and allows each CX4 to be tailored to each user’s specific needs. In the CX4 deployed for this use case each storage processor was populated with four back-end buses to provide 4 Gb connectivity to the DAEs and disk drives. Each storage processor had eight 4 Gb front-end Fibre Channel ports for SAN connectivity. There were also two iSCSI ports on each storage processor which were not used. Nine DAEs were populated with 130 x 300 GB 15k drives, and five 146 GB drives were also used for the vault. The CLARiiON was configured to house a 1 TB production database and two clone copies of that database. The clone copies were utilized as follows:

• Gold copy • Backup copy

Gold copy At various logical checkpoints within the testing process the gold copy was refreshed to ensure there was an up-to-date copy of the database available at all times. This was to ensure that an instantaneous recovery image was always available in the event that any logical corruption occurred during, or as result of, the testing process. If any issue did occur then a reverse synchronization from the SnapView clone gold copy would have made the data available immediately, thereby avoiding having to rebuild the database.

Backup copy The backup clone copy was used for NetWorker proxy backups. The clone copy of the database was mounted to the proxy node and the backups were executed on the proxy node. This is also referred to as the “clone mount host”.

Configuration It is a best practice to use ASM external redundancy for data protection when using

EMC arrays. CLARiiON will also provide protection against loss of media, as well as transparent failover in the event of a specific disk or component failure. The following image shows the CLARiiON layout; the CX4-960 deployed for this solution had four 4 Gb Fibre Channel back-end buses for disk connectivity. The back-end buses are numbered Bus 0 to Bus 3. Each bus connects to a number of DAEs (disk array enclosures). DAEs are numbered using the “Bus X Enc Y” nomenclature, so the first enclosure on Bus 0 is therefore known as Bus 0 Enc 0. Each bus has connectivity to both storage processors for failover purposes. Each enclosure can hold up to 15 disk drives. Each disk drive is numbered in an extension of the Bus Enclosure scheme. The first disk in Bus 0 Enclosure 0 is known as disk 0_0_0.


18


The following image shows how ASM diskgroups were positioned on the CLARiiON

array.

The first enclosure contains the vault area. The first five drives 0_0_0 through 0_0_4 have a portion of the drives reserved for internal use. This reserved area contains the storage processor boot images as well as the cache vault area. Disks 0_0_11 to 0_0_14 were configured as hot spares. Disks 0_0_5 to 0_0_9 were configured as RAID Group 0 with 16 LUNs used for the redo logs. These LUNs were then allocated as an ASM diskgroup, named the redo diskgroup. RAID Group 0 also contained the OCR disk and the Voting disk. The next four enclosures contain three additional ASM diskgroups. The following section explores this in more detail.

ASM diskgroups The database was built using four distinct ASM diskgroups:

• The Data diskgroup contains all datafiles and the first control file. • The Online Redo diskgroup contains online redo logs for the database and a

second control file. Ordinarily, Oracle’s best practice recommendation is for the redo logs files to be placed in the same diskgroup as all the database files (the Data diskgroup in this example). However, it is necessary to separate the online redo logs from the data diskgroup when planning to do recovery from split mirror snap copies since the current redo log files cannot be used to recover the cloned database.

• The Flash Recovery diskgroup contains the archive logs. • The Temp diskgroup contains tempfiles.


19


ASM data area MetaLUNs were chosen for ease of management and future scalability. As the data grows, and consequently the number of ASM disks increases, ASM will have an inherent overhead managing a large number of disks. Therefore, metaLUNs were selected to allow the CLARiiON to manage request queues for large number of LUNs. For the Data diskgroup four striped metaLUNs were created, each containing four members. The selection of members for each metaLUN was chosen to ensure that each member resided on a different back-end bus to ensure maximum throughput. The starting LUN for each metaLUN were also carefully selected to avoid all the metaLUNs starting on the same RAID group. This selection criterion was to avoid starting all the ASM disks on the same set of spindles, and alternating the metaLUN members to even out the LUN residence. This methodology was used to ensure that ASM parallel chunk IOs will not be hitting the same spindles at the same time within the metaLUNs when, or if, Oracle performs a parallel table scan.


20


EMC Backup and Recovery for Oracle 11g OLTP Enabled by EMC CLARiiON, EMC Data Domain, EMC NetWorker,

EMC SnapView SnapView clones were used to create complete copies of the database. A clone copy was used to offload the backup operations to the proxy node. A second clone copy was used as a Gold copy. The following graphic shows an example of a clone LUN's relationship to the source LUN, in this example the clone information for one of the LUNs is contained in the ASM datagroup. SnapView clones create a full bit-for-bit copy of the respective source LUN. A clone was created for each of the LUNs contained within the ASM diskgroups, and all clones were then simultaneously split from their respective sources to provide a point-in-time content consistent replica set. The command naviseccli – h arrayIP snapview –listclonegroup –data1 was used to display information on this clone group. Each of the ASM diskgroup LUNs is added to a clone group becoming the clone source device. Target LUN clones are then added to the clone group. Each clone group is assigned a unique ID and each clone gets a unique clone ID within the group. The first clone added has a clone ID of 010000000000000 and for each subsequent clone added the clone ID increments. The clone ID is then used to specify which clone is selected each time a cloning operation is performed.

As shown above there are two clones assigned to the clone group. Clone ID 01000000000000000 was used as the gold copy and clone ID 0200000000000000 was used for backups. (The Navisphere Manager GUI also shows this information.) When the clones are synchronized they can be split (fractured) from the source LUN to provide an independent point-in-time copy of the database.

and Oracle Recovery Manager using Fibre Channel Proven Solution Guide

21


The LUNs used for the clone copies were configured in a similar fashion to the

source copy to maintain the required throughput during the backup process. The image below shows the clone relationship for two of the metaLUNs.


22


Data Domain

Overview The following sections describe how Data Domain systems ensure data integrity and

provide multiple levels of data compression, reliable restorations and multipath configurations. The Data Domain operating system (DD OS) Data Invulnerability Architecture™ protects against data loss from hardware and software failures.

Data integrity When writing to disk, the DD OS creates and stores checksums and self-describing

metadata for all data received. After writing the data to disk, the DD OS then recomputes and verifies the checksums and metadata. An append-only write policy guards against overwriting valid data. After a backup completes, a validation process looks at what was written to disk to see that all file segments are logically correct within the file system and that the data is the same on the disk as it was before being written to disk. In the background, the Online Verify operation continuously checks that data on the disks is correct and unchanged since the earlier validation process. The back-end storage is set up in a double parity RAID 6 configuration (two parity drives). Additionally hot spares are configured within the system. Each parity stripe has block checksums to ensure that data is correct. The checksums are constantly used during the online verify operation and when data is read from the Data Domain system. With double parity, the system can fix simultaneous errors on up to two disks. To keep data synchronized during a hardware or power failure, the Data Domain system uses NVRAM (non-volatile RAM) to track outstanding I/O operations. An NVRAM card with fully-charged batteries (the typical state) can retain data for a minimum of 48 hours. When reading data back on a restore operation, the DD OS uses multiple layers of consistency checks to verify that restored data is correct.

Data compression

DD OS stores only unique data. Through Global Compression, a Data Domain system pools redundant data from each backup image. Any duplicate data is stored only once. The storage of unique data is invisible to backup software, which sees the entire virtual file system. DD OS data compression is independent of data format. This can be structured, for example, databases, or unstructured, for example, text files. Data can be from file systems or raw volumes. Typical compression ratios are 20:1 on average over many weeks. This assumes weekly full and daily incremental backups. A backup that includes many duplicate or similar files (files copied several times with minor changes) benefits the most from compression. Depending on backup volume, size, retention period, and rate of change, the amount of compression can vary. Data Domain performs inline deduplication only. Inline deduplication ensures:

• Smaller footprint • Longer retention • Faster restore • Faster time to disaster recovery


23


SISL Inline deduplication is enabled by Streams Informed Segment Layout (SISL). SISL

identifies 99 percent of duplicate segments in RAM and ensures that all related segments are stored in close proximity on disk for optimal reads.

Multipath and load-balancing configuration

Multipath configuration and load balancing are supported on Data Domain systems that have at least two HBA ports. In a multipath configuration on a Data Domain system, each of two HBA ports on the system is connected to a separate port on the backup server.


24


SAN topology

SAN topology Oracle layout

The two-node Oracle 11g RAC cluster nodes and the proxy node were cabled and zoned as shown in the following image. Each node contained four two-port HBAs. Four of the available ports were used to connect the nodes to the CX4-960. CLARiiON best practice dictates that single initiator soft zoning is used. Each HBA is zoned to both storage processors. This configuration offers the highest level of protection and may also offer higher performance. It entails the use of full-feature PowerPath software. In this configuration, there are multiple HBAs connected to the host; therefore, there are redundant paths to each storage processor. There is no single point of failure. Data availability is ensured in event of an HBA, cable, switch, or storage processor failure. Since there are multiple paths per storage processor, this configuration benefits from the PowerPath load-balancing feature and thus provides additional performance. The connectivity diagram below shows the two-node Oracle 11g RAC cluster nodes.


25


The remaining four HBA ports in each node were zoned to the Data Domain DD880

appliance. This zoning approach ensured that the primary storage CLARiiON CX4-960 and the backup storage DD880 were on different HBA ports. This ensures that the traffic is segregated during backups as recommended by EMC best practice. Single initiator zoning was also used when zones were created for the DD880. The zoning of the nodes to the DD880 also had redundancy built-in to ensure that tape drives were available in the event of a cable, switch, or HBA failure.


26


NetWorker topology

The EMC NetWorker environment provides the ability to protect your enterprise against the loss of valuable data. In a network environment, where the amount of data grows rapidly, the need to protect data becomes crucial. The EMC NetWorker product gives you the power and flexibility to meet such a challenge.

A Data Domain system integrates into a NetWorker environment as the storage destination for directed backups. In this solution the Data Domain system was configured as a VTL. This takes advantage of the speed of disk and easily integrates with a previously configured NetWorker environment.


27

Chapter 4: Oracle Database Design


Overview

Introduction to Oracle database design

This chapter provides guidelines on the Oracle database design used for this validated solution. The design and configuration instructions apply to the specific revision levels of components used during the development of the solution. Before attempting to implement any real-world solution based on this validated scenario, gather the appropriate configuration documentation for the revision levels of the hardware and software components. Version-specific release notes are especially important.

ASM diskgroups

The database was built with four distinct ASM diskgroups (+DATA, +FRA, +REDO, and +TEMP).

ASM Diskgroup Contents

DATA Data and index tablespaces, controlfile

FRA Archive logs

REDO Online redo log files, controlfile

TEMP Temporary tablespace The ASMCMD CLI lists the diskgroups, showing the state of each one.


28


Control files The Oracle database, in this solution, has two control files, each stored in different

ASM diskgroups.

Redo logs All database changes are written to the redo logs (unless logging is explicitly turned

off) and are therefore very write-intensive. To protect against a failure involving the redo log, the Oracle database was created with multiplexed redo logs so that copies of the redo log can be maintained on different disks. Archive log mode was enabled which automatically created database-generated offline archived copies of online redo log files. Archive log mode enables online backups and media recovery. Note Oracle recommends that archive logging is enabled; the following graphic shows how to check that the database is in archive log mode.


29


The previous graphic shows that once archive log mode is enabled, the archive logs were written out to the FRA diskgroup.

Parameter files A centrally located server parameter file (spfile) was used to store and manage the

database initialization parameters persistently by all RAC instances. Oracle recommends that you create a server parameter file as a dynamic means of maintaining initialization parameters.

EMC Backup and Recand Oracle Recovery Manager using Fibre Channel Proven Solution Guide

30

overy for Oracle 11g OLTP Enabled by EMC CLARiiON, EMC Data Domain, EMC NetWorker,


Swingbench TPC-C-like toolkit

The order entry wizard that is used to create the SOE schema in Swingbench 2.3 limits its size to 100 GB. The reason for this is that it executes as a single-threaded operation and would take an unreasonable time to create a schema any larger. However, Datagenerator is capable of creating larger schemas that would generate much higher levels of I/O (index lookups). The script alterations specific to this solution environment to create a 1 TB database can be seen in the appendix.


31


The Configuration dialog box (see the following image) enables you to change all of

the important attributes that control the size and type of load placed on your server. Four of the most useful are: • Number of Users: This describes the number of sessions that Swingbench will

create against the database. • Min and Max Delay Between Transactions (ms): These values control how long

Swingbench will put a session to sleep between transactions. • Benchmark Run Time: This is the total time Swingbench will run the bench for.

After this time has expired Swingbench will automatically log off the sessions. This graphic shows a typical example with 120 concurrent sessions.


32

Chapter 5: Installation and Configuration


Overview

Introduction to installation and configuration

This chapter provides procedures and guidelines for installing and configuring the components that make up the validated solution scenario. The installation and configuration instructions presented in this chapter apply to the specific revision levels of components used during the development of this solution. Before attempting to implement any real-world solution based on this validated scenario, gather the appropriate installation and configuration documentation for the revision levels of the hardware and software components planned in the solution. Version-specific release notes are especially important.

Contents This chapter contains the following topics:

Topic See Page

Navisphere 34

PowerPath 37

Install Oracle clusterware 41

Data Domain 47

NetWorker 51

Multiplexing 57


33


Navisphere

Overview Navisphere Management Suite enables you to access and manage all CLARiiON

advanced software functionality.

Register hosts The Connectivity Status view in Navisphere, seen in the image below, shows the

new host as logged in but not registered. Install the Navisphere host agent on the host and reboot, and the HBAs will then automatically register.

The Hosts tab shows the host as unknown and the host agent is unreachable; this is because the host is multi-homed, that is, the host has multiple NICs configured, see the following image.


34


A multi-homed host machine has multiple IP addresses on two or more NICs. The host can be physically connected to multiple data links that can be on the same or different networks. When you install Navisphere Host Agent on a multi-homed host, the host agent, by default, binds to the first NIC in the host. If your host is multi-homed, for the host agent to successfully register with the desired CLARiiON storage system, you need to configure the host agent to bind to a specific NIC. This is rectified by setting up an agentID.txt file. To do this stop the Navisphere agent, then rename or delete the HostIdFile.txt file located in /var/log, as shown in the following image.

Create agentID.txt in root; this file should only contain the fully qualified hostname of the host and the IP address HBA/NIC port that the Navisphere agent should use. The agentID.txt file should contain only these two lines and no special characters, as shown in the following image.


35


Then stop and restart the Navisphere agent; this re-creates the HostIdFile.txt file binding the agent to the correct NIC. The host now shows correctly on the Hosts tab.

The host is now shown as registered correctly with Navisphere, as shown in the previous image.


36


PowerPath

Overview EMC PowerPath provides I/O multipath functionality. With PowerPath, a node can

access the same SAN volume via multiple paths (HBA ports), which enables both load balancing across the multiple paths and transparent failover between the paths.

PowerPath policy

After PowerPath has been installed and licensed it is important to set the PowerPath policy to “CLARiiON-Only”. The following image shows the powermt display output prior to setting the PowerPath policy.

The I/O Path Mode is shown to be unlicensed.


37


Once the PowerPath Policy has been set correctly all paths are now alive and

licensed. The previous image shows the powermt set policy command and the powermt display command output for CLARiiON LUN 80. It lists the eight paths for this device. These paths are managed by PowerPath. The LUN is owned by SPA, therefore the four paths to SPA are active and the remaining paths to SPB are passive. All ASM diskgroups were then built using PowerPath pseudo names. Note A pseudo name is a platform-specific value assigned by PowerPath to the PowerPath device. Because of the way in which the SAN devices were discovered on each node, there was a possibility that a pseudo device pointing to a specific LUN on one node might point to a different LUN on another node. The emcpadm command was used to ensure consistent naming of PowerPath devices on all nodes.

The following image shows how to determine the available pseudo names.


38



The next image shows how to change the pseudo names using the following command: emcpadm renamepseudo –s <xxx> – t <yyy>

This table shows the PowerPath names associated with the LUNs used in the ASM diskgroups.

Diskgroup Purpose Diskgroup Name Path CLARiiON

LUN

Data files DATA /dev/emcpowerac 10

/dev/emcpowerad 8

/dev/emcpowerae 2

/dev/emcpoweraf 0

Online Redo Logs REDO /dev/emcpowere 65

/dev/emcpowerf 64

/dev/emcpowerg 63

/dev/emcpowerh 62

/dev/emcpoweri 61

/dev/emcpowerj 60

/dev/emcpowerk 59

/dev/emcpowerl 58

/dev/emcpowerm 57

/dev/emcpowern 56

/dev/emcpowero 55

/dev/emcpowerp 54

/dev/emcpowerq 53

/dev/emcpowerr 52

/dev/emcpowers 50

/dev/emcpowert 51

Temp/Undo TEMP /dev/emcpoweru 22


39


/dev/emcpowerv 20

/dev/emcpowerw 16

/dev/emcpowerx 18

Flash Recovery FRA /dev/emcpowery 23

/dev/emcpowerz 21

/dev/emcpoweraa 19

/dev/emcpoweab 17

High availability health check

To verify that the hosts and CLARiiON are set up for high availability, install and run the naviserverutilcli utility on each node to ensure that everything is set up correctly for failover. To run the utility, use the following command: naviserverutilcli hav –upload – ip 172.<xxxxxxx>

In addition to the standard output, the health check utility also uploads a report to the CLARiiON storage processors that can be retrieved and stored for reference.


40


Install Oracle clusterware

Overview Oracle 11g clusterware was installed and configured for both production nodes. Below are a number of screenshots taken during the installation, showing the configuration of both RAC nodes.

Specify cluster The image below shows the installation summary screen.

Configure ASM and Oracle 11g software and database

Before configuring Oracle and ASM, it is recommended to review the Oracle Database Installation Guide 11g Release 1 (11.1) for Linux. The following general guidelines apply when configuring ASM with EMC technology:

• Use multiple diskgroups, preferably a minimum of four, optimally five. Place the Data, Redo, Temp, and FRA in different (separate) diskgroups.

• Use external redundancy instead of ASM mirroring. • Configure diskgroups so that each contains LUNs of the same size and

performance characteristics. • Distribute ASM diskgroup members over as many spindles as is practical for

the site’s configuration and operational needs.


41


Partition the disks In order to use either file systems or ASM, you must have unused disk partitions available. This section describes how to create the partitions that will be used for new file systems and for ASM. When partitioning the disks it is important that the partition is aligned correctly. Intel-based systems are misaligned due to the metadata written by the BIOS. To correctly align the partition to ensure improved performance, an offset of 64 KB (128 blocks) was used. This example uses /dev/emcpowera (an empty disk with no existing partitions) to create a single partition for the entire disk.


42


ASM diskgroup creation

The Oracle DBCA creates the ASM data diskgroup and the init.ora for the ASM instance. Additional diskgroups can then be created.

ASM uses mirroring for redundancy. Three types of redundancy are supported by ASM. They are:

• External redundancy. • Normal redundancy: 2-way mirrored. At least two failure groups are needed. • High redundancy: 3-way mirrored. At least three failure groups are needed.

EMC recommends using external redundancy as provided by the CLARiiON CX4-940. Refer to the CLARiiON configuration setup.


43



Database installation

Once the ASM diskgroups were created, Oracle Database 11g 11.1.0.6.0 was installed.


44


The Oracle environment was patched to 11.1.0.7.0.


45


Block change tracking

The block change tracking feature for incremental backups improves incremental backup performance by recording changed blocks in each datafile in a block change tracking file. This file is a small binary file stored in the database area. RMAN tracks changed blocks as redo is generated. If you enable block change tracking, RMAN uses the change tracking file to identify changed blocks for an incremental backup, thus avoiding the need to scan every block in the datafile. RMAN only uses block change tracking when the incremental level is greater than 0 because a level 0 incremental backup includes all blocks.


46


Data Domain

Introduction Data Domain DD880 integrates easily into existing data centers and can be

configured for leading backup and archiving applications using NFS, CIFS, OST, or VTL protocols. This solution focuses on VTL over Fibre Channel SAN. The Data Domain appliance was configured with four Fibre Channel connections to the SAN. The largest number of tape drives assigned per channel for this solution was two, giving a maximum number of eight RMAN channels per node.

VTL option The following image shows the Data Domain Enterprise Manager. The VTL option

requires an additional Data Domain license.

To add the license click the Maintenance tab and launch the Configuration Wizard from the Tasks drop-down, as shown in the following image.


47


Select option 1, Licenses.

Add the VTL license and save the configuration.

After the VTL license has been applied, the VTL service can then be enabled. Go to the Data Management page and select VTL – Actions – Enable.


48


After the VTL license was added, the virtual tape library was created.

Virtual tapes were created in a tape pool. For this solution the default pool was selected.

The tapes were then imported into the library.


49


Access groups were created within the DD880 appliance to allow access to individual tape drives and the media changer. The HBA WWNs of the nodes will be present in the Physical Resources tab only if they are correctly zoned on the FC switches. The HBAs are then available to be added to the access groups as initiators. Tape drives are added to the access group as LUNs. The LUNs are assigned primary and secondary ports on the appliance.


50


NetWorker

NetWorker introduction

The following NetWorker components were installed: • NetWorker Server

• NetWorker Server • NetWorker Management Console

• RAC nodes • NetWorker storage node • NetWorker Client • NMO

• Proxy node • NetWorker storage node • NetWorker Client • NMO

NetWorker configuration

After zoning the SAN switches correctly and creating the access groups on the Data Domain appliance, the servers can pick up the tape devices. A SCSI bus rescan is required to achieve this. When selecting tape devices for use it is important to select the SCSI non-rewind devices. The st driver provides the interface to a variety of SCSI tape devices under Linux.

• First (auto-rewind) SCSI tape device name: /dev/st0 • Second (auto-rewind) SCSI tape device name: /dev/st1 • First the non-rewind SCSI tape devices: /dev/nst0 • Second the non-rewind SCSI tape devices: /dev/nst1

The following image shows the devices selected are all non-rewind devices.


51


The three servers were added to NetWorker as storage nodes. The devices can then

be made available to NetWorker by performing a scan for devices.

To change device properties, right-click on the device and select Properties. The parameters used for this solution are described in the following section. The NetWorker device Target sessions and Max sessions parameters were both set to 1. This effectively disabled NetWorker multiplexing, which is important when using data deduplication. Multiplexing, and its effect on data duplication, is described in detail in the “Multiplexing” section.


52


The NetWorker Wizard was used to configure the client backups on each node.

NetWorker Module for Oracle NMO was installed on each node to enable NetWorker integration with Oracle RMAN.

Client configuration, user, and connection details to the database to be backed up are entered along with the RMAN catalog user and connectivity details, as shown in the next graphic.


53


The following parameters were also modified:

• Number of channels • Backup level • Control file backup • Archive redo log backup • Filesperset

Testing was conducted using different numbers of RMAN channels. Backup levels were set at level 0, or full backups, on the weekend and level 1, or differential incremental backups, on weekdays. The control file and archive redo logs were included in the backups. Refer to the “Testing and Validation” section for more details.


54


The filesperset parameter was set to 1; this is explained in detail in the next section.


55


The wizard creates the RMAN script, as shown below, which can be modified if

required.


56


Multiplexing

Overview to NetWorker multiplexing

Today multiplexing is a common practice used in many backup operations. Multiplexing became widely used as traditionally most tape drives had a greater bandwidth than an individual backup client could fill. To ensure a tape drive bandwidth was efficiently utilized, several channels are multiplexed together to keep the tape drive busy. The following image shows six 10 MB/s clients writing to a 60 MB/s device. Without multiplexing the tape device is not used to its capacity, as one client writes the others have to wait for completion, therefore the tape device is underutilized. In this example, the tape device has a bandwidth of 60 MB/s and each backup client has a bandwidth of 10 MB/s.

The image below shows six 10 MB/s clients writing to a 60 MB/s device, with multiplexing enabled, all clients are interleaved together to take advantage of the extra bandwidth of the tape device.


57


RMAN multiplexing

When using a deduplication appliance, such as a DD880, it is best practice to ensure that multiplexing is disabled. When creating backup sets, RMAN can simultaneously read multiple files from disk and then write their blocks into the same backup set. For example, RMAN can read from two datafiles simultaneously, and then combine the blocks from these datafiles into a single backup piece. The combination of blocks from multiple files is called RMAN multiplexing. Similar to NetWorker multiplexing, RMAN multiplexing has the same negative effect on deduplication The parameter that sets up multiplexing within Oracle is filesperset. The filesperset parameter specifies the number of files that will be packaged together and sent on a single channel to a tape device. This has the same effect as mixing bits from many files, and again makes it more difficult to detect segments of data that already exist. Therefore, to take full advantage of data deduplication it is important to have the filesperset parameter set to 1.


58

Chapter 6: Testing and Validation


Overview

Introduction to testing and validation

Storage design is an important element to ensure the successful development of the EMC Backup and Recovery for Oracle 11g OLTP Enabled by EMC CLARiiON, EMC Data Domain, EMC NetWorker, and Oracle Recovery Manager using Fibre Channel solution.

Contents This section contains the following topic:

Topic See Page

Section A: Test results summary and resulting recommendations

60


59


Section A: Test results summary and resulting recommendations

Description of the results summary and conclusions

Testing was conducted using two, four, six, and eight RMAN channels. Backups were run using a backup cycle consisting of level 0, or full backups, on the weekend and level 1, or differential incremental backups, Monday through Thursday. For the purposes of this solution Friday COB was deemed to be the start of the weekend. The starting point for an incremental backup strategy is a level 0 incremental backup, which backs up all blocks in the database. An incremental backup at level 0 is identical in content to a full backup but, unlike a full backup, the level 0 backup is considered part of the incremental backup strategy. A level 1 incremental backup contains only blocks changed after a previous incremental backup. A level 1 backup can be a cumulative incremental backup, which includes all blocks changed since the most recent level 0 backup, or a differential incremental backup, which includes only blocks changed since the most recent incremental backup. Incremental backups are differential by default. Note When using a deduplication appliance such as Data Domain it is often recommended to run only full backups as only the unique data will be stored. However, this was outside the scope of the solution. Archived redo logs and the control file were also backed up as part of each backup that occurred during this solution. Backing up the archived redo logs had a significant impact on the overall change rate of the database. The change rate of the database was 2 percent. However, because the archived log files were backed up on every backup, the change rate observed during incremental backups was actually much higher, closer to 10 percent. Both NetWorker and RMAN multiplexing (filesperset=1) were disabled for all backups, as this has a negative effect on deduplication rates achieved. For this use case a number of tests were carried out on the Oracle 11g OLTP backup and recovery infrastructure. At a high level the tests performed were:

• Orion validation • RMAN channel configuration • Swingbench

• Validate Swingbench profile • Backup from production • SnapView clone copy from production • Backup from proxy

• Data Domain deduplication • Restore


60


Orion validation

Once the disk environment was set up on the CLARiiON CX4-960, the disk configuration was validated using an Oracle toolset called Orion. Orion is the Oracle I/O Numbers Calibration Tool designed to simulate Oracle I/O workloads without having to create and run an Oracle database. It utilizes the Oracle database I/O libraries and can simulate OLTP workloads (small I/Os) or data warehouses (large I/Os). Orion is useful for understanding the performance capabilities of a storage system, either to uncover performance issues or to size a new database installation. Note This is a destructive tool so it should only be run against raw devices prior to installing any database or application. This graph shows total throughput on a single node, with four metaLUNs, consisting of 40 disks.

This demonstrates the desired scaling.


61


RMAN channel configuration

This image shows the average MB/s throughput when backing up the database using different numbers of RMAN channels. Eight RMAN channels proved to be the fastest in this environment.

Shown here is a typical example of NetWorker backing up the database, in this case using four RMAN channels.

All subsequent tests were run using the following base parameters. The methodology for selecting these parameters was explained earlier in this document.

Backup parameters RMAN Channels 8 Filesperset 1 NetWorker Target Session 1 NetWorker Max Session 1 Archived Redo Logs Included in all backups Control File Included in all backups


62


Validate Swingbench profile

The following image shows the processor activity on Node1 with the Swingbench load running against the cluster.

The next image shows the response time of the data metaLUNs (Data ASM

diskgroup) on the CLARiiON array during the same period.

Note The array time is one hour behind the server time.


63


Backup from production

The following graph shows the processor activity on Node 1. Similar to the previous example the Swingbench load is running against the cluster. In addition, an RMAN backup initiated by NetWorker is also running against Node 1. The backup is running against the same LUNs that are serving the Swingbench load. There is an increase in CPU utilization and iowait.

The following graph shows the response times of Data metaLUNs (Data ASM diskgroup) on the CLARiiON. The response time is higher for the duration of the backup.

EMC Backup and Recand Oracle Recovery Manager using Fibre Channel Proven Solution Guide

64

overy for Oracle 11g OLTP Enabled by EMC CLARiiON, EMC Data Domain, EMC NetWorker,


Clone copy from production

The following graph again shows the processor activity on Node 1 under the Swingbench load. In this example, a sync of the proxy clone is started and finished during the time window.

The following graph shows the response time on the Data metaLUNs (Data ASM diskgroup) on the CLARiiON. There is an initial increase in response time and iowait (highlighted) when the sync is started. But the duration of the impact to the production nodes is much shorter when compared to running the backup from one of the nodes.


65



The following graph shows processor activity on Node 1 under Swingbench load. Again, a clone sync is initiated. In this example, the database is also put into hot backup mode after the completion of the sync. No desirable effect can be seen on the RAC node when in hot backup mode.

The following image shows the response time on the metaLUNs (Data ASM diskgroup) on the CLARiiON. The first increase in response time, circa 09:47, corresponds to the start of the clone sync.


66


The second larger increase in response time, between 9:59 and 10:03, occurs when the database is put into hot backup mode. The spike occurs when the database is put into hot backup mode because:

• Any dirty databuffers in the database buffer cache are written out to files and the datafiles are checkpointed.

• The datafile headers are updated to the system change number (SCN) captured when the begin backup command is issued. The SCN is not incremented with checkpoints while a file is in hot backup mode. This lets the recovery process understand which archive redo log files may be required to fully recover this file from that SCN onward.

• The datablocks within the database files continue to be read and written to. • During hot backup, an entire block is written to the redo log files the first time

the data block is changed. Subsequently, only redo vectors (changed bytes) are written to the redo logs.

When the database is taken out of hot backup mode the datafile header and SCN are updated.

This shows that there is less impact on the RAC nodes when using SnapView clones to offload the backup to the proxy host, compared to running the backup directly from the RAC node.


67


Backup from proxy node

The following graph shows processor activity on the proxy node during a backup. The subsequent graph shows the response time on the proxy clone Data metaLUNs (Data ASM diskgroup) on the CLARiiON.


68


The following two graphs show a complete level 0 backup from start to finish. They

show both production RAC nodes under Swingbench load. A clone sync is initiated at approximately 2:50. Again, there is a short period at the beginning of the sync during which an impact is observed. The sync completed at approximately 2:58. The database was put into hot backup mode prior to fracturing the clones; there was no noticeable effect on the production nodes.


69


The following graph shows the proxy node during the same time interval. There was

no load running against the proxy node. To facilitate backing up the clone copy of the database, the instance was started and database mounted on the proxy node.

The next graph shows the response times from the CLARiiON for the duration of the backup. The response time of the production data metaLUNs (the ASM data diskgroup), is tracked on the upper metric. The first peak is the clone sync and the second peak is when the database is in hot backup mode. The lower metric tracks the proxy clone data metaLUNs during the same period; the backup is taken from these LUNs on the proxy node, taking the additional overhead of the backup away from the production nodes.


70


Deduplication The following image shows the data stored on the DD880 after the first weekly

backup cycle. The backup cycle consisted of an RMAN level 0 (full) backup on the weekend and RMAN level 1 backups Monday through Thursday. Oracle Block Change Tracking was enabled to improve incremental backup performance. The database daily change rate is ≈ 2 percent. However, because the archived log files were also backed up, the change rate observed during incremental backups was actually much higher, closer to 10 percent. Note The graphs show the total “Data Written” to the DD880 increasing over time; this is also described as the logical capacity. The “Data Stored” refers to the unique data that is stored on the appliance. The “% Reduction” shows the storage savings gained from using Data Domain.

By eliminating redundant data segments, the Data Domain system allows many more backups to be stored and managed than would normally be possible for a traditional storage server. While completely new data has to be written to disk whenever discovered, the variable-length segment deduplication capability of the Data Domain system makes finding identical segments extremely efficient.


71


The following graph shows the deduplication after two weeks; there is an increase at

the weekend, when the level 0 full backups are run.

A falloff in the deduplication factor can be seen on weekdays when the backup policy is incremental and archived redo logs are contained within every backup.

The graph above shows the backup cycle trend and deduplication factor after three weeks.


72


The graph above illustrates a deduplication factor of 6:1 after five weeks, which yields a storage capacity saving of 84 percent (see the following graph).


73


Restore Data Domain Streams Informed Segment Layout (SISL) technology ensures

balanced backup and restore speeds. The following image shows the duration of a backup of the database compared to a restore of the database. Both the backup and restore processes were initiated on the proxy node using eight RMAN channels.


74


The following image shows the restore throughput of the DD880 appliance. A sustained read throughput of over 630 MB/s was achieved. Comparing this to the data in the subsequent image, which shows the throughput achieved by the DD800 appliance during a backup, this further demonstrates the balance between backup and restore speeds.

The following image shows the DD880 appliance during a backup with a sustained write throughput of over 790 MB/s.


75

Chapter 7: Conclusion


Overview

Introduction to conclusion

This proven solution Reference Architecture details an Oracle infrastructure design leveraging an EMC CLARiiON CX4-960 array, EMC Data Domain DD880, and EMC NetWorker. Also included are various test results, configuration practices, and recommended specific Oracle storage design layouts that meet both capacity and consolidation requirements. Described in this document are many of the technologies that enable the benefits outlined below.


Conclusion Traditional hardware compression provides substantial cost savings in Oracle environments. However, in this solution data deduplication has been shown to significantly reduce the amount of data that needs to be stored over an extended period of time. This offers cost savings both from a management standpoint and in the numbers of disks or tapes required by a customer to achieve their long-term backup strategy. Data deduplication can fundamentally change the way organizations protect backup and nearline data. Deduplication changes the repetitive backup practice of tape, with only unique, new data written to disk. The test results show that, in an environment utilizing RMAN incremental backups, a data deduplication ratio of over 6.3:1, resulting in an 84 percent saving in the storage required to accommodate the backup data, makes it economically practical to retain the savesets for longer periods of time. This reduces the likelihood that a data element must be retrieved from the vault. Both of these factors can significantly improve the RTO. Although cost savings are generally not the initial reason to consider moving to disk backup and deduplication, financial justification is almost always a prerequisite. With the potential cost savings of disk and deduplication, the justification statement becomes, “we can achieve all of these business benefits and save money.” That is a compelling argument. The solution meets the business challenges in the following manner:

• Ability to keep applications up 24x7 • Faster backup and restores – meet more aggressive backup windows,

and restore your key applications in minutes, not days • Reduced backup windows – minimize backup windows to reduce

impact on your application and system availability

• Protect the business information as an asset of the business • Reduced business risk – restore data quickly and accurately with built-

in hardware redundancy and RAID protection • Reduced backup windows – minimize backup windows to reduce

impact on your application and system availability

• Efficient use of both infrastructure and people to support the business • Improved IT efficiency – save hours of staff time and boost user

productivity • Correct costs / reduce costs – match infrastructure costs with changing

information value via efficient, cost-effective tiered storage


76


In summary, utilizing the solution components, in particular CLARiiON technology, EMC Data Domain, and EMC NetWorker software, provides customers with the best possible backup solution to prevent both user and business impact. Business can continue as usual, as if there were no backup taking place at all. In customer environments where, more than ever, there is a trend toward 24x7 activity, this is a critical differentiator that EMC can offer.

Next steps EMC can help to accelerate assessment, design, implementation, and management

while lowering the implementation risks and costs of a backup and recovery solution for an Oracle Database 11g environment. To learn more about this and other solutions contact an EMC representative or visit www.EMC.com/solutions/oracle.


77

Appendix A: Scripts

Appendix A: Scripts

Datagenerator .env file

Alterations specific to use case environment were entered in to the Datagenerator.env file. #!/bin/bash # Set the following to reflect the root directory of your Java installation export JAVAHOME=/usr/java/jdk1.5.0_19/ # Set the following to the directory where you installed datagenerator export DATAGENHOME=/Datagenerator/datagenerator040198/datagenerator # Set the following to the location of your TimesTen install (optional) #export TTHOME=/opt/TimesTen/tt70 # export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/lib:$TTHOME/lib export CLASSPATH=$JAVAHOME/jre/lib/rt.jar:$JAVAHOME/lib/tools.jar:$DATAGENHOME/lib/datagenerator.jar # The following is needed for 10g environments #export CLASSPATH=$CLASSPATH:$ORACLE_HOME/jdbc/lib/ojdbc14.jar # The following is only needed for 11g environments export CLASSPATH=$CLASSPATH:$ORACLE_HOME/jdbc/lib/ojdbc5.jar # The following is only needed for TimesTen environments #export CLASSPATH=$CLASSPATH:$TTHOME/lib/classes15.jar

Datagenerator SOE schema creation script

The following Datagenerator script was used to create and populate the SOE schema used by Swingbench when running a typical TPC-C-type load against the database. -- Datagenerator "SOE" benchmark schema creation -- Author : Dominic Giles -- Created : 1/9/2008 -- set verify off define tablespace=SOE variable tablespace_size number define tablespace_size=1024 define datafile='+DATA' define indextablespace=SOE_INDEX define indextablespace_size=2048 define indexdatafile='+DATA' define username=SOE define password=SOE define indexprefs=nologging define parallelism=16 define parallelclause='' define usecompression='' -- Uncomment the following line to enable compression on the sales and customer -- define usecompression='COMPRESS' define connectstring='//tce-r900-enc01/orcl' variable scale number define scale=1 accept username default &username prompt 'Order entry username [&username] : '


78

Appendix A: Scripts


accept password default &password prompt 'Order entry password [&password] : ' accept tablespace default &tablespace prompt 'Default tablespace [&tablespace] : ' accept datafile default &datafile prompt 'Datafile for tablespace &datafile [&datafile] : ' accept indextablespace default &indextablespace prompt 'Default index tablespace [&indextablespace] : ' accept indexdatafile default &indexdatafile prompt 'Datafile for tablespace &indexdatafile [&indexdatafile] : ' accept scale default &scale prompt 'Enter scale. 1 = 1GB, 100=100GB etc [&scale] : ' accept connectstring default &connectstring prompt 'Enter connectstring for database [&connectstring] : ' accept parallelism default &parallelism prompt 'Enter level of parallelism [&parallelism] : ' pause Generation will now begin. Press return to continue or control C to exit variable t1 number; set termout off col new_tablespace_size new_value tablespace_size col new_indextablespace_size new_value indextablespace_size col new_parallelclause new_value parallelclause select (round(&scale*&tablespace_size))||'M' new_tablespace_size, (round(&scale*&indextablespace_size))||'M' new_indextablespace_size, case when &parallelism <= 1 then '' else 'parallel '||&parallelism end new_parallelclause from dual / set timing on set termout on timing start stage_timer prompt ************************************** prompt Starting tablespace creation prompt ************************************** @@soecreatetablespace.sql prompt ************************************** prompt Completed tablespace creation timing show stage_timer prompt ************************************** @@soecreateuser.sql connect soe/SOE --@@soedrop.sql @@soecreatetables.sql prompt ************************************** prompt Starting data generation prompt ************************************** timing start stage_timer !cd ../..;./datagenerator -c soe.xml -scale &scale -nodrop -ni -nc -cl -db -u &username -p &password -cs &connectstring -tc &parallelism -async -bs 100 -commit 100000 prompt ************************************** prompt Completed data generation timing show stage_timer prompt ************************************** timing start stage_timer prompt ************************************** prompt Creating constraints prompt ************************************** @@soeconstraints.sql prompt **************************************


79

Appendix A: Scripts

prompt Creating indexes prompt ************************************** prompt Creating indexes @@soeindexes.sql prompt ************************************** prompt Completed index and constraint creation timing show stage_timer prompt ************************************** timing start stage_timer @@soeviews.sql @@soepackage.sql prompt ************************************** prompt Started schema analysis prompt ************************************** set timing off exec dbms_stats.gather_schema_stats(SYS_CONTEXT('USERENV','CURRENT_SCHEMA'),degree =>&parallelism); @@soeinvalid.sql prompt ************************************** prompt Completed analysis timing show stage_timer prompt ************************************** prompt prompt Completed building Order Entry schema prompt prompt ************************************** exit


SOE schema installation

The SOE schema was installed manually using the scripts located in the $DATAGEN_HOME/bin/scripts/soe directory. These scripts can be run directly from SQL*Plus as the “SYS” or “SYSTEM” user. The following command is an example of how to start the installation using the soe_install script: − [oracle@TCE-R900-ENC01 soe]$ sqlplus /nolog

SQL*Plus: Release 11.1.0.7.0 - Production on Thu Aug 20 10:47:32 2009 Copyright (c) 1982, 2008, Oracle. All rights reserved. SQL> connect /as sysdba Connected. SQL> @soe_install Order entry username [SOE] : Order entry password [SOE] : Default tablespace [SOE] : Datafile for tablespace +DATA [+DATA] : Default index tablespace [SOE_INDEX] : Datafile for tablespace +DATA [+DATA] : Enter scale. 1 = 1GB, 100=100GB etc [1] : 200 Enter connectstring for database [//tce-r900-enc01/orcl] : Enter level of parallelism [16] : 12 Generation will now begin. Press return to continue or control C to exit This script was used to create the schema of the testing user (in this case, soe) and load all of the data necessary to run the tests.


80

Appendix A: Scripts


81

NetWorker RMAN backup script

The RMAN script below is a typical example of one used to generate backups through the NetWorker console. This example shows an eight-channel incremental level 0 backup to tape. Each backup was assigned a tag ID, which was later used as part of the restore process. RUN { ALLOCATE CHANNEL CH1 TYPE 'SBT_TAPE'; ALLOCATE CHANNEL CH2 TYPE 'SBT_TAPE'; ALLOCATE CHANNEL CH3 TYPE 'SBT_TAPE'; ALLOCATE CHANNEL CH4 TYPE 'SBT_TAPE'; ALLOCATE CHANNEL CH5 TYPE 'SBT_TAPE'; ALLOCATE CHANNEL CH6 TYPE 'SBT_TAPE'; ALLOCATE CHANNEL CH7 TYPE 'SBT_TAPE'; ALLOCATE CHANNEL CH8 TYPE 'SBT_TAPE'; BACKUP INCREMENTAL LEVEL 0 FILESPERSET 1 FORMAT '%d_%U' TAG= 'RUN529' DATABASE PLUS ARCHIVELOG; backup controlfilecopy '+FRA/ORCL/control_backup' tag= 'RUN529_CTL'; RELEASE CHANNEL CH1; RELEASE CHANNEL CH2; RELEASE CHANNEL CH3; RELEASE CHANNEL CH4; RELEASE CHANNEL CH5; RELEASE CHANNEL CH5; RELEASE CHANNEL CH7; RELEASE CHANNEL CH8; }

Oracle RMAN restore script

The restore process consisted of first allocating eight channels then restoring the controlfile, mounting the database, and performing the restore database command using the tag ID assigned earlier. Below is a sample restore script. RUN { ALLOCATE CHANNEL CH1 TYPE 'SBT_TAPE'; ALLOCATE CHANNEL CH2 TYPE 'SBT_TAPE'; ALLOCATE CHANNEL CH3 TYPE 'SBT_TAPE'; ALLOCATE CHANNEL CH4 TYPE 'SBT_TAPE'; ALLOCATE CHANNEL CH5 TYPE 'SBT_TAPE'; ALLOCATE CHANNEL CH6 TYPE 'SBT_TAPE'; ALLOCATE CHANNEL CH7 TYPE 'SBT_TAPE'; ALLOCATE CHANNEL CH8 TYPE 'SBT_TAPE'; restore controlfile from tag'RUN529_CTL'; alter database mount; restore DATABASE from tag'RUN529'; RELEASE CHANNEL CH1; RELEASE CHANNEL CH2; RELEASE CHANNEL CH3; RELEASE CHANNEL CH4; RELEASE CHANNEL CH5; RELEASE CHANNEL CH6; RELEASE CHANNEL CH7; RELEASE CHANNEL CH8; }

emc backup and recovery for oracle 11g oltp backup and recovery for oracle 11g oltp. enabled by emc...

Documents