backup and recovery best practices for oracle 10g with netbackup 6.0 white paper

7/28/2019 Backup and Recovery Best Practices for Oracle 10g With NetBackup 6.0 White Paper

1/45

Backup and recovery best practices for Oracle 10g with NetBackup 6.0white paper

Environment: Oracle 10g on Red Hat AS4, HP Integrity rx7620 server, HP ProLiant DL580 G2 server,using HP StorageWorks EVA8000 and EVA5000 storage arrays and HP StorageWorks EMLE-Series103e and VLS 6510 libraries

Executive summary............................................................................................................................... 3Key findings........................................................................................................................................ 3Overview............................................................................................................................................ 4Components........................................................................................................................................ 5Configuring the hardware..................................................................................................................... 5

Hardware statistics........................................................................................................................... 6Partitioning the Integrity rx7620 server ............................................................................................... 8

Configuring the Management Processor.......................................................................................... 8Creating nPars ............................................................................................................................. 8

Defining the zones ........................................................................................................................... 9Configuring the EVA8000 storage array for primary storage .............................................................. 10Configuring the EVA5000 for disk backups ...................................................................................... 10Configuring the HP StorageWorks EML E-Series 103e Tape Library ..................................................... 11Configuring the HP StorageWorks 6510 Virtual Library System........................................................... 11

Configuring the software .................................................................................................................... 12Configuring the QLogic driver ......................................................................................................... 12

Setting up QLogic Dynamic Load Balancing .................................................................................. 12EVA8000 and EVA5000Active-Active/Active-Passive.................................................................. 12

OCFS2 ......................................................................................................................................... 13OCFS disk configuration ............................................................................................................. 14Setting up the OCFS Clustered File Systems ................................................................................... 15

Working with Benchmark Factory .................................................................................................... 16Oracle parameter changes.......................................................................................................... 18

OLTP workload results..................................................................................................................... 18Oracle backup and restore ................................................................................................................. 19

NetBackup policies ........................................................................................................................ 19Oracle templates............................................................................................................................ 20Setting up the storage units ............................................................................................................. 21Set the master server jobs global attribute ......................................................................................... 21Backup issues ................................................................................................................................ 22Setting up restores.......................................................................................................................... 22


2/45

Backup and restore performance results ............................................................................................... 22Backup methodologies.................................................................................................................... 23

Disk-to-disk backupsBacking up Oracle 10gR2 to EVA5000......................................................... 23Disk-to-virtual tape backupsBacking up Oracle 10gR2 to VLS ....................................................... 23Disk-to-tape backupsBacking up Oracle 10gR2 to EML ................................................................ 23

EVA performance results ................................................................................................................. 23EVA5000 RAW performance characterization............................................................................... 23EVA8000 RAW performance characterization............................................................................... 23

EVA5000 backup and restore results................................................................................................ 23EML E-Series backup and restore performance results ......................................................................... 25

VLS6510 backup and restore performance results.............................................................................. 26

Conclusions ...................................................................................................................................... 28Oracle RMAN ............................................................................................................................... 28Disk-to-disk backup......................................................................................................................... 28Disk-to-tape backup ........................................................................................................................ 28Diskto-VLS backup ........................................................................................................................ 28Server configuration ....................................................................................................................... 29

Best practices .................................................................................................................................... 29Best practices for disk-to-disk backups on the EVA5000 storage array .................................................. 29Best practices for disk-to-tape backups on the EML E-Series Tape Library ............................................... 29Best practices for disk-to-virtual tape backups on the VLS Virtual Tape Library ........................................ 30Best practices for using Oracle Recovery Manager (RMAN)................................................................ 31

Appendix A. Bill of Materials .............................................................................................................. 32Appendix B. Configuring Oracle RMAN .............................................................................................. 34Appendix C. Examples....................................................................................................................... 35Appendix D. Other issues................................................................................................................... 43

Server OS hangs/crashes ............................................................................................................... 43Oracle session hangs ..................................................................................................................... 43NetBackup catalog synchronization ................................................................................................. 43RMAN not backing up archive logs.................................................................................................. 43RMAN specific syntax changes........................................................................................................ 43Imbalanced backups ...................................................................................................................... 43Poorly streaming backups ............................................................................................................... 44RAC issues .................................................................................................................................... 44General Oracle changes................................................................................................................. 44

For more information.......................................................................................................................... 45HP................................................................................................................................................ 45Oracle.......................................................................................................................................... 45Symantec NetBackup 6.0 ............................................................................................................... 45Quest Benchmark Factory 5.0 ......................................................................................................... 45

Open Source Tools ..................................................................................................................... 45


3/45

Executive summary

Many businesses use Oracle databases to store critical data and they need a reliable, robust, andefficient backup and recovery method. Backup and recovery of Oracle databases is a vital part of ITdata protection strategies. Recovery times and backup windows are at the core of establishingrecovery time objectives (RTOs) and recovery point objectives (RPOs). A faster backup method isrequired as data grows to maintain these objectives and allow administrators peace of mind that theimplemented backup and recovery strategy continues to be viable.

The HP StorageWorks Customer Focused Testing Team constructed a Red Hat AS4, Oracle 10genvironment with HP StorageWorks storage arrays to represent an enterprise environment. Thepurpose of the testing was to develop best practices for the backup and restore of an enterpriseenvironment consisting of the deployment of HP StorageWorks 8000 Enterprise Virtual Array(EVA8000), HP StorageWorks 5000 Enterprise Virtual Array (EVA5000), HP StorageWorks 6000

Virtual Library System (VLS6000), and HP StorageWorks Enterprise Modular Library (EML) to providedata protection and recovery operations for Oracle 10g incorporating NetBackup 6.

The objectives for the testing, based on actual customer input, included the following:

Back up an approximate 2.5-TB Oracle database in 2.5 hours or better (approximately 1 TB/hr) Establish best practices for online backup and restore of Oracle databases Determine impact to a transaction workload while backups are running Provide details of NetBackup 6.0 integration with RMANKey findings

Testing successfully provided the following high-level results:

Limited the impact to transaction workloads while optimizing database backup and restore times: HP Integrity RAC VLS backup923 GB/hr (approximately 2 hours, 30 mins) HP Integrity rx7620 RAC EML restore453 GB/hr (approximately 5 hours, 20 mins) HP ProLiant DL580 G2 VLS backup100 GB/hr (approximately 5 hours, 30 mins) HP ProLiant DL580 G2 EML restore60 GB/hr (approximately 7 hours, 40 mins)

Successfully exemplified different EVA5000 configurations. Successfully determined maximum server workloads and capacities for the DL580 and Integrity

rx7620 database servers.

Successfully determined best configurations for each backup methodology: Disk-to-disk Disk-to-tape Disk staging

Important findings uncovered during the tests are documented in the Best practices section.

3


4/45

Overview

The main purpose of the project was to conduct backup and restore testing using various backuptargets in an effort to determine best ways for reducing Oracle database downtime and increasedatabase availability using Symantec NetBackup 6 with RMAN. HP integrated and tested backupand recovery of different Oracle databases with the following objectives:

Demonstrate best practices to back up an approximate 2.5-TB database within 2.5 hours Demonstrate NetBackup 6 integration with Oracle RMAN Determine best practices for backup and recovery to tape, virtual tape, and disk Characterize the impact of online backups on application performanceTesting included full backup of the databases, utilizing staged backups, and performing full andincremental restores. The backup testing included the database data, control files, and archive logswith and without user load. Two different restore tests were performed. In the first test, the fulldatabase was restored after a simulated disaster, such as the loss of an entire storage array. In thesecond test, incremental restores were conducted. Each time a restore was conducted the databasewas opened and checked for data integrity by conducting a simulated workload against the databaseand monitoring the test for errors.

Several options for backup and restore were evaluated as were their impact on database recovery,complexity, and recovery speed:

Integrity and ProLiant Servers to HP StorageWorks EML E-Series Tape LibraryThe scenario utilizeda standard RMAN backup method using an online database and Symantec NetBackup to spool thedata directly to tape. The test goal was to measure the backup time and throughput for a quiescedand busy full database backup. Times for the full backups were recorded.

Integrity and ProLiant Servers to HP StorageWorks VLS6510 Virtual Tape libraryThis scenarioutilized a standard RMAN backup method using an online database and Symantec NetBackup tospool the data directly to a virtual tape library. The test goal was to measure the time taken andthroughput for a quiesced and busy full database backup. Times for the full backups were recorded

Integrity and ProLiant Servers to HP StorageWorks EVA5000 storage arrayThis scenario utilized astandard RMAN backup method using an online database and Symantec NetBackup to spool thedata directly to another physical disk array. The test goal was to measure the time taken andthroughput for a quiesced and busy full database backup. Times for the full backups were recorded

Full restore from HP StorageWorks EML E-Series, VLS6510, and EVA5000 to Integrity and ProLiantServersThis testing provided data for a full restore using each methodology with RMAN andSymantec NetBackup. The test goal was to measure the recovery time using each method with amounted control file. Times for each restore were recorded.

Incremental restore from HP StorageWorks EML E-Series, VLS6510, and EVA5000 to Integrity andProLiant ServersThis testing provided data for an incremental restore using each methodologywith RMAN and Symantec NetBackup. The test goal was to measure the recovery time using eachmethod with a mounted control file. Times for each restore were recorded.

4


5/45

Components

To run these tests, HP configured the system illustrated in Figure 1. The environment was based oninput from customers and is representative of a typical Oracle database environment. The keycomponents include the following:

Oracle 10gBenchmark Factory was used to generate load against the Oracle database, whichwas backed up and restored.

HP rx7620 serverThis server was used to host the Oracle database. Two configurations wereuseda single instance database and a two-node RAC instance.

DL580 serverThis server was used to host multiple database instances. EVA8000The primary SAN-based storage array, which held the Oracle database, logs, and so

on.

EVA5000This storage array was used as a disk-to-disk backup target to show how an older EVAmay be re-deployed within existing infrastructure.

HP StorageWorks EML E-Series 103e Tape LibraryThe EML was configured as a primary tapebackup and restore device.

VLS6510The VLS was configured as a primary tape backup and restore device.

Red Hat AS4 operating systemEnterprise Linux operating system used on both the rx7620 andDL580 servers.

Symantec NetBackup 6.0 was used as the backup application. Benchmark Factory was used to create the OLTP data in the databases and simulate 500- and

1,000-user workloads.

Note:At the time of this writing, the rx7620 server was obsoleted by the rx7640server. The Best Practices outlined in this document are still pertinent.

Configuring the hardware

HP constructed the enterprise configuration using Integrity rx7620 and DL580 servers and EVA8000and EVA5000 storage arrays to best simulate an enterprise environment supporting different Oracledatabases on Itanium- and Xeon-based servers. For the complete list of hardware, seeAppendix

A. Bill of Materials.

Figure 1 shows the configuration of the Oracle RAC database (IA64 RAC and IA64 Single), multipledatabase (IA32 Multi), Benchmark Factory, NetBackup, and command view servers. The 2-Gb/sFibre Channel (FC) links to each system are depicted by the red and blue lines. The white arrowsshow the disk-to-disk backup data flow. The orange arrows show the disk staging data flow, and the

green arrows show the disk-to-tape data flow.

5


6/45

Hardware statistics

The hardware involved in the configuration of this environment includes:

Database Server 1 and 2 HP Integrity rx7620 server

8x IA64 1.6-GHz Processors in one partition

8x QLogic-based HP A6286A Dual Port FC HBAs

64-GB RAMTwo cells and two partitions

Database Server 3 HP ProLiant DL580 G3 server

4x Intel Xeon 3.0-GHz Processors

2x QLogic-based HP FC2214A Dual Port FC HBAs

16-GB RAM

Storage EVA8000 (primary)

144x 300-GB FC drives

Dual Controllers (Active/Active)

One Disk Group (FC)

EVA5000 (D2D, secondary)56x 250GB FATA Drives

Dual Controllers (Active/Active)

One Disk Group (FATA)

Tape devices EML E-Series103e Tape Library

4x LTO3 drives

24x tapes at 400-GB Native

2 FC paths

VLS651024x 250-GB SATA drives

Emulating 12x LTO3 drives with four FC paths

49x tapes at 100-GB each

6


7/45

Figure 1. Environment configuration diagram

7


8/45

Partitioning the Integrity rx7620 server

Configuring the Management ProcessorFor this environment the Integrity rx7620 server was partitioned into two nPars, or server partitions.For partitioning to have been created from a remote system and for remote hardware management,the Management Processor (MP) was configured for remote access:

1. Connect a server running Microsoft Windows or Linux to the MP serial management port.2. Start a terminal program, such as Windows Hyperterminal or Linux Minicom and configure the IP

address with the lc command and follow the on-screen prompts.

NoteYou can configure the MP for DHCP or static IP address. You may alsoenable or disable telnet, SSH, or HTTPS remote access.

Creating nPars

The system was prepared for partitioning and then partitioned:

1. If no partition exists, a new complex must be created with the cc command. Choose cell 0and save the configuration.

2. If a single partition exists, reset the partition for reconfiguration:a. Use the rr command to reset the partition.b. Use the rs command to restart the partition.c. Create a new complex with the cc command.d. Choose cell 0 and save the configuration.

3. On a server installed with nPar utilities, perform the following commands:parcreate -P nextPartition -c 1::: -u Admin h

parstatus u Admin h

This created a partition consisting of a single Cell, and an OS was loaded onto the system. After anOS is loaded, you can install the nPar command line utilities and connect to the MP to create thesecond partition. Alternatively, you can install the nPar utilities on a server running Linux, HP-UX, or

Windows and create the second partition from a remote host.

To increase the process, one partition was created to provide access to all the on-board SCSI disks,and an OS was loaded on the first SCSI disk. The first SCSI disk was then duplicated using the ddutility to the remaining disks. Upon completion of the duplication, the system partitions were re-set andthe rx7620 server was repartitioned into two partitions consisting of one cell each. Each new serverpartition now had an identical bootable OS because of the disk duplication effort. Alternatively, a

network install could have been performed on each partition if a PXE server was set up.Since no PXE server was used, disk duplication was the simplest method of preparing the OS disks foreach server partition.

8


9/45

Defining the zones

Since multiple paths for each device can be mapped, zoning was needed to reduce the total numberof paths. Only one path should be used for tape devices as they are not supported in multipathconfigurations and sometimes cause issues with the backup application installation or configuration.

Since the current release of Red Hat AS4 is limited to 256 busses and multiple busses are generatedfor each path from a host port to an EVA port, bus exhaustion can occur if the EVA is not properlyzoned. After zoning is introduced, it must be used in all cases for the devices to be visible to one

another.Each rx7620 server partition had four dual-port HP A6826A HBAs. Four ports per partition wereexplicitly zoned to the EVA8000 and EVA5000. The last four were zoned to the VLS6510 and two ofthe last four ports were also zoned to the EML E-Series 103e Tape Library.

The DL580 server has two dual-port HP FCA2214A HBAs where two ports were assigned to theEVA8000 and EVA5000 and the other two were assigned to the VLS6510 and the EML E-Series103e Tape Library.

rx7620 server SAN connectivity to EVA8000 storage arrayZones were established for four of the rx7620 server HBA ports and each of the EVA8000 hostports.

DL580 server SAN connectivity to EVA8000 storage arrayZones were established for two of the four DL580 server HBA ports and each of the EVA8000 hostports.

rx7620 server SAN connectivity to EVA5000 storage array for disk-based backupsZones were established for four of the rx7620 server HBA ports and each of the EVA5000 hostports.

DL580 server SAN connectivity to EVA5000 storage array for disk-based backupZones were established for two of the four DL580 server HBA ports and each of the EVA5000 host

ports.

rx7620 server SAN connectivity to VLS6510Zones were established for four of the rx7620 server HBA ports and each of the VLS6510 hostports.

DL580 server SAN connectivity to VLS6510Zones were established for two of the four DL580 server HBA ports and each of the VLS6510 hostports.

rx7620 server SAN connectivity to EML E-Series 103e Tape LibraryZones were established for two of the rx7620 server HBA ports and each of the EML E-Series 103ehost ports.

DL580 server SAN connectivity to EML E-Series 103e Tape LibraryZones were established for two of the four DL580 server HBA ports and each EML E-Series 103ehost ports.

9


10/45

Configuring the EVA8000 storage array for primary storage

The EVA8000 configuration included the following:

EVA VCS Microcode: 5.110An EVA8000 Controller Pair 12 EVA disk shelves 144 300-GB FC disks Three-phase 208-VAC redundant powerThe fiber connections were connected to two Brocade-based HP 2/16N SAN switches and twoBrocade Silkworm 3800s and configured in dual fabrics.

The EVA8000 presented nine RAID1 virtual disks, which were all from a single disk group. The diskgroup included all 144 available FC disks and was configured for double disk failure protection. Thevirtual disks were presented to all host ports that were connected to any port of the EVA. HP/QLogicload balancing driver used the LRU policy for load balancing.

Each used host port was identified on the EVA and the OS type was set to Linux. Each virtual diskfrom the EVA had Preferred Path/Mode set to No Preference and two were set to Path BFailoverOnly, alternating. This equally divided the load across the two controllers according to each host

and LUN mapping.

Configuring the EVA5000 for disk backups

The EVA5000 configuration included the following:

EVA VCS Microcode: 4.001An EVA5000 Controller Pair Eight EVA disk shelves 56 250-GB FATA disks Three-phase 208-VAC redundant powerThe EVA configuration consisted of one HSV110 controller pair and eight disk enclosures, whichwere populated with 56 250-GB FATA drives. Firmware v4.001 (the latest release at time of printing)and v3.028 (previous release) were used for the EVA VCS Microcode.

The EVA5000 presented four RAID0 virtual disks, which were all from a single disk group. The diskgroup included all 56 available FATA disks and was configured for no disk failure protection. Thevirtual disks were presented to all host ports that were connected to the EVA. The HP/QLogic loadbalancing driver was used with the least busy policy for load balancing.

Each used host port was identified on the EVA and the OS type was set to Linux. Two virtual disksfrom the EVA had Preferred Path/Mode set to Path AFailover Only and two were set to Path BFailover Only, alternating. This equally divided the load across the two controllers according to each

host and LUN mapping.

10


11/45

Configuring the HP StorageWorks EML E-Series 103e Tape Library

The HP StorageWorks EML E-Series 103e Tape Library was configured with four FCHP StorageWorks Ultrium 960 tape drives (LTO3).

The e2400-FC Interface Controllers had six FC ports. Four ports were for the back-end tape devices,and the remaining two were for the SAN. All interfaces were 2 GB and each SAN port on theInterface Controller was connected to separate fabrics to distribute the load evenly across fabrics andHBAs.

The tape library was managed from a dedicated SAN Management Server. HP StorageWorksCommand View TL software was installed on the same SAN Management Server used forHP StorageWorks Command View EVA.

Symantec NetBackup 6.0 was used as the backup application and Maintenance Pack 2 (MP2) wasapplied. For clarification, the global catalog server managed the backup images and the mediawhere the images reside. Since there can only be one host per robotic device, NetBackup elects oneof the hosts to be the robotic control host. The robotic control host moves the media to the tape driveswhen backups or restores are activated. Each server in the environment was configured as aNetBackup media server and was responsible for writing data directly to the tape devices. This allowsnetwork backups to be avoided.

Configuring the HP StorageWorks 6510 Virtual Library System

The HP StorageWorks 6510 Virtual Library System (VLS6510) was configured as an Ultrium 960tape library with 50 LTO2 tapes slots and 12 tape drives.

The VLS Interface Controllers had four FC ports and four SCSI ports. The four SCSI ports were for theback-end HP StorageWorks 20 Modular Smart Array (MSA20) disk devices, while the four FC portswere for SAN connectivity. All FC interfaces were 2 GB and each set of FC ports on the VLS InterfaceController was connected to separate fabrics to distribute the load evenly across fabrics and HBAs.

The tape library was managed from a dedicated SAN Management Server. HP StorageWorksCommand View TL software was installed on the same SAN Management Server used for

HP StorageWorks Command View EVA.Symantec NetBackup 6.0 was used as the backup application and MP2 was applied. Forclarification, the global catalog server managed the backup images and the media where the imagesreside. Since there can only be one host per robotic device, NetBackup elects one of the media serverhosts to be the robotic control host. The robotic control host moves the media to the tape drives whenbackups or restores are activated. Each server in the environment was configured as a NetBackupmedia server and was responsible for writing data directly to the tape devices. This allows networkbackups to be avoided.

11


12/45

Configuring the software

For the complete list of software, seeAppendix A. Bill of Materials.

Kernel tuning was applied to accommodate for the Oracle databases running on the hosts. Table 1lists the Linux kernel 2.6 parameters that were modified for this testing. These settings were used asbest practices based on information provided from a previous project. The default values have beenincluded for convenience.

Table 1. Altered kernel parameters

Tunable Default Used

net.core.rmem_default 110592 262144

net.core.rmem_max 131071 262144

kernel.sem 250 32000 32 128 250 32000 100 128

kernel.shmall 2097152 209715200

kernel.shmmax 33554432 24064771072

kernel.shmmni 4096 16384

fs.file-max 232233 658576

fs.aio-max-nr 65536 65535

net.ipv4.ip_local_port_range 32768 61000 1024 65000

vm.swappiness 10 30

Configuring the QLogic driver

Setting up QLogic Dynamic Load Balancing

The Linux QLogic driver supports active-active configuration and an active-passive configuration. TheEVA8000 storage array was configured to load balance across HBAs and controllers with Active-

Active enabled. The EVA5000 required an active-passive configuration but supports load balancingacross different ports of the same controller. The latest QLogic/HP driver can be obtained from the HPwebsite under support and downloads.

EVA8000 and EVA5000Active-Active/Active-Passive

When using the QLogic driver for Linux with the EVA product line, a set of configuration utilities wasinstalled with the driver source. An initial ramdisk (initrd) was created as port of the post-installationtasks of the package. To manually configure the driver options, the hp_qla2300.conf file in /etc wasmodified. Figure 2 shows changes made to the defaults.

12


13/45

The values for the load_balancing parameter are shown in Table 2:

Table 2. QLogic driver load_balancing parameter descriptionsType Policy Description

(0) Static None Finds the first active path or the first active optimized path for each LUN.

(1) Static Automatic Distributes commands across the active paths and available HBAs suchthat one path is used per LUN. Paths are automatically selected by driversfor supported storage systems.

(2) Dynamic Least Recently Used(LRU)

Sends command to the path with the lowest I/O count. Includes specialcommands such as path verification and normal I/O.

(3) Dynamic Least Service Time(LST)

Sends command to the path with the shortest execution time. Does notinclude special commands.

Figure 2. The /etc/hp_qla2300.conf file

qdepth = 16

port_down_retry_count = 30

login_retry_count = 30

failover = 1

load_balancing = 2

auto_restore = 0x80

A value of 2 was used for Least Recently Used (LRU). In this way, the paths selected were balanced

across the HBAs and switches automatically for the EVA8000 and load balanced across HBAs to thesame controller within the same switch for the EVA5000. OCFS2

OCFS is the Oracle Clustered File System and was required when using the RAC configuration wherea file system must be employed. OCFS provided a Distributed Lock Management (DLM) facility tocoordinate writes to the files contained within. OCFS version 2 was used for this environment and iscurrently the only supported version of OCFS.

NoteOCFS, ASM, and RAW devices can all be used with RAC depending on

platform and business requirements.

13


14/45

OCFS disk configuration

The Virtual Disks, VDisks, are zoned so both hosts have visibility to all LUNs for use with RAC. Thesingle LUN containing u06 and u07 are the flash recovery area and Voting and CRS files,respectively. Each of the LUNs has a single partition, which is then formatted OCFS. Figure 3 showsthe hosts and LUN visibility. For OCFS format information, see Figure 4.

Figure 3.Shared OCFS2 LUN presentation to RAC nodes

14


15/45

Setting up the OCFS Clustered File Systems

Each of the five EVA virtual disks was formatted as an OCFS file system with the defaults listed inFigure 4.

Figure 4. Viewing OCFS formatted devices in the OCFS2 console

The OCFS2 Console can simplify the formatting and configuration of the OCFS filesystems and helpsset up lock management and OCFS heartbeat. The cluster size was set at 64 K with a blocksize of

4 K. Likewise the command line is still valuable to mount OCFS filesystems. If this is a RAC and the filesystem will be used to store the Voting Disk file (CRS), Oracle Cluster Registry (CRS), Data files, Redologs, Archive logs or Control files, the file system should be mounted with datavolume and nointroptions. For example:

# mount o _netdev,datavolume,nointr /dev/cciss/c0d7p1 /data

The default /etc/fstab entries were modified to include the _netdev and datavolume options:

LABEL=CFT106_DATA1 /u02 ocfs2 _netdev,datavolume,nointr 0 0

The datavolume option is necessary when putting data files on an OCFS volume. Prior versions of

OCFS would not allow data files to be placed on OCFS volumes. The only allowed file types were theshared Oracle Home. The datavolume option was not added until OCFS version 2.

NoteAll of the OCFS2 file systems need to have the _netdev option specified.This guarantees that the network is started before mounting the file systemsand unmounted before the network is stopped.

15


16/45

For more information, see the OCFS2 Users Guide at:http://oss.oracle.com/projects/ocfs2/dist/documentation/ocfs2_users_guide.pdf

Working with Benchmark Factory

Benchmark Factory 5.1 Beta3 was used to conduct the workload generation. Benchmark Factory is ageneric workload generation utility that can perform DSS and OLTP workloads. Benchmark Factorycan also perform a custom workload based on any trace generated by a database or any customSQL used.

The workloads used for OLTP testing in accordance with the industry standard are defined in Table 3.

Table 3. Benchmark Scale Factors

Host Size User Load

IA64 RAC 2.4 TB 1,000

IA64 Singles 1.5 TB 500

IA32 Multi 750 GB 500

After creating the database, the Oracle spfile parameters were tuned to generate best

performance for backup during workloads. Refer to Table 5 for the specific Oracle parameters thatwere used during the testing. These are the optimal settings for this environment.

Benchmark Factory scale factors are approximate and should not be used as absolute guides. Thefollowing example should make approximately 930 GB of data when in fact this scale factor created1.3 TB of data and 300 GB of indexes. Although this seems somewhat greater than the target size, itdoes not include the actual indexes since size will vary depending on database block size.

16
http://oss.oracle.com/projects/ocfs2/dist/documentation/ocfs2_users_guide.pdfhttp://oss.oracle.com/projects/ocfs2/dist/documentation/ocfs2_users_guide.pdf


17/45

Figure 5. Benchmark scale factors

The scale factor shown is an estimate as stated. The actual size of the database must include indexesas well. This may be as much as 33% of the total data size after the data has been completelygenerated. The generated data alone may be as much as 20% greater than the estimated size.Table 4a and Table 4b show the table and index parameters, respectively.

Table 4a. Benchmark Factory Table/index settings

Object Type Creation Parameters

C_ORDER_LINE Table tablespace bmf parallel (degree default instances default) nologging cache monitoring

C_STOCK Table tablespace bmf parallel (degree default instances default) nologging cache monitoring

C_DISTRICT Table tablespace bmf parallel (degree default instances default) nologging cache monitoring

C_CUSTOMER Table tablespace bmf parallel (degree default instances default) nologging cache monitoring

C_HISTORY Table tablespace bmf parallel (degree default instances default) nologging cache monitoring

C_ORDER Table tablespace bmf parallel (degree default instances default) nologging cache monitoring

C_ITEM Table tablespace bmf parallel (degree default instances default) nologging cache monitoring

C_WAREHOUSE Table tablespace bmf parallel (degree default instances default) nologging cache monitoring

C_NEW_ORDER Table tablespace bmf parallel (degree default instances default) nologging cache monitoring

17


18/45

Table 4b. Benchmark Factory Table/index settingsObject Type Creation Parameters

C_STOCK_I1 Index tablespace bmf parallel (degree default instances default) nologging compute statistics

C_WAREHOUSE_I1 Index tablespace bmf parallel (degree default instances default) nologging compute statistics

C_NEW_ORDER_I1 Index tablespace bmf parallel (degree default instances default) nologging compute statistics

C_CUSTOMER_I2 Index tablespace bmf parallel (degree default instances default) nologging compute statistics

C_ORDER_LINE_I1 Index tablespace bmf parallel (degree default instances default) nologging compute statistics

C_ORDER_I1 Index tablespace bmf parallel (degree default instances default) nologging compute statistics

C_ITEM_I1 Index tablespace bmf parallel (degree default instances default) nologging compute statistics

C_DISTRICT_I1 Index tablespace bmf parallel (degree default instances default) nologging compute statistics

C_CUSTOMER_I1 Index tablespace bmf parallel (degree default instances default) nologging compute statistics

Oracle parameter changes

Table 5 lists the parameter changes that were made and the defaults. These changes were made toaccommodate for the maximum user load and performance during workloads while backups wereoccurring.

Table 5. Changed Oracle parameters

Oracle parameter Default Used

sort_area_size 65536 262144

parallel_max_servers 15 2048

parallel_threads_per_cpu 2 8

db_files 200 1024

Processes 150 1250

Dbwr_io_slaves 0 4

Tape_io_slaves False True

db_file_multiblock_read_count 8 128

Cusor_sharing Exact force

OLTP workload results

After the databases were populated with Benchmark Factory, the OLTP workloads were begun. Thefollowing results show the impact to the OLTP workload generated against each database byBenchmark Factory. This workload is an industry-standard workload, and user levels exercised are500 and 1,000 users, depending on the system. Table 6 shows the transactions rates without a

backup running, OLTP Baseline, and with the backup running.

Table 6. OLTP workload impact by backup method

Host type Databasesize (GB)

OLTPbaseline(TPS)

Disk backup(TPS)

VLS backup(TPS)

EML backup(TPS)

User levels

IA64 RAC 3090 49.5 49.5 49.5 49.45 1000

IA64 Single 1373 24.86 24.5 24.5 24.35 500

IA32 Multi 750 22.38 16.7 21.83 21.8 500

18


19/45

The IA64 RAC system performed best providing no negative impact to users regardless of backupactivity.

The IA64 Single system also shows there would be almost no impact to users and is operating at halfthe transaction rate of the entire RAC.

The IA32 Multi system also shows a flat performance curve with the exception of the EML backup,which dropped by approximately 6 TPS. This was mainly due to the I/O wait incurred by the system,which impacted the TPS. This could be eliminated by adding more RAM or more HBAs. The next

section outlines the backup and restore performance of each methodology in contrast to the OLTPworkload.

Oracle backup and restore

The major goal for the project was to back up the Oracle databases from each server directly to thetape, virtual tape, and disk devices. The backups were conducted in two scenariosthe first without asimulated user load and the second with load. During the first scenario, the database had noworkload applied and the backup was performed to disk, tape, and virtual tape. During the secondscenario, the database was put under a peak OLTP workload and then backed up to disk, tape, andvirtual tape. This was done to observe the interaction between the workload and the backup tounderstand what impact the backup would have on the client experience, and vice versa. A total of200 data files were backed up using RMAN through NetBackup.

A combined total of approximately 5 TB backed up from each server using each of the methodologiesin this paper. Data was sent to multiple drives in multiple streams, where media allowed, to achieveas high a throughput as possible for each configuration. One channel, or stream, was configured pertape device and two streams per disk device (also known as multiplexing) during the backup.

Restores from each backup device were conducted to observe the performance of each methodology.The most important metric captured was the actual speed of the restore as a factor of the overall Time-To-Recover (TTR), which allowed the possible performance within an environment for each of thebackup technologies to be gauged.

NetBackup policiesNetBackup policies determine the way a backup is executed. While each of the backup policies hassimilar attributes, a template or script is the ultimate influence as to how the backup will beperformed. Since the Policy Type used is Oracle, the EML, VLS, and disk backups all share commonpolicy settings. Policy settings used include:

Policy settings Policy TypeSet to Oracle Policy Storage UnitSelect the appropriately configured storage unit for the Volume Pool Policy Volume PoolSet to the volume pool associated with the storage unit device

ScheduleTwo schedule types must exist if automated backups are to work, an Automatic Backuptype and an Application Backup type. The defaults are Full and Default-Application-Backup.

ClientsThis is a list of clients to be backed up by this schedule. At least one client must be definedon an active policy.

Backup SelectionsThis can be a template or script.Multiplexing cannot be set in the Oracle Policy Type since RMAN already performs a level ofmultiplexing and can be specified in the template.

19


20/45

Oracle templates

On the NetBackup Media Server you can create a template using the Java NetBackup AdminConsole as root or a defined administration user:

# /usr/openv/netbackup/bin/jnbsa

When the admin console starts, select the Backup Files tab and then select the checkbox next to the

database name. A dialog box will appear and ask for database connection information. Whenlogged in, the Backup button as well as the rest of the database objects that can be backed up willdisplay. Selecting the entire database implies a database backup, but granular objects can beselected as well.

Clicking the Backup button starts the Backup Wizard. The following information must be provided:

Authentication Select System or Oracle Select RMAN catalog if you are using a catalog

Archived Redo Logs Include archived redo logs and specify the range, if any Delete archived logs after they are backed up, if desired

Configuration Default Existing template

Backup options Backup file name formatSet the format you want to use for backup files Backup set identifierSet the identifier you want the template to employ, if any

Database state OnlineOffline

Configuration variables Backup policy nameAn Oracle policy name on the EMM server Schedule nameA schedule name of type Application Backup on the EMM server Server nameThe EMM server Client nameThe Media Server or remote Oracle client system

Backup limits RMAN defaultsThe currently configured RMAN defaults Specify maximum limits

I/O limitsRead Rate, size of backup piece, number of open files

Backup Set LimitsFilesPerSet, MaxSize, ArchivelogMaxSize

I/O outputNumber of streams (channels) to configure for the backup

The last step is to run the backup, save the backup to a template, both, or cancel. Figure 6 shows theinitial screen after entering the database credentials.

20


21/45

Figure 6. Java NetBackup Admin Console

Setting up the storage unitsIn NetBackup, the storage unit defines the relationship between the host and its storage device. Thestorage unit is where each of the database servers was selected as the hosts for the backup devices.Each of the hosts shares tape devices, and the storage units are modified per host and are treated asseparate storage units. The disk devices are presented explicitly to each host and are truly separate inevery respect.

On the tape storage unit, select Maximum concurrent write drives and Maximum streams perdrive. Maximum concurrent write drives tells the host how many drives may be usedsimultaneously, and Maximum streams per drive tells the host how many streams can besimultaneously sent to any one drive if multiplexing is desired. During these tests only Maximum

concurrent write drives was used and set to a value equaling the number of available drives per tapestorage unit.

On the disk storage unit, set the path in Absolute pathname to directory and Maximum concurrentjobs for the device. You may also select the Enable temporary Staging Area if you want to enablescheduled disk staging. You must also create a schedule for staging with the provided menu button.This storage unit is created when using the Device Configuration wizard within NetBackup. HP andSymantec recommend that the Device Configuration wizard always be used to create tape librariesand their associated devices.

Set the master server jobs global attribute

In the master server host properties, the Maximum Jobs Per Client must be set to avoid job

contention at the master server. This is the total number of active jobs that can run concurrently at anyone time. Since the maximum devices configured for a library is 12, this value had to be set to aminimum of 12, but could be set higher if required.

21


22/45

Backup issues

Since each environment is different, know that issues may arise and require patches or workaroundsto allow proper backup execution. One example encountered was shortly after beginning tests:archivelogs would not backup properly after full backups. Expiring all media and performing thebackup again would allow the archivelogs to be backed up once, but not after. Since this is not anoptimal situation, applying MP2 was required to fix the issue. For other issues, seeAppendix D. Otheissues.

Setting up restores

Three options exist when performing restores to the Oracle database servers in this environment.

Create a restore job on the Media Server with the Java NetBackup Admin Console. Manually create a script on the Media Server and execute the restore with RMAN at the command

line.

Copy a backup template and modify the RMAN script portion to perform restores instead ofbackups. You can then create a backup policy but use the restore template.

For files backed up on one machine that were to be restored to another machine, a parameter mustbe set in the bp.conf file located in the /usr/openv/netbackup directory. The line to be added mustbe in the following format:

FORCE_RESTORE_MEDIA_SERVER = backup_server,restore_server

Where the backup_server is the host name of the server that performed the backup and therestore_server is the host name of the server that needs to perform the restore.

RMAN and NetBackup automated the restore process of de-multiplexing files and mounting therequired backup images, regardless of media type. This is especially helpful with disk backups anddisk staging.

Note:For the restore to be successful, the controlfile from the last backup must beavailable or you must have a RMAN catalog configured.

Backup and restore performance results

The testing effort provided the following results:

Successfully backs up at the rate of approximately 1 TB per hour Successfully recovered the database after simulated catastrophic data corruption

22


23/45

Backup methodologies

Disk-to-disk backupsBacking up Oracle 10gR2 to EVA5000

For this test, RMAN performed a tape backup and NetBackup translated the data streams into filesand wrote them to the defined Disk Storage Units. The I/O load was balanced across HBAs andcontroller ports.

Disk-to-virtual tape backupsBacking up Oracle 10gR2 to VLS

For this test, RMAN performed a tape backup using 12 streams to the VLS. The I/O load wasbalanced across HBAs and controller ports.

Disk-to-tape backupsBacking up Oracle 10gR2 to EML

For this test, RMAN performed a tape backup using four streams to the EML. The I/O load wasbalanced across HBAs and controller ports.

EVA performance results

EVA5000 RAW performance characterization

Raw write testing of the EVA5000 was performed with dd utility, the low-level UNIX tool that onlydoes sequential I/O with a modifiable block size. This testing provided a baseline performance for

the tested configuration. The raw performance of the EVA5000 being accessed from a host was ableto be shown. An example of a dd command is:

dd if=/dev/zero of=/backup1/test.fil bs=256K

The RAW sequential largeblock performance yielded approximately 190 MB/sec for a single port ona single HSV110 Controller. This performance doubled to 380 MB/sec for two simultaneous ddcommands to two different LUNs separated by HSV110 Controllers.

EVA8000 RAW performance characterization

Raw write testing the EVA8000 was performed with dd utility. An example of a dd command is:

dd if=/dev/zero of=/u02/test.fil bs=256K

The RAW sequential largeblock performance yielded approximately 190 MB/sec for a single Vdiskusing multiple HBAs and Controllers. This performance doubled to 380 MB/sec for two simultaneousdd commands to two different LUNs. The I/O load was balanced across HBAs and Controller HostPorts to avoid I/O bottlenecks.

EVA5000 backup and restore results

The following results show the speed of the disk-to-disk backups. These results were generated byperforming a disk backup with a NetBackup block size of 256 KB with and without a workload

applied.

Table 7. Backup results for disk-to-disk backups using EVA5000, VCS 4.001, with OLTP load

Host type Database size(GB)

Backup LUNs Backup time(hrs:min)

Channels Backup rate(GB/hr)

IA64 RAC 2400 2 7:06 12 338.03

IA64 Single 1320 1 4:10 12 316.98

IA32 Multi 600 1 7:42 8 78.43

23


24/45

At first these results might not mean much, but if you examine the backup time, you can understand ifthis meets the backup window requirements. The target of approximately 1 TB per hour was missed.For comparison, Table 10 shows the restore performance for this EVA5000 configuration.

Table 8. Backup results for disk-to-disk backups using EVA5000, VCS 4.001, without OLTP loadHost type Database size

(GB)Backup LUNs Backup time

(hrs:min)Channels Backup rate

(GB/hr)

IA64 RAC 2400 2 5:47 12 415

IA64 Single 1320 1 3:19 12 400

IA32 Multi 600 1 4:15 8 140

The IA64 systems performed best overall but the target of approximately 1 TB/hr was missed. Whencontrasted with the results during OLTP workload, you can see that performing a backup during peakhours is not advised if trying to meet specific backup times. One of the issues causing the results inboth of these backups is the firmware. Table 9 shows the same type of backup but with VCS 3.028and cache mirroring disabled for the backup LUNs.

Table 9. Backup results for disk-to-disk backups using EVA5000, VCS 3.028, without OLTP load

Host type Database size

(GB)

Backup LUNs Backup time

(hrs:min)

RMAN channels Backup rate

(GB/hr)

IA64 RAC 3090 2 4:24 12 702

IA64 Single 1373 1 2:46 12 508

IA32 Multi 750 1 2:10 8 245

Using the backrev firmware improved backup times dramatically, nearly doubling the throughput foreach system. The largest contributing factor was turning off controller cache mirroring.

Table 10. Restore results for disk-to-disk backups using EVA5000, VCS 4.001, offline restore


Backup LUNs Restore time(hrs:min)

RMAN channels Restore rate(GB/hr)

IA64 RAC 2400 2 5:54 12 406.78

IA64 Single 1300 1 2:05 12 624.00

IA32 Multi 750 1 7:36 8 87.21

The two IA64 systems performed best during restores, and much better than the IA32 system overall.The RAC system did not seem to perform as well on the restore as the Single instance database. TheRAC system had two factors causing these results.

The RAC system had imbalanced backups because of NetBackup. The disk I/O was shared on the same HBA for backups.The IA32 system performed poorly due to two similar factors:

The disk I/O was shared on the same HBA for backups. Bigfile tablepsaces were used.The side effect of these were long backups and restores using only one channel because of the limitedcapabilities of the hardware and the bigfile tablespace used to store the entire 150-GB database.

24


25/45

From these results, you can conclude:

Using bigfile tablespaces can hamper overall restore results if you do not have enough tablepsacesto spread across multiple devices.

Using 4.001 or higher code and RAID1 can yield the best protection from failures by eliminatingthe capability to turn off cache mirroring and not taxing controllers with RAID5 parity overhead.

Using 3.028 firmware and RAID0 LUNs yields the best performance by allowing you to turn offcache mirroring, but eliminates any possible recovery from failures.

Properly balancing datafiles/tablespaces to RMAN channels and tape devices yields the bestresults.EML E-Series backup and restore performance results

The following results show the speed of the disk-to-tape backups. These results were generated byperforming a tape backup with NetBackup using a block size of 256 KB with and without a workloadapplied.

Table 11. Backup results for disk-to-tape backup using EML 103e, with OLTP load



RMAN channels Backup rate(GB/hr)

IA64 RAC 2400 2 5:48 4 413.79

IA64 Single 1300 1 2:41 4 484.47

IA32 Multi 600 1 5:07 4 117.26

The two IA64 backups were similar in their rates. There is still an imbalance due to NetBackup, but itis more streamlined on the system bus and there is no contention from the EVA5000 cache. The causeand fix for this imbalance is discussed inAppendix C. Examples.

The IA32 system still has fairly low performance overall, but better than disk-to-disk. This was becauseone of the limitations was removed by backing up from one FC Port to two separate FC Ports.Congestion at the system bus or EVA5000 controller cache was not an issue. Table 12 shows backup

results for the EML while not under load.Table 12. Backup results for disk-to-tape backup using EML 103e, without OLTP load



RMAN channels Backup rate

(GB/hr)

IA64 RAC 2400 2 5:24 4 441.72

IA64 Single 1300 1 2:13 4 586.67

IA32 Multi 600 1 3:20 4 180

25


26/45

The performance characteristics of this test again look different than when performing a backup underload. Each of the backups for the servers had better performance than the previous backup results inTable 11. This again reinforces the fact that a backup while under heavy load is not advised. TheIA32 Multi system saw the greatest percentage increase in performance, though it is not at the level ofthe IA64 systems.

Table 13. Restore results for disk-to-tape backup using EML 103e, offline restore




IA64 RAC 2400 2 3:25 4 702.44

IA64 Single 1300 1 1:42 4 764.71

IA32 Multi 600 1 2:25 4 248.28

Each of these results is fairly good for each server. Each of the restores is done in an offline manner,so each of the servers has more resources available for the restore than they would for the backup.The IA32 system could yield even higher performance if more bigfile tablespaces were employed, orsmallfiles were used for the single 150-GB tablespace.


Tape backup is very sensitive to the layout of the data paths and device visibility.As long as you have proper size files to back up, tape streaming can perform at decent levels of

I/O.

Tape restores are fast, while backups are highly impacted if poor channel balancing occurs.VLS6510 backup and restore performance results

The following results show the speed of the disk-to-tape backups. These results were generated byperforming a tape backup with NetBackup using a block size of 256 KB with and without a workloadapplied.

Table 14. Backup results for Disk to Tape backup using VLS 6510, w/out OLTP load.


Backup LUNs Backup Time(hrs:min)


IA64 RAC 2400 2 2:36 12 923.08

IA64 Single 1320 1 1:44 12 761.54

IA32 Multi 425 1 3:20 8 196.15

The backup results show the high throughput the VLS can deliver. The approximate 1-TB/hr target onthe RAC system is met. Each system had great improvement over disk and tape though duringbackup.

26


27/45

The VLS6510 allows current tape users to enjoy the speed of disk. Since the VLS is emulating any typeof tape device and multiple libraries, you could use this system for consolidation when moving to onetype of tape media and devices.

Table 15. Backup results for disk-to-tape backup using VLS6510, with OLTP load




IA64 RAC 2400 2 2:44 12 883.44

IA64 Single 1320 1 2:43 12 484.47

IA32 Multi 425 1 3:45 8 113.21

Even though these backup results are also very close to the approximate 1-TB/hr target, there is alarge decrease in throughput during workloads. The single largest percent decrease is seen by IA32Multi, with the IA64 Single following closely behind. This further reinforces that backups during peakworkloads should be avoided. Table 16 shows the VLS restore results.

Table 16. Restore results for disk-to-tape backup using VLS6510, without OLTP load




IA64 RAC 2400 2 9:25 12 254.87

IA64 Single 1320 1 2:46 12 477.11

IA32 Multi 600 1 1:48 8 333.33

These results show some startling results. The first of which is the low performance of the restore onIA64 RAC when compared to the backup results. The reason this occurred is the backup imbalancementioned earlier in the paper. When a scripted backup is performed ensuring no RMAN scriptmodification by NetBackup, these results would be similar instead of highly contrasted as seen here.

The backup performance masks the issue but the restore is impacted. The same is true for the IA64Single system, but not to as high a degree. Because a single LUN on IA64 Single is being written to,

there is not as great an impact. Because IA64 RAC is restoring to two LUNs, tape channels are beingwaited on to complete and so the restore finishes with a lesser degree of parallelism than seen onIA64 Single.

The IA32 Multi performed very well for the VLS restore considering that bigfile tablepsaces werebeing used and were again serialized. If several smallfiles or many bigfile tablespaces were to beused, this restore could be improved.


Backups are sensitive to the layout of the data paths, device visibility, and the ratio of the number offiles to RMAN device channels.

The VLS is very easy to use and maintain allowing administrators to incorporate a speedy methodof backup into any tape environment. This allows shorter backup windows than traditional tape.

Configuring many emulated devices on the VLS is essential in achieving maximum performancefrom the VLS.

Using four MSA20 disk shelves, the maximum for the VLS6510, is a must where high throughputbackups are required on the VLS6510. It is also essential when emulating multiple types of libraries,tape devices, and media.

27


28/45

Conclusions

The data derived for the workload, backup, restore, and EVA5000 VCS tests provide anunderstanding to important things about each methodology discussed.

Oracle RMAN

Using few bigfile tablespaces can hamper overall restore results if you do not have enoughtablepsaces to spread across all backup devices.

Preventing poorly balanced channels will improve backup performance. Conducting backups under load is not advised where short backup windows are required.Disk-to-disk backup

Using 4.001 or higher code and RAID1 can yield the best protection from failures by eliminatingthe capability to turn off cache mirroring and not taxing controllers with RAID5 parity overhead.

Using 3.028 firmware and RAID0 LUNs yields the best performance by allowing cache mirroring tobe disabled, but has potential to corrupt the backup if a failure occurs.

Using the same HBA for disk backups can cause contention on the PCI bus resulting in slowerbackups.

The EVA5000 VCS 4.00x cannot disable use of the cache mirror port, which is used for more thanjust host LUN cache mirroring. The need for Active/Active versus Active/Passive capabilities shouldbe weighed before implementation.

Disk-to-tape backup

Tape backup is sensitive to the layout of the data paths and device visibility. Having proper size files to back up will allow tape streaming to perform well. Using OS and hardware tape buffering as well as enabling hardware compression with a proper

block size will allow the tape device to perform at its peak.

Tape restores are fast, while backups are highly impacted if poor channel balancing occurs.Diskto-VLS backup

Backups are sensitive to the layout of the data paths, device visibility, and the ratio of the number offiles to RMAN device channels.

The VLS is very easy to use and maintain allowing administrators to incorporate a speedy methodof backup into any tape environment. This allows shorter backup windows than traditional tape.

Configuring many emulated devices on the VLS is essential in achieving maximum performancefrom the VLS.

Using four MSA20 disk shelves, the maximum for the VLS6510, is a must where high throughputbackups are required on the VLS6510. It is also essential when emulating multiple types of libraries,tape devices, and media.

28


29/45

Server configuration

The DL580 server system was very loaded as a multi-database server. A DL580 server with morememory and HBAs would be advised if performing high levels of transactions to a server of thistype.

System bus contention should be avoided by ensuring servers have enough HBAs or PCI-X busses. If too much swapping occurs during backup, add more RAM to your system until swapping reduces

to acceptable levels.

The latest QLogic driver as of August 11, 2006, supports dynamic load balancing; earlier versionsdo not. This is important because servers can avoid possible I/O contention causing better backupand/or restore performance.

Best practices

During testing, several best practices were developed to improve backup and recovery performancefor each scenario.

Best practices for disk-to-disk backups on the EVA5000 storage array

Use RAID0 for backup target LUNs for best performance on VCS 3.028 firmware Use RAID1 for lowest overhead data protection on VCS 4.001 firmware Use VCS 3.028 firmware for higher performance by disabling cache mirroring for backup LUNs

NoteVCS 4.00x firmware includes several critical bug fixes and otherimprovements. Read the VCS 4.001 and 3.028 release notes carefullybefore choosing to use a down-rev firmware version.

Use as many disks as possible for a given Disk Group. Create Disk Groups of at least 56 FATAdisks for highest throughput per disk group

Use two or more LUNs for each host to spread data streams across controllers for improvedbandwidth

Best practices for disk-to-tape backups on the EML E-Series Tape Library

Several settings were set to achieve optimum performance. These settings should be tuned for thespecific environment, and when possible, validated first within a test environment.

The following is a list of the important tuning options used to achieve good performance during thebackup to tape.

Buffer configurationThe bulk of the NetBackup tuning was done at this level. There were threetouch files that were used within NetBackup to achieve better levels of performance, with respect tothe use of memory buffers. These files will typically be located in the

/usr/openv/netbackup/db/config directory. A document that explains buffer configuration forNetBackup can be found at http://seer.support.veritas.com/docs/183702.htm. Another recommendeddocument regarding buffer configuration for LTO3 and NetBackup can be found athttp://h71028.www7.hp.com/ERC/downloads/5982-9971EN.pdf.

29
http://seer.support.veritas.com/docs/183702.htmhttp://h71028.www7.hp.com/ERC/downloads/5982-9971EN.pdfhttp://h71028.www7.hp.com/ERC/downloads/5982-9971EN.pdfhttp://seer.support.veritas.com/docs/183702.htm


30/45

Block settingsThe OS level tape block settings were configured using the /etc/stinit.def file tospecify settings for the LTO tape devices. The following settings were used in the test environment:

NUMBER_DATA_BUFFERS: The number of buffers used by NetBackup to buffer data beforesending it to the tape drives. The default value is 16 and was set to 32.

SIZE_DATA_BUFFERS: The size of each buffer setup multiplied by the NUMBER_DATA_BUFFERSvalue. The default value is 65536 and was set to 262144.

NUMBER_DATA_BUFFERS_RESTORE: The number of buffers used by NetBackup to buffer databefore writing it to the disk. The default value is 16 and was set to 32. (optional)

Blocksize: This is a stinit.def setting. This is used by stinit to set each tape devices defaults. Asetting of 0 is used so that the blocksize is automatically determined at write time.

Drive-buffering: This is a stinit.def setting. This is used by stinit to set each tape devices defaults.Adding this to the stinit device definition will enable hardware buffering for the LTO3 tape device(this parameter can only be used if the drive is buffer capable).

Best practices for disk-to-virtual tape backups on the VLS Virtual TapeLibrary

Several settings were set to achieve optimum performance. These settings should be tuned for thespecific environment, and when possible, validated first within a test environment. The VLS6510 usesfour MSA20 disk arrays to emulate tape devices.

Buffer configurationThe bulk of the NetBackup tuning was done at this level. There were threetouch files that were used within NetBackup to achieve better levels of performance, with respect tothe use of memory buffers. These files will typically be located in the

/usr/openv/netbackup/db/config directory. A thorough document that explains bufferconfiguration for NetBackup can be found at http://seer.support.veritas.com/docs/183702.htm.

Another recommended document regarding buffer configuration for LTO3 and NetBackup can befound at http://h71028.www7.hp.com/ERC/downloads/5982-9971EN.pdf.

Block settingsThe OS level tape block settings were configured using the /etc/stinit.def file tospecify settings for the LTO tape devices.

Number of devicesThe number of devices emulated can contribute greatly to the overallperformance of the VLS. The best performing configuration is generally a 3 or 4:1 ratio of emulateddevices to MSA20 disk shelves attached to the VLS Interface Controller.

Following are the settings that were used in the test environment:

NUMBER_DATA_BUFFERS: The number of buffers used by NetBackup to buffer data before sendingit to the tape drives. The default value is 16 and was set to 32.

SIZE_DATA_BUFFERS: The size of each buffer setup multiplied by the NUMBER_DATA_BUFFERSvalue. The default value is 65536 and was set to 262144.

Blocksize: This is a stinit.def setting. This is used by stinit to set each tape devices defaults. Asetting of 0 is used so that the blocksize is automatically determined at write time.

Drive-buffering: This is a stinit.def setting. This is used by stinit to set each tape devices defaults.Adding this to the stinit device definition will enable hardware buffering if the drive is capable.

30
http://seer.support.veritas.com/docs/183702.htmhttp://h71028.www7.hp.com/ERC/downloads/5982-9971EN.pdfhttp://h71028.www7.hp.com/ERC/downloads/5982-9971EN.pdfhttp://seer.support.veritas.com/docs/183702.htm


31/45

Best practices for using Oracle Recovery Manager (RMAN)

Use a separate RMAN catalog databaseThis will help in managing your backups by providing redundancy to your NetBackup catalog andretain information otherwise backed up in control files. A catalog database also makes it possibleto restore image copies required if you choose to create an Oracle Data Guard physical standbydatabase, and can add protection against and simplify recovery of lost controlfiles.

Protect your RMAN repository, NetBackup catalog, or bothThe RMAN catalog should be backed up and redundant copies kept on separate media. Create aNetBackup catalog backup policy upon installation. Create a binary copy of the controlfile if notusing a RMAN catalog.

Enable Block Change Tracking to improve the speed of incrementalsThis is especially important when it comes to incremental backups. Without a BCT table anincremental level 1 backup will take as long as a full level 0.

Use flash recovery for online point-in-time recovery and transaction rollback trackingUsing flash recovery is great for online recovery and helps prevent the need to utilize backups for

recovery. Flash recovery should be part of your storage growth planning, if implemented, as it cantake a very large amount of storage. The actual storage required depends on the number of days ofdata and the database transaction rate. Configuring even a small flash recovery area will at leastensure you have a recent copy of a binary control file.

Choose effective backup policy typesIncorporating full and incremental backups should help ease recovery, but one policy does not fitall. Understanding how and when to use cumulative versus differential backups is important for yourrecovery strategy as well.

Maintain your Oracle backup images effectivelyUse Oracle 10gs incremental merge to create a current full image copy from incrementals.

Creating a full backup often to ensure recoverability, even with incrementals handy, does not meanyou can recover, unless you have all the archive logs since the previous full. If using disk backupsas the primary recovery option, move old on-disk backups to tape with the backup databasebackupset command or NetBackup Media Copy to manage space.

Test copies of backupsOne definite way of testing backups is to restore them to an alternate location and attempting tostart the restored database. The next method to check the backup is to use the RMAN validatecommand. Finally, use the RMAN crosscheck command to verify that backup, data, and archivelog files are still located on the target media.

Manage your archived online logs and keep them safeEnsure your archived logs are part of your full and incremental backup. Also, create duplicatecopies of media containing archive logs. Manage the archive log space by deleting archive logs aspart of the backup.

If you cannot use RMAN, NetBackup can helpNetBackup can help your backup strategy by managing online backups with a single point ofmanagement. NetBackup can also provide binary block level incremental backup and restores andmay be slower than RMAN, but the option is available. For additional RMAN configurationinformation, seeAppendix B. Configuring Oracle RMAN.

31


32/45

Appendix A. Bill of Materials

Backup Server 1

Operating system Red Hat Linux AS4 U3

Multi-path solution QLogic Driver Dynamic Load Balancing

Backup Software

Solution

Symantec NetBackup Enterprise Server

6.0 (MP2)

MP E.03.13

BMC 03.47

EFI 03.10

HP Integrity rx7620 server 1

System 03.11

1.6-GHz CPU 8(4 per partition)

GB Memory 64 (32 per partition)

Firmware version: 3.03.150A6826A (dual port HBA)

8 (4 per partition)

Driver 1.42

Backup Server 2

Operating system Red Hat Linux AS4 U3

Multi-path solution QLogic Driver Dynamic Load Balancing

Backup SoftwareSolution

Symantec NetBackup Enterprise Server6.0 (MP2)

HP ProLiant DL580 G2 server 1

3.0-GHz CPU 4

GB Memory 8

Firmware version: 3.21FCA2214 (dual port HBA)

2

Driver 1.45

StorageEVA8000

EVA8000 (2C12D) 1 V5.110

300-GB FC Disk (NDSOMEOMER) 144 HP02

HP StorageWorks SAN 2/16N switches 2 V4.2.0c

Brocade SilkWorm 3800 SAN switches 2 V3.2.0a

HP OpenView Storage ManagementAppliance III

1 V2.1

32


33/45

Disk-to-disk backup targetEVA5000

EVA5000 (2C8D) 1 V4.001 and V3.028

250-GB FATA Disk (ND25058238) 56 HP01

Disk-to-virtual tape backup targetVLS6510

VLS6510 1 V3.020

250-GB SATA Disk (ND12341234) 48 HP02b

HP OpenView Command View TL 1 V3.2

Disk-to-tape backup targetEML 103e

EML E-Series 103e Tape Library 1 V3.020

Ultrium 960 LTO-3 drives (ND25058238) 4 HP01

33


34/45

Appendix B. Configuring Oracle RMAN

This section shows the recommended defaults for each RMAN instance as well as the configurationoptions for configuring the RMAN backup.

Figure 7. RMAN configuration defaults

CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default

CONFIGURE BACKUP OPTIMIZATION OFF; # default

CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default

CONFIGURE CONTROLFILE AUTOBACKUP OFF; # default

CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default

CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default

CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default

CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default

CONFIGURE MAXSETSIZE TO UNLIMITED; # default

CONFIGURE ENCRYPTION FOR DATABASE OFF; # default

CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default

CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default

CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u01/app/oracle/product/10.2.0/db_1/dbs/snapcf_ORDB1.f'; # default

You may want to modify some of the preceding defaults, in particular the Backup Optimization,Default Device Type, Controlfile Autobackup, Parallelism, and Archivelog Deletion Policy.

The following are examples of suggested changes to these settings:

Enable Backup OptimizationIf you plan to use several incrementals and merge them. Only changed blocks will be backed upsince the last backup. Not very useful for Full backups.

Default Device TypeThe default device type may need to be a tape library or worm drive, so setting this may relievesome scripting.

Controlfile AutoackupThis is highly useful to ensure a controlfile backup is done often.

ParallelismWhen writing backup sets, this will stream multiple files together to the same channel if set to avalue greater than one.

Archivelog Deletion PolicySetting this can ease management of scripts since you can set the archivelogs to be deleted at apredefined interval.

34


35/45

Appendix C. Examples

This section shows RMAN scripts examples, NetBackup templates, and NetBackup Screenshots.

Figure 8. RMAN Full Backup script (four channels configured)

RUN {

ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE';




SEND 'NB_ORA_CLIENT=PRIM1,NB_ORA_SERV=EMMSRV,NB_ORA_POLICY=PRIM1-EML,NB_ORA_PC_SCHED=Default-Application-Backup';

BACKUPINCREMENTAL LEVEL=0

FORMAT 'Data_Plus_Arch_%d_u%u_s%s_p%p_t%t'

TAG 'DB1 Full Standby Backup'

DATABASE PLUS ARCHIVELOG ;RELEASE CHANNEL ch00;

RELEASE CHANNEL ch01;



ALLOCATE CHANNEL ch00

TYPE 'SBT_TAPE';

SEND 'NB_ORA_CLIENT=PRIM1,NB_ORA_SERV=EMMSRV,NB_ORA_POLICY=PRIM1-EML, \

NB_ORA_SCHED=Default-Application-Backup';

BACKUPFORMAT 'STBYCTLFILE-_%d_u%u_s%s_p%p_t%t'

CURRENT CONTROLFILE FOR STANDBY;RELEASE CHANNEL ch00;

}

35


36/45

Figure 9. RMAN Duplicate script

run

{

# Auxiliary channels are the only way to restore a database as a duplicate

allocate auxiliary channel ch00 device type 'sbt_tape';




SEND 'NB_ORA_CLIENT=STBY1,NB_ORA_POLICY=STBY1-EML,NB_ORA_SERV=EMMSRV, \

NB_ORA_SCHED=Default-Application-Backup';

duplicate target database for standby;release channel ch00;

release channel ch01;



}

36


37/45

Figure 10. NetBackup 6 Oracle template

#^oracle template configuration file

# Template level: 1.9.0

# Generated on: 06/28/06 16:01:13

# -----------------------------------------------------------------

TEMPLATE_ID1=

TEMPLATE_ID2=

TEMPLATE_OWNER=root

RUN_AS_USER=oracle

# -----------------------------------------------------------------

# BACKUP_TYPE is derived from the schedule type when this script

# is used in a NetBackup scheduled backup. For example, when:

BACKUP_TYPE=INCREMENTAL LEVEL=0

ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1

ORACLE_SID=PRIM1

TARGETDB_LOGIN=sys

TARGETDB_PASSWD=

TARGETDB_TNSNAME=PRIM1

# -----------------------------------------------------------------

# RMAN command section

# -----------------------------------------------------------------

RUN {


SEND 'NB_ORA_CLIENT=Client1,NB_ORA_POLICY=Oracle-Policy, \ NB_ORA_SERV=EmmServer,NB_ORA_SCHED=Default-Application-Backup';

BACKUP

INCREMENTAL LEVEL=0

37


38/45

FILESPERSET 1

MAXOPENFILES 8

FORMAT 'bk_u%u_s%s_p%p_t%t'

DATABASE;


# Backup Archived Logs

sql 'alter system archive log current';



BACKUP

FORMAT 'arch-s%s-p%p-t%t'

ARCHIVELOG

ALL

DELETE INPUT;


# Control file backup

ALLOCATE CHANNEL ch00

TYPE 'SBT_TAPE';


BACKUP

FORMAT 'bk_u%u_s%s_p%p_t%t'

CURRENT CONTROLFILE;


}

38


39/45

Figure 11. NetBackup 6 Policy Properties Dialog

39


40/45

Figure 12. NetBackup Policy Pane

40


41/45

Figure 13. NetBackup 6 Storage Unit Settings

Disk Storage Unit

Tape Storage Unit

41


42/45

Figure 14. Stinit.def configurations for the EML and VLS

# HP Ultrium 960 LTO-3 devices on the EML E-Series 103e

manufacturer="HP" model="Ultrium 3-SCSI" revision="L29S"

{

scsi2logical=1 # Common definitions for all modes

can-bsr drive-buffering can-partitions auto-lock buffer-writes async-writes read-ahead compression

timeout=800

long-timeout=14400

mode1 blocksize=0 density=0x00

}

# HP Ultrium 960 LTO-3 devices emulated on the VLS 6510

manufacturer="HP" model="Ultrium 3-SCSI" revision="R138"

{

scsi2logical=1 # Common definitions for all modes

can-bsr drive-buffering can-partitions auto-lock buffer-writes async-writes read-ahead

timeout=800

long-timeout=14400

mode1 blocksize=0 density=0x00 compression=0

}

42


43/45

Appendix D. Other issues

This section has a set of general issues encountered during testing and a description of suggestedresolutions.

Server OS hangs/crashes

Issue: System hangs under high load.

Resolution: Upgrade from AS4 U1 to U3.

Oracle session hangs

Issue: Oracle instances would stop leaving sessions in hung state during high load.

Resolution: Upgrade Oracle to 10.2.0.2 patch.

NetBackup catalog synchronization

Issue: RMAN backups could not be reliably restored before the expire time.

Resolution: Upgrade NetBackup to MP2.

RMAN not backing up archive logs

Issue: RMAN would not back up archive logs at the end of a full backup.

Resolution: Upgrade NetBackup to MP2.

Resolution (if MP2): Ensure the NetBackup Oracle template has the proper schedule type as part ofthe SEND command for archive logs. It should be Default-Application-Backup or another schedulethat is of type Application Backup.

RMAN specific syntax changes

Issue: RMAN commands are modified before backup, based on specific arguments used.Argument 1: FilesPerSetMaxSetSize, Rate, MaxPieceSize rewritten to the RMAN script.

Argument 2: MaxOpenFilesMaxSetSize rewritten to the RMAN script.

Argument 3: MaxSetSizeRate, MaxPieceSize rewritten to the RMAN script.

Resolution: Set the explicit commands in the template, or create a script on the server and call itdirectly.

Imbalanced backups

Issue: Allocated channels do not back up data evenly resulting in overall decrease in performance of

the backup.Resolution: Use FilesPerSet, DiskRatio, MaxSetSize, or MaxPieceSize arguments to create morebalanced backupsets.

43


44/45

44

Poorly streaming backups

Issue: Backups to tape are not streaming to tape well.

Resolution 1: Check v$views: backup_async_io and backup_sync_io.

Resolution 2: Use the BlkSize parameter of RMAN.

Resolution 3: Adjust NetBackup parameters NUMBER_DATA_BUFFERS or SIZE_DATA_BUFFERS andobserve impact.

Resolution 4: Adjust MaxOpenFiles and/or FilesPerSet RMAN parameters and observe impact.

RAC issues

Issue: OCFS2 timeouts under load.

Resolution: Set default timeo

backup and recovery best practices for oracle 10g with netbackup 6.0 white paper

Documents