remote site recovery of oracle enterprise data warehouse using

38
White Paper EMC SOLUTIONS GROUP Abstract This white paper describes how a 12 TB Oracle data warehouse was transported from a data center in Massachusetts, U.S, to Cork, Ireland. The infrastructure was re-created at the Cork site on EMC ® Symmetrix ® VMAX™ storage and the database was restored and recovered. EMC Data Domain ® Encryption was used to secure the data during transport. July 2011 REMOTE SITE RECOVERY OF ORACLE ENTERPRISE DATA WAREHOUSE USING EMC DATA DOMAIN A Detailed Review

Upload: trannhi

Post on 11-Dec-2016

217 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

White Paper

EMC SOLUTIONS GROUP

Abstract

This white paper describes how a 12 TB Oracle data warehouse was transported from a data center in Massachusetts, U.S, to Cork, Ireland. The infrastructure was re-created at the Cork site on EMC® Symmetrix® VMAX™ storage and the database was restored and recovered. EMC Data Domain® Encryption was used to secure the data during transport.

July 2011

REMOTE SITE RECOVERY OF ORACLE ENTERPRISE DATA WAREHOUSE USING EMC DATA DOMAIN A Detailed Review

Page 2: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

2

Copyright © 2011 EMC Corporation. All Rights Reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

The information in this publication is provided “as is.” EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

All trademarks used herein are the property of their respective owners.

Part Number H8238.1

Page 3: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

3 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

Table of contents

Executive summary ............................................................................................................... 5

Business case .................................................................................................................................. 5

Solution overview ............................................................................................................................ 5

Key results ....................................................................................................................................... 6

Introduction .......................................................................................................................... 7

Purpose ........................................................................................................................................... 7

Scope .............................................................................................................................................. 7

Audience.......................................................................................................................................... 7

Terminology ..................................................................................................................................... 7

Technology overview ............................................................................................................. 8

Overview .......................................................................................................................................... 8

EMC Data Domain DD660 ................................................................................................................. 8

EMC Data Domain Encryption ........................................................................................................... 9

EMC Data Domain file system encryption lock .................................................................................. 9

EMC Symmetrix VMAX ...................................................................................................................... 9

EMC Solutions Enabler ..................................................................................................................... 9

EMC Symmetrix TimeFinder .............................................................................................................. 9

EMC PowerPath .............................................................................................................................. 10

Oracle Database ............................................................................................................................ 10

Solution overview ............................................................................................................... 11

Use case requirements ................................................................................................................... 11

The challenge................................................................................................................................. 11

Nondisruptive to production ...................................................................................................... 11

Distance and latency ................................................................................................................. 11

Database size ............................................................................................................................ 12

The solution ................................................................................................................................... 12

Project timeline .............................................................................................................................. 14

Discovery – primary site ...................................................................................................... 15

Overview ........................................................................................................................................ 15

Security .......................................................................................................................................... 15

Business and technical .................................................................................................................. 16

Primary data center configuration information................................................................................ 16

Remote site infrastructure .............................................................................................................. 17

Architecture ............................................................................................................................... 17

Hardware resources ................................................................................................................... 18

Software resources .................................................................................................................... 18

Page 4: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

4

Backup appliance ...................................................................................................................... 18

Oracle database backup – primary site ................................................................................ 19

Overview ........................................................................................................................................ 19

DD660 setup .................................................................................................................................. 19

Network settings ....................................................................................................................... 19

Encryption ................................................................................................................................. 20

RMAN backup ................................................................................................................................ 20

File system lock .............................................................................................................................. 20

Data compression results ............................................................................................................... 21

Oracle server deployment and storage provisioning – remote site ........................................ 23

Overview ........................................................................................................................................ 23

Deploying the remote site server .................................................................................................... 23

Configuring the remote site storage ............................................................................................... 23

Primary data warehouse configuration information .................................................................... 23

Provisioning storage at the remote site ...................................................................................... 24

Setting up the Oracle database environment.................................................................................. 25

Oracle database recovery – remote site ............................................................................... 27

Overview ........................................................................................................................................ 27

DD660 setup .................................................................................................................................. 27

Network settings ....................................................................................................................... 27

Unlock file system ..................................................................................................................... 28

RMAN restore ................................................................................................................................. 29

Verifying the integrity of the database ............................................................................................ 32

Data Domain consolidation – remote site ............................................................................ 34

Overview ........................................................................................................................................ 34

Replication process ........................................................................................................................ 34

Conclusion ......................................................................................................................... 37

Summary ....................................................................................................................................... 37

Findings ......................................................................................................................................... 37

References .......................................................................................................................... 38

White papers ................................................................................................................................. 38

Product documentation .................................................................................................................. 38

Other documentation ..................................................................................................................... 38

Page 5: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

5 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

Executive summary

EMC offers an extensive series of solutions for quick, efficient, and nondisruptive backup of large Oracle data warehouse environments. Increasingly, customers are looking to repurpose or reuse these backups to facilitate operations such as:

• Migration of Oracle data warehouses to new data center locations

• Seeding of Oracle data warehouse environments at remote disaster recovery locations

• Deployment of remote database copies for test and development purposes

These scenarios present challenges in relation to determining the most appropriate mechanism for transferring the data from the primary site to the remote site, and developing validated procedures for re-creating the production environment and recovering the database at the remote site.

The main considerations in relation to data transfer to the remote site include:

• Security and corporate governance compliance

• Distance and latency

• Database size

• Cost effectiveness

For one-off migration scenarios, where the data warehouse is many terabytes in size and the distance between data centers is great, it is impractical to transfer the data across the WAN. Consequently, customers are looking for an offline transport solution that is economical and secure.

At its headquarters in Hopkinton, Massachusetts, EMC IT maintains a 12 TB Oracle data warehouse, deployed on an Oracle Real Application Clusters (RAC) 11g environment. To demonstrate the benefits and implementation of the solution documented in this white paper, this real-world production database was cloned and backed up, shipped by air from the primary data center in Hopkinton, Massachusetts to the EMC Solutions Center in Cork, Ireland, and then restored and recovered for test and development use.

The remote data center infrastructure consisted of a two-node Oracle RAC 11g cluster that accessed the Oracle database on an EMC® Symmetrix® VMAX™ storage system. This reflects the infrastructure at the primary data center.

The production database in Hopkinton was cloned using EMC TimeFinder® and then backed up to an EMC Data Domain® DD660 appliance using Oracle Recovery Manager (RMAN). Data Domain Encryption and file system encryption locking were used to ensure security and corporate governance compliance during transport.

At the remote site, the DD660 was added to the environment and the file system was unlocked. The database was then restored and recovered to a test system using Oracle RMAN.

Business case

Solution overview

Page 6: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

6

After verifying that the database had been successfully restored, Data Domain Replicator was used to move the database image from the DD660 to a shared Data Domain DD880 appliance in the data center in Cork, so that the DD660 could be repurposed for another project.

The key benefits demonstrated by the solution include:

• Nondisruptive backup

Creating a TimeFinder clone has no impact on production. Mounting the clone to a separate backup server provides a fully-usable, independent, point-in-time copy that can be backed up to the Data Domain appliance with no disruption to the production environment.

• Reduced backup footprint and shipping costs

Data Domain deduplication reduces the footprint of the backup database significantly. In this solution, a 70 percent reduction in capacity requirements enabled use of the midrange DD660 appliance, which minimized shipping costs.

• Seamless Oracle RMAN integration

Data Domain systems integrate seamlessly into Oracle architectures, supporting existing backup strategies, including Oracle RMAN.

• Data security and corporate governance compliance

Data Domain Encryption ensures that all data is physically stored in an encrypted manner and cannot be accessed without first decrypting it. Data Domain file system encryption lock adds a further level of protection that enables shipping on commercial air transport while satisfying corporate governance compliance requirements.

• Validated and documented end-to-end procedures

Database integrity testing at the end of the project demonstrated successful recovery of the database at the remote data center.

This white paper documents the procedures for achieving these results, from the initial discovery phase necessary for designing the infrastructure for the remote site, to a complete restore and recovery of a working data warehouse. These procedures are applicable to any solution involving recovery of an Oracle database.

Key results

Page 7: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

7 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

Introduction

This white paper documents a case study that involved cloning and backing up a real-world 12 TB database, shipping it by air from the primary data center in Hopkinton, Massachusetts to the EMC Solutions Center in Cork, Ireland, and then restoring and recovering the database for test and development use.

The white paper covers the following:

• An overview of the technologies involved in the solution

• The discovery process carried out at the production site to develop a build plan for the remote site

• Re-creation of the data warehouse at the remote site

• Backup and securing of the DD660 for transfer by commercial air transport

• Restore and recovery of the database at the remote site

• Validation of the recovered database

• Replication of the database from the DD660 to a DD880 appliance

This white paper is intended for Oracle, storage, server, and backup administrators who want to understand how to migrate an Oracle database to a remote site and the build requirements at the remote site to enable recovery of the database. It is assumed that the reader is familiar with the following EMC and Oracle products:

• EMC Symmetrix VMAX

• EMC Data Domain

• Oracle Database 11g R1 Enterprise Edition with Oracle Clusterware

Table 1 defines terms used in this white paper.

Table 1. Terminology

Term Definition

AES Advanced Encryption Standard

ASM Automatic Storage Management

DD Data Domain

DD OS Data Domain operating system

RAC Real Application Clusters

RMAN Recovery Manager

SISL™ Stream-Informed Segment Layout architecture

Purpose

Scope

Audience

Terminology

Page 8: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

8

Technology overview

The case study uses the following hardware and software components:

• EMC Data Domain DD660

• EMC Data Domain Encryption

• EMC Data Domain file system encryption lock

• EMC Symmetrix VMAX

• EMC Symmetrix TimeFinder

• EMC PowerPath®

• Oracle Database 11g R1 Enterprise Edition with Oracle Clusterware

• Oracle RMAN

For the solution, an EMC Data Domain DD660 appliance was used to back up the source database and transport it to the recovery site in Cork.

Data Domain deduplication storage systems dramatically reduce the amount of disk storage needed to retain and protect enterprise data, requiring a storage footprint that is 10 to 30 times smaller, on average, than that required by the original data set. The DD660 is a midrange appliance that offers up to 2 TB/hour of aggregate inline deduplication throughput, and up to 700 GB/hour for a single stream. The model used in the case study provides 8 TB of storage capacity, with much higher data storage capacity enabled by deduplication.

Like all Data Domain systems, the DD660 derives its performance advantages from the Data Domain Stream-Informed Segment Layout (SISL) scaling architecture. SISL identifies 99 percent of duplicate segments in RAM, inline, ensuring that only unique data is stored to disk. The storage of unique data is invisible to backup software, which sees the entire virtual file system. The integrity and recoverability of backup data are secured by the Data Domain Operating System (DD OS) and Data Invulnerability Architecture™, which ensures that data is stored and recoverable with continuous write verification, fault-detection, and self-healing mechanisms.

When data enters the Data Domain system, using any of the supported protocols (NFS, CIFS, VTL, OST, and Distributed Segmented Processing), the stream is segmented, fingerprinted, deduplicated (Global Compression™), grouped into multi-segment compression regions, and then locally compressed. When confidentiality is required, deduplicated and compressed data can be encrypted inline and the file system can then be locked.

Data Domain systems integrate seamlessly with Oracle RMAN, with no changes to scripts, backup processes, or system architecture.

Overview

EMC Data Domain DD660

Page 9: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

9 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

Data Domain Encryption protects data if the Data Domain system is stolen or lost during transit, and eliminates accidental exposure during replacement of failed drives. DD Encryption is designed such that if an intruder circumvents other network security controls and gains access to encrypted data, the data is unreadable and unusable to that person without the proper cryptographic keys.

The inline encryption process integrates seamlessly with the inline deduplication process, encrypting data before it is written to disk. Similar to the advantages of inline deduplication, inline encryption requires minimal resources to provide fast, reliable, and secure backup and recovery.

File system encryption lock is specifically designed to lock an encrypted Data Domain appliance and its external storage prior to transporting the appliance between locations.

File system encryption lock requires an encryption passphrase. This is a human-readable key that is used to generate a machine-usable AES256 encryption key that encrypts the Data Domain system encryption key. Locking the file system involves changing the passphrase. This changes the encryption of the Data Domain system key and removes the passphrase from the system. Locking the system allows a Data Domain system to be transported with the encryption keys on the system. The keys are not recoverable until the file system is unlocked using the passphrase (which is not stored on the appliance).

An EMC Symmetrix VMAX provides the storage platform at both the primary and remote data centers.

Symmetrix VMAX storage arrays provide high performance and scalability for demanding enterprise storage environments. Built on the strategy of simple, intelligent, modular storage, the Symmetrix VMAX incorporates a highly scalable Virtual Matrix Architecture™ that enables it to grow seamlessly and cost-effectively from an entry-level configuration into the world’s largest storage system. The VMAX supports Flash drives, Fibre Channel drives, and SATA drives within a single array, as well as an extensive range of RAID types.

EMC Solutions Enabler is the software that provides a host with the Symmetrix Command Line Interface (SYMCLI), which is a comprehensive command set for managing your Symmetrix storage environment. SYMCLI is used in single command line entries and scripts to monitor device configuration and status, and perform control operations on devices and data objects within a managed storage system.

EMC Symmetrix TimeFinder software provides powerful replication capabilities for creating local point-in-time copies that are independent of the hosts and operating systems, applications, and databases. These copies are immediately available for read and write access.

For the solution, a TimeFinder copy of the Oracle database was used to offload backup overhead from the production environment to another server.

EMC Data Domain Encryption

EMC Data Domain file system encryption lock

EMC Symmetrix VMAX

EMC Solutions Enabler

EMC Symmetrix TimeFinder

Page 10: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

10

EMC PowerPath® is server-resident software that enhances performance and application availability by supporting multiple I/O paths to logical devices and intelligently distributing I/O requests across all available paths. PowerPath also provides automatic failover in the event of a hardware failure by automatically detecting the path failure and redirecting I/O to another path.

For the solution, PowerPath provides intelligent path management and automatic path failover at the remote site.

The solution involves an Oracle Database 11g R1 data warehouse and uses the features of Oracle ASMLib, Oracle Database 11g, and Oracle Clusterware 11g such as RAC, Automatic Storage Management (ASM), and RMAN.

EMC PowerPath

Oracle Database

Page 11: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

11 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

Solution overview

This solution is based on a real-world use case and is essentially a case study. The requirements of the use case were to take a production database and transfer it to a lab environment for test and development purposes.

The EMC IT Production DB resides in EMC IT corporate headquarters in Hopkinton, Massachusetts, as shown in Figure 1. The test and development lab is located in the EMC Solutions Center in Cork, Ireland. The database is approximately 12 TB.

Figure 1. The challenge

An economical and secure mechanism was required to transport the data, and the process also had to have zero impact to the EMC IT production system.

Nondisruptive to production As the project involved a real-world, production system, it was critical that the backup procedure did not impact database operations. This factor dictated a nondisruptive backup solution.

Distance and latency The distance between the data centers in Hopkinton and Cork is approximately 4,740 kilometers (2,946 miles). The minimum latency for the round-trip is 47.4 ms, not including other delays introduced by encryption or the network itself. In fact, the average measured ping time between the data centers is 90 ms.

For example, a dedicated WAN link, with 47.4 ms round-trip latency, 1 Gb/s bandwidth, and a TCP window size of 65,536 bytes, would restrict the maximum throughput to 1.3 MB/s and take approximately 112 days to move 12 TB of data.

As this was a one-off project, it was not envisaged that sufficient dedicated bandwidth would be made available on the corporate WAN for the data migration. These factors dictated an offline method for transporting the data.

Use case requirements

The challenge

Page 12: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

12

Database size The size of the database—approximately 12 TB— was a significant challenge and another factor that made it impractical to transport the data across the WAN. Figure 2 indicates just how big 12 TB is in real terms.

Figure 2. What 12 TB looks like

To facilitate transporting the data in an economical and secure fashion, a Data Domain DD660 appliance was chosen. The reasons for this are:

• Small form factor The DD660 used has a 2U form factor and weighs 30 kg (66 lb)—perfectly suited for low shipping costs.

• Data compression The DD660 has 8 TB of data storage space. However, Data Domain systems store only unique data, using Global Compression to pool redundant data from each backup image. This enables much higher data storage rates than the raw capacity of the device.

Figure 3 shows the difference in size between the original database, the database backed up by RMAN, and the deduplicated, encrypted database on the DD660.

Figure 3. Original and deduplicated database sizes

The solution

Page 13: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

13 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

• Data integrity The Data Domain Data Invulnerability Architecture protects against data loss from hardware and software failures.

• Encryption of data at rest Data Domain Encryption protects data if the Data Domain system is stolen or lost during transit.

• Encrypted file system lock Data Domain file system encryption lock protects an encryption-enabled Data Domain system (and its external storage devices) while in transit. The encryption keys are not recoverable without a passphrase, which is not stored with the system.

To ensure that the production environment was not impacted by the project, it was decided to create a TimeFinder clone of the database and mount this to a backup server. This would provide a fully-usable, independent, point-in-time copy of the production database that could be backed up to the DD660 nondisruptively.

Figure 4 illustrates the solution designed for the project and outlines the procedures involved.

Figure 4. The solution

Page 14: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

14

The initial planning and feasibility phase of this project took place over several weeks prior to the official kickoff of the project. After the official kickoff, the entire project took just 22 working days to complete, from commencing discovery to verification of the restored database at the remote site. Figure 5 shows the project schedule.

Predictably, discovery was the lengthiest phase (2 weeks) and was iterative in nature. It progressed in parallel with, and was informed by, the build process at the remote site. The comprehensive information gathering and planning during this phase was key to ensuring successful completion of the project. The result was a complete and successful restore and recovery of the database at the remote site in only 28 hours.

Figure 5. Project timeline

Project timeline

Page 15: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

15 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

Discovery – primary site

This section describes the discovery process carried out at the start of the project to determine the feasibility of the project and to gather infrastructure and configuration information from the production site. This information was used to determine the transport mechanism between the two data centers and to create a build plan for the remote site. This phase of the project involved the steps shown in Figure 6.

Figure 6. Discovery process

The stakeholders involved in the project were:

• Primary site

Stakeholders: EMC IT; EMC IT Security and Compliance

Location: EMC IT headquarters, Hopkinton, Massachusetts, U.S.

Database: EMC IT Production DB

• Remote site

Stakeholders: EMC Global Solutions (Oracle Engineering)

Location: EMC Solutions Center, Cork, Ireland (CSC)

Database: CSC test and development database

The security of the data to be transferred between the data centers was of critical importance to the feasibility of the project. EMC IT and EMC IT Security considered data security issues in terms of:

• Corporate governance and compliance—that is, ensuring data security both in transit and at the remote data center so as to meet legislative requirements and EMC and industry standards.

• Identifying any potential personal/sensitive data for masking/cleaning.

• Choosing a backup/copy mechanism that would be nondisruptive to the production database.

Overview

Security

Page 16: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

16

This feasibility phase concluded that a copy of the EMC IT Production DB would be supplied to EMC Global Solutions under the following conditions:

• The data would be stored in a secure EMC data center at the remote site.

• Access to the data would be limited to specified EMC personnel.

• No data would be made public or available to external parties.

• A Data Domain DD660 would be used to transport the data, with Data Domain Encryption and Data Domain file system encryption lock enabled.

To enable recovery and repurposing of the data warehouse, the infrastructure at the primary site would be re-created at the remote site. In preparation for this, the following business and technical issues were considered:

• Mapping and configuration of the underlying infrastructure—servers, storage, and IP and storage networks

• Identifying hardware and software versions for the:

Server operating system

Oracle database

Symmetrix VMAX storage system

Data Domain system

• Identifying the groups and individuals within EMC IT who could supply information and carry out the necessary tasks in the production environment

A project plan was then agreed to that would minimize the impact on EMC IT staff and would move the data to Cork with minimal delay.

To facilitate the project, EMC IT made several important documents available to the EMC Global Solutions team in Cork. These documents detailed the design and architecture of the EMC IT data warehouse.

The Detailed Solutions Design document for the data warehouse was key to understanding the production environment. This document contains the following information:

• High-level architecture

• Hardware inventory and configuration information for the database and backup servers and cluster, including CPU and memory, internal drives, and HBA and network cards and ports

• List of required Shared Infrastructure Services, including: SAN, storage array, and IP networks—private (Cluster Interconnect), data center, and backup

• Storage rationale and design, hardware and software, and configuration

• Security—user and group access and authentication

• Data center hardware provisioning—environment and cabling details

Business and technical

Primary data center configuration information

Page 17: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

17 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

In addition, the following “live” information was supplied:

• A copy of the current symapi_db.bin file from the production Symmetrix array

• An extract of the Oracle data dictionary views including V$ASM_DISKGROUP and V$ASM_DISK

• A listing of the Oracle ASM disks

• A mapping of the PowerPath devices and paths to the Symmetrix LUNs

• Multiple sample Automatic Workload Repository (AWR) reports

• A listing of the latest operating system and Oracle Clusterware and database patch levels

Discovery was an iterative process, and continued in parallel with, and in response to, the remote site build.

Architecture From the information gathered during the discovery phase, the infrastructure for the remote site was designed as shown in Figure 7.

Figure 7. Remote site architecture

Remote site infrastructure

Page 18: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

18

Hardware resources Table 2 details the hardware environment for the remote site.

Table 2. Hardware resources

Purpose Quantity Resources

Storage array

1 Symmetrix VMAX with:

• 2 x VMAX 64 GB engines

• 8 Gb FC connectivity

• 450 GB 15k FC drives

Oracle RAC database servers 2 Linux server with:

• Quad-core CPU

• 96 GB RAM

FC switches 2 8 GB/s FC switches

Ethernet switches 2 1 Gigabit Ethernet switches

Software resources Table 3 details the software resources for the remote site.

Table 3. Software resources

Software Purpose

Red Hat Enterprise Linux 5.5 Server OS for database and application servers

Oracle Database 11g R1 (11.1.0.7) Database and cluster software

EMC Enginuity™ 5875 Symmetrix VMAX operating environment

Backup appliance Table 4 details the configuration of the Data Domain appliance used for backing up and transporting the database.

Table 4. Data Domain appliance configuration

Appliance Details

EMC Data Domain DD600 • 12 x 1 TB SATA drives

• 1 Gb IP connectivity

• DD OS 4.9

Page 19: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

19 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

Oracle database backup – primary site

The process for configuring the DD660 and backing up the production database at the primary data center involved the steps shown in Figure 8. When the process was complete, the appliance was secure and ready to ship to Cork—that is, the data was encrypted and the file system locked.

Local sitebackup

Create clone copy of production

database

Backup complete

Configure DD660 network settings &

add to network

Enable DD660 file system encryption

RMAN backup to DD660

Lock DD660 file system

Test database restore

DD660 secure & ready to ship Data encrypted File system locked

Figure 8. Oracle database backup process

Network settings The DD660 used for this solution had two 1 Gb network interfaces. To improve network performance and resiliency, these were aggregated and used in parallel. Figure 9 shows the link aggregation setup.

Figure 9. DD660 link aggregation setup

To enable backup of the database, the appliance was then provided with an IP address in the local network.

Overview

DD660 setup

Page 20: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

20

Encryption Data Domain Encryption was enabled prior to enabling the file system on the DD660. Data was encrypted using the 128-bit Advanced Encryption Standard (AES) algorithm (aes_128_cbc), as shown in Figure 10. This option strikes a reasonable balance between security and performance for the use case.

Figure 10. DD660 encryption

On the production site, a fresh clone copy of the storage was created using TimeFinder. An Oracle RMAN backup of this copy was then taken, with the database mounted, but not opened, on a backup server. A recovery catalog was used to record RMAN metadata.

With Data Domain Encryption enabled, the data being backed up to the DD660 was deduplicated and then encrypted inline, before it was written to disk.

The following were backed up to the DD660:

• A full RMAN backup of the Oracle database

• The database control file

• A schema export of the RMAN recovery catalog

The Oracle team on the production site then validated the backup by restoring the control file and database to the backup server. Table 5 shows the duration of the backup and restore operations.

Table 5. Backup and restore times

Process Timings

Database backup to DD660 28 hours and 38 minutes

Database restore (for verification) 30 hours and 2 minutes

The filesys encryption lock command was used to lock the Data Domain file system. This required a passphrase change, as shown in Figure 11, automatically re-encrypts the system encryption key, and removes the new passphrase from the DD system. The new passphrase was communicated separately to the remote site.

RMAN backup

File system lock

Page 21: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

21 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

Figure 11. Data Domain file system encryption lock

Data Domain deduplication eliminated redundant data segments during the backup process. This allowed the Data Domain appliance to store and manage much more data than would be possible with a traditional storage device.

Figure 12 shows a DD OS view that summarizes data storage and compression statistics for the use case and illustrates the value of Data Domain deduplication when transporting an Oracle database. Table 6 also summarizes these statistics.

Figure 12. Use case compression summary

The total physical space available on the appliance is 7.5 TB, the total uncompressed data written by RMAN to the appliance was 10.6 TB, and the space required to store the compressed data was 3.2 TB. This demonstrates a 69.7 percent reduction in the space required to store the database.

Table 6. Use case compression summary

Item Value

Total physical space available 7,528.6 GiB (7.5 TB)

Total data written to appliance (uncompressed) 10,613.6 GiB (10.6 TB)

Data stored on appliance (compressed) 3,217.0 GiB (3.2 TB)

Deduplication factor 3.3

Percent reduction—storage space saving 69.7%

Data compression results

Page 22: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

22

Figure 13 shows the allocated, used, and free space information for the source database. This closely matches the statistics gathered by the DD660 and shown in Figure 12.

Figure 13. SQL output showing database allocated and free space information

The database had 11.6 TB of storage assigned to its data files, not including temp files, redo logs, control files and archive logs. Of this, 10.13 TB was allocated to tables and indexes within the data files, with 1,504 GB of free space available.

Page 23: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

23 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

Oracle server deployment and storage provisioning – remote site

This section describes preparation of the hardware and software infrastructure at the Solutions Center lab in Cork, as outlined in Figure 14.

Figure 14. Remote site preparation process

To simplify the restore process, the Oracle Solutions Engineering team decided to restore the database as a single instance before migrating it to Oracle RAC.

The database server was configured with Red Hat Enterprise Linux 5.5 and prepared for a standard Oracle ASM/Database install. Server configuration involved the following steps:

1. Install and configure EMC PowerPath.

2. Install and configure ASMLIB.

Primary data warehouse configuration information During the discovery phase of the project, EMC IT provided the Oracle Solutions Engineering team in Cork with the configuration database (symapi_db.bin) for the IT data warehouse storage array.

At the remote site, the configuration information for the storage array was extracted by querying symapi_db.bin in offline mode using standard Solutions Enabler display commands. The procedure involved the following steps:

1. Copy symapi_db.bin to a host with Solutions Enabler installed.

Overview

Deploying the remote site server

Configuring the remote site storage

Page 24: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

24

2. Use the following Solutions Enabler commands to set up Solutions Enabler to query the database file:

export SYMCLI_OFFLINE=1 export SYMCLI_DB_FILE=”/DWproject/symapi_db.bin” export SYMCLI_SID=XXX

3. Use the following command to extract the configuration details for all devices in the database storage group (GDWORAPRD), as shown in Figure 15:

symsg show SG_GDWORAPRD

Figure 15. Extracting device configuration information from a storage group

With this information it was possible to understand how the production storage was provisioned. In conjunction with the other information sources supplied by EMC IT during the discovery phase, the Cork team was able to determine how much storage was required on the target Symmetrix VMAX, the Oracle ASM disk groups required, and the number of LUNs in each disk group.

Provisioning storage at the remote site Provisioning storage at the remote site involved the following steps:

1. Create the devices on the target Symmetrix VMAX array, as listed in Table 7.

Table 7. Storage assigned to Symmetrix VMAX array at remote site

ASM disk group Number of devices Device size (MB)

GDWP_TEMP 32 32,263

GDWP_ARCH 16 64,528

GDWP_REDO01 16 8,066

GDWP_REDO02 16 8,066

GDWP_T0_DATA01 16 129,056

GDWP_T1_DATA01 80 129,056

GDWP_T5_DATA01 40 129,056

Total Devices 216

2. Configure each device with a device identifier label that corresponds to the

name of the ASM disk group to which the device will belong. This label can be read from the host using the Inquiry utility, which makes device identification from the host much easier.

Page 25: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

25 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

Figure 16 shows these labels being created with Solutions Enabler.

Figure 16. Setting device identifiers with Solutions Enabler

3. Create a masking view for the devices so that they are visible to the RAC cluster.

At this point, the storage is ready to be incorporated into the Oracle configuration at the remote site.

Setting up the Oracle database environment at the remote site involved the following steps:

1. Partition and assign the PowerPath devices for use with Oracle ASM.

Figure 17 shows the mapping of the Oracle ASM disks to the Symmetrix assigned “device_name”.

Figure 17. Oracle ASM disk mapping to Symmetrix devices

2. Install Oracle Clusterware and the database binaries.

Setting up the Oracle database environment

Page 26: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

26

3. Configure the ASM instance and create the ASM disk groups listed in Table 8.

Table 8. ASM disk groups and sizing

Disk group Allocation unit (MB)

Size (GB)

GDWP_ARCH 1 1,008

GDWP_REDO01 1 126

GDWP_REDO02 1 126

GDWP_T0_DATA01 16 2,016

GDWP_T1_DATA01 16 10,081

GDWP_T5_DATA01 16 5,041

GDWP_TEMP 1 1,008

For example, Figure 18 lists the 16 Oracle ASM disks assigned to disk group GDWP_ARCH.

Figure 18. Oracle ASM disks assigned to disk group GDWP_ARCH

Page 27: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

27 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

Oracle database recovery – remote site

This section describes the restore and recovery of the Oracle database at the remote site, as outlined in Figure 19.

Figure 19. Database restore and recovery process

Network settings When the appliance arrived at the remote data center, the IP address had to be changed to enable connectivity to the remote network. Link aggregation was already in place, so this was simply a case of changing the IP address of the virtual interface, as shown in Figure 20.

Overview

DD660 setup

Page 28: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

28

Figure 20. Reset the IP address

Unlock file system The Data Domain file system was locked during transit and could not be enabled without the passphrase, as indicated in Figure 21. The passphrase was sent to Cork separately from the DD660 appliance.

Figure 21. Data Domain file system enable refused—passphrase required

The filesys encryption unlock command allows the user to enter the passphrase to unlock the file system, as shown in Figure 22.

Figure 22. Unlocking and enabling the Data Domain file system

Page 29: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

29 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

With the Data Domain file system unlocked, the database was then restored from the DD660 using RMAN. This process involved the following steps:

1. Mount the DD660 on the /backup directory, as shown in Figure 23.

Figure 23. Mounting the DD660 on the remote site server

2. Create the Oracle database and build the RMAN recovery catalog from the export file held on the DD660. Figure 24 shows a summary listing of the database and control file backups from the catalog.

Figure 24. Summary of database and control file backups

3. Use RMAN to connect to the target database and the recovery catalog.

4. Start up the Oracle instance without mounting the database (nomount mode), as shown in Figure 25.

RMAN restore

Page 30: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

30

Figure 25. Starting up the Oracle instance in nomount mode

5. Restore the backup control file to the control_file location specified in the SPFILE, as shown in Figure 26.

Figure 26. Restoring the backup database control file

6. Allocate the channels and restore the database, as shown in Figure 27.

Page 31: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

31 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

Figure 27. Database restore

7. Mount and open the database, as shown in Figure 28.

Figure 28. Mounting and opening the database

In this use case, further recovery was not required as the database had been backed up in a consistent state, a backup control file was used, and no further archive logs needed to be applied.

8. Re-create and size the Oracle temp files for the temporary tablespaces TEMP and TEMP1. The SQL commands used to do this are shown in Figure 29.

Figure 29. Re-creating the Oracle temp files

This completed the successful restore and recovery of the database and provided the first step in validating the integrity of the database.

The total time taken for the restore and recovery operations was 27 hours, 8 minutes.

Page 32: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

32

Figure 30 shows a breakdown of the restored database objects and files and their mappings to the Oracle ASM disk groups.

Figure 30. Mapping of Oracle database objects to Oracle ASM disk groups

As part of the discovery process, EMC IT supplied the Cork team with Automatic Workload Repository (AWR) reports from the production database. A number of key SQL queries in the reports were identified to verify the integrity of the restored database. At the remote site, each of these queries was extracted from the AWR reports and run against the restored database.

Verifying the integrity of the database

Page 33: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

33 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

Figure 31 demonstrates this process for one of the queries. The first item is an extract from the SQL ordered by Gets section of an AWR report. The second item is an extract from the AWR report for SQL ID: 0v7knzwvnc9u7. The third item shows the Oracle TKProf output of the trace file gathered for this SQL query when it was run against the restored database.

Figure 31. Testing the integrity of the restored database

SQL query ID

SQL statement

Page 34: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

34

Data Domain consolidation – remote site

The Solutions Center in Cork uses a shared Data Domain DD880 for consolidated backup on many projects within the lab. To ensure consistency, and to maintain a copy of the EMC IT Production DB in its original state, Data Domain Replicator was used to copy the database image to the DD880 at the end of the project, as shown in Figure 32. The DD660 was then repurposed to another project.

Figure 32. Database consolidation to shared DD880

Data Domain Replicator software provides automated, fast, and reliable replication of data for disaster recovery (DR), remote office data protection, and multiple site tape consolidation. Replicator duplicates the compressed, deduplicated data over a network (including WANs), greatly reducing the demands on the network.

Once replication has been configured between a source and a destination, any new data written to the source can be automatically replicated to the destination—for the use case, a one-off copy was made to the DD880 but continued replication was not required.

Setting up replication between the DD660 and DD880 involves simply creating a replication pair by specifying the replica source and destination. This can be done using the DD Enterprise Manager, as shown in Figure 33, or using the DD OS CLI.

Overview

Replication process

EMC Data Domain DD880 Cork Solutions Center shared

backup target

Cork Solutions Center

DD Replicator EMC Data Domain

DD660

Page 35: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

35 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

Figure 33. Creating a replication pair

The Replication view in DD Enterprise Manager provides configuration, status, performance, and topology information for the replication pair. Figure 34 shows this information for the use case replication process at completion.

Figure 34. Data Domain Enterprise Manager—Replication view

Page 36: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

36

Figure 35 shows the timeline for the replication process, which took 23 hours to complete.

Figure 35. Data Domain replication timeline

Page 37: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

37 Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

Conclusion

This white paper captures and describes the process used to transport a 12 TB Oracle data warehouse between two EMC data centers and to restore and recover the database at the target site. The data centers were located on different continents and over 4,000 kilometers apart.

Data security was a critical requirement. If the transport mechanism was not secure, then it would not meet EMC corporate governance compliance and the move could not happen. In addition, as the process involved a real-world data warehouse, the solution had to be nondisruptive to EMC core production activities.

The large size of the database and the long distance involved meant that:

• Transfer over a shared WAN link was neither practical nor efficient

• Creating a dedicated WAN link was neither practical nor economical

In comparison, backing up the data to a Data Domain DD660 appliance, and transporting this by commercial carrier to the target site, proved to be a secure, practical, economical, and efficient solution.

At the primary site, EMC TimeFinder enabled offloading of the backup process to a non-production server, and using an NFS mount allowed simple RMAN backup and restore on the DD660.

The key findings of this solution include:

• Nondisruptive backup

EMC TimeFinder and seamless Data Domain integration with Oracle RMAN ensured zero impact to the production environment.

• Security and corporate governance compliance

Data Domain data integrity, Data Domain Encryption, and Data Domain file system encryption locking secured the system, enabling it to be shipped by standard commercial transport.

• Simple restore and recovery

Using an NFS mount allowed simple RMAN backup and restore from disk. Data Domain technology ensured balanced backup and restore speeds.

The 12 TB data warehouse was available for use within 3 days of arrival in Cork.

• Economic transport

Data Domain deduplication reduced the database storage and transport requirements from 12 TB to 3.3 TB—a small form factor DD660 (2U, 60 kg) minimized shipping costs.

Summary

Findings

Page 38: Remote Site Recovery of Oracle Enterprise Data Warehouse Using

Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain A Detailed Review

38

References

For additional information, see the white papers listed below.

• EMC IT's Migration to the Open, Expandable Oracle BI Grid — Applied Technology

• EMC Backup and Recovery for Oracle Database 11g Data Warehouse — A Detailed Review

For additional information, see the product documents listed below.

• EMC DD OS 4.9 Administration Guide

• EMC Solutions Enabler Symmetrix CLI 7.2 Command Reference

For additional information, see the documents listed below.

• Oracle Clusterware Administration and Deployment Guide 11g Release 1 (11.1)

• Oracle Real Application Clusters Administration and Deployment Guide 11g Release 1 (11.1)

• Oracle Clusterware Installation Guide 11g Release 1 (11.1) for Linux

• Oracle Database Installation Guide 11g Release 1 (11.1) for Linux

• Oracle Real Application Clusters Installation Guide 11g Release 1 (11.1) for Linux and UNIX

• Oracle Database Backup and Recovery Reference 11g Release 1 (11.1)

• Oracle Database Backup and Recovery User's Guide 11g Release 1 (11.1)

White papers

Product documentation

Other documentation