emc vspex oracle computing: oracle database virtualization ... · pdf file4 emc vspex oracle...

IMPLEMENTATION GUIDE

EMC VSPEX ORACLE COMPUTING Oracle Database Virtualization with VMware vSphere and EMC XtremIO Enabled by EMC Data Protection

EMC VSPEX

Abstract

This Implementation Guide describes the high-level steps and best practices required to implement the EMC® VSPEX® Proven Infrastructure for virtualized Oracle Database 11g or 12c on a VSPEX Private Cloud with VMware vSphere enabled by EMC XtremIO™. Usage guidelines and major customer benefits are also included.

July 2015

2 EMC VSPEX Oracle Computing: Oracle Database Virtualization with VMware vSphere and EMC XtremIO Implementation Guide

Copyright © 2015 EMC Corporation. All rights reserved. Published in the USA.

Published July 2015

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

EMC2, EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries. All other trademarks used herein are the property of their respective owners.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

EMC VSPEX Oracle Computing: Oracle Database Virtualization with VMware vSphere and EMC XtremIO Implementation Guide

Part Number H14313

http://www.emc.com/legal/emc-corporation-trademarks.htm

http://www.emc.com/legal/emc-corporation-trademarks.htm

Contents


Contents

Chapter 1 Introduction 6

Purpose of this guide .................................................................................................. 7

Business value ........................................................................................................... 7

Audience .................................................................................................................... 8

Terminology................................................................................................................ 9

Chapter 2 Before You Start 10

Overview .................................................................................................................. 11

Pre-deployment tasks ............................................................................................... 11

Deployment workflow ............................................................................................... 12

Deployment prerequisites ........................................................................................ 13

Essential reading ...................................................................................................... 14

Chapter 3 Solution Overview 15

Overview .................................................................................................................. 16

EMC VSPEX Proven Infrastructure .............................................................................. 16

Solution architecture ................................................................................................ 17

Summary of key components.................................................................................... 18

Chapter 4 Solution Implementation 19

Overview .................................................................................................................. 20

Implementing the network ........................................................................................ 21

Overview .............................................................................................................. 21

Configuring the infrastructure network ................................................................. 21

Configuring the VLANs ......................................................................................... 22

Configuring the storage network .......................................................................... 22

Completing the network cabling........................................................................... 23

Implementing the storage array ................................................................................ 24

Overview .............................................................................................................. 24

Preparing and installing XtremIO.......................................................................... 24

Creating XtremIO volumes for vSphere hosts ....................................................... 25

Implementing ESXi Server and vCenter ..................................................................... 25

Overview .............................................................................................................. 25

Installing vSphere ................................................................................................ 26

Configuring vSphere networking .......................................................................... 27

Tables


Provisioning storage for VMFS datastores ............................................................ 27

Optimizing vSphere for XtremIO ........................................................................... 29

Deploying an Oracle database .................................................................................. 29

Overview .............................................................................................................. 29

Preparing the Oracle virtual machine template..................................................... 30

Deploying and configuring a database ................................................................. 32

Performing the database post creation activities ................................................. 34

Implementing EMC Data Protection........................................................................... 35

Chapter 5 Solution Verification 36

Overview .................................................................................................................. 37

Using the health monitoring tools ........................................................................ 38

Chapter 6 Reference Documentation 39

Reference documentation ......................................................................................... 40

EMC documents ................................................................................................... 40

Oracle documents................................................................................................ 40

VMware documents ............................................................................................. 40

Links .................................................................................................................... 41

Appendix A Configuration Worksheet 42

Customer Configuration Worksheet .......................................................................... 43

Figures Figure 1. VSPEX Proven Infrastructure ................................................................ 16

Figure 2. Architecture of the validated infrastructure .......................................... 17

Figure 3. Example redundant XtremIO network ................................................... 22

Figure 4. Sample FC storage network ................................................................. 23

Figure 5. XtremIO volumes mapping to initiator groups ...................................... 28

Figure 6. Oracle Database 12c Configuration Assistant setup ........................... 32

Figure 7. Storage locations using ASM diskgroup ............................................... 33

Tables Table 1. Terminology........................................................................................... 9

Table 2. Tasks for pre-deployment .................................................................... 11

Table 3. VSPEX workflow for virtualized Oracle database deployment ............... 12

Table 4. Hardware and software requirements .................................................. 13

Table 5. Solution components .......................................................................... 18

Table 6. Implementation processes .................................................................. 20

Table 7. Tasks for switch and network configuration ......................................... 21

Contents


Table 8. Tasks for storage configuration ............................................................ 24

Table 9. XtremIO volumes for storing virtual Oracle servers ............................... 25

Table 10. Tasks for server installation ................................................................. 25

Table 11. Installation and configuration requirements ........................................ 26

Table 12. Tasks for deploying an Oracle database .............................................. 30

Table 13. Example of a virtual machine template specification ........................... 31

Table 14. Storage design to ASM disk group mapping for Oracle database files .. 33

Table 15. Tasks for verifying the VSPEX installation ............................................ 37

Table 16. Tools for monitoring the solution ......................................................... 38

Table 17. Common server information ................................................................ 43

Table 18. ESXi server information ....................................................................... 43

Table 19. XtremIO array information.................................................................... 44

Table 20. Network infrastructure information ...................................................... 44

Table 21. VLAN information ................................................................................ 44

Table 22. Service accounts ................................................................................. 45

Chapter 1: Introduction


Chapter 1 Introduction

This chapter presents the following topics:

Purpose of this guide ................................................................................................. 7

Business value ........................................................................................................... 7

Audience .................................................................................................................... 8

Terminology ............................................................................................................... 9



Purpose of this guide

EMC® VSPEX® Proven Infrastructures are optimized for virtualizing business-critical applications. VSPEX provides modular solutions built with technologies that enable faster deployment, greater simplicity, more choices, higher efficiency, and lower risk.

VSPEX provides partners with the ability to design and implement the virtual assets that are required to support a fully integrated virtualization solution for an Oracle relational database management system (RDBMS) on a VSPEX private cloud infrastructure.

The VSPEX for virtualized Oracle Database 11g or 12c1 infrastructure provides customers with a system capable of hosting a virtualized database solution that is scalable and delivers a constant performance level. This solution uses VMware vSphere 5.5 to provide virtualization supported by the EMC XtremIO™ storage array. The compute and network components, while vendor-definable, are designed to provide redundancy and sufficient power to handle the processing and data needs of the virtual machine environment.

This Implementation Guide describes how to implement the VSPEX Proven Infrastructure for virtualized Oracle Database 11g or 12c on a VSPEX Private Cloud with VMware vSphere. This Implementation Guide describes the high-level steps required to deploy the VSPEX solution for Oracle Database 11g or 12c on a VSPEX Private Cloud for VMware vSphere with EMC XtremIO storage. The infrastructure servers required for this solution are designed to be layered on a VSPEX Private Cloud for VMware vSphere Proven Infrastructure. The same principles and guidelines apply to XtremIO that have been validated as part of the VSPEX program.

Business value

Database management systems software is used by many different types of businesses. Despite the increasing market share of other data management tools, growth in sales is expected to continue and accelerate as customers continue to diversify their infrastructures and supporting technologies and use more appliances and configurations.

This VSPEX Proven Infrastructure is focused on helping EMC partners understand the value that the XtremIO series combined with Oracle bring to customers who often have growing, isolated IT environments running server-centric applications.

1 The Oracle 11g version for this solution is Release 2 (11.2.0.4). We refer to this as 11g R2 throughout this document. The Oracle 12c version for this solution is Release 1 (12.1.0.2). We refer to this as 12c R1 throughout this document.



This solution is designed to meet Oracle database challenges and enable customers to increase performance, scalability, reliability, and automation. By consolidating database applications on XtremIO, customers can consolidate a single centralized storage platform to more effectively manage the exploding data growth that is challenging businesses today. This solution has been sized and proven by EMC to:

Deploy systems faster, saving time and effort with EMC Proven Solutions

Increase performance and scalability out of the box

Minimize storage requirements and reduce costs

Audience

This guide is intended for internal EMC personnel and qualified EMC VSPEX partners. This guide assumes that the VSPEX partners who intend to deploy this solution are:

Qualified by EMC to sell, install, and configure the XtremIO family of storage systems

Qualified to sell, install, and configure the network and server products required for VSPEX Proven Infrastructures

Certified to sell VSPEX Proven Infrastructures

Partners implementing this solution should also have the necessary technical training and background to install and configure:

VMware vSphere 5.5 virtualization platform

Red Hat Enterprise Linux (RHEL) 6.5

Oracle Database 11g R2 or 12c R1

This guide provides external references where applicable. EMC recommends that partners implementing this solution be familiar with these documents. For details, refer to Essential reading and Reference documentation.



Terminology

Table 1 lists the key terms used in this guide.

Table 1. Terminology

Term Definition

System Global Area (SGA)

SGA is a group of shared memory structures containing data and control information from one Oracle database instance.

Process Global Area (PGA)

PGA is memory dedicated to an operating process or thread that is not shared by other processes or threads on the system.

Virtual Machine Disk (VMDK)

VMware VMDK is an open file format that is a container for virtual hard disk drives for virtual machines.

Virtual Machine File System (VMFS)

VMware VMFS is a cluster file system that uses storage virtualization for multiple installations of VMware ESX server.

XtremIO X-Brick An X-Brick is a specialized configuration of the XtremIO All-Flash Array that includes 25 SSD drives for this solution.

Storage Controller (SC) Storage Controller is the computer component of the XtremIO storage array. SCs are used for all aspects of data moving into, out of, and between XtremIO arrays.

Chapter 2: Before You Start


Chapter 2 Before You Start


Overview .................................................................................................................. 11

Pre-deployment tasks .............................................................................................. 11

Deployment workflow .............................................................................................. 12

Deployment prerequisites ........................................................................................ 13

Essential reading ..................................................................................................... 14



Overview

This chapter provides an overview of important information you need to be aware of, documents you need to be familiar with, and tasks you need to perform before you start implementing the VSPEX Oracle Database 11g or 12c solution on XtremIO.

The Design Guide for this solution describes how to size and design the solution. The deployment examples in this Implementation Guide are based on the recommendations and examples in the Design Guide.

Before you deploy Oracle server on a VSPEX Proven Infrastructure, EMC recommends that you complete the pre-deployment tasks as described in the following section.

Pre-deployment tasks

Pre-deployment tasks include those not directly related to the environment installation and configuration, but that are prerequisites for installation. These tasks include collecting hostnames, IP addresses, license keys, and installation media. Perform the tasks in Table 2 to decrease the time required at the customer site.

Table 2. Tasks for pre-deployment

Task Description Reference

Gather documents

Get the related documents listed in Essential reading. These are used throughout the text of this document to provide details on setup procedures and deployment best practices for the various components of the solution.

Essential reading

Gather tools Get the required and optional tools for the deployment. Use the information in Table 4 to confirm that all equipment, software, and appropriate licenses are available before the deployment process.

Deployment prerequisites

Gather data Collect the customer-specific configuration data for networking, arrays, accounts, and so on. Enter this information into the Customer Configuration Worksheet for reference during the deployment process.

In addition, for the most comprehensive array-specific information, complete the relevant XtremIO pre-installation checklist and worksheet. These documents are available on EMC Online Support.

XtremIO Storage Array Pre-Installation Checklist

Configuration Worksheet

http://support.emc.com/

http://support.emc.com/



Deployment workflow

Refer to the process in Table 3 to design and implement a VSPEX workflow for your virtualized Oracle database solution.

Table 3. VSPEX workflow for virtualized Oracle database deployment

Step Action

1 Use the VSPEX qualification worksheet to collect user requirements. The qualification worksheet is provided in Appendix A.

2 Use the EMC VSPEX Sizing Tool to determine the recommended VSPEX Proven Infrastructure for your Oracle Database 11g or 12c solution, based on the user requirements collected in Step 1.

For more information, refer to the VSPEX Sizing Tool on the EMC Business Value Portal.

Note: You need to register the first time you access the tool.

3 Use the Design Guide to determine the final design for your VSPEX solution.

Note: Consider the requirements for all applications, not only the Oracle database.

4 Choose and order the VSPEX Proven Infrastructure components, including the XtremIO array, server systems, and network switches. Refer to the appropriate VSPEX Proven Infrastructure document in Essential reading for guidance.

5 Follow this guide to deploy and test your VSPEX solution.

Note: If you already have a VSPEX Proven Infrastructure environment, you can skip the implementation steps that are already completed.

https://mainstayadvisor.com/bvp_refresh/Landing.aspx





Deployment prerequisites

Table 4 itemizes the hardware and software requirements to configure the solution. For additional information, refer to the hardware and software tables in the deployment prerequisites checklist.

Table 4. Hardware and software requirements

Component Description Version

Hardware Physical servers—Sufficient physical server capacity to host the required number of virtual machines, as recommended by the Design Guide and VSPEX Sizing Tool

N/A

Virtual servers—VMware vSphere servers that host the virtual infrastructure servers

VMware vSphere 5.5

Networking—Switch port capacity and capabilities as required by the virtual server infrastructure

N/A

Storage array—EMC XtremIO array with the required configuration

Note: These requirements might be covered in the existing infrastructure.

EMC XtremIO 4.0

Software EMC XtremIO Management Server N/A

EMC Virtual Storage Integrator (VSI) for VMware vSphere Unified Storage Management

N/A

EMC VSI for VMware vSphere Storage Viewer N/A

VMware vSphere installation media 5.5

VMware vCenter Server installation media 5.5

Oracle database Server 11.2.0.4 or 12.1.0.2

Red Hat Enterprise Linux 6.5

VMware vSphere license keys

(for infrastructure server hosts)

5.5

Licenses VMware vCenter Server license key 5.5

Oracle database Server license key 11g R2 or 12c R1 Standard or Enterprise Edition



Essential reading

Before implementing the solution described in this guide, EMC recommends that you read the following documents, which are available from the VSPEX space on the EMC Community Network or from the VSPEX Proven Infrastructure area on the emc.com website. If you do not have access to a document, contact your EMC representative.

EMC VSPEX Oracle Computing: Oracle Database Virtualization with VMware vSphere and EMC XtremIO Design Guide

Server Virtualization with EMC XtremIO All-Flash Array and VMware vSphere 5.5

EMC Backup and Recovery Options for VSPEX for Virtualized Oracle 11g R2 Design and Implementation Guide

https://community.emc.com/community/partner/vspex?view=overview

https://community.emc.com/community/partner/vspex?view=overview

http://www.emc.com/cloud/vspex/explore.htm?pid=Home-Overview-Hero-013015

Chapter 3: Solution Overview


Chapter 3 Solution Overview


Overview .................................................................................................................. 16

EMC VSPEX Proven Infrastructure ............................................................................. 16

Solution architecture ............................................................................................... 17

Summary of key components ................................................................................... 18



Overview

This chapter provides an overview of the VSPEX for virtualized Oracle Database 11g or 12c solution. This solution has been designed and proven by EMC to provide server virtualization, network, storage, and backup resources. Refer to the VSPEX Proven Infrastructure Guide listed in Essential reading for information on configuring the required infrastructure components.

EMC VSPEX Proven Infrastructure

VSPEX Proven Infrastructures, as shown in Figure 1, are modular, virtualized infrastructures validated by EMC and delivered by EMC VSPEX partners. They include virtualization, server, network, storage, and data protection layers. Partners can choose the virtualization, server, and network technologies that best fit a customer’s environment, while the highly available XtremIO storage systems and EMC data protection technologies provide the storage and data protection layers.

Figure 1. VSPEX Proven Infrastructure

Refer to the VSPEX Proven Infrastructure Guide in Essential reading for information on configuring the required infrastructure components.



Solution architecture

Figure 2 shows the architecture validated for an Oracle Database 11g or 12c server overlay on a VSPEX infrastructure. The XtremIO storage array is an all-flash system based on scale-out architecture. The system uses building blocks, called X-Bricks, which can be clustered to grow performance and capacity as required. The solution supports block storage for virtual Oracle databases using one X-Brick.

Figure 2. Architecture of the validated infrastructure

To validate this solution, we2 completed the following tasks:

Deployed all Oracle Database 11g or 12c servers as virtual machines on VMware vSphere 5.5

Deployed the XtremIO array in multiple configurations to support different resources for both OLTP and DSS workloads on Oracle databases

Determined the recommended storage layout for an Oracle database and the virtual infrastructure pool in the XtremIO series of storage arrays

The solution architecture includes the following:

Storage layer—One X-Brick with a total of 7.58 TB of usable physical capacity 2 In this guide, “we” refers to the EMC Solutions engineering team that validated the solution.



Oracle database layer— Oracle Database 11g or 12c server with multiple differently-sized databases and snapshots

Network layer— SAN/IP switches that are designed to support virtualized environments

Physical servers and virtualization layer—Multiple ESXi servers that enable a high-performing and virtualized approach to deploy Oracle databases

Summary of key components

Table 5 summarizes the key components used in this solution. Refer to the Design Guide for overviews of the individual components.

Table 5. Solution components

Layer Components

Application Oracle Database 11g R2 or 12c R1

Virtualization VMware vSphere 5.5

VMware vCenter Server 5.5

VMware vSphere High Availability

EMC Virtual Storage Integrator

Compute VSPEX defines the minimum amount of compute layer resources required but enables the customer to implement the requirements using any server hardware that meets these requirements.

Network Minimum number of network ports required, as defined by VSPEX

Note: VSPEX defines the minimum number of network ports required and provides general guidance on network architecture, but enables you to implement any network hardware that meets these requirements.

Storage EMC XtremIO with:

EMC XtremIO Management Server

EMC Virtual Storage Integrator for VMware vSphere

VMware vSphere Storage APIs for Array Integration

EMC XtremIO Snapshots

Data protection EMC Data Protection solutions

Chapter 4: Solution Implementation


Chapter 4 Solution Implementation


Overview .................................................................................................................. 20

Implementing the network ....................................................................................... 21

Implementing the storage array ............................................................................... 24

Implementing ESXi Server and vCenter .................................................................... 25

Deploying an Oracle database ................................................................................. 29

Implementing EMC Data Protection .......................................................................... 35



Overview

This chapter describes the steps required to install, configure, and implement the VSPEX for virtualized Oracle Database 11g or 12c solution. If you already have a VSPEX Proven Infrastructure environment, you can skip the sections for the implementation steps. Otherwise, refer to the VSPEX Proven Infrastructure Guide listed in Essential reading for information on configuring the required infrastructure components.

Note: This solution requires certain infrastructure services, as shown in Figure 2. These services can be provided by existing infrastructure at the customer site, by a VSPEX Private Cloud, or by deploying them as dedicated resources as part of this solution.

Table 6 lists the main stages in the solution implementation process.

Table 6. Implementation processes


Prepare the physical network.

Configure the switches and connect to the network.

Implementing the network

Prepare storage. Install and configure the XtremIO array.

Implementing the storage array

Prepare the vSphere hosts.

Install and configure the ESXi Servers and create datastores and virtual machines.

Implementing ESXi Server and vCenter

Configure the virtual network.

Install and configure VMware vCenter and the virtual machine network.


Configure storage for virtual machines.

Configure the datastores for virtual machines.


Prepare the databases. Set up the Oracle Servers and create the databases.

Deploying an Oracle database

Configure the backup. Set up EMC Avamar® for database backup.

Implementing EMC Data Protection



Implementing the network

This section describes requirements for the network infrastructure to support the solution architecture. Table 7 provides a summary of the tasks for switch and network configuration along with references for further information.

Table 7. Tasks for switch and network configuration


Configure the infrastructure network

Configure storage network and host infrastructure networking as specified in the solution reference architecture.

Configuring the infrastructure network

Configure the VLANs

Configure private and public VLANs as required.

Configuring the VLANs

Vendor switch configuration guide

Configure the storage network

Connect Fibre Channel (FC) switch ports and zoning for vSphere hosts and the storage array.

Configuring the storage network

Vendor switch configuration guide

Complete network cabling

Connect the switch interconnect ports and vSphere server ports.

Completing the network cabling

The infrastructure network requires redundant network links for each vSphere host, the storage array, switch interconnect ports, and switch uplink ports. This configuration provides both redundancy and additional network bandwidth.

This configuration is required regardless of whether the network infrastructure for the solution already exists or is being deployed with other components of the solution.

Figure 3 shows an example of redundant network infrastructure for XtremIO storage. It uses redundant switches and links to ensure that no single point of failure exists in network connectivity.

Overview

Configuring the infrastructure network



Figure 3. Example redundant XtremIO network

Ensure that there are adequate switch ports for the storage array and vSphere hosts. EMC recommends that you configure the vSphere hosts with a minimum of two VLANs:

Client access network—Virtual machine networking

Management network—vSphere management and VMware vMotion (private network)

This solution used a dedicated FC network. The FC network required redundant FC switches and links for each vSphere host and the storage array. This configuration provided both redundancy and additional storage network bandwidth. We connected each vSphere host to both FC switches, and each switch to each storage processor on the array. We then placed each FC connection between the vSphere host and the array in a separate FC zone.

Configuring the VLANs

Configuring the storage network



Figure 4 shows a sample FC storage network.

Figure 4. Sample FC storage network

Ensure that all solution servers, storage arrays, switch interconnects, and switch uplinks have redundant connections and are plugged into separate switching infrastructures. Also, ensure that a complete connection is set up to the existing customer network.

Completing the network cabling




This section describes how to configure the XtremIO storage array. Table 8 provides a summary of the tasks for storage configuration and references for further information.

Table 8. Tasks for storage configuration


Prepare and install XtremIO

Install the XtremIO hardware according to the product documentation.

Configure the IP address information and other key parameters on the XtremIO array.

Configure XtremIO event handlers to send notifications concerning array physical space utilization.

EMC XtremIO Storage Array Hardware Installation and Upgrade Guide

Preparing and installing XtremIO

Create XtremIO volumes for vSphere hosts

Create XtremIO volumes that will be presented to the vSphere servers as VMFS datastores hosting the virtual Oracle server and database files.

Connect the VMware datastores to the vSphere hosts deployed for the solution.

Creating XtremIO volumes for vSphere hosts

This solution does not require any specific setup steps to install XtremIO. For instructions on assembly, racking, cabling, and powering the XtremIO array, refer to the following documents:

EMC XtremIO Storage Array Site Preparation Guide

EMC XtremIO Storage Array Pre-Installation Checklist

EMC XtremIO Storage Array Software Installation and Upgrade Guide


After preparing the XtremIO array, configure key information about the existing environment so that the array can communicate with it. Configure the following common items in accordance with your IT data center policies and existing infrastructure information:

Domain Name System (DNS)

Network Time Protocol (NTP)

XMS network interface

Storage network interfaces

Storage network IP addresses

Overview

Preparing and installing XtremIO



With XtremIO, all drives are under XtremIO Data Protection (XDP). Random and sequential I/Os that are typically generated by Oracle databases are treated equally because data is randomized and evenly distributed within the array. Therefore, consideration for different RAID groups is not required. As detailed in Table 9, we configured the required volumes to present to the vSphere servers as VMFS datastores for different types of Oracle files.

Table 9. XtremIO volumes for storing virtual Oracle servers

XtremIO volume Number of volumes

Volume size(GB) Purpose of volume

Datafile_vol 4 1024 Deploying Oracle data files and control files

Logfile_vol 4 50 Deploying Oracle redo log files

Crsfile_vol 2 16 Deploying Oracle Clusterware Ready Service(CRS) files and Voting disk files

Frafile_vol

(optional)

2 512 Deploying Oracle archived log files and backup files


This section provides information about installing and configuring the vSphere hosts and infrastructure servers required to support the architecture. Table 10 describes the tasks to be completed.

Table 10. Tasks for server installation


Install vSphere Install the vSphere hypervisor on the physical servers deployed for the solution.

vSphere Installation and Setup Guide

Installing vSphere

Configure vSphere networking

Configure vSphere networking including network interface card (NIC) trunking, VMkernel ports, and virtual machine port groups.

vSphere Networking

Configuring vSphere networking

Provision storage for VMFS datastores

Add the vSphere hosts to the initiator groups created and provision the storage for VMFS datastores used by ESXi servers


Optimize vSphere

Perform the configuration changes needed to ensure optimal performance of the XtremIO array.

EMC XtremIO Storage Array User Guide

Optimizing vSphere for XtremIO


Overview



On initial power up of the servers being used for vSphere, confirm or enable the hardware-assisted CPU virtualization setting and the hardware-assisted memory management unit (MMU) virtualization setting in the server’s BIOS. If the servers are equipped with a RAID controller, EMC recommends that you configure mirroring on the local disks.

Start the vSphere installation media and install the hypervisor on each of the servers. vSphere hostnames, IP addresses, and a root password are required for installation.

Table 11 describes the installation and configuration requirements to support the architecture, including ESXi hosts, vCenter, and infrastructure servers.

Table 11. Installation and configuration requirements


Install ESXi. Install the ESXi 5.5 hypervisor on the physical servers that are being deployed for the solution.

vSphere Installation and Setup

Configure and deploy vCenter Server.

Install VMware vCenter Server 5.5. vCenter Server and Host Management

Installing vCenter Server 5.5 best practices

Configure ESXi networking.

Configure ESXi networking including NIC trunking, VMkernel port, virtual machine port groups, and Jumbo Frames.

vSphere Networking

Create a virtual data center.

Create a virtual data center. vCenter Server and Host Management

Apply vSphere license keys.

Type the vSphere license keys in the vCenter licensing menu.

vSphere Installation and Setup

Add ESXi hosts. Connect vCenter to ESXi hosts. vCenter Server and Host Management

Configure vSphere clustering.

Create a vSphere cluster. vSphere Resource Management

Install the vCenter Update Manager plug-in.

Install the vCenter Update Manager plug-in from the administration console.

Installing and Administering VMware vSphere Update Manager

Deploy EMC PowerPath®/VE (optional).

Use Update Manager to deploy the PowerPath/VE plug-in to all vSphere hosts.

EMC PowerPath/VE for VMware vSphere Installation and Administration Guide

Install the EMC VSI for VMware vSphere plug-in.

Install the VSI for VMware vSphere plug-in on the administration console.

EMC VSI for VMware vSphere: Unified Storage Management— Product Guide

Install EMC PowerPath Viewer (optional).

Install PowerPath Viewer on the administration console.

EMC PowerPath Viewer Installation and Administration Guide

Installing vSphere




Perform array ESXi host discovery.

Perform ESXi host discovery in the EMC Unisphere® console.

vSphere Resource Management

For details on ESXi and vCenter, refer to the relevant VSPEX Proven Infrastructure Guide in Essential reading.

The VMware vSphere Networking guide describes vSphere networking configuration, including load balancing, link aggregation, and failover options. Select the appropriate load-balancing option for the network infrastructure.

Network interface cards

During the installation of vSphere, a standard virtual switch (vSwitch) is created. By default, vSphere chooses only one physical NIC as a vSwitch uplink. To maintain redundancy and bandwidth requirements, configure an additional NIC, either by using the vSphere console or by connecting to the vSphere host from the VMware vSphere Client.

Each vSphere server should have multiple NICs for each virtual network to ensure redundancy and provide for the use of network load balancing, link aggregation, and network adapter failover.

VMkernel ports

Create the following VMkernel ports as required, based on the infrastructure configuration:

VMkernel port for vMotion

Virtual Oracle Server port groups (used by the virtual Oracle servers to communicate on the network)

The VMware vSphere Networking guide describes the procedure for configuring these settings. Refer to the list of documents in Reference Documentation for more information.

Provisioning volumes in XtremIO is simple and straightforward. Complete the following steps in the EMC XtremIO Storage Management Application to configure volumes on the XtremIO array for storing virtual Oracle servers:

1. Click Configuration.

2. Click Create Volume.

3. In the Create Volumes window, click Add Multiple.

4. In Number of Volumes, type the required number of datastores based on the configuration.

5. In Name, provide an appropriate name. In Size, type the size as required, and then click OK.

Configuring vSphere networking

Provisioning storage for VMFS datastores



To enable vSphere servers to access the XtremIO volumes, configure XtremIO initiator groups and add the appropriate vSphere hosts to each using the following steps in the XtremIO Storage Management Application:

1. Click Configuration.

2. Under Initiator Groups, click Create Initiator Group.

3. In the Create Initiator Group window, Click Add, and then type a name in Initiator Name and Initiator Port Address.

4. In the Create Initiator Group window, repeat step 3 as needed to add all target initiator groups.

5. In the Configuration menu, click Create/Modify Mapping, select the target volumes, and then select the target initiator group, then click Map All. Click Finish to complete the process and grant the vSphere servers access to the selected volumes. Repeat as needed to grant the remaining initiator groups access to the remaining volumes.

6. Perform a Rescan for Datastores operation on the vSphere hosts so that they can immediately discover the XtremIO volumes to which they have access.

Figure 5 shows the volumes presented for different types of Oracle files and the initiator groups mapping.

Figure 5. XtremIO volumes mapping to initiator groups

The VMware vSphere Storage guide provides instructions on how to format the vSphere datastores when XtremIO initiator groups have been configured. Refer to the list of documents in Reference Documentation for more information.



Multiple changes are required to ensure optimal performance of the XtremIO array when used with vSphere. These changes are outlined in the EMC XtremIO Storage Array User Guide and include:

Change the vSphere Disk.SchedQuantum setting to 64 and the Disk.DiskMaxIOSize setting to 4096. This can be done using the following vSphere PowerCLI commands, replacing cluster with the name of the vSphere cluster where the target vSphere hosts reside. Repeat as needed for each cluster.

Get-VMhost -location <cluster> | Set-

VMhostAdvancedConfiguration Disk.SchedQuantum -Value 64

Get-VMhost -location <cluster> | Set-

VMhostAdvancedConfiguration Disk.DiskMaxIOSize -Value 4096

Change the vSphere Disk.SchedNumReqOutstanding setting to 256 for each vSphere datastore. This can be done using the following vSphere PowerCLI script. Repeat as needed for each vSphere cluster, updating the script with the target cluster name.

$vmhosts = get-vmhost –location <cluster_name> foreach ($vmhost in $vmhosts) {

$esxcli = get-esxcli -vmhost $vmhost

$AllLUNs = get-scsilun -vmhost $vmhost | where {$_.vendor -eq

"XtremIO"}

foreach ($lun in $AllLUNs) {

$CN = $lun.canonicalname

$EsxCli.storage.core.device.set($null,

$cn, $null, $null, $null, $null, $null, 256, $on)

}

}

Note: The settings described in this section apply only to vSphere hosts connected to EMC XtremIO arrays, and should not be applied to block datastores hosted on other arrays, including other EMC arrays.

Deploying an Oracle database

Table 12 describes the tasks for deploying the VSPEX for virtualized Oracle Database 11g or 12c solution. When the tasks are complete, Oracle Servers are set up on virtual machines with all the database files—including data files, control files, online redo files, voting disk files, Oracle CRS files, and other files—using the datastores designed for this VSPEX infrastructure.

Note: EMC recommends that you put the operating system (OS) volume for the Oracle Server virtual machine into the VSPEX private cloud pool, if available.

Optimizing vSphere for XtremIO

Overview



Table 12. Tasks for deploying an Oracle database


Prepare the virtual machine template.

Create a virtual machine template with the requirements and prerequisites for the Oracle Database 11g or 12c installation.

Enable the HugePages setting. The HugePages feature enables the Linux kernel to manage large pages of memory in addition to the standard 4KB (on x86 and x86_64) or 16KB (on IA64) page size.

Install the Oracle 11g or 12c CRS and database binaries and patches on the virtual machine.

vSphere Virtual Machine Administration

Preparing the Oracle virtual machine template

Deploy the database and perform the database post-creation activities.

Deploy and configure the database. Creating a database using the Database Configuration Assistant

Configuring the database file layout

Configuring the initialization parameters

Performing the database post creation activities

This section documents the preparation of a VMware virtual machine template to enable rapid deployment of new Oracle Database 11g or 12c environments.

VMware provides the ability to create a template that you can use as a master copy to quickly create and provision virtual machines. By using a template, you can install a guest OS and apply it to a virtual machine with application users and software configured and ready for use with minimal administrative intervention. This minimizes deployment time and avoids repetitive installation and configuration tasks for each virtual machine that is required.

To configure the template with the requirements and prerequisites for the Oracle software installation, complete the following steps:

1. Create a virtual machine with the following resources:

1 vCPU

4 GB of vRAM

100 GB virtual disk

2. Install the OS and Red Hat Package Manager (RPM) packages.

3. Configure the system and kernel parameters.

4. Create the OS groups and users.

5. Install the Oracle CRS and database binaries ready for clone.

Preparing the Oracle virtual machine template



6. Convert the virtual machine to a VMware template.

Table 13 provides an example of a virtual machine template specification.

Table 13. Example of a virtual machine template specification

Component Description

CPU 1 vCPU

Memory 4 GB

Virtual disk 100 GB with:

8 GB for the swap partition

92 GB for the bootable root partition

Operating system Install the RHEL 6.5 64-bit OS and configure the NICs.

Kernel 2.6.32

RPM packages Install or update required RPM packages and configure the kernel with Oracle-specific settings for semaphores, virtual and shared memory, network, and asynchronous I/O.

Check and adjust the resource limits for the Oracle software installation users.

Refer to Oracle Database Quick Installation Guide for Linux x86-64 for more details.

Network interfaces Eth0: public/management IP network

Eth1: Storage network

OS users Username: oracle

OS groups Group names: oinstall and dba

VMware tool Install VMware tool using RPM

Oracle CRS and DB binaries

11.2.0.4 or 12.1.0.2

HugePages HugePages is crucial for faster Oracle database performance on Linux if you have a large RAM and SGA. If your combined database SGAs are large (more than 8 GB), you need to configure HugePages. The size of the SGA is important.

The latest version of My Oracle Support Note ID 1392497.1: USE_LARGE_PAGES To Enable HugePages on the Oracle website provides more details.

You can deploy a virtual machine from a template with custom by manually selecting Deploy Virtual Machine from the template in vCenter. Select Customize and use an existing customization specification.

Use the VSPEX for Virtualized Oracle Qualification Worksheet in the Design Guide to size the virtualization infrastructure for the Oracle database. Determine the size of the individual data files based on the schema design and table space placement of

https://support.oracle.com/epmos/faces/DocumentDisplay?id=1392497.1&displayIndex=1



tables and indexes. Refer to Creating XtremIO Volumes for vSphere Hosts for more details.

This solution used Oracle Automatic Storage Management (ASM) to store the relevant database files, including data files, online redo log files, and CRS files.

Creating a database using the Database Configuration Assistant

The Oracle Database Configuration Assistant (DBCA) is the preferred way to create a database because it is a more automated approach and the database is ready to use when DBCA completes. Start DBCA as a standalone tool to create an Oracle database, as shown in Figure 6.

Figure 6. Oracle Database 12c Configuration Assistant setup

To start DBCA on UNIX, Linux, or at a Windows command-line prompt, type dbca and press Enter.

Note: The dbca utility is typically located in the ORACLE_HOME/bin directory.

Oracle DBCA runs in interactive mode. Using the GUI in Oracle DBCA, you can either modify one of the preconfigured database types or customize the database.

General Purpose/Transaction Processing is a preconfigured database template that you can create and customize during the installation.

Deploying and configuring a database



Configuring the database file layout

After Oracle ASM and Oracle CRS were installed, we deployed an Oracle database using the ASM diskgroup, which stored data on the previously created XtremIO volumes.

Table 14 details the storage design on XtremIO array for an Oracle database.

Table 14. Storage design to ASM disk group mapping for Oracle database files

XtremIO volume

ASM diskgroup Purpose

Datafile_vol +DATA Oracle ASM disks for data files

Logfile_vol +REDO Oracle ASM disks for redo log files

Crsfile_vol +CRS Oracle ASM disks for Cluster Registry and Voting disk files

Frafile_vol

(optional)

+FRA Oracle ASM disks for archived log files and backup files

Figure 7 shows how we configured the database data file using the ASM diskgroup +DATA.

Figure 7. Storage locations using ASM diskgroup

For more information about the storage layout recommendations and design, refer to the Design Guide.



Configuring the initialization parameters

To configure the initialization parameters, you can use one of the following methods:

Automatic Shared Memory Management (ASMM)

Automatic Memory Management (AMM)

I/O operations for file system files

Automatic Shared Memory Management Implement ASMM to automate the management of shared memory structures. The following shared memory structures are automatically managed:

DB_CACHE_SIZE

SHARED_POOL_SIZE

LARGE_POOL_SIZE

JAVA_POOL_SIZE

STREAMS_POOL_SIZE

When implemented, ASMM enables the Oracle database to distribute memory between these components based on workload requirements. Therefore, these components are considered autotuned parameters. The background process, Memory Manager (MMAN), coordinates the sizing of these components and moves memory to where it is needed most.

To implement the ASMM feature, set the following initialization parameters:

SGA_TARGET—Set to a nonzero value.

STATISTICS_LEVEL—Set to TYPICAL (or ALL).

Automatic Memory Management Do not use AMM because AMM is incompatible with HugePages. If you want to use HugePages, ensure that both MEMORY_TARGET and MEMORY_MAX_TARGET initialization parameters are not set.

For more information, refer to My Oracle Support Note ID 749851.1 on the Oracle website.

I/O operations for file system files Other recommended database parameter settings include:

Set DISK_ASYNCH_IO= true. The default value for this asynchronous I/O in both Oracle 11g and 12c is true.

Set FILESYSTEMIO_OPTIONS=SETALL. This setting enables both direct I/O and asynchronous I/O. With asynchronous I/O, processing continues while the I/O request is submitted and processed.

After you create a database as described in Creating a database using the Database Configuration Assistant, the instance is open and available for normal database use. However, the Oracle database will not start up or shut down automatically with the virtual machine.

Performing the database post creation activities



To automate the Oracle database startup and shutdown process, refer toHow to Automate Startup/Shutdown of Oracle Database on Linux (ID 222813.1) for more details.

Implementing EMC Data Protection

All VSPEX solutions are sized and tested with EMC Data Protection products, including EMC Avamar and EMC Data Domain®. If your solution includes backup components, refer to EMC Backup and Recovery Options for VSPEX for Virtualized Oracle Database 11g R2 Design and Implementation Guide for detailed information on implementing these options into your VSPEX solution.

Chapter 5: Solution Verification


Chapter 5 Solution Verification


Overview .................................................................................................................. 37



Overview

This section provides a list to review after configuring this solution. The goal of this section is to verify the configuration and functionality of specific aspects of the solution, and to ensure that the configuration supports core availability requirements.

Table 15 describes the required verification tasks.

Table 15. Tasks for verifying the VSPEX installation


Verify installation with a post-installation checklist

Verify that adequate virtual ports exist on each vSphere host virtual switch.

Verify that each vSphere host has access to the required datastores and VLANs.

vSphere Storage Guide

vSphere Networking

Deploy and test a single virtual Oracle server

Deploy a single virtual machine from the vSphere interface by using the customization specification.

vCenter Server and Host Management

vSphere Virtual Machine Management

Verifying redundancy of the solution components

Restart each storage processor in turn and ensure that connectivity to the VMware datastores is maintained.

Disable each of the redundant switches in turn and verify that the vSphere host, virtual machine, and storage array connectivity remains intact.

On a vSphere host that contains at least one virtual machine, enable maintenance mode and verify that the virtual machine can successfully migrate to an alternate host.

Vendor documentation

vCenter Server and Host Management



Health monitoring is a simplified measurement that reflects the reliability, stability, and performance of the entire VSPEX Proven Infrastructure for a virtualized Oracle database. Table 16 lists some tools you can use to monitor and troubleshoot your VSPEX solution.

Table 16. Tools for monitoring the solution

Tool Description

Esxtop The esxtop tool provides a real-time view (updated every five seconds, by default) of ESXi Hypervisor performance metrics, such as CPU. These performance metrics enable you to measure and troubleshoot Oracle database performance-related issues.

XtremIO Storage Management Application

XtremIO Storage Management Application is preloaded on XtremIO storage systems. It is used to manage the XtremIO array and monitor, collect, and analyze the all aspects of storage related data from it.

Automatic Workload Repository

The Automatic Workload Repository collects, processes, and maintains performance statistics for problem detection and self-tuning purposes. This data is both in memory and stored in the database. The gathered data can be displayed in both reports and views.

EMC recommends that you use these tools to monitor the performance of the entire Oracle database environment. For details about performance verification and test methodology, refer to the Design Guide.

Using the health monitoring tools

Chapter 6: Reference Documentation


Chapter 6 Reference Documentation


Reference documentation ........................................................................................ 40



Reference documentation

The following documents, located on EMC Online Support provide additional and relevant information. Access to these documents depends on your login credentials. If you do not have access to a document, contact your EMC representative.

EMC XtremIO Storage Array User Guide

EMC XtremIO Storage Array Operations Guide

EMC XtremIO Storage Array Software Installation and Upgrade Guide


EMC XtremIO Storage Array Security Configuration Guide

EMC XtremIO Storage Array Pre-Installation Checklist

EMC XtremIO Storage Array Site Preparation Guide

EMC VSI for VMware vSphere: Storage Viewer Product Guide

EMC VSI for VMware vSphere: Unified Storage Management Product Guide

EMC PowerPath/VE for VMware vSphere Installation and Administration Guide

The following documents are available on the Oracle website:

Oracle Database Documentation Library 12c Release 1 (12.1)

Oracle Database Documentation Library 11g Release 2.11.2

Oracle Database Release Notes for Linux

See My Oracle Support for the latest O Patch and latest PSU.

The following documents are available on the VMware website:

VMware vSphere vCenter Server Installation and Setup documentation

Performance Best Practices for VMware vSphere 5.5

VMware vSphere Networking documentation

VMware vSphere Storage documentation

VMware vSphere Resource Management documentation

Automating High Availability (HA) Services with VMware HA Technical Note

Installing VMware Tools in a Linux virtual machine using Red Hat Package Manager (RPM) is available in the VMware Knowledge Base (Article ID: 1018392).

EMC documents

Oracle documents

VMware documents

http://www.oracle.com/

https://support.oracle.com/

http://www.vmware.com/

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1018392&sliceId=1&docTypeID=DT_KB_1_1&dialogID=581560955&stateId=1%200%20581574645

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1018392&sliceId=1&docTypeID=DT_KB_1_1&dialogID=581560955&stateId=1%200%20581574645



The following web pages provide additional references for this solution:

Everything Oracle at EMC: https://community.emc.com/community/connect/everything_oracle

My Oracle Support website: https://support.oracle.com (user account required)

General VMware documentation: http://www.vmware.com

vSphere documentation: http://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-pubs.html

vSphere PowerCLI documentation: http://www.vmware.com/support/developer/PowerCLI/

Links

https://community.emc.com/community/connect/everything_oracle

https://support.oracle.com/

http://www.vmware.com/

http://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-pubs.html

http://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-pubs.html

http://www.vmware.com/support/developer/PowerCLI/

Appendix A: Configuration Worksheet


Appendix A Configuration Worksheet

This appendix presents the following topic:

Customer Configuration Worksheet ......................................................................... 43



Customer Configuration Worksheet

Before configuring an Oracle database for this solution, you need to gather some customer-specific configuration information, such as IP addresses, hostnames, and so on. The following tables provide a worksheet that you can use to record the information. You can also print and use the worksheet as a customer “leave behind” document for future reference.

Table 17. Common server information

Server name Purpose Primary IP address

Domain Controller

DNS Primary

DNS Secondary

DHCP

NTP

SMTP

SNMP

VMware vCenter Server

Oracle database server

Table 18. ESXi server information

Server name Purpose Primary IP address

Private net (storage) addresses

ESXi

Host 1

ESXi

Host 2

…



Table 19. XtremIO array information

Array name

Array Name

XMS IP

Storage Controller IPs

Initiator group names

Datastore names

Virtual machine infrastructure pool name

Virtual machine infrastructure file system name

Data file storage pool name

Data file file system name

Redo log storage pool name

Redo log file system name

CRS storage pool name

CRS file system name

Table 20. Network infrastructure information

Name Purpose IP address Subnet mask Default gateway

Ethernet Switch 1

Ethernet Switch 2

…

Table 21. VLAN information

Name Network purpose VLAN ID Allowed subnets

Client access network

Storage network

Management network



Table 22. Service accounts

Account Purpose Password (optional, secure appropriately)

XtremIO array root account

ESXi server administrator

vCenter Server administrator

Linux Server administrator

Oracle database administrator

emc vspex oracle computing: oracle database virtualization ... · pdf file4 emc vspex oracle...

Documents