best practices upgrading rac

Best Practices for Upgrading Oracle RAC: 10g to 11g

1

Pasha, Mohammed Asmath Capital Market Authority

About Me !!

2

Mohammed Asmath Pasha

Senior Oracle Consultant / Team Lead Capital Market Authority

Over 10 Years of Experience in Banking/Financial Sector

Objectives

Why upgrade Oracle RAC

The benefits of doing an upgrade

Oracle RAC 11gR2 New Features at Glance

Oracle Grid Infrastructure 11gR2

Oracle 11gR2 GI - Clusterware New Features

Oracle 11gR2 GI – ASM New Features

Best Practices – Planning and Upgrading Oracle RAC to 11gR2

How to plan the upgrade

How to minimize downtime during the upgrade

Possible challenges, with workarounds and solutions

Brief details of known issues and bugs

3

Why upgrade ?

4

Oracle Support

New Features

Vendor Support

Bugs/Issues

Organization Policy

Why upgrade ?

Oracle Support

o Running out of premier support

New Features

o To leverage new features

5

Why upgrade ?

Vendor Applications

o New releases of an existing application or limitations in acquiring new applications

Bugs/Issues

o What are the issues are you facing today in your current environment.

o What are the bugs you have encountered in your current environment.

Organization Policy

o Organization policy for software usage

Continued..6

Benefits of doing an upgrade

Reliability got increased further by introducing server pooling.

FAN and FCF or TAF makes it easy for application to mask component failures from user.

Error Detection and Automatic Recovery

Increased Availability by eliminating single point of failures.

7

11gR2 RAC New Features at Glance

8

11gR1/R2

ASM Fast Mirror Resync Grid Infrastructure

ASM Preferred Read Disk Groups Oracle ACFS

ASM Rolling Upgrade Oracle ADVM

RAC One Node

Oracle Restart

Grid Plug and Play

Improved I/O Fencing Processes

Policy-Based Cluster and Capacity Management

GI Configuration Wizard

Role-Separated Management

Enhanced Network Configuration (SCAN using DNS or GNS)

Cluster Health Monitor (CHM)

OCR and Voting disk support in ASM

Redundant Interconnect Usage

Cluster time synchronization service

IPMI (Intelligent Management Platform) for Node Fencing

Enhanced CVU

Enhanced SRVCTL

Minimized downtime patching

11gR2 Grid Infrastructure

Starting 11gR2 ASM and Clusterware/Oracle Restart combined into single distribution called Grid Infrastructure.

Unlike previous releases ASM cannot be installed as separate home

9

Courtesy – Carol Palmer, Oracle.

11gR2 Grid Infrastructure

When Grid Infrastructure is installed on a standalone machine, Oracle automatically installs Oracle Restart component unlike Clusterware environment

Oracle Clusterware and Oracle ASM upgrades are always out-of-place upgrades. With 11g release 2 (11.2), you cannot perform an in-place upgrade of Oracle Clusterware and Oracle ASM to existing homes.

10

Grid Infrastructure – New Features

RAC One Node o Oracle Real Application Clusters One Node (Oracle RAC One Node)

provides enhanced high availability for single-instance databases, protecting them from both planned and unplanned downtime. Oracle RAC One Node provides the following:

o Always-on single-instance database services o Better consolidation for database servers o Enhanced server virtualization o Lower cost development and test platform for full Oracle RAC

11

GI – New Features Contd...

Oracle Restart o Oracle Restart enables the startup and automatic restart of single-

instance Oracle Database through the Cluster Ready Services (CRS) component of Oracle Clusterware on a single server. This provides higher availability and automated management of single-instance Oracle Database and Oracle ASM instances.

o Oracle Restart is integrated with Oracle Data Guard

srvctl config database -d orcl Database unique name: orcl Database name: orcl Oracle home: /u01/app/oracle/product/11.2.0/dbhome_1 Oracle user: oracle Spfile: +DATA/orcl/spfileorcl.ora Domain: ora.com Start options: open Stop options: immediate Database role: Management policy: automatic Disk Groups: DATA

12


Grid Plug and Play o In the past, adding or removing servers in a cluster required extensive

manual preparation. o Grid Plug and Play reduces the costs of installing, configuring, and managing

server nodes by starting a grid naming service within the cluster to allow each node to perform the following tasks dynamically: o Oracle Restart is integrated with Oracle Data Guard o Negotiating appropriate network identities for itself o Acquiring additional information it needs to operate from a

configuration profile o Configuring or reconfiguring itself using profile data, making host

names and addresses resolvable on the network

Improved I/O Fencing Processes o Oracle Clusterware 11g release 2 (11.2) replaces the oprocd and Hangcheck

processes with the cluster synchronization service daemon Agent and Monitor to provide more accurate recognition of hangs and to avoid false termination.

13


Enhanced Network Configuration (SCAN using DNS or GNS) o SCAN (Single Client Access Name) has been introduced in 11gR2 to simplify

the connectivity from the clients to a cluster database. In earlier releases when removal or addition of rac node takes place, the tnsnames.ora needs be updated accordingly.

o Using the SCAN this problem is resolved by using single virtual name for the clients to connect, irrespective of infrastructural changes at the cluster level.

o It can be implemented either using GNS (Grid Naming Service), a new service added in 11gR2 as part of CRS or by DNS.

o In order to implement GNS a subdomain needs to be created in your company with DHCP enabled

14 Courtesy – Oracle


Policy-Based Cluster and Capacity Management o With Oracle Clusterware 11g release 2 (11.2) and later, resources managed

by Oracle Clusterware are contained in logical groups of servers called server pools.

o Enables dynamic capacity assignment when needed to provide server capacity in accordance with the priorities you set with policies

o Enables allocation of resources by importance, so that applications obtain the required minimum resources, whenever possible, and so that lower priority applications do not take resources from more important applications

o Ensures isolation where necessary, so that you can provide dedicated servers in a cluster for applications and databases

15


Role-Separated Management

o Role-separated management for Oracle Clusterware allows certain administrative tasks to be delegated to different people, representing different roles in the company. It is based on the idea of an administrator role, called CRS Administrator that can create server pools and grant administrative tasks on a per resource basis.

Cluster Health Monitor (CHM) o The Cluster Health Monitor (CHM) stores real-time operating system metrics in the

CHM repository that you can use for later triage with the help of Oracle Support should you have cluster issues.

o System Monitor Service (osysmond) runs on each node of the cluster and Cluster logger Service (ologgerd) runs on one node of the cluster. Monitor service from each node send the data to logger service which stores the data in CHM repository.

o Diag Collection automatically retrieves the information from CHM repository to be uploaded to Support. More over diag collection got enhanced to collect the data needed only. For example the incident time and incident duration parameters can be passed while collecting the data.

Grid_home/bin/diagcollection.pl -collect –crshome Grid_home -chmoshome Grid_home -chmos -incidenttime 07/14/201201:00:00 -incidentduration 00:30

16


OCR and Voting Disks (ASM) o OCR and Voting disk now can be stored in ASM, eliminating the need for third-party

cluster volume manager. o Using RAW devices for OCR and Voting is not allowed anymore for fresh install.

o In previous releases dd command was used to backup the voting disks, which is also desupported.

o RAW devices are support only if you are upgrading!!

Cluster time synchronization service (CTSS)

o The Cluster Time Synchronization Service (CTSS) is installed as part of Oracle Clusterware and runs in observer mode if it detects a time synchronization service or a time synchronization service configuration, valid or broken, on the system. If CTSS detects that there is no time synchronization service or time synchronization service configuration on any node in the cluster, then CTSS goes into active mode and takes over time management for the cluster.

Redundant Interconnect Usage

o In previous releases, in order to have redundant connectivity for private interface bonding, teaming or similar technology was used. Grid Infrastructure and Oracle RAC now can make use of redundant network connectivity for private interface without using other network technology to enhance optimal communication in the cluster.

17


Out-of-place Upgrade

o In place upgrades are not available any more, starting 11gr2 patch sets are not released separately in fact the complete bundle released.

o This is mainly to reduce the upgrade time!!!

IPMI (Intelligent Management Platform) for Node Fencing o Oracle Clusterware 11gR2 now supports IPMI, an industry standard management

protocol to provide failure isolation support and to ensure cluster integrity. o You can either configure IPMI by selecting a node-termination protocol during the

installation, or configure after installation using crsctl command.

Enhanced CVU o CVU can generate shell scripts (Fixup scripts) that perform the system configuration

that is required for a successful installation, in addition to identifying system issues that can cause installation failures.

o CVU provides additional checks to address install, configuration and operational issues.

18


Enhanced SRVCTL

o SRVCTL commands have been enhanced to manage the configuration of standalone server using Oracle Restart.

o The new SRVCTL functionality enables you to register a non-cluster database that can be managed by Oracle Clusterware. Once registered, Oracle Clusterware can start, stop, monitor, and restart the database instance.

Minimized downtime patching for Oracle RAC o Opatch now can apply patches in multi-node, in multi-patch fashion. o Opatch now has ability to detect in one of the nodes is not patched.

19

ASM – New Features

Automatic Storage Management Cluster File System (Oracle ACFS) o Oracle Automatic Storage Management Cluster File System (Oracle ACFS) is a new

multi-platform, scalable file system, and storage management design that extends Oracle ASM technology to support all of your application data in both single host and cluster configurations. Additionally, Oracle ACFS provides snapshot functionality for a point-in-time copy of an Oracle ACFS system.

20

ASM – New Features Contd...

ASM Dynamic Volume Manager (Oracle ADVM)

o Oracle ASM Dynamic Volume Manager (Oracle ADVM) provides volume management services and a standard disk device driver interface to clients. File systems and other disk-based applications send I/O requests to Oracle ADVM volume devices as they would to other storage devices on a vendor operating system.

o Oracle Automatic Storage Management Cluster File System (Oracle ACFS) is layered on Oracle ASM through the Oracle ADVM interface. With the addition of Oracle ADVM, Oracle ASM becomes a complete storage solution for both database and non-database file needs.

21

ASM – New Features Contd...

o ASM Rolling Upgrades & ASM Configuration Assistant (ASMCA)

Starting 11.1.0.6 ASMCA performs a rolling upgrade and for earlier releases it performs normal upgrade during which ASMCA will bring down all ASM instances across the cluster.

o Intelligent Data Placement

Intelligent Data Placement enables you to specify disk regions on Oracle ASM disks for best performance. Using the disk region settings, you can ensure that frequently accessed data is placed on the outermost (hot) tracks which have greater speed and higher bandwidth.

o Define Custom Allocation Unit

o New ASMCMD Commands

Access Control Lists ASM File Operations

22

ASM – New Features [Extended/Stretched Clusters]

o ASM Preferred Read

ASM preferred mirror read is designed to indicate to Oracle which disk failover group is the preferred read disk group.

If your ASM disk group is mirrored with local and remote disks, this attribute can be set for RAC instances to read from local disks to ensure better performance.

23

ASM – New Features [Extended/Stretched Clusters] Contd...

o ASM Fast Mirror Resync/Rebalance

In 10g when redundancy is used and if transient failure occurs then the disk is

immediately taken offline and dropped.

Whereas in 11g if transient failure occurs then the disk is taken offline but not dropped until it reaches the attribute value DISK_REPAIR_TIME. ASM keeps track of all modified extents during the failure. After the disk is fixed, only modified extents are resynchronized. (Redundancy failure recovers faster).

Failures are limited to cable, controller etc., and in the case of disk damage or data corruption the complete resync is needed after fixing the disk.

This feature also can be used in preventive maintenance of storage units.

24


o Planning

Kick of meeting (Infrastructure Team, Application Team) It always better to present what you are going to do…

o Review your design document

This is needed to understand what other infrastructural changes that might collide or impact your upgrade plan.

Example: Sys admins are upgrading all the system to RHEL 6.0 To simplify the stack and simplify vendor interactions, Oracle recommends avoiding 3rd party Clusterware, unless absolutely necessary.

25


Contd...26

Current Release Upgrade Path

10.1.0.x

10.2.0.x

11.1.0.x

11.2.0.x

10.1.0.x -> 10.1.0.5 -> 11.2.0.3

10.2.0.x -> 10.2.0.3 -> 11.2.0.3

11.1.0.x -> 11.2.0.3

11.2.0.2 -> 11.2.0.3

Current Release Upgrade Path

9.0.1.x (or earlier)

9.2.0.x

10.1.0.x

10.2.0.x

11.1.0.x

11.2.0.x

9.0.1.x (or earlier) -> 9.0.1.4 -> 10.2.0.3 -> 11.2.0.3

9.2.0.x -> 9.2.0.8 -> 11.2.0.3

10.1.0.x -> 10.1.0.5 -> 11.2.0.3

10.2.0.x -> 10.2.0.3 -> 11.2.0.3

11.1.0.x -> 11.2.0.3

11.2.0.x -> 11.2.0.3

o Select the upgrade path that is needed for your environment Supported Upgrade path for Grid Infrastructure 11gR2

Supported Upgrade patch for Oracle Database 11gR2

Grid Infrastructure 11gR2 supports Oracle Database 11g release 2 (11.2), release 1 (11.1), Oracle Database 10g release 2 (10.2), and release 1 (10.1)


Contd...27

o Verify Verify your Server hardware configuration

Check the compatibility matrix for your O/S

List the additional patches needed for your O/S

Check additional network requirements

Check Storage configurations

Check compatibility with other tools like backup, mail etc.

Create an Issue Log and Change Log (VVV Imp)

# Issue Description Status Comments

1 Kernel issue Kernel does not take affect after reboot In-Progress Will reviewd within RAC installtion

2 VIOs with Oracle RAC Is VIOs supported on Oracle RAC ? Completed Confirmed from SYS Admin

3 Error related to Oracle user account When run verification utility failed to PRVF-4007 Completed Fixed by allow access ssh to same node


Contd...28

o Prepare for Grid Infrastructure 11gR2 Upgrade

Install additional OS patches/packages needed for 11gR2. Setup new BASE location for 11gR2 Setup new HOME for GI installation Setup SCAN IP’s (DNS or GNS)

• 3 IP’s are recommended irrespective of number of nodes • Should be on the subnet of Public/VIP’s • Not recommended to use HOSTS file configuration

o Create additional OS groups needed

/usr/sbin/groupadd -g 502 asmdba /usr/sbin/groupadd -g 503 oper /usr/sbin/groupadd -g 504 asmadmin /usr/sbin/groupadd -g 505 asmoper usermod -g oinstall -G dba,asmdba,oper,asmadmin,asmoper oracle

o Add additional interface on each node of RAC if redundant interconnect is

needed


Contd...29

o Apply additional patches needed for Grid Infrastructure 11gR2

If you are upgrading from 11.1 then you need to apply the patch for bug 7308467 on 11.1 clusterware home.

If you are upgrading from 11.2.0.2 then you must apply patch 11.2.0.2.1 (11.2.0.2 PSU 1) or later.

For more patches that are needed for your environment, please refer to Things to Consider Before Upgrading to 11.2.0.3 Grid Infrastructure/ASM [ID 1363369.1] RAC and Oracle Clusterware Best Practices and Starter Kit (Platform Independent) [ID 810394.1]

o Additional Storage Requirements, Check /tmp space!!

o Check CRS Version, the output should be same for the following: crsctl query crs activeversion crsctl query crs softwareversion


Contd...30

o Run CLUVFY Utility

If upgrading < 11gR2 runcluvfy.sh stage –pre crsinst –n racn1,racn2 –fixup –fixupdir /tmp –verbose

If upgrading > 11gR2(11.2.0) to higher runcluvfy.sh stage –pre crsinst –upgrade –n racn1,racn2 –rolling –src_crshome <current GI Home> -dest_crshome <New GI Home> -dest_version 11.2.0.3 –fixup –fixupdir <dirname> -verbose

o Backup

Backup Clusterware, ASM, Oracle Homes (Full System backup) Backup OCR Backup Voting

Most Importantly Plan your Fallback Strategy


Upgrading Grid Infrastructure to 11gR2

o Login as “oracle”/same user of current Clusterware o UNSET Environment Variables

o $ unset ORACLE_BASE o $ unset ORACLE_HOME o $ unset ORACLE_SID o $ unset ORA_CRS_HOME o $ unset TNS_ADMIN

o Launch Installer

./runInstaller.sh

31


Upgrading Grid Infrastructure to 11gR2 Contd...

32

Choose the desired option according to your upgrade plan.

To minimize the downtime, choose to install the software only. Later use: /u01/app/11.2.0/grid/crs/config/config.sh



o By default English will be selected, Add additional languages as needed.

33

o Node Selection o This step also validates the SSH connectivity between all RAC nodes

o On NEXT, a Pop-up message will appear if your environment is < 11.2.0.1

34



35



Enter the virtual name of SCAN

Enter the SYS Password of your current ASM Instance

36



Installer automatically picks up the groups as per the role

Select the ORACLE_BASE, ORACLE_HOME, that was setup during the preparation

o OCR Integrity check shows failed, You can ignore after verifying as above

o Kernel Parameter value of shmmni is set as 128 as needed, If your fixup script show the same value

then you can safely ignore this.

o Since NTP is not configured CTSS will be in Active mode

o You will encounter this when both “domain” and “search” entries exists in resolv.conf Warning in "Task resolv.conf Integrity" [ID 1271996.1]

37



o Once the software installation is done, execute rootupgrade.sh on local node first once completed,

proceed with rest of the nodes.

38



o Upgrade of Grid Infrastructure (Clusterware & ASM) completed successfully.

Now proceed with Post-Upgrade steps.

39



o Post Upgrade Checks

Verify CRS version on all Cluster nodes [oracle@racn1 bin]$ crsctl query crs activeversion Oracle Clusterware active version on the cluster is [11.2.0.3.0] [oracle@racn1 bin]$ crsctl query crs releaseversion Oracle High Availability Services release version on the local node is [11.2.0.3.0] [oracle@racn1 bin]$ [oracle@racn2 ~]$ crsctl query crs activeversion Oracle Clusterware active version on the cluster is [11.2.0.3.0] [oracle@racn2 ~]$ crsctl query crs releaseversion Oracle High Availability Services release version on the local node is [11.2.0.3.0] [oracle@racn2 ~]$

Check Cluster State across all nodes

[oracle@racn1 ~]$ crsctl check cluster -all ************************************************************** racn1: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** racn2: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online **************************************************************

40



Check the Voting & OCR disk information [oracle@racn2 crsd]$ crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 7200a9a1a244df0abffe040a4f6f8798 (/dev/raw/raw3) [] 2. ONLINE 0d690593a467df03ff181a184e84ffd3 (/dev/raw/raw4) [] 3. ONLINE 7cfd563c5110cf4cbfbda9fa5af30c38 (/dev/raw/raw5) [] Located 3 voting disk(s). [root@racn2 bin]# ./ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 520016 Used space (kbytes) : 6444 Available space (kbytes) : 513572 ID : 870042676 Device/File Name : /dev/raw/raw1 Device/File integrity check succeeded Device/File Name : /dev/raw/raw2 Device/File integrity check succeeded Cluster registry integrity check succeeded

Check if all nodes are Pinned [oracle@racn2 ~]$ olsnodes -t racn1 Pinned racn2 Pinned

41



You need to do this manually if you are performing new 11gR2 GI installation and want to create Pre 11gR2 database

o Post Upgrade Activities

o Migrate OCR to ASM ocrconfig -add +OCR ocrconfig -delete /dev/raw/raw1 ocrconfig -delete /dev/raw/raw2

o Migrate Voting to ASM

crsctl replace votedisk +VOTE

o Adding Redundancy for Interconnect oifcfg getif eth1 10.10.10.0 global cluster_interconnect eth0 192.168.117.0 global public oifcfg setif -global eth2/10.10.1.0:cluster_interconnect oifcfg getif eth1 10.10.10.0 global cluster_interconnect eth0 192.168.117.0 global public eth2 10.10.1.0 global cluster_interconnect How to Modify Private Network Information in Oracle Clusterware [ID 283684.1]

42



A New Interface added on both nodes with subnet 10.10.1.0

o References for the issues during upgrade

o How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on Linux/Unix [ID 969254.1]

o Top 11gR2 Grid Infrastructure Upgrade Issues [ID 1366558.1] o ASMCA Fails When Upgrading to 11.2 due to Different ASM/CRS User or Non-default ASM

Name [ID 1113073.1]

43



o Preparing for 11gR2 Database Installation and Upgrade

Check Space Requirements for new home of database Verify the pre-installation

• cluvfy stage –pre dbinst –n racn1,racn2 -verbose

Time important Queries/Reports

Backup the database

Review the 11gR2 Upgrade Companion • Oracle 11gR2 Upgrade Companion(785351.1)

Run Oracle's Database Pre-Upgrade Utility [ID 884522.1]

Irrespective of the option you want to choose for the database (DBUA/Manual catupgrade)

it is recommended to run Pre-Upgrade Utility (utlu112i.sql) and verify the requirements.

44


Upgrading Database to 11gR2

o Installing 11gR2 Database Login as “oracle” ./runInstaller.sh

45


Upgrading Database to 11gR2 Cont..

46



Choose the option according to your upgrade plan. To minimize the downtime install the software and later use DBUA/Manually to upgrade the database.

o Select if additional language support needed o Select the Edition

47



o Provide the locations for BASE, HOME o Groups will be picked up if created according to recommendations

48



49



Execute root.sh on local node first and then proceed with second node.

50



Once root.sh completed, DBUA will be launched to upgrade the database.

51



52



53



54



Installation, Configuration and Upgrade of database to 11gR2 completed successfully.

o Post-Upgrade Steps

Check for invalid objects Check the status of all database components

SQL> @utlu112s.sql Oracle Database 11.2 Post-Upgrade Status Tool 09-24-2012 14:50:58 Component Current Version Elapsed Time Name Status Number HH:MM:SS Oracle Server VALID 11.2.0.3.0 00:36:02 Oracle Real Application Clusters VALID 11.2.0.3.0 00:00:02 Oracle Workspace Manager VALID 11.2.0.3.0 00:01:30 Gathering Statistics 00:04:11 Total Upgrade Time: 00:41:48

Gather Statistics Upgrade RMAN Catalog (If you are using) Backup the database Modify the COMPATIBLE parameter to 11gR2 Verify the timing of Important Queries /Reports

55



o References

DB Upgrade/Migration-Troubleshooting Tool [MOS] Information Center: Upgrading and Migration Oracle Database [ID 1351112.2] Master Note For Oracle Database Upgrades and Migrations [ID 1152016.1] Oracle Database Upgrade Path Reference List [ID 730365.1] Complete Checklist for Manual Upgrades to 11gR2 [ID 837570.1] Complete Checklist to Upgrade the Database to 11gR2 using DBUA [ID 870814.1]

Oracle Database 11.2 Upgrade Methods http://www.oracle.com/technetwork/products/upgrade/upgrade-11-2-methods-173002.pdf

56



Summary !!!

Planning and Preparation are the keys to a successful upgrade

The DBUA is the recommended method for simplicity and ease-of-use

Take advantage of the Upgrade Planner from MYSUPPORT

Have a fallback strategy and test it

Oracle Grid Infrastructure/Database 11gR2Database 11g Release 2 is a stable release with lots of great features, so go for it!

57

Thank You !!!

Email: [email protected] Blog: http://masmathpasha.wordpress.com/

58

mailto:[email protected]

http://masmathpasha.wordpress.com/

http://masmathpasha.wordpress.com/

best practices upgrading rac

Documents

oracle support o

node oracle rac

oracle asm upgrades

upgrade oracle advm

oracle restart component

11gr2 rac new features

node oracle restart

grid infrastructure