upgrade and migrate on multi terabyte mission critical rac with near zero downtime

Upgrade & Migrate on Multi-Terabyte Mission Critical RAC with Near Zero Downtime

Jane Brown Senior Database Administrator, Oracle Certified Professional (OCP)

Database Management Practice

Purpose

• Minimize planned down time

• Meet the 7x24 availability requirement for Dell Mission Critical Applications

• Streamline Dell Mission Critical databases upgrades to 11g

2


Current Versions

• DB Version: 10.2.0.4

• OS Version: Suze Linux 64-bit

3

Target Upgrade Versions• Database Version: 11.1.0.7

• OS Version: OEL 64-bit


Other Key Data:Data Files: ASMStorage: EMC SymmetricDatabase Flashback: on

High Availability Data Guard Architecture

4

Redo Apply

drdb01-06LTST_DRPhysical Standby

prddb01-06LTSTPrimary

lsdb01-06 LTST_LS Logical Standby

SQL

Ap

ply

Data Center 1 Data Center 2


I. Reimage Logical Standby

5

Other Key Data:Data Files: ASMStorage: EMC SymmetricDatabase Flashback: on

Redo Apply




SQL

Ap

ply



Backup Essential Files

1) Backup spfile to pfile› lsts1> create pfile='/u01/app/oracle/product/10.1.0/db_1/dbs/initltst.ora' from spfile;

2) Backup Oracle Home db_1 and save it on a different host› cd $ORACLE_HOME/..

› /u01/app/oracle/product/10.2.0> tar cvf db_1.tar db_1

› /u01/app/oracle/product/10.2.0> scp db_1.tar prdb01:/dbscratch/preReImange/lsdb01/.

3) Capture admin directories› /u01/app/oracle/admin/ltst> ls

adump bdump cdump udump

4) Capture cluster name› grep -i name $ORA_CRS_HOME/install/cluster.ini

cluster_name= sdr2ppramercrs

5) Capture LUN Mapping – by Linux System Engineer

6


Reimage

• Shutdown database

• Re-image the host cluster servers to OEL 11g image by Linux System Engineer

› Deploy Oracle Enterprise Linux 64-bit OS

› Restore EMC LUN mapping

› Deploy Oracle 11g CRS RPM

› Deploy Oracle 11g Software RPM

7


Start 11g ASM

1) Verify that CRS and Nodeapps are all up› crs_stat.sh

2) Create 11g ASM Initialization file› cp /u01/app/oracle/product/10.2.0/db_1/dbs/init+ASM1.ora /u01/app/oracle/product/11.1.0/asm_1/dbs/.

› remove *_dump_dest from the file

3) Restore password file› cp /u01/app/oracle/product/10.2.0/db_1/dbs/orapw+ASM1 /u01/app/oracle/product/11.1.0/db_1/dbs/.

4) Source ASM environment› export ORACLE_HOME=/u01/app/oracle/product/11.1.0/asm_1

› export PATH=$ORACLE_HOME/bin:/usr/bin:.:/bin:/usr/kerberos/bin:/usr/local/bin:/sbin:/usr/sbin:./

› export LD_LIBRARY_PATH=$ORACLE_HOME/lib

› export ORACLE_SID=+ASM1

5) Startup 11g ASM› sqlplus / as sysasm

› SQL> startup

8


Register ASM to CRS

1. Add ASM to CRS› srvctl add asm -n sdr2prddb01 -i +ASM1 -o /u01/app/oracle/product/11.1.0/asm_1

› # Repeat to add all 6 nodes

2. Copy Initialization and Password files to the rest of RAC nodes › init+ASM1.ora

› orapw+ASM1

3. Start ASM on nodes via srvctl› +ASM1 SQL>shutdown

› srvctl start asm –n lsprddb01 #repeat for node 2-6

9


Start 10g DB

1) Restore 10g Oracle Home to a temporary directory on node1 only› /dbscratch/> tar xvf db_1.tar

2) Create a symbolic link for the original 10g Oracle Home› mkdir –p /u01/app/oracle/product/10.2.0/

› cd /u01/app/oracle/product/10.2.0/

› ln –s /dbscratch/db_1 db_1

3) Create admin directories › mkdir -p /u01/app/oracle/admin/ltst/

› cd /u01/app/oracle/admin/ltst/

› mkdir adump bdump cdump udump

4) Source 10g environment

› . 10gR2.env

5) Startup 10g Database› sqlplus / as sysdba

› SQL> startup

10


Verify Logical Standby logs1. From Logical Standby: Startup 10g Listener

› lsnrctl start LISTENER_LSPRDDB01

2. From Primary: Query the latest online logs › SQL> select THREAD#, SEQUENCE# from v$LOG where status='CURRENT';

THREAD# SEQUENCE#---------- ----------

1 55922 40763 40894 41495 40916 4092

3. From Primary: Force log archive› alter system archive log current;

4. From Logical Standby: Verify that the latest logs are applied› SQL> select THREAD#, MAX(SEQUENCE#), APPLIED from dba_logstdby_log where applied!='YES';

THREAD# MAX(SEQUENCE#) APPLIED---------- -------------- --------

1 5592 YES2 4076 YES3 4089 YES4 4149 YES5 4091 YES6 4092 YES

11

II. Upgrade Logical Standby Database


Pre-Upgrade

1) Run pre-upgrade information tools › SQL> @/u01/app/oracle/product/11.1.0/db_1/utlu111i.sql

2) Resolve any outstanding issue

3) Create a guaranteed restore point› SQL> create restore point pre_upgrade guarantee flashback database;

› SQL> select * from v$restore_point;

4) Stop 10g listener › .~ 10gR2.env

› lsnrctl stop

5) Stop SQL Apply and shutdown 10g database› SQL> alter database stop logical standby apply;

› SQL> shutdown

13


Start 11g Database

1) Create 11g DB Initialization file› cp /u01/app/oracle/product/10.2.0/db_1/dbs/initltst.ora /u01/app/oracle/product/11.1.0/db_1/dbs/.

› Edit initialization file to

– remove *_dump_dest

– set cluster_database=false

– set dg_broker_start=false

2) Restore password file› cp /u01/app/oracle/product/10.2.0/db_1/dbs/orapwltst1 /u01/app/oracle/product/11.1.0/db_1/dbs/.

3) Restore tnsnames.ora› cp /u01/app/oracle/product/10.2.0/db_1/network/admin/tnsnames.ora

/u01/app/oracle/product/11.1.0/asm_1/network/admin/.

4) Startup 11g database › . .bash_profile

› cd $ORACLE_HOME/rdbms/admin

› SQL> startup upgrade

14


Upgrade Database

1. Upgrade database› SQL> spool /tmp/upgrade.log

› SQL> @catupgrd.sql #40 min, db shutdown at then end of upgrade

2. Run post update scripts› ltst1 SQL> startup

› ltst1 SQL> @utlu111s.sql

› ltst1 SQL> @catuppst.sql

› ltst1 SQL> @utlrp.sql

3. Enable cluster database› ltst1 SQL> alter syste set cluster_database=true

4. Restart Instance 1

15


Register Database to CRS

1. Add Database and Instances to CRS › srvctl add database -d ltst_ls -o /u01/app/oracle/product/11.1.0/db_1

› srvctl add instance -d ltst_ls -i ltst1 -n sdrseprddb01 # repeat to add all 6 instance

2. Copy following files to the rest of RAC nodes › tnsnames.ora

› initltst1.ora

› orapwltst1

3. Restart DB› ltst1 SQL> shutdown

› srvctl start database –d ltst_ls

16


III. Synchronize Archive log (10g->11g)

17

Redo Apply




SQL

Ap

ply


11g


Start Apply

1. Query outstanding log in Logical Standby› col file_name for a70

› select thread#, sequence#, file_name, applied from dba_logstdby_log where applied!= ‘YES'

2. Start log Apply › alter database start logical standby apply immediate;

3. Repeat step 1 until all logs are applied

18


IV. Switchover to Logical Standby

19

Redo Apply

drdb01-06LTST_DRBystand Physical Standby

prddb01-06LTSTLogical Standby

lsdb01-06 LTST_LS Primary

SQL

Ap

ply


11g


Transition Primary -> Logical Standby

1. Verify Switchover status› SQL> select switchover_status from v$database;

SWITCHOVER_STATUS --------------------TO STANDBY

If the query returns “SESSIONS ACTIVE’, then ensure it’s ok to shut them down

› SQL > SELECT SID, PROCESS, PROGRAM FROM V$SESSION WHERE TYPE = 'USER' AND SID <> (SELECT DISTINCT SID FROM V$MYSTAT);

2. Disable all RAC instances except instance 1› srvctl stop instance –d ltst –i ltst2; #repeat for lsts2 - 6

› alter database disable thread 2; ; #repeat for thread 2 - 6

3. Switchover› SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO LOGICAL STANDBY;;

20


Transition Logical Standby -> Primary

1. Defer Redo log transfer in Logical Standby› SQL> alter system set

log_archive_dest_3='service="(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=sdrseprddb01-vip.us.dell.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=ltst_XPT.prd.amer.dell.com)(INSTANCE_NAME=ltst1)(SERVER=dedicated)))"',' LGWR ASYNC NOAFFIRM delay=0 OPTIONAL max_failure=0 max_connections=1 reopen=30 db_unique_name="ltst" register net_timeout=15 valid_for=(online_logfile,primary_role)' scope=both

› SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_3=DEFER;

2. Disable all RAC instances except instance 1› srvctl stop instance –d ltst_ls –i ltst2; #repeat for instnace 2-6

› alter database disable thread 2; #repeat for thread 2-6

3. Commit Switchover› SQL> select switchover_status from v$database;

SWITCHOVER_STATUS --------------------TO PRIMARY

› SQL> alter database commit to switchover to primary;

21


Start Rest of RAC Instances

1. New Primary LTST_LS: Enable threads and start RAC instances 2-6› alter database enable thread 2

› srvctl start instance –d ltst_ls –i ltst2;

2. New Logical Standby LTST: Enable threads and start RAC instances 2-6› alter database enable thread 2;

› srvctl start instance –d ltst –i ltst2;

22


V. Reimage Original Primary Hosts

23

Redo Apply




SQL

Ap

ply


11g

10g


Reimage Original Primary Host

• Follow the process used in reimaging logical standby hosts (slice 7 – 10)

• Do not sync archived log from 11g –> 10g (skip slice 11)

24


VI. Upgrade Original Primary to 11g

• Follow the process used in upgrading Logical Standby Database (slice 13–16)

25


VII.Synchronize Archive log (11g->11g)

1. Query outstanding log in Logical Standby› col file_name for a70

› select thread#, sequence#, file_name, applied from dba_logstdby_log where applied!= ‘YES'

2. Start log Apply› alter database start logical standby apply immediate;

3. Repeat step 1 until all logs are applied

26


VIII.Reimage Physical Standby Hosts

27

Redo Apply




SQL

Ap

ply


11g

11g 10g


Reimage Physical Standby Hosts

• Follow the process used in reimaging Logical Standby Database (slice 7 –10, skip step 5 in slice 10)

• Mount 10g Database as standby › sqlplus / as sysdba

› SQL> startup monut

› SQL> alter database mount standby database

28


Start 11g Physical Standby Database

1) Create 11g DB initialization file› cp /u01/app/oracle/product/10.2.0/db_1/dbs/initltst.ora /u01/app/oracle/product/11.1.0/db_1/dbs/.

› Edit initialization file to remove *_dump_dest

2) Create spfile from pfile

3) Restore password file› cp /u01/app/oracle/product/10.2.0/db_1/dbs/orapwltst1 /u01/app/oracle/product/11.1.0/db_1/dbs/.

4) Restore tnsnames.ora› cp /u01/app/oracle/product/10.2.0/db_1/network/admin/tnsnames.ora

c\/u01/app/oracle/product/11.1.0/asm_1/network/admin/.

5) Mount 11g standby database› . .bash_profile

› SQL> startup nomount

› SQL> alter database mount standby database

Note: wait for dg broker configuration to apply log

29


Register Physical Standby Database to CRS

1. Add Database and Instances to CRS › srvctl add database -d ltst_dr -o /u01/app/oracle/product/11.1.0/db_1 –s mount –s physical_standby -m

dr.amer.dell.com

› srvctl add instance -d ltst_dr -i ltst1 -n sdrseprddb01 # repeat to add all 6 instance

2. Copy following files to the rest of RAC nodes › tnsnames.ora

› initltst1.ora

› orapwltst1

3. Restart DB› ltst1 SQL> shutdown

› srvctl start database –d ltst_dr

30


IX. Switchover to Original Primary

31

Redo Apply




SQL

Ap

ply


11g

11g 11g


Switchover via Data Guard Broker

I. Remove old Data Guard Broker configuration› dgmgrl > remove configuration

II. Create new Data Guard Broker configuration› dgmgrl > create configuration dg_ltst as primary database is ltst_ls connect identifier is ltst_ls;

› dgmgrl > add database ltst as connect identifier is ltst_prd maintained as logical ;

III. Switchover to Logical Standby› dgmgrl > switchover to ltst

IV. Add Physical Standby to Data Guard Broker

32


X. Verify Data Guard Status

1) Verify Database Status › dgmgrl > show configuration

2) Verify Log Applying Status in Physical Standby› Run from primary

› @dg_standby_log.sql

3) Verify Log Applying Status in Logical Standby› select thread#, sequence#, file_name, applied from dba_logstdby_log where applied!= 'YES'

33


Summary I. Reimage Logical Standby Hosts to 11g – Linux System Engineer

II. Upgrade Logical Standby DB to 11g –Database Administrator

III. Sync Archive log from 10g Primary to 11g Logical Standby - Database Administrator

IV. Switchover to Logical Standby Manually –Database Administrator

V. Reimage original Primary Hosts to 11g – Linux System Engineer

VI. Upgrade the original Primary DB to 11g – Database Administrator

VII. Sync Archive log from 11g Primary to 11g Logical Standby - Database Administrator

VIII. Reimage Physical Standby Hosts to 11g – Linux System Engineer

IX. Switchover to original Primary via DG Broker- Database Administrator

X. Verify Data Guard Status

34


Reference

• Oracle® Database Upgrade Guide 11g R1 (11.1) B28300-02

• Oracle® Data Guard Concepts and Administration 11g R1 (11.1) B28294-03

• Oracle Data Guard Broker 11g Release 1 (11.1) B28295)

• Database Rolling Upgrade Using Data Guard SQL Apply - Oracle Database 11g and 10gR2

35

Global Marketing36

For more information:

• Visit Dell at Oracle OpenWorld in Moscone South booth 521

• Visit www.dell.com/oracle for more information

• Dell Services Oracle Practice information at http://content.dell.com/us/en/enterprise/oracle-solutions-services.aspx

• Dell presentations at OOW 2010 at http://www.slideshare.net/dellenterprise

• Dell | Oracle white papers at http://content.dell.com/us/en/enterprise/oracle-solutions.aspx#facets=whitepaper&p=1

• Dell | Oracle Tested and Validated documentation at http://content.dell.com/us/en/enterprise/d/solutions/product-configurations.aspx


Questions ?

37


Versions - Production

38

upgrade and migrate on multi terabyte mission critical rac with near zero downtime

Technology

upgrade database

g database sqlplus

g database bash

g asm sqlplus

g image

g environment

ls database flashback

ltst sql