distinguished product manager - oracle€¦ · oracle data guard page 13 of 76 oracle data guard...

Data Guard Hands On Lab

Larry M. Carpenter Distinguished Product Manager

Oracle Data Guard

of 76

Oracle Data Guard

of 76

Data Guard Hands On Lab

Oracle Database 11g Release 2

MMMaaaxxxiiimmmuuummm AAAvvvaaaiiilllaaabbbiiillliiitttyyy AAArrrccchhhiiittteeeccctttuuurrreee (((MMMAAAAAA)))

OOOrrraaacccllleee BBBeeesssttt PPPrrraaaccctttiiiccceeesss FFFooorrr HHHiiiggghhh AAAvvvaaaiiilllaaabbbiiillliiitttyyy

Oracle Data Guard

of 76

Oracle Data Guard

of 76

INTRODUCTION.................................................................................................................................................... 7

OVERVIEW OF THE EXERCISES ..................................................................................................................... 9

SETUP AND CONFIGURATION ....................................................................................................................... 11

ORACLE DATA GUARD .................................................................................................................................... 13

CREATING A PHYSICAL STANDBY ......................................................................................................................... 14

Preparing the Primary Database..................................................................................................................... 14 Enabling Archiving ...................................................................................................................................... 14 Enable Force Logging .................................................................................................................................. 15 The Password File ........................................................................................................................................ 15 Adding Standby Redo Log Files .................................................................................................................. 16

Configuring OracleNet .................................................................................................................................... 17

Configure the Standby System ......................................................................................................................... 20

Creating the required directories .................................................................................................................. 20 Creating the Password File .......................................................................................................................... 21 Creating the Parameter File ......................................................................................................................... 21 Starting up the Standby Instance.................................................................................................................. 22

Creating the Physical Standby Database ........................................................................................................ 22

THE DATA GUARD BROKER ........................................................................................................................... 25

CONFIGURING THE BROKER .................................................................................................................................. 26 Broker Prerequisites ........................................................................................................................................ 26 Configuring the Listener .................................................................................................................................. 28

Create the Broker Configuration ..................................................................................................................... 31 Verifying the Broker Configuration ................................................................................................................. 33

Managing a Broker Configuration .................................................................................................................. 36

TRANSPORT SERVICES .................................................................................................................................... 43

MODIFYING THE TRANSPORT METHOD ................................................................................................................. 46 AUTOMATIC REDO GAP RESOLUTION ................................................................................................................... 48

PROTECTION MODES ....................................................................................................................................... 51

MAXIMUM PERFORMANCE .................................................................................................................................... 51

MAXIMUM AVAILABILITY..................................................................................................................................... 51 MAXIMUM PROTECTION ....................................................................................................................................... 51 CHANGING THE PROTECTION MODE ..................................................................................................................... 52

ROLE MANAGEMENT SERVICES .................................................................................................................. 53

SWITCHOVER ........................................................................................................................................................ 53 Performing a Switchover ................................................................................................................................. 53

FAILOVER ............................................................................................................................................................. 56

Enabling Flashback Database ......................................................................................................................... 56 Performing a Manual Failover ........................................................................................................................ 58 Performing a Manual Reinstate ....................................................................................................................... 60

FAST-START FAILOVER (FSFO) ............................................................................................................................ 63 Enabling Fast-Start Failover ........................................................................................................................... 64 Executing an Automatic Failover .................................................................................................................... 67

CONCLUSION ...................................................................................................................................................... 71

Oracle Data Guard

of 76

RESOURCES ........................................................................................................................................................ 73

APPENDIX A – CREATE STANDBY SCRIPT ................................................................................................ 75

MAKENYC .......................................................................................................................................................... 75

Oracle Data Guard

of 76

Introduction

Oracle Data Guard ensures high availability, data protection, and disaster recovery for enterprise data. Data

Guard provides a comprehensive set of services that create, maintain, manage, and monitor one or more standby

databases to enable production Oracle databases to survive disasters and data corruptions. These standby

databases are maintained as transaction consistent copies of the production database.

If the production database becomes unavailable because of a planned or an unplanned outage, Data Guard can

switch any standby database to the production role, minimizing the downtime associated with the outage. Data

Guard can be used with traditional backup, restoration, and cluster techniques to provide a high level of data

protection and data availability.

A Data Guard configuration consists of one production database and one or more standby databases. The

databases in a Data Guard configuration are connected by Oracle Net and may be dispersed geographically.

There are no restrictions on where the databases are located, provided they can communicate with each other.

For example, you can have a standby database on the same system as the production database, along with two

standby databases on other systems at remote locations.

A standby database can be:

Physical standby database

o Provides a physically identical copy of the primary database, with on disk database structures that

are identical to the primary database on a block-for-block basis. A physical standby database is

kept synchronized with the primary database, through Redo Apply, which recovers the redo data,

received from the primary database and applies the redo to the physical standby database.

Logical standby database

o Contains the same logical information as the production database, although the physical

organization and structure of the data can be different. The logical standby database is kept

synchronized with the primary database using SQL Apply, which transforms the data in the redo

received from the primary database into SQL statements and then executes the SQL statements on

the standby database. Some restrictions apply.

Active Data Guard standby database

o A Physical standby database that is open to read access with up-to-date data from the Primary

database. Ancillary writes are also possible but the writes must be redirected to a read write

database using database links.

Snapshot Standby database

o A fully read write standby database that is created by converting a physical standby database into a

Read write snapshot standby database.

o Receives and archives redo data from the primary database. A snapshot standby database does not

apply the redo data that it receives. The redo data is not applied until the snapshot standby is

converted back into a physical standby database, after first discarding any local updates made to

the snapshot standby database.

This handbook is your guide to the capabilities of Data Guard in Oracle Database 11g Release 2.

Oracle Data Guard

of 76

Oracle Data Guard

of 76

Overview of the Exercises

The exercises are set up to walk you through Oracle Database 11g Data Guard using SQL*Plus, RMAN and the

Broker working with a Physical standby (Redo Apply) database using Data Guard to create a standby for data

protection and to test the various features of Data Guard. The exercises are designed from the top down so that

each step builds upon the previous exercise. It is essential that you follow the exercises in order.

The following is an outline of the exercises that you will perform in Oracle Database 11g.

Create a Physical Standby Database

o Verify that data is getting into the Physical standby.

Configure the Data Guard Broker

Change the Transport mode using Broker Properties

Change the Protection mode to Maximum Availability

Perform a switchover from the Primary to the Standby

Enable Flashback database

Perform Manual Failover from the Primary to the standby.

Enable and use Fast-Start Failover.

Note: While these exercises are run on a single system for simplicity this is NOT the best practice when

implementing Data Guard.

As you go through these exercises you will find the commands that must be done for each step at each line

beginning with the word TASK.

The exercises assume that you have multiple terminal windows set up for SQL*Plus command, DGMGRL

commands or Linux commands. Each TASK line in the book list the commands (SQL*Plus, RMAN, DGMGRL

or Linux) that are to be executed in one of the windows, depending on the type of command. Any DGMGRL

commands are to be executed in your DGMGRL terminal window, SQL commands in your SQL*Plus terminal

window, etc.

For the first exercise, ‘Creating a Standby’, the RMAN command to actually create the standby database at the

end of the section is long so there is one script in the appendix of this book called CreateNYC that you will be

directed to execute at the appropriate time. All of the tasks leading up to this step have to be executed first.

Oracle Data Guard

of 76

Oracle Data Guard

of 76

Setup and Configuration

Install Oracle 11.2.0.3 and configure a database called SFO and enable archiving when it is created.

Oracle Data Guard

of 76

Oracle Data Guard

of 76

Oracle Data Guard

Oracle Data Guard provides the management, monitoring, and automation software infrastructure to create and

maintain one or more standby databases to protect Oracle data from failures, disasters, errors, and data

corruptions. There are two types of standby databases. A physical standby uses Redo Apply to maintain a block

for block, exact replica of the primary database. A logical standby uses SQL Apply and contains the same logical

information as the primary database, although the physical organization and structure of the data can be different.

Administrators can chose either manual or automatic failover of production to a standby system if the primary

fails in order to maintain high availability for mission critical applications. For the purposes of this lab, we will

concentrate on Physical Standby databases (Redo Apply).

A Data Guard configuration includes a production database, referred to as the primary database, and up to 30

standby databases. Primary and standby databases connect over TCP/IP using Oracle Net Services. There are no

restrictions on where the databases are located provided that they can communicate with each other. A standby

database is initially created from a backup copy of the primary database. Data Guard automatically synchronizes

the primary database and all of its standby databases by transmitting primary database redo - the information used

by Oracle to recover transactions - and applying it to the standby database.

Oracle Data Guard

of 76

Creating a Physical Standby

To get started you will need to create your Physical standby database.

Preparing the Primary Database

There are some requirements that you need to meet before starting to create your Physical standby database.

Enabling Archiving

Enabling FORCE LOGGING

Creating the Password file

Adding Standby Redo Log files.

Enabling Archiving

Data Guard uses the redo (generated by the transactions and logged to the online redo log files by the LGWR

process) to maintain the standby database. As such you must enable archiving on your Primary database. In this

exercise, archiving is already enabled on SFO.

The archive logs are being archived to the Flash Recovery Area.

TASK:

In the SFO SQL*Plus window execute:

o sql

o archive log list

o show parameter recovery

SQL> archive log list

Database log mode Archive Mode

Automatic archival Enabled

Archive destination USE_DB_RECOVERY_FILE_DEST

Oldest online log sequence 30

Next log sequence to archive 32

Current log sequence 32

SQL> show parameter log_archive_dest_1

NAME TYPE VALUE

------------------ ------- ------------------------------

log_archive_dest_1 string location=USE_DB_RECOVERY_FILE_

DEST

SQL> show parameter recovery

NAME TYPE VALUE

--------------------------- ----------- ----------------

db_recovery_file_dest string /u01/app/fast_recovery_area

db_recovery_file_dest_size big integer 8G

Oracle Data Guard

of 76

Enable Force Logging

Force logging is a best practice since without the redo from the Primary database otherwise the datafiles on the

Physical standby will become unrecoverable and useless in a switchover or failover situation.

TASK:


o select force_logging from v$database;

o alter database force logging;

o select force_logging from v$database;

The Password File

Data Guard uses Oracle Net sessions to transport redo data and control messages between the members of a Data

Guard configuration. These redo transport sessions are authenticated using either the Secure Sockets Layer (SSL)

protocol or a remote login password file. Your Data Guard configuration is configured to use a remote login

password file. Every physical standby database in the configuration must have an up-to-date copy of the

password file from the primary database which will be copied to the standby database during the creation process

later in this exercise.

Note: Whenever you grant or revoke the SYSDBA or SYSOPER privilege or change the login password of a

user who has these privileges, you must replace the password file at each physical or snapshot standby database

in the configuration with a fresh copy of the password file from the primary database.

TASK:

No task as the Primary already has a password file.

SQL> select force_logging from v$database;

FOR

---

NO

SQL> alter database force logging;

Database altered.

SQL> select force_logging from v$database;

FOR

---

YES

Oracle Data Guard

of 76

Adding Standby Redo Log Files

Data Guard redo transport requires that the standby database have a set of standby redo log files also called the

SRL. An SRL is used to store redo received from the Primary (Chicago) database. If these standby redo log files

are created on the Primary before you create the standby database then they will automatically be added to the

standby database. In addition, having them created on the Primary database means that you are all set up to

function correctly when you perform a switchover. Standby redo log files are structurally identical to redo logs,

and are created and managed using the same SQL statements used to create and manage redo logs with the

‘STANDBY’ qualifier. Redo received from the Primary database via redo transport is written to the current

standby redo log group by a RFS background process. When a log switch occurs on the redo source database,

incoming redo is then written to the next standby redo log group, and the previously used standby redo log group

is archived by an ARCn background process on the standby. Oracle recommends that all redo log files in the

redo log at the redo source database and the standby redo log at a redo transport destination be of the same size.

The Primary has 50MB log files so you will create 50MB Standby redo log files. It is also a best practice for the

standby redo log to have at least one more redo log group per thread than the redo log on the Primary database.

TASK:


o alter database add standby logfile size 50M;




This will create 4 standby redo log files (SRL) at the Primary SFO in the directory specified by the parameter

‘create_file_dest’, which in our case is the ‘oradata’ area. In addition a second copy will automatically be created

in the Fast Recovery Area (FRA).

You can examine the standby redo log files in the v$logfile view where the ‘TYPE’ is ‘STANDBY’.

SQL> alter database add standby logfile size 50M;

Database altered.


Database altered.


Database altered.


Database altered.

Oracle Data Guard

of 76

TASK:


o select member from v$logfile where type=‘STANDBY’;

Configuring OracleNet

To enable your Primary database to talk to your standby database you must define a tnsnames entry on the

Primary system that connects to the standby database. In addition, to enable your standby database to talk to your

Primary database you must define a tnsnames entry on the standby system that connects to the Primary database.

These tnsnames connection descriptors will be used by Data Guard’s Redo Transport mechanism to send redo

from the Primary and to resolve any gaps in the redo when there is a connection failure.

To accomplish this you would first run ‘netmgr’ on your Primary system and add a tnsnames entry called ‘NYC’

pointing to your Standby system. You would then go to the Standby system and, again using ‘netmgr’, add a

tnsnames entry called ‘SFO’ pointing to your Primary system. Since both of your databases will reside on the

same system in this lab you would only need to add the entry for ‘NYC’ as ‘SFO’ already exists.

Since we are using the ‘LOCAL_LISTENER’ parameter in our databases it is also necessary to create tnsname

entries for our local listeners. These are called SFO_LISTENER and NYC_LISTENER.

SQL> select member from v$logfile where type = 'STANDBY';

MEMBER

---------------------------------------------------

/u01/app/oradata/SFO/onlinelog/o1_mf_4_6rppgb4b_.log

/u01/app/fast_recovery_area/SFO/onlinelog/o1_mf_4_6rppgb9z_.log

/u01/app/oradata/SFO/onlinelog/o1_mf_5_6rppghwg_.log

/u01/app/fast_recovery_area/SFO/onlinelog/o1_mf_5_6rppgj0r_.log

/u01/app/oradata/SFO/onlinelog/o1_mf_6_6rppgoqz_.log

/u01/app/fast_recovery_area/SFO/onlinelog/o1_mf_6_6rppgov1_.log

/u01/app/oradata/SFO/onlinelog/o1_mf_7_6rppgv6n_.log

/u01/app/fast_recovery_area/SFO/onlinelog/o1_mf_7_6rppgvb9_.log

8 rows selected.

Oracle Data Guard

of 76

TASK:

At the Linux prompt execute the following command

o cat $ORACLE_HOME/network/admin/tnsnames.ora

NOTE: You will need to ensure that your TNSNAMES.ORA is setup correctly for your system.

bash-3.2$ cat $ORACLE_HOME/network/admin/tnsnames.ora

# tnsnames.ora Network Configuration File:

/u01/app/oracle/product/11.2.0/db_1/network/admin/tnsnames.ora

# Generated by Oracle configuration tools.

SFO =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = SFO)

)

)

LISTENER_SFO =


NYC =

(DESCRIPTION =


(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = NYC)

)

)

LISTENER_NYC =


Oracle Data Guard

of 76

You will also need to add a static listener entry on the Standby system for the ‘NYC’ Physical standby database

so that RMAN will be able to connect to the instance (which will be started in a nomount state) and complete the

Physical standby database creation.

TASK:


o cat $ORACLE_HOME/network/admin/listener.ora

NOTE: You will need to ensure that your LISTENER.ORA is setup correctly for your system as above.

bash-3.2$ cat $ORACLE_HOME/network/admin/listener.ora

# listener.ora Network Configuration File:

/u01/app/oracle/product/11.2.0/db_1/network/admin/listener.ora

# Generated by Oracle configuration tools.

SID_LIST_LISTENER =

(SID_LIST =

(SID_DESC =

(GLOBAL_DBNAME = NYC)

(ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1)

(SID_NAME = NYC)

)

)

LISTENER =

(DESCRIPTION_LIST =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))

)

(DESCRIPTION =


)

)

ADR_BASE_LISTENER = /u01/app

Oracle Data Guard

of 76

Configure the Standby System

On the standby system you need to perform some setup before you can execute the RMAN script to create your

Physical Standby database using the new ‘over the network’ functionality.

Create directories for the Standby

o Audit dump directory and possibly the Data file directories.

Create a password file

Create an initialization parameter file

Startup the standby instance

Creating the required directories

RMAN will create the necessary dump directories for you with the exception of the Auditing directory as that

would normally be something that a DBA would be very careful about where they placed it. In your case you

will put it in the same path as the rest of the dump directories.

TASK:


o mkdir /u01/app/admin/NYC

o mkdir /u01/app/admin/NYC/adump

You also have to create the data file and log file directories because we are using the usual ‘oradata’ structure and

not using Oracle Managed Files (OMF). If the Primary and the Standby are both in ASM there would be no need

to create these directories; RMAN would create a NYC directory in ASM for you. In addition you need to create

the directories in the ‘Fast Recovery Area’ as well.

TASK:

At the Linux prompt execute the following commands

o mkdir /u01/app/oradata/NYC

o mkdir /u01/app/fast_recovery_area/NYC

o mkdir /u01/app/fast_recovery_area/NYC/archivelog

o mkdir /u01/app/fast_recovery_area/NYC/onlinelog

SQL> show parameter audit_file_dest

NAME TYPE VALUE

---------------- ----------- ------------------

audit_file_dest string /u01/app/admin/SFO/adump

bash-3.2$ mkdir /u01/app/admin/NYC

bash-3.2$ mkdir /u01/app/admin/NYC/adump

bash-3.2$ mkdir /u01/app/oradata/NYC

bash-3.2$ mkdir /u01/app/fast_recovery_area/NYC

bash-3.2$ mkdir /u01/app/fast_recovery_area/NYC/archivelog

bash-3.2$ mkdir /u01/app/fast_recovery_area/NYC/onlinelog

Oracle Data Guard

of 76

Creating the Password File

Data Guard needs an exact copy of the Primary database’s password file unless you are using the SSL method of

authentication as mentioned earlier. Normally you would need to copy the password file from the Primary

system to the standby system in order to create your standby database.

With the new RMAN “FROM ACTIVE DATABASE” capability in Oracle Database 11g, you no longer need to

copy the password file to create the standby initially. You only need to create a password file on the standby

system in the parameter directory ‘$ORACLE_HOME/dbs’ that has the same password for the SYS user as the

Primary. RMAN will copy the password file from the Primary to the new Standby for you.

NOTE: Any subsequent changes to SYSDBA user passwords require that a new copy of the Primary

database’s password is copied to each Physical Standby in your configuration.

TASK:

Execute orapwd file=$ORACLE_HOME/dbs/orapwNYC password=oracle

Creating the Parameter File

Normally you would need to obtain a copy of the initialization parameter file from the Primary database and edit

it yourself before you could create the physical standby database. With the RMAN procedure you are going to

use this is no longer necessary. RMAN will obtain a copy of the Primary database’s initialization parameter file

copy it over and edit it for you. You can also perform any additional parameter editing in the RMAN script

without the need for a text initialization parameter file. The resulting spfile will be placed in the default

$ORACLE_HOME/dbs directory.

To get your standby database created you do need to create a text initialization parameter file for the standby but

it only needs to contain the DB_NAME parameter and it does not matter what value that parameter has. For

example, you could ‘cat’ a value for db_name into a temporary init.ora file.

As with the password file, the script you will execute does this task for you.

TASK:

Execute the following:

cat > /u01/app/oracle/product/11.1.0/db_1/dbs/tempini.ora <<EOF

db_name=anyname

EOF

As you will see when you execute the script, very few parameters actually need to be set as the bulk of the Data

Guard specific parameters will be set for you in the next exercise “Configuring the Data Guard Broker”.

However it is possible to set all the necessary parameters in the single RMAN script if you so choose. An

example of this is in Appendix A at the end of this book.

orapwd file=$ORACLE_HOME/dbs/orapwNYC password=oracle


db_name=anyname

EOF

Oracle Data Guard

of 76

Starting up the Standby Instance

For RMAN to attach to your future Physical standby database the instance must be started, but not mounted yet

as there is no control file nor any real set of parameters.

TASK:

Set your environment to NYC and execute the following:

sqlplus '/ as sysdba'

STARTUP PFILE='/u01/app/oracle/product/11.1.0/db_1/dbs/tempini.ora' NOMOUNT;

Creating the Physical Standby Database

Finally, the setup is complete. If you have performed all the tasks in this section then you are ready to execute

the script and create your standby database. As a recap, this script will.

Clean up in case this is a 2nd

run

Create temporary password file

o Must have same password as primary

Create the temporary parameter file

o With only the DB_NAME parameter set to an arbitrary name

Start up the auxiliary instance in nomount mode

Run RMAN to duplicate your Primary database as a Physical standby database

The setup phase of the script is as follows:

TASK:

Already done

sqlplus '/ as sysdba'

SQL> STARTUP PFILE='/u01/app/oracle/product/11.1.0/db_1/dbs/tempini.ora' NOMOUNT;

# Create temporary password file, must have same password as primary

orapwd file=$ORACLE_HOME/dbs/orapwNYC password=oracle

# Create temporary parameter file

cat > $ORACLE_HOME/dbs/tempini.ora <<EOF

db_name=anyname

EOF

# Start up the auxiliary instance nomount

sqlplus '/ as sysdba' <<EOF

SET ECHO ON

STARTUP PFILE='$ORACLE_HOME/dbs/tempini.ora' NOMOUNT;

EOF

Oracle Data Guard

of 76

After those setup steps in your standby creation script, the RMAN script is executed. This script connects to the

Primary database ‘SFO’ (the TARGET) and the not yet created ‘NYC’ standby (the AUXILIARY), allocates

channels for parallelism, creates an in memory script of the spfile edits and as well as the backup procedure and

then executes its own script.

TASK:

Execute the above script from the Linux prompt

You will notice that the only parameters that are expressly set in the standby database are the bare minimum

required parameters and that there are no Data Guard specific parameters being set. This is done so that all of the

Data Guard parameters get created when you create the Data Guard Broker configuration in the next exercise.

One thing to note is that the RMAN ‘FROM ACTIVE DATABASE’ duplication method will place the SPFILE

of the standby in the normal ‘$ORACLE_HOME/dbs’ directory. If you wish you can move it into ASM with the

CREATE SPFILE command in SQL*Plus if you are using ASM.

# Run RMAN to duplicate your database.

rman <<EOF

set echo on

connect target sys/oracle@SFO; # Primary

connect auxiliary sys/oracle@NYC; # Standby: used by primary

run {

# Create channels for parallism

allocate channel prmy1 type disk;





allocate auxiliary channel stby1 type disk;

duplicate target database for standby from active database

spfile

parameter_value_convert 'SFO','NYC'

set 'db_unique_name'='NYC'

nofilenamecheck

;

}

EOF

exit 0

Oracle Data Guard

of 76

When the script has completed you will see the following at the end of the output on your screen.

At this point you have a standby database but it is not receiving any redo from the Primary nor is it ready to apply

any redo. This is due to the fact that we did not set any of the Data Guard parameters in the Standby or in the

Primary yet. You can verify that the Physical Standby database is in fact running as a standby with SQL*Plus.

TASK:

Execute the following commands in your NYC SQL*Plus window

o . oraenv (Enter NYC if it is not already the default)

o sql

To complete the process manually you could now log into the Standby as above and set all the Data Guard

parameters and start Redo Apply. After that you would return to the Primary and setup the same parameters

including the redo transport parameters. Once complete you would switch logs on the Primary and redo would

begin being sent to the Physical standby database. However, in our case we will be using the Broker so we’ll

leave things as they are and let the Broker define everything for us in the next exercise.

You have successfully completed the “Creating a Physical Standby Database” exercise. The next step is to add in

a Data Guard Broker configuration and watch how the Broker takes care of all those parameters for you. No

need for more and more scripts!

RMAN>

Recovery Manager complete.

bash-3.2$ . oraenv

ORACLE_SID = [NYC] ? NYC

The Oracle base remains unchanged with value /u01/app

bash-3.2$ sql

SQL*Plus: Release 11.2.0.2.0 Production on Thu Mar 24 16:48:40 2011

Copyright (c) 1982, 2010, Oracle. All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit…

With the Partitioning, OLAP, Data Mining and Real Application…

You are logged in as sys with sysdba privileges!

Unique Name Current Role Open Mode Protection Mode

---------------- ---------------- ------------ --------------------

NYC PHYSICAL STANDBY MOUNTED MAXIMUM PERFORMANCE

Oracle Data Guard

of 76

The Data Guard Broker

The Oracle Data Guard Broker is a distributed management framework that automates and centralizes the

creation, maintenance, and monitoring of Data Guard configurations. The following list describes some of the

operations the broker automates and simplifies:

Creating Data Guard configurations that incorporate a primary database, one or more (physical, logical, or

snapshot) standby databases, redo transport services, and log apply services. Any of the databases could

be Oracle Real Application Clusters (RAC) databases.

Managing an entire Data Guard configuration, including all databases, redo transport services, and log

apply services, through a client connection to any database in the configuration.

Managing the protection mode for the broker configuration.

Invoking switchover or failover with a single command to initiate and control complex role changes

across all databases in the configuration.

Configuring failover to occur automatically upon loss of the primary database, increasing availability

without manual intervention.

Monitoring the status of the entire configuration, capturing diagnostic information, reporting statistics

such as the redo apply rate and the redo generation rate, and detecting problems quickly with centralized

monitoring, testing, and performance tools.

The Data Guard Broker logically groups primary and standby databases into a broker configuration that allows

the broker to manage and monitor them together as an integrated unit. Administrators may interact with the

Broker using either Enterprise Manager Grid Control or the Broker’s command-line interface (DGMGRL).

Enterprise Manager Grid Control includes wizards that further simplify the creation of a Data Guard

configuration. Key Data Guard metrics such as apply lag, transport lag, redo rate and configuration status, are

included in a new consolidated HA Console, shown below.

Oracle Data Guard

of 76

Enterprise Manager enables historical trend analysis on the Data Guard metrics that it monitors - for example,

how the metric’s performance has been in the last 24 hrs, or last 5 days, etc. Also, through Enterprise Manager, it

is possible to set up notification-alarms such that administrators may be notified in case the metric crosses the

configured threshold value. Use of Enterprise Manager for Data Guard requires that you enable the Broker.

For the rest of the exercises we will be using the Data Guard Broker.

Configuring the Broker

For the purposes of these exercises you will be using the Command Line Interface called ‘dgmgrl’. To get your

Broker configuration created and enabled you will need to execute very few Broker commands as shown below.

Connect

Create configuration

Add database

Enable configuration;

However there are a few things that must be done before you can create and use a Broker configuration.

Broker Prerequisites

The following conditions must be met before you can use the Broker:

1. The primary and standby databases must be using the same version of Oracle Database

2. You must use a server parameter file (SPFILE)

3. Oracle Net Services network files must be set up on the primary database and on the standby database.

4. The value of the DG_BROKER_START initialization parameter must be set to TRUE

5. A service with a specific name must be statically registered with the local listener of each instance.

More information on these prerequisites can be found in Chapter 2 of the Data Guard Broker manual.

Oracle Data Guard

of 76

The first 3 of the above prerequisites have already been met. The other two 2 prerequisites need to be met before

you can execute these commands and create your Data Guard Broker configuration. These two need to be met

for all Broker configurations.

To enable the Broker to create and manage the Data Guard configuration you must first enable the Data Guard

Broker background processes on each database in the configuration. This is done by setting the parameter

“dg_broker_start” to ‘TRUE” on both databases (SFO and NYC).

Before you enable the Broker you first need to check the placement of the Broker configuration files. These are

controlled by the parameters “dg_config_file1” and “dg_config_file2”. These files default to the

$ORACLE_HOME/dbs directory and we will leave them there. On our standby they are as follows.

For the Primary database SFO they would be in the same location but with SFO in the filename.

Normally you would modify these parameters so that both copies of the files on each system in the configuration

reside in different directories for redundancy. And if the database is a Real Application Cluster (RAC) they then

must be modified so that the files are visible across the entire cluster as there can only be one copy of each file

for the entire RAC.

TASK: (Optional)

Execute the following commands in each SQL*Plus window

o show parameter broker

SQL> show parameter broker

NAME TYPE VALUE

---------------------- ----------- ------------------------------

dg_broker_config_file1 string /u01/app/oracle/product/11.2.0

/db_1/dbs/dr1NYC.dat


/db_1/dbs/dr2NYC.dat

dg_broker_start boolean TRUE

SQL> show parameter broker

NAME TYPE VALUE

---------------------- ----------- ------------------------------


/db_1/dbs/dr1SFO.dat


/db_1/dbs/dr2SFO.dat

dg_broker_start boolean TRUE

Oracle Data Guard

of 76

If you had not set these parameters correctly before enabling the Broker (as shown below) your Broker

configuration creation would fail and you would have to reset the ‘dg_broker_start” parameters back to FALSE

before you could change the file location parameters.

TASK:

Execute the following command in the SQL*Plus window of SFO and NYC

o alter system set dg_broker_start=TRUE scope=both;

Configuring the Listener

The second prerequisite is to configure a static listener entry on both systems so that the Broker can attach to any

of the databases in the configuration even when they are shutdown, as in a switchover operation or a restart and

automatic reinstatement of a failed Primary database in a Fast-Start Failover configuration. The Broker tries to

connect to a specific listener entry which is the database unique name with the string ‘_DGMGRL’ and the

domain, as in “SFO_DGMGRL.domain”. You can override this by modifying a Broker property called

‘StaticConnectIdentifier’ but for the purpose we will establish the default static entries in our listener.

Following Oracle best practices you would use the Network Manager utility (netmgr) to add these static entries to

the listener files at the Primary and Standby systems. In our case we’re on one system so we can just run it once

and add both entries.

TASK:

From the Linux prompt start ‘netmgr’ to make the changes to your listener file.

TASK:

Expand the configuration and select the LISTENER.

SQL> alter system set dg_broker_start=TRUE scope=both;

System altered.

SQL>

bash-3.2$ netmgr

Oracle Data Guard

of 76

TASK:

Select the ‘Database Service’ drop down menu at the right.

TASK:

Click the ‘Add Database’ button.

TASK:

Add Database 2 with SFO_DGMGRL as the name and SFO as the SID.

Oracle Data Guard

of 76

TASK:

Click ‘Add Database’ again

Add Database 3 with NYC_DGMGRL as the name and NYC as the SID

o Remember, if you were using two systems you would do this step on the Standby system

TASK:

Select ‘File’ from the menu and click ‘Exit’.

TASK:

And save the configuration changes by clicking the ‘Save’ button.

Oracle Data Guard

of 76

The last thing to do is to reload the listener to enable the changes you just made.

TASK:

Execute the following command at the Linux prompt

o lsnrctl reload listener

Verify that the changes have been made by executing ‘lsnrctl status’. You should see an entry for both SFO and

NYC with the ‘_DGMGRL’ addition.

TASK:

Execute the following command at the Linux prompt

o lsnrctl status

Create the Broker Configuration

Now that the prerequisites have been met it is time to create the Data Guard Broker configuration.

TASK:

Execute the following commands in the DGMGRL window

o dgmgrl sys/oracle@SFO

You must connect to the Primary database to create a configuration.

bash-3.2$ lsnrctl reload listener

.

.

.

The command completed successfully

bash-3.2$ lsnrctl status

… <edited for clarity>

Service "NYC_DGMGRL" has 1 instance(s).

Instance "NYC", status UNKNOWN, …

Service "SFO_DGMGRL" has 1 instance(s).

Instance "SFO", status UNKNOWN, …

bash-3.2$ dgmgrl

DGMGRL for Linux: Version 11.2.0.2.0 - 64bit Production


Welcome to DGMGRL, type "help" for information.

DGMGRL> connect sys/oracle@SFO

Connected.

Oracle Data Guard

of 76

Next you create the configuration, identifying how the Broker should connect to the Primary database by

specifying a tnsnames descriptor that includes all of the Primary instances. In our case the database is a single

instance database so our tnsname points to one host.

TASK:


o create configuration DG2011 as primary database is SFO connect identifier is SFO;

Next you add the standby database that we just created by specifying the tnsname for the standby. As with our

Primary you specify a tnsnames descriptor that includes all of the Standby instances. In our case the database is a

single instance database so our tnsname points to one host.

TASK:


o add database NYC as connect identifier is NYC maintained as physical;

At this point the configuration is complete, a Primary database with our one standby database. But up to now all

we have done is update the Broker configuration files at the Primary. Until we enable the configuration, nothing

will be written to the Standby Broker configuration files nor will any Data Guard parameters be created. So the

next step is to enable the configuration. Once this step is complete Data Guard will be in full operation.

TASK:


o enable configuration;

If you switch to the alert log windows where you should be tailing the alert log files, you can see the various

ALTER SYSTEM and ALTER DATABASE commands that the Broker executes to define all of the Data Guard

parameters, to start redo transport and start redo apply at the standby.

DGMGRL> create configuration DG2011

> as primary database is SFO

> connect identifier is SFO;

Configuration "dg2011" created with primary database "sfo"

DGMGRL>

DGMGRL> add database NYC as connect identifier is NYC

> maintained as physical;

Database "nyc" added

DGMGRL>

DGMGRL> enable configuration;

Enabled.

DGMGRL>

Oracle Data Guard

of 76

Verifying the Broker Configuration

It will take a few minutes for the Broker configuration to be enabled and ready to go. You can check the Broker

configuration with the ‘SHOW CONFIGURATION’ command.

TASK:

Execute the following command in the DGMGRL window

o show configuration;

At this point the Broker has defined all the parameters that you would normally have to do manually. In addition

it has started redo transport from the Primary to the Standby in asynchronous mode and it has started the Redo

Apply process at the standby.

You can verify that the redo is being shipped and applied by using SQL*Plus on the standby to select data from

the ‘V$MANAGED_STANDBY’ view and display it, Primary first, standby second. In the output you will see

that there is an LNS process on the Primary database that is sending the current redo Sequence to our standby.

There would be multiple of ‘LNS’ lines if there were multiple standbys.

TASK:

Execute the following command in the SFO SQL*Plus window

o select client_process, process, sequence#, status from v$managed_standby;

DGMGRL> show configuration;

Configuration - dg2011

Protection Mode: MaxPerformance

Databases:

sfo - Primary database

nyc - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:

SUCCESS

SQL> select client_process,process,sequence#,status

2 from v$managed_standby;

CLIENT_P PROCESS SEQUENCE# STATUS

-------- --------- ---------- ------------

ARCH ARCH 43 CLOSING

ARCH ARCH 0 CONNECTED



LNS LNS 48 WRITING

Oracle Data Guard

of 76

Performing the same command on the standby you will see the receiving RFS process of the redo and a process

called MRP that is coordinating the redo apply services.

TASK:

Execute the following command in the NYC SQL*Plus window

o select client_process, process, sequence#, status from v$managed_standby;

From this output you can see that the Primary is currently shipping sequence #48 and the standby is applying that

same sequence in real time apply mode. Real time apply is enabled because we created standby redo log files on

the primary database before we created the standby. Since they existed on the Primary, RMAN created them for

us on the Standby and the Broker automatically uses Real Time Apply when they exist.

You can examine the standby redo log files on NYC in the v$logfile view where the ‘TYPE’ is ‘STANDBY’.

TASK:

In the NYC SQL*Plus window execute:

o select member from v$logfile where type=‘STANDBY’;

SQL> select client_process,process,sequence#,status

2 from v$managed_standby;

CLIENT_P PROCESS SEQUENCE# STATUS

-------- --------- ---------- ------------



ARCH ARCH 0 CONNECTED


N/A MRP0 48 APPLYING_LOG

LGWR RFS 48 IDLE

UNKNOWN RFS 0 IDLE

UNKNOWN RFS 0 IDLE

ARCH RFS 0 IDLE

SQL> select member from v$logfile where type = 'STANDBY';

MEMBER

---------------------------------------------------------------

/u01/app/oradata/NYC/onlinelog/o1_mf_4_6rqcggh4_.log

/u01/app/fast_recovery_area/NYC/onlinelog/o1_mf_4_6rqcgglm_.log

/u01/app/oradata/NYC/onlinelog/o1_mf_5_6rqcgnd7_.log

/u01/app/fast_recovery_area/NYC/onlinelog/o1_mf_5_6rqcgnhs_.log

/u01/app/oradata/NYC/onlinelog/o1_mf_6_6rqcgvmn_.log

/u01/app/fast_recovery_area/NYC/onlinelog/o1_mf_6_6rqcgvr3_.log

/u01/app/oradata/NYC/onlinelog/o1_mf_7_6rqch4d9_.log

/u01/app/fast_recovery_area/NYC/onlinelog/o1_mf_7_6rqch4hx_.log

8 rows selected.

Oracle Data Guard

of 76

To see which standby redo log is currently being used you can query the view ‘v$standby_log’ and look for

‘ACTIVE’ in the ‘STATUS’ column.

TASK:


o select group#, sequence#, status from v$standby_log;

So from the previous mixture of SQL*Plus and DGMGRL exercises you can see how the Broker simplifies

things as you only had to log in once and use the same window to examine both databases in your Data Guard

configuration. We will discuss Redo Transport and the various processes associated with it later.

SQL> select group#,sequence#,status from v$standby_log;

GROUP# SEQUENCE# STATUS

---------- ---------- ----------

4 0 UNASSIGNED

5 48 ACTIVE

6 0 UNASSIGNED

7 0 UNASSIGNED

Oracle Data Guard

of 76

Managing a Broker Configuration

We can examine the details for each database in our configuration by typing ‘show database verbose <database

name>;’ which will show you the current status of the database and its current property settings. So for SFO we

would see the following (some properties edited out for space).

TASK:

In the DGMGRL window execute:

o show database verbose sfo;

DGMGRL> show database verbose sfo;

Database - sfo

Role: PRIMARY

Intended State: TRANSPORT-ON

Instance(s):

SFO

Properties:

DGConnectIdentifier = 'sfo'

ObserverConnectIdentifier = ''

LogXptMode = 'ASYNC'

DelayMins = '0'

…

ReopenSecs = '300'

NetTimeout = '30'

RedoCompression = 'DISABLE'

LogShipping = 'ON'

PreferredApplyInstance = ''

ApplyInstanceTimeout = '0'

ApplyParallel = 'AUTO'

StandbyFileManagement = 'MANUAL'

ArchiveLagTarget = '0'

LogArchiveMaxProcesses = '4'

LogArchiveMinSucceedDest = '1'

DbFileNameConvert = ''

LogFileNameConvert = ''

FastStartFailoverTarget = ''

InconsistentProperties = '(monitor)'

InconsistentLogXptProps = '(monitor)'

SendQEntries = '(monitor)'

LogXptStatus = '(monitor)'

RecvQEntries = '(monitor)'

…

TopWaitEvents = '(monitor)'

Database Status:

SUCCESS

Oracle Data Guard

of 76

And for NYC we would see the following (some properties edited out for space).

TASK:

In the same DGMGRL window execute:

o show database verbose nyc;

DGMGRL> show database verbose nyc;

Database - nyc

Role: PHYSICAL STANDBY

Intended State: APPLY-ON

Transport Lag: 0 seconds

Apply Lag: 0 seconds

Real Time Query: OFF

Instance(s):

NYC

Properties:

DGConnectIdentifier = 'nyc'

ObserverConnectIdentifier = ''


DelayMins = '0'

…

ReopenSecs = '300'

NetTimeout = '30'

RedoCompression = 'DISABLE'

LogShipping = 'ON'

PreferredApplyInstance = ''

ApplyInstanceTimeout = '0'

ApplyParallel = 'AUTO'

StandbyFileManagement = 'MANUAL'

ArchiveLagTarget = '0'

LogArchiveMaxProcesses = '4'

LogArchiveMinSucceedDest = '1'

DbFileNameConvert = ''

LogFileNameConvert = ''

FastStartFailoverTarget = ''

InconsistentProperties = '(monitor)'

InconsistentLogXptProps = '(monitor)'

SendQEntries = '(monitor)'

LogXptStatus = '(monitor)'

RecvQEntries = '(monitor)'

…

TopWaitEvents = '(monitor)'

Database Status:

SUCCESS

Oracle Data Guard

of 76

From your DGMGRL session you can and MUST effect all changes to your configuration. Once you start using

the Broker any changes you make to Data Guard parameters will be reset by the Broker to the last known value in

the Broker configuration. The list of ‘Properties’ you saw in the preceding exercise are mostly related to init.ora

parameters of the Oracle Database. For example, the Broker considers ‘log_archive_max_processes’ to be a

Data Guard parameter at this point so if we change it in SQL*Plus the Broker will report an error.

TASK:


o show parameter log_archive_max_processes;

o alter system set log_archive_max_processes=6 scope=both;

At this point, when the Broker performs the next health check it will report a parameter inconsistency between

what it think the value should be and what it actually is in the database.

TASK:


o show configuration

SQL> show parameter log_archive_max_processes

NAME TYPE VALUE

------------------------- ----------- -----------------------------

log_archive_max_processes integer 4

SQL> alter system set log_archive_max_processes=6 scope=both;

System altered.

DGMGRL> show configuration



Databases:


Warning: ORA-16792: configurable property value is

inconsistent with database setting




WARNING

Oracle Data Guard

of 76

You can just tell the Broker to fix the problem blindly (which it will do if the database is restarted) or you can

examine the parameter that is ‘inconsistent’. There are a few ‘properties’ in the Broker that are flagged as

‘monitor’ properties. One of these is ‘InconsistentProperties’. The following output is edited.

TASK:


o show database sfo InconsistentProperties;

To clear the error you can either edit the Broker configuration to set the Broker property value to 6 or you can tell

the Broker to set it back by using an ‘enable database’ command.

TASK:


o enable database sfo;


DGMGRL> show database sfo InconsistentProperties;

INCONSISTENT PROPERTIES

INSTANCE_NAME PROPERTY_NAME

SFO LogArchiveMaxProcesses

MEMORY_VALUE SPFILE_VALUE BROKER_VALUE

6 6 4

DGMGRL>

DGMGRL> enable database sfo;

Enabled.




Databases:





SUCCESS

Oracle Data Guard

of 76

If you go to your alert log window you will see that the Broker reset the parameter to what it considers the correct

value based on the configuration setting.

TASK:

Examine the SFO alert log output.

Likewise, any changes to the state of the transport or apply services must be performed with DGMGRL otherwise

you will see errors when you monitor your configuration. For example, stopping the redo apply services using

SQL*Plus instead of the Broker will result in an error state.

TASK:


o alter database recover managed standby database cancel;

SQL> alter database recover managed standby database cancel;

Database Altered

Thu Mar 31 06:05:20 2011

ALTER SYSTEM SET log_archive_trace=0 SCOPE=BOTH SID='SFO';

ALTER SYSTEM SET log_archive_format='%t_%s_%r.dbf' SCOPE=SPFILE

SID='SFO';

ALTER SYSTEM SET standby_file_management='MANUAL' SCOPE=BOTH

SID='*';

ALTER SYSTEM SET archive_lag_target=0 SCOPE=BOTH SID='*';

Thu Mar 31 06:05:20 2011

Shutting down archive processes

Thu Mar 31 06:05:20 2011

ARCH shutting down

ARC4: Archival stopped

Thu Mar 31 06:05:20 2011

ARCH shutting down

ARC5: Archival stopped

ALTER SYSTEM SET log_archive_max_processes=4 SCOPE=BOTH SID='*';

ALTER SYSTEM SET log_archive_min_succeed_dest=1 SCOPE=BOTH SID='*';

Oracle Data Guard

of 76

This will stop the redo apply services but the Broker expects them to be running so it will display an error.

TASK:



A simple ‘ENABLE DATABASE’ command will put things back the way they were supposed to be just like it

did with the parameter we set earlier. Or you can use the ‘EDIT’ command to restart the apply services.

TASK:


o edit database nyc set state=apply-on;




Databases:



Error: ORA-16766: Redo Apply is stopped



ERROR

DGMGRL>

DGMGRL> edit database nyc set state=apply-on;

Succeeded.

DGMGRL>

Oracle Data Guard

of 76

As in the parameter exercise, you can see the Broker restarting the Redo Apply services on the standby NYC by

examining the alert log.

TASK:

Examine the NYC alert log output.

To recap, you should never make any changes to a Broker controlled Data Guard configuration using SQL*Plus.

These are called ‘Out of Band changes’ and will always be flagged by the Broker as errors. To change a

parameter or the ‘state’ of anything in the configuration you must always use the ‘EDIT’ command of DGMGRL

or Enterprise manager. In the above case, to stop the redo apply services on NYC you would execute ‘edit

database nyc set state=apply-off;’ from DGMGRL.

Now that you have created, managed and monitored your Data Guard configuration it is time to move on to

modifying the ‘Redo Transport’ method and ‘Protection Modes’.

Thu Mar 31 06:19:00 2011

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL

SWITCHOVER DISCONNECT USING CURRENT LOGFILE

Attempt to start background Managed Standby Recovery process (NYC)

Thu Mar 31 06:19:00 2011

MRP0 started with pid=29, OS id=11750

MRP0: Background Managed Standby Recovery process started (NYC)

started logmerger process

Thu Mar 31 06:19:05 2011

Managed Standby Recovery starting Real Time Apply

Parallel Media Recovery started with 2 slaves

Waiting for all non-current ORLs to be archived...

All non-current ORLs have been archived.

Media Recovery Waiting for thread 1 sequence 80 (in transit)

Recovery of Online Redo Log: Thread 1 Group 5 Seq 80 Reading mem 0

Mem# 0: /u01/app/oradata/NYC/onlinelog/o1_mf_5_6rqcgnd7_.log

Mem# 1:

/u01/app/fast_recovery_area/NYC/onlinelog/o1_mf_5_6rqcgnhs_.log

Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH

ALL SWITCHOVER DISCONNECT USING CURRENT LOGFILE

Oracle Data Guard

of 76

Transport Services

Data Guard Redo Transport Services coordinate the transmission of redo from a primary database to the standby

database. At the same time that the primary database LGWR process is writing redo to its ORL, a separate Data

Guard process called the Network Server (NSS or NSA) is reading from the redo buffer in SGA and passing redo

to Oracle Net Services for transmission to the standby database. Data Guard’s flexible architecture allows a

primary database to transmit redo directly to a maximum of 30 standby databases. Data Guard is also well

integrated with Oracle RAC. Each primary instance that is active generates its own thread of redo and has its

own NSS/NSA process per standby database to transmit redo to that standby database.

Data Guard transport services transmit the redo to a standby database either synchronously or asynchronously,

where it is written to a standby redo log file (step one in the figure below).

The processes involved in redo transport are outlined in the next figure. Redo records transmitted by the

NSS/NSA are received at the standby database by another Data Guard process called the Remote File Server

(RFS). The RFS receives the redo at the standby database and writes it to a sequential file called a standby redo

log file (SRL). In a multi-standby configuration, the primary database has a separate NSS/NSA process that

manages redo transmissions for each standby database. In a configuration with three standby databases, for

example, three NSS/NSA processes are active on each primary database instance. As users commit transactions

at a primary database, Oracle generates redo records and writes them to a local online log file.

Oracle Data Guard

of 76

Synchronous redo transport (SYNC) causes the primary database to wait for confirmation from the standby

database that redo has been hardened to disk before it will acknowledge commit success to the application -

providing zero data loss protection. Primary database performance is impacted by the sum of the time required

for the standby redo log file I/O to complete and network round-trip time.

Data Guard 11g Release 2 is designed to reduce the impact to primary performance of synchronous transport.

Redo is now transmitted to the remote standby in parallel with the local online log file I/O on the primary

database, effectively eliminating standby I/O from impacting total round trip time. This is shown in the figure

below as Step 1 where the NSS (Network Server Synchronous) process sends the redo at the same time the

LGWR is queuing the I/O to the Online Redo Log (ORL) file. Once the LGWR has the acknowledgement from

the disk I/O and the Standby round trip, Step 2, it will release the commit to the user, Step 3.

Oracle Data Guard

of 76

This enables greater geographic separation between primary and standby databases in a synchronous zero data

loss configuration. On low latency networks it can reduce the impact of SYNC replication on primary database

performance to near zero, making it attractive to complement a remote ASYNC standby with a local SYNC

standby for zero data loss HA protection against component and database failures (SAN failure for example).

Asynchronous redo transport (ASYNC) avoids impact to primary database performance by having the primary

database acknowledge commit success to the application without waiting for acknowledgment that redo has been

received by the standby database. Generally there is no interaction between the NSA process and the LGWR

process other than startup. Data Guard 11g enhancements have virtually eliminated any impact on primary

database performance by shipping directly from the primary log buffer (instead of from an online redo log file),

as well as improving network throughput on high latency wide area networks (WAN). The performance benefit

of ASYNC, however, is accompanied by the potential for a small amount of data loss since there is no guarantee

that all redo has been received by the standby database.

Note: The asynchronous transport method is able to stream redo to a standby database so efficiently with no

impact to the Primary database that the old ‘ARCH’ method of shipping redo was deprecated in Oracle Database

11g and the default transport mode for Data Guard is ASYNC.

Oracle Data Guard

of 76

Modifying the Transport Method

The transport method is shown in the Broker configuration by the “LogXptMode = 'ASYNC'” property

which was displayed when you did the ‘show database verbose’ commands on the two databases. Like the rest of

the properties associated with each database, the LogXptMode is how you specify how redo should be sent to that

standby. The odd thing about this property is that it is never defined on a Standby in reality; it is part of the

‘LOG_ARCHIVE_DEST_n’ that is setup by the Broker on the Primary database to send redo to the standby

database. This is true for all the properties that are actually attributes of the ‘LOG_ARCHIVE_DEST_n’

parameter definition.

DelayMins

LogXptMode

MaxConnections

MaxFailure

NetTimeout

RedoCompression

ReopenSecs

For the purposes of this exercise we are just going to concentrate on the LogXptMode property. To begin with

let’s look at the property as it is configured at present.

TASK:


o show database sfo logxptmode;

o show database nyc logxptmode;

Since SFO is the Primary this property just shows how redo will be sent to SFO when it becomes a Standby. And

the property for NYC, our standby database, shows how the redo is currently being sent to NYC from SFO. You

can verify this by looking at the ‘LOG_ARCHIVE_DEST_n’ parameters as they are currently defined on SFO.

TASK:


o show parameter log_archive_dest_2

DGMGRL> show database sfo logxptmode;


DGMGRL> show database nyc logxptmode;


DGMGRL>

SQL> show parameter 'log_archive_dest_2'

log_archive_dest_2 string service="nyc", LGWR ASYNC NOAF

FIRM delay=0 optional compress

ion=disable max_failure=0 max_

connections=1 reopen=300 db_un

ique_name="nyc" net_timeout=30

, valid_for=(all_logfiles,prim

ary_role)

Oracle Data Guard

of 76

You can see the ‘LGWR ASYNC NOAFFIRM’ which corresponds to the current setting of ‘LogXptMode’ for

the standby ‘NYC, which is visible by the ‘service=”nyc” part of the parameter (the “nyc” is our tnsname we

created and entered as the connect identifier in exercise one). To change the transport mode you would edit the

property on the standby NYC, which will become effective immediately, and also for the Primary SFO, which

will become effective when SFO becomes the standby later on,. You want to change them both at the same time

otherwise you will end up with a configuration that is not the same after you switchover.

TASK:


o edit database sfo set property logxptmode=‘SYNC’;

o edit database nyc set property logxptmode=‘SYNC’;

This change to NYC will become immediately visible on the Primary SFO.

You can see the results of the action by examining the alert log file of the Primary SFO.

TASK:

Examine the SFO alert log output.

Changing the transport mode to ‘SYNC’ is the first step to configuring a ‘Zero Data Loss (ZDL)’ Data Guard

configuration but by itself will not provide ZDL. You must also modify the Protection Mode, something we will

do right after we discuss the last part of Transport Services, Automatic Redo Gap Resolution.

DGMGRL> edit database sfo set property logxptmode='SYNC';

Property "logxptmode" updated

DGMGRL> edit database nyc set property logxptmode='SYNC';

Property "logxptmode" updated

DGMGRL>

SQL> show parameter 'log_archive_dest_2'

log_archive_dest_2 string service="nyc", LGWR SYNC AFFIR

M delay=0 optional compression

=disable max_failure=0 max_con

nections=1 reopen=300 db_uniqu

e_name="nyc" net_timeout=30, v

alid_for=(all_logfiles,primary

_role)

Thu Mar 31 09:48:16 2011

ALTER SYSTEM SET log_archive_dest_2='service="nyc"','LGWR SYNC

AFFIRM delay=0 optional compression=disable max_failure=0

max_connections=1 reopen=300 db_unique_name="nyc"

net_timeout=30','valid_for=(all_logfiles,primary_role)' SCOPE=BOTH;

ALTER SYSTEM SWITCH ALL LOGFILE start (SFO)

Thu Mar 31 09:48:16 2011

Destination LOG_ARCHIVE_DEST_2 is SYNCHRONIZED

Oracle Data Guard

of 76

Automatic Redo Gap Resolution

A log file gap occurs whenever a primary database continues to commit transactions while the NSS or NSA

process has ceased transmitting redo to the standby database. This can occur whenever the network or the standby

database is down, depending on how you have chosen to implement your Data Guard configuration (discussed in

the section “Protection Modes” next). While in this state, the primary database LGWR process continues writing

to the current ORL, fills it, and then switches to a new ORL while an archive (ARCH) process archives the

completed ORL locally. This cycle can repeat itself many times over on a busy system before the connection

between the primary and standby is restored, creating a large log file gap.

Once the standby becomes available again, the ARCH process that has been designated as the ‘Standby Ping

Process’ will discover it and, as long as the specified reopen time (specified by the ‘ReopenSecs’ property) has

passed, immediately begin resolving any missing archive log files that were not sent to that standby. In addition

it will force a log switch which will get the NSS/NSA process for that standby restarted and connected to the

standby so that the current redo stream can also begin to be shipped to the standby.

Oracle Data Guard

of 76

The simplest way to create a redo gap is to shutdown the standby and then perform a couple of log switches at the

Primary database. These archive logs will not be able to be shipped to the standby and will be placed on the

‘Fetch Archive Log (FAL)’ queue to be shipped at a later time when the standby is reachable again. Before we

begin we will reduce the reopen time from the default of 5 minutes to 15 seconds to enable the gap to be resolved

faster.

TASK:


o edit database sfo set property reopensecs=‘15’;

o edit database nyc set property reopensecs=‘15’;

Now, to create the gap we will abort our standby NYC and switch logs twice at the Primary SFO.

TASK:


o shutdown abort

Then switch logs twice at the Primary database.

DGMGRL> edit database sfo set property reopensecs='15';

Property "reopensecs" updated

DGMGRL> edit database nyc set property reopensecs='15';

Property "reopensecs" updated

DGMGRL>

SQL> shutdown abort

ORACLE instance shut down.

SQL> select sequence# from v$log where status like 'CURRENT%';

SEQUENCE#

----------

86

SQL> alter system switch logfile;

System altered.


System altered.

SQL> select sequence# from v$log where status like 'CURRENT%';

SEQUENCE#

----------

88

SQL>

Oracle Data Guard

of 76

TASK:


o select sequence# from v$log where status like ‘CURRENT%’;

o alter system switch logfile;


o select sequence# from v$log where status like ‘CURRENT%’;

We now have a gap where Sequences 86 and 87 have not been shipped nor has the current redo stream in

sequence 88. Within 60 seconds of starting the standby, the ARCH ping process will discover it is back and

immediately begin gap resolution as well as forcing a log switch so that sequence 89 becomes the current stream

and sequence 88 gets sent as a gap after 86 and 87.

TASK:


o startup mount

When you look at the alert log for the standby you will see where those missing log files have been shipped and

applied and the current redo stream (89) is ‘(in transit)’. Your actual sequence numbers will vary.

TASK:

Examine the NYC alert log output.

Now that you understand how Redo transport works we will finish our modifications and change the Protection

Mode of our configuration to a Zero Data Loss mode.

SQL> startup mount

ORACLE instance started.

Total System Global Area 2438529024 bytes

Fixed Size 2228920 bytes

Variable Size 1358957896 bytes

Database Buffers 1056964608 bytes

Redo Buffers 20377600 bytes

Database mounted.

SQL>

/u01/app/fast_recovery_area/NYC/archivelog/2011_03_31/o1_mf_1_86_6s

951xwd_.arc

Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH

ALL SWITCHOVER DISCONNECT USING CURRENT LOGFILE

Media Recovery Log


951xxn_.arc

Media Recovery Log


951y54_.arc

Media Recovery Waiting for thread 1 sequence 89 (in transit)

Oracle Data Guard

of 76

Protection Modes

Data Guard provides three modes of data protection to balance cost, availability, performance, and data

protection. Each mode uses a specific redo transport method, and establishes rules that govern the behavior of the

Data Guard configuration should the primary database ever lose contact with its standby. The following table

outlines the characteristics of each mode.

DATA GUARD PROTECTION MODES

MODE RISK OF DATA LOSS TRANSPORT IF NO ACKNOWLEDGEMENT FROM THE STANDBY DATABASE, THEN:

Maximum Protection Zero data loss Double failure protection

SYNC Stall primary database until acknowledgement is received from the standby database

Maximum Availability Zero data loss Single failure protection

SYNC Stall primary database until acknowledgement is received or NET_TIMEOUT

threshold period expires – then resume processing

Maximum Performance Potential for minimal data loss

ASYNC Primary never waits for standby acknowledgment

Data Guard protection modes implement rules that govern how the configuration will respond to failures,

enabling you to achieve your specific objectives for data protection, availability, and performance. Data Guard

can support multiple standby databases in a single Data Guard configuration, and they may all have the same, or

different, protection mode setting, depending on your requirements. The different Data Guard protection modes

are Maximum Performance, Maximum Availability, and Maximum Protection.

Maximum Performance

This mode emphasizes primary database performance over data protection. It requires ASYNC redo transport so

that the LGWR process never waits for acknowledgement from the standby database. Primary database

performance and availability are not impacted by redo transport, by the status of the network connection between

primary and standby, or by the availability of the standby database.

Maximum Availability

This mode emphasizes availability as its first priority and zero data loss protection as a very close second priority.

It requires SYNC redo transport, thus primary database performance may be impacted by the amount of time

required to receive an acknowledgement from the standby that redo has been written to disk. SYNC transport,

however, guarantees 100-percent data protection during normal operation in the event that the primary database

fails. However, events that have no impact on the availability of the primary database can impact its ability to

transmit redo to the standby.

Maximum Protection

As its name implies, this mode places utmost priority on data protection. It has the same requirements as

Maximum Availability but the primary will not acknowledge a commit to the application unless it receives

acknowledgement from at least one standby database in the configuration that the data needed to recover that

transaction is safely on disk. If the primary does not receive acknowledgement from a SYNC standby database, it

will stall and eventually abort, preventing any unprotected commits from occurring.

Oracle Data Guard

of 76

Changing the Protection Mode

As you can see from this chart, we started with our configuration in ‘Maximum Performance’ mode using the

ASYNC transport. We then changed our transport mode to SYNC but until we change the protection mode to

either ‘Maximum Availability’ or ‘Maximum Protection’ we are not running in a Zero Data Loss mode. If we

had to failover now we would lose data.

Reducing the protection mode has never required a restart of the Primary database but increasing the protection

mode in a Data Guard configuration in older versions required that you bounce the Primary database as the

Protection mode is always controlled from the Primary where the redo is being generated. Now the protection

mode can be elevated or reduced without downtime. If you were not using the Broker you could execute the

SQL*Plus command to raise the Protection mode to Maximum Availability even if you had not yet met the

requirements, you would just be running at the resynchronization level until the requirements had been met.

Moving to Maximum Protection does require that all the requirements are met before the protection mode can be

raised to the highest level otherwise the Primary database would be aborted due to violating the rules.

The Broker requires that the prerequisites for a Protection Mode have been met before it will allow you to change

it. In our case, to get to a zero data loss mode, we need to change the transport mode to SYNC, which we have

done, and we need to have Standby Redo Log files (SRL) at the standby, which we also have since we created

them on the Primary before we created our standby. All that is left is to edit our configuration.

TASK:


o edit configuration set protection mode as MaxAvailability;


DGMGRL> edit configuration set protection mode as MaxAvailability;

Succeeded.



Protection Mode: MaxAvailability

Databases:





SUCCESS

Oracle Data Guard

of 76

Role Management Services

The next area of Data Guard architecture we will discuss are role management services that enable the rapid

transition of a standby database to the primary database role. Data Guard documentation uses the term switchover

to describe a planned role transition, usually for the purpose of minimizing downtime during planned

maintenance activities. The term failover is used to describe a role transition in response to unplanned events.

In both cases, Data Guard Role Management Services quickly transition a designated standby database to the

primary role.

Switchover

A switchover is a planned operation used to reduce downtime during planned maintenance, such as operating

system or hardware upgrades, rolling upgrades of the Oracle database, and other database maintenance.

Regardless of the transport service (SYNC or ASYNC) or protection mode utilized, a switchover is always a zero

data loss operation.

Performing a Switchover

If you were not using the Broker then a switchover would be a combination of several commands on both

databases and a manual database restart of the original primary. The order of the process would be as follows:

1. On the Primary database

a. Shutdown all but one instance of the Primary

b. alter database commit to switchover to standby with session shutdown;

c. shutdown immediate

2. On the Standby Database

a. alter database commit to switchover to primary;

b. alter database open;

3. On the original Primary (now a standby)

a. startup mount

b. alter database recover managed standby database using current logfile disconnect;

c. Startup the other instances

The Broker simplifies all of that process for you down into on command “switchover to <database name>;”




Databases:





SUCCESS

Oracle Data Guard

of 76

As far as the users are concerned, the switchover was finished when the Broker displayed “New primary database

"nyc" is opening...” as the new Primary is up and ready for business at that point. The rest of the time was spent

on getting the old primary up as a standby and Data Guard resynchronized. So, your primary database is now

NYC and SFO is your standby database as shown by another ‘show configuration’ command.

TASK:



o switchover to nyc;


You can verify this by examining the role of the NYC database in your SQL*Plus window.

TASK:


o select db_unique_name, database_role from v$database;

DGMGRL> switchover to nyc;

Performing switchover NOW, please wait...

New primary database "nyc" is opening...

Operation requires shutdown of instance "SFO" on database "sfo"

Shutting down instance "SFO"...


Operation requires startup of instance "SFO" on database "sfo"

Starting instance "SFO"...


Database mounted.

Switchover succeeded, new primary is "nyc"




Databases:

nyc - Primary database

sfo - Physical standby database



SUCCESS

SQL> select db_unique_name, database_role from v$database;

DB_UNIQUE_NAME DATABASE_ROLE

--------------- ------------------

NYC PRIMARY

Oracle Data Guard

of 76

Your original SQL*Plus connection to SFO has been lost because the Broker had to restart SFO as part of the

switchover process. So to examine the role of SFO you will have to connect again.

TASK:



o connect sys/oracle as sysdba


To return to our original configuration, perform a 2nd

switchover and make SFO the Primary database again.

TASK:


o switchover to sfo

Note: Don’t forget to reconnect your NYC SQL*Plus window!


select db_unique_name, database_role from v$database

*

ERROR at line 1:

ORA-03135: connection lost contact

Process ID: 11282

Session ID: 9 Serial number: 5

SQL> connect sys/oracle as sysdba

Connected.



--------------- ------------------

SFO PHYSICAL STANDBY

SQL>

DGMGRL> switchover to sfo;


New primary database "sfo" is opening...

Operation requires shutdown of instance "NYC" on database "nyc"

Shutting down instance "NYC"...


Operation requires startup of instance "NYC" on database "nyc"

Starting instance "NYC"...


Database mounted.

Switchover succeeded, new primary is "sfo"

Oracle Data Guard

of 76

Failover

A failover brings a standby database online as the new primary database during an unplanned outage of the

primary database. As with a switchover, a failover operation does not require the standby database to be restarted

in order to assume the primary role. Also, as long as the database files on the original primary database are intact

and the database can be mounted, the original primary can be reinstated and resynchronized as a standby database

for the new primary using Flashback Database – it does not have to be restored from a backup.

Manual failover is initiated by the administrator using the Oracle Enterprise Manager GUI interface, the Data

Guard Broker’s command line interface, or directly through SQL*Plus. Optionally, Data Guard can perform

automatic failover in a very controlled manner using Fast-Start Failover (FSFO).

Enabling Flashback Database

Flashback Database is part of the Oracle Flashback technologies and is basically a ‘rewind’ button for an Oracle

database. For more information on the various flashback technologies including Flashback Database, please refer

to the Oracle documentation.

Flashback Database plays a very important part in any Data Guard configuration as it allows you to ‘reinstate’ a

failed Primary database as a standby database after a failover has occurred. In many cases it is not a complete

failure of the Primary databases that necessitates a failover. It might have been caused by power, system or site

failures but the actual database files are fine. Or it might have been caused by corruption to some of the data files

of the Primary but the rest of the database is not corrupted. In any case though, without Flashback Database

enabled on the Primary database prior to a failover operation, you would be required to completely rebuild a new

standby database from a backup of the new Primary (aka the old Standby). To reinstate a failed primary database

Data Guard requires 60 minutes of flashback retention time. If you plan on using Flashback Database for more

than just reinstating failed primary databases you can set the retention time to whatever you need.

Flashback database is not mandatory for Data Guard configurations (although strongly recommended) unless you

plan on using the automatic failover capability of the Broker (FSFO). If so, then it is mandatory on the Primary

and the failover target and you will not be able to enable FSFO without it. As FSFO is the last exercise in this lab

we will enable it now on both SFO and NYC.

First set the retention time to 60 minutes on both databases as the default retention period is 24 hours..

TASK:

In the both the SFO and NYC SQL*Plus windows execute:

o show parameter db_flashback_retention_target

o alter system set db_flashback_retention_target=60 scope=both;

SQL> show parameter db_flashback_retention_target

NAME TYPE VALUE

------------------------------ ----------- ------------------------

db_flashback_retention_target integer 1440

SQL> alter system set db_flashback_retention_target=60 scope=both;

System altered.

Oracle Data Guard

of 76

Now we’re ready to enable Flashback Database on both of our databases. In previous versions of Oracle it was

required that the database be in the mount state to enable flashback. With Oracle Database 11g Release 2 (11.2)

this is no longer necessary.

TASK:


o select flashback_on from v$database;

o alter database flashback on;

o select flashback_on from v$database;

However, the standby redo apply services must be stopped first on the standby before you can enable flashback

database. DGMGRL has a SQL command capability but it requires that your session is connected to the database

where you wish to execute the SQL command. Since we want to change the standby we will reconnect to NYC.

TASK:


o connect sys/oracle@NYC

o edit database nyc set state=apply-off;

o sql “alter database flashback on”;

o edit database nyc set state=apply-on;

SQL> select flashback_on from v$database;

FLASHBACK_ON

------------------

NO

SQL> alter database flashback on;

Database altered.

SQL> select flashback_on from v$database;

FLASHBACK_ON

------------------

YES

DGMGRL> connect sys/oracle@NYC

Connected.

DGMGRL> edit database nyc set state=apply-off;

Succeeded.

DGMGRL> sql "alter database flashback on";

Succeeded.

DGMGRL> edit database nyc set state=apply-on;

Succeeded.

DGMGRL>

Oracle Data Guard

of 76

To ensure that there is sufficient information in the Flashback logs both databases, switch log files at the Primary

database SFO.

TASK:



Now we are ready to perform a failover and force NYC to become the Primary database.

Performing a Manual Failover

As with Switchover, if you were not using the Broker then a failover would be several commands that you would

execute on the Standby. The order of the process would be as follows:

1. On the Primary database

a. Nothing, it’s gone!

2. On the Standby Database

a. alter database recover managed standby database finish;

b. alter database commit to switchover to primary;

c. alter database open;

The Broker simplifies all of that process for you down into on command “failover to <database name>”.

To simulate our failure we will abort the Primary and then failover to the standby.

TASK:


o shutdown abort

Panic ensues! Production is down! Never fear, Data Guard is here! You can examine the state of your

configuration with the usual ‘show configuration’ command. The error will be a TNS error because the Broker

has lost its connection with the Primary database. Remember, in the last exercise we reconnected our DGMGRL

session to the standby NYC. If you forgot to do that then you will not be able to examine the state of the

configuration since SFO is down!


System altered.

SQL>

SQL> shutdown abort


SQL>

Oracle Data Guard

of 76

TASK:


o show configuration

This error could be due to the primary being down or just a network disconnect between the Standby and the

Primary. You have to make the decision whether or not to failover. And if you decide to manually failover you

have to be sure that the Primary is actually down and that you prevent it from starting up normally later on

otherwise you will have two Primary databases.

TASK:


o failover to nyc;

You can verify this by examining the database role of NYC.

TASK:


o select db_unique_name,database_role from v$database;




Databases:





ORA-12514: TNS:listener does not currently know of service

requested in connect descriptor

ORA-16625: cannot reach database "sfo"

DGM-17017: unable to determine configuration status

DGMGRL> failover to nyc;

Performing failover NOW, please wait...

Failover succeeded, new primary is "nyc"

DGMGRL>

SQL> select db_unique_name,database_role from v$database;


--------------- ------------------

NYC PRIMARY

Oracle Data Guard

of 76

As with switchover, the failover is complete for the users as soon as the new Primary database is open for

business. In a failover there is no more work to be done since the original Primary database is gone. To get the

old Primary back now as a standby we will have to perform a manual ‘reinstate’.

Performing a Manual Reinstate

A manual reinstate is a sequence of commands that you execute on both the new Primary and the old Primary

using Flashback Database to rewind the failed primary database to a point just before the failover occurred. Then

it is converted into a Physical standby, restarted and the Redo Apply services restarted. At that point Data Guard

will reconnect and resolve the gaps in the redo until the configuration is synchronized again.

The important thing to remember is not to allow the old Primary database to startup normally. There is nothing

in place to stop it from opening read write. Fast-Start Failover is the only thing that can stop that from

happening. So our first step to reinstating the old Primary is to start it up in the MOUNT state.

TASK:


o startup mount

Because we are using the Broker we really just have one command to put the old Primary back as a standby.

Looking at the configuration again you can see that the Broker is telling you to reinstate SFO.

SQL> startup mount







Database mounted.




Databases:


Warning: ORA-16629: database reports a different protection

level from the protection mode

sfo - Physical standby database (disabled)

ORA-16661: the standby database needs to be reinstated



WARNING

Oracle Data Guard

of 76

TASK:



The ORA-16629 above is due to the fact that we are in Maximum Availability mode but have no SYNC standby

to support a Zero Data Loss mode so we are in error. The ORA-16661 tells us why. All that is left is to reinstate

SFO and then, when the configuration is synchronized again, switchover from NYC to SFO to return to our

original Production configuration.

TASK:


o reinstate database sfo;

Once more show configuration to ensure that the configuration is resynchronized and then you can switchover

again returning SFO to the Primary status.

TASK:



DGMGRL> reinstate database sfo;

Reinstating database "sfo", please wait...



ORA-01109: database not open

Database dismounted.





Database mounted.

Continuing to reinstate database "sfo" ...

Reinstatement of database "sfo" succeeded

DGMGRL>




Databases:


sfo - Physical standby database



SUCCESS

Oracle Data Guard

of 76

TASK:


o switchover to sfo;

TASK:



Now that you have gotten this far into this lab book, it is time to take a walk on the wild side. If you thought that

we had a reason talking about the Broker in these exercise you are right. We wanted you to see how the Broker

can simplify the management of Data Guard. But we had an ulterior motive as well. And that is the subject of

our last exercise for this lab – Fast-Start Failover.

DGMGRL> switchover to sfo;


New primary database "sfo" is opening...

Operation requires shutdown of instance "NYC" on database "nyc"

Shutting down instance "NYC"...


Operation requires startup of instance "NYC" on database "nyc"

Starting instance "NYC"...


Database mounted.

Switchover succeeded, new primary is "sfo"

DGMGRL>




Databases:





SUCCESS

Oracle Data Guard

of 76

Fast-Start Failover (FSFO)

Over the years, database administrators have been confronted with the requirement of automating as much as

possible of their disaster recovery plans so that anyone on duty can engineer a move to the disaster recovery site

at the drop of a hat. Everyone is afraid that if only one person knows how to recover the database, that person

won’t be around when the failure happens. However, despite many people’s best efforts and no matter how

scripted you make the process, you still need people to get involved before the transition can take place. Once

people get involved, decisions have to be made, doubt is thrown into the mix, investigations occur, phone calls

happen, and, usually, errors are made. These are not things you want to happen when your databases are down

and your business is on hold. That is why Data Guard implemented Fast-Start Failover. And Fast-Start Failover

requires the Broker. Fast-Start Failover (FSFO) removes the people from the equation. That is not as harsh as it

sounds. It means that when there is a problem, there is no room for error, and people make mistakes, especially

when they are stressed. It also means that you do not end up relying on a script or document that was written by

someone who may no longer work for the company.

FSFO allows Data Guard to automatically fail over to a previously chosen, standby database without requiring

manual intervention to invoke the failover. A Data Guard Observer process continuously monitors the status of a

Fast-Start Failover configuration. If both the Observer and the standby database lose connectivity to the primary

database, the Observer attempts to reconnect to the primary database for a configurable amount of time before

initiating a fast-start failover. Fast-start failover is designed to ensure that out of the three fast-start failover

members - the primary, the standby and the Observer - at least two members agree to major state transitions to

prevent split-brain scenarios from occurring. Once the failed primary is repaired and mounted, it must establish

connection with the Observer process before it can open. When it does, it will be informed that a failover has

already occurred and the original primary is automatically reinstated as a standby of the new primary database.

The simple, yet elegant architecture of fast-start failover makes it excellent for use when both high availability

and data protection is required.

The architecture employed by FSFO is a third-member quorum that ensures that the failover occurs only when

everything meets the rules you have defined, and the failed primary is never allowed to open read write after a

failover to avoid any chance of the split brain scenario. The main two members are the primary database and the

standby database, of course, running in a normal configuration setup as we have setup in this lab.

.

The third member is called the Observer, and its job is to maintain a connection with the primary and the target

standby, monitoring the health of the configuration and performing the failover when required. The Observer is

also responsible for performing the automatic reinstatement of the failed primary database when it comes back

online, if possible. It has rules on where it should be placed but it does not require an Oracle license nor does it

have to be on the same platform or operating system as the databases.

The first thing you need to do is decide where you are going to put the Observer. In our case since we only have

one system we’re going to put it in its own window in the ‘Observer’ workspace of your desktop. In reality you

would want to place it on a 3rd

system with redundant network paths to the Primary and the Standby.

Oracle Data Guard

of 76

Enabling Fast-Start Failover

Now that you have decided where you are going to place the Observer and prepared the system for it, you can

start setting up FSFO. You must meet several prerequisites before setting up FSFO, most of which we have

implemented already:

Use the broker with all of its prerequisites. Done!

Enable Flashback Database on both the primary and the standby. Done!

Set up the configuration correctly for the protection mode. Done!

Standby redo log files on both sides. Done!

Redo transport setup the same in both directions. Done!

Configure the Broker for FSFO. Not Done!

Install the Observer system and configure TNSNAMES. Not Done!

The first thing we can set is the ‘FastStartFailoverTarget’ property for both databases so that no matter which

database is the Primary the other is the FSFO Target standby. Since we only have one standby this is automatic

but we will execute the commands anyway.

TASK:


o edit database sfo set property FastStartFailoverTarget = nyc;

o edit database nyc set property FastStartFailoverTarget = sfo;

We will also set the ‘FastStartFailoverThreshold’ to enable our failover to happen faster. The actual setting for

this property depends on your configuration (if the Primary is a RAC you have to add cluster miscount and

reconfiguration time to the threshold) and how long you want the Primary to stall before aborting and a failover

occurring. There are other properties that relate to FSFO but we’ll just accept the defaults for them.

TASK:


o edit configuration set property FastStartFailoverThreshold=30;

DGMGRL> edit database sfo set property FastStartFailoverTarget = nyc;

Property "faststartfailovertarget" updated

DGMGRL> edit database nyc set property FastStartFailoverTarget = sfo;

Property "faststartfailovertarget" updated

DGMGRL>

DGMGRL> edit configuration set property FastStartFailoverThreshold=30;

Property "faststartfailoverthreshold" updated

DGMGRL>

Oracle Data Guard

of 76

Now we are ready to enable Fast-Start Failover although it will not be ready to go until we start the observer.

TASK:


o enable fast_start failover;

o show fast_start failover;

Finally we start the Observer in its own window and things will be ready to go! In the Observer workspace of

your desktop, or in new terminal window somewhere, connect DGMGRL to the Primary or Standby.

TASK:

In the OBSERVER window execute:

o dgmgrl sys/oracle

o start observer;

DGMGRL> enable fast_start failover;

Enabled.

DGMGRL> show fast_start failover;

Fast-Start Failover: ENABLED

Threshold: 30 seconds

Target: nyc

Observer: (none)

Lag Limit: 30 seconds (not in use)

Shutdown Primary: TRUE

Auto-reinstate: TRUE

Configurable Failover Conditions

Health Conditions:

Corrupted Controlfile YES

Corrupted Dictionary YES

Inaccessible Logfile NO

Stuck Archiver NO

Datafile Offline YES

Oracle Error Conditions:

(none)

bash-3.2$ dgmgrl sys/oracle

DGMGRL for Linux: Version 11.2.0.2.0 - 64bit Production


Welcome to DGMGRL, type "help" for information.

Connected.

DGMGRL> start observer

Observer started

Oracle Data Guard

of 76

The observer runs in a foreground window and will not stop until you execute a ‘stop observer’ from another

DGMGRL session. Enterprise Manager offers high availability for the Observer. In our manual mode we would

have to script it to restart if it fails.

That’s it; FSFO is cocked and ready to go. To verify that all is well we’ll examine the Fast-Start Failover setup

in our configuration.

TASK:


o show fast_start failover;

As you can see there are other properties you can set for a FSFO configuration that we have not explicitly set.

Refer to the Data Guard Broker manual for more information on these properties.

We are now ready for the last exercise. An Automatic Failover!

DGMGRL> show fast_start failover;


Threshold: 30 seconds

Target: nyc

Observer: ip-10-124-233-12

Lag Limit: 30 seconds (not in use)

Shutdown Primary: TRUE

Auto-reinstate: TRUE

Configurable Failover Conditions

Health Conditions:

Corrupted Controlfile YES

Corrupted Dictionary YES

Inaccessible Logfile NO

Stuck Archiver NO

Datafile Offline YES

Oracle Error Conditions:

(none)

Oracle Data Guard

of 76

Executing an Automatic Failover

Remember how we said that FSFO takes the ‘person’ out of a failover? Well, it does but we need you to crash

the Primary. As you will see in the manual, there are several events that will trigger a Fast-Start Failover.

The usual way that a failover will be triggered is when the Observer and the target standby lose their connections

with the primary database, but the Observer and the standby database are still communicating. This can be a

database crash, a system crash, the loss of the network, or an entire primary site outage—as in a complete power

failure of the locality where the primary is running. The threshold will be honored and the failover will not

commence until the timer has expired. Since this is an unexpected event and there is no information as to the

health of the primary database, the Observer will attempt to reinstate the failed database automatically as a

standby when it is restarted.

You can also force a FSFO by performing a SHUTDOWN ABORT on the primary database. Of course, if the

primary is a RAC, a SHUTDOWN ABORT will only cause a failover if the instance being aborted is the last

surviving instance and your FastStartFailoverThreshold is exceeded. As with the normal lost connection event,

the Observer will reinstate the failed primary when it comes back up.

In addition, you can configure FSFO conditions that the Broker will watch for, and, if it sees one of them

happening, FSFO will initiate an immediate abort of the primary and a failover as long as the requirements for the

failover have been met. These are set in the Broker using the ENABLE command and there are five specific

conditions you can configure which are documented in the Broker manual.

So, we’re going to simulate a failure with a SHUTDOWN ABORT.

TASK:



To verify that we really are on the Primary

o shutdown abort

That will trigger a failover to NYC since the Broker (the Observer) recognizes immediately that our last

surviving Primary instance has crashed. It will wait until our threshold has passed and then it will make NYC the

Primary database ‘automatically’!



--------------- ------------------

SFO PRIMARY

SQL> shutdown abort


SQL>

Oracle Data Guard

of 76

If you return to the Observer window you can observe the Observer actually performing the failover.

TASK:

In the OBSERVER window

o Watch the Observer performing the failover.

Returning to the DGMGRL window you can see the current state of the configuration, NYC as the Primary and

SFO needing reinstatement.

TASK:



This is exactly the same state we were in when we performed our manual failover, NYC is the Primary and SFO

needs to be reinstated. The difference is (and it is a big one) that the Primary cannot open read write no matter

how it is started and the Observer will automatically reinstate it as a standby when it does restart.

DGMGRL> start observer

Observer started

11:41:17.61 Friday, April 01, 2011

Initiating Fast-Start Failover to database "nyc"...

Performing failover NOW, please wait...

Failover succeeded, new primary is "nyc"

11:41:21.07 Friday, April 01, 2011




Databases:


Warning: ORA-16817: unsynchronized fast-start failover

configuration

sfo - (*) Physical standby database (disabled)

ORA-16661: the standby database needs to be reinstated



WARNING

Oracle Data Guard

of 76

Return to your SFO SQL*Plus window and execute a ‘startup’, which normally would open the database read

write (since it is still a Primary database at this point).

TASK:


o startup

As you can see, the Broker knows this database was in a FSFO enabled configuration and will prevent it from

opening. Going back to the Observer window you will see that the Observer has detected that SFO is back and

automatically reinstates it as a standby.

TASK:

In the OBSERVER window

o Watch the Observer performing the reinstate.

SQL> startup







Database mounted.

ORA-16649: possible failover to another database prevents this

database from being opened

11:49:26.29 Friday, April 01, 2011

Initiating reinstatement for database "sfo"...

Reinstating database "sfo", please wait...



ORA-01109: database not open

Database dismounted.





Database mounted.

Continuing to reinstate database "sfo" ...

Reinstatement of database "sfo" succeeded

11:50:33.27 Friday, April 01, 2011

Oracle Data Guard

of 76

At this point we are back in business and FSFO is ready to go if NYC should fail. You can check the

configuration with another show configuration command in the DGMGRL window.

TASK:



Success! You can leave things like they are and move production back to SFO when NYC fails or you can

perform a switchover to move production from NYC back to SFO. Of course you could also do a SHUTDOWN

ABORT on NYC and let the Observer move production to SFO.

This ends our exercises with Data Guard but feel free to redo or play around with your configuration.




Databases:


sfo - (*) Physical standby database



SUCCESS

Oracle Data Guard

of 76

Conclusion

This concludes the exercises for the Oracle Database 11g Data Guard Hands On Lab. During this session you

have explored and exercised Data Guard’s unique capabilities for protecting your data. You were able to create,

manage and use Data Guard standby databases in a variety of situations.

You have completed the following exercises.

Created a Physical Standby Database

Configured the Data Guard Broker

Edited Broker Properties

Changed the Protection mode

Performed a Switchover

Performed a Manual Failover

Performed a Manual Reinstate.

Enabled and used Fast-Start Failover.

It is our sincere hope that these exercises have helped to guide you on your path to using Data Guard and

understanding how to use the features that it brings to the Oracle Database world to provide you with Disaster

Protection for your Oracle data.

Thank you for your participation.

Larry M. Carpenter

Oracle Data Guard

of 76

Oracle Data Guard

of 76

Resources

The following documentation is available for Data Guard

Oracle® Data Guard Concepts and Administration 11g Release 1 (11.1)

o http://download.oracle.com/docs/cd/B28359_01/server.111/b28294/toc.htm

Oracle® Data Guard Broker 11g Release 1 (11.1)

o http://download.oracle.com/docs/cd/B28359_01/server.111/b28295/toc.htm

Oracle® Data Guard Concepts and Administration 11g Release 2 (11.2)

o http://download.oracle.com/docs/cd/E11882_01/server.112/e10700/toc.htm

Oracle® Data Guard Broker 11g Release 2 (11.2)

o http://download.oracle.com/docs/cd/E11882_01/server.112/e10702/toc.htm

The Maximum Availability Architecture Best Practices papers are on Oracle OTN.

Best Practices for High Availability -- Maximum Availability Architecture (MAA)

o http://www.oracle.com/goto/maa

Best Practices for Data Guard

o http://www.oracle.com/goto/dataguard

http://download.oracle.com/docs/cd/B28359_01/server.111/b28294/toc.htm




http://www.oracle.com/goto/maa

http://www.oracle.com/goto/dataguard

Oracle Data Guard

of 76

Oracle Data Guard

of 76

Appendix A – Create Standby Script

MakeNYC #!/bin/bash

. oraenv <<EOF

NYC

EOF

#Abort the NYC instance in case it was running from a prior attempt


sqlplus -s '/ as sysdba' <<EOF

SHUTDOWN ABORT

EOF

# Perform some clean up in case this is a 2nd run.

rm -f /u01/app/oracle/product/11.2.0/db_1/dbs/*NYC*

rm -f /u01/app/oracle/product/11.2.0/db_1/dbs/tempini.ora

# Create temporary password file, must have same password as primary

orapwd file=/u01/app/oracle/product/11.2.0/db_1/dbs/orapwNYC password=oracle

# Create temporary parameter file


db_name=anyname

EOF


sqlplus '/ as sysdba' <<EOF

SET ECHO ON

STARTUP PFILE='/u01/app/oracle/product/11.2.0/db_1/dbs/tempini.ora' NOMOUNT;

EOF

# Run RMAN to duplicate your database.

rman <<EOF

set echo on

connect target sys/oracle@SFO; # Primary

connect auxiliary sys/oracle@NYC; # Standby: tnsname used by primary

run {

# Create channels for parallism






allocate auxiliary channel stby1 type disk;

duplicate target database for standby from active database

spfile

parameter_value_convert 'SFO','NYC'

set 'db_unique_name'='NYC'

nofilenamecheck

;

}

EOF

exit 0

Oracle Data Guard

of 76

Data Guard Hands on Lab

January 2013

Authors: Larry M. Carpenter

Oracle USA

World Headquarters

500 Oracle Parkway

Redwood Shores, CA 94065

U.S.A.

Worldwide Inquiries:

Phone: +1.650.506.7000

Fax: +1.650.506.7200

oracle.com

Copyright © 2010, Oracle. All rights reserved.

This document is provided for information purposes only and the contents hereof are subject to change without

notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether

expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a

particular purpose. We specifically disclaim any liability with respect to this document and no contractual

obligations are formed either directly or indirectly by this document. This document may not be reproduced or

transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written

permission. Oracle, JD Edwards, and PeopleSoft are registered trademarks of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

distinguished product manager - oracle€¦ · oracle data guard page 13 of 76 oracle data guard...

Documents