deploying oracle 11g rac r2 on aix6 using raw devices with powerha

Materials may not be reproduced in whole or in part without the prior written permission of IBM. 5.3

© Copyright IBM Corporation 2011

IBM Systems and Technology Group Asia Pacific Technical Enablement Conference

Deploying Oracle RAC 11g R2 on AIX6 using raw devices with PowerHA

SeongLul Son – [email protected]

2© Copyright IBM Corporation 2011

IBM STG Technical Enablement Conference 2011

Agenda

• Architecture changes from Oracle RAC 11g R2• Configure pre-requisites of OS• Install Oracle grid infrastructure with ASM using OUI • Generate scripts with DBCA to create a database• Edit scripts and create a database on raw devices(with

PowerHA) • Convert ocr and vote to raw devices(with PowerHA)• Disabling ASM and scan IP

2

Note: This presentation is based on one of the real reference cases. This scenario can be applied if a customer understands the limitation of using raw devices for ocr, vote in OUI and datafiles in DBCA from Oracle 11g RAC R2, and still wants to utilize raw devices without using migration method . You need to acquire customers’ understanding and consent before applying this method.



3

Architecture changes

Raw devices are not supported for OCR and vote devices with OUI

Raw devices are not supported for database files when creating a database with DBCA

SCAN(Singgle Client Access Name) IP is required when setting up Oracle Grid Infrastructure



4

Architecture changes - shared storage option

Storage option OCR and vote disk Oracle Software Oracle database files

ASM Yes No Yes

GPFS Yes Yes Yes

Raw device (w/ PowerHA)

Yes: Upgrade from previous versions

No: New install with OUI

No Yes: Upgrade from previous versions

No: New creation with DBCA

Local file system No Yes No

Storage option OCR and vote disk Oracle Software Oracle database files

ASM No No Yes

GPFS Yes Yes Yes

Raw device (w/ PowerHA)

Yes No yes

Local file system No Yes No

Oracle RAC10g ~ 11g R1

Oracle RAC 11gR2



Agenda



5



6

Memory and Paging space Requirements

Memory1.5GB for Oracle Grid Infrastructure

2.5GB for Oracle Grid Infrastructure and Oracle RAC

Swap spaceMinimum: 1.5GB

System with 2GB of RAM or less: 1.5 times the amount of RAM

System with 2GB to 16GB of RAM: equal to RAM

System with more than16GB of RAM: 16GB



7

Consideration on OCR and vote devices

With normal redundancy3 OCR and 3 vote devices are needed

each device needs at least 280 MB in size

OCR and vote on raw devicesOCR and vote on raw devices are not supported from 11gR2

ASM or cluster file system are supported



8

Operating System Requirements

Operating system- AIX 7.1 Base - AIX 6.1 TL02 SP1- AIX 5.3 TL09 SP1

Operating system filesets (for AIX6.1)opensshbos.adt.basebos.adt.libbos.adt.libmbos.perf.libperfstatbos.perf.perfstatbos.perf.proctoolsrsct.basic.rtersct.compat.clients.rtexlC.aix61.rte 10.1.0.0



9

Name Resolution

# cat /etc/hosts

127.0.0.1 loopback localhost # loopback (lo0) name/address

130.130.10.115 node15

130.130.10.116 node16

192.168.10.115 node15rac

192.168.10.116 node16rac

130.130.10.215 node15vip

130.130.10.216 node16vip

130.130.10.15 node15scan

Scan IP (Single client access name)-Host name to provide for all clients connecting to the cluster

-The SCAN is a domain name registered to at least one and up to three IP addresses, either in the domain name service (DNS) or the Grid Naming Service (GNS).

-The SCAN eliminates the need to change clients when nodes are added to or removed from the cluster.



10

Configure OS kernel parameters

Tuning Virtual Memory Manager# vmo -p -o minperm%=3# vmo -p -o maxperm%=90# vmo -p -o maxclient%=90# vmo -p -o lru_file_repage=0# vmo -p -o strict_maxclient=1# vmo -p -o strict_maxperm=0

Parameter Value

minperm% 3

maxperm% 90

maxclient% = 90 90

lru_file_repage 0

strict_maxclient 1

strict_maxperm 0

Note : From AIX6.1, these are all default values



11

Configure AIX System Environment

Configuring Shell Limits

- /etc/security/limitsdefault:

fsize = -1data = -1core = -1fsize_hard = -1cpu_hard = -1data_hard = -1stack_hard = 65536core_hard = -1

Shell Limit Item in limits.conf Hard Limit

Maximum number of open file descriptors

nofile -1

Maximum number of processes available to a single user

maxuproc 16384



12

Tuning AIX System Environment

Configuring Network Tuning Parameters Parameter Value

udp_sendspace 65536

udp_recvspace 655360(udp_sendspace * 10)

tcp_sendspace 65536

tcp_recvspace 65536

rfc1323 1

sb_max 4194304

ipqmaxlen 512



13

Users and Groups

Configuring usersoragrid: create a user for installing the Oracle Grid Infrastructure softwareoracle: user for installing the Oracle RAC software and creating database

From 11g, run the following command as root# chuser capabilities=CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE oragrid# chuser capabilities=CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE oracle

# lsuser -a capabilities oragrid (checking capabilities for oragrid )

Configuring groupsdba: primary group for the oragrid and oracle userhagsuser: Required in PowerHA environment



14

Network Time Protocol

Time synchronization is required from Oracle Clusterware 11g Release 2. Choose one of two time synchronization services; Network time protocol (NTP) in AIX or Cluster Time Synchronization Service (CTSS) in oracle.

NTP in AIX-edit /etc/ntp.conf -stop ntpd and restart it with the –x option

# startsrc -s xntpd -a "-x“- edit /etc/rc.tcpip to add the –x option to xntpd on all nodes

start /usr/sbin/xntpd "$src_running" "-x“Note: -x option prevents time from being adjusted backward

Oracle CTSS -Installed in active mode if NTP daemon in AIX is not active-Started in observer mode if NTP is active-In order to confirm that ctssd is active, run the following command$ crsctl stat resource ora.ctssd -t -init



15

Asynchronous I/O

AIX 6.1 or later-AIO device drivers are enabled by default-Increase the number of aioserver processes to 65536 #ioo –p –o aio_maxreqs=65536

AIX 5.x-rootpre.sh enables AIO device driver-use smit aio to change the maxreqs parameter



16

ssh configuration

Required from Oracle Clusterware 11g Release 2.

Manual configuration -configure ssh manually for the grid(oragrid) and oracle user-Need to login to other nodes without prompt for password

Automatic configuration by oracle-OUI creates passwordless ssh connectivity if there is no pre-configured ssh configuration



17

Prepare a disk for ASM

Prepare a disk and change ownership and permission for ASM



18

Create raw devices for OCR, vote and database files

Create a volume group and configure raw devices that PowerHA will varyonconcurrently



19

Change ownership and permission (OCR and vote)

Change ownership and permission of raw devices for OCR and vote



20

Change ownership and permission (Database files)

Change ownership and permission of raw devices for database files



21

More steps before running runInstaller.sh

Configure PowerHA with a concurrent volume group and start PowerHAConfigure graphic environment on AIX (VNC in our test environment)Run cluster verification utility (runclvfy.sh)Execute rootpre.sh as root on both nodes



Agenda



22



23

Install Oracle grid infrastructure software

Run ./runInstaller.sh as an oragrid user and type ‘y’ if you have already run rootpre.sh on both nodes



24

Select installation option

Select ‘Install and Configure Grid Infrastructure for a Cluster’



25

Select installation type

Select ‘Advanced installation’



26

Select product language

Select languages in which your product will run



27

Grid plug and play information

Scan Name must be resolved by DNS or /etc/hosts file

We changed the default SCAN port.



28

Cluster node information

Provide cluster node information



29

Specify network interface usage

Verify interface type and make changes if necessary



30

Storage option information

There is no option to choose raw devices for OCR and vote in OUI. Select ‘ASM’ and we will convert them to raw devices later.



31

Create ASM disk group

Select disk group characteristics and select disks.



32

Specify ASM password

We type ‘oracle’ for ASM password.



33

Specify ASM password

It is prompted due to short password length. Just continue.



34

Privileged operating system groups

Select operating system groups for ASM



35

Privileged operating system group

Click ‘Yes’



36

Specify installation location

Select directories for oracle base and grid infrastructure



37

Create inventory

Select a directory for oracle inventory



38

Perform prerequisite checks

Verity if system requirements are configured correctly



39

Summary

Browse the Summary



40

Run orainstRoot.sh and root.sh

Run the following files in order. orainstRoot.sh on node15 orainstRoot.sh on node16 root.sh on node15 root.sh on node16



41

Finish

End of installation for oracle grid infrastructure



42

Checking status of oracle cluster

Verity that the following services are up and running



43

Install database

We skip detail steps to install database binaries in this presentation because it is basically the same with previous versions.



Agenda



44



45

Create database with DBCA

Run dbca as an oracle user



46


Select ‘Create a Database’



47


Select ‘Custom Database’



48


Provide Database Name and SID



49


Click Next



50


Configure



51


Provide password for DB administration groups



52


It is prompted due to short password length. Just continue.



53


There is no raw devices in the selection list. We choose ASM in this screen and will change to raw devices later.



54


Specify ASMSNMP password



55


Choose recovery options



56


Select components you want to configure



57


Click Next



58


Click Next



59


Click the ‘Generate Database Creation Scripts’ check box



60


Browse summary



61


Remember the name of the directory that scripts were resided in and modify those script files to change ASM to raw devices.



Agenda



62



63

Edit script files to create DB files on raw devices

Edit the following files to create DB files on raw devices, not on ASM disks. (scripts files are placed in the ORACLE_HOME/admin/db_name/scripts/ directory. -init.ora-CreateDB.sql-CreateDBFiles.sql-postDBCreation.sql

Run RAW1.sh (RAW is the name of database) after modifying the scripts mentioned above.



64

Edit script files – init.ora

[oracle@node15:/oracle/db/admin/RAW/scripts]$ vi init.ora

###########################################

# File Configuration

###########################################

#db_create_file_dest=+DATA

control_files=('/dev/rrac_control01','/dev/rrac_control02')

Comment out the following line;

db_create_file_dest=+DATA



65

Edit script files – CreateDB.sql

oracle@node15:/oracle/db/admin/RAW/scripts]$ vi CreateDB.sql

DATAFILE '/dev/rrac_system01' SIZE 500M

EXTENT MANAGEMENT LOCAL

SYSAUX DATAFILE '/dev/rrac_sysaux01' SIZE 500M

SMALLFILE DEFAULT TEMPORARY TABLESPACE TEMP TEMPFILE '/dev/rrac_temp01' SIZE 300M

SMALLFILE UNDO TABLESPACE "UNDOTBS1" DATAFILE '/dev/rrac_undo01' SIZE 500M

CHARACTER SET WE8MSWIN1252

NATIONAL CHARACTER SET AL16UTF16

LOGFILE GROUP 1 ('/dev/rrac_redo01') SIZE 250M,

GROUP 2 ('/dev/rrac_redo02') SIZE 250M

Note: Intentionally omitted some parts of the file that do not need be changed



66

Edit script files – CreateDBFiles.sql

[oracle@node15:/oracle/db/admin/RAW/scripts]$ vi CreateDBFiles.sql

SET VERIFY OFF

connect "SYS"/"&&sysPassword" as SYSDBA

set echo on

spool /oracle/db/admin/RAW/scripts/CreateDBFiles.log append

CREATE SMALLFILE UNDO TABLESPACE "UNDOTBS2" DATAFILE '/dev/rrac_undo02' SIZE 500M ;

CREATE SMALLFILE TABLESPACE "USERS" LOGGING DATAFILE '/dev/rrac_user01' SIZE 120M EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO;

ALTER DATABASE DEFAULT TABLESPACE "USERS";

spool off



67

Edit script files – postDBCreation.sql

[oracle@node15:/oracle/db/admin/RAW/scripts]$ vi postDBCreation.sql

ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/dev/rrac_redo03') SIZE 250M,GROUP 4 ('/dev/rrac_redo04') SIZE 250M;

ALTER DATABASE ENABLE PUBLIC THREAD 2;

set echo on

create spfile='/dev/rrac_spfile01' FROM pfile='/oracle/db/admin/RAW/scripts/init

.ora';

shutdown immediate;

host /oracle/db/11g/bin/srvctl enable database -d RAW;

host /oracle/db/11g/bin/srvctl start database -d RAW;

connect "SYS"/"&&sysPassword" as SYSDBA

Note: Intentionally omitted some parts of the file that do not need be changed



Agenda



68



69

Checking current location - OCR

node15:/#cd /oracle/grid/binnode15:/oracle/grid/bin#ocrcheckStatus of Oracle Cluster Registry is as follows :

Version : 3Total space (kbytes) : 262120Used space (kbytes) : 2252Available space (kbytes) : 259868ID : 416468433Device/File Name : +DATA

Device/File integrity check succeeded

Device/File not configured




Cluster registry integrity check succeeded

Logical corruption check succeeded



70

Moving OCR into raw devices

node15:/oracle/grid/bin# ocrconfig -add /dev/rcrs_ocr01node15:/oracle/grid/bin# ocrconfig -add /dev/rcrs_ocr02node15:/oracle/grid/bin# ocrconfig -delete +DATAnode15:/oracle/grid/bin# ocrcheckStatus of Oracle Cluster Registry is as follows :

Version : 3Total space (kbytes) : 262120Used space (kbytes) : 2252Available space (kbytes) : 259868ID : 416468433Device/File Name : /dev/rcrs_ocr01

Device/File integrity check succeededDevice/File Name : /dev/rcrs_ocr02

Device/File integrity check succeeded



71

Checking current location - Vote

node15:/oracle/grid/bin#crsctl query css votedisk

## STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1. ONLINE 153b3f88253f4f8fbfe762a74a19d52f (/dev/rhdisk3) [DATA]

Located 1 voting disk(s).



72

Moving Vote into raw devices

node15:/oracle/grid/bin# crsctl replace votedisk /dev/rcrs_vote01

Now formatting voting disk: /dev/rcrs_vote01.

CRS-4256: Updating the profile

Successful addition of voting disk 64fa6289057c4fc5bf0f2939027c08ff.

Successful deletion of voting disk 153b3f88253f4f8fbfe762a74a19d52f.

CRS-4256: Updating the profile

CRS-4266: Voting file(s) successfully replaced

node15:/oracle/grid/bin# crsctl add css votedisk /dev/rcrs_vote02


CRS-4603: Successful addition of voting disk /dev/rcrs_vote02.

node15:/oracle/grid/bin# crsctl add css votedisk /dev/rcrs_vote03


CRS-4603: Successful addition of voting disk /dev/rcrs_vote03.



73

Checking new location - Vote

node15:/oracle/grid/bin#crsctl query css votedisk

## STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1. ONLINE 64fa6289057c4fc5bf0f2939027c08ff (/dev/rcrs_vote01) [ ]

2. ONLINE 6b03d0b906f14f9abf129d4e9f54480d (/dev/rcrs_vote02) [ ]

3. ONLINE 6dea9266eb294f2abfe4b28b419cafc7 (/dev/rcrs_vote03) [ ]

Located 3 voting disk(s).



Agenda



74



75

Disabling ASM

[oracle@node15:/home/oracle]$ srvctl status asm

ASM is running on node15,node16

[oracle@node15:/home/oracle]$ srvctl status diskgroup -g data

Disk Group data is running on node15,node16

[oracle@node15:/home/oracle]$ srvctl stop diskgroup -g data

[oracle@node15:/home/oracle]$ srvctl stop asm -n node15

[oracle@node15:/home/oracle]$ srvctl stop asm -n node16

[oracle@node15:/home/oracle]$ srvctl disable diskgroup -g data

[oracle@node15:/home/oracle]$ srvctl disable asm

[oracle@node15:/home/oracle]$ srvctl status diskgroup -g data

Disk Group data is not running

[oracle@node15:/home/oracle]$ srvctl status asm

ASM is not running.



76

Checking SCAN

[oragrid@node15:/home/oragrid]$ netstat -i

Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll

en0 1500 link#2 0.2.55.4f.af.f0 31059683 0 4287605 0 0

en0 1500 130.130.10 node15 31059683 0 4287605 0 0

en0 1500 130.130.10 node15vip 31059683 0 4287605 0 0

en0 1500 130.130.10 node15scan 31059683 0 4287605 0 0

en1 1500 link#3 0.2.55.af.8d.ba 13023820 0 7324113 0 0

en1 1500 192.168.10 node15rac 13023820 0 7324113 0 0

[oragrid@node15:/home/oragrid]$ srvctl status scan_listener

SCAN Listener LISTENER_SCAN1 is enabled

SCAN listener LISTENER_SCAN1 is running on node node15

[oragrid@node15:/home/oragrid]$ srvctl status scan

SCAN VIP scan1 is enabled

SCAN VIP scan1 is running on node node15



77

Disabling SCAN

[oragrid@node15:/home/oragrid]$ srvctl stop scan_listener

[oragrid@node15:/home/oragrid]$ srvctl stop scan

[oragrid@node15:/home/oragrid]$ srvctl disable scan_listener

[oragrid@node15:/home/oragrid]$ su

root's Password:

# cd /oracle/grid/bin

# srvctl disable scan



78

Disabling SCAN

# su - oragrid

[oragrid@node15:/home/oragrid]$ srvctl status scan_listener

SCAN Listener LISTENER_SCAN1 is disabled

SCAN listener LISTENER_SCAN1 is not running

[oragrid@node15:/home/oragrid]$ srvctl status scan

SCAN VIP scan1 is disabled

SCAN VIP scan1 is not running

[oragrid@node15:/home/oragrid]$ netstat -i

Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll

en0 1500 link#2 0.2.55.4f.af.f0 31061585 0 4287992 0 0

en0 1500 130.130.10 node15 31061585 0 4287992 0 0

en0 1500 130.130.10 node15vip 31061585 0 4287992 0 0

en1 1500 link#3 0.2.55.af.8d.ba 13027815 0 7326374 0 0

en1 1500 192.168.10 node15rac 13027815 0 7326374 0 0

Materials may not be reproduced in whole or in part without the prior written permission of IBM. 5.3

© Copyright IBM Corporation 2011

IBM Systems and Technology Group Asia Pacific Technical Enablement Conference

End of Presentation

deploying oracle 11g rac r2 on aix6 using raw devices with powerha

Documents