oracle asm reduces cost of vldb deployment hanan hit, principal database architect lina shabelsky,...

Oracle ASM Reduces Cost of VLDB Deployment

Hanan Hit, Principal Database Architect

Lina Shabelsky, Senior Application and Database Engineer

NOCOUG Winter Conference February 08, 2007

Confidential and Proprietary

Agenda

• Application description• What ASM is and is not• Protocol etc.• Deployment models• Network design • Physical architecture• Oracle operations to reduce required I/O• Index creation stats • Best practices• Lessons learned during VLDB implementation• Backup and recovery

Application Description

Logical Database Structure

Management Db

Operational Data Store

Historical Data Store

ArchiveData Store (SATA)

General Loading Schema

Staging ODS –Last Hour

Data Warehouse

Loader

Aggregators

High Level – Physical Layer

Storage Private Network

CISCOSYSTEMS 2G IP Switch

SkyRider Inc.RAC – Primary / Primary Configuration

With Management DB & ARCH DB

Cache Fusion Private Network

CISCOSYSTEMS 2G IP Switch

Public Network 1GB

FAS 3050 FAS 3050

AMD Dual CPU8GB Machine

AMD Dual CPU 16GB RAM

PRODDB1

PRODDB2

ARCHDB

Cache Fusion Private Network

What ASM is and is not

ASM - Flexible, Efficient, Time Saver

• Vertically integrated file system • Easy file management system • Management of:

-Raw device volume-Oracle data files-Online and archive logs-RMAN backup

• Multi platform availability• Stripes data across all raw volumes• Hot spot detection and correction• Optional mirroring (best with SAN/NAS) but striping is NOT• Oracle 10g Release 1 and above

What ASM is NOT

• A cluster file system• Available for non Oracle files• Available for OCR and voting disks

• General file system

Options prior to ASM

• RawWith LVM Without LVM

• Cooked With LVMWithout LVM

Why We Chose ASM in the First Place?

• New implementation using 10g • Linux RHEL shop• Mid range storage array - full enterprise business needs• “Just trust the hardware to handle it”

Not a workable solution (DBA’s)• Expected thousands of data files

Didn’t want to use the BIG File option (YET).• RAC & non RAC implementations• Obvious need for storage growth with unpredictable limit size• Do more with less

Very few DBAs & sys/storage admin• Block level access to storage• Easy storage provisioning

Protocol

Which Protocol to Use?

• FC – SAN (2/4 GBit/s)• iSCSI – IP-SAN (2/4 GBit/s)• NFS – NAS (2/4 GBit/s)

• FCIP

What is iSCSI Protocol?

• A network protocol standard that allows the use of the SCSI protocol over TCP/IP networks

• A transport layer protocol in the SCSI-3 specifications framework

• Expected to capture more than 10% of storage systems revenue and an even greater percentage of capacity by 2008 - Hot technologies for 2007 (Storage Magazine)

• “iSCSI SAN is definitely happening”- Stephen Foskett, GlassHouse Technologies Inc., Framingham, MA

Wikipedia iSCSI Definitions

• iSCSI initiator in client/server terminology, is akin to a client device that connects to some service offered by the server (in this case an iSCSI target).

• An iSCSI target is akin to a server, in that it provides block level access to its storage media (usually a hard drive, but can be other types of SCSI devices).

• Only one iSCSI initiator can talk to a given iSCSI target at a time (one-to-one).

Host Storage System

Initiator Target

Simple iSCSI system

Deployment Models

• Separate Disk Groups Use storage base features for ASM deployment data management Backup/recovery, cloning etc.

• Shared Disk Groups Use Oracle tools exclusively for all data management methods

Separate Disk Groups

Shared Disk Groups

Shared Disk Group- Multiple RAC Instances

Network Design

Typical Performance in MB/s

System Component Throughput /Performance

Bits Bytes

16 Port Switch8 * 2GBit/s 1200 MB/s

Fibre Channel 2 GBit/s 200 MB/s

Disk Controller 2 GBit/s 180 MB/s

GigE NIC 2 GBit/s 80 MB/s

1 Gbit HBA 1 GBit/s 100 MB/s

2 Gbit HBA 2 GBit/s 200 MB/s

iSCSI End Point Options

Network Configuration

PROPERLY SHUT DOWN SYSTEM BEFORE OPENING CHASSIS.

PCI 1 PCI 3

PCI 4PCI 2

DISCONNECT AC POWER

CORD BEFORE REMOVAL

status

DISCONNECT AC POWER

CORD BEFORE REMOVAL

status

Console

e0a e0b

e0b RLM

e0c e0d

LVD SCSI

NetworkAppliance FAS3020

PCI 1 PCI 3

PCI 4PCI 2

DISCONNECT AC POWER

CORD BEFORE REMOVAL

status

DISCONNECT AC POWER

CORD BEFORE REMOVAL

status

Console

e0a e0b

e0b RLM

e0c e0d

LVD SCSI

PCI 1 PCI 3

PCI 4PCI 2

DISCONNECT AC POWER

CORD BEFORE REMOVAL

status

DISCONNECT AC POWER

CORD BEFORE REMOVAL

status

Console

e0a e0b

e0b RLM

e0c e0d

LVD SCSI

CONSOLE

Catalyst 2948G-L3

PSI RPSU

1 3 5 7 9 1311 15 211917 23 2725 29 31 39373533

41 43 45 47

10/100/100 ETHERNET

LAYER 3 SWITCH

STATUS

1000 Base - XCONSOLE

Catalyst 2948G-L3

PSI RPSU

1 3 5 7 9 1311 15 211917 23 2725 29 31 39373533

41 43 45 47

10/100/100 ETHERNET

LAYER 3 SWITCH

STATUS

1000 Base - X

MLS-A MLS-B

==== Ethernet Configuration ====Ethernet Ports: santana e0a à e0b à e0c à e0d à hendrix e0a à e0b à e0c à e0d à doors e0a à e0b à e0c à e0d à VLAN: 200Jumbo Frame 9216 Bytes

doorshendrixsantana

IP Configuration ====Domain Name: crightcorp.com (or skyrider.com)Default VRRP Router: 172.16.10.1

IP Addresses: MLS-A: 172.16.10.1 MLS-B: 172.16.10.2 santana V3: 172.16.10.16 hendrix V3: 172.16.10.17 doors V3: 172.16.10.18

==== General Info ====

Remote Power Strip (rps1) santana: Bus A: Plug xx Bus B: Port xx hendrix: Bus A: Plug xx Bus B: Plug xx doors: Bus A: Plug xx Bus B: Plug xx

Terminal Server (ts1: 9600/N/1) Port x: santana Port y: hendrix Port z: doors

VIF V1(Standby)

Static 2 Link LAG

VIF V3VIF V2(Active)

Active VRRP Router for VLAN 200

Backup VRRP Router for VLAN 200

9600/N/1

CONSOLE

Catalyst 2948G-L3

PSI RPSU

1 3 5 7 9 1311 15 211917 23 2725 29 31 39373533

41 43 45 47

10/100/100 ETHERNET

LAYER 3 SWITCH

STATUS

1000 Base - X

Management Switch

VIF V1(Standby)

VIF V3VIF V2(Active)

RLM RLMPwr

Bus-BPwr

Physical Architecture

FAS3050 #1

SMRAC2SMRAC1

FAS3050 #1

ASM (VOLMAINDATA)DISKGROUP

“DATA1”

LUN #1DATA #1 100GB

LUN #7TEMP #1 100GB

LUN #3UNDO # 1

LUN #2DATA # 2

LUN #5DATA #3 100GB

LUN #6DATA #4 100GB

LUN #4TEMP #2

MAIN RAC WITH ASM – DATABASE LAYOUT

VOLOCR

MAINOCR 2GB

MGMOCR 2GB

MAINVOTING 2GB

FAS3050 #2

SMRAC2SMRAC1

HENDRIX

ASM (VOLMAINDATA)DISKGROUP

“DATA2”

LUN #1INDEX #1

LUN #3UNDO # 1

LUN #2INDEX # 2

LUN #5INDEX #3

LUN #4TEMP #2

VOLMAINRECOVERY

LUN #1REDO #1

LUN #4REDO #4

LUN #3REDO # 3

LUN #2REDO # 2

LUN #5CONTROLD

iSCSIiSCSI

MAIN & MGM RAC WITH ASM – DATABASE LAYOUT

VOLDATADUMPS

UNC / NFS 600GB

FAS3020

SMARCHDB

ASM (VOLARCHDBATA)DISKGROUP

“DATA1”

LUN #1DATA #1 512GB

LUN #3UNDO # 1

LUN #2INDEX #1

LUN #5TEMP

#2 300GB

VOLARCHDBRECOVERY

LUN #1REDO #1

LUN #4REDO

#4 60GB

LUN #3REDO #

3 60GB

LUN #2REDO # 2

LUN #5CONTRO

LD20GB

ARCHDB WITH ASM – DATABASE LAYOUT

VOLARCHDBARCHLOGS

LUN #4DATA # 2

LUN #6INDEX #

2512GB

VOLMISC

512GBORACLE_HOME

ASM_HOMEADMIN

SMRAC2SMRAC1

VOLMAINARCHLOGS

NFSNFS

ASM (VOLARCHDBATA)DISKGROUP

“DATA2”

LUN #1DATA #1 512GB

LUN #3UNDO # 1

LUN #2INDEX

#1 512GB

LUN #4DATA # 2

LUN #5INDEX #

2512GB

ASM (VOLARCHDB

ATA)DISKGROUP

“DATA3”

LUN #1DATA #1 400GB

LUN #2DATA #1

LUN #3DATA # 2

Database size

Oracle operations to reduce required I/O

Oracle Operations - Reduce Disk I/O

• Index creation on many billions of rows New software version while supporting legacy data

• Gathering Oracle stats

Index creation statistics

Index Creation – New App Version

• Set/unset the SKIP_UNUSABLE_INDEXES When set to TRUE Oracle will not attempt to use or report errors when an index is marked as unusable

• System Level - ALTER SYSTEM SET SKIP_UNUSABLE_INDEXES = TRUE;

• Session Level ALTER SESSION SET SKIP_UNUSABLE_INDEXES = TRUE;

Index Creation – New App Version

• Step 1 - Create the new index with the UNUSABLE attribute• Step 2 - System Level - ALTER SYSTEM SET

SKIP_UNUSABLE_INDEXES = TRUE;• Step 3 - ALTER INDEX <index name> MODIFY PARTITION <partition

name> UNUSABLE; On all partitions. If using a Global Index then mark the entire index as UNUSABLE

• Step 4 - ALTER INDEX <index name> REBUILD PARTITION <new partition name> NOLOGGING;

On the most fresh partition this will enable the application to run with the new index ASAP.

• Step 5 - ALTER INDEX <index name> REBUILD PARTITION <partition name> NOLOGGING;

On all the legacy (non hotspot) partitions. This will be rebuild according the system workload.

• Final Step - System Level - ALTER SYSTEM SET SKIP_UNUSABLE_INDEXES = FALSE;

Gathering Oracle Stats

• Gather partitions statistics (First time) Exec dbms_stats.gather_table_stats(ownname=>‘<schema

name>', tabname => ‘<tab name>', PARTNAME=> ‘<part name>' , estimate_percent => DBMS_STATS.AUTO_SAMPLE_SIZE, cascade=>TRUE, method_opt=>'FOR ALL COLUMNS SIZE AUTO');

Exec dbms_stats.gather_table_stats (ownname=>‘<schema name>' , tabname=>‘<tab name>' , PARTNAME=> '<part name>' , estimate_percent=>1,CASCADE=> true);

• Unlock Statistics (If previously locked)

Exec dbms_stats.unlock_table_stats(OWNNAME=>‘<schema name>' , TABNAME=>‘<tab name>');

Gathering Oracle Stats – More

• Copy statistics Execdbms_stats.COPY_TABLE_STATS(OWNNAME=>‘<schema

name>' , TABNAME=>‘<tab name> ' , SRCPARTNAME=>‘<N’th partition name>' , DSTPARTNAME=>'<N’th+1 partition name>');

• Lock table statistics Exec dbms_stats.lock_table_stats(OWNNAME=>‘<schema name>' ,

TABNAME=>‘<tab name>');

Index Creation Example

Index Creation Example – More

Best Practices

NetApp/ASM – Our Best Practices

• Single aggregate • FlexVol • ASM external redundancy • Separate disk groups• Maximum volume size allowed 16TB while

recommended is not more then 3TB• Use RAID-DP with maximum 16 drives in a single

RAID group• Set minra to off – even on DSS • Jumbo frames – 9K MTU• Single mode VIF

Lessons learned during VLDB implementation

RAC 10gR2 Implementation Learnings

• Implementation of VLDB in RHEL 4.3, software iSCSI, NetApp and ASM environment

• Procedure of provisioning additional storage • Backup procedures with NetApp SnapManager for

Oracle

Network Implementation

• Isolate your database from the rest of the IP traffic• Use separate VLAN for access to the storage• Use oifcfg to verify the setup of your interfaces• Use IP bonding on both – server and filer to achieve

network stability and as an alternative solution to multipathing

• IP bonding allows you to aggregate multiple network interfaces into higher performance network link and provides failover solution

Oracle Clusterware

• Carefully plan your file system layout • If you use ASMLIB with iSCSI disks, don’t map OCR

and voting disk to raw devices in RAC environment – use OCFS or NFS

• Raw devices are not aware of ASMLIB • In RedHat 4 update 3, the use of

/etc/sysconfig/rawdevices is deprecated

Storage

• When planning for the large database consult your storage vendor about RAID configuration, storage provisioning and backup procedures.

• As an example – using NetApp as storage appliance allows DBAs to take advantage of RAID-DP (Double Parity) – fault tolerance of RAID 1 at the price of RAID 4.

• NetApp flexible architecture allows DBAs to provision additional storage with minimal downtime.

• SnapManager for Oracle allows DBAs to combine NetApp snapshot technology with Oracle RMAN to dramatically decrease time taken to backup and restore database.

Storage Provisioning with ASM

• Adding 1TB of storage takes no more than

30 minutes with Netapp• Step1 – Ask your system administrator to

plug in Netapp shelves• Step 2 – login to the filer and add new disks

to the aggregate leaving 2 spares per array• Step 3 – create new volume and LUNs, add

them to the initiator group

Filer view

Adding Storage to the Database

• Step 4 - After new LUNs created on the filer, reboot the database server to automatically discover new LUNS. You can verify that LUNs are accessible from the database server using NetApp host utilities:

Creating ASM Disks

• Step 5 – run fdisk on new devices. • Step 6 – run /etc/init.d/oracleasm createdisk.• Last step – connect to ASM instance and execute

“create diskgroup…” command. You can also use Enterprise Manager db control:

Backup and Recovery

SnapManager for Oracle Backups

• A management tool with a GUI and command-line interface for Oracle Database administrators that simplifies backup, recovery, and cloning for Oracle Databases

• Designed to leverage Oracle Database 10g data management and grid features such as ASM, RAC, and RMAN

• Utilizes NetApp Snapshot technology to create extremely fast and space-efficient backups

Snapshot copies are point-in-time copies of a database that are created nearly instantaneously. These backups can also be registered with Oracle RMAN, which facilitates the use of RMAN to restore and recover the database at finer granularities such as blocks.

How SnapManager Works

• To create a successful backup, SnapManager performs the following operations:

Determines the list of data files, control files, and log files that make up the database. These files can be located on one or more ASM disk groups.

Puts the database in hot backup mode Creates an atomic Snapshot copy of all the filer volumes

that make up the ASM disk group Ends the hot backup mode Clones the Snapshot copy and then renames the cloned

ASM disk group Verifies the backup Registers the cloned ASM disk group and storage

information in the RMAN repository

Disk Backup vs. SnapManager Backup

• To demonstrate the benefit of snapshot backup technology we measured the time taken to backup 70GB database to the disk. Here’s what we found:

• Backup to the disk:• Backup set complete, elapsed time: 00:25:20 • It would take about 18 hours to backup 3 TB

database.

Time Log for SnapManager Backup

• 2006-11-10 17:43:45,396 [INFO ]: SMO-07100: Placing database into online backup mode.

• 2006-11-10 17:46:00,724 [DEBUG]: SMO-12000: Executing SnapDriveCommand

• 2006-11-10 17:46:04,762 [DEBUG]:SMO-12001: Result SnapDriveResult (0:00:04.037) (took 4 seconds to take a snapshot of the volume)

• 2006-11-10 17:55:55,841 [INFO ]: SMO-13037: Successfully completed operation: Backup

• 2006-11-10 17:55:55,947 [INFO ]: SMO-13048: Operation Status: SUCCESS

• 2006-11-10 17:55:56,030 [INFO ]: SMO-13049: Elapsed Time: 0:12:33.238 (including all backup management procedures)

• From database alter log:Fri Nov 10 17:43:45 2006 ALTER DATABASE BEGIN BACKUP Fri Nov 10 17:46:42 2006 ALTER DATABASE END BACKUP

Restore Operation

• Similar to backups, SnapManager for Oracle relieves the DBA from having to worry about the underlying layout of data in terms of storage subsystems, disks, host volumes, or host file systems.

• DBAs just choose the backup they want to restore from, and SnapManager does the rest.

• DBAs can also specify the date and time or the Database SCN to which they would like the database restored.

Thank you

Hanan Hit, Principal Database Architect

Lina Shabelsky, Senior Application and Database Engineer

oracle asm reduces cost of vldb deployment hanan hit, principal database architect lina shabelsky,...

protocol slide

lvm slide

iscsi protocol

ma slide

recovery slide

proprietary asm flexible

gbits fcip slide

proprietary options

Documents

the aditi deductive database system - vldb

nocoug cdc presentation

vldb database school (china) vldb...

oracle database vldb and partitioning guide

tuning database locks & latches hamid r. minoui fritz...

private cloud database consolidation - nocoug · private...

structured approach to database administration - nocoug -...

column-oriented database systems...

1 jim gray & gordon bell: vldb 95 parallel database systems...

column-oriented database systems final · cracking , from...

nocoug journal 201202

logical standby database for reporting mark bole nocoug nov...

flexible database generators nicolas bruno surajit chaudhuri...

oracle help center - vldb and partitioning guide · 2017....

very large database (vldb) backup & recovery best practices

oracle database vldb and partitioning...

spatial database systems - fernuniversität...

oracle® database vldb and partitioning guide 12c release 1...

vldb and partitioning guide - oracle · oracle® database...

new features in database 11gr2 - nocoug - … features in...