1 © 2008 julian dyke juliandyke.com automatic storage management julian dyke independent consultant...
Post on 29-Mar-2015
228 Views
Preview:
TRANSCRIPT
1 © 2008 Julian Dyke juliandyke.com
Automatic StorageManagement
Julian Dyke
Independent Consultant
Web Version - December 2008
2 © 2008 Julian Dyke juliandyke.com
Objectives
1. Understand how Oracle database files are stored in ASM
2. Calculate how long ASM rebalance operations will take
3 © 2008 Julian Dyke juliandyke.com
Agenda
ASM Instances ASM Disk Groups
Metadata Extent Distribution Rebalancing Redundancy
4 © 2008 Julian Dyke juliandyke.com
ASMInstances
5 © 2008 Julian Dyke juliandyke.com
ASM Single Instance Architecture
OracleClusterware
ASMInstance
RDBMSInstance
Server
OCSSDDaemon
Only
DedicatedStorage
6 © 2008 Julian Dyke juliandyke.com
ASM Single Instance Background Processes
PMON
SMON
CKPT
LGWR
DBW0 VKTM PSP0 MMAN RBAL
GMON
X000
DIAG
DIA0
Fixed Area
Variable Area
ASM Cache
Oracle 11.1
7 © 2008 Julian Dyke juliandyke.com
ASM RAC Architecture
Oracle Clusterware
ASMInstance
RDBMSInstance
Node 1
Oracle Clusterware
ASMInstance
RDBMSInstance
Node 2
Oracle Clusterware
ASMInstance
RDBMSInstance
Node 3
Oracle Clusterware
ASMInstance
RDBMSInstance
Node 4
Storage Network
Private Network
PublicNetwork
SharedStorage
8 © 2008 Julian Dyke juliandyke.com
ASM RAC Architecture
PROD1 TEST1
CLUSTERWARE
+ASM1
PROD2 TEST2
CLUSTERWARE
+ASM2
PROD
TEST
Clusterware
ASM Instances
RDBMS Instances
Database Files
9 © 2008 Julian Dyke juliandyke.com
ASM RAC Instance Background Processes
PMON
SMON
CKPT
LGWR
DBW0 VKTM PSP0 MMAN RBAL
GMON
X000
DIAG
DIA0
Fixed Area
Variable Area
ASM Cache
Oracle 11.1
LMON LMS0 LMD0 LCK0 DIAG
MARK KATE
10
© 2008 Julian Dyke juliandyke.com
ASMDisk
Groups
11 © 2008 Julian Dyke juliandyke.com
ASM Disk Groups and Disks
Disk Group 1
Disk Group 2
Disk Group 3
Disk 1
Disk 4
Disk 2
Disk 6
Disk 5
Disk 7
Disk 3
12
© 2008 Julian Dyke juliandyke.com
ASM Disk Groups and Disks
Disk Group 1
Disk 1 Disk 4
Disk Group 2
Disk 2 Disk 5 Disk 6 Disk 7 Disk 3
Disk Group 3
13
© 2008 Julian Dyke juliandyke.com
ASM Disk Groups, Disks and Database Files
Disk Group 1
Disk 1 Disk 4
Disk Group 2
Disk 2 Disk 5 Disk 6 Disk 7
Disk 3
Disk Group 3
File 1
File 2File 3
File 4
File 5
File 5
File 6
14
© 2008 Julian Dyke juliandyke.com
File Extents versus Allocation Units File Extent
Logical unit of ASM file Map to allocation units
One to many mapping
Allocation Unit Physical unit of ASM disk Oracle 10.2 and below
Always 1MB Can be increased using _asm_ausize
Oracle 11.1 and above Variable size 1MB, 2MB ,4MB, 8MB, 16MB, 32MB, 64MB
15
© 2008 Julian Dyke juliandyke.com
X$KFFXP Maps file extents to allocation units Only populated in ASM instance Columns include
Column Name Description
GROUP_KFFXP Disk Group Number
NUMBER_KFFXP File Number
COMPOUND_KFFXP Disk Group Number || File Number
INCARN_KFFXP Incarnation Number
PXN_KFFXP Physical Extent Number (within file)
XNUM_KFFXP Logical Extent Number (within file)
LXN_KFFXP 0=primary, 1=first mirror, 2=second mirror
DISK_KFFXP Disk Number
AU_KFFXP Allocation Unit Number (within disk)
SIZE_KFFXP Size (# allocation units)
16
© 2008 Julian Dyke juliandyke.com
ASM Metadata Metadata is stored in first 256 files in ASM disk group
Space is initially allocated when disk group is created Can be subsequently extended
Metadata allocation units are divided into blocks Each block is 4096 bytes Block size specifed using _asm_blksize
Metadata files includeFile# Description
0 Metadata Header
1 File Directory
2 Disk Directory
3 Active Change Directory
4 Continuing Operations Directory
5 Template Directory
6 Alias Directory
9 Attribute directory (optional)
12 Staleness registry (optional)
17
© 2008 Julian Dyke juliandyke.com
ASM Metadata
Active ChangeDirectory
Continuing OperationsDirectory
Template Directory
Alias Directory
Disk Directory
File Directory
Partner Status Table
Metadata Header
Free Space Table
Allocation Table
Disk Header
Metadata Header
18
© 2008 Julian Dyke juliandyke.com
ASM Metadata Initial Allocation (Single Instance)
File# AU Description # AUs
0 0 Disk Header, Free Space Table, Allocation Table 1
1 Partner Status Table 1
1 File Directory 1
2 Disk Directory 1
3 Active Change Directory 42
4 Continuing Operations Directory 2
5 Template Directory 1
6 Alias Directory 1
Active Change Directory Records changes to metadata Used during recovery of instance or operation failures
Continuing Operations Directory Maintains state of active operations
19
© 2008 Julian Dyke juliandyke.com
ASM Metadata Block Types
Type Description
1 KFBTYP_DISKHEAD
2 KFBTYP_FREESPC
3 KFBTYP_ALLOCTBL
4 KFBTYP_FILEDIR
5 KFBTYP_LISTHEAD
6 KFBTYP_DISKDIR
7 KFBTYP_ACDC
8 KFBTYP_CHNGDIR
9 KFBTYP_CODBGO
10 KFBTYP_TMPLTDIR
11 KFBTYP_ALIASDIR
12 KFBTYP_INDIRECT
Type Description
13 KFBTYP_PST_NONE
14 KFBTYP_HASHNODE
15 KFBTYP_COD_RBO
16 KFBTYP_COD_DATA
17 KFBTYP_PST_META
18 KFBTYP_PST_DTA
19 KFBTYP_HBEAT
20 KFBTYP_SR
21 KFBTYP_STALEDIR
22 KFBTYP_VOLUMEDIR
23 KFBTYP_ATTRDIR
20
© 2008 Julian Dyke juliandyke.com
KFED Utility In Oracle 10.2 and above the kfed utility can be used to inspect and edit the
contents of ASM blocks
[oracle@server3 ~]$ $ORACLE_HOME/bin/kfed -has/mlib ASM Library [asmlib='lib']aun/um AU number to examine or update [AUNUM=number]aus/z Allocation Unit size in bytes [AUSZ=number]blkn/um Block number to examine or update [BLKNUM=number]blks/z Metadata block size in bytes [BLKSZ=number]ch/ksum Update checksum before each write [CHKSUM=YES/NO]cn/t Count of AUs to process [CNT=number]d/ev ASM device to examine or update [DEV=string]o/p KFED operation type [OP=READ/WRITE/MERGE/NEW/FORM/FIND/STRUCT]p/rovnm Name for provisioning purposes [PROVNM=string]s/eek AU number to seek to [SEEK=number]te/xt File name for translated block text [TEXT=string]ty/pe ASM metadata block type number [TYPE=number]
This utility should only be used under the guidance of Oracle Support
21
© 2008 Julian Dyke juliandyke.com
KFED Utility For example to dump blocks in aliases directory in DISKGROUP1 Find group number
SELECT disk_kffxp, au_kffxp FROM x$kffxp WHERE group_kffxp = 1AND number_kffxp = 6AND lxn_kffxp = 0;
Disk Allocation Unit
0 49
Find disk name
SELECT group_number FROM v$asm_diskgroupWHERE name = 'DISKGROUP1';
Alias directory is stored in file number 6
SELECT path FROM v$asm_diskWHERE group_number = 1AND disk_number = 0;
Path/dev/oracleasm/disks/VOL1
22
© 2008 Julian Dyke juliandyke.com
KFED Utility Example (continued)
Allocation unit is 1MB Block size is 4096 Therefore there are 256 blocks per allocation unit Starting block offset = 256 * 49 = 12544
for (( f = 12544 ; f < 12544 + 256 ; f++ ))do
kfed op=read blkn=$f dev='/dev/oracleasm/disks/VOL1' > blk${f}done
23
© 2008 Julian Dyke juliandyke.com
Extent Distribution
24
© 2008 Julian Dyke juliandyke.com
Extent Distribution Creating a disk group:
CREATE DISKGROUP diskgroup1 EXTERNAL REDUNDANCYDISK '/dev/oracleasm/disks/VOL1';
Dropping a disk group:
DROP DISKGROUP diskgroup1 INCLUDING CONTENTS
25
© 2008 Julian Dyke juliandyke.com
Extent Distribution1 disk
Disk 0
Metadata
0
1
2
3
4
5
6
7
Metadata Data
26
© 2008 Julian Dyke juliandyke.com
Extent Distribution2 disks
Disk 0 Disk 1
Metadata
0
2
4
6
8
10
12
14
1
3
5
7
9
11
13
15
Metadata Data
27
© 2008 Julian Dyke juliandyke.com
Extent Distribution4 disks
Disk 0 Disk 1 Disk 2 Disk 3
Metadata
0
4
8
12
16
20
24
28
2
6
10
14
18
22
26
30
1
5
9
13
17
21
25
29
3
7
11
15
19
23
27
31
Metadata Data
28
© 2008 Julian Dyke juliandyke.com
Extent Distribution1 large disk - 1 small disk
Disk 0 Disk 1
Metadata
0
1
3
4
6
7
9
10
2
5
8
11
Metadata Data
29
© 2008 Julian Dyke juliandyke.com
Extent Distribution1 large disk - 3 small disks
Disk 1 Disk 2
1
6
11
16Metadata
0
3
5
8
10
13
15
18
Disk 0 Disk 4
2
7
12
17
4
9
14
19
Metadata Data
30
© 2008 Julian Dyke juliandyke.com
Extent Distribution2 large disks - 2 small disks
Disk 1 Disk 2
3
9
15
21Metadata
0
2
6
8
12
14
18
20
Disk 0 Disk 4
1
4
7
10
5
11
17
23
13
16
19
22
Metadata Data
31
© 2008 Julian Dyke juliandyke.com
Rebalancing
32
© 2008 Julian Dyke juliandyke.com
Extent Distribution Adding a disk:
ALTER DISKGROUP diskgroup1 ADD DISK '/dev/oracleasm/disks/VOL2' REBALANCE POWER 0;
Dropping a disk:
ALTER DISKGROUP diskgroup1 DROP DISK 'DISKGROUP1_0002' REBALANCE POWER 0;
Rebalancing a disk group:
ALTER DISKGROUP diskgroup1 REBALANCE POWER 1;
33
© 2008 Julian Dyke juliandyke.com
RebalancingAdding disks - 1 disks to 2 disks
Disk 0 Disk 1
0
1
2
3
4
5
6
7
5
7
0
1
2
3
4
5
6
7
5
7
1
3
1
3
STOP
Metadata Data
34
© 2008 Julian Dyke juliandyke.com
RebalancingAdding disks - 1 disks to 4 disks
Disk 0 Disk 1 Disk 2 Disk 3
0
1
2
3
4
5
6
7
2
6
1
5
0
4
0
1
2
3
4
5
6
7
1 0
45
2
6
STOP
Metadata Data
35
© 2008 Julian Dyke juliandyke.com
RebalancingAdding disks - 2 disks to 3 disks
Disk 0 Disk 1 Disk 2
1
3
5
7
9
11
13
15
0
2
4
6
8
10
12
14
2
5
8
11
0
1
2
3
4
5
0
31
4
14
17
16
17
Metadata Data
36
© 2008 Julian Dyke juliandyke.com
RebalancingAdding disks - 2 disks to 4 disks
Disk 0 Disk 1 Disk 2 Disk 3
1
3
5
7
9
11
13
15
0
2
4
6
8
10
12
14
1
5
9
13
0
4
8
12
0
1
2
3
4
5
6
7
1 0
45
2
6
STOP
Metadata Data
37
© 2008 Julian Dyke juliandyke.com
RebalancingDropping disks - 3 disks to 1 disk
Disk 0 Disk 1 Disk 2
0
3
6
1
2
4
5
7
4 5
0
3
1
2
4
5
2
5
1
4
8
7 8
1 2
STOP
Metadata Data
38
© 2008 Julian Dyke juliandyke.com
RebalancingMoving disks - 2 disks to 2 disks
Disk 0 Disk 1 Disk 2
1
3
5
7
9
11
13
15
4
6
8
10
12
14
10
6
2
0
4
8
0
1
2
3
4
5
6
7
0
4
2
6
STOP
2
0
12
14
Metadata Data
39
© 2008 Julian Dyke juliandyke.com
RebalancingV$ASM_OPERATION Contains details of ongoing rebalance operations
Column Name Data Type
GROUP_NUMBER NUMBER
OPERATION CHAR(5)
STATE VARCHAR2(4)
POWER NUMBER
ACTUAL NUMBER
SOFAR NUMBER
EST_WORK NUMBER
EST_RATE NUMBER
EST_MINUTE NUMBER
ERROR_CODE VARCHAR2(44)
Estimate of remaining time
40
© 2008 Julian Dyke juliandyke.com
RebalancingPower Limit Power limit can be 0 to 11
0 disables rebalance operation 1 to 11 specifies number of ARBn background processes used for
rebalance
In Oracle 10.2 RBAL manages rebalance operation Each ARBn background process is allocated a range of 128 allocation
units to rebalance When complete another range is requested
AD lock is taken while an allocation unit is being rebalanced Rebalance operations take much longer than theoretically necessary.
Possible reasons include: Locking GES updates with other ASM instances Updates to RDBMS instance
41
© 2008 Julian Dyke juliandyke.com
RebalancingSummary EST_MINUTES column of V$ASM_OPERATION is reasonably accurate
Allow a few minutes for SAN cache to stabilize Check regularly for changes to estimate
ASM rebalance operations do not affect workload Locks are only taken briefly
Lock mechanism has changed in Oracle 11.1 SAN cache and I/O performance will be affected
In Oracle 10.2 rebalancing is fastest if Other ASM instances are shutdown RDBMS instance is shutdown
Estimated completion time will be affected by: Use of SAN cache and I/O by rest of workload Rate of change by applications to blocks in ASM files being rebalanced
42
© 2008 Julian Dyke juliandyke.com
Redundancy
43
© 2008 Julian Dyke juliandyke.com
Redundancy ASM Supports three levels of redundancy
External Redundancy Implemented externally using storage layer Most common configuration in production
Normal Redundancy Two copies of each extent maintained in separate failure groups Used with extended clusters Used occasionally in production e.g. CERN Increases CPU overhead on servers
High Redundancy Three copies of each extent maintained in separate failure groups Very rare in production
44
© 2008 Julian Dyke juliandyke.com
ASM Failure Groups - External Redundancy
Disk Group
Disk 1 Disk 2 Disk 3
45
© 2008 Julian Dyke juliandyke.com
ASM Failure Groups - Normal Redundancy
Failure Group 1 FailureGroup 2
Disk Group
Disk 1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6
46
© 2008 Julian Dyke juliandyke.com
ASM Failure Groups - High Redundancy
Failure Group 1
Disk Group
Disk 1 Disk 2
Failure Group 2
Disk 1 Disk 2
Failure Group 3
Disk 1 Disk 2
47
© 2008 Julian Dyke juliandyke.com
Normal Redundancy1 Disk Per Failure Group
Disk 0 Disk 1
0
1
0
1
2
3
2
3
10
11
Failure Group 1
8
9
67
0
1
2
3
45
9
10
7
8
5
6
0
1
234
11
Failure Group 2
Primary SecondaryMetadata
Data Primary Secondary
48
© 2008 Julian Dyke juliandyke.com
Normal Redundancy2 Disks per Failure Group
Disk 0 Disk 1 Disk 2 Disk 3
0
3
0 1
23
1
2
4
7
5
6
4
7
5
6
20
23
Failure Group 1 Failure Group 2
16
19
12
15
0
3
4
7
8
11
18
21
14
17
10
13
1
2
56
9
22
19
20
15
16
11
12
03
47
8
2321
22
17
18
13
14
1
2
5
69
10
Primary SecondaryMetadata
Data Primary Secondary
49
© 2008 Julian Dyke juliandyke.com
High Redundancy1 Disk per Failure Group
Disk 0 Disk 1 Disk 2
0 0
Failure Group 1
0 0
Failure Group 2 Failure Group 3
0111
2 2 2
3 33
444
5 5 50
11 1
22 2
3 3 3
44 4
55 5
6 6 6
77 7
88 8
9 9 9
Primary Secondary TertiaryMetadata
Data Primary Secondary Tertiary
50
© 2008 Julian Dyke juliandyke.com
References Oracle Automatic Storage Management (Oracle Press)
Nitin Vengurlekar Murali Vallath Rich Long
What ASM and ZFS Can Do For You Jason Arneil - Nominet
A Closer Look Inside Oracle ASM Luca Canali - CERN
Implementing ASM Without HW Raid Luca Canali - CERN
51
© 2008 Julian Dyke juliandyke.com
Thank you for listening
info@juliandyke.com
top related