presentation - oracle exadata architecture overview

Oracle Exadata Storage Server Architecture

Overview

Agenda

Exadata Software Architecture

Exadata Storage Layout

Exadata Storage Scale-O

ut Architecture

2008 Oracle Corporation Proprietary and Confidential

2

ExadataSoftware Architecture


3

Exadata Architecture

DB Server

DB Instance

DBRM

ASM

Single-Instance

Database

RAC

Database

DB Server

DB Instance

DBRM

ASM

DB Server

DB Instance

DBRM

ASM

iDB Protocol over

Enterprise

Manager


4

OEL

CELLSRV

MS

RS

IORM

Exadata Cell

InfiniBand with

Path Failover

InfiniBand Switch/Network

Cell

Control

CLI

OEL

CELLSRV

MS

RS

IORM

Exadata Cell

OEL

CELLSRV

MS

RS

IORM

Exadata Cell

Exadata Storage Server Overview

Exadata Storage Server

Data

Data

Data


5

Network Fabric

Evaluate Disk, CPU, and Network Throughput Standalone


Data

Data

Data

Database Server

RDBMS/ASM instance

SGA

IO Client

IO Layer

ASM layer



6

Network Fabric

ASM layer

?

How do we connect the IO/ASM layer of ASM/DB to Exadata?

?


Database Server

RDBMS/ASM instance

SGA

cellsrv

Data

Meta data

Data

Meta data

Data

Meta data

IO Client

IO Layer

ASM layer



7

Network Fabric

libcell

ASM layer

libcell linked with DB/ASM talks to cellsrv

iDB protocol is born

Multiple threads in cellsrv

Threads perform

asynchronous IO to disks and network


Database Server

RDBMS/ASM instance

SGA

Data

Meta data

Data

Meta data

Data

Meta data

IO Client

IO Layer

ASM layer


cellsrv


8

Network Fabric

libcell

ASM layer

Smart Scan enabled in RDBMS and CELLSRV


Database Server

RDBMS/ASM instance

SGA

Data

Meta data

Data

Meta data

Data

Meta data

/etc/oracle/cell/network-config

IO Client

IO Layer

ASM layer


cellsrv


9

Network Fabric

cellinit.ora

Local IP

/etc/oracle/cell/network

libcell

ASM layer

cellinit.ora decides which network takes Storage traffic


Database Server

RDBMS/ASM instance

SGA

cellip.ora

Cells

Data

Meta data

Data

Meta data

Data

Meta data

IO Client

IO Layer

ASM layer


cellsrv



10

Network Fabric

cellinit.ora

cellip.ora

Local IP

libcell

ASM layer

cellip.ora on database/ASM host maintains list of cells

New cells can be added to cellip.ora dynamically



Database Server

RDBMS/ASM instance

dskm

diskmon

SGA

cellip.ora

Cells

Data

Meta data

Data

Meta data

Data

Meta data

IO Client

IO Layer

ASM layer


cellsrv



11

Network Fabric

cellinit.ora

cellip.ora

Local IP

libcell

ASM layer

Master diskmon (diskmon) starts with CSS and communicates with cellsrv

Slave diskmon (dskm) is part of every instance and communicates with m

aster diskmon

Handles cell failures, IO fencing, IO resource m

anagement plan propagation



Database Server

RDBMS/ASM instance

dskm

diskmon

SGA

cellip.ora

Cells

ms

cellcli

Data

Meta data

Data

Meta data

Data

Meta data

IO Client

IO Layer

ASM layer


cellsrv



12

Network Fabric

cellinit.ora

cellip.ora

Local IP

libcell

ASM layer

Cellcli allows user interaction and configuration

Management Server (M

S) displays and m

anages creation and deletion of

griddisks, changes in hardware, SNMP traps, alerts, email, m

etrics etc



Database Server

RDBMS/ASM instance

dskm

diskmon

SGA

cellip.ora

Cells

ms

cellcli

Data

Meta data

Data

Meta data

Data

Meta data

IO Client

IO Layer

ASM layer


cellsrv



13

Network Fabric

cellinit.ora

cellip.ora

Local IP

rs

libcell

ASM layer

Restart Server (RS) monitors CELLSRV and M

S, Backup RS m

onitors core RS

RS monitors for process aliveness, memory usage etc



Database Server

RDBMS/ASM instance

dskm

diskmon

SGA

cellip.ora

Cells

ms

cellcli

Data

Meta data

Data

Meta data

Data

Meta data

IO Client

IO Layer

ASM layer


cellsrv



14

Network Fabric

cellinit.ora

cellip.ora

Local IP

rsADR

adrci

libcell

ASM layer

Trace files and alert logs in Automatic Diagnostic Repository on cell

alert.log (from RS and CELLSRV), m

s-odl.log, ms-odl.trc, rs*trc, svtrc*.trc



Database Server

RDBMS/ASM instance

dskm

diskmon

SGA

cellip.ora

Exadata cells

ms

cellcli

Data

Meta data

Data

Meta data

Data

Meta data

IO Client

IO Layer

ASM layer


cellsrv



15

Infiniband Fabric

cellinit.ora

cellip.ora

Local IP

rsADR

adrci

libcell

ASM layer

Infiniband fabric requires RPMs from OFED/OEL, RedHat 5.1, OEL 5.1

Exadata Storage Server works only with 11.1.0.7 Database/ASM



From the Enterprise

ms

cellcli

Data

Meta data

Data

Meta data

Data

Meta data

Enterprise M

anager

OMS

Agent

dcli


cellsrv


16

Network Fabric

rsADR

adrci

EM Plugin Provides central location for metrics and alerts across cells

No agent runs on cell

dcli allows user to run commands across cells

ssh/cellcli

ExadataStorage Layout


17

Exadata Storage Layout

Physical disks map to a Cell Disks

Cell Disks partitioned into one or multiple Grid Disks

ASM diskgroups created from Grid Disks

Transparent above the ASM layer


18

Cell

Disk

Sys Area

Sys Area

Grid Disk n

Grid Disk 1

ASM disk

ASM disk

Physical

Disk

Exadata Cell

Exadata Cell

Exadata Storage Layout Example

Cell Disks

Cell

Disk


19

Cell Disk is the entity that represents a physical disk

residing within a Exadata Storage Cell

Automatically discovered and activated

Exadata Cell

Exadata Cell


Grid Disks

Grid

Disk


20

Cell Disks are logically partitioned into Grid Disks

Grid Disk is the entity allocated to ASM as an ASM disk

Minimum of one Grid Disk per Cell Disk

Can be used to allocate hot, warm

and cold regions of a

Cell Disk or to separate databases sharing Exadata Cells


ASM Disk Groups and Mirroring

Exadata Cell

Exadata Cell

Hot

Hot

Hot

Hot

Hot

Hot

Hot ASM

Disk Group

Cold ASM

Disk Group


21

Two ASM disk groups defined

One for the active, or hot portion, of the database and a

second for the cold or inactive portion

ASM striping evenly distributes I/O across the disk group

ASM mirroring is used protect against disk failures

Optional for one or both disk groups

Hot

Hot

Hot

Hot

Hot

Hot

Cold

Cold

Cold

Cold

Cold

Cold


ASM Mirroring and Failure Groups

Exadata Cell

Exadata Cell

Hot

Hot

Hot

Hot

Hot

Hot

ASM

Failure Group

ASM

Failure Group


22

ASM mirroring is used protect against disk failures

ASM failure groups are used to protect against cell failures

Hot

Hot

Hot

Hot

Hot

Hot

Cold

Cold

Cold

Cold

Cold

Cold

ASM

Disk Group

ExadataScale-Out Architecture


23

Scale-Out Architecture

The database, ASM, and Exadata

Cells each play a role in Oracles

scale-out storage architecture

Responsibilities are placed in the

optimal location

DB Business Data Protection

ASM

Reliable Storage

Business

Data Protection

2008 Oracle Corporation

24

DB Business Data Protection

ASM Reliable Storage Pool

Exadata Cell Database Intelligent

Storage

Seamless integration simplifies

management

Reliable Storage

Pool

Exadata

Database

Intelligent

Storage

Exadata Scale-Out Overview

Cells implement Oracles iDB data protocol

Expose storage as collections of intelligent

network disks called Grid Disks to ASM and the

DB

iDB is conceptually similar to iSCSI but has

extensive DB intelligence

iDB is layered on top of ZDP network protocol

Grid Disks


25

iDB is layered on top of ZDP network protocol

To ensure full scale-out benefits, cells

never communicate with each other

Cross cell operations are implemented in ASM or

DB

Cell independence ensures no perform

ance

bottlenecks, and no cascading failures

Key to scalability architecture

iDB data protocol

ZDP net protocol

Exadata Cell Integration for Scale-Out

Cell appliance design elim

inates storage configuration

missteps and administrative overhead

Simple provisioning

Cell grid disks are automatically made visible to ASM

No OS level LUNs or mount points to setup and manage

Cross cell ASM m

irroring is automatically configured


26

Cross cell ASM m

irroring is automatically configured

for grid disks

ASM failure group topology

Multiple Grid disks per physical disk allow m

ultiple

ASMs to use the same cells

Automatic Storage Management

Integrated cluster volume

manager

Flexible data distribution

(striping)

Mirroring

ASM Disk

ASM Disk

ASM Disk

ASM Disk

ASM Disk


27

ASM creates a reliable storage pool

from Exadata Grid disks

Mirroring

Automatic data re-balancing

Free

ASM Scale-out Data Distribution

ASM manages storage in megabyte

allocation units

Each DB file consists of a set of

allocation units

The location of a files allocation units

are individually tracked by ASM

ASM evenly spreads allocation units

DB

File #1

DB

File #2

Files automatically spread

across disks to balance load

ASM


28

ASM evenly spreads allocation units

across all cells and disks in the grid

All disks evenly utilized

Optimal perform

ance

File #1

File #2

ASM Data Re-Distribution

Cells can be easily added or removed

from the storage grid

ASM migrates a fraction of the allocation

units to the new cell

Online and transparent to application

Minimal data m

ovement to new or

removed cells

DB

File #1

DB

File #2

ASM


29

File #1

File #2

Online Provisioning and Capacity Management

New Cell

Protection from Hardware Failure

ASM implements mirroring at the

allocation unit level

Primary and mirror copy of allocation

units are placed on separate storage

cells

Automatically remirrors across all

DB

File #1

DB

File #2

ASM


30

Automatically remirrors across all

remaining cells when a disk or array fails

Failure of disk or array is transparent to

database

Designed to Tolerate Failures of Cells and Disks

Protection from Brownout

ASM temporarily freezes I/O to

unresponsive cells

Redirect read IOs to mirror copy

Track write IOs that are m

issed

Writes replayed when cell begins

responding

Fast mirror resync

No need to remirror all the unchanged

DB

File #1

DB

File #2

ASM


31

No need to remirror all the unchanged

data

Benefits

Fast recovery from transient failures

e.g Cell crash or temporary hang

Can be used for planned m

aintenance

Cell software or component upgrade

Freeze I/O

to Cell

Designed to Gracefully Tolerate Cell Brownouts

Data Protection Solutions

All single points of failure elim

inated by the Exadata Storage

architecture

Hardware Assisted Resilient Data (HARD) built in to Exadata Storage

Prevent data corruption before it happens

Data Guard provides disaster protection and data corruption

protection

Automatically m

aintained second copy of database


32

Flashback provides human error protection

Snapshot-like capabilities to rewind database to before error

Recovery Manager (RMAN) provide backup to disk

Archiving and corruption protection

Can be used with Oracle Secure Backup (OSB) or third party tape

backup software

These work just as they do for traditional non-Exadata storage

Users and database administrator use familiar tools

Exadata Storage Management & Administration

Enterprise Manager

Manage & administer Database and ASM

Exadata Storage Plug-in

Enterprise M

anager Grid Control Plug-in to monitor &

manage Exadata Storage Cells

Comprehensive CLI


33

Comprehensive CLI

Local Exadata Storage cell management

Distributed shell utility to execute CLI across m

ultiple

cells

Integrated Lights Out Management -ILOM

Remote management and administration of hardware


34

presentation - oracle exadata architecture overview

Documents

oracle corporation proprietary

io fencing

asynchronous io

cellsrvslave diskmon

cellsrvidb protocol

ioasm layer of asmdb

bornmultiple threads

databaseasm host