Exadata Management for DBAs

Arup NandaLongtime DBA and Now DMA

2Arup Nanda

What it this About?• How Exadata is different?• Who manages it?– Oracle DBAs– System Admins– Composite Team

• Who does what?• What you need to know about managing it• What must you learn about potential issues

Arup Nanda

What is Exadata• Looks like an appliance• But is not an appliance. Why?– It contains additional software to make it a better

database machine– The components are individually adminstered

• That’s why Oracle calls it a Database Machine (DBM)

• DMA – Database Machine Administrator

3

Arup Nanda

The Magic #1

4

CPU

Memory

Network

I/O Controller

Disk

Get NAME … STATUS =

‘ANGRY’

iDB

Arup Nanda

Magic #2 Storage Cell Server

5

Disk1

Cell Server

Disk2 Disk3

• Cells are Sun Blades• Run Oracle Enterprise

Linux• Software called Exadata

Storage Server (ESS) which understands iDBESS

iDB

Arup Nanda

Magic #3 Storage Indexes

6

1 2 3 4

Disk4

MIN = 3

MAX = 5

MIN = 4

MAX = 5

MIN = 3

MAX = 5

MIN = 1

MAX = 2

Disk1

Cell Server

Disk2 Disk3

ESSStorage Indexes store in memory of the Cell Server the areas on the disk and the MIN/MAX value of the column and whether NULL exists. They eliminate disk I/O.

Storage Index

SELECT …FROM TABLEWHERE COL1 = 1

Arup Nanda

Disk1Disk1

Database Node 7

Cell 1Cell 1Cell 1

Put Together: One Full Rack

7

Database Node 1

Database Node 8

Cell 1 Cell 14

Disk1 Disk 12

InfiniBand Switch

Flash

Network SwitchClients connect to the database nodes.

RAC Cluster

Arup Nanda

Disk Layout

8

Storage Cell

Disk1Disk1

Disk1 Disk 12

Flash

Compute Nodes

• Disks (hard and flash) are connected to the cells.

• The disks are partitioned at the cell

• Some partitions are presented as filesystems

• The rest are used for ASM diskgroups

• All these disks/partitions are presented to the compute nodes

Arup Nanda

Disk Presentation

9

Physical Disk

Partition 1 Partition 2

Celldisk

Griddisk

filesystem

LUN LUN

ASM Diskgroup

ASM Disk

Disk

filesystem

Cell

Node

Arup Nanda

Command Components

10

Storage Cell

Disk1Disk1

Disk1 Disk 12

Flash

Compute Nodes

Disk1

Linux Commands – vmstat, mpstat, fdisk, etc.

Linux Commands – vmstat, mpstat, fdisk, etc.

CellCLI – command line tool to manage the Cell

ASM Commands – SQL*Plus, ASMCMD, ASMCA

Clusterware Commands – CRSCTL, SRVCTL, etc.Database Commands – startup, alter database, etc.

5-part Linux Commands article series http://bit.ly/k4mKQS4-part Exadata Command Reference article series http://bit.ly/lljFl0

Arup Nanda

Disk Failures

11

Cell 1 Cell 2

Datafile

block1 block1

block1

12Arup Nanda

Server Management• Sun Blades and Oracle Enterprise Linux• Normal Sysadmin Work– Shutdown, fdisk, etc.

• ILOM – Integrated Lights Out Management• KVM allows physical access– But you can use ILOM for virtual console

• Needs Pure Linux Skills

13Arup Nanda

Network Management• Two types of network– Ethernet– Infiniband

• Tools– ibstatus– iblinkinfo– verify-topology

14Arup Nanda

Special Oracle Provided Tools• All tools are found at

/opt/oracle.SupportTools• CheckHWnFWProfile– to check the HW profile

• Directory ibdiagtools/opt/MegaRAID/MegaCli/MegaCli64# ipmitool -H prolcel01-ilom -U root chassis

power on# imageinfo# imagehistory

15Arup Nanda

Database and Cluster Management• Cluster– crsctl, srvctl

• ASM– asmcmd, SQL*Plus

• Database– srvctl, SQL*Plus

16Arup Nanda

Storage Management• Two ways to manage the storage– Enterprise Manager– CellCLI

17Arup Nanda

Checking Storage Index Useselect decode(name, 'cell physical IO bytes saved by storage index', 'SI Savings', 'cell physical IO interconnect bytes returned by smart scan', 'Smart Scan' ) as stat_name, value/1024/1024 as stat_valuefrom v$mystat s, v$statname nwhere s.statistic# = n.statistic#and n.name in ( 'cell physical IO bytes saved by storage index', 'cell physical IO interconnect bytes returned by smart

scan’)

18Arup Nanda

Smart Scan Savings• Output

STAT_NAME STAT_VALUE---------- ----------SI Savings 0.000Smart Scan 0.000

• Smart Scan did not yield any savings• Why not?• Disable Smart Scans, if needed– cell_offload_processing = true;– _kcfis_storageidx_disabled = true;

19Arup Nanda

Why Not?• Pre-requisite for Smart Scan– Direct Path– Full Table or Full Index Scan– > 0 Predicates– Simple Comparison Operators

• Other Reasons– Cell is not offload capable• The diskgroup attribute cell.smart_scan_capable set to

FALSE;

– Not on clustered tables, IOTs, etc.

Arup Nanda

One Cluster?

20

1 2 3 4 5 6 7 8One Cluster

QA1 QA2 QA3 Prod1 Prod2 Prod3 Dev1 Int1

QA1 QA2 Prod4 Prod1 Prod2 Prod3 Dev1 Int1

QA1 QA2 QA3 Prod1 Prod2 Prod3 Dev1 Int1

Dev2

Arup Nanda

Many Clusters?

21

1 2 3 4 5 6 7 8QA Cluster

QA1 QA2 QA3 Prod1 Prod2 Prod3 Dev1 Int1

Prod Cluster Dev Int

1 2 3 4 5 6 7 8QA Cluster Prod Cluster Dev Int

22Arup Nanda

DBA

Sys Admin

Network Admin

Cell Admin

Administration SkillsSkill NeededSystem Administrator 10%Storage Administrator 0%Network Administrator 5%Database Administrator 60%Cell Administration 25%

23Arup Nanda

Resources• My Papers– 5-part Linux Commands article series http://bit.ly/k4mKQS– 4-part Exadata Command Reference article series

http://bit.ly/lljFl0• OTN Page on Exadata– http://www.oracle.com/technetwork/database/exadata/inde

x.html• Tutorials– http://www.oracle.com/technetwork/tutorials/index.html

• OTN Exadata Forum– https://forums.oracle.com/forums/forum.jspa?forumID=829

24Arup Nanda

Thank You!