exadata troubleshooting

Slide 1

Last updated Jun 26, 2014 GTPLUS

Exadata Troubleshooting

1

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template2

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template4 ?

Crash Hang

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template5 MOS Notes

888828.1 Exadata environment MOS 1070954.1 - exachk Best Practices DB IB switch . Asrexachk (1450112.1) Snmp .

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template6 ?

DB IORM or DBRM

(.bash_history)

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template7Sundiag

/opt/oracle.SupportTools/sundiag.sh DB .The sundiag tool cellcli ILOM snapshots & Megacli raid card logs .

failure or reboot DB , sundiag .

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template8Sundiag

Sundiag oswatcher dmesg /var/log/messages

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template9ILOM (Integrated Light Out Manager)

History ipmitool sunoem cli "show /SP/console/history ipmitool -I lanplus -H celadm01-ilom -U root -P welcome1 sunoem cli"show /SP/console/history"

ILOM ipmitool -c sunoem cli "show -script /SP/logs/event/list ipmitool -I lanplus H celadm01-ilom -U root -P welcome1 sunoem cli"show -script /SP/logs/event/list

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template10ILOM

ipmitool -I lanplus H celadm01-ilom -U root -Pwelcome1 sunoem cli "show faulty

sundiag ILOM snapshot remote snapshot

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template11ILOM

ILOM ILOM=cell01-ilom HOST=db01 ipmitool sunoem cli "set /SP/diag/snapshot dataset=normal" -H $ILOM-U root P welcome1 ipmitool sunoem cli "set /SP/diag/snapshot dump_uri=sftp://root:welcome1@$HOST/tmp" -H $ILOM -U root -P welcome1 ipmitool sunoem cli "show /SP/diag/snapshot" -H $ILOM -U root -Pwelcome1

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template12ILOM

cel07-c_10.245.20.169_2013-09-20T16-51-21.zipset /SP/diag/snapshot dataset=normal set /SP/diag/snapshot dump_uri=sftp://root:welcome1@172.16.20.1/tmpcd /SP/diag/snapshotshowProperties:dataset = normaldump_uri = (Cannot show property)encrypt_output = false** result = Running **

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template13ILOM

ILOM snapshots , , Fault

ILOM Fault .

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template14DB

OSWatcher ? CPU ? IO ?

ExaWatcher/OSWatcher & .

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template15RAC

$GI_HOME/bin/diagcollect.pl --crs , (default all) --aftertime beforetime

OCR & vote disks ocrcheck crsctl query css votedisk

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template17RAC

Exa/OSWatcher .

Exadata Diagnostic collection . Diagnostic Assistant (201804.1) Trace File Analyzer (1513912.1)

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template16DB Hung

ILOM ILOM overwrite .

MOS 1352805.1 hung SysRq

Attempting to gracefully reboot hung Exadata cell or database node ( ID 1352805.1)

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template18DB Hang

Alertlog ORA-600/7445 I/O .

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template19DB Hang

Hung . ASH AWR ADDM EXA/OSWatcher

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template20DB Hang

DB Hung ?SQL> oradebug g all hanganalyze 1SQL> oradebug g all systemstate 258

Hang , RDA .

DB ASM Disk .

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template21ASM

v$asm_disk offline disk

v$asm_operation

offline v$asm_operation resync (list griddisk checks asm)

(kernel files OSM disk) kfod asm_diskstring='o/*/*' disks=all op=disk

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template22ASM

/etc/oracle/cell/network-config/cellip.ora ASM cellip.ora (with caution)

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template23

. (cell, cell disk, etc.).

METRICDEFINITION . METRICDEFINITION objects describe the metrics.

METRICCURRENT Set .

METRICHISTORY . THRESHOLD alert rule .

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template24

:- Cell metrics CPU , Cell - Cell disk metrics large block - Grid disk metrics - large block - Host interconnection metrics I/O - IORM metrics Category, Database and Consumer Group metrics. IORM

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template25 )Number of requests

to Read Small Blocks

Number of requests

to Write Small Blocks

Number of [Mega]bytes

written in Large Blocks

IO latency for ReadCD_IO_RQ_R_SM

CD_IO_RQ_R_SM_SEC

CD_IO_RQ_W_SM

CD_IO_RQ_W_SM_SEC

CD_IO_BY_W_LG

CD_IO_BY_W_LG_SEC

CD_IO_TM_R_SM_RQC

R

C

R

C

R

RIO req

IO/sec

IO req

IO/sec

Mb

Mb/sec

us/req small Blocks

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template26IORM: DB )Number of requests

for Small Blocks

Number of requests

for Large Blocks

IORM wait time for

read/write Small Blocks

IORM wait time for

read/write Small BlocksDB_IO_RQ _SM

DB_IO_RQ_SM_SEC

DB_IO_RQ_LG

DB_IO_RQ_LG_SEC

DB_IO_WT_SM

DB_IO_WT_SM_RQ

DB_IO_WT_LG

DB_IO_WT_R_LG_ RQC

R

C

R

C

R

C

RIO req

IO/sec

IO req

IO/sec

us

us/req

us

us/req

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template27

cellcli -e list flashcachecontent attributes all|sed -e 's/^[ \t]*//' -e 's/\t/,/g'-e 's/ //g' -e 's/$/,$(date '+%Y%m%d%H%M')/' -e 's/^/${celliphost},/' list metriccurrent CD_IO_TM_W_SM_RQ where metricObjectNamelike 'FD.*' dcli

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template28

Imageinfo

List alerthistory

$CELLTRACE and $LOG_HOME alert history alert.log ms-odl.trc

Copyright 2013, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template29

$CELLTRACE/alert.log file ora-600/7445 or

cellcli list alerthistory