•  Cluster:Ø  Aclusterisagroupofindependent,butinterconnectedcomputersthatactsasasinglesystem.Ø  Deployedtoincreasetheavailability,performanceandtobalanceadynamicallychangedworkloadottobalancetheloadacross

thesystems.Ø  Mostclusterarchitecturesuseadedicatedinterconnectnetworkforcommunica=onandcoordina=onbetweenclusternodes.It

alsousesshareddiskstoragearchitecutre.•  Clusterware:

Ø  Cluterwareisaso@warethatprovidesvariousinterfacesandservicesforacluster.Typically,thisincludescapabili=esthat:Ø  Allowtheclustertobemanagedasasingleen=ty.Ø  Protecttheintegrityoftheclustersothatdataisprotectedandtheclustercon=nuestofunc=onevenifcommunica=onwitha

clusternodeissevered.Ø  Maintainaregistryofresourcessothattheirloca=onisknownacrosstheclusterandsothatdependenciesbetweenresourcesis

maintained.Ø  Dealwithchangestotheclustersuchasnodeaddi=ons,removals,orfailures.Ø  Providesacommonviewofresourcessuchasnetworkaddressandfilesinafilesystem

•  OracleClusterware:Ø  OracleClusterwareprovidesabovecapabili=esandit’sakeypartofaGridInfrastructureandintegratedwithOracleAutoma=c

StorageManagement(ASM)andASMClusterFilesystem(ACFS).Ø  Comina=onofOracleClusterware,ASMandACFSprovidesfounda=onforOracleRealApplica=onClusters(RAC)database.Ø  Servicesprovidedbyoracleclusterwareinclude:

§  ClusterManagement:Allowsclusterservicesandapplica=onresourcestobemonitoredandmanagedfromanynodeincluster.

§  NodeMonitoring:Providesreal-=meinforma=onregardingwhichnodesarecurrentlyavailableandtheresourcestheysupport.Clusterintegrityisalsoprotectedbyevic=nforfencingunresponsivenodes.

§  EventServices:Publishesclustereventssothatapplica=onsareawareofchangesinthecluster.§  TimeSynchroniza=on:Synchronizesthe=meonallnodesofthecluster.§  NetworkManagement:ProvisionsandmanagesVirtualIP(VIP)addressesthatareassociatedwithclusternodesto

provideaconsistentnetworkiden=tyregardlessofwhichnodesareavailable.Inaddi=on,GridNamingService(GNS)managesnetworknamingwithinthecluster

§  HighAvailability:Services,monitorsandrestartsallotherresourcesasrequired.

•  OracleClusterwareStartup:Ø  OracleclusterwareisstartedbytheOSinitdaemon.Ø  Oracleclusterwareinstalla=onmodifiesthe/etc/iniSabfiletorestartohasdintheeventofacrash.Ø  OracleHighAvailabilityServicesdaemon(ohasd)isresponsibleforstar=nginproperorder,monitoring,andrestar=ng

otherlocaloracledaemons,upthroughthecrsddaemon,whichmanagesclusterwideresources.

Ø  OracleClusterwareProcess:Ø  CRS(CRSdaemon):

§  Managesclusterresourcesbasedonconfigura=oninforma=onthatisstoredinOracleClusterRegistry(OCR)foreachresource.Thisincludesstart,stop,monitorandfailoveropera=ons.crsdgenerateseventswhenthestatusofaresourcechanges.

§  Whenaresourcefailesitwilltrytorestartandifconfiguredwillgailovertoanothernodea@erexhaus=ngrestartaSempts.

§  ItisresponsibletomaintaintheOCR.§  CRSDprocesswritestotracefilesin$GRID_HOME/log/<node-name>/crs/crsd.log

Ø  Clustersynchroniza=onService:§  Managestheclusterconfigura=onbycontrollingwhichnodesaremembersoftheclusterandbyno=fyingmembers

whenanodejoinsorleavesthecluster.§  Itmasterthefirstac=venodeandothernodesintheclusteraresynchronizedbyregisteringwithmasternode.§  Itwritestracefiletoin$GRID_HOME/log/<node-name>/css/cssd.log§  CSShasthreeseparateprocess:theCSSdaemon(ocssd),theCSSAgent(cssdagent),andtheCSSMonitor(cssdmonitor).

1.   cssdmonitor=>Monitorsnodehangs(viaoprocdfunc=onality)andmonitorsOCCSDprocesshangs(viaoclsomonfunc=onality)andmonitorsvendorclusterware(viavmonfunc=onality).Thisisthemul=threadedprocessthatrunswithelavatedpriority.

Startupsequence:INIT-->init.ohasd-->ohasd-->ohasd.bin-->cssdmonitor2.   cssdagent=>SpawnedbyOHASDprocess.Previously(10g)oprocd,responsibleforI/Ofencing.Killingthisprocess

wouldcausenodereboot.Stops,startchecksthestatusofoccsd.bindaemonStartupsequence:INIT-->init.ohasd-->ohasd-->ohasd.bin-->cssdagent

3.   occsd.bin=>Managesclusternodemembershiprunsasoragriduser.Failureofthisprocessresultsinnoderestart.

Startupsequence:INIT-->init.ohasd-->ohasd-->ohasd.bin-->cssdagent-->ocssd-->ocssd.binØ  EventManager(EVM):

§  ProcessthatpublishesOracleClusterwareevents.Ithastwoprocessevmdandevmlogger.1.  evmd.bin=>Distributesandcommunicatessomeclustereventstoalloftheclustermemberssothattheyare

awareoftheclusterchanges.2.  evmlogger.bin=>StartedbyEVMD.binreadstheconfigura=onfilesanddetermineswhateventstosubscribeto

fromEVMDanditrunsuserdefinedac=onsforthoseevents.

Ø  DiskMonitor(DiskMonitorDaemon):§  Monitorsandperformsi/ofencingforOracleExadatastorageserver.Diskmondaemonisalwaysstartedwhen

ocssdisstarted.Ø  Mul=castDomainnameservice(mDNS):

§  UsedbyGridPlugandPlaytolocateprofilesinthecluster,aswellasbyGNStoperformnameresolu=on.ThemDNSprocessisabackgroundprocessonLinuxandUNIXandonWindows.

Ø  GridNamingService(GridNamingServiceDaemon):§  IsagatewaybetweentheclustermDNSandexternalDNSservers.TheGNSprocessperformsnameresolu=on

withintheclusterØ  OracleNo=fica=onService(ONS):

§  It’sapublishandsubscribeserviceforcommunica=onFANevents.Ø  Oraagent:

§  ExtendsclusterwaretosupportOracle-specificrequirementsandcomplexresources.ThisprocessrunsservercalloutscriptswhenFANeventsoccur.ThisprocesswasknownasRACGinOracleClusterware11gRelease1(11.1).

§  ohasd'soraagent:o  Performsstart/stop/check/cleanac=onsforora.asm,ora.evmd,ora.gipcd,ora.gpnpd,ora.mdnsd.

§  Crsd'soraagent:o  Performsstart/stop/check/cleanac=onsforora.eons,ora.ons,SCANlistenersandlistener.o  Performsstart/stop/check/cleanforservice,databaseanddiskgroupresources.o  ReceiveseONSevents,andtranslatesandforwardsthemtointerestedclients.

Ø  Clusterkilldaemon(oclskd):§  Handlesinstance/nodeevic=onsrequeststhathavebeenescalatedtoCSS.

Ø  GridIPCdaemon(gipcd):§  Isahelperdaemonforthecommunica=oninfrastructure.

Ø  Orarootagent:§  Aspecializedoraagentprocessthathelpscrsdmanagesresourcesownedbyrootsuchaasthenetwork,andthegrid

virtualipaddress.Ø  ClusterTimeSynchroniza=onservice(CTSS):

§  ProvidesTimeManagementinaclusterforOracleClusterwareØ  GridPlugandPlay(GPNPD):

§  ProvidesaccesstotheGridPlugandPlayprofile,andcoordinatesupdatestotheprofileamondthenodesoftheclustertoensurethatallofthenodeshavethemostrecentprofile.

Ø  SystemMonitor(Osysmond):§  Themonitoringandopera=ngsystemmetriccollec=onservicethatsendsthedatatoclusterloogerservice.The

servicerunsoneverynodeinacluster.•  ClusterwareComponents:

Ø  PrivateInterconnect:§  usedforcommunica=onbetweentheinstances.(fortransferringofblocksacrosstheinstances)§  Itisrecommendedtoisolateprivateinterconnecttousedifferentswitchasitrequiresfastdatatransfer§  privateinterconnectmustbeondifferentsubnetthanpublicandvirualip.

Ø  VirualIP:§  EachDatabasenodeinRACenvironmenthasonenodeipandonenodeVIPaddress.Themaindifferencebetween

thesetwoisnodevipscanmovetoanyothersystemincaseifcurrentworkingsystemiddownbutnodeIPcan'tdothat.Wheneverclientini=atesconnec=onusingSCANnamescanlistenerreplieswiththeleaseloadednode-VIP.

Ø  Vo=ngdisk:§  Vo=ngdiskisusedforpollingmechanismwhereeachandeverynodeintheclusterreportsishealthbypollingthe

disk.§  Italsoarbitratesclusterownershipamongtheinstancesincaseofnetworkfailures.§  Togetvotediskloca=on:

$]crsctlquerycssvotedisk.

Ø  OCR(OracleClusterRegistry):

§  OCRisusedtomaintainclusterresourcesasitstorestheinforma=onaboutclusterresourcesaswellastheresourcesdefinedinthecluster.

§  Registryisafilestoredinsharedstorage.§  EachnodemaintainsacopyofOCRinmemoryonlyoneprocessisdesignatedasmasterprocess§  ReadfromtheOCRisperformedbymasterOCRprocessandrefreshthelocalcopieswhereaswritetoOCRis

performedbythelocalOCRprocess.§  CRSDmasterautoma=callycreatesOCRbackupseveryfourhours,andCRSretainsthelastthreecopies.The

CRSDprocessalsocreatesanOCRbackupatthebeginningofeachdayandofeachweek,andretainsthelasttwocopies.

§  OCRCHECK:--DisplayshealthofOCR(OracleClusterRegistry).§  Togettheinforma=onaboutOCRdisks$]ocrcheck§  Togettheinforma=onaboutOLR(11gR2)

$]ocrcheck-local-config§  OLRresidesoneverynodeintheclusterandmanagesOracleClusterwareconfigura=oninforma=onforeach

par=cularnode§  Tolistocrbackup's$]ocrconfig-showbackup---listsallautoandmanualbackup§  Totakemanualbackup

$]ocrconfig-manualbackup---in11g§  Totakeanexportofocrfile

$]ocrconfig-export<dumpfile>§  TorestoreOCRfrombackup

$]ocrconfig-restore<backupfile>$]ocrconfig-import<dumpfile>

§  Tochangetheautobackuploca=on$]ocrconfig-backuploc<newloca=on>

Ø  SCAN:•  SingleClientAccessName(SCAN)issanewOracleRealApplica=onClusters(RAC)11gRelease2featurethat

provides•  asinglenameforclientstoaccessanOracleDatabaserunninginacluster.•  ThebenefitisclientsusingSCANdonotneedtochangeifyouaddorremovenodesinthecluster.Havingasingle

nametoaccesstheclusterallowsclientstousetheEZConnectclientandthesimpleJDBCthinURLtoaccessanydatabaserunningintheclustersindependentlyofwhichserver(s)intheclusterthedatabaseisac=ve.

•  SCANprovidesloadbalancingandfailoverofclientconnec=onstothedatabase.•  TheSCANworksasanIPaliasforthecluster.•  SCANis=edtomaximumof3ipaddressanditcanbeconfiguredinDNSorGNS.Itworksinroundrobin,.•  EachSCANIP(SCANVIP)hasaSCANLISTENERassociatedwithit.Ifit’sa4nodeRACcluster3SCANLISTENERS

runningon3SCANips'willrunon3nodes.•  AlldatabasesareregisteredwitheachSCANLISTENERintheClusterandPMONupdatesit'sloadtoeachSCAN

LISTENER.EachrequestgothroughusingSCAN_NAME,resolvestoSCANVIPi.e.SCANLISTENER.Now,SCANLISTENERredirectsittoVIPbydecidingusingLoadBalance.

Databaseclient

DNSGNS

Scanlistener

Scanlistener

Scanlistener

Listener

Listener

Rac-scan

192.168.56.211

DatabaselistenerrequestVIPofleast-loadednode

Ø  SharedStorage:§  Datafiles,controlfiles,redologfiles,spfile,OCRandVo=nddiskmustbepresentinthesharedstorage.§  From11gASMisthegotofilesystemforsharedstorage.

Ø  GPnPPofile:§  GPnPProfileisanXMLfilewhichcontainsinforma=onaboutclustername,Networkclassifica=on,storageto

beusedforASManddigitalsignatureofthecluster.•  HowCSSD,CRSandASMcomesupifOCRandVo=ngdiskisstoredinASMdiskgroup:

•  RACDatabase:Ø  SharedResources:

§  Datafiles§  Controlfiles§  Temptablespaces§  Vo=ngdiskandOCR§  RacInterconnect§  VIP's

Ø  Non-sharedresources:§  Onlineredologfiles§  UndoTablespace§  Archivelogfiles§  SGA§  Backgroundprocess§  Alertlog,listener.ora,tnsnames.oraandpfile§  Traceandauditfiles.

Ø  Interconnect-dedicated:§  Internodecommunica=on§  Heartbeats§  Datablocks.

Ø  CacheCoherency:§  Maintainingconsistencyofdatablocksinthebuffercacheofmul=pleinstancesiscalledcachecoherency.

Ø  CacheFusion:§  Cachefusionisamechanismbywhichblocksaretrasferredfromoneinstancetoanotherinstanceviaprivate

interconnect.

•  RACspecificSGAcomponent:Ø  GRD(GlobalResourceDirectory)

§  Itispartofthesharedpool,MaintainedbyGES(GlobalEnqueservices)andGCS(GlocalCacheservice)§  wheneverablockistransferedoutoflocalcachetoanotherinstance'scacheGRDisupdated§  itholdsthemetadataaboutdatablocks,thatareavailbleinDBBC§  itholdsinforma=onlike

1.  SCN(systemchangenumber)2.  DBI(datablockiden=fier)3.  loca=onoftheblock4.  modeoftheblock

I.  null(N)-Nullmodeistheleastrestric=vemode.Itindicatesnoaccessrights.actsasaplaceholder.II.  shared(S)-Sharedmodeindicatethatdatabaseblockisbeingreadandnotmodified.However

anothersessioncanreadthedatablockIII.  exclusive(E)-Exclusivemodeindicateexclusiveaccesstoblock.Otherresourcecannothavewrite

overthisdatablock.Howeveritcanhaveconsistentreadonthisdatablock.5.  roleoftheblock

I.  local-Whenadatablockisfirstreadintotheinstancefromthediskithasalocalrole.Meaningthatonly1copyofdatablockexistsinthecache.Nootherinstancecachehasacopyofthisblock.

II.  Global-Globalroleindicatesthatmul=plecopyofdatablockexistsinclusteredinstance.Forexampleauserconnectedtooneoftheinstancerequestforadatablock.Iftheblockislocalinoneinstanceandrequestedbyanotherinstancethenblockistransferredtothereques=ngnodeandtherolewillbeupdatedasglobal.

6.typesofdatablockimagei.  currentimage-updatedatablockvalueii.  consistentimage-previousdatablockvalueiii.  pastimage-grdupdatedimage

§  itconverttocurrentimagewheninstanceiscrash

•  RACDatabasespecificservices:Ø  GlobalCacheService:

§  GlobalCacheService(GCS)isthemaincomponentofOracleCacheFusiontechnology.ThisisrepresentedbybackgroundprocessLMSn.Therecanbemax10LMSprocessforaninstance.Themainfunc=onofGCSistotrackthestatusandloca=onofdatablocks.Statusofdatablockmeansthemodeandroleofdatablock(Iwillexplainmodeandrolefurther).GCSisthemainmechanismbywhichcachecoherencyamong“mul=plecache”ismaintained.GCSisalsoresponsibleforblocktransferbetweentheinstances.

Ø  GlobalEnqueueService:§  TheGESisprimarilyresponsibleformaintainingcoherencyinthedic=onaryandlibrarycaches.Thedic=onary

cacheconsistsofthedatadic=onarymasterinforma=onforeachnodeinitsSGA(SystemGlobalArea)primarilyforquickerlookupandaccess.AnyDDLcommiSedfromareques=ngnodeneedstobesync’edandwriSenacrossalldatadic=onariesinallnodesoftheRACenvironment.TheGESmakessurethatthechangesremainconsistentacrossthenodesandthattherearenodiscrepancies.Moreover,withthesamedirec=ve,thelocksmustbecreatedandmaintainedacrossthenodesandGESmustensurethattherearenodeadlocksacrossreques=ngnodesoveraccesstothesameobjects.LMON,LCKandLMDprocessesworkintandemtomaketheGESoperateinasmoothandseamlessfashion.

•  RACDatabaseBackgroundProcess:Ø  LMS(GlobalCacheServiceProcess):

§  Maintainsrecordsofdatafilestatusesandeachcachedblockbyrecordinginforma=oninaGRD.§  Controlstheflowofmessagestoremoteinstanceandmanagedglobaldatablockaccessandtransmitblock

imagesbetweenbuffercachesofdifferentinstances.Ø  LMON(GlobalEnqueueServiceMonitor:

§  Detectsinstancetransi=ons(joinsorleavesthecluster)andperformsreconfigura=onofGECandGCS.§  Monitorsglobalenqueuesandresourcesacrosstheclusterandperformglobalenqueuerecoveryopera=ons§  Responsibleforexecu=ngDynamiclocalremasteringevery10mins.

Ø  LMD(GlobalEnqueueserviceDaemon):§  Processesincomingenquerequestmessagesandcontrolsaccesstoglobalresources.§  Performsdistributeddeadlocksdetec=ons.

Ø  LCK0(InstanceEnqueuProcess):§  Managesnon-cachefusionresourcerequestssuchaslibraryandrowcacherequests.

Ø  LMHB:§  MonitorsLMON,LMDandLMSnprocesstoensuretheyarerunningnormallywithoutblockingorspinning.

DataonDisk

DatainRemotecache:3-way

DatainRemotecache:2-wayGlobalBufferCache

•  DynamicResourcereconfigura=on:

Ø  Whenoneinstancedepartsthecluster,theGRDpor=onofthatinstanceneedstoberedistributedtothesurvivingnodes.Similarly,whenanewinstanceentersthecluster,theGRDpor=onsoftheexis=nginstancesmustberedistributedtocreatetheGRDpor=onofthenewinstance.Thisiscalleddynamicresourcereconfigura=on.

•  DynamicResourceremastering:

Ø  InRACeverydatablockismasteredbyaninstance.Masteringablocksimplymeansthatmasterintstancekeepstrackofthestateoftheblockun=lthenextconfigura=oneventoccurs.GCSkeepstrackofthenumberofGCrequestsonaper-instanceandper-objectbasis.Thismeansthatifaninstance,comparedtoanotherisheaviliyaccessingblocksfromthesameobjects,theGCScantakethedecisiontodynamicallymigrateallofthatobject'sresourcestotheinstancethatisaccessingtheobjetmostandmakingitmasterofthoseblocks.LMON,LMDandLMSprocessesareresponsiblefordynamicremastering.

Ø  Remasteringcanbetriggeredasaresultof:§  Manualremastering§  Resourceaffinity(objectaffinity,undoaffinityandfilesaffinity)§  Instancecrash.

Ø  Tofindthecurrentmasterandpreviousmasterofaresource:SQL>selecto.object_name,m.CURRENT_MASTER,m.PREVIOUS_MASTER,m.REMASTER_CNTfromdba_objectso,v$gcspfmaster_infomwhereo.data_object_id=73181andm.data_object_id=73181;SQL>selectkj.kjblname,kj.kjblname2,kj.kjblowner,kj.kjblmasterfrom(selectkjblname,kjblname2,kjblowner,kjblmaster,kjbllockpfromx$kjblwherekjblname='[0x97][0x4],[BL]')kj,x$lelewherele.le_kjbl=kj.kjbllockporderbyle.le_addr;

•  Workingofcachefusion:INSTANCE1 INSTANCE2 INSTANCE3

Lockheld:XGO-------------------ExclGlobal

Nopastimage

Lockheld:none

Lockheld:NG1-------------------NullGlobalWith1past

image

Lockheld:none

Lockheld:SL0-------------------SharedLocalNopastimage

Lockheld:none

Lockheld:none

Lockheld:none

Lockheld:XLO-------------------ExclLocal

Nopastimage

Lockheld:none

Lockheld:SL0-------------------SharedLocalNopastimage

Lockheld:SL0-------------------SharedLocalNopastimage

Instage1datablockisrequestedbyauserCwhoisconnectedtoinstance3.Soadatablockisreadintothebuffercacheofinstance3.SQL>selectsalfromempwhereempno=10;àreturns300

Instage2samedatablockisrequestedbyauserBwhoisconnectedtoinstance2.Astheblockisalreadypresentininstance3itwillbeshippedtoinstance2viaprivateinterconnectusingcachefusion.SQL>selectsalfromempwhereempno=10;àreturns300

Instage3userBwhoisconnectedtoinstance2triestoupdatethesalofemployeeforempno10.Atthisstage,instance2acquiresEXCLUSIVElockforupda=ngthedataatinstance2andSHAREDlockfrominstance3isdowngradedtoNULLlock.roleoftheblockiss=lllocalasitisdir=edforthefirst=meonlyoninstance2SQL>updateempsetsal=400whereempno=10;commit;

Instage4userAwhoisconnectedtoinstance1triestoupdatethesalofemployeeforempno10.Datablockisshippedfrominstance2toinstance1andthelockoninstance2isdowngradedtonullandroleisglobalandhas1pastimage.instance1willhavexclusivelocalwithnopastimageSQL>updateempsetsal=500whereempno=10;

Lockheld:none

Lockheld:XLO-------------------ExclLocal

Nopastimage

Lockheld:none

Lockheld:SG1-------------------SharedGlobalWith1past

image

Lockheld:SG0-------------------SharedGlobalNopastimage

Lockheld:NG1-------------------NullGlobalWith1pastimage

Lockheld:NG1-------------------NullGlobalWith1past

image

Lockheld:XGO-------------------ExclGlobal

Nopastimage

Lockheld:NG1-------------------NullGlobalWith1past

image

Lockheld:SG1-------------------SharedGlobalWith1past

image

Lockheld:SG0-------------------SharedGlobalNopastimage

Lockheld:SG1-------------------SharedLocalWith1past

image

Instage5usercconnectedtoinstance3triedtoselectthesalofempno10.Datablockwillbecopiedfrominstanc1toinstance3andthelockoninstance1andinstance3willbesharedasinstance3requestedsharedlockifinstance3requestedxclusivelockthenoninstance3lockwillbenullSQL>selectsalfromempwhereempno=10;àreturns500

Instage6UserBissuesthesameselectstatementagainsttheemptablewithempno10oninstance2.Instance2willrequestforaconsistentcopyofbufferfromanotherinstance,whichhappenstobethecurrentmaster.Thereforeinstance1willshiptheblocktoinstance2,whereitwillberequiredwithSG1(SHARED,GLOBALwith1PASTIMAGE).Soinstance2modebecomesSG1.

Instage7UserConinstance3updatesthesamerow.Thereforetheinstance3requiresanexclusivelockandinstance1andinstance2willbedowngradedtoNULLlockwithGLOBALroleand1PASTIMAGE.Instance3willhaveEXCLUSIVElock,GLOBALroleandwithnoPASTIMAGES(XG0).

Instage8Thecheckpointisini=atedanda“WritetoDisk”takesplaceatinstance3.Asaresultpreviouspastimageswillbediscarded(astheyarenotrequiredforrecovery)andinstance3willholdthatblockinEXCLUSIVElockLOCALrolewithnoPASTIMAGES(XL0).

•  RACAdministra=on:Ø  Totroubleshootclusterstartup/clusterhealthissues:

§  ReviewthealertlogofGridinfrastructureunder$GRID_HOME/log/<node-name>/alert<node>.log§  Reviewtheclusterbackgroundprocessspecificlogfilesunder$GRID_HOME/log/<node-name>/<process>/

<process-name>.log§  Reviewtheocrchecklogfileunder$GRID_HOME/log/<node-name>/client/ocrcheck_<=mestamp>.logfile§  Nodeevi=onsissuesmightincludesomeaddi=onalclusterwarelogfiles.

/etc/oracle/lastgasp/*or/var/opt/oracle/lastgasp/*§  Iftheabovelogfilesdoesn'tprovidesufficientinforma=ontotroubleshoottheissuewecanenablethetrace/

debugsezngstogetthedetailinforma=on.Usebelowcommandstosetthedebug/tracelevels:$]crsctlgetlogcss/crs/evm/all-àgetthecurrentlevelofthetrace$]crsctlsetlogcrscrsmain=4$]crsctlsetlogcrsall-=3-àyoucansetthelevelfrom1to5higherthevaluehigherthedetailsprovided.$]exportSRVM_TRACE=TRUE

§  CVU(ClusterVerifica=onu=lityisusedtocollectpreandpostclusterconfigura=ondetailsatvariouslevelsandvariouscomponents.Italsoprovidestheabilitytoverifytheclusterhealth.

$]cluvfycomphealthcheck–collectcluster|database$]cluvfycompnodecon–n<node1>,<node2>--toverifynetworkconnec=ons$]cluvfystage-precrsinst–n<node1>,<node2>---toverifyprechecks

§  ClusterDiagnos=cCollec=onTool(diagcollec=on.sh):Asclustermanagesseverallogfilesitmaybe=meconsumingtocollectthefilesmanually.Diagcollec=onu=lityrefersvariousclusterlogfilesandgathersrequiredinforma=ontodiagnosecri=calclusterproblems

$]diagcollec=on.sh–collect–crs$GRID_HOME§  Loga=cketinmyOraclesupporttoreviewthefilesandprovidethecausefortheissue.§  LookaterrorreportinAIX(errpt–a)or/var/log/messages(linux)or/var/adm/messages(Solaris)or/var/adm/

syslog/syslog.log(HP-UX)toseeanyerrorsreportedattheoslevel

•  InterconnectTroubleshoo=ng:Ø  LookforlostblocksinAWRreports.Ø  Lookforerrorsino/pofifconfig–aØ  Lookforpacketreassemblesinnetstat-s

•  Nodeevic=ons:Ø  CSSDmonitorsnodesusing2communica=onchannels:

§  PrivateinterconnectónetworkHeartbeat§  Vo=ngDiskbasedcommunica=onóDiskHeartbeat.

Ø  ForN/Whearbeatnodesmustrespondincssd_misscount=me(defaultis30seconds)Ø  Lookfor"heartbeatfatal"incssd.logfile.Ø  N/Wheartbeatfailuresresultinnodeevic=ons.Ø  Fordiskheartbeateachnodeintheclusterpings(r/w)thevo=ngdiskseverysecond.Ø  Nodesmustreceiveresponsein(long/short)diskTimeout=me.Ø  Diskheartbeatfailureswillleadtonodeevic=onssearchfor"DskHeartbeat"inocssd.logfile.

•  CommandstomanageClusterresources:Ø  Tofindthemasternodeinthecluster:

§  Method#1)o  gotoGRID_HOME/log/<node-name>/cssdo  catocssd.log|grep-i'masternode'|tail-1---thiswillshowus#ofnodesintheclusterlocalnode#and

masternode#§  Method#2)

o  CRSDinMasternodeisresposibleforperformingautobackupofocr,solookingatOCRbackupwecaniden=fythemasternode

o  ocrconfig-showbackupauto---ListthenodeonwhichOCRwasbackedup

Ø  TochecktheCRSstatusonallthenodes:$]crsctlcheckcluster–all$]crsctlstatusserver–f

Ø  Tostop/startCRSonallthenodes(stopseverythingexceptohasd)$]crsctlstop/startcluster–all

Ø  Tostop/startcrsononenode(includingohasd)$crsctlstop/start/checkcrs

Ø  Tocontrolscanlistener:$]srvctlstop/status/startscan_listener$]srvctlstartscan_listener–i<scan_listener#>-n<node-name>$]srvctlstopscan_listener-i<scan_listener#>

Ø  Tostart/stopaspecificclusterresource$]crs_stop<resource-name>-àNeedtorunfromthesamenode.$]crs_start–c<node-name><resource-name>àif–cisusedwithstartwecanrunitfromothernode.

Ø  ToManageASMdiskgroup(WecannotstopthediskgroupusedbyOCRandVo=ngdisk)$]srvctlstart/stop/statusdiskgroup–g<dg-name>-node<node1>,<node2>

Ø  WecannotstoptheASMalone(weneedtobringdowncluster)Ø  Tomanagethelocallistener

$]srvctlstatus/stop/startlistenr–l<LISTENER-NAME>-n<node-name>Ø  Tomanagethedatabaseinstance:

$]srvctlstatus/stop/startinstance–i<instance-name>-d<db-name>Ø  ToManagedatabase:

Ø  Srvctlstatus/stop/startdatabase–d<db-name>-o<open/mount/'readonly'>Ø  Srvctlconfigdatabase–d<db-name>Ø  Srvctlmodifydatabase–d<db-name>-o<ORACLE_HOME>-p<SPFILE-LOCATION>

•  Usefulclusterwarecommands:Ø  Tocheckthenodesinthecluster:

$]olsnodesØ  Tochecktheclustername

$cemutlo–nØ  Toviewsubnetsoftheinterfacesusedandmodifythem

$oifcfgge=fàtogettheinterfacesubnet$oifcfgse=f–globaleth1/<new-subnet>:<interface-type>àToaddnewsubnet$]Ifconfigeth1<new-priv-ip>netmask<value>broadcast<maxvalueforthesubnet>$]oifcfgdelif–globaleth1<old-subnet>:<interface-type>àTooldnewsubnet

Ø  Tocheckthevo=ngdisk$]crsctlquerycssvotedisk

Ø  Tochecktheocrinterity$]ocrcheck

Ø  Tomovevotedisktodifferentdiskgroup$]crsctlreplacevotedisk<new-dg-name>

Ø  TomoveOCR$]ocrconfig–manualbackup$]ocrconfig–add<new-dg-name>$]ocrcheck$]ocrconfig–delete<old-dg-name>

Ø  Tochangethescanlistenerport$]srvctlmodifylistener–p<port#>

•  GlobalCacheWaitEvents

Ø  2-wayBlockRequest:(Itwillbeon2nodeRAC)§  SGA1sendsadirectrequesttoSGA2.SoSGA1waitsonthegccurrentblockrequestevent.§  WhenSGA2receivestherequest,itslocalLGWRprocessmayneedtoflushsomerecoveryinforma=ontoitslocal

redologfiles.§  Then,SGA2sendstherequestedblocktoSGA1.WhentheblockarrivesinSGA1,thewaiteventiscomplete,andis

reflectedasgccurrentblock2-way.

Ø  3-wayblockrequest§  Thisisamodifiedscenarioforaclusterwithmorethantwonodes.However,themasterforthisblockisona

nodethatisdifferentfromthatoftherequestor,andwheretheblockiscached§  Thus,therequestmustbeforwarded.§  WhentheblockarrivesinSGA1,thewaiteventiscomplete,andisreflectedasgccurrentblock3-way.

Ø  2-wayGrant§  Inthisscenario,agrantmessageissentbythemasterbecausetherequestedblockisnotcachedinany

instance.§  Ifthelocalinstanceistheresourcemaster,thegranthappensimmediately.Ifnot,thegrantisalways2-way,

regardlessofthenumberofinstancesinthecluster.§  Theround-triplookssimilartoa2-wayblockround-trip,withthedifferencethatthewire=meisdetermined

byasmallmessage,andtheprocessingdoesnotinvolvethebuffercache

RACInstalla=onTriage:

•  WhyweneedVIPS?Ø  WithoutVIPsifusertriestoconnecttoRACdatabasewithallthelocalip'softheRACinaddresslist.Ifoneofthe

nodeisdownandifuserconnec=onwillwaitun=lit=mesoutbeforemovingtoothernode.TimeoutdependsonOSanditmayvaryfromscondsto2minutes.

Ø  WithVIPSasCRSfailovertheVIPtootherhealthynode.WhenusetriestoconnectwithVIPSitwillgetthenolistenerimmediatelyanditmovestoothernodeimmediately.

Ø  In11gR2evenwithscanvip'sitprovidesthesamefunc=onality.

•  WhySCAN?Ø  WithSCANwheneverweaddordeleteanodeuserdoesn’tneedtomakeanychangestotheconnec=onstring.