zero data loss recovery appliance: best pracfces from ... › technetwork › database › ... ·...

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

ZeroDataLossRecoveryAppliance:BestPracFcesfromCustomerDeployments[CON6535]

MarcoCalmasini,Sr.PrincipalProductManager,OracleJonySafi,ConsulFngMemberofTechnical.Staff,OracleGaganSingh,Sr.DBArchitect,IntelCorporaFon


SafeHarborStatementThefollowingisintendedtooutlineourgeneralproductdirecFon.ItisintendedforinformaFonpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnotacommitmenttodeliveranymaterial,code,orfuncFonality,andshouldnotberelieduponinmakingpurchasingdecisions.Thedevelopment,release,andFmingofanyfeaturesorfuncFonalitydescribedforOracle’sproductsremainsatthesolediscreFonofOracle.

3


RecoveryApplianceProgramAgendaBusinessValues

NewandSimplerWorld

BestPracFces–Manageability

BestPracFces–ConfiguraFon

BestPracFces–BackupandRecovery

BestPracFces–ValidaFon,SecurityandTroubleshooFng

TheIntelexperience–arealcasescenario

1

2

3

4

5

4

6

7



NewandSimplerWorld






1

2

3

4

5

5

6

7


TradiFonalbackupsoluFons

DailyBackupWindowLargeperformanceimpactonproducFon

DataLossExposureLosealldatasincelastbackup

ManySystemstoManageScalebydeployingmorebackupappliances

PoorDatabaseRecoverabilityManyfilesarecopiedbutprotecFonstateofdatabaseisunknown

6


ZeroDataLossRecoveryAppliance

NeedaFundamentallyDifferentApproachtoProtectBusinessCriQcalDatabaseData

7


RecoveryApplianceUniqueBenefitsforBusinessandI.T.

MinimalImpactBackupsProducFondatabasesonlysendchanges.Allbackupandtapeprocessingoffloaded

Savings,BackupTimeShrinks

EliminateDataLossReal-FmeredotransportprovidesinstantprotecFonofongoingtransacFons

ZerotoSub-1sRPO

Cloud-ScaleProtecQonEasilyprotectalldatabasesinthedatacenterusingmassivelyscalableservice

HAandScalableArchitecture

DatabaseLevelRecoverabilityEnd-to-endreliability,visibility,andcontrolofdatabases-notdisjointfiles

RecoveryReadiness,ValidaFons

8


ZeroDataLossRecoveryApplianceOverview

DeltaPush• DBsaccessandsendonlychanges• MinimalimpactonproducFon

• Real-FmeredotransportinstantlyprotectsongoingtransacFons

ProtectedDatabases

ProtectsallDBsinDataCenter• Petabytesofdata• Oracle10.2-12c,anyplagorm• NoexpensiveDBbackupagents

DeltaStore• Backupsvalidated,compression,deduplicaFon• Fastrestorestoanypoint-in-Fmeusingdeltas• BuiltonExadatascaling,HAandresilience• EnterpriseManagerend-to-endcontrol

RecoveryAppliance

ReplicatestoRemoteRecoveryApplianceforDR

9

OffloadsTapeBackup

IntegratedMediaManager/ThirdPartyBackupClientSW


RecoveryApplianceTimeline

10

RAX4HW

RecoveryAppliance–AddressingWeaknessesofBackupAppliances

RAX5HW

8TBdrives–2XCapacity–20XmoreEffecFveStorage

RAX6HW

UpgradedHW–BekerPerformance

FileSystemBackup–FullStackProtecFon

RAX6HW

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.| 11

HundredsofSystemsShippedin2YearsSinceLaunchAcrossAllGeographiesRecoveryAppliance–DataProtecFonAroundtheWorld

Finance Utilities Education Telco Manufacturing



NewandSimplerWorld






1

2

3

4

5

12

6

7


TradiFonalBackupArchitecture

13

ProtectedDBclients• Resourceconsumingagents• Expensivelicenses• Fragmentedmanagement

MediaServers• Expensivelicenses• ManualloaddistribuFon• Frequentlyoverloaded• Disparateinfrastructure• CopiestotapeuFlizesoverloadedmediaservers

TargetDevices•  Ifdiskarray–norealscale-out• OmenlackinginHA•  LimiteddeduplicaFon


ProtectedDBClients• Noagents• Resourcesfullydedicatedtobusiness• Nolicenses• EMmanagesfulllifecycle

MediaServers• Notneeded• HWcanberepurposed• Lessnetworktraffic• Easiermanagement

TargetDevices• Directlyintegrated

Comparison:IntroducingRecoveryAppliance


Pre-Built,OpFmizedandHighlyAvailableOut-of-the-Box

Performan

ceand

HighAv

ailabilityAc

hievem

ent

Time(Days)

Time(Months)

100%

Measure, diagnose

and reconfigure

Recoverability Quality Assemble

dozens of components

Multi-vendor finger

pointing

CustomConfiguraFon

Performan

ceand

HighAv

ailabilityAc

hievem

ent


RecoveryApplianceBestPracFcesJonySafi

16



NewandSimplerWorld






1

2

3

4

5

17

6

7


BestpracFcesrule#1•  TheRecoveryApplianceisnotanExadatamachine,it’sanAPPLIANCE•  Nochangesneededforgreatperformance,HAandresilience

•  FollowthespecificZDLRAdocumentaFonandMOSnotes

•  CreateEMalertsonnewincidents,keymetrics,etc.

•  InformaQonCenter:OverviewZeroDataLossRecoveryAppliance(DocID1683791.2)

•  RunExachkmonthly(OracleExadataDatabaseMachineexachkorHealthCheck(DocID1070954.1)).

•  SubscribetoMOSalertsandreferperiodicallytothefollowingnotes:–  ZeroDataLossRecoveryApplianceCriQcalIssues(DocID1927928.1)forcriQcalissuesalerts–  ZeroDataLossRecoveryApplianceSupportedVersions(DocID1927416.1)forlatestsoewareupdates.

18


BackupManageabilityBestPracFces• Configureprotecteddatabase– UseEnterpriseManagerCloudControl•  SimplestdeploymentandconfiguraFonfor11gand12c

– Stepstobackupdatabase•  CreateProtecFonPolicyonRecoveryAppliance(RA)•  AddProtectedDatabasetoRA•  ConfigureBackupSerngsforProtectedDatabase•  Scheduleone-FmeLevel0(Full)Backupwith“CustomBackup”,thenLevel1swith“Oracle-SuggestedRecoveryApplianceBackup”

19


WhattodowithexisFngbackups?

• RMANbackupstodiskorNFSshare(includingDataDomainshare)– CanbeimportedintotheRecoveryAppliancevia“polling”

• Backupstakenusing3rdpartybackupsomware– LeavetheagentinplaceontheprotectedDBhostsunFlretenFonexpires.– Removingagentssavesystemresources

•  InterimperioddualbackuptargetMOSnotes:– ImplemenFngaDualBackupStrategywithBackupstoDisk/TapeandRecoveryAppliance(DocID2154471.1andDocID2154461.1)– Removingdualbackupseliminatesexpensivestorage,tapehardwareandreducesbackupwindowssignificantly

20



NewandSimplerWorld






1

2

3

4

5

21

6

7


NetworkconfiguraFonopFons

•  10GEandIB– ZDLRAsupports10GigE(default)andInfiniBand.Default10gigEprovides12TB/hourbackupandrestorerates.– Owner’sGuide,Chapter9hasdetailsonhowtoconfigureIngestoverIB.– Note:Real-TimeRedoTransportwilluse10GigEnetworkONLY.

• VLANforsecurity– BackupandrestoretrafficfromdifferentVLANsisnotrouted.– TheRAsupportsVLANtaggingontheingestnetworksoprotectedDBhostsresidingondifferentandisolatedVLANscanbeconnecteddirectlytotheRA.– SeeMOSnote:Enabling8021.QVLANTagginginZeroDataLossRecoveryApplianceOverIngestNetworks(DocID2047411.1)

22


CustomizaFons• DonotmakeanychangestoRecoveryApplianceservers(again,IT’SANAPPLIANCEandit’salreadyopFmized).– SeeMOSnote:ConsequencesofmodifyingtheRecoveryAppliance(DocID2172842.1)-forrestricFonsandsupportedconfiguraFonexcepFons.

• UseMAX_RETENTION_WINDOWtoenforcehardlimitsondataretenFonforalldatabaseswithinaprotecFonpolicy.– Backupsareforciblyremovedamerexceedingwindow– Principalusecaseisforcompliance/regulatoryrequirements– UseRecoveryWindowGoalstomanagebackupspaceconsumpFon.

23



NewandSimplerWorld






1

2

3

4

5

24

6

7


BackupbestpracFces• CreateIncrementalL0(full)asfirstbackup• CreatesubsequentCumulaFveL1backups“IncrementalForever”• UseSecFonSizeforLargeDatafiles(e.g.1+TB)– Use64GBasstarFngpointandevaluateupto(aggregatedatafilessize/#channels)

• VirtualFullBackupCreaFonMonitoring– AmeraL1incrementalbackuptheRAindexesitandbuildsthecorrespondingVirtualFull.CheckEMforerrormessageslike“ORA-64760:DatabaseXYZhashadtasksinorderingwaitstateforoverXdays.”•  RefertoMOSnote:DiagnosQcSQLscriptfortasksinORDERING_WAITstatusonRecoveryAppliance(DocID2095949.1)

25


BackupbestpracFces–Cont.• UseTransparentDataEncrypFon(TDE)insteadofRMANencrypFon• UsenaFvedatabasecompressioninsteadofRMANcompression• Useblockchangetrackingforallprotecteddatabases

26

–  $rmantarget<targetstring>catalog<catalogstring>backupdevicetypesbtcumulaFveincrementallevel1filesperset1secFonsize64gdatabaseplusarchivelognotbackedupfilesperset32;


RestoreandRecoveryBestPracFces

• WhenthereisnovalidateddisasterrecoveryplanandnoRecoveryAppliance,badthingshappen.•  TrueStoryofaRestoreOperaFongoneterriblywrong.– DatabaseFailureOccurs–mustberestoredfrombackup•  Backupwasnotavailableondisk•  Backuprestoredfromtape•  Foundsometapeshadbeenexpiredbymistake–tookdaystore-scanandre-catalogthepieces•  Tapelibraryhadissues–movedtapestoanotherlibrarythatwasonly1GigEconnecFvity•  Taperestoreswerefailingamermanyhours–init.oraparameterwaswrong

– RTO=8daysandRPO>8hours!

WhatisyourRTOandRPOrequirements?

27


RestoreandRecoveryBestPracFces–Cont.•  UseRMANRestoreDatabase/RecoverDatabaseasyouwouldtoday–  NonewRMANcommandstolearn.Intelligentbuilt-inrecoverycataloginRA.–  RMANisawareofthevalidatedbackupsondisk,tapeorreplica.RestoreoperaFonistransparentandsimple.

–  RestoredirectlyfromtapeorRAReplicawithoutstagingdataonlocalRAiflocaldiskbackupsarenotpresent

•  PerformanceconsideraFons– Maximize#ofRMANchannelsforRestoreunlesstheyareotheracFvedatabasesonthetarget.–  RestoreoperaFonsarealwaysprioriFzedautomaFcallywithinRAwithoutprevenFngotherbackupoperaFons

•  BigfileTablespacePracFcesandConsideraFons(recallbackupbestpracFceusingSECTIONSIZE)–  Oracle11gdatabasescanrestoreiniFalL0withSECTIONSIZEtoparallelizesecFonsacrosschannels•  Restoringvirtualfulls(createdfromL1s)doesnotparallelizesecFons

–  Oracle12cdatabasescanrestoreL0andvirtualfullswithSECTIONSIZEparallelismacrosschannels

28


RecoveryApplianceandDataGuard

•  FollowallMAArecommendaFons– OneRecoveryAppliance(RA)perdatacenter– BackupprimaryandstandbydatabasestothelocalRA– NoRAreplicaFonforanydatabaseswitharemotestandbyneeded– RestoreoperaFoncanuseanyRAinanylocaFon

29


RecoveryApplianceandDataGuard

• PostDataGuardroletransiFon– NochangeinbackupoperaFons.ConFnuetobackupboththeprimaryandstandbydatabasestothelocalRA

• DeployingtheZeroDataLossRecoveryApplianceinaDataGuardConfiguraFon– Refertohkp://www.oracle.com/technetwork/database/availability/recovery-appliance-data-guard-2767512.pdforDeployingZeroDataLossRecoveryApplianceinaDataGuardConfiguraFon

MAAWhitePaper

30



NewandSimplerWorld






1

2

3

4

5

31

6

7


ValidaFon,SecurityandTroubleshooFng

• RTOorRPOSLA’snotmet– BadTapes– CorrupFonsinbackups– Missingpieces(archivelogs,datafilesorcontrolfiles)– NoautomaFonorendtoendunderstandingofrestoreandrecoverprocess

• ProblemAvoidance:– WeeklyRMANcrosschecks,– WeeklyormonthlyRMANbackuporrestorevalidate– MonthlyorQuarterlyendtoendrestoreandrecoveryvalidaFontesFngandautomaFon

Topproblemsfacedinthefield

32



•  IngesFngBackups– Validatedatablocksastheyarereadfromsourcedatabaseandsenttoappliance

•  IndexingBackups– Blocksreceivedarevalidated,compressedastheyarewrikentothedeltastore

• OngoingValidaFon– Allbackupsetsarecrosscheckeddaily– AlldatafileblocksareopFmizedweekly(meaningallblocksarereadweekly)– Allbackupsetsarevalidated(thinkrestorevalidate)bi-weekly– MaybemodifiedbyserngofconfiguraFonparameter

HowRecoveryApplianceaddressestheissues

33



• BuiltonExadata– BenefitsfromExadataASMchecksandauto-repairfrommirroredcopy

•  AllchecksrunontheRecoveryAppliance,offloadingaddiFonalloadontheprotecteddatabases

•  End-to-endrecoveryplansFllneedstobetested– DoesnotremovetheneedforperiodicfullrestoreandrecoverytesFngtoprepareoperaFonsteamandvalidateissuesoutsideRA

• MonitorRAforanyalertsoranydatabasenotmeeFngrecoverywindowSLAsandaddressthemearlyon.

HowRecoveryApplianceaddressestheissues(conQnued)

34



• ClienttoRecoveryAppliance,orRecoveryAppliancetoClient– BestEncrypFonatRest–Database/BackupswithTDE– SecurityatFlight-hkps,sqlnetencrypFon&Wallets/CerFficatesintegraFon(staytunedforupcomingMAApaper)

•  SecurityintheRecoveryAppliance– RecoveryApplianceadministratorsresponsibiliFes•  CreateVirtualPrivateCatalog(VPC)User•  AssignprotecteddatabasestoaspecificVPCUser•  TheprotecteddatabaseadministratorcanseealldatabasesthatshareacommonVPCuser

Customersrequiringendtoendsecurity

35



•  FortroubleshooFngRecoveryApplianceissuesreferto– FornetworkperformancerelatedissuesbetweenprotecteddatabasesandtheRecoveryAppliancereferto– ZDLRADetailedTroubleshooQngMethodology(DocID2066528.1)– RecoveryApplianceNetworkTestThroughputscript(DocID2022086.1)

TroubleshooQngNote

36

37

Zero Data Loss Recovery Appliance: Best Practices from Customer Deployments

Gagan Singh Intel Corporation

38

Agenda •  Legacy Backup Environment Overview •  Recovery Appliance – Implementation & learnings •  Recovery Appliance Value Summary

39

Legacy Backup Environment Overview

•  Backups to tier 1 SAN storage •  Different backup strategies (Incr merged , backupsets) •  Backup retentions and validation managed manually •  I/O throttling and staggered backups •  Multiple vendor driver software and constraints

40

RA – Implementation & Learnings

•  Execute test cases in pre-prod – Get Familiar !! •  Backups, Restores, Recovery(PiT/Full), ZDL, different OS •  Stay current on EM plug-ins •  Check N/W bandwidth from protected database to ZDLRA •  DB backup capacity planning – Rate of change •  Daily Incr + archivelog volume for the database(s) •  Leverage different protection policies à”Recovery Window”

is KEY •  Leverage RMAN RESTORE PREVIEW for initial validations

41

..contd • Monitor àZero Data Loss Recovery Appliance Critical Issues (Doc ID 1927928.1) •  Periodic Platinum Patching – Downtime required •  Enable BCT •  Troubleshooting ? •  Follow the relevant MOS note – ZDLRA detailed troubleshooting

methodology (Doc ID 2066528.1) •  ra_incident_log •  Enable EM Notifications and alerts

42

Recovery Appliance Value Summary Goals Result

Reduce resources on production databases •  Offloaded backups and validation to appliance, no more incremental merge

Uniform Backup Environment •  Single backup strategy and type of backup

Flexible Backup Retention •  Leveraged Protection Policies •  Disk Retention | Recovery Windows

Reduce operational overhead •  Enterprise Manager Monitoring , BI Reports , notifications.

Reliability, Availability, & Performance •  Leverages Exadata-based HW •  Scalable •  Flexible to multiple database OS platforms

Backup - Better RPO and RTO •  Virtual full restore eliminates incremental restore+apply time

Reduce vendor footprint •  Ease of patching •  Single point of escalation


Summary/Q&A


Summary•  It’sallaboutsimplificaFon•  Simplerenvironment=bekerstability

•  FollowthebestpracFces–  It’sanAppliance!–  FollowMOSnotes–  Patchregularly

•  ZeroDataLossRecoveryAppliance:EliminateDataProtecFonUncertainFes[CON7405]DonnaCooksey,SalesEnablementLead,Oracle•  Thursday,Sep22,10:45a.m.-11:30a.m.|ParkCentral-MetropolitanIII

•  AcceleraFngDatabaseBackupandRecoverywithZeroDataLossRecoveryAppliance[CON1324]–JavierRuiz,KevinPrendergast,GeorgeMamvura,EnergyTransfer•  Thursday,Sep22,12:00p.m.-12:45p.m.|ParkCentral-MetropolitanIII


MeetusintheShowcaseareaoutsidetheKeynoteHall!

zero data loss recovery appliance: best pracfces from ... › technetwork › database › ... ·...

Documents