Collaborate instant cloning_kyle

Download Collaborate instant cloning_kyle

Post on 27-Jan-2015




0 download

Embed Size (px)




<ul><li> 1. Database VirtualizationandInstant CloningKyle Hailey</li></ul><p> 2. Database Cloning ChallengeBusiness want data now.Business dont understand DBAs.Databases getting bigger &amp; harder to copy.Developers want more copies.Reporting wants more copies.Everyone has storage constraints.If you cant satisfy the businessdemands your process is broken. 3. Two PartsI. Cloning TechnologyII. Accelerate your business 4. Part I : Cloning Technology3. Virtual2. Thin Provision1. Physical 5. =database1. Physical Cloning 6. ProblemDevelopersQA and UATReportsFirstcopyProduction CERN - European Organization for NuclearResearch 145 TB database 75 TB growth each year Dozens of developers want copies. 7. workaroundsDevelopersQA and UATReportsSharedSub set copyProductionMany copies 8. Physical ClonesDatabase SubsetsShared Databases 9. Subsets 10. The ProductionWallClassic problem is that queries thatrun fast on subsets hit the wall inproduction.Developers are unable to test againstall data 11. Shared Full 12. Shared access = Poor ProductivityDevelopers andtester get frustratedDatabases become oldand unrepresentativeof production.Requires complexscheduling andmanagement 13. Never enough environmentsAverage customer makes 12 copies of production- Charles Garry, Database Product Manager Oracle 14. Physical Copies Time &amp; SpaceconsumingTime consumingTime to make copies, days to weeksRMAN backup, archive logs, copy data over, recoverMeetings , days to weeksSystem, Storage ,Database ,Network Admins, manager coordinationSpace consuming12 devs x 10TB production = 120TB20 report DBs x 40 TB = 800TBSlow Clones mean bottlenecks 15. Setup Develop 16. Setup 17. Setup DevelopQA 18. $40M$75M$850M$27,000MStorageITDevelopBusiness 19. ERP Project Failures 2011 NYC CityTime : delays $63 M =&gt; $760 M Montclair Uni: delays sues PeopleSoft Idaho : delays ERP cost millionsStandish : IT Project Failure Rate1994 1996 1998 2000 2002 2004 200931% 40% 28% 23% 15% 18% 24%* 20. Clone 1 Clone 399% of blocks are IdenticalClone 2 21. 2. Thin Provision Cloning 22. Clone 1 Clone 2 Clone 3Thin Provision 23. 2. Thin Provision CloningRequires Cornerstone Technology 24. I. clonedbII. Copy on Write Snapshotsa) EMC BCVb) EMC SRDF or Recover Pointc) VmwareIII. Allocate on Writea) Netapp (EMC VNX)b) ZFSc) DxFS2. Thin Provision Cloning 25. RMANbackupdNFSsparse fileI. clonedb 26. RMANbackupdNFSsparse fileI. clonedb 27. I. CloneDB dNFS cd $ORACLE_HOME/rdbms/lib make -f dnfs_on initSOURCE.ora output.sql MASTER_COPY_DIR="/rman_backup CLONE_FILE_CREATE_DEST="/nfs_mount CLONEDB_NAME="clone" sqlplus / as sysdba @output.sql startup nomount PFILE=initclone.ora Create control file backup location dbms_dnfs.clonedb_renamefile(/backup/file.dbf , /clone/file.dbf); alter database open resetlogs;Tim 28. I. CloneDBRMANphysicalTarget A NFS ServerClone 1Clone 2Read only Clone 3Target BClone 1Clone 2Clone 3 29. I. clonedbII. Copy on Write + snapshotsa) EMC BCVb) EMC SRDF or Recover Pointc) VmwareIII. Allocate on Writea) Netapp (EMC VNX)b) ZFSc) DxFS2. Thin Provision Cloning 30. DActiveFileSystemSnapshotCBAFile System constant - EMCII. Copy on Write a) EMC snapshots 31. DActiveFileSystemSnapshotDCBAChanges written in place - EMCChanges to new area - VMwareII. Copy on Write a) EMC snapshots 32. Goal: backup Create BCV , take snaps of BCV Zone and mask LUN to target host Full copy of disk, snapshot, recover Snapshots 16 snapshots Zero or one snapshots of snapshots Golden Copy EMC uses a save area, area for changes to the snapshot Initial snapshot has to stayII. Copy on Write a) EMC BCV 33. Non-Prod FilerEMC FilerProductionDatabaseDatabaseLunsTarget ATarget BTarget CClone 1Clone 2Clone 3Clone 4SnapshotsMax 16a) EMCBCVIII. Allocate on WriteBCVDay 1snapshotSnapshotor breakDay 2Timefinder needed for snapshotson across multiple LUNs 34. SRDF Symetrix Remote Data Facility Stream Changes to a remote machine Recover Point Capture changes on wire Split changes send to remote siteII. Copy on Write b) EMC SRDF 35. Non-Prod FilerEMC FilerProductionDatabaseDatabaseLunsTarget ATarget BTarget CClone 1Clone 2Clone 3Clone 4File system levelSRDF orRecover PointSnapshotday 1Snapshotday 1Snapshotday 2Snapshotday 2II. Copy on Write b) EMC SRDF 36. DActiveFileSystemSnapshotDCBAChanges written to a new locationChanges to new area - VMwareII. Copy on Write, c) VMware 37. II Copy on Write c) VMwareData Director : Linked Clones Not support for Oracle databases Golden Copy : rebuild after 32 snapshots x86 host databases only Performance issues Having several linked clones can affect the performance of the source database and theperformance of the linked clones. (on ) If you are focused on performance, you should prefer a full clone over a linked clone. Performance worse with more snapshots Performance worse with more concurrent users 38. I. clonedbII. Copy on Writea) EMC BCVb) EMC SRDF or Recover Pointc) VmwareIII. Allocate on Writea) ZFSb) Netapp (EMC VNX)c) DxFS2. Thin Provision Cloning 39. Data Blocksroota.ZFSb.Netapp (EMC VNX)c. DxfsIII. Allocate on Write 40. a) ZFS 1 disk = 1 filesystem ~1990: N disks = 1 FS 2001: ZFS starts 2005: ZFS ships 2008: ZFS storage appliance ships 2010: Delphix heads ZFS open sourceIII. Allocate on Write 41. FS vs. ZFS FS per Volume FS limited bandwidth Storage stranded Many FS in a pool Grow automatically All bandwidthStorage PoolVolumeFSVolumeFSVolumeFS ZFS ZFS ZFS3a) ZFSIII. Allocate on Write 42. a) ZFSSnapshot rootLiverootDelphix Proprietary and ConfidentialZilIntent LogIII. Allocate on Write 43. ZFS Appliance + RMAN1. ZFS Appliance Project db_master Project db_clone 4 file systems: datafile, redo, archive, alerts2. Source Database NFS Mount ZFS Appliance Backup with RMAN3. ZFS Appliance Select db_master Snapshots Then each filesystem on db_master clone it onto db_clone4. Target Host NFS Mount db_clone recover clonecloning-solution-353626.pdfa) ZFSIII. Allocate on Write 44. Oracle ZFS Appliance + RMAN1. physicalZFS Storage ApplianceRMANcopyRMAN CopytoNFS mountTarget AClone 1Clone 1SnapshotNFSa) ZFSIII. Allocate on Writecloning-solution-353626.pdf 45. Oracle ZFS Appliance + DataguardDataguardZFS Storage AppliancedatafilesNFSTarget AClone 1Clone 1SnapshotNFSa) ZFSIII. Allocate on WriteDatafiles viaNFSmaa-db-clone-szfssa-172997.pdf 46. NetApp FilerNetApp FilerProductionDatabaseDatabaseLunssnapshotTarget ATarget BTarget CClone 1Clone 2Clone 3Clone 4File system levelsnapshotclonesSnap mirrorSnapshotManager forOracleFlexcloneb) NetappIII. Allocate on WriteRequires adatabase 47. NetApp FilerNetApp FilerProductionDatabaseDatabaseLunssnapshotTarget ATarget BClone 1Clone 2snapshotsnapshotsnapshotb) NetappIII. Allocate on Write 48. NetApp FilerNetApp FilerPhysicalDatabaseDatabaseLunssnapshotTarget ATarget BClone 1Clone 2snapshotsnapshotsnapshotb) NetappIII. Allocate on Write 49. Compression typically 3x Block sharing DxFS optimized for databases Write optimizations Space allocation and destroy Shared blocks in memoryc) DxFSIII. Allocate on Write 50. I. clonedbII. Copy on Write Snapshotsa) EMC BCVb) EMC SRDF or Recover Pointc) VmwareIII. Allocate on Writea) Netapp (EMC VNX)b) ZFSc) DxFS2. Thin Provision Cloning 51. Clonedb EMC BCV Netapp EMC SRDF EMC VNX2. Review: Thin Provision Cloning 52. 2. Thin Provision Cloning 53. 2. Thin Provision Cloning 54. 3. Database Virtualization Oracle SMU Delphix 55. VirtualizationLayerVirtualizationSMU 56. a) Oracle 12c SMUOracle Snap Management Utility for ZFS Appliance Requires ZFS Appliance Supports Linux , Solaris 10+, Windows 2008+ GUI snapshot source databases provision virtual databases3. Database Virtualization 57. Virtualization Layerx86 hardwareAllocateStorageAny typeSMUChoose your virtualization Layer: Delphix and Oracle SMU automated out of box Netapp and EMC SRDF/Recover Point require massive scripting Delphix and possibly SMU share blocks in memoryZFS Storage Appliance 58. One time backup of source databaseDatabaseProductionInstanceFile systemRMAN APIs 59. Delphix Compress DataDatabaseProductionInstanceFile systemData iscompressedtypically 1/3size 60. Incremental forever change collectionDatabaseProductionInstanceFile systemChanges are collectedautomatically foreverData older than retentionwidow freed 61. Typical ArchitectureDatabaseFile systemProductionInstanceDatabaseFile systemDevelopmentInstanceDatabaseFile systemQAInstanceDatabaseUATInstanceFile system 62. Clones share duplicate blocksDevelopmentInstanceDatabaseProductionInstanceFile systemvDatabaseQAInstanceUATInstancevDatabase vDatabaseNFSSource Database Clone Copies of Source DatabaseFiber Channel 63. Part II: Virtualization Case Studies1. Development2. QA3. Recovery 64. 1. Development Acceleration 65. DatabaseVirtualizationlayer3 clones of same source database 3 virtual clones of same source databaseVirtualization layer orchestrates I/Oaccess between database and storage 66. Source TimeflowSource TimeflowSource DatabaseTarget HostVirtual DatabaseNFSFiberFiberRMAN overTCP 67. SourceFreshVDB 68. SourceVDBVDBVDBFrequent 69. Fresh dataFull clonesFrequent (many)fast 70. Self Service 71. Devv2.6 v2.6v2.6QA UATv2.6Productionv2.6 v2.6v2.6v2.7v2.6 v2.6v2.6v2.8Source Control for the database datav2.6v2.6v2.6v2.6v2.6v2.7v2.6v2.7v2.6v2.8v2.6v2.8 72. DevProd2.6 73. DevQAProd2.6 74. DevQAUATProd2.6 75. DevQAUATProdDevQAUAT2.62.7 76. DevQAUATProdDevQAUAT2.62.7DevQAUAT2.8 77. DevQAUATProdDevQAUAT2.62.7DevQAUAT2.8Data Control = Source Control for the Database 78. Collaborative Provisioning 79. VDB2Source2Source3Source1VDB3VDB1Collaborative sources 80. I looked like a heroTony Young, CIO Informatica 81. 1: Review of Development Provision , Self Service Fast Full Fresh Frequent (Many) Source control for the database Federated cloning 82. 2. Rapid QA 83. dSourcePass to QADeveloper VDBQA VDB 84. SourcePunt to DevDeveloper VDBQA VDBDeveloper VDB 85. dSourceForensic AnalysisVDB 86. dSourceUpgrades, Patches, RATVDB 87. A/B Testing Production vs Virtual invisible index on Prod Creating index on virtual Flashback vs Virtual Confidence testing change reporting SQL compare new code over last 30 days 88. 2: review QA Fork copies off of Dev to QA QA fork copies back to Dev Instant replay performance A/B Upgrade, patching Forensic analysis of Prod 89. 3. Backup and Recovery 90. MONSUN WEDTUE THU SATFRI MONSUN TUEMONSUN WEDTUE THU SATFRI MONSUN TUEDataProcessingRequiredCurrent Volumeof Work 91. SourceRollback / Time machineVDB 92. SourceRollback / Time machineVDBVDB Rolled back 93. SourceRecoveryVDBV2P 94. 3: review Backup Backup 50 days in size of 1 copy logical recovery on prod, dev Restore virtual to production 95. Review Part II: Data Virtualization1. Development: Full, Fresh, Fast, Data SCCS = SDCS2. QA : forking, replay, forensics3. Recovery: 50:1, time machine, V2PReview Part I: Technology1. Full Clones2. Thin Provision : dbclone, EMC, Netapp3. Database Virtualization : SMU, Delphix 96. over 10 times"perhaps the single largest storage consolidationopportunity history 97. Oracle 12c 98. 80MB buffer cache ? 99. 5000Tnxs/minLatency300ms1 5 10 20 30 60 100 200with1 5 10 20 30 60 100 200Users 100. 200GBCache 101. 5000Tnxs/minLatency300ms1 5 10 20 30 60 100 200Users1 5 10 20 30 60 100 200 102. 200GBCache 103. 8000Tnxs/minLatency600ms1 5 10 20 30 60 100 200Users1 5 10 20 30 60 100 200 104. Database Virtualization 105. Matrix of featuresCloneDB ZFSApplianceDelphix DataDirectorNetApp EMCTime Flow No Yes Yes No Yes NoHardwareAgnosticYes No Yes Yes No NoSnapshots No Unlimited Unlimited 31 255 16(96 read only)Snapshots ofsnapshotsNo Unlimited Unlimited 30 255 1AutomatedSnapshotsNo No Yes No Yes NoAutomatedProvisioningNo No Yes No No NoAny O/S Yes Yes Yes Nox86 onlyYes YesMax size None None None ? 16-100TB? 106. Conclusion : Enterprise Solutions VMware Data Director EMC Timefinder offer limited ability to benefit from cloning Clonedb *** fast easy way to create many clones of the same copy limited to and systems with sparse file system capability suffers the golden image problem and performance NetApp Flexclone, Snap Manager for Oracle offers a rolling solution limited database awareness file system clones limited snapshots Vendor lock-in Oracle ZFS Appliance Vendor Lock-in Delphix Agility : Automation, unlimited snapshots, clones, multi-database 107. Appendix CloneDB ZFS ZFS Appliance ZFS setup for Oracle using RMAN using dataguard Data Director linked clone not support for Oracle EMC BCV and SRDF NetApp OEM for Netapp; Oracle SMU Oracle OEM Snapshot Manager for Netapp; Demo starts at 3:10 goes to 19:43 108. Other Cant run multiple clones on same host withASM EMC VNX 109. NetApp LimitsController Size Limit32 bit controllers 16TBFAS3140/FAS3040/FAS3050 40TBFAS3160/FAS3070 50TBFAS6040/FAS3170 70TBFAS6080 100TBAll sources have to be in the same aggregate to besnapshot together.Limit of 255 snapshotssnaps are limited to the same aggregate (storage pool)Aggregates have size limits depending on controller 110. b) Ne...</p>