collaborate instant cloning_kyle

Download Collaborate instant cloning_kyle

If you can't read please download the document

Post on 27-Jan-2015




0 download

Embed Size (px)




  • 1. Database VirtualizationandInstant CloningKyle Hailey

2. Database Cloning ChallengeBusiness want data now.Business dont understand DBAs.Databases getting bigger & harder to copy.Developers want more copies.Reporting wants more copies.Everyone has storage constraints.If you cant satisfy the businessdemands your process is broken. 3. Two PartsI. Cloning TechnologyII. Accelerate your business 4. Part I : Cloning Technology3. Virtual2. Thin Provision1. Physical 5. =database1. Physical Cloning 6. ProblemDevelopersQA and UATReportsFirstcopyProduction CERN - European Organization for NuclearResearch 145 TB database 75 TB growth each year Dozens of developers want copies. 7. workaroundsDevelopersQA and UATReportsSharedSub set copyProductionMany copies 8. Physical ClonesDatabase SubsetsShared Databases 9. Subsets 10. The ProductionWallClassic problem is that queries thatrun fast on subsets hit the wall inproduction.Developers are unable to test againstall data 11. Shared Full 12. Shared access = Poor ProductivityDevelopers andtester get frustratedDatabases become oldand unrepresentativeof production.Requires complexscheduling andmanagement 13. Never enough environmentsAverage customer makes 12 copies of production- Charles Garry, Database Product Manager Oracle 14. Physical Copies Time & SpaceconsumingTime consumingTime to make copies, days to weeksRMAN backup, archive logs, copy data over, recoverMeetings , days to weeksSystem, Storage ,Database ,Network Admins, manager coordinationSpace consuming12 devs x 10TB production = 120TB20 report DBs x 40 TB = 800TBSlow Clones mean bottlenecks 15. Setup Develop 16. Setup 17. Setup DevelopQA 18. $40M$75M$850M$27,000MStorageITDevelopBusiness 19. ERP Project Failures 2011 NYC CityTime : delays $63 M => $760 M Montclair Uni: delays sues PeopleSoft Idaho : delays ERP cost millionsStandish : IT Project Failure Rate1994 1996 1998 2000 2002 2004 200931% 40% 28% 23% 15% 18% 24%* 20. Clone 1 Clone 399% of blocks are IdenticalClone 2 21. 2. Thin Provision Cloning 22. Clone 1 Clone 2 Clone 3Thin Provision 23. 2. Thin Provision CloningRequires Cornerstone Technology 24. I. clonedbII. Copy on Write Snapshotsa) EMC BCVb) EMC SRDF or Recover Pointc) VmwareIII. Allocate on Writea) Netapp (EMC VNX)b) ZFSc) DxFS2. Thin Provision Cloning 25. RMANbackupdNFSsparse fileI. clonedb 26. RMANbackupdNFSsparse fileI. clonedb 27. I. CloneDB dNFS cd $ORACLE_HOME/rdbms/lib make -f dnfs_on initSOURCE.ora output.sql MASTER_COPY_DIR="/rman_backup CLONE_FILE_CREATE_DEST="/nfs_mount CLONEDB_NAME="clone" sqlplus / as sysdba @output.sql startup nomount PFILE=initclone.ora Create control file backup location dbms_dnfs.clonedb_renamefile(/backup/file.dbf , /clone/file.dbf); alter database open resetlogs;Tim 28. I. CloneDBRMANphysicalTarget A NFS ServerClone 1Clone 2Read only Clone 3Target BClone 1Clone 2Clone 3 29. I. clonedbII. Copy on Write + snapshotsa) EMC BCVb) EMC SRDF or Recover Pointc) VmwareIII. Allocate on Writea) Netapp (EMC VNX)b) ZFSc) DxFS2. Thin Provision Cloning 30. DActiveFileSystemSnapshotCBAFile System constant - EMCII. Copy on Write a) EMC snapshots 31. DActiveFileSystemSnapshotDCBAChanges written in place - EMCChanges to new area - VMwareII. Copy on Write a) EMC snapshots 32. Goal: backup Create BCV , take snaps of BCV Zone and mask LUN to target host Full copy of disk, snapshot, recover Snapshots 16 snapshots Zero or one snapshots of snapshots Golden Copy EMC uses a save area, area for changes to the snapshot Initial snapshot has to stayII. Copy on Write a) EMC BCV 33. Non-Prod FilerEMC FilerProductionDatabaseDatabaseLunsTarget ATarget BTarget CClone 1Clone 2Clone 3Clone 4SnapshotsMax 16a) EMCBCVIII. Allocate on WriteBCVDay 1snapshotSnapshotor breakDay 2Timefinder needed for snapshotson across multiple LUNs 34. SRDF Symetrix Remote Data Facility Stream Changes to a remote machine Recover Point Capture changes on wire Split changes send to remote siteII. Copy on Write b) EMC SRDF 35. Non-Prod FilerEMC FilerProductionDatabaseDatabaseLunsTarget ATarget BTarget CClone 1Clone 2Clone 3Clone 4File system levelSRDF orRecover PointSnapshotday 1Snapshotday 1Snapshotday 2Snapshotday 2II. Copy on Write b) EMC SRDF 36. DActiveFileSystemSnapshotDCBAChanges written to a new locationChanges to new area - VMwareII. Copy on Write, c) VMware 37. II Copy on Write c) VMwareData Director : Linked Clones Not support for Oracle databases Golden Copy : rebuild after 32 snapshots x86 host databases only Performance issues Having several linked clones can affect the performance of the source database and theperformance of the linked clones. (on ) If you are focused on performance, you should prefer a full clone over a linked clone. Performance worse with more snapshots Performance worse with more concurrent users 38. I. clonedbII. Copy on Writea) EMC BCVb) EMC SRDF or Recover Pointc) VmwareIII. Allocate on Writea) ZFSb) Netapp (EMC VNX)c) DxFS2. Thin Provision Cloning 39. Data Blocksroota.ZFSb.Netapp (EMC VNX)c. DxfsIII. Allocate on Write 40. a) ZFS 1 disk = 1 filesystem ~1990: N disks = 1 FS 2001: ZFS starts 2005: ZFS ships 2008: ZFS storage appliance ships 2010: Delphix heads ZFS open sourceIII. Allocate on Write 41. FS vs. ZFS FS per Volume FS limited bandwidth Storage stranded Many FS in a pool Grow automatically All bandwidthStorage PoolVolumeFSVolumeFSVolumeFS ZFS ZFS ZFS3a) ZFSIII. Allocate on Write 42. a) ZFSSnapshot rootLiverootDelphix Proprietary and ConfidentialZilIntent LogIII. Allocate on Write 43. ZFS Appliance + RMAN1. ZFS Appliance Project db_master Project db_clone 4 file systems: datafile, redo, archive, alerts2. Source Database NFS Mount ZFS Appliance Backup with RMAN3. ZFS Appliance Select db_master Snapshots Then each filesystem on db_master clone it onto db_clone4. Target Host NFS Mount db_clone recover clonecloning-solution-353626.pdfa) ZFSIII. Allocate on Write 44. Oracle ZFS Appliance + RMAN1. physicalZFS Storage ApplianceRMANcopyRMAN CopytoNFS mountTarget AClone 1Clone 1SnapshotNFSa) ZFSIII. Allocate on Writecloning-solution-353626.pdf 45. Oracle ZFS Appliance + DataguardDataguardZFS Storage AppliancedatafilesNFSTarget AClone 1Clone 1SnapshotNFSa) ZFSIII. Allocate on WriteDatafiles viaNFSmaa-db-clone-szfssa-172997.pdf 46. NetApp FilerNetApp FilerProductionDatabaseDatabaseLunssnapshotTarget ATarget BTarget CClone 1Clone 2Clone 3Clone 4File system levelsnapshotclonesSnap mirrorSnapshotManager forOracleFlexcloneb) NetappIII. Allocate on WriteRequires adatabase 47. NetApp FilerNetApp FilerProductionDatabaseDatabaseLunssnapshotTarget ATarget BClone 1Clone 2snapshotsnapshotsnapshotb) NetappIII. Allocate on Write 48. NetApp FilerNetApp FilerPhysicalDatabaseDatabaseLunssnapshotTarget ATarget BClone 1Clone 2snapshotsnapshotsnapshotb) NetappIII. Allocate on Write 49. Compression typically 3x Block sharing DxFS optimized for databases Write optimizations Space allocation and destroy Shared blocks in memoryc) DxFSIII. Allocate on Write 50. I. clonedbII. Copy on Write Snapshotsa) EMC BCVb) EMC SRDF or Recover Pointc) VmwareIII. Allocate on Writea) Netapp (EMC VNX)b) ZFSc) DxFS2. Thin Provision Cloning 51. Clonedb EMC BCV Netapp EMC SRDF EMC VNX2. Review: Thin Provision Cloning 52. 2. Thin Provision Cloning 53. 2. Thin Provision Cloning 54. 3. Database Virtualization Oracle SMU Delphix 55. VirtualizationLayerVirtualizationSMU 56. a) Oracle 12c SMUOracle Snap Management Utility for ZFS Appliance Requires ZFS Appliance Supports Linux , Solaris 10+, Windows 2008+ GUI snapshot source databases provision virtual databases3. Database Virtualization 57. Virtualization Layerx86 hardwareAllocateStorageAny typeSMUChoose your virtualization Layer: Delphix and Oracle SMU automated out of box Netapp and EMC SRDF/Recover Point require massive scripting Delphix and possibly SMU share blocks in memoryZFS Storage Appliance 58. One time backup of source databaseDatabaseProductionInstanceFile systemRMAN APIs 59. Delphix Compress DataDatabaseProductionInstanceFile systemData iscompressedtypically 1/3size 60. Incremental forever change collectionDatabaseProductionInstanceFile systemChanges are collectedautomatically foreverData older than retentionwidow freed 61. Typical ArchitectureDatabaseFile systemProductionInstanceDatabaseFile systemDevelopmentInstanceDatabaseFile systemQAInstanceDatabaseUATInstanceFile system 62. Clones share duplicate blocksDevelopmentInstanceDatabaseProductionInstanceFile systemvDatabaseQAInstanceUATInstancevDatabase vDatabaseNFSSource Database Clone Copies of Source DatabaseFiber Channel 63. Part II: Virtualization Case Studies1. Development2. QA3. Recovery 64. 1. Development Acceleration 65. DatabaseVirtualizationlayer3 clones of same source database 3 virtual clones of same source databaseVirtualization layer orchestrates I/Oaccess between database and storage 66. Source TimeflowSource TimeflowSource DatabaseTarget HostVirtual DatabaseNFSFiberFiberRMAN overTCP 67. SourceFreshVDB 68. SourceVDBVDBVDBFrequent 69. Fresh dataFull clonesFrequent (many)fast 70. Self Service 71. Devv2.6 v2.6v2.6QA UATv2.6Productionv2.6 v2.6v2.6v2.7v2.6 v2.6v2.6v2.8Source Control for the database datav2.6v2.6v2.6v2.6v2.6v2.7v2.6v2.7v2.6v2.8v2.6v2.8 72. DevProd2.6 73. DevQAProd2.6 74. DevQAUATProd2.6 75. DevQAUATProdDevQAUAT2.62.7 76. DevQAUATProdDevQAUAT2.62.7DevQAUAT2.8 77. DevQAUATProdDevQAUAT2.62.7DevQAUAT2.8Data Control = Source Contr


View more >