kscope 2013 delphix

Database Virtualization and

Instant CloningKyle Hailey

http://dboptimizer.com

Database Cloning Challenge

Business want data now.

Business don’t understand DBAs.

Databases getting bigger & harder to copy.

Developers want more copies.

Reporting wants more copies.

Everyone has storage constraints.

If you can’t satisfy the business demands your process is broken.

Two Parts

I. Cloning TechnologyII. Accelerate your business

Part I : Cloning Technology

3. Virtual2. Thin Provision1. Physical

=

database

1. Physical Cloning

Problem

Developers

QA and UAT

Reports

First copy

Production

• CERN -  European Organization for Nuclear Research

• 145 TB database• 75 TB growth each year• Dozens of developers want copies.

workaroundsDevelopers

QA and UAT

ReportsShared

Sub set copy

Production

Many copies

Physical Clones

Database SubsetsShared Databases

Subsets

ProductionThe Production ‘Wall’

Classic problem is that queries that run fast on subsets hit the wall in production.

Developers are unable to test against all data

Shared Full

Shared access = Poor Productivity

Developers and tester get frustrated

Databases become old and unrepresentative of production.

Requires complex scheduling and management

Never enough environments

Average customer makes 12 copies of production- Charles Garry, Database Product Manager Oracle

Physical CopiesTime consuming

Time to make copies, days to weeksRMAN backup, archive logs, copy data over, recover

Meetings , days to weeksAdmins: System, Storage ,Database ,Network, manager coordination

Space consuming 40 devs x 2.5TB production = 100TB20 report DBs x 40 TB = 800TB

=> bottlenecks

Setup Develop

Setup

Setup Develop

QA

$40M

$75M

$850M

$27,000M

Storage

IT

Develop

Business

ERP Project Failures 2011

• NYC CityTime : delays $63 M => $760 M • Montclair Uni: delays sues PeopleSoft• Idaho : delays ERP cost millions

Standish : IT Project Failure Rate

1994 1996 1998 2000 2002 2004 2009

31% 40% 28% 23% 15% 18% 24%

★http://www.galorath.com/wp/software-project-failure-costs-billions-better-estimation-planning-can-help.php*http://www.pcworld.com/article/246647/10_biggest_erp_software_failures_of_2011.html

Clone 1 Clone 3

99% of blocks are Identical

Clone 2

2. Thin Provision Cloning

Clone 1 Clone 2 Clone 3

Thin Provision

2. Thin Provision Cloning

Cornerstone Technology: File System Snapshots

Thin Cloning

• Snapshot DB Files @ point in time• Copy off production• Snapshot again• Export over NFS or FC to Host• Recovery Database

Netapp tr-3761.pdf

Netapp

Snap Manager

SnapManagerRepository

Protection Manager

Snap Drive

Snap Manager

Snap Mirror

Flex Clone

RMANRepository

Production

Development

DBA

Storage Admin

NetApp Filer - DevelopmentNetApp Filer - Production

Production Database

Database Luns

Target A

Target B

Target C

Clone 1

Clone 2

Clone 3

Clone 4

Snap mirror

Snapshot Manager for Oracle

Flexclone

Repository Database

Netapp

SnapDrive

Protection Manage

2. Thin Provision Cloning

2. Thin Provision Cloning

3. Database Virtualization

Virtualization Layer

38

Virtualization

DatabaseVirtualizationAppliance(DVA)

3 Clones Physical 3 Clones Virtual

Install Delphix on x86 hardware

x86 hardware

Allocate Storage to Delphix

Allocate StorageAny type

One time backup of source database

Database

Production

Instance

File system

RMAN APIs

Delphix Compress Data

Database

Production

Instance

File system

Data is compressed typically 1/3 size

Incremental forever change collection

Database

Production

Instance

File system

Changes are collected automatically foreverData older than retention widow freed

Typical Architecture

Database

File system

Production

Instance

Database

File system

Development

Instance

Database

File system

QA

Instance

Database

UAT

Instance

File system

Clones share duplicate blocks

Development

Instance

Database

Production

Instance

File system

vDatabase

QA

Instance

UAT

Instance

vDatabase vDatabase

Source Database Clone Copies of Source Database

Use Cases

1. Development2. Recovery3. Reporting

1. Development Acceleration

1: Development Acceleration

a) Developer each get a copy– Fast, fresh, full, frequent– Self service

b) Branchingc) Federated

Source

FastSource Database

Target HostVirtual

Database

NFS

Fiber

Fiber

RMAN over TCP

No Data Movement

Source

Fresh

Virtual Database

Fiber

Source

Frequent

Virtual Database

Virtual Database

Target Hosts

Virtual Database

Virtual Database

Fiber

Full clones

Self Service

1 b) Branching and Rapid QA

dSource

1 b) Branching

Developer VDB

QA VDB

Devv2.6 v2.6v2.6

QA UAT

v2.6

v2.6 v2.6v2.6v2.7

v2.6 v2.6v2.6v2.8

v2.6v2.6 v2.6v2.6

v2.6v2.7 v2.6v2.7

v2.6v2.8 v2.6v2.8

Devv2.6 v2.6v2.6

QA UAT

v2.6Production

v2.6 v2.6v2.6v2.7

v2.6 v2.6v2.6v2.8

Source Control for the database data

v2.6v2.6 v2.6v2.6

v2.6v2.7 v2.6v2.7

v2.6v2.8 v2.6v2.8

DevProd

2.6

Dev

QA

Prod

2.6

Dev

QA

UAT

Prod

2.6

Dev

QA

UAT

Prod

Dev

QA

UAT

2.6

2.7

Dev

QA

UAT

Prod

Dev

QA

UAT

2.6

2.7

Dev

QA

UAT2.8

Dev

QA

UAT

Prod

Dev

QA

UAT

2.6

2.7

Dev

QA

UAT2.8

Data Control = Source Control for the Database

Dev

QA

UAT

Dev

QA

UAT

2.6

2.7

Dev

QA

UAT

2.8

Data Control = Source Control for the Database

Production Time Flow

1 c) Federated Cloning

Source2

Source3

Source1

1 c) Federated sources

Virtual Database

Virtual Database

Virtual Database

Virtual Database

“I looked like a hero”Tony Young, CIO Informatica

1. Review Development Use Cases

a) Developer each get a copy

b) QA

c) Federated

2. Recovery, Testing, Forensics

a) Forensicsb) A/B testingc) Recovery

Source

2 a) Forensic Analysis

Virtual Database

Source

2 b) Upgrades, Patches, RAT, A/B

Virtual Database

• Production vs Virtual– invisible index on Prod– Creating index on virtual

• Flashback vs Virtual• Keep tests for compare

2 b) Upgrades, Patches, RAT, A/B

2 c) Recovery

Source

2 c) Logical Recovery Production

Virtual Database

Source

2 c) Logical Recovery Development

Virtual Database

VDB rolled back

Source

Recovery

VDB

V2P

2. Recovery, Testing, Forensics

a) Forensics

b) A/B testing

c) Recovery : Logical and physical

3: reportinga) Fast refreshes

b) Temporal queries

c) Confidence testing

Fast Refreshes

• Refresh in minutes• Without data movement• Faster , cheaper

Temporal Data

3: reportinga) Fast refreshes

b) Temporal queries

c) Confidence testing

Review: Use Cases

1. Developmenta) Full, Fresh, Fast , Self Serveb) Branchingc) Federated

2. Recovery, Testing :a) Forensicsb) Testing : A/B, upgrade, patchc) Recovery: logical, physical

3. Reportinga) Fast refreshb) Temporal Datac) Confidence testing

over 10 times

"perhaps the single largest storage consolidation opportunity history“

Oracle 12c

80MB buffer cache ?

200GBCache

5000

Tnxs

/ m

inLa

tenc

y

300 ms

1 5 10 20 30 60 100 200

with

1 5 10 20 30 60 100 200Users

8000

Tnxs

/ m

inLa

tenc

y

600 ms

1 5 10 20 30 60 100 200Users

1 5 10 20 30 60 100 200

Database Virtualization