Efficient, hot & automatic oracle database cloning

Josep Vidal Canet

Universtitat de València

“Discover how to clone your production database without disruption, in a totally automated & efficient way”

Motivation

• Useful for the DBA or system administrator who wants to give his developers a full-sized TEST and DEV instance by cloning the PROD instance into the development server system

• To train new, inexpert DBAs• To test backup and recovery strategies, workloads,

software upgrades, migrations, etc …

Universitat de Valencia

• Spain’s third-largest university– 45.000 students– 3.500 teachers– 2.000 workers

• Different RDBMS:– DB2 / ZOS mainly OLTP– The biggest amount of data is stored in Oracle /

Unix / Linux• Both OLTP & OLAP

– Postgres

What’s a database clone?

• A database clone is a complete and separate copy of a database system that includes the business data, applications and the DBMS software (wikipedia)

• The cloned DB is both fully functional and separate in its own right

DB cloning strategies

• Physical copy -> same server architecture & oracle version

• Export / import : different architectures or oracle versions

The Big Picture I: GOAL

Prod Instance

SGA

PMON SMON DBWR

LGWR CKPT Others

Prod Server or LPAR(Source)

DatabaseData Files Control

Files

Redo Log Files

Cloned Instance

SGA

PMON SMON DBWR

LGWR CKPT Others

Test Server or LPAR(Target)

DatabaseData Files Control

Files

Redo Log Files

To produce a physical copy of prod DB

AutomaticallyWithout disruptionEfficient ReliableSecure

pclone.sh

• A script that automates the whole process:– Without disrupting production systems

• Remote Hotbackup• If the OS allows it (AIX, HP-UX, Solaris) WLM controls are imposed

to guarantee QoS – Efficient– Easy to schedule as a crontab job – Only differences (modified blocks) from prod to test database

are copied– Parallel programming techniques are applied to reduce the

amount of time required• The whole process is triggered by target system, which

queries source database catalog and launches a remote hotbackup

• Once completed, it is recovered in target system

The Big Picture II: Procedure

Prod Instance

SGA

PMON SMON DBWR

LGWR CKPT Others

Prod Server or LPAR

DatabaseData Files Control

Files

Redo Log Files

Cloned Instance

SGA

PMON SMON DBWR

LGWR CKPT Others

Test Server or LPAR

DatabaseData Files Control

Files

Redo Log Files

1: Issue a remoteHotbackup

2: Automatically recover it on target system

pclone.sh

• No special hardware or software required• Data is moved using a SSL encrypted

connection• Used at UV to clone/update test instances from

production systems in a wide range of platforms (Linux/x86, Solaris/SPARC, AIX/Power).

• Easy to use: pclone.sh DB user/password• The tool can be downloaded from:

– http://www.uv.es/vijo/pclone.sh• Paper:

– http://www.uv.es/vijo/cloning_oracle_database.pdf

Requirements

• Same ORACLE & OS versions & system architecture

• ssh & rsync tools, configured so you can login without password

• Time synchronization in both servers – NTP (Network Time protocol).

• Archive log mode

• Prod instance reachable from test system with tnsping utility

Main steps

• Automated using bash scripting

• Executed in the target system:1. Queries source database catalog to obtain

physical database objects

2. Copies each database object from source to target system ensuring database consistency

3. Starts the source database in the target system (DB recovery)

Step 1: Database physical layout

• Source DB discovery– Find server & $ORACLE_HOME with

tnsping & /etc/oratab– Test source DB status

• Query target DB to determine physical layout– tablespaces, datafiles, redo logs, archive

logs, init.ora, etc …

Step 1: Database physical layout

ArchivedLog Files

ParameterFile

PasswordFile

Oracle Database

Data Files Control Files Redo Log Files

/data01/oracle/oradata/DB/control01.dbf

• The underlying directory structure must be created on target system, before copying physical objects– mkdir -p `dirname $f`

/data03/oracle/oradata/DB/tbs_03.dbf

/OraHome1/dbs/orapw

/OraHome1/dbs/initDB.ora

Step 2: Copying physical database objects between systems

• Goal: To copy each type of database object from source to target ensuring database consistency

• Data transfer utility: rsync– Capable of copying

files between remote computers by transferring only file differences

rsync

• Utility: To copy files between two systems• Performs a block level comparison of two files• Transfers only modified blocks

– huge benefit if you are transferring large files like dafafiles over a network link

• Speeds up file transfers when the destination file already exists

• rsync remote-update protocol– allows rsync to transfer just the differences between

two sets of files across the network link, using an efficient checksum search

Step 2: Copying physical database objects between systems

• Idea: To launch a remote hotbackup from the target system– Physical database objects are copied from the

source to target ensuring consistency– File paths are kept identical in both systems

rsync -taupog source:/$dir/tbs01.dbf target:/$dir/tbs01.dbf

Oracle database main files

ArchivedLog Files

ParameterFile

PasswordFile

Oracle Database

Data Files Control Files Redo Log Files

• Some files (dump, archives, init.ora) can be copied without taking care of database consistency• Others, like datafiles, must be in a consistent state before being copied

Step 2: Moving physical database objects between systems

remote_backup(){

STATUS=`target_db_status`

if [ "$STATUS" == "OK" ]; thenshutdown_db "IMMEDIATE"

fi

sync_dump_dirssync_initorasync_temporary_datafilessync_db_datafilessync_db_ctrl_and_log_files

}

Moving datafiles

Prod Instance

SGA

PMON SMON DBWR

LGWR CKPT Others

Prod Server or LPAR(Source)

Database

tbs01

tbs02

tbs03

Data Files Control

Files

Redo Log Files

alter TBS begin backup

rsync -taupog source:/dir/tbs01.dbftarget:/dir/tbs01.dbf

Alter TBS end backup

Test Server or LPAR(Target)

DatabaseData Files

sqlplus client

shell

rsync -taupog source:/dir/tbs03.dbftarget:/dir/tbs03.dbf

rsync -taupog source:/dir/tbs02.dbftarget:/dir/tbs02.dbf

Moving datafiles

• The algorithm can be summarized as:– for each Tablespace in the source database do:

• Set tablespace in backup mode • Copy (using rsync) each datafile from source to target system using parallel techniques• End tablespace backup mode

• A similar approach is used to move both control and redo log files

Step 2: Copying datafiles

sync_db_datafiles(){TBS=`get_tablespaces_name ${SOURCE_DB_CONNECT_STRING}`switch_db_logfile_ini # Parallelism related variables & locks initializationfor T in $TBS; do DATAFILES=`get_tbs_datafiles ${SOURCE_DB_CONNECT_STRING} ${T}` begin_tbs_backup ${SOURCE_DB_CONNECT_STRING} ${T} for d in $DATAFILES ; do

_maximum_parallelism_barriermkdir -p `dirname $d`(rsync -e 'ssh -c blowfish' -tapogL$HOST:$d $d; _sub;exit;)&

done

_wait_for_all_children_to_finish_barrierend_tbs_backup ${SOURCE_DB_CONNECT_STRING} ${T}

done}

Parallelism• Cloning large databases can take a long time

• Why not just split the amount of objects needed to be copied among different tasks? – For each tablespace:

• Create a process for each datafile to be copied until a maximum level of parallelism is reached (_maximum_parallelism_barrier)

• Once the maximum parallelism level is reached, a new process is created when a running one ends

• The backup state is released for a tablespace when every task is completed (_wait_for_all_children_to_finish_barrier )

• Do the same for the remaining database files (archived logs, dumps, etc …)

Step 3: Recovering database

• Goal: To restore database integrity in target system• How: by recovering the physical copy of the

source database • Source database catalog is queried to determine which

datafiles need to be recovered• For each datafile a ‘media recovery’ is issued against the

physical copy • Main steps:

– Database is mounted on target system– Media recovery is issued for each datafile– Database is opened

Step 3: Recovering database

recover_db(){export ORACLE_SID=$SOURCE_DBstartup_db "mount"TBS=`get_tablespaces_name ${SOURCE_DB_CONNECT_STRING}`for T in $TBS; do

DATAFILES=`get_tbs_datafiles ${SOURCE_DB_CONNECT_STRING}

${T}`for d in $DATAFILES ; do

recover_datafile $d "AUTOMATIC"done

doneopen_db

}

Conclusions

• Database cloning can be used for different purposes like testing, developing, database maintenance tests or DBA training

• pclone.sh automates the whole process without disrupting production systems in an efficient manner

• The whole process, major steps and the necessary code to implement it, has been discussed

Conclusions II

• Performance– The time needed to complete the whole process

depends on:• database size & available computational resources (CPUs &

disks)• database update rate as well as cloning interval

– Example:• 150 GB accounting production database, running in a p570

server uncapped LPAR (6 power5 CPU’s)• After an initial cloning, no additional synchronization was

performed for 6 months• Afterwards, the test database was synchronized from the

prod one• The whole process took less than an hour