openBench Labs

Executive Briefing:

Optimizing Hypervisor-Independent VM Data Protection

Comparison of Host-Level vs. VM-Level Backupwith AppAssure Smart Agents

January 19, 2012

ROADMAP TO THE VIRTUAL ENTERPRISE

For IT managers, data centers are becoming more difficult to manage and protect asmore data and applications are moved into virtual environments. Adding fuel to the fire,CIOs must now deal with corporate mandates to build an IT infrastructure that scales tounknown demand levels and provides service assurance for fluctuating conditions thatcannot be accurately projected. The solution requires a transition to private cloudcomputing characterized by a hypervisor-independent Virtual Infrastructure (VI).Unfortunately, this solution also exacerbates a classic data protection problem.

The problem stems from thereliance software vendors put onclient infrastructure to optimizebackup performance. This is thereal issue behind the “Agent vs.Agentless” marketecture debate—not the hyperbole over licensesand management. In backup andrecovery tests, openBench Labsfound backup server loadconsistently trumped host VirtualMachine (VM) load as the criticalscalability factor. What’smore,host-level agentless VMbackups introduced significantvariability in performance between

VMs configured identically on vSphere and Hyper-V hypervisors.

We measured no difference in VM overhead between AppAssure VM-level SmartAgents and host-level agentless software, which needs to invoke processes on VMs tocollect application item data and implement VSS. Furthermore, by optimizing datatransfers between clients and backup servers, Smart Agents ran incremental backups 9Xfaster on vSphere and 15X faster on Hyper-V. More importantly, Smart Agentsprovided zero-perceived recovery time on both hypervisors without remapping data,which adds I/O overhead until IT runs a consolidation process to finalize the recovery.

In a vSphere VI environment, leveraging the hypervisor equates to leveraging vSphereAPIs for Data Protection, which provide Changed Block Tracking (CBT) and direct SAN

02

ExecutiveReport:

Jack FegreusJanuary 25, 2012

Optimizing Hypervisor-Independent VM Data Protection

Comparison of Host-Level vs. VM-Level Backupwith AppAssure Smart Agents

AppAssure Smart Agent Key Findings1) AppAssure Smart Agents were 3X faster than agentless running 3simultaneous full backups of vSphere VMs without a SAN.

2) AppAssure Smart Agents were 34% faster than agentless running 3simultaneous full backups of vSphere VMs with a SAN.

3) AppAssure Smart Agents were 40% faster than agentless backupsolutions running 3 simultaneous full backups of Hyper-V VMs.

4) AppAssure Smart Agents were 9X faster running incremental backupson vSphere, 15X faster on Hyper-V, and provided zero-perceivedrecovery time on both hypervisors.

5) Legacy agents working on a file-level basis generated incrementalbackup filess 100 times larger than block-level backup files.

6) AppAssure Smart Agents restore data instantly (Zero RTO) withoutremapping data, which must be consolidated to complete the recovery.

access to ESXi datastores. AppAssure, however, takes a radically different approach.AppAssure uses a Smart Agent and filter driver between the client’s file system and OSkernel to capture changes to disk blocks-on any virtual or physical client.

What makes CBT so important is the ability to create incremental backup files thatcontain just the disk blocks that have changed since the previous backup and notcomplete copies of files that contain changed blocks. Not only does this speed theprocessing on incremental backups, it means incremental backup files are nearly free ofduplicated data. By limiting data transfered in an incremental backup to just changeddata blocks, CBT-based backup schemes generate incremental backup files that are afraction of the size of backup files created using full data files containing changed blocks.In our tests legacy agents working on a file-level basis generated incremental backupsthat were 100 times larger than incremental backup files created with AppAssure.

More importantly, this means a CBT-based backup scheme limits the need for datadeduplication to just the contents of each incremental backup file. As a result, there is noneed to implement a complex, CPU-intensive, data deduplication scheme that maintainsa global store of unique blocks that must be checked every time a backup is processed.

Using afilter driver tocollect CBTdata, theAppAssureclient serviceputs negligibleCPU overheadon a VMrunning inproduction.On VMsperformingheavy IOPStransactionprocessing,we measuredAppAssureoverhead tobe just 1%. Asa result, ITcan readilydeployAppAssureacrosss any

hypervisor or physical server environments to garner the performance benefits of CBT-based backup and recovery processing.

03

APPASSURE SMART AGENT OVERHEAD

We observed AppAssure client service activity while we ran an Iometer benchmark that wrote 450MB of dataper second—128KB writes at 3,600 IOPS. During this test, the AppAssure client service was writing 40KB ofdata per second to a log in the System Volume directory. Meanwhile, CPU usage for the AppAssure client servicewas just 0.9%.

Nonetheless, mostvendors of data protectionsoftware designed for VIenvironments simplyleverage the VMware filesystem (VMFS) CBTscheme exposed by thevStorage APIs for DataProtection. By following thisapproach, data protectionsoftware can treat VMs asVMs without any need for aspecial agent to perform astandard VM backup.Working directly with anESXi or ESX hypervisor,however, is not enough tounlock many importantadvanced backup andrecovery features, such asthe recovery of database

tables without having to recover an entire VM.

SMOKE MIRRORS AND AGENTLESS AGENTS

In the black and white marketecture debate of agentless versus agent-based dataprotection, individually licensed and managed agents are the issue. In the real world, thereare many shades of gray and agentless host-level backup is not the only alternative toheavy-weight legacy agents, which simply masquerade VMs as physical systems.

AppAssure Smart Agents focus on adding value to any client—VM or physical—anddo not require a license or come with a management GUI. An AppAssure Smart Agentprovides a backup client with a dynamic CBT mechanism that dramatically enhances thefunctionality and the performance of backup and recovery processes. For CIOs who needto rigorously support a Service Level Agreement (SLA) addressing business continuity,AppAssure creates an new dynamic with respect to a Recovery Time and a RecoveryPoint Objective (RTO and RPO) that works the same way in every environment.

To get a clear picture of the role that an AppAssure Smart Agent can play in a VMbackup scenario, openBench Labs set up three VM servers on vSphere and Hyper-V.Each of the VM server ran Windows 2008 Server R2 and included a Domain Controllerand a SQL Server 2008 R2 database server, For comparison, we set up agentless backupsleveraging vStorage APIs for Data Protection to run over both a network and a SAN.

04

AGENTLESS OVERHEAD PROCESSES

Using an dedicated agentless VI backup package, it is still necessary to configure user logincredentials to run VM-specific tasks, such as the Windows VSS writer for transactionalconsistency and content indexing services to enhance file-level restores. More importantly,backup and restore performance potential was gated by hypervisor communications andperformance overhead.

SUMMARy VM DATA PROTECTIOn PERFORMAnCE By AGEnT TyPE

Data Protection

Key VM MetricsAnalysis

AppAssure v4.7

Smart AgentAgentless Legacy Agents

Host Server

Overhead

Backup overhead on

host governs

backup window

options and directly

impacts backup

RPO.

AppAssure’s CBT

measured less than 1%

CPU utilization during an

Iometer stress test.

Total processing load on

VMs identical with

agentless testing.

For dvanced object-level

data protection within

VMs, “agentless” host -

level backup software

still needs to launch

nonpersistent processes

on VMs.

Masqueraded VMs as

physical systems with

minimal VM optimization.

Backup

Functionality

Backup processing

scalability directly

impacts resource

utilization and

capital expenses.

Multiple simultaneous

backups scaled by

increasing CPU resource

utilization while wall clock

time remained fixed.

AppAssure automatically

discovers protected

applications and adds

daily validation tests for

application recovery.

GUI displays the status of

SQL database and

Exchange mailbox tests.

Any backup can be used

to recover data objects.

Multiple simultaneous

backups scaled by

increasing wall clock

time by a factor of 3X, as

CPU utilization remained

fixed.

Automatically discovered

protected applications.

Any backup can be used

to recover data objects.

Minimum time to set up

and complete an

incremental backup

averaged 2 minutes.

Global block-level

deduplication dramatically

increased backup and

recovery process time.

File-level incremental

backups based on changed

files consumed up to 100X

the disk space of block-level

incremental backups.

RTO and RPO

Capabilities for

Business

Continuity

Update standby

VMs—OS and data—

with production

server disk

snapshots frombackups

Restore applications

and data with no

perceived delay

Smart Agents restoreddisk volumes with no

perceived delay in data

delivery.

Smart Agent-based

incremental backups

started and completed in

under 15 seconds.

Automatically configuredand updated standby VMs,

which were identical to

production servers.

Agentless backup

enables direct recovery

from backup files, which

are read-only, byredirecting writes via VM

snapshots and redo logs.

Full recovery requiresconsolidation of the

original and redirected

data files using vMotion

or a proprietary VM copy

utility.

Legacy agents supported all

of the hot-metal recoveryfeatures on VMs that were

supported by physical

clients.

The key question for sites considering AppAssure comes down to the value of anindependent, driver-based, CBT mechanism. Performing three simultaneous fullbackups in an environment without a high-speed SAN present, AppAssure was 3Xfaster than an agentless backup that exploited the vStorage APIs. More importantly,adding an 8-Gbps SAN to give backup servers direct access to vSphere datastores viavStorage APIs did not trump the advantages provided by AppAssure’s Smart Agents.

Using AppAssure, three VMs could be backed up 34% faster than using agentless

06

SMART AGENT BACKUP SCALABILITY

AppAssure backups with Smart Agents on our quad-core backup server were more efficient than similar backup processes thatrelied on vStorage APIs without agents. A backup of a VM running SQL Server took slightly less time and dramatically less serverCPU resources with Smart Agents. As a result, Smart Agents scaled three simultaneous backups in the same backup window,while an agentless backup of the same three VMs took three times the amount of time as numerous subprocesses came into play.

backup with an 8Gbps SAN in place. This advantage carried over to a Hyper-Venvironment, where AppAssure was 40% faster, As a result, both SMB sites that cannotafford a costly SAN and large enterprise sites with extensive SAN infrastructure canleverage AppAssure to garner considerably smaller backup windows.

The key backup advantage provided by AppAssure’s Smart Agents, however, was notin processing of full backups with lots of data. The key to an effective disk-to-disk (D2D)incremental backup regime with automatic synthetic full backups is the ability togenerate highly efficient incremental backups.

AppAssure Smart Agents processed incremental backups of both vSphere andHyper-V VMs running SQL Server in just 14 seconds and generated backup files,which contained 17MB of compressed and deduplicated data. What’s more, theAppAssure Smart Agents scaled three simultaneous incremental backups in just 19seconds. In sharp contrast, an agentless incremental backup of our SQL Server VMtook 8X longer (2 minutes and 5 seconds) on vSphere and 15X longer (3 minutes and25 seconds) on Hyper-V.

The bottom line for CIOs is that Smart Agents provide unparalleled RPO and RTOfor business continuity. The capability to turn around incremental backups in under oneminute gives IT the flexibility to create more recovery points using Smart Agents. What’smore, that same technology can then be used to complement that RPO advantage withan equally impressive RTO advantage via AppAssure’s Live Recovery™.

It is also important to note that it was the backup server load and not the VM clientload on our host servers that was the controlling factor in our scalability testing. Onall of the tests, the processing load of the three combined test VMs on our ESXi hostremained below 10% with agentless and smart agent processing.

5-MINUTE RPO AND RTO

From the perspective of a Line of Business (LoB) executive, the value of backup andrestore lies entirely in the recovery process. Their attention is focused on an aggressiveRecovery Point Objective (RPO) which limits data loss and an aggressive Recovery TimeObjective (RTO) which limits the time needed to complete a recovery process.

AppAssure doubles down on the basic universality provided by its block-based architecturewith three key technologies: Live Recovery™ to meet near-zero RTO and five-minute RPO;Recovery Assure™ to automatically verify that both a backup is recoverable and thatapplication data within the backup is recoverable; and Universal Recovery™ to supportthe granular recovery of files and application items in virtual-to-virtual (V2V), virtual-to-physical (V2P), physical-to-virtual (P2V), or physical-to-physical (P2P) scenarios. Nodata protection software tested by openBench Labs offers the same range of recoveryoptions from a single incremental backup file.

For improving IT’s image with end users, the most magical and impressive SmartAgent technology is without doubt AppAssure Live Recovery. Live Recovery is a unique

07

volume restoration feature. During a recovery, a Smart Agent monitors block datarequests from users and instructs the backup server to reorder the data being transferedto meet user requests. As a result, users perceive all data as having been restoredimmediately, as they have immediate access to any data on any volume.

With LiveRecovery, therestorationlegerdemain for alogical volumestarts with thetransfer of thevolume’s MasterFile Table (MFT).With the MFT inplace, usersimmediately seeall of the filesthat were on thedrive at the timeof the restorepoint. As a result,end users havethe impressionthat the drive hasbeen magicallyrestored to fulloperationalstratus inseconds. LiveRecovery isdistinctly

different from booting a VM from a backup file.

When a VM is booted from a backup file, its contents are represented by read-onlypointers with no active mechanism to intelligently to handle redirection of writes. As aresult, a static method, such as a VM snapshot or redo logs must be chosen. Unlessspecialized hardware is provided, for example a solid-state disk, only half the normallevel of IOPS will be sustainable. In addition, it will also be necessary to recover thesnapshot or redo log data by running vMotion or a similar proprietary function.

The AppAssure instant recovery scheme is not without serious pitfalls. The mostobvious question is: What happens when an end user clicks on a file that is not there? Ina database-driven environment, there is the distinct risk of corrupting one or morecritical internal tables or causing considerable delays and problems for mission-critical,transaction-processing applications that utilize a database.

08

LIVE RECOVERY OPTIMIZED DATA BLOCK RESTORATION

When we started a restore of a data volume, Live Recovery immediately restored the file structure data andenabled access to the volume. When we accessed any file during the restoration process, the AppAssure Coreserver immediately reordered the block data flow and sent the blocks that we needed. As a result we couldinstantaneously access any file or a even a corrupt Exchange mailbox database.

To avoid these issues, AppAssure Live Recoveryrelies on critical coordination between theAppAssure Smart Agent on the client system and aSmart Service on the AppAssure Core server tointelligently coordinate and reorder I/O during aLive Recovery process. When an applicationattempts to access data queued on the AppAssurebackup server, the requested data addresses aretransferred to the backup server, which reordersthe block stream queued for the client so that therequested blocks are at the top of the queue. As aresult, a large volume file server or an Exchangemailbox database can be restored instantaneouslyfrom the perspective of an end users

To thoroughly test all of the nuances of theAppAssure recovery options, we set up anExchange 2010 server. We placed two databaseswith 100 user accounts each on two separate disks.A third logical disk was used for the Exchange logfiles. Running Jetstress on the Exchangeconfiguration we easily complied with a Jetstressbenchmark designed to simultaneously stress 200accounts with one email message per second. Ourconfiguration easily passed muster with enoughhead room to support 50 to 100 additionalsimultaneously active accounts.

To test Live Recovery, we deleted one of theemail databases and selected a restore pointcorresponding to an incremental backup on ourbackup server. AppAssure automatically generateda synthetic backup for our restore point and begancopying the database to the disk on the Exchangeserver. Once the restore process began, weconfigured and started a Jetstress benchmark thatrequired the presence of the missing database.From the start of the test, the AppAssure clientcoordinated the order in which mailbox data wastransfered from the AppAssure backup server.

Never in the process did a transaction sampledby Jetstress exceed the stringent access timerequirements of Exchange performancebenchmark. What’s more, as the heavy transactionload stressed the database that we were

09

APPASSURE LIVE RECOVERY WITH EXCHANGE

From an Exchange 2010 server, we deleted one of two mailboxdatabase, which contained 100 email accounts. After starting LiveRecovery, we launched Jetstress, which generated a message foreach account every second. Never did a sampled transactionexceed the access time requirements of Jetstress. As the heavytransaction load stressed the database, AppAssure increased thethroughput rate for data recovery measured in MBs per second..During this period, the IOPS rate continuously escalated until itreached about 175 messages per second for each database. Moreimportantly, we reached the steady state point for Jetstress, withonly 60 percent of the Live Recovery process completed.

recovering, AppAssure not only transferred the required data, but also increased thethroughput rate for data recovery measured in MBs transferred per second. During thisperiod of the test, the IOPS rate continuously escalated until it reached the steady state ofabout 175 messages per second for each of the two databases. More importantly, wereached the steady state point for Jetstress, with only 60 percent of the Live Recoveryprocess completed.

What was by far the mostunexpected result of the LiveRecovery email database restoretest turned out to be the Jetstressperformance report: Not only didwe not crash or stall the test, weactually passed certification forsupporting 200 simultaneouslyactive mailboxes. Even moreincredible, the average IOPS rate and access time overhead for the benchmark testturned out to be virtually identical to running Jetstress on our Exchange server withno other processing load.

Live Recovery is just one of the ways that AppAssure works to assuage the concernsof LoB executives over business continuity. In a competitive 24x7 economicenvironment, computer downtime represents more than lost revenue to sales andmarketing executives. These executives equate lengthy computer outages with potentiallosses in customer confidence and market share.

As a result, senior LoB executives expect IT to meet an RTO that is measured inhours rather than days and an RPO that is close to lossless. AppAssure empowers IT tomeet and exceed those stringent RPO and RTO goals. What’s more, with the richhypervisor-independent capabilities of AppAssure, CIOs can get a jump on thecompetition by bringing compelling innovative services to market faster and supportingthem more effectively, once they are rolled out.

NEXT GENERATION CLOUD COMPATIBILITY

Given the CBT architecture of AppAssure, we were able to immediately utilizeAppAssure v4.7 in a next generation hypervisor-independent cloud environment, whichincreases management options and lowers administrative and labor costs dramatically.We used the Release Candidate of Microsoft® System Center Virtual Machine Manager(SCVMM) 2012 to provide a single-pane-of-glass management environment. Specifically,we were able to create multiple private clouds provisioned with VMs hosted on a mix ofvSphere 5 and Hyper-V servers.

In addition, SCVMM provided our Hyper-V hosts with a shared library, in which tostore full VM images and virtual disk files in order to automate the creation andprovisioning of VMs. As a result, IT administrators have the means to create andprovision VMs from a library of disk images created to be compliant with business

10

“Given the CBT architecture of AppAssure,we were able to immediately utilize

AppAssure v4.7 in a next generation hypervisor-independent cloud environment, which increasesmanagement options and lowers administrativeand labor costs dramatically.”

policies. Not only does this provide a means for IT administrators to roll out newapplications and services more accurately, they can do it up to 35 times faster than whenworking within a traditional IT environment.

In thisconfiguration,we were able tocontain thegrowingproblem of VMsprawl byassigning VMsto distinctclouds. Eachcloudrepresented adistinctmanagementzone. Given thepotential tofurther simplifycloudmanagementwith thisoperationalparadigm, it isnot surprisingthat many dataprotectionsoftwarevendors are inthe process ofintroducing

new software releases to work with both vSphere 5 and Hyper-V.

Nonetheless, most of the new data protection software remains highly dependent onleveraging underlying host hypervisors to enhance performance and functionality. As aresult, those packages cannot provide data protection services that integrate acrossheterogeneous hypervisor platforms in order to extend the hypervisor-independentparadigm.

In sharp contrast to competitive offerings, from other vendors, AppAssure v4.7Backup and Replication software extends many of its advanced functions across VMsresident on vSphere 5 and Hyper-V hosts. Specifically, we were able to leverage the lowoverhead of AppAssure backups to create a warm standby VM on any vSphere 5 hostcorresponding to any production VM on a Hyper-V host. What’s more, we were able to

APPASSURE CROSS PLATFORM FUNCTIONALITY

From the AppAssure console, we were able to backup and restore Widows-based VMs without regard to theunderlying host supporting the VM. More importantly, we were able to utilize the Create VM wizard to create awarm standby VM on an ESXi host for any protected system, including any VM running on a Hyper-V host. Wewere also able to use the same wizard to export logical disks from any protected system, including any VMrunning on an ESXi host, as Hyper-V formatted virtual disks for use as templates in the SCVMM library.

11

leverage the cross-platform functionality of AppAssure v4.7 with the SCVMM 2012library to create templates from VMs running on vSphere 5 hosts and then use thetemplates to automate VM provisioning on Hyper-V hosts.

ESXi and ESX hosts can update VMsthat are not in a running state. Thatallows an AppAssure backup server togenerate disk snapshots for a warmstandby VM from incremental backupsof any virtual or physical system.

A key to keeping a warm standby VM updated is the minimal amount of CPUoverhead imposed on both the AppAssure client and target backup server whenprocessing incremental updates. The entire process of sending an incremental backupfrom the client to processing, and saving the update on the AppAssure backup server canbe repeated over intervals measured in minutes, which allows the AppAssure backupserver to quickly send disk snapshots to the vSphere 5 host supporting the warm standbyVM.

The warm standby VM is a complete copy of the original system down to the state ofthe OS. The standby VM boots directly from a standard host datastore, immediately takesthe identity of the original system, and exhibits full disk and network I/O performance.When a protected AppAssure virtual or physical client crashes, the warm standby VM canbe booted in seconds and brought online processing data in minutes. In this way,AppAssure can be used to satisfy the most stringent business continuity SLA with respectto an aggressive Recovery Time Objective (RTO).

What’s more, AppAssure backups are self contained, which allows backups on oneserver to be replicated to another backup server in an HA scenario. Replicating the small,compressed and, deduplicated incremental backups puts minimal stress on a LAN and isvery well suited to WAN infrastructure. IT garners greater efficiency by replicatingincremental backups to an off-site secondary AppAssure backup server via a WAN andusing the off-site server to generate disk snapshots for an off-site, warm standby VM.

We were also able to leverage the AppAssure Create VM wizard to create virtual disksfor use by VMs on a Hyper-V server from any backup file. Specifically, we used backupsof VMs on vSphere 5 to create virtual disks and exported the virtual disks to theSCVMM library.

BACKUP AND RESTORE BENCHMARK SUMMARY

In all of the openBench Labs benchmark tests, we consistently measured nodifference in the overall CPU utilization levels between VM with Smart Agents installedand VMs running in an “agentless” environment running on VMware vSphere andMicrosoft Hyper-V hosts. This highlights the fact that the agentless construct of host-level backups is nothing more than an artifact. To process advanced application-centricfunctions, VM-level and host-level backups need to run processes on the VM.

“When a protected AppAssure virtualor physical client crashes, the warm

standby VM can be booted in seconds andbrought online processing data in minutes.”

12

The key difference lies in theability of a host-level backup toexploit the hypervisor. ForVMware, this comes down to theadvantages provided by thevStorage APIs versus theadvantages that Smart Agents canprovide by dynamicallyoptimizing data flow. In our tests,backup an recovery with SmartAgents generated significantlylower overhead on our backupserver.

As a result, we were able toscale multiple processes withSmart Agents without increasingthe time needed for backupwindows. Using quad-coreservers, we were able to run up tothree simultaneous processes inthe same time frame a singleprocess.

What’s more, unlike traditional legacy backup agents, AppAssure Smart Agents runentirely in the background on their hosts. There are no associated management task,monitoring, or licensing functions.

Jack Fegreus is Managing Director of openBench Labs and consults through RidgetopResearch. He also contributes to InfoStor, Virtual Strategy Magazine, and Open Magazine,and serves as CTO of Strategic Communications. Previously he was Editor in Chief of OpenMagazine, Data Storage, BackOffice CTO, Client/Server Today, and Digital Review. Jack alsoserved as a consultant to Demax Software and was IT Director at Riley Stoker Corp. Jackholds a Ph.D. in Mathematics and worked on the application of computers to symbolic logic.

13

AppAssure Smart Agent Benchmarks1) Minimize Backup Windows on Virtual and Physical Clients: Smart Agents

install a filter driver to track data block changes and accelerate data transfer.

Smart Agents 9X faster than agentless on an incremental backup of avSphere VM—17MB saved in 14 seconds vs. 24MB saved in 125 seconds.

Smart Agents 15X faster than agentless on an incremental backup of aHyper-V VM—17MB saved in 14 seconds vs. 24MB saved in 209 seconds.

Smart Agents 3X faster than agentless running 3 simultaneous fullbackups of vSphere VMs without a SAN—23.3GB saved in 18 minutes 24seconds vs. 30GB saved in 55 minutes 22 seconds.

Smart Agents 30% faster than agentless running 3 simultaneous fullbackups of vSphere VMs with a SAN—23.3GB saved in 18 minutes 26seconds vs 30GB saved in 24 minutes 41 seconds.

Smart Agents 40% faster than agentless running 3 simultaneous fullbackups of Hyper-V VMs—27.8GB saved in 18 minutes 24 seconds vs24.7GB saved in 25 minutes 48 seconds.

2) Zero Perceived Delay on Restores of Disk Volumes: Smart Agents reorderthe transfer of blocks based on user access of files during recovery.

Smart Agents recovered an Exchange mailbox database while runningJetstress—Recovered a mailbox database for 100 users while processingone email transaction per second per user.