vmware backups that work—lessons learned from vadp performance benchmark testing

Disclaimer

This information is about pre-release software. Any unreleased update to the product or other planned modification is subject

to ongoing evaluation by Symantec and therefore subject to change.

This information is provided without warranty of any kind, express or implied. Customers who purchase Symantec

products should make their purchase decision based upon features that are currently available.

1

Participate in the discussion!

2

#SYMCHangout

http://bit.ly/1efz3zB

VMware Backups That Work

George Winter Technical Product Manager, Symantec

Abdul Rasheed Technical Marketing Manager, Symantec

Lessons Learned From VADP Performance Benchmark Testing

3

Agenda – NOTE: UPDATE THIS

Why Perform These Benchmarks?

Benchmark Testing History

Benchmark – NBD Transport

Benchmark – SAN Transport

VADP Process – Quick Review

The backup snapshot process

HW and Backup Configuration Recommendations

Which Method is Fastest?

4

Benchmark Testing History

5

Three VMware Backup Benchmarks

• Requires significant hardware & very slow

• VCB has since been deprecated (not going to discuss)

VMware Consolidated Backup (VCB)

• vStorage API for Data Protection

• NetApp FlexPod, Cisco UCS, NetBackup

NBD (network) benchmark

• vStorage API for Data Protection

• Cisco UCS, NetBackup

SAN transport benchmark

6

Benchmark Environments – Real World Configurations

7

Cisco UCS Blades

1x vCenter, 6x ESXi

Nexus

NetBackup 5220 NetApp FAS 6080 NetApp V-Series 6080

Block Storage

8Gb Fibre Channel

10Gb Ethernet

Why Perform These Benchmarks?

Benchmarks Are A Lot Of Work!

8

Benchmark Goal – Better Backups For You!

Trust but verify – never assume performance

Develop independent performance testing techniques

Tell You What We Learned

Understand hardware and backup API (VADP) limitations

Faster backups

More reliable

Correctly size backup server

Shorter backup window

Restore performance

Restore options

9

10

VADP Backup Process

Quick Review

VADP Backup Process

11

VMware ESXi

Datastore

Backup Server

ESXi Host

VADP Backup Process

12

VMware ESXi

Datastore

Backup Server

VADP Snapshot of Target VM(s)

ESXi Host

VADP Backup Process

13

VMware ESXi

Datastore

Backup Server

VM Backup Sent to Server

ESXi Host

VADP Backup Process

14

VMware ESXi

Datastore

Backup Server

Snapshot Released

ESXi Host


15

#SYMCHangout


16

Backup Snapshots Exposed!

Why Do You Care?

A Closer Look - What Happens During a Backup?

• VSS provider flushes OS buffers in VM

• Snapshot of VM is taken (vmdk(s) are frozen) (SCSI reservation of LUN)

• Redo log created – all writes redirected to redo log (Redo expanded in 16 MB chunks)

• VM is backed up

• Redo log data applied to original vmdk(s)

• Snapshot released - backup completed

During VM backups, VMware creates temp snapshot of VM. Process is as follows:

• Every one of these steps involves significant amount of I/O

• Reducing number of snapshots per Datastore improves backup perf & reliability

• Incrementals are quick and can reduce snapshot impact

Why does this matter?

A Detailed Look at the VADP Snapshot Process

17

VMware ESXi

Reducing Backup I/O Impact – Improve B/U Reliability

• Backup during periods of low VM activity

• Use incremental backups (CBT) liberally

• Limit simultaneous backups per ESX / Datastore (VIP)

Shorten time backup snapshot is open

• Design backup policies to evenly balance load across ESXi / Datastores

• Faster backups = shorter time snapshots are open

Configure NetBackup for optimal performance

• Snapshots more reliable

• Overall backup processing faster

Result:

18

19

VADP Snapshot Pop Quiz

VIP Configuration Exercise – NBD Transport

20

UCS

VMDK VMDK

VM2

ESXi

ESXi

ESXi

ESXi

ESXi

ESXi

Network

Backup Server

• Network = 10GbE

• 6 ESXi Hosts

• VMs = 100 GB

Q: How many streams per ESXi host will provide fastest possible backup? 1, 2, 3, 4, more?

10GbE

1 Stream / ESXi Host (6 streams total)

Network = 10 GbE

100 MB/sec per ESXi

Each VM = 100 MB/sec

Backup 21 VMs / Hour

NBD Transport – How Many Streams?

21

UCS

VMDK VMDK

VM2

ESXi

ESXi

ESXi

ESXi

ESXi

ESXi

3 Streams / ESXi Host (18 streams total)

Network = 10GbE

100 MB/sec per ESXi

Each VM = 33 MB/sec

Backup 21 VMs / Hour

VIP Configuration Example – NBD Transport (Review)

22

UCS

VMDK VMDK

VM2

ESXi

ESXi

ESXi

ESXi

ESXi

ESXi

Network

Backup Server

• Network = 10GbE

• 6 ESXi Hosts

• VMs = 100 GB

10GbE

NBD Transport – Impact Of Snapshots Per Datastore?

23

UCS

VMDK VMDK

VM2

ESXi

ESXi

ESXi

ESXi

ESXi

ESXi


Snapshots / DS = 3

Redo Space Req’d:

45 GB Per Datastore


Snapshot / DS = 1

Redo Space Req’d:

5 GB Per Datastore

???


24

UCS

VMDK VMDK

VM2

ESXi

ESXi

ESXi

ESXi

ESXi

ESXi


Snapshots / DS = 3

Redo Space Req’d:

45 GB Per Datastore


Snapshot / DS = 1

Redo Space Req’d:

5 GB Per Datastore

45 GB


25

UCS

VMDK VMDK

VM2

ESXi

ESXi

ESXi

ESXi

ESXi

ESXi


Snapshots / DS = 3

Redo Space Req’d:

45 GB Per Datastore


Snapshot / DS = 1

Redo Space Req’d:

5 GB Per Datastore

Why 45 GB for 3 simultaneous backups?

• Each VM backs up at slower rate

• Snapshot for each VM is open 3 times as long

• Time to capture data change in redo is longer

• Does not improve overall backup performance!!

45 GB

26

NBD (Network) Transport

Lessons From Benchmark Testing

NBD (Network) Transport – The Good News

Super easy to set up – 5 minutes on a bad day

No software required on any VMware component

Supports any Datastore type

27

NBD (Network) Transport Data Flow – Heat Map

28

LAN VMware ESXi

Backup Server

System Load

• ESXi host does all the work Pulls data from storage Sends data to backup server

NBD (Network) Transport – The Not-So-Good

Direct resource impact on ESXi host

Backup data path directly from ESXi host to Backup system

Backup data transfer through VMkernel port

• Our benchmark testing and VMware have verified this

• First backup stream throughput approximately 100MB/sec

• VMware throttles VADP throughput to reserve VMkernel port bandwidth

VMkernel port is bandwidth limited by VMware (e.g. QoS)

We’ve developed techniques for working around this….

29

VADP Performance – VMkernel Bandwidth Limits

• Single ESXi host – NBD Performance Per Stream

30

# of streams: 1 2 3 4

100

MB/Sec

100 MB/sec

75

MB/Sec

60

MB/Sec 55

MB/Sec

220 MB/sec Creates VMkernel strain on ESXi host

Based on Cisco/VMware/NetBackup

Benchmark Testing 150

MB/S Total

100 MB/S Total

180 MB/S Total

220 MB/S Total

NBD Best Practices For Optimal Backup Performance

• Number of simultaneous backups?

31

Aggregate Backup Speed

Recommended Settings:

• VMware recommends no more than 4 backup streams per ESXi host

• We recommend 1 or 2 streams if possible

• 10 or more ESXi hosts: Number of streams = 1

• Less than 10 ESXi hosts: Number of streams = 2


32

#SYMCHangout


SAN Transport

33

SAN Transport Data Flow – Heat Map

34

VMware ESXi

Backup Server

System Load

• Skip the ESXi host!

SAN Transport – The Good News

Supported with Fibre or iSCSI storage (no NFS support)

Only “off-host” backup for VMware

Near zero impact on ESXi hosts

Backup server does all of the work

More ESXi resources reserved for your VMs

No VMkernel performance limitations

Fastest possible backup performance!

Restore over SAN also possible regardless of backup transport (check with vendor)

35

SAN Transport – The Not-So-Good

Storage shared between ESXi host and backup system

Need to understand the risks

Backup system can overwrite VMware partition

36

SAN Transport – Best Practices For Protecting Storage

• http://technet.microsoft.com/en-us/library/ff715663.aspx

Use Windows SANpolicy:

Restrict Administrator or Root access on Windows/Linux backup system

Disable Disk Management snap-in Group Policy

Disable Windows automount on backup system

Configure LUNs as “read only”

37

http://technet.microsoft.com/en-us/library/ff715663.aspx



HotAdd Transport

38

HotAdd Transport

• Can be fast - bypasses the VMkernel port

• Supports all storage types

• Ideal for remote office environments

The Good News

• Directly impacts the ESXi host with all backup processing

• Proxy VMs need to access the datastore of VMs being protected

• Configuration issues – add SCSI controllers in VM

• Issues with stuck snapshots, CBT on HotAdd VM, etc.

The Not-So-Good

39

How About No Transport?

40

Array Based Snapshot – Heat Map

41

VMware ESXi

Backup Server

System Load

• Zero load on any vSphere component

• No VMkernel bandwidth limit

• Ultimate performance 1 2

Storage Array Based Snapshot – What To Look For

Ability to schedule snapshots just like any other backup

Backup application should automatically handle entire data lifecycle

•Expensive

•Can be required for single file restore

No requirement for additional “backup proxies”

Automatic VM selection based on storage location

•Throttle snapshot creation

•Throttle snapshot deletion

•Critical for limiting I/O “storms”

Control quiesce process – consistent backups

42

Storage Array Based Snapshot – What To Look For

Ability to schedule snapshots just like any other backup

Backup application should automatically handle entire data lifecycle

•Expensive

•Can be required for single file restore

No requirement for additional “backup proxies”

Automatic VM selection based on storage location

•Throttle snapshot creation

•Throttle snapshot deletion

•Critical for limiting I/O “storms”

Control quiesce process – consistent backups

43

NetBackup 7.6

NetBackup 7.6

NetBackup 7.6

NetBackup 7.6

NetBackup 7.6

NetBackup Replication Director For VMware

44

Protect 300 VMs in 300 Seconds!

45

Recommendations

Recommendations - Hardware

• You are paying VMware for CPU & RAM – use these resources for production

Avoid using a VM as backup system

• Two socket, 4 cores per socket plenty of processing power

Modern CPUs are incredibly fast

• 32 GB is minimum

• 64 GB if using deduplication

• More is always better – future proofs this system

RAM

• Performance of internal bus important

• Consider multiple HBA’s for network (teaming) or SAN (DMP)

Pay attention to I/O

46

Recommendations - Software

•Spread backups across all Datastores

•Reduces snapshot impact – I/O and space

Limit active backups per Datastore (Resource Limits)

•Avoid spending resources protecting VMs that don’t need it

•Automatically protect VMs that do need it

Use automatic VM backup selection

•Extremely fast

•Be sure solution controls every aspect:

Storage array snapshots add another dimension of protection

•300 VMs in 300 minutes

Backups, restores, snapshot retention, migration to long term storage

47

If your using full backups, your doing it the wrong way!

48

Advantages of the NetBackup Accelerator for VMware

– Significantly less CPU utilization saves HW and VMware license costs

– Backup is significantly faster – backup more VMs with same/less HW

– Retain all restore functionality – single file, folder, VM, database, application

Backup Processing Impact ESXi Host - CPU Utilization

Standard FULL VADP backup (NBD transport)

NetBackup Accelerator For VMware

49

Advantages of the NetBackup Accelerator Backup (network)

– Significantly less network traffic on already busy VMkernel port (NIC)

– Less backup hardware required

– Shorter backup window

Backup Processing Impact ESXi Host - Network Utilization

Standard FULL VADP backup (NBD transport)

NetBackup Accelerator For VMware

50

NetBackup Accelerator for VMware

• Protect extremely large VMs in seconds

• Faster, more reliable backups

51

Deduplication Technologies

• Benefits to hard to ignore

• Be careful of solutions that are basically compression

Quickly replacing standard disk as backup target

Trade I/O capacity (expensive) for CPU and RAM (cheap)

• Minimums = 64 GB RAM, 2 Sockets, multiple spindles

BYOD (Bring Your Own Disk) solutions

• NetBackup Appliance – performance tuned

• Hardware configuration correctly designed

• Easy – start protecting your VMs in 10 minutes

Deduplication Appliances

52

Thank you!

Copyright © 2013 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.

53

George Winter @vmxgeorge

Abdul Rasheed @AbdulRasheed127

vmware backups that work—lessons learned from vadp performance benchmark testing

Technology