SPO3292

Veeam
Backup &
Replication:
Tips and Tricks
Anton Gostev
Veeam Software
@Gostev
Doug Hazelman
Veeam Software
@VMDoug

#vmworldsponsor

Quick Overview of v6 Architecture

Backup servers
Backup proxy servers
Backup repositories
Automated everything
Intelligent load balancing
Centralized management via Enterprise Manager

Scaling your backups

3 simple rules

Scaling your backups

1. Keep the management server happy
Disable default proxy (requires 6.1 or later)
Allocate enough RAM for job manager processes
Keep concurrent (running) jobs under 100 per management server

2. Backup proxy servers: the more, the better?

No! Too much load on storage and network
Use max concurrent tasks limit on proxies or repositories
Watch for job timed out waiting for resources messages

3. Backup repository considerations

Be careful with the reversed incremental backup mode!
Limit concurrent jobs on backup repositories to a reasonable amount
Use ingest rate throttling for cross-SAN backups

A word on backup
repositories

Dont underestimate the importance of performance!

By far, the most commonly reported bottleneck

What makes the best backup repositories?

Windows or Linux server (can be same as backup proxy server)
Local storage, DAS or SAN mounted for physical server
pRDM disk (vSphere 5+), or iSCSI LUN connected via in-guest iSCSI
for virtual server

Sub-optimal backup storage

NAS or network share
VMDK on VMFS (size and recoverability considerations)

RAID level
If you can afford it, use RAID10 (again, performance)

Processing Modes
All you need to know

Direct SAN Access: The good

Fastest processing mode
Least impact on production
Backup processing is fully offloaded to dedicated backup proxy servers
Backup traffic is isolated to the storage network (aka LAN-free)

Does not impact consolidation ratio, so cheapest too

Direct SAN Access: The bad

Supports block storage only
FC (fibre channel): physical backup proxy server only!
iSCSI: physical and virtual backup proxy servers both supported

Physical backup proxy server requirement for FC SAN

Might not go along well with your virtualization project
Consider repurposing older servers

Might be hard for beginners to setup

See Veeam Forums FAQ for step-by-step guide

Manual datastore mapping might be required

For certain SANs, B&R may not able to detect proxy connectivity

Direct SAN Access: The ugly

Whats the worst that could happen?
Windows re-signaturing your VMFS LUNs!
vSphere will no longer recognize datastores
Dont panic, VMware Support should be able to fix

Three easy ways to get into trouble

1. Windows Explorer automounting new volumes (but not with Veeam)
2. Clicking Disk Management snap-in popup without reading
3. Giving Local Administrator rights to random people

Direct SAN Access: The safe way

Present VMFS LUNs to backup proxy server as read-only
Most SANs support it these dayschase your vendor if yours does not

Disable automount on your backup proxy servers

Do it the right way: use SANPolicy Windows setting!
Veeam backup proxy server setup does this automatically for you

Disable Disk Management snap-in with Group Policy

User Configuration > Administrative Templates
Window Components > Microsoft Management Console >
Restricted/Permitted snap-ins > Disk Management

Keep Local Administrator rights on backup proxy servers

to yourself
Cannot really do this for default proxy due to FLR requirement
Another reason to use dedicated backup proxy server!

Direct SAN Access: Tips & tricks

Got a fast SAN? Get a modern backup proxy server!
Multi-core CPU (compression) and fast RAM (inline deduplication)

Update firmware and drivers across the board

Disabling MPIO may increase performance
iSCSI SAN? Tweak TCP/IP on backup proxy
netsh interface tcp set global autotuning level = disable

Increase read-ahead buffer

Default is 4MB (optimal setting for most SANs)
To change, create the new value in bytes:
VddkPreReadBufferSize (DWORD)

Hot Add: The good

Easy to setupvery little planning involved
Any Windows VM can be made a Hot Add backup proxy

Fast data transfers with any storage

Direct storage access (albeit through ESXi storage stack)

Supports all types of storage (including NFS)

Shared storage: at least 1 backup proxy server per vSphere cluster
Local storage or DAS: at least 1 backup proxy server per host

Use your existing Windows VMs (save on licensing)

Data processing engine process runs with lower priority (6.1)
Further CPU usage reduction in 6.5

Allows for 100% virtual deployment

Hot Add: The bad

Not as mature as other modes
Affects your consolidation ratio
Backup proxy servers take host resources
Ultimately means more ESXi hosts, and more VMware licenses

Hot Add process itself is slooow

Can take up to 12 minutes to complete for each VMadds up quickly!

Hot Add as a vSphere feature has a number of limitations

Good newsmany are being removed as VDDK matures
See FAQ on Veeam forums FAQ for the complete list

Hot Add: The ugly

Snapshot removal problems due to locks
Veeam B&R: multiple hooks in place to work around

CBT must be disabled on backup proxy VM

Prevents stun on Hot Add due to CBT initialization

NFS-specific issue
Extended VM stun on hot remove in some scenarios

Hot Add: Tips & tricks

Add extra virtual SCSI controller to backup proxy server
A single SCSI controller can have a maximum of 16 disks attached
Concurrent jobs on the same backup proxy server can result in more!

Keep vSphere and Veeam up to date

Single block size in VMFS5 removes the most common hot add issue
Latest Veeam Backup & Replication will have latest and greatest
VDDK version

Try increasing read-ahead buffer

Seems to really help with certain NFS storage

Avoid cloning backup proxy VM

For example, to provision additional backup proxies

Network (NBD): The

good

Easy to setupin fact, no setup is required

Any existing server (physical or virtual) would do

Supports all types of storage, including NFS

Server placement does not matter (unlike with Hot Add)

Very quick to initialize data transfer

Can be quite fastwith 10Gb Ethernet

Network (NBD): The ugly

Painfully slow performance on 1Gb Ethernet
Average speed reported is 10-20 MB/s

Leverages ESXi management interface

Network (NBD): Tips & tricks

1 Gb Ethernet
Use for sites with low change rate
Works faster than other processing modes in such conditions

Keep at least one Hot Add backup proxy server around

Full VM and virtual disk restores take forever over NBD

Keep in mind intelligent load balancing algorithms

Network backup proxy servers have lowest priority!

Upgrade to 6.1 or later

Improved network proxy location awareness

One last thing

This hack significantly reduces supportability!
Cut up to 5 minutes of processing time per VM by
disabling VDDK logging
Apply in stable environments only!
Create the new value and set to 1:
DisableVDDKNetworkOutput (DWORD)

Deduplicating Storage
Yes, you can afford it!

Deduplicating storage: The good

What gives? Global dedupe!
Deduplication across backup files from different jobs
Perfect for long-term backup archival

Top hardware appliances among Veeam users

EMC DataDomain
ExaGrid
HP StoreOnce

Top software appliances among Veeam users

ZFS-based appliances
StarWind

Windows Server 2012 dedupe is awesome

Deduplicating storage: The bad

Hardware appliances are expensive
Although they do provide excellent dedupe ratio

Software appliances are resource hogs

Both performance and dedupe ratio are sub-par, too

Windows Server 2012 dedupe is awesome

Included free of chargestart using it today!
Provides very decent dedupe ratio

Deduplicating storage: The ugly

Random access performance is lacking
A problem for all solutions featuring inline deduplication
Typically insufficient out-of-the-box for large-scale vPower usage

Exception: post-process deduplication

ExaGrid
Raw disk landing zone (full-speed vPower from recent backups)
Veeam-specific logic further optimizes performance

Windows Server 2012

Backups land on raw storage at full speed
Only old backup files are deduplicatedgreat for vPower
Decent speed even off already deduped backups

Deduplicating storage: Tips &

tricks

Already own storage with inline deduplication?

Inline data rehydration process is what makes vPower slow

Reduce the block size in Advanced job settings

WAN (256KB) and LAN (512KB)
Reduced block size might impact backup performance

Use Linux-based backup repository

Large client cache, or even caching client file system (FS-cache),
can significantly improve vPower performance

Deduplicating storage: Tips & tricks

(continued)

Keep Veeam dedupe on

Use incremental backup mode
Choice of synthetic or active fulls depending on actual storage

For best dedupe ratio on device side

Disable compression (significantly increases amount of data transferred
from backup proxy server to backup repository over network)

For best backup performance and smallest window

Keep compression at default

If you like to avoid extremes

Set compression to Low (dedupe-friendly)

Deduplicating storage: Tips & tricks

(continued)

Got more than one deduplicating storage device?

Use internal replication to sync backups offsite!
Extremely traffic-efficient approach
Many customers use and report great success!

Keep the backups imported for easy DR

To automate repository refresh in DR site, use:
Get-VBRBackupRepository -Name "DR_Repository" |
Sync-VBRBackupRepository

WAN Accelerators
Your WAN on steroids

WAN accelerators: The good

Two types of WAN accelerators
Caching WAN accelerators provide significant bandwidth savings with
Veeam replication, but are typically quite expensive
Transport layer WAN optimizers are unlikely to offer significant
bandwidth savings with Veeam replication, and are usually cheap

Both improve reliability of TCP

Long distance wireless or satellite links
IPsec rekey operations on a VPN tunnel
WAN links with high jitter, packet loss or occasional drops

Both allow long-running jobs to finish more consistently

For example, initial replication over network

Both improve WAN utilization for most workloads

WAN accelerators: Whats hot?

Top caching WAN accelerators among Veeam users
Cisco WAAS
Riverbed
SilverPeak

Top WAN optimizers among Veeam users

Hyper-IP

WAN optimizers: Tips & tricks

Veeam Backup & Replication leaves little room for
bandwidth reduction by WAN optimizers
Built-in WAN optimizations in v6:
Multiple TCP streams to maximize throughput
Network traffic compression

Consider using on unreliable networks, but dont

expect them to add you extra bandwidth
Update to Backup & Replication 6.1 Patch 1 before evaluating

Veeam Backup & Replication not using all available

bandwidth?
Increase the amount of TCP streams (default is 5)
DownloadStreamsNumber (DWORD)

Caching WAN accelerators: Tips & tricks

Multiple TCP streams can cause issues!
Disable multiple streams in B&R traffic throttling settings

Disable network traffic compression in Veeam

Low (dedupe-friendly) compression level might be a better option

If required, have network admins configure bypass on

Veeam backup proxy servers to avoid polluting caches

Veeam Backup & Replication 6.5

We never stand still

Whats coming in 6.5

Veeam Explorer for Microsoft Exchange
Veeam Explorer for SAN Snapshots
VMware vSphere 5.1 support
Windows Server 2012 support

Veeam Explorer for Microsoft

Exchange

Visibility into Exchange VM backups

Immediate: No need to provision storage, restore the VM or
restore the mailbox store
Agentless
Requires no special backups or metadata collectioneven works with
existing Veeam backups (and SAN snapshots)

Free!
Included in all versions of Veeam Backup & Replication 6.5,
including Free Edition
Eliminates need for expensive standalone tools licensed per-mailbox

Currently available in exclusive beta

Just restored a 145 GB #MSExchange Public
Folder database in 2 min. using @veeam. Then
restored a single item all under 10 min. SWEET!

The new #veeam explorer for exchange

looks veeamy. That's right, I just made up a
new word (it means awesomesauce)

Even the *beta* of @veeam Exchange

Explorer works a treat. Saved literally, hours of
work.. and saved my bacon. #recommend

Really excited with the new @veeam Explorer

beta for #Microsoft Exchange VM backups - we
have lots of interested customers ready for this!

Veeam Explorer for Microsoft Exchange (continued)

Capabilities
Browse: familiar Explorer-type interface
Search: familiar Outlook-like Find, including Advanced Find
Export: export to PST file, MSG file or attachment

Uses cases
E-discovery
Item-level restore: export and send to affected user
Mailbox archive

Supports Exchange Server 2010

Veeam Explorer for SAN Snapshots

Veeam restores from SAN snapshots
Supports tiered data protection strategy
Perform all restores through familiar, easy-to-use
Veeam interface
Supports HP StoreVirtual VSA and HP LeftHand

SAN snapshots + Veeam restore =

Best RPOs and RTOs for operational recovery

Veeam Explorer for SAN Snapshots (continued)

Fast: recover entire VM or individual items in < 2 minutes
Fully automated: clone & promote snapshot, present to vSphere, clean up
Restores directly from VM files on the SAN snapshot: no staging or
intermediate restores required

Flexible
Specific VM
Individual guest files: Windows, Linux, et al
Individual Microsoft Exchange items

Free
Worry-free: automated process eliminates human errors and protects
integrity of SAN snapshots and production LUNS
Agent-free: no agents to deploy on hosts or VMs
Literally free: included in all editions of Veeam Backup & Replication 6.5,
including Free Edition

Questions? Comments?

Thanks for attending!

FILL OUT
A SURVEY
AT WWW.VMWORLD.COM/MOBILE

COMPLETE THE SURVEY

WITHIN ONE HOUR AFTER
EACH SESSION AND YOU WILL
BE ENTERED INTO A DRAW
FOR A GIFT FROM THE
VMWARE COMPANY STORE

SPO3292

Veeam
Backup &
Replication:
Tips and Tricks
Anton Gostev
Veeam Software
@Gostev
Doug Hazelman
Veeam Software
@VMDoug

#vmworldsponsor