IBM Spectrum Protect and VMware
The evolution of Spectrum Protect VMware support
VMware has grown in complexity over the years, and a large organisation can host a lot of VMs on a single ESX, with the size of each VM running to several terabytes. You might need more than one Spectrum Protect storage server and several Data Movers, all of which have to be monitored and managed. First TSM, then Spectrum protect has evolved over the years to cope with all this. The consequence is that some techniques that worked fine for TSM 7 are no longer supported for Spectrum Protect 8.1.8. So how you configure and work with the Data Protection for VMware will depend very much on which version you are using. The historical synopsis below might help.
TSM 6.4 was the first release to provide comprehensive VMware support, but the configuration was manual and complicated, the vCenter plugin was clunky, and we had a few problems with backups and restores.
TSM 7.1 introduced a GUI that was intended to simplify the initial configuration, and that GUI meant you could access the vCenter Server without needing a plugin. You could recover individual MSSQL databases from a VM backup and 'instantly' restore a full VM from a snapshot. Also you could backup and recover a VM that was hosting a Microsoft Active Directory domain controller.
Spectrum Protect V8.1.8 enhancements include:
The ability to monitor your entire Spectrum Protect VMware environment, including being able to view information about all backup schedules and completed backup and restores across multiple backup servers from the Spectrum Protect vSphere plug-in.
When you install IBM Spectrum Protect, you configure an initial default backup server with the setup wizard. This server is used for the web application on the GUI host and cannot be removed from the plug-in. You can configure extra backup servers with the plug-in and assign each backup server to support a data center on a vCenter.
You can configure file restore capabilities on a secondary IBM Spectrum Protect server on a remote Windows system. You can also use either a Windows mount proxy node or a Windows data mover node to run instant restore and instant access operations.
Spectrum Protect V8.1.7 enhancements included:
You can use the -tagschedule option with the backup VM command to invoke an immediate backup or rebalance of all VMs associated with a specified tagged schedule.
All failed VMs now have an associated error code that you can drill into to find out why the VM failed to be backed up. You can use a new panel to display details on the cause of the failed backup, including a preview of the error log for the data mover.
You get a better view of backup schedules that are currently active at the plug-in GUI.
You can select multiple VMs at configuration time, and change their scheduling, data movement assignment and other backup management settings.
Spectrum Protect V8.1.6 enhancements included:
More streamlining of the installation process, including setting up data movers.
A new option to flexibly configure restore operations across multiple VMs using a CSV file
Support for the new NBDSSL (Network block device SSL) protocol compression options.
Support for vSphere 6.5 VM encryption. However if you restore an encrypted VM from backup, it is restored as non-encrypted, so you need to encrypt it again manually.
You can assign individual VMs to a specific schedule using the tagging functions.
DP for VMware V8.1.6 and higher do not support the ability to directly mount a snapshot to view it locally with read-only access on the client system. You must use the recovery agent GUI to perform iSCSI mounts.
Spectrum Protect V8.1.4 enhancements included:
The default selection every time you use the DP for VMware vSphere GUI login page is Configuration mode, so you can set up the configuration before you log into the GUI. Configuration mode allows you to manage the plug-in registration information, and it will either use an available certificate, or if one is not available, then use the default certificate that came as part of the SSL/TLS handshake protocol. From version 8.1.7, configuration mode is only the default option at initial install.
If a schedule does not have an assigned data mover and there is a free data mover available, the data mover is automatically assigned to the schedule when the schedule is selected during configuration.
You are no longer required to manually set the password when you add a data mover. The Set Password option is removed, and the password is generated automatically.
Some VMware basics
VMware Vsphere is the VMware virtualisation platform.
A VM is a Virtual Machine, or simply a virtual Windows or Linux server. VMs are hosted on physical boxes called Hypervisors, or ESX devices and ESX devices are grouped together into clusters.
A VMware vSphere Datacenter contains the physical components needed to host a Vmware system, such as x86 virtualization servers, storage networks and arrays, IP networks, a management server, and desktop clients. Don't confuse this with the popular definition of a datacenter, which is a building that houses IT equipment, an IT datacenter could possibly contain several VMware datacenters.
A VMware vCenter server can be used to centrally manage a Vsphere infrastructure from a single console. Spectrum Protect comes in at this point by providing an interface to the vCenter server, so backup and recovery services can be managed from the same central point.
However, Spectrum Protect continues to provide a standard command line interface to VM, which is much faster and easier to use than the plug-in.
To complete the physical picture, a vStorage Backup Server is a dedicated Windows machine, physical or virtual, that interfaces between the VMware vSphere infrastructure and the Spectrum Protect Server.
back to top
Initial Configuration using the 8.1 vSphere GUI
IBM recommends that you do most of your configuration work using the vSphere GUI, as it is used to run manual and scheduled backups, to recover VMs and to run reports. You can either access this GUI as a plugin to the vCenter Server, or as a stand-alone web browser GUI. You normally install this GUI on the vStorage Backup Server, but you can install it anywhere that has network connectivity to the vStorage Backup Server, the Spectrum Protect server and the vCenter Server. However the GUI performance suffers if you have a large environment with lots of VMs.
First, check your installation documentation for the current list of pre-requisite actions that you need to take before starting the configuration, and complete those tasks. They are mainly to make sure you have a network connection between the component servers, and to set some Windows parameters. Once you have completed this, the install tasks are:
- Log into the VMware vSphere GUI using the vCenter user name and password.
- From the 'getting Started' window go to the Configuration window and click Run Configuration Wizard.
- Follow the instructions on the wizard, installing components, accepting EULAs, checking settings and adding data as necessary and go through to completion.
You have now installed Data Protection for VMware on your server. This is obviously much easier than the manual method described below, but the downside is that you will not have developed the same level of expertise and knowledge as you would have done if you did the install manually.
You should end up with the following:
For a vSphere install, the vCenter node, VMCLI node, datacenter nodes, and data mover nodes are registered on the Spectrum Protect server.
For a vCloud install, the vCloud Director node, VMCLI node, Provider VDC nodes, Organization VDC nodes, and data mover nodes are registered on the Spectrum Protect server.
The proxy relationships are all defined for these nodes, the vmcliprofile is updated and set and the local VMCLI password is set.
You should see four user interfaces, the vSphere GUI, the Recovery Agent, the vCloud GUI and my old favorite, the command-line interface.
The Recovery Agent GUI lets you take snapshots from the Spectrum Protect server and then mount them on the Data Mover Node. If you want to recover a few files, the snapshot content can be viewed in read only mode and the required files copied over to the original. If you want to restore an entire VM, you can use these snapshots to instantly restore a VM, so you access the data from the snapshot while it is being copied in the background.
The vCloud GUI is used to backup and recover vApps and organization vDCs in a vCloud Director environment. This GUI is accessed from a URL, there is more detail about VMware vCloud management below
The vSphere command-line interface is still there. While Tivoli states that the vSphere GUI is the primary way to manage backups and restores I always feel that you get much better control and messages with a command line. The command line works with both vSphere and vCloud environments and I think that the fastest way to manage the environment is just to use the Windows command line and dsmc commands.
The IBM Spectrum Protect VMware Nodes
OK, so we've mentioned lots of nodes above. What do they all do? A Node is a Spectrum Protect client, physical or virtual, which runs some of the Spectrum Protect software necessary to backup VMs to a Spectrum Protect server. Every node is registered on the Spectrum Protect server and has a unique name to identify it. The names will be specific to your site. A typical vSphere environment will have:
At least one data mover node. This node represents a specific Spectrum Protect backup-archive client that "moves data" from one system to another. If you are running a small environment where the VMs are backed up by a single client, the VM data is stored directly under the data mover node. That is, if you run a 'query filespace nodename *' command on the Spectrum Protect server with the name of the data mover node substituted for nodename, you will see all the VMs backed up to that node listed as file spaces.
A Proxy, or Datacenter node. In a larger environment, several data movers are used to back up a complete virtual environment, such as a VMware datacenter. Although the backup work is distributed among multiple data movers, the VM data is stored in a single shared node and this shared node is called the datacenter node or proxy node. The reason for this is that if you backup files to a shared node, then you can recover any VM from any datamover node.
An optional VMCLI node. If you use the vSphere GUI for management, then in a large vSphere virtual environment with several data movers and maybe several datacenters, you need a third node to communicate among the nodes and Spectrum Protect server and this node is the VMCLI node. It connects the command-line interface to the TSM server and the TSM data mover node. As this node is just used for communication, it does not need a Spectrum Protect client acceptor or scheduler service.
Mount Proxy nodes are used to access mounted VM disks through an iSCSI connection, which means that the file systems on the mounted VM disks are accessible as mount points and are typically used by the recovery agent for install full VM restores. A separate dsm.sys file stanza is used for each mount proxy node.
back to top
Configuring a VMware Backup Server manually
Changes needed at the Spectrum Protect server
You will probably need a new policy domain and associated objects for your VM backups at your Spectrum Protect server. Exactly how you set these up will depend on your site standards. You will need two management classes, one to keep the VMware metadata on disk, and one to place the actual data on a storage pool that migrates to tape. Of course you may decide to write direct to tape and even to use LAN free. IBM advises that you store CTL files on disk to speed up restores, and this certainly does help speed them up. In general terms, the size of the metadata disk storage pool needs to be about 1% of the total VM backup capacity.
To make the example configuration below a bit easier to follow, I'm going to propose a single DataCenter called VCSDC001 with 16 ESX Hypervisors called esx001 - esx016. These will be hosted on two vStorage Backup Servers called TSMBACK01 and TSMBACK02, with each server managing 8 ESX hypervisors. The whole lot will be proxied by a single node called VCSDC001_PX.
Client Node Definitions
You need several client nodes as follows.
2 'normal' Spectrum Protect clients just used to backup the data on the vStorage servers.
4 Data Movers, called TSMBACK01_DM, TSMREST01_DM, TSMBACK02_DM, TSMREST02_DM. The reason you need 4 Data Movers clients is because a Data Mover can only be used for one operation at a time, so if it is being used for backups, it cannot be used for restores. So here we have 2 Data Movers dedicated for backups and 2 for restores. Make sure you do not use the userid parameter when defining these special VMware nodes.
The Data Mover nodes need to be defined with a high MAXNUMMPOINTS parameter. The exact value to use depends on how many sessions you will allow to run in parallel and is given by the formula
Total number of sessions = (2 + 2 * VMMAXParallel)
The reason you need this many is because you need 1 session for main thread, 2 sessions per vm operation - one for the consumer and one for the API, and 1 more session for a consumer that runs at the start of the backup, starts, connects to the server, checks that there's nothing to do and closes the session.
So for our example, if we backup 8 VMs in parallel, we would need to set MAXNUMMPOINTS to at least 2+2*8=18.
1 proxy node, called VCSDC001_PX, which is used to collate all the backups from the vStorage servers.
You might need to set up Spectrum Protect client proxy relationships, depending on how many nodes you configure. Each client will store backups in the correct place, and can access all VMs for restores. The proxy commands that you need to run on your Spectrum Protect server to set these relations up are:
grant proxynode target=VCSDC001_DC agent=TSMBACK01_DM, TSMREST01_DM, TSMBACK02_DM, TSMREST02_DM
A typical VMware schedule definitions looks like this - note that all the ESX hypervisors that will be scanned for VMs are explicitly defined in the schedule, and that the schedule uses the -asnodename option so the backups are stored under the VCSDC001_PX proxy node. The Data Mover client is assocated with this schedule.
DEFINE SCHEDULE VMWARE VMWARE_DAILY_INCR Type=Client DESCription="Daily if incremental backup of VM servers" ACTion=Backup
SUBACTion=VM Options='-asnodename=VCSDC001_PX -MODE=IFIncremental' STARTDate=today STARTTime=21:45:00 SCHEDStyle=Enhanced DAYofweek=ANY
DEFINE ASSOCIATION VMWARE VMWARE_DAILY_INCR TSMBACK01_DM
VM, Disk and File Exclusions
To exclude individual VMs from automatic backup, you can include them in the schedule OBJECTS statement with a '-' in front:
UPD SCHEDULE VMWARE VMWARE_DAILY_INCR objects='-VM=VMname1,VMname1'
This will override the settings in the client option DOMAIN.VMFULL statement.
You can exclude multiple VMs separated by commas as shown.
You can use equivalent statements to bind individual files to a specfic management class
include \\*\c$\...\*.mp3 mclass3
Definition at the Windows Clients
Each vStorage server will need a standard dsm.opt file for normal client processing, and also the following clients.
The Data Mover clients, two for each VStorage server. Only one example is given here. The file names will be dsm.tsmback01_DM.opt, dsm.tsmrest01_DM.opt on TSMBACK01, with equivalent names on TSMBACK02.
* DATAMOVER TSM OPT FILE
ERRORLOGRETENTION 30 d
SCHEDLOGRETENTION 30 d
If you chose to use the vSphere interface then you will need opt files for those nodes too, but they are not mentioned here to save a bit of space.
The vSphere password in the above dsm.opt file is commented out as we do not want it to appear in plain text, so it must be created for each option file by command instead.
Run the following commands as admin in D:\SYSTEM\IBM\TSM\baclient, but note that the dsm.opt names are the ones I used above, and may not correspond to your site standards. VSphere_password is your password of course.
dsmc set password -type=vm TSMBACK01.DOMAIN.NAME.COM vSphere_userid vSphere_password -optfile=dsm.TSMBACK01_DM.opt
dsmc set password -type=vm TSMBACK01.DOMAIN.NAME.COM vSphere_userid vSphere_password -optfile=dsm.TSMREST01_DM.opt
Several Windows services need to be configured on the vStorage servers. As well as standard schedule services for the local backups, the following scheduler processes for each datamover node are required. These services are best defined using the dsmcutil command as you have more control over the parameters that you would get using the Spectrum Protect GUI wizard.
dsmcutil install scheduler /name:"Spectrum Protect Scheduler tsmback01 Data Mover"
/errorlog:D:\SYSTEM\IBM\TSM\baclient\dsmerror.tsmback01_DM.log /schedlog:D:\SYSTEM\IBM\TSM\baclient\dsmsched.tsmback01_DM.log /startnow:no /node:TSMBACK01_DM
dsmcutil install cad /name:"Spectrum Protect CAD tsmback01 Data Mover"
/cadschedname:"Spectrum Protect Scheduler tsmback01 Data Mover "
/autostart:yes /password:node_password /startnow:no
Updating the TDP option file
You would only need to do this if you use the vSphere GUI interface. This file is usually held in C:\Program Files (x86)\Common Files\Tivoli\TDPVMware\VMwarePlugin\scripts\vmcliprofile. Most of the parameters can be left at the supplied default values.
The VE_DATACENTER_NAME profile option is case sensitive it must exactly match the datacenter name as used by VMware. You can manage multiple Data Centers from one vCenter TDP, and then they would be specified with multiple VE_DATACENTER_NAME entries. These must be unique and have separate Spectrum Protect datacenter nodenames. The syntax is
Most of the options in this file can be left as the defaults, the ones you need to change are below, but obviously use your own values for userids and port numbers.
..... loads of default options
Once you change this profile, you need to restart the windows Data Protection Services
Now you need to set the TDP password that is used to connect to vSphere. Open a command line as Administrator then run commands like these, with your own values substituted. The password you need here is the one used by your CLI Spectrum Protect client.
cd C:\Program Files (x86)\Common Files\Tivoli\TDPVMware\VMwarePlugin\scripts
vmcli -f set_password -I pwd.txt
The vmcli command will delete that temporary password file. You can test that this works by running vmcli commands, for example
C:\Program Files (x86)\Common Files\Tivoli\TDPVMware\VMwarePlugin\scripts>
Vmcli -f inquire_config -t TSM
The final task is to configure the vCenter plugin on the vCenter GUI.
Start the VMware vCenter client GUI up, navigate to the Home directory from the address bar,
then right at the bottom of the Home screen you will see a Solutions and Applications panel, and in there you should see a Data Protection for VMware vCenter plug-in icon. click on that icon, then edit the Tivoli Storage Manager server settings by clicking Configuration tab > Edit.Configuration. The stuff you need to input includes:
Userid for Proxynode
password for proxy node
Spectrum Protect server address
Spectrum Protect server port
If you have problems with this, go to Windows Services and check that 'IBM WebSphere Application Server V7.0 - TSMVEplugin' and 'Data Protection for VMware command-line interface' are both started.
back to top
Some tips for running backups and restores
Spectrum Protect supports two different backup and restore types
A full VM backup takes a VMware guest disk snapshot then copies the VM configuration information and a block level copy of VM disks to the Spectrum Protect Server. The FULL-VM backup operation does not require a Data Protection for VMware license. Full VM backups are excellent de-dup candidates as they will produce loads of duplicate operating system files
An INCREMENTAL VM backup will backs up only those VM blocks that have changed since the last backup completed, provided that VMware's Change Block Tracking (CBT) is enabled. If CBT is not enabled, a full VM backup is taken and a warning message issued. This backup requires a Data Protection for VMware licence. If CBT is enabled, then Spectrum Protect will not backup empty areas of VM disks either.
Spectrum Protect uses the VMsnap function, which can be used to take a consistent backup of SQL databases hosted on a VM, as it quiesces database activity as part of the snap.
Open up a command line and navigate to d:\system\ibm\tsm\baclient, then start s dsmc session, but you need to point to the data mover dsm.sys and log in as the proxy node like this.
dsmc -optfile=dsm.tsmback01_DM.opt -asnode=VCSDC001_PX
To see what VMs are configured, try the command
show VM all
To see if any backups exist for a VM, use the command
q VM vmname -inact
you can backup a single VM with the following commands.
backup vm vmname -vmbackuptype=fullvm -mode=ifincremental
backup vm vmname -vmbackuptype=fullvm -mode=iffull
You can exclude individual hard disks from a vm backup like this
backup vm "vmname:-vmdk=Hard Disk 3" -vmbackuptype=fullvm -mode=iffull
Spectrum Protect VMcloud backups
The DP for VMware can backup and restore VMCloud templates and vApps.
A VM template is a master image of a VM. The template can include an installed guest operating system and a set of applications.
A vApp is a logical entity that consists of one or more VMs that make up an application. You probably want to consistently backup the whole application together, and if a restore is required, you may want to restore the whole application, that is, all the VMs in the Vapp, to the same point in time. Managing by vApp makes this easier.
Active Directory Backups
Spectrum Protect uses VMware Tools Snapshots for VM backups, and by default it takes the snapshots in quiesced mode. If the VM contains Active Directory components, then Spectrum Protect can have problems trying to take a quiesced snaoshot. Microsoft suggests that the best solution is to backup Active Directory VMs from non-quiesced snapshots. To do this, you need to add a parameter
INCLUDE.VMSNAPSHOTATTEMPTS vmname n1 n2
Where vmname is the name of the VM that contains AD components, n1 is the number of attempts to take a snapshot in quiesced mode and n2 is the number of attempts to take a snapshot in non-quiesced mode. The default values are n1=2, n2=0.
You can use wildcards on the VMname, so if all your Active Directory VMs start AD, you could code the following
INCLUDE.VMSNAPSHOTATTEMPTS AD* 0 2
Data Protection for VMware tagging is used to provide some gramularity to VM backups and scheduling. You can use tags to include or exclude VMs from a schedule, or to assign backups of a VM to a specific management class. You can assign tags to a range of VMware inventory objects, including Datacenter, Folder, Host, Host cluster and Resource pool as well as Virtual machine. When tagging support is enabled, you can assign data protection tags to VMware containers.
Common Backup Errors
If you run a full VM backup on a 32bit Windows OS, it may fail with an ANS9365E error and API return code 60 as shown in this error log extract -
ANS9365E VMware vStorage API error.
API return code : 60
ANS5250E An unexpected error was encountered.
Spectrum Protect function name : vmVddkStartOffloadMount
Spectrum Protect function : snapMoRefP is null
Spectrum Protect return code : 115
Spectrum Protect file : vmbackvddk.cpp (1957)
ANS4151E Failure mounting Virtual Machine 'PASVM002A'. RC=115
ANS4150E Incremental backup of Virtual Machine 'PASVM002A' failed with RC 115
One reason for this error is that there was a timeout, due to lack of memory in the VM guest. Check out how much memory is allocated to the VM, and if it is quite small, say about 2GB, then get it increased.
This command will run an instant restore of a vm called prd047. The temporary datastore is used to store the vm configuration data while the restore is in progress and the name must be unique.
dsmc restore vm prd047 -vmrest=INSTANTRestore -vmtempdatastore=temp_datastore_02
The command to restore a vm called dev25 is shown below. This command will restore from the most recent backup. If you want an earlier backup you can add the -pick option so see a list of available backups and then you can select the one you want.
dsmc restore vm dev25
The vCloud restore command to restore a vApp called vApp_Admin is shown below. You obviously need to substitute your own values for the org name, the orgvdc name and the vapp name.
restore vApp org=org_HR,orgvdc=prod,vapp=vApp_Admin
You cannot restore the Windows system state from a VMware snapshot. The reason is the the system state consists of more than just the files that are backed up by a snapshot, it also consists of the registry.
There are some restrictions on vCloud restores. It is not possible to restore a single VM template as TSM marks the vCloud template object as a single file.
It is possible to restore individual VMs that are contained in a vApp, but if you select a vApp for restore then you restore all the components within that vApp. If you restore vms within a vApp and that vApp still exists, then the restored vms are restored to the vApp. If the vApp does not exist then the VM is restored to the top-level default location on the target ESX host.
you can't restore a single file from the command line, and it is impossible to restore over the top of an existing VM, you need to restore files to a new name
restore vm vmname -vmname="restvmname"
If you want to recover VMs using the GUI then this is quite straightforward, but you will not see error messages or progress messages as you do with the command line. One thing to be aware of is that when you want to select the VM to restore, click on the virtual machine name, not on the check box to the left of the virtual machine. If you do click on the select box you will see an error message like 'This node cannot be selected'.
File level Restores
It is possible to recover an individual from from a snapshot, to do this you mount the backup tape as a 'drive' on your TSM backup server, then you can copy the data over. The process in outline is
- Open up the 'Data Protection for VMware Recovery Agent' GUI
- Select the 'new Tivoli Storage Manager Server' window and fill in the details for your server
- Set the 'Data Access - Storage Type' window as appropriate, usually to 'tape'
- Pick out the VM that you need to restore the file from in the 'Virtual Machine' window, select the date that you want the snapshot from, and select the disk name
- Click on the 'Mount' button, select the drive you want to restore from and the drive that you want to mount to on the backup server. The default mount drive is e:
- Click OK and wait for your tape to be mounted
- If you see a message that the TDP Mount Virtual Volume Driver is not installed, just click 'OK' to get it installed and the tape should mount OK
- Now open Explorer and navigate to the file you want recovered. If you are reading a tape, this could take a little while. Drag and drop the file onto the VM that you are restoring to.
- Finally, go back to the TDP GUI, highlight the drive you were working with and dismount it with the dismount button to free up the drive
back to top
If you want to know more about Spectrum Protect and VMware then read the Installation and User guides:
Version 8.1.8: Data Protection for VMware Installation Guide
Version 8.1.8: Data Protection for VMware User Guide
Monitoring VMware activity
Normally, if you want to find out what backups and restores have been running against a Spectrum Protect client, you would use a command like this, where the ENTITY filter will pick up the clients that you want -
select activity,,NUMBER,entity,SCHEDULE_NAME from summary where ENTITY is like 'VE%' and start_time>(current_timestamp-1 days)
However, if you run this command against VMware clients you will not get the correct data. You need to use the SUMMARY_EXTENDED table instead as this has been specifically introduced for VMware clients. So the correct command to use is
select activity,,NUMBER,entity,SCHEDULE_NAME from SUMMARY_EXTENDED where ENTITY is like 'VE%' and start_time>(current_timestamp-1 days)
back to top