We already reported on the new Feature Pack 3 (FP3) for Backup Exec 15 from Symantec / Veritas a few days ago . Veritas has really added new features to this feature pack and there seems to be a problem. The problem manifests itself such that after the backup a SnapShot can be seen in the Hyper-V Manager console , as you can see here below.
It was inexplicable why the Hyper-V VM now had a checkpoint (snapshot) , although we knew exactly that we had not changed anything on the VM . However, the date and time revealed the culprit, so it was definitely our backup program ” Backup Exec 15 ” from Veritas . The problem with this, however, is that the Hyper-V files .AVHD are created in addition to the existing .VHD or .VHDX files and all changes since the snapshot are written there. This means that significantly more storage space is required and this ultimately led to the full running of the Partition and the failure of all VMs, because these then freeze. Only then did we become aware of this phenomenon.
After that we ran endless tests, but always with the same result that these .AVHD files were created. We then opened a technical ticket on myveritas.com and asked for technical support. After a day we received a call from Veritas and gave us the following statement.
Solved Hyper-V .AVHD Backup Exec problem
By Feature Pack 3 (FP3) from Backup Exec 15 new is feature added that a quick incremental backup allows. This is also Veritas’s recommendation. However, we cannot understand at all why such a new technology is set as standard by Feature Pack 3. Apparently, many users of Backup Exec with FP3 have already complained who had the same problem. Therefore, we would like to recommend all Backup Exec users who have the same problem to switch this setting back to the previous method ” standard method” , as can be seen in the following image.
You can find this setting under
After we changed the backup method of the virtual machines to the “use standard method “, the backup ran again properly and no .AVHD files and no snapshot are created.
Interesting fact is still that this problem seems not only incremental backups occur but also with full backups . The friendly Veritas Supportler then said that he would send us a document containing further information. We want to make this available to you here below and we hope that this will answer most of the questions.
If using the faster method and you see more than 1 Backup Checkpoint, when a job is not running (or more than 2 Backup Checkpoints, when a job is running) then something, outside of Backup Exec’s direct control, blocked the removal of an earlier checkpoint and you should review the Hyper-V-VMMS event log (Event Viewer -> Applications and Services Logs -> Microsoft -> Windows -> Hyper-V-VMMS -> Admin) for errors.
Also if more than the expected number of backup checkpoints are seen, be prepared to use a powershell command to remove the checkpoints to tidy up. If this is a regular problem and / or the powershell command fails to tidy up the snapshot then Microsoft should be involved to help understand and identify why the checkpoint (s) cannot be removed. Note: the first time a faster method full backup takes place then it is expected to see event ids 15070 and 10155 against unable to remove a checkpoint.
The powershell command that appears to be able force the merge of backup snapshots is
Get-VMSnapshot -VMName ‘VM’ -ComputerName ‘HOST’ | Remove-VMSnapshot
The setting choice for faster and standard method Hyper-V backups is against the whole Backup Exec server (and not specific to each job) as such you change it within the Virtual Machines section of the general Backup Exec settings and not inside the job configuration.
If no faster method backups have ever been run and the customer changes the setting to perform standard method backups then the backup checkpoints and AVHDX files removed at the end of the job. However if they ran any faster method backups before running standard method backups then a checkpoint will remain as the active state of the machine. This will cause the avhdx file behind the ‘stuck’ checkpoint to continue growing in size and could cause disk space issues. The powershell command given above has to be used to reset this behavior back to no checkpoints left when jobs are not running.
The new / faster method will merge the older checkpoint into the VHDX file and only keep the newer checkpoint file (AVHDX) at the end of each backup. This checkpoint file will then exist until the end of the next backup when it itself is merged into the vhdx. So when using this method (apart from the first ever full backup), during a backup job 2 checkpoints and 2 AVHDX files will be seen however at the end of the job only the later checkpoint (avhdx file) will remain. As such, as long as regular backups are run, the checkpoint file will not keep growing as the original machine state (vhdx file) is still updated / merged / committed into regularly. Note: The new / faster method backup will create this checkpoint as the full job runs irrespective of whether only full jobs will be run.
Further notes:
At various points in time, whilst backup jobs are active, AutoRecovery Checkpoints (avhdx files) are also created but these are always removed before the job finishes.
Also no matter whether running standard or faster method jobs “Backing up”, “Merge in Progress” and “Creating Checkpoint” statuses can be seen in the Hyper-V manager against the VM.
A lot of additional information about Backup Exec can be found in the following articles.
– Backup Exec 16 from Veritas available
– FP5 for Backup Exec 15 released (revision 1180)
– Backup Exec – Create SDR disk – Part 1
– Copy backup job at Symantec (Veritas) Copy Backup Exec
– Symantec (Veritas) Backup Exec Error V-79 -57344-33932
– Symantec (Veritas) Backup Exec error message (FIXEDB2DDevice, memory could not be deleted)
– Deduplication Option Error with Symantec Backup Exec 2014 Upgrade
– Backup Exec 2012 Snaphot Error 0xe0008526 (V-79-57344-34086)
– Symantec Backup Exec error message 0xe0008488 – Access denied
– The BackupExec Management Service could not be started – .NET Framework update error