Skip to main content
  • Place orders quickly and easily
  • View orders and track your shipping status
  • Enjoy members-only rewards and discounts
  • Create and access a list of your products
  • Manage your Dell EMC sites, products, and product-level contacts using Company Administration.

Avamar: VMware image backups fail with, "Code 10056 and avvcbimage Error <9759>: createSnapshot: snapshot creation failed"

Summary: avvcbimage FATAL <0000>: [IMG0002] The datastore information from VMX '[DATASTOREPATH] abc.vmx' does not permit a restore or backup.

This article may have been automatically translated. If you have any feedback regarding its quality, please let us know using the form at the bottom of this page.

Article Content


Symptoms

VMware client Image backups are failing with Code 10056.
The Snapshot create task fails with one of the following errors:
Issue #1 - NOTE: Avamar 19.4 and below only
avvcbimage Warning <16002>: Too many extra snapshot files (3) were found on the VMs datastore. This can cause a problem for the backup or restore. 
avvcbimage FATAL <16018>: The datastore information from VMX '[MLUN] Datastore/TestMachine.vmx' will not permit a restore or backup.
avvcbimage Info <0000>: Starting graceful (staged) termination, Too many pre-existing snapshots on the datastore will not permit a restore or backup. (wrap-up stage) 
avvcbimage Error <9759>: createSnapshot: snapshot creation failed
The log shows that Avamar did not detect a Snapshot for this VM initially:
avvcbimage Info <14664>: no snapshots are present on this VM.
However, there are Snapshot files on the Datastore:
avvcbimage Info <14677>: All VM file Info:
File:'TEST-000003-ctk.vmdk                   ' last modified on '07-Dec-2015 15:58'  size=3932672
File:'TEST-000003-delta.vmdk                 ' last modified on '07-Dec-2015 15:57'  size=134344704
File:'TEST-000003.vmdk                       ' last modified on '07-Dec-2015 15:55'  size=408
 Issue #2 The Virtual Machine has too many existing snapshots (30 or more).
avvcbimage Info <14665>: 30 snapshots are present on this VM:
   level=1, children=1, ref='snapshot-4223', 'snapshot1':desc='na'
   ...
     *level=30, children=0, ref='snapshot-4252', 'snapshot30':desc='na

avvcbimage FATAL <0000>: [IMG0019] The pre-existing snapshots from VMX '[MLUN] Datastore/TestMachine.vmx does not permit a backup.

Cause

For issue #1 - NOTE: Avamar 19.4 and below only
If there are snapshot files in the Virtual Machine directory and vCenter reported less than were discovered during the scan, the backup fails.

When Avamar attempts to back up a Virtual Machine, it queries vCenter for how many snapshots the Virtual Machine contains and the Datastore directory where the Virtual Machine resides is then scanned.
For Issue #2 
The Avamar proxy does not allow Snapshots of a VM that has 30 existing snapshots.
VMware ESI only permits up to 32 snapshots before the snapshot task fails with: 
Create Virtual Machine Snapshot task fails with a VMware error of:
"Snapshot hierarchy is too deep."
We must take a new "Avamar" snapshot (bring Snapshot count to 31).
At the end of the backup, we must delete this Snapshot. This delete action causes ESXi to create a hidden ESXi Snapshot called consolidated helper (snapshot count would be 32). This could fail due to the ESXi "Snapshot hierarchy is too deep" issue. 

Resolution

Issue #1  - THIS ISSUE IS RESOLVED on Avamar 19.7+.  

For Avamar 19.4 and below
Resolution using GOAV
Snapshot files must be removed from the Virtual Machines Datastore Directory.  The tool GOAV can be used to simplify this.
1. Download and Install the Goav tool:
Avamar: Goav tool
2. Run goav vm snapshot clean.  This removes orphaned snapshots from VMs and any disks that are stuck to a proxy. 
This can be run two ways:
    - This can be run against a single virtual machine with the --name switch.
       - Example:
 goav vm snapshot clean --name winpe-testing-2
    - This can run against ALL VMs, as seen below.
admin@ave:~/>: ./goav vm snapshot clean
===========================================================
GoAv    :        1.49
Avamar  :        19.7
Date    :        03 Mar 2023 10:29 EST
===========================================================
COMMAND :  ./goav vm snapshot clean
NOTE: This is not an official tool
===========================================================
Running "./goav proxy clean" to clean proxies : ConsolidateTask=True
================== Checking Active Vmware Image backups =================
No active Vmware backups found
================== Checking PROXIES for unwanted DISK =================
------------ 192proxy.lab ------------
Proxy 192proxy.lab is clean
------------ 197proxy.lab (19.7.100-82) ------------
Proxy 197proxy.lab (19.7.100-82) is clean
================== Checking Vms for leftover snapshots and disk consolidation =================
....
MANAUL FIX
Snapshot files must be removed from the Virtual Machines Datastore Directory.
Before removing the files, there are some checks that must be completed.
  1. Ensure that the Virtual Machines are not using the snapshot file in question.
  • Looking at the virtual machines Settings > Hardware tab > select each disk and ensure the "Disk file" is not allocated to a Snapshot file. (Snapshots have a suffix name of <vm name>-00000#.vmdk
  1. Ensure that the Avamar proxies do not have the disk attached to them.
  • In vCenter, select a proxy and browse the Settings > Hardware tab

  • Each proxy should have two disks, any additional should be removed

  • To remove an extra disk, select the disk and click "Remove" and leave the "Removal Options" selection to "Remove from Virtual Machine."

  1. Run a Snapshot Consolidation 
  • In vCenter browse the Virtual Machine, then in the tool bar  select > Inventory Virtual Machine > Snapshot > Consolidate

  • Manually remove the Snapshots if Consolidate does not work. Do not do this if there are other valid user Snapshots or if the VM disks are using any Snapshot file.

    1. Browse the Datastore directory of the Virtual Machine

    2. Create a Folder in the VM Directory Called "SS Removal"

    3. Drag and drop the offending snapshot files into the new directory created above.

  1. Inspect the Datastore directory for the Virtual Machine in question and check that the Snapshot files exist.

  2. Once removed perform a backup.

  3. If the issue is not resolved, contact Avamar Support.

For Issue #2.  
In VMware, delete the existing Snapshot. VMware does not recommend using Snapshots as your only backup strategy (For example: Use a product like Avamar). Having more than a small set of short-term Snapshots introduces latency and high stun times during Snapshot operations. Ideally a VM should not have Snapshots.

Additional Information

 

Article Properties


Affected Product

Avamar

Product

Avamar, Avamar Client for VMware

Last Published Date

29 Feb 2024

Version

7

Article Type

Solution