[IDPA 2.1] NDMP Restore fails with error message "Backup is invalid or not a NDMP backup"


   Article Number:     529556                                   Article Version: 2     Article Type:    Break Fix 




Integrated Data Protection Appliance Family,Integrated Data Protection Appliance Software,Integrated Data Protection Appliance 2.1





The following errors can be seen in the NDMP restore job log:     
      2018-12-06 15:03:04 avndmp Info <6444>: [avndmp_assist] Plugin exited with 'code 157: miscellaneous error'           
            2018-12-06 15:03:04 avndmp Error <11799>: [avndmp_assist] Backup is invalid or not a NDMP backup.           
            2018-12-06 15:03:04 avndmp Info <11780>: [avndmp_ctl_sup] Global avndmp_assist exit code is: 161

To Verify the ddboost plugin version on Avamar, run the following command :   
  •         strings /usr/local/avamar/lib/libDDBoost.so | grep "[0-9]\.[0-9]\.[0-9]\.[0-9]"     
  •         grep -i engine ddrmaint.log | tail -1     
    In DDFS log ( /ddr/var/log/debug/ddfs.info) on Data Domain, the following errors can be seen:       
        Note: The NDMP accelerator node IP address can be searched in the ddfs.info logs on DD to confirm the issue.
      12/06 15:02:45.099 (tid 0x90a9b30): nfs_rpc_svc_idx0 accepted cfa 255 from       
        12/06 15:02:45.103 (tid 0x7ff35b151220): nfsproc3_ostmntsec_3_svc: connection failed permission (corrupted credentials) from host - ost_decrypt_mnt_sec_request(): failed to finalize plain text (101077092, error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt)
      This confirms that DD was unable to decrypt the password provided for the NDMP Restore.    






Avamar server 7.5.0 uses DDBoost library version: This issue exists on DDBoost library 3.4.0 and is fixed on 3.4.1.       
        Avamar server version 7.5.1 has DDBoost library version: embedded in it.        
        Starting from 3.4 plugin, Data Domain does authentication using pre-shared-keys(PSKs) generated from the password. The client generates a PSK from the application supplied password, encrypts some authentication information and sends it to DDR. The DDR will try to generate the same PSK (since it already knows the password for a user) and decrypt the contents the client sent and validate.       
        With DDBoost ifgroup enabled, data domain falls into the reconnect path, which calls the PSK connection path and relies on the password crypt_hash stored in the nfs_conn structure. On DDBoost plugin though, DD does not generate the password crypt_hash again during reconnection. So the PSK key is not the same on the client and DDR. Hence the decryption failure.






Workarounds ( Use one of the following methods to get around this problem):       
        Method 1: Use username-password authentication. (Disable session security on the Avamar Server)       
        Follow the KB 484773 to download the SessionSecurityConfiguration avp        
        a.) Pre-checks before installing avp.

  •         Make sure maintenance task, backup, restore, and replication is not running while installing SessionSecurityConfiguration avp     
  •         Make sure Avamar server has a current checkpoint.     
b.) Navigate to the Avamar Installation Manager(AVI) (http://<IP/fqdn of the avamar server/avi) page in a Web Browser and login as root user   
  •         Under the Maintenance tab, select SessionSecurityConfiguration and click on Run :     
User-added image   
  •         Select Disabled mode from the drop down menu under Security settings tab :     
User-added image   
  •         Click on Continue.     
    Please follow KB 492934       
        Method 2: Upgrade the Avamar Client Catalogue to version 7.5.1-101 and then upgrade the NDMP Accelerator node to the same version (7.5.1).       
        Method 3: Disable the DDBoost ifgroups on the Data domain. Note: Please be aware that disabling ifgroups might directly impact the backup and restore performance.       
        Login to the Data Domain.       
        a.) Run the below command to view the existing ifgroups on the DD:       
        ddboost ifgroup show config all       
        Confirm which ifgroups are being utilized for backups. Make a note of the ifgroup name.       
        Group-name    Status        Interface      Clients     Replication       
        -----------   --------      ---------      -------     -----------       
        default       disabled           0         1             0       
        backupgroup   enabled            2         0             0       
        -----------   --------      ---------      -------     -----------       
        Group-name    Status     Interfaces       
        -----------   -------   -----------       
        backupgroup   enabled   10.x.x.x       
        backupgroup   enabled   192.x.x.x       
        -----------   -------   -----------       
        Note: In the above example, the ifgroup named "backupgroup" is enabled and in use.       
        b.) Run the following command to disable the ifgroup in question.       
        ddboost ifgroup disable <group-name>