IDPA Upgrade fails on ESX Compute node with error "Could not stage image profile '(Updated) Dell-EMC-13G-ESXi-6.5.0-update-04': ('VMware_locker_tools-light_6.5.0-1.47.8285314', '[Errno 32] Broken pipe')"

           

   Article Number:     534740                                   Article Version: 2     Article Type:    Break Fix 
   

 


Product:

 

Integrated Data Protection Appliance Family

 

Issue:

 

 

In the ACM GUI, under ESX Compute node upgrade details section, the following errors can be seen.        
       
        The "esxAppLevelUpgrade.log" shows the following error as well :
     
     
      RUNNING,31,Upgrading ESXi x.x.x.x. This may take around 15 minutes       
        RUNNING,32,Installing upgrade packages on ESXi x.x.x.x       
        RUNNING,33,Failed to install upgrade packages on ESXi x.x.x.x       
        RUNNING,33,Failed to upgrade ESXi Server
     
     
      In the esxi_logs.log under "/data01/tmp/patch/logs directory", verify the following error is seen :      
     
      04/06/19 13:40:04  main() Package name: Dell-EMC-13G-ESXi-6.5.0-update-04       
        04/06/19 13:40:04  main() Executing install packages command: esxcli software profile update --depot=https://x.x.x.x:9443/dataprotection-upgrade/esxi_upgrade --profile=Dell-EMC-13G-ESXi-6.5.0-update-04       
        04/06/19 13:40:50   run() Parsing returnCode, output: ['', '[InstallationError]\r\n [Errno 32] Broken pipe\r\n       vibs = VMware_locker_tools-light_6.5.0-1.47.8285314\r\n Please refer to the log file for more details.', 'Status: 1']       
        04/06/19 13:40:50 run_cmd_esxi() Command: esxcli software profile update --depot=https://192.168.100.100:9443/dataprotection-upgrade/esxi_upgrade --profile=Dell-EMC-13G-ESXi-6.5.0-update-04       
        04/06/19 13:40:50 run_cmd_esxi() [InstallationError]       
        04/06/19 13:40:50 run_cmd_esxi()  [Errno 32] Broken pipe       
        04/06/19 13:40:50 run_cmd_esxi()        vibs = VMware_locker_tools-light_6.5.0-1.47.8285314       
        04/06/19 13:40:50 run_cmd_esxi()  Please refer to the log file for more details.       
        04/06/19 13:40:50 run_cmd_esxi() Status: 1       
        04/06/19 13:40:50  main() Install packages command result: 1       
        04/06/19 13:40:50  main() Failed to install packages.
     
     
     
      The Upgrade utility log shows the following exception:      
     
      2019-06-04 14:41:17,777 INFO  [upgrade-workflow-4]-upgradeutil.UpgradeUtil: Getting latest progress percent from: /data01/tmp/patch/logs/status/esxAppLevelUpgrade.log       
        2019-06-04 14:41:17,778 INFO  [upgrade-workflow-4]-upgradeutil.UpgradeUtil: prevPercent: 20 currentPercent: 33       
        2019-06-04 14:41:17,778 ERROR [upgrade-workflow-4]-esx.EsxUpgradeOperations: Failed to upgrade ESXi. responseCode: 1       
        2019-06-04 14:41:17,778 ERROR [upgrade-workflow-4]-esx.EsxUpgradeTask: Failed to upgrade ESXi.       
        com.emc.vcedpa.upgradeutil.common.exception.UpgradeException: Unable to upgrade ESXi.       
                at com.emc.vcedpa.upgradeutil.configure.esx.EsxUpgradeOperations.upgradeEsxHosts(EsxUpgradeOperations.java:298)       
                at com.emc.vcedpa.upgradeutil.configure.esx.EsxUpgradeOperations.upgrade(EsxUpgradeOperations.java:170)       
                at com.emc.vcedpa.upgradeutil.configure.esx.EsxUpgradeTask.run(EsxUpgradeTask.java:46)       
                at com.emc.vcedpa.upgradeutil.configure.impl.UpgradeWorkflowManager.executeAcmAndEsxTask(UpgradeWorkflowManager.java:381)       
                at com.emc.vcedpa.upgradeutil.configure.impl.UpgradeWorkflowManager.executeAcmEsxWhenAllCriticalPpUpgradeSuccessful(UpgradeWorkflowManager.java:334)       
                at com.emc.vcedpa.upgradeutil.configure.impl.UpgradeWorkflowManager.notifyUpgradeTaskStatus(UpgradeWorkflowManager.java:343)       
                at com.emc.vcedpa.upgradeutil.configure.dd.DataDomainUpgradeTask.run(DataDomainUpgradeTask.java:47)       
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)       
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)       
                at java.lang.Thread.run(Thread.java:748)
     
     
     
      This issue maybe caused due to missing vsantrace partitions on affected ESX Host and the mtree is missing on DD for the same host where the VSAN logs are stored.        
       
        On the ESX Host , run the command "df -h" to confirm vsantrace parition is created and listed:       
       
        The output should look like this:
     
     
      [root@dpappliance-esx2:~] df -h       
        Filesystem   Size   Used Available Use% Mounted on       
        vfat       285.8M 285.7M    168.0K 100% /vmfs/volumes/5c82e746-bb00f246-9667-f8f21e473e10       
        vfat       249.7M 191.9M     57.8M  77% /vmfs/volumes/6186f3e0-df715171-4993-8b8091a1977e       
        vfat       249.7M 186.8M     62.9M  75% /vmfs/volumes/718348e6-2c128245-ca05-e9d78fbff849       
        vsan        22.7T  14.9T      7.8T  66% /vmfs/volumes/vsanDatastore
     
     
      A healthy system looks like as shown below:     
     
      [root@dpappliance-esx2:~] df -h       
        Filesystem   Size   Used Available Use% Mounted on       
        NFS        169.5T   2.0G    169.5T   0% /vmfs/volumes/vsantrace2       
        vfat       285.8M 285.7M    168.0K 100% /vmfs/volumes/5c82e746-bb00f246-9667-f8f21e473e10       
        vfat       249.7M 191.9M     57.8M  77% /vmfs/volumes/6186f3e0-df715171-4993-8b8091a1977e       
        vfat       249.7M 186.8M     62.9M  75% /vmfs/volumes/718348e6-2c128245-ca05-e9d78fbff849       
        vsan        22.7T  14.9T      7.8T  66% /vmfs/volumes/vsanDatastore
     
     
     
      Open a ssh session to Data domain and run the following command to confirm the issue:      
     
      sysadmin@example.com# mtree list       
        Name                           Pre-Comp (GiB)   Status       
        ----------------------------   --------------   ------       
        /data/col1/avamar-1xxxxx              0.5   RW       
        /data/col1/backup                         0.0   RW       
        /data/col1/esx1-logs                      0.1   RW/Q       
        ----------------------------   --------------   ------       
         D    : Deleted       
         Q    : Quota Defined       
         RO   : Read Only       
         RW   : Read Write       
         RD   : Replication Destination       
         RLGE : Retention-Lock Governance Enabled       
         RLGD : Retention-Lock Governance Disabled       
         RLCE : Retention-Lock Compliance Enabled
     
     
     
      In the above output, it is clear that the mtree for ESX Host 2 and 3 are missing for IDPA. Therefore we need to recreate the same using the steps in the resolution section. 
   
     
                                                           

 

 

Cause:

 

 

This is a known issue on IDPA 2.1 and Bug 247659 has been created by IDPA engineering team.                                                           

 

 

Resolution:

 

 

   

      Please follow the appropriate section which as per the host having the issue.         
       
        Note: Skip the steps for the hosts which already have the vsantrace partition created and have the mtree for VSAN logs on DD side.        
       
        Check if there are any NFS of DD(vsantrace*) loaded on each ESX.       
        esxcli storage nfs list       
       
        If it is not please perform following steps.        
       
        1: Steps for ESX Host 1 :       
       
        Run the following command on DD:        
        quota capacity enable       
       
        Run the following command on DD for ESX1:
     
           
  •           mtree create /data/col1/esx1-logs quota-hard-limit 2 GiB       
  •        
  •           nfs add /data/col1/esx1-logs 192.168.100.101       
  •      
     
      Run the following commands on ESX Host 1:      
           
  •           esxcli storage nfs remove -v vsantrace1        
  •        
  •           esxcfg-nas -a vsantrace1 -o 192.168.100.109 -s /data/col1/esx1-logs       
  •        
  •           esxcli vsan trace set -p /vmfs/volumes/vsantrace1       
  •        
  •           esxcli vsan trace set --logtosyslog=false       
  •      
     
     
      2: Steps for ESX Host 2 :       
       
        Run the following command on DD for ESX Host 2:
     
           
  •           mtree create /data/col1/esx2-logs quota-hard-limit 2 GiB       
  •        
  •           nfs add /data/col1/esx2-logs 192.168.100.102       
  •      
     
      Run the following commands on ESX Host 2 :      
           
  •           esxcli storage nfs remove -v vsantrace2       
  •        
  •           esxcfg-nas -a vsantrace2 -o 192.168.100.109 -s /data/col1/esx2-logs       
  •        
  •           esxcli vsan trace set -p /vmfs/volumes/vsantrace2       
  •        
  •           esxcli vsan trace set --logtosyslog=false       
  •      
     
     
      3: Steps for ESX Host 3:        
       
        Run the following command on DD for ESX Host 3:
     
           
  •           mtree create /data/col1/esx3-logs quota-hard-limit 2 GiB       
  •        
  •           nfs add /data/col1/esx3-logs 192.168.100.103       
  •      
     
      Run the following commands on ESX Host 3 :      
           
  •           esxcli storage nfs remove -v vsantrace3        
  •        
  •           esxcfg-nas -a vsantrace3 -o 192.168.100.109 -s /data/col1/esx3-logs       
  •        
  •           esxcli vsan trace set -p /vmfs/volumes/vsantrace3       
  •        
  •           esxcli vsan trace set --logtosyslog=false       
  •