IDPA Upgrade to 2.3 fails on Search with error "check_search_version_on_master() Could not verify Search version"

           

   Article Number:     531735                                   Article Version: 3     Article Type:    Break Fix 
   

 


Product:

 

Integrated Data Protection Appliance Family,Integrated Data Protection Appliance Software,Integrated Data Protection Appliance,Integrated Data Protection Appliance 2.3

 

Issue:

 

 

In the Upgrade utility log (/data01/tmp/patch/logs/upgrade-utility.log), the following errors can be seen:     
     
      2019-03-15 23:24:12,150 ERROR [upgrade-workflow-5]-dps.DpsUpgradeOperations: Failed to upgrade DPS. responseCode: 1       
        2019-03-15 23:24:12,150 ERROR [upgrade-workflow-5]-dps.DpsRetryTask: Failed to upgrade DPS.       
        com.emc.vcedpa.upgradeutil.common.exception.UpgradeException: Unable to upgrade Search.       
                at com.emc.vcedpa.upgradeutil.configure.dps.DpsUpgradeOperations.upgrade(DpsUpgradeOperations.java:139)       
                at com.emc.vcedpa.upgradeutil.configure.dps.DpsRetryTask.run(DpsRetryTask.java:68)       
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)       
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)       
                at java.lang.Thread.run(Thread.java:748)
     
     
      In dps_upgrade.log, we see the following error:     
     
      15/03/19 23:23:11 check_status_on_master() Upgrade is still in progress. This may take some time. Please wait       
        15/03/19 23:23:11 check_status_on_master() IDPA proceeding to check the version of Search Node.       
        15/03/19 23:24:11 run_cmd_dps_output() Cmd:ls -l /etc/puppet/status/       
        15/03/19 23:24:11 run_cmd_dps_output() total 0       
        15/03/19 23:24:11 run_cmd_dps_output() Status: 0       
        15/03/19 23:24:12 run_cmd_dps_output() Cmd:cat /usr/local/search/version       
        15/03/19 23:24:12 run_cmd_dps_output() 18.0.0.1065       
        15/03/19 23:24:12 run_cmd_dps_output() Status: 0       
        15/03/19 23:24:12 check_search_version_on_master() 18.0.0.1065       
        15/03/19 23:24:12 check_search_version_on_master() Could not verify Search version
     
     
      The issue is with the puppet service on Search not being able to run upgrade.sh script and it fails with the following error.       
        The puppet_agent.log in the IDPA log bundle can be verified to confirm the failure:
     
     
      2018-10-24T17:14:39,403|slab-dp4400-dpsim01|ERROR|upgrade.sh:(line 45)|Current version is empty!!! Stop upgrade and wait for fixing......       
       
        2019-03-15T20:27:05,813|slab-dp4400-dpsim01|WARN|upgrade.sh:(line 180)|No upgradable package, skip.       
        2019-03-15T20:32:05,824|slab-dp4400-dpsim01|WARN|upgrade.sh:(line 180)|No upgradable package, skip.       
       
        2019-03-16T03:18:12,344|slab-dp4400-dpsim01|ERROR|upgrade.sh:(line 310)|Disable ES shard allocation failed with exit code: 1!       
        2019-03-16T03:23:12,361|slab-dp4400-dpsim01|ERROR|upgrade.sh:(line 310)|Disable ES shard allocation failed with exit code: 1!
   
   
   
     
                                                           

 

 

Resolution:

 

 

1. Login to the ACM GUI.       
       
        2. Click on 'Retry' for the DPS (Search) component.       
       
        3. As soon as the rollback is done for search, login to the Search Master node.       
       
        4. Verify the elasticsearch service is running using following command:       
        service elasticsearch status       
       
        If the service is stopped, restart the service.       
       
        service elasticsearch start       
       
        5. Verify the puppet and puppetmaster service is running.
   

      service puppetmaster status   

   

      service puppet status   

   
    6. Force the puppet agent to perform the upgrade on Search.   

      puppet agent -t   

Note: This step needs to be done fairly quickly after restarting the Search Upgrade. We will get around 3-4 minutes until the Upgrade times out.        
       
        7. Monitor the Search Upgrade in the ACM GUI for completion.