IDPA Upgrade to 2.3 fails on Search with error "check_search_version_on_master() Could not verify Search version"


   Article Number:     531735                                   Article Version: 3     Article Type:    Break Fix 




Integrated Data Protection Appliance Family,Integrated Data Protection Appliance Software,Integrated Data Protection Appliance,Integrated Data Protection Appliance 2.3





In the Upgrade utility log (/data01/tmp/patch/logs/upgrade-utility.log), the following errors can be seen:     
      2019-03-15 23:24:12,150 ERROR [upgrade-workflow-5]-dps.DpsUpgradeOperations: Failed to upgrade DPS. responseCode: 1       
        2019-03-15 23:24:12,150 ERROR [upgrade-workflow-5]-dps.DpsRetryTask: Failed to upgrade DPS.       
        com.emc.vcedpa.upgradeutil.common.exception.UpgradeException: Unable to upgrade Search.       
                at com.emc.vcedpa.upgradeutil.configure.dps.DpsUpgradeOperations.upgrade(       
                at java.util.concurrent.ThreadPoolExecutor.runWorker(       
                at java.util.concurrent.ThreadPoolExecutor$       
      In dps_upgrade.log, we see the following error:     
      15/03/19 23:23:11 check_status_on_master() Upgrade is still in progress. This may take some time. Please wait       
        15/03/19 23:23:11 check_status_on_master() IDPA proceeding to check the version of Search Node.       
        15/03/19 23:24:11 run_cmd_dps_output() Cmd:ls -l /etc/puppet/status/       
        15/03/19 23:24:11 run_cmd_dps_output() total 0       
        15/03/19 23:24:11 run_cmd_dps_output() Status: 0       
        15/03/19 23:24:12 run_cmd_dps_output() Cmd:cat /usr/local/search/version       
        15/03/19 23:24:12 run_cmd_dps_output()       
        15/03/19 23:24:12 run_cmd_dps_output() Status: 0       
        15/03/19 23:24:12 check_search_version_on_master()       
        15/03/19 23:24:12 check_search_version_on_master() Could not verify Search version
      The issue is with the puppet service on Search not being able to run script and it fails with the following error.       
        The puppet_agent.log in the IDPA log bundle can be verified to confirm the failure:
      2018-10-24T17:14:39,403|slab-dp4400-dpsim01|ERROR| 45)|Current version is empty!!! Stop upgrade and wait for fixing......       
        2019-03-15T20:27:05,813|slab-dp4400-dpsim01|WARN| 180)|No upgradable package, skip.       
        2019-03-15T20:32:05,824|slab-dp4400-dpsim01|WARN| 180)|No upgradable package, skip.       
        2019-03-16T03:18:12,344|slab-dp4400-dpsim01|ERROR| 310)|Disable ES shard allocation failed with exit code: 1!       
        2019-03-16T03:23:12,361|slab-dp4400-dpsim01|ERROR| 310)|Disable ES shard allocation failed with exit code: 1!






1. Login to the ACM GUI.       
        2. Click on 'Retry' for the DPS (Search) component.       
        3. As soon as the rollback is done for search, login to the Search Master node.       
        4. Verify the elasticsearch service is running using following command:       
        service elasticsearch status       
        If the service is stopped, restart the service.       
        service elasticsearch start       
        5. Verify the puppet and puppetmaster service is running.

      service puppetmaster status   


      service puppet status   

    6. Force the puppet agent to perform the upgrade on Search.   

      puppet agent -t   

Note: This step needs to be done fairly quickly after restarting the Search Upgrade. We will get around 3-4 minutes until the Upgrade times out.        
        7. Monitor the Search Upgrade in the ACM GUI for completion.