|Article Number: 531735||Article Version: 3||Article Type: Break Fix|
Integrated Data Protection Appliance Family,Integrated Data Protection Appliance Software,Integrated Data Protection Appliance,Integrated Data Protection Appliance 2.3
In the Upgrade utility log (/data01/tmp/patch/logs/upgrade-utility.log), the following errors can be seen:
2019-03-15 23:24:12,150 ERROR [upgrade-workflow-5]-dps.DpsUpgradeOperations: Failed to upgrade DPS. responseCode: 1
2019-03-15 23:24:12,150 ERROR [upgrade-workflow-5]-dps.DpsRetryTask: Failed to upgrade DPS.
com.emc.vcedpa.upgradeutil.common.exception.UpgradeException: Unable to upgrade Search.
In dps_upgrade.log, we see the following error:
15/03/19 23:23:11 check_status_on_master() Upgrade is still in progress. This may take some time. Please wait
15/03/19 23:23:11 check_status_on_master() IDPA proceeding to check the version of Search Node.
15/03/19 23:24:11 run_cmd_dps_output() Cmd:ls -l /etc/puppet/status/
15/03/19 23:24:11 run_cmd_dps_output() total 0
15/03/19 23:24:11 run_cmd_dps_output() Status: 0
15/03/19 23:24:12 run_cmd_dps_output() Cmd:cat /usr/local/search/version
15/03/19 23:24:12 run_cmd_dps_output() 126.96.36.1995
15/03/19 23:24:12 run_cmd_dps_output() Status: 0
15/03/19 23:24:12 check_search_version_on_master() 188.8.131.525
15/03/19 23:24:12 check_search_version_on_master() Could not verify Search version
The issue is with the puppet service on Search not being able to run upgrade.sh script and it fails with the following error.
The puppet_agent.log in the IDPA log bundle can be verified to confirm the failure:
2018-10-24T17:14:39,403|slab-dp4400-dpsim01|ERROR|upgrade.sh:(line 45)|Current version is empty!!! Stop upgrade and wait for fixing......
2019-03-15T20:27:05,813|slab-dp4400-dpsim01|WARN|upgrade.sh:(line 180)|No upgradable package, skip.
2019-03-15T20:32:05,824|slab-dp4400-dpsim01|WARN|upgrade.sh:(line 180)|No upgradable package, skip.
2019-03-16T03:18:12,344|slab-dp4400-dpsim01|ERROR|upgrade.sh:(line 310)|Disable ES shard allocation failed with exit code: 1!
2019-03-16T03:23:12,361|slab-dp4400-dpsim01|ERROR|upgrade.sh:(line 310)|Disable ES shard allocation failed with exit code: 1!
1. Login to the ACM GUI. service puppetmaster status service puppet status puppet agent -t
2. Click on 'Retry' for the DPS (Search) component.
3. As soon as the rollback is done for search, login to the Search Master node.
4. Verify the elasticsearch service is running using following command:
service elasticsearch status
If the service is stopped, restart the service.
service elasticsearch start
5. Verify the puppet and puppetmaster service is running.
6. Force the puppet agent to perform the upgrade on Search.
7. Monitor the Search Upgrade in the ACM GUI for completion.
service puppetmaster status
service puppet status
puppet agent -tNote: This step needs to be done fairly quickly after restarting the Search Upgrade. We will get around 3-4 minutes until the Upgrade times out.