Note: This guide covers upgrades on Ambari 2.1. You should look through the table at the top of the Ever Better Upgrades post to see which HDP upgrade guide is right for you.
Upcoming releases of OneFS will support Ambari's ability to upgrade HDP across the cluster with rolling upgrades. In the meantime, current OneFS releases require jumping through a few hoops to upgrade HDP from 2.2 to 2.3. This guide will walk you through the process.
There are three current OneFS versions affected: 126.96.36.199, 188.8.131.52, and 184.108.40.206. The upcoming release 220.127.116.11 will be similarly impacted. 18.104.22.168 and 22.214.171.124 would only allow upgrade from one release of HDP 2.2 to another maintenance release, eg 2.2.6 to 2.2.8. OneFS 126.96.36.199 introduced compatibility with Ambari 2.1.0 and HDP 2.3, which is the target of this post.
These steps walk through manual upgrade, based on the HDP upgrade guide. HDP 2.3.0 and 2.3.2 are both possibilities, but because of compatibility requirements of OneFS, Ambari 2.1 must be used. Also the steps can be applied to upgrades to maintenance releases such as 2.2.6 to 2.2.8, but they are specifically written for an upgrade from 2.2 to 2.3 and align with those HDP upgrade steps. The examples assume that postgresql is used for the Ambari database, and the steps were confirmed on Ambari Server installed on CentOS 6.6.
In the below steps, these variables appear:
- $ONEFSHOST - This is the host name of your OneFS cluster, as registered in Ambari Server.
- $STACK - This is the 2-number stack of HDP that you are upgrading to. This is "2.3" currently.
- $HDPVERSION-PACKAGE - This is the 4-number release version and 4 digit package version of HDP. The two HDP releases of 2.3 that are available today are 188.8.131.52-2557 and 184.108.40.206-2950.
- $VERSIONDISPLAYNAME - This is the text that appears in bold type in Ambari Server's version page. For example, "HDP-220.127.116.11".
- $CLUSTERNAME - This is the name that you assigned to your cluster during installation, which appears on the top left of Ambari's web UI
Ensure Correct Pre-Install State
Because OneFS is not sending expected information to Ambari Server, in some cases no entry will be created during installation for it in the database table that tracks the state of HDP versions per host. In this sequence you will check if this is true and correct it if so.
- Enter the Ambari database. From the command line of your Ambari Server machine, run the following command.
psql -U ambari
- When prompted for database password input "bigdata"
- Check to make sure that the OneFS host has a CURRENT entry in the host_version table.
SELECT * from host_version LEFT OUTER JOIN hosts ON host_version.host_id = hosts.host_id WHERE hosts.host_name = '$ONEFSHOST';
- In the results, there should be a line with the OneFS host and the state set to CURRENT. This is what it would look like:
- If you don't see a CURRENT entry for OneFS, then create a new entry in the host_version table. You'll need to pick an ID that is not in use, indicated by $NEWID.
insert into host_version (id, repo_version_id, host_id, state) values ('$NEWID', (select repo_version_id from repo_version where version = '$HDPVERSION-PACKAGE'), (select host_id from hosts where host_name = '$ONEFSHOST'), 'CURRENT');
- Exit psql by typing "\q",
In this section you will have Ambari install packages on all of the HDP hosts, and then edit the database to reflect the state as if OneFS had provided installation results to Ambari Server.
- Begin the HDP manual upgrade steps from the Hortonworks upgrade guide.
- Before the final step of "Installing a New Version" (2.2 step 6), the UI will be hung with with the status "Installing" and the progress bar will be full but blue. Clicking on "Installing" will show that all hosts are complete and green. Ambari Server is waiting to hear version information about successful installed on the OneFS host. It will look like this:
- Stop Ambari Server. From the command line:
- Set the OneFS host and the cluster to INSTALLED. Enter the database as described above. Then enter the following queries:
update host_version set state = 'INSTALLED' where state in ('INSTALLING', 'INSTALL_FAILED'); update cluster_version set state = 'INSTALLED' where state in ('INSTALLING', 'INSTALL_FAILED');
- Exit psql as described above and then start Ambari Server with this command:
- Installation in the Ambari Server version page should complete at this point, transitioning to a "Perform Upgrade" button. Do not press it.
In this section you will follow the "Preparing to Upgrade" section of the HDP upgrade guide, skipping a number of irrelevant steps for HDFS. The Ambari database will need to be updated as if OneFS has completed upgrade. The normal step to complete upgrade will not work, so that is replaced with further database edits and an ambari-server command line action.
- Begin 2.3, "Preparing to Upgrade"
- Complete through 2.3.5 step 1
- Skip 2.3.5 steps 2 to 8.
- Resume from 2.3.5 step 9.
- Do not perform 2.4.1 on OneFS, but do it on other HDP hosts.
- At 2.4.4 step 1, move OneFS components to the new version. First stop Ambari Server from the command line then enter the database as described above. Then enter the following queries:
update hostcomponentstate set upgrade_state = 'COMPLETE' where host_id in (select host_id FROM hosts where host_name = '$ONEFSHOST') ; update hostcomponentstate set version = '$HDPVERSION-PACKAGE' where component_name in ('SECONDARY_NAMENODE', 'NAMENODE', 'DATANODE');
- Exit psql and start Ambari Server again.
- In 2.4.4, skip steps 1 to 7.
- At 8(b), perform Start instead of Restart All.
- 8(c) is irrelevant. Skip steps 9 to 13.
- Resume and complete service upgrade tasks.
- Skip 2.4.20.
- After 2.4.21 step 1, set all components to the current stack, move OneFS host and cluster state to UPGRADED.
update hostcomponentstate set current_stack_id = subquery.stack_id from (select stack_id from stack where stack_version = '$STACK') as subquery; update hostcomponentstate set upgrade_state = 'NONE'; update host_version set state = 'UPGRADED' where state IN ('UPGRADING', 'INSTALLED', 'INSTALLING'); update cluster_version set state = 'UPGRADED' where state IN ('UPGRADING', 'INSTALLED', 'INSTALLING');
- Set the new version to CURRENT.
ambari-server set-current --cluster-name=$CLUSTER --version-display-name=$VERSIONDISPLAYNAME
Your upgrade is now complete. By the time you need to upgrade HDP again, OneFS will support rolling upgrade and the process will be much easier.