This article walks through the upgrade of a deployment of Hortonworks Data Platform (HDP) using Ambari 2.2 when Isilon OneFS is used as the data storage for HDFS.
Choosing an upgrade guide
This guide applies to OneFS 220.127.116.11 or higher, OneFS 18.104.22.168 or higher, and later releases of OneFS.
There have been two previous articles on this topic. Last November we described how to upgrade HDP, which required manual changes to work. Use that guide if you are stuck on OneFS 22.214.171.124 or lower. In January OneFS was updated, making the process much easier. Use that guide if you are not yet on OneFS 126.96.36.199 or 188.8.131.52, or if you are not ready to upgrade to Ambari 2.2.
A table will help illustrate.
|2.2 or higher||Current page|
|7.2.1 family equal or greater than 184.108.40.206||2.1 family. But, not 2.1.2 because of JIRA 13414||January 2016 guide|
|220.127.116.11 (with patch) or 18.104.22.168||2.1 family||November 2015 guide|
|22.214.171.124 or 126.96.36.199||2.1.0||No upgrade possible to HDP 2.3|
Ambari 2.2 mainstreamed the previously experimental skip of service checks that we used in the January 2016 guide. Also Ambari 2.2 introduced a new workflow, Express Upgrade, which speeds up the process when there is no problem for your data scientists if you disrupt jobs. This guide describes the improved Rolling Upgrade experience, and what to expect from Express Upgrade.
OneFS still needs to report to Ambari Server that it is at the target version. This step is unchanged. Here's how I described target version last time:
"This is the 4-number release version and 4 digit build number of HDP. The three HDP releases of 2.3 that are available today are 188.8.131.52-2557, 184.108.40.206-2950, and 220.127.116.11-3485. This can be determined from the release notes of the HDP build that you are upgrading to, or running the command 'hadoop version' on an existing deployment."
Currently we're at 18.104.22.168-258. Here's how to make the change:
isi hdfs settings modify --odp-version=[version] --zone=[zone]
Begin Ambari's upgrade steps and complete installation of packages. Click the Perform Upgrade button (it may have changed to a link named Upgrade: In Process; if so, click on this).
The Upgrade Options box will open:
Click the blue link to take a look at list of failed pre-upgrade checks. There will be two Requirements that fail because of the OneFS host configuration. Ambari expects NameNode High Availability enabled, and expects no Secondary NameNode. Those are okay because OneFS provides continuous availability internally. Hadoop's NNHA and Secondary NameNode are not truly applicable. You can ignore these errors.
If the odp-version setting in OneFS was not previously set, this warning may appear. It can be ignored as well. It could also be corrected by setting odp-version on OneFS to the initial HDP version and then restarting the HDFS service in Ambari.
If there are any other required checks or warning messages, resolve those issues before continuing.
Back at the Upgrade Options, select the "Skip all Service Check failures" checkbox, then choose Rolling Upgrade and Proceed.
There may be other actions necessary to complete HDP installation, but the OneFS and HDFS aspects will be complete.
No special instructions are needed for HDP upgrade using Express Upgrade with OneFS in the cluster. You may encounter the warning about the HDFS components not having version stamps, but that can be ignored.
Work through the HDP upgrade wizard and a satisfying "Upgrade finished" dialog is your reward!