Note: This guide covers Rolling Upgrades on Ambari 2.1. You should look through the table at the top of the Ever Better Upgrades post to see which HDP upgrade guide is right for you.

 

 

OneFS now supports rolling upgrades of Hortonworks Data Platform (HDP). Even manual upgrade is now easier. These improvements will be included in the upcoming releases 8.0.0.0 and 7.2.1.2, but for now the functionality is available in patch 164059 for 7.2.1.1. This guide will walk you through the new process, which is a big improvement over the previous procedure documented here.

 

Background

This procedure assumes that you have already installed the patch on 7.2.1.1, or are running a higher build in the same family. There are both manual upgrade and rolling upgrade steps, based on the HDP upgrade guide. Any build of HDP 2.2 or 2.3 can be upgraded from and to. The steps were confirmed on Ambari 2.1.0 with the server installed on CentOS 6.6.

 

In the below steps, these variables appear:

  • $HDP_VERSION_WITH_BUILD - This is the 4-number release version and 4 digit build number of HDP. The three HDP releases of 2.3 that are available today are 2.3.0.0-2557, 2.3.2.0-2950, and 2.3.4.0-3485. This can be determined from the release notes of the HDP build that you are upgrading to, or running the command 'hadoop version' on an existing deployment.
  • $ZONE - This is the name of the OneFS zone where HDFS data is stored.
  • $ONEFSHOST - This is the host name of your OneFS cluster, as registered in Ambari Server.
  • $AMBARIHOSTNAME - This is the host name of the system running Ambari Server.
  • $CLUSTERNAME - This is the name that you assigned to your HDP cluster during installation, which appears on the top left of Ambari's web UI.

 

Configure OneFS

 

Ambari Server expects each agent to send it the full installed HDP package number after completing some commands. Previously OneFS was unable to send that. Starting with this patch and from 7.2.1.2, OneFS has a new setting which stores the value to send to Ambari Server. This is "hdfs-odp-version". In the OneFS CLI set the value with this command:

isi zone zones modify --hdfs-odp-version=$HDP_VERSION_WITH_BUILD --zone $ZONE

 

In some cases Ambari's database is in an invalid state. No entry exists for OneFS in the database table that tracks the state of HDP versions per host. If you have just completed installation of the patch or fixed version of OneFS, first stop and start the HDFS service in Ambari before continuing. Stopping and starting the HDFS service will trigger Ambari to refresh the version for hosts with HDFS components.

 

Rolling Upgrade

Rolling upgrade does not require intervention on each Hadoop cluster machine, and is also non-disruptive. This makes it the easiest way to upgrade the cluster. However, there are a few steps that are necessary because a cluster with OneFS is not configured with the prerequisite NameNode High Availability (NNHA). OneFS provides continuous availability of data and HDFS streams, but does not communicate to Ambari Server as if it has NNHA.

  1. Begin setup and complete installation of packages
  2. Click the Perform Upgrade button (it may have changed to a link named Upgrade: In Progress; if so, click on this), and observe the list of failed pre-upgrade checks. A NameNode High Availability failure is expected. If there is anything else, resolve those issues before continuing.
  3. To skip the NameNode High Availability check, navigate to the Ambari Server experimental features WebUI at http://$AMBARIHOSTNAME:8080/#/experimental/ .
  4. Uncheck the "preUpgradeCheck" option
    Disable-preUpgradeCheck.JPG.jpg
  5. Complete through steps 1.4 or 3.5 of the HDP upgrade guide. Continue with steps 1.5 or 3.5
  6. After upgrade starts, run the following commands on the Ambari Server host's command line, or any machine that has access to it:
curl http://$AMBARIHOSTNAME:8080/api/v1/clusters/$CLUSTERNAME/hosts/$ONEFSHOST/host_components/SECONDARY_NAMENODE -u admin:admin -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo": {"context" :"Stop SNN via REST"}, "Body": {"HostRoles": {"state": "INSTALLED"}}}'

curl http://$AMBARIHOSTNAME:8080/api/v1/clusters/$CLUSTERNAME/hosts/$ONEFSHOST/host_components/SECONDARY_NAMENODE -u admin:admin -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo": {"context" :"Start SNN via REST"}, "Body": {"HostRoles": {"state": "STARTED"}}}'
  1. After the upgrade completes, press the finalize button.

Your upgrade is now complete.

 

A couple of notes. First, if the upgrade needs to be aborted mid-stream, in order to downgrade first set the odp-version in OneFS back to the old version and then complete the downgrade process in HDP.

 

Also, if you waited too long to execute the curl commands, they will be Accepted but enter a paused state in Ambari Server. When you click the Finalize button it will result in a failure such as "Failed on: Save Cluster state". Click the dismiss button on this dialog, and then click on the "op" balloon on the top of the Ambari UI. You may see the operations that were paused, and they will complete without any further interaction. Choose the "Upgrade: Action Required" button in the top middle of the Ambari UI and then the Retry button.

 

Manual Upgrade

There are times where you must manually upgrade your cluster. For those cases, here are the new, simpler steps. It does not require the use of the Ambari experimental feature, but is more labor intensive on each HDP host.

  1. For maintenance releases, eg upgrading from 2.3.0.0 to 2.3.2.0:
    1. Complete through 4.4 step 5 of the HDP upgrade guide
    2. Do not perform 4.4 step 6 on OneFS, but do it on other Hadoop hosts
    3. Complete the remaining tasks including starting each service
  2. For major version upgrades, eg from 2.2.6 to 2.3.0:
    1. Complete through 2.3.5 step 1
    2. Skip 2.3.5 steps 2 to 8
    3. Resume from 2.3.5 step 9
    4. Do not perform 2.4.1 on OneFS, but do it on other Hadoop hosts
    5. In 2.4.4, skip steps 1 to 7. 8(c) is irrelevant. Skip steps 9 to 13
    6. Resume and complete service upgrade tasks
    7. Skip 2.4.20
    8. Complete the upgrade

Your upgrade is now complete.

 

Isilon will continue to work with the Apache community as new features such as Express Upgrade are released to improve the upgrade experience.