Isilon OneFS and Hadoop Known Issues

The following are known issues that exist with OneFS and Hadoop HDFS integrations:

 

 

 

 

May 2019

When using Ambari 2.7 and the Isilon Management Pack, the following is seen in the Isilon hdfs.log:


isilon-3: 2019-05-14T14:34:06-04:00 <30.4> isilon-3 hdfs[95183]: [hdfs] Ambari: Agent for zone 12 got a bad exit code from its Ambari server. The agent will attempt to recover.

isilon-3: 2019-05-14T14:34:06-04:00 <30.6> isilon-3 hdfs[95183]: [hdfs] Ambari: The Ambari server for zone 12 is running a version unsupported by OneFS: 2.7.1.0. Agent will reset and retry until a supported Ambari server version is installed or Ambari is no longer enabled for this zone

isilon-3: 2019-05-14T14:35:12-04:00 <30.4> isilon-3 hdfs[95183]: [hdfs] Ambari: Agent for zone 12 got a bad exit code from its Ambari server. The agent will attempt to recover.

isilon-3: 2019-05-14T14:35:12-04:00 <30.6> isilon-3 hdfs[95183]: [hdfs] Ambari: The Ambari server for zone 12 is running a version unsupported by OneFS: 2.7.1.0. Agent will reset and retry until a supported Ambari server version is installed or Ambari is no longer enabled for this zone

isilon-3: 2019-05-14T14:36:17-04:00 <30.4> isilon-3 hdfs[95183]: [hdfs] Ambari: Agent for zone 12 got a bad exit code from its Ambari server. The agent will attempt to recover.

isilon-3: 2019-05-14T14:36:17-04:00 <30.6> isilon-3 hdfs[95183]: [hdfs] Ambari: The Ambari server for zone 12 is running a version unsupported by OneFS: 2.7.1.0. Agent will reset and retry until a supported Ambari server version is installed or Ambari is no longer enabled for this zone

 

When using Ambari with the Isilon Management Pack, Isilon should not be configured with an Ambari Server or ODP version as they are no longer needed since the Management Pack is in use.

 

If they have been added, remove them from the Isilon hdfs configuration for the zone in question, this only applied to Ambari 2.7 with the Isilon Management Pack, Ambari 2.6 and earlier still require these settings.

 

 

isilon-1# isi hdfs settings view --zone=zone-hdp27

                 Service: Yes

      Default Block Size: 128M

   Default Checksum Type: none

     Authentication Mode: kerberos_only

          Root Directory: /ifs/zone/hdp27/hadoop-root

         WebHDFS Enabled: Yes

           Ambari Server: -

         Ambari Namenode: hdp-27.foo.com

             Odp Version: -

    Data Transfer Cipher: none

Ambari Metrics Collector: centos-01.foo.com

 

 

 

January 2019

Ambari sees LDAPS issue connecting to AD during Kerberization


05 Apr 2018 20:05:14,081 ERROR [ambari-client-thread-38] KerberosHelperImpl:2379 - Cannot validate credentials: org.apache.ambari.server.serveraction.kerberos.KerberosInvalidConfigurationException: Failed to connect to KDC - Failed to communicate with the Active Directory at ldaps://rduvnode217745.west.isilon.com/DC=AMB3,DC=COM: simple bind failed: rduvnode217745.west.isilon.com:636

Make sure the server's SSL certificate or CA certificates have been imported into Ambari's truststore.

 

1.png

 

Review the following KB from Hortonworks on resolving this Ambari Issue:

 

https://community.hortonworks.com/content/supportkb/148572/failed-to-connect-to-kdc-make-sure-the-servers-ssl.html

 

 

 

 

October 2018

HDFS rollup patch for 8.1.2 - Patch-240163:

 

Patch for OneFS 8.1.2.0. This patch addresses issues with the Hadoop Distributed File System (HDFS).

 

********************************************************************************

This patch can be installed on clusters running the following OneFS version:

8.1.2.0

 

This patch deprecates the following patch:

Patch-236288

 

This patch conflicts with the following patches:

Patch-237113

Patch-237483

 

If any conflicting or deprecated patches are installed on the cluster, you must

remove them before installing this patch.

 

********************************************************************************

RESOLVED ISSUES

 

* Bug ID 240177

  The Hadoop Distributed File System (HDFS) rename command did not manage file

  handles correctly and might have caused data unavailability with

  STATUS_TOO_MANY_OPEN_FILES error.

 

* Bug ID 236286

  If a OneFS cluster had the Hadoop Distributed File System (HDFS) configured for Kerberos authentication, WebHDFS requests over curl might have failed to follow a redirect request.

 

 

 

September 2018

WebHDFS issue with Kerberized 8.1.2 - curl requests fail to follow redirects; Service Checks and Ambari Views will fail


 

 

 

August 2018

Isilon HDFS error: STATUS_TOO_MANY_OPENED_FILES causes jobs to fail

 

 

 

 

December 2017

OneFS 8.0.0.X and Cloudera Impala 5.12.X: Impala queries fail with `WARNINGS: TableLoadingException: Failed to load metadata for table: <tablename> , CAUSED BY: IllegalStateException: null`

 

 

 

October 2017

Ambari agent fails to send heartbeats to Ambari server if agent is running on a NANON node

 

 

 

September 2017

NameNode gives out any IP addresses in an access zone, even across pools and subnets; client connection may fail as a result

 

 

 

 

 

Other Known Issues


1. Host Registrations fails on RHEL 7 hosts with openssl issues

 

Modify the ambari-agent.ini file:

/etc/ambari-agent/conf/ambari-agent.ini

 

[security]

force_https_protocol=PROTOCOL_TLSv1_2

 

 

Restart the ambari-server and all ambari-agents

https://community.hortonworks.com/questions/145/openssl-error-upon-host-registration.html




2. Installs fail when using Private Repos with Ambari

Isilon presents itself as a Centos06 host and Ambari expects to see the centos6 repos, without internet access to the public repos installs will fail if a local Centos6 repo is not available and accessible <KB coming on this issue shortly>


 

 

 

 

Using Hadoop with Isilon - Isilon Info Hub

Isilon