The release of OneFS 8.1.1.0 sees the native integration of OneFS support for Cloudera Navigator into the base operating system without the need for any patch to support the Navigator features. https://support.emc.com/docu87518_Isilon-OneFS-8.1.1-Release-Notes.pdf?language=en_US

 

For additional information on Isilon and Navigator integration see the following post: https://community.emc.com/community/products/isilon/blog/2017/10/02/onefs-and-cloudera-navigator-support

This post will review the steps for enabling Navigator integration with OneFS 8.1.1.0 with CDH 5.13 or greater. OneFS 8.1.1.0 will need to be in a committed state to enable the FSImage and INotify functions to be enabled.

 

 

 

1. If Cloudera Navigator is not enabled within Cloudera Manager, install the components. But do not start the Metadata Server (if it is already installed stop the Metadata Server)

 

1.png

 

 

2. Enable FSImage and INotify on the CDH Access Zone, OneFS 8.1.1.0 now has these features added to the WebUI.

 

2.png

 

 

or

 

isi hdfs inotify settings modify --enabled=true --zone=zone2-cdh --verbose

Updated HDFS INotify settings:

enabled: False -> True


isilon01-1# isilon01-1# isi hdfs fsimage settings modify --enabled=true --zone=zone2-cdh --verbose

Updated HDFS FSImage settings:

enabled: False -> True



isilon01-1# isi hdfs inotify settings view --zone=zone2-cdh

      Enabled: Yes

Maximum Delay: 1m

    Retention: 2D


isilon01-1# isi hdfs fsimage settings view --zone=zone2-cdh

Enabled: Yes

 

 

 

3. Review and Modify the Navigator Configuration.

 

Since the Isilon Service is enabled in Cloudera Manager and no HDFS service is present, Navigator is configured for Isilon integration automatically.

3.png

If the Cloudera cluster is Kerberized review the following procedure:


We need to modify the Kerberos Principal that the Metadata Server connects to Isilon as; since the FSimage file and INotify logs are stored outside of the Hadoop root within OneFS, the Principal is required to access the cluster as root to gain access to these log files in a protected part of the ifs file system.


The default account of hue (as seen here) will throw the following errors in the logs.

2-5.png

 

In the Isilon hdfs.log:

 

2018-01-17T16:34:23-05:00 <30.7> isilon01-1 hdfs[3299]: [hdfs] Initializing Connection context AuthType: 81, EffectiveUser: hue/centos-06.foo.com@FOO.COM, RealUser: hue/centos-06.foo.com@FOO.COM

2018-01-17T16:34:23-05:00 <30.6> isilon01-1 hdfs[3299]: [hdfs] ImageTransfer: code 403 error Access denied: user=hue/centos-06.foo.com@FOO.COM desired=1179785 available=1048704 path="/onefs_hdfs/ifs/.ifsvar/modules/hdfs_d/fsimage/3/42"

2018-01-17T16:34:24-05:00 <30.6> isilon01-1 hdfs[3299]: [hdfs] ImageTransfer: code 403 error Access denied: user=hue/centos-06.foo.com@FOO.COM desired=1179785 available=1048704 path="/onefs_hdfs/ifs/.ifsvar/modules/hdfs_d/fsimage/3/42"

 

In the Navigator Metadata Server log

 

4:36:37.194 PM        ERROR        HdfsExtractorShim

[CDHExecutor-0-CDHUrlClassLoader@5aafd51a]: Internal Error while extracting

java.lang.RuntimeException: java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: http://isilon01-cdh.foo.com:8082/imagetransfer?getimage=1&txid=latest&user.name=hue/centos-06.foo.com@FOO.COM, status: 403, message: Forbidden

at com.cloudera.nav.hdfs.extractor.HdfsImageExtractor.doImport(HdfsImageExtractor.java:101)

at com.cloudera.nav.hdfs.extractor.HdfsExtractorShim$1.run(HdfsExtractorShim.java:296)

at com.cloudera.nav.hdfs.extractor.HdfsExtractorShim$1.run(HdfsExtractorShim.java:293)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)

at com.cloudera.cmf.cdh5client.security.UserGroupInformationImpl.doAs(UserGroupInformationImpl.java:44)

at com.cloudera.nav.hdfs.extractor.HdfsExtractorShim.doImport(HdfsExtractorShim.java:293)

at com.cloudera.nav.hdfs.extractor.HdfsExtractorShim.doExtraction(HdfsExtractorShim.java:248)

at com.cloudera.nav.hdfs.extractor.HdfsExtractorShim.run(HdfsExtractorShim.java:144)

at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:221)

at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:211)

at com.cloudera.cmf.cdhclient.CdhExecutor$CallableWrapper.doWork(CdhExecutor.java:236)

at com.cloudera.cmf.cdhclient.CdhExecutor$SecurityWrapper$1.run(CdhExecutor.java:189)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)

at com.cloudera.cmf.cdh5client.security.UserGroupInformationImpl.doAs(UserGroupInformationImpl.java:44)

at com.cloudera.cmf.cdhclient.CdhExecutor$SecurityWrapper.doWork(CdhExecutor.java:186)

at com.cloudera.cmf.cdhclient.CdhExecutor$1.call(CdhExecutor.java:125)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: http://isilon01-cdh.foo.com:8082/imagetransfer?getimage=1&txid=latest&user.name=hue/centos-06.foo.com@FOO.COM, status: 403, message: Forbidden

at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.doGetUrl(TransferFsImage.java:425)

at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:415)

at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.downloadMostRecentImageToDirectory(TransferFsImage.java:98)

at org.apache.hadoop.hdfs.tools.DFSAdmin$1.run(DFSAdmin.java:856)

at org.apache.hadoop.hdfs.tools.DFSAdmin$1.run(DFSAdmin.java:853)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)

at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:477)

at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:471)

at org.apache.hadoop.hdfs.tools.DFSAdmin.fetchImage(DFSAdmin.java:853)

at com.cloudera.cmf.cdh5client.hdfs.DFSAdminImpl.fetchImage(DFSAdminImpl.java:29)

at com.cloudera.nav.hdfs.extractor.HdfsImageFetcherImpl.fetchImage(HdfsImageFetcherImpl.java:15)

at com.cloudera.nav.hdfs.extractor.HdfsImageExtractor.doImport(HdfsImageExtractor.java:81)

... 23 more

Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: http://isilon01-cdh.foo.com:8082/imagetransfer?getimage=1&txid=latest&user.name=hue/centos-06.foo.com@FOO.COM, status: 403, message: Forbidden

at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:286)

at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)

at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:212)

at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:220)

at org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:161)

at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.doGetUrl(TransferFsImage.java:422)

... 36 more

 

 

 

In order to provide access to the fsimage file that resides under /ifs/.ifsvar/modules/hdfs_d/fsimage we need to modify the Kerberos principal that will access the fsimage files. The simplest modification is to use the existing hdfs principal. Since a hdfs => root mapping already exists within the Isilon Access Zone, modifying the Navigator configuration to use the hdfs principal will provide access and allow Navigator to query fsimage correctly.

 

 

3-5.png

 

 

4. Having modified the Navigator principal, start the Navigator MetaDataServer. Initially, it will take time for the different entities to show up as navigator catalogs and links different services into Navigator.

 

Only the cluster is present initially.

4.png

 

Impala added

5.png

 

Hive and Yarn added

7.png

 

The HDFS service(Isilon) often is the last to show up in Navigator based on the polling intervals set and how frequently the FSImage and INotify logs are accessed. Once it shows up HDFS file system events will be present. Create some test files and run some test jobs to validate functionality.

 

8.png

 

 

It may also take additional time for the lineage to link hdfs files and operations, based on how Navigator is configured.

 

9.png

 

 

To recap the Best Practices for using Navigator with Isilon:

- Enable FSImage and INotify prior to Cloudera deployment of Navigator

- There is no known need to adjust the duration of HDFS FSImage and INotify jobs unless instructed to by support

- Do not toggle FSimage and INotify on & off on a zone, once set leave enabled

- Do not enable FSimage and INotify on zones that do not support Cloudera and Navigator

 

 

 

 

 

Russ Stevenson

Isilon

Using Hadoop with Isilon - Isilon Info Hub