The following post continues a series of high level overview posts on Isilon and Hadoop implementations. It provides the core tasks needed to complete the setup and get a basic operational Hadoop cluster running with Isilon, additional topics will be covered later or in upcoming documents. Since the steps to this process are long, I'll break this post up into two parts.

 

 

 

 

This procedure is based on the following:

Isilon OneFS: 8.0.0.1

CDH 5 parcel: 5.7.1-1.cdh5.7.1.p0.11

 

 

OneFS 8.0.0.1 contains a number of updates to facilitate the integration and deployment of hadoop against OneFS, it is highly recommended to use this version. The procedure may requires additional steps prior to 8.0.0.1 not documented in this post.

 

Before installing any Hadoop cluster, the OneFS supportability matrix should be consulted for compatibility: https://community.emc.com/docs/DOC-37101

 

 

This blog assumes the following Isilon Hadoop environment is configured and operational:

 

-Isilon is licensed for HDFS

-A dedicated Isilon Access Zone is in use (not the system zone).

-Isilon HDFS root directory in the Access Zone exists

-The Isilon SmartConnect Zone configuration is implemented per best practice for Isilon HDFS access.

-The Isilon HDFS configuration is correctly configured.

-A simple access model will exists between Hadoop and Isilon; user UID & GID and parity will exist.

 

The best approach to achieving parity is beyond the scope of this post and will be addressed in up coming posts.

 

 

 

Assuming the Isilon is setup and configured for integration with Cloudera, we can begin the deployment of the Cloudera Manager.

 

This post does not address the setup, configuration and deployment of the Linux hosts used to deploy Hadoop services on.  The Cloudera documentation should be consulted to setup and prepare the hosts correctly: Overview of Cloudera and the Cloudera Documentation Set  The post also does not address advanced Cloudera installs, the focus is to highlight the Isilon integration into the installer and how to complete the install.

 

A good overview of the procedure can be found here: Installation Path A - Automated Installation by Cloudera Manager (Non-Production Mode)  This post begins with the download of the bits and installation of CM.

 

 

# wget https://archive.cloudera.com/cm5/installer/latest/cloudera-manager-installer.bin

# chmod u+x cloudera-manager-installer.bin

# ./cloudera-manager-installer.bin

 

0.png

 

On running the installer, you'll get the following:

1.png

Next,

2.png

Next, Accept the Cloudera License

3.png

Yes,

4.png

Next, we will let Cloudera Manager install the JDK

5.png

Yes, Accept the Oracle

6.png

 

7.png

OK, note the URL and the user/pass for the Cloudera Manager WebUI

8.png

OK,

 

You can validate the Cloudera Manager Service is running, if you see problems tail the cloudera-scm-server.log as you start the service.

 

# service cloudera-scm-server status

# cloudera-scm-server (pid  10487) is running...

# tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

 

 

Log in to the Cloudera Manager WebUI;  user:admin, password:admin

a.png

 

Continue,

 

b.png

 

Select the Yes check box to accept the EULA and Continue,

 

c.png

 

Select the version you wish to deploy,

 

d.png

 

Continue,

 

e.png

 

Continue,

 

 

In this post we will deploy to just a single Linux host, but the process is the same when multiple hosts are used in the Hadoop cluster.

 

f.png

 

Add the FQDN of the Linux hosts to be deployed, Search,

 

g.png

 

On completion of the search, select the host(s) to deploy to,

 

h.png

 

Select to use Parcels

Select the CDH Stack you wish to deploy

 

i.png

 

Select the Additional Parcels and Agent configuration as needed and what is supported by Isilon,

 

j.png

 

Continue,

k.png

 

Select install the JDK,

Select install the JUSEP files is you intended to secure this cluster later, Continue,

 

l.png

 

We will not deploy in Single User Mode,  Continue

 

m.png

 

Provide the SSH credentials, either root password or SSH keys depending on how you set your Linux hosts up and wish to manage them, Continue

 

The installation will begin

 

n.png

 

Details,

o.png

 

Installation completes and the installer continues,

 

p.png

 

Parcels being downloaded,

 

q.png

 

Parcels being distributed

 

r.png

 

Parcels unpacked and activated

 

s.png

 

The host inspector will then validate hosts, versions and additional software installed

 

t.png

 

 

The installer will check and validate the hosts, if any deviations are seen recommendation are presented to optimize the hosts. If the validation checker fails it is suggested to follow the recommendations and then re-try the validation.

 

Common errors are seen with:

- transparent_hugepage

- swappiness

 

make the recommended changes to hosts and Run Again,

 

u.pngv.pngw.pngx.pngy.png

 

Finish,

 

 

This completes part 1 of the install, deploying Hadoop services with Cloudera Manager is continued in Part 2.

 

Cloudera and Isilon Implementation Part 2  ---->

 

 

russ_stevenson

Isilon

Using Hadoop with Isilon - Isilon Info Hub