Find Communities by: Category | Product

If Cloudera 5.7 was installed in a process similar to the sequence described in: https://community.emc.com/community/products/isilon/blog/2016/07/07/cloudera-and-isilon-implementation-part-2

 

Then the Impala Service may not start or stay running correctly, a few additional configuration changes will need to be made to get this service running. Also take a look at the Cloudera doc: Using Impala with Isilon Storage

 

 

1.png

 

On further review, the Impala Daemon will not stay started correctly.

2.png

 

The process will continue to exit post start.

3.png

 

 

On reviewing the Impalad log we see the following info and errors.

 

-Short-circuit reads are not enabled

-block tracking is not properly enabled because of dfs.client.file-block-storage-locations.timeout.millis timeout

 

 

5.png

 

 

In order to correct these, from Cloudera Manager search for: dfs_client_read_shortcircuit

 

And select the Isilon : Enable HDFS Short-Circuit Reads

 

6.png

 

1.

Enable HDFS Short-Circuit Read on the Gateway Default Group

 

7.png

 

2.

In HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml and the Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml properties for the Isilon service

 

Add a new value of the dfs.client.file-block-storage-locations.timeout.millis property to 10000

 

hdfs-site.xml

8.png

 

9.png

 

 

core-site.xml

10.png

 

Also add In the Isilon Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml property for the Isilon service

 

3.

Set the value of the hadoop.security.token.service.use_ip property to FALSE.

11.png

 

Save and return to Cloudera Manager Dashboard, since configuration changes have been made we need to redeploy configuration and restart services.

 

12.png

 

Select, to Deploy Client Configuration

13.png

 

 

Deploy,

14.png

 

 

Monitor the deploy

15.png

 

 

We now need to restart the services, affected by the configuration redeployment

 

16.png

 

Restart all the services,

17.png

 

 

Restart,

18.png

 

 

Monitor the restarts,

19.png

 

 

 

You may still see alarms on services based on prior alert triggers

20.png

 

The alarms can be viewed for their status,

21.png

 

 

The alarms will timeout and the status of Impala will return to healthy green.

 

22.png

 

The Impala service is now started and operational.

 

 

 

russ_stevenson

Isilon

Using Hadoop with Isilon - Isilon Info Hub

This blog post continues where Cloudera and Isilon Implementation Part 1 leaves off:

 

 

In order to deploy a Hadoop Cluster with Isilon, we need to select the Custom Services install, this allows us to select the required components.

 

z.png

 

Initially no services are selected,

aa.png

 

Select the Hadoop services you wish to deploy;

 

DO NOT SELECT HDFS, since we are deploying Isilon as the storage & effective NN we don't need the Cloudera HDFS Service.

 

MapReduce is also not recommended, as MapReduce2 is included in Yarn

 

But, if your application is legacy and not written for Yarn MRv1 can be enabled and used.

bb.png

 

Having selected the services continue and assign roles. Since this is a single host all roles are deployed on the same host. Consult the Cloudera documentation for best practices regarding role assignments.

 

cc.pngdd.pngee.png

 

Leave the Isilon role, as default. No need to change it.

ff.png

 

Continue,

 

gg.png

 

hh.png

 

Again this is test host, so all defaults are selected, consult the Cloudera documentation for best practices regarding database.

Test the Database Connections,

ii.png

 

Continue,

 

On the Cluster Setup page, we need to assign the Isilon to the following two parameters, note the ports are different.

 

default_fs_name        hdfs://smartconnectzonename:8020

webhdfs_url                http://smartconnectzonename:8082/webhdfs/v1

 

jj.pngkk.png

ll.png

mm.png

nn.png

 

Assign the two Isilon paramters

oo.png

 

All other settings can be left as default currently,

 

Continue,

 

Cluster Setup will start,

pp.png

 

Review the setup as it runs,

 

qq.png

 

rr.png

Setup will complete, additional details can be seen by opening the specific services

 

The setup will complete,

ss.png

 

Continue, the Hadoop cluster deployment has finished.

tt.png

uu.png

 

Finish and return to the main Cloudera Manager dashboard, review the status

 

 

 

It is not uncommon to see alarms and service down issues on the dashboard. Review the alarms and services and triage as needed. Some services may just need restarting, follow standard protocols in starting these services.

-start the service

-monitor and review logs as needed

-reviewing the Isilon /var/log/hdfs.log  (remember Isilon is a clustered system, so all node logs need reviewing)

 

123.png

 

 

It is also common to see Configuration issues, address and make the required changes as needed to resolve each issue.

124.png

 

Restart services to resolve Alarms following configuration changes.

 

234.png

 

 

The Hadoop cluster and Services are now fully operational and can be tested.

 

vv.png

 

Basic functionality can be tested.

 

Browse the Isilon HDFS root:

hadoop fs -ls /

ww.png

 

Write to the Isilon HDFS root:

xx.png,

 

Run some basic smoketest jobs; PI or teragen/teravalidate/terasort to test mapreduce.

 

xxx.png

 

 

 

With Cloudera 5.7, you may notice that Impala service is not started fully, some additional configuration changes are needed to get this Service started.

 

345.png

 

 

 

The steps needed to get Impala running can be found here  ---- >  Get Cloudera 5.7 Impala starting with Isilon

 

 

Hopefully this doc gives the high level overview of getting Cloudera CDH up and running against Isilon OneFS.

 

 

 

 

russ_stevenson

Isilon

Using Hadoop with Isilon - Isilon Info Hub

The following post continues a series of high level overview posts on Isilon and Hadoop implementations. It provides the core tasks needed to complete the setup and get a basic operational Hadoop cluster running with Isilon, additional topics will be covered later or in upcoming documents. Since the steps to this process are long, I'll break this post up into two parts.

 

 

 

 

This procedure is based on the following:

Isilon OneFS: 8.0.0.1

CDH 5 parcel: 5.7.1-1.cdh5.7.1.p0.11

 

 

OneFS 8.0.0.1 contains a number of updates to facilitate the integration and deployment of hadoop against OneFS, it is highly recommended to use this version. The procedure may requires additional steps prior to 8.0.0.1 not documented in this post.

 

Before installing any Hadoop cluster, the OneFS supportability matrix should be consulted for compatibility: https://community.emc.com/docs/DOC-37101

 

 

This blog assumes the following Isilon Hadoop environment is configured and operational:

 

-Isilon is licensed for HDFS

-A dedicated Isilon Access Zone is in use (not the system zone).

-Isilon HDFS root directory in the Access Zone exists

-The Isilon SmartConnect Zone configuration is implemented per best practice for Isilon HDFS access.

-The Isilon HDFS configuration is correctly configured.

-A simple access model will exists between Hadoop and Isilon; user UID & GID and parity will exist.

 

The best approach to achieving parity is beyond the scope of this post and will be addressed in up coming posts.

 

 

 

Assuming the Isilon is setup and configured for integration with Cloudera, we can begin the deployment of the Cloudera Manager.

 

This post does not address the setup, configuration and deployment of the Linux hosts used to deploy Hadoop services on.  The Cloudera documentation should be consulted to setup and prepare the hosts correctly: Overview of Cloudera and the Cloudera Documentation Set  The post also does not address advanced Cloudera installs, the focus is to highlight the Isilon integration into the installer and how to complete the install.

 

A good overview of the procedure can be found here: Installation Path A - Automated Installation by Cloudera Manager (Non-Production Mode)  This post begins with the download of the bits and installation of CM.

 

 

# wget https://archive.cloudera.com/cm5/installer/latest/cloudera-manager-installer.bin

# chmod u+x cloudera-manager-installer.bin

# ./cloudera-manager-installer.bin

 

0.png

 

On running the installer, you'll get the following:

1.png

Next,

2.png

Next, Accept the Cloudera License

3.png

Yes,

4.png

Next, we will let Cloudera Manager install the JDK

5.png

Yes, Accept the Oracle

6.png

 

7.png

OK, note the URL and the user/pass for the Cloudera Manager WebUI

8.png

OK,

 

You can validate the Cloudera Manager Service is running, if you see problems tail the cloudera-scm-server.log as you start the service.

 

# service cloudera-scm-server status

# cloudera-scm-server (pid  10487) is running...

# tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

 

 

Log in to the Cloudera Manager WebUI;  user:admin, password:admin

a.png

 

Continue,

 

b.png

 

Select the Yes check box to accept the EULA and Continue,

 

c.png

 

Select the version you wish to deploy,

 

d.png

 

Continue,

 

e.png

 

Continue,

 

 

In this post we will deploy to just a single Linux host, but the process is the same when multiple hosts are used in the Hadoop cluster.

 

f.png

 

Add the FQDN of the Linux hosts to be deployed, Search,

 

g.png

 

On completion of the search, select the host(s) to deploy to,

 

h.png

 

Select to use Parcels

Select the CDH Stack you wish to deploy

 

i.png

 

Select the Additional Parcels and Agent configuration as needed and what is supported by Isilon,

 

j.png

 

Continue,

k.png

 

Select install the JDK,

Select install the JUSEP files is you intended to secure this cluster later, Continue,

 

l.png

 

We will not deploy in Single User Mode,  Continue

 

m.png

 

Provide the SSH credentials, either root password or SSH keys depending on how you set your Linux hosts up and wish to manage them, Continue

 

The installation will begin

 

n.png

 

Details,

o.png

 

Installation completes and the installer continues,

 

p.png

 

Parcels being downloaded,

 

q.png

 

Parcels being distributed

 

r.png

 

Parcels unpacked and activated

 

s.png

 

The host inspector will then validate hosts, versions and additional software installed

 

t.png

 

 

The installer will check and validate the hosts, if any deviations are seen recommendation are presented to optimize the hosts. If the validation checker fails it is suggested to follow the recommendations and then re-try the validation.

 

Common errors are seen with:

- transparent_hugepage

- swappiness

 

make the recommended changes to hosts and Run Again,

 

u.pngv.pngw.pngx.pngy.png

 

Finish,

 

 

This completes part 1 of the install, deploying Hadoop services with Cloudera Manager is continued in Part 2.

 

Cloudera and Isilon Implementation Part 2  ---->

 

 

russ_stevenson

Isilon

Using Hadoop with Isilon - Isilon Info Hub

Filter Blog

By date:
By tag: