In part 1 of this blog: SmartConnect, Network Pools and HDFS Racks for Hadoop Part 1 we looked at how Virtual HDFS racks are defined. A NameNode request is made to a SmartConnect IP pool and OneFS if configured to use racks responds with a DataNode node in the IP pool for the clients source IP.

 

Initially racks were all about creating location aware data node connections but with older releases of OneFS it also became useful to split NameNode and DataNode connection into separate pools to create, in effect better load balancing of data node connections even without location awareness. This was accomplished by using racks.

 

 

  • NameNode IP pool – Static allocation with a SmartConnect Name
  • DataNode IP pool – Dynamic allocation
  • /default rack 0.0.0.0 – 255.255.255.255.0 assigned to a DataNode pool

 

A default rack DataNode pool was created to separate NameNode connections from DataNode connection but all source IP’s were in the rack and all Isilon Nodes were in the DataNode pool. This separation of NameNode connections from DataNode connection pools was more effective at balancing connection efficiently to maximize throughput.

 

NameNode Pool

This pool would be assigned a static allocation IP methodology to limit IP usage and not failover IP if interfaces were unavailable, NameNode connections are short lived and it made no sense to failover the IP addresses. The IP pool had an assigned SmartConnect name for the Hadoop compute cluster to connect to as the default.FS.

 

DataNode Pool

This pool would be assigned a Dynamic IP allocation approach with an appropriate number of IP assigned to the pool to ensure equal failover, similar to NFS. See the external connectivity guide for additional information. No SmartConnect name was required on this pool.


1.png

OneFS 8.0.1.x introduced two new hdfs features which changes the way OneFS handles hdfs connection and these changes impact the recommended approaches to IP pool configurations for hadoop clusters.

 

DataNode Load Balancing

Previously the hdfs service on OneFS used a round robin scheme to provide IP address within a SmartConnect zone for new HDFS client to connect to or it leveraged racks. Typically, this works reasonably well but in highly loaded Hadoop cluster. There was a chance that new HDFS clients will be connected to a node that is already highly loaded with DataNode connections potential creating connectivity skew. In 8.0.1, OneFS is built with the intelligence to ensure that new HDFS client will always be given an IP address where the node’s total number of TCP connection count is the lowest, allowing the client to have the best chance to leverage the least loaded node to get the best performance and throughput. This in effect removes the requirement for racks to achieve better load balancing when location awareness is not required.

 

DataNode Write Recovery

Now with HDFS DataNodes the client has the ability to recovery from node failure or network failures by using a feature called Pipeline Recovery. This feature allows the hadoop clients to continue writing their data to a different node in the event that the current node is unreachable or returns an error for some reason. This feature creates greater resiliency with regard to Hadoop clients and Isilon DataNodes, as in effect we respond with three potential DataNodes for the client to connect to as opposed to one in previous versions of OneFS.

 

The implementation of these features provides better load balancing and resiliency within OneFS and allows us to change our recommended approaches to IP pools and the use of racks. We are no longer dependent on using racks to marshal separate NameNode and DataNode connections to optimize load balancing. Racks are still a completely valid configurations and should still be used to create location aware client DataNode connections if the cluster architecture dictates optimization of rack awareness.

 

The primary considerations when defining anIP strategy are:

  • Availability of IP addresses*
  • OneFS node and interface assignment and use
  • Rack and location of compute clients and OneFS nodes with respect to each other

 

* Any deployed pool strategy is dependent on the appropriate number of IP’s being available for assignment within the pool to meet the requirement of allocation or failover.

 

 

IP Pool Strategies for 8.0.1.x and Greater

 

In most simple deployments or if all the compute clients will access all Isilon node interfaces to maximize throughput and performance. It is unlikely we need to leverage location awareness as all the Isilon and compute nodes are using the same top-of-rack switches or where the Isilon nodes and compute clients are entirely deployed in separate racks. We now no longer see any benefit in using racks to load balance DataNode connections.

 

In this configuration it is now recommended to leverage a single IP pool for all hadoop access with a dynamic allocation strategy and a SmartConnect Name. No HDFS virtual racks are required as we are not separating NameNode or DataNode pools by node interfaces or by allocation policy. The improvements in OneFS load balancing make the requirement for separate IP pools redundant and make a single pool, simpler and easier to deploy and manage. This single dynamic pool is now optimized for DataNode load balancing.

 

2.png

Single Rack and All Node Access Architectures (No location awareness)

  • Single NameNode and DataNode Pool – Dynamic, SmartConnect name, all Isilon nodes

 

 

Alternatively if compute nodes and Isilon are deployed in a location aware rack configuration, an example is illustrated below. Then racks and multiple pools should be configured and setup to provide rack aware DataNode interfaces to the compute clients.


3.png

With a two rack solution, we would implement three IP Pools and two racks:

  • NameNode Pool – Static allocation and a SmartConnect name, all Isilon nodes
  • DataNode Pool1 – Dynamic allocation, all Isilon nodes in rack 1, no SmartConnect name
  • DataNode Pool2 – Dynamic allocation, all Isilon nodes in rack 2, no SmartConnect name
  • /rack1 <rack1 client IP range> -  assigned to a DataNode Pool1
  • /rack2 <rack2 client IP range> -  assigned to a DataNode Pool2


If additional racks are leveraged we would add additional DataNode Pools and racks.

 

4.png

With a three rack solution, we would implement three IP Pools and two racks:

  • NameNode Pool – Static allocation and a SmartConnect name, all Isilon nodes
  • DataNode Pool1 – Dynamic allocation, all Isilon nodes in rack 1, no SmartConnect name
  • DataNode Pool2 – Dynamic allocation, all Isilon nodes in rack 2, no SmartConnect name
  • DataNode Pool3 – Dynamic allocation, all Isilon nodes in rack 3, no SmartConnect name
  • /rack1 <rack1 client IP range> -  assigned to a DataNode Pool1
  • /rack2 <rack2 client IP range> -  assigned to a DataNode Pool2
  • /rack3 <rack2 client IP range> -  assigned to a DataNode Pool3

 

 

In an expansion scenario we may need to look at modifying the IP pool strategy and the use of the HDFS racks to optimize the change in architecture.


The cluster was initially deployed as a simple solution, a single rack where all nodes are accessed by all compute clients. This is implemented using a single Dynamic Pool. We now add an additional rack with more Isilon nodes and more computer, this would facilitate a change in the deployed architecture.



5.png



Single Rack

  • NameNode and DataNode Pool – Dynamic, SmartConnect name, all Isilon nodes


Additional Rack Added

  • NameNode Pool – Change this pool to Static allocation, SmartConnect name, all Isilon nodes
  • DataNode Pool1 – Dynamic allocation, all Isilon nodes in rack 1, no SmartConnect name
  • DataNode Pool2 – Dynamic allocation, all Isilon nodes in rack 2, no SmartConnect name
  • /rack1 <rack1 client IP range> -  assigned to a DataNode Pool1
  • /rack2 <rack2 client IP range> -  assigned to a DataNode Pool2

 

 

The primary purpose of this post is to highlight the change in recommended approaches to IP pool deployments with later version of OneFS. The introduction of DataNode Load Balancing and Pipeline Write Recovery in OneFS 8.0.1.0 and greater has shifted the recommended approach when no explicit rack location is enforced.



Recommend HDFS IP Pool Strategy without Location Aware Isilon or Compute

 


 

OneFS 8.0.1.0+  – Single Dynamic IP pool, SmartConnect Name, No racks


 

 

Ultimately, all configurations and designs should be evaluated and implemented to meet the requirements of the specific hadoop and network environment. The selected implemented networking configuration will depend on how your Isilon and compute infrastructure is connected plus the size and scale of the deployment. There are many variables to consider when creating an IP pool strategy and this post hopes to highlight options and potential configurations.



 

Using Hadoop with Isilon - Isilon Info Hub

russ_stevenson

Isilon