As we saw in the first of these blog articles, in OneFS parlance a group is a list of nodes, drives and protocols which are currently participating in the cluster. Under normal operating conditions, every node and its requisite disks are part of the current group, and the group’s status can be viewed by running sysctl on any node of the cluster.


For example, on a three node X200 cluster:

# sysctl <2,288>: { 1-3:0-11, smb: 1-3, nfs: 1-3, hdfs: 1-3, swift: 1-3, all_enabled_protocols: 1-3 }


So, OneFS group info comprises three main parts:


  • Sequence number:  Provides identification for the group (ie.’ <2,288>’ )
  • Membership list:      Describes the group (ie. ‘1-3:0-11’ )
  • Protocol list:             Shows which nodes are supporting which protocol services (ie. ‘smb: 1-3, nfs: 1-3, hdfs: 1-3, swift: 1-3, all_enabled_protocols: 1-3’)


Please note that, for the sake of clarity, the protocol information has been removed from the end of each group string in all the following examples. For example:

<2,288>: { 1-3:0-11, smb: 1-3, nfs: 1-3, hdfs: 1-3, swift: 1-3, all_enabled_protocols: 1-3 }


Will be represented as:

<2,288>: { 1-3:0-11 }

If even more detail is desired, the syscl efs.gmp.current_info command will report extensive current GMP information.

So, the membership list { 1-3:0-11 } represents our three node cluster, with nodes 1 through 3, each containing 12 drives, numbered zero through 11. The numbers before the colon in the group membership string represent the participating Array IDs, and the numbers after the colon are the Drive IDs.


Each node’s info is maintained in the /etc/ifs/array.xml file. For example, the entry for node 1 of the X200 cluster above reads:









<ipaddress name="int-a"></ipaddress>






It’s worth noting that the Array IDs (or Node IDs as they’re also often known) differ from a cluster’s Logical Node Numbers (LNNs). LNNs are the numberings that occur within node names, as displayed by isi stat for example.


Fortunately, the isi_nodes command provides a useful cross-reference of both LNNs and Array IDs:


X200-1# isi_nodes "%{name}: LNN %{lnn}, Array ID %{id}"

x200-1: LNN 1, Array ID 1

x200-2: LNN 2, Array ID 2

x200-3: LNN 3, Array ID 3

As a rule of thumb, LNNs can be re-used within a cluster, whereas Array IDs are never recycled. In this case, node 1 was removed from the X200 cluster and a new node was added instead:


x200-1: LNN 1, Array ID 4


The LNN of node 1 remains the same, but its Array ID has changed to ‘4’. Regardless of how many nodes are replaced, Array IDs will never be re-used.


A node’s LNN, on the other hand, is based on the relative position of its primary backend IP address, within the allotted subnet range.

The numerals following the colon in the group membership string represent drive IDs that, like Array IDs, are also not recycled. If a drive is failed, the node will identify the replacement drive with the next unused number in sequence.


Unlike Array IDs though, Drive IDs (or Lnums, as they’re sometimes known) begin at 0 rather than at 1 and do not typically have a corresponding ‘logical’ drive number. For example:


x200-3# isi devices drive list

Lnn Location  Device    Lnum State   Serial


3    Bay  1 /dev/da1  12    HEALTHY PN1234P9H6GPEX

3    Bay  2 /dev/da2  10    HEALTHY PN1234P9H6GL8X

3    Bay  3 /dev/da3  9     HEALTHY PN1234P9H676HX

3    Bay  4 /dev/da4  8     HEALTHY PN1234P9H66P4X

3    Bay  5 /dev/da5  7     HEALTHY PN1234P9H6GPRX

3    Bay  6 /dev/da6  6     HEALTHY PN1234P9H6DHPX

3    Bay  7 /dev/da7  5     HEALTHY PN1234P9H6DJAX

3    Bay  8 /dev/da8  4     HEALTHY PN1234P9H64MSX

3    Bay  9 /dev/da9  3     HEALTHY PN1234P9H66PEX

3    Bay 10    /dev/da10 2     HEALTHY PN1234P9H5VMPX

3    Bay 11    /dev/da11 1     HEALTHY PN1234P9H64LHX

3    Bay 12    /dev/da12 0     HEALTHY PN1234P9H66P2X


Total: 12


Note that the drive in Bay 5 has an Lnum, or Drive ID, of 7, the number by which it will be represented in a group statement.

Drive bays and device names may refer to different drives at different points in time, and either could be considered a "logical" drive ID. While the best practice is definitely not to switch drives between bays of a node, if this does happen OneFS will correctly identify the relocated drives by Drive ID and thereby prevent data loss.


Depending on device availability, device names ‘/dev/da*’ may change when a node comes up, so cannot be relied upon to refer to the same device across reboots. However, Drive IDs and drive bay numbers do provide consistent drive identification.


Status info for the drives is kept in a node’s /etc/ifs/drives.xml file. Here’s the entry is for drive Lnum 0 on node Lnn 3, for example:


<logicaldrive number="0" seqno="0" active="1" soft-fail="0" ssd="0" purpose="0">66b60c9f1cd8ce1e57ad0ede0004f446</logicaldrive>

For efficiency and ease of reading, group messages combine the xml lists into a pair of numbers separated by dashes to make reporting more efficient and easier to read. For example:


{ 1-3:0-11 }


However, when a replacement disk (Lnum 12) is added to node 2, the list becomes:


{ 1:0-11, 2:0-1,3-12, 3:0-11 }

Unfortunately, changes like these can make cluster groups trickier to read. For example:

{ 1:0-23, 2:0-5,7-10,12-25, 3:0-23, 4:0-7,9-36, 5:0-35, 6:0-9,11-36 }


This describes a  cluster with two node pools. Nodes 1 to 3 are S200s with 24 drives, and nodes 4 through 6 are X400s with 36 drives each. Nodes 1, 3, and 5 contain all their original drives, whereas node 2 has lost drives 6 and 11, and node 6 is missing drive 10.


Accelerator nodes are listed differently in group messages since they contain no disks to be part of the group. They're listed twice, once as a node with no disks, and again explicitly as a ‘diskless’ node. For example, nodes 11 and 12 in the following:


{ 1:0-23, 2,4:0-10,12-24, 5:0-10,12-16,18-25, 6:0-17,19-24, 7:0-10,12-24, 9-10:0-23, 11-12, diskless: 11-12 }


Nodes in the process of SmartFailing are also listed both separately and in the regular group. For example, node 2 in the following:

{ 1-3:0-23, soft_failed: 2 }


However, when the FlexProtect completes, the node will be removed from the group. A SmartFailed node that’s also unavailable will be noted as both down and soft_failed. For example:


{ 1-3:0-23, 5:0-17,19-24, down: 4, soft_failed: 4 …}


Similarly, when a node is offline, the other nodes in the cluster will show that node as down:

{ 1-2:0-23, 4:0-23,down: 3 }


Note that no disks for that node are listed, and that it doesn't show up in the group.


If the node is split from the cluster—that is, if it is online but not able to contact other nodes on its back-end network—that node will see the rest of the cluster as down. Its group might look something like this instead:

{6:0-11, down: 3-5,8-9,12 }


Like nodes, drives may be smartfailed and down, or smartfailed but available. The group statement looks similar to that for a smartfailed or down node, only the drive Lnum is also included. For example:


{ 1-4:0-23, 5:0-6,8-23, 6:0-17,19-24, down: 5:7, soft_failed: 5:7 }

This indicates that node id 5 drive Lnum 7 is both SmartFailed and unavailable. If the drive was SmartFailed but still available, the group would read:


{ 1-4:0-23, 5:0-6,8-23, 6:0-17,19-24, soft_failed: 5:7 }


When multiple devices are down, consolidated group statements can be tricky to read. For example, if node 1 was down, and drive 4 of node 3 was down, the group statement would read:


{ 2:0-11, 3:0-3,5-11, 4-5:0-11, down: 1, 3:4, soft_failed: 1, 3:4 }


As mentioned in the previous GMP blog article, OneFS has a read-only mode. Nodes in a read-only state are clearly marked as such in the group:


{ 1-6:0-8, soft_failed: 2, read_only: 3 }


Node 3 is listed both as a regular group member and called out separately at the end, because it’s still active. It’s worth noting that "read-only" indicates that OneFS will not write to the disks in that node. However, incoming connections to that node are still able write to other nodes in the cluster.


Non-responsive, or dead, nodes appear in groups when a node has been permanently removed from the cluster without SmartFailing the node. For example, node 11 in the following:


{ 1-5:0-11, 6:0-7,9-12, 7-10,12-14:0-11, 15:0-10,12, 16-17:0-11, dead: 11 }


Drives in a dead state include a drive number as well as a node number. For example:

{ 1:0-11, 2:0-9,11, 3:0-11, 4:0-11, 5:0-11, 6:0-11, dead: 2:10 }


In the event of a dead disk or node, the recommended course of action is to immediately start a FlexProtect and contact Isilon Support.

SmartFailed disks appear in a similar manner to other drive-specific states, and therefore include both an array ID and a drive ID. For example:


{ 1:0-11, 2:0-3,5-12, 3-4:0-11, 5:0-1,3-11, 6:0-11, soft_failed: 5:2 }


This shows drive 2 in node 5 to be SmartFailed, but still available. If the drive was physically unavailable or down, the group would show as:

{ 1:0-11, 2:0-3,5-12, 3-4:0-11, 5:0-1,3-11, 6:0-11, down: 5:2, soft_failed: 5:2 }


Stalled drives (drives that don’t respond) are mraked as such, for example:

{ 1:0-2,4-11, 2-4:0-11, stalled: 1:3 }


When a drive becomes responsive again (un-stalled), it simply returns to the group. In this case, the  group status would return to:

{ 1-4:0-11 }


A group displays the sequence number between angle brackets. For example, <3,6><3,6>: { 1-3:0-11 }, the sequence number is <3,6><3,6>.


The first number within the sequence, in this case 3, identifies the node that initiated the most recent group change


In the case of a node leaving the group, the lowest-numbered node remaining in the cluster will initiate the group change and therefore be listed as the first number within the angle brackets. In the case of a node joining the group, the newly-joined node will initiate the change and thus will be the listed node. If the group change involved a single drive joining or leaving the group, the node containing that drive will initiate the change and thus will be the listed node. The second piece of the group sequence number increases sequentially. The previous group would have had a 5 in this place; the next group should have a 7.


A group change occurs when an event changes devices participating in a cluster. These may be caused by drive removals or replacements, node additions, node removals, node reboots or shutdowns, backend (internal) network events, and the transition of a node into read-only mode. For debugging purposes, group change and cluster event messages can be reviewed correlated and to determine whether any devices are currently in a failure state. We will explore this further in the next GMP blog article.


For more information on OneFS groups, please refer to the first article in this series: