Advanced Troubleshooting of an Isilon Cluster Part 3

NOTE: This topic is part of the Uptime Information Hub.

 

< Previous  Next >

 

 

 

Troubleshooting performance issues (cont'd)

This is a continuation of the troubleshooting performance issues series. Performance issues are typically caused by network traffic, network configuration issues, client or cluster processing loads, or a combination thereof. Symptoms include client computers that perform slowly, and/or certain jobs— particularly those than run on the cluster—that either fail or take longer than expected to complete.

Understanding network throughput

If the issue is slow disk writes, for example, and you’ve narrowed it down to a client seeing only 10 MB/s writing over NFSv3 to /ifs/data/tarpit mounted to the IP address that is hosted on node 3, then run the network testing tool iperf at different block and socket buffer sizes writing packets to the same IP address on the cluster, and determine what network throughput can be achieved.


Using iperf, run the following commands on the cluster and client. These commands define a window size that is large enough to indicate whether the network is a potential cause of latency issues.

 

Cluster

iperf -s –w 262144

Client

iperf -c <cluster IP> -w 262144

 

If the throughput is 12 MB/s, the network is likely the culprit. If the throughput is 100 MBs per second, the network is not the likely source of the issue. Next, try running dd or iozone directly on node 3 on the cluster to the same /ifs/data/tarpit directory. Is the performance similar to the client, or markedly better?

Understanding cluster throughput

You can assess cluster throughput by conducting a few tests that measure how much time it takes to read from, and write to, a file. Conduct at least one write test and one read test, as follows:

 

Write test

  1. Open an SSH connection on any node in the cluster and log on using the root account.
  2. Change to the /ifs directory:
    cd /ifs
  3. From the command-line interface (CLI) on the cluster, or from a UNIX or Linux client computer, run the dd command to write a new file to the cluster. Run the following command:
    time dd if=/dev/zero of=1GBfile bs=1024k count=1024
    This command creates a sample 1 GB file and reports how much time it took to write the file to disk.
  4. From the output of this command, extrapolate how many MB per second can be written to disk in single-stream workflows.
  5. If you have a Mac client and want to conduct further analysis:
    • Start Activity Monitor.
    • Run the following command, where pathToFile is the file path of the targeted file:
      cat /dev/zero > /pathToFile
      This command helps you to measure the throughput of write operations on the Isilon cluster. (Although it is possible to run the dd command from a Mac client, results can be inconsistent.)
    • Monitor the results of the command in Activity Monitor's Network tab.


Read test

When measuring the throughput of read operations, be sure not to conduct read tests on the file that you created during the write test. Because that file has been cached, the results of your read tests would be inaccurate. Instead, test a read operation of a file that has not been cached. Find a file on the cluster that is larger than 1GB, and reference that file in the read test.

  1. Open an SSH connection on any node in the cluster and log on using the root account.
  2. From the command-line interface (CLI) on the cluster or from a UNIX or Linux client computer, run the dd command to read a file on the cluster. Run the following command where pathToLargeFile is the file path of the targeted file:
    time dd if=/pathToLargeFile of=/dev/null bs=1024
    This command reads the targeted file and reports how much time it took to read it.
  3. If you have a Mac client and want to conduct further analysis:
    • Start Activity Monitor.
    • Run the following command where pathToFile is the file path of the targeted file:
      time cp /pathToLargeFile > /dev/null
      This command helps you to measure the throughput of read operations on the Isilon cluster. (Although it is possible to run the dd command from a Mac client, results can be inconsistent.)
    • Monitor the results of the command in Activity Monitor's Network tab.

< Previous  Next >