Tips for Deploying Scripts on EMC Isilon OneFS Clusters - part 2

NOTE: This topic is part of the Uptime Information Hub.

 

Back to scripting - part 1

Introduction

 

This article is the second in a two-part series on OneFS scripting and includes some general recommendations regarding putting your own scripts on an Isilon cluster. If you are thinking about creating and running a script to address a product gap, submit a Request For Enhancement (RFE) on EMC Isilon Technical Support.

Scripting best practices

 

The following best practices should be considered when writing and before deploying scripts on a production cluster.

 

First, think about whether you really need to do this. It is important to note that people have brought clusters down with their scripts, so you need to be extra cautious as noted in part 1 of this series.

 

Secondly, examine if you can accomplish your goals with a client using the Isilon OneFS API to interface with the Isilon cluster.  This is particularly true if you are trying to do some type of monitoring task. All of the performance statistics that you can get from the CLI are available through the API. Using the API is much safer if something goes wrong with your script. Isilon Engineering is also working on providing an Isilon SDK to make using the API even easier in the future.

 

Thirdly, consider who will support this script in the future? The customer cannot call EMC Support for help with the script itself. Who will support it if you are moved off of the account?

 

If you still want to go ahead...

  1. Put your script in /ifs.  For example, /ifs/scripts. OneFS does not allow invoking executables directly from /ifs if you are logged into a shell on the cluster itself. If, for example, your script is called 'my-script.sh' located in /ifs/scripts, you could execute it from a shell on the cluster by running the following:
          bash /ifs/scripts/my-script.sh
  2. Put your name, email, and phone number in comments at the top of the script.
  3. Include comments at the top of the script to explain what the script is trying to do and why.
  4. Include the history of changes in comments at the top of the script. When it was installed and by whom. When it or its installation was modified, by whom, and why.
  5. Include logic in you script to check the version of OneFS on the cluster. Ensure that your script exits without doing anything if the version of OneFS is anything other than what you expect and have tested the script on.
  6. Include locking logic to ensure that the script will not run if there are any other instances of it running elsewhere on the cluster.
  7. Every use of an 'isi' command should include a timeout option and the 'degraded' option.  Not all commands include both, but if a command does, then use them.
  8. If you want the script to run under the control of the cron/daemon/service for OneFS 7.x: add lines to /etc/mcp/override/crontab.smbtime on any node as follows. Note that OneFS 8.x is different.
    # Comment to explain what, who, why, and when.
            <min> <hour> <dayofweek> <month> <dayofweek> isi_ropc -s /bin/bash /ifs/scripts/my/scripts.sh
    # End addition by <your name>
    OneFS will propagate the crontab entry to all nodes in the cluster. The isi_ropc -s part tells OneFS to invoke the following command on only one node of the cluster even though the crontab entry exists on all nodes.  The '-s' tells 'isi_ropc' to select the node to run the command on by hashing the command itself.

    If you're running a script as a cron job, be sure to generate a log file to refer to for later analysis.
  9. Test your script extensively. On any versions you want it to run on.  On versions you don't want it to run. Test it in the presence of failed drives and nodes.  Test that it will not run multiple instances.  Test some more.
  10. Get someone both more experiernced with scripting and more experienced with OneFS to review your script and proposed installation method.
  11. Test again.

 

CAUTION!

If you do not understand any of this article, or if you do not know how to perform any of the above, then you should not be creating and running these scripts.

 

Back to scripting - part 1