NOTE: This topic is part of the Uptime Information Hub.
For Unix users, the command line interface (CLI) is a celebrated part of the OneFS experience. Isilon system administrators and other OneFS users often write scripts using the CLI and the application programming interface (API) to automate cluster administration, cluster analysis, and other custom workflows on Isilon clusters. These scripting solutions can deliver tremendous efficiencies, automating tasks that otherwise would have been handled manually. This article presents a lightweight framework for you to consider before you deploy these scripts on an Isilon cluster.
Test your scripts on a test cluster first!
EMC Isilon enables users to run scripts directly on your clusters. Other storage vendors do not always allow for this level of flexibility. With this freedom, however, comes responsibility. Scripts should always be thoroughly tested and debugged on a test cluster before deployment and routinely monitored afterwards.
Always deploy scripts with caution, because running insufficiently tested scripts could cause unexpected results, including performance degradation, data loss, or data unavailability.
One example of a script that caused unexpected results consisted of a script that was used to monitor performance statistics on an Isilon cluster. The script was designed to run multiple times on a node. But because the scripts ran concurrently, rather than waiting for a previous invocation to finish, the cluster experienced high CPU and memory usage until the multiple iterations of those scripts were stopped. Users were significantly affected, because they experienced slower than normal access to the cluster. This impact could have been avoided if the script had been sufficiently tested and debugged on a test cluster before being deployed on the production cluster.
To prevent these kinds of issues in the future, we recommend the following best practices.
Scripting best practices
Consider these best practices when you write, and before you deploy, scripts on a production cluster:
- Always test your scripts on a test (non-production) cluster or virtual node that closely mirrors your production environment before deploying the scripts on your production cluster---unless you want to take the risk of losing live production data! The functionality in the test environment should be configured to be nearly identical to the production cluster.
- Do not run your scripts as the root user. Instead, run them as another user that has only the minimum amount of privileges required to get the work done. This way, in case there are issues with the script, users are less likely to affect other areas of the cluster.
- Restrict scripts to handle only the bare minimum of processes needed to get tasks accomplished, in order to limit any potential issues.
- Error handling: Write errors to a separate log file.
- If you are using the cron daemon/service, run your scripts using the isi_ropc command, which prevents scripts from running on more than one node at a time, even when called through cron. If you're running a script as a cron job, you will want to generate a log file to refer to for later analysis.
- If you are testing scripts with debug on and writing to a log file, be sure to turn debug off before you deploy the scripts, so that disk space is not consumed unnecessarily. Also be sure to delete any test files as you go along, so as not to create extra detritus on disk---for example, a trail of garbage files left behind in your wake.
- Provide comments that explain your code as you write it, so that others can understand it (and so that you can understand it months from now).
- If you are considering upgrading to a newer version of OneFS, check the command line syntax to see if the syntax has changed between versions. The Isilon OneFS CLI Mappings document provides detailed command line syntax changes across OneFS versions, starting with OneFS 6.5.5 through OneFS 7.2.1. Several command structures were changed starting in OneFS 7.x. Customers moving to 7.x should migrate their scripts to the new CLI.
- If you are converting OneFS 6.5.x scripts to 7.x, you can use isi_classic in your existing 6.5.x scripts as a bridge for running scripts with 6.5.x commands on some later OneFS versions. However, you should consider isi_classic only as a temporary stop-gap measure, because isi_classic commands will be removed or restricted in their functionality in upcoming versions. If you are not sure how to convert older scripts, contact EMC Online Support.
- Better yet, rather than relying on CLI structures, migrate your scripts to the OneFS Platform API (PAPI). OneFS PAPI provides programmatic interfaces to OneFS configuration and management using the RESTful HTTPS service as a protocol. Scripts will not break across OneFS versions if you script against the API. Currently not all commands are available through the API in 7.x, but all commands will be available in later versions of OneFS.
- In OneFS 7.0 and later, you can add the –-debug option to the isi command to determine whether a OneFS command is available through PAPI. For more information, see How to determine if a OneFS command is accessible through the OneFS Platform API.
- For more information about API scripting, see the API Info Hub.
To prevent unexpected performance degradation, data loss, or data unavailability events, always fully test scripts in a test environment before you deploy scripts to your production cluster.
Whenever possible, move to a PAPI scripting solution rather than relying on the CLI to automate cluster tasks. For more information about using the OneFS platform API, see these EMC Online Support documents and the API Info Hub.