This blog post is the fifth (and last!) in a series covering basic multiprotocol concepts in Isilon OneFS. Multiprotocol data access provides unified, consistent file access across data access protocols.  Here, we’ll cover how to troubleshoot permissions issues and the OneFS Job Engine’s PermissionRepair job.

The set up

You’ve set up multiprotocol data access.  OneFS is generating identical access tokens for users accessing the cluster using either Windows/SMB or UNIX/NFS, and file access checking determines a user’s access permissions by comparing the access token against permissions on a file. But if something doesn’t work quite as expected, you’ll need to do some troubleshooting.

 

Troubleshooting:  be methodical

The approach to troubleshooting permissions issues can be summed up in two words: Be Methodical. Access tokens and file permissions are where multiprotocol data access starts. Most user permissions problems are the result of a broken or incomplete access token, incorrect file permissions, or a combination of the two.

Always look at the access token first. If the access token doesn’t look right, look at the user mapping. If the user mapping looks good, look at the ID mapping. If the token looks good, check the file permissions. Does the file have the permissions that you think it should? Based on the file permissions and the users’ access tokens, do the users have the access you think they did?

Here are the steps for troubleshooting permissions problems.

  1. Dump the user access token. See the EMC Isilon OneFS CLI Administration Guide for instructions.
  2. Review the file permissions using ls -le and ls -len from the Isilon CLI.
  3. Check your mapping rules.
  4. Review the ID mapping in the ID map database.
  5. Validate your access zones.
  6. Validate your networking.
  7. Review the share and export configurations.
  8. Review extended file and directory permissions from the Isilon CLI.
  9. Review the file permissions using the protocol over which access occurs.

Remember, review permissions directly on the Isilon CLI to troubleshoot multiprotocol access issues or unexpected behavior. Reviewing permissions over SMB or NFS shows you only the SMB view or the NFS view of the permissions, which can be valuable but is definitely not the whole picture. If there’s unexpected or incorrect protocol access behavior, you need to review, validate, and test on the client that has the issue and directly on the Isilon cluster.

 

The troubleshooting loop

In general, a good approach is to loop through the following steps until you see consistent behavior and the permissions or permission management tools are behaving as expected.

  1. View and validate the Isilon permissions and token using the Isilon CLI
  2. Test from the NFS side
  3. Test from the SMB side

 

SMB shares may have share permissions and NFS exports may have export restrictions that can also affect data access and permissions. This series doesn’t address share and export configuration, but share and export configuration can and does affect access to the underlying file system. Share and export configuration must be accounted for in multiprotocol environments and must be consistent with the access that you require. Keep in mind that share and export configuration is separate from actual on-disk file permissions.

 

The Domain Users issue in an LDAP/Active Directory environment

An issue that can come up in an LDAP/Active Directory multiprotocol environment is what happens when OneFS tries to map the Active Directory Domain Users group to an LDAP user group. The Domain Users group generally doesn’t exist in LDAP, so there’s no equivalent ID mapping. The user’s default primary group in that environment should likely be the primary group retrieved from LDAP. In an LDAP/Active Directory multiprotocol environment, make sure that you have a rule in place that specifies that Domain Users is not the default primary group for a particular user. If that rule is not in place, OneFS can potentially set the user access token’s default group as the Domain User’s group, which may not be the baseline requirement for consistent multiprotocol access. See the white paper Identities, Access tokens, and the Isilon OneFS User Mapping Service for details.

 

The Isilon Permission Repair Job

Suppose that you need to update the ownership or permissions for all files and directories in a directory structure with a new set of permissions, or you want to prevent permissions issues that can occur after changing the on-disk identity.  You could issue a recursive chmod command or use Windows Explorer, specifying that the changes apply recursively to all subdirectories and files. But changing the permissions this way is single-threaded and could take a very long time to complete. Instead, you can leverage the Isilon OneFS Job Engine’s PermissionRepair authentication and access control job. Because the Isilon job engine is multi-threaded, changes to permissions on directory trees and very large file systems complete very quickly and are under the control of the Isilon job engine.

If you need to change or update the identity mode for choosing the on-disk identity (UID, SID, or native), run the PermissionRepair job in convert mode.  Convert mode will look at the existing on-disk identity and update it based on the job run. You can update UIDs to SIDs, SIDs to UIDs, or change to native mode so that the cluster makes the appropriate decision.  Convert mode ensures that the changes are fully propagated throughout the cluster. See the EMC Isilon OneFS CLI Administration Guide for more information.

 

Takeaways

Here are some highlights:

  • Most user permissions problems are the result of a broken access token, bad file permissions, or a combination of the two.
  • Be methodical when troubleshooting. Always look at the access token first, then at the user mapping, then at the ID mapping. If the token looks good, look at the file permissions.
  • Obey the troubleshooting loop. Repeat the following steps until you see consistent behavior and the permissions or permission management tools behave as expected.
    1. View and validate the Isilon permissions and token using the Isilon CLI
    2. Test from the NFS side
    3. Test from the SMB side
  • Use the Job Engine’s PermissionRepair job to update the ownership or permissions for all files and directories in a directory structure with a new set of permissions, or to prevent permissions issues that can occur after changing the on-disk identity.

And in the end…

 

Remember that Isilon multiprotocol data access is the relationship of many elements and features of the cluster.

Multi-AIMA.PNG.png

The relationship of all the components defines how and why Isilon multiprotocol data access behaves as it does. Once you understand the relationships, Isilon multiprotocol data access will become simpler and straightforward to work with.


References

  1. Identities, Access tokens, and the Isilon OneFS User Mapping Service white paper
  2. EMC Isilon OneFS CLI Administration Guide
  3. EMC Isilon Multiprotocol Concepts Series: links to the full Multiprotocol Concepts series posts

Let us know

Tell us what you think of this article. Was this level of information useful? Do you have questions that you would like us to cover in future blog posts? Let us know by leaving a comment.