Using the Isilon OneFS isi_auth_expert command to manage authentication issues

NOTE: This topic is part of the Uptime Information Hub.

 

Introduction

 

The isi_auth_expert command was introduced in OneFS 7.1.1.9, OneFS 7.2.1.3, OneFS 8.0.0.1, and OneFS 8.0.1.0. The command can also be installed on clusters running OneFS 7.1.1.0 through 7.1.1.8 and OneFS 7.2.1.0 through 7.2.1.2 by installing patch-164666 which is available for download from the Dell EMC Online Support site.

 

You can run the isi_auth_expert command to examine an Isilon cluster's authentication environment to help ensure that it is properly configured and to identify conditions that could be causing data access latency due to the authentication configuration issues.

 

The isi_auth_expert command runs a series of tests, including network and port connectivity and latency, binding, and clock skew. These results can be used to isolate a problematic configuration or network path that is causing data access issues.

 

You may want to run this tool:

  • When existing or new users experience connection latency or are prompted to enter login crednetials when accessing data.
  • When the cluster is reporting events regarding Active Directory or LDAP offline status.
  • After making changes to authentication provider setup.
  • After configuration changes have affected network paths between a cluster and its authenticiation providers.


NOTE
New checks and parameters were added to the isi_auth_expert command in OneFS 7.2.1.5. See the Additional checks and parameters in OneFS 7.2.1.5 and later section of this article for more information.

 

Instructions

 

To run the isi_auth_expert command on clusters running OneFS 7.1.1.9, OneFS 7.2.1.3, OneFS 8.0.0.1, OneFS 8.0.1.0 or later, run the following command:

isi_auth_expert

 

To run the isi_auth_expert command on a cluster to which the patch has been applied, run the following command:

/usr/local/isi_auth_expert/isi_auth_expert

 

You can also run the command with one or more of the options listed in the table below:

 

OptionExplanation
-h, --helpShow the syntax for this command
-d, --debugDisplay debugging messages
-v, --verboseEnable verbose (more robust) output
--no-colorDisable colored output

 

Example output

wcvirt1-1# isi_auth_expert
Checking authentication process health ... done
Checking LDAP provider 'ldaptest' server connectivity ... done
Checking LDAP provider 'ldaptest' base dn ... done
Checking LDAP provider 'ldaptest' object enumeration support ... done
Checking LDAP provider 'ldaptest' group base dn ... done
Checking LDAP provider 'ldaptest' user base dn ... done
  [ERROR] The configured base user dn 'ou=dne,dc=isilon,dc=com' in LDAP provider
  'ldaptest' was not found on LDAP server ldaptest.west.isilon.com.
Checking AD provider 'WMC-ADA.WEST.ISILON.COM' DC connectivity ... done
Checking AD provider 'WMC-ADA.WEST.ISILON.COM' auth related ports ... done
  [ERROR] Failed to establish a connection to the AD domain controller wmc-ada-dc1
            .wmc-ada.west.isilon.com on port 3268.

 

Implemented tests

 

When you run the isi_auth_expert command, the following checks are performed.

 

Process checks

 

This test confirms that the authentication-related processes (lsass, lwio & netlogon) are running. If any of the processes are not running, an error is returned.

 

Active Directory

 

The following section describes the tests that the isi_auth_expert command performs for each Active Directory (AD) provider.

Check Domain Controller connectivity

Determine whether the cluster has basic network connectivity to at least one domain controller (DC) in the AD domain.

Check DC ports

Verify that for every DC, the cluster can connect to the AD-related ports, and that the ports are accepting connections.

PortExplanationAD UsageTraffic Type
88Port 88 is used for Kerberos authentication traffic.User and Computer Authentication, Forest Level TrustsKerberos
139Port 139 is ued for NetBIOS and NetLogon traffic.User and Computer Authentication, ReplicationDFSN, NetBIOS Session Service, NetLogon
389Port 389 is used for LDAP queries.Directory, Replication, User and Computer Authentication, Group Policy, TrustsLDAP
445Port 445 is used for replication.Replication, User and Computer Authentication, Group Policy, TrustsSMB, CIFS, SMB2, DFSN, LSARPC, NbtSS, NetLogonR, SamR, SrvSvc
3268Port 3268 is used for global catalog LDAP queries. (used if you want the global catalog in the AD provider enabled)Directory, Replication, User and Computer Authentication, Group Policy, TrustsLDAP GC

 

LDAP

 

The following section describes the tests that the isi_auth_expert command performs for each LDAP provider.

LDAP connectivity

Check LDAP server connectivity by making an anonymous LDAP bind and checking the results.

LDAP enumerated objects support

Confirm that each LDAP server supports enumerated objects by checking the LDAP servers' supported controls. OneFS requires either the paged results controls or both the virtual list view and server-side sorting controls.

Validate configured base-dn

Perform a test query against the configured base-dn to ensure configuration compatibility with the LDAP server.

Validate configured user-base-dn

Perform a test query against the configured user-base-dn to ensure configuration compatibility with the LDAP server.

Validate configured group-base-dn

Perform a test query against the configured group-base-dn to ensure configuration compatibility with the LDAP server.

 

Additional checks and parameters in OneFS 7.2.1.5 and later

 

The following checks were added in OneFS 7.2.1.5.

  • Active Directory
    • Domain Controllers latency check
    • Clock Skew and latency check
    • Global Catalog service for user (SFU) check
  • LDAP - User check
  • Kerberos - SPN checks for SmartConnect zones and aliases

 

The isi_auth_expert command can calculate two types of latencies: ping latency and LDAP latency for all of the domain controllers. If the clock skew is less than five minutes, the command will return: "There is minimal or no skew between the AD provider and your machine."

 

The following parameters were also added.

 

OptionExplanation
--ldap-userChecks the LDAP provider for a specified user
--sfu-userChecks the Active Directory Global Catalog for a specified user
--admin-credsEnables you to supply the credentials that are required when checking the Active Directory Global Catalog

 

LDAP user attribute check

 

To run the LDAP user attribute check, you must run the isi_auth_expert command with the --ldap-user=<user> parameter where <user> is the user you want to check. The user name has to be of the form "plain name" for the search to work. The LDAP user attribute check connects to an LDAP server and queries it for the specified user. We can then check the results of the query to ensure that the user has all necessary attributes needed to be authenticated in any domain.

 

Active Directory Global Catalog SFU check

 

A global catalog server is a domain controller that has information not only about the domain it is associated with but also all the other domains in the forest. Much like an LDAP server, the global catalog has a list of data associated with the domain it control in addition to a partial copy of the data it gets from other domain controllers. If it doesn't have all of the data that the domain controllers are sharing, there could be authentication issues.

 

To run the Active Directory Global Catalog SFU check, you must run the isi_auth_expert command with the --sfu-user=<user> and --admin-creds="[('<Domain>', '<User>', '<password>')]" parameters where <user> is the SFU user you want to check and "[('<Domain>', '<User>', '<password>')]" are the credentials the isi_auth_expert command must provide to perform the Global Catalog lookup in the domain controller. Note that we currently have the following limitation when checking the global catalog: You must provide administrator credentials.

 

Server principal name (SPN) check

 

SPNs may cause authentication failures if they are not present when you join a Kerberos provider, or if you change the name of a SmartConnect zone. The isi_auth_expert command determines if SPNs are missing, stale, or incorrect. This feature will automatically run whenever the isi_auth_expert command is executed.

 

This feature is used to check for missing SPNs in both Kerberos providers and also in SmartConnect zones. The command collects all of the SPNs associated with the providers and SmartConnect zones and ensures that the required SPNs are present.

 

If you are using SmartConnect aliases, it also checks against those aliases.  You can use  the isi auth ads spn or isi auth krb5 spn commands to list, check, or fix reported missing SPNs.

 

Let us know

 

Let us know what you think. If you have feedback for us about this or any other Isilon technical content, email us at isicontent@emc.com.