In my post The infamous '401 Authorization Required' error when starting Kerberized services I discussed possible causes of issues starting Kerberized services.


Having completed some additional testing I would now strongly suggest the following are always completed prior to starting any Kerberized services. Without these changes or validation you will likely see errors with startup:



1.Fix the SAMAccount names for hdfs and ambari-qa in AD

Change the auto-generated SAMAccount name field in AD to be the same as the UPN name.




2.Validate the SPN's on Isilon are valid

Make sure the required hdfs & HTTP SPN exist and in the correct location. Duplicate SPN's with Isilon AD Kerberos and Hortonworks prevent services from starting


isi auth ads spn list --provider-name=<domainname>


Fix any issues.



3.Add a mapping rule to map the domain\hdfs to root.

The existing hdfs>=root mapping rules also now needs an additional rule to map the AD hdfs user to root also.


isi zone zones modify --user-mapping-rules="hdfs=>root, domain\hdfs=>root" --zone=<zone-name>

isi zone zones list -v


isi auth mapping token --zone=<zone-name> --user=hdfs

isi auth mapping token --zone=<zone-name>

Both these token need to be mapped to root; UID:0. If they don't troubleshoot further.




4.Validate the permissions on the krb5.conf

If you have upgraded from 8.0.0.x to you may hit this know issue.

KDC Kerberized Yarn Services Fail to Start on 8.0.1 with Ambari via WebHDFS curl calls


Fix any issues




5. Fully validate DNS

Validate all DNS is fully functional and all records are correct. This includes:

-- All hosts in the compute cluster have forward A and reverse PTR records

-- Isilon Smartconnect Name Delegation is correct, NS record

-- All IP's in the pool assigned to the zone have a PTR record


All clients in the computer cluster should be able to resolve all hostnames, smartconnect zone name and reverse IP lookups.


Issues with reverse DNS may be more likely to be seen with WebHDFS as it relies on SPNEGO, you can likely execute successful hadoop kerberized rpc calls# hadoop fs -ls /  but webhdfs calls fail with 401 errors.





Without these updates to Isilon post kerberization you will see errors starting services currently. having validated the configuration above, start the kerberized hdfs services. Additional changes may also be required.



Version of OneFS this post is applicable to: 8.0.x, 8.0.0.x OneFS


Using Hadoop with Isilon - Isilon Info Hub