1 2 3 4 Previous Next 58 Replies Latest reply: Oct 27, 2013 8:12 PM by dynamox RSS

Ask the Expert: AIMA: Everything you wanted to know but were afraid to ask.

MRWA

Welcome to this EMC Support Community Ask the Expert conversation.

YOU MAY ALSO BE INTERESTED ON THESE ATE EVENTS...

Ask the Expert: EMC Isilon Scale-out Data Lake

Ask the Expert: Isilon Performance Analysis

Ask the Expert: Are you ready to manage deep archiving workloads with Isilon’s HD400 node and OneFS 7.2.0? Find out more…

This is an opportunity to learn about and discuss:

What is AIMA?

What is the design philosophy of AIMA, or, in other words, how should I configure it to be successful?

Some common deployment scenarios with best practices and pitfalls.

 

This discussion begins on 9-16-2013 and concludes on 9-27-2013. Get ready by bookmarking this page or signing up to receive email notifications.

 

 

Your host(s): Tim Wright

 

profile-image-display.jspa?imageID=7843&size=350

 

 

Work

Occupation:Technical Support Engineer V
Biography:I have a degree in Computer Science with Mathematics from the University of Bristol, England.
Subsequently, I have spent over 25 years working on complex clustered systems in Customer Support, Professional Services, QA, and Engineering at various companies. I have performed kernel development on both Unix (DYNIX/ptx) and Linux.

 

Because of my breadth and depth of knowledge and experience, I am frequently called upon to diagnose and solve the most complex and difficult problems that involve multiple disparate areas of the product and environment.

 

Outside of work, I enjoy food and wine (both eating out and cooking/entertaining), playing with technology and music (singing and musical typesetting).
Expertise:Extremely broad and deep knowledge of Unix and Linux. Kernel-level development experience with both.
Considerable knowledge and experience in Microsoft Windows architecture and programming.
Significant VMware experience, including VSP and VTSP certification.
Deep knowledge of IP networking, InfiniBand, storage, SANs, Intel-architecture hardware etc.
Programming languages known include C, Python, Perl, Shell, and many others.
Excellent customer skills in support and also pre-sales.
Company:EMC Isilon Storage Division
City:Portland
State:OR
Country:USA
Display Name:Tim Wright
  • 1. Re: Ask the Expert: AIMA: Everything you wanted to know but were afraid to ask.
    MRWA

    This Ask the Expert event is now open for comments and questions. We look forward to an interesting and informative discussion!

  • 3. Re: Ask the Expert: AIMA: Everything you wanted to know but were afraid to ask.
    Tim Wright

    Hello Dynamox,

    I'm glad you asked that. If you can give me about 15 minutes or so, I am about to post an intro which will, I hope, answer that question and possibly others and maybe provoke further questions.

     

    Thanks,

     

    Tim

  • 4. Re: Ask the Expert: AIMA: Everything you wanted to know but were afraid to ask.
    Tim Wright

    Let me start by doing a very brief write-up on the subject:

     

    Q: What is AIMA?

    A: AIMA stands for Authentication, Identity Management, and Authorization. Taken together, it is the functionality in OneFS that talks to the various authentication sources, combines the different personas for a user into an access token, and mediates access to files and directories. Most of the work is performed by the lsassd or lsass process.

     

    Q: Why should I care about this?

    A: AIMA is fundamental to correct operation of the cluster. If your auth environment is simple, then configuring AIMA is correspondingly simple. Where it gets interesting is in the area of multiprotocol.

     

    OneFS supports multiprotocol access i.e. access to the filesystem from both Windows clients that use ACLs and SIDs and also from NFS clients that use POSIX permissions and uids/gids. When correctly configured, it allows users that exist in both "domains" (I'm not referring to AD here) to seamlessly access their data.

     

    AIMA has the concept of auth providers which are directory services where users can be looked up. These include Active Directory, LDAP, NIS as well as cluster-local users. It is probably worth mentioning a few of the design tenets here because they determine how to best configure AIMA.

    • AIMA only ever stores ids (SIDs, gids, uids). It never stores names either in the filesystem or in any of its persistent caches e.g. the idmap database.
    • Only one identity is stored on-disk. The default mode since OneFS 6.5 is "native" which means that if a user has both SIDs and "real" unix ids (more on this later), the unix ids will preferentially be stored on disk.

     

    The underlying reason for the assymetry of the default mode is to accommodate NFSv3. If you are using NFS and you are not using Kerberos, the NFS isn't actually doing any authentication. The protocol simply passes numeric uids and gids in the RPCs on the wire. Given this, if these client are to have access to data, then those uids and gids must somehow allow them access. Because the default settings ensure that Windows users who also exist in a Unix auth provider will store the unix identities, that requirement is met.

     

    One of the challenges to implementing multiprotocol access is the lack of common identities in the on-the-wire protocols. SMB only understands SIDs and ACLs. Basic NFSv3 only understands POSIX permissions and uids/gids (v4 adds support for its own ACL format). Another function of AIMA is to perform this translation. For POSIX ids that do not have an equivalent Windows user, SIDs from special ranges (S-1-5-21 and S-1-5-22) are returned. For SIDs that do not have an equivalent POSIX identity, AIMA constructs "fake" uids/gids and remembers the mapping. This allows e.g. and NFS user to perform stat() (e.g. ls) on a file that is owned by a user that only exists in Windows and get a sane answer.

     

    Documentation

    • The AIMA section of the OneFS Administrator's Guide. Though this is necessarily brief, it is a good introduction.
    • The OneFS multiprotocol whitepaper (found on support.emc.com). A more thorough treatment of the configuration of multiprocol access.
    • The AIMA idmapper paper. This is not yet available but will be published shortly and describes how to use the AIMA identity mapper functionality to map between users where e.g. the names are not consistent.

     

    That's probably enough to start. I will also post some shorter articles focusing on particular tips, tricks and "gotchas".

     

    Thanks,

     

    Tim

  • 5. Re: Ask the Expert: AIMA: Everything you wanted to know but were afraid to ask.
    Peter Serocka

    It seems that lsass(d) in 6.5 never caches queries from an LDAP server.

    Run (on the cluster) stat on 50 files, and 200 full queries, with one new connection per query,

    are sent to the LDAP server within a second or so -- even if only a few distinct

    user/group names appear listed as  file owners.

     

    Sporadicly our cluster (and others have reported the same here)

    complains about the LDAP server accepting no (more) connections.

    Quite understandable with such flooding.

     

    How about better caching? Any options, or am I missing something?

    Are there improvements in 7.0?

     

    Or is id caching regarded as a problem when looking at the whole cluster?

    Could be possible, but not fully obvious. Considering that queries to the

    LDAP server are asynchronous (from multiple nodes)  anyway,

    a few seconds shouldn't do any harm I think; and would help a lot in

    the above example.

     

    -- Peter

  • 6. Re: Re: Ask the Expert: AIMA: Everything you wanted to know but were afraid to ask.
    Tim Wright

    Hello Peter,

    you are quite correct. The LDAP provider in 6.5.x does not cache lookups. Happily, I can report that lsass performs caching for both NIS and LDAP in OneFS 7.x in addition to the caching it has always performed for Active Directory.

     

    By default, the id cache size is 4.77MB and can be modified if required:

    # isi zone zones modify <zone> --cache-size=<size>


    The cache lifetime for both NIS and for LDAP is 15 minutes by default. The duration can also be modified using:

    isi auth ldap modify <provider> --cache-entry-expiry=<duration>


    Your latter point is also important. As implemented today, each node has its own instance of lsass, and its own cache etc. The idmap database for mappings is shared, but there is no globally coherent cache. That obviously means that if you have a large cluster, each node will be performing lookups to the auth providers. Given that in general, a user will connect to only one node at a time, and that the lookups are cached, this should not cause a dramatic increase in traffic, but I do plan to talk to Engineering about where it would be feasible to have some degree of collaboration between the nodes. This would not be trivial to achieve but it certainly warrants consideration.>

     

    Regards,

     

    Tim

  • 7. Re: Ask the Expert: AIMA: Everything you wanted to know but were afraid to ask.
    Peter Serocka

    Tim, thanks a lot!

     

    7.x means starting from 7.0, or "some" x?

     

    Cheers

     

    -- Peter

  • 8. Re: Re: Ask the Expert: AIMA: Everything you wanted to know but were afraid to ask.
    Tim Wright

    Hi Peter,

    sorry for the lack of clarity there. Yes, by 7.x I mean 7.0.1.x or 7.0.2.x or 7.1.x when it ships.

    Current GA versions are, I believe 7.0.1.8 for the 7.0.1 maintenance release train and 7.0.2.1 for the 7.0.2 train. The caching was added in 7.0 and will exist in all future versions.

     

    Regards,

     

    Tim

  • 10. Re: Ask the Expert: AIMA: Everything you wanted to know but were afraid to ask.
    Tim Wright

    OK, today I'd like to talk a little about how the auth token is generated, and one tip for an idmapper rule that may be helpful when managing access by group.

     

    As previously mentioned, AIMA allows you to configure authentication providers such as Active Directory, LDAP etc.

    In OneFS 6.5, there was a single list of providers as a hidden config parameter. With the advent of authentication zones in OneFS 7.0, each zone has its own set of providers. They can be added in any order but, generally speaking, the default order should not be changed. In particular, on 6.5, the idmapper provider (ID) is at the start of the list and should never be moved (this was addressed in 7.0).

    The default order in 7.0 is:

    ActiveDirectory,Local,File,Ldap,Nis

     

    Again, as previously mentioned, there is an assymmetry in the design because non-kerberized NFSv3 clients don't actually perform authentication, and so everything is tailored to allow raw uids and gids to give correct access. The ActiveDirectory provider at the head of the list is "special" insofar as it has an implicit callout to lookup the domain user as a "Unix" user in the other providers. This means that, if you do not disable lookup-users, a user "testuser1" who exists in AD and LDAP will receive a token that contains both the Windows and Unix identities, with the Unix uid as the on-disk identity:

     

    isi7014-1# isi auth mapping token --user='VMTEST\testuser1'

            User

              Name : VMTEST\testuser1

                UID : 1001

                SID : S-1-5-21-4087762976-3323942327-742857095-1118

              ZID: 1

            Zone: System

      Privileges: -

    Primary Group

              Name : VMTEST\domain users

                GID : 1000008

                SID : S-1-5-21-4087762976-3323942327-742857095-513

    Other Groups

              Name : Users

                GID : 1545

                SID : S-1-5-32-545

     

     

              Name : testgroup1

                GID : 1001

                SID : S-1-22-2-1001

     

    Note that for the User, I have both a valid SID and also a UID. However, my primary group is a SID. This is because my primary group in AD is "Domain Users" which doesn't exist in my LDAP provider. If I want the Windows user to pick up my Unix primary group when there is a matching user, I need to add a small mapping rule:

     

    isi7014-1# isi zone zones modify --zone=System --add-user-mapping-rules='VMTEST\* += * [group]'

    We will talk later about the idmapper. What that rule does is find users in the VMTEST domain, and insert (+=) the primary group ([group]) from the matching Unix user.

     

    Now let's look at the token once more:

    isi7014-1# isi auth mapping token --user='VMTEST\testuser1'

             User

               Name : testuser1

                UID : 1001

                SID : S-1-5-21-4087762976-3323942327-742857095-1118

              ZID: 1

             Zone: System

       Privileges: -

    Primary Group

               Name : testgroup1

                GID : 1001

                SID : S-1-22-2-1001

    Other Groups

               Name : VMTEST\domain users

                GID : 1000008

                SID : S-1-5-21-4087762976-3323942327-742857095-513

     

     

               Name : Users

                GID : 1545

                SID : S-1-5-32-545

     

    Notice how my primary group is now Unix group 1001 *testgroup1) and the 'VMTEST\Domain Users' group has been pushed out as a secondary.

     

    That's probably enough for one posting. I welcome comments, questions, complaints, etc. :-)

  • 11. Re: Ask the Expert: AIMA: Everything you wanted to know but were afraid to ask.
    Peter Serocka

    Do I get it correctly, [group] is not a placeholder to be filled in by the CLI admin user,

    but a literal that pulls "the group" from any(?) later provider?

     

    -- Peter

  • 12. Re: Ask the Expert: AIMA: Everything you wanted to know but were afraid to ask.
    Tim Wright

    Hi Peter,

    yes, the CLI syntax for the user mapper allows you to pick and choose which elements of the RHS of the map you wish to pull when doing an insert or append. In this case,"+=" is the insert operation and the '[group]' qualifier means it is pulling the primary group from a match.

     

    When matching, an id of the form "DOMAIN\*" we will look in AD for a domain user, and when matching "*", we will lookup in what I tend to call a Unix provider, i.e. LDAP or NIS (or a local user from the cluster).

     

    I will see if I can find when we expect to publish the usermapper guide as that covers it in a lot of detail.

     

    Tim

  • 13. Re: Ask the Expert: AIMA: Everything you wanted to know but were afraid to ask.
    Tim Wright

    So yesterday, I mentioned that if you want to manage group access using the Unix groups, you can simply force the primary Unix group to be the primary (on-disk) group for the Windows user.

     

    Conversely, if you want the AD groups to be used and you want NFS users to have access, there is a global or per-export NFS setting that will help. This setting is the "map-lookup-uid" setting that can be set in the UI of from the CLI for an individual export or for the default export.

     

    When you enable "map-lookup-uid" on an NFS export, the way that incoming RPCs are treated changes dramatically. Instead of taking the uid, primary gid and up-to-16 supplementary groups that are sent over the wire, OneFS does the following:

    • Takes the incoming uid and performs a reverse-lookup to find a name.
    • Takes the name and looks up the name in the default/primary domain to get the Windows identities for that name.
    • In doing the lookup in the AD provider, that also performs the implicit callout to lookup the name in the Unix auth providers.

     

    The upshot of this is that the NFS server acting on behalf of the client is now running with an auth token that is the same as if the domain user was accessing the cluster via SMB. This actually has two interesting efffects:

    1. The NFS user has all of the Windows groups in its access token and can this access data that e.g. is governed by an ACL that only permits access to a group that is found only in AD and not in LDAP/NIS.
    2. The other side-effect. If the user is a member of more than 16 supplementary Unix groups, these will all be picked up and added to the token.

     

    Both configurations are valid and it comes down to the end-user which one is preferable.

  • 14. Re: Ask the Expert: AIMA: Everything you wanted to know but were afraid to ask.
    unaumann

    Hi Tim,

     

    your explanations is really superb. thanks a lot.

     

    I would ask you about another szenario.

     

    We asume we have local NFS Users (without kerberos or ldap) and on the same dir we would have access over smb with ad authentication for smb users. so we need an construct with one or more groups where the lokal nfs user and the ad user are put together to have access to the same data. (some user should able to read only, other users should be able to have rw access)

    For my opinion, this could not be a AD group, because AD do not know about the local users? Could i add AD Users to a local group?

    Is this possible, without matching user on both sides???

     

    regards Uwe

1 2 3 4 Previous Next