VNX: Event Code : 0xfffffffffffffff9 ; Error HC_BE_14505017506 The following domain file(s) does not exist

           

   Article Number:     485995                                   Article Version: 3     Article Type:    Break Fix 
   

 


Product:

 

VNX/VNXe Family,Unisphere for VNX,VNX1 Series,VNX5300,VNX5500,VNX5700,VNX7500,VNX5400,VNX5600,VNX5800,VNX7600,VNX8000,VNX2 Series

 

Issue:

 

 

1) File Options are grayed out in Unisphere   
   
    2) Login Warning/Error   
    Message :            Logging into 172.28.7.60 encountered an error.   
    Full Description :               An error occurred during the login process. The control station did not log in properly.   
    Recommended Action :   The login error may have occurred because:   
    1. Certificates are not accepted.   
    2. Both storage processors or the control station are not accessible.   
    3. You have logged in to a File or Block system using a local user account not defined on both the File and Block systems.     
      Event Code :       0xfffffffffffffff9     
            
            
      3) nas_checkup error:     
      -------------------------------------Errors-------------------------------------     
      Storage System : Check for domain and federations health on VNX     
      Error HC_BE_14505017506: The following domain file(s) does not exist     
                * /nas/http/domain/domain_list     
                * /nas/http/domain/domain_master     
                * /nas/http/domain/domain_provider_xml     
      Action : Contact your service provider and reference EMC Knowledgebase     
               article emc273714.
   
     
                                                           

 

 

Cause:

 

 

In the most observed cases, Control Station failover/reboot will cause the Domain files to get corrupted.                                                           

 

 

Change:

 

 

Control Station failover/failback                                                           

 

 

Resolution:

 

 

If the control station is failed over to slot_1 then Failback the control station by logging into slot_1 Control Station and rebooting it.   
   
    If you have access to Putty session (ssh) to control station (Primary), try the command(Type "yes" when it prompts):         
         
          [nasadmin@VNX137 ~]$ /nasmcd/sbin/getreason
     
     
      If the result shows that:         
          slot_1 Primary Control station         
          slot_0 Secondary Control station         
         
          You need to fail back the control station by rebooting slot_1 control station. Just login as root (su) and run the following command:         
         
          [nasadmin@VNX137 ~]# reboot         
         
          This would take few minutes to reboot the control station and it to come up in the original configuration.
   
   
   
    Once the control stations are back on original status, check the domain files:       
        [nasadmin@VNX137 ~]$ /nasmcd/sbin/getreason       
        10 - slot_0 primary control station       
        11 - slot_1 secondary control station       
         5 - slot_4 contacted       
         5 - slot_3 contacted       
       
        [nasadmin@VNX137 ~]$ ll /nas/http/domain       
        total 15       
        drwxr-xr-x 2 apache apache 1024 Jun 27 14:35 bak       
        drwxr-xr-x 2 apache apache 1024 Nov 14  2013 ldap       
        -rw-r--r-- 1 apache apache    0 Mar 24 22:04 lock_file       
        [nasadmin@VNX137 ~]$       
       
        Verify the Control Station hostname to be same as in storage domain:       
        [nasadmin@VNX137 ~]$ hostname       
        VNX137       
        [nasadmin@VNX137 ~]$ hostname -i       
        10.31.139.137       
        [nasadmin@VNX137 ~]$ grep initiator /nas/dos/slot_3/boot.cfg       
        setinitiator cs_hostname=Celerra_VNX137 cs_external_ip=10.31.139.137       
       
        [nasadmin@VNX137 ~]$ /nas/sbin/navicli -h spa domain -list       
        Node:                 VNX_137       
        IP Address:           10.31.139.141 (Master)       
        Name:                 SPA       
        Port:                 80       
        Secure Port:          443       
        IP Address:           10.31.139.142       
        Name:                 SPB       
        Port:                 80       
        Secure Port:          443       
        IP Address:           10.31.139.137       
        Name:                 Celerra_VNX137       
        Port:                 80       
        Secure Port:          443       
       
       
        [nasadmin@VNX137 ~]$       
       
        If the external IP of Primary Control Station is not matching then follow the Scenario 3 in KBA https://support.emc.com/kb/448653 otherwise go further.       
       
        As root run below commands:       
        # /nas/http/webui/bin/domain_admin -i <CS_IP>        
        # /nas/sbin/naviseccli -h SPA -user sysadmin -password sysadmin -scope 0 domain -messner -add -system <CS_IP>       
        # /nas/http/webui/bin/update_domain_directory.pl       
       
        Then verify the updated domain files:       
        [nasadmin@VNX137 ~]$ ll /nas/http/domain       
        total 13       
        drwxr-xr-x 2 apache apache 1024 Sep  1 10:20 bak       
        -rw-r--r-- 1 apache apache   64 Sep  8 12:00 domain_list       
        -rw-r--r-- 1 apache apache  194 Sep  8 12:00 domain_master       
        -rw-r--r-- 1 apache apache 6764 Sep  8 12:00 domain_provider_xml       
        -rw------- 1 apache apache   58 Sep  8 12:00 domain_users       
        -rw-r--r-- 1 apache apache   47 Sep  1 10:20 domain_version       
        drwxr-xr-x 2 apache apache 1024 Nov 14  2013 ldap       
        -rw-r--r-- 1 apache apache    0 Aug 23 14:46 lock_file       
       
        Now, login to Unisphere to verify if the alert is cleared and the File options are up.