Avamar Gen4T: Security scanning causes hwfaultd, supervisord, ipmiutil_evt to crash

           

   Article Number:     512581                                   Article Version: 3     Article Type:    Break Fix 
   

 


Product:

 

Avamar Data Store Gen4T,Avamar,Avamar Server

 

Issue:

 

 

Avamar release notes (7.5, 7.4) mentions the following:        
       
        As part of every Avamar release, the product is scanned for vulnerabilities using at least two common vulnerability assessments tools. This release was scanned with Foundstone and Nessus. The Avamar solution has also been scanned by various customers by using tools such as eEye Retina without issue. However, it is possible that the usage of other port/vulnerability scanners might cause disruption to normal operation of the Avamar server. Therefore, it might be necessary to disable scanning of the Avamar server if problems occur.       
       
        For Avamar Gen4T hardware, the vulnerability scanning results in the following two scenarios:        
       
        1. Avamar Administrator GUI shows the following notification message about "BMC Internal Watchdog Trip"        
       
        grep -i "BMC.*Watchdog" /var/log/messages 
   
        

      May 23 04:41:57 test-gen4t-02 ipmiutil: igetevent-gen4t: 006d 05/16/17 05:01:54 MAJ BMC BMC Internal HW Watchdog Trip #e1 Cold Reset Mode (asserted) 6f [00 ff ff]           
            May 23 04:41:58 test-gen4t-02 ipmiutil: igetevent-gen4t: 0082 05/23/17 04:41:43 MAJ BMC BMC Internal HW Watchdog Trip #e1 Cold Reset Mode (asserted) 6f [00 ff ff]
   
   
    Note: In the above output, this message was generated from the node with hostname "test-gen4t-02"        
       
        The above messages also generate an automatic dial home message.        
       
        2. Avamar Administrator GUI repeatedly displays old hardware messages whenever the Avamar hardware monitoring services are restarted due to the vulnerability scanning:        
       
        Following messages (or similar messages) can be seen in the /data01/cur/err.log (on the storage node) 
   
        
      Sep 21 03:14:01 test-gen4t-00 ipmiutil: igetevent-gen4t: 0044 06/09/17 21:15:40 MAJ BIOS Processor Errors #05 CPU Socket  Socket:1 70 [82 01 ff] - ELOG(27) Empty CPU Socket: 00000001           
            Oct 12 00:19:45 test-gen4t-00 ipmiutil: igetevent-gen4t: 0044 06/09/17 21:15:40 MAJ BIOS Processor Errors #05 CPU Socket  Socket:1 70 [82 01 ff] - ELOG(27) Empty CPU Socket: 00000001
   
   
    Note that in the above messages, the actual error occurred on the date "06/09/17" but was repeated again on Sept 21st and Oct 12th       
       
        In both these cases, syslogs will have corresponding messages related to vulnerability scanning.        
       
        grep -i "sshd.*invalid user" /var/log/messages 
   
        
      Sep 21 02:50:04 test-gen4t-00 sshd[4857]: Invalid user NoSuchUser from 10.10.10.245         
          Sep 21 02:50:04 test-gen4t-00 sshd[4857]: Failed none for invalid user NoSuchUser from 10.10.10.245 port 42885 ssh2         
          Sep 21 02:51:02 test-gen4t-00 sshd[6012]: Invalid user NoSuchUser from 10.10.10.245         
          Sep 21 02:51:02 test-gen4t-00 sshd[6012]: Failed none for invalid user NoSuchUser from 10.10.10.245 port 48180 ssh2         
          ...         
          Oct 11 22:03:09 test-gen4t-00 sshd[25493]: Invalid user NoSuchUser from 10.10.10.245         
          Oct 11 22:03:09 test-gen4t-00 sshd[25493]: Failed none for invalid user NoSuchUser from 10.10.10.245 port 50579 ssh2         
          Oct 11 22:04:54 test-gen4t-00 sshd[27676]: Invalid user NoSuchUser from 10.10.10.245         
          Oct 11 22:04:54 test-gen4t-00 sshd[27676]: Failed none for invalid user NoSuchUser from 10.10.10.245 port 36524 ssh2
     
          
   
      Note: In the above output, the security scanning is originating from the IP address 10.10.10.245    
                                                             

 

 

Cause:

 

 

Vulnerability scanning causes Avamar Gen4T hardware monitoring services to crash and report false positive messages.        
        Avamar engineering team has recreated this issue and is working on a permanent fix for this issue. 
                                                           

 

 

Resolution:

 

 

   
    It is recommended to whitelist the Avamar Servers from vulnerability scanning / security scanning / port scanning. That will ensure that the false positive messages are not generated