Network interfaces are not reachable on Data Domain Restorers (DDRs) after upgrading to Data Domain Operating System (DDOS) 5.7

           

   Article Number:     463360                                   Article Version: 5     Article Type:    Break Fix 
   

 


Product:

 

Data Domain, Data Domain Software, DD OS 5.7 5.7.0.4

 

Issue:

 

 

Network communication fails to an existing Data Domain Restorer (DDR) immediately after upgrading to Data Domain Operating System (DDOS) 5.7.       
       
        There are various symptoms associated with this issue which may include (but are not limited to):
   

         
  •         A lack of response to ICMP ping requests sent to some or all of the network interfaces on the DDR from some or all network clients     
  •      
  •         Traceroutes from some or all network clients revealing a 'hole' before reaching the DDR and not being able to reach their ultimate destination     
  •      
  •         Backups to the DDR from some or all network clients fail when they were working without issue immediately prior to the upgrade to DDOS 5.7     
  •    
                                                             

 

 

Cause:

 

 

   

      In versions of DDOS prior to DDOS 5.7 reverse path forwarding was configured in 'loose' mode. This means that:   

   
         
  •         A packet arrives at an interface on the DDR     
  •      
  •         The DDR checks that it has at least one interface which has a route back to the source address from which the packet originated     
  •      
  •         As long as this is true the packet is accepted but any replies may be sent from a different interface on the DDR (which has a route back to the source address)     
  •    
   

      In DDOS 5.7, however, reverse path forwarding is, by default, configured in 'strict' mode. This means that:   

   
         
  •         A packet arrives at an interface on the DDR     
  •      
  •         The DDR checks that the same interface has a route back to the source address from which the packet originated     
  •      
  •         If this is not the case the packet is dropped (even if an alternate interface on the system has a route back to the source address)     
  •    
   

      This issue can prevent network communication in environments where asymmetric routing is configured (the return path for communications is different from the sending path).     
          

                                                             

 

 

Resolution:

 

 

Various steps have been taken to mitigate this issue:   

         
  •         In DDOS 5.7.2.0 and later strict reverse path forwarding is disabled by default     
  •      
  •         In DDOS 5.7.1.0 and later commands are available via the DD CLI (DDSH) to disable strict reverse path forwarding on interfaces     
  •    
As a result:   
         
  •         Affected systems should be upgraded to DDOS 5.7.2.0 or later to completely disable strict reverse path forwarding     
  •    
   
         
  •         If an affected system is running DDOS 5.7.1.0 or later but is not able to be upgraded to DDOS 5.7.2.0 or later strict reverse path forwarding can be disabled for a specific interface as follows:     
  •    
   
      # system show serialno         
          [system serial number displayed]         
          # priv set se         
          [password prompt - enter serial number from above]         
          # net option set net.ipv4.conf.[NIC].rp_filter 0         
          Set "net.ipv4.conf.[NIC].rp_filter" to "0".
       
       
        Note that [NIC] in the above commands should be substituted for the corresponding network interface (NIC) on the DDR.
   
   
         
  •         If a system is running a version of DDOS 5.7.x prior to DDOS 5.7.1.0 and an upgrade is not possible the following steps should be taken:     
  •    
   
      Verify that any communication between network clients (i.e. media/storage nodes) and the DDR is not asymmetrical, i.e. any interface used to receive a packet on the DDR will also be used to send responses back to the source address         
         
          If asymmetric communications/routing are detected the issue can be resolved by adding static routes on the DDR to ensure that the interface receiving communications on the DDR will also be used to send any responses (i.e. avoid asymmetric routing)
   
   
    It is also possible to manually modify /etc/sysctl.conf to disable strict reverse path forwarding for all/a subset of network interfaces. If this is required please contact your contracted support provider.