Data Domain: Reinstate path issue | Abnormal paths in failed_list

           

   Article Number:     539002                                   Article Version: 2     Article Type:    Break Fix 
   

 


Product:

 

Data Domain,Data Domain Deduplication Storage Systems

 

Issue:

 

 

DDOS Bug results in instability on the back-end (SAS) connections to Enclosures/Disks on various Data Domain platforms.   
   
    Applies to:   

         
  •         All Data Domain systems     
  •      
  •         DDOS versions <6.1.2.60 / <6.2.0.40 / <7.0.0.8     
  •    
Symptoms:     
           
Can be a mix of any/all of the following   
         
  •         Single or Multiple failed disks     
  •      
  •         Multiple missing disk or enclosure path alerts     
  •      
  •         Performance degradation     
  •    
Log Entries:   
      Messages.engineering:     
      - Can be viewed in AutoSupport or via DD-CLI command: #log view debug/messages.engineering   
   
      Look out for multiple entries stating:   
   
      ‘PathMon: try to reinstate path’ AND     
      'abnormal paths in failed_list'   
   
     
      Example from 2 different systems:     
      Note: These are posted at 60 Second intervals across the log     
           Jul  9 14:04:19 <DDR1> ssm: INFO: PathMon: try to reinstate path sdkw (dm-171[4.2])       
             Nov  6 19:04:20 <DDR2> ssm: INFO: PathMon: try to reinstate path sdnk (dm-176[5.35])
   
   
      Also entries for ‘abnormal paths in failed_list’        
        Jul  9 14:03:18 <DDR1> ssm: INFO: PathMon: Currently there are 1 abnormal paths in failed_list       
        Nov  6 19:05:21 <DDR2> ssm: INFO: PathMon: Currently there are 8 abnormal paths in failed_list
   
   
          
   
      Kern.info:     
      - Can be viewed in AutoSupport or via DD-CLI command: #log view debug/platform/kern.info   
   
      Multiple DM/multipath ‘reinstating path’ entries for one active path every 60s.     
      Example     
      May  2 23:27:29 <DDR3> kernel: (E6)[   7285385.213946] device-mapper: multipath: Reinstating path sdkw (active:1/degrade:0/insane:0).     
      May  2 23:28:30 <DDR3> kernel: (E6)[   7285446.182758] device-mapper: multipath: Reinstating path sdkw (active:1/degrade:0/insane:0).     
      May  2 23:29:31 <DDR3> kernel: (E6)[   7285507.192361] device-mapper: multipath: Reinstating path sdkw (active:1/degrade:0/insane:0).     
          
                                                             

 

 

Cause:

 

 

When a disk path is failed, one uevent is posted to ssm and pathmon; with ssm reinstating the failed path every 60 seconds.   
    Even if the path is reinstated successfully, another uevent is posted to ssm and pathmon which will not reinstate this path.   
    This results in SAS Back-end instability, unexpected connectivity reporting & behaviour.   
   
   
   
     
                                                           

 

 

Change:

 

 

This is a code defect within a 'list' function of Device-Mapper (DM) in affected DDOS versions which misses the second uevent, resulting in pathmon trying to reinstate the path even if the path has been reinstated already.   
     
                                                           

 

 

Resolution:

 

 

Workaround:   

         
  •         Reboot the DDR unit     
  •    
Fix:   
         
  •         Upgrade to any of the following DDOS versions:                
               
    •             >=6.1.2.60         
    •          
    •             >=6.2.0.40         
    •          
    •             >=7.0.0.8         
    •        
                  
  •      
  •         NOTE: Not ALL DD units support DDOS v7.x  // i.e. Check Release notes     
  •    
                                                             

 

 

Notes:

 

 

This Bug (242087) was originally detected by Engineering Teams & is therefore not a 'Customer Found Defect' (CFD); this means that the fix/change won't be tracked within DDOS Release Notes.   
    The change is tracked internally on 'Change: 626050 Branch: li_6