Data Domain: DDVE Filesystem Down / Unable to Enable DDVE Filesystem

           

   Article Number:     531823                                   Article Version: 4     Article Type:    Break Fix 
   

 


Product:

 

Data Domain,Data Domain Software

 

Issue:

 

 

DDVE filesystem down.   
    Active tier disk is not available.   
    Active tier shows:   

      sysadmin@DD1# filesys show space          
          Active Tier:         
          Resource                  Size GiB       Used GiB     Avail GiB    Use%   Cleanable GiB*         
          ----------------          --------       --------     ---------    ----   --------------         
          /data: pre-comp                -              -           -          -                -         
          /data: post-comp               -              -           -          -                -         
          /ddvar                       49.2            10.0        36.7       21%               -         
          /ddvar/core                 158.5             7.0       143.5        5%               -         
          ----------------          --------         --------  ---------     ----   --------------         
          The filesystem is unavailable at this time.
   

                                                             

 

 

Cause:

 

 

   

      System Environment where DDVE is configured on ESX connected with external storage device.     
     
      DDVE configured on ESX connected with external storage     
     
      The above said issue is occurred when external storage device loses connectivity with ESXi host.     
      The following error reported in ESX server vmkernel log    

   

      Log Location #cd /var/log/vmkernel.log       
       
        cpu4:2098003)ScsiDeviceIO: 3032: Cmd(0x45a2c0b7ca80) 0x2a, CmdSN 0x3bf from world 2305853 to dev "naa.6006016001f049003b705c5cde10cdfc" (vmhba5:C0:T5:L1) failed H:0x8 D:0x0 P:0x0       
        cpu8:2098003)ScsiDeviceIO: 3032: Cmd(0x459ac13b9340) 0x28, CmdSN 0x2e7 from world 2305853 to dev "naa.6006016001f0490045705c5c730bd671"  (vmhba5:C0:T5:L7) failed H:0x8 D:0x0 P:0x0       
        cpu7:2098003)ScsiDeviceIO: 3032: Cmd(0x45a2c0a2a380) 0x88, CmdSN 0x368 from world 2305853 to dev "naa.6006016001f049003d705c5ccf627f86"  (vmhba5:C0:T5:L3) failed H:0x8 D:0x0 P:0x0       
        cpu8:2098003)ScsiDeviceIO: 3032: Cmd(0x45a319d16c00) 0x8a, CmdSN 0x33e from world 2305853 to dev "naa.6006016001f049003f705c5c47140d37" (vmhba5:C0:T5:L5) failed H:0x8 D:0x0 P:0x0       
        cpu5:2098003)ScsiDeviceIO: 3047: Cmd(0x45a31708cb00) 0x2a, CmdSN 0x39a from world 2305853 to dev "naa.6006016007f04900c272965b8bff4a05" (vmhba5:C0:T5:L11) failed H:0x8 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0       
       
        cpu0:66299)ScsiDeviceIO: 2954: Cmd(0x4397ba4ea9c0) 0x4d, CmdSN 0x88555a from world 613487900 to dev "naa.500003981809326d" failed H:0x0D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.       
        cpu10:615767554)<6>usb usb2: resumed       
        cpu20:615767554)<6>usb 2-1: resumed       
        cpu1:66060)<6>usb 2-1: suspended       
        cpu3:66074)<6>usb usb2: suspended       
        cpu10:613606886 opID=9b42f169)World: 12235: VC opID esxcli-4e-1f1a maps to vmkernel opID 9b42f169       
        cpu10:613606886 opID=9b42f169)vmw_ahci[00000011]: scsiDiscover:channel=0, target=0, lun=0, action=0       
        cpu10:613606886 opID=9b42f169)vmw_ahci[00000011]: scsiDiscover:No media       
        cpu10:613606886 opID=9b42f169)vmw_ahci[00000011]: scsiDiscover:channel=0, target=1, lun=0, action=0       
        cpu10:613606886 opID=9b42f169)vmw_ahci[00000011]: scsiDiscover:No media       
        cpu10:613606886 opID=9b42f169)vmw_ahci[0000001f]: scsiDiscover:channel=0, target=0, lun=0, action=0       
        cpu10:613606886 opID=9b42f169)vmw_ahci[0000001f]: scsiDiscover:port initialization not complete       
        cpu10:613606886 opID=9b42f169)vmw_ahci[0000001f]: scsiDiscover:channel=0, target=1, lun=0, action=0
     
          

   

      From the logs you can see the following devices have failed.-   

   

      naa.6006016001f049003b705c5cde10cdfc       
        naa.6006016001f0490045705c5c730bd671       
        naa.6006016001f049003d705c5ccf627f86       
        naa.6006016001f049003f705c5c47140d37       
        naa.6006016007f04900c272965b8bff4a05       
        naa.500003981809326d
   

   

      In addition to the above the following alerts will be posted.   

   

      cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L3 to APM00183214618 is dead.         
          cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L0 to APM00183214618 is dead.         
          cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L7 to APM00183214618 is dead.         
          cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L5 to APM00183214618 is dead.         
          cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L9 to APM00183214618 is dead.         
          cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L11 to APM00183214618 is dead.         
          cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L6 to APM00183214618 is dead.         
          cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L4 to APM00183214618 is dead.         
          cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L2 to APM00183214618 is dead.         
          cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L8 to APM00183214618 is dead.         
          cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L10 to APM00183214618 is dead.         
          cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L1 to APM00183214618 is dead.
     
          

   

      In addition, there will be frame drop events on “vmhbaX” noted.   

   

      vmhba5: FRAME DROP event observed suggests a problem with Fibre Channel link/switch!.   

                                                             

 

 

Resolution:

 

 

DDVE filesystem should get enabled after completing the following two tasks.   

         
  1.         Check the connection path (vmhbaX:C0:T5:LX) and re-establish the connection between switch and external storage device.     
  2.      
  3.         After connection is established, Confirm no further failed devices are reported in "vmkernel.log".     
  4.      
  5.         Reboot DDVE.     
  6.    
   

      If the issue still exists after the above mentioned tasks are complete then collect the following and engage DELL EMC DD Support.   

   
         
  1.         Support bundle from DDVE.     
  2.      
  3.         Generate Bundle from ESXi server.     
  4.      
  5.         Logs from connecting switch.     
  6.