Dell EMC Unity:The storage resource is offline. This may be caused by its storage elements being offline. Please contact your service provider: The storage resource is offline. (User Correctable)

           

   Article Number:     530118                                   Article Version: 2     Article Type:    Break Fix 
   

 


Product:

 

Dell EMC Unity Family,Dell EMC Unity All Flash

 

Issue:

 

 

Thick provisioned File system resource went offline.    
   
     /EMC/backend/log_shared/EMCSystemLogFile.log   
    2019/01/27-12:04:22.196849  18K     7FCAB3651704     cbfs:CBFSA: UFS: 6: UFSCore_Vnode::testAndMakeOwnedMappingChain(): fsid:1073741841 inum:9426: Direct Flag Not Found at level:2<NL>   
    2019/01/27-12:04:22.196851    8     7FCAB3651704     cbfs:CBFSA:  Direct Flag first found at level:3: Mapping:0xffd400100003bfe AU:0x1c41fc ausz:1   
    2019/01/27-12:04:22.196858   13     7FCAB3651704     cbfs:CBFSA: UFS: 3: In UFS64_MappingChainInfo::setDirectFoundInChain: D flag set at unexpected level during re-evaluating the <NL>   
    2019/01/27-12:04:22.196859    7     7FCAB3651704     cbfs:CBFSA: IB chain. Inode=0, FirstLevelDirectFound=3, FirstLevelDirectExpected=1   
    2019/01/27-12:04:22.196866   13     7FCAB3651704     cbfs:CBFSA: UFS: 3: UFSCore_Vnode::testAndMakeOwnedMappingChain(): fsid:1073741841 inum:9426: Direct Flag Unexpectedly Found <NL>   
    2019/01/27-12:04:22.196867    7     7FCAB3651704     cbfs:CBFSA: at level:3 Direct Flag Expected at level:1: Mapping:0x2ffff8010000076b AU:0x1c41fc ausz:1   
    2019/01/27-12:04:22.196872   11     7FCAB3651704     cbfs:CBFSA: UFS: 3: Unmounting fs 1073741841: Reason: Direct Flag Found at wrong level failure   
    2019/01/27-12:04:22.196881   15     7FCAB3651704     cbfs:CBFSA: CBFS Event Id 77579616268, Event Message: FsId: 1073741841 encountered a critical fault, action <NL>   
    2019/01/27-12:04:22.196901   27     7FCAB3651704     cbfs:CBFSA: CBFS Event Id 77579616268, Event Message: (FREEZE_FS_RECOVERY_REQUIRED), FS (File System taken offline (Direct <NL>   
    2019/01/27-12:04:22.196907   11     7FCAB3651704     cbfs:CBFSA: CBFS Event Id 77579616268, Event Message: Flag Found at wrong level failure)) is unmounted internally.   
    2019/01/27-12:04:22.198229 1328     7FCAB3651704     cbfs:CBFSA: CFS: 3: Panic Avoidance Exception Taken on Fs 1073741841   
    2019/01/27-12:04:22.198232    9     7FCAB3651704     cbfs:CBFSA: CFS: 3: Exception callers = 0, 0, 0, 0, 0, 0, 0   
    2019/01/27-12:04:22.198243   17     7FCAB3651704     cbfs:CBFSA: CBFS: 6: internalErrorNotificationCallback fsmo of 1073741841 start to freeze. Recovery needed.   
    2019/01/27-12:04:22.198246   10     7FCAB3651704     cbfs:CBFSA: UFS: 3: Panic Avoidance found 1073741841 corrupted (Direct Flag Found at wrong level failure)   
    2019/01/27-12:04:22.198250   10     7FCAB8E74705      mlu:E MluFSMgrFSObjectCBFSErrorNotify[7389] CBFS Error 0x8, Status 0x31 for FSID  0x40000011   
    2019/01/27-12:04:22.198257   13     7FCAB3651704     cbfs:CBFSA: UFS: 3: UFSCore_Vnode::testAndMakeOwnedMappingChain(): Error fsid:1073741841 inum:9426 PrevState:3 <NL>   
    2019/01/27-12:04:22.198286    7     7FCAB3651704     cbfs:CBFSA: InternalError   
    2019/01/27-12:04:22.198287    8     7FCAB2485702     cbfs:CBFSA: UFS: 6: cancelFlushWaiters fsid:1073741841 finished   
    2019/01/27-12:04:22.198290    9     7FCAB3651704     cbfs:CBFSA: CFS: 3: File_NamingNode::map_for_write map_for_write failed fsid:1073741841 status:InternalError   
    2019/01/27-12:04:22.198291    7     7FCAB2485702     cbfs:CBFSA: UFS: 6: FSDataLog::cleanupDirtySnaps(3157): fsid=1073741841   
    2019/01/27-12:04:22.198301   16     7FCAB3651704     cbfs:CBFSA: CBFS: 3: APIObj:0x7fcc62425d38, FNN:MFW(0x1c41fc0, 0x10) for fsid/ino/flag: 1073741841/9426/0x0  failed.  File <NL>   
    2019/01/27-12:04:22.198302    7     7FCAB3651704     cbfs:CBFSA: status: InternalError   
    2019/01/27-12:04:22.198306   10     7FCAB2485702     cbfs:CBFSA: CBFS: 6: outstandingProcess: process 0x7fcc965212a8 is running which may fail FS unmount, APIObj:0x7fcc622e7858, <NL>   
    2019/01/27-12:04:22.198307    7     7FCAB2485702     cbfs:CBFSA: FSID:1073741841, API:TRUNCATE_FILE, State:Ready, API Status:0   
    2019/01/27-12:04:22.198309    7     7FCAB3651704     cbfs:CBFSA: CBFS: 3: CBFS_APIObject_Mapping (obj:0x7fcc62425d38-MAP_FOR_WRITE): sending error callback: InternalError, <NL>   
    ....   
    2019/01/27-12:04:22.198351    7     7FCAB3651704      mlu:E CompletionRoutine[1139] LV::Complete Irp 0x108f0890, 0xA00000C31 St=0x7E11F07B Info=0x0 dec 0   
    2019/01/27-12:04:22.198352    7     7FCAB2485702     cbfs:CBFSA: STORAGE: 6: SVManagedObject::abort - abort done.   
    2019/01/27-12:04:22.198356   11     7FCAB3651704    redir:MR: G0x7fcd9c6b39c0E Local I/O failed V0x7fcd9c8af2c0, Status 7E11F07B, retryable 1   
    2019/01/27-12:04:22.198360    9     7FCAB3651704      std:DVL-0xf4c92f0 IO failed. status=0x7e11f07b type=IRP_MJ_INTERNAL_DEVICE_CONTROL block=29630400 bytes=8192 panic=1 drop=0   
    2019/01/27-12:04:22.198382   28     7FCAB3651704     sade:STORAGE: 4:[core]  DVLD-DVL20000c2f IO failed. status=0x2 type=4 block=29630400 bytes=8192   
    2019/01/27-12:04:22.198398   22     7FCAB3651704     sade:STORAGE: 3:[core]  RawIO_IRP failed, deviceId is 18446744073709551615, irp start is 29630400, irp op length is 8192, BlockIO_St   
    2019/01/27-12:04:22.198399    8     7FCAB3651704     sade:STORAGE: 3:[core]  atus is 2.   
    2019/01/27-12:04:22.198435   42     7FC99BF3D705     sade:UFS: 3:[core]  UFSCore_VNode::blockChainWriteInternal(1221): Error fsid=536874031 inum=98570 IO error - Status = 2(DeviceError)   
    2019/01/27-12:04:22.198437    8     7FC99BF3D705     sade:UFS: 3:[core]   
    2019/01/27-12:04:22.198449   18     7FCAB8B7D704     cbfs:CBFSA: UFS: 3: cleanUnmount fs is marked corrupt fs 1073741841   
    2019-01-27T12:05:14.733Z" "xxxxxxxx_spa" "Kittyhawk_safe" "4843" "unix/spa/root" "INFO" "13:10330003" :: "NDMP Session 990 (thread ndmp990) will start backup for /zzzz/zzzz. NDMP version: 4, backup type: dump." :: Category=System Component=DART_NDMP    
    "2019-01-27T12:06:52.100Z" "xxxxxxxxxx_spa" "Neo_CEM" "23331" "N/A" "CRIT" "14:600f4" :: "File system  zzzzz  is offline." :: Category=User Component=Health   
   
   
     
                                                           

 

 

Cause:

 

 

Code issue that affects 'Thick' provison file system. Thick provisioned File system have a (DLU) Lower Deck, issue is manifested after the last snapshot of a Thick LUN or a user filesystem is deleted, it triggers an internal process called Defragmentation  (also know as File Mode Conversion Prototcol or FMCP). When the Defragmentation operation failed for Misallocation, it  hit the "Direct Flag Not Found" issue  which caused the File system  to go offline.   
   
     
                                                           

 

 

Change:

 

 

 Unity OE was upgraded to 4.3.1.1531187824.1531200778  and user  created a File system as Thick File system (DLU)                                                           

 

 

Resolution:

 

 

Fix:   
    A permanent fix for this issue will be included in a future Unity OE release.   
   
    Workaround:    
    The workaround for a THICK file system going offline during defragmentation is to create a User Snap [Thin] to avoid the exposure to this bug.   
    Refrain from deleting the last snapshot of a Thick LUN or a user filesystem. This avoids the defragmentation process.   
    If the user has already deleted the last snapshot and a defragmentation is in progress, then create a dummy snapshot. Creating a snapshot immediately stops defragmentation.   
   
     
                                                           

 

 

Notes: