ECS: CAS Migration Errors (-10014) Due to Prev.Clip Metadata

           

   Article Number:     534005                                   Article Version: 2     Article Type:    Break Fix 
   

 


Product:

 

ECS SD,Elastic Cloud Storage,ECS Appliance Hardware Series,ECS Appliance Software with Encryption,ECS Appliance Software without Encryption,ECS Software,ECS Software 2.2,ECS Software 3.1,ECS Software 3.2

 

Issue:

 

 

At code versions lower than 3.3 GP1 an ECS non-native CAS migration could have failures due to the prev.clip value in the c-clip metadata.    
   
    During the migration -10014 errors would be sent by the ECS and those would correspond with errors in dataheadsvc.log of the nodes that received this transaction.   

[1186605060,sdk,app/59/WRITE_CLIP,-49da99c0:169b6429916:5adb,1] INFO CasTransactionLogger.java (line 24) WRITE MIGRATIONAPP 54JGRJQE4U9I7e756JSU146JT05NOTAREALCLIP0B6EGMUNFALQFHC -10014 76 1859    
                                                             

 

 

Cause:

 

 

This issue happens because during the write process of a c-clip that has the metadata value prev.clip populated, the ECS dataheadsvc sees this value and tries to confirm that the previous clip exists. However, since this is a new write that check fails and then causes the entire write to fail which is the source of the -10014 error. In the example below the object C5EP91SF2U31HeNOTACLIPDK7P8G41AHJUV7FA0B6EGMUNFALQFHC was a value in the metadata of clip 54JGRJQE4U9I7e756JSU146JT05NOTAREALCLIP0B6EGMUNFALQFHC and gets rejected in the logs with the error NOT_FOUND.   
        

[1186605060,sdk,app/73/WRITE_CLIP,-49da99c9:169b6429016:5ae6,1-0a325806:1696888061c:227a:37a32] INFO ObjectControllerExceptionHelper.java (line 244) Method setHeadSystemMetadataForObject ObjectInfo: [id=9e18b175b50175e7f727527anotanobjectba2798fce6119ef4c8eb, CoS=urn:storageos:VirtualArray:ec07772e-3608-4b99-bb1c-4575891d611c.urn:storageos:ReplicationGroupInfo:xadhttreal-109b-4fce-af68-d2db86a948fc:global, FAMode=disabled, PreviousFAMode=, FAStartVersion=0, FAEndVersion=0, ObjectOwnerZone=null, objectOwnerRecordListnull, keyName=C5EP91SF2U31HeNOTACLIPK7P8G41AHJUV7FA0B6EGMUNFALQFHC] hit error, failed due to com.emc.storageos.data.object.exception.ObjectControllerException: directory server returns error ERROR_OBJECT_NOT_FOUND requestId 0a325806:1696888061c:227a:37a33[1186605060,sdk,c003n01/73/WRITE_CLIP,-49da99c9:169b6429016:5ae6,1] ERROR AwaitEOF.java (line 263) app/73/WRITE_CLIP: Failed to update object metadata, error is NOT_FOUND[1186605060,sdk,app/73/WRITE_CLIP,-49da99c9:169b6429016:5ae6,1] ERROR AbstractTransaction.java (line 136) Failed to execute the transaction, aborting.[1186605060,sdk,app/59/WRITE_CLIP,-49da99c9:169b6429016:5adb,1] INFO CasTransactionLogger.java (line 24) WRITE MIGRATIONAPP 54JGRJQE4U9I7e756JSU146JT05NOTAREALCLIP0B6EGMUNFALQFHC -10014 76 1859    
                                                             

 

 

Resolution:

 

 

This issue is resolved after an upgrade to 3.3 GP1.