ECS: Transformation Errors and or Performance issues caused by Centera Nodes being overloaded[1]

           

   Article Number:     533591                                   Article Version: 9     Article Type:    Break Fix 
   

 


Product:

 

Elastic Cloud Storage,ECS Appliance,ECS Software,ECS Appliance Software with Encryption,ECS Appliance Software without Encryption,ECS Appliance Hardware Series

 

Issue:

 

 

1. Transformation is experience performance issues or is seeing errors in Transformation Tool   
    2. Customer is experiencing application errors during Centera to ECS Migration    
     
                                                           

 

 

Cause:

 

 

Transformation performance or errors may occur when the Centera access nodes are overloaded with connections.    
   
    On the Centera, each access node can support up to 100 connections simultaneously. Any additional connections made to the access node will be dropped.   
   
    If the customer applications are still operating on the Centera, or if the ECS transformation threads are misconfigured to a high number, then the access nodes may get overloaded. This in turn will  affect the performance of the ECS Transformation. 
                                                           

 

 

Resolution:

 

 

                                                                       

First, confirm that the Centera access nodes are indeed overloaded. Reference Level 40 Internal KB 471191, Centera: SDK errors due to access nodes reaching max established connections.           
                          
               Once confirmed, then the ECS Transformation thread count needs to be reduced using the following steps. (Please get customer approval first! as this may affect customer read/writes to the Centera done through the ECS Native Transformation)                          
                 
  1.                 Pause the Transformation activity (for example Indexing) and wait for 60 minutes as the threads may need time to finish. Click on Abort to pause the Transformation. The active threads should come down to zero once you Abort/Pause it.               
                                                            
    # viprexec -i "grep 'the active thread' /opt/emc/caspian/fabric/agent/services/object/main/log/transformsvc.log|tail -1"2018-03-19T09:25:42,410 [TaskScheduler-transform-DefaultScheduler-006]  INFO  TaskScheduler  Thread stats - task scheduler TRANSFORMATION_TASK_RUNNER: the active thread 100, core pool size 100, pool size 100, queued tasks 0                
                                     
  2.            
                2. Open a ticket with ECS Support, and Refrence KB 522186 to make a change to drop the amount of  concurrent connections. The value is to be determined depending on current load on the Centera, how many Centera access nodes exist, and how many ECS nodes exist.  For example, if the Centera has 4 access nodes, and the ECS has 8 nodes, then the Centera would be receiving 800 connections also it can only handle 400 between the four nodes (100 connection per node). In that case, the value of the parameter should be at maximum 50. If other customer applications are still accessing the Centera, we should choose an even smaller value to accommodate the application connections to the Centera.                           

             
                 Please provide transformation info:           

                          
curl -s "$(sudo netstat -nap | grep 9101 | grep LISTEN | head -1 | awk '{print $4}')/diagnostic/TT/0/DumpAllKeys/TRANSFORMATION_INFO" |grep -i centera