VPLEX: Slowness in GUI/CLI operations on VS6 VPLEX Metro with the GeoSynchrony version 6.1.x

           

   Article Number:     533574                                   Article Version: 4     Article Type:    Break Fix 
   

 


Product:

 

VPLEX Series,VPLEX Local,VPLEX Metro,VPLEX GeoSynchrony 6.1,VPLEX GeoSynchrony 6.1 Patch 1,VPLEX GeoSynchrony 6.1 Patch 2,VPLEX VS6

 

Issue:

 

 

Slowness in GUI/CLI operations seen on VS6 following upgrade to GeoSynchrony 6.1.x, below is a list of operations impacted:   

         
  •         Addition of virtual volume to a Storage View ( local/distributed )     
  •      
  •         Removing a virtual volume from a Storage View ( local/distributed )     
  •      
  •         Listing of Virtual Volume ( local/distributed )     
  •      
  •         Attaching a mirror to a device     
  •      
  •         Discard a mirror from a device     
  •      
  •         Add virtual volume to Storage-View and/or CG     
  •      
  •         Remove a virtual volume from a Storage-View and/or CG     
  •      
  •         Add Initiator to a Storage View     
  •      
  •         Remove Initiator from Storage View     
  •      
  •         System Status/Cluster Status     
  •    
        
    1. The CPU utilization of the Java process goes beyond 100% - 150% during the operations   
        executed from GUI   
   
        Example:   
      service@ManagementServer:/ ~> top -c     
      21660 service 20 0 4224m 1.7g 31m S 241 46.2 92:00.54 java   
        
    2. Memory utilization is consistently high - +90%   
          
        Example:   
        
      service@ManagementServer:/ ~> top -c     
      Mem: 3911272k total, 3417448k used, 493824k free, 133116k buffers   
        
    3. Operations like listing of virtual volumes and addition/deletion of virtual volumes to a storage   
        view take time.   
        
    4. Number of REST calls on a given day from each IP is very high.   
   
        Example:   
      service@ManagementServer:/var/log/VPlex/cli> grep -i "REST" <latest vplex-ms-stats file> | wc -l     
      29934         
          service@ManagementServer:/var/log/VPlex/cli> grep -i "REST" <latest vplex-ms-stats file> | grep "GET" | awk -F':' '{print $7}' | sort | uniq
     
      xx.xx.xx.153     
      xx.xx.xx.154   
        
    5. Number of FAPI* calls on a given day is very high   
          
         Note: FAPI is related to when RecoverPoint is configured on the VPLEX   
   
        Example:   
      service@ManagementServer:/var/log/VPlex/cli>grep -i "FAPI" <latest vplex-ms-stats file> | wc -l     
      28679   
                                                             

 

 

Cause:

 

 

   

         
  •         Following upgrade to GeoSynchrony 6.1.x we start to see slowness/performance issues in the GUI, this is due to checks being made for the eLicense.     
  •      
  •         FAPI calls are SOAP calls over HTTPS made from VPLEX to RP to fetch/refresh RP properties. High number of FAPI calls made to RP and REST calls made by RP which are adding to the performance issue noted on VPLEX.     
  •      
  •         The RecoverPoint (RP) is behaving as per design, to be clear it is just the amount of polling made by RP which is adding to the performance issue noted on VPLEX.     
  •      
  •         Operations like addition/deletion of multiple virtual volumes from the VPLEX GUI is serialized. Below scenario will explain this.     
  •    
   
         
  1.         When we try to add/remove one virtual-volume (VV) it results in 6 FAPI calls to RP to  fetch/refresh.     
  2.      
  3.         RP properties.     
  4.      
  5.         Now assume the add/remove operations done via the GUI for 10 VV, this would result in (10 * 6 = 60) FAPI calls to fetch/refresh RP properties. Since the operation from the GUI is serialized for each virtual volume operation VPLEX will make 6 FAPI calls thus  resulting in a total of 60 calls for 10 volumes. This results in lot of handshake exchanges between VPLEX and RP and eventually results in performance issue.     
  6.      
  7.          The add/remove operations done via CLI for 10 VV, would result in only in ONE set  of 6 FAPI calls to the RP.     
  8.    
   
           Hence we will see a definite performance gain for the end user post de-registering the RP     
           from VPLEX.   
                                                             

 

 

Resolution:

 

 

1. Apply the actions listed in the Notes section below. Implementation of those actions will make   
        the response time of the cluster status and the GUI refreshes faster.   
   
        Info about the "license show" and "security list-certificates" commands were   
        added checks as part of 'cluster status' in 6.1.x accounts for the slowness of these    
        commands.   
          
        There is a script that can be loaded that will bypass the license and certificate checks,   
         thus improving the response time. You will need to engage a Dell EMC Customer Service    
         to load the script. Mention this article when contacting support.    
   
        There will be no impact seen and end users can verify the status of license and certificates   
         using the commands "license show" and "security list-certificates" respectively on the   
         VPlexcli level.   
   
    2. Make sure to have the least possible number of active open GUI sessions at a given time.   
        Try to have ONE at a time.   
   
        Reason : For each active GUI session there will be regular GUI refreshes initiated on the   
        system adding to the load on the system.   
   
    3. Check if the number of REST calls on a given day from each IP are very high, the   
        recommendation is to reduce the polling to at-least a gap of 20 minutes.   
   
    4. De-register RPAs from both VPLEX cluster.   
   
        Example:   
        Get the RPA cluster IP address(es) as follows:   
        

      VPlexcli:/> ls recoverpoint/rpa-clusters   
   
     
           De-register RPA clusters: (to be done for all the registered RP clusters):   
   
     
      VPlexcli:/recoverpoint/rpa-clusters> rpa-cluster remove -r <rp-cluster-ip>     
          
          Confirm that all the registered RPA clusters are de-registered (Below command should not   
          list any RP clusters)   
        
      VPlexcli:/> ls recoverpoint/rpa-clusters     
          
    Note: Impact of de-registering RP from VPLEX   
        ================================================   
         
  •         When you unregister RPA clusters from VPLEX the control path ceases to exit. So management operations will get affected, including enabling RP protection for the CG. The I/O path replication will not be impacted. Once de-registered, to manage the RPA you will need to login to the RPA. Same way, if you need to manage VPLEX, you will need to login to VPLEX to do the management operations.     
  •      
  •         Operations/steps that would be no longer possible to the RPA from the GUI or CLI due to the fact that of the RPA de-registration on the VPLEX.        
               
    1.             recoverpoint/rpa-clusters> ll          
    2.          
    3.             recoverpoint/rpa-clusters> rp summary         
    4.          
    5.             recoverpoint/rpa-clusters> rp validate-configuration         
    6.          
    7.             ndu will not put RPAs to maintenance mode with the RPA de-registered, the VPLEX will not detect it in the ndu pre-check so this is why the RPAs are not placed in maintenance mode during an NDU, as would normally occur if they were registered and seen by the VPLEX.         
    8.        
                  
  •    
   
                           NOTE:  The RPAs need to be re-registered prior to a VPLEX NDU to ensure the           
                                 VPLEX can put them into maintenance mode at the start of the NDU.  This is
   
   
                           critical to ensure a successful NDU.)   
   
         
  •         After re-registration of the RPA(s), auto-sync up will take place if there was a disk already provisioned before the de-registration was done.     
  •    
    If after following the above steps if the issue persists contact Dell EMC Customer Support for further assistance and mention this article.