VPLEX : VPlex cluster shutdown script hung

           

   Article Number:     536538                                   Article Version: 2     Article Type:    Break Fix 
   

 


Product:

 

VPLEX Series,VPLEX for All Flash,VPLEX Geo,VPLEX GeoSynchrony,VPLEX Local,VPLEX Metro,VPLEX VS1,VPLEX VS2,VPLEX VS6

 

Issue:

 

 

In a situation where user is performing VPlex cluster shutdown (Metro/Local), the shutdown script gets hung on either of the cluster (example: cluster-1) and show's below output :   
   
    VPlexcli:/> cluster shutdown --cluster cluster-1     
      Warning: Shutting down a VPlex cluster may cause data unavailability. Please refer to the VPlex documentation for the recommended procedure for shutting down a cluster. To show that you understand the impact, enter 'shutdown': shutdown     
     
      You have chosen to shutdown 'cluster-1'. To confirm, enter 'cluster-1': cluster-1     
     
      Status Description     
      ------------ -----------------------------     
      In progress. Shutdown already in progress.
   
   
    NOTE: In this scenario, user ran shutdown script on Cluster-1 and waited for 3-5 minutes (recommended) for shutdown to complete. However, user finds that script is hung and not getting completed.   
        
    Type the following command to display the cluster summary in which cluster-1 still shows connected as "true" as follows:   
   
    VPlexcli:/> cluster summary     
      Clusters:     
        Name       Cluster ID  TLA             Connected  Expelled  Operational Status  Health State     
        ---------  ----------  --------------  ---------  --------  ------------------  ------------     
        cluster-1  1           FNM0019xxxxxxx  true       false     unknown             minor-failure      
        cluster-2  2           FNM0018xxxxxxx  true       false       ok                  ok     
     
     
      Islands:     
        Island ID  Clusters     
        ---------  --------------------     
        1          cluster-1, cluster-2
   
   
    NOTE: In above scenario , user is trying to shutdown only cluster-1 not cluster-2 which is a part of Metro configuration   
     
                                                           

 

 

Cause:

 

 

The shutdown script got stuck on affected cluster while shutting down for maintenance or relocation.                                                           

 

 

Resolution:

 

 

In-order to resume the shutdown script on affected cluster (example: cluster-1 as shown above), please reboot the affected management server of affected cluster (example: cluster-1 as shown above) by running below mentioned command's and resume the shutdown procedure again. Rebooting the management server kills the running script and allows us to run the command again. Please be informed that rebooting the management server will not make any impact it will just kill the existing running process.    
   
    1) Please follow below KB article to reboot the affected management server (ex. cluster-1) as follows:   
   
    330568 : VPLEX: How to reboot the VPLEX management server.    
   
    2) Please check if the affected management server is back online by re-logging into CLI session. Once confirmed, please resume shutdown script on affected cluster (ex. cluster-1) as follows:    
   
    Login as: service     
      Using keyboard-interactive authentication.     
      Password:     
      service@ManagementServer:~> vplexcli     
      Trying ::1...     
      Connected to localhost.     
      Escape character is '^]'.     
      creating logfile:/var/log/VPlex/cli/session.log_service_localhost_T23474_20190XXXXXXXXX     
     
      VPlexcli:/> cluster shutdown --cluster cluster-1     
      Warning: Shutting down a VPlex cluster may cause data unavailability. Please refer to the VPlex documentation for the recommended procedure for shutting down a cluster. To show that you understand the impact, enter 'shutdown': shutdown     
     
      You have chosen to shutdown 'cluster-1'. To confirm, enter 'cluster-1': cluster-1     
     
      Status Description     
      ------------ -----------------------------     
      In progress. Shutdown already in progress.
   
   
    3) Please wait for 3-5 minutes (recommended) for the shutdown to complete. Please run below command to check the cluster's status and confirm that "operational-status" for affected cluster (ex. cluster-1) is "not-running" state:   
   
    VPlexcli:/> cluster status     
      Cluster cluster-1       
        operational-status:     not-running
     
      transitioning-indications:     
      transitioning-progress:     
      health-state:           unknown     
      health-indications:     
      local-com:              failed to validate local-com: Firmware command     
      error.     
      communication error recently.     
     
      Cluster cluster-2     
      operational-status:      degraded     
      transitioning-indications: suspended exports,suspended volumes     
      transitioning-progress:     
      health-state:            minor-failure     
      health-indications:      37 suspended Devices     
                               6 unhealthy Devices or storage-volumes     
      storage-volume           unreachable     
      local-com:               ok
   
   
    4) Run below command to display the cluster summary to check affected cluster (ex. cluster-1) "connected" is showing "false" and island is showing the running the cluster (ex.cluster-2) as follows:   
   
    VPlexcli:/> cluster summary     
      Clusters:     
        Name       Cluster ID  TLA             Connected  Expelled  Operational Status  Health State     
        ---------  ----------  --------------  ---------  --------  ------------------  ------------     
        cluster-1  1           FNM0019xxxxxxx  false       -          -                   -       
        cluster-2  2           FNM0018xxxxxxx  true       false       ok                  ok     
     
     
      Islands:     
        Island ID  Clusters     
        ---------  --------------------     
        1          cluster-2
   
   
    5) If the issue still persists, please follow below KB article and check if it resolves the issue:   
   
    333670 : VPLEX: How to shut down a cluster cleanly when the cluster is not in contact with other clusters in a Metro-PLEX   
   
    473796 : VPLEX: When following the procedure to shut down both clusters in a VPlex Metro system, one cluster does not shut down correctly