VPLEX: 3-way VPN configuration fails due to incorrect ip-address

           

   Article Number:     537741                                   Article Version: 3     Article Type:    Break Fix 
   

 


Product:

 

VPLEX for All Flash,VPLEX Geo,VPLEX GeoSynchrony,VPLEX Local,VPLEX Metro,VPLEX Series,VPLEX Sizing Tool,VPLEX VS1,VPLEX VS2,VPLEX Virtual Edition,VPLEX VS6,VPLEX GeoSynchrony 6.1,VPLEX GeoSynchrony 6.0,VPLEX GeoSynchrony 5.5

 

Issue:

 

 

   

      Issue Description:     
     
      Configuring 3-way VPN connection between VPlex management server (either cluster-1 or/both cluster-2) and cluster-witness server fails with below error message:     
     
      VPlexcli:/> configuration cw-vpn-configure -i xx.xx.xx.xx --force       
        Please enter the IP address of the remote cluster management server that will be included in the 3-way VPN setup: yy.yy.yy.yy       
        Verifying the VPN status between the management servers...       
        IPSEC is UP       
        Remote Management Server at IP Address yy.yy.yy.yy is reachable       
        Remote Internal Gateway addresses are reachable       
        Verifying the VPN status between the management server and the cluster witness server...       
        IPSEC is not UP       
        Cluster Witness Server at IP Address 128.221.254.3 is not reachable       
       
        Error during Cluster Witness VPN Establishment: IPSEC configuration failed: Cannot proceed to configure the Cluster Witness Server IPSec configuration.       
       
        It is possible that a 3-way VPN has already been established and you have given a wrong Cluster Witness Server public IP address.       
        Please re-run the cluster witness server vpn configuration with the right public IP address.       
        Resetting the Cluster Witness VPN configuration       
        Resetting the Cluster Witness Server VPN configuration       
       
        .       
        .       
        .
     
      <./truncated>   
                                                             

 

 

Cause:

 

 

There could be 2(two) scenario's during which configuration of 3-way VPN connection between VPlex management server (either cluster-1 or/both cluster-2) and cluster-witness server can fail as follows:   
   
    1) User has changed VPlex management server IP address (eth3) (either cluster-1 or/both cluster-2) but cluster-witness server IPsec configuration file still contains an old IP address of the affected management server.   
   
    AND/OR,   
   
    2) User has changed cluster-witness IP address but VPlex management server(either cluster-1 or/both cluster-2) IPsec configuration file still contains old IP address of cluster-witness server.
                                                           

 

 

Change:

 

 

User has changed/updated VPlex management server IP address(either cluster-1 or/both cluster-2) or cluster-witness IP address                                                           

 

 

Resolution:

 

 

Please go through below scenario details and resolution step's in-order to resolve this issue:   
   
    Scenario 1:  User has changed VPlex management server IP address (eth3) (either cluster-1 or/both cluster-2) but cluster-witness server IPsec configuration file still contains an old IP address of the affected management server   
   
    NOTE: In below example's, user has changed VPlex management server IP address of both cluster-1 and cluster-2.   
   
    1) Please gather correct IP-address's assigned to VPlex management server (both cluster-1 and cluster-2) as follows:   
   
    Cluster-1:   
    VPlexcli:/> vpn status   

      Verifying the VPN status between the management servers...IPSEC is UPRemote Management Server at IP Address 14N.NNN.N.NNN is reachableRemote Internal Gateway addresses are reachable.</truncated>    
   
    Cluster-2:   
      VPlexcli:/> vpn statusVerifying the VPN status between the management servers...IPSEC is UPRemote Management Server at IP Address 14M.MMM.M.MMM is reachableRemote Internal Gateway addresses are reachable.</truncated>    
   
   
    2) Please perform SSH to the cluster-witness server using its public IP address :   
   
    2.a) To find the public IP-address of cluster-witness server, run the following VPlexcli command:   
   
    Example:   
    VPlexcli:/> ll /cluster-witness/     
      /cluster-witness:     
      Attributes:     
      Name                Value     
      ------------------  -------------     
      admin-state         unknown     
      private-ip-address  128.221.254.3     
      public-ip-address   XX.XX.XX.XX      <<< Cluster-Witness server public IP-address
   
   
   
    2.b) Please perform SSH to the public IP-address of cluster-witness obtained from step 1.a as follows:   
   
    service@ManagementServer:~> ssh <cluster-witness-public-IP-address>   
   
    Example:   
    service@ManagementServer:~> ssh xx.xx.xx.xx       >> cluster-witness-public-IP-address     
      Warning: Permanently added 'xx.xx.xx.xx' (ECDSA) to the list of known hosts.     
      Last login: Mon Jun 06 15:33:14 2016 from xx.xx.xx.xx     
      service@ClusterWitness:~>
   
   
    3) Cat the “IPsec.config” file and search for VPlex management server cluster-1 and cluster-2 IP address’s as follows:   
    Note: Before carrying out to step(3) confirm the actual IP address of VPlex management server's from vpn status output using step(1). Once this information is gathered, please compare it with the "IPsec.config" file mentioned below to check/confirm if it is matching or not.   
   
    Example:   
    service@ClusterWitness:~> cat /etc/ipsec.conf     
      # Add connections here.     
      # Setup a tunnel between the management servers and the Cluster Witness Server     
      # "left" means local, "right" means remote.
   
    # Connection between Cluster Witness Server and Management Server     
      conn witness-cluster2     
      type=tunnel     
      keyexchange=ikev2     
      mobike=no     
      reauth=no     
      left=%defaultroute     
      leftsubnet=128.221.254.3/32     
      leftcert=hostCert.pem     
      right=15Y.YYY.Y.YYY             <<========== Old/incorrect IP address of VPlex management server-2     
      rightsubnet=128.221.252.64/27,128.221.253.64/27     
      rightid="C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN: CKxxxxxxxxxxxx, E=support@emc.com"     
      ike=3des-sha256-modp2048     
      esp=aes256-sha256     
      auto=start     
     
      # Connection between Cluster Witness Server and Management Server     
      conn witness-cluster1     
      type=tunnel     
      keyexchange=ikev2     
      mobike=no     
      reauth=no     
      left=%defaultroute     
      leftsubnet=128.221.254.3/32     
      leftcert=hostCert.pem     
      right=15X.XXX.X.XXX         <<========== Old/incorrect IP address of VPlex management server-1     
      rightsubnet=128.221.252.32/27,128.221.253.32/27     
      rightid="C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN: CKyyyyyyyyyyyy, E=support@emc.com"     
      ike=3des-sha256-modp2048     
      esp=aes256-sha256     
      auto=start
   
   
    4) As from the above example, we found that “IPsec.config” file of cluster-witness is still carrying old IP-address of VPlex management server-1 and cluster-2. Therefore, please edit the file “IPsec.config” of cluster-witness server using vi editor to update the correct IP-address of VPlex management server-1 and cluster-2.   
    NOTE: Place the correct IP address of VPlex cluster-1 and cluster-2 after the equals sign with no space between the equal sign and IP address, save and exit the file.   
   
    Example:   
    service@ClusterWitness:~> vi /etc/ipsec.conf     
      # Add connections here.     
      # Setup a tunnel between the management servers and the Cluster Witness Server     
      # "left" means local, "right" means remote.
   
    # Connection between Cluster Witness Server and Management Server       
        conn witness-cluster1
     
      type=tunnel     
      keyexchange=ikev2     
      mobike=no     
      reauth=no     
      left=%defaultroute     
      leftsubnet=128.221.254.3/32     
      leftcert=hostCert.pem     
      right=14M.MMM.M.MMM       <<========== Add/update the correct IP address of VPlex cluster-1.     
      rightsubnet=128.221.252.32/27,128.221.253.32/27     
      rightid="C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN: CKyyyyyyyyyyyy, E=support@emc.com"     
      ike=3des-sha256-modp2048     
      esp=aes256-sha256     
      auto=start     
     
      # Connection between Cluster Witness Server and Management Server       
        conn witness-cluster2
     
      type=tunnel     
      keyexchange=ikev2     
      mobike=no     
      reauth=no     
      left=%defaultroute     
      leftsubnet=128.221.254.3/32     
      leftcert=hostCert.pem     
      right=14N.NNN.N.NNN       <<========== Add/update the correct IP address of VPlex cluster-2.     
      rightsubnet=128.221.252.64/27,128.221.253.64/27     
      rightid="C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN: CKxxxxxxxxxxxx, E=support@emc.com"     
      ike=3des-sha256-modp2048     
      esp=aes256-sha256     
      auto=start
   
   
    5) Restart IPSEC service in both cluster-witness server and VPlex management server’s(both cluster-1 and cluster-2) as follows:   
   
   
    service@ClusterWitness:~> sudo /usr/sbin/ipsec restart     
      service@ManagementServer:~> sudo /usr/sbin/ipsec restart
   
   
    5.a) Verify the status of IPsec service as follows:   
   
    service@ClusterWitness:~> sudo /usr/sbin/ipsec status     
      service@ManagementServer:~> sudo /usr/sbin/ipsec status
   
   
    6) Re-perform the below command to re-configure 3-way VPN connection between VPlex management server's and cluster-witness server as follows:   
   
    Example:   
    VPlexcli in Cluster-1:   
   
    VPlexcli:/> configuration cw-vpn-configure -i <cluster-witness-public-IP> –force   

      Then from VPlexcli in Cluster-2   

VPlexcli:/> configuration cw-vpn-configure -i <cluster-witness-public-IP> –force   
   
   
    7) Verify the status of cluster-witness following step-1(both cluster-1 and cluster-2) as follows:   
   
    Example:   
      VPlexcli:/>  ll /cluster-witness/* /cluster-witness/components:Name               ID  Admin State  Operational State    Mgmt Connectivity-----------------  --  -----------  -------------------  -----------------cluster-1          1   enabled      in-contact           okcluster-2          2   enabled      in-contact           okserver             -   enabled      clusters-in-contact  ok    
   
     
   
    8) Check VPN connectivity using command vpn status(both cluster-1 and cluster-2) as follows:   
   
    Example:   
    Cluster-1:   
    VPlexcli:/> vpn status   
      Verifying the VPN status between the management servers...IPSEC is UPRemote Management Server at IP Address 14N.NNN.N.NNN is reachableRemote Internal Gateway addresses are reachableVerifying the VPN status between the management server and the cluster witness server...IPSEC is UP Cluster Witness Server at IP Address 128.221.254.3 is reachable     
   
    Cluster-2:   
      VPlexcli:/> vpn statusVerifying the VPN status between the management servers...IPSEC is UPRemote Management Server at IP Address 14M.MMM.M.MMM is reachableRemote Internal Gateway addresses are reachableVerifying the VPN status between the management server and the cluster witness server...IPSEC is UP Cluster Witness Server at IP Address 128.221.254.3 is reachable     
   
   
   
    Scenario 2:  User has changed cluster-witness IP address but VPlex management server(either cluster-1 or/both cluster-2) IPsec configuration file still contains old IP address of cluster-witness server.   
   
   
    9)  Verify the correct cluster-witness public IP-address by following the below command:   
                
VPlexcli:/> ll /cluster-witness/**/cluster-witness:Attributes:Name                Value------------------  -------------admin-state         enabledprivate-ip-address  128.221.254.3public-ip-address   xx.xx.xx.65   <<< Cluster-Witness server public IP-addressContexts:Name        Description----------  --------------------------components  Cluster Witness Components    
   
   
   
    10) From VPlex management server(either cluster-1 or cluster-2) cat the file “ipsec.conf” as follows:   
    Note: Before carrying out to step(10) confirm the actual IP address of VPlex management server's from vpn status output using step(9). Once this information is gathered, please compare it with the "IPsec.config" file mentioned below to check/confirm if it is matching or not.   
   
    Example:   
    service@Managementserver:~> cat /etc/ipsec.conf     >> Cluster-1     
      # Add connections here.     
      # Setup a tunnel between the management servers and their networks     
      # "left" means local, "right" means remote.     
      # Connection between Cluster Witness Server and Management Server       
        conn net-witness
     
      type=tunnel     
         keyexchange=ikev2     
         mobike=no     
         reauth=no     
         left=%defaultroute     
         leftsubnet=128.221.252.64/27,128.221.253.64/27     
         leftcert=hostCert.pem     
         right=xx.xx.xx.45    <<========== Old/incorrect IP address of cluster-witness     
         rightsubnet=128.221.254.3/32     
         rightid="C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN CWS, E=support@emc.com"     
         ike=3des-sha256-modp2048     
         esp=aes128-sha1     
         auto=start     
     
      # Connection between Management Server 1 and Management Server 2     
      conn net-net     
         type=tunnel     
         keyexchange=ikev2     
         mobike=no     
         reauth=no     
         left=%defaultroute     
         leftsubnet=128.221.252.64/27,128.221.253.64/27     
         leftcert=hostCert.pem     
         right=14N.NNN.N.NNN   <<========== IP address of remote management server      
         rightsubnet=128.221.252.32/27,128.221.253.32/27     
         rightid="C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN:CKxxxxxxxxxxxx, E=support@emc.com"     
         ike=3des-sha256-modp2048     
         esp=aes256-sha256     
         auto=start
   
   
     11) As from the above example, we found that “IPsec.config” file of VPlex management server (either cluster-1 or/both cluster-2) is still carrying old IP-address of cluster-witness server. Therefore, please edit the file “IPsec.config” using vi editor to update the correct IP-address of cluster-witness server.   
    NOTE: Place the correct IP address of Cluster-witness server on affected VPlex management server after the equals sign with no space between the equal sign and IP address, save and exit the file.   
   
    Repeat step's from 4 to 8 from scenario 1 in-order to resolve this issue.