PowerPath: Running the command "powermig sync -handle X" or "powermig cleanup -handle X" returns an error "PPME error(79): Not all nodes have reached the correct state to complete this operation, please try this or the recover command again later"

           

   Article Number:     500749                                   Article Version: 4     Article Type:    Break Fix 
   

 


Product:

 

PowerPath for Windows 5.7,PowerPath for Windows 5.7 SP1,PowerPath for Windows 5.7 SP2,PowerPath for Windows 5.7 SP3,PowerPath for Windows 5.7 SP4,PowerPath for Windows 6.0,PowerPath for Windows 6.0 SP1,PowerPath for Windows 6.0 SP2

 

Issue:

 

 

Running the command "powermig sync -handle X" returns an error "PPME error(79): Not all nodes have reached the correct state to complete this operation, please try this or the recover command again later"   
    Running the command "powermig cleanup -handle X" returns an error "PPME error(79): Not all nodes have reached the correct state to complete this operation, please try this or the recover command again later"   
   
    We are trying to use PowerPath Migration Enabler(PPME) to migrate the physical disk resources that are part of the Cluster group "XYZ-ClusterResources" which is online on node NODE-3   
   
    C:\Windows\system32>powermigcl display   
    Group name | Current owner | Status |   
    ================================|=================|===================|   
    Available Storage              | NODE-2 | Unconfigured |   
    XYZ-ClusterResources      | NODE-3 | Configured |   
    LMN-CLuster Resources   | NODE-2 | Unconfigured |   
    ABC_08CLDTC                 | NODE-2 | Unconfigured |   
    Cluster Group                    | NODE-2 | Unconfigured |   
   
    We are able to run the setup command:-   
    C:\Windows\system32>powermig setup -techType hostcopy -src harddisk6 -tgt harddisk16 -cluster   
   
    Setup migration? [yes]/no: y   
    Migration Handle = 9   
   
    C:\Windows\system32>powermig setup -techType hostcopy -src harddisk7 -tgt harddisk21 -cluster   
   
    Setup migration? [yes]/no: y   
    Migration Handle = 10   
   
    C:\Windows\system32>powermig info -all   
    ================================================   
    Hnd Source Target Tech State   
    === ========== ========== ============ =====   
    9 harddisk6 harddisk16 HostCopy(cl) setup   
    10 harddisk7 harddisk21 HostCopy(cl) setup   
    11 harddisk8 harddisk20 HostCopy(cl) setup   
    12 harddisk9 harddisk18 HostCopy(cl) setup   
    13 harddisk10 harddisk15 HostCopy(cl) setup   
    14 harddisk11 harddisk14 HostCopy(cl) setup   
    15 harddisk12 harddisk17 HostCopy(cl) setup   
    16 harddisk13 harddisk19 HostCopy(cl) setup   
   
    As Soon as we run the sync command, it gives this error:-   
    C:\Windows\system32>powermig sync -handle 9   
   
    Start sync for handle 9 ? [yes]/no: y   
   
    PPME error(79): Not all nodes have reached the correct state to complete this operation, please try this or the recover command again later   
   
    Trying to cleanup the migration also gives the same error:-   
   
    C:\Windows\system32>powermig cleanup -all   
   
    Cleanup migration for handle 9 ? [yes]/no: y   
   
    PPME error(79): Not all nodes have reached the correct state to complete this operation, please try this or the recover command again later   
   
    At that stage cleanup -force also gives the same error:-   
    C:\Windows\system32>powermig cleanup -handle 9 -force   
   
    Cleanup migration for handle 9 ? [yes]/no: y   
   
    PPME error(79): Not all nodes have reached the correct state to complete this operation, please try this or the recover command again later   
   
    C:\Windows\system32>powermig info -all   
    =================================================================   
    Hnd Source Target Tech State   
    === ========== ========== ============ ======================   
    9 harddisk6 harddisk16 HostCopy(cl) needsRecovery(cleanup)   
    10 harddisk7 harddisk21 HostCopy(cl) setup   
    11 harddisk8 harddisk20 HostCopy(cl) setup   
    12 harddisk9 harddisk18 HostCopy(cl) setup   
    13 harddisk10 harddisk15 HostCopy(cl) setup   
    14 harddisk11 harddisk14 HostCopy(cl) setup   
    15 harddisk12 harddisk17 HostCopy(cl) setup   
    16 harddisk13 harddisk19 HostCopy(cl) setup   
   
    C:\Windows\system32>powermig cleanup -handle 9 -force   
   
    Cleanup migration for handle 9 ? [yes]/no: y   
   
    PPME error(74): Cluster open failed, cluster migration must be intiate from a cluster node   
   
    In Cluster.log, there is below PPME error instead of "connection error"   
   
    ERR [RES] ppme <PPME 2404-RESOURCE-036-TEMPDB>: Cannot set funnel state in non-mig node: error:fffffffd:81c0   
    0000111c.00001b20::2014/03/14-19:50:24.850 ERR [RES] ppme <PPME 2404-RESOURCE-036-TEMPDB>: Cannot set funnel state in non-mig node: error:fffffffd:81c0   
    0000111c.00001b20::2014/03/14-19:50:27.869 ERR [RES] ppme <PPME 2404-RESOURCE-036-TEMPDB>: Cannot set funnel state in non-mig node: error:fffffffd:81c0   
   
    Or you see the following logs in the Cluster logs:-In cluster logs on node on PH1CRMSGSQL0A we see the following Logs:-   
   
    00001ba8.000048b0::2014/03/24-13:30:50.879 WARN [RCM] Possible owners: <vector len='3'>   
    00001ba8.000048b0::2014/03/24-13:30:50.879 WARN <item>1</item>   
    00001ba8.000048b0::2014/03/24-13:30:50.879 WARN <item>3</item>   
    00001ba8.000048b0::2014/03/24-13:30:50.879 WARN <item>2</item>   
    00001ba8.000048b0::2014/03/24-13:30:50.879 WARN </vector>   
    00001ba8.000048b0::2014/03/24-13:30:50.957 WARN [RCM] DllIsPresent( 4 ) returned false for restype ppme.   
    00001ba8.000048b0::2014/03/24-13:30:50.957 WARN [RCM] PPME Disk L cannot be hosted on node 4   
    00001ba8.000048b0::2014/03/24-13:30:50.957 WARN [RCM] Possible owners: <vector len='0'>   
    00001ba8.000048b0::2014/03/24-13:30:50.957 WARN </vector>   
    00001ba8.000048b0::2014/03/24-13:30:50.957 WARN [RCM] Disk O cannot be hosted on node 4   
    00001ba8.000048b0::2014/03/24-13:30:50.957 WARN [RCM] Possible owners: <vector len='3'>   
    00001ba8.000048b0::2014/03/24-13:30:50.957 WARN <item>2</item>   
    00001ba8.000048b0::2014/03/24-13:30:50.957 WARN <item>3</item>   
    00001ba8.000048b0::2014/03/24-13:30:50.957 WARN <item>1</item>   
    00001ba8.000048b0::2014/03/24-13:30:50.957 WARN </vector>   
    00001ba8.000048b0::2014/03/24-13:30:50.957 WARN [RCM] SQL Server (INT) cannot be hosted on node 4   
    00001ba8.000048b0::2014/03/24-13:30:50.957 WARN [RCM] Possible owners: <vector len='3'>   
    00001ba8.000048b0::2014/03/24-13:30:50.957 WARN <item>2</item>   
    00001ba8.000048b0::2014/03/24-13:30:50.957 WARN <item>1</item>   
    00001ba8.000048b0::2014/03/24-13:30:50.957 WARN <item>3</item>   
    00001ba8.000048b0::2014/03/24-13:30:50.957 WARN </vector>   
    00001ba8.000048b0::2014/03/24-13:30:51.035 WARN [RCM] DllIsPresent( 4 ) returned false for restype ppme.   
    00001ba8.000048b0::2014/03/24-13:30:51.035 WARN [RCM] PPME Disk M cannot be hosted on node 4   
    00001ba8.000048b0::2014/03/24-13:30:51.035 WARN [RCM] Possible owners: <vector len='0'>   
    00001ba8.000048b0::2014/03/24-13:30:51.035 WARN </vector>   
    00001ba8.000034a8::2014/03/24-13:30:51.129 WARN [RCM] PPME Disk O cannot be hosted on node 4   
    00001ba8.000034a8::2014/03/24-13:30:51.129 WARN [RCM] Possible owners: <vector len='0'>   
    00001ba8.000034a8::2014/03/24-13:30:51.129 WARN </vector>
                                                           

 

 

Cause:

 

 

Target devices are not presented to all the nodes of the cluster   
    All the cluster nodes are not up and running.   
    Cluster service is not running on all the cluster nodes.   
    Failover nodes have funneling and access control set on for the devices.
                                                           

 

 

Resolution:

 

 

Please make sure :   
   
    1. PowerPath version is same on all the nodes of the Cluster.   
   
    2. Find the total number of nodes in the cluster from SERVERNAME_CLUSTER_MPS_INFORMATION.TXT in the Cluster Folder in EMC reports   
   
    3, The cluster service is running on all cluster nodes.   
   
    4. The target device is connected and is available to all the nodes of the cluster.   
   
    5. Once the above is taken care of, Run the powermig recover command on node where PPME commands were run.Then try sync.   
   
    6. If it still fails with the same error message, check the registry key   
    "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\EmcPowerPath" on all the nodes of the cluster and perform the following action after taking the backup of the registry.:-   
   
    All dm-related KMD entries should be removed on all servers in the cluster.   
    1. Delete the UMD by deleting the files from C:\Program Files\EMC\PPME\db\*.*   
   
    2. Delete all the subkeys with Prefix "dm_" EXCEPT for dev_conf under   
    "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\EmcPowerPath\KMD_0" (or   
    "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\EmcPowerPath\KMD_1"). The Keys would be dm_ac, dm_control_io_to_clones, dm_funnel_io, dm_wc.   
   
    3. Delete the subkey umd_dev_conf (NOT dev_conf) under   
    "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\EmcPowerPath\KMD_0" (or   
    "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\EmcPowerPath\KMD_1").   
   
    4. Reboot Each node of the cluster after performing the above steps on that node.   
   
    5. For better results, reinstall powerpath on each node and reboot.