ViPR Controller : Create Volume and Datastore fails with "Error 4000: Token not found or invalid token provided"

           

   Article Number:     533755                                   Article Version: 2     Article Type:    Break Fix 
   

 


Product:

 

ViPR Controller,ViPR Controller Controller 3.6 SP2,VMware ESXi,VMware vCenter Server

 

Issue:

 

 

A ViPR Controller order to create a volume and Datastore errors after several hours with the error   
   
    "Error 4000: Token not found or invalid token provided"   
   
   
    In the ViPR Controller UI, we can see that the order fails at the "Create VMFS Datastore" stage   
   
    Create VMFS Datastore     Create VMFS datastore 'DatstoreName' on ESX host 'hostname' using LUN DGC Fibre Channel Over Ethernet Disk (naa.600xxxxxxxxxxxxxxxxxxxxxxxxxx)     
     
      com.iwave.ext.vmware.VMWareException     
         at com.iwave.ext.vmware.HostStorageAPI.createVmfsDatastore(HostStorageAPI.java:516)     
         at com.emc.sa.service.vmware.block.tasks.CreateVmfsDatastore.tryExecute(CreateVmfsDatastore.java:33)     
         at com.emc.sa.service.vmware.block.tasks.CreateVmfsDatastore.tryExecute(CreateVmfsDatastore.java:17)
   
   
   
    Although the ViPR Controller order errors, the block volume is created and exported to the VMWare cluster and is not rolled back after order failure.   
     
                                                           

 

 

Cause:

 

 

ViPR Controller creates the block volume, exports it to each node in the VMware cluster but the "Create VMFS datastore" VMware task takes a long time to complete the VMFS re-scan on some hosts and eventually errors with :       
       
       
        VMware vSphere Client UI error  (Cluster - Monitor - Tasks & Events)       
       
        "An error occurred while communicating with the remote host"
   
   
   
    During the "Create VMFS Datastore" phase, an attempt is made to discover all devices presented to an ESXi host during the device claiming phase.   
   
    However, Microsoft Cluster Server (MSCS) LUNs that have a permanent SCSI reservation cause this process to lengthen as the ESXi host cannot interrogate the LUN due to the persistent SCSI reservation placed on a device by an active MSCS Node hosted on another ESXi host.   
   
    These delays eventually cause the VIPR Controller order to fail / time-out.   
     
                                                           

 

 

Resolution:

 

 

   

     
      1. Please follow VMware KB 1016106 for resolution.     
     
      2. In ViPR Controller, create a new volume and datastore again to test.     
     
          

                                                             

 

 

Notes:

 

 

Any volumes created as a consequence of this issue will not be rolled back after order failure.   
    The volumes are created as block volumes (not datastores) and are still exported to the ESX cluster/host.   
   
    If these volumes are to be maintained as datastores, then they should be tagged as VMFS datastores.   
    ViPR Controller: How to add Virtual Machine File System (VMFS) tags to VMware datastore volumes   
   
    Alternatively, delete the block volumes from ViPR Controller and recreate them as VMFS datastores after following the resolution above.