XtremIO: Can't set LUN Queue Depth following an upgrade to VMware vSphere 6.7U1

           

   Article Number:     537989                                   Article Version: 2     Article Type:    Break Fix 
   

 


Product:

 

XtremIO Family,XtremIO X1,XtremIO X2

 

Issue:

 

 

Following an upgrade to vSphere 6.7U1 and above running on top of Cisco UCS servers, Queue Depth can't be tuned.   
   
    Note: This Issue only impacts Cisco UCS servers running with nfnic version 4.0.0.28 (or earlier versions).
                                                           

 

 

Cause:

 

 

VMWare introduced in vSphere 6.7U1 (and later versions) a Native driver integrated into vSphere to replace the obsolete fnic drivers.   
        
    When upgrading to vSphere 6.7U1 (or later versions), the legacy driver (fnic) is converted to a Native driver (nfnic). The nfnic version that comes embedded with vSphere 6.7U1 (4.0.0.14) has an issue with configuring Queue Depth, a minimum driver of 4.0.0.28 is required to resolve the issue.   
   
    According to CSCvo09082 the default Queue Depth will be 32 if not configured otherwise.
                                                           

 

 

Resolution:

 

 

To Resolve the issue either upgrade to vSphere 6.7U3 (contains driver 4.0.0.29), or upgrade the nfnic driver to a minimum of 4.0.0.28.   
   
    The following instructions can be used as guidelines to upgrade the nfnic driver:   

         
  1.         Identify the current driver layout (native or legacy)     
  2.    
   
[root@ucs4-esx65:~] esxcfg-scsidevs -avmhba0  nfnic             link-up   fc.20000025b5cc000f:20000025b5a0000f    (0000:08:00.0) Cisco Systems Inc Cisco VIC FCoE HBA Drivervmhba1  nfnic             link-up   fc.20000025b5cc000f:20000025b5b0000f    (0000:0f:00.0) Cisco Systems Inc Cisco VIC FCoE HBA Driver    
   
      Note: from the above we can see the current active driver is nfnic driver.   
   
         
  1.         Identify the installed driver versions     
  2.    
   
[root@ucs4-esx65:~] esxcli software vib list | grep nfnicnfnic                          4.0.0.14-0vmw.670.1.28.10302608       VMW     VMwareCertified   2019-02-12    
   
      Note: the current installed version is 4.0.0.14 which is impacted by the issue described in this KB.   
   
         
  1.         Run the following command to identify the vHBA details     
  2.    
   
[root@ucs4-esx65:~] vmkchdev -l |grep vmhba10000:0f:00.0 1137:0045 1137:012c vmkernel vmhba1    
   
      Note: from the above we can identify the vHBA information (VID=1137, DID=0045, SVID=1137, SDID=012c).   
   
         
  1.         Open VMWare Compatibility Guide (Link), and enter the above details (VID, DID, SVID, SDID), this will identify the vHBA model     
  2.    
   
      User-added image   
   
         
  1.         According to the above we have Cisco UCS VIC-1340 card      
  2.      
  3.         Choose the appropriate OS (in our case vSphere 6.7U1)     
  4.      
  5.         Choose a minimum driver version of 4.0.0-28, and click the link to download the driver     
  6.      
  7.         Review the release notes for the specified driver (review known issues, bugs fixed, enhancements introduced)     
  8.    
   
         
  1.         Once you have downloaded the offline bundle (for example: VMW-ESX-6.7.0-nfnic-4.0.0.40-14303978.zip), extract the VIB file (example: nfnic-4.0.0.40-1OEM.670.0.0.8169922.x86_64.vib) and upload to any datastore on relevant ESX host.     
  2.      
  3.         Follow KB#2137854 to install the driver.     
  4.      
  5.         The below is an example for installing nfnic version 4.0.0-40 on vSphere 6.7U1:        
               
    1.             Place the host into maintenance mode         
    2.        
                  
  6.    
   
[root@ucs4-esx65:~] esxcli system maintenanceMode set --enable=true    
   
         
  1.         Run the following command to install the driver from the VIB file:     
  2.    
   
[root@ucs4-esx65:~] esxcli software vib install -v /vmfs/volumes/4e166578-e1e7f8b7-c6e2-27d0173f56f7/nfnic-4.0.0.40-1OEM.670.0.0.8169922.x86_64.vibInstallation Result   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.   Reboot Required: true   VIBs Installed: Cisco_bootbank_nfnic_4.0.0.40-1OEM.670.0.0.8169922   VIBs Removed: VMW_bootbank_nfnic_4.0.0.29-0vmw.670.3.73.14320388   VIBs Skipped:    
   
         
  1.         Reboot the host     
  2.    
   
[root@ucs4-esx65:~] esxcli system shutdown reboot -r "Upgrade NFNIC driver to 4.0.0.40"    
   
         
  1.         Once the host is up, confirm the driver version     
  2.    
   
[root@ucs4-esx65:~] esxcli software vib list | grep nfnicnfnic                          4.0.0.40-1OEM.670.0.0.8169922         Cisco   VMwareCertified   2019-10-17    
   
         
  1.         Confirm the relevant driver is activated against the relevant vHBA:     
  2.    
   
[root@ucs4-esx65:~] esxcfg-scsidevs -avmhba0  nfnic              link-up   fc.20000025b5cc000f:20000025b5a0000f    (0000:08:00.0) Cisco Systems Inc Cisco VIC FCoE HBA Drivervmhba1  nfnic              link-up   fc.20000025b5cc000f:20000025b5b0000f    (0000:0f:00.0) Cisco Systems Inc Cisco VIC FCoE HBA Driver    
   
         
  1.         Exit maintenance mode     
  2.    
   
[root@ucs4-esx65:~] esxcli system maintenanceMode set --enable=false    
   
         
  1.         Configure the LUN Queue Depth for the nfnic vHBAs     
  2.    
   
[root@ucs4-esx65:~] esxcli system module parameters set -m nfnic -p lun_queue_depth_per_path=128    
   
         
  1.         Confirm the Queue Depth has been updated     
  2.    
   
[root@ucs4-esx65:~] esxcli system module parameters list -m nfnicName                      Type   Value  Description------------------------  -----  -----  --------------------------------------------------------------lun_queue_depth_per_path  ulong  128    nfnic lun queue depth per path: Default = 32. Range [1 - 1024]    
   
                                                                

 

 

Notes:

 

 

Please Follow XtremIO Host Configuration Guide for up to date instructions