VPLEX: Unexpected IP WAN-COM behavior due to LLDP streaming when a VPLEX VS2 and VS6 exist on the same VLAN

           

   Article Number:     530700                                   Article Version: 3     Article Type:    Break Fix 
   

 


Product:

 

VPLEX Series,VPLEX VS2,VPLEX VS6,VPLEX Local,VPLEX Metro,VPLEX GeoSynchrony 5.5,VPLEX GeoSynchrony 5.5 Patch 1,VPLEX GeoSynchrony 5.5 Service Pack 1,VPLEX GeoSynchrony 5.5 Service Pack 1 Patch 1,VPLEX GeoSynchrony 5.5 Service Pack 1 Patch 2

 

Issue:

 

 

Impacted VPLEX Hardware:    
    EMC Hardware: VPLEX Series   
    EMC Hardware: VPLEX VS2   
    EMC Hardware: VPLEX VS6   
    EMC Hardware: VPLEX-Local   
    EMC Hardware: VPLEX-Metro   
   
    Impacted VPLEX GeoSynchrony code versions:   
    EMC Software: GeoSynchrony 5.X   
    EMC Software: GeoSynchrony 6.X   
        

         
  •         High frequency of LLDP (Link Layer Discover Protocol) multicasts are seen on the XG00 orts of all directors on one cluster. The XG01 ports on the the same directors of this one cluster, and all directors XG ports at peer cluster, do not have the LLDP multicast storm problem.      
  •    
   
         
  •         Network captures (TCP dumps) from the VPLEX IP WAN-COM ports show very high frequencies of LLDP frames (almost at the order of 1000 frames every minute).     
  •    
   
         
  •         End Users IP network is affected due to an LLDP multicast storm.     
  •    
   
         
  •         Possible reboot of the VPLEX directors due to the way the NIC (Network Interface Card) on the VPLEX handles the LLDP multicast storm.     
  •    
                                                             

 

 

Cause:

 

 

   

         
  •         Link Layer Discovery Protocol (LLDP) is a layer-2 standard protocol which defines messages encapsulated in Ethernet frames to allow devices to announce basic device information to other (neighboring) devices in the LAN (Local Area Network). LLDP is a vendor-independent protocol (not proprietary to any vendor).     
  •    
   
         
  •         This is achieved through periodic transmissions by the Network Interface Card (NIC) every 30 seconds (by default).     
  •    
   
         
  •         The VPLEX VS6 cluster utilizes the Broadcom Emulex OneConnect OCe14000 Series NIC for it's IP WAN-COM module.     
  •    
   
         
  •         The VPLEX VS2 cluster utilizes the QLogic Heatwave EP8112 NIC for it's IP WAN-COM module.     
  •    
   
         
  •         Both the Broadcom and Qlogic NICs support LLDP as the chipsets support iSCSI, even though VPLEX does not support iSCSI.     
  •    
   
         
  •         TCP dumps captured during the incident reveal that the LLDP frames are being streamed at a very high frequency and not per the default value of 30 seconds. This is because whenever a NIC receives an LLDP frame with different settings, it is applying and processing the settings and re-transmitting the frame with the new settings. The LLDP multicast frames from all the NICs are forwarded to all other NICs within the same VLAN causing the LLDP multicast storm.     
  •    
                                                             

 

 

Change:

 

 

The VPLEX VS6 and VS2 are left configured on the same VLAN (post-GenU, or for whatever other reason)                                                            

 

 

Resolution:

 

 

Resolution:   
    Disable LLDP forwarding on the switch-ports to which the VPLEX WAN-COM ports are connected to.
                                                           

 

 

Notes:

 

 

If the TCPdump captures from the VS6 WAN-COM ports are reviewed, you may observe that the MAC (Media Access Control) address of the origin of the LLDP frames is 1 more than the MAC address of the IP WAN-COM port. This is because the VS6 WAN-COM ports utilize the Broadcom OneConnect OCe14000 Series adapters as their NIC which is capable of having 128 MAC/VLAN addresses per port. This should avoid any confusion as to the origins of the LLDP frames. For example, consider the following MAC address of a VPLEX IP WAN-COM port;   
   
    VPlexcli:/engines/engine-1-1/directors/director-1-1-A/hardware/ports/A2-XG00> ll     
     
      Name                Value     
      ------------------  -----------------     
      address             X.X.X.X     
      config-status       applied     
      enabled             true     
      mac-address         00:11:22:33:44:aa     
      mtu                 1500     
      operational-status  ok     
      option-set          optionset-com-0     
      port-status         up     
      protocols           [udt]     
      role                wan-com     
      speed               10000     
      subnet              cluster-1-SN00
   
   
   
    The source MAC address from the TCP dumps would report the address as 00:11:22:33:44:a(notice that the source address is the mac-address + 1).   
        
    Another example for consideration;   
   
    VPlexcli:/engines/engine-1-1/directors/director-1-1-A/hardware/ports/A2-XG01> ll     
      Name                Value     
      ------------------  -----------------     
      address             X.X.X.X     
      config-status       applied     
      enabled             true     
      mac-address         00:11:22:33:44:a1     
      mtu                 1500     
      operational-status  ok     
      option-set          optionset-com-1     
      port-status         up     
      protocols           [udt]     
      role                wan-com     
      speed               10000     
      subnet              cluster-1-SN01
   
   
   
    The TCP dumps reports would reveal the source mac-address as 00:11:22:33:44:a(mac-address + 1).   
        
    The VS2 WAN-COM ports run Heatwave EP8112 NIC firmware which does not have a feature to disable LLDP. The only option provided by Qlogic is to disable LLDP on the switch.