Dell EMC VxRail: How to determine if a storage drive should be replaced in a VxRail vSAN cluster and preparation steps.[1]

           

   Article Number:     529853                                   Article Version: 3     Article Type:    How To 
   

 


Product:

 

VxRail Appliance Family,VxRail Appliance Series,VxRail 460 and 470 Nodes,VxRail E Series Nodes,VxRail G Series Nodes,VxRail Gen 5,VxRail Gen2 Hardware,VxRail P Series Nodes,VxRail S Series Nodes,VxRail V Series Nodes,VxRail Software

 

Instructions:

 

 

About VxRail Remote Support's failed drive troubleshooting and hardware replacement process:   
         Timely resolutions are a main focus of VxRail Support's approach to situations where drives may need to be replaced. In general, all that is required to send replacement drives is to identify a faulted disk and ensure the correct part(s) are matched up for dispatch. In most cases, this can be done quickly and arrangements should be made to send any needed replacements very quickly.   
   
         It is also important to ensure that drive replacement is the appropriate solution and the correct hardware information has been identified to avoid issues with sending the correct hardware and unexpected problems during/after replacement. Sometimes, it may appear that a drive has failed because vCenter or host clients show a disk or diskgroup in an unexpected state. Even if a drive shows 'Permanent Device Loss' (PDL), that drive may not actually be faulted, and replacement might not resolve the issue.   
   
         The below steps are intended to be quick but to also maximize the chance of identifying the root issue, appropriate resolution, and any additional impact or problems that should be addressed. It is expected that all verification steps (barring complications) can be done and a determination can be made to send a disk replacement in less than 30 minutes, unless checks indicate an issue that requires a different approach.    
   
    Steps to determine if hardware replacement should resolve the issue:   
    1. Check in IDRAC (Dell) or BMC (Phoenix/Quanta) or their hardware logs for drives marked as failed. Other than SATADOM, M.2, or other devices unrelated to the vSAN storage, if drives are marked as failed here, this generally means that the hardware really is faulted and needs to be replaced.    
         ***Note that even if a drive shows failure in hardware sensors and on the vSAN level, sometimes it shows the same even after replacing the drive. In these cases, the expected next point of troubleshooting is the backplane on the host.    
        
    2. Check vCenter and the host to determine if a vSAN drive is marked PDL. On a host with a suspected drive failure you can use command line:   

          vdq -qH                                      (check for PDL of '1' indicating drive failure)     
          esxcli vsan storage list       (if 'CMMDS' parameter is "false" then communication to that disk is lost)     
          
    3. Determine if the 'failed' device is real/if it's the actual faulted drive:    
         
  •         If the 'Name' of a drive is missing (name should start with "naa.") and just the UUID appears, this can occur due to bad device partitions/diskgroup metadata. This often doesn't include any actual hardware faults. See https://support.emc.com/kb/529488 about these "ghost devices"/"phantom drives" and how to address them. In some cases, if steps in the article don't resolve these situations, a factory reset of the node will be the best way to move forward.     
  •      
  •         For clusters with 'Deduplication and Compression' enabled, failure may be indicated on a cache drive when a fault is present on a capacity drive in the diskgroup. Refer to article  https://support.emc.com/kb/495439     
  •      
  •         When diskgroups are formatted to be able to de-duplicate (Deduplication and Compression' enabled in the cluster), a failed capacity drive can be removed from the vSAN, but the remaining drives from its former diskgroup no longer function normally in the vSAN without it there. You might see failures/alerts related to other drives' vSAN operational state. This does not mean that other drives have hardware faults. With de-duplication, the entire disk group needs to be re-created. See the process at https://support.emc.com/kb/528355       
  •    
   
      User-added image  In this image, the bottom drive is a "ghost device" where a failed capacity drive used to be. The top one shows failures in 'Operational...' columns but doesn't actually indicate a hardware fault. Removing the diskgroup from the vSAN should result in the physically healthy disks showing up normally again. You can re-create the diskgroup later, after replacing any drives that are actually failed.   
   
    Steps AFTER determining that drive replacement should resolve the issue:     
    1. The VxRail Support Engineer (TSE) checks for and notes any additional issues that might impact drive replacement or that will need to be addressed afterwards.   
   
    2. Collect and upload logs to the Service Request (SR) if not done already. Expected logs are TSR logs (if Dell hardware environments), the VxRail Manager log bundle, and vCenter + host logs. *See information about collecting VxRail logs at https://support.emc.com/kb/333684    
   
    3. TSE verifies customer contact and shipping details are correct.   
   
    4. Customer chooses between replacing drives themselves or if a field resource needs to be sent on site to do it.   
   
    5. TSE updates case notes with details about the situation, upcoming disk replacement, and any special considerations.    
   
    6. TSE provides node replacement documentation (customer can also download from Solve). If 'De-duplication and Compression' is enabled on the cluster, TSE should attach https://support.emc.com/kb/528355 to the SR, ensure it is provided to the customer, and include it in 'CE Instructions' if a field resource will be sent.   
   
    7. The VxRail Support Engineer creates a work order for dispatch of part(s) and labor (if needed). At this point, the SR ownership moves from VxRail 'Remote Support' to 'Field Support'. If the original SR was not specifically for a failed drive, the TSE should create a new SR for the dispatch and retain ownership of the original one.   
   
    8. Someone from Dell EMC's scheduling team reaches out to the customer contact to finalize shipping/dispatch details.   
   
    9. VxRail Remote Support may be re-engaged as needed by the customer/partner or by on-site field resources through the original SR or a new one.   
   
                                                                                                                                

 

 

Notes: