avexvss Exchange Backup randomly failing with Powershell limit exceeded errors

           

   Article Number:     485835                                   Article Version: 3     Article Type:    Break Fix 
   

 


Product:

 

Avamar Client for Windows 7.2.100-401

 

Issue:

 

 

   

      Randomly customer seeing their Exchange Plugin backups failing with the error:     
      2016/06/13-23:15:34.22300 [avexvss_snapup_assist] Root domain context 'DC=xyz1,DC=com', default naming context 'DC=xyz,DC=com'.     
      2016-06-13 19:16:00 avexvss Error : Connecting to remote server failed with the following error message : The WS-Management service cannot process the request. This user is allowed a maximum number of 25 concurrent shells, which has been exceeded. Close existing shells or raise the quota for this user. For more information, see the about_Remote_Troubleshooting Help topic.     
      2016-06-13 19:16:00 avexvss Error : Unable to initialize Powershell interface process -- cannot continue. 2016-06-13 19:16:00 avexvss Error : Unable to complete backup. At the same time ps_exec.log also shows the error as:     
      2016-06-13 19:16:00.213 ps_exec.exe Info :: ERROR: Connecting to remote server failed with the following error message : The WS-Management service cannot process the request. This user is allowed a maximum number of 25 concurrent shells, which has been exceeded. Close existing shells or raise the quota for this user. For more information, see the about_Remote_Troubleshooting Help topic. Customer is using iisreset /noforce command to reset the state so that new backups not impacted.   
                                                             

 

 

Cause:

 

 

Engineering engaged via Escalation process:   
   
    Escalation 26591 - avexvss Exchange Backup randomly failing with Powershell limit exceeded errors   
   
    It was determined that in case of successful scenario where a PS session gets created without   
    failures, WinRM logs of the same shows ,   
    - When new-pssession is called, the WinRM calls WSMan API Initialize and WSMan Session initialize.   
    PS session get created.   
    - When remove-pssesion is called, the WinRM calls WSMan API Deinitialize and WSMan Session   
    Deinitialize(Closing WSMan Session completed successfully).   
   
    But during a failure case from the application logs found that,   
   
    - When new-pssession is called, the WinRM calls WSMan API Initialize and WSMan Session initialize.   
    PS session get created.   
    - When remove-pssesion is called, the WinRM calls WSMan API Deinitialize. But WSMan Session Deinitialize is not being called.   
   
    These were Microsoft API calls and they provided a solution as described below
                                                           

 

 

Change:

 

 

None                                                           

 

 

Resolution:

 

 

Steps to resolve this configuration issue were:   

         
  •         Disable the MaxShellPerUser group policy in place.     
  •      
  •         Upgrade the Windows Management Framework 2.0 to 3.0. The default version of windows management framework with windows 2008 r2 is 2.0. By upgrading, we make sure the remoteshell is upgraded and  stabilize the WMF compared to older version. Details at: https://www.microsoft.com/en-in/download/details.aspx?id=3459     
  •    
                                                             

 

 

Notes:

 

 

Referenced following links that provide useful leads related to this issue: