IDPA DP4400 - changeAvamarPasswords - Failed to execute change-passwords command on avamar utility node

           

   Article Number:     534860                                   Article Version: 3     Article Type:    Break Fix 
   

 


Product:

 

DP4400 Appliance,Integrated Data Protection Appliance

 

Issue:

 

 

IDPA DP4400 Changing password from the ACM Dashboard fails to change Avamar Utility Node OS passwords and reports errors:   

         
  •         [FAILED] Backup Server - Failed to change password for Avamar OS user.     
  •      
  •         [FAILED] Backup Server - Failed to change password for Avamar server user.      
  •    
   
    Additional Details:   
         
  •         All other passwords update successfully     
  •      
  •         ACM /usr/local/dataprotection/var/configmgr/server_data/logs/server.log file reports error:     
  •    
   
      ERROR [Thread-256364]-avadapter.AvamarUtil: changeAvamarPasswords -->Failed to execute change-passwords command on avamar utility node   
                                                             

 

 

Cause:

 

 

When the change password option is invoked from the ACM Dashboard, the ACM connects to each DP4400 component and carries out the relevant password change.  For the Avamar virtual component AVE, (Avamar Virtual Edition), on the DP4400 , it connects via ssh to the AVE Utility node as user "root" and executes the local "change-password" script. In this instance, the change-password script failed because the prompt expected, after "su", did not prompt "Password" in timely manner and hence it failed with a timeout, as noted in the ACM log file: /usr/local/dataprotection/var/configmgr/server_data/logs/server.log  Below is the excerpt:   
   
    2019-05-29 17:06:13,100 INFO  [Thread-256364]-util.SSHUtil: Creating session using SSH parameters:      Host     : [10.99.2.214]     User     : [admin]     Password : [**********]   
    2019-05-29 17:06:13,100 INFO  [Thread-256364]-util.SSHUtil: Connecting to host [10.99.2.214] using provided credentials.   
    2019-05-29 17:06:13,331 INFO  [Thread-256364]-util.SSHUtil: Connected to host [10.99.2.214] using provided credentials.   
    2019-05-29 17:06:13,331 INFO  [Thread-256364]-util.SSHUtil: executeCmdWithChannelShell --> Opening a channel for communication.   
    2019-05-29 17:06:13,331 INFO  [Thread-256364]-util.SSHUtil: executeCmdWithChannelShell --> Channel for communication opened successfully.   
    2019-05-29 17:06:13,332 INFO  [Thread-256364]-util.SSHUtil: executeCmdWithChannelShell --> Channel connected successfully.   
    2019-05-29 17:06:13,332 INFO  [Thread-256364]-util.SSHUtil: executeCmdWithChannelShell -->  Executing command su   
    2019-05-29 17:36:13,340 ERROR [Thread-256364]-util.SSHUtil: executeCmdWithChannelShell -->  Error in command execution Expect operation fails (timeout: 1800000 ms) for matcher: contains('Password:')   
    2019-05-29 17:36:13,340 INFO  [Thread-256364]-util.SSHUtil: executeCmdWithChannelShell -->  Executing command ssh-agent bash   
    2019-05-29 17:36:13,341 ERROR [Thread-256364]-avadapter.AvamarUtil: changeAvamarPasswords -->Failed to execute change-passwords command on avamar utility node
                                                           

 

 

Resolution:

 

 

The workaround to correct this issue is to manually execute the "change-password" script as user "root" from the Avamar Utility node of the AVE. The "change-password" script is menu driven and once the script concludes, the ACM service will recognize that a password change has taken place and hence prompt, on the ACM Dashboard,  for the AVE component: "Backup Server", that the 'Backup Server OS 'admin' user password is out of synch. Please update the latest password'. Once that message appears, you simply click on it and provide the new common ACM password:   
        

      User-added image   
   
   
    Notes:   
   
    In addition to updating the the OS users admin and root users passwords, on the Avamar Utility Node AVE, the "change-password" script will also prompt to update four Avamar Server Users password. These are:   
         
  1.         MCUser     
  2.      
  3.         viewuser     
  4.      
  5.         root     
  6.      
  7.         repluser     
  8.    
    The end result of updating these four additional user passwords is that another message will appear on the ACM dashboard, once the first  'Backup Server OS admin user password is out of synch' has been synched up. This second message is: 'Backup Server 'root', 'MCUser', 'repluser' and 'viewuser' user password is out of synch. Please update the latest password.' :   
        
      User-added image     
          
   
     
      STEPS:   
   
         
  1.         Logged into the ACM and ensure ensure components show green. If there is any component that is not showing green, ideally that issue needs to be fixed first     
  2.      
  3.         Using an ssh client such as putty.exe, ssh to the Avamar Utility Node as user admin. The password will be the original password prior to the password change was invoked from the ACM     
  4.      
  5.         Issue command "unset TMOUT" to stop the admin ssh session from timing out     
  6.      
  7.         Switch user to user root - issue command:  "su -"     
  8.      
  9.         Issue command "unset TMOUT" to stop the root ssh session from timing out     
  10.      
  11.         Next duplicate the ssh session as user admin to the to the Avamar Utility Node and then switch user to user root, i.e. repeat steps 2,3,4 & 5. This is important,  as a precaution, to ensure you do not get locked out from the Avamar Utility Node after changing the OS root and admin user passwords.     
  12.      
  13.         From this second ssh session to the Avamar Utility Node, issue command: "change_password"  and follow the prompts.       
  14.      
  15.         For the very first prompt, answer no:     
  16.    
   
Do you wish to specify one or more additional SSH passphrase-less    private keys that are authorized for root operations?Answer n(o) here unless there are known inconsistencies in    ~root/.ssh/authorized_keys files among the various nodes.Note that the following key will be used automatically (i.e., there is    no need to re-specify it here):      /root/.ssh/rootidy(es), n(o), h(elp), q(uit/exit): no    
   
          
   
         
  1.          The remaining prompts will ask to update the OS admin and root passwords. Once that concludes, it will prompt to update the additional four internal Avamar user passwords for users, MCUser, root, repluser & viewuser. In addition, it will ask to update the ssh keys for the OS root users. Ensure to select yes  to update the ssh keys for the OS users root and admin. Below are the remaining updates:     
  2.    
        
--------------------------------------------------------The following is a test of OS root authorization with the currently    loaded SSH key(s).    If the authorization test fails, then you might be missing an    appropriate private key, e.g., rootid or dpnid.        -> In that event, re-run this program and, when prompted,           specify as many SSH private key files as are necessary           in order to complete root operations.Starting root authorization test with 600 second timeout...End of root authorization test.--------------------------------------------------------Change OS (login) passwords?y(es), n(o), q(uit/exit): yeschange-passwords: INFO: Each OS password will be changed locally without further prompting as soon as you have (twice) entered a valid password.--------------------------------------------------------Change OS password for "admin"?y(es), n(o), q(uit/exit): yesEnter a new OS (login) password for user "admin".(Entering an empty (blank) line twice quits/exits.)>Enter the same OS password again.(Entering an empty (blank) line twice quits/exits.)>Changing password for admin.change-passwords: INFO: the password for OS user admin has been updated on _this_ host.change-passwords: INFO: the password will not be reverted if you later decline to update passwords/passphrases.Accepted OS password for "admin".--------------------------------------------------------Change OS password for "root"?y(es), n(o), q(uit/exit): yesEnter a new OS (login) password for user "root".(Entering an empty (blank) line twice quits/exits.)>Enter the same OS password again.(Entering an empty (blank) line twice quits/exits.)>Changing password for root.change-passwords: INFO: the password for OS user root has been updated on _this_ host.change-passwords: INFO: the password will not be reverted if you later decline to update passwords/passphrases.Accepted OS password for "root".--------------------------------------------------------Generate new SSH keys?y(es), n(o), h(elp), q(uit/exit): yes--------------------------------------------------------Change Avamar Server passwords?y(es), n(o), q(uit/exit): yes--------------------------------------------------------Please enter the CURRENT server password for "root"(Entering an empty (blank) line twice quits/exits.)>Checking Avamar Server root password (1200 second timeout)...Avamar Server current root password accepted.--------------------------------------------------------Change Avamar Server password for "MCUser"?y(es), n(o), q(uit/exit): yesPlease enter a new Avamar Server password for user "MCUser".(Entering an empty (blank) line twice quits/exits.)>Enter the same Avamar Server password again.(Entering an empty (blank) line twice quits/exits.)>Accepted Avamar Server password for "MCUser".--------------------------------------------------------Change Avamar Server password for "root"?y(es), n(o), q(uit/exit): yesPlease enter a new Avamar Server password for user "root".(Entering an empty (blank) line twice quits/exits.)>Enter the same Avamar Server password again.(Entering an empty (blank) line twice quits/exits.)>Accepted Avamar Server password for "root".--------------------------------------------------------Change Avamar Server password for "repluser"?y(es), n(o), q(uit/exit): yesPlease enter a new Avamar Server password for user "repluser".(Entering an empty (blank) line twice quits/exits.)>Enter the same Avamar Server password again.(Entering an empty (blank) line twice quits/exits.)>Accepted Avamar Server password for "repluser".--------------------------------------------------------Change the viewuser password?y(es), n(o), h(elp), q(uit/exit): yesChecking Administrator Server status...EUA2-Enter the NEW viewuser password.Enter ? or help for help.(Entering an empty (blank) line twice quits/exits.)>For verification, re-enter the NEW viewuser password.Enter ? or help for help.(Entering an empty (blank) line twice quits/exits.)>ERROR: your entries for "viewuser" did not match.        Please try again.Enter the NEW viewuser password.Enter ? or help for help.(Entering an empty (blank) line twice quits/exits.)>For verification, re-enter the NEW viewuser password.Enter ? or help for help.(Entering an empty (blank) line twice quits/exits.)>--------------------------------------------------------Do you wish to proceed with your changes on the selected node?        Answering y(es) will proceed to make changes.        Answering n(o) or q(uit) will not proceed.y(es), n(o), q(uit/exit): yesChanging OS passwords...[Logging to /usr/local/avamar/var/change-passwords.log...]Done changing OS passwords...Changing Avamar Server passwords...Suspending maintenance cron jobsChecking Administrator Server status...Stopping Administrator Server...Starting process of updating Administrator and Enterprise Manager configurations...Running script to update Administrator and Enterprise Manager configurations on node 0.s...[Logging to /usr/local/avamar/var/change-passwords.log...]Done with updating Administrator configuration on node 0.s...Starting process of updating client configurations...Running script to update client configuration on all+...[Logging to /usr/local/avamar/var/change-passwords.log...]Updating client configuration on node 0.0...Done updating client configuration on 0.0...Starting process of updating mccli configuration files...Running script to update mccli configuration files on node set "0.0"...[Logging to /usr/local/avamar/var/change-passwords.log...]Done with updating mccli configuration files on node 0.0...Checking Administrator Server status...Starting Administrator Server...Stopping dtlt (DT/LT) subsystemStarting dtlt (DT/LT) subsystemResuming maintenance cron jobsStarting process of changing SSH keys...Running script to update SSH keys on node 0.s...[Logging to /usr/local/avamar/var/change-passwords.log...]Done with updating SSH keys on node 0.s...Starting process of updating viewuser password...Checking Administrator Server status...Stopping Administrator Server...Running script to update mcdb viewuser password on node 0.0...[Logging to /usr/local/avamar/var/change-passwords.log...]Done with updating mcdb viewuser password on node 0.0...Checking Administrator Server status...Starting Administrator Server...--------------------------------------------------------Done.NOTES:- If you had custom public keys present in the      authorized_keys files of any Avamar OS users      (admin, root) be aware that      you may need to re-add your custom keys.- If mccli (the Administrator command line interface)      is used from any remote user accounts, then please update      the password in each remote account's copy of the mccli      preferences/configuration file, typically      ~USER/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml.- Please be sure to resume schedules via the        Administrator GUI or via 'dpnctl start sched'.root@idpa-avamar-lj:~/#:    
   
         
  1.                 
              As per the messages, it is important to confirm the the mccli preferences configuration file did get updated. This is best tested by issue a mccli command as user admin and as user root. Below is an example as user root. Both command should work. If they do not, then you will need to update the ~USER/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml file: root@idpa-avamar-lj:~/#: mccli user show0,23000,CLI command completed successfully.Name          Role                     Domain Authenticator------------- ------------------------ ------ ---------------------------MCUser        Administrator            /      Axion Authentication Systembackuponly    Back up Only User        /      Axion Authentication Systembackuprestore Back up/Restore User     /      Axion Authentication Systemrepluser      Replication User         /      Axion Authentication Systemrestoreonly   Restore (Read) Only User /      Axion Authentication Systemroot          Administrator            /      Axion Authentication Systemroot@idpa-avamar-lj:~/#:        
                  
  2.      
  3.         Additionally, as per the message, it is also important to re-start the backup schedular as this is disabled when the MCS service is stopped and restarted duirng the password change:     
  4.    
   
      root@idpa-avamar-lj:~/#: dpnctl status     
      Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)     
      dpnctl: INFO: gsan status: up     
      dpnctl: INFO: MCS status: up.     
      dpnctl: INFO: emt status: up.     
      dpnctl: INFO: Backup scheduler status: down.     
      dpnctl: INFO: Maintenance windows scheduler status: enabled.     
      dpnctl: INFO: Unattended startup status: disabled.     
      dpnctl: INFO: avinstaller status: up.     
      dpnctl: INFO: ConnectEMC status: up.     
      dpnctl: INFO: ddrmaint-service status: up.     
     
      root@idpa-avamar-lj:~/#: dpnctl start sched     
      Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)     
      dpnctl: INFO: Resuming backup scheduler...     
      dpnctl: INFO: Backup scheduler resumed.     
      dpnctl: INFO: No /usr/local/avamar/var/dpn_service_status exist.     
      root@idpa-avamar-lj:~/#:     
     
          
   
         
  1.         Finally, verify that the OS admin and root user passwords updated successfully. To do this, open a new ssh session to the Avamar Backup server, logon as user admin with the updated common password. Once successfully logged on, switch user to user root using the "su -" and enter the same updated common password. Once both users have been verified,     
  2.      
  3.         Switch over to the ACM dashboard and refresh the screen. You should see the following message. Click on the message and enter the same common password used for the admin user and save the password. Once successfully saved, you will see a message "Backup Server, password updated successfully":     
  4.    
   
      User-added image   
   
         
  1.         Once the backup server password has been updated successfully, you will see the following 2nd message. Click on the message and enter the same common password used for the admin user and save the password. Once successfully saved, you will see a message that the password was updated successfully:     
  2.    
   
      User-added image   
        
          
   
         
  1.         At this point, the ACM dasboard should show all components as  green. This concludes the workaround for the OS password for the DP4400 AVE issue      
  2.    
                                                             

 

 

Notes:

 

 

At the time of writing this article, the prompts for the change-password script were as per this article. These prompt may change a little from one version to the next and as such, it is advised that the prompts are carefully read to ensure the wrong selection is is specified.