IDPA 2.1 Post Update Fails During DPSearch Update.sh

           

   Article Number:     531162                                   Article Version: 2     Article Type:    Break Fix 
   

 


Product:

 

Integrated Data Protection Appliance,Integrated Data Protection Appliance Software

 

Issue:

 

 

During the Idpa_post_update_2.1.0.599286 the DPsearch update.sh fails with Puppet cert Clean errors.    
   
    the following errors can be seen in the update logs:    

19/02/19 15:10:24 run_cmd_dps() [mNotice: Finished catalog run in 192.07 seconds[0m19/02/19 15:10:24 run_cmd_dps() Status: 219/02/19 15:10:24 run_puppet_agent_on_data_index() Running puppet agent on data index xx.xx.91.63('Error: [', "Warning: Permanently added 'xx.xx.91.63' (ECDSA) to the list of known hosts.", ']')19/02/19 15:10:28 run_cmd_dps() Cmd:rm -rf /etc/puppet/ssl && puppet agent -t19/02/19 15:10:28 run_cmd_dps() [0;32mInfo: Creating a new SSL key for dpsavdpsn02[0m19/02/19 15:10:28 run_cmd_dps() [0;32mInfo: Caching certificate for ca[0m19/02/19 15:10:28 run_cmd_dps() [0;32mInfo: Caching certificate for dpsavdpsn02[0m19/02/19 15:10:28 run_cmd_dps() [1;31mError: Could not request certificate: The certificate retrieved from the master does not match the agent's private key.19/02/19 15:10:28 run_cmd_dps() Certificate fingerprint: XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX19/02/19 15:10:28 run_cmd_dps() To fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certficate.19/02/19 15:10:28 run_cmd_dps() On the master:19/02/19 15:10:28 run_cmd_dps()   puppet cert clean dpsavdpsn0219/02/19 15:10:28 run_cmd_dps() On the agent:19/02/19 15:10:28 run_cmd_dps()   rm -f /etc/puppet/ssl/certs/dpsavdpsn02.pem19/02/19 15:10:28 run_cmd_dps()   puppet agent -t19/02/19 15:10:28 run_cmd_dps() [0m19/02/19 15:10:28 run_cmd_dps() Exiting; failed to retrieve certificate and waitforcert is disabled19/02/19 15:10:28 run_cmd_dps() Status: 119/02/19 15:10:28 log_failure() Failed to Upgrade Data Protection Search due to following error.19/02/19 15:10:28 log_failure() Failed to run update.sh on Data Protection search    
   
                                                                

 

 

Cause:

 

 

the DPSearch nodes are returning the Same hostname, it is a naming resolution.    
    This is either caused by the DNS entries or the /etc/hosts files.    
   
    Further research will need to be done. to determine where the hostnames are being incorrectly resolved.    
   
   
     in the DPS_upgrade.log we will see the following errors:    
        

we delete puppet certs for data nodes on master.    
   
19/02/19 15:03:11 delete_cert_entries() Deleting puppet certificate entries from master node xx.xx.91.60    
        
As you can see from the log below, puppet cert clean was called for multiple times for dpsavdpsn02    
   
('Error: [', "Warning: Permanently added 'xx.xx.91.60' (ECDSA) to the list of known hosts.", ']')19/02/19 15:03:13 run_cmd_dps() Cmd:puppet cert clean dpsavdpsn0119/02/19 15:03:13 run_cmd_dps() [1;31mError: Could not find a serial number for dpsavdpsn01[0m    (these are hostnames of the dpsearch nodes)19/02/19 15:03:13 run_cmd_dps() Status: 24('Error: [', "Warning: Permanently added 'xx.xx.91.60' (ECDSA) to the list of known hosts.", ']')19/02/19 15:03:14 run_cmd_dps() Cmd:puppet cert clean dpsavdpsn0219/02/19 15:03:14 run_cmd_dps() [1;31mError: Could not find a serial number for dpsavdpsn02[0m    (these are hostnames of the dpsearch nodes)('Error: [', "Warning: Permanently added 'xx.xx.91.60' (ECDSA) to the list of known hosts.", ']')19/02/19 15:03:16 run_cmd_dps() Cmd:puppet cert clean dpsavdpsn0219/02/19 15:03:16 run_cmd_dps() [1;31mError: Could not find a serial number for dpsavdpsn02[0m    (these are hostnames of the dpsearch nodes)19/02/19 15:03:16 run_cmd_dps() Status: 24('Error: [', "Warning: Permanently added 'xx.xx.91.60' (ECDSA) to the list of known hosts.", ']')19/02/19 15:03:17 run_cmd_dps() Cmd:puppet cert clean dpsavdpsn0219/02/19 15:03:17 run_cmd_dps() [1;31mError: Could not find a serial number for dpsavdpsn02[0m     (these are hostnames of the dpsearch nodes)    
        
Index Data 2 : xx.xx.91.62 - got hostname dpsavdpsn02Index Data 3 : xx.xx.91.63 - got hostname dpsavdpsn02 , should have been dpsavdpsn03Index Data 4 : xx.xx.91.64 - got hostname dpsavdpsn02 , should have been dpsavdpsn04The reason it failed is because "puppet cert clean dpsavdpsn03 and dpsavdpsn04 was not called"    
                                                             

 

 

Resolution:

 

 

on the ACM you can check the IP address and hostname resolutions with the following script:    
        

testgrid:/usr/local/dataprotection # cat ~/abc.py#!/usr/bin/env pythonimport socketprint socket.gethostbyaddr("dpsearch node 1 IPaddress")print socket.gethostbyaddr("dpsearch node 2 IPaddress")print socket.gethostbyaddr("dpsearch node 3 IPaddress")    
   
    your output after the script is run will look like the following:    
        
acm01:~ # python ~/abc.py('dpsavdpsm.scasurgery.net', [], ['xx.xx.91.60'])('dpsavdpsn01.scasurgery.net', [], ['xx.xx.91.61'])('dpsavdpsn02.scasurgery.net', [], ['xx.xx.91.62'])('dpsavdpsn02.scasurgery.net', [], ['xx.xx.91.63'])('dpsavdpsn02.scasurgery.net', [], ['xx.xx.91.64'])    
   
    update the DNS entries or host files on the DPSearch nodes to give the correct Hostnames for each IP address.