Data Protection Search: Upgrade of Search to version 18.2.0.2410 shows up as complete within ACM whereas validate_search_install.sh script reports failures

           

   Article Number:     534316                                   Article Version: 3     Article Type:    Break Fix 
   

 


Product:

 

Integrated Data Protection Appliance Family,Integrated Data Protection Appliance Software,Integrated Data Protection Appliance,Integrated Data Protection Appliance 2.3,Integrated Data Protection Appliance 2.4,Data Protection Search

 

Issue:

 

 

After upgrading Search on Integrated Data Protection Appliance (IDPA) to version 18.2.0.2410, components of search nodes becomes inaccessible.  ACM GUI will report upgrade for search with completed state.   
   
    If validate_search_install.sh script for search is run output similar to the following is reported:   
        

idpa-search:/home/search # bash validate_search_install.sh 18.2.0.2410 admin Idpa_12345|Check Item                   |Expect                          |Actual---------------------------------------------------------------------------------------------|SEARCH_HOME                  |Exist                           |Exist|CIS_HOME                     |Exist                           |Exist|Search    Version            |18.2                            |18.2.0.2410|lgtoxtdclnt Version          |9.2.1.0                         |9.2.1.0|lgtoclnt    Version          |9.2.1.0                         |9.2.1.0|Java        Version          |1.8.0_192                       |1.8.0_192|Ruby        Version          |2.1.9p490                       |2.1.9p490|Nginx       Version          |1.12.2                          |1.12.2|Elasticsearch Version        |5.2.2                           |5.2.2---------------------------------------------------------------------------------------------|slapd                        |Running                         |Running|nginx                        |Running                         |Not Running|elasticsearch                |Running                         |Running|search-cis-core              |Running                         |Running|search-cis-schedule          |Running                         |Running|search-api                   |Running                         |Running|search-worker                |Running                         |Running|search-adminapi              |Running                         |Running|search-avamar-action         |Running                         |Running|search-avamar-worker         |Running                         |Running|search-networker-action      |Running                         |Running|search-networker-worker      |Running                         |Running---------------------------------------------------------------------------------------------|slapd                        |Enabled                         |Enabled|nginx                        |Enabled                         |Disabled|elasticsearch                |Enabled                         |Enabled|search-cis-core              |Enabled                         |Enabled|search-cis-schedule          |Enabled                         |Enabled|search-api                   |Enabled                         |Disabled|search-worker                |Enabled                         |Disabled|search-adminapi              |Enabled                         |Disabled|search-avamar-action         |Enabled                         |Disabled|search-avamar-worker         |Enabled                         |Disabled|search-networker-action      |Enabled                         |Disabled|search-networker-worker      |Enabled                         |Disabled---------------------------------------------------------------------------------------------|defaultPasswordPolicy        |dn:cn=defaultPasswordPolicy...  |dn: cn=defaultPasswordPolicy,ou=Policy,dc=example,dc=com|AdminGroup                   |dn:cn=AdminGroup...             |dn: cn=AdminGroup,ou=Group,dc=example,dc=com|Admin  login                 |"message":ok                    |"message":authenticate failed|Search login                 |"message":ok                    |"message":authenticate failed---------------------------------------------------------------------------------------------    
    Attempts to start nginx service manually will fail.                                                            

 

 

Cause:

 

 

This problem is caused by software defect.                                                           

 

 

Resolution:

 

 

Allow 1 hour for upgrade to finish behind the scenes, only then reboot the Search node. After reboot of search node, run bash script "validate_search_install.sh" again to make sure all components are reporting correct version and services are running as expected.    
        

idpa-search:/home/search # bash validate_search_install.sh 18.2.0.2410 admin Idpa_12345|Check Item                   |Expect                          |Actual---------------------------------------------------------------------------------------------|SEARCH_HOME                  |Exist                           |Exist|CIS_HOME                     |Exist                           |Exist|Search    Version            |18.2.0.2410                     |18.2.0.2410|lgtoxtdclnt Version          |9.2.1.0                         |9.2.1.0|lgtoclnt    Version          |9.2.1.0                         |9.2.1.0|Java        Version          |1.8.0_192                       |1.8.0_192|Ruby        Version          |2.1.9p490                       |2.1.9p490|Nginx       Version          |1.12.2                          |1.12.2|Elasticsearch Version        |5.2.2                           |5.2.2---------------------------------------------------------------------------------------------|slapd                        |Running                         |Running|nginx                        |Running                         |Running|elasticsearch                |Running                         |Running|search-cis-core              |Running                         |Running|search-cis-schedule          |Running                         |Running|search-api                   |Running                         |Running|search-worker                |Running                         |Running|search-adminapi              |Running                         |Running|search-avamar-action         |Running                         |Running|search-avamar-worker         |Running                         |Running|search-networker-action      |Running                         |Running|search-networker-worker      |Running                         |Running---------------------------------------------------------------------------------------------|slapd                        |Enabled                         |Enabled|nginx                        |Enabled                         |Enabled|elasticsearch                |Enabled                         |Enabled|search-cis-core              |Enabled                         |Enabled|search-cis-schedule          |Enabled                         |Enabled|search-api                   |Enabled                         |Enabled|search-worker                |Enabled                         |Enabled|search-adminapi              |Enabled                         |Enabled|search-avamar-action         |Enabled                         |Enabled|search-avamar-worker         |Enabled                         |Enabled|search-networker-action      |Enabled                         |Enabled|search-networker-worker      |Enabled                         |Enabled    
                                                             

 

 

Notes:

 

 

ACM didn't wait long enough for all nodes to be upgraded before marking search upgrade with completed state. In future releases, ACM will be modified to wait for 1 hour for all search nodes to finish upgrade or search-upgrade-18.2.0.2410.zip-allcompleted file to be created instead of search-upgrade-18.2.0.2410.zip-completed file that ACM uses currently.