Failed to Import Data Domain CA certificate intermittently due to I/O constraints on the Data Domain

Environment:

Avamar

Data Domain

 

Description:

Import of the Data Domain CA certificate typically works, but the failed to import Data Domain CA certificate error pops up in the Avamar GUI intermittently.

Import of CA certificate fails

grep -i certificate mcserver.log.0

<snip>
07/14-01:57:29.00811 com.avamar.mc.datadomain.DdrCache.ConfigCerts
FINE: Unable to get DD imported ca certificate.
07/14-01:57:29.00811 com.avamar.mc.util.MCThreadPool.enqueueWork
FINE: com.avamar.mc.util.MCThreadPool-dbev queue depth: 1
07/14-01:57:29.00811 com.avamar.mc.datadomain.DdrCache.updateDdrPersistentStoreEx
FINE: No need to update persistent storage.
07/14-01:57:29.00811 com.avamar.mc.datadomain.DdrCache.importCertCaAppForDdboostEx
FINE: Deleting imported ca certificate...
07/14-01:57:29.00811 com.avamar.mc.datadomain.DdrSsh.checkSshClient
WARNING: Ssh client is not connected.
07/14-01:57:29.00811 com.avamar.mc.datadomain.DdrCache.importCertCaAppForDdboostEx
FINE: Importing ca certificate...
07/14-01:57:29.00811 com.avamar.mc.datadomain.DdrSsh.checkSshClient
WARNING: Ssh client is not connected.
07/14-01:57:29.00811 com.avamar.mc.datadomain.DdrCache.importCertCaAppForDdboostEx
WARNING: Failed to import ca certificate.
WARNING: Failed to import ca certificate.
<snip>


When reviewing the system show performance section of the Data Domain autosupports it appears that the Data Domain is under high I/O conditions during the time frame that the import fails.

-----------------------------------
Time Stamp    ||Utilization       
              ||    CPU      Disk 
  Date   Time ||  avg/max     max 
MM/DD/YY HH:MM||     %         %  
-------- -----  ----------- -------
07/14/16 01:36    22/ 35[0]  91[03]
07/14/16 01:46    22/ 35[0]  88[08]
07/14/16 01:56    14/ 23[0]  83[01]
07/14/16 02:06    13/ 21[0]  85[03]
07/14/16 02:16    12/ 21[0]  85[03]
07/14/16 02:26    11/ 17[0]  76[08]
07/14/16 02:36   10/ 17[12]  76[10]
07/14/16 02:46   11/ 18[12]  81[08]
07/14/16 02:56    12/ 19[0]  76[10]
07/14/16 03:06    15/ 25[0]  84[02]


Later on the import of the CA certificate succeeds again when the Data Domain is no longer under such high strain.

grep -i --context=1 "adminaccess certificate" mcserver.log.0

07/14-02:56:32.00572 com.avamar.mc.datadomain.DdrSsh.executeDdrCommand
FINE: Executing ddr command. host: testdd.emc.com cmd: adminaccess certificate show imported-host application ddboost...
07/14-02:56:32.00598 com.avamar.mc.datatap.ConnectionPool.getConnection
--
07/14-02:56:35.00631 com.avamar.mc.datadomain.DdrSsh.executeDdrCommand
FINE: Executing ddr command. host: testdd.emc.com cmd: adminaccess certificate export imported-ca-for-host application ddboost...
07/14-02:56:35.00639 com.avamar.mc.cr.bcConnection.readClientMessage
--
07/14-02:56:38.00502 com.avamar.mc.datadomain.DdrSsh.executeDdrCommand
FINE: Executing ddr command. host: testdd.emc.com cmd: adminaccess certificate show imported-host application ddboost...
07/14-02:56:41.00103 com.avamar.mc.cr.bcConnection.readClientMessage
--
07/14-02:56:44.00070 com.avamar.mc.datadomain.DdrSsh.executeDdrCommand
FINE: Executing ddr command. host: testdd.emc.com cmd: adminaccess certificate export imported-ca-for-host application ddboost...
07/14-02:56:44.00074 com.avamar.mc.cr.bcConnection.readClientMessage

 

 

Resolution:

An error occurs due to some I/O limitations on Data Domain and fails to import Data Domain CA certificate. During the time frame, in the Data Domain autosupports it appears that the Data Domain is under high I/O conditions that causes the import fails and later when the Data Domain is under normal I/O conditions it succeeds.

 

For a detailed step by step resolution please refer to EMC Support Solution 487121 https://support.emc.com/kb/487121