Determining Whether an Avamar System is Experiencing a Time Synchronization (NTP) issue.

If nodes within an Avamar system are not time synchronized, we can expect the following types of behavior:

• The Avamar server is unable to start
• Nodes go offline
• HFScheck fails with MSG_ERR_CGSAN_FAILED
• HFScheck fails with MSG_ERR_HFSCHECKERRORS
• Checkpoints fail
• Garbage Collection fails
• Data consistency issues (if the time changes during garbage collection)

Examples of error messages commonly reported as a result of loss of time synchronization:

• samconn::checkallsucceed request failed DPNTIMECHECK=230 
• FATAL ERROR: <0001> dpn time mismatch: synchronize clocks and retry
• ERROR: <0001> dpncheckmanager::verifyStartup cgsan died unexpectedly. terminating  
• not enough valid responses received in time

This article helps the reader determine whether the Avamar system is experiencing a time synchronization issue. Resolving the issue is beyond the scope of this article.

There are many websites which cover NTP troubleshooting and the reader is encouraged to investigate them. Helpful web URLs available at the time of writing are listed in the 'external links' section.

To proceed:
1. Log in to the Avamar server as admin per KB Avamar: How to Log in to an Avamar Server and Load Various Keys..

2. To determine whether Avamar nodes are time synchronized, check the current time and date of each node on the Avamar system. See APPENDIX A for sample output.

mapall --all --parallel '/bin/date'

When all nodes report the same date and time this means that the time is fully synchronized between all the nodes on this system.

3. To keep time synchronized on the nodes, Avamar uses Network Time Protocol (NTP). The Linux command "ntpq -pn" returns the state of time synchronization. See APPENDIX B for sample output.

mapall --all --noerror '/usr/sbin/ntpq -p'

4. General Avamar Server Observations:

• All nodes are set to prefer 128.xxx.xxx.xx as the primary time source.
• The secondary time source for all nodes is the local BIOS clock on "avmtest1" (node 0.s).
• The tertiary time source is set to be avmtest2 (node 0.0) which is itself referring to avmtest1.
• All nodes are synchronizing with avmtest1. The time server marked with an asterisk (*) is the one that the node is currently synchronizing with.
• In this case, 128.xxx.xxx.xx is located remotely. It has a 'reach' value of 0 (currently unreachable). It is useless as a time source.
• avmtest1 and avmtest2 both have a reachability register of octal 377. This is the highest figure attainable. Therefore, the nodes are all synchronizing with the secondary source.

5. Looking at the ntpq output for node 0.2;

(0.2) ssh  -x  admin@10.64.18.164 '/usr/sbin/ntpq -p'

     remote           refid      st t when poll reach   delay   offset  jitter

==============================================================================
 128.xxx.xxx.xx  .INIT.          16 u    - 1024    0    0.000    0.000 4000.00
*avmtest1.emcvmw LOCAL(0)         9 u   54  256  377    0.085   -0.116   0.002
+avmtest2.emcvmw xx.xx.xx.xxx    10 u   56  256  377    0.090    0.073   0.012

• Node 0.2 is polling avmtest1 every 256 seconds
• Node 0.2 is currently synchronizing with avmtest1
• avmtest1 is at stratum 9, implying that node 0.2 is at stratum 10.
• Node 0.2 is polling avmtest1 once every 256 seconds.
• The reachability register for avmtest1 is octal 376.
• The clock on avmtest1 is 0.116 milliseconds (or 116 microseconds) behind the clock on avmtest1.
• The roundtrip delay to avmtest1 is 85 milliseconds.
• The measurement of the variance in latency on the network (jitter) between node 0.2 and avmtest1 is 2 milliseconds.

NTP configuration (/etc/ntp.conf):
If reviewing the /etc/ntp.conf file on node 0.2  it corresponds to the ntpq output above.

#Customer premises / external time servers.

#
server xxx.xxx.xxx.xx     <--  Primary time source (this is an external server located remote to the Avamar grid)
# - - - - -
# DPN time servers here and in the other module(s).
#
server xx.xx.xx.xxx   <--  Secondary time source (this is the utility node)
server xx.xx.xx.xxx   <--  Tertiary time source (this is node 0.0)

Logging:
NTP logging is directed to the /var/log/messages file.
To view NTP-related logging, grep the contents of /var/log/messages* for 'ntp'

If an external time server is unreliable, as in the example given above, it is acceptable to use an internal time server. The internal time may drift slowly from UTC, but the most important consideration is that data nodes are time synchronized with one another.

The Avamar utility asktime tool can be used to select new, preferred time sources for NTP.
See Avamar: How to configure NTP on an Avamar Server using asktime 

Additional Information:
http://support.microsoft.com/kb/939322 - Windows Domain controllers should not be used for good time keeping.