Here are few most frequently asked questions and most common issues related to Avamar Replication.
There is an Avamar source and an Avamar target system. Capacity utilisation is higher on one system than the other.
Replication fails with errors linked to permissions.
2010/11/24-18:02:11 rununtil -82672 avmgr newd --acnt=/REPLICATE --duplicates-okay --hfsaddr=avamar_target.emc.com --id=repluser --ap=*****
2010/11/24-18:02:11 REPLICATE ERROR: Command returned 5120 (0x1400): rununtil -82672 avmgr newd --acnt=/REPLICATE --duplicates-okay --hfsaddr=avamar_target.emc.com --id=repluser --ap=*****
2010/11/24-18:02:12 rununtil -82671 avmgr newd --acnt=/REPLICATE/avamar_source.emc.com --duplicates-okay --hfsaddr=avamar_target.emc.com --id=repluser --ap=*****
2010/11/24-18:02:12 REPLICATE ERROR: Command returned 5120 (0x1400): rununtil -82671 avmgr newd --acnt=/REPLICATE/avamar_source.emc.com --duplicates-okay --hfsaddr=avamar_target.emc.com --id=repluser --ap=*****
Calculating how much data Avamar can replicate on a daily basis
Replication is not completing within the backup window. The 'how to' below describes the factors involved in understanding how much data you can expect the Avamar system to be able to replicate each day.
Please refer to KB Article 110852 to see resolution for this issue.
Avamar replication troubleshooting - questions to ask and data to gather
The scope of this KB article is to describe which logs and information are needed in order to perform general troubleshooting of an Avamar replication job which is failing.
Please refer to KB Article 117124 to see resolution for this issue.
Unencrypted Backups & Replication Fail in Avamar 7.1 Due to Increased Firewall Restrictions
The issue that is occurring is the avfirewall is filtering backups over port 27000.
Networker 8.x is not compatible with Data Deduplication with Avamar 7.1.
This ONLY applies to new Avamar 7.1 installation and does not apply to Avamar server upgraded to version 7.1.
Cron based replication to an Avamar 7.1 server fails with one or more of following error in /usr/local/avamar/var/cron/replicate.log (where target.local is the replication target server).
2014/10/14-17:36:00 ERROR: avmaint: hfscreatetime: cannot connect to server target.local at 192.168.10.10:27000
2014/10/14-17:36:01 avmgr logn --hfsaddr=target.local --id=repluser --ap=*****
2014/10/14-17:36:11 0 ERROR! Exit code 15: Cannot connect to Avamar dispatcher
2014/10/14-17:36:11 REPLICATE ERROR: Command returned 3840 (0xf00): avmgr logn --hfsaddr=target.local --id=repluser --ap=*****
2014/10/14-17:36:11 ABORT: Could not login to destination DPN. Check flags or local config files (repl_cron.cfg)
2014/10/14-17:36:11 dpncron.pm::docmdout got exit code 256 from 'replicate --flagfile=/usr/local/avamar/etc/repl_cron.cfg', returning 0
2014/10/14-17:36:11 ======== failed replication (pid=28702) ========
2014/10/14-17:36:11 mccli event publish --code=4602 --message='repl_cron - failed replication (pid=28702)'
2014/10/14-17:36:19 0,23000,CLI command completed successfully.
2014/10/14-17:36:19 avmaint infomessage --avamaronly --errcode=4602 'repl_cron - failed replication (pid=28702)'
2014/10/14-17:36:20 ============================== Finished repl_cron ==============================
Unencrypted Avamar client backups (over port 27000) fail with the following messages in the client avtar logs.
avmaint Info <5562>: - - Connect: Trying <IP>:27000
avmaint Info <5694>: - Failed initial handshake, trying again
avtar Info <5552>: Connecting to Avamar Server (<AV Server>)
avtar Info <5554>: Connecting to one node in each datacenter
avtar Info <5694>: - Failed initial handshake, trying again
avtar Info <6063>: - Communication error: Could not create connection to Server
Telnet\curl from client to server over port 27000 also fails.
Netstat shows we are listening on port 27000 for the appropriate IP's.
netstat -anlp | grep 27000
(No info could be read for "-p": geteuid()=500 but you should be root.)
tcp 0 0 <IP>:27000 :::* LISTEN -
On the target Avamar server the firewall is running:
Output from Gen3 Hardware:
rpm -qa | grep avfwb
service avfirewall status
avfirewall is running...
Output from Gen4/Gen4s Hardware:
rpm -qa | grep avfw
service avfirewall status
Checking for service avfirewall running
Please refer to KB Article 193138 to see resolution for this issue.
Avamar Administrator does not allow /REPLICATE to be selected when running backend capacity report
When configuring the backend report through the Avamar Administrator interface the /REPLICATE domain is not displayed along
with the other domains when attempting to choose which domains and clients should be included in the report.
If the backend report is called from the Avamar command line and the /REPLICATE domain is specified,
no meaningful data is produced. See below for example output
nohup backendreport --include=/REPLICATE/sourceavamar.domain.com/domain/testclient.domain.com --avtar=--clearcache --avtar=--hashcachemax=-10 --small-client-mb=0 --verbose > backendreport-repl.out &
admin@avmtest1:~/nric/>: tail -f backendreport-repl.out
Changing working directory to /tmp/replicate
INFO: Inclusion patterns are: "/REPLICATE/sourceavamar.domain.com/domain/testclient.domain.com"
1 Request succeeded
-1 privilege level (enabled,create,read,backup,access,move,delete,maint,manage,fullmanage,noticketrequired,
2 block type (directory)
avmgr cpdb --path=/ --mvpath=/REPLICATE/ --out=replnew.sh --avmgr-path=avmgr --failover
Script "replnew.sh" for target path /REPLICATE copies 36 users in 115 accounts (81 clients)
Accounting changes: 0 differences
DONE Replicate to successful, elapsed 0000:00:08 (Wed Jan 22 09:42:18 GMT 2014)
<backendreport version="6.1.2-46" reporttime="2014/01/22-09:42:18" bytessent="0" totaltime="0">
completed backendreport (pid=10074)
============================== Finished backendreport ========================================
Please refer to KB Article 190414 to see resolution for this issue.
Multiple replication streams with different clients are failing with the following error message "Cache file for "client1" is locked, probably due to another replication -- Aborting"
Multiple replication streams are configured.
Secondary replication stream is configured.
Both replication streams are configured with different clients.
For instance, the replication streams could be configured as follows:
Replication Stream 1
Parsed flags: --hfsaddr=avamargrid.com --dstaddr= avamargrid.com --dstid=root --dstpassword=***** --dpnname=sourcegrid.com --id=root --ap=***** --srcpath=//domain/client1 --dstpath=/REPLICATE/sourcegrid.com/domain/client1
Replication Stream 2
Parsed flags: --hfsaddr=avamargrid.com --dstaddr= avamargrid.com --dstid=root --dstpassword=***** --dpnname=sourcegrid.com --id=root --ap=***** --srcpath=/domain/client2 --dstpath=/REPLICATE/sourcegrid.com/domain/client2
Although replication streams have different clients configured, both streams attempt to replicate the same client.
This can be observed by examining the logs for both streams. For instance, the logs would show the following:
Replication Stream 1 log snippet:
INFO: Replicating "domain/client1" to "/REPLICATE/sourcegrid.com/domain/client1/" (approx. 0 MB: small client)
Replication Stream 2 lot snippet:
INFO: Replicating "domain/client1" to "/REPLICATE/sourcegrid.com/domain/client1(approx. 0 MB: small client)
In the above example, both streams are replicating client1 ; one of the streams will show the following error message:
avtar Error <5064>: Cannot open file "/usr/local/avamar/var/p_sourcegrid.com-client1.dat"
avtar Info <5065>: Creating new cache file /usr/local/avamar/var/sourcegrid.com-client1.dat (1,573,408 bytes)
avtar Error <5803>: Error writing 32-byte header to cache file /usr/local/avamar/var sourcegrid.com-client1.dat. Possibly out of disk space
avtar Error <6044>: Cache file for "client1" is locked, probably due to another replication -- Aborting
One stream will fail with above error messages and the other stream will continue to run.
Please refer to KB Article 123886 to see resolution for this issue.
Replication Failing with Error <10772>: The target server doesn't have repository attached for DDR replication
Replication for clients not designated to backup to Data Domain are successful. ONLY clients designated to backup to Data Domain are failing replication. Replication is configured correctly on the Avamar source grid as confirmed by the configuration in /usr/local/avamar/etc/repl_cron.cfg.
The following message is seen in the /usr/local/avamar/var/cron/replicate.log
2012/04/01-12:50:29 avtar Error <10772>: The target server doesn't have repository attached for DDR replication.
The following messages are seen in the /usr/local/avamar/var/ddrmaintlogs/ddrmaint.log
Apr 19 10:02:51 <server> ddrmaint.bin: Info: ============================= get-ddrindex cmd start v6.0.1-66 =============================
Apr 19 10:02:51 <server> ddrmaint.bin: Info: get-ddrindex cmdline: '/usr/local/avamar/bin/ddrmaint.bin --flagfile=/usr/local/avamar/etc/usersettings.cfg --password=********** --vardir=/usr/local/avamar/var --server=<server> --id=root --bindir=/usr/local/avamar/bin --vardir=/usr/local/avamar/var --bindir=/usr/local/avamar/bin --sysdir=/usr/local/avamar/etc get-ddrindex --dpnid=1297284169 --server=<server> --id=repluser --password=********** --client=/REPLICATE/<SERVER>/clients'
Apr 19 10:02:51 <server> ddrmaint.bin: Info: get-ddrindex::get_ddrindex - Found 1 ddrconfig records in ddr_info.
Apr 19 10:02:51 <server> ddrmaint.bin: Error: get-ddrindex::get_ddrindex - Did not find a ddr index for client path "/REPLICATE/<SERVER>/clients" and there is no ddr marked as the default.
Both the source and target grids have a Data Domain system configured. This can be confirmed by running the following command:
> ddrmaint read-ddr-info
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<avamar dpnid="1297113326" version="2">
<ddrconfig client-map-default="true" ddos-version="126.96.36.199-282511" ddrcreatetime="1331065279" ddrid="62BC434C95E571A75F2DA6B04DB238C12405A991" hostname="<ddr_server>" index="1" max-streams="50" modelno="DD670" password="<password>" serialno="2FA0904066" username="boost">
<ports getter-setter="161" trap="163">
When logged into the Data Domain system replication statistics reveal no replication is running.
> ddboost file-replication show stats
Network bytes sent: 0
Pre-compressed bytes sent: 0
Bytes after filtering: 0
Bytes after low bandwidth optimization: 0
Bytes after local compression: 0
Compression ratio: 0
Please refer to KB Article 123567 to see resolution for this issue.
How to configure Avamar plugin based replication to allow clients to run longer than 24hrs (overtime)
Avamar plugin based replication jobs can be configured to run for a maximum of 24 hours.
On a correctly configured system 24 hours should be sufficient time for all replication jobs to complete. However, this may be a problem in circumstances where the time required may exceed this.
- First time replication is occurring and the target system is being 'seeded' with a large amount of source data
- Replication has failed to run for a period of time and needs to catch up
- A large amount of new data has been added to the replication source and the large amount of new data requires longer than normal to replicate to the target
Please refer to KB Article 192507 to see resolution for this issue.
How to restore the client data from a replication target from the /REPLICATE domain
How to restore client data from a replication target from /REPLICATE domain.
Please refer to KB Article 93490 to see resolution for this issue.