FAQ And Issues On Avamar Replication

FAQ And Issues On Avamar Replication

Environment:

Avamar 7.x

Avamar 6.x

Avamar 5.x

 

 

Description:

Here are few most frequently asked questions and most common issues related to Avamar Replication.

 

Avamar replication systems - source and target differ in capacity utilization

 

Issue –

There is an Avamar source and an Avamar target system. Capacity utilisation is higher on one system than the other.

 

Please refer to KB Article 115944 to see resolution for this issue.

 

Replication fails with Exit code 20: Insufficient privileges

 

Issue –

Replication fails with errors linked to permissions.

 

You can see the following in the logs:

 

2010/11/24-18:02:11 rununtil -82672 avmgr newd --acnt=/REPLICATE --duplicates-okay --hfsaddr=avamar_target.emc.com --id=repluser --ap=*****

2010/11/24-18:02:11 0  ERROR! Exit code 20: Insufficient privileges

2010/11/24-18:02:11 alarm 82672

2010/11/24-18:02:11 REPLICATE ERROR: Command returned 5120 (0x1400): rununtil -82672 avmgr newd --acnt=/REPLICATE --duplicates-okay --hfsaddr=avamar_target.emc.com --id=repluser --ap=*****

2010/11/24-18:02:12

2010/11/24-18:02:12 rununtil -82671 avmgr newd  --acnt=/REPLICATE/avamar_source.emc.com --duplicates-okay --hfsaddr=avamar_target.emc.com --id=repluser --ap=*****

2010/11/24-18:02:12 0  ERROR! Exit code 20: Insufficient privileges

2010/11/24-18:02:12 alarm 82671

2010/11/24-18:02:12 REPLICATE ERROR: Command returned 5120 (0x1400): rununtil -82671 avmgr newd --acnt=/REPLICATE/avamar_source.emc.com --duplicates-okay --hfsaddr=avamar_target.emc.com --id=repluser --ap=*****

2010/11/24-18:02:12

2010/11/24-18:02:12 Could not open recoverdb.sh

 

Please refer to KB Article 115681 to see resolution for this issue.

 

Calculating how much data Avamar can replicate on a daily basis

 

Issue –

Replication is not completing within the backup window. The 'how to' below describes the factors involved in understanding how much data you can expect the Avamar system to be able to replicate each day.

 

Please refer to KB Article 110852 to see resolution for this issue.

 

Avamar replication troubleshooting - questions to ask and data to gather

 

Issue –

The scope of this KB article is to describe which logs and information are needed in order to perform general troubleshooting of an Avamar replication job which is failing.

 

Please refer to KB Article 117124 to see resolution for this issue.

 

Unencrypted Backups & Replication Fail in Avamar 7.1 Due to Increased Firewall Restrictions

 

Issue –

The issue that is occurring is the avfirewall is filtering backups over port 27000.

 

Networker 8.x is not compatible with Data Deduplication with Avamar 7.1.

This ONLY applies to new Avamar 7.1 installation and does not apply to Avamar server upgraded to version 7.1.

 

Cron based replication to an Avamar 7.1 server fails with one or more of following error in /usr/local/avamar/var/cron/replicate.log (where target.local is the replication target server).

 

2014/10/14-17:36:00 ERROR: avmaint: hfscreatetime: cannot connect to server target.local at 192.168.10.10:27000

<snip...>

2014/10/14-17:36:01 avmgr logn --hfsaddr=target.local --id=repluser --ap=*****

2014/10/14-17:36:11 0  ERROR! Exit code 15: Cannot connect to Avamar dispatcher

2014/10/14-17:36:11 REPLICATE ERROR: Command returned 3840 (0xf00): avmgr logn --hfsaddr=target.local --id=repluser --ap=*****

2014/10/14-17:36:11 ABORT: Could not login to destination DPN.  Check flags or local config files (repl_cron.cfg)

2014/10/14-17:36:11 dpncron.pm::docmdout got exit code 256 from 'replicate --flagfile=/usr/local/avamar/etc/repl_cron.cfg', returning 0

2014/10/14-17:36:11 ======== failed replication (pid=28702) ========

2014/10/14-17:36:11 mccli event publish --code=4602 --message='repl_cron - failed replication (pid=28702)'

2014/10/14-17:36:19 0,23000,CLI command completed successfully.

2014/10/14-17:36:19

2014/10/14-17:36:19 avmaint infomessage  --avamaronly --errcode=4602 'repl_cron - failed replication (pid=28702)'

2014/10/14-17:36:20 ============================== Finished repl_cron ==============================

 

Unencrypted Avamar client backups (over port 27000) fail with the following messages in the client avtar logs.

 

avmaint Info <5562>: - - Connect: Trying <IP>:27000

avmaint Info <5694>: - Failed initial handshake, trying again

 

and/or

 

avtar Info <5552>: Connecting to Avamar Server (<AV Server>)

avtar Info <5554>: Connecting to one node in each datacenter

avtar Info <5694>: - Failed initial handshake, trying again

avtar Info <6063>: - Communication error: Could not create connection to Server

 

 

Telnet\curl from client to server over port 27000 also fails.

Netstat shows we are listening on port 27000 for the appropriate IP's.

 

netstat -anlp | grep 27000

(No info could be read for "-p": geteuid()=500 but you should be root.)

tcp 0      0 <IP>:27000     :::* LISTEN      -

 

On the target Avamar server the firewall is running:

 

Output from Gen3 Hardware:

rpm -qa | grep avfwb

avfwb-2.1.0-4

 

service avfirewall status

avfirewall is running...

 

Output from Gen4/Gen4s Hardware:

rpm -qa | grep avfw

avfwb-3.0.0-20

 

service avfirewall status

Checking for service avfirewall                                     running

 

Please refer to KB Article 193138 to see resolution for this issue.

 

Avamar Administrator does not allow /REPLICATE to be selected when running backend capacity report

 

Issue –

When configuring the backend report through the Avamar Administrator interface the /REPLICATE domain is not displayed along

with the other domains when attempting to choose which domains and clients should be included in the report.

 

If the backend report is called from the Avamar command line and the /REPLICATE domain is specified,

no meaningful data is produced.  See below for example output

 

nohup backendreport --include=/REPLICATE/sourceavamar.domain.com/domain/testclient.domain.com  --avtar=--clearcache --avtar=--hashcachemax=-10 --small-client-mb=0 --verbose > backendreport-repl.out &

[1] 10074

admin@avmtest1:~/nric/>: tail -f backendreport-repl.out

Changing working directory to /tmp/replicate

 

INFO: Inclusion patterns are: "/REPLICATE/sourceavamar.domain.com/domain/testclient.domain.com"

 

avmgr logn

1 Request succeeded

-1 privilege level (enabled,create,read,backup,access,move,delete,maint,manage,fullmanage,noticketrequired,

superuser,ignoreacls,readdir,mclogin,opt1,opt2)

2 block type  (directory)

 

avmgr cpdb --path=/ --mvpath=/REPLICATE/ --out=replnew.sh --avmgr-path=avmgr --failover

--list-snapups

Script "replnew.sh" for target path /REPLICATE copies 36 users in 115 accounts (81 clients)

Accounting changes: 0 differences

 

 

DONE Replicate to  successful, elapsed 0000:00:08   (Wed Jan 22 09:42:18 GMT 2014)

 

<backendreport version="6.1.2-46" reporttime="2014/01/22-09:42:18" bytessent="0" totaltime="0">

</backendreport>

completed backendreport (pid=10074)

============================== Finished backendreport ========================================

 

Please refer to KB Article 190414 to see resolution for this issue.

 

Multiple replication streams with different clients are failing with the following error message "Cache file for "client1" is locked, probably due to another replication -- Aborting"

 

Issue –

Multiple replication streams are configured.

Secondary replication stream is configured.

 

Both replication streams are configured with different clients.

For instance, the replication streams could be configured as follows:

 

Replication Stream 1

 

Parsed flags: --hfsaddr=avamargrid.com --dstaddr= avamargrid.com --dstid=root --dstpassword=***** --dpnname=sourcegrid.com --id=root --ap=***** --srcpath=//domain/client1 --dstpath=/REPLICATE/sourcegrid.com/domain/client1

 

Replication Stream 2

 

Parsed flags: --hfsaddr=avamargrid.com --dstaddr= avamargrid.com --dstid=root --dstpassword=***** --dpnname=sourcegrid.com --id=root --ap=***** --srcpath=/domain/client2 --dstpath=/REPLICATE/sourcegrid.com/domain/client2

 

Although replication streams have different clients configured, both streams attempt to replicate the same client.

 

This can be observed by examining the logs for both streams. For instance, the logs would show the following:

 

Replication Stream 1 log snippet:

 

INFO: Replicating "domain/client1" to "/REPLICATE/sourcegrid.com/domain/client1/" (approx. 0 MB: small client)

 

Replication Stream 2 lot snippet:

 

INFO: Replicating "domain/client1" to "/REPLICATE/sourcegrid.com/domain/client1(approx. 0 MB: small client)

 

In the above example, both streams are replicating client1 ; one of the streams will show the following error message:

 

avtar Error <5064>: Cannot open file "/usr/local/avamar/var/p_sourcegrid.com-client1.dat"

avtar Info <5065>: Creating new cache file /usr/local/avamar/var/sourcegrid.com-client1.dat (1,573,408 bytes)

avtar Error <5803>: Error writing 32-byte header to cache file /usr/local/avamar/var sourcegrid.com-client1.dat.  Possibly out of disk space

avtar Error <6044>: Cache file for "client1" is locked, probably due to another replication -- Aborting

 

One stream will fail with above error messages and the other stream will continue to run.

 

Please refer to KB Article 123886 to see resolution for this issue.

 

Replication Failing with Error <10772>: The target server doesn't have repository attached for DDR replication

 

Issue –

Replication for clients not designated to backup to Data Domain are successful. ONLY clients designated to backup to Data Domain are failing replication. Replication is configured correctly on the Avamar source grid as confirmed by the configuration in /usr/local/avamar/etc/repl_cron.cfg.

 

The following message is seen in the /usr/local/avamar/var/cron/replicate.log

 

2012/04/01-12:50:29 avtar Error <10772>: The target server doesn't have repository attached for DDR replication.

 

The following messages are seen in the /usr/local/avamar/var/ddrmaintlogs/ddrmaint.log

 

Apr 19 10:02:51 <server> ddrmaint.bin[1179]: Info: ============================= get-ddrindex cmd start v6.0.1-66 =============================

Apr 19 10:02:51 <server> ddrmaint.bin[1179]: Info: get-ddrindex cmdline: '/usr/local/avamar/bin/ddrmaint.bin --flagfile=/usr/local/avamar/etc/usersettings.cfg --password=********** --vardir=/usr/local/avamar/var --server=<server> --id=root --bindir=/usr/local/avamar/bin --vardir=/usr/local/avamar/var --bindir=/usr/local/avamar/bin --sysdir=/usr/local/avamar/etc get-ddrindex --dpnid=1297284169 --server=<server> --id=repluser --password=********** --client=/REPLICATE/<SERVER>/clients'

Apr 19 10:02:51 <server> ddrmaint.bin[1179]: Info: get-ddrindex::get_ddrindex - Found 1 ddrconfig records in ddr_info.

Apr 19 10:02:51 <server> ddrmaint.bin[1179]: Error: get-ddrindex::get_ddrindex - Did not find a ddr index for client path "/REPLICATE/<SERVER>/clients" and there is no ddr marked as the default.

 

Both the source and target grids have a Data Domain system configured.  This can be confirmed by running the following command:

 

> ddrmaint read-ddr-info

 

Example output:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<avamar dpnid="1297113326" version="2">

<datadomain count="1">

<ddrconfig client-map-default="true" ddos-version="5.1.0.9-282511" ddrcreatetime="1331065279" ddrid="62BC434C95E571A75F2DA6B04DB238C12405A991" hostname="<ddr_server>" index="1" max-streams="50" modelno="DD670" password="<password>" serialno="2FA0904066" username="boost">

<snmp community="avamar">

<ports getter-setter="161" trap="163">

</ports>

</snmp>

<client-map>

</client-map>

</ddrconfig>

</datadomain>

</avamar>

 

When logged into the Data Domain system replication statistics reveal no replication is running.

 

> ddboost file-replication show stats

 

Example output:

Direction:                                Outbound

Network bytes sent:                       0

Pre-compressed bytes sent:                0

Bytes after filtering:                    0

Bytes after low bandwidth optimization:   0

Bytes after local compression:            0

Compression ratio:                        0

 

Please refer to KB Article 123567 to see resolution for this issue.

 

How to configure Avamar plugin based replication to allow clients to run longer than 24hrs (overtime)

 

Issue –

Avamar plugin based replication jobs can be configured to run for a maximum of 24 hours.

 

On a correctly configured system 24 hours should be sufficient time for all replication jobs to complete.  However, this may be a problem in circumstances where the time required may exceed this.

 

For example;

  • First time replication is occurring and the target system is being 'seeded' with a large amount of source data
  • Replication has failed to run for a period of time and needs to catch up
  • A large amount of new data has been added to the replication source and the large amount of new data requires longer than normal to replicate to the target

 

  Please refer to KB Article 192507 to see resolution for this issue.


How to restore the client data from a replication target from the /REPLICATE domain

 

Issue –

How to restore client data from a replication target from /REPLICATE domain.

 

Please refer to KB Article 93490 to see resolution for this issue.