For good or ill, Avamar overloaded the term "Domain". An Avamar domain is not related to an ActiveDirectory domain or a DNS domain so it's unlikely that the issue is related to domain membership.
It is very possible to back up remote systems with the Avamar solution -- in fact, this is one of Avamar's strengths.
Make sure the clients in Domain B can reach the Avamar server on ports 27000, 29000 and 28001 and that they're registered and activated to the Avamar Server. Make sure the avagent process is running on the clients. It should be listening on port 28002.
Could you post a snip from the client logs showing the authentication error?
Yes, I can see the client fine and have those ports. What happens is when I the job starts it fails due to some kind of authentication issue. I have used the same DataSet on clients within the same domain and they backup fine. Here is a copy of the log from one of my test clients on the untrusted domain.'
2012-11-07 15:14:32 avtar Info <6555>: Initializing connection (Avamar Deduplication Engine v2.0.0)
2012-11-07 15:14:32 avtar Info <5552>: Connecting to Avamar Server (##.##.##.##)
2012-11-07 15:14:32 avtar Info <5554>: Connecting to one node in each datacenter
2012-11-07 15:14:54 avtar Info <5694>: - Failed initial handshake, trying again
2012-11-07 15:15:15 avtar Info <5694>: - Failed initial handshake, trying again
2012-11-07 15:15:36 avtar Info <5694>: - Failed initial handshake, trying again
2012-11-07 15:15:57 avtar Info <5694>: - Failed initial handshake, trying again
2012-11-07 15:16:18 avtar Info <5694>: - Failed initial handshake, trying again
2012-11-07 15:16:39 avtar Info <5694>: - Failed initial handshake, trying again
2012-11-07 15:16:39 avtar Info <6063>: - Communication error: Could not create connection to Server
2012-11-07 15:16:39 avtar Info <5557>: No connections available
server address and login credentials.
address and login credentials.
2012-11-07 15:16:39 avtar Info <6149>: Error summary: 2 errors: 8604, 8941
2012-11-07 15:16:39 avtar Info <8468>: Sending wrapup message to parent
2012-11-07 15:16:39 avtar Info <5314>: Command failed (2 errors, exit code 10008: cannot establish connection with server (possible network or DNS failure))
From what I can tell it looks like the client process (avtar) can't authenticate with the Storage Node? I will double check with my firewall team to make sure that ports 27000 and 29000 are open from this network to the network the Avamar server is on.
The "Failed initial handshake" message generally means that the client can't reach the server on port 27000 (or port 29000 for SSL connections). The client is picking up the workorder so some traffic is getting through from the client to the Avamar server, meaning routing is probably not an issue.
As a quick test, you can try to telnet from the client to the server on port 27000. If the connection times out, port 27000 may be blocked at the firewall.
It does look like ports 27000 and 29000 and 28001 is open. 280002 looks to be closed.
Querying target system called:
Attempting to resolve IP address to a name...
Failed to resolve IP address to name
TCP port 27000 (unknown service): LISTENING
TCP port 29000 (unknown service): LISTENING
TCP port 28002 (unknown service): FILTERED
TCP port 28001 (unknown service): LISTENING
portqry.exe -n ##.##.##.## -e 27000,29000,28002,28001 -p TCP exits with return code 0x00000000.
If the server is a multi-node grid, keep in mind that ports 27000 and 29000 on each data node must be reachable from the client. If you're not able to make any progress on the issue with that information, I would recommend opening a service request. To speed up troublshooting, I would recommend that you edit the dataset to enable debugging information, run a new backup, and attach the resulting log to the service request.