12 Replies Latest reply: Jan 18, 2017 2:44 PM by Rud RSS

Avamar-DD replication

avmaint

hi

If we configure replication from Avamar LSU on source DD to Avamar LSU on target DD (or other mtree) what additional steps we need to take during disaster of primary site?

  • 1. Re: Avamar-DD replication
    Ian Anderson

    It sounds like you may be trying to use Data Domain native replication here. Is that the case? If so, this will not work because the metadata stored on the Avamar side will not be replicated.

     

    Replication in an Avamar / Data Domain integrated configuration must be managed from the Avamar side. If you configure replication from Avamar, all the data required to perform a disaster recovery of the source systems will be replicated to the target automatically. During a disaster, the clients are normally failed over to the target system and back up to that system instead.


    Assuming the source system was lost, once it has been repaired or replaced, a replicate restore needs to be performed in order to recover the source system's configuration settings. Once this is done, the clients can be failed back. At that time, backups can also be restored from the target back to the source but this does has some potential pitfalls (Avamar replication can't replicate deletions, for example). In any case, I would recommend working with EMC to recover a system that has been completely lost.

  • 2. Re: Avamar-DD replication
    dynamox

    Ian Anderson wrote:

     

     

    Replication in an Avamar / Data Domain integrated configuration must be managed from the Avamar side.

     

    Ian,

     

    can you please explain how that would work ? If  I were setting up a brand new environent, 2 sites where each site has an Avamar Grid/Data Domain configuraiton, all backups are stored on DD, metadata on the grid.

     

    1) What/How/Where is replication setup ?

    2) Can we setup bi-directional replication ?

     

    Thank you

  • 3. Re: Avamar-DD replication
    avmaint

    OK fine, for a moment I have forgotten that DD is a passive device which holds data and needs a tool separately to administer and manage it.

    What is we replicate the metadata from Avamar to Avamar and just Data from DD to DD?

  • 4. Re: Avamar-DD replication
    Ian Anderson

    dynamox,

     

    Replication in Avamar / DD environments uses Data Domain "Managed File Replication". Replication is configured through the Avamar Administrator GUI in the usual way but in addition to the Avamar-to-Avamar replication of metadata, the source Avamar server also sends DDBoost commands to its attached Data Domain device(s) with instructions about what to replicate across to the target DD. The Avamar system is in charge but the Data Domain is doing most of the heavy lifting.

     

    The process to set up a net new bi-direction replication pair would be something like this (assuming there are two sites -- one in Toronto and one Calgary):

    1. Rack and install the Toronto Avamar, Toronto Data Domain, Calgary Avamar and Calgary Data Domain
    2. Log into the Avamar Administrator GUI on each system and perform the following steps:
      1. Attach the local Data Domain to the Avamar Server. This will create an Avamar MTree on the local Data Domain.
      2. Add the target Avamar server as a new replication destination.
      3. Configure one or more replication groups.

     

    The metadata (and any data on the GSAN back-end) will be replicated Avamar to Avamar and any data on the DD back-end will be replicated DD to DD. That's pretty much it.

     

    avmaint, I hope this answers your question as well.

  • 5. Re: Avamar-DD replication
    dynamox

    Thank you Ian.

     

    Now let's take two DR situation, how would you go about resuming backup operations

     

    1) Toronto datacenter that hosted Avamar/DD fell through the ground because of a sinkhole , servers were hosted in another facility so they survived but need to start backup up to Calgary now.

     

    2) Toronto datacenter that hosted Avamar/DD and servers fell through the ground because of a sinkhole. New hardware was delivered to Calgary and we need to restore all backups to these new servers and start backing them up again.

     

    In both scenarios we still need to be able to backup servers that were located at Calgary datacenter. I assume somewhere we'll need to resort to DNS aliases to handle different grid names ..etc

     

    Thank you and Happy Holidays Ian.

  • 6. Re: Avamar-DD replication
    Ian Anderson

    I'm not sure I like this sinkhole scenario.

    1) Toronto datacenter that hosted Avamar/DD fell through the ground because of a sinkhole , servers were hosted in another facility so they survived but need to start backup up to Calgary now.

    Assuming the grids are named catoav01 and cacaav01, the existing catoav01 clients and their data are stored under the /REPLICATE/CATOAV01 domain on cacaav01. The domain structure of the source is replicated across to this domain during replication. For example, if you have a client called /clients/catodc01, you will find it in /REPLICATE/CATOAV01/clients/catodc01 on the replication partner.

     

    During a disaster, servers that need to continue backing up would have to be reactivated to the Calgary grid and new backups would be written to the new accounts outside the /REPLICATE domain. To help keep thing straight, it's helpful to create a domain specifically for the DR event -- in our case, something like /catodr -- and activate the clients in the same domain structure they had on the source. Continuing with the example above, new backups for catodc01 would be written to /catodr/clients/catodc01. The usual limitations on backing up clients to a Data Domain system across a WAN would apply.

    2) Toronto datacenter that hosted Avamar/DD and servers fell through the ground because of a sinkhole. New hardware was delivered to Calgary and we need to restore all backups to these new servers and start backing them up again.

    This scenario would be handled more or less the same as above, except you'd have to set up the hardware and either do a clean install of the OS and Avamar client software, or use the System State Restore ISO. Once the clients were back up and running, you'd have to reactivate them to the Calgary system.

     

    In both cases, you would use redirected restore to restore older (replicated) data to the clients.

     

    I'm sure this doesn't sound like a walk in the park. The developers are aware of the pain involved in reactivating large numbers of clients in a DR scenario and are considering options to make this process easier in future. If you do find yourself in a DR scenario, I would recommend getting in touch with support as they have access to some tools to do things like automate copying of datasets that can help ease some of this pain.

     

    Also, I'm going to leave aside operational root-to-root replication in this discussion because a) it requires an RPQ, and b) one of the (numerous) limitations of root-to-root replication is that it can only be used in a one-to-one unidirectional replication configuration -- a root-to-root target must be a dedicated DR target with no clients of its own.

     

    My pleasure! Happy Holidays.

  • 7. Re: Avamar-DD replication
    dynamox

    Hi Ian,

     

    1) so what you are saying is that there is no way to "adapt" backups in case of DR, all clients will have to perform full backups (DDBoost obviously will help with WAN traffic) but full file system scan will un-avoidable ?

     

    2) Let's say the sinkhole is fixed, new hardware is purchased for Toronto. How do i take my "/REPLICATE/CATOAV01" and copy it back to Toronto and restart replication ?

  • 8. Re: Avamar-DD replication
    Ian Anderson

    1. Because the backups are referenced in the cache using their (immutable) root hash, the client will find the older backups still on the system (under a different account but this doesn't matter to the client) and the caches will still be used. No need for a full.

     

    2. This is a "replicate restore" and I'd recommend working with support. There are some non-obvious pitfalls.

     

    Once the new hardware is racked and the Avamar software installed, the replicator is run in restore mode on the Calgary system to push the GSAN user accounting structure and the most recent flushes of the system accounts (MCS, EMS, etc.) to the replacement Toronto system. Once that's done, the flushes are restored and another replicate restore is run to push the backup data to the replacement system. I'm leaving out a lot of detail here, so nobody go and try this at home.

  • 9. Re: Avamar-DD replication
    dynamox

    2) Ok so we push it back to Toronto, does it have to be restored into /REPLCIATE/CATOAV01 path or can it be restored back into /CATOAV01. At this point we can turn around and begin replication from Toronto to Calgary again (incremental only) ?

     

    3) What happens to all the backups that were done by Toronto clients that were temporally backing up to Calgary. How do i get those backups back to Toronto ?

     

    Thank you

  • 10. Re: Avamar-DD replication
    Ian Anderson

    2. A replicate restore is a reverse replicate -- everything that was replicated from / to /REPLICATE/CATOAV01 will be replicated back from /REPLICATE/CATOAV01 to /.

     

    3. There are different ways to handle this. Replicating everything from /catodr to /REPLICATE/CACAAV01/catodr would reduce the upfront complexity of the replication at a cost of potentially having to hunt for a backup based on what date it was created should you need to perform a restore.

     

    Another option would be to use account-to-account replication to replicate the backups from the client account in /catodr on the Calgary system to the corresponding account in /clients on the Toronto system. This is more involved up front but would put the backups all in one place, simplifying later restores.

     

    Either way, I'd recommend working with support.

  • 11. Re: Avamar-DD replication
    avmaint

    Do we have a way to replicate just few snapups by labelID?

  • 12. Re: Avamar-DD replication
    Rud

    Hi all,

     

    What about restore VM from replicate using emergency restore if vcenter on primary site is down ?

    On site-2, all We have in this case is the "Avamar-2 + DataDomain-2" replication target.

     

    I am looking for a solution to restore VCenter or few VM directly to an esx host.

     

    Kind regards,

    Ted