13 Replies Latest reply: Jun 10, 2016 3:17 AM by nr RSS

How to setup simultaneous multiple backups for Windows host ?

suhasbhide411

How to setup simultaneous multiple backups for Windows host ?

 

We have a host with 10 file systems each with around 150 GB backup data (total around 1.3 TB). The backup takes around 36-38 hours. As such, I was looking at splitting the backup to 3 simultaneous streams so that it can complete in 12 hours or so. Is there any way to do this ? I know there are other options (such as not to backup all drives on daily basis, but am currently not looking at that option). Reason for taking so long time for backup is that there are many changes happening on the server applications and the directory structure is too deep. Server team is looking at that but it will take long time. So, was looking at running 3 simultaneous streams on same host.

 

Thanks.

  • 1. Re: How to setup simultaneous multiple backups for Windows host ?
    Gautam G Prakash

    You can create multiple instances of the Client and run multiple backups. Only problem with that approach is that you cannot run 2 backups from the same client at a time. You can try to increase the no. of sessions in more options and see if that helps.

  • 2. Re: How to setup simultaneous multiple backups for Windows host ?
    dynamox

    separate windows drives/mount points ?   Yes, there is a way to setup and run backups on multiple drives at the same time. Talk to support and they will give you the procedure. I had a Windows 2k8 R2 cluster with 8 drives, each drive was 10TB so you can imagine running one backup would not cut it. Support gave me the procedure that allowed me to run 8 backup simultaneously. You do have to be careful with CPU/Memory/Disk I/O  utilization when you do that. Mine was running on pretty beefy Cisco UCS blade so it was fine.

  • 3. Re: How to setup simultaneous multiple backups for Windows host ?
    suhasbhide411

    Hi dynamox....

     

    I checked with EMC Support and was told that it's not possible on Windows systems. Hence, posted the query on community. Can you send me the steps to make this happen ?

  • 4. Re: How to setup simultaneous multiple backups for Windows host ?
    dynamox

    it's absolutely possible,  whoever has your ticket ask them to escalate it.  You will need to get the procedure from support, i can't share it here.

  • 5. Re: How to setup simultaneous multiple backups for Windows host ?
    sconstable_illumina

    You may want to check if you AV scanner or something is interfering, a 1.3T file system with a 100% change rate shouldn't take that long to backup.

  • 6. Re: How to setup simultaneous multiple backups for Windows host ?
    suhasbhide411

    No problem. I have already followed up extensively with EMC on this. So, I will wait for someone else to share the information.

  • 7. Re: How to setup simultaneous multiple backups for Windows host ?
    dynamox

    i am not sure you are talking to the right people or they understand what you are trying to do. Level 2 support had the procedure. The procedure basically involves setting up multiple avamar services running on the same Windows host, each service will be using a different port, appear as different avamar client and require special dataset for each drive letter you want to backup.

  • 8. Re: How to setup simultaneous multiple backups for Windows host ?
    dgian

    You will need work with EMC-support and at an escalation. The Two parameters to put into the Dataset, those values are hidden, purposely not documented, but only in internal documents, as it may cause other issues like over CPU and RAM utilization.

    Starting in 5.x (year 2009 or about); in your dataset for the file-backup, please , parallel=true; multicore=true – both have to be set. Now keeping in mind as the avtar will be processing multiple drives (# of streams based on the number of drives you have in the dataset), thus we engage support is that CPU utilization on the "client" doing the backup will go up higher during the backup- window. So you may want to experiment by putting in 3 independent drives in the dataset and seeing (example f:\ g:\, h:\)

    Also the other issue you may be having; as the version of Avamar Client has not been documented in your posting, also if Avamar is integrated to Data Domain, that the client "cache files", may be "off" which is based on the number of files and that "cache" files May need tuning as well. Need to look at the daily backup log, also back in the Dataset, at least enable "Report Advanced statistics" .

    So in general if the RAM on the client is not limited and the CPU is not, one should expect at least 100 GB/HR of backup and lets say 1 Million files per 1TB (1000GB)- even without parallel turned on ... so there is something else going on in your environment, may be client-cashes that may need tuning, so please review the "backup log" for your backup and  search for the word, "booted" (notes my MAC for 273 GB has ~1 Million files, yet only 327.5 MB was unique / changed data)

     

    Here is my Mac backup o May 7th, backup log

    2016-05-07 20:24:21 avtar Info <5156>: Backup #308 timestamp 2016-05-07 20:19:48, 1,034,180 files, 133,483 directories, 273.8 GB (3,053 files, 327.5 MB, 0.12% new)

    2016-05-07 20:24:21 avtar Info <7539>: Label "Mac-Laptop-Sch-630pm-130am-Laptop-Mac-Policy-1462660200004", scheduled to expire after 07/06/16 (2016-07-06 22:30:00 UTC), daily backup

    2016-05-07 20:24:21 avtar Info <6083>: Backed-up 273.8 GB in 109.80 minutes: 150 GB/hour (565,134 files/hour)

     

    Also look at:

     

    2016-05-07 20:24:21 avtar Info <5587>: Updating cache files in /usr/local/avamar/var

    2016-05-07 20:24:21 avtar Info <5069>: - Writing cache file "/usr/local/avamar/var/f_cache.dat"

    2016-05-07 20:24:22 avtar Info <5546>: Cache update complete /usr/local/avamar/var/f_cache.dat (352.0 MiB of 1024 MiB max)

    2016-05-07 20:24:22 avtar Stats <6151>: File cache: entries added/updated 1,032,839, booted 0

    2016-05-07 20:24:22 avtar Stats <18857>: Backup statistics for filename cache (monolithic) in '/usr/local/avamar/var/f_cache.dat'

    2016-05-07 20:24:22 avtar Stats <0000>:  -- elements: 1,032,839 entered, 1,032,839 looked up, 0 booted (not recorded)

    2016-05-07 20:24:22 avtar Stats <18859>:  -- elements (lookup results): 3,051 missed

    2016-05-07 20:24:22 avtar Stats <18860>:  -- elements (hash table probes): minimum 1, maximum 21, mean 1.211 sdev 0.662

    2016-05-07 20:24:22 avtar Info <5069>: - Writing cache file "/usr/local/avamar/var/p_cache.dat"

    2016-05-07 20:24:24 avtar Info <5546>: Cache update complete /usr/local/avamar/var/p_cache.dat (384.0 MiB of 512 MiB max)

    2016-05-07 20:24:24 avtar Stats <6152>: Hash cache: entries added/updated 184,376, booted 0

     

    - Feel free to review the Avamar Operational Best Practices Document that is on Avamar System http://Youravamarsystem, ignore the browser certificate warning and continue, then "Documentation" link, then Search or scroll for "Avamar Operational Best Practices" guide to review the "cache" guidelines and or as I started to get EMC support engaged.

  • 9. Re: How to setup simultaneous multiple backups for Windows host ?
    Ian Anderson

    Slow backup performance is most often caused by some factor other than lack of parallelism. Implementing and managing parallelism has a number of caveats and challenges, especially for methods like running multiple avtars in parallel. The reason people are insisting you work with support is that support can help you analyze your environment and determine if there is some other factor that needs to be addressed before opening the parallelism can of worms. There are also several different ways to implement parallelism and if parallelism is actually needed, they can point you toward the appropriate type to use.

     

    Shooting in the dark is not going to help you. Please work with support on this. You will likely need to work with L2 support.

     

    FYI, setting --parallel for Data Domain integrated backups is known to cause issues and is not supported.

  • 10. Re: How to setup simultaneous multiple backups for Windows host ?
    suhasbhide411

    None of the reply has been helpful as yet. I have tried my options with EMC support but they have refused stating this is not possible. Does anyone want to provide me the answer ?

  • 11. Re: How to setup simultaneous multiple backups for Windows host ?
    Ian Anderson

    Ask the SR owner to transfer your case to an L2.

  • 12. Re: How to setup simultaneous multiple backups for Windows host ?
    nr

    Hi Suhas - I see that you've opened an SR with support.  A colleague of mine has taken ownership of it and I'll be working with him to get a better understanding of the situation.

     

    In order to facilitate this you will be asked to provide a full set of information as described in the following article

    • KB 303720 - Information required by Support when investigating Avamar client backup performance issues

     

    Please provide this information as soon as you can and in its entirety so we have the best opportunity to be effective.

     

    Note that there are many many possible causes of backups not completing within the available backup window and we'll be using the following article as a general troubleshooting template. I recommend reviewing it to understand more about these types of challenges and the things we will look for

    • KB 335029 - How to understand Avamar Client backup performance and identify performance bottlenecks

     

    In some cases a client might be backing up with a high rate of performance, but the amount of data to be handled means that there simply isn't enough time within the available window.  If that's the case, there may be a need for some careful optimisation.

     

    Finally, I should add that if this client has never previously backed up successfully within the backup window, then we will need to engage the Professional Services team to work directly on it with you as a client implementation.  In that case, Support will collaborate with the Professional Services delivery person.

     

    Thanks,

    Nick

  • 13. Re: How to setup simultaneous multiple backups for Windows host ?
    nr

    Just to wrap this one up...

     

    The main reason the client was experiencing slow backup performance was due to file cache misses. 

    2016-05-15 07:24:31 avtar Warning <5650>: CAPACITY WARNING: The file cache is full; filecachemax=511MB, should be at least 2817MB

     

    Due to the large size of the dataset and the client having a 32 bit architecture it wasn't possible to size the monolithic cache large enough to accommodate entries for all files that were being backed up.


    The Operational Best Practices Guide documents a suggestion for this type of scenario "using cacheprefix"

     

    Also refer to

    • KB 317443 - How to correctly set the size of the Avamar client caches to avoid cache overflows and backup performance problems