Why do CommVault customers store their backups on Isilon?  Because Isilon is scale-out, efficient, simple to setup, and easy to maintain.  Yes there are other reasons (https://www.emc.com/collateral/handout/h12842-ho-top-reasons-simpana-and-isilon.pdf) but the scale-out architecture and operational simplicity are most important.  A single scale-out NAS filesystem that dynamically grows with online capacity additions is the most flexible type of on-premises storage media for CommVault. 


What exactly does this mean?  Why is scale-out better than traditional scale-up storage?  Storage architectures that are scale-up will always have some type of size limit that prevents significant growth of a single filesystem.  Every legacy scale-up storage array will have its own unique capacity limit that will prevent this type of filesystem growth on NAS or SAN, whether it is a filesystem limitation or a RAID level construct.  Yes there are tricks you can use to get around this for each vendor but they are still compromises.  Scale-up storage will only grow to its limit and will not scale-out a single filesystem like Isilon. 


Why is it so important to have a single scale-out filesystem anyway?  What is wrong with using multiple NAS or SAN filesystems?  Both add overhead (capacity and operational) which is unnecessary.  Isilon allows you to setup a single SMB share for CommVault Windows MediaAgents that can dynamically grow as you add Isilon nodes.  A single SMB share for the entire backup environment that never runs out of space as long as you manage the Isilon capacity per best practices.  Anything else is a compromise, you will have to manage more paths, add more silos of storage, induce more usable capacity overhead, and create a more complicated architecture.


But what about Linux MediaAgents?  Scale-out is best for all CommVault disk libraries, even with Linux MediaAgents.  Isilon brings the same value to both Windows and Linux with an ever expanding scale-out disk library for backup storage.  Linux MediaAgents are configured differently than Windows and have different architectural concerns but are still simple to setup and require less sysadmin time to run in production than SAN or scale-up NAS. 

 

The goal of this post is to walk those unfamiliar with managing CommVault through he process of setting up Isilon as a disk library for CommVault Windows and Linux MediaAgents.  Walking through the setup process will demonstrate the simplicity of this architecture and the value Isilon scale-out NAS brings to CommVault.

 

Quick Review

 

I posted a previous blog (https://community.emc.com/blogs/keith/2017/05/01/isilon-as-a-commvault-backup-target--planning-and-sizing) covering the basics of the CommVault architecture and how they relate to Isilon. Take a look to get an in-depth overview since I won't repeat all the details in this post. 


To quickly recap, a CommVault deployment will look like the diagram below, the Isilon will act as a "library".  Specifically a disk library that can be shared among multiple MediaAgents not just a single MediaAgent like the diagram. 

 

CommVault architecture overview

commcell_logical.png

Lab Components

 

I've setup CommVault along with Isilon in a lab environment with the components below running virtualized on a single ESXi host. 

 

CommVault Simpana v11

CommServe - Win2K8R2 ("CS")

Windows MediaAgent - Win2K8R2 ("MA")

Linux MediaAgent - CentOS 7 ("Linux MA")

Client - Win2K8R2

Isilon - OneFS 8.0.0.4

DNS server (AD domain controller)  - Win2KR2

 

VM lab components - vSphere client

Screen Shot 2017-08-02 at 8.09.17 AM.png

 

Windows MediaAgent setup

General OneFS setup

 

First assume a standard Isilon cluster setup.  Nothing special has to be done on OneFS for CommVault but there are some key standard components that need to be functional for optimal performance.  Specifically SmartConnect with DNS which load balances end client SMB connections across multiple Isilon nodes.  Before anything else, get SmartConnect and DNS working correctly even before configuring Isilon as a disk library (https://youtu.be/zVwZDpyPedA).  All CommVault infrastructure servers (CommServe and all MediaAgents) should get rotating "round-robin" IP addresses returned from the Isilon with consecutive 'nslookup' commands.  Think of this as the number one Isilon best practice you can implement for CommVault.

 

Why is SmartConnect so important?  CommVault Windows MediaAgents will open multiple SMB connections to the entire Isilon cluster across multiple Isilon nodes when thing are working correctly.  This will spread out the backup load and take advantage of all Isilon cluster resources.  Without SmartConnect, the cluster can get unbalanced with performance hot spots - certain Isilon nodes can get overloaded with too many SMB connections and overall backup performance will suffer.  It's easy to get SmartConnect working, just do it! 

 

While we are on the topic of SmartConnect, leave the default "round-robin" connection policy in place.  This policy can always get tuned later when CommVault is up and running.  It's best to leave the default SmartConnect connection policy and confirm a baseline level of performance with CommVault backups.  Get a baseline with the defaults then tune the SmartConnect connection policy later if necessary.

 

Create a CommVault OneFS SMB share

 

This is where Isilon makes a huge difference to a storage sysadmin.  All we need is a single SMB share created on the Isilon for all CommVault Windows MediaAgents.  Simple right?  All backups from all CommVault Windows MediaAgents will pass through this single share.  Other storage solutions will have many steps of creating/masking/mapping many LUNs or exporting many storage array filesystems and shares which take significantly longer to setup, troubleshoot, and maintain.  Not with Isilon, a single SMB share is used for the disk library by all Windows MediaAgents in the CommCell. 

 

Just login to the Isilon webUI and create a new SMB share in the desired access zone (use the system zone by default if you like).  There is nothing special about this share, it's a standard SMB share with a name, path, and share permissions (persona).  You can even use the Isilon "root" account to authenticate if you like.  I created a local "commvault" account (with a local Isilon password) to use as the share persona.  Put an optional SmartQuota directory quota on the share root directory if you like (not shown below). 

 

Don't create multiple SMB shares for multiple Windows MediaAgents, it's not necessary!  The value in this architecture is using an single SMB share for a single Isilon disk library shared by all Windows MediaAgents. 

 

Create new SMB share - OneFS WebUI

Screen+Shot+2017-07-24+at+2.17.35+PM.png

 

Add Persona (account) to new SMB share - OneFS webUI

Screen Shot 2017-07-24 at 2.19.24 PM.png

 

CommVault Setup of an Isilon disk library

 

At this point the Isilon sysadmin is complete.  The CommVault sysadmins now take over and configure CommVault to use the single SMB share as a disk library.  This is also very easy, just create a new disk library, use the share persona account used when creating the share (my example is a local 'commvault' account), share account password, and Windows UNC path to the share (something like \\isilon.fully.qualified.domain.name.com\commvault).


Don't map Windows explorer drives to the share and don't use drive letters to the SMB share.  When a connection to the disk library is needed, CommVault will mount the UNC path dynamically with this configuration and load balancing across Isilon nodes will happen with OneFS SmartConnect.  You also need to select a single MediaAgent ("MA" in screenshot) to initially use the disk library which can be shared with other Windows MediaAgents later.

 

See the screenshots below that walk through an example. My Isilon disk library is given a name, assigned to a Windows MediaAgent ("MA"), authenticated with an local Isilon "commvault" account/password, and mapped to a UNC path (\\isilon.keith.com\commvault).  That's it, the setup of the disk library is done!


Again, this is why Isilon is great for CommVault, all of the sysadmin work (both Isilon and CommVault) on the disk library is done at this point and is future proof.  This single share can grow the library indefinitely without any additional configuration aside from Isilon node additions.  Keep adding Isilon nodes and the SMB share will continue to grow over the life of the disk library.  Nothing will fill up as long as the Isilon cluster capacity is managed per best practices.  No future migrations, no LUNs, no SAN networks, no management of multiple filesystem paths, just a simple SMB share with a single SMB mount from CommVault.

 

Isilon disk library setup

 

Add disk library - CommVault webUI

Screen Shot 2017-07-19 at 10.51.30 AM.png

Add disk library configuration - CommVault webUI

Screen Shot 2017-07-19 at 10.52.34 AM.png

Completed Isilon disk library - CommVault webUI

Screen Shot 2017-07-19 at 10.53.33 AM.png

 

Isilon disk library properties

 

Let's look at the Isilon disk library properties from the CommVault webUI.  We don't want to change anything at this point, keep the defaults until you have a reason to tune options.  This section is not intended as documentation of all the options, for that see the CommVault documentation (http://documentation.commvault.com/commvault/v11/article?p=features/disk_library/advanced.htm).  I just want to highlight a few options as they relate to Isilon to help understand how the disk library works.

 

General - This is the main screen for the disk library where you can take the library offline/online using the "enable library" checkbox for maintenance.  I can also can mark archive files as read-only which is integrated between CommVault v11 and OneFS SmartLock.  This is a very cool feature see the CommVault documentation link above for more details. 

 

Mount Paths - Here I find the settings mostly used for storage systems that need use multiple mount paths.  This is not necessary for Windows MediaAgents using Isilon.

 

Associations - This lists the storage policies associated with the disk library along with the storage policy copy name.  All storage policies have a "primary" copy for backups and optionally can have a "secondary" copy, usually for offsite or long term retention.

 

Security - CommVault webUI accounts with permissions to the library GUI tree component of the webUI, screenshot omitted

 

Disk Usage - Gives a nice summary of the disk library space from a CommVault perspective.

 

Space Management - Tab that allows a CommVault sysadmin to set thresholds for warning events when the disk library is running low on space.  Can also automatically spawn a data aging job when space is tight to remove old backups immediately on hitting the threshold since they are normally scheduled to run daily.

 

Isilon disk library "general" properties - CommVault webUI

Screen Shot 2017-07-19 at 10.55.35 AM.png

 

Isilon disk library "mount path" properties - CommVault webUI

Screen Shot 2017-07-19 at 10.55.46 AM.png

 

Isilon disk library "associations" properties - CommVault webUI

Screen Shot 2017-07-26 at 8.37.44 AM.png

 

Isilon disk library "disk usage" properties - CommVault webUI

Screen Shot 2017-07-19 at 10.56.12 AM.png

 

Isilon disk library "space management" properties - CommVault webUI

Screen Shot 2017-07-19 at 10.56.20 AM.png

 

Mount Path Properties

 

The mount path properties are different than the library properties, let's look at these also. Again, see the CommVault docs for full documentation (link above) since I am only highlighting of what is interesting from an Isilon perspective.

 

General - We find some useful space information on the mount path along with another option to enable/disable the disk library mount path for maintenance.

 

Allocation Policy - Here CommVault sets some reserved space on the mount path to avoid running a path/filesystem completely out of space (default of 2GB reserved space).  Isilon mount paths will grow automatically as capacity is added to the Isilon cluster or as quotas are increased on the SMB share, keep the 2GB default. 

 

Deduplication DBs - Screenshot omitted, summarizes the MediaAgent deduplication databases for the mount path, more on this later.

 

Sharing - Allows sharing of mount path/library between multiple MediaAgents.  If you want to add a second or third MediaAgent to this Isilon disk library mount path, simply click "share" and add the other Windows MediaAgents using the same UNC path and credentials.  MediaAgents will each need their own storage policy for load balancing but can all share the same Isilon mount path. 

 

Isilon mount path "general" properties - CommVault webUI

Screen Shot 2017-07-19 at 10.56.47 AM.png

 

Isilon mount path "allocation policy" properties - CommVault webUI

Screen Shot 2017-07-19 at 10.56.53 AM.png

 

Isilon mount path "sharing" properties - CommVault webUI

Screen Shot 2017-07-19 at 10.57.08 AM.png

 

Creation of a Storage Policy

 

We now have an Isilon disk library owned by a Windows MediaAgent and have seen all the properties for our various configuration options.  How do we now start using the library?


We have to create a CommVault "storage policy" to send client backups to the Isilon through the Windows MediaAgent(s).  This is simply a shared policy that clients use to send a library backups through a specific MediaAgent with a few attributes like the backup retention time.  See the wizard screenshots below for an example, this is a very basic configuration required for any storage library.  The "data aging" (or pruning) process is a scheduled job that runs and enforces each storage policy retention time by deleting expired backups.

 

Wizard walkthrough:

 

Data protection and archiving - We will create a storage policy for backups ("data protection and archiving"), the other option is a DR backup for the CommServe database

 

Storage policy name - self explanatory, leave the other options unchecked

 

Select library - select the Isilon disk library we created earlier

 

MediaAgent - select the MediaAgent that will push data from the client to the Isilon for this storage policy.

 

Here is an important point to clear up any confusion.  Multiple MediaAgents can all share the same Isilon disk library.  However, we would typically assign a single MediaAgent to a single storage policy.  Then assign clients to various storage policies to load balance across MediaAgents.  So use multiple storage policies each with a unique MediaAgent (all using the Isilon library) and split your client backups between these storage policies.

 

Streams and retention criteria - each stream will spawn a new SMB connection to the Isilon library, leave the default but it can be tuned later.  The retention policy is how long this storage policy keeps backups, measured in days and cycles.  A cycle is a full backup plus incrementals, a second cycle starts when a second full is run.  Again, the data aging (pruning) job enforces the retention settings on a storage policy by deleting old backups.

 

Another important point to clear up confusion.  Performance bottlenecks are a fact of life.  Once you have a significant number of clients running backups (say 100, 200, 500 etc) there will a bottleneck somewhere in the architecture that slow things down.  This could be the MediaAgent (CPU, RAM, DDB speed), the network, or the storage library.  Bottlenecks can induce degraded performance and are either fixed by adding more resources (more MediaAgents, more network, more storage resources) or tuning the system to ease the stress on the bottleneck.  This tuning can be as simple as reducing concurrency (simultaneous client backups) by throttling the number of writers and backup streams.

 

Let's take an example.  Lets say when running nightly backups my MediaAgent CPU is pegged @ 100% CPU and the host gets unresponsive.  I'd much rather run the system at a maximum of 80% CPU which gives me few choices for remediation.  I can invest in another MediaAgent, create another storage policy, and balance clients across multiple MediaAgents and reduce the CPU load on the first MediaAgent.  But what if I don't have another MediaAgent available?  In that case, I can artificially limit (reduce) the writers to the library and/or the streams (reduce) on the storage policy to ease the concurrency load during backups.  Less writers and less streams means less simultaneous backup activity which could help improve my MediaAgent performance. 

 

Let's take another example.  When running my nightly backups my MediaAgents are running fine but my storage system is pegged @ 100% CPU.  Or the disk I/O subsystem is 100% busy, the disk wait time is very high, and the disk queues are more than full.  This causes the client SMB latency to spike dramatically and I'd rather run the storage with less load and better response time.  I can invest in more storage resources and increase performance (add an Isilon node).  But what if I have more than enough capacity and don't want to invest in more storage?  Again, I would decrease the concurrency by tuning (reducing) the max writers on the library or tuning (reducing) the streams on each storage policy.

 

Contact CommVault for help with this type of tuning.

 

Advanced (software encryption) - self explanitory

 

Deduplication DB - each MediaAgent will use a separate deduplication database (DDB) for each storage policy and store them locally on the MediaAgent.  This database is best hosted on block flash storage (either shared or local), avoid putting this on an Isilon NAS share.   

 

Review - review selections and create the storage policy

 

New storage policy - CommVault webUI

Screen Shot 2017-07-19 at 10.58.06 AM.png

 

New storage policy wizard - CommVault webUI

Screen Shot 2017-07-19 at 10.58.17 AM.png

 

New storage policy wizard - CommVault webUI

Screen Shot 2017-07-19 at 10.58.39 AM.png\


New storage policy wizard - CommVault webUI

Screen Shot 2017-07-19 at 10.58.49 AM.png

 

New storage policy wizard - CommVault webUI

Screen Shot 2017-07-19 at 10.58.56 AM.png

 

New storage policy wizard - CommVault webUI

Screen Shot 2017-07-19 at 10.59.21 AM.png

 

New storage policy wizard - CommVault webUI

Screen Shot 2017-07-19 at 10.59.30 AM.png

 

New storage policy wizard - CommVault webUI

Screen Shot 2017-07-19 at 10.59.37 AM.png

 

New storage policy wizard - CommVault webUI

Screen Shot 2017-07-19 at 10.59.48 AM.png

 

New storage policy wizard - CommVault webUI

Screen Shot 2017-07-19 at 11.00.30 AM.png

 

New storage policy wizard - CommVault webUI

Screen Shot 2017-07-19 at 11.00.41 AM.png

 

New storage policy wizard - CommVault webUI

Screen Shot 2017-07-19 at 11.00.47 AM.png

 

Newly created storage policy - CommVault webUI

Screen Shot 2017-07-19 at 11.01.06 AM.png

 

Assign Storage Policy to a client and run a backup

 

The disk library is configured and a new storage policy has been created to use our Windows MediaAgent with our Isilon disk library.  All that has to be done now is to assign a backup client to this storage policy and run a backup.  I've omitted screenshots of this process since it is a standard operation for all CommVault sysadmins. 

 

Linux MediaAgent setup


Isilon works just as well for Linux MediaAgents as a CommVault disk library as it does for Windows MediaAgents.  Both benefit from a scale-out architecture and result in a disk library that automatically grows as Isilon capacity is added.  However, there are some differences between how Windows mounts SMB shares and how Linux mounts NFS shares that impact the sysadmin steps required to create an Isilon disk library for Linux MediaAgents. 

 

Linux MediaAgent Differences

 

What is the difference between using a CommVault Windows MediaAgent versus a Linux MediaAgent with an Isilon disk library?  Both take advantage of an Isilon scale-out architecture which allows rapid data growth with minimal administration effort.  However, Linux MediaAgents differ in that they do not dynamically mount a UNC path when they need to write to a disk library like a Windows MediaAgent.  Linux hosts need to mount shared storage with NFS when creating an Isilon disk library on Linux MediaAgents.  This raises some questions and concerns.  How will this work with SmartConnect client load balancing across the Isilon cluster?  How can a single NFS mount take advantage of scale-out storage? Can Isilon bring any benefit to Linux MediaAgents?


Multiple mount paths


We don't want an entire disk library mounted with a single NFS mount on a single Linux host since it will become a bottleneck.  Why?  That configuration would only use a single IP address to NFS mount a single Isilon node and wouldn't balance the workload across all nodes in the Isilon cluster.  The Isilon architecture prefers concurrency of many connections across all Isilon nodes in a cluster, not a single NFS connection to a single node.  So we need our Linux MediaAgent to connect to more than a single Isilon node IP address when writing large amounts of data during a CommVault backup.


Is a single NFS connection to a single Isilon node a bad thing?  It will function fine and protect the data when written to OneFS.  The Linux host will connect to an NFS mount over a single IP address to a single Isilon node.  Data written to the NFS mount will be erasure coded and spread across all Isilon nodes in the OneFS nodepool through the back-end cluster network.  However, the process of writing data to the cluster its coordinated by the Isilon node that accepts the initial NFS connection.  This node performs the erasure coding and sends the data to other nodes using the coordinator node hardware resources (CPU/RAM/network/infiniband).  If this coordinator Isilon node becomes overwhelmed with write requests it can become a bottleneck since it will be very busy while the other nodes in the cluster may not be as busy.  More work can be done if a client connects to multiple Isilon nodes for writing rather than a single Isilon node, especially for a large amount of backup data.


The best way to load balance across the Isilon cluster is to configure the Linux MediaAgent to use multiple NFS mounts to multiple Isilon nodes.  SmartConnect will balance the NFS mount requests across the Isilon cluster by returning the client a rotating list of IP addresses that reside in the OneFS IP address pool.  Repeated mount commands will cycle through IP addresses on the Isilon via SmartConnect and multiple Isilon nodes will service client mount read/write requests once the mounts are made. The Linux host will use these fixed IP addresses returned from SmartConnect through the life of each NFS mount until the host is rebooted or the mount unmounted and remounted.

 

Mount path usage and balancing


Can a Commvault MediaAgent take advantage of multiple NFS mount points for backups?  Absolutely!  Remember CommVault MediaAgents also support block SAN storage which usually is allocated with multiple LUNs each formatted with a host filesystem.  The software needs a method for the MediaAgent to spread backups across several separate local filesystems which could be local DAS, SAN, or in the case of Linux, hard NFS mounts. CommVault takes care of this local mount path load balancing in the MediaAgent software and can utilize multiple mount paths efficiently. 


This load balancing feature is a property of the disk library.  Multiple paths can be balanced by using a "fill and spill mount paths..." or a "spill and fill mount paths..." algorithm to use multiple block or NAS filesystems on a MediaAgent.  "Fill and spill..." simply means fill the first mount path (capacity) and then start using the next mount path until it is full.  "Spill and fill..." is the opposite where each mount is used in more of a round-robin fashion and all mounts are filled just about equally.  See the screenshots later in the configuration walkthrough that demonstrates the mount path option.  We are going to select "Spill and fill..." for Isilon because we want to use the multiple rotating mount paths for I/O load balancing.  The goal is to spread the I/O across multiple Isilon nodes which means creating multiple mount paths that are balanced across the Isilon cluster by SmartConnect. 

 

Performance tuning of clients to use multiple mount paths


Disk libraries with multiple mount paths and a round-robin type load balancing configuration may need some performance tuning of the backup clients.  Why?  A backup single stream will only use a single MediaAgent mount point.  More specifically, each backup client will have "readers" (read threads) that will write to MediaAgent mount paths with "writers" (write threads).  A large backup with a single reader won't use our multi-mount path round-robin Linux MediaAgent to its full potential.  So we may need to tune clients with large amounts of backup data to take advantage of multiple paths.  We do this by adjusting the number of client readers and possibly also allowing multiple readers to access a single client volume/mount.

 

Before you start tuning, think about the concurrency in your architecture.  Will this Linux MediaAgent funnel backup data to the disk library from a single backup client?  Several backup clients?  Many backup clients?  Client tuning is probably not necessary for many clients, they will naturally balance across the Linux MediaAgent mount paths using the "spill and fill" algorithm.  However, single clients or only a few client may need tuning since they may not use all MediaAgent mount paths equally. 

 

Let's look at an example.  Say you have four (4) Isilon NFS mount paths on a single Linux MediaAgent.  A single (Windows) client backup using this MediaAgent will default to two (2) readers and will not use multiple readers within a single drive letter or volume mount.  So at most we only use two (2) mounts paths out of four (4) total on the Linux MediaAgent.  Performance tuning is necessary in this case, we want to use all mount paths for load balancing.

 

Take another example.  Say you have the same four (4) Isilion NFS mount paths on the Linux MediaAgent but instead you have twenty (20) clients to backup.  Clients will default to two (2) readers each but since we have more clients we will naturally load balance across the mount paths without any performance tuning.  More clients and more concurrency will most likely not require any performance tuning since the multiple mount paths will be more fully utilized. 

 

Disk library capacity reporting and OneFS quotas


Isilon NFS exports without quotas can seem confusing.  Say I have 1PB of total capacity in my Isilon cluster.  I create an NFS export with no quota and mount it from a Linux host.  The Linux host will see the entire 1PB of capacity from the NFS client mount when running a 'df' command.  Now create several more NFS exports on the same Isilon and mount them from the same Linux host.  Linux will see 1PB of total capacity for each of the NFS mounts made from the Isilon.  The used/available/total capacity will adjust with usage but each NFS mount will report the entire capacity of the Isilon cluster.  This is why we use the OneFS SmartQuota feature, to control the export size as reported by the end client. 

 

A CommVault disk library will view the total capacity of the library as the sum of its mount paths. In our example above, we have a 1PB Isilon without quotas.  Each client NFS mount will report the entire 1PB filesystem capacity.  What if the Linux host mounted four (4) NFS mounts for the CommVault disk library?  Without quotas, Linux would report 4PB across the four (4) paths and CommVault would report that the disk library was 4PB in total capacity.  Each separate NFS mount will magnify the total library capacity without quotas.  This is not necessarily a problem, backups will work fine.  But this can get confusing if the storage team is managing Isilon capacity using OneFS and the CommVault team is managing capacity from the CommVault webUI.

 

What if you don't want each NFS mount to assume it owns the entire Isilon cluster capacity?  What if you don't want the mount path magnification of the total size and inflate the total size by the number of mount paths?  Then use quotas.  Quotas on Isilon are very granular and can be set on any directory in the OneFS filesystem.  A quota can be placed on an NFS export OneFS directory or on any subdirectory within that NFS export.  Want a disk library to start at with 400TB of total capacity?  Put a 100TB hard quota on the NFS export parent directory and each mount path will be 100TB in size, four (4) mount paths will create a 400TB CommVault disk library.  Or put a 100TB quota on each mount path nested under the NFS export parent directory. 


Hard directory quotas are required to force the Linux NFS mount to report on the quota size.  When utilization of the NFS exports starts to get too high, simply increase the quotas when more capacity is required.  The CommVault Linux MediaAgent queries the mount path every 30 minutes and updates the capacity utilization.  So increase the quota and wait 30 minutes and your disk library will grow in size.  There is very minimal effort required to grow a library, no need for additional NFS export, no additional LUNs, no Linux or CommVault sysadmin work necessary. 

 

Does Isilon still benefit Linux MediaAgents?


Does Isilon still benefit Linux MediaAgents?  Does scale-out work better than scale-up for this use case?  The initial configuration of a Linux MediaAgent with SAN or scale-up NAS will be similar to Isilon.  Allocate a few LUNs or NFS exports to a Linux host and mount for use with CommVault.  The difference comes when we need to grow the library.  Isilon requires very little effort.  The NFS exports to  the Linux hosts will always have capacity as long as the Isilon capacity is managed per best practices.  Isilon NFS exports will dynamically grow with the Isilon cluster and require little to no sysadmin work.  Add Isilon nodes to the Isilon cluster and the shares get larger.  If quotas are used, increase a quota to dynamically grow an NFS export and the disk library automatically.  Capacity allocation is performed only once during the initial configuration which is a unique benefit, SAN or scale-up NAS will always require more sysadmin work to grow the environment when capacity is running low. 


Scale-out also adds performance with capacity.  SAN or scale-up NAS will scale capacity up to a certain point when the array controller becomes a bottleneck and can no longer service additional workloads.  As a CommVault environment grows, it needs more performance not less.  A disk library that grows from 1PB to 2PB will need more performance with the additional capacity.  Since Isilon is scale-out we automatically get this performance bump with each capacity addition.  This is a much simpler solution than SAN or scale-up NAS for a customer and avoids having multiple storage arrays for a single disk library or performance troubleshooting of certain filesystems, RAID groups, aggregates, etc.

 

General OneFS setup


Linux MediaAgents rely on a few key features in OneFs.  Primarily SmartConnect which we discussed earlier for Windows MediaAgents.  Get SmartConnect working before you start any CommVault configuration.  Make sure your Linux Media Agent receives unique "round-robin" IP addresses from the Isilon with consecutive 'nslookup' commands.

 

Windows MediaAgents should use a OneFS "static" IP address pool on Isilon.  Linux MediaAgents should use a OneFS "dynamic" IP address pool.  Why?  SMB2 is a stateful protocol and works best with fixed IP addresses that remain associated with a specific ethernet interface on an Isilon node.  NFSv3 is a stateless protocol and can benefit from "floating" IP addresses that move between interfaces when nodes get added or removed from the Isilon cluster (or rebooted).  So if you are sharing the Isilon cluster between both Windows and Linux MediaAgents, create a static IP address pool for Windows MediaAgents with a unique SmartConnect zone name.  Create a separate dynamic IP address pool for Linux MediaAgents with a separate unique SmartConnect zone name.  Both pools can be on the same subnet, but it is best to separate them and use static IPs for SMB2 (Windows) and dynamic IPs for NFSv3 (Linux).

 

The default "round-robin" SmartConnect client connection balancing policy is most likely best for most CommVault deployments.   Linux MediaAgents will mount their NFS shares infrequently and probably won't query SmartConnect very often.  The policy can always be adjusted if necessary but it's best to keep the default "round-robin" and baseline performance prior to making any changes.

 

Create a CommVault OneFS NFS export

 

OneFS has flexibly for hosts mounting NFS exports.  Hosts can directly mount OneFS NFS exports or can mount subdirectories within the parent NFS export.  Linux MediaAgents will benefit from load balancing I/O across multiple Isilon mount paths.  So we have a choice, we can create an NFS export for each CommVault mount path or create a single parent NFS export for the Linux host which individually mounts export subdirectories for each mount path.  Either will work but we can save some time by just creating a single export per Linux host.  Then just create subdirectories within this parent export for the Linux host to mount as CommVault mount paths.

 

How many Isilon mount paths should be used by a Linux MediaAgent?  It depends!  There are no simple answers here, the number of Isilon nodes and the horsepower of the Linux hosts will determine how many mount paths are optimal.  A Linux host may be very busy with only a few mounts paths and not need more I/O paths to the Isilon.  On the other hand, the number of mount paths may be the bottleneck and the host may be able to push more data with more mount paths.  There is no broad statement to make here, talk to CommVault to get official sizing for the application.  CommVault can estimate the amount of data the MediaAgent is capable of pushing to a disk library which can then be spread across a few mount paths.  There is a feature in CommVault to "validate" a mount path which tests I/O.  Get a baseline for each mount path and size accordingly for the total number of mount paths required.


Don't bypass SmartConnect by mounting the Isilon cluster by IP address to attempt to manually load balance I/O.  Dynamic IP addresses will shuffle across Isilon node interfaces over time due to node additions, removals, and reboots.  The IP address residing on a particular Isilon node will over time move to another node.  Use SmartConnect DNS name when mounting NFS shares to allow the cluster to load balance connections.

 

In my example below I have a single Linux MediaAgent.  I have create a parent directory for this MediaAgent (/ifs/linuxma) and I have created four (4) subdirectories under this parent directory for my Linux host to mount which will give me four (4) mount paths to present to CommVault.  The parent directory is exported with the default options.  The Isilon sysadmin work is done, we configure the Linux host next.

 

Isilon CLI


# pwd

/ifs/linuxma

eightdotfour-1# ls

mountfour mountone mountthree mounttwo

 

# isi nfs exports view 2

                     ID: 2

                   Zone: System

                  Paths: /ifs/linuxma

            Description: -

                Clients: -

           Root Clients: -

      Read Only Clients: -

     Read Write Clients: -

               All Dirs: Yes

             Block Size: 8.0k

           Can Set Time: Yes

       Case Insensitive: No

        Case Preserving: Yes

       Chown Restricted: No

    Commit Asynchronous: No

Directory Transfer Size: 128.0k

               Encoding: DEFAULT

<snip>

 

CommVault Setup of an Isilon disk library


Configuring a Linux MediaAgent is a two step process.  We first mount the NFS shares from the Linux CLI and then finish by configuring the disk library in the CommVault webUI.

 

Mounting Isilon NFS export mount paths on Linux

 

I'll use /mnt to mount the Isilon exports and I've created a subdirectory under /mnt for each of the four (4) Isilon subdirectories under my single NFS export. The /mnt directories "commvaultM1", "commvaultM2", "commvaultM3", "commvaultM4" will mount the "mountone", "mounttwo", "mountthree", and "mount four" directories under the /ifs/linuxma/ export.  The OneFS /ifs/linuxma/mountone directory will get mounted as /mnt/commvaultM1, /ifs/linuxma/mounttwo will get mounted as /mnt/commvaultM2, etc.

 

Each mount command will cycle through the Isilon SmartConnect IP pool assuming SmartConnect is working correctly.  A single IP address for all mounts will not load balance I/O, multiple mounts each need a unique IP address returned from SmartConnect.  My lab has an IP address pool range of 192.168.0.51-53 and each of my mount commands will use a unique IP address.  Dynamic IP addresses will get shuffled across Isilon nodes over time, the distribution doesn't have to be perfectly balanced across the Isilon cluster but should be balanced enough to avoid overwhelming a single Isilon node with write I/O. 


What mount options should I use for CommVault NFS?  I don't use any below, I'd rather get a baseline for performance and possibly adjust the mount options later.  Again, CommVault provides a function to "validate" a mount path, test I/O with various mount options if you like to see if any provide performance improvements.  Start with the basics and make tuning adjustments later.

 

Linux MediaAgent CLI

 

# ls -lah /mnt

total 4.0K

drwxr-xr-x.  6 root root   78 Jul 31 14:45 .

dr-xr-xr-x. 17 root root 4.0K Jul 31 11:53 ..

drwxr-xr-x   2 root root    6 Jul 31 12:04 commvaultM1

drwxr-xr-x   2 root root    6 Jul 31 14:45 commvaultM2

drwxr-xr-x   2 root root    6 Jul 31 14:45 commvaultM3

drwxr-xr-x   2 root root    6 Jul 31 14:45 commvaultM4

 

# mount isilon.keith.com:/ifs/linuxma/mountone /mnt/commvaultM1

# mount isilon.keith.com:/ifs/linuxma/mounttwo /mnt/commvaultM2

# mount isilon.keith.com:/ifs/linuxma/mountthree /mnt/commvaultM3

# mount isilon.keith.com:/ifs/linuxma/mountfour /mnt/commvaultM4

 

# mount

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)

<snip>

isilon.keith.com:/ifs/linuxma/mountone on /mnt/commvaultM1 type nfs (rw,relatime,vers=3,rsize=131072,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0.53,mountvers=3,mountport=300,mountproto=udp,local_lock=none,addr=192.168.0.53)

isilon.keith.com:/ifs/linuxma/mounttwo on /mnt/commvaultM2 type nfs (rw,relatime,vers=3,rsize=131072,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0.51,mountvers=3,mountport=300,mountproto=udp,local_lock=none,addr=192.168.0.51)

isilon.keith.com:/ifs/linuxma/mountthree on /mnt/commvaultM3 type nfs (rw,relatime,vers=3,rsize=131072,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0.53,mountvers=3,mountport=300,mountproto=udp,local_lock=none,addr=192.168.0.53)

isilon.keith.com:/ifs/linuxma/mountfour on /mnt/commvaultM4 type nfs (rw,relatime,vers=3,rsize=131072,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0.52,mountvers=3,mountport=300,mountproto=udp,local_lock=none,addr=192.168.0.52)

 

Add Isilon disk library

 

Use the CommVault webUI to complete the configuration of the Linux MediaAgent disk library.  I've included a screenshot of the Linux MediaAgent properties to start with, I have a CentOS 7 host installed with the CommVault MediaAgent software.

 

Following the screenshots below, we add a disk library, give it a name, associate with an existing Linux MediaAgent, and add the first Isilon NFS mount path.  A single mount path is added for the initial creation, additional mount paths are added immediately after with the "add mount path" GUI dialog.  The last screenshot is a summary of all four (4) Isilon NFS mount paths added to this disk library associated with the Linux MediaAgent.

 

LinuxMediaAgent properties - CommVault webUI

Screen Shot 2017-08-01 at 10.19.51 AM.png

 

Add disk library - CommVault webUI

Screen Shot 2017-08-01 at 10.20.03 AM.png

 

Add disk library - name and select MediaAgent - CommVault webUI

Screen+Shot+2017-08-01+at+10.20.28+AM.png

 

Add disk library - browse for local path - CommVault webUI

Screen Shot 2017-08-01 at 10.20.40 AM.png

 

Add disk library - select first mount path - CommVault webUI

Screen Shot 2017-08-01 at 10.20.53 AM.png

 

Add disk library complete - CommVault webUI

Screen Shot 2017-08-01 at 10.21.04 AM.png

 

Add additional mount path (repeat for all available mount paths)  - CommVault webUI

Screen Shot 2017-08-01 at 10.23.26 AM.png

 

Final view of all mount paths - CommVault webUI

Screen Shot 2017-08-01 at 10.24.30 AM.png

 

Change library properties

 

Our disk library has all mount paths added and it at full capacity.  We now need to adjust our load balancing algorithm to balance I/O across all mount paths equally.  This setting can be views from the "mount paths" tab of the library properties.  Change the "mount path usage" from "fill and spill..." to "spill and fill...".  "Spill and fill..." will cycle I/O across the mount paths equally rather than filling up a single mount path before moving to the next.

 

After making this change, the disk library is ready to use.  I have not included screenshots but simply create a storage policy like we did earlier using this Linux MediaAgent and disk library.  Then assign the storage policy to backup client to start using the library and MediaAgent.

 

Isilon library properties - CommVault webUI

Screen Shot 2017-08-01 at 10.27.15 AM.png

 

Mount path properties for Isilon library - "Fill and spill" - CommVault webUI

Screen Shot 2017-08-01 at 10.27.25 AM.png

 

Mount path properties for Isilon library - change to "Spill and fill" - CommVault webUI

Screen Shot 2017-08-01 at 10.27.37 AM.png

 

Performance tuning - readers and writers

 

As discussed earlier, you may want to tune client performance if you only have a single backup client or a small number of backup clients.  Adjust performance when you don't have a large degree of concurrency (many backup clients) or aren't fully utilizing the available Linux mount paths.  The default is two (2) readers (threads) per Windows client and also by default will not allow multiple readers per drive or mount point.  So if you want a client to run more than 2 threads or run multiple threads per filesystem see the steps below. 

 

In my example, I have four (4) mount paths configured on my Linux MediAgent disk library.  I'm configuring a Windows client to backup to the Linux MediaAgent and Isilon disk library.  The default settings on the Windows backup client will not take advantage of the four (4) paths (default to two (2) readers and does not allow multiple readers per mount).  So we are going to adjust the number of readers to four (4) and we will allow multiple readers per drive/mount.

 

My client is going to backup up a single Windows directory with a single large file and some small files.  The default settings would only run a single stream since my content resides on a single drive and would only use a single MediaAgent mount path.  By tuning performance I can drive my backup data from a single Windows client across my four (4) Isilon mount paths using the new tuning parameters.

 

The screenshots below also show a running and completed backup of the Windows client after making the performance tuning adjustments.  The running job shows four (4) streams running to each Isilon mount path and the job history shows four (4) streams.  I have a single large ~600MB file on my Windows client which is sent to a single mount path.  The other mount paths are used for the small files within the client backup content.  A single large file is not going to be split across multiple mount paths in this configuration, a single file is sent to a single path.


There is a feature in CommVault to "view contents" of each mount path which will be unique in our configuration.  Each stream will send unique data to each mount path which can be validated by viewing each mount path's contents (no screenshots). 

 

Subclient properties - advanced button - CommVault webUI

Screen Shot 2017-08-01 at 10.30.51 AM.png

 

Performance tab - advanced sub client properties - CommVault webUI

Screen Shot 2017-08-02 at 2.48.39 PM.png

 

Performance tab - change advanced sub client properties - CommVault webUI

Screen Shot 2017-08-01 at 10.31.01 AM.png

 

Running backup of client with performance changes - streams - CommVault webUI

Screen+Shot+2017-08-01+at+10.32.38+AM.png

 

Client backup completed job details - CommVaultwebUI

Screen Shot 2017-08-01 at 10.34.22 AM.png

 

Disk library capacity and quotas


Isilon NFS shares without quotas can be deceiving to CommVault as we mentioned earlier.  Every NFS mount will "see" the entire Isilon cluster capacity if no quotas are used.  Mount several NFS mounts from an Isilon cluster and each will seem to own entire cluster capacity.  Combine NFS mounts to create a a disk library and it will appear to be the size of the entire Isilon cluster magnified by the number of mount paths.

 

Does this really matter?  CommVault will use a mount path until the free space on the mount path reaches the reserved space set on that path.  By default, CommVault will reserve 2GB per mount path and will stop writing to that mount path when it hits this 2GB free reserve.  Keep the Isilon total capacity healthy per best practices and the NFS mounts will never run out of space, they will continue to grow as Isilon nodes are added for additional capacity.   The disk library when viewed from the CommVault software will appear to be larger than the physical Isilon but there will be no loss of functionality, backups will run just fine.

 

But what if I don't want the disk library to appear larger than the available Isilon capacity?  What if I want to put a limit on how much data the disk library can write to the Isilon?  Both are reasonable concerns and are why the SmartQuotas license is recommended for CommVault customers using Isilon.  Use quotas to limit the capacity of the NFS export either at a parent directory or subdirectory level for Linux mount paths.  Hard directory quota limits are required to force Linux to see the NFS mount as a fixed size dictated by the quota.  CommVault will stop writing to the mount paths when the used capacity reaches the hard quota limit (minus the 2GB reserve).

 

What if I fill all mount paths and need to increase the capacity of the disk library?  Hard quotas will run out of space eventually right?  Yes, just increase the quota and the share will automatically grow on both the Linux host and on the CommVault disk library mount paths.  CommVault checks for disk space updates every 30 minutes by default.  So increase the quota and just give it some time, the available capacity will increase automatically after the quotas are increased. 

 

Linux mount paths with no quotas

 

In my example below, I have an Isilon cluster that is ~56GB in total capacity and has ~20GB of free space.  I use four (4) NFS mounts from my Linux client when creating the disk library so I have four (4) mounts each seeing the entire capacity and availability of the Isilon cluster by default with no quotas.  CommVault will report 4 x 56GB total capacity for the disk library (224GB) total and 4 x 20GB free space (80GB) total.  CommVault will report this total as capacity consumed and capacity free for all the library mount paths.

 

This will work fine but is confusing to the Linux and CommVault sysadmins.  The capacity free and consumed will update from the Isilon every 30 minutes so any Isilon cluster capacity increases will automatically grow the shares.  And the consumed/free will be accurate for each mount path, the disk library "disk usage" tab will just be inaccurate since it will total all mount paths associated with the library and magnify the total actual capacity. 

 

Isilon disk library properties - no quotas - CommVault webUI

Screen Shot 2017-08-02 at 4.27.30 PM.png

 

Isilon disk library mount paths - no quotas - CommVault webUI

Screen Shot 2017-08-02 at 4.27.44 PM.png

 

Adding OneFS quotas to Linux mount paths

 

I set the quota below on the Linux MediaAgent parent NFS export directory (/ifs/linuxma) to 10GB.  This can also be done on each of the individual mount paths but it will save time to set a single quota at the parent directory.  Each of the NFS mounts will appear to the Linux host as 10GB total per mount instead of the 56GB previously above with no quota.  See the CLI output for my quota on the Isilon cluster and then see the Linux 'df' output for the NFS mounts. 

 

After waiting for approximately 30 minutes, the CommVault webUI updates the library properties and the mount path disk space usage.  Notice how the CommVault webUI now sees the total library space as 40GB (4 x 10GB mounts) and also updated the mount paths to show 10GB per mount path.  Ignore the 64% consumed above and the 0% consumed below, I had to delete a few things between screenshots .

 

Quota on LinuxMA root directory - OneFS CLI

 

# isi quota quota view /ifs/linuxma directory

                       Path: /ifs/linuxma

                       Type: directory

                  Snapshots: No

Thresholds Include Overhead: No

                      Usage

                          Files: 21

                  With Overhead: 134.50k

                   W/O Overhead: 32.66k

                       Over: -

                   Enforced: Yes

                  Container: Yes

                     Linked: -

                 Thresholds

                 Hard Threshold: 10.00G

                  Hard Exceeded: No

             Hard Last Exceeded: 1969-12-31T19:00:00

                       Advisory: -

              Advisory Exceeded: No

         Advisory Last Exceeded: -

                 Soft Threshold: -

                  Soft Exceeded: No

             Soft Last Exceeded: -

                     Soft Grace: -

 

Linux MA CLI

 

# df -h /mnt/*

Filesystem                                Size  Used Avail Use% Mounted on

isilon.keith.com:/ifs/linuxma/mountone     10G     0   10G   0% /mnt/commvaultM1

isilon.keith.com:/ifs/linuxma/mounttwo     10G     0   10G   0% /mnt/commvaultM2

isilon.keith.com:/ifs/linuxma/mountthree   10G     0   10G   0% /mnt/commvaultM3

isilon.keith.com:/ifs/linuxma/mountfour    10G     0   10G   0% /mnt/commvaultM4

 

Isilon disk library properties - with quotas - CommVault webUI

Screen Shot 2017-08-02 at 4.23.57 PM.png

 

Isilon disk library mount paths - with quotas - CommVault webUI

Screen+Shot+2017-08-02+at+3.07.12+PM.png

Summary

 

Isilon for Commvault is easy.  Create a single SMB share for all your Windows MediaAgents to mount via UNC path and you are done.  Create an NFS export for each of your Linux MediaAgents and NFS mount a few mounts paths for load balancing.  Grow the Isilon cluster and both your Windows and Linux MediaAgent disk libraries grow automatically.  Your sysadmins will have much less busy work when compared to a SAN or scale-up NAS alternative. 


Thanks for reading, comments welcome!