Isilon and ECS for unstructured data storage
CommVault data protection software generates massive amounts of unstructured data. What makes this CommVault backup data unstructured data? Because the backup data is not stored in a structured CommVault database but as flat files in a filesystem or objects in an object store. While CommVault does use databases to run its CommServe and MediaAgents (http://bit.ly/2oQX3al), the backup data is only indexed in a database. The actual backup data chunks are stored outside the CommVault databases which makes it unstructured data.
Dell EMC is the industry leader when it comes to unstructured data storage. For the second year in a row (http://bit.ly/2z6J3dP) Dell EMC has been recognized as the leader in Gartner's magic quadrant for distributed file systems and object storage. Isilon is the distributed filesystem platform (NAS) and ECS (Elastic Cloud Storage) is the object storage platform. Both combined offer an industry leading approach to storing unstructured data.
In previous blog posts I have covered in detail how to use Isilon as a CommVault backup target (http://bit.ly/2vqtlJl). Isilon scale-out NAS is a great choice as a CommVault backup target due to its ease of management and storage efficiency. But when would I compliment an Isilon backup target with an ECS object store? When would I use ECS in general for CommVault? How do I configure ECS object storage with CommVault? I'll cover all these questions and run through some configuration examples in this blog post.
Gartner 2017 Distribute File Systems and Object Storage Magic Quadrant
What is ECS?
Let's talk about ECS and object storage. ECS is an object storage platform that lets you host cloud storage within your datacenter. Cloud storage (to me) is a storage platform that allows read/write access via industry standard-object APIs. Typically the S3 protocol is used but can also be the Swift, Atmos, or CAS protocols (http://bit.ly/2jHVcCU). All are web-based protocols that use HTTP(S) and a REST-based API to read/write data to the object storage platform. ECS allows you to provide your business object storage without sending your users and developers to the public cloud.
ECS looks like a storage appliance. It has a node-based architecture, 10GbE connectivity, JBOD attached storage, erasure coding data protection, and the familiar black with blue stripe front bezel you recognize from other storage Dell EMC appliances (http://bit.ly/2hNYhjR). What makes ECS different is how data is read and written to the appliance. This is not a block storage array with LUNs nor is this an ethernet NAS storage appliance (although we have some flexibility here). ECS requires an application to interact with the data using an API rather than a filesystem or LUN. To use ECS you need an application (like CommVault) that supports REST-based API access to object storage.
Object storage latency
NAS or SAN-based storage is designed to work on a local network and its workloads typically have a requirement for low latency. A highly transactional database might be running on all-flash SAN storage due to a latency requirement of 1 millisecond (or less). Or you might have scale-out NAS storage (like Isilon) that provides applications access to unstructured data with response time of less than 10 milliseconds. Both workloads are much different than an object storage use case because SAN and NAS-based storage is intended to stay within the same datacenter as the application, often within a single SAN network or ethernet dedicated LAN network.
Object storage has a different use case and performance characteristic than standard NAS or SAN-based storage. Object storage is intended for web traffic using HTTP as the vehicle for transport. Which means your application isn't expecting extremely low response times, often you will see web HTTP traffic with response times of 100 milliseconds. You don't expect a workload with low latency requirements to use object storage. But you will see plenty of new applications developed to take advantage of object storage for data that does not require low latency.
Object storage replication
Traditional SAN/NAS storage was built for hosting a workload in a single datacenter and replicate this data offsite for protection. Which means traditional storage is replicated in an active/passive manner, you have an active site with "live" data and a passive site with a read-only copy. You can fail over traditional storage between sites and reverse the roles but you don't typically have the same data active and available in both sites. Yes, you can clone read-only DR data with snapshots to create a writable copy but the architecture remains active/passive.
The ECS architecture is much more flexible than a traditional active/passive architecture since the ECS platform was built for more modern applications. With ECS object storage you can have geo-replication across multiple sites which means an active/active architecture for two sites, or even an active/active/active architecture for three sites. With the aid of a load balancer (ie, http://bit.ly/2zUbe2B), users and applications can connect to any available site and read/write any object data. Applications that need REST-based API storage using HTTP (and the associated latency) can access ECS content on with a global URL no matter where they are located. Which makes ECS great as a global object content repository since the architecture is not limited by the location of the user or application. Applications can take advantage of an active/active object storage architecture which does not necessarily need to be colocated in the same datacenter as the application.
ECS and CommVault
CommVault is a data protection software platform I have written about previously with a focus on how it can interact with Isilon file-based protocols like SMB and NFS (http://bit.ly/2vqtlJl). So why would I bother with ECS and object storage for CommVault, why not just use a NAS platform like Isilon?
Generally, CommVault has a need to store some backups short-term while retaining other backups long-term. This is referred to as the backup retention cycle and can vary depending on the type of data being protected. Say you had 500 virtual machines that needed daily backups run and retained for a month. Most data restore requests usually happen within a month of the backup so this short-term retention works well operationally. However, say you also had a legal requirement to store one full backup a month of each VM for 7 years. This means you need another strategy for keeping certain data for longer amounts of time which is very infrequently accessed.
Tape backups used to be the standard for long-term storage. Tapes were cheap and could be easily used for long-term retention of backups and archival purposes. But these days, object storage is a much more attractive alternative as a medium for long-term storage. Think of a global object content repository accessible from any datacenter in your company. Instead of having local tape libraries in each of your datacenters with physical tapes to managed, a geo-federated set of ECS appliances could provide this type of long-term backup storage at a very low cost and with less physical management than a tape library.
What about ECS for short-term retention? What about using ECS for all your CommVault backup activity? Not really a good fit, think about the latency. Your short-term daily backups need fast storage to complete their backup cycle quickly and meet your SLAs. 10 millisecond or less response time will typically keep your backup application (like CommVault) happy and performing well. Using HTTP(S) and REST-based APIs like S3 are not intended to provide this type of low latency since they are intended to service web-based traffic. Even if your ECS appliance is in the same datacenter as your CommVault infrastructure, the latency profile will be different and will not perform the same as a NAS-based appliance using NAS protocols. Stick with NAS scale-out storage (like Isilon) for short-term daily backups and consider ECS for long-term backup/archival storage due to the higher latency values.
What about using the ECS NFS gateway for short-term backups? Or the CIFS-ECS application for Windows access to ECS? While these tools provide the convenience of NAS protocols for Linux and Windows hosts, they are not intended for workloads that require NAS (LAN-based) latency. Think of the ECS NAS gateways as a convenient way to upload/download and share data but not as substitute for NAS storage when implementing with performance based workloads. Backup applications will do best with NAS-based storage for the daily backup cycle because they need the ~10ms or less response time when running active backups. ECS can provide plenty of bandwidth is but if you don't have decent response time (low latency) your legacy applications written for SAN/NAS storage will suffer. Use ECS for long-term backup storage and use a platform like Isilon for short-term backup storage.
Lastly, why do I need a separate object storage platform like ECS for long-term backup storage? Why not just retain long-term backups on the same storage where I keep my short-term backups? Why use both Isilon for short-term and ECS for long-term backup storage?
You can certainly use a single scale-out storage platform like Isilon for all backup storage. But what if your retention requirements are forcing you to store petabytes of old data? What if you want to start taking advantage of object storage and offload some of this content to a geo-distributed object archive rather than continue to grow your Isilon? What if you want to keep your Isilon lean with high performance (low latency) and move all old data out to an object store? All of these would be great reasons to explore object storage to compliment Isilon storage with a platform for short-term low latency storage and a second for long-term object storage.
Configuring a CommVault library to use ECS
It's very easy to use object storage with CommVault. CommVault supports various cloud storage "libraries" and lists EMC ECS as a supported S3 vendor (http://bit.ly/2pC1KFt). This means you can add ECS as a "cloud storage library" and start moving long-term backups to this library just as you would with any other tape or disk library.
We'll make the assumption that you already have ECS up and running in your environment and have also created an ECS namespace and bucket for CommVault to use. A namespace is simply a means to provide multi-tenancy within the ECS object store. Multiple tenants typically use multiple namespaces while a single organization with no multi-tenancy requirements can use a single namespace for their ECS object store. A bucket is an object storage container within a namespace that offers further granularity for user/group access, quotas, and retention. See the ECS admin guide (http://bit.ly/2znYvWY) for more details. For the purposes of this blog, we'll assume you have a namespace and bucket already created for use by CommVault backups, something like https://firstname.lastname@example.org:9021 for a "commvault" bucket on an ECS appliance named ecs.keith.com using port 9021.
Create a cloud storage library for ECS by opening the CommVault GUI and adding a new library of type "Cloud Storage Library" shown below in the diagram. Give your ECS library a friendly name and set the "type" in the drop down list to "Amazon S3". Also assign an existing MediaAgent host to control this library. Next complete the access information for CommVault to communicate with ECS. This includes:
- Authentication set to "Access & Secret Keys"
- Service host set to your HTTPS URL for ECS using port 9021
- Access Key ID set during creation of your CommVault bucket
- Secret Access Key set during creation of your CommVault bucket
- Bucket name on ECS for CommVault
- Storage class set to "standard"
See the next few screenshots for example configuration. Once all the fields are completed we click "OK" and create the library.
CommVault GUI - Add Cloud Storage Library
CommVault GUI - Cloud Storage Properties
Lets take a look at a few of the ECS library properties just for fun. We see below that the ECS library is of cloud vendor type "Amazon S3" which means it will use S3 APIs to read/write to the object storage. We also see that the library is a single mount path by default associated with a single MediaAgent and using the maximum allowed writers. Keep the defaults, these are fine for our use case of long-term backup storage.
CommVault GUI - ECS library general properties
CommVault GUI - ECS library mount paths
Configuring a CommVault storage policy to use an ECS library
Our intention is to use ECS as a long-term backup archive that will complement whatever storage we are using for daily short-term backup storage. We use a NAS-based platform like Isilon for the daily backups and retain those backups for a standard time, say 15 days in the screenshots below. Then we configure CommVault to send off backups for long-term retention to ECS, say 120 days in the screenshots below. All this is very easy to configure using CommVault storage policy copies and the auxillary copy job to move the data.
For a recap on CommVault storage policies and CommVault's aux copy process see my previous blog post (http://bit.ly/2oQX3al). A storage policy is CommVault's way of combining a storage library, MediaAgent, and retention settings all into a policy to assign to clients for a shared configuration. A storage policy will have a primary copy where the backups run during the daily backup cycle and would typically use faster storage like Isilon for NAS-based access (could be SAN as well). A storage policy can also have secondary copies with different backup retention settings for longer term backup storage. CommVault's aux copy process moves backups between the primary and secondary copies when the aux copy job is run.
Simply put, we have a primary storage policy copy on Isilon and we will create a secondary storage policy copy on ECS using the ECS cloud library we configured earlier. I already have a primary copy on Isilon in the screenshot below retaining backups for 15 days and 2 backup cycles (time between full backups).
CommVault GUI - storage policy primary copy general settings
CommVault GUI - storage policy primary copy retention settings
We want to create a secondary copy for long-term backups since we want to retain some backups for longer than the 15 day / 2 cycle retention on the primary copy. All we need to do is to create a new storage policy copy and assign the ECS library along with the MediaAgent we used to control the ECS (Windows "MA" host below). We then define a longer retention policy than the primary copy, I've used 120 days and 4 cycles as an example below. I also keep the copy policy to set to "all backups" and enable deduplication (more on this below).
CommVault GUI - Create new copy
CommVault GUI - new copy general properties
CommVault GUI - new copy retention settings
CommVault GUI - new copy policy
I'm using deduplication so I need to configure the MediaAgent deduplication database (DDB) for this storage policy copy. This DDB stores the hashes used for deduplication blocks and should be placed on fast flash storage local to the MediaAgent or on an all-flash block based array. I'll keep the defaults and point the DDB to local storage on my MediaAgent. See the screenshots below for a walkthrough of the DDB configuration.
CommVault has a great feature called "DASH copy" which is a deduplication-aware secondary copy. Say you had your backups nicely deduped on your primary copy, wouldn't you also want to preserve those capacity savings on your secondary copies? If so, use the DASH copy which is configured along with the DDB during the creation of the storage policy copy on ECS. Make sure the "Enable DASH Copy" is selected on your secondary copy and use either a disk read optimized copy or a network read optimized copy. Ask your CommVault team which is right for you, I have the default disk read copy, see the screenshots below.
CommVault GUI - new copy deduplication settings
CommVault GUI - DDB access path
CommVault GUI - completed DDB configuration with local paths
CommVault GUI - copy deduplication completed settings
CommVault GUI - enable DASH for storage policy copy
Using CommVault Auxillary copy to move long-term backups to ECS
The configuration work is done, now we just use the CommVault auxillary copy job to move backups from our primary copy to our secondary ECS copy. The MediaAgent that "owns" the storage policy will perform the data movement between copies while maintaining the deduplication scheme (if the option was enabled for the secondary copy).
For our example, we have a storage policy named "Isilon.windows.mediaagent" with a "primary" copy on Isilon and a "ecs - long term" secondary copy on ECS. When we view the backups on the Isilon primary copy we can see a completed client backup. When we view the backups on the secondary ECS copy we see the same job with a "to be copied" status since we have not yet run an aux copy.
CommVault GUI - example storage policy with two copies, Isilon and ECS
CommVault GUI - view jobs on primary Isilon copy
CommVault GUI - completed job #1486 on primary copy
CommVault GUI - view jobs on ECS secondary copy
CommVault GUI - job #1486 to be copied to ECS storage policy copy
So how do we move our backups between primary and secondary copies? Easy, just run an "auxillary copy" job or wait for the next scheduled aux copy job to run. We'll leave the default options and move "all copies". Below is the process to kick off an aux copy and also a few screenshots of the running job status and the completed job.
CommVault GUI - run axillary copy
CommVault GUI - aux copy job options
CommVault GUI - aux copy job details of running job
CommVault GUI - completed aux copy job
I now have my CommVault backups on two different storage libraries with different levels of performance and retention. My primary copy is on Isilon and intended to absorb the daily backup "churn" and store short-term backups with higher performance. My secondary copy is using ECS object storage for long-term backup storage. I run my operational SLA-driven backups on my primary copy and move my archival backups to an object store. The bulk of my restores will most likely come from my primary copy but I have the flexibility to also restore from any long-term copy on the ECS storage.
Think about using ECS for CommVault long-term backup storage. You can effectively build a cloud storage archive that is globally accessible across multiple datacenters and share this platform across multiple CommVault environments. You can scale an object storage platform to multiple petabytes and eliminate the hassles of trying to send this data out to the public cloud. And you can keep your primary short-term storage high performing while moving you long-term backups to an industry leading object storage platform.
Thanks for reading, comments welcome!