Find Communities by: Category | Product

trimbn

OneFS SmartQuotas and Dedupe

Posted by trimbn Dec 17, 2019

Got a question from the field asking whether a deduplicated file gets reported by and counted against SmartQuotas, and if there’s a performance penalty accessing that deduplicated file.


With OneFS, deduplicated files appear no differently than regular files to standard quota policies, regardless of whether the file has been deduplicated by SmartDedupe or OneFS in-line deduplication – or both. This is also true if the file is a clone or has been containerized by OneFS Small File Storage Efficiency (SFSE), both of which also use shadow stores, and also for in-line compression.


However, if the quota accounting is configured for ‘physical size’, which includes data-protection overhead, the additional space used by the shadow store will not be accounted for by the quota.

 

In OneFS 8.2.1, SmartQuotas has been enhanced to report the capacity saving from in-line data reduction as a storage efficiency ratio. SmartQuotas reports efficiency as a ratio across the desired data set as specified in the quota path field. The efficiency ratio is for the full quota directory and its contents, including any overhead, and reflects the net efficiency of deduplication (plus in-line compression, if available and enabled). On a cluster with licensed and configured SmartQuotas, this efficiency ratio can be easily viewed from the WebUI by navigating to ‘File System > SmartQuotas > Quotas and Usage’.


dedupe-quota-1.png


Similarly, the same data can be accessed from the OneFS command line via is ‘isi quota quotas list’ CLI command. For example:


# isi quota quotas list

Type      AppliesTo Path           Snap  Hard Soft  Adv  Used Efficiency

-----------------------------------------------------------------------------

directory DEFAULT    /ifs           No    - -     -    2.3247T 1.29 : 1

-----------------------------------------------------------------------------

Total: 1

 

More detail, including both the physical (raw) and logical (effective) data capacities, is also available via the ‘isi quota quotas view <path> <type>’ CLI command. For example:


# isi quota quotas view /ifs directory

                        Path: /ifs

                        Type: directory

Snapshots: No

Thresholds Include Overhead: No

                       Usage

                           Files: 4245818

Physical(With Overhead): 1.80T

Logical(W/O Overhead): 2.33T

Efficiency(Logical/Physical): 1.29 : 1

Creating and configuring a directory quota is a simple procedure and can be performed from the WebUI, as follows:


Navigate to ‘File System > SmartQuotas > Quotas and Usage’ and select ‘Create a Quota’. In the create pane, field, set the Quota type to ‘Directory quota’, add the preferred top-level path to report on, select ‘File system logical size’ for Quota Accounting, and set the Quota Limits to ‘Track storage without specifying a storage limit’. Finally, select the ‘Create Quota’ button to confirm the configuration and activate the new directory quota.

 

dedupe-quota-2.png

 

To configure SmartQuotas for in-line data efficiency reporting create a directory quota at the top-level file system directory of interest, for example /ifs. The efficiency ratio is a single, current-in time efficiency metric that is calculated per quota directory and includes the sum of in-line compression, zero block removal, in-line dedupe and SmartDedupe. This is in contrast to a history of stats over time, as reported in the ‘isi statistics data-reduction’ CLI command output, described above. As such, the efficiency ratio for the entire quota directory will reflect what is actually there.


In addition to SmartQuotas, OneFS provides several other reporting methods for obtaining efficiency information about deduplication, and data reduction in general. The most comprehensive of the data reduction reporting CLI utilities is the ‘isi statistics data-reduction’ command. For example:


# isi statistics data-reduction

Recent Writes (5 mins)              Cluster Data Reduction

----------------------------------  -----------------------------------------

Logical data            339.50G     Est. logical data             1.37T

Zero-removal saved      112.00k

Deduplication saved     432.00k     Dedupe saved                  1.41G

Compression saved       146.64G     Est. compression saved        199.82G

Preprotected physical   192.87G     Est. preprotected physical    1.18T

Protection overhead     157.26G     Est. protection overhead      401.22G

Protected physical      350.13G     Protected physical            1.57T

Deduplication ratio     1.00:1      Est. dedupe ratio             1.00:1

Compression ratio       1.76:1      Est. compression ratio        1.17:1

Data reduction ratio    1.76:1      Est. data reduction ratio     1.17:1

Efficiency ratio        0.97:1      Est. storage efficiency ratio 0.87:1

 

The ‘recent writes’ data to the left of the output provides precise statistics for the five-minute period prior to running the command. By contrast, the ‘cluster data reduction’ metrics on the right of the output are slightly less real-time but reflect the overall data and efficiencies across the cluster. This is designated by the ‘Est.’ prefix, denoting an ‘estimated’ value.

The ratio data in each column is calculated from the values above it. For instance, to calculate the data reduction ratio, the ‘logical data’ (effective) is divided by the ‘preprotected physical’ (usable) value. From the output above, this would be:


339.50 / 192.87 = 1.76    Or a Data Reduction ratio of 1.76:1


Similarly, the ‘efficiency ratio’ is calculated by dividing the ‘logical data’ (effective) by the ‘protected physical’ (raw) value. From the output above, this yields:


339.50 / 350.13 = 0.97    Or an Efficiency ratio of 0.97:1


In-line dedupe and post-process SmartDedupe both deliver very similar end results, just at different stages of data ingestion. Since both features use the same core components, the results are combined. As such, the isi dedupe stats output reflects the sum of both in-line dedupe and SmartDedupe efficiency.


# isi dedupe stats

      Cluster Physical Size: 86.14T

          Cluster Used Size: 4.44T

  Logical Size Deduplicated: 218.81G

             Logical Saving: 182.56G

Estimated Size Deduplicated: 271.92G

  Estimated Physical Saving: 226.88G

 

Similarly, the WebUI’s deduplication savings histogram combines the efficiency savings from both in-line dedupe and SmartDedupe.


dedupe-quota-3.png

 

OneFS’ WebUI cluster dashboard now displays a storage efficiency tile, which shows physical and logical space utilization histograms and reports the capacity saving from in-line data reduction as a storage efficiency ratio. This dashboard view is displayed by default when opening the OneFS WebUI in a browser and can be easily accessed by navigating to ‘File System > Dashboard > Cluster Overview’.

 

dedupe-quota-4.png

 

SmartDedupe also deduplicates common blocks within the same file, resulting in even better data efficiency.

 

InsightIQ, Isilon’s multi-cluster reporting and trending analytics suite, is also integrated with and able to report in detail on SmartDedupe. This is available from the performance reporting section of IIQ, by selecting “Deduplication” as the “Report Type”. Also, included in the data provided by the File Systems Reporting section, is a report detailing the space savings efficiency delivered by deduplication.


So how does SmartDedupe play with the other storage management and data protection tools in OneFS portfolio? Let’s take a look:


When deduplicated files are replicated to another Isilon cluster via SyncIQ, or backed up to a tape device, the deduplicated files are inflated (or rehydrated) back to their original size, since they no longer share blocks on the target Isilon cluster. However, once replicated data has landed, SmartDedupe can be run on the target cluster to provide the same space efficiency benefits as on the source.


Shadows stores are not transferred to target clusters or backup devices. Because of this, deduplicated files do not consume less space than non-deduplicated files when they are replicated or backed up. To avoid running out of space on target clusters or tape devices, it is important to verify that the total amount of storage space saved and storage space consumed does not exceed the available space on the target cluster or tape device. To reduce the amount of storage space consumed on a target Isilon cluster, you can configure deduplication for the target directories of your replication policies. Although this will deduplicate data on the target directory, it will not allow SyncIQ to transfer shadow stores. Deduplication is still performed post-replication, via a deduplication job running on the target cluster.


Because files are backed up as if the files were not deduplicated, backup and replication operations are not faster for deduplicated data. You can deduplicate data while the data is being replicated or backed up. It’s also worth noting that OneFS NDMP backup data won’t be deduped unless deduplication is provided by the backup vendor’s DMA software. However, compression is often provided natively by the backup tape or VTL device instead.


SmartDedupe does not deduplicate the data stored in a snapshot. However, snapshots can be created of deduplicated data. If a snapshot is taken of a deduplicated directory, and then the contents of that directory are modified, the shadow stores will be transferred to the snapshot over time. Because of this, more space will be saved on a cluster if deduplication is run prior to enabling snapshots. If deduplication is enabled on a cluster that already has a significant amount of data stored in snapshots, it will take time before the snapshot data is affected by deduplication. Newly created snapshots will contain deduplicated data, but older snapshots will not.


It is also good practice to revert a snapshot before running a deduplication job. Restoring a snapshot will cause many of the files on the cluster to be overwritten. Any deduplicated files are reverted back to normal files if they are overwritten by a snapshot revert. However, once the snapshot revert is complete, deduplication can be run on the directory again and the resulting space savings will persist on the cluster.


Dedupe is also fully compatible with SmartLock, OneFS’ data retention and compliance product. SmartDedupe delivers storage efficiency for immutable archives and write once, read many (or WORM) protected data sets.

However, OneFS will not deduplicate files that span SmartPools pools or tiers, or that have different protection levels set. This is to avoid potential performance or protection asymmetry which could occur if portions of a file live on different classes of storage.


InsightIQ, Isilon’s multi-cluster reporting and trending analytics suite, is also integrated with and able to report in detail on SmartDedupe. This is available from the performance reporting section of IIQ, by selecting “Deduplication” as the “Report Type”. Also, included in the data provided by the File Systems Reporting section, is a report detailing the space savings efficiency delivered by deduplication.

trimbn

OneFS Snapshot Scheduling

Posted by trimbn Dec 11, 2019

One question that frequently crops up from the field is what snapshot schedule to configure on a particular cluster.


SnapshotIQ scheduling allows cluster administrators to automatically generate snapshots according to a pre-defined itinerary. While there definitely isn’t a ‘one size fits all’ recommendation to make, three of main drivers for this decision are:

 

  • Recovery point objective (RPO)
  • Available cluster capacity
  • Dataset rate of change

 

An organization’s data security, availability, and disaster recovery policy will often answer the first question – how much? Many companies define explicit service level requirements (SLAs) around the availability of their data. RPO is the acceptable amount of data loss that can be tolerated. With an RPO of 30-minutes, for example, a half hour is the maximum amount of time that can elapse since the last backup or snapshot was taken.


While OneFS does not require any cluster capacity to be exclusively reserved for snapshots, obviously snaps do consume space. Furthermore, this space will grow the more HEAD data changes, and as more snapshots are retained.


OneFS snapshot schedules can be configured at daily, weekly, monthly or yearly intervals, with single or multiple job frequency per schedule, and down to a per-minute granularity.

There are two main strategies for snapshot scheduling:

  • Ordered Deletion
  • Unordered Deletion


Ordered deletion is suited to data sets with a low rate of change, such as archive or other cold data; whereas unordered deletion, which retains considerably fewer snapshots, is recommended for more active data, or clusters with limited capacity available.

The following table provides a recommended snapshot schedule for both ordered and unordered deletion configurations:

snap_schedule2.png

 

The following CLI command will create a schedule for hourly snapshots of the /ifs/data/prod directory and its contents, plus a one month retention setting:


# isi snapshot schedules create hourly /ifs/data/media HourlyBackup_%m-%d-%Y_%H:%M "Every day every hour" --duration 1M


To configure a similar schedule from the WebUI, navigating to Data Protection > Snapshots > Snapshot Schedules and clicking on the ‘Create a Schedule’ button.


snapshot_sched_1.png


On the other hand, the following commands create an unordered deletion schedule for /ifs/data/prod that generate snapshots at hourly, daily, weekly and monthly cadences:

 

# isi snapshot schedules create every-other-hour /ifs/data/prod EveryOtherHourBackup_%m-%d-%Y_%H:%M "Every day every 2 hours" --duration 1D


# isi snapshot schedules create daily /ifs/data/prod Daily_%m-%d-%Y_%H:%M "Every day at 12:00 AM" --duration 1W


# isi snapshot schedules create weekly /ifs/data/prod Weekly_%m-%d-%Y_%H:%M "Every Saturday at 12:00 AM" --duration 1M


# isi snapshot schedules create monthly /ifs/data/prod Monthly_%m-%d-%Y_%H:%M "The 1 Saturday of every month at 12:00 AM" --duration 3M

 

Existing snapshot schedules can be viewed from the CLI with the following command:

# isi snapshot schedules list

ID Name

---------------------

1 every-hour

2 daily

3 weekly

4 monthly

---------------------

 

More detailed information about a particular snapshot is also available. For example, the following command will display more context about the ‘every-hour’ schedule above:

# isi snapshot schedules view every-hour

ID: 1

Name: every-other-hour

Path: /ifs/data/media

Pattern: EveryOtherHourBackup_%m-%d-%Y_%H:%M

Schedule: Every day every 2 hours

Duration: 1D

Alias: -

Next Run: 2019-12-10T17:00:00

Next Snapshot: EveryHourBackup_12-10-2019_17:00

 

Another important consideration when configuring snapshot schedules at any level of scale is the snapshot naming convention. If you schedule snapshots to be automatically generated, either according to a snapshot schedule or a replication policy, a snapshot naming pattern that determines how the snapshots are named. Snapshot naming patterns contain variables that include information about how and when the snapshot was created.

The following variables can be included in a snapshot naming pattern:

 

Variable

Description

%A

Day of the week

%a

Abbreviated day of the week. Ie. if the snapshot is generated on a Sunday, %a will have value ‘Sun’

%B

Name of the month

%b

Abbreviated name of the month. Ie. if the snapshot is generated in September, %b will have value ‘Sep’

%C

First two digits of the year

%c

The time and day. This variable is equivalent to specifying %a %b %e %T %Y

%d

Two-digit day of the month

%e

Day of the month. A single digit day is preceded by a blank space

%F

The date. This variable is equivalent to %Y-%m-%d

%G

The year. This variable is equivalent to specifying %Y. However, if the snapshot is created in a week that has less than four days in the current year, the year that contains the majority of the days of the week is displayed. The first day of the week is calculated as Monday. Ie, if a snapshot is created on Sunday, January 1, 2020, %G is replaced with 2019, because only one day of that week is in 2019.

%g

The abbreviated year. This variable is equivalent to specifying %y.

%H

The hour. The hour is represented on the 24-hour clock. Single-digit hours are preceded by a zero. For example, if a snapshot is created at 1:45 AM, %H is replaced with 01

%I

The hour represented on the 12-hour clock. Single-digit hours are preceded by a zero. Ie. if a snapshot is created at 1:45 PM, %I is replaced with 01

%j

The numeric day of the year. Ie. If a snapshot is created on February 1, %j is replaced with 32 .

%k

The hour represented on the 24-hour clock.

%l

The hour represented on the 12-hour clock. Single-digit hours are preceded by a blank space. Ie, if a snapshot is created at 1:45 AM, %I is replaced with 1

%M

Two-digit minute

%m

Two-digit month

%p

AM or PM

%{PolicyName}

Name of the replication policy that the snapshot was created for. This variable is only valid if specifying a snapshot naming pattern for a replication policy.

%R

The time. This variable is equivalent to specifying %H:%M

%r

The time. This variable is equivalent to specifying %I:%M:%S %p

%S

Two-digit second

%s

The second represented in POSIX time

%{SrcCluster}

The name of the source cluster of the replication policy that the snapshot was created for. Valid only if specifying a snapshot naming pattern for a replication policy

%T

The time. Equivalent to %H: %M: %S

%U

Two-digit numerical week of the year

%u

Numerical day of the week. Ie. If a snapshot is created on Sunday, %u has value 7

%V

Two-digit numerical week

%v

Day of snapshot creation. Equivalent to %a-%b-%Y

%W

Two-digit numerical week of the year that the snapshot was created in

%X

Time that snapshot was created. Equivalent to %H: %M: %S

%Y

Year the snapshot was created

%y

Last two digits of snapshot creation year

%Z

Time zone the snapshot was created in

%z

Offset from UTC time of time zone snapshot was created in

%+

Time and date of snapshot creation. Equivalent to %a %b %e %X %Z %Y

 

 

Similarly, automatic snapshot deletion can also be configured per defined schedule at an hourly through yearly range.

Snapshot attributes such as name and expiry date can easily be changed. For example, the following command will cause the snapshot ‘HourlyBackup_06-15-2018_22:00 to expire at 2:30 PM on 30th December 2019:


# isi snapshot snapshots modify HourlyBackup_06-15-2018_22:00 --expires 2019-12-30T02:30

A snapshot schedule can also be easily modified. However, any changes to a schedule are applied only to snapshots generated after the modifications are made. Existing snapshots are not affected by schedule modifications. If the alias of a snapshot schedule is modified, the alias is assigned to the next snapshot generated based on the schedule. However, the old alias is not removed from the last snapshot that it was assigned to. Unless you manually remove the old alias, the alias will remain attached to the last snapshot that it was assigned to.

For example, the following command causes snapshots created according to the snapshot schedule hourly_prod_snap to be deleted 15 days after they are created:

# isi snapshot schedules modify hourly_prod_snap --duration 15D


Similarly, deleting a snapshot schedule will not remove snapshots that were previously generated according to the schedule.

The following command will delete the snapshot schedule named ‘hourly_prod_snap’:

# isi snapshot schedules delete hourly_prod_snap


You can configure a snapshot schedule to assign a snapshot alias to the most recent snapshot created by the schedule. As such, the alias will be  assigned to the next snapshot generated based on the schedule. However, the old alias is not automatically removed from the last snapshot that it was assigned to. Unless you manually remove the old alias, the alias will remain attached to the last snapshot that it was assigned to.

For example, the following command will configure the the snapshot schedule WeeklySnapshot to use the alias ‘LatestWeekly’:

# isi snapshot schedules modify WeeklySnapshot –alias LatestWeekly


It’s worth noting that a snapshot schedule cannot span multiple days. For example, you cannot specify to begin generating snapshots at 5:00 PM Monday and end at 5:00 AM Tuesday. Instead, to continuously generate snapshots for a period greater than a day, two individual snapshot schedules are required.

 

In order to generate snapshots from 5:00 PM Monday to 5:00 AM Tuesday, for example, create one schedule that generates snapshots from 5:00 PM to 11:59 PM on Monday, and another schedule that generates snapshots from 12:00 AM to 5:00 AM on Tuesday.

 

For mixed node clusters, associated with the snapshot schedule frequency question may also be a decision as to which storage tier of a cluster to house the snapshots on.  This can be set, along with a specific protection level and SSD strategy for the snapshot, in the SmartPools file pool policy configuration. For example, from the WebUI browse to File System > Storage Pools > File Pools and select the desired policy.

 

snapshot_sched_3.png

 

SnapshotIQ also provides a number of global snapshot settings, including:


  • Control of auto-creation of scheduled snapshots
  • Deletion of expired snapshots
  • The ability to enable and disable the snapshot service
  • Per-protocol and complete control of snapshot visibility and and accessibility

 

These global snapshot storage settings can be accessed and configured in the WebUI by browsing to Data Protection > Snapshots > Settings:


snapshot_sched_4.png

 

Or from the CLI, via:


# isi snapshot settings view

 

The following table provides a description of the global snapshot configuration settings:

 

Attribute

Description

Autodelete

Determines whether snapshots are automatically deleted according to their expiration dates.

Reserve

Specifies the percentage of disk space on the cluster that is reserved for snapshots.

NFS Root Accessible

Determines whether snapshot directories are accessible through NFS

NFS Root Visible

Determines whether snapshot directories are visible through NFS.

NFS Subdir Accessible

Determines whether snapshot subdirectories are accessible through NFS.

SMB Root Accessible

Determines whether snapshot directories are accessible through SMB. 

SMB Root Visible

Determines whether snapshot directories are visible through SMB.

SMB Subdir Accessible

Determines whether snapshot subdirectories are accessible through SMB.

Local Root Accessible

Determines whether snapshot directories are accessible through an SSH connection or the local console.

Local Root Visible

Determines whether snapshot directories are visible through the an SSH connection or the local console.

Local Subdir Accessible

Determines whether snapshot subdirectories are accessible through an SSH connection or the local console.

Received an interesting snapshot restore inquiry from the field and thought it was worth incorporating into a blog article. The scenario this this: A large amount of data needs to be restored on a cluster. Unfortunately, the SnapshotIQ policies are configured at the root /ifs level and it is not feasible to restore every subdirectory under the snapshot. Although the files themselves are not that large, the subdirectories contain anywhere from thousands to tens of millions of files. Restores are taking a very long time when copying the directories manually.


So, there are two main issues at play here:


  • Since the snapshot is taken at a lower level in the directory tree and the entire snapshot cannot be restored in place, using the SnapRevert job is not an option here.
  • The sheer quantity of files involved mean that a manual, serial restore of the data will be incredibly time consuming.


Fortunately, there is a solution that involves using replication. SyncIQ allows for snapshot subdirectories to be included or excluded, plus also provides the performance benefit of parallel job processing.


SyncIQ contains an option only available via the command line (CLI) which allows replicate out of a snapshot.


The procedure is as follows:


1)     Create a snapshot of a root directory.

# isi snapshot snapshots create --name snaptest3 /ifs/data


2)     List the available snapshots and select the desired instance.

 

For example:


# isi snapshot list

ID Name Path

----------------------------------------------------

6 FSAnalyze-Snapshot-Current-1529557209 /ifs

8    snaptest3                             /ifs/data

----------------------------------------------------

Total: 2


Note that there are a couple of caveats:


  • The subdirectory to be restored must still exist in the HEAD filesystem (ie. not have been deleted since the snapshot was taken).
  • You cannot replicate data from a SyncIQ generated snapshot.

 

3)     Create a local SyncIQ replication policy with the snapshot source as the original location and a new directory location on ‘localhost’ as the destination. The ‘—source-include-directories’ argument lists the desired subdirectory(s) to restore.

 

For example, via the CLI:

 

# isi sync policies create snapshot_sync3 sync /ifs/data localhost /ifs/file_sync3 --source-include-directories /ifs/data/local_qa

 

Or via the WebUI:

 

SyncIQ_snapshot_replication_1.png

 

Note:  You cannot configure the snapshot into the policy, or set source=snapshot.


4)     Next, run the sync job to replicate a subset of a snapshot. This step is CLI only (not WebUI) since the SyncIQ policy needs to be executed with ‘--source-snapshot’ argument specified.

 

For example:


# isi sync job start snapshot_sync3 --source-snapshot=snaptest3


Note: This command is essentially a change root for the single run of the SyncIQ Job.


5)     Finally, rename the original directory to something else with mv, and then rename the restore location to the original name.

 

For example:

 

# mv /ifs/data/local_qa /ifs/data/local_qa_old

# mv /ifs/file_sync3/local_qa /ifs/data/local_qa


If you do not have a current replication license on your cluster, you can enable the OneFS SyncIQ trial license from the WebUI by browsing to Cluster Management > Licensing.


Using SyncIQ in this manner is a very efficient way to recover large amounts of data from within snapshots. However, this scenario also illustrates one of the drawbacks of taking snapshots at the root directory level. Consider whether it’s more advantageous to configure snapshot schedules to capture at the subdirectory directory level instead.

Filter Blog

By date:
By tag: