Received a recent enquiry about snapshot deletion, and thought it was worth elaborating upon in a blog article:

OneFS Snapshots are created at the directory-level instead of the volume-level, thereby providing a high degree of granularity. However, they are a point in time immutable copy of a specified subset of OneFS’ data, so can’t be altered in any way once taken - beyond a full deletion. As such, removing a portion of an existing snapshot is not an option: Deleting an Isilon snapshot is an all-or-nothing event.


There are a couple of useful OneFS CLI commands that show how much space is consumed by snapshots:


First up, the ‘isi_classic snapshot usage’ command will display the existing snapshots and their disk usage. For example:


# isi_classic snapshot usage

FSAnalyze-Snapshot-Current-1530077114             51G     n/a (R) 0.00% (T)

SIQ-Failover-snapshot_sync3-2019-10-22            4.0K     n/a (R) 0.00% (T)

SIQ-Failover-snapsync-2019-10-22_12-02            4.0K     n/a (R) 0.00% (T)

Snapshot: 2019Oct22, 08:20:05 AM                  1.9G     n/a (R) 0.00% (T)

[snapid 57, delete pending]                          0     n/a (R) 0.00% (T)

snaptest1 6.0K     n/a (R)    0.00% (T)

snaptest2 70K     n/a (R)    0.00% (T)

snaptest3                                         1.3M     n/a (R) 0.00% (T)


In addition to the name of the snapshot and the amount of space the snapshot takes up, the percentage of the snapshot reserved space this accounts for (R), and the percentage of the total filesystem space this accounts for (T) are also displayed.


Secondly, the ‘isi snapshot view’ command can be used to find more detailed info for an individual snapshot. This includes the snapshot path, alias, ID, whether there are any locks, expiry date, etc. For example:

 

# isi snapshot view FSAnalyze-Snapshot-Current-1530077114

ID: 56

Name: FSAnalyze-Snapshot-Current-1530077114

Path: /ifs

Has Locks: No

Schedule: -

  Alias Target ID: -

Alias Target Name: -

Created: 2019-10-26T22:25:14

Expires: -

Size: 50.764G

Shadow Bytes: 0

% Reserve: 0.00%

     % Filesystem: 0.00%

State: active

 

 

Snapshots can be automatically deleted on a preconfigured schedule, or manually deleted via the ‘isi snapshot snapshots delete’ CLI command.



Usage:

    isi snapshot snapshots delete { <snapshot> | --schedule <string> | --type

(alias | real) | --all }

[{--force | -f}]

[{--verbose | -v}]

[{--help | -h}]

 

Options:

<snapshot>

Delete a single snapshot.

<schedule>

Delete all snapshots created by the given schedule.

    <type> (alias | real)

Delete all snapshots of the specified type.

    --all

Delete all snapshots.

 

 

Let’s look at a simple example:


1)  The following snapshot usage command lists the available snapshots and their size, ordered by age:


# isi_classic snapshot usage

CBsnapshot                                      85K      n/a (R)    0.00% (T)

Hourly - prod 6.0K     n/a (R)    0.00% (T)

SIQ-Failover-CBpolicy1-2019-10-29_13-0          6.0K     n/a (R) 0.00% (T)

Daily_2019-11-12_12:00                          584M     n/a (R) 0.00% (T)

Weekly_2019-11-11_12:00                         6.0K     n/a (R) 0.00% (T)

 

From this output, we can see the snapshot ‘Daily_2019-11-12_12:00’ is 584MB in size and appears to be a viable candidate for deletion.

 

2)  The following CLI command will return the snapshot ID.   


# isi snapshot snapshots list | grep “Daily_2019-11-12_12:00” | awk '{print $1}'

110


3)  Next, we can use the snap ID to verify the snapshot details to ensure its deletion is desirable:


# isi snapshot snapshots view `isi snapshot snapshots list | grep Daily_2019-11-12_12:00| awk '{print $1}'`

 

ID: 110

Name: Daily_2019-11-12_12:00

Path: /ifs

        Has Locks: No

Schedule: Daily @ Noon

  Alias Target ID: -

Alias Target Name: -

          Created: 2019-11-12T12:00:06

          Expires: -

Size: 582.45M

     Shadow Bytes: 0

        % Reserve: 0.00%

     % Filesystem: 0.00%

State: active

 

The output confirms that it’s the correct snapshot, its size, and that it’s not locked, etc.

 

4)  The following syntax will delete the snapshot ID 110, after prompting for confirmation:

 

# isi snapshot snapshots delete 110

Are you sure? (yes/[no]):


5)  A detailed report of the SnapshotDelete job can then be viewed from the WebUI. This can be found by browsing to Job Operations > Job Reports, filtering for ‘SnapshotDelete’, and selecting ‘View Details’ for the desired job.

 

When it comes to deleting snapshots, there are a couple of rules of thumb to keep in mind:

 

  • The oldest snapshot can be removed very quickly. An ordered deletion is the deletion of the oldest snapshot of a directory, and is a recommended best practice for snapshot management. An unordered deletion is the removal of a snapshot that is not the oldest in a directory, and can often take approximately twice as long to complete and consume more cluster resources than ordered deletions.

 

  • Where possible, avoid deleting snapshots from the middle of a time range. Newer snapshots are mostly pointers to older snapshots, and they look larger than they really are. Removing the newer snapshots will not free up much space. Deleting the oldest snapshot ensures you will actually free up the space. You can determine snapshot order (if not by name or date) by using the isi snapshot list -l command. The snapshot IDs (first column) are non-conserved, serial values.

  • Avoid deleting SyncIQ snapshots (snapshots with names that start with SIQ), unless the only remaining snapshots on the cluster are SyncIQ snapshots, and the only way to free up space is to delete those SyncIQ snapshots. Deleting SyncIQ snapshots resets the SyncIQ policy state, which requires a reset of the policy and potentially a full sync or initial diff sync. A full sync or initial diff sync could take many times longer than a regular snapshot-based incremental sync.

 

So what happens under the hood? Upon deleting a snapshot, OneFS immediately modifies some of the tracking data and the snapshot disappears from view. However, the actual behind-the-scenes clean-up of the snapshot can involve a fair amount of work, which is performed in the second phase of the SnapshotDelete job. There is no requirement for reserved space for snapshots in OneFS. Snapshots can use as much or little of the available file system space as desirable.

In the example below, snapshot ID 100 is being deleted. To accomplish this, any changes will likely need to be moved to the prior snapshot (ID 98), because that snapshot will no longer be able to read forward.

 

snap_delete_1.png

 

Snapshot 100 has two changed blocks: block 0 and block 4.  Snapshot 98 was changed after snapshot 98 was taken, so block 4 can be deleted, but block 0 needs to be moved over to snapshot 98.


snap_delete_2.png

 

It’s worth noting that SnapshotDelete will only run if the cluster is in a fully available state, i.e., no drives or nodes are down.

 

If you have old, large snapshots consuming space and the cluster does not have a current SnapshotIQ license, contact Dell EMC Isilon Technical Support to discuss your options and assistance with deleting the old snapshots.