Welcome back to VMAX & OpenStack Ocata: An Inside Look! Last time around we looked at basic operations in OpenStack using VMAX as the block-storage back end, if you would like to read that article again you can do so by clicking here. Topics included volume management, cloning a volume, and attaching or detaching a volume from a Nova compute instance. This time we are going to look at Snapshots & Backups in OpenStack, following a similar format to the previous articles, after breaking down the functionality, I will give a demonstration using CLI commands and provide a video demonstration at the end using the Horizon dashboard.
Note: Cinder backups are not available through the Horizon dashboard in OpenStack Ocata, to make use of the Cinder backup service you must use CLI commands
What are Cinder Volume Snapshots?
A Cinder volume snapshot is a read-only, point-in time copy of a block-storage volume. Snapshots can be created from existing block-storage volumes that in the state 'available' or 'in-use', meaning that the snapshot can be created when it is attached to a running instance in OpenStack. A snapshot in Cinder can be used as the source for a new Cinder volume, allowing for a quick and easy way to either restore an volume to a previous point-in-time, or used almost as an 'image' from which to provision new volumes from. Cinder volume snapshots are not to be confused with Nova instance snapshots. Instance snapshots are not only the attached persistent volumes that live on the Virtual Machine but also the flat files that make up the virtual machine itself. So when you take a snapshot of the VM its the exact point in time of the OS, all of it's files etc.
What are Cinder Volume Backups?
A Cinder volume backup is an exact copy, a full clone, that can reside at the same site as its read/write source, or in a secondary site for disaster recovery solutions. A Cinder volume backup can be viewed as all volume copies from the source storage array that are stored externally of the source storage array. A backup captures the state of its source at the time the backup is made, and its contents cannot change. The Cinder-Backup service two back ends to backup to: Ceph and Swift. For the VMAX Cinder block storage drivers, Swift is the chosen back end to backup to.
Backups vs Snapshots
Although very similar in their purpose in OpenStack, there are subtle differences between both and how they should best be used. Cinder snapshots serve as a way to continually take exact point-in-time copies of the data within a volume that resides on the same storage device that the source volume lives on. Cinder backups are full-copy volume clones which are in most circumstances stored on a different platter or storage node from which the source volume resides. Best practices for Cinder backups would ideally result in the backup being stored on a storage node in a data-centre is a different geo-location for disaster-recovery situations.
Creating & Managing Volume Snapshots
Snapshot operations in OpenStack are performed using the cinder snapshot-[command] format, where command is replaced by operations such as 'create', 'delete', or 'list'. A full list of Cinder Snapshot CLI commands can be found in the Cinder CLI reference guide here. To create a snapshot of a volume in OpenStack you will need a volume, so I will create a 1GB bootable volume using the CirrOS image again.
Using the available bootable volume 'cirros_vol', we can create a point-in-time snapshot of the volume.
$ cinder snapshot-create --name [snapshot_name] [volume_name/volume_id]
$ cinder snapshot-create --name cirros_snap_1 cirros_vol
To list your available OpenStack snapshots, use the cinder snapshot-list CLI command. To view specific details about a particular snapshot, use the cinder snapshot-show command.
$ cinder snapshot-show [snapshot_name/snapshot_id]
$ cinder snapshot-show bb27b18b-c135-481c-939f-f3945c5cd070
Another possibility with snapshot management is the ability to rename the snapshot or add a description to it.
$ cinder snapshot-rename [snapshot_name/snapshot_id] [new_snapshot_name]
$ cinder snapshot-rename cirros_snap_1 cirros_snap_june
To take the rename command even further, if you want to add a description to the snapshot, you can either repeat the command exactly as above and add the --description [user_description] parameter and value, or you can leave out the new snapshot name entirely (example in screenshot below).
$ cinder snapshot-rename [snapshot_name/snapshot_id] [new_snapshot_name] --description [user_description]
$ cinder snapshot-rename cirros_snap_1 cirros_snap_june --description "June 2nd 2017"
$ cinder snapshot-rename cirros_snap_june --description "June 2nd 2017"
Deleting a snapshot in OpenStack using the CLI only requires a simple cinder snapshot-delete command and the snapshot name or ID. You can also additionally add the --force parameter to the command if a snapshot of a volume is in a state other that 'available' or 'error'.
$ cinder snapshot-delete [snapshot_name/snapshot_id][--force]
$ cinder snapshot-delete cirros_snap_june
Creating a Volume from a Snapshot
Creating a volume from a snapshot is almost identical to creating a blank volume with no source, the only difference being the inclusion of the --snapshot-id parameter.
$ cinder create --snapshot-id [snapshot_id]
$ cinder create --snapshot-id 136d7787-7a4d-46ea-9337-a5ce6cbeddb5 --name vmax_vol1 --volume-type VMAX_ISCSI_DIAMOND 1
Creating & Managing Cinder Volume & Snapshot Backups
The Cinder CLI provides the tools for creating a volume backup. You can restore a volume from a backup as long as the backup’s associated database information (or backup metadata) is intact in the Block Storage database. In advance of using Cinder's backup service, you will need to make sure that you have the backup service correctly configured. The configuration will be dependent on the type of service which will interact with Cinder backup to store the backups (in this example we will be using Swift object storage service). For more information on configuring your environment for Cinder backup, please follow the detailed instructions at the official OpenStack website for 'Install and configure the backup service'. At this point we will assume that you have your backups service installed and configured, and you have a running instance in OpenStack with a Cinder backed bootable volume attached.
To create a backup of a volume use the following command:
$ cinder backup-create [--name <name>] [--incremental] [--force] <volume>
$ cinder backup-create --name cirros_backup --force cirros_boot_vol
Where volume is the name or ID of the volume,
incremental is a flag that indicates whether an incremental backup should be performed, and
force is a flag that allows or disallows backup of a volume when the volume is attached to an instance.
incremental flag, a full backup is created by default. With the
incremental flag, an incremental backup is created. The incremental backup is based on a parent backup which is an existing backup with the latest timestamp. The parent backup can be a full backup or an incremental backup depending on the timestamp.
force flag, the volume will be backed up only if its status is
available. With the
force flag, the volume will be backed up whether its status is
in-use. A volume is
in-use when it is attached to an instance. The backup of an
in-use volume means your data is crash consistent. The
force flag is False by default.
Note: The first backup of a volume has to be a full backup. Attempting to do an incremental backup without any existing backups will fail. There is an
is_incremental flag that indicates whether a backup is incremental when showing details on the backup. Another flag,
has_dependent_backups, returned when showing backup details, will indicate whether the backup has dependent backups. If it is
true, attempting to delete this backup will fail.
To list all of the backups currently available in your environment use the cinder backup-list command, to show specific details about any given backup, use the command cinder backup-show <volume> where volume is the backup name or ID.
To restore a volume backup, there are number of options to take into consideration depending on the type of restoration you require. The CLI command is structured as follows:
$ cinder backup-restore [--volume <volume>] [--name <name>] <backup>
$ cinder backup-restore --name cirros_restored cirros_backup
Both the volume and name parameters in the backup-restore command are optional, specifying only the backup to restore will create a new volume with a randomly generated name assigned. Supplying the name will give the new volume a user-specified name, and including the volume parameter tells OpenStack which existing volume is to be restored using the backup.
Note: When restoring from a full backup, a full restore will be carried out. Also, for a restore to proceed successfully on a pre-existing volume, the target volume must have an 'available' status.
Deleting a Cinder volume backup is handled similarly to all other delete operations in OpenStack, you need only supply the name or ID of the resource you want to delete, multiple backups can be specified to delete more than one backup in the same CLI command.
$ cinder backup-delete [--force] <backup> [<backup> ...]
$ cinder backup-delete cirros_backup
The force flag is used in the delete command to remove backups which have a status other than 'available' or 'in-use'.
Users can take snapshots from the volumes as a way to protect their data. These snapshots reside on the storage back end itself. Providing a way to backup snapshots directly allows users to protect the snapshots taken from the volumes on a backup device, separately from the storage back end. There are users who have taken many snapshots and would like a way to protect these snapshots. The functionality to backup snapshots provides another layer of data protection. To create a backup of a snapshot, use the same command format as creating a normal Cinder volume backup, but this time include the --snapshot-id parameter:
$ cinder backup-create [--name <name>] --snapshot-id [snapshot] [--force] <volume>
$ cinder backup-create --name cirros_snap_backup --snapshot-id df942bd8-f509-4535-9e37-90a4958e0ea0 --force cirros_boot_vol
When backing up a snapshot, it is not enough to just specify the snapshot ID, you must also specify the ID of the source volume also. In addition, if the source volume is 'in-use' the --force parameter must also be included. When specifying the snapshot ID, only the snapshot ID will suffice, it is not possible to use the snapshot name.
Restoring a backup of a snapshot uses exactly the same process as restoring a standard backup, you specify the snapshot backup ID instead of a volume backup ID.
Important Note on Cinder Backup Service
Cinder volume backups are dependent on the Block Storage database, meaning that if the database becomes unavailable or corrupted your backups won't be of any use to you. In order to prevent against unusable Cinder volume backups, you must also back up your Block Storage database regularly to ensure data recovery. To learn more about backing up OpenStack databases please read the Official OpenStack documentation 'Backup and Recovery'.
Alternatively, you can export and save the metadata of selected volume backups. Doing so precludes the need to back up the entire Block Storage database. This is useful if you need only a small subset of volumes to survive a catastrophic database failure. For more information about how to export and import volume backup metadata, see the Official OpenStack documentation 'Export and import backup metadata'.
To show all of the above operations using the Horizon dashboard, I have created a short demo video. Full-screen viewing of the embedded video is not possible from the DECN website, to view the video full-screen click here or click on the YouTube logo on the video controls to redirect to the video on the YouTube website.
Troubleshooting issues with Snapshots and Backups
There is very little that can go wrong with snapshots and backups in terms of setup and configuration, if the setup from part 1 of this series is correct, and you can provision volumes with no issues, then there won't be many areas that need looked at if something goes wrong.
As mentioned in the introductory article in this series, there are a number of licenses which are required for a VMAX to function correctly in an OpenStack environment. These licenses are directly related to the functionality which has been described in this article on snapshots and backups so it is essential that they are installed into your VMAX before OpenStack can be used to its full potential.
The list of required licenses is very briefly touched on in the table below, but I would highly recommend getting in contact with your VMAX account manager for more information on licensing and in particular licensing for OpenStack.
|Advanced Suite||Local Replication Suite||Remote Replication Suite|
|Cinder Snapshots||Required||Required||Not required|
|Cinder Backups (local)||Required||Required||Not required|
|Cinder Backups (remote)||Required||Required||Not required|
|Cinder Replication (SRDF/S)||Required||Not required||Required|
If the setup is correct and all required software licenses are installed there should be no issues with creating snapshots of VMAX volumes in OpenStack, however, if you believe you have found a bug in the Snapshot functionality get in contact to let us know through your Dell EMC rep. If you are having issues with Cinder backups, make sure the service is correctly installed and configured as per the instructions in the guide 'Install and configure the backup service'. Configuring services other than Cinder do not fall under the scope of our series here, but there is ample resources on the internet for setting up Cinder backup with services such as Ceph or Swift.
What's coming up in part 5 of 'VMAX & OpenStack Ocata: An Inside Look'...In the next in the series of 'VMAX & OpenStack Ocata: An inside Look' we will be looking at managing and unmanaging volumes. These operations will allow us to manage existing VMAX volumes into OpenStack for use in the environment, or unmanage volumes whereby OpenStack volumes are removed from the environment but still remain on the VMAX for later use. See you then!