Welcome back! In the previous article we looked at managing and unmanaging VMAX volumes in OpenStack, if you would like to read that article again click here. This time around it is part 6 of VMAX & OpenStack Ocata: An Inside Look.

 

Today we will be looking at Consistency Groups (CGs) in OpenStack, not to be confused with VMAX Consistency Groups which is an SRDF feature. So far in our adventures into OpenStack & VMAX functionality everything has happened at the volume level. With Consistency Groups, the functionality of snapshots is taken further by taking a group of volumes and creating a consistent snapshot of the volumes at that point in time to ensure data consistency. Consistency groups are needed to group volumes together for the purpose of data protection (snapshots, backups) and remote replication for disaster recovery.

 

Imagine this scenario, a mission critical system has two attached block storage devices each having their own database, each relying on data from the other.If you wanted to create a snapshot of this system using standard snapshots you would have to manually create snapshots of each volume.

 

The biggest problem with this is that due to the manual nature of this process, the snapshot of each volume will not leave the volume snapshots in a consistent state. For example, in the time that it takes to send the command to snapshot volume 1, data will have been written to the database in volume 2. When it comes to taking a snapshot of volume 2, the data within does not correlate with the exact point in time as the snapshot of volume 1. In a restore process, this could have the knock on effect of data being inconsistent or missing, data written to volume 2 may not have the correct corresponding data in volume 1 as it had been written just fractionally after the snapshot of volume 1 was created.

 

This is where CGs come in. By adding both volumes to a CG, when a CG snapshot is taken of that group the snapshots represent the state of both volumes at that exact point in time, keeping your volume data consistent across both in the event of restoring the volumes or creating new volumes.

 

Consistency Group Behaviour & Required Setup

At present, using CGs in OpenStack is achievable only via the Cinder CLI, so there won't be an accompanying video as all of the features of the CLI will be demonstrated using screenshots and command examples. Before working with CGs, there are a few things to take into consideration about the behaviour of the feature:

  • A CG can support more than one volume type, the Cinder scheduler is responsible for finding a back end that can support all volume types
  • A CG can only contain volumes hosted by the same back end
  • A CG is empty when created, volumes are added later
  • The following operations are not allowed if a volume is in a CG:
    • Volume migration
    • Volume retype
    • Volume deletion (CG has to be deleted as a whole with all the volumes)
  • CGs can only be deleted if all the volumes in the CG are deleted along with it, or if the CG is empty
  • Volume snapshot deletion is not allowed if the volume snapshot is in a CG snapshot (CG snapshot has to be deleted as a whole with all the volume snapshots)

 

Before using CGs in OpenStack, you need to change the policies for the CG APIs in the /etc/cinder/policy.json file. By default, these CG APIs are disabled, so make sure to enable them before running any CG operations. To enable CG functionality, remove 'group:nobody' from the policy entries for CGs:

 

{
    "consistencygroup:create": "group:nobody"
    "consistencygroup:delete": "group:nobody",
    "consistencygroup:update": "group:nobody",
    "consistencygroup:get": "group:nobody",
    "consistencygroup:get_all": "group:nobody",

    "consistencygroup:create_cgsnapshot" : "group:nobody",
    "consistencygroup:delete_cgsnapshot": "group:nobody",
    "consistencygroup:get_cgsnapshot": "group:nobody",
    "consistencygroup:get_all_cgsnapshots": "group:nobody",
}















Becomes...

{
    "consistencygroup:create": "",
    "consistencygroup:delete": "",
    "consistencygroup:update": "",
    "consistencygroup:get": "",
    "consistencygroup:get_all": "",

    "consistencygroup:create_cgsnapshot" : "",
    "consistencygroup:delete_cgsnapshot": "",
    "consistencygroup:get_cgsnapshot": "",
    "consistencygroup:get_all_cgsnapshots": "",
}















 

After making the above changes, make sure to restart all Cinder services for the changes to take effect.

 

Creating & Managing Consistency Groups

The first step in using CGs is to create one. As I was mentioning earlier, a CG must be created before volumes can be added to it, it is not possible to create a CG and add volumes at the same time in the one command. Additionally, the only way to access CG functionality it through the Cinder CLI, any other CLIs, even those that interact with block storage, do not support CGs.

 

Creating a Consistency Group

To create a Cinder CG use the cinder consisgroup-create command:

Command format:

$ cinder consisgroup-create [--name <name>] [--description <description>] [--availability-zone <availability-zone>] <volume-types>

Command example:

$ cinder consisgroup-create --name vmax_demo_cg --availability-zone nova VMAX_ISCSI_DIAMOND

CgCreate.PNG.png


Show Consistency Group details

To check what CGs are currently present in your environment, use the cinder consisgroup-list and use the cinder consisgroup-list <consistencygroup> command to show details of a given CG.


CgListShow.PNG.png


Adding existing volumes to a Consistency Group

With the CG now created and available for use, we can add volumes to it. Managing existing CGs is possible through the cinder consisgroup-update command, using it allows us to add & remove volumes, change the name of the CG, or add a description.  For now we will just stick with adding a volume.

Command format:

$ cinder consisgroup-update --add-volumes <uuid1,uuid2,...>] <consistencygroup>

Command example:

$ cinder consisgroup-update --add-volumes \

07a23299-a10a-4bfb-bd88-736095564bed,6a6040f2-4724-4425-bbff-124455a4b0f9,\

6d2fd0ca-63a2-44fb-a10e-8f77efa068bc,a53b723a-abbd-4752-b446-8f76a66781c2,\

e633bfce-f11d-4a6f-9e6e-8409c991ed0e vmax_demo_cg


CgAddVol.PNG.png

 

Update Consistency Group

Unfortunately, once volumes have been added to a CG there is no way to determine exactly what volumes exist within it. When the command cinder consisgroup-show <consistencygroup> is used, only the summary information is provided which does not include a list of volumes. To get around this oversight, I recommend setting a description to the CG so that you can tell from there what volumes are present.

Command format:

$ cinder consisgroup-update --description <description> <consistencygroup>

Command example:

$ cinder consisgroup-update --description \

"Volumes in CG: boot1, boot2, vmax_vol_1, vmax_vol_2, vmax_vol_3" vmax_demo_cg


CgUpdateDescription.PNG.png

 

Adding new volumes to a Consistency Group

Another way to add a volume to a CG is to create a volume within a CG, this is very similar to how a standard volume is created the only difference being the inclusion of the --consis-group parameter within the cinder create command.

Command format:

$ cinder create [--consisgroup-id <consistencygroup-id>] [--volume-type <volume-type>]

[--availability-zone <availability-zone>] [--name <name>] [<size>]

Command example:

$ cinder create --consisgroup-id dffc8eae-195e-41c7-ae4c-27f5cd38ab97 \

--volume-type VMAX_ISCSI_DIAMOND --availability-zone nova --name new_volume_1 1


CgCreateVol.PNG.png


Removing volumes from a Consistency Group

Removing volumes from a CG is handled in the same way as adding volumes, the only difference being the use of --remove-volumes instead of --add-volumes. Although not included in the example below, if you are removing volumes and you have added a description with a list of volumes to the CG, you can update the description at the same time to reflect the removal of the volumes.

Command format:

$ cinder consisgroup-update [--remove-volumes <uuid3,uuid4,...>] <consistencygroup>

Command example:

$ cinder consisgroup-update --remove-volumes \

07a23299-a10a-4bfb-bd88-736095564bed,6a6040f2-4724-4425-bbff-124455a4b0f9,\

6d2fd0ca-63a2-44fb-a10e-8f77efa068bc,a53b723a-abbd-4752-b446-8f76a66781c2,\

b6fc62fb-c921-4aa0-b380-71cb6b850493,e633bfce-f11d-4a6f-9e6e-8409c991ed0e \

vmax_demo_cg


CgRemoveVol.PNG.png


Deleting a Consistency Group

Earlier in the article I mentioned that a CG cannot be deleted if certain attributes of the CG are not met. However, this deletion can be forced if your need to delete it without making it valid for a standard deletion. There is no need to outline the format of both scenarios here as they are almost identical, they only differ through the use of the --force flag, however I will include a screenshot of a deletion using this force flag.. First up is the standard deletion of a CG with no volumes within:

Command format:

$ cinder consisgroup-delete [--force] <consistencygroup> [<consistencygroup> ...]

Command example:

$ cinder consisgroup-delete vmax_demo_cg


CgDelete.PNG.png

 

If you cannot delete the CG and instead get an error message back, if the CG can be deleted using the delete-volumes (force) flag you will be told so. Run the same command again but this time including --force before the name of the CG(s):

 

CgDeleteForce.PNG.png

 

Create a copy of an existing Consistency Group

The last piece of functionality to cover for CGs is the ability to create a copy of an existing CG, or creating a CG from a source CG. To do this we use the cinder consisgroup-create-from-src command instead of specifying the source CG as an extra parameter in the standard CG creation command. The command is structured as follows:

Command format:

$ cinder consisgroup-create-from-src [--source-cg <source-cg>] [--name <name>]

[--description <description>]

Command example:

$ cinder consisgroup-create-from-src --source-cg dffc8eae-195e-41c7-ae4c-27f5cd38ab97 \

--name 'vmax_demo_cg_clone' --description 'new cg cloned from vmax_demo_cg'


CgCreateFromCG.PNG.png


Using Consistency Group Snapshots

With your CG setup and volumes added to it, we can start to use the CG to take consistent point-in-time snapshots for future use. CG snapshots behave in the same way as standard snapshots, the only difference being there is more of them. For every volume in your CG, you will have a corresponding snapshot when you create a snapshot of a CG. If there are 5 volumes in a CG when you snapshot it, you will have 5 snapshots appear in your environment afterwards. You will need to take this into consideration when determining your environment defaults also, if  you are doing things in multiples, it might be easier to hit the limits set in your environment quicker than you anticipated.

 

Creating a Consistency Group Snapshot

To create a CG snapshot, or even interact with CG snapshots for that matter, the CLI command vary slightly from the commands in the previous section. Now when we do anything with CG snapshots we use the command base cinder cgsnapshot instead of cinder consisgroup.  To create a snapshot of a CG we use the cinder cgsnapshot-create command:

Command format:

$ cinder cgsnapshot-create [--name <name>] [<consistencygroup> ...]

Command example:

$ cinder cgsnapshot-create --name vmax_cg_snap vmax_demo_cg


CgSnapshotCreate.PNG.png

 

Show Consistency Group Snapshot details

To list all available CG snapshots in your environment and show details of a specific CG snapshot we use the cinder cgsnapshot-list and cinder cgsnapshot-show <cgsnapshot> commands. You can see from the example below that the CG snapshot references the source CG from which it was created, in this case 'vmax_demo_cg' is the source of 'vmax_cg_snap'.


CgSnapshotShow.PNG.png

 

Create a Consistency Group from a Consistency Group Snapshot

If you have an existing CG snapshot in your environment, you can leverage that snapshot to create another CG. The process is almost identical to creating a CG from another source CG, this time --cgsnapshot <cgsnapshot> is used instead of --source-cg <source-cg>.

Command format:

$ cinder consisgroup-create-from-src [--cgsnapshot <cgsnapshot>] [--name <name>]

[--description <description>]

Command example:

$ cinder consisgroup-create-from-src --cgsnapshot 59ce55cb-206f-4557-a784-ce8437c05efd \

--name 'vmax_demo_cg_clone' --description 'new cg cloned from vmax_demo_cg'

CgCreateFromSnapshot.PNG.png

 

Deleting a Consistency Group Snapshot

Deleting a CG snapshot is carried out practically in the same way as deleting any other resource is handled using the CLI:

Command format:

$ cinder cgsnapshot-delete <cgsnapshot> [<cgsnapshot> ...]

Command example:

$ cinder cgsnapshot-delete vmax_cg_snap

 

CgSnapshotDelete.PNG.png

 

Troubleshooting issues with Consistency Groups

You may run in to some issues with CGs when using them in OpenStack, thankfully for us, troubleshooting these issues is fairly straightforward as there is not much configuration required to enable the feature. Most problems encountered with CGs are usually around attempting operations which are not permitted (outlined previously in this article but ill go through them again here).

 

If you are having any issues with CGs in OpenStack the first thing to check is the policy configuration. Check the correct config is present in /etc/cinder/policy.json and "group:nobody" is removed from each CG related policy. Once confirmed, restart all Cinder services to ensure all policy changes are applied, then test using CGs again.


If at this point you still cant get CGs to work, but you have confirmed the policies are correct and have restarted all Cinder services, you need to make sure that the operation you are trying to perform with a CG meets the following criteria to be CG valid operation:

  • Are all volumes to be added to CG using the same VMAX back end?
  • Are you attempting an operation that isn’t allowed when a volume is in a CG?
    • Volume migration
    • Volume retype
    • Volume deletion, CG has to be deleted as a whole with all the volumes
  • Are you trying to delete a CG with volumes in it? CGs can only be deleted if all the volumes in the CG are deleted along with it, or if the CG is empty
  • Are you trying to delete a volume snapshot which is part of a CG snapshot? A CG snapshot has to be deleted as a whole with all the volume snapshots

 

What's coming up in part 7 of 'VMAX & OpenStack Ocata: An Inside Look'...

In the next in the series of 'VMAX & OpenStack Ocata: An inside Look' we will be looking at volume replication. Volume replication will allow us to take our data protection considerations even further that what is available using snapshots and backups. See you then!