Find Communities by: Category | Product

Welcome back! In our series we have taken an in-depth look into all of the various features supported by VMAX in OpenStack Ocata. We have looked at everything from installation and configuration, snapshots to volume replication. Today we are going to look at live migration in OpenStack when a VMAX is the storage back end. Whilst no additional configuration is required from a VMAX perspective, and primarily it is a Nova compute operation, it is an interesting topic and is worth taking a better look at. I will not go into detail about configuring your environment for Live Migration as it differs from environment to environment, but I do recommend you take a look at the official OpenStack documentation for configuring live migrations.


What is Live Migration?

Live migration is the process whereby a virtual machine instance is moved to a different OpenStack compute host whilst the instance continues running causing no disruption. A typical use-case for migrating an instance is for planned maintenance on a compute server, but it may be also for resource load distribution when many virtual machines are running on a compute server.






Imagine a scenario using the illustration above. Hosts A, B & C are all running virtual machines, controlled by a cloud administrator (client). Host B over the course of a working day starts see resource contention and as a consequence the VMs running on the host start to slow down in terms of RAM and CPU allocation. To alleviate the resource contention, the cloud admin decides to move instances to host A & C and spread the load more evenly across their environment. Instead of shutting down the instances on host B, the cloud admin can migrate the contested instances to host A & C using live migration so there is no impact to the end user and no-one is aware there has been a change in the VM host location. If anything, the end-user will notice their services are running better than they did a few minuted before.


There are a few different methods of migrating your instances from one compute host to another in OpenStack, dependent on the requirements of the migration and the storage type used for the instance to be migrated. The different types of migration are:

  • Cold migration (non-live migration or simply migration) - In cold migrations the running instance is shut down and moved to another compute server and restarted when the transfer is complete. As the instance is shut down in this scenario, there is a disruption to any services it provided.
  • Live migration - During live migrations the instance keeps running throughout the entire process, which is especially useful if it is not possible to shut down the instance or have a disruption to running services. Live migrations can be classified further by how it handles the storage back end of the running instance:
    • Shared storage-based live migration - The instance has ephemeral disks that are located on storage shared between the source and destination hosts.
    • Block live migration - The instance has ephemeral disks that are not shared between the source and destination compute hosts.
    • Volume-backed live migration - The instance uses a volume rather than ephemeral disks for storage. The volume is located on storage shared between the source and destination hosts. This is the migration type used by OpenStack when VMAX is used as the storage back end of the instance.


Shared-storage and volume-backed live migrations are distinctly more advantageous than block migrations as they do not require disks to be copied between hosts. Block live migrations take considerably more time to complete and put additional load on the network.


How does Live Migration work?

When a live migration process is requested to move an instance to another compute node there are a number of steps required. Gaining a better insight into these allows for better understanding of the process itself and help determine if a live migration is going to be successful or not. The most important aspects of these checks will determine:

  • If the target host has enough resources available to support the instance when migration is complete
  • If the source and target host use shared storage to support moving the volume-backed instance between the hosts seamlessly


The steps carried out during each and every OpenStack live migration process are as follows:

  1. Pre-migration - Check memory, CPU, and storage availability on target host
  2. Reservation - Reserve the required instance resources on the target host and mount required disks
  3. Pre-copy - Copy instance memory from the source to target host
  4. Stop and copy - Pause instance on source host and copy dirty memory and CPU state
  5. Committent - Instance confirmed running on the target hosts
  6. Clean-up - Unmount disks to remove old connections to source host from storage, delete instance on source host


Performing Live Migrations

Migrating an instance from one compute host to another using OpenStack is very straightforward. Once you know the name of your target host that is pretty much it in terms of what you need to do in advance of the operation!  The command nova list will show you the Nova compute hosts in your environment, it is up to you to make sure that the target host is accessible from the source host and back end storage device.


Nova list show.PNG.png


Note: There has been output from the nova show command removed as it is not relevant to the LM process, all we are interested in here is the host on which the instance resides. In this case the instance resides on a host ending in vm20.


Once you have selected your target host, issue the command to migrate your chosen instance to it.


Command Format:

$ nova live-migration <instance_id> <target_host_id>

Command Example:

$ nova live-migration 659012c5-bfe0-4264-86a3-18cecdf2bf5b test_host_vm10


Once the command has been issued you will not get any notification back that it has been successful when performing the operation from the CLI. To check if the live migration process was successful, the easiest thing to do is to check the status of the migrated instance instead, if the process was successful the instance should list the target host in the migration process as its current host.

Command Format:

$ nova show <instance_id>

Command Example:

$ nova show 659012c5-bfe0-4264-86a3-18cecdf2bf5b

nova live-migrate.PNG.png


From the example above, the 'cirros_vm' instance was live migrated to from vm20 to vm10 seamlessly without any interruption to connectivity or services.

Live Migration Gotchas

When performing live migrations in your environment there are a few important things which need to be taken into consideration.


There is very little (or at times non-existent) feedback on the live migration process to the user, if something goes wrong it is not always instantly obvious. An instance will just not move to the target host and keep running as if there was never a live migration process started in the first place. A good place to go look for potential issues with live migration is in your Nova & Cinder logs with debug mode enabled in their respective .conf files.


Depending on your environment setup, infrastructure capabilities, and a number of other environment variables at any given time, there may be on occasion a time where your live migration process times out. If this is the case, increase the value of 'rpc_response_timeout' in all cinder and nova configuration files (cinder.conf & nova.conf) across all nodes until you find a value which is best suited to your environment. When you change this value (or any other value for that matter in any OpenStack configuration file) make sure you restart all respective services for the changes to take effect.


Your environment may have HAProxy configured to assist with load-balancing and high-availability nodes. If so, you will need to ensure that time settings in HAProxy do not interfere with operations in OpenStack. If you see errors in the logs which appear to be gateway timeouts, this usually points towards HAProxy being the culprit. To resolve this, increase the value of 'timeout server' in '/etc/haproxy/haproxy.cfg', by default it is set to 60s, you could set this to the same value of 'rpc_response_timeout' in the previous paragraph. Once you have changed this value, make sure to restart the HAProxy service using the command 'crm resource restart p_haproxy' (confirmed on Ubuntu OS only).


It might seem obvious at this stage, but as I mentioned earlier, you need to make sure that your target host has sufficient resources available to support the instance to be migrated to it. If there are no resources available, or insufficient resources, the process will be cancelled during the pre-migration checks.  To confirm you have enough available resources for your target host, use the command 'nova hypervisor-show <hostname>' to see what is available for use in a given host.


Hypervisor show host.PNG.png


Next time on 'An Inside Look'....

Next time we return it won't be to look at Cinder and block storage using VMAX, but instead the newly supported Manila! Manila is the file share service project for OpenStack. Manila provides the management of file shares for example, NFS and CIFS as a core service to OpenStack. Come back again to go over the ins and outs of Manila and VMAX, including an overview, configuration and usage, see you then!

The VMAX drivers for OpenStack Cinder support the ability to migrate volumes between multiple VMAX back-ends (volume types). Migrating a volume in this way will move the volume from its current back-end to a new one. As noted in the official OpenStack documentation, 'this is an administrator function, and can be used for functions including storage evacuation (for maintenance or decommissioning), or manual optimizations (for example, performance, reliability, or cost)'.

This feature is supported by VMAX-3 series arrays and gives the user the option to switch volumes between various Service Level & Workload combinations. For example, if a VMAX backed volume in OpenStack has a Diamond & OLTP combination, it can be migrated to a volume type with a Silver & OLTP combination.

There are 3 workflows for volume migration in OpenStack:

  1. If the VMAX is able to, it will migrate the volume between volume types on its own. If not, it will follow one of the two following workflows:
    1. If the volume is not attached to an instance, Cinder will create a volume using the target volume type and copies the data from the source volume to the new volume
    2. If the volume is attached to an instance, Cinder will create a volume using the target volume type but this time calls Nova (the Compute Service) to copy the data from the source volume to the new volume


Configuring your environment for volume retype

To support volume retype in your VMAX & OpenStack environment, there are some changes which need to be made to the Cinder configuration file and the back-end XML configuration file.


1.  Add the parameter multi_pool_support to the configuration group in the /etc/cinder/cinder.conf file and set it to True


volume_driver = cinder.volume.drivers.dell_emc.vmax.fc.EMCVMAXFCDriver
cinder_emc_config_file = /etc/cinder/cinder_emc_config_CONF_GROUP_FC.xml
volume_backend_name = VMAX_backend
multi_pool_support = True


The next step is to configure a single back-end per VMAX Storage Resource Pool (SRP). This is different from the regular configuration where one back-end is configured per service level/workload combination.

2. Create the file /etc/cinder/cinder_emc_config_CONF_GROUP_VMAX.xml (as linked to by the cinder_emc_config_file parameter in step 1) and add the following lines. The difference from the previous XML file created during initial setup is the removal of the <ServiceLevel> and <Workload> tags, with retype these values are no longer required here. Note: The XML filename is still in the naming convention of cinder_emc_config_[CONF_GROUP].xml


<?xml version = "1.0" encoding = "UTF-8" ?>


3. With cinder.conf updated and the new VMAX back-end config file created, restart the Cinder volume service for the changes to take effect:

Ubuntu: $ sudo service cinder-volume restart

RedHat/CentOS/SLES/openSUSE: $ sudo systemctl restart cinder-volume

4. Now the VMAX back-end is configured for volume retype, we can proceed with creating our new retype supported volume types. The example below demonstrates creating a volume type for the Diamond Service Level and OLTP workload

$ openstack volume type create VMAX_FC_DIAMOND_OLTP

$ openstack volume type set --property volume_backend_name=FC_backend VMAX_FC_DIAMOND_OLTP

$ openstack volume type set --property pool_name=Diamond+OLTP+SRP_1+111111111111

Repeat step 4 to create as many volume types with the various Service Level/Workload combinations that you need for your environment to provision volumes with.  The additional property set in the last command, pool_name, is where you define your Service Level/Workload combination. It uses the following format: <ServiceLevel>+<Workload>+<SRP>+<Array ID>


5. Once you have created all of the volume types required, you can check they were added successfully with the command:

$ cinder type-list


CLI Pool.PNG.png


Retyping your volume from one volume type to another

For migrating a volume from one Service Level or Workload combination to another, use volume retype with the migration-policy set to on-demand. The target volume type should have the same volume_backend_name configured and should have the desired pool_name to which you are trying to retype to.

Command Structure

$ cinder retype --migration-policy on-demand <volume> <volume-type>

Command Example

$ cinder retype --migration-policy on-demand vmax_vol1 VMAX_SILVER_OLTP




The example above shows a volume which has just been retyped to a new volume type. You will notice that the host attribute still retains the name of the back end used which all the service level/workload pools are associated with, and the volume type uses multi-pool now so multiple volume types can be created to all use the same back end. This differs from previous examples as one back end used to be associated only with one volume type in a 1-to-1 relationship, this relationship using multi-pool allows us to use a 1-to-many relationship for back ends to volume types.


Troubleshooting retyping VMAX volumes

There are only a number of checks which need to be carried out if you find that VMAX volume retype is not working as expected:

  • Is the back end stanza in cinder.conf for your VMAX backend correctly configured and with the additional 'multi_pool_support = True' parameter included?
  • Did you restart all Cinder services after making the change in the previous point?
  • Is the XML configuration file associated with the back end in cinder.conf correct? Did you remove the service level from it?
  • Are the OpenStack volume types correctly set up? Is the pool_name parameter correct?


Coming up next time on 'VMAX & Ocata: An Inside Look'...

Next time around we are going to take a look at live migration in OpenStack, where running instances using VMAX block storage volumes are migrated from one compute host to another without any breaks to operations or services, pretty nifty! As always, if you have any comments, recommendations, or questions, leave them in the comments section below or send me a private message! See you next time!

Welcome back! Today it's part 7 of 'An Inside Look' and we are going to be looking at volume replication using SRDF with VMAX & OpenStack Ocata. Before we get stuck in, if you would like to revisit last week's article on Consistency Groups using VMAX & OpenStack click here. This week's article will get fairly heavy and technical in places, so I will do my best to break it down into logical, easy to digest chunks of information so as not to overwhelm you too much!


What is SRDF?

Symmetrix Remote Data Facility (SRDF) is the gold standard for providing disaster recovery (DR) and data mobility solutions for VMAX arrays.  SRDF provides external protection of data at a remote site over synchronous and asynchronous distances and also in an Active/Active model.  SRDF has been around a long time continuously improving and protecting the worlds most mission critical systems providing disaster recover and disaster avoidance solutions. There is a lot of reading material out there if you are not familiar and the SRDF Family CLI User Guide goes through all this in detail so I won't attempt to cover off here, only just what is relevant to us using VMAX with OpenStack.  The latest 8.4 SRDF Family CLI User Guide guide can be found here.


VMAX, SRDF & OpenStack

The VMAX drivers for OpenStack use synchronous SRDF (SRDF/S) as the replication type, at present VMAX Cinder drivers can support a single replication target per back-end, concurrent SRDF or cascaded SRDF are unsupported. Also, in order for replication to work with VMAX & OpenStack you will need the correct licenses added to your VMAX, namely the Advanced Suite and Remote Replication Suite (more information can be found in the introductory article of this series).


What is Synchronous SRDF (SRDF/S)

SRDF synchronous (SRDF/S) mode maintains a real-time copy at arrays generally located within 200 kilometres (dependent upon application workload, network latency, and block size). Writes from the production host are acknowledged from the local array when they are written to cache at the remote array creating a real time mirror of the primary devices.


SRDF disaster recovery solutions, including SRDF synchronous, traditionally use active, remote mirroring and dependent-write logic to create consistent copies of data. Dependent-write consistency ensures transactional consistency when the applications are restarted at the remote location.


An SRDF device is a logical device paired with another logical device that resides in a second array. The arrays are connected by SRDF links. R1 devices are the member of the device pair at the primary (production) site. R1 devices are generally read/write accessible to the host. R2 devices are the members of the device pair at the secondary (remote) site. During normal operations, host I/O writes to the R1 device are mirrored over the SRDF links to the R2 device.



SRDF Links.png


Traditionally, data on R2 devices are not available to the host while the SRDF relationship is active. In SRDF synchronous mode, an R2 device is typically in read only mode (write disabled) that allows a remote host to read from the R2 devices. In a typical open systems host environment, the production host has read/write access to the R1 device. A host connected to the R2 device has read only access to the R2 device. To access the R2 device of a traditional synchronous relationship, a manual failover command must be performed to write enable the R2 site to accept host writes.


If you would like to find out about this process in a lot finer detail, the SRDF CLI user guide linked earlier has great sections on full & incremental SRDF pairings and the failover procedure from a VMAX perspective. This info can be found in the section 'Basic SRDF Control Operations'. There is too much to include here in this article so I would recommend there as a good starting point to read up more on the topic.


Disaster Recovery Considerations with VMAX & OpenStack Environments

When preparing your environment for SRDF replication and DR there are a number of things which must be taken in to consideration if the situation arises where your local environment becomes inaccessible.

  • For full failover functionality, the source and target VMAX arrays must be discovered and managed by the same SMI-S/ECOM server, locally connected for example. This SMI-S/ ECOM server cannot be embedded - it can be installed on a physical server or a VM hosted by an ESX server only.
  • With both arrays being managed by the one SMI-S server, it is the cloud storage administrators responsibility to account for a DR scenario where the management (SMI-S) server goes down as well as the primary array. In that event, the details and credentials of a back-up SMI-S server can be passed in to the XML file, and the VMAX cinder driver can be rebooted. It would be advisable to have the SMI-S server at a third location (separate from both arrays) if possible.
  • If the source and target arrays are not managed by the same management server (that is, the target array is remotely connected to server), in the event of a full disaster scenario (for example, the primary array is completely lost and all connectivity to it is gone), the SMI-S server would no longer be able to contact the target array. In this scenario, the volumes would be automatically failed over to the target array, but administrator intervention would be required to either; configure the target (remote) array as local to the current SMI-S server, or enter the details to the XML file of a second SMI-S server, which is locally connected to the target array, and restart the cinder volume service.


Configuring your Environment for Volume Replication

The first step in configuring your OpenStack environment with volume replication abilities is to setup the associated SRDF group that will be used by OpenStack to replicate volumes between source and target VMAXs. To create an SRDF group:

  1. Select your source VMAX in Unisphere and navigate to 'Data Protection' > 'Create SRDF Group'
  2. Select your communication protocol type and give the group a label (name)
  3. It is possible to configure multiple SRDF groups on the same source and target, so to differentiate between each a group number is assigned. Select an unused group number and enter it into the dialogue box
  4. Select the director ports on the primary VMAX, hold CTRL whilst selecting to select multiple director:port combinations
  5. In the remote section, click 'scan' to show all available VMAXs and pick your target
  6. Enter your SRDF group number and select the target director ports
  7. Click 'OK' to create the SRDF group



Once the SRDF group has been created that is it, there is no more configuration required on the VMAX side of things. The VMAX Cinder drivers will handle the rest of the setup on the first volume replication task.


With your SRDF group configured, the next step is create a replication enabled volume type in OpenStack. As with any volume type, creating them involves configuring a back end stanza in /etc/cinder/cinder.conf for it and an associated XML configuration file. When configuring your replication back end stanza in cinder.conf the following rules apply:

  • The source array is defined in the XML configuration file, this file uses the same format as any other VMAX back end
  • The target array is defined in the replication_device parameter
  • Only one target array can be defined per back end
  • If SSL is required it will also need to be configured in the back end stanza, along with any other back end specific configurations as standard


enabled_backends = VMAX_ISCSI_REPLICATION
volume_driver = cinder.volume.drivers.dell_emc.vmax.fc.EMCVMAXFCDriver
cinder_emc_config_file = /etc/cinder/cinder_emc_config_VMAX_ISCSI_REPLICATION.xml
volume_backend_name = VMAX_ISCSI_REPLICATION
replication_device = target_device_id:000197811111, remote_port_group:os-failover-pg,\
remote_pool:SRP_1, rdf_group_label: 28_11_07, allow_extend:False


Note: The replication_device parameter must be defined all in one line, it is only split across two lines in the example above for ease of reading.


  • target_device_id is a unique VMAX array serial number of the target array. For full failover functionality, the source and target VMAX arrays must be discovered and managed by the same SMI-S/ECOM server. That is, locally connected. Follow the instructions in the SMI-S release notes.
  • remote_port_group is the name of a VMAX port group that has been pre-configured to expose volumes managed by this backend in the event of a failover. Make sure that this port group contains either all FC or all iSCSI port groups (for a given back end), as appropriate for the configured driver (iSCSI or FC).
  • remote_pool is the unique pool name for the given target array.
  • rdf_group_label is the name of a VMAX SRDF group (Synchronous) that has been pre-configured between the source and target arrays.
  • allow_extend is a flag for allowing the extension of replicated volumes. To extend a volume in an SRDF relationship, this relationship must first be broken, both the source and target volumes are then independently extended, and then the replication relationship is re-established. As the SRDF link must be severed, due caution should be exercised when performing this operation. If not explicitly set, this flag defaults to False.


The last step is to create a replication-enabled volume type. Once the replication_device parameter has been entered in the VMAX backend entry in the cinder.conf, a corresponding volume type needs to be created replication_enabled property set.

$ openstack volume type create VMAX_FC_REPLICATION

$ openstack volume type set --property volume_backend_name=Replication_backend VMAX_FC_REPLICATION

$ openstack volume type set --property replication_enabled='<is> True' VMAX_FC_REPLICATION




With the volume type created for use in OpenStack that is all the steps complete, any volumes created using this new replication enabled volume type will be automatically replicated between your source and target VMAXs. If you look in Unisphere at the SRDF group you created in the first step, all volumes created using the replication volume type will show the source volume and the replicated target volume.




An attempt will be made to create a storage group on the target array with the same service level (SL) and workload combination as the primary. However, if this combination is unavailable on the target (for example, in a situation where the source array is a Hybrid, the target array is an All Flash, and an All Flash incompatible SL like Bronze is configured), no SL will be applied.


Volume replication interoperability with other features

Most features are supported except for the following:

  • There is no OpenStack Consistency Group support for replication-enabled VMAX volumes.
  • Storage-assisted retype operations on replication-enabled VMAX volumes (moving from a non-replicated type to a replicated-type and vice-versa. Moving to another SLO/workload combination, for example) are not supported.
  • The image volume cache functionality is supported (enabled by setting image_volume_cache_enabled = True), but one of two actions must be taken when creating the cached volume:  This is because the initial boot volume is created at the minimum required size for the requested image, and then extended to the user specified size.
    • The first boot volume created on a back end (which will trigger the cached volume to be created) should be the smallest necessary size. For example, if the minimum size disk to hold an image is 5GB, create the first boot volume as 5GB.
    • Alternatively, ensure that the allow_extend option in the replication_device parameter is set to True.


Failover host

In the event of a disaster, or where there is required downtime, upgrade of the primary array for example, the administrator can issue the failover host command to failover to the configured target. When making these changes to/from failover backends using the CLI you will get no confirmation back that the switch was successful, however, if you monitor the Cinder volume logs you will see the change being carried out there and the remote array becoming the source and vice-versa.

$ cinder failover-host cinder_host@VMAX_FC_REPLICATION



If the primary array becomes available again, you can initiate a fail back using the same command and specifying --backend_id default:

$ cinder failover-host cinder_host@VMAX_FC_REPLICATION --backend_id default 



Troubleshooting issues with VMAX Replication in OpenStack

As with the potential issues faced with features discussed up until this point, the most likely place to look when something goes wrong is in the configuration and setup of replication for use in OpenStack. There are a few moving parts which must come together in order for replication to work as intended, I will not cover the ECOM related considerations surrounding DR as they are discussed earlier in this article.


If you are having issues with a VMAX replication enabled volume type in OpenStack check the following:

  • Is the backend stanza in cinder.conf configured correctly?
  • Is the OpenStack volume type correctly configured and extra spec for replication added?
  • Is the RDF group correctly configured in Unisphere or SE?
  • Are you trying an unsupported operation?
    • There is no CG support for replicated volumes
    • There is no volume retype support


Coming up next time...

Next time in 'VMAX & Openstack Ocata: An Inside Look' we will be looking at storage assisted volume migration, or othewise known as volume retype!  As always thanks for reading and if you have any comments, suggestions, document fixes, or questions, feel free to contact me directly or via the comments section below!

Welcome back! In the previous article we looked at managing and unmanaging VMAX volumes in OpenStack, if you would like to read that article again click here. This time around it is part 6 of VMAX & OpenStack Ocata: An Inside Look.


Today we will be looking at Consistency Groups (CGs) in OpenStack, not to be confused with VMAX Consistency Groups which is an SRDF feature. So far in our adventures into OpenStack & VMAX functionality everything has happened at the volume level. With Consistency Groups, the functionality of snapshots is taken further by taking a group of volumes and creating a consistent snapshot of the volumes at that point in time to ensure data consistency. Consistency groups are needed to group volumes together for the purpose of data protection (snapshots, backups) and remote replication for disaster recovery.


Imagine this scenario, a mission critical system has two attached block storage devices each having their own database, each relying on data from the other.If you wanted to create a snapshot of this system using standard snapshots you would have to manually create snapshots of each volume.


The biggest problem with this is that due to the manual nature of this process, the snapshot of each volume will not leave the volume snapshots in a consistent state. For example, in the time that it takes to send the command to snapshot volume 1, data will have been written to the database in volume 2. When it comes to taking a snapshot of volume 2, the data within does not correlate with the exact point in time as the snapshot of volume 1. In a restore process, this could have the knock on effect of data being inconsistent or missing, data written to volume 2 may not have the correct corresponding data in volume 1 as it had been written just fractionally after the snapshot of volume 1 was created.


This is where CGs come in. By adding both volumes to a CG, when a CG snapshot is taken of that group the snapshots represent the state of both volumes at that exact point in time, keeping your volume data consistent across both in the event of restoring the volumes or creating new volumes.


Consistency Group Behaviour & Required Setup

At present, using CGs in OpenStack is achievable only via the Cinder CLI, so there won't be an accompanying video as all of the features of the CLI will be demonstrated using screenshots and command examples. Before working with CGs, there are a few things to take into consideration about the behaviour of the feature:

  • A CG can support more than one volume type, the Cinder scheduler is responsible for finding a back end that can support all volume types
  • A CG can only contain volumes hosted by the same back end
  • A CG is empty when created, volumes are added later
  • The following operations are not allowed if a volume is in a CG:
    • Volume migration
    • Volume retype
    • Volume deletion (CG has to be deleted as a whole with all the volumes)
  • CGs can only be deleted if all the volumes in the CG are deleted along with it, or if the CG is empty
  • Volume snapshot deletion is not allowed if the volume snapshot is in a CG snapshot (CG snapshot has to be deleted as a whole with all the volume snapshots)


Before using CGs in OpenStack, you need to change the policies for the CG APIs in the /etc/cinder/policy.json file. By default, these CG APIs are disabled, so make sure to enable them before running any CG operations. To enable CG functionality, remove 'group:nobody' from the policy entries for CGs:


    "consistencygroup:create": "group:nobody"
    "consistencygroup:delete": "group:nobody",
    "consistencygroup:update": "group:nobody",
    "consistencygroup:get": "group:nobody",
    "consistencygroup:get_all": "group:nobody",

    "consistencygroup:create_cgsnapshot" : "group:nobody",
    "consistencygroup:delete_cgsnapshot": "group:nobody",
    "consistencygroup:get_cgsnapshot": "group:nobody",
    "consistencygroup:get_all_cgsnapshots": "group:nobody",


    "consistencygroup:create": "",
    "consistencygroup:delete": "",
    "consistencygroup:update": "",
    "consistencygroup:get": "",
    "consistencygroup:get_all": "",

    "consistencygroup:create_cgsnapshot" : "",
    "consistencygroup:delete_cgsnapshot": "",
    "consistencygroup:get_cgsnapshot": "",
    "consistencygroup:get_all_cgsnapshots": "",


After making the above changes, make sure to restart all Cinder services for the changes to take effect.


Creating & Managing Consistency Groups

The first step in using CGs is to create one. As I was mentioning earlier, a CG must be created before volumes can be added to it, it is not possible to create a CG and add volumes at the same time in the one command. Additionally, the only way to access CG functionality it through the Cinder CLI, any other CLIs, even those that interact with block storage, do not support CGs.


Creating a Consistency Group

To create a Cinder CG use the cinder consisgroup-create command:

Command format:

$ cinder consisgroup-create [--name <name>] [--description <description>] [--availability-zone <availability-zone>] <volume-types>

Command example:

$ cinder consisgroup-create --name vmax_demo_cg --availability-zone nova VMAX_ISCSI_DIAMOND


Show Consistency Group details

To check what CGs are currently present in your environment, use the cinder consisgroup-list and use the cinder consisgroup-list <consistencygroup> command to show details of a given CG.


Adding existing volumes to a Consistency Group

With the CG now created and available for use, we can add volumes to it. Managing existing CGs is possible through the cinder consisgroup-update command, using it allows us to add & remove volumes, change the name of the CG, or add a description.  For now we will just stick with adding a volume.

Command format:

$ cinder consisgroup-update --add-volumes <uuid1,uuid2,...>] <consistencygroup>

Command example:

$ cinder consisgroup-update --add-volumes \



e633bfce-f11d-4a6f-9e6e-8409c991ed0e vmax_demo_cg



Update Consistency Group

Unfortunately, once volumes have been added to a CG there is no way to determine exactly what volumes exist within it. When the command cinder consisgroup-show <consistencygroup> is used, only the summary information is provided which does not include a list of volumes. To get around this oversight, I recommend setting a description to the CG so that you can tell from there what volumes are present.

Command format:

$ cinder consisgroup-update --description <description> <consistencygroup>

Command example:

$ cinder consisgroup-update --description \

"Volumes in CG: boot1, boot2, vmax_vol_1, vmax_vol_2, vmax_vol_3" vmax_demo_cg



Adding new volumes to a Consistency Group

Another way to add a volume to a CG is to create a volume within a CG, this is very similar to how a standard volume is created the only difference being the inclusion of the --consis-group parameter within the cinder create command.

Command format:

$ cinder create [--consisgroup-id <consistencygroup-id>] [--volume-type <volume-type>]

[--availability-zone <availability-zone>] [--name <name>] [<size>]

Command example:

$ cinder create --consisgroup-id dffc8eae-195e-41c7-ae4c-27f5cd38ab97 \

--volume-type VMAX_ISCSI_DIAMOND --availability-zone nova --name new_volume_1 1


Removing volumes from a Consistency Group

Removing volumes from a CG is handled in the same way as adding volumes, the only difference being the use of --remove-volumes instead of --add-volumes. Although not included in the example below, if you are removing volumes and you have added a description with a list of volumes to the CG, you can update the description at the same time to reflect the removal of the volumes.

Command format:

$ cinder consisgroup-update [--remove-volumes <uuid3,uuid4,...>] <consistencygroup>

Command example:

$ cinder consisgroup-update --remove-volumes \



b6fc62fb-c921-4aa0-b380-71cb6b850493,e633bfce-f11d-4a6f-9e6e-8409c991ed0e \



Deleting a Consistency Group

Earlier in the article I mentioned that a CG cannot be deleted if certain attributes of the CG are not met. However, this deletion can be forced if your need to delete it without making it valid for a standard deletion. There is no need to outline the format of both scenarios here as they are almost identical, they only differ through the use of the --force flag, however I will include a screenshot of a deletion using this force flag.. First up is the standard deletion of a CG with no volumes within:

Command format:

$ cinder consisgroup-delete [--force] <consistencygroup> [<consistencygroup> ...]

Command example:

$ cinder consisgroup-delete vmax_demo_cg



If you cannot delete the CG and instead get an error message back, if the CG can be deleted using the delete-volumes (force) flag you will be told so. Run the same command again but this time including --force before the name of the CG(s):




Create a copy of an existing Consistency Group

The last piece of functionality to cover for CGs is the ability to create a copy of an existing CG, or creating a CG from a source CG. To do this we use the cinder consisgroup-create-from-src command instead of specifying the source CG as an extra parameter in the standard CG creation command. The command is structured as follows:

Command format:

$ cinder consisgroup-create-from-src [--source-cg <source-cg>] [--name <name>]

[--description <description>]

Command example:

$ cinder consisgroup-create-from-src --source-cg dffc8eae-195e-41c7-ae4c-27f5cd38ab97 \

--name 'vmax_demo_cg_clone' --description 'new cg cloned from vmax_demo_cg'


Using Consistency Group Snapshots

With your CG setup and volumes added to it, we can start to use the CG to take consistent point-in-time snapshots for future use. CG snapshots behave in the same way as standard snapshots, the only difference being there is more of them. For every volume in your CG, you will have a corresponding snapshot when you create a snapshot of a CG. If there are 5 volumes in a CG when you snapshot it, you will have 5 snapshots appear in your environment afterwards. You will need to take this into consideration when determining your environment defaults also, if  you are doing things in multiples, it might be easier to hit the limits set in your environment quicker than you anticipated.


Creating a Consistency Group Snapshot

To create a CG snapshot, or even interact with CG snapshots for that matter, the CLI command vary slightly from the commands in the previous section. Now when we do anything with CG snapshots we use the command base cinder cgsnapshot instead of cinder consisgroup.  To create a snapshot of a CG we use the cinder cgsnapshot-create command:

Command format:

$ cinder cgsnapshot-create [--name <name>] [<consistencygroup> ...]

Command example:

$ cinder cgsnapshot-create --name vmax_cg_snap vmax_demo_cg



Show Consistency Group Snapshot details

To list all available CG snapshots in your environment and show details of a specific CG snapshot we use the cinder cgsnapshot-list and cinder cgsnapshot-show <cgsnapshot> commands. You can see from the example below that the CG snapshot references the source CG from which it was created, in this case 'vmax_demo_cg' is the source of 'vmax_cg_snap'.



Create a Consistency Group from a Consistency Group Snapshot

If you have an existing CG snapshot in your environment, you can leverage that snapshot to create another CG. The process is almost identical to creating a CG from another source CG, this time --cgsnapshot <cgsnapshot> is used instead of --source-cg <source-cg>.

Command format:

$ cinder consisgroup-create-from-src [--cgsnapshot <cgsnapshot>] [--name <name>]

[--description <description>]

Command example:

$ cinder consisgroup-create-from-src --cgsnapshot 59ce55cb-206f-4557-a784-ce8437c05efd \

--name 'vmax_demo_cg_clone' --description 'new cg cloned from vmax_demo_cg'



Deleting a Consistency Group Snapshot

Deleting a CG snapshot is carried out practically in the same way as deleting any other resource is handled using the CLI:

Command format:

$ cinder cgsnapshot-delete <cgsnapshot> [<cgsnapshot> ...]

Command example:

$ cinder cgsnapshot-delete vmax_cg_snap




Troubleshooting issues with Consistency Groups

You may run in to some issues with CGs when using them in OpenStack, thankfully for us, troubleshooting these issues is fairly straightforward as there is not much configuration required to enable the feature. Most problems encountered with CGs are usually around attempting operations which are not permitted (outlined previously in this article but ill go through them again here).


If you are having any issues with CGs in OpenStack the first thing to check is the policy configuration. Check the correct config is present in /etc/cinder/policy.json and "group:nobody" is removed from each CG related policy. Once confirmed, restart all Cinder services to ensure all policy changes are applied, then test using CGs again.

If at this point you still cant get CGs to work, but you have confirmed the policies are correct and have restarted all Cinder services, you need to make sure that the operation you are trying to perform with a CG meets the following criteria to be CG valid operation:

  • Are all volumes to be added to CG using the same VMAX back end?
  • Are you attempting an operation that isn’t allowed when a volume is in a CG?
    • Volume migration
    • Volume retype
    • Volume deletion, CG has to be deleted as a whole with all the volumes
  • Are you trying to delete a CG with volumes in it? CGs can only be deleted if all the volumes in the CG are deleted along with it, or if the CG is empty
  • Are you trying to delete a volume snapshot which is part of a CG snapshot? A CG snapshot has to be deleted as a whole with all the volume snapshots


What's coming up in part 7 of 'VMAX & OpenStack Ocata: An Inside Look'...

In the next in the series of 'VMAX & OpenStack Ocata: An inside Look' we will be looking at volume replication. Volume replication will allow us to take our data protection considerations even further that what is available using snapshots and backups. See you then!

Welcome back to part 5 of VMAX & OpenStack Ocata: An Inside Look! In my last post I covered snapshots and backups in Cinder using VMAX, if you would like to see that blog article again please click here. Today we will look at managing and unmanaging VMAX volumes in OpenStack Ocata.


Manage Volumes

Managing volumes in OpenStack is the process whereby a volume which exists on the storage device is imported into OpenStack to be made available for use in the OpenStack environment.  For a volume to be valid for managing into OpenStack, the following prerequisites must be met:

  • The volume exists in a Cinder managed pool
  • The volume is not part of a Masking View
  • The volume is not part of an SRDF relationship
  • The volume is configured as a TDEV (thin device)
  • The volume is set to FBA emulation


For a volume to exist in a Cinder managed pool, it must reside in in the same Storage Resource Pool (SRP) as the back end which is configured for use in OpenStack. For the purposes of this article, my configured back end will be using the Diamond service level with no workload type specified. Specifying the pool correctly can be entered manually as it follows the same format each time:

Pool format: <service_level>+<workload_type>+<srp>+<array_id>

Pool example 1: Diamond+DSS+SRP_1+111111111111

Pool example 2: Diamond+SRP_1+111111111111


service_level - The service level of the volume to be managed

workload - The workload type of the volume to be managed

srp - The Storage Resource Pool configured for use by the back end

array_id - The numerical VMAX ID


It is also possible to get the pool name using the Cinder CLI command cinder get-pools.  Running this command will return the available pools from your configured back ends. Each of the pools returned will also have the host name and back end name specified, these values are needed in the next step.


With your pool name defined you can now manage the volume into OpenStack, this is possible with the CLI command cinder manage. The bootable parameter is optional in the command, if the volume to be managed into OpenStack is not bootable leave this parameter out. OpenStack will also determine the size of the value when it is managed so there is no need to specify the volume size.

Command Format:

$ cinder manage --name <new_volume_name> --volume-type <vmax_vol_type> --availability-zone <av_zone> <--bootable> <host> <identifier>

Command Example:

$ cinder manage --name vmax_managed_volume --volume-type VMAX_ISCSI_DIAMOND

--availability-zone nova demo@VMAX_ISCSI_DIAMOND#Diamond+SRP_1+111111111111 031D8


After the above command has been run, the volume will be available for use in the same way as any other OpenStack VMAX volume.

Managing Volumes with Replication Enabled

Whilst it is not possible to manage volumes into OpenStack that are part of a SRDF relationship, it is possible to manage a volume into OpenStack and enable replication at the same time. This is done by having a replication enabled VMAX volume type (discussed in pt. 7 of this series), during the manage volume process you specify the replication volume type as the chosen volume type. Once managed, replication will be enabled for that volume.

Unmanaging a Volume

Unmanaging a volume is not the same as deleting a volume. When a volume is deleted from OpenStack, it is also deleted from the VMAX at the same time. Unmanaging a volume is the process whereby a volume is removed from OpenStack but it remains for further use on the VMAX. The volume can also be managed back into OpenStack at a later date using the process discussed in the previous section. Unmanaging volume is carried out using the cinder unmanage CLI command:

Command Format:

$ cinder unmanage <volume_name/volume_id>

Command Example:

$ cinder unmanage vmax_test_vol


Once unmanaged from OpenStack, the volume can still be retrieved using its device ID or OpenStack volume ID. Within Unisphere you will also notice that the 'OS-' prefix has been removed, this is another visual indication that the volume is no longer managed by OpenStack.


Troubleshooting issues with Managing & Unmanaging Volumes

When managing & unmanaging volumes there are a number of things which may contribute to the operation not being carried out successfully. Unmanaging volumes is the most straight forward of the two operations so I will cover that first. Basically, when a volume is unmanaged from OpenStack, all that is happening is the volume is removed from the Cinder database and the volume UUID changed to remove the OS- prefix. The only thing that can go wrong here from a VMAX perspective is if the connection to the ECOM server goes down during the operation to rename the volume. Check the status of your ECOM server and try the operation again, restarting the ECOM if necessary. The process of removing the volume from the Cinder database involves just an SQL command being run in the back ground to remove it from the required tables.

Managing a volume into OpenStack is trickier as there are more moving parts and prerequisites which must be met first in order for it to work successfully. The first thing to check is that your volume meets all of the following requirements:

  • The volume exists in a Cinder managed pool
  • The volume is not part of a Masking View
  • The volume is not part of an SRDF relationship
  • The volume is configured as a TDEV (thin device)
  • The volume is set to FBA emulation


If all of the above requirements are met then you need to make sure that the host is properly specified in the manage command itself. The host parameter takes the format:

Format: <hostname>@<volume_type>#<service_level>+<srp_id>+<array_id>

Example: demo@VMAX_ISCSI_DIAMOND#Diamond+SRP_1+111111111111


If the host is defined correctly, is the device identifier correct? You can double check this easily in by checking the value of the volume in Unisphere or through Solutions Enabler. With everything correctly configured and checked up until this point you should have no problems managing the volume in to OpenStack.


There may be some confusion over the replication enabled ability for VMAX volumes managed into OpenStack. What is meant here is that the volume being managed into OpenStack is not a part of an SRDF relationship before it is managed, but after being managed it is assigned an OpenStack volume type with replication enabled, so that newly managed volume gets added to an OpenStack replication volume type, meaning it is replicated from that point forward.


What's coming up in part 6 of 'VMAX & OpenStack Ocata: An Inside Look'...

In the next in the series of 'VMAX & OpenStack Ocata: An inside Look' we will be looking at OpenStack Consistency Groups. Not to be confused with VMAX Consistency Groups (which is an SRDF feature), OpenStack Consistency Groups allow snapshots of multiple volumes in the same consistency group to be taken at the same point-in-time to ensure data consistency. See you then!

Welcome back to VMAX & OpenStack Ocata: An Inside Look! Last time around we looked at basic operations in OpenStack using VMAX as the block-storage back end, if you would like to read that article again you can do so by clicking here. Topics included volume management, cloning a volume, and attaching or detaching a volume from a Nova compute instance.  This time we are going to look at Snapshots & Backups in OpenStack, following a similar format to the previous articles, after breaking down the functionality, I will give a demonstration using CLI commands and provide a video demonstration at the end using the Horizon dashboard.

Note: Cinder backups are not available through the Horizon dashboard in OpenStack Ocata, to make use of the Cinder backup service you must use CLI commands


What are Cinder Volume Snapshots?

A Cinder volume snapshot is a read-only, point-in time copy of a block-storage volume. Snapshots can be created from existing block-storage volumes that in the state 'available' or 'in-use', meaning that the snapshot can be created when it is attached to a running instance in OpenStack.  A snapshot in Cinder can be used as the source for a new Cinder volume, allowing for a quick and easy way to either restore an volume to a previous point-in-time, or used almost as an 'image' from which to provision new volumes from. Cinder volume snapshots are not to be confused with Nova instance snapshots.  Instance snapshots are not only the attached persistent volumes that live on the Virtual Machine but also the flat files that make up the virtual machine itself. So when you take a snapshot of the VM its the exact point in time of the OS, all of it's files etc.

What are Cinder Volume Backups?

A Cinder volume backup is an exact copy, a full clone, that can reside at the same site as its read/write source, or in a secondary site for disaster recovery solutions.  A Cinder volume backup can be viewed as all volume copies from the source storage array that are stored externally of the source storage array. A backup captures the state of its source at the time the backup is made, and its contents cannot change.  The Cinder-Backup service two back ends to backup to: Ceph and Swift.  For the VMAX Cinder block storage drivers, Swift is the chosen back end to backup to.


Backups vs Snapshots

Although very similar in their purpose in OpenStack, there are subtle differences between both and how they should best be used. Cinder snapshots serve as a way to continually take exact point-in-time copies of the data within a volume that resides on the same storage device that the source volume lives on.  Cinder backups are full-copy volume clones which are in most circumstances stored on a different platter or storage node from which the source volume resides.  Best practices for Cinder backups would ideally result in the backup being stored on a storage node in a data-centre is a different geo-location for disaster-recovery situations. 

Creating & Managing Volume Snapshots

Snapshot operations in OpenStack are performed using the cinder snapshot-[command] format, where command is replaced by operations such as 'create', 'delete', or 'list'. A full list of Cinder Snapshot CLI commands can be found in the Cinder CLI reference guide here. To create a snapshot of a volume in OpenStack you will need a volume, so I will create a 1GB bootable volume using the CirrOS image again.




Using the available bootable volume 'cirros_vol', we can create a point-in-time snapshot of the volume.

Command Format:

$ cinder snapshot-create --name [snapshot_name] [volume_name/volume_id]

Command Example:

$ cinder snapshot-create --name cirros_snap_1 cirros_vol




To list your available OpenStack snapshots, use the cinder snapshot-list CLI command. To view specific details about a particular snapshot, use the cinder snapshot-show command.

Command Format:

$ cinder snapshot-show [snapshot_name/snapshot_id]

Command Example:

$ cinder snapshot-show bb27b18b-c135-481c-939f-f3945c5cd070




Another possibility with snapshot management is the ability to rename the snapshot or add a description to it.

Command Format:

$ cinder snapshot-rename [snapshot_name/snapshot_id] [new_snapshot_name]

Command Example:

$ cinder snapshot-rename cirros_snap_1 cirros_snap_june




To take the rename command even further, if you want to add a description to the snapshot, you can either repeat the command exactly as above and add the --description [user_description] parameter and value, or you can leave out the new snapshot name entirely (example in screenshot below).

Command Format:

$ cinder snapshot-rename [snapshot_name/snapshot_id] [new_snapshot_name] --description [user_description]

Command Example:

$ cinder snapshot-rename cirros_snap_1 cirros_snap_june --description "June 2nd 2017"

$ cinder snapshot-rename cirros_snap_june --description "June 2nd 2017"




Deleting a snapshot in OpenStack using the CLI only requires a simple cinder snapshot-delete command and the snapshot name or ID. You can also additionally add the --force parameter to the command if a snapshot of a volume is in a state other that 'available' or 'error'.

Command Format:

$ cinder snapshot-delete [snapshot_name/snapshot_id][--force]

Command Example:

$ cinder snapshot-delete cirros_snap_june




Creating a Volume from a Snapshot

Creating a volume from a snapshot is almost identical to creating a blank volume with no source, the only difference being the inclusion of the --snapshot-id parameter.

Command Format:

$ cinder create --snapshot-id [snapshot_id]

Command Example:

$ cinder create --snapshot-id 136d7787-7a4d-46ea-9337-a5ce6cbeddb5 --name vmax_vol1 --volume-type VMAX_ISCSI_DIAMOND 1




Creating & Managing Cinder Volume & Snapshot Backups

The Cinder CLI provides the tools for creating a volume backup. You can restore a volume from a backup as long as the backup’s associated database information (or backup metadata) is intact in the Block Storage database. In advance of using Cinder's backup service, you will need to make sure that you have the backup service correctly configured. The configuration will be dependent on the type of service which will interact with Cinder backup to store the backups (in this example we will be using Swift object storage service). For more information on configuring your environment for Cinder backup, please follow the detailed instructions at the official OpenStack website for 'Install and configure the backup service'. At this point we will assume that you have your backups service installed and configured, and you have a running instance in OpenStack with a Cinder backed bootable volume attached.


To create a backup of a volume use the following command:

Command Format:

$ cinder backup-create [--name <name>] [--incremental] [--force] <volume>

Command Example:

$ cinder backup-create --name cirros_backup --force cirros_boot_vol




Where volume is the name or ID of the volume, incremental is a flag that indicates whether an incremental backup should be performed, and force is a flag that allows or disallows backup of a volume when the volume is attached to an instance.


Without the incremental flag, a full backup is created by default. With the incremental flag, an incremental backup is created. The incremental backup is based on a parent backup which is an existing backup with the latest timestamp. The parent backup can be a full backup or an incremental backup depending on the timestamp.


Without the force flag, the volume will be backed up only if its status is available. With the force flag, the volume will be backed up whether its status is available or in-use. A volume is in-use when it is attached to an instance. The backup of an in-use volume means your data is crash consistent. The force flag is False by default.


Note: The first backup of a volume has to be a full backup. Attempting to do an incremental backup without any existing backups will fail. There is an is_incremental flag that indicates whether a backup is incremental when showing details on the backup. Another flag, has_dependent_backups, returned when showing backup details, will indicate whether the backup has dependent backups. If it is true, attempting to delete this backup will fail.

To list all of the backups currently available in your environment use the cinder backup-list command, to show specific details about any given backup, use the command cinder backup-show <volume> where volume is the backup name or ID.




To restore a volume backup, there are number of options to take into consideration depending on the type of restoration you require. The CLI command is structured as follows:

Command Format:

$ cinder backup-restore [--volume <volume>] [--name <name>] <backup>

Command Example:

$ cinder backup-restore --name cirros_restored cirros_backup



Both the volume and name parameters in the backup-restore command are optional, specifying only the backup to restore will create a new volume with a randomly generated name assigned. Supplying the name will give the new volume a user-specified name, and including the volume parameter tells OpenStack which existing volume is to be restored using the backup.

Note: When restoring from a full backup, a full restore will be carried out. Also, for a restore to proceed successfully on a pre-existing volume, the target volume must have an 'available' status.

Deleting a Cinder volume backup is handled similarly to all other delete operations in OpenStack, you need only supply the name or ID of the resource you want to delete, multiple backups can be specified to delete more than one backup in the same CLI command.

Command Format:

$ cinder backup-delete [--force] <backup> [<backup> ...]

Command Example:

$ cinder backup-delete cirros_backup



The force flag is used in the delete command to remove backups which have a status other than 'available' or 'in-use'.

Users can take snapshots from the volumes as a way to protect their data. These snapshots reside on the storage back end itself. Providing a way to backup snapshots directly allows users to protect the snapshots taken from the volumes on a backup device, separately from the storage back end. There are users who have taken many snapshots and would like a way to protect these snapshots. The functionality to backup snapshots provides another layer of data protection. To create a backup of a snapshot, use the same command format as creating a normal Cinder volume backup, but this time include the --snapshot-id parameter:

Command Format:

$ cinder backup-create [--name <name>] --snapshot-id [snapshot] [--force] <volume>

Command Example:

$ cinder backup-create --name cirros_snap_backup --snapshot-id df942bd8-f509-4535-9e37-90a4958e0ea0 --force cirros_boot_vol




When backing up a snapshot, it is not enough to just specify the snapshot ID, you must also specify the ID of the source volume also. In addition, if the source volume is 'in-use' the --force parameter must also be included. When specifying the snapshot ID, only the snapshot ID will suffice, it is not possible to use the snapshot name.

Restoring a backup of a snapshot uses exactly the same process as restoring a standard backup, you specify the snapshot backup ID instead of a volume backup ID.


Important Note on Cinder Backup Service

Cinder volume backups are dependent on the Block Storage database, meaning that if the database becomes unavailable or corrupted your backups won't be of any use to you.  In order to prevent against unusable Cinder volume backups, you must also back up your Block Storage database regularly to ensure data recovery. To learn more about backing up OpenStack databases please read the Official OpenStack documentation 'Backup and Recovery'.


Alternatively, you can export and save the metadata of selected volume backups. Doing so precludes the need to back up the entire Block Storage database. This is useful if you need only a small subset of volumes to survive a catastrophic database failure.  For more information about how to export and import volume backup metadata, see the Official OpenStack documentation 'Export and import backup metadata'.



Video Demonstration

To show all of the above operations using the Horizon dashboard, I have created a short demo video. Full-screen viewing of the embedded video is not possible from the DECN website, to view the video full-screen click here or click on the YouTube logo on the video controls to redirect to the video on the YouTube website.




Troubleshooting issues with Snapshots and Backups

There is very little that can go wrong with snapshots and backups in terms of setup and configuration, if the setup from part 1 of this series is correct, and you can provision volumes with no issues, then there won't be many areas that need looked at if something goes wrong.


As mentioned in the introductory article in this series, there are a number of licenses which are required for a VMAX to function correctly in an OpenStack environment. These licenses are directly related to the functionality which has been described in this article on snapshots and backups so it is essential that they are installed into your VMAX before OpenStack can be used to its full potential.


The list of required licenses is very briefly touched on in the table below, but I would highly recommend getting in contact with your VMAX account manager for more information on licensing and in particular licensing for OpenStack.


Advanced SuiteLocal Replication SuiteRemote Replication Suite
Cinder SnapshotsRequiredRequiredNot required
Cinder Backups (local)RequiredRequiredNot required
Cinder Backups (remote)RequiredRequiredNot required
Cinder Replication (SRDF/S)RequiredNot requiredRequired


If the setup is correct and all required software licenses are installed there should be no issues with creating snapshots of VMAX volumes in OpenStack, however, if you believe you have found a bug in the Snapshot functionality get in contact to let us know through your Dell EMC rep.  If you are having issues with Cinder backups, make sure the service is correctly installed and configured as per the instructions in the guide 'Install and configure the backup service'. Configuring services other than Cinder do not fall under the scope of our series here, but there is ample resources on the internet for setting up Cinder backup with services such as Ceph or Swift.


What's coming up in part 5 of 'VMAX & OpenStack Ocata: An Inside Look'...

In the next in the series of 'VMAX & OpenStack Ocata: An inside Look' we will be looking at managing and unmanaging volumes. These operations will allow us to manage existing VMAX volumes into OpenStack for use in the environment, or unmanage volumes whereby OpenStack volumes are removed from the environment but still remain on the VMAX for later use. See you then!

In my last post I went over over-subscription, quality of service (QoS), and compression. If you would like to see that blog article again click here. This time around I will be going over the basic operations that you can carry out using the VMAX drivers for OpenStack Cinder.  The operations I will be looking at are:

  • Create, list, delete, attach, and detach volumes
  • Copying images to/from volumes
  • Clone a volume
  • Extend a volume


When going through each of the operations, I will include examples of performing that operation using the OpenStack Cinder CLI and at the end of the article there will be a video going over all functionality using the Horizon dashboard. The hope is that by the end you will be fully familiar with each operation and confident to carry them out on your own! With the formalities out of the way, lets begin...


Note: At this point it is assumed that you have fully configured your OpenStack environment for your VMAX and created a VMAX back-end with one or more volume types to provision volumes with.


Create, list, delete, attach, and detach volumes

Provisioning volumes and managing them in OpenStack couldn't be easier using the Cinder CLI or dashboard, it is intuitive and at no point are you left guessing what to do next. The first operation we will look at is creating a volume.

Creating & deleting volumes

When creating a volume there are only a few things you need to know in advance:

  • The size of the volume (in GB)
  • The name of the volume
  • The volume type


If you want to list the volume types you have created in OpenStack use the command cinder type-list. To view your volume types using the dashboard, navigate to Admin>Volume>Volume Types.


To then create a volume, use the following CLI command:

Command Format:

$ cinder create --name <volume_name> --volume-type <vmax_vol_type> <size>

Command Example:

$ cinder create --name vmax_vol1 --volume-type VMAX_ISCSI_DIAMOND 10



To view the details of your newly created volume, use the command:

Command Format:

$ cinder show <volume_id>/<volume_name>

Command Example:

$ cinder show vmax_vol1



To view all of the volumes you have created use the command cinder list. If you want to delete any of the volumes you have created the CLI command is as follows:

Command Format:

$ cinder delete <volume_id>/<volume_name> [<volume_id>/<volume_name>...]

Command Example:

$ cinder delete vmax_vol1



You will notice that in the cinder delete command there is the option to add another volume name or ID, if you want to delete more than one volume at a time you only need list the volumes in the one command, separating each with a space.

Command Example:

$ cinder delete vmax_vol1 vmax_vol2 vmax_vol3

Attaching & detaching volumes

When you have created a volume for use in OpenStack, it is likely that you will want to then attach it to a compute instance for later use. It is assumed here that you already have a running Nova compute instance, it is that instance we will be attaching our VMAX volume to.  In advance of this operation you need only to know the instance ID that you would like to attach the volume to, and the volume ID to be attached. If no device mount-point is specified, or the value is set to auto, OpenStack will automatically pick the next available mount-point to attach the volume to.

The required IDs can be obtained using the Nova & Cinder commands # nova list and # cinder list




Taking these IDs, you can attach a volume to the instance using the below Nova command:

Command Format:

$ nova volume-attach <instance_id> <volume_id> [<device>/auto]

Command Example:

$ nova volume-attach ee356911-3855-4197-a1ef-aba3437f6571 28021528-e641-4af6-8984-a6ea0815df7f /dev/vdb




To detach a volume you need only replace 'attach' in the command above with 'detach, there is no need to specify the device mount-point as it is already attached to a mount-point and so not required for detach operations:

Command Format:

$ nova volume-detach <instance_id> <volume_name/volume_id>

Command Example

$ nova volume-detach ee356911-3855-4197-a1ef-aba3437f6571 vmax_vol1




Copying images to/from volume

Images in OpenStack are pre-configured virtual-machine images which can be used to provision volumes. This is especially useful in production environments, where you can create a master image of a volume, and use that image to provision multiple identical volumes, saving you countless time setting up each.


For this operation it is assumed that you have an image already in OpenStack Glance (image service), any image will do but for this example we are going to use a lightweight CirrOS image. The process is almost identical to provisioning a blank volume, the only difference is the addition of the --image parameter where we can specify the image name we would like to use.

Command Format:

$ cinder create --name <volume_name> --volume-type <vmax_vol_type> --image <image_name>/<image_id> <size>

Command Example:

$ cinder create --name vmax_cirros --volume-type VMAX_ISCSI_DIAMOND --image cirros-0.3.5-x86_64-disk 10




The outcome of the above command will be a new 10GB VMAX volume called 'vmax_cirros', volume type VMAX_ISCSI_DIAMOND, with the CirrOS system image copied to it. When this volume is attached to a Nova compute instance, it can be powered on to run a CirrOS virtual machine.

After setting up this virtual machine with all the necessary packages and environment configuration, it is in a state which can be used for whatever purposes the user has. To avoid having to go through the process of copying the system image to the volume each time, and configuring it as necessary, it is possible to take the configured virtual-machine and turn its present state into an image to be used to provision new virtual machines. To copy a volume to a Glance image, use the command:

Command Format:

$ cinder upload-to-image <volume> <image-name>

Command Example:

$ cinder upload-to-image vmax_cirros cirros_full_config




Cloning a volume

Cloning a volume does exactly what it says on the tin, it take a volume and copies it to make a new volume. The new volume is an exact replica of the volume at that point in time and can be used straight away to perform operations on such as attaching to a compute instance.  Just like creating a volume from an image, creating a cloned volume is very similar to the create volume command. The command is the same but this time the additional parameter is --source-volid which lets us specify the source volume that we would like to create a clone from. To create a cloned volume, the command is as follows:

Command Format:

$ cinder create --name <volume_name> --volume-type <vmax_vol_type> --source-volid <volume_id> <size>

Command Example:

$ cinder create --name vmax_cirros_clone --volume-type VMAX_ISCSI_DIAMOND --source-volid 54249fab-3629-4017-becf-1134c7e56cf0 20




The example above will take the CirrOS volume from part 2 of this guide, and creates a bigger volume (20GB) called 'vmax_cirros_clone' with the same volume type as before.


Extending a volume

Extending a volume takes a pre-existing volume and increases its size to a size of your choosing. It is only possible to increase the size of a volume, it is not possible to make a volume smaller than its current size. Extending a volume is really straight forward using the CLI, you only have to specify the volume ID and the size you want to increase it to.

Command Format:

$ cinder extend <volume_id> <new_size>

Command Example:

$ cinder extend 54249fab-3629-4017-becf-1134c7e56cf0 30




Video Demonstration

To show all of the above operations using the Horizon dashboard, I have created a short demo video. Full-screen viewing of the embedded video is not possible from the DECN website, to view the video full-screen click here or click on the YouTube logo on the video controls to redirect to the video on the YouTube website.



Troubleshooting issues with Volume Management

Working with volumes are one of the most basic operations that can be performed, so luckily when troubleshooting issues in this area we can focus on just a few areas to diagnose what is going on.


The most obvious of issues you could face during volume management is the inability to provision a volume for use. We are going to keep things uncomplicated and say that this volume is a blank volume for now, we can look at bootable volumes later (and if you can't create a bootable volume you should always check if you can still create a blank volume).  If you can't create a blank volume you should check the following:

  • Is your configuration correct (as outlined in part 1 of this series of blog articles)?
  • Have you got the required capacity to provision the volume on the VMAX?
  • Have you hit the upper limit for volume storage in OpenStack?
  • Have you hit the upper limit for volumes in OpenStack?
    • To check your limits from the CLI use the cinder absolute-limits command. You will see the upper limits of each quota and your current usage across all volume types defined in Cinder




If you can create a volume but you can't attach it to a Nova compute instance you need to check the setup of your storage network and if there is successful connectivity between your OpenStack controller/compute nodes and your VMAX storage array. The same checks can be carried out if you are having issues creating a bootable volume, in both cases there is a point in the operation where the VMAX volume needs to be mounted, either for attachment to an instance, or to copy an image to a volume so it can be made bootable. If you can create the volume but fail at the next stage, then it is most likely a storage network issue and most often this issue is connectivity problems between nodes.


What's coming up in part 4 of 'VMAX & OpenStack Ocata: An Inside Look'...

Now you can create and manage basic operations, we can start to look at some of the advanced functionality, next on the list is volume snapshots. We will be taking a look at what snapshots are, the advantages of having SnapVX backed snapshots, and how to work with them using the same methods we covered in this article.