Hi! I wanted to put something together for using the Dell EMC VMAX technical add-on (TA) & front-end app for Splunk Enterprise 6.5 (and above) to give you all a bit more information about it, setting it up, getting the data into the front-end app and getting you ready for analysing your VMAX storage systems.  I will try to go over all of the functionality so there is no unanswered questions, but if I do manage to miss something by all means let me know in the comments or by private message and I will be sure to follow up on it!  I will include most of the information from the VMAX TA User Guide for Splunk , with some extra bits and pieces here and there that I feel will benefit you in setting it all up on your end.

 

About the Splunk Technical Add-On & App for VMAX

The VMAX TA allows a Splunk admin to collect inventory, performance information, and summary information from VMAX storage arrays. You can then directly analyse the data or use it as a contextual data feed to correlate with other operational or security data in Splunk Enterprise. The VMAX App for Splunk Enterprise allows admins to take the data ingested into Splunk and analyse it to gain insight into VMAX Array inventory and performance data.

 

Currently the TA is version 1.0.1 and the app is version 1.0, both are configured to work with VMAX-3 and All-Flash arrays using Unisphere 8.3. Support for Unisphere 8.4 is coming in the near future! Watch this space for more information when it becomes available

 

Note: I have created a script you can run in your environment to check connectivity to Unisphere, registered VMAX-3 series arrays, performance metrics registration & timestamp confirmation. Click here to get that script.


Data Collection & Source Types

The VMAX TA provides the index-time and search-time knowledge for inventory, performance metrics, and summary information. By default, all VMAX data is indexed into the default Splunk index, this is the ‘main’ index unless changed by the admin.


The Splunk VMAX TA is configured to report events in 5 minute intervals which is the lowest possible granularity for performance metrics reporting.  Event metric values are representative of the value recorded at that point in time on the VMAX. Values shown for an event in Splunk at 10:00am represent their respective values at 10:00am on the VMAX.


The add-on collects many different kinds of events for VMAX including performance, inventory, and summary metrics. Depending on the activity of the Port Groups & Initiators in your environment, there may be events where there are no performance metrics collected. This can be confirmed if there is a metric present in the event named ‘perf_data’ with a value of ‘false. To limit the amount of data collected and stored on a VMAX, only active Port Groups & Initiators are reported against, so it is intended behaviour to have no performance metrics for those which have been inactive for some time.


The source type used for the Splunk Add-on for VMAX is 'dellemc:vmax:rest'. All events are in key=value pair formats. All events have an assigned 'reporting_level' which indicates the level at which the event details, along with the associated VMAX array ID & if reporting at lower levels, the object ID e.g. Storage Group, Director, Host.


Hardware and software requirements for the Splunk TA & App for VMAX

To install and configure the VMAX TA & App, you must have Splunk admin privileges. Because this add-on runs on Splunk Enterprise, all of the Splunk Enterprise system requirements apply.


There are no specific hardware or software requirements for the VMAX TA, it will point towards your existing environment and Unisphere to gather metrics. For the VMAX app you will need to install two additional Splunk apps; Splunk Status Indicator, and Splunk Treemap. These are both available through the Splunkbase or through the Splunk web UI.


Single Instance/Distributed Environment Installations

In a distributed deployment, install the Splunk VMAX TA to your search heads and heavy forwarders. This TA does not support universal forwarders because the TA requires Python. The add-on does not need to be installed on indexers because it does not support universal forwarders or light forwarders, thus parsing occurs on the heavy forwarder rather than on indexers. The app only needs to be installed on the search heads, and requires no additional configuration.

 

For a detailed single/distributed installation instructions, refer to Splunk's "Installing add-ons" that describes how to install an add-on in the following deployment scenarios:

  • Single-instance Splunk Enterprise
  • Distributed Splunk Enterprise
  • Splunk Cloud
  • Splunk Light


Note: I am aware that at present the Splunk TA only allows collection of metrics from a single instance of Unisphere. To get around this, have a distributed Splunk deployment where each forwarded is pointed at a different instance of Unisphere, allowing your indexers and search heads to collect data from more than one instance of Unisphere at a time. This functionality is set to change in future iterations of the Splunk TA.


VMAX TA Installation Considerations

The add-on does not require the ability to modify VMAX configuration. It is highly recommended that you create a read-only user account with proper read capabilities in Unisphere for VMAX.


The VMAX TA works through the RESTful communications between Splunk and Unisphere, so it is necessary to have Unisphere setup and running in your environment with your arrays added. I wont go into details about REST here, but if you would like to know more about it my colleague Paul Martin has put together a great series of blog articles on REST & VMAX to get you started. The first article in that series is 'Getting Started with the REST API'.


Performance of data collection is dependent on many factors, such as VMAX system load, Splunk Enterprise system load, and environmental factors such as network latency.  Before any metrics can be collected from a VMAX you must also ensure that the VMAX is registered to collect performance metrics. This is enabled from within the Unisphere for VMAX Web UI. For more information on enabling performance metrics collection, please see the ‘Registering Storage Systems’ in the ‘Performance Management > Settings’ section of the ‘Unisphere for VMAX Online Help’ guide.


Installing the VMAX TA for Splunk Enterprise

Once you have Splunk set up and running in your environment, there is very little required to get the VMAX TA set up and collecting information. There are no additional requirements or dependencies, so once you have the VMAX TA downloaded from the Splunkbase website or through the app store from within Splunk you are good to go with set up! I am going to go through the process of setting up the TA first, adding VMAX arrays as data inputs afterwards, then finally setting up the VMAX app to start viewing meaningful analysis of your environment through Splunk.


1. Within the Splunk Web UI navigate to Apps > Manage Apps. Click the button ‘Install App from file’


Install1.png


2. Click ‘Choose File’ and select the Splunk VMAX TA. Once selected, click ‘Upload


Install2.png

 

3. Once the upload is complete you will be prompted to restart Splunk to complete the installation, click ‘Restart now’. After Splunk has restarted and you have logged back in, you will get an ‘Install successful’ message. Click ‘Set up now’ to proceed to configuring the Splunk VMAX TA.


Install3.png

 

4. The Splunk VMAX TA configuration screen will ask you for the following environment details:

  1. Unisphere IP Address
  2. Unisphere Port
  3. Unisphere Username
  4. Unisphere Password

Enter these details and click ‘Save’ when complete.


Install4.png


5. To add VMAX data inputs to Splunk, navigate to Settings > Data inputs > Dell EMC VMAX REST, click either on the Add-on name or click ‘Add new’ to add a new VMAX data input

 

 

Install5.png

 

 

6. When adding a VMAX data input, you will be required to enter two values:

  1. Name (A Splunk Web UI name for your own reference)
  2. VMAX Array ID (VMAX Numerical ID)

When you are ready to add the data input, click ‘Next’ to continue

 

 

Install6.png

 

 

7. Within the ‘Dell EMC VMAX REST’ data input you will now see your VMAX listed as an input. To add another input, repeat steps 5 & 6 until all desired inputs have been added (Note: Array IDs removed for this article)


Install7.png

 

 

8. Once your VMAX data inputs have been added to the TA they will start ingesting summary and performance metrics into your specified Splunk index. To start viewing this data straight away navigate to Splunk search and load the VMAX data index chosen when adding the data inputs.

 

 

Install8.PNG.png

 

 

Installing the VMAX App for Splunk

In addition to just having a TA which ingests VMAX data into Splunk, there is a front end app for the Splunk UI which gives you a number of dashboards to analyse your data easily. You can also take the queries from these dashboards and use them to build your own look at the app as a basis from which to build your own full feature dashboard to monitor your entire environment!


Installing and configuring the VMAX app for Splunk is just as easy as installing the VMAX TA...


1. Download the app from Splunkbase first then install it in the same way as the TA, navigate to Apps > Manage Apps. Click the button ‘Install App from file’. Click ‘Choose File’ and select the Splunk VMAX TA. Once selected, click ‘Upload.


2. (Optional) If you are using an index for VMAX data other than the default index you will need to tell the app where to look in order for it to start analysis the data and running the queries against it. To do this, on your Splunk host navigate to:

{splunk_install_location}/etc/apps/Dell-EMC-app-VMAX/default/

 

Copy all of the settings within and create a new macros.conf file in

{splunk_install_location}/etc/apps/Dell-EMC-app-VMAX/local/

 

The macros.conf file allows us to designate specific environment settings for our app (for a full breakdown of the macros.conf file click here). Within the macros.conf file you will see a number of VMAX configuration groups, each of which has a index= value. For each VMAX configuration group change this value to the name of the index you are using for the VMAX performance data and restart Splunk. Defining these index values in the local directory will override the settings defined in the default directory.

 

app1.PNG.png

 

 

To restart Splunk use the Splunk CLI or the web interface, once restarted the VMAX app will be reading the data from your chosen index and presenting all of the information in the various tables, charts, and info-graphics!

 

 

app2.PNG.png

 

app3.PNG.png

 

app4.PNG.png

 

 

Troubleshooting the VMAX TA

Note: I have created a script you can run in your environment to check connectivity to Unisphere, registered VMAX-3 series arrays, performance metrics registration & timestamp confirmation. Click here to get that script.

 

To diagnose problems with your Splunk & VMAX environment, the first place to look for answers is in the log files for the TA and for Splunk itself.  The two log files can be found in $SPLUNK_HOME/var/log/splunk under the names:

ta_DellEMC_vmax_DellEMC_vmax_rest.log

splunkd.log


Before the add-on successfully runs for the first time, error logs go to splunkd.log. After the add-on successfully runs, error logs go to ta_DellEMC_vmax_DellEMC_vmax_rest.log.


The Splunk VMAX TA has been developed to give the end-user as much detail as possible about the activity of the add-on in their environment.  All add-on logged events will either be marked as ‘info’ or ‘error’, depending on the nature of the event. If you are having any issues with the add-on, the logs will be able to give you precise information as to the cause of the problem. These issues could be related, but not limited to:

  • Incorrect add-on configuration
  • Incorrect Array ID
  • VMAX is not performance registered
  • Performance metrics timestamp is not up-to-date


Performance Data Gaps

Depending on the activity of the Port Groups & Initiators in your environment, there may be events where there are no performance metrics collected. This can be confirmed if there is a metric present in the event named ‘perf_data’ with a value of ‘false’. To limit the amount of data collected and stored on a VMAX, only active Port Groups & Initiators are reported against, so it is intended behavior to have no performance metrics for those which have been inactive for some time.


Splunkd Timeout Issues

In environments where there are VMAX storage arrays which are moderately loaded with resources such as Storage Groups, there will be occasions where Splunk cannot gather all of the required data in time before the splunkd service will timeout.

By default, this timeout value is set to 30 seconds in the Splunk configuration file ‘web.conf’. In order to increase this default setting to something more realistic for VMAX data collection, please follow these steps:

  1. Navigate to $SPLUNK_HOME/etc/system/local/
  2. Create a file called 'web.conf' and enter the following (the value of 1200 is for example purposes, the more resources on a VMAX the higher this number may be):
    [settings]
    splunkdConnectionTimeout = 1200
  3. Restart Splunk once the file has been created

 

Now it's up to you!

That is all of the required functionality covered to get you up and running with Splunk in your VMAX environments, as long as you have Unisphere set up and configured beforehand there is next to no effort required to get Splunk set up with it. Once you have your environment details specified and add the data inputs the TA does the rest of the heavy lifting and the app displays all of the data in a very neat and tidy fashion!

 

What's Next....

Both the VMAX TA and app are both in their very first iterations, both offerings are still in their version 1.0.x stage, so at this point in time we are open to any and all suggestions on what you believe should be included, fixed, made better, removed, whatever, in future releases! Without input from the people that use these offerings day in day out it is best guess as to what we think would work best, so any feedback is always welcomed Even if you only installed the TA to ingest data and build your own SPL (Splunk Processing Language) queries and dashboards, we would still very much like to hear from you! You can send me a message through the community network or even better e-mail the VMAX Splunk support alias at  vmax.splunk.support@emc.com. Many Thanks and I hope to hear from you soon!