How to gather the necessary information for a CLARiiON or VNX performance Analysis ?

Product:

 

VNX

 

Description:

 

Gather the necessary information for a CLARiiON or VNX performance analysis.

 

Fix

 

For each event, Technical Support needs to know the following, in the description of the performance problem:

  • The time (with the time zone GMT/UTC difference) and date.
  • The server names or CLARiiON Storage Groups that own the LUNs.
  • The LUNs that the affected application was using at that time.
  • Is it reading or writing performance that is suffering?

 

  Obtain the basic diagnostic files that must cover the period when the problems occur:

  • SPcollects for both Storage Processors, which were run after the problem occurred (so that their logs would include the time of the incident).
  • Navisphere (or Unisphere) Analyzer .NAR or .NAZ files, which cover the whole period when the problem was occurring.
  • Host Grabs and Switch logs may also be required

 

To obtain SPcollects, see Navisphere (or Unisphere) Help article "Generating and transferring SP log files" or:

  • See EMC Knowledgebase article emc60493 for instructions on how to obtain SPcollects on a CLARiiON running Navisphere.
  • See EMC Knowledgebase article emc253191 for instructions on how to obtain SPcollects on a VNX or CLARiiON running Unisphere.

 

  To gather Navisphere (or Unisphere) Analyzer .NAR or .NAZ files:

  • If the problem occurs throughout the day, include NAR / NAZ files, supply a total period of approximately 24 hours (for a day when the issues occurred):
    • NAR and NAZ files contain the same data, but only NAR files can be opened in Navisphere / Unisphere Analyzer.  If the CLARiiON has an Analyzer license ('-NavisphereAnalyzer' in the software list) then NAR files will be produced; otherwise data logging produces NAZ files.  NAZ files are for use by EMC support only, to assist with break/fix performance diagnostics.
    • Uploading more NAR files than are needed can allow support to get more background information to an issue and see if it has been going on for some time, but usually a performance analysis will concentrate on a single NAR file which contains the most relevant data.
  • To gather the performance data it is first necessary access the Analyzer menu and the method for this differs depending on the version of Unisphere or Navisphere:
    • For Navisphere (CLARiiONs running Release 29 or lower), select the Tools drop-down menu (at the top of the Navisphere Window), then select Analyzer. A sub-menu will then appear with the Navisphere Analyzer options.
    • For CX4 Unisphere (CLARiiON CX4 running Release 30), go to the System List view and select the CLARiiON you wish to monitor.  Highlight the Monitoring Tab and select Analyzer, which will bring up a page with all of the available Analyzer options (which will depend on whether the Analyzer enabler is installed).
    • For VNX Unisphere, after selecting the VNX system, click on the System Tab. Click on ‘Monitoring and Alerts’, followed by ‘Statistics for Block’.  This will bring up a page with all of the available Analyzer options (which will depend on whether the Analyzer enabler is installed).  In other words, the folders under the taskbar at the top should be listed as: This_VNX > System > Monitoring and Alerts > Statistics for Block. 
  • Before starting the data logging, the archive interval should be set appropriately:
    • Click on the 'Performance Data Logging' option, in the Navisphere Analyzer menu (see above) to check the current settings or to change them.
    • Before making any changes it is best to stop the data logging first (if it is already running), which will close the current file.
    • A typical NAR (or NAZ) file will contain 155 polling intervals of data. Therefore, if the archive polling interval was 120 seconds, there would be 5 hours and 10 minutes of data in each file.  Once each file is full (or sooner if periodic archiving is enabled), it will be closed and a new one started.  These can later be merged together, in order to produce graphs covering a longer period, but only upload the original files to Powerlink for a performance analysis (i.e. not any merged files).
    • The archive interval is set to 120 seconds by default, in order the produce a detailed record of performance.  However, large CX3 and CX4 configurations may take longer than 120 seconds to poll, in which case the interval would need to be set to a higher interval, such as 240 seconds.  Having a higher polling interval can reduce the impact of data logging on the SP Utilization (see article emc207795), but will reduce the amount of detail in each NAR file.
    • On VNX, SP Utilization levels are less likely to be an issue and too high a polling interval can miss details about heavily loaded drives in a Pool.  Therefore with VNX, it is recommended to have the archive interval set at 60 seconds for testing and no more than 300 seconds for regular monitoring.
    • It is usually recommended to also set the real-time interval to the same value as the archive interval.  The impact of Analyzer on the SP Utilization (which is typically around 1% of additional utilization) would be dictated by the lower of these two values.
    • The results recorded will be averages of the performance over archive intervals.  Therefore if the archive polling interval was set at 600 seconds, performance problems which only lasted a few minutes, would be hard to diagnose.
    • For FLARE release 22 and earlier, you will need to be in Engineering Mode to change the Archive Interval (see article emc255033)
    • To change the archive interval using the CLI, use the following command: naviseccli ~ analyzer -set -narinterval 120  (where ~ would be where the arguments -h <IP address>, -user <username> etc. may need to included)
  • To start data logging:
    • Click the 'Performance Data Logging' option, in the Navisphere Analyzer menu (see above).  In the data logging window that opens, click on the Start button.
    • Alternatively use the following CLI command: naviseccli ~ analyzer -start  (where ~ would include the SP IP address and the Navisphere login)
  • A NAR (or NAZ) file will not be available to download while it is still open for data logging.  Therefore the list of NAR files will only include the files which have been completed (which normally happens when there are 155 polling intervals contained in them), not the latest file.
    • To obtain the latest data, there is a 'Create New' button in the 'Retrieve Archive' window.  Alternatively the data logging can be stopped and then started again.  Either option will close the current NAR file and then start a new one (although you may need to wait for a minute or so for the current file to be closed).
    • The following CLI command will start a new NAR file and make the latest NAR file available to download:
      • naviseccli ~ analyzer -archive -new  (where ~ would include the SP IP address and the Navisphere login)
  • To retrieve the completed archives (after they have had time to gather sufficient information and the performance issues were occurring during this period):
    • Click 'Retrieve Archive' from the Analyzer menu (also referred to as 'Statistics for Block')
      • Make sure the correct CLARiiON is selected in the top drop-down box.
      • Select SP A.
      • Select the relevant archives and click Retrieve.
      • Repeat for SP B (although usually the same data will be present in the the NAR or NAZ files from either SP)
    • The path for saving the files can be changed, but the default path is set in the 'Customize Charts' window (using the Archives tab), on the Analyzer menu.
    • Alternatively use the following CLI command to retrieve all the archives from one SP: naviseccli -h <SP IP> analyzer -archive -all
    • If you have remote access to the SP, via EMC Remote or Remotely Anywhere, then the NAR/NAZ files can be transferred directly from C:\EMC\Archives folder. (Flare release 24 and higher)
    • See article emc255033 for further information on using Navisphere / Unisphere to collect NAR/NAZ files.
    • For Release 22 and earlier, see article emc227465.
  • Do not use the 'naviseccli ~ analyzer -archiveretrieve' in Release 24 or higher.  Instead use 'naviseccli ~ analyzer -archive <-all or -file filenames> - see emc240526
  • The NAR/NAZ file title contains the date and time (in GMT) of the last data point logged.  It is important to stress that the Navisphere files have to be for the period when the problems were actually occurring.
  • Upload files to Powerlink in the same form that they were produced on the SP.  Do not merge NAR files and do not decrypt NAZ files.  Any changes to the files can introduce errors, so it is important that the original files are supplied.
  • A guide to running Navisphere Analyzer can be found by going to 'Help' in Navisphere.  Select 'Help topics', followed by 'Analyzing storage-system performance using Analyzer'
  • A guide to using Naviseccli for retrieving NAR files can be found on Powerlink, titled: 'EMC® Navisphere® Analyzer Command Line Interface (CLI)'.
  • For creating a script on a host to automate the creation and collection of archive files, see Support Solution emc148982 "How do I automate collecting .NAR files?"

 

  Host Grab (or EMC Report) output files may be needed, if the issue is unique to specific hosts.

  • Select this link to EMC Support which will search for the Grab utilities for different OS types.
  • Download the utility and the README file.  Follow the README file to run the utility for the host operating system chosen.
  • Alternatively, many of these grabs can also be downloaded from ftp://ftp.emc.com/pub/emcgrab/.

 

  Switch logs may be needed, especially if the performance problem only exists on certain paths.  Please refer to the following EMC knowledgebase articles based on the model switch being used:

 

 

For more information, Refer EMC Knowledgebase article emc161922 .