Had a couple of recent field inquiries on how to gather performance statistics on a cluster, and thought it might be an interesting topic for a blog article:

 

Before planning or undertaking any performance tuning on a cluster (or its attached clients):

1)  First, record the original cluster settings before making any configuration changes to OneFS or its data services.


2)  Next, measure and analyze how the various workloads in your environment interact with and consume storage resources.


Performance measurement is typically done by gathering statistics about the common file sizes and I/O operations, including CPU and memory load, network traffic utilization, and latency.

 

Key metrics and wall-clock timing data for delete, renew lease, create, remove, set user data, get entry, and other file system operations, can be obtained from OneFS' VOP (virtual operation) counters. To access these stats, connect to a node via SSH and run the following command as root to enable the vopstat system control:

 

# sysctl efs.util.vopstats.record_timings=1

 

After enabling vopstats, they can be viewed by running the ‘sysctl efs.util.vopstats’ command as root. Here's an example of the command's output:


# sysctl efs.util.vopstats

efs.util.vopstats.ifs_snap_set_userdata.initiated: 26 efs.util.vopstats.ifs_snap_set_userdata.fast_path: 0 efs.util.vopstats.ifs_snap_set_userdata.read_bytes: 0

efs.util.vopstats.ifs_snap_set_userdata.read_ops: 0

efs.util.vopstats.ifs_snap_set_userdata.raw_read_bytes:   0

efs.util.vopstats.ifs_snap_set_userdata.raw_read_ops:   0

efs.util.vopstats.ifs_snap_set_userdata.raw_write_bytes:   0

efs.util.vopstats.ifs_snap_set_userdata.raw_write_ops:   0 

efs.util.vopstats.ifs_snap_set_userdata.timed: 0

efs.util.vopstats.ifs_snap_set_userdata.total_time: 0

efs.util.vopstats.ifs_snap_set_userdata.total_sqr_time:   0

efs.util.vopstats.ifs_snap_set_userdata.fast_path_timed:   0

efs.util.vopstats.ifs_snap_set_userdata.fast_path_total_time:   0

efs.util.vopstats.ifs_snap_set_userdata.fast_path_total_sqr_time:   0


The time data captures the number of operations that cross a OneFS clock tick, which is 10 milliseconds in duration. Independent of the number of events, the total_sq_time provides little in the way of actionable information because of the short event window. Instead, the 'total_time' stats are a much better bet for analyzing the operations.


The following CLI command will display just the vopstats 'total_time' metrics:


# sysctl efs.util.vopstats | grep –e "total_ time: [ ^0]"

efs.util.vopstats.access_rights.total_time: 40000

efs.util.vopstats.lookup.total_time:  30001

efs.util.vopstats.unlocked_ write_mbuf.total_time:  340006

efs.util.vopstats.unlocked_write_mbuf.fast_path_total_time:  340006

efs.util.vopstats.commit.total_time: 3940137

efs.util.vopstats.unlocked_getattr.total_time: 280006

efs.util.vopstats.unlocked_getattr.fast_path_total_time: 50001

efs.util.vopstats.inactive.total_time: 100004

efs.util.vopstats.islocked.total_time:   30001

efs.util.vopstats.lock1.total_time:   280005

efs.util.vopstats.unlocked_read_mbuf.total_time: 11720146

efs.util.vopstats.readdir.total_time: 20000

efs.util.vopstats.setattr.total_time: 220010

efs.util.vopstats.unlock.total_time: 20001

efs.util.vopstats.ifs_snap_delete_resume.timed: 77350

efs.util.vopstats.ifs_snap_delete_resume.total_time: 720014

efs.util.vopstats.ifs_snap_delete_resume.total_sqr_time: 7200280042

 

The ‘isi statistics’ CLI command is another useful tool for performance analysis - and its output is current (ie. real time). Isi statistics has the the following sub-command-level options:

 

Statistics

Category

Details

Client

Display cluster usage statistics organized according to cluster hosts and users

Drive

Show performance by drive

Heat

Identify the most accessed files/directories

List

List valid arguments to given option

Protocol

Display cluster usage statistics organized by communication protocol

Pstat

Generate detailed protocol statistics along with CPU, OneFS, network & disk stats

Query

Query for specific statistics. There are current and history modes

System

Display general cluster statistics (Op rates for protocols, network & disk traffic (kB/s)

 

Full command syntax and a description of the options can be accessed via isi statistics --help or via the man page (man isi-statistics).

 

The ‘isi statistics pstat’ command provides statistics per protocol operation, client connections, and the file system. For example, for  NFSv3:

 

# isi statistics pstat --protocol=nfs3

 

The ‘isi statistics client’ CLI command  provides I/O and timing data by client name and/or IP address, depending on options - plus the username if it can be determined. For example, to generate a list of the top NFSv3 clients on a cluster, the following command can be used:

 

# isi statistics client --protocols=nfs3 –-format=top

 

Or, for SMB clients:

 

# isi statistics client --protocols=smb2 –-format=top

 

SMB2 and SMB3 current connections are both displayed in the following stats command:

 

# isi statistics query current --stats node.clientstats.active.smb2


Or for SMB2 + 3 historical connection data:

 

# isi statistics query history --stats node.clientstats.active.smb2


The following command will total by users, as opposed to node. This can be helpful when investigating HPC workloads, or other workflows involving compute clusters:

 

# isi statistics client --protocols=nfs3 –-format=top --numeric --totalby=username --sort=ops,timemax

 

This ‘heat’ command option can be useful for viewing the files that are being utilized:

 

# isi statistics heat --long --classes=read,write,namespace_read,namespace_write | head -10

 

This shows the amount of contention where parallel user(s) operations are targeting the same object.

 

# isi statistics heat --long --classes=read,write,namespace_read,namespace_write --events=blocked,contended,deadlocked | head -10

 

It can also be useful to constrain statistics reporting to a single node. For example, the following command will show the fifteen hottest files on node 3.

 

# isi statistics heat --limit=15 --nodes=3

 

It’s worth noting that isi statistics doesn’t directly tie a client to a file or directory path. Both isi statistics heat and isi statistics client provide some of this information, but not together. The only directory/file related statistics come from the ‘heat’ stats, which track the hottest accesses in the filesystem.

 

The system and drive statistics can also be useful for performance analysis and troubleshooting purposes. For example:


# isi statistics system -–nodes=all --oprates --nohumanize


This output will give you the per-node OPS over protocol, network and disk. On the disk side, the sum of DiskIn (writes)  and DIskOut (reads) gives the total IOPS for all the drives per node.


For the next level of granularity, the drive statistics command provides individual disk info.


# isi statistics drive -nall -–long --type=sata --sort=busy | head -20


If most or all the drives have high busy percentages, this indicates a uniform resource constraint, and there is a strong likelihood that the cluster is spindle bound. If, say, the top five drives are much busier than the rest, this would suggest a workflow hot-spot.


This topic of gathering OneFS drive statistics is covered in more depth in the following article:


https://community.emc.com/community/products/isilon/blog/2017/09/29/onefs-drive-statistics