Had a couple of recent field inquiries on how to gather performance statistics on a cluster, and thought it might be an interesting topic for a blog article:
Before planning or undertaking any performance tuning on a cluster (or its attached clients):
1) First, record the original cluster settings before making any configuration changes to OneFS or its data services.
2) Next, measure and analyze how the various workloads in your environment interact with and consume storage resources.
Performance measurement is typically done by gathering statistics about the common file sizes and I/O operations, including CPU and memory load, network traffic utilization, and latency.
Key metrics and wall-clock timing data for delete, renew lease, create, remove, set user data, get entry, and other file system operations, can be obtained from OneFS' VOP (virtual operation) counters. To access these stats, connect to a node via SSH and run the following command as root to enable the vopstat system control:
# sysctl efs.util.vopstats.record_timings=1
After enabling vopstats, they can be viewed by running the ‘sysctl efs.util.vopstats’ command as root. Here's an example of the command's output:
# sysctl efs.util.vopstats
efs.util.vopstats.ifs_snap_set_userdata.initiated: 26 efs.util.vopstats.ifs_snap_set_userdata.fast_path: 0 efs.util.vopstats.ifs_snap_set_userdata.read_bytes: 0
The time data captures the number of operations that cross a OneFS clock tick, which is 10 milliseconds in duration. Independent of the number of events, the total_sq_time provides little in the way of actionable information because of the short event window. Instead, the 'total_time' stats are a much better bet for analyzing the operations.
The following CLI command will display just the vopstats 'total_time' metrics:
# sysctl efs.util.vopstats | grep –e "total_ time: [ ^0]"
efs.util.vopstats.unlocked_ write_mbuf.total_time: 340006
The ‘isi statistics’ CLI command is another useful tool for performance analysis - and its output is current (ie. real time). Isi statistics has the the following sub-command-level options:
Display cluster usage statistics organized according to cluster hosts and users
Show performance by drive
Identify the most accessed files/directories
List valid arguments to given option
Display cluster usage statistics organized by communication protocol
Generate detailed protocol statistics along with CPU, OneFS, network & disk stats
Query for specific statistics. There are current and history modes
Display general cluster statistics (Op rates for protocols, network & disk traffic (kB/s)
Full command syntax and a description of the options can be accessed via isi statistics --help or via the man page (man isi-statistics).
The ‘isi statistics pstat’ command provides statistics per protocol operation, client connections, and the file system. For example, for NFSv3:
# isi statistics pstat --protocol=nfs3
The ‘isi statistics client’ CLI command provides I/O and timing data by client name and/or IP address, depending on options - plus the username if it can be determined. For example, to generate a list of the top NFSv3 clients on a cluster, the following command can be used:
# isi statistics client --protocols=nfs3 –-format=top
Or, for SMB clients:
# isi statistics client --protocols=smb2 –-format=top
SMB2 and SMB3 current connections are both displayed in the following stats command:
# isi statistics query current --stats node.clientstats.active.smb2
Or for SMB2 + 3 historical connection data:
# isi statistics query history --stats node.clientstats.active.smb2
The following command will total by users, as opposed to node. This can be helpful when investigating HPC workloads, or other workflows involving compute clusters:
# isi statistics client --protocols=nfs3 –-format=top --numeric --totalby=username --sort=ops,timemax
This ‘heat’ command option can be useful for viewing the files that are being utilized:
# isi statistics heat --long --classes=read,write,namespace_read,namespace_write | head -10
This shows the amount of contention where parallel user(s) operations are targeting the same object.
# isi statistics heat --long --classes=read,write,namespace_read,namespace_write --events=blocked,contended,deadlocked | head -10
It can also be useful to constrain statistics reporting to a single node. For example, the following command will show the fifteen hottest files on node 3.
# isi statistics heat --limit=15 --nodes=3
It’s worth noting that isi statistics doesn’t directly tie a client to a file or directory path. Both isi statistics heat and isi statistics client provide some of this information, but not together. The only directory/file related statistics come from the ‘heat’ stats, which track the hottest accesses in the filesystem.
The system and drive statistics can also be useful for performance analysis and troubleshooting purposes. For example:
# isi statistics system -–nodes=all --oprates --nohumanize
This output will give you the per-node OPS over protocol, network and disk. On the disk side, the sum of DiskIn (writes) and DIskOut (reads) gives the total IOPS for all the drives per node.
For the next level of granularity, the drive statistics command provides individual disk info.
# isi statistics drive -nall -–long --type=sata --sort=busy | head -20
If most or all the drives have high busy percentages, this indicates a uniform resource constraint, and there is a strong likelihood that the cluster is spindle bound. If, say, the top five drives are much busier than the rest, this would suggest a workflow hot-spot.
This topic of gathering OneFS drive statistics is covered in more depth in the following article: