Received a couple of recent enquiries from the field regarding SmartQuotas performance. In this article we'll explore one of the more obscure tuning parameters of SmartQuotas. But first, a quick refresher on the OneFS quotas architecture.
From the file system point of view, there are three main elements to SmartQuotas:
Define which files and directories belong to a quota.
The quantity being limited
Specify the limits and what actions are taken when those thresholds are exceeded
Each OneFS Quota Domain includes a set of usage levels, limits, and configuration options. Most of this information is organized and managed by the file system and stored in the Quota Database. This database is represented in a B-tree structure, known as the Quota Tree, and provides both scalability and fast random access. Because of its importance, the Quota Database is protected at highest level for metadata in OneFS. The Quota Accounting Blocks (QABs) within individual records are protected at the same level as the associated directory.
A Quota Domain is made up of the following principle parts:
Data structure that stores the QDR
Quota domain record (QDR)
Stores all configuration and state associated with a domain
Quota domain key
Where the unique identifier for the domain is stored
Quota domain header (QDH)
Contains various state and configuration information that affects the domain as a whole
Quota domain enforcements
Manages quota limits, including whether they have been hit or exceeded, notification information, and the quota grace period
Quota domain account (QDA)
Handles tracking of usage levels for the domain. Tracks physical, logical, and file resource types for each domain
Resource allocation and governance changes are recorded in the quota operation associated with a transaction, totaled and applied persistently to the QDRs.
The Quota Domain Record can be broken down into three elements:
Fields within quota config, such as whether the domain is a container. Despite the name, this includes some state fields like the Ready flag
A list of quota enforcements, which include the limit, grace period, and notification state. Although the structure is flexible, only three enforcements are allowed, and only for a single resource
The quota account for the domain
The on-disk format of the QDR is shown in the following diagram. The structure is dynamic, based on the configured enforcements and state of the account, so the on-disk structures look much different than the in-memory structures.
Quota domain locks synchronize access to quota domain records in the QDB. The main challenge for quota domain locks is that the need to lock quota domains exclusively is not known until the accounting is fully determined. In fact, it may not be until responses from transaction deltas are received before this is reported to the initiator. To address this, Quota Domain Locks use optimistic restarts.
Within the SmartQuotas database, quota data is maintained in Quota Accounting Blocks (QABs). Each QAB contains a large number of Quota Accounting records, which need to be updated whenever a particular user adds or removes data from the quota domain, the area of the filesystem on which quotas are enabled. If a large quantity of clients are simultaneously accessing the quota domain, these blocks can become highly contended and a potential bottleneck. Similarly, if a single client (or small number of clients) consistently makes a large number of small writes to files within a single quota, write performance could again be impacted.
To address this, quota accounts (as of OneFS 184.108.40.206 and later) have a mechanism to help avoid hot spots on those nodes which are storing QABs. This problem is solved using Quota Accounting Constituents, or QACs, which help parallelize the accounting. QACs can boost the performance of quota accounting by creating additional QAB mirrors, which are distributed across the cluster.
QAC configuration is via the sysctl 'efs.quota.reorganize.qac_ratio', which increases the number of accounting constituents, which are in turn spread across a much larger number of nodes and drives. This provides better scalability by increasing aggregate throughput and reduces latencies on heavy create/delete activities when quotas are configured.
Using this parameter, the internally calculated QAC count for each quota is multiplied by the specified value. If a workflow experiences write performance issues, and it has many writes to files or directories governed by a single quota, then increasing the QAC ratio may significantly improve write performance.
The qac_ratio can be reconfigured to from its default value of ‘1’ up to the maximum value of 8 via the following CLI command:
# isi_sysctl_cluster efs.quota.reorganize.qac_ratio=8
To verify the persistent change, run:
# cat /etc/mcp/override/sysctl.conf | grep qac_ratio
efs.quota.reorganize.qac_ratio=8 #added by script
Although increasing the QAC count via this sysctl can improve performance on write heavy quota domains, some amount of experimentation may be required until the ideal QAC ratio value is found.
Adjusting the ‘qac_ratio’ sysctl parameter can adversely affect write performance if you apply a value that is too high, or if you apply the parameter in an environment that does not have diminished write performance due to quota contention.
To help assess write performance while tuning the QAC ratio, write latency (TimeAvg) for NFSv3, for example, can be continuously monitored by running the following CLI command:
# isi statistics protocol --protocols nfs3 --classes write --output TimeAvg --format top