Since last week’s cache persistence blog article, there have been several inquiries around the relative latency of the different levels of the OneFS read caching infrastructure.
Typically, L2 cache is more valuable than L1, because an L2 hit avoids a higher latency operation. An L1 cache hit negates the need for a back-end round-trip to fetch the data. However, an L2 cache hit avoids a SATA disk seek in the worst case. This is a substantial difference in both relative and absolute terms.
For SATA drives, an L2 miss is two orders of magnitude above a hit compared to one for L1, and a single back-end round-trip is typically a small portion of a full front-end operation.
The following table shows the relative latency of OneFS Cache Hits and Misses. These values are shown in microseconds (us) and miliseconds (ms):
L3, (or Hard Disk)
Note: These figures are for comparative guidance, and may vary in an active cluster.
L2 cache is also preferable because it is accessible to all nodes. Assuming a workflow with any overlap among nodes, it is preferable to have the cluster’s DRAM holding L2 data rather than L1. In L2, a given data block is only cached once and invalidated much less frequently. This is why storage nodes are configured with a drop-behind policy on file data. Nodes without disks, like for example the A100 performance accelerator, will not drop behind since there is no L2 data to cache.
L3 cache contains file data and metadata blocks evicted from L2 cache, effectively increasing L2 cache capacity, albeit with double the latency. However, this is considerably preferable to the order of magnitude or more latency that a hard disk drive seek incurs.
For related information, please refer to the read cache persistence blog article, mentioned above: