How we test enterprise SSDs

How we test enterprise SSDs

(Image credit: Tom's Hardware) (Image credit: Tom's Hardware) (Image credit: Tom's Hardware) (Image credit: Tom's Hardware) Image 1 of 4 View Original

Testing enterprise drives in a standard PCIe expansion slot is more convenient. For that purpose, Icy Dock has provided us with the ExpressSlot Elite MB221E5P-B and MB303E5P-B to accept one and two E1.S SSDs. Unlike other generic designs on the market, these adapters feature tool-less installation mechanisms that don't require removing them from the PCIe expansion slot, and they include 50mm fans for active cooling.

The list also included the ToughArmor MB218E5P-B , a 5.25-inch drive bay that supports up to eight E1.S SSDs for future projects exploring RAID arrays. To test E3.S drives, we use a JMT ADT-R27C adapter strapped to a Noctua NF-F12 industrialPPC-24V-3000 IP67 PWM cooling fan, which can run at 12V and still provide substantial airflow to actively cool the SSDs under evaluation.

A spec sheet only tells you what a drive can do on its best day. The real world is not as pretty. The drive is mostly full, garbage collection is constantly at work, and multiple workloads are hitting it simultaneously. Our enterprise benchmark suite is built to measure the drive in service: at steady state, under load, through write bursts, and with other jobs running. Every drive gets the same automated run on the same test bench. Every result is stored with the exact parameters that produced it, so we can trace how we got there.

This can take some time. A full run takes most of a day on a 7.68TB PCIe 5.0 drive, and it writes the drive end-to-end many times over. To make life a bit easier on us, the benchmark can be run unattended, and it’s crash-resumable from the last stage. A shorter express mode handles bench checks and quick triage. When a result looks odd, express runs and variant runs, which repeat just the tests in question or change one setting, help us find the cause before we take it to the vendor.

We run the bench on a custom Debian 13 image that we build and maintain ourselves, with the suite copied in at build time. This image gets installed on the bench’s OS drive, and the kernel is pinned so every drive has the same software stack. Right now that’s Linux 7.2.6 from Debian backports, because Debian 13’s stock 6.12 kernel predates support for the Asus Pro WS W890E-Sage SE’s Intel E610 10GbE controller. The suite is also designed to be run remotely if needed, but we need a tester at the bench to switch the drives, of course.

Debian is our pick over Ubuntu for a few reasons: it has fewer background services, no Snap, and no unattended upgrades by default. We hold the kernel so nothing changes between runs. We remove AppArmor and turn off scheduled jobs that could fire mid-run, like the weekly TRIM timer and the drive-health service that polls every drive, and the suite records any that are still running. The suite also turns off NVMe autonomous power-state transitions (APST) at the start of every run and checks that the drive reports it off, so no drive drops into a low-power state during an idle gap and adds wake-up latency to the next I/O.

Every test runs in fio 3.42, using io_uring and direct I/O against the raw block device. This means no filesystem and no page cache sitting between fio and the drive; the bench’s 64GB of RAM never caches a byte of test data. We compile fio from source because Debian’s packaged version lacks SPRandom, which we’ll get to in short order.

Some enterprise storage reviews run application profiles in Oracle’s Vdbench, but we decided to go full fio. This makes for a more consistent suite with one set of latency accounting across every chart. And fio is open source and actively maintained, while Vdbench hasn’t had a new release since 2018. Vdbench profiles are built from the same parts fio uses, such as read/write ratio, block-size mix, and random or sequential access, and our mixed-workload sweeps cover the common read/write ratios at 4K and 8K, though not mixed-block-size profiles. Running Vdbench ourselves wouldn’t make our results comparable to other outlets’ anyway, since they use different hardware, preconditioning, and profile definitions.

A key element of benchmarking SSDs is making sure the test drive is at steady state, with the performance it can sustain rather than the peak you can see after a fresh erase. Our first stage is a secure erase using nvme format. If a drive refuses, the suite falls back to a plain format, then a full-drive discard, and records which one it used.

Next comes preconditioning. SPRandom, SanDisk’s open-source contribution to fio, is a faster alternative to filling the drive over and over for hours. It takes user capacity and over-provisioning (OP) as inputs, which SanDisk says lets it reach a steady-state layout in just one pass. Overprovisioning (OP) is inferred from the capacity point or given by the tester. NVMe drives don’t report their raw flash, so a 7.68TB drive like our first one, assumed to sit on 8TiB of actual NAND, works out to having about 14.5% OP.

We don’t take SPRandom’s result on faith. A steady-state gate follows, running 4K random writes at queue depth 256 (QD256) until the drive’s write speed stops moving. It uses the two tests from the Storage Networking Industry Association’s (SNIA) steady-state rule, with limits ten times tighter, applied to the last 30 minutes: split into five six-minute rounds, the round averages must sit within 2% of one another, and the trend line across the window can’t move more than 1%.

That takes a while, but that’s the point: an earlier, looser version of the gate passed our first drive while its write speed was still climbing. The whole settling curve is saved with every run, and a drive that doesn’t settle within three hours is marked as not reaching steady state, though the run continues. As a backstop, the suite repeats the four-corner 4K write after the random sweep, hours later, and flags the run if the two results differ by more than 3%.

The suite also checks its own results as it goes. If a test’s slowest moment falls below 70% of its average, that test is run again on the spot. If a point in the random sweep lands more than 10% off the curve set by its neighbors, it’s measured again before the drive is erased for the sequential tests. Both results are kept, and any re-test is noted in the review.

Following the random tests, the tool erases the drive again and writes it end to end once, with 128K sequential writes at QD64. We run it before the sequential and AI tests, so the drive is conditioned with the kind of I/O we’re about to measure. It matters because sequential writes can swing wildly on a randomly written drive: on our first test drive, the fastest result was more than three times the slowest. After a sequential fill, they held flat from QD2 through QD1024, at both 128K and 1MB.

Drives are formatted to 512-byte logical blocks when they support it, so our block size sweep can go down to 512-byte transfers, which a 4K-native namespace can’t issue. SNIA’s enterprise IOPS test uses 512-byte transfers too, and Kioxia pitches 512-byte access for its GP Series, which is aimed at GPU-initiated storage. Drives without a 512-byte format are tested as shipped, and their sub-4K results are marked unsupported. The switch happens in the same format command as the secure erase, so it adds no time.

A fast enough SSD will happily benchmark your host, or your tools, instead, and our early runs turned up three ways that could happen. The first is CPU idle states, or C-states: waking our Xeon 658X cores from their deepest idle state can take up to 170 microseconds, and at QD1 that directly impacts the results. Holding the cores awake for the whole run cut the first drive’s average 4K QD1 read latency by about a sixth.

The second is the CPU itself: a single fio job can saturate its core well short of what a PCIe 5.0 drive can manage. Split across eight jobs, the same test runs several times faster, so random tests now spread their queue depth across up to eight fio jobs. That’s also why fast cores, and enough of them, matter for this test bench.

The third is fio itself. By default, fio seeds its random generator the same way every run, so it replays the same sequence of random offsets, and each random write test rewrites the previous test’s blocks in the same order. On our first drive, that was the main reason 4K random writes came in at more than twice the sustained figure. Every random test now runs with a fresh random seed.

What we don’t change matters, too. Completions arrive by interrupt rather than polling, and the IOMMU stays in its default mode, because that’s how most applications use a drive. On our first drive, polling and IOMMU passthrough would each have shaved some latency off 4K QD1 writes.

Interrupts also set a ceiling at the other end of the scale: this bench tops out near 2.8 million 4K random read IOPS, and adding fio jobs doesn’t lift it, though polling does. So on the fastest drives, our 4K random read result can sit below the spec sheet’s, and that’s our bench’s limit, not the drive’s. Our low queue depth latencies, writes especially, run higher than a tuned host would show, but they’re closer to what you’d actually see. To show the drive’s limit at both ends, the suite repeats three tests with polled I/O at the end of the random stage: the 4K random read peak, plus 4K random reads and writes at QD1. Those results get their own charts and never mix into the main ones.

4K random read and write at QD256; 128K and 1MB sequential read and write at QD64

The headline numbers, measured on a preconditioned drive instead of fresh out of the box

4K and 8K, QD1 to QD4096, eight read/write mixes from 100% read to 100% write

Key considerations

  • Investor positioning can change fast
  • Volatility remains possible near catalysts
  • Macro rates and liquidity can dominate flows

Reference reading

More on this site

Informational only. No financial advice. Do your own research.

Leave a Comment