IOPS degradation as a leading indicator before capacity loss

The standard story about drive failure is incomplete

The dominant mental model for drive failure in most infrastructure teams is: drive shows SMART errors, drive fails, data is gone. The intervention point in this model is the SMART error. If you catch it fast enough, you replace the drive before it stops serving reads.

That model misses a class of failures that shows up in the performance layer first. Before a drive loses capacity, before it starts reporting reallocated sectors at scale, before the SMART health check turns red, there is often a period of IOPS degradation. The drive is still healthy from the binary perspective, but it is doing more work per I/O than it used to, and the extra work is measurable.

We analyzed IOPS performance data alongside SMART telemetry for three drive model families across a six-month period in 2025 to understand what this degradation looks like, how early it appears, and how to distinguish it from normal workload variation.

What drives IOPS degradation before failure

For SATA HDDs, IOPS degradation ahead of failure is typically caused by increasing head seek time. As sectors develop read errors and the drive queues them for reallocation, the read-retry logic adds latency per I/O. A single bad sector that the drive retries three times before succeeding adds 15 to 40 milliseconds to that I/O. On a drive handling 200 IOPS, a handful of consistently slow sectors raises average latency enough to be visible in percentile metrics without causing any errors.

For SSDs and NVMe, the mechanism is different. Flash cells at end of life require more read voltage to produce reliable data. The controller compensates by reading at a higher voltage and retrying marginal cells. Each retry is fast relative to an HDD head seek, but at high IOPS the aggregate effect of more frequent retries becomes measurable in tail latency, specifically at P95 and P99.

Both mechanisms produce the same observable symptom at the application layer: stable average IOPS with rising tail latency percentiles, followed by gradual reduction in total IOPS as more of the drive's internal bandwidth is consumed by retry operations.

What the degradation pattern looks like across three drive families

We looked at three drive model families in our dataset: a Seagate Exos 4TB SATA HDD, a Samsung PM983 NVMe, and a Toshiba MG06 4TB SATA HDD. For each family, we identified drives that subsequently failed and traced their IOPS history for 60 days before the failure event.

Seagate Exos SATA: Visible IOPS degradation started on average 23 days before failure. The signature was a gradual increase in P95 read latency while average IOPS remained relatively stable. The P95/P50 latency ratio widened from approximately 3:1 (normal operating range for this model under production load) to 8:1 or higher in the 14 days before failure. Total IOPS drop was modest until the final 4 to 5 days when pending sector count was already high.

Samsung PM983 NVMe: IOPS degradation was subtler and started later, approximately 12 days before failure. The primary indicator was P99 write latency increasing by a factor of 3 to 5 while the available spare percentage was still above 15%. Read IOPS degradation was minimal until 5 days before failure. The write-latency increase was detectable before any SMART/NVMe telemetry moved significantly.

Toshiba MG06 SATA: This family showed the most gradual degradation. Elevated P95 latency was detectable 35 days before failure in some cases. IOPS degradation was slow enough that without a per-drive baseline, it could easily be attributed to workload change rather than drive health.

Correlating IOPS signals with SMART telemetry

Days to failure    SMART-197   P95 latency   IOPS (% of baseline)
-60                0           1.2x          100%
-40                0           1.3x          99%
-23                0           1.8x          97%
-14                2           2.9x          93%
-7                 8           5.1x          84%
-3                 16          8.6x          71%
0                  --          --            device removed

This is a representative trajectory from a Seagate Exos drive in the dataset. The IOPS and latency signals started moving 23 days before failure. The SMART-197 counter did not move until 14 days before failure. For this drive, IOPS degradation gave 9 additional days of lead time compared to SMART attribute changes.

This is not universal. In some failures, SMART attributes moved first. In others, performance degradation was the only precursor before a sudden failure. The practical implication is that both signal channels are needed, and they should be combined into a single risk score rather than monitored independently with separate alert rules.

Why most monitoring setups miss this

We are not saying IOPS monitoring is easy to implement as a failure-prediction signal. The challenge is that IOPS and latency are workload-dependent. A drive whose latency increased because the application started running heavier queries looks identical to a drive whose latency increased because of physical wear without a per-drive baseline and workload correlation. Acting on raw IOPS degradation without that baseline produces too many false positives to be operationally useful.

Most Prometheus setups track aggregate disk I/O per host, not per physical drive. The standard node_exporter metrics (node_disk_io_time_seconds_total, node_disk_read_time_seconds_total) do expose per-device data, but teams rarely write per-drive latency percentile alerts. The tooling to compute P95/P99 per physical drive and compare against that drive's baseline requires a time-series aggregation that most monitoring setups do not implement.

Crest tracks per-drive IOPS and latency metrics as part of the agent telemetry stream alongside SMART data. The failure model uses the combined signal: when both IOPS degradation and SMART attribute movement are present, the prediction window is tighter and confidence is higher. When only IOPS degradation is present without SMART movement, the model applies more conservative prediction windows because the signal is less specific, but it still contributes to the overall risk score rather than being ignored.

What to do with this in practice

If you want to add IOPS-based early warning to an existing monitoring setup without deploying a dedicated agent, the minimal viable approach is: track node_disk_read_time_seconds_total and node_disk_reads_completed_total per device, compute average read latency per device over 10-minute windows, and alert on per-device latency that exceeds 150% of the prior 30-day average for that device. This will catch the late-stage degradation but miss the early signal.

The more complete approach is what Crest does: combine the per-device latency trend with the SMART and NVMe telemetry trends into a unified risk model per drive. The performance signal alone is noisy. The telemetry signal alone misses some failure paths. Combined, they produce a prediction signal that is both earlier and more reliable than either channel in isolation.