Why off-the-shelf anomaly detection is the wrong starting point
The first version of the prediction pipeline we built was a straightforward SMART anomaly detector. Read the 24 standard attributes, train a statistical model on normal behavior, flag deviations. It looked reasonable and performed poorly. The recall rate on actual failures was around 40%. We were missing more than half the failures we would later find with a more targeted approach.
The core problem is that drive telemetry is not anomaly data in the usual sense. Most drives exhibit highly variable SMART attribute behavior throughout their lifecycle that is completely normal. Temperature variance, read error rates, and command retry counts fluctuate based on workload, thermal conditions, and firmware behavior in ways that have nothing to do with impending failure. An anomaly detector trained on the aggregate will flag healthy drives constantly and miss the specific degradation trajectories that predict actual failure. Precision tanks, SREs stop trusting the output, and you end up worse than no model at all.
The right framing is not anomaly detection. It is survival analysis: given this drive's current telemetry trajectory, what is the probability that it fails within the next N days? This is a different model architecture, trained differently, evaluated differently, and it requires treating the problem from the outset as a time-to-event prediction problem rather than a classification problem.
Feature engineering: what to include and what to discard
The raw SMART attribute values are almost never the right input features. Raw normalized values are smoothed and bounded by the drive vendor. The information content is in the derived signals. For each attribute we include in the model, we compute several derived features over sliding windows.
For attributes that accumulate over time (reallocated sector count, media errors, command timeouts), the most predictive features are the rate of change over 7-day and 30-day windows and the acceleration (rate of change of the rate of change). A drive whose reallocated sector count jumped from 0 to 4 this week is very different from a drive that has accumulated exactly 4 over two years. The current value is a weak signal; the velocity and whether velocity is itself increasing are strong signals.
For attributes that represent health estimates (percentage life remaining, available spare percentage, wear leveling count), we compute the weekly decline rate and compare it against the expected decline rate for that specific drive model given its reported write workload. A drive burning through available spare at 3x the predicted rate for its write volume pattern is meaningfully different from a drive tracking within expected bounds at the same absolute spare level.
For temperature and power cycle data, we compute variance and trend rather than absolute values. Temperature mean at steady workload is not predictive. Rising temperature variance at constant workload, or a trend break where temperature starts running hotter without a corresponding workload increase, is predictive.
After feature construction, we apply feature importance testing against the labeled failure dataset. On NVMe drives, reallocated block count velocity, available spare decline rate, and media data integrity error count turn out to carry the most weight. On SATA SSDs, attribute 187 (Reported Uncorrectable Errors) velocity, attribute 177 decline trajectory, and command timeout rate are the strongest contributors. The feature importance ranking is different between protocol families, which is one reason you cannot use a single model across both without losing predictive accuracy on one or the other.
Model architecture: why we chose survival analysis
Cox proportional hazards regression is the primary modeling layer. It is interpretable, handles right-censored data correctly (drives that haven't failed yet but are under observation), and produces a hazard ratio per feature that makes it possible to reason about which signals are contributing most to a drive's risk score at any given time. That last property matters operationally: when Crest surfaces a risk alert, we want to be able to say which specific telemetry attribute is driving the elevated risk, not just output a risk score from a black-box model.
The survival model outputs a probability distribution over failure time for each drive, conditioned on its current telemetry trajectory. From this distribution we extract two numbers: the risk score (probability of failure within the prediction window) and the predicted failure window (the time range where failure probability exceeds the alert threshold). Both are updated every time we ingest new telemetry for the drive.
Cox PH has a proportional hazards assumption that does not hold perfectly for all drive telemetry features. Some attributes interact nonlinearly with time-on-service. We handle this with time-varying covariates for the high-interaction features and stratification by drive model family to allow the baseline hazard to differ across model groups. Without per-model stratification, the model applies a single baseline hazard rate that fits no drive model well and fits the highest-failure-rate family in the training set best.
For drives where the Cox model produces a high-uncertainty output (wide confidence interval on the predicted failure window), we apply a gradient-boosted tree model trained on the same feature set as a secondary signal. When both models agree that risk is elevated, confidence in the alert is high. When they diverge, we surface the alert at a lower severity level and flag it as requiring human review before action. This two-model approach reduces both false positives and missed alerts compared to relying on either model alone.
Per-model normalization: why this matters more than any algorithm choice
Drive failure telemetry has a significant confound: different drive models have fundamentally different attribute distributions even in healthy operation. A reallocated sector count of 8 means something different on a Seagate Barracuda 4TB than on a WD Red 6TB. A wear leveling count normalized value of 150 means something different on a Samsung 860 Pro than on a Kioxia CM5 enterprise NVMe.
Training a single model on the combined telemetry of multiple drive models without per-model normalization produces a model that learns the characteristics of the most-represented drive model in the training set. It under-weights the failure patterns of less-represented models and produces poor recall on those families even if overall recall looks acceptable.
Our approach is to normalize each SMART attribute to a per-model z-score distribution before training. This requires a large enough labeled dataset per drive model to compute stable distribution parameters. For drive models with fewer than roughly 200 historical drive-months of telemetry in our dataset, we apply a partial pooling approach: normalize to the vendor family distribution first, then apply a drive-model correction factor based on the available data for that specific SKU. This lets the model generalize to less-common drive models while still benefiting from the richer data available for high-volume drive families.
Evaluation: precision-recall and what the operational cost of each error type is
Precision and recall are not equally important for drive failure prediction. The cost of a false negative (missing a failure) is high: a drive that the model said was healthy fails unplanned, triggering an incident and potentially data loss. The cost of a false positive (flagging a healthy drive) is lower: an SRE investigates the drive, finds nothing wrong, and moves on. But false positives accumulate as operational load and erode trust in the alerting system if they are too frequent.
We evaluate the model at multiple operating points along the precision-recall curve and set the default alert threshold at the operating point where false positive rate stays below 15% while recall on 30-day failures is at or above 80%. This means roughly 8 in 10 drives that will fail within 30 days receive an alert, with a false positive rate of 1 in 7 alerts being a drive that does not actually fail within the window.
We are not claiming those numbers are achievable on all drive model families with all fleet sizes. They reflect performance on the drive telemetry in our training and validation dataset. Drive model families with less representation in the training data, particularly niche enterprise SAS models and very new NVMe SKUs, have wider confidence intervals on these estimates. The performance numbers degrade on drive models we have seen fewer than 50 failure events for, and we surface that uncertainty in the model's confidence output rather than hiding it behind a single aggregate accuracy figure.
What the labeled dataset problem looks like in practice
Training a drive failure model requires labeled examples: telemetry from drives that failed, with the failure event timestamped so you know what the attribute values looked like in the hours, days, and weeks before failure. This labeled data is genuinely difficult to acquire. Drives in production fleets fail infrequently enough that accumulating thousands of labeled failure events from a single fleet takes years.
The labeled dataset we built for Crest's initial model combined drives from a handful of sources: drives we monitored in a storage infrastructure environment before starting Crest, drives contributed by early users with detailed failure logs, and augmentation using published research datasets (Google, Backblaze) where the attribute schema overlapped with our feature set. For the public datasets, we had to handle the fact that the monitoring interval, attribute selection, and drive model composition do not exactly match what the Crest agent collects. Aligning those datasets required careful feature mapping and removing attributes that were collected differently in the external datasets.
The consequence of a small labeled dataset is that rare failure modes are underrepresented. Controller failures that do not produce SMART attribute changes, for example, appear in our training data but not in large enough quantities to train a model that reliably catches them. We surface those cases as the second-tier "unusual telemetry pattern" alerts rather than as high-confidence predictive failure alerts. Being explicit about what the model is trained to detect and what it is not is more useful than claiming complete coverage.
Retraining cadence and drift handling
Drive model portfolios change. New NVMe SKUs enter production fleets, drive firmware updates change attribute behavior, and the write workload on a fleet evolves. A model trained on 2023 telemetry and left static will drift relative to the drives it is being asked to score in 2025.
The Crest model retrains monthly for drive models with sufficient new labeled data, and triggers an emergency retrain when a known failure event produces telemetry that the current model scored as low risk. That second trigger catches cases where a new drive model SKU or firmware revision has introduced attribute behavior the current model has not seen. The emergency retrain uses the new failure event as a high-weight training example and updates the affected model strata without retraining the full model from scratch.
The useful operational test for model drift is not a summary accuracy metric. It is whether recent confirmed failure events appear in the model's top-quintile risk scores in the week before failure. If the last 10 confirmed failures in your fleet were all scored in the bottom two quintiles the week before they failed, your model has drifted. Monitoring that signal actively is more useful than periodic batch evaluation against a static held-out test set.