Wiring predictive storage alerts into PagerDuty

The problem with existing PagerDuty storage integrations

Most PagerDuty storage integrations trigger on events that have already happened. RAID degradation after a drive drop. Filesystem errors after I/O failures. Replication lag after data loss begins. By the time these events fire, the incident is already in progress. You are managing recovery, not preventing the failure.

The standard SMART monitoring integrations are a partial improvement. A smartmontools alert that fires when a drive reports SMART FAILED is better than waiting for the RAID card to notice the drop, but it still fires at the moment of failure rather than before it. The signal you want in PagerDuty is: "this drive will likely fail in the next 8 to 18 days, here is the physical drive ID, here is the host, here is which volume it backs."

That is what we built the Crest PagerDuty connector to deliver. This post explains how it works, how we structured the incident payload, and the severity mapping decisions we made.

How the connector creates incidents

Crest uses the PagerDuty Events API v2. When a drive's risk score crosses a configured threshold, the connector sends a trigger event. The dedup key is set to the physical drive identifier, so if the same drive continues to degrade, subsequent events update the existing incident rather than creating duplicates.

A typical trigger event payload from Crest looks like this:

{
  "routing_key": "YOUR_INTEGRATION_KEY",
  "event_action": "trigger",
  "dedup_key": "crest-drive-storage-node-04-sdc",
  "payload": {
    "summary": "Drive /dev/sdc on storage-node-04 predicted to fail in 12-18 days",
    "severity": "warning",
    "source": "crest-agent@storage-node-04",
    "component": "/dev/sdc",
    "group": "storage",
    "class": "drive-health",
    "custom_details": {
      "drive_id": "/dev/sdc",
      "host": "storage-node-04",
      "model": "Seagate ST4000NM005A",
      "serial_hash": "a3f7c2819b",
      "health_score": 68,
      "risk_level": "watch",
      "predicted_failure_window": "12-18 days",
      "confidence": "high",
      "primary_signal": "SMART-197 pending sector count trending +2/day over 8 days",
      "crest_dashboard_url": "https://app.gocrest.org/drives/storage-node-04-sdc"
    }
  }
}

The dedup key matters in practice. Drive degradation is gradual. Without deduplication, a drive that stays in the "Watch" risk band for 10 days generates 10 incidents, which trains the on-call team to ignore storage alerts. With deduplication, there is one incident that updates as the situation evolves. The incident closes automatically when the drive is replaced and the risk score drops back to healthy range.

Severity mapping: getting it wrong trains bad habits

The most consequential configuration decision for a PagerDuty storage integration is severity mapping. If you map all drive health alerts to critical, you get paged at 3am for a drive that has an 18-day failure window. Your on-call team learns to treat storage alerts as low-signal noise and starts acknowledging them without acting. By the time the drive is days from failure, the alert gets the same treatment as all the other storage alerts that never turned into actual incidents.

The mapping we recommend for Crest alerts:

• Risk score 60-74, predicted window 15-30 days: warning severity, route to low-priority service. Business hours response acceptable. Team reviews weekly.
• Risk score 75-89, predicted window 7-14 days: error severity, route to the storage team's working-hours escalation policy. Action expected within 2 business days.
• Risk score 90+, predicted window under 7 days: critical severity, on-call page. This is the only tier that should wake someone up.

This tiering keeps the on-call pager quiet except for genuine urgency, while creating an actionable queue for the working-hours team to process.

Resolve events and incident lifecycle

The Crest connector sends a resolve event when a drive's risk score returns to healthy range. This happens in two scenarios: the drive was replaced and the agent re-registered the new drive at baseline health, or the degradation trend reversed (less common, but happens with pending sector counts that stabilize after a bad batch of writes).

We are not recommending you auto-resolve incidents without verifying the replacement was actually completed. In practice, we have seen cases where a drive gets replaced, the agent registers the new drive, but the replacement drive was a different model that the agent needed additional time to baseline. Auto-resolved incidents that reopen 48 hours later create confusion. A simple "verify replacement completed" step in your resolution runbook is worth the two minutes it takes.

Integration setup in the Crest dashboard

Setting up the PagerDuty integration in Crest requires a PagerDuty integration key from a service configured with the Events API v2 integration type. In the Crest dashboard under Settings, Integrations, PagerDuty:

1. Paste the integration key for the target PagerDuty service
2. Set the risk score thresholds for each severity tier
3. Configure the dedup key prefix if you have multiple Crest-monitored environments routing to the same PagerDuty service
4. Send a test event to verify routing is working before saving

The connector uses the PagerDuty Events API endpoint at events.pagerduty.com/v2/enqueue over HTTPS. No inbound firewall rule changes required on your infrastructure; all traffic is outbound from the Crest backend to PagerDuty. For on-premise Crest deployments, the agent needs outbound HTTPS access to the Crest API endpoint, which then handles the PagerDuty relay.

What to do after the first week of alerts

After a week of running the PagerDuty integration, you will have a sense of whether the threshold configuration needs adjustment. If you are seeing too many warning-severity incidents that the team never acts on, raise the warning threshold from 60 to 65 or 70. If you want more lead time on critical severity, lower the critical threshold window from 7 days to 10.

The common adjustment we see in practice: teams start with the default thresholds, get a handful of warning incidents in the first week (usually drives that have been degrading slowly and were never previously visible), act on the two or three most urgent ones, and find that the configuration is roughly right for their tolerance level. The drives that showed up in the first week are typically the ones that would have been the next 3am incidents under the old monitoring setup.

Connecting the Crest Prometheus metrics to PagerDuty via Alertmanager

For teams that already use Prometheus Alertmanager as their alert routing layer, an alternative path is to use the Crest Prometheus metrics endpoint rather than the direct PagerDuty connector. The crest_drive_risk_score gauge is available for scrape from the Crest agent, and Alertmanager's PagerDuty receiver handles the escalation routing.

This approach has one advantage and one limitation. The advantage: your alert routing config lives in Alertmanager alongside all other infrastructure alerts, so storage incidents go through the same deduplication and grouping logic as everything else. The limitation: the Alertmanager PagerDuty receiver uses a generic alert payload. The Crest direct connector produces a richer PagerDuty incident with the predicted failure window, drive model, and dashboard URL pre-populated in the custom details. The Alertmanager path can be configured to surface these via annotations, but it requires templating work that the native connector handles automatically.

For teams with a mature Alertmanager setup, the Prometheus path is often the cleaner integration. For teams where PagerDuty is the primary incident system and Prometheus is used mainly for metrics, the native connector is less friction. Both are supported and the configuration for each is documented in the Crest integration settings panel.