Observability: an alert that does not say what to do ends up ignored

An operations team receiving forty alerts a day handles none of them. Within weeks it learns which ones are noise, and it ends up ignoring them all — including the one that matters.

That is the most common failure mode of observability, and it is organisational rather than technical.

The alert rule

An alert must correspond to an immediate human action. If nobody needs to do anything on receiving it, it is not an alert, it is a metric. It belongs on a dashboard, not in a notification.

That criterion generally eliminates the majority of alerts in an existing configuration.

The four attributes of a useful alert

A named recipient. Not a distribution list, not a general channel. An identified on-call person, with a defined escalation if they do not respond.

A link to a runbook. The runbook says what to check, in what order, and what to do depending on the result. An alert without a runbook forces the on-call person to reinvent the diagnosis at three in the morning.

A justified threshold. Why 85% and not 90%? If nobody knows, the threshold was copied from a template and will fire at the wrong moment. A threshold should come from observation: above what value did a real incident occur?

A duration. Alerting on an instantaneous value produces noise; alerting on a value sustained for five minutes produces a signal.

What to alert on first

The order that works, from most to least useful:

1. Symptoms visible to the user. Request error rate, 95th percentile response time, failure of a scheduled business job. That is what actually matters.
2. Capacity thresholds approaching. Disk space, quotas, database connections: the ones that give you lead time before the incident.
3. Background job failures. Backups, imports, nightly jobs. A silent backup failure over three weeks is a classic scenario.

Alerting on a machine's CPU usage, by contrast, is almost always noise: a machine at 90% that responds correctly is not a problem.

Periodic pruning

Once a quarter, list the alerts that fired and classify them into three categories: those that led to an action, those acknowledged with no action, and those never acknowledged at all.

The last two categories should be removed or reconfigured. An alerting system that is never pruned degrades mechanically.

The link to your service agreement

If you have an availability commitment with a provider, check that the measurement comes from a source you control, or at minimum a shared one. A commitment measured only by the provider's own tooling is an unverifiable commitment, and they know it.