Your alerts are noise and your noise is an outage

I was on a call with a team in Hamburg whose on-call engineer had ignored three pages in the last hour. Not silenced them. Not acknowledged them. Ignored them, in the sense that he saw the notification on his phone, decided it was not real, and went back to what he was doing. The fourth page was real. Their payment provider webhook endpoint was timing out, orders were failing, and customers were tweeting. He saw that page too, because it came through the same channel with the same priority as the three false ones, and he treated it the same way. He assumed it was noise. It was not noise. It was the beginning of a 90-minute outage that cost them roughly EUR 28,000 in failed orders.

I asked him afterward why he ignored the first three. He said, “Because they’re always noise. CPU goes above 80% for two minutes and then it comes back. It happens twenty times a day.” I looked at their Alertmanager. He was right. In the last 30 days, their alerting system had fired 847 times. Of those, 812 were acknowledged and closed without action. That is a 96% false positive rate. Their alerting system was a noise machine that occasionally transmitted a signal, and the signal was indistinguishable from the noise.

This is the most common observability problem I see at startups. Not missing metrics. Not missing dashboards. Not missing alerts. The opposite. Too many alerts, configured by people who confused coverage with safety, producing a system that desensitizes the on-call engineer to the point where the real alert gets ignored.

Here is what I find, why it happens, and what I do about it.

The coverage fallacy

The instinct that produces alert noise is the belief that more alerts means more safety. If we alert on CPU, memory, disk, queue depth, error rate, latency, connection count, and pod restarts, we will catch everything. This is backwards. You do not catch everything. You catch nothing, because the human at the end of the pipeline stops paying attention.

The coverage fallacy treats alerting as a monitoring problem. It is not. It is a human attention problem. The scarce resource is not metrics. You can collect thousands of metrics for cents. The scarce resource is the on-call engineer’s attention, and every alert you send consumes some of it. An alert that fires and resolves without action has not improved safety. It has reduced the attention available for the next alert, which might be real.

I worked with a team in Amsterdam that had 62 alert rules in Prometheus. I asked them to pull the firing history for 90 days. 58 of the 62 rules had fired at least once. 41 had fired more than 10 times. 19 had fired more than 50 times. The team could not tell me, for 23 of the rules, what action they would take if the alert fired. “We would investigate” is not an action. An action is a specific thing you do: roll back, scale up, failover, page a specific team, open a ticket. If you cannot name the action, the alert is not actionable, and an alert that is not actionable is noise with a pager attached.

I deleted 34 of their 62 rules. The team was nervous. They felt they were losing visibility. They were not losing visibility. They were losing noise. The metrics were still collected. The dashboards still showed CPU, memory, disk, queue depth. What disappeared was the pager firing for things that did not require a human at 3am.

The two questions every alert must answer

Before I install or keep any alert rule, it must answer two questions. If it cannot answer both, it does not belong in the alerting system.

Question one: is this a symptom of a customer-facing problem, or a cause that might become one?

Symptom alerts tell you the system is broken in a way the user already sees. Error rate is up. Latency is up. The checkout endpoint is returning 500s. These are the alerts that matter, because they are the alerts that correspond to incidents. The customer is already affected. You need to know.

Cause alerts tell you something is wrong that might lead to a customer-facing problem. Disk is filling. Queue depth is growing. Certificate expires in 7 days. These are useful, but they are a different class of alert. They should not page. They should ticket. They should route to a channel where someone will handle them during business hours, not at 3am.

The mistake teams make is routing both classes through the same pager with the same priority. The on-call engineer gets paged for a disk that is at 82% and will hit 90% in four days. He gets paged at 3am for a four-day problem. He learns that pages are not urgent. Then the checkout breaks and he gets a page with the same urgency, and he treats it the same way.

Question two: what specific action will the on-call engineer take?

If the answer is “investigate,” the alert is a dashboard. Dashboards do not page people. They wait for people to look at them. If the answer is “page the database team,” the alert should route to the database team, not to the general on-call. If the answer is “scale the deployment,” the alert should trigger an autoscaler, not a human. If the answer is “there is no action, we just want to know,” the alert should be a log line, not a page.

These two questions eliminate most alert noise. The rules that survive are the ones that correspond to real incidents with real actions.

The rule I install: alert on symptoms, ticket on causes

Every team I work with gets the same alerting structure. Symptom alerts page. Cause alerts ticket. Dashboards show everything else. The structure looks like this in Alertmanager.

# alertmanager.yml
route:
  receiver: default
  group_by: ["alertname", "severity"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Pages: customer-facing symptoms only
    - matchers:
        - severity = page
      receiver: pagerduty
      group_wait: 10s
      repeat_interval: 30m
    # Tickets: causes that need handling, not paging
    - matchers:
        - severity = ticket
      receiver: slack-ops
      group_wait: 5m
      repeat_interval: 12h
    # Info: logged, no human action needed
    - matchers:
        - severity = info
      receiver: slack-info
      repeat_interval: 24h

receivers:
  - name: pagerduty
    pagerduty_configs:
      - routing_key: <key>
        severity: critical
  - name: slack-ops
    slack_configs:
      - api_url: <webhook>
        channel: "#ops-tickets"
        send_resolved: true
  - name: slack-info
    slack_configs:
      - api_url: <webhook>
        channel: "#ops-info"
        send_resolved: false
  - name: default
    slack_configs:
      - api_url: <webhook>
        channel: "#ops-info"

Three tiers. Page, ticket, info. The page tier is reserved for symptoms that mean a customer is affected right now. The ticket tier is for causes that need human action but not urgently. The info tier is for things that should be visible but do not require action. The routing is explicit. The on-call engineer knows that a page means something is broken. A Slack message means something needs attention. A dashboard means something to look at when you have time.

The difference between this and what most teams have is that most teams have one tier. Everything is a page. Everything has the same urgency. The on-call engineer cannot triage because there is nothing to triage against. Everything is red.

What actually pages: SLO burn rate

The symptom alerts that page are built on SLO burn rate, not on raw thresholds. This is the single biggest change I make to a team’s alerting, and it is the one that produces the most pushback, because it requires the team to define what “good” means before they can alert on “bad.”

An SLO is a promise about the user experience. “99.9% of requests succeed within 500ms over 30 days.” Once you have that, you can alert on burn rate, which is the rate at which you are consuming your error budget. If your SLO is 99.9% (0.1% error budget over 30 days), and your current error rate is 1%, you are burning budget 10x faster than you are allowed to. At that rate, you will exhaust your 30-day budget in 3 days. That is worth knowing. That is worth paging.

The Prometheus rules I install for every team.

# slo-alerts.yml
groups:
  - name: slo-burn-rate
    rules:
      # Fast burn: 14.4x burn rate over 1h window
      # Pages immediately. This means you exhaust your
      # 30-day budget in 2 hours.
      - alert: SLOBurnRateFast
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[1h]))
            /
            sum(rate(http_requests_total[1h]))
          ) > (14.4 * 0.001)
          and
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) > (14.4 * 0.001)
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "Fast SLO burn rate on {{ $labels.service }}"
          description: "Error rate is consuming 30-day budget in 2h. Page now."

      # Slow burn: 6x burn rate over 6h window
      # Tickets. This means you exhaust your 30-day budget
      # in 5 days. Handle during business hours.
      - alert: SLOBurnRateSlow
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[6h]))
            /
            sum(rate(http_requests_total[6h]))
          ) > (6 * 0.001)
          and
          (
            sum(rate(http_requests_total{status=~"5.."}[30m]))
            /
            sum(rate(http_requests_total[30m]))
          ) > (6 * 0.001)
        for: 15m
        labels:
          severity: ticket
        annotations:
          summary: "Slow SLO burn rate on {{ $labels.service }}"
          description: "Error rate is consuming 30-day budget in 5 days. Ticket."

Two windows, two severities. The fast burn alert pages because the budget is being exhausted in hours. The slow burn alert tickets because the budget is being exhausted in days. Both are based on the user experience, not on infrastructure internals. A CPU spike that does not affect error rate does not fire either alert. A database slowdown that causes 500s fires the fast burn alert. The alert corresponds to the incident, not to the cause.

The multi-window comparison (1h and 5m for fast, 6h and 30m for slow) prevents flapping. The long window confirms the trend. The short window confirms the current state. Both must be burning for the alert to fire. This eliminates the transient spike that resolves in 90 seconds and produces 812 false positives in 30 days.

The 14.4x and 6x multipliers come from the Google SRE workbook. 14.4x means you consume 2% of your 30-day budget in 1 hour, which is the threshold where paging is justified. 6x means you consume 1.5% of your 30-day budget in 6 hours, which is the threshold where a ticket is justified. These are not magic numbers. They are derived from the budget math, and they work because they are tied to a business-relevant quantity (the error budget) rather than an infrastructure metric that may or may not matter.

What does not page

Here is what I remove from every team’s alerting, and why.

CPU usage above a threshold. CPU at 85% is not a problem. CPU at 100% is not a problem if the application is still meeting its SLO. CPU is a resource, not a symptom. If high CPU is causing latency or errors, the SLO burn rate alert will fire. If high CPU is not causing latency or errors, there is nothing to do. Alerting on CPU directly produces pages for situations where the system is working fine, which teaches the on-call engineer to ignore pages.

Memory usage above a threshold. Same logic. Memory at 90% with a working garbage collector and no OOM kills is fine. Memory at 90% with rising GC pause times and increasing latency will trigger the latency SLO alert. Alert on the symptom.

Pod restarts. A pod restarting is Kubernetes doing its job. A pod restarting 10 times in 5 minutes might indicate a problem, but only if it affects the user. If the service stays available through the restarts, the restarts are not an incident. If the service degrades, the SLO alert fires. I see teams page on any pod restart, which means they page every time a node drains, every time a deployment rolls out, and every time a horizontal pod autoscaler replaces a pod. That is noise.

Disk usage below 90%. Disk filling is a real problem, but it is a cause, not a symptom. It gets a ticket, not a page. The one exception is disk usage above 95% on a database volume, because that can cause the database to stop accepting writes, which is a symptom. But even then, the SLO alert will catch the resulting errors. The disk alert is a heads-up that arrives earlier, and earlier is good for tickets, not pages.

Certificate expiry more than 7 days out. This is a ticket. Certificates expiring in 3 days is a page, because a certificate expiration will take down TLS and that is a customer-facing symptom. But 30 days out is a ticket. I see teams page on 30-day cert expiry, which means they page for something that has a month of runway. The on-call engineer learns that pages have a month of runway. Then the checkout breaks and he treats it like a cert renewal.

The latency SLO, because error rate is not enough

Error rate is the most obvious symptom, but latency is the second one. A request that takes 12 seconds is not a 500, but it is a failed request from the user’s perspective. They abandoned it. Your error rate looks fine. Your users are leaving.

I add a latency SLO alongside the error rate SLO. The SLO is “99% of requests complete within 500ms over 30 days.” The alerting is the same burn-rate structure, but the metric is a histogram quantile instead of an error counter.

      - alert: LatencySLOBurnRateFast
        expr: |
          (
            sum(rate(http_request_duration_seconds_bucket{le="0.5"}[1h]))
            /
            sum(rate(http_request_duration_seconds_count[1h]))
          ) < (1 - (14.4 * 0.01))
          and
          (
            sum(rate(http_request_duration_seconds_bucket{le="0.5"}[5m]))
            /
            sum(rate(http_request_duration_seconds_count[5m]))
          ) < (1 - (14.4 * 0.01))
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "Latency SLO burn rate on {{ $labels.service }}"
          description: "p99 latency exceeds 500ms budget at 14.4x burn rate. Page now."

The expression is inverted because we are measuring success rate (requests under 500ms) instead of error rate. (1 - success_rate) gives us the failure rate, and we compare that against the budget burn. The logic is identical to the error rate alert. The symptom is the same. The user is having a bad experience. Page.

This is the alert that catches the problem the Hamburg team missed. Their payment webhook was timing out at 12 seconds. Their error rate alert did not fire because the requests were not returning 5xx. They were returning 200 with a 12-second latency, or they were returning 504 after a timeout, which their monitoring classified as a client error, not a server error. Their latency SLO would have caught it in under 2 minutes. Instead, they caught it in 40 minutes, when a customer tweeted at them.

The cause alerts that earn their place

Not all cause alerts are noise. A small number of cause alerts are worth keeping because they give you enough runway to prevent the symptom from ever reaching the user. These are the ones I keep, with strict conditions.

Disk fill rate. If disk is filling at a rate that will hit 100% in under 24 hours, that is a ticket. If it will hit 100% in under 4 hours, that is a page, because a full disk on a database or a log volume will cause a symptom very quickly and the runway is too short for a ticket workflow.

      - alert: DiskWillFillIn24h
        expr: |
          predict_linear(node_filesystem_avail_bytes[2h], 24 * 3600) < 0
        for: 30m
        labels:
          severity: ticket
        annotations:
          summary: "Disk {{ $labels.device }} on {{ $labels.instance }} will fill in 24h"

predict_linear is the function most teams do not know about. It extrapolates the current trend forward. Instead of alerting on “disk is at 80%,” it alerts on “disk will be at 100% in 24 hours based on the last 2 hours of fill rate.” This is actionable. The on-call engineer knows the urgency from the prediction, not from an arbitrary threshold.

Queue depth growing without drain. If a background job queue is growing and the processing rate is not keeping up, that is a ticket if the queue will overflow in days, a page if it will overflow in hours. The symptom (jobs not running) will eventually surface, but the cause alert gives runway.

Certificate expiry under 7 days. Ticket at 30 days. Page at 7 days. The page at 7 days is justified because cert renewal automation can fail silently, and a cert expiration is a hard outage that is trivial to prevent and embarrassing to explain.

These cause alerts survive because they have a specific action (clear disk, scale workers, renew cert) and a specific urgency (the runway before the symptom hits). The cause alerts I delete are the ones without a specific action or without a meaningful runway. “CPU is high” has no specific action. “Disk is at 80%” has no meaningful runway. Both go.

The cleanup process I run with every team

When I arrive at a team with alert noise, I do not rewrite their alerting on day one. I run a four-step process that takes about two weeks.

Step one: pull the firing history. I export 90 days of Alertmanager firing data. For each rule, I count how many times it fired, how long it lasted, and whether it was acknowledged or auto-resolved. This produces a ranked list of the noisiest rules. The top 20% of rules typically produce 80% of the noise. Those are the first targets.

# Export firing history from Alertmanager (if logged to Loki)
logcli query --limit=10000 \
  '{job="alertmanager"} |= "resolve" or {job="alertmanager"} |= "firing"' \
  --since=2160h --output=jsonl > alert-history.jsonl

# Count firings per alert name
cat alert-history.jsonl | jq -r '.labels.alertname' | \
  sort | uniq -c | sort -rn | head -20

Step two: classify each rule. For every rule, I ask the team: what action does this trigger? Is it a symptom or a cause? What severity should it be? Rules that cannot answer the action question get deleted. Rules that are causes get demoted to ticket severity. Rules that are symptoms get kept at page severity.

Step three: implement the three-tier routing. I install the Alertmanager config with page, ticket, and info tiers. Every surviving rule gets a severity label. The routing ensures pages go to the pager, tickets go to Slack, and info goes to a low-priority channel.

Step four: measure for two weeks. I leave the team with a dashboard that tracks alert volume by severity. After two weeks, we review. If page volume is more than 1-2 pages per on-call shift, we look at what fired and whether those rules need tuning. The target is not zero pages. The target is that every page is real.

The Hamburg team I started this with went from 847 alerts in 30 days to 11. Of those 11, 9 were real incidents that the on-call engineer acted on. 2 were false positives that we tuned out in the following week. The on-call engineer who had ignored the payment webhook alert told me, a month later, that he now trusts the pager. “When it goes off, I know it’s real. I don’t have to decide whether to look at it. I just look.”

That is what alerting is supposed to do. Not transmit information. Transmit decisions. The decision the on-call engineer makes when the pager fires should be “something is broken and I need to act,” not “is this real or is this another CPU spike.” If the pager cannot give you that decision, it is not an alerting system. It is a noise machine with a pager attached, and the noise is costing you the signal you actually need.

The number that matters

I give every team one metric to track after the cleanup. Page-to-action ratio. Of the pages that fire in a given month, what percentage resulted in the on-call engineer taking a real action (rollback, scale, failover, manual fix). The target is 80% or higher. Below 80%, the pager is producing noise. Above 80%, the pager is producing signal.

Most teams I audit start at 5-15%. The Hamburg team was at 4%. After the cleanup, they were at 82%. The metric is simple to track and hard to game. If your page-to-action ratio is low, your alerting is broken. If it is high, your alerting is doing its job. Everything else is dashboard material.

The point of alerting is not to know everything. The point is to know the things that require a human to act, and to deliver those things in a way the human trusts. Trust is the scarce resource. Every false page spends it. Every real page earns it. The system you want is the one where the real page arrives and the human acts immediately, because they have learned, through repeated experience, that the pager does not lie.