The 4am pager: why your on-call rotation is a cultural problem, not a technical one

The first thing I see when I audit an SRE practice is the alerting rules. The second thing I see is the on-call schedule. By the third thing, I already know whether the team is healthy or not.

It is never the tools. Everyone has PagerDuty or Opsgenie. Everyone has a runbook wiki. Everyone has a Slack channel for incidents. The tools are solved. The culture is not.

The team that called me

A Series B startup in Berlin. Twelve engineers. A healthy PagerDuty rotation in theory: two primaries, two secondaries, weekly rotation. On paper, everyone was on call one week out of six. In practice, one engineer, the senior who had built the original infrastructure, was getting paged 80% of the time because the alerts were all routed to the person who could fix them fastest.

He was burning out. He told me he had not slept through a full weekend in four months. His partner was losing patience. He was thinking about quitting.

The CTO called me because the on-call rotation was not working. What he meant was: the senior engineer was about to leave and nobody else could keep the system running.

The diagnosis

I spent a week reviewing their alerting. Here is what I found:

• 340 active alerting rules across three Prometheus instances.
• 187 of those rules had fired in the last 30 days.
• 42 of those were actionable. The rest were noise, duplicate alerts, or alerts for systems that were deprecated but never cleaned up.
• The runbook wiki had 89 pages. 14 were updated in the last year. The rest referenced infrastructure that no longer existed.
• The on-call schedule had the senior engineer as primary 4 out of every 6 weeks because he was the only one who knew which alerts to ignore.

The problem was not the rotation. The problem was that nobody had done the work of making the system operable by anyone other than the person who built it.

The fix is boring, which is the point

I have run this same audit for teams in Dubai, Amsterdam, Tallinn, and Singapore. The pattern is always the same. The fix is always the same. And it is always boring.

Step one: kill the noise

We went through all 340 alerting rules. For each one, I asked two questions:

1. Does this alert require human action?
2. If yes, can any engineer on the team take that action, or does it require tribal knowledge?

If the answer to question one was no, the alert was deleted. If the answer to question two was “only the senior engineer knows,” the alert either got a runbook or got deleted.

We went from 340 rules to 78 in three days. The page count dropped by 85% in the first week.

Step two: write runbooks that a stranger could follow

A runbook that says “restart the payment service” is not a runbook. It is a hint. A runbook says:

The payment-service pod may enter a crash loop when the Stripe webhook
endpoint returns 5xx for more than 60 seconds. The liveness probe will
restart the pod, but the underlying issue is the Stripe integration, not
the pod.

1. Check kubectl logs -n payments payment-service-* | grep stripe
2. If you see "connection refused", check the Stripe status page:
   https://status.stripe.com
3. If Stripe is down, set the payment-service replicas to 0 and enable
   the maintenance page: kubectl apply -f overlays/maintenance/payment.yaml
4. Page the payments lead: @payments-oncall
5. Do NOT roll back the deployment. This is an upstream issue.

That is a runbook. A new engineer reading it at 4am can take the right action without understanding the entire system.

Step three: rotate the knowledge, not just the pager

The on-call rotation was broken because the knowledge was not rotating. Only one person knew the system deeply enough to respond to the hard alerts.

We fixed this with shadow on-call. Every week, one engineer was paired with the on-call primary as a shadow. They were paged at the same time. They responded together. The senior engineer explained what he was doing and why, in the incident channel, in real time.

After four weeks of shadowing, every engineer on the team had been through at least two real incidents with the senior engineer present. The tribal knowledge started to spread.

Step four: blameless postmortems that actually change something

They had postmortems. They were blameless in the sense that nobody got fired. But they were also useless in the sense that the same incident types kept recurring.

I changed one thing about their postmortems: every one had to end with at least one action item that would prevent this class of incident, not just this instance. Not “we will monitor the Stripe endpoint more closely.” But “we will add a circuit breaker to the payment service that trips after 5 consecutive 5xx responses and enables the maintenance page automatically.”

Action items had owners and deadlines. If an action item was not closed in two weeks, it escalated to the CTO. Not as punishment, as visibility.

What on-call should feel like

When on-call is working, the pager goes off and the engineer thinks: I know what this is, I know what to do, and I know I will be back asleep in ten minutes.

When on-call is not working, the pager goes off and the engineer thinks: I do not know what this is, I do not know who to ask, and I might be up for two hours.

The difference between those two experiences is not the tooling. It is the work someone put into making the system operable by people who did not build it. That work is boring, unglamorous, and invisible until it is missing. But it is the difference between a team that can sustain on-call and a team that loses its best engineer at 4am.

The real cost of bad on-call

That senior engineer in Berlin did not quit. He got his weekends back. He started sleeping through the night. The team’s mean time to resolution dropped from 47 minutes to 12, not because they got faster, but because the noise stopped drowning the signal.

The CTO told me later that the thing that surprised him most was not the reduction in pages. It was the change in the team’s mood. Engineers who had been quiet in incident channels started speaking up. Engineers who had been afraid of on-call started volunteering for extra shifts.

The system did not change. The culture did. And the culture changed because someone did the boring work of making the system safe to operate at 4am by someone who did not build it.