Why your startup needs runbooks before it needs a hire

A founder in Berlin told me he needed to hire a DevOps engineer. His startup had just closed a Series A, the team was twelve engineers, and the infrastructure was starting to hurt. Deploys were manual. The database had been down twice in the last quarter. Nobody knew how the SSL certificates were renewed, because the contractor who had set them up had left six months ago. The founder’s solution was to hire someone who would fix all of this. I asked him what would happen if the database went down right now, today, while the requisition was open. He said the backend lead would figure it out. I asked the backend lead. He said he would Google it.

This is the pattern. The team wants to hire their way out of operational fragility. The hire will fix it. Except the hire arrives into an environment with no documentation, no runbooks, no recorded knowledge of how anything works, and spends the first six months reverse-engineering the infrastructure they were hired to improve. The runbooks get written, eventually, by the person who was hired to do higher-level work, at the salary of a senior engineer, doing junior-level documentation. This is the most expensive way to write runbooks I have seen, and I have seen it at six companies.

The runbooks should exist before the hire. Not because the current team has time to write them, but because the process of writing them surfaces the problems the hire needs to solve. A team that writes runbooks discovers that nobody knows how the certificates renew. A team that writes runbooks discovers that the database failover has never been tested. A team that writes runbooks discovers that the deploy process has seven undocumented steps that live in one engineer’s head. These are the things you want to know before you write the job description, not after you make the hire.

What a runbook actually is

A runbook is not documentation. Documentation describes how a system works. A runbook describes what to do when it does not. The distinction matters because the audience is different. Documentation is read by someone who has time to learn. A runbook is read by someone who is being paged at 3am and has ten minutes to stop the bleeding.

I have audited runbooks at roughly thirty companies. The most common failure is that they are written like documentation. They explain the architecture, describe the components, and provide background context. None of this helps at 3am. The person being paged does not need to understand the architecture. They need to know which command to run, which service to restart, and which metric to watch to confirm the fix worked.

A real runbook has four sections. Symptom: what the alert looks like, what the user experience is. Impact: what is affected and what is not. Mitigation: the specific steps to reduce or eliminate the impact, in order, with exact commands. Escalation: who to call if the mitigation does not work, with phone numbers, not Slack handles.

Here is a runbook I wrote for a database failover. Not a theoretical one. The one that sits in the repo of a company I worked with, and has been used twice.

# Runbook: Primary database unavailable

## Symptom
- Alert: `pg_primary_down` firing in PagerDuty
- Application returns 500 on all write requests
- Read requests may succeed if replica is reachable

## Impact
- All user actions that write to the database fail
- Read-only pages (catalog, public profiles) continue working
- Estimated affected users: all active sessions

## Mitigation

### Step 1: Confirm the primary is down (2 minutes)
ssh to the bastion:
  ssh ops@bastion.prod.internal
Check connectivity:
  psql -h db-primary.internal -U healthcheck -c "SELECT 1"
If this returns within 5 seconds, the primary is up. The alert is a false
positive. Acknowledge and close.

### Step 2: Promote the replica (5 minutes)
Run from the bastion:
  pg_ctl promote -D /var/lib/postgresql/data
Verify the replica is now accepting writes:
  psql -h db-replica.internal -U healthcheck -c "SELECT pg_is_in_recovery()"
Must return: f (false means it is now a primary)

### Step 3: Update the connection string
Update the DNS record to point db.internal to the promoted replica:
  aws route53 change-resource-record-sets \
    --hosted-zone-id Z12345678 \
    --change-batch '{"Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"db.internal","Type":"CNAME","TTL":30,"ResourceRecords":[{"Value":"db-replica.internal"}}]}}]'
Wait 30 seconds for DNS propagation.

### Step 4: Verify application recovery
curl -X POST https://app.example.com/health/write
Must return: {"status":"ok"}
If this fails, see Escalation.

## Escalation
- On-call DBA: +49 30 12345678 (rotation: check PagerDuty schedule)
- Aayan Mateen (contractor): +49 ... (available for severe incidents)
- If neither is reachable and the application is down > 15 minutes,
  initiate the disaster recovery procedure in runbooks/dr-full-restore.md

This runbook is not elegant. It is not clever. It is a list of commands that a stressed engineer can copy and paste at 3am. Every command has been tested. Every path has been walked. The DNS TTL is set to 30 seconds because we tested failover and 30 seconds was the propagation time. The health check endpoint includes a write, not just a read, because a read-only health check would pass even if the database connection was broken.

The runbook took four hours to write. It took another two hours to test, because we actually ran the failover in staging to confirm every step worked. Six hours total. The last time it was used in production, the database failover took seven minutes from alert to recovery. Without the runbook, the same failover took forty-three minutes, because the on-call engineer had to figure out the DNS update, the promotion command, and the verification step from scratch while the application was down.

The five runbooks every startup needs

I install the same five runbooks at every startup I work with. They cover the failure modes that actually happen, not the failure modes that are theoretically interesting. If you only write five, write these.

One: database unavailable

The runbook above. Database failover, connection string update, verification. This is the runbook that gets used most often, because database issues are the most common cause of full-platform outages at startups. The runbook must include the exact failover command, the DNS update, and a write-path health check. If your database failover has never been tested, the runbook is fiction. Test it.

Two: certificate expiry

SSL certificates expire. Let’s Encrypt certificates expire every 90 days. If your renewal is automated with certbot, the expiry is a silent failure that you discover when the browser shows a security warning. If your renewal is manual, the expiry is a calendar event that someone forgot.

# Runbook: SSL certificate expired

## Symptom
- Browser shows "Your connection is not private" on all pages
- Alert: `cert_expiry_7d` or `cert_expired` in PagerDuty
- curl to the domain returns: SSL certificate problem: certificate has expired

## Mitigation

### Step 1: Check which certificate expired
ssh to the load balancer:
  ssh ops@lb.prod.internal
Check the certificate:
  openssl s_client -connect localhost:443 -servername app.example.com </dev/null 2>/dev/null \
    | openssl x509 -noout -dates
Look at the notAfter date. If it is in the past, the cert is expired.

### Step 2: Renew with certbot
  sudo certbot renew --nginx
If this fails because the DNS challenge cannot complete:
  sudo certbot certonly --manual --preferred-challenges dns -d app.example.com
Follow the prompts to add the TXT record.

### Step 3: Verify
  curl -vI https://app.example.com 2>&1 | grep "expire date"
Must show a date in the future.

## Escalation
- Aayan Mateen: +49 ...
- If the domain registrar is unreachable, the fallback is to switch the
  DNS to the backup load balancer which has a valid certificate.

I have seen certificate expiry take down a fintech startup for three hours because the renewal script had silently broken six weeks earlier and nobody had noticed. The runbook exists so that when the alert fires, the fix is seven minutes of copy-paste, not three hours of investigation.

Three: deployment rollback

# Runbook: Rollback a bad deployment

## Symptom
- Error rate spike after a recent deploy
- Alert: `http_5xx_rate` above 5% in the last 10 minutes
- A deploy was made in the last 30 minutes (check ArgoCD or CI history)

## Mitigation

### Step 1: Identify the last good version
Check the deployment history:
  kubectl rollout history deployment/api -n production
Find the revision before the current one.

### Step 2: Roll back
  kubectl rollout undo deployment/api -n production
If using ArgoCD, change the image tag in the Git repo and let ArgoCD sync:
  # In the infra repo:
  sed -i 's/api:v1.4.3/api:v1.4.2/' apps/production/api.yaml
  git commit -am "rollback: api to v1.4.2"
  git push

### Step 3: Verify
  kubectl rollout status deployment/api -n production
  curl https://app.example.com/health
Error rate should drop within 2 minutes.

## Escalation
- If the rollback fails, the previous image may be missing from the registry.
  Check: docker pull ghcr.io/team/api:v1.4.2
  If the pull fails, rebuild from the previous commit:
    git checkout <previous-commit-sha>
    docker build -t ghcr.io/team/api:v1.4.2 .
    docker push ghcr.io/team/api:v1.4.2
  Then retry the rollback.

The rollback runbook is the one that should be tested most often and is tested least. Teams assume rollback works because the pipeline has a rollback button. The button works when the previous image exists in the registry. It does not work when the registry retention policy deleted the old image, or when the deployment was a config change, not an image change. Test the rollback. I have watched teams discover, mid-outage, that their rollback button does not work because the previous image was garbage-collected two weeks ago.

Four: disk full

# Runbook: Disk space full on a production node

## Symptom
- Alert: `node_disk_usage > 90%`
- Pods failing to start with: CreateContainerError
- Logs show: no space left on device

## Mitigation

### Step 1: Identify what is using the space
ssh to the node (or use kubectl debug):
  df -h
  du -sh /var/lib/docker/* | sort -rh | head -10
  du -sh /var/log/* | sort -rh | head -10

### Step 2: Clear Docker artifacts
  docker system prune -a --volumes --filter "until=48h"
This removes unused images, stopped containers, and old volumes.
Do NOT run without the filter. It will remove everything including
running container data if misused.

### Step 3: Rotate logs
  journalctl --vacuum-time=2d
  truncate -s 0 /var/log/app/*.log

### Step 4: Verify
  df -h
Disk usage should be below 70%.

## Escalation
- If disk usage is from the database volume, do NOT prune. Contact the DBA.
- If the node is unrecoverable, cordon and drain it, then provision a
  replacement from the autoscaling group.

Disk full is the most boring alert and the one that causes the most avoidable outages. The runbook exists because the instinct is to rm -rf something, and the something is often the database. The runbook constrains the response to safe operations.

Five: DNS outage

# Runbook: DNS resolution failing

## Symptom
- Application cannot reach external services (Stripe, SendGrid, etc.)
- Error: "lookup api.stripe.com on 10.0.0.2:53: no such host"
- Internal services may also fail if they use DNS for discovery

## Mitigation

### Step 1: Check the DNS server
  nslookup api.stripe.com
  nslookup api.stripe.com 8.8.8.8
If the second command works, the internal DNS server is the problem.

### Step 2: Restart CoreDNS (if using Kubernetes)
  kubectl rollout restart deployment/coredns -n kube-system

### Step 3: Check Route53 (if using AWS)
  aws route53 get-health-check-status --health-check-id <id>
If the health check is failing, the issue is upstream. Check the
AWS status page.

### Step 4: Fallback to IP addresses
If DNS is fully down, update the application config to use direct IPs
for critical external services. This is a last resort and should be
reverted within 24 hours.

## Escalation
- If the domain registrar is the issue, contact them directly.
- Keep a printed copy of critical IP addresses in the office.

DNS is the infrastructure layer that everyone depends on and nobody monitors. I have seen a startup go down for two hours because their Route53 health check was misconfigured and nobody noticed until the application stopped resolving. The runbook is short because the fix is usually short, but the diagnosis without a runbook is long.

The hire argument

The Berlin founder hired his DevOps engineer. Good person, three years of experience, comfortable with Kubernetes and Terraform. The engineer’s first month was spent documenting the infrastructure. Not building, not improving, documenting. The SSL renewal process. The database failover procedure. The deploy pipeline. The certificate chain. The DNS configuration. The things that should have been written down before the hire, by the people who built the system, while they still remembered how it worked.

The engineer’s second month was spent testing the runbooks. Running the failover in staging. Running the rollback. Simulating a disk full event. The things that should have been tested before the hire, because testing them revealed three failures that would have been outages in production. The database failover did not work because the replica was six hours behind on replication. The rollback did not work because the previous image had been garbage-collected. The disk full alert did not fire until 97% because the threshold was set too high.

These were not exotic problems. They were the standard problems of a startup that had grown past its initial infrastructure and had not invested in operational readiness. The hire found them. The hire fixed them. But the hire cost EUR 75,000 in salary for the first six months, and the first six months were spent on work that the existing team could have done, and should have done, before the requisition opened.

The runbooks cost six hours each to write. Five runbooks, thirty hours. A week of engineering time. The team had twelve engineers. The runbooks could have been written in an afternoon if each engineer took one. Instead, the startup paid a senior engineer EUR 37,500 to write them over three months. The math is not complicated.

What I tell founders

When a founder tells me they need to hire a DevOps engineer, I ask them two questions. Do you have runbooks? Have you tested them? If the answer to either is no, the hire is premature. Not because the hire is not needed, but because the team is not ready to receive the hire. A DevOps engineer arriving into an undocumented environment spends their first quarter writing documentation, not improving the system. The improvement starts in month four. The documentation could have been written in week one, by the team, before the hire arrived.

Write the five runbooks. Test them in staging. Fix the failures the tests reveal. Then open the requisition. The hire will start on real work on day one instead of month four, and the team will already know where the bodies are buried, because the runbooks forced them to look.

If you are a startup reading this and recognizing the pattern, Basecamp exists for exactly this reason: digitalaultis.com/basecamp

The runbooks are not the goal. The runbooks are the forcing function. The act of writing them is what surfaces the problems. The act of testing them is what proves the fixes work. The hire is what scales the fixes. But the runbooks come first, and they always come first, because you cannot hire your way out of a problem you have not yet defined.