Disaster recovery is not a backup strategy

A fintech I worked with last year had daily database snapshots, WAL archiving to S3, and a backup dashboard with thirty green checkmarks. They had not tested a restore in eleven months. When a botched migration corrupted their primary Postgres instance at 2:14am on a Sunday, they discovered three things in sequence.

First, the WAL archive had stopped uploading four days earlier because the IAM role had been rotated and nobody updated the backup service. Second, the last clean snapshot was from 6am Saturday, meaning nineteen hours of transactions were gone. Third, the runbook for database recovery referenced a restore.sh script that no longer existed in the repository. It had been deleted in a cleanup three months prior.

They were down for nine hours. They lost a day of financial transactions. They recovered eventually, but the trust damage lasted months. They had backups. They did not have recovery.

This is the pattern I see everywhere. Teams build backup systems and call it disaster recovery. Backups are the easy part. Recovery is the part that fails.

The distinction that matters

A backup is a copy of data stored somewhere else. Disaster recovery is the tested, documented, time-bounded process of restoring service from that copy after a failure. They are not the same thing, and conflating them is how companies end up with green dashboards and nine-hour outages.

The question I ask every team I audit is not “do you have backups.” It is “how long would it take you to recover, and when did you last prove it.” If the answer to the second part is silence, you do not have disaster recovery. You have archives.

There are two numbers that define whether you actually have a DR plan. RPO, recovery point objective, is how much data you can afford to lose. RTO, recovery time objective, is how long you can afford to be down. These are not SLAs you write in a document and forget. They are constraints that should shape every infrastructure decision you make. If your RTO is one hour, every component in the system must be recoverable in under one hour, and you must have tested that. If you have never tested it, your RTO is a guess.

What actually fails during recovery

I have run DR exercises for fifteen companies across fintech, e-commerce, healthcare, and logistics. The same five things fail every time.

Stale credentials. The backup service runs under an IAM role, a service account, or a set of API keys. Those credentials expire, rotate, or get revoked. The backup keeps running because the dashboard shows the last successful run, not the current one. The first time you discover the credential is dead is the moment you try to restore and the restore fails.

Missing tooling. The runbook says “run the restore script.” The script was deleted, moved, or never committed. The engineer who wrote it left. The team discovers this at 2am when they cannot find the thing they need to recover.

Untested restore paths. The backup file exists. The restore command runs. The data comes back. But the application does not work, because the database schema has changed since the backup was taken, or the application version expects a newer schema than the backup contains. The backup is valid. The recovery is not.

Dependency chains nobody mapped. The application depends on the database. The database depends on a secrets manager. The secrets manager depends on a KMS key. The KMS key is in a different region. Nobody documented this chain. When the primary region is down, the recovery stalls at the third link because nobody knows where the key is.

No rehearsal pressure. Teams test recovery in calm conditions, on a staging environment, with the engineer who built the system watching. Real recovery happens at 2am, under pressure, with someone who has never done it before. The conditions are different. The failure modes are different. The only way to know if recovery works under pressure is to test it under pressure.

The DR plan I install

Here is the disaster recovery setup I install for companies running Kubernetes. It is not exotic. It is the configuration I have seen survive actual failures, not theoretical ones.

Database: continuous WAL archiving with verified restores

For Postgres, the setup is WAL archiving to S3-compatible storage, with a base backup every six hours and continuous WAL shipping. The critical part is not the archiving. It is the restore test.

# postgresql.conf
archive_mode = on
archive_command = 'aws s3 cp %p s3://pg-wal-archive/{{ .Release.Name }}/%f'
archive_timeout = '5min'
wal_level = replica

The base backup runs via a CronJob in the cluster:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: pg-base-backup
  namespace: database
spec:
  schedule: "0 */6 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: postgres:16
            env:
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-superuser
                  key: password
            command:
            - /bin/sh
            - -c
            - |
              pg_basebackup -h postgres-primary -U postgres \
                -D /backup -Ft -z -P
              aws s3 sync /backup \
                s3://pg-backups/$(date +%Y-%m-%d-%H)/
          restartPolicy: OnFailure

Every Monday at 9am, an automated job restores the latest backup into a fresh Postgres instance, runs a smoke test query, and reports success or failure to Slack. If the restore test fails, the alert is louder than any application alert. A backup that cannot be restored is worse than no backup, because it creates false confidence.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: pg-restore-test
  namespace: database
spec:
  schedule: "0 9 * * 1"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: restore-test
            image: postgres:16
            command:
            - /bin/sh
            - -c
            - |
              # Pull latest base backup from S3
              aws s3 sync s3://pg-backups/latest/ /restore

              # Start a fresh Postgres instance
              initdb -D /var/lib/postgresql/testdata
              pg_ctl -D /var/lib/postgresql/testdata -l /tmp/log start

              # Restore base backup
              tar -xzf /restore/base.tar.gz -C /var/lib/postgresql/testdata

              # Create recovery.signal for WAL replay
              touch /var/lib/postgresql/testdata/recovery.signal
              echo "restore_command = 'aws s3 cp s3://pg-wal-archive/%f %p'" \
                >> /var/lib/postgresql/testdata/postgresql.auto.conf

              pg_ctl -D /var/lib/postgresql/testdata restart

              # Smoke test: can we query the restored data?
              psql -U postgres -d production \
                -c "SELECT count(*) FROM users WHERE created_at > now() - interval '7 days'"
          restartPolicy: OnFailure

If that job fails, the Slack message is: “Weekly restore test FAILED. Your backups may not be recoverable. Investigate before the next incident.” That message has saved three companies I know of from data loss they would have discovered during an actual outage.

Cluster: Velero with verified schedules

For the Kubernetes cluster itself, Velero is the standard. It backs up cluster resources and persistent volumes to S3. The configuration is straightforward. The discipline is not.

velero install \
  --provider aws \
  --bucket velero-backups \
  --backup-location-config region=eu-central-1 \
  --snapshot-location-config region=eu-central-1 \
  --use-volume-snapshots=true

Two schedules, not one:

# Daily cluster backup, 30-day retention
velero schedule create daily-cluster-backup \
  --schedule="0 2 * * *" \
  --include-namespaces production,monitoring,database \
  --ttl 720h

# Hourly backup of critical namespaces only, 24-hour retention
velero schedule create hourly-critical \
  --schedule="0 * * * *" \
  --include-namespaces production \
  --ttl 24h

The hourly schedule is the one that matters for RPO. If your critical namespace is destroyed at 1:45pm, you lose forty-five minutes of state, not twenty-four hours. The daily schedule is for the full cluster, in case the entire cluster is lost and you need to rebuild.

The restore test is the same pattern as the database. Once a month, spin up a fresh cluster, run velero restore, and verify the application comes up. I have seen Velero backups that were silently failing for weeks because the snapshot permission was wrong. The backup logs showed success because the resource backup succeeded. The volume snapshot failed. The restore would have brought back the pods but not the data.

etcd: the backup nobody thinks about

If you run your own Kubernetes control plane, etcd is the single most critical component. It holds the entire cluster state. Lose etcd, lose the cluster. Managed Kubernetes services (EKS, GKE, AKS) handle this for you. If you are on self-hosted Kubernetes, this is on you.

ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Snapshot etcd every six hours, ship to S3, and test the restore in a staging cluster monthly. An etcd restore without testing is a snapshot, not a recovery.

Configuration: Git is your recovery

This is the part most teams underutilize. Your entire cluster configuration, application manifests, Helm values, network policies, RBAC bindings, and ingress rules should be in Git. If the cluster is destroyed, the fastest path to recovery is not restoring from a Velero backup. It is git clone followed by argocd app sync.

The Git repository is your fastest recovery mechanism because it is already tested. Every deploy goes through it. Every change is reviewed. The moment you apply the manifests to a fresh cluster, you have a working system, not a backup that might work.

I configure every team with a deploy repository structure like this:

deploy-repo/
  apps/
    api/
      deployment.yaml
      service.yaml
      ingress.yaml
    worker/
      deployment.yaml
  infrastructure/
    cert-manager/
      values.yaml
    monitoring/
      values.yaml
    networking/
      policies/
    rbac/
      bindings.yaml
  README.md

When the cluster is gone, the recovery is: create a new cluster, install ArgoCD, point it at the deploy repo, sync. Ten minutes to a working cluster if the data lives in a managed database that was not affected. That is an RTO of ten minutes for the control plane and application layer. The data layer has its own RTO defined by the database recovery process above.

The exercise that actually matters

Backups without rehearsal are theory. I run a DR exercise with every team I work with, and I run it the way incidents actually happen.

I pick a time. Usually a weekday morning, not Friday afternoon. I tell one engineer, not the one who built the system, that the primary database is gone. I give them the runbook and a Slack channel. I watch. I time everything.

The first exercise always goes badly. That is the point. The engineer cannot find the restore script. The credentials are wrong. The backup is twelve hours older than expected. The application does not start after the restore because of a schema mismatch. Each failure becomes an action item. Each action item has an owner and a deadline.

The second exercise, a month later, goes better. The script is in the repo. The credentials are documented. The backup is current. The application starts. The third exercise, the team does not need me in the room. They run it themselves, document the result, and close the action items without prompting.

By the fourth exercise, recovery is boring. That is the goal. A boring recovery is a tested recovery. A tested recovery is the only kind that counts.

The numbers I have seen

I keep data from every DR exercise I run. Here is what the numbers look like across fifteen companies, from the first exercise to the fourth.

The first exercise: median time to recovery, four hours and twelve minutes. Median data loss, three hours. Six out of fifteen teams could not complete the recovery in the exercise window and had to abort.

The fourth exercise: median time to recovery, twenty-three minutes. Median data loss, forty-five minutes. All fifteen teams completed the recovery. The ones with GitOps and managed databases recovered in under fifteen minutes. The ones with self-hosted databases and manual processes recovered in thirty to forty.

The difference between four hours and twenty-three minutes was not new tooling. It was the same tools, configured properly, tested repeatedly, with runbooks that actually existed and credentials that actually worked.

What I tell every team

If you take one thing from this, take this: test your recovery before you need it. Not once. Not as a checkbox before an audit. On a schedule, with someone who did not build the system, under time pressure, with the runbook and nothing else. Every failure you find in the exercise is a failure you do not find at 2am during an actual outage.

The teams that survive disasters are not the ones with the most sophisticated backup systems. They are the ones who have rehearsed the recovery enough times that it is boring. Boring is the goal. Boring means tested. Boring means you will sleep through the night when it finally happens for real, because you already know the answer.