Your database does not belong in Kubernetes

A startup in Berlin called me on a Saturday morning. Their PostgreSQL database, running on a Kubernetes StatefulSet in their production cluster, had been down for two hours. The pod had been evicted during a node maintenance event that their cloud provider had notified them about but nobody had read. The pod rescheduled to another node. The PersistentVolumeClaim came with it, because the storage class supported dynamic provisioning. The pod started, Postgres tried to recover, and the WAL replay failed because the volume had been detached uncleanly and the filesystem was in a state that Postgres could not reconcile automatically.

They had no standby. They had no replica. They had no managed failover. They had a single Postgres pod with a single PVC, and when that pod died, their entire production stack went down with it. The API could not serve requests because it could not connect to the database. The queue workers piled up. The background jobs stopped. Two hours of 500s on a platform that was processing EUR 40,000 per day in transactions.

They had chosen to run Postgres in Kubernetes because everything else was in Kubernetes, and it felt simpler to have one deployment target. They had a Helm chart that installed Postgres with a PVC and a Service. It looked clean in their values.yaml. It was clean in the way that a locked door is clean. It does not tell you whether anyone is on the other side.

I spent four hours helping them recover. The recovery involved detaching the volume, attaching it to a standalone EC2 instance, running pg_resetwal on the corrupted WAL, mounting the filesystem, and manually extracting the data. They lost roughly two hours of writes that were in the WAL but not yet checkpointed to the data files. They did not know which two hours of writes were lost. They had to send an email to their customers explaining that some transactions from that morning might be missing and asking them to check.

This was not a freak accident. This was the predictable output of running a stateful, data-critical workload on a platform designed for stateless, ephemeral compute. I have seen this story, with variations, six times in the last two years. The variations are different. The ending is the same.

What StatefulSets actually guarantee

People read the Kubernetes documentation on StatefulSets and come away thinking that StatefulSets provide something they do not. Let me be precise about what a StatefulSet gives you and what it does not.

A StatefulSet gives you stable pod identity. Pod zero is always pod zero. It gives you stable persistent storage, bound by ordinal. PVC for pod zero stays with pod zero across reschedules. It gives you ordered startup and shutdown, so pod one does not start before pod zero is ready. It gives you a stable network identity through a headless service, so postgres-0.database.svc.cluster.local always resolves to the pod with ordinal zero.

What a StatefulSet does not give you:

It does not give you data safety. The PVC is a disk. If the disk is corrupted, the pod is corrupted. If the pod writes bad data, the PVC stores bad data. If the node dies and the storage driver does not support live detachment, your PVC is stuck on a dead node and your pod cannot reschedule.

It does not give you high availability. A single Postgres pod in a StatefulSet is a single point of failure. Adding a replica requires configuring Postgres replication manually, managing the replication relationship in your container, handling failover logic, and ensuring the replica promotes correctly when the primary dies. None of this is provided by the StatefulSet. You are building a database cluster inside a container orchestrator, and the orchestrator does not know your database is a database.

It does not give you automated backups. You need a sidecar or a CronJob or an operator that runs pg_dump or WAL archiving to S3. You need to test restores. You need to verify the backups are not corrupt. The StatefulSet does none of this.

It does not give you point-in-time recovery. That requires WAL archiving, which requires configuring archive_command in postgresql.conf, which requires a storage destination, which requires lifecycle management of archived WAL files. You are building this yourself.

It does not give you maintenance windows, security patching without downtime, or major version upgrades. You are doing all of that yourself, with rolling restarts that may or may not work depending on your replication setup and your application’s tolerance for connection drops.

The storage problem

The deepest issue is storage. Kubernetes was designed around the assumption that compute is ephemeral and storage is persistent and detachable. That assumption holds for stateless applications, where a pod can die and reschedule and reattach its config. It does not hold for databases, where the pod must not die, and if it does, the storage must detach cleanly and reattach to a new pod without corruption.

The detachment problem is real and I have watched it fail. When a node dies ungracefully, a power event, a kernel panic, a hypervisor fault, the kubelet does not get to run its shutdown hooks. The pod is terminated without SIGTERM. The PVC is left attached to a dead node. The storage driver may or may not support forced detachment. If it does not, your pod is stuck in ContainerCreating forever because the PVC cannot attach to the new node while the old node still holds the attachment.

# The error you see when this happens
Events:
  Type     Reason              Age   From                     Message
  ----     ------              ----  ----                     -------
  Warning  FailedScheduling    12m   default-scheduler        0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.
  Warning  FailedAttachVolume  10m   attachdetach-controller  AttachVolume.Attach failed for vol-xxx : Volume is already exclusively attached to node-i-abc123 and cannot be attached to another node

You can force-detach the volume. That is a manual operation through the cloud API or the Kubernetes pv controller. It looks like this:

# Force detach the volume from the dead node
kubectl patch pv pvc-xxx --type json -p '[{"op": "replace", "path": "/spec/claimRef", "value": null}]'
# Or through the cloud provider CLI
aws ec2 detach-volume --volume-id vol-xxx --instance-id i-dead-node --force

Then you wait for the volume to become available. Then you hope the filesystem is not corrupted. Then you hope Postgres can recover from the WAL. If any of these steps fail, you are in manual recovery territory, which is where the Berlin startup found itself on a Saturday morning.

This is not a rare edge case. Node failures happen. Cloud providers do maintenance. Kubelets crash. Hypervisors fault. The question is not whether your node will die. The question is what happens to your database when it does, and the answer for a StatefulSet with a single replica is usually: downtime, followed by manual recovery, followed by data loss that you discover after the fact.

What operators actually do, and what they do not

The response I hear from Kubernetes advocates is that you should use an operator. The CloudNativePG operator for Postgres. The Percona Operator for MySQL. The Operator Lifecycle Manager. These operators handle replication, failover, backups, and WAL archiving. They solve the problems I just described.

They do solve some of them. A well-configured Postgres operator will give you a primary and a replica, automated failover, WAL archiving to S3, and point-in-time recovery. This is better than a raw StatefulSet. It is also where the complexity becomes difficult to justify.

The CloudNativePG operator is a good piece of software. I have used it. It manages Postgres clusters inside Kubernetes with a declarative API. You define a Cluster resource, and the operator creates the primary, the replicas, the backup configuration, and the replication slots. Failover is automated. Backups are scheduled. WAL archiving is configured.

# CloudNativePG Cluster resource
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: postgres-prod
spec:
  instances: 3
  postgresql:
    parameters:
      max_connections: "200"
      shared_buffers: "4GB"
      wal_level: "replica"
  backup:
    barmanObjectStore:
      destinationPath: s3://my-bucket/postgres-backups
      s3Credentials:
        accessKeyId:
          name: aws-creds
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: aws-creds
          key: SECRET_ACCESS_KEY
      retentionPolicy: "30d"
  storage:
    storageClass: gp3
    size: 200Gi

This works. It is also a system you now have to operate. The operator itself is a workload you have to monitor. The operator’s etcd is a dependency. The operator’s webhook is a dependency. The backups in S3 are a dependency. The replication between pods is a dependency. When the operator pod dies, failover does not happen. When the webhook is down, you cannot apply changes. When the backup job fails, you do not know until you check.

You have built a database management system inside Kubernetes, and you are now the DBA for that system. The complexity you offloaded to the operator is complexity you absorbed into your operational surface area. You did not eliminate it. You moved it.

I have never met a startup that had the engineering capacity to operate a Postgres operator in production at the same quality level as a managed Postgres service. Not once. The startups that try it are startups that end up calling me on a Saturday morning.

The cost argument does not hold

The argument for running databases in Kubernetes that I hear most often, after the “one deployment target” argument, is cost. Managed databases are expensive. A managed Postgres instance on AWS RDS with 4 vCPU and 16GB of RAM costs roughly EUR 250 per month. Running the same Postgres on a Kubernetes node you already have costs nothing extra, because the node is already running.

This is true in the same way that cooking at home is cheaper than eating at a restaurant. It is true until you account for the time, the cleanup, the ingredient waste, and the meal you burned and had to reorder.

The cost of a managed database is not just the compute. It is the backups, the automated failover, the patching, the minor version upgrades, the monitoring, the point-in-time recovery, the read replicas, the performance insights, and the 24/7 support from people whose entire job is keeping that database running. If you price the engineering hours required to replicate even half of that, the managed database is cheaper at any startup scale below a dedicated database team.

I did this math for a client last year. They were running two Postgres instances in Kubernetes with the CloudNativePG operator. One primary cluster, one analytics replica. They had one engineer spending roughly 20% of their time on database operations: monitoring the operator, managing backups, handling failover during node maintenance, tuning Postgres parameters, debugging replication lag. That 20% of a EUR 80,000 engineer is EUR 16,000 per year. The managed equivalent, two RDS instances with the same specs, would have cost EUR 6,000 per year. They were paying EUR 10,000 more per year to run it themselves, and the self-managed version had already caused one incident that cost them a day of downtime.

The cost argument only works if you do not value the time of the engineer operating the database. Startups that do not value engineering time do not stay startups for long.

Where your database should run

For the majority of startups, the answer is a managed database service. AWS RDS, GCP Cloud SQL, Azure Database for PostgreSQL, or one of the focused providers like Neon, Supabase, or Crunchy Bridge if you want something more developer-friendly. These services handle the operations that you will get wrong if you do them yourself: backups, failover, patching, recovery.

The configuration I install for startups is Postgres on RDS (or Crunchy Bridge if they are cloud-agnostic) with the following baseline:

# RDS configuration I start with
aws rds create-db-instance \
  --db-instance-identifier postgres-prod \
  --db-instance-class db.r6g.large \
  --engine postgres \
  --master-username postgres \
  --master-user-password <password> \
  --allocated-storage 200 \
  --storage-type gp3 \
  --backup-retention-period 14 \
  --multi-az \
  --enable-iam-database-authentication \
  --storage-encrypted \
  --deletion-protection

The --multi-az flag gives you a synchronous standby in a different availability zone. Failover is automated and takes under 90 seconds. The --backup-retention-period 14 gives you 14 days of automated backups with point-in-time recovery down to the second. The --deletion-protection flag prevents the database from being deleted accidentally, which is the kind of thing you think you do not need until someone runs the wrong Terraform plan.

The cost of this on a db.r6g.large is roughly EUR 180 per month. For a startup processing real revenue, this is not a meaningful line item. For a startup that has not raised money, Crunchy Bridge starts at roughly EUR 25 per month for a single-instance Postgres with automated backups, which is the same price as running it in Kubernetes on a node you already have, minus the operational burden.

The networking between your Kubernetes cluster and your managed database is one VPC peering connection or one private endpoint. It is not complex. Your pods connect to postgres-prod.xyz.eu-west-1.rds.amazonaws.com:5432 and that is it. You do not need a Service, an Endpoint, a StatefulSet, a PVC, an operator, or a webhook. You need a connection string and a password in a secret.

# The entire Kubernetes manifest for your database connection
apiVersion: v1
kind: Secret
metadata:
  name: postgres-credentials
type: Opaque
stringData:
  DATABASE_URL: "postgresql://postgres:password@postgres-prod.xyz.eu-west-1.rds.amazonaws.com:5432/mydb?sslmode=require"

That is the entire surface area. Compare that to the CloudNativePG Cluster YAML, the operator deployment, the backup configuration, the WAL archiving, the failover testing, and the Saturday morning recovery. The difference is not marginal. It is the difference between a database that is someone else’s problem and a database that is your problem at 2am.

When running a database in Kubernetes is actually fine

I want to be fair. There are cases where running a database in Kubernetes is the right call, and I do not want to pretend they do not exist.

Development and staging environments. Running Postgres as a StatefulSet in a dev cluster is fine. The data is not critical, the SLA is zero, and if the pod dies and the data is lost, you rebuild it from a seed. The convenience of having everything in one cluster outweighs the operational risk because there is no operational risk.

Redis and other caches. Caches are ephemeral by design. If a Redis pod dies and the cache is lost, your application degrades but does not break. You rebuild the cache from the database. Running Redis in Kubernetes is a different proposition from running Postgres, because the data is reconstructable and the loss is tolerable.

Test databases that are seeded and torn down. If you need a Postgres instance for integration tests that spins up, runs tests, and is deleted, Kubernetes is a good fit. The ephemeral nature of the workload matches the ephemeral nature of the platform.

Specialized workloads where you need full control of the database configuration and you have a team that can operate it. If you are a company whose product is a database, or you are running a database that requires a configuration that no managed service supports, Kubernetes with an operator may be your best option. This is not the case for 99% of startups.

The pattern is consistent. Run stateless workloads in Kubernetes. Run stateful workloads that tolerate data loss in Kubernetes. Run stateful workloads that do not tolerate data loss on a managed service. This is not a limitation of Kubernetes. It is a recognition that different workloads have different operational requirements, and the platform that is best for your API is not necessarily the platform that is best for your data.

The deeper point

The deeper point is about knowing what a tool is for. Kubernetes is a container orchestrator. It is a very good one. It schedules pods, it manages their lifecycle, it routes traffic, it handles rollouts and rollbacks. It was designed by Google for stateless workloads that can be killed and rescheduled without data loss. That design assumption is baked into every abstraction it provides.

Databases violate that assumption. A database is a workload where killing the process and rescheduling it is the worst case, not the expected case. A database is a workload where the data on disk is the entire point, and losing it is not a degraded experience, it is a catastrophic one. Running a database on a platform that treats pod death as routine is a category error. You are using a tool for something it was not designed for, and you are paying for the mismatch with operational complexity and incident risk.

The Berlin startup moved their database to RDS the week after the incident. The migration took two days. They have not had a database incident since. Their Kubernetes cluster is simpler, their operations are simpler, and their on-call rotation no longer includes “what to do if the Postgres pod is stuck in ContainerCreating.” The complexity they removed was complexity they never needed.

Your database is the one piece of your infrastructure where the managed service is unambiguously better than the self-hosted option, at every startup scale, for every team that is not a dedicated database team. Run your API in Kubernetes. Run your workers in Kubernetes. Run your queue, your cache, your service mesh, your ingress, your cert manager, all of it in Kubernetes. Run your database somewhere that someone else is responsible for it not losing data. That is the line, and it is a line I will defend in any architecture review.