Your graceful shutdown is not graceful enough

A team in Berlin invited me in after their service started dropping requests every time they deployed. Not a lot. Maybe one in five hundred requests failed during a rollout. Their deploy was a standard rolling update in Kubernetes. Six replicas, maxSurge: 1, maxUnavailable: 0. They had a SIGTERM handler in their Go service that closed the HTTP server, then closed the database connection pool. They had read the documentation. They had done everything the guides tell you to do. And they were still dropping requests on every deploy.

I watched a deploy happen. A new pod came up. An old pod entered Terminating. Within two seconds, the old pod’s logs showed received SIGTERM, shutting down. The HTTP server closed immediately. The database pool closed. The process exited in under a second. Kubernetes removed the pod. Clean, fast, correct. Except that during those two seconds, the cluster’s load balancer was still sending new requests to the pod that was already shutting down. Those requests hit a closed HTTP server and got a connection refused. The client saw a 502. The deployment succeeded. The metrics looked fine because one failed request in five hundred does not move the error rate needle. But the users who hit those requests saw a broken page, and some of them were in the middle of a checkout flow.

This is the most common shutdown problem I see, and it is not a bug in the application. The application does exactly what the SIGTERM handler says. The problem is that the SIGTERM handler is correct in isolation and wrong in the context of how Kubernetes terminates pods. The gap between those two things is where requests die.

What Kubernetes actually does when it terminates a pod

Most engineers I talk to think pod termination works like this: Kubernetes sends SIGTERM, the application shuts down, Kubernetes removes the pod from the service. That is roughly the right shape but the ordering is wrong, and the ordering is the whole problem.

Here is what actually happens, step by step.

Step one. The pod enters Terminating. Kubernetes sets pod.metadata.deletionTimestamp and begins the shutdown sequence.

Step two, in parallel. Kubernetes sends SIGTERM to the container’s PID 1. At the same time, Kubernetes begins removing the pod from the endpoint lists of all services that select it. These two things happen concurrently. They are not coordinated. The SIGTERM is delivered immediately. The endpoint removal is an API call that propagates through the control plane to kube-proxy on every node, and from kube-proxy to the iptables or IPVS rules that actually route traffic. This propagation takes time. On a healthy cluster it takes one to three seconds. On a busy cluster, or a cluster with many services, or a cluster where the endpoint controller is behind, it takes longer.

Step three. The application receives SIGTERM and starts shutting down. If the application closes its HTTP server immediately, which is what most SIGTERM handlers do, it stops accepting connections within milliseconds of receiving the signal.

Step four. The kube-proxy on a node that has a client making a request still has the old pod’s IP in its iptables rules. The endpoint removal has not propagated yet. The client’s next request gets routed to the pod that just closed its HTTP server. Connection refused.

This is the race. The application shuts down faster than the control plane propagates the endpoint removal. The window is small, usually one to three seconds, but it is nonzero on every deploy and every scale-down event. On a service doing a hundred requests per second, a two-second window is two hundred failed requests. On a service doing a thousand requests per second, it is two thousand. Most teams never notice because their error rate dashboards average over windows long enough to hide it. Their users notice because they are the ones hitting the failed requests.

The preStop hook is not a hack, it is the mechanism

The fix for this race is a preStop hook. I keep meeting engineers who think preStop hooks are a workaround, a hack for applications that do not handle SIGTERM correctly. They are not. The preStop hook is the mechanism Kubernetes provides specifically for this race. It runs before SIGTERM is delivered, and its purpose is to give the control plane time to propagate the endpoint removal before the application starts shutting down.

Here is the configuration I install for every HTTP service in Kubernetes.

spec:
  terminationGracePeriodSeconds: 30
  containers:
    - name: app
      image: ghcr.io/digitalaultis/app:latest
      lifecycle:
        preStop:
          exec:
            command: ["sleep", "5"]
      ports:
        - containerPort: 8080

The preStop hook runs sleep 5. During those five seconds, Kubernetes has already started removing the pod from endpoint lists. The application is still running, still accepting requests, still serving traffic. After five seconds, the endpoint removal has propagated to every node, the iptables rules no longer route new connections to this pod, and only then does Kubernetes deliver SIGTERM. The application shuts down, and no new requests are arriving.

Five seconds is not a magic number. It is a default that works on most clusters I have operated. If your cluster is large, has many services, or has a slow endpoint controller, you may need more. You can check the actual propagation time by watching endpoint updates during a deploy. On a 50-node cluster with a healthy control plane, I see endpoint removals propagate in two to four seconds. On a 200-node cluster with heavy service churn, I have seen it take eight seconds. Measure it on your cluster and set the preStop delay accordingly.

The terminationGracePeriodSeconds: 30 is the total time Kubernetes waits between sending SIGTERM and sending SIGKILL. The default is 30 seconds. The preStop sleep counts against this grace period. So with a five-second preStop and a 30-second grace period, the application has 25 seconds after SIGTERM to finish in-flight requests and exit before it gets killed. For most services, 25 seconds is more than enough. For services with long-running requests, like file uploads or streaming endpoints, you may need to increase the grace period.

What the SIGTERM handler should actually do

The preStop hook solves the inbound traffic race. The SIGTERM handler solves the in-flight request problem. These are separate concerns and they need separate handling.

A correct SIGTERM handler for an HTTP service does three things in order.

One. Stop accepting new requests. Call Shutdown() on the HTTP server, not Close(). Shutdown() stops the listener from accepting new connections while allowing in-flight requests to complete. Close() kills everything immediately, including requests in the middle of being served. In Go, this is srv.Shutdown(ctx). In Node, it is server.close(). In Python with uvicorn, it is built into the lifespan shutdown event. Every framework has this. Most engineers call the wrong one.

Two. Wait for in-flight requests to drain, with a deadline. You cannot wait forever. The grace period is finite, and if you exceed it, Kubernetes sends SIGKILL and the process dies without cleanup. Set a deadline that is shorter than terminationGracePeriodSeconds minus your preStop delay. With a 30-second grace period and a 5-second preStop, I set the shutdown deadline to 20 seconds. That leaves 5 seconds of buffer for the process to exit cleanly after the drain completes.

ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
if err := srv.Shutdown(ctx); err != nil {
    log.Printf("shutdown completed with error: %v", err)
}

Three. Close downstream connections. After the HTTP server has drained, close database connection pools, message broker connections, and any other external resources. Order matters. If you close the database pool before the HTTP server finishes draining in-flight requests, those requests will fail when they try to query the database. I have seen this exact bug in three different codebases. The SIGTERM handler closed the database connection first, then tried to drain HTTP requests, and every in-flight request that touched the database got a connection error.

The correct order is: stop accepting, drain in-flight, then close downstream. Not the other way around.

The connection draining problem that preStop does not solve

There is a second race that the preStop hook does not address, and it affects services with long-lived connections. WebSockets, gRPC streams, database connections from a connection pool, anything that holds a connection open past a single request.

When Kubernetes removes a pod from the endpoint list, new connections stop arriving. Existing connections are not killed. They stay open. The client on the other end does not know the pod is terminating. It will keep using the connection until the application closes it or until the pod is finally killed after the grace period.

If your application has a WebSocket connection that stays open for hours, a five-second preStop and a 30-second grace period will not drain it. The connection is still open when SIGKILL arrives, and the client gets a broken connection. For a WebSocket, this might be a user who gets disconnected from a live dashboard. For a gRPC stream, it might be a service that loses its subscription to an event feed. For a database connection pool, it might be a client that tries to use a stale connection and gets a query error.

For these cases, the SIGTERM handler needs to actively close existing connections, not just stop accepting new ones. In Go, srv.Shutdown() closes keep-alive HTTP connections but does not close WebSockets or gRPC streams. You need to track them and close them explicitly. A common pattern is to maintain a registry of active long-lived connections and close them all during shutdown, with a grace period that lets them finish their current operation.

var activeStreams sync.WaitGroup

// when a stream starts
activeStreams.Add(1)
defer activeStreams.Done()

// during shutdown
done := make(chan struct{})
go func() {
    activeStreams.Wait()
    close(done)
}()
select {
case <-done:
    log.Println("all streams drained")
case <-time.After(20 * time.Second):
    log.Println("stream drain timed out, forcing close")
    closeAllStreams()
}

This is not over-engineering. If you run WebSockets or gRPC streams in Kubernetes and you do not do this, you are relying on your clients to reconnect after a broken connection. Some clients do this well. Many do not. The ones that do not are the ones that produce the support tickets.

The readiness probe trap during shutdown

There is a third pattern I see that makes shutdown worse, not better. Some teams set their readiness probe to fail when the application receives SIGTERM, on the theory that failing readiness will remove the pod from the endpoint list faster.

This does not work the way they think it does. Failing the readiness probe removes the pod from endpoints, yes, but it does so on the probe schedule, which is typically every 10 seconds with a failure threshold of 3. That is 30 seconds before Kubernetes removes the pod, which is slower than the endpoint removal that already happens when the pod enters Terminating. The readiness probe failure does not speed up endpoint removal. It runs on top of a process that is already happening, and it adds noise to your dashboards because the probe failures show up as health check errors.

Worse, if your liveness probe is also configured to hit the same endpoint, and the endpoint starts returning failures during shutdown, Kubernetes may kill the pod with SIGKILL before the graceful shutdown completes. I watched this happen to a team in Munich. Their liveness and readiness probes both hit /health. During shutdown, the handler returned 503 to signal that the pod was going down. The liveness probe saw the failure, decided the pod was unhealthy, and killed it with SIGKILL four seconds into the shutdown. In-flight requests were dropped. The graceful shutdown they had carefully implemented never ran to completion.

The correct behavior during shutdown is to let the endpoint removal happen naturally via the termination sequence and the preStop hook. Do not manipulate the readiness probe during shutdown. Do not return failures from the health endpoint during SIGTERM. Let the preStop hook hold the pod in a serving state until endpoints have propagated, then let the SIGTERM handler drain and exit.

The full configuration I install

Here is the complete shutdown configuration for a typical HTTP service. I install some version of this for every service that handles user traffic.

spec:
  terminationGracePeriodSeconds: 30
  containers:
    - name: app
      image: ghcr.io/digitalaultis/app:latest
      lifecycle:
        preStop:
          exec:
            command: ["sleep", "5"]
      livenessProbe:
        httpGet:
          path: /healthz
          port: 8080
        periodSeconds: 10
        failureThreshold: 3
      readinessProbe:
        httpGet:
          path: /readyz
          port: 8080
        periodSeconds: 5
        failureThreshold: 3
      ports:
        - containerPort: 8080

And the shutdown handler, in Go.

func gracefulShutdown(srv *http.Server, db *sql.DB) {
    ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
    defer cancel()

    // stop accepting new connections, drain in-flight requests
    if err := srv.Shutdown(ctx); err != nil {
        log.Printf("http shutdown error: %v", err)
    }

    // close downstream connections after http drain
    if err := db.Close(); err != nil {
        log.Printf("db close error: %v", err)
    }

    log.Println("shutdown complete")
}

/healthz and /readyz are separate endpoints. Liveness checks whether the process is alive. Readiness checks whether the process can serve requests. During shutdown, neither endpoint changes its behavior. The endpoint removal is handled by the termination sequence, not by probe manipulation.

Why this matters more than you think

The Berlin team I started with was dropping one in five hundred requests on every deploy. They deployed eight times a day. At their request volume, that was roughly thirty failed requests per day, every day, from deploys alone. Their error budget was being consumed not by bugs or traffic spikes but by their own deployment process. They had a good CI/CD pipeline, a good rolling update strategy, and a shutdown handler that was correct in isolation. The only thing missing was a five-second preStop hook.

I added the hook. I changed the SIGTERM handler to drain before closing the database. I separated the liveness and readiness endpoints. The deploy-time failures went to zero on the next rollout and stayed there. The whole change took two hours. The team had been living with the problem for six months because the error rate was low enough to ignore and the failures were distributed across users who rarely reported them.

This is the pattern I see over and over. Teams build correct applications and incorrect shutdown sequences, and the gap between the two produces a low-grade failure rate that never triggers an alert but degrades every user’s experience during deploys. The fix is not complex. It is a preStop hook, a drain-before-close ordering, and a grace period that matches your request durations. The hard part is knowing that the race exists, because Kubernetes does not warn you about it. It just silently routes traffic to a pod that is no longer listening.