Your health checks are lying to you

I was on a call with a team in Munich whose API had been restarting every 90 seconds for three hours. The logs showed nothing wrong. The application started, served traffic for about a minute and a half, then got killed by Kubernetes. The cycle repeated. Users saw intermittent 502s. The team had been debugging the application code for two hours and found nothing because the problem was not in the application code. It was in the liveness probe.

Their liveness probe hit /health, which ran a database query. The query was simple: SELECT 1. It returned in 2ms most of the time. But under load, when the connection pool was saturated, the query took 6 seconds. The probe timeout was set to 3 seconds. Kubernetes interpreted the timeout as a failed health check and killed the pod. The pod restarted, served traffic for 90 seconds until the pool saturated again, and got killed again. The liveness probe, which was supposed to detect failures and recover, was the failure. It was causing the outages it was designed to prevent.

This is the most common health check problem I see, and it is not the only one. I have audited the Kubernetes configurations of roughly forty teams over the last three years. The health check configuration is wrong in almost every one. Not subtly wrong. Structurally wrong. Wrong in ways that make outages worse, not better.

Here is what I find, why it happens, and what to do instead.

The three probes, and why nobody knows which one to use

Kubernetes has three probe types. Most teams use two and confuse them. Some teams use one for everything. Almost nobody uses the third, which is the one that would actually solve their problem.

Liveness probe: determines whether the container should be restarted. If it fails, Kubernetes kills the container and starts a new one. This is for detecting deadlocks, infinite loops, and states where the process is running but not functioning.

Readiness probe: determines whether the container should receive traffic. If it fails, Kubernetes removes the pod from the service endpoint list. The pod keeps running. It just does not get requests. This is for detecting states where the application is alive but cannot handle requests, like a warming cache or a lost database connection.

Startup probe: determines whether the container has started successfully. It runs once at startup and disables liveness and readiness probes until it passes. This is for slow-starting applications that need time to initialize and would fail liveness probes during that window.

The confusion between liveness and readiness is the source of most problems I see. Teams use liveness when they should use readiness, readiness when they should use liveness, or the same endpoint for both. The result is that Kubernetes either restarts pods that should be removed from traffic, or removes pods from traffic that should be restarted.

The rule I give teams is simple. If the application can recover on its own, use readiness. If it cannot, use liveness. A database connection that drops and reconnects is a readiness problem. The application will reconnect. Removing it from traffic during the reconnection window is correct. Restarting it is wrong, because restarting does not fix a database connectivity issue. It just adds startup overhead.

A deadlock or a goroutine leak that causes the process to hang is a liveness problem. The application cannot recover. Restarting is the only fix.

Most teams I audit use liveness for both. When the database goes down, their pods restart instead of being removed from traffic. The restarts do not help. They just add noise, increase startup load on the database when it comes back, and delay recovery.

The health endpoint that lies

The second problem is what the health endpoint actually checks. I have seen three patterns, and two of them are wrong.

Pattern one: the endpoint returns 200 unconditionally.

func healthHandler(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
}

This is not a health check. It is a process check. It tells you the HTTP server is listening. It does not tell you whether the application can handle requests. A service with a dead database connection, a full connection pool, or a corrupted cache will return 200 on this endpoint while failing every real request. I have seen this pattern in production at a fintech that was returning 200s while every payment request was timing out.

Pattern two: the endpoint checks every dependency.

func healthHandler(w http.ResponseWriter, r *http.Request) {
    if err := db.Ping(); err != nil {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    if err := cache.Ping(); err != nil {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    if err := queue.Ping(); err != nil {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
}

This is better, and it is also wrong for a different reason. If your health check depends on your database, and your database is down, your health check fails. If this is a liveness probe, Kubernetes restarts your pods. Restarting pods does not fix a database outage. If this is a readiness probe, Kubernetes removes all your pods from traffic. Your entire service goes down because a dependency is down. You have turned a partial outage into a full outage.

The problem is that your health check and your dependency check should not be the same thing. They answer different questions. “Is this application process healthy?” is not the same as “Can this application serve requests right now?” And neither question is the same as “Are all dependencies available?”

Pattern three, the one I install: the endpoint checks whether the application process can do its core work, which is usually “can I serve a request from my own state.” It does not check dependencies directly. It checks the application’s own internal state.

func healthHandler(w http.ResponseWriter, r *http.Request) {
    // Check internal state, not external dependencies.
    // Can we acquire a connection from the pool?
    // Can we read from our local cache?
    // Are we not in a shutting-down state?
    
    select {
    case <-shutdownSignal:
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    default:
    }
    
    if dbPool.Stats().AvailableConnections == 0 && dbPool.Stats().WaitCount > 0 {
        // Pool is exhausted. We are alive but cannot serve.
        // This is a readiness failure, not a liveness failure.
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    
    w.WriteHeader(http.StatusOK)
}

The distinction matters. The health endpoint should tell you whether the application itself is functioning. It should not tell you whether the database is up, because the database being down does not mean the application is broken. It means the application cannot do its job right now, which is a readiness problem, not a liveness problem.

The cascading restart problem

The Munich team’s problem was a specific instance of a broader pattern I see in roughly a third of the clusters I audit. The liveness probe checks a dependency. The dependency gets slow under load. The probe times out. Kubernetes restarts the pod. The new pod adds startup load to the already-struggling dependency. The dependency gets slower. More pods fail their liveness probes. More pods restart. The system enters a restart loop where the recovery mechanism accelerates the failure.

I saw this at its worst with a team in Amsterdam running 24 replicas of a payment service. Their database got slow during a traffic spike. All 24 pods failed their liveness probes simultaneously. Kubernetes restarted all 24 pods. Each restart triggered a new connection pool initialization, which hit the database with 24 x 20 = 480 new simultaneous connections. The database, already struggling, collapsed under the connection load. The pods failed their startup probes, got killed again, and the cycle continued for 40 minutes until someone manually cordoned the nodes and stopped the restart loop.

The fix was three changes.

First, the liveness probe stopped checking the database. It checked only whether the process could respond to HTTP. If the process was alive and listening, the liveness probe passed. The database being slow was not a reason to restart the process.

livenessProbe:
  httpGet:
    path: /livez
    port: 8080
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 3

Second, the readiness probe checked whether the application could serve requests, which included checking that the connection pool was usable. When the database got slow, pods were removed from traffic instead of restarted. The service degraded gracefully. Some requests failed, but the system did not enter a restart loop.

readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 2

Third, a startup probe was added so that slow-starting pods were not killed during initialization.

startupProbe:
  httpGet:
    path: /readyz
    port: 8080
  periodSeconds: 5
  failureThreshold: 12
  # Up to 60 seconds to start before Kubernetes gives up

After these changes, the same traffic spike that caused a 40-minute outage caused a 3-minute degradation. Pods were removed from traffic when the database got slow, and they came back when the database recovered. No restarts. No cascading failure. The system self-healed instead of self-destructing.

The timeout problem

The third problem I see everywhere is probe timeouts that do not match reality. Teams copy defaults from tutorials or from other services, and the timeouts are wrong for their specific application.

A liveness probe with a 10-second timeout is not a health check. It is a test of whether your application can respond to an HTTP request in 10 seconds. If your application normally responds in 5ms, a 10-second timeout means the probe will pass even when the application is 2,000x slower than normal. The probe will not detect degradation until the application is completely broken.

A liveness probe with a 1-second timeout on an application that occasionally takes 800ms to respond will produce false positives. The probe will fail intermittently during normal operation, causing spurious restarts that are harder to debug than the problems they are trying to catch.

The timeout should be set based on your application’s observed behavior, not a default. I set liveness probe timeouts to 2x the p99 response time of the health endpoint, measured under normal load. If the health endpoint’s p99 is 50ms, the timeout is 100ms. If something is wrong enough that the health endpoint takes more than 100ms, the probe should fail.

# Measure the health endpoint's response time distribution
# Run this over a few hours of normal traffic
while true; do
  curl -w "%{time_total}\n" -o /dev/null -s http://localhost:8080/livez
  sleep 0.1
done | sort -n | awk '
{ a[NR]=$1 }
END {
  print "p50:", a[int(NR*0.50)]
  print "p90:", a[int(NR*0.90)]
  print "p99:", a[int(NR*0.99)]
}'

If the p99 is 30ms, set the timeout to 100ms. If the p99 is 300ms, set it to 1s. The point is that the timeout should be tight enough to detect problems but loose enough to avoid false positives. Copy-pasted defaults are neither.

The period and threshold problem

The fourth problem is the interaction between periodSeconds and failureThreshold. Most teams leave these at defaults: period 10 seconds, threshold 3. This means Kubernetes will restart a pod 30 seconds after it starts failing its liveness probe.

Thirty seconds is a long time for some systems and too short for others. The defaults assume a certain cadence that may not match your application.

For a high-throughput API where 30 seconds of a broken pod means thousands of failed requests, the threshold should be lower. I use period 5 and threshold 2 for liveness, which means 10 seconds from failure to restart. For a background worker where 30 seconds of a broken pod is irrelevant, the defaults are fine.

For readiness, the threshold should be low. I use period 3 and threshold 1. The moment a pod cannot serve traffic, it should be removed. There is no benefit to waiting. The pod is not going to recover faster because you checked it three more times.

# Liveness: 10 seconds from failure to restart (period 5 x threshold 2)
livenessProbe:
  httpGet:
    path: /livez
    port: 8080
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 2

# Readiness: 3 seconds from failure to removal (period 3 x threshold 1)
readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  periodSeconds: 3
  timeoutSeconds: 1
  failureThreshold: 1

# Startup: 60 seconds to initialize (period 5 x threshold 12)
startupProbe:
  httpGet:
    path: /readyz
    port: 8080
  periodSeconds: 5
  failureThreshold: 12

These numbers are not universal. They are starting points that I tune based on the application. The point is that the defaults are guesses, and you should replace them with measurements.

The /healthz, /livez, /readyz convention

I use three separate endpoints for every service I configure. This is the convention Kubernetes itself uses for its own components, and it is worth copying.

/livez is the liveness endpoint. It checks only whether the process is alive and can respond to HTTP. It does not check dependencies. It does not check internal state beyond “is the HTTP server listening.” If this endpoint fails, the process is broken in a way that requires a restart.

/readyz is the readiness endpoint. It checks whether the application can serve requests. This includes checking the connection pool, the local cache, and any internal state that determines whether the application can do its job. It does not check external dependencies directly. If this endpoint fails, the application is alive but not ready, and should be removed from traffic.

/healthz is the aggregate endpoint that some load balancers and monitoring tools expect. It combines both checks. It returns 200 only if both livez and readyz would return 200. I include it for compatibility with tools that expect a single health endpoint, but I configure Kubernetes probes to use the specific endpoints.

func registerHealthChecks(mux *http.ServeMux, app *Application) {
    mux.HandleFunc("/livez", func(w http.ResponseWriter, r *http.Request) {
        // Process is alive if we got here. The HTTP server is listening.
        w.WriteHeader(http.StatusOK)
    })

    mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
        if !app.IsReady() {
            w.WriteHeader(http.StatusServiceUnavailable)
            return
        }
        w.WriteHeader(http.StatusOK)
    })

    mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
        // Aggregate for external tools
        if !app.IsReady() {
            w.WriteHeader(http.StatusServiceUnavailable)
            return
        }
        w.WriteHeader(http.StatusOK)
    })
}

func (a *Application) IsReady() bool {
    // Can we acquire a DB connection without blocking?
    if a.dbPool.Stats().AvailableConnections == 0 && a.dbPool.Stats().WaitCount > 0 {
        return false
    }
    // Is our local cache initialized?
    if !a.cache.IsInitialized() {
        return false
    }
    // Are we shutting down?
    select {
    case <-a.shutdownCh:
        return false
    default:
    }
    return true
}

The separation matters because it forces you to think about what “healthy” means for your specific application. A single endpoint used for both liveness and readiness means you have not thought about the difference. That is the state most teams are in when I arrive, and it is the state that produces cascading restarts, false outages, and probes that cause more incidents than they prevent.

What I install

When I configure health checks for a team, the baseline is four things.

Three separate endpoints. /livez checks process health only. /readyz checks application readiness including internal state but not external dependencies. /healthz aggregates both for compatibility.

Probes with timeouts tuned to observed response times. Liveness at 2x p99 of the health endpoint. Readiness at 1x p99. Startup with a generous window that covers the slowest observed cold start plus 50%.

A readiness probe that removes pods from traffic, not a liveness probe that restarts them, for dependency-related issues. The liveness probe should fail only when the process itself is broken. Dependency problems are readiness problems.

A startup probe on every slow-starting application. If your application takes more than 5 seconds to initialize, you need a startup probe. Without it, your liveness probe will kill the pod during initialization, and the pod will never start.

None of this is complicated. It is a set of endpoints, a set of probe configurations, and the discipline to measure your application’s behavior instead of copying defaults. The complicated part is accepting that health checks are not a checkbox. They are a system that can help you or hurt you, and the difference is whether you thought about what “healthy” means for your specific application.

The team in Munich has not had a probe-related outage since the fix. Their database still gets slow under load. Their pods still lose database connections. But the pods are removed from traffic, not restarted. The service degrades instead of collapsing. The probes, which were the problem, became the solution. The difference was not the tooling. It was understanding what the probes actually do and configuring them to do it.