Your HPA is not scaling and your metrics are why

I have audited the autoscaling setup for roughly fifteen startups. In twelve of those, the HPA was configured, committed, and not working. Not broken in an obvious way. Broken in a way where the pods eventually scaled, hours after the traffic spike, and everyone assumed that was just how Kubernetes worked.

It is not how Kubernetes works. It is how a misconfigured HPA works. The difference is whether you understand the metrics pipeline that feeds the autoscaler, and almost nobody does.

The startup that scaled too late

A logistics startup in Rotterdam. Three services, Kubernetes on EKS, HPA on every deployment. They ran a flash sale on a Tuesday morning. Traffic went from 200 requests per second to 4,000 in under two minutes. The HPA was set to scale at 70% CPU utilization with a minimum of 3 replicas and a maximum of 20.

The pods scaled. They went from 3 to 5 replicas. Not 20. Five. The checkout service was returning 503s for 40% of requests for eleven minutes until someone manually ran kubectl scale deployment checkout --replicas=15.

After the incident, the platform lead sent me the HPA config and asked why it only added two pods. The config was fine. The metrics were not.

Here was their HPA:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Standard. The kind of YAML you find in every tutorial. The problem is in what this config does not say, and what it does not say is the part that kills you in production.

Problem one: CPU utilization is the wrong signal

The HPA scales on CPU utilization, which means it scales on how close your pods are to their CPU request. Not their limit. Their request. If you set a CPU request of 500m and a limit of 2000m, the HPA sees 70% utilization as 350m of usage. Your pod can use up to 2000m, but the autoscaler starts adding replicas at 350m.

This is the first thing nobody explains. CPU utilization in the HPA is actual_usage / cpu_request. It is not actual_usage / cpu_limit. It is not actual_usage / node_capacity. It is a ratio against a number you set, and if you set that number wrong, the autoscaler is wrong.

The Rotterdam team had set CPU requests at 1000m (1 full core) for the checkout service. The pods were using 700m during the traffic spike, which is 70% utilization. The HPA saw 70%, which matched the threshold, so it added just enough replicas to bring the average back to 70%. Two pods. The threshold is not a trigger. It is a target. The HPA does not say “scale up when CPU hits 70%.” It says “keep CPU at 70% by adding or removing replicas.” If you are at 72%, it adds one pod. If you are at 95%, it adds more. But the number of pods it adds is proportional to how far above target you are, and it is capped by a stabilization window that prevents rapid scaling.

# What the HPA actually computed
# desiredReplicas = ceil(currentReplicas * (currentMetric / targetMetric))
desiredReplicas = ceil(3 * (720m / 700m)) = ceil(3.08) = 4

Four pods. Not twenty. The HPA did exactly what the math told it to do. The math was based on a metric that did not represent the problem.

The checkout service was not CPU-bound. It was I/O-bound. It spent most of its time waiting on a PostgreSQL database and a payment provider API. CPU was at 72% because the event loop was busy juggling connections, not because the pod was doing computation. Adding pods did not help because each new pod also waited on the same database and the same payment provider, both of which were now the bottleneck.

The fix is to scale on a metric that represents the actual constraint. For an I/O-bound service, that is request latency, or queue depth, or concurrent connections. Not CPU.

metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Pods
    pods:
      metric:
        name: http_requests_in_flight
      target:
        type: AverageValue
        averageValue: "50"

This HPA scales on two signals. CPU stays as a floor. The second metric, http_requests_in_flight, is a custom metric exposed by the application that represents the number of requests currently being processed by each pod. When the average hits 50, the HPA adds pods. This is a direct measure of load, not a proxy.

But custom metrics require a metrics pipeline, and that is where most teams stop. You cannot just write http_requests_in_flight in the HPA and expect Kubernetes to find it. You need a metrics adapter, and you need the application to expose the metric. I will get to that.

Problem two: CPU throttling hides the spike

There is a more insidious version of the CPU problem. It happens when you set a CPU limit that is lower than what the pod actually needs, and the kernel throttles the pod before the HPA ever sees the real usage number.

CPU throttling in Linux cgroups works like a token bucket. The pod gets a quota of CPU time per period. The default period is 100ms. If you set a limit of 500m, the pod gets 50ms of CPU time per 100ms period. If the pod tries to use more than 50ms in the first 80ms of the period, it is throttled for the remaining 20ms. It sits idle. The scheduler reports the throttled time, but the metric the HPA consumes is average CPU utilization over a window, and the throttling can average out to a number that looks fine.

I saw this at a fintech in Luxembourg. Their API pods were throttling 40% of the time during peak hours. The HPA was not scaling because the CPU metric, averaged over the five-minute window the HPA uses by default, showed 65% utilization. Under the threshold. The pods were burning 40% of their time in cgroup throttle, the application was slow, and the autoscaler was asleep.

The fix here is not to raise the limit. The fix is to remove the CPU limit entirely and rely on requests for scheduling.

resources:
  requests:
    cpu: 500m
    memory: 512Mi
  # No limits. Let the pod burst when it needs to.
  # The HPA scales on request utilization, which is now an honest number.

I know this is controversial. Every managed Kubernetes provider’s default template includes CPU limits. Every cost optimization guide tells you to set limits to prevent noisy neighbors. I have read those guides. I have also sat in incident reviews where the root cause was CPU throttling on a service that was trying to handle a traffic spike. The noisy neighbor problem is real. The throttling problem is more real, and it is harder to diagnose because it does not show up as an error. It shows up as latency.

If you must keep limits, set them high enough that the pod never hits them under normal load. 2x to 4x the request is a starting point. The limit should be a safety net for pathological cases, not a ceiling on normal operation. And monitor throttling directly:

# CPU throttling ratio per pod
rate(container_cpu_cfs_throttled_seconds_total[5m])
  / rate(container_cpu_cfs_periods_total[5m])

If that ratio is above 0.1, your pods are spending more than 10% of their time throttled, and your HPA is making decisions on a metric that does not reflect what the pod is actually experiencing.

Problem three: the metrics pipeline is missing

This is the part that catches every team. You write a custom metric in the HPA. You deploy it. The HPA stays at the minimum replica count and prints this in its events:

Warning  FailedComputeMetricsReplicas  6s (x12 over 8m)
  horizontal-pod-autoscaler  unable to get metric http_requests_in_flight:
  unable to fetch metrics from custom metrics API:
  no custom metrics API registered

The custom metrics API is not a built-in. Kubernetes does not know about your application metrics. You need an adapter that sits between your metrics store (usually Prometheus) and the Kubernetes custom metrics API. The adapter translates Prometheus queries into the format the HPA expects.

The two options I have used in production are the Prometheus Adapter and KEDA. They solve the same problem differently, and the choice matters.

The Prometheus Adapter is the standard approach. It runs as a Deployment in your cluster, talks to Prometheus, and exposes custom metrics through the Kubernetes API. You configure it with a mapping rules file that tells it which Prometheus queries correspond to which Kubernetes metrics.

# adapter config
rules:
  - seriesQuery: 'http_requests_in_flight{namespace!="",pod!=""}'
    resources:
      overrides:
        namespace: {resource: "namespace"}
        pod: {resource: "pod"}
    name:
      matches: "^(.*)_in_flight"
      as: "${1}_in_flight"
    metricsQuery: 'avg(http_requests_in_flight{namespace="<<.Namespace>>",pod=~"<<.PodName>>.*"}) by (pod)'

This is dense, and it is the reason most teams give up on custom metrics. The mapping rules are a YAML DSL that translates between two metric naming conventions, and getting it wrong produces silent failures. The HPA does not error. It just does not scale, because the metric it is asking for does not exist in the adapter’s response.

I once spent four hours debugging an HPA that would not scale on a custom metric. The adapter was running. Prometheus had the metric. The mapping rule was wrong. It was matching http_requests_inflight instead of http_requests_in_flight because of a regex I had copied from a blog post. Four hours for a missing underscore.

KEDA is the alternative, and it is the one I install for teams that do not want to maintain adapter config. KEDA is an operator that handles the scaling logic itself. Instead of the HPA reading from the custom metrics API, KEDA creates and manages the HPA for you, and it talks to metric sources directly.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: checkout
spec:
  scaleTargetRef:
    name: checkout
  minReplicaCount: 3
  maxReplicaCount: 20
  pollingInterval: 30
  cooldownPeriod: 300
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc:9090
        metricName: http_requests_in_flight
        threshold: "50"
        query: avg(http_requests_in_flight{namespace="checkout"}) by (pod)

No adapter. No mapping rules. KEDA reads the Prometheus query directly and scales the deployment. The pollingInterval is 30 seconds, which is faster than the default HPA sync interval of 15 seconds to 60 seconds depending on the controller manager flags. The cooldownPeriod controls how long KEDA waits before scaling down, which prevents flapping.

The tradeoff is that KEDA is another operator in your cluster, with its own CRDs, its own upgrade cycle, and its own failure modes. For a team that already runs Prometheus and wants custom metric autoscaling without writing adapter config, KEDA is the right choice. For a team that wants to stay close to upstream Kubernetes and is comfortable with the adapter, the Prometheus Adapter is fine. I have run both in production. Both work. KEDA is less work to maintain.

Problem four: the scale-up is too slow even when the metric is right

The Rotterdam team fixed their metrics. They added http_requests_in_flight as a custom metric through KEDA. They ran another flash sale. The HPA scaled. It took four minutes to go from 3 replicas to 15. The traffic spike lasted two minutes.

Four minutes is too slow. The spike was over before the new pods were ready. The pods took 90 seconds to start because the image was 1.2GB and the Java application took 40 seconds to initialize. The autoscaler reacted in 30 seconds (the KEDA polling interval), but the pod startup time was the bottleneck.

This is the problem the HPA cannot solve. The HPA controls when pods are created. It does not control how fast pods become ready. If your pod takes 90 seconds to start, your autoscaling floor is 90 seconds plus the time it takes the autoscaler to react. For a two-minute traffic spike, that is the entire window.

The fixes are unglamorous and they have nothing to do with the HPA.

Reduce image size. The 1.2GB image was a Spring Boot fat jar running on a generic JRE base image. We switched to a distroless Java base with jlink and got it to 280MB. Pod start time dropped from 45 seconds (pulling the image) to 12 seconds.

Reduce application startup time. The Spring Boot app was doing 40 seconds of context initialization. We enabled lazy initialization for non-critical beans and deferred the database migration check to a post-start hook. Startup dropped to 18 seconds.

Overprovision the floor. If your traffic pattern includes spikes that last less than five minutes, your minimum replica count needs to handle the spike without scaling. The HPA is for sustained load, not burst load. For burst load, you keep extra pods warm.

minReplicaCount: 8  # warm capacity for burst traffic
maxReplicaCount: 20  # scale further for sustained load

Eight warm pods costs more. It costs less than eleven minutes of 503 responses during a flash sale. The Rotterdam team did the math. Eight pods cost them EUR 340 per month. The flash sale incident cost them an estimated EUR 18,000 in lost transactions. The math is not close.

Problem five: scale-down is destroying your cost savings

The other side of the HPA is scale-down, and it is where I see teams undo their own work. The HPA scales down based on the same metric it scales up on, but the default behavior is conservative. The scaleDown.stabilizationWindowSeconds defaults to 300 seconds for scale-down (and 0 for scale-up in autoscaling/v2). This means the HPA waits five minutes of low utilization before removing pods. That is fine.

The problem is teams that override this to be aggressive. I saw a startup in Munich that set the stabilization window to 30 seconds because they wanted to save money. Their traffic pattern was spiky. Pods scaled down 30 seconds after a spike, then scaled back up 30 seconds later when the next spike arrived. They were churning pods constantly. The cluster autoscaler was creating and destroying nodes every few minutes. Their EC2 bill went up, not down, because of the node churn and the repeated image pulls.

The default five-minute stabilization window exists for a reason. It is not arbitrary. It is long enough to smooth out most traffic fluctuations without keeping warm capacity you do not need. If your traffic is bursty enough that five minutes is too long, you need warm pods, not a shorter stabilization window.

behavior:
  scaleDown:
    stabilizationWindowSeconds: 300
    policies:
      - type: Percent
        value: 25
        periodSeconds: 60

This says: wait five minutes of low utilization, then remove at most 25% of pods per minute. No sudden drops. No churning. The pod count ramps down the same way it ramps up.

What I install

For every startup I work with, the autoscaling setup ends up at the same place.

CPU requests set to the actual steady-state usage, measured over a week of production traffic. No CPU limits, or limits at 3x the request if the cluster has noisy neighbor pressure. The HPA on CPU as a floor, with a custom metric from KEDA as the real signal. The custom metric is request concurrency or queue depth, something the application exposes directly. Scale-up stabilization at 0 seconds. Scale-down stabilization at 300 seconds with a 25% per minute policy. Image size under 400MB. Application startup under 30 seconds.

And one more thing that is not in any YAML: a load test that proves the autoscaling works. Not a synthetic test against staging. A controlled load test against production during a low-traffic window, with real traffic, that verifies the pods scale before the latency budget is exhausted.

# Verify HPA behavior under load
# Watch the replica count and the metric simultaneously
watch -n 2 'kubectl get hpa checkout -o jsonpath="{.status.currentReplicas} replicas, { .status.currentMetrics[0].resource.current.averageUtilization }% CPU"'

# In another terminal, apply load
k6 run --vus 500 --duration 120s loadtest.js

If the replica count does not move before latency degrades, your autoscaling is not working. It does not matter what the config says. It does not matter what the dashboard shows. The load test is the truth. Everything else is a claim.

The Rotterdam team runs this load test every month now. It takes twenty minutes. It has caught three HPA regressions that would have been production incidents. The test is not sophisticated. It is a k6 script that ramps traffic and checks whether the pods scale in time. The sophistication is in running it at all, because most teams trust their HPA config and never verify it under load. The HPA is not a configuration. It is a system, and systems need to be tested.