Your retry logic is making your outage worse

A team in Vienna called me in after an outage that should not have happened. Their payment provider had a slow minute, maybe 90 seconds of elevated latency on one endpoint. No errors, just slow. Their platform went down for 40 minutes. Not because of the provider. Because of their retry logic.

The provider’s API started responding in 3 seconds instead of the usual 200 milliseconds. Their payment service had a client-side timeout of 5 seconds, so most calls succeeded. But a few hit the timeout. The retry logic kicked in, retried immediately, and those retries also timed out. The retry logic retried again. Meanwhile, the service was still accepting new requests, queuing them behind the retries, and the thread pool filled up. The health check started failing. Kubernetes restarted the pods. The new pods did the same thing. The entire payment service was down, and because every other service depended on it, the whole platform followed.

The slow minute at the provider lasted 90 seconds. The outage lasted 40 minutes. The retry logic was the difference.

What your retry logic actually does

Most retry logic I see in production looks like this:

import requests
from tenacity import retry

@retry(stop=stop_after_attempt(3))
def call_payment_provider(order_id: str):
    response = requests.post(
        "https://api.provider.com/v1/charge",
        json={"order_id": order_id},
        timeout=5,
    )
    response.raise_for_status()
    return response.json()

This is the default that every retry library gives you. It retries three times, immediately, with no delay. It looks responsible. It is the most dangerous pattern in distributed systems.

Here is what happens when the provider gets slow. The first call times out at 5 seconds. The retry fires immediately. The provider is still slow, so the retry also times out at 5 seconds. The third retry times out. Total time per request: 15 seconds. During those 15 seconds, the thread is blocked, the connection is held open, and the caller is waiting. If the caller also has retry logic, it retries its call to your service, which retries its call to the provider. The load multiplies at every layer.

Now scale this across 50 concurrent requests. The provider gets 150 calls instead of 50. If it was slow before, it is now slower. The retries made the problem worse. This is called a retry storm, and it is the most common cause of cascading failures in microservice architectures.

The four things your retry logic must do

I install the same four patterns in every codebase I work on. They are not optional. Each one prevents a specific failure mode.

One: exponential backoff with a cap

Retries must wait between attempts, and the wait must grow. The first retry waits 1 second, the second waits 2, the third waits 4. This gives the downstream service time to recover. Without backoff, retries pile on instantly and guarantee that the downstream service stays overloaded.

But the backoff must have a cap. If you retry five times with pure exponential backoff, the fifth retry waits 16 seconds. That is a long time to hold a request open. I cap backoff at 10 seconds for most services, 30 seconds for batch jobs.

@retry(
    stop=stop_after_attempt(4),
    wait=wait_exponential(multiplier=1, max=10),
)
def call_payment_provider(order_id: str):
    ...

The cap matters more than people think. I once audited a service that retried 10 times with uncapped exponential backoff. The tenth retry waited 512 seconds. The caller had a timeout of 30 seconds. The retry never actually ran because the caller had already given up and returned an error to the user. The thread was still held open, waiting for a retry that would never complete. The service had 200 threads, all stuck in this state, and it stopped accepting new connections.

Two: jitter

Exponential backoff solves the pile-on problem. It does not solve the thundering herd. If 100 requests fail at the same time and all retry after 1 second, 100 requests hit the downstream service at the same moment. The service is still recovering. The 100 requests fail again. All 100 retry after 2 seconds, at the same time. The synchronization is the problem.

Jitter breaks the synchronization. Each retry adds a random delay on top of the backoff. Instead of all 100 retries hitting at the 1-second mark, they spread out between 1 and 2 seconds. The downstream service sees a smooth ramp instead of a spike.

@retry(
    stop=stop_after_attempt(4),
    wait=wait_exponential_jitter(multiplier=1, max=10, jitter=2),
)
def call_payment_provider(order_id: str):
    ...

The jitter value should be roughly equal to the backoff value. If the backoff is 1 second, the jitter should be between 0 and 1 second. This spreads the retries across a window roughly the size of the backoff itself. The exact distribution matters less than the presence of randomness. Equal jitter (half fixed, half random) is better than full jitter for most cases, because it preserves some of the backoff structure while still breaking synchronization.

I can tell within five minutes of looking at a system whether it has jitter. If the downstream service’s request rate looks like a staircase, with sharp steps at each backoff interval, there is no jitter. If it looks like a smooth curve, there is.

Three: a circuit breaker

Backoff and jitter make retries safer. They do not make them smart. If a downstream service is down, not slow but down, retrying is pointless. You are spending resources to get the same error. The circuit breaker stops you from trying.

A circuit breaker has three states. Closed: requests pass through normally. Open: requests fail immediately without hitting the downstream service. Half-open: a limited number of requests are allowed through to test if the downstream service has recovered.

The transition from closed to open happens when failures exceed a threshold. I usually set this at 50% of requests failing over a 10-second window. The transition from open to half-open happens after a cooldown period, typically 30 seconds. If the half-open requests succeed, the breaker closes. If they fail, it opens again.

from circuitbreaker import circuit

@circuit(failure_threshold=10, recovery_timeout=30)
@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential_jitter(multiplier=1, max=10, jitter=1),
)
def call_payment_provider(order_id: str):
    ...

The circuit breaker and the retry logic work together. The retry handles transient failures. The circuit breaker handles sustained failures. Without the circuit breaker, a 40-minute outage at the provider means 40 minutes of useless retries consuming threads, connections, and memory. With the circuit breaker, the requests fail fast after 10 consecutive failures, the service degrades gracefully, and the system stays up.

The Vienna team did not have a circuit breaker. Their payment service spent 40 minutes retrying calls to a slow provider. If they had a circuit breaker, the service would have stopped calling the provider after 10 failures, returned a “payment temporarily unavailable” response to the caller, and the rest of the platform would have stayed up. The outage would have been a degraded experience for one feature, not a full platform outage.

Four: retry only on idempotent operations

This is the one that catches everyone. Retry logic assumes that the retried operation is safe to repeat. If the first call failed after the downstream service received it but before you got the response, the retry sends the same request again. If the operation is a payment, the customer gets charged twice.

The question is not whether the call failed. It is whether the downstream service processed it before the failure. You cannot know. The HTTP request might have reached the server, the server might have processed it, and the response might have been lost on the network. Your client sees a timeout. The server sees a success. The retry sends a duplicate.

The fix is idempotency keys. Every retry-able request includes a unique key. The downstream service stores the key and the result. If a request arrives with a key it has already processed, it returns the stored result instead of processing again.

import uuid

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential_jitter(multiplier=1, max=10, jitter=1),
)
def call_payment_provider(order_id: str, amount: int):
    response = requests.post(
        "https://api.provider.com/v1/charge",
        json={"order_id": order_id, "amount": amount},
        headers={
            "Idempotency-Key": str(uuid.uuid5(uuid.NAMESPACE_URL, f"charge:{order_id}")),
            # Deterministic UUID: same order always produces same key
            # so retries with the same order_id are deduplicated by the provider
        },
        timeout=5,
    )
    response.raise_for_status()
    return response.json()

The key is a deterministic UUID, not a random one. A random UUID means each retry has a different key, which defeats the purpose. The key must be derived from the request identity, so the same logical operation always produces the same key. For a payment, that is the order ID. For a webhook delivery, it is the webhook ID plus the attempt number, or better, just the webhook ID if the provider deduplicates.

If the downstream service does not support idempotency keys, you have two options. You can make the operation idempotent by design, which means the operation is safe to repeat (a GET request, a status check, a read). Or you do not retry, and you handle the failure explicitly. The worst option is to retry a non-idempotent operation without an idempotency key. That is how double charges happen.

I have seen this exact bug at three different companies. A payment service retries on timeout. The provider processes the charge. The response is lost. The retry charges again. The customer sees two charges, disputes both, and the company eats the cost. The fix is one line: an idempotency key derived from the order ID.

The retry budget

Even with all four patterns, retries amplify load. If every service retries 3 times, a 10% failure rate at the bottom of the call chain becomes a 30% increase in load at that layer, which can push it over the edge. The retry budget is a system-level constraint on how much retrying is allowed.

The concept is simple. The system allows a certain percentage of retries relative to original requests. If the budget is 10%, then for every 100 original requests, the system allows 10 retries. Once the budget is exhausted, retries are rejected. This prevents retries from overwhelming a system that is already struggling.

In practice, retry budgets are hard to implement at the application level. They require a shared counter across instances, which means a distributed counter, which is its own failure mode. I implement them at the infrastructure level instead, using rate limiting. The service mesh (Linkerd, Istio) or an API gateway (Envoy) can enforce a retry budget across all instances of a service. This is one of the few cases where I recommend a service mesh. The retry budget is worth the complexity.

Without a service mesh, the application-level approximation is to make the circuit breaker aggressive. A 10% failure threshold with a 30-second recovery timeout effectively caps retries at 10% of traffic for 30 seconds. It is not precise, but it prevents the retry storm.

What I install

Here is the complete pattern I put in every service that calls a downstream API. It is not clever. It is not exotic. It is the minimum that prevents cascading failures.

import uuid
import requests
from tenacity import retry, stop_after_attempt, wait_exponential_jitter
from circuitbreaker import circuit

@circuit(failure_threshold=10, recovery_timeout=30)
@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential_jitter(multiplier=1, max=10, jitter=1),
    retry=retry_if_exception_type(
        (requests.ConnectionError, requests.Timeout)
    ),
)
def call_downstream(endpoint: str, payload: dict, idempotency_key: str):
    response = requests.post(
        endpoint,
        json=payload,
        headers={"Idempotency-Key": idempotency_key},
        timeout=5,
    )
    response.raise_for_status()
    return response.json()

Four things are happening here. The circuit breaker prevents calls to a service that is demonstrably down. The retry handles transient failures with backoff and jitter. The retry only fires on connection errors and timeouts, not on HTTP 4xx or 5xx responses, because a 4xx is not going to succeed on retry and a 5xx might be a permanent server error. The idempotency key prevents duplicate side effects.

The retry condition is the part most teams get wrong. They retry on any exception, including HTTPError from a 500 response. A 500 from the downstream service might be transient, but retrying immediately makes it worse. If the service is returning 500s because it is overloaded, your retries are part of the problem. I retry only on connection errors and timeouts, which indicate a network-level issue that might resolve. For HTTP 5xx, I let the circuit breaker handle it. If the 5xx rate exceeds the failure threshold, the breaker opens and the calls stop.

The Vienna postmortem

I sat with the Vienna team for the postmortem. The timeline was clear. The payment provider had 90 seconds of elevated latency starting at 14:03:12. The payment service started timing out at 14:03:15. The retry logic fired, immediately, on every timed-out request. By 14:03:30, the thread pool was exhausted. By 14:03:45, the health check was failing. By 14:04:00, Kubernetes had restarted the pods. The new pods loaded, connected to the provider (still slow), and repeated the cycle. The provider recovered at 14:04:42. The platform did not recover until 14:43, because the retry storms kept the thread pools exhausted even after the provider was healthy.

The fix took one afternoon. We added exponential backoff with jitter, a circuit breaker with a 10-failure threshold and 30-second recovery, and idempotency keys on the payment calls. We also changed the retry condition to only fire on timeouts, not on HTTP errors. The total change was 40 lines of code.

We tested it by intentionally slowing down the payment provider in staging. The circuit breaker opened after 10 failures. The payment service returned a “temporarily unavailable” response to the checkout flow. The checkout showed a banner. The rest of the platform stayed up. When the provider recovered, the breaker went to half-open, the test request succeeded, and the breaker closed. The entire degradation lasted 35 seconds and affected one feature.

The team’s CTO asked me why nobody had caught this before. The honest answer is that retry logic is invisible until it breaks. It sits in the code, looks responsible, and passes every code review. The library name is reassuring. The parameters look safe. Nobody questions it because it is a library, not hand-rolled logic. But the defaults are wrong for production, and the defaults are what most teams ship.

The defaults are the problem

Every retry library I have used ships with defaults that are dangerous in production. Tenacity defaults to no wait between retries. Resilience4j defaults to a fixed 500ms wait with no jitter. The Go standard library’s net/http has no built-in retry at all, which means teams write their own, and the ones they write are always naive.

The defaults are designed for the happy path. They demonstrate the library’s API. They are not designed for the case where the downstream service is having a bad day, which is the only case where retry logic matters. Retry logic that works when everything is fine is useless. Retry logic that works when everything is broken is the only kind worth having.

I treat retry logic the same way I treat health checks. If you have one, it needs to be correct, because a wrong one is worse than none. A service with no retry logic fails fast and degrades cleanly. A service with naive retry logic fails slow, amplifies the load, and takes down the services around it. The Vienna team would have been better off with no retries at all. The payment provider’s slow minute would have caused some payment failures, but the platform would have stayed up. Their retry logic turned a minor incident into a major outage.

If you are writing retry logic today, ask yourself four questions. Does it wait between retries? Does the wait have jitter? Does it stop retrying when the downstream service is down? Is the operation safe to repeat? If the answer to any of these is no, your retry logic is a loaded gun. It has not gone off yet. It will.