Canary deployments are a distraction from the fact you cannot roll back

A startup in Amsterdam asked me to help them set up canary deployments. They had a payments API in Kubernetes, twelve replicas, and they wanted to ship changes to 5% of traffic, watch the metrics for ten minutes, and then promote to 100% if nothing looked wrong. They had been reading about progressive delivery. They had a demo of Argo Rollouts working on their laptops. They wanted it in production by Friday.

I asked them one question. “When was the last time you rolled back a deployment?”

The lead engineer thought about it for a while. “We have never rolled back. We just push a fix forward.”

I asked a second question. “What happens to your database when you roll back?”

He did not have an answer. Nobody in the room had an answer. They had spent three weeks evaluating canary deployment tools and zero weeks understanding what happens to their schema when they need to go backward. The canary deployment was going to give them the ability to shift 5% of traffic to a new version and watch metrics. It was not going to help them when the new version ran a migration that could not be reversed, which is the scenario that actually kills startups.

This is the pattern I see repeatedly. Teams chase canary deployments because canary deployments are the deployment strategy that sounds the most sophisticated. They are what Google and Netflix and the big tech companies write about. The tooling is impressive. The dashboards are beautiful. The idea of catching a bad deploy at 5% traffic instead of 100% traffic is compelling. It is also, for the vast majority of startups, solving the wrong problem first.

The problem canary solves, and the problems it does not

Canary deployment solves a specific problem: detecting a regression in a new version by exposing it to a small percentage of traffic before promoting it fully. The assumption is that the regression is detectable in the metrics you are watching, that the metrics are available fast enough to act on, and that the rollback is clean.

Each of those assumptions is a potential failure. Let me take them one at a time.

The regression is detectable in your metrics. This is true for regressions that manifest as latency spikes, error rate increases, or throughput drops. It is not true for regressions that manifest as incorrect data. A payment that is processed with the wrong fee structure does not show up in your error rate. It shows up in your revenue reconciliation three days later. A canary that routes 5% of traffic to a version with a subtle calculation bug will process 5% of your transactions incorrectly, and your metric-based canary analysis will not flag it because the requests are returning 200 OK. The canary catches the failures you instrumented for. It does not catch the failures you did not instrument for, which are most of them.

The metrics are available fast enough. Argo Rollouts and Flagger both analyze metrics over a time window. The default is usually one to two minutes of data after the canary is live. If your p99 latency metric is aggregated over a one-minute window, and the regression manifests as a p99.9 tail latency that only shows up at higher percentiles, the canary analysis window may not catch it. You need a high enough request volume in the canary window for the metric to be statistically meaningful. A startup doing 50 requests per second does 150 requests in a 5% canary over one minute. That is not enough data to distinguish a 2% error rate increase from noise. The canary will promote because the signal is not strong enough to stop it.

The rollback is clean. This is the one that actually matters. Rolling back a canary means routing traffic back to the old version. If the new version made a database change, the old version may not work with the new schema. This is the scenario that breaks startups, and canary deployments make it worse, not better, because the canary created a window where some traffic ran against the new schema and some against the old. If the migration is irreversible, you cannot roll back. If the migration is reversible but the new version wrote data in a format the old version cannot read, you cannot roll back. The canary gave you a fancy traffic shifting mechanism and a broken rollback path.

What I see when teams adopt canary too early

The Amsterdam startup is not an outlier. I have watched four teams adopt canary deployments in the last year. Here is what happened to each.

Team one installed Argo Rollouts, configured a 5% canary with a five-minute analysis window, and shipped their first canary deployment. The canary promoted to 100% because the metrics looked fine. The deployment included a migration that added a NOT NULL column without a default. The old version of the application, which was still running on 95% of traffic during the canary, did not know about the column and was inserting rows that violated the constraint. The database started rejecting writes. The canary analysis was watching error rates on the canary pods, which were fine because the canary pods had the new code. The old pods were failing silently. By the time the canary promoted, the production database had been rejecting writes for six minutes. The rollback failed because the column could not be dropped without losing the data that had been written to it.

Team two set up Flagger with Prometheus analysis. Their canary watched error rate and p99 latency. They deployed a change that fixed a bug in their discount calculation. The fix was wrong in a different way. Discounts were being applied at 15% instead of 10%. The canary promoted because error rates and latency were normal. The bug affected 5% of orders for ten minutes, then 100% of orders for two hours until a customer reported it. The canary tooling was watching the wrong signals.

Team three got canary working correctly. Traffic shifted, metrics were analyzed, the rollback button existed. Then they tried to roll back after a bad deploy and discovered that the new version had consumed messages from their RabbitMQ queue and transformed them into a format the old version did not understand. Rolling back the application meant rolling back to code that could not process the messages that had already been processed. The queue was in an inconsistent state. They spent eight hours manually reconciling messages.

Team four is the one that succeeded. They had canary deployments, but they also had reversible migrations, feature flags for every behavioral change, a tested rollback procedure that they ran every week, and an observability stack that caught the actual failure modes their application produced. The canary was the last piece they added, not the first. It sat on top of a deployment foundation that was already solid. The canary made a good system slightly better. It did not make a bad system good.

The deployment maturity ladder

I think about deployment maturity as a ladder with five rungs. Canary is the fifth rung. Most startups I work with are on rung two or three and are trying to jump to rung five.

Rung one: repeatable deploys. Every deploy is the same process. No SSH, no manual steps, no “run this script then this script then restart this thing.” A merge to main triggers a pipeline that builds an artifact, runs tests, and deploys. If you cannot deploy by merging a pull request, you are on rung zero, and canary deployments are not your problem.

Rung two: instant rollback. You can roll back to the previous version in under two minutes by promoting a prior artifact. No database restoration, no manual fixup, no “we need to reverse the migration by hand.” If your rollback takes longer than your deploy, you do not have a deployment pipeline. You have a one-way valve.

Rung three: safe migrations. Every database migration is reversible. Every schema change follows the expand-contract pattern. No deploy includes a destructive migration. Destructive changes are split across two deploys with a waiting period between them. If you cannot roll back your database schema, you cannot roll back your application, and your canary deployment is a theater that ends when the first bad migration hits.

Rung four: feature flags. Behavioral changes are gated behind flags. You can disable a feature in production without rolling back the deployment. This means a canary is not your only safety mechanism. You can ship code that is dormant and activate it independently. If you do not have feature flags, every deploy is a binary choice between the old behavior and the new behavior, and canary deployments just give you a more complex way of making that binary choice.

Rung five: progressive delivery. Canary, blue-green, traffic shifting, automated analysis. This is where Argo Rollouts and Flagger live. This is the top of the ladder. It is valuable, but it is valuable because it builds on the four rungs below it. Without instant rollback, a canary that detects a problem cannot recover. Without safe migrations, a canary that detects a problem cannot recover from schema changes. Without feature flags, a canary cannot isolate behavioral changes from code changes.

The startups that try to jump from rung two to rung five end up with canary deployments that detect problems they cannot recover from, using metrics that do not capture their actual failure modes, on top of a migration strategy that makes rollback impossible. They have built a sophisticated detection system on top of a broken recovery system. The detection does not help if the recovery is broken.

What I build instead

When I work with a startup that asks for canary deployments, here is what I actually build, in order.

Rolling updates with readiness probes. Kubernetes gives you rolling updates by default. A new pod starts, the readiness probe passes, traffic is routed to the new pod, an old pod is terminated. This is a progressive deployment. It is not a canary, but it is not a big-bang deploy either. The rollout happens one pod at a time. If the new pods crash on startup, the rollout stops. The old pods keep serving traffic. You get automatic detection of startup failures without any additional tooling.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
spec:
  replicas: 12
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    spec:
      containers:
        - name: api
          image: payments-api:2026.07.23
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 3

The maxUnavailable: 0 means no pod is removed until the new pod is ready. The maxSurge: 1 means only one new pod is created at a time. This gives you a rolling deployment that is safe by default. If the new version fails its readiness probe, the rollout halts. The old pods continue serving. You have time to investigate.

For most startups, this is sufficient. The failure mode where a new version starts but serves bad requests is the failure mode that canary catches, and that failure mode is better addressed by feature flags and integration tests than by traffic shifting.

Instant rollback via immutable artifacts. Every deploy produces a versioned container image. The image tag is the git SHA. Rolling back means updating the image tag in the deployment manifest and letting ArgoCD reconcile. No build, no pipeline, no waiting.

# Rollback to a previous version
kubectl set image deployment/payments-api \
  api=ghcr.io/team/payments-api:7a3f2e1 \
  -n production

# Or with ArgoCD, just change the tag in Git
git revert HEAD --no-edit
git push origin main
# ArgoCD detects the change and rolls back automatically

The rollback takes under 90 seconds. The image already exists. The manifest change is a single line. There is no rebuild, no retest, no uncertainty about what is being deployed. The previous version is the previous version, byte for byte.

Reversible migrations. Every migration has an up and a down. The down migration is tested. Destructive migrations are never deployed in the same release as the code that stops using the destructively changed object. The expand-contract pattern is the baseline.

-- Expand: add the new column, nullable
ALTER TABLE orders ADD COLUMN payment_method_new VARCHAR(50);

-- Deploy code that writes to both columns
-- Wait two weeks. Verify the new column is populated.

-- Contract: stop writing to the old column
-- Deploy code that only uses the new column
-- Wait one week. Verify no code references the old column.

-- Cleanup: drop the old column
ALTER TABLE orders DROP COLUMN payment_method_old;

This is the pattern that makes rollback possible. If the expand migration is deployed and the new code has a bug, you roll back the code. The new column stays, nullable and unused. The old column still exists. The old code works. No data is lost. The cleanup migration is a separate deploy, weeks later, after you are confident the old column is truly unnecessary.

If your migrations are not reversible, canary deployments give you a false sense of safety. You can detect a problem at 5% traffic and still not be able to recover from it, because the schema change that came with the canary cannot be undone. You are stranded at 5% with a broken version and no path back.

Feature flags for behavioral changes. Not every change is a code change. Some changes are behavioral. A new pricing calculation. A different checkout flow. A changed API response format. These changes should be gated behind flags, not deployed via canary. A feature flag lets you enable a behavior for a specific percentage of users, for a specific segment, or for internal testing only. It is more granular than a canary and it does not require a new deployment to change the percentage.

# Feature flag check in the application
if flags.is_enabled("new_discount_calculation", user_id):
    discount = calculate_discount_v2(order)
else:
    discount = calculate_discount_v1(order)

The flag can be toggled without a deploy. It can be enabled for 1% of users, watched for a day, and then ramped up. It can be disabled instantly if something goes wrong, without rolling back the deployment. This is progressive delivery at the behavior level, which is where most startup regressions actually live. Canary deployments are progressive delivery at the infrastructure level, which is where fewer startup regressions live.

When canary is actually the right answer

I am not against canary deployments. I am against canary deployments as the first thing you build instead of the last. There are situations where canary is the correct strategy, and I want to be clear about what they are.

You have a high-traffic service where a 5% canary generates statistically meaningful data in a few minutes. If you are doing 10,000 requests per second, a 5% canary is 500 requests per second, and you get enough data to detect a 1% error rate increase in under a minute. At that scale, canary analysis is fast and accurate. At 50 requests per second, a 5% canary is 2.5 requests per second, and you need ten minutes to get enough data to distinguish signal from noise. By then, the canary has been running for ten minutes and the damage is done.

You have a mature migration strategy. Every migration is reversible. Every deploy is independently rollable. The canary is not your rollback mechanism. It is your detection mechanism on top of a rollback mechanism that already works.

You have metrics that capture your actual failure modes. Not just error rate and latency, but business metrics. Payment success rate. Order completion rate. Conversion rate. The metrics that tell you whether the application is doing the right thing, not just whether it is responding to requests. If your canary analysis is watching http_requests_total and http_request_duration_seconds, it will catch infrastructure regressions and miss application logic regressions. You need both.

You have the engineering capacity to maintain the canary tooling. Argo Rollouts and Flagger are not set-and-forget. They require metric analysis configuration, traffic shifting tuning, rollback behavior testing, and ongoing maintenance. A startup with two engineers does not have this capacity. A startup with ten engineers and a dedicated platform person does.

For everyone else, rolling updates with good readiness probes, instant rollback via immutable artifacts, reversible migrations, and feature flags will catch more real regressions than a canary ever would. The boring deployment strategies work because they address the failure modes that actually happen. Canary deployments address a failure mode that is real but rare, and they do it at the cost of ignoring the failure modes that are common and catastrophic.

The question I should have asked first

The Amsterdam startup did not need canary deployments. They needed to be able to roll back. Their database migrations were irreversible. Their deployment artifacts were mutable. Their rollback procedure was “push a fix forward.” The canary deployment they wanted would have given them a 5% traffic shift and a metric analysis dashboard on top of a system that could not recover from any problem the canary detected.

I spent two weeks with them building the things they actually needed. Reversible migrations. Versioned container images. A rollback that worked in 90 seconds. Feature flags for the two behavioral changes they had coming up. By the end of those two weeks, their deployment confidence was higher than it would have been with canary, because they could ship a change and know that if it was wrong, they could go back.

The canary can come later. It will be more useful later. It will sit on top of a system that can actually recover from the problems it detects. Until then, the most sophisticated deployment strategy is the one that lets you undo what you just did. Everything else is a detection system for a fire you cannot put out.