When manual deployments break: a startup postmortem

A startup in Munich deployed to production by SSHing into a server and running git pull. They did this for eighteen months. It worked. They shipped features fast, they had no pipeline to maintain, and they did not spend engineering time on tooling they felt they did not need. Their investor was happy. Their customers were happy. The team was four engineers and they were moving.

Then on a Tuesday afternoon, someone ran git pull on the wrong server.

The command was meant for the staging box. The engineer had three SSH sessions open in tmux, one for each environment. The panes looked identical. The prompt did not show the hostname because nobody had configured it. He pulled the staging branch onto the production server, ran the database migration that came with it, and restarted the application. The migration dropped a column that production still depended on. The application started, served 500 errors to every request, and the engineer realized what he had done roughly ninety seconds later.

He could not roll back. There was no artifact to roll back to. There was no previous version of the database schema to restore. The git pull had overwritten the working tree, the migration had altered the schema, and the only recovery path was to restore the database from the last backup, which was fourteen hours old. They lost fourteen hours of user data. The platform was down for six hours. It was a Wednesday before everything was fully restored and verified.

I got called in on Thursday. Not to fix the immediate problem, which they had handled by then, but to make sure it could not happen again. The founder said, “We need CI/CD.” I said, “You need more than CI/CD. You need to understand why this happened, because CI/CD would have prevented this specific incident, but the failure mode that caused it is broader than the deployment process.”

This is the postmortem.

What happened, precisely

The engineer had three tmux panes open. Left pane: production, ssh prod-server. Middle pane: staging, ssh staging-server. Right pane: his local machine. The prompts in all three panes were user@ip-10-0-x-x:~$. The only difference was the last octet of the IP. He had been switching between panes all day. He intended to pull the staging branch onto the staging server. He clicked into the left pane instead of the middle pane and ran:

git pull origin staging
npm run migrate
pm2 restart api

Three commands. The first brought the staging code onto the production server. The second ran a migration that included this:

ALTER TABLE orders DROP COLUMN legacy_payment_ref;

The legacy_payment_ref column had been removed in staging two weeks earlier. The migration was a cleanup that had been sitting in the staging branch waiting to go to production. But it went to production out of order, before the application code that stopped referencing the column. The production application still queried legacy_payment_ref on every order. The column was gone. Every query failed.

The third command restarted the application, which immediately began serving 500s.

The engineer noticed when his own Slack bot, which monitored the health endpoint, posted an alert. He looked at the pane he had run the commands in, saw the production hostname in the SSH known_hosts output from the initial connection (which had scrolled off screen), and realized the mistake. He tried to revert:

git checkout main
npm run migrate
pm2 restart api

The checkout reverted the code. The migration tried to run the up migration for the production branch, which did not include recreating the dropped column. The migration framework tracked which migrations had been applied, and since the staging migration had already run, it did not attempt to re-apply it. The column stayed dropped.

He manually ran the SQL to recreate the column:

ALTER TABLE orders ADD COLUMN legacy_payment_ref VARCHAR(255);

The column came back, but it was empty. The data that was in it was gone. The DROP COLUMN had destroyed it. PostgreSQL does not keep dropped column data. There was no way to recover it without restoring from backup.

The backup was fourteen hours old because their backup script ran once a day at 2am. The incident happened at 4pm. Fourteen hours of orders, user profile updates, and transaction records were gone.

The five failures, not one

The team’s initial reaction was “we need to be more careful.” This is the wrong conclusion. Carefulness is a policy, not a control. You cannot engineer safety out of individual attention. The incident was not caused by a lack of care. It was caused by five separate failures, each of which was individually survivable but collectively catastrophic.

Failure one: no environment isolation. The engineer could SSH into production and staging from the same machine with the same key. There was no bastion host, no separate IAM role, no IP-based restriction. The access model assumed that engineers would always know which environment they were in. The access model was wrong.

Failure two: no deployment artifact. The deployment was a git pull, which means the server was building from source at deploy time. There was no versioned artifact, no container image, no immutable release. You cannot roll back a git pull reliably because the working tree is mutable state. The previous version is not stored anywhere. The only way to go back is to git checkout <previous-commit>, which assumes you know the previous commit and that the working tree is clean.

Failure three: no migration safety. The migration framework tracked applied migrations by filename, not by content. The staging migration that dropped the column was a different file than the production migrations. When it ran on production, the framework applied it because it had not seen that filename before. There was no check for whether the migration was safe for the current schema. There was no down migration. There was no migration review process.

Failure four: no backup frequency. The backup ran once a day. For a platform processing orders, once a day is not a backup strategy. It is a data loss strategy with extra steps. The RPO (recovery point objective) was 24 hours, which means in the worst case you lose 24 hours of data. For a startup, 24 hours of data loss can be a company-ending event.

Failure five: no rollback path. Even if the engineer had noticed the mistake before running the migration, there was no quick rollback. The deployment process was a sequence of manual commands with no transactional boundary. If command one succeeded and command two failed, you were in a half-deployed state with no automated way to recover.

Why CI/CD would have prevented this, and what it would not

A CI/CD pipeline would have prevented this specific incident in three ways.

First, the deployment would not have been a git pull on a server. It would have been a container image or a versioned artifact promoted through environments. The engineer would not have had SSH access to production, because the deployment would have been triggered by the pipeline, not by a human on the server. The wrong-server mistake becomes impossible when you cannot SSH to the server.

Second, the migration would have been part of the artifact. CI would have run the migration against a copy of the production schema in a test database. The migration would have been reviewed in a pull request. The reviewer would have seen the DROP COLUMN and asked whether the application still referenced it. The review might not have caught it, but the migration test would have, because the test database would have been seeded from a recent production snapshot.

Third, the rollback would have been a button. Promote the previous artifact. The pipeline redeploys the previous version. The database migration is the hard part, and this is where CI/CD alone is not enough.

# ArgoCD Application: what production deployment looks like instead of SSH
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-prod
spec:
  source:
    repoURL: ghcr.io/aayanmtn/api
    targetRevision: 1.4.2  # the version you want
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

The version is pinned. To roll back, you change 1.4.2 to 1.4.1 and ArgoCD reconciles. No SSH. No git pull. No tmux panes. No human running commands on a server they should not be touching.

What CI/CD would not have prevented is the bad migration. A DROP COLUMN without a prior deploy that stops referencing the column is a migration design error. CI/CD makes it easier to catch, through migration testing and review, but it does not make it impossible. The migration discipline has to come from the engineering team, not from the pipeline. This is why the expand-contract pattern matters, and why I keep telling startups to learn it before they need it, not after.

The migration that broke the Munich startup should have been two migrations, weeks apart:

-- Migration 1: deploy code that stops writing to legacy_payment_ref
ALTER TABLE orders ALTER COLUMN legacy_payment_ref DROP NOT NULL;
-- Application stops reading the column. Deploy. Wait two weeks.

-- Migration 2: after confirming no code references the column
ALTER TABLE orders DROP COLUMN legacy_payment_ref;

The column is dropped only after the application has stopped using it. If the migration runs out of order, the worst case is an unused column, not a production outage.

The recovery

The recovery took six hours. Here is what happened, because the details matter.

The engineer restored the database from the 2am backup. This took forty minutes. The restore brought the database to its state at 2am, which meant every order, every user action, and every transaction between 2am and 4pm was gone.

The team had to manually reconcile the missing data. They pulled payment records from their payment provider’s API (Stripe, which retains transaction data). They matched transactions to users by email. They re-created order records. This took four hours of manual work by two engineers.

They then had to notify customers whose orders were affected. Roughly 340 orders had been placed during the 14-hour window. Of those, 180 were recoverable from Stripe. 160 were not, because the order metadata (which items, which shipping address) was stored only in their database. They refunded the 160 orders and sent an apology email. The refunds cost roughly EUR 12,000.

The total cost of the incident: six hours of downtime, four hours of engineering time, EUR 12,000 in refunds, and a customer trust hit that is harder to quantify but real. The founder told me two customers churned within a month, citing reliability concerns. Their combined ARR was EUR 38,000.

The cost of preventing it: a CI/CD pipeline, a migration review process, and a backup frequency change. I will get to the numbers.

What I built

I worked with the team for three weeks. The changes, in order of priority:

Backups. This was the first thing I fixed, on day one, because it was the highest risk. I moved them from a daily cron job to continuous WAL archiving with a 5-minute RPO.

# PostgreSQL WAL archiving to S3, every 5 minutes max data loss
archive_command = 'aws s3 cp %p s3://backups/wal/%f'
archive_timeout = 300  # force a WAL switch every 5 minutes

If they have another incident, they lose at most 5 minutes of data, not 14 hours. This single change reduced their risk more than anything else I did.

CI/CD pipeline. I set up a GitHub Actions pipeline that built a container image on every merge to main, ran the migration against a production-schema copy in a test database, and promoted the image to staging. Promotion to production required a manual approval in the pipeline, not an SSH session.

# .github/workflows/deploy.yml
name: Deploy
on:
  push:
    branches: [main]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build image
        run: docker build -t ghcr.io/team/api:${{ github.sha }} .
      - name: Run migrations against test DB
        run: |
          docker run --rm \
            -e DATABASE_URL=postgres://test:5432/testdb \
            ghcr.io/team/api:${{ github.sha }} \
            npm run migrate
      - name: Push image
        run: docker push ghcr.io/team/api:${{ github.sha }}
  deploy-staging:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to staging
        run: kubectl set image deployment/api api=ghcr.io/team/api:${{ github.sha }} -n staging
  deploy-production:
    needs: deploy-staging
    environment: production  # requires manual approval
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to production
        run: kubectl set image deployment/api api=ghcr.io/team/api:${{ github.sha }} -n production

The manual approval is a button in GitHub. It is not an SSH session. The person approving does not have server access. They have a button. The button deploys a specific, tested, versioned artifact. The button cannot deploy the wrong branch to the wrong server because the pipeline knows which branch goes where.

SSH access removal. I removed SSH access to production for all engineers. Production deployments happen through the pipeline. If an engineer needs to debug something in production, they use kubectl exec into the pod, which is audited and time-limited. No more tmux panes with identical prompts.

Migration review process. Every migration now requires a pull request review that specifically asks: does this migration drop or rename a column? If yes, has the application stopped referencing it? Is there a down migration? The review checklist is in the PR template, and it is enforced by a required check.

## Migration Review Checklist
- [ ] Does this migration drop or rename any column or table?
- [ ] If yes, has the application stopped referencing it? (PR link)
- [ ] Is there a reversible down migration?
- [ ] Has the migration been tested against a production-schema copy?

The checklist is not sophisticated. It is a forcing function. The question “has the application stopped referencing it?” is the question that would have prevented the Munich incident. It was not asked because nobody had written it down.

What it cost

The CI/CD pipeline, the backup changes, and the SSH access restructuring took three weeks of my time and two weeks of the team’s engineering time. The infrastructure cost increase was negligible. GitHub Actions was already in their plan. The container registry was free at their scale. The S3 bucket for WAL archiving cost EUR 3 a month. The only meaningful cost was engineering time, which is always the real cost of infrastructure work.

The team’s deployment frequency went from “whenever someone has time to SSH in” to four to six deploys a day. The lead time from merge to production went from hours to under twenty minutes. The rollback time went from “we cannot” to under two minutes. These are not vanity metrics. They are the difference between a team that ships with confidence and a team that ships with dread.

The deeper failure

The deeper failure was not the git pull on the wrong server. The deeper failure was the eighteen months of manual deployments that preceded it. The team had built a deployment process that depended on human precision, and human precision is not reliable. It is not reliable in a two-person startup and it is not reliable in a two-hundred-person engineering organization. The difference is that the two-hundred-person organization has already learned this lesson, usually through the same kind of incident, and has invested in the controls that the two-person startup has not.

The Munich startup was at Series A. They had revenue, customers, and investors. They had the resources to build a proper deployment pipeline. They chose not to because manual deployments were working, and “working” is a word that means “has not failed yet.” It does not mean “will not fail.” It means you have not hit the failure mode yet, and you will.

I see this pattern at every startup that grows past ten engineers with a deployment process designed for three. The process that was fine when the team was small and everyone knew which server was which becomes dangerous when the team grows and nobody knows which server is which. The process does not scale with the team. It scales with the risk. More engineers means more deploys, more sessions, more opportunities for the mistake that was always possible but had not happened yet.

The transition from manual deployments to CI/CD is not about speed. The Munich startup was already fast. They shipped features quickly. The transition is about removing the human from the critical path of a destructive operation. A deployment is a destructive operation. It changes the state of production. If it goes wrong, it can destroy data, break availability, and cost real money. You do not want that operation to depend on which tmux pane someone clicked into.

If you are a startup reading this and recognizing the pattern, Basecamp exists for exactly this reason: digitalaultis.com/basecamp

The postmortem that matters

The team wrote a postmortem. It was good. It identified the five failures, the corrective actions, and the timeline. It was blameless in the real sense, not the performative sense. The engineer who ran the wrong command was not blamed, because the system allowed him to run the wrong command, and the system is what failed.

The last paragraph of the postmortem said: “We treated our deployment process as a convenience. It was a risk. We got lucky for eighteen months. We will not rely on luck again.”

They have not had a deployment incident in the eight months since. The pipeline runs. The backups run. The migrations are reviewed. The SSH sessions are gone. The tmux panes are gone. The human is no longer in the critical path of a destructive operation.

That is the postmortem that matters. Not the document. The change.