Terraform drift is the incident you are not detecting

A startup in Helsinki called me on a Thursday afternoon. Their Terraform plan had been failing for three days. Nobody had noticed because the pipeline was set to only run Terraform on infrastructure changes, and nobody had pushed an infrastructure change since Monday. On Thursday, someone tried to add a new security group rule. The plan failed with an error about a subnet that Terraform thought existed but AWS reported as missing.

Here is what happened. On Monday night, a contractor logged into the AWS console to investigate a networking issue. He found a subnet that looked misconfigured. He deleted it. He recreated it manually with what he thought were the same settings. He did not update Terraform. He did not tell anyone. He fixed the immediate problem, closed his laptop, and went home.

The subnet he created had a different subnet ID. The route table association was different. The CIDR block was the same but the availability zone was different. Terraform’s state file still referenced the old subnet ID. Every resource that depended on that subnet, three security groups and a NAT gateway, was now pointing at an ID that no longer existed in AWS.

The application kept running because the EC2 instances were still up, still attached to the old ENIs, still routing traffic through the NAT gateway that the contractor had manually reconfigured. The drift was invisible. The infrastructure was working. The state file was wrong. For three days, the gap between what Terraform believed and what AWS actually had was growing, and nobody was looking.

When the engineer tried to add the security group rule on Thursday, Terraform tried to reconcile state against reality. It found a subnet in state that did not exist in AWS. It tried to plan around it. The plan failed. Now the team could not make any infrastructure changes through Terraform until someone reconciled the state file with the manual changes the contractor had made.

It took six hours to fix. Not because the fix was complicated. The fix was a terraform import and a few state moves. It took six hours because nobody knew what the contractor had changed, the contractor was on a flight and unreachable, and the team had to reverse-engineer the current AWS state by comparing it against the Terraform state file line by line.

This is what infrastructure drift looks like. It is not dramatic. It does not trigger alarms. It accumulates in the gap between your state file and reality, and it surfaces at the worst possible moment, which is when you need Terraform to work and it cannot.

What drift actually is

Drift is the difference between what Terraform’s state file says exists and what actually exists in your cloud provider. Terraform state is a snapshot. It is a record of what the world looked like the last time Terraform ran successfully. It is not a live view. It does not update when someone changes something outside Terraform.

There are three ways drift enters your infrastructure, and I see all three in every company I work with.

The first is manual console changes. Someone logs into AWS, GCP, or Azure and makes a change directly. They add a security group rule, they resize an instance, they modify a load balancer configuration. They do this because it is faster than writing Terraform, or because they do not know Terraform, or because it is 2am and the production site is down and they will fix the Terraform later. They never fix the Terraform later.

The second is out-of-band automation. A script that someone wrote six months ago modifies cloud resources directly. A Lambda function that scales instances. A cron job that rotates SSL certificates. A Kubernetes controller that provisions load balancers. These tools do not know about Terraform, and Terraform does not know about them. They change reality without updating state.

The third is provider-side changes. AWS deprecates an instance type and replaces it. GCP renames a service. Azure migrates a resource to a new API version. These changes happen silently, and Terraform state does not update to reflect them until you run a plan and discover that the resource you defined no longer matches what the provider expects.

The result is the same in all three cases. Your state file is lying to you. It says the world is one way, and the world is actually another way. The lie is harmless until it is not, and the moment it becomes harmful is the moment you need Terraform to make a change and it cannot reconcile the lie with reality.

Why nobody detects it

The reason drift goes undetected is simple. Teams run terraform plan before they apply changes. They do not run it when there are no changes to make. The plan command is treated as a pre-deploy check, not as a monitoring tool.

This means drift can exist for days, weeks, or months before anyone notices. I audited a company in Berlin last year that had 47 drifted resources in their Terraform state. Forty-seven. They had been accumulating for eight months. Security group rules that had been added manually. An S3 bucket policy that had been changed by a compliance script. A DNS record that had been updated by their CDN provider’s automation. Three instance types that AWS had silently upgraded during a maintenance window.

Nobody knew because nobody was looking. The Terraform pipeline ran on infrastructure PRs, and nobody had opened an infrastructure PR in six weeks. When they finally did, the plan showed 47 changes, most of which were Terraform trying to revert manual changes back to what the state file said. Half of those reverts would have broken things, because the manual changes were actually correct and the Terraform configuration was outdated.

This is the dangerous part of drift. When you finally run a plan and see unexpected changes, you do not know which side is correct. Is the state file right and the cloud wrong? Is the cloud right and the state file wrong? Did someone make a deliberate manual change that should be preserved, or did someone make a mistake that should be reverted?

You cannot answer that question without a history of what changed and why. And you do not have that history because the changes happened outside Terraform, outside version control, and outside any audit trail.

The detection baseline

The fix is to run drift detection continuously, not just before deploys. Terraform has a flag for this that most teams do not know about.

terraform plan -detailed-exitcode

The -detailed-exitcode flag changes the exit code behavior of plan. Exit code 0 means no changes. Exit code 2 means there are changes, which means either someone modified the configuration or there is drift. Exit code 1 means an error.

If you run this on a schedule, with no configuration changes, exit code 2 means drift. That is your signal.

Here is the CI pipeline I set up for every team I work with. It runs every four hours against every environment, and it posts to Slack if it detects drift.

# .github/workflows/drift-detection.yml
name: Terraform Drift Detection
on:
  schedule:
    # Every 4 hours
    - cron: "0 */4 * * *"
  workflow_dispatch:

jobs:
  detect-drift:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        environment: [staging, production]
    steps:
      - uses: actions/checkout@v4

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.TERRAFORM_ROLE_ARN }}
          aws-region: eu-central-1

      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: "1.7.0"

      - name: Terraform Init
        working-directory: terraform/environments/${{ matrix.environment }}
        run: terraform init -input=false

      - name: Detect Drift
        working-directory: terraform/environments/${{ matrix.environment }}
        run: |
          terraform plan -detailed-exitcode -input=false -out=drift.tfplan
          # Exit code 0: no drift. Exit code 2: drift detected.
          # We want to fail the job on drift, so invert the logic.
        continue-on-error: true
        id: plan

      - name: Notify on Drift
        if: steps.plan.outcome == 'failure'
        run: |
          # Generate a readable diff of what drifted
          terraform show -json drift.tfplan > drift.json
          jq -r '.resource_changes[] | select(.change.actions != ["no-op"]) | "\(.change.actions | join(",")): \(.address)"' drift.json > drift-summary.txt

          curl -X POST ${{ secrets.SLACK_WEBHOOK }} \
            -H 'Content-Type: application/json' \
            -d "{
              \"text\": \"Terraform drift detected in ${{ matrix.environment }}\",
              \"blocks\": [{
                \"type\": \"section\",
                \"text\": {
                  \"type\": \"mrkdwn\",
                  \"text\": \"*Terraform drift detected in ${{ matrix.environment }}*\n\n$(cat drift-summary.txt | head -20)\"
                }
              }]
            }"

      - name: Fail on Drift
        if: steps.plan.outcome == 'failure'
        run: exit 1

This pipeline does three things. It runs plan with -detailed-exitcode on a schedule. If it detects drift, it generates a summary of which resources changed. It posts that summary to Slack and fails the job so the drift shows up in your CI dashboard.

The four-hour interval is a judgment call. Every hour is noisy. Every day is too slow. Four hours means you will know about drift within half a workday, which is fast enough to catch it while the context of what changed is still fresh in someone’s head.

The Slack notification is the important part. Drift detection that writes to a log file nobody reads is not drift detection. It is drift logging. The notification needs to go somewhere engineers actually look, and it needs to include enough detail that someone can start investigating without running a plan themselves.

What to do when you find drift

Detecting drift is half the problem. The other half is deciding what to do about it. There are three options, and the wrong one is to blindly run terraform apply.

Option one: the drift is a mistake. Someone made a manual change that should not have been made. You run terraform apply to revert the cloud to match the state file. This is the correct response when the manual change was wrong, like a security group rule that opened port 22 to the internet.

Option two: the drift is correct and the Terraform configuration is outdated. Someone made a manual change that was necessary, and the Terraform code should be updated to reflect it. You update the Terraform configuration, run terraform plan to confirm it now matches reality, and commit the updated configuration. This is the correct response when the manual change was a legitimate fix that should be permanent.

Option three: the drift is correct but temporary. Someone scaled up instances for a traffic spike, or added a temporary security group rule for debugging. You acknowledge the drift, document it, and set an expiry date. When the temporary change is no longer needed, you remove it and let Terraform reconcile.

The problem is that most teams do not know which option applies. They see a plan with unexpected changes, panic, and either apply it blindly or cancel it and hope it goes away. Neither is correct. You need to investigate, and investigation requires knowing who made the change and why.

This is where cloud audit logs matter. Every cloud provider has an audit log. AWS has CloudTrail. GCP has Cloud Audit Logs. Azure has Activity Logs. If you have drift, the first thing you do is check the audit log for the resource that drifted.

# AWS CloudTrail: who modified this security group in the last 24 hours?
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=sg-0abc123def456 \
  --start-time $(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%S) \
  --max-results 20 \
  --query 'Events[].{Time:EventTime,User:Username,Action:EventName}' \
  --output table

If your drift detection pipeline includes this step, you do not just know that drift happened. You know who did it, when, and through what interface. That changes the conversation from “something drifted” to “the contractor modified the subnet at 2am on Monday through the console,” which is a conversation you can actually act on.

The cultural problem

The technical fix for drift is a scheduled plan and a Slack notification. The cultural fix is harder. Drift happens because people make changes outside Terraform, and they do that because Terraform is slow, because they do not know Terraform, or because the situation is urgent and the console is faster.

You cannot solve this by banning console access. Engineers need to be able to respond to incidents, and sometimes the console is the right tool. The contractor in Helsinki was not wrong to fix the networking issue through the console at 2am. He was wrong to not update Terraform afterward and not tell anyone.

The cultural fix is a process, not a rule. It has two parts.

The first part is a break-glass process for manual changes. When someone makes a manual change to infrastructure, they create a ticket within one hour that describes what they changed, why, and what needs to happen in Terraform to reconcile. The ticket is not optional. It is not a best practice. It is a requirement, and the drift detection pipeline is the enforcement mechanism. If drift is detected and there is no ticket, someone has a conversation with their manager.

The second part is making Terraform fast enough that people prefer it to the console. If adding a security group rule through Terraform takes thirty minutes because the pipeline is slow, the state file is large, and the plan takes ten minutes to run, people will use the console. If it takes two minutes, they will use Terraform. Speed is a security feature. A Terraform pipeline that takes twenty minutes to apply a one-line change is a pipeline that creates drift.

I worked with a team in Tallinn whose Terraform plan took twelve minutes to run. Their state file had 3,400 resources. Every plan refreshed every resource. Engineers used the console for everything because Terraform was too slow to be practical. We split the Terraform configuration into smaller workspaces by environment and service. The plan time dropped from twelve minutes to ninety seconds. Console usage dropped by 80% in the following month, and drift detections went from three per week to one per month.

The lesson is that drift is not a discipline problem. It is a tooling problem. If your tools are slow enough that people work around them, the workaround becomes the drift.

The state file is not a database

There is a deeper issue here that I want to name explicitly. Teams treat the Terraform state file as a source of truth. It is not. It is a cache. It is a cache of what the cloud looked like the last time Terraform ran. Caches go stale. The state file goes stale. The question is not whether your state file is accurate. The question is how quickly you notice when it stops being accurate.

This is why drift detection matters more than state file management. You can use S3 and DynamoDB for state locking. You can use workspaces, you can use modules, you can structure your state file perfectly. None of that prevents drift. Drift comes from outside Terraform, and no amount of state file hygiene will detect it.

The teams that handle drift well are the ones that treat the state file with suspicion. They know it is a cache. They know caches lie. They verify the cache against reality on a schedule, and they have a process for when the verification fails.

The teams that handle drift poorly are the ones that trust the state file. They run terraform plan, see no changes, and believe the world is fine. They do not realize that “no changes” means “no changes relative to the state file,” not “no changes relative to reality.” If the state file is wrong, a clean plan is not reassuring. It is a lie told by a stale cache.

What I install

When I set up infrastructure for a team, the drift detection baseline is four things.

A scheduled terraform plan -detailed-exitcode that runs at least every four hours against every environment. Not just production. Staging drifts too, and staging drift that goes undetected becomes production drift when someone promotes the staging configuration.

A Slack or Teams notification that fires on drift, with a summary of which resources changed. The notification includes a link to the CI job so someone can investigate without context-switching.

A CloudTrail or equivalent audit log query that runs alongside the drift detection and identifies who made the change and when. The notification includes the actor and the timestamp. “Drift detected” is a signal. “Drift detected, changed by contractor-x at 2:14am via console” is an investigation.

A break-glass ticket process that requires anyone making a manual infrastructure change to document it within one hour. The drift detection pipeline cross-references open tickets. If drift is detected and no ticket exists, the Slack message says so explicitly. That message is the trigger for a conversation, not a punishment.

None of this is complicated. It is a scheduled CI job, a webhook, an API call, and a ticketing convention. The complicated part is accepting that your state file is a cache, that caches go stale, and that the staleness will hurt you if you do not watch for it.

The Helsinki startup now detects drift within four hours. They have not had a drift-related incident since the pipeline went in. They still get drift notifications, roughly twice a month, usually from the same contractor who has now learned to create tickets. The notifications are smaller. The investigations are faster. The state file is still a cache, but it is a cache they verify, and that is the difference between infrastructure you trust and infrastructure that surprises you.