Terraform State Corruption at 3 AM: How We Recovered and What We Changed

It was a Tuesday, 03:14 UTC, and our Terraform apply had been running for eleven minutes when the runner crashed. When the on-call engineer re-ran it, Terraform told him the state was locked. When he forced the unlock — which, yes, we'll get to — the next plan wanted to destroy the production RDS cluster. That's the story. Here's what we learned.

What actually broke

We run our infra on AWS with an S3 backend for Terraform state and DynamoDB for state locking. Standard setup, been fine for years. The apply in question was a routine change: adding two IAM policies and rotating a KMS key alias. Nothing exotic.

What happened, in order:

1. GitHub Actions runner spun up, acquired the DynamoDB lock, started the apply.
2. Roughly nine minutes in, the runner hit its 10-minute idle timeout on a spot instance and was reclaimed mid-apply.
3. The terraform apply process was killed before it could write the updated state back to S3.
4. The DynamoDB lock, however, was still held — the process never got to release it.
5. On-call ran terraform force-unlock <LOCK_ID> and re-ran the plan.
6. Terraform read the pre-apply state from S3, compared it to reality (which had drifted because half the apply had gone through), and produced a plan that included destroying resources it thought were duplicates.

The engineer caught it. Barely. The plan output had a - next to aws_db_instance.primary and he stopped before typing yes. That's the only reason this is a blog post and not a much longer incident report.

Why the lock wasn't the real problem

Everyone's first instinct after this kind of near-miss is "the lock system failed". It didn't. DynamoDB did exactly what it was told. The problem is that Terraform's locking protects the state file, not the real infrastructure. If your process dies after mutating cloud resources but before writing state, you have drift, and no lock in the world fixes that.

A Terraform state lock is a mutex for the state file. It is not a transaction across your cloud provider. Treat it that way and you'll stop being surprised.

The recovery, step by step

We didn't recover cleanly on the first try. We recovered on the third. Here's the sequence that finally worked, sanitised.

1. Freeze everything

First move: disable the CI workflow that runs Terraform. Not pause, not comment out — disable in the GitHub UI so nobody can trigger it by re-running a failed job. Post in the incident channel that IaC is frozen. This buys you time to think without a colleague in another timezone "just trying something".

2. Snapshot the state before you touch it

S3 versioning was on (please, please turn this on if it isn't), so we had every historical version of the state file. We pulled the last known-good version and the current broken version to local disk:

# Pull current (post-crash) state
aws s3api get-object \
  --bucket tf-state-prod \
  --key env/prod/terraform.tfstate \
  ./state.broken.json

# List versions and pull the last known-good
aws s3api list-object-versions \
  --bucket tf-state-prod \
  --prefix env/prod/terraform.tfstate

aws s3api get-object \
  --bucket tf-state-prod \
  --key env/prod/terraform.tfstate \
  --version-id <VERSION_ID_BEFORE_APPLY> \
  ./state.lastgood.json

Both files went into a private incident bucket with a timestamp. Do not skip this. If your next command corrupts things further, this is your rollback.

3. Reconcile state with reality, not the other way around

The temptation is to restore state.lastgood.json and re-apply. Don't. The last-good state doesn't know about the IAM policies that did get created during the partial apply. Restoring it means Terraform will try to create them again, hit "already exists" errors, and you'll be doing surgery under pressure.

Instead we did this:

1. Restored state.lastgood.json as the active state.
2. For every resource in the changed module, ran terraform plan -target=<resource> to see what Terraform thought vs. what actually existed in AWS.
3. For resources that already existed in AWS but weren't in state, used terraform import to bring them in.
4. For resources in state but not in AWS, used terraform state rm to drop them.

This took about forty minutes for a module with roughly a dozen resources. Slow, deliberate, -target flags everywhere, one resource at a time.

4. Run a full plan against production and read every line

Once reconciled, we ran a plan against the whole environment — not just the affected module — and read the entire diff out loud on a call. Two engineers. If either of us saw a destroy we weren't expecting, we'd stop. The final plan showed zero changes. That's when we knew we were back.

What we changed afterwards

Recovery is the boring part. The interesting part is what stops this happening again.

Kill the spot runner for apply jobs

We were running Terraform apply on spot GitHub runners because they were cheap. The savings, in our case, were somewhere in the tens of dollars a month. The near-miss cost us hours of senior engineer time and a lot of trust. Apply jobs now run on on-demand runners with a hard 45-minute timeout and no idle-eviction. Plan jobs can stay on spot — they're read-only.

Split plan and apply into separate jobs with an artifact hand-off

We now generate the plan file explicitly, upload it as an artifact, and the apply job downloads and applies that specific plan. If the apply job dies, the plan is still on disk and we can inspect exactly what was in-flight.

- name: Terraform Plan
  run: |
    terraform plan -out=tfplan.binary
    terraform show -json tfplan.binary > tfplan.json

- name: Upload plan
  uses: actions/upload-artifact@v4
  with:
    name: tfplan-${{ github.run_id }}
    path: |
      tfplan.binary
      tfplan.json
    retention-days: 30

Add a drift detector that runs on a schedule

Once a day, a job runs terraform plan -detailed-exitcode against every environment and posts to Slack if the exit code is 2 (drift detected). This would have caught the partial apply within 24 hours even if the on-call had gone home. In our experience, catching drift on a daily cadence dramatically reduces the blast radius when something does go wrong.

Require a human in the loop for destroys

We added an OPA policy in our CI pipeline that fails any plan containing a destroy action on a resource type in an allow-list (RDS, DynamoDB tables, S3 buckets, EKS clusters, KMS keys). It can be overridden with a signed commit message, but the default is: no, you cannot destroy stateful infrastructure without a human explicitly saying so.

Stop using force-unlock as a reflex

The real mistake in the incident wasn't the runner dying — that's going to happen. It was force-unlocking without first asking "why is this locked?". We now have a runbook that says: if you see a stale lock, first check whether the associated CI job is actually still running. If it isn't, pull the state file, diff it against the last version, and only then consider unlocking.

The bit nobody talks about

Terraform's failure modes are boring until they aren't. Ninety-nine applies out of a hundred will complete cleanly, and you'll build up the intuition that state and reality are the same thing. They aren't. State is a cache of reality, updated on a best-effort basis, and any time your apply process is interrupted you should assume the cache is stale.

The teams I've seen handle this best treat every apply as potentially interruptible and design their pipelines around that assumption. The teams I've seen handle it worst are the ones with a Slack channel full of terraform force-unlock commands and no one asking why.

Where we'd start

If you inherit a Terraform setup tomorrow, do these four things in the first week, in this order: turn on S3 versioning for the state bucket, move apply jobs off spot/preemptible runners, split plan and apply with an artifact hand-off, and add a scheduled drift detector. That's maybe a day of work and it eliminates the top three ways state gets corrupted in the first place.

We write more about IaC pipelines and reliability work on the 72Technologies blog, and if you want a second pair of eyes on your Terraform setup before it bites you at 3 AM, that's part of what our DevOps engagements cover.