The Terraform State Lock That Held Our Deploys Hostage for 6 Hours

On a Tuesday morning last quarter, every Terraform apply in the company started hanging on Acquiring state lock. Twelve teams, one shared backend, zero deploys. It took us six hours to fully recover — not because the fix was hard, but because our first three attempts made things worse.

This is the postmortem, minus the names and account IDs. If you run Terraform at any scale with an S3 + DynamoDB backend, the same failure mode is waiting for you.

The setup

We run Terraform (0.14 through 1.7 across various repos, don't judge) with the standard AWS remote backend: S3 for state, DynamoDB for the lock table. One lock table, terraform-locks, shared across roughly 40 state files spanning platform, product, and data infra.

CI runs applies through GitHub Actions on self-hosted runners in EKS. Each apply job requests the lock, holds it while planning and applying, releases it on exit. Under normal load we see maybe 30 – 60 applies per day across the org, with occasional bursts during release windows.

The lock table itself is trivial:

resource "aws_dynamodb_table" "terraform_locks" {
  name         = "terraform-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

That table had been running untouched for about three years.

What we saw first

At 09:14 a platform engineer pinged #infra-help: her apply on the networking state had been sitting at Acquiring state lock for eight minutes. By 09:20 three more people had piped up. By 09:30 the on-call (me) was staring at a Slack channel that looked like a denial-of-service.

The symptoms:

• Every apply against any state file hung on lock acquisition.
• terraform plan with -lock=false worked fine.
• The DynamoDB table showed normal metrics — no throttling, no 5xx, read/write capacity fine on PAY_PER_REQUEST.
• CloudTrail showed successful PutItem and GetItem calls against the table.

That last point is what threw us. If the table was healthy, why couldn't anyone get a lock?

The actual failure

When we finally scanned the table (should have been step one, was step four), we found roughly 180 lock rows. Every state file had between one and twelve stale locks. The oldest was from a CI job that had been killed by a node eviction the previous evening.

Here's what nobody had internalised: Terraform's lock isn't a mutex on the table. It's a conditional PutItem on a row keyed by the state file path. If the row exists, you can't acquire. If your process dies without releasing, the row stays. Forever. There is no TTL on Terraform lock rows by default.

We had been accumulating stale locks for months. Most were harmless because they were on rarely-touched states. But something — we still aren't 100% sure what — caused a cascade the previous night where a runner pool restart killed maybe 40 in-flight applies simultaneously. Those 40 stale locks tipped enough shared modules into a state where any dependent apply also blocked.

Why our runbook made it worse

Our runbook for stuck locks said, verbatim: "Run terraform force-unlock <LOCK_ID> after confirming no active apply is running."

Three problems with that.

First, force-unlock operates on one lock at a time and requires you to grab the lock ID from the error message. When you have 180 stale locks, that's not a runbook, it's a punishment.

Second, "confirming no active apply is running" is harder than it sounds when your CI is a self-hosted runner pool with ephemeral pods. We had no reliable way to correlate a lock row to a specific job.

Third — and this is the one that actually hurt us — one of our senior engineers, trying to help, wrote a quick script to force-unlock every lock in the table. It worked. It also nuked the locks of the four legitimate applies that were running at that moment, one of which was halfway through a VPC change. That apply corrupted a state file. Recovering it took another two hours and a very careful terraform state pull / manual JSON edit / terraform state push dance that I would not recommend to anyone.

Rule we now enforce: no bulk lock operations, ever. If you're deleting more than one lock, you stop and page a second engineer.

What actually fixed it

Once we accepted that we needed to be surgical, the recovery was mechanical:

1. Snapshot the DynamoDB table (point-in-time recovery was on — thank you, past us).
2. Query the table for lock rows, join against the CI system's recent job history.
3. For each lock, confirm the associated job was terminated (not still running).
4. Delete the row directly via the AWS CLI, one at a time, with a second engineer confirming.

A sample of the check we ran per lock:

aws dynamodb get-item \
  --table-name terraform-locks \
  --key '{"LockID":{"S":"my-bucket/env/prod/networking.tfstate-md5"}}' \
  --query 'Item.Info.S' \
  --output text | jq .

The Info field contains JSON with Operation, Who, Created, and — critically — the CI job identifier if you populate it. We didn't populate it consistently. That's now mandatory.

The guardrails we added

We spent the following week making sure this class of incident couldn't repeat. Nothing here is clever; it's all stuff we should have had on day one.

1. Lock metadata that actually helps

Every CI job now sets TF_HTTP_USERNAME and a custom lock info header that includes the workflow run URL. When a lock is stale, you can click through to the exact job that owned it in about ten seconds.

2. A scheduled stale-lock reaper — with brakes

A Lambda runs every 15 minutes, scans the lock table, and flags any lock older than 90 minutes. It does not delete anything. It posts to Slack with the lock ID, the owning job URL, and a one-click button that triggers a second Lambda to delete that specific lock after checking the CI job status via the GitHub API.

The important design choice: the reaper cannot bulk-delete. Ever. Each action is one lock, one confirmation.

3. Splitting the lock table

We moved from one shared terraform-locks table to per-team tables. This was mildly annoying to migrate but massively reduced blast radius. A stuck lock in the data team's backend no longer creates a Slack panic in the platform team's channel.

If you're setting this up fresh, the pattern is boring:

terraform {
  backend "s3" {
    bucket         = "acme-tfstate-platform"
    key            = "networking/prod.tfstate"
    region         = "eu-west-1"
    dynamodb_table = "terraform-locks-platform"
    encrypt        = true
  }
}

One lock table per bucket, one bucket per team or domain. It's not glamorous but it contains failures.

4. Kill-signal handling in CI

Our runners now trap SIGTERM and attempt a graceful terraform exit before the pod dies. It doesn't always work — if the node itself dies you get nothing — but it catches the common case of a scale-down event, which was the trigger the night before our outage.

5. A real runbook

The new runbook has three paths: single stuck lock (use the Slack button), multiple stuck locks on unrelated states (page a second engineer, work through them one at a time), and "the whole table looks broken" (freeze all applies via a repo-level flag, then investigate). No option involves a loop.

Things we considered and rejected

A few ideas came up in the postmortem that sound reasonable but that we chose not to implement.

DynamoDB TTL on lock rows. Tempting, but it hides the underlying problem. A lock that expires silently at 24 hours means someone's apply could resume against a state that another job has since modified. We'd rather the lock stay and someone investigate.

Moving to a different backend (Terraform Cloud, Spacelift, etc.). We evaluated it. For our volume the cost and migration risk didn't justify it. If you're smaller, or greenfield, honestly — just use a managed runner. The S3+DynamoDB backend is fine, but it's fine in the way that self-hosting Postgres is fine: you own every failure mode.

A cross-team "apply freeze" bot. We already have this as a manual flag. Automating it felt like adding a system that would itself need a runbook.

Where we'd start

If you're running Terraform on the S3+DynamoDB backend and you've never audited your lock table, do this today:

1. Scan the table. Count rows older than an hour. That's your stale lock count.
2. Check whether your CI populates the lock Info field with something you can trace back to a job. If not, fix that this week.
3. Delete your "force-unlock everything" script if one exists. Replace it with a one-lock-at-a-time process that requires a human.
4. Consider splitting your lock table by team or domain. It's an afternoon of work and it caps your blast radius.

Stuck locks are one of those failures that feels rare until the day it isn't. The DynamoDB row model is elegantly simple, which is also why it fails silently and accumulates cruft. Treat the lock table as a first-class piece of infrastructure — monitored, alerted on, and boring — and this incident stays hypothetical.

If you want a hand auditing your IaC and CI setup before it bites, our platform engineering team does this kind of work with clients who'd rather not learn these lessons the way we did.