The Terraform State Lock That Froze Our Production Deploys for 6 Hours

It was 4:47 PM on a Friday when the deploy pipeline hung on Acquiring state lock. Nobody panicked at first — state locks clear in seconds. By 5:30 PM three engineers were on a call, and by 10 PM we'd learned more about DynamoDB conditional writes than any of us wanted to.

This is the postmortem, with the parts that are actually useful if you run Terraform against an S3 + DynamoDB backend at any real scale.

The setup

We run Terraform 1.6 with the standard AWS backend: state in S3 with versioning on, locks in a DynamoDB table with LockID as the hash key. CI runs plans and applies from GitHub Actions. One workspace per environment, roughly 40 workspaces total across three AWS accounts.

The apply that triggered the incident was tiny — a security group rule change on a staging workspace. The runner picked up the job, authenticated via OIDC, started terraform apply, and then just sat there.

Acquiring state lock. This may take a few moments...

Ten minutes later, same message. We cancelled the job and retried. Same result. Then we tried a different workspace in the same account. Also stuck.

What the lock actually is

If you've never looked, a Terraform lock in DynamoDB is a single item that looks roughly like this:

{
  "LockID": "my-bucket/env/staging/terraform.tfstate",
  "Info": "{\"ID\":\"7f3c...\",\"Operation\":\"OperationTypeApply\",\"Who\":\"runner@ci\",\"Created\":\"2026-...\"}"
}

Terraform writes this item with a conditional expression (attribute_not_exists(LockID)). If the item already exists, the write fails and Terraform retries with backoff. It will keep retrying until you Ctrl+C or hit the configured timeout.

So "stuck acquiring lock" always means: something else holds the lock, or looks like it does.

What we found

First instinct was a dead runner holding the lock. We queried the table:

aws dynamodb scan \
  --table-name tf-locks-prod \
  --filter-expression "contains(LockID, :env)" \
  --expression-attribute-values '{":env":{"S":"staging"}}'

There was a lock. The Who field pointed to a GitHub Actions runner that had been reaped 40 minutes earlier. Classic stale lock. Easy — terraform force-unlock <ID> and move on.

Except it wasn't easy, because the next apply hung too. And the one after that. Something was reacquiring the lock faster than humans could work.

The real cause

We'd merged a change earlier that week to our drift-detection workflow. It ran terraform plan across every workspace every 15 minutes and shipped drift reports to Slack. Harmless in review. In production, two things went wrong:

1. The workflow had no concurrency group, so overlapping runs stacked up when plans took longer than 15 minutes.
2. A recent provider upgrade made plans against one workspace take 11+ minutes because of a slow aws_iam_policy_document data source lookup pattern.

Result: at any given moment, three to five drift-detection runs were queued or running against the same workspace, each grabbing the lock, holding it for the duration of the plan, releasing it, and the next one grabbing it 200 ms later. Our Friday apply never got a window.

terraform plan taking a lock surprises some people. It does, because the plan refresh phase reads and potentially updates the state file. You can disable it with -lock=false, but then concurrent plans can race on refresh. We'd been running with locks on, which was correct — it just meant plans competed with applies.

The six hours

Here's roughly how the incident went:

• 5:00 PM — Confirmed stale lock, ran force-unlock, retried. Still stuck.
• 5:45 PM — Realized drift workflow was involved. Disabled it in the GitHub Actions UI. Lock cleared, apply started.
• 6:10 PM — Apply failed mid-way with a provider error. State was left in a partially-applied condition. New lock, held by the failed apply, now actually stale.
• 6:30 PM — Force-unlocked again. Attempted re-apply. Hit a state version mismatch because someone had manually imported a resource an hour earlier and the S3 object version we were reading was behind.
• 8:00 PM — Pulled the state down with terraform state pull, inspected it, confirmed no corruption, pushed it back.
• 9:15 PM — Clean apply finally went through. We spent another hour verifying no drift and writing up what happened.

The outage wasn't customer-facing — nothing in production broke. But we couldn't ship anything for six hours, including a security patch that was supposed to go out that evening. That counts.

What force-unlock actually does, and when it's dangerous

terraform force-unlock <LOCK_ID> deletes the DynamoDB item. That's it. It does not touch the state file in S3.

It's safe when:

• The process that acquired the lock is definitely dead.
• No other process is currently mid-apply against that state.

It's dangerous when:

• The lock holder is still running. You'll get two concurrent applies writing to the same state, and the last writer wins. State corruption is real here — resources Terraform created can become orphaned, and resources it thinks it manages can get double-provisioned.

Before force-unlocking, we now always:

1. Check the Who and Created fields on the lock item.
2. Confirm the referenced CI job is actually terminated (GitHub API, not just "it looks dead").
3. Grab the current S3 state object version ID and save it locally. If something goes wrong, you can restore from that version because the bucket has versioning on. (You do have versioning on, right?)

aws s3api list-object-versions \
  --bucket tf-state-prod \
  --prefix env/staging/terraform.tfstate \
  --max-items 5

The guardrails we added

None of these are clever. They're the things we should have had on day one.

Concurrency groups on every Terraform workflow

GitHub Actions supports this directly. One line:

concurrency:
  group: terraform-${{ matrix.workspace }}
  cancel-in-progress: false

For drift detection, cancel-in-progress: true is fine — a stale drift report is worse than no new one. For applies, keep it false so you don't abort a half-finished apply.

Lock timeout, not infinite wait

We set -lock-timeout=5m on every Terraform command in CI. If a lock can't be acquired in five minutes, the job fails loudly instead of silently waiting an hour.

A tf-lock-status command

Small internal CLI that scans the lock table, correlates each lock to a GitHub Actions run, and flags any lock whose owning run has finished. Runs every 10 minutes and posts to a #infra-alerts channel.

Separate drift detection from the apply path

Drift detection now runs against a read replica concept: we terraform state pull to a temporary location, run plan with -lock=false against a copied state, and throw the result away. It can't block applies because it never touches the real lock. The tradeoff is that drift detection can't catch state-level issues like checksum mismatches, which we accept.

Separate state files for high-churn resources

The staging workspace that caused the slow plan had 400+ resources, including a huge IAM policy module. We split it into three states: network, compute, and IAM. Plans dropped from 11 minutes to under two. Smaller blast radius too.

Where we'd start if you're reading this before your own incident

If you run Terraform in CI against a shared backend:

• Add -lock-timeout to every command today. It's the single highest-leverage change.
• Put concurrency groups on every workflow that touches state. Match the group to the state file path, not the repo.
• Make sure S3 state bucket versioning is on, and that your CI role can list and restore prior versions. Test the restore once, in staging, before you need it.
• If you have drift detection, measure how long its plans actually take and compare to its schedule interval. If plan time is more than half the interval, you're going to stack.
• Write down the force-unlock runbook before you need it. Include who's allowed to run it (we require two engineers on a call).

Terraform's locking model is fine. It's small, understandable, and works the way the docs say. What bit us was treating locks as something that always clears quickly instead of as a shared resource that queued workloads compete for. Once you see it that way, the fixes are obvious — and if you're interested in how we design CI pipelines around this kind of shared infrastructure, our DevOps and cloud services page has more on how we approach it.