All articles
DevOps & CloudNovember 17, 2026 6 min read

Terraform State Locking Failed Mid-Apply: What We Learned About DynamoDB and S3 Backends

A half-applied Terraform run, a stuck DynamoDB lock, and a VPC that almost lost its NAT. Here's how state locking actually fails, and the backend hygiene we wish we'd had six months earlier.

A Terraform apply hung for 40 minutes on a Friday afternoon. The engineer running it killed the process. Monday morning, nobody could plan against that workspace — and the half-applied change had already deleted a route table association we very much needed.

This is the story of how we got there, what the S3 + DynamoDB backend actually guarantees (and what it doesn't), and the backend hygiene we now treat as non-negotiable.

The setup: a backend we thought was boring

Our networking stack lived in a single Terraform workspace, backed by the usual AWS pattern:

terraform {
  backend "s3" {
    bucket         = "acme-tfstate-prod"
    key            = "networking/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

That DynamoDB table is the lock. S3 holds the state; DynamoDB holds a single item with a LockID that Terraform writes on plan/apply and deletes when it's done. If someone else tries to run against the same state, they get the familiar:

Error: Error acquiring the state lock
Lock Info:
  ID:        9f4b...
  Operation: OperationTypeApply
  Who:       alex@laptop
  Created:   2026-01-17 14:03:11 UTC

For three years, this was fine. Then it wasn't.

What actually went wrong

The engineer ran terraform apply from a laptop over a flaky coffee-shop Wi-Fi connection. The apply started modifying a VPC: removing one route table association, planning to add a new one. Partway through, the TCP connection to the AWS API dropped long enough that Terraform's AWS provider started retrying. The laptop went to sleep. When it woke up, the Terraform process was gone — but the DynamoDB lock item was still there, and the state in S3 was now mid-mutation.

Two problems compounded:

  1. The lock wasn't released. Terraform only deletes the lock item on clean exit. SIGKILL, laptop sleep, or a dropped SSH session all leave the lock behind.
  2. The state file reflected a partial reality. Some resources had been updated in S3; others hadn't. The real AWS infrastructure was a third state, different from both.

What the S3 + DynamoDB backend actually guarantees

It's worth being precise here, because the docs are precise and most blog posts aren't.

  • DynamoDB gives you mutual exclusion on the lock item. A conditional write on LockID means two concurrent Terraform runs can't both think they hold the lock. That's it. That's the guarantee.
  • S3 gives you durability and (if you enable it) versioning. Versioning is the single most important setting on your state bucket and we'll come back to this.
  • Neither gives you transactional state updates. Terraform writes the new state file to S3 in one PutObject, but the "apply resources, then write state" sequence is not atomic. If Terraform dies between applying a resource and persisting that fact to S3, you have drift.

So the lock isn't a magic consistency boundary. It's just a mutex. If the process holding the mutex dies ungracefully, the mutex stays held, and whatever it was doing is now in an unknown state.

Recovering from a stuck lock (carefully)

The tempting first move is terraform force-unlock <LOCK_ID>. Don't do it yet. The lock being stuck is a symptom. The real question is: what state is the infrastructure in, and what state does S3 think it's in?

Our recovery runbook now looks like this:

1. Confirm no process actually holds the lock

Before anything else, check CI, check every engineer's terminal, check any automation. We've seen "stuck locks" that were actually a long-running terraform plan in a GitHub Actions job nobody remembered scheduling. If you force-unlock a live run, you will corrupt state.

aws dynamodb get-item \
  --table-name terraform-locks \
  --key '{"LockID":{"S":"acme-tfstate-prod/networking/terraform.tfstate"}}'

Look at the Created timestamp and the Who field. If it's 40 minutes old and the person named is on vacation, it's probably stuck.

2. Pull the current state and the previous version

Because we had S3 versioning on, we could do this:

aws s3api list-object-versions \
  --bucket acme-tfstate-prod \
  --prefix networking/terraform.tfstate

We downloaded both the current (post-crash) state and the version from immediately before the apply started. Then we diffed them. That diff told us exactly which resources Terraform thought it had modified before dying.

3. Reconcile with reality

With the diff in hand, we went to the AWS console and checked the actual state of each resource. In our case:

  • The old route table association was gone (deleted by the apply).
  • The new one had not been created (the apply died before that step).
  • The state file reflected the deletion but not the creation.

So real AWS and state S3 actually agreed on the route table association. The problem wasn't drift — it was that the apply hadn't finished its job. We needed to re-run, not roll back.

4. Force-unlock and re-plan

Only now:

terraform force-unlock 9f4b...
terraform plan -out=tfplan

We reviewed the plan line by line before applying. It wanted to create the missing route table association. Good. We applied it from a server in the same region this time, inside tmux, over a stable connection.

The backend hygiene we now enforce

After the incident, we tightened five things. None are clever. All would have made that Friday a non-event.

S3 versioning is mandatory

If you have one setting on your state bucket, make it this. Without versioning, a corrupted state write is permanent. With it, you can always roll back to the last known-good state as an object version.

resource "aws_s3_bucket_versioning" "tfstate" {
  bucket = aws_s3_bucket.tfstate.id
  versioning_configuration {
    status = "Enabled"
  }
}

Also enable MFA delete if your threat model justifies the operational pain. Ours doesn't, but we do have a bucket policy that denies DeleteObject outside a break-glass role.

No applies from laptops

This was the biggest cultural change. All applies now run from a CI runner — GitHub Actions for most workspaces, a self-hosted runner in a private subnet for anything that touches the networking stack. Laptops can plan, can read state, cannot apply. The runner has a stable network path to AWS and doesn't go to sleep.

If you want a managed version of this, Terraform Cloud and Spacelift both do it well. We stuck with GitHub Actions plus a careful IAM boundary because we already had the plumbing.

Lock timeouts and retries, configured explicitly

Terraform accepts -lock-timeout=10m on plan and apply. We default to 10 minutes in CI. That way a legitimate concurrent run waits politely instead of failing immediately, and a stuck lock surfaces quickly instead of blocking a queue silently.

Split the blast radius

Our networking stack used to be one workspace. It's now three: VPC + subnets, route tables + NAT, security groups. Each has its own state file and its own lock. A stuck lock on security groups doesn't block a NAT change. This is the oldest advice in Terraform and we still ignored it until it hurt.

If you're wrestling with how to carve up state, our platform engineering notes have a longer write-up on workspace boundaries.

A tested recovery runbook

We wrote the recovery steps above into a runbook, then deliberately broke a staging state file and asked a different engineer to recover it using only the runbook. They found two steps that were wrong. We fixed them. A runbook you've never executed is fiction.

What we'd do on day one of a new AWS account

If we were setting up Terraform state for a new project tomorrow, in order:

  1. Create the S3 bucket with versioning on, public access blocked, and SSE-KMS encryption.
  2. Create the DynamoDB lock table with LockID as the hash key and on-demand billing — it's cheaper than provisioned for this workload and removes a capacity-planning headache.
  3. Put a bucket policy in place that denies DeleteObject and DeleteObjectVersion except to a break-glass role.
  4. Wire terraform apply to a CI runner with OIDC-based AWS credentials. Revoke iam:* and *:Delete* from any human IAM user that could run apply locally.
  5. Split workspaces by blast radius before the first apply, not after the first incident.

State locking works. It just doesn't protect you from the things you probably think it does. Treat the lock as a mutex, treat the state file as a database that needs backups and migration discipline, and keep the humans away from the apply button.

#Terraform#AWS#IaC#DevOps#Reliability

Want a team like ours?

72Technologies builds production software for the kind of teams who actually read this blog.

Start a project