Why We Moved Our Terraform Backend from S3 to Terraform Cloud (and Half-Regretted It)

Four years ago we set up the standard Terraform backend: an S3 bucket for state, a DynamoDB table for locks, KMS for encryption, and a small mountain of IAM policy to keep it honest. It worked. Then we migrated 40+ workspaces across three AWS accounts and two GCP projects to HCP Terraform (formerly Terraform Cloud). Six months in, we have opinions.

This isn't a pitch for either side. It's the tradeoff sheet we wish we'd had before we started.

Why we moved at all

The S3 backend wasn't broken. It was tedious.

Our pain points, in rough order:

• Credential sprawl in CI. Every GitHub Actions runner needed AWS creds scoped tightly enough to touch state but not blow up production. We used OIDC, but each new workspace meant another trust policy edit.
• No native plan visibility. Plans lived in CI logs. Reviewers had to hunt through Actions output to find the actual +/-/~ lines. We built a bot to post plans as PR comments; it was fine, not great.
• Drift detection was homemade. A nightly job ran terraform plan across workspaces and Slacked us when the exit code was 2. It missed things when workspaces failed to init.
• Locking edge cases. DynamoDB locks are reliable until they aren't. We had two incidents in four years where a killed runner left a lock behind and someone had to force-unlock at 11pm.

HCP Terraform promised to absorb most of this. The pricing was the sticking point — the Standard tier is billed per resource under management (RUM), and we had a lot of small resources (IAM policies, DNS records, Cloudflare rules) inflating the count.

The math we did up front

Rough numbers, not a benchmark:

• ~4,200 resources under management across all workspaces
• ~15 engineers touching Terraform weekly
• ~120 plans/week, ~40 applies/week

At the time, the Standard tier came out to a mid-four-figure monthly bill. Our S3 + DynamoDB + KMS costs were under $30/month. The delta was real, but so was the ~6 hours/week our platform team spent on IaC plumbing.

We decided the platform time was worth more than the license. That math might not hold for you.

The migration itself

Migrating state is boring when you do it right and terrifying when you don't. We did it in waves — non-prod workspaces first, then prod, one team at a time.

The actual state move is a two-line backend swap:

# Before
terraform {
  backend "s3" {
    bucket         = "acme-tfstate-prod"
    key            = "platform/networking.tfstate"
    region         = "us-east-1"
    dynamodb_table = "acme-tflocks"
    encrypt        = true
  }
}

# After
terraform {
  cloud {
    organization = "acme"
    workspaces { name = "platform-networking-prod" }
  }
}

Then terraform init -migrate-state, confirm the prompt, and you're done. In practice, we scripted it and ran it against every workspace in a batch, because doing it by hand invites typos.

The things the docs undersell

A few surprises worth knowing:

1. Variable migration is manual. Anything you had in .tfvars or CI environment variables has to be re-created as workspace variables or variable sets. We wrote a small script that parsed our existing GitHub Actions secrets and pushed them via the TFE provider. Save yourself an afternoon and do this before the state migration, not after.
2. Provider auth changes shape. We had been passing AWS creds through env vars in CI. In HCP Terraform, we moved to dynamic provider credentials (OIDC from HCP into AWS). This is genuinely nicer, but it means a fresh trust policy per workspace unless you consolidate.
3. Sentinel/OPA policies are opt-in but sneaky. We enabled a couple of "no public S3 buckets" policies and immediately had two workspaces fail apply because of legacy exceptions. Plan for a grace period.

What actually got better

Six months later, here's what we'd fight to keep.

PR-driven plans without the glue code

The VCS integration posts plan output directly onto PRs, gated by workspace. Reviewers see the resource diff without leaving GitHub. We deleted about 400 lines of custom GitHub Actions and a small Node.js bot that had been posting plan summaries. That alone is worth something.

Drift detection that works

HCP Terraform's health assessments run plans on a schedule and flag drift explicitly. Ours catches roughly one drift event per week — usually someone touching a Route53 record in the console during an incident. Before, we'd find out weeks later when the next apply tried to revert it.

No more lock incidents

Zero force-unlocks in six months. Not a huge sample size, but the failure mode is genuinely different — the run queue handles concurrency at the workspace level, and killed runs release cleanly.

Audit trail we didn't have to build

Every plan and apply is stored with the initiating user, the VCS commit, the plan output, and the apply log. For SOC 2 evidence collection, this went from a scavenger hunt to a URL.

What got worse

Not everything is upside.

The bill is real

RUM billing punishes fine-grained resources. A workspace managing 400 IAM policies costs the same as one managing 400 EC2 instances. We ended up refactoring a few modules to use aws_iam_policy_document and inline policies instead of separate aws_iam_policy resources, purely to reduce RUM count. That's a weird incentive.

If your IaC estate is heavy on IAM, DNS, or config-style resources, model the RUM cost carefully before you commit. It's the single biggest surprise for teams we've talked to.

Local dev is more awkward

Running terraform plan locally now goes through the remote backend by default, which means every local plan queues on HCP. For quick iteration on a module, that's slower than the old S3 flow. You can set TF_CLOUD_ORGANIZATION and use terraform plan -no-color with local execution mode, but it's not the default and new engineers trip on it.

Provider version pinning matters more

Remote runs use whatever Terraform version the workspace is pinned to. We had a handful of workspaces pinned to 1.5.x that we'd been meaning to upgrade. The migration forced us to actually do it, which was fine, but it added a week to the timeline we hadn't planned for.

You are now dependent on someone else's uptime

HCP Terraform has had incidents. When it's degraded, you can't apply. We hit this twice — once for about 40 minutes, once for a couple of hours. Neither was during an outage on our side, but it's worth knowing that your ability to ship infra changes now depends on a vendor's status page.

The hybrid pattern we landed on

We don't run everything in HCP. A few workspaces stayed on S3:

• Bootstrap workspaces. The Terraform that creates the AWS accounts, KMS keys, and OIDC providers that HCP itself uses. Circular dependencies are not fun.
• Break-glass workspaces. One workspace per environment that can rebuild critical networking if HCP is down. Rarely touched, but we want the option.
• Experimental modules. Engineers spike on new modules in a local S3-backed workspace before graduating them.

This felt like overkill when we designed it. Then HCP had its first outage and we were glad.

When we'd actually recommend the move

Straight talk: HCP Terraform is worth it if you meet at least two of these:

• You have 10+ engineers regularly running Terraform
• You spend real time maintaining CI glue around plans, locks, or drift
• You need audit evidence for compliance
• Your RUM count is dominated by "real" resources, not IAM/DNS noise

If you're a team of three with five workspaces, the S3 backend is still the right answer. The plumbing tax is low at that scale, and the license cost isn't.

Where we'd start

If you're weighing the move today, do these three things before you decide:

1. Run terraform state list | wc -l across every workspace and total it. That's your RUM ceiling. Multiply by the current per-resource price and see if the number is survivable.
2. Timebox one week to migrate two non-prod workspaces end-to-end, including VCS integration and dynamic credentials. You'll learn more from that than from any comparison blog post — including this one.
3. Decide up front which workspaces stay on S3 as your escape hatch. Don't skip this. Future-you, at 2am during a vendor incident, will be grateful.

If you want a hand modelling the migration or designing the hybrid split, that's the kind of work our platform engineering team does regularly — happy to compare notes.