Pulumi's Automation API Rewrote Our Preview Environments: A 6-Month Report

Six months ago we tore out a Terraform-plus-bash rig that provisioned per-PR preview environments and replaced it with a small Go service built on Pulumi's Automation API. The short version: it works, we ship faster, and we introduced two failure modes we didn't anticipate. This is the honest report.

Why we stopped using Terraform for previews

Our old setup was fine on paper. Every PR triggered a GitHub Actions job that ran terraform workspace new pr-<num>, applied a module, and tore it down on PR close. In practice, three things kept biting us:

• State lock contention. Two PRs merging within seconds would fight over the same backend. We'd already moved to per-workspace state files, but the workspace metadata itself was a shared object.
• Shell glue. Around the terraform apply there were roughly 400 lines of bash doing DNS registration, seeding a Postgres schema, warming a CDN, and posting a preview URL to the PR. Every one of those steps had its own retry logic. Or didn't.
• No programmatic introspection. We wanted to answer "how many previews are alive right now, what do they cost, and which are older than 48 hours?" from a dashboard. Parsing terraform state list in CI is not a dashboard.

We considered Terraform Cloud's run tasks and Atlantis. Both are legitimate answers. We picked Pulumi's Automation API because we wanted the orchestrator to be a real program, not a YAML pipeline calling a CLI.

What the Automation API actually is

If you've only used Pulumi through its CLI, the Automation API is the same engine exposed as a library. You can create stacks, set config, run up, preview, destroy, and stream events — all from Go, TypeScript, Python, or .NET. No subprocess calls to pulumi. No parsing stdout.

Here's the shape of what our controller does when a PR opens:

stackName := fmt.Sprintf("preview-pr-%d", pr.Number)
s, err := auto.UpsertStackInlineSource(ctx, stackName, "previews", definePreview(pr))
if err != nil {
    return err
}

_ = s.SetConfig(ctx, "aws:region", auto.ConfigValue{Value: "eu-west-1"})
_ = s.SetConfig(ctx, "preview:branch", auto.ConfigValue{Value: pr.Branch})
_ = s.SetConfig(ctx, "preview:sha",    auto.ConfigValue{Value: pr.SHA})

res, err := s.Up(ctx, optup.EventStreams(eventCh), optup.Parallel(8))
if err != nil {
    return fmt.Errorf("up failed: %w", err)
}

previewURL := res.Outputs["url"].Value.(string)
return github.PostComment(pr, previewURL)

That's the whole hot path. No shell. The definePreview function returns a pulumi.RunFunc that defines the ECS service, RDS branch, Route53 record, and CloudFront behavior. The controller service is deployed once and listens for GitHub webhooks.

Why inline sources beat project directories

We started with UpsertStackLocalSource, pointing at a preview/ directory of Pulumi code. It worked, but every controller pod needed the code checked out, Node/Go toolchains installed, and versions kept in sync. Switching to UpsertStackInlineSource — where the stack's program is a function inside the controller binary — collapsed the deploy story. One binary, one image, one version.

The tradeoff: your infra code and your controller code now ship together. If you want product teams to edit preview definitions independently, inline sources are worse. For us, the previews team is the platform team, so it was fine.

Real numbers, honestly framed

A few measurements from our environment, which is a moderately busy monorepo pushing 40 – 90 PRs per week. Take these as our shape, not a benchmark.

• Cold preview creation (new PR, empty cache): 3 – 5 minutes, dominated by ECS task pull and RDS logical database creation. Terraform was 4 – 7 minutes for the same workload.
• Warm update (push to existing PR): 25 – 60 seconds. This is the real win. Pulumi's diff engine plus a long-lived controller (no cold-start CLI) is genuinely faster.
• Destroy on PR close: 40 – 90 seconds.
• Controller cost: one t4g.small on ECS, plus the Pulumi Cloud team tier. Under $80/month combined.
• Preview cost: we cap at ~$0.12/hour per active preview by sharing an RDS cluster with per-PR logical databases and using Fargate Spot. A preview that lives 8 hours costs under a dollar.

What we didn't measure well until month three: how long dead previews were lingering. More on that below.

The two failure modes we didn't see coming

1. The controller became a stateful service we forgot to treat as one

The Automation API doesn't store your stack state — Pulumi Cloud (or your chosen backend) does. But the controller does hold in-memory context: active Stack handles, event channels, in-flight Up operations. When we rolled the controller during a deploy, any preview mid-provision would end up with orphan resources.

The fix was boring and correct: treat the controller like any other stateful worker. Drain on SIGTERM, persist a small "in-flight operations" table in DynamoDB, and reconcile on startup. If a PR's last known state was provisioning and no controller is currently working it, requeue it.

We should have designed this on day one. We didn't, because "it's just IaC" felt lighter than it was.

2. GitHub webhooks lie about PR closure

About 4% of PR-close webhooks never arrived, or arrived after we'd already garbage-collected them. Previews would live for days. Our AWS bill for one week in month two was 30% over baseline before anyone noticed.

We now run a reconciler every 15 minutes that:

1. Lists all stacks in the previews project via the Automation API.
2. Cross-references with open PRs from the GitHub API.
3. Destroys any stack whose PR is closed or whose last update is older than 72 hours.

stacks, _ := workspace.ListStacks(ctx)
for _, st := range stacks {
    prNum := parsePRNumber(st.Name)
    if !github.IsOpen(prNum) || olderThan(st.LastUpdate, 72*time.Hour) {
        s, _ := auto.SelectStackInlineSource(ctx, st.Name, "previews", definePreview(prNum))
        _, _ = s.Destroy(ctx)
        _ = s.Workspace().RemoveStack(ctx, st.Name)
    }
}

This one loop paid for the entire migration in the first month it ran.

Where Pulumi genuinely beats Terraform for this use case

• You get a real programming language. Loops over PR labels, conditional resources based on which services the PR touched, dynamic tagging — all trivial. In Terraform we had a 200-line locals block computing the same things with merge() and for expressions.
• Streaming events. optup.EventStreams gives you a channel of typed events. We pipe them to a per-PR log in S3, so a developer can see exactly what happened when their preview failed. Terraform's JSON output is fine, but you're always parsing it out-of-band.
• Programmatic destroy is safe. SelectStack + Destroy is a supported, first-class flow. In Terraform we were shelling out and hoping the working directory was right.

Where Terraform is still better

Be honest: Pulumi isn't the right answer for everyone.

• Provider maturity for edge cases. We hit two bugs in the AWS provider where a resource property that Terraform's AWS provider handled cleanly required a workaround in Pulumi. Both were fixed within a release cycle, but it happened.
• Hiring. More engineers know Terraform. If your infra team is small and rotating, HCL's constraints are a feature.
• Read-only auditability. A Terraform plan in a PR comment is easier for a security reviewer to skim than a Pulumi preview against inline Go code.

For long-lived production infrastructure, we still use Terraform. For the preview system — a program that manages other programs — Pulumi's Automation API is the right shape.

Things we'd tell past us

• Version-pin the Pulumi SDK aggressively. Minor version bumps changed event stream shapes twice. Pin, upgrade deliberately, test in a staging controller.
• Don't share stacks across services. We tried a single "preview" stack with all services as resources. Diffs got slow and blast radius scared people. One stack per PR per service is cleaner.
• Budget for Pulumi Cloud or self-host early. The free tier is generous but you will outgrow it once you have ephemeral stacks churning constantly. Decide before you're surprised by a quota.
• Instrument the controller with OpenTelemetry from day one. We added it in month four. Should have been day one. Every Up, Preview, and Destroy is a span; every stack is a trace attribute. Debugging without it was guessing.

Where we'd start

If you're rebuilding preview environments in 2026, start with a single-binary controller in whatever language your platform team already writes. Use UpsertStackInlineSource. Build the reconciler before you build the happy path — dead previews will eat your budget faster than any bug. And keep the scope small: one project, per-PR stacks, aggressive TTLs. You can always add more.

If you want to see how we approach platform work end-to-end, our DevOps and cloud services page has the longer version, and there's more incident-driven writing on the blog.