The Terraform State Lock That Held Our Deploys Hostage for 6 Hours
A stuck DynamoDB lock froze every pipeline in the org. Here's what actually happened, why our runbook made it worse, and the guardrails we put in afterward.
August 18, 2026 7 min
← All articles
Category · DevOps & Cloud
DevOps and cloud engineering guides
Vercel, AWS, GCP, observability, IaC, and reliability engineering.
The S3 Egress Bill That Doubled Overnight: Tracing a Rogue Prefetch
A quiet Tuesday, a doubled AWS bill, and a client-side prefetch nobody remembered shipping. Here's how we traced 3.2 TB of surprise S3 egress back to a single React hook.
August 12, 2026 7 min
GCP Cloud Run vs AWS Fargate for Bursty APIs: What We Learned Running Both
We ran the same Node API on Cloud Run and Fargate for six months. Cold starts, egress costs, autoscaling behaviour, and the operational papercuts nobody warns you about.
August 10, 2026 6 min
The DynamoDB Hot Partition That Only Showed Up on Black Friday
A DynamoDB table that behaved perfectly in load tests fell over during a Black Friday spike. Here's the partition key mistake we made, how we found it, and the redesign that stopped it happening again.
August 7, 2026 6 min
Sentry Release Health Lied to Us: Debugging a Silent Crash Rate
Sentry said our release was healthy at 99.6% crash-free sessions. Users disagreed. Here's what we found when we stopped trusting the dashboard and started reading the SDK config.
August 4, 2026 6 min
Pulumi vs Terraform in 2026: A Migration We Half-Regretted
We migrated a mid-size AWS + Vercel estate from Terraform to Pulumi. Six months in, here's what actually broke, what got better, and the parts we'd do differently.
August 2, 2026 6 min
OpenTelemetry Sampling in Production: The Config That Saved Our Trace Bill
Head sampling threw away the traces we needed. Tail sampling blew up our collector memory. Here's the sampling config we landed on after six months in production.
July 30, 2026 6 min
CloudFront to Vercel: The Cache Header Mismatch That Cost Us a Weekend
We fronted a Vercel app with CloudFront to satisfy a compliance requirement. Two weeks later, stale checkouts and missing Set-Cookie headers taught us how differently these two CDNs think about caching.
July 27, 2026 6 min
Vercel Edge Middleware Latency: What We Measured When We Moved Auth to the Edge
We moved auth checks from a Node API route to Vercel Edge Middleware expecting free speed. Some routes got faster, some got slower, and the bill moved in ways we didn't predict.
July 25, 2026 6 min
Terraform State Corruption at 3 AM: How We Recovered and What We Changed
A stale lock, a partial apply, and a state file that no longer matched reality. Here's the recovery playbook we now hand every new engineer — and the guardrails that make sure we never do it live again.
July 22, 2026 6 min
AWS NAT Gateway Bills Ate Our Margins. Here's How We Cut Them 78%.
A single misconfigured VPC route turned our NAT Gateway into a five-figure monthly line item. Here's the audit trail, the fixes, and what we'd do differently.
July 19, 2026 6 min
GCP Cloud Run vs AWS Lambda for Bursty APIs: What Broke, What Held
We ran the same bursty checkout API on Cloud Run and Lambda for six months. Cold starts, concurrency, and billing quirks all bit us in ways the marketing pages don't mention.
July 14, 2026 6 min
Vercel Preview Deployments Are Leaking Secrets. Audit Yours Now.
Preview URLs are treated like staging by developers and like production by attackers. Here's how we found real secrets exposed across three client accounts, and the guardrails we now enforce by default.
July 11, 2026 6 min
Sentry Performance Costs Doubled Overnight. Here's What We Found.
Our Sentry bill jumped from ~$900 to ~$2,100 in a single billing cycle with no traffic change. Here's the investigation, the culprits we found, and the sampling strategy we settled on.
July 9, 2026 6 min
Pulumi vs Terraform in 2026: A Migration Story We Almost Regretted
We migrated a mid-sized AWS + Vercel estate from Terraform to Pulumi, hit real walls, and rolled part of it back. Here's what actually happened and when Pulumi is worth it.
July 6, 2026 7 min
OpenTelemetry Sampling: Why Head-Based Cost Us Real Incidents
We ran head-based sampling in OpenTelemetry for a year and it burned us during two real incidents. Here's what tail sampling actually costs, what it saved, and how we'd configure it from scratch.
July 3, 2026 6 min
CloudWatch Logs to Grafana Loki: A Migration That Actually Paid Off
We moved a mid-size AWS workload off CloudWatch Logs and onto self-hosted Loki. Here's what broke, what the bill actually looked like, and when we'd tell you not to do it.
July 1, 2026 6 min
Vercel Edge Middleware Cold Starts Wrecked Our p95. Here's the Fix.
Edge middleware promised sub-50ms execution. Our p95 said otherwise. Here's what we found when we instrumented it properly, and the three changes that brought latency back under control.
June 25, 2026 6 min
Terraform State Locking Failed Mid-Apply. Here's What We Learned.
A DynamoDB throttle event left our Terraform state half-written and locked. Here's the postmortem, the recovery steps, and the guardrails we added so it doesn't happen again.
June 23, 2026 6 min
Our AWS NAT Gateway Bill Hit $4k/Month. Here's How We Cut It by 80%.
A single NAT Gateway line item quietly ate our cloud budget. Here's the traffic audit, the VPC endpoint rollout, and the gotchas nobody mentions in the AWS docs.
June 20, 2026 7 min
GCP Cloud Run vs AWS Lambda for a Real Next.js Backend: What We Picked and Why
We ran the same Next.js API workload on Cloud Run and Lambda for three months. Cold starts, cost, observability, and one nasty timeout bug shaped the decision.
June 17, 2026 6 min
Our Vercel Cron Jobs Silently Stopped Firing for 6 Hours. Here's the Postmortem.
A scheduled job that hadn't fired in six hours, no alert, no error in Sentry, and a billing email that didn't get sent. Here's exactly what broke, how we caught it, and the cron monitoring pattern we run now.
June 15, 2026 6 min
Sentry Performance Quotas Blew Up Our Bill: What We Changed
A war story about Sentry transactions, span ingestion, and a 6x bill spike — plus the dynamic sampling, SDK config, and quota guardrails we now ship by default.
June 12, 2026 6 min
Pulumi vs Terraform in 2026: A Real Migration, Not a Bake-Off
We moved part of a production AWS estate from Terraform to Pulumi over six months. Here's what actually changed, what broke, and where we'd quietly stay on HCL.
June 9, 2026 6 min
OpenTelemetry Sampling at Scale: Why Tail-Based Bit Us First
We rolled out OpenTelemetry across a Node and Go fleet, picked tail-based sampling because everyone said to, and learned why head-based wins for most teams. Here's the tradeoff we wish someone had drawn for us.
June 7, 2026 6 min
We Moved Our Vercel ISR Cache to S3 + CloudFront. Here's the Math.
Vercel's bandwidth and function invocation costs got loud at scale. We moved the hot read path to S3 + CloudFront while keeping the DX. Here's the architecture, the numbers, and what broke.
June 4, 2026 6 min
The CloudFront-to-Vercel Edge Migration That Almost Broke Auth
We moved a Next.js app from CloudFront + Lambda@Edge to Vercel and learned the hard way that signed cookies, edge regions, and middleware ordering don't translate cleanly. Here's what bit us.
June 1, 2026 7 min
Migrating a Terraform Monorepo to Stacks: What We'd Do Differently
We moved a 40-module Terraform monorepo to HCP Terraform Stacks. Here's what broke, what we gained, and the four decisions we'd reverse if we started over.
May 30, 2026 7 min
How We Cut Our AWS Multi-AZ RDS Failover Time From 90s to 12s
A real story of trimming RDS failover from a customer-visible 90 seconds down to roughly 12. The fix wasn't a bigger instance — it was DNS, connection pools, and Proxy.
May 27, 2026 6 min
The Day Our GCP Cloud Run Cold Starts Took Down Checkout
A Cloud Run service that ran fine for eighteen months started timing out checkout on a Friday afternoon. The fix wasn't more CPU — it was a misread of how min-instances, concurrency, and startup CPU boost actually interact.
May 24, 2026 6 min
The AWS NAT Gateway Bill That Ate Our Margin
A quiet $40/day NAT Gateway line item turned into the second-largest cost on our AWS account. Here's how we found it, what was actually driving it, and the VPC endpoint plumbing that fixed it.
May 22, 2026 6 min
The Sentry Bill That Tripled Overnight: A Quota Postmortem
A single deploy turned a calm Sentry account into a $4k surprise. Here's what happened, what we changed, and how to stop event floods before finance notices.
May 19, 2026 6 min
Terraform vs Pulumi in 2026: A Migration We Half-Finished
We spent six months partially migrating a production AWS estate from Terraform to Pulumi. Here's what we kept, what we rolled back, and the boring reasons IaC choices rarely come down to language.
May 16, 2026 6 min
When OpenTelemetry Sampling Bit Us: Debugging a Phantom p99
A war story about how head-based sampling in OpenTelemetry quietly hid a real latency regression for three weeks, and the tail-sampling setup we landed on after the incident.
May 14, 2026 6 min
Cutting a Vercel Bill in Half: What Actually Moved the Needle
A client's Vercel invoice tripled in a quarter. We spent two weeks tearing apart their Next.js app and shaved roughly 55% off the bill. Here's what mattered, what didn't, and what we'd do differently.
May 12, 2026 6 min