Our AWS NAT Gateway Bill Tripled Overnight: Tracing a Rogue S3 Egress Path

A Monday cost alert told us NAT Gateway data processing had jumped from around $180/day to just under $600/day, and had been climbing quietly for six days before the threshold tripped. Nothing in our deploy log looked dramatic. The culprit turned out to be a single Terraform refactor that removed an S3 gateway endpoint nobody remembered was load-bearing.

This is the write-up we wish we'd had before we started digging.

The shape of the bill

Our setup is unremarkable: a handful of ECS Fargate services in private subnets across three AZs, one NAT Gateway per AZ, and a mix of internal APIs that read and write to S3 for user uploads, cached model artefacts, and log shipping. Traffic patterns are steady on weekdays with a modest evening bump.

The cost breakdown before and after, pulled from Cost Explorer with the UsageType dimension:

• NatGateway-Bytes (data processed): went from ~1.1 TB/day to ~3.6 TB/day
• NatGateway-Hours: unchanged
• DataTransfer-Out-Bytes to internet: barely moved

That last point is what made it interesting. If we were suddenly serving more traffic to the public internet, egress-out would have climbed too. It hadn't. Something inside the VPC was pushing a lot of bytes through the NAT for a destination that wasn't really "the internet" in the billing sense.

S3 was the obvious suspect, but suspecting and proving are different jobs.

Proving it with VPC Flow Logs

We already ship Flow Logs to S3 in the parquet-friendly format and query them with Athena. If you don't have this set up, it's the single highest-leverage thing you can do for network cost debugging. The schema we use includes srcaddr, dstaddr, bytes, pkt-dstaddr, and flow-direction.

The query that broke the case open:

SELECT
  pkt_dstaddr,
  SUM(bytes) / 1024 / 1024 / 1024 AS gb
FROM vpc_flow_logs
WHERE day BETWEEN '2026/01/12' AND '2026/01/18'
  AND flow_direction = 'egress'
  AND srcaddr LIKE '10.40.%'   -- private subnet CIDR
GROUP BY pkt_dstaddr
ORDER BY gb DESC
LIMIT 25;

The top rows were all AWS-owned IP ranges in eu-west-1, and a reverse lookup against the published ip-ranges.json confirmed the service: S3. Roughly 2.4 TB/day of S3 traffic that used to travel over the gateway endpoint was now hairpinning through the NAT Gateway.

At the current published NAT data processing rate in our region, that's the difference between essentially free (gateway endpoints for S3 and DynamoDB don't charge data processing) and a five-figure monthly line item.

The Terraform diff that did it

The change was three weeks old. Someone had cleaned up what looked like an unused module and removed this:

resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = aws_route_table.private[*].id
}

The PR description said "remove orphaned endpoint, no references found". Technically true. Gateway endpoints don't get "referenced" by resources; they inject prefix-list routes into route tables. When the endpoint went away, so did the routes, and every S3 request from a private subnet quietly fell back to the default route: 0.0.0.0/0 via the NAT.

Nothing broke. Nothing alerted. The application logs looked identical because from the app's perspective, they were identical.

Why our guardrails missed it

We had a few things in place that should have caught this earlier. They didn't, and it's worth being honest about why.

Budget alerts fired late. We had a monthly budget alert at 80% of forecast. Because the spike started early in the billing cycle, the forecast absorbed a lot of the increase before the threshold tripped. Daily anomaly detection would have caught it in 48 hours; we didn't have it enabled on this account.

Terraform plan output was noisy. The PR touched networking modules across three environments. The -aws_vpc_endpoint.s3 line was there, on line 340-something of a 900-line plan. Nobody flagged it in review because nobody knew what depended on it.

We had no test for the route table. Our infra tests checked that subnets existed, that the NAT existed, that security groups had the right rules. Nothing asserted that the S3 prefix list was present in the private route tables.

The fix, and the guardrail

The immediate fix was a one-line revert. Within about ten minutes of the endpoint being re-applied, NAT bytes-processed dropped back to baseline. CloudWatch's BytesOutToDestination metric on the NAT is the fastest signal here — it updates in near real-time and you don't have to wait for Cost Explorer to catch up.

The more interesting question was what to add so this doesn't happen again. We ended up with three things.

1. A conftest policy for the S3 endpoint

We run conftest against terraform show -json output in CI. The rule is deliberately blunt:

package main

deny[msg] {
  input.resource.aws_vpc.main
  not input.resource.aws_vpc_endpoint.s3
  msg := "VPC must have an S3 gateway endpoint attached to private route tables"
}

deny[msg] {
  input.resource.aws_vpc.main
  not input.resource.aws_vpc_endpoint.dynamodb
  msg := "VPC must have a DynamoDB gateway endpoint"
}

Gateway endpoints for S3 and DynamoDB are free. There is no scenario in our architecture where removing them is the right call. Encoding that as a policy is cheaper than remembering it.

2. A CloudWatch alarm on NAT bytes-processed

One alarm per NAT, on the BytesOutToDestination metric, with a threshold set to roughly 1.5x our rolling weekly p95. Not clever, but it would have paged us on day one instead of day six.

3. A weekly Athena report on top NAT destinations

A scheduled Athena query, results posted to a Slack channel, showing the top 20 destination IPs by NAT-egressed bytes, annotated with the AWS service each IP belongs to. This makes anomalies visually obvious the moment they start.

What we'd tell someone starting from scratch

A few opinions we now hold more strongly than we did a month ago.

Gateway endpoints for S3 and DynamoDB should be baseline, not optional. They cost nothing, they reduce blast radius during an internet outage, and they remove a whole category of surprise bills. Put them in your VPC module and make removing them require a policy exception.

Interface endpoints are a different conversation. They have per-hour and per-GB charges, and for low-volume services (Secrets Manager fetched once at boot, for example) the NAT path can genuinely be cheaper. Do the maths per service; don't reflexively endpoint everything.

Treat NAT bytes-processed as a first-class SLI. It's a proxy for "how much of our internal traffic is accidentally leaving the VPC". Sudden changes are almost always a bug, not a feature.

Flow Logs pay for themselves the first time you need them. The storage cost is real but modest; the debugging leverage is enormous. If you're not shipping them somewhere queryable, that's the first thing to fix.

Where we'd start

If you inherited an AWS account tomorrow and had one afternoon, we'd do this in order: enable Flow Logs to S3 with Athena on top, add a CloudWatch alarm on NAT BytesOutToDestination per gateway, confirm S3 and DynamoDB gateway endpoints exist and are attached to every private route table, and add a Cost Explorer daily anomaly monitor on the NatGateway-Bytes usage type. That's maybe three hours of work and it would have saved us the entire incident.

If you want a second pair of eyes on your AWS networking or your Terraform guardrails, that's the kind of thing we do — see our services or browse more incident write-ups on the blog.