Our AWS NAT Gateway Bill Hit $4K/Month Before We Noticed

Nobody notices a NAT Gateway until the finance team does. Ours sat quietly at the bottom of the AWS bill for about eight months, growing at roughly $300/month, until it crossed four figures and someone finally asked what it was. By the time we looked, the single line item was the second-largest cost in our production account, behind RDS and ahead of every EC2 instance combined.

This is the walkthrough of how we traced it, what was actually sending traffic, and the three changes that stuck.

The line item nobody owns

NAT Gateway pricing has two components that bite: an hourly charge (around $0.045/hr in us-east-1, so roughly $33/month per gateway just to exist) and a data processing charge of $0.045 per GB. The hourly charge is boring and predictable. The data processing charge is where things get interesting, because it applies to every byte that passes through — in or out, regardless of destination.

Our setup had three NAT Gateways (one per AZ, as the docs recommend for HA) across two VPCs. The hourly cost was around $200/month total. The remaining ~$3,800 was data processing. That means something in our private subnets was pushing about 85 TB/month through NAT.

For context, our application traffic to end users went through an ALB and CloudFront — none of that touches NAT. NAT traffic is strictly outbound-initiated from private subnets: pulling container images, hitting third-party APIs, writing to S3 from the wrong endpoint, and so on.

Why it stayed invisible

Three reasons we missed it:

1. Cost Explorer defaults to grouping by service. NAT Gateway rolls up under "EC2 - Other," which is a bucket of miscellaneous network and storage charges. Nobody scrolls into "EC2 - Other" during a casual review.
2. No per-team tagging on network infrastructure. NAT Gateways are shared. There's no natural owner to flag the growth.
3. Linear growth is the hardest to spot. A spike triggers an alert. A 10% month-over-month climb just looks like business as usual.

Finding what's actually using it

The first instinct is to enable VPC Flow Logs and start grepping. That works, but it's slow and the logs themselves cost money. We did two things in parallel.

VPC Flow Logs to S3, queried with Athena

We already had Flow Logs on for security auditing, dumped to S3 in Parquet. Athena made the aggregation cheap:

SELECT
  srcaddr,
  dstaddr,
  SUM(bytes) / 1024 / 1024 / 1024 AS gb_total
FROM vpc_flow_logs
WHERE day BETWEEN '2026/01/01' AND '2026/01/07'
  AND action = 'ACCEPT'
  AND srcaddr LIKE '10.20.%'
  AND dstaddr NOT LIKE '10.%'
GROUP BY srcaddr, dstaddr
ORDER BY gb_total DESC
LIMIT 50;

The top rows told the story within minutes. Two IPs stood out immediately, both ECS task IPs that were short-lived but reappeared constantly because the service was autoscaling. The destinations were S3 public IPs and a specific third-party analytics endpoint.

CloudWatch metrics on the gateways themselves

BytesOutToDestination and BytesOutToSource on each NAT Gateway, broken down by AZ, showed one AZ carrying about 60% of the total. That was a hint: it correlated with where our largest ECS service's tasks tended to land because of a stale placement constraint.

What was actually in the traffic

Three culprits, in rough order of cost:

1. S3 traffic going through NAT instead of a Gateway Endpoint. This was the single biggest one, around 45 TB/month. We had a worker service that pulled large Parquet files from S3, processed them, and wrote results back. The VPC had no S3 Gateway Endpoint, so every byte went: ECS task → NAT Gateway → S3 public endpoint → back. We paid $0.045/GB on the way out and on the way in. S3 Gateway Endpoints are free. This was a pure unforced error.

2. Container image pulls on every task start. Our ECS tasks pulled images from ECR on cold start. With autoscaling events several times per hour and image sizes around 400 MB (bloated Node images with every build tool left in), this added up fast. Rough math: 300 cold starts/day × 400 MB × 30 days ≈ 3.5 TB/month, pulled through NAT because ECR had no interface endpoint in the VPC.

3. A chatty third-party analytics SDK. One of our backend services fired per-event payloads to a third-party analytics API. Each event was small (~2 KB), but at a few hundred events per second across the fleet, it summed to about 8 TB/month. This one we couldn't eliminate — it's an external service — but we could batch it.

The fixes, in the order we shipped them

S3 and DynamoDB Gateway Endpoints

Ten-minute change. Zero ongoing cost. In Terraform:

resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = aws_route_table.private[*].id

  tags = {
    Name = "s3-gateway-endpoint"
  }
}

The route table association is the whole trick. Once that lands, anything in the private subnets routing to S3's prefix list goes via the endpoint for free. We saw NAT traffic drop by about 50% within an hour of deploy.

ECR Interface Endpoints

Interface endpoints are not free — they cost roughly $0.01/hr per AZ per endpoint plus $0.01/GB data processing. For ECR you need three: ecr.api, ecr.dkr, and the S3 gateway endpoint (ECR stores layers in S3). Across three AZs that's about $65/month in endpoint hourly charges.

Worth it? For us, yes. We were pushing roughly $160/month on ECR pulls through NAT. Net savings: around $90/month, plus faster cold starts because the endpoint path has lower latency than going out to the public ECR endpoint.

If you pull less than ~1.5 TB/month from ECR, the math flips and the interface endpoint costs more than it saves. Run the numbers for your own traffic before you cargo-cult this.

Slimmer container images

Separate problem, but worth mentioning because it compounded everything. We moved from a single-stage Node Dockerfile to multi-stage with node:20-alpine as the runtime base. Image size went from ~400 MB to ~90 MB. Even with the ECR endpoint in place, this saved cold start time and reduced the interface endpoint data processing charge.

Batching the analytics SDK

This one lived with the application team. They moved from per-event HTTP POSTs to a 5-second batch window with gzip compression. Payload volume dropped by roughly 70%. We considered a VPC-level egress proxy with compression for all third-party traffic, but decided the maintenance burden wasn't worth it for one caller.

What the bill looks like now

After the changes settled, NAT Gateway data processing dropped from ~85 TB to ~12 TB/month. Combined with the ECR interface endpoint cost, the new total NAT-related spend is around $750/month versus the original $4,000. We did not touch the HA topology — still three gateways, one per AZ — because the hourly cost is the cheap part.

What we'd watch for next time

• A CloudWatch alarm on BytesOutToDestination per NAT Gateway with a month-over-month threshold, not a spike threshold.
• A monthly automated report that breaks "EC2 - Other" into its sub-components and posts to a Slack channel.
• A pre-prod checklist line item: does this VPC have S3 and DynamoDB Gateway Endpoints? It's free. There's no reason not to.

Where we'd start

If you've never looked at your NAT bill specifically, do this today: open Cost Explorer, filter to UsageType containing NatGateway-Bytes, group by linked account, and look at the last six months. If the number surprises you, enable Flow Logs for a week and run the Athena query above. Nine times out of ten it's S3 traffic missing a Gateway Endpoint, and that's a one-afternoon fix.

If you want help auditing cloud spend or rebuilding VPC topology without downtime, that's the kind of work we do under DevOps & Cloud services. The NAT Gateway is rarely the whole story — it's usually the first thread you pull.