AWS NAT Gateway Costs Ate Our Margins: How We Cut Them 80% Without Rewriting the VPC

Our monthly AWS bill had a quiet freeloader on it: NAT Gateway data processing charges that had grown roughly 6x in eighteen months without anyone noticing. The compute line items were watched like hawks. The networking line was treated like weather. This is the audit we ran, what we actually changed, and the parts that didn't go smoothly.

Why NAT Gateway bills sneak up on you

AWS charges for NAT Gateway in two ways: an hourly rate per gateway (modest, predictable) and a per-GB data processing fee on everything that flows through it. In most regions that per-GB number sits in the $0.045 range at time of writing — roughly the same as cross-AZ data transfer. The hourly cost is boring. The per-GB cost is where margins go to die.

The trap: every private subnet workload that talks to an AWS service over the public endpoint — S3, ECR, CloudWatch Logs, Secrets Manager, STS, DynamoDB, SQS — pays that per-GB fee twice if you're not careful. Once egressing through NAT, and sometimes again on the response path depending on how you count. Your ECS tasks pulling images, your Lambda functions writing logs, your nightly backups to S3: all of it routes through NAT by default when it originates in a private subnet.

We had assumed "it's AWS to AWS, it stays on the backbone, it's basically free." That assumption cost us.

The audit: finding where the bytes actually go

Before touching anything, we needed to know what was flowing through NAT. VPC Flow Logs are the honest answer, but raw flow logs are a swamp. We enabled them to an S3 bucket in Parquet format and queried with Athena.

The query that mattered looked roughly like this:

SELECT
  pkt_dst_aws_service,
  SUM(bytes) / 1024 / 1024 / 1024 AS gb_total,
  COUNT(*) AS flow_count
FROM vpc_flow_logs
WHERE year = '2026' AND month = '02'
  AND srcaddr LIKE '10.0.%'          -- private subnet CIDR
  AND dstaddr NOT LIKE '10.0.%'      -- leaving the VPC
  AND log_status = 'OK'
GROUP BY pkt_dst_aws_service
ORDER BY gb_total DESC;

You need Flow Logs v5 fields enabled for pkt_dst_aws_service to be populated. If you only have v2, you're stuck resolving IPs against the AWS IP ranges JSON — doable, but tedious.

The breakdown surprised us. Roughly:

• S3: ~45% of NAT egress bytes. Mostly asset pulls and log shipping from a legacy service.
• ECR: ~22%. Image pulls on autoscaling events and deploys.
• CloudWatch Logs: ~11%. Chatty applications logging full request bodies at INFO.
• Secrets Manager + STS: ~4%. Small payloads, lots of calls.
• Third-party APIs (actual public internet): ~12%.
• Everything else: ~6%.

Roughly 82% of what we were paying NAT fees for was traffic to AWS services that could, in principle, bypass NAT entirely.

A note on measurement honesty

Flow Logs count bytes at the ENI, so your Athena totals won't match the Cost Explorer "NAT Gateway processed bytes" line to the cent. In our case they were within ~7%. Close enough to make decisions. If you need them to match perfectly, you're overthinking it.

The fix: endpoints, in the right order

VPC endpoints come in two flavours, and the distinction matters for cost planning.

Gateway endpoints are free. They exist for S3 and DynamoDB only. You attach one to your VPC, update route tables, done. There is no per-hour and no per-GB charge. If you have an S3 gateway endpoint and you're still paying NAT fees for S3 traffic, something is misrouted.

Interface endpoints (PrivateLink) cost roughly $0.01 per hour per endpoint per AZ, plus around $0.01 per GB processed. That per-GB number is less than NAT's, but not zero. For small, chatty services the hourly cost can outweigh the savings if you're not careful.

We rolled out in this order:

1. S3 and DynamoDB gateway endpoints first

No-brainer. Free, immediate impact. The only gotcha: if you have bucket policies with aws:SourceIp conditions, they will stop matching once traffic comes from the VPC endpoint. Use aws:SourceVpce or aws:SourceVpc instead. We tripped on this with a bucket that had an explicit deny tied to the NAT EIP — the fix took ten minutes once we figured out why uploads were suddenly 403ing.

Terraform snippet, nothing exotic:

resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = aws_route_table.private[*].id

  policy = data.aws_iam_policy_document.s3_endpoint.json
}

2. ECR (api + dkr) and S3 — yes, S3 again

ECR image pulls hit three things: ecr.api, ecr.dkr, and the underlying S3 bucket where layers live. If you add ECR interface endpoints but forget the S3 gateway endpoint, layer pulls still go through NAT. We'd done step 1 already, so this was just the two interface endpoints. Image pull times also dropped noticeably — not something we were optimising for, but a welcome side effect.

3. CloudWatch Logs

This one needed a judgement call. The interface endpoint has a per-GB fee. For our log volume (~400 GB/month through this path), the maths worked: NAT at ~$0.045/GB vs interface endpoint at ~$0.01/GB + ~$22/month for the endpoint itself across two AZs. Break-even was around 30 GB/month. We were well past that.

We also used the opportunity to drop log verbosity on the chattiest service. Half the fix was networking; the other half was telling a service to stop logging the full response body at INFO.

4. Secrets Manager and STS

Small payloads, so the per-GB savings were trivial. We added these endpoints anyway because the latency improvement on cold-path auth calls was meaningful (STS AssumeRole in particular). This was the one decision we made on something other than cost.

What we deliberately did not do

We did not add interface endpoints for every service AWS offers. There's a long tail — Step Functions, EventBridge, Kinesis, SNS — where our traffic was low enough that the fixed hourly cost of the endpoint would have exceeded the NAT savings. The rule we settled on:

If a service accounts for less than ~10 GB/month of NAT egress, leave it on NAT. The endpoint will cost more than it saves.

We also did not try to eliminate NAT entirely. We still have genuine third-party API traffic — payment processors, email providers, a couple of SaaS webhooks going outbound. NAT is still the right tool for that. The goal was never zero NAT; it was zero wasteful NAT.

The result and the honest caveats

NAT Gateway data processing charges dropped by roughly 80% month-over-month after the full rollout. Total AWS networking spend dropped by a smaller percentage — around 55% — because interface endpoints have their own costs and some traffic genuinely belongs on NAT.

Things that didn't go clean:

• A Lambda function in a private subnet started timing out intermittently after we added the Secrets Manager endpoint. Root cause: it was using an old SDK version that cached DNS aggressively and held onto the public endpoint IP. Forcing a cold start fixed it; upgrading the SDK fixed it properly.
• One service's health checks broke for about four minutes during an endpoint rollout because the security group on the interface endpoint didn't allow 443 from the right subnet CIDR. Classic. Test in staging first, which we did — but staging had a flatter network topology, so the issue didn't surface until prod.
• Cost Explorer took about 48 hours to show the drop clearly. If you're reporting this to finance, wait for the data to settle before claiming victory.

Where we'd start if we were doing this again

Turn on VPC Flow Logs v5 with the AWS service field, run the Athena query above, and sort by GB descending. If S3 or DynamoDB is in the top three, you have a free win sitting there. If ECR is in the top three, your next win is two interface endpoints plus the S3 gateway endpoint you probably already need.

Don't chase every service. The 80/20 is real here — three or four endpoints will get you most of the way, and the long tail isn't worth the operational surface area. And budget half a day for the IAM policy conditions and security group rules, because that's where the actual time goes, not the endpoint resources themselves.

If you'd rather have someone else run this audit against your account, our cloud and DevOps team does exactly this kind of work.