The S3 Egress Bill That Doubled Overnight: Tracing a Rogue Prefetch

The AWS cost anomaly email hit at 06:42. Our S3 data transfer line had doubled in 24 hours, from a steady baseline to something that would blow the quarter's infra budget if it ran for a week. Nothing had been deployed. No traffic spike. No new customer onboarded. That's the worst kind of alert — the one where the graph moves and your changelog doesn't.

This is the story of how we found it, what we changed, and the observability we wish we'd had in place before it happened.

The shape of the spike

The first thing we did — before touching any code — was stare at Cost Explorer with a grouping by usage type. Two lines mattered:

• DataTransfer-Out-Bytes on S3
• Requests-Tier1 on S3 (PUT/COPY/POST/LIST)

Egress was up ~110%. GET requests were up ~180%. That ratio was the first real clue. If a legitimate feature had gone viral, we'd expect egress and requests to scale roughly together, weighted by average object size. A request count growing faster than bytes meant something was pulling lots of small objects, or pulling the same objects repeatedly without cache hits.

CloudFront cache hit ratio in the same window had dropped from ~94% to ~71%. So the origin — S3 — was seeing traffic it normally wouldn't. That reframed the problem: this wasn't an S3 bug, it was a CDN cache-miss storm.

Ruling out the obvious

Before going deep, we ran the boring checklist:

• No new deployments in the last 72 hours (checked Vercel and our ECS pipelines).
• No lifecycle policy changes on the bucket.
• No new IAM principals accessing the bucket (CloudTrail confirmed).
• No replication rule flipped on.
• No signed-URL leak on GitHub (we grep public gists and run a scheduled secret scan).

All clean. The traffic was legitimate — coming through our CDN, from real user agents. Which meant our own code was doing this to us.

Turning CloudFront logs into a query

We already ship CloudFront access logs to S3. What we hadn't done, embarrassingly, was keep an Athena table pointed at them with partitions up to date. Twenty minutes of Terraform later, we had one.

SELECT
  cs_uri_stem,
  COUNT(*) AS hits,
  SUM(sc_bytes) / 1024 / 1024 AS mb_out,
  SUM(CASE WHEN x_edge_result_type = 'Miss' THEN 1 ELSE 0 END) AS misses,
  SUM(CASE WHEN x_edge_result_type = 'Hit'  THEN 1 ELSE 0 END) AS hits_cache
FROM cloudfront_logs
WHERE date BETWEEN DATE '2026-01-13' AND DATE '2026-01-14'
GROUP BY cs_uri_stem
ORDER BY mb_out DESC
LIMIT 50;

The top row was a single path pattern: /product-media/hero/*.mp4. Hero videos. Three of them, each between 8 and 14 MB, autoplayed muted on the product listing page. Not unusual on their own.

What was unusual: the misses column for those URIs was almost equal to hits. Meaning CloudFront kept fetching them from S3 despite them being served millions of times.

Why was cache missing?

We checked the response headers on one of those objects:

Cache-Control: public, max-age=31536000, immutable

That's fine. So the object was cacheable. But then we looked at the request headers coming in:

Range: bytes=0-1
Range: bytes=0-524287
Range: bytes=524288-1048575

Range requests. Lots of them, with wildly varying byte ranges. CloudFront does cache range requests, but only when the ranges are consistent. When a client asks for arbitrary byte windows — especially small ones — the edge frequently has to go back to origin.

Something was hammering the videos with byte-range GETs.

Finding the caller

We cross-referenced User-Agent and cs_referer in the same Athena query. The traffic wasn't coming from bots. It was coming from real Chrome and Safari sessions, referred from our own product listing page.

That narrowed it to the frontend. We opened the listing page in Chrome DevTools, filtered Network by mp4, and hit reload. On the surface, nothing looked wrong — three videos loaded, played, life went on. But the Network tab showed 47 separate requests for those three videos on a single page load. Some completed. Most were cancelled after a few hundred KB.

Git blame on the video component pointed at a well-intentioned PR from three weeks earlier titled "Preload hero videos for smoother autoplay". It introduced a hook that, on hover of any product card, would start prefetching the associated hero video via fetch() with a small initial range, then abort if the user moved on.

useEffect(() => {
  if (!isHovered) return;
  const controller = new AbortController();
  fetch(videoUrl, {
    headers: { Range: 'bytes=0-524287' },
    signal: controller.signal,
  });
  return () => controller.abort();
}, [isHovered, videoUrl]);

On paper: reasonable. In production: catastrophic. The listing page renders ~40 cards. A user scanning the page hovers over a dozen in a couple of seconds. Each hover triggers a range GET. Each aborted fetch still counts as an origin request if CloudFront hadn't cached that exact range yet. And because Chrome's own video element also issues range requests once autoplay kicks in, we were doubling the fetches on any card the user actually stopped on.

Multiply that by the ~180k daily sessions on that page, and you get 3.2 TB of extra egress in a day.

Why it didn't show up in staging

Staging traffic is synthetic. Our Playwright tests don't hover-scan a grid of 40 cards the way a human does. And our load tests hit API endpoints, not the marketing surface. This is a recurring pattern: cost bugs live in places load tests don't visit.

The fix, and what we changed around it

The immediate fix was a two-liner: replace the fetch() prefetch with a <link rel="preload" as="video"> gated by IntersectionObserver, and only for the first four cards above the fold. Egress dropped back to baseline within an hour of the deploy.

But a fix isn't a lesson. Here's what we actually changed:

• Cost anomaly alerts at the service level, not just account level. AWS Cost Anomaly Detection now watches S3 and CloudFront separately, with a 15% threshold instead of the default. We route those to the same Slack channel as Sentry alerts, because a cost spike is a bug.
• CloudFront cache hit ratio as a first-class SLO. We publish it to our Grafana instance from CloudWatch and alert when it drops below 90% for 30 minutes. Cache hit ratio is a leading indicator; the bill is a lagging one.
• Athena table on CloudFront logs, kept warm. Partition projection on date, so we don't need to run MSCK REPAIR TABLE ever again. When something weird happens, we can query it in minutes, not hours.
• A network-budget check in CI. For the marketing routes, our Playwright suite now measures total bytes fetched per page load and fails the build if it exceeds a threshold. Crude, but it would have caught this PR.
• A frontend rule: no fetch() with Range headers without review. Range requests bypass most caching intuitions. If you need one, you need to think about what the CDN will do with it.

The uncomfortable part

The PR that caused this passed review. Two senior engineers looked at it. It made the site feel faster in dev. It shipped a real UX improvement. The cost side of the tradeoff was completely invisible at review time — because nothing in our tooling surfaced it. That's on us as a team, not on the author.

One heuristic we now apply: any code that triggers a network request in response to a mouse event, especially a passive one like hover, is a cost-and-reliability decision as much as a UX one. Treat it accordingly.

What we'd do if this happened to you tomorrow

If you're staring at a doubled S3 or egress bill right now, in order:

1. Group Cost Explorer by usage type. Look at the ratio of request count growth to byte growth.
2. Check CloudFront (or your CDN's) cache hit ratio for the same window. If it dropped, the problem is cache misses, not new traffic.
3. Point Athena at your access logs. Find the top URIs by bytes and by origin fetches.
4. Look at request headers — Range, unusual query strings, cache-busting parameters — on the top offenders.
5. Grep your frontend for fetch( against those URLs. Nine times out of ten, the culprit is code you shipped and forgot about.

And once the bleeding stops: put the alerting in place so next time, you find out from a dashboard instead of a finance email. If you want a hand wiring up CloudFront logs, Athena, and cost anomaly detection into something usable, that's the kind of thing our DevOps and cloud team does day in and day out.