The CloudFront Cache Key That Cost Us $4K in a Weekend

On a Friday afternoon in October, someone shipped a two-line change to add a UTM-style tracking parameter to our marketing links. By Sunday night our CloudFront bill had a fresh four-figure spike and our origin was quietly on fire. Nothing was down. Nothing paged. That was the problem.

This is the story of how a cache key ate a weekend, what the cost actually looked like, and the cache policy pattern we now apply by default on every new distribution.

The setup

We run a fairly standard architecture for one of our e-commerce clients: a Next.js app served from an ALB behind CloudFront, with static assets on S3 (also behind the same distribution via multiple origins). The cache hit ratio on the marketing pages hovered around 92–95% in a normal week. Origin costs were dominated by the API, not HTML.

The distribution had been created years ago with the legacy behaviour of forwarding all query strings to the origin and including them in the cache key. Nobody had touched it because it worked.

The change that broke things

Marketing wanted better attribution, so an engineer added a sid (session ID) parameter to every outbound campaign link. Not utm_*, which we already stripped at the edge — a new one. The value was a random UUID per click.

Every visitor now hit a URL like:

https://shop.example.com/collections/autumn?sid=8b3f...c1d2

Because sid was part of the cache key, every single click was a unique cache entry. Cache hit ratio for the marketing paths collapsed from ~94% to under 3% within hours. The pages still rendered. The origin scaled. Latency crept up but stayed within SLO. No alerts fired.

What the bill actually looked like

We caught it Monday morning when the FinOps digest landed. A few observations from the invoice detail, without pretending these numbers generalise:

• CloudFront request charges roughly tripled for the weekend
• Origin data transfer out of the ALB was up about 8x versus the previous weekend
• Lambda@Edge invocations (we run a small viewer-request function) spiked in lockstep with requests
• Total incremental spend for the ~60-hour window landed just north of $4,000

The painful part: revenue was flat. We paid four grand to serve identical HTML to people who would have hit the cache under the old link format.

Why nothing paged

We had alerts on 5xx rates, p95 latency, and origin health. We did not have an alert on cache hit ratio. In hindsight this is obvious — hit ratio is a leading indicator for both cost and eventual latency degradation. It's on our default alert bundle now.

We also had a CloudWatch anomaly detector on origin request count, but the ramp was gradual enough (Friday evening traffic is low) that it didn't trip until Sunday, and by then the damage was done.

The fix, and why it wasn't just "strip the param"

The reflex fix is: strip sid at the edge. That works, but it's whack-a-mole. Marketing will add another parameter next quarter. Someone will build a new distribution and forget. The real fix is to invert the default: the cache key should contain nothing except what you explicitly opt in.

CloudFront's newer Cache Policy and Origin Request Policy split solves this cleanly. The cache key is separate from what gets forwarded to the origin. You can forward every query string to the origin (so your analytics still work) while including none of them in the cache key.

Here's the Terraform we now use as a baseline for HTML behaviours:

resource "aws_cloudfront_cache_policy" "html_minimal" {
  name        = "html-minimal-cache-key"
  default_ttl = 60
  max_ttl     = 3600
  min_ttl     = 0

  parameters_in_cache_key_and_forwarded_to_origin {
    enable_accept_encoding_brotli = true
    enable_accept_encoding_gzip   = true

    query_strings_config {
      query_string_behavior = "whitelist"
      query_strings {
        items = ["page", "sort", "filter"]
      }
    }

    headers_config {
      header_behavior = "whitelist"
      headers {
        items = ["Accept-Language", "CloudFront-Viewer-Country"]
      }
    }

    cookies_config {
      cookie_behavior = "none"
    }
  }
}

resource "aws_cloudfront_origin_request_policy" "html_forward_all" {
  name = "html-forward-all-query-strings"

  query_strings_config {
    query_string_behavior = "all"
  }

  headers_config {
    header_behavior = "allViewer"
  }

  cookies_config {
    cookie_behavior = "all"
  }
}

The cache policy whitelists three query strings that actually change the rendered HTML. Everything else — sid, future marketing IDs, whatever product wants to add next — gets forwarded to the origin for analytics but doesn't fragment the cache.

The gotcha with Vary and cookies

We learned two things the hard way while rolling this out:

1. If your origin sends Set-Cookie on HTML responses (session bootstrap, CSRF tokens, A/B test assignment), CloudFront will refuse to cache the response by default. We moved session bootstrap to a separate /api/session call and stopped setting cookies on the HTML path. Cache hit ratio jumped another few points as a bonus.
2. Vary: Accept-Encoding from the origin combined with CloudFront's own compression handling can create surprise cache misses. Let CloudFront handle compression (enable brotli and gzip in the cache policy) and have the origin serve uncompressed. Don't send Vary: Accept-Encoding yourself.

What we monitor now

After the incident we added four things to the default observability bundle for any distribution we own or manage:

• Cache hit ratio alert at both the distribution level and per-behaviour, with a threshold of "drops more than 10 points from 7-day baseline". This is the alert that would have caught the incident within an hour.
• Origin request count anomaly detection with a tighter sensitivity than the default. CloudWatch's built-in detector is fine but it's tuned conservatively.
• Bytes-downloaded-from-origin as a cost proxy. Requests can lie (small responses), bytes don't.
• A weekly report listing the top 20 cache keys by miss count. If you see UUIDs or timestamps in there, something is wrong.

We wire all of this through the same OpenTelemetry pipeline the rest of the app uses, with CloudFront real-time logs shipped to a Kinesis stream and aggregated. It's not free, but at the volume where a bad cache key costs $4k in a weekend, the observability spend pays for itself in one prevented incident.

The org-level fix

The technical fix is the easy part. The harder question is: how did a marketing parameter change ship without anyone thinking about the CDN? The honest answer is that nobody owned the cache. The frontend team owned the app, the platform team owned the distribution, and marketing owned the links. The cache key sat in the seam.

We now require any new query string parameter that appears in a URL served through the CDN to be reviewed against the cache policy whitelist. It's a two-line checklist item in the PR template. It's not glamorous. It works.

Where we'd start

If you run anything non-trivial behind CloudFront and haven't looked at your cache and origin request policies in a year, do these three things this week:

1. Pull your current cache hit ratio per behaviour from CloudWatch. Anything under 80% on cacheable content deserves an explanation.
2. Audit what's in your cache key. If it says "forward all query strings" or "forward all headers", you're one marketing campaign away from our weekend.
3. Add a cache hit ratio alert before you do anything else. It's the single highest-leverage alert we've added in the last two years.

If you'd rather have someone else audit this for you, that's the kind of unglamorous reliability work our DevOps and cloud team does — usually starting with the boring stuff that nobody's looked at since the distribution was created.