OpenTelemetry Sampling: Why Head-Based Cost Us Real Incidents

We spent about a year on head-based sampling with the OpenTelemetry Collector because the docs made it sound like the sensible default. Then two incidents in the same quarter showed us exactly what we'd traded away. This is what we changed, what it cost, and what we'd tell a team starting from zero in 2026.

The setup that seemed fine

Our backend is a mix of Next.js on Vercel, a couple of Go services on GCP Cloud Run, and a Postgres cluster on AWS RDS. We instrumented everything with the OpenTelemetry SDKs and shipped spans to a self-hosted Collector, which forwarded to a commercial backend (the specific vendor doesn't matter here — the sampling problem is the same across Honeycomb, Datadog, Grafana Cloud, and Tempo).

At roughly 40M spans/day, sending 100% was not going to happen on our budget. So we did what most teams do: set a parentbased_traceidratio sampler at 10% at the SDK level. Simple, cheap, and it worked out to about 4M spans/day landing in the backend.

# what we started with, roughly
processors:
  batch:
    send_batch_size: 1024

exporters:
  otlp:
    endpoint: ingest.example.com:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

The SDK made the sampling decision at the root span using the trace ID, and every child service respected it via parentbased. Clean, consistent, cheap.

Why head-based is seductive

Head-based sampling has real advantages you shouldn't dismiss:

• The decision is made once, at the start of the trace. No buffering.
• Every downstream service agrees, so you never get half-traces.
• Collector memory is basically flat regardless of traffic spikes.
• Cost is predictable to the span.

If you're early stage, or your traffic is uniform and your errors are common enough to show up in 10% of traces, head-based is genuinely fine. Don't let anyone tell you it's amateur hour. It's not.

The two incidents that changed our minds

Incident one: the checkout 500s we couldn't see

A payment provider changed a response shape. Roughly 0.3% of checkouts started failing with a 500. Support flagged it before our error budget alerts did, because the volume was low but the customer impact was high (these were paying customers, not bots probing endpoints).

We went to look at traces. At 10% sampling, we had roughly 30 traces per hour of the actual failure — but the specific merchant/currency combination that triggered the bug appeared in maybe 1 in 8 of those. So we had maybe 4 useful traces per hour, and half of them were missing the downstream span because a proxy in the middle wasn't propagating context correctly.

We found the bug from logs, not traces. The traces were theoretically there. In practice, sampling had thrown away most of what we needed, and the ones we kept weren't the interesting ones.

Incident two: the slow tail nobody saw

A background job hit a p99.9 latency of about 40 seconds on a specific tenant. p50 and p95 were fine. Our sampler, being uniformly random at 10%, kept 10% of the fast requests and 10% of the slow ones. Since slow requests were rare to begin with, we had almost no traces of the actual pathology.

The fix took us three days longer than it should have because we were reasoning from aggregate metrics instead of concrete traces.

What tail-based sampling actually gives you

Tail-based sampling waits until a trace is complete (or a timeout fires), then decides whether to keep it. That means you can make the decision based on things you couldn't possibly know at the head: the final status code, the total duration, whether any span had an exception, whether a specific attribute is present.

The cost is real. Concretely:

• The Collector has to buffer every span for every in-flight trace until the decision window closes.
• You need a single logical Collector instance (or a load balancer that routes by trace ID) so all spans for a trace end up in the same place.
• Memory scales with traffic and with your decision window.

In our environment, moving from head-based at the SDK to tail-based at the Collector roughly tripled Collector memory and added one more pod tier we had to run. Call it a 2-3x infra cost bump on the Collector layer specifically. Backend ingest cost dropped, because we got much smarter about what we kept.

A config that actually works

Here's a stripped-down version of what we run now. The real one has more policies, but this is the shape:

processors:
  tail_sampling:
    decision_wait: 30s
    num_traces: 100000
    expected_new_traces_per_sec: 2000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow-traces
        type: latency
        latency:
          threshold_ms: 1500
      - name: checkout-always
        type: string_attribute
        string_attribute:
          key: http.route
          values: ["/api/checkout", "/api/payment/.*"]
          enabled_regex_matching: true
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

A few things worth pointing out:

• decision_wait: 30s is the ceiling on how long we hold spans. Longer means better decisions and more memory. 30s covers 99%+ of our traces.
• The baseline probabilistic policy keeps 5% of everything, so we still see healthy traffic patterns. Without it, you get a beautifully broken sample where only errors and slow requests exist and your p50 dashboard looks like a horror film.
• Business-critical routes are always kept. This is the single highest-leverage change we made.

The load balancer problem

If you run more than one Collector replica — and you should, for availability — you need trace-ID-aware routing. Otherwise spans from the same trace land on different Collectors, and neither has enough context to decide.

The OTel Collector ships a loadbalancing exporter for this. You run a tier of "gateway" Collectors that just do routing, and a tier of "sampling" Collectors that do the decision. It's more moving parts, but it's the only way tail sampling works at scale.

exporters:
  loadbalancing:
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      dns:
        hostname: otel-tail-collector.internal

The tradeoffs nobody tells you upfront

You lose true statistical sampling

With head-based at 10%, you can multiply span counts by 10 and get honest traffic estimates. With tail-based, your kept sample is heavily biased toward errors and slow requests. If you use traces for volume metrics, you'll be wrong. Push volume metrics through actual metrics (Prometheus, OTel metrics), not through counting spans.

Decision windows create real latency

A 30-second decision wait means traces show up in your backend 30 seconds later than they used to. If your on-call reflex is to open the traces UI while the incident is still unfolding, that lag matters. Not a dealbreaker, but a real change to muscle memory.

Memory pressure is now your problem

We've had two Collector OOMs since switching, both during traffic spikes we hadn't provisioned for. Set memory limits, alert on Collector restarts, and treat the sampling tier as a real production service, not observability plumbing that runs itself.

Where we'd start

If we were building this fresh today:

1. Start with head-based at whatever ratio your budget tolerates. Don't over-engineer on day one.
2. The moment you have a real incident where sampling burned you, do the migration. Not before — you need the scar tissue to configure tail policies well.
3. When you switch, always keep 100% of errors, always keep slow traces above a route-specific threshold, always keep business-critical routes, and keep a probabilistic baseline of 3-10% for everything else.
4. Budget for 2-3x Collector infra. Anyone quoting less either has trivial traffic or hasn't hit their first spike yet.
5. Move traffic-volume metrics out of traces and into real metrics before you switch, or your dashboards will lie to you the next morning.

Sampling is one of those decisions that looks like a config knob and turns out to be an architecture choice. Treat it that way and you'll spend a lot less time in postmortems explaining why the traces you needed weren't there. If you want to talk through your own setup, our team writes about this kind of thing regularly on the 72Technologies blog.