OpenTelemetry Sampling at 1%: How We Lost the One Trace That Mattered

We shipped OpenTelemetry across a dozen services with a tidy 1% head sampler and congratulated ourselves on a reasonable bill. Six weeks later, during a 40-minute checkout outage, we opened our tracing backend and found exactly zero traces for the failing request path. The 99% we threw away was, of course, the 99% we needed.

This is the story of how we rebuilt sampling, what it cost, and the specific OTel Collector config we landed on.

Why head sampling looked like the adult choice

When you first wire up OpenTelemetry, the sampling decision feels academic. The SDK defaults to ParentBased(AlwaysOn), your dev traffic is small, and everything just works. Then you deploy to production, watch your tracing vendor's bill, and panic-set TRACES_SAMPLER=traceidratio with a ratio of 0.01.

Head sampling has real virtues:

• The decision is made once, at the root span, and propagated via tracestate.
• Downstream services don't waste CPU on spans that will be dropped.
• Egress to your backend drops roughly 100x.
• Your bill becomes predictable.

The tradeoff is the one everyone glosses over in the docs: you decide what to keep before you know whether it's interesting. A 1% sampler keeps 1% of boring 200s and 1% of catastrophic 500s. During an incident, if your error rate goes from 0.1% to 4%, you'll still only see a thin slice of the failures, and you have no way to pull the rest back.

The specific moment it broke

Our incident was a Stripe webhook retry loop interacting with a stale idempotency key cache. The failing requests were a tiny fraction of overall traffic — maybe 20 per minute against a baseline of 3,000 RPS. At 1% head sampling, the collector saw roughly one failing trace every three minutes, and because our tail_sampling processor wasn't yet in place, even those were randomly dropped.

We ended up reconstructing the failure from Sentry breadcrumbs and raw application logs. It worked, but it took an extra 25 minutes we didn't have.

The three sampling strategies, honestly compared

There are really only three shapes of sampling worth discussing in 2026, and the OTel docs tend to describe them as interchangeable. They aren't.

Head sampling

Decision made at the root. Cheap, simple, propagates cleanly. Terrible for incidents. Fine for steady-state capacity planning and latency histograms where you only need statistical signal.

Tail sampling

Decision made after the full trace is assembled, in the Collector. You can keep every error, every slow request, every trace touching a specific tenant or route. The cost: the Collector has to buffer spans for a decision window (typically 10 – 30 seconds), which eats memory and adds a stateful component you now have to run reliably.

In our experience, a tail_sampling processor handling ~5k spans/second needs 2 – 4 GB of memory per Collector replica with a 20-second decision wait. Not catastrophic, but not free.

Probabilistic + tail hybrid

What we actually run now. A low-rate probabilistic sampler at the edge for baseline coverage, plus tail-based rules that force-keep anything matching error, latency, or business-critical predicates. You get a predictable bill floor and a sharp signal when things go wrong.

The Collector config we landed on

After two more near-misses, here's roughly what we deployed. This runs in a dedicated Collector tier — not as a sidecar — because tail sampling needs to see all spans for a given trace ID in one place.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    send_batch_size: 1024
    timeout: 2s

  tail_sampling:
    decision_wait: 20s
    num_traces: 100000
    expected_new_traces_per_sec: 5000
    policies:
      - name: errors-always
        type: status_code
        status_code: { status_codes: [ERROR] }

      - name: slow-requests
        type: latency
        latency: { threshold_ms: 1500 }

      - name: checkout-critical
        type: string_attribute
        string_attribute:
          key: http.route
          values: ["/api/checkout", "/api/payments/.*"]
          enabled_regex_matching: true

      - name: enterprise-tenants
        type: string_attribute
        string_attribute:
          key: tenant.tier
          values: ["enterprise"]

      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 2 }

exporters:
  otlphttp/vendor:
    endpoint: https://otel.your-vendor.example
    headers:
      authorization: ${env:OTEL_VENDOR_TOKEN}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlphttp/vendor]

A few things worth calling out, because the defaults will bite you.

decision_wait is not free latency

It doesn't delay your users — spans are already exported from the app. But it does mean a 20-second gap between when a trace completes and when it appears in your backend. During an active incident, 20 seconds feels long. We tried 10 seconds and dropped roughly 3% of traces because long-running background jobs hadn't finished emitting spans. Twenty was the sweet spot for our workload.

num_traces is a memory ceiling, not a target

It caps the in-memory trace buffer. If your traffic spikes past expected_new_traces_per_sec, the Collector will evict traces before the decision window closes, and those evictions are silent unless you're scraping otelcol_processor_tail_sampling_sampling_trace_dropped_too_early. Scrape it. Alert on it.

Policies are evaluated as OR

If any policy says keep, the trace is kept. This is why the baseline probabilistic policy sits at the bottom — it's a floor, not a filter. The error, latency, and route policies are additive on top.

What it actually cost us

Moving from 1% head sampling to this hybrid setup roughly tripled our ingested span volume. Our tracing vendor bill went up about 2.4x. In exchange:

• Every 5xx response now has a full trace, no exceptions.
• Every request slower than 1.5s is retained, which caught two N+1 regressions in the first month.
• Enterprise tenant traffic is fully traced, which turned out to matter more for support than for engineering.
• The Collector tier added two m6i.large equivalents and about $140/month in infrastructure.

Is 2.4x worth it? For us, yes — one avoided 40-minute outage pays for roughly two years of the delta. For a side project or a service with no revenue tied to uptime, probably not. Keep the 1% head sampler and move on.

The things we got wrong on the way

A few mistakes worth not repeating.

We ran tail sampling as a sidecar first. It doesn't work. Spans from the same trace can land on different Collector instances, so the tail processor sees fragments and makes bad decisions. You need a load balancer that routes by trace ID (the loadbalancingexporter in a two-tier Collector setup) or a single Collector tier that all services export to.

We forgot about tracestate propagation. Our Next.js edge functions on Vercel were dropping the W3C trace context when calling internal APIs, so half our traces were orphans. The fix was explicit propagator config in the OTel SDK; worth auditing if you run anything on edge runtimes.

We over-indexed on error sampling and under-indexed on latency. Our first config kept every error but used a 3-second latency threshold. We missed a slow-burn regression where P95 crept from 400ms to 1.2s over a week because nothing tripped the threshold. Set latency thresholds relative to your current P95, not an absolute number.

Where we'd start

If you're staring at a 1% head sampler right now and wondering whether to change anything, do this in order:

1. Add a Prometheus scrape on your Collector and watch otelcol_processor_tail_sampling_count_traces_sampled and _dropped_too_early. You need the data before you can argue for the budget.
2. Stand up a second Collector tier (not sidecars) with a basic tail_sampling processor that keeps errors and samples 1% of the rest. Point one non-critical service at it and measure the span volume delta.
3. Add latency and route-based policies once you have a baseline. Resist the urge to add a policy for every team's pet route — the policy list gets unmaintainable fast.
4. Alert on dropped-too-early traces. That metric going non-zero means your sampling decisions are lying to you.

If you want a hand wiring this into an existing AWS or GCP stack, that's the kind of work our DevOps and cloud team does regularly. Otherwise, the OTel Collector docs are solid — just read the tail sampling section twice before you ship.