The OpenTelemetry Sampling Config That Hid Our Worst Latency Bug

A support ticket said checkout was taking twelve seconds. Our dashboards said P99 was 480ms. Both were true, and the reason lived inside a single line of our OpenTelemetry collector config.

This is the story of how we misused head-based sampling, what tail sampling actually costs in a mid-sized production system, and the collector setup we landed on after the dust settled.

The setup that looked reasonable on paper

We run a Node and Go backend on AWS — a mix of Fargate services and Lambda behind API Gateway, with Postgres on RDS and a few third-party APIs in the hot path (payments, address validation, tax). Instrumentation was OpenTelemetry SDKs auto-instrumenting HTTP, gRPC, and pg calls, exporting to an OTel Collector running as a sidecar, which forwarded to our vendor backend.

When we first rolled this out, our vendor bill was climbing fast. The obvious lever was sampling. We picked the default recommendation everyone gives when they see the invoice: probabilistic sampling at the SDK level, 5%.

# what we had (simplified)
processors:
  probabilistic_sampler:
    sampling_percentage: 5

Head-based, decided at the root span, propagated downstream via the traceparent header. Clean, cheap, standard. And exactly wrong for the problem we were about to have.

Why head-based sampling fails you here

Head-based sampling makes the keep-or-drop decision before the request has run. That means the interesting spans — the slow ones, the errored ones, the ones that touched a degraded dependency — are sampled at the same rate as the boring 200s. If 1 in 500 requests to checkout is slow, and you keep 1 in 20, you'll see the slow one maybe once every few hours, drowned in a sea of healthy traces.

Worse, our P99 dashboard was computed from the sampled data, then extrapolated. So a genuine latency spike affecting 0.2% of traffic looked like statistical noise. The dashboard wasn't lying — we had asked it a question it couldn't answer.

The bug we couldn't see

A customer support escalation forced the issue. A specific merchant's checkouts were timing out. We had their order IDs but no traces for any of them, because none had been sampled.

We turned sampling up to 100% for that merchant using a header attribute rule, waited a few hours, and finally got a trace. The culprit was our tax API integration: a specific combination of address country and product tax category triggered a retry loop inside the vendor SDK, with exponential backoff that maxed out at 8 seconds per attempt, three attempts deep. Total added latency: ~11 seconds on the affected requests.

The fix was one line — a shorter timeout and no retry on that specific error class. But we'd been shipping that bug for roughly three weeks. Three weeks of a subset of customers seeing broken checkouts that never made it into our sampled traces.

What we changed: tail sampling, and what it actually costs

Tail sampling makes the keep-or-drop decision after the trace is complete, so you can keep the interesting ones. The OTel Collector's tail_sampling processor supports this, with policies like latency thresholds, status codes, and attribute matches.

Here's a trimmed version of what we ended up with:

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    expected_new_traces_per_sec: 500
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow-traces
        type: latency
        latency:
          threshold_ms: 1500
      - name: checkout-full
        type: string_attribute
        string_attribute:
          key: http.route
          values: ["/checkout", "/checkout/confirm"]
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 3

Four policies: keep every error, keep everything slower than 1.5s, keep 100% of checkout routes, and sample 3% of everything else as a baseline for trend data. A trace is kept if any policy matches.

The tradeoffs nobody warns you about

Tail sampling isn't free, and the OTel docs undersell the operational cost. In our experience:

• Memory. The collector has to buffer every span for every trace until decision_wait expires or the trace is judged complete. Our collector RAM usage roughly quadrupled. Budget for it. We moved from a sidecar model to a dedicated collector deployment on Fargate with more headroom.
• You need a gateway topology. Tail sampling only works if all spans for a trace land on the same collector instance. Sidecar per service breaks this. We run a load-balanced collector gateway with trace-ID-aware routing (the loadbalancing exporter in front of the tail-sampling collectors).
• Decision wait tuning. Set it too short and you drop late spans, so your slow traces get truncated — exactly the ones you wanted to keep. Set it too long and memory balloons. We started at 30s, ended at 10s after measuring actual trace completion times.
• Cost math is not obvious. Yes, we dropped total spans shipped by ~60%. But the value per retained span went up dramatically, because the retained ones are now the ones you'd actually query. Our vendor bill went down about 40% net after the extra collector infrastructure.

The collector topology we settled on

The final layout, roughly:

1. Application → OTel SDK → local collector agent (sidecar or DaemonSet). Agents do batching and resource enrichment only. No sampling decisions here.
2. Agents → collector gateway (load-balanced, trace-ID-consistent hashing via the loadbalancing exporter).
3. Gateway collectors → tail sampling → vendor backend.

The gateway is where the memory lives and the intelligence sits. Agents stay dumb and cheap.

# agent config (simplified)
exporters:
  loadbalancing:
    protocol:
      otlp:
        tls:
          insecure: false
    resolver:
      dns:
        hostname: otel-gateway.internal
    routing_key: traceID

Without the routing_key: traceID, tail sampling silently makes wrong decisions because it never sees the full trace. This bit us in staging and we caught it because a synthetic slow trace was being dropped roughly half the time.

What we monitor on the collector itself

The collector became a tier-1 dependency, so we treat it that way. What we alert on:

• otelcol_processor_tail_sampling_sampling_trace_dropped_too_early — spans arriving after the decision was made. If this climbs, decision_wait is too short.
• otelcol_processor_tail_sampling_new_trace_id_received vs actual traffic — if these diverge, load balancing is broken.
• Collector heap. We page at 80% sustained.
• Export failures to the vendor. Backpressure here can drop everything.

We also keep a small debug exporter path writing 0.1% of decisions to CloudWatch Logs, so when someone asks "why was this trace dropped", we can actually answer.

Where head-based still makes sense

Not every system needs this. If your traffic is uniform, your services are few, and your bill is small, head-based at 10% is fine and much simpler to run. Tail sampling earns its keep when:

• You have long-tail latency that matters to specific customers.
• Error rates are low enough that random sampling misses them.
• Your traces span multiple services and dependencies with variable behaviour.
• You care about being able to explain a specific user's specific bad experience.

If none of those apply, don't add the complexity.

Where we'd start

If you're setting this up from scratch in 2026, do this in order:

1. Instrument first, sample second. Get full traces flowing at 100% in staging so you know what "complete" looks like per service. Measure your actual trace completion times before picking decision_wait.
2. Build the gateway topology from day one. Retrofitting load-balanced trace-ID routing into a sidecar-only setup is painful. Start with a gateway even if it's a single instance.
3. Write your policies against real incidents. Look at your last five production issues. What would you have needed to keep to debug them faster? That's your policy list, not a copy-paste from the docs.
4. Alert on the collector as a service. Dropped-too-early and export failure metrics should page someone. Silent observability failure is worse than no observability, because you trust the empty dashboard.

If you want help auditing an existing OTel setup or designing a tail-sampling topology for a system that's grown past its original config, that's the kind of work our DevOps and Cloud team does regularly. The dashboards you trust should be the ones you've tried to break.