OpenTelemetry Sampling in Production: The Config That Saved Our Trace Bill

Our trace ingestion bill tripled the week after we rolled OpenTelemetry out to a fourth service. The kicker: 94% of the traces we were paying to store were successful health checks and cache hits. This is the story of how we fixed sampling without going blind during incidents.

Why Default Sampling Fails You

The OpenTelemetry SDKs ship with a ParentBased(AlwaysOn) sampler by default. That's fine for a demo. In production it means every span from every request ends up in your backend, and your vendor invoice starts to look like a Series A fundraise.

The obvious fix is head sampling — decide at the start of a trace whether to keep it, usually with TraceIdRatioBased(0.1) for 10%. It's cheap, it's stateless, and it's wrong for anything that matters.

Here's why: head sampling is random. It doesn't know that this particular request is going to 500 out three services deep. It doesn't know the user is a paying customer on an enterprise plan. It doesn't know the span will run for 12 seconds. By the time you'd know any of that, the decision was already made and the trace is either fully captured or fully gone.

We ran head sampling at 5% for about a month. Every incident post-mortem had the same line: "we don't have a trace for the failing request." That's not observability, that's a lottery.

The Tail Sampling Promise (and Its Teeth)

Tail sampling flips the model. The OTel Collector buffers spans for a trace, waits until it thinks the trace is complete, then decides. You can keep every error, every slow request, every trace touching a specific tenant — and drop the boring successes.

The teeth: the collector has to hold every span in memory until the decision window closes. For a service doing 4k RPS with 20 spans per trace and a 10-second wait, you're buffering roughly 800k spans at any given moment. Our first tail-sampling collector OOM-killed itself within 20 minutes of going live.

The Load Balancing Problem

There's a second tooth that bit us harder. Tail sampling only works if all spans for a single trace land on the same collector instance. If you run three collector replicas behind a normal round-robin load balancer, spans for the same trace get sharded across instances, and each instance sees a partial trace. It makes a bad decision because it doesn't have the full picture.

The fix is a two-tier collector setup: a stateless front tier that routes by trace ID, and a stateful sampling tier that does the actual buffering.

# front-tier collector config (stateless, routes by trace id)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      k8s:
        service: otel-sampler.observability
        ports: [4317]

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [loadbalancing]

The loadbalancing exporter hashes the trace ID and pins it to a specific downstream collector. Now every span for a given trace lands on the same sampler instance, and tail sampling can actually work.

The Policy Set That Held Up

After three iterations, this is the policy set we've been running for about six months. It's not fancy. It's deliberately boring.

# sampler-tier collector config
processors:
  tail_sampling:
    decision_wait: 8s
    num_traces: 100000
    expected_new_traces_per_sec: 2000
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }

      - name: slow-requests
        type: latency
        latency: { threshold_ms: 750 }

      - name: enterprise-tenants
        type: string_attribute
        string_attribute:
          key: tenant.tier
          values: [enterprise]

      - name: checkout-flow
        type: string_attribute
        string_attribute:
          key: http.route
          values: [/api/checkout, /api/checkout/confirm]

      - name: probabilistic-baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 3 }

      - name: drop-healthchecks
        type: string_attribute
        string_attribute:
          key: http.route
          values: [/health, /ready, /metrics]
          invert_match: false

A few things worth calling out:

• Errors always win. Any span with status_code=ERROR keeps the whole trace. This is non-negotiable.
• Latency threshold is per-service tuned. 750ms is our API gateway default. Our worker services use 5s. Don't copy-paste a single latency threshold across your whole fleet.
• The 3% probabilistic baseline is the safety net. Without it, you have no idea what "normal" looks like. When p99 latency drifts up, that baseline is how you'll notice.
• Health checks are dropped explicitly. They pass the probabilistic filter otherwise and pollute the baseline signal.

Tuning num_traces and decision_wait

These two knobs are where the memory bill lives. num_traces is the max in-flight trace count per collector instance. decision_wait is how long you buffer before deciding.

Our rough sizing rule: expect around 2-4 KB of memory per buffered span, and multiply by average spans-per-trace and expected traces-per-second times the decision wait. If you're seeing collector memory climb until OOM, either lower num_traces, shorten decision_wait, or scale out the sampler tier horizontally. In our experience, shortening decision_wait from 30s to 8s made the biggest single improvement — most of our traces complete well under a second, and the long tail we were waiting for wasn't giving us much extra signal.

What We Actually Saw

Moving from 5% head sampling to this policy set:

• Trace volume dropped by roughly 60% versus our brief AlwaysOn experiment, and landed about 40% higher than 5% head sampling — but we now keep 100% of errors and slow requests.
• Collector memory sits around 4-6 GB per sampler instance at peak, with three replicas handling our load. That's steady-state, not the OOM lottery we started with.
• Incident MTTR improved noticeably because "we don't have a trace" stopped appearing in retros. We didn't measure this rigorously — anyone who claims a precise MTTR delta from a sampling change is selling you something.
• Vendor bill: down meaningfully from the AlwaysOn peak, up modestly from the head-sampling floor. The tradeoff was obvious once we had the data.

The Gotchas Nobody Warns You About

Async spans break decision_wait. If your service kicks off a background job that lives longer than your decision window, the sampler will decide before the async span arrives. That late span shows up orphaned. Either raise the wait for those specific traces (using a policy on span name) or accept the orphans.

Sampling decisions don't propagate to logs. Your traces are sampled, but your log volume is not. If you correlate logs to traces via trace ID, you'll have logs for traces that got dropped. Consider a log processor that drops logs whose trace ID wasn't sampled — or accept the mismatch.

Collector upgrades can silently change memory behavior. We got bitten by a minor version bump that changed how the tail sampling processor handled trace eviction. Pin your collector image tag and test upgrades in a staging environment that gets real traffic mirroring. "It's just a patch version" is famous last words.

Sentry and OTel double-billing. If you're running Sentry for errors and OTel for traces, make sure you're not paying for the same error span twice. We route Sentry SDK data through a Sentry-specific pipeline and keep it out of the OTel error-sampling policy.

Where We'd Start

If you're staring at a trace bill you can't defend, don't jump to tail sampling on day one. Start here:

1. Turn on head sampling at something conservative — 10% — and measure your bill baseline for a week.
2. Instrument a single high-value service with tail sampling behind a load-balancing collector. Just one. Learn the memory profile before you scale it.
3. Write your policies in this order: errors, latency, business-critical routes, probabilistic baseline, explicit drops. Resist the urge to get clever.
4. Set an alert on sampler collector memory at 70% of limit. You want to know before the OOM, not after.

If you want a hand designing this for a specific stack, our team does this work regularly — see our DevOps and cloud services or dig through more posts on observability. Sampling isn't glamorous, but it's one of the highest-leverage config decisions you'll make this year.