OpenTelemetry Sampling at 100 RPS: What We Keep, What We Drop, and Why
Head sampling is cheap and lies to you. Tail sampling is honest and expensive. Here's the sampling config we actually run in production, and the traces we deliberately throw away.
The first time we turned on OpenTelemetry tracing across a mid-sized Node and Go stack, our observability bill tripled in a week. The second time, we were smarter about sampling — and we still got it wrong, just in a more interesting way. This is what we've landed on for services running between 50 and 500 requests per second, and the tradeoffs we accepted to get there.
Why sampling is the only real knob
A single HTTP request in a modern system produces somewhere between 8 and 40 spans once you include the inbound handler, outbound HTTP calls, database queries, cache lookups, and any async work. At 200 RPS, that's roughly 2 to 8 million spans per hour per service. Most trace backends — Grafana Tempo, Honeycomb, Datadog APM, New Relic — charge on ingested spans or events. You can argue about storage tiers and compression, but the unit economics are the same: fewer spans in means less money out.
So you sample. The question is just when and how.
Head sampling: cheap, fast, lies by omission
Head sampling decides at the start of a trace whether to keep it. The TraceIdRatioBased sampler in the OTel SDK is the canonical example — pick a ratio like 0.1 and keep 10% of traces. The decision propagates through traceparent, so every downstream service agrees.
The upside: zero buffering, zero collector memory, trivial to reason about. The downside: you make the keep/drop decision before you know whether the request failed, took 12 seconds, or triggered a retry storm. At 10% head sampling, you will lose 90% of your 5xx traces. Your error rate dashboard is fine because that comes from metrics, but the actual trace you want to stare at during an incident is probably one of the ones you threw away.
Tail sampling: honest, expensive, fiddly
Tail sampling happens in the OTel Collector after all spans for a trace are received. You can keep every errored trace, every slow trace, every trace matching a specific attribute, and drop the uninteresting middle. The cost is the collector has to buffer spans in memory for a decision_wait window — typically 10 to 30 seconds — and you need enough collector capacity to hold the firehose.
In our experience, tail sampling collectors need roughly 2–4x the CPU and 3–6x the memory of a pass-through collector at the same throughput. That's not nothing, but it's usually cheaper than paying the backend for everything.
The config we actually run
Here's a slimmed-down version of the tail sampling processor we use in the OTel Collector for a typical web backend. The real thing has more routes and more exclusions, but the shape is honest.
processors:
tail_sampling:
decision_wait: 15s
num_traces: 100000
expected_new_traces_per_sec: 300
policies:
- name: errors
type: status_code
status_code: { status_codes: [ERROR] }
- name: slow-requests
type: latency
latency: { threshold_ms: 800 }
- name: critical-routes
type: and
and:
and_sub_policy:
- name: route-match
type: string_attribute
string_attribute:
key: http.route
values: ["/checkout", "/auth/login", "/api/payments/.*"]
enabled_regex_matching: true
- name: sample-half
type: probabilistic
probabilistic: { sampling_percentage: 50 }
- name: health-and-metrics
type: string_attribute
string_attribute:
key: http.route
values: ["/healthz", "/metrics", "/ready"]
invert_match: false
# this policy's matches get dropped via the composite below
- name: baseline
type: probabilistic
probabilistic: { sampling_percentage: 5 }
A few things worth calling out:
decision_wait: 15sis a compromise. Shorter means less memory but you'll miss slow traces that haven't finished. Longer means you buffer more. 15 seconds covers the P99 of most synchronous traffic we see.num_traces: 100000is the in-memory cache size. If you blow past this, the collector evicts and your sampling decisions get weird. Size it for roughlydecision_wait * expected_new_traces_per_sec * 1.5.- Baseline 5% keeps a trickle of healthy traffic so you can still answer "what does a normal /checkout look like?" without going to logs.
Health checks and metrics scrapes we actually drop entirely in a separate filter processor upstream — they'd otherwise dominate the baseline.
What this costs vs. keeping everything
On a service doing 180 RPS average, the above config kept roughly 11–14% of traces by span count. Errors and slow traces are a small fraction of total volume, so most of that is the 5% baseline plus the weighted critical routes. Against a backend charging per-span, that's an 85%+ reduction in trace ingest compared to no sampling, while keeping 100% of errors and 100% of anything over 800ms.
The collector itself ran on two c7g.large instances behind an NLB, averaging about 35% CPU. Not free, but the math worked out to roughly one-fifth of the backend savings.
The things we learned the hard way
Head sampling and tail sampling do not mix well
We tried running head sampling at 50% in the SDK and tail sampling in the collector as a "belt and braces" cost control. Then we spent an afternoon wondering why half our error traces were missing. Because the SDK had already dropped the trace before any span reached the collector, there was nothing to tail-sample. Error retention is only as good as the weakest sampler in the chain.
If you tail sample, set the SDK to AlwaysOn (or ParentBased(AlwaysOn)) and let the collector decide. If the SDK volume is too high to ship to the collector, scale the collector or add more of them — don't claw back cost by head sampling, because you'll silently degrade exactly the data you're paying to collect.
Latency-based policies need a sanity floor
Our first latency policy was threshold_ms: 500. On a service with a batch endpoint that legitimately takes 2–4 seconds, we were capturing 100% of batch traces and blowing the ingest budget. Either exclude those routes explicitly or raise the threshold. We now set per-route latency policies for the handful of endpoints where 500ms is pathological, and a looser global one at 800ms.
Async work breaks decision_wait
If a trace kicks off a background job via a queue and that job finishes 45 seconds later, the collector has already made its decision and flushed the parent trace. The late spans arrive as a separate, truncated trace. You can't fix this with a longer decision_wait without exploding memory. We mark long-running async spans with a trace.async=true attribute and query them as their own thing, which is uglier but honest.
Context propagation is still where bugs hide
Tail sampling only works if all spans for a trace reach the same collector instance. If you load-balance across collectors with round-robin, you'll split traces across instances and each one will make a partial decision. Use a load balancer that hashes on traceID — the collector has a built-in loadbalancing exporter for exactly this, and it's the piece most teams forget on day one.
exporters:
loadbalancing:
protocol:
otlp:
tls: { insecure: true }
resolver:
dns:
hostname: otel-tail-collector.internal
routing_key: traceID
Put that in a first-tier collector (which does no sampling), and point it at a pool of tail-sampling collectors behind the DNS name. Now every span for a given trace lands on the same tail collector.
When head sampling is still the right answer
If your service is doing 5,000+ RPS, or you're on a tight collector budget, or your trace backend is already cheap per-span (self-hosted Tempo on S3, for example), tail sampling may not pay for itself. A flat 1–5% head sample plus aggressive metric-based alerting works fine. You'll lose most error traces, but you'll have exemplars from your metrics pipeline pointing at specific trace IDs — and if your AlwaysOn sampler for a small percentage of users or tenants catches the rest, you've got enough to debug with.
The mistake is picking a sampling strategy once and never revisiting it. Traffic shape changes, backend pricing changes, the shape of your incidents changes. Re-tune every quarter at minimum.
Where we'd start
If you're setting this up from scratch on a service doing under 500 RPS, start with AlwaysOn in the SDK, a two-tier collector setup with loadbalancing in front of tail sampling, and three policies: keep all errors, keep anything over your P95 latency, and 5% baseline. Measure the ingest reduction for a week, then add route-specific policies for the endpoints that dominate your bill. Don't touch head sampling until you can prove the tail setup is the bottleneck. If you want a second pair of eyes on an observability setup that's drifting, our DevOps and cloud team does exactly this kind of review.
Want a team like ours?
72Technologies builds production software for the kind of teams who actually read this blog.
Start a projectKeep reading
Terraform State Locking Failed Mid-Apply: What We Learned About DynamoDB and S3 Backends
A half-applied Terraform run, a stuck DynamoDB lock, and a VPC that almost lost its NAT. Here's how state locking actually fails, and the backend hygiene we wish we'd had six months earlier.
CloudWatch to Grafana Cloud: What Our Logging Bill Actually Did
We moved 2.4 TB/month of application logs off CloudWatch Logs onto Grafana Cloud. The bill dropped, but not by as much as the slide decks promised, and two things broke in ways we didn't expect.
Vercel ISR vs On-Demand Revalidation: When Each One Actually Breaks
ISR and on-demand revalidation look interchangeable until traffic shifts or a CMS webhook misfires. Here's what we've learned shipping both patterns across e-commerce and content sites.
