Sentry Release Health Lied to Us: A Post-Mortem on Trusting Crash-Free Sessions

We shipped a release on a Tuesday afternoon. Sentry's release health dashboard glowed green: 99.7% crash-free sessions, no new issues, adoption climbing normally. Six hours later, support was on fire because roughly a quarter of returning users couldn't finish checkout — and none of it showed up as a crash.

This is the story of why we stopped using crash-free sessions as our primary deploy gate, and what we replaced it with.

What crash-free sessions actually measure

Sentry's release health is a genuinely useful feature, but the name oversells it. A "session" in Sentry is a period of user activity bounded by SDK lifecycle events. A session is marked as crashed when the SDK captures an unhandled error that terminates it, errored when a handled error occurs, and healthy otherwise. Crash-free sessions is the ratio of non-crashed sessions to total sessions.

The critical word there is unhandled. If your code catches an exception, logs it, shows a friendly toast, and returns early — that session is not crashed. It might be errored, depending on how you called captureException, but it will not drag down your crash-free rate.

On the browser, this gets worse. A lot of what breaks real user flows isn't a thrown exception at all:

• A fetch that returns a 500 and gets swallowed by a .catch that just sets loading = false
• A third-party script that fails to load, so a button silently does nothing
• A feature flag that resolves to the wrong variant and hides a form
• A React error boundary that renders a fallback (handled = not a crash)

All of those can tank conversion without moving the crash-free needle by a single basis point.

Our specific failure mode

The release in question changed how we called our payment intent endpoint. A refactor moved the Stripe key lookup from build-time env vars to a runtime config fetch. That config fetch worked fine for new sessions, but for users who had an older service worker cached, the request hit a stale route and returned an empty body.

Our code did roughly this:

try {
  const config = await fetchRuntimeConfig();
  const stripe = await loadStripe(config.stripePublishableKey);
  // ...
} catch (err) {
  Sentry.captureException(err, { level: 'warning' });
  setCheckoutError('Payment is temporarily unavailable. Please refresh.');
}

Notice three things. The exception was caught. It was reported at warning level, which Sentry does not count toward crash-free sessions by default. And the user got a message telling them to refresh — which, because the service worker was still stale, did nothing.

From Sentry's perspective, the release was healthy. From the user's perspective, checkout was broken.

Why the alerts didn't fire either

We had alerts. They were the standard ones you set up on day one and forget:

• Alert if crash-free sessions drop below 99% over 1 hour
• Alert if a new issue is seen more than 100 times in 10 minutes
• Alert on any regression in an existing resolved issue

The first didn't fire because crash-free sessions never dropped. The second didn't fire because the warning was grouped with an existing, previously-seen issue from an unrelated flow, and its rate was elevated but not extreme — checkout is a small fraction of overall traffic. The third didn't fire because the issue wasn't marked resolved.

The issue was visible in Sentry. It was sitting in the issues list, sorted by recency, with a rising events graph. Nobody was looking at it because the dashboard we had wired to Slack said everything was fine.

The lesson: crash-free sessions is a smoke detector, not a health check

Crash-free sessions is great at catching the class of bugs it was designed to catch: a new build that throws on load, a native crash on a specific OS version, a null reference in a hot path. It is nearly useless for silent business-logic failures, which in our experience are the majority of revenue-impacting incidents on mature codebases.

The deeper problem is that a single top-line metric encourages exactly the wrong operational habit. You end up treating a green number as evidence of health, when really it is evidence of the absence of one specific kind of unhealth.

What we changed

We kept Sentry. We changed what we trusted it to tell us, and we added signals it was never going to produce on its own.

1. Per-flow success rate as the real gate

For every business-critical flow — signup, checkout, subscription upgrade, file upload — we instrument the start and the successful completion as explicit spans, and we compute the ratio in our metrics backend. The deploy gate is now: for the top three flows, did completion rate stay within 2 percentage points of the trailing 24-hour baseline?

That number will move even if nothing throws. It moved sharply during the incident above, from about 91% to 67% within twenty minutes. If we had been watching it, we would have caught the regression before the first support ticket.

2. Alert on handled exceptions in critical paths

We tagged every captureException call in checkout, auth, and billing with a critical_path: true tag. Then we set a Sentry alert on event volume for that tag alone, regardless of grouping, regardless of level. If handled exceptions in a critical path double over 15 minutes compared to the previous 24 hours, someone gets paged.

Sentry.captureException(err, {
  level: 'warning',
  tags: { critical_path: 'checkout', flow: 'payment_intent' },
});

This is noisier than crash-free sessions and requires occasional tuning, but it fires on the failure mode we actually care about.

3. Progressive rollout tied to the flow metric, not release health

Our previous rollout used Sentry's release adoption and crash-free rate to auto-promote from 10% to 50% to 100% traffic. We now gate promotion on the per-flow success rate delta, checked at 15-minute intervals. Sentry release health is still displayed on the same dashboard, but it is advisory, not authoritative.

4. A weekly review of warning-level issues in critical paths

The issue that caused this incident was sitting in Sentry, visible, ignored. We now have a 20-minute Friday review where someone walks the list of critical_path tagged issues from the past week, regardless of level or resolution status. Roughly one in four of those reviews surfaces something worth fixing that no alert caught.

What we would tell our past selves

A few things worth internalising if you are running Sentry in production:

• Read the definition of every metric you gate on. Crash-free sessions is a specific technical measurement, not a general health score. The same goes for Apdex, Web Vitals pass rates, and anything else that gets rendered as a big green number.
• The most dangerous bug is the one your code handles gracefully but wrong. Try/catch that logs a warning and shows a fallback UI can hide a total feature outage. Instrument the success of the flow, not just the failure.
• Grouping is a feature and a liability. Sentry's fingerprinting collapses similar events, which is what keeps the issues list readable. It also means a new failure mode can hide inside an old, familiar issue. Use custom fingerprints on critical paths.
• Every top-line SLO needs a companion drill-down. If your only view is aggregate, you will miss regressions that affect a slice of users. Segment by release, region, device class, and — especially — by business flow.

None of this is a criticism of Sentry as a tool. Sentry did exactly what it says on the tin. The mistake was ours: we outsourced a judgement call about whether a release was healthy to a metric that was designed to answer a narrower question.

Where we'd start

If you're running Sentry today and using release health as a deploy signal, spend an afternoon doing this:

1. List your three most revenue-critical user flows.
2. For each one, add an explicit success event at the terminal step and a start event at the entry point. Emit them to whatever metrics backend you already have — Datadog, Grafana, CloudWatch, doesn't matter.
3. Build one chart: completion rate per flow, per release, over the last 24 hours.
4. Put that chart above your crash-free sessions chart on the deploy dashboard.
5. Move your rollout gate to the completion rate delta. Keep crash-free as a secondary signal.

If you want a second pair of eyes on your observability setup or your rollout pipeline, our team does this kind of work regularly — see our DevOps and cloud services for what that engagement looks like.