Sentry Release Health Lied to Us: Debugging a Silent Crash Rate

Our Sentry dashboard said the new release had a 99.6% crash-free session rate. Support tickets said otherwise. For three days we argued with the graphs before we found the actual bug — and it wasn't in the product code. It was in how we'd wired up session tracking, and it had been quietly under-reporting crashes across two previous releases too.

This is the story of that debugging session, what the SDK was actually doing versus what we thought it was doing, and the config changes we now enforce on every project.

The Setup That Looked Fine

We run a React Native app with a Node.js backend on AWS, with Sentry wired into both. Release health had been enabled for months. The mobile team relied on the crash-free sessions and crash-free users metrics to gate releases — anything below 99.5% blocked promotion to the next rollout ring.

Release 4.12 shipped on a Tuesday. By Thursday morning, we had:

• 47 support tickets mentioning app freezes or force-closes
• App Store rating dropping from 4.6 to 4.3 in 48 hours
• A Sentry dashboard showing 99.63% crash-free sessions
• Issue counts in Sentry that looked roughly flat versus 4.11

That gap — real user pain versus a healthy-looking dashboard — is the worst kind of observability failure. A noisy dashboard you can tune. A quiet one that lies gives you false confidence.

The first wrong assumption

Our first instinct was that the tickets were unrelated: maybe a network issue, maybe a specific device model outside our tested matrix. We filtered Sentry by OS, device, and region. Nothing spiked. We even considered that support was miscategorising complaints.

That cost us about a day. Lesson: when qualitative signal (tickets, reviews, sales calls) disagrees with quantitative signal (dashboards), assume the dashboard is wrong until proven otherwise. Dashboards have config. Users don't.

What Release Health Actually Measures

Here's the thing we'd internalised wrong. Crash-free sessions is not "percentage of sessions where nothing bad happened." It's "percentage of tracked sessions that were not terminated by a crash the SDK recognised as a crash."

There are three quiet failure modes in that sentence:

1. Tracked sessions. If sessions aren't being started or ended correctly, the denominator is wrong.
2. Terminated by a crash. ANRs, freezes, and hangs on iOS/Android are not, by default, counted as crashes in every SDK version. They can be logged as issues without marking the session as crashed.
3. SDK recognised as a crash. Native crashes require the native layer to be initialised before the crash. JS errors caught by an error boundary and swallowed don't count.

We were hit by all three, in different proportions.

Reading the SDK init, line by line

When we finally sat down and read our Sentry.init call as if it were a stranger's code, the problem started to show:

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  release: `myapp@${version}+${buildNumber}`,
  environment: 'production',
  tracesSampleRate: 0.1,
  autoSessionTracking: true,
  enableAutoSessionTracking: true,
  sessionTrackingIntervalMillis: 30000,
  enableNdk: true,
  integrations: [
    new Sentry.ReactNativeTracing({
      routingInstrumentation,
    }),
  ],
  beforeSend(event) {
    if (event.exception && isKnownNoise(event)) {
      return null;
    }
    return event;
  },
});

Three problems, all subtle:

• beforeSend returned null for events we'd classified as noise months earlier. That filter had grown to cover a specific native module error that was, in fact, causing real hangs on Android 13+. We were dropping the very events that would have marked sessions as crashed.
• sessionTrackingIntervalMillis: 30000 meant a session was considered ended after 30 seconds in the background. Users who backgrounded the app during a crash-triggering flow and returned later were starting fresh sessions with no crash attached. The crash happened, but not "in a session" from Sentry's perspective.
• ANR detection was on, but we hadn't enabled enableAppHangTracking on iOS. Freezes on iPhones were being reported as issues but not counted against release health.

The headline number was doing exactly what we'd configured it to do. It just wasn't measuring what we thought.

The Debugging Loop That Worked

Once we stopped trusting the summary metric, the process got faster. We treated Sentry itself as a system under test.

Step 1: Cross-reference with a source of truth

We pulled crash counts from three places for the same 24-hour window:

• Sentry issues (grouped by release)
• Google Play Console → Android vitals → Crash rate
• App Store Connect → Crashes

Google Play reported roughly 3.1% of sessions crashing on Android 13 devices for 4.12. Sentry reported 0.4% for the same slice. That 8x gap was the smoking gun.

In our experience, Play Console and App Store Connect are the closest thing to ground truth for mobile crashes, because they measure at the OS level and don't depend on your SDK being alive at the moment of the crash. Always keep them in the loop, even if your team lives in Sentry.

Step 2: Instrument the instrumentation

We added a debug build variant that logged every session start, session end, and event submission to a local file, then diffed those against what actually arrived in Sentry. About 12% of events we tried to send were being dropped by beforeSend. Of those, roughly a third were the Android 13 native module error.

Step 3: Fix and re-baseline

We made four changes:

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  release: `myapp@${version}+${buildNumber}`,
  environment: 'production',
  tracesSampleRate: 0.1,
  autoSessionTracking: true,
  sessionTrackingIntervalMillis: 5000,
  enableNdk: true,
  enableAppHangTracking: true,
  appHangTimeoutInterval: 2,
  enableAutoPerformanceTracing: true,
  integrations: [
    new Sentry.ReactNativeTracing({ routingInstrumentation }),
  ],
  beforeSend(event, hint) {
    if (event.level === 'fatal' || hint?.originalException) {
      return event;
    }
    if (event.exception && isKnownNoise(event)) {
      return null;
    }
    return event;
  },
});

The key changes: beforeSend never drops fatal events or events with an original exception. Session interval dropped to 5 seconds so foreground-background-crash flows stayed attached to the correct session. App hang tracking on with a 2-second threshold. And we deleted the isKnownNoise rule that was masking the real bug — turns out it was never really noise.

After shipping 4.12.1 with these changes, the reported crash-free session rate dropped to 96.8%. That was the honest number. It felt worse. It was better.

What We Enforce Now

A few things stuck as team standards after this incident:

• No beforeSend filter goes in without a review and a comment linking to why. Filters accrue. Every six months, we re-justify each one or delete it.
• Release health metrics are cross-checked against store-provided crash data on every release. If they disagree by more than 2x, we treat the SDK config as suspect, not the store.
• Session tracking intervals are documented per platform with the reasoning. 5 seconds isn't a magic number — it's the number that fit our user's typical background-return pattern. Yours may differ.
• App hang and ANR tracking are on by default for any new mobile project we ship.
• We alert on the ratio between Sentry-reported issues and total sessions, not just on absolute issue counts. A dashboard that goes suspiciously quiet is as much a signal as one that spikes.

If you're using Sentry across a fleet of services, the same principle applies to backend release health: session semantics are different (typically request- or process-scoped), but the trap is identical. The number you see is a function of your init config, not a physical measurement.

Where We'd Start

If you're reading this because your release health numbers feel too good, do three things this week. Pull the last 30 days of crash data from your app stores and diff it against Sentry's crash-free sessions rate for the same window — any gap over 2x is worth a Friday afternoon. Print out your Sentry.init call and read every option aloud in a room with one other engineer; the ones you can't explain are the ones lying to you. And add store-console data as a required check on your release checklist, right next to the Sentry dashboard, so a single source can never quietly drift.

We help teams wire up honest observability across mobile, web, and backend as part of our DevOps and reliability work. If your dashboards feel a little too green this quarter, that's usually where we'd start looking.