Streaming Structured Output: Why Your JSON Parser Is Blocking the UI

Streaming works beautifully when you're rendering prose. Tokens arrive, you append them to a <div>, and the user feels a fast product. Then someone asks you to stream a JSON response into a live UI — a table, a form, a card list — and the whole illusion collapses. JSON.parse throws on every chunk. You buffer until the end. Your "streaming" product now feels slower than the non-streaming one.

This is one of the most common LLM UX bugs we see in code reviews. Below is the pattern we use to fix it, plus the tradeoffs across Claude, OpenAI, and Gemini.

Why standard JSON parsing breaks on streams

JSON.parse is all-or-nothing. It needs a complete, syntactically valid document. A stream of tokens from an LLM gives you fragments like:

{"items": [{"name": "Wid
{"items": [{"name": "Widget", "pri
{"items": [{"name": "Widget", "price": 12

Each of those throws. So most teams do one of three things:

1. Wait for the full response, defeating the point of streaming.
2. Regex-hack the partial string to find complete objects, which works until a value contains a } or an escaped quote.
3. Ask the model to emit newline-delimited JSON (NDJSON), one object per line, and parse line by line.

Option 3 is genuinely good when your schema is a flat list. It falls apart when you have nested structures, or when you want the UI to show a half-built object filling in field by field.

For the general case, you need an incremental JSON parser — one that accepts partial input and returns the best-effort valid subtree so far.

The incremental parsing pattern

The core idea: on every chunk, take the accumulated buffer, attempt to "close" any open brackets, quotes, and commas, then parse the repaired string. Libraries like partial-json, best-effort-json-parser, and jsonrepair all do variants of this. You can also write it yourself in about 150 lines if you only care about a subset of JSON.

Here's the shape of the loop we use in a React/Next.js app talking to Claude:

import { parse } from 'partial-json';

async function streamStructured<T>(
  response: Response,
  onUpdate: (partial: Partial<T>) => void,
) {
  const reader = response.body!.getReader();
  const decoder = new TextDecoder();
  let buffer = '';

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    buffer += decoder.decode(value, { stream: true });

    // Extract just the JSON payload from SSE frames, tool_use
    // blocks, or whatever wrapper your provider uses.
    const jsonSlice = extractJsonSlice(buffer);
    if (!jsonSlice) continue;

    try {
      const partial = parse(jsonSlice) as Partial<T>;
      onUpdate(partial);
    } catch {
      // Not yet parseable even with repair; wait for more.
    }
  }
}

Two things matter here.

First, extractJsonSlice is provider-specific. If you're using Anthropic's tool_use streaming, the JSON arrives inside input_json_delta events and you concatenate the partial_json fields (see Anthropic's streaming docs). OpenAI's Responses API emits response.output_text.delta events, or response.function_call_arguments.delta for tool calls. Gemini streams functionCall.args progressively via generateContentStream.

Second, onUpdate should be idempotent. Every call replaces the current state — never appends. This is what lets the UI cleanly re-render a growing tree.

Making the UI tolerant of half-baked data

Your renderer now receives objects that mutate on every chunk. Fields appear, strings grow character by character, arrays gain items. A few rules that save pain:

• Never destructure with required fields. Use optional chaining everywhere.
• Key list items by a stable id if one exists, otherwise by index. Index keying is fine here because items only get appended, never reordered mid-stream.
• Render placeholders for missing fields, not empty strings. An undefined price should show a skeleton, not $0.
• Debounce expensive renders. If your update rate hits 50/sec on a large tree, batch with requestAnimationFrame or a 30 – 50ms throttle.

Schema validation without killing the stream

If you're using Zod, Pydantic, or the provider's native structured output mode, don't validate the partial. Validate only when the stream completes. Partial data will fail required-field checks every time.

What we do instead:

const PartialSchema = FullSchema.deepPartial();

onUpdate: (partial) => {
  const safe = PartialSchema.safeParse(partial);
  if (safe.success) setState(safe.data);
},

onComplete: (full) => {
  const result = FullSchema.safeParse(full);
  if (!result.success) handleValidationFailure(result.error);
},

The deepPartial trick gives you type safety in the UI without rejecting mid-stream data. At the end, you run the strict schema and decide what to do on failure — retry, repair with a follow-up call, or surface an error.

Provider-specific gotchas

All three major providers stream structured output, but the mechanics differ enough to matter.

Anthropic (Claude)

Claude streams tool inputs as input_json_delta events. The partial_json string is a raw fragment — you concatenate them in order and parse the accumulated string. Anthropic's docs note that the JSON may be split at arbitrary character boundaries, including inside a UTF-8 codepoint, so decode as a stream ({ stream: true } on TextDecoder) rather than per-chunk.

Claude's tool-use path is generally the most reliable way to get structured output in a stream. Ask for prose JSON in a plain text response and you'll occasionally get markdown fences you have to strip.

OpenAI

OpenAI's structured outputs (response_format: { type: 'json_schema' }) guarantee schema conformance on the final response, per OpenAI's documentation. Streaming still emits partials though — you'll see response.output_text.delta for text or response.function_call_arguments.delta for tool calls. The guarantee applies to the completed message, not to intermediate chunks, so your parser still needs to tolerate garbage mid-stream.

One quirk: with strict JSON schema mode, OpenAI's server-side validation can occasionally retry internally, which shows up as a brief pause in the stream. Budget for it.

Google (Gemini)

Gemini supports responseSchema on generateContentStream, and streams functionCall.args as a growing object. In our experience the chunks arrive in larger, less frequent batches than Claude or OpenAI — fewer updates, but each one is closer to a complete subtree. That's actually easier to render, at the cost of the stream feeling less "live".

When streaming structured output is the wrong tool

A few cases where we tell clients to skip this pattern entirely:

• Short responses (under ~500 tokens). The time-to-first-token savings don't justify the complexity. Just wait for the full response.
• Responses that drive side effects. If the JSON is going straight into a database write or an API call, stream nothing. Wait for validation.
• Deeply nested schemas with strict business logic. Partial rendering will show impossible intermediate states (an order with items but no customer). Users read that as a bug.

Streaming structured output shines in three places: chat UIs that render rich cards, generative form-filling where users watch fields populate, and agent traces where you want to show the plan as it forms.

Measuring whether it actually helped

Don't ship this without measuring. The metrics we track:

• Time to first meaningful paint — when the first parseable partial hits the UI, not the first token.
• Update frequency — chunks per second. Below 2/sec feels choppy; above 30/sec you're wasting CPU.
• Parse failure rate — how often the incremental parser can't recover. Should be under 5% of chunks after the first parseable one.
• Final validation pass rate — how often the completed response satisfies the strict schema. If this drops below 98%, your prompt or schema needs work, not your parser.

We've had cases where switching to streamed structured output cut perceived latency by more than half, and cases where it made things feel worse because the UI thrashed. The metrics tell you which one you're in.

Where we'd start

If you're adding this to an existing product, do it in this order:

1. Pick one screen where the response is long enough to matter and the schema is flat enough to render safely.
2. Add an incremental parser (partial-json is a fine default) behind a feature flag.
3. Split your Zod/Pydantic schema into a deepPartial for the stream and a strict version for the final validation.
4. Instrument the four metrics above before you touch the UI.
5. Only then decide whether to roll it out further, or whether the non-streaming version was quietly fine.

If you want a hand wiring this into a production app, our AI engineering team does this kind of work weekly — and most of the wins come from knowing when not to stream.