Streaming Tool Calls: Making Agents Feel Fast Without Lying to Users

Users don't judge your agent by tokens per second. They judge it by whether something on the screen moved in the first 400ms, and whether the thing that moved was honest. Streaming tool calls done well is the difference between an agent that feels alive and one that feels like a loading spinner with a personality.

This is the pattern we keep landing on across Claude, GPT, and Gemini deployments — what to stream, what to hide, and how to not paint yourself into a corner when the model changes its mind mid-response.

Why streaming tool calls is harder than streaming text

Streaming a chat completion is easy: chunks come in, you concatenate, you render. Streaming a tool-calling agent is not that. A single user turn might produce:

1. A short thinking preamble (text)
2. A tool call with partial JSON arguments
3. A tool result (from your backend)
4. Another tool call
5. Final text answer

Each step arrives as a stream of deltas, and the model can interleave text and tool calls in the same turn. If you naively render every delta, you'll show users half-formed JSON, tool names that get renamed a token later, or a "final answer" that turns out to be step 2 of 5.

The three big vendors expose this differently:

• OpenAI Responses API streams response.output_text.delta, response.function_call_arguments.delta, and lifecycle events like response.output_item.added (see OpenAI's streaming docs).
• Anthropic Messages API streams content_block_start, content_block_delta (with input_json_delta for tool inputs), and content_block_stop per block.
• Google Gemini streams GenerateContentResponse chunks with parts that may contain text or functionCall.

The shapes differ, but the mental model is the same: blocks with lifecycles, not a flat token stream.

The four things worth streaming to the UI

We've tried streaming everything. It's a mess. These are the only signals that consistently improve perceived quality:

1. Intent, as soon as the first tool name is known

The moment you know the model is calling search_orders, tell the user something honest: "Looking up your recent orders…" Not the raw tool name. Not the JSON. Just the intent, mapped from a small dictionary you control.

const TOOL_LABELS: Record<string, string> = {
  search_orders: 'Looking up your recent orders',
  get_invoice: 'Fetching the invoice',
  refund_order: 'Preparing a refund',
};

This is the single highest-leverage change we make on new agent projects. It cuts perceived latency more than any model swap.

2. Progressive text, but only from the final assistant block

Don't stream reasoning preambles as if they're the answer. If the model emits "Let me check that for you" before the tool call, either suppress it or render it in a de-emphasized style. Stream tokens verbatim only once you're in the post-tool-result assistant block.

3. Tool status transitions

Three states, no more: calling, succeeded, failed. Users understand these. "Executing function with arguments {...}" is not for humans.

4. Structured artifacts, when the schema stabilises

If your tool returns a table, a chart, or a card, render a skeleton the moment the tool call starts and hydrate it when the result lands. Don't try to stream the tool's JSON output field-by-field unless you have a genuine reason — it's fragile and rarely worth it.

Handling partial JSON without exploding

Tool argument streaming is where most teams get burned. The model emits arguments as a JSON string, one delta at a time:

{"quer
{"query": "ref
{"query": "refund for order 88
{"query": "refund for order 8821"}

If you JSON.parse on every delta, you'll throw on 90% of them. Three options, in order of how much we like them:

Option A: Don't parse until the block closes. The model finishes the tool call, you get a content_block_stop (Anthropic) or response.function_call_arguments.done (OpenAI), and then you parse. Simple, boring, correct. Use this unless you have a reason not to.

Option B: Use a partial JSON parser. Libraries like partial-json or best-effort-json-parser will happily give you { query: "refund for order 88" } mid-stream. Useful if you want to render a search box or a filter chip live as the model types the query.

Option C: Structured streaming via schema. Both OpenAI (response_format with JSON schema) and Gemini (responseSchema) constrain output. Combined with a partial parser, you get reliable field-by-field hydration. This is what we use for form-filling agents.

import { parse } from 'partial-json';

let buffer = '';
for await (const event of stream) {
  if (event.type === 'response.function_call_arguments.delta') {
    buffer += event.delta;
    try {
      const partial = parse(buffer);
      if (partial?.query) updateSearchChip(partial.query);
    } catch {
      // still incomplete, that's fine
    }
  }
}

The retry problem nobody warns you about

Streaming and retries fight each other. If you've already streamed 200 tokens to the user and the connection drops, what do you do? Three patterns, each with a tradeoff:

• Restart from scratch, hide the previous partial. Cleanest. Wastes tokens. Users see a brief flicker.
• Buffer server-side, replay on reconnect. Requires a session store (Redis, Durable Objects). Best UX, most infrastructure.
• Idempotent tool calls + resume. If your tools are idempotent, you can re-run the turn and dedupe. Elegant but requires discipline about tool design.

We default to pattern two on anything customer-facing. A small Redis-backed event log keyed by turn_id, with a 10-minute TTL, is enough. The client sends Last-Event-ID on reconnect (standard SSE), and the server replays.

Don't stream past a tool call boundary you can't undo

The worst bug we've shipped in this space: the model streamed "I've cancelled your subscription" before the cancel_subscription tool actually returned. The tool failed. We had already told the user it worked.

Rule: never stream text that makes a claim about a tool result until the tool has returned successfully. Enforce this in your prompt ("only describe actions in past tense after the tool result is present") and by chunking the assistant response into pre-action and post-action segments. If you can't guarantee it via prompting, add a validator that scans streamed text for action verbs before releasing the chunk.

Cost and latency: what streaming actually buys you

Streaming doesn't make your agent cheaper. It makes it feel faster while costing roughly the same. In our experience:

• Time-to-first-token (TTFT) matters more than total tokens/sec for perceived quality up to about the 2-second mark.
• Streaming tool calls adds ~50–150ms of overhead per turn versus buffered responses, mostly from SSE framing and client rehydration. Worth it.
• The real cost lever is turn count, not streaming. An agent that resolves in one tool call beats one that streams beautifully across five.

If you're picking a model partly for streaming behaviour: Claude's input_json_delta events are the most granular for tool arguments, GPT's Responses API has the cleanest lifecycle semantics, and Gemini streams fastest on short turns in our tests but with coarser chunking. Check the current vendor docs before committing — this changes every few months.

A minimal server pattern that scales

On the server, we treat each turn as a state machine emitting typed events over SSE. The client renders based on event type, not raw model output.

type AgentEvent =
  | { type: 'intent'; label: string }
  | { type: 'text_delta'; text: string }
  | { type: 'tool_start'; id: string; label: string }
  | { type: 'tool_end'; id: string; ok: boolean }
  | { type: 'artifact'; id: string; data: unknown }
  | { type: 'done' }
  | { type: 'error'; message: string };

That's it. The vendor-specific stream parser lives in one module and emits these events. The rest of the system — logging, evals, UI — talks only to AgentEvent. When a new model version breaks its streaming format, you fix one file.

Where we'd start

If you're adding streaming to an existing agent, don't try to stream everything on day one. Ship in this order:

1. Intent labels on tool start. One dictionary, one event type. Biggest UX win per hour of work.
2. Buffered tool arguments, streamed final text. Parse arguments only on block close. Stream the assistant's final answer token-by-token.
3. Server-side replay for reconnects. Redis, turn_id, 10-minute TTL. Turn on before you hit real traffic.
4. Partial JSON parsing for artifacts. Only for surfaces where progressive hydration genuinely helps — forms, search filters, tables.
5. Post-action gating. A validator that prevents streaming claims about tool results until the tool returns.

Skip step 4 until you've done 1–3. Most teams jump straight to partial parsing, wire up something clever, and ship an agent that lies confidently at 60 tokens per second. Don't be that team.

If you want help designing the agent boundary — tools, evals, streaming contract — that's a lot of what we do on the AI engineering side.