Streaming Tool Calls: How to Keep Agents Snappy Without Breaking Your Parser

Agents feel slow because most of the wall-clock time is spent waiting for a tool call argument that the model has already decided on. If you're not streaming those arguments, you're paying full generation latency before you can even start the tool. Streaming tool calls fixes that — but the parser you write matters more than the flag you flip.

This is a war-story-style walkthrough of how we ship streaming tool calls in production agents, what breaks, and what we do differently on Claude, OpenAI, and Gemini.

Why streaming tool calls is worth the complexity

A typical agent turn looks like this: the model emits a tool call, your runtime executes it, the result goes back, and the model continues. If the tool arguments are 400 tokens of JSON, you're waiting on the full generation before you can do anything useful. On a slow day that's 3 – 6 seconds of dead air.

With streaming, you can:

• Start warming caches or opening DB connections as soon as the tool name is known.
• Validate arguments incrementally and abort early when the model is clearly going off the rails.
• Show the user a "Searching invoices…" affordance the moment the tool name arrives, not after the arguments finish.
• Cancel a runaway generation mid-stream and save both tokens and seconds.

In our experience, streaming tool calls trims perceived latency by roughly 20 – 40% on argument-heavy tools, and materially more on chained agents where every hop compounds.

When it's not worth it

If your tool arguments are tiny (a single ID, a short query string), streaming buys you almost nothing and adds parser risk. Keep it simple and wait for the completed call. The rule of thumb: stream when arguments routinely exceed ~150 tokens or when the tool has meaningful setup cost.

The three provider shapes you'll deal with

Every major provider streams tool calls differently. You need to know the shape before you write the parser.

Anthropic Claude emits content_block_start, content_block_delta, and content_block_stop events. Tool arguments arrive as input_json_delta chunks — partial JSON strings you concatenate. Claude guarantees the final concatenation is valid JSON, but any intermediate slice almost certainly is not (see the Anthropic messages streaming docs).

OpenAI streams tool calls as deltas on choices[0].delta.tool_calls, with function.arguments arriving as string fragments. Multiple tool calls in parallel are distinguished by index. The arguments field is a raw string you must accumulate per index.

Google Gemini streams functionCall parts, but historically has been chunkier — you often get the full args object in one go rather than character-level deltas. That's simpler to parse but gives you less latency headroom.

The practical implication: your parser needs to handle partial JSON for Claude and OpenAI, and a whole-object-at-once path for Gemini. Don't pretend they're the same interface.

Partial JSON parsing without tears

The naive approach — JSON.parse on every delta — throws on every chunk until the very last one. That's fine if you only care about the final value, but then you've thrown away the whole point of streaming.

What you actually want is a tolerant parser that returns the best-effort structured value at each step. Two options:

1. Roll your own state machine. Tractable if your tool schemas are shallow. Painful once you have nested objects and arrays.
2. Use a partial JSON library. partial-json (npm) or json-stream-parser handle the common cases — unterminated strings, missing closing brackets, trailing commas.

Here's the pattern we use in a TypeScript agent runtime:

import { parse as parsePartial } from 'partial-json';

interface ToolCallState {
  id: string;
  name: string;
  argsBuffer: string;
  lastParsed: unknown;
  startedAt: number;
}

const calls = new Map<string, ToolCallState>();

function onDelta(index: number, id: string, name: string, argsChunk: string) {
  const key = `${index}:${id}`;
  let state = calls.get(key);
  if (!state) {
    state = { id, name, argsBuffer: '', lastParsed: null, startedAt: Date.now() };
    calls.set(key, state);
    onToolStart(name); // warm caches, show UI affordance
  }
  state.argsBuffer += argsChunk;
  try {
    const partial = parsePartial(state.argsBuffer);
    if (partial && !shallowEqual(partial, state.lastParsed)) {
      state.lastParsed = partial;
      onToolArgsProgress(name, partial);
    }
  } catch {
    // still not parseable — ignore, wait for more bytes
  }
}

The onToolArgsProgress callback is where the real wins live. If your tool is search_documents({ query, filters, limit }), you can kick off the search the moment query stabilises, even before limit arrives. In our RAG pipelines, we begin embedding the query string as soon as it appears complete (usually detectable by the parser transitioning from an open string to a closed one).

Guardrails for early execution

Executing on partial arguments is powerful and dangerous. Rules we enforce:

• Never execute side-effecting tools (writes, sends, payments) until the full call is received and validated against the schema.
• Only speculate on read-only, idempotent tools where a wasted call costs cents at most.
• Cap speculative executions per turn (we use 1) so a hallucinated tool storm can't fan out.
• Attach a cancellation token so you can abort the speculative call if the final arguments differ from what you speculated on.

Handling cancellation properly

Streaming means you can stop the model mid-generation. That's a feature — until you realise you've cancelled a stream while a tool call was half-parsed.

Our rule: cancellation is a first-class event, not an exception. When the user hits stop or a timeout fires, we:

1. Close the upstream provider stream.
2. Emit a synthetic tool_call_aborted event for every in-flight call.
3. Persist the partial state to the turn log so replays don't get confused.
4. Return a structured error to the caller — never a half-populated tool result.

Skipping step 3 is the source of the nastiest bug we've hit in this space: an agent replay loaded a half-parsed tool call from the log, tried to execute it, and hit a NOT NULL constraint on a field the model hadn't finished streaming.

Cost and observability

Streaming doesn't change token cost, but it changes what you can observe. Log these per turn:

• Time-to-first-tool-token (TTFT for tool calls specifically, not just text).
• Time between tool name arrival and full arguments.
• Parser attempts vs successes (a high failure ratio means your library is weak or the model is emitting weird escapes).
• Speculative-execution hit rate — how often did we speculate on args that matched the final call?

A speculative hit rate below ~70% means your parser is jumping too early. Above ~95% and you're probably being too conservative and leaving latency on the table.

A note on evals

Streaming introduces a whole class of failures that don't show up in offline evals, because offline evals typically operate on the completed response. Add streaming-specific eval cases: mid-stream cancellation, malformed deltas (inject them in a mock provider), and partial-arg race conditions. If you want the broader picture on agent evals, we wrote about that in our blog.

Provider-specific gotchas we've hit

• Claude occasionally emits input_json_delta chunks that split a Unicode escape sequence across two deltas. Buffer at the byte level, not the character level, or you'll corrupt strings.
• OpenAI can send tool_calls deltas interleaved with content deltas when the model narrates before calling. Your parser must not assume tool-call mode is exclusive.
• Gemini sometimes returns the entire functionCall in a single chunk even in streaming mode. Don't build UI that requires progressive updates — degrade gracefully to "appeared instantly."

Where we'd start

If you're adding streaming tool calls to an existing agent, do it in this order:

1. Add a tolerant partial-JSON parser and log parse attempts. Ship nothing user-visible yet.
2. Wire up onToolStart to render UI affordances the moment the tool name arrives. This alone is a huge perceived-latency win with near-zero risk.
3. Identify your top two read-only tools by frequency and add speculative execution behind a feature flag.
4. Instrument speculative hit rate and TTFT for tool calls. Tune thresholds.
5. Only then start considering speculative execution for anything with side effects — and honestly, we usually don't.

Streaming tool calls is one of those features that looks like a small optimisation and ends up reshaping how your agent feels. Get the parser right first, and the latency wins follow. If you want help wiring this into a production agent stack, that's the kind of work our team does on AI engagements.