Structured Outputs vs Tool Calls: Which One Should Your LLM Feature Use?

Every LLM feature eventually hits the same fork in the road: do you ask the model to emit a strict JSON object, or do you expose a set of tools and let it call them? Both approaches return typed data. Both are supported by every major provider. And yet picking the wrong one will cost you a week of refactoring and a surprising amount of latency.

We've shipped both patterns across extraction pipelines, chat copilots, and autonomous-ish agents. Here's the mental model we use when a client asks which one their feature should use.

The two mechanisms, briefly

Structured outputs and tool calls look similar on the wire — both hand you JSON that conforms to a schema — but they exist for different reasons.

Structured outputs (OpenAI's response_format: { type: "json_schema" }, Anthropic's tool-use-as-JSON pattern, Gemini's responseSchema) constrain the model's final answer to a schema. The model is always producing that one object. There's no branching, no choice, no side effects. It's a typed return value.

Tool calls (a.k.a. function calling) expose one or more named tools with input schemas. The model decides whether to call a tool, which one, and with what arguments. You then execute the tool and, usually, feed the result back for another turn. It's a typed function invocation, not a return value.

The distinction sounds pedantic until you're debugging why your extraction pipeline is randomly returning empty objects because the model "chose" not to call the tool.

When structured outputs are the right call

Use structured outputs when the model's job is to produce data, not to decide what to do.

Classic fits:

• Extracting fields from a document, email, or transcript
• Classifying an input into one of N enums
• Rewriting free text into a normalized shape (invoice line items, address components)
• Generating a UI payload for a fixed component tree
• Any endpoint where the answer is "always return an object like this"

With OpenAI's strict mode and Anthropic's tool-based JSON pattern, you get guaranteed schema conformance — the decoder is constrained during sampling, so you don't need a Zod retry loop wrapping every call. That alone removes a whole class of production incidents.

import OpenAI from "openai";
import { z } from "zod";
import { zodResponseFormat } from "openai/helpers/zod";

const Invoice = z.object({
  vendor: z.string(),
  total_cents: z.number().int(),
  currency: z.enum(["USD", "EUR", "GBP"]),
  line_items: z.array(z.object({
    description: z.string(),
    quantity: z.number().int(),
    unit_price_cents: z.number().int(),
  })),
});

const client = new OpenAI();
const res = await client.chat.completions.parse({
  model: "gpt-4.1-mini",
  messages: [
    { role: "system", content: "Extract invoice data." },
    { role: "user", content: rawInvoiceText },
  ],
  response_format: zodResponseFormat(Invoice, "invoice"),
});

const invoice = res.choices[0].message.parsed;

That's the entire integration. No try/catch around JSON.parse. No "the model added a preamble again" bug. If the schema is satisfiable, you get a valid object.

The trap: schemas that punish the model

Structured outputs constrain generation. If your schema has 40 required fields and half of them aren't in the source document, the model will hallucinate to satisfy the constraint. Make fields optional (or use nullable unions) when they might genuinely be absent, and the quality problem disappears.

When tool calls are the right call

Reach for tool calls when the model needs to choose — between actions, between data sources, or between doing something and doing nothing.

Good fits:

• A chat agent that can look up orders, issue refunds, or escalate to a human
• A RAG system where the model decides whether to search, and with what query
• Anything with side effects (writing to a DB, calling an external API)
• Multi-step workflows where the next step depends on the previous result

The key property is optionality. The model is picking from a menu, and "don't call anything, just answer" is often a valid choice. Structured outputs can't express that — they force a shape onto every response.

tools = [
    {
        "type": "function",
        "function": {
            "name": "search_orders",
            "description": "Search a customer's order history by date range or status.",
            "parameters": {
                "type": "object",
                "properties": {
                    "customer_id": {"type": "string"},
                    "status": {"type": "string", "enum": ["open", "shipped", "returned"]},
                },
                "required": ["customer_id"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "issue_refund",
            "description": "Refund an order. Requires human approval for amounts over $200.",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {"type": "string"},
                    "amount_cents": {"type": "integer"},
                    "reason": {"type": "string"},
                },
                "required": ["order_id", "amount_cents", "reason"],
            },
        },
    },
]

The model might call search_orders, then issue_refund, then produce a natural-language reply. Or it might just answer "your order shipped yesterday" without touching a tool. That branching is the whole point.

Force it when you have to

All three major providers let you force a specific tool call (tool_choice: { type: "function", name: "..." } on OpenAI, tool_choice: { type: "tool", name: "..." } on Anthropic). This is genuinely useful — but if you find yourself always forcing the same tool, you've written structured outputs the long way. Switch.

The blurry middle: single-tool "extractors"

Before strict structured outputs existed, the community's workaround was to define one tool, force the model to call it, and treat the arguments as the return value. Anthropic still recommends a variant of this pattern for reliable JSON with Claude, since their tool-use path is the most constrained decoding surface they expose.

That's fine. It's mechanically identical to structured outputs from your app's perspective. The rule of thumb:

• One tool, always called, no side effects → conceptually a structured output. Name your abstraction accordingly.
• Multiple tools, or optional invocation, or real side effects → genuinely a tool-call architecture.

Don't let the vendor's SDK surface trick you into thinking a single forced tool is somehow "agentic." It isn't. It's a typed function return.

Latency, cost, and failure modes

A few practical differences we've hit in production:

Latency. Structured outputs are one round trip. Tool-calling flows are at least two — model decides to call, you execute, model reads result and replies. For simple extractions, tool-calling doubles your p95 for no benefit.

Streaming. Structured outputs stream cleanly as partial JSON that you can incrementally parse (useful for populating a UI as fields arrive). Tool-call arguments also stream, but you can't act on them until the call is complete, and streaming through a multi-turn tool loop is a parser you'd rather not write from scratch.

Failure modes differ. Structured outputs fail by producing plausible-but-wrong values inside a valid schema. Tool calls fail by picking the wrong tool, or by hallucinating an argument that references a nonexistent ID. Your evals need to cover the right failure surface — see our note on evaluating LLM features if you're starting from scratch.

Cost. Tool schemas count against your input tokens on every turn. A sprawling tool catalog with verbose descriptions can quietly add 1–3k tokens per request. If you're only ever using two of ten tools in a given user journey, route to a smaller tool subset upstream.

A decision checklist

When a new feature lands on the backlog, we run through this in about two minutes:

1. Does the model need to choose between actions? → tool calls.
2. Are there side effects (writes, external calls, money movement)? → tool calls, with human-in-the-loop on the dangerous ones.
3. Is the answer always the same shape, always required? → structured outputs.
4. Is it a classification, extraction, or normalization task? → structured outputs.
5. Multi-step reasoning across data sources? → tool calls (probably with a planner).
6. "I want typed JSON back" and nothing else? → structured outputs. Stop overengineering.

One more thing: mix them. A tool-calling agent can have tools whose arguments are constrained by rich JSON schemas, and a structured-output pipeline can include an optional needs_human_review boolean that triggers a downstream workflow. These aren't mutually exclusive layers — they're different verbs for different jobs.

Where we'd start

If you're building something new tomorrow: default to structured outputs. Most "LLM features" are really extraction or generation tasks wearing an agent costume, and structured outputs will ship faster, cost less, and fail more predictably. Add tool calls the moment you have a real branching decision or a real side effect — and when you do, invest in evals for tool selection specifically, not just for the final answer. That's usually where the bugs hide.

If you want a hand designing the boundary for your product, our AI engineering team does this work all day.