Structured Outputs vs Tool Calls: Which One Should Your LLM Actually Use?

Every few months a client asks us the same question in a slightly different form: should this endpoint return a JSON object from the model, or should we let the model call a function? On paper the two features look like siblings. In production they behave very differently, and picking the wrong one bakes latency, retries, and weird edge cases into your stack for months.

This is the mental model we use on real projects, plus the specific cases where each one shines and the ones where they quietly hurt you.

The two features, honestly described

Both OpenAI, Anthropic, and Google now offer a way to constrain a model's output to a schema. OpenAI calls it Structured Outputs (backed by a constrained decoder), Anthropic documents JSON output via tool use in the Claude tool use guide, and Google exposes responseSchema in Gemini's controlled generation. Tool calling, meanwhile, lets the model emit a request to invoke a named function with typed arguments — the model chooses whether and when to call it, and your code executes the actual work.

Here's the part most tutorials skip: structured outputs are a way to shape the model's final answer. Tool calls are a way to shape a mid-conversation intent. That single distinction resolves 80% of the design arguments we have on kickoff calls.

A quick concrete example

Say you're building an intake form for a legal ops tool. The user pastes a contract clause and you need { risk_level, rationale, suggested_edit } back.

{
  "type": "object",
  "properties": {
    "risk_level": { "enum": ["low", "medium", "high"] },
    "rationale": { "type": "string" },
    "suggested_edit": { "type": "string" }
  },
  "required": ["risk_level", "rationale", "suggested_edit"],
  "additionalProperties": false
}

That's a structured output. There's no side effect, no external system, no decision the model needs to make about whether to respond. It just needs to answer in a shape your frontend can render.

Now imagine the same product also needs to pull the client's prior contracts from your vector store before answering. That retrieval is a tool call. The model has to decide: do I have enough context, or do I need to fetch more?

When to reach for structured outputs

We default to structured outputs whenever all three of these are true:

• The model has everything it needs already (from the prompt, RAG context, or prior turns)
• The output goes straight into a UI, a database row, or a downstream deterministic pipeline
• There is exactly one shape of answer

Extraction, classification, summarisation with fields, form autofill, moderation labels, code diff generation with metadata — all of these live comfortably here. The wins are real:

1. No parsing gymnastics. With OpenAI's strict: true mode or Gemini's responseSchema, the decoder enforces the grammar. You stop writing defensive try/except json.loads code.
2. Lower latency. One round trip, no tool-execution hop, no second model call to summarise a tool result.
3. Cheaper. You pay for one completion, not two or three.

The trap: engineers keep piling optional fields into the schema until the model is essentially doing routing inside a single JSON object (action_type: "search" | "answer" | "escalate", then a giant conditional payload). At that point you've reinvented tool calls, badly, because now your code has to branch on a string field the model chose freely.

The 'discriminated union' smell

If your schema has a type field and half the other fields only make sense for certain values of type, stop. That's a tool call wearing a costume. Split it.

When tool calls earn their keep

Tool calls are the right answer when the model needs to decide — between actions, between fetching more data or answering, between calling one API or another. Concretely:

• Retrieval that isn't always needed. Cheap queries answered from context, expensive ones triggering a vector search or SQL lookup.
• Multi-step workflows. Book a meeting → check calendar → check the other calendar → propose slots.
• Guarded side effects. Sending an email, writing to a CRM, charging a card. You want the model to propose the action so your code can validate, log, and require confirmation before executing.
• Parallel work. All three major vendors now support parallel tool calls, which is genuinely useful when the model needs to hit multiple independent lookups.

The reason to route decisions through tool calls rather than a free-form action field is that the tool-calling path gets first-class treatment in the training data and in the API. Retries, streaming, and audit logs all cleanly separate the model's intent from the system's execution.

tools = [{
    "name": "search_contracts",
    "description": "Search the client's historical contracts. Use only when the current clause references prior agreements or the user asks about precedent.",
    "input_schema": {
        "type": "object",
        "properties": {
            "query": {"type": "string"},
            "client_id": {"type": "string"},
            "top_k": {"type": "integer", "minimum": 1, "maximum": 10}
        },
        "required": ["query", "client_id"]
    }
}]

Notice the description does real work. Tool descriptions are prompts. If your model is calling the tool too often (or not enough), the fix is almost always in the description, not the temperature.

The hybrid pattern we use most often

Most real endpoints end up being tool calls until the model is ready, then a structured output for the final answer. This is how we build agent-shaped features without the agent chaos.

The loop looks like this:

1. First turn: model gets the user query, a set of tools, and a system prompt that ends with something like "When you have enough information, respond with your final answer using the submit_answer tool."
2. Middle turns: model calls retrieval or lookup tools, you execute them, feed results back.
3. Final turn: model calls the submit_answer tool, whose input schema is your structured output shape.

Why wrap the final answer in a tool call instead of just letting the model emit JSON at the end? Because it gives you a single, unambiguous signal that the loop is done. No parsing the assistant message to guess whether it's a tool call or the final reply. The model either called submit_answer or it called something else and the loop continues.

This pattern also gives you a clean place to run output validation and, if you want, a cheap eval hook — every final answer flows through one function.

The failure modes nobody warns you about

Structured outputs make hallucinations more confident. If the schema demands a citation_url and the model doesn't have one, it will make one up rather than fail validation. Add nullable: true and explicit instructions to leave fields empty when unknown. Better: return citations: [] and let downstream code decide what to do with zero results.

Tool call loops. We've written about the death spiral separately on our blog, but the short version: cap the number of tool calls per turn, require the model to justify repeat calls, and treat identical consecutive calls as a bug signal.

Schema drift between vendors. OpenAI's strict mode disallows additionalProperties and requires every property in required. Gemini is looser but has its own quirks with anyOf. Claude accepts standard JSON Schema through its tool interface. If you're building a model-agnostic layer, write your schemas to the strictest common subset or you'll spend a sprint chasing per-vendor edge cases.

Token cost of schemas. A large schema is repeated in every request. If your responseSchema is 800 tokens and you're serving thousands of requests, that adds up fast. Prompt caching helps (where supported), but the honest fix is to keep schemas lean.

A decision cheatsheet

Situation	Reach for
Extract fields from a document	Structured output
Classify with a fixed label set	Structured output
Model might need to look something up	Tool calls
Any real-world side effect	Tool calls
Multi-step reasoning with external data	Tool calls, submit_answer at the end
Streaming a long-form answer with metadata	Structured output with streamed fields
You have three optional payload shapes	Split into three tools

Where we'd start

On a new project, we build the simplest structured-output version first, even for problems we suspect will need tools. It forces us to be honest about what context the model actually needs — and half the time, once RAG is doing its job, the tool-calling layer never gets built. When it does, we add one tool at a time, each with a description that reads like an instruction to a junior engineer, and we always wrap the terminal answer in a submit_answer tool so the control flow stays boring. Boring control flow is the whole game. If you'd like a hand designing this layer for a specific product, our AI engineering services page is the right place to start.