Structured Outputs in 2026: JSON Mode, Tool Schemas, and When to Just Parse It Yourself

Every team that ships LLM features eventually hits the same wall: the model gives you mostly valid JSON, until the one request where it wraps the output in backticks, apologises mid-object, or hallucinates a field that breaks your Pydantic validator at 2am. Structured output features from OpenAI, Anthropic, and Google were supposed to end this. Mostly they have — but not always, and not for free.

This is a working guide to what each provider actually offers in 2026, where the sharp edges are, and when we still reach for a plain parser instead of the vendor's constrained decoding.

What "structured output" actually means now

There are three overlapping mechanisms, and vendors use the terms loosely:

1. JSON mode — the model is biased to produce syntactically valid JSON, but not against a specific schema. You still validate shape yourself.
2. Schema-constrained output — you pass a JSON Schema, and the decoder is constrained so the output cannot violate it. OpenAI calls this Structured Outputs with strict: true; Google calls it responseSchema on Gemini; Anthropic enforces this via tool-use schemas rather than a top-level response format.
3. Tool calling — the model emits arguments for a named function, validated against your schema. This is the dominant pattern for agents.

The third is where most production code lives, because it composes cleanly with retrieval, actions, and multi-step agents.

Why constrained decoding changes the calculus

When the provider enforces the schema at the sampling layer (grammar-constrained decoding), you stop needing retry loops for JSON parse errors. You still need semantic validation — the model can emit a schema-valid object that is factually wrong — but you can delete a surprising amount of defensive code. OpenAI's Structured Outputs documentation describes this as guaranteeing the output matches the supplied schema; in practice, that holds when you stay inside the subset of JSON Schema they support (no oneOf at the root, every property required, additionalProperties: false).

The three providers, honestly

OpenAI: strict mode is the default answer

OpenAI's response_format: { type: "json_schema", strict: true } is the strongest guarantee of the three major vendors. The constraints are real: supported types are a subset of JSON Schema, every object must declare all properties as required, and additionalProperties must be false. If you need an optional field, you model it as type: ["string", "null"].

from openai import OpenAI
client = OpenAI()

schema = {
    "name": "invoice_extraction",
    "strict": True,
    "schema": {
        "type": "object",
        "additionalProperties": False,
        "required": ["invoice_number", "total_cents", "currency", "line_items"],
        "properties": {
            "invoice_number": {"type": "string"},
            "total_cents": {"type": "integer"},
            "currency": {"type": "string", "enum": ["USD", "EUR", "GBP"]},
            "line_items": {
                "type": "array",
                "items": {
                    "type": "object",
                    "additionalProperties": False,
                    "required": ["description", "amount_cents"],
                    "properties": {
                        "description": {"type": "string"},
                        "amount_cents": {"type": "integer"}
                    }
                }
            }
        }
    }
}

resp = client.responses.create(
    model="gpt-5.1",
    input=[{"role": "user", "content": pdf_text}],
    response_format={"type": "json_schema", "json_schema": schema}
)

The first request against a new schema incurs a compilation step — in our experience, somewhere between a few hundred milliseconds and a couple of seconds of added latency. Subsequent calls with the same schema are cached. If you're A/B testing schemas, warm them during deploy.

Anthropic: everything is a tool

Claude doesn't have a separate JSON response format. The idiomatic way to get structured output is to define a tool and use tool_choice to force it. The schema is standard JSON Schema and Claude is extremely good at honouring it, though Anthropic's docs frame the guarantee more cautiously than OpenAI's — treat it as very reliable, not grammar-enforced.

message = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=2048,
    tools=[{
        "name": "record_invoice",
        "description": "Record a parsed invoice.",
        "input_schema": schema["schema"]
    }],
    tool_choice={"type": "tool", "name": "record_invoice"},
    messages=[{"role": "user", "content": pdf_text}]
)
parsed = message.content[0].input

The upside: tool schemas support richer constructs than OpenAI strict mode — unions, optional fields without null hacks, nested conditionals. The downside: no grammar guarantee, so keep a validator on the receiving end.

Gemini: responseSchema plus responseMimeType

Gemini supports responseMimeType: "application/json" and an optional responseSchema. The schema is a subset of OpenAPI 3.0 schema, which is close to but not identical to JSON Schema — format, enum, and property ordering via propertyOrdering are all honoured. Google's docs are explicit that property ordering influences generation quality, which is a nice detail most people miss.

Gemini's structured output is reliable for flat and moderately nested objects. For deeply recursive structures (think ASTs), we've seen it degrade faster than Claude or GPT.

When to skip all of this and just parse

Constrained decoding is not free. You pay in:

• Latency, especially the first call with a new schema.
• Expressiveness — strict modes forbid patterns you may actually need.
• Debuggability — when the model wants to say "I don't know" but the schema requires a value, it will fabricate one.

Hand-rolled parsing still wins in a few cases:

Case 1: single-field extraction

If the output is "one short string," don't bother with a schema. Ask for it, strip whitespace, validate with a regex. A JSON wrapper doubles your token count for no reliability gain.

Case 2: streaming UX with partial objects

If you're streaming structured output to a UI (think a form that fills in as the model thinks), you need to parse partial JSON. OpenAI's SDK exposes partial parses, but if you're on a vendor that doesn't, a tolerant parser like partial-json or your own bracket-counter will serve you better than waiting for a complete, schema-valid blob.

Case 3: when the schema itself is dynamic

Schema compilation overhead becomes painful if every request has a different schema (e.g., "extract these N fields the user just defined"). In that scenario, JSON mode without strict enforcement plus server-side validation often beats paying compile cost per request.

Patterns we actually use

Pattern: schema versioning

Schemas are code. Check them into the repo, give them a version string, and include the version in logs. When evals regress, the first question is "did the schema change?" and you want a one-line answer.

SCHEMAS = {
    "invoice.v3": {...},
    "invoice.v4": {...},
}

Pattern: nullable over optional

Under OpenAI strict mode, you can't mark fields optional — everything is required. Model "I didn't find this" as an explicit null with a sibling confidence field. This also forces the model to acknowledge missing data instead of hallucinating.

Pattern: enum your way out of free text

Any field with a bounded set of values should be an enum. We've seen hallucination rates on category fields drop sharply when switching from type: string with instructions to enum: [...]. The decoder physically cannot emit anything else.

Pattern: validate semantically, not just structurally

Schema-valid ≠ correct. A total_cents: 0 is valid JSON and probably wrong. Layer a semantic validator — a cheap second LLM call, a rules engine, or human review for high-stakes outputs. For our take on building that layer, see our notes on evals and LLM test suites.

Cost and latency notes

Schema-constrained output is roughly token-cost-neutral — you're not paying more per token, but you may generate more tokens because the model is forced to emit every required field. If you have ten optional-ish fields and the model usually only needs three, strict mode will cost you the other seven as null values. Not catastrophic, but it adds up across millions of calls.

For high-volume extraction pipelines, we often run a cheaper model (Haiku, Flash, GPT-5 mini) with a strict schema and reserve the frontier model for cases that fail semantic validation. That tiering routinely cuts costs 60–80% versus using the top-tier model everywhere.

Where we'd start

If you're adding structured output to a product today: pick one provider, define one schema in version control, enable strict mode or forced tool choice, and add a semantic validator behind it. Measure parse-failure rate and semantic-failure rate separately — they tell you different things. Only reach for multi-provider fallbacks once you've seen your primary fail in production, not before. The simplest pipeline that validates both syntax and meaning will beat a clever one that trusts the vendor to be perfect.

If you want a second pair of eyes on an extraction or agent pipeline you're building, our AI engineering team does exactly this kind of work.