Structured Outputs in Production: JSON Mode, Tool Schemas, and When to Just Parse Text

Every team building on LLMs eventually hits the same wall: the model gives you almost the right JSON. A trailing comma, a hallucinated field, a markdown fence wrapped around the payload. You write a regex. Then another. Then you discover your PM shipped a demo where the parser silently dropped 8% of records.

Structured outputs are the fix — but there are now three or four competing mechanisms across vendors, each with different guarantees. This is how we choose between them at 72Technologies, and the cases where we still, deliberately, parse plain text.

The three mechanisms, honestly compared

There are essentially three families of "make the model return structured data" features in 2026:

1. Loose JSON mode — the model is nudged to produce JSON. No schema enforcement. Old OpenAI response_format: { type: "json_object" } was this.
2. Strict schema enforcement — the decoder is constrained so the output provably matches a schema. OpenAI's Structured Outputs with strict: true, Gemini's responseSchema with responseMimeType: application/json, and grammar-based decoders (llama.cpp GBNF, Outlines) all live here.
3. Tool / function calling — the model emits arguments for a named tool, typed against a JSON Schema. Under the hood on modern OpenAI and Anthropic APIs, this is often the same constrained-decoding machinery, just wrapped in a different UX.

The distinction that matters in production is enforcement vs suggestion. Loose JSON mode will still hallucinate keys. Strict modes will not — but they come with their own tax.

What "strict" actually costs

Constrained decoding narrows the token distribution at each step to only tokens that keep the output valid against the grammar. That has real consequences:

• Latency on first token can spike the first time a schema is seen, because the vendor compiles it. OpenAI documents this and caches compiled schemas afterwards (see the Structured Outputs guide).
• Quality can degrade if your schema is over-specified. A model forced to emit "category": "A" | "B" | "C" when the right answer is genuinely "none of these" will pick one anyway. Enforcement doesn't create knowledge; it just constrains form.
• Refusals get weird. With strict schemas the model can't say "I don't know" unless you gave it a field for that. Always include an unknown enum value or a nullable notes field.

Our default: tool schemas, not raw JSON mode

For anything more complex than "extract these five fields," we reach for tool calling with a JSON Schema, even when there's only one tool and we don't actually plan to execute it.

Why? Three reasons:

• Tool schemas are first-class across Claude, GPT, and Gemini, with consistent request/response shapes.
• The models are trained heavily on tool-use traces, so quality inside a tool call is often better than free-form JSON of the same shape.
• You get a natural place for the model to emit reasoning before the structured payload — the assistant message content — without polluting the JSON itself.

Here's the pattern we use for extraction tasks:

const tools = [
  {
    type: "function",
    function: {
      name: "record_invoice",
      description: "Record a parsed invoice. Call exactly once.",
      strict: true,
      parameters: {
        type: "object",
        additionalProperties: false,
        required: ["vendor", "total_cents", "currency", "line_items", "confidence"],
        properties: {
          vendor: { type: "string" },
          total_cents: { type: "integer", minimum: 0 },
          currency: { type: "string", enum: ["USD", "EUR", "GBP", "unknown"] },
          line_items: {
            type: "array",
            items: {
              type: "object",
              additionalProperties: false,
              required: ["description", "amount_cents"],
              properties: {
                description: { type: "string" },
                amount_cents: { type: "integer" }
              }
            }
          },
          confidence: { type: "string", enum: ["high", "medium", "low"] }
        }
      }
    }
  }
];

Notes on the details, because they matter:

• additionalProperties: false and required listing every property are mandatory for OpenAI's strict mode. Miss one and the API rejects the schema at request time, which is actually the failure mode you want.
• total_cents as an integer avoids the classic float-rounding disaster on money. Never let the model emit total: 19.99.
• The confidence enum gives the model a socially acceptable way to say "I'm guessing." Downstream, anything below high routes to human review.
• The unknown currency value is the escape hatch. Without it, the model will confidently assert USD on a Polish invoice.

When plain-text parsing is still the right call

Contrarian take: constrained decoding is not always the right answer.

We've shipped features where we deliberately let the model return prose and parsed it ourselves. The cases:

Streaming UX where structure would block rendering

If the user is watching text appear token-by-token, a strict JSON schema means they see nothing until the object is well-formed enough to parse. For a chat summarization sidebar, that felt worse than parsing headings out of markdown as they arrived. We used a simple contract: ## Summary, ## Action items, ## Risks. Regex over streamed chunks. Total parser code: 30 lines.

Very long outputs where schemas hurt quality

On one project we tried forcing a 2,000-word structured report into a nested JSON schema. Quality tanked — the model kept truncating sections to stay within array shape expectations. Switching to markdown output with a post-hoc extraction pass (a cheaper model called in a second step to structure the result) was both cheaper and better.

When the schema changes per request

Dynamic schemas defeat vendor-side caching. If you're generating a fresh JSON Schema on every call, the compile-time overhead adds up. In those cases we sometimes prompt for JSON, validate with Zod or Pydantic, and retry with the validation error inlined into the next prompt. Two attempts, then fall back to a human queue.

Vendor differences worth knowing

A quick field guide, based on shipping across all three in the last year:

• OpenAI Structured Outputs (strict: true on tools, or response_format: { type: "json_schema" }) is the strictest. It rejects schemas at request time if they violate its subset (no oneOf at the root, every property must be in required, etc.). Once accepted, the guarantee is real.
• Anthropic Claude tool use does not currently offer the same hard grammar guarantee on all models, but in our experience adherence is very high when the schema is clean and the description fields are informative. Claude particularly rewards good field descriptions — treat them as inline documentation the model reads at inference.
• Google Gemini supports responseSchema with responseMimeType: application/json and, separately, function calling. The schema subset supported has grown but check the current docs before assuming $ref or anyOf work.

Don't take our word for feature parity — vendors iterate monthly. Check the current docs before you write portability abstractions.

The portability trap

We tried building a vendor-neutral schema layer once. It was a mistake. The dialect differences (which JSON Schema keywords are supported, how enums interact with nullability, whether additionalProperties: false is required or forbidden) are large enough that the abstraction leaked constantly. Now we keep three thin adapters and one canonical Zod schema per feature. Adapters convert Zod to each vendor's dialect. Total code is smaller than the failed abstraction.

Evals for structured outputs

Structured doesn't mean correct. A schema-valid response can still be semantically wrong. Our eval harness for any extraction feature checks three things separately:

1. Schema validity — did the output parse? With strict mode this should be 100%. Below that means an API error, not a model error.
2. Field-level accuracy — for each field, exact-match or fuzzy-match against ground truth. Track per-field, because average accuracy hides that vendor is 98% and currency is 71%.
3. Refusal calibration — when the model emits confidence: low or currency: unknown, is it actually the hard cases? If low-confidence outputs are just as accurate as high, the model isn't using the signal and you need to prompt harder.

If you haven't built an eval harness yet, that's a bigger gap than your schema strategy. We wrote about the pattern in our engineering blog and it's the first thing we set up on any LLM project.

Where we'd start

If you're picking up an existing feature that parses LLM text with regex and prayer:

1. Define one Zod (or Pydantic) schema per output shape. This becomes your source of truth.
2. Convert it to a tool schema for your primary vendor. Turn on strict mode.
3. Add an unknown / low_confidence escape hatch on every enum and every field the model might not know.
4. Log every response, run a weekly eval against 100 – 500 labelled examples, and split accuracy by field.
5. Only reach for grammar-based local decoding (Outlines, GBNF) if you're self-hosting or need schemas the hosted APIs don't support.

And if the feature is a streaming chat UI or a 2,000-word report — consider whether structure is actually helping the user, or just helping you feel safe. Sometimes the right answer is markdown and a small parser. We're happy to argue about it; that's usually where the interesting product decisions live.

If you want a second pair of eyes on a schema or an eval setup, our AI engineering team does this work daily.