All articles
AI & LLMsNovember 13, 2026 6 min read

Structured Outputs in 2026: When to Trust the Schema and When to Validate Twice

Structured output modes from OpenAI, Anthropic, and Google look identical on the slide deck. In production they fail in very different ways. Here's how we handle schemas without getting burned.

Every vendor now ships a "structured outputs" mode and every vendor's marketing page implies it Just Works. In practice, the three major providers enforce schemas differently, fail differently, and bill you differently when they do. If you're wiring an LLM into a typed backend, those differences matter more than the benchmark scores.

This is the pattern we use on client projects when the model's output has to feed a database, a downstream API, or a UI component that will crash on a missing field.

What "structured output" actually means at each vendor

The three implementations look similar but have real differences worth knowing before you pick one.

  • OpenAI exposes response_format: { type: "json_schema", strict: true } on the Responses and Chat Completions APIs. When strict is true, the sampler is constrained so the output is guaranteed to match the schema — but only a subset of JSON Schema is supported (no minLength, no pattern, no format: "email" enforcement, all fields must be in required, additionalProperties must be false). See OpenAI's Structured Outputs docs.
  • Anthropic doesn't have a strict JSON mode in the same sense. The idiomatic path is tool use: define an input_schema on a tool, set tool_choice to force that tool, and the model emits arguments matching the schema. Validation is best-effort — Claude is usually right, but you still need to validate.
  • Google Gemini supports responseSchema with responseMimeType: "application/json". It accepts a subset of OpenAPI schema. Enum constraints and types are enforced at decode time; more exotic constraints are not.

Translation: OpenAI gives you the hardest guarantee but the narrowest schema dialect. Anthropic gives you the richest schema flexibility but no decode-time guarantee. Gemini sits in the middle.

The three failure modes you'll actually hit

  1. Schema accepted, output semantically wrong. The JSON parses, every field is present, and price is 0 because the model didn't find a price and guessed. No API error will catch this.
  2. Schema accepted at request time, rejected at generation. You pass something the vendor's schema linter allows but the sampler can't satisfy — usually deeply nested anyOf with recursive refs. You get a generation error 20 seconds in.
  3. Schema "honored" but truncated. The model hits max_tokens mid-object. Strict mode on OpenAI will usually refuse; Anthropic and Gemini will happily hand you an unclosed brace.

Design the schema for the model, not for your ORM

The biggest mistake we see is teams passing their Prisma or SQLAlchemy model shape directly to the LLM. That schema was designed for a relational store, not for a token-by-token generator.

A few rules we apply:

  • Flatten aggressively. Deep nesting costs tokens and increases the chance the model loses track. If you need a tree, generate a flat list and reassemble server-side.
  • Put required reasoning fields first. Models write left to right. If summary depends on findings, findings must come first in the property order.
  • Use enums for anything categorical. status: "pending" | "approved" | "rejected" is strictly better than status: string and the vendors enforce it.
  • Add a confidence or source field when the output will be stored. It's cheap and gives you a filterable signal later.

Here's a schema we used recently for extracting line items from scanned invoices. Note the ordering and the explicit null handling:

{
  "type": "object",
  "additionalProperties": false,
  "required": ["vendor_name", "invoice_date", "line_items", "total", "extraction_notes"],
  "properties": {
    "vendor_name": { "type": "string" },
    "invoice_date": { "type": "string", "description": "ISO 8601 date, or empty string if not found" },
    "line_items": {
      "type": "array",
      "items": {
        "type": "object",
        "additionalProperties": false,
        "required": ["description", "quantity", "unit_price", "line_total"],
        "properties": {
          "description": { "type": "string" },
          "quantity": { "type": "number" },
          "unit_price": { "type": "number" },
          "line_total": { "type": "number" }
        }
      }
    },
    "total": { "type": "number" },
    "extraction_notes": {
      "type": "string",
      "description": "Free-text notes about ambiguity, missing fields, or low-confidence values."
    }
  }
}

The extraction_notes field is a trick we use everywhere. It gives the model a legal place to express uncertainty instead of inventing data to fill required fields. Read it in logs; you'll learn more about your pipeline from those notes than from any eval dashboard.

Validate twice, even when the vendor says you don't have to

Strict mode from OpenAI is genuinely strict at the token level. But "matches the schema" is not the same as "is correct." We run two layers of validation on every structured response, regardless of vendor.

Layer 1: schema + type coercion

Use Zod, Pydantic, or your language's equivalent. Don't just check — coerce. Models occasionally emit "42" where you asked for a number, especially with Anthropic tool use.

import { z } from "zod";

const LineItem = z.object({
  description: z.string().min(1),
  quantity: z.coerce.number().positive(),
  unit_price: z.coerce.number().nonnegative(),
  line_total: z.coerce.number().nonnegative(),
});

const Invoice = z.object({
  vendor_name: z.string().min(1),
  invoice_date: z.union([z.string().date(), z.literal("")]),
  line_items: z.array(LineItem),
  total: z.coerce.number().nonnegative(),
  extraction_notes: z.string(),
});

export function parseInvoice(raw: unknown) {
  const result = Invoice.safeParse(raw);
  if (!result.success) {
    return { ok: false as const, issues: result.error.issues };
  }
  return { ok: true as const, data: result.data };
}

Layer 2: business rules

These are checks the schema can't express. For invoices: do the line items sum to the total (within rounding)? Is the invoice date not in the future? Does the vendor name match any entry in your known-vendors list?

Failures here don't necessarily mean reject — they mean route to human review or retry with a sharper prompt. We log both layers separately so we can see whether our problem is model behavior or prompt scope.

The retry loop that doesn't bankrupt you

When validation fails, the temptation is to re-ask the model with the error message appended. That works, but naively implemented it's expensive. Our rule: one retry max, with the specific validation error, and only if the first attempt was non-strict. If you were already using OpenAI strict mode and still got a semantic failure, retrying rarely helps — the model doesn't know why total was wrong, just that it was.

def extract_with_retry(client, document, schema):
    first = call_model(client, document, schema)
    parsed = validate(first)
    if parsed.ok:
        return parsed.data

    # One targeted retry with the error surfaced
    second = call_model(
        client,
        document,
        schema,
        extra_instructions=f"Previous attempt failed validation: {parsed.error}. Fix only the invalid fields.",
    )
    parsed = validate(second)
    if parsed.ok:
        return parsed.data

    # Hand off, don't loop
    raise NeedsHumanReview(first, second, parsed.error)

If you find yourself needing more than one retry regularly, the fix isn't more retries — it's a better schema, a better prompt, or a bigger model for that specific call. Model routing applies here too.

Vendor-specific gotchas worth knowing

  • OpenAI strict mode caches the compiled schema. The first call with a new schema pays a latency tax (sometimes a few seconds). Keep schemas stable across requests and you avoid it.
  • Anthropic tool use counts tool definitions as input tokens on every call. Big schemas get expensive fast if you're high-volume. Consider prompt caching for the tool definitions.
  • Gemini's responseSchema doesn't support oneOf cleanly in our experience. If you need discriminated unions, emit a type enum and a loose payload, then narrow server-side.
  • All three vendors handle description fields differently. Treat them as part of the prompt, not as documentation. The model reads them and acts on them.

Where this fits in a real system

Structured outputs are the seam between the probabilistic part of your stack and the deterministic part. Treat that seam with suspicion. Log every raw response (not just the parsed one) for at least 30 days — when a bug report lands, you'll want to see what the model actually said, not what your validator reconstructed.

If you're building extraction, classification, or form-filling flows, structured outputs will carry 80% of the work. The other 20% is the validation, retry, and human-review scaffolding around them. We cover more of that in our AI engineering services work.

Where we'd start

Pick one high-volume structured call in your product. Write a Zod or Pydantic schema for its output. Add business-rule validation as a second layer. Log the raw response, the parsed response, and both validation results separately. Run it for a week and look at the diff between "schema valid" and "business valid." That gap is where your real work is — and no amount of switching models will close it for you.

#AI#LLMs#RAG#Engineering#Structured Outputs

Want a team like ours?

72Technologies builds production software for the kind of teams who actually read this blog.

Start a project