Structured Outputs in 2026: When to Trust the Schema and When to Validate Twice
Structured output modes from OpenAI, Anthropic, and Google look identical on the slide deck. In production they fail in very different ways. Here's how we handle schemas without getting burned.
Every vendor now ships a "structured outputs" mode and every vendor's marketing page implies it Just Works. In practice, the three major providers enforce schemas differently, fail differently, and bill you differently when they do. If you're wiring an LLM into a typed backend, those differences matter more than the benchmark scores.
This is the pattern we use on client projects when the model's output has to feed a database, a downstream API, or a UI component that will crash on a missing field.
What "structured output" actually means at each vendor
The three implementations look similar but have real differences worth knowing before you pick one.
- OpenAI exposes
response_format: { type: "json_schema", strict: true }on the Responses and Chat Completions APIs. Whenstrictis true, the sampler is constrained so the output is guaranteed to match the schema — but only a subset of JSON Schema is supported (nominLength, nopattern, noformat: "email"enforcement, all fields must be inrequired,additionalPropertiesmust be false). See OpenAI's Structured Outputs docs. - Anthropic doesn't have a
strictJSON mode in the same sense. The idiomatic path is tool use: define aninput_schemaon a tool, settool_choiceto force that tool, and the model emits arguments matching the schema. Validation is best-effort — Claude is usually right, but you still need to validate. - Google Gemini supports
responseSchemawithresponseMimeType: "application/json". It accepts a subset of OpenAPI schema. Enum constraints and types are enforced at decode time; more exotic constraints are not.
Translation: OpenAI gives you the hardest guarantee but the narrowest schema dialect. Anthropic gives you the richest schema flexibility but no decode-time guarantee. Gemini sits in the middle.
The three failure modes you'll actually hit
- Schema accepted, output semantically wrong. The JSON parses, every field is present, and
priceis0because the model didn't find a price and guessed. No API error will catch this. - Schema accepted at request time, rejected at generation. You pass something the vendor's schema linter allows but the sampler can't satisfy — usually deeply nested
anyOfwith recursive refs. You get a generation error 20 seconds in. - Schema "honored" but truncated. The model hits
max_tokensmid-object. Strict mode on OpenAI will usually refuse; Anthropic and Gemini will happily hand you an unclosed brace.
Design the schema for the model, not for your ORM
The biggest mistake we see is teams passing their Prisma or SQLAlchemy model shape directly to the LLM. That schema was designed for a relational store, not for a token-by-token generator.
A few rules we apply:
- Flatten aggressively. Deep nesting costs tokens and increases the chance the model loses track. If you need a tree, generate a flat list and reassemble server-side.
- Put required reasoning fields first. Models write left to right. If
summarydepends onfindings,findingsmust come first in the property order. - Use enums for anything categorical.
status: "pending" | "approved" | "rejected"is strictly better thanstatus: stringand the vendors enforce it. - Add a
confidenceorsourcefield when the output will be stored. It's cheap and gives you a filterable signal later.
Here's a schema we used recently for extracting line items from scanned invoices. Note the ordering and the explicit null handling:
{
"type": "object",
"additionalProperties": false,
"required": ["vendor_name", "invoice_date", "line_items", "total", "extraction_notes"],
"properties": {
"vendor_name": { "type": "string" },
"invoice_date": { "type": "string", "description": "ISO 8601 date, or empty string if not found" },
"line_items": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": ["description", "quantity", "unit_price", "line_total"],
"properties": {
"description": { "type": "string" },
"quantity": { "type": "number" },
"unit_price": { "type": "number" },
"line_total": { "type": "number" }
}
}
},
"total": { "type": "number" },
"extraction_notes": {
"type": "string",
"description": "Free-text notes about ambiguity, missing fields, or low-confidence values."
}
}
}
The extraction_notes field is a trick we use everywhere. It gives the model a legal place to express uncertainty instead of inventing data to fill required fields. Read it in logs; you'll learn more about your pipeline from those notes than from any eval dashboard.
Validate twice, even when the vendor says you don't have to
Strict mode from OpenAI is genuinely strict at the token level. But "matches the schema" is not the same as "is correct." We run two layers of validation on every structured response, regardless of vendor.
Layer 1: schema + type coercion
Use Zod, Pydantic, or your language's equivalent. Don't just check — coerce. Models occasionally emit "42" where you asked for a number, especially with Anthropic tool use.
import { z } from "zod";
const LineItem = z.object({
description: z.string().min(1),
quantity: z.coerce.number().positive(),
unit_price: z.coerce.number().nonnegative(),
line_total: z.coerce.number().nonnegative(),
});
const Invoice = z.object({
vendor_name: z.string().min(1),
invoice_date: z.union([z.string().date(), z.literal("")]),
line_items: z.array(LineItem),
total: z.coerce.number().nonnegative(),
extraction_notes: z.string(),
});
export function parseInvoice(raw: unknown) {
const result = Invoice.safeParse(raw);
if (!result.success) {
return { ok: false as const, issues: result.error.issues };
}
return { ok: true as const, data: result.data };
}
Layer 2: business rules
These are checks the schema can't express. For invoices: do the line items sum to the total (within rounding)? Is the invoice date not in the future? Does the vendor name match any entry in your known-vendors list?
Failures here don't necessarily mean reject — they mean route to human review or retry with a sharper prompt. We log both layers separately so we can see whether our problem is model behavior or prompt scope.
The retry loop that doesn't bankrupt you
When validation fails, the temptation is to re-ask the model with the error message appended. That works, but naively implemented it's expensive. Our rule: one retry max, with the specific validation error, and only if the first attempt was non-strict. If you were already using OpenAI strict mode and still got a semantic failure, retrying rarely helps — the model doesn't know why total was wrong, just that it was.
def extract_with_retry(client, document, schema):
first = call_model(client, document, schema)
parsed = validate(first)
if parsed.ok:
return parsed.data
# One targeted retry with the error surfaced
second = call_model(
client,
document,
schema,
extra_instructions=f"Previous attempt failed validation: {parsed.error}. Fix only the invalid fields.",
)
parsed = validate(second)
if parsed.ok:
return parsed.data
# Hand off, don't loop
raise NeedsHumanReview(first, second, parsed.error)
If you find yourself needing more than one retry regularly, the fix isn't more retries — it's a better schema, a better prompt, or a bigger model for that specific call. Model routing applies here too.
Vendor-specific gotchas worth knowing
- OpenAI strict mode caches the compiled schema. The first call with a new schema pays a latency tax (sometimes a few seconds). Keep schemas stable across requests and you avoid it.
- Anthropic tool use counts tool definitions as input tokens on every call. Big schemas get expensive fast if you're high-volume. Consider prompt caching for the tool definitions.
- Gemini's
responseSchemadoesn't supportoneOfcleanly in our experience. If you need discriminated unions, emit atypeenum and a loose payload, then narrow server-side. - All three vendors handle
descriptionfields differently. Treat them as part of the prompt, not as documentation. The model reads them and acts on them.
Where this fits in a real system
Structured outputs are the seam between the probabilistic part of your stack and the deterministic part. Treat that seam with suspicion. Log every raw response (not just the parsed one) for at least 30 days — when a bug report lands, you'll want to see what the model actually said, not what your validator reconstructed.
If you're building extraction, classification, or form-filling flows, structured outputs will carry 80% of the work. The other 20% is the validation, retry, and human-review scaffolding around them. We cover more of that in our AI engineering services work.
Where we'd start
Pick one high-volume structured call in your product. Write a Zod or Pydantic schema for its output. Add business-rule validation as a second layer. Log the raw response, the parsed response, and both validation results separately. Run it for a week and look at the diff between "schema valid" and "business valid." That gap is where your real work is — and no amount of switching models will close it for you.
Want a team like ours?
72Technologies builds production software for the kind of teams who actually read this blog.
Start a projectKeep reading
Tool Use That Doesn't Hallucinate: Designing Function Schemas LLMs Actually Respect
Function calling fails in boring, predictable ways: invented arguments, wrong enums, nested JSON that half-parses. Here's how we design tool schemas so Claude, GPT, and Gemini stop improvising.
Prompt Caching in Practice: What Actually Lands on the Bill
Prompt caching sounds like free money until you read the invoice. Here's how Anthropic, OpenAI, and Gemini caches actually behave, and the patterns that make them pay off.
Agent Loops That Don't Spiral: Budgets, Stop Conditions, and Fallbacks
Most agent failures aren't reasoning failures — they're loop failures. Here's how we design budgets, stop conditions, and fallbacks so our agents don't burn $40 proving they can't book a flight.
