How teams ship real products with Claude, GPT, Gemini, RAG, agents, and evals.
Autonomous agents fail in two ways: they stop too early, or they never stop. Here's how we design loops that terminate cleanly, stay under budget, and tell you why they quit.
Semantic chunking sounds smarter than splitting on 512 tokens, but in production it often loses to boring fixed-size chunks. Here's when each one wins, and what we ship by default.
When Anthropic goes down for two hours, your product still needs to answer. Here's how we design fallback chains across Claude, GPT, and Gemini without blowing latency budgets or shipping garbage responses.
Most LLM eval suites are theatre. Here's how we build ones that actually block bad deploys — with graders that agree with humans, deterministic seeds, and a scoring rubric your PM can read.
A 200K context window sounds like freedom until latency spikes and answers get worse. Here's how we allocate tokens across system, retrieval, history, and scratch in real apps.
Streaming tokens feel fast until you try to render structured JSON. Here's how to parse partial LLM output as it arrives, without the UI freezing or the schema blowing up mid-stream.
Prompt caching can cut LLM bills by 50–90% on the right workloads — and quietly do nothing on the wrong ones. Here's how we structure prompts, pick TTLs, and measure real savings on Claude and Gemini.
Structured outputs and tool calls look interchangeable until you ship them. Here's how we decide which one to reach for, based on latency, reliability, and how much the model needs to think.
Rerankers are the most oversold and undermeasured piece of the RAG stack. Here's when adding one earns its keep, when it's dead weight, and how to prove it with a small eval.
Agents that call the same tool twelve times in a row aren't reasoning — they're panicking. Here's how we detect, prevent, and recover from tool call loops in production LLM agents.
Semantic caching promises free hits at 95% similarity. In practice it hands users wrong answers with confidence. Here's how to build a similarity layer you can actually trust in production.
Fixed-size chunking is the default for a reason: it's easy. It's also why your RAG answers are mediocre. Here's how we chunk documents on real projects, and when we throw the rulebook out.
Most LLM eval suites tell you nothing useful. Here's the harness structure we use to catch real regressions before they hit production — golden sets, judges, and CI gates that don't lie.
Long context windows are a trap if you treat them like free storage. Here's how we budget tokens across system prompts, RAG chunks, and tool outputs without wrecking quality.
A practical pattern for cutting LLM spend 40–70% without hurting quality: route to a small model first, escalate to a frontier model only when confidence is low. Here's how to build it, measure it, and avoid the traps.
Prompt caching sounds like free money — bill the big system prompt once, pay pennies after. Reality is messier. Here's what caching actually buys you, and the failure modes that show up once you ship.
JSON mode, strict tool schemas, grammar-constrained decoding — three ways to force LLMs into structured output, each with different failure modes. Here's how we pick, and when we give up and parse text.
Naive exponential backoff will either burn your budget or drop requests your users actually care about. Here's how we design retry logic for Claude, GPT, and Gemini in production.
Users don't care about your token/sec chart. They care whether the UI moved. Here's how we stream tool-calling agents so they feel responsive without faking progress or breaking retries.
Pure vector search misses exact matches. Pure BM25 misses meaning. Here's how we wire them together in production RAG without turning the retrieval layer into a tangle of glue code.
Fixed-size chunking is the default because it's easy. Semantic chunking is trendy because it sounds smart. Here's what actually changes retrieval quality in production RAG systems, and how to decide which one you need.
Most teams write prompts, ship, and pray. Here's how we build eval harnesses that actually catch regressions before a model swap or prompt tweak breaks production.
One runaway agent loop can eat a week of margin. Here's how we set per-request token budgets, enforce them at the SDK layer, and keep product features profitable without lobotomising the model.
Gemini and Claude now ship million-token windows. That doesn't mean you should stuff everything into the prompt. Here's how we decide between long context and RAG on real projects.
Prompt caching sounds like free money — until your hit rate collapses at 3am. Here's what we've learned shipping cached prompts across Claude, OpenAI, and Gemini, and where the savings actually come from.
Both structured outputs and tool calls give you typed JSON from an LLM, but they solve different problems. Here's how we decide which one to reach for on real projects.
Rerankers can turn a mediocre RAG stack into a sharp one — or add 400ms of latency for a rounding-error quality bump. Here's how to tell which situation you're in.
Streaming tool calls can shave seconds off agent latency, but only if your parser can handle partial JSON, mid-flight cancellations, and provider-specific quirks. Here's how we ship it in production.
Anthropic, OpenAI, and Google all go down. Here's how to build a fallback chain that survives outages without doubling your p95 latency or your bill.
Exact-match caches barely help LLM apps because users phrase things differently every time. Here's how to build a semantic cache that actually cuts spend without shipping stale or wrong answers.
Unit tests don't catch agent regressions. Here's how to build an eval harness that runs in CI, fails fast on real breakages, and doesn't bankrupt you on token spend.
Retrieval pipelines quietly inflate prompts until margins disappear. Here's how we set token budgets per stage, enforce them in code, and catch regressions before they hit production invoices.
Long-context models tempt you to stuff everything into the prompt. That's how you end up with slow, expensive, and weirdly dumb responses. Here's how we budget tokens in production.
Prompt caching looks like free money: stuff a giant system prompt once, pay pennies forever. The reality is messier. Here's when it actually saves you cost and latency, and when it quietly costs more than it saves.
JSON mode, strict schemas, and tool calls all promise reliable structured output from LLMs. They behave differently under load, failure, and schema drift. Here's how we pick between them.
Rerankers fix the recall-precision gap in RAG, but they cost latency and money. Here's when a cross-encoder actually pays off, and when you should tune retrieval instead.
Streaming tool calls feel like a free win until your agent state diverges, your UI flickers, and your retries double-charge users. Here's how to ship it without the footguns.
Picking one frontier model and praying is not a strategy. Here's how we route requests across Claude, GPT, and Gemini in production — by task shape, cost, and failure mode.
Semantic caching promises huge cost wins for LLM apps, but naive implementations leak wrong answers across users. Here's how we build cache layers that actually hold up.
Most LLM eval setups measure the wrong things and miss the regressions that actually break production. Here's how we build harnesses that catch silent failures before users do.
Pure vector search loses on acronyms, product codes, and rare names. Here's how we mix BM25 with embeddings to fix recall without rewriting the stack.
Agent loops quietly balloon context until a single task costs dollars instead of cents. Here's how we budget tokens per turn, per tool, and per task — with code you can paste into your own runner.
Prompt caching looks like free money until your hit rate collapses or your cached context goes stale. Here's how we decide when to turn it on, and when to leave it off.
JSON mode and structured outputs feel like a silver bullet until they aren't. Here's what actually breaks in production, and how we decide between strict schemas, function calling, and plain prompting.
Agents fail mid-run, networks flap, and tool calls retry. Here's how we design LLM agents that can be safely re-executed without double-charging cards, duplicating tickets, or corrupting state.
Streaming tool calls look great in demos and fall apart in production. Here's what we learned wiring them into agents that handle real traffic, retries, and partial failures.
We've shipped agents on all three frontier APIs in the last year. Here's how we actually decide which one runs in production — tool use, latency, cost ceilings, and the boring stuff that breaks at 3am.
RAG bills balloon fast once you ship to real users. Here's how we trim retrieval and generation spend by half or more without watching recall collapse.
Most teams write LLM evals once, watch them pass, and ship blind. Here's how we structure eval suites that fail loudly when a prompt tweak or model swap quietly breaks production.
