Tool-Call Retries: Why Your Agent Keeps Calling the Wrong Function Twice

A client called us last spring because their support agent had refunded the same order three times in forty seconds. The LLM wasn't hallucinating — it was retrying a tool call that had actually succeeded, because the first response timed out at the HTTP layer. The model saw no result, assumed failure, and tried again. And again.

This is one of the most common and least-discussed failure modes in agent systems. Everyone writes about prompt engineering and RAG. Almost nobody writes about what happens when process_refund returns a 504 but still ran.

The failure mode nobody talks about

Function-calling agents — whether you're using OpenAI's tool calling, Anthropic's tool_use blocks, or Gemini's function declarations — all share the same basic loop:

1. Model emits a tool call
2. Your code executes it
3. You feed the result back as a tool message
4. Model decides what to do next

The fragile part is step 2 to step 3. Networks fail. Databases lock. Third-party APIs rate-limit. And when they do, you have three bad options:

• Return an error to the model and let it retry. Risk: duplicate side effects if the first call partially succeeded.
• Retry silently in your runtime before returning. Risk: same duplicate problem, plus the model has no visibility.
• Hard-fail the whole agent run. Safe, but your UX dies on every transient blip.

Most teams pick option 1 or 2 without thinking about it, then discover the consequences in production.

Why models make it worse

LLMs are optimistic retriers. If you return {"error": "timeout"}, Claude and GPT will almost always call the same tool again with the same arguments. That's usually correct behaviour for read operations. It's catastrophic for write operations that aren't idempotent.

Worse, if you don't shape the error message carefully, the model may "improve" the retry by changing arguments — turning a transient failure into a semantic bug. We've seen GPT-4 class models decide that if charge_card(amount=100) timed out, maybe it should try charge_card(amount=50) to "test if the service is up". Yes, really.

The idempotency key pattern

The fix borrowed from payments infrastructure is straightforward: every tool call that mutates state gets an idempotency key, and your tool implementation is responsible for deduping.

The trick is where the key comes from. You have two choices:

1. Model-generated: Add an idempotency_key parameter to the tool schema and let the LLM mint a UUID. Simple, but models are terrible at generating truly unique values and will sometimes reuse keys across semantically different calls.
2. Runtime-generated: Your agent runtime hashes the tool name, arguments, and a per-turn nonce. The model never sees the key.

We strongly prefer the second approach. Here's a sketch:

import hashlib
import json
from dataclasses import dataclass

@dataclass
class ToolCall:
    name: str
    arguments: dict
    turn_id: str  # stable per agent turn, not per retry

def idempotency_key(call: ToolCall) -> str:
    payload = json.dumps(
        {"name": call.name, "args": call.arguments, "turn": call.turn_id},
        sort_keys=True,
    )
    return hashlib.sha256(payload.encode()).hexdigest()[:32]

async def execute_with_retry(call: ToolCall, max_attempts: int = 3):
    key = idempotency_key(call)
    for attempt in range(max_attempts):
        try:
            return await TOOL_REGISTRY[call.name](
                **call.arguments,
                _idempotency_key=key,
            )
        except TransientError as e:
            if attempt == max_attempts - 1:
                return {"error": "transient_failure", "detail": str(e), "retried": attempt + 1}
            await backoff(attempt)

The turn_id is critical. If you use a per-call nonce, every retry gets a new key and defeats the purpose. If you use something too stable (like a session ID), the model can't legitimately call the same tool twice in one conversation.

What the tool itself has to do

The idempotency key is useless unless the downstream tool honours it. For your own services, that means a dedup table:

CREATE TABLE tool_executions (
  idempotency_key TEXT PRIMARY KEY,
  tool_name TEXT NOT NULL,
  result JSONB NOT NULL,
  created_at TIMESTAMPTZ DEFAULT now()
);

On each call: INSERT ... ON CONFLICT DO NOTHING RETURNING result. If the row already exists, return the stored result. If not, do the work and store it.

For third-party APIs, check their docs. Stripe, for example, accepts an Idempotency-Key header on every mutating endpoint. Many internal corporate APIs don't support this — in which case you need a proxy layer that adds it, or you accept the risk and log loudly when retries happen.

Shaping errors so the model behaves

Assume a tool call fails after all your retries. What you send back to the model matters enormously. Compare these two tool results:

{"error": "500 Internal Server Error"}

{
  "status": "failed",
  "error_type": "transient",
  "retriable": false,
  "message": "The refund service is temporarily unavailable. Do not retry this tool. Inform the user and offer to escalate.",
  "retry_after_seconds": null
}

The second one tells the model exactly what to do. We've found that explicit retriable: false plus a natural-language instruction cuts duplicate retries dramatically. Anthropic's tool-use documentation and OpenAI's function-calling guide both note that models respond well to structured error schemas — treat the tool result as a prompt, because that's what it is.

The three error categories we use

We classify every tool failure into one of three buckets, and the schema makes the category explicit:

• transient: Network blip, rate limit, timeout. The runtime already retried. Model should inform the user and move on, not retry.
• permanent: Invalid arguments, auth failure, resource not found. Model should re-plan or ask the user for clarification.
• degraded: The tool returned something, but it's incomplete. Model should decide whether to proceed or ask.

This maps cleanly to how models actually reason. A permanent error with "the user_id you provided does not exist" reliably triggers a clarifying question. A transient error with "do not retry" reliably stops the loop.

When to let the model re-plan vs retry

Not every failure deserves a retry. Some deserve a re-plan. The heuristic we use:

• Retry at the runtime level for network and infrastructure errors. Keep it invisible to the model.
• Return to the model for semantic errors (wrong argument, missing permission, business rule violation).
• Abort the whole turn when you've hit a hard budget — too many tool calls, too much latency, too many tokens.

The third case is where a lot of agent frameworks fall down. If you let the model keep calling tools after a failure cascade, you can burn through a $5 budget in one turn. We cap total tool calls per turn, total wall-clock time, and total tokens consumed. When any budget is exceeded, we inject a synthetic tool result that says "budget exceeded, summarise what you've learned and stop" — and we strip the tools from the next model call so it physically cannot continue.

Observability is not optional

If you're running agents in production without per-tool-call tracing, you are flying blind. At minimum, log:

• Tool name, arguments (redact PII), idempotency key
• Attempt number, latency, outcome
• The model's reasoning text immediately before the call (if available)
• The exact tool result returned to the model

We tag every tool call with the agent run ID and the turn ID so we can reconstruct the full loop later. When something goes wrong — and it will — this is the difference between a two-hour debug session and a two-day one.

The uncomfortable tradeoff

All of this adds latency and complexity. A naive agent that just retries on any error ships in a day. An agent with idempotency keys, error classification, budget enforcement, and tracing takes a sprint or two to get right. For a weekend prototype, skip it. For anything touching money, user data, or external systems with real-world consequences, don't.

The teams that get this wrong usually learn the same way: a postmortem where someone has to explain to a customer why they were charged three times, or why their calendar has eleven duplicate meetings, or why a ticket got escalated to seven different agents.

Where we'd start

If you're building an agent today and haven't thought about tool-call retries, do these three things this week:

1. Audit every tool and mark which ones are safe to retry. The unsafe ones need idempotency keys end-to-end.
2. Replace generic error strings with the structured error schema above. Even without full idempotency, this cuts duplicate calls noticeably.
3. Add per-turn budgets — tool call count, wall time, tokens. Enforce them in the runtime, not the prompt.

If you want help designing agent architecture that holds up in production, that's part of what our AI engineering team does day to day. The patterns above come straight from systems we've shipped and the ones we've had to rescue.