Handling API timeouts and retries when implementing LLM function calling

Function calling is a game-changer for giving LLMs "hands," but the biggest lie we tell ourselves during the demo phase is that the API is stable. In production, you'll hit 504 Gateway Timeouts or 429 Too Many Requests exactly when your agent is mid-loop, leaving your state machine hanging and your user staring at a loading spinner.

The mistake most people make is wrapping the entire LLM call in a generic try-catch. That's too blunt. You need to distinguish between a model failure (hallucinated arguments) and a transport failure (timeout).

I've found that the most resilient pattern is implementing an exponential backoff with "jitter" specifically for the tool-execution phase. If the LLM decides to call a function that hits a slow third-party API, and that API times out, you can't just retry the LLM call—you have to retry the tool execution and then feed the result back.

Here is how I handle this in TypeScript using tenacity-style logic. I avoid simple loops and use a dedicated retry wrapper for the tool handlers:

async function withRetry<T>(fn: () => Promise<T>, retries = 3, delay = 1000): Promise<T> {
  try {
    return await fn();
  } catch (error) {
    if (retries <= 0) throw error;
    // Add jitter to prevent thundering herd problem
    const jitter = Math.random() * 100; 
    await new Promise(resolve => setTimeout(resolve, delay + jitter));
    return withRetry(fn, retries - 1, delay * 2);
  }
}

// Usage inside the function calling loop
const toolResult = await withRetry(async () => {
  return await executeTool(toolCall.name, toolCall.args);
});

A critical "gotcha" I hit with Cursor's composer while building this: the AI often forgets that LLM providers have strict timeouts on their end. If your tool takes 20 seconds to run, the LLM connection might drop before you can send the tool_outputs back.

To fix this, I moved to an asynchronous polling pattern for heavy tools. Instead of the tool executing the work, the tool returns a job_id. The agent then has to call a check_status tool. This prevents the HTTP connection from idling out.

My current config for production reliability:

• Hard Timeout: Set a timeout value in your Axios/Fetch config that is shorter than the LLM provider's gateway timeout (usually 30-60s). If you hit 25s, kill it and retry manually rather than waiting for a generic 504.
• Idempotency Keys: If your function calls trigger payments or database writes, pass an idempotency_key generated at the start of the LLM turn. This ensures that if a retry happens after a partial timeout, you don't charge the customer twice.
• Fallback Prompts: When a tool fails after all retries, don't just throw an error to the user. Feed the error back to the LLM:

   tool 'get_weather' failed due to a timeout. Please inform the user that the service is currently unavailable and suggest an alternative.

If you're using Claude 3.5 Sonnet, it's remarkably good at "self-healing" if you pass the error message back into the conversation history. It will often try to call a different tool or simplify the query to get a result.