Handling Complex JSON Schemas for Reliable Tool Use in Function Calling

The biggest headache with LLM function calling isn't the logic—it's the hallucinated JSON. When you start passing complex, nested schemas to Claude or GPT-4o, the model often forgets a required field or messes up the type of a nested object, causing your backend to crash during parsing.

I've spent the last month fighting this in a project where I need the AI to generate multi-layered API queries for a legacy database. If the JSON isn't 100% compliant with the schema, the whole request fails.

The "naive" way is to just describe the schema in the tool definition. The "pro" way is to treat the schema as code and enforce it via a strict validation loop.

The "Schema Flattening" Trick
Deeply nested objects are where LLMs trip up. I found that if I flatten the structure in the tool definition but reconstruct it in the handler, the reliability jumps. Instead of a filters object containing an options object containing a date_range object, I define them as:
filter_start_date, filter_end_date, filter_operator.

Strict Type Enforcement with Pydantic
If you're using Python, stop manually parsing JSON. Wrap your tool outputs in Pydantic models. This doesn't just validate; it gives you a clean way to handle retries.

from pydantic import BaseModel, Field, ValidationError
from typing import List, Optional

class QueryFilter(BaseModel):
    field: str
    operator: str = Field(description="One of: 'eq', 'gt', 'lt', 'contains'")
    value: str

class ComplexSearch(BaseModel):
    query: str
    filters: List[QueryFilter]
    limit: int = Field(gt=0, lt=100)

def handle_tool_call(tool_input_json):
    try:
        # This catches the hallucinated types immediately
        validated_data = ComplexSearch.model_validate_json(tool_input_json)
        return execute_search(validated_data)
    except ValidationError as e:
        # Feed the error back to the AI to self-correct
        return f"JSON Schema Error: {e.json()}. Please fix the format and try again."

Optimizing the Prompt for Tool Accuracy
When using Cursor's .cursorrules or a system prompt, I've noticed that explicitly telling the AI how to think about the JSON helps. I use a "Schema Anchor" in my system prompt:

When calling the 'complex_search' tool:
- Ensure 'filters' is always an array, even for a single filter.
- The 'operator' must be strictly lowercase.
- Never omit the 'limit' field; default to 20 if not specified.

The "Gotchas" I encountered
Enum drift: If your enum values change in the code but not in the tool description, the AI will keep sending the old values. I now auto-generate my tool definitions directly from my Pydantic models to keep them in sync.
Token bloat: Massive JSON schemas eat into your context window and can actually confuse the model. If your schema is over 50 lines, break the tool into smaller, atomic functions. Instead of one update_user_profile tool with 30 optional fields, use update_user_contact and update_user_preferences.

Productivity Gain
By implementing the Pydantic validation loop and flattening my nested schemas, my "failed tool call" rate dropped from about 15% to under 2%. I stopped spending time debugging "KeyError: 'date'" and started focusing on the actual feature logic.