How to handle nested JSON schemas in OpenAI function calling for complex APIs

Deeply nested JSON schemas in function calling often lead to "hallucinated" structures or the model flattening your data unexpectedly. I hit this wall while building a dynamic reporting tool where the AI needed to pass a complex filter object with nested logic (AND/OR groups) to a backend API. The standard "just describe it in the description" approach fails once you go three levels deep.

The secret to getting this right is strict typing and using the required array at every single nesting level. If you leave a nested object's properties as optional, Claude or GPT-4o will occasionally omit the wrapper object entirely and just give you the leaf values.

Here is the schema pattern that actually works for complex nesting:

{
  "name": "get_analytics_data",
  "description": "Fetch reports with complex filtering",
  "parameters": {
    "type": "object",
    "properties": {
      "filters": {
        "type": "object",
        "description": "The filter criteria",
        "properties": {
          "logical_operator": { "type": "string", "enum": ["AND", "OR"] },
          "conditions": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "field": { "type": "string" },
                "operator": { "type": "string", "enum": ["eq", "gt", "lt"] },
                "value": { "type": "string" }
              },
              "required": ["field", "operator", "value"]
            }
          }
        },
        "required": ["logical_operator", "conditions"]
      }
    },
    "required": ["filters"]
  }
}

One massive productivity gain I've found is using Pydantic (if you're on Python) to generate these schemas. Writing raw JSON is a recipe for syntax errors. Since OpenAI's API expects a specific JSON Schema dialect, Pydantic's .model_json_schema() gets you 90% of the way there.

To push the accuracy further, I've started adding "Example Values" directly into the property descriptions. Instead of saying "type": "string", "description": "The date range", I use "description": "The date range in ISO 8601 format (e.g., 2023-01-01)". This acts as a few-shot prompt embedded within the schema itself.

A few gotchas to watch out for:

The Token Tax: Complex schemas eat into your context window. If you have 20+ nested functions, you'll notice the model starts ignoring the system prompt. I solve this by using "Dynamic Tool Selection"—I use a cheap model (GPT-4o-mini) to decide which 3-4 tools are relevant, then pass only those schemas to the heavy-lifter model.

The "Null" Trap: OpenAI models sometimes pass null for nested objects they don't understand. Always implement a validation layer in your code to catch None values before they hit your API, or the whole request will crash with a 500 error.

Schema Over-specification: Don't over-engineer. If you can flatten the schema (e.g., using filter_field_1 instead of filters: { field: 1 }), do it. The shallower the tree, the higher the reliability.

If you are using Cursor, I highly recommend keeping your Pydantic models in a separate schemas.py file and referencing them in your prompts. It makes it way easier to iterate on the structure without scrolling through 200 lines of JSON.