How to enforce tool calls in Gemma’s auto mode with a two-pass fallback

The problem persists in a local-first assistant using the Gemma model, where responses include a vague "I'll check that for you" without triggering any actual tool calls—just fabricated text. Even with tool_choice="auto", the system fails to execute required actions reliably.

The root issue stems from the model’s flexibility allowing tool omissions, which isn’t just a prompt flaw but a structural oversight in how tool selection is enforced.

A two-pass recovery method ensures mandatory tool execution

The solution involves intercepting the stream and forcing a second attempt when needed. Here’s how it works:

1. Initial stream processing checks for tool calls in real time

   response = client.chat.completions.create(
       model="gemma-3-27b",
       messages=messages,
       tools=local_tools,
       tool_choice="auto",
       stream=True
   )

A buffered list collects chunks, and any tool_calls detected halts further checks.

1. Conditional retry triggers only when the user’s request demands local data

   if needs_local_data(user_msg) and not tool_called:
       recovery = client.chat.completions.create(
           model="gemma-3-27b",
           messages=messages,
           tools=local_tools,
           tool_choice="required",
           stream=False
       )

This second attempt enforces a tool call before execution begins.

1. Validation ensures the tool matches the task

   if is_valid_local_tool(recovery.choices[0].message.tool_calls[0]):
       execute_tool_flow(recovery)
   else:
       yield from buffered  # revert to initial output with anomaly logged

Critical safeguards include:

• Limiting retries to one instance to avoid cycles
• Skipping forced tools for non-data requests
• Rejecting mismatched tool types (e.g., write_note when the query asked for notes)

The fallback preserves the original stream’s output for users until the system stabilizes, preventing them from seeing the hallucinated placeholder response. Test results confirm the workflow reliably transitions between auto and required modes without losing context.

This approach highlights that while models decide what to do, proper tooling enforcement must sit at the application layer to meet user expectations.