Why does my local LLM keep hallucinating API endpoints that

Spent the weekend wrestling with a 7B model running via Ollama, trying to get it to generate valid FastAPI route handlers. Every third response invents a RequestValidationError import from fastapi.exceptions that was deprecated two versions ago, or suggests response_model=List[User] without importing List from typing. The model confidently outputs code that looks syntactically correct but fails at import time.

Tried a few approaches:

1. Added a system prompt with version-pinned docs — pasted the FastAPI 0.110 reference into the context window. Helped with imports but the model still invents parameter names like request_body instead of body for Body(...).

2. Few-shot with 5 corrected examples — better, but now it overfits to the pattern and repeats the same CRUD structure even when I ask for a webhook handler.

3. RAG with the actual codebase — indexed my project with langchain + chroma, retrieval works but the context window fills fast. 7B model only has 4k context (8k if I push num_ctx), and the retrieved chunks eat 2k tokens before the prompt.

# Current workaround: post-generation lint loop
import subprocess
import ast

def validate_python(code: str) -> tuple[bool, str]:
    try:
        ast.parse(code)
        result = subprocess.run(
            ["ruff", "check", "--select=F401,F821", "-"],
            input=code.encode(),
            capture_output=True,
            timeout=5
        )
        return result.returncode == 0, result.stderr.decode()
    except SyntaxError as e:
        return False, str(e)

Run the generated code through this, feed errors back as a follow-up prompt, max 3 iterations. Gets me to ~85% compilable on first try, but the latency adds up — 12-18 seconds per usable snippet.

Questions for anyone doing this in production:

• Are you fine-tuning a small model on your framework's patterns, or just accepting the retry loop?
• Has anyone tried guidance / lmql style constrained generation to force valid imports?
• For local models, is 7B just too small for reliable codegen, or am I prompting wrong?

The num_ctx bump to 8192 helps retrieval but slows inference noticeably on my 24GB VRAM. Considering switching to a 13B quant (q4_k_m) and accepting slower tokens for better reasoning.