Building a Custom Evaluation Benchmark for Domain-Specific Python Code Generation

Stop relying on HumanEval or MBPP for domain-specific logic; they are too generic to tell you if your LLM actually understands your proprietary SDK or a niche scientific library. When I was building a custom wrapper for a financial data API, I found that Claude 3.5 Sonnet could write "correct" Python, but it constantly hallucinated method signatures because the library's documentation wasn't in its core training set.

The only way to stabilize this is a private evaluation benchmark. Instead of manual "vibe checks," I built a lightweight test harness that treats LLM outputs as candidates for a unit test suite.

The core architecture is simple: a JSON file containing Prompt, Expected Output (or Test Case), and Reference Implementation.

[
  {
    "id": "fin_calc_01",
    "prompt": "Calculate the CAGR for a portfolio given initial_value=1000 and final_value=1500 over 5 years using the internal FinanceLib.calculate_growth function.",
    "test_code": "def test_cagr():\n    res = FinanceLib.calculate_growth(1000, 1500, 5)\n    assert abs(res - 0.0844) < 0.001"
  }
]

To automate this, I wrote a runner script that pipes the prompt to Cursor's Composer (or via API) and executes the resulting code in a sandboxed subprocess. Here is the logic I used to handle the execution and capture failures:

import subprocess

def evaluate_candidate(generated_code, test_code):
    # Combine the generated code and the test case
    full_script = f"{generated_code}\n\n{test_code}\n\ntest_cagr()"
    
    try:
        process = subprocess.run(
            ["python3", "-c", full_script], 
            capture_output=True, 
            text=True, 
            timeout=5
        )
        return process.returncode == 0, process.stderr
    except subprocess.TimeoutExpired:
        return False, "Timeout"

A few hard-won tips on making this actually work:

The "Context Window" Trap
Don't just test the prompt. To get these benchmarks to pass, I had to create a .cursorrules file that explicitly mapped the library's API surface. I found that providing a "Cheat Sheet" of method signatures in the system prompt increased the pass rate from 40% to 85%.

Handling Non-Deterministic Output
LLMs love to add conversational filler. Use a regex or a specific delimiter like `python to strip everything except the code. If you're using Claude Code, you can force a concise output by adding "Output only the code, no explanations" to the prompt, but the regex approach is more robust.

The Golden Set
Build a "Golden Set" of 50-100 cases. Every time I update my prompts or switch from GPT-4o to Claude 3.5, I run the entire suite. This is the only way to know if a "fix" for one bug actually broke three other things in the domain logic.

Productivity Gain
Since implementing this, I've stopped wasting time manually running snippets in a REPL. I can now iterate on a complex prompt, run the benchmark script, and see a "Pass/Fail" percentage in seconds. It turns AI coding from a guessing game into an engineering process.