Building a Custom Eval Benchmark for Domain-Specific Python Code Generation

Generic benchmarks like HumanEval are useless for professional work because they test "LeetCode" logic, not how an AI handles a 10k-line proprietary codebase or a niche library like PyTorch Lightning or FastAPI. To actually measure if Claude 3.5 Sonnet or GPT-4o is getting better at my specific domain, I built a "Golden Set" eval pipeline that runs locally.

The core idea is to stop eyeballing the results and start treating AI prompts like unit tests. I created a directory of test_cases where each case is a JSON file containing a prompt, a reference implementation, and a suite of pytest-style assertions.

Here is the basic structure of my eval script. I use a simple Python wrapper that iterates through the cases, hits the API, and pipes the output directly into a temporary file to be executed by pytest.

import subprocess
import json
from anthropic import Anthropic

client = Anthropic(api_key="your_key")

def run_eval(case):
    # Extract only the code from the AI response
    response = client.messages.create(
        model="claude-3-5-sonnet-20240620",
        max_tokens=1024,
        messages=[{"role": "user", "content": case['prompt']}]
    )
    code = extract_code(response.content[0].text)
    
    with open("temp_solution.py", "w") as f:
        f.write(code)
    
    # Run the associated test file against the generated solution
    result = subprocess.run(["pytest", case['test_file'], "temp_solution.py"], 
                            capture_output=True, text=True)
    return result.returncode == 0

The biggest "gotcha" I found was prompt leakage. If you put the expected output in the prompt to "guide" the AI, your benchmark is fake. I shifted to a "Zero-Shot" approach where the prompt only contains the business requirement and the API documentation for the internal libraries being used.

To make this actually productive, I integrated it with a .cursorrules file. Once I identify a common failure point in my custom benchmark (e.g., the AI constantly forgets to handle async sessions in my specific wrapper), I add a constraint to the .cursorrules and rerun the entire eval set to see if the "Pass Rate" increases.

My current setup for maximizing accuracy:

Context Injection: Instead of letting the AI guess the API, I feed it a condensed api_summary.md containing only the method signatures and docstrings of the domain-specific classes.

Execution Sandbox: I run the evals inside a Docker container. Since I'm testing code generation, letting an LLM execute arbitrary code on my host machine is a recipe for disaster.

Failure Analysis: I don't just track pass/fail. I log the "diff" between the AI's output and the reference solution. Often, the AI is logically correct but fails because of a naming convention—this tells me I need to tighten the prompt, not change the model.

This workflow turned my "I think it's getting dumber" feelings into actual data. I discovered that switching from a generic system prompt to one that explicitly defines my project's type-hinting standards boosted my domain-specific pass rate from 62% to 84%.