Building a Custom Evaluation Benchmark for Domain-Specific Code Generation Tasks

Generic benchmarks like HumanEval are useless once you move into specialized domains—like writing custom DSLs for hardware description or complex financial state machines. I spent the last month building a private eval set for a proprietary API integration project because Claude 3.5 Sonnet was hallucinating method signatures that didn't exist in our internal SDK.

The core problem is that "correctness" in domain-specific code isn't just about passing a unit test; it's about adhering to architectural constraints that the LLM doesn't see in its training data.

My approach was to move away from manual "vibe checks" and build a lightweight automated pipeline. I created a JSONL dataset where each entry contains a prompt, a golden_reference (the ideal implementation), and a test_suite (a Python script that executes the generated code against specific edge cases).

To make this work with Cursor, I leveraged the .cursorrules file to force the model to adhere to the benchmark's constraints during the iterative refinement phase. Here is the logic I used to automate the scoring:

import subprocess

def evaluate_generation(generated_code, test_script):
    # Write generated code to a temporary file
    with open("temp_gen.py", "w") as f:
        f.write(generated_code)
    
    # Run the test script against the generated code
    result = subprocess.run(["python3", test_script], capture_output=True, text=True)
    
    if result.returncode == 0:
        return "PASS"
    else:
        return f"FAIL: {result.stderr}"

One major gotcha: LLMs often wrap code in markdown blocks or add conversational filler, which breaks the execution script. I had to write a regex pre-processor to strip everything except the raw code before passing it to the evaluator.

To actually improve the model's performance, I used a "Failure-Driven Prompting" loop. Whenever a prompt failed the benchmark, I fed the error message and the golden_reference back into the system prompt as a "Few-Shot" example.

My current config for the evaluation pipeline:

• Dataset format: JSONL with input, expected_output, and validation_script.
• Execution environment: Dockerized containers to prevent the AI from accidentally running rm -rf / during a benchmark run.
• Metric: Pass@1 (since in a real workflow, I'm not accepting 10 variations of the same function).

The productivity gain was immediate. Instead of spending two hours manually testing if a prompt change fixed a bug, I can now run a script across 50 domain-specific cases and see a precision percentage. It turns out that adding "Always use the AsyncSession context manager for DB calls" to the system prompt increased my Pass@1 rate from 60% to 92% for our specific codebase.

If you're doing this, don't over-engineer the benchmark. Start with 20 "hard" cases that the AI consistently fails. That's your baseline. Once you hit 100% on those, add the next 20. Using a tool like Claude Code to help write the validation_scripts themselves is a massive time-saver, as long as you verify the tests are actually testing the right thing.