Building a Custom Evaluation Benchmark for Domain Specific Code Generation

Generic benchmarks like HumanEval are useless once you start working with proprietary frameworks or niche domain languages because the LLM might know Python, but it doesn't know your company's internal DataPipeline API. I spent the last month building a private evaluation harness to stop "vibes-based" testing and actually measure if my Cursor rules and Claude 3.5 Sonnet prompts were improving code quality or just making the code look cleaner.

The biggest mistake most people make is trying to write a massive test suite manually. Instead, I used a "synthetic-to-real" pipeline. I fed 50 high-quality existing PRs into Claude, asked it to strip the implementation and turn them into "Challenge Prompts" with corresponding unit tests.

Here is the basic architecture of the eval runner I built in Python:

import subprocess
import json

def run_eval(prompt, ground_truth_test):
    # 1. Get generation from AI (via API or cached file)
    generated_code = get_llm_response(prompt) 
    
    # 2. Write to a temporary file
    with open("temp_eval.py", "w") as f:
        f.write(generated_code)
    
    # 3. Run the test case against the generated code
    result = subprocess.run(["pytest", "temp_eval.py"], capture_output=True)
    
    return "PASS" if result.returncode == 0 else "FAIL"

To make this actually work for domain-specific code, you have to solve the "Context Gap." If the AI doesn't have the API docs, it will hallucinate method names. I found that adding a .cursorrules file isn't enough for benchmarking; you need a dedicated context_injection step in your eval script.

My current benchmark workflow:

• The Golden Set: A JSON file containing 100 pairs of (prompt, test_case, required_context_files).
• Context Injection: The script reads the required_context_files and prepends them to the prompt as "Reference Documentation."
• Pass@1 Metric: I run each prompt once. If it fails, I log the traceback. This is the only metric that matters for developer productivity.

One major gotcha: LLMs love to wrap code in markdown blocks (

 ...

). If your eval script just pipes the output to a file, it'll crash. I had to write a regex cleaner to strip the markdown and handle the cases where the AI adds "Here is the code:" text at the top.

import re

def clean_code(raw_output):
    pattern = r"

(?:python)?\s(.?)\s*

"
    match = re.search(pattern, raw_output, re.DOTALL)
    return match.group(1) if match else raw_output

The real productivity gain happened when I started using this to test my system prompts. I realized that telling the AI to "be concise" actually dropped my pass rate by 12% because it started skipping necessary imports. By having a local benchmark, I can iterate on my prompt instructions and know exactly when I've broken a specific edge case.

If you're doing this, don't bother with complex frameworks. A simple Python script that triggers pytest or npm test on generated snippets is all you need to move from "I think this prompt is better" to "This prompt increased accuracy from 65% to 82%."