How to build a custom evaluation benchmark for domain-specific LLM tasks

Most "off-the-shelf" benchmarks like MMLU or GSM8K are basically useless once you move into specialized domains like legal discovery or high-frequency trading logs because they measure general intelligence rather than domain precision. If you're trying to decide between Claude 3.5 Sonnet and GPT-4o for a niche pipeline, you can't trust the leaderboard; you need a "golden dataset" that actually reflects your edge cases.

The biggest mistake I see is people relying on "vibes-based" evaluation—prompting a model ten times and deciding it "feels" better. To get actual delta measurements, you need a structured evaluation harness.

Start by curating 50-100 high-quality pairs of inputs and ground-truth outputs. Don't over-engineer the quantity; 50 gold-standard examples are worth more than 1,000 noisy ones. For domain-specific tasks, I prefer using a "LLM-as-a-Judge" pattern but with a twist: you must provide the judge model with a strict rubric. If you just ask "which is better," the judge will lean toward the longer, more polite response (the "verbosity bias").

Here is how I structure my evaluation prompts for the judge (usually using GPT-4o or Claude 3 Opus since they are the most critical):

System: You are a domain expert in [Insert Domain]. Evaluate the candidate response based on the ground truth.
Rubric:
- Accuracy: Does it contradict the ground truth? (Score 0-5)
- Hallucination: Does it introduce external facts not present in the context? (Score 0-5)
- Conciseness: Is there unnecessary filler? (Score 0-5)

Output only a JSON object: {"accuracy": X, "hallucination": X, "conciseness": X, "reasoning": "..."}

When I ran this against a set of complex SQL generation tasks, the results were eye-opening. GPT-4o tended to be faster and more syntactically correct, but Claude 3.5 Sonnet was significantly better at handling the complex joins and ambiguous column names specific to my schema. If I had relied on general benchmarks, I would have picked the wrong model.

For the actual implementation, avoid manual spreadsheets. I use a simple Python script to iterate through the test set and log results into a JSONL file.

Performance metrics to track:

• Exact Match (EM): Only for deterministic outputs (like IDs or specific codes).
• Semantic Similarity: Use cosine similarity via embeddings if the phrasing doesn't matter as much as the meaning.
• Failure Rate: Specifically track where the model "hallucinates" versus where it simply "fails to follow instructions."

The most frustrating part of this process is the "benchmark leak" problem. If your domain data is public, the model might have seen it during training. To combat this, I always include "synthetic perturbations"—slightly altering the names, dates, or values in my test set to ensure the model is actually reasoning and not just recalling a training sample.

If you're choosing between DeepSeek and the giants for a coding-heavy domain, you'll notice DeepSeek is incredibly competitive on raw logic, but often fails on the "nuance" of business requirements compared to Claude. Building your own eval is the only way to quantify exactly where that breaking point is.