JevBench proves typed decision models beat LLMs on speed and cost

Standard large language models are notoriously slow and expensive for simple classification tasks. If you just need a bounded choice with a probability attached, you shouldn't be paying for full text generation. That’s why I started looking closely at Jev-class models, which return discrete outputs rather than streaming tokens. To see how the real Jev model stacks up against open-source lookalikes and other competitors, I used JevBench, a new reproducible benchmark designed specifically for typed decision models.

The core idea is straightforward: measure accuracy, latency, and price simultaneously, then weigh them according to your priorities. JevBench doesn't just look at raw accuracy; it combines chance-corrected Intelligence, Calibration, Speed, and Cost into a single composite score. This gives a much clearer picture of practical utility than standard leaderboards that often ignore inference time or API costs.

A full benchmark run evaluates 534 English decision tasks. The current version, v1.3, aggregates these metrics into a weighted final score. When I ran through the latest data, the hierarchy was quite tight at the top, but the gap between the proprietary model and the open-source alternatives was visible.

Here is the current leaderboard breakdown:

• Jev: 74.4
• SemIf: 73.1
• djev: 73.0
• Winnow-12B Q8: 71.2
• reflex 4B: 70.3

The fact that Jev sits at #1 with a 74.4 score suggests it holds an edge, but SemIf and djev are dangerously close. The difference between Jev and SemIf is only 1.3 points. Given the limitations listed by the creator, where ~1-point gaps can be attributed to noise, these results indicate that high-quality open-source or "fake lookalike" projects are genuinely competitive. The drop-off happens more noticeably with Winnow-12B Q8 and reflex 4B, which lag behind by over 3 points.

The benchmarking process is open and reproducible. The harness uses MIT licensing, and all items, frozen artifacts, and scoring code are public. You can even inspect the per-task outcomes yourself. This transparency is crucial for verifying claims in a space where marketing often obscures actual performance.

There are some constraints to keep in mind when interpreting these numbers. The benchmark is English-only, which limits its applicability for multilingual decision tasks. Latency measurements were taken from a single server located in Germany. The author applies a disclosed ×2 adjustment for local or demo latency, adding a flat 150 ms buffer on their servers to account for network variance. This is an informed assumption, but it means direct millisecond comparisons with your local setup might vary. Additionally, held-out prompts still reach the evaluated services during testing, though the impact appears minimal given the controlled environment.

If you want to test the models yourself without setting up the full harness, there are two no-signup demos available. One checks general decision correctness via who-is-right.app.mintapis.com, and the other focuses on detecting AI-generated content via is-it-ai-slop.app.mintapis.com. These are useful for quick sanity checks before committing to a heavier integration.

The key takeaway is that if your workload involves structured decisions—like routing queries, classifying intents, or filtering content—typed decision models offer a distinct advantage in speed and cost. While Jev currently leads the board, the tight clustering with SemIf and djev shows that the open-source ecosystem is catching up fast. You don't need to pay premium prices for marginal gains unless your specific calibration needs align perfectly with the top-tier model. For most applications, the sub-73 models provide a compelling balance of performance and efficiency.