JevBench confirms typed decision models surpass LLMs in both speed and cost efficiency

JevBench's typed decision models outpace LLMs on cost and speed.

Standard large language models drag behind on even simple classification work, burning budget on full text generation whenever a bounded choice plus a probability would suffice. That mismatch led to testing Jev-class models, which emit discrete outputs rather than streaming tokens, and measuring them with JevBench, a reproducible benchmark built for typed decision models.

Rather than ranking models on accuracy alone, JevBench blends chance-corrected Intelligence, Calibration, Speed, and Cost into a weighted composite score for each run. Version v1.3 consolidates results from 534 English decision tasks into that final number, giving a fuller picture of practical value than leaderboards that ignore inference time or API bills.

The latest data shows a tight cluster at the top, with a sharper drop below it:

• Jev: 74.4
• SemIf: 73.1
• djev: 73.0
• Winnow-12B Q8: 71.2
• reflex 4B: 70.3

Jev holds first place by 1.3 points over SemIf, and with djev a tenth behind, the gap sits within the noise range flagged by the creator, where roughly one point can come down to randomness. Winnow-12B Q8 and reflex 4B fall further behind, trailing by over three points, marking a clearer performance decline.

The harness runs openly under an MIT license, exposing every task, frozen artifact, and scoring script, so per-task breakdowns can be inspected directly. Readers should note the benchmark covers English only, so multilingual workloads fall outside its scope. Latency figures trace to a single German server, with the author applying a stated ×2 adjustment for local or demo runs and adding a flat 150 ms buffer to absorb network variance, meaning local comparisons may shift. Held-out prompts still reach the evaluated endpoints during testing, though the effect appears minor here.

For a quick sanity check before heavier integration, two no-signup demos are available: who-is-right.app.mintapis.com for general decision correctness, and is-it-ai-slop.app.mintapis.com for AI-generated content flagging.

The takeaway is that typed decision models gain speed and cost on structured workloads like query routing, intent classification, or content filtering. With Jev, SemIf, and djev bunched near the top, open-source options now sit close enough that paying premium rates for small gains rarely pays off unless calibration needs line up precisely with the leader. Models below 73 points keep a strong balance between efficiency and performance for most uses.