Why your LLM architecture is failing your accuracy targets

Your medical coding recall will never hit 90% through prompting alone. We hit a wall at 60% trying to force a probabilistic model to act like a deterministic rulebook. We were building a platform for CPT and ICD assignment, and despite feeding the model more context and upgrading to larger models, we couldn't bridge that gap.

The mistake is assuming the model lacks "intelligence" when the reality is a mismatch in logic types. LLMs guess the next likely token; they don't inherently respect the National Correct Coding Initiative (NCCI) constraints. These administrative rules live in regulatory tables, not in the clinical notes the model is reading. You can fine-tune all day, but if the model doesn't "know" a specific code pair is forbidden by a hard rule, it will keep hallucinating valid-looking but incorrect combinations.

We stopped wasting money on larger context windows and bigger models and built a deterministic validation layer instead. The new workflow is simple: the LLM extracts candidate codes from the unstructured text, then a hard-coded engine checks those candidates against the full NCCI edit set.

Moving the logic out of the prompt and into a post-processing pipeline took our recall from 45% to 92%. If you're trying to solve a regulatory or math-heavy problem with better prompting, you're just throwing compute at a structural flaw. Stop trying to make the model smarter and start making the architecture more rigorous.

https://www.cms.gov/medicare/coding-billing/ncci-edits