Detecting medical misdiagnosis via coherence scoring

Twelve million US adults hit with diagnostic errors every year is a stat that basically haunts my dreams. When you realize that's one in every twenty outpatient visits, you realize the stakes aren't just "cool tech project"—they're massive. I spent my senior year trying to see if we could build a "Misdiagnosis Detector" to act as a skeptical second opinion, essentially a bot that flags when symptoms and conclusions are straight-up vibing in different directions.

The initial dev phase was a total "bruh" moment. I tried the standard student move: supervised learning. I scraped PubMed to train a classifier to map symptoms to diagnoses. On paper, it looked cracked—90% accuracy. But once I dug deeper, I realized the model was basically cheating by pattern-matching the publication bias in medical journals. It was great at the easy stuff but completely useless at catching the actual errors that matter. It was all hype and no substance.

I had to pivot hard. Instead of trying to predict what a doctor should say, I started training the model to score "coherence." This meant ditching the obvious mismatches and hunting for "hard negatives"—cases that sound plausible, like a migraine versus meningitis, but are fundamentally wrong.

Since I didn't have a billion-dollar budget to train a massive medical LLM from scratch (RIP my bank account), I went the smart route and leveraged PubMedBERT embeddings. By using those pre-trained biomedical embeddings, I could finally train a layer on top that actually understood the nuances of medical text. It shifted the whole project from a basic pattern matcher to a tool that senses the tension between symptoms and conclusions. It’s still a WIP, but moving from raw text to high-quality medical embeddings was the real value-for-money play here.

https://pubmed.ncbi.nlm.nih.gov/