How to Build a Custom LLM-Based Auto-Labeling Pipeline for Medical Datasets

Medical data is a nightmare to label because you can't just hire a cheap crowd-sourced workforce; you need actual clinicians who charge by the hour. I've been using Cursor paired with Claude 3.5 Sonnet to build an auto-labeling pipeline that uses a "LLM-as-a-Judge" pattern to pre-process thousands of radiology reports, leaving only the ambiguous cases for human review.

The biggest mistake people make is sending a raw document to an LLM and asking for a label. That leads to hallucinated categories. Instead, I built a verification loop. I use a local Python script to slice the data, send it to the LLM with a strict schema, and then run a second "critic" prompt to verify the logic.

Here is the core logic I used to structure the prompts for high-precision labeling. I found that providing a "Reasoning" field before the "Label" field (Chain-of-Thought) drastically reduces errors in medical entity extraction:

import openai

def label_medical_text(text, guidelines):
    prompt = f"""
    Guidelines: {guidelines}
    Patient Note: {text}
    
    Task: Analyze the note. 
    1. Extract relevant evidence.
    2. Determine if the evidence meets the criteria for 'Positive' or 'Negative'.
    3. Output strictly in JSON format.
    
    Format:
    {{
      "reasoning": "step-by-step analysis of the text",
      "label": "Positive/Negative/Uncertain",
      "confidence": 0.0-1.0
    }}
    """
    # Using Claude 3.5 Sonnet via API for superior medical nuance
    response = openai.chat.completions.create(
        model="claude-3-5-sonnet",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"}
    )
    return response.choices[0].message.content

To make this production-ready, I integrated a "Confidence Threshold" filter. If the LLM returns a confidence score below 0.8, the record is automatically flagged for manual review in a CSV. This turned a 3-month labeling project into a 2-week task.

A few config tips and gotchas I hit along the way:

The "System Prompt" Trap: Don't put the medical guidelines in the system prompt. I found that putting them in the user prompt as a reference block makes the model follow the constraints more rigidly.

Token Window Management: Medical reports can be rambling. I used Cursor's @Codebase feature to quickly write a preprocessing script that strips out boilerplate headers (hospital names, timestamps) before sending the text to the API. This saved me about 20% on token costs.

Handling Hallucinations: I implemented a "Negative Constraint" in the prompt. I explicitly told the model:

 the text does not explicitly mention X, you MUST label it as 'Uncertain' rather than guessing based on context.

This is critical for medical data where "absence of evidence is not evidence of absence."

Batching Strategy: Don't loop API calls one by one in a standard for loop. I used asyncio to hit the API in parallel batches of 10. It’s the only way to process 10k+ records without spending your entire day watching a progress bar.

import asyncio

async def process_batch(records):
    tasks = [label_medical_text(rec, guidelines) for rec in records]
    return await asyncio.gather(*tasks)

The productivity gain here isn't just in the speed of labeling, but in the consistency. Humans get tired and change their labeling criteria halfway through a dataset. The LLM doesn't. Once I locked in the prompt version, the labels were perfectly consistent across the entire corpus.