Building a Custom Python Tool for Automated LLM Dataset Labeling

Cursor's "Composer" mode (Cmd+I) is a cheat code for building internal tooling like this, but the real trick is how you feed it the context of your data schema to avoid "hallucinated" labels. I spent the weekend building a Python script to label 5,000 rows of messy customer feedback for a sentiment-aspect project, and I managed to hit 94% alignment with human labels by using a "few-shot validator" loop.

The core architecture is a simple Python script using pandas and litellm (to easily swap between Claude 3.5 Sonnet and GPT-4o). The biggest gotcha is that LLMs tend to drift in their labeling consistency over long files. If you just loop through a CSV, the model starts getting "lazy" halfway through.

To fix this, I implemented a sliding window prompt that injects 3-5 gold-standard examples into every request. Here is the basic logic I used to structure the labeling loop:

import pandas as pd
from litellm import completion

def label_data(row, gold_examples):
    prompt = f"""Label the following text based on these examples:
    {gold_examples}
    
    Text: {row['text']}
    Label: """
    
    response = completion(
        model="claude-3-5-sonnet-20240620", 
        messages=[{"role": "user", "content": prompt}],
        temperature=0
    )
    return response.choices[0].message.content.strip()

# Load data and gold set
df = pd.read_csv("raw_data.csv")
gold_set = "Text: 'Too slow' -> Label: Performance\nText: 'UI is ugly' -> Label: UX"

df['label'] = df.apply(lambda x: label_data(x, gold_set), axis=1)

The productivity jump happened when I stopped writing the boilerplate myself. I used Cursor to generate the error handling and the CSV chunking logic. I specifically used a .cursorrules file in the project root to tell the AI: "Always use type hints, prefer f-strings, and implement a checkpoint system that saves the CSV every 100 rows so I don't lose progress if the API times out."

If you're doing this, don't trust the first pass. I found that adding a "reasoning" step significantly boosts accuracy. Instead of asking for just the label, I force the model to output a JSON object:

{
  "reasoning": "The user mentions the app crashes on startup, which is a technical failure.",
  "label": "Stability"
}

I then used a regex to extract the label. This forces the LLM to "think" before committing to a category, which is basically a manual implementation of Chain-of-Thought.

Pro tips for your config:

• Temperature to 0: This is non-negotiable for labeling. You want deterministic outputs, not creativity.
• Batching: Don't send one request per row if your provider supports batch APIs; it'll cut your costs by 50%.
• The "Unknown" Category: Always give the AI an "Other" or "Unsure" label. If you force it to choose from a fixed list, it will hallucinate a category just to satisfy the prompt.

The biggest productivity gain wasn't the labeling itself, but using Claude Code to write a quick evaluation script that compared the AI labels against a small human-verified test set. It flagged exactly which categories the model was confusing (e.g., "Pricing" vs "Value"), allowing me to refine the gold examples in the prompt until the accuracy plateaued.