How to build a custom semi-automated data labeling tool using LLMs

The bottleneck in most ML projects isn't the model architecture, it's the ground truth data. I got tired of spending hours manually labeling JSON objects for a niche sentiment analysis project, so I built a "human-in-the-loop" pipeline using Cursor and a lightweight Streamlit frontend. The goal was to let an LLM do the first pass, and have me only intervene when the model's confidence was low.

The architecture is simple: a Python script that reads raw data, sends it to Claude 3.5 Sonnet with a strict rubric, and saves the output to a local SQLite DB with a verified flag set to false.

To make this work, the prompt is everything. If you just ask an LLM to "label this," you'll get inconsistent results. I used a structured prompt that forced the model to provide a reasoning chain before the final label. This actually makes the labels more accurate and, more importantly, helps me understand why the AI made a mistake when I'm reviewing.

# Example of the labeling prompt structure I used
prompt = f"""
Analyze the following text based on these categories: [Urgent, Routine, Spam].
Rules:
- Urgent: Requires action within 2 hours.
- Routine: General inquiry.
- Spam: Marketing or irrelevant.

Text: {user_input}

Output format:
Reasoning: <brief explanation>
Label: <Category>
Confidence: <0.0 to 1.0>
"""

I used Cursor's @Codebase feature to rapidly scaffold the Streamlit UI. I basically told Cursor: "Create a dashboard that queries the SQLite DB for all rows where verified=0, shows the text and the LLM's label, and gives me two buttons: 'Confirm' and 'Correct'."

One major productivity gain came from implementing a "Bulk Confirm" feature. Since the LLM gets about 85% of the easy cases right, I filtered the view to only show items where the LLM's self-reported Confidence was below 0.8. This cut my manual review time by nearly 70%.

Key config tips for this setup:

Use JSON mode for the LLM output. If you're using the API, set response_format={"type": "json_object"}. It prevents the parser from crashing when the LLM decides to add "Here is the label:" at the start of the response.

Batching requests is a must. Don't send one request per row if you have 10k items; you'll hit rate limits and waste tokens. I grouped 20 items into a single prompt and asked for a JSON array back.

Implement a 'Gold Set'. I manually labeled 50 examples first. Every time I update the prompt, I run the LLM against this gold set. If the accuracy drops, I know the prompt tweak broke something.

The biggest gotcha was "label drift." I noticed that after 500 rows, the LLM started interpreting "Urgent" more broadly. To fix this, I injected 3-5 "anchor" examples (few-shot prompting) into every batch request to keep the model calibrated.

# Quick start for the environment
pip install streamlit openai pandas
streamlit run app.py

This setup transforms the task from "data entry" to "data auditing." Instead of staring at a blank box, I'm just clicking 'Yes' or 'No' on a suggested label, which is mentally much lighter and significantly faster.