Can LLMs actually grade their own homework to get better results?

Most of us are stuck in a frustrating trade-off when it comes to prompt engineering. You either go the few-shot route, where you spend hours hand-crafting perfect examples only to find that changing one word in a demo tanked your accuracy, or you go zero-shot and just pray the "think step-by-step" magic keeps the model from hallucinating a confident lie. One path is a manual labor nightmare; the other is a gamble on reliability.

Consistency-based Self-adaptive Prompting (COSP) basically solves this by turning the LLM into its own data annotator. Instead of you writing the examples, the model generates them, and then uses a consistency check to figure out which ones are actually worth keeping.

The logic behind the loop

The core insight here is that variance in LLM outputs isn't just "noise"—it's a signal. If you ask a model a complex reasoning question five times and it gives you the same answer four times, it's likely correct. If it gives you five different answers, it's guessing.

COSP leverages this by treating the model's own consistent answers as "pseudo-labels." It effectively creates its own few-shot prompt on the fly without a human ever having to write a single example. You get the guidance of few-shot prompting with the zero-effort setup of zero-shot.

A real-world AI workflow for COSP

If you want to implement this logic into your own LLM agent or pipeline, it follows a specific two-stage process:

1. The Generation Phase: You feed the model a batch of queries. For each query, you generate multiple independent responses (sampling with a higher temperature).
2. The Filtering Phase: You look for consistency. Any response that matches the majority vote for that specific query is flagged as "high confidence."
3. The Adaptive Prompting Phase: You take those high-confidence pairs (the original question and the consistent answer) and feed them back into the prompt as in-context examples for the remaining unsolved queries.
4. Final Vote: The model generates a final set of answers based on these self-generated examples, and you take the majority vote again.

Performance Breakdown

Comparing this to standard methods, the shift in reliability is noticeable, especially in reasoning-heavy tasks:

• Zero-shot: Low effort, but high variance. The model often wanders off-track.
• Manual Few-shot: High accuracy if examples are perfect, but extremely brittle and doesn't scale across different task types.
• COSP: Low effort (automated) and higher stability. It dynamically adapts the examples to the specific distribution of the current dataset.

For anyone building a production AI workflow, this is a huge win because it removes the "prompt tuning" bottleneck. You stop guessing which examples the model likes and let the model tell you what it finds consistent. It's a practical tutorial in self-correction that makes LLMs feel significantly more robust.