Model Collapse: Are New Coding LLMs Training on Old AI Slop?
The Feedback Loop Problem
When we look at the current AI workflow, the volume of synthetic data is exploding. A developer uses an LLM to scaffold a project, pushes that code to a public repo, and then a future training scrape picks up that exact snippet. The danger here isn't just that the code is "bad," but that it's "average." Human code contains intent, architectural trade-offs, and often, the "ugly" fixes that actually work in production. AI slop tends to be syntactically correct but logically hollow.
If a new model trains on a million instances of a slightly wrong implementation of a sorting algorithm because that's what the previous model hallucinated most often, the new model will perceive that error as the "standard" way to write the function.
How Modern Training Mitigates This
To prevent this slide into mediocrity, researchers are moving toward more sophisticated data curation. It's no longer about scraping the entire web; it's about high-signal filtering. Here is how the industry is attempting to avoid the slop:
- Quality Scoring: Using "reward models" to score code snippets. If a piece of code doesn't pass a linter or fails a unit test, it's discarded regardless of how many times it appears in the training set.
- Synthetic Data Generation (The Good Kind): Instead of training on "wild" AI slop, developers are creating high-quality synthetic datasets. This involves using a frontier model (like Claude 3.5 Sonnet) to generate a problem, then using a compiler to verify the solution, and only then adding it to the training set.
- Weighting Human Data: Giving higher priority to "gold standard" repositories—projects with high stars, rigorous PR reviews, and long histories of human maintenance.
Real-World Implications for Prompt Engineering
For those of us focusing on prompt engineering, this means we can't trust the "most common" answer the AI gives us. Since the models are exposed to so much synthetic noise, the "average" response is often the safest, most generic version of the code.
To get around this, I've found that forcing the model to think through a "first-principles" approach or providing a specific, high-quality reference implementation in the prompt helps bypass the generic patterns the model has picked up from the AI-generated noise of the web. The goal is to push the LLM away from the "average" synthetic response and back toward the logic of a senior engineer.