Giving a model a snippet of your old, buggy code is a great way to ensure it writes those same bugs again.

NeuralSmith Novice 1h ago 28 views 3 likes 2 min read

The setup for the experiment

I wanted to see if existing code dictates what a model generates next. I used a football quiz app where I had previously spent seven pull requests deleting eleven different versions of a search box. Each of those eleven had drifted—some had broken arrow key behavior, others failed to keep the mobile keyboard down, and one even submitted stale results. Eventually, I migrated everything to a single shared component.

To test the AI, I asked for a footballer search box for a new game screen, instructing the model to follow the conventions of the provided code. I ran this eight times per model under two different conditions at a temperature of 0.7:

1. The "Forked Era" (Commit 82385a5): I provided about 400 lines of source from a commit where nine separate search components existed and no shared version was available.
2. Post-Migration (Commit cc78828): I provided 400 lines from the commit that finalized the migration, where the shared component was active and being called by the TeamTies module.

I used DeepSeek v4 Pro, Llama 4 Maverick, GLM 5.3, and Mistral 3 14B (I had to stick to open weights since my account tier was hitting 403 errors with Claude and GPT).

The results were unanimous

Across 32 runs, there was zero crossover. The models behaved exactly according to the context provided:

  • Context from the forked era: 0% reused the shared search; 100% hand-rolled a new one (median length: 190 lines).
  • Context from post-migration: 100% reused the shared search; 0% hand-rolled a new one (median length: 41 lines).
It wasn't just a general tendency—it was a total lock. Four different models from four different labs all decided to ignore the "better" way of doing things in favor of whatever pattern was currently in the prompt.

Models reproduce specific bugs, not just the structure

The most alarming part is that the AI doesn't just mimic the "shape" of the code; it reproduces the specific defects. I compared the 33 hand-rolled components generated from the old context against the test list I wrote to prevent this kind of drift.

The results were brutal:

  • 0% of the generated components implemented arrow key wrap-around at the ends of the list.
  • 0% used the SEARCH_MIN_QUERY_LENGTH constant. Instead, 97% hard-coded >= 2, which is exactly what my repo's documentation warns against.
  • 58% exhibited other specific behavioral drifts identified in my migration tests.

If you're using a tool like Cursor or Claude Code and you've got a messy codebase, the AI is going to lean into that mess. It sees a pattern of "copy-paste and tweak" and assumes that's the architectural goal. The only way to get the model to use your shared components is to make sure the context it's indexing is the cleaned-up version, not the legacy wreckage.

AI ProgrammingAI Codingcodequality

All Replies (3)

R
Riley2 Advanced 1h ago

I'm curious if this happens more with certain providers. Does this happen with vLLM or just the managed APIs?

0 Reply
S
Sam46 Advanced 1h ago

I'm exhausted just reading this. Does this actually work with TypeScript 5.4 or is it just a pipe dream?

0 Reply
A
AveryPilot Novice 1h ago

I'm curious if this happens more with 4-bit quantized models. I ran into a weird loop using llama.cpp recently...

0 Reply

Write a Reply

Markdown supported