Giving a model a snippet of your old, buggy code is a great way to ensure it writes those same bugs again.
The setup for the experiment
I wanted to see if existing code dictates what a model generates next. I used a football quiz app where I had previously spent seven pull requests deleting eleven different versions of a search box. Each of those eleven had drifted—some had broken arrow key behavior, others failed to keep the mobile keyboard down, and one even submitted stale results. Eventually, I migrated everything to a single shared component.
To test the AI, I asked for a footballer search box for a new game screen, instructing the model to follow the conventions of the provided code. I ran this eight times per model under two different conditions at a temperature of 0.7:
1. The "Forked Era" (Commit 82385a5): I provided about 400 lines of source from a commit where nine separate search components existed and no shared version was available.
2. Post-Migration (Commit cc78828): I provided 400 lines from the commit that finalized the migration, where the shared component was active and being called by the TeamTies module.
I used DeepSeek v4 Pro, Llama 4 Maverick, GLM 5.3, and Mistral 3 14B (I had to stick to open weights since my account tier was hitting 403 errors with Claude and GPT).
The results were unanimous
Across 32 runs, there was zero crossover. The models behaved exactly according to the context provided:
- Context from the forked era: 0% reused the shared search; 100% hand-rolled a new one (median length: 190 lines).
- Context from post-migration: 100% reused the shared search; 0% hand-rolled a new one (median length: 41 lines).
Models reproduce specific bugs, not just the structure
The most alarming part is that the AI doesn't just mimic the "shape" of the code; it reproduces the specific defects. I compared the 33 hand-rolled components generated from the old context against the test list I wrote to prevent this kind of drift.
The results were brutal:
- 0% of the generated components implemented arrow key wrap-around at the ends of the list.
- 0% used the
SEARCH_MIN_QUERY_LENGTHconstant. Instead, 97% hard-coded>= 2, which is exactly what my repo's documentation warns against. - 58% exhibited other specific behavioral drifts identified in my migration tests.
If you're using a tool like Cursor or Claude Code and you've got a messy codebase, the AI is going to lean into that mess. It sees a pattern of "copy-paste and tweak" and assumes that's the architectural goal. The only way to get the model to use your shared components is to make sure the context it's indexing is the cleaned-up version, not the legacy wreckage.
All Replies (3)
I'm exhausted just reading this. Does this actually work with TypeScript 5.4 or is it just a pipe dream?
I'm curious if this happens more with 4-bit quantized models. I ran into a weird loop using llama.cpp recently...
I'm curious if this happens more with certain providers. Does this happen with vLLM or just the managed APIs?