AI COBOL-to-Java Migration: Bugs Included

A regional insurer's core policy administration system was just ported from COBOL to Java by an LLM in a single automated pass. The output compiled on the first try, passed the original test suite, and carried over every known bug — including a premium-rounding error that had been living in a COBOL COMPUTE statement since 1994.

That's not a failure. That's the whole point of legacy migration.

When you're moving a system that has been running for decades, the existing behavior is the specification. The accounting department has built workflows around that rounding error. The CRM team has a workaround for the leap-year glitch. If the AI had "fixed" those quirks, the migrated system would break downstream processes in ways that have nothing to do with the code itself. So preserving bugs is the correct outcome — it means the LLM understood the contract, not just the syntax.

Here's how the team actually pulled it off:

1. They treated the old COBOL as ground truth. No prompts like "modernize this" or "improve performance." Every instruction was scoped to a specific program, with the output format defined and the runtime environment pinned.
2. They built a golden-test harness first. Instead of relying on the AI to verify its own work, they fed it thousands of input-output pairs from the production logs. The AI was told to match those outputs exactly, down to the decimal point.
3. They used a file-by-file conversion pipeline. A small agent orchestrated the migration — reading the COBOL section, generating the Java class, compiling it, running the tests, and reporting diffs. The human reviewer only got involved when the diff count crossed a threshold.

That's where the prompt engineering mattered. The key wasn't a clever system prompt. It was embedding the test harness and the error history into the context so the model knew why the code looked the way it did. The AI still hallucinated two new off-by-one errors in a date-conversion routine — but the static analysis step caught them before anyone noticed.

If you're thinking about doing this yourself, here's the real lesson:

• The old code is the spec. Don't ask the LLM to improve, refactor, or clean up anything during migration. You'll lose behavior you didn't even know you depended on.
• The test suite is your safety net. The AI reproducing the old bugs is proof it learned the system's quirks. The test suite is what tells you it didn't invent new ones.
• You still need a human to review the diffs. The LLM's migration was impressive, but it flagged two suspicious loops that turned out to be its own mistakes. A human who knows the domain had to confirm those were hallucinations, not legacy behavior.

The real risk with LLM-based migration isn't that it copies bugs. It's that it silently produces new ones while making the old code look clean. The team's approach — treat the output with suspicion, compare it against a large corpus of real transactions, and keep the migration pipeline reversible — handled that risk well.

If you're planning something similar, start small. Pick one program, build a golden-test harness, run the migration, and diff the behavior. Once you see the AI reproduce