Solving the hallucination loop in AI generated unit tests

Morgan42 Novice 2h ago 279 views 6 likes 4 min read

I spent four hours last Thursday fighting a ghost in my test suite. I was using Claude 3.5 Sonnet via Cursor to generate unit tests for a TypeScript payment gateway module. The AI kept generating tests that looked perfect—beautifully structured, covering edge cases, and using Jest mocks—but they failed every single time I ran them.

Solving the hallucination loop in AI generated unit tests

The error was always some variation of:
TypeError: Cannot read properties of undefined (reading 'processPayment')

The wild part? The AI looked at the error, apologized, and gave me the exact same code back with a different comment. I was stuck in a "hallucination loop" where the AI assumed my mock was being injected correctly when, in reality, the dependency injection in my actual project used a custom singleton pattern that the LLM didn't grasp from the provided context.

Why the AI kept lying about the mocks

The problem wasn't the logic; it was the context window. I had attached the service file, but not the dependency-container.ts file where the actual instantiation happened. The AI assumed I was using standard Jest jest.mock() behavior, but my project required a manual override of the singleton instance.

I tried the "just fix it" prompt three times. Waste of time.

The breakthrough happened when I stopped asking the AI to "fix the error" and instead asked it to "explain how it thinks the PaymentService is being instantiated in the test environment." It admitted it was guessing based on common patterns.

Here is the comparison of what it gave me versus what actually worked:

| Attempt | AI's Assumption | Actual Reality | Result |
| :--- | :--- | :--- | :--- |
| 1 | jest.mock('./service') works automatically | Project uses a custom ServiceLocator | TypeError |
| 2 | Mock should be passed in constructor | Service is a singleton accessed via .getInstance() | TypeError |
| 3 | (After I provided the container file) Manual override of singleton | Manual override of singleton | Pass |

The fix that actually stopped the loop

Instead of letting the AI guess, I had to be explicit about the mocking strategy. I stopped using the generic "generate tests" prompt and switched to a specific implementation pattern.

The solution was to explicitly mock the singleton instance before the beforeEach block.

AI Learning Community, AI generated unit tests
// This is what actually worked after 4 hours of failure
import { ServiceLocator } from '../container';
import { PaymentService } from './payment.service';

jest.mock('../container');

describe('PaymentGateway', () => {
  let mockPaymentService: jest.Mocked<PaymentService>;

  beforeEach(() => {
    mockPaymentService = {
      processPayment: jest.fn(),
    } as any;
    
    // The critical missing piece the AI ignored:
    (ServiceLocator.getInstance as jest.Mock).mockReturnValue({
      paymentService: mockPaymentService
    });
  });

  it('should handle declined cards', async () => {
    mockPaymentService.processPayment.mockRejectedValue(new Error('Card Declined'));
    // ... rest of the test
  });
});

If you're hitting a wall where the AI keeps giving you the same broken code, stop prompting for the solution. Prompt for the assumption. Ask: "What do you believe is the current state of the variable X at line Y?" Usually, you'll find the AI is hallucinating a version of your architecture that doesn't exist.

Stop guessing and use a shared knowledge base

Doing this alone is a slog. I realized that while I was fighting this singleton bug, someone in the PromptCube community had already documented a similar struggle with the same model and the same architecture pattern.

The value of an AI learning community isn't just "getting prompts"; it's finding the people who have already failed in the specific way you are currently failing. When you're deep in AI Coding, you realize that 80% of the struggle is context management. Knowing how to structure your files so the AI doesn't hallucinate your dependency tree is a skill you don't learn from a documentation page—you learn it from other devs who have spent four hours on a Tuesday fighting a TypeError.

Improving the generation workflow

To stop this from happening again, I changed my entire approach to generating tests. I no longer ask for "unit tests for this file." I now provide a "Mocking Guide" as a .md file in my project root and reference it in every prompt.

My new Workflows for testing look like this:
1. Attach the target file.
2. Attach the MOCKING_GUIDE.md (which explains the singleton pattern).
3. Command: "Generate tests following the mocking patterns defined in MOCKING_GUIDE.md."

The result? The failure rate of generated tests dropped from about 60% (mostly mock-related errors) to under 10%.

If you're tired of the "apology loop" where the AI says "I apologize for the mistake" and then repeats the mistake, you need to move beyond basic prompting. Joining a community like PromptCube gives you access to these battle-tested patterns and a place to dump your failure logs so others don't repeat them. You can join by visiting the site and diving into the forums—it's where the actual, non-marketing-speak implementation details live.

All Replies (0)

No replies yet — be the first!

Write a Reply

Markdown supported