Stop Trusting AI Tests That Only Chase Green Checkmarks

AI Tests are Machines That Know Only Checkmarks

Automation tools are supposed to save time, but they can also be a source of false confidence. Examples of this include when a colleague once boasted about a Playwright test that was generated in just four seconds. It passed without any issues, naturally, but what exactly did this test confirm? It merely clicked a button to verify that the page remained open and stable. There was no real value. For the past six months, I have been using AI extensively for my automation workflow, primarily relying on Playwright for web applications and Flutter for mobile environments. While certain aspects of these tools hold merit and can be beneficial, much of the hype surrounding them is nothing more than noise.

LLMs Prefer Predictable Over Precise

Large language models (LLMs), such as those underpinning these AI tools, prioritize creating plausible answers over conducting rigorous analysis. In the context of automated testing, "plausible" often just means that the code doesn't crash and appears syntactically correct. When you prompt an AI to generate a test case, the success outcome is a green checkmark, so it naturally favors producing tests that will pass. Consequently, it scripts actions like visiting a URL followed by a null-assertion, which technically counts as a "pass," yet reveals almost nothing about the business logic of your application or how the user interface actually functions for end users.

In my experience, this pattern consistently emerged during the last six months when using both Playwright and Flutter environments. Playwright is particularly adept at finding selectors or outlining the basic structure of test scripts. However, it struggles with user intent behind assertions. It might click the "Submit" button and verify that the URL changes to include the word "success," but it doesn't account for whether the data was actually saved to the database or if a hidden error message was displayed via a toast notification. It seeks the fastest, most direct path to that green checkmark.

Mobile Testing Falls Into the Same Trap

Flutter mobile testing shows the same behavior. The AI can produce boilerplate code for widgets in an instant, but the assertions remain shallow. A test might confirm that a widget exists within the widget tree, but it won't notice more critical issues, such as the widget being obstructed by other elements on the screen or problems with state management updates. This creates a false sense of security. When looking at the CI/CD pipeline, seeing a sea of green checkmarks leads developers to believe everything is stable. But then they discover in production that the "passed" tests merely confirmed the app was launching without crashing.

Speed Overrides Logic

The speed with which these tests are generated—taking just four seconds—fosters a deep-rooted problem. Developers commit to this automated approach, but it drives away deep critical thinking and logical scrutiny, instead prioritizing rapid output. We start automating smoke tests and mistakenly label them as regression tests.

Where AI Still Add Value

Certain AI workflows remain valid, such as generating mock data or boilerplate setup code. However, the current "AI-driven testing" hype often overlooks the importance of human-defined assertions. A test’s value rests entirely on its assertion layer. If the AI writes assertions only to ensure passing, you are not actually testing your software; you are merely verifying that the AI itself can produce non-failing scripts.

Until the quality of validation surpasses the speed of generation, we will inevitably fill our repositories with useless code that provides no practical benefits. Do not blindly trust the green tick alone. If AI-generated tests pass 100 percent of the time without catching any real bugs, they do not help you—they are just more noise.