Evaluating AI scientists by final output alone risks overlooking flawed reasoning and unrepeatable luck.

Judging an AI’s scientific ability based solely on its published results or code resembles grading a student’s exam by checking only the final answers—without reviewing how they arrived there. A correct conclusion does not prove the model followed logical steps; it could stem from random chance or an unexamined shortcut. Current benchmarks create a blind spot: high scores on output-only tests may not reflect true scientific rigor or a reliable method. By focusing exclusively on the end result, these metrics ignore the process—the very foundation where science is built.

To address this gap, the OpenDiscoveryTrace project has released 558 complete records of AI agents performing research in drug discovery, genomics, and materials science. Each trajectory captures every decision, tool interaction, and mistake made during the experiment. Rather than treating the final answer as the sole measure of success, this dataset highlights the distinction between a correct result and a correct method. In critical fields like drug development or genomics, a "correct" output derived from flawed logic or coincidental patterns holds no real-world value—it cannot be trusted for practical use or scaled effectively.

Because the records are exhaustive, researchers can now dissect how an AI navigates the scientific method. Instead of a static endpoint, each entry traces an iterative cycle: the starting hypothesis, the specific tools used to gather data, the analysis of findings, and adjustments made after errors. By preserving every thought and misstep, OpenDiscoveryTrace reveals whether an AI truly integrates knowledge or merely repurposes memorized patterns that coincidentally match expected outcomes.

Take materials science as an example. Discovering new materials demands understanding complex chemical relationships and physical constraints. An output-only benchmark might label a model "correct" if it suggests a material with the desired properties—but this says nothing about whether the model grasped the underlying science or simply guessed a known compound. A full trajectory, however, exposes whether the AI actively tested its hypothesis through tool calls or reached conclusions without evidence.

The inclusion of errors may be the dataset’s most important contribution. Traditional benchmarks treat mistakes as simple failures, but OpenDiscoveryTrace reframes them as critical data points. By tracking how an AI recovers from errors—or fails to—researchers can distinguish between genuine problem-solving and brittle, luck-dependent performance. Many "successful" results in current AI science benchmarks likely arise from random alignment with training data rather than reliable reasoning.

The broader implication is clear: treating AI scientists as black-box paper generators distorts their true capabilities. Assessing an AI’s potential in genomics, drug discovery, or materials science requires visibility into its reasoning steps. Without the 558 trajectories provided by this project, determining whether an AI acts as a scientist or a sophisticated pattern matcher remains speculative. Output-only benchmarks obscure the fragility of the underlying logic, offering a misleading measure of progress. The solution lies in examining not just what an AI produces, but how it gets there.