Catalyst: Building an AI Scientist for Theory Discovery

The gap between data analysis and actual scientific discovery usually requires a human to spot a pattern and formulate a hypothesis. Catalyst attempts to automate this loop by treating theory discovery as a search problem, using LLMs to propose hypotheses and then validating them against data in a semi-autonomous cycle. Instead of just asking an AI to "explain this data," Catalyst creates a structured environment where the agent can iterate on a theory, fail, and refine its approach based on evidence.

For those of us into LLM agents, the real value here isn't just the "scientist" label—it's the architecture. It's designed to handle the "discovery loop": observing a phenomenon, proposing a formal theory, testing that theory via code, and updating the theory based on the results.

Getting Started with Catalyst

To get this running locally, you'll need a Python environment (3.10+ recommended) and an API key for your chosen LLM (Claude 3.5 Sonnet is typically the best performer for this kind of reasoning).

1. Clone the repository and install dependencies:

git clone https://github.com/imbue-ai/catalyst.git
cd catalyst
pip install -r requirements.txt

2. Configure your environment variables. Create a .env file in the root directory:

ANTHROPIC_API_KEY=your_api_key_here
OPENAI_API_KEY=your_api_key_here

3. Run a basic discovery task. You can point the agent at a dataset and a specific question you're trying to solve:

python main.py --dataset path/to/your/data.csv --query "Find the relationship between X and Y"

Technical Deep Dive: The Discovery Loop

The core of the system is a state-machine-like loop. Most "AI scientists" just prompt a model once; Catalyst maintains a "Theory State" that evolves.

• Hypothesis Generation: The agent analyzes the data distribution and proposes a potential rule or mathematical relationship.
• Verification Step: It writes a Python script to test if the proposed theory holds across the entire dataset.
• Refinement: If the test fails (e.g., the code throws an error or the result contradicts the theory), the error trace is fed back into the prompt as a "counter-example," forcing the model to pivot.

Real-World Performance and Constraints

From a practical standpoint, this is a powerful tool for prompt engineering researchers or data scientists who are tired of manually iterating through hypotheses. However, it has specific limitations:

• Compute Cost: Because it iterates (Hypothesis -> Test -> Refine), a single discovery task can trigger 10-20 LLM calls. If you're using GPT-4o or Claude 3.5, the token spend adds up quickly.
• Dependency on Tooling: The agent is only as good as the libraries it can use. If your theory requires a niche physics library that isn't installed in the environment, the agent will hallucinate a solution or get stuck in a loop.

If you want to customize the agent's behavior, you can modify the system prompts in the prompts/ directory. I found that adding a constraint to "prioritize Occam's Razor" in the theory generation prompt significantly reduced the number of overly complex, overfitted hypotheses the agent was producing.

For a full walkthrough of the implementation, you can check the specific documentation here:

https://github.com/imbue-ai/catalyst

This is a solid foundation for anyone trying to move from simple RAG workflows to a full-scale AI workflow for research. It shifts the AI from a "chatbot that knows things" to an "agent that finds things."