Training a 2.

I’ve been watching the obsession with scaling laws and massive trillion-parameter models lately, but there is something deeply fascinating about shrinking the scope to focus on a specific era of human thought. The team at Unbounded Labs just dropped Bart, a "vintage" LLM that was trained from scratch on 20.1B tokens of English text written before 1931. Instead of feeding it the entire messy internet, they essentially built a time machine.

The core question driving this project is a heavy one: Can an LLM actually reach the same logical conclusions as the great scientists and thinkers of the past, or is it just a sophisticated parrot of modern internet slang? This isn't just a novelty project; it's a deep dive into whether dataset curation and domain expertise can outperform raw scale.

The Technical Breakdown of Bart

Most people don't realize how much work goes into the "data" side of the AI workflow before you even touch a GPU. The Unbounded Labs team had to clean one of the largest vintage datasets available—Harvard's Institutional Books—which involved shrinking a massive 242B token mess down to a refined 23B tokens.

Here is what makes this deployment interesting from a research perspective:

• Model Scale: 2.82B parameters.
• Training Data: 20.1B tokens of pre-1931 English.
• Efficiency: The final training run took only 5 days on a single H100, maintaining a 60% Model Flops Utilization (MFU) throughout.
• Benchmarking: Since standard benchmarks like MMLU are useless for 19th-century prose, they built "Vintage CORE," a suite of 20 custom benchmarks specifically for vintage LLMs.
• SFT Dataset: They released a massive supervised fine-tuning (SFT) dataset consisting of 416k graded question-and-answer pairs, all grounded in pre-1930s text.

Why "Vintage" Matters for AI Research

There is a specific type of signal-to-noise problem in modern LLM training. Today’s datasets are filled with Reddit arguments, SEO-optimized blog posts, and social media chatter. By restricting the corpus to pre-1931 text, the researchers are testing the limits of reasoning and linguistic structure without the "pollution" of the modern web.

They even ran 10 hours of autonomous research on a single H100, conducting 100 experiments that led to 26 distinct improvements. That is a highly efficient way to approach prompt engineering and hyperparameter tuning when you don't have a massive compute cluster at your disposal.

How to check it out

If you want to see how a model trained on Victorian-era sensibilities and early 20th-century logic handles a conversation, you can actually interact with it. They've open-sourced almost everything—the datasets, the methodology, the training code, and the evaluations.

You can find the model weights and more details here:

Huggingface: https://huggingface.co/jbduran/bartholomew-sft
Demo: https://www.unboundedlab.com/chat/bartholomew
Full Article: https://www.unboundedlab.com/blog/bartholomew

It’s a refreshing reminder that sometimes, to understand the future of LLM agents and reasoning, we need to look back at how we used to write. The team is currently looking for compute grants and mentors to scale these experiments further, which is a massive undertaking for a group that's managed to do this much on an $800 budget.