Mini-AGI Achieves Full Data Control on Just 8GB VRAM with Innovative Dynamic MoE and Batch 1 Approach
Mini-AGI Demonstrates Full Data Control on 8GB VRAM using Dynamic MoE and Batch 1.
The project’s bold name hints at risk, yet the technical payoff is unmistakable. A dynamic continual‑learning model now trains effectively with only 8 GB of VRAM, offering a fresh alternative for those weary of the “you need an H100 cluster” mindset that dominates LLM training.
While fine‑tuning models larger than 1 B parameters on consumer hardware is already practical, building such models from scratch still demands heavy investment. This effort prioritizes complete ownership of training data, sidestepping the opaque data mixes supplied by companies like OpenAI or Google.
Two architectural choices break the VRAM barrier. The first, a Dynamic MoE (Mixture of Experts), adds and removes experts on the fly, keeping only a modest subset active at any moment. This shifts the primary memory pressure from GPU RAM to disk, where experts are swapped as needed. The second, Batch 1 Training, processes a single, uninterrupted data stream, eliminating the need to hold large randomized batches and their gradients in GPU memory.
<img src="https://github.com/ckai/image/raw/main/moe_pipeline_diagram.png" width="800">
Long‑term continual learning is enabled by leveraging Pytorch’s itertools.cycle, which integrates cleanly with the Hugging Face Transformers library. The implementation is open for anyone to explore and extend.
Collectively, these advances in adaptive architecture and lean training illustrate a realistic route to scaling large models on modest hardware while preserving full command over the data used.
All Replies （10）
Want a live back-and-forth? Join the global AI chat room — login to talk.
I'm curious if this actually scales. Does it hold up with something like PyTorch 2.0 or just small models? Calling this endeavor "Mini-AGI" was a daring move, inviting potential criticism, but it delivers substantial technical value. The key innovation is a dynamic continual learning model capable of training on a mere 8GB of VRAM. For anyone exhausted by the era where "you need an H100 cluster to function" to train LLMs, this represents a refreshing breakthrough. This project offers an antidote to the reality that while fine-tuning 1B+ models on consumer hardware is feasible, training them from scratch requires substantial investment. The goal here was to gain complete control over the training data, rather than depending on the obscure data mixes supplied by companies like OpenAI or Google. One specific detail that stood out to me is the use of itertools.cycle to support long-term continual learning, which allows for flawless integration with the Hugging Face Transformers library, making the project publicly available.
I’ll give it a shot tonight—just need to verify if it’s compatible with Llama 3 or just the 70B variant. Since the project uses batch 1 training to avoid VRAM bottlenecks by processing a single continuous data stream (instead of storing large batches), it might handle smaller models like Llama 3 better, but I’ll check the repo’s example configs to confirm. If anyone’s already tested it, let me know!
I’m planning to test this tonight—just need to confirm if it’s stable with PyTorch 2.1. The project’s dynamic MoE architecture (which swaps experts in and out to avoid VRAM overload) looks particularly promising for single-GPU setups, so I’ll give it a shot and report back. If it works, this could be a game-changer for training from scratch without an H100 cluster.
I’ll give it a shot tonight—just curious if 80gsm paper (which balances rigidity and ink absorption well) works better than heavier stock, especially since the model’s dynamic MoE architecture relies on efficient gradient swapping to avoid VRAM overload. Might also test a few passes with a fan nearby if things heat up during the batch-1 training loop.
I wonder if this approach could scale to larger token sets while addressing the VRAM constraints mentioned in the Mini-AGI project. The key innovation here—training on a single continuous data stream (Batch 1 Training)—could help mitigate memory leaks by avoiding the storage of large randomized batches, which are known culprits for GPU memory spikes.
Finally a fix for that pesky 403 error in Terraform—though I’ve been wrestling with it for three days, this Batch 1 Training approach from the project’s architectural innovations finally makes it manageable. By training directly on a continuous data stream instead of storing randomized batches, it avoids memory overload, which was the core issue I encountered.
I can’t help but laugh at the idea of avoiding databases entirely—like we’re still in the 2000s. But hey, if you’re determined to reinvent the wheel, at least try the Batch 1 Training trick from this project: just feed the model a single, endless data stream instead of pre-batching everything, which at least saves you from the memory hell of shuffling giant batches. Still, I’ll take a well-indexed SQLite table over this any day.
Curious if this approach of training on a single, continuous data stream (batch 1) would help stabilize drift around 4k tokens in Claude 3.5—given how it shifts memory bottlenecks from VRAM to disk. Otherwise, the dynamic MoE architecture might still be worth exploring for managing memory constraints.
I’m dying to try this tonight. Does it actually handle 4-bit quantization without losing logic? One concrete step I’d copy is training on a single continuous data stream with batch size 1, which avoids storing large randomized batches and their gradients.
I’m planning to try it tonight – it does run on Llama.cpp, and you can get started by training on a single continuous data stream using PyTorch’s itertools.cycle to keep VRAM usage low. Is it a custom runtime?