Building a Transformer from Scratch: My 6.
The goal was simple: create a model capable of reasoning about recipes and ingredients without relying on external vendors, rate limits, or per-token costs.
The Architecture
I designed RasavedaGPT as a tiny, decoder-only transformer. Since the domain is narrow, I didn't need a behemoth; I sized the model specifically for the task so it could run comfortably on a CPU within a FastAPI backend.
- Total parameters: 6,392,320
- Vocabulary size: 6,000 (custom BPE)
- Context length: 512 tokens
- Embedding dimension: 256
- Attention heads: 8
- Transformer layers: 6
- Feed-forward dim: 1,024
Training Workflow
I found that starting from random initialization and jumping straight into recipe data was too difficult for the model to grasp both fluency and domain knowledge. Instead, I used a two-stage AI workflow.
Stage 1: General Language Acquisition
I pre-trained on WikiText-2 for 3 epochs using a cosine learning rate schedule. This was purely to teach the model the basic structure of English. By the end of this stage, the perplexity dropped from 345.3 to 114.3.
Stage 2: Domain Specialization
I then shifted to fine-tuning on a specific dataset of 2,139 recipe examples. I ran this for 12 epochs, repeating the data 8x per epoch to ensure the model locked in the patterns. The loss dropped significantly, from 2.439 in the first epoch to 0.787 by the third.
Real-World Takeaways
Building this from scratch provided a level of insight that prompt engineering simply can't. When your loss curve refuses to budge, you're forced to dive deep into positional embeddings and attention mechanisms to figure out why.
For anyone looking for a practical tutorial on LLM internals, I highly recommend trying a small-scale deployment like this. It proves that for specific, narrow tasks, a well-trained 6M parameter model can be more efficient and predictable than a general-purpose giant.
# The project is available here
https://github.com/cherimedz/Rasaveda