How I Fixed My GPT‑2 Reproducibility Nightmare (Part 2)

A few weeks ago I ran the standard GPT‑2 training script and got stuck with weights that were consistently worse than OpenAI’s published checkpoints. After a hundred hours of debugging, the culprit turned out to be a one‑line error in the weight initialisation routine. Here’s the exact cause, how I isolated it, and the fix that finally put my metrics in line with their reported numbers.

The symptom – loss curves looked healthy for the first few thousand steps, then plateaued 0.2 nats above the OpenAI baseline while the validation perplexity refused to cross the 35‑point mark. I blamed the dataset mix, batch size, even the tokeniser caching. Each check came up clean.

The culprit – in model.py the normal_ call for embedding layers had a missing std argument, falling back to a default of 1.0 instead of the 0.02 used in the original papers. That blew out the variance in every embedding table, forcing the transformer to spend precious capacity correcting the initial noise rather than learning meaningful representations.

# Broken
nn.init.normal_(self.wte.weight)

# Fixed – note explicit std
nn.init.normal_(self.wte.weight, mean=0.0, std=0.02)

I also found the same regression in the linear projection layers after the attention block. Without a consistent std=0.02 the residual stream accumulated variance at every layer, making the final hidden states twice as scattered as they should have been.

Diagnosis path – I dumped the norms of each weight matrix at step 0 and compared them against a reference run from the original gpt-2 repo. The embedding layer was already 8× larger in Frobenius norm. From there a git bisect on my config changes pointed straight to an overlooked line in the init function.

Re‑run results – after the fix the model hit the reported perplexity at 40k steps, and the validation loss matched the curve published in the GPT‑2 blog post. The training time didn’t change noticeably, so the fix is purely about fidelity, not efficiency.

For anyone trying to reproduce old transformers from scratch: always verify weight initialisation against the original codebase, not just the paper. The papers often omit the exact std values, and the reference implementations can get damaged by refactoring. A two‑minute sanity check on initial norms can save you two weeks of head‑scratching.

If you want the full diff of all the other small corrections I found along the way, drop a reply. I’ll put together a clean patch set if there’s interest.