AI alignment should be integrated into pretraining rather than treated as a surface layer

Most developers stick to a predictable routine: pretrain a model on the chaotic internet, then spend months attempting to fix its personality via SFT and RLHF to prevent it from suggesting users eat glue. This approach fails because by the time alignment begins, the model has already internalized a trillion ingrained priors. It is like trying to teach manners to a teenager who has spent fifteen years in the wildest corners of Reddit.

Synthetic Persona Pretraining (SPP) suggests baking good behavior into the model from token zero. Instead of the standard pretrain then align pipeline, SPP integrates value-aligned reflections directly into the pretraining data.

The SPP Workflow

The process functions essentially as a three-act play:

1. Value Annotation: Standard pretraining documents are paired with first-person reflections based on a normative value constitution. This functions like a diary that constantly reminds the model how to be a helpful, aligned entity while it learns basic language.
2. The Blend: The model undergoes pretraining using standard cross-entropy loss on both raw data and these synthetic reflections. The objective is not to create a saint, but to install a specific persona alongside the other noise being absorbed.
3. Persona Binding: In this final step, dialogue data is used to tell the model that the polite persona learned during pretraining is its actual identity.

Does it actually stop jailbreaks?

The results for models up to 3B parameters are quite interesting. By shifting alignment to the pretraining phase, models showed improved constitution following and better jailbreak robustness.

When facing out-of-distribution moral dilemmas—the edge cases that typically cause an LLM agent to leak its system prompt or have a meltdown—SPP models stayed on track more frequently. If values are rooted in the model weights rather than a thin layer of instruction-tuning, they are much harder to shake.

The most damning finding for the traditional approach is that introducing SPP only at the end of pretraining is far less effective. Early intervention is the key factor. The more compute dedicated to the pretraining phase, the stronger the alignment becomes.

We have been trying to patch leaks in a sinking ship when we should have built a hull that does not leak. This is a much more elegant AI workflow than praying that RLHF does not accidentally lobotomize the model's reasoning capabilities.