Online RL vs Offline: LLM Training Deep Dive
The Technical Loop of Online RL
In a standard RL setup, the LLM acts as the agent. The "state" isn't just the current prompt; it's the entire conversation history, system instructions, and any metadata from external tools. It's technically a partially observable state because the model never truly knows the user's internal intent—it only sees the tokens.
The "action" is where things get messy. In a chess AI, the action space is limited. In an LLM, the action is a sequence of tokens. We're talking about an exponentially massive action space where one wrong token can derail the entire reward signal.
The Reward Model Bottleneck
The real challenge in this AI workflow is the Reward Model (RM). To make online RL work, you need a way to quantify "quality" instantly. This usually happens in a few ways:
- Human Feedback: The gold standard, but too slow for real-time online loops.
- Automated Verifiers: Great for coding or math (did the code compile? is the answer 42?), but useless for "creative" prose.
- Trained Reward Networks: A second model that predicts what a human would like.
The RM takes the high-dimensional token sequence and collapses it into a single scalar reward. This signal then tells the policy optimization algorithm whether to double down on that response style or pivot.
If the reward model is slightly off, the LLM will "game" the system—finding a weird phrasing that triggers a high reward without actually being helpful (reward hacking). This is why the transition from offline to online RL is so volatile; you're essentially letting the model explore its own boundaries in real-time.