Claude Code: Is a Shorter System Prompt Better for Small LLMs?
In my experience with prompt engineering, there is a common misconception that small models require exhaustive, step-by-step instructions to avoid hallucinating. While that's true for very basic models, many mid-sized LLMs actually get "lost" in long prompts. When you overload a small context window with repetitive constraints, the model often suffers from "lost in the middle" syndrome, ignoring the core objective in favor of following a minor formatting rule.
If you're trying to optimize a local deployment or a lightweight agent, try this streamlined approach to prune your prompts:
1. Strip the Politeness: Remove "Please," "I would like you to," and "It would be helpful if."
2. Convert Prose to Constraints: Instead of "Try to make sure the output is concise," use "Constraint: Max 2 sentences."
3. Remove Redundancy: If you tell a model it's a "Senior Python Developer," you don't need to tell it to "write professional, clean code"—that's implied by the persona.
4. Use Delimiters: Use clear markers like ### Instructions or --- to separate the system role from the task.
The goal is to maximize the signal-to-noise ratio. For smaller models, a tight, high-density system prompt often results in faster inference and more precise adherence to the primary task because the model isn't wasting attention heads on filler text.
For anyone building an LLM agent from scratch, I'd suggest starting with the leanest possible prompt and only adding constraints when the model consistently fails a specific test case. It's much easier to add a rule than to figure out which part of a 2,000-word prompt is confusing the model.