Stop treating your prompt like a foolproof contract if you're
The problem is that while these smaller models are surprisingly capable at the actual task, they are absolute nightmares when it comes to following constraints. I spent way too long trying to "prompt engineer" my way out of hallucinations and formatting errors before realizing that the only real solution is a robust parser.
The failure of "absolute" rules
I hit a wall where the model kept insisting on inserting the Chinese particle "了" into sentences that were already grammatically correct. I tried every trick in the book to stop it:
1. I added a judgment rule telling it to only flag genuine errors.
2. I provided negative examples of what not to flag.
3. I moved the instruction to the end of the system prompt.
4. I used all-caps "ABSOLUTE" rules.
None of it worked consistently. The model would seem fixed for a few hours, then start hallucinating corrections again. The only thing that actually solved it was moving the logic out of the prompt and into a Go-based parsing boundary. Now, the code simply drops any "no-op" edits or pure "了-insertion" edits before they ever touch the database. The prompt still asks the model to be accurate, but the parser is the one actually enforcing the quality.
Why the system prompt isn't enough
I dealt with another issue where the model would occasionally drift from the user's UI language (e.g., Russian) into the target language (e.g., Finnish) for the explanations. When I tested this with 21 runs of the same input, about 29% failed.
I tried dropping the temperature to 0.0, but that did almost nothing. This proved to me that this isn't a sampling noise issue; it's an instruction-following failure. When the input text is heavily dominated by Finnish, the "answer in Russian" instruction in the system prompt just fades away.
Interestingly, repeating the rule as the very last line of the user turn—immediately after the learner's text—dropped the error rate to 0/21 in my sample. For small models, proximity to the generation point is everything.
Even so, I don't trust a 0/21 sample size. I implemented a "dumb" guard that checks the Unicode script of the explanation. If the UI is Cyrillic and the output is Latin, it triggers a single retry. If the retry fails, I keep the wrong-language answer anyway because a slightly broken correction is better than a blank screen.
The "Production-Ready" Prompt Logic
If you're building an AI workflow with a small model, stop relying on the system prompt for critical constraints. Use this structure instead:
### System Prompt
You are a linguistic expert. Your goal is to identify errors in the provided text.
Return a JSON array of objects with the following keys: "original", "corrected", "explanation".
[Insert specific category constraints here]
### User Turn
Text to analyze: "{{USER_TEXT}}"
REMINDER: You must write the "explanation" field exclusively in {{UI_LANGUAGE}}.
Do not invent errors. Return ONLY valid JSON.The takeaway for any real-world deployment is simple: the prompt is a suggestion, but the parser is the law. If you can validate a constraint with a regex or a script check in your backend, do it there and stop wasting tokens trying to convince the LLM to behave.