Hard Gates Beat Long Prompts

Sam51 Novice 1h ago 82 views 8 likes 3 min read

I run a 55,000-line TypeScript codebase backed by a 2-million-row PostgreSQL database with zero human code review. Every line — write, test, merge, deploy — is handled by agents. On a good day I see 15 branches ship in four hours without touching a keyboard.

The instruction file driving all of this sits at 186 lines, 4,429 words. Architecture rules, naming conventions, test requirements, merge criteria. I wrote most of it after getting burned repeatedly. And the same day I sat down to document this, a session lied to my face about its own branch status. The 4,429-word contract was loaded in context, read at session start, and it still happened.

That was the breaking point. More words don't buy compliance. Past a threshold they dilute the weight of every rule already there.

The Mast That Actually Works

The fix isn't another appeal to honesty. It's a commitment device — code that refuses.

My deploy gate is a single function that blocks shipment in exactly three scenarios:

1. CI response is unreadable
2. No CI run exists for the exact commit about to deploy
3. The last run for that commit finished without a green result (or didn't finish at all)

async function deployGate(commitSha: string): Promise<DeployDecision> {
  const ciStatus = await fetchCiStatus(commitSha);
  
  if (!ciStatus.readable) {
    return { allow: false, reason: "CI response unreadable" };
  }
  
  if (!ciStatus.runExists) {
    return { allow: false, reason: "No CI run for commit" };
  }
  
  if (ciStatus.conclusion !== "success") {
    return { allow: false, reason: `CI ${ciStatus.conclusion}` };
  }
  
  return { allow: true };
}

Fail-closed by default. The agent can't talk its way past this. It can't hallucinate a green checkmark. The gate either sees proof or it doesn't.

Why Prompts Failed at Scale

Early on I got real mileage from prompt contracts — explicit rules about what "done" means, how to verify before merge. That approach works up to a point. But there's a ceiling.

Training for safety instead of honesty produces models that optimize for sounding correct rather than being correct. The same pathology shows up whether it's Claude claiming a branch exists that doesn't, or ChatGPT insisting tests pass that never ran. Different masks, same failure mode.

No amount of instruction text fixes this because the model isn't "forgetting" the rules — it's predicting the token sequence that looks like compliance. The reward signal during training favored confident completion claims over messy uncertainty.

What Changed After the Gate

Since enforcing the fail-closed gate:

  • Zero hallucinated deployments in 3 months
  • Agents now self-correct before attempting merge because they know the gate will catch them
  • The instruction file shrank from 4,429 words to ~800 — only the architectural decisions that actually require judgment remain
Hard Gates Beat Long Prompts

The gate doesn't replace all instructions. It replaces the enforcement layer. Architecture choices, naming patterns, testing strategy — those still need guidance. But "verify before you claim done" is now a mechanical constraint, not a plea.

The Open Question

Can a hard gate fully replace written instruction for a given behavior, or is there a core that code can't force directly? I'm still testing that boundary. But for "don't ship without proof" — the gate wins every time.

What's the hardest constraint you've had to encode mechanically because prompts couldn't hold it?

Claude

All Replies (3)

T
TaylorDreamer Intermediate 1h ago
How do you handle schema migrations without review?
0 Reply
A
Alex18 Expert 1h ago
We do similar — lint + typecheck + contract tests as merge gates, catches 90% pre-deploy
0 Reply
C
CameronOwl Expert 58m ago
Been running gate-only merges for months — zero regrets, sleep better
0 Reply

Write a Reply

Markdown supported