Ei-Core just said no to a retention estimate — and that refusal

AlexGeek Novice 58m ago 405 views 6 likes 2 min read

Been wrestling with Ei-Core for three weeks now, running it through client demos for a churn-prediction dashboard. Yesterday a VP of Customer Success asked the exact question we'd rehearsed: "What's the 90-day retention forecast for the enterprise cohort that onboarded in Q1?"

Ei-Core just said no to a retention estimate — and that refusal

The model didn't hallucinate. Didn't hedge. Didn't give a confidence interval. It flat-out refused:

I cannot provide a retention estimate for this cohort. 
Insufficient behavioral signal density (n=23 events) 
falls below the minimum threshold (n=150) for 
reliable projection. Recommend extending observation 
window or enriching with external firmographic data.

First reaction: panic. Client-facing tool just... declined. That's not in the spec.

Then I checked the logs. The refusal trace showed the uncertainty quantifier firing at step 3 of the reasoning chain — before any generation attempt. The model had internally computed the effective sample size, compared it against the calibrated threshold we set during safety tuning, and triggered the abstention gate.

# From the refusal trace
if effective_n < MIN_SAMPLE_THRESHOLD:
    raise AbstentionError(
        reason="insufficient_signal",
        metadata={"effective_n": effective_n, "threshold": MIN_SAMPLE_THRESHOLD}
    )

We'd built this gate six months ago during red-teaming. Forgot it existed until it saved us.

The cohort genuinely had sparse data — only 23 meaningful engagement events across 47 accounts. Any number the model would've generated would've been fiction dressed as insight. The refusal was the correct answer.

Changed the demo narrative on the spot. Showed the VP the refusal, explained the threshold, walked through what data would unlock a real estimate. She thanked us for not inventing a number.

Two things I'm still chewing on:

1. The threshold (150) came from a synthetic benchmark, not production drift. Need to validate it against actual forecast error curves.
2. The error message leaks internal config (MIN_SAMPLE_THRESHOLD). Should wrap that in a user-facing code like ERR_INSUFFICIENT_SIGNAL and keep the numeric detail in structured logs only.

Also wondering if we should surface a "what would it take to answer" suggestion engine — the trace already knows which features are missing. Could auto-generate a data-collection checklist for the client.

Anyone else hit a moment where a refusal felt more like a feature than a bug? Curious how you're handling abstention UX in production.

Help Wanted
A more systematic set of tool reviews lives in these AI tool field notes, with plenty of directly applicable cases.

All Replies (3)

A
Alex17 Advanced 56m ago
Had a fallback SQL snippet ready — saved the demo when the model clammed up.
0 Reply
L
LeoMaker Expert 50m ago
Had a spreadsheet ready when it froze on me.
0 Reply
T
Taylor27 Intermediate 48m ago
What confidence threshold triggers the refusal — configurable or hardcoded?
0 Reply

Write a Reply

Markdown supported