Agnost AI: Product Analytics for AI Agents

Traditional web analytics are useless when your primary interface is a chat box. In a standard SaaS app, you track clicks, funnels, and button presses to understand user intent. But when you deploy a voice or chat agent, those metrics fall apart. A user might stay on the page for ten minutes (good retention metric?) but spend that entire time "rageprompting"—cursing at the agent or repeatedly rephrasing the same request because the LLM isn't grasping their intent.

I've been following the rollout of Agnost AI (a YC S26 company) because they tackle the "black box" problem of agent UX. While observability tools tell you about latency and errors, and eval frameworks test known edge cases, they don't actually tell you what the users want.

They've built what is essentially a PostHog for conversations. Instead of monitoring technical traces, they monitor behavioral failures.

How the Discovery Engine Works

The technical challenge with analyzing millions of conversational messages is the cost and latency of LLMs. You can't just pipe every single chat log into GPT-4 to ask "is the user happy?"

The team implemented a multi-layered clustering approach to handle scale:

Segmentation: They split conversations based on cosine drift to isolate specific segments.

Compression: They use BIRCH to compress the candidate space, making it manageable.

Clustering: They apply HDBSCAN-like clustering on the smaller, compressed set.

Classification: For matching existing intents, they use embeddings and BERT-style models, only falling back to LLMs for the truly ambiguous stuff.

Real-World Feature Discovery

The most interesting part is the "hidden intent" detection. They had a case with an AI video editor company where users were asking for "auto-subtitles" but using highly varied language like "add this text in this frame" or "give me a transcript."

A standard keyword search would have missed the pattern, but Agnost's clustering identified the intent across different languages and phrasing. This turned "unstructured chat noise" into a concrete product roadmap item.

Integration and Implementation

If you are building an agentic workflow, you don't have to rebuild your entire data stack. They provide SDKs via PyPI and npm, and they support OpenTelemetry (OTel) for ingestion.

Data Privacy: They use customer data exclusively for that specific customer (SOC 2 Type 1 compliant).

Scaling: They are currently ingesting roughly 1M messages per day.

Deployment:

# Available via standard package managers
pip install agnost-ai
npm install @agnost/sdk

For teams moving from "testing prompts" to "shipping agents in production," this is the layer that actually bridges the gap between technical stability and product-market fit.