Using LLMs for Customer Sentiment: Why Surveys Lie

NPS and CSAT scores are often a performance. For years, we saw 8s and 9s and polite feedback, yet our churn rate remained stubbornly high. The numbers said everyone was happy, but the bank account said otherwise.

To find the disconnect, I dumped a massive batch of raw sales and support call transcripts into ChatGPT. I didn't ask for a summary or a sentiment score; I asked it to tell me what customers actually think of us, specifically requesting that it strip away all the diplomacy.

The results were brutal. While surveys called us "responsive," the actual call transcripts revealed customers complaining that we take days to reply. While surveys suggested the product was "great," the raw audio transcripts showed people feeling we oversell features and then walk them back later. The most damning part was the repetition—customers explicitly stating they had to explain the same problem to three different people before anything happened.

This was a massive wake-up call in AI workflow implementation. A survey is a 10-second interaction where people are polite because they have a meeting to get to. A support call is where the unfiltered truth lives. By using an LLM agent to analyze raw text instead of sanitized checkboxes, we finally got a real-world view of our operational failures.

I've shifted this into a monthly deep dive. It's impossible to ignore a systemic issue once you've read a customer describe it in their own words.

For those wondering about the pipeline, the transcription tool is secondary to the prompt engineering. I used BuildBetter for the transcripts, though tools like Fathom or Otter do the same thing. The real value comes from how you prompt the LLM to ignore the "polite" layer of corporate communication and extract the actual pain points.