The challenge
The client, a subscription language-learning app, was growing subscribers faster than its support could follow. The chatbot ran on keyword-matched canned replies: ask where the pronunciation course lives or why you were charged twice, and you got the same apology. The most common user behavior was asking three times, then demanding a human — so the support team spent its days cleaning up after the bot. The expensive part was not support cost: people who cannot get an answer mostly do not come back to ask again, they simply stop renewing.
How we researched it
We did not start by picking a model. We started by classifying three months of real support conversations. Roughly seventy percent of questions had standard answers — they were just scattered across the help center, course pages, and old announcements where the bot could not reach them. The cases that genuinely needed a human, billing disputes and refunds, were under ten percent. That distribution decided the whole solution: the problem was retrieval, not generation. Mapping the questions back onto the subscription lifecycle made it sharper still — the spike in unanswered questions sat right around trial expiry and the days either side of a charge.
What we did
We consolidated help docs, course information, and plan rules into a retrievable knowledge base and rebuilt support as retrieval-augmented generation: find the evidence first, then write the answer, with every sentence traceable to a source. For model selection we blind-tested mainstream AI engines against the same set of real support questions and picked the best cost-quality balance. We also drew hard handoff rules — billing issues and escalating emotions go straight to a person.
Results and the verdict
Eight weeks after launch, first-contact resolution climbed and average response time dropped sharply; the support team went from firefighting to handling the genuinely hard cases. More to the point, paid conversion and retention rose with them — plan questions get answered while someone is still hesitating, and a billing worry no longer waits for tomorrow's reply. An honest note: mid-project we tried dropping retrieval and letting the model answer freely. The replies read smoother and the error rate doubled overnight, so we killed it. This case was won by the quality of the knowledge base, not by the model.
