AI customer service: the four metrics, and the two that mean nothing
At the end of the first month a number arrives: "the AI resolved 85% of requests". It's the number in every deck, and it's also the one that inflates itself. It's worth understanding how, because the two metrics that actually matter are different ones.
TL;DR
- "Resolved by AI" usually means "conversations with no escalation" — and that bucket contains the customers who gave up.
- First response time is always excellent for an assistant: a metric that cannot get worse isn't measuring anything.
- A low escalation rate is not good news by itself: it depends on what happens in the conversations that don't escalate.
- The only one useful without interpretation is the list of recurring topics, because it isn't a score: it's a list of things to fix.
1. "Resolved by AI" — how it inflates
Nearly everyone computes it the same way: total conversations minus the ones handed to a human. Which means three very different things land in the "resolved" column:
- the customer got their answer and went on to buy;
- the customer got a wrong answer and never came back;
- the customer stopped writing halfway through.
In a chart, the last two look a lot like success.
The question to ask: how do you tell a resolved conversation from an abandoned one? If the answer is that you can't, then that number is a count of the conversations that didn't bother anyone — legitimate information, but not what it appears to be.
The check you can run yourself: take twenty "resolved" conversations at random and read them. It takes half an hour and tells you more than the dashboard.
2. First response time
An assistant answers in under a second, always, at any hour. The metric is green by construction, and a number that can't get worse isn't telling you how things are going.
Two different times actually matter, and neither is the first response:
- How long it takes to reach the answer, measured in messages exchanged. An assistant that replies in three hundred milliseconds but needs eight messages to understand the question is worse than an email answered in two hours.
- How long it takes a human to arrive when one is needed. That's where the experience really breaks: not when the bot doesn't know, but when it doesn't know and doesn't hand over.
3. The escalation rate, read backwards
Instinct says: the lower the better. But a low rate has two opposite explanations, and the dashboard doesn't tell them apart:
| Escalations are low because… | What it means |
|---|---|
| the assistant genuinely resolves | it works |
| nobody knows they can ask for a human | difficult customers leave quietly |
An honest assistant offers the handover when it realises it's out of its depth, instead of waiting for the customer to ask. If your dashboard says escalations are 2%, look at how many were started by the bot and how many by an exasperated customer. If nearly all come from the customer, 2% is a problem, not a result.
4. Recurring topics — the one I read first
The most frequent questions of the first fortnight are the most useful thing this tool produces, and it isn't a performance metric: it's your website's shopping list.
If thirty people asked about delivery times, that information isn't where they look for it. If twenty asked about fit, the product pages are incomplete — and you're also losing the orders of everyone who asked nothing and left.
In the first month, the value of an assistant isn't the load it takes off you: it's that it tells you where your shop fails to explain itself.
The comparison almost nobody makes
Before switching anything on, take half an hour and write down three numbers: how many support requests you get in a week, how long you take on average to answer, and what the three most frequent questions are.
Without those three numbers, after a month all you can say is that the dashboard is green. With them, you can say whether anything changed.
FAQ
What's a good resolution rate?
There's no number to chase until you know how it's calculated. A 60% measured by checking whether the customer bought is worth more than a 90% measured by counting conversations nobody was handed.
Can the AI get products wrong?
Yes, if it isn't connected to the catalogue. Connected, it answers from your data — as of the last sync, not in real time: I've written about it here.
What if the customer wants a person?
They have to be able to get one immediately, and the assistant should offer it when it realises it's out of its depth. On WhatsApp that counts double: why.
How long before I see something?
Two weeks for the recurring topics, which are the part that's useful straight away. The resolution numbers take longer, and require deciding up front how they're counted.
ShopBrain answers from your real store data, hands over to an operator who sees the full history, and collects the recurring questions. If you're still choosing, continue with how not to get fooled by fake AI chatbots.
Related articles
- When the posting and the interview describe different rolesA hiring process produces four documents — the posting, the technical exercise, the final interview, the rejection email — and each says something different about the real role. How to read them before investing weeks.
- When the interview measures a conflict that isn’t yoursIt wasn't the first time I'd seen this pattern at a large, well-run company: whoever writes the posting wants one thing, whoever evaluates you needs another. How to spot it before you sign.
- The worst bug does not break the feature. It breaks the view26 tools connected and working in production, the dashboard showed zero. The bug had been there for months for another product too, and nobody had seen it because whoever could notice had admin permissions.