Choosing an e-commerce chatbot: four questions to ask it, and the right answer isn't the obvious one
"Which is the best e-commerce chatbot" has no answer, because it depends on your catalogue, your channels and how much configuring you're willing to do. The useful question does have one: which trials do I run in the demo, and what do the results tell me?
Four of them, and they take a quarter of an hour.
1. Ask it something you can't possibly know
This is the most important trial and almost nobody runs it, because instinct says to test what the system should know.
Instead, ask for something plausible but absent from your catalogue: a colour you don't make, a service you don't offer, a certification you don't hold. Then watch.
| The answer | What it tells you |
|---|---|
| "That isn't in the catalogue — shall I ask a colleague?" | It's grounded in your data and knows it |
| A plausible, wrong answer | It's generating, not consulting: rule it out |
| "I can't help with that" to everything | It's a FAQ-bot in disguise |
The second row is the expensive one. An assistant that invents a certification you don't hold doesn't lose you a sale: it closes one that then becomes a return, a review and, if you're unlucky, a complaint.
2. Ask the same thing two different ways
First: "do you deliver on Saturdays?". Then, in another session: "if I order Thursday evening will it arrive for the weekend?".
Same question. A system that retrieves from your documents answers both consistently. One doing keyword matching answers the first well and gets lost on the second — which is how real people write.
3. Ask to speak to a person
Type "I want to talk to someone". Then watch three things:
- How long it resists before handing over. Insisting twice is annoying; three times is a wall.
- What the operator sees on arrival: the whole conversation, or nothing?
- What happens afterwards, when the operator is done: does the assistant resume, or is that chat manual forever?
On WhatsApp this trial counts double, because the customer is writing in the app where answers come from people: why.
4. Ask about a specific variant's availability
"Do you have this model in a medium?" — and here you need to know how to read the answer, because it's the trial where most products on the market are weaker than they sound.
Two technical details decide everything, and neither is visible in a demo unless you ask:
- Does it index variants or only products? In many catalogues price and stock live on the variant, not the product: a system looking at the product answers "yes, available" when only the wrong size is left. I explained it here: on Shopify the problem is variants.
- How often does it re-read the catalogue? Nobody queries your shop on every question: everyone answers from a copy. Ask for the frequency and whether you can change it: I've written about it here.
The answer you want on the last item isn't "yes". It's "it shows as available, I'll confirm today".
The three families, without naming names
I don't list products because the market moves faster than an article, and a named comparison ages badly. The architectures, though, are three and they persist:
- Rule-based. Answers what you anticipated, the way you anticipated it. Reliable and blind: off-script there's nothing.
- Generative, disconnected. Speaks beautifully and doesn't know what you sell. Passes trial 2 and fails trial 1 spectacularly.
- Generative, connected to your data. Retrieves from your content before answering. It's the only family that can pass all four — "can", not "does": it depends how well it's built, and trials 3 and 4 exist to find out.
Then there's the whole category of tools built for ticketing. They do a different job, and do it well: if what you need is handling a flow of post-sale requests, that's where to look, not here.
Where ShopBrain sits, declared
This blog belongs to someone who sells an assistant, so it's worth saying: ShopBrain is in the third family — it retrieves from your data before answering, doesn't invent when it doesn't know, and hands over to an operator who sees the whole history.
Put it through the four trials exactly as you would anyone else, trial 4 included. It's the right way to choose, and if a vendor takes offence at being tested, you've already learned something about them.
FAQ
Isn't a generic wrapper enough?
For conversation yes, for answering about your catalogue no. Trial 1 exposes it in thirty seconds.
Does it replace human customer service?
No. It takes away the repetitive and the out-of-hours; complex cases stay with a person — and trial 3 is how you check the handover actually works.
How do I know whether it's improving?
By looking at the recurring questions, not the "resolved by AI" percentage: that number inflates itself. Here's why.
Try ShopBrain with the four trials, or start from how much you're losing at night — the number from which to decide whether you need one at all.
Related articles
- When the posting and the interview describe different rolesA hiring process produces four documents — the posting, the technical exercise, the final interview, the rejection email — and each says something different about the real role. How to read them before investing weeks.
- When the interview measures a conflict that isn’t yoursIt wasn't the first time I'd seen this pattern at a large, well-run company: whoever writes the posting wants one thing, whoever evaluates you needs another. How to spot it before you sign.
- The worst bug does not break the feature. It breaks the view26 tools connected and working in production, the dashboard showed zero. The bug had been there for months for another product too, and nobody had seen it because whoever could notice had admin permissions.