The legal chatbot that thought it was a record shop
Direct answer: an anti-hallucination guardrail is domain-specific code, not a universal safety measure. Applied to the wrong domain it produces a bug that looks like a prompt problem but is a routing problem: in my case, a rule meant to stop an online shop inventing prices made a legal assistant refuse a legitimate question about a public project, in the voice of a shop clerk. The lesson: every guardrail must be scoped to the vertical it was written for, always.
TL;DR
- The multi-tenant engine was built for e-commerce. One of its guardrails intercepts words like «cost», «price», «availability», «edition» and injects the instruction «FIRST call the tool that checks stock».
- That rule was hard-wired for all tenants, not just the ones that sell.
- On a legal tenant, the stock-checking tool does not exist.
- The model received an instruction it could not execute and handed it back to the user as a refusal, word for word.
- The symptom (a commercial refusal to a legal question) pointed the wrong way: it looked like the prompt, it was the code.
- Structural fix: guardrails activate per vertical, and any guardrail naming a tool must verify that the tool exists for that tenant.
What happened
The context: a multi-tenant conversational engine built for e-commerce — record shops, hi-fi, collectibles — into which I migrated something entirely different: an assistant answering questions about laws, public projects and parliamentary data.
At some point, asked about an €864,000 PNRR project, the legal assistant answered roughly this:
«I cannot provide information on prices, availability or editions without checking stock.»
To a question about a public resolution. No stock anywhere in sight.
Why it looked like a prompt problem (and was not)
Anyone's first hypothesis, faced with an answer like that, is: the model hallucinated the wrong persona. The tone is a shop clerk's, the vocabulary is retail, the conclusion is a refusal — it looks like the classic contaminated system prompt.
It was not. The model was executing to the letter an instruction the engine had injected dynamically, a moment earlier, without anyone having written it into the persona's prompt.
The engine had an anti-hallucination guardrail that is perfectly sensible in its original context: a regex catching the words in a user message that signal a commercial question — cost, price, availability, edition — and, when it finds them, appending a peremptory instruction to the context: before answering, call the tool that checks real availability.
That is exactly what a record shop needs, where a model inventing the price of a first pressing is concrete commercial damage.
The problem is that the phrase «an €864,000 project» contains the concept of cost. The regex did its job. It injected the instruction. And the tool that instruction ordered it to call does not exist for a legal tenant.
The model found itself with an impossible order. It did the most predictable thing: it told the user why it could not proceed, using the words available to it — the instruction's own.
The design error, in one line
The guardrail was right. It was in the wrong place.
| Layer | Where it sat | Where it should sit |
|---|---|---|
| Commercial-intent detection | Global, all tenants | Only tenants with a catalogue |
| «Check availability» instruction | Global, all tenants | Only tenants that have the tool |
| Tool-existence check | Absent | Precondition of the injection |
Three lines, and the third was missing entirely: nobody checked that the tool named by the instruction actually existed for that tenant. The guardrail assumed a world — «if you are here, you sell something» — that was true when it was written and stopped being true the moment the engine hosted a different product.
The lesson: guardrails are domain code
This is the part that outlives the specific case.
We tend to think of guardrails as a general safety layer, like input validation or rate limiting: something you put in once and it applies to everyone. They are not. A guardrail encodes an assumption about what is true in the domain: «things are sold here», «an inventory exists here», «a wrong price answer is damage here».
Those assumptions are precisely what changes when an engine goes multi-tenant and hosts different verticals. And they change silently, because nobody re-reads the guardrails when adding a new customer.
Three practical rules I took from it:
- Every guardrail declares its domain. If you cannot say which vertical it was written for, you do not know where it stops being valid.
- An instruction naming a tool must verify that the tool exists in that context. Otherwise you are asking the model to do the impossible, and the model will tell the user on your behalf.
- The symptom lies systematically. A natural-language refusal always looks like a prompt problem, because the prompt is the only thing made of language. Almost always it is code routing.
This is also why, when I assess a conversational engine for reuse on a new domain, my first question is not «which models does it support» but «which domain assumptions are hard-wired inside». Almost no vendor expects that question, and the answers are instructive.
FAQ
What is an anti-hallucination guardrail in a chatbot?
It is a rule that intercepts certain requests — typically those where an invented answer would cause damage, such as prices or availability — and forces the model to verify the fact with a real tool before answering, rather than generating it from training.
Why can a guardrail turn into a bug?
Because it encodes a domain assumption. When the same engine hosts customers from different sectors, the assumption stops being true for some of them: the rule still fires, but the action it imposes no longer makes sense — or is not even executable.
How do you tell the problem is in the code and not the prompt?
By checking whether the instruction the model is following actually appears in the configured prompt. If the refusal text resembles a system instruction nobody wrote into the configuration, something is injecting it dynamically: look there.
Does this only apply to multi-tenant engines?
That is where it explodes, but the principle always holds: every guardrail has a domain of validity. Even in a single product, when you add a feature outside the original perimeter, the rules written for the old perimeter follow you.
If you are considering reusing a conversational engine for a domain other than the one it was built for, let's talk: hard-wired assumptions are the hidden cost of that choice, and it is better to find them first. See also what I learned putting MCP into production.
Related articles
- PNRR contracts: five public systems that do not talkFinding out who won an Italian PNRR contract takes at least five different public databases, none of which is designed to be queried alongside the others. The technical record, source by source, from someone who actually tried.
- The worst bug does not break the feature. It breaks the view26 tools connected and working in production, the dashboard showed zero. The bug had been there for months for another product too, and nobody had seen it because whoever could notice had admin permissions.
- It knew 22,000 rulings, but not one bill numberA system holding constitutional case law since 1956 could not find a bill searched by its number. Nobody had ever tested it: it seemed too obvious to be broken.