A customer asks whether they can cancel a contract without a fee. The AI says yes, confidently. It sounds plausible. It's wrong. In a low-stakes creative application, that's an annoyance. In customer service it's incorrect policy information handed out as fact, a promise the company can't actually deliver, a complaint, sometimes exposure that takes a human agent hours to unwind. No generative system can be talked into never hallucinating, so that's not really the goal here. The goal is a system where unsupported answers are rarer, easier to catch, and cheaper to fix when one gets through.

An AI hallucination is when a generative system states something false, unsupported or inconsistent as though it were reliable. That's the term most people in the industry reach for. NIST's Generative AI Profile prefers “confabulation” and defines it as the production of confidently stated but erroneous or false content. It can be plainly wrong, it can be invented outright, it can be an assertion the system has no basis for, or it can simply contradict something the model already said. NIST treats confabulation as one of twelve risk categories that are either unique to generative AI or made significantly worse by it.

The label is less important than what's actually going wrong. Is the model wrong? Has it made something up entirely? Has it asserted something it can't back up? Has it contradicted itself? For a CX team, these are the questions worth asking.

The base model has no built-in access to live company policy: A large language model generates statistically likely text from patterns it learned during training. It has no inherent connection to a live, authoritative source of your current policy, because there isn't one built in. That gap is what retrieval, databases, APIs and business rules exist to close, which is why the rest of this article is really about those layers rather than the model on its own.

Information can be stale in two different places: A model's own training can lag behind a price change, a policy update, a new product. But there's a separate, often bigger risk: the external knowledge fed to the model at the moment of response can itself be out of date, even if the model handling it is perfectly capable. Both look identical from the customer's side of the conversation.

Retrieval fails: In a retrieval-augmented generation (RAG) setup, the model is only as good as what it's handed. Feed it something irrelevant, outdated, contradictory or incomplete, and it can produce a confident wrong answer just as easily as if there'd been no retrieval at all. RAG lowers the odds of hallucination. It doesn't remove them.

Customer questions are ambiguous: People rarely ask precisely what they mean. Faced with a gap, a model can fill it with an inference rather than admit it doesn't know, though whether it actually does this depends on prompting, training, retrieval quality and whatever confidence checks are built in. It isn't some fixed trait of language models generally.

The system has too much autonomy: A model that only produces text is one kind of risk. An agent that can act on a hallucinated premise, processing a refund, updating an account, cancelling a service, is a different order of problem entirely. OWASP's 2026 Top 10 for LLM Applications, published on 4 August 2026, moved Excessive Agency from sixth place to third, the biggest jump on the list. It's not hard to see why: a wrong output stops being a wrong sentence on a screen and becomes a wrong action once something is empowered to act on it.

Customer service AI has stopped being just a content-generation exercise. It's shaping decisions and actions on the company's behalf, which turns accuracy into an operational requirement rather than a metric you check on a dashboard once a quarter.

Without some external grounding mechanism, a model is generating an answer from patterns baked into its parameters rather than consulting anything current. Retrieval-augmented generation changes that: the system retrieves information intended to be relevant and current, and hands it to the model before generation happens. Google Cloud describes grounding as giving models access to specific, verifiable data sources at the point of response, which reduces hallucination and improves the odds the answer holds up.

RAG is a control, not a guarantee, and that qualification matters as much as the technology itself. Retrieval can still fail in all the ways already covered above, and when it does, you get the same wrong answer, just now with a citation attached to make it look trustworthy. For a deeper look at how the retrieval layer actually works, see Retrieval-Augmented Generation for Customer Experience Explained.

1. Ground responses in authoritative sources: Point the system at approved, current company knowledge instead of letting it fall back on general model training.

2. Keep the knowledge base current, and assign clear ownership: A knowledge base doesn't stay accurate on its own. It stays accurate because someone is responsible for it. See How to Build an AI Knowledge Base for Customer Support.

3. Let the system say “I don't know”: A model that can admit it lacks enough information is a genuinely useful control, and one companies underuse given how cheap it is to build.

4. Check grounding, not just fluency: Where possible, check whether a generated claim is actually supported by what was retrieved. Google Cloud's grounding check tool scores claims against source documents and can filter out anything under a support threshold. That's a measure of support against the sources supplied, not a verdict on absolute truth.

5. Restrict what the AI is allowed to do: Apply least-privilege access. Don't hand an agent permissions or tool access it doesn't need for the task in front of it. This is the excessive-agency risk OWASP flags, and it limits the blast radius of every other failure mode above.

6. Use human escalation for higher-risk situations: Complex complaints, consequential financial decisions, legal or regulatory questions, vulnerable customers, policy exceptions: these are candidates for escalation wherever an organisation's risk framework calls for it. It isn't a rule that every one of these must always involve a human, but it's a category worth designing for deliberately. See Human in the Loop: When Should Humans Oversee Customer-Facing AI?

7. Test real interactions continuously, not just before launch: Passing pre-launch testing tells you nothing about whether the system stays safe once it's live. Combine sample-based evaluation, adversarial testing and ongoing production monitoring, and check specifically whether answers are supported by the system's sources rather than just whether customers rated the chat highly. It's the same underlying discipline that's pushing enterprise AI more broadly to be judged on outcomes rather than deployment, as covered in Enterprise AI Is Entering Its Measurement Era.

No single technology prevents hallucinations. Reliability comes from several controls working together: authoritative data, knowledge retrieval, generation, validation against sources, guardrails on what the system can do, human escalation for higher-risk cases, ongoing monitoring. They don't have to run as one strict linear pipeline, and no single layer can carry the whole thing on its own. There isn't one universally correct architecture here. What there is, is a need for several safeguards working at once, including the kind of AI guardrails that constrain what the system can actually do.

You can't manage a risk you're not measuring, and “accuracy” on its own is too blunt an instrument to act on. Worth pulling apart:

●        Factual accuracy: is the answer actually correct?

●        Groundedness: is the answer supported by what the system was given, regardless of whether that source was itself correct?

●        Policy compliance: does the answer conform to current company rules?

●        Unsupported-claim rate: how often does the model introduce claims its sources don't back up?

●        Escalation rate: how often does the system hand a case to a human?

●        Correction rate: how often does a human or customer catch and correct an AI answer?

●        Repeat contact: how often does an AI interaction fail to actually resolve the issue?

This is still a long way from a full evaluation framework. Without these signals tracked separately, a hallucination problem stays invisible until a customer or a regulator finds it for you.

The goal probably isn't a model that never produces an unsupported answer. It's a system that knows what it can trust, recognises when the evidence runs out, has limited authority to act alone, and hands the hard cases to a person. As customer service AI moves from chatbots toward autonomous agents, the line between suggesting and acting starts to matter a great deal (see AI Agents vs AI Copilots: What's the Difference?). A wrong output no longer just misinforms. It acts.

Hallucinations aren't going away because the models get better. For customer service, the answer was never to expect infallibility from the model. It's to build reliable knowledge, effective retrieval, the right context, validation, guardrails, limited permissions and human oversight around it. The most trustworthy customer service AI will be the one that understands and admits the limits of its knowledge.