Glossary
AI hallucination
An AI hallucination is an answer from an AI tool that states something false as though it were true. NIST, the US standards agency, calls it confabulation.
What it is, and why it happens
NIST's profile of the risks of generative AI names it confabulation: content that is stated with confidence but is wrong or false. In everyday speech it gets called hallucination or fabrication, and NIST notes that it can mislead or deceive the people who read it (NIST).
NIST explains it as a natural result of how these models are built. They write by predicting the next word from patterns in the text they were trained on. That can produce a correct answer. It can also produce a wrong one that reads just as smoothly. NIST says this happens especially with open-ended questions and with subjects that need specialist knowledge (NIST).
Why it matters when the bot speaks for you
NIST points out that the risk comes when people believe a false answer and act on it, often because it sounds sure of itself, and that a model can make up the reasoning or the citations that seem to back it up (NIST).
A chatbot on your website talks in your name. If it invents a price or an opening time, the visitor often cannot tell the invented answer from a real one, and your staff hear about it later. Even Google says its own AI Overviews can and will make mistakes (Google Search Help).
What keeps a chatbot on your site in check
No setting removes the risk, but a few limits narrow it. Have the bot answer only from your own pages, so its answers start from what you wrote. When the answer is not there, it should say so and hand the visitor to a person. And it should tell every visitor plainly that it is a bot, so nobody mistakes its answers for your staff's.
What to test before any chatbot goes live
Ask it the questions your customers actually ask, in the words they use, including the awkward ones such as whether you can come out today. Compare each answer with what you would say. Then ask about things your site never mentions, and watch whether it hands off or improvises.
Keep your site current, because a bot trained on your pages repeats them. A service you no longer offer, still described on a page you forgot about, can come back as an answer. The free scan catches one kind of mismatch for you: L1 flags a site whose tappable phone links and structured data give different numbers.
Health and legal practices
Medicine and law need specialist knowledge, the kind of subject where NIST says confabulation is especially likely, and the harm it describes comes when people act on what they are told (NIST). A bot on a practice or firm website should give no medical or legal advice, and should pass those questions straight to a person.
Our work keeps a short list of things that always wait for your sign-off, and this is one of them: "Any medical, legal or financial claim." That covers anything our chatbot is set up to say on those subjects.
Questions
Can a chatbot be set up so it never makes anything up?
No one can promise that, and be wary of anyone who does. NIST treats confabulation as a natural result of how these models work (NIST). You can narrow what the bot may answer, have it hand off when your pages do not cover a question, and test it again whenever your site changes.
Should a chatbot answer questions about price?
Only with prices you have published and approved, in the words you approved. A price the bot works out for itself is a guess that looks like a quote. Anything beyond your published prices, such as the cost of one particular job, should go to a person.
Is a chatbot lying when it hallucinates?
Not in the way a person lies, knowing the truth and saying otherwise. NIST describes a model predicting likely next words, which can come out false while sounding certain (NIST). The customer who acted on the answer is misled all the same, so the limits above still matter.
Where this fits in our work
We handle this as part of AI chatbot: A chatbot trained on your own site that answers, qualifies and hands off to your team, and always says it is a bot.
Related reading
Posts that use this term
- Can I see what ChatGPT or Google's AI says about my business? No assistant shows you the words it used about your business, so you ask. A list of questions to put to each one, and what to record each round.
Related terms
- Chatbot disclosure: Telling visitors they are talking to a bot, what some state laws require, and how to say it plainly.
- Business associate agreement (BAA): The HIPAA contract a covered health practice signs with a vendor that handles its patient information, such as the company behind a patient chatbot.
Sources
- National Institute of Standards and Technology: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1). Checked September 26, 2026.
- Google Search Help: Find information in faster & easier ways with AI Overviews in Google Search. Checked September 26, 2026.