What should a website chatbot do when it doesn't know the answer?

A chat widget does not say "I think". It states your prices, your refund policy and your opening hours in your business's name, and a visitor cannot tell a confident wrong answer from a right one. The review pages that rank for "best AI chatbot" list "occasionally incorrect responses" as a downside and move on. This post covers how a website chatbot decides what to say, where it goes wrong, and what to check.

How does a website chatbot decide what to say?

It does not learn your website the way a new employee would. When a visitor asks a question, the tool searches the pages it has read for the passages closest in meaning to the question. It then hands those passages to a language model with instructions to answer from them.

"Trained on your website" in marketing copy usually means this search step. That is good news for you: when you change a page, the answers can change after the tool next reads the site, and nothing has to be retrained.

Where does it go wrong?

In two places. The search can hand over the wrong passages, or none that are useful. Or the model can write something the passages do not say.

The second one is what people call hallucination. A language model produces the most likely-sounding reply, and if nobody has told it to stop when the passages do not cover the question, it answers from general knowledge. "Our standard package is $35 a month" sounds likely whether or not your site says it.

What five things should a good chatbot do when it doesn't know?

1. Drop weak matches. A relevance cut-off throws away passages that are not close enough to the question, so a loose match does not become a confident answer.
2. Say so. When nothing passes the cut-off, it says it is not sure and offers a person or a contact form, instead of answering from general knowledge.
3. Show its sources. Links under the answer to the pages it used, so the visitor and you can check.
4. Tell you what it could not answer. A list of unanswered questions, so you can add the missing information to your site.
5. Take corrections. You can fix a wrong answer and see the fix used the next time someone asks.

Vendors describe these in different words, so test them rather than asking. Questions 16 to 25 of our 30-question chatbot test are built for this: prices, areas and discounts your site does not mention, and risky questions it should hand to a person.

How does AI Chat for Website handle it?

For each question it takes the six passages from your site that are closest in meaning and drops any that are not close enough. If none pass, the model is told exactly that: nothing on the website matched the question.

The model is instructed that if the passages do not cover the question, it should say it is not sure, offer to pass the question to your team, and report the question as a gap.

Every answer then goes through a check in code before the visitor sees it. Links the model was not shown are removed. An answer that repeats its own instructions, contains HTML, links off your site or runs too long is withheld, and the visitor sees this instead: "I couldn't put together a reliable answer to that. Try asking another way, or ask for a person." You can read the withheld text in your transcripts.

Under an answer, the widget shows up to three "From:" links: the pages that were retrieved for that question. Questions it could not answer appear in your Gaps list. From there you can add the missing page or write the answer yourself, and "Fix this answer" on any transcript saves your correction as a question-and-answer pair.

Messages that are not about your business get a fixed reply ("I can only help with questions about [your business]"). They never reach the answering model. An attempt to change its instructions gets "I can't change how I work", and the third attempt ends the chat.

What can't it guarantee?

The check in code does not verify facts. It catches links, leaked instructions, markup and length, not a wrong sentence. So the accurate claim is that the assistant is told not to guess and cannot link outside your site. It does not follow that it never gets anything wrong.

A question-and-answer pair you write is used when it is retrieved for a similar question, and it gets a small head start over web pages in that search. It is not a hard override. The "From:" links show the pages that were retrieved, which is not proof that the model used every one of them.

That is why we publish the test rather than a promise. Run it on us and on anything else you are considering.

What can I do to cut wrong answers?

• Put the facts visitors ask about on pages the tool reads: prices, the area you serve, hours, refund terms. If a fact is only in your head, the bot cannot have it.
• Before going live, ask five questions your site does not answer and read every reply.
• Check the list of unanswered questions every week for the first month, and answer each one on your site or directly in the tool.
• Re-test after big site changes, and keep the old answers so you can compare.