ai insights3 min read
Why our AI assistants are built to say "I don't know"
A confident wrong answer costs more than no answer. Here is how we build assistants that only answer from your documents, show their source, and decline the rest, and how we test that they do.
Most AI chatbots have the same flaw: they answer every question with the same confidence, whether they know the answer or not. In a demo that looks impressive. In a business, it is a liability. A customer who gets a wrong refund policy, or a tenant who gets the wrong notice period, does not care that the AI sounded sure.
So we build assistants the other way round. Here is what that means in practice.
1. Find the passage first, then answer from it
The assistant does not answer from the AI model’s general knowledge. It first searches your own documents for the passages that match the question, using two kinds of search together: one that matches meaning and one that matches exact words. A second model then re-ranks the results so the best passage comes first.
Only then does the AI write an answer, and it is instructed to use those passages and nothing else.
2. Two chances to decline
The first check happens before the AI is called. If even the best matching passage is a poor match, the assistant declines straight away: no AI cost, and no chance to guess. That threshold is measured, not picked by feel: we score real questions and unrelated ones and set the line between them.
The second check is in the AI’s instructions. Some questions match documents on the surface without being answered by them, so the AI is told to decline when the passages it was given don’t contain the answer.
On our Naija Law assistant, which answers from the 1999 Constitution, federal tax law and the Lagos Tenancy Law, that is what happened with all 5 out-of-scope questions in its test set, such as the penalty for armed robbery and the naira exchange rate. They looked close enough to legal text to pass the first check, and the second check declined them.
3. Citations copied from the source, not typed by the AI
Every claim in an answer links to the exact passage it came from. The quote is copied from the document by the system, not written by the AI, so it cannot be invented. In testing, 11 out of 11 citations on Naija Law, and 18 out of 18 on our Helpdesk assistant, matched the source text exactly.
Copying the quote mechanically also makes problems visible. We noticed some quotes began mid-word (“onths notice…”), because the documents had been cut into chunks by character count. We now cut on whole sentences, and every quote starts cleanly. The test scores did not change, but the answers read better, and it was the citations that showed us the problem.
4. A miss you can explain
One answerable question in the Naija Law test set was declined: the education requirement to be elected President. The Constitution uses the same wording for four different offices, and the assistant would not risk attributing the wrong one. We count that as a miss in the scores, but for a legal tool it is the right behaviour.
That is the point of testing: you see exactly where a system is cautious and where it is confident, before your customers do.
What this means for your business
If you are considering an AI assistant for customer questions, staff questions or internal documents, ask three things of whoever builds it:
- What happens when the answer is not in the documents? It should decline, not guess.
- Can I see where each answer came from? Every answer should show its source.
- What were the test results? You should see a written test set and the scores before launch.
Every assistant we build meets all three. You can try two of them right now: Naija Law RAG and Helpdesk RAG.