The problem
The assistant answered from internal documents, and it answered confidently even when the documents didn’t hold the answer.
What I built
I made the system routed and evaluated.
- Routed. Each question goes to the right retrieval path, or to “I don’t know” when the documents can’t answer it.
- Better retrieval. Hybrid search with reranking, so the passages behind an answer are the ones that matter.
- Evaluated. The system is tested against a set of real questions before every release and after every change, with RAGAS.
What changed
A prototype that invented answers became a system that cites its source or says it doesn’t know, with an evaluation set rerun after every change.
I build and fix systems like this. See RAG chatbots and AI agents.
Is your chatbot making things up?
A few lines is enough: what it answers from, and where it goes wrong.
From the $150 scoping session: 60 minutes and a written plan, credited toward the build.