Home » Work » A RAG chatbot that invented answers now cites its source

A RAG chatbot that invented answers now cites its source

A team’s internal document assistant that answered confidently without the facts now cites its source or says it doesn’t know.

Client
A team’s internal document assistant
Stack
  • Python
  • LangChain
  • Hybrid search
  • Reranking
  • RAGAS evaluation
Outcome
Cites its source or says it doesn’t know, with an evaluation set rerun after every change

The problem

The assistant answered from internal documents, and it answered confidently even when the documents didn’t hold the answer.

What I built

I made the system routed and evaluated.

  • Routed. Each question goes to the right retrieval path, or to “I don’t know” when the documents can’t answer it.
  • Better retrieval. Hybrid search with reranking, so the passages behind an answer are the ones that matter.
  • Evaluated. The system is tested against a set of real questions before every release and after every change, with RAGAS.

What changed

A prototype that invented answers became a system that cites its source or says it doesn’t know, with an evaluation set rerun after every change.

I build and fix systems like this. See RAG chatbots and AI agents.

Is your chatbot making things up?

A few lines is enough: what it answers from, and where it goes wrong.

From the $150 scoping session: 60 minutes and a written plan, credited toward the build.