How do you tell whether retrieval or answer generation is at fault?
Answer out loud or in your head first, then compare.
Answer
I check separately what was found and what the model did with it. If the right passage isn't in the material passed on, I start with document fetching, chunking and retrieval. If it is, and the answer is still wrong, I look at the instruction and the model's behaviour.
I prepare questions with the needed documents marked. I can count what share of the important passages made it into the top results. A test with hand-picked correct context also helps: if the answer is still wrong, the problem isn't only retrieval. I judge the answer on truthfulness and agreement with the sources. The tests include questions with no answer, too.
Why it matters: A key skill in diagnosing RAG.
Possible follow-ups: What if the answer needs several different documents, but only one was found?
How did it go? Be honest — “Not yet” puts the question on your review list.
Start with
- G-P01 · What is RAG? — RAG and retrieval
- G-P02 · What are embeddings? — RAG and retrieval
Practise in code
- Does the search find the right material? — runs tests in your browser
Further reading
- Anthropic: Demystifying evals for AI agents(opens in a new tab)09.01.2026. Outcome vs transcript, repeated trials, evaluating the whole agent and the harness.
- pgvector: official repository and documentation(opens in a new tab)Docs; accessed 22.10.2026. Exact vs ANN search, indexes, and how filters affect search.