Does “task completed” in an agent's answer mean success?
Answer out loud or in your head first, then compare.
Answer
No. I have to check the outcome in the environment, not just the claim. If the agent was meant to create a request, I check that the right request exists and has correct data. If it was meant to fix code, I check the app's behaviour and the tests.
A trace of the run helps you understand how it got there, but it can also contain false statements. A valid alternative path isn't wrong just because it differs from the reference. At the same time, a good final result doesn't excuse unauthorised actions along the way. The evaluation should check both what was achieved and whether the required constraints were respected.
Why it matters: The basic difference between evaluating text and evaluating an agent.
Possible follow-ups: How do you check that no extra, unwanted action was performed?
How did it go? Be honest — “Not yet” puts the question on your review list.
Start with
- J-P01 · What is evaluating an AI system, i.e. an eval? — Evaluation, quality and production AI
- J-P02 · How does an eval differ from an ordinary code test? — Evaluation, quality and production AI
Further reading
- Anthropic: Demystifying evals for AI agents(opens in a new tab)09.01.2026. Outcome vs transcript, repeated trials, evaluating the whole agent and the harness.