How do you tell a real regression from an accidentally worse answer?
Answer out loud or in your head first, then compare.
Answer
A regression is a deterioration after a change, e.g. a new prompt or model. Since results vary, I compare versions on the same tasks and run enough attempts. I look at the spread, the kinds of errors and how significant the change is, not just one result.
If the new version phrases one sentence slightly worse, that's weak evidence. If it revealed someone else's document, I don't wait for the average to become statistically clearly worse. That's a separate serious problem. I set the criteria and acceptable differences in advance. I don't endlessly tweak the prompt on the final test set, because it would stop testing behaviour on new cases.
Why it matters: Lets you judge changes without over-trusting a single run.
Possible follow-ups: What if one group of tasks improved and another got worse?
How did it go? Be honest — “Not yet” puts the question on your review list.
Start with
- J-P01 · What is evaluating an AI system, i.e. an eval? — Evaluation, quality and production AI
- J-P02 · How does an eval differ from an ordinary code test? — Evaluation, quality and production AI
Further reading
- Anthropic: Demystifying evals for AI agents(opens in a new tab)09.01.2026. Outcome vs transcript, repeated trials, evaluating the whole agent and the harness.