What is evaluating an AI system, i.e. an eval?
Answer out loud or in your head first, then compare.
Answer
It's checking how well a system handles prepared tasks against agreed criteria. I give it sample inputs, collect the results and judge them: are they correct, useful and safe? The impression from one successful conversation isn't enough.
If the system extracts data from invoices, I can compare the result with the correct values across many documents. If it answers questions, I check that the answer follows from the sources. Some judgements are done by code, some by a person, and sometimes by a helper model. A good set also contains hard cases and ones where the system should say it has no grounds to answer.
Why it matters: The offer requires measuring quality, not just calling an API.
Possible follow-ups: How do you avoid evaluating only on easy examples?
How did it go? Be honest — “Not yet” puts the question on your review list.
Further reading
- Claude Platform: Define success criteria and build evaluations(opens in a new tab)Docs; accessed 22.10.2026. Success criteria, representative data and kinds of graders.