How do you check whether a judging model can be trusted?
Review note: The answer refers to the original author's CV or target job offer — adapt it to your own experience.
Answer out loud or in your head first, then compare.
Answer
I give it clear criteria and compare its scores with human scores on examples similar to the target tasks. I look at where they agree and which mistakes repeat. It's not enough that the model sounds confident or gives a similar average.
It may prefer a longer answer, a certain style or the first of two candidates. When comparing, you can swap the order and hide the author. Clear-cut conditions, such as a maximum amount or access, I check with code. If one answer violates security, a nice style shouldn't lift it to a “good” score. Changing the judging model also requires re-checking its behaviour.
Why it matters: Teaches the limits of the LLM-as-a-Judge approach from your CV.
Possible follow-ups: Why can a weighted average hide an error that should block everything?
How did it go? Be honest — “Not yet” puts the question on your review list.
Start with
- J-P01 · What is evaluating an AI system, i.e. an eval? — Evaluation, quality and production AI
- J-P02 · How does an eval differ from an ordinary code test? — Evaluation, quality and production AI
Further reading
- Claude Platform: Define success criteria and build evaluations(opens in a new tab)Docs; accessed 22.10.2026. Success criteria, representative data and kinds of graders.