AI & Agents Engineer
J · Evaluation, quality and production AI
14 questions · 1 coding challenge
14 questions
Coding challenges
Learning paths
- Agents and control over actions · 7 steps
- AI quality and your own projects · 5 steps
Questions
- J-P01What is evaluating an AI system, i.e. an eval?BasicSTART 30
- J-P02How does an eval differ from an ordinary code test?Basic
- J-P03What does LLM-as-a-Judge mean?Basic
- J-P04What do precision and recall mean?Basic
- J01How do you prepare a set of tasks to evaluate an AI system?Advanced
- J02How do you check whether a judging model can be trusted?Advanced
- J04Does “task completed” in an agent's answer mean success?Advanced
- J08How do you choose quality metrics and a decision threshold?Intermediate
- J03How do you tell a real regression from an accidentally worse answer?Advanced
- J05How do you test security when you can't think of every attack?Advanced
- J06How do you leave an audit trail without copying confidential data into every log?Advanced
- J07When do you consider an AI system ready for users?Advanced
- J09What is drift, and how do you notice quality dropping?Advanced
- J10How do you avoid a test that an agent can game instead of solving the task?Advanced