What is drift, and how do you notice quality dropping?
Answer out loud or in your head first, then compare.
Answer
Drift is a change in data or conditions that makes an earlier evaluation stop describing how the system works now. Users may start sending a different type of document, another language or harder tasks, even without the model changing.
So I watch the kinds of inputs, errors, time and quality results on a current sample. When changing provider, I also check how schemas, tools, refusals and streaming are handled. A shared API library doesn't mean the models behave identically. I keep configuration versions and the ability to go back to an earlier one. A rise in technically successful responses alone doesn't prove the content is equally correct.
Why it matters: A product has to work after data and dependencies change too.
Possible follow-ups: What can be spotted automatically, and what needs new human evaluations?
How did it go? Be honest — “Not yet” puts the question on your review list.
Start with
- J-P01 · What is evaluating an AI system, i.e. an eval? — Evaluation, quality and production AI
- J-P02 · How does an eval differ from an ordinary code test? — Evaluation, quality and production AI
Further reading
- Microsoft Foundry: Observability in generative AI(opens in a new tab)Docs; accessed 22.10.2026. Evaluation, monitoring and tracing across the solution's lifecycle.