What does inference mean, and what determines a model's response time?
Answer out loud or in your head first, then compare.
Answer
Inference means using an already-trained model to get a result. It's a different stage from training, during which the model learns from data. When an app calls a model API, it's usually using inference.
It's not only the network that affects the time to a result. The input length, the output length, the service's load and extra steps, e.g. document retrieval or running tools, all matter. The first fragment can appear quickly while the full task finishes much later. So I measure the time to the first text and the time to a finished, verified result separately.
Why it matters: Helps you understand the latency and costs of an AI product.
Possible follow-ups: Does showing text earlier always shorten the time to complete the task?
How did it go? Be honest — “Not yet” puts the question on your review list.
Further reading
- Microsoft Foundry: Observability in generative AI(opens in a new tab)Docs; accessed 22.10.2026. Evaluation, monitoring and tracing across the solution's lifecycle.