What do you do when jobs arrive faster than the system can process them?
Answer out loud or in your head first, then compare.
Answer
You need to limit the inflow or the processing rate, not just keep adding new attempts. A queue can hold a temporary surplus, but if I accept more than I process all day, the backlog will keep growing. I watch the number of jobs and the age of the oldest waiting one.
With an AI model, the constraint may be the number of tokens processed per unit of time. More workers won't raise the provider's limit and may cause more 429 errors. I introduce per-user limits and priorities. An import of thousands of documents shouldn't block all the employees' short questions. This is practical control of the inflow of work, called backpressure.
Why it matters: Shows why a queue isn't unlimited system capacity.
Possible follow-ups: When do you have to refuse a new job or defer it?
How did it go? Be honest — “Not yet” puts the question on your review list.
Start with
- D-P01 · What is HTTP, and what do a request and a response look like? — Backend, data and system design
- D-P10 · What is a database transaction? — Backend, data and system design
- D-P13 · What is a message queue, and why do we use one? — Backend, data and system design
Further reading
- Node.js: Backpressuring in Streams(opens in a new tab)Docs; accessed 22.10.2026. Flow, drain and limiting memory use.