Continuous batching
Serving many users at once by adding and removing requests from a running batch at every generation step.
Also: in-flight batching
Graphics processors are fastest when doing many calculations together, so servers group requests into batches. With simple batching, a batch waits until its longest answer is finished. Continuous batching, described in the Orca paper of 2022, rebuilds the batch at every step: finished requests leave and new ones join immediately. This keeps chips busy and greatly raises throughput without making short answers wait for long ones. It is standard in vLLM, Nvidia's TensorRT-LLM and other serving engines.